feat(databricks): create tables with IF NOT EXISTS and reconcile columns under concurrent loads

[mucio-at-untitled-data-company]

Feature description

When several dlt runs load into the same Databricks table concurrently and the table doesn't exist yet, each run independently decides the table is new and emits a plain CREATE TABLE. The first run wins; the others fail with "table already exists". dlt should create user tables with CREATE TABLE IF NOT EXISTS on Databricks and reconcile columns so concurrent runs converge.

Are you a dlt user?

Yes, I run dlt in production.

Use case

Parallel orchestration (multiple Airflow tasks, parallel pipeline processes, partitioned backfills) regularly lands several runs on the same not-yet-existing table at once. The DDL phase races:

1. Every run calls get_storage_tables() → table missing → generate_alter = False (dlt/destinations/job_client_impl.py:688).
2. Every run emits CREATE TABLE foo (...).
3. First commit wins; the rest raise "table already exists" and the load fails.

A subtler second bug hides behind the crash: a run carrying an extra column c would see its create become a no-op against the table another run already made, then fail when its load job writes c — a column that was never added.

Proposed solution

Scoped to the Databricks destination:

1. Emit CREATE TABLE IF NOT EXISTS for user tables. Today IF NOT EXISTS is applied only to dlt internal tables in _make_create_table (dlt/destinations/job_client_impl.py:715), gated by the existing supports_create_table_if_not_exists capability.
2. Reconcile columns: after the (possibly no-op) create, re-read the actual table columns and emit ALTER TABLE … ADD COLUMN for any missing ones. Databricks has no ADD COLUMN IF NOT EXISTS, so reconciliation must re-read storage and tolerate the "column already exists" error two runs can hit when adding the same column concurrently.

This keeps the common identical-schema case race-free and lets divergent-schema runs converge.

Related issues

None known.