go/adbc/driver/bigquery: HTTP 409 on job submission should attach to existing job instead of erroring

[will-sargent-dbtlabs]

Summary

When BigQuery returns HTTP 409 Already Exists during job submission, the Go BigQuery ADBC driver surfaces the error to the caller. The correct behaviour is to attach to the already-created job (using the driver's own job ID) and read its results — the same job ran successfully, we just need to reconnect to it.

This matters whenever job submission is retried after a transient network timeout: the server created the job on the first attempt, but the client never received the 200 OK. A subsequent query.Run() with the same job ID hits the 409 path and the driver has no recovery.

Affected code

go/adbc/driver/bigquery/record_reader.go — runQuery

job, err := query.Run(ctx)
if err != nil {
    return nil, -1, err   // ← 409 falls through here unchanged
}

Expected behaviour

On a *googleapi.Error with .Code == 409, call client.JobFromIDLocation(ctx, query.JobID, query.Location) to attach to the existing job and continue normally.

Second issue: job ID is not refreshed per execution

runQuery previously only assigned a driver-owned UUID when query.JobID == "":

if query.JobID == "" {
    query.JobID = "adbc-" + uuid.NewString()
}

A *bigquery.Query is reused for each row when bound parameters are in play (queryRecordWithSchemaCallback). Because query.JobID is non-empty after the first execution, subsequent rows reuse the same ID — which either collides with the live job or makes the 409 recovery attach to the wrong execution.

The fix is to assign a fresh UUID unconditionally at the start of every runQuery call.

Proposed fix

func runQuery(ctx context.Context, client *bigquery.Client, query *bigquery.Query, ...) (...) {
    // Fresh ID per execution — safe for bound-parameter reuse.
    query.JobID = "adbc-" + uuid.NewString()

    job, err := query.Run(ctx)
    var apiErr *googleapi.Error
    if errors.As(err, &apiErr) && apiErr.Code == http.StatusConflict {
        job, err = client.JobFromIDLocation(ctx, query.JobID, query.Location)
    }
    if err != nil {
        return nil, -1, err
    }
    // ... rest unchanged
}

Threading *bigquery.Client through runQuery / runPlainQuery / queryRecordWithSchemaCallback / newRecordReader is required because the recovery call needs the client directly.

A regression test (TestRunQueryRecoversExistingJobAfterDuplicateInsert) uses an httptest.Server that always returns 409 on submission, then serves the job on the subsequent GET, and verifies two consecutive executions each produce distinct job IDs and both return results.

Fix in dbt-labs fork

A patch implementing this fix (including the regression test) is available at:
dbt-labs#153

Happy to port it upstream as a PR against this repo if that would be helpful.

Environment

• Go BigQuery ADBC driver (go/adbc/driver/bigquery)
• Reproducible via the httptest-based unit test in the patch above (no real GCP credentials needed)

[lidavidm]

The BigQuery driver is no longer maintained in this repo. The Driver Foundry maintains it instead; do you mind filing it there?

[will-sargent-dbtlabs]

Thanks for the pointer! Refiled at adbc-drivers/bigquery#247.