Intake wdl

[piotor87]

• removed a bunch of legacy folders from wdl since we have releases where one can look
• edited intake to now process all columns (we will decide later what to keep anyways if needed)
• change _rowid to ROWID since it will be already that downstream anyways
• added intake.wdl/json to process the data and outputs parquet +tsv.gz file

[vincent-octo]

Looks good at this point. :)

I will add a command to run the whole thing end-to-end in order to keep the WDL small.

One thing we should discuss at some point: should we just keep _rowid_source instead of ROWID. The advantage of _rowid_source is that it's a direct lookup in the intermediate files generated.

[piotor87]

These are the changes

src/kanta/intake/tidyup.py — the actual OOM fix
  - The old code built one big lazy plan (join SEX + with_row_index + sanitize + sink_parquet) over all buckets concatenated together.
  maintain_order="left" on the join and the global counter in with_row_index both disable Polars' streaming engine, forcing full materialization in
  memory.
  - Now each bucket is joined, indexed (with a running ROWID offset carried across buckets), sanitized, and sunk to its own temp parquet file individually
  — bounded memory per bucket regardless of total dataset size.
  - Final buckets (and duplicate buckets) are concatenated into the output file via a new concatenate_parquet_files helper that streams row batches
  through pyarrow directly, since sink_parquet over a multi-file scan_parquet isn't reliably streaming either.
  - Docstring note updated to mention the join+sanitize stage also processes one bucket at a time now.
 
  docker/Dockerfile
  - Bumped Python 3.12 → 3.14.
  - Replaced manual uv venv + uv pip install with uv sync --python 3.14 --locked (via UV_PROJECT_ENVIRONMENT), so the venv is built straight from the
  lockfile.

[vincent-octo]

This line

output.check_safe_write(args.output_file.with_name(f"{args.output_file.stem}_duplicates.parquet"))

should also be added to

kanta_lab_preprocessing/src/kanta/intake/__main__.py

Lines 54 to 56 in 9bf7288

	output.check_safe_write(output_file_assemble_stage)
	output.check_safe_write(output_file_tidyup_stage)

[piotor87]

Ok i can add it

    output.check_safe_write(output_file_assemble_stage)
    output.check_safe_write(output_file_tidyup_stage)
    output.check_safe_write(
        output_file_tidyup_stage.with_name(f"{output_file_tidyup_stage.stem}_duplicates.parquet")
    )

[piotor87]

I've also split the tsv generation in the wdl because it made sense structurally. Otherwise I think we're good to go