Materialize typed Parquet datasets with provenance

[JacobLinCool]

Renamed by #23. This issue was written while the project was called Android Data
Collector. Its terminology has been updated to the Particeps identity now in force. The
old spellings are rejected inputs, not an earlier dialect.

Summary

Materialize the validated event stream from particeps-analysis as deterministic, strictly typed Parquet datasets with complete source provenance.

This issue implements Parquet only. It defines a narrow DatasetSink extension point for later work but does not add Postgres, D1, dashboards, or any other database connector.

Requirements

Strict schema materialization

• Consume only the validated/reassembled stream from Build offline bundle inventory, verification, and reassembly tooling #13.
• Resolve every collector, payload type, field, type, unit, and nullability rule through the registry in Create a machine-readable collector and event-schema registry #10.
• Convert Protocol v1 decimal-string 64-bit values to lossless Arrow/Parquet integer types after strict bounds checking.
• Parse enumerations, booleans, integers, fixed binary, and strings according to the declared schema; do not infer types from observed values.
• Reject/quarantine unknown collector or payload schemas, undeclared fields, invalid enums, lossy values, unit mismatches, and nullability violations. Do not emit a generic map/JSON fallback column.
• Preserve wall time, continuous/monotonic time, time-zone context, boot-session identity, sequence numbers, and producer metadata at full declared precision.

Dataset layout and provenance

• Partition safely and deterministically by experiment, configuration, collector, schema, and payload type, with path-safe encodings that cannot traverse the output root.
• Within files, use stable ordering and a documented row-group/file sizing policy.
• Make each row or row group traceable to:
  • source ciphertext SHA-256;
  • bundle ID;
  • configuration digest;
  • parser/catalog version;
  • source object/path;
  • identical-duplicate source set where applicable.
• Keep sensitive identifiers inside the protected dataset and out of filenames unless the documented data-handling policy explicitly permits them.
• Emit a dataset manifest containing schema fingerprints, partitions, row/file counts, source inventory digest, tool versions, conflicts/quarantines, and reproducibility metadata.
• Define a typed DatasetSink interface around validated records, but implement only the Parquet sink in this issue.

Quality summaries

Produce offline, evidence-bounded summaries for:

• sequence gaps, overlaps, identical duplicates, and conflicts;
• boot sessions and incomparable monotonic intervals;
• wall-clock and time-zone changes;
• requested versus achieved sampling rates where the configuration and data support the calculation;
• collector start/stop/access/quota states;
• intervention/survey occurrence lifecycle.

Do not infer participant awareness, notification visibility, travel, presence, orientation, activity, or other behavior the recorded events cannot prove.

Acceptance criteria

• Every shipped registry schema has deterministic Arrow/Parquet schema fixtures.
• Unknown schemas or invalid typed values fail closed without a generic fallback partition.
• Full-range supported 64-bit sequence/time values round-trip without floating-point conversion or precision loss.
• Identical validated inputs, catalog version, and options produce identical logical schemas, manifests, partition paths, ordering, and quality summaries.
• Any output row can be traced to its source ciphertext, bundle, configuration, parser/catalog version, and source object.
• Local and R2-derived validated streams produce equivalent Parquet datasets.
• Conflicts and quarantined bundles are reported and never silently materialized as authoritative rows.
• The shipped package has no database client or D1/Postgres-specific implementation.

Documentation

Document Parquet schemas and partitioning, provenance columns/sidecars, manifest format, quality-metric definitions and limits, secure output handling, reproducibility workflow, and the future DatasetSink contract.

Dependencies

Blocked by #10 (schema registry) and #13 (validated reassembly stream).