Expose Parquet statistics

[kylebarron]

Ideally we'll expose statistics as Arrow; this is much simpler to use. Ref #313

However, it doesn't yet work for struct columns, ref apache/arrow-rs#7364

    #[pyo3(signature = (column_name, *, missing_null_counts_as_zero=true))]
    fn statistics(
        &self,
        column_name: &str,
        missing_null_counts_as_zero: bool,
    ) -> Arro3IoResult<Arro3RecordBatch> {
        let parquet_meta = self.meta.metadata();
        let converter = StatisticsConverter::try_new(
            column_name,
            self.meta.schema(),
            self.meta.parquet_schema(),
        )?
        .with_missing_null_counts_as_zero(missing_null_counts_as_zero);

        let min_values = converter.row_group_mins(parquet_meta.row_groups())?;
        let max_values = converter.row_group_maxes(parquet_meta.row_groups())?;
        let null_counts = converter.row_group_null_counts(parquet_meta.row_groups())?;

        let schema = Arc::new(Schema::new(vec![
            Field::new("min", min_values.data_type().clone(), true),
            Field::new("max", max_values.data_type().clone(), true),
            Field::new("null_count", null_counts.data_type().clone(), true),
        ]));
        let record_batch =
            RecordBatch::try_new(schema, vec![min_values, max_values, Arc::new(null_counts)])?;
        Ok(record_batch.into())
    }

    fn page_statistics(&self) {
        todo!()
    }

    def statistics(
        self, column_name: str, *, missing_null_counts_as_zero: bool = True
    ) -> core.RecordBatch: ...

[yohplala]

Hello,
I have a half related question.
Please, is there a way to retrieve per row group number of rows and filepath / filename?
Albeit defined per row group, this data is not stored in statistics as far as i know.

[kylebarron]

Only once you've read the row groups as Arrow.

[yohplala]

Hello,
Exposing statistics would be of noticable interest in a new project of mine. Please, do you have some idea when it could be made available in arro3 project?
Thanks for your reply, best regards,

[kylebarron]

I don't have any active plans to implement this, but we could discuss a PR if you were interested in submitting one

[yohplala]

thanks for your reply, I will try to have a look.

[yohplala]

I have had Claude working out an implementation of this ready on a fork (yohplala:claude/arro3-feature-planning-jet5w) that follows the prototype in the issue body: ParquetFile.open(path).statistics(column_name, *, missing_null_counts_as_zero=True) returning an Arrow RecordBatch with min / max / null_count columns, one row per row group. It uses ArrowReaderMetadata::load + StatisticsConverter (parquet 58), with no new dependencies.

Before opening a PR, since #313 is also adding a ParquetFile class, would you prefer this land as:

1. A standalone PR against main that introduces a minimal ParquetFile (just open, num_rows, num_row_groups, num_columns, schema_arrow, __repr__, and statistics) — and #313 adds the async / row-filter surface on top when it lands, or
2. A follow-up commit on top of kyle/parquet-api-updates that just adds the statistics method to your existing ParquetFile, or
3. Something else entirely (e.g., a free function rather than a class).

Some parts of #313 have been deliberately mirrored (meta: ArrowReaderMetadata, module = "arro3.io", schema_arrow getter name, #[classmethod] fn open(...), subclass, frozen) to keep the merge path short in case you pick option 1.