Skip to content

fix: support parquet floating point columns as varchar - #814

Merged
guhaiyan0221 merged 1 commit into
bytedance:mainfrom
guhaiyan0221:fix_double_to_varchar
Aug 3, 2026
Merged

fix: support parquet floating point columns as varchar#814
guhaiyan0221 merged 1 commit into
bytedance:mainfrom
guhaiyan0221:fix_double_to_varchar

Conversation

@guhaiyan0221

@guhaiyan0221 guhaiyan0221 commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Allow REAL and DOUBLE Parquet columns to be read as VARCHAR. For ordinary Hive Parquet splits, reject incompatible VARCHAR value filters before they reach floating-point readers. When only metadata pruning is involved, disable the metadata filter and evaluate the residual predicate after conversion. Keep IS NULL and IS NOT NULL pushdown unchanged, and exclude Paimon splits from this handling.

What problem does this PR solve?

Issue Number: close #796

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 🚀 Performance improvement (optimization)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
  • 🔨 Refactoring (no logic changes)
  • 🔧 Build/CI or Infrastructure changes
  • 📝 Documentation only

Description

  • Support reading Parquet FLOAT and DOUBLE columns as VARCHAR.
  • Reject incompatible VARCHAR value filters before they reach floating-point readers.
  • Skip incompatible metadata-filter leaves while preserving null checks and compatible row-group pruning.
  • Cover scalar, filter-only, nested map, and metadata-filter scenarios.

Performance Impact

  • No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).

  • Positive Impact: I have run benchmarks.

    Click to view Benchmark Results
    Paste your google-benchmark or TPC-H results here.
    Before: 10.5s
    After:   8.2s  (+20%)
    
  • Negative Impact: Explained below (e.g., trade-off for correctness).

Release Note

Please describe the changes in this PR

Release Note:

Release Note:
- fix: support parquet floating point columns as varchar

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Breaking Changes

  • No

  • Yes (Description: ...)

    Click to view Breaking Changes
    Breaking Changes:
    - Description of the breaking change.
    - Possible solutions or workarounds.
    - Any other relevant information.
    

Allow REAL and DOUBLE Parquet columns to be read as VARCHAR. For ordinary Hive Parquet splits, reject incompatible VARCHAR value filters before they reach floating-point readers. When only metadata pruning is involved, disable the metadata filter and evaluate the residual predicate after conversion. Keep IS NULL and IS NOT NULL pushdown unchanged, and exclude Paimon splits from this handling.

@zhangxffff zhangxffff left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@guhaiyan0221
guhaiyan0221 added this pull request to the merge queue Aug 3, 2026
Merged via the queue into bytedance:main with commit 2617ce8 Aug 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] schema mismatch error raised during runtime

3 participants