Skip to content

[Json] Support FixedSizeList in json decoder - #9715

Merged
alamb merged 2 commits into
apache:mainfrom
liamzwbao:issue-9714-fixed-size-list-decoder
Apr 21, 2026
Merged

[Json] Support FixedSizeList in json decoder#9715
alamb merged 2 commits into
apache:mainfrom
liamzwbao:issue-9714-fixed-size-list-decoder

Conversation

@liamzwbao

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Implement decoder for FixedSizeList

Are these changes tested?

Yes

Are there any user-facing changes?

New decoding type supported

@github-actions github-actions Bot added the arrow Changes to the arrow crate label Apr 14, 2026
@liamzwbao
liamzwbao force-pushed the issue-9714-fixed-size-list-decoder branch from 58a95fc to 9ed71fc Compare April 14, 2026 15:53
@liamzwbao
liamzwbao marked this pull request as ready for review April 14, 2026 15:53

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a fun little degenerate test case here:

    #[test]
    fn test_fixed_size_list_zero_size() {
        let buf = r#"
        {"a": []}
        {"a": null}
        {"a": []}
        "#;

        let field = Field::new_list_field(DataType::Int32, true);
        let schema = Arc::new(Schema::new(vec![Field::new(
            "a",
            DataType::FixedSizeList(Arc::new(field), 0),
            false,
        )]));

        let batches = do_read(buf, 1024, false, false, schema);
        assert_eq!(batches.len(), 1);

        let col = batches[0].column(0).as_fixed_size_list();
        assert_eq!(col.len(), 3);
        assert_eq!(col.value_length(), 0);

        let values = col.values().as_primitive::<Int32Type>();
        assert!(values.values().is_empty());
    }

It seems to panic within struct array decoding. Could be worth exploring in this PR if there is capacity, though considering this PR is adding a new feature I wouldn't block it on this case.


let mut nulls = self
.is_nullable
.then(|| BooleanBufferBuilder::new(pos.len()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth considering using NullBufferBuilder here instead?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb

alamb commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

It seems to panic within struct array decoding. Could be worth exploring in this PR if there is capacity, though considering this PR is adding a new feature I wouldn't block it on this case.

Filed a ticket to track

@alamb
alamb merged commit f73922b into apache:main Apr 21, 2026
23 checks passed
@alamb

alamb commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Thank you @liamzwbao and @Jefffrey

@liamzwbao

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @Jefffrey and @alamb! Was a bit occupied last week, will address the feedbacks in a followup PR

@alamb

alamb commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Thank you ! I was trying to finalize the release content and keep things moving.

@liamzwbao
liamzwbao deleted the issue-9714-fixed-size-list-decoder branch April 24, 2026 21:25
Rich-T-kid pushed a commit to Rich-T-kid/arrow-rs that referenced this pull request Jun 2, 2026
# Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax.
-->

- Closes apache#9714.

# Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

# What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

Implement decoder for FixedSizeList

# Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Yes

# Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please call them out.
-->

New decoding type supported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support FixedSizeList in arrow-json reader

3 participants