bug(table): filters on evolved columns ignore initial-default for old files

[mattfaltyn]

Apache Iceberg version

main at 9097100f4aa3bae419644e3b1f25d3fe29ce8fb7. The affected code is also present in v0.6.0.

Please describe the bug

A filtered Arrow scan can silently drop rows written before a column was added, even when that column's InitialDefault satisfies the filter.

For example, start with a v3 table containing two rows, add optional new_col int with an initial default of 42, and scan the old file:

• An unfiltered projection returns both rows with new_col = 42, as expected.
• A scan filtered by new_col == 42 returns no rows and no error.
• PlanFiles retains the old data file, so the loss occurs during read-time filtering rather than file planning.

Expected: both old rows are returned because their projected value is 42.

Actual: the result is empty.

The following package-local regression test reproduces the problem using the existing buildV3TableWithRows helper:

func collectInitialDefaultScan(
	t *testing.T,
	scan *Scan,
	tasks []FileScanTask,
) (ids []int64, defaults []int32) {
	t.Helper()

	_, records, err := scan.ReadTasks(t.Context(), tasks)
	require.NoError(t, err)

	for rec, recErr := range records {
		require.NoError(t, recErr)
		require.NotNil(t, rec)
		if rec.NumRows() > 0 {
			idCol := rec.Column(0).(*array.Int64)
			defaultCol := rec.Column(1).(*array.Int32)
			for i := range int(rec.NumRows()) {
				ids = append(ids, idCol.Value(i))
				defaults = append(defaults, defaultCol.Value(i))
			}
		}
		rec.Release()
	}

	return ids, defaults
}

func TestInitialDefaultFilterDropsOldFileRepro(t *testing.T) {
	tbl := buildV3TableWithRows(t, `[{"id":1,"data":"a"},{"id":2,"data":"b"}]`)

	txn := tbl.NewTransaction()
	require.NoError(t, txn.UpdateSchema(true, false).
		AddColumn(
			[]string{"new_col"},
			iceberg.PrimitiveTypes.Int32,
			"",
			false,
			iceberg.Int32Literal(42),
		).
		Commit())
	var err error
	tbl, err = txn.Commit(t.Context())
	require.NoError(t, err)

	control := tbl.Scan(WithSelectedFields("id", "new_col"))
	controlTasks, err := control.PlanFiles(t.Context())
	require.NoError(t, err)
	require.Len(t, controlTasks, 1)
	controlIDs, controlDefaults := collectInitialDefaultScan(t, control, controlTasks)
	require.Equal(t, []int64{1, 2}, controlIDs)
	require.Equal(t, []int32{42, 42}, controlDefaults)

	matching := tbl.Scan(
		WithSelectedFields("id", "new_col"),
		WithRowFilter(iceberg.EqualTo(iceberg.Reference("new_col"), int32(42))),
	)
	matchingTasks, err := matching.PlanFiles(t.Context())
	require.NoError(t, err)
	require.Len(t, matchingTasks, 1, "planning retains the old file")
	matchingIDs, matchingDefaults := collectInitialDefaultScan(t, matching, matchingTasks)

	t.Logf("control IDs=%v defaults=%v", controlIDs, controlDefaults)
	t.Logf("matching-filter IDs=%v defaults=%v", matchingIDs, matchingDefaults)
	require.Equal(t, controlIDs, matchingIDs,
		"every old row projects new_col=42 and must satisfy new_col == 42")
	require.Equal(t, controlDefaults, matchingDefaults)
}

Required imports:

import (
	"testing"

	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/iceberg-go"
	"github.com/stretchr/testify/require"
)

Run:

go test ./table -run '^TestInitialDefaultFilterDropsOldFileRepro$' -count=2 -v

Observed on both runs:

control IDs=[1 2] defaults=[42 42]
matching-filter IDs=[] defaults=[]

Root cause

TranslateColumnNames treats a field missing from the file schema as null: it folds IsNull to true and every other predicate to AlwaysFalse, without consulting the evolved field's InitialDefault.

getRecordFilter turns that AlwaysFalse into dropFile = true, and this happens before projection. Projection later materializes the correct InitialDefault, but the file has already been discarded.

Suggested fix

When translating a bound predicate for a field absent from the file schema, evaluate the predicate against the field's InitialDefault and fold it to the corresponding constant result. Preserve the existing null behavior only when no non-null initial default applies.

A regression matrix should cover matching and non-matching predicates, null predicates, and both old and newly written files.

Willingness to contribute

• I can contribute a fix for this bug independently