Skip to content

Wrong result: grouped MAX(NULL) row is dropped with ORDER BY aggregate NULLS FIRST LIMIT #23440

Description

@Fly-a-Kite

Describe the bug

DataFusion 54.0.0 returns an empty result for a grouped aggregate query where the aggregate value is NULL, followed by ORDER BY ... NULLS FIRST LIMIT.

The grouped aggregate should still produce one output row. The final ORDER BY / LIMIT should not remove it.

This looks related to #22190, but I can still reproduce it on the latest datafusion==54.0.0 with a smaller reproducer.

To Reproduce

  import datafusion
  import pyarrow as pa
  from datafusion import SessionContext

  print("datafusion", datafusion.__version__)

  batch = pa.record_batch(
      [
          pa.array(["gamma"], type=pa.string()),
          pa.array([None], type=pa.float64()),
      ],
      names=["s", "y"],
  )

  ctx = SessionContext()
  ctx.register_record_batches("t0", [[batch]])

  query = """
  SELECT max_y
  FROM (
    SELECT s, MAX(y) AS max_y
    FROM t0
    GROUP BY s
  )
  ORDER BY max_y DESC NULLS FIRST
  LIMIT 3
  """

  result = ctx.sql(query).to_pandas()
  print(result)
  print("row count:", len(result))

### Expected behavior

  ### Expected behavior

  The query should return one row:

     max_y
  0   NULL

  Reason: the input contains one group, s = 'gamma'. MAX(y) over a group where all y values are NULL should produce one grouped output row with max_y = NULL.

  ### Actual behavior

  DataFusion 54.0.0 returns zero rows:

  Empty DataFrame
  Columns: [max_y]
  Index: []
  row count: 0

### Additional context

 ### Environment

  - datafusion==54.0.0
  - pyarrow==24.0.0
  - Python 3.12.3
  - Linux x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions