Skip to content

Fix kafka recipe: correct federated-query timestamps misread as nanoseconds - #558

Open
claudespice wants to merge 1 commit into
spiceai:trunkfrom
claudespice:fix/kafka-to-timestamp-output
Open

Fix kafka recipe: correct federated-query timestamps misread as nanoseconds#558
claudespice wants to merge 1 commit into
spiceai:trunkfrom
claudespice:fix/kafka-to-timestamp-output

Conversation

@claudespice

Copy link
Copy Markdown
Contributor

Summary

The Federated Query Example in the kafka recipe selects TO_TIMESTAMP(o.order_ts) as timestampt, but its documented output shows every row at 1970-01-01T00:00:01.75600827x — epoch-second values being read as nanoseconds.

  • What the recipe said: timestampt renders as 1970-01-01T00:00:01.756008276.
  • What the code does: producer.py sets order_ts = time.time() — a float of epoch seconds. DataFusion 54's to_timestamp documents that "Integers, unsigned integers, and doubles are interpreted as seconds since the unix epoch", and the Float64 arm multiplies by 1_000_000_000.0 in both the scalar and array branches. So 1756008276.x renders as 2025-08-24, not 1970.
  • What changed: the timestampt column in that one output block now shows the correctly-interpreted timestamps, derived from the same epoch values already encoded in the stale output (17560082762025-08-24T04:04:36, and so on). No query, config, or other column was touched.

The stale block is also self-contradicting: the KPI Example immediately below filters WHERE TO_TIMESTAMP(order_ts) >= CAST(NOW() AS TIMESTAMP) - INTERVAL '1' HOUR and reports 617 orders. That result is only possible if TO_TIMESTAMP(order_ts) yields present-day timestamps — with 1970 values the window would match zero rows. The two blocks in the same README cannot both be right, and the KPI block is the one consistent with the current runtime.

Verified against

spiceai/spiceai at v2.1.2Cargo.toml pins datafusion = "54.0.0".
Function semantics: datafusion/functions/src/datetime/to_timestamp.rs @ branch-54.
Producer type: kafka/producer.py:30order_ts = time.time().

Evidence

Static review only — this recipe needs Docker (Kafka broker + producer), so it was not executed. The correction rests on three independent, mutually consistent signals: the producer's value type, the DataFusion 54 source, and the arithmetic of the recipe's own KPI output.

One note for the reviewer: to_timestamp on a Float64 returns nanosecond precision, so real output carries sub-second digits. Those digits are not recoverable from the stale capture (the old nanosecond reading truncated them), so the corrected column is shown at whole-second precision.

…econds

The federated query aliases TO_TIMESTAMP(o.order_ts) as timestampt, but the
documented output shows 1970-01-01 dates. producer.py sets order_ts via
time.time() (float epoch seconds), and DataFusion 54 multiplies Float64 input
by 1_000_000_000 to convert seconds to nanoseconds, so these rows render as
2025-08-24, not 1970.

The stale block also contradicts the KPI query later in the same README, which
reports 617 orders within a 1-hour window -- impossible if the same expression
returned 1970 timestamps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant