Fix kafka recipe: correct federated-query timestamps misread as nanoseconds - #558
Open
claudespice wants to merge 1 commit into
Open
Fix kafka recipe: correct federated-query timestamps misread as nanoseconds#558claudespice wants to merge 1 commit into
claudespice wants to merge 1 commit into
Conversation
…econds The federated query aliases TO_TIMESTAMP(o.order_ts) as timestampt, but the documented output shows 1970-01-01 dates. producer.py sets order_ts via time.time() (float epoch seconds), and DataFusion 54 multiplies Float64 input by 1_000_000_000 to convert seconds to nanoseconds, so these rows render as 2025-08-24, not 1970. The stale block also contradicts the KPI query later in the same README, which reports 617 orders within a 1-hour window -- impossible if the same expression returned 1970 timestamps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Federated Query Example in the
kafkarecipe selectsTO_TIMESTAMP(o.order_ts) as timestampt, but its documented output shows every row at1970-01-01T00:00:01.75600827x— epoch-second values being read as nanoseconds.timestamptrenders as1970-01-01T00:00:01.756008276.producer.pysetsorder_ts = time.time()— a float of epoch seconds. DataFusion 54'sto_timestampdocuments that "Integers, unsigned integers, and doubles are interpreted as seconds since the unix epoch", and theFloat64arm multiplies by1_000_000_000.0in both the scalar and array branches. So1756008276.xrenders as 2025-08-24, not 1970.timestamptcolumn in that one output block now shows the correctly-interpreted timestamps, derived from the same epoch values already encoded in the stale output (1756008276→2025-08-24T04:04:36, and so on). No query, config, or other column was touched.The stale block is also self-contradicting: the KPI Example immediately below filters
WHERE TO_TIMESTAMP(order_ts) >= CAST(NOW() AS TIMESTAMP) - INTERVAL '1' HOURand reports 617 orders. That result is only possible ifTO_TIMESTAMP(order_ts)yields present-day timestamps — with 1970 values the window would match zero rows. The two blocks in the same README cannot both be right, and the KPI block is the one consistent with the current runtime.Verified against
spiceai/spiceaiatv2.1.2—Cargo.tomlpinsdatafusion = "54.0.0".Function semantics:
datafusion/functions/src/datetime/to_timestamp.rs@branch-54.Producer type:
kafka/producer.py:30—order_ts = time.time().Evidence
Static review only — this recipe needs Docker (Kafka broker + producer), so it was not executed. The correction rests on three independent, mutually consistent signals: the producer's value type, the DataFusion 54 source, and the arithmetic of the recipe's own KPI output.
One note for the reviewer:
to_timestampon aFloat64returns nanosecond precision, so real output carries sub-second digits. Those digits are not recoverable from the stale capture (the old nanosecond reading truncated them), so the corrected column is shown at whole-second precision.