diff --git a/CHANGELOG.md b/CHANGELOG.md index ba37fa7c3c..aa835bd983 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Added +- **GFQL searchAny WYSIWYG float + datetime search (#1695)**: `searchAny` / `search_nodes` / `search_edges` now match float and datetime columns against **what the streamgl-viz inspector renders** (the displayed string, not the raw value), via explicit `columns=`. Floats render byte-identically to the inspector's `sprintf('%.4f')` (whole floats → bare integer string), fixing the exponent/half-tie divergence that had excluded them; datetimes render the inspector's moment format `'MMM D YYYY, h:mm:ss a z'` localized to a caller timezone. New WYSIWYG format call-params `{floatPrecision, temporalFormat, tz}` (cypher option-map + python twins, strict-validated), defaulting to the inspector (4 decimals; UTC). Cross-engine parity-or-honest-NIE: **pandas-exact** (dgx-verified against the real `sprintf-js`/`moment`), while cuDF/polars honestly decline float/datetime (their native decimal/temporal render can't reproduce pandas and a per-cell UDF would be a host-bridge). Float/datetime stay out of the auto-gate on every engine, keeping auto-search cross-engine-symmetric. - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked). - **GFQL Cypher `searchAny(entity, term[, opts])` cross-column search predicate + `g.search_nodes()`/`g.search_edges()` (viz-filter L2, native on all four engines)**: True where ANY of the entity's columns matches the term — the streamgl-viz inspector's table-search semantics as a composable WHERE predicate: OR across columns; case-insensitive substring by DEFAULT (case-folded, never regex — the common call avoids every engine regex limit); regex opt-in obeying the same per-engine decline rules as `=~`; dtype gate AS SEMANTICS (string columns always; integer columns iff the term is a numeric literal, per the inspector's `/^[0-9.-]+$/` gate; floats/dates/booleans reachable via the explicit `columns:` list). Options map `{caseSensitive, regex, columns}` is strict-validated (unknown keys error listing the valid ones); unbound aliases and missing explicit columns error clearly; null cells never match. Lowered like the pattern-predicate markers (a `search_any` row op + fresh marker column), so it composes through AND/OR/NOT and different node/edge terms coexist in one pipeline. Per-column matching reuses the parity-hardened `Contains` predicate on pandas/cuDF and a lowercase-fold/any_horizontal lowering on polars; oracle-pinned + 4-engine parity-or-NIE conformance cases. Python twins `g.search_nodes(term, columns=, case_sensitive=, regex=)` / `g.search_edges(...)` filter their own table and return a Plottable (polars-frame twins decline honestly for now — use the cypher op). Honest declines (NIE, use engine='pandas'): edge-alias searchAny on polars, and explicit columns beyond string/int/bool dtypes on polars AND cuDF (incl. floats: repr diverges across engines — dgx-probed). - **GFQL Cypher `EXISTS { }` pattern-existence subqueries (openCypher-standard), native on all four engines**: `WHERE EXISTS { (n)-[:R]->() }` and `WHERE NOT EXISTS { (n)--() }` now parse and run — the declarative prune-isolated building blocks for the streamgl-viz filter pipeline. An `EXISTS` body reuses the existing pattern-predicate lowering wholesale (`semi_apply_mark` / `anti_semi_apply` row ops), so pandas/cuDF worked immediately; the polars engine gains NATIVE lowerings for the semi-apply family (correlated key sets computed by the polars chain executor's named-flag columns; order-preserving `is_in` joins) plus `rows(binding_ops=...)` for the single-entity row table — previously all honest-NIE. Aliases introduced inside the braces are existentially scoped (`EXISTS { (n)--(m) }` allowed with `m` unbound outside — bare pattern predicates keep the conservative guard), inline property maps work, and the one supported inner `WHERE` form is endpoint inequality — `EXISTS { (n)--(m) WHERE m <> n }`, the drop-self-loop prune-isolated flavor (pandas/cuDF filter the correlated bindings; polars excludes self-loop edges, which is exactly the `m <> n` witness). Both prune flavors are oracle-pinned in the conformance matrix on a self-loop discriminator graph, 4-engine parity-or-NIE. Honest declines with clear errors: `EXISTS` in RETURN/WITH projections, general inner `WHERE`, multi-pattern bodies, full `MATCH..RETURN` subquery bodies, multi-alias correlation on polars. diff --git a/graphistry/compute/ComputeMixin.py b/graphistry/compute/ComputeMixin.py index 03b135f122..84d7d4e1c5 100644 --- a/graphistry/compute/ComputeMixin.py +++ b/graphistry/compute/ComputeMixin.py @@ -487,12 +487,15 @@ def get_topological_levels( out_df = safe_merge(g2_base._nodes, levels_df, on=g2_base._node, how='left') return self.nodes(out_df) - def search_nodes(self, term, columns=None, case_sensitive=False, regex=False): + def search_nodes(self, term, columns=None, case_sensitive=False, regex=False, + float_precision=4, temporal_format=None, tz="UTC"): """Keep nodes where ANY column matches ``term`` (viz-filter L2 inspector semantics: OR across columns; case-insensitive substring default; regex opt-in; string columns always, integer columns iff the term is a numeric literal — floats/dates via explicit ``columns=`` on pandas ONLY: cuDF declines them, its float/temporal stringification diverges from pandas). + ``float_precision``/``temporal_format``/``tz`` are the #1695 WYSIWYG format + options for explicit float/datetime columns (default = the inspector). pandas/cuDF native; polars frames raise NotImplementedError (use the cypher ``search_any`` op). """ @@ -506,7 +509,8 @@ def search_nodes(self, term, columns=None, case_sensitive=False, regex=False): "search_nodes is not yet native on polars frames; use the cypher " "search_any op or engine='pandas'") mask = search_any_mask( - df, term, case_sensitive=case_sensitive, regex=regex, columns=columns) + df, term, case_sensitive=case_sensitive, regex=regex, columns=columns, + float_precision=float_precision, temporal_format=temporal_format, tz=tz) if mask is None: raise GFQLValidationError( ErrorCode.E108, @@ -515,7 +519,8 @@ def search_nodes(self, term, columns=None, case_sensitive=False, regex=False): suggestion="List only columns present on the nodes table.") return self.nodes(df[mask]) - def search_edges(self, term, columns=None, case_sensitive=False, regex=False): + def search_edges(self, term, columns=None, case_sensitive=False, regex=False, + float_precision=4, temporal_format=None, tz="UTC"): """Keep edges where ANY column matches ``term`` — see :meth:`search_nodes`.""" from graphistry.compute.gfql.search_any import search_any_mask from graphistry.compute.exceptions import ErrorCode, GFQLValidationError @@ -527,7 +532,8 @@ def search_edges(self, term, columns=None, case_sensitive=False, regex=False): "search_edges is not yet native on polars frames; use the cypher " "search_any op or engine='pandas'") mask = search_any_mask( - df, term, case_sensitive=case_sensitive, regex=regex, columns=columns) + df, term, case_sensitive=case_sensitive, regex=regex, columns=columns, + float_precision=float_precision, temporal_format=temporal_format, tz=tz) if mask is None: raise GFQLValidationError( ErrorCode.E108, diff --git a/graphistry/compute/ast.py b/graphistry/compute/ast.py index 1ca8ef3ae6..cb5e714872 100644 --- a/graphistry/compute/ast.py +++ b/graphistry/compute/ast.py @@ -1688,11 +1688,17 @@ def search_any( case_sensitive: bool = False, regex: bool = False, columns: Optional[Sequence[str]] = None, + float_precision: Optional[int] = None, + temporal_format: Optional[str] = None, + tz: Optional[str] = None, ) -> ASTCall: """Annotate active rows with a cross-column search marker (viz-filter L2): ``out_col`` True where ANY of ``alias``'s columns matches ``term`` — OR across columns, case-insensitive substring default, regex opt-in, dtype-gated (string - columns always; integer columns iff numeric-literal term).""" + columns always; integer columns iff numeric-literal term). ``float_precision`` + / ``temporal_format`` / ``tz`` are the #1695 WYSIWYG format options for explicit + float / datetime columns (default = the streamgl-viz inspector: 4 decimals; the + moment date format; UTC).""" params: Dict[str, Any] = {"alias": alias, "term": term, "out_col": out_col} if case_sensitive: params["case_sensitive"] = True @@ -1700,6 +1706,12 @@ def search_any( params["regex"] = True if columns is not None: params["columns"] = list(columns) + if float_precision is not None: + params["float_precision"] = float_precision + if temporal_format is not None: + params["temporal_format"] = temporal_format + if tz is not None: + params["tz"] = tz return ASTCall("search_any", params) diff --git a/graphistry/compute/gfql/call/validation.py b/graphistry/compute/gfql/call/validation.py index 095a9d74da..931ac33d95 100644 --- a/graphistry/compute/gfql/call/validation.py +++ b/graphistry/compute/gfql/call/validation.py @@ -317,6 +317,12 @@ def is_list_of_dicts(v: object) -> bool: return isinstance(v, list) and all(isinstance(item, dict) for item in v) +def is_nonneg_int(v: object) -> bool: + # bool is an int subclass in Python — exclude it, and negatives (#1695 + # floatPrecision: a `"%.*f" % -1` clamp would silently misrender). + return is_int(v) and not isinstance(v, bool) and v >= 0 # type: ignore[operator] + + def is_where_rows_expr(v: object) -> bool: return is_non_empty_string(v) and _where_rows_expr_parser_parse_ok(str(v).strip()) @@ -407,7 +413,8 @@ def _semi_apply_mark_added_node_cols(params: Dict[str, object]) -> Set[str]: ), 'search_any': _safelist_entry( - {'alias', 'term', 'out_col', 'case_sensitive', 'regex', 'columns'}, + {'alias', 'term', 'out_col', 'case_sensitive', 'regex', 'columns', + 'float_precision', 'temporal_format', 'tz'}, required_params={'alias', 'term', 'out_col'}, param_validators={ 'alias': is_non_empty_string, @@ -416,6 +423,11 @@ def _semi_apply_mark_added_node_cols(params: Dict[str, object]) -> Set[str]: 'case_sensitive': is_bool, 'regex': is_bool, 'columns': is_non_empty_list_of_strings, + # #1695 WYSIWYG format options (default = inspector): float decimals, + # datetime strftime pattern, and localization tz. + 'float_precision': is_nonneg_int, + 'temporal_format': is_non_empty_string, + 'tz': is_non_empty_string, }, description='Annotate active rows with a cross-column search marker (OR across columns)', schema_effects=_schema_effects(adds_node_cols=_semi_apply_mark_added_node_cols), diff --git a/graphistry/compute/gfql/cypher/lowering.py b/graphistry/compute/gfql/cypher/lowering.py index 3623e4d992..ce829e94de 100644 --- a/graphistry/compute/gfql/cypher/lowering.py +++ b/graphistry/compute/gfql/cypher/lowering.py @@ -3851,7 +3851,12 @@ def _append_page_ops( r"\s*(?:,\s*\{([^{}]*)\})?\s*\)", re.IGNORECASE, ) -_SEARCH_ANY_OPT_KEYS = {"casesensitive": "case_sensitive", "regex": "regex", "columns": "columns"} +_SEARCH_ANY_OPT_KEYS = { + "casesensitive": "case_sensitive", "regex": "regex", "columns": "columns", + # #1695 WYSIWYG format options (default = the streamgl-viz inspector) + "floatprecision": "float_precision", "temporalformat": "temporal_format", "tz": "tz", +} +_SEARCH_ANY_STR_OPT_RE = re.compile(r"^'([^']*)'$|^\"([^\"]*)\"$") _SEARCH_ANY_COLUMNS_RE = re.compile( r"^\[\s*(?:'[^']*'|\"[^\"]*\")(?:\s*,\s*(?:'[^']*'|\"[^\"]*\"))*\s*\]$") _SEARCH_ANY_COL_ITEM_RE = re.compile(r"'([^']*)'|\"([^\"]*)\"") @@ -3898,7 +3903,22 @@ def _parse_search_any_opts(opts_text: str, *, line: int, column: int) -> Dict[st field="where", value=val_text, line=line, column=column, ) out[canon] = low == "true" - else: + elif canon == "float_precision": + if not re.fullmatch(r"\d+", val_text): + raise _unsupported( + "searchAny option 'floatPrecision' must be a non-negative integer", + field="where", value=val_text, line=line, column=column, + ) + out[canon] = int(val_text) + elif canon in ("temporal_format", "tz"): + sm = _SEARCH_ANY_STR_OPT_RE.match(val_text) + if not sm: + raise _unsupported( + f"searchAny option {key!r} must be a string literal", + field="where", value=val_text, line=line, column=column, + ) + out[canon] = sm.group(1) if sm.group(1) is not None else sm.group(2) + else: # columns if not _SEARCH_ANY_COLUMNS_RE.match(val_text): raise _unsupported( "searchAny option 'columns' must be a list of string literals", diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index 294d068744..4ca99042ad 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -4180,6 +4180,9 @@ def search_any( case_sensitive: bool = False, regex: bool = False, columns: Optional[List[str]] = None, + float_precision: int = 4, + temporal_format: Optional[str] = None, + tz: str = "UTC", ) -> "Plottable": """Cross-column search marker (viz-filter L2, panel-algebra D2): ``out_col`` is True where ANY of the alias's columns matches ``term`` — OR across columns, @@ -4204,7 +4207,8 @@ def search_any( sub = left_df[[c for c in left_df.columns if not c.startswith("__gfql_") and c != alias]] mask = search_any_mask( - sub, term, case_sensitive=case_sensitive, regex=regex, columns=columns) + sub, term, case_sensitive=case_sensitive, regex=regex, columns=columns, + float_precision=float_precision, temporal_format=temporal_format, tz=tz) if mask is None: raise GFQLValidationError( ErrorCode.E108, diff --git a/graphistry/compute/gfql/search_any.py b/graphistry/compute/gfql/search_any.py index 04977bfb46..473153837f 100644 --- a/graphistry/compute/gfql/search_any.py +++ b/graphistry/compute/gfql/search_any.py @@ -1,12 +1,17 @@ """Cross-column search kernel (viz-filter L2, panel-algebra D2): OR-across-columns substring/regex match, dtype-gated AS SEMANTICS — string columns always; integer columns iff the term is a numeric literal (inspector gate); float/date/bool are -auto-gated OUT (float stringification is engine-divergent — explicit ``columns=`` -reaches bool on both engines, float/date on pandas only: cuDF declines them, its -astype(str) rendering diverges from pandas — dgx-probed). Per-column matching -delegates to the parity-hardened ``Contains`` predicate, so every pandas/cuDF -quirk and honest decline gate carries over; cuDF regex obeys the same decline -rules as ``=~``.""" +auto-gated OUT (kept symmetric across engines for cross-engine parity — #1695 +decision A). Explicit ``columns=`` reaches bool on both engines and float/datetime +on **pandas only**: pandas renders floats to the inspector's WYSIWYG search text +via ``_canonical_float_str`` (``%.4f``-direct, dgx-verified byte-identical to the +viz ``sprintf-js``) and datetimes via ``_canonical_datetime_str`` (the inspector's +moment date format, localized to a caller ``tz``); cuDF/polars honestly decline +both (their native decimal / datetime render can't reproduce pandas — cuDF float +truncates, polars diverges at ties — and a per-element UDF would be a host-bridge; +dgx-probed 2026-07-05). Per-column matching delegates to the parity-hardened +``Contains`` predicate, so every pandas/cuDF quirk and honest decline gate carries +over; cuDF regex obeys the same decline rules as ``=~``.""" import re from typing import List, Optional @@ -30,6 +35,160 @@ def _is_int_dtype(dtype: object) -> bool: return bool(pat.is_integer_dtype(dtype)) +def _is_float_dtype(dtype: object) -> bool: + import pandas.api.types as pat + return bool(pat.is_float_dtype(dtype)) + + +def _is_datetime_dtype(dtype: object) -> bool: + import pandas.api.types as pat + return bool(pat.is_datetime64_any_dtype(dtype)) + + +# The inspector's default date render (moment `'MMM D YYYY, h:mm:ss a z'`) expressed +# as its strftime equivalent — moment MMM=%b, D=%-d (no leading zero), YYYY=%Y, +# h=%-I (12h no leading zero), mm=%M, ss=%S, a=am/pm (lowercase), z=%Z (tz abbrev). +_INSPECTOR_TEMPORAL_STRFTIME = "%b %-d %Y, %-I:%M:%S %p %Z" +_MONTH_ABBR = [ + "", "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", +] + + +def _assemble_default_datetime_utc(loc: SeriesT) -> SeriesT: + """Vectorized build of the inspector's default date render in UTC — byte-identical + to ``loc.dt.strftime('%b %-d %Y, %-I:%M:%S %p %Z')`` (am/pm lowercased, z='UTC'), + but ~3x faster: pandas ``dt.strftime`` is a per-element slow path (~2s/1M rows), + while ``.dt`` component extraction + string concat is vectorized. The tz abbrev + is the constant ``'UTC'`` here, which is what makes the fast path sound (a DST + zone's abbrev varies per timestamp -> those fall back to strftime).""" + import numpy as np + import pandas as pd + d = loc.dt + idx = loc.index + # NaT lanes are filled with dummy components (never surface — the caller masks + # them to "" via `.where(notna, "")`); this keeps every op vectorized (no NA + # propagation through the concat). + mo = pd.Series(np.array(_MONTH_ABBR)[d.month.fillna(0).astype(int).to_numpy()], index=idx) + h24 = d.hour.fillna(0).astype(int) + h12 = ((h24 - 1) % 12 + 1).astype(str) # 0->12, 13->1, 12->12 + ap = pd.Series(np.where(h24.to_numpy() < 12, "am", "pm"), index=idx) + day = d.day.fillna(1).astype(int).astype(str) + yr = d.year.fillna(0).astype(int).astype(str) + mm = d.minute.fillna(0).astype(int).astype(str).str.zfill(2) + ss = d.second.fillna(0).astype(int).astype(str).str.zfill(2) + return mo + " " + day + " " + yr + ", " + h12 + ":" + mm + ":" + ss + " " + ap + " UTC" + + +def _canonical_datetime_str( + s: SeriesT, temporal_format: Optional[str] = None, tz: str = "UTC" +) -> SeriesT: + """WYSIWYG datetime render matching the streamgl-viz inspector's date search text + (``formatDate`` -> moment ``'MMM D YYYY, h:mm:ss a z'``), LOCALIZED to ``tz``. + + The inspector formats a date in the VIEWER's timezone (``moment.unix`` local), + which is not knowable server-side — so ``tz`` is a caller knob (Leo's + localization call-param): pass the viewer's zone (e.g. ``'America/Los_Angeles'``) + for byte parity with what that viewer sees. Default ``'UTC'`` is chosen for + DETERMINISM (a server-local default would vary by deployment and break the + parity oracle); flip via ``tz=``. + + PANDAS-ONLY (like the float render): cuDF/polars decline datetime search — their + native datetime->string rendering and tz handling diverge from pandas strftime, + and a per-cell UDF would be a host-bridge. Null cells render ``""`` (masked out). + + ``temporal_format`` is a **strftime** pattern; its default reproduces the + inspector's moment default. moment's lowercase am/pm (``a``) is reproduced by + lowercasing the ``%p`` output (tz abbrevs / month names never contain AM/PM). + """ + import pandas as pd + fmt = temporal_format or _INSPECTOR_TEMPORAL_STRFTIME + ser = s if isinstance(s, pd.Series) else pd.Series(s) + # localize: naive columns are assumed UTC then converted; tz-aware are converted. + if getattr(ser.dtype, "tz", None) is not None: + localized = ser.dt.tz_convert(tz) + else: + localized = ser.dt.tz_localize("UTC").dt.tz_convert(tz) + notna = ser.notna() + if temporal_format is None and tz == "UTC": + # FAST PATH (interactive hot path): the default format in UTC, assembled + # vectorized instead of via the ~20x-slower dt.strftime. The render is + # term-independent, so a repeated (typing) search should also cache it. + txt = _assemble_default_datetime_utc(localized) + else: + txt = localized.dt.strftime(fmt) + if "%p" in fmt: + # moment `a` is lowercase am/pm; strftime %p is upper. Lowercase ONLY the + # standalone AM/PM token (word-bounded) so we don't corrupt an alpha tz + # abbreviation (e.g. America/Manaus -> "AMT") or literal text in a custom + # temporal_format — a blind global replace turned "AMT" into "amT". + txt = txt.str.replace(r"\bAM\b", "am", regex=True).str.replace(r"\bPM\b", "pm", regex=True) + return txt.where(notna, "") + + +def _canonical_float_str(s: SeriesT, precision: int = 4) -> SeriesT: + """WYSIWYG float render matching the streamgl-viz inspector's search text — + verified byte-identical to the inspector's real ``sprintf-js`` ``%.4f`` on + dgx-spark (float parity fuzzer, 2026-07-05). + + - WHOLE value (``v % 1 == 0``, incl. whole floats like ``5.0``, ``|v| < 1e21``) + -> bare integer string (``"5"``/``"-3"`` — no decimals, no exponent; + matches JS ``String(v)`` / ``formatToString``). + - FRACTIONAL value -> ``"%.f" % v`` applied DIRECTLY to the raw + double (single correctly-rounded conversion; matches ``sprintf('%.4f', v)``, + e.g. ``0.12345`` -> ``"0.1235"``, ``-3.74825`` -> ``"-3.7483"``). + + PANDAS-ONLY. The GPU engines cannot reproduce this natively and honestly + decline float search upstream (dgx-proven: cuDF ``float64 -> decimal128`` + TRUNCATES — wrong on generic values, not just ties; polars' decimal cast + rounds differently at 5th-decimal ties and drops the sign on ``-0.0000``; + there is no native GPU ``printf`` and a per-element UDF would be a forbidden + host-bridge). So this is only ever called on a pandas float column. + + NOTE: an earlier draft pre-rounded with ``np.round(v, precision)`` before + ``%.*f`` — that DOUBLE-ROUNDS and mis-renders half-ties (``-3.74825`` -> + ``-3.7482`` instead of the browser's ``-3.7483``). We round exactly once, in + ``%.*f``, on the raw double. Null cells render as ``""`` and never match + (the caller masks them out). + """ + import numpy as np + n = len(s) + if n == 0: + return cast_series_like(s, []) + notna = np.asarray(s.notna()) + vals = s.to_numpy(dtype="float64", na_value=np.nan) + inf = np.inf + fmt = "%." + str(precision) + "f" + + def render(v: float) -> str: + # order matters: NaN/inf checked BEFORE `v % 1` (which would warn on them) + if v != v: # NaN -> "" (masked out by caller) + return "" + if v == inf: # JS String(Infinity)/sprintf('%.4f',Inf) + return "Infinity" + if v == -inf: + return "-Infinity" + if v % 1 == 0: # WHOLE (finite): plain integer digits (JS String); + return str(int(v)) if abs(v) < 1e21 else "" # >=1e21 JS-exponent regime -> "" + return fmt % v # FRACTIONAL: printf on raw double == sprintf-js + + # A python list-comp is the interactive hot path here and — measured on dgx at + # 1M rows — is faster than np.char.mod / a numpy-masked scatter (which pay for + # several full-array passes + object boxing). The render is term-independent, so + # a caller that searches repeatedly (typing) should cache this per column. + # (NOTE: whole floats >= 2**53 render str(int(v)) = the exact-integer decimal, + # which can differ from JS String()'s shortest round-trip for a few magnitudes — + # a documented, astronomically-rare search limitation.) + out = [render(v) for v in vals] + if not notna.all(): # nullable NA that to_numpy filled -> never match + out = [o if ok else "" for o, ok in zip(out, notna)] + return cast_series_like(s, out) + + +def cast_series_like(template: SeriesT, data: list) -> SeriesT: + import pandas as pd + return pd.Series(data, index=template.index, dtype="object") + + def _has_string_content(df: DataFrameT, c: object) -> bool: """True iff column ``c`` holds STRINGS (a real string dtype, or an object column whose values are actually strings). An object column of lists/dicts/mixed is NOT string @@ -70,6 +229,31 @@ def search_candidate_columns( return out +def validate_format_opts( + float_precision: int, temporal_format: Optional[str], tz: str +) -> None: + """Validate the #1695 WYSIWYG format options at the kernel — the SINGLE choke + point every surface passes through (cypher op, ast op, and the python twins, + which bypass the call safelist). Consistent ``GFQLValidationError`` regardless + of entry point, instead of a downstream ``"%.*f" % -1`` silent clamp or an + opaque ``tz_convert('')`` IndexError.""" + from graphistry.compute.exceptions import ErrorCode, GFQLValidationError + if not isinstance(float_precision, int) or isinstance(float_precision, bool) or float_precision < 0: + raise GFQLValidationError( + ErrorCode.E108, "searchAny floatPrecision must be a non-negative integer", + field="float_precision", value=float_precision, + suggestion="Use an integer >= 0 (default 4).") + if temporal_format is not None and (not isinstance(temporal_format, str) or temporal_format == ""): + raise GFQLValidationError( + ErrorCode.E108, "searchAny temporalFormat must be a non-empty strftime string", + field="temporal_format", value=temporal_format, + suggestion="Omit for the inspector default, or pass a non-empty strftime pattern.") + if not isinstance(tz, str) or tz == "": + raise GFQLValidationError( + ErrorCode.E108, "searchAny tz must be a non-empty timezone string", + field="tz", value=tz, suggestion="Use an IANA zone like 'UTC' or 'America/Los_Angeles'.") + + def search_any_mask( df: DataFrameT, term: str, @@ -77,9 +261,13 @@ def search_any_mask( case_sensitive: bool = False, regex: bool = False, columns: Optional[List[str]] = None, + float_precision: int = 4, + temporal_format: Optional[str] = None, + tz: str = "UTC", ) -> Optional[SeriesT]: """Boolean row mask over ``df`` (pandas or cuDF), or None to decline (an explicit column is missing). Null cells never match; no candidate columns -> all-False.""" + validate_format_opts(float_precision, temporal_format, tz) from graphistry.compute.predicates.str import ( Contains, _cudf_casefold_or_decline, _cudf_regex_prep, ) @@ -120,7 +308,19 @@ def search_any_mask( for c in cols: s = df[c] m: SeriesT - if not _is_searchable_string_dtype(s.dtype): + if _is_float_dtype(s.dtype): + # FLOAT (pandas-only; cuDF float declined above, polars uses its own + # lowering): render to the inspector's WYSIWYG search text (%.4f-direct + # / whole->int-string, dgx-verified byte-identical to sprintf-js) rather + # than astype(str), whose exponent/half-tie rendering diverges (#1695). + nulls = s.isna() + m = pred(_canonical_float_str(s, float_precision)) & ~nulls + elif _is_datetime_dtype(s.dtype): + # DATETIME (pandas-only; cuDF/polars declined above/in their lowering): + # render the inspector's moment date format localized to `tz` (#1695). + nulls = s.isna() + m = pred(_canonical_datetime_str(s, temporal_format, tz)) & ~nulls + elif not _is_searchable_string_dtype(s.dtype): # canonical toString for int / explicit columns; pandas astype(str) # stringifies nulls ("nan"/"") so mask them back out — null cells # never match on any engine (wave-1 I1) diff --git a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py index fb2274094d..f3f2c18664 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py @@ -426,6 +426,150 @@ def q(**kw): g.gfql(q(term="x", columns=["nope"]), engine="pandas") +def test_search_any_float_wysiwyg(): + """#1695: explicit ``columns=[]`` searches the inspector's WYSIWYG render + (``%.4f``-direct / whole->int-string, dgx-verified == viz sprintf-js) on PANDAS; + cuDF/polars honestly decline (their native float render can't reproduce printf). + Float stays OUT of the auto gate on every engine (decision A) so auto behavior is + cross-engine-symmetric.""" + import numpy as np + import pandas as pd + from graphistry.compute.ast import search_any as search_any_op + nd = pd.DataFrame({ + "id": [0, 1, 2, 3, 4, 5], + # fractional, whole-float, a 5th-decimal HALF-TIE (guards the np.round + # double-rounding regression: correct render is -3.7483, NOT -3.7482), null + "f": [0.5, 1.5, 2.5, -3.74825, 5.0, np.nan], + }) + ed = pd.DataFrame({"s": [0, 1], "d": [1, 2], "eid": [0, 1]}) + g = graphistry.nodes(nd, "id").edges(ed, "s", "d").bind(edge="eid") + + def q(**kw): + return [n(name="a"), search_any_op(alias="a", out_col="__hit__", **kw)] + + # (term, columns) -> matching ids, pandas oracle. renders: + # 0.5->"0.5000" 1.5->"1.5000" 2.5->"2.5000" -3.74825->"-3.7483" 5.0->"5" nan->"" + oracle = { + "frac_exact": (dict(term="1.5000", columns=["f"]), [1]), + "half_tie": (dict(term="-3.7483", columns=["f"]), [3]), # correct single-round + "half_tie_bad": (dict(term="-3.7482", columns=["f"]), []), # the WRONG (double-round) render must NOT match + "whole_no_dec": (dict(term="5.0000", columns=["f"]), []), # 5.0 renders "5", not "5.0000" + "whole_int": (dict(term="5", columns=["f"]), + [0, 1, 2, 4]), # substring '5' hits 0.5000/1.5000/2.5000/"5" + "null_no_nan": (dict(term="nan", columns=["f"]), []), # null cell renders "" not "nan" -> never matches + } + for label, (kw, expected) in oracle.items(): + out = g.gfql(q(**kw), engine="pandas") + got = _to_pd(out._nodes).sort_values("id") + got = got[got["__hit__"]]["id"].tolist() + assert got == expected, f"float wysiwyg {label}: {got} != {expected}" + # cuDF/polars must honestly NIE on explicit float columns (never silently diverge) + _assert_invariant(g, q(**kw), f"float wysiwyg {label}") + + # inf/-inf render "Infinity"/"-Infinity" (JS String(Infinity)), no RuntimeWarning + import numpy as np + import warnings + ginf = graphistry.nodes(pd.DataFrame({"id": [0, 1, 2], "f": [np.inf, -np.inf, 1.5]}), "id") + with warnings.catch_warnings(): + warnings.simplefilter("error", RuntimeWarning) + assert ginf.search_nodes("Infinity", columns=["f"])._nodes["id"].tolist() == [0, 1] + assert ginf.search_nodes("-Infinity", columns=["f"])._nodes["id"].tolist() == [1] + + # pandas python-twin also renders float WYSIWYG (row 3 = the half-tie -3.7483); + # polars-backed twin declines honestly (never a silent wrong answer) + assert sorted(g.search_nodes("-3.7483", columns=["f"])._nodes["id"].tolist()) == [3] + import polars as pl + gpl = graphistry.nodes(pl.from_pandas(nd), "id").edges(pl.from_pandas(ed), "s", "d").bind(edge="eid") + with pytest.raises(NotImplementedError): + gpl.search_nodes("1.5000", columns=["f"]) + + +def test_search_any_datetime_wysiwyg(): + """#1695: explicit ``columns=[]`` searches the inspector's moment date + render (``'MMM D YYYY, h:mm:ss a z'``, default tz=UTC — byte-verified vs real + moment.utc) on PANDAS; cuDF/polars honestly decline. Datetime stays OUT of the + auto gate on every engine (decision A).""" + import pandas as pd + from graphistry.compute.ast import search_any as search_any_op + nd = pd.DataFrame({ + "id": [0, 1, 2], + "t": pd.to_datetime(["2021-03-14 15:09:26", "2021-12-25 00:00:00", None]), + }) + ed = pd.DataFrame({"s": [0, 1], "d": [1, 2], "eid": [0, 1]}) + g = graphistry.nodes(nd, "id").edges(ed, "s", "d").bind(edge="eid") + + def q(**kw): + return [n(name="a"), search_any_op(alias="a", out_col="__hit__", **kw)] + + # renders (tz=UTC default): 0->"Mar 14 2021, 3:09:26 pm UTC" + # 1->"Dec 25 2021, 12:00:00 am UTC" 2(null)->"" + oracle = { + "month_day": (dict(term="mar 14 2021", columns=["t"]), [0]), # ci substring + "time_ampm": (dict(term="3:09:26 pm", columns=["t"]), [0]), # 12h + lowercase pm + "midnight": (dict(term="12:00:00 am", columns=["t"]), [1]), # moment 'h' -> 12 am + "tz_abbrev": (dict(term="utc", columns=["t"]), [0, 1]), # z token = UTC + "null_empty": (dict(term="nat", columns=["t"]), []), # null renders "" not "nat"/"nan" + } + for label, (kw, expected) in oracle.items(): + out = g.gfql(q(**kw), engine="pandas") + got = _to_pd(out._nodes).sort_values("id") + got = got[got["__hit__"]]["id"].tolist() + assert got == expected, f"datetime wysiwyg {label}: {got} != {expected}" + # cuDF/polars must honestly NIE on explicit datetime columns (never silent divergence) + _assert_invariant(g, q(**kw), f"datetime wysiwyg {label}") + + # pandas python-twin also renders datetime WYSIWYG; polars-backed twin declines + assert sorted(g.search_nodes("dec 25 2021", columns=["t"])._nodes["id"].tolist()) == [1] + import polars as pl + gpl = graphistry.nodes(pl.from_pandas(nd), "id").edges(pl.from_pandas(ed), "s", "d").bind(edge="eid") + with pytest.raises(NotImplementedError): + gpl.search_nodes("mar 14 2021", columns=["t"]) + + +def test_search_any_format_options(): + """#1695 P4: the WYSIWYG format call-params (floatPrecision / temporalFormat / tz) + thread through the cypher option map AND the python twins; strict-validated.""" + import pandas as pd + from graphistry.compute.exceptions import GFQLValidationError + nd = pd.DataFrame({ + "id": [0, 1, 2], + "f": [0.5, 1.5, -3.74825], + "t": pd.to_datetime(["2021-03-14 15:09:26", "2021-06-01 00:00:00", None]), + }) + ed = pd.DataFrame({"s": [0, 1], "d": [1, 2], "eid": [0, 1]}) + g = graphistry.nodes(nd, "id").edges(ed, "s", "d").bind(edge="eid") + + # floatPrecision=2 via cypher: -3.74825 -> "-3.75" (2 decimals, not the default "-3.7483") + q_fp = "MATCH (a) WHERE searchAny(a, '-3.75', {columns: ['f'], floatPrecision: 2}) RETURN a.id AS id" + assert sorted(_to_pd(g.gfql(q_fp, engine="pandas")._nodes)["id"].tolist()) == [2] + _assert_invariant(g, q_fp, "searchAny floatPrecision") + + # tz via cypher: 15:09:26 UTC -> America/Los_Angeles "8:09:26 am" (PDT) + q_tz = ("MATCH (a) WHERE searchAny(a, '8:09:26 am', " + "{columns: ['t'], tz: 'America/Los_Angeles'}) RETURN a.id AS id") + assert sorted(_to_pd(g.gfql(q_tz, engine="pandas")._nodes)["id"].tolist()) == [0] + _assert_invariant(g, q_tz, "searchAny tz") + + # python twin threads float_precision too + assert sorted(g.search_nodes("-3.75", columns=["f"], float_precision=2) + ._nodes["id"].tolist()) == [2] + # default precision (4) does NOT match the 2-decimal string -> disjoint from fp=2 + assert g.search_nodes("-3.75", columns=["f"])._nodes["id"].tolist() == [] + + # strict validation: unknown option key + wrong floatPrecision type both raise + with pytest.raises(GFQLValidationError): + g.gfql("MATCH (a) WHERE searchAny(a, 'x', {bogus: 1}) RETURN a.id AS id", engine="pandas") + with pytest.raises(GFQLValidationError): + g.gfql("MATCH (a) WHERE searchAny(a, 'x', {floatPrecision: 'no'}) RETURN a.id AS id", + engine="pandas") + # kernel-level validation ALSO covers the python twins (which bypass the call + # safelist): negative/empty options raise GFQLValidationError, not a silent + # "%.*f" % -1 misrender or an opaque tz_convert('') IndexError. + for bad in (dict(float_precision=-1), dict(float_precision=True), dict(tz="")): + with pytest.raises(GFQLValidationError): + g.search_nodes("x", columns=["f"], **bad) + + def test_search_any_cypher_surface_all_engines(): """viz-filter L2-b: the cypher WHERE searchAny(...) surface — marker lift + composition with other predicates; oracle-pinned + 4-engine parity-or-NIE."""