Skip to content

fix: escape member names in normalized paths - #123

Open
gaoflow wants to merge 1 commit into
hiltontj:mainfrom
gaoflow:fix-normalized-path-escaping
Open

fix: escape member names in normalized paths#123
gaoflow wants to merge 1 commit into
hiltontj:mainfrom
gaoflow:fix-normalized-path-escaping

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 27, 2026

Copy link
Copy Markdown

Display for NormalizedPath destructures PathElement::Name and writes the &str with write!(f, "['{name}']"). That formats the raw string, not the PathElement, so it never reaches the RFC 9535 §2.7 escaper that #113 added to Display for PathElement. The escaper has been unreachable from the public API since it landed.

The example from #113's own description still reproduces on main:

let value = json!({"O'Reilly": true});
let path = JsonPath::parse("$.*")?;
let loc = path.query_located(&value).exactly_one()?.location().to_string();
// main:  $['O'Reilly']
// after: $['O\'Reilly']

Serialize for NormalizedPath goes through to_string(), so serialized locations were affected identically.

Why it went unnoticed

On #113 you wrote:

Another important thing that this made me realize is that I don't think the compliance test suite is providing a set of tests for validating how a library forms normalized paths.

The CTS added exactly that 11 days later (jsonpath-standard/jsonpath-compliance-test-suite@aaa53f2, "Test normalized paths"). The pinned submodule carries result_paths on 447 cases and results_paths on 9. tests/compliance.rs deserializes only result/results/invalid_selector, and its located branch calls q.nodes() and drops q.locations(), so none of those 456 expectations were ever read.

That is measurable rather than inferred. Replacing the result_paths/results_paths value of all 456 cases with ["$['TOTALLY_BOGUS_PATH']"]:

runner corrupted corpus
main compliance_test_suite ... ok
this PR fails

The class

Delegating to PathElement's Display exposes two further defects in that escaper, both in the same match:

  1. '\u{005C}' => write!(f, r#"\"#) emits one backslash. §2.7 lists "\" under normal-escapable, i.e. ESC + \. The visible effect is the worst case in the set, because it fails silently rather than loudly: member name a\b produced $['a\b'], which re-parses fine and selects 0 nodes, the \b having been re-read as a backspace.
  2. The hex arm hard-coded a \u000 prefix (write!(f, "\\u000{:x}", c as i32)), so it structurally could not emit \u001x and U+0010–U+001F fell through to _ => write!(f, "{c}") raw. §2.7 allows exactly this range:
normal-hexchar = "0" "0"
                 (
                    ("0" %x30-37) / ; "00"-"07"
                    ("0" %x62) /    ; "0b"
                    ("0" %x65-66) / ; "0e"-"0f"
                    ("1" normal-HEXDIG)
                 )

Replaced with c if c < '\u{0020}' => write!(f, "\\u{:04x}", c as u32), which covers the whole range uniformly and drops the prefix hazard. normal-unescaped omits %x0-1F entirely, so every remaining control code must be escaped, and normal-HEXDIG is DIGIT / %x61-66, so the digits are lowercase.

I audited the rest of the path-formatting surface and found nothing else: to_json_pointer implements RFC 6901, not §2.7, and its ~-then-/ ordering is correct.

Extent

Every code point U+0000–U+02FF as a member name, alone and as a<c>b, plus quote/backslash cases — 1551 names, checked against three oracles: a §2.7 ABNF validator over the emitted string, a round-trip (re-parse the path, must re-select that node), and Serialize vs Display.

before after
ABNF violations 71 0
normalized paths that fail to re-parse 70 0
re-parse silently selecting the wrong set 5 0
Serialize disagreeing with Display 0 0
CTS cases with wrong result_paths 14 / 456 0 / 456

Tests

  • tests/compliance.rs: result_paths/results_paths are now required fields on the TestResult variants and asserted against locations(), so the detector cannot silently disarm again. Also a round-trip assertion per located node — a normalized path is a singular query, so re-running it must return exactly the node it names.
  • tests/normalized_paths.rs: round-trip over U+0000–U+001F and the quote/backslash cases. The corpus cannot cover these; its result_paths contain only \b \f \n \r \t \' \\ and no \uXXXX at all.
  • path.rs: added normalized_path_fmt, and extended normalized_element_fmt to U+0010–U+001F.

One existing assertion had to change. normalized_element_fmt's "escapes" case fed r#"'\b\f\n\r\t\\'"# — a raw string, so the input is literal backslash-letter pairs, not the control characters the case name implies. Under the lone-backslash bug each \ passed through unchanged and the pairs came out looking like the intended escapes, so the case passed by coincidence and would have passed with no escaping of \ whatsoever. Correcting \ to \\ makes the expected value \'\\b\\f\\n\\r\\t\\\\\'. I split it in two rather than just updating the string: "escapes" now uses the actual control characters, and "literal_backslashes" keeps the original input with the RFC-correct expectation.

Mutation-tested; every mutant was confirmed to compile before its verdict was taken.

mutant caught by
revert NormalizedPath::Display to the raw-name write unit, CTS, round-trip
\ → one backslash (pre-fix) unit, CTS, round-trip
\ → four backslashes (over-fix) unit, CTS, round-trip
restore the U+0010–U+001F hole unit, round-trip
hex-escape printable ASCII too (over-fix) unit, CTS, round-trip
{:04X} uppercase hex unit only
{:03x} hex width unit, round-trip

Uppercase hex is pinned by the unit tests alone — the corpus has no \uXXXX expectation to catch it with.

cargo test --workspace --all-features, cargo clippy --all-targets -- -D warnings, cargo fmt --all -- --check and cargo doc --no-deps all pass.

`Display for NormalizedPath` destructured `PathElement::Name` and wrote the
`&str` directly, bypassing the RFC 9535 section 2.7 escaper that hiltontj#113 added to
`Display for PathElement`. Format each element through its own `Display`, and
fix two defects that only that delegation exposes: U+005C was written as a
single backslash instead of `\\`, and the hex-escape arm hard-coded a `\u000`
prefix, so U+0010-U+001F fell through unescaped.

Also deserialize the compliance suite's `result_paths`/`results_paths`, which
the runner had never read, and check that every normalized path re-parses and
re-selects the node it names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant