Make @base64d tolerant of newlines

[robin-a-meade]

It would be nice if @base64d was tolerant of newlines.

$ echo -n 'Hello, World!' | jaq -sRr @base64 | jaq -sRr @base64d
Error: "Invalid symbol 10, offset 20."

Maybe do something like https://docs.rs/testserver/0.1.1/src/testserver/lib.rs.html#56-61

Or use https://crates.io/crates/data-encoding .

[01mf02]

Hi @robin-a-meade, sorry for having taking such a long time to respond!

To understand this issue:

$ echo -ne 'a\n' | jaq -sRr @base64 | jaq -rR '.'
YQo=
$ echo -ne 'a\n' | jaq -sRr @base64 | jaq -srR '.'
YQo=

$ # notice the empty line before!

So slurping in the input (via -s) creates a newline.
May I ask why you're using -s in the first place in the jaq invocation of @base64d? Without that, your filter works fine.

Apart from that, there seem to be a few differences w.r.t. base64 decoding between jq and jaq. For example:

$ echo -n 'a' | jaq -sRr @base64
YQ==
$ echo -n 'YQ==YQ==' | jq -sR '@base64d' # let's just double the previous output
"a"
$ echo -n 'YQ==YQ==' | jaq -sR '@base64d'
Error: "Invalid symbol 61, offset 2."

So jq seems to parse until the end of the base64-encoded string, and discards everything after it.
I imagine that this could easily lead to silent data corruption.
In contrast, jaq's @base64d only succeeds if the whole string passed to it is valid base64 data.

[robin-a-meade]

Hi @01mf02 , sorry for the late response, thank you for your comments.

I slurped the input (via -s) because I wanted to do a test of jaq's base64 encoding and decoding and figured a good test would be to attempt a roundtrip of encoding and decoding a bunch of text.

For example:

lorem.txt

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus nec justo sit
amet erat porta condimentum vitae ut risus. Nam eu elit et massa commodo
scelerisque quis eget ex. In sit amet mi sem. Phasellus efficitur ultricies ex,
vel interdum sem dignissim vel. Integer posuere nibh risus.

The base64 utility from GNU CoreUtils is able to roundtrip base64 encoding/decoding:

base64 <lorem.txt | base64 -d

But when I plug in jaq for the decoding, it gives an error:

$ base64 <lorem.txt | jaq -sRr @base64d
Error: "Invalid symbol 10, offset 76."

If I eliminate the newlines using the -w 0 option, it works:

$ base64 -w 0 <lorem.txt | jaq -sRr @base64d
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus nec justo sit
amet erat porta condimentum vitae ut risus. Nam eu elit et massa commodo
scelerisque quis eget ex. In sit amet mi sem. Phasellus efficitur ultricies ex,
vel interdum sem dignissim vel. Integer posuere nibh risus

I researched it and learned that whether newlines should be tolerated in base64 encoded text is debatable.

The marshallpierce/rust-base64 library, that jaq depends on, has a FAQ about it.

In java world, java.util.Base64.getDecoder() does not tolerate newlines when decoding base64 encoded data, but [java.util.Base64.getMimeDecoder()](https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html) does.

NodeJS Buffer.from(base64String, 'base64').toString('utf-8'); tolerates. "Whitespace characters such as spaces, tabs, and new lines contained within the base64-encoded string are ignored." https://nodejs.org/api/buffer.html

Browser Window.atob() is forgiving https://infra.spec.whatwg.org/#forgiving-base64

I'll defer to you in in the case of jaq.

[polarathene]

I think this is working as intended? You initially used -s/--slurp which failed, and -j/--join-output didn't seem to be discussed, yet for this input/output scenario it seems like appropriate solution?

# `-s` on base64 decode causes failure:
$ echo -n 'Hello, World!' | jaq -Rr @base64 | jaq -sRr @base64d
Error: "Invalid symbol 10, offset 20."

# Works (NOTE: Use `-j` at final jaq to match `base64 -d` without a trailing newline):
$ echo -n 'Hello, World!' | jaq -Rr @base64 | jaq -Rr @base64d
Hello, World!

# Without `-s` base64 decode works here too, but `base64` output inserts newlines:
$ base64 <lorem.txt | jaq -Rr @base64d
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Vivamus nec justo sit
amet erat porta condimentum vitae u
t risus. Nam eu elit et massa commodo
scelerisque quis eg
et ex. In sit amet mi sem. Phasellus efficitur ultricies
ex,
vel interdum sem dignissim vel. Integer posuere nibh
risus.

# Above issue with newlines due to this input to jaq which contains `\n`:
$ base64 <lorem.txt
TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdC4g
Vml2YW11cyBuZWMganVzdG8gc2l0CmFtZXQgZXJhdCBwb3J0YSBjb25kaW1lbnR1bSB2aXRhZSB1
dCByaXN1cy4gTmFtIGV1IGVsaXQgZXQgbWFzc2EgY29tbW9kbwpzY2VsZXJpc3F1ZSBxdWlzIGVn
ZXQgZXguIEluIHNpdCBhbWV0IG1pIHNlbS4gUGhhc2VsbHVzIGVmZmljaXR1ciB1bHRyaWNpZXMg
ZXgsCnZlbCBpbnRlcmR1bSBzZW0gZGlnbmlzc2ltIHZlbC4gSW50ZWdlciBwb3N1ZXJlIG5pYmgg
cmlzdXMuCg==

# Use `-j` / `--join-output` to concatenate newline outputs from `jaq`,
# whilst the base64 encoded newlines are preserved:
$ base64 <lorem.txt | jaq -Rrj @base64d
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus nec justo sit
amet erat porta condimentum vitae ut risus. Nam eu elit et massa commodo
scelerisque quis eget ex. In sit amet mi sem. Phasellus efficitur ultricies ex,
vel interdum sem dignissim vel. Integer posuere nibh risus.

---

FWIW jq is known to be bad at handling some base64 content when it decodes to binary instead of UTF-8 string: jqlang/jq#1931

$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64
57432c2f68706f20efbfbd1acd8032efbfbd4029efbfbdefbfbd3e5f0eefbfbd7e0e580122efbfbdefbfbd19550a

$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d19550a

# Use `-j` to avoid trailing newline `0a` from output
# (NOTE: `-j` is still required, even with `echo -n` that avoids appending a trailing new line to the base64 given to jaq):
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jaq -Rrj '@base64d' | xxd -p -c64
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d1955

# jaq 3.0.0 is required, earlier releases aren't compatible with decoding bytes:
# https://github.com/01mf02/jaq/releases/tag/v3.0.0-alpha
$ echo -n "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jaq-2.3 -Rrj '@base64d'
Error: "invalid utf-8 sequence of 1 bytes from index 8"

The linked issue also has a comparison with gojq tested against a larger HTML document instead of binary base64 which it handles properly while jq does not. gojq uses -j which results in the same success with jaq when using -j, as such it seems jaq is already matching expectations for correctness?

[tianon]

I researched it and learned that whether newlines should be tolerated in base64 encoded text is debatable.

FWIW, RFC 4648 (the base64 RFC) has the following to say about newlines in base64:

Implementations MUST NOT add line feeds to base-encoded data unless the specification referring to this document explicitly directs base encoders to add line feeds after a specific number of characters.

So while many encoders add newlines (base64 --wrap=0 ✊) and many decoders can handle them, they are "officially" actively discouraged unless the wrapping format requires them (MIME, etc), and that's probably part of the justification for jq's own @base64d not supporting them:

$ printf 'hello this is a long line of text that should be long enough to cause newlines in the base64\n' \
| base64

aGVsbG8gdGhpcyBpcyBhIGxvbmcgbGluZSBvZiB0ZXh0IHRoYXQgc2hvdWxkIGJlIGxvbmcgZW5v
dWdoIHRvIGNhdXNlIG5ld2xpbmVzIGluIHRoZSBiYXNlNjQK

$ printf 'hello this is a long line of text that should be long enough to cause newlines in the base64\n' \
| base64 \
| jq-1.8.1 --slurp --raw-input '@base64d'

jq: error (at <stdin>:2): string ("aGVsbG8gdG...) is not valid base64 data

$ # arguably easily "fixed" via gsub
$ printf 'hello this is a long line of text that should be long enough to cause newlines in the base64\n' \
| base64 \
| jq-1.8.1 --slurp --raw-input 'gsub("\n+"; "") | @base64d'

"hello this is a long line of text that should be long enough to cause newlines in the base64\n"

(ie, I think this is maybe more of a jq feature request than being jaq-specific, right?)

It's also worth noting that the RFC only affords newlines, not arbitrary whitespace or any non-base64 characters. 👍

[polarathene]

arguably easily "fixed" via gsub

If you do not use --slurp but instead -j / --join-output you do not need the gsub:

$ printf 'hello this is a long line of text that should be long enough to cause newlines in the base64\n' | base64 | jq -j --raw-input '@base64d'

hello this is a long line of text that should be long enough to cause newlines in the base64

# Otherwise without `-j` you get individual base64 lines input decoded:
$ printf 'hello this is a long line of text that should be long enough to cause newlines in the base64\n' | base64 | jq --raw-input '@base64d'

"hello this is a long line of text that should be long eno"
"ugh to cause newlines in the base64\n"

Which is what my previous comment pointed out with the equivalent jaq command before/after where -s / --slurp was the problem 😅

[tianon]

Sure, that's true, but in my use case, the base64 I'm passing to @base64d doesn't come from another program but is a pre-existing field in a JSON object and might contain newlines (see https://oci.dag.dev/?image=tianon/test@sha256:cd10299aa738f0e60080632a09493daa783b38a2dc67e51ecad7374b91604fac for my crimes -- however with my OCI image-spec maintainer hat on, I'd lean very heavily on that RFC 4648 text and say this is actually non-compliant and any client would be 100% valid to treat this the same as it would any other invalid data field contents 😄).

Edit: again with example output, to make this more clear:

$ crane manifest tianon/test@sha256:cd10299aa738f0e60080632a09493daa783b38a2dc67e51ecad7374b91604fac \
| jq-1.8.1 -r '.manifests[].data | @base64d'

jq: error (at <stdin>:28): string ("\n\n\n\n\n...) is not valid base64 data

$ crane manifest tianon/test@sha256:cd10299aa738f0e60080632a09493daa783b38a2dc67e51ecad7374b91604fac \
| jq-1.8.1 -r '.manifests[].data | gsub("\n+"; "") | @base64d'

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json",...