Erroneous auto-resolving percent encoded characters within URLs

[rgchris]

This is a longstanding issue in Redbol languages. Take the URL https://duckduckgo.com/?q=hergé+%26+tintin. In most variants, this URL is transformed by the loader:

rebol2> https://duckduckgo.com/?q=hergé+%26+tintin
== https://duckduckgo.com/?q=hergé+&+tintin

rebol3> https://duckduckgo.com/?q=hergé+%26+tintin
== https://duckduckgo.com/?q=herg%C3%A9+&+tintin

red> https://duckduckgo.com/?q=hergé+%26+tintin
== https://duckduckgo.com/?q=herg%C3%A9+&+tintin

ren-c> https://duckduckgo.com/?q=hergé+%26+tintin
== https://duckduckgo.com/?q=hergé+%26+tintin

The imperative here is data integrity (Redbol is, after all, a data-first format). In auto-resolving the percent encoding, the URL changes value and means something different when resolved. Also note that in a browser, while the é may be converted to percent encoding as part of the request, it is still displayed as é in the location bar. This behaviour may also be incompatible with non-HTTP URL schemes. My contention is that URLs should be dumb strings that round-trip input-to-output and should only be resolved when passed to a scheme—that includes URLs that would appear to be malformed (e.g. meta:thing=100% or http://latin1.com/?q=herg%E9); anything else is incorrect.

I have an open ticket on Red and have made the case for Ren-C (which now exhibits the correct behaviour). There are related tickets for Rebol 3 (#1644, #2012 and #2207).

[Oldes]

This is difficult topic... problem is, that your url is mixing url encoded value with not encoded.

Correct way how to get the correct data in above case is:

query: "hergé & tintin" ;; notice that it is human readable content
url: join https://duckduckgo.com/?q= enhex/uri query
;== https://duckduckgo.com/?q=herg%25C3%25A9+%2526+tintin
as string! url
;== "https://duckduckgo.com/?q=herg%C3%A9+%26+tintin"
parse read url [thru "<title>" return copy title to "<"]
;== "hergé &amp; tintin at DuckDuckGo" ;<-- expected query result

[rgchris]

URLs (and emails) are always going to be a challenge in determining how much content they consume. Personally, I'd be ok with them eating square brackets or parenthesis if there's a reasonable amount of data that supports that usage. (I think parenthesis are more common—used by Wikipedia, for example)

(from RFC 3986)

[rgchris]

Wikipedia example: https://en.wikipedia.org/wiki/Red_(programming_language)

[Oldes]

So you think that Rebol's URL scaner should accept such an URL?

[Oldes]

Actually it is not possible.. as the lexer is using [( chars as delimiters, so the Scan_URL would not see these.

[rgchris]

So you think that Rebol's URL scaner should accept such an URL?

Maybe one day. Unless someone has a strong use case for them (the Wikipedia example is the only common one I'm aware of). The &=@ (etc) and unicode handling is more important.

[Oldes]

@rgchris when I modify the code not to provide automatic conversion, then decode-url fails, because it accepts only valid URLs...
https://duckduckgo.com/?q=hergé is not valid (because é is not encoded)

So do you really think, that current version is the wrong one?

[rgchris]

I believe so.

My inclination is that it's up to each individual protocol to interpret and/or validate each URL. For those that use DECODE-URL, that would include 'pre-flight' conversion to percent encoding for non-ascii characters, possibly even rejecting URLs that are non-conformant. Even then, you can't assume that the destination server will conform exactly to URL conventions.

I'll add to this that 'pre-flight' conversion may be different dependent on which part of the URL the non-ascii characters appear: in the PATH, TARGET, or QUERY, this would be percent encoding, for the HOST value, this could be a conversion to Punycode. Thus the following behaviour currently exhibited by Red/Rebol 3 is misleading or incorrect:

>> http://müsic.example/motörhead
== http://m%C3%BCsic.example/mot%C3%B6rhead

Again, the precedent for this is how browser location bars handle URLs and how that shapes common expectations of how URLs are presented and shared. URLs are to an extent an human-to-network interface, it would seem that LOAD aggressively converting Unicode characters to percent-encoding is less useful in general. Round-trip is the safest policy.

This answer has a bunch of information too: Can urls have UTF-8 characters?

Tests

URLs should in most (if not all) cases round-trip.

Some tests of DECODE-URL:

decode-url http://user%40rebol.com:blah@www.rebol.com/  ; Note 1
[scheme: 'http user: "user%40rebol.com" pass: "blah" host: "www.rebol.com" path: %/]  ; Note 2

decode-url http://müsic.example/motörhead
[scheme: 'http host: "müsic.example" path: %/ target: %motörhead]

decode-url http://xn--msic-0ra.example/mot%C3%B6rhead
[scheme: 'http host: "xn--msic-0ra.example" path: %/ target: "mot%C3%B6rhead"]  ; Note 3

decode-url https://duckduckgo.com/?q=herg%C3%A9+%26+tintin
[scheme: 'https host: "duckduckgo.com" path: %/ query: "q=herg%C3%A9+%26+tintin"]  ; Note 4

decode-url https://duckduckgo.com/?q=hergé+%26+tintin
[scheme: 'https host: "duckduckgo.com" path: %/ query: "q=hergé+%26+tintin"]  ; Note 5

decode-url https://duckduckgo.com/?q=herg%E9+%26+tintin
[scheme: 'https host: "duckduckgo.com" path: %/ query: "q=herg%E9+%26+tintin"]  ; Note 6

Note 1: From ticket #1644

Note 2: Percent decoding could be used on the USER/PASSWORD fields.

Note 3: If you convert this to a file, you are de-facto percent-decoding the value as FILE! uses percent-encoding internally. I'd also be inclined to say that there should be no equivalence in the way that FILE! and URL! handles percent-encoding—the dynamics of each datatype are different.

Note 4: If you decode the percent encoding as a part of DECODE-URL then you run the risk of changing the meaning of the URL (e.g. %26 -> &)

Note 5: I don't think it is the business of DECODE-URL to percent-encode non-ascii characters. If that is to be done, it should be by—in this case—the HTTP scheme after receiving the decoded URL block.

Note 6: DuckDuckGo in this case assumes Latin 1 and will process as if %E9 = é.

[Oldes]

It would be more useful if you provided expected results... for example this is output of the new modified version:

decode-url http://user%40rebol.com:blah@www.rebol.com/  ; Note 1
;== [scheme: 'http user: "user@rebol.com" pass: "blah" host: "www.rebol.com" path: %/]

decode-url http://müsic.example/motörhead
;== [scheme: 'http host: "müsic.example" path: %/ target: %motörhead]

decode-url http://xn--msic-0ra.example/mot%C3%B6rhead
;== [scheme: 'http host: "xn--msic-0ra.example" path: %/ target: %motörhead]

decode-url https://duckduckgo.com/?q=herg%C3%A9+%26+tintin
;== [scheme: 'https host: "duckduckgo.com" path: %/ query: "q=herg%C3%A9+%26+tintin"]

decode-url https://duckduckgo.com/?q=hergé+%26+tintin
;== [scheme: 'https host: "duckduckgo.com" path: %/ query: "q=herg%C3%A9+%26+tintin"]

decode-url https://duckduckgo.com/?q=herg%E9+%26+tintin
;== [scheme: 'https host: "duckduckgo.com" path: %/ query: "q=herg%E9+%26+tintin"]

[Oldes]

And

parse read decode-url https://duckduckgo.com/?q=herg%E9+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

parse read decode-url https://duckduckgo.com/?q=hergé+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

parse read decode-url https://duckduckgo.com/?q=herg%C3%A9+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

[Oldes]

parse read https://duckduckgo.com/?q=herg%E9+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

parse read https://duckduckgo.com/?q=hergé+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

parse read https://duckduckgo.com/?q=herg%C3%A9+%26+tintin [thru "<title>" return to "<"]
;== "hergé &amp; tintin at DuckDuckGo"

[Oldes]

@rgchris Another issue, when the URL scanner does not normalize the input:

>> c: 1 f2: try [load mold f1: append copy a:/ to char! c]
== a:/%01

>> as string! f1
== "a:/^A"

>> as string! f2
== "a:/%01"

>> f1
== a:/%01 ;<--- because MOLD does not allow to produce an invalid URL

>> f2
== a:/%01

>> f1 = f2
== #(false) ;<--- is this acceptable?

As I said... this topic is problematic.

[Oldes]

Which would be more logical... because for example there is another part of this issue and that is email! datatype, which is now using same escaping like url! although it does not make sense:

>> as email! "hergé"
== herg%C3%A9 ;<-- #(email! "hergé") would be better result

[rgchris]

Yeah, I think the answer is to have a separate set of functions for PERCENT-ENCODING. Module/Library/Whatever

[Oldes]

There are enhex and dehex already for the percent encoding. But important part of this issue is url-parser, which is used in all schemes to convert an URL to a port specification.

[Oldes]

Ok... it should be resolved in these 3 commits:
f442f8a (not using mold in decode-url)
047f9b1 (using construction syntax when needed for emails and urls)
c2abb8b (not using mold to encode HTTP request path)

[Oldes]

And this one b9e0511 which allows decoding URL where is used % char not used for escaping. I'm not sure if it is good, but maybe better than having none as a result.

[Oldes]

I've pushed changes squashed in this commit: ebe607c
Let's see if there will be any issues with it.

[rgchris]

I built with c34a39c and so far looks good.

Using the following: load/as https://httpbun.org/anything/?q=hergé+%26+tintin 'json

c34a39c

#[
    method: "GET"
    args: #[
        q: "hergé & tintin"
    ]
    headers: #[
        Accept: "*/*"
        Accept-Charset: "utf-8"
        Accept-Encoding: "gzip,deflate"
        Host: "httpbun.org"
        User-Agent: "rebol/3.19.2"
        Via: "1.1 Caddy"
    ]
    origin: "..."
    url: {https://httpbun.org/anything/?q=herg%C3%A9+%26+tintin}
    form: #[]
    data: ""
    json: #(none)
    files: #[]
]

e907201 (3.19.0 download)

#[
    method: "GET"
    args: #[
        tintin: ""
        q: "herg�A9 "
    ]
    headers: #[
        Accept: "*/*"
        Accept-Charset: "utf-8"
        Accept-Encoding: "gzip,deflate"
        Host: "httpbun.org"
        User-Agent: "rebol/3.19.0"
        Via: "1.1 Caddy"
    ]
    origin: "..."
    url: "https://httpbun.org/anything/?q=herg%C3A9+&+tintin"
    form: #[]
    data: ""
    json: #(none)
    files: #[]
]

(and red/red@982be03 (Red))

#[
    method: "GET"
    args: #[
        " tintin" ""
        q: "hergé "
    ]
    headers: #[
        Accept: "*/*"
        Accept-Charset: "UTF-8"
        Accept-Encoding: "gzip"
        Host: "httpbun.org"
        User-Agent: "Mozilla/5.0 (Windows NT 6.1; Win64; x64)"
        Via: "1.1 Caddy"
    ]
    origin: "..."
    url: "https://httpbun.org/anything/?q=hergé+&+tintin"
    form: #[]
    data: ""
    json: none
    files: #[]
]