attempt to autodetect an s3 compatible URL

[ianhi]

Addresses part of #3495 by attempting to autodetect if a URL is S3 compatible or not.
With this change the following "just works"

import zarr
z = zarr.open("https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr", storage_options={"anon":True})
print(list(z.keys()))

This is equivalent to:

  zarr.open(
      "s3://idr/zarr/v0.5/idr0062A/6001240_labels.zarr",
      storage_options={
          "anon": True,
          "client_kwargs": {"endpoint_url": "https://uk1s3.embassy.ebi.ac.uk"}
      }
  )

but now zarr handles that splitting for the user

• Add unit tests and/or doctests in docstrings
• [NA?] Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[d-v-b]

how reliable is this?

[d-v-b]

@martindurant and @kylebarron relevant to your interests.

[martindurant]

I don't understand: why would someone provide an HTTP URL for something that you need the S3 protocol for? The storage options only make sense to s3, not http, so the caller already knows.

[d-v-b]

I'm inclined against the changes in this PR -- "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr" is unambiguously an https URL, and so it makes sense to use an HTTP-based storage backend to access it. For example, "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr/zarr.json" works just fine. Directory listing does not work, but that's pretty common for zarr data over http.

[ianhi]

I ran into this today when trying to load data from: https://idr.github.io/ome-ngff-samples/

the "copy s3 url" button gives a one line URL: https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

which I naively expected to work in zarr. So this PR is basically protection against that situation for other naive users possibly sourcing URLs that look like that.

Ideally zarr would just work and protect me from this, or at least it should have thrown an error, rather silently failing by not returning any data but otherwise working

[d-v-b]

the "copy s3 url" button gives a one line URL: https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

IMO a button that says "copy s3 url" should give you an s3 url, not an http url

[ianhi]

IMO a button that says "copy s3 url" should give you an s3 url, not an http url

I agree! but I suspect this is not the only place this is happening and we can't fix it everywhere

Also is it possible to specify the endpoint url and bucket and path in one s3: url?

[martindurant]

Exactly what @d-v-b says. That URL is just wrong, it doesn't point to anything! If you poke it, the returned headers (of the 404) does contain an "x-amz-*" key, but that might be true from a real website that is hosted somewhere within AWS.

[ianhi]

hmm. ok. So should zarr have raised an error for what I did here: #3495 when it loaded as an http store that didn't fully behave like one?

and am I right that there's no one liner for that website to return becuase you can't specify an endpoint-url within the s3: syntax?

[ianhi]

I am updating the URLS over there: IDR/ome-ngff-samples#28

going to close this now as I hear the arguments against this change. I would sitll like to find a way to make this more transparent to users but this is clearly not the right approach.

[joshmoore]

and am I right that there's no one liner for that website to return becuase you can't specify an endpoint-url within the s3: syntax?

That's my understanding.