Failed to determine total size of zarr3 shard using head

[dchen116]

When trying to access the https://s3proxy.janelia.org/funceworm/embryo/20220426i_Janelia_sampleA_OTO_6x6x6nm.zarr/ dataset with neuroglancer (https://neuroglancer-demo.appspot.com/), the following error is encountered:

Error retrieving chunk [object Object]:33,181,119: Error: Failed to determine total size of https://s3proxy.janelia.org/funceworm/embryo/20220426i_Janelia_sampleA_OTO_6x6x6nm.zarr/c/3/5/1 in order to fetch suffix {"suffixLength":524292}

This error comes from:

neuroglancer/src/kvstore/http/read.ts

Line 95 in 186674a

`Failed to determine total size of ${store.getUrl(key)} in order to fetch suffix ${JSON.stringify(byteRangeRequest)}`,

This was introduced in #696.

The head response is as follows:

>curl --head https://s3proxy.janelia.org/funceworm/embryo/20220426i_Janelia_sampleA_OTO_6x6x6nm.zarr/c/3/5/1
HTTP/2 200 
server: nginx
date: Wed, 29 Jan 2025 20:16:54 GMT
content-type: application/octet-stream
content-disposition: attachment; filename="1"
x-proxy-cache: MISS
access-control-allow-origin: *
access-control-allow-credentials: true
access-control-allow-methods: GET, HEAD, OPTIONS
access-control-allow-headers: *
access-control-expose-headers: Range, Content-Range

The head response does not contain content-length information.

[mkitti]

We have resolved this at Janelia in https://github.com/JaneliaSciComp/x2s3 by returning the content-length information in head requests.

If a HEAD request fails to determine the length of the shard, perhaps a suffix range request should be attempted?

[jbms]

@mkitti Just to be clear, are you saying that the original issue reported by @dchen116 is now resolved because it has been fixed in the server?

Potentially we can add more workarounds but if we can fix the server to behave more consistently that is better. HEAD is supposed to return the same response headers as GET.

Suffix length requests are not CORS safelisted and therefore require an OPTIONS preflight request (one additional round trip), and furthermore often servers don't handle those correctly.

A GET response with a single-byte byte range is probably a better bet as a workaround since some servers don't support HEAD at all.

[mkitti]

As I mentioned in #704, we have an internal storage system that has some limited S3 capabilities. On top of this, we (specifically @krokicki) have built a proxy service which expands those capabilities when we want to expose data to the outside world. However, this requires some whitelisting of header information.

This issue arose because we had previously implemented the capability for the negative byte range requests, including the CORS requests, but had not implemented delivering content-length via HEAD. Thus datasets which previously worked with Neuroglancer were no longer viewable after the recent changes.

[jbms]

It would be easy to implement a workaround for the issue in #704 as discussed there.

However, I'm trying to understand if this current issue can be closed given your comment that you have "resolved this at Janelia".

[mkitti]

I'm trying to understand if this current issue can be closed given your comment that you have "resolved this at Janelia".

Yes. I'm not aware of anyone else having this specific issue. The ticket can be closed.