Does Zarr.jl support authentication ?

[alex-s-gardner]

Looking around it doesn't look like Zarr.jl supports authentication. I'm looking to lazily load the ERA5-land hourly analysis ready data. Is this possible with Zarr.jl?

[lazarusA]

check:

• https://juliaio.github.io/Zarr.jl/dev/tutorials/s3examples
• https://github.com/JuliaCloud/AWS.jl#modifying-functionality

[alex-s-gardner]

perfect, thanks!

[lazarusA]

so, how does it work? if you have a MWE it would be nice to add it to the documentation.

[alex-s-gardner]

I vibed up an AuthenticatedHTTPStore <: Zarr.AbstractStore Type, Should something like this live in Zarr?

[lazarusA]

I think we should add it to the docs at least. Also note the zarr.jl already provides a CachingStore, so that you don't need to code your own, https://juliaio.github.io/Zarr.jl/dev/UserGuide/caching#Persistent-store-caching.

[alex-s-gardner]

Not sure this makes for a good example... it's too complicated with the current Zarr implimentation:

using Zarr                                                                                               
  using HTTP
  using OpenSSL: OpenSSL                                                                                   
  using Dates                                               
                                                                                                           
  # --- Authentication ---
  # ERA5-Land ARCO stores require a free CDS API key:                                                      
  #   1. Register at https://cds.climate.copernicus.eu/                                                    
  #   2. Go to your profile page: https://cds.climate.copernicus.eu/profile                                
  #   3. Copy your "Personal Access Token"                                                                 
  #   4. Set it as an environment variable:                                                                
  #        export CDS_API_KEY="your-token-here"                                                            
  token = ENV["CDS_API_KEY"]                                                                               
                                                                                                           
  # ERA5-Land geo-chunked 2m temperature store                                                             
  url = "https://arco.datastores.ecmwf.int/cadl-arco-geo-007/arco/reanalysis_era5_land/sfc-2m-temperature/g
  eoChunked.zarr"                                                                                          
   
  # --- Minimal authenticated store (Zarr.AbstractStore subtype) ---                                       
  struct MyHTTPStore <: Zarr.AbstractStore                  
      url::String                                                                                          
      headers::Vector{Pair{String,String}}                                                                 
  end
                                                                                                           
  function Base.getindex(s::MyHTTPStore, k::String)                                                        
      r = HTTP.request("GET", "$(s.url)/$(k)", s.headers;
                       status_exception=false, socket_type_tls=OpenSSL.SSLStream)                          
      r.status == 404 && return nothing                     
      r.status >= 300 && error("HTTP $(r.status): $(String(r.body))")                                      
      return r.body                                                                                        
  end                                                                                                      
                                                                                                           
  # Required interface stubs for read-only access           
  Zarr.storagesize(::MyHTTPStore, ::String) = 0
  Zarr.subdirs(::MyHTTPStore, ::String) = String[]                                                         
  Zarr.subkeys(::MyHTTPStore, ::String) = String[]                                                         
  Zarr.isinitialized(s::MyHTTPStore, i::String) = s[i] !== nothing                                         
  Base.setindex!(::MyHTTPStore, v, k::String) = error("read-only")                                         
                                                                                                           
  # --- Open the store ---                                  
  store = MyHTTPStore(url, ["Authorization" => "Bearer $token"])                                           
  zg = Zarr.zopen(store; consolidated=true, fill_as_missing=false)                                         
                                                                                                           
  # Inspect what's inside                                                                                  
  println("Variables: ", keys(zg.arrays))                                                                  
  println("Subgroups: ", keys(zg.groups))                                                                  
   
  # Read coordinates                                                                                       
  lat = zg["latitude"][:]                                   
  lon = zg["longitude"][:]                                                                                 
  time_raw = zg["time"][:]
                                                                                                           
  println("Lat range: $(extrema(lat))")                     
  println("Lon range: $(extrema(lon))")                                                                    
  println("Time units: ", zg["time"].attrs["units"])        
  println("Array shape (t2m): ", size(zg["t2m"]))                                                          
                                                                                                           
  # Pick a point: Greenland summit (~72.6°N, -38.5°E)                                                      
  target_lat, target_lon = 72.6, -38.5                                                                     
  lat_idx = argmin(abs.(lat .- target_lat))                                                                
  lon_idx = argmin(abs.(lon .- target_lon))                                                                
  println("Nearest grid point: $(lat[lat_idx])°N, $(lon[lon_idx])°E")                                      
                                                                                                           
  # Read first 24 hours of t2m at that point                                                               
  # Storage order is (longitude, latitude, time) despite metadata saying (time, lat, lon)                  
  t2m = Float64.(zg["t2m"][lon_idx, lat_idx, 1:24])                                                        
  println("First 24h of t2m (K): ", round.(t2m; digits=1))