String too large to store inside array when writing Zarr store

[WeilerP]

Report

Description

When I try saving a TreeData as a zarr ZipStore, writing the obst slot fails. My lineage trees (stage/condition dependent) are stored as networkx DiGraphs. The object I am trying to save has approx 1.8 million cells if that could be an issue.

CCing @ilan-gold since the issue is in the anndata writer, so maybe you have a suggestion.

import warnings
import zarr
import treedata as td

tdata = td.read_h5td("tdata.h5td")

with warnings.catch_warnings():
    warnings.simplefilter("ignore", UserWarning)
    store = zarr.storage.ZipStore("tdata.zarr.zip", mode="w")
    data.write_zarr(store=store)
    store.close()

Traceback:

Traceback (most recent call last):
  File ".env/tdatabug-py312/lib/python3.12/site-packages/treedata/_core/treedata.py", line 531, in write_zarr
    write_zarr(store, self, chunks=chunks)
  File ".env/tdatabug-py312/lib/python3.12/site-packages/treedata/_core/write.py", line 199, in write_zarr
    _write_tdata(f, tdata, filename, chunks, **kwargs)
  File ".env/tdatabug-py312/lib/python3.12/site-packages/treedata/_core/write.py", line 87, in _write_tdata
    _write_elem(f, key, _serialize_axis_trees(getattr(tdata, key)), dataset_kwargs=kwargs)
  File ".env/tdatabug-py312/lib/python3.12/site-packages/treedata/_core/write.py", line 48, in _write_elem
    ad.io.write_elem(f, k, elem, dataset_kwargs=dataset_kwargs)
  File ".env/tdatabug-py312/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 518, in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
  File ".env/tdatabug-py312/lib/python3.12/site-packages/anndata/_io/utils.py", line 272, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".env/tdatabug-py312/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 389, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/tdatabug-py312/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 77, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".env/tdatabug-py312/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 1321, in write_scalar_zarr
    a[...] = np.array(value)
             ^^^^^^^^^^^^^^^
TypeError: string too large to store inside array.
Error raised while writing key 'obst' of <class 'zarr.core.group.Group'> to /

Versions

| Package  | Version |
| -------- | ------- |
| treedata | 0.2.5   |

| Dependency         | Version        |
| ------------------ | -------------- |
| typing_extensions  | 4.15.0         |
| session-info2      | 0.4            |
| google-crc32c      | 1.8.0          |
| scipy              | 1.16.3         |
| charset-normalizer | 3.4.4          |
| fast-array-utils   | 1.3.1          |
| cloudpickle        | 3.1.2          |
| PyYAML             | 6.0.3          |
| packaging          | 26.0           |
| scanpy             | 1.12           |
| donfig             | 0.8.1.post1    |
| setuptools         | 82.0.0         |
| psutil             | 7.2.2          |
| natsort            | 8.4.0          |
| Jinja2             | 3.1.6          |
| pyarrow            | 23.0.0         |
| h5py               | 3.15.1         |
| numcodecs          | 0.16.5         |
| six                | 1.17.0         |
| llvmlite           | 0.46.0         |
| threadpoolctl      | 3.6.0          |
| anndata            | 0.12.10        |
| cuda-pathfinder    | 1.3.4 (12.9.4) |
| xarray             | 2026.1.0       |
| pytz               | 2025.2         |
| MarkupSafe         | 3.0.3          |
| fsspec             | 2026.2.0       |
| sparse             | 0.17.0         |
| zarr               | 3.1.5          |
| pandas             | 2.3.3          |
| python-dateutil    | 2.9.0.post0    |
| networkx           | 3.6.1          |
| numba              | 0.63.1         |
| legacy-api-wrap    | 1.5            |
| dask               | 2026.1.2       |
| numpy              | 2.3.5          |
| toolz              | 1.1.0          |
| cuda-bindings      | 12.9.4         |

| Component | Info                                                     |
| --------- | -------------------------------------------------------- |
| Python    | 3.12.12 (main, Feb  3 2026, 22:51:04) [Clang 21.1.4 ]    |
| OS        | Linux-4.18.0-425.19.2.el8_7.x86_64-x86_64-with-glibc2.28 |
| Updated   | 2026-04-21 15:29                                         |

[ilan-gold]

Presumably this is an issue with hdf5 files as well:

import numpy
np.array("a" * (2 ** 29)).shape

but

import numpy
np.array("a" * (2 ** 28)).shape

works (at least on my machine, the numbers seem kind of arbitrary).

Are you storing base pairs or something in uns?

[colganwi]

Currently TreeData stores nx.DiGraphs as json strings which become massive for >1M cell datasets, exceeding the size limit allowed by Zarr. The fix will be creating a proper nx.DiGraph writer that stores elements in separate Zarr groups.

[ilan-gold]

Currently TreeData stores nx.DiGraphs as json strings which become massive for >1M cell datasets, exceeding the size limit allowed by Zarr. The fix will be creating a proper nx.DiGraph writer that stores elements in separate Zarr groups.

@colganwi Is this stored in uns? scverse/anndata#1979 might be of relevance.

UPDATE:

treedata/src/treedata/_core/read.py

Lines 93 to 95 in 0091e87

	for k in ["obst", "vart"]:
	if k in f:
	d[k] = _parse_axis_trees(_read_elem(f[k]))

ok it's the elements themselves

[colganwi]

#86 should fix this issue.

@ilan-gold the nx.DiGraphs are stored in top level obst and vart fields rather than uns. The serializing code is very similar to the way uns is handled.

[ilan-gold]

@colganwi Just spitballing, would awkward arrays (or recarrays) work?

from __future__ import annotations

import awkward as ak
import networkx as nx

G = nx.Graph()
G.add_edges_from([
    (0, 1, {"weight": 1.0}),
    (1, 2, {"weight": 2.0}),
    (2, 0, {"weight": 1.0}),
])
d = nx.to_dict_of_dicts(G)
print(ak.from_iter([{str(k): v for k, v in row.items()} for row in d.values()]))

[ilan-gold]

import numpy as np
import networkx as nx

G = nx.balanced_tree(r=2, h=5, create_using=nx.DiGraph)

# Build edge list (source, target)
edges = [(u, v) for u, v in G.edges()]

# Define structured dtype
dtype = np.dtype([("source", np.int64), ("target", np.int64)])

# Create structured array
rec = np.array(edges, dtype=dtype)

# Optional: view as recarray for attribute-style access
rec = rec.view(np.recarray)

print(rec)
print(rec.source)
print(rec.target)

Maybe recarrays would work as well? zarr is converging toward an "official" structured data type implementation: zarr-developers/zarr-python#3781, which could be a nice target for this.

The catch is recarrays are fixed length per-row. I think awkward would not require this.

[colganwi]

Thanks for the suggestion @ilan-gold! I've switched to awkward arrays for storing nx.DiGraphs in zarr files, utilizing ad.io.write_elem (#86). It's a bit slower than the json string but won't suffer from the large string issue. What do you think of this approach?

[WeilerP]

Thanks both, for the quick input! @colganwi, I can save the TreeData object to a zarr ZipStore with your fix!

[colganwi]

I just released the updated zarr format with v0.3.0.

@ilan-gold thanks for your feedback! Node/edge attributes are unstructured so not a good fit for recarrays. Awkward arrays work but I don't think they are worth the added dependency. I've decided to just json encode the elements in a numpy array. Here is the spec. There is typically a small number of keys (i.e. depth, time) which are defined for all nodes so the overhead of converting to columns should be fairly low.

@WeilerP Since the format has changed you'll need to convert any zarr files you have. Sorry for the inconvenience.