ZEP11 - Evolving Zarr Governance - discussion thread

[rabernat]

This issue is for the community to comment on, and the stakeholders to vote on, ZEP11.

Actual updates to governance docs are in zarr-developers/governance#52 zarr-developers/governance#54.

Timelines

ZEP0 does not prescribe a specific timeline for review. This has been a problem in the past. To make sure this process resolves, we will follow this timeline.

• Two weeks for discussion - March 5 - March 19. During this time we will take comments and the authors will revise the ZEP into its final form
• Two weeks for voting - March 19 - April 2

[maxrjones]

Thanks for working on this!

Prompted by Davis' Zarr Summit talk, my understanding of the Zarr specification shifted from defining an "on-disk" file layout to defining the meaning of keys in a storage backend, more like a protocol or API. This understanding aligned with @TomNicholas and others saying that Icechunk does not disagree with the Zarr specification (e.g., zarr-developers/zarr-developers.github.io#131). But https://zarr.dev/blog/governance-update/ and zarr-developers/governance#52 states that the Zarr specification is a collection of documents defining the on-disk Zarr format. Can anyone clarify whether the Zarr spec does/does not define the physical storage layout?

I was uncertain if this is the right place to ask, so please direct me elsewhere if appropriate.

[rabernat]

Max, sorry to kick the can down the road, but I would say that is a question for the proposed spec committee to address. The whole point here is to separate project governance from spec governance. This update is about the project as a whole, not the spec.

[maxrjones]

Max, sorry to kick the can down the road, but I would say that is a question for the proposed spec committee to address. The whole point here is to separate project governance from spec governance. This update is about the project as a whole, not the spec.

That makes sense, thanks for your response. Given that it's unsettled, might I suggest changing the language to "The Zarr specification defines the Zarr format for N-dimensional typed arrays", which is the language used in the specification document? I think that would leave it to the Zarr spec committee to address and eliminate any confusion from the blog/governance document.

[rabernat]

Yes, that feedback can be incorporated in the final accepted ZEP11. There is no intention to define / redefine the spec in this ZEP.

Let's keep the conversation here focused on practical issues related to governance.

[maxrjones]

I've read through the document in detail and think it's a huge improvement! thanks for proposing the changes. I particularly appreciate the formation of the Spec Committee.

I have some suggestions, but would prefer the existing changes be merged soon over protracted discussion about these comments.

ZSC members who do not actively engage with the ZSC duties are expected to resign.

The only mechanism listed for a ZSC members term ending is resignation. While hopefully this is never needed, I think there should also be an involuntary mechanism for a term ending. I'd recommend at least the inactivity model from NumPy.

(AI-generated summary) How other open source projects handle involuntary removal of steering council members

The Zarr governance doc only mentions resignation — here's how several well-known projects handle involuntary removal:

NumPy / SciPy / MNE-Python / SciML

These all distinguish two removal triggers:

1. Inactivity — If a council member becomes inactive for one year, they are approached first. If they don't plan to return, they are removed by council vote. If they expect to return, they get a one-year grace period before removal proceeds without further grace period.
2. Harmful conduct — The council reserves the right to eject members deemed actively harmful to the project's well-being, after attempts at communication and conflict resolution have failed.

Python (PEP 8016)

Python distinguishes between ejecting a core team member and removing a council member:

• Core team removal — Requires a steering council vote with at least a two-thirds supermajority (4:1 with 5 members; a 3:2 vote is insufficient). Intended for egregious and ongoing code of conduct violations.
• Council removal — In exceptional circumstances, the broader core team may remove a sitting council member, or the entire council, via a vote of no confidence. Triggered when one core team member calls for it publicly and another seconds it. The vote lasts two weeks; a two-thirds majority of voters is required for it to succeed.

ClearlyDefined (Open Source Initiative)

Members are removed if they resign or are inactive for more than 6 months — no council vote required.

SLSA (OpenSSF)

Focuses less on the removal trigger and more on ensuring replacement: when a member resigns or is removed, an election must be held, with nominations and decisions publicly recorded for transparency.

---

Suggested pattern for Zarr

The most common approach across the scientific Python ecosystem is two removal tracks:

1. Inactivity — after a defined period (e.g. 1 year), with a check-in first, the council votes to remove the inactive member.
2. Harmful conduct — the council can eject a member by vote (potentially requiring a supermajority) after communication and conflict resolution have failed.

The Python model adds a useful third track — no-confidence votes by the broader community — which provides recourse if the council itself is the problem. Given the ZSC has only five members, it's also worth specifying whether a member being considered for removal must recuse themselves from that vote, and whether a supermajority is required.

Manage responsibilities which do not belong to a specific Affiliated Software Project (e.g. the Zarr website, the zarr-developers GitHub organization, social media accounts, and similar Zarr-owned resources) in order to ensure smooth operations and effective collaboration.

I think the Zarr Project would be better suited if the Zarr website was considered a project, rather than a responsibility of the ZSC. Concretely, I expect that ZSC reviews would be a bottleneck for a revamp of the website (e.g., zarr-developers/zarr-developers.github.io#167), which is direly needed. I would be glad to recruit a few people from across software projects to participate in a Core Developer Group for the website. I think there should be an avenue for other aspects independent of software to become projects if new people who are skilled and trusted want to lead them (e.g., social media) and an avenue for them to be reclaimed by the ZSC if the project model fails or gets abandoned. The website project could serve as a trial/model for other non-software projects.

Centralized organization: All Affiliated Software Projects live within the zarr-developers GitHub organization, providing a central entry point and allowing projects to share resources.

It's currently ambiguous whether non-affiliated software projects can also exist within the zarr-developers GitHub organization. In practice, it seems obvious that non-affiliated projects can exist in the organization, but it seems worth stating and clarifying how they manage membership under this governance model.

[d-v-b]

thanks for working on this @rabernat I agree that the proposed changes are an improvement.

I would prefer as much as possible to model our governance structure on something that has worked for similar projects, with departures from that model as needed. I don't know much about this so I read the NumPy governance docs and the Apache governance docs (used by Arrow, and presumably Parquet). Some notes from my reading:

• Both the Apache and NumPy model are explicitly meritocratic. Positions of responsibility (Steering council membership for NumPy / PMC membership in the Apache model) are filled by people who contribute extensively to the project. Why don't we adopt the same model for the Zarr Steering Council? We don't currently define criteria for membership.

• I share Max's concern about the website being a responsibility for the ZSC. NumPy has a web team. I think we could do something similar.

[joshmoore]

Thanks to @rabernat for having kicked off this effort and gotten it out to the community 🙌🏽 Thanks also to @d-v-b and @maxrjones for feedback. 👍 (More welcome!)

A few specific responses from my side:

Affiliated projects / the website

I had originally proposed treating the website as an affiliated project as well. However, I came to agree with Ryan that this is something we can adopt later if needed. For now, I would prefer to focus on making the minimal changes required so that responsibility for the website can move cleanly from the ZSC to the appropriate project, rather than blocking getting that project set up.

For example, I'd suggest changing “which do not belong to a specific Affiliated Software Project (e.g. the Zarr website” to “...e.g., at the time of this writing, the website…”. Once ZEP11 is active, engaged individuals can be gathered together to take over new or existing repositories — that’s pretty much the whole point!

Unaffiliated projects in the GitHub organization

I would strongly prefer to avoid allowing unaffiliated projects to live directly within the zarr-developers GitHub organization. Doing so makes the relationship between the organization’s admins and the projects unclear, which could lead to confusion about responsibilities, permissions, and oversight. Supporting unaffiliated projects would likely require a separate set of governance guidelines, which seems unnecessarily complicated at this stage. My preference is that every repository in the zarr-developers org should either clearly belong to an affiliated project or, if not, default to belonging under the ZSC. This keeps the organizational structure straightforward and ensures accountability is well defined.

Voting rules and other structural changes

I agree that some of the suggestions—particularly around voting mechanisms—are topics that we are going to need to spend more time on. However, I see a practical difficulty in trying to resolve them within this ZEP. At the same time that we would be defining new voting procedures, we are also changing who could vote through the introduction of new projects and roles. That makes it difficult to settle those questions here in a stable way. My strong preference would be to defer those broader governance decisions to a later proposal once the new structure is in place.

Additional mechanisms such as votes of no confidence

Mechanisms like a vote of no confidence are sensible governance tools in principle. That said, introducing them right now could interact in unexpected ways with the current proposal. For example, we are intentionally making the bar for projects to join fairly low in order to encourage participation. If stronger governance mechanisms are introduced simultaneously, we might find ourselves needing to reconsider what threshold projects must meet before joining. That becomes a much larger governance discussion. In the interest of unblocking the specification work, I would again strongly prefer to first establish the new specification project and then iterate on governance once we have practical experience with the structure.

Adopting governance models from other projects

And I agree it is always best to look to governance structures from other communities. At the same time, we should be careful that any model we adopt matches the “topography” of the Zarr ecosystem. For example, in the Python governance model (AFAIK) the Steering Council oversees a single body of core developers and is therefore clearly accountable to that group. Our structure is likely somewhat different, since we are coordinating multiple projects and implementations (i.e., more like the Apache model, in which the core devs group rather than the ZSC maps to the PMC.) Because of that, direct adoption may not always translate cleanly, but definitely happy to read through any concrete proposals.

---

Overall, my hope with this ZEP (as I think is true of everyone in the ZSC) is to resolve the immediate governance bottleneck around specification development, and then we can continue iterating on broader governance questions in follow-up proposals once the new structure is operating.

[d-v-b]

Our structure is likely somewhat different, since we are coordinating multiple projects and implementations (i.e., more like the Apache model, in which the core devs group rather than the ZSC maps to the PMC.) Because of that, direct adoption may not always translate cleanly, but definitely happy to read through any concrete proposals.

As you note, the Apache model seems like a good starting point.

[d-v-b]

I found references to the Apache process in the zep itself and the updated governance docs, so I think I'm satisfied on that point.

[maxrjones]

Thanks @joshmoore, I agree with prioritizing fixing the most pressing governance need of the spec evolution process.

There's a risk that some pressing spec issues (e.g., zarr-developers/zarr-specs#371) will actually take longer to resolve since the Spec Committee is tasked with setting up its own governance structure after ZEP11 is reviewed and approved. To me, this risk feels especially acute given that the initial Spec Committee draws from the same pool of volunteers who were stretched thin under the existing structure. This risk could be mitigated by either the ZSC fixing regressions like zarr-developers/zarr-specs#371 during the ZEP11 feedback solicitation process, or a guidance on a transition process for the time it takes for the Spec Committee to self organize, so that progress is not put on hold entirely.

[joshmoore]

Thanks for raising this — I agree that we should avoid any situation where pressing specification issues are blocked while governance transitions are happening.

The intent of this ZEP is primarily to restore a workable voting structure for the specification, rather than to fully define the long-term governance of that structure. In practice, the first step is to adopt the ZEP so that we can move to a voting body that is small enough to function effectively. One of the immediate follow-ups would then be to reduce the membership to reflect currently active participants so that votes are not blocked by inactive members.

Once that is in place, we would again be in a functional decision-making state for specification issues. From there, the Spec Committee could iterate on its own governance in a more deliberate way, including questions around voting procedures, membership expectations, and similar topics.

More generally, I think it is useful to think of governance changes as moving from one functioning voting state to another, rather than trying to design the final structure all at once. The priority here is to re-establish a decision-making group that can resolve specification questions. Once that is operating, governance improvements can continue incrementally without blocking ongoing work.

Regarding the transition period specifically, I agree it would be helpful to ensure that specification progress is not put on hold. If a pressing issue arises during that window, the goal would be for the newly established Spec Committee to address it as soon as it is operational. My expectation is that this should be a relatively short transition, since the ZEP mainly establishes the body rather than requiring a large amount of additional process design before it can begin operating.

Overall, the goal with this proposal is simply to unblock specification governance, and then continue improving the structure through subsequent iterations once we are again able to make decisions efficiently.

[maxrjones]

thanks again for your detailed responses @joshmoore!

What's the timeline for adopting ZEP11?