Epic: Feature Flags Migration

[RayBB]

Epic: Feature Flags Migration

Goal: Replace the existing feature-flag system with a lightweight, pydantic-settings-based approach that we fully control — making it easier to migrate HTML pages and deprecate the old internal-tooling dependency.

Context

Current System (Infogami)

Right now Open Library's feature flags live in conf/openlibrary.yml under a top-level features: key. Each flag is evaluated per-request via infogami.utils.features:

Flag	Config Value	Effective Gate	Used Where
debug	enabled	always on	Python: features.is_enabled("debug")
dev	enabled	always on	Templates + Python ("dev" in ctx.features)
history_v2	admin	admin users only	Templates ("history_v2" in ctx.features)
lists	enabled	always on	Templates + Python ("lists" in ctx.features)
merge-authors	{filter: usergroup, usergroup: /usergroup/librarians}	librarians + admins (code fallback)	Templates + Python
publishers	enabled	always on	Templates + Python ("publishers" in ctx.features)
recentchanges_v2	enabled	always on	Python: features.is_enabled("recentchanges_v2")
stats	enabled	always on	Template only ("stats" in ctx.features)
stats-header	enabled	always on	Python: "stats-header" in web.ctx.features
superfast	enabled	always on	Template only ("superfast" in ctx.features)
undo	enabled	always on	Python: features.is_enabled("undo")

How the old system works:

1. At startup, infogami.utils.delegate calls features.set_feature_flags(config.get("features", {})) — stores the raw YAML dict as feature_flags.
2. Per request, features.loadhook() runs find_enabled_features() which evaluates each flag's filter spec (string like "enabled" → filter_enabled → always True; dict like {filter: "usergroup", ...} → filter_usergroup → checks group membership) and writes the resulting set into web.ctx.features and context.features.
3. Templates check membership: $if "lists" in ctx.features:
4. Python code uses either features.is_enabled("name") or "name" in web.ctx.features.
5. web.ctx.features defaults to [] (set in setup_context_defaults() in openlibrary/plugins/openlibrary/code.py:1145).

Dev config file: conf/openlibrary.yml (in-repo, used by Docker).
Prod configs: Managed per-environment (not in this repo).

New System (Example Code)

The following is example code — not committed, not wired into the application. No production code imports from openlibrary.core.features. Treat it as a reference for the pattern, not as work already done.

openlibrary/core/features.py

"""Pydantic-settings based feature flags.

POC for replacing the legacy ``infogami.utils.features`` module. Features
are loaded from the ``features:`` section of a YAML file (typically
``openlibrary.yml``) using explicit pydantic-settings fields.

A single process-wide ``features`` instance is loaded at import time from
the path given by the ``OL_FEATURES_YAML_PATH`` environment variable
(falling back to ``conf/openlibrary.yml``). Access flags via dot notation,
which gives full IDE autocomplete and type checking::

    from openlibrary.core.features import features

    if features.debug:
        ...
"""

from __future__ import annotations

import os
from pathlib import Path

import yaml
from pydantic_settings import BaseSettings


class Features(BaseSettings):
    """OpenLibrary feature flags.

    Each field is a boolean flag. Override values by passing them as kwargs
    or setting the corresponding ``OL_FEATURE_<NAME>`` environment variable.
    Unknown keys in the YAML (e.g. admin/usergroup flags from the legacy
    config) are silently ignored.
    """

    model_config = {"extra": "ignore"}

    debug: bool = False
    dev: bool = False
    lists: bool = True
    publishers: bool = False
    recentchanges_v2: bool = False
    stats: bool = True
    stats_header: bool = False
    superfast: bool = False
    undo: bool = True
    # loops: bool

    @classmethod
    def from_yaml(cls, path: Path | str) -> Features:
        """Load feature flags from the ``features:`` section of a YAML file.

        YAML keys are expected to match field names (kebab-case keys like
        ``stats-header`` are normalized to snake_case ``stats_header``).
        Values should be native YAML booleans (``true``/``false``).
        """
        data = yaml.safe_load(Path(path).read_text()) or {}
        features_dict = data.get("features") or {}
        normalized = {key.replace("-", "_"): bool(value) for key, value in features_dict.items()}
        return cls(**normalized)


def _load_features() -> Features:
    """Load the process-wide Features instance from the configured YAML file."""
    path = os.environ.get("OL_FEATURES_YAML_PATH", "conf/openlibrary.yml")
    return Features.from_yaml(path)


# Process-wide singleton -- loaded once at import time.
# Access flags via dot notation: ``features.debug``, ``features.stats_header``.
features: Features = _load_features()  # type: ignore[assignment]

openlibrary/tests/core/test_features.py

"""Tests for the pydantic-settings based feature flag POC."""

from __future__ import annotations

from pathlib import Path
from textwrap import dedent

import pytest

from openlibrary.core import features as features_module
from openlibrary.core.features import Features


def _write_test_config(tmp_path: Path, body: str) -> Path:
    """Write a test openlibrary.yml and assign it to the module-level
    ``features`` instance. Returns the config path.
    """
    config = tmp_path / "openlibrary.yml"
    config.write_text(dedent(body))
    features_module.features = Features.from_yaml(config)
    return config


@pytest.fixture(autouse=True)
def _reset_features_to_defaults(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    """Ensure each test starts with a known-good in-memory ``features``
    instance and no stray env var pointing at a real config file.
    """
    monkeypatch.delenv("OL_FEATURES_YAML_PATH", raising=False)
    _write_test_config(
        tmp_path,
        """\
        features:
            debug: false
            dev: false
            lists: true
            publishers: false
            recentchanges_v2: false
            stats: true
            stats-header: false
            superfast: false
            undo: true
        """,
    )


class TestDefaults:
    def test_default_values(self):
        f = Features()
        assert f.debug is False
        assert f.dev is False
        assert f.lists is True
        assert f.publishers is False
        assert f.recentchanges_v2 is False
        assert f.stats is True
        assert f.stats_header is False
        assert f.superfast is False
        assert f.undo is True

    def test_kwargs_override_defaults(self):
        f = Features(debug=True, lists=False)
        assert f.debug is True
        assert f.lists is False


class TestFromYaml:
    def test_loads_enabled_flags(self, tmp_path: Path):
        config = tmp_path / "openlibrary.yml"
        config.write_text(
            dedent(
                """\
                features:
                    debug: true
                    dev: true
                    lists: true
                    publishers: true
                    recentchanges_v2: true
                    stats: true
                    stats-header: true
                    superfast: true
                    undo: true
                """
            )
        )
        f = Features.from_yaml(config)
        assert f.debug is True
        assert f.stats_header is True  # kebab-case YAML key normalized

    def test_unset_flags_use_defaults(self, tmp_path: Path):
        config = tmp_path / "openlibrary.yml"
        config.write_text("features:\n    debug: true\n")
        f = Features.from_yaml(config)
        assert f.debug is True
        assert f.lists is True  # default
        assert f.undo is True  # default

    def test_ignores_non_features_keys(self, tmp_path: Path):
        config = tmp_path / "openlibrary.yml"
        config.write_text(
            dedent(
                """\
                site: openlibrary.org
                features:
                    debug: true
                """
            )
        )
        f = Features.from_yaml(config)
        assert f.debug is True
        assert "site" not in Features.model_fields

    def test_missing_features_section_uses_defaults(self, tmp_path: Path):
        config = tmp_path / "openlibrary.yml"
        config.write_text("site: openlibrary.org\n")
        f = Features.from_yaml(config)
        assert f.debug is False
        assert f.lists is True


class TestModuleInstance:
    def test_features_is_a_features_instance(self):
        assert isinstance(features_module.features, Features)

    def test_features_singleton_is_shared(self):
        assert features_module.features is features_module.features

    def test_dot_notation_autocomplete_friendly(self):
        # The whole point of the refactor: callers should get full type
        # checking and IDE autocomplete via dot notation, not a stringly-typed
        # ``is_enabled("flag_name")`` helper.
        assert features_module.features.debug is False
        assert features_module.features.lists is True
        assert features_module.features.stats_header is False
        assert features_module.features.undo is True

    def test_reflects_yaml_reload(self, tmp_path: Path):
        _write_test_config(tmp_path, "features:\n    debug: true\n")
        assert features_module.features.debug is True

---

Phase 1 — Python-only Feature Flag (Stats)

Scope: Pick the simplest existing flag — the stats flag, which is only used in Python code and requires no user-group logic.

Proposed approach (following the example code above):

1. Set up a pydantic-settings config reader that loads a YAML file
2. Define the stats flag in the new config
3. Provide a clean import path so Python code can check it directly
4. Wire the existing stats flag usage paths to the new source
5. Remove the old stats flag definition

Wiring targets:

• "stats" in ctx.features in openlibrary/templates/site/footer.html:16
• "stats-header" in web.ctx.features in openlibrary/plugins/openlibrary/stats.py:68

Success criteria:

• Stats flag setting is read from YAML via pydantic-settings
• No templates or user-group checks involved
• Old internal-tooling-generated flag is gone for this feature

---

Phase 2 — Boolean Feature Flag in HTML Templates

Scope: Pick a flag that is only used in HTML templates where the check is a simple enabled/disabled boolean.

Candidate flags for Phase 2 (boolean-only, no group gating):

Flag	Template(s)	Check Pattern
superfast	viewpage.html:22, lib/history.html:4	"superfast" in ctx.features
undo	recentchanges/header.html:15	"undo" in ctx.features
lists	lists/widget.html:4, recentchanges/index.html:32, type/edition/view.html:535, type/user/view.html:12,33,65, type/author/view.html:178	"lists" in ctx.features
publishers	type/edition/view.html:266, publishers/view.html:89	"publishers" in ctx.features
dev	login.html:11, site/head.html:89, home/index.html:38	"dev" in ctx.features

These are all simple membership checks — no user-group logic involved. The dev flag is also checked in Python (openlibrary/plugins/openlibrary/code.py:369, openlibrary/plugins/openlibrary/home.py:72,112, openlibrary/core/admin.py:91, openlibrary/plugins/openlibrary/deprecated_handler.py:18).

Steps:

1. Make the new pydantic-settings config available in template context (e.g. add to context.defaults and inject there, or use a template global)
2. Update the chosen flag's template usage to read from the new source
3. Remove the old flag definition

Success criteria:

• A template can check a boolean feature flag from the new config without touching user-group logic
• Clear pattern established for converting template-only flags

---

Phase 3 — Research & Decision: Do We Need User-Group Feature Flags?

Question: Should the new system support librarian / admin / super_librarian style feature gating at all?

Group-Based Flags Audit

Two flags in conf/openlibrary.yml use group-based gating:

Flag	Config Gate	Code Fallback	Duplicates Permission Logic?
history_v2	admin	None (templates only)	Needs investigation — no code-level permission check exists
merge-authors	usergroup: /usergroup/librarians	user.is_admin() in both GET and POST handlers (merge_authors.py:215,319)	Yes — code already checks admin status independently

Analysis:

• merge-authors: The config restricts to librarians, but the code already allows any admin as a fallback. The usergroup filter is redundant. Dropping the group gating and relying on the code-level check would work — the code is the right place for this permission.
• history_v2: Only used in two templates. Under the old system, non-admin users never see it in ctx.features. If we drop group gating, we need to either:
  • Make it a simple boolean and let it be visible to all (if appropriate for the feature), or
  • Add an explicit admin check in the template

Ray's recommendation (unchanged):

• Avoid group-based feature flags in the config layer
• Keep permission logic in one place (the code) rather than splitting it between code and config
• User-group flags are unnecessary overhead

Possible outcomes:

• Option A (Recommended): Drop user groups entirely; feature flags become simple booleans
• Option B: Keep minimal group support but only for flags that genuinely need it
• Option C: Status quo — replicate the old group system in the new layer

Deliverable: A brief decision doc (can live in this note) with a recommendation and a migration plan for any affected flags.

---

Phase 4 — Migrate All Remaining Feature Flags

Scope: Move every remaining flag to the new pydantic-settings system.

• Convert boolean-only flags using the Phase 2 pattern
• Handle any group-based flags using the Phase 3 decision
• Clean up the old config-generator dependency once all flags are migrated

Success criteria:

• No feature flags read from the old internal-tooling-generated config
• Every HTML template using a feature flag reads from the new source
• Old feature-flag system can be fully retired

---

Settled Questions

• [x] Where does the current config file live? — conf/openlibrary.yml in the repo; per-environment configs managed outside the repo.
• [x] What does the YAML schema look like? — Top-level features: key with flag-name → filter-spec mappings. Simple flags use "enabled" / "disabled" strings; group-gated flags use "admin" or a dict {filter: "usergroup", usergroup: "/usergroup/librarians"}. See table in Context section above.
• [x] How does the current app surface feature flags into templates? — infogami.utils.features.loadhook() runs per-request, evaluates filter specs, and stores the enabled set in web.ctx.features (and context.features). Templates check membership via "flag_name" in ctx.features.
• [x] Does the new system need environment-aware overrides (e.g. dev vs. prod)? — No, because configs are already per-environment (different openlibrary.yml per deployment). The OL_FEATURES_YAML_PATH env var is sufficient.

[mekarpeles]

This is a well-structured epic. A few findings from the codebase that may help contributors get started:

Context

• openlibrary/core/features.py does not yet exist in master — the example code in this issue is not committed. Phase 1 contributors start from scratch using that code as a reference.

• pydantic-settings is a new dependency — requirements.txt currently has pydantic==2.13.4 only. pydantic_settings.BaseSettings lives in a separate package (pydantic-settings) that must be added to requirements.txt before the example code can run. This is a Phase 1 prerequisite.

• Direct predecessor: Audit Feature Flag Usage #12882 ("Audit Feature Flag Usage") produced a full per-flag inventory. Two cleanup PRs from that audit have already merged: Replace debug feature flag with query_param in work_search template #12883 (replaced debug in the work_search template with query_param) and Remove unused feature flags from openlibrary.yml #12888 (removed 6 entirely unused flags). Both are prerequisites to this migration and are already done.

• Four Python files import infogami.utils.features — these are the Python-side wiring targets that must be updated as flags move to the new system: openlibrary/plugins/openlibrary/code.py, openlibrary/plugins/upstream/utils.py, openlibrary/plugins/upstream/recentchanges.py, openlibrary/admin/utils.py.

• Phase 2 note on history_v2: The issue lists it as "admin-gated" and notes no code-level permission check exists. Confirmed — the only references are in templates/history/comment.html:5 and templates/lib/history.html:4. If the new system drops group gating (Phase 3 Option A), this flag would become visible to all users, which may need explicit admin-check logic in the templates.

Full triage checklist (maintainers / Richy)

• Triage — correct labels applied or removed
• Problem / opportunity — problem is clear, audience identified, actionable
• Scope — issue is well-scoped; phased breakdown isolates complexity
• Justification — infogami dependency removal and IDE-friendly dot notation are clear benefits
• Success & testing criteria — each phase defines explicit success criteria
• Proposal & breakdown — concrete phased plan with example code and wiring targets
• Risks, concerns, open questions — pydantic-settings new dependency not flagged inline; history_v2 group-gating removal may require template logic changes
• Richy context — relevant code, commits, and predecessor issues reviewed
• References — config file, template paths, and Python call sites all documented inline

Note

This comment was automatically generated by Richy, via PAM, Open Library's Project AI Manager, on behalf of @mekarpeles. Richy is designed to provide status visibility, perform basic project management functions and relevant codebase research, and provide actionable feedback so contributors aren't left waiting.

[Sanket17052006]

Thanks @RayBB for the detailed description, will be starting with phase 1 as per the provided references!