Optimize object moves by reindexing only context-aware indexes

[mamico]

• Optimize object moves by reindexing only context-aware indexes instead of a full unindex + reindex. The catalog RID is preserved across the move, avoiding stale references and unnecessary work. A new moveObject(object, old_path, idxs) method is added to CatalogTool and ICatalogTool to implement the path remap.

• Add IContextAwareIndexProvider, a named-utility interface that contributes catalog index names which must be reindexed on object moves (path-sensitive, security-sensitive, etc.). Register a named utility providing this interface to extend the default set (path, getId, id, allowedRolesAndUsers). The helper CMFCatalogAware.get_context_aware_indexes() aggregates all registered providers.

Based on / inspirated by:

1. Draft: Experiment with optimizing moves plone/Products.CMFPlone#3834 (comment)
2. https://github.com/4teamwork/ftw.copymovepatches

• I signed and returned the Zope Contributor Agreement, and received and accepted an invitation to join a team in the zopefoundation GitHub organization.
• I verified there aren't any other open pull requests for the same change.
• I followed the guidelines in Developer guidelines.
• I successfully ran code quality checks on my changes locally.
• I successfully ran tests on my changes locally.
• If needed, I added new tests for my changes.
• If needed, I added documentation for my changes.
• I included a change log entry in my commits.

[mamico]

Potential breaking change

By default this change treats catalog indexes as non-context-aware: on a move, only the indexes contributed by registered IContextAwareIndexProvider utilities (currently path, getId, id, allowedRolesAndUsers) are reindexed, while all other indexes keep their previous values. Any custom index whose value depends on the object's location in the tree, but that is not registered as a provider, will therefore become stale after a move. This is potentially breaking, in the same way the earlier work referenced here is.

Alternatives to reduce the risk

1. Experimental feature flag. Gate the optimization behind an opt-in experimental flag (env var / registry) and keep the current full unindex + reindex as the default. This lets the new behavior be rolled out gradually and validated in the field before it becomes the default.
2. Invert the default. Treat every index as context-aware by default (i.e. keep reindexing all of them on a move, preserving today's behavior and avoiding any stale-data risk) and instead register via utility the indexes that are context-independent. Optimization then becomes opt-in per index and safe by default, at the cost of requiring packages to declare their context-independent indexes in order to gain the performance benefit.

Note on trade-off: in a typical catalog most indexes are context-independent (title, description, subject, ...) and only a few are context-aware. The current approach registers the few context-aware ones (fast by default, but risky if one is missed); the inverted approach registers the many context-independent ones (safe by default, but the speed-up only materializes once they're declared).

[yurj]

What happens to indexes added via ZMI?

[mamico]

What happens to indexes added via ZMI?

Currently, the new index defaults to a context-unaware behavior. To modify this within the current implementation, a custom utility must be written.

Another option is to move the configuration into the index schema, providing the ability to modify it also through the web interface.

[mamico]

Initial benchmark here: https://github.com/mamico/cmfcore-move-benchmark

[mamico]

Design question: how to declare context-aware indexes

The problem

This optimization treats catalog indexes as non-context-aware by default: on a move, only indexes contributed by registered IContextAwareIndexProvider utilities are reindexed; currently path, getId, id, allowedRolesAndUsers. All other indexes keep their previous values unchanged.

Any custom index whose value depends on the object's location in the tree but is not registered as a provider will become silently stale after a move. This is a potentially breaking change for sites with custom path-sensitive indexes.

Alternatives

A — Feature flag (opt-in, safe rollout)
Gate the optimization behind an env var or a persistent registry flag; keep full unindex + reindex as the default. Sites opt in explicitly after verifying their catalog has no unregistered context-aware indexes.

• ✅ zero risk of regression, gradual rollout
• ❌ performance benefit requires active opt-in; the flag needs a deprecation/removal story

B — Invert the default (opt-out model)
Treat every index as context-aware by default (preserve today's full-reindex behavior). Register via utility the indexes that are context-independent (e.g. IContextIndependentIndexProvider). The optimization activates per-index only when packages declare themselves safe to skip.

• ✅ safe by default; no silent staleness risk
• ❌ speed-up only materializes once packages declare their context-independent indexes; in a typical catalog most indexes are context-independent (Title, Description, SearchableText, ...) so the burden of declaration is high

C — Marker interface on the index object itself
Define IContextAwareIndex in ZCatalog (or CMFCore). Index implementations implement it to signal they need reindexing on moves. PathIndex and FieldIndex based security indexes would implement it; plain field/text indexes would not.

• ✅ self-describing; no separate registry; the index knows its own semantics
• ❌ requires changes to ZCatalog index classes; third-party indexes need updating; adds a CMFCore dependency in ZCatalog (or requires an adapter layer)

D — Explicit list on CatalogTool
Store a context_aware_indexes frozenset directly on portal_catalog, configurable TTW or via GenericSetup. Ships with the known defaults; site admins can extend it for their custom indexes.

• ✅ per-site flexibility; fully inspectable and auditable
• ❌ requires a migration step on upgrade; package-provided indexes need an upgrade step or GenericSetup profile to register themselves; easy to forget

Trade-off summary

	Safe by default	Zero config for common case	Extensible by packages	Per-site control
A — flag	✅	❌ (opt-in)	✅	—
B — opt-out	✅	❌ (many declarations needed)	✅	—
C — marker on index	✅ (if ZCatalog ships defaults)	✅	✅	—
D — per-catalog list	✅	✅	⚠️ (upgrade step)	✅
current (opt-in utility)	❌	✅	✅	—

Thoughts?

[dataflake]

I am mostly neutral here because I don't have any Plone projects, but one thing sticks out:

C — Marker interface on the index object itself Define IContextAwareIndex in ZCatalog (or CMFCore). Index implementations implement it to signal they need reindexing on moves. PathIndex and FieldIndex based security indexes would implement it; plain field/text indexes would not.

* ✅ self-describing; no separate registry; the index knows its own semantics

* ❌ requires changes to ZCatalog index classes; third-party indexes need updating; 
     adds a CMFCore dependency in ZCatalog (or requires an adapter layer)

Adding a CMFCore dependency in ZCatalog sounds like an absolutely terrible idea no matter how I look at it.

[yurj]

What if I add something to the title related to the path, with a subscriber? This solution would not index the title with the new value. It is a corner case but this means I know I've to mark the title as context aware.

I would calculate the number of object to move. If the number is more than a critical value, I would ask the user if they want to optimize the move, with a warning about the cases in which this could be problematic.

Move a huge amount of object is usually some admins do, not regular users do. A new permission "Move many objects" could be implemented. But this seems complicated and can depend on the instance performances.

Move is a special operation, so need a special path:

• warn the user that only the context aware indexes will be updated
• add an option to index things anyway, selected by default

But this touch the UI, giving the choice to the user.

The simplest option is to limit the number of writes. If the result of the indexing is the same (same metadata, same index), don't update the index. I don't see problems in changing the RID but I would investigate why this happen on a reindex.

The big problem in the path index is the repeated values. I mean, if I've 10.000 objects under /Plone/afolder, I've an index with tokenization of /Plone/afolder repeated 10.000 times.I would alias the path index to the uuidindex and have a map path:[list of pathuids] stored in the index (or we can add this map to the UUIDIndex). So when moving objects, the list has just to be splitted, getting part of the UUIDs on the new path entry. Move would be fast as removing a list from a list, there are algorithms and data structures that do this in a very optimized way (O(N)).

The problem lies in the Zope Folder implementation, where object contained are stored in the folder instead of being stored in a central index, just like filesystems do. This would be the real solution. Folder should ask a central, fast index what are its contents, this would makevery fast the indexing (just a proxy here), moving (fs moving on the same filesystem get microseconds or less) and searching (ls on filesystem is very fast).