[DOC] Missing guidance on proper replication shutdown order and orphaned persistent tasks cleanup

[rapcore2]

Describe the issue:

The current cross-cluster replication documentation does not cover two critical operational scenarios that can lead to orphaned persistent tasks in cluster state:

1. Proper shutdown order is not documented — The documentation describes how to stop replication (POST /_plugins/_replication/{index}/_stop) and how to remove a remote connection, but does not explicitly state that _stop must be called for every replicated index before removing the remote connection. Reversing this order results in orphaned persistent tasks in cluster state that cannot be removed through any documented API.

2. Deleting autofollow rule does not stop existing replication — The documentation mentions this briefly, but does not warn that if you delete an autofollow rule and then remove the connection without calling _stop on existing indexes, you will end up with orphaned persistent tasks.

3. No recovery procedure for orphaned persistent tasks — If replication is stopped incorrectly (connection removed before _stop), persistent tasks remain in cluster state with executor_node: null and continuously generate IndexNotFoundException errors in logs. There is no documented procedure to clean up this state.

Steps to reproduce orphaned tasks:

1. Set up cross-cluster replication for one or more indexes
2. Remove the remote connection (cluster.remote.{alias}.seeds: null) without calling _stop first
3. Check GET /_cluster/state/metadata?filter_path=metadata.persistent_tasks — orphaned tasks are present
4. Attempt POST /_plugins/_replication/{index}/_stop — returns No replication in progress
5. Attempt POST /_tasks/_cancel?actions=*replication* — returns empty nodes {}

The tasks are not visible via GET /_tasks because executor_node is null, making them impossible to cancel through the Tasks API.

Expected behavior:

The documentation should include:

1. A clear warning that _stop must be called for all replicated indexes before removing the remote connection
2. A warning that deleting an autofollow rule does NOT stop existing replication
3. A recovery procedure for orphaned persistent tasks:
   • Restore the remote connection temporarily
   • Create a temporary empty index matching the orphaned task name
   • Call _stop on the index
   • Verify tasks are removed from cluster state
   • Remove the remote connection again

Suggested complete shutdown procedure:

Step 1 — Delete autofollow rule (stops new indexes from being replicated)
DELETE /_plugins/_replication/_autofollow
{
  "leader_alias": "my-connection-alias",
  "name": "my-replication-rule"
}

Step 2 — Stop replication for all existing indexes
POST /_plugins/_replication/{index}/_stop
{}

Step 3 — Verify persistent tasks are gone
GET /_cluster/state/metadata?filter_path=metadata.persistent_tasks
# tasks array should be empty

Step 4 — Only now remove the remote connection
PUT /_cluster/settings
{
  "persistent": {
    "cluster.remote.my-connection-alias.seeds": null
  }
}

Additional findings:

• POST /_plugins/_replication/{index}/_stop does not support wait_for_completion parameter — returns 400 unrecognized parameter
• Orphaned tasks continuously log IndexNotFoundException errors which pollute logs but do not affect cluster functionality
• The .replication-metadata-store index retains metadata for orphaned tasks and should also be cleaned up after recovery

Affected documentation pages:

• https://docs.opensearch.org/latest/tuning-your-cluster/replication-plugin/getting-started/
• https://docs.opensearch.org/latest/tuning-your-cluster/replication-plugin/auto-follow/

[kolchfa-aws]

@ankitkala @mohit10011999 @gbbafna @monusingh-1 @soosinha Could you verify that the proposed changes are accurate?