[fix][broker]Do not trigger topic GC if replication is still active

[poorbarcode]

Motivation

Replication is stuck due to the Topic GC

• Enable binary way replication between the primary cluster and backup cluster
• primary cluster: Publishing messages into primary cluster
  • backup cluster: No consumer/producer is registered.
• backup cluster: Check topic GC
  • No subscriptions
  • No producers except remote producer
  • The topic should be GC
    • Disable replication
    • The topic will not be deleted since the remote-side producer is still registered
• backup cluster: the topic GC progress is waiting for the remote-producer to be disconnected.
  • It will not be executed since no one wants to delete the topic.
• backup cluster: backlog increases because the replicator was closed
  • Although the messages copied from the remote end will not be copied back repeatedly, the replicator still needs to perform a check and then mark delete.

Modifications

• Topic GC will only be triggered if there is no producer(includes remote producer) and no replicator
• Replicator producer will be closed after there are no messages to be replicated anymore.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[codelipenghui]

@poorbarcode I'm thinking another solution

1. Close the replicator producer if the producer is idle for a while (e.g. 10 mins)
2. Check all the producers for detecting the inactive topic (now it's more complicated to skip the replicator producer)
3. Only delete the inactive topic if there is no producers (including the replicator producer)

Now, your solution added 7 days delay for inactive topic deletion if geo-replication enable. But if there is no messages from last 7 days, the issue can still happen, right?

[void-ptr974]

@poorbarcode I'm thinking another solution

1. Close the replicator producer if the producer is idle for a while (e.g. 10 mins)
2. Check all the producers for detecting the inactive topic (now it's more complicated to skip the replicator producer)
3. Only delete the inactive topic if there is no producers (including the replicator producer)

Now, your solution added 7 days delay for inactive topic deletion if geo-replication enable. But if there is no messages from last 7 days, the issue can still happen, right?

I agree with this concern. A replicated topic can be quiet for longer than the threshold while the replication relationship is still valid.

Using latestPublishTime here makes topic deletion depend on the traffic pattern instead of the producer lifecycle. It seems cleaner to let the replicator close its producer explicitly when it is really idle, and let topic GC only check whether producers still exist.

[poorbarcode]

@codelipenghui @void-ptr974 Changed the solution as @codelipenghui suggested, please review again

[poorbarcode]

@codelipenghui @lhotari @void-ptr974

Please review the PR again. I have changed the solution:

• Topic GC will only be triggered if there is no producer(includes remote producer) and no replicator
• Replicator producer will be closed after there are no messages to be replicated, it does not rely on the Persistent Topic anymore