[fix][ml] fix potential data lose when enable lazy cursor recovery

[lifepuzzlefun]

Motivation

when user enable lazyCursorRecovery.

if one cursor success recovered and ack some message, and the other cursor recovery is delayed for sometime.

if the finished cursor markdeletepos moved it may trigger internalTrimLedger and delete ledger only decide by this cursor.

so after the other cursor finished recovery, these cursor may occur "data lose" because of the early trim.

Modifications

1. check if there is lazy recovery before trim ledgers.
2. add programmed readEntry delay in PulsarMockLedgerHandle
3. add CursorRecoveryInProcessException

Verifying this change

This change added tests and can be verified as follows:

add unit test in ManagedLedgerTest.onlyTrimCursorAfterLazyRecoverFinished

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[github-actions]

@lifepuzzlefun Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[lifepuzzlefun]

@poorbarcode @mattisonchao Could you take a look this pr ?

[poorbarcode]

Nice catch!

[poorbarcode]

Since the variable uninitializedCursors is tracing the cursor, which is in initialize progress. I think the new variable lazyRecoveryInProcess is not needed.

[lifepuzzlefun]

Thanks for review, I agree with you. The idea to use this variable here is to avoid get the synchronized lock of the managedledger instance. current uninitializedCursors is not thread safe. so each time a call to get synchronized lock is needed. I submit another to make this map thread safe. #20674. Need some guide to handle this. If acquire synchronized lock is acceptable. I will change the code in this style.

[github-actions]

The pr had no activity for 30 days, mark with Stale label.