TXM: Rpc liveness

[GabrielMartinezRodriguez]

Linked Issues

• closes https://linear.app/happychain/issue/HAPPY-366/txm-add-rpc-liveness-monitor

Description

Added a liveness monitor to the Transaction Manager

• Include all relevant context (but no need to repeat the issue's content).
• Draw attention to new, noteworthy & unintuitive elements.

Toggle Checklist

Checklist

Basics

• B1. I have applied the proper label & proper branch name (e.g. norswap/build-system-caching).
• B2. This PR is not so big that it should be split & addresses only one concern.
• B3. The PR targets the lowest branch it can (ideally master).

Reminder: PR review guidelines

Correctness

• C1. Builds and passes tests.
• C2. The code is properly parameterized & compatible with different environments (e.g. local,
  testnet, mainnet, standalone wallet, ...).
• C3. I have manually tested my changes & connected features.

< INDICATE BROWSER, DEMO APP & OTHER ENV DETAILS USED FOR TESTING HERE >

< INDICATE TESTED SCENARIOS (USER INTERFACE INTERACTION, CODE FLOWS) HERE >

• C4. I have performed a thorough self-review of my code after submitting the PR,
  and have updated the code & comments accordingly.

Architecture & Documentation

• D1. I made it easy to reason locally about the code, by (1) using proper abstraction boundaries,
  (2) commenting these boundaries correctly, (3) adding inline comments for context when needed.
• D2. All public-facing APIs & meaningful (non-local) internal APIs are properly documented in code
  comments.
• D3. If appropriate, the general architecture of the code is documented in a code comment or
  in a Markdown document.
• D4. An appropriate Changeset has been generated (and committed) for changes that touch npm published packages (currently packages/core and packages/react), see here for more info.

[cloudflare-workers-and-pages]

Deploying happychain with    Cloudflare Pages

Latest commit:	c884815
Status:	✅  Deploy successful!
Preview URL:	https://2608acea.happychain.pages.dev
Branch Preview URL:	https://gabriel-txm-rpc-liveness.happychain.pages.dev

View logs

[GabrielMartinezRodriguez]

• Randomness monitor service #584
• Txm traces #521
• fix(txm): returned nonce queue order #578
• fix(txm): not process the same block multiple times #577
• Fix: Viem sends random undefined blocks #575
• fix(txm): heap out of memory #573
• Avoid initiating new attempts if the gas conditions persist unchanged #571
• Dynamic priority fee #570
• TXM: Rpc liveness #562 👈 (View in Graphite)
• fix(txm): nonce gap #561
• PoC: Add metrics to TXM #503
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

HAPPY-366 TXM: Add RPC liveness monitor

Goal: avoid creating an unbounded amount of attempts when the RPC is down, which then creates a lot of load on the service or on the RPC. At the same time, we want to retry once the RPC comes back up.

This service can receive pings from other components (e.g. block monitor or tx submitter) to determine if the service is alive.

This is a policy that could be customized by the user.

[not-reed]

onSuccess/onFailure sound like callback listeners to me

onSuccess(args => console.log("success", args))

maybe trackSuccess() or something?

[GabrielMartinezRodriguez]

Yes, much better!

[aodhgan]

i think this was supposed to be used in conjunction with livenessSuccessCount but dont see it anywhere

[GabrielMartinezRodriguez]

Yup, good catch. It was renamed to livenessCheckInterval, and I forgot to remove livenessPingInterval

[norswap]

I'm wondering if that doesn't add a lot of overhead? If we're making 1000 RPC calls per second (which is not insane, each tx might require a few of these calls), then we're calling this 1000/s, and there are ~10k events in the event window, so 1000 times per second we're filtering through a list of 10,000 events.

Maybe we could maintain a rolling list of counters? Like for 10s period, 10 counters for success events, 10 counters for error events. And just maintain a single timestamp corresponding to the second-granularity timestamp of the oldest counters?

[norswap]

cc @aodhgan for an opinion here

[GabrielMartinezRodriguez]

I think you're right. I refactored the code, and I believe this approach is much better

[norswap]

Nice! If we ever need to optimize this more, we can replace the object by an array, and we could also update the count dynamically instead of recomputing it in ratioOfSuccess but this works nicely, merging this.