# of host modules in core is increasing faster due to more frequent protocol releases

[sisuresh]

Now that we're aiming for a stable protocol build every 60 days, we're adding host modules at a faster rate. I think this is increasing compilation time, and the problem will only get worse as we add more.

We should figure out a way to either compact older host modules, or some other solution that would let us reduce the number of older host modules when needed. @graydon and I discussed this a bit here.

[dmkozh]

To me the worst problem is compilation time. Given that 99.9% of the time we probably don't care about compiling the older envs for anything but the parallel catchup, I suggest we just by default publish core with e.g. just 1-3 host versions: vPrev (optionally), vCurr, vNext (only in vNext builds). The development builds would use the same 1-3 host versions as well. We also do a short catchup round with these builds that start at the lowest protocol version.

For major releases we can publish the special 'catchup' builds. These don't even necessarily need to be published for minor releases - these would be catchup-exclusive builds, and they shouldn't really care about most of the bug fixes and improvements. These of course would be larger and the build for them may take a while, but since we only need to actually build these rarely, I think the lift is not really going to be significant.

The reasons for why I would prefer this over 'host collapse' solutions:

• Single time effort vs an ongoing maintenance process that we'd need to go through on every protocol release (I also suspect that the maintenance process itself might take a while depending on how tricky the changes are and how hard it is to identify all of them in replay).
• More straightforward implementation, no need for bespoke 'bug maintenance', per-transaction budget behavior fudging etc.
• Constant number of hosts with guaranteed build time (for most of the builds). I can imagine some features can't be easily 'collapsed', so there is a risk that the number of hosts would still grow.

[graydon]

Some thoughts:

• Background: we were originally going to do "1-2 soroban hosts at a time" (with a complex prev/curr/next gating dance to do at upgrades) and protocol-gate any residual backwards compatibility issues required for replay once we had stable history-replay segments, the same way we do in core. We discovered in practice that soroban metering is so fragile that we had to put protocol-gating everywhere and so we decided to adopt this simpler strategy of "an ever-growing list of hosts", kicking the can of "how to compact them" down the road. We are perhaps now at the "down the road" state of that decision.
• An overarching qualifier: it's good in general to be able to replay the past, but it's not 100% clear to me that there are a lot of users to make much use of the ability, especially "deeper into the past" (years). Like we do it and we catch bugs with it, but most builds are probably not that concerned with how (or even "if") this functionality works. I think it's mostly for our benefit, for QA.
• That aside: I think there are .. maybe 4 major options on how to deal with this issue (besides just continuing to ignore it):
  1. Make it possible to "merge" some of the soroban hosts in such a way that catchup for protocol N runs on host N+1 or N+2, and catchup works but it loses a little bit of fidelity. Since the vast majority of observable protocol drift is in metering and the only way it's observable is "if a contract ran out of gas", it's fairly easy to reduce replay fidelity on this one axis and "follow the historical record" when it comes to out-of-gas errors, maintaining fidelity on everything else.
  2. Reduce the range of supported replay, but maintain exact fidelity on the part we support. That is: tell users they can only replay from the past year, or 2 years, or whatever of history. Possibly we can still release a "full-replay capable" build that's just a larger binary.
  3. Make a development configuration that only builds recent sorobans, but keep the release configuration building all of them. This has the advantage that we can iterate quickly on our laptops and workstations, without affecting anyone else.
  4. Spend some time just trying to make the build faster in general. There are a lot of knobs to turn here:
     • What looks like "linking" in the build process is actually us doing single-CGU LTO on all the soroban crates when combining them. This is expensive! We don't necessarily gain a lot by doing it either. We have never measured it in detail. We could try turning that off, or switching to multi-CGU / ThinLTO
     • We do release+debug builds of everything. This is also really expensive (and really big, like debuginfo is 10x or more the size of the binary itself). We could do less of this, or use a debuginfo level that is smaller (90% of the use we get from debuginfo is just backtraces, which use almost-none of the debuginfo). Maybe we can at least put reduced debuginfo into our release packages, if users are finding the binaries uncomfortably large?
     • We could push a little harder on getting sccache working across the group, so we're able to reuse build artifacts more readily.
     • We could switch to faster linkers. There are several now (mold and wild in particular seem to be quite fast).

I tend to lean towards options 3 and 4 since they perturb other parties the least and solves the immediate build pain.

I also don't mind option 1. As mentioned above, we are the main consumers of replay and I'm pretty sure that both us and all the other consumers of replay won't mind if we replay (still bit-identical results!) by running the history through a newer soroban and overriding the results when-and-if there's a diverging out-of-gas error. That's a very small edge case and the two main benefits of replay (checking we didn't break code elsewhere + letting people replay history for event streams) would still work.

I feel like option 2 is .. perhaps intellectually the most honest, and I agree it puts a bound on how much work it's going to cost us going forward, but is also potentially the most disruptive, most likely to annoy the few people who might rely on replay, and cause the most up-front work for us in terms of adjusting our build and release pipelines (we'll need to be releasing a whole new package variant and image).

[dmkozh]

I think for now option 3 might provide the immediate development process improvement without much time investment.

If the build size becomes a concern, we can reconsider option 2. I'm not sure I agree on the amount of work necessary - it seems to me like adding a new build configuration in the pipelines is not that hard. This might indeed be more disruptive, but I would be surprised if someone runs full catchup frequently (if at all). I feel like that can be compensated with the docs and some helper scripts.

[MonsieurNicolas]

given the nature of historical replay -- could we throw an agentic loop to merge modules for old protocol versions?

[dmkozh]

given the nature of historical replay -- could we throw an agentic loop to merge modules for old protocol versions?

I think the question is not whether we could, but whether we should. Yes, we could try that and it may even work to some degree, but the general issue is that we'd need to do compactions periodically, which would make the code more and more messy with every protocol.

Also if there are budget observation discrepancies (which would be the main thing that may be different between the protocol versions), then we'd need to do some nasty hacks that either would require us to maintain a really messy module where every operation has a bunch of conditional branches, or maintain a hardcoded list of 'divergent' transactions where we'd need to fudge the budget observation to achieve the desired replay result (which sounds quite sketchy to me).

And let's not forget that even a single 'merged' module still has to be compiled and linked on every build. It seems to me like any option that just skips building most of the code most of the time would be preferable as it's going to be faster.

[MonsieurNicolas]

I don't think we care so much about the quality of the merged modules -- we just need them to replay old versions, being able to snap them to latest toolchains/deps (that over time risk getting pulled) is the main benefit.

[sisuresh]

I actually have a WIP branch that's the result of Claude compacting older host modules, but it hasn't been tested yet. I agree that the quality of those modules is irrelevant as long as then can pass historical catchup.

I think a solution with both what @dmkozh is suggesting (option 3) and compacting host modules is ideal. Option 3 allows us to not have to care as much about the growing list of host modules on a continuous basis, so we can compact host modules on a much slower cadence as needed.

[dmkozh]

I agree that the quality of those modules is irrelevant as long as then can pass historical catchup.

I'm worried about keeping around unaudited unmaintainable code, even if we technically don't need to worry about it today. But let's actually test if it works at all - a working branch would give us a much better idea of how good/bad this is.