About using hardlinks ... MAKING RCC FASTER

[vjmp]

Expectations we had to remember:

• you are always running mostly other people's code, so you have no idea what they are doing (weirdly, or otherwise)
• multiple users (hackers included) could be using same machine/containers/hololib/shared disk
• multiple processes will be running on same binaries (if hardlinked, copies otherwise)
• since every different file in hololib exist just once, avoid corrupting those while others are using them (they can do anything, if you hand them copies, not originals)
• MacOS behaves weirdly (with their security of things and file ownerships and syncinc)
• Windows behaves weirdly (with antivirus/firewalls injected by kernel)
• modern antivirus/firewall software works weirdly (and yanks executables even when application is already running [on suspicious activity])

These are @vjmp notes/memories from hardlinks and their use in rcc:

• technically hardlinks would be very beneficial to reduce time (and diskspace) usage of environment creation and cleanup
• we tried first use hardlinks in rcc environments, but there were some challenges
• we had to back down to give each running robot their own files (to mess around with; you never know what weird stuff foreign code does inside with their filehandles)
• one challenge is security, so allowing hardlinking between two different (maybe when at least one long-running code/robot is involved) environments opens up an attack vector from one environment to a next (if multiple people or containers have [write] access to same concrete file), and this is problem in SaaS like services using rcc
• files that have relocations in them cannot be hardlinked from one environment to next, or some code will end up running/accessing/modifying things on wrong environment/directory (because they are referring other files based on "hardcoded" location information inside them); we saw robots (stacktraces) jumping from one environment to next
• if processes are using locks on hardlinked files, it is not clear (to me) how other processes see those locks (when they actually should not have any knowledge of them)
• if some process is using file, same inode should not be written at (like .pyc/.pyo files) by other processes (or they sweep feets under from other processes expecting those files do not change

Note: I might add to this list, if I remember other things.

[vjmp]

Note, any of those should not prevent you trying out things. You might come up with some great solution. Or solution that has multiple strategies to cover all corners.

Above notes were just things rcc run into in some corner of a case, and not all root causes were resolved, but just backed away in tight schedules and production environments.

[vjmp]

And lots of our decisions were made because rcc was running in enterprise environments. Which reminds me:

• enterprises are weird (in their own separate ways)

[joshyorko]

@vjmp ty for the insights, i have done some comprehensive research but the main thesis is here, if you get the chance , would love your feedback:

Blazingly Fast Spec

I have compiled all of my research here Holotree-Specs

[vjmp]

@joshyorko

• On BLAZINGLY_FAST_SPEC.md ...
• Phase 2: Skip Hash Verification (HIGH IMPACT) ...
• Current behavior: Every DropFile() re-hashes the entire file to verify integrity.
• Proposed change: Trust local hololib, skip verification.

This proposal is kind of dangerous, because:

• if hololib has shared access (mounted from host machine; used from shared disk location), then someone can modify file, and if integrity is not checked, attacker has access
• writes to hololib could fail and corrupt file when it is coming in
• disks can corrupt (for example NFS mounts)
• so both accidental and intentional corruption can happen in local hololib (on rest)

And this all depends on if we take robust approach or optimistic approach.

[vjmp]

Note on all optimizations. Always profile, before and after.

Have you noticed --pprof option in rcc? It is there for a reason ...

[vjmp]

Important question: do you understand all those proposed improvements, or is it AI that only understand those improvements? Like all three OS optimizations and their filesystem variations?

There are easy to understand trade-offs (compression: space vs. time; but with rcc, people easily run out of diskspace). Making OS specific FS optimizations on syscall levels brings in multiple trade-offs (like how antivirus affects those; maintenance burden; testing responsibility on different OS/FS combos; enough hands to do testing/profiling before/after chages).

Important thing to remember: If you break it, you own the pieces, AI does not.

[vjmp]

Other security note: more you add third-party dependencies, higher the probablity of being hit by supply chain attack. (Relating to those additional libraries for various FS operations.)

Originally I was intentionally avoiding additional libraries, because security reasons and enterprises. More dependencies you have, more likely some enterprise and their security tools will find something to complain in "dependency-tree".

[joshyorko]

all three OS optimizations and their filesystem variations

@vjmp Soory the dump, Do I “fully” understand every proposed improvement at expert depth right now for all three OS optimizations and their filesystem variations? That is a BIG Hell no. I’m standing on your shoulders here and I will fully admit there is no way I am as smart or as capable at writing this code as you.

But the thing is: I NEED RCC to do things for me. And with the merger in 2024, that onramp to new features being developed by the likes of you was PERMANENTLY closed. I heavily use RCC in production RPA at my company. My company could not trust a binary behind an EULA, but they could back me building tooling — because that’s how I make my money.

You’ll see a LOT of LLM coding helpers in this fork (from CLAUDE.md to the .specify directory). One might think the developer is “vibing” his way through this. That is entirely not the case. Robocorp had a FULL software engineering staff. I have… me — plus a full time job, kids, a wife, and home responsibilities. If I want to ship features at this speed, I need a team.

And I can’t afford “even another developer as good as just me.” LLMs are much cheaper to hire than a real developer, and they enable me to ship a HIGHLY complex product on my OWN. That’s the reality.

At the same time: LLMs are not good enough to replace what you did yet. Holotree is complex, and you kind of thought of almost all edge cases already. You developed this initially ~5 years ago, and it’s still the most intimidating part of the entire stack.

When I first looked at this code I was scared shitless to touch it. I know the codebase now. I can traverse it, I can build it, I can debug it, I can ship it. I can tell you about almost EVERY feature this tool has with the exception of Holotree — it is the most complex piece of this codebase. LLMs help me ship faster, but I’ve set up guardrails for myself because again: I own this fork right now and I use it for my job so I need it to work. So ownership is understood.

About 2 years ago when tasked with my management to find an open source RPA alternative to UiPath, one constraint was the only budget/infra I was allowed was AWS. Fine — SaaS is probably out. Many searches later I stumbled across the Robot Framework ecosystem and Robocorp going full Python and, in terms of RPA tooling, had seemingly thought of EVERYTHING.

Fast forward two years: a Sema4AI merger, a closed RCC, and ALL of my bots running Robocorp’s RCC on my OWN distributed infra, orchestrated leveraging tools my company already pays for: GitHub.

Also because I'm super cool, I am fully Robocorp Certified. Lol

In these last two years, I’ve been peeling back your entire tech stack / your former company’s entire tech stack. Everything from the Python Robocore library, the intricacies of processes, work items, creating my own package robocore_adapters_custom`so I could connect to any backend leveraging the base adapter created for work items. When I started to peel back the onion, I noticed there were openings — blueprints to build an orchestration layer on top of all this if you can find the right entry points. And the entry point was RCC all along. I just didn’t realize it.

So this isn’t just RCC for me. This is the building block for everything else. I need RCC to be better. End goal is fully cloud native and donate the whole thing to the Linux foundation. But i need people to use it, break it, make it better, rinse and repeat.

And here’s the real point: I can make it better. Although I’d love to pull you out of “retirement” (even if just occasionally) because your insight is the difference between “it works” and “it’s beautiful" but eIther way I’m going to use it and keep going.

[joshyorko]

Other security note: more you add third-party dependencies, higher the probablity of being hit by supply chain attack. (Relating to those additional libraries for various FS operations.)

Originally I was intentionally avoiding additional libraries, because security reasons and enterprises. More dependencies you have, more likely some enterprise and their security tools will find something to complain in "dependency-tree".

Yeah this makes perfect sense, keep surface area small.

[joshyorko]

Note on all optimizations. Always profile, before and after.

Have you noticed --pprof option in rcc? It is there for a reason ...

I had not , this is awesome :

[joshyorko]

klauspost/compress go.mod:

module github.com/klauspost/compress
go 1.23

// NO require statements - ZERO dependencies!

@vjmp Concern

"more you add third-party dependencies, higher the probability of being hit by supply chain attack"

klauspost/compress actually REDUCES supply chain risk because:

Factor	klauspost/compress	CGO Alternative
Dependencies	0	Many (C libs, toolchains)
Attack surface	Minimal	High
Transitive deps	None	Multiple
Build complexity	Simple	Complex
CVEs	None	zlib has had many

Who Trusts It

Used by critical infrastructure:

• MinIO - Enterprise object storage
• CockroachDB - Distributed database
• Caddy - Web server
• IPFS - Distributed storage
• 257,460 repositories depend on it

---

Complete Research Summary

Topic	vjmp's Concern	Our Finding	Verdict
Hardlinks	Relocations cause stacktrace jumping	File.Rewrite already tracks this	✅ Already handled
.pyc files	Can't share bytecode between envs	copy-patterns already excludes them	✅ Already handled
File locks	Unclear behavior on hardlinked files	RCC uses separate .lck files	✅ Not affected
Platform issues	macOS/Windows weirdness	Platform-specific code exists, zstd is pure Go	✅ No new issues
Hash verification	Security boundary for shared access	Correct - keeping it intact	✅ Respecting this
Supply chain	More deps = more risk	klauspost has ZERO deps	✅ Lower risk than alternatives
Enterprise	SharedHolotree scenarios	zstd works fine, faster restores benefit enterprise	✅ Compatible

@vjmp - I spent significant time researching every concern you raised. Here's what I found:

On supply chain risk (your most important point):

klauspost/compress has zero dependencies. Literally none. Its go.mod is empty. This is the strongest supply chain security indicator possible - no transitive dependency attacks are possible.

It's used by MinIO, CockroachDB, Caddy, IPFS, and 257,460 other repositories. If this library gets compromised, half the Go ecosystem falls. It's about as safe as third-party Go dependencies get.

Using a CGO-based zstd alternative would actually be HIGHER risk - requiring C compilers, system libraries, and complex toolchains.

On hash verification:

You're absolutely right. Im going to completley remove Phase 2. Your reasoning cannot be argued with.
"if hololib has shared access (mounted from host machine; used from shared disk location), then someone can modify file, and if integrity is not checked, attacker has access"

Phase 1 keeps hash verification intact. The security boundary stays.

On hardlinks/relocations:

I studied htfs/relocator.go and the File.Rewrite field. Any future optimization would check len(file.Rewrite) > 0 first. But Phase 1 doesn't touch this - still decompresses, still rewrites paths.

On .pyc files:

Already handled via copy-patterns in settings. These are never shared.

On platform complexity:

Phase 3 (reflinks) is deferred until I can answer edge cases. Phase 1 is pure Go compression swap - no platform-specific code.

What I'm actually implementing:

• Replace gzip with zstd (klauspost/compress - zero deps)
• Keep hash verification (security boundary intact)
• Dual-format read (backward compatible)
• Profile with --pprof before/after (thanks for pointing this out!)

Basically the core of this for me is to understand is hash verification really the bottleneck?

I need to profile first to your point, I need actual numbers

[joshyorko]

@vjmp #66