more clear design trade-off explanations and roadmap

[matu3ba]

Motivation
This project looks very interesting and I am curious, if "universal undo" could be used for incremental verification (or other incremental program related executions like static/dynamic analysis or faster interactive proof inputs) by tracking all editor changes between user-specified snapshots to at least estimate or ideally compute the semantic difference between program changes.
Afaiu, currently the best one can get is decl/fn based invalidation and transitive recomputation of according proofs.
Not sure, if this is the correct place to do in-depth discussion and how you can be reached. Feel free to close or move to "Discussions".

That said, I have a few questions

1. "git compatible" sounds reasonable until you have a big rebase forcing a specific patch order meaning "to squash snapshots for the rebase". The text does not explain this shortcoming of jujutsu very clearly and if or how you plan to address it.
2. "combine snapshot-based and patch-based version control" is missing a summary sentence for "Based on the above, we can conclude that neither snapshots nor patches are "better"; they are better at different things. An ideal system would combine them to get the benefits of both." Is such ideal system possible or not and what trade-offs were chosen (for now)?
3. "built-in TUI" sounds like wip, but good idea, if there is corresponding API functionality with tests.
4. "store large/binary files efficiently" sounds good, but missing interface on how to mark files at binary and how to store large files out of band as required or optional
5. "universal undo" is missing memory usage/growth over time and what happens on data base deletion. I'd expect that it contains user-added intermediate states, so there would need to be some "cleanup-logic", which so far is not explained (theoretically). Or is my understanding incorrect? Also unclear is if/how undo-tree functionality vim implements is planned or feasible. It reads like a few edge cases are missing, like what happens if remote commits were reordered to rebase feature branch on master by a colleague or bot.
6. "clean implementation" sounds good

It would be nice, if there could be a roadmap and/or general overview of what things work and what things not to set properly user expectations.

[xeubie]

1. "git compatible" sounds reasonable until you have a big rebase forcing a specific patch order meaning "to squash snapshots for the rebase". The text does not explain this shortcoming of jujutsu very clearly and if or how you plan to address it.

Hmm can you elaborate on this question? I am not sure which shortcoming of jujutsu you are talking about; I don't use it, and I only mentioned its approach to git compatibility in the docs. Do you mean that jujutsu encourages rebase-heavy usage, which can affect collaboration with git users because your history diverges?

Xit doesn't exactly have a strong answer to this issue because I haven't even implemented rebase yet :^D It will not be that hard, though...rebase is pretty much just reset + cherry-pick, both of which I have already implemented.

2. "combine snapshot-based and patch-based version control" is missing a summary sentence for "Based on the above, we can conclude that neither snapshots nor patches are "better"; they are better at different things. An ideal system would combine them to get the benefits of both." Is such ideal system possible or not and what trade-offs were chosen (for now)?

I think a combination of these two systems is possible, and that's what I'm going for with xit. As far as I know, xit is the very first VCS that can do patch-based merging while also is compatible with git. This is possible for one simple reason...patches can always be derived from commits. There is no extra information needed than what is provided in standard git objects. Nonetheless, I don't know of anyone that has tried this, so it's uncharted territory.

3. "built-in TUI" sounds like wip, but good idea, if there is corresponding API functionality with tests.

Pretty much all the functionality in the TUI will also be available via the CLI and programmatically via the Repo struct. By the way, since you mentioned tests, one cool thing I added recently to my unit tests was actual embedded TUI assertions. I'm not sure if this is amazing or insane but embedding the TUI itself in my unit tests put a smile on my face for some reason.

4. "store large/binary files efficiently" sounds good, but missing interface on how to mark files at binary and how to store large files out of band as required or optional

Files are automatically considered binary if they contain invalid UTF-8 data. Maybe that's too narrow of a definition but it was an expedient choice, and easy to change. But actually, this distinction is only relevant for when xit creates a diff (for xit diff) and when it creates patches. It does not affect how the object is actually stored.

I have no plans for storing files out of band, though. For now I'm only solving the client-side half of the problem -- how to best store large binary files in the object store. I think storing files out of band requires server-side work which I'm not prepared to do right now.

5. "universal undo" is missing memory usage/growth over time and what happens on data base deletion. I'd expect that it contains user-added intermediate states, so there would need to be some "cleanup-logic", which so far is not explained (theoretically). Or is my understanding incorrect?

In xitdb data is organized using the hash array mapped trie from Phil Bagwell. So, each transaction shares data with the previous transaction; only the parts that changed require more disk usage. Nonetheless, there is no garbage collection right now; the plan is to have a feature such as xit gc that rebuilds the database from scratch and includes only the data referenced by the latest transaction, which effectively would nuke all the undo history.

Also unclear is if/how undo-tree functionality vim implements is planned or feasible. It reads like a few edge cases are missing, like what happens if remote commits were reordered to rebase feature branch on master by a colleague or bot.

I'll keep the implementation pretty stupid at first; I'm not sure I'll be able to make it safely undo arbitrary actions from the past rather than just the last N transactions in the db. The latter is much more straight-forward; in fact, in xitdb it is just a pointer copy. Making it undo some arbitrary transaction from the past is much harder, because as you point out, it can conflict with something that happened after.

It would be nice, if there could be a roadmap and/or general overview of what things work and what things not to set properly user expectations.

I just updated the readme to add a "status:" line under each major bullet point. It's not that specific but hopefully it helps set expectations. I'm not sure how specific I want to be about roadmap. For at least the next several months, I am just going to focus on getting things to work and getting universal undo out the door; it's hard to look further into the future than that.

I have every intention of making a serious git successor, and my approach is "evolution, not revolution". I don't want to change everything about git. I actually think they made a lot of great decisions. I'd rather choose a few key areas to improve (built-in TUI, patch-based merging, handling large files, undo) and keep most of the rest the same. After building a solid foundation, we can start to depart from git in other ways, but it's a bad idea to change too many things at once.

[ilyagr]

Re jj and xit comparison, there's a nice write-up at https://lobste.rs/s/mltpfg/xit_is_coming#c_5gekio.

[matu3ba]

Thanks for your fast and in-depth reply and status.

Hmm can you elaborate on this question?

Say person A and B work on a feature branch f1 and person A has a xit history xa, person B has xb. Person A has pushed, person B rebases f1 and person A now tries to reorder things without merge conflicts.
Person A has not the database history etc from person B, because it is only stored locally.

Potentially a worse problem appears, if person B would squash its own changes into f1, for example because person A introduced an untested regression on some arcane platform.

A similar problem or question appears on deciding, when person A should actually clean its data base. To me this indicates some additional locking/info is needed to specify to the server, when the server should merge, clean or indefinitely keep between-commit histories.

Jujutsu (like your tool) offers a local optimum regarding its solution, which makes it to me not justified to learn yet another tool. If jujutsu/your tool would at least theoretically describe how such merging logic can work, then I'd use it to eventually get the benefits from my expectation that it will be eventually solved.

Does this make sense to you?

I think storing files out of band requires server-side work which I'm not prepared to do right now.

Sounds reasonable for most use cases, since fetching/syncing out of band data can also be solved via a good build system.

Making it undo some arbitrary transaction from the past is much harder, because as you point out, it can conflict with something that happened after.

Sounds reasonable to me.

[xeubie]

Say person A and B work on a feature branch f1 and person A has a xit history xa, person B has xb. Person A has pushed, person B rebases f1 and person A now tries to reorder things without merge conflicts.
Person A has not the database history etc from person B, because it is only stored locally.

Well we need to be more specific about what database history refers to here. Are you referring to the local undo history? Or to the actual git history (the tree of commits)?

I'm not sure how the local undo history would present a unique problem here. When you perform a fetch in xit, it creates a new transaction in xitdb and thus a new item will appear in your undo history. If you fetch a feature branch that has been rebased, that transaction will include the change to that remote ref. Patch-based merging will still work fine, even if the commits from that remote ref were rebased; it may need to generate new patches but it will still work.

In xit, all repo state is stored in .xit/db, but currently the information it stores is basically identical to what git stores (with the exception of state related to patches). So, if someone force-pushes changes to a feature branch, xit will face the same issue as in git. The remote branch's history has diverged from your local branch. Right now xit doesn't have any special solution to this, but it doesn't face any additional problem beyond what you'd deal with in git.

[matu3ba]

what database history refers to here. Are you referring to the local undo history?

I wanted to refer to the local undo history.

[xeubie]

Gotcha. I'll move this to a discussion. Thanks for the questions.