Reducing size of the repository / remove vendoring

[selfhoster1312]

Currently a full checkout is ~350MB on disk or ~100MB transfer (with compression).

That's too much for people to get started reasonably. The biggest part of it is the vendor folder.

We should consider:

• adding vendoring only in source release tarballs for new versions
  • is that even useful in the golang ecosystem?
  • is that useful for packaging for distros?
• dropping or rewriting some git history removing the vendoring from the repository
  • if we can make sure the repository stays buildable on every previously-buildable commit, that's a plus (that's only 1700 commits to test)
  • it's important if we do this to provide a script that transparently compares with upstream to verify that we didn't rewrite any other part of history to alter the source code (even in non-malicious ways)

[poVoq]

Some of the libraries are also very outdated custom forks. Especially the xmpp library has been updated a lot since wim42 made their fork.

[selfhoster1312]

When performing initial review of matterbridge-org/matterbridge#5, and especially this commit: 04c5018b833a85796626744a4e2504ac16e1682d, i realized that this way of vendoring makes it really hard to follow what happens in the PR because the diff is so huge.

I understand the advantages of vendoring, but in this case, i believe it may be a disadvantage: it's much easier for an untrusted contributor to sneak in malware inside the vendor directory (which we're never going to review properly) than in the rest of the codebase.

All in all, i believe i'd be in favor of removing this vendor repository altogether.

[poVoq]

All in all, i believe i'd be in favor of removing this vendor repository altogether.

If feasible I agree, but I assume wim42 did run into issues requiring it for some of the dependencies?

[selfhoster1312]

All in all, i believe i'd be in favor of removing this vendor repository altogether.

If feasible I agree, but I assume wim42 did run into issues requiring it for some of the dependencies?

I'd say usually the idea with vendoring is to:

• pin the dependencies so you don't have unintended API breakage and/or supply chain attacks
• provide source tarballs where all you need is an appropriate compiler, and no internet connection is required to build a working binary

The first problem is now solved by go.sum which was only stabilized in 2020 so that's probably why it's not used by matterbridge upstream.

The second problem is solved by vendoring inside the source tarballs you produce for release/packaging, not keeping the vendoring inside the repository.

I can also see additional minor considerations for/against vendoring. But in a big project with a lot of dependencies and drive-by contributors, i think i see more value in removing the vendor folder because:

• the repository is much lighter
• go build . works as expected, where as vendoring requires go build . -mod=vendor or may produce different results when vendoring is not properly kept in sync
• there's much less risk to introduce xz-style backdoors deep in the vendor folder where noone will review the changes

I'm just not sure how Github will react to rewriting the history: force-pushing should be OK (probably nobody has cloned the repo yet except us), but how to make sure the git blobs are not kept on Github side and redownloaded everytime? I'm not sure… i guess i can try on my personal branch/fork to see what happens.

Of course i'm not pushing for that just yet i'm waiting for more opinions :)

[poVoq]

I think there is no need to rewrite git history if we plan to move away from Github sooner or later anyways. We can do a clean import into Codeberg once that happens, no?

[selfhoster1312]

We can do a clean import into Codeberg once that happens, no?

I think the transition would be much smoother if the history rewriting took place before moving away from Github, because then all you have to do is git remote set-url on your copies and it will be transparent.

[selfhoster1312]

Looks like CI passes on matterbridge-org/matterbridge#6

@poVoq do i have your blessing to force push on master and put this question behind us? this will make it much easier to review "big" PRs like bibanon/matterbridge#25

[poVoq]

Yes, lets try that.

[selfhoster1312]

OK i've forced pushed. For further reference, here's how to verify we didn't change anything apart from removing the vendor folder:

git clone https://github.com/matterbridge-org/matterbridge matterbridge-compare
cd matterbridge-compare
git remote add upstream https://github.com/42wim/matterbridge
git fetch upstream
git diff --diff-filter=d c4157a4d5b49fce79c80a30730dc7c404bacd663 bf39c6547ca4e4b1d9af3bc86a6cf1d7877eefc5

This command shows the diff between the two commits, ignoring removed files/folders. Where bf39 is the rewritten commit on this repository, and c415 is the last upstream commit at the time of writing.

The output of this command should be empty! :-)

[selfhoster1312]

It is now done and merged. I'll close this issue!

[lil5]

This has made merging other forks unnecessarily complicated.