Shrinking the repo - important but disruptive chore

[michaelneale]

the cloned repo is around 1.7G or so at time of writing, which is unfortunately large and probably 4x at least what it should be.

Some reasons:
1 one blog post has very large full res photos which could be compressed
2 some dead directories which had checked in binaries (for windows and other things) and some accidental node_modules commits
3 other large bits of content as the repo also acts as a bit of a CMS for docs/content

The largest one to fix is 2, but is most disruptive:

Filter out old paths, rewrite history, force push
This does mean that every PR basically will break, everyone has to clone fresh, but otherwise all history is kept
How to do this:

• Run git filter-repo --invert-paths --path temporal-service/ --path pr-preview/ --path documentation/prompt-library/node_modules/ --path extensions-site/node_modules/ --path crates/goose-demo/ --path crates/goose-ffi/ --path ui/desktop/src/bin/goosed --path ui/desktop/src/bin/goose_ffi.dll --path ui/desktop/src/bin/goose_llm.dll --path ui/desktop/package-lock.json --path input.wav
• Force-push rewritten history
• Everyone re-clones (all SHAs change, old forks/PRs break)

This will shave off 500+ gig of totally dead noise.

for 1: documentation/blog/2025-04-17-goose-goes-to-NY/ has 256M of photos - could be 3m, so really should do this as part of 2 above - shrink those, and then filter out the old ones from the history so we don't keep the large ones.

for others: there are videos which could be moved to somewhere else, could also use git LFS to prevent, and perhaps we should consider docs/blogs into another peer repo next to goose as part of it aaif-goose/goose-docs for example

With all the above, I think can get it down to a few hundred meg at most, a vastly better experience, but does require some pain.

Anyone up for this?

[jh-block]

This would cause an enormous amount of churn for contributors with clones / forks / active PRs. Is it really necessary now that git has good tooling like git clone --filter=blob:none to keep things small? If we stop the bleeding by removing large files from the current state of the repo, and then people use --filter=blob:none to clone, those large files won't be fetched unless they actually check out an older ref.

Also: it's currently possible to cryptographically verify that every commit (or at least every recent commit, I didn't check back to the start of the repo) in main came from a GitHub PR, because we use squash merge so the commits are created and signed by GitHub. If we rewrite history locally and force-push, that will no longer be the case.

[michaelneale]

yeah I was wondering how disruptive - yeah I didn't think of forks, not just branches/PRs, would that mean they have to fork fresh? (not ideal).

yeah --filter blob is a good idea - we could put that in AGENTS.md and even the readme etc... and yes can do that, that doesn't break things.

[michaelneale]

came from a GitHub PR, because we use squash merge so the commits are created and signed by GitHub

I wonder if that would apply across the board, I didn't think it would (wouldn't go all the way back, no, but I guess certainly past year or so?)

[michaelneale]

git clone --filter=blob:none git@github.com:aaif-goose/goose.git goose_s

currently leaves me withg 500M ish - but that is all the blobs in there right now, so if we stopped the bleeding, then could get that down much smaller, I guess is that acceptable? Really wish there was a safe way to cauterise irrelevant mistaken history

[jamadeo]

This is the output of git-sizer. The biggest issues it flags are the gh-pages related artifacts. We have a few ways to trim this down but the real solution is moving to publish using a workflow instead of a branch.

| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Commits                    |           |                                |
|   * Count                    |  18.1 k   |                                |
|   * Total size               |  10.1 MiB |                                |
| * Trees                      |           |                                |
|   * Count                    |   536 k   |                                |
|   * Total size               |   209 MiB |                                |
|   * Total tree entries       |  5.00 M   |                                |
| * Blobs                      |           |                                |
|   * Count                    |   530 k   |                                |
|   * Total size               |  21.9 GiB | **                             |
| * Annotated tags             |           |                                |
|   * Count                    |     0     |                                |
| * References                 |           |                                |
|   * Count                    |  2.41 k   |                                |
|     * Branches               |   503     |                                |
|     * Tags                   |   222     |                                |
|     * Remote-tracking refs   |  1.44 k   |                                |
|     * Git stash              |     1     |                                |
|     * Other                  |   248     |                                |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Commits                    |           |                                |
|   * Maximum size         [1] |  10.9 KiB |                                |
|   * Maximum parents      [2] |     2     |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [3] |  6.75 k   | ******                         |
| * Blobs                      |           |                                |
|   * Maximum size         [4] |  93.6 MiB | *********                      |
|                              |           |                                |
| History structure            |           |                                |
| * Maximum history depth      |  4.36 k   |                                |
| * Maximum tag depth          |     0     |                                |
|                              |           |                                |
| Biggest checkouts            |           |                                |                                                                                            | * Number of directories  [5] |  9.86 k   | ****                           |
| * Maximum path depth     [6] |    13     | *                              |
| * Maximum path length    [7] |   151 B   | *                              |                                                                                            | * Number of files        [7] |  31.9 k   |                                |
| * Total size of files    [8] |  11.9 GiB | ************                   |                                                                                            | * Number of symlinks     [9] |    57     |                                |
| * Number of submodules       |     0     |                                |

                                                                                            [1]  69abe69874badb22f603dd2d09c8962be4aa8a09
[2]  2b6ac7b3907a91e35fe6befe70cae6a9492e7727                                                                                                                            [3]  80ca1b90de9ea435517ffa1e17fc11db108455bc (refs/remotes/fork/gh-pages:assets/js)
[4]  c87175ea85f5b8c32b85245a6f6703d8cce489d3 (refs/remotes/origin/windows-fix-test:ui/desktop/src/bin/goose_ffi.dll)
[5]  d960ee81d84eb67e6486f9122bbfad045fbd7d49 (refs/remotes/origin/gh-pages^{tree})
[6]  19cb88fca1d27760cd23d6b0373808c6bf2eda02 (refs/remotes/origin/smart-approval-docs^{tree})                                                                           [7]  5823790c58153d1cfa0b1c8769b67f33cf6dad6e (a4cdca1056bdd1f6958b82b12a09e7ecf18fd6c0^{tree})
[8]  53ef4d1cd477c32d5c159bd886bedcf12cbe0693 (24f8da7d39eb3623c645af4a3ae50a3ae1f13625^{tree})
[9]  4fd480bc2eb1edba3c2a649e46264d1550dc2c88 (a4cdca1056bdd1f6958b82b12a09e7ecf18fd6c0:documentation/prompt-library/node_modules)