Index rework overview

[consideRatio]

Done / Preliminary done tasks

1. DONE
   Add client support for publisher field

   • Respect a publisher field in .project.json
   • Fixed by feat: add support for publisher field in .project.json #212

2. PRELIMINARY DONE
   Define index structure

   • Fixed by https://gitlab.com/sensmetry/internal2/tech/syside/sysand/home/-/merge_requests/1
   • Fixed by breaking: implement the new sysand index read protocol #288

   The index structure has been defined like this, and it is different from the environment structure.

    # index_root
    # ├──index.json:
    # │    {
    # │      "projects": [
    # │        {
    # │          i = "pkg:sysand/publisher1/name1",
    # │          v = ["0.0.1", "0.0.2"]
    # │        }
    # │      ]
    # │    }
    # └──publisher1
    #     └──name1
    #         ├──0.0.1
    #         ├──0.0.2
    #         └──meta.json:
    #             {
    #                 "iri": "pkg:sysand/publisher1/name1",
    #                 "versions": ["0.0.1", "0.0.2"]
    #             }
   

3. PRELIMINARY DONE
   Define index client->server upload/publish API
   A preliminary strategy is implemented in the sysand-index, sending a POST request with multipart form-data, including a PURL and the .kpar data blob (with bundled README).

4. DONE
   Add README.md bundling to build command

   • Resolved by build command: package a README.md, from /README.md, or from a sysand.toml configurable source #86

5. DONE
   Add client publish command

   • Add publish command #249

6. DONE
   Change client's index reading behavior (breaking)

   • Tracked by Change client's index reading behavior (breaking) #286
   • Fixed by breaking: implement the new sysand index read protocol #288

7. DONE
   Decisions about index server paths
   Currently beta.sysand.org exposes things directly under its root path / (https://beta.sysand.org/entries.txt), but the new index currently doesn't (https://redacted.domain/index/index.json), it instead exposes it under /index/.

   A client reading the index should read from https://redacted.domain/index/, and a client publishing something should do it to https://redacted.domain/ where api/v1/upload is appended.

   https://crates.io makes use of a config.json file where the client looks up where it should do various things, looking like https://github.com/rust-lang/crates.io-index/blob/master/config.json:

   {
     "dl": "https://static.crates.io/crates",
     "api": "https://crates.io"
   }

   I'm considering if we perhaps should providing something like that, allowing us to reference a single URL as "the index" when users are picking where to download and upload to. It would also be a signal that the index is "smart" rather than dumb, if there is a response to such endpoint.

   EDIT: sysand-index-config.json is now respected with index_root and api_root like crates.io.

   • Fixed by breaking: implement the new sysand index read protocol #288

8. Jonas assigned
   Add index management command

   • Tracked by Add index management command #279

9. Pending, Simonas is working this I think
   Add standard library to the new index

   • An organization should be created at the new index.
   • Standard libraries with publisher field set should be uploaded

10. BLOCKED until UX for [WIP] Add new usage types #281 is decided on
    Support publisher/name shorthand for pkg:sysand/publisher/name in sysand add?
    Could improve the UX of sysand, but its a fancy thing beyond the core need. sysand and the index should align, currently the index suggests using the shorthand.

11. BLOCKED by 8, pending assignment
    Update this repo's docs on self-hosting index
    We have docs at docs/src/hosting_index.md that should be updated with the sysand index management command.

12. DONE
    Deploy new deployment of sysand-index to test.sysand.com

13. DISCARDED TASK
    Rework sysand-index's concept of organization to publisher

14. Erik assigned
    Write user friendly docs on the official sysand-index about publishing a package.

15. BLOCKED by 14, pending assignment
    Update this repo's docs on publishing
    We have docs at docs/src/publishing.md that should be updated to point to sysand-index's servers docs on how to publish.

16. DONE
    Decide if and how to change how we compute the lockfile's checksum

    • Discussion Locking down what our project checksum is #308
    • We don't change it before June, but before v1.

17. DONE
    Decide if we should remove project_digest from versions.json

    • Tracked by Remove project_digest from an index's versions.json #325
    • Yes, we remove it. In order to do so, we need to have lockfile's checksum optional.

18. Pending, Andrius assigned
    Make lockfile's checksum optional, and Source variants specific

    • Tracked by Remove checksum in Project struct and introduce project_digest and kpar_digest fields in Source variants #326

19. Erik assigned
    Deploy keycloak at accounts.sensmetry.com and make us sign in with that on sysand.com