Make sources configurable

[sagikazarmark]

Eg. disable GitHub? (there are often issues with rate limiting)

Maybe batching requests with graphQL could help?

[nikpivkin]

Hi @sagikazarmark ! I could try to take it on myself

[sagikazarmark]

@nikpivkin sounds great! Let me know if you need any help!

[nikpivkin]

@sagikazarmark I researched the Github GraphQL API and it's easy to implement. Should I replace the use of REST API with GraphQL or is there a reason to leave REST?

[sagikazarmark]

@nikpivkin I don't see a reason to stick to REST from the top of my head.

One thing I'd suggest is keeping the "old" REST implementation around in case we need to (partially) roll back.

I wonder if it would make sense to introduce a "batch" interface for license checking (ie. if a source implements the interface, batch license checks together?).

I don't know how much work that would be.

In any case, happy to review PRs.

[sagikazarmark]

@nikpivkin random though: maybe we could look at https://pkg.go.dev as a license source. They do pull the license from somewhere. No idea if they have an API though.

[nikpivkin]

@sagikazarmark I didn't find a public api at https://pkg.go.dev, but I did come across https://docs.deps.dev/api/v3alpha/. This service is hosted and developed by Google. We can use the following endpoint to obtain a project license:

curl -s "https://api.deps.dev/v3alpha/systems/go/packages/github.com%2Fgoph%2Flicensei/versions/v0.9.0" | jq '.licenses'
[
  "MIT"
]

Or use gRPC (does that make sense?) https://github.com/google/deps.dev/tree/main/api/v3alpha.

I created an issue to find out what rate limits this API has, since I didn't find this information in the documentation.

[nikpivkin]

@sagikazarmark Also I would like to clarify, is it necessary to add a flag to select sources?