Extensible Source Providers

[patrick-east]

Extensible source providers

Problem

OCP's built-in source types are hard coded (eg git, http/s3 datasources, sql, etc). Downstream users of OCP that want to build bundles from sources OCP doesn't know about have only two options today: fork OCP, or upstream every new kind. Neither option is ideal for a variety of reasons.

What would be really nice to have is the ability to extend the set of source types supported programmatically when using OCP as a library (ie, no runtime/binary plugins, just a way to register them through the public go interfaces).

Proposed Config Changes

Source accepts two config formats; the top level container type determines a persisted schema version.

v1 (current) — map of typed fields:

Example:

sources:
  app:
    git:
      repo: https://... 
    datasources:
      - name: users
        type: http
        url: ...

v2 (new) — list of map entries with a type: discriminator:

Example:

sources:
  app:
    - type: git
      repo: https://...
    - type: git              # >1 of same kind allowed
      name: tests-repo
      repo: https://...
    - type: http
      name: users
      url: https://api/...
    - type: my-custom-source # registered with OCP Service via SDK integration
      param1: ... 

We could make it so name is required only when more than one entry of the same type exists in a source; otherwise it defaults to type (similar behavior as today where like git configs don't have a name)

In this proposal a source's schema format would be sticky for its lifetime. PUTs against an existing source must match the stored shape; mismatches return 400. Schema version is persisted internally and never appears on the API responses.

Supporting changes

(TBD on the exact ordering)

1. Refactor the public interface in pkg/sync for the Synchronizer. It doesn't appear to be used, and is a little off from what the service workers require. This would be either a breaking change or we play it safe and add a new interface (or set of methods on the existing one). This time refactoring the existing sources to actually adhere to it, and the workers to use it.

   Something like...

   type Synchronizer interface {
       Execute(ctx context.Context) (map[string]any, error)
       Close(ctx context.Context)
   }

2. Add a new SourceProvider public interface in pkg/sync along with implementations for the builtin types (registered by default).

   Something along the lines of:

    type SourceProvider interface {
        // Type returns the discriminator value used in v2 source entries.
        // e.g. "my-custom-source"
         Type() string
        
        // ValidateConfig parses and validates the entry's config blob.
        // Called at config-load time and on REST PUT, before any sync.
        ValidateConfig(raw json.RawMessage) error
        
        // MetadataFields declares which keys the synchronizer can produce
        // in its Execute return value (e.g. ["hash"]). The worker uses this
        // together with the bundle's revision template to decide which
        // fields to actually compute — same opt-in pattern as
        // httpsync.WithMetadataFields today.
        MetadataFields() []string
        
        // New constructs a Synchronizer for one entry instance.
        New(ctx context.Context, p ProviderParams) (Synchronizer, error)
    }
    
    type ProviderParams struct {
        SourceName     string             // owning source's name
        InstanceName   string             // the entry's `name:` (or its `type` if omitted)
        Config         json.RawMessage    // already passed ValidateConfig
        Dir            string             // {bundleDir}/sources/{src}/entries/{instance}
        SecretProvider SecretProvider     // tenant-scoped; may be nil
        MetadataFields []string           // subset of MetadataFields() the worker actually wants
        Logger         *slog.Logger
    }

3. Refactor the Service to use the registry when setting up workers.

4. Add a new Service constructor option to supply external SourceProviders that get registered

   reg, err := sync.NewSourceProviderRegistry().       // pre-populated with OCP built-ins
                WithExternalSourceProvider(myCustomSource.New(...))  // Optionally allows external providers
   
   svc := service.New().
           WithConfig(cfg).
           WithSourceProviders(reg).                 // new builder option
   ...

5. Update the sources table schema with a couple new columns

   1. schema_version INT NOT NULL DEFAULT 1
   2. json_config TEXT NULL
      v1 rows continue using existing columns; v2 rows store entries as JSON in json_config

6. Update config.Source to support both styles of configuration with a list of more opaque config entries

7. Refactor the worker to adapt v1 source configs into the newer style opaque entries. Goal being that the workers shouldn't care what type it was configured with. TBD if we need another intermediate source config struct for this.

8. Slight TBD on how we change the JSON schema to aggregate Source and make it more like oneOf: [SourceV1, SourceV2]. We may need to get a little bit creative on the schema generation as well as the validation to dispatch to each SourceProvider to perform their own config validation.

9. TBD if we need to change anything with the revision templates. We could just expose the structures kind of as-is and expect that the template matches the config they've provided, eg v1 sources keep existing paths: input.sources["x"].git.commit, input.sources["x"].http["ds-name"].hash. v2 sources use something like "entry"-keyed paths: input.sources["x"].entries["name"].field

Some Alternatives

1. Extend existing Source with extension/extra/etc field

This field could then allow for adhoc source types to be referenced. All the registration stuff stays about the same as the proposed approach, but the config doesn't change as much.

Example:

sources:
  my-src:
    git:
      repo: https://github.com/example/app-policy.git
      reference: refs/heads/main
    datasources:
      - name: s3-datasource
        type: http
        path: data/from/s3
    extensions: # or whatever
      - type: my-custom-source
        param1: ...

The main downside here is that we've got two styles for defining sources, somewhat complicating the UX. Somewhat of a vanity concern is also makes them look different from the builtin ones. If these are registered with a library integration the aim would be to try and treat them as peers of the builtin sources.

2. Extend the Source type in place

We could try and allow for dynamic top level fields on the Source config, more like:

sources:
  my-src:
    git:
      repo: https://github.com/example/app-policy.git
      reference: refs/heads/main
    datasources:
      - name: s3-datasource
        type: http
        path: data/from/s3
    my-custom-source:
        param1: ...

This gives a super native feel for the new source, primarily at the cost of complicating the code using these configs. It also doesn't make it very easy to support >1 entry (eg, If a user wanted two git repos, or multiple custom source entries). Switching to a list lets us unify the config between datasources and other sources.