Skip to content

feat: VendorImport CRD and Ramp adapter#16

Closed
mattdjenkinson wants to merge 1 commit into
mainfrom
feat/ramp-vendor-import
Closed

feat: VendorImport CRD and Ramp adapter#16
mattdjenkinson wants to merge 1 commit into
mainfrom
feat/ramp-vendor-import

Conversation

@mattdjenkinson

@mattdjenkinson mattdjenkinson commented May 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a VendorImport CRD and a reconciler that pulls accounting vendors from Ramp and upserts them as Draft Vendor records, so the compliance team doesn't have to hand-type every vendor before reviewing its DPA. This is Phase 2 of the compliance ingestion plan; Phase 1 (contract OCR in the staff portal) is already on the staff-portal `feat/compliance-ui` branch.

  • CRD `compliance.miloapis.com/v1alpha1.VendorImport` (cluster-scoped, Platform-context) with a discriminated `spec.source` (today only `ramp`), the Ramp credentials Secret reference, optional endpoint/token URL overrides for sandboxes, and a default 24h `resyncInterval`.
  • Ramp adapter under `internal/adapter/ramp`: OAuth2 client_credentials with on-demand token caching, paginated listing of `/developer/v1/accounting/vendors`, and a deterministic `BuildResourceName(displayName, vendorID)` that produces RFC 1123-safe names that fit the 63-char DNS-label limit.
  • Reconciler that resolves the Secret (missing -> `Ready=False, Reason=CredentialsNotConfigured`, requeue), lists Ramp vendors, indexes existing imports via the `compliance.miloapis.com/imported-from=ramp` label, creates Drafts for new vendors, refreshes identity fields on existing Drafts, and records Active vendors in `status.skippedActiveRefs` rather than overwriting them. `status.conditions` carries both `Ready` and `ImportComplete`; `status.lastSyncTime` is set on every successful pass.
  • IAM: new ProtectedResource for VendorImport plus `vendorimports.{create,update,delete,patch,list,get,watch}` on `compliance-admin` / `compliance-viewer`.
  • Controller RBAC gains the new resource verbs plus `secrets.{get,list,watch}`.

Test plan

  • `go build ./...`, `go vet ./...`, `go test ./...` all clean. Unit tests cover deterministic resource naming, DNS-1123 cap, fallback slug for non-alphanumeric input, suffix divergence for different vendor IDs, and rejection of blank input.
  • `kustomize build config/base/crd` and `kustomize build config/components/iam` both render cleanly with the new VendorImport CRD and IAM resources.
  • After this lands and the OCI bundle ships, apply a `VendorImport` with a Secret of sandbox Ramp credentials and confirm:
    • `kubectl get vendorimports` shows `Ready=True`,
    • `kubectl get vendors -l compliance.miloapis.com/imported-from=ramp` returns the imported Drafts,
    • re-applying the CR re-syncs with no duplicates,
    • flipping an imported Vendor to `phase: Active` and re-applying the CR records it in `status.skippedActiveRefs` rather than overwriting.

Follow-ups

  • Country of incorporation default: Ramp's accounting vendors endpoint doesn't expose a country, so new imports default `countryOfIncorporation` to `UN`. Operators must update it before activating the profile. Worth revisiting once we know which Ramp field carries the address.
  • Schema completeness: only `id`, `name`, `code`, `remote_id`, `is_active` are consumed today. Once we have a real tenant we should extend `AccountingVendor` with whatever Ramp returns for tax IDs / address so the mapping can populate more of the Vendor spec.
  • Staff-portal UI: Phase 3 of the plan wires an "Import from Ramp" button into the vendor list and polls the VendorImport status until `ImportComplete=True`. Will land on the staff-portal `feat/compliance-ui` branch as a separate commit.

@mattdjenkinson
feat: VendorImport CRD and Ramp adapter
fd7e858 
Add a `VendorImport` CRD plus a reconciler that pulls accounting
vendors from Ramp and upserts them as Draft Vendor records, so the
compliance team doesn't have to hand-type every vendor before reviewing
its DPA.

Key features:
- New `VendorImport` CRD in api/v1alpha1/vendorimport_types.go. The
  spec carries the source ("ramp" today, extensible to other adapters),
  a Ramp credentials Secret reference (`client_id` / `client_secret`),
  optional endpoint + token URL overrides for sandboxes, and a
  resyncInterval (default 24h) that controls automatic refresh.
- Ramp adapter under internal/adapter/ramp implementing OAuth2
  client_credentials with on-demand token caching, paginated listing
  of /developer/v1/accounting/vendors via the cursor-style `start` +
  `page_size` parameters, and a hand-rolled `MappedVendor` shape that
  the controller turns into either a new Vendor or a patch.
- VendorImportReconciler watches VendorImport CRs. For each pass it:
    - resolves the credentials Secret (missing -> Ready=False,
      Reason=CredentialsNotConfigured, requeue after resyncInterval),
    - lists active Ramp vendors,
    - lists existing imported Vendors via the
      `compliance.miloapis.com/imported-from=ramp` label so subsequent
      syncs find them in O(1),
    - creates a Vendor for each unseen ramp-vendor-id, refreshes the
      identity fields on existing Draft Vendors, and records Active
      Vendors in `status.skippedActiveRefs` rather than overwriting
      operator-curated data.
- Resource names are derived deterministically via
  `BuildResourceName(displayName, vendorID)` — slug + 8-char SHA-1
  suffix of the vendor ID — so re-runs always produce the same K8s
  name and the slug never breaks the DNS-1123 63-char limit.
- Country of incorporation defaults to "UN" on creation (Ramp doesn't
  expose it); operators must update it during review before activating
  the compliance profile. Re-syncs respect operator edits to the
  country and never clobber a non-default value.
- IAM updated: new ProtectedResource for VendorImport plus
  vendorimports.{create,update,delete,patch,list,get,watch} permissions
  on the existing compliance-admin / compliance-viewer Roles.
- Controller RBAC ClusterRole gains vendorimports + secrets perms.

Tests cover the mapping helper: deterministic resource naming,
DNS-1123 cap, fallback slug for non-alphanumeric input, suffix-based
collision avoidance for different vendor IDs, and rejection of blank
input.
@scotwells

scotwells commented May 11, 2026

Copy link
Copy Markdown
Contributor

This functionality (vendor management) should exist in a ramp-provider repo since it's specific to ramp. We shouldn't need a VendorImport CRD since configuring the connection to Ramp is a deployment time concern, not something that needs to change at runtime.

mattdjenkinson added a commit to milo-os/ramp-provider that referenced this pull request May 12, 2026
@mattdjenkinson
feat: initial ramp-provider scaffold
b08b97f 
A standalone controller that polls Ramp's accounting-vendors endpoint
on a schedule and upserts Draft Vendor resources in Milo so the
compliance team has a starting point instead of hand-typing every
record. Lifts the Ramp client + mapping logic from the abandoned
milo-os/compliance#16 branch but drops the VendorImport CRD entirely:
configuration is deployment-time (flags + env + a mounted credentials
Secret), with nothing about a particular Ramp account stored in the
control plane. Operators that don't pay vendors through Ramp simply
don't deploy this controller.

Key features:
- internal/ramp ports the OAuth2 client_credentials token cache, the
  paginated /developer/v1/accounting/vendors lister, and the
  deterministic BuildResourceName helper (slug + 8-char SHA-1 suffix,
  RFC 1123-safe, fits the 63-char DNS-label limit). MapVendor no
  longer references the compliance API package; it emits a small
  provider-owned MappedVendor so the Vendor schema can evolve without
  forcing churn here.
- internal/syncer is a controller-runtime Runnable that fires once on
  startup and then on a configurable interval (default 24h). It lists
  imported Vendors via the compliance.miloapis.com/imported-from=ramp
  label, creates new Drafts for unseen Ramp IDs, refreshes identity
  fields on existing Drafts, and records Active Vendors in the log
  without overwriting them. Vendors are written through
  unstructured.Unstructured so the provider doesn't import the
  compliance API types.
- cmd/main.go exposes --milo-kubeconfig, --leader-election-namespace,
  --ramp-* credential flags (with file-mount / env equivalents so the
  client_secret never has to live in process args), and
  --resync-interval. Leader election routes through the same Milo
  control plane the syncer writes to.
- config/base/manager + components/{controller_rbac,leader_election,
  namespace} mirror compliance-service's layout, with the manager
  deployment mounting a `ramp-credentials` Secret at /etc/ramp by
  default and ClusterRole permissions limited to
  compliance.miloapis.com/vendors.{get,list,watch,create,update,patch}.
- Dockerfile, Makefile, and .github/workflows/{publish,test,
  validate-kustomize} match the milo-os/compliance conventions so the
  publish-kustomize-bundle workflow can package this provider the same
  way.

Tests cover the mapping helper: deterministic naming, DNS-1123 cap,
fallback slug for non-alphanumeric input, suffix divergence across
different vendor IDs, and rejection of blank input.
@mattdjenkinson

mattdjenkinson commented May 12, 2026

Copy link
Copy Markdown
Collaborator Author

Good call. Reworked as a standalone provider per the architectural feedback: the Ramp adapter, the periodic-sync loop, and the deployment manifests now live in milo-os/ramp-provider. There's no VendorImport CRD any more — the Ramp connection is purely a deployment-time concern, configured via flags / env vars and a mounted ramp-credentials Secret in the provider deployment. Operators that don't pay vendors through Ramp just don't deploy that controller; nothing about it leaks into compliance-service.

Imported Vendors still carry the compliance.miloapis.com/imported-from=ramp label and compliance.miloapis.com/ramp-vendor-id=<id> annotation so re-syncs stay idempotent and other tools can tell where a Vendor came from. The provider writes Vendors as unstructured.Unstructured so it doesn't import the compliance API types, decoupling the two repos from schema churn.

Closing this in favour of the new repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@scotwells scotwells Awaiting requested review from scotwells
@JoseSzycho JoseSzycho Awaiting requested review from JoseSzycho

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mattdjenkinson @scotwells