Skip to content

add path field and optimize bulk record fetching#88

Open
matthewpeterkort wants to merge 16 commits into
developmentfrom
feature/optimize
Open

add path field and optimize bulk record fetching#88
matthewpeterkort wants to merge 16 commits into
developmentfrom
feature/optimize

Conversation

@matthewpeterkort
Copy link
Copy Markdown
Contributor

@matthewpeterkort matthewpeterkort commented May 26, 2026

PR Summary: feature/optimize

Overview

This branch is a broad optimization and cleanup pass across Syfon server runtime, generated API contracts, client/CLI behavior, storage and path handling, auth middleware, metrics, and reviewer-facing docs.

At a high level, the branch does five things at once:

  1. restructures large runtime packages into smaller responsibility-focused files
  2. improves object read/write and path-scoped behavior for project browsing and bulk operations
  3. expands and corrects the internal bucket/project management API surface
  4. modernizes client and CLI transfer behavior, auth flows, and package layout
  5. updates code generation, docs, and CI/release plumbing to match the new surfaces

The raw diff is large, but a meaningful part of the size comes from generated API output and large package splits rather than entirely new product behavior.

Main Themes

1) OpenAPI and generated API contract cleanup

The branch reorganizes apigen inputs and outputs so generation is clearer and closer to how the runtime actually serves specs:

  • local OpenAPI specs move under apigen/openapi
  • codegen configs move under apigen/codegen
  • old .openapi-generator artifacts and stale generated model packages are removed
  • generated DRS, bucket, and internal client/server packages are refreshed

This also adds the missing bucket-management generated surfaces for:

  • deleting a bucket scope
  • deleting all Syfon project data plus attached bucket scopes

The bucket client config now uses response-type-suffix: Resp so schema model names can coexist with ClientWithResponses wrapper names without producing invalid Go.

2) Internal API refactor by responsibility

Several oversized runtime packages are split into smaller units:

  • internal/api/drsapi
    • split into access.go, mutations.go, objects.go, registration.go, routes.go
  • internal/api/internaldrs
    • upload/download/data logic is reorganized around transfer, bucket helpers, and scoped route tests
  • internal/api/metrics
    • old monolithic metrics routing is replaced with smaller focused files such as access, file usage, transfer ingest, and transfer reports
  • internal/api/middleware
    • authn/authz and mode handling are reorganized into clearer local/gen3/token/plugin-specific files

This is largely maintainability work, but it changes enough routing and request handling code that reviewers should treat it as behavioral risk, not a mechanical move.

3) Auth and middleware semantics

The branch changes how Syfon reasons about authenticated access, especially in Gen3 mode:

  • member/owner semantics are corrected in middleware checks
  • the old authz cache is removed
  • built-in and local authn modes are made more explicit
  • JWT/session handling is expanded and better isolated
  • CSV-backed local auth support is added

This area matters because it affects who is allowed to read, mutate, and enumerate scoped resources, especially around organization/project behavior.

4) Storage, scope, and object-model improvements

A large part of the branch improves how Syfon handles project-scoped objects and their backing storage:

  • provider-aware bucket identity fixes duplicate-name issues across providers
  • path fields are added and normalized more consistently
  • bucket scope management is tightened
  • project cleanup now deletes both Syfon objects and linked bucket scopes
  • logic is added to remove bucket credentials when their last scope is removed
  • object reads, object mutations, and storage deletion are separated into clearer core modules
  • scoped browse/index behavior is improved for repo-style project views

This is the main product-level behavior bucket for the branch.

5) Index and browsing performance work

The branch includes multiple changes aimed at large-project browsing and bulk record access:

  • indexing for path queries
  • better scoped object selection and hydration paths
  • support for project hierarchy/repo-style browsing improvements
  • configurable or higher-value paging assumptions for large project views
  • optimized bulk record fetching

Several supporting docs in this branch explain the intended behavior around:

  • project hierarchy browsing
  • configurable /index page sizing
  • org/project storage metrics

6) Client and CLI restructuring

The client side is heavily reorganized:

  • client/conf becomes client/config
  • client/syfonclient becomes client/services
  • client/xfer becomes client/transfer

This is not only a rename pass. The branch also adds or improves:

  • cloud inspection helpers
  • transfer retry and progress plumbing
  • upload orchestration
  • download engine behavior
  • CLI auth support
  • progress rendering
  • command behavior for upload, download, list, rm, metrics, and related tests

Reviewers should expect both package-structure cleanup and end-user CLI behavior changes here.

7) Metrics surface changes

Metrics handling is significantly reworked:

  • old generated metrics API artifacts are removed
  • runtime metrics code is rewritten into smaller focused modules
  • transfer-oriented metrics ingestion/reporting is added
  • file usage and scope-aware metrics handling are expanded

This appears to be a shift away from one monolithic metrics route set toward narrower, composable metrics behaviors.

8) Documentation, deployment, and CI alignment

The branch also updates non-runtime surfaces so the new behavior is operable:

  • README.md
  • CONTRIBUTING.md
  • MkDocs nav and multiple operator/problem-solution docs
  • local deployment and Kubernetes deployment docs
  • CI, docs, and release workflows
  • dependency versions and CVE-related updates

This is important because the codegen and runtime changes would otherwise be difficult to operate or review correctly.

Biggest Reviewer Buckets

If someone is reviewing this PR end to end, the highest-value buckets are:

A) Runtime API behavior

Focus on:

  • internal/api/internaldrs
  • internal/api/drsapi
  • internal/api/middleware
  • internal/api/metrics

Questions to ask:

  • did any route contract change silently?
  • did auth checks become stricter or looser than intended?
  • do new delete and scope behaviors match expectations?

B) Storage and object correctness

Focus on:

  • internal/core
  • internal/db/postgres
  • internal/db/sqlite
  • internal/common

Questions to ask:

  • are object IDs, paths, and scope boundaries canonicalized consistently?
  • do provider-aware bucket identifiers avoid collisions without breaking older assumptions?
  • are deletes and bulk operations scoped exactly to {organization, project}?

C) Client and CLI compatibility

Focus on:

  • client/services
  • client/transfer
  • cmd/*

Questions to ask:

  • do the renamed/restructured packages preserve the same public behavior where required?
  • do progress, retry, and auth changes preserve existing scripts and user workflows?
  • are CLI tests covering the changed flows?

D) Generated contract drift

Focus on:

  • apigen/openapi/*
  • apigen/codegen/*
  • apigen/client/*
  • apigen/server/*

Questions to ask:

  • do generated diffs match actual intended API changes?
  • are there any remaining name collisions or stale generated artifacts?
  • does make gen now produce stable, compilable output?

Suggested Reading Order

For a reviewer trying to get through the branch efficiently:

  1. operator-guide-code-generation.md
  2. problem-solution-bucket-project-cleanup-codegen.md
  3. internal/api/internaldrs
  4. internal/api/middleware
  5. internal/core
  6. internal/db/postgres and internal/db/sqlite
  7. client/services, client/transfer, and cmd/*
  8. generated apigen output last, after understanding the source OpenAPI and runtime intent

Testing and Risk Notes

The branch carries risk in four areas:

  • authz/authn regressions
  • scoped delete behavior
  • generated client/server contract drift
  • CLI transfer regressions under real cloud providers

Testing focus should include:

  • make gen
  • go build .
  • route-level tests in internal/api/internaldrs
  • auth/middleware tests
  • client transfer tests
  • CLI tests for upload/download/list/remove flows

Bottom Line

This is not one feature. It is a branch-wide optimization and reorganization pass that also ships real behavior changes in:

  • bucket/project cleanup APIs
  • auth middleware semantics
  • path-scoped object handling
  • transfer and CLI behavior
  • metrics/reporting internals

The right way to review it is by subsystem, with generated output treated as a consequence of the OpenAPI and runtime changes rather than as the main event.

@netlify
Copy link
Copy Markdown

netlify Bot commented May 26, 2026

Deploy Preview for drs-server ready!

Name Link
🔨 Latest commit 357a4b0
🔍 Latest deploy log https://app.netlify.com/projects/drs-server/deploys/6a24b5b781a082000926eaa2
😎 Deploy Preview https://deploy-preview-88--drs-server.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8dae9f7977

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cmd/migrate/main.go
Comment on lines +134 to +135
if strings.TrimSpace(targetProfile) == "" {
targetProfile = cliauth.ResolvedProfile()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid auto-filling target profile for basic-auth imports

Auto-populating targetProfile from cliauth.ResolvedProfile() causes targetAuthFromInputs to reject valid --target-basic-user/--target-basic-password runs with --target-profile cannot be combined... whenever a root profile is set (for example via SYFON_PROFILE), even though the user did not request profile auth. This is a functional regression in migrate import for local/basic-auth targets and blocks imports in common environments where profile defaults are configured.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant