From 60f102833b0a311ae28174968895ebbd699240bd Mon Sep 17 00:00:00 2001 From: michaelkedar Date: Wed, 20 May 2026 06:45:16 +0000 Subject: [PATCH 1/4] refactor: unify go Dockerfiles and begin setting up Go OSV API --- .gitignore | 4 +- AGENTS.md | 34 +++++- Makefile | 8 +- deployment/build-and-stage.yaml | 20 ++-- go/.golangci.yaml | 1 + go/Dockerfile | 135 ++++++++++++++++++++++ go/cmd/api-devserver/main.go | 151 +++++++++++++++++++++++++ go/cmd/api/main.go | 32 ++++++ go/cmd/custommetrics/Dockerfile | 31 ----- go/cmd/exporter/Dockerfile | 31 ----- go/cmd/first_package_finder/Dockerfile | 31 ----- go/cmd/generatesitemap/Dockerfile | 31 ----- go/cmd/gitter/Dockerfile | 37 ------ go/cmd/importer/Dockerfile | 34 ------ go/cmd/osv-linter-worker/Dockerfile | 1 + go/cmd/recordchecker/Dockerfile | 31 ----- go/cmd/relations/Dockerfile | 31 ----- go/cmd/worker/Dockerfile | 31 ----- go/go.mod | 11 +- go/go.sum | 16 +-- go/internal/api/server.go | 52 +++++++++ 21 files changed, 436 insertions(+), 317 deletions(-) create mode 100644 go/Dockerfile create mode 100644 go/cmd/api-devserver/main.go create mode 100644 go/cmd/api/main.go delete mode 100644 go/cmd/custommetrics/Dockerfile delete mode 100644 go/cmd/exporter/Dockerfile delete mode 100644 go/cmd/first_package_finder/Dockerfile delete mode 100644 go/cmd/generatesitemap/Dockerfile delete mode 100644 go/cmd/gitter/Dockerfile delete mode 100644 go/cmd/importer/Dockerfile delete mode 100644 go/cmd/recordchecker/Dockerfile delete mode 100644 go/cmd/relations/Dockerfile delete mode 100644 go/cmd/worker/Dockerfile create mode 100644 go/internal/api/server.go diff --git a/.gitignore b/.gitignore index 2d896a85840..8b232e74d70 100644 --- a/.gitignore +++ b/.gitignore @@ -16,4 +16,6 @@ hurl-scripts/ temp/* **/tmp/** gcp/api/v1/osv/** -.hypothesis \ No newline at end of file +.hypothesis +go/api-devserver +esp.log \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 3660718a89e..5cbe47518ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -160,44 +160,68 @@ Many tests use expected outputs saved directly in the source tree: ``` - Add custom mock testcases inside [`gcp/website/testdata/osv/`](gcp/website/testdata/osv/). +### Local API Server Development (Go-native) +- To run the public OSV API server locally using the native Go implementation alongside the ESPv2 proxy (which transcodes HTTP/JSON REST requests to gRPC): + ```bash + make run-go-api + ``` + --- ## Go Component Architecture (`go/`) The Go component contains the active and migrated services for the OSV database. It is structured with executables in `cmd/` and shared libraries in `internal/`. +### Go Monorepo Docker Builds +All Go microservices are compiled using a single, unified multi-target Dockerfile (`go/Dockerfile`) which shares the workspace compilation setup. +- Due to the `replace` directive in `go/go.mod` pointing to the sibling `bindings/` library, Go Docker builds **must** mount the `bindings/` folder into the build context. +- Since the production Cloud Build steps run inside `dir: 'go'`, they leverage BuildKit's `--build-context` flag to cleanly map the sibling folder without importing other monorepo files: + ```bash + docker build -t osv/importer --target importer --build-context bindings=../bindings -f Dockerfile . + ``` +- For local dev testing, you can use the exact same command inside the `go/` directory. + > [!IMPORTANT] > **Python to Go Migration**: We are actively migrating core services from Python to Go. For example, the new Go-based worker (`go/cmd/worker/`) replaces the legacy Python worker (`gcp/workers/worker/`). Always prefer modifying the Go implementation if both exist, unless instructed otherwise. ### Executables (`go/cmd/`) -1. **`importer`**: +1. **`api`**: + - The public OSV gRPC API server. + - Defined in `go/cmd/api` as a thin wrapper calling into the `go/internal/api` library. + +2. **`api-devserver`**: + - Development orchestrator command for local testing. + - Spawns the Go API server natively in a background thread while concurrently running the `osv-esp` (ESPv2) docker container to perform HTTP/JSON to gRPC transcoding. + +3. **`importer`**: - Run as a cron job. - Reads from each vulnerability data source (defined as `SourceRepository` in Datastore or mapped in [`source.yaml`](source.yaml) / [`source_test.yaml`](source_test.yaml)). - Detects new or deleted vulnerability records. - Dispatches processing tasks via **GCP Pub/Sub** to the worker. -2. **`worker`**: +4. **`worker`**: - Daemon that subscribes to Pub/Sub tasks. - Ingests and enriches vulnerability records. - Computes affected Git ranges for commit-based querying. - Writes the enriched records to the database (GCS/Datastore). - Powered by a modular processing pipeline defined in [`go/internal/worker/pipeline/`](go/internal/worker/pipeline/). -3. **`exporter`**: +5. **`exporter`**: - Exports the entire database to a public GCS bucket. - Generates a root `all.zip` file containing all records. - Generates ecosystem-specific `all.zip` files (e.g., `PyPI/all.zip`). - Outputs individual vulnerability JSON files in their respective ecosystem folders (e.g., `PyPI/GHSA-abcd-efgh.json`). -4. **`relations`**: +6. **`relations`**: - Populates relationships between vulnerabilities in the database. - Calculates transitive and reflective `aliases`, reflective `related` vulnerabilities, and transitive `upstream` fields. -5. **`gitter`**: +7. **`gitter`**: - Git client daemon/utility to precompute and cache git operations required by other services. - Performs intensive Git tasks like computing commit graphs and generating patch IDs. ### Internal Shared Libraries (`go/internal/`) +- **`api/`**: Shared package containing the core gRPC public server implementation of the OSV API. - **`worker/`**: Core engine and subscriber logic for the Go worker. - **`database/`**: Shared Datastore client and repository models (specifically [`go/internal/database/datastore/`](go/internal/database/datastore/)). - *Design Pattern*: Models here **mirror** the Datastore models defined in the Python library ([`osv/models.py`](osv/models.py)). diff --git a/Makefile b/Makefile index 1ff80f2c390..11f7446fb4d 100644 --- a/Makefile +++ b/Makefile @@ -107,7 +107,13 @@ run-website-emulator: run-api-server: test -f $(HOME)/.config/gcloud/application_default_credentials.json || (echo "GCP Application Default Credentials not set, try 'gcloud auth application-default login'"; exit 1) cd gcp/api && docker build -f Dockerfile.esp -t osv/esp:latest . - cd gcp/api && $(install-cmd) && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities $(run-cmd) python test_server.py $(HOME)/.config/gcloud/application_default_credentials.json $(ARGS)# Run with `make run-api-server ARGS=--no-backend` to launch esp without backend. + cd gcp/api && $(install-cmd) && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities $(run-cmd) python test_server.py $(HOME)/.config/gcloud/application_default_credentials.json $(ARGS) + +# Run the native Go developer server orchestrator (launches both ESPv2 and the Go API server). +run-go-api: + test -f $(HOME)/.config/gcloud/application_default_credentials.json || (echo "GCP Application Default Credentials not set, try 'gcloud auth application-default login'"; exit 1) + docker inspect osv/esp:latest >/dev/null 2>&1 || (cd gcp/api && docker build -f Dockerfile.esp -t osv/esp:latest .) + @cd go && go build -o ./api-devserver ./cmd/api-devserver && (GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities ./api-devserver $(ARGS); EXIT_CODE=$$?; rm -f ./api-devserver; exit $$EXIT_CODE) run-api-server-test: test -f $(HOME)/.config/gcloud/application_default_credentials.json || (echo "GCP Application Default Credentials not set, try 'gcloud auth application-default login'"; exit 1) diff --git a/deployment/build-and-stage.yaml b/deployment/build-and-stage.yaml index faa024b956f..323aac9fbaa 100644 --- a/deployment/build-and-stage.yaml +++ b/deployment/build-and-stage.yaml @@ -104,7 +104,7 @@ steps: id: 'pull-importer' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/importer:latest', '-t', 'gcr.io/oss-vdb/importer:$COMMIT_SHA', '-f', 'cmd/importer/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/importer:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/importer:latest', '-t', 'gcr.io/oss-vdb/importer:$COMMIT_SHA', '--target', 'importer', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/importer:latest', '--pull', '.'] dir: 'go' id: 'build-importer' waitFor: ['pull-importer'] @@ -118,7 +118,7 @@ steps: id: 'pull-osv-worker' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/osv-worker:latest', '-t', 'gcr.io/oss-vdb/osv-worker:$COMMIT_SHA', '-f', 'cmd/worker/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/osv-worker:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/osv-worker:latest', '-t', 'gcr.io/oss-vdb/osv-worker:$COMMIT_SHA', '--target', 'worker', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/osv-worker:latest', '--pull', '.'] dir: 'go' id: 'build-osv-worker' waitFor: ['pull-osv-worker'] @@ -132,7 +132,7 @@ steps: id: 'pull-exporter' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/exporter:latest', '-t', 'gcr.io/oss-vdb/exporter:$COMMIT_SHA', '-f', 'cmd/exporter/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/exporter:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/exporter:latest', '-t', 'gcr.io/oss-vdb/exporter:$COMMIT_SHA', '--target', 'exporter', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/exporter:latest', '--pull', '.'] dir: 'go' id: 'build-exporter' waitFor: ['pull-exporter'] @@ -146,7 +146,7 @@ steps: id: 'pull-record-checker' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/record-checker:latest', '-t', 'gcr.io/oss-vdb/record-checker:$COMMIT_SHA', '-f', 'cmd/recordchecker/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/record-checker:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/record-checker:latest', '-t', 'gcr.io/oss-vdb/record-checker:$COMMIT_SHA', '--target', 'recordchecker', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/record-checker:latest', '--pull', '.'] dir: 'go' id: 'build-record-checker' waitFor: ['pull-record-checker'] @@ -160,7 +160,7 @@ steps: id: 'pull-relations' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/relations:latest', '-t', 'gcr.io/oss-vdb/relations:$COMMIT_SHA', '-f', 'cmd/relations/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/relations:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/relations:latest', '-t', 'gcr.io/oss-vdb/relations:$COMMIT_SHA', '--target', 'relations', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/relations:latest', '--pull', '.'] dir: 'go' id: 'build-relations' waitFor: ['pull-relations'] @@ -174,7 +174,7 @@ steps: id: 'pull-generatesitemap' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/generatesitemap:latest', '-t', 'gcr.io/oss-vdb/generatesitemap:$COMMIT_SHA', '-f', 'cmd/generatesitemap/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/generatesitemap:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/generatesitemap:latest', '-t', 'gcr.io/oss-vdb/generatesitemap:$COMMIT_SHA', '--target', 'generatesitemap', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/generatesitemap:latest', '--pull', '.'] dir: 'go' id: 'build-generatesitemap' waitFor: ['pull-generatesitemap'] @@ -188,7 +188,7 @@ steps: id: 'pull-custommetrics' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/custommetrics:latest', '-t', 'gcr.io/oss-vdb/custommetrics:$COMMIT_SHA', '-f', 'cmd/custommetrics/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/custommetrics:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/custommetrics:latest', '-t', 'gcr.io/oss-vdb/custommetrics:$COMMIT_SHA', '--target', 'custommetrics', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/custommetrics:latest', '--pull', '.'] dir: 'go' id: 'build-custommetrics' waitFor: ['pull-custommetrics'] @@ -202,7 +202,7 @@ steps: id: 'pull-gitter' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/gitter:latest', '-t', 'gcr.io/oss-vdb/gitter:$COMMIT_SHA', '-f', 'cmd/gitter/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/gitter:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/gitter:latest', '-t', 'gcr.io/oss-vdb/gitter:$COMMIT_SHA', '--target', 'gitter', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/gitter:latest', '--pull', '.'] dir: 'go' id: 'build-gitter' waitFor: ['pull-gitter'] @@ -222,7 +222,7 @@ steps: # Build/push osv-linter images to gcr.io/oss-vdb-test. - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb-test/osv-linter:latest', '-t', 'gcr.io/oss-vdb-test/osv-linter:$COMMIT_SHA', '-f', 'cmd/osv-linter-worker/Dockerfile', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb-test/osv-linter:latest', '-t', 'gcr.io/oss-vdb-test/osv-linter:$COMMIT_SHA', '--build-context', 'bindings=../bindings', '-f', 'cmd/osv-linter-worker/Dockerfile', '.'] dir: 'go' id: 'build-osv-linter' waitFor: ['build-worker'] @@ -344,7 +344,7 @@ steps: id: 'pull-first-package-finder' waitFor: ['setup'] - name: gcr.io/cloud-builders/docker - args: ['build', '-t', 'gcr.io/oss-vdb/run_first_package_finder:latest', '-t', 'gcr.io/oss-vdb/run_first_package_finder:$COMMIT_SHA', '-f', 'cmd/first_package_finder/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/run_first_package_finder:latest', '--pull', '.'] + args: ['build', '-t', 'gcr.io/oss-vdb/run_first_package_finder:latest', '-t', 'gcr.io/oss-vdb/run_first_package_finder:$COMMIT_SHA', '--target', 'first_package_finder', '--build-context', 'bindings=../bindings', '-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/run_first_package_finder:latest', '--pull', '.'] dir: 'go' id: 'build-first-package-finder' waitFor: ['pull-first-package-finder'] diff --git a/go/.golangci.yaml b/go/.golangci.yaml index c66c97b37a0..4d7c582435d 100644 --- a/go/.golangci.yaml +++ b/go/.golangci.yaml @@ -7,6 +7,7 @@ linters: - forbidigo - paralleltest - tparallel + - gomoddirectives # Allow local replace directives since this is a private module - cyclop # - depguard # Too annoying - err113 # will re-add later (another-rex) diff --git a/go/Dockerfile b/go/Dockerfile new file mode 100644 index 00000000000..8c43877bdd8 --- /dev/null +++ b/go/Dockerfile @@ -0,0 +1,135 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ======================================================== +# Shared Builder Stage +# ======================================================== +FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS builder + +WORKDIR /workspace + +# Mount the 'bindings' context into the container's filesystem +COPY --from=bindings . /workspace/bindings/ + +# Copy the main Go module (strictly go/) +COPY . /workspace/go/ + +WORKDIR /workspace/go +RUN go mod download && go mod verify + +# ======================================================== +# Target: Importer +# ======================================================== +FROM builder AS importer-build +RUN CGO_ENABLED=0 go build -o /app/importer ./cmd/importer/ + +FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 AS importer +# Need to install the full tar package, to not use the busybox version, which doesn't have --zstd support. +RUN apk add --no-cache git zstd tar +COPY --from=importer-build /app/importer / +ENTRYPOINT ["/importer"] + +# ======================================================== +# Target: Worker +# ======================================================== +FROM builder AS worker-build +RUN CGO_ENABLED=0 go build -o /app/worker ./cmd/worker/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS worker +COPY --from=worker-build /app/worker / +ENTRYPOINT ["/worker"] + +# ======================================================== +# Target: Exporter +# ======================================================== +FROM builder AS exporter-build +RUN CGO_ENABLED=0 go build -o /app/exporter ./cmd/exporter/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS exporter +COPY --from=exporter-build /app/exporter / +ENTRYPOINT ["/exporter"] + +# ======================================================== +# Target: Record Checker +# ======================================================== +FROM builder AS recordchecker-build +RUN CGO_ENABLED=0 go build -o /app/recordchecker ./cmd/recordchecker/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS recordchecker +COPY --from=recordchecker-build /app/recordchecker / +ENTRYPOINT ["/recordchecker"] + +# ======================================================== +# Target: Relations +# ======================================================== +FROM builder AS relations-build +RUN CGO_ENABLED=0 go build -o /app/relations ./cmd/relations/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS relations +COPY --from=relations-build /app/relations / +ENTRYPOINT ["/relations"] + +# ======================================================== +# Target: Generate Sitemap +# ======================================================== +FROM builder AS generatesitemap-build +RUN CGO_ENABLED=0 go build -o /app/generatesitemap ./cmd/generatesitemap/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS generatesitemap +COPY --from=generatesitemap-build /app/generatesitemap / +ENTRYPOINT ["/generatesitemap"] + +# ======================================================== +# Target: Custom Metrics +# ======================================================== +FROM builder AS custommetrics-build +RUN CGO_ENABLED=0 go build -o /app/custommetrics ./cmd/custommetrics/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS custommetrics +COPY --from=custommetrics-build /app/custommetrics / +ENTRYPOINT ["/custommetrics"] + +# ======================================================== +# Target: Gitter +# ======================================================== +FROM builder AS gitter-build +RUN CGO_ENABLED=0 go build -o /app/gitter ./cmd/gitter/ + +FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 AS gitter +# Need to install the full tar package, to not use the busybox version, which doesn't have --zstd support. +RUN apk add --no-cache bash git zstd tar +# Set maintenance.autoDetach to false so git's automatic maintenance will not detach and run in the background +RUN git config --system maintenance.autoDetach false +COPY --from=gitter-build /app/gitter / +ENTRYPOINT ["/gitter"] + +# ======================================================== +# Target: First Package Finder +# ======================================================== +FROM builder AS first_package_finder-build +RUN CGO_ENABLED=0 go build -o /app/first_package_finder ./cmd/first_package_finder/ + +FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 AS first_package_finder +COPY --from=first_package_finder-build /app/first_package_finder / +ENTRYPOINT ["/first_package_finder"] + +# ======================================================== +# Target: OSV API +# ======================================================== +FROM builder AS api-build +RUN CGO_ENABLED=0 go build -o /app/api ./cmd/api/ + +FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 AS api +COPY --from=api-build /app/api / +ENTRYPOINT ["/api"] diff --git a/go/cmd/api-devserver/main.go b/go/cmd/api-devserver/main.go new file mode 100644 index 00000000000..924a28b6a0f --- /dev/null +++ b/go/cmd/api-devserver/main.go @@ -0,0 +1,151 @@ +// Package main implements the native developer server orchestrator. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "os" + "os/exec" + "os/signal" + "path/filepath" + "syscall" + + "github.com/google/osv.dev/go/internal/api" + "github.com/google/osv.dev/go/logger" +) + +const ( + defaultESPPort = 8080 + defaultBackendPort = 8000 + dockerImage = "osv/esp:latest" + containerName = "osv-esp" +) + +func main() { + if err := run(); err != nil { + os.Exit(1) + } +} + +func run() error { + logger.InitGlobalLogger() + defer logger.Close() + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + espPort := flag.Int("port", defaultESPPort, "ESP listener port") + backendPort := flag.Int("backend-port", defaultBackendPort, "Backend server port") + credPath := flag.String("cred", "", "Path to GCP Service Account credential JSON (defaults to local Application Default Credentials)") + noBackend := flag.Bool("no-backend", false, "Start ESPv2 without launching the Go API backend server") + flag.Parse() + + // Auto-discover Application Default Credentials (ADC) + if *credPath == "" { + home, err := os.UserHomeDir() + if err == nil { + adcPath := filepath.Join(home, ".config/gcloud/application_default_credentials.json") + if _, err := os.Stat(adcPath); err == nil { + *credPath = adcPath + } + } + } + + if *credPath == "" { + err := errors.New("path to GCP Service Account credential JSON file is required (please run 'gcloud auth application-default login' or specify via -cred)") + logger.ErrorContext(ctx, err.Error()) + + return err + } + + // Stop any orphaned container from a previous run + _ = exec.CommandContext(ctx, "docker", "stop", containerName).Run() + + if !*noBackend { + logger.InfoContext(ctx, "Starting Go API backend natively", "port", *backendPort) + go func() { + if err := api.RunServer(ctx, *backendPort); err != nil { + logger.ErrorContext(ctx, "Go API server exited", "error", err) + } + }() + } + + logger.InfoContext(ctx, "Starting ESPv2 container", "port", *espPort, "backendPort", *backendPort) + + credDir := filepath.Dir(*credPath) + credName := filepath.Base(*credPath) + + dockerArgs := []string{ + "run", + "--name", containerName, + "--network=host", + "--rm", + "-v", credDir + ":/esp:ro", + fmt.Sprintf("--publish=%d", *espPort), + dockerImage, + "--disable_tracing", + "--service=api-test.osv.dev", + "--rollout_strategy=managed", + "--underscores_in_headers", + fmt.Sprintf("--listener_port=%d", *espPort), + fmt.Sprintf("--backend=grpc://localhost:%d", *backendPort), + "--service_account_key=/esp/" + credName, + "--non_gcp", + "--enable_debug", + "--transcoding_preserve_proto_field_names", + "--envoy_connection_buffer_limit_bytes=104857600", + } + + // Redirect ESPv2 output to esp.log to keep the console clean + espLog, err := os.OpenFile("esp.log", os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666) + if err != nil { + err = fmt.Errorf("failed to create esp.log: %w", err) + logger.ErrorContext(ctx, err.Error()) + + return err + } + defer espLog.Close() + + espCmd := exec.CommandContext(ctx, "docker", dockerArgs...) + espCmd.Stdout = espLog + espCmd.Stderr = espLog + + if err := espCmd.Start(); err != nil { + err = fmt.Errorf("failed to start ESPv2: %w", err) + logger.ErrorContext(ctx, err.Error()) + + return err + } + + defer func() { + logger.InfoContext(ctx, "Stopping ESPv2 docker container...") + // Use background context to ensure cleanup runs even if parent context is cancelled + _ = exec.Command("docker", "stop", containerName).Run() + }() + + // Wait for the ESP container to stop, or for context to be cancelled + select { + case err := <-runCmdAsync(espCmd): + if err != nil { + err = fmt.Errorf("ESPv2 exited with error: %w", err) + logger.ErrorContext(ctx, err.Error()) + + return err + } + case <-ctx.Done(): + logger.InfoContext(ctx, "Received termination signal, shutting down cleanly...") + } + + return nil +} + +func runCmdAsync(cmd *exec.Cmd) <-chan error { + out := make(chan error, 1) + go func() { + out <- cmd.Wait() + }() + + return out +} diff --git a/go/cmd/api/main.go b/go/cmd/api/main.go new file mode 100644 index 00000000000..fcc2725203a --- /dev/null +++ b/go/cmd/api/main.go @@ -0,0 +1,32 @@ +// Package main implements the entry point for the production OSV API server. +package main + +import ( + "context" + "flag" + "os" + "os/signal" + "syscall" + + "github.com/google/osv.dev/go/internal/api" + "github.com/google/osv.dev/go/logger" +) + +func main() { + if err := run(); err != nil { + os.Exit(1) + } +} + +func run() error { + logger.InitGlobalLogger() + defer logger.Close() + + port := flag.Int("port", 8000, "port for the OSV API") + flag.Parse() + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + return api.RunServer(ctx, *port) +} diff --git a/go/cmd/custommetrics/Dockerfile b/go/cmd/custommetrics/Dockerfile deleted file mode 100644 index 2e857b07892..00000000000 --- a/go/cmd/custommetrics/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o custommetrics ./cmd/custommetrics - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/custommetrics / - -ENTRYPOINT ["/custommetrics"] diff --git a/go/cmd/exporter/Dockerfile b/go/cmd/exporter/Dockerfile deleted file mode 100644 index e031d52b8bf..00000000000 --- a/go/cmd/exporter/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o exporter ./cmd/exporter - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/exporter / - -ENTRYPOINT ["/exporter"] diff --git a/go/cmd/first_package_finder/Dockerfile b/go/cmd/first_package_finder/Dockerfile deleted file mode 100644 index 256a1ce707b..00000000000 --- a/go/cmd/first_package_finder/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o first_package_finder ./cmd/first_package_finder/ - -FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 - -WORKDIR /src -COPY --from=build /src/first_package_finder ./first_package_finder - -ENTRYPOINT ["./first_package_finder"] diff --git a/go/cmd/generatesitemap/Dockerfile b/go/cmd/generatesitemap/Dockerfile deleted file mode 100644 index 3d819c653a4..00000000000 --- a/go/cmd/generatesitemap/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o generatesitemap ./cmd/generatesitemap - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/generatesitemap / - -ENTRYPOINT ["/generatesitemap"] diff --git a/go/cmd/gitter/Dockerfile b/go/cmd/gitter/Dockerfile deleted file mode 100644 index 74d592e256d..00000000000 --- a/go/cmd/gitter/Dockerfile +++ /dev/null @@ -1,37 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o gitter ./cmd/gitter/ - -FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 - -# Need to install the full tar package, to not use the busybox version, which doesn't have --zstd support. -RUN apk add --no-cache bash git zstd tar - -# Set maintenance.autoDetach to false so git's automatic maintenance will not detach and run in the background -RUN git config --system maintenance.autoDetach false - -COPY --from=build /src/gitter / - -ENTRYPOINT ["/gitter"] diff --git a/go/cmd/importer/Dockerfile b/go/cmd/importer/Dockerfile deleted file mode 100644 index 39ab9d7fc1b..00000000000 --- a/go/cmd/importer/Dockerfile +++ /dev/null @@ -1,34 +0,0 @@ -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o importer ./cmd/importer/ - -FROM alpine:3.23@sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11 - -# Need to install the full tar package, to not use the busybox version, which doesn't have --zstd support. -RUN apk add --no-cache git zstd tar - -COPY --from=build /src/importer / - -ENTRYPOINT ["/importer"] diff --git a/go/cmd/osv-linter-worker/Dockerfile b/go/cmd/osv-linter-worker/Dockerfile index f275928dfbc..b5206b63fc1 100644 --- a/go/cmd/osv-linter-worker/Dockerfile +++ b/go/cmd/osv-linter-worker/Dockerfile @@ -31,6 +31,7 @@ RUN go generate ./... RUN CGO_ENABLED=0 GOOS=linux go build -o /usr/local/bin/osv-linter ./cmd/osv # Build the worker binary +COPY --from=bindings . /src/osv.dev/bindings/ WORKDIR /src/osv.dev/go COPY go.mod go.sum ./ RUN go mod download diff --git a/go/cmd/recordchecker/Dockerfile b/go/cmd/recordchecker/Dockerfile deleted file mode 100644 index 96999edc9d9..00000000000 --- a/go/cmd/recordchecker/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o recordchecker ./cmd/recordchecker/ - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/recordchecker / - -ENTRYPOINT ["/recordchecker"] diff --git a/go/cmd/relations/Dockerfile b/go/cmd/relations/Dockerfile deleted file mode 100644 index 779add86a65..00000000000 --- a/go/cmd/relations/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o relations ./cmd/relations - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/relations / - -ENTRYPOINT ["/relations"] diff --git a/go/cmd/worker/Dockerfile b/go/cmd/worker/Dockerfile deleted file mode 100644 index c7dc8ca4071..00000000000 --- a/go/cmd/worker/Dockerfile +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM golang:1.26.3-alpine@sha256:91eda9776261207ea25fd06b5b7fed8d397dd2c0a283e77f2ab6e91bfa71079d AS build - -WORKDIR /src - -COPY ./go.mod /src/go.mod -COPY ./go.sum /src/go.sum -RUN go mod download && go mod verify - - -COPY ./ /src/ -RUN CGO_ENABLED=0 go build -o worker ./cmd/worker - -FROM gcr.io/distroless/static-debian12@sha256:9c346e4be81b5ca7ff31a0d89eaeade58b0f95cfd3baed1f36083ddb47ca3160 - -COPY --from=build /src/worker / - -ENTRYPOINT ["/worker"] diff --git a/go/go.mod b/go/go.mod index c3ed18f0f2e..342c6251dcd 100644 --- a/go/go.mod +++ b/go/go.mod @@ -2,6 +2,8 @@ module github.com/google/osv.dev/go go 1.26.3 +replace osv.dev/bindings/go => ../bindings/go + require ( charm.land/lipgloss/v2 v2.0.3 cloud.google.com/go/datastore v1.23.0 @@ -28,9 +30,11 @@ require ( golang.org/x/sync v0.20.0 google.golang.org/api v0.279.0 google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 + google.golang.org/grpc v1.81.0 google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af gopkg.in/dnaeon/go-vcr.v4 v4.0.6 k8s.io/apimachinery v0.36.1 + osv.dev/bindings/go v0.0.0-20251029235023-a02c549eeac2 ) require ( @@ -56,10 +60,10 @@ require ( github.com/clipperhouse/displaywidth v0.11.0 // indirect github.com/clipperhouse/uax29/v2 v2.7.0 // indirect github.com/cloudflare/circl v1.6.3 // indirect - github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 // indirect + github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect github.com/emirpasic/gods v1.18.1 // indirect - github.com/envoyproxy/go-control-plane/envoy v1.36.0 // indirect - github.com/envoyproxy/protoc-gen-validate v1.3.0 // indirect + github.com/envoyproxy/go-control-plane/envoy v1.37.0 // indirect + github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect github.com/go-git/gcfg/v2 v2.0.2 // indirect github.com/go-git/go-billy/v6 v6.0.0-alpha.1 // indirect @@ -101,7 +105,6 @@ require ( golang.org/x/time v0.15.0 // indirect google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect - google.golang.org/grpc v1.80.0 // indirect sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/go/go.sum b/go/go.sum index edb88acf2ac..953c1c995db 100644 --- a/go/go.sum +++ b/go/go.sum @@ -69,8 +69,8 @@ github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJ github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8= github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= -github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 h1:6xNmx7iTtyBRev0+D/Tv1FZd4SCg8axKApyNyRsAt/w= -github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5/go.mod h1:KdCmV+x/BuvyMxRnYBlmVaq4OLiKW6iRQfvC62cvdkI= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os8IaYg++6uMOdKK83QtkkvJik= +github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= @@ -88,13 +88,13 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/go-control-plane v0.14.0 h1:hbG2kr4RuFj222B6+7T83thSPqLjwBIfQawTkC++2HA= github.com/envoyproxy/go-control-plane v0.14.0/go.mod h1:NcS5X47pLl/hfqxU70yPwL9ZMkUlwlKxtAohpi2wBEU= -github.com/envoyproxy/go-control-plane/envoy v1.36.0 h1:yg/JjO5E7ubRyKX3m07GF3reDNEnfOboJ0QySbH736g= -github.com/envoyproxy/go-control-plane/envoy v1.36.0/go.mod h1:ty89S1YCCVruQAm9OtKeEkQLTb+Lkz0k8v9W0Oxsv98= +github.com/envoyproxy/go-control-plane/envoy v1.37.0 h1:u3riX6BoYRfF4Dr7dwSOroNfdSbEPe9Yyl09/B6wBrQ= +github.com/envoyproxy/go-control-plane/envoy v1.37.0/go.mod h1:DReE9MMrmecPy+YvQOAOHNYMALuowAnbjjEMkkWOi6A= github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI= github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/envoyproxy/protoc-gen-validate v1.3.0 h1:TvGH1wof4H33rezVKWSpqKz5NXWg5VPuZ0uONDT6eb4= -github.com/envoyproxy/protoc-gen-validate v1.3.0/go.mod h1:HvYl7zwPa5mffgyeTUHA9zHIH36nmrm7oCbo4YKoSWA= +github.com/envoyproxy/protoc-gen-validate v1.3.3 h1:MVQghNeW+LZcmXe7SY1V36Z+WFMDjpqGAGacLe2T0ds= +github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0= github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM= github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE= github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= @@ -324,8 +324,8 @@ google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyac google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= -google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= -google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= +google.golang.org/grpc v1.81.0 h1:W3G9N3KQf3BU+YuCtGKJk0CmxQNbAISICD/9AORxLIw= +google.golang.org/grpc v1.81.0/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= diff --git a/go/internal/api/server.go b/go/internal/api/server.go new file mode 100644 index 00000000000..ffc703fcff7 --- /dev/null +++ b/go/internal/api/server.go @@ -0,0 +1,52 @@ +// Package api implements the public gRPC server API for OSV. +package api + +import ( + "context" + "fmt" + "net" + + "github.com/google/osv.dev/go/logger" + "google.golang.org/grpc" + pb "osv.dev/bindings/go/api" +) + +type server struct { + pb.UnimplementedOSVServer +} + +// RunServer starts the gRPC server and handles graceful shutdown. +func RunServer(ctx context.Context, port int) error { + lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) + if err != nil { + logger.ErrorContext(ctx, "failed to listen", "error", err) + return err + } + + s := grpc.NewServer() + pb.RegisterOSVServer(s, &server{}) + + logger.InfoContext(ctx, "server listening", "port", port) + + serveErr := make(chan error, 1) + go func() { + serveErr <- s.Serve(lis) + }() + + select { + case err := <-serveErr: + if err != nil { + logger.ErrorContext(ctx, "server failed to serve", "error", err) + return err + } + case <-ctx.Done(): + logger.InfoContext(ctx, "received shutdown signal, shutting down server gracefully") + s.GracefulStop() + if err := <-serveErr; err != nil { + logger.ErrorContext(ctx, "server failed during shutdown", "error", err) + return err + } + } + + return nil +} From 491b7f0d860cb03df451ffaf06bf2cc9ea7ec632 Mon Sep 17 00:00:00 2001 From: michaelkedar Date: Thu, 21 May 2026 00:36:08 +0000 Subject: [PATCH 2/4] SIGINT --- go/cmd/api-devserver/main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/go/cmd/api-devserver/main.go b/go/cmd/api-devserver/main.go index 924a28b6a0f..81c48be9f07 100644 --- a/go/cmd/api-devserver/main.go +++ b/go/cmd/api-devserver/main.go @@ -33,7 +33,7 @@ func run() error { logger.InitGlobalLogger() defer logger.Close() - ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) defer stop() espPort := flag.Int("port", defaultESPPort, "ESP listener port") From f18e6c2e47b754006687f8c92e2060c124547c1c Mon Sep 17 00:00:00 2001 From: michaelkedar Date: Thu, 21 May 2026 00:39:44 +0000 Subject: [PATCH 3/4] =?UTF-8?q?=F0=9F=AA=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTRIBUTING.md | 10 ++++++++++ go/Dockerfile | 14 ++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c70ebcd348f..e198c7ef441 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -159,6 +159,16 @@ make update-api-snapshots and check the git diff to see if the API result changes are expected. +#### Go Monorepo Docker Builds + +All Go microservices are compiled using a single, unified multi-target Dockerfile (`go/Dockerfile`). +- Due to the `replace` directive in `go/go.mod` pointing to the sibling `bindings/` library, Go Docker builds **must** mount the `bindings/` folder into the build context. +- To build a Go service container (e.g. `importer`) locally, run the `docker build` command inside the `go/` directory using BuildKit's `--build-context` flag to map the sibling directory: + ```shell + cd go + docker build -t osv/importer --target importer --build-context bindings=../bindings -f Dockerfile . + ``` + ### Frontend development #### Running a local UI instance (maintainers only) diff --git a/go/Dockerfile b/go/Dockerfile index 8c43877bdd8..bfd206cc9fe 100644 --- a/go/Dockerfile +++ b/go/Dockerfile @@ -12,6 +12,20 @@ # See the License for the specific language governing permissions and # limitations under the License. +# ==================================================================================== +# HOW TO BUILD THIS DOCKERFILE LOCALLY: +# +# Because go/go.mod contains a relative replacement directive for bindings/go, +# you must build this Dockerfile from the go/ directory and explicitly map +# the sibling bindings/ folder in BuildKit using --build-context: +# +# cd go +# docker build -t osv/importer --target importer --build-context bindings=../bindings -f Dockerfile . +# +# Select which service to build using the --target flag (e.g. importer, worker, exporter, +# relations, recordchecker, generatesitemap, custommetrics, gitter, first_package_finder, api). +# ==================================================================================== + # ======================================================== # Shared Builder Stage # ======================================================== From 0031d55df79e856b35d53d4fcc705c1303283685 Mon Sep 17 00:00:00 2001 From: michaelkedar Date: Thu, 21 May 2026 00:45:14 +0000 Subject: [PATCH 4/4] copy Makefile comment over --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index 11f7446fb4d..3f61fc74532 100644 --- a/Makefile +++ b/Makefile @@ -110,6 +110,7 @@ run-api-server: cd gcp/api && $(install-cmd) && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities $(run-cmd) python test_server.py $(HOME)/.config/gcloud/application_default_credentials.json $(ARGS) # Run the native Go developer server orchestrator (launches both ESPv2 and the Go API server). +# Run with `run-go-api ARGS=--no-backend` to launch esp without backend. run-go-api: test -f $(HOME)/.config/gcloud/application_default_credentials.json || (echo "GCP Application Default Credentials not set, try 'gcloud auth application-default login'"; exit 1) docker inspect osv/esp:latest >/dev/null 2>&1 || (cd gcp/api && docker build -f Dockerfile.esp -t osv/esp:latest .)