Exploit Intelligence Component Syncer

The Exploit Intelligence Component Syncer processes Git repository content and synchronizes it to S3-compatible storage. It clones repositories, serializes documents into pickle files, and uploads the processed content to an S3-compatible bucket.

ExploitIQ client triggers this service through Knative Events and JobSink. For development and testing, you can run it locally without a Knative cluster.

Features

• Clones a Git repository and serializes processed documents into a pickle file.
• Synchronizes repository content to an S3-compatible bucket.
• Uses commit SHA metadata to skip synchronization when content is already up to date.

Prerequisites

• Python 3.12
• Git
• Docker or Podman
• uv package manager

Installation

1. Create a Virtual Environment

uv venv --python 3.12
source .venv/bin/activate

2. Install Dependencies

GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

Note

GIT_LFS_SKIP_SMUDGE=1 is required because the exploit_iq_commons dependency is sourced from a repository that uses Git LFS. Without this flag, the installation fails when Git attempts to download LFS objects.

Running the Service

The service reads a Knative event file, resolves the current commit SHA for each component, and synchronizes repository content to S3.

K_EVENT_PATH=events/event_single.json uv run python src/main.py

The service performs the following steps:

1. Reads the Knative event file and parses the list of components to process.
2. For each component, resolves the current commit SHA from the remote Git repository.
3. Checks whether the S3 bucket already contains a pickle file with a matching commit SHA.
4. If the content is up to date, then the service skips processing.
5. If the content is outdated or missing, then the service clones the repository, collects documents, uploads all repository files to S3, and stores the pickle file with the commit SHA as metadata.
6. Reports results to the backend API.

To run the service, you need an S3-compatible storage backend and a backend API endpoint. Refer to the Local Development section if you do not have these available.

Local Development

This section describes how to run the service locally without a Knative cluster or a production backend.

1. Set Up S3 Storage

The service requires an S3-compatible storage backend.

• MinIO (recommended for local development): See docs/minio.md for automated and manual setup instructions.
• AWS S3: See docs/aws-s3.md for an end-to-end guide covering bucket creation, IAM configuration, and verification.

2. Configure Environment Variables

Create a .env file in the project root:

S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
API_ENDPOINT=http://localhost:8080/api/v1
AUTH_TOKEN=local-dev-token

Environment Variable Reference

Variable	Required	Default	Description
S3_ACCESS_KEY	✓	—	Access key for S3-compatible storage
S3_SECRET_KEY	✓	—	Secret key for S3-compatible storage
API_ENDPOINT	✓	—	Backend API endpoint for report submission
AUTH_TOKEN	✓*	—	Bearer token sent in the Authorization header for all backend API requests. For local development with a mock backend, any non-empty string works.
AUTH_TOKEN_PATH	✓*	/var/run/secrets/kubernetes.io/serviceaccount/token	Path to a file containing the Bearer token. On OpenShift or Kubernetes, the pod's service account token is automatically mounted at the default path, so you do not need to set this variable.
S3_ENDPOINT		http://localhost:9000	S3-compatible storage endpoint (include scheme)
S3_BUCKET_NAME		cloud-syncer	S3 bucket name
S3_VERIFY_TLS		true	Verify TLS certificates for S3 connections. Set to false for self-signed certificates.
S3_CA_BUNDLE		—	Path to a custom CA bundle PEM file. Overrides S3_VERIFY_TLS when set.
S3_REGION		—	AWS region for the S3 bucket (required for non-us-east-1 AWS S3 buckets)
S3_RETRIES		3	Number of retries for S3 operations
S3_UPLOAD_WORKERS		1	Number of parallel upload threads. Increase to 10–20 for AWS S3.
JOB_COMPLETION_INDEX		0	Index of the current worker pod (0-based)
K_EVENT_PATH		/etc/jobsink-event	Path to the Knative event file
LOG_LEVEL		INFO	Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
NUMBER_OF_WORKERS		1	Total number of worker pods
VERIFY_TLS		true	Verify TLS certificates for backend API requests

*Either AUTH_TOKEN or AUTH_TOKEN_PATH must be set.

3. Start a Mock Backend

If you do not have a running ExploitIQ backend, start a local mock server that accepts all POST requests and returns HTTP 202. The service requires a reachable API_ENDPOINT to report processing results.

python3 -c "
from http.server import HTTPServer, BaseHTTPRequestHandler
class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(202)
        self.end_headers()
        print(f'POST {self.path} -> 202')
    def log_message(self, *a): pass
print('Mock backend listening on :8080')
HTTPServer(('0.0.0.0', 8080), H).serve_forever()
"

Keep this running in a separate terminal before starting the service.

4. Trigger Without Knative

In production, Knative delivers events to the service by writing an event file to the pod. To replicate this locally, use one of the following methods.

Method 1 — Direct file reference (simplest):

K_EVENT_PATH=events/event_single.json uv run python src/main.py

Method 2 — HTTP trigger server (simulates Knative delivery):

1. Start the trigger server:

   TRIGGER_PORT=18080 TRIGGER_ENDPOINT=/trigger python trigger.py

2. Send an event:

   curl -X POST http://localhost:18080/trigger \
     -H "Content-Type: application/json" \
     --data-binary @events/event_single.json

Running Tests

# Unit tests (no external services required)
make test

# Integration tests against a real MinIO instance (requires Docker)
make test-integration

# Integration tests against a real AWS S3 bucket
S3_TEST_BUCKET=my-bucket make test-s3

Building the Container Image

make build   # auto-detects Docker or Podman
make push