Skip to content

feat(log-ingestor): Add realtime OTLP log ingestion#2329

Draft
junhaoliao wants to merge 3 commits into
y-scope:mainfrom
junhaoliao:feat/realtime-log-ingestion
Draft

feat(log-ingestor): Add realtime OTLP log ingestion#2329
junhaoliao wants to merge 3 commits into
y-scope:mainfrom
junhaoliao:feat/realtime-log-ingestion

Conversation

@junhaoliao

@junhaoliao junhaoliao commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

This PR adds a v1 realtime log ingestion path for CLP-S:

  • log-ingestor can receive OTLP/HTTP logs at /v1/logs, normalize OTLP records into JSONL, and persist them into filesystem-backed hot segments.
  • Hot segment metadata is tracked in SQL through hot_log_segments, including committed byte offsets, record counts, timestamp ranges, and segment lifecycle state.
  • Query scheduling can include hot JSONL segment targets alongside archive targets. Search workers execute hot targets from a stable snapshot offset and write compatible Mongo results-cache documents.
  • API/WebUI search requests include hot segments by default for normal searches. Timeline aggregation remains archive-only for v1.
  • Docker Compose and Helm mount shared realtime hot storage into log-ingestor, query workers, and compression workers.
  • realtime-logs-plan.md documents the implementation plan, v1 limitations, delivery semantics, and validation instructions. Package validation must use build/clp-package/sbin/start-clp.sh and build/clp-package/sbin/stop-clp.sh; do not use raw docker compose up/down for package lifecycle validation.

V1 limitations:

  • Hot storage is filesystem-backed and must be visible to log-ingestor, query workers, and compression workers.
  • Built-in log-ingestor auth, direct JSONL ingestion, Fluent Bit integration, and object-storage hot segments are future work.
  • Hot aggregation is disabled; aggregation jobs force archive-only search.
  • Hot JSONL search currently uses a temporary CLP-S snapshot archive per hot segment target instead of a native raw JSONL C++ search path.

Impact Assessment

This affects log-ingestor, the package config schema, orchestration DB initialization, query scheduling/task dispatch, API/WebUI search request handling, Docker Compose deployment, and Helm deployment. The change is needed so freshly ingested structured logs can become searchable before they are compacted into CLP-S archives.

The main runtime implication is a new shared hot-storage volume and SQL metadata table. Ingestion success means records have been appended and flushed to the hot segment file; query workers only search up to the SQL-published committed_end_offset. The system provides at-least-once ingestion semantics: acknowledged records should not be lost under the v1 durability policy, but duplicate upstream delivery can still produce duplicate records unless a future dedupe layer is added.

Operationally, realtime log ingestion is independent of S3 ingestion and anonymous telemetry. The OpenTelemetry Collector can forward logs to log-ingestor, but it is not responsible for writing CLP hot segment files.

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Task: Stop the existing package before rebuilding

Command:

cd build/clp-package && ./sbin/stop-clp.sh

Output:

 Container clp-package-clp-runtime-run-0f9df5e563c7 Creating 
 Container clp-package-clp-runtime-run-0f9df5e563c7 Created 
2026-06-11T19:38:32.328 INFO [controller] Stopping all CLP containers using Docker Compose...
...
 Network clp-package-526de957-69c7-4cde-943d-e7f6a427983e_default Removed 
2026-06-11T19:38:54.805 INFO [controller] Stopped CLP.

Explanation: Stopped the running package through the package script before rebuilding. This avoids using raw Docker Compose lifecycle commands.

Task: Build the CLP package with CPU-matched build parallelism

Command:

CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN) task package

Output:

...
    Finished `release` profile [optimized] target(s) in 2m 15s
✓ 6151 modules transformed.
rendering chunks...
computing gzip size...
../../../build/webui/client/index.html                             0.94 kB │ gzip:   0.46 kB
../../../build/webui/client/assets/codicon-B_Z2XQ3P.ttf           90.73 kB
../../../build/webui/client/assets/editor.worker-CHMq8uMJ.js     250.72 kB
../../../build/webui/client/assets/index-BLrcVHjM.css              3.92 kB │ gzip:   1.43 kB
../../../build/webui/client/assets/monaco-editor-C2NR8L7s.css    141.33 kB │ gzip:  22.82 kB
../../../build/webui/client/assets/sql-BdTr02Mf.js                10.16 kB │ gzip:   3.79 kB
../../../build/webui/client/assets/monaco-editor-CAmGTfSf.js   2,794.29 kB │ gzip: 720.82 kB
../../../build/webui/client/assets/index-CAItAq9N.js           3,268.90 kB │ gzip: 987.67 kB
...
#30 naming to moby-dangling@sha256:dc993737514437d5bc28ebd5c843034df5a090bfb79845d1a6962c4001f80a3e done
#30 unpacking to moby-dangling@sha256:dc993737514437d5bc28ebd5c843034df5a090bfb79845d1a6962c4001f80a3e 3.0s done
#30 DONE 7.6s
Removing previous image sha256:b3aa54ee8c10548461ec3724b0a1d852af60bd90562a724c276e3e96e8e2f28f.
Untagged: clp-package:dev-junhao-b3aa
Deleted: sha256:b3aa54ee8c10548461ec3724b0a1d852af60bd90562a724c276e3e96e8e2f28f
...
task: [package] echo '0.12.1-dev' > '/home/junhao/.codex/worktrees/5ef4/clp/build/clp-package/VERSION'

Explanation: Rebuilt the package after Rust/TypeScript autofix. getconf _NPROCESSORS_ONLN returned 8, so C++ package build tasks ran with parallelism 8.

Task: Start the rebuilt package

Command:

cd build/clp-package && ./sbin/start-clp.sh

Output:

 Container clp-package-clp-runtime-run-81e3afd54a5a Creating 
 Container clp-package-clp-runtime-run-81e3afd54a5a Created 
2026-06-11T19:44:34.062 INFO [controller] Setting up environment for telemetry...
2026-06-11T19:44:34.076 INFO [controller] Setting up environment for log_ingestor...
2026-06-11T19:44:34.219 INFO [controller] Starting CLP using Docker Compose (full deployment)...
...
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-log-ingestor-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-database-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-db-table-creator-1 Exited 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-redis-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-queue-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-results-cache-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-api-server-1 Healthy 
 Container clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601-compression-worker-1 Healthy 
2026-06-11T19:45:08.308 INFO [controller] Started CLP.

Explanation: Started the rebuilt package through sbin/start-clp.sh with realtime logs enabled in build/clp-package/etc/clp-config.yaml.

Task: Compress archive data for mixed hot/archive search

Command:

cd build/clp-package && ./sbin/compress.sh --timestamp-key timestamp ~/samples/postgresql.jsonl

Output:

 Container clp-package-clp-runtime-run-63c1d93711c6 Creating 
 Container clp-package-clp-runtime-run-63c1d93711c6 Created 
2026-06-11T19:45:29.289 INFO [compress] Compression job 1 submitted.
2026-06-11T19:45:35.346 INFO [compress] Compression finished.
2026-06-11T19:45:35.346 INFO [compress] Compressed 389.02MB into 10.05MB (38.70x). Speed: 67.21MB/s.

Task: Verify log-ingestor and API health

Command:

curl -sS -i http://localhost:3002/health
curl -sS -i http://localhost:3001/health

Output:

HTTP/1.1 200 OK
content-type: text/plain; charset=utf-8
vary: origin, access-control-request-method, access-control-request-headers
access-control-allow-origin: *
content-length: 23
date: Thu, 11 Jun 2026 19:45:58 GMT

log-ingestor is running
HTTP/1.1 200 OK
content-type: text/plain; charset=utf-8
vary: origin, access-control-request-method, access-control-request-headers
access-control-allow-origin: *
content-length: 21
date: Thu, 11 Jun 2026 19:45:58 GMT

API server is running

Task: Ingest one realtime OTLP/HTTP JSON log

Command:

curl -sS -i -X POST http://localhost:3002/v1/logs \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON'
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {"key": "clp.dataset", "value": {"stringValue": "default"}},
          {"key": "service.name", "value": {"stringValue": "codex-realtime-smoke"}}
        ]
      },
      "scopeLogs": [
        {
          "scope": {"name": "codex.manual", "version": "1.0.0"},
          "logRecords": [
            {
              "timeUnixNano": "1679891195719000000",
              "observedTimeUnixNano": "1679891195719000000",
              "severityText": "INFO",
              "severityNumber": 9,
              "body": {"stringValue": "codex_hot_mix_20260611_r2 PostgreSQL realtime validation log"},
              "attributes": [
                {"key": "validation.case", "value": {"stringValue": "hot_archive_mixed_search"}}
              ]
            }
          ]
        }
      ]
    }
  ]
}
JSON

Output:

HTTP/1.1 200 OK
content-type: application/json
vary: origin, access-control-request-method, access-control-request-headers
access-control-allow-origin: *
content-length: 23
date: Thu, 11 Jun 2026 19:45:58 GMT

{"partialSuccess":null}

Task: Verify hot JSONL storage and SQL metadata

Command:

find build/clp-package/var/data/realtime-logs -type f -maxdepth 3 -print -exec sed -n '1,3p' {} \;

db_container=$(docker ps --format '{{.Names}}' | grep 'clp-package-.*-database-1' | head -1)
db_pass=$(docker exec "$db_container" env | sed -n 's/^MYSQL_PASSWORD=//p')
docker exec "$db_container" mariadb -uclp-user -p"$db_pass" clp-db \
  -e 'SELECT id,dataset,status,storage_type,locator,size_bytes,record_count,committed_end_offset,begin_timestamp,end_timestamp FROM hot_log_segments; SELECT id,begin_timestamp,end_timestamp,uncompressed_size,size FROM clp_default_archives;'

Output:

build/clp-package/var/data/realtime-logs/default/segment-1781207158445-1.jsonl
{"timestamp":1679891195719,"observed_timestamp":1679891195719,"severity_text":"INFO","severity_number":9,"body":"codex_hot_mix_20260611_r2 PostgreSQL realtime validation log","trace_id":"","span_id":"","attributes":{"validation.case":"hot_archive_mixed_search"},"resource":{"clp.dataset":"default","service.name":"codex-realtime-smoke"},"scope":{"name":"codex.manual","version":"1.0.0","attributes":{}},"_clp":{"ingest_time":1781207158444,"source_format":"otlp","dataset":"default"}}

id	dataset	status	storage_type	locator	size_bytes	record_count	committed_end_offset	begin_timestamp	end_timestamp
1	default	0	fs	/var/data/realtime-logs/default/segment-1781207158445-1.jsonl	484	1	484	1679891195719	1679891195719
id	begin_timestamp	end_timestamp	uncompressed_size	size
642272af-7272-48e1-a08b-5c9faab24284	1679876795719	1679877135936	407917670	10541012

Explanation: The OTLP record was normalized into hot JSONL, and SQL published a searchable committed offset for the open hot segment.

Task: Verify mixed hot/archive search through the WebUI API

Command:

curl -sS -i http://localhost:4000/api/search/query \
  -H 'Content-Type: application/json' \
  --data '{"datasets":["default"],"ignoreCase":true,"includeHotSegments":true,"queryString":"*PostgreSQL*","timeRangeBucketSizeMillis":60000,"timestampBegin":1679876790000,"timestampEnd":1679891200000}'

Output:

HTTP/1.1 201 Created
x-ratelimit-limit: 1000
x-ratelimit-remaining: 999
x-ratelimit-reset: 60
content-type: application/json; charset=utf-8
content-length: 38
Date: Thu, 11 Jun 2026 19:46:42 GMT
Connection: keep-alive
Keep-Alive: timeout=72

{"searchJobId":1,"aggregationJobId":2}

Command:

db_container=$(docker ps --format '{{.Names}}' | grep 'clp-package-.*-database-1' | head -1)
db_pass=$(docker exec "$db_container" env | sed -n 's/^MYSQL_PASSWORD=//p')
docker exec "$db_container" mariadb -uclp-user -p"$db_pass" clp-db \
  -e "SELECT id,type,status,num_tasks,num_tasks_completed,duration FROM query_jobs WHERE id IN (1,2); SELECT id,job_id,status,archive_id,target_type,target_id,hot_segment_id,snapshot_end_offset FROM query_tasks WHERE job_id IN (1,2) ORDER BY id;"

Output:

id	type	status	num_tasks	num_tasks_completed	duration
1	0	2	2	2	0.887577
2	0	2	1	1	0.761032
id	job_id	status	archive_id	target_type	target_id	hot_segment_id	snapshot_end_offset
1	1	0	hot:1	hot_jsonl_segment	1	1	484
2	1	0	642272af-7272-48e1-a08b-5c9faab24284	archive	642272af-7272-48e1-a08b-5c9faab24284	NULL	NULL
3	2	0	642272af-7272-48e1-a08b-5c9faab24284	archive	642272af-7272-48e1-a08b-5c9faab24284	NULL	NULL

Explanation: The search job selected both a hot JSONL segment target and a CLP-S archive target. The aggregation job selected only the archive target.

Task: Verify mixed hot/archive results in Mongo results cache

Command:

results_container=$(docker ps --format '{{.Names}}' | grep 'clp-package-.*-results-cache-1' | head -1)
docker exec "$results_container" mongosh --quiet --eval '
const d = db.getSiblingDB("clp-query-results");
printjson(d.getCollection("results-metadata").find({_id: "1"}).toArray());
printjson(d.getCollection("1").aggregate([
  {$match: {message: /codex_hot_mix_20260611_r2/}},
  {$project: {_id: 0, dataset: 1, archive_id: 1, timestamp: 1, message_prefix: {$substrBytes: ["$message", 0, 170]}}}
]).toArray());
printjson(d.getCollection("1").aggregate([
  {$match: {message: /starting PostgreSQL 15.2/}},
  {$limit: 1},
  {$project: {_id: 0, dataset: 1, archive_id: 1, timestamp: 1, message_prefix: {$substrBytes: ["$message", 0, 170]}}}
]).toArray());
'

Output:

[
  {
    _id: '1',
    errorMsg: null,
    errorName: null,
    lastSignal: 'resp-done',
    queryEngine: 'clp-s',
    numTotalResults: 18
  }
]
[
  {
    timestamp: Long('1679891195719'),
    archive_id: '2900f418-0d81-4b58-9701-e93dc5ccdc1d',
    dataset: 'default',
    message_prefix: '{"timestamp":1679891195719,"observed_timestamp":1679891195719,"severity_text":"INFO","severity_number":9,"body":"codex_hot_mix_20260611_r2 PostgreSQL realtime validation '
  }
]
[
  {
    timestamp: Long('1679876795719'),
    archive_id: '642272af-7272-48e1-a08b-5c9faab24284',
    dataset: 'default',
    message_prefix: '{"timestamp":"2023-03-27 00:26:35.719","pid":7808,"session_id":"64211afb.1e80","line_num":2,"session_start":"2023-03-27 00:26:35 EDT","txid":0,"error_severity":"LOG","mes'
  }
]

Task: Verify the browser WebUI search flow with playwright-cli

Command:

playwright-cli goto http://localhost:4000/search
playwright-cli fill "input[placeholder='Enter your query']" "*PostgreSQL*"
playwright-cli click "button[type='submit']"

Output:

### Ran Playwright code
await page.goto('http://localhost:4000/search');
### Page
- Page URL: http://localhost:4000/search
- Page Title: YScope CLP

### Ran Playwright code
await page.locator('input[placeholder=\'Enter your query\']').fill('*PostgreSQL*');

### Ran Playwright code
await page.locator('button[type=\'submit\']').click();
### Page
- Page URL: http://localhost:4000/search
- Page Title: YScope CLP

Command:

playwright-cli --raw eval "document.body.innerText"

Output:

"Ingest\nSearch\nDataset\ndefault\nAa\nSearch\nSuccess - search job 3 found 18 results in 0.654 seconds (624.0 MB/s)\nTimestamp\n\tMessage\t\n2023-Mar-27 04:26:35\n{\"timestamp\":1679891195719,\"observed_timestamp\":1679891195719,\"severity_text\":\"INFO\",\"severity_number\":9,\"body\":\"codex_hot_mix_20260611_r2 PostgreSQL realtime validation log\",\"trace_id\":\"\",\"span_id\":\"\",\"attributes\":{\"validation.case\":\"hot_archive_mixed_search\"},\"resource\":{\"clp.dataset\":\"default\",\"service.name\":\"codex-realtime-smoke\"},\"scope\":{\"name\":\"codex.manual\",\"version\":\"1.0.0\",\"attributes\":{}},\"_clp\":{\"ingest_time\":1781207158444,\"source_format\":\"otlp\",\"dataset\":\"default\"}}\n\n Original file\n2023-Mar-27 00:27:32\n{\"timestamp\":\"2023-03-27 00:27:32.217\",\"pid\":7894,\"session_id\":\"64211b34.1ed6\",\"line_num\":5,\"session_start\":\"2023-03-27 00:27:32 EDT\",\"txid\":0,\"error_severity\":\"LOG\",\"message\":\"connection authenticated: identity=\\\"postgres\\\" method=peer (/etc/postgresql/15/main/pg_hba.conf:90)\",\"query_id\":0,\"backend_type\":\"client backend\",\"vxid\":\"10/1\",\"remote_host\":\"[local]\",\"ps\":\"authentication\",\"user\":\"postgres\",\"dbname\":\"example\"}\n\n Original file\n2023-Mar-27 00:27:32\n{\"timestamp\":\"2023-03-27 00:27:32.213\",\"pid\":7893,\"session_id\":\"64211b34.1ed5\",\"line_num\":5,\"session_start\":\"2023-03-27 00:27:32 EDT\",\"txid\":0,\"error_severity\":\"LOG\",\"message\":\"connection authenticated: identity=\\\"postgres\\\" method=peer (/etc/postgresql/15/main/pg_hba.conf:90)\",\"query_id\":0,\"backend_type\":\"client backend\",\"vxid\":\"9/1\",\"remote_host\":\"[local]\",\"ps\":\"authentication\",\"user\":\"postgres\",\"dbname\":\"example\"}\n\n Original file\n2023-Mar-27 00:27:32\n{\"timestamp\":\"2023-03-27 00:27:32.209\",\"pid\":7892,\"session_id\":\"64211b34.1ed4\",\"line_num\":5,\"session_start\":\"2023-03-27 00:27:32 EDT\",\"txid\":0,\"error_severity\":\"LOG\",\"message\":\"connection authenticated: identity=\\\"postgres\\\" method=peer (/etc/postgresql/15/main/pg_hba.conf:90)\",\"query_id\":0,\"backend_type\":\"client backend\",\"vxid\":\"8/1\",\"remote_host\":\"[local]\",\"ps\":\"authentication\",\"user\":\"postgres\",\"dbname\":\"example\"}\n\n Original file"

Explanation: The browser-driven WebUI search found the hot OTLP record and archived PostgreSQL records in one query.

Task: Verify browser-submitted mixed query target selection and results cache

Command:

db_container=$(docker ps --format '{{.Names}}' | grep 'clp-package-.*-database-1' | head -1)
db_pass=$(docker exec "$db_container" env | sed -n 's/^MYSQL_PASSWORD=//p')
docker exec "$db_container" mariadb -uclp-user -p"$db_pass" clp-db \
  -e "SELECT id,type,status,num_tasks,num_tasks_completed,duration FROM query_jobs WHERE id IN (3,4); SELECT id,job_id,status,archive_id,target_type,target_id,hot_segment_id,snapshot_end_offset FROM query_tasks WHERE job_id IN (3,4) ORDER BY id;"

Output:

id	type	status	num_tasks	num_tasks_completed	duration
3	0	2	2	2	0.653765
4	0	2	1	1	0.746224
id	job_id	status	archive_id	target_type	target_id	hot_segment_id	snapshot_end_offset
4	3	0	hot:1	hot_jsonl_segment	1	1	484
5	3	0	642272af-7272-48e1-a08b-5c9faab24284	archive	642272af-7272-48e1-a08b-5c9faab24284	NULL	NULL
6	4	0	642272af-7272-48e1-a08b-5c9faab24284	archive	642272af-7272-48e1-a08b-5c9faab24284	NULL	NULL

Command:

results_container=$(docker ps --format '{{.Names}}' | grep 'clp-package-.*-results-cache-1' | head -1)
docker exec "$results_container" mongosh --quiet --eval '
const d = db.getSiblingDB("clp-query-results");
printjson(d.getCollection("results-metadata").find({_id: "3"}).toArray());
printjson(d.getCollection("3").aggregate([
  {$match: {message: /codex_hot_mix_20260611_r2/}},
  {$project: {_id: 0, dataset: 1, archive_id: 1, timestamp: 1, message_prefix: {$substrBytes: ["$message", 0, 170]}}}
]).toArray());
printjson(d.getCollection("3").aggregate([
  {$match: {message: /starting PostgreSQL 15.2/}},
  {$limit: 1},
  {$project: {_id: 0, dataset: 1, archive_id: 1, timestamp: 1, message_prefix: {$substrBytes: ["$message", 0, 170]}}}
]).toArray());
'

Output:

[
  {
    _id: '3',
    errorMsg: null,
    errorName: null,
    lastSignal: 'resp-done',
    queryEngine: 'clp-s',
    numTotalResults: 18
  }
]
[
  {
    timestamp: Long('1679891195719'),
    archive_id: 'd6066adf-cb5f-4c55-9642-3785cec5f2ae',
    dataset: 'default',
    message_prefix: '{"timestamp":1679891195719,"observed_timestamp":1679891195719,"severity_text":"INFO","severity_number":9,"body":"codex_hot_mix_20260611_r2 PostgreSQL realtime validation '
  }
]
[
  {
    timestamp: Long('1679876795719'),
    archive_id: '642272af-7272-48e1-a08b-5c9faab24284',
    dataset: 'default',
    message_prefix: '{"timestamp":"2023-03-27 00:26:35.719","pid":7808,"session_id":"64211afb.1e80","line_num":2,"session_start":"2023-03-27 00:26:35 EDT","txid":0,"error_severity":"LOG","mes'
  }
]

Task: Stop the package after validation

Command:

cd build/clp-package && ./sbin/stop-clp.sh

Output:

 Container clp-package-clp-runtime-run-255a7dc061a6 Creating 
 Container clp-package-clp-runtime-run-255a7dc061a6 Created 
2026-06-11T19:49:54.167 INFO [controller] Stopping all CLP containers using Docker Compose...
...
 Network clp-package-ad789ec6-ec47-4586-9794-bd397ef2c601_default Removed 
2026-06-11T19:50:16.568 INFO [controller] Stopped CLP.

Task: Rust lint/autofix and formatting

Command:

CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN) task lint:fix-rust

Output:

task: Task "toolchains-rust" is up to date
task: [lint:cargo-workspace-clippy] . "/home/junhao/.codex/worktrees/5ef4/clp/build/toolchains/rust/env"
cargo +nightly clippy --all-targets --all-features --fix --allow-dirty --allow-staged
    Checking clp-rust-utils v0.12.1-dev (/home/junhao/.codex/worktrees/5ef4/clp/components/clp-rust-utils)
    Checking log-ingestor v0.12.1-dev (/home/junhao/.codex/worktrees/5ef4/clp/components/log-ingestor)
    Checking api-server v0.12.1-dev (/home/junhao/.codex/worktrees/5ef4/clp/components/api-server)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 17.25s
task: [lint:rust-lint-configs] cp \
  /home/junhao/.codex/worktrees/5ef4/clp/tools/yscope-dev-utils/exports/lint-configs/.rustfmt.toml \
  /home/junhao/.codex/worktrees/5ef4/clp/.rustfmt.toml
task: [lint:cargo-workspace-fmt] . "/home/junhao/.codex/worktrees/5ef4/clp/build/toolchains/rust/env"
cargo +nightly fmt --all --

Task: JavaScript/TypeScript lint autofix

Command:

task lint:fix-js

Output:

task: Task "toolchains:remote:curl-/home/junhao/.codex/worktrees/5ef4/clp/build/nodejs-22.tar.gz" is up to date
task: Task "toolchains:remote:download-and-extract-tar-/home/junhao/.codex/worktrees/5ef4/clp/build/nodejs-22" is up to date
task: Task "webui-node-modules" is up to date
task: [lint:js] cd "/home/junhao/.codex/worktrees/5ef4/clp/components/webui"
PATH="/home/junhao/.codex/worktrees/5ef4/clp/build/nodejs-22/bin":$PATH npm run "lint:fix"
...
> @webui/server@0.1.0 lint:check
> eslint . --max-warnings 0 --fix

[@stylistic/eslint-plugin]: You are using deprecated rule("jsx-sort-props"), please use "sort-jsx-props" in "eslint-plugin-perfectionist" instead.

Task: Python validation

Command:

task lint:fix-py

Output:

task: Task "lint:venv" is up to date
task: [lint:py] uv run --directory "/home/junhao/.codex/worktrees/5ef4/clp/components/clp-mcp-server" ruff check --fix --exit-zero .
uv run --directory "/home/junhao/.codex/worktrees/5ef4/clp/components/clp-mcp-server" ruff format  .
uv run --directory "/home/junhao/.codex/worktrees/5ef4/clp/components/clp-mcp-server" mypy . || true
Using CPython 3.12.3 interpreter at: /usr/bin/python3
Creating virtual environment at: .venv
error: Distribution `mysql-connector-python==9.6.0 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform

hint: You're using CPython 3.12 (`cp312`), but `mysql-connector-python` (v9.6.0) only has wheels with the following Python ABI tags: `cp310`, `cp311`
task: Failed to run task "lint:fix-py": task: Failed to run task "lint:py": exit status 2

Explanation: The repo-level Python autofix task is blocked by the current mysql-connector-python wheel availability for local CPython 3.12. The generated .venv under components/clp-mcp-server was removed after this failure.

Command:

python3 -m py_compile \
  components/clp-package-utils/clp_package_utils/controller.py \
  components/clp-py-utils/clp_py_utils/clp_config.py \
  components/clp-py-utils/clp_py_utils/initialize-orchestration-db.py \
  components/job-orchestration/job_orchestration/executor/query/fs_search_task.py \
  components/job-orchestration/job_orchestration/scheduler/constants.py \
  components/job-orchestration/job_orchestration/scheduler/job_config.py \
  components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py

Output:

Task: YAML lint

Command:

task lint:yaml

Output:

task: Task "lint:venv" is up to date
task: [lint:yaml] . "/home/junhao/.codex/worktrees/5ef4/clp/build/lint-venv/bin/activate"
yamllint --strict \
  --config-file "tools/yscope-dev-utils/exports/lint-configs/.yamllint.yml" \
  ".github" \
  "components/core/.clang-format" \
  "components/core/config" \
  "components/core/tools/" \
  "components/package-template/src/etc" \
  "docs" \
  "taskfile.yaml" \
  "taskfiles" \
  "tools/deployment/package" \
  "tools/deployment/presto-clp"

Task: Helm package and lint

Command:

CLP_CPP_MAX_PARALLELISM_PER_BUILD_TASK=$(getconf _NPROCESSORS_ONLN) task helm:package

Output:

Successfully packaged chart and saved it to: /home/junhao/.codex/worktrees/5ef4/clp/build/clp-package-helm/clp-0.3.2-dev.11.tgz

Command:

task lint:helm

Output:

Linting chart "clp => (version: \"0.3.2-dev.11\", path: \"tools/deployment/package-helm\")"
Checking chart "clp => (version: \"0.3.2-dev.11\", path: \"tools/deployment/package-helm\")" for a version bump...
Error: failed linting charts: failed processing charts
Old chart version: 0.3.2-dev.11
New chart version: 0.3.2-dev.11

------------------------------------------------------------------------------------------------------------------------
 ✖︎ clp => (version: "0.3.2-dev.11", path: "tools/deployment/package-helm") > chart version not ok. Needs a version bump! 
------------------------------------------------------------------------------------------------------------------------
failed linting charts: failed processing charts
task: Failed to run task "lint:helm": exit status 1

Explanation: Helm packaging succeeds. Helm lint is blocked by the chart-testing version-bump policy because this local branch has not bumped tools/deployment/package-helm/Chart.yaml.

Task: Whitespace check

Command:

git diff --check

Output:

@coderabbitai

coderabbitai Bot commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: a6ba969a-80b8-43e5-a8d8-6b4b9154e62c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@junhaoliao
Merge branch 'main' of https://github.com/y-scope/clp into feat/realt…
c8480e4 
…ime-log-ingestion

# Conflicts:
#	components/webui/client/src/pages/SearchPage/SearchControls/Native/SearchButton/SubmitButton/index.tsx
#	components/webui/client/src/pages/SearchPage/SearchResults/SearchResultsTimeline/Native/NativeResultsTimeline.tsx
#	components/webui/common/src/schemas/search.ts
#	components/webui/server/src/routes/api/search/index.ts
@junhaoliao
chore: fix realtime logs CI outputs
7a08d9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@junhaoliao