Skip to content

test(integration): harden pre-existing shard-5/6 CI flakes #1504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dawsontoth wants to merge 4 commits into from
Closed

test(integration): harden pre-existing shard-5/6 CI flakes #1504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
abc95f8
test(integration): harden shard-5/6 flakes (early-hints restart + job…
dawsontoth Jun 26, 2026
28574cb
test(integration): log full error in early-hints deploy guard
dawsontoth Jun 26, 2026
49179af
perf(test): cut round-trips in HNSW vector-index integration tests
dawsontoth Jun 26, 2026
551c8f9
perf(test): deploy HNSW vector tables once, cut per-test worker restarts
dawsontoth Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions integrationTests/apiTests/terminology.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,15 @@ import { awaitJobCompleted } from './utils/operations.mjs';
// Resolve the CSV fixture path relative to this file so Harper can read it.
const SUPPLIERS_CSV = path.join(path.dirname(fileURLToPath(import.meta.url)), 'data/Suppliers.csv');

// On Bun shard 2, embed-directive tear-down (HNSW flush) starves the job processor,
// so csv_data_load can sit IN_PROGRESS past the default 30s. Same pattern as northwind.
// Job-processor starvation can leave these async jobs IN_PROGRESS well past a tight timeout.
// On Bun shard 2 the embed-directive tear-down (HNSW flush) starves it; in CI the Node shard
// 5/6 run showed the same — the slow drop/teardown ahead of these jobs pushed the csv / delete /
// export jobs past the old 30s limit and failed ~10 of them in a single cascade. Give every
// runtime generous headroom so a slow-but-completing job processor doesn't fail the suite (the
// jobs do complete; they were just starved). This does not mask the separate drop_database
// failure ahead of them — that assertion is left intact.
const isBunRuntime = process.env.HARPER_RUNTIME === 'bun';
const JOB_TIMEOUT_SECONDS = isBunRuntime ? 120 : 30;
const JOB_TIMEOUT_SECONDS = isBunRuntime ? 120 : 90;

suite('Terminology aliases (database / primary_key)', (ctx) => {
let client;
Expand Down
30 changes: 22 additions & 8 deletions integrationTests/components/early-hints.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,28 @@ suite('Component: early-hints', (ctx: ContextWithHarper) => {
before(async () => {
await startHarper(ctx);

const deployBody = await sendOperation(ctx.harper, {
operation: 'deploy_component',
project: 'early-hints',
package: join(__dirname, '../fixtures/template-early-hints-2.0.0.tgz'),
restart: true,
});
strictEqual(deployBody.message, 'Successfully deployed: early-hints, restarting Harper');
ok(typeof deployBody.deployment_id === 'string', `expected deployment_id, got ${deployBody.deployment_id}`);
// `restart: true` makes Harper restart right after accepting the deploy, which on a slow
// runner can delay or drop the HTTP response to this call — surfacing as a fetch
// `HeadersTimeoutError`. That is not a deploy failure: the readiness polling below (which
// waits for the /hints endpoint + seed data) is the authoritative success check. So
// tolerate a missing/failed response here instead of failing the whole suite in `before`.
let deployBody: any;
try {
deployBody = await sendOperation(ctx.harper, {
operation: 'deploy_component',
project: 'early-hints',
package: join(__dirname, '../fixtures/template-early-hints-2.0.0.tgz'),
restart: true,
});
} catch (e: any) {
// Log the full error (not just the message) so a genuine deploy failure — bad
// package, invalid operation — is debuggable when the readiness poll below times out.
console.log('[early-hints] deploy response not received; relying on readiness poll', e);
}
Comment thread
dawsontoth marked this conversation as resolved.
if (deployBody) {
strictEqual(deployBody.message, 'Successfully deployed: early-hints, restarting Harper');
ok(typeof deployBody.deployment_id === 'string', `expected deployment_id, got ${deployBody.deployment_id}`);
}

// poll until /hints endpoint is registered and seed data is loaded
const seedDeadline = Date.now() + 60_000;
Expand Down
191 changes: 73 additions & 118 deletions integrationTests/server/vector-index-integrity.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,9 +226,49 @@ async function waitFor(predicate: () => Promise<boolean>, timeoutMs = 60_000, in
suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper) => {
let client: any;

// All four tables are deployed by a SINGLE component with ONE worker restart in `before`.
// Previously every test deployed its own component and restarted the HTTP workers (~5 restarts
// across the suite); on slow runners that restart is the dominant per-test cost (a test doing
// 3 inserts still took ~265s on Windows — almost entirely restart/setup). Consolidating to one
// shared restart here — plus the single restart `reindex` needs for its mid-test index-add — is
// the bulk of this suite's wall-time. The tables use distinct names/databases so the tests stay
// isolated despite sharing one component; `reindex` runs last and is the only test that mutates
// the schema afterward, so re-asserting the others on its redeploy is a harmless no-op.
const PROJECT = 'vecintegsuite';

@claude claude Bot Jun 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared component/schema state creates an implicit test-ordering dependency: reindex must run last because it redeploys SCHEMA_REINDEX_INDEXED, which adds the HNSW index to ReindexTable. node:test runs suites in declaration order in practice, but that's a convention, not a guarantee enforced by the framework. If a future contributor reorders the test(...) calls — or node:test ever runs them in parallel — ReindexTable could have an index unexpectedly present for earlier tests.

At minimum, a comment near the test declarations noting "order matters — reindex must run last" (or a runtime guard that asserting the reindex test runs after the others) would make the dependency explicit. This is low-risk given the current codebase, but worth a brief note in code.

// ReindexTable starts WITHOUT an index — the reindex test adds it mid-test to exercise backfill.
const SCHEMA_INITIAL = [
makeSchemaWithIndex('DelEPTable', 'vecinteg1'),
makeSchemaWithIndex('ChurnTable', 'vecinteg2'),
makeSchemaWithIndex('ThreshTable', 'vecinteg3'),
makeSchemaWithoutIndex('ReindexTable', 'vecinteg4'),
].join('\n');
// Same schema with ReindexTable now indexed — redeployed by the reindex test to trigger backfill.
const SCHEMA_REINDEX_INDEXED = [
makeSchemaWithIndex('DelEPTable', 'vecinteg1'),
makeSchemaWithIndex('ChurnTable', 'vecinteg2'),
makeSchemaWithIndex('ThreshTable', 'vecinteg3'),
makeSchemaWithIndex('ReindexTable', 'vecinteg4'),
].join('\n');

before(async () => {
await startHarper(ctx, { config: {}, env: {} });
client = createApiClient(ctx.harper);
await client
.req()
.send({ operation: 'add_component', project: PROJECT })
.expect((r: any) => {
ok(
JSON.stringify(r.body).includes('Successfully added project') ||
JSON.stringify(r.body).includes('Project already exists'),
r.text
);
});
await client
.req()
.send({ operation: 'set_component_file', project: PROJECT, file: 'schema.graphql', payload: SCHEMA_INITIAL })
.expect(200);
// One restart loads all four tables; probing one ready endpoint confirms the workers are back.
await restartHttpWorkers(client, '/DelEPTable/');
});

after(async () => {
Expand All @@ -241,39 +281,13 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
// getRange and skips the node being deleted.
// Pre-fix: getRange ran outside the write-set → re-elected the deleted node
// as EP → dangling entry point → subsequent searches returned [].
const DB = 'vecinteg1';
const TABLE = 'DelEPTable';
const PROJECT = 'vecintegsuite1';
const DIMS = 8;
const N = 50;
const SURVIVORS = 10;
const deleteCount = N - SURVIVORS;

await client
.req()
.send({ operation: 'add_component', project: PROJECT })
.expect((r: any) => {
ok(
JSON.stringify(r.body).includes('Successfully added project') ||
JSON.stringify(r.body).includes('Project already exists'),
r.text
);
});
await client
.req()
.send({
operation: 'set_component_file',
project: PROJECT,
file: 'schema.graphql',
payload: makeSchemaWithIndex(TABLE, DB),
})
.expect(200);

await restartHttpWorkers(client, `/${TABLE}/`);

const httpURL = ctx.harper.httpURL;
const headers = client.headers;
const path = `/${TABLE}/`;
const path = '/DelEPTable/';

// Insert N records — the first-inserted node becomes the initial entry point.
for (let i = 0; i < N; i++) {
Expand Down Expand Up @@ -312,38 +326,12 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
// where the old connection existed, not the full 0..l sweep (correct for DELETE).
// The broader sweep destroyed reverse edges that addConnection had just re-added,
// accumulating asymmetry with every re-embed round.
const DB = 'vecinteg2';
const TABLE = 'ChurnTable';
const PROJECT = 'vecintegsuite2';
const DIMS = 8;
const N = 30;
const ROUNDS = 5;

await client
.req()
.send({ operation: 'add_component', project: PROJECT })
.expect((r: any) => {
ok(
JSON.stringify(r.body).includes('Successfully added project') ||
JSON.stringify(r.body).includes('Project already exists'),
r.text
);
});
await client
.req()
.send({
operation: 'set_component_file',
project: PROJECT,
file: 'schema.graphql',
payload: makeSchemaWithIndex(TABLE, DB),
})
.expect(200);

await restartHttpWorkers(client, `/${TABLE}/`);

const httpURL = ctx.harper.httpURL;
const headers = client.headers;
const path = `/${TABLE}/`;
const path = '/ChurnTable/';

for (let i = 0; i < N; i++) {
await insertRecord(httpURL, headers, path, { id: `churn-${i}`, embedding: seedVector(i, DIMS) });
Expand All @@ -360,12 +348,17 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
}

// Each record's final vector is seedVector(i*100 + ROUNDS, dims).
// Searching for that exact vector must return it in the top-5.
// Searching for that exact vector must return it in the top-5. These are
// independent read-only searches, so run them concurrently rather than
// serializing N round-trips.
const churnResults = await Promise.all(

@claude claude Bot Jun 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Launching 30 concurrent QUERY requests via Promise.all on a CI runner that's already strained enough to hit 30s job timeouts may cause a wave of simultaneous HTTP connections to the same Harper instance. On an overloaded runner this could trigger per-connection timeouts or starve other in-flight requests, producing flaky misses that look like data-integrity failures rather than infra noise. A small concurrency cap (e.g. 5 in-flight at a time) preserves most of the speedup while bounding the burst:

Suggested change
const churnResults = await Promise.all(
const CONCURRENCY = 5;
const churnResults: any[][] = [];
for (let start = 0; start < N; start += CONCURRENCY) {
const batch = await Promise.all(
Array.from({ length: Math.min(CONCURRENCY, N - start) }, (_, j) =>
vectorSearch(httpURL, headers, path, seedVector((start + j) * 100 + ROUNDS, DIMS), { limit: 5 })
)
);
churnResults.push(...batch);
}

Array.from({ length: N }, (_, i) =>
vectorSearch(httpURL, headers, path, seedVector(i * 100 + ROUNDS, DIMS), { limit: 5 })
)
);
let misses = 0;
for (let i = 0; i < N; i++) {
const finalVec = seedVector(i * 100 + ROUNDS, DIMS);
const results = await vectorSearch(httpURL, headers, path, finalVec, { limit: 5 });
if (!results.some((r: any) => r.id === `churn-${i}`)) misses++;
if (!churnResults[i].some((r: any) => r.id === `churn-${i}`)) misses++;
}
const allowed = Math.ceil(N * 0.1);
ok(misses <= allowed, `expected ≤${allowed} misses after ${ROUNDS} churn rounds; got ${misses}/${N}`);
Expand All @@ -375,35 +368,9 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
test('threshold queries: le includes exact boundary; le(~0) returns exact-match only', async () => {
// Guards fix #6b: le comparator now uses <= (was <).
// Pre-fix: records at exactly the threshold distance were excluded by le.
const DB = 'vecinteg3';
const TABLE = 'ThreshTable';
const PROJECT = 'vecintegsuite3';

await client
.req()
.send({ operation: 'add_component', project: PROJECT })
.expect((r: any) => {
ok(
JSON.stringify(r.body).includes('Successfully added project') ||
JSON.stringify(r.body).includes('Project already exists'),
r.text
);
});
await client
.req()
.send({
operation: 'set_component_file',
project: PROJECT,
file: 'schema.graphql',
payload: makeSchemaWithIndex(TABLE, DB),
})
.expect(200);

await restartHttpWorkers(client, `/${TABLE}/`);

const httpURL = ctx.harper.httpURL;
const headers = client.headers;
const path = `/${TABLE}/`;
const path = '/ThreshTable/';

// 2-D vectors at three well-separated cosine distances from [1,0]:
// [1,0] → distance 0 (exact; representable in float32, so exactly 0)
Expand Down Expand Up @@ -464,42 +431,26 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
// on structural index change so runIndexing starts clean).
const DB = 'vecinteg4';
const TABLE = 'ReindexTable';
const PROJECT = 'vecintegsuite4';
const DIMS = 8;
const N = 40;
const httpURL = ctx.harper.httpURL;
const headers = client.headers;
const path = '/ReindexTable/';

// Step 1: Deploy schema WITHOUT the HNSW index.
await client
.req()
.send({ operation: 'add_component', project: PROJECT })
.expect((r: any) => {
ok(
JSON.stringify(r.body).includes('Successfully added project') ||
JSON.stringify(r.body).includes('Project already exists'),
r.text
);
});
// Step 1: ReindexTable was deployed WITHOUT an HNSW index by the shared `before` setup.
// Step 2: Insert N records (plain [Float], no HNSW index yet). Bulk-insert in a
// single request — there is no index during insert, so per-record ordering is
// irrelevant here; the backfill (Step 3) builds the index from the stored rows.
await client
.req()
.send({
operation: 'set_component_file',
project: PROJECT,
file: 'schema.graphql',
payload: makeSchemaWithoutIndex(TABLE, DB),
operation: 'insert',
database: DB,
table: TABLE,
records: Array.from({ length: N }, (_, i) => ({ id: `reindex-${i}`, embedding: seedVector(i, DIMS) })),
})
.expect(200);

await restartHttpWorkers(client, `/${TABLE}/`);

const httpURL = ctx.harper.httpURL;
const headers = client.headers;
const path = `/${TABLE}/`;

// Step 2: Insert N records (plain [Float], no HNSW index yet).
for (let i = 0; i < N; i++) {
await insertRecord(httpURL, headers, path, { id: `reindex-${i}`, embedding: seedVector(i, DIMS) });
}

// Sanity-check: records exist.
const hashCheck = await client
.req()
Expand All @@ -513,14 +464,16 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
.expect(200);
ok(Array.isArray(hashCheck.body) && hashCheck.body.length === 1, 'reindex-0 must exist before reindex');

// Step 3: Alter schema to ADD the HNSW index → triggers runIndexing backfill.
// Step 3: Alter schema to ADD the HNSW index on ReindexTable → triggers runIndexing
// backfill. Redeploy the full schema (the other three tables are unchanged, so this is
// a no-op re-assert for them); reindex runs last, so restarting them here is harmless.
await client
.req()
.send({
operation: 'set_component_file',
project: PROJECT,
file: 'schema.graphql',
payload: makeSchemaWithIndex(TABLE, DB),
payload: SCHEMA_REINDEX_INDEXED,
})
.expect(200);

Expand All @@ -537,12 +490,14 @@ suite('HNSW vector-index data-integrity (integration)', (ctx: ContextWithHarper)
}
}, 60_000);

// Step 5: All N pre-existing records must be reachable.
// Step 5: All N pre-existing records must be reachable. Independent read-only
// searches — run them concurrently rather than serializing N round-trips.
const reindexResults = await Promise.all(
Array.from({ length: N }, (_, i) => vectorSearch(httpURL, headers, path, seedVector(i, DIMS), { limit: 5 }))
);
let misses = 0;
for (let i = 0; i < N; i++) {
const vec = seedVector(i, DIMS);
const results = await vectorSearch(httpURL, headers, path, vec, { limit: 5 });
if (!results.some((r: any) => r.id === `reindex-${i}`)) misses++;
if (!reindexResults[i].some((r: any) => r.id === `reindex-${i}`)) misses++;
}
const allowed = Math.ceil(N * 0.1);
ok(misses <= allowed, `expected ≤${allowed} misses after backfill; got ${misses}/${N}`);
Expand Down
Loading