Skip to content

Add CommandApiPerformanceBenchmarkTest for save/load comparison vs 0.12.8 baseline #107

Description

@jtnelson

Summary

Add a local-only benchmark that times save and load operations on a large object graph against both the 0.12.8 baseline (no Command API support) and the latest (with supportsBulkCommands) Concourse server, so the round-trip wins from #103, #104, and #105 can be empirically measured. Modeled on the existing PreSelectPerformanceBenchmarkTest.java. @Ignored so it does not run on CI — meant to be unignored locally for comparison runs.

Scope

New file: src/test/java/com/cinchapi/runway/CommandApiPerformanceBenchmarkTest.java

Shape

Use the existing benchmark as a template. Key elements:

@Versions({ "0.12.8", "latest" })
@Ignore
public class CommandApiPerformanceBenchmarkTest extends CrossVersionTest {

    @Override
    public void beforeEachTest() {
        // Build a large graph: e.g., 5000 Users, each linking one Company
        // and one Address (mirror the existing benchmark's shape, perhaps
        // larger to amplify per-record round-trip differences).
    }

    @Test
    public void testBatchSavePerformance() {
        // Time a single db.save(record1, ..., recordN) call where N is
        // large. After #104 lands, the 1.0.0 row should drop sharply
        // because per-field writes collapse into one submit per save.
    }

    @Test
    public void testGraphSavePerformance() {
        // Time saving a deeply-linked graph (root + many children +
        // grandchildren). Validates that recursive
        // saveModifiedReferenceWithinTransaction also flows through the
        // CommandGroup once #104 lands.
    }

    @Test
    public void testBulkLoadPerformance() {
        // Time db.load(User.class) on the large graph. After #103 lands,
        // the 1.0.0 row should drop because the multi-selection dispatch
        // collapses to one submit instead of thread-pool-fanned-out
        // per-class queries.
    }

    @Test
    public void testMultiSelectionPerformance() {
        // Time a select(s1, s2, ..., sN) call with selections spanning
        // multiple classes (which today exercises both combinable and
        // isolated branches). After #103, all N collapse to one submit.
    }

    // Inner Record classes (User, Company, Address, etc.) — same as the
    // existing benchmark.
}

For each @Test, use Benchmark from com.cinchapi.common.profile.Benchmark (the existing pattern) and record(...) to log:

Why 0.12.8 specifically

0.12.8 is the most recent server release before the 1.0.0 RC, so the comparison isolates the Command API impact rather than mixing in unrelated improvements between e.g. 0.11 and 1.0.

Why @ignore'd

The benchmark needs both versions of the server available locally (CI runners may not have both, and the test runtime is significant). The benchmark is for local validation, not CI gating — mirrors the existing PreSelectPerformanceBenchmarkTest's @Ignore.

Dependencies

None — the test is valid against the current code (it'll show the 0.12.8 vs latest difference is small / zero before the Command API work lands). Recommended workflow:

  1. Land this ticket.
  2. Run locally to establish baseline numbers for 0.12.8 and 1.0.0-rc against the current Runway code.
  3. As Collapse multi-selection reads into one prepare()/submit() round trip #103, Bulk-submit field writes during Runway.save() #104, Fold stage / reads / writes / commit into a 2-RT save #105 land, re-run to capture the round-trip reduction on the 1.0.0-rc row. The 0.12.8 row should stay flat since the bulk path is gated on supportsBulkCommands.

The before/after deltas on the 1.0.0-rc row are the success metric for the umbrella.

Acceptance

Parent

Part of #100 — Adopt Concourse 1.0.0 Command API in Runway.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions