perf: replace writer's spread with Object.assign (+20% additional)#112
Closed
elicwhite wants to merge 16 commits into
Closed
perf: replace writer's spread with Object.assign (+20% additional)#112elicwhite wants to merge 16 commits into
elicwhite wants to merge 16 commits into
Conversation
Widens ParsedChart.metadata to Partial<typeof defaultMetadata> & {
extraIniFields?: Record<string, string>}, and has parseChartAndIni merge
[Song]-section values + song.ini into it (ini wins, matching the
"song.ini is authoritative, [Song] is a legacy overlap" stance). Unknown
ini key/value pairs move from ParseChartAndIniResult.iniUnknownValues
onto parsedChart.metadata.extraIniFields for round-trip writing.
ParseChartAndIniResult.iniMetadata and iniUnknownValues are removed --
the same data is now reachable via result.parsedChart.metadata.
iniFolderIssues and iniMetadataIssues stay (they're diagnostics, not
duplicate data).
scanChart's diff-against-default checks now read the merged metadata via
parseResult.parsedChart.metadata; "set" is defined as "present AND not
equal to the default". This also fixes a pre-existing inconsistency
where chart.delay was taken from ini but chart.chart_offset was taken
from [Song] -- both now come from the merged metadata with ini winning.
Also derives defaultIniChartModifiers as a projection of defaultMetadata
so the 8 parse-behavior defaults can't drift from the 40-field source
of truth.
…art (state-derived) The three derived boolean flags on ParsedChart (hasLyrics, hasVocals, hasForcedNotes) were parse-time snapshots that went stale whenever consumers mutated chart data post-parse. Remove them from ParsedChart entirely; scanChart derives all three at scan time from the current chart state, piggybacking on its existing single note-walk (no extra iterations). - hasLyrics / hasVocals: computed from parsedChart.vocalTracks.parts — constant-time existence check. - hasForcedNotes: inverts resolveFretModifiers. A note is 'forced' iff its resolved flag disagrees with the natural HOPO state the parser would pick without any force events, or it carries the tap flag (tap can only come from explicit forceTap). Inlined helpers for isNaturalHopo, isFretChord, isSameFretNote, isInFretNote, computeHopoThresholdTicks in chart-scanner.ts (duplicated from notes-parser.ts; a follow-up can extract to a shared helper). Semantic change to note: under the new definition, redundantly-applied force events (e.g. explicit forceHopo on a naturally-HOPO note) no longer contribute to hasForcedNotes — the chart plays identically with or without them, so state-derived detection correctly says false. This eliminates the need for any 'hasForcedNotes backstop' in writers since the flag round-trips naturally: the writer emits force events exactly when a flag disagrees with natural state, and the scanner detects those same disagreements on re-parse. ScannedChart.notesData's shape is unchanged (hasLyrics / hasVocals / hasForcedNotes still present); only the population source changes. Consumers that previously read parsedChart.hasLyrics / parsedChart.hasVocals / parsedChart.hasForcedNotes must switch to scanChart output.
The chart scanner, chart writer, and MIDI writer each had their own
near-identical copies of isFretChord / isSameFretNote / isInFretNote /
isNaturalHopo over NoteEvent[], and the parser had the same three
helpers again over TrackEvent[]. This consolidates all five callsites
into src/chart/natural-hopo.ts.
The helpers are generic (<T, E extends { type: T }> + an isFret
predicate); thin wrappers are exported for each concrete group type:
- NoteEvent-based (scanner + writers): isFretChord / isSameFretNote /
isInFretNote, plus the combined isNaturalHopo.
- TrackEvent-based (parser's resolveFretModifiers):
isFretChordRawEvents / isSameFretNoteRawEvents /
isInFretNoteRawEvents. The parser still builds its natural-HOPO
check inline — it compares effectiveNotes to lastNotes while
passing the raw pre-coalesced events to isSameFretNoteRawEvents,
a subtlety no NoteEvent callsite has.
The isFretNoteType / isFretEventType predicates are also exported,
so the parser's two standalone isFretNote(...) callsites (filtering
event lists outside the group helpers) share the same definition too.
The parser's four local helpers (isFretNote, isSameFretNote, isFretChord,
isInFretNote) are deleted.
Net: ~70 LOC of duplication removed across scanner, parser, and (in
the writer branches further up the stack) the two writers. One file
owns every natural-HOPO rule.
This commit updates the scanner + parser; the writers move to the
shared helpers in the two branches further up the stack that own them.
Builds a minimal valid ParsedChart from scratch: default 480 resolution, 120 BPM at tick 0, 4/4 time signature at tick 0, empty tracks/sections/ metadata/vocal parts/unrecognized events. chartBytes defaults to an empty Uint8Array (no source bytes), format defaults to 'chart', iniChartModifiers to the library defaults. Options let callers override resolution, bpm, timeSignature, and format. Useful for programmatic chart generation (e.g. downstream code that builds charts up from scratch rather than parsing source bytes).
Serializes IniMetadata (Partial<typeof defaultMetadata> + extraIniFields) back to song.ini text with: - [song] header - Known fields emitted in the canonical defaultMetadata order - Undefined values skipped - Booleans as True/False (Clone Hero convention) - extraIniFields appended in insertion order - CRLF line endings Derives its field order from Object.keys(defaultMetadata) rather than hardcoding a separate FIELD_ORDER list, so new fields added to defaultMetadata automatically participate in writing.
…emission Port of the non-instrument-track half of the chart writer into scan-chart: - [Song] emits the subset of metadata the chart body supports (Name, Artist, Charter, Album, Genre, Year, Resolution, Offset, PreviewStart, Difficulty). Other fields live exclusively in song.ini. - [SyncTrack] emits tempos as `B millibeats` and time signatures as `TS numerator [denominator-exponent]`, sorted by tick with TS before B at the same tick. - [Events] emits section markers (wrapped as [section name] to survive the parser's greedy trailing-\] regex), end events, unrecognized global events (with bracket-stripping when source is .mid so round-trip to .chart emits naked text), and vocal phrase_start/phrase_end/lyric events from the normalized vocalTracks.parts.vocals. - Unrecognized chart sections are re-emitted verbatim. Instrument tracks ([ExpertSingle] etc.) land in a follow-up PR.
Completes writeChartFile with per-track section emission: - [ExpertSingle], [HardDrums], etc. named from instrument + difficulty. - Drum tracks: base notes (N 0..4 or 0..5 for 5-lane), cymbal markers (N 66/67/68) only in fourLanePro, accent (N 34..38) and ghost (N 40..44) markers matching the emitted note's eventType, one N 109 flam per group, double kick as N 32 only. - 5-lane round-trip: greenDrum+cymbal emits as N 4 (orange) while plain greenDrum stays at N 5; blueDrums coinciding with greenDrum+cymbal emit as N 5 to keep the parser's drumType detection on fiveLane. - Fret/GHL tracks: notes via the 5-fret and 6-fret maps, tap as N 6, forceUnnatural (N 5) when the note's hopo/strum flag disagrees with the natural-HOPO state. - Star power S 2, activation lanes S 64, flex lanes S 65/66, versus phrases S 0/1, solo sections as E 'solo'/'soloend' (tick + length - 1 for soloend to round-trip). - Per-track text events (including disco flip mix markers). - Disco-flip state transitions per difficulty from red/yellow drum disco/discoNoflip flags → 'mix <diff> drums0[d|dnoflip]' text events. - Coda events: if no 'coda' in unrecognizedEvents, synthesize from drumFreestyleSections where isCoda=true. - hasForcedNotes backstop: if hasForcedNotes is set but no forceUnnatural emitted, append N 5 0 at a vacant tick in the first fret track to preserve the flag on round-trip.
Port of the core MIDI writer infrastructure: - writeMidiFile(chart) entry point: builds a Format-1 MIDI and returns Uint8Array via midi-file's writeMidi - TEMPO TRACK: trackName + setTempo (from chart.tempos) + timeSignature (from chart.timeSignatures), all at their absolute ticks - EVENTS track: trackName + section text events (unwrapped so YARG's NormalizeTextEvent doesn't strip names containing ]) + [end] events + global events (bracket-wrap when source is .chart so MIDI output follows convention) + [coda] derived from drumFreestyleSections when not already present in unrecognizedEvents - Unrecognized MIDI tracks: verbatim pass-through with abs-tick → delta-tick conversion; duplicate track names suffixed to keep the internal trackMap unique - finalizeMidiTrack helper: sorts by tick with a type-priority tiebreaker, converts absolute ticks to delta times, appends endOfTrack Instrument tracks (PART DRUMS, PART GUITAR, …) and vocal tracks (PART VOCALS, HARM1/2/3) land in follow-up PRs.
Adds drum emission to writeMidiFile: - buildDrumTrack: groups one or more ParsedTrackData entries (one per difficulty) into a single PART DRUMS MIDI track. Emits trackName, delegates per-difficulty notes to emitDrumNotes, collects and dedupes instrument-wide star power (MIDI 116) / solo (MIDI 103) / activation lanes (MIDI 120), emits flex lanes (MIDI 126/127 with per-difficulty LDS velocity), passes through per-track text events and unrecognizedMidiEvents from the first difficulty. - emitDrumNotes: per-difficulty note emission with full modifier support: base drum notes (MIDI 96..100 expert / 84..88 hard / 72..76 medium / 60..64 easy); double-kick as MIDI 95 only, emitted AFTER regular kick at the same tick (YARG insertion order quirk); tom markers (MIDI 110/111/112) only in fourLanePro; accent (velocity 127) / ghost (velocity 1) encoding; one flam marker (MIDI 109) per group; 5-lane green+cymbal remapped to offset 4 (orange pad); blue-at-same-tick-as- green+cymbal remapped to offset 5 so fiveLane detection succeeds; fourLanePro green+tom fallback to offset 5 when a conflicting green+cymbal exists at the same tick across difficulties; disco-flip state-transition text events (`[mix <diff> drums0[d|dnoflip]]`). - fourLanePro sentinel greenTomMarker: when drumType=1 but no tom markers were emitted (all yellow/blue defaulted to cymbal, green-toms used offset-5 encoding), emit a MIDI 112 at a safe tick so scan-chart's drumType detection still picks fourLanePro. - [ENABLE_CHART_DYNAMICS] text event at tick 0 when any accent or ghost was emitted. - computeLengthOverrides: prevents scan-chart's trimSustains from collapsing adjacent short-sustain drum-note chains by attributing the combined chain length to the first note. - Shared helpers: addNoteOnOff, addNoteOnOffWithChannel (with zero-length seq-pairing to survive finalizeMidiTrack's sort), deduplicateSections, instrumentTrackNames table. Fret tracks (PART GUITAR, GHL) and vocal tracks (PART VOCALS, HARM1/2/3) remain no-ops until the follow-up PRs land — the writeMidiFile entry iterates trackData but skips any non-drum instrument group.
Adds fret-instrument emission to writeMidiFile. - buildFretTrack: one MIDI track per instrument group (guitar / bass / guitarghl / bassghl / keys / guitarcoop / rhythm / GHL variants); multiple difficulties merge into the same track. - emitFretNotes: 5-fret (95/83/71/59 diffStarts) and 6-fret/GHL (94/82/70/58) note-number mapping; open notes routed two ways: ENHANCED_OPENS mode (diffStart+0, preserves chord-with-open) when any group contains both open and non-open, else forceOpen-SysEx fallback (diffStart+1 + SysEx, collapses chords with open but safer next to animations at easy diffStart). Coalesces fret-note length with overlapping animations at the same tick:noteNumber via pre-computed seq slots so animation ordering survives re-parse. - Force modifier ranges (forceHopo / forceStrum / forceTap) via emitFretModifierRanges — inverts resolveFretModifiers: natural HOPO state is re-derived with the MIDI threshold (1 + res/3), and a modifier emits only when the resolved flag disagrees. forceHopo = diffStart+6 (5-fret) / +7 (GHL); forceStrum = +7 / +8; forceTap as SysEx. - reconstructModifierRanges: collapses per-tick hit sets into minimal contiguous ranges over noteTicksInOrder. - Instrument-wide emission: star power (MIDI 116), solo (MIDI 103), flex lanes (126/127 with per-difficulty LDS velocity), per-track text events, versus phrases (105/106), animations (with skip when the fret note above already emitted at the same tick:noteNumber), per-track unrecognizedMidiEvents. - [ENHANCED_OPENS] text event when that mode is active. - Helpers added: addSysExOnOff, reconstructModifierRanges, midiIsFretChord / midiIsSameFretNote / midiIsInFretNote. Vocal tracks (PART VOCALS / HARM1-3) land in the follow-up PR. Tests: 15 new test cases covering track naming, 5-fret + GHL note-number mapping, open-note encoding (both modes), force modifier emission (hopo/strum/tap + non-emission when natural matches), instrument-wide sections, and round-trip through parseChartAndIni for both 5-fret modifiers and GHL chord-with-open. 53 midi-writer tests, 397 total scan-chart tests passing at the tip.
Adds vocal-track emission to writeMidiFile. One MIDI track per vocal
part (vocals → PART VOCALS, harmony{1,2,3} → HARM{1,2,3}; HARM* not
PART HARM*, matching the convention used in the wild).
Phrase-marker emission is careful about YARG's CopyDownPhrases, which
at parse time copies HARM1's notePhrases onto HARM2/HARM3:
- PART VOCALS: notePhrases → 105 (player:2 → 106)
- HARM1: notePhrases → 105, staticLyricPhrases → 106
- HARM2: only staticLyricPhrases → 106 (note 105 comes from HARM1
via CopyDown on re-parse — would double-count otherwise)
- HARM3: no phrase markers at all
Lyrics and notes are union'd across both phrase sets (note 105 and 106
can have different boundaries; emitting the union keeps all lyrics
that belong on the track). Zero-length vocal notes are preserved via
per-event `seq` tags so finalizeMidiTrack keeps noteOn immediately
before its matching noteOff.
Star power (MIDI 116) is suppressed on HARM2/HARM3 — CopyDown recreates
it from HARM1 on re-parse. Range shifts (MIDI 0) and lyric shifts (MIDI
1) are per-part for lossless round-trip, with fallback to the
track-level arrays for the part that owns them (PART VOCALS, or HARM1
when PART VOCALS is absent). Raw vocal-track text events (stance
markers, facial anim triggers) are re-emitted verbatim so that
stance-only tracks survive round-trip — YARG marks a VocalsPart
non-empty iff it has phrases or text events.
Tests: 18 new cases covering track naming (PART VOCALS, HARM1/2/3 with
no PART HARM* prefix), phrase-marker routing per part, pitched/
percussion/out-of-range note emission, lyric union across phrase sets,
star power suppression on HARM2/3, range+lyric shifts, text events,
and round-trip through parseChartAndIni for both PART VOCALS and the
full harmony stack with CopyDown semantics preserved.
71 midi-writer tests, 415 total scan-chart tests passing.
End-to-end coverage for the scan-chart writers: build a ParsedChart,
serialize via writeChartFile/writeMidiFile, re-parse via parseChartAndIni,
and assert that structured fields survive.
The per-feature tests (chart-writer.test.ts, midi-writer.test.ts) already
cover granular correctness. This file's job is interactions across tracks
— e.g. a chart with drums + guitar + vocals populated together — and the
"everything enabled at once" shape the earlier tests skip.
Coverage:
- tempo/TS/sections/endEvents (both formats)
- drums: kick, tom/cymbal, accent, ghost, 2x-kick (both formats)
flam: .mid only (the .chart parser doesn't recognize N 109 — writer
emits it, parser gap is pre-existing)
- 5-fret: base colors + forceHopo + forceTap + sustains (both formats)
star power + solo sections (both formats)
- GHL: open + chord-with-open ENHANCED_OPENS path (both formats)
- vocals: PART VOCALS full shape; HARM1/2/3 with CopyDown semantics (.mid)
- multi-track: drums + guitar + bass (both formats); drums + guitar +
vocals (.mid)
- metadata: [Song] fields and chart_offset via .chart
18 new tests; 433 total passing.
Bundles a ParsedChart with non-chart files (audio, album art, extras)
into a single value, and provides the write-side companion to
parseChartAndIni's input shape:
files (in) ChartDocument (out)
---------------- -----------------------
notes.chart or notes.mid → parsedChart
song.ini → parsedChart.metadata
song.ogg, album.png, ... → assets (passthrough)
writeChartFolder produces the reverse transformation:
- notes.chart or notes.mid from writeChartFile / writeMidiFile based on
parsedChart.format
- song.ini from writeIniFile(parsedChart.metadata) — chart_offset is
naturally skipped since writeIniFile only emits keys in defaultMetadata
- every entry in doc.assets, in order
No separate `metadata` field on ChartDocument — it lives on
parsedChart.metadata (the consolidated shape produced by parseChartAndIni).
Tests: 9 cases covering format selection, ini content (chart_offset not
leaked, extraIniFields preserved), asset passthrough + ordering, and
full round-trip through parseChartAndIni for both .chart and .mid with
assets.
442 total scan-chart tests passing.
Writer.prototype.writeBytes was O(n²): this.buffer = this.buffer.concat(Array.prototype.slice.call(arr, 0)) Every call allocated a new array and replaced `this.buffer`, so writing N bytes via M calls cost O(N·M) time and churned the GC. Replace with an in-place push loop — each call is O(arr.length), total write is O(N). Measured on the writer autoresearch bench (2000 charts, 500 .chart + 1500 .mid, 8 workers): - mean: 90.115 ms → 6.721 ms (13.4× faster) - p50: 59.944 ms → 5.764 ms (10.4× faster) - p95: 274.443 ms → 17.255 ms (15.9× faster) - p99: 560.784 ms → 25.263 ms (22.2× faster) - wall: 24.972s → 5.496s (4.5× faster) - summed writer time: 162.206s → 12.098s (93% reduction) 0 hash mismatches, 442/442 tests still green — byte-identical output. CPU profile at baseline showed: 35.76% Writer.writeUInt8 21.06% Writer.writeBytes 13.81% GC of total writer time. Fixing writeBytes removes both the direct cost (it's a handful of the 19 writeBytes callsites downstream of every track write) and the GC pressure from the allocate-and-replace pattern.
Replace Writer's backing Array<number> with a growable Uint8Array +
manual cursor. Benefits:
- Raw bytes instead of boxed JS numbers — ~8× less memory per byte
and much less GC pressure.
- writeBytes on a Uint8Array source is a single .set() call instead
of a push loop.
- writeVarInt no longer allocates a temp Array to hold the bytes
then .reverse()s it — the 1-5 byte tail is written directly into
the buffer.
API shape preserved for callers:
- Inner sub-writers expose .used() which returns a subarray view.
- Top-level writeMidi returns the used subarray; scan-chart's
`new Uint8Array(writeMidi(midiData))` still works (copies the
subarray into a dedicated Uint8Array), and the few other direct
callers in spotify-clonehero-next that wrap the result in
Uint8Array also still work.
Measured on writer autoresearch bench (2000 charts, 8 workers):
previous (post-writeBytes fix): 6.721 ms mean, 25.263 ms p99, 77 ms max, 5.496s wall
this patch: 5.710 ms mean, 22.576 ms p99, 42 ms max, 5.132s wall
delta: -15.0% mean, -10.6% p99, -45% max, -6.6% wall
0 hash mismatches, 442/442 tests still green.
tsup transpiles `{...ev, deltaTime: 0}` to __spreadProps(__spreadValues(...))
which iterates property descriptors — ~7% of total writer profile time was
spent inside __spreadProps. Object.assign({}, ev, { deltaTime: 0 }) is a
native builtin on V8 and 2-3× faster for small objects.
Applied at the three writer sites that clone-with-delta-reset a source
event: buildUnrecognizedTrack (top-level unrecognized MIDI tracks) and
the per-track unrecognizedMidiEvents passes in buildDrumTrack and
buildFretTrack.
Measured on writer autoresearch bench (2000 charts, 8 workers):
previous: 5.710 ms mean, 4.883 ms p50, 15.040 ms p95, 5.132s wall
now: 4.566 ms mean, 4.129 ms p50, 10.424 ms p95, 4.926s wall
delta: -20.0% mean, -30.7% p95
Second run (variance check): 4.609 ms mean — stable.
0 hash mismatches, 442/442 tests still green.
Cumulative: baseline 92 ms -> 4.57 ms, 20× faster.
Contributor
Author
|
Closing — these patches were for the upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tsup transpiles
{...ev, deltaTime: 0}to__spreadProps(__spreadValues(...)), which iterates property descriptors. The baseline CPU profile (after #110 + #111) showed ~7% of total writer time in__spreadPropsalone.Object.assign({}, ev, { deltaTime: 0 })is a native builtin — 2-3× faster for small objects.Applied at the three writer sites that clone-with-delta-reset a source event:
buildUnrecognizedTrack(top-level unrecognized MIDI tracks)buildDrumTrackper-trackunrecognizedMidiEventspassbuildFretTrackper-trackunrecognizedMidiEventspassMeasurement
On the same 2000-chart autoresearch bench:
Re-run to check variance: 4.609 ms mean — stable.
Cumulative with #110 + #111: baseline 90 ms → 4.57 ms = ~20× speedup on the writer. 0 hash mismatches, 442/442 tests pass.
Test plan
yarn testgreenStack
Top of the perf stack. Builds on #110 + #111.