writeChartFile: re-quote extraChartSongFields per .chart spec types#124
Open
elicwhite wants to merge 15 commits into
Open
writeChartFile: re-quote extraChartSongFields per .chart spec types#124elicwhite wants to merge 15 commits into
elicwhite wants to merge 15 commits into
Conversation
Builds a minimal valid ParsedChart from scratch: default 480 resolution, 120 BPM at tick 0, 4/4 time signature at tick 0, empty tracks/sections/ metadata/vocal parts/unrecognized events. chartBytes defaults to an empty Uint8Array (no source bytes), format defaults to 'chart', iniChartModifiers to the library defaults. Options let callers override resolution, bpm, timeSignature, and format. Useful for programmatic chart generation (e.g. downstream code that builds charts up from scratch rather than parsing source bytes).
Serializes IniMetadata (Partial<typeof defaultMetadata> + extraIniFields) back to song.ini text with: - [song] header - Known fields emitted in the canonical defaultMetadata order - Undefined values skipped - Booleans as True/False (Clone Hero convention) - extraIniFields appended in insertion order - CRLF line endings Derives its field order from Object.keys(defaultMetadata) rather than hardcoding a separate FIELD_ORDER list, so new fields added to defaultMetadata automatically participate in writing. Tests exercise writeIniFile only via round-trip through parseChartAndIni, per reviewer feedback: build a metadata object, write it, re-parse, and assert on parsedChart.metadata. No assertions about the serialized ini text itself (CRLF, field order, True/False formatting, quoting) — those are implementation details.
…emission Port of the non-instrument-track half of the chart writer into scan-chart: - [Song] emits the subset of metadata the chart body supports (Name, Artist, Charter, Album, Genre, Year, Resolution, Offset, PreviewStart, Difficulty). Other fields live exclusively in song.ini. - [SyncTrack] emits tempos as `B millibeats` and time signatures as `TS numerator [denominator-exponent]`, sorted by tick with TS before B at the same tick. - [Events] emits section markers (wrapped as [section name] to survive the parser's greedy trailing-\] regex), end events, unrecognized global events (with bracket-stripping when source is .mid so round-trip to .chart emits naked text), and vocal phrase_start/phrase_end/lyric events from the normalized vocalTracks.parts.vocals. - Unrecognized chart sections are re-emitted verbatim. Instrument tracks ([ExpertSingle] etc.) land in a follow-up PR.
Completes writeChartFile with per-track section emission: - [ExpertSingle], [HardDrums], etc. named from instrument + difficulty. - Drum tracks: base notes (N 0..4 or 0..5 for 5-lane), cymbal markers (N 66/67/68) only in fourLanePro, accent (N 34..38) and ghost (N 40..44) markers matching the emitted note's eventType, one N 109 flam per group, double kick as N 32 only. - 5-lane round-trip: greenDrum+cymbal emits as N 4 (orange) while plain greenDrum stays at N 5; blueDrums coinciding with greenDrum+cymbal emit as N 5 to keep the parser's drumType detection on fiveLane. - Fret/GHL tracks: notes via the 5-fret and 6-fret maps, tap as N 6, forceUnnatural (N 5) when the note's hopo/strum flag disagrees with the natural-HOPO state. - Star power S 2, activation lanes S 64, flex lanes S 65/66, versus phrases S 0/1, solo sections as E 'solo'/'soloend' (tick + length - 1 for soloend to round-trip). - Per-track text events (including disco flip mix markers). - Disco-flip state transitions per difficulty from red/yellow drum disco/discoNoflip flags → 'mix <diff> drums0[d|dnoflip]' text events. - Coda events: if no 'coda' in unrecognizedEvents, synthesize from drumFreestyleSections where isCoda=true. - hasForcedNotes backstop: if hasForcedNotes is set but no forceUnnatural emitted, append N 5 0 at a vacant tick in the first fret track to preserve the flag on round-trip.
Port of the core MIDI writer infrastructure: - writeMidiFile(chart) entry point: builds a Format-1 MIDI and returns Uint8Array via midi-file's writeMidi - TEMPO TRACK: trackName + setTempo (from chart.tempos) + timeSignature (from chart.timeSignatures), all at their absolute ticks - EVENTS track: trackName + section text events (unwrapped so YARG's NormalizeTextEvent doesn't strip names containing ]) + [end] events + global events (bracket-wrap when source is .chart so MIDI output follows convention) + [coda] derived from drumFreestyleSections when not already present in unrecognizedEvents - Unrecognized MIDI tracks: verbatim pass-through with abs-tick → delta-tick conversion; duplicate track names suffixed to keep the internal trackMap unique - finalizeMidiTrack helper: sorts by tick with a type-priority tiebreaker, converts absolute ticks to delta times, appends endOfTrack Instrument tracks (PART DRUMS, PART GUITAR, …) and vocal tracks (PART VOCALS, HARM1/2/3) land in follow-up PRs.
Adds drum emission to writeMidiFile: - buildDrumTrack: groups one or more ParsedTrackData entries (one per difficulty) into a single PART DRUMS MIDI track. Emits trackName, delegates per-difficulty notes to emitDrumNotes, collects and dedupes instrument-wide star power (MIDI 116) / solo (MIDI 103) / activation lanes (MIDI 120), emits flex lanes (MIDI 126/127 with per-difficulty LDS velocity), passes through per-track text events and unrecognizedMidiEvents from the first difficulty. - emitDrumNotes: per-difficulty note emission with full modifier support: base drum notes (MIDI 96..100 expert / 84..88 hard / 72..76 medium / 60..64 easy); double-kick as MIDI 95 only, emitted AFTER regular kick at the same tick (YARG insertion order quirk); tom markers (MIDI 110/111/112) only in fourLanePro; accent (velocity 127) / ghost (velocity 1) encoding; one flam marker (MIDI 109) per group; 5-lane green+cymbal remapped to offset 4 (orange pad); blue-at-same-tick-as- green+cymbal remapped to offset 5 so fiveLane detection succeeds; fourLanePro green+tom fallback to offset 5 when a conflicting green+cymbal exists at the same tick across difficulties; disco-flip state-transition text events (`[mix <diff> drums0[d|dnoflip]]`). - fourLanePro sentinel greenTomMarker: when drumType=1 but no tom markers were emitted (all yellow/blue defaulted to cymbal, green-toms used offset-5 encoding), emit a MIDI 112 at a safe tick so scan-chart's drumType detection still picks fourLanePro. - [ENABLE_CHART_DYNAMICS] text event at tick 0 when any accent or ghost was emitted. - computeLengthOverrides: prevents scan-chart's trimSustains from collapsing adjacent short-sustain drum-note chains by attributing the combined chain length to the first note. - Shared helpers: addNoteOnOff, addNoteOnOffWithChannel (with zero-length seq-pairing to survive finalizeMidiTrack's sort), deduplicateSections, instrumentTrackNames table. Fret tracks (PART GUITAR, GHL) and vocal tracks (PART VOCALS, HARM1/2/3) remain no-ops until the follow-up PRs land — the writeMidiFile entry iterates trackData but skips any non-drum instrument group.
Adds fret-instrument emission to writeMidiFile. - buildFretTrack: one MIDI track per instrument group (guitar / bass / guitarghl / bassghl / keys / guitarcoop / rhythm / GHL variants); multiple difficulties merge into the same track. - emitFretNotes: 5-fret (95/83/71/59 diffStarts) and 6-fret/GHL (94/82/70/58) note-number mapping; open notes routed two ways: ENHANCED_OPENS mode (diffStart+0, preserves chord-with-open) when any group contains both open and non-open, else forceOpen-SysEx fallback (diffStart+1 + SysEx, collapses chords with open but safer next to animations at easy diffStart). Coalesces fret-note length with overlapping animations at the same tick:noteNumber via pre-computed seq slots so animation ordering survives re-parse. - Force modifier ranges (forceHopo / forceStrum / forceTap) via emitFretModifierRanges — inverts resolveFretModifiers: natural HOPO state is re-derived with the MIDI threshold (1 + res/3), and a modifier emits only when the resolved flag disagrees. forceHopo = diffStart+6 (5-fret) / +7 (GHL); forceStrum = +7 / +8; forceTap as SysEx. - reconstructModifierRanges: collapses per-tick hit sets into minimal contiguous ranges over noteTicksInOrder. - Instrument-wide emission: star power (MIDI 116), solo (MIDI 103), flex lanes (126/127 with per-difficulty LDS velocity), per-track text events, versus phrases (105/106), animations (with skip when the fret note above already emitted at the same tick:noteNumber), per-track unrecognizedMidiEvents. - [ENHANCED_OPENS] text event when that mode is active. - Helpers added: addSysExOnOff, reconstructModifierRanges, midiIsFretChord / midiIsSameFretNote / midiIsInFretNote. Vocal tracks (PART VOCALS / HARM1-3) land in the follow-up PR. Tests: 15 new test cases covering track naming, 5-fret + GHL note-number mapping, open-note encoding (both modes), force modifier emission (hopo/strum/tap + non-emission when natural matches), instrument-wide sections, and round-trip through parseChartAndIni for both 5-fret modifiers and GHL chord-with-open. 53 midi-writer tests, 397 total scan-chart tests passing at the tip.
Adds vocal-track emission to writeMidiFile. One MIDI track per vocal
part (vocals → PART VOCALS, harmony{1,2,3} → HARM{1,2,3}; HARM* not
PART HARM*, matching the convention used in the wild).
Phrase-marker emission is careful about YARG's CopyDownPhrases, which
at parse time copies HARM1's notePhrases onto HARM2/HARM3:
- PART VOCALS: notePhrases → 105 (player:2 → 106)
- HARM1: notePhrases → 105, staticLyricPhrases → 106
- HARM2: only staticLyricPhrases → 106 (note 105 comes from HARM1
via CopyDown on re-parse — would double-count otherwise)
- HARM3: no phrase markers at all
Lyrics and notes are union'd across both phrase sets (note 105 and 106
can have different boundaries; emitting the union keeps all lyrics
that belong on the track). Zero-length vocal notes are preserved via
per-event `seq` tags so finalizeMidiTrack keeps noteOn immediately
before its matching noteOff.
Star power (MIDI 116) is suppressed on HARM2/HARM3 — CopyDown recreates
it from HARM1 on re-parse. Range shifts (MIDI 0) and lyric shifts (MIDI
1) are per-part for lossless round-trip, with fallback to the
track-level arrays for the part that owns them (PART VOCALS, or HARM1
when PART VOCALS is absent). Raw vocal-track text events (stance
markers, facial anim triggers) are re-emitted verbatim so that
stance-only tracks survive round-trip — YARG marks a VocalsPart
non-empty iff it has phrases or text events.
Tests: 18 new cases covering track naming (PART VOCALS, HARM1/2/3 with
no PART HARM* prefix), phrase-marker routing per part, pitched/
percussion/out-of-range note emission, lyric union across phrase sets,
star power suppression on HARM2/3, range+lyric shifts, text events,
and round-trip through parseChartAndIni for both PART VOCALS and the
full harmony stack with CopyDown semantics preserved.
71 midi-writer tests, 415 total scan-chart tests passing.
End-to-end coverage for the scan-chart writers: build a ParsedChart,
serialize via writeChartFile/writeMidiFile, re-parse via parseChartAndIni,
and assert that structured fields survive.
The per-feature tests (chart-writer.test.ts, midi-writer.test.ts) already
cover granular correctness. This file's job is interactions across tracks
— e.g. a chart with drums + guitar + vocals populated together — and the
"everything enabled at once" shape the earlier tests skip.
Coverage:
- tempo/TS/sections/endEvents (both formats)
- drums: kick, tom/cymbal, accent, ghost, 2x-kick (both formats)
flam: .mid only (the .chart parser doesn't recognize N 109 — writer
emits it, parser gap is pre-existing)
- 5-fret: base colors + forceHopo + forceTap + sustains (both formats)
star power + solo sections (both formats)
- GHL: open + chord-with-open ENHANCED_OPENS path (both formats)
- vocals: PART VOCALS full shape; HARM1/2/3 with CopyDown semantics (.mid)
- multi-track: drums + guitar + bass (both formats); drums + guitar +
vocals (.mid)
- metadata: [Song] fields and chart_offset via .chart
18 new tests; 433 total passing.
Bundles a ParsedChart with non-chart files (audio, album art, extras)
into a single value, and provides the write-side companion to
parseChartAndIni's input shape:
files (in) ChartDocument (out)
---------------- -----------------------
notes.chart or notes.mid → parsedChart
song.ini → parsedChart.metadata
song.ogg, album.png, ... → assets (passthrough)
writeChartFolder produces the reverse transformation:
- notes.chart or notes.mid from writeChartFile / writeMidiFile based on
parsedChart.format
- song.ini from writeIniFile(parsedChart.metadata) — chart_offset is
naturally skipped since writeIniFile only emits keys in defaultMetadata
- every entry in doc.assets, in order
No separate `metadata` field on ChartDocument — it lives on
parsedChart.metadata (the consolidated shape produced by parseChartAndIni).
Tests: 9 cases covering format selection, ini content (chart_offset not
leaked, extraIniFields preserved), asset passthrough + ordering, and
full round-trip through parseChartAndIni for both .chart and .mid with
assets.
442 total scan-chart tests passing.
…ognizedEventsTrackMidiEvents
The parser preserves three post-writer-stack round-trip buckets that the
writers weren't yet emitting:
- `metadata.extraChartSongFields` (unknown `[Song]` keys, .chart source)
- `unrecognizedSyncTrackEvents` (non-B/TS `[SyncTrack]` lines, .chart)
- `unrecognizedEventsTrackMidiEvents` (non-text MIDI events on the EVENTS
track — e.g. RB practice-mode
assist-sample notes 24/25/26)
Emit them:
- `writeChartFile` appends `extraChartSongFields` at the tail of `[Song]`
(values written verbatim — no quoting added or stripped). Editors and
consumers should treat these as opaque and never synthesize them; this
is strictly a round-trip aid for tools like Moonscraper that author
deprecated fields (`Player2`, `HoPo`, `PreviewEnd`, audio-stream
filenames, etc.). Any audio file discovery should go through folder
scan, not via `[Song].*Stream` values.
- `writeChartFile` emits `unrecognizedSyncTrackEvents` alongside tempos +
time signatures in `[SyncTrack]`, sorted by tick (TS < B < raw at the
same tick for determinism). Text is written verbatim as
`${tick} = ${text}`, so tempo anchors (`A <microseconds>`) and any
future SyncTrack event types survive a parse → write loop without
parser updates.
- `writeMidiFile` appends `unrecognizedEventsTrackMidiEvents` to the
EVENTS track after sections / end events / unrecognized text events /
coda. Events arrive with absolute-tick `deltaTime` (scan-chart's
post-process) and get re-deltified by `finalizeMidiTrack`.
`.chart`-only fields (`extraChartSongFields`, `unrecognizedSyncTrackEvents`)
are dropped when writing to `.mid`, and `unrecognizedEventsTrackMidiEvents`
is dropped when writing to `.chart` — tests pin both contracts so a future
writer change doesn't accidentally smuggle them through as something else.
11 new tests in `round-trip-unrecognized-extras.test.ts` cover legacy-field
preservation, tempo-anchor round-trip, forward-compat for unknown SyncTrack
types, co-occurrence with tempos/TS at the same tick, practice-assist note
round-trip on `.mid`, and the cross-format drop contracts. All 1020 tests
pass (1009 existing + 11 new).
Moonscraper/Clone Hero's MIDI reader matches the literal prefix `[section ` and silently drops sections written as plain `section name`. Emit `[section <name>]` to match the .chart writer and the RBN convention; YARG accepts both forms.
The chart-format spec (Chart-File-Formats/chart-format/Format-Overview.md, "Basic Global Events" table + example) shows sections in .chart files written as plain `section <name>`. The Rock Band / GH1+2 MIDI Practice-Sections docs say parsers should accept both `[section <name>]` and plain `section <name>`. A scan of 15,525 charts in the wild confirms it: .chart 93,034 plain events, 0 bracketed (across 6,490 charts) .mid 9 plain events, 133,046 bracketed (across 9,047 charts) Updates: - chart-writer.ts: emit plain `section <name>`. Moonscraper / Clone Hero's .chart reader explicitly checks for `"section` (no leading bracket) at the start of the E-event payload — bracketed sections in .chart silently dropped on import. - chart-parser.ts: strict plain-only. Bracketed sections in .chart fall through to unrecognizedTextEvents instead of being promoted to typed sections. - midi-writer.ts (already in ec5668c) keeps emitting `[section …]`, the MIDI canonical form. - midi-parser.ts (unchanged) stays lenient on input — accepts both forms per the spec. - per-track-data.test.ts: pin the new .chart strictness.
The writer was emitting every key in `defaultMetadata` whose value was populated, which after parseChartAndIni's getIniInteger / getIniBoolean helpers fall back to defaults for missing keys meant a round-tripped song.ini grew ~18 default lines (`diff_keys = -1`, `loading_phrase = `, `modchart = False`, etc.) that the source never had. Skip keys whose value equals defaultMetadata[key]. The re-parser fills missing keys back from defaults, so the round-trip stays lossless — present-but-default and absent-on-source produce equivalent metadata. Verified against a real-world chart: source song.ini had 26 lines, the written output now also has 26 (same set), down from 44.
The parser strips enclosing quotes from `[Song]` values on read, so `extraChartSongFields` always holds bare strings. Emitting them verbatim produced `MusicStream = song.ogg` (unquoted) — Moonscraper rejects that shape and fails to load the audio. Re-apply quoting on write per the spec's field-type table: - `Player2` (the lone `bare string` field) stays bare. - Numeric and boolean primitives stay bare via value-shape check. - Everything else gets quoted. The default-quote-strings rule is verified across a 62K-chart corpus: every `string` / `file path` field is quoted ≥99.97% of the time, and the only de-facto boolean field (`OriginalArtist = false`) appears unquoted in 171/171 occurrences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on top of the writer PRs (#108-#123). Standalone of the stack, the change is ~30 lines in
chart-writer.tsplus tests.Summary
The chart parser strips enclosing quotes from
[Song]values on read (chart-parser.ts'schartSongMetaRegex = /^(.+?) = "?(.*?)"?$/), sometadata.extraChartSongFieldsalways holds bare strings. The writer was emitting them verbatim — which producesMusicStream = song.ogg(unquoted). Moonscraper rejects that shape and silently fails to load audio, so any chart edited and re-saved viawriteChartFilecame back broken.Changes
writeChartFilere-applies quoting on emit per the .chart spec's field-type table at Format-Overview.md (lines 78-85, 143-199):Player2— the spec's lonebare stringfield — stays bare via a small allowlist./^(?:-?\d+(?:\.\d+)?|true|false)$/).stringandfile pathfield is quoted) and mirrors what readers (Moonscraper, scan-chart) tolerate.The parser is untouched, so the
extraChartSongFieldsshape stays "always bare" — callers don't have to think about quoting.Why these rules
Verified across a 62,697-chart corpus (Encore + remaining-charts dumps). For every well-known string-typed
[Song]key, ≥99.97% of occurrences are quoted; outlier counts are 3-5 per key (sloppy authoring, not a real divergence). Stream filenames specifically are quoted 100% of the time. The only de-facto boolean field in the corpus,OriginalArtist, appears asOriginalArtist = falsein 171/171 charts that use it — covered by the boolean branch of the value-shape regex.Test plan
round-trip-unrecognized-extras.test.ts:MusicStreamemits with quotes Moonscraper expects (pins on-disk text).Player2emits unquoted (bare-string field per spec).HoPo,PreviewEnd) emit unquoted.