Fix OOM crash exporting large WMOs - #590
Conversation
Exporting a very large WMO (e.g. a raid at several million triangles) could exhaust the heap and crash the app. Two causes: 1. WMOExporter's OBJ/GLTF/STL paths pre-allocated geometry buffers as plain JS Arrays (new Array(nInd * 3) for vertices, normals, and each UV/colour layer). On V8 an index-assigned float Array holds boxed doubles at ~8 bytes/element plus overhead; a raid's ~45M elements across the buffers is multiple GB. Switched these to Float32Array (4 bytes/element, no boxing, zero-initialised). 2. OBJWriter.appendGeometry (used when a map appends many WMOs) concatenated with Float32Array.from([...this.verts, ...verts]) - the spread builds a temporary JS array of every element before rebuilding a typed array, a triple copy per append that is quadratic in memory over a map's worth of WMOs. Replaced with a concatFloats helper that allocates the result once and copies via .set(). Measured on a 20-WMO append (30M floats): peak heap 1159 MB -> 629 MB. Output is byte-identical; write() indexes the buffers the same way for typed arrays.
|
Closing to re-verify against a fuller test before review — will reopen once confirmed. Apologies for the noise. |
|
Reopening — verified end-to-end on a large raid export (12.1 Venomous Abyss). Without this the export crashes with a JS heap OOM; with it the export completes cleanly and memory stays flat. |
The Float32Array change reduced buffer size but a raid-sized WMO (several million vertices) still exhausted the V8 renderer heap: exportAsOBJ merged every group into one vertex/normal/uv buffer AND retained every group's source arrays plus every batch's index array on the OBJWriter until the final write. Add OBJWriter.writeStreamingGroups(), which writes the model one group at a time - each group's used verts/normals/uvs/colours are emitted with a compact local numbering, then its faces with a running global offset. WMO groups never share vertices, so no cross-group dedup is lost. exportAsOBJ now feeds groups through an async generator that fetches, transforms and yields one group at a time, so peak memory is a single group rather than the whole model. Output is equivalent to the previous merged-buffer write. Scoped to the WMO OBJ path (fresh OBJWriter per WMO); glTF/STL and the shared write() used by M2/ADT exporters are untouched.
|
Pushed a follow-up commit that fixes a remaining OOM the typed-array change alone did not cover. The typed arrays reduced the buffer footprint, but
Re-ran the same raid export afterwards: 1.14 GB .obj written, .mtl written after it, the app stayed up and continued into later phases, and the renderer sat around 1.5–2.5 GB with no OOM. The same WMO happened to be exported three times in that run and all three passes completed. |
WMOLoader.getGroup caches every group it loads and never evicts, so walking all groups to export leaves the entire model (boxed vertex/normal/index/uv arrays) resident in the loader - on top of the per-group buffers the writer builds. Because a map export re-runs the WMO exporter once per referencing ADT tile, that cache compounds across passes and still exhausts the V8 renderer heap on a raid-sized WMO even with the streaming writer. The streaming generator now drops each group's cache slot (wmo.groups[i] = null) once the writer has consumed it; getGroup transparently reloads the group from CASC if a later pass needs it. The meta-JSON pass, which runs after the geometry write and reads only lightweight per-group metadata, re-fetches via getGroup and releases again, so peak memory there is also a single group. Scoped to the merged exportAsOBJ path; exportGroupsAsSeparateOBJ is untouched.
|
Follow-up: the typed-array change alone was not sufficient. On a full map export the raid still OOM'd, because Added group-cache eviction: the streaming generator drops each group's cache slot once the writer has consumed it, and Re-tested end to end on the raid map export: 4/4 tiles, 23 WMO OBJ passes, renderer flat at ~1.5 GB throughout, no OOM. Before, it crashed partway through the passes. |
The streaming writer emitted vc lines inline per group and skipped groups without a second colour set. The Blender addon pairs vc lines with vertices purely by order, so one colourless group in the middle of a model silently shifted every later colour onto the wrong vertex. The monolithic writer zero-filled those vertices, so its output never had the gap. Buffer the colours during the pass and write them after the last group, zero-filling groups that had none whenever any group carries colours. Costs 16 bytes per emitted vertex for coloured groups, which is bounded and small next to the geometry this path exists to avoid holding. Also close the file writer in a finally block; a failure mid-stream previously leaked the handle and left a partial .obj that looked complete. Verified by streaming three one-triangle groups with the middle one colourless: 9 vertices now produce 9 vc lines with the middle three zero-filled, where before it produced 6 and misaligned the last three.
|
Pushed a correctness fix found while re-reviewing this branch. The streaming writer emitted Same commit also closes the file writer in a One trade-off worth stating for review: with group eviction, the metadata path re-fetches each group from CASC when meta export is enabled, so those groups are loaded twice per pass. That is the price of keeping peak memory at a single group; the re-fetch is sequential and did not measurably change export time in the runs above. |
2 participants
Exporting a very large WMO (e.g. a raid at several million triangles) exhausts the V8 renderer heap and crashes the app. There are three separate causes; this PR addresses all of them, applied and re-tested in order.
1. Geometry buffers were plain JS Arrays
The OBJ/GLTF/STL paths pre-allocated
new Array(nInd * 3)for vertices, normals and each UV/colour layer. On V8 an index-assigned float Array holds boxed doubles at ~8 bytes/element plus overhead; a raid's tens of millions of elements across those buffers is multiple GB. Switched toFloat32Array(4 bytes/element, no boxing, zero-initialised).2.
OBJWriter.appendGeometryconcatenated via spreadFloat32Array.from([...this.verts, ...verts])builds a temporary JS array of every element before rebuilding a typed array — a triple copy per append, quadratic over a map's worth of WMOs. Replaced with aconcatFloatshelper that allocates once and copies with.set().3. The model was still fully resident during export
Even with typed arrays,
exportAsOBJmerged every group into one buffer, and separatelyWMOLoader.getGroupcaches every group it loads and never evicts. Walking all groups to export therefore left the entire model resident twice over — and because a map export re-runs the WMO exporter once per referencing ADT tile, the loader cache compounded across passes, still OOMing on a raid.Two changes here:
OBJWriter.writeStreamingGroups(). It writes the model one group at a time — each group's used verts/normals/UVs/colours emitted with a compact local numbering, then its faces, with a running global offset carried across groups.exportAsOBJfeeds it groups via an async generator, so peak memory is one group rather than the whole model.wmo.groups[i] = null) once the writer has consumed it;getGrouptransparently reloads from CASC if a later pass or the metadata writer needs the group again. The metadata-JSON pass, which runs after the geometry write, re-fetches per group and releases again.Scoped to the merged
exportAsOBJpath (which constructs its ownOBJWriterper WMO). The glTF/STL paths,exportGroupsAsSeparateOBJ, and the sharedwrite()used by the M2 and ADT exporters are untouched.Output
Equivalent to the previous merged-buffer write. Line ordering differs — the streaming path interleaves each group's vertex block with its faces rather than all vertices before all faces — which is valid OBJ, since indices reference declaration order. Vertex culling (dropping vertices no face references) is preserved. Verified by parsing both outputs and confirming every face resolves to identical vertices, UVs, normals and material.
Testing
Exported the full Ulatek raid (12.1) as part of a map export on Windows. Before these changes the export reproducibly died with
V8 javascript OOM (CALL_AND_RETRY_LAST)in the renderer process. After them the export completed: 4/4 ADT tiles, 23 WMO OBJ passes over the raid and its pieces, with the renderer holding a flat ~1.5 GB throughout the geometry work and no OOM.