Skip to content

java

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

java

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
jni
jni
 
 
src
src
 
 
README.md
README.md
 
 
build.gradle.kts
build.gradle.kts
 
 
gradle.properties
gradle.properties
 
 
pom.xml
pom.xml
 
 
settings.gradle.kts
settings.gradle.kts
 
 

README.md

Java FFM bindings — whiteout

Pure-Java bindings to whiteout_lib, routed through the C ABI shared library (whiteout_native.dll / libwhiteout_native.so / libwhiteout_native.dylib) using the JDK's Foreign Function & Memory API. JDK 22 or newer is required (FFM was preview before that; JDK 24+ is recommended to avoid the restricted-method warnings).

Every .java file under src/main/java/whiteout/ is autogenerated by tools/codegen/emit_java.py from the same IR that drives the pybind11, Embind, and C ABI emitters. Don't hand-edit them — the next codegen run overwrites them.

Package Surface
whiteout.common Vector{2,3,4}f, Quaternion, Matrix33f, Matrix44f
whiteout.textures Texture, BLP/PNG/JPEG/DDS/BMP/TGA parsers + writers, GIF writer, APNG
whiteout.mdx Warcraft III MDX / MDL model + parser + writer
whiteout.m2 WoW M2 model + parser + writer (multi-file aware)
whiteout.m3 StarCraft II / HotS M3 model + parser + writer
whiteout.mpq MPQ archive read + write
whiteout.casc CASC archive read (local + online)
whiteout.host OsFileSystem, SimpleThreadPool, SimpleHttpHandler
whiteout.utils VertexBuffer, VertexBufferBuilder, free math functions
whiteout.interfaces Abstract bases (VirtualPathFileSystem, WorkerPool, HttpHandler)

Architecture

java code  ──►  whiteout.<module>.*                 (public API — AutoCloseable handles)
                       │
                       ▼
            whiteout.<module>.internal.Native       (FFM MethodHandles per module)
                       │
                       ▼
            whiteout.common.internal.NativeCommon   (lookup + invokeNative helper)
                       │
                       ▼
            whiteout_native (C ABI)                 (extern "C" shim from bindings/c/)
                       │
                       ▼
            whiteout_lib (C++)                      (real implementations, exception-free)
  • The C++ library is compiled with exceptions disabled (-fno-exceptions / /EHs-c-). The C ABI in bindings/c/ is a flat wrapper that preserves that no-throw guarantee.
  • That C ABI is mechanically wrapped by FFM MethodHandles — one Native.java per package, hidden in a non-exported internal subpackage.
  • A single NativeCommon.invokeNative(handle, args...) helper in whiteout.common.internal invokes each handle and re-throws any Throwable as an unchecked RuntimeException. Generated wrapper methods carry no inline try / catch boilerplate as a result — every call is a clean one-liner.

JPMS layout

The generated module-info.java:

module whiteout {
    requires java.base;

    exports whiteout.common;
    exports whiteout.textures;
    exports whiteout.mdx;
    exports whiteout.m2;
    exports whiteout.m3;
    exports whiteout.utils;
    exports whiteout.host;
    exports whiteout.interfaces;
    exports whiteout.mpq;
    exports whiteout.casc;
}

Every whiteout.<module>.internal subpackage is deliberately not exported. Strong encapsulation is enforced at compile time: user code that tries to import whiteout.common.internal.Handles fails with

package whiteout.common.internal is declared in module whiteout,
which does not export it

There is no public method on the handle classes that hands a MemorySegment back to user code.

Lifecycle: everything is AutoCloseable

Every generated class implements AutoCloseable. The instance owns a native allocation (or holds a borrowed view of one) and must be released — try-with-resources is the natural shape:

import whiteout.textures.PngParser;
import whiteout.textures.PngWriter;
import whiteout.textures.Texture;

byte[] roundtrip(byte[] pngBytes) {
    try (PngParser parser = new PngParser();
         Texture   tex    = parser.parse(pngBytes).orElseThrow();
         PngWriter writer = new PngWriter()) {
        return writer.write(tex);
    }
}

Rules:

  1. Owned vs borrowed. A handle returned by a new, a static factory (Vector3f.of(...)), or a method whose Javadoc reads "a fresh X owning a native allocation" is owned — closing it frees memory. Sub-slices returned by field getters (e.g. node.getPivotPoint()) are borrowed views; their close() is a no-op, but the slice must not outlive the parent.
  2. Don't share across threads. Native types make no synchronisation guarantees. Coordinate externally if you need to hand a handle across threads.
  3. Optional<T> for fallible parsers. Parsers return Optional<Texture> (etc.) so the caller chooses how to react — .orElseThrow(), .ifPresent(...), .map(...). The library never throws on bad input; an empty Optional paired with parser.getIssues() is how recoverable failure surfaces.

Quick start: format-by-format

import whiteout.mdx.*;
import whiteout.textures.*;
import whiteout.host.OsFileSystem;
import java.nio.file.*;

byte[] mdxBytes = Files.readAllBytes(Path.of("Footman.mdx"));

// ── MDX (Warcraft III) ─────────────────────────────────────────────────
try (MdxParser parser = new MdxParser()) {
    Model model = parser.parseBufferFormat(mdxBytes, MdxMDLXFormat.MDX);
    try (model) {
        try (MdxWriter writer = new MdxWriter()) {
            // MDL text, HiveWorkshop dialect:
            byte[] mdlText = writer.writeMdxFormatMdlFormat(
                model, MdxMDLXFormat.MDL, MdxMdlFormat.HIVEWORKSHOP);
            Files.write(Path.of("Footman.mdl"), mdlText);
        }
    }
    if (parser.hasIssues()) {
        parser.getIssues().forEach(msg -> System.err.println("warn: " + msg));
    }
}

// ── M2 (WoW) — multi-file via OsFileSystem ─────────────────────────────
import whiteout.m2.M2Parser;
import whiteout.m2.Model;

try (OsFileSystem fs    = OsFileSystem.create("C:/wow/Models");
     M2Parser     parse = new M2Parser();
     Model        model = parse.parse(fs, "character/HumanMale.m2")) {
    System.out.printf("%d bones, %d textures%n",
        model.getBones().size(), model.getTextures().size());
}

// ── Textures ───────────────────────────────────────────────────────────
try (BlpParser bp = new BlpParser();
     Texture   t  = bp.parse(Files.readAllBytes(Path.of("texture.blp"))).orElseThrow();
     PngWriter pw = new PngWriter()) {
    Files.write(Path.of("texture.png"), pw.write(t));
}

Archives & threading

import whiteout.mpq.MpqStorage;
import whiteout.casc.CascStorage;
import whiteout.host.SimpleThreadPool;
import whiteout.host.SimpleHttpHandler;

// MPQ
try (MpqStorage mpq = MpqStorage.open("war3.mpq", null /* WorkerPool */).orElseThrow()) {
    byte[] blp = mpq.readFile("textures/character/footman.blp").orElseThrow();
}

// CASC — local install with an 8-thread BLTE pool
try (SimpleThreadPool pool = new SimpleThreadPool(8);
     CascStorage      casc = CascStorage.openLocal("/path/to/Game/Data", pool).orElseThrow()) {
    System.out.println("files indexed: " + casc.fileCount());
    byte[] adt = casc.readFileByName("World/Maps/Azeroth/Azeroth_31_46.adt").orElseThrow();
}

// CASC — online (HTTP-backed)
try (SimpleHttpHandler http = new SimpleHttpHandler();   // WinHTTP / libcurl
     CascStorage       casc = CascStorage.openOnline("wow", http).orElseThrow()) {
    byte[] manifest = casc.readManifest();
}

SimpleThreadPool is a Java handle to the native pthread pool that the C++ library's pool-aware APIs (Texture.generateMipmapsPool, Texture.format(fmt, pool), BLTE batch decoding, …) fan out across. The parameter is always optional — pass null for single-threaded behaviour.

Shared math: zero-roundtrip field access

The math types use a MemoryLayout + VarHandle so per-component access doesn't pay for an FFM call:

try (Vector3f v = Vector3f.of(1f, 2f, 3f)) {
    v.setY(v.getY() * 2f);          // pure VarHandle reads/writes
    System.out.println(v.length()); // VarHandle for in/out, FFM for the C math kernel
}

The JIT inlines the VarHandle accessors — a tight loop does not cross the JNI boundary per component.

For nested POD fields (e.g. MipLevel.getExtent()) the codegen recognises (via libclang's is_pod()) that a bitwise copy is safe and emits a MemorySegment.copy setter; non-POD fields like Node::name (which contains a std::string) route through the C wrapper so the real C++ assignment operator runs.

Reading parser output: idiomatic Java patterns

  • List views. std::vector<T> field getters surface as List<T> backed by the native container. Iteration is cheap; element access lazily wraps a borrowed view.
  • Typed primitive arrays. Free functions that pack a vector of primitives (e.g. vertex buffers) hand back float[] / int[] / long[], not opaque byte[]. Conversion happens once on the C side.
  • equals / hashCode / toString. Math types compare via the native comparator (so NaN / -0.0 follow the C++ rules) and hash on component values. Non-math handle classes intentionally don't override these — identity is the native pointer.

Native library lookup

NativeCommon checks the system property whiteout.native.path first; if set, it loads that absolute path. Otherwise it falls back to System.loadLibrary("whiteout_native"), which uses java.library.path.

Both forms need --enable-native-access=ALL-UNNAMED (or a --enable-native-access=whiteout flag in modular runs) — FFM is not enabled by default. Sample invocation:

java --enable-native-access=whiteout \
     -Djava.library.path=packages/java/lib \
     -p packages/java -m whiteout/com.example.Main

Layout

bindings/java/
├── README.md                                    (this file)
├── jni/                                         JNI shim for trampoline subclasses
│                                                (HttpHandler / VirtualPathFileSystem
│                                                implemented in Java that the C++ side
│                                                calls back into).
└── src/main/java/
    ├── module-info.java                         JPMS module (generated)
    └── whiteout/
        ├── common/        + common/internal/    Math types + FFM glue, Handles bridge
        ├── interfaces/                          Abstract bases (WorkerPool, VFS, HttpHandler)
        ├── textures/      + textures/internal/  Texture + parsers/writers
        ├── mdx/           + mdx/internal/       MDX model + parser + writer
        ├── m2/            + m2/internal/        M2 model + parser + writer
        ├── m3/            + m3/internal/        M3 model + parser + writer
        ├── mpq/           + mpq/internal/       MPQ archive types
        ├── casc/          + casc/internal/      CASC archive types
        ├── host/          + host/internal/      OsFileSystem, ThreadPool, HttpHandler
        └── utils/         + utils/internal/     VertexBuffer, math free functions

The internal/ subpackage in each tree holds the FFM glue (Native.java — one per module). They are private to the JPMS module and not exposed in the published module-info.java.

Building

.\scripts\build-java.ps1

The script:

  1. Runs tools/codegen/codegen.py for every module across the C and Java backends (c-common-header, c-common, c-header, c-source, java-common, java).
  2. Builds whiteout_native.dll (the C ABI shim + whiteout_lib) via CMake into build-java/c-dist/.
  3. (Optional) javac + jar the generated sources into a modular jar under packages/java/.
  4. Stages the jar + native library so they're discoverable via --module-path packages/java + -Djava.library.path=packages/java/lib.

Codegen-only:

$env:Path = "C:\Projects\WhiteoutLib\.venv\Scripts;$env:Path"
python -m tools.codegen.codegen textures --backend c-common-header
python -m tools.codegen.codegen textures --backend c-common
python -m tools.codegen.codegen textures --backend java-common
foreach ($mod in 'mdx','m2','m3','textures','mpq','utils','host','casc') {
    python -m tools.codegen.codegen $mod --backend c-header
    python -m tools.codegen.codegen $mod --backend c-source
    python -m tools.codegen.codegen $mod --backend java
}

Memory & exception model

  • The C++ library never throws. Built with -fno-exceptions; parsing / writing corrupt input yields a best-effort result plus issues on getIssues(). The Java side's Optional<T> is the recoverable failure channel.
  • MethodHandle.invoke() does declare throws Throwable. That's a Java type-system requirement; the C side never produces one in practice. NativeCommon.invokeNative() centralises the catch + sneaky-rethrow so every generated wrapper is a clean one-liner instead of try/catch boilerplate.
  • AutoCloseable is the contract. Use try-with-resources or call close() explicitly — every owned handle leaks native memory otherwise. Closing a handle that doesn't own its allocation (borrowed = true) is a no-op.

What's not bound

The codegen filters out methods whose signature can't be safely lowered through the C ABI:

  • return type is std::string (paths only — Optional<String> works for byte payloads that happen to be UTF-8)
  • return type is std::vector<T> for T != u8 and T not a bound struct
  • return type is a raw pointer (e.g. const u8* dataPtr())
  • parameters take ownership of a moved value

The pybind11 and Embind backends apply the same filters; if you need a new shape exposed, the IR change applies to every binding at once.

Troubleshooting

Symptom Cause / fix
UnsatisfiedLinkError: no whiteout_native in java.library.path Set -Djava.library.path=… to the directory holding the .dll / .so, or set the whiteout.native.path system property to the absolute file path.
IllegalCallerException: Illegal native access Add --enable-native-access=whiteout (modular) or --enable-native-access=ALL-UNNAMED (classpath).
WARNING Restricted method java.lang.foreign.… (JDK 24+) Same fix — --enable-native-access.
RuntimeException: symbol not found: whiteout_X Native library is older than the generated Java. Rebuild whiteout_native + re-stage; or check the whiteout.native.path actually points at the latest.
NullPointerException from a parsed model field The native allocation was freed (close() was called) before the field access. Owned handles in try-with-resources should outlive any borrowed view.
Subclassing HttpHandler from Java leaks native callbacks Trampoline keeps Python/Java alive on the C++ side. Hold a strong reference to the handler until every CascStorage using it is closed.
Compile error: package whiteout.X.internal is not exported Working as intended — the internal packages are hidden. Use the public API in whiteout.X instead.

Extending

To add a new C++ type to the Java surface:

  1. Annotate the C++ declaration with @bind (see docs/BINDINGS_STRATEGY.md).
  2. Add a ModuleConfig entry in tools/codegen/modules/ if it's a new module; otherwise no Python change is needed.
  3. Run .\scripts\build-java.ps1.
  4. The new Javadoc'd handle class shows up in bindings/java/src/main/java/whiteout/<module>/.

Hand-writing a .java file under bindings/java/src/main/java/ is unsupported — the next codegen run will overwrite it.

See also