Not Enough SQL Exporter

The exporter module for NESQL. The server module can be found here. Still WIP.

The exported database is an HSQLDB database, and can also be queried directly with the HSQLDB client.

Currently supported exported data:

• NEI item list
• Forge ore dictionary
• Forge fluids and fluid container data
• Minecraft crafting table and furnace recipes
• GTNH GT5 recipe maps
• BetterQuesting quests
• Thaumcraft aspects for items

For GTNH, the exported database file is ~600 MB, and the export process takes my computer ~60 min. There are ~150k exported images, which take up a total of ~90 MB raw; more on disk due to file overhead.

Instructions

1. Place NESQL-Exporter-*.jar and NESQL-Exporter-*-deps.jar into your mods/ folder. NotEnoughItems is the only required dependency.
2. If you have bugtorch-1.7.10-*.jar in your mods/ folder, move it somewhere else temporarily, as it conflicts with rendering enchanted items. See below.
3. Start Minecraft and join a world. It is recommended that you use a new creative, single-player world for this. It is recommended to use a new world because the exporter uses your current player state, so e.g. if you have Spice of Life installed, tooltips will reflect which foods you have eaten.
4. If you're using the GTNH version of NotEnoughItems, open your inventory and view the NEI item list so that it gets loaded. If you forget to do this, some items might not be exported.
5. If you are exporting Thaumcraft data, it is recommended that you acquire all Thaumcraft knowledge, as missing knowledge may cause some data not to be exported properly. See below for detailed instructions.
6. Run the command /nesql. You may optionally specify a repository name with /nesql your_repository_name.
7. You can now pause the game, and the export will continue while the game is paused. Doing this can make things a little bit faster.
8. Wait for the export process to finish. It can take a very long time depending on how many mods you have installed. The exported database can be found in .minecraft/nesql.
9. Once the export is finished, you can delete the two mod jars. Remember to replace bugtorch-1.7.10-*.jar if you moved it earlier.

You may see Forge complaining about a call to System.exit() in the logs. Please disregard it; this is due to one of Hibernate's libraries containing a call to System.exit().

Memory allocation

The exporter holds a lot of recipe/item data in the JPA entity manager before flushing, and the recipe post-processing pass in particular is memory-hungry on modpacks with many recipes.

• Minimum: 6 GB max heap on a fresh GTNH world.
• Recommended: 10 GB max heap for GTNH — exports with hundreds of thousands of recipes have been observed to OOM at 4 GB during recipe post-processing (around 99% through that phase).

In Prism Launcher / MultiMC: Edit Instance → Settings → Java → Memory. In the vanilla launcher: edit the JVM -Xmx argument in the profile.

If the export crashes with OutOfMemoryError, the HSQLDB transaction is never committed, so the partial output in .minecraft/nesql/ can be deleted and the run restarted from scratch after raising the heap.

Monitoring progress

The exporter prints high-level milestones (start of each plugin, "Data exported!", "Committing database...", "Commit complete!") to in-game chat. It also writes intermittent per-plugin progress lines to Minecraft's log file — these are useful when the game is minimized or paused and you can't see chat.

Tail the log and filter for exporter lines (Linux/macOS):

tail -F .minecraft/logs/fml-client-latest.log \
  | grep -E "Not Enough SQL Exporter|nesql" \
  | grep -vE "WrappedPrintStream|^\s*at "

The second grep drops stack-trace frames from non-fatal warnings that the exporter prints via printStackTrace().

Lines look like:

[Not Enough SQL Exporter/base/]: Post-processed recipe 200000 of 225344
[Not Enough SQL Exporter/quest/]: Processed quest 3700 of 3739

so you get an X-of-Y readout for each phase. The frequency is controlled by the logging_frequency config option (default 100; lower for more frequent updates, set to ≤0 to disable).

If the log stops advancing for many minutes while Minecraft itself is still running, check the log for OutOfMemoryError — that indicates the export thread crashed and won't recover.

Enchanted item rendering issue

If enchanted items (anything with that purple glint overlay) are showing up as blank images for you, then there is likely a conflict with the BugTorch mod.

I believe that this line is causing the problem, but it probably can't be fixed by NESQL Exporter since it is a mix-in. I recommend just temporarily removing BugTorch from your mods/ folder until after export is complete.

Thaumcraft knowledge

If you are exporting Thaumcraft data, some data may not be exported properly if your character is missing Thaumcraft knowledge. It is recommended that you do the following to acquire all knowledge before exporting:

1. Read the creative Thaumonomicon
2. Run the following commands to purge all warp:
   • /tc warp @p set 0
   • /tc warp @p set 0 PERM
   • /tc warp @p set 0 TEMP