TUI menu, configurable JDBC connections, drivers, docs + CI#1
Merged
Conversation
- load operation.properties as UTF-8 (was ISO-8859-1 -> mangled Korean confirm text) - set javac encoding + stdout/stderr to UTF-8 (applicationDefaultJvmArgs) for Windows - copyright headers + README -> DevsLab Co., Ltd. (주식회사 데브스랩), https://devslab.kr - show destructive confirm text in the CLI list
- i18n/messages_<lang>.properties loaded as UTF-8 (English base + language overlay; missing keys fall back to English, then to the key itself) - Messages helper (MessageFormat args); language from application.yml options.language (blank = system locale) - external + drop-in editable (add a language by dropping a file); no ResourceBundle, so a GraalVM native image stays simple - CLI `i18n [lang]` to inspect; verified en/ko load correctly
- header logo loaded from an external ASCII-art file -> swap branding without rebuild (consistent with sql/, i18n/, application.yml) - Logo loader falls back to a plain "DataLinq" wordmark if the file is missing/empty - default banner: "DataLinq" (figlet slant); CLI `logo` to preview
- application.yml now holds multiple NAMED datasources (any can be source or target); an operation picks by name (source=/target=), falling back to defaults.source/target - operation.properties: ETL target table moved from `target=` to `table=` (so `source` and `target` now mean datasource names, not the table) - MigrationEngine resolves + opens the operation's source/target by name; SCRIPT needs only the target; handlers can open extra datasources via ctx.connection(name) - options.sql-dir makes the migration folder location configurable (blank = ./sql)
- MigrationHandler: queryFrom (read a named datasource for multi-source joins),
lookup (2-column reference map), insertBatch (batched writes),
execute (raw UPDATE/DELETE for second-pass forward-FK fills)
- OrderImportHandler example ("more complex than master/detail"): one flat result set
-> customers (dedup) + orders (master, FK resolved via a lookup map) + order_items
(detail) - three target tables in one transaction
- bump build to Java 25 (release 25): best virtual-thread + JDBC behaviour, no synchronized pinning (JEP 491) - BatchRunner runs INDEPENDENT operations concurrently on virtual threads, bounded by options.max-parallel (a Semaphore) so it never exceeds the DB's connection capacity; structured via a try-with-resources virtual-thread executor (not the preview StructuredTaskScope). Per-operation failures are captured, never abort the batch. - JUnit tests verify the mechanics (all-run, concurrency bound, failure capture) without a database.
Built per tamboui's recommended MVC: - DataLinqController: all state + commands/queries, no TamboUI dependency, unit-tested (7 tests: menu build, About nav, destructive confirm flow, cancel, immediate run, failure logging, parallel run-all) - DataLinqApp: pure View (render reads the controller) + key dispatch - header(logo + mode) / menu(base items + scanned migrations) / center(output|About|confirm) / footer; UI strings via Messages (i18n en/ko) - run flow: select -> destructive i18n confirm -> MigrationEngine -> output panel; failures are logged, never crash. 'a' runs all migrations in parallel via BatchRunner (vthreads) - Main launches the TUI by default; CLI subcommands (list/config/i18n/logo/run) still work Known follow-up: a migration run is currently synchronous (blocks the UI during the run); async (virtual thread + redraw) is the next improvement.
Replace options.release=25 with a languageVersion-25 toolchain, so Gradle locates a JDK 25 (auto-detected, e.g. from ~/.jdks) for compile + run regardless of which JDK launches Gradle. `./gradlew run` no longer needs JAVA_HOME pointed at a JDK 25.
Add --enable-native-access=ALL-UNNAMED to the application JVM args so JLine's native terminal calls don't emit the Java 25 restricted-method warning (and keep working once those methods become enforced).
The app uses only Java 21 features (virtual threads are stable since 21; StructuredTaskScope is not used), so compile to 21 - it then runs on the user's default JDK without needing a JDK 25 at runtime. Running on JDK 24+/25 still avoids virtual-thread pinning in JDBC drivers (JEP 491): a runtime throughput bonus, not a requirement. Fixes UnsupportedClassVersionError.
- add a Quit base-menu entry (controller quitRequested -> the View calls quit()); i18n menu.quit (en/ko) - About and the destructive confirm now render as centered dialog() popups overlaid via stack() on the main UI, instead of replacing the centre panel - controller test for Quit; base-menu indices updated (4 base items now)
Enable mouse capture (TuiConfig.mouseCapture(true)) and dispatch input to the controller so every menu item is reachable three ways: - Left-click a menu row to select+activate it (hit-tests the list's renderedArea, maps the clicked y to a row index). - Number keys 1-9 select+activate the matching entry; labels carry a leading number prefix so the binding is discoverable. - Enter activates the highlighted row; wheel scroll is left to the list. - Single-key actions: d (toggle dry-run), r (rescan), a (run all), q (quit), and Y/N inside the destructive-confirm dialog. activateIndex(int) on the controller selects-then-activates so the number and click paths share the existing activate() flow (confirm gate for destructive ops, About popup, Quit). footer.keys updated in both locales.
The launcher loaded i18n/, branding/, sql/, and application.yml relative to user.dir, so running bin/datalinq.bat from the bin/ folder showed raw message keys and the fallback wordmark instead of the translations and logo. - Add Home: locates the running jar via the code source and treats its install dir (parent of lib/) as the primary resource base, with the working dir as a fallback. resolve(name) returns whichever base actually has the resource, so dev runs (exploded classes -> CWD = project root) and packaged runs (CWD = anywhere -> install dir) both find their files. - Bundle i18n/, branding/, sql/, and application.example.yml into the distribution image so the install dir contains the defaults. application.yml stays out of the image (gitignored, holds credentials). - AppConfig.load(readFrom, saveTo) seeds from application.example.yml on first run while saving edits to application.yml. - Main resolves every external path through Home instead of user.dir.
…rces Replace the bin/lib install tree as the primary deliverable with one self-contained datalinq.jar, runnable from anywhere via java -jar or jbang. - Shadow plugin builds datalinq.jar. mergeServiceFiles() merges the JDBC java.sql.Driver descriptors so BOTH drivers register via ServiceLoader; this needs duplicatesStrategy = INCLUDE, otherwise the default drops MariaDB's descriptor before the transformer runs and only MSSQL stays registered. Signature files are stripped to avoid the signed-jar SecurityException. - processResources bakes i18n/, branding/, and application.example.yml into the jar so a bare jar renders the right logo + translations with zero setup. - The loaders now fall back to those classpath defaults: Messages overlays an external i18n/ file over the bundled one; Logo prefers an external logo.txt then the bundled one; AppConfig.loadResource seeds config from the bundled example while still saving to the user's application.yml. - Home also treats the jar's own directory as a resource base, so a dropped jar picks up sql/ and overrides placed beside it. - jbang-catalog.json aliases 'datalinq' to the jar with the UTF-8 / enable-native-access java-options. Verified non-interactively from a scratch dir holding only the jar: logo, ko translations, and example config all resolve from the classpath; the merged java.sql.Driver lists both com.microsoft.sqlserver.jdbc.SQLServerDriver and org.mariadb.jdbc.Driver.
Running the jar needs no setup - the defaults live on the classpath. But to customise the logo, translations, or config you previously had to hand-create the files next to the jar. 'datalinq init' writes them into the current folder in one step: i18n/, branding/logo.txt, application.example.yml, and an empty sql/ with a README documenting the NN_Title folder + operation.properties convention. Existing files are skipped, so it is safe to re-run. Verified from a scratch dir holding only the jar: first run creates 5 files (Korean kept byte-for-byte through the copy), second run skips them all.
The footer hard-coded "1-9" but only as many digits as there are menu
entries do anything (4 base + N migrations). Parameterise footer.keys with
{0} and fill it with the real range (e.g. "1-8", or "1"), capped at 9 since
only 1-9 are bound.
get() returns the raw Object, which is exactly right for pass-through ETL -
the value flows back into an insert uncast. But handlers that compute with a
value (arithmetic, comparisons) had to write ((Number) r.get("qty")).longValue().
Add getLong/getInt/getBigDecimal/getBool that coerce instead of cast: the same
column can arrive as Integer, Long, or BigDecimal depending on the driver, so a
strict cast - or a <T> get(col, Class<T>) generic - would break on that
variance. BigDecimal goes via the string form to stay exact. All return boxed
types so a SQL NULL stays null. str() already covered the String/key case.
MigrationHandler's Object signatures are unchanged and intentional:
values(Object...) alternates (String key, any value) which no type parameter
can express, and the row maps / bind args are heterogeneous SQL values.
Activating the DB Connection menu entry now opens a dedicated screen instead of logging 'coming soon'. Controller (TUI- and JDBC-free, unit-tested): - Screen enum (MAIN / DB_CONNECTION) drives which full screen the View renders. - A DatasourceGateway port abstracts read / test / persist so the controller stays testable; AppConfigDatasourceGateway backs it with AppConfig + DriverManager (5s login timeout) in production. - Commands: open/close, datasource navigation (clamped), testConnection and saveDatasource update a volatile dbStatus line. 4 new controller tests with an in-memory fake gateway (12 total in the class, 24 overall). View (DataLinqApp): a single focusable panel owns all keys - the toolkit intercepts Tab for focus navigation and a per-element keyHandler fires regardless of focus, so custom focus juggling is fragile. Instead a unified Up/Down cursor runs over [datasource rows..., URL, USER, PASSWORD]; typing edits the active field, Up/Down on a datasource row reseeds the buffers (only on an actual change, so edits survive), F5 tests (on a virtual thread so the UI does not freeze), Enter saves, Esc returns. Bilingual db.* strings added. Follow-ups: add/rename/delete datasources, set defaults from the UI, mask the password field, async test status already shown via markTesting.
These are classpath resources baked into the jar, so the conventional Java/ Gradle home is src/main/resources - the plugin bundles them automatically, no manual processResources copy needed. The external-override behaviour is unchanged: Home still reads an external i18n//branding/ next to the jar (or in the CWD) ahead of the bundled defaults, and datalinq init still extracts them from the same classpath paths (/i18n/, /branding/logo.txt). application.example.yml stays at the project root as a user-facing config template (.env.example style) and is still copied into the jar for the first-run seed; sql/ stays at the root as sample external data the scanner discovers, not a bundled resource.
Two real problems from testing: 1. Editing was unreachable. The unified Up/Down cursor started on the datasource list, so reaching the fields meant pressing Down past every datasource - not discoverable, and typing did nothing until you got there. Replace it with an explicit two-pane model: Left/Right switches between the datasource list and the edit fields; Up/Down moves within the active pane (highlighted by a cyan border); typing edits the active field. F5/Enter/Esc unchanged. Footer hint updated in both locales. 2. Labels misaligned. The field labels were padded by String.length(), but a monospace terminal renders Hangul in two cells, so 'URL' (3) and '사용자' (3 chars / 6 cells) pushed the colons to different columns. Extract a TextWidth helper that measures East-Asian wide glyphs as two cells and pad to a column width; the colons now line up in any locale. 5 unit tests. 29 tests total.
Two bits of feedback from testing the form: - The editable values did not look editable. Render each value as an underlined input line (its own styled span via row(), so only the value is underlined, not the label) padded to a minimum width so even an empty field shows the line. The active field stays yellow with the block cursor. - Password masking should be a setting (some users want it visible). Add options.mask-password (default true, so the password now masks by default instead of showing in plain) read into the controller as togglable state; the password field renders bullets while still saving the real value. The Settings screen will expose the toggle; for now it is read from application.yml. 1 controller test for the toggle (30 total). Label alignment already works via TextWidth, so the labels were left as-is rather than switched to a form-field widget (which would force Tab-based focus navigation).
Activating the Settings menu entry now opens an editor for the options.* block instead of logging 'coming soon'. Controller (TUI/config-free, unit-tested): - Screen.SETTINGS + a SettingsGateway port (prod: AppConfig; test: fake). - Edit buffers for language, dry-run-default, mask-password, batch-size, max-parallel, sql-dir, seeded on open. Up/Down moves rows; toggle cycles the choice/boolean rows; typing edits the text rows (digits only for the numbers). - saveSettings() persists all via the gateway; mask-password applies live, and a changed sql-dir triggers rescan() so the menu reflects the new folder right away - the other half of 'drop a folder -> menu appears'. 4 controller tests (17 in the class, 34 total). Wiring: the operation provider now resolves sql-dir from config on every scan, so a Settings change rescans the new directory. AppConfig gains setOption(); the existing AppConfig gateway also implements SettingsGateway (one object, both ports). View: a single focusable panel, same robust model as the DB screen - Up/Down rows, Space/Left/Right toggles, type to edit (underlined value rows), Enter saves, Esc returns. Bilingual settings.* / field.* strings. Language / batch-size / max-parallel take effect next launch (noted on screen).
Three issues from testing the Settings screen: - Space saved instead of toggling. The default bindings map Actions.SELECT to BOTH Enter and Space, so e.matches(SELECT) caught Space before the toggle check. Save now triggers on KeyCode.ENTER only (same fix applied to the DB screen, where Space should type a space, not save). - Rows were cramped. The options column now uses spacing(1) for a blank line between rows. - The sql folder took raw text. Add a directory picker: Space on the sql-dir row opens a browser rooted at the current value (or the working dir); Up/Down moves, Enter descends into a subfolder / goes up via '..' / selects the current folder via '.', Esc cancels. Selecting sets the buffer; Enter then saves and rescans. The browse logic lives in the controller (java.nio, unit-tested with a @tempdir) so it stays View-free. Manual typing still works as a fallback. Bilingual picker.* strings added. 35 tests (controller 18).
spacing(1) put a blank line between every row, which read as too airy. Use a single blank line between the toggle group and the value group instead - tight within each group, one break between. TUI gaps are whole lines, so this is the middle ground between fully tight and a gap on every row.
Esc was unhandled on the main screen, so tamboui's router cleared focus and the follow-up redraw threw; its error handler then tried to load RenderError and, if the jar had been overwritten by a rebuild while running, failed with NoClassDefFoundError. Handle Esc on the main screen directly - it now quits cleanly (consumed before the focus-clear), so the render-error path is never entered. The DB/Settings screens already consume Esc (back), so only the main screen was affected.
Apply spacing(1) to the datasource list and the form fields on the DB Connection screen so the rows get the same breathing room as the Settings screen, instead of being packed tight.
The inline [S]/[T] markers read like section headers ('everything below is the
source'), but they actually just flag the one default-source and one
default-target datasource. Replace them:
- The datasource list is now a plain connection list (no markers).
- Below it, an explicit line: 'default source: X' / 'default target: Y'.
- S / T keys set the selected datasource as the default source / target and
persist immediately (new DatasourceGateway.setDefaultSource/Target).
This also matches the model the user kept: datasources are role-agnostic
connections; an operation picks its source/target by name and the defaults are
just the fallback when it doesn't. So one source + many targets works fine -
each operation routes its own target; the 'default target' is only the
fallback. 1 controller test (19 in the class, 36 total).
Instead of one raw URL, the DB form is now IntelliJ-style: - A DB type selector (Space cycles MS SQL Server / MariaDB-MySQL / Custom). - Structured types show Host / Port / Database fields and derive the JDBC URL (shown as a live preview); Custom keeps a hand-written URL. - JdbcUrls builds the URL per type (default ports 1433 / 3306). Only the bundled drivers are offered; the MariaDB driver also connects to MySQL servers. Storage: application.yml datasources now carry type/host/port/database; the URL is derived. AppConfig.url() derives for structured types and returns the stored url for custom or legacy (no type) entries, so old 'url:' configs keep working. DatasourceGateway/AppConfig gain structured accessors + saveStructured; the controller exposes them and saveDatasourceStructured. Also: removed the hardcoded 'MS SQL Server -> MariaDB' line from the About box (the tool is JDBC-generic). Bilingual field.type/host/port/database strings. Verified: structured config derives the right URLs (sqlserver + mariadb), legacy url: still resolves. JdbcUrlsTest (4) + a structured save test (controller 20); 41 total.
Bundle the PostgreSQL driver (BSD, ~1MB) as a third structured type
(jdbc:postgresql://host:port/db, default 5432). The fat jar now merges three
java.sql.Driver entries (mssql, mariadb, postgresql).
For any other database, IntelliJ-style on-demand drivers:
- DriverManager ignores drivers loaded by a foreign class loader, so an
externally-loaded driver is wrapped in a DriverShim (loaded by the app class
loader, delegating to the real driver) and registered - that is the one real
source of complexity here.
- Drivers.loadExternal() (called at startup) scans ~/.datalinq/drivers/*.jar,
loads them in a URLClassLoader and registers each via the shim.
- 'datalinq driver <name>' downloads a catalog driver (mysql / postgresql / h2 /
sqlite / oracle) from Maven Central into that dir; 'datalinq driver' lists them.
Users may also just drop a driver jar there. Such drivers are used from the
Custom type + a hand-written URL.
Avoided bundling Oracle (proprietary) and MySQL Connector/J (GPL) - the MariaDB
driver already connects to MySQL servers, and the rest are one download away.
Verified end-to-end on the real machine: 'driver h2' downloads the jar, and
DriverManager.getConnection("jdbc:h2:mem:test") then connects through the shim
(H2 2.3.232). JdbcUrlsTest (5) + DriversTest (4); 46 total.
drivers: only load external JDBC drivers for the tui/run commands (the ones that actually open connections). loadExternal() opens a URLClassLoader over the jars, which on Windows keeps the files locked for the JVM's lifetime; doing it on every command made `driver <name>` fail to overwrite an already-downloaded jar with a cryptic FileSystemException - the real reason an Oracle re-download appeared to "fail". download() now reports HTTP status clearly (404 -> actionable message) and stages to a .part file before an atomic move, so a failed download never leaves a truncated jar. ui: localise the output panel title (was hardcoded "output"); ellipsise the derived URL preview instead of hard-cutting it at the panel border; indent the default source/target lines to line up with the datasource list; hide the menu scrollbar when the list fits (AS_NEEDED) so short menus stay clean.
README said a fixed "Source = MS SQL Server, Target = MariaDB" and "TUI wired next", both stale: connections are now configurable by type (sqlserver / mariadb / postgresql) + host/port/database or a custom URL, drivers are bundled/downloadable, and the TUI is done. Reframe as cross-vendor, document the structured config, the `driver` download flow, and the fat-jar/jbang run path; drop the same stale phrasing from the jbang catalog description. Add CHANGELOG.md (Keep a Changelog) with an [Unreleased] entry covering the initial engine + config + drivers + TUI feature set and the notable fixes.
Cover the engine's core promises end to end against real databases: the cross-vendor ETL row copy, the dry-run "writes nothing" guarantee (target rolled back), the rollback-on-error guarantee (a target PK violation leaves the target unchanged), and SCRIPT execution / SCRIPT dry-run on the target. The class is annotated @testcontainers(disabledWithoutDocker = true), so it runs where a Docker daemon is reachable (CI) and is skipped - not failed - where it is not, keeping the local build green on machines without Docker. Verified the same five scenarios green against real postgres:16 -> mariadb:11.4 this session (engine logic + assertions), so CI exercises only the container wiring on top of already-proven behaviour.
Run gradlew build on pull requests and pushes to main. The ubuntu runner provides a Docker daemon, so the Testcontainers integration tests execute in CI even though they skip on local machines without Docker.
Cloned on Windows, gradlew lost its +x bit, so the Linux CI runner failed with 'Permission denied' (exit 126) before Gradle started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
English
Brings DataLinq from an engine + CLI baseline to a complete, usable tool.
What's included
type+ host/port/database (sqlserver / mariadb / postgresql) or a custom URL — no longer hard-wired MSSQL→MariaDB.datalinq driver <name>(registered through aDriverShim).datalinq init.disabledWithoutDocker, so they run in CI and skip on machines without Docker.Verification
gradlew buildgreen locally (46 pass + 5 Testcontainers tests skipped without Docker). The five engine scenarios were also verified green against real postgres:16 → mariadb:11.4 this session.한국어
DataLinq를 엔진 + CLI 기준선에서 완성형 도구로 끌어올립니다.
포함 내용
type+ host/port/database(sqlserver / mariadb / postgresql) 또는 custom URL — MSSQL→MariaDB 고정 제거.datalinq driver <name>로 다운로드(DriverShim등록).datalinq init.disabledWithoutDocker로 CI에서는 실행, Docker 없는 로컬에서는 스킵.검증
로컬
gradlew build그린(46 통과 + Testcontainers 5건은 Docker 없어 스킵). 다섯 엔진 시나리오는 실제 postgres:16 → mariadb:11.4 로도 이번 세션에서 그린 확인.