Cluster helper doesn't preserve core dumps

[locker]

On my setup, core dumps are written to the file core in the current directory:

$ cat /proc/sys/kernel/core_pattern
core

Turns out the luatest.cluster simply removes all server directories, no matter if the test failed or passed, including all core dumps.

To reproduce the issue, apply the following diff:

diff --git a/test/cluster_test.lua b/test/cluster_test.lua
index 54331cdb2e11..e5ed0bda8f84 100644
--- a/test/cluster_test.lua
+++ b/test/cluster_test.lua
@@ -65,6 +65,12 @@ g.test_start_stop = function()
     assert_instance_is_ro(c, 'instance-y1', false)
     assert_instance_is_ro(c, 'instance-y2', true)
 
+    c['instance-x1']:exec(function()
+        local ffi = require('ffi')
+        local p = ffi.new('int[?]', 1)
+        p[123456] = 123
+    end)
+
     c:stop()
 
     assert_instance_stopped(c, 'instance-x1')

(applied on top of 09a6276)

Then run the test:

bin/luatest -l /tmp/luatest.log -p start_stop test/cluster_test.lua

The test fails, which is expected:

$ bin/luatest -l /tmp/luatest.log -p start_stop test/cluster_test.lua
Tarantool version is 3.8.0-entrypoint-43-g7430a8f64a5e
started logging into a pipe, SIGHUP log rotation disabled
E.

Tests with errors:
------------------

1) cluster.test_start_stop
{
    base_type = "ClientError",
    code = 77,
    message = "Peer closed",
    name = "NO_CONNECTION",
    trace = {},
    type = "ClientError",
}
stack traceback:
        builtin/box/net_box.lua:949: in function '_request'
        builtin/box/net_box.lua:996: in function 'eval'
        /home/vlad/src/tarantool/luatest/test/cluster_test.lua:68: in function 'cluster.test_start_stop'
artifacts:
        instance-y2 -> /tmp/t/artifacts/instance-y2-GxgNxlTU_8sb
        instance-x2 -> /tmp/t/artifacts/instance-x2-0bga8GJZikTU
        instance-y1 -> /tmp/t/artifacts/instance-y1-hcyi-ZJU24Ls
        instance-x1 -> /tmp/t/artifacts/instance-x1-jfcK2A4LLmDf
Ran 2 tests in 3.180 seconds, 1 succeeded, 1 errored, 9 not selected

Failed tests:

cluster.test_start_stop

If you check the logs, you'll see that a server instance crashed:

instance-x1 | Segmentation fault
instance-x1 |   code: SEGV_MAPERR
instance-x1 |   addr: 0x40647a98
instance-x1 |   context: 0x5dd62eb887c0
instance-x1 |   siginfo: 0x5dd62eb888f0
instance-x1 |   rax      0x40647a98         1080326808
instance-x1 |   rbx      0x418a2544         1099572548
instance-x1 |   rcx      0x4                4
instance-x1 |   rdx      0x7b               123
instance-x1 |   rsi      0xf436fff5fff7f02  1099845876626456322
instance-x1 |   rdi      0x418ad678         1099617912
instance-x1 |   rsp      0x72e542c80260     126328993481312
instance-x1 |   rbp      0x405cda00         1079826944
instance-x1 |   r8       0x405cdda8         1079827880
instance-x1 |   r9       0x2                2
instance-x1 |   r10      0x0                0
instance-x1 |   r11      0xfffffffc         4294967292
instance-x1 |   r12      0x0                0
instance-x1 |   r13      0x0                0
instance-x1 |   r14      0x418a30a8         1099575464
instance-x1 |   r15      0x5dd61bbe832a     103174169854762
instance-x1 |   rip      0x5dd61bc6cb28     103174170397480
instance-x1 |   eflags   0x10246            66118
instance-x1 |   cs       0x33               51
instance-x1 |   gs       0x0                0
instance-x1 |   fs       0x0                0
instance-x1 |   cr2      0x40647a98         1080326808
instance-x1 |   err      0x6                6
instance-x1 |   oldmask  0x0                0
instance-x1 |   trapno   0xe                14
instance-x1 | Current time: 1780994171
instance-x1 | Please file a bug at https://github.com/tarantool/tarantool/issues
instance-x1 | Attempting backtrace... Note: since the server has already crashed, 
instance-x1 | this may fail as well
instance-x1 | #1  0x5dd61bb2e1be in crash_collect+256
instance-x1 | #2  0x5dd61bb2ebfa in crash_signal_cb+100
instance-x1 | #3  0x72e545845330 in __sigaction+80
instance-x1 | #4  0x5dd61bc6cb28 in lj_cconv_ct_ct+2009
instance-x1 | #5  0x5dd61bc706d3 in lj_cconv_ct_tv+4451
instance-x1 | #6  0x5dd61bc74b49 in lj_cdata_set+1493
instance-x1 | #7  0x5dd61bbe84a9 in lj_cf_ffi_meta___newindex+383
instance-x1 | #8  0x5dd61bb9dc97 in lj_BC_FUNCC+70
instance-x1 | #9  0x5dd61bba0f9d in jit_secure_call+622
instance-x1 | #10 0x5dd61bbab449 in lua_call+272
instance-x1 | #11 0x5dd61bab1b6a in execute_lua_eval+289
instance-x1 | #12 0x5dd61bb9dc97 in lj_BC_FUNCC+70
instance-x1 | #13 0x5dd61bbab818 in lua_pcall+969
instance-x1 | #14 0x5dd61bb0a9dc in luaT_call+45
instance-x1 | #15 0x5dd61bab2a8d in box_process_lua+476
instance-x1 | #16 0x5dd61bab2c47 in box_lua_eval+89
instance-x1 | #17 0x5dd61ba0b908 in box_process_eval+306
instance-x1 | #18 0x5dd61b86be71 in tx_process_call(cmsg*)+473
instance-x1 | #19 0x5dd61bb44680 in cmsg_deliver+52
instance-x1 | #20 0x5dd61bb45810 in fiber_pool_f+817
instance-x1 | #21 0x5dd61b85715c in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*)+34
instance-x1 | #22 0x5dd61bb3b1be in fiber_loop+219
instance-x1 | #23 0x5dd61bf1ca28 in coro_init+118

However, there's no core dump anywhere in /tmp, i.e. the following command returns nothing:

find /tmp -name core 2>/dev/null

Actually, if you run the command while the test is running, you'll see that there was a core dump that existed for a brief moment in time:

$ find /tmp -name core 2>/dev/null
/tmp/Wjzrsg/core

But it was removed when the test ended:

$ grep '/tmp/Wjzrsg' /tmp/luatest.log
luatest | 2026-06-09 11:36:10.901 [1607313] main/104/luatest/luatest.log treegen.lua:55 I> Creating a directory: /tmp/Wjzrsg
luatest | 2026-06-09 11:36:10.902 [1607313] main/104/luatest/luatest.log treegen.lua:58 I> Writing a content: /tmp/Wjzrsg/config.yaml
luatest | 2026-06-09 11:36:14.048 [1607313] main/104/luatest/luatest.log treegen.lua:110 I> Recursively removing: /tmp/Wjzrsg

Shouldn't it be saved to artifacts on failure? Also, shouldn't we use the VARDIR (/tmp/t) for storing all files created in tests?

[locker]

Root cause

Two different directories are in play for each instance, and the core dump lands in the one that isn't archived:

1. chdir — the process's actual working directory. For cluster instances, Cluster:new() sets chdir = dir where dir is the treegen temp dir (/tmp/Wjzrsg), and spawns the process with that cwd (server.lua:457).
2. workdir — defaults to <vardir>/<id> (/tmp/t/instance-x1-…) since the cluster never sets it (server.lua:264-266). This is the only directory Server:save_artifacts() copies (server.lua:534: fio.copytree(self.workdir, self.artifacts)).

With a relative core_pattern (core), the kernel writes the core into the process cwd = chdir = treegen dir. The cluster config sets no process.work_dir, so the instance never chdirs away — its snapshots, xlogs, and the core all land in the treegen dir, while self.workdir stays essentially empty. So:

• save_artifacts() archives the wrong (near-empty) directory — the core is never collected.
• treegen.clean() (treegen.lua after_all_preloaded hook) does fio.rmtree(dir) on the treegen dir, deleting the core. That's exactly the Recursively removing: /tmp/Wjzrsg line in the log.

Why a plain server:new() doesn't have this bug: the default instance script runs box.cfg{work_dir = TARANTOOL_WORKDIR}, which chdirs the process into workdir. So the core lands in workdir and is archived. The cluster path breaks the invariant that chdir/effective-cwd == workdir.

On the two questions

Saving the core to artifacts on failure — the fix is to make the directory the instance actually runs in (and writes its core into) the same one save_artifacts() copies. The cleanest options:

• A. Give each cluster server its own per-instance dir and set both chdir and workdir to it (currently all instances share one chdir, which is also why the core's owning instance is ambiguous). This keeps save_artifacts() unchanged and gives one artifacts dir per instance, core included.
• B. Less invasive but messier: have save_artifacts() (or the cluster's drop) also sweep chdir for a core file. Avoid — it papers over the chdir ≠ workdir mismatch.

Note a hook-ordering subtlety: artifacts are saved synchronously in runner.lua save_failed_test_artifacts during update_status, which runs well before the after_all treegen.clean(), so copying-then-cleaning is safe as long as the copy source is correct.

Using VARDIR (/tmp/t) instead of TMPDIR — same underlying problem from a different angle. treegen.prepare_directory() uses fio.tempdir(), which follows TMPDIR and scatters everything into /tmp/… instead of under Server.vardir. Rooting treegen dirs under VARDIR would (a) keep all test files in one collectable place and (b) make leftover cores land somewhere predictable even before the per-instance fix lands.

Recommendation

Option A + rooting treegen under VARDIR addresses both questions properly. This touches cluster.lua (per-instance dirs), treegen.lua (base dir + the shared chdir), and possibly the artifacts prefix logic — a real design change, not a one-liner.

---

_{🤖 Analysis by Claude (Claude Code)}