Skip to content

Troubleshooting

Jump to bottom
gomathishankar37 edited this page Jun 30, 2026 · 4 revisions

Troubleshooting

Practical checks aligned with current repo behavior.

1) Nothing uploads

Check for dumps:

ls -lh /minidumps/*.dmp 2>/dev/null
ls -lh /opt/minidumps/*.dmp 2>/dev/null
ls -lh /var/lib/systemd/coredump/* 2>/dev/null

Check services:

systemctl status coredump-upload.service
systemctl status minidump-on-bootup-upload.service
systemctl status minidump-on-bootup-upload.timer

Check logs:

journalctl -u coredump-upload.service -n 100 --no-pager
journalctl -u minidump-on-bootup-upload.service -n 100 --no-pager
tail -n 100 /opt/logs/core_log.txt 2>/dev/null
tail -n 100 /rdklogs/logs/core_log.txt 2>/dev/null

2) Lock contention

C path lock files:

  • /tmp/.uploadMinidumps
  • /tmp/.uploadCoredumps

Inspect and clear stale locks only if no uploader is active:

ls -l /tmp/.uploadMinidumps /tmp/.uploadCoredumps 2>/dev/null
ps -ef | grep -E 'crashupload|uploadDumps|runDumpUpload' | grep -v grep
rm -f /tmp/.uploadMinidumps /tmp/.uploadCoredumps

Use wait_for_lock to avoid immediate lock-exit behavior.

3) Build issues

./cov_build.sh
./cov_build.sh --clean
./cov_build.sh

4) Test issues

./run_ut.sh
./cov_build.sh --l2-test
./run_l2.sh

Expected reports:

  • Unit: /tmp/Gtest_Report
  • L2: /tmp/l2_test_report

5) Binary vs script routing confusion

Verify DEVICE_TYPE and dispatch in uploadDumps.sh:

  • mediaclient: binary-first
  • broadband and extender: legacy path

Check force-legacy marker:

ls -l /tmp/.legacy_crashuploader

6) Rate-limit and cooldown diagnostics

Inspect rate-limit state files:

ls -l /tmp/.deny_dump_uploads_till /tmp/.minidump_upload_timestamps 2>/dev/null
cat /tmp/.deny_dump_uploads_till 2>/dev/null
tail -n 20 /tmp/.minidump_upload_timestamps 2>/dev/null
date +%s

Interpretation:

  • /tmp/.deny_dump_uploads_till is a global cooldown timestamp.
    • If current epoch is less than the file value, both minidump and coredump runs are blocked.
  • /tmp/.minidump_upload_timestamps is the minidump-only rolling counter source.
    • It does not apply to coredump directly, but can trigger global cooldown creation.

Navigation

Clone this wiki locally