BackStats crash

[kslong]

Although run the current version of BackStats on many datasets it is failing on LMC_c50 T01. It fails in a way that destroys the terminal on my linux box. The last thing printed out is the following:

Starting c4d_250123_034312_ooi_N673_v1_S4.fits  with 31 matches
BackStats: Finished  27 x-matches for  DECam_BACK/LMC_c50/T01/c4d_250123_034312_ooi_N673_v1_S4.fits 
Starting c4d_250123_034312_ooi_N673_v1_S5.fits  with 38 matches
BackStats: Finished  33 x-matches for  DECam_BACK/LMC_c50/T01/c4d_250123_034312_ooi_N673_v1_S5.fits 
Starting c4d_250123_034312_ooi_N673_v1_S6.fits  with 19 matches
BackStats: Finished  17 x-matches for  DECam_BACK/LMC_c50/T01/c4d_250123_034312_ooi_N673_v1_S6.fits 
<string>:1: SyntaxWarning: invalid decimal literal
<string>:1: SyntaxWarning: invalid escape sequence '\�'
<string>:1: SyntaxWarning: invalid escape sequence '\�'
<string>:1: SyntaxWarning: invalid decimal literal

Unlike most errors in python there is no traceback. The error occurs with and without multiprocessing.

I believe I have seen these error messages on other datasets, but they have not completely crashed the system, though I have noticed that the number of processing threads drops to 1 for awhile.

[kslong]

This issue turned out to be complicated. There were multiple issues.

The Syntax warning errors seemed to be due various combinations of the file system not being completely synchronized before file was read, and memory issues associated with using eval

The actual crash appears to have been due to a known bug in astropy join for large tables, specifically a memory leak

All this is took place toward the end of BackStat.py where the final output files were being prepped.

I have modified BackStats.py in ways that avoid this problems, at least for LMC_c50 T01, and run the revised version of the progam on LMC_c50 and 51 all tiles.

The currrent of BackStats contains many more diagnostic messages that should make it easier to diagnose the problem if it reappears.

I am attaching my chat with "Claude", which may be helpful if the problem reappears

Note that part of the recommendation was to use pandas for tables instead of astropy.tables, but I have not made such a change.

Astropy Table Writing Syntax Warnings - Claude.pdf

[kslong]

This problem reappeared this week while I was attempting to process the 2025 OIII data. While running BackStats, typically, overnight, I come back in the morning to find either Gnome has crashed or my linux box has rebooted. Inspection of the system logs suggests that the crashes appear during times linux is doing various automatic system activities, and that the Gnome crashes and system reboots are initiated not from within BackStats itself. Typically, I run BackStarts with np 8 (on my 24 processor machine). I found that reducing np to 4 seemed to help. My suspicion, if this needs to be followed up in future, is that the reason BackStars is the culprit is associated with the fact that it opens so many fits images, and in some cases is opening the same image at the same time from two different threads.

If this is some kind of system interaction, it might explain why not anyone else has seen this, and I have not seen it on my Mac

If anyone else notices, kred just going away, either causing a system reboot or terminal crash, please look in the kred log files, to see what was happening when the crash occurred, and note the time to see if is only occurring at a specific time of day or night. In ubuntu apparently, the default time to do automatic system maintenance is 11pm, and this is when most of my problems were occurring.