SVM-Based Glitch Detection and Interval Removal

[katefoyle]

This pull request transforms the glitch detection pipeline into a machine learning approach (with SVM classifier) capable of finding instrumental glitches at lower S/N thresholds and removes only those with glitch likelihoods that exceed a specified threshold (set to .45).

Key Changes

Added SVM glitch detection:

• Added SVM_dpi_eval function to analyze 64ms DPI segments and flag likely glitches
• Lowered SNR candidates ((stds > 6) and (stds / np.abs(stds2)) > 2) (previously hard-coded to 10 and 3)
• Glitch probabilities of candidates logged
• Those with glitch likelihood > .45 get flagged and their interval removed from event GTI

Interval handling:

• Updated get_btis_for_glitches to set customizable S/N thresholds
• t_min, t_max tuples are returned and used to update GTI

Code integration:

• Imported pickle
• SVM loaded from config
• Glitch likelihood logged

Why?

• Lowering the heat pipe glitch parameters allows for more to be detected by the SVM, which has proven highly effective in separating these types of glitches from GRBs, SGRs, and plain background.
• More rejection lowers overall noise at any S/N, increasing the sensitivity to GRBs in ground-based searches.
• Avoids data loss caused by previously hard-coded glitch parameters.

@jjd330 to review

[jak574]

Something seriously screwy with this PR, it seems like every file is completely changed. This can't be right. Can you fix this?

[jjd330]

Looks like your text editor (VS code) may have changed the format of your line endings, making it look like every file got changed.

Try changing your "Eol" setting in VS code to "\n" (or it could be called "LF"). Hopefully that's enough to fix it, if not you may need to edit your Git settings, as is shown in the first answer here https://stackoverflow.com/questions/74936583/git-nothing-changed-but-mark-every-line-changed

[jak574]

Looks like your text editor (VS code) may have changed the format of your line endings, making it look like every file got changed.

Try changing your "Eol" setting in VS code to "\n" (or it could be called "LF"). Hopefully that's enough to fix it, if not you may need to edit your Git settings, as is shown in the first answer here https://stackoverflow.com/questions/74936583/git-nothing-changed-but-mark-every-line-changed

Yep, that's it. Another solution would be to run the unix dos2unix tool on everything to convert back.

[Tohuvavohu]

@jjd330 can you link any more info about the approach, how it works, and what the yield is?

[katefoyle]

@jak574 @jjd330 @Tohuvavohu Thanks, I’ll fix this when I’m in tomorrow!

[katefoyle]

@Tohuvavohu I have my repo about the SVM approach here if this’ll help: https://github.com/katefoyle/SWIFT-BAT-Machine-Learning-Glitch-Detections

[jjd330]

Thanks for linking your other repo! Do you have a link to your slides that you showed on the call a few weeks ago too? I'm pretty sure that showed the gain in catching weaker glitches and less false alarms on things like SGRs.

[katefoyle]

https://1drv.ms/p/c/54477a59e4de02c6/Ed5XSQHFib1NrH0bD6sMITwB1LMLQ5vuWKoT4TAvatXpiA?e=cVWdhu https://1drv.ms/p/c/54477a59e4de02c6/ETMVFzi82w5MvbrBD0wwSLsB6I45Ynis67rfyAZpTYiRJA?e=7U1tqG first link is the most recent recap presentation when I finished finding the SVM threshold, second link is the overview of the entire project