Multiview particle stacks - #573
Closed
bHimes wants to merge 12 commits into
Closed
Conversation
…ner.json file with dot prefix
- gpu debugging tools were previously not being enabled b/c the define was only being passed to the host compiler.
- rather than having defines from cistem_config.h (set by AC_DEFINE in config.ac/other .m4) cistem_config is now FORCE included for all builds in config.ac
- the use of many distinct cuda streams left several silent(ish) bugs in the main GpuImage class, these are resolved hear
- to cope with occasional floating point issues, extends Kahan summation beyond the buffered mip stack (~20 mips) to the full inner loop. Costs tracking/loading of 2 additional arrays
- removes histogram.cu which was superceded by template_matching_empirical_distribution a long while ago
# File details
.vscode_shared/CistemDev/c_cpp_properties.json
- sync to config options so profile in vscode dims the appropriate define blocks
.vscode_shared/CistemDev/tasks.json
- adds debug level to config
configure.ac
- adds check that cistem_config.h is force included (ensure all defines are everywere as expected, viz gpu_debug)
- adds option to configure gpu debug level
- renames config define RIGOROUS_SOCKET -> RIGOROUS_SOCKET_CHECK
m4/ax_cuda.m4
- Cuda MUST be > 11 now (previous warn now error)
src/Makefile.am
- ensure build system knows to rebuild when gpu headers are modified (since we have a make rule outside of trad autotools, we previously had to touch a .cu for make to know to rebuild)
src/core/core_headers.h
- Walk -> MyWalk to silence shadowing warnings on build
src/gpu/GpuImage.cu
- fixes outdated assumptions in stream semantics
- postcheck debugs take stream arg
- NppInit and bufferInit now stream ordered (not just cudaStreamPerThread)
- cuFFT checks on set stream and has safety checks now as workspace is not managed and could lead to race conditions
src/gpu/TemplateMatchingCore.cu
- applies updates for stream ordering and clarifies/documents gotchas
- remove test code for debugging and local mip normalization
src/gpu/projection_queue.cuh
- applies updates for stream ordering and clarifies/documents gotchas
- adds saftey checks in destructor to ensure work is done and streams/events okay to be destroyed
src/gpu/template_matching_empirical_distribution.cu
- applies updates for stream ordering and clarifies/documents gotchas'
- extends previous partial kahan summation to be more complete reducing errors in sum/sum sqs calculations at the expense of persiting the sum_error and sum_sq_errors needed beyond the number of images in a stack of mips (20 by default) to the full inner loop now.
- manages tracking of double buffers which had been mixed responsibility with TemplateMatchingCore, now more logical ownership model and safer updating
- Adds multiview parameters to database, and imports/refinement tracking
- particle group, pre/post exposure
- just intended for particles from tilt-series right now, but the same logic would applie to extracting particles from movie frames
- attempts to minimize DB impact by supplementing tables (new table with join) rather than extending existing tables
- imports/package creation now check for these data in starfiles and add or use default values. Option to exclude particles based on total exposure threshold
File details:
src/core/database.cpp
- multiview table (supplements rather than extending directly)
- refinement results had to be extended, writes default values if no multiview data
src/core/database.h
- adds missing include guards
src/core/database_schema.h
- adds missing include guards
- adds REFINEMENT_PACKAGE_CONTAINED_PARTICLES_MULTI_VIEW_ (supplements REFINEMENT_PACKAGE_CONTAINED_PARTICLES_)
- removes duplicate entry for REFINEMENT_PACKAGE_CONTAINED_PARTICLES_
src/core/image.h/cpp
- adds method IsFinite (rather than only checking for nans)
- NOTE: newest intel compiler was adding --fast-math which can invalidate std::isnan, current build system is okay though.
src/core/particle.cpp
- adds multiview parameters to class attributes
- adds method ApplyExposureDecayToSSNRCurve to (experimentally) us in refinment where particles do not yet have exposure filtering
src/core/refinement_package.cpp
- adds multiview params and simple method to see if they are present (ContainsMultiViewData)
updates refinment results writing in ::ProcessJobResult in classes
- src/gui/AbInitio3DPanel.cpp
- src/gui/AutoRefine3dPanel.cpp
- src/gui/MyRefine3DPanel.cpp
- src/gui/ResampleDialog.cpp
-
src/gui/CombineRefinementPackagesWizard.cpp
src/gui/ImportRefinementPackageWizard.cpp
src/gui/MyNewRefinementPackageWizard.cpp
src/gui/MatchTemplatePanel.cpp
- removes temporary disabling of non FastFFT gpu path
MatchTemplateApp::CalcGlobalCCCScalingFactor
- explicitly ignores FFTW padding rather than relying on it being set to zero. (which shoulid have been fine)
src/programs/reconstruct3d/reconstruct3d.cpp
- cleans up usage of multiview parameters and deadcode comments
src/programs/refine3d/refine3d.cpp
- adds experimental filtering of non-expsoure filtered particles during refinment, ifdef'ed out in this commit
…properly maintain in the main repo.
…ound the source, unrelated to changes in this commit
… happend to ensure critical symbols are defined, especially for multicompiler (gpu) codepaths.
…ed the output psi angles and had a small impact on overall avg/std stats images affecting exact peak values.
- Fixes bug in reconstruct3d when applying the exposure filter for multi_view particles, the wrong image was passed when creating the exposure filter, which works much of the time, but can lead to segfaults when the memory sizes do not match. (just an oversight.)
- Changes fixed MIP batch size to depend on image size in TM empirical dist. Should probably also consider the hardware it is running on TODO.
- Removes optional trimming of which mip values were included in the stats images, part of alignment with some of Raisons work. Returns to ALL values tracked.
- Affects TM empirical dist and match_template.cpp
- Removes to else // comments that were blocking formatting in recosntruct 3d. (moved down a line so it becomes else { \n //)
- Adds experimental define in AutoRefine3dPanel for testing skipping the global search (for running autorefine on multi_view/tomo stacks, e.g. automated local refine. "works" but the results are no good, search space must be different thatn running Refine3d manually several times. NOT enabled)
- Adds experimental define in MatchTemplatePanel to allow an iterative run easily for a fixed set of resolutions. Disabled here, but leaving as it is a nice idea for how we might easily include other batched experiments, e.g. over multiple templates. #define BATCH_HIGH_RES_EXPERIMENT
- Removes github copilot configs from settings
- CHANGES behavior of TM results to never ignore pixels near the edge. There are too many complications with binning and it was already a bit of a tenuous topic anyway. (The SNR values should be too low close to the edge so the are hard to compare to other peaks even if valid in their own right. We ignored them so end users wouldn't have to consider this, but it is hopefully not a big deal)
- removed config for this in the test utils: scripts/testing/programs/cistem_test_utils/args.py, scripts/testing/programs/cistem_test_utils/make_tmp_runfile.py
- removed spec in constants.h
- removed option and action in src/programs/make_template_result/make_template_result.cpp, refine_template, prepare_stack_matchtemplate
- Removes no longer used Sum2/SumSq2 arrays in TemplateMatchingCore - these were part of the "cascading sum" to help prevent numerical error with the stats arrays and half precision, but we are now using Kahan summation and single precision
…lean approach setup here that I do in downstream cisTEMx
…andle peak extraction and processing for tm, tm stack, refine tm. However I also ended up adding a new method that is orderS of magnitude faster for finding peaks to the image class, and that scrambled some things. Resampling in that method is untested and currently working around the extraction/projection of results in match template. Need to entirely replace the GetNextPeak method and just pass it the peak list for post processing.
… template results, refine template and prepare stack match templates (the latter two with both images and coord files) and then also peak upsampling needs to be debugged. DO NOT USE THIS COMMIT for work.
Collaborator
Author
|
superceded by #575 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Summary
Details
More file based details in the commits.
These are mainly for tilt-series but could be used for movies as well. They are generally speaking needed for cases where we cannot apply exposure filtering until reconstruction. They would also allow better constraints during refinment, however, that is beyond the scope of the current work.
This is lower level stuff not worth detailing here. Extensive doxygen docs added.
I had previously moved many of our manual define (-Dsymbol) to happen with config.ac which places defines in a build/dir/cistem_config.h and then this header was included in core_headers. Given the complexity of the build process, I decided it is easier to force include this header for all compilers so it is present in all compilation units in the project. It is lightweight. Also added a extra redundant check to ensure this is included everywhere based on a required symbol defined in config.ac.
The newer "skills" arch from Anthropic fully supplants the nested CLAUDE.md structure and given it could all change in two months again anyway, I elected to keep the tooling local to my repos and not add technical debt to the main repo.
I have rebased my feature branch to be current with the master branch using to minimize conflicts and headaches
Which compilers were tested
These changes are isolated to the
How has the functionality been tested?
Please describe the tests that you ran to verify your changes. Please also note any relevant details for your test configuration.
Checklist: