We've been testing ndreg as an alternative registration package for our Clarity imaging efforts, but we're running in to an issue with the registration failing due to an OOM error in what seems to be the last stage of the registration.
The tutorial / example data provided runs fine - this is failing on our data. The first several steps seems to complete as expected:
Step 0: alpha=0.05, beta=0.05, scale=0.0625
E, E_velocity, E_rate, E_image (E_image %), LearningRate
0. 26814.1, 62.2496, 0, 26751.8 (87.9049%), 5.500000e-03
1. 26397.5, 78.2021, 0, 26319.3 (86.4834%), 1.512500e-03
2. 25807.2, 95.4081, 0, 25711.7 (84.4872%), 1.663750e-03
3. 24154.6, 121.688, 0, 24032.9 (78.9707%), 1.830125e-03
4. 22683.3, 155.63, 0, 22527.7 (74.0246%), 2.013138e-03
5. 20909.5, 202.948, 0, 20706.6 (68.0405%), 2.214451e-03
6. 20573.7, 233.436, 0, 20340.3 (66.837%), 1.217948e-03
7. 20334.7, 268.579, 0, 20066.1 (65.9359%), 1.339743e-03
8. 19753.9, 289.534, 0, 19464.4 (63.9586%), 7.368587e-04
9. 19294.7, 312.791, 0, 18981.9 (62.3734%), 8.105445e-04
10. 19167.6, 326.595, 0, 18841 (61.9104%), 4.457995e-04
11. 19000.9, 341.867, 0, 18659.1 (61.3125%), 4.903794e-04
12. 18915.4, 359.172, 0, 18556.3 (60.9747%), 5.394174e-04
13. 18887.1, 378.791, 0, 18508.3 (60.8171%), 5.933591e-04
14. 18867.5, 400.87, 0, 18466.6 (60.6802%), 6.526950e-04
15. 18848, 425.853, 0, 18422.1 (60.534%), 7.179645e-04
16. 18847.8, 454.67, 0, 18393.1 (60.4386%), 7.897610e-04
E = 18847.8 (60.4386%)
Length = 20.1142
Time = 8.62127s (0.143688m)
Step 1: alpha=0.05, beta=0.05, scale=0.125
E, E_velocity, E_rate, E_image (E_image %), LearningRate
0. 28493.2, 65.7145, 0, 28427.5 (97.5929%), 5.500000e-03
1. 27603.1, 59.5668, 0, 27543.6 (94.5585%), 7.562500e-04
2. 27411.8, 56.9109, 0, 27354.9 (93.9108%), 8.318750e-04
3. 27277, 57.2933, 0, 27219.7 (93.4468%), 9.150625e-04
4. 27067.2, 61.968, 0, 27005.2 (92.7103%), 1.006569e-03
5. 26750.5, 72.5735, 0, 26678 (91.5868%), 1.107226e-03
6. 26592.5, 87.0846, 0, 26505.4 (90.9945%), 1.217948e-03
7. 26565.4, 102.462, 0, 26463 (90.8487%), 1.339743e-03
8. 26564.9, 110.63, 0, 26454.3 (90.819%), 7.368587e-04
9. 26562.8, 120.35, 0, 26442.4 (90.7782%), 8.105445e-04
10. 26420, 131.428, 0, 26288.6 (90.25%), 8.915990e-04
11. 26406.2, 134.154, 0, 26272.1 (90.1934%), 2.451897e-04
12. 26402.6, 134.51, 0, 26268.1 (90.1797%), 3.371359e-05
13. 26400.8, 134.707, 0, 26266.1 (90.1729%), 1.854247e-05
14. 26400.8, 134.71, 0, 26266.1 (90.1729%), 3.186987e-07
15. 26400.8, 134.71, 0, 26266.1 (90.1729%), 2.191054e-08
16. 26400.8, 134.71, 0, 26266.1 (90.1729%), 1.205080e-08
17. 26400.8, 134.71, 0, 26266.1 (90.1729%), 6.627938e-09
E = 26411.1 (90.2084%)
Length = 19.269
Time = 9.69856s (0.161643m)
Step 2: alpha=0.05, beta=0.05, scale=0.25
E, E_velocity, E_rate, E_image (E_image %), LearningRate
0. 26447.6, 15.0638, 0, 26432.6 (94.9078%), 5.500000e-03
1. 26355.9, 14.3033, 0, 26341.6 (94.581%), 3.781250e-04
2. 26279.6, 14.1924, 0, 26265.5 (94.3077%), 4.159375e-04
3. 26159.6, 14.7062, 0, 26144.9 (93.8748%), 4.575313e-04
4. 26040.3, 15.8101, 0, 26024.5 (93.4425%), 5.032844e-04
5. 25829.6, 17.3934, 0, 25812.2 (92.6803%), 5.536128e-04
6. 25683.2, 19.2385, 0, 25663.9 (92.1479%), 6.089741e-04
7. 25631.8, 21.1908, 0, 25610.6 (91.9566%), 6.698715e-04
8. 25623.7, 23.3273, 0, 25600.4 (91.9197%), 7.368587e-04
9. 25623.7, 23.3942, 0, 25600.3 (91.9194%), 2.532952e-05
10. 25623.7, 23.4126, 0, 25600.3 (91.9193%), 6.965617e-06
11. 25623.7, 23.4128, 0, 25600.3 (91.9193%), 5.986077e-08
12. 25623.7, 23.4128, 0, 25600.3 (91.9193%), 1.646171e-08
E = 25623.7 (91.9193%)
Length = 8.34867
Time = 10.7544s (0.17924m)
Step 3: alpha=0.05, beta=0.05, scale=0.5
E, E_velocity, E_rate, E_image (E_image %), LearningRate
0. 35048.8, 7.49631, 0, 35041.3 (99.3777%), 5.500000e-03
1. 34209.6, 4.8356, 0, 34204.7 (97.005%), 3.025000e-03
2. 34100, 7.29589, 0, 34092.7 (96.6872%), 1.663750e-03
3. 34081.6, 8.84533, 0, 34072.8 (96.6308%), 1.830125e-03
4. 34053.3, 11.4954, 0, 34041.8 (96.543%), 2.013138e-03
5. 33971, 14.1881, 0, 33956.8 (96.3019%), 2.214451e-03
6. 33951, 18.4157, 0, 33932.6 (96.2331%), 2.435896e-03
7. 33867.6, 22.7148, 0, 33844.9 (95.9844%), 2.679486e-03
8. 33836.3, 26.0582, 0, 33810.3 (95.8864%), 1.473717e-03
9. 33792.9, 29.1741, 0, 33763.7 (95.7543%), 1.621089e-03
10. 33760.4, 33.28, 0, 33727.1 (95.6506%), 1.783198e-03
11. 33718.6, 37.7322, 0, 33680.9 (95.5193%), 1.961518e-03
12. 33696.3, 43.4092, 0, 33652.9 (95.44%), 2.157670e-03
13. 33651.8, 49.7127, 0, 33602.1 (95.296%), 2.373436e-03
14. 33636.2, 57.863, 0, 33578.3 (95.2285%), 2.610780e-03
15. 33522.6, 66.4359, 0, 33456.2 (94.8822%), 2.871858e-03
16. 33522.6, 66.4386, 0, 33456.2 (94.8822%), 7.712510e-07
E = 33522.6 (94.8822%)
Length = 11.1824
Time = 41.7592s (0.695987m)
Step 4: alpha=0.05, beta=0.05, scale=1.0
E, E_velocity, E_rate, E_image (E_image %), LearningRate
0. 31299.2, 1.72881, 0, 31297.4 (97.985%), 2.750000e-03
1. 31128.1, 0.965935, 0, 31127.2 (97.4519%), 1.512500e-03
2. 30994.7, 1.89466, 0, 30992.8 (97.0314%), 1.663750e-03
3. 30984.2, 2.91225, 0, 30981.3 (96.9954%), 1.830125e-03
4. 30983.3, 3.05556, 0, 30980.2 (96.9919%), 2.516422e-04
5. 30983.3, 3.05556, 0, 30980.2 (96.9919%), 1.689492e-08
E = 30983.3 (96.9919%)
Length = 2.2311
Time = 155.781s (2.59635m)
However, after Step 4 the memory usage blows up (during steps 0-4 monitoring with docker stats memory usage stays below 4GB, but then balloons to use all available memory, >130GB) and the process is killed.
This is being run on Windows 10.
We've been testing ndreg as an alternative registration package for our Clarity imaging efforts, but we're running in to an issue with the registration failing due to an OOM error in what seems to be the last stage of the registration.
The tutorial / example data provided runs fine - this is failing on our data. The first several steps seems to complete as expected:
However, after Step 4 the memory usage blows up (during steps 0-4 monitoring with
docker statsmemory usage stays below 4GB, but then balloons to use all available memory, >130GB) and the process is killed.Full output from
dmesgis available hereDocker Info:
Version 2.0.0.3 (31259)
Channel: stable
Build: 8858db3
This is being run on Windows 10.