demo working ds1 refinement through annulus worker

[dwpaley]

No description provided.

[nksauter]

Let's leave this as a pull request for now pending the following:

1. Use the originally simulated images with mosaic spread full width=0.08 degrees
2. Make sure the ds1 analysis is scaled up to 100000 images.
3. provide LBFGS target functional, parameters, and gradient to stdout
4. check DS1 timings per image.

[dwpaley]

Nick, some updates:

1. The sim in sim_dials_ds1.sh now uses 25 mosaic domains.
2. Currently expect to process 100k images in 90 minutes on 100 nodes. See point 4.
3. Functional, gradient magnitude, and refined parameters are now written to stderr. This branch now also requires branch exercise_ds1 for both cctbx_project and psii_spread.
4. On 1 Perlmutter node, we now process 100 images in 9 minutes using the annulus worker running ds1. Part of the long-ish time is from poor convergence when working on the mosaic images. When the simulation is with MOS_DOM=1 and oversample=4, then we process 100 images in 5 minutes.

[nksauter]

Dan, thanks, this looks great. This seems like a big step forward. It is a lot to take in so it will take me some time to look it over. One thing I'd like to understand is whether the branch to cctbx_project is really necessary. I've spoken to Derek in the past about redirecting the logger to stdout, and he has a way of doing it. That is, without modifying the library code.

[dwpaley]

The cctbx_project branch is necessary to add logging of refined parameters while computing the functional and gradients. I think this has to be done where it is because the loop over refinement cycles is in Scipy code and we just provide the target function. I haven't discussed this with @dermen though.

The logging is pretty rough at present. I think it gets configured and re-configured repeatedly. It will have to be cleaned up. For now the ds1 per-cycle output ends up in out/rank_xx.err.

[dwpaley]

Sorry, I misunderstood your question about the cctbx branch. The purpose is to add logging of parameter values using the existing logger. I did seem to recall using some similar functionality a while ago, but I couldn't find it. @dermen can you weigh in? The code in question is here: cctbx/cctbx_project@9012469

[dermen]

Before hopper_utils.py existed, there was a different diffBragg refinement script (at the python level) that logged parameters at each iteration.. @dwpaley perhaps you recall that.

If this is just a way to optionally dump the parameters to the refinement debug log, it looks good. Maybe we could add customizable print formatters to each parameter (that would be added in diffBragg/refiners/parameters.py, and setup in e.g. hopper_utils.py) but for now this looks ok. Is the current use-case to write these logs to files, optionally in an MPI setting ?

[dwpaley]

Hi @dermen I'm just getting back to this after a couple hectic beamtimes. I think you're right about all of this. The logging that I remember must have been in the old script. I agree that custom print formatting would be nice but this already gets us most of what we need. Yes currently these are written to files, which I think is setup by calling mpi_logger.setup_logging_from_params(self.params.diffBragg) in the client code.