get_symbolic_mapping example code doesn't work; how to download rmaps?

[mperrin]

Hi, I'm trying to learn to use crds for various queries about reference files, and have ran into a small issue with example code from the docstring. This is quite likely due to some ignorance or missed step on my part, so I'll be grateful for any pointers. Here's what I'm trying, from the docstring:

    >> get_symbolic_mapping("jwst-miri-flat-2015-01-01T00:20:05")
    ReferenceMapping('jwst_miri_flat_0012.rmap')

When I try that, I get a FileNotFound error that that rmap is not in my local cache:

FileNotFoundError: [Errno 2] No such file or directory: 'crds_cache/mappings/jwst/jwst_miri_flat_0012.rmap'

The error persists even if I change the hard-coded date in the example code to some more recent date (e.g. after launch and commissioning). It only works if I use a mapping name chosen such that the relevant rmap file is already in my local cache.

What would be the necessary sequence of steps to ask crds to download the past rmap file, if it's not already in the cache?

The task I'm ultimately trying to do is going to involve scripting checks for changes in reference files over time, so being able to say e.g. "what was the active reference file for instrument X, mode Y, on date Z" is exactly what I'd like to be able to do. Thanks in advance for any tips!

[hbrown-ST]

Hello Marshall, first thing I would ask you to check is that your CRDS_SERVER_URL is defined. You can do that with

$ echo $CRDS_SERVER_URL

or running

crds list --status.

It should be 'https://jwst-crds.stsci.edu', so if it is not you can set it with

$ export CRDS_SERVER_URL=https://jwst-crds.stsci.edu

If you had to follow these steps, you can try again and it should work. However it does not, I would suggest running the following command and posting what you get back:

$ crds sync -v --files jwst_miri_flat_0012.rmap

[mperrin]

Thanks much. Yes the $CRDS_SERVER_URL is defined and has the expected value. And FYI, running pipeline reductions of science data work as expected; the pipeline automatically retrieves mappings and reference files as needed. However, even though that variable is defined, it doesn't seem like the get_symbolic_mapping function invokes whichever bit of code would retrieve and cache that file.

The CRDS sync command did work to download that one particular file. So that works for cases in which I know the desired filename ahead of time. However that doesn't work for the more general case where I don't know the rmap filename ahead of time.

Maybe there is another better way to do the type of search task I'm trying to do, which is to e.g. find a list of all available (current and past) reference files for a given instrument, mode, and reference file type. On the CRDS website this is easy: For example one can select the miri section, then the flat reference file type, and get a list of all available reference files of that type. And the Search box allows easily filtering that list by filter or detector, for instance. I would like to be able to script similar retrievals of calibration file lists, and had hoped that the crds package would provide some straightforward interface for doing so. But I have not yet been able to identify a simple interface for doing so via the documented portions of the API. Thanks for any advice.

[hbrown-ST]

What would likely be helpful for at least checking other files you don't know ahead of time is to run

crds sync --all

to set your cache. This would ideally handle any file you choose to input for get_symbolic_mapping. What is likely the case as you pointed out is that get_symbolic_mapping does not perform the cache sync for the file you are checking implicitly.

As for your objective, you might try using

crds.api.get_reference_names('<your pmap or rmap here>')

If you enter a pmap value, you will get the entire mapping structure back, which could indicate the rmap you need to know. If you already know it, you can run for example

crds.api.get_reference_names('jwst_miri_flat_0012.rmap')
['jwst_miri_flat_0006.fits', 'jwst_miri_flat_0007.fits', 'jwst_miri_flat_0008.fits', 'jwst_miri_flat_0009.fits', 'jwst_miri_flat_0010.fits', 'jwst_miri_flat_0011.fits', 'jwst_miri_flat_0012.fits', 'jwst_miri_flat_0013.fits', 'jwst_miri_flat_0014.fits', 'jwst_miri_flat_0015.fits', 'jwst_miri_flat_0023.fits', 'jwst_miri_flat_0024.fits', 'jwst_miri_flat_0025.fits', 'jwst_miri_flat_0026.fits', 'jwst_miri_flat_0027.fits', 'jwst_miri_flat_0028.fits', 'jwst_miri_flat_0029.fits', 'jwst_miri_flat_0030.fits']

I would imagine you would know the pmap from the date you want to use, pull out the relevant rmap based on the instrument and mode you are using, and then could get all of the references you are looking for.

[hbrown-ST]

Just curious if you have had a chance to try this out Marshall.