Group 5 Discussion Thread

[susannah-slv]

A discussion space for group 5.

[Annabel-MQ]

Hi Susannah. Thanks for creating the discussion

[amhanson9]

For our discussion of APIs next time, here are the University of Georgia scripts that make use of APIs. FYI, I use a lot of comments when I code, since I don't work on any given script that frequently and future maintainers will also likely not be professional developers. I hope that might help you get oriented to what each piece of code is trying to accomplish. 

https://github.com/uga-libraries/web-archive-it-api gets metadata from Archive-It (json) and transforms it to different CSVs for reporting and analysis. Look for a get_metadata() function for the part that interacts with the API.

https://github.com/uga-libraries/web-aip gets metadata from Archive-It (json) and uses it to select WARCs and metadata reports to download for ingest into our preservation system. We do this quarterly. Several functions use the API.

https://github.com/uga-libraries?q=ASpace We have 4 repos for ArchivesSpace scripts, which use that API. I did not write these, but can ask my colleague who did if you have questions.

[amhanson9]

Here's the repo about how we use GitHub at the University of Georgia that I mentioned during our first meeting: https://github.com/uga-libraries/GitHub_Standards

[susannah-slv]

Thanks so much for those resources Adriane - it's so great to see such well documented and clear code! I've found the collaborative development workflow helpful to get started on our web api project with my colleagues. - I actually am still getting the hang of Git and forgot to reply and just added a comment, so you may not have seen.

[susannah-slv]

Hi everyone,

I've been working on the API's and have a functioning get request for MMS Ids in Alma - the code is in the public repo for the project: https://github.com/StateLibraryVictoria/metadata-clean-up

Does anyone have any experience using https://pymarc.readthedocs.io/en/latest/? I am looking at this as a next step for parsing the MARC XML. Any advice would be awesome. How is everyone else's work going?

[lars-unimelb]

Hi Susannah, I used to work with marc files back in my old job and I frequently used lxml (https://lxml.de/) to handle the xml from the API. It is a bit of a learning curve, especially working with the different namespaces.

[susannah-slv]

Hi Lars, thanks for the tip I will look into it. :)

[harrylloyd-bl]

Hi Susannah - mentor here, I use PyMARC to handle xmls every now and then, the easiest route I found was with the pymarc.marcxml.parse_xml_to_array function, which returns a pymarc Record. Let me know if you try it/have any questions

[amhanson9]

Just stumbled across a few articles in Code4Lib about using pymarc, that might give you some ideas: https://journal.code4lib.org/?s=pymarc

[susannah-slv]

Thanks everyone for your advice - it turned out I had a bug, where the metadata had some stray unprintable bytes in a field. I could get the root element with lxml, but pymarc would not parse to array. ElementTree's ET.parse() ended up having error messages that was specific enough to figure out what was going on and now I've got MARC running in Python! Super exciting!!

[Annabel-MQ]

Apologies, I cannot make today's meeting. Cheers, Annabel

[amhanson9]

No worries, most people couldn't make it! To summarize: Susannah and I shared our progress. Susannah is working on MARC data cleanup and dealing with encoding errors. I'm doing a proof of concept with topic modelling for describing born-digital archives and am trying to decide what libraries to focus on.

[amhanson9]

March potential topic: unit testing. Based on our February meeting, Susannah was going to try unit tests in Python for next time. Unit tests are a way to automatically test if a function or script is producing the output you expect given different input. It helps you write the script in the first place and then makes it fast to check if future changes have broken anything. If you want to give that a try, here is the tutorial I used to learn them https://realpython.com/python-testing/ and here is my most recent script's set of unit tests: https://github.com/uga-libraries/format-report/tree/main/tests

We can also do more general sharing around whatever project or tutorial you're working on and any questions you have about them.

[Annabel-MQ]

Apologies, I cannot make it to the meeting today. Work committments keeping me away for the next few months, so will withdraw from the group. Best wishes for the continued work on python! Cheers, Annabel

[susannah-slv]

Hey group 5,
An update to say that I've implemented MarcEdit rules validation and MarcBreaker functionality to my scripts in the Metadata Cleanup Project repo. See src/shared_functions.py for the output_file_validation() function which draws on the src/transform_marc_file.py functions break_marc_record() and validate_marc_record() to run cmarcedit.exe. These require the person running the script has MarcEdit installed and have added the path to cmarcedit.exe and the MarcEdit rules file into the .env file.
Currently I think it might save cataloguers a bit of time breaking and validating the records - and I'm toying with the idea of having a function that generates a 'qa' file on a random subset of processed records to make it easier to check if things are functioning as expected.
It's likely very buggy as this is the most complicated bit of code I've written thus far - but thought I'd share after our discussions at the last meeting!