Instructions for uploading V3 atlases to AWS

[IgorTatarnikov]

Before submitting a pull request (PR), please read the contributing guide.

Please fill out as much of this template as you can, but if you have any problems or questions, just leave a comment and we will help out :)

Description

What is this PR

• Bug fix
• Addition of a new feature
• Other

Why is this PR needed?
brainglobe-atlsapi V3 uses AWS to host the atlases. The current instructions refer to GIN as the remote source.

What does this PR do?
Adds basic instructions for uploading new altases to V3.

How has this PR been tested?

Commands ran locally.

Checklist:

• The code has been tested locally
• The documentation has been updated to reflect any changes
• The code has been formatted with pre-commit

[PolarBean]

When following the instructions

aws login

returns

No AWS region has been configured. The AWS region is the geographic location of your AWS resources.

If you have used AWS before and already have resources in your account, specify which region they were created in. If you have not created resources in your account before, you can pick the region closest to you: https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html.

You are able to change the region in the CLI at any time with the command "aws configure set region NEW_REGION".

The instructions should let the user know the best response

[PolarBean]

(base) harrycarey@snowy:~/brainglobe_workingdir/hoops_dragon/brainglobe-atlasapi$ aws s3 sync --dry-run . s3://brainglobe/atlas

aws: [ERROR]: An error occurred (ParamValidation): Unknown options: --dry-run

[PolarBean]

the cli is apparently dryrun with no hyphen

[PolarBean]

using dot is so dangerous lol, we're going to get a lot of people uploading their home directories. can we ask people to spell out the directory they want to upload

[PolarBean]

ok something im confused about. say we have the allen which is generated at 10, 25, 50, and 100. in the 2022 version some of these meshes are reused across all resolutions and some are generated at each resolution.

Do we store meshes for each resolution?

Would i combine all these generated folders into one before uploading?

Is there an order i should follow to copy them into the same directory, for instance so that anything overwritten is overwritten with the highest resolution copy of that file?

[adamltyson]

Do we store meshes for each resolution?

The meshes don't have a resolution in the same way that the images do (they're in real space). In principle I think we should probably only generate them from the highest resolution atlas?

[IgorTatarnikov]

I agree with @adamltyson, the intention would be that meshes are generated once, at the highest resolution.

The wrapup function should be able to take a multi-resolution set of templates/annotations and run the mesh generation once on the highest resolution. The resulting output can be uploaded whole sale to AWS once generated.