Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# CODEOWNERS file (from GitHub template at
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners)
# Each line is a file pattern followed by one or more owners.

##############################################################################
# These owners will be the default owners for everything in the repo. Unless a
# later match takes precedence, @ghukill will be requested for review
# when someone opens a pull request.This is commented out in favor of using a
# team as the default (see below). It is left here as a comment to indicate
# the primary expert for this code.
# * @ghukill

# Teams can be specified as code owners as well. Teams should be identified in
# the format @org/team-name. Teams must have explicit write access to the
# repository.
* @mitlibraries/dataeng

# We set the senior engineer in the team as the owner of the CODEOWNERS file as
# a layer of protection for unauthorized changes.
/.github/CODEOWNERS @ghukill
18 changes: 5 additions & 13 deletions .github/pull-request-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ Describe the overall purpose of the PR changes and any useful background context

### How can a reviewer manually see the effects of these changes?
Explain how to see the proposed changes in the application if possible.

Delete this section if it isn't applicable to the PR.

### Includes new or updated dependencies?
Expand All @@ -15,15 +14,8 @@ YES | NO
### What are the relevant tickets?
- Include links to Jira Software and/or Jira Service Management tickets here.

### Developer
- [ ] All new ENV is documented in README
- [ ] All new ENV has been added to staging and production environments
- [ ] All related Jira tickets are linked in commit message(s)
- [ ] Stakeholder approval has been confirmed (or is not needed)

### Code Reviewer(s)
- [ ] The commit message is clear and follows our guidelines (not just this PR message)
- [ ] There are appropriate tests covering any new functionality
- [ ] The provided documentation is sufficient for understanding any new functionality introduced
- [ ] Any manual tests have been performed or provided examples verified
- [ ] New dependencies are appropriate or there were no changes
### Code review
* Code review best practices are documented
[here](https://mitlibraries.github.io/guides/collaboration/code_review.html)
and you are encouraged to have a constructive dialogue with your reviewers
about their preferences and expectations.
10 changes: 7 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
name: CI
on: push
permissions: read-all
on:
pull_request:
paths-ignore:
- '.github/**'
jobs:
test:
uses: mitlibraries/.github/.github/workflows/python-shared-test.yml@main
uses: mitlibraries/.github/.github/workflows/python-uv-shared-test.yml@main
lint:
uses: mitlibraries/.github/.github/workflows/python-shared-lint.yml@main
uses: mitlibraries/.github/.github/workflows/python-uv-shared-lint.yml@main
27 changes: 14 additions & 13 deletions .github/workflows/dev-build.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
### This is the Terraform-generated dev-build.yml workflow for the oai-pmh-harvester-dev app repository ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
### If the container requires any additional pre-build commands, uncomment and edit ###
### the PREBUILD line at the end of the document. ###
name: Dev Container Build and Deploy
name: Dev Image Build and Deploy
on:
workflow_dispatch:
pull_request:
Expand All @@ -11,14 +7,19 @@ on:
paths-ignore:
- '.github/**'

permissions:
id-token: write
contents: read

jobs:
deploy:
name: Dev Container Deploy
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-dev.yml@main

build-push:
name: Dev Build and Push
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-dev.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
GHA_ROLE: "oai-pmh-harvester-gha-dev"
ECR: "oai-pmh-harvester-dev"
# FUNCTION: ""
# PREBUILD:
AWS_REGION: us-east-1
GHA_ROLE: oai-pmh-harvester-gha-dev
ECR: oai-pmh-harvester-dev
# DOCKERFILE: # only if the name of the Dockerfile is not "Dockerfile"!
# PREBUILD: # only if there is some pre-build dependency
24 changes: 13 additions & 11 deletions .github/workflows/prod-promote.yml
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
### This is the Terraform-generated prod-promote.yml workflow for the oai-pmh-harvester-prod repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document. ###
name: Prod Container Promote
name: Prod Image Promote
on:
workflow_dispatch:
release:
types: [published]

permissions:
id-token: write
contents: read

jobs:
deploy:
name: Prod Container Promote
uses: mitlibraries/.github/.github/workflows/ecr-shared-promote-prod.yml@main

promote:
name: Prod promote
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-promote-prod.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
AWS_REGION: us-east-1
GHA_ROLE_STAGE: oai-pmh-harvester-gha-stage
GHA_ROLE_PROD: oai-pmh-harvester-gha-prod
ECR_STAGE: "oai-pmh-harvester-stage"
ECR_PROD: "oai-pmh-harvester-prod"
# FUNCTION: ""

ECR_STAGE: oai-pmh-harvester-stage
ECR_PROD: oai-pmh-harvester-prod
# DEFAULT_BRANCH: # Only if the default branch is not "main"!
27 changes: 14 additions & 13 deletions .github/workflows/stage-build.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
### This is the Terraform-generated dev-build.yml workflow for the oai-pmh-harvester-stage app repository ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
### If the container requires any additional pre-build commands, uncomment and edit ###
### the PREBUILD line at the end of the document. ###
name: Stage Container Build and Deploy
name: Stage Image Build and Deploy
on:
workflow_dispatch:
push:
Expand All @@ -11,14 +7,19 @@ on:
paths-ignore:
- '.github/**'

permissions:
id-token: write
contents: read

jobs:
deploy:
name: Stage Container Deploy
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-stage.yml@main

build-push:
name: Stage Build and Push
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-stage.yml@main
secrets: inherit
with:
AWS_REGION: "us-east-1"
GHA_ROLE: "oai-pmh-harvester-gha-stage"
ECR: "oai-pmh-harvester-stage"
# FUNCTION: ""
# PREBUILD:
AWS_REGION: us-east-1
GHA_ROLE: oai-pmh-harvester-gha-stage
ECR: oai-pmh-harvester-stage
# DOCKERFILE: # only if the name of the Dockerfile is not "Dockerfile"!
# PREBUILD: # only if there is some pre-build dependency
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -108,5 +108,7 @@ out.xml
.DS_Store
.vscode
.idea
.arch_tag
.tool-versions

output/
29 changes: 15 additions & 14 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,29 +1,30 @@
default_language_version:
python: python3.11 # set for project python version
python: python3.13

default_stages:
- pre-push

repos:
- repo: local
hooks:
- id: black-apply
name: black-apply
entry: pipenv run black
- id: ruff-format
name: ruff-format
entry: uv run ruff format --diff
language: system
pass_filenames: true
types: ["python"]

- id: mypy
name: mypy
entry: pipenv run mypy
entry: uv run mypy
language: system
pass_filenames: true
types: ["python"]
exclude: "tests/"
- id: ruff-apply
name: ruff-apply
entry: pipenv run ruff check --fix
exclude: "(tests/|output/|migrations/)"

- id: ruff-check
name: ruff-check
entry: uv run ruff check
language: system
pass_filenames: true
types: ["python"]
- id: pip-audit
name: pip-audit
entry: pipenv run pip-audit
language: system
pass_filenames: false
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.12
3.13
24 changes: 13 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
FROM python:3.12-slim AS py3
RUN pip install pipenv
FROM python:3.13-slim

RUN apt-get update && \
apt-get install -y --no-install-recommends git ca-certificates && \
rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
ENV UV_SYSTEM_PYTHON=1

FROM py3 AS wheel
WORKDIR /app
COPY . .
RUN python3 setup.py bdist_wheel

FROM py3
COPY Pipfile* /
RUN pipenv install --system --ignore-pipfile --deploy
COPY --from=wheel /app/dist/harvest*-py3-none-any.whl .
RUN pip3 install harvest*-py3-none-any.whl
COPY pyproject.toml uv.lock* ./
COPY harvester ./harvester

RUN uv pip install --system .

ENTRYPOINT ["oai"]
CMD ["--help"]
CMD []
Loading
Loading