From 4159da15c332c5724c4c867e82d6891e782dcf89 Mon Sep 17 00:00:00 2001 From: Kimberly Robasky Date: Tue, 2 Dec 2025 16:02:20 -0500 Subject: [PATCH] Improve deployment automation and documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makefile improvements: - Add automatic cleanup of ROLLBACK_COMPLETE stacks before deployment - Add automatic SAM deployment bucket creation (one-time account setup) - Add automatic API Gateway CloudWatch Logs role creation - Add node_modules check and auto-generate schemas in start-frontend - Fix test-jwt-aws to use $(AWS_REGION) instead of $AWS_REGION - Improve test-jwt-aws with better JWT validation and user info display Documentation improvements: - Update first-deployment.md to use ${ORG} and ${ENV} shell variables - Document automatic account-level resource setup - Add authentication testing section with proper examples - Users can now set ORG/ENV once and paste all commands without editing Add CLAUDE.md with comprehensive project overview for AI assistants 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- CLAUDE.md | 296 ++++++++++++++++++ Makefile | 91 +++++- .../quickstart/first-deployment.md | 187 +++++++---- 3 files changed, 509 insertions(+), 65 deletions(-) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..34b6682 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,296 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +SYNDI (Synthetic Intelligence for Data Integrity) is an automated lab data capture system enabling AI-ready data. The system consists of three core components: + +1. **PAUL** (Protocol Automation Librarian) - Extracts SOPs from written protocols using AI +2. **SAM** (SOP Authoring Manager) - Validates and optimizes SOPs +3. **CLAIRE** (Compliant Ledger-based Automation for Integrated Reporting and Export) - ELN data capture system with dynamic forms + +The architecture is AWS-based (Lambda, S3, Cognito, CloudFront, API Gateway) with a FastAPI Python backend and React/TypeScript frontend. + +## Essential Build Commands + +All commands require explicit `ENV` and `ORG` parameters (no defaults for security): + +### Development +```bash +# Start backend (FastAPI with hot reload) +make start-backend ENV=dev ORG= + +# Start frontend (Vite dev server with hot reload) +make start-frontend ENV=dev ORG= + +# Start both servers together +make start-dev ENV=dev ORG= + +# Serve lambda locally (requires Python 3.9 syndi conda environment) +conda activate syndi && make serve-lambda ENV=dev ORG= + +# Stop all local servers +make stop-all +``` + +### Testing +```bash +# All tests default to ORG=testorg for isolation +make test-all # Run all tests +make test-frontend # Vitest unit tests +make test-backend # pytest unit tests +make test-e2e # Playwright end-to-end tests +make test-ci # CI test suite with coverage + +# Run single backend test file +cd backend && pytest tests/unit/test_auth.py -v + +# Run single frontend test +cd frontend && npm test -- tests/unit/MyComponent.test.tsx +``` + +### Building & Deployment +```bash +# Build frontend for production +make build-frontend ENV=prod ORG= + +# Build backend Lambda package +make rs-build ENV=stage ORG= + +# Deploy to AWS (initial setup with resource creation) +ENABLE_AUTH=true CREATE_BUCKETS=true make rs-deploy ENV=stage ORG= + +# Deploy Lambda code updates only (fast, 95% of deployments) +make rs-deploy-function ENV=stage ORG= + +# Sync configs from CloudFormation outputs (after deployment) +make sync-configs ENV=stage ORG= +``` + +## Environment Setup + +### Python Environment +```bash +# Create syndi environment (Python 3.9 - matches AWS Lambda) +conda env create -f environment.yml + +# Activate for production simulation (serve-lambda) +conda activate syndi + +# For development, you can use any Python 3.9+ environment +# The syndi environment is required ONLY for serve-lambda +``` + +### Configuration System +SYNDI uses a three-tier configuration system: + +1. **Base configs**: `infra/.config/lambda/{env}.json` and `infra/.config/webapp/{env}.json` - Environment defaults +2. **Org-specific overrides**: `infra/.config/lambda/{env}-{org}.json` - Deployment-specific settings +3. **Merged output**: Generated automatically by `make config` command + +**Important**: Never manually edit merged configs in `backend/rawscribe/.config/` or `frontend/public/config.json`. Always edit source files in `infra/.config/`. + +After AWS deployment, always run `make sync-configs` to update org-specific files with CloudFormation outputs (API endpoints, Cognito IDs, bucket names). + +## Architecture Overview + +### Backend Structure (`/backend/rawscribe/`) +- **main.py** - FastAPI application entry point with Mangum handler for Lambda +- **routes/** - API endpoints: + - `auth.py` - Authentication endpoints + - `drafts.py` - Draft SOP management + - `eln.py` - Electronic Lab Notebook data + - `sops.py` - Standard Operating Procedures + - `files.py` - File upload/download with S3 + - `user_management.py` - User/role management + - `config.py` - Runtime configuration endpoint +- **utils/** - Shared utilities: + - `auth_providers/` - Authentication provider abstraction (Cognito/JWT) + - `storage_factory.py` - Storage backend abstraction (S3/Local) + - `config_loader.py` - Configuration loading with S3 fallback + - `rbac_enforcement.py` - Role-based access control + - `schema_utils.py` - JSON schema validation + +### Frontend Structure (`/frontend/src/`) +- **claire/** - CLAIRE ELN application + - `components/` - React components (form renderers, data tables) + - `views/` - Page-level components + - `hooks/` - Custom React hooks + - `lib/` - API clients and utilities + - `types/` - TypeScript type definitions +- **sam/** - SAM SOP authoring interface +- **paul/** - PAUL protocol extraction interface +- **shared/** - Shared components and utilities + +### Storage Architecture +- **S3 Buckets** (format: `rawscribe-{type}-{env}-{org}-{account}`): + - `lambda` - Configuration files + - `forms` - Published SOP schemas + - `eln` - Electronic lab notebook data with audit trails + - `drafts` - Draft SOPs (mutable, versioned) +- **Local Storage** - File-based storage for development (`.local/s3/`) + +### Authentication +- **AWS Cognito** - Production auth (AWS Lambda deployments) +- **JWT** - Local development auth with configurable secret +- **Provider Abstraction** - `auth_providers/` allows switching between Cognito/JWT +- **RBAC** - Four roles: Admin, Lab Manager, Researcher, Clinician + +## Key Patterns & Conventions + +### Configuration Loading +The backend loads config in this order: +1. S3 bucket (`rawscribe-lambda-{env}-{org}-{account}/config.json`) +2. Local file (`backend/rawscribe/.config/config.json`) +3. Fallback defaults (logged as warnings) + +Always check `app.state.config` for runtime configuration access. + +### Environment Validation +The backend enforces strict environment/provider rules: +- AWS Lambda MUST use Cognito (security critical - API Gateway rejects JWT) +- Stage/prod MUST use secure auth (Cognito or JWT with strong secret) +- Dev defaults to JWT for local development + +See `main.py` lifespan function for enforcement logic. + +### Storage Backend Selection +Use `StorageFactory` to get the appropriate storage backend: +```python +from rawscribe.utils.storage_factory import StorageFactory +storage = StorageFactory.get_storage(config) +# Returns S3Storage or LocalStorage based on config +``` + +### Testing Isolation +All tests use `ORG=testorg` by default. This provides complete isolation from dev/stage/prod data. Test configs are in `infra/.config/lambda/test-testorg.json` and `infra/.config/webapp/test-testorg.json`. + +## Deployment Best Practices + +### Initial Infrastructure Setup (once per env/org) +1. Deploy with resource creation: + ```bash + ENABLE_AUTH=true CREATE_BUCKETS=true make rs-deploy ENV=stage ORG=myorg + ``` +2. Sync configs to capture CloudFormation outputs: + ```bash + make sync-configs ENV=stage ORG=myorg + ``` +3. Review and commit org-specific configs + +### Regular Code Updates (frequent) +1. Use fast Lambda-only updates: + ```bash + make rs-deploy-function ENV=stage ORG=myorg + ``` +2. OR full stack update if infrastructure changed: + ```bash + ENABLE_AUTH=true CREATE_BUCKETS=false make rs-deploy ENV=stage ORG=myorg + ``` + +**Critical**: Never set `CREATE_BUCKETS=true` on existing deployments unless you want to recreate buckets (destroys data). + +### CloudFormation Stack Management +- Stack naming: `rawscribe-{env}-{org}` +- Build isolation: `.aws-sam-{env}-{org}/` directories +- Each env/org combination is a separate CloudFormation stack +- Rollback automatic on deployment failures + +## Common Pitfalls + +### Authentication Mismatches +- **Problem**: JWT tokens don't work in AWS Lambda +- **Cause**: API Gateway Cognito Authorizer only accepts Cognito tokens +- **Solution**: Always use `provider: cognito` in stage/prod configs + +### Config Out of Sync +- **Problem**: Frontend shows wrong API endpoint or auth errors +- **Cause**: Org-specific config not updated after deployment +- **Solution**: Always run `make sync-configs` after AWS deployments + +### Build Directory Conflicts +- **Problem**: `sam build` fails with cache errors +- **Cause**: Multiple env/org builds sharing same directory +- **Solution**: Each env/org uses isolated `.aws-sam-{env}-{org}/` directory + +### Missing ORG Parameter +- **Problem**: Commands fail with "ORG must be specified" +- **Cause**: No default ORG for security reasons +- **Solution**: Always provide `ORG=` parameter (except test commands which default to `testorg`) + +## AWS SAM Template + +The `template.yaml` defines all AWS resources: +- Lambda function with dependency layer +- API Gateway with Cognito authorizer +- S3 buckets for storage +- Cognito User Pool (optional - can attach existing) +- CloudFront distribution (optional) +- IAM roles and policies + +Parameters control behavior: +- `Environment` (dev/test/stage/prod) +- `Organization` (required, no default) +- `EnableAuth` (true/false - toggles Cognito) +- `CreateBuckets` (true/false - manages S3 lifecycle) +- `CognitoUserPoolId` / `CognitoClientId` (optional - use existing pool) + +## Code Style + +### Python (Backend) +- FastAPI patterns with async/await +- Type hints required (Pydantic models) +- Snake_case for variables/functions +- Import from local utils using relative imports +- Structured logging with `logging` module +- HTTP exceptions with proper status codes + +### TypeScript (Frontend) +- React 18 functional components +- Custom hooks for state management +- PascalCase for components +- camelCase for variables/functions +- React Query for API calls +- Radix UI for accessible components +- TailwindCSS for styling + +## Testing Strategy + +### Backend Testing (pytest) +- Located in `backend/tests/` +- `conftest.py` provides shared fixtures +- Async mode enabled (`asyncio_mode = auto`) +- Mock S3/Cognito with moto library +- Test isolation with testorg organization + +### Frontend Testing +- **Unit tests**: Vitest with React Testing Library +- **E2E tests**: Playwright for full user flows +- Located in `frontend/tests/` +- Test utils in `frontend/tests/helpers/` + +## Documentation + +Comprehensive documentation in `/docs/`: +- **System Admin** - Deployment, configuration, monitoring +- **CLAIRE User** - ELN usage guides +- **SAM User** - SOP authoring guides +- **PAUL User** - Protocol extraction guides + +Built with Sphinx and hosted on Read the Docs: https://syndi.readthedocs.io/ + +To build docs locally: +```bash +make docs +``` + +## Important Files + +- **Makefile** - All build, test, deployment commands (extensive help: `make help`) +- **template.yaml** - AWS SAM CloudFormation template +- **environment.yml** - Conda environment for Python 3.9 +- **AGENTS.md** - Development guide (complementary to this file) +- **build.toml** - SAM build configuration +- **pytest.ini** - Backend test configuration +- **playwright.config.ts** - E2E test configuration diff --git a/Makefile b/Makefile index 45ceb61..7336665 100644 --- a/Makefile +++ b/Makefile @@ -621,9 +621,18 @@ start-frontend: echo "❌ ORG parameter required. Usage: make start-frontend ENV=dev ORG="; \ exit 1; \ fi + @if [ ! -d "frontend/node_modules" ]; then \ + echo "❌ Frontend dependencies not installed"; \ + echo ""; \ + echo "Run: cd frontend && npm install"; \ + echo ""; \ + exit 1; \ + fi @echo "ENV=$(ENV) ORG=$(ORG)" @echo "To exit, press Ctrl+C" @$(MAKE) config ENV=$(ENV) ORG=$(ORG) + @echo "📋 Generating TypeScript schemas..." + @$(MAKE) schemas cd frontend && npm run dev start-dev: @@ -1144,6 +1153,63 @@ rs-deploy: rs-build $(eval AWS_REGION := $(shell $(GET_AWS_REGION))) $(eval ACCOUNT_NUMBER := $(shell $(GET_ACCOUNT_NUMBER))) $(eval STACK_NAME := rawscribe-$(ENV)-$(ORG)) + @# Check for failed stack and clean up automatically + @STACK_STATUS=$$(aws cloudformation describe-stacks --stack-name $(STACK_NAME) --region $(AWS_REGION) --query 'Stacks[0].StackStatus' --output text 2>/dev/null || echo "NONE"); \ + if [ "$$STACK_STATUS" = "ROLLBACK_COMPLETE" ] || [ "$$STACK_STATUS" = "ROLLBACK_FAILED" ]; then \ + echo "⚠️ Stack in $$STACK_STATUS state - automatic cleanup required"; \ + echo " Deleting failed stack: $(STACK_NAME)..."; \ + aws cloudformation delete-stack --stack-name $(STACK_NAME) --region $(AWS_REGION); \ + echo " Waiting for deletion to complete..."; \ + aws cloudformation wait stack-delete-complete --stack-name $(STACK_NAME) --region $(AWS_REGION); \ + echo "✅ Failed stack deleted, proceeding with fresh deployment"; \ + elif [ "$$STACK_STATUS" != "NONE" ] && [ "$$STACK_STATUS" != "CREATE_COMPLETE" ] && [ "$$STACK_STATUS" != "UPDATE_COMPLETE" ]; then \ + echo "⚠️ Stack is in $$STACK_STATUS state"; \ + echo " You may need to wait for current operation to complete or manually clean up"; \ + fi + @# Ensure SAM deployment bucket exists (one-time account setup) + @if ! aws s3 ls s3://rawscribe-sam-deployments-$(ACCOUNT_NUMBER) --region $(AWS_REGION) >/dev/null 2>&1; then \ + echo "📦 Creating SAM deployment bucket (one-time account setup)..."; \ + echo " Bucket: rawscribe-sam-deployments-$(ACCOUNT_NUMBER)"; \ + aws s3 mb s3://rawscribe-sam-deployments-$(ACCOUNT_NUMBER) --region $(AWS_REGION); \ + aws s3api put-bucket-versioning \ + --bucket rawscribe-sam-deployments-$(ACCOUNT_NUMBER) \ + --versioning-configuration Status=Enabled \ + --region $(AWS_REGION); \ + echo "✅ SAM deployment bucket created and versioning enabled"; \ + else \ + echo "✅ SAM deployment bucket exists: rawscribe-sam-deployments-$(ACCOUNT_NUMBER)"; \ + fi + @# Ensure API Gateway CloudWatch Logs role exists (one-time account setup) + @CURRENT_ROLE=$$(aws apigateway get-account --query 'cloudwatchRoleArn' --output text --region $(AWS_REGION) 2>/dev/null || echo "None"); \ + if [ "$$CURRENT_ROLE" = "None" ] || [ -z "$$CURRENT_ROLE" ]; then \ + echo "🔧 Configuring API Gateway CloudWatch Logs role (one-time account setup)..."; \ + if ! aws iam get-role --role-name APIGatewayCloudWatchLogsRole >/dev/null 2>&1; then \ + echo " Creating IAM role: APIGatewayCloudWatchLogsRole"; \ + aws iam create-role \ + --role-name APIGatewayCloudWatchLogsRole \ + --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"apigateway.amazonaws.com"},"Action":"sts:AssumeRole"}]}' \ + --output text >/dev/null 2>&1 || echo " (Role may already exist)"; \ + echo " Attaching CloudWatch Logs policy..."; \ + aws iam attach-role-policy \ + --role-name APIGatewayCloudWatchLogsRole \ + --policy-arn arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs \ + 2>&1 | grep -v "attached" || true; \ + echo " Waiting for IAM role to propagate (15 seconds)..."; \ + sleep 15; \ + fi; \ + ROLE_ARN=$$(aws iam get-role --role-name APIGatewayCloudWatchLogsRole --query 'Role.Arn' --output text); \ + echo " Setting API Gateway account role: $$ROLE_ARN"; \ + if aws apigateway update-account \ + --patch-operations op=replace,path=/cloudwatchRoleArn,value=$$ROLE_ARN \ + --region $(AWS_REGION) >/dev/null 2>&1; then \ + echo "✅ API Gateway CloudWatch Logs role configured"; \ + else \ + echo "⚠️ API Gateway role set failed - may need more propagation time"; \ + echo " This is non-critical and will be configured on next deployment"; \ + fi; \ + else \ + echo "✅ API Gateway CloudWatch Logs role already configured"; \ + fi @# Auto-detect CREATE_BUCKETS based on stack existence @# Once a stack is created, we ALWAYS use CREATE_BUCKETS=true to keep CloudFormation managing buckets @STACK_EXISTS=$$(aws cloudformation describe-stacks --stack-name $(STACK_NAME) --region $(AWS_REGION) 2>/dev/null && echo "true" || echo "false"); \ @@ -2413,12 +2479,12 @@ test-jwt-aws: $(eval AWS_REGION := $(shell $(GET_AWS_REGION))) @if [ -z "$(ORG)" ]; then echo "Error: ORG parameter required. Usage: make test-jwt-aws ENV=stage ORG=pwb"; exit 1; fi @echo "Testing JWT for $(ORG) on AWS..." - @API_ID=$$(aws apigateway get-rest-apis --query "items[?name=='rawscribe-$(ENV)-$(ORG)-api'].id | [0]" --output text --region $$AWS_REGION); \ + @API_ID=$$(aws apigateway get-rest-apis --query "items[?name=='rawscribe-$(ENV)-$(ORG)-api'].id | [0]" --output text --region $(AWS_REGION)); \ if [ -z "$$API_ID" ] || [ "$$API_ID" = "None" ]; then \ echo "API Gateway not found for rawscribe-$(ENV)-$(ORG)-api"; \ exit 1; \ fi; \ - API_URL="https://$$API_ID.execute-api.$$AWS_REGION.amazonaws.com/$(ENV)"; \ + API_URL="https://$$API_ID.execute-api.$(AWS_REGION).amazonaws.com/$(ENV)"; \ echo "Found API Gateway: $$API_URL"; \ TEST_USER=$${$(shell echo $(ORG) | tr '[:lower:]' '[:upper:]')_TEST_USER}; \ TEST_PASS=$${$(shell echo $(ORG) | tr '[:lower:]' '[:upper:]')_TEST_PASSWORD}; \ @@ -2427,15 +2493,28 @@ test-jwt-aws: echo " export $(shell echo $(ORG) | tr '[:lower:]' '[:upper:]')_TEST_PASSWORD="; \ exit 1; \ fi; \ - POOL_ID=$$(aws cognito-idp list-user-pools --max-results 60 --query "UserPools[?contains(Name,'rawscribe-$(ENV)-$(ORG)')].Id | [0]" --output text --region $$AWS_REGION); \ + POOL_ID=$$(aws cognito-idp list-user-pools --max-results 60 --query "UserPools[?contains(Name,'rawscribe-$(ENV)-$(ORG)')].Id | [0]" --output text --region $(AWS_REGION)); \ if [ -z "$$POOL_ID" ] || [ "$$POOL_ID" = "None" ]; then \ echo "User Pool not found for rawscribe-$(ENV)-$(ORG)"; \ exit 1; \ fi; \ - CLIENT_ID=$$(aws cognito-idp list-user-pool-clients --user-pool-id $$POOL_ID --query "UserPoolClients[0].ClientId" --output text --region $$AWS_REGION); \ - JWT=$$(aws cognito-idp admin-initiate-auth --user-pool-id $$POOL_ID --client-id $$CLIENT_ID --auth-flow ADMIN_USER_PASSWORD_AUTH --auth-parameters USERNAME=$$TEST_USER,PASSWORD=$$TEST_PASS --region $$AWS_REGION --query 'AuthenticationResult.IdToken' --output text); \ + CLIENT_ID=$$(aws cognito-idp list-user-pool-clients --user-pool-id $$POOL_ID --query "UserPoolClients[0].ClientId" --output text --region $(AWS_REGION)); \ + JWT=$$(aws cognito-idp admin-initiate-auth --user-pool-id $$POOL_ID --client-id $$CLIENT_ID --auth-flow ADMIN_USER_PASSWORD_AUTH --auth-parameters "USERNAME=$$TEST_USER,PASSWORD=$$TEST_PASS" --region $(AWS_REGION) --query 'AuthenticationResult.IdToken' --output text); \ + if [ -z "$$JWT" ]; then \ + echo "❌ Failed to get JWT token"; \ + exit 1; \ + fi; \ + echo "✅ JWT token obtained (length: $${#JWT})"; \ echo "Testing protected endpoint..."; \ - curl -s -H "Authorization: Bearer $$JWT" $$API_URL/api/config/private | jq '.lambda.auth.cognito' + RESPONSE=$$(curl -s -H "Authorization: Bearer $$JWT" $$API_URL/api/config/private); \ + if echo "$$RESPONSE" | jq -e '.user' >/dev/null 2>&1; then \ + echo "✅ Authentication successful!"; \ + echo "$$RESPONSE" | jq '.user'; \ + else \ + echo "❌ Unexpected response:"; \ + echo "$$RESPONSE" | jq .; \ + exit 1; \ + fi test-jwt-regression: @echo "Running JWT authentication regression tests..." diff --git a/docs/source/shared/system-admin/quickstart/first-deployment.md b/docs/source/shared/system-admin/quickstart/first-deployment.md index 0b9a715..190fa39 100644 --- a/docs/source/shared/system-admin/quickstart/first-deployment.md +++ b/docs/source/shared/system-admin/quickstart/first-deployment.md @@ -52,14 +52,58 @@ aws sts get-caller-identity # Should output your account ID and user/role ``` +## Automatic Account-Level Setup + +During your first deployment, `make rs-deploy` automatically creates two account-wide resources if they don't exist: + +### 1. SAM Deployment Bucket + +**Name**: `rawscribe-sam-deployments-{account-id}` +**Purpose**: SAM uses this bucket to upload Lambda artifacts (code and layers) before CloudFormation deployment. +**Scope**: Account-wide - shared by all environments and organizations +**When created**: Automatically on first `make rs-deploy` in your AWS account + +This bucket is separate from the application S3 buckets and is used exclusively by AWS SAM for deployment artifacts. + +### 2. API Gateway CloudWatch Logs Role + +**Name**: `APIGatewayCloudWatchLogsRole` +**Purpose**: Allows API Gateway to write access logs and execution logs to CloudWatch Logs. +**Scope**: Account-wide - required for all API Gateway stages with logging enabled +**When created**: Automatically on first `make rs-deploy` in your AWS account + +This IAM role grants API Gateway permission to create log groups and streams in CloudWatch. + +### What You'll See + +On first deployment, you'll see output like: +``` +📦 Creating SAM deployment bucket (one-time account setup)... + Bucket: rawscribe-sam-deployments-804320730777 +✅ SAM deployment bucket created and versioning enabled + +🔧 Configuring API Gateway CloudWatch Logs role (one-time account setup)... + Creating IAM role: APIGatewayCloudWatchLogsRole + Setting API Gateway account role: arn:aws:iam::804320730777:role/APIGatewayCloudWatchLogsRole +✅ API Gateway CloudWatch Logs role configured +``` + +On subsequent deployments: +``` +✅ SAM deployment bucket exists: rawscribe-sam-deployments-804320730777 +✅ API Gateway CloudWatch Logs role already configured +``` + +**These are one-time setups** - once created, they're reused for all future SYNDI deployments in your AWS account. + ## Step-by-Step Deployment ### Step 1: Choose Environment and Organization ```bash # Set your parameters -ENV=stage # or prod -ORG=myorg # Your organization identifier +export ENV=stage # or prod +export ORG=myorg # Your organization identifier (change to your org) # Account ID (automatic) ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) @@ -83,9 +127,9 @@ Deploy all resources with one command: ```bash # Deploy with authentication and bucket creation ENABLE_AUTH=true CREATE_BUCKETS=true \ - ADMIN_USERNAME=admin@myorg.com \ + ADMIN_USERNAME=admin@${ORG}.com \ ADMIN_PASSWORD=SecurePassword2025! \ - ORG=myorg ENV=stage make rs-deploy + make rs-deploy ``` **Parameters explained:** @@ -108,7 +152,7 @@ ENABLE_AUTH=true CREATE_BUCKETS=true \ **Expected output:** ``` -🚀 Deploying to stage for myorg with SAM... +🚀 Deploying to stage for ${ORG} with SAM... Building Lambda with SAM... ✅ Copied requirements.txt to layer directory 🔄 Building with layer caching... @@ -117,7 +161,7 @@ Building Lambda with SAM... Deploying to AWS... [CloudFormation progress...] -👤 Creating admin user admin@myorg.com... +👤 Creating admin user admin@${ORG}.com... 👥 Ensuring admin group exists... 🔗 Adding user to admin group... 🔐 Setting permanent password... @@ -129,21 +173,21 @@ Deploying to AWS... SOPs list: ✅ Found 0 SOPs (none uploaded yet) ════════════════════════════════════════════════ -🎉 Deployment Complete: stage/myorg +🎉 Deployment Complete: stage/${ORG} ════════════════════════════════════════════════ 📡 API Endpoint: https://abc123def.execute-api.us-east-1.amazonaws.com/stage 🔐 User Pool ID: us-east-1_ABC123DEF 🔑 Client ID: abc123def456ghi789 -👤 Admin User: admin@myorg.com +👤 Admin User: admin@${ORG}.com 🔒 Admin Pass: [set successfully] 📋 Next Steps: 1. Upload SOPs to S3: - aws s3 cp your-sop.yaml s3://rawscribe-forms-stage-myorg-288761742376/sops/ + aws s3 cp your-sop.yaml s3://rawscribe-forms-stage-${ORG}-288761742376/sops/ 2. Check deployment status: - ORG=myorg ENV=stage make check-rs + make check-rs 3. View CloudWatch logs: - aws logs tail /aws/lambda/rawscribe-stage-myorg-backend --follow + aws logs tail /aws/lambda/rawscribe-stage-${ORG}-backend --follow ════════════════════════════════════════════════ ``` @@ -152,19 +196,19 @@ Deploying to AWS... After deployment, sync configuration files from CloudFormation outputs: ```bash -make sync-configs ENV=stage ORG=myorg +make sync-configs ``` **What it does:** 1. Queries CloudFormation stack for outputs 2. Extracts API endpoint, Cognito User Pool ID, Client ID -3. Updates `infra/.config/webapp/stage-myorg.json` -4. Updates `infra/.config/lambda/stage-myorg.json` +3. Updates `infra/.config/webapp/stage-${ORG}.json` +4. Updates `infra/.config/lambda/stage-${ORG}.json` 5. Preserves custom fields you may have added **Output:** ``` -🔍 Fetching outputs from stack: rawscribe-stage-myorg +🔍 Fetching outputs from stack: rawscribe-stage-${ORG} 📋 CloudFormation Outputs: ApiEndpoint: https://abc123.execute-api.us-east-1.amazonaws.com/stage @@ -173,17 +217,17 @@ make sync-configs ENV=stage ORG=myorg CloudFrontURL: https://d1234567.cloudfront.net 📝 Updating configuration files... -✅ Updated org-specific config: infra/.config/webapp/stage-myorg.json +✅ Updated org-specific config: infra/.config/webapp/stage-${ORG}.json (Custom fields preserved, CloudFormation values updated) -✅ Updated org-specific lambda config: infra/.config/lambda/stage-myorg.json +✅ Updated org-specific lambda config: infra/.config/lambda/stage-${ORG}.json (Custom fields preserved, CloudFormation values updated) ✅ Configuration sync complete! 📌 Next steps: - 1. Review changes: git diff infra/.config/webapp/stage-myorg.json - 2. Test frontend: make start-frontend ENV=stage ORG=myorg - 3. Commit if correct: git add infra/.config/webapp/stage-myorg.json + 1. Review changes: git diff infra/.config/webapp/stage-${ORG}.json + 2. Test frontend: make start-frontend + 3. Commit if correct: git add infra/.config/webapp/stage-${ORG}.json ``` ### Step 4: Review and Commit Configs @@ -192,21 +236,21 @@ Review the auto-generated configuration files: ```bash # View webapp config -cat infra/.config/webapp/stage-myorg.json | jq +cat infra/.config/webapp/stage-${ORG}.json | jq -# View lambda config -cat infra/.config/lambda/stage-myorg.json | jq +# View lambda config +cat infra/.config/lambda/stage-${ORG}.json | jq # See what changed -git diff infra/.config/webapp/stage-myorg.json -git diff infra/.config/lambda/stage-myorg.json +git diff infra/.config/webapp/stage-${ORG}.json +git diff infra/.config/lambda/stage-${ORG}.json ``` **If using private config repo:** ```bash cd infra/.config -git add webapp/stage-myorg.json lambda/stage-myorg.json -git commit -m "Add stage-myorg configs with deployed resource IDs" +git add webapp/stage-${ORG}.json lambda/stage-${ORG}.json +git commit -m "Add stage-${ORG} configs with deployed resource IDs" git push cd ../.. ``` @@ -218,14 +262,14 @@ Upload Standard Operating Procedures to the forms bucket: ```bash # Upload single SOP aws s3 cp your-sop.yaml \ - s3://rawscribe-forms-stage-myorg-${ACCOUNT_ID}/sops/ + s3://rawscribe-forms-stage-${ORG}-${ACCOUNT_ID}/sops/ # Or upload directory of SOPs aws s3 sync ./sops-directory \ - s3://rawscribe-forms-stage-myorg-${ACCOUNT_ID}/sops/ + s3://rawscribe-forms-stage-${ORG}-${ACCOUNT_ID}/sops/ # Verify upload -aws s3 ls s3://rawscribe-forms-stage-myorg-${ACCOUNT_ID}/sops/ +aws s3 ls s3://rawscribe-forms-stage-${ORG}-${ACCOUNT_ID}/sops/ ``` ### Step 6: Test the Deployment @@ -234,29 +278,29 @@ Verify everything works: ```bash # Check deployment status -ORG=myorg ENV=stage make check-rs +make check-rs ``` **Output shows:** ``` -=== myorg Resources (stage) === -Lambda: rawscribe-stage-myorg-backend -API Gateway: rawscribe-stage-myorg-api +=== ${ORG} Resources (stage) === +Lambda: rawscribe-stage-${ORG}-backend +API Gateway: rawscribe-stage-${ORG}-api API Endpoint: https://abc123.execute-api.us-east-1.amazonaws.com/stage/ -Stack Name: rawscribe-stage-myorg +Stack Name: rawscribe-stage-${ORG} User Pool: us-east-1_ABC123 Client ID: abc123def456 S3 Buckets: - lambda: rawscribe-lambda-stage-myorg-288761742376 - forms: rawscribe-forms-stage-myorg-288761742376 - ELN: rawscribe-eln-stage-myorg-288761742376 - ELN drafts: rawscribe-eln-drafts-stage-myorg-288761742376 + lambda: rawscribe-lambda-stage-${ORG}-288761742376 + forms: rawscribe-forms-stage-${ORG}-288761742376 + ELN: rawscribe-eln-stage-${ORG}-288761742376 + ELN drafts: rawscribe-eln-drafts-stage-${ORG}-288761742376 ``` **Test API endpoint:** ```bash API_ENDPOINT=$(aws cloudformation describe-stacks \ - --stack-name rawscribe-stage-myorg \ + --stack-name rawscribe-stage-${ORG} \ --query 'Stacks[0].Outputs[?OutputKey==`ApiEndpoint`].OutputValue' \ --output text) @@ -268,20 +312,45 @@ curl ${API_ENDPOINT}/health **Test authentication:** ```bash -make test-jwt-aws ENV=stage ORG=myorg +# Set test credentials (use single quotes for passwords with special characters) +export ${ORG^^}_TEST_USER=testresearcher@example.com +export ${ORG^^}_TEST_PASSWORD='TestResearch123!' + +# Run authentication test +make test-jwt-aws ``` +**Expected output:** +``` +Testing JWT for ${ORG} on AWS... +Found API Gateway: https://abc123.execute-api.us-east-1.amazonaws.com/stage +✅ JWT token obtained (length: 1171) +Testing protected endpoint... +✅ Authentication successful! +{ + "id": "default-user", + "email": "default@localhost", + "username": "default", + "name": "Default User", + "groups": ["admin"], + "permissions": ["*"], + "isAdmin": true +} +``` + +**Note:** Use **single quotes** for the password to prevent bash from interpreting special characters like `!`. The `${ORG^^}` syntax uppercases the ORG variable (e.g., if `ORG=myorg`, then `${ORG^^}_TEST_USER` becomes `MYORG_TEST_USER`). + ### Step 7: Test Frontend Start frontend locally pointing to deployed backend: ```bash # Frontend will use config.json with deployed API endpoint -make start-frontend ENV=stage ORG=myorg +make start-frontend ``` Navigate to `http://localhost:3000` and: -1. Login with admin@myorg.com / SecurePassword2025! +1. Login with admin@${ORG}.com / SecurePassword2025! 2. Verify SOP list loads 3. Test creating a draft 4. Test submitting an ELN @@ -342,11 +411,11 @@ Example: `rawscribe-stage-myorg` **Solution:** ```bash # Check existing stack -aws cloudformation describe-stacks --stack-name rawscribe-stage-myorg +aws cloudformation describe-stacks --stack-name rawscribe-stage-${ORG} # Either use different ORG name or delete existing stack -aws cloudformation delete-stack --stack-name rawscribe-stage-myorg -aws cloudformation wait stack-delete-complete --stack-name rawscribe-stage-myorg +aws cloudformation delete-stack --stack-name rawscribe-stage-${ORG} +aws cloudformation wait stack-delete-complete --stack-name rawscribe-stage-${ORG} ``` ### "Bucket name already taken" @@ -356,7 +425,7 @@ aws cloudformation wait stack-delete-complete --stack-name rawscribe-stage-myorg **Solution:** ```bash # Use CREATE_BUCKETS=false if buckets exist -CREATE_BUCKETS=false ENABLE_AUTH=true ORG=myorg ENV=stage make rs-deploy +CREATE_BUCKETS=false ENABLE_AUTH=true make rs-deploy # Or choose different ORG name ``` @@ -407,13 +476,13 @@ After deployment, verify: **Verification commands:** ```bash # Stack status -make check-rs-stack-status ENV=stage ORG=myorg +make check-rs-stack-status # Complete check -make check-rs ENV=stage ORG=myorg +make check-rs # Test auth -make test-jwt-aws ENV=stage ORG=myorg +make test-jwt-aws ``` ## Next Steps @@ -422,7 +491,7 @@ After successful first deployment: 1. **Create Additional Users**: See [User Management](../authentication/user-management.md) 2. **Upload SOPs**: Add your organization's SOPs to forms bucket -3. **Customize Configs**: Edit `infra/.config/lambda/stage-myorg.json` for org-specific settings +3. **Customize Configs**: Edit `infra/.config/lambda/stage-${ORG}.json` for org-specific settings 4. **Deploy Frontend**: Build and deploy frontend to CloudFront (TBD) 5. **Setup Monitoring**: Configure CloudWatch alarms and dashboards 6. **Document Procedures**: Create runbook for your organization @@ -433,11 +502,12 @@ After successful first deployment: For production deployment (first time): ```bash -# Production deployment with confirmation +# Production deployment with confirmation (set ENV=prod first) +export ENV=prod ENABLE_AUTH=true CREATE_BUCKETS=true \ - ADMIN_USERNAME=admin@myorg.com \ + ADMIN_USERNAME=admin@${ORG}.com \ ADMIN_PASSWORD=ProductionPassword2025! \ - ORG=myorg ENV=prod make rs-deploy + make rs-deploy ``` **Production differences:** @@ -461,17 +531,16 @@ After initial deployment, use different commands for updates: ```bash # Code changes only (30 seconds) -make rs-deploy-function ENV=stage ORG=myorg +make rs-deploy-function # Config changes (1-2 minutes) -make rs-deploy-only ENV=stage ORG=myorg +make rs-deploy-only # Infrastructure changes (5-7 minutes) -ENABLE_AUTH=true CREATE_BUCKETS=false \ - ORG=myorg ENV=stage make rs-deploy +ENABLE_AUTH=true CREATE_BUCKETS=false make rs-deploy # Always sync after infrastructure changes -make sync-configs ENV=stage ORG=myorg +make sync-configs ``` **Note:** Use `CREATE_BUCKETS=false` for subsequent deployments (buckets already exist).