DS5220 Data Project 2: NOAA Tide Tracking Pipeline

Overview

This repository implements a scheduled data pipeline that collects NOAA water-level observations for The Battery, NY (station 8518750), stores each observation in DynamoDB, and publishes visualization artifacts to S3.

At a high level:

• A Kubernetes CronJob runs hourly.
• The Python application requests the latest water-level reading from NOAA CO-OPS.
• The reading is written to DynamoDB.
• The full stored history is loaded, plotted, and exported as CSV.
• plot.png and data.csv are uploaded to S3.

Repository Structure

• tide-app/app.py: Data collection, DynamoDB persistence, plotting, and S3 upload logic.
• tide-app/Dockerfile: Container image definition for the tide tracking job.
• tide-app/requirements.txt: Python dependencies.
• tide-job.yaml: Kubernetes CronJob specification (hourly schedule).
• main.tf: Terraform resources for AWS infrastructure (IAM role/policy, EC2 instance profile, security group, EC2 instance, and Elastic IP).

Data Source Chosen and Why

The data source is the NOAA CO-OPS Data API (https://api.tidesandcurrents.noaa.gov/api/prod/datagetter) with:

• product=water_level
• station=8518750 (The Battery, NY)
• date=latest
• units=metric
• time_zone=gmt

This source was selected because NOAA CO-OPS is the authoritative public source for U.S. coastal water-level observations and provides frequent, machine-readable updates suitable for scheduled ingestion.

What the Data Shows Over a 72-Hour Window

With an hourly CronJob schedule, a complete 72-hour window produces up to 72 observations (assuming no missed runs).

Typical characteristics expected in this dataset:

• A repeating tidal oscillation rather than a monotonic trend.
• Alternating local highs and lows consistent with tidal cycles.
• Periodic peaks and troughs that should recur across the 72-hour interval.

Potential surprises or spikes:

• Short-term deviations caused by weather, pressure, or local conditions.
• Missing or duplicate timestamps if jobs fail/retry or run out of schedule.

In this repository, the plotting logic visualizes all available historical records currently in DynamoDB; it does not automatically trim to the most recent 72 points.

Secrets vs Plain Environment Variables

Kubernetes Secrets and plain environment variables both provide runtime configuration, but they differ in intent and handling:

• Plain environment variables are best for non-sensitive configuration (for example, station ID or AWS region).
• Kubernetes Secrets are designed for sensitive values, with API-level object separation and optional controls such as encryption at rest.

Why this matters:

• Putting credentials directly into plain env vars or manifests increases accidental exposure risk (source control, logs, manifests, and debugging output).
• Secrets improve operational hygiene for sensitive data and reduce leakage risk when managed correctly.

How CronJob Pods Access AWS Without Static Credentials in Files

AWS SDK calls in app.py use boto3, which resolves credentials from the runtime environment. In a Kubernetes deployment on AWS, this is typically provided by attached IAM role credentials rather than hard-coded keys in files.

For this project as currently defined:

• Terraform creates an IAM role and instance profile for an EC2 host.
• The Kubernetes manifest does not define a dedicated service account annotation for IRSA.
• Therefore, pods would typically rely on node-level IAM credentials (if the cluster is configured that way), and no static AWS key file is required in the container image or repository.

One Production Improvement

If this were a production pipeline, the first major improvement would be workload-level IAM isolation (IRSA) with a dedicated Kubernetes service account and least-privilege policies per job.

That change would:

• Avoid broad node-level credential sharing.
• Make permissions explicit per workload.
• Improve auditability and reduce blast radius.

Current Configuration Notes (Accuracy and Gaps)

The current code and infrastructure definitions are close, but not fully aligned:

• app.py writes to DynamoDB table tide-tracking.
• main.tf IAM policy references iss-tracking.
• tide-job.yaml sets DYNAMODB_TABLE and S3_BUCKET, but app.py currently hard-codes those names and does not read those env vars.

These items should be aligned before treating the deployment as fully production-ready.

Running the Application Locally

From tide-app:

pip install -r requirements.txt
python app.py

Required runtime assumptions:

• AWS credentials are available through standard boto3 resolution.
• DynamoDB table exists and is accessible.
• S3 bucket exists and is writable.
• Network access to NOAA API is available.