Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Integration Tests

on:
push:
branches: [main]
pull_request:
workflow_dispatch:
inputs:
node-version:
description: 'Node.js version'
required: true
type: choice
default: 'all'
options:
- 'all'
- '22'
- '24'
- '26'

jobs:
generate-node-version-matrix:
name: Generate Node Version Matrix
runs-on: ubuntu-latest
outputs:
node-versions: ${{ steps.set-node-versions.outputs.node-versions }}

steps:
- name: Checkout code
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

- name: Set Node versions
id: set-node-versions
env:
NODE_VER: ${{ github.event.inputs.node-version }}
run: |
if [ "$NODE_VER" == "all" ] || [ -z "$NODE_VER" ]; then
echo "node-versions=[22, 24, 26]" >> $GITHUB_OUTPUT
else
echo "node-versions=[$NODE_VER]" >> $GITHUB_OUTPUT
fi

integration-tests:
name: Integration Tests (Node ${{ matrix.node-version }})
needs: [generate-node-version-matrix]
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
node-version: ${{ fromJSON(needs.generate-node-version-matrix.outputs.node-versions) }}

steps:
- name: Checkout code
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: ${{ matrix.node-version }}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding cache: npm to the setup-node step for faster CI runs

The removed test.yaml workflow used cache: npm with setup-node. The new workflow omits it, so every matrix run does a cold npm ci (cold install of the full harper + native sharp binaries). Adding cache: npm to this step:

      - name: Set up Node.js
        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
        with:
          node-version: ${{ matrix.node-version }}
          cache: npm

...will cache the npm download cache between runs, reducing install time significantly (especially for sharp with its optional native binaries). Low-effort win.


- name: Install dependencies
run: npm ci

- name: Build TypeScript
run: npm run build

- name: Run integration tests
run: npm run test:integration

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old npm test (unit tests) is no longer run in CI

The removed test.yaml workflow ran npm test, which executes api/test/resources.test.js. That file still exists and includes a test case not covered by the new integration suite: uploading via a ReadableStream (fs.createReadStream + duplex: 'half'). The new workflow runs only npm run test:integration.

Consider adding a step before this one to also run npm test (or restructure so both test suites run), otherwise the stream-upload path has no CI coverage.

env:
HARPER_INTEGRATION_TEST_LOG_DIR: /tmp/harper-test-logs
FORCE_COLOR: '1'

- name: Upload Harper logs on failure
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: harper-logs-node-${{ matrix.node-version }}
path: /tmp/harper-test-logs/
retention-days: 7
79 changes: 0 additions & 79 deletions .github/workflows/test.yaml

This file was deleted.

12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ flowchart TD
## Getting Started

```
git clone https://github.com/HarperDB/image-optimizer.git
git clone https://github.com/HarperFast/image-optimizer.git

cd image-optimizer

Expand Down Expand Up @@ -95,7 +95,7 @@ Or in Postman:

- Set method to GET
- Set URL to `http://localhost:9926/ImageVariant?id=abc123_400_2_webp`
- In Authorization tab, select "Basic Auth" and enter your HarperDB username and password
- In Authorization tab, select "Basic Auth" and enter your Harper username and password

### Upload an image (POST)

Expand All @@ -114,7 +114,7 @@ Or in Postman:
- Set URL to `http://localhost:9926/Images`
- In Body, select `binary` and choose your image file
- Set header `Content-Type: image/png` (or your image type)
- In Authorization tab, select "Basic Auth" and enter your HarperDB username and password
- In Authorization tab, select "Basic Auth" and enter your Harper username and password

### Upload or update an image with a specific ID (PUT)

Expand All @@ -133,15 +133,15 @@ Or in Postman:
- Set URL to `http://localhost:9926/Images?id=IMAGE_ID`
- In Body, select `binary` and choose your image file
- Set header `Content-Type: image/png` (or your image type)
- In Authorization tab, select "Basic Auth" and enter your HarperDB username and password
- In Authorization tab, select "Basic Auth" and enter your Harper username and password

## Testing

This application uses Node.js' built-in test runner for integration tests. The tests interact with a running Harper instance and the API endpoints, simulating real-world usage. Key flows covered include image upload, variant generation and caching, image updates, and error handling.

- **Test Environment Setup:**
- Harper is started and configured before tests run (see `.github/workflows/test.yaml` for CI setup).
- The required database (`ImageOptimization`) and tables (`images`, `image_variants`) are created automatically using the HarperDB operations API.
- The required database (`ImageOptimization`) and tables (`images`, `image_variants`) are created automatically via the Harper operations API.
- A valid test image is placed in the test directory and used for upload scenarios.

- **Test Execution:**
Expand All @@ -154,7 +154,7 @@ This application uses Node.js' built-in test runner for integration tests. The t
- Run `npm test` to execute the test suite.

- **Running Tests in CI:**
- The GitHub Actions workflow (`.github/workflows/test.yaml`) automates HarperDB setup, database/table creation, and test execution.
- The GitHub Actions workflow (`.github/workflows/integration-tests.yml`) automates Harper setup, database/table creation, and test execution.
- Logs are uploaded for debugging if any tests fail.

> **Note:** Integration tests require a valid image file named test-image.png in the test directory. Make sure this file exists and is a real image (not empty or corrupted) before running tests locally or in CI.
2 changes: 1 addition & 1 deletion api/harper.d.ts
Original file line number Diff line number Diff line change
@@ -1 +1 @@
declare module 'harperdb';
declare module 'harper';
7 changes: 4 additions & 3 deletions api/resources.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import sharp from 'sharp';
import { Resource, databases, logger } from 'harperdb';
import { Resource, databases, logger, createBlob } from 'harper';
import { parseCacheKey, formatToContentType } from './utils/index.js';
import type { User } from './types/index.js';
import { randomUUID } from 'crypto';
Expand Down Expand Up @@ -302,7 +302,7 @@ export class Images extends Resource {
for (const variant of variants || []) {
const vId = (variant as any)?.id ?? variant;
if (typeof vId === 'string') {
await VariantsTable.delete(vId);
await VariantsTable.invalidate(vId);
}
}
Comment on lines 302 to 307

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The loop performs sequential asynchronous invalidate operations on each variant. If an image has many cached variants, this can significantly slow down the PUT request response time. Running these invalidations in parallel using Promise.all will improve performance.

			const invalidations = (variants || []).map(async (variant) => {
				const vId = (variant as any)?.id ?? variant;
				if (typeof vId === 'string') {
					await VariantsTable.invalidate(vId);
				}
			});
			await Promise.all(invalidations);

} catch (err: any) {
Expand All @@ -317,4 +317,5 @@ export class Images extends Resource {
}
}

VariantsTable.sourcedFrom(ImagesTable);
// VariantsTable variant generation is handled manually in ImageVariant.get();
// sourcedFrom is not used here because variant IDs differ from image IDs.
2 changes: 1 addition & 1 deletion api/utils/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export function parseCacheKey(rawId: string) {
const [imageId, widthRaw, dprRaw, formatRaw] = parts;
const format = formatRaw?.toLowerCase() as VariantFormat;

if (!imageId || !/^[a-z0-9_-]+$/i.test(imageId)) return null;
if (!imageId || !/^[a-z0-9-]+$/i.test(imageId)) return null;

try {
assertFormat(format);
Expand Down
Loading