utils for agents, sql, redis, etc

.ai/ai-agents.md

@@ -0,0 +1,29 @@
+# AI Agents
+
+We have a custom package for making AI Agents that uses the OpenAI Agents SDK (`packages/agentic-workflows`).
+
+## Agent Service
+
+The core is an agent service (`packages/agentic-workflows/src/service/ai-agent-service.ts`) with multiple
+implementations:
+
+- `ai-agent-service-openai`: production implementation using the OpenAI API
+- `ai-agent-service-local`: mocked implementation for testing non-AI portions of workflows (orchestration, task
+  management, error handling, etc)
+- `ai-agent-service-ollama`: hits a local ollama deployment for development/testing with local models
+
+## Tools
+
+Tools can be passed into `AgentConfig` to give agents access to external capabilities:
+
+```ts
+tools: [getWebContentTool({ tavilyService: this.tavilyService })],
+```
+
+Tool definitions live in `packages/tavily-utils/src/tools/`.
+
+## Update Checklist
+
+Every time you update an agent, you must also update:
+- Any fixtures
+- Any tests that assume a certain agent structure

.ai/bull-queues.md

@@ -0,0 +1,92 @@
+# BullMQ Crons & Async Workers
+
+We use "bullmq" to run cron jobs and async workflows. Bull uses Redis under the hood.
+
+## Prerequisites
+
+The consuming service must have a configured redis instance.
+For testing, boot a redis container:
+
+```ts
+const redisContainer = new RedisContainer();
+await redisContainer.start();
+```
+
+Our utilities for interacting with Bull: `packages/commons/src/bull`.
+
+## Queue + Worker Pattern
+
+Every bull queue must have a `*-queue` and a `*-worker`.
+
+### Queue
+
+A queue wraps a BullMQ `Queue` and exposes an `addAndAwait()` method for enqueuing jobs:
+
+```ts
+@Injectable()
+export class ExampleQueue {
+  private queue: Queue;
+
+  constructor(private redisConnectionConfig: RedisConnectionConfig) {
+    this.queue = new Queue("example-queue", {
+      connection: redisConnectionConfig.getConnection()
+    });
+  }
+
+  async addAndAwait(data: ExampleRequest): Promise<ExampleResponse> {
+    return await this.queue.add("example-job", data as any);
+  }
+}
+```
+
+### Worker
+
+A worker processes jobs from the queue:
+
+```ts
+@Injectable()
+export class ExampleWorker {
+  private worker: Worker;
+
+  constructor(private redisConnectionConfig: RedisConnectionConfig) {}
+
+  start(): void {
+    this.worker = new Worker("example-queue", async (job) => {
+      // process job.data
+    }, {
+      connection: this.redisConnectionConfig.getConnection()
+    });
+  }
+}
+```
+
+## Cron Jobs
+
+The queue is injected and launched via `CronLauncher`:
+
+```ts
+this.cronLauncher = new CronLauncher([
+  {
+    queue: exampleCronQueue,
+    pattern: config.cronPattern,
+    interval: config.interval,
+    jobId: EXAMPLE_CRON_JOB_ID
+  }
+]);
+this.cronLauncher.launchCrons();
+```
+
+**All queues must have globally unique names in the codebase.**
+
+## Async Processes
+
+Bull is also used for general async processing of multi-stage workflows with retries.
+
+Trigger an async task:
+
+```ts
+await this.exampleQueue.addAndAwait(request);
+```
+
+This adds an item to the queue, triggers the worker, and returns a response. There are default retry mechanisms on any
+bull worker set up like this.

.ai/configuration.md

@@ -0,0 +1,45 @@
+# Configuration
+
+## Configuration Loading
+
+Configuration is read via `packages/commons/src/config/config-utils.ts`. This defaults to reading from a YAML file at
+the application root (e.g., `assets/config.yaml`).
+
+Overrides can be set as environment variables with the `app_` prefix:
+
+```json
+{
+  "name": "app_remoteClient___port",
+  "value": "8080"
+},
+{
+  "name": "app_remoteClient___address",
+  "value": "localhost"
+}
+```
+
+### Type Suffixes
+
+Env vars are parsed as strings. For typed values, use suffixes:
+- `_numeric`: `"app_defaultTokenExpirySeconds_numeric"` → `600`
+- `_boolean`: `"app_someFeatureEnabled_boolean"` → `true`
+
+## Dynamic Configuration
+
+`*DynamicConfig` classes are simple, injectable config holders that:
+
+- Load initial values from YAML at startup via `loadConfig(...)`.
+- Expose **getter** methods for production code.
+- Expose **setter** methods so tests can override values at runtime.
+
+Typical structure:
+- Static `CONFIG_KEY` field pointing to the YAML section
+- Private fields for each config value
+- Constructor: `loadConfig` using `ApplicationConfig.applicationRoot` and `CONFIG_KEY`
+- Public getters and setters mirroring those fields
+
+Best practices:
+- Always save and restore original values (use `try/finally`) in tests.
+- Keep fields private, exposed only via getters/setters.
+- Use consistent naming: `getX()` / `setX(...)`.
+- Always mark these classes as `@Injectable()`.

.ai/dependency-injection.md

@@ -0,0 +1,74 @@
+# NestJS Dependency Injection
+
+All dependencies in our object graph are managed via NestJS.
+We add the `@Injectable()` tag to all classes that are created/managed via DI.
+
+## Module Structure
+
+We create a `Module` for each logical group of functionality:
+
+```ts
+import { Closer, ConfigModule, PostgresConnectionModule } from "@zeroshotbuilders/commons";
+import { Module } from "@nestjs/common";
+import { ExampleConfig } from "./config/example-config";
+import { ExampleDao } from "./repository/example-dao";
+import { ExampleRepository } from "./repository/example-repository";
+import { ExampleServiceImpl } from "./service/example-service-impl";
+
+@Module({
+  providers: [
+    ExampleServiceImpl,
+    ExampleConfig,
+    Closer,
+    ExampleDao,
+    ExampleRepository
+  ],
+  imports: [
+    ConfigModule.forApplicationRoot(__dirname),
+    PostgresConnectionModule.forApplicationRoot(__dirname)
+  ]
+})
+export class ExampleModule {}
+```
+
+## Provider Registration
+
+All classes that participate in DI must be:
+1. Decorated with `@Injectable()`
+2. Listed in the `providers` array of their module
+3. Constructor-injected with their dependencies
+
+```ts
+@Injectable()
+export class ExampleRepository {
+  constructor(
+    private readonly exampleDao: ExampleDao,
+    private readonly config: ExampleConfig
+  ) {}
+}
+```
+
+## Cron Job Wiring
+
+Services that need cron jobs wire them up via `CronLauncher`:
+
+```ts
+@Injectable()
+export class ServiceComponent {
+  constructor(
+    private readonly cronQueue: ExampleCronQueue,
+    private readonly config: ExampleConfig,
+    private readonly closer: Closer
+  ) {
+    this.cronLauncher = new CronLauncher([
+      {
+        queue: cronQueue,
+        pattern: config.cronPattern,
+        interval: config.interval,
+        jobId: EXAMPLE_CRON_JOB_ID
+      }
+    ]);
+    this.cronLauncher.launchCrons();
+  }
+}
+```

.ai/skills/testing-guide.md

@@ -0,0 +1,57 @@
+---
+name: testing-guide
+description: Reference guide for writing integration and unit tests
+---
+
+Reference guide for writing integration and unit tests.
+
+Read these docs for full details:
+- `.ai/testing.md` for integration test structure, unit tests, and testcontainers
+- `.ai/configuration.md` for dynamic config and overriding config in tests
+
+## Quick Reference
+
+### Integration Test Structure
+- Boot dependencies via testcontainers (`PostgresContainer`, `RedisContainer` from `@zeroshotbuilders/commons-testing`)
+- Wire into a NestJS test module using `.overrideProvider()`
+- Run assertions against the real stack
+
+### Testcontainers
+```ts
+import { PostgresContainer, RedisContainer } from "@zeroshotbuilders/commons-testing";
+
+const postgresContainer = new PostgresContainer();
+await postgresContainer.start();
+
+const redisContainer = new RedisContainer();
+await redisContainer.start();
+```
+
+### Overriding Config in Tests
+```ts
+const testModule = await Test.createTestingModule({
+  imports: [ServiceModule],
+})
+  .overrideProvider(RedisConnectionConfig)
+  .useValue(redisContainer.getConnectionConfig())
+  .overrideProvider(PostgresConnectionConfig)
+  .useValue(postgresContainer.getConnectionConfig())
+  .compile();
+```
+
+### Dynamic Config in Tests
+```ts
+const original = dynamicConfig.getX();
+try {
+  dynamicConfig.setX(testValue);
+  // test logic
+} finally {
+  dynamicConfig.setX(original);
+}
+```
+
+### Unit Tests
+Required for all utility functions, especially in `packages/commons`.
+Good example: `packages/commons/test/port/port-utils.spec.ts`.
+
+$ARGUMENTS

.ai/sql-decorators.md

@@ -0,0 +1,87 @@
+# sql-decorators Framework
+
+We have built a custom in-house framework for interacting with the SQL database. It allows us to use actual SQL queries.
+The logic lives in the `sql-decorators` package. **100% of DML must use this `sql-decorators` construct.** The queries should all live
+in a DAO.
+
+The framework uses "sequelize" to actually execute the queries.
+
+## Implementation Details
+
+For implementation details, look at `packages/sql-decorators/src/decorators.ts`. This is where our main decorators live that
+contain the logic for translating TypeScript functions on the DAO into actual SQL calls. It is also responsible for
+running the actual underlying query and managing parameterized replacements.
+
+For examples of how to use each decorator, look in `packages/sql-decorators/test`.
+
+## Data Access Objects (DAOs)
+
+A DAO encapsulates all interaction with a data source behind a clean interface. DAOs handle queries, inserts, updates,
+deletes, and data-mapping so the rest of the application never deals directly with SQL, ORMs, or database drivers.
+
+In a well-structured backend, services depend on DAOs rather than on raw database primitives.
+
+## DAO Structure
+
+```ts
+@Dao({
+  queryDirectory: `${__dirname}/queries`
+})
+export class NoteDao extends DaoBase {
+  @SqlQuery<NoteModel>({
+    queryType: QueryTypes.SELECT,
+    clazz: NoteModel,
+    returnList: true
+  })
+  getNotes(
+    noteIds: string[] | undefined,
+    customerIds: string[] | undefined
+  ): Promise<List<NoteModel>> {
+    return null;
+  }
+}
+```
+
+## Transactions
+
+When you need to perform multiple inserts, mutations, or anything that would require a SQL Transaction, we have a
+`@SqlTransaction` decorator. This generates a `Transaction` object that must be passed into each of the methods that
+must be part of the same transaction.
+
+Look in `packages/sql-decorators/test/repository/enum-dao.ts` for an example on how to do this.
+
+## Streaming with @StreamSelect
+
+The `@StreamSelect` decorator handles paginating through a database table for large result sets:
+
+```typescript
+@StreamSelect<TreatmentModel>({
+  clazz: TreatmentModel,
+  batchSize: 1000
+})
+streamTreatments(
+  createdSince?: number | undefined
+): StreamIterator<TreatmentModel> {
+  return null; // Implementation is injected by the decorator
+}
+```
+
+### How StreamSelect Works
+
+- **Batching**: Executes the SQL query multiple times with `LIMIT` and `OFFSET` clauses.
+- **Lazy Fetching**: Fetches the first `batchSize` records, then auto-fetches the next batch when consumed.
+- **Termination**: Stops when a batch returns fewer records than `batchSize`.
+
+### CRITICAL: Sort Order
+
+Because `StreamSelect` relies on `LIMIT` and `OFFSET`, **the underlying SQL query MUST have a deterministic `ORDER BY`
+clause.** Without it, records may be duplicated or skipped between batches.
+
+**Always use a unique, immutable column (like `created_at`) in your `ORDER BY` clause for streaming.**
+
+```sql
+SELECT *
+FROM treatments
+WHERE (:createdSince IS NULL OR created_at > :createdSince)
+ORDER BY created_at
+```