diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 013d795b6..e42c280d1 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -63,6 +63,9 @@ * [Consuming Substreams](how-to-guides/sinks/sinks.md) * [Hosted Sinks](how-to-guides/sinks/hosted-sinks/hosted-sinks.md) * [Managing Your Sink](how-to-guides/sinks/hosted-sinks/manage-your-sink.md) + * [Supabase](how-to-guides/sinks/hosted-sinks/supabase.md) + * [Neon](how-to-guides/sinks/hosted-sinks/neon.md) + * [ClickHouse Cloud](how-to-guides/sinks/hosted-sinks/clickhouse-cloud.md) * [Substreams:SQL](how-to-guides/sinks/sql/sql.md) * [Using Relational Mappings](how-to-guides/sinks/sql/relational-mappings.md) * [Using Database Changes](how-to-guides/sinks/sql/db_out.md) diff --git a/docs/how-to-guides/sinks/hosted-sinks/clickhouse-cloud.md b/docs/how-to-guides/sinks/hosted-sinks/clickhouse-cloud.md new file mode 100644 index 000000000..c793fe32d --- /dev/null +++ b/docs/how-to-guides/sinks/hosted-sinks/clickhouse-cloud.md @@ -0,0 +1,78 @@ +# Hosted Sink: ClickHouse Cloud + +This walkthrough connects a Hosted Sink to a [ClickHouse Cloud](https://clickhouse.com/cloud) managed ClickHouse service. + +## 1. Create a ClickHouse Cloud Service + +1. Sign in to [clickhouse.cloud](https://clickhouse.cloud) and open your organization. +2. Click **New service**, choose a cloud provider and region close to StreamingFast's infrastructure, and select a service tier. Hosted Sinks are provided from us-central1 in Iowa. +3. Set the **default user** password during creation — save it immediately, it is only shown once. +4. Wait for the service to reach **Running** status (typically 1–3 minutes). + +## 2. Get Connection Credentials + +In your service, go to **Connect → Native (TCP)**. + +| Field | Example | +|---|---| +| Host | `abc123def456.us-east-1.aws.clickhouse.cloud` | +| Port | `9440` (native TLS) | +| Database | `default` | +| User | `default` | +| Password | Set during service creation | + +{% hint style="info" %} +ClickHouse Cloud exposes the native protocol on port **9440** (TLS) rather than the standard `9000`. Enable **Secure (TLS)** in the sink configuration. +{% endhint %} + +## 3. Open the IP Access List + +By default, ClickHouse Cloud restricts inbound connections. You must allow StreamingFast's infrastructure to reach your service. Hosted Sinks runs in **us-central1 (Iowa)**. + +1. In your service, go to **Settings → Security → IP Access List**. +2. Click **Add entry** and add `0.0.0.0/0` to allow connections from any IP. Note, if you require a restricted IP range, please contact us at support@streamingfast.io. +3. Save the access list. + +## 4. Create a Dedicated User and Database (Recommended) + +Connect to your service using the ClickHouse Cloud SQL console (**Connect → SQL console**) and run: + +```sql +CREATE DATABASE IF NOT EXISTS substreams; + +CREATE USER substreams_sink + IDENTIFIED BY 'a-strong-password'; + +GRANT SELECT, INSERT, CREATE TABLE, DROP TABLE, ALTER, TRUNCATE + ON substreams.* + TO substreams_sink; +``` + +Replace `substreams` with your intended database name. + +## 5. Configure the Hosted Sink + +In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: + +| Field | Value | +|---|---| +| Host | `..aws.clickhouse.cloud` | +| Port | `9440` | +| Database | `substreams` (or your database name) | +| User | `substreams_sink` | +| Password | Password for that user | +| Secure (TLS) | **Enabled** | + +## 6. Deploy + +Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your ClickHouse Cloud service and begin writing data. + +## Troubleshooting + +**Connection refused** — Confirm the IP Access List includes StreamingFast's egress IPs or is set to allow `0.0.0.0/0`. Also confirm port `9440` is selected, not `9000`. + +**Authentication failed** — Double-check the user and password. ClickHouse Cloud passwords are case-sensitive. If you reset the default user password in the console, the old credential is immediately invalidated. + +**TLS handshake error** — Ensure **Secure (TLS)** is enabled. ClickHouse Cloud's native port `9440` requires TLS; connections without it will fail. + +**Database or table not found** — Ensure the sink user has been granted `CREATE TABLE` on the target database so the sink can initialize the schema on first run. diff --git a/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md b/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md index bbe8e3f27..e0925bcd2 100644 --- a/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md +++ b/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md @@ -38,7 +38,8 @@ Choose how you want to supply the `.spkg`: ### 3. Output -Choose **Postgres** or **ClickHouse** and fill in the connection details. +Choose **Postgres** or **ClickHouse** and fill in the connection details. +See walkthroughs provided below for different SQL services you can rely on, with guided setups to run your Hosted Sink. **Postgres fields:** @@ -86,6 +87,17 @@ Credentials are stored securely and used only to connect the sink process to you Click **Deploy sink**. The sink enters the `Deploying` state while the infrastructure provisions. Once pods are ready it transitions to `Deployed` and begins indexing. +## Provider Walkthroughs + +Step-by-step guides for connecting a Hosted Sink to common managed database providers: + +**Postgres** +- [Supabase](./supabase.md) — managed Postgres with a generous free tier. +- [Neon](./neon.md) — serverless Postgres that scales to zero. + +**ClickHouse** +- [ClickHouse Cloud](./clickhouse-cloud.md) — fully managed ClickHouse from the creators of ClickHouse. + ## Next Steps - [Monitor and manage your sink](./manage-your-sink.md) — check status, view logs, edit config, stop, or delete. diff --git a/docs/how-to-guides/sinks/hosted-sinks/neon.md b/docs/how-to-guides/sinks/hosted-sinks/neon.md new file mode 100644 index 000000000..d421f14db --- /dev/null +++ b/docs/how-to-guides/sinks/hosted-sinks/neon.md @@ -0,0 +1,76 @@ +# Hosted Sink: Neon + +This walkthrough connects a Hosted Sink to a [Neon](https://neon.tech) serverless Postgres database. + +## 1. Create a Neon Project + +1. Sign in to [console.neon.tech](https://console.neon.tech) and click **New project**. +2. Choose a name, Postgres version, and a region close to StreamingFast's infrastructure. +3. Neon creates a default database (`neondb`) and branch (`main`) automatically. + +## 2. Get Connection Credentials + +In your project, go to the **Dashboard** and open the **Connection Details** panel. Select: +- **Branch**: `main` +- **Database**: your target database (default: `neondb`) +- **Role**: `neondb_owner` or a role you create + +The connection string looks like: + +``` +postgres://:@.neon.tech/neondb?sslmode=require +``` + +Pull the individual fields from it: + +| Field | Example | +|---|---| +| Host | `ep-quiet-forest-a1b2c3d4.us-east-2.aws.neon.tech` | +| Port | `5432` | +| Database | `neondb` | +| User | `neondb_owner` | +| Password | shown once — copy it now | + +## 3. Create a Dedicated Role and Schema (Recommended) + +Open the Neon **SQL Editor** and run: + +```sql +CREATE ROLE substreams_sink WITH LOGIN PASSWORD 'a-strong-password'; +CREATE SCHEMA IF NOT EXISTS substreams; +GRANT ALL PRIVILEGES ON SCHEMA substreams TO substreams_sink; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA substreams TO substreams_sink; +ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink; +``` + +Replace `substreams` with your intended schema name. + +## 4. Configure the Hosted Sink + +In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: + +| Field | Value | +|---|---| +| Host | `ep-..aws.neon.tech` | +| Port | `5432` | +| Database | `neondb` (or your database name) | +| Schema | `substreams` (or the schema you created above) | +| User | `substreams_sink` | +| Password | Password for that role | +| SSL Mode | `require` | + +{% hint style="warning" %} +Neon endpoints **suspend after a period of inactivity** on the free tier. A Hosted Sink running continuously will keep the endpoint active, but if the sink is stopped for an extended period the first reconnect may be slow. Consider upgrading to a paid Neon plan for production workloads. +{% endhint %} + +## 5. Deploy + +Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your Neon database and begin writing data. + +## Troubleshooting + +**Endpoint suspended on first connect** — Neon free-tier endpoints auto-suspend. The sink will retry on reconnect but may log an initial connection error. This is transient; the endpoint wakes within a few seconds. + +**SSL required** — Neon requires SSL. Set SSL Mode to `require`; `disable` will be rejected. + +**Permission denied on new tables** — Run `ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink;` so that tables created by the sink are automatically accessible to the role. diff --git a/docs/how-to-guides/sinks/hosted-sinks/supabase.md b/docs/how-to-guides/sinks/hosted-sinks/supabase.md new file mode 100644 index 000000000..ac27374c3 --- /dev/null +++ b/docs/how-to-guides/sinks/hosted-sinks/supabase.md @@ -0,0 +1,71 @@ +# Hosted Sink: Supabase + +This walkthrough connects a Hosted Sink to a [Supabase](https://supabase.com) Postgres database. + +## 1. Create a Supabase Project + +1. Sign in to [supabase.com](https://supabase.com) and open your organization. +2. Click **New project**, choose a region close to StreamingFast's infrastructure, and set a strong database password. +3. Wait for the project to finish provisioning (roughly 1–2 minutes). + +## 2. Get Connection Credentials + +In your project, go to **Settings → Database → Connection parameters**. + +| Supabase field | Value to copy | +|---|---| +| Host | `db..supabase.co` | +| Port | `5432` | +| Database name | `postgres` (default) | +| Username | `postgres` (or a dedicated user you create) | +| Password | The password you set during project creation | + +{% hint style="info" %} +Use the **direct connection** host (`db..supabase.co`) rather than the Session or Transaction pooler hosts. The pooler ports (5432/6543 on `aws-0-.pooler.supabase.com`) use PgBouncer, which can cause issues with the sink's prepared statements. +{% endhint %} + +## 3. Create a Dedicated Database User (Recommended) + +Run the following in the Supabase **SQL Editor** to create a user scoped to your target schema: + +```sql +CREATE USER substreams_sink WITH PASSWORD 'a-strong-password'; +CREATE SCHEMA IF NOT EXISTS substreams; +GRANT ALL PRIVILEGES ON SCHEMA substreams TO substreams_sink; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA substreams TO substreams_sink; +ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink; +``` + +Replace `substreams` with your intended schema name. + +## 4. Allow External Connections + +Supabase Postgres accepts connections from the public internet by default. No additional firewall steps are needed for the Hosted Sink to reach your database. + +## 5. Configure the Hosted Sink + +In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: + +| Field | Value | +|---|---| +| Host | `db..supabase.co` | +| Port | `5432` | +| Database | `postgres` | +| Schema | `substreams` (or the schema you created above) | +| User | `substreams_sink` | +| Password | Password for that user | +| SSL Mode | `require` | + +Supabase requires SSL — use `require` at minimum. Use `verify-full` if you want certificate validation. + +## 6. Deploy + +Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your Supabase database and begin writing data. + +## Troubleshooting + +**Connection refused / timeout** — Confirm you are using the direct connection host, not the pooler host. Check that port `5432` is not blocked by any additional network policy in your Supabase project (**Settings → Database → Network restrictions**). + +**Permission denied** — Ensure the sink user has been granted privileges on the target schema and that `ALTER DEFAULT PRIVILEGES` was applied so future tables are also accessible. + +**SSL errors** — Supabase requires SSL. Set SSL Mode to `require` or higher; `disable` will be rejected.