-
Notifications
You must be signed in to change notification settings - Fork 90
docs(hosted-sinks): add provider walkthroughs for Supabase, Neon, and ClickHouse Cloud #807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Changes from all commits
6030b57
a4bf765
2017c2b
f54fc99
5a12a9b
0122bb3
47839e5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Hosted Sink: ClickHouse Cloud | ||
|
|
||
| This walkthrough connects a Hosted Sink to a [ClickHouse Cloud](https://clickhouse.com/cloud) managed ClickHouse service. | ||
|
|
||
| ## 1. Create a ClickHouse Cloud Service | ||
|
|
||
| 1. Sign in to [clickhouse.cloud](https://clickhouse.cloud) and open your organization. | ||
| 2. Click **New service**, choose a cloud provider and region close to StreamingFast's infrastructure, and select a service tier. Hosted Sinks are provided from us-central1 in Iowa. | ||
| 3. Set the **default user** password during creation — save it immediately, it is only shown once. | ||
| 4. Wait for the service to reach **Running** status (typically 1–3 minutes). | ||
|
|
||
| ## 2. Get Connection Credentials | ||
|
|
||
| In your service, go to **Connect → Native (TCP)**. | ||
|
|
||
| | Field | Example | | ||
| |---|---| | ||
| | Host | `abc123def456.us-east-1.aws.clickhouse.cloud` | | ||
| | Port | `9440` (native TLS) | | ||
| | Database | `default` | | ||
| | User | `default` | | ||
| | Password | Set during service creation | | ||
|
|
||
| {% hint style="info" %} | ||
| ClickHouse Cloud exposes the native protocol on port **9440** (TLS) rather than the standard `9000`. Enable **Secure (TLS)** in the sink configuration. | ||
| {% endhint %} | ||
|
|
||
| ## 3. Open the IP Access List | ||
|
|
||
| By default, ClickHouse Cloud restricts inbound connections. You must allow StreamingFast's infrastructure to reach your service. Hosted Sinks runs in **us-central1 (Iowa)**. | ||
|
|
||
| 1. In your service, go to **Settings → Security → IP Access List**. | ||
| 2. Click **Add entry** and add `0.0.0.0/0` to allow connections from any IP. Note, if you require a restricted IP range, please contact us at support@streamingfast.io. | ||
| 3. Save the access list. | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Everywhere we have this 0.0.0.0, add a hint to contact us if someone has increased enterprise needs where they need to restrict to a given IP range
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added here. Reviewing if found in other pages. |
||
| ## 4. Create a Dedicated User and Database (Recommended) | ||
|
|
||
| Connect to your service using the ClickHouse Cloud SQL console (**Connect → SQL console**) and run: | ||
|
|
||
| ```sql | ||
| CREATE DATABASE IF NOT EXISTS substreams; | ||
|
|
||
| CREATE USER substreams_sink | ||
| IDENTIFIED BY 'a-strong-password'; | ||
|
|
||
| GRANT SELECT, INSERT, CREATE TABLE, DROP TABLE, ALTER, TRUNCATE | ||
| ON substreams.* | ||
| TO substreams_sink; | ||
| ``` | ||
|
|
||
| Replace `substreams` with your intended database name. | ||
|
|
||
| ## 5. Configure the Hosted Sink | ||
|
|
||
| In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: | ||
|
|
||
| | Field | Value | | ||
| |---|---| | ||
| | Host | `<hash>.<region>.aws.clickhouse.cloud` | | ||
| | Port | `9440` | | ||
| | Database | `substreams` (or your database name) | | ||
| | User | `substreams_sink` | | ||
| | Password | Password for that user | | ||
| | Secure (TLS) | **Enabled** | | ||
|
|
||
| ## 6. Deploy | ||
|
|
||
| Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your ClickHouse Cloud service and begin writing data. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| **Connection refused** — Confirm the IP Access List includes StreamingFast's egress IPs or is set to allow `0.0.0.0/0`. Also confirm port `9440` is selected, not `9000`. | ||
|
|
||
| **Authentication failed** — Double-check the user and password. ClickHouse Cloud passwords are case-sensitive. If you reset the default user password in the console, the old credential is immediately invalidated. | ||
|
|
||
| **TLS handshake error** — Ensure **Secure (TLS)** is enabled. ClickHouse Cloud's native port `9440` requires TLS; connections without it will fail. | ||
|
|
||
| **Database or table not found** — Ensure the sink user has been granted `CREATE TABLE` on the target database so the sink can initialize the schema on first run. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # Hosted Sink: Neon | ||
|
|
||
| This walkthrough connects a Hosted Sink to a [Neon](https://neon.tech) serverless Postgres database. | ||
|
|
||
| ## 1. Create a Neon Project | ||
|
|
||
| 1. Sign in to [console.neon.tech](https://console.neon.tech) and click **New project**. | ||
| 2. Choose a name, Postgres version, and a region close to StreamingFast's infrastructure. | ||
| 3. Neon creates a default database (`neondb`) and branch (`main`) automatically. | ||
|
|
||
| ## 2. Get Connection Credentials | ||
|
|
||
| In your project, go to the **Dashboard** and open the **Connection Details** panel. Select: | ||
| - **Branch**: `main` | ||
| - **Database**: your target database (default: `neondb`) | ||
| - **Role**: `neondb_owner` or a role you create | ||
|
|
||
| The connection string looks like: | ||
|
|
||
| ``` | ||
| postgres://<user>:<password>@<endpoint-host>.neon.tech/neondb?sslmode=require | ||
| ``` | ||
|
|
||
| Pull the individual fields from it: | ||
|
|
||
| | Field | Example | | ||
| |---|---| | ||
| | Host | `ep-quiet-forest-a1b2c3d4.us-east-2.aws.neon.tech` | | ||
| | Port | `5432` | | ||
| | Database | `neondb` | | ||
| | User | `neondb_owner` | | ||
| | Password | shown once — copy it now | | ||
|
|
||
| ## 3. Create a Dedicated Role and Schema (Recommended) | ||
|
|
||
| Open the Neon **SQL Editor** and run: | ||
|
|
||
| ```sql | ||
| CREATE ROLE substreams_sink WITH LOGIN PASSWORD 'a-strong-password'; | ||
| CREATE SCHEMA IF NOT EXISTS substreams; | ||
| GRANT ALL PRIVILEGES ON SCHEMA substreams TO substreams_sink; | ||
| GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA substreams TO substreams_sink; | ||
| ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink; | ||
| ``` | ||
|
|
||
| Replace `substreams` with your intended schema name. | ||
|
|
||
| ## 4. Configure the Hosted Sink | ||
|
|
||
| In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: | ||
|
|
||
| | Field | Value | | ||
| |---|---| | ||
| | Host | `ep-<name>.<region>.aws.neon.tech` | | ||
| | Port | `5432` | | ||
| | Database | `neondb` (or your database name) | | ||
| | Schema | `substreams` (or the schema you created above) | | ||
| | User | `substreams_sink` | | ||
| | Password | Password for that role | | ||
| | SSL Mode | `require` | | ||
|
|
||
| {% hint style="warning" %} | ||
| Neon endpoints **suspend after a period of inactivity** on the free tier. A Hosted Sink running continuously will keep the endpoint active, but if the sink is stopped for an extended period the first reconnect may be slow. Consider upgrading to a paid Neon plan for production workloads. | ||
| {% endhint %} | ||
|
|
||
| ## 5. Deploy | ||
|
|
||
| Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your Neon database and begin writing data. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| **Endpoint suspended on first connect** — Neon free-tier endpoints auto-suspend. The sink will retry on reconnect but may log an initial connection error. This is transient; the endpoint wakes within a few seconds. | ||
|
|
||
| **SSL required** — Neon requires SSL. Set SSL Mode to `require`; `disable` will be rejected. | ||
|
|
||
| **Permission denied on new tables** — Run `ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink;` so that tables created by the sink are automatically accessible to the role. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Hosted Sink: Supabase | ||
|
|
||
| This walkthrough connects a Hosted Sink to a [Supabase](https://supabase.com) Postgres database. | ||
|
|
||
| ## 1. Create a Supabase Project | ||
|
|
||
| 1. Sign in to [supabase.com](https://supabase.com) and open your organization. | ||
| 2. Click **New project**, choose a region close to StreamingFast's infrastructure, and set a strong database password. | ||
| 3. Wait for the project to finish provisioning (roughly 1–2 minutes). | ||
|
|
||
| ## 2. Get Connection Credentials | ||
|
|
||
| In your project, go to **Settings → Database → Connection parameters**. | ||
|
|
||
| | Supabase field | Value to copy | | ||
| |---|---| | ||
| | Host | `db.<project-ref>.supabase.co` | | ||
| | Port | `5432` | | ||
| | Database name | `postgres` (default) | | ||
| | Username | `postgres` (or a dedicated user you create) | | ||
| | Password | The password you set during project creation | | ||
|
|
||
| {% hint style="info" %} | ||
| Use the **direct connection** host (`db.<project-ref>.supabase.co`) rather than the Session or Transaction pooler hosts. The pooler ports (5432/6543 on `aws-0-<region>.pooler.supabase.com`) use PgBouncer, which can cause issues with the sink's prepared statements. | ||
| {% endhint %} | ||
|
|
||
| ## 3. Create a Dedicated Database User (Recommended) | ||
|
|
||
| Run the following in the Supabase **SQL Editor** to create a user scoped to your target schema: | ||
|
|
||
| ```sql | ||
| CREATE USER substreams_sink WITH PASSWORD 'a-strong-password'; | ||
| CREATE SCHEMA IF NOT EXISTS substreams; | ||
| GRANT ALL PRIVILEGES ON SCHEMA substreams TO substreams_sink; | ||
| GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA substreams TO substreams_sink; | ||
| ALTER DEFAULT PRIVILEGES IN SCHEMA substreams GRANT ALL ON TABLES TO substreams_sink; | ||
| ``` | ||
|
|
||
| Replace `substreams` with your intended schema name. | ||
|
|
||
| ## 4. Allow External Connections | ||
|
|
||
| Supabase Postgres accepts connections from the public internet by default. No additional firewall steps are needed for the Hosted Sink to reach your database. | ||
|
|
||
| ## 5. Configure the Hosted Sink | ||
|
|
||
| In [The Graph Market](https://thegraph.market/sinks/new), create a new sink and fill in the **Output** section: | ||
|
|
||
| | Field | Value | | ||
| |---|---| | ||
| | Host | `db.<project-ref>.supabase.co` | | ||
| | Port | `5432` | | ||
| | Database | `postgres` | | ||
| | Schema | `substreams` (or the schema you created above) | | ||
| | User | `substreams_sink` | | ||
| | Password | Password for that user | | ||
| | SSL Mode | `require` | | ||
|
|
||
| Supabase requires SSL — use `require` at minimum. Use `verify-full` if you want certificate validation. | ||
|
|
||
| ## 6. Deploy | ||
|
|
||
| Complete the remaining sink configuration (package, execution settings) and click **Deploy sink**. The sink will connect to your Supabase database and begin writing data. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| **Connection refused / timeout** — Confirm you are using the direct connection host, not the pooler host. Check that port `5432` is not blocked by any additional network policy in your Supabase project (**Settings → Database → Network restrictions**). | ||
|
|
||
| **Permission denied** — Ensure the sink user has been granted privileges on the target schema and that `ALTER DEFAULT PRIVILEGES` was applied so future tables are also accessible. | ||
|
|
||
| **SSL errors** — Supabase requires SSL. Set SSL Mode to `require` or higher; `disable` will be rejected. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it could be turn into a
mailto:link which usually in browser auto open email client, easy to check on GitBook preview