diff --git a/src/content/docs/integrations/index.mdx b/src/content/docs/integrations/index.mdx index 9e3899017..21b41734d 100644 --- a/src/content/docs/integrations/index.mdx +++ b/src/content/docs/integrations/index.mdx @@ -34,7 +34,8 @@ communication over numerous protocols and APIs: nats, amazon/sqs, amqp - **Databases**: snowflake, - clickhouse, mysql + clickhouse, mysql, + microsoft/sql-server - **Network protocols**: tcp, udp, http, syslog diff --git a/src/content/docs/integrations/microsoft/sql-server.mdx b/src/content/docs/integrations/microsoft/sql-server.mdx new file mode 100644 index 000000000..04e6f234c --- /dev/null +++ b/src/content/docs/integrations/microsoft/sql-server.mdx @@ -0,0 +1,110 @@ +--- +title: Microsoft SQL Server +--- + +import Op from '@components/see-also/Op.astro'; + +This guide shows you how to read events from Microsoft SQL Server and Azure SQL +Database with Tenzir. + +Use from_microsoft_sql to connect to Microsoft SQL Server over the +Tabular Data Stream (TDS) protocol. The operator can read a full table, execute +a custom query, list table metadata, or poll a table for newly inserted rows. + +## Connect to Azure SQL + +Azure SQL Database accepts TDS connections on port `1433` and requires TLS. Use +the server host name from Azure, usually in the form +`.database.windows.net`, and set `tls=true`. + +```tql +from_microsoft_sql query="SELECT TOP 10 * FROM dbo.events", + host="example.database.windows.net", + port=1433, + user="tenzir_reader", + password=secret("azure-sql-password"), + database="security", + tls=true +``` + +The operator currently supports SQL authentication. Microsoft Entra +authentication, managed identities, Windows authentication, and Kerberos are not +supported. + +## Read from SQL Server + +Use `table` to read every row from a table: + +```tql +from_microsoft_sql table="dbo.events", + host="sql.example.com", + user="tenzir_reader", + password=secret("sql-server-password"), + database="security", + tls=true +``` + +Use `query` or `sql` when you want to select, filter, or join data in SQL +Server before Tenzir receives it: + +```tql +from_microsoft_sql query=r"SELECT id, created_at, source, message + FROM dbo.events + WHERE severity >= 3", + host="sql.example.com", + user="tenzir_reader", + password=secret("sql-server-password"), + database="security", + tls=true +``` + +## Stream new rows + +Set `live=true` to poll a table for newly inserted rows. Live mode uses a +monotonic integer tracking column as a watermark. If you don't specify +`tracking_column`, the operator tries to detect a single integer identity column +or a single integer primary-key column. + +```tql +from_microsoft_sql table="dbo.events", + live=true, + tracking_column="id", + host="sql.example.com", + user="tenzir_reader", + password=secret("sql-server-password"), + database="security", + tls=true +``` + +Live mode initializes its watermark from the current maximum tracking value and +then emits rows with greater values. It doesn't emit an initial snapshot, and it +doesn't capture updates or deletes. + +## Inspect metadata + +List all base tables in the selected database: + +```tql +from_microsoft_sql show="tables", + host="sql.example.com", + user="tenzir_reader", + password=secret("sql-server-password"), + database="security", + tls=true +``` + +List columns for a table: + +```tql +from_microsoft_sql show="columns", + table="dbo.events", + host="sql.example.com", + user="tenzir_reader", + password=secret("sql-server-password"), + database="security", + tls=true +``` + +## See Also + +- from_microsoft_sql diff --git a/src/content/docs/reference/operators.mdx b/src/content/docs/reference/operators.mdx index 405786af5..87d394072 100644 --- a/src/content/docs/reference/operators.mdx +++ b/src/content/docs/reference/operators.mdx @@ -391,6 +391,10 @@ operators: description: 'Reads events from a Microsoft Graph collection.' example: 'from_microsoft_graph "auditLogs/signIns", auth={…}' path: 'reference/operators/from_microsoft_graph' + - name: 'from_microsoft_sql' + description: 'Reads events from Microsoft SQL Server or Azure SQL Database.' + example: 'from_microsoft_sql table="dbo.events", host="sql.example.com", database="security", tls=true' + path: 'reference/operators/from_microsoft_sql' - name: 'from_mysql' description: 'Reads events from a MySQL database.' example: 'from_mysql table="users", host="db.example.com", database="mydb"' @@ -1526,6 +1530,14 @@ from_microsoft_graph "auditLogs/signIns", auth={…} + + +```tql +from_microsoft_sql table="dbo.events", host="sql.example.com", database="security", tls=true +``` + + + ```tql diff --git a/src/content/docs/reference/operators/from_microsoft_sql.mdx b/src/content/docs/reference/operators/from_microsoft_sql.mdx new file mode 100644 index 000000000..43a849c4a --- /dev/null +++ b/src/content/docs/reference/operators/from_microsoft_sql.mdx @@ -0,0 +1,261 @@ +--- +title: from_microsoft_sql +category: Inputs/Events +example: 'from_microsoft_sql table="dbo.events", host="sql.example.com", database="security", tls=true' +--- + +import Op from '@components/see-also/Op.astro'; +import Integration from '@components/see-also/Integration.astro'; + +Reads events from Microsoft SQL Server or Azure SQL Database. + +```tql +from_microsoft_sql [table=string], [sql=string], [query=string], + [show=string], [live=bool], + [tracking_column=string], [host=string], [port=int], + [user=string], [password=secret], + [database=string], [tls=bool|record] +``` + +## Description + +The `from_microsoft_sql` operator connects to Microsoft SQL Server-compatible +databases over TDS. You can read a table, execute a custom query, retrieve +metadata, or poll a table for newly inserted rows. + +The operator supports four query modes: + +1. **Table mode**: Read all rows from a table using the `table` parameter. +2. **SQL mode**: Execute a custom SQL query using the `sql` parameter. +3. **Query mode**: Execute a custom SQL query using the `query` parameter. +4. **Show mode**: List database metadata using the `show` parameter. + When `show="columns"`, also set `table` to the table name. + +### `table = string (optional)` + +The name of the table to read from. Use the `schema.table` form to select a +schema-qualified table, such as `dbo.events`. + +This is mutually exclusive with `sql` and `query`. When `show="columns"`, set +`table` to the table name. + +### `sql = string (optional)` + +A raw SQL query to execute. This is mutually exclusive with `table`, `query`, +and `show`. + +Use raw strings for complex queries: + +```tql +from_microsoft_sql sql=r"SELECT id, created_at FROM dbo.events WHERE severity >= 3" +``` + +### `query = string (optional)` + +A raw SQL query to execute. This is equivalent to `sql` and exists as a more +descriptive spelling. This is mutually exclusive with `table`, `sql`, and +`show`. + +### `show = string (optional)` + +Retrieve database metadata. This is mutually exclusive with `sql`, `query`, and +`live=true`. When `show="columns"`, set `table` to the table name. + +Supported values: + +- `"tables"`: List all base tables in the database. +- `"columns"`: List all columns for the table specified in `table`. + +### `live = bool (optional)` + +Enables continuous polling for new rows from a table. The operator tracks +progress using a watermark on an integer column and polls every second for rows +above the last-seen value. Live mode is mutually exclusive with `sql`, `query`, +and `show`, and requires `table`. + +Defaults to `false`. + +Live mode initializes the watermark from the current maximum tracking value and +then emits rows with greater values. It doesn't emit an initial snapshot, and it +doesn't capture updates or deletes. + +### `tracking_column = string (optional)` + +The integer column to use for watermark tracking in live mode. The operator +queries for rows where this column exceeds the last-seen watermark. + +When omitted, the operator detects a suitable column from the table metadata. It +prefers a single integer identity column and falls back to a single integer +primary-key column. Requires `live=true`. + +### `host = string (optional)` + +The hostname or IP address of the SQL Server. + +Defaults to `"localhost"`. + +### `port = int (optional)` + +The TCP port of the SQL Server. + +Defaults to `1433`. + +### `user = string (optional)` + +The username for SQL authentication. + +Defaults to `"sa"`. + +### `password = secret (optional)` + +The password for SQL authentication. Use `secret` for secure credential +management. + +Defaults to `""`. + +### `database = string (optional)` + +The database to connect to. + +Defaults to `""`. + +### `tls = bool|record (optional)` + +TLS configuration for the SQL Server connection. Defaults to `false` (no TLS). + +Use `tls=true` to enable TLS with default settings and certificate +verification, or provide a record to customize specific options: + +```tql +{ + skip_peer_verification: bool, // Skip certificate verification. + cacert: string, // CA bundle to verify peers. + certfile: string, // Client certificate to present. + keyfile: string, // Private key for the client certificate. +} +``` + +Azure SQL Database requires TLS. Use the Azure SQL server host name, usually in +the form `.database.windows.net`, and set `tls=true`. + +The operator currently supports SQL authentication. Microsoft Entra +authentication, managed identities, Windows authentication, and Kerberos are not +supported. + +## Types + +The operator maps Microsoft SQL Server types to types as +follows: + +| SQL Server type | Tenzir type | Notes | +| :--------------------------------------------- | :---------- | :----------------------------------- | +| `tinyint` | `uint64` | | +| `smallint`, `int`, `bigint` | `int64` | | +| `bit` | `bool` | | +| `real`, `float` | `double` | | +| `decimal`, `numeric` | `double` | May lose precision. | +| `smallmoney`, `money` | `double` | May lose precision. | +| `date`, `smalldatetime`, `datetime` | `time` | | +| `datetime2`, `datetimeoffset` | `time` | The offset isn't preserved. | +| `time` | `duration` | | +| `char`, `varchar`, `text` | `string` | | +| `nchar`, `nvarchar`, `ntext`, `xml` | `string` | | +| `binary`, `varbinary`, `image` | `blob` | | +| `uniqueidentifier` | `string` | | + +## Examples + +### Read all rows from a table + +```tql +from_microsoft_sql table="dbo.users", + host="sql.example.com", + database="app", + tls=true +``` + +### Execute a custom SQL query + +```tql +from_microsoft_sql query=r"SELECT id, name, created_at + FROM dbo.users + WHERE active = 1", + host="sql.example.com", + database="app", + tls=true +``` + +### Use secure credentials + +```tql +from_microsoft_sql table="dbo.events", + host="sql.example.com", + user="tenzir_reader", + password=secret("mssql-password"), + database="security", + tls=true +``` + +### Connect to Azure SQL + +```tql +from_microsoft_sql query="SELECT TOP 10 * FROM dbo.events", + host="example.database.windows.net", + port=1433, + user="tenzir_reader", + password=secret("azure-sql-password"), + database="security", + tls=true +``` + +### List all tables in a database + +```tql +from_microsoft_sql show="tables", + host="sql.example.com", + database="app", + tls=true +``` + +### List columns for a specific table + +```tql +from_microsoft_sql show="columns", + table="dbo.users", + host="sql.example.com", + database="app", + tls=true +``` + +### Stream new rows from a table + +```tql +from_microsoft_sql table="dbo.events", + live=true, + tracking_column="id", + host="sql.example.com", + database="security", + tls=true +``` + +### Connect with TLS but skip peer verification + +```tql +from_microsoft_sql table="dbo.events", + host="sql.example.com", + database="security", + tls={skip_peer_verification: true} +``` + +### Connect with TLS using a CA certificate + +```tql +from_microsoft_sql table="dbo.events", + host="sql.example.com", + database="security", + tls={cacert: "/path/to/ca.pem"} +``` + +## See Also + +- microsoft/sql-server diff --git a/src/sidebar.ts b/src/sidebar.ts index ddbffd657..3527be575 100644 --- a/src/sidebar.ts +++ b/src/sidebar.ts @@ -360,6 +360,7 @@ export const integrations = [ "integrations/microsoft/azure-event-hubs", "integrations/microsoft/defender", "integrations/microsoft/graph", + "integrations/microsoft/sql-server", "integrations/microsoft/sentinel-log-analytics", "integrations/microsoft/windows-event-logs", ],