diff --git a/.gitbook.yaml b/.gitbook.yaml index 43f895a5f..aeec83ec0 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -58,6 +58,7 @@ redirects: output/chronicle: ./pipeline/outputs/chronicle.md output/cloudwatch: ./pipeline/outputs/cloudwatch.md output/datadog: ./pipeline/outputs/datadog.md + output/doris: ./pipeline/outputs/doris.md output/es: ./pipeline/outputs/elasticsearch.md output/fabric: ./pipeline/outputs/azure_kusto.md output/file: ./pipeline/outputs/file.md diff --git a/SUMMARY.md b/SUMMARY.md index 18d8948ee..29d7e3596 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -182,6 +182,7 @@ * [Counter](pipeline/outputs/counter.md) * [Dash0](pipeline/outputs/dash0.md) * [Datadog](pipeline/outputs/datadog.md) + * [Doris](pipeline/outputs/doris.md) * [Dynatrace](pipeline/outputs/dynatrace.md) * [Elasticsearch](pipeline/outputs/elasticsearch.md) * [Exit](pipeline/outputs/exit.md) diff --git a/administration/transport-security.md b/administration/transport-security.md index 12c1cfc5f..c8c258d88 100644 --- a/administration/transport-security.md +++ b/administration/transport-security.md @@ -50,6 +50,7 @@ The following output plugins can take advantage of the TLS feature: - [BigQuery](../pipeline/outputs/bigquery.md) - [Dash0](../pipeline/outputs/dash0.md) - [Datadog](../pipeline/outputs/datadog.md) +- [Doris](../pipeline/outputs/doris.md) - [Elasticsearch](../pipeline/outputs/elasticsearch.md) - [Forward](../pipeline/outputs/forward.md) - [GELF](../pipeline/outputs/gelf.md) diff --git a/installation/downloads/source/build-and-install.md b/installation/downloads/source/build-and-install.md index 89c7610ac..c7bcdf1e9 100644 --- a/installation/downloads/source/build-and-install.md +++ b/installation/downloads/source/build-and-install.md @@ -283,6 +283,7 @@ The following table describes the output plugins available: | [`FLB_OUT_CLOUDWATCH_LOGS`](../../../pipeline/outputs/cloudwatch.md) | Enable Amazon CloudWatch output plugin | `On` | | [`FLB_OUT_COUNTER`](../../../pipeline/outputs/counter.md) | Enable Counter output plugin | `On` | | [`FLB_OUT_DATADOG`](../../../pipeline/outputs/datadog.md) | Enable Datadog output plugin | `On` | +| [`FLB_OUT_DORIS`](../../../pipeline/outputs/doris.md) | Enable Apache Doris output plugin | `On` | | [`FLB_OUT_ES`](../../../pipeline/outputs/elasticsearch.md) | Enable Elastic Search output plugin | `On` | | [`FLB_OUT_EXIT`](../../../pipeline/outputs/exit.md) | Enable Exit output plugin | `On` | | [`FLB_OUT_FILE`](../../../pipeline/outputs/file.md) | Enable File output plugin | `On` | diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md new file mode 100644 index 000000000..e7266d579 --- /dev/null +++ b/pipeline/outputs/doris.md @@ -0,0 +1,98 @@ +--- +description: Send logs to Apache Doris +--- + +# Apache Doris + +The _Apache Doris_ output plugin lets you ingest your records into an +[Apache Doris](https://doris.apache.org) database. To use this plugin, you must have an +operational Doris service running in your environment. + +## Configuration parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `host` | HTTP address of the target Doris frontend (FE) or backend (BE). | `127.0.0.1` | +| `port` | HTTP port of the target Doris frontend (FE) or backend (BE). | `8030` | +| `user` | Username for Doris access. | _none_ | +| `password` | Password for Doris access. | _none_ | +| `database` | The target Doris database. | _none_ | +| `table` | The target Doris table. | _none_ | +| `label_prefix` | Label prefix of Doris stream load, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` | +| `time_key` | The name of the time key in the output record. | `date` | +| `header` | Headers of Doris stream load. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ | +| `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` | +| `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` | +| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | + +### TLS / SSL + +The Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md) +for more details about the supported properties and general configuration. + +## Get started + +To insert records into a Doris database, run the plugin from the command line or define a configuration file: + +### Command line + +The Doris plugin can read the parameters from the command through the `-p` argument, + +as shown in the following example: + +```shell copy +fluent-bit -i cpu -t cpu -o doris \ + -m '*' \ + -p host=127.0.0.1 \ + -p port=8030 \ + -p user=admin \ + -p password=admin \ + -p database=d_fb \ + -p table=t_fb \ + -p header='columns date, cpu_p, log=cast(cpu_p as string)' +``` + +### Configuration file + +In your main configuration file, append the following `Input` and `Output` sections. + +{% tabs %} +{% tab title="fluent-bit.conf" %} +```python copy +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + name doris + match * + host 127.0.0.1 + port 8030 + user admin + password admin + database d_fb + table t_fb + header columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} + +{% tab title="fluent-bit.yaml" %} +```yaml copy +pipeline: + inputs: + - name: cpu + tag: cpu + outputs: + - name: doris + match: '*' + host: 127.0.0.1 + port: 8030 + user: admin + password: admin + database: d_fb + table: t_fb + header: + - columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} +{% endtabs %} diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 19a34ebee..1171af91a 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -59,6 +59,7 @@ deprovisions Devo distroless DogStatsD +Doris downsample downsampled downsamples