Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitbook.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ redirects:
output/chronicle: ./pipeline/outputs/chronicle.md
output/cloudwatch: ./pipeline/outputs/cloudwatch.md
output/datadog: ./pipeline/outputs/datadog.md
output/doris: ./pipeline/outputs/doris.md
output/es: ./pipeline/outputs/elasticsearch.md
output/fabric: ./pipeline/outputs/azure_kusto.md
output/file: ./pipeline/outputs/file.md
Expand Down
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@
* [Counter](pipeline/outputs/counter.md)
* [Dash0](pipeline/outputs/dash0.md)
* [Datadog](pipeline/outputs/datadog.md)
* [Doris](pipeline/outputs/doris.md)
* [Dynatrace](pipeline/outputs/dynatrace.md)
* [Elasticsearch](pipeline/outputs/elasticsearch.md)
* [Exit](pipeline/outputs/exit.md)
Expand Down
1 change: 1 addition & 0 deletions administration/transport-security.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ The following output plugins can take advantage of the TLS feature:
- [BigQuery](../pipeline/outputs/bigquery.md)
- [Dash0](../pipeline/outputs/dash0.md)
- [Datadog](../pipeline/outputs/datadog.md)
- [Doris](../pipeline/outputs/doris.md)
- [Elasticsearch](../pipeline/outputs/elasticsearch.md)
- [Forward](../pipeline/outputs/forward.md)
- [GELF](../pipeline/outputs/gelf.md)
Expand Down
1 change: 1 addition & 0 deletions installation/downloads/source/build-and-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ The following table describes the output plugins available:
| [`FLB_OUT_CLOUDWATCH_LOGS`](../../../pipeline/outputs/cloudwatch.md) | Enable Amazon CloudWatch output plugin | `On` |
| [`FLB_OUT_COUNTER`](../../../pipeline/outputs/counter.md) | Enable Counter output plugin | `On` |
| [`FLB_OUT_DATADOG`](../../../pipeline/outputs/datadog.md) | Enable Datadog output plugin | `On` |
| [`FLB_OUT_DORIS`](../../../pipeline/outputs/doris.md) | Enable Apache Doris output plugin | `On` |
| [`FLB_OUT_ES`](../../../pipeline/outputs/elasticsearch.md) | Enable Elastic Search output plugin | `On` |
| [`FLB_OUT_EXIT`](../../../pipeline/outputs/exit.md) | Enable Exit output plugin | `On` |
| [`FLB_OUT_FILE`](../../../pipeline/outputs/file.md) | Enable File output plugin | `On` |
Expand Down
98 changes: 98 additions & 0 deletions pipeline/outputs/doris.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
description: Send logs to Apache Doris
---

# Apache Doris

Check warning on line 5 in pipeline/outputs/doris.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Apache Doris' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Apache Doris' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/doris.md", "range": {"start": {"line": 5, "column": 3}}}, "severity": "INFO"}

The _Apache Doris_ output plugin lets you ingest your records into an
[Apache Doris](https://doris.apache.org) database. To use this plugin, you must have an
operational Doris service running in your environment.

## Configuration parameters

| Key | Description | Default |
| :--- | :--- | :--- |
| `host` | HTTP address of the target Doris frontend (FE) or backend (BE). | `127.0.0.1` |
| `port` | HTTP port of the target Doris frontend (FE) or backend (BE). | `8030` |
| `user` | Username for Doris access. | _none_ |
| `password` | Password for Doris access. | _none_ |
| `database` | The target Doris database. | _none_ |
| `table` | The target Doris table. | _none_ |
| `label_prefix` | Label prefix of Doris stream load, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` |
| `time_key` | The name of the time key in the output record. | `date` |
| `header` | Headers of Doris stream load. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ |
| `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` |
| `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` |
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |

### TLS / SSL

The Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md)
for more details about the supported properties and general configuration.

## Get started

To insert records into a Doris database, run the plugin from the command line or define a configuration file:

### Command line

The Doris plugin can read the parameters from the command through the `-p` argument,

as shown in the following example:

```shell copy
fluent-bit -i cpu -t cpu -o doris \
-m '*' \
-p host=127.0.0.1 \
-p port=8030 \
-p user=admin \
-p password=admin \
-p database=d_fb \
-p table=t_fb \
-p header='columns date, cpu_p, log=cast(cpu_p as string)'
```

### Configuration file

In your main configuration file, append the following `Input` and `Output` sections.

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python copy
Comment thread
joker-star-l marked this conversation as resolved.
[INPUT]
Name cpu
Tag cpu

[OUTPUT]
name doris
match *
host 127.0.0.1
port 8030
user admin
password admin
database d_fb
table t_fb
header columns date, cpu_p, log=cast(cpu_p as string)
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml copy
pipeline:
inputs:
- name: cpu
tag: cpu
outputs:
- name: doris
match: '*'
host: 127.0.0.1
port: 8030
user: admin
password: admin
database: d_fb
table: t_fb
header:
- columns date, cpu_p, log=cast(cpu_p as string)
```
{% endtab %}
{% endtabs %}
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ deprovisions
Devo
distroless
DogStatsD
Doris
downsample
downsampled
downsamples
Expand Down