Creating pipelines

You can create and edit sources and pipelines with CLI or with file configs:

CLI commands:

agent source create
agent source edit
agent pipeline create
agent pipeline edit
agent pipeline create-raw

Commands for file configs:

Source

You can create or edit multiple sources at once with a single configuration file in JSON format.

agent create source -f path/to/file.json

agent edit source -f path/to/file.json

The content of the file must be an array of objects. Each object represents a single source.

Specifications are available on the corresponding source page in this wiki

Example:

[
  {
    "type": "influx",
    "name": "test_influx",
    "config": {
      "host": "http://influx:8086",
      "db": "test"
    }
  },
  {
    "type": "kafka",
    "name": "test_kafka",
    "config": {
      "kafkaConfigBean.metadataBrokerList": "kafka:29092",
      "kafkaConfigBean.topic": "test"
    }
  }
]

Pipelines

You can create or edit multiple pipelines at once with a configuration file in JSON format.

agent create pipeline -f path/to/file.json

agent edit pipeline -f path/to/file.json

The content of the file must be an array of objects. Each object represents a single pipeline.

Example:

[
  {
    "source": "test_influx",
    "pipeline_id": "test_influx_pipeline",
    "measurement_name": "cpu_test",
    "value": ["usage_active", "usage_idle"],
    "dimensions": ["cpu", "host", "zone"]
  },
  {
    "source": "test_mongo",
    "pipeline_id": "test_mongo_pipeline",
    "measurement_name": "clicks",
    "value": {
      "type": "constant",
      "value": "1"
    },
    "dimensions": {
      "required": ["ver", "Country"],
      "optional": ["Exchange", "optional_dim"]
    },
    "timestamp": {
      "type": "string",
      "name": "timestamp_string",
      "format": "M/d/yyyy H:mm:ss"
    },
    "target_type": "counter",
    "properties": {"test": "val"}
  }
]

Specifications are available on the corresponding source page in this wiki

Raw pipelines

You can create a pipeline, that will pull data from a source and keep it without any transformations in your local file system. It might be used for debugging purposes or in order to see in what format data is stored.

To do that use the command

agent pipeline create-raw -f path/to/file.json

Requirements for such pipelines are the same as for regular pipelines, except that some fields are not needed, such as dimensions or values. You can find a list of required fields for raw pipelines on the corresponding integration description pages.

Desired output directory might be configured via LOCAL_DESTINATION_OUTPUT_DIR environment variable in the agent docker container, by default it's /usr/src/app/local-output.