Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 58 additions & 8 deletions docs/converters.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,21 @@
# Converters

Converters service provides bulk CSV ingestion by parsing uploaded CSV files and publishing their rows as platform messages. It is intended for batch import of historical or bulk-collected sensor data without requiring device-level publishing.
Converters service provides bulk CSV and JSON ingestion by parsing uploaded files and publishing their records as platform messages. It is intended for batch import of historical or bulk-collected sensor data without requiring device-level publishing.

Two input formats are supported:
Four input formats are supported:

- **SenML** — the first row is the header where column names become measurement names. The first column must be a Unix timestamp. All remaining columns are treated as numeric measurement values.
- **JSON** — column names become JSON field names. The value from the column named by the thing's profile transformer `time_field` setting is parsed as a numeric Unix timestamp and stored as a `Created` field in the payload. The first column is excluded from the payload. Values are auto-parsed as numbers where possible, otherwise treated as strings.
- **CSV → SenML** — the first row is the header where column names become measurement names. The first column must be a Unix timestamp. All remaining columns are treated as numeric measurement values.
- **CSV → JSON** — column names become JSON field names. The value from the column named by the thing's profile transformer `time_field` setting is parsed as a numeric Unix timestamp and stored as a `Created` field in the payload. The first column is excluded from the payload. Values are auto-parsed as numbers where possible, otherwise treated as strings.
- **JSON → SenML** — a JSON array of objects. Each object must have a `t` key (Unix timestamp) and one or more numeric measurement keys.
- **JSON → JSON** — a JSON array of objects. The key matching the thing's profile transformer `time_field` is stored as `Created`. All other fields are passed through as-is.

Large files are processed in batches with a 30-second pause between batches to avoid overloading the message broker. For SenML, a batch is flushed every 50,000 SenML records (one record per value column per row). For JSON, a batch is flushed every 50,000 rows.

## Authentication

Requests must be authenticated with the device's internal key using the `Authorization: Thing <key>` header.

## SenML CSV
## CSV file to SenML Messages

The first column must be a Unix timestamp. Subsequent columns are measurement names taken from the header row.

Expand All @@ -29,12 +31,12 @@ time,voltage,current,power
curl -s -S -i -X POST \
-H "Authorization: Thing <thing_key>" \
-F "file=@readings.csv" \
https://localhost/converters/csv/senml
"https://localhost/converters/csv?to=senml"
```

Rows are collected and published as batched SenML messages. Each SenML record carries the measurement name (`n`), value (`v`), and timestamp (`t`) from its column and row.

## JSON CSV
## CSV file to JSON Messages

Column names become JSON field names. If the profile transformer `time_field` is set and matches a column name, that column's value is parsed as a numeric Unix timestamp (seconds or nanoseconds as a float) and stored as a `Created` field in the payload. The first column is always excluded from the payload. Values that cannot be parsed as numbers are kept as strings.

Expand All @@ -50,7 +52,7 @@ time,temperature,humidity,status
curl -s -S -i -X POST \
-H "Authorization: Thing <thing_key>" \
-F "file=@events.csv" \
https://localhost/converters/csv/json
"https://localhost/converters/csv?to=json"
```

Rows are collected and published as batched JSON messages. The `time` column value is parsed as a float and stored as `Created` in the payload. The remaining columns — `temperature`, `humidity`, and `status` — become fields in the JSON payload. The resulting record looks like:
Expand All @@ -59,4 +61,52 @@ Rows are collected and published as batched JSON messages. The `time` column val
{"Created": 1709635200, "temperature": 21.5, "humidity": 60, "status": "ok"}
```

## JSON file to SenML Messages

Each object in the array must contain a `t` key (Unix timestamp as a float) and one or more numeric measurement keys. Each measurement key produces one SenML record.

Example file (`readings.json`):

```json
[
{"t": 1709635200.0, "voltage": 120.1, "current": 1.2},
{"t": 1709635260.0, "voltage": 119.8, "current": 1.3}
]
```

```bash
curl -s -S -i -X POST \
-H "Authorization: Thing <thing_key>" \
-F "file=@readings.json" \
"https://localhost/converters/json?to=senml"
```

Records are collected and published as batched SenML messages. Each SenML record carries the measurement name (`n`), value (`v`), and timestamp (`t`).

## JSON file to JSON Messages

Each object in the array becomes one JSON payload record. If the profile transformer `time_field` is set and matches a key, that key's value is stored as a `Created` field. All other keys are passed through with their original types.

Example file (`events.json`), assuming `time_field` is set to `time`:

```json
[
{"time": 1709635200.0, "temperature": 21.5, "humidity": 60, "status": "ok"},
{"time": 1709635260.0, "temperature": 22.1, "humidity": 58, "status": "ok"}
]
```

```bash
curl -s -S -i -X POST \
-H "Authorization: Thing <thing_key>" \
-F "file=@events.json" \
"https://localhost/converters/json?to=json"
```

The `time` field value is stored as `Created` in the payload. The resulting record looks like:

```json
{"time": 1709635200, "Created": 1709635200, "temperature": 21.5, "humidity": 60, "status": "ok"}
```

For the full API reference, see the [API documentation](https://mainfluxlabs.github.io/docs/swagger/).