DataSet-Go

This repository holds the source code for the Go wrapper around DataSet API.

To learn more about DataSet, visit https://www.dataset.com.

Features

The DataSet API supports following API calls:

• addEvents - This endpoint inserts one or more structured or unstructured log events. It can scale horizontally to ingest 100s of TBs per day.

Examples

// create logger
logger := zap.Must(zap.NewDevelopment())
// read configuration from env variables
cfg, err := config.New(config.FromEnv())
if err != nil { panic(err) }

// build client
cl, err := client.NewClient(cfg, &http.Client{}, logger, nil, nil)
if err != nil { panic(err) }

// send bundles
err = cl.AddEvents(makeBundles())
if err != nil { panic(err) }

// Finish event processing
err = cl.Shutdown()
if err != nil { panic(err) }

Full example can be found in examples/readme/

All examples:

Examples are located in the examples folder.

• simple client - simple client
• readme - full example from readme

Release Notes and Changelog

For release notes please see RELEASE_NOTES.md document and for changelog, see CHANGELOG.md document.

Developing

Install Dev Tools

Run following script - install-dev-tools.sh - to install tools needed by pre-commit.

Pre-Commit Hooks

Pre-commit is used to automatically run checks including Black formatting prior to a git commit.

To use pre-commit:

• Use one of the Installation Methods from the documentation.
• Install the hooks with pre-commit install.
• To manually execute the pre-commit hooks (including black), run pre-commit run --all-files ( run it on all the files) or pre-commit run (run it only on staged files).

Pre-commit Configuration

• .pre-commit-config.yaml configures the scripts run by pre-commit

To update the Pre-commit hooks , run pre-commit autoupdate. This will update .pre-commit-config.yaml and will need to be committed to the repository.

Updating the Go version

The Go version used by CI is derived from a single source of truth: the go directive in the root go.mod. The GitHub workflows do not hard-code any version — they read .github/go-versions.json, which is generated from go.mod by update-go-version.sh:

• primary — the <major>.<minor> from go.mod, used by single-version jobs (e.g. code quality).
• matrix — the last 3 minor versions, used by the test matrices (unit and stress tests).

To bump the Go version:

1. Edit the go directive in go.mod (and the examples/*/go.mod modules if you keep them in lockstep).
2. Run make update-go-version (or ./scripts/update-go-version.sh) to regenerate .github/go-versions.json.
3. Commit both go.mod and .github/go-versions.json.

CI guards against drift: the code-quality workflow runs ./scripts/update-go-version.sh --check and fails if .github/go-versions.json is out of sync with go.mod.

Contributing

In the future, we will be pushing guidelines on how to contribute to this repository. For now, please just feel free to submit pull requests to the main branch and we will work with you.

Copyright, License, and Contributors Agreement

Copyright 2023 SentinelOne, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.apache.org/licenses/LICENSE-2.0

By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the project, present and future, pursuant to the license of the project.

Architecture

see detailed Architecture description