Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/vale/styles/Vocab/OpenSearch/Products/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Google Normalized Distance
Google Workspace
Gradle
Grafana
Grok Debugger
Hanja
Helm
Hjson
Expand Down
18 changes: 17 additions & 1 deletion _dashboards/discover/run-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,4 +151,20 @@ To update your preferences, select **Settings** from the top menu:

## Using keyboard shortcuts

To view all available keyboard shortcuts, select **Help** from the top menu.
To view all available keyboard shortcuts, select **Help** from the top menu.

## Using the Grok Debugger

The **Grok Debugger** is a built-in tool in Dev Tools that helps you build and test [Grok patterns]({{site.url}}{{site.baseurl}}/ingest-pipelines/processors/grok/) before using them in ingest pipelines. Grok patterns parse unstructured log data into structured fields.

To use the Grok Debugger, follow these steps:

1. On the top menu in Dev Tools, select **Grok Debugger**.
1. Enter a sample log message in the **Sample Data** field.
1. Enter your Grok pattern in the **Grok Pattern** field.
1. Select **Simulate** to test the pattern.

The debugger displays the parsed fields in the output pane, as shown in the following image.

![Grok Debugger]({{site.url}}{{site.baseurl}}/images/dev-tools/grok-debugger.png)

10 changes: 5 additions & 5 deletions _ingest-pipelines/processors/grok.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,24 @@
layout: default
title: Grok
parent: Ingest processors
nav_order: 140
nav_order: 120
---

This documentation describes using the `grok` processor in OpenSearch ingest pipelines. Consider using the [Data Prepper `grok` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/), which runs on the OpenSearch cluster, if your use case involves large or complex datasets.
{: .note}

# Grok processor

The `grok` processor is used to parse and structure unstructured data using pattern matching. You can use the `grok` processor to extract fields from log messages, web server access logs, application logs, and other log data that follows a consistent format.

This documentation describes using the `grok` processor in OpenSearch ingest pipelines. Consider using the [Data Prepper `grok` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/), which runs on the OpenSearch cluster, if your use case involves large or complex datasets.
{: .note}

## Grok basics

The `grok` processor uses a set of predefined patterns to match parts of the input text. Each pattern consists of a name and a regular expression. For example, the pattern `%{IP:ip_address}` matches an IP address and assigns it to the field `ip_address`. You can combine multiple patterns to create more complex expressions. For example, the pattern `%{IP:client} %{WORD:method} %{URIPATHPARM:request} %{NUMBER:bytes %NUMBER:duration}` matches a line from a web server access log and extracts the client IP address, the HTTP method, the request URI, the number of bytes sent, and the duration of the request.

For a list of available predefined patterns, see [Grok patterns](https://github.com/opensearch-project/OpenSearch/blob/main/libs/grok/src/main/resources/patterns/grok-patterns).
{: .tip}

The `grok` processor is built on the [Oniguruma regular expression library](https://github.com/kkos/oniguruma/blob/master/doc/RE) and supports all the patterns from that library. You can use the [Grok Debugger](https://grokdebugger.com/) tool to test and debug your grok expressions.
The `grok` processor is built on the [Oniguruma regular expression library](https://github.com/kkos/oniguruma/blob/master/doc/RE) and supports all the patterns from that library. You can use the built-in [Grok Debugger]({{site.url}}{{site.baseurl}}/dashboards/discover/run-queries/#using-the-grok-debugger) in OpenSearch Dashboards Dev Tools to test and debug your grok expressions.

Note that patterns are *not anchored*. For performance and reliability, include a start-of-line anchor (`^`) in your pattern.
{: .note}
Expand Down
Binary file added images/dev-tools/grok-debugger.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading