[zoom] Add support for the meeting activity data stream

packages/zoom/_dev/build/docs/README.md

@@ -1,18 +1,123 @@
-# Zoom Webhook Integration
+# Zoom Integration for Elastic
 
-This integration creates an HTTP listener that accepts incoming webhook
-callbacks from Zoom.
+## Overview
 
-To configure Zoom to send webhooks to this integration, please follow the
-[Zoom Documentation](https://developers.zoom.us/docs/api/rest/webhook-only-app).
+[Zoom](https://www.zoom.com/) is a unified communications platform that provides meetings, webinars, phone, team chat, and Zoom Rooms. The Zoom integration for Elastic enables you to collect Zoom event and audit data so you can monitor user activity, investigate security incidents, and analyze platform usage in Elastic.
 
-The agent running this integration must be able to accept requests from the
-Internet in order for Zoom to be able connect. Zoom requires that the webhook
-accept requests over HTTPS. So you must either configure the integration with
-a valid TLS certificate or use a reverse proxy in front of the integration.
+This integration collects data using two complementary methods:
 
-## Compatibility
+- **Webhook**: a real-time HTTP listener that receives event notifications pushed by Zoom (meeting, webinar, recording, user, account, phone, team chat, and Zoom Rooms events).
+- **REST API**: a periodic poll of the Zoom REST API to collect the **meeting activity** logs report for an account.
 
-This integration is compatible with the Zoom Platform API as of September 2020.
+### Compatibility
+
+- The **meeting_activity** data stream uses the Zoom REST API [`GET /report/meeting_activities`](https://developers.zoom.us/docs/api/meetings/#tag/reports/get/report/meeting_activities) endpoint. It is available for Paid or ZMP accounts, and the meeting audit trail log feature must be enabled for the account by Zoom Support.
+
+### How it works
+
+The **webhook** data stream creates an HTTP listener that accepts incoming webhook callbacks from Zoom. The Elastic Agent running this integration must be reachable from the internet so that Zoom can connect to it. Zoom requires that webhooks are delivered over HTTPS, so you must either configure the integration with a valid TLS certificate or place a reverse proxy that terminates TLS in front of the integration. Incoming events are then routed to the appropriate ingest pipeline based on the Zoom event type.
+
+The **meeting_activity** data stream periodically queries the Zoom REST API using Server-to-Server OAuth. On each interval it requests meeting activity logs within a date window (a maximum of one month per request), paginates through the results, and advances a cursor so that subsequent runs collect only new activity.
+
+## What data does this integration collect?
+
+The Zoom integration collects the following data:
+
+- `webhook`: real-time Zoom event notifications, including account, team chat (channel and message), meeting, phone, recording, user, webinar, and Zoom Rooms events.
+- `meeting_activity`: meeting activity logs from the Zoom REST API reports endpoint, such as a meeting being created or started, a user joining or leaving, in-meeting chat, remote control, and a meeting ending.
+
+### Supported use cases
+
+Integrating Zoom with Elastic SIEM provides centralized visibility into collaboration and administrative activity. Webhook events support real-time monitoring and detection across meetings, recordings, users, and administrative changes, while the meeting activity logs report provides an audit trail of meeting lifecycle and participant activity for investigating incidents, monitoring meeting usage, and meeting compliance requirements.
+
+## What do I need to use this integration?
+
+### From Zoom
+
+#### Collecting data via Webhook
+
+1. Create a Webhook-only app in the [Zoom App Marketplace](https://marketplace.zoom.us/) by following the [Zoom webhook documentation](https://developers.zoom.us/docs/api/webhooks/).
+2. Add the event types you want to receive and set the event notification endpoint URL to the public HTTPS address where this integration is reachable.
+3. Note the **Secret Token** generated by Zoom. It is used for CRC endpoint validation and to verify the authenticity of incoming events.
+
+#### Collecting data from the Zoom REST API
+
+1. Create a **Server-to-Server OAuth** app in the [Zoom App Marketplace](https://marketplace.zoom.us/) by following the [Server-to-Server OAuth documentation](https://developers.zoom.us/docs/internal-apps/s2s-oauth/).
+2. Record the app's **Account ID**, **Client ID**, and **Client Secret**.
+3. Add the `report:read:admin` scope (or the granular `report:read:meeting_activity_log:admin` scope) to the app and activate it. The account must be a Paid or ZMP account, and the meeting audit trail log feature must be enabled by Zoom Support.
+
+## How do I deploy this integration?
+
+### Agent-based deployment
+
+Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host.
+
+Elastic Agent is required to receive the Zoom webhook callbacks or to poll the Zoom REST API, and to ship the data to Elastic, where the events are then processed via the integration's ingest pipelines.
+
+### Onboard / configure
+
+1. In the top search bar in Kibana, search for **Integrations**.
+2. In the search bar, type **Zoom**.
+3. Select the **Zoom** integration from the search results.
+4. Select **Add Zoom** to add the integration.
+5. Enable and configure only the collection methods which you will use.
+
+    * To **Collect Zoom logs via Webhook**, you'll need to:
+
+        - Configure the **Listen Address**, **Listen Port**, and **Webhook path** where the integration accepts requests.
+        - Optionally enable **CRC validation** and provide the **Zoom Secret Token**, and/or configure a custom header to verify incoming requests.
+        - Provide a valid **TLS** certificate (or front the integration with a TLS-terminating reverse proxy), since Zoom requires HTTPS.
+
+    * To **Collect Zoom logs via REST API**, you'll need to:
+
+        - Configure the **Account ID**, **Client ID**, and **Client Secret** of your Server-to-Server OAuth app.
+        - Adjust the integration configuration parameters if required, including the **Interval**, **Initial Interval** (lookback), and **Activity Type**, to enable data collection.
+
+6. Select **Save and continue** to save the integration.
+
+### Validation
+
+#### Dashboards populated
+
+1. In the top search bar in Kibana, search for **Dashboards**.
+2. In the search bar, type **Zoom**.
+3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated.
+
+## Troubleshooting
+
+For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
+
+## Scaling
+
+For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation.
+
+## Reference
+
+### webhook
+
+This is the `webhook` data stream. It collects real-time event notifications pushed by Zoom over an HTTP endpoint.
+
+{{event "webhook"}}
 
 {{fields "webhook"}}
+
+### meeting_activity
+
+This is the `meeting_activity` data stream. It collects meeting activity logs from the Zoom REST API.
+
+{{event "meeting_activity"}}
+
+{{fields "meeting_activity"}}
+
+### Inputs used
+
+These inputs are used in this integration:
+
+- [http_endpoint](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-http_endpoint)
+- [cel](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-cel)
+
+### API usage
+
+This integration uses the following APIs:
+
+- `meeting_activity`: [Get a meeting activities report](https://developers.zoom.us/docs/api/meetings/#tag/reports/get/report/meeting_activities).

packages/zoom/_dev/deploy/docker/docker-compose.yml

@@ -10,6 +10,19 @@ services:
       - STREAM_ADDR=http://elastic-agent:9080/zoom
       - STREAM_WEBHOOK_HEADER=Authorization=abc123
     command: log --start-signal=SIGHUP --delay=5s /sample_logs/account-ndjson.log
+  zoom-meeting-activity:
+    image: docker.elastic.co/observability/stream:v0.20.0
+    hostname: zoom-meeting-activity
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config-meeting_activity.yml
   zoom-webhook-https:
     image: docker.elastic.co/observability/stream:v0.20.0
     volumes:

packages/zoom/_dev/deploy/docker/files/config-meeting_activity.yml

@@ -0,0 +1,133 @@
+rules:
+  # Zoom S2S OAuth token endpoint (account_credentials grant).
+  - path: /oauth/token
+    methods: [POST]
+    query_params:
+      grant_type: account_credentials
+      account_id: test-account-id
+    request_headers:
+      Authorization:
+        - "Basic dGVzdC1jbGllbnQtaWQ6dGVzdC1jbGllbnQtc2VjcmV0"
+      Content-Type:
+        - "application/x-www-form-urlencoded"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |
+          {"access_token":"test-access-token","token_type":"Bearer","expires_in":3600,"scope":"report:read:admin"}
+  # Pagination page 2 — token from page 1.
+  - path: /v2/report/meeting_activities
+    methods: [GET]
+    query_params:
+      from: "{from:.*}"
+      to: "{to:.*}"
+      activity_type: "{activity_type:.*}"
+      page_size: "{page_size:.*}"
+      next_page_token: mock-page-2-token
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {
+            "meeting_activity_logs": [
+              {
+                "meeting_number": "982 610 0285",
+                "activity_time": "{{ .request.vars.from }} 07:35:12:880",
+                "operator": "Bob Brown",
+                "operator_email": "bob@example.com",
+                "activity_category": "User left",
+                "activity_detail": "Bob Brown left the meeting"
+              }
+            ],
+            "page_size": 300,
+            "next_page_token": "mock-page-3-token"
+          }
+  # Pagination page 3 — terminal page for round 1.
+  - path: /v2/report/meeting_activities
+    methods: [GET]
+    query_params:
+      from: "{from:.*}"
+      to: "{to:.*}"
+      activity_type: "{activity_type:.*}"
+      page_size: "{page_size:.*}"
+      next_page_token: mock-page-3-token
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {
+            "meeting_activity_logs": [
+              {
+                "meeting_number": "982 610 0285",
+                "activity_time": "{{ .request.vars.from }} 07:40:00:120",
+                "operator": "Jill Chill",
+                "operator_email": "jillchill@example.com",
+                "activity_category": "Meeting ended",
+                "activity_detail": "Meeting ended"
+              }
+            ],
+            "page_size": 300
+          }
+  # Initial page (round 1) and cursor-resumed page (round 2).
+  - path: /v2/report/meeting_activities
+    methods: [GET]
+    query_params:
+      from: "{from:.*}"
+      to: "{to:.*}"
+      activity_type: "{activity_type:.*}"
+      page_size: "{page_size:.*}"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - "application/json"
+        body: |-
+          {{ if eq .req_num 1 }}
+          {
+            "meeting_activity_logs": [
+              {
+                "meeting_number": "982 610 0285",
+                "activity_time": "{{ .request.vars.from }} 07:09:03:216",
+                "operator": "Jill Chill",
+                "operator_email": "jillchill@example.com",
+                "activity_category": "Meeting Started",
+                "activity_detail": "Meeting Started"
+              },
+              {
+                "meeting_number": "982 610 0285",
+                "activity_time": "{{ .request.vars.from }} 07:09:45:002",
+                "operator": "Bob Brown",
+                "operator_email": "bob@example.com",
+                "activity_category": "User joined",
+                "activity_detail": "Bob Brown joined the meeting"
+              }
+            ],
+            "page_size": 300,
+            "next_page_token": "mock-page-2-token"
+          }
+          {{ else if eq .req_num 2 }}
+          {
+            "meeting_activity_logs": [
+              {
+                "meeting_number": "112 233 4455",
+                "activity_time": "{{ .request.vars.from }} 09:00:10:500",
+                "operator": "Alice Adams",
+                "operator_email": "alice@example.com",
+                "activity_category": "Meeting created",
+                "activity_detail": "Meeting created"
+              }
+            ],
+            "page_size": 300
+          }
+          {{ else }}
+          {
+            "meeting_activity_logs": [],
+            "page_size": 300
+          }
+          {{ end }}

packages/zoom/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.24.0"
+  changes:
+    - description: Add support for the `meeting_activity` data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/19554
 - version: "1.23.0"
   changes:
     - description: Map `user.email` and `source.ip` from available fields.

packages/zoom/data_stream/meeting_activity/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_original_event

packages/zoom/data_stream/meeting_activity/_dev/test/pipeline/test-zoom-meeting-activity.log

@@ -0,0 +1,5 @@
+{"meeting_number":"982 610 0285","activity_time":"2024-03-21 07:09:03:216","operator":"Jill Chill","operator_email":"jillchill@example.com","activity_category":"Meeting Started","activity_detail":"Meeting Started"}
+{"meeting_number":"982 610 0285","activity_time":"2024-03-21 07:09:45:002","operator":"Bob Brown","operator_email":"bob@example.com","activity_category":"User joined","activity_detail":"Bob Brown joined the meeting"}
+{"meeting_number":"982 610 0285","activity_time":"2024-03-21 07:35:12:880","operator":"Bob Brown","operator_email":"bob@example.com","activity_category":"User left","activity_detail":"Bob Brown left the meeting"}
+{"meeting_number":"982 610 0285","activity_time":"2024-03-21 07:40:00:120","operator":"Jill Chill","operator_email":"jillchill@example.com","activity_category":"Meeting ended","activity_detail":"Meeting ended"}
+{"meeting_number":"112 233 4455","activity_time":"2024-03-22 09:00:10:500","operator":"Alice Adams","activity_category":"Meeting created","activity_detail":"Meeting created"}