From 18e1541f0c8ff81fa539b5c1c391329f11d38a53 Mon Sep 17 00:00:00 2001 From: "Erwin Vrolijk (technimad-splunk)" Date: Tue, 26 May 2026 19:43:43 +0200 Subject: [PATCH 1/2] updated skill to support the new display framework in splunk 10.4 --- .claude/skills/splunk-viz/SKILL.md | 81 ++- .../splunk-viz/references/dse-framework.md | 574 ++++++++++++++++++ build.sh | 169 +++++- 3 files changed, 786 insertions(+), 38 deletions(-) create mode 100644 .claude/skills/splunk-viz/references/dse-framework.md diff --git a/.claude/skills/splunk-viz/SKILL.md b/.claude/skills/splunk-viz/SKILL.md index 8d64e91..efecffd 100644 --- a/.claude/skills/splunk-viz/SKILL.md +++ b/.claude/skills/splunk-viz/SKILL.md @@ -1,23 +1,55 @@ --- name: splunk-viz -description: Scaffold and build Splunk custom visualizations using Canvas 2D. Use this skill whenever the user wants to create, modify, debug, fix, or package a Splunk custom visualization app — including visualization_source.js, formatter.html, visualizations.conf, savedsearches.conf, webpack config, harness.json, or anything involving SplunkVisualizationBase. Also triggers for Splunk Cloud vetting errors (check_for_trigger_stanza, check_for_prohibited_files), blurry/HiDPI canvas issues, custom font embedding in Splunk vizs, Dashboard Studio custom viz integration, and scaffolding parent apps that bundle multiple vizs. +description: Scaffold and build Splunk custom visualizations using Canvas 2D. Use this skill whenever the user wants to create, modify, debug, fix, or package a Splunk custom visualization app — including visualization_source.js, formatter.html, visualizations.conf, savedsearches.conf, webpack config, harness.json, or anything involving SplunkVisualizationBase or the Dashboard Studio Extension API (@splunk/dashboard-studio-extension, VisualizationAPI, addDataSourcesListener). Also triggers for Splunk Cloud vetting errors (check_for_trigger_stanza, check_for_prohibited_files, check_meta_default_write_access), blurry/HiDPI canvas issues, custom font embedding in Splunk vizs, Dashboard Studio custom viz integration on Splunk 10.4+, repaint-on-refresh bugs in Dashboard Studio, and scaffolding parent apps that bundle multiple vizs. --- -You are an expert Splunk developer specializing in custom visualizations built with the Splunk Visualization Framework (Canvas 2D rendering, AMD modules, webpack). You generate production-ready code, not prototypes. +You are an expert Splunk developer specializing in custom visualizations built with the Splunk Visualization Framework (Canvas 2D rendering, AMD modules, webpack — Splunk 10.2 legacy) and the newer Dashboard Studio Extension framework (iframe-based, ESM, esbuild — Splunk 10.4+). You generate production-ready code, not prototypes. ## Architecture Overview -**Requires Splunk Enterprise 10.2+ or Splunk Cloud.** The `visualizations.conf` configuration and custom viz framework were significantly improved in 10.2. The target platform (Cloud, Enterprise, or both) is determined in Step 1 and affects which vetting constraints are applied — see the **Platform Differences** table. +A Splunk custom visualization is a standalone Splunk app that renders search results using Canvas 2D. **Two different Splunk frameworks are available**, and the right one depends on which Splunk version the user is targeting: -A Splunk custom visualization is a standalone Splunk app that renders search results using Canvas 2D. It consists of: +| Track | Framework | Splunk version | Dashboard type | Notes | +|-------|-----------|----------------|----------------|-------| +| **A** | Legacy `SplunkVisualizationBase` (AMD + webpack) | 10.2+ (works on 10.4) | Simple XML **and** Dashboard Studio | Default and most widely supported. Visibly repaints during scheduled refreshes on Dashboard Studio in 10.4 — see Track B if that matters. | +| **B** | Dashboard Studio Extension (`@splunk/dashboard-studio-extension`, ESM + esbuild, iframe-based) | 10.4+ **only** | Dashboard Studio only | Refresh-stable: subscribes to an explicit `loading` signal and holds the previous frame during refreshes. Will not load on 10.2 or in Simple XML dashboards. | -1. **App scaffolding** — Splunk app config files (`app.conf`, `visualizations.conf`, `savedsearches.conf`) -2. **Formatter UI** — HTML form that exposes user-configurable settings in the Splunk dashboard editor -3. **Visualization source** — JavaScript AMD module that extends `SplunkVisualizationBase` with Canvas 2D rendering -4. **Build tooling** — webpack bundles the source into a single `visualization.js` AMD module -5. **Build/deploy scripts** — Shell scripts to build, package, and deploy the app +**Always ask the user which target Splunk version to support before scaffolding.** See [Step 0: Choose Framework](#step-0-choose-framework). -## Step 1: Gather Requirements +A Splunk custom viz consists of: + +1. **App scaffolding** — Splunk app config files (`app.conf`, `visualizations.conf`, `savedsearches.conf` for Track A; `package/app/app.conf` for Track B). `visualizations.conf` declares `framework_type = legacy_visualization` (Track A) or `framework_type = studio_visualization` (Track B). +2. **Settings UI** — Track A: HTML form (`formatter.html`) using `` elements. Track B: JSON schema in `config.json` (`optionsSchema` + `editorConfig`). +3. **Visualization source** — Track A: AMD module extending `SplunkVisualizationBase`. Track B: ESM module registering listeners on `VisualizationAPI`. +4. **Build tooling** — Track A: webpack → single AMD `visualization.js`. Track B: esbuild → single ESM `visualization.js` plus auto-generated `default/visualizations.conf` and `metadata/default.meta`. +5. **Build/deploy scripts** — Track A: shared `./build.sh` at repo root. Track B: per-app `npm run build && npm run package` producing a `.spl`. + +## Step 0: Choose Framework + +**Always run this step first.** Ask the user (or extract from context) which Splunk version they need to support, then pick the track: + +1. **What is the minimum Splunk version that must run this viz?** (`10.2` / `10.4`) +2. **Which dashboard type(s)?** (Simple XML / Dashboard Studio / both) +3. **(If 10.4 + Studio only)**: Does refresh-stability matter? (i.e. does the viz appear on dashboards with scheduled searches where a brief repaint would look like a bug?) + +Decision matrix: + +| Min version | Dashboard type | Refresh stability? | → Track | +|-------------|----------------|--------------------|----| +| 10.2 (or unsure) | any | — | **A** (legacy) | +| 10.4+ | Simple XML or both | — | **A** (legacy — required for Simple XML) | +| 10.4+ | Dashboard Studio only | no / don't care | **A** (legacy) — simpler tooling | +| 10.4+ | Dashboard Studio only | yes — refreshes are visible | **B** (studio extension) | + +When in doubt, **default to Track A**. It supports the widest Splunk version range and the simpler tooling is easier to maintain. Only recommend Track B when the user explicitly targets 10.4-or-later and either asks about the refresh repaint issue or operates dashboards where scheduled refreshes are user-visible. + +Once selected, proceed with the rest of this skill for **Track A** in-line below, or read [`references/dse-framework.md`](references/dse-framework.md) for the complete **Track B** scaffolding, build pipeline, listener model, and Splunk Cloud appinspect notes. + +> **Track A is documented in the remainder of this file. Everything from Step 1 onward — directory structure, file templates, formatter.html, visualization_source.js, webpack, harness.json, all 32 Critical Rules — applies to Track A only.** Track B has its own structure, its own settings schema, its own runtime model, and its own appinspect quirks. Do not mix them. + +## Step 1: Gather Requirements (Track A — Legacy Framework) + +> Skip this step if Step 0 selected Track B. Use [`references/dse-framework.md`](references/dse-framework.md) instead. Before generating code, ask the user (or extract from context): @@ -404,6 +436,8 @@ When creating a new viz, read `references/core-template.md` for the complete AMD ## Critical Rules +> The rules below are written for **Track A** (legacy `SplunkVisualizationBase`). The Track B summary at the top of this file lists which of these rules also apply to the Studio Extension framework and which are superseded by listener-based equivalents. + 1. **Use `var`, not `const`/`let`**. Webpack targets AMD for Splunk's RequireJS environment. Some Splunk versions run older JS engines. Stick with ES5 (`var`, `function`, `for` loops). No arrow functions, no template literals, no destructuring. 2. **Always handle HiDPI displays**. Set `canvas.width/height` to `rect.width * dpr` and call `ctx.scale(dpr, dpr)`. All drawing math uses the CSS pixel dimensions (`rect.width`, `rect.height`), NOT `canvas.width/height`. @@ -784,7 +818,9 @@ Do not try to parallelize file generation itself — the config files, formatter ## Step 3: Generate Build Script -The repo uses a single shared `build.sh` at the root. **Do not generate per-viz build scripts** — use the existing `build.sh` instead. Do not generate deploy scripts — apps should be installed via the Splunk UI (Manage Apps → Install app from file). +The repo uses a single shared `build.sh` at the root. It auto-detects whether the app is Track A (legacy) or Track B (studio extension) by looking for `default/app.conf` versus `package/app/app.conf`, and dispatches to the right tooling. **Do not generate per-viz build scripts for Track A** — use the existing `build.sh` instead. Track B apps DO have a per-app `build.mjs` + `package.mjs` (vendored from `@splunk/create` with bug patches), but `./build.sh` still works as the unified entry point — it just delegates to `npm run build:prod && npm run package` inside the app. + +Do not generate deploy scripts — apps should be installed via the Splunk UI (Manage Apps → Install app from file). ### Usage @@ -838,6 +874,8 @@ For any viz type, always include a "no data" state. Ask the user whether they wa **For new vizs**, verify all files are generated. **For modifications to existing vizs**, update all affected files — code changes that add/remove data fields, settings, or features MUST be reflected in `README.md`, `savedsearches.conf`, `savedsearches.conf.spec`, `harness.json`, and `formatter.html`. Never change the JS without updating the documentation and config files to match. +> **Track B (Studio Extension):** use the checklist in [`references/dse-framework.md` → Verify Completeness](references/dse-framework.md#verify-completeness) instead. The list below is Track A specific. + Before presenting the generated code, verify: - [ ] `README.md` exists with description, install, columns, search, configuration, drilldown (if applicable), time range, and build sections @@ -868,6 +906,27 @@ Before presenting the generated code, verify: - [ ] **If modifying an existing viz**: `savedsearches.conf` search query includes any new data fields - [ ] **If modifying an existing viz**: `harness.json` updated with new field controls and data columns +## Track B: Studio Extension Framework (Splunk 10.4+) + +If Step 0 selected Track B, **stop reading this file** for scaffolding instructions and read [`references/dse-framework.md`](references/dse-framework.md) end-to-end. That reference contains the full workflow: + +- Project layout (very different from Track A — no `appserver/`, no `default/visualizations.conf` to hand-write, no `formatter.html`, no `savedsearches.conf`) +- `package.json` + `build.mjs` + `package.mjs` build pipeline (and the three upstream `@splunk/create` bugs that must be patched locally — see [`upstream-bugfix/`](../../../upstream-bugfix/)) +- `config.json` schema (replaces `formatter.html` + `savedsearches.conf` defaults) +- `visualization.js` listener model (`addDataSourcesListener`, `addOptionsListener`, `addDimensionsListener`) +- Column-major data shape (`{ fields, columns }` instead of `{ fields, rows }`) +- `package/app/app.conf` stanza requirements for Splunk Cloud +- `metadata/default.meta` global write-access stanza requirement +- The refresh-stability pattern (silently skip render during `loading: true`) + +A complete worked example lives at `examples/steampunk_gauge_studio/`. Refer to it for any detail not covered by the reference doc. + +**Critical Rules that still apply to Track B** (numbered in the Track A section below): 2 (HiDPI), 3 (zero-size canvas guard), 4 (null ctx), 5–6 (canvas state cleanup), 10 (font conventions), 12 (no `this` in helpers), 14 (XSS prevention if touching DOM), 24 (label alignment), 27 ("no data" via `_status` field — adapted for column-major shape), 29 (text readability), 30 (README markdown linting), 31 (Python venv for preview generation), 32 (real-time smoothing). + +**Critical Rules that do NOT apply to Track B**: 1 (Track B is ESM, not AMD — modern syntax encouraged), 7–8 (no `formatData`/`VisualizationError` — use `setError`/`clearError` from VisualizationAPI), 9 (settings come typed, not always strings, when `optionsSchema` declares `type: number` or `type: boolean`), 11 (no AMD JSON loader — use ESM `import`), 13 (no `this`-bound timer — use module-scope state), 16–17 (no `invalidate*`/`reflow` methods — listeners drive re-render), 18–21 (no `getPropertyNamespaceInfo`, no namespaced config keys, no `formatData` caching needed — the listener model + cached state already prevents re-render during loading), 22 (no `savedsearches.conf.spec` — defaults live in `optionsSchema`), 23 (`/_bump` still works but you typically re-install the `.spl` via the UI), 28 (Dashboard Studio JSON `options` block — Track B writes well-typed values directly, no stale-key issue). + +For the harness, see the [Studio Extension Harness](references/dse-framework.md#testing) section of the reference doc — the legacy Step 5 harness pattern below does not apply to Track B vizs. + ## Step 5: Generate Test Harness Config (MANDATORY) Every new viz MUST have a `harness.json` file and be added to `harness-manifest.json`. When modifying an existing viz, update its `harness.json` to match. diff --git a/.claude/skills/splunk-viz/references/dse-framework.md b/.claude/skills/splunk-viz/references/dse-framework.md new file mode 100644 index 0000000..a4e5489 --- /dev/null +++ b/.claude/skills/splunk-viz/references/dse-framework.md @@ -0,0 +1,574 @@ +# Dashboard Studio Extension Framework (Splunk 10.4+) — Track B Reference + +Use this reference when [Step 0 of the parent skill](../SKILL.md#step-0-choose-framework) selects **Track B** — that is, when the target Splunk version is 10.4+ **and** the viz only needs to run inside Dashboard Studio (not Simple XML), and refresh-stability matters. + +The complete worked example is `examples/steampunk_gauge_studio/`. Read it alongside this document. + +## Why this track exists + +Splunk 10.4 Dashboard Studio repaints `legacy_visualization` panels visibly on every scheduled-search refresh: the canvas briefly clears, the empty shell is drawn, then the next frame paints. On 10.2 the panel held the previous frame; on 10.4 it does not. The fix is not in the legacy viz code — it is in the framework choice. + +The new `@splunk/dashboard-studio-extension` framework runs each viz in an iframe injected by the host dashboard and exposes an explicit `loading` boolean alongside the data. When `loading` is true we **silently skip the redraw** and keep the previous frame on screen. The Dashboard Studio host overlays its own "refreshing" indicator in the corner of the panel, so the user still sees that data is loading — they just no longer see the panel flicker. + +Trade-offs: + +- Will not load on Splunk 10.2 or in Simple XML dashboards. +- Build pipeline is more complex (esbuild + custom packaging script, with upstream bugs that must be patched locally). +- Settings UI uses a JSON schema rather than HTML, which makes some advanced layouts (multi-tab formatter, conditional fields) harder. + +## Project layout + +``` +examples/{app_name}/ + README.md + .gitignore (excludes node_modules, dist, stage) + package.json (npm + esbuild + tar deps; build scripts) + package-lock.json + build.mjs (esbuild driver — vendored from @splunk/create) + package.mjs (assembles .spl — vendored from @splunk/create, patched) + build-plugins/ + css-and-size.mjs (esbuild plugin: inlines CSS, warns on >2MB bundles) + package/ + app/ + app.conf (source of truth for the [id]/[package]/[launcher] stanzas) + metadata/ (optional — packager writes default.meta automatically) + visualizations/ + {app_name}/ + config.json (replaces formatter.html AND savedsearches.conf defaults) + src/ + visualization.js (ESM module — VisualizationAPI listeners) + visualization.css (imported from visualization.js — inlined by esbuild) + dist/ (build output: dist/{app_name}/visualization.js[.map]) + stage/ (packager staging area — gitignored) +``` + +Notes: + +- **No `appserver/` directory in source.** The packager (`package.mjs`) creates `appserver/static/visualizations/{app_name}/` inside the staged tarball and copies `dist/{app_name}/visualization.js` and `config.json` into it. +- **No hand-written `default/visualizations.conf`.** The packager generates it from the discovered `config.json` files, including `framework_type = studio_visualization`. +- **No `formatter.html`, no `savedsearches.conf`, no `savedsearches.conf.spec`.** All settings live in `config.json` → `optionsSchema` (with defaults) + `editorConfig` (UI layout). +- **No `harness.json`.** Studio extension vizs are tested via `npm run dev` (esbuild watch) + installing the `.spl` in a real Splunk 10.4 instance. The Track A test-harness pattern does not apply. + +## Step-by-step scaffolding + +### 1. Initialise the project + +From the repo root: + +```bash +cd examples +npx -y @splunk/create@latest dashboard-studio-extension {app_name} +``` + +This produces a viable starting layout, but its `package.mjs` and `package/app/app.conf` contain bugs that block Splunk Cloud appinspect. Apply the fixes documented in [`upstream-bugfix/`](../../../../upstream-bugfix/) immediately: + +1. **`package/app/app.conf`** — add the `[id]` stanza and `check_for_updates = false`. See [App configuration](#app-configuration) below. +2. **`package.mjs` → `stageConfFiles()`** — `mkdirSync(defaultDir, { recursive: true })` before writing `visualizations.conf` (the upstream template assumes `default/` already exists, which it does not by the time this code runs). +3. **`package.mjs` → `main()`** — reorder so `app.conf` is parsed first, then `stage/{appId}/` is wiped and re-created, then `stageAppConf()` runs. The upstream sequence writes `app.conf` and immediately deletes it. +4. **`package.mjs` → `generateDefaultMeta()`** — prepend a global `[]\naccess = read : [ * ], write : [ admin, sc_admin ]` stanza so Splunk Cloud's `check_meta_default_write_access` rule passes. + +Easier: copy `examples/steampunk_gauge_studio/{build.mjs,package.mjs,build-plugins/}` into the new project as-is. Those copies already include the fixes. + +### 2. Configure `package.json` + +```json +{ + "name": "{app_name}", + "version": "1.0.0", + "description": "{description}", + "author": "{author}", + "type": "module", + "scripts": { + "build": "node build.mjs --entry=visualization.js", + "build:prod": "NODE_ENV=production node build.mjs --entry=visualization.js", + "dev": "node build.mjs --entry=visualization.js --watch", + "package": "node package.mjs" + }, + "devDependencies": { + "@splunk/dashboard-studio-extension": "^1.0.0", + "chalk": "^5.3.0", + "esbuild": "^0.27.3", + "tar": "^7.4.3" + } +} +``` + +Keep `"type": "module"` — the build/package scripts are ESM. + +### 3. Author `config.json` + +`config.json` is the single source of truth for: + +- The visualization's identity (`name`, `description`, `category`, `icon`) +- Its data contract (`requiredDataSources`, `optionalDataSources`) +- Default panel size (`initialWidth`, `initialHeight`) +- All user-configurable options and their default values (`optionsSchema`) +- The Format-panel UI layout that exposes those options (`editorConfig`) + +Full schema: + +```json +{ + "showTitleAndDescription": true, + "includeInToolbar": true, + "includeInVizSwitcher": true, + "showDrilldown": false, + "canSetTokens": [], + "hasEventHandlers": false, + "config": { + "name": "{Display Label}", + "description": "{One-line description}", + "category": "Custom", + "icon": null, + "dataContract": { + "requiredDataSources": ["primary"], + "optionalDataSources": [] + }, + "size": { + "initialWidth": 320, + "initialHeight": 320 + }, + "optionsSchema": { + "{optionName}": { "type": "string|number|boolean", "default": } + }, + "editorConfig": [ + { + "label": "{Tab label}", + "layout": [ + [{ "editor": "editor.text", "label": "{Field label}", "option": "{optionName}" }] + ] + } + ] + } +} +``` + +**`optionsSchema` field types:** + +| `type` | Editor surface | Notes | +|--------|----------------|-------| +| `"string"` | `editor.text`, `editor.color`, `editor.select` | Free text or one of a fixed list | +| `"number"` | `editor.number` | Comes through as a JS number when the user has set one. Falls back to schema `default` if unset; `undefined` if no default. | +| `"boolean"` | `editor.toggle` | Comes through as a JS boolean. | + +If a setting needs no default value (only enabled when the user fills it in), omit `default` — the option will be `undefined` in the runtime payload and your code should handle that case. + +**`editorConfig`** — an array of tabs. Each tab has `label` and `layout`. Each `layout` is an array of rows; each row is an array of editor objects: + +| `editor` | Use for | Required keys | +|----------|---------|---------------| +| `editor.text` | Strings | `label`, `option` | +| `editor.number` | Numeric input | `label`, `option` | +| `editor.color` | Hex colour picker | `label`, `option` | +| `editor.toggle` | Boolean switch | `label`, `option` | +| `editor.select` | Drop-down | `label`, `option`, `editorProps.values` (array of `{ label, value }`) | + +Avoid baking colour palettes or fonts into `editorConfig` — the host applies the user's theme around the iframe. Use `editor.color` and accept any hex. + +**Defaults discipline:** the value sent to the viz at runtime is always `optionsSchema[name].default` (if set) until the user changes it in the Format panel. Your `visualization.js` code should still defend with `||` / `??` fallbacks because: + +- The user may have explicitly set the option to `null` or empty string. +- Options without a `default` will be `undefined` on first load. + +### 4. Author `visualization.js` + +```js +import { VisualizationAPI } from '@splunk/dashboard-studio-extension'; +import './visualization.css'; + +// ── Module-scope state survives across listener invocations because +// the iframe host keeps a single execution context per viz panel. +const state = { + loading: false, + rawData: null, + options: {}, + width: 0, + height: 0, + statusMsg: null, +}; + +// ── DOM setup runs once at module load. +const root = document.getElementById('root') || document.body; +const canvas = document.createElement('canvas'); +canvas.style.cssText = 'width:100%;height:100%;display:block;'; +root.appendChild(canvas); + +function sizeCanvas(w, h) { + const dpr = window.devicePixelRatio || 1; + const targetW = Math.max(1, Math.floor(w * dpr)); + const targetH = Math.max(1, Math.floor(h * dpr)); + if (canvas.width !== targetW || canvas.height !== targetH) { + canvas.width = targetW; + canvas.height = targetH; + } + canvas.style.width = w + 'px'; + canvas.style.height = h + 'px'; +} + +function draw() { + const { width: w, height: h } = state; + if (w <= 0 || h <= 0) return; + sizeCanvas(w, h); + const ctx = canvas.getContext('2d'); + if (!ctx) return; + const dpr = window.devicePixelRatio || 1; + ctx.setTransform(dpr, 0, 0, dpr, 0, 0); + ctx.clearRect(0, 0, w, h); + // ... draw using state.rawData and state.options ... +} + +VisualizationAPI.addDimensionsListener( + ({ width, height }) => { + state.width = width || 0; + state.height = height || 0; + draw(); + }, + { invokeImmediately: true } +); + +VisualizationAPI.addOptionsListener( + ({ options }) => { + state.options = options || {}; + draw(); + }, + { invokeImmediately: true } +); + +VisualizationAPI.addDataSourcesListener( + ({ dataSources, loading }) => { + // The refresh-stability fix: keep the previous frame on screen + // while the data source is reloading. The Dashboard Studio host + // shows its own refresh indicator in the corner. + state.loading = loading; + if (loading) return; + + const raw = dataSources?.primary?.data ?? null; + if (!raw) { + // First load with no data yet — render the empty shell. + if (!state.rawData) draw(); + return; + } + state.rawData = raw; + draw(); + }, + { invokeImmediately: true } +); +``` + +#### Listener cookbook + +| API | Payload | Use for | +|-----|---------|---------| +| `addDimensionsListener` | `{ width, height }` | Panel resize. Always pass `{ invokeImmediately: true }` so the first frame has dimensions. | +| `addOptionsListener` | `{ options }` | Format panel changes. `options` is keyed by the names in `optionsSchema`. Values are already type-coerced. | +| `addDataSourcesListener` | `{ dataSources, loading }` | New data arrived, or a refresh started. **Always early-return when `loading` is true** — this is the whole reason for using Track B. | +| `VisualizationAPI.setError({ title, message })` | — | Surface a user-visible error inside the panel. The host renders this; you do not draw it yourself. | +| `VisualizationAPI.clearError()` | — | Clear a previously set error. Call this when valid data starts flowing again. | +| `VisualizationAPI.setTrellisGroupBy(field)` | — | Optional — for vizs that participate in trellis layouts. Most custom vizs do not need this. | + +#### Data shape (very different from Track A) + +Dashboard Studio passes data **column-major**, not row-major: + +```js +dataSources.primary.data === { + fields: [{ name: 'value' }, { name: 'label' }, { name: '_status' }], + columns: [ + ['42'], // value column + ['Pressure'], // label column + ['Awaiting telemetry'] // _status column (only present if SPL emitted it) + ] + // Optionally also: meta, requestParams +} +``` + +To extract values: + +```js +function parseData(data, options) { + if (!data?.fields || !data?.columns) return null; + const idx = {}; + for (let i = 0; i < data.fields.length; i++) { + idx[data.fields[i].name] = i; + } + const vCol = data.columns[idx[options.valueField || 'value']]; + if (!vCol || vCol.length === 0) return null; + return { value: parseFloat(vCol[vCol.length - 1]) }; +} +``` + +A worked example with `_status` fallback handling lives in `examples/steampunk_gauge_studio/visualizations/steampunk_gauge_studio/src/visualization.js` — search for `parseData`. + +#### `_status` no-data pattern + +The SPL `appendpipe` trick from Track A rule 27 still works, but the JS side detects it via the column-major shape: + +```js +if (idx._status !== undefined) { + const col = data.columns[idx._status]; + if (col?.length && col[col.length - 1]) { + return { status: String(col[col.length - 1]) }; + } +} +``` + +When `parseData` returns a `{ status }` sentinel, draw a centred status message on the canvas instead of the normal viz. See `drawStatusMessage()` in the steampunk gauge studio for an emoji-plus-text implementation that auto-scales to fit 85% of the panel width. + +#### Smoothing / animation + +`requestAnimationFrame` works exactly as in Track A. State lives in module scope rather than `this`: + +```js +const state = { /* ... */, target: 0, current: 0, animFrame: null, idleFrames: 0, lastFrameTs: 0 }; + +function startAnim() { + if (state.animFrame !== null) return; + state.lastFrameTs = 0; + const step = (ts) => { + if (!state.lastFrameTs) state.lastFrameTs = ts; + const dt = Math.min(0.25, Math.max(0, (ts - state.lastFrameTs) / 1000)); + state.lastFrameTs = ts; + const smoothness = state.options.smoothness ?? 8; + const diff = state.target - state.current; + state.current += diff * (1 - Math.exp(-smoothness * dt)); + draw(); + if (Math.abs(diff) < 1e-3) { + if (++state.idleFrames > 6) { state.current = state.target; state.animFrame = null; return; } + } else { + state.idleFrames = 0; + } + state.animFrame = requestAnimationFrame(step); + }; + state.animFrame = requestAnimationFrame(step); +} +``` + +The host iframes are torn down when the panel is removed, so explicit cleanup of the rAF handle is rarely necessary in practice, but cancel it on visibility change if you observe sustained background CPU. + +### 5. Author `visualization.css` + +The CSS file is imported from `visualization.js` and inlined into the bundle by the `cssInjectAndSizeWarnPlugin` esbuild plugin. Keep it minimal and transparent so the dashboard background shows through: + +```css +html, body { + margin: 0; + padding: 0; + width: 100%; + height: 100%; + background: transparent; + overflow: hidden; +} + +#root, +.{app_name}-viz { + width: 100%; + height: 100%; + background: transparent; +} + +.{app_name}-viz canvas { + display: block; + width: 100%; + height: 100%; +} +``` + +The iframe inherits the host theme; do not set explicit text colours in CSS — pick them at draw time based on a setting or a theme-detection helper. + +## App configuration + +### `package/app/app.conf` + +```ini +[id] +name = {app_name} +version = 1.0.0 + +[install] +is_configured = 0 +build = <@- buildNumber @> + +[package] +id = {app_name} +check_for_updates = false + +[ui] +is_visible = 0 +label = {Display Label} +show_in_nav = 0 + +[launcher] +author = {author} +description = {description} +version = 1.0.0 + +[manifest] +category = Custom +``` + +Key requirements: + +- **`[id]` stanza is mandatory.** Without it, Splunk Cloud's `check_version_is_valid_semver` rule fails. `name` must match `[package] id`, `version` must be valid SemVer. +- **`check_for_updates = false`** in `[package]` keeps Splunk Cloud's `check_app_update_uri` quiet for private apps. +- **`<@- buildNumber @>`** is a template placeholder that `package.mjs` replaces with the build number it derives from the git short hash. Leave it as-is. +- **`is_visible = 0` and `show_in_nav = 0`** — a custom viz app is invisible in the launcher; users discover it through the viz picker. + +The packager's `parseAppConf()` only reads the keys it cares about (id, version, author, description, label, category). Any extra keys are preserved in the staged file but ignored by the build. + +### `metadata/` (optional in source) + +The packager generates `metadata/default.meta` from the discovered visualizations. The patched `generateDefaultMeta()` (see [Upstream bugs](#upstream-bugs)) writes: + +```ini +[] +access = read : [ * ], write : [ admin, sc_admin ] + +[visualizations/{app_name}] +export = system +``` + +If you need extra metadata stanzas, add a `package/metadata/default.meta` source file. The packager will copy it instead of regenerating. + +## Build and package + +Two equivalent entry points. Either works. + +**From the app directory** (useful during development for `npm run dev` watch mode): + +```bash +npm install # one-time +npm run build # dev build (sourcemaps, no minify) → dist/{app_name}/visualization.js +npm run build:prod # production build (minified, no maps) +npm run dev # esbuild watch mode for iterative development +npm run package # builds .spl into dist/{app_name}-{version}-{git}.spl +``` + +**From the repo root** (unified across Track A and Track B): + +```bash +./build.sh {app_name} # auto-detects Track B, runs build:prod + package, + # copies the .spl to dist/ at the repo root +./build.sh # builds every app in examples/, mixing Track A and B +``` + +`build.sh` invokes `npm run build:prod` followed by `npm run package` for Track B apps, then copies the per-app `.spl` to the top-level `dist/` so all outputs end up in one place. Use `npm run dev` directly when iterating — `build.sh` does not support watch mode. + +The packager's flow: + +1. Validate project structure (`package.json`, `visualizations/*/config.json`, `dist/*/visualization.js`). +2. Parse `package/app/app.conf` for `id` and `version`. +3. Clear `stage/{appId}/`, recreate it. +4. Stage `default/app.conf` (with `buildNumber` interpolated). +5. Copy `dist/{viz}/visualization.js` and `config.json` to `stage/{appId}/appserver/static/visualizations/{viz}/`. +6. Generate `default/visualizations.conf` listing every viz with `framework_type = studio_visualization`. +7. Generate `metadata/default.meta` with the global `[]` stanza and per-viz `[visualizations/{name}]` stanzas. +8. Generate `app.manifest` (Splunk app metadata). +9. Create `dist/{appId}-{version}-{shortHash}.spl` (gzipped tar). + +The `.spl` filter excludes dotfiles, so any `.DS_Store` / `.gitkeep` left behind in `stage/` will not end up in the tarball. + +## Install in Splunk + +1. Apps → Manage Apps → Install app from file +2. Upload `dist/{app_name}-{version}-{git}.spl` +3. Splunk may prompt for a restart — restart if asked +4. Open a Dashboard Studio dashboard, then Visualization picker → Custom → **{Display Label}** + +## Upstream bugs + +The `@splunk/create@11.0.0` template that scaffolds these projects has four known defects. The vendored `package.mjs` and `package/app/app.conf` in `examples/steampunk_gauge_studio/` already work around all four. See [`upstream-bugfix/BUG_REPORT.md`](../../../../upstream-bugfix/BUG_REPORT.md) for the full reproduction, root cause, and patch. + +| # | File | Defect | Workaround | +|---|------|--------|------------| +| 1 | `package.mjs` → `stageConfFiles` | Writes `default/visualizations.conf` without creating `default/`. Build fails with `ENOENT`. | `mkdirSync(defaultDir, { recursive: true })` before the write. | +| 2 | `package.mjs` → `main()` | Writes `app.conf`, then `rmSync`s the entire stage dir, deleting it. The `.spl` ships without `default/app.conf`. Splunk rejects the install. | Parse `app.conf` first, clear stage, then write `app.conf`. | +| 3 | `package.mjs` → `generateDefaultMeta` | Emits per-viz stanzas only. No global `[]` access stanza. Splunk Cloud `check_meta_default_write_access` fails. | Prepend `[]\naccess = read : [ * ], write : [ admin, sc_admin ]`. | +| 4 | `package/app/app.conf.template` | Missing `[id]` stanza and `check_for_updates`. Splunk Cloud `check_version_is_valid_semver` warns. | Add `[id]` with `name` and `version`; add `check_for_updates = false` to `[package]`. | + +When generating a new Track B project, copy the patched files from the steampunk gauge studio rather than re-running `@splunk/create` and re-patching by hand. + +## Splunk Cloud appinspect + +Run before submission: + +```bash +splunk-appinspect inspect dist/{app_name}-{version}-{git}.spl --mode test --included-tags cloud +``` + +A correctly built Track B app passes with no failures and no warnings, provided: + +- `app.conf` has the `[id]` stanza with valid SemVer. +- `metadata/default.meta` has the global `[]` write-access stanza (the patched `generateDefaultMeta` handles this). +- No dotfiles (`.DS_Store`, `.gitignore`) ended up in the tarball (the packager's `.spl` filter handles this). +- `visualizations.conf` uses `framework_type = studio_visualization` (the packager generates this). + +## Testing + +The Track A `harness.json` + `test-harness.html` pattern does **not** work for Track B vizs — they require the `@splunk/dashboard-studio-extension` runtime which only the real Dashboard Studio host provides. + +Recommended dev loop: + +1. `npm run dev` in the viz directory — esbuild watches `src/` and rebuilds `dist/{app_name}/visualization.js` on every save. +2. `npm run package` — re-creates the `.spl`. +3. Re-upload the `.spl` in Splunk (Apps → Manage Apps → Install app from file → "Upgrade app" if it already exists). +4. Hard-refresh the dashboard tab (`Cmd+Shift+R`). + +A faster inner loop, once the app is installed: edit the file directly inside `$SPLUNK_HOME/etc/apps/{app_name}/appserver/static/visualizations/{app_name}/visualization.js`, then `/_bump` and hard-refresh. Reserve this for tight tweaks — always re-package + re-install before submitting to Splunk Cloud. + +## Verify Completeness + +For a new Track B viz, check: + +- [ ] `package.json` has `"type": "module"`, depends on `@splunk/dashboard-studio-extension`, `esbuild`, `chalk`, `tar`. +- [ ] `build.mjs`, `package.mjs`, `build-plugins/css-and-size.mjs` copied from the patched reference (see [Upstream bugs](#upstream-bugs)). +- [ ] `package/app/app.conf` has `[id]` stanza with matching `name` and SemVer `version`. +- [ ] `package/app/app.conf` `[package]` includes `check_for_updates = false`. +- [ ] `visualizations/{app_name}/config.json` has `name`, `description`, `category`, `dataContract`, `optionsSchema` (with defaults), and `editorConfig`. +- [ ] Every option referenced from `editorConfig` exists in `optionsSchema`. +- [ ] `visualizations/{app_name}/src/visualization.js` uses `import { VisualizationAPI } from '@splunk/dashboard-studio-extension'`. +- [ ] `addDataSourcesListener` early-returns when `loading` is true (refresh-stability fix). +- [ ] All three listeners pass `{ invokeImmediately: true }` so first paint has dimensions, options, and data. +- [ ] Canvas sizing is HiDPI-aware and only resizes when target dimensions change. +- [ ] `_status` SPL fallback handled if a custom no-data message is requested. +- [ ] `visualization.css` keeps `background: transparent` on root and canvas. +- [ ] `npm run build:prod && npm run package` produces a `.spl` in `dist/`. +- [ ] (For Cloud) `splunk-appinspect inspect dist/...spl --mode test --included-tags cloud` reports no failures and no warnings. +- [ ] README documents the columns, SPL examples, and the 10.4-only minimum version. + +## Migrating a Track A viz to Track B + +If you already have a Track A viz and want a Track B counterpart (rather than a wholesale migration), build it as a **separate sibling app** and ship them side by side. Users on 10.2 install the legacy app; users on 10.4 install the studio app; the picker shows both as distinct entries. This is what `examples/steampunk_gauge/` and `examples/steampunk_gauge_studio/` do. + +Porting checklist (legacy → studio): + +| Legacy concept | Studio equivalent | +|---|---| +| `formatter.html` `` | `config.json` `editorConfig` entry | +| `formatter.html` `value="..."` default | `config.json` `optionsSchema[name].default` | +| `savedsearches.conf` `display.visualizations.custom.*` | Not needed — defaults come from `optionsSchema` | +| `savedsearches.conf.spec` | Not needed | +| `SplunkVisualizationBase.extend({...})` | Module-scope code + listeners | +| `initialize` | Top-level module code | +| `getInitialDataParams` | Not needed — Splunk hands you the full data each refresh | +| `formatData` | `parseData(rawData, options)` called inside `addDataSourcesListener` | +| `updateView` | `draw()` called from any listener | +| `reflow` | `addDimensionsListener` | +| `destroy` | iframe teardown; rAF cleanup if needed | +| `this._lastGoodData` cache | Module-scope `state.rawData` (the `loading` guard already prevents the flashing the cache was working around) | +| `throw VisualizationError(...)` | `VisualizationAPI.setError({ title, message })` | +| Row-major data (`data.rows`) | Column-major data (`data.columns[idx[fieldName]]`) | +| `config[ns + 'name']` | `options.name` (no namespace prefix) | +| `parseFloat(config[ns + 'n']) \|\| 0` | `options.n ?? 0` (already typed if schema says so) | + +The drawing helpers themselves (Canvas 2D primitives) port over verbatim — they were already pure functions of `ctx`, dimensions, and data in Track A (per legacy Rule 12). Modernise the JS to ESM `const`/`let`/arrow functions while you're there — Track B targets `es2017` via esbuild, so the ES5-only Rule 1 of Track A no longer applies. + +## Reference example + +`examples/steampunk_gauge_studio/` — production-ready studio extension viz with: + +- Refresh-stable rendering (no visible repaint on scheduled refresh in Splunk 10.4) +- All Track A patterns ported: zones, wear seed derived from config, smoothing animation, `_status` no-data message, HiDPI canvas +- All four upstream `@splunk/create` bugs patched +- Passes Splunk Cloud appinspect with no failures or warnings + +Read the source alongside this reference whenever you need a concrete example of any pattern described above. diff --git a/build.sh b/build.sh index e793525..2c749bb 100755 --- a/build.sh +++ b/build.sh @@ -4,45 +4,76 @@ set -euo pipefail # # build.sh — Build and package Splunk custom visualization apps # +# Auto-detects the framework track per app: +# Track A (legacy SplunkVisualizationBase, AMD + webpack) +# → expects examples/{app}/default/app.conf +# → runs `npm run build` inside appserver/static/visualizations/{app}/ +# → packages a .tar.gz under dist/ +# Track B (Dashboard Studio Extension, ESM + esbuild) +# → expects examples/{app}/package/app/app.conf +# → runs `npm run build:prod && npm run package` at the app root +# → copies the per-app .spl into dist/ +# # Usage: -# ./build.sh # Build all viz apps in examples/ -# ./build.sh custom_single_value # Build a specific viz app +# ./build.sh # Build all viz apps in examples/ +# ./build.sh steampunk_gauge # Build a specific viz app # -# Output: {app_name}-{version}.tar.gz in the dist/ directory +# Output: dist/{app_name}-{version}.tar.gz (Track A) +# dist/{app_name}-{version}-{git}.spl (Track B) # SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" EXAMPLES_DIR="$SCRIPT_DIR/examples" OUTPUT_DIR="$SCRIPT_DIR/dist" -# Optional: path to shared font CSS to prepend to visualization.css +# Optional: path to shared font CSS to prepend to legacy visualization.css. +# Track B inlines its own CSS via esbuild and ignores this. FONT_CSS="$SCRIPT_DIR/shared/fonts.css" mkdir -p "$OUTPUT_DIR" -build_app() { - local APP_NAME="$1" - local APP_DIR="$EXAMPLES_DIR/$APP_NAME" - local VIZ_DIR="$APP_DIR/appserver/static/visualizations/$APP_NAME" +# --------------------------------------------------------------------------- +# Track detection +# --------------------------------------------------------------------------- - if [ ! -d "$APP_DIR" ]; then - echo "Error: App directory not found: $APP_DIR" - return 1 +detect_track() { + local APP_DIR="$1" + if [ -f "$APP_DIR/package/app/app.conf" ]; then + echo "B" + elif [ -f "$APP_DIR/default/app.conf" ]; then + echo "A" + else + echo "unknown" fi +} - if [ ! -f "$APP_DIR/default/app.conf" ]; then - echo "Error: No app.conf found in $APP_DIR/default/" - return 1 - fi +read_version() { + # First `^version` line wins. Both Track A and Track B keep [id]/version + # at the top of app.conf, so head -1 picks the canonical app version. + grep -E '^version' "$1" | head -1 | cut -d= -f2 | tr -d ' ' +} + +# --------------------------------------------------------------------------- +# Track A — legacy SplunkVisualizationBase +# --------------------------------------------------------------------------- + +build_app_legacy() { + local APP_NAME="$1" + local APP_DIR="$EXAMPLES_DIR/$APP_NAME" + local VIZ_DIR="$APP_DIR/appserver/static/visualizations/$APP_NAME" local VERSION - VERSION=$(grep '^version' "$APP_DIR/default/app.conf" | head -1 | cut -d= -f2 | tr -d ' ') + VERSION=$(read_version "$APP_DIR/default/app.conf") local TARBALL="$OUTPUT_DIR/${APP_NAME}-${VERSION}.tar.gz" - echo "=== Building: $APP_NAME v$VERSION ===" + echo "=== Building (Track A / legacy): $APP_NAME v$VERSION ===" echo "" - # Step 1: Install dependencies + if [ ! -d "$VIZ_DIR" ]; then + echo "Error: legacy viz directory missing: $VIZ_DIR" + return 1 + fi + if [ ! -d "$VIZ_DIR/node_modules" ]; then echo "[1/3] Installing npm dependencies..." (cd "$VIZ_DIR" && npm install --silent) @@ -50,11 +81,9 @@ build_app() { echo "[1/3] Dependencies already installed, skipping." fi - # Step 2: Build webpack bundle echo "[2/3] Building visualization bundle..." (cd "$VIZ_DIR" && npm run build --silent) - # Step 3: Optionally prepend shared font CSS local CSS_MODIFIED=false local ORIGINAL_CSS="" local VIZ_CSS="$VIZ_DIR/visualization.css" @@ -66,10 +95,8 @@ build_app() { CSS_MODIFIED=true fi - # Step 4: Package tarball echo "[3/3] Packaging $TARBALL..." - # Build tar flags (macOS needs extra flags to suppress resource forks) local TAR_FLAGS=() if [[ "$(uname)" == "Darwin" ]]; then xattr -rc "$APP_DIR" 2>/dev/null || true @@ -88,7 +115,6 @@ build_app() { -C "$EXAMPLES_DIR" \ "$APP_NAME" - # Restore original CSS if we modified it if [ "$CSS_MODIFIED" = true ]; then echo "$ORIGINAL_CSS" > "$VIZ_CSS" fi @@ -99,20 +125,109 @@ build_app() { echo "" } +# --------------------------------------------------------------------------- +# Track B — Dashboard Studio Extension +# --------------------------------------------------------------------------- + +build_app_studio() { + local APP_NAME="$1" + local APP_DIR="$EXAMPLES_DIR/$APP_NAME" + + local VERSION + VERSION=$(read_version "$APP_DIR/package/app/app.conf") + + echo "=== Building (Track B / Studio Extension): $APP_NAME v$VERSION ===" + echo "" + + if [ ! -f "$APP_DIR/package.json" ]; then + echo "Error: $APP_DIR/package.json is missing — Track B requires it at the app root." + return 1 + fi + if [ ! -f "$APP_DIR/package.mjs" ] || [ ! -f "$APP_DIR/build.mjs" ]; then + echo "Error: $APP_DIR is missing build.mjs or package.mjs." + echo " Copy the patched scripts from examples/steampunk_gauge_studio/." + return 1 + fi + + if [ ! -d "$APP_DIR/node_modules" ]; then + echo "[1/3] Installing npm dependencies..." + (cd "$APP_DIR" && npm install --silent) + else + echo "[1/3] Dependencies already installed, skipping." + fi + + echo "[2/3] Building production bundle (esbuild)..." + (cd "$APP_DIR" && npm run build:prod --silent) + + echo "[3/3] Packaging .spl..." + (cd "$APP_DIR" && npm run package --silent) + + # The Track B packager (package.mjs) writes .spl into the per-app + # examples/{app}/dist/ directory and names the file with the git short + # hash so multiple builds accumulate. Pick the newest one and copy it + # to the repo-level dist/ so all build.sh output lives in one place. + local LATEST_SPL="" + if compgen -G "$APP_DIR/dist/*.spl" > /dev/null; then + # shellcheck disable=SC2012 + LATEST_SPL=$(ls -t "$APP_DIR/dist/"*.spl | head -1) + fi + if [ -z "$LATEST_SPL" ] || [ ! -f "$LATEST_SPL" ]; then + echo "Error: no .spl produced under $APP_DIR/dist/" + return 1 + fi + + cp -f "$LATEST_SPL" "$OUTPUT_DIR/" + local COPIED="$OUTPUT_DIR/$(basename "$LATEST_SPL")" + + echo "" + echo "Done! Install with:" + echo " \$SPLUNK_HOME/bin/splunk install app $COPIED" + echo "" +} + +# --------------------------------------------------------------------------- +# Dispatcher +# --------------------------------------------------------------------------- + +build_app() { + local APP_NAME="$1" + local APP_DIR="$EXAMPLES_DIR/$APP_NAME" + + if [ ! -d "$APP_DIR" ]; then + echo "Error: App directory not found: $APP_DIR" + return 1 + fi + + local TRACK + TRACK=$(detect_track "$APP_DIR") + + case "$TRACK" in + A) build_app_legacy "$APP_NAME" ;; + B) build_app_studio "$APP_NAME" ;; + *) + echo "Error: $APP_NAME has neither default/app.conf (Track A)" + echo " nor package/app/app.conf (Track B). Skipping." + return 1 + ;; + esac +} + +# --------------------------------------------------------------------------- # Main +# --------------------------------------------------------------------------- + if [ $# -gt 0 ]; then - # Build specific app(s) for app in "$@"; do build_app "$app" done else - # Build all apps in examples/ echo "Building all visualization apps..." echo "" for app_dir in "$EXAMPLES_DIR"/*/; do app_name=$(basename "$app_dir") - if [ -f "$app_dir/default/app.conf" ]; then - build_app "$app_name" + if [ -f "$app_dir/default/app.conf" ] || [ -f "$app_dir/package/app/app.conf" ]; then + build_app "$app_name" || echo "(continuing with next app)" + echo "" fi done fi From cecf8ec7c9e5a859cbf612ed752bc31a2df95a2e Mon Sep 17 00:00:00 2001 From: "Erwin Vrolijk (technimad-splunk)" Date: Tue, 26 May 2026 20:04:57 +0200 Subject: [PATCH 2/2] update test harnass to support the new visualization framework --- .claude/skills/splunk-viz/SKILL.md | 2 +- .../splunk-viz/references/dse-framework.md | 72 +++++++++-- test-harness.html | 118 ++++++++++++++++++ 3 files changed, 184 insertions(+), 8 deletions(-) diff --git a/.claude/skills/splunk-viz/SKILL.md b/.claude/skills/splunk-viz/SKILL.md index efecffd..8f62451 100644 --- a/.claude/skills/splunk-viz/SKILL.md +++ b/.claude/skills/splunk-viz/SKILL.md @@ -925,7 +925,7 @@ A complete worked example lives at `examples/steampunk_gauge_studio/`. Refer to **Critical Rules that do NOT apply to Track B**: 1 (Track B is ESM, not AMD — modern syntax encouraged), 7–8 (no `formatData`/`VisualizationError` — use `setError`/`clearError` from VisualizationAPI), 9 (settings come typed, not always strings, when `optionsSchema` declares `type: number` or `type: boolean`), 11 (no AMD JSON loader — use ESM `import`), 13 (no `this`-bound timer — use module-scope state), 16–17 (no `invalidate*`/`reflow` methods — listeners drive re-render), 18–21 (no `getPropertyNamespaceInfo`, no namespaced config keys, no `formatData` caching needed — the listener model + cached state already prevents re-render during loading), 22 (no `savedsearches.conf.spec` — defaults live in `optionsSchema`), 23 (`/_bump` still works but you typically re-install the `.spl` via the UI), 28 (Dashboard Studio JSON `options` block — Track B writes well-typed values directly, no stale-key issue). -For the harness, see the [Studio Extension Harness](references/dse-framework.md#testing) section of the reference doc — the legacy Step 5 harness pattern below does not apply to Track B vizs. +**Track B vizs work in the test harness** (as of harness v2). The harness mounts each studio viz in an `