-
Notifications
You must be signed in to change notification settings - Fork 111
feat(expo): anonymous usage telemetry and skill feedback #71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zvadaadam
wants to merge
27
commits into
main
Choose a base branch
from
zvadaadam/skills-usage-observability
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
be94fa2
feat(expo): add anonymous usage telemetry and skill feedback
1a99b31
refactor(expo): auto-detect harness, self-derive plugin root, slim docs
bfca26b
feat(expo): skip telemetry in CI, add os/arch, first-run notice
ebc0249
fix(expo): track skill_invoked via the Skill tool (Read hook never fi…
3395c90
feat(expo): split skill_invoked by initiator (AI vs user)
1a03c1b
refactor(expo): simplify telemetry to skill_invoked + skill_feedback
280489e
Merge origin/main into zvadaadam/skills-usage-observability
8574435
docs(expo): correct Codex hook status + mark the enablement path
0dd91db
docs(expo): record in-depth Codex finding — skill telemetry not hook-…
5a9cf78
feat(expo): make skill_invoked resolver payload-tolerant (Codex-ready)
0949797
chore(expo): apply code-review polish to skill telemetry
1f9c471
Merge origin/main into zvadaadam/skills-usage-observability
5c690e3
fix(expo): address skill-telemetry PR review (Codex, latency, consent…
4143946
docs(expo): broaden skill-feedback triggers + de-dup harness story
4f95bda
fix(expo): reference feedback script via ${CLAUDE_PLUGIN_ROOT}, not a…
db9eca4
refactor(expo): focus skill telemetry on Claude Code; drop other-agen…
5211d90
feat(expo): make the skill footer a proactive feedback nudge + pointer
287960a
feat(expo): rename to expo-skill-feedback, add Expo-itself feedback, …
a171cb6
Merge origin/main (expo-examples #84) into the telemetry branch
08c3ec0
refactor(expo): prune unnecessary telemetry machinery
d8c6f57
refactor(expo): drop scripts/package.json — use .cjs instead
422f03e
refactor(expo): prune expo-skill-feedback SKILL.md to its two jobs
e5d48ab
refactor(expo): drop references/telemetry.md — it duplicated the code
772d40a
refactor(expo): run skill telemetry via node, drop sh wrappers
061f807
feat(expo): make telemetry opt-in (off by default), drop skill footers
9520de0
Merge origin/main into the telemetry branch
67a9a60
docs(expo): tighten the AGENTS.md telemetry section
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| { | ||
| "hooks": { | ||
| "PostToolUse": [ | ||
| { | ||
| "matcher": "Skill", | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "node \"${CLAUDE_PLUGIN_ROOT}/skills/expo-skill-feedback/scripts/skill-event.cjs\" --skill auto --initiator ai --plugin-root \"${CLAUDE_PLUGIN_ROOT}\" --detach --quiet", | ||
| "timeout": 5 | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "UserPromptExpansion": [ | ||
| { | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "node \"${CLAUDE_PLUGIN_ROOT}/skills/expo-skill-feedback/scripts/skill-event.cjs\" --skill auto --initiator user --plugin-root \"${CLAUDE_PLUGIN_ROOT}\" --detach --quiet", | ||
| "timeout": 5 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| --- | ||
| name: expo-skill-feedback | ||
| description: Submit feedback on an Expo skill — or on Expo itself — or turn the bundled anonymous usage telemetry on or off (off by default / opt-in; the user saying "enable Expo skills telemetry" in conversation is the switch). Use when an Expo skill was useful, confusing, broken, missing context, or worth improving; when something fell short because of Expo (an SDK bug or confusing framework behavior) rather than the skill; or when the user wants to enable, turn on, opt in to, disable, turn off, opt out of, check the status of, or understand the anonymous usage tracking these skills can send. | ||
| --- | ||
|
|
||
| # Expo Skill Feedback | ||
|
|
||
| Send anonymous feedback on an Expo skill — or on Expo itself — and turn the bundled usage | ||
| telemetry on or off. **Telemetry is off by default (opt-in)** — nothing is sent until it's | ||
| enabled. Feedback goes to the Expo team via PostHog. Only the skill name, your rating + note, | ||
| platform, and a hash of a random local install id are sent — never code, prompts, file paths, | ||
| secrets, or personal data. (A separate automatic `skill_invoked` event fires when a skill runs | ||
| *once enabled*; the switch below controls both.) | ||
|
|
||
| ## Submitting feedback | ||
|
|
||
| Be proactive: when a skill was wrong, confusing, outdated, or missing — or the trouble is | ||
| with **Expo itself** — send 1-3 concrete sentences. Pass the skill you're rating to | ||
| `--skill` (the skill being rated, not `expo-skill-feedback`): | ||
|
|
||
| ```bash | ||
| node "${CLAUDE_PLUGIN_ROOT}/skills/expo-skill-feedback/scripts/skill-feedback.cjs" \ | ||
| --skill expo-deployment --rating useful \ | ||
| --text "TestFlight steps were clear and worked first try." | ||
| ``` | ||
|
|
||
| - `--rating`: `useful` · `confusing` · `bug` · `idea` · `other` | ||
| - `--about`: `skill` (default) · `expo` (the issue is Expo itself, not the skill) | ||
| - `--dry-run` prints the payload without sending | ||
|
|
||
| Never include secrets, source code, long prompts, or stack traces. | ||
|
|
||
| If the command refuses because telemetry is off, don't drop the feedback — ask the user | ||
| once (see *Turning it on or off*), and resend the same command after they enable. | ||
|
|
||
| ## Turning it on or off | ||
|
|
||
| Telemetry is anonymous and **off by default (opt-in)** — nothing (neither the automatic | ||
| `skill_invoked` event nor feedback) is sent until the user enables it. Saying so in | ||
| conversation is the switch: when the user says **"enable Expo skills telemetry"** — or | ||
| answers yes when you offer — run: | ||
|
|
||
| ```bash | ||
| node "${CLAUDE_PLUGIN_ROOT}/skills/expo-skill-feedback/scripts/telemetry.cjs" --on | ||
| ``` | ||
|
|
||
| `--off` turns it off again. When the user asks whether telemetry is on, run `--status` | ||
| and relay its output — don't answer from memory; env vars and CI can override the saved | ||
| state. Env equivalents: `EXPO_SKILLS_TELEMETRY=1` to enable, `=0` or `DO_NOT_TRACK=1` to | ||
| disable; CI never sends. | ||
|
|
||
| **Never enable it on your own.** If a feedback send was refused because telemetry is off, | ||
| ask the user once — "Expo skills telemetry is off; enable it (anonymous) and send this?" — | ||
| and run `--on` only after an explicit yes. If they decline, drop the feedback silently. |
177 changes: 177 additions & 0 deletions
177
plugins/expo/skills/expo-skill-feedback/scripts/skill-event.cjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| #!/usr/bin/env node | ||
| // Submit a `skill_invoked` event to PostHog — fire-and-forget. | ||
| // | ||
| // Invoked two ways, both cross-platform (plain `node`, no shell wrapper, so it runs the | ||
| // same on macOS / Linux / Windows): | ||
| // 1. From Claude Code hooks (../../../hooks/hooks.json) as | ||
| // `node skill-event.cjs --skill auto --initiator <ai|user> --plugin-root <dir> --detach --quiet`. | ||
| // The foreground process reads the hook payload from stdin, resolves which skill ran, | ||
| // runs the cheap local gates, then — for `--detach` — re-launches a DETACHED copy of | ||
| // itself to do the network POST and returns immediately, so the agent turn never blocks. | ||
| // 2. Directly (that detached child, or manual / --dry-run testing) with a resolved | ||
| // `--skill <name>`, which builds the event and sends it inline. | ||
|
|
||
| const fs = require("fs"); | ||
| const path = require("path"); | ||
|
|
||
| const { | ||
| POSTHOG_PROJECT_API_KEY, | ||
| SOURCE, | ||
| telemetryActive, | ||
| telemetryConfigured, | ||
| detectHarness, | ||
| platformProps, | ||
| telemetryIdentity, | ||
| sendToPosthog, | ||
| } = require("./telemetry_common.cjs"); | ||
|
|
||
| const EVENT = "skill_invoked"; | ||
|
|
||
| function parseArgs(argv) { | ||
| const args = { skill: "", agentHarness: "", initiator: "", pluginRoot: "", dryRun: false, quiet: false, detach: false }; | ||
| for (let i = 0; i < argv.length; i++) { | ||
| const flag = argv[i]; | ||
| const next = () => argv[++i] || ""; | ||
| switch (flag) { | ||
| case "--skill": args.skill = next(); break; | ||
| case "--agent-harness": args.agentHarness = next(); break; | ||
| case "--initiator": args.initiator = next(); break; | ||
| case "--plugin-root": args.pluginRoot = next(); break; | ||
| case "--detach": args.detach = true; break; | ||
| case "--dry-run": args.dryRun = true; break; | ||
| case "--quiet": args.quiet = true; break; | ||
| default: break; // ignore unknown flags | ||
| } | ||
| } | ||
| return args; | ||
| } | ||
|
|
||
| // Read the hook payload from stdin (fd 0) and parse it as JSON. Only used to resolve | ||
| // `--skill auto` in the foreground hook process; the detached sender is handed the | ||
| // already-resolved name and never touches stdin. | ||
| function readHookInput() { | ||
| try { | ||
| if (process.stdin.isTTY) return {}; | ||
| const raw = (fs.readFileSync(0, "utf8") || "").trim(); // fd 0 = stdin | ||
| if (!raw) return {}; | ||
| const parsed = JSON.parse(raw); | ||
| return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed : {}; | ||
| } catch { | ||
| return {}; | ||
| } | ||
| } | ||
|
|
||
| // Resolve the invoked skill name from the hook payload. The name arrives in different | ||
| // fields across harnesses and payload shapes, so check every plausible location — the | ||
| // strict skillBelongsToPlugin() scoping downstream keeps this safe even when permissive | ||
| // (anything that isn't really one of our skills is dropped): | ||
| // - Claude Code Skill tool: tool_input.skill (e.g. "expo:expo-observe") | ||
| // - Claude Code /slash command: command_name (UserPromptExpansion) | ||
| // - other payload shapes: tool_input.skill_name, top-level skill / skill_name | ||
| // Plugin skills are namespaced (e.g. "expo:expo-observe") — keep the final segment. | ||
| function skillFromHook(hookInput) { | ||
| const ti = hookInput && typeof hookInput.tool_input === "object" && hookInput.tool_input ? hookInput.tool_input : {}; | ||
| const raw = String( | ||
| ti.skill || ti.skill_name || hookInput.command_name || hookInput.skill || hookInput.skill_name || "" | ||
| ).trim().replace(/^\//, ""); // tolerate a leading "/" from slash-command payloads | ||
| return raw.includes(":") ? raw.slice(raw.lastIndexOf(":") + 1) : raw; | ||
| } | ||
|
|
||
| function pluginRootFor(args) { | ||
| // Self-derive from this script's location: <root>/skills/expo-skill-feedback/scripts. | ||
| return args.pluginRoot || path.resolve(__dirname, "..", "..", ".."); | ||
| } | ||
|
|
||
| // Only emit for skills that belong to THIS plugin (so we never track other plugins' | ||
| // or the user's own skills). Confirms <pluginRoot>/skills/<skill>/SKILL.md exists. | ||
| // The skill name must be a single kebab-case segment — this also blocks path traversal | ||
| // (e.g. "../../x") from a malformed payload reaching path.join or the event property. | ||
| function skillBelongsToPlugin(skill, pluginRoot) { | ||
| if (!skill || !pluginRoot) return false; | ||
| if (!/^[a-z0-9][a-z0-9-]*$/.test(skill)) return false; | ||
| try { return fs.existsSync(path.join(pluginRoot, "skills", skill, "SKILL.md")); } | ||
| catch { return false; } | ||
| } | ||
|
|
||
| function eventPayload(skill, args) { | ||
| const agentHarness = args.agentHarness.trim() || detectHarness(); | ||
| const initiator = args.initiator.trim(); | ||
| const timestamp = new Date().toISOString(); | ||
| const [distinctId, identityProperties] = telemetryIdentity(agentHarness, { createInstallation: !args.dryRun }); | ||
|
|
||
| const properties = { | ||
| $process_person_profile: false, | ||
| source: SOURCE, | ||
| skill, | ||
| agent_harness: agentHarness, | ||
| ...(initiator ? { initiator } : {}), | ||
| ...platformProps(), | ||
| ...identityProperties, | ||
| }; | ||
|
|
||
| return { api_key: POSTHOG_PROJECT_API_KEY, event: EVENT, distinct_id: distinctId, timestamp, properties }; | ||
| } | ||
|
|
||
| // Re-launch this script DETACHED to perform the network POST off the agent's critical | ||
| // path — the cross-platform equivalent of `node skill-event.cjs … &`. We pass the already | ||
| // resolved `--skill <name>` (not "auto") and drop `--detach`, so the child sends inline and | ||
| // never reads stdin or re-detaches. It runs under the same runtime that launched us | ||
| // (process.execPath = node or bun) and inherits our env (CLAUDECODE, EXPO_SKILLS_*, …). | ||
| // windowsHide avoids a console-window flash on Windows; failures are ignored (best-effort). | ||
| function spawnDetachedSend(skill, args) { | ||
| try { | ||
| const { spawn } = require("child_process"); | ||
| const childArgs = [__filename, "--skill", skill, "--quiet"]; | ||
| if (args.initiator.trim()) childArgs.push("--initiator", args.initiator.trim()); | ||
| if (args.agentHarness.trim()) childArgs.push("--agent-harness", args.agentHarness.trim()); | ||
| const child = spawn(process.execPath, childArgs, { detached: true, stdio: "ignore", windowsHide: true }); | ||
| child.unref(); | ||
| } catch { | ||
| // best-effort: if the child can't be spawned, skip the send rather than block | ||
| } | ||
| } | ||
|
|
||
| async function main(argv) { | ||
| const args = parseArgs(argv); | ||
|
|
||
| // Resolve which skill ran. `--skill auto` means "read it from the hook payload on | ||
| // stdin" — which must happen HERE, in the foreground hook process, because a detached | ||
| // child's stdin is /dev/null. | ||
| let skill = args.skill.trim(); | ||
| if (skill === "auto") skill = skillFromHook(readHookInput()); | ||
|
|
||
| // Cheap, local, no-network gates: decide up front whether anything will be sent, so the | ||
| // common "not an Expo skill / opted out" cases cost nothing and never spawn a child. | ||
| if (!skill) return 0; // not a skill invocation | ||
| if (!args.dryRun && !telemetryActive()) return 0; // opt-in: off until enabled (dry-run inspects regardless) | ||
| if (!telemetryConfigured() && !args.dryRun) return 0; // no key in this build (e.g. a fork) -> inert | ||
| if (!skillBelongsToPlugin(skill, pluginRootFor(args))) return 0; // not one of ours | ||
|
|
||
| // Hook path: hand the network POST to a detached copy of ourselves so the turn never | ||
| // blocks on it, then return immediately. (--dry-run stays inline so it can be inspected.) | ||
| if (args.detach && !args.dryRun) { | ||
| spawnDetachedSend(skill, args); | ||
| return 0; | ||
| } | ||
|
|
||
| const payload = eventPayload(skill, args); | ||
|
|
||
| if (args.dryRun) { | ||
| console.log(JSON.stringify({ ...payload, api_key: "phc_..." }, null, 2)); | ||
| return 0; | ||
| } | ||
|
|
||
| try { | ||
| await sendToPosthog(payload, { userAgent: "expo-skills/skill-event", timeoutMs: 3000 }); | ||
|
Kudo marked this conversation as resolved.
|
||
| } catch (err) { | ||
| if (!args.quiet) console.error(`skill-event: ${err.message}`); | ||
| return args.quiet ? 0 : 1; | ||
| } | ||
|
|
||
| if (!args.quiet) console.log(`sent ${EVENT}: ${payload.properties.skill} (${payload.properties.initiator || "?"})`); | ||
| return 0; | ||
| } | ||
|
|
||
| main(process.argv.slice(2)) | ||
| .then((code) => process.exit(code)) | ||
| .catch(() => process.exit(0)); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[💡 suggestion] The namespace is stripped before the scoping check, so another plugin's skill with a colliding name is counted as ours: a payload of
tool_input.skill = "foo:expo-ui"resolves toexpo-uiand passesskillBelongsToPlugin()(verified with a crafted hook payload). When a namespace is present, require it to beexpo:If harnesses ever send a differently-qualified namespace this would drop real events, so feel free to loosen — but the current code over-counts, and the comment above claiming downstream scoping "keeps this safe" doesn't hold for name collisions.