Skip to content
BranchesTags

Repository files navigation

JSTestabilityStudy

JavaScript Testability Metrics (Language-Aware, Type-Agnostic)

These metrics capture the testability dimensions of JavaScript code (controllability, observability, complexity, asynchrony, encapsulation, and side effects) in a way that reflects the dynamic and event-driven nature of the language.
Each metric includes its purpose, definition, formula, and static analysis approach.

📋 Summary of JavaScript Testability Metrics

Metric Purpose Definition / Formula Key Static Analysis Features
C: Controllability Measures how easily inputs can steer execution. C = wP·#params + wD·#defaults + wE·isExported − wG·#globalRefs − wCl·#closureRefs − wSI·#staticImportsNewables Count parameters and defaults; identify global/closure refs; detect exports and hard-coded dependencies.
O: Observability Measures how easily outputs or effects can be observed. O = wR·hasReturnValue + wL·#logWrites + wIO·#ioWrites − wH·#hiddenStateWrites Detect return statements, console/DOM/FS/network writes, and hidden state mutations.
BC: Branching Complexity Captures control-flow complexity that increases test effort. BC = (#if + #elseIf + #switch + #ternary + #tryCatch + #loops) + nestingDepth Count branching and loop constructs; compute nesting depth from AST.
AC: Asynchronous Complexity Quantifies complexity from async constructs. AC = #await + #thenCatchFinally + #PromiseAllRaceAny + #timers + chainDepth Identify async/await, promises, callbacks, and timers.
EC: Event-Driven Complexity Measures reactivity and event-based control. EC = #domListeners + #nodeEventEmitter + #customEmitter + #addEventListenerCalls + #reactHandlers + #routerHandlers Detect addEventListener, .on/.emit, Express routes, React/Vue event handlers.
EL: Encapsulation Level Indicates reliance on hidden or private state. EL = #closureReads + #closureWrites + #moduleSingletonAccess + #prototypePrivateHeuristics Track closure access, singleton mutations, and private/prototype field usage.
SEI: Side-Effect Index Measures side-effect density relative to function size. SEI = #sideEffectCalls / LOCf Identify effectful APIs (fs, fetch, DOM, timers, randomness) and compute per-LOC ratio.
ER: Entry-Point Reachability Marks whether function is reachable from an external entry point. ER ∈ {0,1} Detect exported functions, CLI commands, route handlers, and reachable nodes in call graph.
CTS: Composite Testability Score Aggregates normalized metrics into a single testability measure. CTS = α·Ĉ + β·Ō − γ·BĈ − δ·ÂĈ − ε·ÊĈ − ζ·ĖĻ − η·ŜĖĨ + κ·ER Normalize metrics (0–1), assign weights, and combine controllability, observability, and complexity dimensions.

Legend:

  • Positive metrics (↑ improves testability): C, O, ER
  • Negative metrics (↑ decreases testability): BC, AC, EC, EL, SEI
  • Composite score (CTS) summarizes overall testability across all dimensions.

1. Controllability (C)

Purpose:
How easily can tests steer the execution of the code through inputs?

Definition:
Measures how exposed the function is to external control (parameters, exports) vs. hidden dependencies (globals, closures).

Formula:
C = wP·#params + wD·#defaults + wE·isExported − wG·#globalRefs − wCl·#closureRefs − wSI·#staticImportsNewables

Static Analysis:

  • Count parameters and default values.
  • Identify references to undeclared variables (globals/closures).
  • Detect exported functions (export, module.exports, exports.*).
  • Penalize direct instantiations of imported classes or singletons.

2. Observability (O)

Purpose:
How easily can a test observe the function’s effects or outputs?

Definition:
Measures return values, logging, and visible effects (DOM, FS, network).

Formula:
O = wR·hasReturnValue + wL·#logWrites + wIO·#ioWrites − wH·#hiddenStateWrites

Static Analysis:

  • Check for return statements with values.
  • Detect console.*, DOM updates, file or network operations.
  • Penalize mutations to hidden or closure-scoped state.

3. Branching Complexity (BC)

Purpose:
Quantifies internal control-flow complexity.

Definition:
Counts conditional and looping structures, plus nesting.

Formula:
BC = (#if + #elseIf + #switch + #ternary + #tryCatch + #loops) + nestingDepth

Static Analysis:

  • Count control structures (IfStatement, SwitchStatement, TryStatement, For*, While*).
  • Track nesting depth during AST traversal.

4. Asynchronous Complexity (AC)

Purpose:
Captures how difficult it is to coordinate and observe asynchronous behavior.

Definition:
Based on async/await usage, promise chains, and timers.

Formula:
AC = #await + #thenCatchFinally + #PromiseAllRaceAny + #timers + chainDepth

Static Analysis:

  • Count AwaitExpression, .then/.catch/.finally calls.
  • Identify Promise.all, Promise.race, etc.
  • Count timers (setTimeout, setInterval, queueMicrotask).

5. Event-Driven Complexity (EC)

Purpose:
Reflects difficulty from callbacks, listeners, and reactive flows.

Definition:
Counts event-driven constructs in browser, Node.js, or frameworks.

Formula:
EC = #domListeners + #nodeEventEmitter + #customEmitter

  • #addEventListenerCalls + #reactHandlers + #routerHandlers

Static Analysis:

  • Detect addEventListener, on*= handlers.
  • Identify .on, .emit, .once calls on EventEmitters.
  • Heuristically detect Express routes or React/Vue handler props.

6. Encapsulation Level (EL)

Purpose:
Estimates how dependent the function is on hidden or private state.

Definition:
Measures closure variable access, private/prototype fields, and module singletons.

Formula:
EL = #closureReads + #closureWrites + #moduleSingletonAccess + #prototypePrivateHeuristics

Static Analysis:

  • Identify free variables (not declared locally).
  • Detect writes to top-level objects or private class fields (#name, _name).
  • Count access to shared state (module scope or singleton patterns).

7. Side-Effect Index (SEI)

Purpose:
Quantifies how side-effect-prone a function is relative to its size.

Definition:
Ratio of side-effecting calls to total lines of code (LOC).

Formula:
SEI = #sideEffectCalls / LOCf

Static Analysis:

  • Recognize known effectful APIs:
    fs.*, fetch, axios, process.*, localStorage, DOM writes, timers, Math.random, Date.now.
  • Compute LOC per function.

8. Entry-Point Reachability (ER)

Purpose:
Determines whether a function is reachable from a real application entry point.

Definition:
Boolean reachability flag based on exports, event handlers, or call graph traversal.

Formula:
ER ∈ {0, 1}

Static Analysis:

  • Mark exported or top-level invoked functions as roots.
  • Use an intra-project call graph to mark reachable functions.

9. Composite Testability Score (CTS)

Purpose:
Combines all metrics into a single normalized score for ranking and comparison.

Formula:
CTS = α·Ĉ + β·Ō − γ·BĈ − δ·ÂĈ − ε·ÊĈ − ζ·ĖĻ − η·ŜĖĨ + κ·ER

Where each metric is normalized (m̂ ∈ [0,1]) and weighted.

Interpretation:
Higher CTS → easier to control, observe, and test;
Lower CTS → complex, hidden, and hard to validate.

Recommended Tooling Stack

Component Purpose Library
Parser Parse JS files @babel/parser
Traversal Walk and analyze AST @babel/traverse
Scope tracking Identify local vs. closure/global path.scope.hasBinding(name)
Config Store API allowlists, thresholds, weights JSON/YAML
Output Structured metric results JSON per function/file/project

Example Function Output

{
  "id": "src/utils/format.js:formatDate",
  "file": "src/utils/format.js",
  "metrics": {
    "C": { "params": 2, "globals": 0, "closures": 1, "score": 2.0 },
    "O": { "hasReturn": true, "logs": 0, "ioWrites": 0, "score": 1.0 },
    "BC": { "branches": 3, "nesting": 1, "score": 4 },
    "AC": { "awaits": 0, "thens": 0, "timers": 0, "score": 0 },
    "EC": { "listeners": 0, "emitters": 0, "score": 0 },
    "EL": { "closureReads": 1, "closureWrites": 0, "score": 1 },
    "SEI": { "calls": 0, "loc": 22, "score": 0 },
    "ER": 1,
    "CTS": 0.83
  }
}

How These Metrics Capture JavaScript Testability

These metrics collectively capture the multi-dimensional nature of testability in JavaScript, accounting for control, observation, complexity, and hidden behavior.
Each metric contributes to explaining why certain code is easier or harder to test.

Pattern / Property Captured By Example / Explanation
Heavy global or closure use C, EL Reduces controllability; behavior depends on hidden or shared state (window.config, process.env).
Lack of explicit return / observable output O, EL Tests cannot easily assert outcomes when only side effects occur.
Deep branching or nested logic BC Increases number of possible execution paths that need testing.
Complex async chains AC Promises, callbacks, and timers make sequencing and observation difficult.
Event-driven behavior EC Asynchronous and reactive patterns make triggering conditions non-deterministic.
Hidden shared state / module singletons EL, SEI Introduces coupling between tests and execution order.
High side-effect density SEI More external interactions (I/O, network, DOM) make testing expensive or flaky.
Unreachable or unused code ER Lowers practical test relevance; unreachable functions are not worth testing.
High composite score (CTS) — Indicates functions that are easy to test in isolation, with clear inputs and outputs.
Low composite score (CTS) — Signals functions that likely require refactoring or special testing strategies.

Normalization & Reporting

To compare testability metrics across functions, files, and projects, values should be normalized and aggregated systematically.

1. Normalization

  • Normalize each metric to a 0–1 scale per project using:
    • Min–Max normalization:
      [ m_{norm} = \frac{m - m_{min}}{m_{max} - m_{min}} ]
    • Or z-score normalization for large datasets: [ m_{z} = \frac{m - \mu_m}{\sigma_m} ]
  • Ensure consistent “directionality”:
    • For positive metrics (C, O, ER) → higher is better.
    • For negative metrics (BC, AC, EC, EL, SEI) → invert before aggregation.

2. Aggregation Levels

Level Description Aggregation
Function-level Base metrics per function. Raw values.
File-level Summarizes testability per file. Mean, median, stddev, min, max.
Project-level Summarizes entire project. Weighted mean across files (by LOC or function count).

3. Composite Score (CTS)

After normalization: [ CTS = α·Ĉ + β·Ō − γ·BĈ − δ·ÂĈ − ε·ÊĈ − ζ·ĖĻ − η·ŜĖĨ + κ·ER ] Weights can be equal (default) or empirically tuned.

4. Reporting

Each project’s report should include:

  • Statistical summary: mean, median, stddev for each metric.
  • Histograms: distribution of BC, AC, EC, CTS.
  • Scatter plots:
    • C vs. O → controllability–observability relationship.
    • BC vs. CTS → effect of complexity on testability.
  • Top/bottom N functions: rank by CTS to identify refactoring candidates.
  • Cross-project comparison: correlate metrics with project metadata (size, domain, test coverage).

5. Interpretation

  • High variability in C or O → inconsistent coding/testability practices.
  • High AC/EC + low O → async/event-heavy code is harder to test.
  • High EL + SEI → strong coupling or hidden side effects.
  • Stable high CTS across files → project with modular, observable, and controllable design.

Goal:
Normalization and reporting turn raw metric values into actionable insights that explain what factors make JavaScript code testable in practice — enabling both quantitative comparison across projects and qualitative interpretation for developers.

Methodology: Measuring JavaScript Testability in the Wild

This is a structured, end-to-end plan for collecting projects, computing JS testability metrics, aggregating results, and reporting study findings.
Each step lists inputs → actions → outputs and the best tools to use.

0) Study Setup & Configuration

Goal: Make runs reproducible and configurable.

  • Inputs: testability.config.json (paths, excludes, thresholds, weights, side-effect/event API lists)
  • Actions: Load config, validate schema, set defaults.
  • Outputs: In-memory Config object used by all steps.
  • Tools: zod or jsonschema for config validation.

1) Project Selection & Acquisition

Goal: Build a diverse dataset of JS repositories.

Could reuse/start with TestPilot benchmarks.

  • Inputs: Candidate repo list (GitHub URLs), inclusion criteria.
  • Actions: Clone/shallow-clone or download archives; record metadata (stars, domain, LOC).
  • Outputs: projects/ directory; dataset.csv with project metadata.
  • Tools: simple-git, nodegit, cloc (or sloc), GitHub REST API (optional).

2) Source Discovery & Filtering

Goal: Find analyzable JS files and exclude noise.

  • Inputs: Project root, config excludes.
  • Actions: Glob files; filter node_modules, dist, coverage, generated code.
  • Outputs: List of JS files to parse.
  • Tools: fast-glob or globby.

3) Parsing & AST Construction

Goal: Parse JS to ESTree/Babel AST.

  • Inputs: JS files, parser options.
  • Actions: Parse with tolerant options; support ESM/CJS, JSX if present.
  • Outputs: AST per file.
  • Tools: @babel/parser (sourceType: 'unambiguous', plugins: jsx, classProperties, classPrivateProperties, optionalChaining, dynamicImport).

4) AST Traversal & Scope Resolution

Goal: Walk ASTs, know what’s local vs. closure/global.

  • Inputs: ASTs.
  • Actions: Traverse; compute lexical scope; resolve identifiers (local/outer/global).
  • Outputs: Visitor hooks and scope helpers for metric passes.
  • Tools: @babel/traverse, @babel/types; scope via path.scope.hasBinding(name); optional eslint-scope for explicit scope graphs.

5) Function & Entry Artifact Discovery

Goal: Enumerate all functions/methods and entry points.

  • Inputs: ASTs.
  • Actions: Find FunctionDeclaration, function expressions, arrow functions, class methods; detect exports; tag likely entry points (routes, CLI, handlers).
  • Outputs: functions.jsonl (one row/object per function) with stable IDs: file:lineStart:functionName.
  • Tools: Babel traverse; heuristics for exports (export, module.exports), Express routes, EventEmitter calls.

6) Lightweight Call Graph (Intra-/Inter-file)

Goal: Enable reachability (ER) and slice context.

  • Inputs: Function nodes, call expressions, import/export graph.
  • Actions: Map callers → callees; resolve same-file symbols; basic inter-file via import names.
  • Outputs: call-graph.json (adjacency list).
  • Tools: Babel traverse; module graph via resolve, enhanced-resolve, or madge (optional).

7) Metric Passes (Per Function)

Compute raw counts first; normalize later. Keep counts and a score per metric if needed.

7.1 Controllability (C)

  • Actions: Count params/defaults, exported flag; penalize closure/global refs, new on imported classes.
  • Outputs: { params, defaults, globals, closures, newables, score }
  • Tools: Babel traverse; scope checks; import table.

7.2 Observability (O)

  • Actions: Detect returned values; console.*; DOM writes; Node I/O/network; hidden state writes.
  • Outputs: { hasReturn, logs, ioWrites, hiddenWrites, score }
  • Tools: API allowlists (config); pattern match known sinks.

7.3 Branching Complexity (BC)

  • Actions: Count if/switch/ternary/try/loops; track nestingDepth.
  • Outputs: { branches, nestingDepth, score }
  • Tools: Traverse; maintain depth counter stack.

7.4 Asynchronous Complexity (AC)

  • Actions: Count await; .then/.catch/.finally; Promise.all/race/any; timers; estimate chainDepth.
  • Outputs: { awaits, thens, timers, chainDepth, score }
  • Tools: Traverse; simple chain analysis.

7.5 Event-Driven Complexity (EC)

  • Actions: Count DOM listeners; EventEmitter .on/.emit; Express routes; React/Vue handler heuristics.
  • Outputs: { domListeners, emitters, routerHandlers, reactHandlers, score }
  • Tools: Allowlist of event APIs; framework signatures.

7.6 Encapsulation Level (EL)

  • Actions: Count closure reads/writes, module singleton access, private/prototype heuristics.
  • Outputs: { closureReads, closureWrites, moduleSingleton, privateHeuristics, score }
  • Tools: Scope checks; class/private fields detection.

7.7 Side-Effect Index (SEI)

  • Actions: Count calls to effectful APIs; compute LOCf; ratio.
  • Outputs: { sideEffectCalls, loc, sei }
  • Tools: Allowlist for side effects; LOC via ranges.

7.8 Entry-point Reachability (ER)

  • Actions: Mark reachable from roots (exports, routes, CLI) via call graph traversal.
  • Outputs: 0/1
  • Tools: BFS/DFS over call-graph.json.

8) Normalization & Composite Score

Goal: Make metrics comparable and combine into CTS.

  • Inputs: Raw per-function metrics across a project.
  • Actions: Min–max normalize per metric to [0,1] (invert “negative” metrics: BC/AC/EC/EL/SEI); compute CTS with weights.
  • Outputs: Updated per-function records with normalized fields and CTS.
  • Tools: In-repo JS utility; (optional) ml-matrix or simple math.

9) Aggregation (File → Project)

Goal: Summaries for analysis and reporting.

  • Inputs: Per-function results.
  • Actions:
    • File level: mean/median/stddev min/max; % low-testability; list worst functions.
    • Project level: same stats over all functions; predefined histogram buckets.
  • Outputs: files.json, project-summary.json (with histograms & top/bottom lists).
  • Tools: In-repo JS utilities; optional duckdb for larger datasets.

10) (Optional) Dynamic Signals

Goal: Correlate static metrics with runtime testing signals.

  • Inputs: Test runs & coverage reports.
  • Actions: Run test suite and collect coverage; map covered lines to functions; (optional) mutation testing scores.
  • Outputs: coverage-map.json, augmented summaries.
  • Tools: nyc/Istanbul, vitest --coverage, jest --coverage, stryker (optional).

11) Reporting

Goal: Produce human-readable project reports and dataset exports.

  • Inputs: project-summary.json, files.json, config.
  • Actions: Generate Markdown report; export CSVs; (optional) HTML with charts.
  • Outputs: reports/<project>-summary.md, exports/*.csv.
  • Tools: Markdown template with mustache/handlebars; (optional) vega-lite for charts.

12) QA: Tests & Fixtures

Goal: Keep the pipeline stable.

  • Inputs: Synthetic JS fixtures covering patterns (pure functions, closures, async chains, events, heavy side effects).
  • Actions: Unit tests for each metric pass; snapshot tests for file/project aggregation.
  • Outputs: npm test green; snapshots under __snapshots__/.
  • Tools: vitest or jest.

13) CLI & Automation

Goal: One command, full pipeline.

  • Inputs: CLI args (--path, --out, --threshold, --buckets, --exclude).
  • Actions: Run steps 2→11; write outputs.
  • Outputs: output/ + reports/ directories per project.
  • Tools: commander or yargs; GitHub Actions workflow for CI runs.

14) Study Analysis & Synthesis

Goal: Turn numbers into findings.

  • Inputs: All project summaries + metadata.
  • Actions: Compute cross-project statistics; correlations; comparative plots; identify patterns & anti-patterns.
  • Outputs: Paper figures/tables; “Implications for practice” and “Threats to validity”.
  • Tools:
    • JS: arquero, vega-lite
    • or Python: pandas, scipy, seaborn (optional offline analysis)

Artifacts & Schemas

  • docs/schemas/js-function.md — function record schema
  • docs/schemas/file-summary.md — file summary schema (freeze)
  • docs/schemas/project-summary.md — project summary schema (freeze)
  • docs/metrics.md — metric definitions & formulas
  • docs/task-plans/ — weekly plans

Minimal Outputs per Project (expected layout)

output/ functions.jsonl # 1 JSON per function (raw + normalized) files.json # file-level aggregates project-summary.json # project-level summary + histograms reports/ -summary.md # human-readable report exports/ functions.csv files.csv

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages