diff --git a/README.md b/README.md index 8322cea..6ecbc9e 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,25 @@ Then ask Claude: *"Run a conformance check against the o2c-simple reference mode Full walkthrough: **[QUICKSTART.md](QUICKSTART.md)** · Five-question demo: **[scripts/demo-walkthrough.md](scripts/demo-walkthrough.md)** · Pattern discovery: **[pattern-discovery/README.md](pattern-discovery/README.md)** +### Full portfolio dashboard — 6 analysis lenses + +Beyond the synthetic demo, the **[portfolio dashboard](demo/portfolio/index.html)** +(live at `/portfolio/`) presents six forensic lenses over **real** datasets. It was +consolidated here from the now-archived `transaction-forensics` repo so everything +lives in one place. + +| Lens | Data source | Scale | Key finding | +|---|---|---|---| +| **Overview** | Architecture + thesis | — | Structured-vs-unstructured gap analysis | +| **CRM Pipeline** | Kaggle CRM Sales Opportunities | 8,800 opportunities | Win rates, velocity, quarter-end compression | +| **BPI Challenge** | BPI Challenge 2019 (4TU.ResearchData) | 251,734 POs · 1.6M events | 57K payment blocks, resource concentration | +| **IDES Compliance** | SAP IDES demo system | 3,132 cases (O2C + P2P) | **7 compliance violations** in SAP's own reference data | +| **Client Cases** | 3 anonymized engagements | 3M+ ERP records | $103K savings, credit-hold overrides, SOD violations | +| **NLP Patterns** | Salesforce/HERB (HuggingFace) | 37,064 documents | 11 communication clusters, approval bottlenecks | + +The NLP pipeline behind the last tab lives in **[`herb-nlp/`](herb-nlp/)** +(TF-IDF + KMeans, optional BERTopic, network + temporal analysis). + ## Proven in the Field Three real consulting engagements motivated this tool. Anonymized, but the figures diff --git a/demo/index.html b/demo/index.html index f4528c1..c43d581 100644 --- a/demo/index.html +++ b/demo/index.html @@ -265,7 +265,8 @@

SAP Transaction Forensics

diff --git a/demo/portfolio/index.html b/demo/portfolio/index.html new file mode 100644 index 0000000..30ff13a --- /dev/null +++ b/demo/portfolio/index.html @@ -0,0 +1,2606 @@ + + + + + +Transaction Forensics — Enterprise Process Analysis + + + + + + +
+
+
+
TF
+
+
Transaction Forensics
+
ERP Access, Inc. — Christopher Bailey — Built with Claude Code (AI pair-programming)
+
+
+
+ + + + + + +
+
+ +
+
1,639
Tests Passing
+
19
Extraction Paths
+
11
MCP Tools
+
+ + + + + + + + + + +
+
+
+ + +
+
+ AI Disclosure: + Architecture and analysis methodology by Christopher Bailey. Implementation pair-programmed with Claude Code (Anthropic). All commits co-authored — see git history for full attribution. +
+
+ + +
+
+ + +
+
Structured data tells you what happened.
Unstructured text tells you why.
+
+ Transaction Forensics is a cross-system process forensics platform that combines ERP transaction logs with the unstructured text that surrounds them — emails, Slack messages, progress reports, call transcripts, and order notes — to surface the discrepancies between what organizations report and what actually happened. Built by ERP Access, Inc. +
+
+ + +
+
The Core Insight
+
+
+ Every enterprise system generates two kinds of data. Structured transactions — timestamps, amounts, stage changes, user IDs — tell you the official story. Unstructured text — the emails, Slack threads, meeting notes, timesheets, SOWs, and progress reports that surround those transactions — tell you what actually happened. The gap between them is where fraud, waste, and dysfunction hide. +
+ + +
+ + +
+
Structured Data Says
+ +
+
+
CRM OPPORTUNITY
+
"Deal in Negotiation for 6 months"
+
+
+
SAP P2P
+
"Purchase Order created 03/15"
+
+
+
TIMESHEET
+
"40 hours billed to Project Alpha"
+
+
+
PROJECT STATUS
+
"Phase 2: On Track, Green"
+
+
+
+ + +
vs
+ + +
+
Unstructured Text Reveals
+ +
+
+
SLACK THREAD
+
"Customer said not ready — but Sales moved it forward anyway. No documented sign-off."
+
+
+
EMAIL CHAIN
+
"Requisition wasn't approved yet. Create the PO now, we'll get the paperwork later."
+
+
+
PROGRESS REPORT
+
"Assigned to Project Alpha but worked on Beta all week. SOW deliverables not started."
+
+
+
MEETING TRANSCRIPT
+
"We're 3 weeks behind. Tell the client we're on track while we figure this out."
+
+
+
+ +
+ + +
Evidence From Our Analysis
+
+ +
+
SAP IDES: Retroactive Documentation
+
Structured data shows PO and PR both exist. Timestamps reveal the PO was created before the PR — approval was documented after the fact. Only detectable by cross-referencing temporal sequence.
+
See IDES Compliance tab →
+
+ +
+
HERB: Approvals in Slack, Not Systems
+
37,064 enterprise documents analyzed (32.8K Slack messages, 3.6K pull requests, 400 docs, 321 transcripts). 1,226 "LGTM/Approved" messages found in Slack channels — informal approvals with no audit trail. Structured approval workflows show no record of these decisions.
+
See NLP Patterns tab →
+
+ +
+
BPI: 57K Payment Blocks — Why?
+
Structured data shows 22.7% of POs hit payment blocks. The event log can't explain why — that answer lives in vendor correspondence, invoice discrepancy notes, and buyer emails that aren't in the event log.
+
See BPI Challenge tab →
+
+ +
+
+ + +
+
+ 20 years of ERP consulting taught me this: In every engagement where something went wrong — billing disputes, project failures, compliance gaps — the structured transaction data looked clean. The truth was always in the unstructured layer: the email where someone said "skip the approval," the timesheet that didn't match the progress report, the SOW deliverable that was marked complete but never started. This tool automates finding those discrepancies at scale. +
+
+
+ + +
+
System Architecture
+
+ + +
+ + +
+
Data Sources
+
+
SAP ERP (IDES/ECC)
+
Salesforce CRM
+
BPI Challenge (XES)
+
Slack / HERB Comms
+
NetSuite ERP
+
CSV / Custom
+
+
+ + +
+
→
+
+ + +
+
7 Adapters
+
+
+ IDataAdapter
interface
+ Normalize to
unified event log +
+
+
BPI, CSV, ECC,
S4, SALT, SFDC,
Synthetic
+
+ + +
+
→
+
+ + +
+
Analysis Engines
+
+
+
Conformance Checker
+
Token-based replay
van der Aalst algorithms
Process model builder
+
+
+
Temporal Analyzer
+
Throughput times
Bottleneck detection
Delay probability
+
+
+
Pattern Clustering
+
TF-IDF + K-Means
Effect sizing (Cohen's d)
Stability bootstrap
+
+
+
Cross-System Resolver
+
Entity matching
Levenshtein + proximity
Gap detection
+
+
+
+ + +
+
→
+
+ + +
+
Forensic Output
+
+
Compliance
Violations
+
Bottleneck
Reports
+
Pattern
Cards
+
Evidence
Ledger
+
+
+ +
+ + +
+
TypeScript (MCP Server)
+
Python 3.11 (Pattern Engine)
+
1,639 tests across 70 suites
+
Deterministic (seed=42)
+
Zero frontend dependencies
+
+
+
+ + +
+
Four Forensic Lenses
+
+ +
+
8,800
+
CRM Pipeline Forensics
+
Sales opportunity analysis — win rates, velocity patterns, agent performance, quarter-end compression. Kaggle real-world CRM data.
+
Explore →
+
+ +
+
251K
+
BPI Challenge 2019
+
Real purchase-to-pay from a multinational. 1.6M events, payment blocks, process variability, resource concentration risks.
+
Explore →
+
+ +
+
7
+
SAP IDES Compliance
+
Compliance violations in SAP's own demo system. Maverick buying, retroactive documentation, segregation of duties risks.
+
Explore →
+
+ +
+
37K
+
NLP Pattern Analysis
+
Salesforce HERB — 37K documents (Slack, PRs, transcripts) clustered into communication patterns. Network graphs, bridge users, team dynamics.
+
Explore →
+
+ +
+
+ + +
+
+
Real-World Client Data
+
3 anonymized engagements — 3M+ ERP records, $103K savings, ITGC violations, 28.6% RMA rate
+
License audit + ticket forensics + high-growth hardware company ERP forensics with credit hold overrides and SOD violations
+
+
View Cases →
+
+ + +
+
Design Principles
+
+ +
+
Adapter Pattern
+
Every data source implements IDataAdapter — normalize once, analyze everywhere. Adding a new ERP means writing one adapter, not rewriting analysis logic. Currently: SAP, Salesforce, NetSuite, BPI (XES/OCEL), CSV, synthetic.
+
+ +
+
Deterministic Reproducibility
+
All analysis uses seed=42. Every pattern card, every cluster, every statistical test can be reproduced exactly. Run make demo and get identical output. No non-determinism in the forensic chain.
+
+ +
+
Evidence-Based Findings
+
Every claim links to an evidence ledger entry with source files, row counts, timestamps, and reproducibility parameters. Effect sizes use Cohen's d with 95% CI. Weak results are labeled as exploratory — no overclaiming.
+
+ +
+
+ + +
+
Evidence Platform Capabilities
+
+ +
+
12
+
Contradiction Types
+
AMOUNT_DIVERGENCE, TEMPORAL_IMPOSSIBILITY, SOD_VIOLATION, STATUS_CONTRADICTION, APPROVAL_BYPASS, DUPLICATE_PAYMENT, PHANTOM_VENDOR, RETROACTIVE_CHANGE, MISSING_RECEIPT, THRESHOLD_SPLIT, POLICY_OVERRIDE, GHOST_EMPLOYEE
+
+ +
+
19
+
Extraction Paths
+
Deterministic extraction across SAP O2C, FI/CO, P2P + Salesforce + NetSuite pipelines. Schema validator with 19-table IDES reference (438 fields). Reality-gap detector: 3-way comparison of reference vs documented vs actual.
+
+ +
+
8
+
Finding Lifecycle States
+
DETECTED → CONFIRMED → ASSIGNED → INVESTIGATING → ESCALATED → REMEDIATED → VERIFIED → RESOLVED. Field-level provenance with SHA-256 replay verification. Reviewer handoff packets as self-contained audit artifacts.
+
+ +
+ +
+ +
+
7 Evidence Systems
+
+ Contradiction Engine (12 types)
+ Schema Validator (438 IDES fields)
+ Reality-Gap Detector (3-way diff)
+ Finding Lifecycle (8-state machine)
+ Reviewer Handoff Packets
+ Field-Level Provenance (SHA-256)
+ MCP Tool Integration (11 tools) +
+
+ +
+
Platform Stats
+
+ 1,639 tests across 70 test suites
+ 7 data adapters + 7 evidence systems
+ 19 deterministic extraction paths
+ 12-type contradiction engine
+ 19-table IDES schema reference
+ 11 MCP tools for agent integration
+ TypeScript + Python 3.11 +
+
+ +
+
+ + +
+
How AI Was Used — Honest Accounting
+
+
+
+
What I Did (Christopher Bailey)
+
+ • Defined the problem space and research questions
+ • Selected data sources and licensed datasets
+ • Designed the adapter architecture and analysis pipeline
+ • Chose conformance algorithms (van der Aalst token replay)
+ • Interpreted findings and wrote forensic narratives
+ • Determined what's a real finding vs. a statistical artifact
+ • Real-world client engagement and domain expertise (20 yrs ERP) +
+
+
+
What Claude Code Did (AI Pair-Programmer)
+
+ • Implemented data adapters and parsers (TypeScript)
+ • Built pattern engine and clustering pipeline (Python)
+ • Wrote conformance checking engine
+ • Generated test suites (1,639 tests across 70 suites)
+ • Built this dashboard (vanilla HTML/CSS/JS)
+ • Statistical computations (effect sizing, CI, p-values)
+ • All code visible in git history with co-author tags +
+
+
+
+ The honest version: Claude Code is a force multiplier. The 1,639-test, 7-adapter, 7-evidence-system platform you see here was built in weeks, not months. But the AI doesn't know what's worth finding — it doesn't know that a PO-before-PR is a Sarbanes-Oxley risk, or that 22.7% payment block rates are 4x industry norms. Domain expertise decides what to look for; AI makes looking fast. +
+
+
+ + +
+
+
Data Sources
+
+ Kaggle CRM Sales Opportunities — Apache 2.0
+ BPI Challenge 2019 — 4TU.ResearchData, CC BY 4.0
+ SAP IDES — sap-extractor, MIT License
+ Salesforce HERB — HuggingFace, CC-BY-NC-4.0
+ Client data — anonymized, used with permission +
+
+
+
Reproduce This
+
+ git clone github.com/chrbailey/SAP-Transaction-Forensics
+ make demo      # one-command bootstrap
+ make test      # 1,639 tests
+ make demo-kaggle # real Kaggle CRM data
+ View on GitHub → +
+
+
+ +
+ + + +
+ + +
+ +
+ + +
+
Pipeline Overview
+
+
+
63.2%
+
Win Rate
+
4,238 Won / 6,711 Closed
+
+
+
$2,361
+
Avg Deal Size
+
Across all closed-won deals
+
+
+
57 days
+
Median Velocity
+
Time from open to close
+
+
+
2,089
+
Open Pipeline
+
Opportunities in progress
+
+
+
+ + +
+
Conformance Analysis
+
+
New Business Pipeline Model
+
Expected stage sequence for opportunity progression
+
+
+
Prospecting
+ → +
+
+
Qualification
+ → +
+
+
Needs Analysis
+ → +
+
+
Value Proposition
+ → +
+
+
Id. Decision Makers
+ → +
+
+
Perception Analysis
+ → +
+
+
Proposal / Price Quote
+ → +
+
+
Negotiation / Review
+ → +
+
+
Closed Won
+
+
+
+
+
+
0.05
+
Avg Fitness Score
+
Average fitness 0.05 — expected when real-world CRM data is measured against an aspirational 8-stage model. Most organizations skip stages; low conformance is typical, not alarming.
+
+
+
53,862
+
Deviations Detected
+
Stage skips, reversals, and out-of-sequence transitions detected across 8,300 opportunities (closed + open) with stage history.
+
+
+
8,300
+
Cases Analyzed
+
All opportunities with stage history (closed + open) analyzed for sequence conformance against the defined process model.
+
+
+
+
+ + +
+ +
+
Quarter-End Compression
+
Monthly close distribution — QE months highlighted in orange
+
+
+
Jan
+
—
+
0
+
+
+
Feb
+
—
+
0
+
+
+
Mar
+
647
+
647
+
+
+
Apr
+
586
+
586
+
+
+
May
+
805
+
805
+
+
+
Jun
+
641
+
641
+
+
+
Jul
+
627
+
627
+
+
+
Aug
+
785
+
785
+
+
+
Sep
+
635
+
635
+
+
+
Oct
+
566
+
566
+
+
+
Nov
+
768
+
768
+
+
+
Dec
+
651
+
651
+
+
+
+ 38.4% close in quarter-end months (Mar/Jun/Sep/Dec) — 1.15× the expected baseline of 33.3% (4 of 12 months). Slight concentration but within normal range for B2B sales cycles. +
+
+ +
+
Deal Velocity Distribution
+
Time-to-close buckets — closed won opportunities only
+
+
+
0–30 days
+
1,817
+
42.9%
+
+
+
31–60 days
+
377
+
8.9%
+
+
+
61–90 days
+
1,078
+
25.4%
+
+
+
91–180 days
+
966
+
22.8%
+
+
+
+
Velocity Insights
+
+
+
57
+
Median Days
+
+
+
4,238
+
Deals Measured
+
+
+
+ The bimodal distribution — 43% closing in under 30 days, 23% taking 91–180 days — suggests two distinct deal types: quick transactional sales and extended enterprise negotiations. +
+
+
+ +
+ + +
+ +
+
Top Agents by Win Rate
+
Minimum 50 closed opportunities — top 10 performers
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AgentWin RateWonClosedRevenue
1Hayden Neloms70.4%107152$272K
2Maureen Marcano70.0%149213$350K
3Wilburn Farren69.6%5579$158K
4Cecily Lampkin66.9%107160$230K
5Versie Hillebrand66.7%176264$188K
6Moses Frase66.2%129195$207K
7Boris Faz66.0%101153$262K
8James Ascencio65.5%135206$414K
9Corliss Cosme65.5%150229$421K
10Reed Clapper65.4%155237$438K
+
+ +
+
Product Revenue Mix
+
Closed-won revenue by product — total $10,005K
+
+
+
GTXPro
+
$3,510K
+
35.1%
+
+
+
GTX Plus Pro
+
$2,630K
+
26.3%
+
+
+
MG Advanced
+
$2,216K
+
22.2%
+
+
+
GTX Plus Basic
+
$705K
+
7.1%
+
+
+
GTX Basic
+
$499K
+
5.0%
+
+
+
GTK 500
+
$401K
+
4.0%
+
+
+
MG Special
+
+
0.4%
+
+
+
+
Concentration
+
Top 3 products (GTXPro, GTX Plus Pro, MG Advanced) account for 83.6% of total revenue. MG Special at $44K represents tail inventory with minimal impact.
+
+
+ +
+ + +
+
Account Concentration
+
+
+
Top Accounts by Revenue
+
Closed-won revenue across 85 accounts
+
+ + + + + +
+
+
+
+
Diversification Assessment
+
CR5 concentration ratio (top-5 revenue share)
+
+
+
12.1%
+
Top 5 Share
+
+
+
85
+
Total Accounts
+
+
+
+
+ Healthy distribution. Top 5 accounts represent only 12.1% of revenue — well below typical key-account concentration risk thresholds. No single customer dependency. +
+
+
+
+ + +
+
Key Findings
+
+ +
+
+
Quarter-End Concentration
+ Low +
+
38.4%
+
38.4% of closed deals close in quarter-end months (Mar/Jun/Sep/Dec), vs. a 33.3% uniform baseline (4 of 12 months). The 1.15× ratio suggests mild concentration — typical in B2B sales where fiscal quarters influence buyer and seller timing. Not a strong anomaly signal on its own.
+
+ +
+
+
Stage Conformance Gap (Expected)
+ Expected +
+
Avg fitness: 0.05
+
This is an expected result, not an anomaly. The 8-stage New Business pipeline (Prospecting → Qualification → Needs Analysis → ... → Negotiation/Review → Closed Won) is an aspirational reference model, not an operational requirement. Real-world CRM data typically records only 2-4 stages per deal. The 53,862 "missing_activity" deviations across 8,300 opportunities (closed + open) quantify the gap between prescribed process and actual practice — useful for process improvement, but not indicative of control failures. Average fitness score: 0.05.
+
+ +
+
+
Agent Performance Spread
+ Low +
+
15-pt spread
+
Win rates among active agents range from approximately 55% to 70.4%, a 15-point spread. Top performers (Hayden Neloms, Maureen Marcano) sustain 70%+ win rates across 150–200+ closed deals, suggesting reproducible behavioral patterns worth codifying as playbooks.
+
+ +
+
+
Account Diversification
+ Low +
+
12.1% top-5
+
The top 5 accounts (Kan-code, Konex, Condax, Cheers, Hottechi) represent only 12.1% of total closed-won revenue across 85 accounts. Revenue is broadly distributed, reducing customer concentration risk. No single account exceeds 3.4% of total revenue.
+
+ +
+
+ +
+ + +
+
+ +
+ + +
+
CSV Ingest
+
→
+
SFDC Normalize
+
→
+
Event Log Build
+
→
+
Conformance Check
+
→
+
Pattern Detection
+
→
+
Report
+
+ + +
+
Stage 1: CSV Ingest
+
+ Source: Kaggle CRM Sales Opportunities (innocentmfa)
+ Files: sales_pipeline.csv (8,800 rows), accounts.csv (85), products.csv (7), sales_teams.csv (35)
+ Converter: convert_kaggle_crm.py maps CSV fields to SFDC JSON schema +
+
+ +
+
Stage 2: SFDC Normalization
+
+ Adapter: SFDCSyntheticAdapter implements IDataAdapter (8 methods)
+ Field mapper: Opportunity.Id → VBELN, Account.Id → KUNNR, Amount → NETWR
+ Pipeline models: Prospecting → Qualification → Needs Analysis → ... → Closed Won
+ Stage mapping: "Engaging" → "Qualification", "Won" → "Closed Won", "Lost" → "Closed Lost" +
+
+ +
+
Stage 3: Event Log Construction
+
+ Records: 49,408 events from 8,800 opportunities
+ Stage transitions: 31,787 entries (with synthetic intermediate stages for Won deals)
+ Activities: 17,621 task records (product-related subjects)
+ Format: case_id, activity, timestamp, resource, attributes +
+
+ +
+
Stage 4: Conformance Checking
+
+ Algorithm: Token-based replay (van der Aalst, 2016)
+ Model: sfdc_new_business — 8-stage pipeline with mandatory transitions
+ Cases: 8,300 analyzed | Fitness range: 0.00 – 0.10 (real data lacks full stage coverage)
+ Deviations: 53,862 total (45,386 missing_activity, 8,476 skipped_activity) +
+
+ +
+
Stage 5: Pattern Detection
+
+ Quarter-end compression: 38.4% of closes in QE months (1.15x the 33.3% baseline)
+ Deal velocity bimodal: 42.9% close within 30 days, 22.8% take 91-180 days
+ Agent spread: 15-point win rate range (55%–70%) across 30 agents
+ Account concentration: Top 5 = 12.1% of revenue (healthy diversification) +
+
+ + +
+
Technology Stack
+
+ MCP Server: TypeScript (ESM, strict mode), 7 data adapters + 7 evidence systems
+ Pattern Engine: Python 3.11, scikit-learn, scipy
+ Conformance: Token-based replay, ProcessModelBuilder, van der Aalst algorithms
+ Cross-System: Entity resolver (Levenshtein + proximity), unified event log
+ Evidence: 12-type contradiction engine, SHA-256 provenance, 8-state finding lifecycle
+ Tests: 1,639 passing across 70 test suites
+ Data: Kaggle CRM Sales Opportunities (Apache 2.0 license)
+ Frontend: Vanilla HTML/CSS/JS (zero dependencies)
+ Deployment: Vercel (static)
+ Source: github.com/chrbailey/SAP-Transaction-Forensics +
+
+ +
+
+
+ + + +
+ + +
+ +
+ +
+
+ Cluster Quality Note: Global silhouette score is 0.028 (KMeans) / 0.09 (BERTopic), indicating weak cluster separation. Findings should be treated as exploratory signals, not confirmed patterns. See Pipeline Transparency for methodology details. HERB dataset timestamps extend to 2027 (synthetic); temporal patterns reflect relative timing. +
+
+ +
+ + + + +
+ +
+
+
⏳

Loading forensic analysis...

+
+
+ + +
+
+ + +
+
+ + +
+
+
SF
+
+
View Analysis Code on GitHub
+
This is an analysis tool, not a hosted service. Clone the repo to run the pattern engine on your own Slack exports, case comments, or CRM data with text fields and timestamps.
+
+ View on GitHub → +
+
+ + + +
+ + +
+
+ + +
+ Data Source: BPI Challenge 2019 — 4TU.ResearchData (CC BY 4.0). Real purchase-to-pay event log from a multinational coatings and paints company. 251,734 purchase order items, 1,595,923 events, 628 unique resources. Period: Jan 2018 — Jan 2019. +
+ + +
+
Process Overview
+
+
+
64.3
+
Median Throughput (days)
+
Average 72.3 days; max 25,670 days (stale POs)
+
+
+
32%
+
Top 2 Variants Coverage
+
Only 2 of hundreds of paths cover a third of cases
+
+
+
57,136
+
Payment Blocks
+
22.7% of all POs required manual block removal
+
+
+
2,835
+
Incomplete Cases
+
POs that never progressed past creation
+
+
+
+ + +
+
Most Common Process Paths
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#Process PathCases%
1Create PO → Vendor Invoice → Goods Receipt → Invoice Receipt → Clear Invoice50,28620.0%
2Create PO → Goods Receipt → Vendor Invoice → Invoice Receipt → Clear Invoice30,79812.2%
3Create PO → Goods Receipt (incomplete)9,4433.8%
4Create PO → Vendor Invoice → GR → IR → Remove Payment Block → Clear6,9312.8%
5Create PO (abandoned — never progressed)2,8351.1%
+
+
+ Key finding: Variants 1 and 2 differ only in whether the vendor invoice arrives before or after goods receipt — a classic "3-way match" ordering question in P2P. Variant 4 shows 2.8% of cases require payment block removal, indicating invoice discrepancies. +
+
+ + +
+
Anomalies Detected
+
+
+
Process Issues
+
+
+ Payment blocks requiring intervention57,136 +
+
+ Quantity changes after PO creation21,449 +
+
+ Price changes after PO creation12,423 +
+
+ Deleted / cancelled orders5,298 +
+
+ Abandoned POs (single event)2,835 +
+
+
+
+
Resource Concentration
+
Top 10 resources handle 63% of all 1.6M events, creating operational risk if key personnel are unavailable.
+
+
+ System +
+ 25.0% +
+
+ user_002 +
+ 10.4% +
+
+ user_001 +
+ 6.0% +
+
+ batch_001 +
+ 4.6% +
+
+
user_002 alone handles 166,353 events (10.4%). Single-person bottleneck risk.
+
+
+
+ + +
+
Document & Matching Types
+
+
+
Purchase Order Types
+
+
Standard PO152,562 (60.6%)
+
+
Framework Order62,543 (24.9%)
+
+
Consignment36,629 (14.6%)
+
+
+
+
+
Invoice Matching Strategy
+
+
3-way match, invoice after GR164,874 (65.5%)
+
+
3-way match, invoice before GR67,583 (26.9%)
+
+
2-way match19,277 (7.7%)
+
+
+
3-way matching compares PO, goods receipt, and invoice. The 26.9% "invoice before GR" cases are where vendors bill before physical delivery — common but higher fraud risk.
+
+
+
+ + +
+
Forensic Insight
+
+
+

This dataset comes from a real multinational company (anonymized for BPI Challenge 2019). The process mining community uses it as a benchmark for purchase-to-pay analysis. Our forensic engine processed the full 1.6M event log and surfaced several structural concerns:

+

Payment blocks (22.7%) — Nearly 1 in 4 purchase orders hit a payment block requiring human intervention. This signals systematic issues in invoice matching or vendor master data quality. In a healthy P2P process, payment block rates should be under 5%.

+

Process variability — With the top 2 variants covering only 32% of cases, the remaining 68% follow hundreds of different paths. This "spaghetti process" pattern makes it difficult to automate, audit, or optimize. Industry benchmarks target 80%+ coverage in the top 5 variants.

+

Resource concentration — user_002 processes 10.4% of all events. If this person is unavailable (vacation, resignation), the bottleneck could cascade across the entire P2P process. This is a classic single-point-of-failure that process mining can identify but traditional audits miss.

+
+
+
+ +
+ + + +
+ + +
+
+ + +
+ Compliance Alert: 7 process violations detected in SAP's own IDES demo system. IDES (Internet Demonstration and Evaluation System) is SAP's official reference environment used for training and certification. These violations exist in SAP's reference data — our engine found them through automated conformance checking. +
+ + +
+
Violations Detected
+
+
+
7
+
Total Violations
+
In SAP's own reference data
+
+
+
6
+
Missing Purchase Requisition
+
PO created without prior approval workflow
+
+
+
1
+
Retroactive Documentation
+
PO created before PR — approval was backdated
+
+
+
1
+
Segregation of Duties Risk
+
Same resource created both PR and PO
+
+
+
+ + +
+
Violation Breakdown
+
+ + +
+
+ HIGH + MISSING_PR — Purchase Order Without Purchase Requisition + 6 cases +
+
+
+ Six purchase orders were created directly without an associated Purchase Requisition. In a compliant Procure-to-Pay process, every PO must originate from an approved PR to ensure proper authorization and budget control. +
+
+
+
Root Cause
+
Maverick buying — users bypassing requisition approval workflow to create POs directly
+
+
+
Business Risk
+
Unauthorized spending, budget overruns, audit findings. Bypasses approval controls designed to prevent fraud.
+
+
+
Remediation
+
Enforce system control: block PO creation without linked PR. Add validation rule in SAP transaction ME21N.
+
+
+
+
+ + +
+
+ CRITICAL + PO_BEFORE_PR — Retroactive Approval Documentation + 1 case +
+
+
+ One Purchase Order was created before its associated Purchase Requisition. This means the order was placed first and the approval was documented after the fact — a clear compliance violation where the authorization workflow was bypassed and retroactively papered over. +
+
+
+
Root Cause
+
Approval workflow bypassed — order placed first, requisition created afterward to satisfy documentation requirements
+
+
+
Business Risk
+
Fraudulent documentation, Sarbanes-Oxley violations, audit failure. Represents intentional control circumvention.
+
+
+
Remediation
+
Implement sequential control: system must enforce PR approval timestamp < PO creation timestamp. Add automated alert.
+
+
+
+
+ + +
+
+ MEDIUM + SOD_VIOLATION — Segregation of Duties Risk + 1 case +
+
+
+ One case where the same resource created both the Purchase Requisition (request) and the Purchase Order (fulfillment). Proper segregation of duties requires different individuals to request and approve purchases to prevent self-dealing. +
+
+
+
Root Cause
+
Missing role separation — user has authorization for both ME51N (create PR) and ME21N (create PO)
+
+
+
Business Risk
+
Self-dealing, fictitious vendor schemes. One person can request, approve, and fulfill without oversight.
+
+
+
Remediation
+
Review SAP role assignments (PFCG). Ensure PR creator and PO creator roles are mutually exclusive.
+
+
+
+
+ +
+
+ + +
+
IDES Process Comparison — O2C vs P2P
+
+
+
Order-to-Cash (O2C)
+
Sales side — from customer order to payment
+
+
Cases analyzed646
+
Total events5,708
+
Unique activities8
+
Process variants158
+
Median duration2.7 days
+
Max duration6,578 days
+
Normal completion rate83.6%
+
Invoice cancellation delay17.7 days avg
+
+
+ Key finding: 158 unique process variants for just 8 activities and 646 cases — extreme variability. The max duration of 6,578 days (18 years) indicates stale/orphaned orders in the demo system that were never closed. +
+
+
+
Procure-to-Pay (P2P)
+
Purchasing side — from requisition to vendor payment
+
+
Cases analyzed2,486
+
Total events7,420
+
Unique activities20
+
Process variants142
+
Average duration45.2 days
+
Max duration1,027 days
+
Compliance violations7
+
Batch processing outlier2,181 events
+
+
+ Key finding: 7 compliance violations in SAP's own demo data. The "PO before PR" case is particularly notable — it represents retroactive documentation, a pattern that in production systems is a red flag for fraud investigators. +
+
+
+
+ + +
+
Why This Matters
+
+
+

SAP IDES is not production data — it's SAP's official demo and training environment. Thousands of consultants learn SAP using this system. Yet our automated conformance checker found 7 compliance violations that exist in the reference data itself.

+

This demonstrates two things: (1) Automated process mining catches what manual review misses, even in well-known systems. (2) If reference data contains these patterns, production systems — with real users under real deadline pressure — almost certainly contain more.

+

The conformance checking engine uses token-based replay (van der Aalst algorithm) to compare actual event sequences against expected process models. For P2P, the expected model requires: PR → PO → Goods Receipt → Invoice → Payment. Any deviation is flagged, measured, and classified by severity.

+

The O2C analysis reveals a different problem: 158 process variants from just 8 activities. This is a "spaghetti process" — technically functional but impossible to audit or optimize at scale. Combined with the 6,578-day max duration (stale orders from the 1990s still open), it paints a picture of a system that works but accumulates technical debt in its process layer.

+
+
+
+ + +
+
Analysis Methodology
+
+
+ Conformance Engine: Token-based replay, ProcessModelBuilder, van der Aalst algorithms
+ Data Adapters: BPI (XES/OCEL), SAP IDES (sap-extractor, MIT), Synthetic (seed=42)
+ Pattern Engine: Python 3.11, scikit-learn, scipy (TF-IDF + K-Means + effect sizing)
+ Temporal Analysis: Throughput time, bottleneck detection, delay probability
+ Tests: 1,639 passing across 70 test suites
+ Source: github.com/chrbailey/SAP-Transaction-Forensics +
+
+
+ +
+ + + +
+ + +
+
+ + +
+ Client Data Notice: All data on this page comes from real consulting engagements. Company names, individual names, and email addresses have been anonymized. Financial figures, ticket counts, and category distributions are actual. Used with permission for educational purposes. +
+ + +
+
Case 1 — Healthcare Company: NetSuite License Optimization
+
+ +
+
+
+
Engagement: ERP User License Audit
+
289-user NetSuite environment — automated classification found $103,896 in annual savings
+
+
+
14.4x
+
ROI
+
+
+
+ +
+
+
289
+
Total Users
+
+
+
69
+
Eliminable
+
+
+
$103,896
+
Annual Savings
+
+
+
0.8 mo
+
Payback Period
+
+
+ +
+
+
Savings by Category
+
+
Dormant full-access (8 users, no login 90+ days)$46,464
+
Departed employee center (est. 53)$31,800
+
Approval-only users (4, replace w/ SuiteFlow)$23,232
+
Deprecated integrations (est. 4 of 8)$2,400
+
+
+
+
What Structured Data Shows vs. What We Found
+
+

Structured: NetSuite user list shows 289 active users with assigned roles. Looks clean.

+

Unstructured signals: Login timestamps reveal 8 full-access users ($5,808/yr each) haven't logged in for 90+ days. Cross-referencing with HR termination dates shows ~53 Employee Center users are departed employees still consuming licenses. 4 users' entire activity consists of clicking "Approve" on purchase orders — replaceable by an email-based workflow that costs nothing.

+

The gap: $103,896/year in waste invisible to anyone looking at the user list alone.

+
+
+
+
+
+ + +
+
Case 2 — MedTech Manufacturer: Help Desk Ticket Forensics During Acquisition
+
+ +
+
Engagement: NetSuite Implementation + Post-Acquisition Support
+
2,525 help desk tickets reveal organizational stress invisible in ERP transaction data
+
Diagnostics manufacturer acquired by Fortune 500. Structured data showed normal operations. Tickets told a different story.
+
+ + +
+
+
2,525
+
Help Tickets
+
+
+
11
+
Categories
+
+
+
38%
+
Uncategorized
+
+
+
3,992
+
ERP Users
+
+
+
1,423
+
Inventory Items
+
+
+ +
+ + +
Ticket Category Distribution
+
+ +
+
+ Uncategorized +
+ 956 +
+
+ Finance +
+ 469 +
+
+ Access +
+ 257 +
+
+ Procurement +
+ 215 +
+
+ Inventory +
+ 119 +
+
+ +
+
+ Manufacturing +
+ 107 +
+
+ Warehouse +
+ 103 +
+
+ Cost Accounting +
+ 84 +
+
+ Quality +
+ 77 +
+
+ Order Mgmt +
+ 66 +
+
+ +
+ + +
What Ticket Text Reveals — Unstructured Signals from Real Tickets
+
+ +
+
DATA INTEGRITY
+
"How did 20413 turn into 20433?"
+
Inventory team can't explain item number mutation. Structured data shows both items exist. The ticket reveals someone doesn't trust the data — and they're right to question it.
+
+ +
+
SYSTEM WORKAROUNDS
+
"Explore creating dummy transactions for MRP"
+
Manufacturing is building fake transactions to work around MRP limitations. Structured data will show these as real — auditors would never know.
+
+ +
+
ESCALATION CULTURE
+
"Bill Payment Email Notification for Vendors — URGENT"
+
Multiple "URGENT" tickets for routine vendor payments. Finance team is under pressure. Transaction data shows payments made on time — the stress is invisible.
+
+ +
+
ACQUISITION CHAOS
+
257 "Request for NetSuite Access" tickets
+
10% of all tickets are access requests — many from acquiring company email domains. IT is drowning in onboarding during the acquisition. ERP data shows users; tickets show the churn.
+
+ +
+ + +
+
+
ERP Transaction Data
+
+ • 3,992 employees in system
+ • 1,044 active customers
+ • 1,423 inventory items tracked
+ • 307 bills of materials
+ • 465 GL accounts
+ • 5,035 warehouse bin locations
+ Status: Operational +
+
+
vs
+
+
Ticket Text Analysis
+
+ • 38% of tickets uncategorized (overwhelmed)
+ • Dummy transactions created as workarounds
+ • Item numbers mutating unexplainably
+ • "URGENT" escalation culture in Finance
+ • Acquiring company flooding access requests
+ • Lot traceability questions (FDA compliance)
+ Status: Organization under stress +
+
+
+ + +
+ Available data for forensic analysis: 2,525 help desk tickets (with summary, assignee, priority, category, timestamps, response/close times) + complete NetSuite master data (employees, customers, vendors, items, BOM, chart of accounts, inventory, financial statements). The combination of structured ERP data with unstructured ticket text is exactly the dual-layer forensic approach this tool is designed for. +
+
+
+
+ + +
+
Case 3 — Connected Hardware Manufacturer: High-Growth ERP Forensics
+
+ +
+
Engagement: ERP Migration Assessment + ITGC Audit + International Expansion (multi-year engagement)
+
3M+ ERP records forensically analyzed — credit hold overrides, 28.6% return rate, SOD violations, approval chain complexity
+
High-growth hardware manufacturer during rapid scaling. Legacy ERP → enterprise ERP migration. Structured transaction data + ITGC audit findings + process documentation.
+
+ + +
+
+
3M+
+
CSV Rows
+
+
+
102K
+
Sales Orders
+
+
+
97K
+
RMA Returns
+
+
+
43K
+
Vendors
+
+
+
10K
+
Customers
+
+
+
28.6%
+
RMA Rate
+
+
+ +
+ + +
Data Sources Analyzed
+
+
+ Master Data
+ 10K customers, 43K vendors
8.7K fixed assets, 5K contacts +
+
+ Transaction Data
+ 102K sales orders, 1M+ EDI lines
97K RMAs, 164K credit memos +
+
+ Governance / Text
+ ITGC audit, SOD analysis
7.6K deductions, PES call notes +
+
+ + +
Forensic Findings — What the Data Revealed
+
+ +
+
ITGC VIOLATIONS (Deloitte Audit)
+
7 users with Administrator role. Terminated employee still active.
+
153 active users, 40 unique roles. SOD violations at both role and user level. 4 generic shared accounts. No formalized change management policy. Admin access to both dev and prod environments. No post-implementation review process. Critical gaps for a publicly traded company.
+
Source: Deloitte SOD Role Definition Analysis
+
+ +
+
CREDIT HOLD OVERRIDES
+
Sales orders shipped despite "Customer On Credit Hold" flag
+
Sales order headers contain both "Customer On Credit Hold" and "Shipment Hold Released by Finance" fields. Cross-referencing reveals orders where credit holds were manually overridden — Finance releasing shipments to customers already flagged for credit risk. The structured status says "shipped." The override field tells you it shouldn't have been.
+
Source: Sales Order Header — 102K records
+
+ +
+
RETURN RATE ANOMALY
+
28.62% of customer accounts had return events — only 67.5% on-time delivery
+
Forensic case analysis of 1,090 customer accounts: 312 had at least one RMA event (28.62%). 97K total RMA line items in the extract across 6 types: Open Box, Closed Box, Destroyed in Field, Stock Rotation, Warranty, Error Shipment. Transaction data shows the returns; memo fields and reason codes hint at systemic quality or logistics failures the structured data can't explain.
+
Source: Case Outcome Analysis (1,090 accounts, 21,099 events) + RMA Extract (97K line items)
+
+ +
+
APPROVAL CHAIN COMPLEXITY
+
7,610 customer deductions with multi-approver routing and rerouting
+
Marketing deductions (MDF) routed through "Next Approver" and "Set Rerouted Next Approver" chains. Multiple email notification flags. Deductions linked to DFI invoices, credit memos, and proof-of-performance documents. The approval chain is so complex that the rerouting field exists specifically because the normal chain fails regularly.
+
Source: Customer Deductions — 7,610 records, 50+ columns
+
+ +
+ + +
+
+
Structured ERP Data Says
+
+ • 102K sales orders processed
+ • 97K returns authorized
+ • 43K vendors in master data
+ • 153 active users, roles assigned
+ • Orders shipped, invoiced, cleared
+ • International entities operational
+ Status: ERP Functioning +
+
+
vs
+
+
Governance + Text Layer Reveals
+
+ • Credit holds overridden to ship anyway
+ • 28.6% returns — systemic product/logistics issue
+ • 7 users with admin (SOX risk, public co.)
+ • Terminated employee still accessing system
+ • Approval chains so broken a "reroute" field exists
+ • 144K PO changes — constant purchasing churn
+ Status: Controls Gap, SOX Exposure +
+
+
+ + +
+ Engagement scope: Multi-year consulting engagement spanning ERP vendor selection, enterprise ERP migration assessment, ITGC audit (Big Four SOD analysis), international tax restructuring (European entity), regional expansion (APAC), and PCI/SOX compliance programs. Data sources: legacy ERP production extracts, audit firm findings, change management logs, 24+ project status call notes, SOW/FRD documentation. All company names, customer IDs, employee names, and identifying details anonymized. +
+ +
+
+
+ + +
+
The Pattern Across All Three Cases
+
+
+ Case 1 (license audit): Structured user data hides $103K in waste — login timestamps and role assignments alone told the story. Case 2 (acquisition tickets): When structured data looks normal, 2,525 help desk tickets reveal an organization under stress — workarounds, data trust issues, and IT drowning in access requests. Case 3 (high-growth ERP): 3 million rows of clean-looking transaction data mask credit hold overrides, a 28.6% return rate, SOD violations in a rapidly scaling company, and approval chains so dysfunctional that a "reroute approver" field was built into the system. In every case, the structured data said "operational." The unstructured layer — tickets, audit findings, override fields, memo text — told the real story. +
+
+
+ +
+ + + +
+ + + + diff --git a/demo/portfolio/pattern_cards.json b/demo/portfolio/pattern_cards.json new file mode 100644 index 0000000..d941c0c --- /dev/null +++ b/demo/portfolio/pattern_cards.json @@ -0,0 +1,4234 @@ +{ + "metadata": { + "generated_at": "2026-03-26T09:21:01.140849", + "author": "Christopher Bailey", + "engine": "Transaction Forensics Pattern Engine v3.0", + "source": "Salesforce/HERB", + "source_url": "https://huggingface.co/datasets/Salesforce/HERB", + "license": "CC-BY-NC-4.0", + "products_analyzed": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "total_messages_scanned": 32781, + "total_documents_scanned": 400, + "total_transcripts_scanned": 321, + "total_prs_analyzed": 3562, + "total_customers": 120, + "total_team_members": 18 + }, + "pipeline": { + "version": "2.0", + "stages": [ + { + "name": "Ingest", + "duration_seconds": 0.188, + "documents_loaded": 37064, + "sources": { + "slack": 32781, + "transcript": 321, + "document": 400, + "pull_request": 3562 + }, + "relational_joins": "employee_map, customer_map loaded for cross-referencing" + }, + { + "name": "Normalize", + "description": "Text cleaning + 22 abbreviation expansions + URL/mention stripping" + }, + { + "name": "Vectorize", + "method": "TF-IDF", + "features": 5000, + "ngram_range": [ + 1, + 3 + ], + "max_features": 5000 + }, + { + "name": "Cluster", + "method": "KMeans", + "optimal_k": 15, + "silhouette_score": 0.0278, + "k_scores": { + "8": 0.021, + "9": 0.0213, + "10": 0.0219, + "11": 0.0235, + "12": 0.0239, + "13": 0.0227, + "14": 0.0235, + "15": 0.0278, + "16": 0.0268 + }, + "per_cluster_silhouette": { + "3": 0.1882, + "10": 0.0993, + "6": -0.004, + "2": -0.0034, + "8": 0.0021, + "14": 0.0536, + "7": 0.0565, + "12": 0.0524, + "13": 0.0076, + "11": 0.0678, + "1": 0.1317, + "5": 0.1554, + "4": 0.0546, + "9": 0.4798, + "0": 0.1365 + }, + "duration_seconds": 22.22, + "cluster_sizes": { + "3": 891, + "10": 930, + "6": 12002, + "2": 5369, + "8": 5384, + "14": 1138, + "7": 1452, + "12": 888, + "13": 3951, + "11": 1800, + "1": 686, + "5": 802, + "4": 1226, + "9": 297, + "0": 248 + } + }, + { + "name": "Stabilize", + "method": "Bootstrap (10 runs, Jaccard similarity)", + "threshold": 0.5, + "stable_clusters": 11, + "pruned_clusters": 4, + "duration_seconds": 7.207 + }, + { + "name": "Measure", + "method": "Evidence-based scoring (NO keyword heuristics)", + "metrics_computed": [ + "cross_team_entropy", + "source_diversity", + "no_reply_rate", + "author_gini", + "customer_density", + "temporal_span_days", + "reaction_rate" + ], + "severity_method": "Weighted metric combination (customer_density x cross_team, author_gini, no_reply_rate)", + "type_method": "Dominant metric signal (highest computed score wins)", + "patterns_surfaced": 11, + "patterns_pruned": 4, + "duration_seconds": 0.257 + }, + { + "name": "Network", + "method": "Communication graph (NetworkX), Louvain communities, centrality analysis", + "n_nodes": 521, + "n_edges": 8406, + "communities": 7, + "bridge_users": [ + "slack_admin_bot", + "'eid_3b36c220'", + "eid_3adf7b8a", + "eid_8658e19", + "'eid_46e00bc4'" + ], + "density": 0.06205521925291599 + }, + { + "name": "Temporal", + "method": "Change-point detection (ruptures PELT, RBF kernel)", + "total_days": 549, + "change_points": 0, + "busiest_day": { + "date": "2026-10-08", + "count": 208 + } + } + ], + "total_duration_seconds": 29.872, + "pruned_clusters": [ + { + "cluster_id": 2, + "size": 5369, + "reason": "unstable (stability 45% < 50% threshold)", + "stability": 0.449 + }, + { + "cluster_id": 14, + "size": 1138, + "reason": "unstable (stability 47% < 50% threshold)", + "stability": 0.469 + }, + { + "cluster_id": 12, + "size": 888, + "reason": "unstable (stability 25% < 50% threshold)", + "stability": 0.253 + }, + { + "cluster_id": 0, + "size": 248, + "reason": "unstable (stability 7% < 50% threshold)", + "stability": 0.073 + } + ], + "network": { + "n_nodes": 521, + "n_edges": 8406, + "n_communities": 7, + "graph_density": 0.06205521925291599, + "bridge_users": [ + "slack_admin_bot", + "'eid_3b36c220'", + "eid_3adf7b8a", + "eid_8658e19", + "'eid_46e00bc4'" + ], + "isolated_products": [], + "duration_seconds": 1.9393, + "top_product_overlaps": { + "AnomalyForce|CollaborateForce": 69, + "AnomalyForce|VizForce": 61, + "CollaborateForce|VizForce": 61, + "SearchFlow|SummarizeForce": 52, + "ContentForce|ProposalForce": 51, + "AnomalyForce|InsightForce": 50, + "CollaborateForce|InsightForce": 50, + "ActionGenie|SearchFlow": 49, + "ForecastForce|VizForce": 49, + "InsightForce|VizForce": 49 + } + }, + "temporal": { + "daily_volume": [ + { + "date": "2026-01-14", + "count": 2 + }, + { + "date": "2026-01-16", + "count": 12 + }, + { + "date": "2026-01-18", + "count": 13 + }, + { + "date": "2026-01-21", + "count": 16 + }, + { + "date": "2026-01-22", + "count": 2 + }, + { + "date": "2026-01-23", + "count": 12 + }, + { + "date": "2026-01-24", + "count": 1 + }, + { + "date": "2026-01-25", + "count": 1 + }, + { + "date": "2026-01-26", + "count": 19 + }, + { + "date": "2026-01-27", + "count": 10 + }, + { + "date": "2026-01-28", + "count": 16 + }, + { + "date": "2026-01-29", + "count": 50 + }, + { + "date": "2026-01-30", + "count": 32 + }, + { + "date": "2026-01-31", + "count": 3 + }, + { + "date": "2026-02-01", + "count": 51 + }, + { + "date": "2026-02-02", + "count": 1 + }, + { + "date": "2026-02-03", + "count": 15 + }, + { + "date": "2026-02-04", + "count": 1 + }, + { + "date": "2026-02-06", + "count": 10 + }, + { + "date": "2026-02-07", + "count": 31 + }, + { + "date": "2026-02-08", + "count": 10 + }, + { + "date": "2026-02-09", + "count": 15 + }, + { + "date": "2026-02-10", + "count": 30 + }, + { + "date": "2026-02-11", + "count": 23 + }, + { + "date": "2026-02-12", + "count": 1 + }, + { + "date": "2026-02-13", + "count": 14 + }, + { + "date": "2026-02-14", + "count": 29 + }, + { + "date": "2026-02-15", + "count": 3 + }, + { + "date": "2026-02-16", + "count": 2 + }, + { + "date": "2026-02-17", + "count": 38 + }, + { + "date": "2026-02-18", + "count": 31 + }, + { + "date": "2026-02-20", + "count": 3 + }, + { + "date": "2026-02-21", + "count": 14 + }, + { + "date": "2026-02-22", + "count": 30 + }, + { + "date": "2026-02-23", + "count": 28 + }, + { + "date": "2026-02-24", + "count": 2 + }, + { + "date": "2026-02-25", + "count": 15 + }, + { + "date": "2026-02-26", + "count": 1 + }, + { + "date": "2026-02-27", + "count": 29 + }, + { + "date": "2026-02-28", + "count": 6 + }, + { + "date": "2026-03-01", + "count": 49 + }, + { + "date": "2026-03-02", + "count": 11 + }, + { + "date": "2026-03-03", + "count": 18 + }, + { + "date": "2026-03-04", + "count": 65 + }, + { + "date": "2026-03-05", + "count": 14 + }, + { + "date": "2026-03-06", + "count": 16 + }, + { + "date": "2026-03-07", + "count": 33 + }, + { + "date": "2026-03-08", + "count": 23 + }, + { + "date": "2026-03-09", + "count": 27 + }, + { + "date": "2026-03-10", + "count": 27 + }, + { + "date": "2026-03-11", + "count": 2 + }, + { + "date": "2026-03-12", + "count": 47 + }, + { + "date": "2026-03-13", + "count": 26 + }, + { + "date": "2026-03-14", + "count": 77 + }, + { + "date": "2026-03-15", + "count": 35 + }, + { + "date": "2026-03-16", + "count": 72 + }, + { + "date": "2026-03-17", + "count": 5 + }, + { + "date": "2026-03-18", + "count": 19 + }, + { + "date": "2026-03-19", + "count": 12 + }, + { + "date": "2026-03-20", + "count": 45 + }, + { + "date": "2026-03-21", + "count": 45 + }, + { + "date": "2026-03-22", + "count": 59 + }, + { + "date": "2026-03-23", + "count": 49 + }, + { + "date": "2026-03-24", + "count": 71 + }, + { + "date": "2026-03-25", + "count": 45 + }, + { + "date": "2026-03-26", + "count": 41 + }, + { + "date": "2026-03-27", + "count": 55 + }, + { + "date": "2026-03-28", + "count": 50 + }, + { + "date": "2026-03-29", + "count": 91 + }, + { + "date": "2026-03-30", + "count": 85 + }, + { + "date": "2026-03-31", + "count": 100 + }, + { + "date": "2026-04-01", + "count": 92 + }, + { + "date": "2026-04-02", + "count": 90 + }, + { + "date": "2026-04-03", + "count": 77 + }, + { + "date": "2026-04-04", + "count": 41 + }, + { + "date": "2026-04-05", + "count": 82 + }, + { + "date": "2026-04-06", + "count": 82 + }, + { + "date": "2026-04-07", + "count": 83 + }, + { + "date": "2026-04-08", + "count": 81 + }, + { + "date": "2026-04-09", + "count": 70 + }, + { + "date": "2026-04-10", + "count": 71 + }, + { + "date": "2026-04-11", + "count": 106 + }, + { + "date": "2026-04-12", + "count": 74 + }, + { + "date": "2026-04-13", + "count": 56 + }, + { + "date": "2026-04-14", + "count": 21 + }, + { + "date": "2026-04-15", + "count": 45 + }, + { + "date": "2026-04-16", + "count": 100 + }, + { + "date": "2026-04-17", + "count": 78 + }, + { + "date": "2026-04-18", + "count": 28 + }, + { + "date": "2026-04-19", + "count": 64 + }, + { + "date": "2026-04-20", + "count": 94 + }, + { + "date": "2026-04-21", + "count": 80 + }, + { + "date": "2026-04-22", + "count": 90 + }, + { + "date": "2026-04-23", + "count": 40 + }, + { + "date": "2026-04-24", + "count": 85 + }, + { + "date": "2026-04-25", + "count": 101 + }, + { + "date": "2026-04-26", + "count": 30 + }, + { + "date": "2026-04-27", + "count": 97 + }, + { + "date": "2026-04-28", + "count": 82 + }, + { + "date": "2026-04-29", + "count": 63 + }, + { + "date": "2026-04-30", + "count": 50 + }, + { + "date": "2026-05-01", + "count": 53 + }, + { + "date": "2026-05-02", + "count": 116 + }, + { + "date": "2026-05-03", + "count": 105 + }, + { + "date": "2026-05-04", + "count": 134 + }, + { + "date": "2026-05-05", + "count": 104 + }, + { + "date": "2026-05-06", + "count": 40 + }, + { + "date": "2026-05-07", + "count": 63 + }, + { + "date": "2026-05-08", + "count": 137 + }, + { + "date": "2026-05-09", + "count": 95 + }, + { + "date": "2026-05-10", + "count": 43 + }, + { + "date": "2026-05-11", + "count": 110 + }, + { + "date": "2026-05-12", + "count": 69 + }, + { + "date": "2026-05-13", + "count": 59 + }, + { + "date": "2026-05-14", + "count": 139 + }, + { + "date": "2026-05-15", + "count": 105 + }, + { + "date": "2026-05-16", + "count": 49 + }, + { + "date": "2026-05-17", + "count": 73 + }, + { + "date": "2026-05-18", + "count": 51 + }, + { + "date": "2026-05-19", + "count": 82 + }, + { + "date": "2026-05-20", + "count": 142 + }, + { + "date": "2026-05-21", + "count": 62 + }, + { + "date": "2026-05-22", + "count": 103 + }, + { + "date": "2026-05-23", + "count": 113 + }, + { + "date": "2026-05-24", + "count": 90 + }, + { + "date": "2026-05-25", + "count": 146 + }, + { + "date": "2026-05-26", + "count": 125 + }, + { + "date": "2026-05-27", + "count": 136 + }, + { + "date": "2026-05-28", + "count": 60 + }, + { + "date": "2026-05-29", + "count": 107 + }, + { + "date": "2026-05-30", + "count": 94 + }, + { + "date": "2026-05-31", + "count": 51 + }, + { + "date": "2026-06-01", + "count": 100 + }, + { + "date": "2026-06-02", + "count": 104 + }, + { + "date": "2026-06-03", + "count": 76 + }, + { + "date": "2026-06-04", + "count": 59 + }, + { + "date": "2026-06-05", + "count": 85 + }, + { + "date": "2026-06-06", + "count": 88 + }, + { + "date": "2026-06-07", + "count": 87 + }, + { + "date": "2026-06-08", + "count": 84 + }, + { + "date": "2026-06-09", + "count": 59 + }, + { + "date": "2026-06-10", + "count": 66 + }, + { + "date": "2026-06-11", + "count": 91 + }, + { + "date": "2026-06-12", + "count": 65 + }, + { + "date": "2026-06-13", + "count": 170 + }, + { + "date": "2026-06-14", + "count": 37 + }, + { + "date": "2026-06-15", + "count": 53 + }, + { + "date": "2026-06-16", + "count": 72 + }, + { + "date": "2026-06-17", + "count": 139 + }, + { + "date": "2026-06-18", + "count": 170 + }, + { + "date": "2026-06-19", + "count": 83 + }, + { + "date": "2026-06-20", + "count": 122 + }, + { + "date": "2026-06-21", + "count": 104 + }, + { + "date": "2026-06-22", + "count": 178 + }, + { + "date": "2026-06-23", + "count": 130 + }, + { + "date": "2026-06-24", + "count": 110 + }, + { + "date": "2026-06-25", + "count": 66 + }, + { + "date": "2026-06-26", + "count": 92 + }, + { + "date": "2026-06-27", + "count": 112 + }, + { + "date": "2026-06-28", + "count": 98 + }, + { + "date": "2026-06-29", + "count": 185 + }, + { + "date": "2026-06-30", + "count": 61 + }, + { + "date": "2026-07-01", + "count": 116 + }, + { + "date": "2026-07-02", + "count": 140 + }, + { + "date": "2026-07-03", + "count": 69 + }, + { + "date": "2026-07-04", + "count": 109 + }, + { + "date": "2026-07-05", + "count": 167 + }, + { + "date": "2026-07-06", + "count": 72 + }, + { + "date": "2026-07-07", + "count": 155 + }, + { + "date": "2026-07-08", + "count": 174 + }, + { + "date": "2026-07-09", + "count": 51 + }, + { + "date": "2026-07-10", + "count": 168 + }, + { + "date": "2026-07-11", + "count": 146 + }, + { + "date": "2026-07-12", + "count": 91 + }, + { + "date": "2026-07-13", + "count": 129 + }, + { + "date": "2026-07-14", + "count": 116 + }, + { + "date": "2026-07-15", + "count": 109 + }, + { + "date": "2026-07-16", + "count": 119 + }, + { + "date": "2026-07-17", + "count": 104 + }, + { + "date": "2026-07-18", + "count": 100 + }, + { + "date": "2026-07-19", + "count": 87 + }, + { + "date": "2026-07-20", + "count": 114 + }, + { + "date": "2026-07-21", + "count": 118 + }, + { + "date": "2026-07-22", + "count": 77 + }, + { + "date": "2026-07-23", + "count": 106 + }, + { + "date": "2026-07-24", + "count": 91 + }, + { + "date": "2026-07-25", + "count": 143 + }, + { + "date": "2026-07-26", + "count": 70 + }, + { + "date": "2026-07-27", + "count": 125 + }, + { + "date": "2026-07-28", + "count": 103 + }, + { + "date": "2026-07-29", + "count": 121 + }, + { + "date": "2026-07-30", + "count": 133 + }, + { + "date": "2026-07-31", + "count": 58 + }, + { + "date": "2026-08-01", + "count": 82 + }, + { + "date": "2026-08-02", + "count": 107 + }, + { + "date": "2026-08-03", + "count": 123 + }, + { + "date": "2026-08-04", + "count": 89 + }, + { + "date": "2026-08-05", + "count": 115 + }, + { + "date": "2026-08-06", + "count": 114 + }, + { + "date": "2026-08-07", + "count": 64 + }, + { + "date": "2026-08-08", + "count": 85 + }, + { + "date": "2026-08-09", + "count": 81 + }, + { + "date": "2026-08-10", + "count": 83 + }, + { + "date": "2026-08-11", + "count": 75 + }, + { + "date": "2026-08-12", + "count": 116 + }, + { + "date": "2026-08-13", + "count": 108 + }, + { + "date": "2026-08-14", + "count": 76 + }, + { + "date": "2026-08-15", + "count": 68 + }, + { + "date": "2026-08-16", + "count": 75 + }, + { + "date": "2026-08-17", + "count": 97 + }, + { + "date": "2026-08-18", + "count": 88 + }, + { + "date": "2026-08-19", + "count": 126 + }, + { + "date": "2026-08-20", + "count": 78 + }, + { + "date": "2026-08-21", + "count": 121 + }, + { + "date": "2026-08-22", + "count": 101 + }, + { + "date": "2026-08-23", + "count": 108 + }, + { + "date": "2026-08-24", + "count": 115 + }, + { + "date": "2026-08-25", + "count": 102 + }, + { + "date": "2026-08-26", + "count": 135 + }, + { + "date": "2026-08-27", + "count": 129 + }, + { + "date": "2026-08-28", + "count": 146 + }, + { + "date": "2026-08-29", + "count": 121 + }, + { + "date": "2026-08-30", + "count": 144 + }, + { + "date": "2026-08-31", + "count": 107 + }, + { + "date": "2026-09-01", + "count": 169 + }, + { + "date": "2026-09-02", + "count": 141 + }, + { + "date": "2026-09-03", + "count": 124 + }, + { + "date": "2026-09-04", + "count": 169 + }, + { + "date": "2026-09-05", + "count": 128 + }, + { + "date": "2026-09-06", + "count": 86 + }, + { + "date": "2026-09-07", + "count": 123 + }, + { + "date": "2026-09-08", + "count": 181 + }, + { + "date": "2026-09-09", + "count": 93 + }, + { + "date": "2026-09-10", + "count": 158 + }, + { + "date": "2026-09-11", + "count": 64 + }, + { + "date": "2026-09-12", + "count": 147 + }, + { + "date": "2026-09-13", + "count": 62 + }, + { + "date": "2026-09-14", + "count": 180 + }, + { + "date": "2026-09-15", + "count": 151 + }, + { + "date": "2026-09-16", + "count": 122 + }, + { + "date": "2026-09-17", + "count": 103 + }, + { + "date": "2026-09-18", + "count": 116 + }, + { + "date": "2026-09-19", + "count": 131 + }, + { + "date": "2026-09-20", + "count": 87 + }, + { + "date": "2026-09-21", + "count": 110 + }, + { + "date": "2026-09-22", + "count": 102 + }, + { + "date": "2026-09-23", + "count": 104 + }, + { + "date": "2026-09-24", + "count": 142 + }, + { + "date": "2026-09-25", + "count": 118 + }, + { + "date": "2026-09-26", + "count": 119 + }, + { + "date": "2026-09-27", + "count": 105 + }, + { + "date": "2026-09-28", + "count": 79 + }, + { + "date": "2026-09-29", + "count": 96 + }, + { + "date": "2026-09-30", + "count": 97 + }, + { + "date": "2026-10-01", + "count": 160 + }, + { + "date": "2026-10-02", + "count": 129 + }, + { + "date": "2026-10-03", + "count": 111 + }, + { + "date": "2026-10-04", + "count": 102 + }, + { + "date": "2026-10-05", + "count": 140 + }, + { + "date": "2026-10-06", + "count": 111 + }, + { + "date": "2026-10-07", + "count": 84 + }, + { + "date": "2026-10-08", + "count": 208 + }, + { + "date": "2026-10-09", + "count": 81 + }, + { + "date": "2026-10-10", + "count": 52 + }, + { + "date": "2026-10-11", + "count": 133 + }, + { + "date": "2026-10-12", + "count": 128 + }, + { + "date": "2026-10-13", + "count": 86 + }, + { + "date": "2026-10-14", + "count": 86 + }, + { + "date": "2026-10-15", + "count": 88 + }, + { + "date": "2026-10-16", + "count": 130 + }, + { + "date": "2026-10-17", + "count": 94 + }, + { + "date": "2026-10-18", + "count": 127 + }, + { + "date": "2026-10-19", + "count": 124 + }, + { + "date": "2026-10-20", + "count": 92 + }, + { + "date": "2026-10-21", + "count": 118 + }, + { + "date": "2026-10-22", + "count": 76 + }, + { + "date": "2026-10-23", + "count": 125 + }, + { + "date": "2026-10-24", + "count": 86 + }, + { + "date": "2026-10-25", + "count": 87 + }, + { + "date": "2026-10-26", + "count": 89 + }, + { + "date": "2026-10-27", + "count": 67 + }, + { + "date": "2026-10-28", + "count": 105 + }, + { + "date": "2026-10-29", + "count": 133 + }, + { + "date": "2026-10-30", + "count": 145 + }, + { + "date": "2026-10-31", + "count": 98 + }, + { + "date": "2026-11-01", + "count": 85 + }, + { + "date": "2026-11-02", + "count": 99 + }, + { + "date": "2026-11-03", + "count": 100 + }, + { + "date": "2026-11-04", + "count": 86 + }, + { + "date": "2026-11-05", + "count": 61 + }, + { + "date": "2026-11-06", + "count": 111 + }, + { + "date": "2026-11-07", + "count": 107 + }, + { + "date": "2026-11-08", + "count": 111 + }, + { + "date": "2026-11-09", + "count": 78 + }, + { + "date": "2026-11-10", + "count": 152 + }, + { + "date": "2026-11-11", + "count": 92 + }, + { + "date": "2026-11-12", + "count": 112 + }, + { + "date": "2026-11-13", + "count": 88 + }, + { + "date": "2026-11-14", + "count": 80 + }, + { + "date": "2026-11-15", + "count": 107 + }, + { + "date": "2026-11-16", + "count": 95 + }, + { + "date": "2026-11-17", + "count": 84 + }, + { + "date": "2026-11-18", + "count": 58 + }, + { + "date": "2026-11-19", + "count": 109 + }, + { + "date": "2026-11-20", + "count": 73 + }, + { + "date": "2026-11-21", + "count": 102 + }, + { + "date": "2026-11-22", + "count": 106 + }, + { + "date": "2026-11-23", + "count": 98 + }, + { + "date": "2026-11-24", + "count": 60 + }, + { + "date": "2026-11-25", + "count": 83 + }, + { + "date": "2026-11-26", + "count": 62 + }, + { + "date": "2026-11-27", + "count": 102 + }, + { + "date": "2026-11-28", + "count": 49 + }, + { + "date": "2026-11-29", + "count": 110 + }, + { + "date": "2026-11-30", + "count": 52 + }, + { + "date": "2026-12-01", + "count": 106 + }, + { + "date": "2026-12-02", + "count": 55 + }, + { + "date": "2026-12-03", + "count": 83 + }, + { + "date": "2026-12-04", + "count": 92 + }, + { + "date": "2026-12-05", + "count": 77 + }, + { + "date": "2026-12-06", + "count": 60 + }, + { + "date": "2026-12-07", + "count": 48 + }, + { + "date": "2026-12-08", + "count": 47 + }, + { + "date": "2026-12-09", + "count": 82 + }, + { + "date": "2026-12-10", + "count": 47 + }, + { + "date": "2026-12-11", + "count": 56 + }, + { + "date": "2026-12-12", + "count": 53 + }, + { + "date": "2026-12-13", + "count": 69 + }, + { + "date": "2026-12-14", + "count": 49 + }, + { + "date": "2026-12-15", + "count": 85 + }, + { + "date": "2026-12-16", + "count": 42 + }, + { + "date": "2026-12-17", + "count": 68 + }, + { + "date": "2026-12-18", + "count": 62 + }, + { + "date": "2026-12-19", + "count": 61 + }, + { + "date": "2026-12-20", + "count": 38 + }, + { + "date": "2026-12-21", + "count": 123 + }, + { + "date": "2026-12-22", + "count": 79 + }, + { + "date": "2026-12-23", + "count": 18 + }, + { + "date": "2026-12-24", + "count": 92 + }, + { + "date": "2026-12-25", + "count": 39 + }, + { + "date": "2026-12-26", + "count": 69 + }, + { + "date": "2026-12-27", + "count": 88 + }, + { + "date": "2026-12-28", + "count": 21 + }, + { + "date": "2026-12-29", + "count": 73 + }, + { + "date": "2026-12-30", + "count": 80 + }, + { + "date": "2026-12-31", + "count": 79 + }, + { + "date": "2027-01-01", + "count": 36 + }, + { + "date": "2027-01-02", + "count": 40 + }, + { + "date": "2027-01-03", + "count": 35 + }, + { + "date": "2027-01-04", + "count": 68 + }, + { + "date": "2027-01-05", + "count": 70 + }, + { + "date": "2027-01-06", + "count": 47 + }, + { + "date": "2027-01-07", + "count": 35 + }, + { + "date": "2027-01-08", + "count": 51 + }, + { + "date": "2027-01-09", + "count": 72 + }, + { + "date": "2027-01-10", + "count": 60 + }, + { + "date": "2027-01-11", + "count": 56 + }, + { + "date": "2027-01-12", + "count": 48 + }, + { + "date": "2027-01-13", + "count": 39 + }, + { + "date": "2027-01-14", + "count": 56 + }, + { + "date": "2027-01-15", + "count": 54 + }, + { + "date": "2027-01-16", + "count": 18 + }, + { + "date": "2027-01-17", + "count": 56 + }, + { + "date": "2027-01-18", + "count": 58 + }, + { + "date": "2027-01-19", + "count": 25 + }, + { + "date": "2027-01-20", + "count": 32 + }, + { + "date": "2027-01-21", + "count": 48 + }, + { + "date": "2027-01-22", + "count": 38 + }, + { + "date": "2027-01-23", + "count": 49 + }, + { + "date": "2027-01-24", + "count": 70 + }, + { + "date": "2027-01-25", + "count": 42 + }, + { + "date": "2027-01-26", + "count": 33 + }, + { + "date": "2027-01-27", + "count": 51 + }, + { + "date": "2027-01-28", + "count": 26 + }, + { + "date": "2027-01-29", + "count": 56 + }, + { + "date": "2027-01-30", + "count": 11 + }, + { + "date": "2027-01-31", + "count": 41 + }, + { + "date": "2027-02-01", + "count": 40 + }, + { + "date": "2027-02-02", + "count": 40 + }, + { + "date": "2027-02-03", + "count": 45 + }, + { + "date": "2027-02-04", + "count": 65 + }, + { + "date": "2027-02-05", + "count": 12 + }, + { + "date": "2027-02-06", + "count": 93 + }, + { + "date": "2027-02-07", + "count": 37 + }, + { + "date": "2027-02-08", + "count": 20 + }, + { + "date": "2027-02-09", + "count": 47 + }, + { + "date": "2027-02-10", + "count": 68 + }, + { + "date": "2027-02-11", + "count": 49 + }, + { + "date": "2027-02-12", + "count": 19 + }, + { + "date": "2027-02-13", + "count": 38 + }, + { + "date": "2027-02-14", + "count": 45 + }, + { + "date": "2027-02-15", + "count": 45 + }, + { + "date": "2027-02-16", + "count": 34 + }, + { + "date": "2027-02-17", + "count": 68 + }, + { + "date": "2027-02-18", + "count": 10 + }, + { + "date": "2027-02-19", + "count": 25 + }, + { + "date": "2027-02-20", + "count": 41 + }, + { + "date": "2027-02-21", + "count": 22 + }, + { + "date": "2027-02-22", + "count": 53 + }, + { + "date": "2027-02-23", + "count": 8 + }, + { + "date": "2027-02-24", + "count": 24 + }, + { + "date": "2027-02-25", + "count": 26 + }, + { + "date": "2027-02-26", + "count": 44 + }, + { + "date": "2027-02-27", + "count": 41 + }, + { + "date": "2027-02-28", + "count": 48 + }, + { + "date": "2027-03-01", + "count": 63 + }, + { + "date": "2027-03-02", + "count": 47 + }, + { + "date": "2027-03-03", + "count": 43 + }, + { + "date": "2027-03-04", + "count": 40 + }, + { + "date": "2027-03-05", + "count": 26 + }, + { + "date": "2027-03-06", + "count": 28 + }, + { + "date": "2027-03-07", + "count": 41 + }, + { + "date": "2027-03-08", + "count": 53 + }, + { + "date": "2027-03-09", + "count": 34 + }, + { + "date": "2027-03-10", + "count": 19 + }, + { + "date": "2027-03-11", + "count": 14 + }, + { + "date": "2027-03-12", + "count": 33 + }, + { + "date": "2027-03-13", + "count": 19 + }, + { + "date": "2027-03-14", + "count": 9 + }, + { + "date": "2027-03-15", + "count": 59 + }, + { + "date": "2027-03-16", + "count": 11 + }, + { + "date": "2027-03-17", + "count": 30 + }, + { + "date": "2027-03-18", + "count": 31 + }, + { + "date": "2027-03-19", + "count": 16 + }, + { + "date": "2027-03-20", + "count": 18 + }, + { + "date": "2027-03-21", + "count": 19 + }, + { + "date": "2027-03-22", + "count": 22 + }, + { + "date": "2027-03-23", + "count": 37 + }, + { + "date": "2027-03-24", + "count": 11 + }, + { + "date": "2027-03-25", + "count": 27 + }, + { + "date": "2027-03-26", + "count": 19 + }, + { + "date": "2027-03-27", + "count": 32 + }, + { + "date": "2027-03-28", + "count": 24 + }, + { + "date": "2027-03-29", + "count": 17 + }, + { + "date": "2027-03-30", + "count": 41 + }, + { + "date": "2027-03-31", + "count": 9 + }, + { + "date": "2027-04-01", + "count": 32 + }, + { + "date": "2027-04-02", + "count": 26 + }, + { + "date": "2027-04-04", + "count": 27 + }, + { + "date": "2027-04-05", + "count": 19 + }, + { + "date": "2027-04-06", + "count": 12 + }, + { + "date": "2027-04-07", + "count": 27 + }, + { + "date": "2027-04-08", + "count": 11 + }, + { + "date": "2027-04-09", + "count": 41 + }, + { + "date": "2027-04-10", + "count": 1 + }, + { + "date": "2027-04-11", + "count": 24 + }, + { + "date": "2027-04-12", + "count": 21 + }, + { + "date": "2027-04-13", + "count": 11 + }, + { + "date": "2027-04-14", + "count": 9 + }, + { + "date": "2027-04-15", + "count": 30 + }, + { + "date": "2027-04-18", + "count": 21 + }, + { + "date": "2027-04-19", + "count": 9 + }, + { + "date": "2027-04-20", + "count": 21 + }, + { + "date": "2027-04-22", + "count": 13 + }, + { + "date": "2027-04-23", + "count": 19 + }, + { + "date": "2027-04-24", + "count": 7 + }, + { + "date": "2027-04-26", + "count": 8 + }, + { + "date": "2027-04-27", + "count": 9 + }, + { + "date": "2027-04-28", + "count": 11 + }, + { + "date": "2027-04-29", + "count": 10 + }, + { + "date": "2027-04-30", + "count": 7 + }, + { + "date": "2027-05-01", + "count": 8 + }, + { + "date": "2027-05-02", + "count": 16 + }, + { + "date": "2027-05-05", + "count": 20 + }, + { + "date": "2027-05-06", + "count": 7 + }, + { + "date": "2027-05-07", + "count": 12 + }, + { + "date": "2027-05-08", + "count": 19 + }, + { + "date": "2027-05-11", + "count": 12 + }, + { + "date": "2027-05-12", + "count": 20 + }, + { + "date": "2027-05-13", + "count": 14 + }, + { + "date": "2027-05-15", + "count": 18 + }, + { + "date": "2027-05-17", + "count": 11 + }, + { + "date": "2027-05-19", + "count": 20 + }, + { + "date": "2027-05-22", + "count": 20 + }, + { + "date": "2027-05-23", + "count": 8 + }, + { + "date": "2027-05-24", + "count": 15 + }, + { + "date": "2027-05-25", + "count": 7 + }, + { + "date": "2027-05-26", + "count": 19 + }, + { + "date": "2027-05-29", + "count": 21 + }, + { + "date": "2027-05-30", + "count": 18 + }, + { + "date": "2027-06-01", + "count": 23 + }, + { + "date": "2027-06-02", + "count": 11 + }, + { + "date": "2027-06-03", + "count": 11 + }, + { + "date": "2027-06-05", + "count": 20 + }, + { + "date": "2027-06-06", + "count": 34 + }, + { + "date": "2027-06-07", + "count": 1 + }, + { + "date": "2027-06-08", + "count": 11 + }, + { + "date": "2027-06-09", + "count": 20 + }, + { + "date": "2027-06-14", + "count": 10 + }, + { + "date": "2027-06-15", + "count": 9 + }, + { + "date": "2027-06-16", + "count": 7 + }, + { + "date": "2027-06-17", + "count": 10 + }, + { + "date": "2027-06-18", + "count": 10 + }, + { + "date": "2027-06-19", + "count": 21 + }, + { + "date": "2027-06-20", + "count": 16 + }, + { + "date": "2027-06-24", + "count": 10 + }, + { + "date": "2027-06-25", + "count": 11 + }, + { + "date": "2027-06-27", + "count": 11 + }, + { + "date": "2027-06-28", + "count": 7 + }, + { + "date": "2027-06-29", + "count": 8 + }, + { + "date": "2027-06-30", + "count": 9 + }, + { + "date": "2027-07-04", + "count": 9 + }, + { + "date": "2027-07-05", + "count": 6 + }, + { + "date": "2027-07-06", + "count": 7 + }, + { + "date": "2027-07-08", + "count": 1 + }, + { + "date": "2027-07-09", + "count": 11 + }, + { + "date": "2027-07-10", + "count": 11 + }, + { + "date": "2027-07-11", + "count": 8 + }, + { + "date": "2027-07-14", + "count": 12 + }, + { + "date": "2027-07-16", + "count": 9 + } + ], + "change_points": [], + "per_product_trends": { + "MonitorForce": { + "total": 894, + "daily_avg": 1.63, + "peak_day": "2026-01-29", + "trend": "decreasing" + }, + "WorkFlowGenie": { + "total": 1072, + "daily_avg": 1.95, + "peak_day": "2026-02-10", + "trend": "decreasing" + }, + "ConnectForce": { + "total": 977, + "daily_avg": 1.78, + "peak_day": "2026-05-08", + "trend": "decreasing" + }, + "SearchFlow": { + "total": 1235, + "daily_avg": 2.25, + "peak_day": "2026-05-04", + "trend": "decreasing" + }, + "ContentForce": { + "total": 1066, + "daily_avg": 1.94, + "peak_day": "2026-03-04", + "trend": "decreasing" + }, + "InsightForce": { + "total": 1262, + "daily_avg": 2.3, + "peak_day": "2026-03-04", + "trend": "decreasing" + }, + "TrendForce": { + "total": 1011, + "daily_avg": 1.84, + "peak_day": "2026-03-16", + "trend": "decreasing" + }, + "SupportForce": { + "total": 925, + "daily_avg": 1.68, + "peak_day": "2026-03-24", + "trend": "decreasing" + }, + "ForecastForce": { + "total": 1079, + "daily_avg": 1.97, + "peak_day": "2026-04-10", + "trend": "decreasing" + }, + "LeadForce": { + "total": 1243, + "daily_avg": 2.26, + "peak_day": "2026-04-09", + "trend": "decreasing" + }, + "VizForce": { + "total": 1371, + "daily_avg": 2.5, + "peak_day": "2026-06-13", + "trend": "decreasing" + }, + "SentimentForce": { + "total": 1136, + "daily_avg": 2.07, + "peak_day": "2026-04-02", + "trend": "decreasing" + }, + "CoachForce": { + "total": 1111, + "daily_avg": 2.02, + "peak_day": "2026-10-01", + "trend": "decreasing" + }, + "FlowForce": { + "total": 877, + "daily_avg": 1.6, + "peak_day": "2026-04-24", + "trend": "decreasing" + }, + "SecurityForce": { + "total": 1072, + "daily_avg": 1.95, + "peak_day": "2026-04-24", + "trend": "decreasing" + }, + "FeedbackForce": { + "total": 998, + "daily_avg": 1.82, + "peak_day": "2026-06-30", + "trend": "decreasing" + }, + "AnomalyForce": { + "total": 925, + "daily_avg": 1.68, + "peak_day": "2026-05-04", + "trend": "decreasing" + }, + "PersonalizeForce": { + "total": 964, + "daily_avg": 1.76, + "peak_day": "2026-05-23", + "trend": "decreasing" + }, + "ExplainabilityForce": { + "total": 1276, + "daily_avg": 2.32, + "peak_day": "2026-10-08", + "trend": "stable" + }, + "KnowledgeForce": { + "total": 1113, + "daily_avg": 2.03, + "peak_day": "2026-05-30", + "trend": "decreasing" + }, + "CollaborateForce": { + "total": 1491, + "daily_avg": 2.72, + "peak_day": "2026-06-02", + "trend": "stable" + }, + "ActionGenie": { + "total": 1047, + "daily_avg": 1.91, + "peak_day": "2026-10-30", + "trend": "stable" + }, + "AutoTuneForce": { + "total": 1136, + "daily_avg": 2.07, + "peak_day": "2026-06-29", + "trend": "decreasing" + }, + "CollaborationForce": { + "total": 844, + "daily_avg": 1.54, + "peak_day": "2026-06-22", + "trend": "decreasing" + }, + "PitchForce": { + "total": 1187, + "daily_avg": 2.16, + "peak_day": "2026-07-08", + "trend": "stable" + }, + "EdgeForce": { + "total": 975, + "daily_avg": 1.78, + "peak_day": "2026-08-27", + "trend": "increasing" + }, + "ContextForce": { + "total": 1022, + "daily_avg": 1.86, + "peak_day": "2026-09-17", + "trend": "increasing" + }, + "SearchForce": { + "total": 1206, + "daily_avg": 2.2, + "peak_day": "2027-03-23", + "trend": "increasing" + }, + "SummarizeForce": { + "total": 1190, + "daily_avg": 2.17, + "peak_day": "2026-09-27", + "trend": "increasing" + }, + "ProposalForce": { + "total": 1076, + "daily_avg": 1.96, + "peak_day": "2026-12-21", + "trend": "increasing" + } + }, + "activity_windows": { + "earliest": "2026-01-14", + "latest": "2027-07-16", + "total_days": 549, + "active_days": 510 + }, + "busiest_day": { + "date": "2026-10-08", + "count": 208 + }, + "quietest_day": { + "date": "2026-01-24", + "count": 1 + }, + "duration_seconds": 0.1468 + } + }, + "cards": [ + { + "id": "PAT-C64B1E", + "title": "Programming / Programming Interface", + "description": "Cluster of 1,800 documents across 30 products. Cross-team entropy: 0.97. Author concentration (Gini): 0.48. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "MEDIUM", + "confidence_score": 0.6, + "severity": "high", + "severity_reasoning": "Computed severity score: 2.0/5.0. Based on: customer density (0.000), cross-team entropy (0.97), author Gini (0.48), no-reply rate (100%).", + "top_phrases": [ + "api", + "programming", + "programming interface", + "api application", + "api application programming", + "application programming" + ], + "occurrence": "1,800 of 37,064 documents (4.9%)", + "effect": "Spans 30 products, 304 authors, 535 days. Sources: {'slack': 1373, 'pull_request': 425, 'document': 2}.", + "evidence": { + "source_count": 1800, + "source_breakdown": { + "slack": 1373, + "pull_request": 425, + "document": 2 + }, + "sample_messages": [ + "Great points, both of you! Next, let's look at Rocket.Chat's real-time event API integration. This feature facilitates seamless interactions with external applications. Here's the PR link: https://git", + "Thanks for the input, everyone! I'll take a closer look at these features and see how we can adapt some of their strategies for our Slack Events API integration. I'll keep you all updated on my findin", + "Alright, I've gone through the changes. The Slack Events API integration looks solid, and I can see that the endpoints are correctly set up to handle incoming events. Nice work! \ud83d\udc4d", + "Sure, I can provide some context. The new direction focuses on enhancing our core features rather than expanding third-party integrations. The Slack Events API integration was meant to support interac" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [], + "time_range": "2026-01-21 to 2027-07-10" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.969, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.75, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.478, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 304, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 535, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.43, + "communication": 4.41, + "anomaly": 2.0 + }, + "sample_snippets": [ + "Great points, both of you! Next, let's look at Rocket.Chat's real-time event API integration. This f", + "Thanks for the input, everyone! I'll take a closer look at these features and see how we can adapt s" + ], + "caveats": "Confidence 60% = stability (96%) x silhouette (0.068). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 1,800 documents. Dominant metric signal: communication (4.4). Top phrases: api, programming, programming interface.", + "computation": { + "cluster_id": 11, + "cluster_size": 1800, + "method": "TF-IDF + KMeans", + "silhouette": 0.0678, + "stability": { + "mean": 0.958, + "std": 0.024, + "min": 0.919, + "max": 0.994, + "stable": true + }, + "severity_score": 2.02, + "top_tfidf_terms": [ + "api", + "programming", + "programming interface", + "api application", + "api application programming", + "application programming" + ] + } + }, + { + "id": "PAT-6C0A38", + "title": "Pr Pull Request / Pr Pull", + "description": "Cluster of 3,951 documents across 30 products. Cross-team entropy: 0.99. Author concentration (Gini): 0.39. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "MEDIUM", + "confidence_score": 0.52, + "severity": "high", + "severity_reasoning": "Computed severity score: 2.0/5.0. Based on: customer density (0.000), cross-team entropy (0.99), author Gini (0.39), no-reply rate (100%).", + "top_phrases": [ + "pr", + "pr pull request", + "pr pull", + "pull request", + "pull", + "request" + ], + "occurrence": "3,951 of 37,064 documents (10.7%)", + "effect": "Spans 30 products, 378 authors, 476 days. Sources: {'slack': 3711, 'pull_request': 240}.", + "evidence": { + "source_count": 3951, + "source_breakdown": { + "slack": 3711, + "pull_request": 240 + }, + "sample_messages": [ + "First up, we have Mattermost's integration of webhooks for third-party app interactions. This feature allows Mattermost to exchange data in real-time with other applications. You can check out the PR ", + "Absolutely, Bob! Lastly, we have Zulip's enhancement of their bot framework to handle events from external services. This improves automation and integration capabilities. Check out the PR here: https", + "Great points, @eid_49aa3b00 and Charlie! Next up is TensorFlow's feature. They have a PR titled 'Efficient Data Pipeline for Model Training' which enhances data handling efficiency during model traini", + "Agreed, both of you. Lastly, let's look at Apache Airflow's feature. They have a PR titled 'Dynamic Task Scheduling for Data Pipelines' which adds dynamic scheduling capabilities. Here's the link: htt" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [ + "BlueWave" + ], + "time_range": "2026-03-20 to 2027-07-10" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.99, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.5, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.394, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 378, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [ + "BlueWave" + ], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 476, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.18, + "communication": 3.97, + "anomaly": 2.0 + }, + "sample_snippets": [ + "First up, we have Mattermost's integration of webhooks for third-party app interactions. This featur", + "Absolutely, Bob! Lastly, we have Zulip's enhancement of their bot framework to handle events from ex" + ], + "caveats": "Confidence 52% = stability (86%) x silhouette (0.008). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 3,951 documents. Dominant metric signal: communication (4.0). Top phrases: pr, pr pull request, pr pull.", + "computation": { + "cluster_id": 13, + "cluster_size": 3951, + "method": "TF-IDF + KMeans", + "silhouette": 0.0076, + "stability": { + "mean": 0.864, + "std": 0.063, + "min": 0.755, + "max": 0.978, + "stable": true + }, + "severity_score": 1.99, + "top_tfidf_terms": [ + "pr", + "pr pull request", + "pr pull", + "pull request", + "pull", + "request" + ] + } + }, + { + "id": "PAT-385A70", + "title": "User / Think / Great", + "description": "Cluster of 12,002 documents across 30 products. Cross-team entropy: 0.99. Author concentration (Gini): 0.37. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "LOW", + "confidence_score": 0.42, + "severity": "high", + "severity_reasoning": "Computed severity score: 2.5/5.0. Based on: customer density (0.000), cross-team entropy (0.99), author Gini (0.37), no-reply rate (100%).", + "top_phrases": [ + "user", + "think", + "great", + "good", + "sounds", + "maybe" + ], + "occurrence": "12,002 of 37,064 documents (32.4%)", + "effect": "Spans 30 products, 516 authors, 546 days. Sources: {'slack': 10781, 'transcript': 22, 'document': 10, 'pull_request': 1189}.", + "evidence": { + "source_count": 12002, + "source_breakdown": { + "slack": 10781, + "transcript": 22, + "document": 10, + "pull_request": 1189 + }, + "sample_messages": [ + "Thanks for sharing, Charlie! @eid_13fdff84 I think the Executive Summary is clear, but it might benefit from a bit more detail on the specific AI technologies used. This could help differentiate us fr", + "Good point, Emma. Also, in the Market Overview section, it might be helpful to include some recent statistics or trends about the growth of AI in collaboration tools. This could strengthen our argumen", + "I agree with George. Adding some data could really bolster the Market Overview. Additionally, in the Target Audience section, maybe we could specify more about the size of businesses we're targeting. ", + "The Competitive Analysis is solid, but maybe we could add a bit more about our unique selling proposition. What makes our AI suggestions more contextually relevant than others? This could be a key dif" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [ + "DataSolutions", + "TechCorp", + "SmartData" + ], + "time_range": "2026-01-16 to 2027-07-16" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.995, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 1.0, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.371, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 516, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [ + "DataSolutions", + "TechCorp", + "SmartData" + ], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 546, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.11, + "communication": 4.98, + "anomaly": 2.0 + }, + "sample_snippets": [ + "Thanks for sharing, Charlie! @eid_13fdff84 I think the Executive Summary is clear, but it might bene", + "Good point, Emma. Also, in the Market Overview section, it might be helpful to include some recent s" + ], + "caveats": "Confidence 42% = stability (70%) x silhouette (-0.004). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 12,002 documents. Dominant metric signal: communication (5.0). Top phrases: user, think, great.", + "computation": { + "cluster_id": 6, + "cluster_size": 12002, + "method": "TF-IDF + KMeans", + "silhouette": -0.004, + "stability": { + "mean": 0.703, + "std": 0.123, + "min": 0.569, + "max": 0.922, + "stable": true + }, + "severity_score": 2.48, + "top_tfidf_terms": [ + "user", + "think", + "great", + "good", + "sounds", + "maybe" + ] + } + }, + { + "id": "PAT-85D069", + "title": "Data / Time / Processing", + "description": "Cluster of 5,384 documents across 30 products. Cross-team entropy: 1.00. Author concentration (Gini): 0.51. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "LOW", + "confidence_score": 0.38, + "severity": "high", + "severity_reasoning": "Computed severity score: 2.0/5.0. Based on: customer density (0.000), cross-team entropy (1.00), author Gini (0.51), no-reply rate (100%).", + "top_phrases": [ + "data", + "time", + "processing", + "real time", + "real", + "model" + ], + "occurrence": "5,384 of 37,064 documents (14.5%)", + "effect": "Spans 30 products, 449 authors, 546 days. Sources: {'slack': 3287, 'document': 388, 'pull_request': 1708, 'transcript': 1}.", + "evidence": { + "source_count": 5384, + "source_breakdown": { + "slack": 3287, + "document": 388, + "pull_request": 1708, + "transcript": 1 + }, + "sample_messages": [ + "@here Explore the capabilities of the spaCy library, a powerful tool for natural language processing that can be used to build features like onForceX's smart action suggestions. https://github.com/exp", + "Sounds good, George. One more thing, in the Security and Authentication section, should we mention any specific compliance standards we're aiming to meet, like GDPR or CCPA? This could be important fo", + "I agree with @eid_fd8cecea. The integration with Spaces discussions is a unique feature. It could really set them apart. But I'm curious about how they handle data privacy. \ud83e\udd14", + "Yeah, @eid_4df5d4b7, I noticed that too. They seem to have a strong focus on natural language processing for understanding context in DMs and group chats." + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [], + "time_range": "2026-01-16 to 2027-07-16" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.996, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 1.0, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.513, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 449, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 546, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.54, + "communication": 4.99, + "anomaly": 2.0 + }, + "sample_snippets": [ + "@here Explore the capabilities of the spaCy library, a powerful tool for natural language processing", + "Sounds good, George. One more thing, in the Security and Authentication section, should we mention a" + ], + "caveats": "Confidence 38% = stability (63%) x silhouette (0.002). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 5,384 documents. Dominant metric signal: communication (5.0). Top phrases: data, time, processing.", + "computation": { + "cluster_id": 8, + "cluster_size": 5384, + "method": "TF-IDF + KMeans", + "silhouette": 0.0021, + "stability": { + "mean": 0.63, + "std": 0.259, + "min": 0.242, + "max": 0.916, + "stable": true + }, + "severity_score": 2.05, + "top_tfidf_terms": [ + "data", + "time", + "processing", + "real time", + "real", + "model" + ] + } + }, + { + "id": "PAT-8B2C9F", + "title": "Let Know / Know", + "description": "Cluster of 1,452 documents across 30 products. Cross-team entropy: 1.00. Author concentration (Gini): 0.43. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "LOW", + "confidence_score": 0.37, + "severity": "high", + "severity_reasoning": "Computed severity score: 1.5/5.0. Based on: customer density (0.000), cross-team entropy (1.00), author Gini (0.43), no-reply rate (100%).", + "top_phrases": [ + "let know", + "know", + "let", + "looking forward", + "forward", + "let know need" + ], + "occurrence": "1,452 of 37,064 documents (3.9%)", + "effect": "Spans 30 products, 377 authors, 546 days. Sources: {'slack': 1452}.", + "evidence": { + "source_count": 1452, + "source_breakdown": { + "slack": 1452 + }, + "sample_messages": [ + "Sounds good, @eid_fd8cecea. Looking forward to it! \ud83d\udc4d", + "Looking forward to it! Thanks for the great discussion, everyone. \ud83d\ude0a", + "On it too! Looking forward to seeing how you've tackled the integration with our existing systems. \ud83d\udc4d", + "Sounds good, @eid_13cb0e90. Once that's done, we can re-evaluate. Let us know if you need any help setting up the tests. \ud83d\udcaa" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [], + "time_range": "2026-01-16 to 2027-07-16" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.998, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.25, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.432, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 377, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 546, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.3, + "communication": 3.49, + "anomaly": 2.0 + }, + "sample_snippets": [ + "Sounds good, @eid_fd8cecea. Looking forward to it! \ud83d\udc4d", + "Looking forward to it! Thanks for the great discussion, everyone. \ud83d\ude0a" + ], + "caveats": "Confidence 37% = stability (57%) x silhouette (0.057). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 1,452 documents. Dominant metric signal: communication (3.5). Top phrases: let know, know, let.", + "computation": { + "cluster_id": 7, + "cluster_size": 1452, + "method": "TF-IDF + KMeans", + "silhouette": 0.0565, + "stability": { + "mean": 0.573, + "std": 0.177, + "min": 0.365, + "max": 0.959, + "stable": true + }, + "severity_score": 1.51, + "top_tfidf_terms": [ + "let know", + "know", + "let", + "looking forward", + "forward", + "let know need" + ] + } + }, + { + "id": "PAT-14537D", + "title": "Approved / Lgtm / Lgtm Approved", + "description": "Cluster of 1,226 documents across 30 products. Cross-team entropy: 0.99. Author concentration (Gini): 0.39. Customer references: 0.002/doc.", + "type": "communication", + "confidence": "LOW", + "confidence_score": 0.33, + "severity": "high", + "severity_reasoning": "Computed severity score: 2.0/5.0. Based on: customer density (0.002), cross-team entropy (0.99), author Gini (0.39), no-reply rate (100%).", + "top_phrases": [ + "approved", + "lgtm", + "lgtm approved", + "appreciate quick", + "appreciate", + "quick" + ], + "occurrence": "1,226 of 37,064 documents (3.3%)", + "effect": "Spans 30 products, 255 authors, 437 days. Sources: {'slack': 1226}.", + "evidence": { + "source_count": 1226, + "source_breakdown": { + "slack": 1226 + }, + "sample_messages": [ + "Thanks, @eid_4bcfb482 and Hannah! Appreciate the quick review and feedback. I'll merge it now. \ud83d\ude0a", + "Everything looks good to me. LGTM, approved! \u2705", + "Thanks, @Hannah Johnson and @Charlie Jones! Appreciate the quick review and feedback. I'll merge it now. \ud83d\ude4c", + "Yep, I can confirm that too. The integration looks seamless. LGTM, approved! \u2705" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [ + "TechCorp", + "CloudSync" + ], + "time_range": "2026-03-14 to 2027-05-26" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.989, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.25, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.388, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 255, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.002, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [ + "TechCorp", + "CloudSync" + ], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 437, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.02, + "bottleneck": 3.16, + "communication": 3.47, + "anomaly": 2.0 + }, + "sample_snippets": [ + "Thanks, @eid_4bcfb482 and Hannah! Appreciate the quick review and feedback. I'll merge it now. \ud83d\ude0a", + "Everything looks good to me. LGTM, approved! \u2705" + ], + "caveats": "Confidence 33% = stability (51%) x silhouette (0.055). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 1,226 documents. Dominant metric signal: communication (3.5). Top phrases: approved, lgtm, lgtm approved.", + "computation": { + "cluster_id": 4, + "cluster_size": 1226, + "method": "TF-IDF + KMeans", + "silhouette": 0.0546, + "stability": { + "mean": 0.508, + "std": 0.174, + "min": 0.072, + "max": 0.612, + "stable": true + }, + "severity_score": 1.99, + "top_tfidf_terms": [ + "approved", + "lgtm", + "lgtm approved", + "appreciate quick", + "appreciate", + "quick" + ] + } + }, + { + "id": "PAT-1E4237", + "title": "Joined / Joined Develop / Joined Planning", + "description": "Cluster of 891 documents across 30 products. Cross-team entropy: 0.99. Author concentration (Gini): 0.00. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "MEDIUM", + "confidence_score": 0.65, + "severity": "medium", + "severity_reasoning": "Computed severity score: 1.1/5.0. Based on: customer density (0.000), cross-team entropy (0.99), author Gini (0.00), no-reply rate (100%).", + "top_phrases": [ + "joined", + "joined develop", + "joined planning", + "develop", + "planning", + "joined invite" + ], + "occurrence": "891 of 37,064 documents (2.4%)", + "effect": "Spans 30 products, 1 authors, 540 days. Sources: {'slack': 891}.", + "evidence": { + "source_count": 891, + "source_breakdown": { + "slack": 891 + }, + "sample_messages": [ + "@eid_13fdff84 joined planning-onForceX. Also, @eid_d96fb219, @eid_2589c967, @eid_ec3b5d65 joined via invite.", + "@eid_4df5d4b7, @eid_8658e19c, @eid_fa6ec727, @eid_13cb0e90 joined planning-onForceX.", + "@eid_fd8cecea, @eid_eb1756e0 joined planning-onForceX.", + "@eid_5346d7a5 joined planning-onForceX." + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [], + "time_range": "2026-01-14 to 2027-07-08" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.991, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.25, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.0, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 1, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 540, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 2.0, + "communication": 3.47, + "anomaly": 2.0 + }, + "sample_snippets": [ + "@eid_13fdff84 joined planning-onForceX. Also, @eid_d96fb219, @eid_2589c967, @eid_ec3b5d65 joined via", + "@eid_4df5d4b7, @eid_8658e19c, @eid_fa6ec727, @eid_13cb0e90 joined planning-onForceX." + ], + "caveats": "Confidence 65% = stability (95%) x silhouette (0.188). Cluster stability tested across 10 bootstrap runs. Severity computed from metrics, not keyword matching.", + "recommendation": "Investigate 891 documents. Dominant metric signal: communication (3.5). Top phrases: joined, joined develop, joined planning.", + "computation": { + "cluster_id": 3, + "cluster_size": 891, + "method": "TF-IDF + KMeans", + "silhouette": 0.1882, + "stability": { + "mean": 0.95, + "std": 0.149, + "min": 0.502, + "max": 1.0, + "stable": true + }, + "severity_score": 1.1, + "top_tfidf_terms": [ + "joined", + "joined develop", + "joined planning", + "develop", + "planning", + "joined invite" + ] + } + }, + { + "id": "PAT-F39316", + "title": "Open Source / Source / Open", + "description": "Cluster of 930 documents across 30 products. Cross-team entropy: 0.99. Author concentration (Gini): 0.43. Customer references: 0.000/doc.", + "type": "communication", + "confidence": "MEDIUM", + "confidence_score": 0.57, + "severity": "medium", + "severity_reasoning": "Computed severity score: 1.3/5.0. Based on: customer density (0.000), cross-team entropy (0.99), author Gini (0.43), no-reply rate (100%).", + "top_phrases": [ + "open source", + "source", + "open", + "hi team", + "hi", + "team" + ], + "occurrence": "930 of 37,064 documents (2.5%)", + "effect": "Spans 30 products, 279 authors, 537 days. Sources: {'slack': 930}.", + "evidence": { + "source_count": 930, + "source_breakdown": { + "slack": 930 + }, + "sample_messages": [ + "Hi team, I've shared the for onForceX. Let's discuss and refine it together. Looking forward to you", + "Hi team, I wanted to discuss the Market Research Report on X-Genie. It's pretty interesting and I think we can learn a lot from it. \ud83d\udcca", + "Hi team, I've shared the for onForceX. Let's discuss and refine it to ensure it aligns with our s", + "Hi team, I wanted to discuss the X-Genie Product Vision Document. It's quite interesting and could have some implications for us. Thoughts? \ud83e\udd14" + ], + "affected_products": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce", + "CollaborationForce", + "ConnectForce", + "ContentForce", + "ContextForce", + "EdgeForce", + "ExplainabilityForce", + "FeedbackForce", + "FlowForce", + "ForecastForce", + "InsightForce", + "KnowledgeForce", + "LeadForce", + "MonitorForce", + "PersonalizeForce", + "PitchForce", + "ProposalForce", + "SearchFlow", + "SearchForce", + "SecurityForce", + "SentimentForce", + "SummarizeForce", + "SupportForce", + "TrendForce", + "VizForce", + "WorkFlowGenie" + ], + "affected_teams": [ + "ActionGenie", + "AnomalyForce", + "AutoTuneForce", + "CoachForce", + "CollaborateForce" + ], + "customers_referenced": [], + "time_range": "2026-01-18 to 2027-07-10" + }, + "metrics": { + "cross_team_entropy": { + "value": 0.995, + "method": "Shannon entropy of product distribution, normalized to [0,1]", + "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product" + }, + "source_diversity": { + "value": 0.25, + "method": "Unique source types / 4 (slack, transcript, document, PR)", + "interpretation": "1.0 = present in all source types" + }, + "no_reply_rate": { + "value": 1.0, + "method": "Fraction of Slack messages with 0 thread replies", + "interpretation": "Higher = less engagement / potential communication gap" + }, + "avg_thread_depth": { + "value": 0.0, + "method": "Mean thread reply count for Slack messages in cluster" + }, + "author_gini": { + "value": 0.434, + "method": "Gini coefficient of message authorship distribution", + "interpretation": "Higher = more concentrated (fewer authors dominating)" + }, + "unique_authors": { + "value": 279, + "method": "Count of distinct user IDs in cluster" + }, + "customer_density": { + "value": 0.0, + "method": "Customer name mentions per document in cluster" + }, + "customers_referenced": { + "value": [], + "method": "Customer names found via relational join with customer master data" + }, + "temporal_span_days": { + "value": 537, + "method": "Days between earliest and latest timestamped document" + }, + "reaction_rate": { + "value": 0.0, + "method": "Fraction of messages with emoji reactions" + }, + "product_count": { + "value": 30, + "method": "Distinct products represented in cluster" + } + }, + "metric_signals": { + "compliance": 1.0, + "bottleneck": 3.3, + "communication": 3.48, + "anomaly": 2.0 + }, + "sample_snippets": [ + "Hi team, I've shared the dict: + """Load all HERB data with relational metadata for joins.""" + t0 = time.time() + base = Path(base_path) + + with open(base / "metadata" / "customers_data.json") as f: + customers = json.load(f) + with open(base / "metadata" / "salesforce_team.json") as f: + team = json.load(f) + with open(base / "metadata" / "employee.json") as f: + employees_raw = json.load(f) + + # Build employee lookup for relational joins + # employee.json is a dict keyed by eid + employee_map = {} + if isinstance(employees_raw, dict): + employee_map = employees_raw + employees = list(employees_raw.values()) + else: + employees = employees_raw + for emp in employees: + if isinstance(emp, dict): + employee_map[emp.get("employee_id", emp.get("id", ""))] = emp + + # Build customer lookup + customer_map = {} + for cust in customers: + customer_map[cust.get("id", "")] = cust + customer_map[cust.get("name", "").lower()] = cust + + products = {} + for p in sorted((base / "products").iterdir()): + if p.suffix == ".json": + with open(p) as f: + products[p.stem] = json.load(f) + + documents = [] + for prod_name, prod_data in products.items(): + # Track which customers are associated with this product + prod_customers = set(prod_data.get("customers", [])) + + for msg in prod_data.get("slack", []): + user = msg.get("Message", {}).get("User", {}) + text = user.get("text", "") + if text and len(text) >= MIN_DOC_LENGTH and "created this channel" not in text: + uid = user.get("userId", "") + documents.append({ + "text": text, + "source": "slack", + "product": prod_name, + "channel": msg.get("Channel", {}).get("name", ""), + "user": uid, + "timestamp": user.get("timestamp", ""), + "thread_replies": len(msg.get("ThreadReplies", []) or []), + "has_reactions": bool(msg.get("Message", {}).get("Reactions")), + "employee_info": employee_map.get(uid), + "product_customers": prod_customers, + }) + + for t_item in prod_data.get("meeting_transcripts", []): + transcript = t_item.get("transcript", "") + if transcript and len(transcript) >= MIN_DOC_LENGTH: + attendees = [] + lines = transcript.split("\n") + if lines and lines[0].startswith("Attendees"): + attendees = [a.strip() for a in lines[1].split(",") if a.strip()] + documents.append({ + "text": transcript[:2000], + "source": "transcript", + "product": prod_name, + "channel": "", + "user": "", + "timestamp": "", + "thread_replies": 0, + "has_reactions": False, + "attendee_count": len(attendees), + "attendees": attendees, + "product_customers": prod_customers, + }) + + for doc in prod_data.get("documents", []): + content = doc.get("content", "") + if content and len(content) >= MIN_DOC_LENGTH: + documents.append({ + "text": content[:2000], + "source": "document", + "product": prod_name, + "channel": "", + "user": "", + "timestamp": "", + "thread_replies": 0, + "has_reactions": False, + "product_customers": prod_customers, + }) + + for pr in prod_data.get("prs", []): + title = pr.get("title", "") + summary = pr.get("summary", "") + combined = f"{title}. {summary}" if summary else title + if combined and len(combined) >= MIN_DOC_LENGTH: + documents.append({ + "text": combined[:500], + "source": "pull_request", + "product": prod_name, + "channel": "", + "user": "", + "timestamp": "", + "thread_replies": 0, + "has_reactions": False, + "product_customers": prod_customers, + }) + + ingest_time = time.time() - t0 + return { + "documents": documents, + "customers": customers, + "customer_map": customer_map, + "team": team, + "employees": employees, + "employee_map": employee_map, + "products": list(products.keys()), + "products_data": products, + "stats": { + "total_documents": len(documents), + "by_source": dict(Counter(d["source"] for d in documents)), + "by_product": dict(Counter(d["product"] for d in documents)), + "ingest_time_seconds": round(ingest_time, 3), + } + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# STAGE 2: NORMALIZE +# ═══════════════════════════════════════════════════════════════════════════ + +SF_ABBREVIATIONS = { + "SFDC": "Salesforce", "SOQL": "Salesforce Object Query Language", + "LWC": "Lightning Web Component", "SLDS": "Salesforce Lightning Design System", + "CPQ": "configure price quote", "MQL": "marketing qualified lead", + "ACV": "annual contract value", "ARR": "annual recurring revenue", + "TAM": "total addressable market", "NPS": "net promoter score", + "SLA": "service level agreement", "PII": "personally identifiable information", + "GDPR": "General Data Protection Regulation", "RBAC": "role-based access control", + "NLP": "natural language processing", "API": "application programming interface", + "MVP": "minimum viable product", "KPI": "key performance indicator", + "SWOT": "strengths weaknesses opportunities threats", "ROI": "return on investment", + "PR": "pull request", "CI/CD": "continuous integration continuous deployment", +} + +def normalize_text(text: str) -> str: + text = re.sub(r"<[^>]+\|([^>]+)>", r"\1", text) + text = re.sub(r"https?://\S+", "", text) + for abbr, expansion in SF_ABBREVIATIONS.items(): + text = re.sub(rf"\b{re.escape(abbr)}\b", f"{abbr} ({expansion})", text, count=1) + text = text.lower().strip() + text = re.sub(r"\s+", " ", text) + return text + + +# ═══════════════════════════════════════════════════════════════════════════ +# STAGE 3: VECTORIZE + CLUSTER +# ═══════════════════════════════════════════════════════════════════════════ + +def cluster_documents(documents: list[dict], n_clusters: int = N_CLUSTERS) -> dict: + t0 = time.time() + texts = [normalize_text(d["text"]) for d in documents] + + vectorizer = TfidfVectorizer( + max_features=TFIDF_MAX_FEATURES, ngram_range=TFIDF_NGRAM_RANGE, + stop_words="english", min_df=2, max_df=0.95, + ) + tfidf_matrix = vectorizer.fit_transform(texts) + feature_names = vectorizer.get_feature_names_out() + + # Optimal k via silhouette + k_range = range(max(2, n_clusters - 4), n_clusters + 5) + best_k, best_score = n_clusters, -1 + k_scores = {} + for k in k_range: + km = KMeans(n_clusters=k, random_state=42, n_init=10, max_iter=300) + labels = km.fit_predict(tfidf_matrix) + score = silhouette_score(tfidf_matrix, labels, sample_size=min(5000, len(texts))) + k_scores[k] = round(score, 4) + if score > best_score: + best_k, best_score = k, score + + # Final clustering + kmeans = KMeans(n_clusters=best_k, random_state=42, n_init=10, max_iter=300) + cluster_labels = kmeans.fit_predict(tfidf_matrix) + + # Per-sample silhouette scores for cluster quality + sample_silhouettes = silhouette_samples(tfidf_matrix, cluster_labels) + + # Top phrases per cluster + cluster_phrases = {} + cluster_docs = defaultdict(list) + cluster_silhouette_means = {} + for i, label in enumerate(cluster_labels): + cluster_docs[label].append(i) + + for label, doc_indices in cluster_docs.items(): + cluster_vectors = tfidf_matrix[doc_indices].toarray() + mean_scores = cluster_vectors.mean(axis=0) + top_indices = mean_scores.argsort()[-10:][::-1] + phrases = [feature_names[idx] for idx in top_indices if mean_scores[idx] > 0] + cluster_phrases[label] = phrases + cluster_silhouette_means[label] = round(float(np.mean(sample_silhouettes[doc_indices])), 4) + + cluster_time = time.time() - t0 + return { + "labels": cluster_labels.tolist(), + "n_clusters": best_k, + "silhouette_score": round(best_score, 4), + "k_scores": k_scores, + "cluster_sizes": dict(Counter(cluster_labels.tolist())), + "cluster_phrases": {str(k): v for k, v in cluster_phrases.items()}, + "cluster_docs": {str(k): v for k, v in cluster_docs.items()}, + "cluster_silhouette_means": {str(k): v for k, v in cluster_silhouette_means.items()}, + "tfidf_features": len(feature_names), + "tfidf_ngram_range": list(TFIDF_NGRAM_RANGE), + "cluster_time_seconds": round(cluster_time, 3), + "tfidf_matrix": tfidf_matrix, + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# STAGE 4: STABILITY — bootstrap cluster validation +# ═══════════════════════════════════════════════════════════════════════════ + +def compute_stability(tfidf_matrix, best_k: int, base_labels: list[int], + n_runs: int = STABILITY_RUNS) -> dict: + """Run clustering n_runs times with different seeds. + For each cluster, measure what fraction of its members stay together.""" + t0 = time.time() + n_docs = tfidf_matrix.shape[0] + base_arr = np.array(base_labels) + + # Build base cluster membership sets + base_clusters = defaultdict(set) + for i, label in enumerate(base_labels): + base_clusters[label].add(i) + + stability_scores = {} + run_details = [] + + for run in range(n_runs): + seed = run * 7 + 13 # Deterministic but varied seeds + km = KMeans(n_clusters=best_k, random_state=seed, n_init=5, max_iter=200) + alt_labels = km.fit_predict(tfidf_matrix) + + # For each base cluster, find the best-matching alt cluster (Jaccard similarity) + alt_clusters = defaultdict(set) + for i, label in enumerate(alt_labels): + alt_clusters[label].add(i) + + run_matches = {} + for base_label, base_members in base_clusters.items(): + best_jaccard = 0 + for alt_label, alt_members in alt_clusters.items(): + intersection = len(base_members & alt_members) + union = len(base_members | alt_members) + jaccard = intersection / union if union > 0 else 0 + if jaccard > best_jaccard: + best_jaccard = jaccard + run_matches[base_label] = round(best_jaccard, 3) + run_details.append(run_matches) + + # Average stability per cluster across all runs + for label in base_clusters: + scores = [rd.get(label, 0) for rd in run_details] + stability_scores[str(label)] = { + "mean": round(float(np.mean(scores)), 3), + "std": round(float(np.std(scores)), 3), + "min": round(float(np.min(scores)), 3), + "max": round(float(np.max(scores)), 3), + "stable": float(np.mean(scores)) >= STABILITY_THRESHOLD, + } + + stability_time = time.time() - t0 + return { + "scores": stability_scores, + "n_runs": n_runs, + "seeds_used": [r * 7 + 13 for r in range(n_runs)], + "threshold": STABILITY_THRESHOLD, + "stable_count": sum(1 for s in stability_scores.values() if s["stable"]), + "pruned_count": sum(1 for s in stability_scores.values() if not s["stable"]), + "duration_seconds": round(stability_time, 3), + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# STAGE 5: MEASURE — computed metrics replacing keyword heuristics +# ═══════════════════════════════════════════════════════════════════════════ + +def compute_cluster_metrics(documents: list[dict], doc_indices: list[int], + all_docs: list[dict], employee_map: dict, + customer_map: dict) -> dict: + """Compute MEASURED evidence-based metrics for a cluster. + No keyword matching — only counts, ratios, and statistical measures.""" + + cluster_docs = [documents[i] for i in doc_indices] + total_docs = len(all_docs) + cluster_size = len(doc_indices) + + # ── 1. Cross-team duplication frequency ── + products = Counter(d["product"] for d in cluster_docs) + n_products = len(products) + product_entropy = 0.0 + for count in products.values(): + p = count / cluster_size + if p > 0: + product_entropy -= p * math.log2(p) + max_entropy = math.log2(n_products) if n_products > 1 else 1 + cross_team_score = round(product_entropy / max_entropy, 3) if max_entropy > 0 else 0 + + # ── 2. Source diversity (multi-channel signal) ── + sources = Counter(d["source"] for d in cluster_docs) + n_sources = len(sources) + source_diversity = round(n_sources / 4.0, 2) # 4 possible sources + + # ── 3. Response latency proxy (thread reply density) ── + thread_counts = [d.get("thread_replies", 0) for d in cluster_docs if d["source"] == "slack"] + avg_thread_depth = round(np.mean(thread_counts), 2) if thread_counts else 0 + no_reply_rate = round(sum(1 for t in thread_counts if t == 0) / len(thread_counts), 3) if thread_counts else 0 + + # ── 4. Unique author concentration (knowledge silo risk) ── + authors = Counter(d["user"] for d in cluster_docs if d["user"]) + n_authors = len(authors) + author_gini = 0.0 + if n_authors > 1: + sorted_counts = sorted(authors.values()) + n = len(sorted_counts) + cumsum = np.cumsum(sorted_counts) + author_gini = round((2 * sum((i + 1) * c for i, c in enumerate(sorted_counts)) / + (n * sum(sorted_counts))) - (n + 1) / n, 3) + + # ── 5. Customer mention density ── + customer_mentions = 0 + customer_names_found = set() + for d in cluster_docs: + text_lower = d["text"].lower() + for cust in (d.get("product_customers") or []): + cid = customer_map.get(cust, {}) + if isinstance(cid, dict): + cname = cid.get("name", "").lower() + company = cid.get("company", "").lower() + if cname and cname in text_lower: + customer_mentions += 1 + customer_names_found.add(cid.get("company", cust)) + elif company and company in text_lower: + customer_mentions += 1 + customer_names_found.add(company) + customer_density = round(customer_mentions / cluster_size, 3) + + # ── 6. Temporal spread (longer = more systemic) ── + timestamps = [d["timestamp"] for d in cluster_docs if d["timestamp"]] + temporal_span_days = 0 + if len(timestamps) >= 2: + ts_sorted = sorted(timestamps) + try: + t_start = datetime.fromisoformat(ts_sorted[0].replace("Z", "+00:00")) + t_end = datetime.fromisoformat(ts_sorted[-1].replace("Z", "+00:00")) + temporal_span_days = (t_end - t_start).days + except (ValueError, TypeError): + pass + + # ── 7. Reaction rate (engagement signal) ── + reaction_docs = [d for d in cluster_docs if d.get("has_reactions")] + reaction_rate = round(len(reaction_docs) / cluster_size, 3) if cluster_size > 0 else 0 + + # ── COMPUTED SEVERITY (no keywords, only metrics) ── + # Score is a weighted sum of normalized metrics. + # Each metric contributes 0-1 to the total; weights reflect forensic importance. + severity_score = 0.0 + + # Customer data exposure: customer mentions in cross-team context + if customer_density > 0.05: + severity_score += min(customer_density * 10, 2.0) # Up to 2.0 + elif len(customer_names_found) > 0: + severity_score += 0.5 + + # Knowledge silo: high author concentration with few authors + if n_authors > 0: + silo_signal = author_gini * (1.0 if n_authors < 5 else 0.5) + severity_score += silo_signal # Up to ~1.0 + + # Cross-team spread: high entropy = systemic issue (not always bad, but worth noting) + if n_products >= 10: + severity_score += cross_team_score * 0.5 # Up to 0.5 for huge spread + elif n_products >= 5: + severity_score += cross_team_score * 0.3 + + # Source diversity: patterns appearing across multiple source types = more real + if source_diversity >= 0.75: + severity_score += 0.5 + + # Large cluster = more impactful + if cluster_size > 1000: + severity_score += 0.5 + elif cluster_size > 500: + severity_score += 0.3 + + # Temporal persistence + if temporal_span_days > 20: + severity_score += 0.3 + + # Scale to 5.0 max + severity_score = min(round(severity_score, 2), 5.0) + + if severity_score >= 2.5: + computed_severity = "critical" + elif severity_score >= 1.5: + computed_severity = "high" + elif severity_score >= 0.8: + computed_severity = "medium" + else: + computed_severity = "low" + + # ── COMPUTED TYPE (from dominant metric, not keywords) ── + metric_signals = { + "compliance": customer_density * 10 + (1 if cross_team_score > 0.5 else 0), + "bottleneck": author_gini * 3 + no_reply_rate * 2, + "communication": cross_team_score * 3 + source_diversity * 2, + "anomaly": (1 - reaction_rate) * 2 + (1 if temporal_span_days < 3 else 0), + } + computed_type = max(metric_signals, key=lambda k: metric_signals[k]) + + return { + "severity": computed_severity, + "type": computed_type, + "severity_score": round(severity_score, 2), + "metrics": { + "cross_team_entropy": {"value": cross_team_score, "method": "Shannon entropy of product distribution, normalized to [0,1]", "interpretation": "1.0 = evenly distributed across all products, 0.0 = single product"}, + "source_diversity": {"value": source_diversity, "method": "Unique source types / 4 (slack, transcript, document, PR)", "interpretation": "1.0 = present in all source types"}, + "no_reply_rate": {"value": no_reply_rate, "method": "Fraction of Slack messages with 0 thread replies", "interpretation": "Higher = less engagement / potential communication gap"}, + "avg_thread_depth": {"value": avg_thread_depth, "method": "Mean thread reply count for Slack messages in cluster"}, + "author_gini": {"value": author_gini, "method": "Gini coefficient of message authorship distribution", "interpretation": "Higher = more concentrated (fewer authors dominating)"}, + "unique_authors": {"value": n_authors, "method": "Count of distinct user IDs in cluster"}, + "customer_density": {"value": customer_density, "method": "Customer name mentions per document in cluster"}, + "customers_referenced": {"value": list(customer_names_found)[:5], "method": "Customer names found via relational join with customer master data"}, + "temporal_span_days": {"value": temporal_span_days, "method": "Days between earliest and latest timestamped document"}, + "reaction_rate": {"value": reaction_rate, "method": "Fraction of messages with emoji reactions"}, + "product_count": {"value": n_products, "method": "Distinct products represented in cluster"}, + }, + "metric_signals": {k: round(v, 2) for k, v in metric_signals.items()}, + "products": dict(products), + "sources": dict(sources), + "n_authors": n_authors, + "customer_names_found": list(customer_names_found)[:10], + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# STAGE 6: REPORT — assemble pattern cards with full provenance +# ═══════════════════════════════════════════════════════════════════════════ + +def build_cards(documents: list[dict], clustering: dict, stability: dict, + employee_map: dict, customer_map: dict) -> tuple[list[dict], list[dict], float]: + """Build pattern cards using computed metrics. Returns (cards, pruned, time).""" + t0 = time.time() + cards = [] + pruned = [] + + for label_str, doc_indices in clustering["cluster_docs"].items(): + label = int(label_str) + if len(doc_indices) < MIN_CLUSTER_SIZE: + pruned.append({"cluster_id": label, "size": len(doc_indices), "reason": f"below minimum size ({MIN_CLUSTER_SIZE})"}) + continue + + stab = stability["scores"].get(label_str, {}) + if not stab.get("stable", False): + pruned.append({"cluster_id": label, "size": len(doc_indices), + "reason": f"unstable (stability {stab.get('mean', 0):.0%} < {STABILITY_THRESHOLD:.0%} threshold)", + "stability": stab.get("mean", 0)}) + continue + + cluster_docs = [documents[i] for i in doc_indices] + phrases = clustering["cluster_phrases"].get(label_str, []) + cluster_sil = clustering["cluster_silhouette_means"].get(label_str, 0) + + # Compute evidence-based metrics + metrics_result = compute_cluster_metrics( + documents, doc_indices, documents, employee_map, customer_map + ) + + # Confidence = stability * silhouette quality + stability_mean = stab.get("mean", 0.5) + confidence_score = round(stability_mean * 0.6 + min(max(cluster_sil, 0), 1) * 0.4, 2) + confidence_score = min(confidence_score, 0.95) + confidence = "HIGH" if confidence_score >= 0.7 else "MEDIUM" if confidence_score >= 0.5 else "LOW" + + # Title from top phrases + title_phrases = [p for p in phrases[:3] if len(p) > 3] + title = " / ".join(w.title() for w in title_phrases[:3]) if title_phrases else f"Cluster {label}" + + # Time range + timestamps = [d["timestamp"] for d in cluster_docs if d["timestamp"]] + time_range = "" + if timestamps: + ts_sorted = sorted(timestamps) + time_range = f"{ts_sorted[0][:10]} to {ts_sorted[-1][:10]}" + + # Sample messages + samples = [d["text"][:200] for d in cluster_docs if d["source"] == "slack"][:4] + if not samples: + samples = [d["text"][:200] for d in cluster_docs[:3]] + + card = { + "id": f"PAT-{hash(title + str(len(doc_indices))) % 0xFFFFFF:06X}", + "title": title, + "description": ( + f"Cluster of {len(doc_indices):,} documents across {metrics_result['metrics']['product_count']['value']} products. " + f"Cross-team entropy: {metrics_result['metrics']['cross_team_entropy']['value']:.2f}. " + f"Author concentration (Gini): {metrics_result['metrics']['author_gini']['value']:.2f}. " + f"Customer references: {metrics_result['metrics']['customer_density']['value']:.3f}/doc." + ), + "type": metrics_result["type"], + "confidence": confidence, + "confidence_score": confidence_score, + "severity": metrics_result["severity"], + "severity_reasoning": ( + f"Computed severity score: {metrics_result['severity_score']:.1f}/5.0. " + f"Based on: customer density ({metrics_result['metrics']['customer_density']['value']:.3f}), " + f"cross-team entropy ({metrics_result['metrics']['cross_team_entropy']['value']:.2f}), " + f"author Gini ({metrics_result['metrics']['author_gini']['value']:.2f}), " + f"no-reply rate ({metrics_result['metrics']['no_reply_rate']['value']:.0%})." + ), + "top_phrases": phrases[:6], + "occurrence": f"{len(doc_indices):,} of {len(documents):,} documents ({len(doc_indices)/len(documents):.1%})", + "effect": ( + f"Spans {metrics_result['metrics']['product_count']['value']} products, " + f"{metrics_result['n_authors']} authors, " + f"{metrics_result['metrics']['temporal_span_days']['value']} days. " + f"Sources: {dict(metrics_result['sources'])}." + ), + "evidence": { + "source_count": len(doc_indices), + "source_breakdown": metrics_result["sources"], + "sample_messages": samples, + "affected_products": list(metrics_result["products"].keys()), + "affected_teams": list(metrics_result["products"].keys())[:5], + "customers_referenced": metrics_result["customer_names_found"], + "time_range": time_range or "Not timestamped", + }, + "metrics": metrics_result["metrics"], + "metric_signals": metrics_result["metric_signals"], + "sample_snippets": [s[:100] for s in samples[:2]], + "caveats": ( + f"Confidence {confidence_score:.0%} = stability ({stability_mean:.0%}) x silhouette ({cluster_sil:.3f}). " + f"Cluster stability tested across {STABILITY_RUNS} bootstrap runs. " + f"Severity computed from metrics, not keyword matching." + ), + "recommendation": ( + f"Investigate {len(doc_indices):,} documents. " + f"Dominant metric signal: {max(metrics_result['metric_signals'], key=lambda k: metrics_result['metric_signals'][k])} " + f"({max(metrics_result['metric_signals'].values()):.1f}). " + f"Top phrases: {', '.join(phrases[:3])}." + ), + "computation": { + "cluster_id": label, + "cluster_size": len(doc_indices), + "method": "TF-IDF + KMeans", + "silhouette": cluster_sil, + "stability": stab, + "severity_score": metrics_result["severity_score"], + "top_tfidf_terms": phrases[:6], + } + } + cards.append(card) + + sev_order = {"critical": 0, "high": 1, "medium": 2, "low": 3} + cards.sort(key=lambda c: (sev_order.get(c["severity"], 9), -c["confidence_score"])) + + build_time = time.time() - t0 + return cards, pruned, round(build_time, 3) + + +def generate_report(ingest_result: dict, clustering: dict, stability: dict, + cards: list[dict], pruned: list[dict], build_time: float) -> dict: + # Remove non-serializable tfidf_matrix + clustering_clean = {k: v for k, v in clustering.items() if k != "tfidf_matrix"} + + return { + "metadata": { + "generated_at": datetime.now().isoformat(), + "author": "Christopher Bailey", + "engine": "Transaction Forensics Pattern Engine v2.0", + "source": "Salesforce/HERB", + "source_url": "https://huggingface.co/datasets/Salesforce/HERB", + "license": "CC-BY-NC-4.0", + "products_analyzed": ingest_result["products"], + "total_messages_scanned": ingest_result["stats"]["by_source"].get("slack", 0), + "total_documents_scanned": ingest_result["stats"]["by_source"].get("document", 0), + "total_transcripts_scanned": ingest_result["stats"]["by_source"].get("transcript", 0), + "total_prs_analyzed": ingest_result["stats"]["by_source"].get("pull_request", 0), + "total_customers": len(ingest_result["customers"]), + "total_team_members": len(ingest_result["team"]), + }, + "pipeline": { + "version": "2.0", + "stages": [ + {"name": "Ingest", "duration_seconds": ingest_result["stats"]["ingest_time_seconds"], + "documents_loaded": ingest_result["stats"]["total_documents"], + "sources": ingest_result["stats"]["by_source"], + "relational_joins": "employee_map, customer_map loaded for cross-referencing"}, + {"name": "Normalize", "description": f"Text cleaning + {len(SF_ABBREVIATIONS)} abbreviation expansions + URL/mention stripping"}, + {"name": "Vectorize", "method": "TF-IDF", + "features": clustering_clean["tfidf_features"], + "ngram_range": clustering_clean["tfidf_ngram_range"], + "max_features": TFIDF_MAX_FEATURES}, + {"name": "Cluster", "method": "KMeans", + "optimal_k": clustering_clean["n_clusters"], + "silhouette_score": clustering_clean["silhouette_score"], + "k_scores": clustering_clean.get("k_scores", {}), + "per_cluster_silhouette": clustering_clean.get("cluster_silhouette_means", {}), + "duration_seconds": clustering_clean["cluster_time_seconds"], + "cluster_sizes": clustering_clean["cluster_sizes"]}, + {"name": "Stabilize", "method": f"Bootstrap ({STABILITY_RUNS} runs, Jaccard similarity)", + "threshold": STABILITY_THRESHOLD, + "stable_clusters": stability["stable_count"], + "pruned_clusters": stability["pruned_count"], + "duration_seconds": stability["duration_seconds"]}, + {"name": "Measure", "method": "Evidence-based scoring (NO keyword heuristics)", + "metrics_computed": ["cross_team_entropy", "source_diversity", "no_reply_rate", + "author_gini", "customer_density", "temporal_span_days", "reaction_rate"], + "severity_method": "Weighted metric combination (customer_density x cross_team, author_gini, no_reply_rate)", + "type_method": "Dominant metric signal (highest computed score wins)", + "patterns_surfaced": len(cards), + "patterns_pruned": len(pruned), + "duration_seconds": build_time}, + ], + "total_duration_seconds": round( + ingest_result["stats"]["ingest_time_seconds"] + + clustering_clean["cluster_time_seconds"] + + stability["duration_seconds"] + build_time, 3 + ), + "pruned_clusters": pruned, + }, + "cards": cards, + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# MAIN +# ═══════════════════════════════════════════════════════════════════════════ + +def main(): + print("=" * 60) + print("Transaction Forensics — Pattern Engine v3.0") + print("Author: Christopher Bailey") + print("Data: Salesforce/HERB (HuggingFace)") + print("=" * 60) + + # ── Stage 1: Ingest ── + print("\n[1/8] Ingesting HERB dataset...") + ingest_result = ingest_herb(HERB_BASE) + stats = ingest_result["stats"] + print(f" {stats['total_documents']:,} documents in {stats['ingest_time_seconds']}s") + print(f" Sources: {stats['by_source']}") + print(f" Products: {len(ingest_result['products'])}") + print(f" Employees: {len(ingest_result['employees'])}, Customers: {len(ingest_result['customers'])}") + + # ── Stage 2: Normalize ── + print(f"\n[2/8] Normalizing ({len(SF_ABBREVIATIONS)} abbreviation expansions)...") + texts = [normalize_text(d["text"]) for d in ingest_result["documents"]] + + # ── Stage 3: BERTopic Clustering (with TF-IDF fallback) ── + bertopic_result = None + try: + from bertopic_cluster import cluster_with_bertopic + print(f"\n[3/8] BERTopic clustering (SBERT + HDBSCAN)...") + bertopic_result = cluster_with_bertopic(texts, min_topic_size=20) + print(f" Topics: {bertopic_result['n_topics']} (outliers: {bertopic_result['outlier_count']})") + print(f" Silhouette: {bertopic_result['silhouette_score']}") + print(f" Time: {bertopic_result['duration_seconds']}s") + for tid, info in list(bertopic_result['topic_representations'].items())[:5]: + print(f" Topic {tid}: {', '.join(info[:5])}") + except Exception as e: + print(f"\n[3/8] BERTopic failed ({e}), falling back to TF-IDF + KMeans...") + + # TF-IDF + KMeans (always run for stability analysis) + print(f"\n[4/8] TF-IDF + KMeans clustering (for stability baseline)...") + clustering = cluster_documents(ingest_result["documents"]) + print(f" Features: {clustering['tfidf_features']}, k={clustering['n_clusters']} (silhouette: {clustering['silhouette_score']})") + + # ── Stage 4: Network Analysis ── + network_result = None + try: + from network_analysis import build_communication_graph + print(f"\n[5/8] Network analysis (communication graph)...") + network_result = build_communication_graph(ingest_result["documents"]) + print(f" Nodes: {network_result.get('n_nodes', '?')}, Edges: {network_result.get('n_edges', '?')}") + print(f" Communities: {network_result.get('n_communities', '?')}") + print(f" Density: {network_result.get('graph_density', '?')}") + if network_result.get('bridge_users'): + print(f" Bridge users: {[u[:15] for u in network_result['bridge_users'][:3]]}") + if network_result.get('isolated_products'): + print(f" Isolated products: {network_result['isolated_products'][:3]}") + except Exception as e: + print(f"\n[5/8] Network analysis failed: {e}") + + # ── Stage 5: Temporal Analysis ── + temporal_result = None + try: + from temporal_analysis import analyze_temporal_patterns + print(f"\n[6/8] Temporal change-point detection...") + temporal_result = analyze_temporal_patterns(ingest_result["documents"]) + print(f" Window: {temporal_result.get('activity_windows', {}).get('total_days', '?')} days") + print(f" Change points: {len(temporal_result.get('change_points', []))}") + if temporal_result.get('busiest_day'): + print(f" Busiest day: {temporal_result['busiest_day']}") + except Exception as e: + print(f"\n[6/8] Temporal analysis failed: {e}") + + # ── Stage 6: Bootstrap Stability ── + print(f"\n[7/8] Bootstrap stability ({STABILITY_RUNS} runs)...") + stability = compute_stability(clustering["tfidf_matrix"], clustering["n_clusters"], clustering["labels"]) + print(f" Stable: {stability['stable_count']}, Pruned: {stability['pruned_count']}") + + # ── Stage 7: Measure + Build Cards ── + print(f"\n[8/8] Computing evidence-based metrics + building cards...") + cards, pruned, build_time = build_cards( + ingest_result["documents"], clustering, stability, + ingest_result["employee_map"], ingest_result["customer_map"] + ) + print(f" {len(cards)} patterns surfaced, {len(pruned)} pruned") + for c in cards: + print(f" {c['severity']:8} | {c['type']:12} | {c['confidence']} ({c['confidence_score']:.0%}) | {c['title'][:45]}") + + # ── Generate Report ── + report = generate_report(ingest_result, clustering, stability, cards, pruned, build_time) + + # Add BERTopic results to report + if bertopic_result: + # Don't serialize embeddings (numpy array) + bt_clean = {k: v for k, v in bertopic_result.items() if k != 'embeddings'} + report["pipeline"]["bertopic"] = bt_clean + report["pipeline"]["stages"].insert(2, { + "name": "BERTopic", + "method": "Sentence-Transformers (all-MiniLM-L6-v2) + HDBSCAN", + "n_topics": bertopic_result["n_topics"], + "outliers": bertopic_result["outlier_count"], + "silhouette_score": bertopic_result["silhouette_score"], + "duration_seconds": bertopic_result["duration_seconds"], + "top_topics": {str(k): v[:5] for k, v in list(bertopic_result["topic_representations"].items())[:8]}, + }) + + # Add network results + if network_result: + # Don't serialize full centrality dicts (too large) + net_summary = { + "n_nodes": network_result.get("n_nodes"), + "n_edges": network_result.get("n_edges"), + "n_communities": network_result.get("n_communities"), + "graph_density": network_result.get("graph_density"), + "bridge_users": network_result.get("bridge_users", [])[:5], + "isolated_products": network_result.get("isolated_products", []), + "duration_seconds": network_result.get("duration_seconds"), + } + # Product overlap for top pairs + overlap = network_result.get("product_overlap_matrix", {}) + top_overlaps = sorted(overlap.items(), key=lambda x: x[1], reverse=True)[:10] + net_summary["top_product_overlaps"] = {k: v for k, v in top_overlaps} + + report["pipeline"]["network"] = net_summary + report["pipeline"]["stages"].append({ + "name": "Network", + "method": "Communication graph (NetworkX), Louvain communities, centrality analysis", + "n_nodes": net_summary["n_nodes"], + "n_edges": net_summary["n_edges"], + "communities": net_summary["n_communities"], + "bridge_users": net_summary["bridge_users"], + "density": net_summary["graph_density"], + }) + + # Add temporal results + if temporal_result: + report["pipeline"]["temporal"] = temporal_result + report["pipeline"]["stages"].append({ + "name": "Temporal", + "method": "Change-point detection (ruptures PELT, RBF kernel)", + "total_days": temporal_result.get("activity_windows", {}).get("total_days"), + "change_points": len(temporal_result.get("change_points", [])), + "busiest_day": temporal_result.get("busiest_day"), + }) + + # Recalculate total duration + total_time = sum( + s.get("duration_seconds", 0) for s in report["pipeline"]["stages"] + if isinstance(s.get("duration_seconds"), (int, float)) + ) + report["pipeline"]["total_duration_seconds"] = round(total_time, 3) + report["metadata"]["engine"] = "Transaction Forensics Pattern Engine v3.0" + + output_path = OUTPUT_DIR / "pattern_cards.json" + with open(output_path, "w") as f: + json.dump(report, f, indent=2) + + print(f"\n Output: {output_path}") + print(f" Total: {report['pipeline']['total_duration_seconds']}s") + print("=" * 60) + print("Done.") + + +if __name__ == "__main__": + main() diff --git a/herb-nlp/bertopic_cluster.py b/herb-nlp/bertopic_cluster.py new file mode 100644 index 0000000..7faed36 --- /dev/null +++ b/herb-nlp/bertopic_cluster.py @@ -0,0 +1,113 @@ +"""BERTopic-based document clustering for transaction forensics.""" +from __future__ import annotations + +import time +from typing import Any + +import numpy as np +from bertopic import BERTopic +from sentence_transformers import SentenceTransformer +from sklearn.metrics import silhouette_score as sk_silhouette_score + + +def cluster_with_bertopic( + texts: list[str], min_topic_size: int = 20 +) -> dict[str, Any]: + """Cluster documents using BERTopic with sentence-transformer embeddings. + + Args: + texts: List of document strings to cluster. + min_topic_size: Minimum number of documents per topic. + + Returns: + Dict with topics, topic_info, n_topics, outlier_count, + silhouette_score, topic_representations, embeddings, duration_seconds. + """ + start = time.time() + + if len(texts) < min_topic_size: + print(f"[bertopic] Only {len(texts)} docs (< min_topic_size={min_topic_size}). Returning all as outliers.") + embeddings = SentenceTransformer("all-MiniLM-L6-v2").encode(texts, show_progress_bar=True) + return { + "topics": [-1] * len(texts), + "topic_info": [{"topic_id": -1, "count": len(texts), "name": "Outlier", "representation": []}], + "n_topics": 0, + "outlier_count": len(texts), + "silhouette_score": -1.0, + "topic_representations": {}, + "embeddings": np.array(embeddings), + "duration_seconds": round(time.time() - start, 2), + } + + print("[bertopic] Encoding documents with all-MiniLM-L6-v2 ...") + embedding_model = SentenceTransformer("all-MiniLM-L6-v2") + embeddings = embedding_model.encode(texts, show_progress_bar=True) + + print(f"[bertopic] Fitting BERTopic (min_topic_size={min_topic_size}) ...") + topic_model = BERTopic( + embedding_model=embedding_model, + min_topic_size=min_topic_size, + verbose=True, + ) + + try: + topics, _probs = topic_model.fit_transform(texts, embeddings=embeddings) + except Exception as exc: + print(f"[bertopic] BERTopic failed: {exc}. Returning all as outliers.") + return { + "topics": [-1] * len(texts), + "topic_info": [{"topic_id": -1, "count": len(texts), "name": "Outlier", "representation": []}], + "n_topics": 0, + "outlier_count": len(texts), + "silhouette_score": -1.0, + "topic_representations": {}, + "embeddings": np.array(embeddings), + "duration_seconds": round(time.time() - start, 2), + } + + # Build topic info from BERTopic's internal table + info_df = topic_model.get_topic_info() + topic_info = [] + for _, row in info_df.iterrows(): + tid = int(row["Topic"]) + words = topic_model.get_topic(tid) + top_words = [w for w, _ in words[:10]] if words and words != -1 else [] + topic_info.append({ + "topic_id": tid, + "count": int(row["Count"]), + "name": str(row.get("Name", f"Topic_{tid}")), + "representation": top_words, + }) + + # Topic representations dict (exclude outlier topic -1) + topic_representations = {} + for entry in topic_info: + if entry["topic_id"] != -1: + topic_representations[entry["topic_id"]] = entry["representation"] + + topics_arr = np.array(topics) + outlier_count = int((topics_arr == -1).sum()) + n_topics = len(set(topics) - {-1}) + + # Silhouette score on non-outlier documents + sil_score = -1.0 + mask = topics_arr != -1 + if mask.sum() > 1 and n_topics > 1: + try: + sil_score = float(sk_silhouette_score(embeddings[mask], topics_arr[mask])) + except Exception: + pass + + duration = round(time.time() - start, 2) + print(f"[bertopic] Done. {n_topics} topics, {outlier_count} outliers, silhouette={sil_score:.3f}, {duration}s") + + return { + "topics": [int(t) for t in topics], + "topic_info": topic_info, + "n_topics": n_topics, + "outlier_count": outlier_count, + "silhouette_score": sil_score, + "topic_representations": topic_representations, + "embeddings": np.array(embeddings), + "duration_seconds": duration, + } diff --git a/herb-nlp/network_analysis.py b/herb-nlp/network_analysis.py new file mode 100644 index 0000000..633494e --- /dev/null +++ b/herb-nlp/network_analysis.py @@ -0,0 +1,109 @@ +"""Network Analysis — Communication graph and metrics from enterprise message data. +Detects communities, identifies bridge users, and finds product silos. +Standalone module. No imports from analyze.py. +""" +from __future__ import annotations + +import time +from collections import defaultdict +from itertools import combinations +from typing import Dict, List, Optional, Set, Tuple + +import networkx as nx +from networkx.algorithms.community import louvain_communities + + +def build_communication_graph(documents: List[Dict]) -> Dict: + """Build a communication graph from enterprise message documents. + Each document has: user, product, channel, timestamp, text. + Returns dict with centrality, communities, bridge users, product overlaps, etc. + """ + t0 = time.time() + if not documents: + return _empty_result(time.time() - t0) + + # --- Collect relationships --- + channel_users: Dict[str, Set[str]] = defaultdict(set) + user_products: Dict[str, Set[str]] = defaultdict(set) + chan_msgs: Dict[Tuple[str, str], int] = defaultdict(int) + all_users: Set[str] = set() + for doc in documents: + user = doc.get("user") + if not user: + continue + all_users.add(user) + channel, product = doc.get("channel"), doc.get("product") + if channel: + channel_users[channel].add(user) + chan_msgs[(user, channel)] += 1 + if product: + user_products[user].add(product) + + if len(all_users) < 2: + return _empty_result(time.time() - t0, users=all_users) + + # --- Build user graph (edges = shared channels, weight = combined msg count) --- + G = nx.Graph() + G.add_nodes_from(all_users) + edge_weights: Dict[Tuple[str, str], int] = defaultdict(int) + for channel, users in channel_users.items(): + su = sorted(users) + for i, u1 in enumerate(su): + for u2 in su[i + 1:]: + edge_weights[(u1, u2)] += chan_msgs[(u1, channel)] + chan_msgs[(u2, channel)] + for (u1, u2), w in edge_weights.items(): + G.add_edge(u1, u2, weight=w) + + # --- Centrality & communities --- + degree_cent = nx.degree_centrality(G) + betweenness_cent = nx.betweenness_centrality(G, weight="weight") + if G.number_of_edges() > 0: + communities = [set(c) for c in louvain_communities(G, weight="weight", seed=42)] + else: + communities = [{u} for u in all_users] + + # --- Bridge users (top-5 betweenness, excluding zero) --- + sorted_bw = sorted(betweenness_cent.items(), key=lambda x: x[1], reverse=True) + bridge_users = [u for u, s in sorted_bw[:5] if s > 0] + + # --- Product overlap & isolated products --- + all_products: Set[str] = set() + prod_users: Dict[str, Set[str]] = defaultdict(set) + for user, prods in user_products.items(): + all_products.update(prods) + for p in prods: + prod_users[p].add(user) + product_overlap: Dict[str, int] = {} + connected: Set[str] = set() + for p1, p2 in combinations(sorted(all_products), 2): + shared = len(prod_users[p1] & prod_users[p2]) + if shared > 0: + product_overlap[f"{p1}|{p2}"] = shared + connected.update((p1, p2)) + + return { + "n_nodes": G.number_of_nodes(), + "n_edges": G.number_of_edges(), + "degree_centrality": degree_cent, + "betweenness_centrality": betweenness_cent, + "communities": communities, + "n_communities": len(communities), + "bridge_users": bridge_users, + "isolated_products": sorted(all_products - connected), + "product_overlap_matrix": product_overlap, + "graph_density": nx.density(G), + "duration_seconds": round(time.time() - t0, 4), + } + + +def _empty_result(elapsed: float, users: Optional[Set[str]] = None) -> Dict: + """Zeroed-out result for empty or single-user input.""" + u = users or set() + cent = {x: 0.0 for x in u} + return { + "n_nodes": len(u), "n_edges": 0, + "degree_centrality": dict(cent), "betweenness_centrality": dict(cent), + "communities": [set(u)] if u else [], "n_communities": 1 if u else 0, + "bridge_users": [], "isolated_products": [], "product_overlap_matrix": {}, + "graph_density": 0.0, "duration_seconds": round(elapsed, 4), + } diff --git a/herb-nlp/requirements.txt b/herb-nlp/requirements.txt new file mode 100644 index 0000000..fd00834 --- /dev/null +++ b/herb-nlp/requirements.txt @@ -0,0 +1,10 @@ +scikit-learn>=1.3.0 +pandas>=2.0.0 +numpy>=1.24.0 +datasets>=2.14.0 +bertopic>=0.17.0 +sentence-transformers>=5.0.0 +hdbscan>=0.8.0 +umap-learn>=0.5.0 +networkx>=3.0 +ruptures>=1.1.0 diff --git a/herb-nlp/temporal_analysis.py b/herb-nlp/temporal_analysis.py new file mode 100644 index 0000000..4bda2cf --- /dev/null +++ b/herb-nlp/temporal_analysis.py @@ -0,0 +1,92 @@ +"""Temporal change-point detection for enterprise communication data.""" +from __future__ import annotations + +import time +from collections import Counter, defaultdict +from datetime import datetime, timedelta +from typing import Any + + +def analyze_temporal_patterns(documents: list[dict]) -> dict: + """Analyze temporal patterns in communication documents with change-point detection.""" + start = time.time() + empty = { + "daily_volume": [], "change_points": [], "per_product_trends": {}, + "activity_windows": {"earliest": "", "latest": "", "total_days": 0, "active_days": 0}, + "busiest_day": {"date": "", "count": 0}, "quietest_day": {"date": "", "count": 0}, + "duration_seconds": 0.0, + } + # Parse timestamps, skipping invalid ones + parsed: list[tuple[datetime, dict]] = [] + for doc in documents: + try: + parsed.append((datetime.fromisoformat(doc.get("timestamp", "")), doc)) + except (ValueError, TypeError): + continue + if not parsed: + empty["duration_seconds"] = round(time.time() - start, 4) + return empty + + parsed.sort(key=lambda x: x[0]) + earliest, latest = parsed[0][0], parsed[-1][0] + total_days = (latest.date() - earliest.date()).days + 1 + date_range = [(earliest.date() + timedelta(days=i)).strftime("%Y-%m-%d") for i in range(total_days)] + + # Build daily and per-product counts + daily_counts: Counter[str] = Counter() + product_daily: dict[str, Counter[str]] = defaultdict(Counter) + for ts, doc in parsed: + day = ts.strftime("%Y-%m-%d") + daily_counts[day] += 1 + product_daily[doc.get("product", "unknown")][day] += 1 + + all_days = sorted(daily_counts) + daily_volume = [{"date": d, "count": daily_counts[d]} for d in all_days] + signal = [float(daily_counts.get(d, 0)) for d in date_range] + + # Change-point detection via ruptures + change_points: list[dict[str, Any]] = [] + try: + import numpy as np + import ruptures as rpt + if len(signal) >= 3: + algo = rpt.Pelt(model="rbf").fit(np.array(signal).reshape(-1, 1)) + for cp in algo.predict(pen=10): + if cp >= len(date_range): + continue + before = np.mean(signal[max(0, cp - 3):cp]) if cp > 0 else 0 + after = np.mean(signal[cp:min(len(signal), cp + 3)]) if cp < len(signal) else 0 + mag = after - before + change_points.append({ + "date": date_range[min(cp, len(date_range) - 1)], + "type": "increase" if mag >= 0 else "decrease", + "magnitude": round(abs(mag), 2), + }) + except Exception: + pass + + # Per-product trends (first-half vs second-half average comparison) + per_product_trends: dict[str, dict] = {} + for product, counts in product_daily.items(): + total = sum(counts.values()) + vals = [counts.get(d, 0) for d in date_range] + mid = len(vals) // 2 + diff = (sum(vals[mid:]) / max(len(vals) - mid, 1)) - (sum(vals[:mid]) / max(mid, 1)) + per_product_trends[product] = { + "total": total, "daily_avg": round(total / max(total_days, 1), 2), + "peak_day": max(counts, key=counts.get), # type: ignore[arg-type] + "trend": "increasing" if diff > 0.5 else ("decreasing" if diff < -0.5 else "stable"), + } + + busiest = max(daily_volume, key=lambda x: x["count"]) + quietest = min(daily_volume, key=lambda x: x["count"]) + return { + "daily_volume": daily_volume, "change_points": change_points, + "per_product_trends": per_product_trends, + "activity_windows": { + "earliest": earliest.strftime("%Y-%m-%d"), "latest": latest.strftime("%Y-%m-%d"), + "total_days": total_days, "active_days": len(all_days), + }, + "busiest_day": busiest, "quietest_day": quietest, + "duration_seconds": round(time.time() - start, 4), + }