Skip to content

📚 Automated Documentation Update #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
github-actions wants to merge 7 commits into
base: master
Choose a base branch
from
+11,674 −11,575
Open

📚 Automated Documentation Update #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0b82b5a
docs: Initialize documentation pipeline [skip ci]
github-actions[bot] Jun 4, 2026
53b47f6
chore(docs): Clean slate - remove all documentation (101 files) [skip…
github-actions[bot] Jun 4, 2026
485a40a
docs: Stage 1 - Inline documentation (235 files) [skip ci]
github-actions[bot] Jun 4, 2026
43fa950
chore(docs): Remove 1 orphaned inline files [skip ci]
github-actions[bot] Jun 4, 2026
a6c6924
docs: Stage 2 - Architecture analysis (14 files) [skip ci]
github-actions[bot] Jun 4, 2026
fe891bb
docs: Stage 3 - Tutorial generation (9 files) [skip ci]
github-actions[bot] Jun 4, 2026
3a7033d
docs: Stage 4 - Repository documentation (4 files) [skip ci]
github-actions[bot] Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
473 changes: 226 additions & 247 deletions CONTRIBUTING.md
View file
Open in desktop

Large diffs are not rendered by default.

304 changes: 153 additions & 151 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

148 changes: 81 additions & 67 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# osquery — OpenFrame Edition Documentation
# osquery (OpenFrame-Enhanced) — Documentation

Welcome to the documentation for **osquery with OpenFrame** — the cross-platform OS instrumentation framework extended with AI-driven MSP automation by [Flamingo](https://flamingo.run) and [OpenFrame](https://openframe.ai).
Welcome to the documentation for the OpenFrame-enhanced distribution of **osquery**, a cross-platform OS instrumentation framework that exposes system state as relational data using SQL. This distribution is part of the [Flamingo](https://flamingo.run) / [OpenFrame](https://openframe.ai) platform for intelligent MSP automation.

---

Expand All @@ -11,108 +11,122 @@ Welcome to the documentation for **osquery with OpenFrame** — the cross-platfo
- [Reference Architecture](#-reference-architecture)
- [Architecture Diagrams](#-architecture-diagrams)
- [Quick Links](#-quick-links)
- [Community](#-community)

---

## 🚀 Getting Started

New to osquery? Start here.

| Guide | Description |
|---|---|
| [Introduction](./getting-started/introduction.md) | What is osquery with OpenFrame? Features and target audience |
| [Prerequisites](./getting-started/prerequisites.md) | System requirements, supported platforms, required software |
| [Quick Start](./getting-started/quick-start.md) | Clone, build, and run osquery in under 10 minutes |
| [First Steps](./getting-started/first-steps.md) | Explore virtual tables, scheduled packs, FIM, extensions, and OpenFrame |

**Recommended reading order:** Introduction → Prerequisites → Quick Start → First Steps
| Document | Description |
|----------|-------------|
| [Introduction](./getting-started/introduction.md) | What is osquery, key features, OpenFrame extensions, and architecture overview |
| [Prerequisites](./getting-started/prerequisites.md) | System requirements, required build tools, and environment verification |
| [Quick Start](./getting-started/quick-start.md) | Clone, build, and run osquery in minutes |
| [First Steps](./getting-started/first-steps.md) | Explore tables, run security queries, create query packs, and configure OpenFrame integration |

---

## 🛠 Development

Guides for contributors and developers building on or extending osquery.
Guides for contributors and developers working on the codebase.

### Setup
| Document | Description |
|----------|-------------|
| [Development Overview](./development/README.md) | Project structure, technology stack, and navigation index |
| [Environment Setup](./development/setup/environment.md) | IDE configuration (CLion, VS Code), compiler setup, ccache, clang-format |
| [Local Development](./development/setup/local-development.md) | Build modes, running locally, incremental builds, debugging with LLDB/GDB |
| [Architecture Overview](./development/architecture/README.md) | High-level system design, component diagrams, data flow, OpenFrame auth pipeline |
| [Security Best Practices](./development/security/README.md) | Auth patterns, AES-256-GCM encryption, secrets management, TLS configuration |

| Guide | Description |
|---|---|
| [Environment Setup](./development/setup/environment.md) | IDE recommendations, clangd, ccache, editor extensions |
| [Local Development](./development/setup/local-development.md) | Build configurations, debug flags, GDB/LLDB, VS Code debugging |
---

### Architecture
## 📖 Reference Architecture

| Guide | Description |
|---|---|
| [Architecture Overview](./development/architecture/README.md) | High-level module breakdown, runtime lifecycle, design decisions |
Detailed technical documentation for each core module, auto-generated from source code analysis.

### Quality
### Core Runtime

| Guide | Description |
|---|---|
| [Testing Guide](./development/testing/README.md) | GTest/GMock structure, running tests, writing unit and integration tests |
| [Security Best Practices](./development/security/README.md) | Auth patterns, AES-256-GCM, SQL authorizer, TLS, secrets management |
| Document | Description |
|----------|-------------|
| [Core Runtime And Lifecycle](./reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md) | Bootstrapping, flag system, process modes, watchdog model, shutdown coordination |

### Contributing
### Query Engine

| Guide | Description |
|---|---|
| [Contributing Guidelines](./development/contributing/guidelines.md) | Code style, branch naming, commit format, PR process, virtual table creation |
| Document | Description |
|----------|-------------|
| [SQL Engine And Virtual Tables](./reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) | Embedded SQLite, virtual table integration, constraint pushdown, query planning, security authorizer |

---
### Configuration

## 📖 Reference Architecture
| Document | Description |
|----------|-------------|
| [Configuration And Packs](./reference/architecture/configuration-and-packs/configuration-and-packs.md) | Query packs, scheduling, discovery logic, denylisting, hash-based change detection |
| [Config Plugins](./reference/architecture/config-plugins/config-plugins.md) | FilesystemConfigPlugin, OptionsConfigParserPlugin, ViewsConfigParserPlugin, DecoratorsConfigParserPlugin |

Deep technical documentation for each core subsystem — generated directly from source code analysis.

| Module | Description |
|---|---|
| [Core Init And Runtime](./reference/architecture/core-init-and-runtime/core-init-and-runtime.md) | Process bootstrap, flags, watcher/worker model, watchdog |
| [SQL Engine And Virtual Tables](./reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) | SQLite engine, authorizer, virtual table binding, constraint pushdown |
| [Configuration And Packs](./reference/architecture/configuration-and-packs/configuration-and-packs.md) | Config loading, packs, schedulers, decorators, change detection |
| [Eventing Framework And Subscriptions](./reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md) | Publisher/subscriber system, inotify, BPF, FSEvents, ETW |
| [Extensions And IPC](./reference/architecture/extensions-and-ipc/extensions-and-ipc.md) | Apache Thrift IPC, runtime plugin model, UUID routing |
| [Distributed Querying](./reference/architecture/distributed-querying/distributed-querying.md) | Remote SQL orchestration, TLS transport, fleet denylisting |
| [Remote HTTP Client](./reference/architecture/remote-http-client/remote-http-client.md) | Boost.Asio/Beast HTTPS client, TLS handshake, peer verification |
| [Logging And Query Observability](./reference/architecture/logging-and-query-observability/logging-and-query-observability.md) | Differential result tracking, JSON serialization, pluggable backends |
| [Database And Storage Plugins](./reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md) | RocksDB persistent backend, ephemeral in-memory store, key-value interface |
| [Filesystem And Path Utilities](./reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md) | Cross-platform file abstraction, glob resolution, permission enforcement |
### Storage

---
| Document | Description |
|----------|-------------|
| [Database Backend](./reference/architecture/database-backend/database-backend.md) | RocksDB persistent backend, ephemeral in-memory backend, domain-based key–value storage, schema migration |

## 🗺 Architecture Diagrams
### Observability

Visual Mermaid diagrams for each subsystem are available in:
| Document | Description |
|----------|-------------|
| [Logging And Query Metadata](./reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md) | QueryLogItem, DiffResults, differential vs snapshot logging, JSON serialization |
| [Events Core And Subscriptions](./reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md) | Publisher/subscriber event framework, event persistence, schedule-aware expiration |
| [Osquery Shell Devtooling](./reference/architecture/osquery-shell-devtooling/osquery-shell-devtooling.md) | `osqueryi` interactive shell, meta commands, pretty-print formatting, CPU profiling |

```text
docs/diagrams/architecture/
```
### Distribution & Extensions

Diagrams cover all major subsystems including the SQL engine, eventing framework, configuration pipeline, distributed querying, OpenFrame auth layer, and more. Open any `.mmd` file in a Mermaid-compatible viewer.
| Document | Description |
|----------|-------------|
| [Distributed Querying](./reference/architecture/distributed-querying/distributed-querying.md) | Pull-based distributed queries, discovery gating, denylisting, result batching |
| [Distributed TLS Plugin](./reference/architecture/distributed-tls-plugin/distributed-tls-plugin.md) | Secure HTTPS transport for distributed queries via `TLSDistributedPlugin` |
| [Extensions Framework](./reference/architecture/extensions-framework/extensions-framework.md) | Thrift IPC, dynamic plugin injection, UUID-based routing, health monitoring |

### Networking & Filesystem

| Document | Description |
|----------|-------------|
| [Remote HTTP Client](./reference/architecture/remote-http-client/remote-http-client.md) | Boost.Asio + Boost.Beast HTTP client, TLS validation, timeout handling, proxy support |
| [Filesystem And Fileops Core](./reference/architecture/filesystem-and-fileops-core/filesystem-and-fileops-core.md) | Cross-platform file abstraction, secure permission validation, globbing, portable stat |

---

## 🔗 Quick Links
## 🗺 Architecture Diagrams

| Resource | Link |
|---|---|
| [Project README](../README.md) | Main project overview, quick start, and features |
| [Contributing Guide](../CONTRIBUTING.md) | How to contribute code, docs, and virtual tables |
| [License](../LICENSE.md) | License information |
Mermaid-format architecture diagrams are available under `docs/diagrams/architecture/`. Key diagrams include:

| Diagram | Description |
|---------|-------------|
| `docs/diagrams/architecture/core-runtime-and-lifecycle.mmd` | Core runtime lifecycle flow |
| `docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd` | SQL engine and virtual table layer |
| `docs/diagrams/architecture/extensions-framework.mmd` | Extensions IPC and lifecycle |
| `docs/diagrams/architecture/events-core-and-subscriptions.mmd` | Event publisher/subscriber pipeline |
| `docs/diagrams/architecture/distributed-querying.mmd` | Distributed query execution flow |
| `docs/diagrams/architecture/distributed-tls-plugin.mmd` | TLS-secured distributed transport |
| `docs/diagrams/architecture/database-backend.mmd` | Database backend architecture |
| `docs/diagrams/architecture/configuration-and-packs.mmd` | Configuration and scheduling flow |
| `docs/diagrams/architecture/config-plugins.mmd` | Config plugin chain |
| `docs/diagrams/architecture/remote-http-client.mmd` | HTTP client TLS flow |
| `docs/diagrams/architecture/filesystem-and-fileops-core.mmd` | Filesystem abstraction layer |
| `docs/diagrams/architecture/logging-and-query-metadata.mmd` | Logging pipeline |
| `docs/diagrams/architecture/osquery-shell-devtooling.mmd` | Interactive shell architecture |

---

## 💬 Community

> We do **not** use GitHub Issues or GitHub Discussions. All support and collaboration happens on **OpenMSP Slack**.
## 🔗 Quick Links

| Resource | Link |
|---|---|
| OpenMSP Community Slack | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) |
| OpenMSP Website | [https://www.openmsp.ai/](https://www.openmsp.ai/) |
| OpenFrame Platform | [https://openframe.ai](https://openframe.ai) |
| Flamingo | [https://flamingo.run](https://flamingo.run) |
|----------|------|
| [Project README](../README.md) | Main project README with Quick Start |
| [Contributing Guide](../CONTRIBUTING.md) | How to contribute, code style, PR process |
| [OpenMSP Community Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | Community support (primary channel) |
| [OpenMSP Platform](https://www.openmsp.ai/) | OpenMSP community platform |
| [Flamingo](https://flamingo.run) | Flamingo AI-powered MSP platform |
| [OpenFrame](https://openframe.ai) | OpenFrame unified MSP tooling platform |

---

Expand Down
136 changes: 73 additions & 63 deletions docs/development/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,96 +1,106 @@
# Development Documentation

Welcome to the osquery with OpenFrame development documentation. This section covers everything you need to build, run, test, and contribute to the project.
Welcome to the osquery (OpenFrame-enhanced) development documentation. This section covers everything you need to understand, build, modify, and contribute to this project.

---

## Overview

osquery is a C++ cross-platform OS instrumentation framework. The codebase is organized around a modular, plugin-driven architecture where every major subsystem is independently testable and extensible.

The **OpenFrame** layer adds authentication and encryption components that integrate with the [Flamingo/OpenFrame MSP platform](https://openframe.ai).
This osquery distribution extends the upstream open-source osquery framework with OpenFrame-specific authentication, token management, and AES-256-GCM encryption services. The codebase is written in **C++17** and built with **CMake**, targeting Linux, macOS, and Windows.

---

## Documentation Index

| Guide | Description |
|---|---|
| [Environment Setup](setup/environment.md) | IDE recommendations, development tools, editor extensions |
| [Local Development](setup/local-development.md) | Clone, build, run locally, debug configuration |
| [Architecture Overview](architecture/README.md) | High-level architecture diagrams and core component descriptions |
| [Security Best Practices](security/README.md) | Auth patterns, encryption, secrets management |
| [Testing Guide](testing/README.md) | Test structure, running tests, writing new tests |
| [Contributing Guidelines](contributing/guidelines.md) | Code style, branch naming, PR process, commit format |
### Setup

---
| Document | Description |
|----------|-------------|
| [Environment Setup](setup/environment.md) | IDE configuration, development tools, editor extensions |
| [Local Development](setup/local-development.md) | Cloning, building, running locally, debug configuration |

## Technology Stack
### Architecture

| Layer | Technology |
|---|---|
| **Language** | C++17 |
| **Build System** | CMake 3.21+ with Ninja |
| **SQL Engine** | SQLite (embedded, in-memory) |
| **IPC** | Apache Thrift (UNIX sockets / named pipes) |
| **Encryption** | OpenSSL (AES-256-GCM via OpenFrame layer) |
| **Networking** | Boost.Asio + Boost.Beast |
| **Event Systems** | inotify (Linux), FSEvents (macOS), ETW (Windows), BPF (Linux) |
| **Database** | RocksDB (persistent), ephemeral in-memory store |
| **Code Generation** | Python scripts for table schema, API, and amalgamation |
| **Testing** | Google Test + Google Mock |
| Document | Description |
|----------|-------------|
| [Architecture Overview](architecture/README.md) | High-level system design, component diagrams, data flow |

### Security

| Document | Description |
|----------|-------------|
| [Security Best Practices](security/README.md) | Auth patterns, encryption, secrets management, vulnerability mitigations |

### Testing

| Document | Description |
|----------|-------------|
| [Testing Overview](testing/README.md) | Test structure, running tests, writing new tests, coverage |

### Contributing

| Document | Description |
|----------|-------------|
| [Contributing Guidelines](contributing/guidelines.md) | Code style, branch naming, PR process, commit format |

---

## Repository Structure
## Project Structure

```text
osquery/
├── osquery/ # Core C++ source — SQL, eventing, config, logging
│ ├── core/ # Initialization, flags, watcher/worker model
│ ├── sql/ # SQLite engine, virtual tables, authorizer
│ ├── config/ # Configuration loading, packs, parsers
│ ├── events/ # Eventing framework (publishers/subscribers)
│ ├── database/ # Storage backend abstraction (RocksDB, ephemeral)
│ ├── distributed/ # Distributed query orchestration
│ ├── extensions/ # Thrift-based extension/IPC framework
│ ├── remote/ # HTTP client, TLS transport
│ ├── logger/ # Logging plugins and observability
│ └── tables/ # Virtual table implementations
├── openframe/ # OpenFrame authentication and encryption layer
├── plugins/ # Logger, config, database, and distributed plugins
├── libraries/ # CMake-managed third-party dependencies
├── tools/ # Codegen scripts, CI tools, formatting
├── tests/ # Integration test suite
└── external/ # Extension examples
├── osquery/ # Core osquery source (C++17)
│ ├── core/ # Runtime lifecycle, flags, watchdog
│ ├── sql/ # SQL engine and virtual table layer
│ ├── config/ # Configuration and query packs
│ ├── database/ # RocksDB and ephemeral backends
│ ├── events/ # Event publisher/subscriber system
│ ├── extensions/ # Thrift-based extension framework
│ ├── distributed/ # Distributed query engine
│ ├── logger/ # Logging and query metadata
│ └── remote/ # HTTP client for TLS endpoints
├── openframe/ # OpenFrame-specific extensions (C++)
│ ├── openframe_authorization_manager.*
│ ├── openframe_encryption_service.*
│ ├── openframe_token_extractor.*
│ └── openframe_token_refresher.*
├── plugins/ # Config, logger, database plugins
├── libraries/ # Vendored third-party libraries (CMake)
├── tools/ # Code generation, CI scripts
├── external/ # Extension SDK examples
└── tests/ # Integration tests
```

---

## Quick Commands
## Technology Stack

```bash
# Configure build
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo
| Layer | Technology |
|-------|-----------|
| **Language** | C++17 |
| **Build System** | CMake ≥ 3.21 |
| **SQL Engine** | SQLite (embedded) |
| **Persistent Database** | RocksDB |
| **IPC / Extensions** | Apache Thrift |
| **Networking** | Boost.Asio + Boost.Beast |
| **Encryption** | OpenSSL (AES-256-GCM) |
| **Logging** | Google glog |
| **Testing** | Google Test (gtest) |
| **Flags** | Google gflags |
| **Documentation Pipeline** | VoltAgent + Anthropic AI |

# Build everything
cmake --build build --parallel $(nproc)
---

# Run interactive shell
./build/osquery/osqueryi
## Quick Navigation

# Run tests
cmake --build build --target osquery_tests
cd build && ctest --output-on-failure
```
- New to the codebase? Start with [Architecture Overview](architecture/README.md).
- Setting up your machine? See [Environment Setup](setup/environment.md).
- Ready to code? Follow [Local Development](setup/local-development.md).
- Writing a fix or feature? Read [Contributing Guidelines](contributing/guidelines.md).
- Security concern? See [Security Best Practices](security/README.md).

---

## Getting Help

All development discussions happen in the **OpenMSP Slack community**:

- [Join OpenMSP Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA)
- [https://www.openmsp.ai/](https://www.openmsp.ai/)
## Community

> We do not use GitHub Issues or GitHub Discussions. All support and collaboration is on Slack.
All development discussions happen in the [OpenMSP Slack community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA). This project does not use GitHub Issues or GitHub Discussions.
Loading
Loading