diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5e5092f04d5..ab57f0302f1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,365 +1,344 @@ -# Contributing to osquery β€” OpenFrame Edition +# Contributing to osquery (OpenFrame-Enhanced) -Thank you for contributing to osquery with OpenFrame! This guide covers everything you need to get started: code style, branching conventions, commit format, testing requirements, and the pull request process. +Thank you for your interest in contributing to the OpenFrame-enhanced distribution of osquery! This guide covers everything you need to get started: setting up your environment, understanding the codebase, following code style conventions, and submitting changes. --- ## Community First -All collaboration happens on the **OpenMSP Slack community** β€” not GitHub Issues or GitHub Discussions. +This project is managed through the **OpenMSP Slack community**. GitHub Issues and GitHub Discussions are **not used**. -| Resource | Link | -|---|---| -| πŸ’¬ OpenMSP Community Slack | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | -| 🌐 OpenMSP Website | [https://www.openmsp.ai/](https://www.openmsp.ai/) | +- **Slack**: [Join OpenMSP](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) +- **Platform**: [https://www.openmsp.ai/](https://www.openmsp.ai/) -Before starting significant work, please discuss your changes in Slack to align with the team's roadmap. +Before opening a pull request, please discuss your planned change in the Slack community. This ensures alignment on scope, approach, and priority. --- -## Development Setup +## Development Environment Setup -### Hardware Requirements +### System Requirements -| Tier | RAM | CPU Cores | Disk | -|---|---|---|---| -| **Minimum** | 24 GB | 6 cores | 50 GB | -| **Recommended** | 32 GB | 12 cores | 100 GB | +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| **RAM** | 24 GB | 32 GB | +| **CPU Cores** | 6 cores | 12 cores | +| **Disk Space** | 50 GB | 100 GB | ### Required Tools -| Tool | Minimum Version | Purpose | -|---|---|---| -| CMake | 3.21+ | Build system generator | -| Python | 3.8+ | Code generation scripts | -| Git | 2.x | Source control | -| C++ Compiler | GCC 9+ / Clang 10+ / MSVC 2019+ | C++17 compilation | -| Ninja | 1.10+ | Fast parallel builds | -| OpenSSL | 1.1.1+ | TLS + AES-256-GCM (OpenFrame layer) | -| clang-format | β€” | Code formatting (CI enforced) | +| Tool | Version | Purpose | +|------|---------|---------| +| **CMake** | β‰₯ 3.21 | Build system generator | +| **Clang** | β‰₯ 13 | C++ compiler (C++17 required) | +| **Python 3** | β‰₯ 3.6 | Code generation scripts | +| **OpenSSL** | β‰₯ 1.1.1 | AES-256-GCM encryption | +| **Git** | β‰₯ 2.x | Source control | -### Quick Setup +### Install Dependencies + +**Linux (Ubuntu/Debian):** + +```bash +sudo apt-get update && sudo apt-get install -y \ + build-essential \ + cmake \ + clang-13 \ + clang++-13 \ + lld-13 \ + python3 \ + python3-pip \ + git \ + libssl-dev \ + ninja-build \ + ccache \ + clang-format-13 +``` + +**macOS:** + +```bash +xcode-select --install +brew install cmake ninja ccache openssl clang-format +``` + +**Windows:** + +1. Install [Visual Studio 2022](https://visualstudio.microsoft.com/) with the **Desktop development with C++** workload +2. Install [CMake](https://cmake.org/download/) β‰₯ 3.21 +3. Install [Git for Windows](https://git-scm.com/download/win) +4. Install [Python 3](https://www.python.org/downloads/windows/) + +--- + +## Cloning and Building ```bash -# Clone +# Clone the repository git clone https://github.com/flamingo-stack/osquery.git cd osquery -# Configure (Debug build for development) -cmake -S . -B build \ - -G Ninja \ +# Initialize submodules +git submodule update --init --recursive + +# Create build directory (always build out-of-source) +mkdir build && cd build + +# Configure for development (Debug mode + compile_commands.json) +cmake .. \ + -DCMAKE_C_COMPILER=clang \ + -DCMAKE_CXX_COMPILER=clang++ \ -DCMAKE_BUILD_TYPE=Debug \ - -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ - -DOSQUERY_BUILD_TESTS=ON + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON # Build -cmake --build build --parallel $(nproc) +cmake --build . --parallel $(nproc) -# Run -./build/osquery/osqueryi +# Symlink compile_commands.json for IDE/clangd support +cd .. +ln -s build/compile_commands.json compile_commands.json ``` -For full environment setup instructions see the [Development Documentation](./docs/development/README.md). +### Useful CMake Flags for Development + +| Flag | Description | +|------|-------------| +| `-DCMAKE_BUILD_TYPE=Debug` | Include debug symbols | +| `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` | Generate `compile_commands.json` for IDE intelligence | +| `-DOSQUERY_DISABLE_DATABASE=ON` | Skip RocksDB build (faster for table development) | +| `-DCMAKE_C_COMPILER_LAUNCHER=ccache` | Enable ccache for faster incremental builds | +| `-DCMAKE_CXX_COMPILER_LAUNCHER=ccache` | Enable ccache for C++ compilation | --- -## Code Style and Conventions +## Project Structure -### C++ Standards +```text +osquery/ +β”œβ”€β”€ osquery/ # Core osquery C++ source (C++17) +β”‚ β”œβ”€β”€ core/ # Runtime lifecycle, flags, watchdog +β”‚ β”œβ”€β”€ sql/ # SQL engine and virtual table layer +β”‚ β”œβ”€β”€ config/ # Configuration and query packs +β”‚ β”œβ”€β”€ database/ # RocksDB and ephemeral backends +β”‚ β”œβ”€β”€ events/ # Event publisher/subscriber system +β”‚ β”œβ”€β”€ extensions/ # Thrift-based extension framework +β”‚ β”œβ”€β”€ distributed/ # Distributed query engine +β”‚ β”œβ”€β”€ logger/ # Logging and query metadata +β”‚ └── remote/ # HTTP client for TLS endpoints +β”œβ”€β”€ openframe/ # OpenFrame-specific extensions (C++) +β”‚ β”œβ”€β”€ openframe_authorization_manager.* +β”‚ β”œβ”€β”€ openframe_encryption_service.* +β”‚ β”œβ”€β”€ openframe_token_extractor.* +β”‚ └── openframe_token_refresher.* +β”œβ”€β”€ plugins/ # Config, logger, database, distributed plugins +β”œβ”€β”€ libraries/ # Vendored third-party libraries +β”œβ”€β”€ tools/ # Code generation and CI scripts +β”œβ”€β”€ external/ # Extension SDK examples +└── tests/ # Integration tests +``` + +--- -- Use **C++17** features where appropriate -- Follow the existing code style in each file you modify -- All new code must pass `clang-format` with the repository's `.clang-format` config +## Code Style -### Formatting +### C++ Standards -osquery enforces `clang-format`. Run it before every commit: +- The codebase uses **C++17** +- Follow the existing code style in the file you are editing +- Use `clang-format` to format all C++ files before submitting ```bash -# Format a single file -clang-format -i path/to/your/file.cpp +# Format a file in-place +clang-format -i openframe/openframe_encryption_service.cpp -# Format all changed files (compared to main branch) -git diff --name-only main | grep -E '\.(cpp|h)$' | xargs clang-format -i - -# Check without modifying -clang-format --dry-run --Werror path/to/your/file.cpp +# Check formatting without modifying +clang-format --dry-run --Werror openframe/openframe_encryption_service.cpp ``` ### Naming Conventions -| Item | Convention | Example | -|---|---|---| -| Classes | `PascalCase` | `EventSubscriberPlugin` | -| Methods | `camelCase` | `generateRows()` | -| Member variables | `snake_case_` (trailing underscore) | `running_` | -| Constants | `kPascalCase` | `kSQLOpcodes` | -| Macros | `UPPER_SNAKE_CASE` | `DECLARE_FLAG` | -| Namespaces | `snake_case` | `osquery` | -| Files | `snake_case.cpp` / `snake_case.h` | `event_subscriber.cpp` | +| Construct | Convention | Example | +|-----------|-----------|---------| +| Classes | PascalCase | `OpenframeEncryptionService` | +| Methods | camelCase | `extractToken()` | +| Variables | snake_case | `token_path` | +| Constants | UPPER_SNAKE_CASE | `MAX_RETRY_COUNT` | +| Files | snake_case | `openframe_token_extractor.cpp` | -### Include Order +### General Guidelines -Follow this include order with blank lines between groups: +- Write self-documenting code; add comments for non-obvious logic +- Keep functions focused on a single responsibility +- Prefer `const` and RAII patterns +- Do not introduce new dependencies without discussion in the community Slack +- Follow existing patterns in the module you are modifying -```cpp -// 1. Standard library -#include -#include -#include - -// 2. Third-party libraries -#include -#include - -// 3. osquery headers -#include "osquery/core/core.h" -#include "osquery/sql/sql.h" - -// 4. Local headers (same directory) -#include "my_local_header.h" -``` +--- -### Code Organization +## Adding New OpenFrame Source Files -- Keep headers (`*.h`) minimal β€” forward declare where possible -- Use the `osquery` namespace for all production code -- Place tests in `tests/` subdirectories alongside the source -- New virtual tables go in `osquery/tables//` +When adding a new C++ source file to the `openframe/` directory: ---- +1. Create the `.h` and `.cpp` files +2. Add the source to the relevant `CMakeLists.txt`: -## Branch Naming +```cmake +target_sources(openframe_lib PRIVATE + openframe/openframe_new_component.cpp +) +``` -Always branch from the latest `main`: +3. Re-run CMake and rebuild: ```bash -git checkout main -git pull origin main -git checkout -b feature/my-new-feature +cd build +cmake .. +cmake --build . --parallel $(nproc) ``` -| Type | Pattern | Example | -|---|---|---| -| Feature | `feature/` | `feature/bpf-socket-events` | -| Bug fix | `fix/` | `fix/config-refresh-race` | -| OpenFrame integration | `openframe/` | `openframe/token-refresh-retry` | -| Documentation | `docs/` | `docs/virtual-table-guide` | -| Refactor | `refactor/` | `refactor/sql-authorizer` | -| Test | `test/` | `test/events-integration` | -| Release | `release/v` | `release/v5.13.0` | - --- -## Commit Message Format - -```text -(): - - +## Running Tests - -``` +```bash +cd build -### Types - -| Type | When to Use | -|---|---| -| `feat` | New feature or capability | -| `fix` | Bug fix | -| `docs` | Documentation changes only | -| `style` | Code formatting, no logic change | -| `refactor` | Code refactoring without behavior change | -| `test` | Adding or fixing tests | -| `perf` | Performance improvement | -| `chore` | Build, CI, tooling changes | -| `openframe` | OpenFrame platform-specific changes | - -### Scopes - -| Scope | Area | -|---|---| -| `core` | Core init and runtime | -| `sql` | SQL engine and virtual tables | -| `config` | Configuration and packs | -| `events` | Eventing framework | -| `logger` | Logging and observability | -| `db` | Database and storage | -| `distributed` | Distributed querying | -| `extensions` | Extension IPC | -| `http` | Remote HTTP client | -| `openframe` | OpenFrame auth layer | -| `tables` | Virtual table implementations | - -### Examples +# Run all unit tests +ctest --parallel $(nproc) --output-on-failure -```text -feat(events): add BPF socket event publisher for Linux +# Run a specific test binary +./osquery/osquery_tests --gtest_filter=ConfigTests.* -Implements a new BPF-based publisher that captures socket connect/accept -events and exposes them via the bpf_socket_events virtual table. +# Run with verbose output +./osquery/osquery_tests --gtest_filter="*" --gtest_verbose=all ``` -```text -fix(openframe): handle token refresh failure with exponential backoff +### Writing Tests -When OpenframeTokenRefresher encounters a network error, it now retries -with exponential backoff instead of immediately stopping the refresh loop. -``` +Tests use **Google Test (gtest)**. When modifying security-sensitive code in `openframe/`, always add unit tests: -```text -test(config): add pack discovery query unit tests +```cpp +// Test that decryption fails on tampered ciphertext +TEST(EncryptionServiceTest, TamperedCiphertextThrows) { + OpenframeEncryptionService svc("your-32-byte-secret-key-here!!"); + EXPECT_THROW(svc.decrypt("tampered_base64_payload"), std::runtime_error); +} ``` --- -## Testing - -Tests use **Google Test** and **Google Mock**. Always build with `-DOSQUERY_BUILD_TESTS=ON`: - -```bash -cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DOSQUERY_BUILD_TESTS=ON -cmake --build build --parallel $(nproc) - -# Run all tests -cd build && ctest --output-on-failure +## Security Guidelines -# Run in parallel -cd build && ctest --output-on-failure --parallel $(nproc) +Security is a top priority for all contributions. Follow these rules strictly: -# Run a specific suite -cd build && ctest -R "osquery_sql_tests" --output-on-failure -``` +- **Never log token values** at any severity level +- **Never hardcode secrets** β€” use environment variable injection or a secrets manager +- **Always validate file paths** before opening them +- **Always join background threads** on shutdown +- **Never disable TLS peer verification** in production code +- Use `OpenframeAuthorizationManagerProvider` to access the authorization manager β€” direct construction is private by design -### Test Categories +### Code Review Security Checklist -| Category | Location | Description | -|---|---|---| -| Unit Tests | `osquery/*/tests/` | Fast, isolated, no OS dependencies | -| Integration Tests | `tests/integration/tables/` | Live table queries against the real OS | -| Extension Tests | `osquery/extensions/tests/` | IPC and Thrift round-trips | -| Plugin Tests | `plugins/*/tests/` | Logger, config, and database plugins | +Before submitting a PR that touches `openframe/`, `osquery/remote/`, or authentication logic: -When contributing a new virtual table, a corresponding integration test in `tests/integration/tables/` is expected. +- [ ] No secrets or tokens appear in log output +- [ ] Secret keys are not hardcoded in source or tests +- [ ] Encryption errors throw exceptions and do NOT silently return empty strings +- [ ] All file paths are validated before use +- [ ] Background threads are properly joined on shutdown +- [ ] No new unsafe SQL execution paths are introduced +- [ ] TLS peer verification is not disabled --- ## Pull Request Process -### Before Submitting - -- [ ] Branch is up-to-date with `main` -- [ ] All tests pass: `cd build && ctest --output-on-failure` -- [ ] Code is formatted: `clang-format --dry-run --Werror` -- [ ] Copyright headers are present on new files -- [ ] New virtual tables have integration tests in `tests/integration/tables/` -- [ ] OpenFrame-specific changes include updated documentation -- [ ] Discussed in OpenMSP Slack (for significant changes) - -### PR Title Format +1. **Discuss first** β€” Open a thread in the [OpenMSP Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) before starting significant work +2. **Branch naming** β€” Use descriptive branch names: `feature/openframe-token-rotation`, `fix/rocksdb-shutdown-hang` +3. **Commit messages** β€” Write clear, imperative commit messages: `Add AES key rotation support to OpenframeEncryptionService` +4. **Keep PRs focused** β€” One logical change per PR; avoid mixing refactors with feature work +5. **Run tests** β€” Ensure `ctest` passes before submitting +6. **Format code** β€” Run `clang-format` on all modified files +7. **Update documentation** β€” If you change a public interface or behavior, update the relevant docs -Use the same format as commit messages: +### Commit Message Format ```text -feat(sql): add query result caching for repeated virtual table scans +: + + ``` -### PR Description Template +Types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `security` -```markdown -## Summary - +**Examples:** -## Changes - +```text +feat: add OpenframeTokenRefresher configurable interval -## Testing - +fix: resolve RocksDB file lock during rapid daemon restarts -## Platform Support - +security: enforce TLS peer verification in all remote HTTP calls -## Checklist -- [ ] Tests pass (ctest) -- [ ] clang-format applied -- [ ] Documentation updated (if applicable) -- [ ] Discussed in OpenMSP Slack (if significant change) +docs: update prerequisites with OpenSSL version requirement ``` --- -## Copyright Headers +## Running Locally in Development Mode -All new source files must include a copyright header: +Use the in-memory database for rapid iteration to avoid RocksDB file locking: -```cpp -/** - * Copyright (c) 2014-present, The osquery authors - * - * This source code is licensed in accordance with the terms specified in - * the LICENSE file found in the root directory of this source tree. - */ +```bash +./build/osquery/osqueryd \ + --config_path=/tmp/dev_osquery.conf \ + --ephemeral=true \ + --disable_database=true \ + --verbose ``` -The CI script `tools/ci/scripts/check_copyright_headers.py` enforces this on all pull requests. - --- -## Adding a New Virtual Table +## Debugging -1. Define the table schema in `osquery/tables//.table` -2. Run the code generator: +### LLDB (macOS / Linux) ```bash -python3 tools/codegen/gentable.py osquery/tables//.table +lldb ./build/osquery/osqueryi +(lldb) breakpoint set --name OpenframeEncryptionService::decrypt +(lldb) run --verbose ``` -3. Implement the `generate()` method in `.cpp` -4. Register in the CMakefile for your category -5. Add an integration test in `tests/integration/tables/.cpp` -6. Test locally: +### GDB (Linux) ```bash -cmake --build build --target osqueryi -./build/osquery/osqueryi -osquery> SELECT * FROM ; +gdb ./build/osquery/osqueryi +(gdb) break OpenframeTokenExtractor::extractToken +(gdb) run --verbose ``` --- -## Security Guidelines - -- **Never** hardcode secrets, credentials, or tokens in source code -- **Never** log JWT token values, even at debug level -- SQL inputs must pass through the SQLite authorizer β€” do not bypass it -- AES-GCM nonces must be generated freshly (never reused for the same key) -- TLS peer verification must **not** be disabled in production code -- New config keys must include size/depth validation -- Thread-shared state must use proper synchronization primitives +## Environment Variables -Report security vulnerabilities directly to the Flamingo team via the **OpenMSP Slack community** β€” do not open public GitHub Issues for security issues. +| Variable | Purpose | Notes | +|----------|---------|-------| +| `CC` | C compiler path | e.g., `/usr/bin/clang-13` | +| `CXX` | C++ compiler path | e.g., `/usr/bin/clang++-13` | +| `CCACHE_DIR` | ccache storage directory | e.g., `$HOME/.ccache` | +| `OPENFRAME_TOKEN_PATH` | Path to encrypted token file | For OpenFrame integration | +| `OPENFRAME_SECRET_KEY` | AES-256 key for token decryption | Inject from secrets manager; never hardcode | --- -## Reviewer Checklist - -When reviewing a PR: - -- [ ] Logic is correct and all error paths are handled -- [ ] No secrets or credentials in source -- [ ] SQL inputs are validated through the authorizer -- [ ] Thread safety considered for shared state -- [ ] Platform-specific code is properly guarded with `#ifdef` -- [ ] Tests added for new functionality -- [ ] No debug/temporary code left in -- [ ] Commit messages follow format convention -- [ ] Performance impact considered for hot paths (scheduler, SQL engine) - ---- +## Getting Help -## Code of Conduct +All development discussions happen in the [OpenMSP Slack community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA). This is the fastest way to get answers, discuss architecture decisions, or ask about contribution guidelines. -Be respectful, collaborative, and constructive. All contributors are expected to maintain a professional and welcoming environment in both Slack and code reviews. +- **OpenMSP Community**: [https://www.openmsp.ai/](https://www.openmsp.ai/) +- **Flamingo**: [https://flamingo.run](https://flamingo.run) +- **OpenFrame**: [https://openframe.ai](https://openframe.ai) --- diff --git a/README.md b/README.md index a707452a43e..bd46c88681c 100644 --- a/README.md +++ b/README.md @@ -10,238 +10,240 @@ License

-# osquery β€” OpenFrame Edition +# osquery β€” OpenFrame-Enhanced Distribution -**osquery** is a cross-platform operating system instrumentation framework that exposes live system state and activity as relational data β€” enabling you to query your operating system using SQL. +**osquery** is a cross-platform operating system instrumentation framework that exposes system state as relational data using **SQL**. This distribution extends the upstream osquery project with **OpenFrame**-specific authentication, token management, and AES-256-GCM encryption services β€” part of the [Flamingo](https://flamingo.run) / [OpenFrame](https://openframe.ai) platform for intelligent MSP automation. -Built and extended by the [Flamingo / OpenFrame](https://openframe.ai) platform, this distribution integrates osquery's powerful SQL telemetry engine with OpenFrame's AI-driven MSP automation infrastructure. Every host becomes a **SQL-queryable telemetry node** β€” no proprietary log parsers, no fragile scripts, just standard SQL. +Instead of parsing log files or writing custom scripts, you query the operating system like a database: ```sql --- Which processes are listening on ports? -SELECT pid, name, port, protocol FROM listening_ports JOIN processes USING (pid); +SELECT name, pid, path FROM processes WHERE on_disk = 0; +``` --- What Chrome extensions are installed? -SELECT u.username, e.name, e.version, e.permissions -FROM users u, chrome_extensions e -WHERE e.uid = u.uid; +```sql +SELECT address, mac FROM interface_addresses; +``` + +```sql +SELECT * FROM users WHERE uid = 0; ``` +--- + +## Watch: Introduction to osquery + +[![osquery Introduction](https://img.youtube.com/vi/bRd3JCJZ1vc/hqdefault.jpg)](https://www.youtube.com/watch?v=bRd3JCJZ1vc) + +--- + ## Features -- **SQL Querying** β€” Query live OS data with standard SQL via an embedded, hardened SQLite engine -- **300+ Virtual Tables** β€” Platform-specific tables across Linux, macOS, and Windows covering processes, sockets, packages, users, registry, hardware, and more -- **Event Monitoring** β€” inotify, BPF, FSEvents, ETW, and OpenBSM events exposed as queryable tables -- **Scheduled Query Packs** β€” Configuration-driven query scheduling with differential result tracking (only changed rows are logged) -- **Distributed Fleet Queries** β€” Remote SQL orchestration across entire fleets via TLS -- **Extension System** β€” Runtime plugin model via Apache Thrift IPC β€” add custom tables, loggers, and config plugins as external processes -- **OpenFrame Auth Layer** β€” AES-256-GCM encrypted JWT token management (`OpenframeAuthorizationManager`, `OpenframeEncryptionService`, `OpenframeTokenRefresher`) for seamless integration with the Flamingo/OpenFrame MSP platform -- **Cross-Platform** β€” Linux (x86\_64, aarch64), macOS (Intel + Apple Silicon), and Windows x86\_64 -- **Security-First** β€” SQLite authorizer allowlists only safe opcodes, Watcher/Worker process isolation, peer-verified TLS everywhere +- **SQL-based OS querying** β€” Interact with 300+ virtual tables covering processes, files, users, network, and kernel state +- **Cross-platform** β€” Runs on Linux, macOS, and Windows with a unified SQL interface +- **Scheduled query packs** β€” Define queries that run on configurable schedules +- **Event-driven tables** β€” Subscribe to OS events: file changes, process launches, network connections +- **Distributed querying** β€” Push queries to an entire fleet from a central control plane over TLS +- **Plugin extensions** β€” Add custom tables, loggers, and config sources via Thrift IPC without modifying the binary +- **OpenFrame integration** β€” Secure bearer-token authentication with the OpenFrame/Flamingo platform +- **AES-256-GCM encryption** β€” Built-in encrypted token storage and retrieval at rest +- **Differential logging** β€” Only changed rows (added/removed) are logged by default, drastically reducing output volume +- **Watchdog supervision** β€” Watcher/worker process separation for resilient daemon operation +- **Pluggable persistence** β€” RocksDB for durability or in-memory ephemeral backend for lightweight use --- -## Architecture +## System Architecture ```mermaid -graph TD - CLI["osqueryi / osqueryd"] --> Core["Core Init And Runtime"] +flowchart TD + OpenFrame["OpenFrame Auth Layer"] --> Core["Core Runtime And Lifecycle"] Core --> Config["Configuration And Packs"] Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework"] - Core --> Logging["Logging And Observability"] - Core --> DB["Database And Storage Plugins"] - Core --> Dist["Distributed Querying"] - Core --> Ext["Extensions And IPC"] - Core --> HTTP["Remote HTTP Client"] - Core --> OF["OpenFrame Auth Layer"] + Core --> DB["Database Backend"] + Core --> Logging["Logging And Query Metadata"] + Core --> Events["Events Core And Subscriptions"] + Core --> Extensions["Extensions Framework"] + Core --> Distributed["Distributed Querying"] + Config --> SQL Config --> Events + Config --> Logging + SQL --> Logging + SQL --> DB + Events --> DB - Dist --> SQL - Dist --> HTTP - Ext --> SQL - Ext --> DB - OF --> HTTP + Events --> Logging + + Distributed --> SQL + Distributed --> DB + Distributed --> Logging + + Extensions --> SQL + Extensions --> Config + Extensions --> Distributed ``` -| Module | Location | Responsibility | -|---|---|---| -| Core Init And Runtime | `osquery/core/` | Process lifecycle, flags, watcher/worker model | -| SQL Engine And Virtual Tables | `osquery/sql/` | SQLite engine, authorizer, virtual table binding | -| Configuration And Packs | `osquery/config/` | Scheduled queries, packs, decorators | -| Eventing Framework | `osquery/events/` | Publisher/subscriber event system | -| Logging And Observability | `osquery/logger/` | Differential logging, JSON serialization | -| Database And Storage Plugins | `osquery/database/` | RocksDB persistent + ephemeral in-memory stores | -| Distributed Querying | `osquery/distributed/` | Remote fleet orchestration | -| Extensions And IPC | `osquery/extensions/` | Apache Thrift-based runtime extensions | -| Remote HTTP Client | `osquery/remote/` | Boost.Asio/Beast HTTPS with strict TLS | -| OpenFrame Auth Layer | `openframe/` | JWT token management + AES-256-GCM encryption | +### Architectural Layers + +| Layer | Responsibility | +|-------|---------------| +| **Core Runtime** | Bootstrapping, flags, lifecycle, watchdog, shutdown | +| **SQL Engine** | Embedded SQLite, virtual table layer, query planning (300+ tables) | +| **Configuration** | Packs, scheduling, discovery, dynamic updates | +| **Database Backend** | Persistent RocksDB and ephemeral in-memory key–value storage | +| **Logging** | Differential result serialization and logger plugin emission | +| **Events** | Real-time OS event ingestion exposed as SQL tables | +| **Distributed** | Pull-based remote query retrieval and TLS result submission | +| **Extensions** | Thrift IPC for external plugin injection | +| **OpenFrame Auth** | Token lifecycle, AES-256-GCM encryption, background refresh | --- ## Technology Stack | Layer | Technology | -|---|---| -| Language | C++17 | -| Build System | CMake 3.21+ with Ninja | -| SQL Engine | SQLite (embedded, in-memory) | -| IPC | Apache Thrift (UNIX sockets / named pipes) | -| Encryption | OpenSSL (AES-256-GCM) | -| Networking | Boost.Asio + Boost.Beast | -| Event Systems | inotify, BPF, FSEvents, ETW, OpenBSM | -| Database | RocksDB (persistent), ephemeral in-memory | -| Testing | Google Test + Google Mock | +|-------|-----------| +| **Language** | C++17 | +| **Build System** | CMake β‰₯ 3.21 | +| **SQL Engine** | SQLite (embedded) | +| **Persistent Database** | RocksDB | +| **IPC / Extensions** | Apache Thrift | +| **Networking** | Boost.Asio + Boost.Beast | +| **Encryption** | OpenSSL (AES-256-GCM) | +| **Logging** | Google glog | +| **Testing** | Google Test (gtest) | +| **Flags** | Google gflags | --- ## Hardware Requirements -| Tier | RAM | CPU Cores | Disk | -|---|---|---|---| -| **Minimum** | 24 GB | 6 cores | 50 GB | -| **Recommended** | 32 GB | 12 cores | 100 GB | - -> Building from source is resource-intensive. The recommended configuration significantly reduces build times and prevents out-of-memory failures during compilation. - ---- - -## Supported Platforms - -| Platform | Architecture | Notes | -|---|---|---| -| Linux | x86\_64, aarch64 | Ubuntu 20.04+, RHEL 8+, Debian 11+ | -| macOS | x86\_64, aarch64 | macOS 12+ (Intel + Apple Silicon) | -| Windows | x86\_64 | Windows 10/11, Server 2019+ | +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| **RAM** | 24 GB | 32 GB | +| **CPU Cores** | 6 cores | 12 cores | +| **Disk Space** | 50 GB | 100 GB | --- ## Quick Start -### 1. Install Prerequisites +### Prerequisites -**Linux (Debian/Ubuntu):** - -```bash -sudo apt-get update -sudo apt-get install -y \ - build-essential cmake ninja-build python3 python3-pip \ - git openssl libssl-dev clang-format ccache -``` +- CMake β‰₯ 3.21 +- Clang β‰₯ 13 or GCC β‰₯ 10 (C++17 required) +- Python 3 β‰₯ 3.6 +- OpenSSL β‰₯ 1.1.1 (for OpenFrame encryption) +- Git -**macOS:** +Verify your environment: ```bash -xcode-select --install -brew install cmake ninja python3 openssl git ccache +cmake --version +clang --version +python3 --version +openssl version ``` -**Windows:** - -1. Install [Visual Studio 2022](https://visualstudio.microsoft.com/) with the **Desktop development with C++** workload -2. Install [CMake 3.21+](https://cmake.org/download/), [Git](https://git-scm.com/download/win), [Python 3.8+](https://www.python.org/downloads/windows/) -3. Download the CLI binary directly: [openframe-cli\_windows\_amd64.zip](https://github.com/flamingo-stack/openframe-cli/releases/latest/download/openframe-cli_windows_amd64.zip) - -### 2. Clone and Build +### Build from Source ```bash -# Clone +# 1. Clone the repository git clone https://github.com/flamingo-stack/osquery.git cd osquery -# Configure (Linux/macOS) -cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo -G Ninja +# 2. Create the build directory +mkdir build && cd build -# Configure (Windows) -cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo -G "Visual Studio 17 2022" +# 3. Configure with CMake +cmake .. \ + -DCMAKE_C_COMPILER=clang \ + -DCMAKE_CXX_COMPILER=clang++ \ + -DCMAKE_BUILD_TYPE=Release -# Build β€” uses all available CPU cores -cmake --build build --parallel $(nproc) +# 4. Build (uses all available CPU cores) +cmake --build . --parallel $(nproc) + +# 5. Run the interactive shell +./osquery/osqueryi ``` -### 3. Run Your First Query +### Windows (AMD64) -```bash -./build/osquery/osqueryi -``` +1. Download: [openframe-cli_windows_amd64.zip](https://github.com/flamingo-stack/openframe-cli/releases/latest/download/openframe-cli_windows_amd64.zip) +2. Extract the archive +3. Run the installer following the same steps as other operating systems + +### Your First Query -```text -Using a virtual database. Need help, type '.help' -osquery> +```bash +./osquery/osqueryi ``` ```sql -osquery> SELECT hostname, cpu_brand, physical_memory FROM system_info; -osquery> SELECT pid, name, port, protocol FROM listening_ports LIMIT 10; -osquery> SELECT uid, username, shell FROM users; -osquery> .tables +-- List running processes +SELECT pid, name, path FROM processes LIMIT 10; + +-- Check listening network ports +SELECT pid, port, protocol, address FROM listening_ports; + +-- List all available tables +.tables + +-- Exit +.exit ``` -### 4. Run as a Daemon +### Run the Daemon (Continuous Monitoring) ```bash -sudo mkdir -p /etc/osquery -sudo tee /etc/osquery/osquery.conf <<'EOF' -{ - "options": { - "logger_plugin": "filesystem", - "schedule_splay_percent": 10 - }, - "schedule": { - "system_info": { - "query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;", - "interval": 3600 - } - } -} -EOF - -sudo ./build/osquery/osqueryd --config_path=/etc/osquery/osquery.conf +./osquery/osqueryd --config_path=/etc/osquery/osquery.conf ``` --- -## OpenFrame Integration +## OpenFrame Extensions -This distribution extends osquery with a secure authentication and encryption layer for the [OpenFrame platform](https://www.flamingo.run/openframe): +This distribution adds the following components on top of upstream osquery: | Component | Description | -|---|---| -| `OpenframeAuthorizationManager` | Lifecycle-controlled JWT token singleton (non-copyable) | -| `OpenframeAuthorizationManagerProvider` | Sole factory and owner of the token manager | -| `OpenframeEncryptionService` | AES-256-GCM symmetric encryption via OpenSSL | -| `OpenframeTokenExtractor` | Token acquisition from OpenFrame services | -| `OpenframeTokenRefresher` | Background thread for seamless token renewal | +|-----------|-------------| +| `OpenframeAuthorizationManager` | Singleton manager for securely storing and serving the OpenFrame bearer token | +| `OpenframeEncryptionService` | AES-256-GCM encryption/decryption service that secures tokens at rest | +| `OpenframeTokenExtractor` | Reads and decrypts authentication tokens from an encrypted token file on disk | +| `OpenframeTokenRefresher` | Background thread that periodically re-extracts and refreshes the authentication token | + +--- + +## Two Execution Modes -Obtain your OpenFrame credentials from your platform administrator. Token lifecycle is fully automated once configured. Refer to your environment configuration for connection details. +| Mode | Binary | Description | +|------|--------|-------------| +| **Interactive Shell** | `osqueryi` | Ad-hoc SQL queries against the local system | +| **Daemon** | `osqueryd` | Continuous scheduled query execution | --- ## Documentation -πŸ“š See the [Documentation](./docs/README.md) for comprehensive guides. +πŸ“š See the [Documentation](./docs/README.md) for comprehensive guides, architecture references, and development workflows. -- [Introduction](./docs/getting-started/introduction.md) β€” What is osquery + OpenFrame? -- [Prerequisites](./docs/getting-started/prerequisites.md) β€” System and software requirements -- [Quick Start](./docs/getting-started/quick-start.md) β€” Clone, build, and run -- [First Steps](./docs/getting-started/first-steps.md) β€” Explore tables, packs, FIM, extensions -- [Architecture Overview](./docs/development/architecture/README.md) β€” Module deep-dives -- [Local Development](./docs/development/setup/local-development.md) β€” Build configurations and debugging -- [Contributing Guidelines](./docs/development/contributing/guidelines.md) β€” Code style and PR process +- [Introduction](./docs/getting-started/introduction.md) β€” What is osquery and how does it work +- [Prerequisites](./docs/getting-started/prerequisites.md) β€” System requirements and build tool setup +- [Quick Start](./docs/getting-started/quick-start.md) β€” Build and run in minutes +- [First Steps](./docs/getting-started/first-steps.md) β€” Explore tables, security queries, and OpenFrame integration --- -## Community and Support +## Community & Support -> We do **not** use GitHub Issues or GitHub Discussions. All support and collaboration happens on the **OpenMSP Slack community**. +This project is managed through the **OpenMSP Slack community** β€” GitHub Issues and Discussions are not used. -| Resource | Link | -|---|---| -| πŸ’¬ OpenMSP Community Slack | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | -| 🌐 OpenMSP Website | [https://www.openmsp.ai/](https://www.openmsp.ai/) | -| πŸš€ OpenFrame Platform | [https://openframe.ai](https://openframe.ai) | -| 🦩 Flamingo | [https://flamingo.run](https://flamingo.run) | +- **Slack**: [Join OpenMSP](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) +- **OpenMSP Platform**: [https://www.openmsp.ai/](https://www.openmsp.ai/) +- **Flamingo**: [https://flamingo.run](https://flamingo.run) +- **OpenFrame**: [https://openframe.ai](https://openframe.ai) --- diff --git a/docs/README.md b/docs/README.md index 19d2db24188..efca391b0b9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ -# osquery β€” OpenFrame Edition Documentation +# osquery (OpenFrame-Enhanced) β€” Documentation -Welcome to the documentation for **osquery with OpenFrame** β€” the cross-platform OS instrumentation framework extended with AI-driven MSP automation by [Flamingo](https://flamingo.run) and [OpenFrame](https://openframe.ai). +Welcome to the documentation for the OpenFrame-enhanced distribution of **osquery**, a cross-platform OS instrumentation framework that exposes system state as relational data using SQL. This distribution is part of the [Flamingo](https://flamingo.run) / [OpenFrame](https://openframe.ai) platform for intelligent MSP automation. --- @@ -11,7 +11,6 @@ Welcome to the documentation for **osquery with OpenFrame** β€” the cross-platfo - [Reference Architecture](#-reference-architecture) - [Architecture Diagrams](#-architecture-diagrams) - [Quick Links](#-quick-links) -- [Community](#-community) --- @@ -19,100 +18,115 @@ Welcome to the documentation for **osquery with OpenFrame** β€” the cross-platfo New to osquery? Start here. -| Guide | Description | -|---|---| -| [Introduction](./getting-started/introduction.md) | What is osquery with OpenFrame? Features and target audience | -| [Prerequisites](./getting-started/prerequisites.md) | System requirements, supported platforms, required software | -| [Quick Start](./getting-started/quick-start.md) | Clone, build, and run osquery in under 10 minutes | -| [First Steps](./getting-started/first-steps.md) | Explore virtual tables, scheduled packs, FIM, extensions, and OpenFrame | - -**Recommended reading order:** Introduction β†’ Prerequisites β†’ Quick Start β†’ First Steps +| Document | Description | +|----------|-------------| +| [Introduction](./getting-started/introduction.md) | What is osquery, key features, OpenFrame extensions, and architecture overview | +| [Prerequisites](./getting-started/prerequisites.md) | System requirements, required build tools, and environment verification | +| [Quick Start](./getting-started/quick-start.md) | Clone, build, and run osquery in minutes | +| [First Steps](./getting-started/first-steps.md) | Explore tables, run security queries, create query packs, and configure OpenFrame integration | --- ## πŸ›  Development -Guides for contributors and developers building on or extending osquery. +Guides for contributors and developers working on the codebase. -### Setup +| Document | Description | +|----------|-------------| +| [Development Overview](./development/README.md) | Project structure, technology stack, and navigation index | +| [Environment Setup](./development/setup/environment.md) | IDE configuration (CLion, VS Code), compiler setup, ccache, clang-format | +| [Local Development](./development/setup/local-development.md) | Build modes, running locally, incremental builds, debugging with LLDB/GDB | +| [Architecture Overview](./development/architecture/README.md) | High-level system design, component diagrams, data flow, OpenFrame auth pipeline | +| [Security Best Practices](./development/security/README.md) | Auth patterns, AES-256-GCM encryption, secrets management, TLS configuration | -| Guide | Description | -|---|---| -| [Environment Setup](./development/setup/environment.md) | IDE recommendations, clangd, ccache, editor extensions | -| [Local Development](./development/setup/local-development.md) | Build configurations, debug flags, GDB/LLDB, VS Code debugging | +--- -### Architecture +## πŸ“– Reference Architecture -| Guide | Description | -|---|---| -| [Architecture Overview](./development/architecture/README.md) | High-level module breakdown, runtime lifecycle, design decisions | +Detailed technical documentation for each core module, auto-generated from source code analysis. -### Quality +### Core Runtime -| Guide | Description | -|---|---| -| [Testing Guide](./development/testing/README.md) | GTest/GMock structure, running tests, writing unit and integration tests | -| [Security Best Practices](./development/security/README.md) | Auth patterns, AES-256-GCM, SQL authorizer, TLS, secrets management | +| Document | Description | +|----------|-------------| +| [Core Runtime And Lifecycle](./reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md) | Bootstrapping, flag system, process modes, watchdog model, shutdown coordination | -### Contributing +### Query Engine -| Guide | Description | -|---|---| -| [Contributing Guidelines](./development/contributing/guidelines.md) | Code style, branch naming, commit format, PR process, virtual table creation | +| Document | Description | +|----------|-------------| +| [SQL Engine And Virtual Tables](./reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) | Embedded SQLite, virtual table integration, constraint pushdown, query planning, security authorizer | ---- +### Configuration -## πŸ“– Reference Architecture +| Document | Description | +|----------|-------------| +| [Configuration And Packs](./reference/architecture/configuration-and-packs/configuration-and-packs.md) | Query packs, scheduling, discovery logic, denylisting, hash-based change detection | +| [Config Plugins](./reference/architecture/config-plugins/config-plugins.md) | FilesystemConfigPlugin, OptionsConfigParserPlugin, ViewsConfigParserPlugin, DecoratorsConfigParserPlugin | -Deep technical documentation for each core subsystem β€” generated directly from source code analysis. - -| Module | Description | -|---|---| -| [Core Init And Runtime](./reference/architecture/core-init-and-runtime/core-init-and-runtime.md) | Process bootstrap, flags, watcher/worker model, watchdog | -| [SQL Engine And Virtual Tables](./reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) | SQLite engine, authorizer, virtual table binding, constraint pushdown | -| [Configuration And Packs](./reference/architecture/configuration-and-packs/configuration-and-packs.md) | Config loading, packs, schedulers, decorators, change detection | -| [Eventing Framework And Subscriptions](./reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md) | Publisher/subscriber system, inotify, BPF, FSEvents, ETW | -| [Extensions And IPC](./reference/architecture/extensions-and-ipc/extensions-and-ipc.md) | Apache Thrift IPC, runtime plugin model, UUID routing | -| [Distributed Querying](./reference/architecture/distributed-querying/distributed-querying.md) | Remote SQL orchestration, TLS transport, fleet denylisting | -| [Remote HTTP Client](./reference/architecture/remote-http-client/remote-http-client.md) | Boost.Asio/Beast HTTPS client, TLS handshake, peer verification | -| [Logging And Query Observability](./reference/architecture/logging-and-query-observability/logging-and-query-observability.md) | Differential result tracking, JSON serialization, pluggable backends | -| [Database And Storage Plugins](./reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md) | RocksDB persistent backend, ephemeral in-memory store, key-value interface | -| [Filesystem And Path Utilities](./reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md) | Cross-platform file abstraction, glob resolution, permission enforcement | +### Storage ---- +| Document | Description | +|----------|-------------| +| [Database Backend](./reference/architecture/database-backend/database-backend.md) | RocksDB persistent backend, ephemeral in-memory backend, domain-based key–value storage, schema migration | -## πŸ—Ί Architecture Diagrams +### Observability -Visual Mermaid diagrams for each subsystem are available in: +| Document | Description | +|----------|-------------| +| [Logging And Query Metadata](./reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md) | QueryLogItem, DiffResults, differential vs snapshot logging, JSON serialization | +| [Events Core And Subscriptions](./reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md) | Publisher/subscriber event framework, event persistence, schedule-aware expiration | +| [Osquery Shell Devtooling](./reference/architecture/osquery-shell-devtooling/osquery-shell-devtooling.md) | `osqueryi` interactive shell, meta commands, pretty-print formatting, CPU profiling | -```text -docs/diagrams/architecture/ -``` +### Distribution & Extensions -Diagrams cover all major subsystems including the SQL engine, eventing framework, configuration pipeline, distributed querying, OpenFrame auth layer, and more. Open any `.mmd` file in a Mermaid-compatible viewer. +| Document | Description | +|----------|-------------| +| [Distributed Querying](./reference/architecture/distributed-querying/distributed-querying.md) | Pull-based distributed queries, discovery gating, denylisting, result batching | +| [Distributed TLS Plugin](./reference/architecture/distributed-tls-plugin/distributed-tls-plugin.md) | Secure HTTPS transport for distributed queries via `TLSDistributedPlugin` | +| [Extensions Framework](./reference/architecture/extensions-framework/extensions-framework.md) | Thrift IPC, dynamic plugin injection, UUID-based routing, health monitoring | + +### Networking & Filesystem + +| Document | Description | +|----------|-------------| +| [Remote HTTP Client](./reference/architecture/remote-http-client/remote-http-client.md) | Boost.Asio + Boost.Beast HTTP client, TLS validation, timeout handling, proxy support | +| [Filesystem And Fileops Core](./reference/architecture/filesystem-and-fileops-core/filesystem-and-fileops-core.md) | Cross-platform file abstraction, secure permission validation, globbing, portable stat | --- -## πŸ”— Quick Links +## πŸ—Ί Architecture Diagrams -| Resource | Link | -|---|---| -| [Project README](../README.md) | Main project overview, quick start, and features | -| [Contributing Guide](../CONTRIBUTING.md) | How to contribute code, docs, and virtual tables | -| [License](../LICENSE.md) | License information | +Mermaid-format architecture diagrams are available under `docs/diagrams/architecture/`. Key diagrams include: + +| Diagram | Description | +|---------|-------------| +| `docs/diagrams/architecture/core-runtime-and-lifecycle.mmd` | Core runtime lifecycle flow | +| `docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd` | SQL engine and virtual table layer | +| `docs/diagrams/architecture/extensions-framework.mmd` | Extensions IPC and lifecycle | +| `docs/diagrams/architecture/events-core-and-subscriptions.mmd` | Event publisher/subscriber pipeline | +| `docs/diagrams/architecture/distributed-querying.mmd` | Distributed query execution flow | +| `docs/diagrams/architecture/distributed-tls-plugin.mmd` | TLS-secured distributed transport | +| `docs/diagrams/architecture/database-backend.mmd` | Database backend architecture | +| `docs/diagrams/architecture/configuration-and-packs.mmd` | Configuration and scheduling flow | +| `docs/diagrams/architecture/config-plugins.mmd` | Config plugin chain | +| `docs/diagrams/architecture/remote-http-client.mmd` | HTTP client TLS flow | +| `docs/diagrams/architecture/filesystem-and-fileops-core.mmd` | Filesystem abstraction layer | +| `docs/diagrams/architecture/logging-and-query-metadata.mmd` | Logging pipeline | +| `docs/diagrams/architecture/osquery-shell-devtooling.mmd` | Interactive shell architecture | --- -## πŸ’¬ Community - -> We do **not** use GitHub Issues or GitHub Discussions. All support and collaboration happens on **OpenMSP Slack**. +## πŸ”— Quick Links | Resource | Link | -|---|---| -| OpenMSP Community Slack | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | -| OpenMSP Website | [https://www.openmsp.ai/](https://www.openmsp.ai/) | -| OpenFrame Platform | [https://openframe.ai](https://openframe.ai) | -| Flamingo | [https://flamingo.run](https://flamingo.run) | +|----------|------| +| [Project README](../README.md) | Main project README with Quick Start | +| [Contributing Guide](../CONTRIBUTING.md) | How to contribute, code style, PR process | +| [OpenMSP Community Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | Community support (primary channel) | +| [OpenMSP Platform](https://www.openmsp.ai/) | OpenMSP community platform | +| [Flamingo](https://flamingo.run) | Flamingo AI-powered MSP platform | +| [OpenFrame](https://openframe.ai) | OpenFrame unified MSP tooling platform | --- diff --git a/docs/development/README.md b/docs/development/README.md index 83ce9e027c8..b4124b57c45 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,96 +1,106 @@ # Development Documentation -Welcome to the osquery with OpenFrame development documentation. This section covers everything you need to build, run, test, and contribute to the project. +Welcome to the osquery (OpenFrame-enhanced) development documentation. This section covers everything you need to understand, build, modify, and contribute to this project. --- ## Overview -osquery is a C++ cross-platform OS instrumentation framework. The codebase is organized around a modular, plugin-driven architecture where every major subsystem is independently testable and extensible. - -The **OpenFrame** layer adds authentication and encryption components that integrate with the [Flamingo/OpenFrame MSP platform](https://openframe.ai). +This osquery distribution extends the upstream open-source osquery framework with OpenFrame-specific authentication, token management, and AES-256-GCM encryption services. The codebase is written in **C++17** and built with **CMake**, targeting Linux, macOS, and Windows. --- ## Documentation Index -| Guide | Description | -|---|---| -| [Environment Setup](setup/environment.md) | IDE recommendations, development tools, editor extensions | -| [Local Development](setup/local-development.md) | Clone, build, run locally, debug configuration | -| [Architecture Overview](architecture/README.md) | High-level architecture diagrams and core component descriptions | -| [Security Best Practices](security/README.md) | Auth patterns, encryption, secrets management | -| [Testing Guide](testing/README.md) | Test structure, running tests, writing new tests | -| [Contributing Guidelines](contributing/guidelines.md) | Code style, branch naming, PR process, commit format | +### Setup ---- +| Document | Description | +|----------|-------------| +| [Environment Setup](setup/environment.md) | IDE configuration, development tools, editor extensions | +| [Local Development](setup/local-development.md) | Cloning, building, running locally, debug configuration | -## Technology Stack +### Architecture -| Layer | Technology | -|---|---| -| **Language** | C++17 | -| **Build System** | CMake 3.21+ with Ninja | -| **SQL Engine** | SQLite (embedded, in-memory) | -| **IPC** | Apache Thrift (UNIX sockets / named pipes) | -| **Encryption** | OpenSSL (AES-256-GCM via OpenFrame layer) | -| **Networking** | Boost.Asio + Boost.Beast | -| **Event Systems** | inotify (Linux), FSEvents (macOS), ETW (Windows), BPF (Linux) | -| **Database** | RocksDB (persistent), ephemeral in-memory store | -| **Code Generation** | Python scripts for table schema, API, and amalgamation | -| **Testing** | Google Test + Google Mock | +| Document | Description | +|----------|-------------| +| [Architecture Overview](architecture/README.md) | High-level system design, component diagrams, data flow | + +### Security + +| Document | Description | +|----------|-------------| +| [Security Best Practices](security/README.md) | Auth patterns, encryption, secrets management, vulnerability mitigations | + +### Testing + +| Document | Description | +|----------|-------------| +| [Testing Overview](testing/README.md) | Test structure, running tests, writing new tests, coverage | + +### Contributing + +| Document | Description | +|----------|-------------| +| [Contributing Guidelines](contributing/guidelines.md) | Code style, branch naming, PR process, commit format | --- -## Repository Structure +## Project Structure ```text osquery/ -β”œβ”€β”€ osquery/ # Core C++ source β€” SQL, eventing, config, logging -β”‚ β”œβ”€β”€ core/ # Initialization, flags, watcher/worker model -β”‚ β”œβ”€β”€ sql/ # SQLite engine, virtual tables, authorizer -β”‚ β”œβ”€β”€ config/ # Configuration loading, packs, parsers -β”‚ β”œβ”€β”€ events/ # Eventing framework (publishers/subscribers) -β”‚ β”œβ”€β”€ database/ # Storage backend abstraction (RocksDB, ephemeral) -β”‚ β”œβ”€β”€ distributed/ # Distributed query orchestration -β”‚ β”œβ”€β”€ extensions/ # Thrift-based extension/IPC framework -β”‚ β”œβ”€β”€ remote/ # HTTP client, TLS transport -β”‚ β”œβ”€β”€ logger/ # Logging plugins and observability -β”‚ └── tables/ # Virtual table implementations -β”œβ”€β”€ openframe/ # OpenFrame authentication and encryption layer -β”œβ”€β”€ plugins/ # Logger, config, database, and distributed plugins -β”œβ”€β”€ libraries/ # CMake-managed third-party dependencies -β”œβ”€β”€ tools/ # Codegen scripts, CI tools, formatting -β”œβ”€β”€ tests/ # Integration test suite -└── external/ # Extension examples +β”œβ”€β”€ osquery/ # Core osquery source (C++17) +β”‚ β”œβ”€β”€ core/ # Runtime lifecycle, flags, watchdog +β”‚ β”œβ”€β”€ sql/ # SQL engine and virtual table layer +β”‚ β”œβ”€β”€ config/ # Configuration and query packs +β”‚ β”œβ”€β”€ database/ # RocksDB and ephemeral backends +β”‚ β”œβ”€β”€ events/ # Event publisher/subscriber system +β”‚ β”œβ”€β”€ extensions/ # Thrift-based extension framework +β”‚ β”œβ”€β”€ distributed/ # Distributed query engine +β”‚ β”œβ”€β”€ logger/ # Logging and query metadata +β”‚ └── remote/ # HTTP client for TLS endpoints +β”œβ”€β”€ openframe/ # OpenFrame-specific extensions (C++) +β”‚ β”œβ”€β”€ openframe_authorization_manager.* +β”‚ β”œβ”€β”€ openframe_encryption_service.* +β”‚ β”œβ”€β”€ openframe_token_extractor.* +β”‚ └── openframe_token_refresher.* +β”œβ”€β”€ plugins/ # Config, logger, database plugins +β”œβ”€β”€ libraries/ # Vendored third-party libraries (CMake) +β”œβ”€β”€ tools/ # Code generation, CI scripts +β”œβ”€β”€ external/ # Extension SDK examples +└── tests/ # Integration tests ``` --- -## Quick Commands +## Technology Stack -```bash -# Configure build -cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo +| Layer | Technology | +|-------|-----------| +| **Language** | C++17 | +| **Build System** | CMake β‰₯ 3.21 | +| **SQL Engine** | SQLite (embedded) | +| **Persistent Database** | RocksDB | +| **IPC / Extensions** | Apache Thrift | +| **Networking** | Boost.Asio + Boost.Beast | +| **Encryption** | OpenSSL (AES-256-GCM) | +| **Logging** | Google glog | +| **Testing** | Google Test (gtest) | +| **Flags** | Google gflags | +| **Documentation Pipeline** | VoltAgent + Anthropic AI | -# Build everything -cmake --build build --parallel $(nproc) +--- -# Run interactive shell -./build/osquery/osqueryi +## Quick Navigation -# Run tests -cmake --build build --target osquery_tests -cd build && ctest --output-on-failure -``` +- New to the codebase? Start with [Architecture Overview](architecture/README.md). +- Setting up your machine? See [Environment Setup](setup/environment.md). +- Ready to code? Follow [Local Development](setup/local-development.md). +- Writing a fix or feature? Read [Contributing Guidelines](contributing/guidelines.md). +- Security concern? See [Security Best Practices](security/README.md). --- -## Getting Help - -All development discussions happen in the **OpenMSP Slack community**: - -- [Join OpenMSP Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) -- [https://www.openmsp.ai/](https://www.openmsp.ai/) +## Community -> We do not use GitHub Issues or GitHub Discussions. All support and collaboration is on Slack. +All development discussions happen in the [OpenMSP Slack community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA). This project does not use GitHub Issues or GitHub Discussions. diff --git a/docs/development/architecture/README.md b/docs/development/architecture/README.md index a7b03cb1fce..db2125bf672 100644 --- a/docs/development/architecture/README.md +++ b/docs/development/architecture/README.md @@ -1,239 +1,234 @@ # Architecture Overview -osquery is built as a modular, plugin-driven system. Each subsystem is independently composable, testable, and extensible. The codebase is written in C++17 and targets Linux, macOS, and Windows. +This document describes the high-level architecture of osquery (OpenFrame-enhanced), including core components, data flow, and the OpenFrame authentication layer. --- -## High-Level Architecture +## System Architecture + +osquery is structured as a modular, plugin-driven runtime. All major subsystems communicate through a **Registry** β€” a pluggable dispatch mechanism. The **Core Runtime** bootstraps every subsystem in a defined order and coordinates their lifecycle. ```mermaid -graph TD - CLI["osqueryi / osqueryd"] --> Core["Core Init And Runtime"] - Core --> Config["Configuration And Packs"] +flowchart TD + Core["Core Runtime And Lifecycle"] --> Config["Configuration And Packs"] Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework"] - Core --> Logging["Logging And Observability"] - Core --> DB["Database And Storage Plugins"] - Core --> Dist["Distributed Querying"] - Core --> Ext["Extensions And IPC"] - Core --> HTTP["Remote HTTP Client"] - Core --> OF["OpenFrame Auth Layer"] + Core --> DB["Database Backend"] + Core --> Logging["Logging And Query Metadata"] + Core --> Events["Events Core And Subscriptions"] + Core --> Extensions["Extensions Framework"] + Core --> Distributed["Distributed Querying"] + Config --> SQL Config --> Events + Config --> Logging + SQL --> Logging + SQL --> DB + Events --> DB - Dist --> SQL - Dist --> HTTP - Ext --> SQL - Ext --> DB - OF --> HTTP + Events --> Logging + + Distributed --> SQL + Distributed --> DB + Distributed --> Logging + + Extensions --> SQL + Extensions --> Config + Extensions --> Distributed + + OpenFrame["OpenFrame Auth Layer"] --> Core + OpenFrame --> Distributed ``` --- ## Core Components -| Module | Location | Description | -|---|---|---| -| **Core Init And Runtime** | `osquery/core/` | Process bootstrap, flags, watchdog, shutdown | -| **SQL Engine And Virtual Tables** | `osquery/sql/` | SQLite engine, authorizer, virtual table binding | -| **Configuration And Packs** | `osquery/config/` | Config loading, packs, parsers, scheduled queries | -| **Eventing Framework** | `osquery/events/` | Pub/sub system for OS events (inotify, BPF, ETW) | -| **Logging And Observability** | `osquery/logger/` | Differential logging, JSON serialization, logger plugins | -| **Database And Storage Plugins** | `osquery/database/` | Key-value storage (RocksDB, ephemeral) | -| **Distributed Querying** | `osquery/distributed/` | Remote SQL orchestration across fleets | -| **Extensions And IPC** | `osquery/extensions/` | Apache Thrift-based extension framework | -| **Remote HTTP Client** | `osquery/remote/` | Boost.Asio/Beast HTTPS client with TLS | -| **OpenFrame Auth Layer** | `openframe/` | JWT token management, AES-256-GCM encryption | -| **Virtual Tables** | `osquery/tables/` | 300+ OS-specific table implementations | +| Module | Path | Responsibility | +|--------|------|---------------| +| **Core Runtime** | `osquery/core/` | Bootstrapping, flags, watchdog, lifecycle, signal handling | +| **SQL Engine** | `osquery/sql/` | Embedded SQLite, virtual table layer, query planning | +| **Configuration** | `osquery/config/` | Query packs, scheduling, discovery, denylisting | +| **Database Backend** | `osquery/database/` | RocksDB persistence and ephemeral in-memory storage | +| **Logging** | `osquery/logger/` | Result diffing, QueryLogItem, serialization, logger plugins | +| **Events** | `osquery/events/` | Publisher/subscriber system, event persistence | +| **Extensions** | `osquery/extensions/` | Thrift IPC, extension registration, health monitoring | +| **Distributed** | `osquery/distributed/` | Remote query pull, execution, result submission | +| **Remote HTTP** | `osquery/remote/` | Boost.Asio/Beast HTTP client for TLS endpoints | +| **OpenFrame Auth** | `openframe/` | Token management, AES-256-GCM encryption, background refresh | --- -## Runtime Lifecycle +## Query Execution Flow ```mermaid sequenceDiagram - participant Main - participant Initializer - participant Registry - participant Config - participant Extensions - participant Events - - Main->>Initializer: Construct(argc, argv) - Initializer->>Initializer: Parse flags - Initializer->>Registry: registryAndPluginInit() - Initializer->>Extensions: startExtensionManager() - Initializer->>Config: load() - Initializer->>Events: attachEvents() - Initializer->>Initializer: start() + participant Sched as "Scheduler" + participant SQL as "SQL Engine" + participant VT as "Virtual Tables" + participant DB as "Database Backend" + participant Log as "Logger Plugin" + + Sched->>SQL: Submit SQL query + SQL->>VT: Execute xFilter / xNext + VT-->>SQL: Return rows + SQL->>DB: getPreviousQueryResults() + DB-->>SQL: Previous rows + SQL->>SQL: Compute DiffResults (added/removed) + SQL->>Log: Emit QueryLogItem + SQL->>DB: saveQueryResults() ``` -osquery can operate in four runtime modes: - -| Mode | Binary | Description | -|---|---|---| -| **Interactive Shell** | `osqueryi` | REPL for ad-hoc SQL queries | -| **Daemon** | `osqueryd` | Background daemon with scheduled queries | -| **Watcher** | Internal | Supervisor process that monitors the worker | -| **Extension** | External | External plugin process communicating via Thrift | - --- -## SQL Engine and Virtual Tables +## Event Pipeline -The heart of osquery is its SQL engine β€” an in-memory SQLite instance hardened with a strict authorizer. +Event-based tables (file events, process events, network events) use a publish/subscribe model: ```mermaid -graph TD - Query["SQL Query"] --> SQLPlugin["SQLiteSQLPlugin"] - SQLPlugin --> DBManager["SQLiteDBManager"] - DBManager --> DBInst["SQLiteDBInstance"] - DBInst --> SQLiteCore["In-Memory SQLite Engine"] - SQLiteCore --> VTabModule["sqlite3_module"] - VTabModule --> VirtualTable["VirtualTable Wrapper"] - VirtualTable --> TablePlugin["TablePlugin"] - TablePlugin --> OS["Operating System Data"] +flowchart TD + Publisher["Event Publisher Plugin"] --> Subscription["Subscription"] + Subscription --> Subscriber["EventSubscriberPlugin"] + Subscriber --> DB["Database Backend"] + Subscriber --> VTable["Event Virtual Table"] + VTable --> Query["SELECT FROM event_table"] + Query --> Logging["Logger Plugin"] ``` -**Key properties:** -- All queries run against an in-memory SQLite database -- A strict authorizer allowlists only safe SQL opcodes -- Virtual tables are attached dynamically from the registry -- Constraint pushdown and projection optimization reduce OS calls +### Event Model Key Facts ---- +- Publishers run in dedicated threads, collecting OS-level events +- Subscribers transform events into rows and persist them to the database +- SQL queries retrieve time-bounded results via `generateRows()` +- Event expiration windows are schedule-aware -## Eventing Framework +--- -Event-driven monitoring is handled by a publish/subscribe system: +## Distributed Query Flow ```mermaid -graph TD - Publisher["EventPublisher"] --> EventFactory["EventFactory"] - Subscriber["EventSubscriber"] --> EventFactory - EventFactory --> Callback["EventCallback"] - Callback --> Storage["Database Storage"] - SQL["SELECT from events tables"] --> Subscriber +flowchart TD + Server["Remote Server"] --> TLSPlugin["Distributed TLS Plugin"] + TLSPlugin --> DistEngine["Distributed Querying"] + DistEngine --> SQL["SQL Engine"] + SQL --> Results["DistributedQueryResult"] + Results --> TLSPlugin + TLSPlugin --> Server ``` -| Platform | Event Technology | -|---|---| -| Linux | inotify, BPF, Audit netlink | -| macOS | FSEvents, EndpointSecurity, OpenBSM | -| Windows | ETW, USN Journal, Windows Event Log | +Pull-based model β€” osquery polls the server for pending queries, executes them locally, and submits results back over TLS. --- -## Configuration and Scheduling +## Extensions Framework + +Extensions allow third-party processes to register new tables, config providers, and loggers without modifying the osquery binary: ```mermaid -graph TD - Source["Config Source"] --> ConfigCore["Config Singleton"] - ConfigCore --> Parsers["ConfigParserPlugins"] - Parsers --> Schedule["Scheduled Queries"] - Schedule --> Scheduler["Query Scheduler"] - Scheduler --> SQL["SQL Engine"] - SQL --> Logging["Logging And Observability"] +flowchart TD + Core["osquery Core"] --> Manager["Extension Manager"] + Manager --> Registry["RegistryFactory"] + ExtA["Extension Process A"] -->|"registerExtension()"| Manager + ExtB["Extension Process B"] -->|"registerExtension()"| Manager + Core -->|"callExtension()"| ExtA + Core -->|"callExtension()"| ExtB ``` -Configuration sources: -- **FilesystemConfigPlugin** β€” local JSON file with `.d/` fragment support -- **TLS Config Plugin** β€” remote configuration over HTTPS -- **Extension Config Plugin** β€” configuration provided by an extension process +Communication uses **Apache Thrift** over UNIX domain sockets (Linux/macOS) or named pipes (Windows). --- ## OpenFrame Authentication Layer -The `openframe/` directory contains the Flamingo/OpenFrame integration: +The OpenFrame extensions add a secure authentication pipeline: ```mermaid -graph LR - Provider["AuthorizationManagerProvider"] --> Manager["AuthorizationManager"] - Extractor["TokenExtractor"] --> Manager - Refresher["TokenRefresher"] --> Extractor - Manager --> Token["JWT Bearer Token"] - Token --> HTTP["Remote HTTP Client"] - EncSvc["EncryptionService"] --> OpenSSL["OpenSSL AES-256-GCM"] +flowchart LR + TokenFile["Encrypted Token File\n/etc/openframe/token.enc"] --> Extractor["OpenframeTokenExtractor"] + EncSvc["OpenframeEncryptionService\nAES-256-GCM"] --> Extractor + Extractor --> Refresher["OpenframeTokenRefresher\n(background thread)"] + Refresher --> Manager["OpenframeAuthorizationManager\n(singleton)"] + Manager --> OutboundReq["Authenticated HTTP Requests"] ``` | Component | Responsibility | -|---|---| -| `OpenframeAuthorizationManager` | Singleton token store with provider-controlled lifecycle | -| `OpenframeAuthorizationManagerProvider` | Sole factory for the token manager | -| `OpenframeEncryptionService` | AES-256-GCM decryption via OpenSSL | -| `OpenframeTokenExtractor` | Fetches tokens from OpenFrame services | -| `OpenframeTokenRefresher` | Background thread for token renewal | - ---- +|-----------|---------------| +| `OpenframeEncryptionService` | AES-256-GCM decryption with OpenSSL, Base64 decode | +| `OpenframeTokenExtractor` | Reads encrypted token file, returns plaintext token | +| `OpenframeTokenRefresher` | Background thread that periodically calls `extractToken()` | +| `OpenframeAuthorizationManager` | Singleton that stores and serves the current bearer token | +| `OpenframeAuthorizationManagerProvider` | Only class permitted to construct the manager (friend pattern) | -## Extension System - -Extensions communicate with osquery core via Apache Thrift over UNIX domain sockets (Linux/macOS) or named pipes (Windows): +### Token Lifecycle ```mermaid -graph LR - Core["osquery Core"] --> Manager["Extension Manager"] - Manager --> Registry["RegistryFactory"] - ExtProc["Extension Process"] --> Manager - Manager --> ExtProc +flowchart TD + Start["Process Startup"] --> Init["OpenframeTokenExtractor initialized"] + Init --> Extract["extractToken() β€” read + decrypt file"] + Extract --> Store["AuthorizationManager.updateToken()"] + Store --> Active["Token active β€” used in HTTP headers"] + Active --> Refresh["OpenframeTokenRefresher wakes up (interval)"] + Refresh --> Extract ``` -Extensions can provide: -- Custom virtual tables -- Logger plugins -- Config plugins -- Distributed plugins +--- + +## Database Domains + +The Database Backend organizes all persistent state into well-known domains: + +| Domain | Used By | +|--------|---------| +| `configurations` | Configuration And Packs | +| `queries` | Query state diffing | +| `events` | Events Core | +| `logs` | Logger plugins | +| `carves` | File carver | +| `distributed` | Distributed Querying | +| `query_performance` | Performance tracking | --- -## Data Flow: Query Execution to Log +## Process Modes -```mermaid -sequenceDiagram - participant Scheduler - participant SQL as SQL Engine - participant DB as Database - participant Logger - - Scheduler->>SQL: Execute scheduled query - SQL->>SQL: Run against virtual tables - SQL-->>Scheduler: QueryData (current results) - Scheduler->>DB: Load previous results - DB-->>Scheduler: Previous QueryData - Scheduler->>Scheduler: Compute DiffResults - Scheduler->>Logger: logQueryLogItem(diff) - Logger->>Logger: Serialize to JSON - Logger-->>Logger: Forward to backend sink -``` +osquery runs in several distinct modes: + +| Mode | Binary | Description | +|------|--------|-------------| +| **Shell** | `osqueryi` | Interactive SQL shell for ad-hoc queries | +| **Daemon** | `osqueryd` | Scheduled query execution in background | +| **Watcher** | (internal) | Supervises worker process and restarts on failure | +| **Worker** | (internal) | Actual query execution, spawned by watcher | +| **Extension** | custom binary | External process registering new capabilities | --- ## Key Design Decisions -| Decision | Rationale | -|---|---| -| **In-memory SQLite** | Zero persistent SQL state; each query is a fresh execution | -| **Plugin/Registry pattern** | All major subsystems (loggers, config, tables) are hot-swappable | -| **Watcher/Worker isolation** | Worker crashes don't bring down the watchdog supervisor | -| **Thrift for extensions** | Language-agnostic, versioned, cross-platform RPC | -| **Differential logging** | Only changed rows are logged, dramatically reducing volume | -| **AES-256-GCM encryption** | OpenFrame credentials protected with authenticated encryption | +1. **Registry-based plugins** β€” All major subsystems (SQL, config, logger, database, distributed) are pluggable via the Registry pattern, enabling replacement without core changes. + +2. **Virtual table layer** β€” OS data is never stored unless queried; virtual tables are instantiated on demand, keeping memory usage low. + +3. **Differential logging** β€” Only changed rows (added/removed) are logged by default, reducing output volume significantly. + +4. **Out-of-process extensions** β€” Thrift IPC isolates extension crashes from the core process. + +5. **Watchdog supervision** β€” Watcher/worker separation ensures daemon resilience; a crashing worker is restarted automatically. + +6. **OpenFrame auth separation** β€” The authentication layer is implemented as a separate C++ module under `openframe/`, keeping it isolated from the core osquery codebase. --- ## Reference Documentation -For deep dives into each module, see the reference architecture docs: +For detailed per-module documentation, see the auto-generated reference architecture docs: -- [Core Init And Runtime](./reference/architecture/core-init-and-runtime/core-init-and-runtime.md) +- [Core Runtime And Lifecycle](./reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md) - [SQL Engine And Virtual Tables](./reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) - [Configuration And Packs](./reference/architecture/configuration-and-packs/configuration-and-packs.md) -- [Eventing Framework And Subscriptions](./reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md) -- [Extensions And IPC](./reference/architecture/extensions-and-ipc/extensions-and-ipc.md) +- [Database Backend](./reference/architecture/database-backend/database-backend.md) +- [Events Core And Subscriptions](./reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md) +- [Extensions Framework](./reference/architecture/extensions-framework/extensions-framework.md) - [Distributed Querying](./reference/architecture/distributed-querying/distributed-querying.md) +- [Logging And Query Metadata](./reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md) - [Remote HTTP Client](./reference/architecture/remote-http-client/remote-http-client.md) -- [Logging And Query Observability](./reference/architecture/logging-and-query-observability/logging-and-query-observability.md) -- [Database And Storage Plugins](./reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md) -- [Filesystem And Path Utilities](./reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md) diff --git a/docs/development/contributing/guidelines.md b/docs/development/contributing/guidelines.md deleted file mode 100644 index de082c732cb..00000000000 --- a/docs/development/contributing/guidelines.md +++ /dev/null @@ -1,272 +0,0 @@ -# Contributing Guidelines - -Thank you for contributing to osquery with OpenFrame! This guide covers code style, branching, commit messages, and the pull request process. - ---- - -## Community First - -All collaboration happens on the **OpenMSP Slack community** β€” not GitHub Issues or GitHub Discussions. - -- πŸ’¬ **Join Slack**: [OpenMSP Community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) -- 🌐 **OpenMSP**: [https://www.openmsp.ai/](https://www.openmsp.ai/) - -Before starting significant work, please discuss your changes in the Slack community to align with the team's roadmap. - ---- - -## Code Style and Conventions - -### C++ Standards - -- Use **C++17** features where appropriate -- Follow the existing code style in each file you modify -- All new code must pass `clang-format` with the repository's `.clang-format` config - -### Formatting - -osquery enforces `clang-format`. Run it before every commit: - -```bash -# Format a single file -clang-format -i path/to/your/file.cpp - -# Format all changed files (compared to main branch) -git diff --name-only main | grep -E '\.(cpp|h)$' | xargs clang-format -i - -# Check without modifying -clang-format --dry-run --Werror path/to/your/file.cpp -``` - -### Naming Conventions - -| Item | Convention | Example | -|---|---|---| -| Classes | `PascalCase` | `EventSubscriberPlugin` | -| Methods | `camelCase` | `generateRows()` | -| Member variables | `snake_case_` (trailing underscore) | `running_` | -| Constants | `kPascalCase` | `kSQLOpcodes` | -| Macros | `UPPER_SNAKE_CASE` | `DECLARE_FLAG` | -| Namespaces | `snake_case` | `osquery` | -| Files | `snake_case.cpp` / `snake_case.h` | `event_subscriber.cpp` | - -### Code Organization - -- Keep headers (`*.h`) as minimal as possible β€” forward declare where possible -- Use the `osquery` namespace for all production code -- Place tests in `tests/` subdirectories alongside the source -- New virtual tables go in `osquery/tables//` - -### Include Order - -Follow this include order, with blank lines between groups: - -```cpp -// 1. Standard library -#include -#include -#include - -// 2. Third-party libraries -#include -#include - -// 3. osquery headers -#include "osquery/core/core.h" -#include "osquery/sql/sql.h" - -// 4. Local headers (same directory) -#include "my_local_header.h" -``` - ---- - -## Branch Naming - -| Type | Pattern | Example | -|---|---|---| -| Feature | `feature/` | `feature/bpf-socket-events` | -| Bug fix | `fix/` | `fix/config-refresh-race` | -| OpenFrame integration | `openframe/` | `openframe/token-refresh-retry` | -| Documentation | `docs/` | `docs/virtual-table-guide` | -| Refactor | `refactor/` | `refactor/sql-authorizer` | -| Test | `test/` | `test/events-integration` | -| Release | `release/v` | `release/v5.13.0` | - -Always branch from the latest `main`: - -```bash -git checkout main -git pull origin main -git checkout -b feature/my-new-feature -``` - ---- - -## Commit Message Format - -Use the following format for all commits: - -```text -(): - - - - -``` - -### Types - -| Type | When to Use | -|---|---| -| `feat` | New feature or capability | -| `fix` | Bug fix | -| `docs` | Documentation changes only | -| `style` | Code formatting, no logic change | -| `refactor` | Code refactoring without behavior change | -| `test` | Adding or fixing tests | -| `perf` | Performance improvement | -| `chore` | Build, CI, tooling changes | -| `openframe` | OpenFrame platform-specific changes | - -### Scopes - -| Scope | Area | -|---|---| -| `core` | Core init and runtime | -| `sql` | SQL engine and virtual tables | -| `config` | Configuration and packs | -| `events` | Eventing framework | -| `logger` | Logging and observability | -| `db` | Database and storage | -| `distributed` | Distributed querying | -| `extensions` | Extension IPC | -| `http` | Remote HTTP client | -| `openframe` | OpenFrame auth layer | -| `tables` | Virtual table implementations | - -### Examples - -```text -feat(events): add BPF socket event publisher for Linux - -Implements a new BPF-based publisher that captures socket connect/accept -events and exposes them via the bpf_socket_events virtual table. - -Closes #1234 -``` - -```text -fix(openframe): handle token refresh failure with exponential backoff - -When OpenframeTokenRefresher encounters a network error, it now retries -with exponential backoff instead of immediately stopping the refresh loop. -``` - -```text -test(config): add pack discovery query unit tests - -Adds unit tests for the discovery query evaluation logic in Pack::shouldPackExecute() -covering platform, version, and shard constraints. -``` - ---- - -## Pull Request Process - -### Before Submitting - -- [ ] Branch is up-to-date with `main` -- [ ] All tests pass: `cd build && ctest --output-on-failure` -- [ ] Code is formatted: `clang-format --dry-run --Werror` -- [ ] Copyright headers are present on new files -- [ ] New virtual tables have integration tests in `tests/integration/tables/` -- [ ] OpenFrame-specific changes include updated documentation - -### PR Title - -Use the same format as commit messages: - -```text -feat(sql): add query result caching for repeated virtual table scans -``` - -### PR Description Template - -```markdown -## Summary - - -## Changes - - -## Testing - - -## Platform Support - - -## Checklist -- [ ] Tests pass (ctest) -- [ ] clang-format applied -- [ ] Documentation updated (if applicable) -- [ ] Discussed in OpenMSP Slack (if significant change) -``` - ---- - -## Copyright Headers - -All new source files must include a copyright header. Use the format already present in existing source files: - -```cpp -/** - * Copyright (c) 2014-present, The osquery authors - * - * This source code is licensed in accordance with the terms specified in - * the LICENSE file found in the root directory of this source tree. - */ -``` - -The CI script `tools/ci/scripts/check_copyright_headers.py` enforces this on all pull requests. - ---- - -## Review Checklist for Reviewers - -When reviewing a PR: - -- [ ] Logic is correct and handles error paths -- [ ] No secrets or credentials in source -- [ ] SQL inputs validated (if applicable) -- [ ] Thread safety considered for shared state -- [ ] Platform-specific code properly guarded with `#ifdef` -- [ ] Tests added for new functionality -- [ ] No debug/temporary code left in -- [ ] Commit messages follow format convention -- [ ] Performance impact considered for hot paths (scheduler, SQL engine) - ---- - -## Adding a New Virtual Table - -1. Define the table schema in `osquery/tables//.table` -2. Run the code generator: - ```bash - python3 tools/codegen/gentable.py osquery/tables//.table - ``` -3. Implement the `generate()` method in `.cpp` -4. Register in the CMakefile for your category -5. Add an integration test in `tests/integration/tables/.cpp` -6. Test locally: - ```bash - cmake --build build --target osqueryi - ./build/osquery/osqueryi - osquery> SELECT * FROM ; - ``` - ---- - -## Code of Conduct - -Be respectful, collaborative, and constructive. All contributors are expected to maintain a professional and welcoming environment in both Slack and code reviews. diff --git a/docs/development/security/README.md b/docs/development/security/README.md index 87ae6702cae..f3820920dff 100644 --- a/docs/development/security/README.md +++ b/docs/development/security/README.md @@ -1,239 +1,227 @@ # Security Best Practices -This guide covers security patterns used in osquery with OpenFrame, including authentication, encryption, secrets management, input validation, and common vulnerability mitigations. +This document covers authentication patterns, encryption, secrets management, input validation, and security testing guidelines for osquery (OpenFrame-enhanced). --- ## Authentication and Authorization -### OpenFrame JWT Token Management +### OpenFrame Token Authentication -The OpenFrame layer uses a provider-controlled singleton pattern to manage JWT authentication tokens: +The OpenFrame authentication system uses a **bearer token** model secured by AES-256-GCM encryption at rest. + +**Key principles:** + +- Tokens are never stored in plaintext on disk. +- The encrypted token file is read by `OpenframeTokenExtractor`, which decrypts it using a secret key passed at construction time. +- The plaintext token is held only in memory via the `OpenframeAuthorizationManager` singleton. +- A background thread (`OpenframeTokenRefresher`) rotates the in-memory token periodically, limiting exposure if the in-memory token is somehow leaked. ```mermaid -graph LR - Provider["AuthorizationManagerProvider"] --> Manager["AuthorizationManager"] - Extractor["TokenExtractor"] --> Manager - Refresher["TokenRefresher"] --> Extractor - Manager --> Token["Bearer JWT Token"] - Token --> HTTP["Outbound HTTPS Requests"] +flowchart LR + Disk["Encrypted Token File\n(AES-256-GCM at rest)"] --> Extractor["OpenframeTokenExtractor\n(read + decrypt)"] + SecretKey["Secret Key\n(env var / secrets manager)"] --> EncSvc["OpenframeEncryptionService"] + EncSvc --> Extractor + Extractor --> Manager["AuthorizationManager\n(in-memory only)"] + Manager --> Request["Outbound HTTP Request\nAuthorization: Bearer ..."] ``` -**Key principles:** - -- `OpenframeAuthorizationManager` is **non-copyable** (`boost::noncopyable`) β€” tokens cannot be accidentally duplicated -- Only `OpenframeAuthorizationManagerProvider` can construct or destroy the manager (private constructor/destructor) -- The `OpenframeTokenRefresher` runs in a dedicated background thread using `std::atomic` for lock-free lifecycle control -- Token updates and reads are centralized through `updateToken()` and `getToken()` +### Security Rules for Token Handling -**Never:** -- Store JWT tokens in plain text files or environment variables without appropriate OS-level access controls -- Log token values, even at debug level -- Pass tokens as command-line arguments +- **Do NOT** log the token value at any severity level. +- **Do NOT** print the secret key or token in error messages. +- **Do NOT** store decrypted tokens in configuration files or environment variable dumps. +- **ALWAYS** use `OpenframeAuthorizationManagerProvider` to access the manager β€” direct construction is private by design. +- **ALWAYS** call `stop()` on `OpenframeTokenRefresher` before destroying it to avoid dangling threads. --- ## Encryption -### AES-256-GCM via OpenSSL - -The `OpenframeEncryptionService` provides symmetric encryption using AES-256-GCM β€” an authenticated encryption scheme: - -```cpp -// Initialize with a 256-bit secret key -OpenframeEncryptionService enc("my-32-byte-secret-key-goes-here!"); - -// Decrypt a Base64-encoded AES-256-GCM ciphertext -std::string plaintext = enc.decrypt(ciphertext_base64); -``` +### AES-256-GCM (OpenframeEncryptionService) -**Encryption properties:** +The `OpenframeEncryptionService` provides: -| Property | Value | -|---|---| +| Parameter | Value | +|-----------|-------| | Algorithm | AES-256-GCM | -| Key size | 32 bytes (256-bit) | -| IV (nonce) size | 12 bytes (96-bit) | -| Authentication tag | 16 bytes (128-bit) | +| Key size | 256 bits (32 bytes) | +| IV size | 96 bits (12 bytes) | +| Auth tag | 128 bits (16 bytes) | | Encoding | Base64 | -**Security rules:** -- The symmetric key must be exactly 32 bytes -- GCM authentication tag is verified on every decryption β€” tampered ciphertext throws `std::runtime_error` -- Never reuse nonces for the same key -- Store keys using OS keystore mechanisms or secret management systems, not in source code +**Authentication tag verification** is enforced by AES-GCM β€” any tampered ciphertext will cause decryption to fail with a `std::runtime_error`. This provides both **confidentiality and integrity**. ---- - -## SQL Security β€” The Authorizer - -osquery's embedded SQLite engine includes a strict **authorizer** (`sqliteAuthorizer`) that allowlists only safe SQL operations: +#### Correct Usage -```mermaid -graph TD - Prepare["sqlite3_prepare_v2"] --> Authorizer["sqliteAuthorizer"] - Authorizer -->|"Allowed"| Continue["Execute Statement"] - Authorizer -->|"Denied"| Reject["SQLITE_DENY"] +```cpp +// Initialize with a 256-bit (32-byte) secret key +OpenframeEncryptionService enc_service("your-32-byte-secret-key-here!!"); + +try { + std::string plaintext = enc_service.decrypt(encrypted_payload_base64); + // Use plaintext; do NOT log it +} catch (const std::runtime_error& e) { + LOG(ERROR) << "Token decryption failed: " << e.what(); + // Do NOT include the ciphertext or key in the log message +} ``` -**Explicitly allowlisted:** -- `SELECT`, `READ` operations -- Controlled `INSERT`, `UPDATE`, `DELETE` (virtual tables only) -- Virtual table creation and drop -- Limited `PRAGMA` commands +#### Security Prohibitions -**Explicitly denied:** -- `SQLITE_ATTACH` β€” cannot attach external database files -- Any non-allowlisted opcode +```cpp +// ❌ NEVER do this β€” logs the plaintext token +LOG(INFO) << "Token: " << plaintext; + +// ❌ NEVER do this β€” hardcodes the secret key in source code +OpenframeEncryptionService enc("hardcoded-secret-key-bad!!!!!"); -This prevents osquery from being abused to read arbitrary files via SQLite's `ATTACH` mechanism or execute unsafe operations. +// βœ… Correct β€” inject from a secure source +std::string secret = getFromSecretsManager(); +OpenframeEncryptionService enc(secret); +``` --- -## Input Validation +## Secrets Management -### SQL Query Validation +### Rules -All SQL submitted through the distributed querying or config channels is: -1. Parsed by the SQLite authorizer before execution -2. Validated against the virtual table schema -3. Size-limited by configuration constraints +1. **Never hardcode secrets** β€” Secret keys, API tokens, and encryption keys must never appear in source code, configuration files, or commit history. -### Configuration Validation +2. **Use environment variables or a secrets manager** β€” Provide the OpenFrame secret key via: + - A secrets manager (HashiCorp Vault, AWS Secrets Manager, etc.) + - A secure environment variable injected at process start time + - A protected file with `chmod 600` ownership-restricted to the osquery process user -The `Config` singleton enforces: -- JSON max depth limits -- JSON max document size limits -- Comment stripping (non-standard JSON is rejected) -- Hash-based change detection to prevent replay attacks +3. **Restrict file permissions** on token files: -### Extension Registration +```bash +# Token file must be readable only by the osquery service user +sudo chown osquery:osquery /etc/openframe/token.enc +sudo chmod 600 /etc/openframe/token.enc +``` -Extension processes are validated during registration: -- SDK version compatibility is enforced -- Duplicate extension names are rejected -- Route UUIDs are generated server-side (not client-controlled) +4. **Rotate tokens regularly** β€” The `OpenframeTokenRefresher` background thread handles periodic rotation of the in-memory token. Ensure your token files are also rotated at the platform level. ---- +5. **Audit secret access** β€” Log access to the token file path (not its content) to detect unauthorized access attempts. -## Secrets Management +--- -### Environment-Level Secrets +## Input Validation and Sanitization -Never embed secrets in: -- Source code -- CMake configuration -- Build scripts -- Log output +### SQL Injection Prevention -Use OS-level secrets management: +osquery's SQL engine uses an **authorizer** (`sqliteAuthorizer`) that enforces a strict allowlist of SQL operations: -| Platform | Recommended Tool | -|---|---| -| Linux | systemd credentials, HashiCorp Vault, AWS Secrets Manager | -| macOS | Keychain, AWS Secrets Manager | -| Windows | Windows Credential Manager, Azure Key Vault | +| Allowed | Denied | +|---------|--------| +| `SELECT`, `READ` | `ATTACH DATABASE` | +| `INSERT`, `UPDATE`, `DELETE` | Unauthorized `PRAGMA` calls | +| `CREATE/DROP TABLE`, `CREATE/DROP VIEW` | File write operations | +| `FUNCTION` calls | Anything not on the allowlist | -### osquery Flag Files +When writing extension tables or custom plugins, follow these guidelines: -Sensitive flags (TLS certificates, enrollment secrets) should be stored in a flagfile with restricted permissions: +```cpp +// βœ… Validate all column constraint values before use +if (context.constraints["pid"].existsAndMatches("1")) { + // safe to process +} -```bash -# Create flagfile with restricted permissions -sudo touch /etc/osquery/osquery.secret.flags -sudo chmod 600 /etc/osquery/osquery.secret.flags -sudo chown root:root /etc/osquery/osquery.secret.flags - -# Write sensitive flags -echo "--tls_server_certs=/etc/osquery/server.pem" | sudo tee -a /etc/osquery/osquery.secret.flags -echo "--enroll_secret_path=/etc/osquery/enroll.secret" | sudo tee -a /etc/osquery/osquery.secret.flags +// βœ… Use parameterized operations, not string concatenation +// ❌ NEVER build SQL strings from user input ``` -Reference the flagfile: +### Extension Plugin Input Validation -```bash -sudo ./osqueryd --flagfile=/etc/osquery/osquery.secret.flags -``` +When implementing a `TablePlugin`: + +- Validate all incoming `QueryContext` constraints before acting on them. +- Reject negative or out-of-range integer constraints. +- Sanitize filesystem paths to prevent path traversal. +- Limit result set sizes to prevent memory exhaustion. --- -## TLS / HTTPS Communication +## Network Security (TLS Configuration) -### Remote HTTP Client Security +The Remote HTTP Client (`osquery/remote/`) supports comprehensive TLS options: -The `Remote HTTP Client` module (`osquery/remote/`) enforces: -- TLS by default for all remote communication -- Peer certificate verification -- Configurable cipher suites -- Custom CA certificate paths -- Timeout enforcement (no hanging connections) +| Option | Recommendation | +|--------|---------------| +| `always_verify_peer` | Set to `true` in production | +| Custom CA path | Pin to your organization's CA bundle | +| TLS version | Use TLS 1.2 or higher | +| Client certificate | Use mutual TLS (mTLS) for sensitive endpoints | -```text -Client::Options options; -options.ssl_connection(true) - .verify_peer(true) - .ca_verify_path("/etc/ssl/certs/ca-certificates.crt") - .timeout(30); -``` +**Never** disable peer verification in production: -**Never disable peer verification in production.** The `verify_peer` flag should only be `false` in isolated development environments. +```bash +# ❌ Insecure β€” never use in production +osqueryd --tls_server_certs="" --disable_ciphers="..." + +# βœ… Secure β€” pin to your CA bundle +osqueryd --tls_server_certs=/etc/ssl/openframe-ca.pem +``` --- ## Common Vulnerabilities and Mitigations -| Vulnerability | Mitigation in osquery | -|---|---| -| **SQL Injection** | Authorizer blocks unsafe opcodes; only allowlisted operations execute | -| **Credential Theft** | JWT tokens non-copyable, stored in singleton with restricted access | -| **Ciphertext Tampering** | AES-256-GCM authentication tag verified on every decrypt | -| **Privilege Escalation** | Watcher/Worker model isolates query execution from the supervisor | -| **External DB Injection** | `SQLITE_ATTACH` explicitly denied by authorizer | -| **Token Replay** | `OpenframeTokenRefresher` rotates tokens on a schedule | -| **Extension Impersonation** | SDK version check and UUID server-assignment prevent spoofing | -| **Resource Exhaustion** | Watchdog enforces CPU/memory limits and respawns workers | -| **Config Tampering** | Hash-based change detection and JSON size/depth limits | +| Vulnerability | Mitigation | +|--------------|------------| +| Token plaintext logged | Never log token values; review all log calls in auth code | +| Hardcoded secrets in source | Use secrets manager / environment injection; enforce in code review | +| Path traversal in token file path | Validate and sanitize all file paths before opening | +| Memory exposure of decrypted token | Minimize lifetime of plaintext token; zero memory on deallocation where possible | +| TOCTOU on token file | Read file atomically; hold exclusive lock during decrypt | +| SQL injection via distributed queries | Rely on the SQLite authorizer; never execute raw user input without validation | +| Extension process spoofing | Validate extension SDK version and UUID at registration | --- -## Security Testing +## Security Testing Guidelines -### Running Security-Relevant Tests +### Unit Tests for Security-Sensitive Code -```bash -# Build with tests enabled -cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DOSQUERY_BUILD_TESTS=ON -cmake --build build --parallel $(nproc) - -# Run all tests -cd build && ctest --output-on-failure +When writing or modifying `openframe/` components: -# Run specific security-related tests -cd build && ctest -R "tls" --output-on-failure -cd build && ctest -R "config" --output-on-failure -cd build && ctest -R "sql" --output-on-failure +```cpp +// Test that decryption fails on tampered ciphertext +TEST(EncryptionServiceTest, TamperedCiphertextThrows) { + OpenframeEncryptionService svc("your-32-byte-secret-key-here!!"); + EXPECT_THROW(svc.decrypt("tampered_base64_payload"), std::runtime_error); +} + +// Test that empty token file path throws +TEST(TokenExtractorTest, EmptyPathThrows) { + auto enc = std::make_shared("key..."); + OpenframeTokenExtractor extractor(enc, ""); + EXPECT_THROW(extractor.extractToken(), std::runtime_error); +} ``` -### Code Review Checklist for Security +### Code Review Checklist for Security PRs -Before submitting any PR that touches security-sensitive code: +Before merging any PR touching `openframe/`, `osquery/remote/`, or authentication logic: -- [ ] No secrets or credentials hardcoded -- [ ] All SQL inputs pass through the authorizer -- [ ] AES-GCM nonces are generated freshly (not reused) -- [ ] TLS peer verification is not disabled -- [ ] Error paths do not leak sensitive information in logs -- [ ] New extension APIs validate input size and type -- [ ] Thread-shared state uses proper synchronization primitives -- [ ] New config keys have size/depth validation +- [ ] No secrets or tokens appear in log output. +- [ ] Secret keys are not hardcoded in source or tests. +- [ ] Encryption errors throw exceptions and do NOT silently return empty strings. +- [ ] All file paths are validated before use. +- [ ] Background threads are properly joined on shutdown. +- [ ] No new unsafe SQL execution paths are introduced. +- [ ] TLS peer verification is not disabled. --- -## Reporting Security Issues - -Security vulnerabilities should be reported directly to the Flamingo team through the **OpenMSP Slack community**: +## Environment Variables and Secrets at Runtime -- πŸ’¬ [Join OpenMSP Slack](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) -- 🌐 [https://www.openmsp.ai/](https://www.openmsp.ai/) +| Variable | Handling | +|----------|---------| +| OpenFrame secret key | Inject via secrets manager; do NOT put in shell history | +| Token file path | Configure in osquery flags; restrict file permissions to `600` | +| TLS certificate paths | Store in `/etc/osquery/` with `640` permissions, `osquery` group | -> Do not open public GitHub Issues for security vulnerabilities. +> **Reminder:** This project uses the [OpenMSP Slack community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) for security discussions. For responsible disclosure, reach out there directly. diff --git a/docs/development/setup/environment.md b/docs/development/setup/environment.md index 882e48c34fb..965e8ef393d 100644 --- a/docs/development/setup/environment.md +++ b/docs/development/setup/environment.md @@ -1,200 +1,239 @@ # Development Environment Setup -This guide covers recommended IDE configurations, development tools, editor extensions, and environment variables for contributing to osquery with OpenFrame. +This guide covers how to configure your development environment for working on osquery (OpenFrame-enhanced), including IDE setup, recommended tools, and required extensions. --- -## Recommended IDEs +## Recommended IDE: CLion or VS Code -### Visual Studio Code (All Platforms) +### Option A β€” CLion (Recommended for C++) -VS Code with the C++ extension pack is the recommended cross-platform IDE for osquery development. +[CLion](https://www.jetbrains.com/clion/) provides native CMake integration and is the most productive IDE for this C++17 codebase. -**Required Extensions:** +**Setup steps:** -| Extension | ID | Purpose | -|---|---|---| -| C/C++ | `ms-vscode.cpptools` | IntelliSense, debugging, syntax highlighting | -| CMake Tools | `ms-vscode.cmake-tools` | CMake integration, build configuration | -| CMake | `twxs.cmake` | CMake syntax highlighting | -| clangd | `llvm-vs-code-extensions.vscode-clangd` | Fast code navigation, diagnostics | +1. Install CLion (commercial license or free trial). +2. Open the repository root directory β€” CLion auto-detects `CMakeLists.txt`. +3. Configure the CMake profile: + - Navigate to **Settings β†’ Build, Execution, Deployment β†’ CMake** + - Add a profile: `Debug` with generator `Ninja` or `Unix Makefiles` + - Set CMake options: -**Optional but Recommended:** +```text +-DCMAKE_C_COMPILER=clang +-DCMAKE_CXX_COMPILER=clang++ +-DCMAKE_BUILD_TYPE=Debug +``` -| Extension | ID | Purpose | -|---|---|---| -| GitLens | `eamodio.gitlens` | Enhanced Git history and annotations | -| Clang-Format | `xaver.clang-format` | Auto-formatting on save | -| Error Lens | `usernamehakki.error-lens` | Inline error display | +4. Enable **clangd** for code intelligence under **Settings β†’ Languages & Frameworks β†’ C/C++ β†’ Clangd**. -**Install all at once:** +--- -```bash -code --install-extension ms-vscode.cpptools -code --install-extension ms-vscode.cmake-tools -code --install-extension twxs.cmake -code --install-extension llvm-vs-code-extensions.vscode-clangd -code --install-extension eamodio.gitlens -``` +### Option B β€” Visual Studio Code -**Recommended `.vscode/settings.json`:** +VS Code works well with the following extensions installed: + +| Extension | ID | Purpose | +|-----------|-----|---------| +| **C/C++ Extension Pack** | `ms-vscode.cpptools-extension-pack` | C++ IntelliSense, debugging | +| **CMake Tools** | `ms-vscode.cmake-tools` | CMake integration | +| **clangd** | `llvm-vs-code-extensions.vscode-clangd` | Fast code intelligence via clang | +| **CodeLLDB** | `vadimcn.vscode-lldb` | LLDB debugger integration | +| **GitLens** | `eamodio.gitlens` | Enhanced Git integration | + +**VS Code workspace settings** (create `.vscode/settings.json`): ```json { + "cmake.configureArgs": [ + "-DCMAKE_C_COMPILER=clang", + "-DCMAKE_CXX_COMPILER=clang++" + ], "cmake.buildDirectory": "${workspaceFolder}/build", - "cmake.generator": "Ninja", - "cmake.buildType": "Debug", - "editor.formatOnSave": true, "clangd.arguments": [ "--compile-commands-dir=${workspaceFolder}/build", - "--header-insertion=iwyu", + "--background-index", "--clang-tidy" ], "C_Cpp.intelliSenseEngine": "disabled" } ``` -> When using `clangd`, disable the built-in IntelliSense engine to avoid conflicts. - --- -### CLion (JetBrains) +## Required Development Tools + +### Linux (Ubuntu/Debian) + +```bash +sudo apt-get update && sudo apt-get install -y \ + build-essential \ + cmake \ + clang-13 \ + clang++-13 \ + lld-13 \ + python3 \ + python3-pip \ + git \ + libssl-dev \ + ninja-build \ + ccache +``` -CLion provides first-class CMake support. Open the project root and CLion will auto-detect the `CMakeLists.txt`. +### macOS -**Recommended Settings:** +```bash +# Install Xcode command-line tools +xcode-select --install -- Set the CMake build directory to `build/` -- Enable `clang-format` in **Settings β†’ Editor β†’ Code Style β†’ C/C++** -- Use the built-in CMake tab for build configuration +# Install Homebrew +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +# Install tools +brew install cmake ninja ccache openssl +``` + +### Windows + +1. Install [Visual Studio 2022](https://visualstudio.microsoft.com/) with: + - **Desktop development with C++** workload + - **Windows 11 SDK** + - **LLVM/Clang compiler** (optional, but recommended) +2. Install [CMake](https://cmake.org/download/) β‰₯ 3.21. +3. Install [Git for Windows](https://git-scm.com/download/win). +4. Install [Python 3](https://www.python.org/downloads/windows/). --- -### Xcode (macOS Only) +## Compiler Configuration + +osquery requires **C++17** support. The recommended compiler configuration: -Generate an Xcode project from CMake: +| Platform | Compiler | Version | +|----------|---------|---------| +| Linux | Clang | β‰₯ 13 | +| macOS | Apple Clang | β‰₯ 13 (Xcode β‰₯ 13) | +| Windows | MSVC | Visual Studio 2019+ | + +Set compiler explicitly to avoid version mismatches: ```bash -cmake -S . -B build-xcode -G Xcode -open build-xcode/osquery.xcodeproj +export CC=clang-13 +export CXX=clang++-13 ``` --- -## Development Tools +## ccache (Build Cache) -Install these tools before starting development: +`ccache` significantly speeds up incremental and clean builds by caching compiled objects. -### All Platforms +Install and enable: ```bash -# Python tools for code generation -pip3 install jinja2 pexpect six +# Linux +sudo apt-get install ccache + +# macOS +brew install ccache + +# Verify +ccache --version ``` -### Linux +Configure in your shell profile: ```bash -sudo apt-get install -y \ - clang-format \ - clang-tidy \ - ccache \ - valgrind \ - gdb +export CCACHE_DIR="$HOME/.ccache" +export PATH="/usr/lib/ccache:$PATH" ``` -### macOS +CMake integration: ```bash -brew install \ - clang-format \ - ccache \ - llvm +cmake .. \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CXX_COMPILER_LAUNCHER=ccache ``` -### Windows +--- -- Install [LLVM](https://releases.llvm.org/) for `clang-format` and `clang-tidy` -- Configure `PATH` to include the LLVM `bin` directory +## clang-format (Code Formatting) ---- +The project uses `clang-format` for consistent C++ code style. -## Setting Up ccache (Highly Recommended) +Install: -`ccache` dramatically speeds up incremental rebuilds: +```bash +# Linux +sudo apt-get install clang-format-13 + +# macOS +brew install clang-format +``` + +Format a file: ```bash -# Install ccache -sudo apt-get install ccache # Linux -brew install ccache # macOS - -# Enable in CMake -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_BUILD_TYPE=Debug \ - -DCMAKE_C_COMPILER_LAUNCHER=ccache \ - -DCMAKE_CXX_COMPILER_LAUNCHER=ccache +clang-format -i openframe/openframe_encryption_service.cpp +``` + +Check formatting without modifying: + +```bash +clang-format --dry-run --Werror openframe/openframe_encryption_service.cpp ``` --- -## Generate compile_commands.json for clangd +## Python Tools (Code Generation) -For full IntelliSense support with `clangd`, generate the compilation database: +osquery uses Python scripts for table code generation and CI tasks: ```bash -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_EXPORT_COMPILE_COMMANDS=ON - -# Link to project root (required by clangd) -ln -sf build/compile_commands.json compile_commands.json +pip3 install jinja2 six ``` --- ## Environment Variables for Development -| Variable | Purpose | Example | -|---|---|---| -| `OSQUERY_EXTENSIONS_SOCKET` | Extension manager socket path | `/tmp/osquery.em` | -| `OSQUERY_CONFIG_PATH` | Override default config location | `./test.conf` | -| `OSQUERY_DB_PATH` | Custom database path for development | `/tmp/osquery-dev.db` | -| `OPENSSL_ROOT_DIR` | OpenSSL installation path (Windows/macOS) | `/usr/local/opt/openssl` | +| Variable | Purpose | Example Value | +|----------|---------|--------------| +| `CC` | C compiler path | `/usr/bin/clang-13` | +| `CXX` | C++ compiler path | `/usr/bin/clang++-13` | +| `CCACHE_DIR` | ccache storage directory | `$HOME/.ccache` | +| `OPENFRAME_TOKEN_PATH` | Path to encrypted token file | `/etc/openframe/token.enc` | +| `OPENFRAME_SECRET_KEY` | AES-256 secret key for token decryption | Refer to your environment configuration | + +--- + +## Generating compile_commands.json -Set in your shell profile or export before building: +The `compile_commands.json` file is required by clangd for code intelligence: ```bash -export OPENSSL_ROOT_DIR="/usr/local/opt/openssl@3" +cd build +cmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ``` -On Windows: +Then symlink to the project root so clangd and editors find it: ```bash -set OPENSSL_ROOT_DIR=C:\OpenSSL-Win64 +ln -s build/compile_commands.json compile_commands.json ``` --- -## Code Formatting - -osquery enforces formatting with `clang-format`. The configuration is in `.clang-format` at the repository root. +## Verifying Your Environment ```bash -# Format a single file -clang-format -i osquery/core/init.cpp +# Confirm compiler works +clang++ --std=c++17 -o /dev/null -x c++ /dev/null -# Check formatting without modifying -clang-format --dry-run --Werror osquery/core/init.cpp +# Confirm CMake finds Clang +cmake .. -DCMAKE_CXX_COMPILER=clang++ --trace-source=CMakeLists.txt 2>&1 | head -20 -# Format all C++ files in a directory -find osquery/core -name "*.cpp" -o -name "*.h" | xargs clang-format -i +# Confirm OpenSSL is available for the OpenFrame encryption service +pkg-config --libs openssl ``` - -The CI pipeline enforces formatting. Run format checks before submitting PRs. - ---- - -## Next Steps - -Once your environment is set up, proceed to the [Local Development Guide](local-development.md) for instructions on building, running, and debugging osquery locally. diff --git a/docs/development/setup/local-development.md b/docs/development/setup/local-development.md index da84e7144e8..d6572fc51dd 100644 --- a/docs/development/setup/local-development.md +++ b/docs/development/setup/local-development.md @@ -1,6 +1,6 @@ # Local Development Guide -This guide walks you through cloning, building, running, and debugging osquery with OpenFrame on your local machine. +This guide covers cloning the repository, building locally, running osquery in development mode, and configuring a debugger. --- @@ -11,176 +11,185 @@ git clone https://github.com/flamingo-stack/osquery.git cd osquery ``` +Initialize submodules if any are present: + +```bash +git submodule update --init --recursive +``` + +--- + +## Directory Structure + +```text +. +β”œβ”€β”€ osquery/ # Core osquery C++ source +β”œβ”€β”€ openframe/ # OpenFrame authentication extensions +β”œβ”€β”€ plugins/ # Config, logger, database, distributed plugins +β”œβ”€β”€ libraries/ # Vendored third-party libraries +β”œβ”€β”€ tools/ # Code generation scripts, CI tooling +β”œβ”€β”€ external/ # Extension SDK examples +β”œβ”€β”€ tests/ # Integration test suite +β”œβ”€β”€ CMakeLists.txt # Root CMake configuration +└── package.json # Documentation pipeline tooling (not a build dep) +``` + --- -## Build Configurations +## Build for Local Development -### Debug Build (Recommended for Development) +### 1. Create a Build Directory + +Always build out-of-source: + +```bash +mkdir build +cd build +``` + +### 2. Configure CMake (Debug Mode) ```bash -cmake -S . -B build \ - -G Ninja \ +cmake .. \ + -DCMAKE_C_COMPILER=clang \ + -DCMAKE_CXX_COMPILER=clang++ \ -DCMAKE_BUILD_TYPE=Debug \ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON - -cmake --build build --parallel $(nproc) ``` -> Debug builds include full debug symbols and disable optimizations, making debugging straightforward. +> **Tip:** `CMAKE_EXPORT_COMPILE_COMMANDS=ON` generates `compile_commands.json` for IDE code intelligence. -### RelWithDebInfo (Recommended for Testing) +### 3. Build All Targets ```bash -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo - -cmake --build build --parallel $(nproc) +cmake --build . --parallel $(nproc) ``` -### Release Build +### 4. Build Specific Targets -```bash -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_BUILD_TYPE=Release +Build only the interactive shell: -cmake --build build --parallel $(nproc) +```bash +cmake --build . --target osqueryi --parallel $(nproc) ``` -### Build with Tests Enabled +Build only the daemon: ```bash -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_BUILD_TYPE=Debug \ - -DOSQUERY_BUILD_TESTS=ON +cmake --build . --target osqueryd --parallel $(nproc) +``` + +Build the test suite: -cmake --build build --parallel $(nproc) +```bash +cmake --build . --target osquery_tests --parallel $(nproc) ``` --- -## Running osquery Locally +## Running Locally -### Interactive Shell (`osqueryi`) +### Interactive Shell (osqueryi) ```bash -./build/osquery/osqueryi +./osquery/osqueryi ``` -```sql -osquery> SELECT hostname, cpu_brand FROM system_info; -osquery> .tables -osquery> .schema processes -osquery> .quit +With verbose output: + +```bash +./osquery/osqueryi --verbose ``` -### Daemon (`osqueryd`) +Disable logging for cleaner development output: ```bash -# Create a minimal development config -cat > /tmp/osquery-dev.conf <<'EOF' +./osquery/osqueryi --disable_logging=true +``` + +### Daemon (osqueryd) + +Create a minimal development config: + +```bash +cat > /tmp/dev_osquery.conf << 'EOF' { "options": { + "config_plugin": "filesystem", "logger_plugin": "filesystem", - "logger_path": "/tmp/osquery-dev-logs", - "database_path": "/tmp/osquery-dev.db", + "logger_path": "/tmp/osquery_dev_logs", + "disable_logging": false, "schedule_splay_percent": 0 }, "schedule": { "system_info": { - "query": "SELECT hostname, cpu_brand FROM system_info;", - "interval": 60 + "query": "SELECT hostname, cpu_type FROM system_info;", + "interval": 10 } } } EOF - -mkdir -p /tmp/osquery-dev-logs - -./build/osquery/osqueryd \ - --config_path=/tmp/osquery-dev.conf \ - --verbose \ - --ephemeral ``` ---- - -## Development Flags - -These flags are useful during local development: - -| Flag | Purpose | -|---|---| -| `--verbose` | Enable verbose logging output | -| `--ephemeral` | Use in-memory database (no disk writes) | -| `--disable_watchdog` | Disable resource watchdog (easier debugging) | -| `--disable_events` | Disable event publishers (faster startup) | -| `--disable_logging` | Suppress all logger output | -| `--allow_unsafe` | Allow loading unsigned extensions | -| `--extensions_timeout=10` | Extension connection timeout in seconds | - -Example for a minimal debug session: +Run the daemon in the foreground (useful during development): ```bash -./build/osquery/osqueryi \ - --verbose \ - --disable_events \ - --ephemeral +./osquery/osqueryd \ + --config_path=/tmp/dev_osquery.conf \ + --ephemeral=true \ + --disable_database=true \ + --verbose ``` +> **Note:** `--ephemeral=true --disable_database=true` uses the in-memory database, avoiding RocksDB file locking issues during rapid iteration. + --- ## Incremental Builds -After making source changes, only rebuild changed targets: +After modifying a source file, CMake only recompiles the changed translation units: ```bash -# Rebuild only the osqueryi binary -cmake --build build --target osqueryi +cd build +cmake --build . --parallel $(nproc) +``` -# Rebuild only the osqueryd binary -cmake --build build --target osqueryd +After adding a new source file, re-run CMake first: -# Rebuild a specific test target -cmake --build build --target osquery_core_tests +```bash +cmake .. +cmake --build . --parallel $(nproc) ``` --- -## Debugging +## Debug Configuration -### Using GDB (Linux) +### Using LLDB (macOS / Linux) ```bash -# Build with debug symbols -cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -cmake --build build --parallel $(nproc) +lldb ./osquery/osqueryi +(lldb) run --verbose +``` -# Launch with GDB -gdb ./build/osquery/osqueryi +Set a breakpoint: -# In GDB: -# (gdb) set args --verbose --ephemeral -# (gdb) break osquery::Initializer::start -# (gdb) run +```bash +(lldb) breakpoint set --name OpenframeEncryptionService::decrypt +(lldb) run ``` -### Using LLDB (macOS) +### Using GDB (Linux) ```bash -lldb ./build/osquery/osqueryi - -# In LLDB: -# (lldb) settings set -- target.run-args "--verbose" "--ephemeral" -# (lldb) b osquery::Initializer::start -# (lldb) run +gdb ./osquery/osqueryi +(gdb) break OpenframeTokenExtractor::extractToken +(gdb) run --verbose ``` -### Using VS Code Debugger +### VS Code Debug Configuration -Add to `.vscode/launch.json`: +Create `.vscode/launch.json`: ```json { @@ -188,23 +197,43 @@ Add to `.vscode/launch.json`: "configurations": [ { "name": "Debug osqueryi", - "type": "cppdbg", + "type": "lldb", "request": "launch", "program": "${workspaceFolder}/build/osquery/osqueryi", - "args": ["--verbose", "--ephemeral", "--disable_events"], - "stopAtEntry": false, + "args": ["--verbose", "--disable_database=true"], "cwd": "${workspaceFolder}", - "environment": [], - "externalConsole": false, - "MIMode": "gdb", - "miDebuggerPath": "/usr/bin/gdb", - "setupCommands": [ - { - "description": "Enable pretty-printing", - "text": "-enable-pretty-printing", - "ignoreFailures": true - } - ] + "preLaunchTask": "cmake-build" + }, + { + "name": "Debug osqueryd", + "type": "lldb", + "request": "launch", + "program": "${workspaceFolder}/build/osquery/osqueryd", + "args": [ + "--config_path=/tmp/dev_osquery.conf", + "--verbose", + "--ephemeral=true" + ], + "cwd": "${workspaceFolder}" + } + ] +} +``` + +Create `.vscode/tasks.json` for the pre-launch build task: + +```json +{ + "version": "2.0.0", + "tasks": [ + { + "label": "cmake-build", + "type": "shell", + "command": "cmake --build build --parallel $(nproc)", + "group": { + "kind": "build", + "isDefault": true + } } ] } @@ -212,68 +241,91 @@ Add to `.vscode/launch.json`: --- -## Developing the OpenFrame Layer +## Running Unit Tests + +```bash +cd build -The OpenFrame components live in the `openframe/` directory: +# Run all unit tests +ctest --parallel $(nproc) --output-on-failure -```text -openframe/ -β”œβ”€β”€ openframe_authorization_manager.h/.cpp # JWT token lifecycle -β”œβ”€β”€ openframe_authorization_manager_provider.h -β”œβ”€β”€ openframe_encryption_service.h/.cpp # AES-256-GCM via OpenSSL -β”œβ”€β”€ openframe_token_extractor.h/.cpp # Token acquisition -└── openframe_token_refresher.h/.cpp # Background token renewal +# Run a specific test binary +./osquery/osquery_tests --gtest_filter=ConfigTests.* + +# Run with verbose output +./osquery/osquery_tests --gtest_filter=* --gtest_verbose=all ``` -When modifying OpenFrame components, rebuild only the affected target: +--- + +## Watching for Changes + +osquery doesn't have a built-in watch mode. Use `entr` or a similar tool to trigger rebuilds on file changes: ```bash -cmake --build build --target openframe_auth +# Linux/macOS - rebuild on any .cpp or .h change under osquery/ or openframe/ +find osquery/ openframe/ -name '*.cpp' -o -name '*.h' | \ + entr -c cmake --build build --parallel $(nproc) --target osqueryi ``` -Test the encryption service independently by linking against it in a test binary. +Install `entr` if needed: ---- +```bash +# Ubuntu/Debian +sudo apt-get install entr -## Working with Virtual Tables +# macOS +brew install entr +``` -To add a new virtual table: +--- -1. Create schema and implementation in `osquery/tables//` -2. Register the table in the appropriate CMake target -3. Run the code generator to update schema definitions: +## Cleaning the Build + +Remove build artifacts without reconfiguring: ```bash -python3 tools/codegen/gentable.py osquery/tables/my_category/my_table.table +cmake --build build --target clean ``` -4. Rebuild and test: +Full clean (delete build directory): ```bash -cmake --build build --target osqueryi -./build/osquery/osqueryi -osquery> .tables my_table +rm -rf build/ ``` --- -## Log Output Locations +## Adding a New OpenFrame Source File -| Mode | Default Log Path | -|---|---| -| Daemon (Linux) | `/var/log/osquery/` | -| Daemon (macOS) | `/var/log/osquery/` | -| Daemon (Windows) | `C:\ProgramData\osquery\log\` | -| Development override | Set `--logger_path=/tmp/my-logs` | +When adding a new C++ source file to the `openframe/` directory: ---- +1. Create the `.h` and `.cpp` files. +2. Add the source to the relevant `CMakeLists.txt`: -## Clean Rebuild +```cmake +target_sources(openframe_lib PRIVATE + openframe/openframe_new_component.cpp +) +``` -If you encounter stale build artifacts: +3. Re-run CMake and rebuild: ```bash -rm -rf build/ -cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -cmake --build build --parallel $(nproc) +cd build +cmake .. +cmake --build . --parallel $(nproc) ``` + +--- + +## Useful CMake Flags for Development + +| Flag | Description | +|------|-------------| +| `-DCMAKE_BUILD_TYPE=Debug` | Include debug symbols | +| `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` | Generate `compile_commands.json` | +| `-DOSQUERY_DISABLE_DATABASE=ON` | Skip RocksDB build (faster for table dev) | +| `-DOSQUERY_NO_DEBUG_SYMBOLS=ON` | Reduce binary size | +| `-DCMAKE_C_COMPILER_LAUNCHER=ccache` | Enable ccache for C compilation | +| `-DCMAKE_CXX_COMPILER_LAUNCHER=ccache` | Enable ccache for C++ compilation | diff --git a/docs/development/testing/README.md b/docs/development/testing/README.md deleted file mode 100644 index 550d7e68a8e..00000000000 --- a/docs/development/testing/README.md +++ /dev/null @@ -1,310 +0,0 @@ -# Testing Guide - -osquery uses **Google Test** (GTest) and **Google Mock** (GMock) for unit and integration testing. Tests are organized alongside the source code in `tests/` subdirectories within each module. - ---- - -## Test Structure and Organization - -```text -osquery/ -β”œβ”€β”€ core/ -β”‚ β”œβ”€β”€ tests/ -β”‚ β”‚ β”œβ”€β”€ flags_tests.cpp -β”‚ β”‚ β”œβ”€β”€ query_tests.cpp -β”‚ β”‚ β”œβ”€β”€ system_test.cpp -β”‚ β”‚ └── tables_tests.cpp -β”œβ”€β”€ sql/ -β”‚ └── tests/ -β”‚ β”œβ”€β”€ sql.cpp -β”‚ └── virtual_table.cpp -β”œβ”€β”€ events/ -β”‚ └── tests/ -β”‚ β”œβ”€β”€ events_tests.cpp -β”‚ └── linux/ -β”‚ β”œβ”€β”€ inotify_tests.cpp -β”‚ └── bpf/ -β”œβ”€β”€ config/ -β”‚ └── tests/ -β”‚ β”œβ”€β”€ config_tests.cpp -β”‚ └── packs.cpp -β”œβ”€β”€ database/ -β”‚ └── tests/ -β”œβ”€β”€ distributed/ -β”‚ └── tests/ -└── extensions/ - └── tests/ - -tests/ -└── integration/ - └── tables/ # Integration tests for all virtual tables - β”œβ”€β”€ processes.cpp - β”œβ”€β”€ users.cpp - β”œβ”€β”€ listening_ports.cpp - └── ... - -plugins/ -└── */tests/ # Plugin-level tests -``` - -### Test Categories - -| Category | Location | Description | -|---|---|---| -| **Unit Tests** | `osquery/*/tests/` | Fast, isolated, no OS dependencies | -| **Integration Tests** | `tests/integration/tables/` | Live table queries against the real OS | -| **Benchmark Tests** | `osquery/*/benchmarks/` | Performance measurements | -| **Extension Tests** | `osquery/extensions/tests/` | IPC and Thrift extension round-trips | -| **Plugin Tests** | `plugins/*/tests/` | Logger, config, and database plugins | - ---- - -## Building Tests - -Tests must be explicitly enabled at CMake configure time: - -```bash -cmake -S . -B build \ - -G Ninja \ - -DCMAKE_BUILD_TYPE=Debug \ - -DOSQUERY_BUILD_TESTS=ON - -cmake --build build --parallel $(nproc) -``` - ---- - -## Running Tests - -### Run All Tests - -```bash -cd build -ctest --output-on-failure -``` - -### Run Tests in Parallel - -```bash -cd build -ctest --output-on-failure --parallel $(nproc) -``` - -### Run a Specific Test Suite - -```bash -cd build - -# Run core tests -ctest -R "osquery_core_tests" --output-on-failure - -# Run SQL tests -ctest -R "osquery_sql_tests" --output-on-failure - -# Run config tests -ctest -R "osquery_config_tests" --output-on-failure - -# Run events tests -ctest -R "osquery_events_tests" --output-on-failure - -# Run integration table tests -ctest -R "integration" --output-on-failure -``` - -### Run Tests by Keyword - -```bash -cd build -ctest -R "tls" --output-on-failure -ctest -R "database" --output-on-failure -ctest -R "extension" --output-on-failure -``` - -### Verbose Test Output - -```bash -cd build -ctest -V -R "osquery_core_tests" -``` - ---- - -## Writing New Tests - -### Unit Test Template - -```cpp -#include -#include - -#include "osquery/my_module/my_class.h" - -namespace osquery { - -class MyClassTest : public ::testing::Test { - protected: - void SetUp() override { - // Per-test setup - } - - void TearDown() override { - // Per-test cleanup - } -}; - -TEST_F(MyClassTest, BasicOperation) { - MyClass obj; - EXPECT_EQ(obj.doSomething(), expected_value); -} - -TEST_F(MyClassTest, HandlesEdgeCase) { - MyClass obj; - EXPECT_THROW(obj.doSomethingInvalid(), std::runtime_error); -} - -} // namespace osquery -``` - -### Virtual Table Integration Test Template - -```cpp -#include "tests/integration/tables/helper.h" - -namespace osquery { -namespace table_tests { - -class MyTableTest : public testing::Test {}; - -TEST_F(MyTableTest, test_sanity) { - // Validate table schema and basic query - auto const data = execute_query("SELECT * FROM my_table;"); - ASSERT_GT(data.size(), 0ul); - - // Validate specific column presence and types - ValidateSchema( - data, - { - {"column_one", BIGINT_TYPE, CHECK_GT, 0, CHECK_LT, INT_MAX}, - {"column_two", TEXT_TYPE, CHECK_NON_EMPTY}, - } - ); -} - -} // namespace table_tests -} // namespace osquery -``` - -### Using GMock for Dependencies - -```cpp -#include -#include "osquery/database/database.h" - -class MockDatabase : public IDatabaseInterface { - public: - MOCK_METHOD(Status, putBatch, - (const std::string& domain, - const DatabaseStringValueList& data), - (override)); - - MOCK_METHOD(Status, get, - (const std::string& domain, - const std::string& key, - std::string& value), - (const, override)); -}; - -TEST(MyServiceTest, PersistsData) { - MockDatabase db; - EXPECT_CALL(db, putBatch(testing::_, testing::_)) - .Times(1) - .WillOnce(testing::Return(Status::success())); - - MyService service(db); - auto result = service.persist("key", "value"); - EXPECT_TRUE(result.ok()); -} -``` - ---- - -## Test Helpers - -The integration test suite includes shared helpers in `tests/integration/tables/helper.h`: - -| Helper | Purpose | -|---|---| -| `execute_query(sql)` | Execute SQL and return rows | -| `ValidateSchema(data, validators)` | Assert column types and value ranges | -| `CHECK_GT`, `CHECK_LT` | Numeric range validators | -| `CHECK_NON_EMPTY` | String non-emptiness check | - ---- - -## Benchmark Tests - -Benchmarks measure performance of hot paths: - -```bash -# Build benchmarks -cmake -S . -B build -G Ninja \ - -DCMAKE_BUILD_TYPE=Release \ - -DOSQUERY_BUILD_BENCHMARKS=ON -cmake --build build --parallel $(nproc) - -# Run SQL benchmarks -./build/osquery/sql/benchmarks/sql_benchmarks - -# Run events benchmarks -./build/osquery/events/benchmarks/events_benchmarks -``` - ---- - -## Coverage Requirements - -While no mandatory coverage threshold is enforced in CI currently, the project encourages: - -- **Unit tests** for all new core module functionality -- **Integration tests** for every new virtual table -- **Error path coverage** for all error handling branches -- **Mock-based tests** for components with external dependencies (network, database) - -When contributing new virtual tables, a corresponding integration test in `tests/integration/tables/` is expected. - ---- - -## Platform-Specific Tests - -Some tests only run on specific platforms: - -```cpp -// Linux-only test -#ifdef __linux__ -TEST_F(AuditTest, LinuxAuditPublisher) { - // Linux-specific audit test -} -#endif - -// macOS-only test -#ifdef __APPLE__ -TEST_F(FSEventsTest, FseventsPublisher) { - // macOS FSEvents test -} -#endif -``` - -The CMake build system automatically includes platform-appropriate test targets. - ---- - -## Continuous Integration - -Tests run automatically in CI on every pull request. The CI pipeline: - -1. Builds with `-DOSQUERY_BUILD_TESTS=ON` -2. Runs `ctest --output-on-failure` -3. Enforces `clang-format` compliance -4. Checks copyright headers (via `tools/ci/scripts/check_copyright_headers.py`) - -All tests must pass before a PR can be merged. diff --git a/docs/diagrams/architecture/README-10.mmd b/docs/diagrams/architecture/README-10.mmd deleted file mode 100644 index 9de352b6a66..00000000000 --- a/docs/diagrams/architecture/README-10.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart TD - Caller["Module"] --> Client["HTTP Client"] - Client --> TLS["TLS Handshake"] - TLS --> Remote["Remote Server"] diff --git a/docs/diagrams/architecture/README-11.mmd b/docs/diagrams/architecture/README-11.mmd deleted file mode 100644 index dbc77c3eaf7..00000000000 --- a/docs/diagrams/architecture/README-11.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart TD - Caller["Subsystem"] --> FS["Filesystem API"] - FS --> POSIX["POSIX Implementation"] - FS --> Windows["Windows Implementation"] diff --git a/docs/diagrams/architecture/README-2.mmd b/docs/diagrams/architecture/README-2.mmd index 15cf680aafe..01f65151dd1 100644 --- a/docs/diagrams/architecture/README-2.mmd +++ b/docs/diagrams/architecture/README-2.mmd @@ -1,11 +1,14 @@ flowchart TD - Start["Process Start"] --> Init["Initializer"] - Init --> Flags["Parse Flags"] - Init --> Registry["Initialize Registry"] - Init --> DBInit["Initialize Database"] - Init --> ExtMgr["Start Extension Manager"] - Init --> ConfigLoad["Load Configuration"] - ConfigLoad --> Schedule["Build Schedule"] - Schedule --> QueryExec["Execute Queries"] - QueryExec --> Log["Log Results"] - Log --> End["Runtime Loop"] + Source["Scheduled Or Distributed Query"] --> Scheduler["Configuration And Packs"] + Scheduler --> SQLPlugin["SQLiteSQLPlugin"] + SQLPlugin --> SQLiteCore["SQLite Engine"] + SQLiteCore --> VTables["Virtual Tables"] + VTables --> TablePlugins["Table Plugins"] + + SQLiteCore --> QueryState["Query State And Diff"] + QueryState --> DB["Database Backend"] + QueryState --> Diff["DiffResults"] + + Diff --> LogItem["QueryLogItem"] + LogItem --> Logger["Logger Plugin"] + Logger --> External["External Logging System"] diff --git a/docs/diagrams/architecture/README-3.mmd b/docs/diagrams/architecture/README-3.mmd index 2bec5262176..e9aaea002ea 100644 --- a/docs/diagrams/architecture/README-3.mmd +++ b/docs/diagrams/architecture/README-3.mmd @@ -1,7 +1,6 @@ flowchart TD - Query["SQL Query"] --> SQLite["SQLite Engine"] - SQLite --> Module["sqlite3_module"] - Module --> VirtualTable["VirtualTable Wrapper"] - VirtualTable --> TablePlugin["TablePlugin"] - TablePlugin --> OS["Operating System Data"] - OS --> Rows["Rows Returned"] + Publisher["Event Publisher Plugin"] --> Subscriber["EventSubscriberPlugin"] + Subscriber --> DB["Database Backend"] + Subscriber --> SQLTable["Event Virtual Table"] + SQLTable --> Query["SELECT From Event Table"] + Query --> Logging["Logging And Query Metadata"] diff --git a/docs/diagrams/architecture/README-4.mmd b/docs/diagrams/architecture/README-4.mmd index 7cb68c8186b..fbfca6eb432 100644 --- a/docs/diagrams/architecture/README-4.mmd +++ b/docs/diagrams/architecture/README-4.mmd @@ -1,6 +1,7 @@ flowchart TD - Source["Config Source (File / TLS / Extension)"] --> ConfigCore["Config Singleton"] - ConfigCore --> Parsers["ConfigParserPlugins"] - Parsers --> Schedule["Scheduled Queries"] - Schedule --> Scheduler["Query Scheduler"] - Scheduler --> SQL["SQL Engine"] + Server["Remote Server"] --> TLSPlugin["Distributed TLS Plugin"] + TLSPlugin --> DistributedEngine["Distributed Querying"] + DistributedEngine --> SQL["SQL Engine"] + SQL --> Results["DistributedQueryResult"] + Results --> TLSPlugin + TLSPlugin --> Server diff --git a/docs/diagrams/architecture/README-5.mmd b/docs/diagrams/architecture/README-5.mmd deleted file mode 100644 index 15b3d86b3cc..00000000000 --- a/docs/diagrams/architecture/README-5.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - Publisher["EventPublisher"] --> EventFactory["EventFactory"] - Subscriber["EventSubscriber"] --> EventFactory - EventFactory --> Callback["EventCallback"] - Callback --> Storage["Database Storage"] - SQL["SELECT *_events"] --> Subscriber diff --git a/docs/diagrams/architecture/README-6.mmd b/docs/diagrams/architecture/README-6.mmd deleted file mode 100644 index f6966640dec..00000000000 --- a/docs/diagrams/architecture/README-6.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - Exec["Query Execution"] --> Diff["DiffResults"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serialize["JSON Serialization"] - Serialize --> Logger["LoggerPlugin"] - Logger --> Sink["External Backend"] diff --git a/docs/diagrams/architecture/README-7.mmd b/docs/diagrams/architecture/README-7.mmd deleted file mode 100644 index b995aa1bd07..00000000000 --- a/docs/diagrams/architecture/README-7.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Modules["Core Modules"] --> Interface["IDatabaseInterface"] - Interface --> Plugin["Database Plugin"] - Plugin --> RocksDB["Persistent Backend"] - Plugin --> Ephemeral["In-Memory Backend"] diff --git a/docs/diagrams/architecture/README-8.mmd b/docs/diagrams/architecture/README-8.mmd deleted file mode 100644 index f90914fba8e..00000000000 --- a/docs/diagrams/architecture/README-8.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Server["Central Server"] --> TLS["TLS Distributed Plugin"] - TLS --> Core["Distributed Core"] - Core --> SQL["SQL Engine"] - SQL --> Results["Results"] - Results --> TLS - TLS --> Server diff --git a/docs/diagrams/architecture/README-9.mmd b/docs/diagrams/architecture/README-9.mmd deleted file mode 100644 index 05f3210aec0..00000000000 --- a/docs/diagrams/architecture/README-9.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart LR - Core["osquery Core"] --> Manager["Extension Manager"] - Extension["External Extension"] --> Manager - Manager --> Registry["RegistryFactory"] - Core --> Extension diff --git a/docs/diagrams/architecture/README.mmd b/docs/diagrams/architecture/README.mmd index abd03ad683d..9d4393b5aa6 100644 --- a/docs/diagrams/architecture/README.mmd +++ b/docs/diagrams/architecture/README.mmd @@ -1,22 +1,26 @@ flowchart TD - CLI["osqueryi / osqueryd"] --> Core["Core Init And Runtime"] - - Core --> Config["Configuration And Packs"] + Core["Core Runtime And Lifecycle"] --> Config["Configuration And Packs"] Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework"] - Core --> Logging["Logging And Observability"] - Core --> DB["Database And Storage Plugins"] + Core --> DB["Database Backend"] + Core --> Logging["Logging And Query Metadata"] + Core --> Events["Events Core And Subscriptions"] + Core --> Extensions["Extensions Framework"] Core --> Distributed["Distributed Querying"] - Core --> Extensions["Extensions And IPC"] - Core --> HTTP["Remote HTTP Client"] - Core --> FS["Filesystem And Path Utilities"] Config --> SQL Config --> Events + Config --> Logging + SQL --> Logging + SQL --> DB + Events --> DB + Events --> Logging + Distributed --> SQL - Distributed --> HTTP - Logging --> DB + Distributed --> DB + Distributed --> Logging + Extensions --> SQL - Extensions --> DB + Extensions --> Config + Extensions --> Distributed diff --git a/docs/diagrams/architecture/config-plugins-2.mmd b/docs/diagrams/architecture/config-plugins-2.mmd new file mode 100644 index 00000000000..87c41852990 --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-2.mmd @@ -0,0 +1,7 @@ +flowchart TD + Start["Start genConfig()"] --> CheckFile["Check config_path exists"] + CheckFile -->|"valid"| LoadDir["Resolve config_path.d/*.conf"] + LoadDir --> SortFiles["Sort files"] + SortFiles --> Merge["Append main config"] + Merge --> Output["Return map of source to JSON"] + CheckFile -->|"invalid"| Fail["Return failure Status"] diff --git a/docs/diagrams/architecture/config-plugins-3.mmd b/docs/diagrams/architecture/config-plugins-3.mmd new file mode 100644 index 00000000000..3f2e1e62041 --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-3.mmd @@ -0,0 +1,14 @@ +flowchart TD + ConfigLoad["Config Updated"] --> ParseDecorators["Parse decorators JSON"] + ParseDecorators --> StoreQueries["Store queries by point"] + StoreQueries --> RunLoad["Run load decorators"] + + Scheduler["Query Scheduler"] --> RunAlways["Run always decorators"] + Scheduler --> RunInterval["Run interval decorators"] + + RunLoad --> SQLExec["Execute SQL"] + RunAlways --> SQLExec + RunInterval --> SQLExec + + SQLExec --> DecorationStore["Store first row columns"] + DecorationStore --> Logger["Attach to logs"] diff --git a/docs/diagrams/architecture/config-plugins-4.mmd b/docs/diagrams/architecture/config-plugins-4.mmd new file mode 100644 index 00000000000..58d3717a3bc --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-4.mmd @@ -0,0 +1,13 @@ +flowchart TD + ConfigUpdate["Config Update"] --> RemoveOld["Remove existing files for source"] + RemoveOld --> ParseFilePaths["Parse file_paths"] + RemoveOld --> ParseQuery["Parse file_paths_query"] + RemoveOld --> ParseAccess["Parse file_accesses"] + RemoveOld --> ParseExclude["Parse exclude_paths"] + + ParseFilePaths --> Register["Config::addFile()"] + ParseQuery --> SQLExec["Execute SQL query"] + SQLExec --> Register + + ParseAccess --> AccessMap["Build access map"] + ParseExclude --> MergeExclude["Merge exclude paths"] diff --git a/docs/diagrams/architecture/config-plugins-5.mmd b/docs/diagrams/architecture/config-plugins-5.mmd new file mode 100644 index 00000000000..2a68339bac7 --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-5.mmd @@ -0,0 +1,6 @@ +flowchart TD + ParseOptions["Parse options JSON"] --> Validate["Validate flag name"] + Validate -->|"invalid"| Warn["Log warning"] + Validate -->|"valid"| UpdateFlag["Flag::updateValue()"] + UpdateFlag --> CheckVerbose["Is verbosity option?"] + CheckVerbose -->|"yes"| Reconfigure["setVerboseLevel()"] diff --git a/docs/diagrams/architecture/config-plugins-6.mmd b/docs/diagrams/architecture/config-plugins-6.mmd new file mode 100644 index 00000000000..a02f3e3e54c --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-6.mmd @@ -0,0 +1,7 @@ +flowchart TD + ParseViews["Parse views JSON"] --> LoadExisting["Scan database keys"] + LoadExisting --> Compare["Compare old vs new"] + Compare -->|"changed or new"| DropOld["DROP VIEW"] + DropOld --> CreateNew["CREATE VIEW AS query"] + CreateNew --> Persist["Store in database"] + Compare -->|"removed"| Remove["Delete database entry"] diff --git a/docs/diagrams/architecture/config-plugins-7.mmd b/docs/diagrams/architecture/config-plugins-7.mmd new file mode 100644 index 00000000000..bfd2c01c15e --- /dev/null +++ b/docs/diagrams/architecture/config-plugins-7.mmd @@ -0,0 +1,10 @@ +flowchart TD + Source["Filesystem or Remote Source"] --> ConfigPlugin["ConfigPlugin"] + ConfigPlugin --> ConfigCore["Config Core"] + ConfigCore --> ParserPlugins["ConfigParserPlugins"] + ParserPlugins --> RuntimeState["Runtime State"] + + RuntimeState --> Scheduler["Query Scheduler"] + RuntimeState --> Logger + RuntimeState --> Events + RuntimeState --> SQL diff --git a/docs/diagrams/architecture/config-plugins.mmd b/docs/diagrams/architecture/config-plugins.mmd new file mode 100644 index 00000000000..43a0e8b56eb --- /dev/null +++ b/docs/diagrams/architecture/config-plugins.mmd @@ -0,0 +1,18 @@ +flowchart TD + CLI["CLI Flags"] --> FilesystemPlugin["FilesystemConfigPlugin"] + FilesystemPlugin --> ConfigCore["Config Core"] + UpdatePlugin["UpdateConfigPlugin"] --> ConfigCore + + ConfigCore --> DecoratorsParser["DecoratorsConfigParserPlugin"] + ConfigCore --> EventsParser["EventsConfigParserPlugin"] + ConfigCore --> FilePathsParser["FilePathsConfigParserPlugin"] + ConfigCore --> OptionsParser["OptionsConfigParserPlugin"] + ConfigCore --> ViewsParser["ViewsConfigParserPlugin"] + + DecoratorsParser --> SQL["SQL Engine"] + FilePathsParser --> SQL + ViewsParser --> SQL + + ViewsParser --> DB["Database Backend"] + DecoratorsParser --> Logger["Logger"] + EventsParser --> EventsCore["Events Framework"] diff --git a/docs/diagrams/architecture/configuration-and-packs-2.mmd b/docs/diagrams/architecture/configuration-and-packs-2.mmd index 058d1366de2..35594d81ffa 100644 --- a/docs/diagrams/architecture/configuration-and-packs-2.mmd +++ b/docs/diagrams/architecture/configuration-and-packs-2.mmd @@ -1,6 +1,6 @@ flowchart TD - Start["ConfigRefreshRunner Start"] --> Wait["Sleep refresh_sec"] - Wait --> Check["Interrupted?"] - Check -->|No| Refresh["Config.refresh()"] - Refresh --> Wait - Check -->|Yes| End["End"] + Start["Thread Start"] --> Pause["Sleep refresh_sec"] + Pause --> Check["Interrupted?"] + Check -->|"Yes"| End["Stop"] + Check -->|"No"| Refresh["Config.refresh()"] + Refresh --> Pause diff --git a/docs/diagrams/architecture/configuration-and-packs-3.mmd b/docs/diagrams/architecture/configuration-and-packs-3.mmd index 14e23b18fcf..16fb7c1cd66 100644 --- a/docs/diagrams/architecture/configuration-and-packs-3.mmd +++ b/docs/diagrams/architecture/configuration-and-packs-3.mmd @@ -1,7 +1,9 @@ flowchart TD - ConfigUpdate["Config Update"] --> BuildPacks["Create Pack Objects"] - BuildPacks --> DiscoveryCheck["Discovery Queries"] - DiscoveryCheck --> ActivePack["Active Pack"] - ActivePack --> Scheduler["ScheduledQuery Execution"] - Scheduler --> Performance["Query Performance Recording"] - Scheduler --> Denylist["Denylist On Failure"] + Gen["Registry.call genConfig"] --> Update["Config.update()"] + Update --> Purge["Purge stale queries"] + Purge --> UpdateSource["updateSource(source)"] + UpdateSource --> Validate["Validate JSON"] + Validate --> Extract["Extract schedule and packs"] + Extract --> ApplyParsers["Apply config parsers"] + ApplyParsers --> Reconfigure["Reconfigure registries"] + Reconfigure --> Done["Config Valid"] diff --git a/docs/diagrams/architecture/configuration-and-packs-4.mmd b/docs/diagrams/architecture/configuration-and-packs-4.mmd index 31827beb892..3c90561d8cf 100644 --- a/docs/diagrams/architecture/configuration-and-packs-4.mmd +++ b/docs/diagrams/architecture/configuration-and-packs-4.mmd @@ -1,8 +1,5 @@ flowchart TD - ConfigLoad["Config Load"] --> RunLoad["Run Load Decorators"] - QueryExec["Query Execution"] --> RunAlways["Run Always Decorators"] - IntervalTick["Interval Timer"] --> RunInterval["Run Interval Decorators"] - RunLoad --> Decorations[("Decoration Store")] - RunAlways --> Decorations - RunInterval --> Decorations - Decorations --> Logger["Status And Query Logs"] + Platform["Check Platform"] --> Version["Check Version"] + Version --> Shard["Check Shard"] + Shard --> Discovery["Run Discovery Queries"] + Discovery --> Active["Pack Active"] diff --git a/docs/diagrams/architecture/configuration-and-packs-5.mmd b/docs/diagrams/architecture/configuration-and-packs-5.mmd index 91bbcadc931..9d94b00e37c 100644 --- a/docs/diagrams/architecture/configuration-and-packs-5.mmd +++ b/docs/diagrams/architecture/configuration-and-packs-5.mmd @@ -1,11 +1,6 @@ flowchart TD - Source["Config Source"] --> GenConfig["genConfig()"] - GenConfig --> Update["Config.update()"] - Update --> Purge["Purge Stale State"] - Purge --> UpdateSource["updateSource()"] - UpdateSource --> ParseJSON["Validate And Parse JSON"] - ParseJSON --> BuildSchedule["Create Packs And Schedule"] - BuildSchedule --> ApplyParsers["Apply ConfigParserPlugins"] - ApplyParsers --> Reconfigure["Registry.configure()"] - Reconfigure --> EventNotify["EventFactory.configUpdate()"] - EventNotify --> Ready["Runtime Updated"] + Failure["Query Failure"] --> Persist["Persist to Database"] + Persist --> Denylist["Add to Denylist"] + Denylist --> Skip["Skip During Scheduling"] + Skip --> Expire["Expiration Check"] + Expire -->|"Expired"| Reactivate["Allow Execution"] diff --git a/docs/diagrams/architecture/configuration-and-packs-6.mmd b/docs/diagrams/architecture/configuration-and-packs-6.mmd new file mode 100644 index 00000000000..36700dbe79c --- /dev/null +++ b/docs/diagrams/architecture/configuration-and-packs-6.mmd @@ -0,0 +1,4 @@ +flowchart TD + Update["Successful Update"] --> Backup["Backup Config Sources"] + Failure["Refresh Failure"] --> Restore["Restore From Database"] + Restore --> Update diff --git a/docs/diagrams/architecture/configuration-and-packs-7.mmd b/docs/diagrams/architecture/configuration-and-packs-7.mmd new file mode 100644 index 00000000000..4a142d2e3a1 --- /dev/null +++ b/docs/diagrams/architecture/configuration-and-packs-7.mmd @@ -0,0 +1,4 @@ +flowchart LR + SourceA["Source A Hash"] --> Combine["Combine Hashes"] + SourceB["Source B Hash"] --> Combine + Combine --> Global["Global Config Hash"] diff --git a/docs/diagrams/architecture/configuration-and-packs.mmd b/docs/diagrams/architecture/configuration-and-packs.mmd index 3a720d6d6c3..66ce753920e 100644 --- a/docs/diagrams/architecture/configuration-and-packs.mmd +++ b/docs/diagrams/architecture/configuration-and-packs.mmd @@ -1,15 +1,10 @@ flowchart TD - ConfigPlugin["Config Plugin Registry"] --> ConfigCore["Config Singleton"] - ConfigCore --> RefreshRunner["ConfigRefreshRunner"] - ConfigCore --> Schedule["Schedule"] - Schedule --> Pack["Pack"] - ConfigCore --> Parsers["ConfigParserPlugins"] - Parsers --> OptionsParser["Options Parser"] - Parsers --> FilePathsParser["File Paths Parser"] - Parsers --> DecoratorsParser["Decorators Parser"] - Parsers --> EventsParser["Events Parser"] - Parsers --> ViewsParser["Views Parser"] - ConfigCore --> Database[("Persistent Database")] - Schedule --> Logging["Logging And Query Observability"] - Schedule --> SQLEngine["SQL Engine And Virtual Tables"] - Parsers --> EventFramework["Eventing Framework"] + ConfigPlugin["Config Plugin Registry"] -->|"genConfig"| ConfigCore["Config Singleton"] + ConfigCore -->|"updateSource"| JSONValidation["JSON Validation"] + JSONValidation --> Packs["Pack Objects"] + Packs --> Schedule["Schedule Manager"] + Schedule --> SQL["SQL Engine"] + Schedule --> Logger["Logger Plugins"] + ConfigCore --> Database["Database Backend"] + ConfigCore --> EventFactory["Event Factory"] + ConfigCore --> Registry["Plugin Registry"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-2.mmd b/docs/diagrams/architecture/core-init-and-runtime-2.mmd deleted file mode 100644 index 12303b48e3e..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-2.mmd +++ /dev/null @@ -1,15 +0,0 @@ -sequenceDiagram - participant Main - participant Initializer - participant Registry - participant Config - participant Extensions - participant Events - - Main->>Initializer: Construct(argc, argv) - Initializer->>Initializer: Parse flags - Initializer->>Registry: registryAndPluginInit() - Initializer->>Extensions: startExtensionManager() - Initializer->>Config: load() - Initializer->>Events: attachEvents() - Initializer->>Initializer: start() diff --git a/docs/diagrams/architecture/core-init-and-runtime-3.mmd b/docs/diagrams/architecture/core-init-and-runtime-3.mmd deleted file mode 100644 index 6f1c2a8d3fd..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-3.mmd +++ /dev/null @@ -1,8 +0,0 @@ -flowchart LR - Macro["FLAG / CLI_FLAG Macros"] --> FlagCreate["Flag::create()"] - FlagCreate --> FlagRegistry["Internal Flag Map"] - - CLI["Command Line"] --> Parse["ParseCommandLineFlags"] - Config["Config Options"] --> Update["Flag::updateValue()"] - - FlagRegistry --> Runtime["Runtime Access"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-4.mmd b/docs/diagrams/architecture/core-init-and-runtime-4.mmd deleted file mode 100644 index 3464c298d66..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-4.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Watcher["Watcher Process"] --> Spawn["Spawn Worker"] - Spawn --> Monitor["Monitor CPU And Memory"] - Monitor --> Check{{"Limit Exceeded?"}} - Check -->|"No"| Continue["Continue Monitoring"] - Check -->|"Yes"| Kill["Stop Worker"] - Kill --> Respawn["Respawn Worker"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-5.mmd b/docs/diagrams/architecture/core-init-and-runtime-5.mmd deleted file mode 100644 index 50b08395e59..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-5.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - AnyThread["Any Thread"] --> Request["requestShutdown()"] - Request --> Signal["Condition Variable"] - Signal --> MainThread["Main Thread"] - MainThread --> StopServices["Dispatcher::stopServices()"] - StopServices --> Join["Join Services"] - Join --> StopEvents["EventFactory::end()"] - StopEvents --> CloseDB["shutdownDatabase()"] - CloseDB --> Exit["Return Exit Code"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-6.mmd b/docs/diagrams/architecture/core-init-and-runtime-6.mmd deleted file mode 100644 index 83e50573800..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-6.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart LR - Flag["host_identifier Flag"] --> Mode{{"Mode"}} - Mode -->|"uuid"| Hardware["Hardware UUID"] - Mode -->|"instance"| Instance["Instance UUID"] - Mode -->|"ephemeral"| Ephemeral["Random UUID"] - Mode -->|"specified"| Specified["Configured Identifier"] - Mode -->|"default"| Hostname["System Hostname"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-7.mmd b/docs/diagrams/architecture/core-init-and-runtime-7.mmd deleted file mode 100644 index 0533bd82ae3..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-7.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - SQL["SQL Query"] --> SQLite["SQLite Virtual Table API"] - SQLite --> QueryContext["QueryContext"] - QueryContext --> ConstraintMap["ConstraintMap"] - ConstraintMap --> ConstraintList["ConstraintList"] - ConstraintList --> TablePlugin["TablePlugin::generate()"] diff --git a/docs/diagrams/architecture/core-init-and-runtime-8.mmd b/docs/diagrams/architecture/core-init-and-runtime-8.mmd deleted file mode 100644 index dd5a802fe24..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime-8.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - Core["Core Init And Runtime"] - - Core --> Config["Configuration And Packs"] - Core --> Logging["Logging And Query Observability"] - Core --> Database["Database And Storage Plugins"] - Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework And Subscriptions"] - Core --> Extensions["Extensions And IPC"] diff --git a/docs/diagrams/architecture/core-init-and-runtime.mmd b/docs/diagrams/architecture/core-init-and-runtime.mmd deleted file mode 100644 index 75bf4fac1e6..00000000000 --- a/docs/diagrams/architecture/core-init-and-runtime.mmd +++ /dev/null @@ -1,16 +0,0 @@ -flowchart TD - Main["Main Entry Point"] --> Initializer["Initializer"] - - Initializer --> Flags["Flag System"] - Initializer --> Registry["Registry And Plugins"] - Initializer --> Database["Database Initialization"] - Initializer --> ExtensionManager["Extension Manager"] - Initializer --> Events["Event Factory"] - Initializer --> Watchdog["Watcher / Worker Model"] - - Watchdog --> Worker["Worker Process"] - Watchdog --> Extensions["Extension Processes"] - - Worker --> Tables["Virtual Tables"] - Worker --> Scheduler["Query Scheduler"] - Worker --> Logger["Logger Plugins"] diff --git a/docs/diagrams/architecture/core-runtime-and-lifecycle-2.mmd b/docs/diagrams/architecture/core-runtime-and-lifecycle-2.mmd new file mode 100644 index 00000000000..c0245a19ed1 --- /dev/null +++ b/docs/diagrams/architecture/core-runtime-and-lifecycle-2.mmd @@ -0,0 +1,5 @@ +flowchart LR + Define["FLAG Macro"] --> Create["Flag::create()"] + Create --> Registry["Flag Internal Map"] + Registry --> Query["getValue() / getType()"] + Registry --> Print["printFlags()"] diff --git a/docs/diagrams/architecture/core-runtime-and-lifecycle-3.mmd b/docs/diagrams/architecture/core-runtime-and-lifecycle-3.mmd new file mode 100644 index 00000000000..19fc4631fc7 --- /dev/null +++ b/docs/diagrams/architecture/core-runtime-and-lifecycle-3.mmd @@ -0,0 +1,11 @@ +flowchart TD + Initializer["Initializer"] --> Mode{{"ToolType"}} + Mode --> Shell["Shell Mode"] + Mode --> Daemon["Daemon Mode"] + Mode --> Extension["Extension Mode"] + + Daemon --> Watchdog{{"Watchdog Enabled?"}} + Watchdog -->|"Yes"| Watcher["Watcher Process"] + Watchdog -->|"No"| DirectRun["Single Process Runtime"] + + Watcher --> Worker["Worker Process"] diff --git a/docs/diagrams/architecture/core-runtime-and-lifecycle-4.mmd b/docs/diagrams/architecture/core-runtime-and-lifecycle-4.mmd new file mode 100644 index 00000000000..74c005f4a0c --- /dev/null +++ b/docs/diagrams/architecture/core-runtime-and-lifecycle-4.mmd @@ -0,0 +1,6 @@ +flowchart TD + Signal["SIGTERM / SIGINT / SIGUSR1"] --> Handler["signalHandler()"] + Handler --> Request["requestShutdown()"] + Request --> Notify["Condition Variable Notify"] + Notify --> MainWait["waitForShutdown()"] + MainWait --> GracefulStop["Dispatcher::stopServices()"] diff --git a/docs/diagrams/architecture/core-runtime-and-lifecycle-5.mmd b/docs/diagrams/architecture/core-runtime-and-lifecycle-5.mmd new file mode 100644 index 00000000000..adbe3c7a76d --- /dev/null +++ b/docs/diagrams/architecture/core-runtime-and-lifecycle-5.mmd @@ -0,0 +1,7 @@ +flowchart TD + ShutdownStart["Initializer::shutdown()"] --> StartAlarm["Spawn AlarmRunnable"] + StartAlarm --> StopServices["Stop & Join Services"] + StopServices --> EndEvents["EventFactory::end()"] + EndEvents --> ShutdownDB["shutdownDatabase()"] + ShutdownDB --> CancelAlarm["Interrupt AlarmRunnable"] + CancelAlarm --> Exit["Return Exit Code"] diff --git a/docs/diagrams/architecture/core-runtime-and-lifecycle.mmd b/docs/diagrams/architecture/core-runtime-and-lifecycle.mmd new file mode 100644 index 00000000000..6a3cef53adf --- /dev/null +++ b/docs/diagrams/architecture/core-runtime-and-lifecycle.mmd @@ -0,0 +1,24 @@ +flowchart TD + Entry["Process Entry Point"] --> Init["Initializer Constructor"] + Init --> ParseFlags["Parse Flags & CLI"] + ParseFlags --> SetupRegistry["Registry & Plugin Initialization"] + SetupRegistry --> ModeCheck{{"Shell / Daemon / Extension?"}} + + ModeCheck -->|"Shell"| ShellInit["Shell Initialization"] + ModeCheck -->|"Daemon"| DaemonInit["Daemon Initialization"] + ModeCheck -->|"Extension"| ExtInit["Extension Initialization"] + + ShellInit --> StartRuntime["Initializer.start()"] + DaemonInit --> StartRuntime + ExtInit --> StartRuntime + + StartRuntime --> LoadConfig["Load Config Plugin"] + LoadConfig --> ActivateLogger["Activate Logger Plugin"] + ActivateLogger --> ActivateDistributed["Activate Distributed Plugin"] + ActivateDistributed --> AttachEvents["Attach Event Subscribers"] + AttachEvents --> Running["Runtime Active"] + + Running -->|"Signal / Error"| ShutdownReq["requestShutdown()"] + ShutdownReq --> Graceful["Dispatcher Stop & Join"] + Graceful --> AlarmGuard["AlarmRunnable Guard"] + AlarmGuard --> Exit["Process Exit"] diff --git a/docs/diagrams/architecture/database-and-storage-plugins-2.mmd b/docs/diagrams/architecture/database-and-storage-plugins-2.mmd deleted file mode 100644 index 605504b8b71..00000000000 --- a/docs/diagrams/architecture/database-and-storage-plugins-2.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart TD - DB["EphemeralDatabasePlugin"] --> Map["std::map>"] - Map --> DomainMap["Domain Map"] - DomainMap --> KeyValue["boost::variant"] diff --git a/docs/diagrams/architecture/database-and-storage-plugins-3.mmd b/docs/diagrams/architecture/database-and-storage-plugins-3.mmd deleted file mode 100644 index cea2ef35f02..00000000000 --- a/docs/diagrams/architecture/database-and-storage-plugins-3.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - Start["Startup"] --> CheckFlag{{"disable_database?"}} - CheckFlag -->|"No"| UseRocksDB["Activate RocksDB Plugin"] - CheckFlag -->|"Yes"| UseEphemeral["Activate Ephemeral Plugin"] - UseRocksDB --> InitDone["Database Initialized"] - UseEphemeral --> InitDone diff --git a/docs/diagrams/architecture/database-and-storage-plugins-4.mmd b/docs/diagrams/architecture/database-and-storage-plugins-4.mmd deleted file mode 100644 index 721f5928c67..00000000000 --- a/docs/diagrams/architecture/database-and-storage-plugins-4.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - ResetCall["resetDatabase()"] --> LockWrite["Acquire Write Lock"] - LockWrite --> TearDown["Plugin tearDown()"] - TearDown --> SetUp["Plugin setUp()"] - SetUp --> Unlock["Release Lock"] diff --git a/docs/diagrams/architecture/database-and-storage-plugins-5.mmd b/docs/diagrams/architecture/database-and-storage-plugins-5.mmd deleted file mode 100644 index eed79ca554a..00000000000 --- a/docs/diagrams/architecture/database-and-storage-plugins-5.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - ReadVersion["Read results_version"] --> Compare["Compare with target version"] - Compare -->|"Older"| RunMigration["Execute migrateVxVy()"] - RunMigration --> UpdateVersion["Persist new version"] - UpdateVersion --> Compare - Compare -->|"Match"| Done["Migration Complete"] diff --git a/docs/diagrams/architecture/database-and-storage-plugins.mmd b/docs/diagrams/architecture/database-and-storage-plugins.mmd deleted file mode 100644 index a4efc77fa93..00000000000 --- a/docs/diagrams/architecture/database-and-storage-plugins.mmd +++ /dev/null @@ -1,19 +0,0 @@ -flowchart TD - CoreModules["Core And Feature Modules"] --> DBInterface["IDatabaseInterface"] - DBInterface --> OsqueryDB["OsqueryDatabase"] - OsqueryDB --> RegistryLayer["Database Plugin Registry"] - RegistryLayer --> RocksDBPlugin["RocksDB Plugin (Persistent)"] - RegistryLayer --> EphemeralPlugin["Ephemeral Database Plugin (In-Memory)"] - - subgraph storage_domains["Logical Storage Domains"] - PersistentSettings["configurations"] - Queries["queries"] - Events["events"] - Logs["logs"] - Carves["carves"] - Distributed["distributed"] - QueryPerformance["query_performance"] - end - - RocksDBPlugin --> storage_domains - EphemeralPlugin --> storage_domains diff --git a/docs/diagrams/architecture/database-backend-2.mmd b/docs/diagrams/architecture/database-backend-2.mmd new file mode 100644 index 00000000000..ea3265df9e4 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-2.mmd @@ -0,0 +1,8 @@ +flowchart TD + Start["Process Startup"] --> Init["initDatabasePlugin()"] + Init --> CheckDisable{"disable_database flag?"} + CheckDisable -->|"Yes"| UseEphemeral["Set Active Plugin: Ephemeral"] + CheckDisable -->|"No"| UseRocksDB["Set Active Plugin: RocksDB"] + UseEphemeral --> MarkInit["kDBInitialized = true"] + UseRocksDB --> MarkInit + MarkInit --> Ready["Database Ready"] diff --git a/docs/diagrams/architecture/database-backend-3.mmd b/docs/diagrams/architecture/database-backend-3.mmd new file mode 100644 index 00000000000..933143948d2 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-3.mmd @@ -0,0 +1,5 @@ +flowchart TD + Caller["Higher Level Module"] --> Helper["getDatabaseValue()"] + Helper --> Lock["Acquire Read Lock"] + Lock --> Plugin["Active DatabasePlugin"] + Plugin --> Result["Return Status and Value"] diff --git a/docs/diagrams/architecture/database-backend-4.mmd b/docs/diagrams/architecture/database-backend-4.mmd new file mode 100644 index 00000000000..ccf661641c7 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-4.mmd @@ -0,0 +1,5 @@ +flowchart LR + Client["Component Using IDatabaseInterface"] --> Interface["IDatabaseInterface"] + Interface --> OsqueryDB["OsqueryDatabase"] + OsqueryDB --> GlobalHelpers["Global Database Helpers"] + GlobalHelpers --> ActivePlugin["Active DatabasePlugin"] diff --git a/docs/diagrams/architecture/database-backend-5.mmd b/docs/diagrams/architecture/database-backend-5.mmd new file mode 100644 index 00000000000..7b0d1612c78 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-5.mmd @@ -0,0 +1,3 @@ +flowchart TD + DB["Ephemeral DB"] --> DomainMap["Map<Domain, Map<Key, Value>>"] + DomainMap --> KeyValue["Key -> Variant(int or string)"] diff --git a/docs/diagrams/architecture/database-backend-6.mmd b/docs/diagrams/architecture/database-backend-6.mmd new file mode 100644 index 00000000000..6de7e0315f0 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-6.mmd @@ -0,0 +1,6 @@ +flowchart TD + ResetCall["resetDatabase()"] --> RegistryCall["Registry::call action=reset"] + RegistryCall --> WriteLock["Acquire Write Lock"] + WriteLock --> TearDown["Plugin tearDown()"] + TearDown --> SetUp["Plugin setUp()"] + SetUp --> Done["Database Reinitialized"] diff --git a/docs/diagrams/architecture/database-backend-7.mmd b/docs/diagrams/architecture/database-backend-7.mmd new file mode 100644 index 00000000000..e84462568b4 --- /dev/null +++ b/docs/diagrams/architecture/database-backend-7.mmd @@ -0,0 +1,7 @@ +flowchart TD + Start["upgradeDatabase(target)"] --> ReadVersion["Read results_version"] + ReadVersion --> Compare{"Current != Target?"} + Compare -->|"Yes"| Migrate["Run Migration Step"] + Migrate --> UpdateVersion["Persist New Version"] + UpdateVersion --> Compare + Compare -->|"No"| Done["Upgrade Complete"] diff --git a/docs/diagrams/architecture/database-backend.mmd b/docs/diagrams/architecture/database-backend.mmd new file mode 100644 index 00000000000..8d428609973 --- /dev/null +++ b/docs/diagrams/architecture/database-backend.mmd @@ -0,0 +1,11 @@ +flowchart TD + CoreRuntime["Core Runtime and Lifecycle"] --> DatabaseBackend["Database Backend"] + + DatabaseBackend --> Registry["Registry Factory"] + DatabaseBackend --> RocksDBPlugin["RocksDB Plugin"] + DatabaseBackend --> EphemeralPlugin["Ephemeral Database Plugin"] + + DatabaseBackend --> ConfigModule["Configuration and Packs"] + DatabaseBackend --> LoggingModule["Logging and Query Metadata"] + DatabaseBackend --> DistributedModule["Distributed Querying"] + DatabaseBackend --> EventsModule["Events Core and Subscriptions"] diff --git a/docs/diagrams/architecture/distributed-querying-2.mmd b/docs/diagrams/architecture/distributed-querying-2.mmd index af1e3628736..9a1e365a4b4 100644 --- a/docs/diagrams/architecture/distributed-querying-2.mmd +++ b/docs/diagrams/architecture/distributed-querying-2.mmd @@ -1,11 +1,8 @@ flowchart TD - Start["Pull Updates"] --> Accept["acceptWork()"] - Accept --> Pending{"Pending Queries?"} - Pending -->|"Yes"| Run["runQueries()"] - Run --> Deny{"Denylisted?"} - Deny -->|"No"| Execute["Execute via SQL Engine"] - Execute --> Record["recordQueryPerformance()"] - Record --> Queue["addResult()"] - Queue --> Flush["flushCompleted()"] - Deny -->|"Yes"| Skip["Skip Execution"] - Flush --> End["Cycle Complete"] + Start["Start Loop"] --> Pull["pullUpdates()"] + Pull --> Check["Pending Queries?"] + Check -->|"Yes"| Run["runQueries()"] + Check -->|"No"| Sleep["Wait Interval"] + Run --> Flush["flushCompleted()"] + Flush --> Sleep + Sleep --> Pull diff --git a/docs/diagrams/architecture/distributed-querying-3.mmd b/docs/diagrams/architecture/distributed-querying-3.mmd index 2850535d935..e573649ebb8 100644 --- a/docs/diagrams/architecture/distributed-querying-3.mmd +++ b/docs/diagrams/architecture/distributed-querying-3.mmd @@ -1,6 +1,7 @@ flowchart TD - Incoming["Incoming Query"] --> Check["checkAndSetAsRunning()"] - Check --> Running{"Already Running?"} - Running -->|"Within Duration"| Block["Denylisted"] - Running -->|"Expired"| Allow["Allow Execution"] - Check -->|"First Time"| Allow + ReceiveWork["acceptWork()"] --> HasDiscovery["Has Discovery Query?"] + HasDiscovery -->|"No"| Enqueue["Enqueue Query"] + HasDiscovery -->|"Yes"| RunDiscovery["Execute Discovery"] + RunDiscovery --> DiscoveryResult["Rows Returned?"] + DiscoveryResult -->|"Yes"| Enqueue + DiscoveryResult -->|"No"| Skip["Skip Query"] diff --git a/docs/diagrams/architecture/distributed-querying-4.mmd b/docs/diagrams/architecture/distributed-querying-4.mmd index da9275ef2f4..f6217cbf1da 100644 --- a/docs/diagrams/architecture/distributed-querying-4.mmd +++ b/docs/diagrams/architecture/distributed-querying-4.mmd @@ -1,11 +1,7 @@ -sequenceDiagram - participant Node - participant TLS as "TLS Distributed Plugin" - participant Server - - Node->>TLS: pullUpdates() - TLS->>Server: POST read endpoint - Server->>TLS: JSON queries - TLS->>Node: return JSON - Node->>TLS: writeResults(JSON) - TLS->>Server: POST write endpoint +flowchart TD + Execute["runQueries()"] --> CheckRun["checkAndSetAsRunning"] + CheckRun -->|"Already Running"| WithinWindow["Within Denylist Duration?"] + WithinWindow -->|"Yes"| Skip["Skip Execution"] + WithinWindow -->|"No"| ExecuteQuery["Execute Query"] + CheckRun -->|"Not Running"| ExecuteQuery + ExecuteQuery --> Complete["setAsNotRunning"] diff --git a/docs/diagrams/architecture/distributed-querying-5.mmd b/docs/diagrams/architecture/distributed-querying-5.mmd new file mode 100644 index 00000000000..63ba0fbdd40 --- /dev/null +++ b/docs/diagrams/architecture/distributed-querying-5.mmd @@ -0,0 +1,5 @@ +flowchart TD + RunQuery["Execute SQL"] --> Measure["Measure Duration"] + Measure --> CountRows["Count Result Rows"] + CountRows --> Record["recordQueryPerformance"] + Record --> Store["Performance Map"] diff --git a/docs/diagrams/architecture/distributed-querying-6.mmd b/docs/diagrams/architecture/distributed-querying-6.mmd new file mode 100644 index 00000000000..49b3b814b91 --- /dev/null +++ b/docs/diagrams/architecture/distributed-querying-6.mmd @@ -0,0 +1,6 @@ +flowchart TD + AddResult["addResult()"] --> Buffer["results_ Vector"] + Buffer --> FlushCall["flushCompleted()"] + FlushCall --> Serialize["serializeResults()"] + Serialize --> PluginWrite["DistributedPlugin.writeResults()"] + PluginWrite --> Clear["Clear results_"] diff --git a/docs/diagrams/architecture/distributed-querying-7.mmd b/docs/diagrams/architecture/distributed-querying-7.mmd new file mode 100644 index 00000000000..716f98275d1 --- /dev/null +++ b/docs/diagrams/architecture/distributed-querying-7.mmd @@ -0,0 +1,11 @@ +flowchart TD + Server["Remote Server"] --> PluginPull["getQueries()"] + PluginPull --> Accept["acceptWork()"] + Accept --> Queue["Internal Queue"] + Queue --> Execute["runQueries()"] + Execute --> SQL["SQL Engine"] + SQL --> Results["DistributedQueryResult"] + Results --> Buffer["results_"] + Buffer --> Flush["flushCompleted()"] + Flush --> PluginPush["writeResults()"] + PluginPush --> Server diff --git a/docs/diagrams/architecture/distributed-querying.mmd b/docs/diagrams/architecture/distributed-querying.mmd index c57619110a1..939864f9ad9 100644 --- a/docs/diagrams/architecture/distributed-querying.mmd +++ b/docs/diagrams/architecture/distributed-querying.mmd @@ -1,10 +1,9 @@ flowchart TD - Server["Central Server"] -->|"HTTPS"| TLSPlugin["TLS Distributed Plugin"] - TLSPlugin -->|"getQueries()"| DistributedCore["Distributed Core"] - DistributedCore -->|"runQueries()"| SQLEngine["SQL Engine"] - SQLEngine -->|"QueryData"| DistributedCore - DistributedCore -->|"writeResults()"| TLSPlugin - TLSPlugin -->|"HTTPS"| Server - - DistributedCore -->|"recordQueryPerformance"| Observability["Query Observability"] - DistributedCore -->|"running state"| Storage["Database Plugins"] + RemoteService["Remote Service"] --> DistributedPlugin["Distributed Plugin"] + DistributedPlugin --> DistributedManager["Distributed"] + DistributedManager --> SQLExecutor["SQL Engine"] + SQLExecutor --> VirtualTables["Virtual Tables"] + DistributedManager --> DatabaseState["Database Backend"] + DistributedManager --> PerformanceTracker["Query Performance"] + DistributedManager --> ResultSerializer["Result Serialization"] + ResultSerializer --> DistributedPlugin diff --git a/docs/diagrams/architecture/distributed-tls-plugin-2.mmd b/docs/diagrams/architecture/distributed-tls-plugin-2.mmd new file mode 100644 index 00000000000..2a1a3bd6768 --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-2.mmd @@ -0,0 +1,7 @@ +flowchart TD + Start["Start setUp()"] --> ReadFlag["Read distributed_tls_read_endpoint"] + ReadFlag --> MakeReadURI["TLSRequestHelper makeURI()"] + MakeReadURI --> WriteFlag["Read distributed_tls_write_endpoint"] + WriteFlag --> MakeWriteURI["TLSRequestHelper makeURI()"] + MakeWriteURI --> StoreURIs["Store read_uri and write_uri"] + StoreURIs --> EndNode["END"] diff --git a/docs/diagrams/architecture/distributed-tls-plugin-3.mmd b/docs/diagrams/architecture/distributed-tls-plugin-3.mmd new file mode 100644 index 00000000000..5b740a2a0a5 --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-3.mmd @@ -0,0 +1,11 @@ +sequenceDiagram + participant Engine + participant TLSPlugin + participant TLSTransport + participant Remote + Engine->>TLSPlugin: getQueries() + TLSPlugin->>TLSTransport: go(JSONSerializer) + TLSTransport->>Remote: HTTPS POST read_uri + Remote-->>TLSTransport: JSON query payload + TLSTransport-->>TLSPlugin: Serialized JSON + TLSPlugin-->>Engine: Query JSON diff --git a/docs/diagrams/architecture/distributed-tls-plugin-4.mmd b/docs/diagrams/architecture/distributed-tls-plugin-4.mmd new file mode 100644 index 00000000000..008d01437df --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-4.mmd @@ -0,0 +1,7 @@ +flowchart TD + StartWrite["Start writeResults()"] --> ParseJSON["Parse input JSON"] + ParseJSON --> ValidCheck{{"Valid JSON?"}} + ValidCheck -->|"No"| ReturnError["Return error Status"] + ValidCheck -->|"Yes"| SendTLS["TLSRequestHelper go()"] + SendTLS --> IgnoreResp["Ignore server response"] + IgnoreResp --> EndWrite["END"] diff --git a/docs/diagrams/architecture/distributed-tls-plugin-5.mmd b/docs/diagrams/architecture/distributed-tls-plugin-5.mmd new file mode 100644 index 00000000000..79aa8fbf773 --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-5.mmd @@ -0,0 +1,5 @@ +flowchart LR + Scheduler["Distributed Scheduler"] --> Request["DistributedQueryRequest"] + Request --> TLSPlugin["Distributed Tls Plugin"] + TLSPlugin --> Result["DistributedQueryResult"] + Result --> Scheduler diff --git a/docs/diagrams/architecture/distributed-tls-plugin-6.mmd b/docs/diagrams/architecture/distributed-tls-plugin-6.mmd new file mode 100644 index 00000000000..7cb156ea3bb --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-6.mmd @@ -0,0 +1,7 @@ +flowchart TD + StartRetry["Start TLS request"] --> Attempt["Attempt HTTPS call"] + Attempt --> SuccessCheck{{"Success?"}} + SuccessCheck -->|"Yes"| ReturnOK["Return Status OK"] + SuccessCheck -->|"No"| RetryCheck{{"Attempts left?"}} + RetryCheck -->|"Yes"| Attempt + RetryCheck -->|"No"| ReturnFail["Return Status Error"] diff --git a/docs/diagrams/architecture/distributed-tls-plugin-7.mmd b/docs/diagrams/architecture/distributed-tls-plugin-7.mmd new file mode 100644 index 00000000000..6a38fad3127 --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin-7.mmd @@ -0,0 +1,3 @@ +flowchart LR + Registry["Registry"] --> DistributedCategory["distributed Category"] + DistributedCategory --> TLSPlugin["tls Plugin"] diff --git a/docs/diagrams/architecture/distributed-tls-plugin.mmd b/docs/diagrams/architecture/distributed-tls-plugin.mmd new file mode 100644 index 00000000000..7e0bf65e6e8 --- /dev/null +++ b/docs/diagrams/architecture/distributed-tls-plugin.mmd @@ -0,0 +1,5 @@ +flowchart LR + DistributedEngine["Distributed Query Engine"] -->|"getQueries()"| TLSPlugin["Distributed Tls Plugin"] + TLSPlugin -->|"HTTPS POST"| RemoteServer["Remote TLS Server"] + RemoteServer -->|"JSON Queries"| TLSPlugin + TLSPlugin -->|"writeResults()"| DistributedEngine diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-2.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-2.mmd deleted file mode 100644 index 0f4dd5e4690..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-2.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - RegisterSub["registerEventSubscriber()"] --> SetupSub["setUp()"] - SetupSub --> InitSub["init()"] - InitSub --> RunningSub["EVENT_RUNNING"] - - RegisterPub["registerEventPublisher()"] --> SetupPub["setUp()"] - SetupPub --> ReadyPub["EVENT_SETUP"] diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-3.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-3.mmd deleted file mode 100644 index 22c93bf1ac5..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-3.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Delay["delay()"] --> ThreadSpawn["spawn thread per publisher"] - ThreadSpawn --> RunLoop["run(type_id)"] - RunLoop --> PublisherRun["publisher->run()"] - PublisherRun --> Pause["pause(200ms)"] - Pause --> RunLoop - RunLoop -->|"isEnding()"| TearDown["tearDown()"] diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-4.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-4.mmd deleted file mode 100644 index 13f5eab16f8..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-4.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Callback["EventCallback"] --> AddBatch["addBatch()"] - AddBatch --> AssignID["generateEventIdentifier()"] - AssignID --> Persist["Database write"] - Persist --> IndexUpdate["update time index"] - - ExpireCheck["expireEventBatches()"] --> RemoveOld["delete expired batches"] diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-5.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-5.mmd deleted file mode 100644 index e165a93a604..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-5.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - SQLQuery["SELECT * FROM example_events"] --> GenTable["genTable()"] - GenTable --> GenerateRows["generateRows()"] - GenerateRows --> FilterTime["apply start/stop window"] - FilterTime --> YieldRows["yield Row to SQL engine"] diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-6.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-6.mmd deleted file mode 100644 index 2de3bd7f42b..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-6.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart LR - FactoryLock["factory_lock_"] --> ProtectPub["event_pubs_"] - FactoryLock --> ProtectSub["event_subs_"] - - SubscriberLock1["event_id_lock_"] --> IDGen["EventID generation"] - SubscriberLock2["event_record_lock_"] --> RecordWrite["record time bins"] - SubscriberLock3["event_query_record_"] --> QueryTracking["query usage tracking"] diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions-7.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions-7.mmd deleted file mode 100644 index 58804b2f0a6..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions-7.mmd +++ /dev/null @@ -1,16 +0,0 @@ -sequenceDiagram - participant Config - participant Factory as EventFactory - participant Pub as EventPublisher - participant Sub as EventSubscriber - participant DB as Database - participant SQL - - Config->>Factory: configUpdate() - Factory->>Sub: setMinExpiry() - Factory->>Pub: spawn run loop - Pub->>Sub: fire event (callback) - Sub->>DB: addBatch() - SQL->>Sub: SELECT from event table - Sub->>DB: generateRows() - Sub->>SQL: yield rows diff --git a/docs/diagrams/architecture/eventing-framework-and-subscriptions.mmd b/docs/diagrams/architecture/eventing-framework-and-subscriptions.mmd deleted file mode 100644 index 169b0f41f29..00000000000 --- a/docs/diagrams/architecture/eventing-framework-and-subscriptions.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - Config["Configuration And Packs"] -->|"scheduled queries"| EventFactory["EventFactory"] - EventFactory -->|"registers"| Publisher["EventPublisherPlugin"] - EventFactory -->|"registers"| Subscriber["EventSubscriberPlugin"] - Subscriber -->|"creates"| SubscriptionObj["Subscription"] - Publisher -->|"dispatches"| Callback["EventCallback"] - Callback -->|"addBatch()"| Storage["Database And Storage Plugins"] - SQL["SQL Engine And Virtual Tables"] -->|"SELECT from _events tables"| Subscriber - Subscriber -->|"generateRows()"| SQL diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-2.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-2.mmd new file mode 100644 index 00000000000..b61aad6bc2e --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-2.mmd @@ -0,0 +1,7 @@ +flowchart TD + RegisterSub["registerEventSubscriber()"] --> Setup["setUp()"] + Setup --> Init["init()"] + Init --> Running["EVENT_RUNNING"] + + RegisterPub["registerEventPublisher()"] --> PubSetup["publisher.setUp()"] + PubSetup --> Stored["stored in factory map"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-3.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-3.mmd new file mode 100644 index 00000000000..a302d003aee --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-3.mmd @@ -0,0 +1,6 @@ +flowchart TD + Delay["EventFactory.delay()"] --> Spawn["spawn thread per publisher"] + Spawn --> RunLoop["EventFactory.run(type)"] + RunLoop --> Loop["publisher.run() loop"] + Loop --> Pause["pause 200ms"] + Loop -->|"error"| TearDown["tearDown()"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-4.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-4.mmd new file mode 100644 index 00000000000..f56d012279f --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-4.mmd @@ -0,0 +1,5 @@ +flowchart TD + Context["Context"] --> Namespace["database_namespace"] + Context --> Index["event_index"] + Context --> LastTime["last_query_time"] + Context --> LastEvent["last_event_id"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-5.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-5.mmd new file mode 100644 index 00000000000..e292f00a045 --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-5.mmd @@ -0,0 +1,4 @@ +flowchart TD + Publisher["EventPublisher"] --> Callback["EventCallback"] + Callback --> AddBatch["addBatch(rows)"] + AddBatch --> DBWrite["Database Backend"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-6.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-6.mmd new file mode 100644 index 00000000000..906f5ac8095 --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-6.mmd @@ -0,0 +1,5 @@ +flowchart TD + SQLQuery["SELECT * FROM event_table"] --> GenTable["genTable()"] + GenTable --> GenerateRows["generateRows()"] + GenerateRows --> Filter["time window filtering"] + Filter --> Yield["RowYield callback"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-7.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-7.mmd new file mode 100644 index 00000000000..25723ef1def --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-7.mmd @@ -0,0 +1,4 @@ +flowchart LR + Sub["EventSubscriber"] -->|"creates"| Subscription + Subscription -->|"context + callback"| Publisher + Publisher -->|"fires"| Callback diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-8.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-8.mmd new file mode 100644 index 00000000000..63776ff8a38 --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-8.mmd @@ -0,0 +1,5 @@ +flowchart TD + Scheduled["Scheduled Queries"] --> Scan["scan for event tables"] + Scan --> Compute["compute max interval + count"] + Compute --> SetExpiry["setMinExpiry()"] + SetExpiry --> ResetCount["resetQueryCount()"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions-9.mmd b/docs/diagrams/architecture/events-core-and-subscriptions-9.mmd new file mode 100644 index 00000000000..cfe73edd0be --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions-9.mmd @@ -0,0 +1,8 @@ +flowchart TD + Start["Startup"] --> Register["Register Publishers & Subscribers"] + Register --> Config["Apply Config Enable/Disable"] + Config --> Delay["Spawn Publisher Threads"] + Delay --> Running["Event Loop Running"] + Running --> Query["SQL Query Execution"] + Query --> Expire["Expiration & Optimization"] + Expire --> Shutdown["EventFactory.end()"] diff --git a/docs/diagrams/architecture/events-core-and-subscriptions.mmd b/docs/diagrams/architecture/events-core-and-subscriptions.mmd new file mode 100644 index 00000000000..3598c240e1c --- /dev/null +++ b/docs/diagrams/architecture/events-core-and-subscriptions.mmd @@ -0,0 +1,20 @@ +flowchart TD + Config["Configuration And Packs"] -->|"scheduled queries"| EventFactory["EventFactory"] + + subgraph publishers["Event Publishers"] + Pub1["EventPublisherPlugin"] + end + + subgraph subscribers["Event Subscribers"] + Sub1["EventSubscriberPlugin"] + end + + EventFactory -->|"register"| Pub1 + EventFactory -->|"register"| Sub1 + + Sub1 -->|"create"| Subscription["Subscription"] + Subscription -->|"attach"| Pub1 + + Pub1 -->|"fires events"| Sub1 + Sub1 -->|"persist rows"| DB["Database Backend"] + Sub1 -->|"expose table"| SQL["SQL Engine And Virtual Tables"] diff --git a/docs/diagrams/architecture/extensions-and-ipc-2.mmd b/docs/diagrams/architecture/extensions-and-ipc-2.mmd deleted file mode 100644 index 566b1b1c48c..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-2.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Client["ExtensionClient"] --> Transport["Thrift Transport"] - Transport --> Socket["UNIX Socket or Named Pipe"] - Socket --> Server["ExtensionRunner or ManagerRunner"] - Server --> Handler["ExtensionHandler or ExtensionManagerHandler"] - Handler --> Interface["ExtensionInterface / ExtensionManagerInterface"] - Interface --> Registry["RegistryFactory"] diff --git a/docs/diagrams/architecture/extensions-and-ipc-3.mmd b/docs/diagrams/architecture/extensions-and-ipc-3.mmd deleted file mode 100644 index 9aabee40835..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-3.mmd +++ /dev/null @@ -1,10 +0,0 @@ -sequenceDiagram - participant Ext as Extension Process - participant EM as Extension Manager - participant Reg as RegistryFactory - - Ext->>EM: registerExtension(info, registry) - EM->>EM: Validate name uniqueness - EM->>EM: Check SDK compatibility - EM->>Reg: addBroadcast(uuid, registry) - EM-->>Ext: Return UUID diff --git a/docs/diagrams/architecture/extensions-and-ipc-4.mmd b/docs/diagrams/architecture/extensions-and-ipc-4.mmd deleted file mode 100644 index 642a20aeb10..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-4.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart LR - SQL["SQL Engine"] --> ExternalSQL["ExternalSQLPlugin"] - ExternalSQL --> ManagerClient["ExtensionManagerClient"] - ManagerClient --> Extension["Extension Process"] diff --git a/docs/diagrams/architecture/extensions-and-ipc-5.mmd b/docs/diagrams/architecture/extensions-and-ipc-5.mmd deleted file mode 100644 index a4a24d258ed..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-5.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Watcher["ExtensionManagerWatcher"] --> UUIDs["Registered UUIDs"] - UUIDs --> Ping["Ping Extension"] - Ping -->|"Success"| Keep["Keep Registered"] - Ping -->|"Failure x2"| Remove["Remove Broadcast Route"] diff --git a/docs/diagrams/architecture/extensions-and-ipc-6.mmd b/docs/diagrams/architecture/extensions-and-ipc-6.mmd deleted file mode 100644 index 1844fc82272..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-6.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Start["Core Start"] --> StartManager["startExtensionManager()"] - StartManager --> Watcher["ExtensionManagerWatcher"] - Watcher --> Runner["ExtensionManagerRunner"] - Runner --> Listen["Thrift Listen"] diff --git a/docs/diagrams/architecture/extensions-and-ipc-7.mmd b/docs/diagrams/architecture/extensions-and-ipc-7.mmd deleted file mode 100644 index 82e492bb100..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc-7.mmd +++ /dev/null @@ -1,6 +0,0 @@ -flowchart TD - ExtStart["Extension Binary Start"] --> Connect["Connect to Manager Socket"] - Connect --> Register["registerExtension()"] - Register --> UUID["Receive UUID"] - UUID --> StartServer["Start ExtensionRunner"] - StartServer --> Serve["Serve Thrift Requests"] diff --git a/docs/diagrams/architecture/extensions-and-ipc.mmd b/docs/diagrams/architecture/extensions-and-ipc.mmd deleted file mode 100644 index 343a33e6b65..00000000000 --- a/docs/diagrams/architecture/extensions-and-ipc.mmd +++ /dev/null @@ -1,12 +0,0 @@ -flowchart LR - Core["osquery Core"] -->|"Starts"| Manager["Extension Manager"] - Manager -->|"Registers routes"| Registry["RegistryFactory"] - - ExtensionProc["Extension Process"] -->|"registerExtension()"| Manager - Manager -->|"Assign UUID"| ExtensionProc - - Core -->|"callExtension()"| Manager - Manager -->|"Route to UUID"| ExtensionProc - - ExtensionProc -->|"Response"| Manager - Manager -->|"PluginResponse"| Core diff --git a/docs/diagrams/architecture/extensions-framework-2.mmd b/docs/diagrams/architecture/extensions-framework-2.mmd new file mode 100644 index 00000000000..ec4abee316d --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-2.mmd @@ -0,0 +1,5 @@ +flowchart LR + Ext["Extension Process"] -->|"registerExtension"| EM["ExtensionManagerInterface"] + EM --> UUID["Assign RouteUUID"] + EM --> Registry["RegistryFactory.addBroadcast"] + EM --> Store["Store ExtensionInfo"] diff --git a/docs/diagrams/architecture/extensions-framework-3.mmd b/docs/diagrams/architecture/extensions-framework-3.mmd new file mode 100644 index 00000000000..4a33447d53d --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-3.mmd @@ -0,0 +1,5 @@ +flowchart TD + Client["ExtensionClient"] --> Socket["UNIX Socket / Named Pipe"] + Socket --> ThriftServer["TThreadedServer"] + ThriftServer --> Handler["ExtensionHandler or ExtensionManagerHandler"] + Handler --> Interface["ExtensionInterface Logic"] diff --git a/docs/diagrams/architecture/extensions-framework-4.mmd b/docs/diagrams/architecture/extensions-framework-4.mmd new file mode 100644 index 00000000000..9d44736668b --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-4.mmd @@ -0,0 +1,5 @@ +flowchart TD + Start["startExtensionManager"] --> Check["Check socketExists"] + Check --> Watcher["Start ExtensionManagerWatcher"] + Watcher --> Runner["Start ExtensionManagerRunner"] + Runner --> Ready["Manager Listening"] diff --git a/docs/diagrams/architecture/extensions-framework-5.mmd b/docs/diagrams/architecture/extensions-framework-5.mmd new file mode 100644 index 00000000000..ca43ee332c5 --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-5.mmd @@ -0,0 +1,6 @@ +flowchart TD + ExtStart["Extension startExtension"] --> Wait["extensionPathActive"] + Wait --> Register["registerExtension"] + Register --> UUID["Receive RouteUUID"] + UUID --> Watcher["Start ExtensionWatcher"] + Watcher --> Runner["Start ExtensionRunner"] diff --git a/docs/diagrams/architecture/extensions-framework-6.mmd b/docs/diagrams/architecture/extensions-framework-6.mmd new file mode 100644 index 00000000000..a6a39233e65 --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-6.mmd @@ -0,0 +1,4 @@ +flowchart LR + ManagerWatcher["ExtensionManagerWatcher"] --> PingExt["Ping Extension"] + PingExt -->|"Success"| Healthy["Keep Registered"] + PingExt -->|"Failure x2"| Remove["removeBroadcast(uuid)"] diff --git a/docs/diagrams/architecture/extensions-framework-7.mmd b/docs/diagrams/architecture/extensions-framework-7.mmd new file mode 100644 index 00000000000..d2f32cfee44 --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework-7.mmd @@ -0,0 +1,7 @@ +flowchart TD + CoreCall["RegistryFactory.call"] --> Resolve["Resolve RouteUUID"] + Resolve --> IPC["callExtension()"] + IPC --> Socket["Thrift RPC"] + Socket --> ExtHandler["ExtensionHandler.call"] + ExtHandler --> Plugin["Local Plugin Execution"] + Plugin --> Response["Return PluginResponse"] diff --git a/docs/diagrams/architecture/extensions-framework.mmd b/docs/diagrams/architecture/extensions-framework.mmd new file mode 100644 index 00000000000..1e2d8b3aa13 --- /dev/null +++ b/docs/diagrams/architecture/extensions-framework.mmd @@ -0,0 +1,12 @@ +flowchart TD + Core["osquery Core"] --> Manager["Extension Manager"] + Manager --> Registry["RegistryFactory"] + + ExtensionA["Extension Process A"] -->|"registerExtension()"| Manager + ExtensionB["Extension Process B"] -->|"registerExtension()"| Manager + + Core -->|"callExtension()"| ExtensionA + Core -->|"callExtension()"| ExtensionB + + Manager -->|"health checks"| ExtensionA + Manager -->|"health checks"| ExtensionB diff --git a/docs/diagrams/architecture/filesystem-and-fileops-core-2.mmd b/docs/diagrams/architecture/filesystem-and-fileops-core-2.mmd new file mode 100644 index 00000000000..ee7ab355e54 --- /dev/null +++ b/docs/diagrams/architecture/filesystem-and-fileops-core-2.mmd @@ -0,0 +1,6 @@ +flowchart TD + Create["Create PlatformFile"] --> Open["Open Native Handle"] + Open --> Valid{"Valid Handle?"} + Valid -->|Yes| Operate["Read or Write or Seek"] + Valid -->|No| Error["Return Failure"] + Operate --> Close["Destructor Closes Handle"] diff --git a/docs/diagrams/architecture/filesystem-and-fileops-core-3.mmd b/docs/diagrams/architecture/filesystem-and-fileops-core-3.mmd new file mode 100644 index 00000000000..522761208c3 --- /dev/null +++ b/docs/diagrams/architecture/filesystem-and-fileops-core-3.mmd @@ -0,0 +1,7 @@ +flowchart TD + Pattern["Input Pattern"] --> Normalize["Canonicalize Base Path"] + Normalize --> Expand["Apply Wildcards"] + Expand --> DetectLoop{"Symlink Loop?"} + DetectLoop -->|Yes| Stop["Stop Recursion"] + DetectLoop -->|No| Recurse["Continue Expansion"] + Recurse --> Filter["Apply Folder or File Filter"] diff --git a/docs/diagrams/architecture/filesystem-and-fileops-core-4.mmd b/docs/diagrams/architecture/filesystem-and-fileops-core-4.mmd new file mode 100644 index 00000000000..6a2705e5f8f --- /dev/null +++ b/docs/diagrams/architecture/filesystem-and-fileops-core-4.mmd @@ -0,0 +1,7 @@ +flowchart TD + Caller["Caller"] --> PF["PlatformFile"] + PF --> OSCheck{"Operating System"} + OSCheck -->|POSIX| POSIXIO["open read write lseek"] + OSCheck -->|Windows| WINIO["CreateFile ReadFile WriteFile"] + POSIXIO --> Result["Normalized Result"] + WINIO --> Result diff --git a/docs/diagrams/architecture/filesystem-and-fileops-core.mmd b/docs/diagrams/architecture/filesystem-and-fileops-core.mmd new file mode 100644 index 00000000000..8c585216854 --- /dev/null +++ b/docs/diagrams/architecture/filesystem-and-fileops-core.mmd @@ -0,0 +1,14 @@ +flowchart TD + HighLevel["High-Level Consumers"] --> FilesystemAPI["Filesystem API"] + FilesystemAPI --> PlatformFile["PlatformFile"] + FilesystemAPI --> GlobEngine["Glob and Pattern Resolver"] + FilesystemAPI --> PermissionChecks["Permission and Safety Checks"] + + PlatformFile --> PosixImpl["POSIX Implementation"] + PlatformFile --> WindowsImpl["Windows Implementation"] + + GlobEngine --> PosixGlob["POSIX glob()"] + GlobEngine --> WindowsGlob["Windows FindFirstFile"] + + PermissionChecks --> PosixPerms["POSIX chmod and stat"] + PermissionChecks --> WindowsACL["Windows ACL and SID Logic"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-2.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-2.mmd deleted file mode 100644 index 58aae8b080a..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-2.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Open["Open PlatformFile"] --> Validate["Handle Valid?"] - Validate -->|No| Fail["Return Error"] - Validate -->|Yes| IO["Read / Write / Seek"] - IO --> Close["Destructor Closes Handle"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-3.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-3.mmd deleted file mode 100644 index 26f4fd78970..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-3.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart LR - Path["File Path"] --> StatCall["platformStat()"] - StatCall -->|POSIX| POSIXStat["struct stat"] - StatCall -->|Windows| WinStat["WINDOWS_STAT"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-4.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-4.mmd deleted file mode 100644 index 8f8a50b070d..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-4.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Pattern["Input Pattern"] --> Normalize["replaceGlobWildcards()"] - Normalize --> Expand["platformGlob()"] - Expand --> Filter["Apply GLOB_FILES / GLOB_FOLDERS"] - Filter --> Results["Resolved Paths"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-5.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-5.mmd deleted file mode 100644 index 9efad4a8202..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-5.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - ReadCall["readFile()"] --> Open["PlatformFile"] - Open --> SizeCheck["checkFileReadLimit()"] - SizeCheck --> Loop["Block Read Loop"] - Loop --> Predicate["Callback / Buffer"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-6.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-6.mmd deleted file mode 100644 index 8b63dd13550..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-6.mmd +++ /dev/null @@ -1,3 +0,0 @@ -flowchart LR - getMounted["getMountedFilesystems()"] --> MountInfo["MountInformation"] - MountInfo --> StatFS["StatFsInfo"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-7.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-7.mmd deleted file mode 100644 index 60fca930f80..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-7.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart TD - Proc["/proc"] --> Enumerate["procEnumerateProcesses()"] - Enumerate --> FDEnum["procEnumerateProcessDescriptors()"] - FDEnum --> SocketMap["SocketInodeToProcessInfoMap"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities-8.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities-8.mmd deleted file mode 100644 index 1359e2b787f..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities-8.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - File["Candidate File"] --> Exists["Exists?"] - Exists -->|No| Reject["Reject"] - Exists --> Owner["Owner Root or Current User?"] - Owner -->|No| Reject - Owner --> Writable["World Writable?"] - Writable -->|Yes| Reject - Writable --> Exec["Executable Required?"] - Exec --> Accept["Safe"] diff --git a/docs/diagrams/architecture/filesystem-and-path-utilities.mmd b/docs/diagrams/architecture/filesystem-and-path-utilities.mmd deleted file mode 100644 index 44c2abfbebc..00000000000 --- a/docs/diagrams/architecture/filesystem-and-path-utilities.mmd +++ /dev/null @@ -1,21 +0,0 @@ -flowchart TD - Caller["Core / SQL / Extensions"] --> FSAPI["Filesystem API"] - - subgraph abstraction_layer["Cross-Platform Abstraction"] - FSAPI --> PlatformFile["PlatformFile"] - FSAPI --> Glob["platformGlob()"] - FSAPI --> Perms["platformChmod()"] - FSAPI --> Access["platformAccess()"] - FSAPI --> Stat["platformStat() / lstat()"] - end - - subgraph posix_layer["POSIX Implementation"] - PlatformFile --> POSIXFile["open/read/write/lstat"] - Glob --> POSIXGlob["glob()"] - end - - subgraph windows_layer["Windows Implementation"] - PlatformFile --> WinFile["CreateFile / ReadFile"] - Glob --> WinGlob["FindFirstFileW"] - Perms --> WinACL["ACL Manipulation"] - end diff --git a/docs/diagrams/architecture/logging-and-query-metadata-2.mmd b/docs/diagrams/architecture/logging-and-query-metadata-2.mmd new file mode 100644 index 00000000000..e6882d71af5 --- /dev/null +++ b/docs/diagrams/architecture/logging-and-query-metadata-2.mmd @@ -0,0 +1,7 @@ +flowchart TD + Start["Scheduled Query Execution"] --> FetchOld["getPreviousQueryResults()"] + FetchOld --> Exec["Execute SQL"] + Exec --> Diff["diff(old, new)"] + Diff --> Store["saveQueryResults()"] + Store --> BuildLog["Construct QueryLogItem"] + BuildLog --> End["Emit Log"] diff --git a/docs/diagrams/architecture/logging-and-query-metadata-3.mmd b/docs/diagrams/architecture/logging-and-query-metadata-3.mmd new file mode 100644 index 00000000000..bd750be4d45 --- /dev/null +++ b/docs/diagrams/architecture/logging-and-query-metadata-3.mmd @@ -0,0 +1,4 @@ +flowchart LR + Logger["LoggerPlugin"] -->|"usesLogStatus()"| StatusHandling["Handle Glog Status"] + Logger -->|"usesLogEvent()"| EventHandling["Handle Event Forwarding"] + Logger -->|"logSnapshot()"| SnapshotHandling["Snapshot Optimization"] diff --git a/docs/diagrams/architecture/logging-and-query-metadata-4.mmd b/docs/diagrams/architecture/logging-and-query-metadata-4.mmd new file mode 100644 index 00000000000..e8a3d61c178 --- /dev/null +++ b/docs/diagrams/architecture/logging-and-query-metadata-4.mmd @@ -0,0 +1,12 @@ +flowchart TD + Config["Configuration And Packs"] --> SchedulerExec["Scheduled Query"] + SchedulerExec --> SQLEngine["SQL Engine"] + SQLEngine --> QueryState["Query"] + QueryState --> DiffLayer["DiffResults"] + QueryState --> Perf["QueryPerformance"] + DiffLayer --> LogItem["QueryLogItem"] + Perf --> LogItem + LogItem --> Serializer["JSON Serialization"] + Serializer --> Logger["LoggerPlugin"] + Logger --> Sink["External Sink"] + QueryState --> Database["Database Backend"] diff --git a/docs/diagrams/architecture/logging-and-query-metadata.mmd b/docs/diagrams/architecture/logging-and-query-metadata.mmd new file mode 100644 index 00000000000..be37dcd0ffd --- /dev/null +++ b/docs/diagrams/architecture/logging-and-query-metadata.mmd @@ -0,0 +1,9 @@ +flowchart TD + Scheduler["Configuration And Packs"] -->|"creates"| ScheduledQuery + ScheduledQuery -->|"executes"| SQLEngine["SQL Engine And Virtual Tables"] + SQLEngine -->|"returns rows"| QueryLayer["Query"] + QueryLayer -->|"computes diff"| DiffResultsNode["DiffResults"] + QueryLayer -->|"builds"| QueryLogItemNode["QueryLogItem"] + QueryLogItemNode -->|"serialize"| LoggerPluginNode["LoggerPlugin"] + LoggerPluginNode -->|"emits"| ExternalSystem["External Logging System"] + QueryLayer -->|"store previous"| DatabaseBackend["Database Backend"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-2.mmd b/docs/diagrams/architecture/logging-and-query-observability-2.mmd deleted file mode 100644 index 2feabab2fc7..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-2.mmd +++ /dev/null @@ -1,7 +0,0 @@ -flowchart TD - Init["System Initialization"] --> Buffer["Buffer Early Status Logs"] - Buffer --> LoadConfig["Load Configuration"] - LoadConfig --> Discover["Discover Logger Plugin"] - Discover --> InitLogger["LoggerPlugin.init()"] - InitLogger --> Flush["Flush Buffered StatusLogLine Entries"] - Flush --> Runtime["Runtime Logging"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-3.mmd b/docs/diagrams/architecture/logging-and-query-observability-3.mmd deleted file mode 100644 index 7c5a848377e..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-3.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart LR - Old["Previous Results"] --> DiffEngine["diff()"] - New["Current Results"] --> DiffEngine - DiffEngine --> Added["Added Rows"] - DiffEngine --> Removed["Removed Rows"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-4.mmd b/docs/diagrams/architecture/logging-and-query-observability-4.mmd deleted file mode 100644 index 93386e9e467..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-4.mmd +++ /dev/null @@ -1,4 +0,0 @@ -flowchart TD - Diff["DiffResults"] --> Serialize["serializeDiffResults()"] - Serialize --> JSONDoc["JSON Document"] - JSONDoc --> Logger["LoggerPlugin"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-5.mmd b/docs/diagrams/architecture/logging-and-query-observability-5.mmd deleted file mode 100644 index 30d12735a02..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-5.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - Item["QueryLogItem"] --> JSON1["serializeQueryLogItem()"] - Item --> JSON2["serializeQueryLogItemJSON()"] - Item --> Events1["serializeQueryLogItemAsEvents()"] - Item --> Events2["serializeQueryLogItemAsEventsJSON()"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-6.mmd b/docs/diagrams/architecture/logging-and-query-observability-6.mmd deleted file mode 100644 index 99c77b8ea17..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-6.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - Config["ScheduledQuery"] --> Exec["SQL Execution"] - Exec --> Results["QueryDataTyped"] - Results --> QueryState["Query.addNewResults()"] - QueryState --> Diff["DiffResults"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serialize["JSON Serialization"] - Serialize --> Logger["LoggerPlugin"] - Logger --> Sink["External Log System"] diff --git a/docs/diagrams/architecture/logging-and-query-observability-7.mmd b/docs/diagrams/architecture/logging-and-query-observability-7.mmd deleted file mode 100644 index 88b889b3b8e..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability-7.mmd +++ /dev/null @@ -1,5 +0,0 @@ -flowchart TD - SQ["ScheduledQuery snapshot=true"] --> Exec["Execute Query"] - Exec --> Full["Full Result Set"] - Full --> LogItem["QueryLogItem isSnapshot=true"] - LogItem --> Logger["logSnapshot()"] diff --git a/docs/diagrams/architecture/logging-and-query-observability.mmd b/docs/diagrams/architecture/logging-and-query-observability.mmd deleted file mode 100644 index 82b06f91f54..00000000000 --- a/docs/diagrams/architecture/logging-and-query-observability.mmd +++ /dev/null @@ -1,15 +0,0 @@ -flowchart TD - Config["Configuration And Packs"] --> Scheduler["Scheduled Query"] - Scheduler --> SQ["ScheduledQuery Struct"] - SQ --> QueryObj["Query State Manager"] - QueryObj --> DB["Database And Storage Plugins"] - - QueryObj --> Diff["DiffResults Engine"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serializer["JSON Serialization"] - Serializer --> Logger["LoggerPlugin"] - - StatusLogs["Internal Status Events"] --> StatusLine["StatusLogLine"] - StatusLine --> Logger - - Logger --> External["External Logging Backend"] diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-2.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-2.mmd new file mode 100644 index 00000000000..834d4774cf0 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-2.mmd @@ -0,0 +1,10 @@ +flowchart TD + A["Read Input Line"] --> B{"Starts With Dot?"} + B -->|"Yes"| C["Meta Command Handler"] + B -->|"No"| D["Accumulate SQL"] + D --> E{"Complete Statement?"} + E -->|"No"| A + E -->|"Yes"| F["Execute SQL"] + F --> G["shell_callback"] + G --> H["Render Output Mode"] + H --> A diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-3.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-3.mmd new file mode 100644 index 00000000000..530e32ff190 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-3.mmd @@ -0,0 +1,11 @@ +classDiagram + class callback_data { + +int echoOn + +int mode + +int showHeader + +char separator[20] + +char nullvalue[20] + +FILE* out + +sqlite3_stmt* pStmt + +prettyprint_data* prettyPrint + } diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-4.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-4.mmd new file mode 100644 index 00000000000..5d790f5bb66 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-4.mmd @@ -0,0 +1,8 @@ +classDiagram + class prettyprint_data { + +QueryData results + +vector columns + +map lengths + } + + callback_data --> prettyprint_data diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-5.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-5.mmd new file mode 100644 index 00000000000..7f4952e1b8c --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-5.mmd @@ -0,0 +1,5 @@ +flowchart TD + Start["BEGIN_TIMER"] --> CaptureStart["Capture rusage + Wall Time"] + CaptureStart --> Execute["Run Query"] + Execute --> CaptureEnd["Capture End rusage"] + CaptureEnd --> Print["Print real, user, sys"] diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-6.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-6.mmd new file mode 100644 index 00000000000..a6241074696 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-6.mmd @@ -0,0 +1,5 @@ +flowchart LR + Query["SQL Query"] --> Check{"Connect Flag Set?"} + Check -->|"No"| LocalExec["shell_exec Local"] + Check -->|"Yes"| RemoteExec["shell_exec_remote"] + RemoteExec --> ExtClient["ExtensionManagerClient"] diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-7.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-7.mmd new file mode 100644 index 00000000000..8323c135a2b --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-7.mmd @@ -0,0 +1,5 @@ +flowchart TD + Input[".command args"] --> Tokenize["Tokenize Arguments"] + Tokenize --> Match["Command Switch"] + Match --> Execute["Meta Handler Function"] + Execute --> Output["Print To Shell"] diff --git a/docs/diagrams/architecture/osquery-shell-devtooling-8.mmd b/docs/diagrams/architecture/osquery-shell-devtooling-8.mmd new file mode 100644 index 00000000000..ce5479fad67 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling-8.mmd @@ -0,0 +1,5 @@ +flowchart TD + Start["Launch With Pack Flag"] --> LoadConfig["Load Packs From Config"] + LoadConfig --> Iterate["Iterate Scheduled Queries"] + Iterate --> Run["runQuery"] + Run --> Output["Formatted Results"] diff --git a/docs/diagrams/architecture/osquery-shell-devtooling.mmd b/docs/diagrams/architecture/osquery-shell-devtooling.mmd new file mode 100644 index 00000000000..c5ef8802b28 --- /dev/null +++ b/docs/diagrams/architecture/osquery-shell-devtooling.mmd @@ -0,0 +1,13 @@ +flowchart TD + User["User Terminal"] --> Shell["Osquery Shell Devtooling"] + Shell --> Parser["Meta Command Parser"] + Shell --> Executor["SQL Executor"] + + Executor --> SQLite["SQLite Engine"] + SQLite --> VTables["Virtual Tables"] + VTables --> Registry["Table Registry"] + + Executor --> DBManager["SQLiteDBManager"] + Executor --> Extensions["Extension Manager Client"] + + Extensions --> RemoteExt["Remote Extension Socket"] diff --git a/docs/diagrams/architecture/remote-http-client-2.mmd b/docs/diagrams/architecture/remote-http-client-2.mmd index 810bffb2210..5a3e5abaccf 100644 --- a/docs/diagrams/architecture/remote-http-client-2.mmd +++ b/docs/diagrams/architecture/remote-http-client-2.mmd @@ -1,5 +1,6 @@ flowchart LR - OldOpts["Current Options"] --> Compare["operator=="] - NewOpts["New Options"] --> Compare - Compare -->|"Different"| Reconfigure["Reinitialize Connection"] - Compare -->|"Same"| Reuse["Reuse Existing Settings"] + Options["Client Options"] --> SSL["SSL Configuration"] + Options --> Proxy["Proxy Settings"] + Options --> Timeout["Timeout Control"] + Options --> Redirects["Redirect Handling"] + Options --> KeepAlive["Keep Alive"] diff --git a/docs/diagrams/architecture/remote-http-client-3.mmd b/docs/diagrams/architecture/remote-http-client-3.mmd index 6445d351559..3afc2ca34cf 100644 --- a/docs/diagrams/architecture/remote-http-client-3.mmd +++ b/docs/diagrams/architecture/remote-http-client-3.mmd @@ -1,4 +1,4 @@ flowchart TD - Response["HTTP_Response"] --> Headers["Headers"] - Headers --> Iterator["Iterator"] - Iterator --> Pair["(name, value)"] + Request["HTTP Request"] --> URI["URI Parser"] + Request --> Headers["Header Injection"] + Request --> Protocol["Scheme Detection"] diff --git a/docs/diagrams/architecture/remote-http-client-4.mmd b/docs/diagrams/architecture/remote-http-client-4.mmd index 52c275c5be9..ee0bac1f3a3 100644 --- a/docs/diagrams/architecture/remote-http-client-4.mmd +++ b/docs/diagrams/architecture/remote-http-client-4.mmd @@ -1,14 +1,4 @@ -sequenceDiagram - participant Caller - participant Client - participant Resolver - participant Server - - Caller->>Client: get(Request) - Client->>Resolver: async_resolve() - Resolver-->>Client: endpoints - Client->>Server: async_connect() - Client->>Server: async_write(request) - Server-->>Client: HTTP response - Client->>Server: async_read() - Client-->>Caller: Response +flowchart TD + Response["HTTP Response"] --> Status["Status Code"] + Response --> Body["Response Body"] + Response --> HeaderAccess["Header Iterator"] diff --git a/docs/diagrams/architecture/remote-http-client-5.mmd b/docs/diagrams/architecture/remote-http-client-5.mmd index 1f2bd2fd1e2..72a3062d05b 100644 --- a/docs/diagrams/architecture/remote-http-client-5.mmd +++ b/docs/diagrams/architecture/remote-http-client-5.mmd @@ -1,6 +1,11 @@ flowchart TD - Connect["TCP Connect"] --> TLSCheck{"SSL Enabled?"} - TLSCheck -->|"Yes"| Handshake["TLS Handshake"] - TLSCheck -->|"No"| Plain["Plain HTTP"] - Handshake --> Send["Send Request"] - Plain --> Send + Start["Start Request"] --> Init["Initialize HTTP Request"] + Init --> Resolve["DNS Resolve"] + Resolve --> Connect["TCP Connect"] + Connect --> TLSCheck{{"TLS Enabled?"}} + TLSCheck -->|"Yes"| Handshake["SSL Handshake"] + TLSCheck -->|"No"| Write + Handshake --> Write["Async Write"] + Write --> Read["Async Read"] + Read --> Complete["Post Response Handler"] + Complete --> EndNode["END"] diff --git a/docs/diagrams/architecture/remote-http-client-6.mmd b/docs/diagrams/architecture/remote-http-client-6.mmd deleted file mode 100644 index 01586684843..00000000000 --- a/docs/diagrams/architecture/remote-http-client-6.mmd +++ /dev/null @@ -1,9 +0,0 @@ -flowchart TD - Start["Start Network Call"] --> Timer["Start Deadline Timer"] - Timer --> Async["Async Operation"] - Async --> Complete{"Completed?"} - Complete -->|"Yes"| CancelTimer["Cancel Timer"] - Complete -->|"No"| Timeout["Timer Fires"] - Timeout --> Abort["Abort Socket"] - CancelTimer --> Return["Return Response"] - Abort --> Return diff --git a/docs/diagrams/architecture/remote-http-client.mmd b/docs/diagrams/architecture/remote-http-client.mmd index f6bb4f64dff..0cd12bdd75f 100644 --- a/docs/diagrams/architecture/remote-http-client.mmd +++ b/docs/diagrams/architecture/remote-http-client.mmd @@ -1,13 +1,8 @@ flowchart TD - Caller["Caller Module"] --> Client["Client"] - Client --> Options["Client::Options"] - Client --> Request["HTTP_Request"] - Client --> Response["HTTP_Response"] + ConfigPlugins["Config Plugins"] -->|"Fetch config"| RemoteHttpClient["Remote Http Client"] + DistributedTLS["Distributed TLS Plugin"] -->|"Send queries"| RemoteHttpClient + DistributedQuerying["Distributed Querying"] -->|"Submit results"| RemoteHttpClient - Client --> Resolver["TCP Resolver"] - Client --> Socket["TCP Socket"] - Client --> TLSSocket["SSL Stream"] - Client --> Timer["Deadline Timer"] - - Request --> URI["URI Parser"] - Response --> Headers["Header Iterator"] + RemoteHttpClient -->|"TCP"| Asio["Boost Asio"] + RemoteHttpClient -->|"HTTP"| Beast["Boost Beast"] + RemoteHttpClient -->|"TLS"| OpenSSL["OpenSSL"] diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables-2.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables-2.mmd index 09946020084..b4acebdb726 100644 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables-2.mmd +++ b/docs/diagrams/architecture/sql-engine-and-virtual-tables-2.mmd @@ -1,5 +1,9 @@ -flowchart LR - Caller["Caller"] --> GetConn["SQLiteDBManager.get()"] - GetConn --> PrimaryCheck{"Primary Available?"} - PrimaryCheck -->|Yes| Primary["Primary SQLiteDBInstance"] - PrimaryCheck -->|No| Transient["Transient SQLiteDBInstance"] +flowchart TD + Start["Query Request"] --> GetDB["SQLiteDBManager.get()"] + GetDB --> UseCache["Set Cache Mode"] + UseCache --> Prepare["sqlite3_prepare_v2"] + Prepare --> Step["sqlite3_step"] + Step --> Rows["readRows()"] + Rows --> Finalize["sqlite3_finalize"] + Finalize --> Clear["Clear Affected Tables"] + Clear --> End["Return Status + Results"] diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables-3.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables-3.mmd index 6bded7bfd12..ff4c1bd8570 100644 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables-3.mmd +++ b/docs/diagrams/architecture/sql-engine-and-virtual-tables-3.mmd @@ -1,4 +1,7 @@ flowchart TD - Prepare["sqlite3_prepare_v2"] --> Authorizer["sqliteAuthorizer"] - Authorizer -->|Allowed| Continue["Execute Statement"] - Authorizer -->|Denied| Reject["SQLITE_DENY"] + Request["get() or getConnection()"] --> CheckPrimary{"Primary DB Exists?"} + CheckPrimary -->|"No"| Init["openOptimized()"] + CheckPrimary -->|"Yes"| Reuse["Reuse Primary"] + Init --> Attach["attachVirtualTables()"] + Reuse --> ReturnPrimary["Return Primary or Transient"] + Attach --> ReturnPrimary diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables-4.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables-4.mmd index d79fe38ce6a..fc480c5a593 100644 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables-4.mmd +++ b/docs/diagrams/architecture/sql-engine-and-virtual-tables-4.mmd @@ -1,5 +1,9 @@ flowchart TD - Query["SQL Query"] --> ExplainPlan["EXPLAIN QUERY PLAN"] - Query --> Explain["EXPLAIN"] - Explain --> OpcodeMap["kSQLOpcodes Mapping"] - OpcodeMap --> TypeInference["Apply Column Types"] + Create["xCreate or xConnect"] --> Declare["sqlite3_declare_vtab"] + Declare --> BestIndex["xBestIndex"] + BestIndex --> Filter["xFilter"] + Filter --> Next["xNext"] + Next --> Column["xColumn"] + Column --> Rowid["xRowid"] + Rowid --> Eof["xEof"] + Eof --> Close["xClose"] diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables-5.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables-5.mmd index 5a569905d36..270bc393296 100644 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables-5.mmd +++ b/docs/diagrams/architecture/sql-engine-and-virtual-tables-5.mmd @@ -1,12 +1,6 @@ -sequenceDiagram - participant SQLite - participant Module as "sqlite3_module" - participant Table as "TablePlugin" - - SQLite->>Module: xBestIndex - Module->>SQLite: Constraint plan - - SQLite->>Module: xFilter - Module->>Table: generate(context) - Table-->>Module: Row set - Module-->>SQLite: Rows +flowchart TD + Query["User Query"] --> ExplainPlan["EXPLAIN QUERY PLAN"] + ExplainPlan --> Explain["EXPLAIN"] + Explain --> Parse["Parse Opcode Rows"] + Parse --> ApplyTypes["applyTypes()"] + ApplyTypes --> Columns["Typed Column Metadata"] diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables-6.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables-6.mmd deleted file mode 100644 index ebfb533e38c..00000000000 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables-6.mmd +++ /dev/null @@ -1,10 +0,0 @@ -flowchart TD - Start["Query Submitted"] --> Prepare["sqlite3_prepare_v2"] - Prepare --> Execute["sqlite3_step Loop"] - Execute -->|Virtual Table Access| VTab - VTab --> xBestIndex - xBestIndex --> xFilter - xFilter --> Generate["TablePlugin.generate()"] - Generate --> Rows["Return Rows"] - Rows --> Finalize["sqlite3_finalize"] - Finalize --> End["Results Returned"] diff --git a/docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd b/docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd index f435ddde934..9a037ea9a38 100644 --- a/docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd +++ b/docs/diagrams/architecture/sql-engine-and-virtual-tables.mmd @@ -1,11 +1,8 @@ flowchart TD - Client["SQL Caller (Scheduler / CLI / Extension)"] --> SQLPlugin["SQLiteSQLPlugin"] + Client["SQL Caller\nScheduler, Shell, API"] --> SQLPlugin["SQLiteSQLPlugin"] SQLPlugin --> DBManager["SQLiteDBManager"] DBManager --> DBInstance["SQLiteDBInstance"] - DBInstance --> SQLiteCore[("In-Memory SQLite Engine")] - - SQLiteCore --> VTabModule["sqlite3_module (Virtual Table Module)"] - VTabModule --> VirtualTable["VirtualTable Wrapper"] - VirtualTable --> TablePlugin["TablePlugin (Registry)"] - - SQLiteCore --> Planner["QueryPlanner"] + DBInstance --> SQLiteCore[("SQLite Engine")] + SQLiteCore --> VTableModule["sqlite3_module\nVirtual Table Layer"] + VTableModule --> TablePlugin["TablePlugin\nRegistry"] + TablePlugin --> DataSource["System Data Sources\nFilesystem, Processes, Network"] diff --git a/docs/getting-started/first-steps.md b/docs/getting-started/first-steps.md index 3b002d66b32..c76f6b8bf25 100644 --- a/docs/getting-started/first-steps.md +++ b/docs/getting-started/first-steps.md @@ -1,178 +1,244 @@ # First Steps -Now that osquery is running, here are the first things to explore and configure. +Welcome to osquery! This guide walks you through the first five things to do after a successful installation. --- ## 1. Explore Available Tables -osquery ships with hundreds of virtual tables. Discover them with the `.tables` command in the interactive shell: +osquery exposes 300+ virtual tables across Linux, macOS, and Windows. Start by discovering what's available in the interactive shell: ```bash -./build/osquery/osqueryi +./osquery/osqueryi ``` +List all available tables: + +```sql +.tables +``` + +Search for tables matching a keyword: + +```sql +.tables process +``` + +Get the schema of a specific table: + +```sql +.schema processes +``` + +Expected output: + +```text +CREATE VIRTUAL TABLE processes USING osquery( + pid BIGINT, + name TEXT, + path TEXT, + cmdline TEXT, + state TEXT, + cwd TEXT, + root TEXT, + uid BIGINT, + ... +); +``` + +--- + +## 2. Run Your First Security Queries + +Try these essential security-focused queries: + +**Find processes running from unusual locations:** + ```sql --- List all available tables -osquery> .tables +SELECT pid, name, path +FROM processes +WHERE path NOT LIKE '/usr/%' + AND path NOT LIKE '/bin/%' + AND path NOT LIKE '/sbin/%' + AND on_disk = 0; +``` --- Search for tables by name -osquery> .tables process -osquery> .tables network -osquery> .tables user +**List all listening network ports:** --- Inspect the schema of any table -osquery> .schema processes -osquery> .schema listening_ports -osquery> .schema users +```sql +SELECT pid, port, protocol, address +FROM listening_ports +WHERE address != '127.0.0.1'; ``` -Tables are organized by platform: +**Find recently modified files in system directories:** -| Category | Example Tables | -|---|---| -| **System** | `system_info`, `uptime`, `cpu_info`, `memory_info` | -| **Processes** | `processes`, `process_open_files`, `process_open_sockets` | -| **Networking** | `listening_ports`, `interface_addresses`, `arp_cache`, `routes` | -| **Users & Groups** | `users`, `groups`, `logged_in_users`, `sudoers` | -| **Packages** | `deb_packages`, `rpm_packages`, `homebrew_packages`, `chocolatey_packages` | -| **Events** | `file_events`, `process_events`, `socket_events` | -| **Security** | `authorized_keys`, `certificates`, `secureboot`, `disk_encryption` | +```sql +SELECT path, mtime, size +FROM file +WHERE directory = '/etc' + AND mtime > (SELECT unix_time - 3600 FROM time); +``` + +**Check logged-in users:** + +```sql +SELECT username, tty, host, time +FROM logged_in_users; +``` --- -## 2. Write Your First Scheduled Query Pack +## 3. Create Your First Query Pack -Create a configuration pack to run queries on a schedule: +Query packs allow you to schedule queries to run at regular intervals. Create a configuration file: ```bash -sudo tee /etc/osquery/packs/first-steps.json <<'EOF' +mkdir -p /etc/osquery +``` + +```bash +cat > /etc/osquery/osquery.conf << 'EOF' { - "queries": { - "users_snapshot": { - "query": "SELECT uid, username, shell, directory FROM users;", - "interval": 3600, - "description": "Snapshot of all local users every hour" + "options": { + "config_plugin": "filesystem", + "logger_plugin": "filesystem", + "logger_path": "/var/log/osquery", + "schedule_splay_percent": 10 + }, + "schedule": { + "running_processes": { + "query": "SELECT pid, name, path, cmdline FROM processes;", + "interval": 60, + "description": "Track all running processes every minute" }, - "listening_ports": { - "query": "SELECT pid, port, protocol, address FROM listening_ports;", - "interval": 300, - "description": "Check open ports every 5 minutes" + "network_connections": { + "query": "SELECT pid, local_address, local_port, remote_address, remote_port FROM process_open_sockets WHERE state = 'ESTABLISHED';", + "interval": 30, + "description": "Track established network connections every 30 seconds" }, - "new_processes": { - "query": "SELECT pid, name, path, cmdline, start_time FROM processes;", - "interval": 60, - "description": "Poll running processes every minute" + "user_activity": { + "query": "SELECT username, tty, host, time FROM logged_in_users;", + "interval": 300, + "description": "Track user logins every 5 minutes" } } } EOF ``` -Reference it in your `osquery.conf`: +Start the daemon with this configuration: -```json -{ - "options": { - "logger_plugin": "filesystem", - "schedule_splay_percent": 10 - }, - "packs": { - "first-steps": "/etc/osquery/packs/first-steps.json" - } -} +```bash +./osquery/osqueryd \ + --config_path=/etc/osquery/osquery.conf \ + --pidfile=/var/run/osqueryd.pid \ + --daemonize=true ``` --- -## 3. Enable File Integrity Monitoring +## 4. Understand Event-Based Tables -osquery can monitor filesystem changes using platform-native event APIs (inotify on Linux, FSEvents on macOS, ETW on Windows): +Some tables capture events in real time rather than querying static state. To enable file integrity monitoring (FIM): -```json +Add file paths to watch in your configuration: + +```bash +cat > /etc/osquery/osquery.conf << 'EOF' { - "file_paths": { - "etc": ["/etc/%%"], - "homes": ["/root/.ssh/%%", "/home/%/.ssh/%%"], - "sensitive": ["/usr/bin/%%", "/usr/sbin/%%"] + "options": { + "config_plugin": "filesystem", + "logger_plugin": "filesystem", + "logger_path": "/var/log/osquery", + "disable_events": false }, "schedule": { "file_events": { - "query": "SELECT * FROM file_events;", + "query": "SELECT target_path, action, time FROM file_events;", "interval": 30 } + }, + "file_paths": { + "etc": [ + "/etc/%%" + ], + "home": [ + "/home/%/.ssh/%%" + ] } } +EOF ``` -Then query the event table: +Query file events interactively: ```sql -osquery> SELECT time, target_path, action, md5 FROM file_events; +SELECT target_path, action, time +FROM file_events +ORDER BY time DESC +LIMIT 20; ``` --- -## 4. Connect to OpenFrame (Flamingo Platform) - -If you are part of the [OpenFrame](https://openframe.ai) / [Flamingo](https://flamingo.run) platform: +## 5. Configure OpenFrame Integration -1. Obtain your OpenFrame credentials from your platform administrator -2. Configure the TLS enrollment endpoint in your daemon flags -3. The `OpenframeAuthorizationManager` handles token lifecycle automatically -4. The `OpenframeTokenRefresher` keeps sessions active in the background +To connect this osquery instance with the OpenFrame/Flamingo platform: -The OpenFrame layer uses AES-256-GCM encryption via the `OpenframeEncryptionService` β€” all credential handling is fully automated once configured. +### Obtain Your Token File -Refer to your environment configuration for the specific connection details. +After authenticating with the OpenFrame platform, you'll receive an encrypted token file. Store it securely: ---- +```bash +sudo mkdir -p /etc/openframe +sudo chmod 700 /etc/openframe +# Copy your encrypted token file here +sudo cp openframe-token.enc /etc/openframe/token.enc +sudo chmod 600 /etc/openframe/token.enc +``` -## 5. Load an Extension +### Configure the Token Path -osquery's extension system lets you add custom tables, loggers, and configuration plugins as separate processes: +Set the token file path in your osquery configuration: ```bash -# Start osqueryd with extension support enabled -sudo ./build/osquery/osqueryd \ - --config_path=/etc/osquery/osquery.conf \ - --extensions_socket=/var/osquery/osquery.em \ - --extensions_autoload=/etc/osquery/extensions.load - -# In a separate terminal, run your extension -./my_extension --socket=/var/osquery/osquery.em +cat > /etc/osquery/osquery.conf << 'EOF' +{ + "options": { + "config_plugin": "tls", + "logger_plugin": "tls", + "tls_hostname": "your-openframe-endpoint.openframe.ai", + "enroll_secret_path": "/etc/openframe/token.enc" + } +} +EOF ``` -Extensions communicate via Apache Thrift over UNIX domain sockets (Linux/macOS) or named pipes (Windows). +Refer to your OpenFrame environment configuration for the correct hostname and secret key values. --- -## Key Configuration Flags +## Key Configuration Options Reference -| Flag | Description | Default | -|---|---|---| -| `--config_path` | Path to the JSON configuration file | Platform-specific | -| `--logger_plugin` | Logging backend (`filesystem`, `tls`, `syslog`) | `filesystem` | -| `--schedule_splay_percent` | Random splay to distribute query load | `10` | -| `--extensions_socket` | Path to the Thrift extension socket | Platform-specific | -| `--config_refresh` | How often (seconds) to refresh remote config | `0` (disabled) | -| `--watchdog_level` | Resource limit profile (0=normal, 1=restrictive, -1=disabled) | `0` | +| Option | Description | Example | +|--------|-------------|---------| +| `config_plugin` | Configuration source (`filesystem`, `tls`) | `"filesystem"` | +| `logger_plugin` | Log destination (`filesystem`, `tls`, `syslog`) | `"filesystem"` | +| `logger_path` | Directory for filesystem logs | `"/var/log/osquery"` | +| `schedule_splay_percent` | Random schedule jitter (%) | `10` | +| `disable_events` | Disable event-based tables | `false` | +| `config_refresh` | Config refresh interval (seconds) | `60` | --- ## Where to Get Help | Resource | Link | -|---|---| -| **OpenMSP Community Slack** | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | -| **OpenMSP Website** | [https://www.openmsp.ai/](https://www.openmsp.ai/) | -| **Flamingo Platform** | [https://flamingo.run](https://flamingo.run) | -| **OpenFrame** | [https://openframe.ai](https://openframe.ai) | - -> All support and discussions are managed on the **OpenMSP Slack community** β€” not GitHub Issues or GitHub Discussions. - ---- - -## Watch: osquery in Action +|----------|------| +| **Community Slack** | [OpenMSP](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | +| **Flamingo Platform** | [flamingo.run](https://flamingo.run) | +| **OpenFrame Platform** | [openframe.ai](https://openframe.ai) | +| **OpenMSP Community** | [openmsp.ai](https://www.openmsp.ai/) | -[![osquery Overview](https://img.youtube.com/vi/1UcWGiHbLVo/hqdefault.jpg)](https://www.youtube.com/watch?v=1UcWGiHbLVo) +> **Tip:** The OpenMSP Slack community is the primary support channel. GitHub Issues and Discussions are not used for this project. diff --git a/docs/getting-started/introduction.md b/docs/getting-started/introduction.md index a15e0308dbb..e0a87137857 100644 --- a/docs/getting-started/introduction.md +++ b/docs/getting-started/introduction.md @@ -1,113 +1,169 @@ -# Introduction to osquery with OpenFrame +# Introduction to osquery -**osquery** is a cross-platform operating system instrumentation framework that exposes live system state and activity as relational data β€” enabling you to query your operating system using SQL. +> **osquery** is a cross-platform operating system instrumentation framework that exposes system state as relational data using SQL. Part of the [Flamingo](https://flamingo.run) / [OpenFrame](https://openframe.ai) platform, this distribution extends the upstream osquery project with OpenFrame-specific authentication, token management, and encryption services. -Built and extended by the [Flamingo / OpenFrame](https://openframe.ai) platform, this distribution integrates osquery's powerful SQL telemetry engine with OpenFrame's AI-driven MSP automation infrastructure. +[![osquery Introduction](https://img.youtube.com/vi/bRd3JCJZ1vc/hqdefault.jpg)](https://www.youtube.com/watch?v=bRd3JCJZ1vc) --- -## What is osquery? +## What Is osquery? -osquery transforms every host into a **SQL-queryable telemetry node**. Instead of parsing flat log files or calling proprietary APIs, you write standard SQL queries to explore: - -- Running processes and open sockets -- Installed packages and browser extensions -- Kernel modules and hardware inventory -- Filesystem events and user activity -- Network interfaces and firewall rules -- Windows Registry, launchd, systemd units, and much more +osquery lets operators, security teams, and infrastructure engineers **query the operating system as if it were a database**. Instead of parsing log files or writing custom scripts, you write standard SQL: ```sql --- Which processes are listening on ports? -SELECT pid, name, port, protocol FROM listening_ports JOIN processes USING (pid); +SELECT name, pid, path FROM processes WHERE on_disk = 0; +``` --- What Chrome extensions are installed? -SELECT u.username, e.name, e.version, e.permissions -FROM users u, chrome_extensions e -WHERE e.uid = u.uid; +```sql +SELECT address, mac FROM interface_addresses; ``` ---- +```sql +SELECT * FROM users WHERE uid = 0; +``` -## OpenFrame Integration +Under the hood, osquery embeds **SQLite** and implements a virtual table layer that maps OS primitives β€” processes, files, users, network sockets, kernel state, and more β€” into relational tables queryable in real time. -This repository extends osquery with the **OpenFrame** authentication and encryption layer: +--- -- **`OpenframeAuthorizationManager`** β€” Secure, lifecycle-controlled JWT token management -- **`OpenframeEncryptionService`** β€” AES-256-GCM symmetric encryption via OpenSSL -- **`OpenframeTokenExtractor`** β€” Token acquisition from OpenFrame services -- **`OpenframeTokenRefresher`** β€” Background thread for seamless token renewal +## Key Features -These components are part of the [OpenFrame platform](https://www.flamingo.run/openframe) β€” the unified MSP interface that integrates IT tools under a single AI-driven experience. +| Feature | Description | +|---------|-------------| +| **SQL-based OS querying** | Interact with 300+ virtual tables covering every aspect of the OS | +| **Cross-platform** | Runs on Linux, macOS, and Windows with a unified interface | +| **Scheduled queries** | Define query packs that run on a configurable schedule | +| **Event-driven tables** | Subscribe to OS events (file changes, process launches, network connections) | +| **Distributed querying** | Push queries to an entire fleet from a central control plane | +| **Plugin extensions** | Add custom tables, loggers, and config sources without modifying the binary | +| **OpenFrame integration** | Secure token-based authentication with the OpenFrame/Flamingo platform | +| **AES-256-GCM encryption** | Built-in encrypted token storage and retrieval | --- -## Key Features +## OpenFrame Extensions -| Feature | Description | -|---|---| -| **SQL Querying** | Query live OS data with standard SQL via embedded SQLite | -| **Virtual Tables** | 300+ platform-specific tables across Linux, macOS, and Windows | -| **Event Monitoring** | inotify, BPF, FSEvents, ETW, audit β€” all exposed as queryable tables | -| **Scheduled Queries** | Pack-based query scheduling with differential result tracking | -| **Distributed Queries** | Remote SQL orchestration across fleets via TLS | -| **Extension System** | Runtime plugin model with Apache Thrift IPC | -| **OpenFrame Auth** | AES-256-GCM encrypted JWT token management for OpenFrame services | -| **Cross-Platform** | Linux (x86_64, aarch64), macOS, and Windows support | +This distribution adds the following components on top of upstream osquery: + +- **`OpenframeAuthorizationManager`** β€” Singleton manager for securely storing and retrieving the OpenFrame bearer token used to authenticate with OpenFrame services. +- **`OpenframeEncryptionService`** β€” AES-256-GCM encryption/decryption service that secures tokens at rest. +- **`OpenframeTokenExtractor`** β€” Reads and decrypts authentication tokens from an encrypted token file on disk. +- **`OpenframeTokenRefresher`** β€” Background thread that periodically re-extracts and refreshes the authentication token. --- ## Target Audience -osquery with OpenFrame is designed for: +osquery is designed for: -- **IT Operations Teams** using Flamingo/OpenFrame for managed service delivery -- **Security Engineers** building detection and response workflows -- **Fleet Administrators** who need SQL-level visibility across endpoints -- **Developers** building extensions, plugins, or custom table providers -- **MSP Technicians** leveraging Mingo AI and Fae automation through the OpenFrame platform +- **Security engineers** building endpoint detection and response (EDR) capabilities +- **Infrastructure operators** requiring real-time system observability +- **MSP technicians** using the Flamingo/OpenFrame platform for managed IT operations +- **Developers** extending osquery with custom tables and plugins --- -## Architecture Overview +## System Architecture Overview ```mermaid -graph TD - CLI["osqueryi / osqueryd"] --> Core["Core Init And Runtime"] - Core --> Config["Configuration And Packs"] +flowchart TD + Core["Core Runtime And Lifecycle"] --> Config["Configuration And Packs"] Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework"] - Core --> Logging["Logging And Observability"] - Core --> DB["Database And Storage Plugins"] - Core --> Dist["Distributed Querying"] - Core --> Ext["Extensions And IPC"] - Core --> HTTP["Remote HTTP Client"] - Core --> OF["OpenFrame Auth Layer"] + Core --> DB["Database Backend"] + Core --> Logging["Logging And Query Metadata"] + Core --> Events["Events Core And Subscriptions"] + Core --> Extensions["Extensions Framework"] + Core --> Distributed["Distributed Querying"] + Config --> SQL Config --> Events + Config --> Logging + SQL --> Logging + SQL --> DB + Events --> DB - Dist --> SQL - Dist --> HTTP - Ext --> SQL - OF --> HTTP + Events --> Logging + + Distributed --> SQL + Distributed --> DB + Distributed --> Logging + + Extensions --> SQL + Extensions --> Config + Extensions --> Distributed + + OpenFrame["OpenFrame Auth Layer"] --> Core +``` + +### Architectural Layers + +| Layer | Responsibility | +|-------|---------------| +| **Core Runtime** | Bootstrapping, flags, lifecycle, watchdog, shutdown | +| **SQL Engine** | Query parsing, execution, virtual tables (300+ tables) | +| **Configuration** | Packs, scheduling, dynamic updates | +| **Database Backend** | Persistent RocksDB and ephemeral key–value storage | +| **Logging** | Structured result serialization and emission | +| **Events** | Real-time event ingestion and exposure as tables | +| **Distributed** | Remote query retrieval and result submission | +| **Extensions** | External plugin injection via Thrift IPC | +| **OpenFrame Auth** | Token lifecycle, AES-256-GCM encryption, background refresh | + +--- + +## Query Execution Flow + +```mermaid +sequenceDiagram + participant Scheduler as "Scheduler / Shell" + participant SQL as "SQL Engine" + participant VTable as "Virtual Tables" + participant DB as "Database Backend" + participant Logger as "Logger Plugin" + + Scheduler->>SQL: Submit SQL query + SQL->>VTable: Execute against virtual tables + VTable-->>SQL: Return rows + SQL->>DB: Fetch previous results + DB-->>SQL: Previous result set + SQL->>SQL: Compute diff (added/removed) + SQL->>Logger: Emit QueryLogItem + Logger-->>Scheduler: Log confirmation +``` + +--- + +## Two Execution Modes + +**`osqueryi`** β€” Interactive shell for ad-hoc queries: + +```bash +osqueryi +osquery> SELECT name, version FROM os_version; +``` + +**`osqueryd`** β€” Daemon for continuous scheduled execution: + +```bash +osqueryd --config_path=/etc/osquery/osquery.conf ``` --- ## Getting Started -- Review system and software prerequisites in the [Prerequisites Guide](prerequisites.md) -- Follow the [Quick Start Guide](quick-start.md) to build and run osquery in minutes -- Explore common workflows in [First Steps](first-steps.md) +- Review the [Prerequisites](prerequisites.md) guide to ensure your environment is ready. +- Follow the [Quick Start](quick-start.md) to get osquery running in minutes. +- Complete [First Steps](first-steps.md) to explore key features and initial configuration. --- -## Community and Support +## Community & Support -osquery with OpenFrame is part of the [Flamingo](https://flamingo.run) open-source ecosystem. +Join the open MSP community for questions, discussions, and contributions: -- πŸ’¬ **Community Slack**: [OpenMSP Community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) -- 🌐 **OpenMSP**: [https://www.openmsp.ai/](https://www.openmsp.ai/) -- πŸš€ **OpenFrame Platform**: [https://openframe.ai](https://openframe.ai) -- 🦩 **Flamingo**: [https://flamingo.run](https://flamingo.run) +- **Slack**: [OpenMSP Community](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) +- **Platform**: [https://www.openmsp.ai/](https://www.openmsp.ai/) +- **Flamingo**: [https://flamingo.run](https://flamingo.run) +- **OpenFrame**: [https://openframe.ai](https://openframe.ai) diff --git a/docs/getting-started/prerequisites.md b/docs/getting-started/prerequisites.md index 8e6027a2725..f18a058dde1 100644 --- a/docs/getting-started/prerequisites.md +++ b/docs/getting-started/prerequisites.md @@ -1,167 +1,160 @@ # Prerequisites -Before building or deploying osquery with OpenFrame, ensure your environment meets the requirements below. +Before building or running this OpenFrame-enhanced distribution of osquery, ensure your system and environment meet the following requirements. --- ## System Requirements -| Tier | RAM | CPU Cores | Disk Space | -|---|---|---|---| -| **Minimum** | 24 GB | 6 cores | 50 GB | -| **Recommended** | 32 GB | 12 cores | 100 GB | +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| **RAM** | 24 GB | 32 GB | +| **CPU Cores** | 6 cores | 12 cores | +| **Disk Space** | 50 GB | 100 GB | +| **OS** | Linux, macOS, or Windows | Linux (Ubuntu 20.04+) | -> Building osquery from source is resource-intensive. The recommended configuration significantly reduces build times and prevents out-of-memory failures during compilation. +> **Note:** The build system compiles osquery and all vendored libraries from source. Insufficient RAM will cause build failures or extremely slow compilation times. --- ## Supported Operating Systems -| Platform | Architecture | Notes | -|---|---|---| -| Linux | x86_64, aarch64 | Ubuntu 20.04+, RHEL 8+, Debian 11+ | -| macOS | x86_64, aarch64 (Apple Silicon) | macOS 12+ recommended | -| Windows | x86_64, aarch64 | Windows 10/11, Server 2019+ | +| OS | Architectures | Notes | +|----|--------------|-------| +| **Linux** | x86_64, aarch64 | Ubuntu 20.04+, CentOS 8+, Fedora 34+ | +| **macOS** | x86_64, aarch64 | macOS 12 (Monterey) or newer | +| **Windows** | x86_64, aarch64 | Windows 10 / Server 2019 or newer | --- -## Required Software +## Required Build Tools -| Tool | Minimum Version | Purpose | -|---|---|---| -| **CMake** | 3.21+ | Build system generator | -| **Python** | 3.8+ | Code generation, tooling scripts | -| **Git** | 2.x | Source control | -| **C++ Compiler** | GCC 9+ / Clang 10+ / MSVC 2019+ | Compiling C++17 sources | -| **Ninja** | 1.10+ | Fast parallel builds (recommended) | -| **OpenSSL** | 1.1.1+ | TLS support and AES-256-GCM encryption (OpenFrame) | +### All Platforms ---- +| Tool | Version | Purpose | +|------|---------|---------| +| **CMake** | β‰₯ 3.21 | Build system generator | +| **Python 3** | β‰₯ 3.6 | Code generation scripts | +| **Git** | β‰₯ 2.x | Source control | +| **Clang / GCC** | Clang β‰₯ 13 or GCC β‰₯ 10 | C++ compiler (C++17 required) | -## Optional but Recommended +### Linux -| Tool | Purpose | -|---|---| -| **ccache** | Speeds up incremental C++ builds significantly | -| **clang-format** | Code formatting (enforced by CI) | -| **Docker** | Isolated build environments for Linux targets | +| Tool | Version | Purpose | +|------|---------|---------| +| **Clang** | β‰₯ 13 | Preferred compiler | +| **lld** | β‰₯ 13 | LLVM linker (faster builds) | +| **build-essential** | β€” | Base build tools | +| **libssl-dev** | β‰₯ 1.1.1 | OpenSSL (required for OpenFrame encryption) | ---- +### macOS -## Platform-Specific Notes +| Tool | Version | Purpose | +|------|---------|---------| +| **Xcode Command Line Tools** | Latest | Compiler toolchain | +| **Homebrew** | Latest | Package manager | -### Linux +### Windows -On Debian/Ubuntu, install build essentials: +| Tool | Version | Purpose | +|------|---------|---------| +| **Visual Studio** | 2019 or 2022 | MSVC compiler | +| **Windows SDK** | β‰₯ 10.0.18362 | Platform headers | -```bash -sudo apt-get update -sudo apt-get install -y \ - build-essential \ - cmake \ - ninja-build \ - python3 \ - python3-pip \ - git \ - openssl \ - libssl-dev \ - clang-format -``` +--- -On RHEL/CentOS/Fedora: +## OpenSSL Requirement -```bash -sudo dnf install -y \ - gcc-c++ \ - cmake \ - ninja-build \ - python3 \ - git \ - openssl-devel \ - clang -``` +The OpenFrame encryption service (`OpenframeEncryptionService`) requires **OpenSSL** with AES-GCM support. This is used for: -### macOS +- AES-256-GCM encryption and decryption of authentication tokens +- Base64 encoding/decoding of encrypted payloads -Install Xcode Command Line Tools and Homebrew tools: +Verify OpenSSL is available: ```bash -xcode-select --install -brew install cmake ninja python3 openssl git +openssl version ``` -### Windows - -1. Install [Visual Studio 2019 or 2022](https://visualstudio.microsoft.com/) with the **Desktop development with C++** workload -2. Install [CMake](https://cmake.org/download/) 3.21+ -3. Install [Git for Windows](https://git-scm.com/download/win) -4. Install [Python 3.8+](https://www.python.org/downloads/windows/) -5. Optionally install [Ninja](https://github.com/ninja-build/ninja/releases) - ---- - -## OpenFrame Platform Requirements - -If you are connecting to the [OpenFrame platform](https://openframe.ai), you also need: +Expected output example: -| Requirement | Description | -|---|---| -| **OpenFrame Account** | Active account on the Flamingo/OpenFrame platform | -| **Network Access** | Outbound HTTPS (port 443) to OpenFrame services | -| **AES-256 Key** | Symmetric secret key for the `OpenframeEncryptionService` | -| **JWT Token** | Authorization token managed by `OpenframeAuthorizationManager` | - -Refer to your environment configuration and OpenFrame administrator for credential details. +```text +OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022) +``` --- -## Environment Variables +## Node.js / JavaScript Tools (Documentation Pipeline Only) -The following environment variables may be required depending on your deployment: +The `package.json` at the repository root lists tooling dependencies for the documentation generation pipeline β€” these are **not** required to build or run osquery itself: -| Variable | Purpose | -|---|---| -| `OSQUERY_FLAGS_FILE` | Path to a flagfile for osquery runtime configuration | -| `OSQUERY_EXTENSIONS_SOCKET` | Path to the Thrift extension manager socket | -| `OSQUERY_CONFIG_PATH` | Custom configuration file path (overrides default) | +| Package | Purpose | +|---------|---------| +| `@voltagent/core` | Documentation agent (pipeline tool) | +| `@ai-sdk/anthropic` | AI SDK (pipeline tool) | +| `@anthropic-ai/sdk` | Anthropic SDK (pipeline tool) | +| `glob` | File globbing (pipeline tool) | +| `zod` | Schema validation (pipeline tool) | -Set variables using your shell profile or deployment tooling. For example: - -```bash -export OSQUERY_FLAGS_FILE="/etc/osquery/osquery.flags" -export OSQUERY_EXTENSIONS_SOCKET="/var/osquery/osquery.em" -``` +> **Important:** These packages support documentation generation tooling only. They are not runtime or build dependencies for osquery. --- -## Verifying Your Environment +## Verification Checklist -Run the following checks to confirm your build environment is ready: +Run these commands to verify your environment before starting: ```bash -# Check CMake version +# Check CMake version (need >= 3.21) cmake --version -# Check C++ compiler -c++ --version +# Check Clang version (need >= 13) +clang --version -# Check Python +# Check Python version (need >= 3.6) python3 --version # Check Git git --version -# Check OpenSSL +# Check OpenSSL (required for OpenFrame encryption) openssl version -# Check Ninja (if installed) -ninja --version +# Check disk space (need >= 50 GB free) +df -h . + +# Check available memory (need >= 24 GB) +free -h ``` -All tools should report versions meeting the minimums in the table above. +--- + +## Account and Access Requirements + +| Resource | Required For | Notes | +|----------|-------------|-------| +| **OpenFrame account** | OpenFrame platform integration | Sign up at [openframe.ai](https://openframe.ai) | +| **Flamingo account** | MSP platform features | Sign up at [flamingo.run](https://flamingo.run) | +| **OpenMSP Slack** | Community support | [Join here](https://join.slack.com/t/openmsp/shared_invite/zt-36bl7mx0h-3~U2nFH6nqHqoTPXMaHEHA) | + +--- + +## Environment Variables + +The following environment variables may be required depending on your build and deployment configuration: + +| Variable | Purpose | Required | +|----------|---------|----------| +| `OPENFRAME_TOKEN_PATH` | Path to the encrypted OpenFrame token file | For OpenFrame integration | +| `OPENFRAME_SECRET_KEY` | 256-bit secret key for AES-256-GCM token decryption | For OpenFrame integration | + +> **Security Note:** Never hardcode secret keys in shell scripts or configuration files. Use a secrets manager or secure environment injection. --- ## Next Steps -Once prerequisites are confirmed, proceed to the [Quick Start Guide](quick-start.md) to clone, build, and run osquery. +Once your environment is verified: + +- Follow the [Quick Start](quick-start.md) to build and run osquery in minutes. +- Complete [First Steps](first-steps.md) to configure your first query packs. diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md index 0ab657e8ce7..d67dea47055 100644 --- a/docs/getting-started/quick-start.md +++ b/docs/getting-started/quick-start.md @@ -1,170 +1,206 @@ # Quick Start -Get osquery with OpenFrame running in under 10 minutes. +Get osquery built and running in minutes with this step-by-step guide. --- -## TL;DR β€” Fastest Path +## TL;DR β€” 5-Minute Setup ```bash # 1. Clone the repository git clone https://github.com/flamingo-stack/osquery.git cd osquery -# 2. Configure the build -cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo -G Ninja +# 2. Create the build directory +mkdir build && cd build -# 3. Build osquery (this takes 5–30 minutes depending on hardware) -cmake --build build --parallel $(nproc) +# 3. Configure with CMake +cmake .. -# 4. Run the interactive shell -./build/osquery/osqueryi -``` +# 4. Build osquery +cmake --build . --parallel $(nproc) -> **Hardware note**: Building from source requires at minimum 24 GB RAM and 6 CPU cores. See the [Prerequisites Guide](prerequisites.md) for full system requirements. +# 5. Run the interactive shell +./osquery/osqueryi +``` --- -## Step 1: Clone the Repository +## Step-by-Step Instructions + +### 1. Clone the Repository ```bash git clone https://github.com/flamingo-stack/osquery.git cd osquery ``` ---- - -## Step 2: Configure the Build - -### Linux / macOS +### 2. Configure the Build ```bash -cmake -S . -B build \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -G Ninja +mkdir build +cd build +cmake .. ``` -### Windows (Visual Studio) +Common CMake options: + +| Option | Default | Description | +|--------|---------|-------------| +| `-DCMAKE_BUILD_TYPE=Release` | `RelWithDebInfo` | Build type (Release/Debug/RelWithDebInfo) | +| `-DCMAKE_C_COMPILER=clang` | System default | C compiler | +| `-DCMAKE_CXX_COMPILER=clang++` | System default | C++ compiler | +| `-DOSQUERY_DISABLE_DATABASE=ON` | OFF | Disable RocksDB (use ephemeral backend) | + +Example with explicit Clang: ```bash -cmake -S . -B build ^ - -DCMAKE_BUILD_TYPE=RelWithDebInfo ^ - -G "Visual Studio 17 2022" +cmake .. \ + -DCMAKE_C_COMPILER=clang \ + -DCMAKE_CXX_COMPILER=clang++ \ + -DCMAKE_BUILD_TYPE=Release ``` -### Common CMake Options +### 3. Build -| Option | Description | -|---|---| -| `-DCMAKE_BUILD_TYPE=RelWithDebInfo` | Optimized build with debug symbols (recommended) | -| `-DCMAKE_BUILD_TYPE=Debug` | Full debug build (slower, larger binaries) | -| `-DCMAKE_BUILD_TYPE=Release` | Fully optimized production build | -| `-G Ninja` | Use Ninja for faster parallel builds | -| `-DOSQUERY_BUILD_TESTS=ON` | Include test suite in the build | +```bash +# Use all available CPU cores for parallel compilation +cmake --build . --parallel $(nproc) +``` ---- +> **Tip:** On a 6-core system, this typically takes 20–40 minutes on first build. Subsequent incremental builds are much faster. -## Step 3: Build +### 4. Verify the Build ```bash -# Linux/macOS β€” use all available CPU cores -cmake --build build --parallel $(nproc) - -# Windows β€” use all available CPU cores -cmake --build build --parallel %NUMBER_OF_PROCESSORS% +# Check the binaries were produced +ls -la osquery/osqueryi osquery/osqueryd ``` -The first build downloads and compiles all third-party dependencies. Subsequent builds with `ccache` are significantly faster. +Expected output: + +```text +-rwxr-xr-x ... osquery/osqueryi +-rwxr-xr-x ... osquery/osqueryd +``` --- -## Step 4: Run Your First Query +## Your First Query -Launch the osquery interactive shell: +Launch the interactive shell: ```bash -./build/osquery/osqueryi +./osquery/osqueryi ``` -Expected output: +Try a simple query: + +```sql +SELECT name, version FROM os_version; +``` + +Expected output (example on Ubuntu): ```text -Using a virtual database. Need help, type '.help' -osquery> ++----------+---------+ +| name | version | ++----------+---------+ +| Ubuntu | 22.04 | ++----------+---------+ ``` -Now try your first query: +Try querying running processes: ```sql -osquery> SELECT hostname, cpu_brand, physical_memory FROM system_info; +SELECT pid, name, path FROM processes LIMIT 5; ``` -Example output: +Try querying network interfaces: -```text -+------------------+-------------------------------+------------------+ -| hostname | cpu_brand | physical_memory | -+------------------+-------------------------------+------------------+ -| my-linux-host | Intel(R) Core(TM) i9-12900K | 34207285248 | -+------------------+-------------------------------+------------------+ +```sql +SELECT interface, address, mask FROM interface_addresses; ``` ---- - -## Step 5: Explore More Tables +List all available tables: ```sql --- List all running processes -osquery> SELECT pid, name, cmdline FROM processes LIMIT 10; - --- Check listening network ports -osquery> SELECT pid, port, protocol, address FROM listening_ports LIMIT 10; +.tables +``` --- List installed packages (Linux) -osquery> SELECT name, version, arch FROM deb_packages LIMIT 10; +Exit the shell: --- Show users on the system -osquery> SELECT uid, gid, username, shell FROM users; +```sql +.exit ``` --- -## Step 6: Run as a Daemon +## Running osqueryd (Daemon Mode) + +The daemon mode runs scheduled queries continuously using a configuration file. -To run osquery as a background daemon with scheduled queries: +Create a minimal configuration file: ```bash -# Create a basic configuration -sudo mkdir -p /etc/osquery -sudo tee /etc/osquery/osquery.conf <<'EOF' +cat > /tmp/osquery.conf << 'EOF' { "options": { + "config_plugin": "filesystem", "logger_plugin": "filesystem", - "schedule_splay_percent": 10 + "logger_path": "/tmp/osquery_logs", + "disable_logging": false }, "schedule": { "system_info": { - "query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;", - "interval": 3600 + "query": "SELECT hostname, cpu_type, physical_memory FROM system_info;", + "interval": 60 } } } EOF +``` + +Run the daemon: -# Run the daemon -sudo ./build/osquery/osqueryd --config_path=/etc/osquery/osquery.conf +```bash +./osquery/osqueryd --config_path=/tmp/osquery.conf --pidfile=/tmp/osquery.pid ``` +Check logs: + +```bash +ls /tmp/osquery_logs/ +cat /tmp/osquery_logs/osqueryd.results.log +``` + +--- + +## Windows Installation + +On Windows (AMD64), you can download the pre-built CLI: + +1. Download: [openframe-cli_windows_amd64.zip](https://github.com/flamingo-stack/openframe-cli/releases/latest/download/openframe-cli_windows_amd64.zip) +2. Extract the archive +3. Run the installer following the same steps as other operating systems + --- -## Expected Results Summary +## Expected Build Output + +A successful build produces: + +```text +[100%] Built target osqueryi +[100%] Built target osqueryd +``` + +The following binaries are available: -| Command | What You Should See | -|---|---| -| `osqueryi` | Interactive SQL prompt | -| `SELECT * FROM system_info;` | Host identity, CPU, RAM details | -| `SELECT * FROM processes LIMIT 5;` | Running process list | -| `osqueryd` | Daemon starts, logs to `/var/log/osquery/` | +| Binary | Description | +|--------|-------------| +| `osquery/osqueryi` | Interactive SQL shell | +| `osquery/osqueryd` | Continuous monitoring daemon | --- @@ -172,6 +208,5 @@ sudo ./build/osquery/osqueryd --config_path=/etc/osquery/osquery.conf After completing this quick start: -- Follow the [First Steps Guide](first-steps.md) to explore key features and common workflows -- Review the [Prerequisites Guide](prerequisites.md) for full system and software requirements -- Explore the [Development section](../development/README.md) to understand the architecture and contribute +- Read [First Steps](first-steps.md) to explore key features and configuration options. +- Review [Prerequisites](prerequisites.md) if you encountered build issues. diff --git a/docs/reference/architecture/README.md b/docs/reference/architecture/README.md index d8bef8be03c..7b3540a2a03 100644 --- a/docs/reference/architecture/README.md +++ b/docs/reference/architecture/README.md @@ -1,366 +1,412 @@ # osquery Repository Overview -## 1. Purpose of the Repository +## Purpose -**osquery** is a cross-platform operating system instrumentation framework that exposes system state and activity as relational data. It embeds a hardened SQLite engine and represents operating system concepts (processes, files, sockets, users, registry keys, etc.) as **virtual tables**, enabling users to query live system data using SQL. +**osquery** is a cross-platform operating system instrumentation framework that exposes system state as relational data using **SQL**. It allows operators, security teams, and infrastructure engineers to: -The repository implements: +- Query system information (processes, files, users, network, kernel state) +- Schedule and log queries continuously +- Collect and persist system events +- Execute distributed queries from a central control plane +- Extend functionality dynamically through plugins and extensions -- A secure in-memory SQL engine -- A virtual table plugin framework -- Configuration-driven scheduled querying -- Differential result logging and observability -- Distributed query orchestration -- Event-driven monitoring infrastructure -- Persistent and ephemeral storage backends -- Remote HTTP/TLS communication -- Extension-based runtime plugin model - -At a high level, osquery transforms a host into a **SQL-queryable telemetry node** with local and distributed control capabilities. +At its core, osquery embeds SQLite and implements a virtual table layer that maps operating system primitives into relational tables. --- -# 2. End-to-End Architecture +# End-to-End Architecture -The system is modular and plugin-driven. Below is the full end-to-end architecture across core subsystems. +osquery is structured as a modular runtime composed of subsystems that cooperate through registries, plugins, and well-defined interfaces. -## 2.1 System-Level Architecture +## High-Level System Architecture ```mermaid flowchart TD - CLI["osqueryi / osqueryd"] --> Core["Core Init And Runtime"] - - Core --> Config["Configuration And Packs"] + Core["Core Runtime And Lifecycle"] --> Config["Configuration And Packs"] Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework"] - Core --> Logging["Logging And Observability"] - Core --> DB["Database And Storage Plugins"] + Core --> DB["Database Backend"] + Core --> Logging["Logging And Query Metadata"] + Core --> Events["Events Core And Subscriptions"] + Core --> Extensions["Extensions Framework"] Core --> Distributed["Distributed Querying"] - Core --> Extensions["Extensions And IPC"] - Core --> HTTP["Remote HTTP Client"] - Core --> FS["Filesystem And Path Utilities"] Config --> SQL Config --> Events + Config --> Logging + SQL --> Logging + SQL --> DB + Events --> DB + Events --> Logging + Distributed --> SQL - Distributed --> HTTP - Logging --> DB + Distributed --> DB + Distributed --> Logging + Extensions --> SQL - Extensions --> DB + Extensions --> Config + Extensions --> Distributed ``` ---- +### Architectural Layers -# 3. Runtime Execution Model +| Layer | Responsibility | +|-------|----------------| +| Core Runtime | Bootstrapping, flags, lifecycle, watchdog, shutdown | +| SQL Engine | Query parsing, execution, virtual tables | +| Configuration | Packs, scheduling, dynamic updates | +| Database Backend | Persistent and ephemeral key–value storage | +| Logging | Structured result serialization and emission | +| Events | Real-time event ingestion and exposure as tables | +| Distributed | Remote query retrieval and result submission | +| Extensions | External plugin injection via Thrift IPC | -osquery operates in several runtime modes: +--- -- **Daemon (`osqueryd`)** -- **Interactive shell (`osqueryi`)** -- **Watcher / Worker model** -- **Extension process** +# Query Execution Flow -## Runtime Lifecycle +This diagram shows how a scheduled or distributed query travels through the system. ```mermaid flowchart TD - Start["Process Start"] --> Init["Initializer"] - Init --> Flags["Parse Flags"] - Init --> Registry["Initialize Registry"] - Init --> DBInit["Initialize Database"] - Init --> ExtMgr["Start Extension Manager"] - Init --> ConfigLoad["Load Configuration"] - ConfigLoad --> Schedule["Build Schedule"] - Schedule --> QueryExec["Execute Queries"] - QueryExec --> Log["Log Results"] - Log --> End["Runtime Loop"] + Source["Scheduled Or Distributed Query"] --> Scheduler["Configuration And Packs"] + Scheduler --> SQLPlugin["SQLiteSQLPlugin"] + SQLPlugin --> SQLiteCore["SQLite Engine"] + SQLiteCore --> VTables["Virtual Tables"] + VTables --> TablePlugins["Table Plugins"] + + SQLiteCore --> QueryState["Query State And Diff"] + QueryState --> DB["Database Backend"] + QueryState --> Diff["DiffResults"] + + Diff --> LogItem["QueryLogItem"] + LogItem --> Logger["Logger Plugin"] + Logger --> External["External Logging System"] ``` -The **Core Init And Runtime** module governs this lifecycle and ensures safe startup, resource enforcement (watchdog), and graceful shutdown. +### Execution Steps -πŸ“˜ Reference: -`osquery/core` β†’ *Core Init And Runtime* +1. Query originates from: + - Scheduled configuration pack + - Distributed remote request + - Interactive shell +2. SQLite engine executes against virtual tables. +3. Previous results are fetched from the database. +4. Differential results are computed. +5. Structured log artifact is built. +6. Results are serialized and emitted via logger plugins. --- -# 4. SQL Engine and Virtual Table Layer +# Event Pipeline -The SQL engine is built around an in-memory SQLite instance hardened with a strict authorizer. All system data is exposed via dynamically attached **virtual tables** backed by registry plugins. +Event-based tables follow a publisher/subscriber model. ```mermaid flowchart TD - Query["SQL Query"] --> SQLite["SQLite Engine"] - SQLite --> Module["sqlite3_module"] - Module --> VirtualTable["VirtualTable Wrapper"] - VirtualTable --> TablePlugin["TablePlugin"] - TablePlugin --> OS["Operating System Data"] - OS --> Rows["Rows Returned"] + Publisher["Event Publisher Plugin"] --> Subscriber["EventSubscriberPlugin"] + Subscriber --> DB["Database Backend"] + Subscriber --> SQLTable["Event Virtual Table"] + SQLTable --> Query["SELECT From Event Table"] + Query --> Logging["Logging And Query Metadata"] ``` -Key characteristics: - -- In-memory execution -- Strict opcode allowlisting -- Constraint pushdown -- Projection optimization -- Extension-backed writable tables +### Event Model -πŸ“˜ Reference: -`osquery/sql` β†’ *Sql Engine And Virtual Tables* +- Publishers collect OS-level events (e.g., file, process, network). +- Subscribers transform events into rows. +- Rows are persisted and indexed. +- SQL queries retrieve time-bounded results. +- Expiration is schedule-aware. --- -# 5. Configuration and Packs +# Distributed Query Flow -Configuration defines: - -- Scheduled queries -- Packs (query groups) -- Decorators -- File monitoring rules -- Event subscriptions -- Logger configuration -- Runtime options +Distributed querying enables fleet-wide remote execution. ```mermaid flowchart TD - Source["Config Source (File / TLS / Extension)"] --> ConfigCore["Config Singleton"] - ConfigCore --> Parsers["ConfigParserPlugins"] - Parsers --> Schedule["Scheduled Queries"] - Schedule --> Scheduler["Query Scheduler"] - Scheduler --> SQL["SQL Engine"] + Server["Remote Server"] --> TLSPlugin["Distributed TLS Plugin"] + TLSPlugin --> DistributedEngine["Distributed Querying"] + DistributedEngine --> SQL["SQL Engine"] + SQL --> Results["DistributedQueryResult"] + Results --> TLSPlugin + TLSPlugin --> Server ``` -Supports: +Key mechanisms: + +- Pull-based query retrieval +- Discovery query filtering +- SHA-based denylisting +- Performance tracking +- Batched result flushing + +--- + +# Core Modules Documentation + +Below is a structured overview of the repository’s major modules and their documentation paths. +--- + +## 1. Core Runtime And Lifecycle +**Path:** `osquery/core` + +Responsible for: + +- Process bootstrapping +- Flag management (`FlagDetail`, `FlagInfo`) +- Watchdog and worker supervision +- Plugin activation +- Signal handling and shutdown (`ShutdownData`) +- Forced termination guard (`AlarmRunnable`) + +This module orchestrates startup, runtime state transitions, and graceful termination. + +--- + +## 2. Configuration And Packs +**Path:** `osquery/config` + +Responsible for: + +- Loading configuration via plugins +- Parsing JSON configs +- Managing scheduled query packs +- Discovery logic and sharding +- Denylisting unstable queries - Hash-based change detection -- Background refresh runner -- Query denylisting -- Dynamic registry reconfiguration -πŸ“˜ Reference: -`osquery/config` β†’ *Configuration And Packs* +Core components: + +- `ConfigRefreshRunner` +- `PackStats` +- `Schedule` + +Acts as the control plane for scheduled execution. --- -# 6. Eventing Framework +## 3. SQL Engine And Virtual Tables +**Path:** `osquery/sql` -The eventing subsystem provides a publish/subscribe model for event-driven monitoring. +Implements: -```mermaid -flowchart TD - Publisher["EventPublisher"] --> EventFactory["EventFactory"] - Subscriber["EventSubscriber"] --> EventFactory - EventFactory --> Callback["EventCallback"] - Callback --> Storage["Database Storage"] - SQL["SELECT *_events"] --> Subscriber -``` +- Embedded SQLite engine +- Virtual table integration (`sqlite3_module`) +- Query planning and type inference +- Constraint pushdown +- Security authorizer -Capabilities: +Core components: -- Thread-isolated publishers -- Time-window indexing -- Config-driven retention -- Persistent event buffering -- Optimized incremental queries +- `SQLiteSQLPlugin` +- `SQLiteDBManager` +- `VirtualTable` +- `BaseCursor` -πŸ“˜ Reference: -`osquery/events` β†’ *Eventing Framework And Subscriptions* +This module turns OS data sources into relational tables. --- -# 7. Logging and Query Observability +## 4. Database Backend +**Path:** `osquery/database` -osquery converts query executions into structured log artifacts. +Provides: -```mermaid -flowchart TD - Exec["Query Execution"] --> Diff["DiffResults"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serialize["JSON Serialization"] - Serialize --> Logger["LoggerPlugin"] - Logger --> Sink["External Backend"] -``` +- Pluggable database interface +- Domain-based key–value storage +- Persistent RocksDB backend +- Ephemeral in-memory backend +- Schema migration framework -Features: +Core components: -- Differential result tracking -- Snapshot support -- Structured JSON output -- Performance telemetry -- Pluggable logging backends +- `OsqueryDatabase` +- `EphemeralDatabasePlugin` -πŸ“˜ Reference: -`osquery/core` β†’ *Logging And Query Observability* +Stores query state, performance metrics, configuration, events, and distributed state. --- -# 8. Database and Storage Plugins +## 5. Logging And Query Metadata +**Path:** `osquery/core` -All persistent state is stored through a pluggable key-value interface. +Transforms execution results into structured logs: -```mermaid -flowchart TD - Modules["Core Modules"] --> Interface["IDatabaseInterface"] - Interface --> Plugin["Database Plugin"] - Plugin --> RocksDB["Persistent Backend"] - Plugin --> Ephemeral["In-Memory Backend"] -``` +- `QueryLogItem` +- `DiffResults` +- `QueryPerformance` +- `StatusLogLine` + +Handles: + +- Snapshot vs differential logging +- JSON serialization +- Logger plugin integration + +--- + +## 6. Events Core And Subscriptions +**Path:** `osquery/events` -Used for: +Implements: -- Query state -- Event buffers -- Distributed query tracking -- Configuration backup -- Performance metrics +- Publisher/subscriber event framework +- Event persistence and indexing +- Schedule-aware expiration +- Subscriber context isolation -πŸ“˜ Reference: -`osquery/database` β†’ *Database And Storage Plugins* +Core components: + +- `Subscription` +- `GenerateRowsResult` +- `SubscriberExpirationDetails` + +Enables real-time OS monitoring as SQL tables. --- -# 9. Distributed Querying +## 7. Extensions Framework +**Path:** `osquery/extensions` -Enables centralized orchestration of SQL across fleets. +Provides: -```mermaid -flowchart TD - Server["Central Server"] --> TLS["TLS Distributed Plugin"] - TLS --> Core["Distributed Core"] - Core --> SQL["SQL Engine"] - SQL --> Results["Results"] - Results --> TLS - TLS --> Server -``` +- Thrift-based IPC +- Dynamic plugin injection +- RouteUUID-based routing +- Extension health monitoring +- Registry broadcast system -Includes: +Core components: -- Discovery query gating -- SHA256-based denylisting -- Execution performance tracking -- Pluggable transport interface +- `ExtensionManagerHandler` +- `ImplExtensionClient` +- `UuidGenerator` +- `ExtensionManagerWatcher` -πŸ“˜ Reference: -`osquery/distributed` β†’ *Distributed Querying* +Allows external binaries to extend osquery safely. --- -# 10. Extensions and IPC +## 8. Distributed Querying +**Path:** `osquery/distributed` -osquery supports runtime plugin extensions via Apache Thrift IPC. +Manages: -```mermaid -flowchart LR - Core["osquery Core"] --> Manager["Extension Manager"] - Extension["External Extension"] --> Manager - Manager --> Registry["RegistryFactory"] - Core --> Extension -``` +- Pull-based distributed queries +- Discovery gating +- Denylisting and expiration +- Result batching and flushing -Capabilities: +Core components: -- Dynamic table plugins -- Remote SQL execution -- UUID-based routing -- Health monitoring -- Secure socket validation +- `DistributedQueryRequest` +- `DistributedQueryResult` -πŸ“˜ Reference: -`osquery/extensions` β†’ *Extensions And IPC* +Acts as the fleet execution engine. --- -# 11. Remote HTTP Client +## 9. Remote HTTP Client +**Path:** `osquery/remote` -Provides secure HTTP/TLS transport used by: +Implements: -- Distributed querying -- Remote logging -- Remote configuration plugins +- Boost.Asio + Boost.Beast HTTP client +- TLS configuration and validation +- Timeout handling +- Proxy support -```mermaid -flowchart TD - Caller["Module"] --> Client["HTTP Client"] - Client --> TLS["TLS Handshake"] - TLS --> Remote["Remote Server"] -``` +Used by distributed and config TLS plugins. -Built on Boost.Asio and Boost.Beast with strict TLS configuration. +--- -πŸ“˜ Reference: -`osquery/remote` β†’ *Remote Http Client* +## 10. Config Plugins +**Path:** `plugins/config` + +Includes: + +- `FilesystemConfigPlugin` +- `OptionsConfigParserPlugin` +- `ViewsConfigParserPlugin` +- `DecoratorsConfigParserPlugin` + +Transforms raw config into runtime state. --- -# 12. Filesystem and Path Utilities +## 11. Distributed TLS Plugin +**Path:** `plugins/distributed` -Cross-platform abstraction for: +Implements secure HTTPS transport for distributed queries. -- Safe file access -- Glob resolution -- Permission enforcement -- Linux `/proc` inspection -- Windows ACL and metadata handling +Core component: -```mermaid -flowchart TD - Caller["Subsystem"] --> FS["Filesystem API"] - FS --> POSIX["POSIX Implementation"] - FS --> Windows["Windows Implementation"] -``` +- `TLSDistributedPlugin` + +--- + +## 12. Filesystem And Fileops Core +**Path:** `osquery/filesystem` + +Provides: -Security-first design ensures safe extension loading and configuration handling. +- Cross-platform file abstraction +- Secure permission validation +- Globbing and recursive path expansion +- Portable `stat` implementation -πŸ“˜ Reference: -`osquery/filesystem` β†’ *Filesystem And Path Utilities* +Used across config, extensions, logging, and tables. --- -# 13. Repository Structure Summary - -| Module | Path | Responsibility | -|--------|------|----------------| -| Core Init And Runtime | `osquery/core` | Process lifecycle, flags, watchdog | -| Configuration And Packs | `osquery/config` | Scheduled queries, packs, decorators | -| SQL Engine And Virtual Tables | `osquery/sql` | SQLite engine, virtual table layer | -| Eventing Framework | `osquery/events` | Publisher/subscriber event system | -| Logging And Query Observability | `osquery/core` | Result diffing, structured logs | -| Database And Storage Plugins | `osquery/database` | Persistent state abstraction | -| Distributed Querying | `osquery/distributed` | Remote fleet orchestration | -| Extensions And IPC | `osquery/extensions` | Thrift-based runtime extensions | -| Remote HTTP Client | `osquery/remote` | Secure HTTP/TLS transport | -| Filesystem And Path Utilities | `osquery/filesystem` | Cross-platform file abstraction | +## 13. Osquery Shell Devtooling +**Path:** `osquery/devtools` + +Implements `osqueryi` interactive shell: + +- SQL execution loop +- Meta commands +- Pretty-print formatting +- Remote extension execution +- CPU profiling (`rusage`) + +Core components: + +- `callback_data` +- `prettyprint_data` --- -# 14. Architectural Principles +# Architectural Characteristics -1. **SQL as a Universal Interface** -2. **Plugin-Driven Extensibility** -3. **Security-First Execution (SQLite authorizer + permission checks)** -4. **In-Memory Analytical Core** -5. **Config-Driven Behavior** -6. **Differential Logging Efficiency** -7. **Distributed Fleet Control** -8. **Cross-Platform Abstraction Layer** +- βœ… SQLite-powered relational abstraction +- βœ… Strong plugin and registry model +- βœ… Pluggable persistence backend +- βœ… Differential logging for efficiency +- βœ… Schedule-aware event retention +- βœ… Extension-based modularity +- βœ… Secure TLS distributed execution +- βœ… Cross-platform filesystem abstraction --- -# 15. Conclusion +# Conclusion -The `osquery` repository implements a secure, extensible, SQL-based operating system telemetry platform. +The **osquery** repository implements a modular, extensible, SQL-based operating system instrumentation platform. -It combines: +At runtime, it combines: -- A hardened SQLite execution engine -- A virtual table plugin architecture -- Event-driven monitoring -- Differential logging -- Distributed query orchestration -- Runtime extension via IPC -- Secure remote communication +- A secure embedded SQL engine +- A dynamic configuration and scheduling layer +- A pluggable storage backend +- A structured logging pipeline +- A real-time event system +- A distributed query execution engine +- A Thrift-based extension framework -Together, these modules form a scalable, production-grade system observability framework capable of operating both locally and as part of a centrally managed fleet. \ No newline at end of file +Together, these subsystems transform low-level OS primitives into structured, queryable, and remotely orchestratable data β€” enabling observability, compliance, security monitoring, and fleet-wide introspection at scale. \ No newline at end of file diff --git a/docs/reference/architecture/config-plugins/config-plugins.md b/docs/reference/architecture/config-plugins/config-plugins.md new file mode 100644 index 00000000000..6f9c0a8c0f1 --- /dev/null +++ b/docs/reference/architecture/config-plugins/config-plugins.md @@ -0,0 +1,347 @@ +# Config Plugins + +The **Config Plugins** module is responsible for sourcing, parsing, transforming, and dynamically updating osquery configuration data. It acts as the bridge between raw configuration inputs (files or remote sources) and the structured runtime state consumed by the core engine. + +This module is tightly integrated with: + +- [Configuration and Packs](../configuration-and-packs/configuration-and-packs.md) – orchestrates config refresh and pack scheduling +- [SQL Engine and Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) – executes decorator and file path queries +- [Database Backend](../database-backend/database-backend.md) – persists generated views and config state +- [Events Core and Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) – consumes parsed event configuration +- [Logging and Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) – enriched by decorators + +The Config Plugins module is built around two extension points: + +1. **ConfigPlugin** – produces raw configuration blobs (e.g., filesystem, remote, update routing). +2. **ConfigParserPlugin** – parses specific top-level configuration keys and updates runtime state. + +--- + +## Architecture Overview + +```mermaid +flowchart TD + CLI["CLI Flags"] --> FilesystemPlugin["FilesystemConfigPlugin"] + FilesystemPlugin --> ConfigCore["Config Core"] + UpdatePlugin["UpdateConfigPlugin"] --> ConfigCore + + ConfigCore --> DecoratorsParser["DecoratorsConfigParserPlugin"] + ConfigCore --> EventsParser["EventsConfigParserPlugin"] + ConfigCore --> FilePathsParser["FilePathsConfigParserPlugin"] + ConfigCore --> OptionsParser["OptionsConfigParserPlugin"] + ConfigCore --> ViewsParser["ViewsConfigParserPlugin"] + + DecoratorsParser --> SQL["SQL Engine"] + FilePathsParser --> SQL + ViewsParser --> SQL + + ViewsParser --> DB["Database Backend"] + DecoratorsParser --> Logger["Logger"] + EventsParser --> EventsCore["Events Framework"] +``` + +### Key Responsibilities + +- Load configuration from local files or remote sources. +- Merge layered configuration files. +- Parse specific configuration sections. +- Update flags and runtime settings. +- Dynamically create SQL views. +- Register file paths and event subscriptions. +- Enrich logs with runtime decorations. + +--- + +# ConfigPlugin Implementations + +Config plugins generate configuration data for the core engine. + +## FilesystemConfigPlugin + +**Component:** `osquery.plugins.config.filesystem_config.FilesystemConfigPlugin` + +The Filesystem Config Plugin loads configuration from disk. + +### Responsibilities + +- Read the primary config file defined by the `config_path` flag. +- Load layered configuration files from a `.d` directory. +- Support multi-pack generation using glob patterns. +- Validate file existence before parsing. + +### Layered Config Resolution + +```mermaid +flowchart TD + Start["Start genConfig()"] --> CheckFile["Check config_path exists"] + CheckFile -->|"valid"| LoadDir["Resolve config_path.d/*.conf"] + LoadDir --> SortFiles["Sort files"] + SortFiles --> Merge["Append main config"] + Merge --> Output["Return map of source to JSON"] + CheckFile -->|"invalid"| Fail["Return failure Status"] +``` + +### Multi-Pack Support + +If a pack name is `*`, the plugin: + +- Resolves a file pattern. +- Reads each JSON file. +- Builds a synthetic JSON object keyed by filename. +- Emits a merged pack blob. + +This allows external pack directories to behave like embedded pack definitions. + +--- + +## UpdateConfigPlugin + +**Component:** `osquery.plugins.config.update.UpdateConfigPlugin` + +The Update Config Plugin is a special routing plugin that enables: + +- Extension-driven configuration updates. +- Asynchronous config refresh propagation. + +It does not generate configuration directly. Instead, it provides a mechanism for extensions to call `Config::update`, allowing core osquery to refresh configuration state. + +This enables safe plugin-to-core state mutation without violating registry boundaries. + +--- + +# ConfigParserPlugin Implementations + +Parser plugins operate on specific top-level configuration keys and update internal runtime structures. + +--- + +## DecoratorsConfigParserPlugin + +**Component:** `osquery.plugins.config.parsers.decorators.DecoratorsConfigParserPlugin` + +Decorators allow SQL queries to append additional key-value pairs to: + +- Query results +- Snapshot logs +- Status logs + +### Supported Decoration Points + +- `load` – executed when config is loaded. +- `always` – executed before every query. +- `interval` – executed on configured intervals. + +### Decoration Execution Flow + +```mermaid +flowchart TD + ConfigLoad["Config Updated"] --> ParseDecorators["Parse decorators JSON"] + ParseDecorators --> StoreQueries["Store queries by point"] + StoreQueries --> RunLoad["Run load decorators"] + + Scheduler["Query Scheduler"] --> RunAlways["Run always decorators"] + Scheduler --> RunInterval["Run interval decorators"] + + RunLoad --> SQLExec["Execute SQL"] + RunAlways --> SQLExec + RunInterval --> SQLExec + + SQLExec --> DecorationStore["Store first row columns"] + DecorationStore --> Logger["Attach to logs"] +``` + +### Key Design Considerations + +- Only the first row of a decorator query is used. +- Duplicate column names cause undefined behavior. +- Interval decorators must be multiples of 60 seconds. +- Thread-safe access is enforced via mutexes. +- Can be disabled using the `disable_decorators` flag. + +Decorations directly enrich data consumed by the [Logging and Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) module. + +--- + +## EventsConfigParserPlugin + +**Component:** `osquery.plugins.config.parsers.events_parser.EventsConfigParserPlugin` + +Parses the `events` configuration key and publishes structured event configuration into the shared config state. + +### Responsibilities + +- Copy `events` JSON subtree into parser data. +- Expose structured event configuration to the [Events Core and Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) module. + +This parser performs minimal transformation and acts primarily as a structured pass-through. + +--- + +## FilePathsConfigParserPlugin + +**Component:** `osquery.plugins.config.parsers.file_paths.FilePathsConfigParserPlugin` + +Manages file-based monitoring configuration. + +### Supported Keys + +- `file_paths` +- `file_paths_query` +- `file_accesses` +- `exclude_paths` + +### Processing Pipeline + +```mermaid +flowchart TD + ConfigUpdate["Config Update"] --> RemoveOld["Remove existing files for source"] + RemoveOld --> ParseFilePaths["Parse file_paths"] + RemoveOld --> ParseQuery["Parse file_paths_query"] + RemoveOld --> ParseAccess["Parse file_accesses"] + RemoveOld --> ParseExclude["Parse exclude_paths"] + + ParseFilePaths --> Register["Config::addFile()"] + ParseQuery --> SQLExec["Execute SQL query"] + SQLExec --> Register + + ParseAccess --> AccessMap["Build access map"] + ParseExclude --> MergeExclude["Merge exclude paths"] +``` + +### Notable Behaviors + +- Glob wildcards are normalized before registration. +- SQL-driven path discovery must return a `path` column. +- Exclusions are merged across sources. +- File registrations are source-aware and can be removed on update. + +This parser integrates closely with filesystem monitoring and the event subsystem. + +--- + +## OptionsConfigParserPlugin + +**Component:** `osquery.plugins.config.parsers.options.OptionsConfigParserPlugin` + +Handles runtime flag configuration via the `options` key. + +### Responsibilities + +- Merge configuration options across sources. +- Validate flag existence. +- Ignore CLI-only flags. +- Update flag values dynamically. +- Trigger verbosity reconfiguration if needed. + +### Flag Update Flow + +```mermaid +flowchart TD + ParseOptions["Parse options JSON"] --> Validate["Validate flag name"] + Validate -->|"invalid"| Warn["Log warning"] + Validate -->|"valid"| UpdateFlag["Flag::updateValue()"] + UpdateFlag --> CheckVerbose["Is verbosity option?"] + CheckVerbose -->|"yes"| Reconfigure["setVerboseLevel()"] +``` + +### Important Constraints + +- Unknown flags are rejected. +- CLI-only flags cannot be set via config. +- Custom flags must begin with `custom_`. + +This parser directly affects behavior across the entire osquery runtime. + +--- + +## ViewsConfigParserPlugin + +**Component:** `osquery.plugins.config.parsers.views.ViewsConfigParserPlugin` + +Creates and manages dynamic SQL views defined in configuration. + +### Responsibilities + +- Parse `views` dictionary. +- Create or update SQL views. +- Persist view definitions in the database. +- Drop views removed from configuration. + +### View Lifecycle + +```mermaid +flowchart TD + ParseViews["Parse views JSON"] --> LoadExisting["Scan database keys"] + LoadExisting --> Compare["Compare old vs new"] + Compare -->|"changed or new"| DropOld["DROP VIEW"] + DropOld --> CreateNew["CREATE VIEW AS query"] + CreateNew --> Persist["Store in database"] + Compare -->|"removed"| Remove["Delete database entry"] +``` + +### Integration Points + +- Executes SQL using the [SQL Engine and Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md). +- Persists metadata in the [Database Backend](../database-backend/database-backend.md). + +The plugin ensures idempotency by skipping recreation when definitions have not changed, except during first initialization. + +--- + +# End-to-End Configuration Flow + +```mermaid +flowchart TD + Source["Filesystem or Remote Source"] --> ConfigPlugin["ConfigPlugin"] + ConfigPlugin --> ConfigCore["Config Core"] + ConfigCore --> ParserPlugins["ConfigParserPlugins"] + ParserPlugins --> RuntimeState["Runtime State"] + + RuntimeState --> Scheduler["Query Scheduler"] + RuntimeState --> Logger + RuntimeState --> Events + RuntimeState --> SQL +``` + +### Lifecycle Summary + +1. A Config Plugin generates raw configuration blobs. +2. The Config Core merges and normalizes them. +3. Each Config Parser Plugin processes relevant keys. +4. Runtime state is updated atomically. +5. Dependent subsystems react (scheduler, logger, SQL engine, events). + +--- + +# Design Characteristics + +- **Registry-based extensibility** – Plugins register themselves dynamically. +- **Source-aware merging** – Each config source is tracked independently. +- **Thread-safe decoration handling** – Mutex-protected shared state. +- **Idempotent updates** – Avoid unnecessary re-creation of views or flags. +- **Loose coupling** – Parsers modify only their scoped domain. + +--- + +# Relationship to Other Modules + +| Concern | Module | +|----------|--------| +| Config refresh scheduling | [Configuration and Packs](../configuration-and-packs/configuration-and-packs.md) | +| Query execution | [SQL Engine and Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) | +| View persistence | [Database Backend](../database-backend/database-backend.md) | +| Log enrichment | [Logging and Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) | +| Event subscription behavior | [Events Core and Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) | + +--- + +# Conclusion + +The **Config Plugins** module forms the dynamic configuration backbone of osquery. It transforms static or remote JSON configuration into structured runtime behavior, enabling: + +- Flexible deployment models +- Dynamic reconfiguration +- Cross-module feature activation +- Extension-driven updates + +By cleanly separating configuration sourcing from configuration parsing, this module ensures extensibility, modularity, and safe runtime evolution of system behavior. \ No newline at end of file diff --git a/docs/reference/architecture/configuration-and-packs/configuration-and-packs.md b/docs/reference/architecture/configuration-and-packs/configuration-and-packs.md index f9ca515ae14..6fe3415ebb0 100644 --- a/docs/reference/architecture/configuration-and-packs/configuration-and-packs.md +++ b/docs/reference/architecture/configuration-and-packs/configuration-and-packs.md @@ -1,383 +1,365 @@ # Configuration And Packs -The **Configuration And Packs** module is responsible for loading, validating, parsing, distributing, and refreshing osquery configuration data. It transforms raw configuration sources (files, extensions, or remote plugins) into: +## Overview -- Scheduled queries and packs -- File monitoring rules -- Event subscriptions -- Logger and runtime options -- SQL views -- Decorations and metadata +The **Configuration And Packs** module is responsible for: -This module acts as the *control plane* for osquery runtime behavior. It connects configuration sources with the scheduler, SQL engine, eventing framework, logging pipeline, and database layer. +- Loading configuration data from pluggable configuration providers +- Parsing and validating JSON configuration documents +- Building and maintaining the scheduled query packs +- Managing discovery queries, sharding, and platform/version constraints +- Tracking query performance and denylisting unstable queries +- Coordinating reconfiguration across the system when configuration changes ---- +This module acts as the **control plane** for scheduled execution. It translates raw configuration into executable query schedules and propagates changes to other subsystems such as: + +- [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) +- [Logging And Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) +- [Database Backend](../database-backend/database-backend.md) +- [Events Core And Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) +- [Extensions Framework](../extensions-framework/extensions-framework.md) -## Architecture Overview +--- -At a high level, the module consists of: +## Architectural Overview -- A **Config registry** for loading configuration sources -- A **ConfigRefreshRunner** background service -- A **Schedule** abstraction built from Packs -- Multiple **ConfigParserPlugin** implementations -- Backup, hashing, and validation logic +At a high level, configuration flows through the system as follows: ```mermaid flowchart TD - ConfigPlugin["Config Plugin Registry"] --> ConfigCore["Config Singleton"] - ConfigCore --> RefreshRunner["ConfigRefreshRunner"] - ConfigCore --> Schedule["Schedule"] - Schedule --> Pack["Pack"] - ConfigCore --> Parsers["ConfigParserPlugins"] - Parsers --> OptionsParser["Options Parser"] - Parsers --> FilePathsParser["File Paths Parser"] - Parsers --> DecoratorsParser["Decorators Parser"] - Parsers --> EventsParser["Events Parser"] - Parsers --> ViewsParser["Views Parser"] - ConfigCore --> Database[("Persistent Database")] - Schedule --> Logging["Logging And Query Observability"] - Schedule --> SQLEngine["SQL Engine And Virtual Tables"] - Parsers --> EventFramework["Eventing Framework"] + ConfigPlugin["Config Plugin Registry"] -->|"genConfig"| ConfigCore["Config Singleton"] + ConfigCore -->|"updateSource"| JSONValidation["JSON Validation"] + JSONValidation --> Packs["Pack Objects"] + Packs --> Schedule["Schedule Manager"] + Schedule --> SQL["SQL Engine"] + Schedule --> Logger["Logger Plugins"] + ConfigCore --> Database["Database Backend"] + ConfigCore --> EventFactory["Event Factory"] + ConfigCore --> Registry["Plugin Registry"] ``` +### Key Responsibilities + +| Layer | Responsibility | +|--------|----------------| +| Config Plugin | Supplies raw configuration (filesystem, TLS, etc.) | +| Config | Validates, parses, hashes, and updates runtime state | +| Pack | Represents a logical bundle of scheduled queries | +| Schedule | Filters active packs and manages denylisting | +| Database | Persists performance stats, hashes, backups | +| Registry | Reconfigures plugins after config updates | + --- -## Core Responsibilities +## Core Components -### 1. Configuration Loading +### 1. ConfigRefreshRunner -The module defines a `config` registry that supports pluggable configuration sources: +**Component:** `osquery.osquery.config.config.ConfigRefreshRunner` -- `FilesystemConfigPlugin` (default) -- `UpdateConfigPlugin` (internal chaining for extensions) +This internal runnable is responsible for periodic configuration refresh. -Each plugin implements: +#### Responsibilities -- `genConfig()` β†’ produce configuration map -- `genPack()` β†’ resolve pack references -- `update()` β†’ apply configuration updates +- Sleeps for `config_refresh` interval +- Calls `Config::refresh()` +- Accelerates retry interval when refresh fails +- Stops when shutdown is requested -The active plugin is selected via the `config_plugin` flag. +#### Refresh Lifecycle ---- +```mermaid +flowchart TD + Start["Thread Start"] --> Pause["Sleep refresh_sec"] + Pause --> Check["Interrupted?"] + Check -->|"Yes"| End["Stop"] + Check -->|"No"| Refresh["Config.refresh()"] + Refresh --> Pause +``` -### 2. Config Singleton +Refresh interval behavior: -The `Config` class is a process-wide singleton that: +- Normal interval: `config_refresh` +- Failure interval: `config_accelerated_refresh` +- Optional backup restore on first failure -- Loads configuration -- Maintains the active Schedule -- Applies parser plugins -- Tracks source hashes -- Backs up and restores configuration -- Coordinates plugin reconfiguration +--- -Key behaviors: +### 2. Config -- JSON validation (max depth and size constraints) -- Comment stripping -- Hash-based change detection -- Safe concurrent access using mutexes -- Performance tracking per scheduled query +The `Config` class is a **singleton control center** for configuration state. ---- +#### Primary Responsibilities -### 3. Refresh Lifecycle +- Call active config plugin (`genConfig`) +- Validate JSON structure and depth +- Hash sources to detect changes +- Build and update packs +- Apply config parser plugins +- Purge stale query results +- Trigger registry reconfiguration +- Manage performance stats and denylisting -The `ConfigRefreshRunner` runs in the background when `config_refresh` is enabled. +#### Configuration Update Flow ```mermaid flowchart TD - Start["ConfigRefreshRunner Start"] --> Wait["Sleep refresh_sec"] - Wait --> Check["Interrupted?"] - Check -->|No| Refresh["Config.refresh()"] - Refresh --> Wait - Check -->|Yes| End["End"] + Gen["Registry.call genConfig"] --> Update["Config.update()"] + Update --> Purge["Purge stale queries"] + Purge --> UpdateSource["updateSource(source)"] + UpdateSource --> Validate["Validate JSON"] + Validate --> Extract["Extract schedule and packs"] + Extract --> ApplyParsers["Apply config parsers"] + ApplyParsers --> Reconfigure["Reconfigure registries"] + Reconfigure --> Done["Config Valid"] ``` -Features: - -- Normal refresh interval -- Accelerated retry interval on failure -- Optional backup restoration on first failure -- Dynamic reconfiguration of registries and loggers - --- ## Packs and Scheduling -The Schedule is composed of `Pack` objects. +### 3. Pack -### Pack +**Component:** `osquery.osquery.config.packs.Pack` -A **Pack** represents a group of scheduled queries plus optional discovery logic. +A Pack represents a logical bundle of scheduled queries. -Capabilities: +Each pack may contain: +- Discovery queries +- Scheduled queries - Platform constraints - Version constraints - Shard percentage -- Discovery queries -- ScheduledQuery collection -- Execution eligibility (`shouldPackExecute()`) -### Schedule +#### Pack Execution Logic -The Schedule: +A pack is considered active when: -- Maintains active packs -- Filters packs via discovery checks -- Tracks denylisted queries -- Restores failed queries on restart -- Provides query iteration to the scheduler +1. Platform check passes +2. Version check passes +3. Shard requirement matches +4. Discovery queries return expected results ```mermaid flowchart TD - ConfigUpdate["Config Update"] --> BuildPacks["Create Pack Objects"] - BuildPacks --> DiscoveryCheck["Discovery Queries"] - DiscoveryCheck --> ActivePack["Active Pack"] - ActivePack --> Scheduler["ScheduledQuery Execution"] - Scheduler --> Performance["Query Performance Recording"] - Scheduler --> Denylist["Denylist On Failure"] + Platform["Check Platform"] --> Version["Check Version"] + Version --> Shard["Check Shard"] + Shard --> Discovery["Run Discovery Queries"] + Discovery --> Active["Pack Active"] ``` -### Denylist Logic - -If a scheduled query: - -- Causes watchdog termination -- Was executing during shutdown - -It is denylisted temporarily and persisted in the database. - -Expiration logic considers: - -- Time-based expiry -- Event-based exemptions -- Explicit `denylist=false` option +Pack state is cached to avoid repeatedly executing discovery queries unnecessarily. --- -## Config Parser Plugins +### 4. Schedule -The module defines a `config_parser` registry. Each parser consumes specific top-level configuration keys. +The `Schedule` class maintains all active packs. -Parsers are applied in this order: +It: -1. Options parser (always first) -2. Remaining parsers in registry order +- Stores pack objects +- Filters packs using `shouldPackExecute()` +- Tracks failed queries +- Maintains denylist with expiration +- Restores denylist from persistent storage -### FilesystemConfigPlugin +#### Denylist Behavior -Loads configuration from: +When a scheduled query crashes or exceeds watchdog limits: -- Primary JSON file -- Optional `.d` directory fragments -- Multi-pack glob patterns +- It is written to the database (`failed_queries`) +- Added to in-memory denylist +- Skipped during scheduling +- Automatically expires after a timeout -Produces a map of `source β†’ JSON content`. +```mermaid +flowchart TD + Failure["Query Failure"] --> Persist["Persist to Database"] + Persist --> Denylist["Add to Denylist"] + Denylist --> Skip["Skip During Scheduling"] + Skip --> Expire["Expiration Check"] + Expire -->|"Expired"| Reactivate["Allow Execution"] +``` --- -### OptionsConfigParserPlugin +## Pack Statistics -Handles the `options` key. +### 5. PackStats -- Updates runtime flags -- Prevents modification of CLI-only flags -- Reconfigures logger verbosity dynamically -- Supports `custom_` prefixed flags +**Component:** `osquery.osquery.config.packs.PackStats` ---- +Tracks discovery query effectiveness: -### FilePathsConfigParserPlugin +| Field | Meaning | +|-------|----------| +| total | Total discovery executions | +| hits | Successful discovery matches | +| misses | Discovery failures | -Handles: +These metrics help determine whether packs are frequently inactive due to discovery logic. -- `file_paths` -- `file_paths_query` -- `file_accesses` -- `exclude_paths` +--- -Responsibilities: +## Configuration Parsers -- Adds file monitoring rules to Config -- Executes SQL-based path discovery -- Normalizes wildcard globs -- Maintains access category mappings +The module defines a **config_parser registry**. -This integrates directly with the **Filesystem And Path Utilities** and **Eventing Framework And Subscriptions** modules. +Parser plugins: ---- +- Receive JSON fragments by key +- Update subsystem state +- Do not expose call actions +- Operate through structured property trees -### DecoratorsConfigParserPlugin +Execution order: -Handles the `decorators` key. +1. `options` parser (always first) +2. Remaining parsers -Decorators: +This ensures flags and options are configured before dependent parsers execute. -- Execute SQL queries -- Extract first-row column values -- Attach results to logs +--- + +## Backup and Restore -Supported execution points: +If `config_enable_backup` is enabled: -- `load` -- `always` -- `interval` +- Each config source is persisted with prefix `config_persistence.` +- On first refresh failure, backup is restored +- Old keys are pruned on update ```mermaid flowchart TD - ConfigLoad["Config Load"] --> RunLoad["Run Load Decorators"] - QueryExec["Query Execution"] --> RunAlways["Run Always Decorators"] - IntervalTick["Interval Timer"] --> RunInterval["Run Interval Decorators"] - RunLoad --> Decorations[("Decoration Store")] - RunAlways --> Decorations - RunInterval --> Decorations - Decorations --> Logger["Status And Query Logs"] + Update["Successful Update"] --> Backup["Backup Config Sources"] + Failure["Refresh Failure"] --> Restore["Restore From Database"] + Restore --> Update ``` -Thread safety is enforced with dedicated mutexes. - --- -### EventsConfigParserPlugin +## Query Performance Tracking -Consumes the `events` key and exposes configuration to event publishers and subscribers. - -Acts as a bridge to the **Eventing Framework And Subscriptions** module. - ---- +The module integrates tightly with [Logging And Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md). -### ViewsConfigParserPlugin +### Recorded Metrics -Handles the `views` key. +- Wall time +- User time +- System time +- Memory usage delta +- Output size +- Execution count +- Last executed timestamp -- Creates SQL views -- Drops outdated views -- Persists view definitions -- Ensures idempotent updates +Performance is stored in the database under `kQueryPerformance`. -This integrates with the **SQL Engine And Virtual Tables** module. +When queries are modified or removed, performance data is cleared to avoid stale metrics. --- -## Backup and Persistence +## Purge and Expiration -If `config_enable_backup` is enabled: - -- Config sources are stored in the database -- Failed refresh attempts restore backup -- Backup keys are namespaced with `config_persistence.` +During updates, the module: -The module also persists: +- Scans persisted query result sets +- Removes entries for queries no longer in schedule +- Expires results older than one week -- Query performance statistics -- Executing query markers -- Denylist entries -- Query result timestamps - -Database interactions use the **Database And Storage Plugins** module. +This prevents unbounded database growth and stale result accumulation. --- ## Hashing and Change Detection -Each configuration source is hashed (SHA1): +Each configuration source is hashed using SHA1. -- Prevents unnecessary reconfiguration -- Enables selective updates -- Supports global config hash generation +If the hash does not change: -If a source hash does not change: +- The source is skipped +- Reconfiguration is avoided -- Packs are not rebuilt -- Parsers are not re-applied +A global configuration hash can be generated by combining source hashes. + +```mermaid +flowchart LR + SourceA["Source A Hash"] --> Combine["Combine Hashes"] + SourceB["Source B Hash"] --> Combine + Combine --> Global["Global Config Hash"] +``` --- -## Interaction With Other Modules +## Thread Safety -### Logging And Query Observability +The module uses multiple mutexes: -- Query performance stats recorded -- Decorations attached to logs -- Denylist logging +- `config_hash_mutex_` +- `config_refresh_mutex_` +- `config_backup_mutex_` +- `config_schedule_mutex_` +- `config_files_mutex_` +- `config_performance_mutex_` -### SQL Engine And Virtual Tables +This ensures: -- Discovery queries -- Scheduled queries -- View creation +- Safe concurrent refresh +- Safe schedule iteration +- Consistent performance updates +- Atomic configuration transitions -### Eventing Framework And Subscriptions +--- -- Event configuration via `events` -- File path monitoring integration +## Integration with Other Modules -### Database And Storage Plugins +### Core Runtime And Lifecycle -- Config backup -- Query stats -- Denylist persistence -- Result expiration +- Refresh thread runs through dispatcher +- Shutdown interrupts refresh loop -### Extensions And IPC +### Database Backend -- External extensions can call `Config::update()` -- UpdateConfigPlugin enables update chaining -- External registries propagate updates back to core +Used for: ---- +- Denylist persistence +- Performance metrics +- Backup storage +- Query timestamp tracking -## Configuration Processing Flow +### Extensions Framework -```mermaid -flowchart TD - Source["Config Source"] --> GenConfig["genConfig()"] - GenConfig --> Update["Config.update()"] - Update --> Purge["Purge Stale State"] - Purge --> UpdateSource["updateSource()"] - UpdateSource --> ParseJSON["Validate And Parse JSON"] - ParseJSON --> BuildSchedule["Create Packs And Schedule"] - BuildSchedule --> ApplyParsers["Apply ConfigParserPlugins"] - ApplyParsers --> Reconfigure["Registry.configure()"] - Reconfigure --> EventNotify["EventFactory.configUpdate()"] - EventNotify --> Ready["Runtime Updated"] -``` +- External processes may call `update` +- Updates are forwarded to core +- Extension configs propagate to main runtime ---- +### Events Core And Subscriptions -## Concurrency and Safety +After configuration changes: -The module uses: +- Event publishers and subscribers are reconfigured +- `EventFactory::configUpdate()` is triggered -- Recursive mutexes for schedule and files -- Dedicated mutexes for: - - Hash map - - Backup store - - Performance stats - - Decorations +--- -Design goals: +## Summary -- Safe asynchronous refresh -- Deterministic schedule rebuilds -- Minimal runtime disruption -- Atomic config transitions +The **Configuration And Packs** module is the orchestration layer that: ---- +- Converts configuration into executable schedules +- Controls pack activation and filtering +- Protects runtime stability through denylisting +- Maintains deterministic scheduling behavior +- Coordinates dynamic reconfiguration across subsystems -## Summary +It is central to runtime adaptability, reliability, and consistency of scheduled query execution. -The **Configuration And Packs** module is the orchestration layer of osquery runtime behavior. It: +Without this module, the system would lack: -- Loads configuration from pluggable sources -- Builds and manages query packs -- Applies runtime options -- Coordinates parser plugins -- Manages refresh and backup logic -- Integrates deeply with scheduler, SQL engine, event system, logging, and database layers +- Dynamic configuration refresh +- Safe scheduling guarantees +- Distributed pack management +- Query lifecycle observability -Without this module, osquery would have no dynamic runtime control plane. It is the central authority that transforms static configuration data into active system behavior. +It forms the backbone of runtime query orchestration within the platform. \ No newline at end of file diff --git a/docs/reference/architecture/core-init-and-runtime/core-init-and-runtime.md b/docs/reference/architecture/core-init-and-runtime/core-init-and-runtime.md deleted file mode 100644 index 03460755d71..00000000000 --- a/docs/reference/architecture/core-init-and-runtime/core-init-and-runtime.md +++ /dev/null @@ -1,428 +0,0 @@ -# Core Init And Runtime - -The **Core Init And Runtime** module is the foundational execution layer of osquery. It is responsible for: - -- Process initialization and bootstrap -- Command-line flag management -- Worker / watcher lifecycle orchestration -- Graceful and forced shutdown handling -- Platform and identity utilities (UUID, host identity, privilege control) -- Virtual table execution context and constraint handling -- Watchdog-based resource governance - -This module defines the runtime contract that all higher-level subsystems rely on, including: - -- [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md) -- [Logging And Query Observability](../logging-and-query-observability/logging-and-query-observability.md) -- [Database And Storage Plugins](../database-and-storage-plugins/database-and-storage-plugins.md) -- [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) -- [Eventing Framework And Subscriptions](../eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md) -- [Extensions And IPC](../extensions-and-ipc/extensions-and-ipc.md) - ---- - -## 1. Architectural Overview - -At runtime, osquery can operate in multiple modes: - -- **Daemon** (`osqueryd`) -- **Shell** (`osqueryi`) -- **Watcher (watchdog)** -- **Worker** (child process of watcher) -- **Extension process** - -The Core Init And Runtime module orchestrates how these modes are selected, started, monitored, and shut down. - -### High-Level Runtime Architecture - -```mermaid -flowchart TD - Main["Main Entry Point"] --> Initializer["Initializer"] - - Initializer --> Flags["Flag System"] - Initializer --> Registry["Registry And Plugins"] - Initializer --> Database["Database Initialization"] - Initializer --> ExtensionManager["Extension Manager"] - Initializer --> Events["Event Factory"] - Initializer --> Watchdog["Watcher / Worker Model"] - - Watchdog --> Worker["Worker Process"] - Watchdog --> Extensions["Extension Processes"] - - Worker --> Tables["Virtual Tables"] - Worker --> Scheduler["Query Scheduler"] - Worker --> Logger["Logger Plugins"] -``` - -The **Initializer** class is the central orchestrator of this lifecycle. - ---- - -## 2. Initialization Lifecycle - -The `Initializer` class (defined in `system.h` and implemented in `init.cpp`) governs startup behavior. - -### Core Responsibilities - -- Parse CLI flags and flagfiles -- Determine tool type (daemon, shell, extension) -- Configure runtime defaults -- Initialize registries and plugins -- Start extension manager -- Load configuration -- Attach event publishers -- Activate logging and distributed plugins -- Launch watcher/worker model if enabled - -### Startup Sequence - -```mermaid -sequenceDiagram - participant Main - participant Initializer - participant Registry - participant Config - participant Extensions - participant Events - - Main->>Initializer: Construct(argc, argv) - Initializer->>Initializer: Parse flags - Initializer->>Registry: registryAndPluginInit() - Initializer->>Extensions: startExtensionManager() - Initializer->>Config: load() - Initializer->>Events: attachEvents() - Initializer->>Initializer: start() -``` - -The `start()` method finalizes runtime activation after construction. - ---- - -## 3. Flag System - -Core Components: - -- `FlagDetail` -- `FlagInfo` -- `Flag` - -The flag system wraps **Google GFlags** to provide: - -- Structured metadata (shell-only, CLI-only, hidden, extension-only) -- Default value tracking -- Runtime updates -- Custom flags from configuration - -### Flag Architecture - -```mermaid -flowchart LR - Macro["FLAG / CLI_FLAG Macros"] --> FlagCreate["Flag::create()"] - FlagCreate --> FlagRegistry["Internal Flag Map"] - - CLI["Command Line"] --> Parse["ParseCommandLineFlags"] - Config["Config Options"] --> Update["Flag::updateValue()"] - - FlagRegistry --> Runtime["Runtime Access"] -``` - -Important behaviors: - -- Flags can be set via CLI, flagfile, or config (unless CLI-only) -- `Flag::isDefault()` detects overridden values -- Aliases support backwards compatibility - -Flags influence: - -- Database activation -- Watchdog enablement -- Logging and distributed plugins -- OpenFrame mode - ---- - -## 4. Worker and Watcher Model - -Core Components: - -- `LimitDefinition` -- `PerformanceChange` - -The watcher/worker architecture isolates execution for reliability and resource control. - -### Roles - -| Role | Responsibility | -|------|----------------| -| Watcher | Monitors worker and extensions | -| Worker | Executes queries and plugins | -| Extension | External plugin process | - -### Watchdog Control Flow - -```mermaid -flowchart TD - Watcher["Watcher Process"] --> Spawn["Spawn Worker"] - Spawn --> Monitor["Monitor CPU And Memory"] - Monitor --> Check{{"Limit Exceeded?"}} - Check -->|"No"| Continue["Continue Monitoring"] - Check -->|"Yes"| Kill["Stop Worker"] - Kill --> Respawn["Respawn Worker"] -``` - -### Resource Limits - -`LimitDefinition` defines profiles: - -- Normal -- Restrictive -- Disabled - -Enforced metrics: - -- Memory footprint (MB) -- CPU utilization percentage -- Sustained latency duration -- Respawn frequency - -`PerformanceChange` calculates CPU window utilization based on interval and CPU count. - -If limits are exceeded: - -- Worker is gracefully terminated -- Forced kill occurs if needed -- Restart logic applies exponential backoff - ---- - -## 5. Shutdown Coordination - -Core Component: - -- `ShutdownData` - -Shutdown is centralized and thread-safe. - -### Shutdown Model - -```mermaid -flowchart TD - AnyThread["Any Thread"] --> Request["requestShutdown()"] - Request --> Signal["Condition Variable"] - Signal --> MainThread["Main Thread"] - MainThread --> StopServices["Dispatcher::stopServices()"] - StopServices --> Join["Join Services"] - Join --> StopEvents["EventFactory::end()"] - StopEvents --> CloseDB["shutdownDatabase()"] - CloseDB --> Exit["Return Exit Code"] -``` - -Key properties: - -- Shutdown can only be requested once -- Exit code is stored atomically -- `AlarmRunnable` enforces maximum shutdown time -- `shutdownNow()` performs immediate `_Exit()` - -This guarantees deterministic teardown even during deadlocks. - ---- - -## 6. Platform and System Utilities - -Core Components: - -- `PrivateData` -- `stat` -- `tm` - -### Host Identity - -The module manages multiple UUID strategies: - -- Hardware UUID -- Instance UUID (persistent) -- Ephemeral UUID -- Specified UUID -- Hostname fallback - -Flow: - -```mermaid -flowchart LR - Flag["host_identifier Flag"] --> Mode{{"Mode"}} - Mode -->|"uuid"| Hardware["Hardware UUID"] - Mode -->|"instance"| Instance["Instance UUID"] - Mode -->|"ephemeral"| Ephemeral["Random UUID"] - Mode -->|"specified"| Specified["Configured Identifier"] - Mode -->|"default"| Hostname["System Hostname"] -``` - -Persistent identifiers are stored via the database layer. - -### Privilege Dropping (POSIX) - -- Temporarily reduce effective UID/GID -- Restore original groups on destruction -- Used when interacting with filesystem-backed tables - -### Thread Naming - -Cross-platform thread naming for observability and debugging. - ---- - -## 7. Virtual Table Execution Context - -Core Components: - -- `Constraint` -- `ConstraintList` -- `QueryContext` -- `VirtualTableContent` - -This subsystem forms the bridge between: - -- SQLite virtual table engine -- osquery table plugins - -### Constraint Model - -```mermaid -flowchart TD - SQL["SQL Query"] --> SQLite["SQLite Virtual Table API"] - SQLite --> QueryContext["QueryContext"] - QueryContext --> ConstraintMap["ConstraintMap"] - ConstraintMap --> ConstraintList["ConstraintList"] - ConstraintList --> TablePlugin["TablePlugin::generate()"] -``` - -`ConstraintList` supports: - -- Operator-aware filtering -- Affinity-aware matching (TEXT, INTEGER, BIGINT) -- Efficient pre-filter evaluation - -`QueryContext` provides: - -- Column usage tracking -- Constraint expansion -- Cache control flags - -### VirtualTableContent - -Stores: - -- Column definitions -- Attributes (CACHEABLE, EVENT_BASED, etc.) -- Aliases -- Query-scoped caches - -This metadata allows high-performance virtual table execution without repeated registry lookups. - ---- - -## 8. Table Plugin Integration - -Although fully described in [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md), Core Init And Runtime provides: - -- Context serialization -- Cache freshness checks -- Generator-based row streaming -- Extension table attachment - -Caching model: - -- Interval-based freshness -- Disabled when constraints are complex -- Serialized into backing database - ---- - -## 9. Signal Handling - -Signals handled: - -- `SIGTERM` -- `SIGINT` -- `SIGUSR1` - -Behavior: - -- `SIGUSR1` β†’ mark resource limit hit -- `SIGTERM` / `SIGINT` β†’ request graceful shutdown -- Alarm timeout enforces upper bound on shutdown time - ---- - -## 10. OpenFrame Mode Integration - -Additional flags: - -- `openframe_mode` -- `openframe_secret` -- `openframe_token_path` - -When enabled: - -- Initializes encryption service -- Extracts token -- Starts token refresher -- Updates authorization manager - -This integrates osquery runtime with OpenFrame-managed authentication. - ---- - -## 11. How This Module Fits the System - -The Core Init And Runtime module: - -- Bootstraps **all subsystems** -- Owns lifecycle and teardown guarantees -- Provides runtime isolation (watchdog model) -- Supplies identity and environment context -- Enables safe virtual table execution -- Enforces resource governance - -Every other module depends on it either directly or indirectly. - -### Dependency Positioning - -```mermaid -flowchart TD - Core["Core Init And Runtime"] - - Core --> Config["Configuration And Packs"] - Core --> Logging["Logging And Query Observability"] - Core --> Database["Database And Storage Plugins"] - Core --> SQL["SQL Engine And Virtual Tables"] - Core --> Events["Eventing Framework And Subscriptions"] - Core --> Extensions["Extensions And IPC"] -``` - -Without this module: - -- No flags would be parsed -- No plugins would activate -- No scheduler would run -- No graceful shutdown would occur - -It is the execution spine of the osquery system. - ---- - -## Summary - -The **Core Init And Runtime** module is responsible for transforming a process invocation into a fully operational, resource-governed, plugin-enabled osquery runtime. - -It provides: - -- Deterministic initialization -- Mode-aware execution (daemon, shell, extension) -- Resource watchdog enforcement -- Safe shutdown semantics -- Host identity management -- Virtual table execution scaffolding - -All higher-level capabilities build on top of this foundational runtime layer. \ No newline at end of file diff --git a/docs/reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md b/docs/reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md new file mode 100644 index 00000000000..22b49f507dc --- /dev/null +++ b/docs/reference/architecture/core-runtime-and-lifecycle/core-runtime-and-lifecycle.md @@ -0,0 +1,281 @@ +# Core Runtime And Lifecycle + +The **Core Runtime And Lifecycle** module is responsible for bootstrapping, configuring, supervising, and gracefully shutting down the osquery process. It defines: + +- The global flag system and runtime configuration surface +- Process initialization for daemon, shell, watcher, and worker modes +- Signal handling and shutdown coordination +- Resource limit enforcement and forced termination safeguards + +This module acts as the orchestration layer that connects configuration, logging, extensions, database, distributed querying, and event subsystems into a coherent runtime. + +--- + +## 1. Architectural Overview + +At a high level, the runtime lifecycle follows this sequence: + +```mermaid +flowchart TD + Entry["Process Entry Point"] --> Init["Initializer Constructor"] + Init --> ParseFlags["Parse Flags & CLI"] + ParseFlags --> SetupRegistry["Registry & Plugin Initialization"] + SetupRegistry --> ModeCheck{{"Shell / Daemon / Extension?"}} + + ModeCheck -->|"Shell"| ShellInit["Shell Initialization"] + ModeCheck -->|"Daemon"| DaemonInit["Daemon Initialization"] + ModeCheck -->|"Extension"| ExtInit["Extension Initialization"] + + ShellInit --> StartRuntime["Initializer.start()"] + DaemonInit --> StartRuntime + ExtInit --> StartRuntime + + StartRuntime --> LoadConfig["Load Config Plugin"] + LoadConfig --> ActivateLogger["Activate Logger Plugin"] + ActivateLogger --> ActivateDistributed["Activate Distributed Plugin"] + ActivateDistributed --> AttachEvents["Attach Event Subscribers"] + AttachEvents --> Running["Runtime Active"] + + Running -->|"Signal / Error"| ShutdownReq["requestShutdown()"] + ShutdownReq --> Graceful["Dispatcher Stop & Join"] + Graceful --> AlarmGuard["AlarmRunnable Guard"] + AlarmGuard --> Exit["Process Exit"] +``` + +### Key Responsibilities + +| Area | Responsibility | +|------|----------------| +| Flags | Define and track runtime options | +| Initialization | Configure process mode and environment | +| Plugin Activation | Select and activate config, logger, distributed plugins | +| Watchdog Model | Supervise worker processes | +| Shutdown | Coordinate graceful or forced termination | + +--- + +## 2. Flag System + +**Core Components:** +- `osquery.osquery.core.flags.FlagDetail` +- `osquery.osquery.core.flags.FlagInfo` + +The flag subsystem wraps Google GFlags and adds metadata tracking specific to osquery. + +### 2.1 FlagDetail + +Defines visibility and behavioral attributes: + +- `description` +- `shell` (shell-only) +- `external` (extension-only) +- `cli` (CLI-only, not config-settable) +- `hidden` + +### 2.2 FlagInfo + +Provides runtime inspection data: + +- Type +- Description +- Default value +- Current value +- Associated `FlagDetail` + +### 2.3 Flag Registry + +All flags are tracked centrally via the `Flag` singleton: + +```mermaid +flowchart LR + Define["FLAG Macro"] --> Create["Flag::create()"] + Create --> Registry["Flag Internal Map"] + Registry --> Query["getValue() / getType()"] + Registry --> Print["printFlags()"] +``` + +This allows: + +- Runtime introspection +- Controlled updates via `updateValue()` +- Alias support +- Differentiation between CLI, shell, and extension flags + +Flags directly influence: + +- Plugin activation +- Watchdog behavior +- Database enablement +- Distributed querying +- OpenFrame mode activation + +--- + +## 3. Process Initialization (Initializer) + +**Core Component:** +- `osquery.osquery.core.init.AlarmRunnable` + +The `Initializer` orchestrates runtime startup. It: + +1. Detects tool type (shell, daemon, extension) +2. Parses flags +3. Initializes registries and plugins +4. Sets up database and configuration +5. Attaches event loops +6. Transitions to active runtime + +### 3.1 Tool Modes + +The runtime distinguishes between: + +- **Shell** (`osqueryi`) +- **Daemon** (`osqueryd`) +- **Extension** process +- **Watcher** (watchdog supervisor) +- **Worker** (spawned by watcher) + +```mermaid +flowchart TD + Initializer["Initializer"] --> Mode{{"ToolType"}} + Mode --> Shell["Shell Mode"] + Mode --> Daemon["Daemon Mode"] + Mode --> Extension["Extension Mode"] + + Daemon --> Watchdog{{"Watchdog Enabled?"}} + Watchdog -->|"Yes"| Watcher["Watcher Process"] + Watchdog -->|"No"| DirectRun["Single Process Runtime"] + + Watcher --> Worker["Worker Process"] +``` + +### 3.2 Watchdog Model + +If watchdog is enabled: + +- A **Watcher** supervises a **Worker** process +- Worker executes core logic +- Watcher restarts worker on failure +- CPU and I/O priority adjustments are applied + +This model improves resilience and resource isolation. + +--- + +## 4. Plugin and Subsystem Activation + +During `Initializer::start()`: + +1. Database plugin is initialized +2. Extension manager starts +3. Config plugin is activated +4. Logger plugin is activated +5. Distributed plugin is activated +6. Event threads are attached + +Relevant subsystems are documented separately: + +- [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md) +- [Logging And Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) +- [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) +- [Database Backend](../database-backend/database-backend.md) +- [Extensions Framework](../extensions-framework/extensions-framework.md) +- [Events Core And Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) +- [Distributed Querying](../distributed-querying/distributed-querying.md) + +The Core Runtime And Lifecycle module does not implement these subsystems directly; it activates and coordinates them. + +--- + +## 5. Signal Handling and Shutdown Coordination + +**Core Component:** +- `osquery.osquery.core.shutdown.ShutdownData` + +Shutdown is centralized and thread-safe. + +### 5.1 ShutdownData + +Encapsulates: + +- `condition_variable` for coordination +- `mutex` for synchronization +- `atomic` for shutdown state +- Exit code storage + +```mermaid +flowchart TD + Signal["SIGTERM / SIGINT / SIGUSR1"] --> Handler["signalHandler()"] + Handler --> Request["requestShutdown()"] + Request --> Notify["Condition Variable Notify"] + Notify --> MainWait["waitForShutdown()"] + MainWait --> GracefulStop["Dispatcher::stopServices()"] +``` + +### 5.2 AlarmRunnable (Forced Shutdown Guard) + +`AlarmRunnable` runs in a separate thread during shutdown: + +- Waits in 200 ms intervals +- Monitors elapsed time +- If timeout exceeds `alarm_timeout`, calls `shutdownNow()` + +This prevents indefinite hangs during teardown. + +```mermaid +flowchart TD + ShutdownStart["Initializer::shutdown()"] --> StartAlarm["Spawn AlarmRunnable"] + StartAlarm --> StopServices["Stop & Join Services"] + StopServices --> EndEvents["EventFactory::end()"] + EndEvents --> ShutdownDB["shutdownDatabase()"] + ShutdownDB --> CancelAlarm["Interrupt AlarmRunnable"] + CancelAlarm --> Exit["Return Exit Code"] +``` + +--- + +## 6. OpenFrame Integration + +The module optionally enables OpenFrame mode using runtime flags: + +- `openframe_mode` +- `openframe_secret` +- `openframe_token_path` + +When enabled: + +1. Encryption service is created +2. Token extractor retrieves initial token +3. Authorization manager is updated +4. Token refresher thread starts + +OpenFrame is initialized during startup but remains decoupled from the core lifecycle logic. + +--- + +## 7. Resource Limits and Safety + +The initializer: + +- Sets file descriptor limits (POSIX) +- Adjusts scheduling priority when watchdog is enabled +- Supports resource limit detection via `resourceLimitHit()` + +If limits are exceeded, the shutdown mechanism can be triggered. + +--- + +## 8. Lifecycle Summary + +The Core Runtime And Lifecycle module provides: + +- A structured startup pipeline +- Mode-aware runtime behavior +- Plugin activation sequencing +- Robust shutdown guarantees +- Watchdog-based resilience +- Centralized flag and configuration handling + +It is the execution backbone of the system, ensuring that all higher-level modules operate within a controlled, observable, and recoverable runtime environment. + +For subsystem-specific behavior, refer to the corresponding module documentation listed above. \ No newline at end of file diff --git a/docs/reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md b/docs/reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md deleted file mode 100644 index c3210d09ee9..00000000000 --- a/docs/reference/architecture/database-and-storage-plugins/database-and-storage-plugins.md +++ /dev/null @@ -1,357 +0,0 @@ -# Database And Storage Plugins - -## Overview - -The **Database And Storage Plugins** module provides the persistent and ephemeral key-value storage abstraction used by osquery core. It exposes a unified database interface that: - -- Persists configuration state, scheduled query metadata, event buffers, logs, and distributed query state. -- Abstracts the underlying storage engine (e.g., RocksDB or in-memory ephemeral storage). -- Provides safe concurrent access with reset and migration support. -- Enables plugin-based database backends via the internal registry system. - -At runtime, this module acts as the storage backbone for higher-level modules such as: - -- [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md) -- [Logging And Query Observability](../logging-and-query-observability/logging-and-query-observability.md) -- [Distributed Querying](../distributed-querying/distributed-querying.md) -- [Eventing Framework And Subscriptions](../eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md) -- [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) - -It is implemented using a registry-driven plugin architecture, allowing the storage backend to be replaced or disabled. - ---- - -## Architectural Overview - -### High-Level Architecture - -```mermaid -flowchart TD - CoreModules["Core And Feature Modules"] --> DBInterface["IDatabaseInterface"] - DBInterface --> OsqueryDB["OsqueryDatabase"] - OsqueryDB --> RegistryLayer["Database Plugin Registry"] - RegistryLayer --> RocksDBPlugin["RocksDB Plugin (Persistent)"] - RegistryLayer --> EphemeralPlugin["Ephemeral Database Plugin (In-Memory)"] - - subgraph storage_domains["Logical Storage Domains"] - PersistentSettings["configurations"] - Queries["queries"] - Events["events"] - Logs["logs"] - Carves["carves"] - Distributed["distributed"] - QueryPerformance["query_performance"] - end - - RocksDBPlugin --> storage_domains - EphemeralPlugin --> storage_domains -``` - -The module separates concerns into three layers: - -1. **Public Database Interface** – Used by the rest of the system. -2. **Plugin Registry Layer** – Selects and manages the active database backend. -3. **Concrete Database Plugins** – Implement actual storage logic. - ---- - -## Core Components - -### OsqueryDatabase - -**Component:** `osquery.osquery.database.database.OsqueryDatabase` - -`OsqueryDatabase` implements the `IDatabaseInterface` and serves as the primary entry point for all database operations inside osquery. - -It provides: - -- `getDatabaseValue` (string and integer variants) -- `setDatabaseValue` -- `setDatabaseBatch` -- `deleteDatabaseValue` -- `deleteDatabaseRange` -- `scanDatabaseKeys` - -Internally, it delegates all operations to free functions such as: - -- `getDatabaseValue(...)` -- `setDatabaseValue(...)` -- `scanDatabaseKeys(...)` - -These functions: - -- Acquire read/write locks to protect against concurrent resets. -- Route calls through the database plugin registry. -- Support both internal and external (extension-based) registry execution. - -This design ensures that the rest of the system depends only on the interface, not the concrete backend. - ---- - -### EphemeralDatabasePlugin - -**Component:** `osquery.osquery.database.ephemeral.EphemeralDatabasePlugin` - -The `EphemeralDatabasePlugin` is an in-memory implementation of the `DatabasePlugin` interface. - -It is: - -- Registered internally as the `ephemeral` database plugin. -- Used when the `disable_database` flag is enabled. -- Used for testing and fallback scenarios. - -#### Internal Data Structure - -```mermaid -flowchart TD - DB["EphemeralDatabasePlugin"] --> Map["std::map>"] - Map --> DomainMap["Domain Map"] - DomainMap --> KeyValue["boost::variant"] -``` - -Storage model: - -- Outer map: `domain β†’ domain_map` -- Inner map: `key β†’ boost::variant` - -Supported operations: - -- `get` -- `put` -- `putBatch` -- `remove` -- `removeRange` -- `scan` - -Since it is purely in-memory: - -- No persistence across restarts. -- No disk I/O. -- Ideal for unit testing and ephemeral deployments. - ---- - -## Database Domains - -The module defines several logical domains used throughout osquery: - -| Domain | Purpose | -|--------|----------| -| `configurations` | Persistent settings and metadata | -| `queries` | Scheduled query state and results metadata | -| `events` | Event framework buffers | -| `logs` | Buffered logging data | -| `carves` | File carving state | -| `distributed` | Distributed query tasks | -| `distributed_running` | Active distributed query tracking | -| `query_performance` | Performance metrics for queries | - -Domains act as logical namespaces within a single database backend. - ---- - -## Plugin Lifecycle and Initialization - -### Initialization Flow - -```mermaid -flowchart TD - Start["Startup"] --> CheckFlag{{"disable_database?"}} - CheckFlag -->|"No"| UseRocksDB["Activate RocksDB Plugin"] - CheckFlag -->|"Yes"| UseEphemeral["Activate Ephemeral Plugin"] - UseRocksDB --> InitDone["Database Initialized"] - UseEphemeral --> InitDone -``` - -Key behaviors: - -- If `disable_database` is false, the internal persistent plugin (RocksDB) is selected. -- If `disable_database` is true, the `ephemeral` plugin is activated. -- Initialization retries up to 25 times to handle file locks or shutdown races. -- `kDBInitialized` ensures safe access. - ---- - -## Concurrency and Reset Safety - -To protect database operations: - -- A global mutex `kDatabaseReset` is used. -- Read locks guard standard operations. -- Write locks guard reset flows. - -### Reset Flow - -```mermaid -flowchart TD - ResetCall["resetDatabase()"] --> LockWrite["Acquire Write Lock"] - LockWrite --> TearDown["Plugin tearDown()"] - TearDown --> SetUp["Plugin setUp()"] - SetUp --> Unlock["Release Lock"] -``` - -If reset fails: - -- The registry falls back to the `ephemeral` backend. -- A warning is logged. - -This prevents total system failure when persistent storage is corrupted. - ---- - -## Database Migrations - -The module supports schema/data migrations via version tracking. - -- Version key: `results_version` -- Stored under domain: `configurations` - -### Migration Flow - -```mermaid -flowchart TD - ReadVersion["Read results_version"] --> Compare["Compare with target version"] - Compare -->|"Older"| RunMigration["Execute migrateVxVy()"] - RunMigration --> UpdateVersion["Persist new version"] - UpdateVersion --> Compare - Compare -->|"Match"| Done["Migration Complete"] -``` - -Examples of migrations: - -- Converting legacy property-tree JSON to RapidJSON format. -- Renaming event publisher keys. - -If migration fails: - -- Error is logged. -- Database upgrade aborts. - ---- - -## Interaction with Other Modules - -### Configuration And Packs - -Configuration refreshers persist: - -- Pack execution counters. -- Schedule metadata. -- Decorator state. - -All stored under structured domains using `setDatabaseValue` and `scanDatabaseKeys`. - ---- - -### Logging And Query Observability - -This module persists: - -- Scheduled query state. -- Query performance metrics. -- Result differentials. - -The `query_performance` domain ensures metrics survive restarts when persistent storage is enabled. - ---- - -### Distributed Querying - -The distributed module uses domains: - -- `distributed` -- `distributed_running` - -These track: - -- Pending tasks. -- Active query execution state. - ---- - -### Eventing Framework And Subscriptions - -Event publishers and subscribers use the `events` domain to: - -- Persist event cursors. -- Maintain audit or publisher state. - ---- - -## Extension and External Registry Behavior - -If osquery is running as an **extension process**: - -- It does not directly access the database backend. -- Database requests are routed via `Registry::call("database", ...)`. -- The main daemon handles storage. - -This enforces: - -- Centralized persistence. -- No extension-level database corruption risk. - ---- - -## Key Design Characteristics - -### 1. Plugin-Based Storage - -- Storage backend selected via registry. -- Easily replaceable or disabled. -- Uniform request interface (`get`, `put`, `scan`, etc.). - -### 2. Domain Isolation - -Logical partitioning prevents key collisions across features. - -### 3. Migration Safety - -- Explicit version tracking. -- Incremental migration steps. -- Failure-safe version commit. - -### 4. Reset and Recovery Support - -- Safe reset workflow. -- Automatic fallback to ephemeral storage. -- Multi-attempt initialization. - -### 5. Thread Safety - -- Reader/writer locking around reset flows. -- Atomic flags guarding initialization state. - ---- - -## When to Use Each Backend - -| Backend | Use Case | -|----------|----------| -| RocksDB (Persistent) | Production deployments requiring durable state | -| Ephemeral | Testing, debugging, or stateless environments | - -Testing environments typically use: - -```text -initDatabasePluginForTesting() -``` - -Which: - -- Forces `disable_database = true` -- Activates ephemeral storage -- Resets the database - ---- - -## Summary - -The **Database And Storage Plugins** module provides the foundational storage abstraction for osquery. It: - -- Decouples storage implementation from core logic. -- Enables plugin-based extensibility. -- Protects against corruption via reset and migration controls. -- Supports both persistent and in-memory storage. - -Every major subsystem in osquery relies on this module for durable state, making it a critical backbone component of the overall architecture. \ No newline at end of file diff --git a/docs/reference/architecture/database-backend/database-backend.md b/docs/reference/architecture/database-backend/database-backend.md new file mode 100644 index 00000000000..cadb15f5aaa --- /dev/null +++ b/docs/reference/architecture/database-backend/database-backend.md @@ -0,0 +1,272 @@ +# Database Backend + +The **Database Backend** module provides the persistent and ephemeral key–value storage layer for osquery. It abstracts the underlying storage engine behind a pluggable interface and exposes a uniform API used by configuration, logging, distributed querying, and event subsystems. + +At runtime, the Database Backend is responsible for: + +- Selecting and initializing the active database plugin (e.g., RocksDB or Ephemeral) +- Providing domain-scoped key–value operations (get, put, remove, scan) +- Handling database resets and migrations +- Coordinating safe access through locking and lifecycle guards +- Offering a stable `IDatabaseInterface` for higher-level components + +--- + +## 1. Architectural Overview + +The Database Backend sits at the core of osquery’s state management. Other modules such as Configuration and Packs, Logging and Query Metadata, Distributed Querying, and Events rely on it for durable or transient storage. + +### High-Level Architecture + +```mermaid +flowchart TD + CoreRuntime["Core Runtime and Lifecycle"] --> DatabaseBackend["Database Backend"] + + DatabaseBackend --> Registry["Registry Factory"] + DatabaseBackend --> RocksDBPlugin["RocksDB Plugin"] + DatabaseBackend --> EphemeralPlugin["Ephemeral Database Plugin"] + + DatabaseBackend --> ConfigModule["Configuration and Packs"] + DatabaseBackend --> LoggingModule["Logging and Query Metadata"] + DatabaseBackend --> DistributedModule["Distributed Querying"] + DatabaseBackend --> EventsModule["Events Core and Subscriptions"] +``` + +### Key Concepts + +- **DatabasePlugin Registry**: A registry category named `database` allows pluggable storage engines. +- **Domains**: Logical namespaces (e.g., `queries`, `events`, `logs`) partition stored data. +- **Global Accessors**: Helper functions (e.g., `getDatabaseValue`, `setDatabaseValue`) provide thread-safe access. +- **Migration Framework**: Handles on-disk schema evolution between database versions. + +--- + +## 2. Core Components + +This module contains two primary components: + +- `osquery.osquery.database.database.OsqueryDatabase` +- `osquery.osquery.database.ephemeral.EphemeralDatabasePlugin` + +Together, they define the public database interface and a built-in in-memory implementation. + +--- + +## 3. DatabasePlugin Abstraction Layer + +The Database Backend uses a plugin-based architecture: + +- A `DatabasePlugin` registry is created under the name `database`. +- The active plugin is selected at initialization time. +- By default, the internal RocksDB implementation is used unless disabled. +- The Ephemeral plugin acts as a fallback or test implementation. + +### Initialization Flow + +```mermaid +flowchart TD + Start["Process Startup"] --> Init["initDatabasePlugin()"] + Init --> CheckDisable{"disable_database flag?"} + CheckDisable -->|"Yes"| UseEphemeral["Set Active Plugin: Ephemeral"] + CheckDisable -->|"No"| UseRocksDB["Set Active Plugin: RocksDB"] + UseEphemeral --> MarkInit["kDBInitialized = true"] + UseRocksDB --> MarkInit + MarkInit --> Ready["Database Ready"] +``` + +### Concurrency Controls + +- `kDatabaseReset` (reader/writer mutex): + - **Write lock** during reset operations + - **Read lock** during normal get/put operations +- Atomic flags: + - `kDBInitialized` + - `kDBAllowOpen` + - `kDBChecking` + +This ensures safe access during plugin switching, resets, and shutdown. + +--- + +## 4. Domains and Data Organization + +The Database Backend organizes state into well-known domains: + +- `configurations` +- `queries` +- `events` +- `logs` +- `carves` +- `distributed` +- `distributed_running` +- `query_performance` + +These domains are used by: + +- [Configuration and Packs](../configuration-and-packs/configuration-and-packs.md) +- [Logging and Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) +- [Distributed Querying](../distributed-querying/distributed-querying.md) +- [Events Core and Subscriptions](../events-core-and-subscriptions/events-core-and-subscriptions.md) + +Each domain is logically isolated but handled through the same plugin interface. + +--- + +## 5. Public Database API + +The Database Backend exposes global helper functions wrapping the active plugin: + +- `getDatabaseValue(domain, key, value)` +- `setDatabaseValue(domain, key, value)` +- `setDatabaseBatch(domain, data)` +- `deleteDatabaseValue(domain, key)` +- `deleteDatabaseRange(domain, low, high)` +- `scanDatabaseKeys(domain, keys, prefix, max)` + +### Internal Call Flow + +```mermaid +flowchart TD + Caller["Higher Level Module"] --> Helper["getDatabaseValue()"] + Helper --> Lock["Acquire Read Lock"] + Lock --> Plugin["Active DatabasePlugin"] + Plugin --> Result["Return Status and Value"] +``` + +If running inside an **extension context**, calls are routed through the registry instead of direct plugin access, since extensions do not host active database instances. + +--- + +## 6. OsqueryDatabase (IDatabaseInterface Implementation) + +`OsqueryDatabase` implements `IDatabaseInterface` and delegates all operations to the global helper functions. + +### Responsibilities + +- Provide a stable, dependency-injectable interface +- Abstract away registry and plugin details +- Support both string and integer value overloads + +### Delegation Pattern + +```mermaid +flowchart LR + Client["Component Using IDatabaseInterface"] --> Interface["IDatabaseInterface"] + Interface --> OsqueryDB["OsqueryDatabase"] + OsqueryDB --> GlobalHelpers["Global Database Helpers"] + GlobalHelpers --> ActivePlugin["Active DatabasePlugin"] +``` + +This ensures that higher-level components do not depend on specific plugin implementations. + +--- + +## 7. Ephemeral Database Plugin + +`EphemeralDatabasePlugin` is a built-in in-memory implementation of `DatabasePlugin`. + +### Storage Model + +```mermaid +flowchart TD + DB["Ephemeral DB"] --> DomainMap["Map<Domain, Map<Key, Value>>"] + DomainMap --> KeyValue["Key -> Variant(int or string)"] +``` + +### Characteristics + +- Backed by nested `std::map` containers +- Supports both `int` and `std::string` values via `boost::variant` +- No persistence across restarts +- Used for: + - Testing (`initDatabasePluginForTesting()`) + - Fallback when persistent storage fails + - Explicit `--disable_database` usage + +### Supported Operations + +- `get()` with type checking +- `put()` and `putBatch()` +- `remove()` +- `removeRange()` with lexicographic bounds +- `scan()` with prefix filtering and optional maximum + +The Ephemeral plugin is registered internally under the name `ephemeral`. + +--- + +## 8. Reset, Dump, and Lifecycle + +### Reset Flow + +```mermaid +flowchart TD + ResetCall["resetDatabase()"] --> RegistryCall["Registry::call action=reset"] + RegistryCall --> WriteLock["Acquire Write Lock"] + WriteLock --> TearDown["Plugin tearDown()"] + TearDown --> SetUp["Plugin setUp()"] + SetUp --> Done["Database Reinitialized"] +``` + +If a reset fails, the system falls back to the Ephemeral plugin. + +### Dumping Contents + +When the `database_dump` flag is enabled, all domains are scanned and printed to stdout for debugging. + +### Shutdown + +`shutdownDatabase()` removes all registered database plugins from the registry, ensuring a clean termination. + +--- + +## 9. Database Versioning and Migration + +The Database Backend supports schema evolution via a version key stored in the `configurations` domain. + +- Version key: `results_version` +- Migration path examples: + - V0 β†’ V1: Convert legacy property tree JSON to RapidJSON + - V1 β†’ V2: Rename specific event keys + +### Migration Loop + +```mermaid +flowchart TD + Start["upgradeDatabase(target)"] --> ReadVersion["Read results_version"] + ReadVersion --> Compare{"Current != Target?"} + Compare -->|"Yes"| Migrate["Run Migration Step"] + Migrate --> UpdateVersion["Persist New Version"] + UpdateVersion --> Compare + Compare -->|"No"| Done["Upgrade Complete"] +``` + +Migration failures abort the upgrade and return an error status. + +--- + +## 10. Interaction with Other Modules + +The Database Backend underpins several subsystems: + +- **Configuration and Packs**: Stores persistent configuration state and metadata. +- **Logging and Query Metadata**: Persists query results, scheduled query counters, and performance metrics. +- **Distributed Querying**: Tracks distributed query tasks and results. +- **Events Core and Subscriptions**: Maintains event buffers and publisher state. + +It also integrates with: + +- **Core Runtime and Lifecycle** for startup/shutdown orchestration. +- **Extensions Framework** for registry-based request forwarding in extension mode. + +--- + +## 11. Design Strengths + +- βœ… Pluggable storage engines +- βœ… Domain-based logical isolation +- βœ… Thread-safe reset and access model +- βœ… Backward-compatible migration framework +- βœ… Extension-aware routing logic + +The Database Backend provides a stable foundation for osquery’s persistent and transient state, enabling modular growth while preserving strong encapsulation and operational safety. diff --git a/docs/reference/architecture/distributed-querying/distributed-querying.md b/docs/reference/architecture/distributed-querying/distributed-querying.md index 37c3bf57f6c..fdbfa114ff2 100644 --- a/docs/reference/architecture/distributed-querying/distributed-querying.md +++ b/docs/reference/architecture/distributed-querying/distributed-querying.md @@ -1,376 +1,400 @@ # Distributed Querying -The **Distributed Querying** module enables remote orchestration and execution of SQL queries across a fleet of osquery nodes. It allows a central server to: +The **Distributed Querying** module enables osquery nodes to receive SQL queries from a remote service, execute them locally, and return structured results back to the requester. It acts as the orchestration layer between remote query providers (via distributed plugins), the SQL engine, the database backend, and the logging and performance subsystems. -- Push SQL queries to enrolled agents -- Collect query results asynchronously -- Monitor execution performance -- Denylist problematic or long-running queries - -This module acts as the execution bridge between: - -- The **SQL Engine and Virtual Tables** subsystem (local query execution) -- The **Database and Storage Plugins** subsystem (state management) -- The **Logging and Query Observability** subsystem (performance + result logging) -- Remote transport implementations such as the TLS-based plugin - -At its core, Distributed Querying defines the request/response model, execution lifecycle, denylist logic, and plugin interface for retrieving and submitting distributed work. +Unlike scheduled queries defined in configuration, distributed queries are dynamic and externally driven. This module manages their lifecycle, execution state, denylisting, performance tracking, and result serialization. --- -## Architectural Overview +## 1. Purpose and Responsibilities -```mermaid -flowchart TD - Server["Central Server"] -->|"HTTPS"| TLSPlugin["TLS Distributed Plugin"] - TLSPlugin -->|"getQueries()"| DistributedCore["Distributed Core"] - DistributedCore -->|"runQueries()"| SQLEngine["SQL Engine"] - SQLEngine -->|"QueryData"| DistributedCore - DistributedCore -->|"writeResults()"| TLSPlugin - TLSPlugin -->|"HTTPS"| Server - - DistributedCore -->|"recordQueryPerformance"| Observability["Query Observability"] - DistributedCore -->|"running state"| Storage["Database Plugins"] -``` +The Distributed Querying module is responsible for: + +- Pulling pending queries from a remote distributed plugin +- Parsing and validating incoming work +- Enqueuing and executing queries through the SQL engine +- Preventing duplicate or runaway execution via denylisting +- Tracking performance metrics per query +- Serializing and batching results for remote delivery +- Cleaning up expired or stale running queries -### Key Responsibilities +It operates as a bridge between: -| Component | Responsibility | -|------------|----------------| -| Distributed Core | Lifecycle, queuing, denylisting, performance tracking | -| Distributed Plugin | Abstract interface for remote work retrieval and result submission | -| TLS Distributed Plugin | HTTPS implementation of Distributed Plugin | -| SQL Engine | Executes queries against virtual tables | -| Database Plugins | Persist running state and deduplication | -| Observability | Tracks performance and execution metrics | +- The Extensions Framework (plugin abstraction) +- The SQL Engine and Virtual Tables +- The Database Backend (state tracking) +- Logging and Query Metadata (performance and result representation) --- -# Core Data Structures +## 2. Core Data Structures -## DistributedQueryRequest +### 2.1 DistributedQueryRequest -Represents a single unit of distributed work. +Represents a single distributed query instruction. ```text Fields: -- id: Unique server-assigned identifier -- query: SQL string to execute +- query : SQL string to execute +- id : Unique identifier assigned by remote service ``` -This structure is serialized/deserialized using: +This structure is: -- `serializeDistributedQueryRequest` -- `deserializeDistributedQueryRequest` -- JSON string helpers for transport safety +- Deserialized from plugin JSON payloads +- Stored internally before execution +- Attached to results for traceability -### Example Server Payload +Serialization and deserialization helpers: -```json -{ - "queries": { - "id1": "select * from osquery_info", - "id2": "select * from processes" - } -} -``` +- serializeDistributedQueryRequest +- serializeDistributedQueryRequestJSON +- deserializeDistributedQueryRequest +- deserializeDistributedQueryRequestJSON -Each key becomes a `DistributedQueryRequest` instance. +These ensure consistent JSON transport between node and server. --- -## DistributedQueryResult +### 2.2 DistributedQueryResult -Encapsulates execution output and metadata. +Represents the outcome of executing a distributed query. ```text Fields: -- request: Original DistributedQueryRequest -- results: QueryData (rows) -- columns: ColumnNames -- status: Execution Status -- message: Optional error or informational message +- request : Original DistributedQueryRequest +- results : QueryData (rows) +- columns : ColumnNames +- status : Execution Status +- message : Additional status or error message ``` -Serialized using: +This structure: -- `serializeDistributedQueryResult` -- `deserializeDistributedQueryResult` +- Captures execution output +- Includes metadata required by the remote server +- Is serialized before transmission -This object ensures transport-neutral packaging of results. +Serialization helpers mirror those for requests: ---- +- serializeDistributedQueryResult +- serializeDistributedQueryResultJSON +- deserializeDistributedQueryResult +- deserializeDistributedQueryResultJSON -# Distributed Execution Lifecycle +--- -The `Distributed` class orchestrates the runtime behavior. +## 3. High-Level Architecture ```mermaid flowchart TD - Start["Pull Updates"] --> Accept["acceptWork()"] - Accept --> Pending{"Pending Queries?"} - Pending -->|"Yes"| Run["runQueries()"] - Run --> Deny{"Denylisted?"} - Deny -->|"No"| Execute["Execute via SQL Engine"] - Execute --> Record["recordQueryPerformance()"] - Record --> Queue["addResult()"] - Queue --> Flush["flushCompleted()"] - Deny -->|"Yes"| Skip["Skip Execution"] - Flush --> End["Cycle Complete"] + RemoteService["Remote Service"] --> DistributedPlugin["Distributed Plugin"] + DistributedPlugin --> DistributedManager["Distributed"] + DistributedManager --> SQLExecutor["SQL Engine"] + SQLExecutor --> VirtualTables["Virtual Tables"] + DistributedManager --> DatabaseState["Database Backend"] + DistributedManager --> PerformanceTracker["Query Performance"] + DistributedManager --> ResultSerializer["Result Serialization"] + ResultSerializer --> DistributedPlugin ``` +### Components + +- Distributed Plugin: Provides transport abstraction (TLS or custom plugin). +- Distributed: Core orchestration class. +- SQL Engine: Executes SQL against virtual tables. +- Database Backend: Tracks running and denylisted queries. +- Query Performance: Stores execution metrics. + --- -## 1. Pull Phase +## 4. DistributedPlugin Abstraction -```cpp -Status pullUpdates(); -``` +`DistributedPlugin` extends the generic Plugin interface and defines two core operations: -- Calls the active `DistributedPlugin` -- Retrieves raw JSON work payload -- Delegates parsing to `acceptWork()` +### 4.1 getQueries ---- +Returns JSON formatted work: -## 2. Work Acceptance +```text +{ + "queries": { + "id1": "select * from osquery_info", + "id2": "select * from processes" + } +} +``` + +### 4.2 writeResults + +Accepts JSON formatted results: -```cpp -Status acceptWork(const std::string& work); +```text +{ + "queries": { + "id1": [ + {"col1": "val1"} + ] + } +} ``` -Behavior: +The plugin transport may use TLS, HTTP, or another extension-based mechanism. The Distributed Querying module is transport-agnostic. + +--- + +## 5. Distributed Class Lifecycle + +The `Distributed` class manages the full query lifecycle. -- Parses `queries` section -- Evaluates optional `discovery` queries -- Enqueues only valid work +### 5.1 Continuous Execution Loop -### Discovery Query Logic +```mermaid +flowchart TD + Start["Start Loop"] --> Pull["pullUpdates()"] + Pull --> Check["Pending Queries?"] + Check -->|"Yes"| Run["runQueries()"] + Check -->|"No"| Sleep["Wait Interval"] + Run --> Flush["flushCompleted()"] + Flush --> Sleep + Sleep --> Pull +``` -- If no discovery query exists β†’ enqueue -- If discovery returns rows β†’ enqueue -- If discovery returns zero rows β†’ skip +Core public methods: -This prevents unnecessary execution on nodes where the query is irrelevant. +- pullUpdates +- getPendingQueries +- runQueries +- serializeResults +- getCompletedCount +- cleanupExpiredRunningQueries --- -## 3. Denylisting Protection +## 6. Work Acceptance and Discovery Logic -To prevent repeated execution of problematic queries, the module includes: +Incoming work may include: -- `checkAndSetAsRunning()` -- `setAsNotRunning()` -- `denylistedQueryTimestampExpired()` -- `denylistDuration()` -- `hashQuery()` +- Direct queries +- Discovery queries -### Denylist Mechanism +Discovery queries determine whether a corresponding query should run. + +### Discovery Behavior ```mermaid flowchart TD - Incoming["Incoming Query"] --> Check["checkAndSetAsRunning()"] - Check --> Running{"Already Running?"} - Running -->|"Within Duration"| Block["Denylisted"] - Running -->|"Expired"| Allow["Allow Execution"] - Check -->|"First Time"| Allow + ReceiveWork["acceptWork()"] --> HasDiscovery["Has Discovery Query?"] + HasDiscovery -->|"No"| Enqueue["Enqueue Query"] + HasDiscovery -->|"Yes"| RunDiscovery["Execute Discovery"] + RunDiscovery --> DiscoveryResult["Rows Returned?"] + DiscoveryResult -->|"Yes"| Enqueue + DiscoveryResult -->|"No"| Skip["Skip Query"] ``` -The denylist key is derived from: +Rules: -```text -SHA256(query) -``` +- No discovery query β†’ enqueue directly +- Discovery returns rows β†’ enqueue +- Discovery returns no rows β†’ skip -This ensures consistent identification across restarts and transports. +This mechanism enables conditional execution across distributed fleets. --- -## 4. Query Execution +## 7. Execution and Denylisting -```cpp -Status runQueries(); -``` +To prevent duplicate or runaway queries, Distributed Querying implements a denylisting mechanism. + +### 7.1 Running State Tracking -For each pending request: +Methods: -1. Pop request from internal storage -2. Execute via SQL engine -3. Measure execution time -4. Record performance -5. Store result -6. Clear running state +- checkAndSetAsRunning +- setAsNotRunning +- setKeyAsNotRunning +- cleanupExpiredRunningQueries -Execution uses: +### 7.2 Denylist Flow -```cpp -SQL monitorNonnumeric(const std::string& name, const std::string& query); +```mermaid +flowchart TD + Execute["runQueries()"] --> CheckRun["checkAndSetAsRunning"] + CheckRun -->|"Already Running"| WithinWindow["Within Denylist Duration?"] + WithinWindow -->|"Yes"| Skip["Skip Execution"] + WithinWindow -->|"No"| ExecuteQuery["Execute Query"] + CheckRun -->|"Not Running"| ExecuteQuery + ExecuteQuery --> Complete["setAsNotRunning"] ``` ---- +Supporting utilities: -## 5. Performance Recording +- hashQuery: SHA256 of query string +- denylistedQueryTimestampExpired +- denylistDuration -Performance metrics are stored in: +The hash is used as a stable key to track running queries across restarts and flush cycles. -```text -std::map performance_ -``` +--- -Tracked metrics include: +## 8. Query Execution and Performance Monitoring -- Execution duration -- Row count -- Sample process table deltas +Queries are executed through the SQL engine using: -These integrate with the **Logging and Query Observability** module. +- monitorNonnumeric +- recordQueryPerformance ---- +Performance metrics stored per query include: -## 6. Result Flushing +- Execution time in milliseconds +- Number of rows returned +- Differential row comparison (for sampled tables) -```cpp -Status flushCompleted(); +```mermaid +flowchart TD + RunQuery["Execute SQL"] --> Measure["Measure Duration"] + Measure --> CountRows["Count Result Rows"] + CountRows --> Record["recordQueryPerformance"] + Record --> Store["Performance Map"] ``` -- Serializes accumulated `DistributedQueryResult` objects -- Sends them to the plugin via `writeResults()` -- Clears local result queue +The performance data integrates with the broader query metadata subsystem and can influence operational visibility and diagnostics. --- -# Plugin Interface +## 9. Result Collection and Flushing + +Executed query results are appended to an internal vector: -## DistributedPlugin (Abstract) +```text +std::vector results_ +``` -Defines the extension point for remote orchestration. +### Result Handling Flow -```cpp -virtual Status getQueries(std::string& json) = 0; -virtual Status writeResults(const std::string& json) = 0; +```mermaid +flowchart TD + AddResult["addResult()"] --> Buffer["results_ Vector"] + Buffer --> FlushCall["flushCompleted()"] + FlushCall --> Serialize["serializeResults()"] + Serialize --> PluginWrite["DistributedPlugin.writeResults()"] + PluginWrite --> Clear["Clear results_"] ``` -The `call()` method acts as the unified entry point for plugin requests. +Key behavior: -Any transport (TLS, filesystem, custom RPC) can implement this interface. +- Results are batched +- JSON is constructed via serialization helpers +- Buffer is cleared after successful flush --- -# TLS Distributed Plugin - -The **TLS Distributed Plugin** provides an HTTPS implementation of the plugin interface. +## 10. Current Request Tracking -Registered as: +A static member tracks the ID of the currently executing distributed query: ```text -REGISTER(TLSDistributedPlugin, "distributed", "tls") +static std::string currentRequestId_ ``` -## Configuration Flags +Accessors: -```text ---distributed_tls_read_endpoint ---distributed_tls_write_endpoint ---distributed_tls_max_attempts -``` +- getCurrentRequestId +- setCurrentRequestId -## Setup Phase +This enables: -```cpp -Status setUp(); -``` +- Correlation with logs +- Traceability in error handling +- Integration with logging and metadata subsystems -- Builds read/write URIs -- Uses TLS request helper utilities +--- -## Query Retrieval +## 11. Interaction with Other Modules -```cpp -Status getQueries(std::string& json); -``` +Distributed Querying interacts closely with several major subsystems: -- Sends POST to read endpoint -- Returns JSON payload containing work +### SQL Engine and Virtual Tables -## Result Submission +- Executes distributed SQL statements +- Retrieves rows from virtual tables +- Uses SQL abstraction for safe execution -```cpp -Status writeResults(const std::string& json); -``` +### Database Backend -- Parses JSON -- POSTs to write endpoint -- Ignores server response body +- Tracks running query keys +- Maintains denylist timestamps +- Persists state across iterations ---- +### Logging and Query Metadata -## TLS Data Flow +- Records performance metrics +- Associates status and message fields with results +- Enables operational insight -```mermaid -sequenceDiagram - participant Node - participant TLS as "TLS Distributed Plugin" - participant Server - - Node->>TLS: pullUpdates() - TLS->>Server: POST read endpoint - Server->>TLS: JSON queries - TLS->>Node: return JSON - Node->>TLS: writeResults(JSON) - TLS->>Server: POST write endpoint -``` +### Extensions Framework + +- Provides Plugin abstraction +- Allows pluggable distributed transports +- Supports external TLS-based or custom implementations --- -# State Management +## 12. End-to-End Data Flow -Distributed Querying relies on persistent state to: +```mermaid +flowchart TD + Server["Remote Server"] --> PluginPull["getQueries()"] + PluginPull --> Accept["acceptWork()"] + Accept --> Queue["Internal Queue"] + Queue --> Execute["runQueries()"] + Execute --> SQL["SQL Engine"] + SQL --> Results["DistributedQueryResult"] + Results --> Buffer["results_"] + Buffer --> Flush["flushCompleted()"] + Flush --> PluginPush["writeResults()"] + PluginPush --> Server +``` -- Track running queries -- Store denylist timestamps -- Maintain queued work +This flow illustrates the complete lifecycle: -This state is handled through the Database and Storage Plugins subsystem. +1. Remote server issues queries +2. Node pulls and parses work +3. Queries are conditionally enqueued +4. SQL execution occurs +5. Results are serialized +6. Results are flushed back to server --- -# Security Model - -Key security properties: +## 13. Design Characteristics -1. Query integrity validated via SHA-256 hashing -2. HTTPS transport in TLS plugin -3. Configurable retry attempts -4. Denylist prevents execution storms +### Transport Agnostic -The denylist duration is configurable via runtime flags. +The module does not embed networking logic. All communication is delegated to DistributedPlugin implementations. ---- +### Safe Execution -# Integration Points +- Denylisting prevents duplicate execution storms +- Hash-based tracking ensures stable identity +- Expired running queries are cleaned automatically -Distributed Querying integrates with: +### Observability-Oriented -- SQL Engine and Virtual Tables (local execution) -- Database and Storage Plugins (persistent state) -- Logging and Query Observability (performance tracking) -- Extensions and IPC (optional distributed plugin implementations) -- Remote HTTP Client (transport layer headers and networking) +- Query performance metrics are recorded +- Status and error messages are preserved +- Current request ID enables traceability -Sibling module reference: +### Scalable by Design -- [Remote HTTP Client](../remote-http-client/remote-http-client.md) +- Pull-based model supports fleet scaling +- Batched result submission reduces overhead +- Discovery queries minimize unnecessary work --- # Summary -The **Distributed Querying** module provides: - -- A structured request/response model -- Pluggable transport abstraction -- Controlled query lifecycle management -- Built-in denylisting and performance tracking -- Secure TLS-based remote orchestration +The **Distributed Querying** module orchestrates remote query execution across osquery nodes. It abstracts transport through plugins, enforces safe execution with denylisting, integrates tightly with the SQL engine and database backend, and provides structured, serialized results enriched with performance metadata. -It transforms osquery from a local query engine into a centrally orchestrated distributed telemetry system while preserving execution safety and observability guarantees. +It is the core execution engine for dynamic, centrally managed queries in distributed deployments. \ No newline at end of file diff --git a/docs/reference/architecture/distributed-tls-plugin/distributed-tls-plugin.md b/docs/reference/architecture/distributed-tls-plugin/distributed-tls-plugin.md new file mode 100644 index 00000000000..0f8f43f50c6 --- /dev/null +++ b/docs/reference/architecture/distributed-tls-plugin/distributed-tls-plugin.md @@ -0,0 +1,287 @@ +# Distributed Tls Plugin + +The **Distributed Tls Plugin** module implements the TLS-based distributed query transport for osquery. It enables secure retrieval of distributed queries from a remote service and submission of query results back to that service over HTTPS. + +This module provides a concrete implementation of the distributed plugin interface and acts as the secure bridge between the local distributed query engine and a remote TLS endpoint. + +--- + +## Purpose and Responsibilities + +The Distributed Tls Plugin is responsible for: + +- Retrieving distributed queries from a remote HTTPS endpoint +- Submitting distributed query results back to a remote HTTPS endpoint +- Handling retry logic for network communication +- Serializing and deserializing JSON payloads +- Integrating with the enrollment and TLS transport stack + +It extends the distributed query abstraction defined in the [Distributed Querying](distributed-querying/distributed-querying.md) module and relies on the TLS transport utilities provided by the remote HTTP layer. + +--- + +## Core Component + +### TLSDistributedPlugin + +**Namespace:** `osquery.plugins.distributed.tls_distributed` + +The `TLSDistributedPlugin` class derives from `DistributedPlugin` and is registered under the name: + +- Registry type: `distributed` +- Plugin name: `tls` + +This makes it selectable as the distributed backend when TLS-based communication is desired. + +### Key Methods + +- `setUp()` – Initializes TLS endpoints +- `getQueries(std::string& json)` – Fetches distributed queries from remote server +- `writeResults(const std::string& json)` – Submits query results to remote server + +--- + +## Configuration Flags + +The Distributed Tls Plugin is configured using runtime flags: + +- `distributed_tls_read_endpoint` – HTTPS endpoint for retrieving queries +- `distributed_tls_write_endpoint` – HTTPS endpoint for submitting results +- `distributed_tls_max_attempts` – Maximum retry attempts for requests +- `tls_node_api` – Enables node-specific TLS behavior + +These flags are part of the runtime configuration system described in the Core Runtime and Lifecycle module. + +--- + +## Architectural Position + +The Distributed Tls Plugin sits between the distributed query engine and the remote HTTPS service. + +```mermaid +flowchart LR + DistributedEngine["Distributed Query Engine"] -->|"getQueries()"| TLSPlugin["Distributed Tls Plugin"] + TLSPlugin -->|"HTTPS POST"| RemoteServer["Remote TLS Server"] + RemoteServer -->|"JSON Queries"| TLSPlugin + TLSPlugin -->|"writeResults()"| DistributedEngine +``` + +### Relationships + +- **Upstream Dependency:** [Distributed Querying](distributed-querying/distributed-querying.md) +- **Transport Layer:** Remote HTTP client and TLS request utilities +- **Serialization:** JSON serializer +- **Enrollment:** Integrates with node identity and enrollment utilities + +--- + +## Internal Flow + +### Setup Phase + +During initialization: + +1. Read endpoint flags are retrieved +2. URIs are constructed using TLS helpers +3. Internal read and write URIs are stored + +```mermaid +flowchart TD + Start["Start setUp()"] --> ReadFlag["Read distributed_tls_read_endpoint"] + ReadFlag --> MakeReadURI["TLSRequestHelper makeURI()"] + MakeReadURI --> WriteFlag["Read distributed_tls_write_endpoint"] + WriteFlag --> MakeWriteURI["TLSRequestHelper makeURI()"] + MakeWriteURI --> StoreURIs["Store read_uri and write_uri"] + StoreURIs --> EndNode["END"] +``` + +--- + +### Query Retrieval Flow + +The `getQueries()` method performs a POST request to retrieve distributed queries. + +```mermaid +sequenceDiagram + participant Engine + participant TLSPlugin + participant TLSTransport + participant Remote + Engine->>TLSPlugin: getQueries() + TLSPlugin->>TLSTransport: go(JSONSerializer) + TLSTransport->>Remote: HTTPS POST read_uri + Remote-->>TLSTransport: JSON query payload + TLSTransport-->>TLSPlugin: Serialized JSON + TLSPlugin-->>Engine: Query JSON +``` + +### Key Characteristics + +- Always uses HTTP POST +- Serializes parameters using JSON +- Retries up to configured maximum attempts +- Returns raw JSON payload to distributed engine + +--- + +### Result Submission Flow + +The `writeResults()` method submits distributed query results. + +```mermaid +flowchart TD + StartWrite["Start writeResults()"] --> ParseJSON["Parse input JSON"] + ParseJSON --> ValidCheck{{"Valid JSON?"}} + ValidCheck -->|"No"| ReturnError["Return error Status"] + ValidCheck -->|"Yes"| SendTLS["TLSRequestHelper go()"] + SendTLS --> IgnoreResp["Ignore server response"] + IgnoreResp --> EndWrite["END"] +``` + +### Important Behavior + +- Validates JSON before transmission +- Uses the write endpoint +- Ignores server response body +- Returns status of TLS request operation + +--- + +## Interaction with Distributed Querying Module + +The Distributed Tls Plugin implements the transport layer for the distributed query lifecycle: + +```mermaid +flowchart LR + Scheduler["Distributed Scheduler"] --> Request["DistributedQueryRequest"] + Request --> TLSPlugin["Distributed Tls Plugin"] + TLSPlugin --> Result["DistributedQueryResult"] + Result --> Scheduler +``` + +The data structures used by this plugin originate from the [Distributed Querying](distributed-querying/distributed-querying.md) module: + +- `DistributedQueryRequest` +- `DistributedQueryResult` + +The plugin itself does not interpret SQL; it only transports JSON payloads. + +--- + +## Retry and Error Handling + +The plugin delegates network execution to `TLSRequestHelper::go()` which: + +- Performs HTTPS communication +- Applies retry logic +- Uses the configured maximum attempt count +- Returns a `Status` object + +```mermaid +flowchart TD + StartRetry["Start TLS request"] --> Attempt["Attempt HTTPS call"] + Attempt --> SuccessCheck{{"Success?"}} + SuccessCheck -->|"Yes"| ReturnOK["Return Status OK"] + SuccessCheck -->|"No"| RetryCheck{{"Attempts left?"}} + RetryCheck -->|"Yes"| Attempt + RetryCheck -->|"No"| ReturnFail["Return Status Error"] +``` + +This separation ensures: + +- Clean transport abstraction +- Centralized TLS handling +- Consistent retry semantics across plugins + +--- + +## Security Considerations + +The Distributed Tls Plugin relies on: + +- HTTPS transport +- TLS certificate validation +- Enrollment identity +- Secure JSON serialization + +Security is enforced at the TLS transport layer rather than inside the plugin itself. + +Key security characteristics: + +- No plaintext query transport +- No local storage of distributed results +- No custom cryptographic implementation +- Delegation to hardened TLS transport utilities + +--- + +## Registry Integration + +The plugin is registered using the osquery registry system: + +- Registry category: `distributed` +- Plugin name: `tls` + +This allows dynamic selection of distributed backends at runtime. + +```mermaid +flowchart LR + Registry["Registry"] --> DistributedCategory["distributed Category"] + DistributedCategory --> TLSPlugin["tls Plugin"] +``` + +The registry framework is defined in the broader core runtime and extension infrastructure. + +--- + +## Data Contracts + +The plugin communicates exclusively via JSON. + +### Incoming (Query Retrieval) + +- JSON payload containing distributed queries +- Delivered as raw JSON string to the distributed engine + +### Outgoing (Result Submission) + +- JSON representation of distributed query results +- Must parse successfully before transmission + +No SQL parsing, validation, or transformation occurs inside this module. + +--- + +## Integration with Other Modules + +The Distributed Tls Plugin integrates with: + +- [Distributed Querying](distributed-querying/distributed-querying.md) – Defines distributed query structures and lifecycle +- Remote HTTP client utilities – Provides TLS transport and request helpers +- Core runtime flags system – Supplies configuration values + +It does not: + +- Execute SQL (handled by SQL engine modules) +- Persist results (handled by database backend modules) +- Schedule queries (handled by distributed engine and scheduler) + +--- + +## Design Principles + +The Distributed Tls Plugin follows several architectural principles: + +1. **Single Responsibility** – Only handles TLS-based distributed transport +2. **Transport Abstraction** – Delegates networking to TLS helpers +3. **Stateless Operation** – No persistent internal state beyond URIs +4. **Retry Delegation** – Centralized retry logic in transport layer +5. **JSON Boundary** – Uses JSON as strict interface contract + +--- + +## Summary + +The **Distributed Tls Plugin** is the secure transport implementation for distributed queries over HTTPS. It connects the distributed query engine to a remote control plane, handling retrieval and submission of query data through TLS-protected endpoints. + +By isolating network communication from query execution and scheduling logic, it maintains a clean separation of concerns while enabling secure, scalable distributed query workflows. \ No newline at end of file diff --git a/docs/reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md b/docs/reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md deleted file mode 100644 index a18e9b822dc..00000000000 --- a/docs/reference/architecture/eventing-framework-and-subscriptions/eventing-framework-and-subscriptions.md +++ /dev/null @@ -1,363 +0,0 @@ -# Eventing Framework And Subscriptions - -## Overview - -The **Eventing Framework And Subscriptions** module provides the core publish/subscribe infrastructure for osquery’s event-driven data model. It enables: - -- Event publishers to monitor operating system activity (e.g., file changes, process activity). -- Event subscribers to register interest in specific event types. -- Persistent storage of event data for later retrieval via virtual tables. -- Automatic lifecycle, expiration, and optimization of event data based on query schedules. - -This module acts as the bridge between: - -- The SQL Engine And Virtual Tables module (which exposes event tables to queries). -- The Database And Storage Plugins module (which persists event data). -- The Configuration And Packs module (which defines scheduled queries and event usage). - ---- - -## High-Level Architecture - -```mermaid -flowchart TD - Config["Configuration And Packs"] -->|"scheduled queries"| EventFactory["EventFactory"] - EventFactory -->|"registers"| Publisher["EventPublisherPlugin"] - EventFactory -->|"registers"| Subscriber["EventSubscriberPlugin"] - Subscriber -->|"creates"| SubscriptionObj["Subscription"] - Publisher -->|"dispatches"| Callback["EventCallback"] - Callback -->|"addBatch()"| Storage["Database And Storage Plugins"] - SQL["SQL Engine And Virtual Tables"] -->|"SELECT from _events tables"| Subscriber - Subscriber -->|"generateRows()"| SQL -``` - -### Core Roles - -- **EventFactory** – Central coordinator for publishers and subscribers. -- **EventPublisherPlugin** – Produces system events. -- **EventSubscriberPlugin** – Consumes events and exposes them as queryable tables. -- **Subscription** – Binds subscriber logic to publisher events. -- **PathSet utilities** – Optimized path pattern matching for filesystem-related subscriptions. - ---- - -## EventFactory - -The `EventFactory` is a singleton responsible for managing the lifecycle and coordination of all event publishers and subscribers. - -### Responsibilities - -- Registering and deregistering event publishers and subscribers. -- Creating and forwarding subscriptions. -- Spawning publisher run-loop threads. -- Coordinating shutdown and cleanup. -- Applying configuration updates (e.g., scheduled query analysis). - -### Publisher and Subscriber Registration - -```mermaid -flowchart TD - RegisterSub["registerEventSubscriber()"] --> SetupSub["setUp()"] - SetupSub --> InitSub["init()"] - InitSub --> RunningSub["EVENT_RUNNING"] - - RegisterPub["registerEventPublisher()"] --> SetupPub["setUp()"] - SetupPub --> ReadyPub["EVENT_SETUP"] -``` - -During registration: - -- Subscribers are optionally enabled/disabled via configuration (`events.enable_subscribers`, `events.disable_subscribers`). -- Publishers are validated by type and initialized if events are enabled. -- State transitions are enforced (`EVENT_NONE`, `EVENT_SETUP`, `EVENT_RUNNING`, `EVENT_PAUSED`, `EVENT_FAILED`). - -### Publisher Execution Model - -Each publisher runs in its own thread: - -```mermaid -flowchart TD - Delay["delay()"] --> ThreadSpawn["spawn thread per publisher"] - ThreadSpawn --> RunLoop["run(type_id)"] - RunLoop --> PublisherRun["publisher->run()"] - PublisherRun --> Pause["pause(200ms)"] - Pause --> RunLoop - RunLoop -->|"isEnding()"| TearDown["tearDown()"] -``` - -Key behaviors: - -- The factory prevents publisher restarts once started. -- A default cool-off pause avoids CPU thrashing. -- Shutdown via `end()` gracefully deregisters publishers and joins threads. - ---- - -## Subscription Model - -The `Subscription` structure binds a subscriber to a publisher. - -### Subscription Structure - -```text -Subscription - β”œβ”€β”€ subscriber_name - β”œβ”€β”€ context (SubscriptionContextRef) - └── callback (EventCallback) -``` - -Defined in: - -- `osquery.osquery.events.subscription.Subscription` - -### EventCallback - -```text -Status(const EventContextRef&, const SubscriptionContextRef&) -``` - -When an event fires: - -1. The publisher evaluates matching subscriptions. -2. The associated callback executes. -3. The subscriber stores structured rows using `addBatch()`. - -Subscriptions are usually created via: - -```cpp -EventFactory::addSubscription("PublisherType", subscription); -``` - ---- - -## EventSubscriberPlugin - -Defined in: - -- `osquery.osquery.events.eventsubscriberplugin.Context` -- `osquery.osquery.events.eventsubscriberplugin.GenerateRowsResult` - -The `EventSubscriberPlugin` class encapsulates event storage, indexing, expiration, and query-time retrieval. - -### Core Responsibilities - -- Registering subscriptions in `init()`. -- Storing events using `addBatch()`. -- Managing time-based indexing and expiration. -- Exposing events as virtual tables via `genTable()`. - -### Internal Context - -```text -Context - β”œβ”€β”€ database_namespace - β”œβ”€β”€ event_index - β”œβ”€β”€ last_query_time - └── last_event_id -``` - -The context ensures: - -- Namespace isolation between subscribers. -- Efficient indexing by time and event identifier. -- Thread-safe updates using mutexes. - -### Event Storage and Expiration - -```mermaid -flowchart TD - Callback["EventCallback"] --> AddBatch["addBatch()"] - AddBatch --> AssignID["generateEventIdentifier()"] - AssignID --> Persist["Database write"] - Persist --> IndexUpdate["update time index"] - - ExpireCheck["expireEventBatches()"] --> RemoveOld["delete expired batches"] -``` - -Expiration behavior is governed by: - -- `getEventsExpiry()` – Default from flags. -- `getEventBatchesMax()` – Max retained batches. -- `setMinExpiry()` – Adjusted dynamically based on scheduled query intervals. - -### Query-Time Retrieval - -When a query selects from an event table: - -```mermaid -flowchart TD - SQLQuery["SELECT * FROM example_events"] --> GenTable["genTable()"] - GenTable --> GenerateRows["generateRows()"] - GenerateRows --> FilterTime["apply start/stop window"] - FilterTime --> YieldRows["yield Row to SQL engine"] -``` - -The `GenerateRowsResult` structure tracks: - -- Whether the scan reached the end. -- The last processed timestamp. -- The last processed event identifier. - -This enables optimized incremental queries. - ---- - -## Configuration-Driven Expiration Logic - -One of the most advanced responsibilities of this module is aligning event retention with scheduled queries. - -### SubscriberExpirationDetails - -Defined in: - -- `osquery.osquery.events.eventfactory.SubscriberExpirationDetails` - -Structure: - -```text -SubscriberExpirationDetails - β”œβ”€β”€ max_interval - └── query_count -``` - -During `configUpdate()`: - -1. The schedule is scanned for queries referencing `_events` tables. -2. For each subscriber table: - - The maximum interval is recorded. - - The number of queries referencing it is counted. -3. The minimum expiration window is set to: - -```text -min_expiry = (max_interval * 3), rounded to next minute -``` - -This ensures: - -- Events are retained long enough for scheduled queries to observe them. -- Storage is not retained unnecessarily. - ---- - -## PathSet and Pattern Matching - -Defined in: - -- `osquery.osquery.events.pathset.Compare` - -The `PathSet` template provides a thread-safe multiset implementation for path-based matching. - -### Supported Patterns - -- `*` – Match a single path component. -- `**` – Recursive wildcard. - -Example equivalence: - -```text -/This/Path/is -/This/Path/* -/This/Path/** -``` - -### Comparison Logic - -The `Compare` functor: - -- Compares path components lexicographically. -- Treats `*` as a wildcard. -- Treats `**` as recursive match. -- Maintains ordering semantics required by `std::multiset`. - -This is especially important for filesystem event publishers that need efficient path matching across many subscriptions. - ---- - -## Concurrency Model - -```mermaid -flowchart LR - FactoryLock["factory_lock_"] --> ProtectPub["event_pubs_"] - FactoryLock --> ProtectSub["event_subs_"] - - SubscriberLock1["event_id_lock_"] --> IDGen["EventID generation"] - SubscriberLock2["event_record_lock_"] --> RecordWrite["record time bins"] - SubscriberLock3["event_query_record_"] --> QueryTracking["query usage tracking"] -``` - -Key thread-safety mechanisms: - -- Recursive locks in `EventFactory`. -- Mutex-protected event indexing. -- Atomic counters for event identifiers and expiration windows. -- Dedicated threads per publisher. - ---- - -## Integration With Other Modules - -### SQL Engine And Virtual Tables - -- Event subscribers expose data through `genTable()`. -- `_events` tables are queried like normal virtual tables. - -### Database And Storage Plugins - -- Event rows are persisted using `IDatabaseInterface`. -- Time-based indexing enables efficient windowed queries. - -### Configuration And Packs - -- Scheduled queries determine retention windows. -- Config parsers can enable/disable subscribers. - -### Logging And Query Observability - -- Events may be forwarded using `addForwarder()` and `forwardEvent()`. -- Logger plugins can receive structured event payloads. - ---- - -## End-to-End Flow - -```mermaid -sequenceDiagram - participant Config - participant Factory as EventFactory - participant Pub as EventPublisher - participant Sub as EventSubscriber - participant DB as Database - participant SQL - - Config->>Factory: configUpdate() - Factory->>Sub: setMinExpiry() - Factory->>Pub: spawn run loop - Pub->>Sub: fire event (callback) - Sub->>DB: addBatch() - SQL->>Sub: SELECT from event table - Sub->>DB: generateRows() - Sub->>SQL: yield rows -``` - ---- - -## Design Principles - -1. **Separation of Concerns** – Publishers generate, subscribers transform and persist, SQL retrieves. -2. **Config-Aware Retention** – Event expiration adapts to scheduled queries. -3. **Thread Isolation** – Each publisher runs independently. -4. **Optimized Query Windows** – Incremental scanning via time and ID indexes. -5. **Extensibility** – New publishers and subscribers can be registered dynamically. - ---- - -## Summary - -The **Eventing Framework And Subscriptions** module is the backbone of osquery’s event-driven capabilities. It provides: - -- A robust publish/subscribe infrastructure. -- Persistent and optimized event storage. -- Config-driven lifecycle management. -- Tight integration with the SQL engine and scheduler. - -Through careful coordination between `EventFactory`, `EventSubscriberPlugin`, `Subscription`, and `PathSet`, this module enables scalable, queryable system event monitoring across platforms. \ No newline at end of file diff --git a/docs/reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md b/docs/reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md new file mode 100644 index 00000000000..e52aa39dbbc --- /dev/null +++ b/docs/reference/architecture/events-core-and-subscriptions/events-core-and-subscriptions.md @@ -0,0 +1,316 @@ +# Events Core And Subscriptions + +The **Events Core And Subscriptions** module implements the publish/subscribe eventing infrastructure used by osquery to collect, persist, and expose real-time and near-real-time system events as virtual tables. + +It connects: + +- **Event publishers** (low-level OS or subsystem monitors) +- **Event subscribers** (table-oriented processors of events) +- **Subscriptions** (scoped event filters and callbacks) +- **Persistent storage and expiration logic** + +This module is the backbone of event-based tables (e.g., file events, process events) and integrates tightly with: + +- [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md) +- [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) +- [Database Backend](../database-backend/database-backend.md) +- [Logging And Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) + +--- + +## 1. Architectural Overview + +The module implements a centralized **EventFactory** that coordinates publishers and subscribers. + +```mermaid +flowchart TD + Config["Configuration And Packs"] -->|"scheduled queries"| EventFactory["EventFactory"] + + subgraph publishers["Event Publishers"] + Pub1["EventPublisherPlugin"] + end + + subgraph subscribers["Event Subscribers"] + Sub1["EventSubscriberPlugin"] + end + + EventFactory -->|"register"| Pub1 + EventFactory -->|"register"| Sub1 + + Sub1 -->|"create"| Subscription["Subscription"] + Subscription -->|"attach"| Pub1 + + Pub1 -->|"fires events"| Sub1 + Sub1 -->|"persist rows"| DB["Database Backend"] + Sub1 -->|"expose table"| SQL["SQL Engine And Virtual Tables"] +``` + +### Core Responsibilities + +| Component | Responsibility | +|------------|---------------| +| EventFactory | Lifecycle and orchestration of publishers and subscribers | +| EventSubscriberPlugin | Consumes events and persists them as table rows | +| Subscription | Binds a subscriber callback to publisher-specific criteria | +| Context | Maintains indexing and namespace state | +| GenerateRowsResult | Supports optimized time-bounded table scans | + +--- + +## 2. EventFactory + +The **EventFactory** is a singleton that manages: + +- Publisher registration +- Subscriber registration +- Subscription routing +- Thread lifecycle for publishers +- Schedule-driven expiration tuning + +### 2.1 Registration Flow + +```mermaid +flowchart TD + RegisterSub["registerEventSubscriber()"] --> Setup["setUp()"] + Setup --> Init["init()"] + Init --> Running["EVENT_RUNNING"] + + RegisterPub["registerEventPublisher()"] --> PubSetup["publisher.setUp()"] + PubSetup --> Stored["stored in factory map"] +``` + +Key behaviors: + +- Validates plugin type +- Applies configuration-based enable/disable rules +- Transitions subscriber state (`EVENT_SETUP`, `EVENT_RUNNING`, `EVENT_PAUSED`, `EVENT_FAILED`) +- Prevents duplicate publisher types + +### 2.2 Subscription Routing + +Subscriptions are added via: + +- `EventFactory::addSubscription(type_id, name_id, context, callback)` +- Routed to the appropriate publisher + +If the publisher does not exist, the operation fails safely. + +### 2.3 Publisher Execution Model + +Each publisher runs inside its own thread. + +```mermaid +flowchart TD + Delay["EventFactory.delay()"] --> Spawn["spawn thread per publisher"] + Spawn --> RunLoop["EventFactory.run(type)"] + RunLoop --> Loop["publisher.run() loop"] + Loop --> Pause["pause 200ms"] + Loop -->|"error"| TearDown["tearDown()"] +``` + +Important behaviors: + +- Publishers cannot be restarted once started +- Controlled shutdown via `isEnding()` +- Automatic teardown on loop exit +- Threads joined or detached via `end(join)` + +--- + +## 3. EventSubscriberPlugin + +`EventSubscriberPlugin` is the abstraction that: + +- Registers subscriptions during `init()` +- Receives events via callbacks +- Converts them into `Row` objects +- Persists them in the backing database +- Serves them through `genTable()` + +### 3.1 Internal Context + +Each subscriber maintains a `Context`: + +```mermaid +flowchart TD + Context["Context"] --> Namespace["database_namespace"] + Context --> Index["event_index"] + Context --> LastTime["last_query_time"] + Context --> LastEvent["last_event_id"] +``` + +The context ensures: + +- Namespace isolation between subscribers +- Unique event identifiers +- Optimized time-based indexing +- Safe concurrency via mutexes + +### 3.2 Event Storage Flow + +When an event fires: + +```mermaid +flowchart TD + Publisher["EventPublisher"] --> Callback["EventCallback"] + Callback --> AddBatch["addBatch(rows)"] + AddBatch --> DBWrite["Database Backend"] +``` + +Key concepts: + +- `addBatch()` is preferred over deprecated `add()` +- Events are indexed by time and EventID +- Expiration and overflow enforcement applied + +### 3.3 Query-Time Retrieval + +At SQL execution time: + +```mermaid +flowchart TD + SQLQuery["SELECT * FROM event_table"] --> GenTable["genTable()"] + GenTable --> GenerateRows["generateRows()"] + GenerateRows --> Filter["time window filtering"] + Filter --> Yield["RowYield callback"] +``` + +`GenerateRowsResult` provides: + +- `isEnd` flag +- `last_time` +- `last_id` + +This supports incremental and optimized scans. + +--- + +## 4. Subscription + +A **Subscription** binds: + +- A subscriber name +- A publisher-specific context +- A callback + +```mermaid +flowchart LR + Sub["EventSubscriber"] -->|"creates"| Subscription + Subscription -->|"context + callback"| Publisher + Publisher -->|"fires"| Callback +``` + +### Structure + +| Field | Purpose | +|--------|--------| +| subscriber_name | Identifies subscriber | +| context | Publisher-specific filter configuration | +| callback | Event processing function | + +Subscriptions: + +- Scope publisher workload +- Allow fine-grained filtering +- Decouple publishers from subscriber logic + +--- + +## 5. Schedule-Aware Expiration Logic + +The module dynamically adjusts expiration windows based on scheduled queries. + +During `configUpdate()`: + +1. All scheduled queries are scanned. +2. Queries referencing `_events` tables are identified. +3. Per-subscriber metrics are computed: + - Maximum interval + - Query count +4. Minimum expiration is set to `max_interval * 3`, rounded to minute boundary. + +```mermaid +flowchart TD + Scheduled["Scheduled Queries"] --> Scan["scan for event tables"] + Scan --> Compute["compute max interval + count"] + Compute --> SetExpiry["setMinExpiry()"] + SetExpiry --> ResetCount["resetQueryCount()"] +``` + +This ensures: + +- Data persists long enough for slow schedules +- Subscribers expire events safely +- Over-aggressive expiration is prevented + +--- + +## 6. Forwarding and Logging Integration + +EventFactory supports forwarding serialized events to logger plugins: + +- `addForwarder(logger_name)` +- `forwardEvent(event_string)` + +This integrates with: + +- [Logging And Query Metadata](../logging-and-query-metadata/logging-and-query-metadata.md) + +Allowing event streams to be: + +- Logged +- Forwarded to remote systems +- Integrated with distributed pipelines + +--- + +## 7. Lifecycle Summary + +```mermaid +flowchart TD + Start["Startup"] --> Register["Register Publishers & Subscribers"] + Register --> Config["Apply Config Enable/Disable"] + Config --> Delay["Spawn Publisher Threads"] + Delay --> Running["Event Loop Running"] + Running --> Query["SQL Query Execution"] + Query --> Expire["Expiration & Optimization"] + Expire --> Shutdown["EventFactory.end()"] +``` + +--- + +## 8. Relationship to Other Modules + +| Module | Integration Role | +|---------|-----------------| +| Configuration And Packs | Drives scheduled queries and subscriber tuning | +| SQL Engine And Virtual Tables | Exposes event data as queryable tables | +| Database Backend | Persists indexed event rows | +| Logging And Query Metadata | Emits event logs and execution metadata | + +The **Events Core And Subscriptions** module acts as the real-time data ingestion and event persistence layer between OS-level monitoring and SQL-based introspection. + +--- + +## 9. Design Characteristics + +- Thread-isolated publishers +- Subscriber-driven storage model +- Time-indexed database persistence +- Schedule-aware expiration tuning +- Plugin-based extensibility +- Configuration-driven enable/disable logic +- Safe teardown and deregistration + +--- + +## Conclusion + +The **Events Core And Subscriptions** module provides a scalable, extensible event processing framework built around: + +- Publisher threads +- Subscription-based filtering +- Subscriber-managed persistence +- SQL-driven retrieval + +It enables osquery to transform asynchronous OS signals into consistent, queryable relational data while maintaining performance, safety, and configurability across diverse platforms. \ No newline at end of file diff --git a/docs/reference/architecture/extensions-and-ipc/extensions-and-ipc.md b/docs/reference/architecture/extensions-and-ipc/extensions-and-ipc.md deleted file mode 100644 index b6bf3ce6127..00000000000 --- a/docs/reference/architecture/extensions-and-ipc/extensions-and-ipc.md +++ /dev/null @@ -1,375 +0,0 @@ -# Extensions And Ipc - -The **Extensions And Ipc** module provides the inter-process communication (IPC) layer and extension framework that allows osquery to be extended at runtime. It enables external processes (extensions) to: - -- Register new registry plugins (tables, config plugins, logger plugins, etc.) -- Execute SQL queries via the core engine -- Expose custom functionality to the osquery core -- Communicate securely over platform-specific IPC channels - -This module is the foundation for osquery’s pluggable architecture, separating the core daemon from externally developed functionality while maintaining a controlled and versioned API boundary. - ---- - -## 1. Purpose and Design Goals - -The Extensions And Ipc module is designed to: - -- βœ… Allow third-party or internal extensions to run as separate processes -- βœ… Provide a stable RPC interface using Apache Thrift -- βœ… Enforce SDK compatibility and version checks -- βœ… Isolate crashes or faults from the core daemon -- βœ… Support cross-platform IPC (UNIX domain sockets on POSIX, named pipes on Windows) - -At a high level, it implements: - -- An **Extension Manager** (running inside osquery core) -- One or more **Extension Processes** (external binaries) -- A **Thrift-based RPC layer** -- Health monitoring and watchdog services - ---- - -## 2. High-Level Architecture - -The Extensions And Ipc module sits between: - -- The **Registry subsystem** (plugin routing) -- The **SQL engine** (query delegation) -- The **Dispatcher and runtime threads** -- External extension processes - -### 2.1 Core–Extension Interaction Model - -```mermaid -flowchart LR - Core["osquery Core"] -->|"Starts"| Manager["Extension Manager"] - Manager -->|"Registers routes"| Registry["RegistryFactory"] - - ExtensionProc["Extension Process"] -->|"registerExtension()"| Manager - Manager -->|"Assign UUID"| ExtensionProc - - Core -->|"callExtension()"| Manager - Manager -->|"Route to UUID"| ExtensionProc - - ExtensionProc -->|"Response"| Manager - Manager -->|"PluginResponse"| Core -``` - -### 2.2 IPC and Thrift Layer - -```mermaid -flowchart TD - Client["ExtensionClient"] --> Transport["Thrift Transport"] - Transport --> Socket["UNIX Socket or Named Pipe"] - Socket --> Server["ExtensionRunner or ManagerRunner"] - Server --> Handler["ExtensionHandler or ExtensionManagerHandler"] - Handler --> Interface["ExtensionInterface / ExtensionManagerInterface"] - Interface --> Registry["RegistryFactory"] -``` - ---- - -## 3. Core Components - -### 3.1 Extension Metadata - -**Component:** `ExtensionInfo` - -Defined in `extensions.h`, this struct mirrors the Thrift `InternalExtensionInfo` type and contains: - -- `name` -- `version` -- `sdk_version` -- `min_sdk_version` - -This metadata is validated during registration and stored in an `ExtensionList` keyed by a transient `RouteUUID`. - ---- - -### 3.2 Extension Manager (Core Side) - -**Key Classes:** - -- `ExtensionManagerInterface` -- `ExtensionManagerHandler` -- `ExtensionManagerRunner` -- `ExtensionManagerWatcher` - -#### Responsibilities - -1. Accept extension registrations -2. Assign unique `RouteUUID` values (via `UuidGenerator`) -3. Validate SDK compatibility -4. Maintain active extension metadata -5. Route plugin calls to registered extensions -6. Monitor extension health - -#### Registration Flow - -```mermaid -sequenceDiagram - participant Ext as Extension Process - participant EM as Extension Manager - participant Reg as RegistryFactory - - Ext->>EM: registerExtension(info, registry) - EM->>EM: Validate name uniqueness - EM->>EM: Check SDK compatibility - EM->>Reg: addBroadcast(uuid, registry) - EM-->>Ext: Return UUID -``` - -If duplicate names or incompatible SDK versions are detected, registration fails. - ---- - -### 3.3 Extension Process (External Binary) - -**Key Classes:** - -- `ExtensionRunner` -- `ExtensionInterface` -- `ExtensionHandler` - -An extension process: - -1. Connects to the Extension Manager socket -2. Registers its broadcasted registry routes -3. Receives a `RouteUUID` -4. Starts a Thrift server loop -5. Waits for calls from the core - -Each extension serves the `Extension` Thrift service defined in the generated files. - ---- - -### 3.4 Thrift RPC Layer - -**Key Files:** - -- `impl_thrift.cpp` -- Generated files under `thrift/gen/` - -The module uses Apache Thrift to define two services: - -1. `Extension` – implemented by extension processes -2. `ExtensionManager` – implemented by osquery core - -#### Server-Side Wrappers - -- `ExtensionHandler` implements `ExtensionIf` -- `ExtensionManagerHandler` implements `ExtensionManagerIf` - -These handlers: - -- Translate Thrift structures into internal `PluginRequest`/`PluginResponse` -- Delegate logic to `ExtensionInterface` or `ExtensionManagerInterface` - -#### Client-Side Wrappers - -- `ExtensionClient` -- `ExtensionManagerClient` - -They abstract: - -- Transport creation -- Timeout configuration -- Method invocation -- Status translation - ---- - -### 3.5 Extension API Contracts - -The module defines abstract APIs: - -- `ExtensionAPI` -- `ExtensionManagerAPI` - -These interfaces enforce a separation between: - -- Thrift transport concerns -- Business logic - -This design allows alternative RPC backends in the future while preserving core logic. - ---- - -### 3.6 UUID Management - -**Component:** `UuidGenerator` - -- Generates 16-bit UUID values -- Tracks active UUIDs in a thread-safe set -- Removes UUIDs on deregistration - -UUIDs uniquely identify extension routes inside `RegistryFactory`. - ---- - -### 3.7 External SQL Plugin - -**Component:** `ExternalSQLPlugin` - -This special plugin allows the core SQL registry to delegate SQL queries to extensions. - -```mermaid -flowchart LR - SQL["SQL Engine"] --> ExternalSQL["ExternalSQLPlugin"] - ExternalSQL --> ManagerClient["ExtensionManagerClient"] - ManagerClient --> Extension["Extension Process"] -``` - -Used when tables or query logic are implemented outside the core. - ---- - -### 3.8 Watchers and Health Monitoring - -**Key Classes:** - -- `ExtensionWatcher` -- `ExtensionManagerWatcher` - -#### ExtensionWatcher - -- Runs inside extension processes -- Periodically pings the manager -- Exits fatally if the manager disappears - -#### ExtensionManagerWatcher - -- Runs inside core -- Pings all registered extensions -- Removes routes if extensions become unreachable - -```mermaid -flowchart TD - Watcher["ExtensionManagerWatcher"] --> UUIDs["Registered UUIDs"] - UUIDs --> Ping["Ping Extension"] - Ping -->|"Success"| Keep["Keep Registered"] - Ping -->|"Failure x2"| Remove["Remove Broadcast Route"] -``` - ---- - -### 3.9 Autoloading and Security - -Extensions can be autoloaded using: - -- `--extensions_autoload` -- `--extension` (shell-only) - -Security checks include: - -- File extension validation (`.ext`, `.exe` on Windows) -- Directory permission safety checks -- Ownership validation - -Unsafe or improperly permissioned binaries are rejected. - ---- - -### 3.10 IPC Channel Integration - -On POSIX systems, IPC is primarily UNIX domain sockets. Additionally, worker IPC may use pipe-based channels. - -**Component:** `GetChannelType` - -This specialization maps a `PipeChannelFactory` to its underlying `PipeChannel` type, integrating extension communication with the worker IPC framework. - -This ensures consistent abstraction across: - -- Extension IPC -- Worker-table IPC -- Internal task communication - ---- - -## 4. Lifecycle Overview - -### 4.1 Core Startup - -```mermaid -flowchart TD - Start["Core Start"] --> StartManager["startExtensionManager()"] - StartManager --> Watcher["ExtensionManagerWatcher"] - Watcher --> Runner["ExtensionManagerRunner"] - Runner --> Listen["Thrift Listen"] -``` - -### 4.2 Extension Startup - -```mermaid -flowchart TD - ExtStart["Extension Binary Start"] --> Connect["Connect to Manager Socket"] - Connect --> Register["registerExtension()"] - Register --> UUID["Receive UUID"] - UUID --> StartServer["Start ExtensionRunner"] - StartServer --> Serve["Serve Thrift Requests"] -``` - -### 4.3 Shutdown Flow - -- Extension calls `shutdown()` or receives manager failure -- Manager deregisters UUID -- `RegistryFactory::removeBroadcast()` is invoked -- Stale socket paths are cleaned via `removeStalePaths()` - ---- - -## 5. Interaction with Other Subsystems - -The Extensions And Ipc module interacts with: - -- **RegistryFactory** – route registration and broadcast -- **SQL Engine** – delegated queries -- **Dispatcher** – background service threads -- **Flags subsystem** – option propagation -- **Filesystem utilities** – socket and permission checks - -It acts as a boundary layer rather than a business-logic module. - ---- - -## 6. Error Handling and Status Model - -All RPC responses use: - -- `ExtensionStatus` -- `ExtensionResponse` -- `ExtensionCode` (SUCCESS, FAILED, FATAL) - -This ensures: - -- Consistent status propagation across process boundaries -- Clear failure semantics for SDK mismatches and duplicate registrations - ---- - -## 7. Security Considerations - -The module enforces: - -- SDK version compatibility checks -- Duplicate extension prevention -- File permission validation for autoload -- Socket path isolation per UUID -- Controlled shutdown behavior - -On Windows, named pipes are secured using explicit security descriptors. - ---- - -## 8. Summary - -The **Extensions And Ipc** module is the runtime extension backbone of osquery. It: - -- Implements a bidirectional Thrift RPC system -- Manages extension lifecycle and UUID assignment -- Routes registry plugin calls across process boundaries -- Monitors extension health -- Enforces compatibility and security constraints - -Without this module, osquery would be a static binary. With it, osquery becomes a dynamic, pluggable platform capable of safely integrating externally developed functionality at runtime. diff --git a/docs/reference/architecture/extensions-framework/extensions-framework.md b/docs/reference/architecture/extensions-framework/extensions-framework.md new file mode 100644 index 00000000000..2a764738780 --- /dev/null +++ b/docs/reference/architecture/extensions-framework/extensions-framework.md @@ -0,0 +1,377 @@ +# Extensions Framework + +The **Extensions Framework** enables osquery to be dynamically extended at runtime through external processes. It provides a secure, version-aware, and Thrift-based inter-process communication (IPC) layer that allows third-party or custom components to register new registry plugins, SQL tables, configuration providers, and loggers without modifying the core binary. + +At its core, the Extensions Framework: + +- Starts and manages an **Extension Manager** inside osquery core +- Allows external binaries (extensions) to **register registry routes** +- Provides a **Thrift IPC layer** for RPC-style communication +- Monitors extension health and lifecycle +- Enforces SDK compatibility and uniqueness constraints + +--- + +## 1. Architectural Overview + +The Extensions Framework is built around a manager–extension model using UNIX domain sockets (or Windows named pipes) and Apache Thrift. + +```mermaid +flowchart TD + Core["osquery Core"] --> Manager["Extension Manager"] + Manager --> Registry["RegistryFactory"] + + ExtensionA["Extension Process A"] -->|"registerExtension()"| Manager + ExtensionB["Extension Process B"] -->|"registerExtension()"| Manager + + Core -->|"callExtension()"| ExtensionA + Core -->|"callExtension()"| ExtensionB + + Manager -->|"health checks"| ExtensionA + Manager -->|"health checks"| ExtensionB +``` + +### Key Roles + +- **Extension Manager**: Runs inside osquery core and exposes the Thrift API for extension registration and coordination. +- **Extension Process**: External binary that registers registry routes and serves plugin calls. +- **RegistryFactory**: Maintains broadcast routes and resolves plugin calls between core and extensions. +- **Extension Watchers**: Monitor connectivity and liveness of the manager and registered extensions. + +--- + +## 2. Core Components + +### 2.1 ExtensionInfo + +**Component:** `osquery.osquery.extensions.extensions.ExtensionInfo` + +`ExtensionInfo` encapsulates metadata for each registered extension: + +- `name` +- `version` +- `sdk_version` +- `min_sdk_version` + +This metadata is validated during registration to: + +- Prevent duplicate extension names +- Enforce minimum SDK compatibility +- Track active extensions by `RouteUUID` + +Internally, extensions are tracked as: + +- `ExtensionList = std::map` + +--- + +### 2.2 ExtensionManagerInterface + +The `ExtensionManagerInterface` implements the core logic for managing extension lifecycle and registry broadcasts. + +Key responsibilities: + +- Registering extensions (`registerExtension`) +- Deregistering extensions (`deregisterExtension`) +- Tracking metadata (`extensions()`) +- Enforcing SDK compatibility +- Exposing osquery flags as `OptionList` +- Executing SQL on behalf of extensions + +```mermaid +flowchart LR + Ext["Extension Process"] -->|"registerExtension"| EM["ExtensionManagerInterface"] + EM --> UUID["Assign RouteUUID"] + EM --> Registry["RegistryFactory.addBroadcast"] + EM --> Store["Store ExtensionInfo"] +``` + +During registration: + +1. Duplicate extension names are rejected. +2. SDK compatibility is validated. +3. A new `RouteUUID` is generated via `UuidGenerator`. +4. Registry routes are broadcast into `RegistryFactory`. +5. Metadata is stored for tracking and health monitoring. + +--- + +### 2.3 UuidGenerator + +**Component:** `osquery.osquery.extensions.interface.UuidGenerator` + +The `UuidGenerator` assigns unique `RouteUUID` values to extensions at registration time. + +Characteristics: + +- Thread-safe via mutex protection +- Prevents UUID reuse +- Ensures uniqueness across active extensions +- Removes UUID on deregistration + +This UUID is: + +- Embedded in extension socket paths +- Used for routing registry calls +- Used by watchers for health checks + +--- + +### 2.4 Thrift Layer + +The Thrift layer provides the RPC mechanism between core and extensions. + +Key components: + +- `ExtensionManagerHandler` +- `ExtensionHandler` +- `ExtensionRunner` +- `ExtensionManagerRunner` +- `ImplExtensionClient` +- `ImplExtensionRunner` + +```mermaid +flowchart TD + Client["ExtensionClient"] --> Socket["UNIX Socket / Named Pipe"] + Socket --> ThriftServer["TThreadedServer"] + ThriftServer --> Handler["ExtensionHandler or ExtensionManagerHandler"] + Handler --> Interface["ExtensionInterface Logic"] +``` + +### ExtensionManagerHandler + +**Component:** `osquery.osquery.extensions.impl_thrift.ExtensionManagerHandler` + +Implements Thrift endpoints for: + +- `registerExtension` +- `deregisterExtension` +- `extensions` +- `options` +- `query` +- `getQueryColumns` + +It translates Thrift-generated types into internal osquery structures and delegates to `ExtensionManagerInterface`. + +### ImplExtensionClient and ImplExtensionRunner + +These provide platform-specific Thrift transport implementations: + +- UNIX sockets on Linux/macOS +- Named pipes on Windows +- Buffered transport +- Binary protocol +- Threaded server model + +--- + +## 3. Lifecycle Management + +### 3.1 Starting the Extension Manager + +`startExtensionManager()`: + +1. Validates socket path availability. +2. Starts `ExtensionManagerWatcher`. +3. Starts `ExtensionManagerRunner` (Thrift server). +4. Optionally enforces required extensions via `extensions_require`. + +```mermaid +flowchart TD + Start["startExtensionManager"] --> Check["Check socketExists"] + Check --> Watcher["Start ExtensionManagerWatcher"] + Watcher --> Runner["Start ExtensionManagerRunner"] + Runner --> Ready["Manager Listening"] +``` + +--- + +### 3.2 Starting an Extension + +`startExtension()` performs: + +1. Wait for manager availability +2. Broadcast registry routes +3. Register with manager +4. Receive assigned `RouteUUID` +5. Start `ExtensionWatcher` +6. Start `ExtensionRunner` (Thrift server) + +```mermaid +flowchart TD + ExtStart["Extension startExtension"] --> Wait["extensionPathActive"] + Wait --> Register["registerExtension"] + Register --> UUID["Receive RouteUUID"] + UUID --> Watcher["Start ExtensionWatcher"] + Watcher --> Runner["Start ExtensionRunner"] +``` + +--- + +### 3.3 Health Monitoring + +#### ExtensionWatcher + +Monitors: + +- Manager socket availability +- Registration presence +- Ping response + +If the manager disappears and `fatal` is true, the extension exits. + +#### ExtensionManagerWatcher + +Monitors: + +- Each registered extension's socket +- Repeated ping failures +- Removes broadcast routes for dead extensions + +```mermaid +flowchart LR + ManagerWatcher["ExtensionManagerWatcher"] --> PingExt["Ping Extension"] + PingExt -->|"Success"| Healthy["Keep Registered"] + PingExt -->|"Failure x2"| Remove["removeBroadcast(uuid)"] +``` + +This ensures that: + +- Stale registry routes are cleaned up +- Dead extensions are removed from routing tables + +--- + +## 4. Registry Routing and Plugin Calls + +Extensions expose registry routes via broadcast during registration. + +When a registry call occurs: + +1. `RegistryFactory` resolves the active plugin +2. If routed to an extension, `callExtension()` is invoked +3. Thrift client sends request to extension socket +4. ExtensionHandler translates request and executes locally +5. Response is returned to core + +```mermaid +flowchart TD + CoreCall["RegistryFactory.call"] --> Resolve["Resolve RouteUUID"] + Resolve --> IPC["callExtension()"] + IPC --> Socket["Thrift RPC"] + Socket --> ExtHandler["ExtensionHandler.call"] + ExtHandler --> Plugin["Local Plugin Execution"] + Plugin --> Response["Return PluginResponse"] +``` + +This allows extensions to implement: + +- SQL tables +- Config plugins +- Logger plugins +- Distributed plugins + +Without linking into the core binary. + +--- + +## 5. SQL Delegation + +Extensions do not embed SQLite directly. Instead: + +- Complex queries are forwarded to core +- Column metadata is retrieved via `getQueryColumns` +- Execution is performed via `ExtensionManagerInterface::query` + +The `ExternalSQLPlugin` allows the core to delegate SQL execution to the manager when appropriate. + +--- + +## 6. Option Synchronization + +`ExtensionManagerInterface::options()` exports all osquery flags as: + +- `OptionList = std::map` + +This allows extensions to: + +- Discover active config plugin +- Discover logger plugin +- Respect runtime configuration + +Options are transferred using Thrift-generated structures. + +--- + +## 7. Error and Status Model + +The framework uses `ExtensionCode` to standardize status: + +- `EXT_SUCCESS` +- `EXT_FAILED` +- `EXT_FATAL` + +These are mirrored between: + +- Internal C++ enums +- Thrift-generated enums + +Each Thrift response includes: + +- `code` +- `message` +- `uuid` + +This ensures consistent cross-process error reporting. + +--- + +## 8. Security Considerations + +The Extensions Framework includes several safeguards: + +- Safe permission checks for autoloaded extensions +- SDK version enforcement +- Duplicate extension name rejection +- Secure socket handling +- Windows named pipe ACL enforcement +- Removal of stale socket paths + +Autoloaded extensions must: + +- Have safe directory permissions +- Use allowed binary extensions (e.g., `.ext`, `.exe`) + +--- + +## 9. Relationship to Other Modules + +The Extensions Framework integrates with: + +- Registry system for plugin routing +- SQL engine for query delegation +- Dispatcher for service threading +- Core runtime for shutdown handling +- Filesystem utilities for socket management + +It acts as the boundary layer between: + +- Internal osquery core +- External extension processes + +--- + +# Summary + +The **Extensions Framework** is a robust IPC and lifecycle orchestration layer that enables osquery to be modular and extensible at runtime. + +It provides: + +- Dynamic plugin injection +- Secure extension registration +- Thrift-based RPC communication +- Health monitoring and cleanup +- SQL and option delegation +- Cross-platform transport support + +Through this design, osquery maintains a small, stable core while allowing flexible extension through isolated, independently deployable binaries. \ No newline at end of file diff --git a/docs/reference/architecture/filesystem-and-fileops-core/filesystem-and-fileops-core.md b/docs/reference/architecture/filesystem-and-fileops-core/filesystem-and-fileops-core.md new file mode 100644 index 00000000000..6191785e3bd --- /dev/null +++ b/docs/reference/architecture/filesystem-and-fileops-core/filesystem-and-fileops-core.md @@ -0,0 +1,345 @@ +# Filesystem And Fileops Core + +## Overview + +The **Filesystem And Fileops Core** module provides the cross-platform abstraction layer for file system access, file metadata inspection, permissions handling, and glob-based path resolution within osquery. + +It unifies POSIX and Windows behaviors behind a common interface so that higher-level components (SQL tables, extensions, configuration loaders, logging, and distributed features) can interact with files and directories in a platform-agnostic way. + +At its heart is the `PlatformFile` abstraction, supported by platform-specific implementations and a set of helper utilities for: + +- File open, read, write, seek, and size operations +- Permission and ownership validation +- Safe execution and module loading checks +- Globbing and recursive path resolution +- Cross-platform `stat`-like metadata retrieval +- Secure handling of sockets and temporary directories + +--- + +## Architectural Overview + +The module is structured around a portable interface with OS-specific backends. + +```mermaid +flowchart TD + HighLevel["High-Level Consumers"] --> FilesystemAPI["Filesystem API"] + FilesystemAPI --> PlatformFile["PlatformFile"] + FilesystemAPI --> GlobEngine["Glob and Pattern Resolver"] + FilesystemAPI --> PermissionChecks["Permission and Safety Checks"] + + PlatformFile --> PosixImpl["POSIX Implementation"] + PlatformFile --> WindowsImpl["Windows Implementation"] + + GlobEngine --> PosixGlob["POSIX glob()"] + GlobEngine --> WindowsGlob["Windows FindFirstFile"] + + PermissionChecks --> PosixPerms["POSIX chmod and stat"] + PermissionChecks --> WindowsACL["Windows ACL and SID Logic"] +``` + +### Design Principles + +1. **Single Logical Interface** – `PlatformFile` and `platform*` helpers hide OS differences. +2. **Security First** – Explicit safe-permission validation before executing or loading files. +3. **Non-blocking Support** – Emulated asynchronous semantics across platforms. +4. **Controlled Resource Usage** – Read limits and block-based file reading. +5. **Globbing Compatibility** – Approximate POSIX glob behavior on Windows. + +--- + +## Core Components + +### 1. AsyncEvent + +**Component:** `osquery.osquery.filesystem.fileops.AsyncEvent` + +`AsyncEvent` encapsulates Windows asynchronous I/O state. It is used to simulate POSIX-style non-blocking file semantics on Windows using the overlapped I/O API. + +Responsibilities: + +- Maintain `OVERLAPPED` structure +- Track active asynchronous reads +- Manage temporary buffers +- Detect incomplete I/O operations + +On POSIX systems, non-blocking behavior relies on `O_NONBLOCK` and `EAGAIN`. + +--- + +### 2. PlatformTime + +**Component:** `osquery.osquery.filesystem.fileops.PlatformTime` + +`PlatformTime` abstracts access and modification timestamps across platforms: + +- POSIX: `timeval` +- Windows: `FILETIME` + +Used by `PlatformFile::getFileTimes()` to normalize time retrieval. + +--- + +### 3. stat Implementations + +Components: + +- `osquery.osquery.filesystem.fileops.stat` +- `osquery.osquery.filesystem.filesystem.stat` +- `osquery.osquery.filesystem.posix.fileops.stat` + +The module provides layered metadata access: + +- POSIX: `lstat`, `fstat`, `struct stat` +- Windows: `platformStat` populating `WINDOWS_STAT` + +Windows-specific metadata includes: + +- File ID and inode mapping +- SID-based UID/GID translation +- NTFS attributes +- Version information +- Original filename from version resource + +This allows higher-level SQL tables (e.g., file tables) to present consistent schemas. + +--- + +### 4. WindowsFindFiles + +**Component:** `osquery.osquery.filesystem.windows.fileops.WindowsFindFiles` + +Encapsulates `FindFirstFileW` and `FindNextFileW` logic to enumerate directory contents. + +Used internally by the Windows glob implementation to: + +- Enumerate matching directories +- Apply brace and wildcard expansions +- Normalize returned paths + +--- + +## PlatformFile Abstraction + +`PlatformFile` is the central class for file I/O. + +### Responsibilities + +- Open files using portable mode flags +- Support read, write, seek, and size operations +- Detect special files +- Track pending non-blocking operations +- Validate ownership and permissions +- Provide access to native file handles + +### Mode Abstraction + +Portable flags are defined to emulate cross-platform open modes: + +- `PF_READ` +- `PF_WRITE` +- `PF_CREATE_NEW` +- `PF_OPEN_EXISTING` +- `PF_NONBLOCK` +- `PF_APPEND` + +These are translated internally into: + +- POSIX: `open()` flags (`O_RDONLY`, `O_CREAT`, etc.) +- Windows: `CreateFileW()` access masks and disposition flags + +### Lifecycle + +```mermaid +flowchart TD + Create["Create PlatformFile"] --> Open["Open Native Handle"] + Open --> Valid{"Valid Handle?"} + Valid -->|Yes| Operate["Read or Write or Seek"] + Valid -->|No| Error["Return Failure"] + Operate --> Close["Destructor Closes Handle"] +``` + +--- + +## File Reading and Writing + +### Controlled Reads + +`readFile()` enforces a configurable maximum read size. + +Key behaviors: + +- Uses non-blocking mode when supported +- Reads in blocks when file size unknown +- Aborts if size exceeds configured limit +- Supports streaming via predicate callback + +This prevents excessive memory usage and improves resilience. + +### Writing + +`writeTextFile()`: + +- Opens file with requested permissions +- Applies `platformChmod()` to enforce mode +- Writes full content +- Verifies byte counts + +--- + +## Globbing and Path Resolution + +The module implements a portable glob engine. + +### POSIX + +- Uses native `glob()` +- Supports `GLOB_TILDE`, `GLOB_MARK`, `GLOB_BRACE` + +### Windows + +- Converts glob to regex when needed +- Uses `FindFirstFileW` +- Implements brace expansion heuristics +- Emulates tilde expansion via `USERPROFILE` + +### Recursive Globs + +Double-star patterns (`**`) are expanded iteratively with loop detection. + +```mermaid +flowchart TD + Pattern["Input Pattern"] --> Normalize["Canonicalize Base Path"] + Normalize --> Expand["Apply Wildcards"] + Expand --> DetectLoop{"Symlink Loop?"} + DetectLoop -->|Yes| Stop["Stop Recursion"] + DetectLoop -->|No| Recurse["Continue Expansion"] + Recurse --> Filter["Apply Folder or File Filter"] +``` + +--- + +## Permission and Safety Model + +Security-sensitive logic ensures that executables and modules are safe to load. + +### Ownership Checks + +- `isOwnerRoot()` +- `isOwnerCurrentUser()` + +On Windows, this maps to SID comparisons for: + +- Administrators +- Local System + +### Executable and Safe Permissions + +`hasSafePermissions()` verifies: + +- No unsafe write permissions +- Parent directory is protected +- File not world-writable + +On Windows, this involves: + +- Parsing DACLs +- Detecting allow-write ACEs +- Ensuring deny precedence where required + +### Safe Database Permissions + +`platformSetSafeDbPerms()` enforces restricted access: + +- POSIX: `0700` +- Windows: Full control to SYSTEM and Administrators only + +--- + +## Directory and Path Utilities + +The module provides portable utilities for: + +- `pathExists()` +- `isDirectory()` +- `createDirectory()` +- `removePath()` +- `movePath()` +- `platformAccess()` +- `platformIsTmpDir()` +- `platformIsFileAccessible()` + +These functions abstract: + +- `boost::filesystem` +- POSIX `stat()` and `access()` +- Windows ACL-based access checks + +--- + +## Socket Handling + +`socketExists()` abstracts UNIX domain sockets and Windows named pipes. + +Behavior differs: + +- POSIX: filesystem path existence + writability +- Windows: `WaitNamedPipeW()` with timeout + +This is critical for extension manager communication. + +--- + +## Home and System Directory Resolution + +`getHomeDirectory()` and `getSystemRoot()` normalize: + +- POSIX: `$HOME`, `/` +- Windows: `USERPROFILE`, Windows directory + +`osqueryHomeDirectory()` ensures a writable `.osquery` directory or falls back to a temporary directory. + +--- + +## Cross-Platform I/O Flow + +```mermaid +flowchart TD + Caller["Caller"] --> PF["PlatformFile"] + PF --> OSCheck{"Operating System"} + OSCheck -->|POSIX| POSIXIO["open read write lseek"] + OSCheck -->|Windows| WINIO["CreateFile ReadFile WriteFile"] + POSIXIO --> Result["Normalized Result"] + WINIO --> Result +``` + +--- + +## Integration Within osquery + +Filesystem And Fileops Core underpins: + +- SQL virtual tables accessing files +- Configuration loaders reading JSON and packs +- Logging output targets +- Extension manager sockets +- Distributed query artifact access +- Safe executable and module loading + +By isolating OS-specific complexity in this module, the rest of the system can rely on consistent file semantics and robust security checks. + +--- + +## Summary + +The **Filesystem And Fileops Core** module is the portability and safety backbone of osquery’s file interaction layer. + +It provides: + +- Unified file I/O abstraction +- Cross-platform metadata retrieval +- Robust permission validation +- Secure globbing and path expansion +- Defensive read limits and safe directory checks + +Through careful emulation of POSIX semantics on Windows and strict permission enforcement, it ensures reliable and secure file operations across supported operating systems. diff --git a/docs/reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md b/docs/reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md deleted file mode 100644 index 579007ad3ba..00000000000 --- a/docs/reference/architecture/filesystem-and-path-utilities/filesystem-and-path-utilities.md +++ /dev/null @@ -1,338 +0,0 @@ -# Filesystem And Path Utilities - -## Overview - -The **Filesystem And Path Utilities** module provides a cross-platform abstraction layer for file operations, path handling, globbing, permissions, and filesystem metadata inspection across Linux, macOS (POSIX), and Windows systems. - -It is a foundational module used by higher-level components to: - -- Read and write files safely -- Enforce secure permission models -- Resolve glob patterns -- Inspect file metadata -- Enumerate directories and mounts -- Interact with `/proc` on Linux -- Provide consistent APIs across POSIX and Windows - -This module hides operating system differences while preserving security guarantees and performance characteristics. - ---- - -## Architectural Overview - -```mermaid -flowchart TD - Caller["Core / SQL / Extensions"] --> FSAPI["Filesystem API"] - - subgraph abstraction_layer["Cross-Platform Abstraction"] - FSAPI --> PlatformFile["PlatformFile"] - FSAPI --> Glob["platformGlob()"] - FSAPI --> Perms["platformChmod()"] - FSAPI --> Access["platformAccess()"] - FSAPI --> Stat["platformStat() / lstat()"] - end - - subgraph posix_layer["POSIX Implementation"] - PlatformFile --> POSIXFile["open/read/write/lstat"] - Glob --> POSIXGlob["glob()"] - end - - subgraph windows_layer["Windows Implementation"] - PlatformFile --> WinFile["CreateFile / ReadFile"] - Glob --> WinGlob["FindFirstFileW"] - Perms --> WinACL["ACL Manipulation"] - end -``` - -The abstraction layer ensures that callers use a unified API regardless of the underlying OS. - ---- - -## Core Components - -### 1. PlatformFile - -**Component:** `osquery.osquery.filesystem.fileops.PlatformFile` - -`PlatformFile` is the central abstraction for file I/O. - -It provides: - -- Cross-platform file opening modes (`PF_READ`, `PF_WRITE`, etc.) -- Non-blocking I/O support -- Ownership validation -- Executable checks -- Safe permission verification -- Seek and size inspection -- File time retrieval - -### Lifecycle Model - -```mermaid -flowchart TD - Open["Open PlatformFile"] --> Validate["Handle Valid?"] - Validate -->|No| Fail["Return Error"] - Validate -->|Yes| IO["Read / Write / Seek"] - IO --> Close["Destructor Closes Handle"] -``` - -Platform-specific implementations exist for: - -- POSIX (`open`, `read`, `write`, `fstat`) -- Windows (`CreateFileW`, `ReadFile`, overlapped I/O) - ---- - -### 2. File Metadata and Stat Structures - -#### POSIX -- `osquery.osquery.filesystem.posix.fileops.stat` - -Uses: -- `stat` -- `lstat` -- `fstat` - -#### Windows -- `osquery.osquery.filesystem.fileops.WINDOWS_STAT` -- `osquery.osquery.filesystem.fileops.win_stat` -- `osquery.osquery.filesystem.windows.fileops.WindowsFindFiles` - -Windows provides: -- File ID -- Volume serial -- ACL-derived ownership -- File version metadata -- Attributes (hidden, system, archive) - -```mermaid -flowchart LR - Path["File Path"] --> StatCall["platformStat()"] - StatCall -->|POSIX| POSIXStat["struct stat"] - StatCall -->|Windows| WinStat["WINDOWS_STAT"] -``` - ---- - -### 3. Globbing and Pattern Resolution - -**Core APIs:** -- `platformGlob()` -- `resolveFilePattern()` -- `replaceGlobWildcards()` - -Supports: -- `*` and `?` -- `**` recursive globs -- `%` SQL-style wildcard translation -- Brace expansion (Windows regex translation) - -Recursive globbing includes loop detection using inode tracking. - -```mermaid -flowchart TD - Pattern["Input Pattern"] --> Normalize["replaceGlobWildcards()"] - Normalize --> Expand["platformGlob()"] - Expand --> Filter["Apply GLOB_FILES / GLOB_FOLDERS"] - Filter --> Results["Resolved Paths"] -``` - ---- - -### 4. Permissions and Security Enforcement - -Security is a major responsibility of this module. - -#### Ownership Checks -- `isOwnerRoot()` -- `isOwnerCurrentUser()` - -#### Safe Permission Enforcement -- `hasSafePermissions()` -- `platformSetSafeDbPerms()` - -On Windows, safe permissions require: -- Only Administrators and SYSTEM have write privileges -- Explicit ACL validation - -On POSIX, safety approximates: -- Mode `0700` for sensitive files -- No world-writable flags - -#### Execution Safety - -The `safePermissions()` helper ensures: - -- File exists -- Not located in temporary directory -- Proper ownership -- Proper executable bits - -This protects extension loading and module execution paths. - ---- - -### 5. File Reading and Writing Utilities - -High-level helpers include: - -- `readFile()` (streaming and full-buffer versions) -- `writeTextFile()` -- `isReadable()` -- `isWritable()` -- `pathExists()` -- `removePath()` -- `movePath()` - -A configurable limit (`read_max`) prevents excessive file reads. - -```mermaid -flowchart TD - ReadCall["readFile()"] --> Open["PlatformFile"] - Open --> SizeCheck["checkFileReadLimit()"] - SizeCheck --> Loop["Block Read Loop"] - Loop --> Predicate["Callback / Buffer"] -``` - ---- - -### 6. Home Directory and System Paths - -Cross-platform resolution of: - -- Current user home directory (`getHomeDirectory()`) -- Osquery working directory (`osqueryHomeDirectory()`) -- System root (`getSystemRoot()`) - -Fallback behavior: - -- Uses environment variables first -- Falls back to system APIs -- Falls back to temp directory if needed - ---- - -## Linux-Specific Extensions - -### Mounted Filesystems - -**Component:** `osquery.osquery.filesystem.linux.mounts.MountInformation` - -Provides structured information for mounted filesystems: - -- Device path -- Filesystem type -- Mount flags -- `statfs` block and inode counts - -```mermaid -flowchart LR - getMounted["getMountedFilesystems()"] --> MountInfo["MountInformation"] - MountInfo --> StatFS["StatFsInfo"] -``` - ---- - -### /proc Parsing and Socket Inspection - -**Components:** -- `osquery.osquery.filesystem.linux.proc.SocketInfo` -- `osquery.osquery.filesystem.linux.proc.SocketProcessInfo` - -Capabilities: - -- Enumerate processes via `/proc` -- Map socket inode to process -- Decode hex-encoded network addresses -- Parse `/proc/net/*` files - -```mermaid -flowchart TD - Proc["/proc"] --> Enumerate["procEnumerateProcesses()"] - Enumerate --> FDEnum["procEnumerateProcessDescriptors()"] - FDEnum --> SocketMap["SocketInodeToProcessInfoMap"] -``` - -This enables socket tables and network observability features. - ---- - -## Windows-Specific Enhancements - -Windows includes additional capabilities: - -- NTFS ACL manipulation -- File version extraction -- Original filename extraction from PE metadata -- Overlapped (async) I/O (`AsyncEvent`) -- Named pipe detection (`socketExists()`) - -Windows globbing is implemented using: -- `FindFirstFileW` -- Regex translation for brace patterns - ---- - -## Security Model Summary - -```mermaid -flowchart TD - File["Candidate File"] --> Exists["Exists?"] - Exists -->|No| Reject["Reject"] - Exists --> Owner["Owner Root or Current User?"] - Owner -->|No| Reject - Owner --> Writable["World Writable?"] - Writable -->|Yes| Reject - Writable --> Exec["Executable Required?"] - Exec --> Accept["Safe"] -``` - -This ensures: - -- No unsafe extension loading -- No world-writable execution -- No loading from temp directories - ---- - -## Integration Within the System - -The Filesystem And Path Utilities module is used by: - -- SQL virtual tables for file metadata -- Configuration loading -- Extension management -- Logging subsystems -- Distributed query execution -- Event subscribers - -It acts as a **trusted boundary layer** between: - -- User-supplied file paths -- The operating system -- Internal execution logic - ---- - -## Key Design Principles - -1. **Cross-platform consistency** -2. **Security-first permission enforcement** -3. **Non-blocking support where possible** -4. **Recursive glob loop detection** -5. **Minimal exposure of OS-specific details** - ---- - -## Conclusion - -The **Filesystem And Path Utilities** module is a foundational cross-platform subsystem responsible for: - -- File I/O abstraction -- Secure permission validation -- Glob resolution -- Filesystem metadata access -- Linux `/proc` inspection -- Windows ACL and metadata handling - -It enables higher-level components to operate safely and consistently across heterogeneous operating systems while enforcing strict security guarantees. \ No newline at end of file diff --git a/docs/reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md b/docs/reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md new file mode 100644 index 00000000000..c8d899e9bee --- /dev/null +++ b/docs/reference/architecture/logging-and-query-metadata/logging-and-query-metadata.md @@ -0,0 +1,314 @@ +# Logging And Query Metadata + +## Overview + +The **Logging And Query Metadata** module is responsible for transforming query executions and internal runtime events into structured, serializable, and transportable log artifacts. + +It acts as the bridge between: + +- The SQL execution engine and scheduler +- The persistent query state stored in the database backend +- The pluggable logging framework +- External logging and distributed systems + +This module defines: + +- Query result representation (`QueryLogItem`) +- Differential result computation (`DiffResults`) +- Scheduled query metadata (`ScheduledQuery`) +- Query execution performance metrics (`QueryPerformance`) +- Logger plugin contracts and status log lines (`StatusLogLine`) + +It does **not** execute SQL directly (see the SQL Engine And Virtual Tables module) and does **not** manage persistent storage directly (see the Database Backend module). Instead, it orchestrates how results and metadata are formatted, diffed, tracked, and emitted. + +--- + +## Architectural Positioning + +```mermaid +flowchart TD + Scheduler["Configuration And Packs"] -->|"creates"| ScheduledQuery + ScheduledQuery -->|"executes"| SQLEngine["SQL Engine And Virtual Tables"] + SQLEngine -->|"returns rows"| QueryLayer["Query"] + QueryLayer -->|"computes diff"| DiffResultsNode["DiffResults"] + QueryLayer -->|"builds"| QueryLogItemNode["QueryLogItem"] + QueryLogItemNode -->|"serialize"| LoggerPluginNode["LoggerPlugin"] + LoggerPluginNode -->|"emits"| ExternalSystem["External Logging System"] + QueryLayer -->|"store previous"| DatabaseBackend["Database Backend"] +``` + +### Key Relationships + +- **Configuration And Packs** defines scheduled queries. +- **SQL Engine And Virtual Tables** executes SQL and produces row sets. +- **Database Backend** persists historical results. +- **Logging And Query Metadata**: + - Computes diffs between executions + - Tracks metadata (epoch, counter, timestamps) + - Serializes results + - Sends structured logs to logger plugins + +--- + +## Core Responsibilities + +### 1. Scheduled Query Metadata + +**Component:** `ScheduledQuery` + +Represents the runtime metadata of a scheduled query: + +- Query name and SQL +- Pack name +- Execution interval and startup priority +- Denylisting state +- Query options (`snapshot`, `removed`, etc.) + +Key behaviors: + +- `isSnapshotQuery()` determines if the query emits full results every run. +- `reportRemovedRows()` determines whether removed rows are included in logs. + +This metadata drives how results are processed and serialized downstream. + +--- + +### 2. Historical State and Diffing + +**Components:** + +- `Query` +- `DiffResults` + +The `Query` class is the stateful interface between execution results and the persistent database. + +#### Lifecycle of a Scheduled Query Execution + +```mermaid +flowchart TD + Start["Scheduled Query Execution"] --> FetchOld["getPreviousQueryResults()"] + FetchOld --> Exec["Execute SQL"] + Exec --> Diff["diff(old, new)"] + Diff --> Store["saveQueryResults()"] + Store --> BuildLog["Construct QueryLogItem"] + BuildLog --> End["Emit Log"] +``` + +### `DiffResults` + +Encapsulates the difference between: + +- Previous result set +- Current result set + +Structure: + +- `added`: newly observed rows +- `removed`: rows no longer present + +Utility: + +- `hasNoResults()` avoids emitting empty logs. +- Serialization helpers convert diffs into JSON. + +The diff mechanism ensures efficient logging by emitting only state transitions instead of full snapshots (unless configured otherwise). + +--- + +### 3. Query Log Representation + +**Component:** `QueryLogItem` + +`QueryLogItem` is the canonical log artifact emitted by the system. + +It contains: + +- `isSnapshot`: snapshot vs differential mode +- `results`: differential results (`DiffResults`) +- `snapshot_results`: full result set +- `name`: scheduled query name +- `identifier`: host identity +- `time` and `calendar_time` +- `epoch`: logical configuration version +- `counter`: execution counter +- `decorations`: additional metadata fields + +#### Serialization Paths + +The module provides multiple serialization formats: + +- `serializeQueryLogItem()` β†’ JSON document +- `serializeQueryLogItemJSON()` β†’ JSON string +- `serializeQueryLogItemAsEvents()` β†’ event-based representation +- `serializeQueryLogItemAsEventsJSON()` β†’ event list JSON strings + +This flexibility enables: + +- Batch logging +- Event-driven logging +- Snapshot logging +- Differential logging + +--- + +### 4. Query Performance Tracking + +**Component:** `QueryPerformance` + +Tracks runtime characteristics of each query: + +- Execution count +- Wall time (seconds and milliseconds) +- User and system CPU time +- Memory consumption +- Output size + +It supports: + +- CSV serialization (`toCSV()`) +- CSV deserialization constructor + +Performance metrics are essential for: + +- Detecting expensive queries +- Observability dashboards +- Adaptive scheduling logic +- Operational debugging + +--- + +### 5. Logging Plugin Interface + +**Components:** + +- `StatusLogLine` +- `LoggerPlugin` + +This module defines the contract between internal logging and pluggable logging backends. + +#### `StatusLogLine` + +Represents structured status logs: + +- Severity (`O_INFO`, `O_WARNING`, `O_ERROR`, `O_FATAL`) +- Filename and line number +- Message +- Timestamps (UNIX and calendar) +- Host identifier + +These are buffered during early startup and delivered to the logger during initialization. + +--- + +### `LoggerPlugin` + +A pluggable interface allowing integration with: + +- Syslog +- Splunk +- Kafka pipelines +- Cloud logging systems + +Core extension points: + +- `logString()` (mandatory) +- `logStatus()` (optional) +- `logSnapshot()` (optional) +- `logEvent()` (optional) +- `logStringBatch()` (batch event handling) + +Feature negotiation: + +```mermaid +flowchart LR + Logger["LoggerPlugin"] -->|"usesLogStatus()"| StatusHandling["Handle Glog Status"] + Logger -->|"usesLogEvent()"| EventHandling["Handle Event Forwarding"] + Logger -->|"logSnapshot()"| SnapshotHandling["Snapshot Optimization"] +``` + +The logger can opt into: + +- Handling Glog statuses directly +- Receiving individual events +- Receiving batched events +- Handling large snapshot payloads separately + +This decouples core execution from output transport. + +--- + +## End-to-End Data Flow + +```mermaid +flowchart TD + Config["Configuration And Packs"] --> SchedulerExec["Scheduled Query"] + SchedulerExec --> SQLEngine["SQL Engine"] + SQLEngine --> QueryState["Query"] + QueryState --> DiffLayer["DiffResults"] + QueryState --> Perf["QueryPerformance"] + DiffLayer --> LogItem["QueryLogItem"] + Perf --> LogItem + LogItem --> Serializer["JSON Serialization"] + Serializer --> Logger["LoggerPlugin"] + Logger --> Sink["External Sink"] + QueryState --> Database["Database Backend"] +``` + +--- + +## Interaction With Other Modules + +This module collaborates closely with: + +- **Configuration And Packs** (scheduled query definitions) +- **SQL Engine And Virtual Tables** (query execution) +- **Database Backend** (persistent historical state) +- **Distributed Querying** (remote execution results) +- **Extensions Framework** (custom logger plugins) + +It intentionally does not: + +- Parse SQL +- Implement storage engines +- Perform network transport directly + +Instead, it standardizes metadata and logging semantics. + +--- + +## Design Principles + +### 1. Differential Logging First +Default behavior minimizes output volume by logging only changes. + +### 2. Snapshot Flexibility +Queries can explicitly request snapshot semantics. + +### 3. Pluggable Logging +Logging transport is fully decoupled via plugin interfaces. + +### 4. Persistent State Awareness +Every scheduled query execution is contextualized by: + +- Epoch (configuration version) +- Counter (execution count) +- Previous result set + +### 5. Observability Built-In +Performance metrics are tracked alongside result data. + +--- + +## Summary + +The **Logging And Query Metadata** module is the structured observability layer of the system. + +It: + +- Tracks query execution metadata +- Computes result deltas +- Serializes structured logs +- Maintains historical state +- Exposes extensible logging interfaces + +By separating execution, storage, and transport concerns, it enables scalable, efficient, and flexible logging architectures while preserving rich metadata for auditing, debugging, and analytics. \ No newline at end of file diff --git a/docs/reference/architecture/logging-and-query-observability/logging-and-query-observability.md b/docs/reference/architecture/logging-and-query-observability/logging-and-query-observability.md deleted file mode 100644 index cf98d585c16..00000000000 --- a/docs/reference/architecture/logging-and-query-observability/logging-and-query-observability.md +++ /dev/null @@ -1,409 +0,0 @@ -# Logging And Query Observability - -## Overview - -The **Logging And Query Observability** module is responsible for transforming raw query executions and internal status events into structured, serialized, and transport-ready log artifacts. - -It sits at the intersection of: - -- The **SQL execution layer** (see [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md)) -- The **persistent state layer** (see [Database And Storage Plugins](../database-and-storage-plugins/database-and-storage-plugins.md)) -- The **configuration and scheduling layer** (see [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md)) -- The **pluggable logging infrastructure** - -This module provides: - -- Query result diffing and snapshot handling -- Query execution metadata and performance tracking -- Structured log serialization (JSON, event-based) -- Pluggable logger interface for external sinks (TLS, filesystem, syslog, etc.) -- Status log forwarding and event streaming support - ---- - -## Architectural Position - -```mermaid -flowchart TD - Config["Configuration And Packs"] --> Scheduler["Scheduled Query"] - Scheduler --> SQ["ScheduledQuery Struct"] - SQ --> QueryObj["Query State Manager"] - QueryObj --> DB["Database And Storage Plugins"] - - QueryObj --> Diff["DiffResults Engine"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serializer["JSON Serialization"] - Serializer --> Logger["LoggerPlugin"] - - StatusLogs["Internal Status Events"] --> StatusLine["StatusLogLine"] - StatusLine --> Logger - - Logger --> External["External Logging Backend"] -``` - -The Logging And Query Observability module acts as a transformation and routing layer: - -1. Queries execute. -2. Results are compared with historical state. -3. Changes are wrapped in structured log items. -4. Logs are serialized. -5. A `LoggerPlugin` implementation forwards them upstream. - ---- - -## Core Components - -This module consists of five primary building blocks: - -- **LoggerPlugin** and **StatusLogLine** – Pluggable logging interface -- **QueryLogItem** – Structured representation of query output -- **DiffResults** – Result set comparison engine -- **QueryPerformance** – Execution metrics and telemetry -- **ScheduledQuery** – Query scheduling metadata model - -Each component addresses a specific concern in the observability pipeline. - ---- - -## LoggerPlugin And StatusLogLine - -### Purpose - -The logging subsystem provides a pluggable abstraction for forwarding: - -- Query results -- Snapshot outputs -- Event batches -- Internal status logs - -### StatusLogLine - -`StatusLogLine` represents a single internal status entry emitted by the runtime. It includes: - -- Severity (`O_INFO`, `O_WARNING`, `O_ERROR`, `O_FATAL`) -- Source file and line number -- Human-readable message -- UNIX timestamp and calendar time -- Host identifier - -This structure decouples internal logging (e.g., Glog) from external log sinks. - -### LoggerPlugin - -`LoggerPlugin` is an abstract plugin interface that allows custom logging backends. - -Key extensibility points: - -- `logString` (required) – Primary logging entrypoint -- `logStatus` – Handles structured status logs -- `logSnapshot` – Special handling for large snapshot queries -- `logEvent` / `logStringBatch` – Direct event forwarding -- `usesLogStatus` – Opt-in takeover of internal status handling -- `usesLogEvent` – Opt-in direct event streaming - -### Logger Feature Flags - -Logger plugins can advertise support for: - -- `LOGGER_FEATURE_LOGSTATUS` -- `LOGGER_FEATURE_LOGEVENT` - -This enables selective routing decisions at runtime. - -### Logger Lifecycle - -```mermaid -flowchart TD - Init["System Initialization"] --> Buffer["Buffer Early Status Logs"] - Buffer --> LoadConfig["Load Configuration"] - LoadConfig --> Discover["Discover Logger Plugin"] - Discover --> InitLogger["LoggerPlugin.init()"] - InitLogger --> Flush["Flush Buffered StatusLogLine Entries"] - Flush --> Runtime["Runtime Logging"] -``` - -Before the logger initializes, early status logs are buffered and later flushed during `init()`. - -This ensures no observability gaps during startup. - ---- - -## ScheduledQuery - -`ScheduledQuery` models the runtime attributes of a scheduled query. - -Key attributes: - -- `pack_name` – Configuration pack source -- `name` – Unique identifier -- `query` – SQL statement -- `interval` – Execution frequency -- `startup_priority` – Startup execution control -- `denylisted` – Configuration-level suppression -- `options` – Behavioral flags (e.g., snapshot, removed) - -### Behavioral Flags - -- `snapshot` β†’ Always log full results -- `removed` β†’ Control reporting of removed rows - -This structure originates from configuration parsing (see [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md)) and feeds directly into execution and logging logic. - ---- - -## Query State Management - -### Query - -The `Query` class maintains persistent state for a scheduled query. - -Responsibilities: - -- Retrieve previous results from storage -- Compute diffs between executions -- Track query epoch -- Maintain execution counter -- Persist updated result sets - -It interacts with the database abstraction (see [Database And Storage Plugins](../database-and-storage-plugins/database-and-storage-plugins.md)). - -### Differential Execution Model - -```mermaid -flowchart LR - Old["Previous Results"] --> DiffEngine["diff()"] - New["Current Results"] --> DiffEngine - DiffEngine --> Added["Added Rows"] - DiffEngine --> Removed["Removed Rows"] -``` - -The `addNewResults` method: - -1. Loads historical state. -2. Computes a `DiffResults` structure. -3. Updates persistent storage. -4. Returns differential output for logging. - -This avoids repeatedly logging identical datasets and reduces log volume. - ---- - -## DiffResults - -`DiffResults` represents the delta between two query executions. - -Structure: - -- `added` – Rows present in new results but not old -- `removed` – Rows present in old results but not new - -### Properties - -- Move-only semantics (efficient handling of large result sets) -- Equality comparison support -- JSON serialization helpers - -### Serialization Flow - -```mermaid -flowchart TD - Diff["DiffResults"] --> Serialize["serializeDiffResults()"] - Serialize --> JSONDoc["JSON Document"] - JSONDoc --> Logger["LoggerPlugin"] -``` - -Diff serialization ensures: - -- Stable output format -- Optional numeric preservation -- Compatibility with downstream analytics pipelines - ---- - -## QueryLogItem - -`QueryLogItem` is the canonical structured representation of a query execution. - -It includes: - -- `isSnapshot` – Snapshot vs differential indicator -- `results` – `DiffResults` -- `snapshot_results` – Full results (if snapshot) -- `name` – Query name -- `identifier` – Host ID -- `time` / `calendar_time` -- `epoch` -- `counter` -- `decorations` – Arbitrary metadata - -### Equality Semantics - -Two `QueryLogItem` instances are equal if: - -- Their `DiffResults` match -- Their query names match - -This enables deduplication logic in higher-level processing. - -### Serialization Variants - -The module supports multiple output formats: - -1. Structured JSON object -2. JSON string -3. Event-style records (split per action) - -```mermaid -flowchart TD - Item["QueryLogItem"] --> JSON1["serializeQueryLogItem()"] - Item --> JSON2["serializeQueryLogItemJSON()"] - Item --> Events1["serializeQueryLogItemAsEvents()"] - Item --> Events2["serializeQueryLogItemAsEventsJSON()"] -``` - -Event-based serialization allows each row addition/removal to be emitted as an individual log record. - ---- - -## QueryPerformance - -`QueryPerformance` provides execution telemetry. - -Metrics tracked: - -- Total executions -- Last execution time -- Wall time (total and last) -- User/system CPU time -- Memory usage -- Output size - -### CSV Serialization - -Performance data is stored and transferred using a CSV representation: - -- Constructor accepts CSV -- `toCSV()` exports current state - -This lightweight format reduces overhead in persistent storage. - -### Observability Use Cases - -- Detect slow queries -- Identify resource-heavy packs -- Enforce denylisting via configuration -- Feed monitoring dashboards - ---- - -## End-To-End Query Logging Flow - -```mermaid -flowchart TD - Config["ScheduledQuery"] --> Exec["SQL Execution"] - Exec --> Results["QueryDataTyped"] - Results --> QueryState["Query.addNewResults()"] - QueryState --> Diff["DiffResults"] - Diff --> LogItem["QueryLogItem"] - LogItem --> Serialize["JSON Serialization"] - Serialize --> Logger["LoggerPlugin"] - Logger --> Sink["External Log System"] -``` - -### Snapshot Query Flow - -```mermaid -flowchart TD - SQ["ScheduledQuery snapshot=true"] --> Exec["Execute Query"] - Exec --> Full["Full Result Set"] - Full --> LogItem["QueryLogItem isSnapshot=true"] - LogItem --> Logger["logSnapshot()"] -``` - -Snapshot queries bypass diff computation and emit complete result sets. - ---- - -## Relationship With Other Modules - -### SQL Engine And Virtual Tables - -Query execution originates from the SQL engine layer: - -- Virtual table resolution -- SQLite execution -- Result materialization - -See: [SQL Engine And Virtual Tables](../sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md) - -### Database And Storage Plugins - -Persistent state for query history and counters is managed by the storage abstraction. - -See: [Database And Storage Plugins](../database-and-storage-plugins/database-and-storage-plugins.md) - -### Configuration And Packs - -Scheduled queries and behavioral flags originate from configuration packs. - -See: [Configuration And Packs](../configuration-and-packs/configuration-and-packs.md) - -### Distributed Querying - -When queries are executed via distributed frameworks, results still flow through `QueryLogItem` and the logger interface. - -See: [Distributed Querying](../distributed-querying/distributed-querying.md) - ---- - -## Design Principles - -### 1. Pluggability - -Logging is abstracted via `LoggerPlugin` to allow: - -- TLS streaming -- Filesystem logging -- Syslog integration -- Custom enterprise pipelines - -### 2. Efficiency - -- Differential logging reduces noise -- Move semantics for result sets -- CSV encoding for performance stats - -### 3. Deterministic Serialization - -All log structures have explicit JSON serializers to guarantee: - -- Backward compatibility -- Stable schema -- Predictable downstream parsing - -### 4. Observability-First Model - -Every query execution produces: - -- State delta -- Execution metadata -- Performance metrics -- Host attribution - -This ensures complete auditability of query behavior. - ---- - -## Summary - -The **Logging And Query Observability** module transforms raw SQL execution results and runtime status events into structured, serialized, and pluggable log outputs. - -It provides: - -- Differential result tracking (`DiffResults`) -- Query metadata encapsulation (`QueryLogItem`) -- Performance telemetry (`QueryPerformance`) -- Scheduling metadata (`ScheduledQuery`) -- Extensible logging backends (`LoggerPlugin`) - -By cleanly separating execution, state management, serialization, and transport, this module forms the backbone of osquery’s observability and audit pipeline. \ No newline at end of file diff --git a/docs/reference/architecture/osquery-shell-devtooling/osquery-shell-devtooling.md b/docs/reference/architecture/osquery-shell-devtooling/osquery-shell-devtooling.md new file mode 100644 index 00000000000..8e1ed61e43c --- /dev/null +++ b/docs/reference/architecture/osquery-shell-devtooling/osquery-shell-devtooling.md @@ -0,0 +1,353 @@ +# Osquery Shell Devtooling + +## Overview + +The **Osquery Shell Devtooling** module implements the interactive `osqueryi` shell experience. It provides: + +- An interactive SQLite-based SQL console +- Multiple output rendering modes (pretty, column, line, list, CSV, JSON) +- Meta-commands (e.g., `.tables`, `.schema`, `.show`, `.connect`) +- Integration with extensions and remote query execution +- Runtime profiling via CPU and wall-clock timing + +This module acts as a developer-facing entry point into the osquery runtime, bridging the SQL engine, registry, configuration system, extensions framework, and database backend into a cohesive CLI tool. + +Core components documented here: + +- `callback_data` +- `prettyprint_data` +- `rusage` (platform-dependent CPU accounting structure) + +--- + +## Architectural Position + +The shell sits on top of the SQL engine and database manager while optionally delegating execution to extensions. + +```mermaid +flowchart TD + User["User Terminal"] --> Shell["Osquery Shell Devtooling"] + Shell --> Parser["Meta Command Parser"] + Shell --> Executor["SQL Executor"] + + Executor --> SQLite["SQLite Engine"] + SQLite --> VTables["Virtual Tables"] + VTables --> Registry["Table Registry"] + + Executor --> DBManager["SQLiteDBManager"] + Executor --> Extensions["Extension Manager Client"] + + Extensions --> RemoteExt["Remote Extension Socket"] +``` + +### Key Responsibilities + +1. Accept and parse interactive input +2. Distinguish SQL statements from meta-commands +3. Execute SQL locally or remotely +4. Format and render result sets +5. Provide runtime diagnostics and environment introspection + +--- + +## Execution Flow + +The shell processes user input in a loop, accumulating SQL until it forms a complete statement or identifying a meta-command. + +```mermaid +flowchart TD + A["Read Input Line"] --> B{"Starts With Dot?"} + B -->|"Yes"| C["Meta Command Handler"] + B -->|"No"| D["Accumulate SQL"] + D --> E{"Complete Statement?"} + E -->|"No"| A + E -->|"Yes"| F["Execute SQL"] + F --> G["shell_callback"] + G --> H["Render Output Mode"] + H --> A +``` + +### Major Functions + +- `process_input` – Main input loop +- `do_meta_command` – Parses and dispatches dot commands +- `shell_exec` – Executes SQL locally +- `shell_exec_remote` – Executes SQL through extensions +- `shell_callback` – Row-level result handler + +--- + +## Core Data Structures + +### callback_data + +`callback_data` is the primary state container for the shell runtime. + +It maintains: + +- Output mode (`MODE_Pretty`, `MODE_Column`, etc.) +- Header visibility +- Field separator +- Column widths +- Null display value +- Output file handle +- Current prepared statement +- Pretty-print state + +```mermaid +classDiagram + class callback_data { + +int echoOn + +int mode + +int showHeader + +char separator[20] + +char nullvalue[20] + +FILE* out + +sqlite3_stmt* pStmt + +prettyprint_data* prettyPrint + } +``` + +This structure is passed through execution layers and into `shell_callback`, allowing output behavior to remain configurable and consistent. + +--- + +### prettyprint_data + +`prettyprint_data` buffers result rows before formatting when in `MODE_Pretty`. + +It contains: + +- `QueryData results` – Collected rows +- `vector columns` – Column order +- `map lengths` – Computed width per column + +```mermaid +classDiagram + class prettyprint_data { + +QueryData results + +vector columns + +map lengths + } + + callback_data --> prettyprint_data +``` + +This buffering allows the shell to compute column widths dynamically before rendering formatted output. + +--- + +### rusage + +The `rusage` structure tracks CPU time for the `.timer` feature. + +- On Unix: uses `getrusage` +- On Windows: wraps `FILETIME` + +The timing workflow: + +```mermaid +flowchart TD + Start["BEGIN_TIMER"] --> CaptureStart["Capture rusage + Wall Time"] + CaptureStart --> Execute["Run Query"] + Execute --> CaptureEnd["Capture End rusage"] + CaptureEnd --> Print["Print real, user, sys"] +``` + +The timer reports: + +- Real (wall-clock) time +- User CPU time +- System CPU time + +--- + +## Output Modes + +The shell supports multiple rendering strategies. + +| Mode | Description | +|------|-------------| +| Pretty | Default aligned table format | +| Column | Fixed-width columns | +| Line | One column per line | +| List | Separator-delimited rows | +| CSV | Quoted CSV output | +| JSON | JSON output via flags | + +### Rendering Strategy + +All result rows are passed through: + +```text +shell_exec β†’ shell_callback β†’ output mode switch β†’ formatting logic +``` + +In Pretty mode: + +1. Rows are buffered +2. Column widths are computed +3. Final formatted output is rendered +4. Buffers are cleared + +--- + +## Local vs Remote Execution + +The shell can execute queries either: + +- Directly against the embedded SQLite database +- Through an extension socket using the extension client + +```mermaid +flowchart LR + Query["SQL Query"] --> Check{"Connect Flag Set?"} + Check -->|"No"| LocalExec["shell_exec Local"] + Check -->|"Yes"| RemoteExec["shell_exec_remote"] + RemoteExec --> ExtClient["ExtensionManagerClient"] +``` + +### Remote Path + +- Connect via `FLAGS_connect` +- Use `ExtensionManagerClient` +- Retrieve rows and column types +- Replay results through `shell_callback` + +This design preserves formatting behavior regardless of execution backend. + +--- + +## Meta Commands + +Meta-commands begin with `.` and are handled separately from SQL. + +Examples: + +- `.tables` – List registered tables +- `.schema` – Show CREATE statements +- `.show` – Display runtime configuration +- `.features` – Show enable or disable flags +- `.connect` / `.disconnect` – Manage extension sockets +- `.timer ON` – Enable profiling + +### Meta Command Processing + +```mermaid +flowchart TD + Input[".command args"] --> Tokenize["Tokenize Arguments"] + Tokenize --> Match["Command Switch"] + Match --> Execute["Meta Handler Function"] + Execute --> Output["Print To Shell"] +``` + +Meta commands may: + +- Interact with the registry +- Query internal virtual tables +- Adjust shell runtime flags +- Invoke SQL indirectly + +--- + +## Input Handling and History + +Interactive input uses `linenoise` for: + +- Line editing +- Command history +- Table name completion + +Completion is implemented by querying the table registry and matching prefixes. + +Non-interactive mode supports piped input and file-driven execution. + +--- + +## Integration with the Larger System + +The shell integrates tightly with: + +- SQL Engine and Virtual Tables +- Database Backend (via `SQLiteDBManager`) +- Extensions Framework +- Configuration and Packs (via `runPack`) +- Flags and Runtime Options + +### Pack Execution + +When `--pack` is provided: + +```mermaid +flowchart TD + Start["Launch With Pack Flag"] --> LoadConfig["Load Packs From Config"] + LoadConfig --> Iterate["Iterate Scheduled Queries"] + Iterate --> Run["runQuery"] + Run --> Output["Formatted Results"] +``` + +This allows the shell to function as a lightweight pack executor for development and debugging. + +--- + +## Error Handling + +Errors are managed at multiple levels: + +- SQLite prepare and step failures +- Extension RPC failures +- Incomplete SQL detection +- Interrupt handling via SIGINT + +Behavior is influenced by: + +- `.bail ON|OFF` +- Interactive vs non-interactive mode + +--- + +## Key Design Characteristics + +### 1. Stateful Rendering Model + +All output behavior is driven by `callback_data`, enabling: + +- Consistent formatting +- Mode switching at runtime +- File redirection support + +### 2. Backend-Agnostic Execution + +The shell abstracts execution so that: + +- Local SQLite +- Remote extension queries + +produce identical output paths. + +### 3. Developer-Centric Tooling + +Features like: + +- `.types` +- `.features` +- `.show` +- `.timer` + +make the shell a debugging surface for the entire osquery stack. + +--- + +## Summary + +The **Osquery Shell Devtooling** module provides a fully featured interactive SQL interface on top of the osquery runtime. + +It: + +- Bridges user input to the SQL engine +- Supports local and extension-based execution +- Implements flexible and extensible output formatting +- Integrates with configuration, packs, and flags +- Provides runtime diagnostics and profiling + +Architecturally, it is the developer-facing gateway into osquery’s SQL engine, virtual tables, extension framework, and database layer β€” combining them into a cohesive, interactive experience. \ No newline at end of file diff --git a/docs/reference/architecture/remote-http-client/remote-http-client.md b/docs/reference/architecture/remote-http-client/remote-http-client.md index 83b5f95de29..77c72143d13 100644 --- a/docs/reference/architecture/remote-http-client/remote-http-client.md +++ b/docs/reference/architecture/remote-http-client/remote-http-client.md @@ -1,345 +1,310 @@ # Remote Http Client -The **Remote Http Client** module provides a general-purpose HTTP and HTTPS client implementation used across the system for secure remote communication. Built on top of Boost.Asio and Boost.Beast, it enables synchronous-style HTTP operations backed by asynchronous networking primitives, with full TLS support, timeout handling, redirect control, and proxy configuration. +## Overview -This module acts as the foundational networking layer for components that need to communicate with remote services, such as: +The **Remote Http Client** module provides a general-purpose HTTP and HTTPS client implementation built on top of Boost.Asio and Boost.Beast. It is responsible for outbound network communication between osquery and remote services such as configuration endpoints, logging backends, distributed query servers, and metadata services. -- [Distributed Querying](distributed-querying/distributed-querying.md) -- [Logging and Query Observability](logging-and-query-observability/logging-and-query-observability.md) -- [Configuration and Packs](configuration-and-packs/configuration-and-packs.md) - -It abstracts low-level socket management, TLS setup, request serialization, and response parsing into a reusable and secure HTTP client interface. - ---- +This module abstracts: -## 1. Purpose and Responsibilities +- TCP connection management +- TLS/SSL negotiation and certificate handling +- HTTP request/response lifecycle +- Header manipulation and URI parsing +- Timeout and error handling +- Optional proxy support -The Remote Http Client module is responsible for: +It serves as a foundational networking layer used by higher-level modules such as: -- Creating and managing TCP and TLS connections -- Executing HTTP methods: `GET`, `POST`, `PUT`, `HEAD`, `DELETE` -- Managing request headers and body serialization -- Parsing HTTP responses -- Enforcing timeouts and connection lifecycle rules -- Handling TLS verification, certificates, and cipher configuration -- Supporting optional proxy routing -- Supporting redirect and keep-alive behavior - -It provides a high-level API while internally orchestrating asynchronous Boost networking operations. +- [Distributed TLS Plugin](distributed-tls-plugin/distributed-tls-plugin.md) +- [Distributed Querying](distributed-querying/distributed-querying.md) +- [Config Plugins](config-plugins/config-plugins.md) --- -## 2. High-Level Architecture - -The module is centered around three main abstractions: +## Architectural Context -- `Client` – Connection lifecycle and HTTP execution engine -- `HTTP_Request` – URI-aware request wrapper -- `HTTP_Response` – Response wrapper with header iteration support - -### 2.1 Architectural Overview +Within the osquery architecture, the Remote Http Client sits below plugin implementations that require remote communication but above the raw networking libraries (Boost.Asio, OpenSSL). ```mermaid flowchart TD - Caller["Caller Module"] --> Client["Client"] - Client --> Options["Client::Options"] - Client --> Request["HTTP_Request"] - Client --> Response["HTTP_Response"] - - Client --> Resolver["TCP Resolver"] - Client --> Socket["TCP Socket"] - Client --> TLSSocket["SSL Stream"] - Client --> Timer["Deadline Timer"] - - Request --> URI["URI Parser"] - Response --> Headers["Header Iterator"] + ConfigPlugins["Config Plugins"] -->|"Fetch config"| RemoteHttpClient["Remote Http Client"] + DistributedTLS["Distributed TLS Plugin"] -->|"Send queries"| RemoteHttpClient + DistributedQuerying["Distributed Querying"] -->|"Submit results"| RemoteHttpClient + + RemoteHttpClient -->|"TCP"| Asio["Boost Asio"] + RemoteHttpClient -->|"HTTP"| Beast["Boost Beast"] + RemoteHttpClient -->|"TLS"| OpenSSL["OpenSSL"] ``` -The `Client` orchestrates the network stack and delegates URI parsing to `HTTP_Request`, while wrapping Boost.Beast responses inside `HTTP_Response`. +The module provides a clean interface for sending HTTP requests while encapsulating: + +- Asynchronous IO coordination +- SSL context configuration +- Request serialization and response parsing +- Timeout enforcement --- -## 3. Core Components +## Core Components -### 3.1 Client +The Remote Http Client module revolves around the following key abstractions: -The `Client` class implements a general-purpose HTTP/HTTPS client built on: +### 1. Client -- `boost::asio::io_context` -- `boost::asio::ip::tcp::resolver` -- `boost::asio::ip::tcp::socket` -- `boost::asio::ssl::stream` -- `boost::beast::http` +`osquery::http::Client` is the main entry point for executing HTTP operations. -It exposes the following public methods: +It supports: -- `get(Request&)` -- `post(Request&, body, content_type)` -- `put(Request&, body, content_type)` -- `head(Request&)` -- `delete_(Request&)` +- `GET` +- `POST` +- `PUT` +- `HEAD` +- `DELETE` Each method: -1. Initializes the request -2. Establishes or reuses a connection -3. Sends the request asynchronously -4. Waits for response or timeout -5. Returns a `Response` object -The destructor ensures that open sockets are closed safely. +- Accepts a `Request` +- Optionally accepts a body and content type +- Returns a `Response` by value + +The Client encapsulates: + +- `boost::asio::io_context` +- TCP resolver +- TCP socket +- Optional TLS socket wrapper +- Deadline timer for timeouts --- -### 3.2 Client Options +### 2. Client Options -The `Client::Options` class drives client behavior. +The nested `Client::Options` class controls behavior and security settings. -It supports configuration for: +Supported configuration areas: -- TLS enablement (`ssl_connection`) -- Keep-alive behavior -- Redirect following -- Peer verification enforcement -- Timeout configuration +- TLS enable/disable (`ssl_connection`) +- Certificate validation (`always_verify_peer`) +- Custom CA verification path +- Client certificate and private key - Cipher selection -- Certificate and private key files -- CA verify path - Proxy hostname -- Remote hostname and port overrides - -#### Options Comparison +- Remote hostname override +- Remote port override +- Timeout configuration +- Redirect handling +- Keep-alive support -The equality operator allows the `Client` to detect configuration changes: +Options are compared for equality to determine whether connection state must be refreshed. ```mermaid flowchart LR - OldOpts["Current Options"] --> Compare["operator=="] - NewOpts["New Options"] --> Compare - Compare -->|"Different"| Reconfigure["Reinitialize Connection"] - Compare -->|"Same"| Reuse["Reuse Existing Settings"] + Options["Client Options"] --> SSL["SSL Configuration"] + Options --> Proxy["Proxy Settings"] + Options --> Timeout["Timeout Control"] + Options --> Redirects["Redirect Handling"] + Options --> KeepAlive["Keep Alive"] ``` -This prevents unnecessary TLS reconfiguration and socket churn. +This design allows higher-level plugins to define secure communication policies without interacting directly with OpenSSL or Boost internals. --- -### 3.3 HTTP_Request - -`HTTP_Request` extends Boost.Beast request objects with: +### 3. HTTP Request -- URI parsing support -- Host, port, path extraction -- Protocol detection -- Header insertion via operator overloading +`HTTP_Request` extends Boost.Beast request types and adds URI parsing capabilities. -#### URI Handling +Key features: -The request wraps an internal `Uri` object and exposes: +- Parses full URLs via `osquery::Uri` +- Extracts host, port, path, query, fragment +- Determines protocol (http or https) +- Supports header injection via operator overloading -- `remoteHost()` -- `remotePort()` -- `remotePath()` -- `protocol()` +Example header usage conceptually: -This allows the `Client` to resolve and connect without requiring manual URL parsing. +- Create `Header(name, value)` +- Stream into request using `operator<<` -#### Header Injection +This allows concise header manipulation while maintaining strong typing. -Headers are added using a helper struct: - -```text -Request req("https://example.com/api"); -req << HTTP_Request::Header("Authorization", "Bearer token"); +```mermaid +flowchart TD + Request["HTTP Request"] --> URI["URI Parser"] + Request --> Headers["Header Injection"] + Request --> Protocol["Scheme Detection"] ``` -This keeps header logic encapsulated within the request abstraction. - --- -### 3.4 HTTP_Response +### 4. HTTP Response -`HTTP_Response` extends Boost.Beast response objects and provides: +`HTTP_Response` wraps the Beast response and provides: -- `status()` – numeric HTTP status -- `body()` – response body string -- `headers()` – iterable header collection +- `status()` for HTTP status code +- `body()` for response payload +- `headers()` iterator interface -#### Header Iteration Model +The nested `Headers` helper exposes: + +- Indexed header access +- Iterable header enumeration ```mermaid flowchart TD - Response["HTTP_Response"] --> Headers["Headers"] - Headers --> Iterator["Iterator"] - Iterator --> Pair["(name, value)"] + Response["HTTP Response"] --> Status["Status Code"] + Response --> Body["Response Body"] + Response --> HeaderAccess["Header Iterator"] ``` -Example usage pattern: - -```text -for (const auto& header : resp.headers()) { - header.first; - header.second; -} -``` - -This abstraction avoids exposing raw Boost iterators directly. - --- -## 4. Request Lifecycle - -The `Client` wraps asynchronous Boost operations into a controlled flow. - -### 4.1 End-to-End Flow - -```mermaid -sequenceDiagram - participant Caller - participant Client - participant Resolver - participant Server - - Caller->>Client: get(Request) - Client->>Resolver: async_resolve() - Resolver-->>Client: endpoints - Client->>Server: async_connect() - Client->>Server: async_write(request) - Server-->>Client: HTTP response - Client->>Server: async_read() - Client-->>Caller: Response -``` - -### 4.2 TLS Flow (HTTPS) +## Request Lifecycle -If `ssl_connection` is enabled: +The Client orchestrates a structured asynchronous workflow: ```mermaid flowchart TD - Connect["TCP Connect"] --> TLSCheck{"SSL Enabled?"} - TLSCheck -->|"Yes"| Handshake["TLS Handshake"] - TLSCheck -->|"No"| Plain["Plain HTTP"] - Handshake --> Send["Send Request"] - Plain --> Send + Start["Start Request"] --> Init["Initialize HTTP Request"] + Init --> Resolve["DNS Resolve"] + Resolve --> Connect["TCP Connect"] + Connect --> TLSCheck{{"TLS Enabled?"}} + TLSCheck -->|"Yes"| Handshake["SSL Handshake"] + TLSCheck -->|"No"| Write + Handshake --> Write["Async Write"] + Write --> Read["Async Read"] + Read --> Complete["Post Response Handler"] + Complete --> EndNode["END"] ``` -TLS configuration respects: +### Key Internal Mechanisms -- Cipher restrictions -- Verify path -- Server certificate pinning -- Client certificate and private key +- `createConnection()` selects direct or proxy connection +- `encryptConnection()` upgrades TCP socket to SSL stream +- `callNetworkOperation()` wraps async calls and timeout control +- `timeoutHandler()` aborts operations if deadline expires +- `postResponseHandler()` treats certain TLS short-read conditions as success -Legacy SSL versions and MD5 are explicitly disabled at compile time. +This architecture ensures: ---- +- Non-blocking network behavior +- Deterministic timeout handling +- Controlled error propagation -## 5. Timeout and Error Handling +--- -The client uses a `boost::asio::deadline_timer` to enforce network timeouts. +## Timeout and Error Handling Model -### 5.1 callNetworkOperation Wrapper +The module uses a `boost::asio::deadline_timer` to enforce timeouts. -The `callNetworkOperation` method: +Network operations are wrapped inside `callNetworkOperation()` which: 1. Starts the timeout timer 2. Executes the async operation -3. Waits for completion or timeout -4. Cancels the opposing operation -5. Sets the final error code +3. Waits for either completion or timeout +4. Cancels the alternate event -```mermaid -flowchart TD - Start["Start Network Call"] --> Timer["Start Deadline Timer"] - Timer --> Async["Async Operation"] - Async --> Complete{"Completed?"} - Complete -->|"Yes"| CancelTimer["Cancel Timer"] - Complete -->|"No"| Timeout["Timer Fires"] - Timeout --> Abort["Abort Socket"] - CancelTimer --> Return["Return Response"] - Abort --> Return -``` +Error codes are stored in an internal `boost::system::error_code` field. -### 5.2 TLS Short Read Handling +This ensures: -During TLS shutdown, some servers may not perform proper `shutdown()` calls. The client treats certain short-read conditions as success in post-response handling to avoid false-negative errors. +- Unified timeout behavior +- Safe cancellation +- Clear separation between network and protocol errors --- -## 6. Proxy and Metadata Support +## TLS and Security Model -### 6.1 Proxy Routing +Security behavior is controlled via `Client::Options`. -If a proxy hostname is configured: +Security capabilities include: -- Connection is established to the proxy -- Request routing logic is adapted accordingly +- Disable legacy SSL versions +- Custom OpenSSL cipher configuration +- Client certificate authentication +- Custom CA verification paths +- Enforced peer verification -This enables deployment behind corporate proxies. +The module explicitly disables insecure OpenSSL features such as SSLv2, SSLv3, MD5, and deprecated APIs. -### 6.2 Cloud Metadata Authority +Special handling is included for cloud metadata endpoints such as: -The constant `kInstanceMetadataAuthority` (`169.254.169.254`) identifies the authority used by cloud metadata services (e.g., EC2, Azure). +- `169.254.169.254` -This allows safe and consistent access to instance metadata endpoints when required by higher-level modules. +This authority constant supports integration with instance metadata services. --- -## 7. Interaction with Other Modules +## Interaction with Other Modules -The Remote Http Client module acts as an infrastructure dependency for modules that require outbound HTTP communication. +### Distributed TLS Plugin -### 7.1 Distributed Querying +The [Distributed TLS Plugin](distributed-tls-plugin/distributed-tls-plugin.md) uses the Remote Http Client to: -The [Distributed Querying](distributed-querying/distributed-querying.md) module typically relies on HTTP transport (e.g., TLS-based distributed plugins) to: - -- Fetch remote queries +- Fetch distributed queries - Submit query results +- Maintain secure TLS sessions -The Remote Http Client provides the secure communication backbone for these operations. - -### 7.2 Logging and Query Observability +### Distributed Querying -The [Logging and Query Observability](logging-and-query-observability/logging-and-query-observability.md) module may use HTTP/TLS transports for remote log submission. +The [Distributed Querying](distributed-querying/distributed-querying.md) module relies on this client for: -The client ensures: +- Remote query retrieval +- Result delivery to orchestration servers -- TLS enforcement -- Certificate verification -- Controlled timeout behavior +### Config Plugins -### 7.3 Configuration and Packs +The [Config Plugins](config-plugins/config-plugins.md) leverage this module to: -The [Configuration and Packs](configuration-and-packs/configuration-and-packs.md) module can retrieve configuration from remote endpoints via HTTP plugins. The Remote Http Client supplies: +- Fetch remote configuration +- Support dynamic config refresh -- Secure GET/POST support -- Redirect handling -- Proxy awareness +By isolating HTTP behavior in this module, higher-level components remain focused on business logic rather than networking. --- -## 8. Security Model +## Design Characteristics + +### 1. Abstraction Over Boost + +Consumers do not interact with: + +- Boost.Asio sockets +- SSL stream configuration +- HTTP parser internals + +All networking complexity is encapsulated. -The module enforces multiple security measures: +### 2. Explicit Security Configuration -- SSLv2 and SSLv3 disabled -- MD5 disabled -- Deprecated OpenSSL features disabled -- Optional strict peer verification -- Optional custom cipher suites -- Support for certificate pinning +TLS behavior is not implicit. It must be enabled and configured via options. -Security behavior is fully driven by `Client::Options`, allowing different modules to enforce different policies. +### 3. Stateless Usage Model + +Each request: + +- Initializes request state +- Establishes connection if needed +- Executes operation +- Cleans up socket on destruction + +### 4. Header and URI Convenience + +The extended request/response wrappers provide ergonomic access while preserving Beast compatibility. --- -## 9. Summary +## Summary -The **Remote Http Client** module provides a robust, TLS-capable HTTP client abstraction built on Boost.Asio and Boost.Beast. It encapsulates: +The **Remote Http Client** module provides the secure, asynchronous HTTP transport layer for osquery. It abstracts Boost networking primitives and OpenSSL configuration into a clean, high-level interface used by configuration, logging, and distributed execution subsystems. -- Connection lifecycle management -- TLS negotiation and verification -- Asynchronous networking orchestration -- Timeout enforcement -- Request and response abstraction +Its responsibilities include: + +- Secure HTTP and HTTPS communication +- Timeout and error orchestration +- Proxy and redirect support +- TLS certificate management +- URI parsing and header handling -By centralizing HTTP communication logic, it ensures consistent, secure, and configurable remote connectivity across the system, serving as the networking foundation for distributed querying, logging, and remote configuration workflows. +By centralizing outbound HTTP behavior, the module ensures consistent security posture and networking semantics across the entire osquery system. \ No newline at end of file diff --git a/docs/reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md b/docs/reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md index e450a4f1d9d..4ae39df4826 100644 --- a/docs/reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md +++ b/docs/reference/architecture/sql-engine-and-virtual-tables/sql-engine-and-virtual-tables.md @@ -2,22 +2,16 @@ ## Overview -The **Sql Engine And Virtual Tables** module is the execution core of osquery. It embeds and configures SQLite as an in-memory analytical engine and exposes operating system data through dynamically attached virtual tables. +The **Sql Engine And Virtual Tables** module is the execution backbone of osquery. It embeds and manages an in-memory SQLite engine, exposes osquery tables as SQLite virtual tables, enforces SQL safety policies, and provides query planning and column type introspection. -This module is responsible for: +This module acts as the bridge between: -- Managing and optimizing SQLite database instances -- Enforcing SQL security through authorizers and allowlists -- Attaching and detaching virtual tables backed by TablePlugin implementations -- Planning and introspecting queries -- Executing SQL statements and returning typed results -- Bridging SQLite’s virtual table API with osquery’s plugin system +- The **SQLite core engine** (query parsing, planning, execution) +- The **Table Plugin framework** (C++ and extension-based tables) +- The **Registry system** (plugin discovery and invocation) +- Higher-level components such as logging, scheduling, and distributed querying -It acts as the execution layer between: - -- Table plugins (from the Registry) -- The scheduler and query interfaces -- Extensions providing additional tables +At runtime, nearly every query flows through this module. --- @@ -25,26 +19,25 @@ It acts as the execution layer between: ```mermaid flowchart TD - Client["SQL Caller (Scheduler / CLI / Extension)"] --> SQLPlugin["SQLiteSQLPlugin"] + Client["SQL Caller\nScheduler, Shell, API"] --> SQLPlugin["SQLiteSQLPlugin"] SQLPlugin --> DBManager["SQLiteDBManager"] DBManager --> DBInstance["SQLiteDBInstance"] - DBInstance --> SQLiteCore[("In-Memory SQLite Engine")] - - SQLiteCore --> VTabModule["sqlite3_module (Virtual Table Module)"] - VTabModule --> VirtualTable["VirtualTable Wrapper"] - VirtualTable --> TablePlugin["TablePlugin (Registry)"] - - SQLiteCore --> Planner["QueryPlanner"] + DBInstance --> SQLiteCore[("SQLite Engine")] + SQLiteCore --> VTableModule["sqlite3_module\nVirtual Table Layer"] + VTableModule --> TablePlugin["TablePlugin\nRegistry"] + TablePlugin --> DataSource["System Data Sources\nFilesystem, Processes, Network"] ``` -### Flow Summary +### Key Responsibilities -1. A SQL query is submitted via the SQL registry. -2. `SQLiteSQLPlugin` obtains a database connection from `SQLiteDBManager`. -3. The query is prepared and executed using `queryInternal`. -4. When a virtual table is accessed, SQLite invokes the `sqlite3_module` callbacks. -5. The virtual table implementation forwards execution to a `TablePlugin`. -6. Results are returned to SQLite and then to the caller as typed rows. +| Layer | Responsibility | +|--------|----------------| +| SQLiteSQLPlugin | Public SQL interface and plugin registration | +| SQLiteDBManager | Connection lifecycle and resource management | +| SQLiteDBInstance | Per-query SQLite wrapper with attach safety | +| QueryPlanner | Query analysis and type inference | +| sqlite3_module | SQLite virtual table implementation | +| BaseCursor / VirtualTable | Runtime cursor and table state | --- @@ -52,300 +45,283 @@ flowchart TD ### SQLiteSQLPlugin -Implements the internal `sql` registry plugin. +**Namespace:** `osquery.osquery.sql.sqlite_util.SQLiteSQLPlugin` + +The `SQLiteSQLPlugin` implements the `SQLPlugin` registry interface and provides: + +- `query()` – Execute SQL queries +- `getQueryColumns()` – Infer result column names and types +- `getQueryTables()` – Extract referenced tables +- `attach()` – Attach virtual tables dynamically +- `detach()` – Detach virtual tables -**Responsibilities:** +It is registered as an internal plugin under the name `sql`. -- Execute SQL queries (`query`) -- Introspect query columns (`getQueryColumns`) -- Determine scanned tables (`getQueryTables`) -- Attach/detach virtual tables dynamically +### Execution Flow -It acts as the primary entry point for SQL execution inside osquery. +```mermaid +flowchart TD + Start["Query Request"] --> GetDB["SQLiteDBManager.get()"] + GetDB --> UseCache["Set Cache Mode"] + UseCache --> Prepare["sqlite3_prepare_v2"] + Prepare --> Step["sqlite3_step"] + Step --> Rows["readRows()"] + Rows --> Finalize["sqlite3_finalize"] + Finalize --> Clear["Clear Affected Tables"] + Clear --> End["Return Status + Results"] +``` --- -### SQLiteDBManager +## SQLiteDBManager + +**Namespace:** `osquery.osquery.sql.sqlite_util.SQLiteDBManager` -Singleton responsible for lifecycle and concurrency management of SQLite instances. +The `SQLiteDBManager` is the central access point for SQLite resources. -**Key Features:** +### Responsibilities -- Maintains a primary in-memory SQLite database -- Creates transient database instances under contention -- Applies memory optimizations (PRAGMA tuning) -- Attaches all registered virtual tables -- Enforces table enable/disable flags +- Maintain the primary SQLite database instance +- Provide transient instances during contention +- Attach virtual tables automatically +- Enforce enabled/disabled table policies +- Manage SQLite memory limits + +### Connection Strategy ```mermaid -flowchart LR - Caller["Caller"] --> GetConn["SQLiteDBManager.get()"] - GetConn --> PrimaryCheck{"Primary Available?"} - PrimaryCheck -->|Yes| Primary["Primary SQLiteDBInstance"] - PrimaryCheck -->|No| Transient["Transient SQLiteDBInstance"] +flowchart TD + Request["get() or getConnection()"] --> CheckPrimary{"Primary DB Exists?"} + CheckPrimary -->|"No"| Init["openOptimized()"] + CheckPrimary -->|"Yes"| Reuse["Reuse Primary"] + Init --> Attach["attachVirtualTables()"] + Reuse --> ReturnPrimary["Return Primary or Transient"] + Attach --> ReturnPrimary ``` -This design minimizes resource usage while ensuring thread safety. +If contention occurs, a transient `SQLiteDBInstance` is created with its own in-memory database. --- -### SQLiteDBInstance +## SQLiteDBInstance + +**Namespace:** `osquery.osquery.sql.sqlite_util.SQLiteDBInstance` -RAII wrapper around a `sqlite3*` pointer. +A RAII wrapper around `sqlite3*` that tracks: -**Responsibilities:** +- Query cache usage +- Attached tables +- Table attributes (such as EVENT_BASED) +- Per-query constraint state -- Provide safe access to SQLite handle -- Manage locking and attach mutexes -- Track affected virtual tables per query -- Clear per-query table state -- Support query result caching +### Important Capabilities -Each query receives a scoped database instance. On destruction: +- `attachLock()` – Safe attach operations +- `addAffectedTable()` – Track accessed tables +- `clearAffectedTables()` – Reset state post-query +- `useCache()` – Enable virtual table caching -- Transient connections are closed -- Primary connections remain managed +This isolation allows safe concurrent execution across threads. --- -### Security Enforcement (Authorizer) +## SQLite Security Model -The module enforces strict SQL safety using `sqliteAuthorizer`. +The module enforces strict SQLite authorizer rules. -**Allowlisted Actions:** +### Authorizer: `sqliteAuthorizer` -- `SELECT`, `READ` -- Controlled `INSERT`, `UPDATE`, `DELETE` -- Virtual table creation and drop -- Limited `PRAGMA` commands +Allowed operations include: -**Explicitly Denied:** +- SELECT +- READ +- INSERT, UPDATE, DELETE +- CREATE/DROP TABLE or VIEW +- CREATE/DROP VIRTUAL TABLE +- FUNCTION calls +- RECURSIVE queries +- TRANSACTION -- `SQLITE_ATTACH` -- Any non-allowlisted opcode +Explicitly denied: -```mermaid -flowchart TD - Prepare["sqlite3_prepare_v2"] --> Authorizer["sqliteAuthorizer"] - Authorizer -->|Allowed| Continue["Execute Statement"] - Authorizer -->|Denied| Reject["SQLITE_DENY"] -``` +- ATTACH DATABASE +- Non-allowlisted PRAGMA calls + +If a forbidden action is attempted: + +- An error is logged +- SQLite returns `SQLITE_DENY` -This ensures osquery cannot be abused to modify arbitrary files or attach external databases. +This prevents arbitrary file writes or unsafe database attachment. --- -### QueryPlanner +## Virtual Table Integration -Performs query introspection using: +**Namespace:** `osquery.osquery.sql.virtual_table` -- `EXPLAIN QUERY PLAN` -- `EXPLAIN` +osquery tables are implemented as SQLite virtual tables. -Used for: +### Core Structures -- Inferring column types for expressions -- Determining scanned tables -- Inspecting SQLite opcodes +- `sqlite3_module` – SQLite callback table +- `VirtualTable` – Wrapper around sqlite3_vtab +- `BaseCursor` – Per-query cursor state -The planner maps SQLite opcodes to osquery `ColumnType` using `kSQLOpcodes`. +### Virtual Table Lifecycle ```mermaid flowchart TD - Query["SQL Query"] --> ExplainPlan["EXPLAIN QUERY PLAN"] - Query --> Explain["EXPLAIN"] - Explain --> OpcodeMap["kSQLOpcodes Mapping"] - OpcodeMap --> TypeInference["Apply Column Types"] + Create["xCreate or xConnect"] --> Declare["sqlite3_declare_vtab"] + Declare --> BestIndex["xBestIndex"] + BestIndex --> Filter["xFilter"] + Filter --> Next["xNext"] + Next --> Column["xColumn"] + Column --> Rowid["xRowid"] + Rowid --> Eof["xEof"] + Eof --> Close["xClose"] ``` -This enables accurate schema introspection even when SQLite cannot directly determine expression types. - --- -### SQLInternal +## Constraint Pushdown and Query Planning -A lower-level SQL execution wrapper used internally. +### xBestIndex -**Capabilities:** +`xBestIndex` evaluates WHERE constraints and determines: -- Executes queries bypassing registry lookup -- Returns typed results (`QueryDataTyped`) -- Detects event-based tables -- Escapes non-printable bytes -- Calculates result size +- Which columns are indexed +- Required column presence +- Estimated cost +- Constraint ordering -It is used when deeper inspection of query attributes is required. +It builds: ---- +- ConstraintSet +- UsedColumns +- Bitset of accessed columns -## Virtual Table Subsystem +If required constraints are missing, cost is set to a high sentinel value. -The Virtual Table subsystem connects SQLite’s virtual table API with osquery’s `TablePlugin` abstraction. +### xFilter -### VirtualTable +`xFilter`: -Wraps: - -- `sqlite3_vtab` -- `VirtualTableContent` (metadata, schema, constraints) -- Associated `SQLiteDBInstance` - -It maintains: - -- Column definitions -- Aliases -- Attributes (EVENT_BASED, USER_BASED, etc.) -- Constraint tracking +- Applies constraints to a `QueryContext` +- Validates required columns +- Executes the associated `TablePlugin` +- Returns row sets or generator-backed cursors --- -### BaseCursor +## QueryPlanner and Type Inference -Represents an active scan of a virtual table. +**Namespace:** `osquery.osquery.sql.sqlite_util.QueryPlanner` -Tracks: +Used primarily for column introspection when SQLite cannot determine expression types. -- Row set or generator -- Cursor position -- Current row -- Planner ID +### Strategy -Supports both: +1. Execute `EXPLAIN QUERY PLAN` +2. Execute `EXPLAIN` +3. Parse opcodes +4. Map SQLite opcodes to osquery column types -- Pre-generated row sets -- Streaming generator-based tables - ---- +The `Opcode` structure maps: -### sqlite3_module Implementation +- Register position (P1, P2, P3) +- Resulting ColumnType -Implements the SQLite virtual table callbacks: +This enables inference of types for: -- `xCreate` -- `xBestIndex` -- `xFilter` -- `xNext` -- `xColumn` -- `xRowid` -- `xUpdate` +- Arithmetic operations +- Aggregates +- Comparators +- CAST expressions ```mermaid -sequenceDiagram - participant SQLite - participant Module as "sqlite3_module" - participant Table as "TablePlugin" - - SQLite->>Module: xBestIndex - Module->>SQLite: Constraint plan - - SQLite->>Module: xFilter - Module->>Table: generate(context) - Table-->>Module: Row set - Module-->>SQLite: Rows +flowchart TD + Query["User Query"] --> ExplainPlan["EXPLAIN QUERY PLAN"] + ExplainPlan --> Explain["EXPLAIN"] + Explain --> Parse["Parse Opcode Rows"] + Parse --> ApplyTypes["applyTypes()"] + ApplyTypes --> Columns["Typed Column Metadata"] ``` --- -## Constraint Optimization +## SQLInternal -`xBestIndex` evaluates constraints provided by SQLite and: +`SQLInternal` is a lower-level execution wrapper that: -- Identifies indexed and required columns -- Calculates estimated cost -- Rewrites `IN` constraints for optimized handling -- Tracks used columns for projection pushdown +- Returns `QueryDataTyped` +- Tracks whether results are EVENT_BASED +- Allows result escaping +- Computes result size -If required constraints are missing: +### Additional Features -- Cost is set to a maximum -- Query may fail with `SQLITE_CONSTRAINT` +- `escapeResults()` – Escapes non-printable bytes +- `getSize()` – Computes result memory size +- `eventBased()` – Detect event-based tables -This enables efficient filtering at the table implementation level. +This is used by higher-level components that need typed results. --- -## Query Execution Flow - -```mermaid -flowchart TD - Start["Query Submitted"] --> Prepare["sqlite3_prepare_v2"] - Prepare --> Execute["sqlite3_step Loop"] - Execute -->|Virtual Table Access| VTab - VTab --> xBestIndex - xBestIndex --> xFilter - xFilter --> Generate["TablePlugin.generate()"] - Generate --> Rows["Return Rows"] - Rows --> Finalize["sqlite3_finalize"] - Finalize --> End["Results Returned"] -``` - ---- - -## Extension and Writable Tables - -If a table originates from an extension: - -- `xUpdate` is enabled -- INSERT / UPDATE / DELETE are forwarded -- Parameters are serialized into JSON -- Responses are validated - -This allows controlled writable virtual tables. - ---- +## Attach and Detach Mechanism -## Memory and Performance Optimizations +Virtual tables can be dynamically attached: -When opening the in-memory database: +- `attachTableInternal()` registers a SQLite module +- `detachTableInternal()` drops the temp virtual table -- `journal_mode=OFF` -- `synchronous=OFF` -- `auto_vacuum=FULL` -- Custom function extensions registered -- Authorizer installed +Attachment occurs during: -Additionally: +- Startup +- Extension loading +- Explicit attach requests -- SQLite soft heap limit configured -- Memory released after each query -- Transient DB instances created only under contention +All operations are guarded by `attachLock()`. --- -## Interaction with Other Modules +## Integration with Other Modules The Sql Engine And Virtual Tables module integrates closely with: -- Table plugins (Registry) -- Extensions (for remote virtual tables) -- Logging and query observability (for execution results) -- Eventing framework (for event-based tables) -- Core initialization (for flag configuration) +- Core runtime for shutdown handling and flags +- Logging for error reporting +- Extensions framework for external tables +- Database backend for persistent storage tables +- Events framework for EVENT_BASED table semantics -It does not duplicate table logic; it provides the execution infrastructure. +It forms the execution boundary between SQL logic and system instrumentation. --- -## Key Design Principles +## Design Principles -1. **Security First** – Strict SQLite authorizer and allowlists -2. **In-Memory Execution** – No persistent database files -3. **Plugin-Based Tables** – Data sources abstracted behind registry -4. **Query-Aware Optimization** – Constraint and projection pushdown -5. **Thread-Safe Resource Management** – Managed primary DB with transient fallback -6. **Extension-Friendly** – Writable virtual tables supported +1. Security-first SQL execution +2. Deterministic virtual table behavior +3. Thread-safe database access +4. Efficient constraint pushdown +5. Minimal SQLite surface exposure +6. Strong separation between SQL layer and table logic --- ## Summary -The **Sql Engine And Virtual Tables** module is the execution backbone of osquery. +The **Sql Engine And Virtual Tables** module transforms osquery into a SQL-powered operating system instrumentation engine. -It transforms SQLite into a secure, extensible, in-memory analytics engine that: +It: -- Executes SQL safely -- Dynamically attaches system-backed virtual tables -- Optimizes query plans -- Bridges plugins and extensions with SQLite’s execution engine +- Embeds and secures SQLite +- Converts table plugins into virtual tables +- Plans and executes queries +- Infers types for expressions +- Enforces constraint semantics +- Provides safe attach/detach mechanisms -Without this module, osquery would not be able to expose system data through a unified SQL interface. +Without this module, osquery would be a collection of system probes. With it, those probes become a relational query engine capable of complex joins, filtering, aggregation, and introspection. \ No newline at end of file diff --git a/libraries/cmake/source/augeas/generated/linux/aarch64/code/.parser.md b/libraries/cmake/source/augeas/generated/linux/aarch64/code/.parser.md index a9ebb2bd29b..cc323666012 100644 --- a/libraries/cmake/source/augeas/generated/linux/aarch64/code/.parser.md +++ b/libraries/cmake/source/augeas/generated/linux/aarch64/code/.parser.md @@ -1,46 +1,34 @@ -Auto-generated Bison parser interface header for the Augeas lens language (`augl`) parser, defining token types, semantic value types, and location tracking structures used by the generated LALR(1) parser. +Auto-generated Bison parser interface header for the `augl` language parser, defining token types, semantic value types, and location tracking structures used by the Yacc/Bison-generated parser. ## Key Components -### Token Enum (`yytokentype`) -Defines all terminal tokens recognized by the lexer: - -| Token | Value | Description | -|---|---|---| -| `DQUOTED` | 258 | Double-quoted string literal | -| `REGEXP` | 259 | Regular expression literal | -| `LIDENT` / `UIDENT` / `QIDENT` | 260–262 | Lower/Upper/Qualified identifiers | -| `ARROW` | 263 | `->` arrow operator | -| `KW_MODULE` … `KW_AFTER` | 264–275 | Language keywords | - -### Semantic Value Union (`YYSTYPE`) -Holds the value associated with each parsed token or grammar rule: - -- `struct term *term` β€” AST expression node -- `struct type *type` β€” Type annotation node -- `struct ident *ident` β€” Identifier reference -- `struct tree *tree` β€” Parse tree node -- `char *string` β€” String value -- `regexp` β€” Struct with `pattern` (string) and `nocase` (int) flag -- `int intval` β€” Integer literal -- `enum quant_tag quant` β€” Quantifier (`*`, `+`, `?`) - -### Location Type (`YYLTYPE`) -Tracks source positions with `first_line`, `first_column`, `last_line`, `last_column` for error reporting. - -### Parser Entry Point -```c -int augl_parse(struct term **term, yyscan_t scanner); -``` - -### Scanner State (`struct state`) -```c -struct state { - struct info *info; // Source file info - unsigned int comment_depth; // Nested comment tracking -}; -``` +**Token Enum (`yytokentype`)** +Defines lexical tokens for the `augl` grammar: +- Literals: `DQUOTED` (258), `REGEXP` (259) +- Identifiers: `LIDENT`, `UIDENT`, `QIDENT` +- Keywords: `KW_MODULE`, `KW_AUTOLOAD`, `KW_LET`, `KW_LET_REC`, `KW_IN`, `KW_STRING`, `KW_REGEXP`, `KW_LENS`, `KW_TEST`, `KW_GET`, `KW_PUT`, `KW_AFTER` +- Operators: `ARROW` + +**`YYSTYPE` (Semantic Value Union)** +Holds the value associated with each parsed token: +- `struct term *term` β€” AST term node +- `struct type *type` β€” type annotation +- `struct ident *ident` β€” identifier reference +- `struct tree *tree` β€” parse tree node +- `char *string` β€” string literal value +- `regexp` β€” struct with `nocase` flag and `pattern` string +- `int intval` β€” integer value +- `enum quant_tag quant` β€” quantifier tag + +**`YYLTYPE` (Location Type)** +Tracks source location (line/column range) for error reporting. + +**`struct state`** +Custom scanner state holding a `struct info *` and `comment_depth` counter for nested comment tracking. + +**`augl_parse()`** +Entry point for the parser. ## Usage Example @@ -48,18 +36,19 @@ struct state { #include "parser.h" #include "info.h" -struct term *result = NULL; -yyscan_t scanner; +int parse_augeas_lens(const char *filename) { + struct term *term = NULL; + yyscan_t scanner; -augl_lex_init(&scanner); -// configure scanner input ... + /* Initialize scanner with custom state */ + struct state st = { .info = make_info(filename), .comment_depth = 0 }; -int rc = augl_parse(&result, scanner); -if (rc == 0 && result != NULL) { - // process the parsed AST term -} + augl_lex_init(&scanner); + augl_set_extra(&st, scanner); -augl_lex_destroy(scanner); -``` + int result = augl_parse(&term, scanner); -> **Note:** This file is auto-generated by GNU Bison 3.0.4 from `parser.y`. Do not edit directly β€” modify the grammar source and regenerate. \ No newline at end of file + augl_lex_destroy(scanner); + return result; /* 0 on success */ +} +``` \ No newline at end of file diff --git a/libraries/cmake/source/augeas/generated/linux/x86_64/code/.parser.md b/libraries/cmake/source/augeas/generated/linux/x86_64/code/.parser.md index b54e83a94b8..cc997ce3172 100644 --- a/libraries/cmake/source/augeas/generated/linux/x86_64/code/.parser.md +++ b/libraries/cmake/source/augeas/generated/linux/x86_64/code/.parser.md @@ -1,74 +1,60 @@ -Auto-generated Bison parser interface header defining token types, semantic value union (`YYSTYPE`), location tracking (`YYLTYPE`), and scanner state for parsing Augeas lens definition files. +Auto-generated Bison parser interface header defining token types, semantic value unions, and location tracking structures for the Augeas lens language parser. ## Key Components ### Token Definitions (`yytokentype` enum) +Lexer tokens for the lens grammar: + | Token | Value | Description | -|---|---|---| +|-------|-------|-------------| | `DQUOTED` | 258 | Double-quoted string literal | | `REGEXP` | 259 | Regular expression literal | -| `LIDENT` / `UIDENT` / `QIDENT` | 260–262 | Lower/Upper/Qualified identifiers | +| `LIDENT` | 260 | Lowercase identifier | +| `UIDENT` | 261 | Uppercase identifier | +| `QIDENT` | 262 | Qualified identifier | | `ARROW` | 263 | `->` arrow operator | -| `KW_MODULE` … `KW_AFTER` | 264–275 | Language keywords | - -### `YYSTYPE` β€” Semantic Value Union -Holds the value associated with each parsed token or grammar rule: - -```c -union YYSTYPE { - struct term *term; // Expression/term node - struct type *type; // Type annotation - struct ident *ident; // Identifier reference - struct tree *tree; // Parse tree node - char *string; // String value - struct { - int nocase; // Case-insensitive flag - char *pattern; // Regex pattern string - } regexp; - int intval; // Integer literal - enum quant_tag quant; // Quantifier (*, +, ?) -}; -``` - -### `YYLTYPE` β€” Location Tracking +| `KW_MODULE`–`KW_AFTER` | 264–275 | Language keywords | + +### `YYSTYPE` (Semantic Value Union) +Holds per-token semantic data during parsing: +- `term` β€” parsed expression/term node +- `type` β€” type annotation node +- `ident` β€” identifier node +- `tree` β€” AST tree node +- `string` β€” raw string value +- `regexp` β€” struct with `pattern` (char*) and `nocase` (int) flag +- `intval` β€” integer literal value +- `quant` β€” quantifier tag (`*`, `+`, `?`) + +### `YYLTYPE` (Location Tracking) Tracks source positions for error reporting: +- `first_line` / `first_column` β€” token start +- `last_line` / `last_column` β€” token end -```c -typedef struct YYLTYPE { - int first_line; - int first_column; - int last_line; - int last_column; -} YYLTYPE; -``` - -### `state` β€” Custom Scanner State -Carries context between the lexer and parser: - -```c -struct state { - struct info *info; // Source file metadata - unsigned int comment_depth; // Nested comment tracking -}; -``` +### `struct state` +Custom scanner state (defined via `%code provides`): +- `info` β€” pointer to `struct info` (file/source metadata) +- `comment_depth` β€” tracks nested comment depth for lexer ## Usage Example ```c #include "parser.h" - -// Access token location after parsing -void report_error(YYLTYPE *loc, const char *msg) { - fprintf(stderr, "Error at line %d, col %d: %s\n", - loc->first_line, loc->first_column, msg); +#include "info.h" + +/* Access token location in a grammar action */ +YYLTYPE loc = yylloc; +fprintf(stderr, "Error at line %d, col %d\n", + loc.first_line, loc.first_column); + +/* Inspect semantic value type */ +YYSTYPE val = yylval; +if (current_token == REGEXP) { + printf("Pattern: %s (nocase: %d)\n", + val.regexp.pattern, + val.regexp.nocase); } - -// Initialize scanner state before invoking yyparse() -struct state scanner_state = { - .info = load_info("myfile.aug"), - .comment_depth = 0 -}; ``` -> **Note:** This file is auto-generated by GNU Bison 2.4.1 from `parser.y`. Do not edit directly β€” modify the grammar source instead. \ No newline at end of file +> **Note:** This file is auto-generated by GNU Bison 2.4.1 from `parser.y`. Edit `parser.y` directly rather than modifying this header. \ No newline at end of file diff --git a/libraries/cmake/source/augeas/generated/macos/aarch64/code/.parser.md b/libraries/cmake/source/augeas/generated/macos/aarch64/code/.parser.md index e4d32a0c30c..20a23379fda 100644 --- a/libraries/cmake/source/augeas/generated/macos/aarch64/code/.parser.md +++ b/libraries/cmake/source/augeas/generated/macos/aarch64/code/.parser.md @@ -1,41 +1,33 @@ -Auto-generated Bison parser interface header for the `augl` grammar, defining token types, semantic value unions, and location tracking structures used by the Augeas lens language parser. +Auto-generated Bison parser interface header for the `augl` language parser, defining token types, semantic value types, and location tracking structures used by the Augeas lens grammar. ## Key Components -### Token Enum (`yytokentype`) +**Token Enum (`yytokentype`)** Defines all terminal tokens recognized by the lexer: - -| Token | Value | Description | -|---|---|---| -| `DQUOTED` | 258 | Double-quoted string literal | -| `REGEXP` | 259 | Regular expression literal | -| `LIDENT` / `UIDENT` / `QIDENT` | 260–262 | Lower/upper/qualified identifiers | -| `ARROW` | 263 | `->` arrow operator | -| `KW_MODULE` … `KW_AFTER` | 264–275 | Language keywords | - -### Semantic Value Union (`YYSTYPE`) -Holds the value of each parsed token/rule: - -| Field | Type | Purpose | -|---|---|---| -| `term` | `struct term *` | AST term node | -| `type` | `struct type *` | Type annotation | -| `ident` | `struct ident *` | Identifier reference | -| `tree` | `struct tree *` | Parse tree node | -| `regexp` | `struct { int nocase; char *pattern; }` | Regex with case flag | -| `quant` | `enum quant_tag` | Quantifier (`*`, `+`, `?`) | - -### Location Type (`YYLTYPE`) -Tracks source positions with `first_line`, `first_column`, `last_line`, `last_column` for error reporting. - -### Scanner State (`struct state`) -Custom lexer state carrying an `info` context pointer and `comment_depth` for nested comment handling. - -### Entry Point -```c -int augl_parse(struct term **term, yyscan_t scanner); -``` +- Literals: `DQUOTED` (258), `REGEXP` (259) +- Identifiers: `LIDENT` (260), `UIDENT` (261), `QIDENT` (262) +- Keywords: `KW_MODULE`, `KW_AUTOLOAD`, `KW_LET`, `KW_LET_REC`, `KW_IN`, `KW_STRING`, `KW_REGEXP`, `KW_LENS`, `KW_TEST`, `KW_GET`, `KW_PUT`, `KW_AFTER` +- Operators: `ARROW` (263) + +**`YYSTYPE` Union** +Semantic value type carrying one of: +- `struct term *` β€” AST term node +- `struct type *` β€” type annotation +- `struct ident *` β€” identifier +- `struct tree *` β€” tree node +- `char *string` β€” string literal +- `regexp` struct (pattern + `nocase` flag) +- `int intval`, `enum quant_tag quant` + +**`YYLTYPE` Struct** +Source location tracking with `first_line`, `first_column`, `last_line`, `last_column`. + +**`struct state`** +Custom scanner state holding an `info` pointer and `comment_depth` counter for nested comment handling. + +**`augl_parse()`** +Main parser entry point. ## Usage Example @@ -44,24 +36,17 @@ int augl_parse(struct term **term, yyscan_t scanner); #include "info.h" int parse_lens_file(const char *filename) { - struct term *root = NULL; + struct term *result = NULL; yyscan_t scanner; - /* Initialize scanner state with source info */ - struct state st = { - .info = make_info(filename), - .comment_depth = 0 - }; - - augl_lex_init(&scanner); - augl_set_extra(&st, scanner); - - /* Run the parser; result stored in root */ - int rc = augl_parse(&root, scanner); - - augl_lex_destroy(scanner); - return rc; /* 0 on success */ + /* Initialize scanner, then parse */ + int rc = augl_parse(&result, scanner); + if (rc != 0) { + /* handle parse error */ + } + /* use result AST term */ + return rc; } ``` -> **Note:** Do not use any symbols prefixed with `yy_` or `YY_` β€” these are private Bison internals subject to change without notice. \ No newline at end of file +> **Note:** This file is auto-generated by GNU Bison 3.8.2 from `parser.y`. Edit the grammar source, not this header directly. \ No newline at end of file diff --git a/libraries/cmake/source/augeas/generated/macos/x86_64/code/.parser.md b/libraries/cmake/source/augeas/generated/macos/x86_64/code/.parser.md index aebbb9ace67..ce8566c7439 100644 --- a/libraries/cmake/source/augeas/generated/macos/x86_64/code/.parser.md +++ b/libraries/cmake/source/augeas/generated/macos/x86_64/code/.parser.md @@ -1,38 +1,36 @@ -Auto-generated Bison parser interface header for the Augeas lens language (`augl`) grammar, defining token types, semantic value union, and location tracking structures used by the LALR(1) parser. +Auto-generated Bison parser interface header for the `augl` grammar, defining token types, semantic value types, and location tracking structures used by the Augeas lens language parser. ## Key Components -### Token Enum (`yytokentype`) +### Token Types (`yytokentype` enum) Defines all terminal symbols recognized by the lexer: - -| Token | Value | Description | -|---|---|---| -| `DQUOTED` | 258 | Double-quoted string literal | -| `REGEXP` | 259 | Regular expression literal | -| `LIDENT` / `UIDENT` / `QIDENT` | 260–262 | Lower/upper/qualified identifiers | -| `ARROW` | 263 | `->` operator | -| `KW_MODULE` … `KW_AFTER` | 264–275 | Language keywords | - -### Semantic Value Union (`YYSTYPE`) -Holds the parsed value for each grammar symbol: - -- `struct term *term` β€” AST expression node -- `struct type *type` β€” type annotation node -- `struct ident *ident` β€” identifier node -- `struct tree *tree` β€” parse tree node -- `char *string` β€” raw string value -- `regexp` β€” struct with `nocase` flag and `pattern` string -- `int intval` β€” integer literal -- `enum quant_tag quant` β€” quantifier (`*`, `+`, `?`) - -### Location Type (`YYLTYPE`) -Tracks source positions with `first_line`, `first_column`, `last_line`, `last_column` for error reporting. - -### Scanner State (`struct state`) -Custom lexer state attached to the reentrant scanner, carrying an `info` context and `comment_depth` counter for nested comment handling. - -### Parser Entry Point +- **Literals:** `DQUOTED` (258), `REGEXP` (259) +- **Identifiers:** `LIDENT` (260), `UIDENT` (261), `QIDENT` (262) +- **Operators:** `ARROW` (263) +- **Keywords:** `KW_MODULE`, `KW_AUTOLOAD`, `KW_LET`, `KW_LET_REC`, `KW_IN`, `KW_STRING`, `KW_REGEXP`, `KW_LENS`, `KW_TEST`, `KW_GET`, `KW_PUT`, `KW_AFTER` + +### `YYSTYPE` (Semantic Value Union) +Holds the value associated with each parsed token or grammar rule: + +| Field | Type | Purpose | +|-------|------|---------| +| `term` | `struct term *` | AST term node | +| `type` | `struct type *` | Type annotation | +| `ident` | `struct ident *` | Identifier reference | +| `tree` | `struct tree *` | Parse tree node | +| `string` | `char *` | String literal value | +| `regexp` | struct | Pattern + `nocase` flag | +| `intval` | `int` | Integer literal | +| `quant` | `enum quant_tag` | Quantifier (`*`, `+`, `?`) | + +### `YYLTYPE` (Location Tracking) +Tracks source positions for error reporting via `first_line`, `first_column`, `last_line`, `last_column`. + +### `struct state` +Custom scanner state carrying `struct info *` and a `comment_depth` counter for nested comment handling. + +### Entry Point ```c int augl_parse(struct term **term, yyscan_t scanner); ``` @@ -47,18 +45,16 @@ int parse_lens_file(const char *filename) { struct term *result = NULL; yyscan_t scanner; - /* Initialize reentrant scanner, then invoke parser */ + /* Initialize scanner, then invoke parser */ augl_lex_init(&scanner); - int rc = augl_parse(&result, scanner); + augl_lex_destroy(scanner); + if (rc == 0 && result != NULL) { - /* Walk the resulting AST term */ - process_term(result); + /* result points to the parsed AST term */ } - - augl_lex_destroy(scanner); return rc; } ``` -> **Note:** This file is auto-generated by GNU Bison 3.8.2 from `parser.y`. Do not edit directly β€” modify the grammar source instead and regenerate. \ No newline at end of file +> **Note:** This file is auto-generated by GNU Bison 3.8.2 from `parser.y`. Do not edit directly β€” modify the grammar source instead. \ No newline at end of file diff --git a/openframe/.openframe_authorization_manager.md b/openframe/.openframe_authorization_manager.md index 5203e846d2e..8f4e178397e 100644 --- a/openframe/.openframe_authorization_manager.md +++ b/openframe/.openframe_authorization_manager.md @@ -1,34 +1,35 @@ -Manages the OpenFrame authorization token used for authentication with OpenFrame services. This singleton-style class provides controlled access to storing and retrieving the auth token via a provider-based pattern. +Singleton-style manager class for storing and retrieving the OpenFrame authorization token used to authenticate with OpenFrame services. ## Key Components | Symbol | Type | Description | -|---|---|---| -| `OpenframeAuthorizationManager` | Class | Non-copyable token manager; instantiation restricted to `OpenframeAuthorizationManagerProvider` | -| `updateToken(token)` | Method | Replaces the stored authorization token | -| `getToken()` | Method | Returns the current authorization token | -| `token_` | Private Field | Internal string storage for the token | -| `OpenframeAuthorizationManagerProvider` | Friend Class | Sole class permitted to construct/destroy instances | +|--------|------|-------------| +| `OpenframeAuthorizationManager` | Class | Non-copyable manager for the OpenFrame auth token | +| `updateToken(token)` | Method | Replaces the currently stored authorization token | +| `getToken()` | Method | Returns the current authorization token as a `std::string` | +| `token_` | Private field | Internal storage for the authorization token | +| `OpenframeAuthorizationManagerProvider` | Friend class | Sole class permitted to construct this manager | ## Usage Example ```cpp -// Access is granted only through the provider β€” never constructed directly. -// Typical usage via the provider pattern: +#include "openframe_authorization_manager.h" + +// Access is provided via the friend provider β€” direct construction is private. +// Typical usage through the provider: OpenframeAuthorizationManager& manager = provider.getManager(); // Store a new token received from OpenFrame auth flow manager.updateToken("Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."); -// Retrieve the token for an outbound API request +// Retrieve token for use in outbound requests std::string token = manager.getToken(); -httpClient.setHeader("Authorization", token); ``` -## Design Notes +## Notes -- **Non-copyable** (`boost::noncopyable`): Prevents accidental token duplication across object copies. -- **Private constructor/destructor**: Enforces that only `OpenframeAuthorizationManagerProvider` can create or destroy instances, ensuring centralized lifecycle management. -- **`osquery` namespace**: Integrates with the broader osquery agent infrastructure used by the OpenFrame platform. \ No newline at end of file +- The class inherits from `boost::noncopyable`, preventing accidental copy or assignment β€” enforcing a single shared instance pattern. +- Construction and destruction are `private`, delegating lifecycle management exclusively to `OpenframeAuthorizationManagerProvider`. +- Lives within the `osquery` namespace, integrating with the broader osquery/OpenFrame telemetry and endpoint management stack. \ No newline at end of file diff --git a/openframe/.openframe_encryption_service.md b/openframe/.openframe_encryption_service.md index 7319497e8f5..88abf86af71 100644 --- a/openframe/.openframe_encryption_service.md +++ b/openframe/.openframe_encryption_service.md @@ -1,17 +1,16 @@ -AES-256-GCM decryption service header for the OpenFrame platform, providing symmetric key encryption utilities via OpenSSL. +Declares the `OpenframeEncryptionService` class, providing AES-256-GCM decryption and Base64 decoding utilities for securing data within the OpenFrame platform. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `OpenframeEncryptionService` | Class | Main encryption service wrapping OpenSSL EVP AES-GCM operations | -| `OpenframeEncryptionService(secret)` | Constructor | Initializes the service with a symmetric secret key | -| `decrypt(data)` | Method | Decrypts a Base64-encoded AES-256-GCM ciphertext, returns plaintext string | -| `base64Decode(encoded)` | Method | Decodes a Base64 string into raw bytes | -| `handleOpenSSLError()` | Private Method | Throws `std::runtime_error` with OpenSSL error details on failure | +| Member | Type | Description | +|--------|------|-------------| +| `OpenframeEncryptionService(secret)` | Constructor | Initializes the service with a secret key string | +| `decrypt(data)` | Public method | Decrypts a Base64-encoded AES-GCM payload; throws `std::runtime_error` on failure | +| `base64Decode(encoded)` | Public method | Decodes a Base64 string into raw bytes | +| `handleOpenSSLError()` | Private method | Extracts and throws OpenSSL error details | | `KEY_SIZE` | Constant | 32 bytes (256-bit AES key) | -| `IV_SIZE` | Constant | 12 bytes (96-bit GCM nonce) | +| `IV_SIZE` | Constant | 12 bytes (96-bit GCM IV) | | `TAG_SIZE` | Constant | 16 bytes (128-bit GCM authentication tag) | ## Usage Example @@ -22,22 +21,24 @@ AES-256-GCM decryption service header for the OpenFrame platform, providing symm int main() { // Initialize with a 256-bit secret key - OpenframeEncryptionService enc("my-32-byte-secret-key-goes-here!"); + OpenframeEncryptionService enc_service("your-32-byte-secret-key-here!!"); - // Decrypt Base64-encoded AES-256-GCM payload + // Decrypt a Base64-encoded AES-GCM payload try { - std::string ciphertext = "BASE64_ENCODED_ENCRYPTED_PAYLOAD=="; - std::string plaintext = enc.decrypt(ciphertext); + std::string encrypted_payload = "BASE64_ENCODED_ENCRYPTED_DATA"; + std::string plaintext = enc_service.decrypt(encrypted_payload); std::cout << "Decrypted: " << plaintext << std::endl; } catch (const std::runtime_error& e) { std::cerr << "Decryption failed: " << e.what() << std::endl; } - // Standalone Base64 decode - std::vector raw = enc.base64Decode("SGVsbG8gV29ybGQ="); - - return 0; + // Decode raw Base64 data + std::vector raw = enc_service.base64Decode("SGVsbG8gV29ybGQ="); } ``` -> **Note:** The secret passed to the constructor must produce a 256-bit (32-byte) derived key. Any OpenSSL failure during decryption throws `std::runtime_error` via `handleOpenSSLError()`. \ No newline at end of file +## Notes + +- Requires **OpenSSL** (`libssl-dev`) linked at build time (`-lssl -lcrypto`) +- The secret passed to the constructor must produce a valid 256-bit key after internal derivation +- Authentication tag verification is enforced by AES-GCM, ensuring both confidentiality and integrity \ No newline at end of file diff --git a/openframe/.openframe_token_extractor.md b/openframe/.openframe_token_extractor.md index 2d3e20928e6..8d6291dbf74 100644 --- a/openframe/.openframe_token_extractor.md +++ b/openframe/.openframe_token_extractor.md @@ -1,16 +1,11 @@ -Provides a class interface for reading and decrypting authentication tokens stored in encrypted files, bridging file I/O with the OpenFrame encryption layer. +Declares the `OpenframeTokenExtractor` class, responsible for reading and decrypting authentication tokens from a file using an injected encryption service. ## Key Components -### `OpenframeTokenExtractor` (class) - -| Member | Type | Description | -|--------|------|-------------| -| `OpenframeTokenExtractor(encryption_service, token_file_path)` | Constructor | Initializes the extractor with a shared encryption service and the path to the token file | -| `extractToken()` | `std::string` | Reads the token file, decrypts its contents via the injected `OpenframeEncryptionService`, and returns the plaintext token | -| `token_file_path_` | `std::string` (private) | Stored path to the encrypted token file on disk | -| `encryption_service_` | `std::shared_ptr` (private) | Shared ownership of the encryption service used for decryption | +- **`OpenframeTokenExtractor`** β€” Main class that wraps token file access and decryption logic. +- **Constructor** β€” Accepts a shared `OpenframeEncryptionService` instance and the path to the token file. +- **`extractToken()`** β€” Public method that reads the token file and returns the decrypted token string. ## Usage Example @@ -18,20 +13,15 @@ Provides a class interface for reading and decrypting authentication tokens stor #include "openframe_token_extractor.h" #include "openframe_encryption_service.h" -// Create a shared encryption service -auto enc_service = std::make_shared(/* config */); - -// Instantiate the extractor with the service and token file path -OpenframeTokenExtractor extractor(enc_service, "/etc/openframe/auth.token"); +auto enc_service = std::make_shared(/* ... */); +OpenframeTokenExtractor extractor(enc_service, "/etc/openframe/token.enc"); -// Extract and decrypt the token std::string token = extractor.extractToken(); - -// Use the plaintext token for authenticated API calls +// Use token for authenticated API calls ``` ## Notes -- Uses shared ownership (`std::shared_ptr`) for the encryption service, making it safe to share one service instance across multiple extractors. -- The destructor is defaulted, so no manual resource cleanup is required. -- Depends on `openframe_encryption_service.h` β€” ensure the encryption service is properly initialized before calling `extractToken()`. \ No newline at end of file +- Depends on `OpenframeEncryptionService` via dependency injection, keeping encryption logic decoupled from file I/O. +- The encryption service is held as a `std::shared_ptr`, allowing shared ownership across multiple consumers. +- The token file path is stored at construction time; create a new instance to switch paths. \ No newline at end of file diff --git a/openframe/.openframe_token_refresher.md b/openframe/.openframe_token_refresher.md index 3ec04ff3e39..0c3acadcf64 100644 --- a/openframe/.openframe_token_refresher.md +++ b/openframe/.openframe_token_refresher.md @@ -1,17 +1,14 @@ -Manages background token refresh for OpenFrame authentication by running a periodic refresh loop in a dedicated thread. +Manages background token refresh for the OpenFrame authorization system, periodically re-extracting authentication tokens at a fixed interval using a dedicated thread. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `OpenframeTokenRefresher` | Class | Background service that periodically refreshes authentication tokens | -| `OpenframeTokenRefresher(extractor)` | Constructor | Accepts a shared `OpenframeTokenExtractor` instance | -| `start()` | Method | Spawns the background refresh thread | -| `stop()` | Method | Signals the refresh loop to terminate and joins the thread | -| `process()` | Private Method | Core refresh loop executed on the background thread | -| `running_` | `std::atomic` | Thread-safe flag controlling the refresh loop lifecycle | -| `extractor_` | `shared_ptr` | Token extraction dependency injected at construction | +- **`OpenframeTokenRefresher`** β€” Main class responsible for running a background refresh loop + - `OpenframeTokenRefresher(extractor)` β€” Constructor accepting a shared `OpenframeTokenExtractor` instance + - `start()` β€” Launches the background refresh thread + - `stop()` β€” Signals the thread to terminate and joins it cleanly + - `process()` β€” Private method containing the periodic refresh loop logic +- **Thread-safety primitives** β€” Uses `std::atomic running_` for safe shutdown signaling and `std::mutex mutex_` for state protection ## Usage Example @@ -19,20 +16,21 @@ Manages background token refresh for OpenFrame authentication by running a perio #include "openframe_token_refresher.h" #include "openframe_token_extractor.h" -// Inject extractor and start background refresh +// Create the extractor and pass it to the refresher auto extractor = std::make_shared(); osquery::OpenframeTokenRefresher refresher(extractor); -refresher.start(); // begins periodic token refresh loop +// Start background token refresh +refresher.start(); -// ... application runs ... +// ... application runs, tokens are refreshed automatically ... -refresher.stop(); // gracefully stops the background thread +// Gracefully stop the refresh thread on shutdown +refresher.stop(); ``` ## Notes -- Lives in the `osquery` namespace alongside other OpenFrame authentication components -- Depends on `OpenframeTokenExtractor` for token retrieval and `OpenframeAuthorizationManager` for token storage/validation -- Uses `std::atomic` for lock-free lifecycle control and `std::mutex` for any shared state access within `process()` -- `stop()` should always be called before destruction to avoid detached thread behavior \ No newline at end of file +- The refresher holds a `shared_ptr` to the extractor, ensuring the extractor remains alive for the full lifetime of the refresh thread +- `stop()` should always be called before destruction to avoid dangling thread resources (the destructor calls it implicitly via RAII) +- Operates within the `osquery` namespace, integrating with the broader OpenFrame authorization pipeline via `openframe_authorization_manager.h` \ No newline at end of file diff --git a/osquery/carver/.carver.md b/osquery/carver/.carver.md index 50e1e949180..3f2931b66f1 100644 --- a/osquery/carver/.carver.md +++ b/osquery/carver/.carver.md @@ -3,38 +3,32 @@ Defines the file carving subsystem for osquery, providing classes and utilities ## Key Components -### `CarverRunnable` -Abstract base class extending `InternalRunnable`. Manages the carver dispatch lifecycle via a static `running_` atomic flag. The `start()` method serially processes all pending carve requests; subclasses must implement `doCarve()`. - -### `CarverRunner` -Templated concrete runner that instantiates a carver of type `T` for each request. Designed to allow test-fake injection by swapping the carver type. Tracks the number of carves attempted in the current run cycle via `carves()`. - -### `Carver` -Core carving class. Constructed with a set of file paths, a GUID, and a request ID. Orchestrates the full pipeline: - -| Method | Description | -|---|---| -| `carve()` | End-to-end entry point: carve β†’ compress β†’ POST | -| `createPaths()` | Sets up temp directory, archive, and compressed output paths | -| `carveAll()` | Blockwise-copies source files into a temp directory | -| `blockwiseCopy()` | Low-level copy using `FLAGS_carver_block_size` (default 8 KB) | -| `postCarve()` | POSTs the compressed archive to the configured TLS endpoint | - -### `scheduleCarves()` -Free function called by the scheduler to dispatch a `CarverRunner` if one is not already running. +| Symbol | Type | Description | +|--------|------|-------------| +| `CarverRunnable` | Abstract class | `InternalRunnable` subclass that serially processes pending carve requests; exposes a static `running()` flag | +| `CarverRunner` | Class template | Concrete `CarverRunnable` that instantiates a carver of type `T` per request and tracks carve count | +| `Carver` | Base class | Implements the full carve lifecycle: path setup, blockwise file copy, tar/zstd compression, and HTTP POST | +| `scheduleCarves()` | Free function | Entry point called by the scheduler; dispatches a `CarverRunner` if none is active | ## Usage Example ```cpp -// Dispatch carves from the scheduler +// Dispatch pending carves from the scheduler osquery::scheduleCarves(); -// Direct use of the Carver pipeline (e.g., in tests via CarverRunner) -std::set paths = {"/etc/passwd", "/var/log/syslog"}; -osquery::Carver carver(paths, "guid-1234", "request-5678"); +// --- Or use CarverRunner directly in tests --- +std::set paths = {"/etc/hosts", "/etc/passwd"}; +std::string guid = "abc-123"; +std::string requestId = "req-456"; + +osquery::CarverRunner runner; +osquery::Status status = runner.doCarve(paths, guid, requestId); -Status status = carver.carve(); // carve β†’ archive β†’ compress β†’ POST if (!status.ok()) { LOG(ERROR) << "Carve failed: " << status.getMessage(); } -``` \ No newline at end of file + +size_t attempted = runner.carves(); // 1 +``` + +> **Note:** `Carver::postCarve()` is `virtual`, allowing test subclasses to override the HTTP POST step without hitting a real endpoint. The carve block size is controlled by the `FLAGS_carver_block_size` flag (default 8 KB). \ No newline at end of file diff --git a/osquery/carver/.carver_utils.md b/osquery/carver/.carver_utils.md index da5baa2fa91..840ecb42fa9 100644 --- a/osquery/carver/.carver_utils.md +++ b/osquery/carver/.carver_utils.md @@ -1,40 +1,40 @@ -Utility header providing core primitives for osquery's file carving subsystem β€” constants, database helpers, and scheduling functions used across carver components. +Utility header for osquery's file carving subsystem, providing constants, a template function, and declarations for scheduling and managing file carve operations. ## Key Components ### Constants - -| Constant | Value | Purpose | -|---|---|---| -| `kCarvePathPrefix` | `"osquery_carve_"` | Temp filesystem directory prefix | -| `kCarveNamePrefix` | `"carve_"` | Tar archive filename prefix | -| `kCarverDBPrefix` | `"carves."` | Database key namespace prefix | -| `kCarverStatusSuccess` | `"SUCCESS"` | Terminal success state | -| `kCarverStatusScheduled` | `"SCHEDULED"` | Pending/queued state | - -### Globals - -- **`kCarverPendingCarves`** (`std::atomic`) β€” Optimization flag for `CarverRunner`; avoids spinning up scheduler threads when no carves are queued. +| Name | Description | +|------|-------------| +| `kCarvePathPrefix` | Temp filesystem directory prefix (`osquery_carve_`) | +| `kCarveNamePrefix` | Tar archive name prefix (`carve_`) | +| `kCarverDBPrefix` | Database key prefix for carve entries (`carves.`) | +| `kCarverStatusSuccess` | Status string for completed carves (`SUCCESS`) | +| `kCarverStatusScheduled` | Status string for pending carves (`SCHEDULED`) | +| `kCarverPendingCarves` | Atomic bool flag used by `CarverRunner` to skip idle thread cycles | ### Functions -- **`updateCarveValue(guid, key, value)`** β€” Template function that reads a carve entry from the database by GUID, parses its JSON, updates a single key, and writes it back. -- **`createCarveGuid()`** β€” Generates and returns a new UUID string for identifying a carve request. -- **`carvePaths(paths, request_id, carve_guid)`** β€” Schedules a deferred file carve for the given paths; populates `carve_guid` on success and returns a `Status`. +- **`updateCarveValue`** β€” Template function that updates a single key-value attribute on a carve entry stored in the database, identified by GUID. Handles JSON deserialization, mutation, and re-serialization internally. +- **`createCarveGuid()`** β€” Generates and returns a new UUID string to identify a carve request. +- **`carvePaths()`** β€” Schedules a file carve for a set of paths. Carving is intentionally deferred to avoid blocking query execution or causing parallel carve contention. ## Usage Example -```c -// Schedule a carve for specific paths -std::set targets = {"/etc/passwd", "/var/log/auth.log"}; -std::string guid; +```cpp +#include -Status s = carvePaths(targets, "incident-42", guid); +// Schedule a carve for a set of paths +std::string carve_guid; +std::set paths = {"/etc/passwd", "/var/log/syslog"}; + +Status s = osquery::carvePaths(paths, "incident-42", carve_guid); if (s.ok()) { - // Update a metadata field on the scheduled carve - updateCarveValue(guid, "priority", std::string("high")); + LOG(INFO) << "Carve scheduled with GUID: " << carve_guid; } + +// Later, update an attribute on that carve entry +osquery::updateCarveValue(carve_guid, "status", osquery::kCarverStatusSuccess); ``` -> **Note:** `carvePaths` is non-blocking by design β€” actual carving is deferred to the scheduler to avoid blocking query execution or parallelism issues. \ No newline at end of file +> **Note:** `carvePaths` is non-blocking by design β€” actual carving is dispatched by the `CarverRunner` scheduler, not executed inline. The `kCarverPendingCarves` atomic flag controls whether the runner thread activates on its 60-second cycle. \ No newline at end of file diff --git a/osquery/config/.config.md b/osquery/config/.config.md index c963af8f4d6..168ea85f750 100644 --- a/osquery/config/.config.md +++ b/osquery/config/.config.md @@ -1,52 +1,34 @@ -Programmatic representation of osquery's runtime configuration, providing a singleton interface for loading, updating, validating, and querying configuration state including scheduled queries, file watches, and performance metrics. +Programmatic representation of osquery's runtime configuration, providing a singleton `Config` class that manages query schedules, file watches, config source hashing, performance tracking, and plugin-driven config parsing. ## Key Components -### `Config` (Singleton Class) -The core configuration manager. Access via `Config::get()`. - -**Public Methods:** - -| Method | Description | -|---|---| -| `update(config)` | Apply new configuration data from a source map | -| `load()` | Check for a registered config plugin and trigger initial load | -| `refresh()` | Re-invoke the config plugin's `genConfig` (supports periodic refresh) | -| `recordQueryPerformance(...)` | Store pre/post process metrics for a scheduled query | -| `recordQueryStart(name)` | Mark a query as "dirty" (running) for crash detection | -| `genHash(hash)` | Compute SHA1 over all config sources | -| `hashSource(source, content)` | Hash a single source; returns `true` if content changed | -| `scheduledQueries(predicate)` | Iterate scheduled queries via callback | -| `packs(predicate)` | Iterate all loaded packs via callback | -| `files(predicate)` | Iterate watched file categories via callback | -| `getPerformanceStats(name, predicate)` | Retrieve `QueryPerformance` stats for a named query | -| `getParser(name)` | Fetch a registered `ConfigParserPlugin` by name | -| `validateConfig(document)` | Validate JSON depth and root type | -| `restoreConfigBackup()` | Retrieve config persisted in the backing database | - -### `kExecutingQuery` -Global string constant identifying the currently executing scheduled query. - ---- +- **`Config`** β€” Singleton class (non-copyable) encapsulating all configuration state +- **`Config::get()`** β€” Static singleton accessor +- **`Config::update()`** β€” Applies new config data from a source map +- **`Config::recordQueryPerformance()`** β€” Records pre/post execution metrics for scheduled queries +- **`Config::recordQueryStart()`** β€” Marks a query as "dirty" (in-flight) in the backing store +- **`Config::genHash()` / `hashSource()`** β€” SHA1 hashing of config content per source +- **`Config::scheduledQueries()`** β€” Iterates scheduled queries via a predicate callback +- **`Config::packs()`** β€” Iterates registered query packs via a predicate callback +- **`Config::files()`** β€” Iterates watched file categories via a predicate callback +- **`Config::getParser()`** β€” Retrieves a named `ConfigParserPlugin` from the registry +- **`Config::validateConfig()`** β€” Validates JSON document structure and depth +- **`Config::restoreConfigBackup()`** β€” Retrieves config persisted in the database +- **`kExecutingQuery`** β€” Global string holding the currently executing query name ## Usage Example ```cpp // Access the singleton -auto& cfg = Config::get(); +auto& c = Config::get(); -// Iterate all scheduled queries -cfg.scheduledQueries([](std::string name, const ScheduledQuery& query) { - LOG(INFO) << "Query: " << name << " interval: " << query.interval; -}); +// Load and apply config from a plugin source +c.load(); -// Iterate watched file categories -cfg.files([](const std::string& category, - const std::vector& paths) { - for (const auto& p : paths) { - LOG(INFO) << category << ": " << p; - } +// Iterate all scheduled queries +c.scheduledQueries([](std::string name, const ScheduledQuery& query) { + LOG(INFO) << "Scheduled query: " << name; }); // Retrieve performance stats for a specific query @@ -54,9 +36,9 @@ Config::getPerformanceStats("my_query", [](const QueryPerformance& perf) { LOG(INFO) << "Executions: " << perf.executions; }); -// Restore backed-up config from database -auto result = Config::restoreConfigBackup(); -if (result) { - cfg.update(result.take()); -} +// Iterate watched file categories +c.files([](const std::string& category, + const std::vector& paths) { + LOG(INFO) << "Category: " << category; +}); ``` \ No newline at end of file diff --git a/osquery/config/.packs.md b/osquery/config/.packs.md index 153833cf1fe..d9cc7768789 100644 --- a/osquery/config/.packs.md +++ b/osquery/config/.packs.md @@ -1,51 +1,50 @@ -Defines the `Pack` class and related utilities for managing osquery query packs β€” collections of scheduled queries with platform/version constraints and discovery logic. +Defines the `Pack` class and related utilities for managing osquery query packs β€” groupings of scheduled queries with discovery conditions, platform requirements, and execution scheduling logic. ## Key Components -### `PackStats` struct -Tracks discovery query execution statistics: `total`, `hits`, and `misses` counts. +### `PackStats` +Simple struct tracking discovery query execution metrics: `total`, `hits`, and `misses` counts. -### `Pack` class +### `Pack` (class) Non-copyable class representing a query pack loaded from configuration. | Method | Description | |---|---| | `initialize()` | Parses and loads pack content from a `rapidjson::Value` | -| `shouldPackExecute()` | Determines if the pack should run on this host | -| `checkPlatform()` | Validates platform compatibility | +| `shouldPackExecute()` | Determines if the pack is eligible to run on this host | +| `checkPlatform()` | Validates OS compatibility | | `checkVersion()` | Validates minimum osquery version requirement | -| `checkDiscovery()` | Runs discovery queries to determine pack applicability | -| `isActive()` | Returns active state without triggering discovery queries | -| `getSchedule()` | Returns the vector of `ScheduledQuery` entries | -| `getStats()` | Returns accumulated `PackStats` | +| `checkDiscovery()` | Runs discovery queries to confirm pack applicability | +| `isActive()` | Returns cached active state without re-running discovery | +| `getSchedule()` | Returns the list of `ScheduledQuery` entries in the pack | +| `getStats()` | Returns discovery execution statistics | ### Free Functions | Function | Description | |---|---| -| `splayValue(original, splay_percent)` | Generates a splayed interval from a base interval and percent | -| `restoreSplayedValue(name, interval)` | Retrieves or generates a cached splay for a name/interval pair | +| `splayValue(original, splay_percent)` | Generates a randomized interval offset within a given splay percentage | +| `restoreSplayedValue(name, interval)` | Retrieves or generates a deterministic cached splay for a query name/interval pair | ## Usage Example ```cpp #include -// Construct a pack from parsed JSON -rapidjson::Value packObj; // populated from config -osquery::Pack pack("my_pack", "config_source", packObj); +// Construct a pack from a JSON object +rapidjson::Document doc; +doc.Parse(packJsonString.c_str()); -// Check if the pack should run on this host +osquery::Pack pack("my_pack", "config_source", doc); + +// Check if this pack should run on the current host if (pack.shouldPackExecute()) { - for (const auto& query : pack.getSchedule()) { - // queue query for execution - } + for (const auto& query : pack.getSchedule()) { + // Process each scheduled query + } } -// Generate a splayed interval (e.g., 10% splay on 60s interval) -uint64_t splayed = osquery::splayValue(60, 10); - -// Retrieve or create a stable cached splay -uint64_t stable = osquery::restoreSplayedValue("my_pack_query", 60); +// Get a splayed interval for a 60s query with 10% splay +uint64_t interval = osquery::restoreSplayedValue("my_query", 60); ``` \ No newline at end of file diff --git a/osquery/config/tests/.test_utils.md b/osquery/config/tests/.test_utils.md index 0771b51ebad..f5c3ebfb923 100644 --- a/osquery/config/tests/.test_utils.md +++ b/osquery/config/tests/.test_utils.md @@ -1,37 +1,37 @@ -Utility header providing test infrastructure helpers for osquery configuration and pack loading in unit tests. +Utility header providing test infrastructure helpers for osquery's configuration and pack-related unit tests. ## Key Components | Symbol | Type | Description | -|---|---|---| -| `getTestConfigDirectory()` | Function | Returns path to the test configuration directory | -| `getTestHelperScriptsDirectory()` | Function | Returns path to the test helper scripts directory | -| `getTestConfigMap()` | Function | Builds a config map from a file, mapping source name to JSON content | -| `getExamplePacksConfig()` | Function | Returns a sample packs configuration as a `JSON` object | -| `getUnrestrictedPack()` | Function | Returns a pack with no platform/version restrictions | -| `getRestrictedPack()` | Function | Returns a pack with platform/version restrictions applied | -| `getPackWithDiscovery()` | Function | Returns a pack containing discovery queries | -| `getPackWithValidDiscovery()` | Function | Returns a pack with a discovery query expected to pass | -| `getPackWithFakeVersion()` | Function | Returns a pack targeting a non-existent/fake osquery version | +|--------|------|-------------| +| `getTestConfigDirectory()` | Function | Returns the filesystem path to the test configuration directory | +| `getTestHelperScriptsDirectory()` | Function | Returns the filesystem path to test helper scripts | +| `getTestConfigMap()` | Function | Builds a config map from a file, mapping a static source name to its JSON content | +| `getExamplePacksConfig()` | Function | Returns a `JSON` object representing an example packs configuration | +| `getUnrestrictedPack()` | Function | Returns a `JSON` object for a pack with no restrictions | +| `getRestrictedPack()` | Function | Returns a `JSON` object for a pack with restrictions applied | +| `getPackWithDiscovery()` | Function | Returns a `JSON` object for a pack containing discovery queries | +| `getPackWithValidDiscovery()` | Function | Returns a `JSON` object for a pack with valid discovery queries | +| `getPackWithFakeVersion()` | Function | Returns a `JSON` object for a pack targeting a non-existent/fake version | ## Usage Example ```c #include "test_utils.h" -// Resolve test config files at runtime -auto config_dir = osquery::getTestConfigDirectory(); -auto config_map = osquery::getTestConfigMap("example_config.conf"); +// Resolve test asset paths +boost::filesystem::path configDir = osquery::getTestConfigDirectory(); +boost::filesystem::path scriptsDir = osquery::getTestHelperScriptsDirectory(); -// Load fixture packs for scheduler/config unit tests -osquery::JSON unrestricted = osquery::getUnrestrictedPack(); -osquery::JSON restricted = osquery::getRestrictedPack(); -osquery::JSON discovered = osquery::getPackWithValidDiscovery(); -``` +// Load a config map from a test file +auto configMap = osquery::getTestConfigMap("example.conf"); -## Notes +// Access fixture pack configurations +osquery::JSON unrestrictedPack = osquery::getUnrestrictedPack(); +osquery::JSON restrictedPack = osquery::getRestrictedPack(); +osquery::JSON discoveryPack = osquery::getPackWithValidDiscovery(); +osquery::JSON fakePack = osquery::getPackWithFakeVersion(); +``` -- All functions are declared inside the `osquery` namespace. -- Path helpers return `const` references β€” do not store the result beyond test scope. -- Pack fixture functions return `JSON` objects (from `osquery/utils/json/json.h`) and are intended exclusively for use in unit/integration tests. \ No newline at end of file +> All symbols are declared within the `osquery` namespace. This header is intended exclusively for use in test targets and should not be included in production code. \ No newline at end of file diff --git a/osquery/core/.flags.md b/osquery/core/.flags.md index 3168ef7b20f..9192a889854 100644 --- a/osquery/core/.flags.md +++ b/osquery/core/.flags.md @@ -1,55 +1,54 @@ -Declares osquery's flag management system, wrapping gflags to support categorized, queryable, and runtime-updatable configuration options across shell, CLI, extension, and hidden flag contexts. +Defines osquery's flag management system, wrapping Google's gflags library with additional metadata tracking for shell, extension, CLI, and hidden flag categories. ## Key Components ### Structs -- **`FlagDetail`** β€” Metadata for a flag: description, and boolean markers (`shell`, `external`, `cli`, `hidden`) -- **`FlagInfo`** β€” Full flag snapshot including type, description, default value, current value, and its `FlagDetail` +- **`FlagDetail`** β€” Metadata for a flag: description, and boolean attributes (`shell`, `external`, `cli`, `hidden`) +- **`FlagInfo`** β€” Full flag info including type, description, default value, current value, and a `FlagDetail` -### Class: `Flag` (singleton, non-copyable) +### Class: `Flag` (Singleton, noncopyable) | Method | Description | -|---|---| -| `create()` | Registers a new named flag with its detail metadata | -| `createAlias()` | Registers a gflags alias pointing to an existing flag | -| `flags()` | Returns a map of all registered flags and their info | -| `getValue()` / `getInt32Value()` | Retrieves flag value by name | -| `getDefaultValue()` | Retrieves the default value for a flag | -| `updateValue()` | Updates a flag's value at runtime | -| `isDefault()` | Checks if a flag still holds its default value | -| `getType()` / `getDescription()` | Introspection helpers | -| `isCLIOnlyFlag()` | Returns true if the flag is CLI/flagfile-only | +|--------|-------------| +| `create()` | Registers a new flag with its metadata | +| `createAlias()` | Creates a gflags alias pointing to an existing flag | +| `flags()` | Returns all registered flags as a `map` | +| `getValue()` / `getInt32Value()` | Retrieves current flag value by name | +| `getDefaultValue()` | Retrieves the default value, returns `Status` | +| `updateValue()` | Updates a flag's runtime value | +| `isDefault()` | Checks if a flag is still at its default | +| `isCLIOnlyFlag()` | Returns true if flag cannot be set via config options | | `printFlags()` | Prints help output, filterable by shell/external/cli | -| `resetCustomFlags()` | Clears custom flags (used by fuzzers) | +| `resetCustomFlags()` | Clears custom flags map (used by fuzzers) | ### Macros -| Macro | Behavior | -|---|---| -| `FLAG(t,n,v,d)` | Standard flag β€” settable via flagfile, CLI, or config options | -| `SHELL_FLAG` | Only available in `osqueryi` | +| Macro | Description | +|-------|-------------| +| `FLAG(t,n,v,d)` | Standard flag usable in flagfile, CLI, or config options | +| `SHELL_FLAG` | Only available in `osqueryi` shell | | `EXTENSION_FLAG` | Only available to extensions | -| `CLI_FLAG` | Cannot be set via config `options` | +| `CLI_FLAG` | CLI/flagfile only β€” not settable via config `"options"` | | `HIDDEN_FLAG` | Excluded from `--help` output | ## Usage Example -```cpp -// Declare flags in a .cpp file -FLAG(bool, enable_feature, false, "Enable the experimental feature"); -CLI_FLAG(string, config_path, "/etc/osquery/osquery.conf", "Path to config file"); -SHELL_FLAG(int32, query_timeout, 300, "Query timeout in seconds (osqueryi only)"); - -// Read a flag value at runtime -std::string path = Flag::getValue("config_path"); - -// Update a flag value programmatically -Status s = Flag::updateValue("enable_feature", "true"); -if (!s.ok()) { - LOG(WARNING) << "Failed to update flag: " << s.getMessage(); -} - -// Check if a flag was explicitly set or still at default -if (Flag::isDefault("query_timeout")) { - LOG(INFO) << "Using default query timeout"; -} +```c +// Declare flags in any translation unit +FLAG(bool, enable_monitor, false, "Enable the system monitor"); +SHELL_FLAG(string, output_format, "table", "Output format for osqueryi"); +CLI_FLAG(uint64, config_refresh, 60, "Seconds between config refreshes"); +HIDDEN_FLAG(bool, debug_internal, false, "Internal debug toggle"); + +// Read and update flag values at runtime +std::string val = Flag::getValue("enable_monitor"); // "false" +Flag::updateValue("enable_monitor", "true"); + +std::string def; +Flag::getDefaultValue("config_refresh", def); // def = "60" + +bool unchanged = Flag::isDefault("config_refresh"); // true +bool cliOnly = Flag::isCLIOnlyFlag("config_refresh"); // true + +// Print only shell flags (osqueryi --help) +Flag::printFlags(/*shell=*/true); ``` \ No newline at end of file diff --git a/osquery/core/.query.md b/osquery/core/.query.md index 5cfec8773ce..fad25396e46 100644 --- a/osquery/core/.query.md +++ b/osquery/core/.query.md @@ -1,60 +1,59 @@ -Defines the `Query` class and `QueryLogItem` struct for managing scheduled query execution history, differential results, and JSON serialization within the osquery framework. +Defines the `Query` class and `QueryLogItem` struct for managing scheduled query execution history, result diffing, and JSON serialization within the osquery framework. ## Key Components -### `QueryLogItem` struct +### `QueryLogItem` (struct) +Holds metadata and results for a single query execution log entry, supporting both differential and snapshot result modes. + | Field | Type | Description | -|---|---|---| +|-------|------|-------------| | `isSnapshot` | `bool` | Flags snapshot vs. differential results | -| `results` | `DiffResults` | Differential query results | -| `snapshot_results` | `QueryDataTyped` | Full snapshot results | -| `name` / `identifier` | `std::string` | Query name and host identifier | +| `results` | `DiffResults` | Differential result set | +| `snapshot_results` | `QueryDataTyped` | Full snapshot result set | | `epoch` / `counter` | `uint64_t` | Execution epoch and invocation counter | -| `decorations` | `map` | Additional log line metadata | +| `decorations` | `map` | Extra fields appended to log output | -### Serialization Functions -- `serializeQueryLogItem()` β€” Serializes a `QueryLogItem` into a JSON document -- `serializeQueryLogItemJSON()` β€” Serializes a `QueryLogItem` into a JSON string -- `serializeQueryLogItemAsEvents()` β€” Serializes as an event-list JSON document -- `serializeQueryLogItemAsEventsJSON()` β€” Serializes as a vector of event JSON strings +### `Query` (class) +Manages persistent storage and retrieval of scheduled query results via RocksDB. -### `Query` class | Method | Description | -|---|---| +|--------|-------------| | `getPreviousQueryResults()` | Loads prior results from RocksDB into a `QueryDataSet` | -| `saveQueryResults()` | Persists updated results JSON and epoch to the database | +| `saveQueryResults()` | Persists serialized JSON results and epoch to RocksDB | | `addNewResults()` | Stores new results and computes differential vs. last run | | `addNewEvents()` | Variant of `addNewResults` for event-based queries | | `getQueryCounter()` | Returns/increments the invocation counter, handling resets | -| `isNewQuerySql()` | Detects if the query SQL has changed since last run | -| `getStoredQueryNames()` | Static; returns all query names persisted in RocksDB | +| `isNewQuerySql()` | Detects if the query SQL has been altered since last run | +| `getStoredQueryNames()` | Static β€” returns all query names persisted in RocksDB | + +### Serialization Functions +Free functions for converting `QueryLogItem` to JSON: +- `serializeQueryLogItem()` / `serializeQueryLogItemJSON()` β€” standard serialization +- `serializeQueryLogItemAsEvents()` / `serializeQueryLogItemAsEventsJSON()` β€” event-list form ## Usage Example ```c -// Initialize and run a scheduled query, capturing differential results -ScheduledQuery sq; -sq.query = "SELECT * FROM processes;"; - -Query q("process_monitor", sq); +// Construct and execute a query diff cycle +ScheduledQuery sq{"SELECT * FROM processes", 60, {}}; +Query q("proc_monitor", sq); +QueryDataTyped new_results = /* run query */; uint64_t counter = 0; DiffResults diff; -QueryDataTyped results = executeQuery(sq.query); -Status status = q.addNewResults(std::move(results), current_epoch, counter, diff); +Status s = q.addNewResults(std::move(new_results), current_epoch, counter, diff); -if (status.ok()) { +if (s.ok()) { QueryLogItem item; - item.name = "process_monitor"; + item.name = "proc_monitor"; item.results = diff; item.epoch = current_epoch; item.counter = counter; - item.isSnapshot = false; - std::string json; - serializeQueryLogItemJSON(item, json); - logger.log(json); + std::string json_out; + serializeQueryLogItemJSON(item, json_out); + // send json_out to logger } ``` \ No newline at end of file diff --git a/osquery/core/.shutdown.md b/osquery/core/.shutdown.md index 6480b232439..d9c48360938 100644 --- a/osquery/core/.shutdown.md +++ b/osquery/core/.shutdown.md @@ -1,47 +1,49 @@ -Declares the osquery shutdown coordination API, providing functions to request, wait for, and inspect application shutdown state across daemon services and threads. +Declares the osquery shutdown coordination API, providing functions to request, monitor, and wait for graceful application termination across daemon threads and services. ## Key Components | Function | Description | -|---|---| -| `requestShutdown(int, string)` | Primary shutdown trigger; signals all dispatched services to stop gracefully | -| `waitForShutdown()` | Blocks the calling thread until shutdown is requested | -| `waitTimeoutOrShutdown(milliseconds)` | Interruptible sleep β€” returns early if shutdown is requested | -| `shutdownRequested()` | Non-blocking check for shutdown state (shell/tool use only) | -| `getShutdownExitCode()` | Retrieves the stored exit code set during shutdown request | -| `setShutdownExitCode(int)` | Directly sets the exit code without triggering shutdown | -| `resetShutdown()` | Resets shutdown state to `false`; intended for internal/testing use only | +|----------|-------------| +| `requestShutdown(retcode)` | Primary shutdown trigger; signals all dispatched services to stop | +| `requestShutdown(retcode, system_log)` | Overload that also writes a message to the system log before shutting down | +| `waitForShutdown()` | Blocks the calling thread until `requestShutdown` is invoked | +| `waitTimeoutOrShutdown(timeout)` | Blocks until shutdown is requested **or** the given timeout elapses; returns `true` if shutdown was requested | +| `shutdownRequested()` | Non-blocking check of shutdown state; primarily for shell tools | +| `getShutdownExitCode()` | Retrieves the exit code set by the last `requestShutdown` call | +| `setShutdownExitCode(retcode)` | Directly sets the exit code without triggering shutdown | +| `resetShutdown()` | Resets shutdown state; intended for internal use and testing only | ## Usage Example ```cpp #include -// In main thread β€” block until shutdown is signaled +// Signal handler or service stop callback +void onSignal(int sig) { + osquery::requestShutdown(EXIT_SUCCESS); +} + +// Main thread β€” blocks until shutdown is requested void runDaemon() { - waitForShutdown(); - // Proceed with cleanup after shutdown is requested - Initializer::shutdown(getShutdownExitCode()); + osquery::waitForShutdown(); + // Perform cleanup after shutdown is signalled } -// In a retry loop β€” sleep interruptibly +// Retry loop interruptible by shutdown void enrollWithRetry() { - while (!shutdownRequested()) { - bool done = attemptEnrollment(); - if (done) break; - - // Wait 30s or abort early if shutdown is requested - if (waitTimeoutOrShutdown(std::chrono::milliseconds(30000))) { - break; // Shutdown was requested - } + using namespace std::chrono_literals; + bool should_stop = osquery::waitTimeoutOrShutdown(30000ms); + if (should_stop) { + return; // Shutdown was requested during wait } + // Continue retry logic } -// Request shutdown from an error handler -void onFatalError(const std::string& reason) { - requestShutdown(EXIT_FAILURE, reason); +// Unrecoverable error with system log entry +void onFatalError() { + osquery::requestShutdown(EXIT_FAILURE, "Unrecoverable config error"); } ``` -> **Note:** Prefer `requestShutdown()` over direct `exit()` calls to ensure all event threads and Thrift service pools are properly notified before process termination. \ No newline at end of file +> **Note:** Prefer `waitForShutdown()` over polling `shutdownRequested()`. Call `waitForShutdown()` before `Initializer::shutdown` in the main thread. \ No newline at end of file diff --git a/osquery/core/.system.md b/osquery/core/.system.md index 50183a002ee..60ef4425bcc 100644 --- a/osquery/core/.system.md +++ b/osquery/core/.system.md @@ -1,52 +1,43 @@ -Defines the `Initializer` class and supporting utilities for bootstrapping osquery process lifecycle, identity, and platform setup within the `osquery` namespace. +Provides the core initialization, lifecycle management, and system identity utilities for osquery processes, defining the `Initializer` class and related helper functions used across daemon, shell, worker, and watcher process types. ## Key Components -### `Initializer` Class -The primary entry point for osquery process initialization. Non-copyable, intended for use in `main()`. - -| Method | Description | -|---|---| -| `Initializer(argc, argv, tool, init_glog)` | Constructor; configures flags, logging, and tool type | -| `initDaemon()` | Sets up process as an osquery daemon | -| `initShell()` | Sets up process as an interactive shell | -| `initWorkerWatcher(name)` | Spawns and monitors child worker processes | -| `shutdown(retcode)` | Cleanly stops all services and returns exit code | -| `requestShutdown(retcode)` | Static; requests graceful shutdown | -| `shutdownNow(retcode)` | Static; immediate exit without dispatcher teardown | -| `isWorker()` / `isWatcher()` | Static; detects process role via environment | - -### UUID & Identity Utilities -| Function | Description | -|---|---| -| `generateNewUUID()` | Generates a random UUID string | -| `getInstanceUUID(ident)` | Retrieves persistent instance UUID | -| `getEphemeralUUID(ident)` | Retrieves session-scoped UUID | -| `getHostUUID(ident)` | Retrieves hardware-based host UUID | -| `getHostIdentifier()` | Returns configured unique machine identifier | -| `isPlaceholderHardwareUUID(uuid)` | Detects invalid/placeholder UUIDs | - -### Platform & Process Utilities -- `platformSetup()` / `platformTeardown()` β€” COM/platform init on Windows -- `isUserAdmin()` β€” Checks admin/root privileges -- `setThreadName(name)` β€” Names the current thread -- `getStartTime()` / `setStartTime(st)` β€” Tracks tool start time +**`Initializer` class** β€” Primary entry point for osquery process setup (non-copyable): +- `Initializer(argc, argv, tool, init_glog)` β€” Bootstraps process state; call from `main()` +- `initDaemon()` β€” Configures process as an osquery daemon with mutex and config validation +- `initShell()` β€” Lightweight shell mode initialization +- `initWorkerWatcher(name)` β€” Spawns and monitors worker child processes with CPU/memory bounds +- `start()` β€” Begins active work after initialization +- `shutdown(retcode)` β€” Stops all services/threads cleanly; returns most appropriate exit code +- `requestShutdown(retcode)` / `shutdownNow(retcode)` β€” Static shutdown triggers +- `isWorker()` / `isWatcher()` β€” Detect current process role via environment variable + +**UUID & Identity helpers:** +- `generateNewUUID()` β€” Creates a random UUID string +- `getInstanceUUID()` / `getEphemeralUUID()` / `getHostUUID()` β€” Scoped UUID getters returning `Status` +- `getHardwareUUID()` / `getHostIdentifier()` β€” Machine-level identification +- `isPlaceholderHardwareUUID(uuid)` β€” Filters known invalid/placeholder UUIDs + +**System utilities:** +- `isUserAdmin()` β€” Checks if the process runs with admin privileges +- `setThreadName(name)` β€” Platform-aware thread naming +- `getStartTime()` / `setStartTime(st)` β€” Tool start timestamp accessors +- `platformSetup()` / `platformTeardown()` β€” Platform-specific init/teardown (e.g., COM on Windows) ## Usage Example -```cpp -#include - +```c int main(int argc, char* argv[]) { - osquery::Initializer runner(argc, argv, osquery::ToolType::DAEMON); + osquery::Initializer runner(argc, argv, osquery::ToolType::DAEMON); + + runner.initDaemon(); - runner.initDaemon(); - runner.initWorkerWatcher("osqueryd"); + // Watcher forks workers; only workers return here + runner.initWorkerWatcher("osqueryd"); - // Only worker processes return here - runner.start(); + runner.start(); - return runner.shutdown(EXIT_SUCCESS); + return runner.shutdown(EXIT_SUCCESS); } ``` \ No newline at end of file diff --git a/osquery/core/.tables.md b/osquery/core/.tables.md index 7dabd713bd7..cf1dd80096a 100644 --- a/osquery/core/.tables.md +++ b/osquery/core/.tables.md @@ -1,43 +1,34 @@ -Defines the core data structures and type system for osquery's virtual table infrastructure, including SQLite type affinity macros, constraint handling, and table metadata used when implementing and querying osquery tables. +Defines core data structures and types for osquery's virtual table system, providing the foundational abstractions for SQL query constraint handling, column type affinities, and table attribute management used by all osquery table implementations. ## Key Components -### Type Affinity Macros -- `SQL_TEXT(x)`, `INTEGER(x)`, `BIGINT(x)`, `UNSIGNED_BIGINT(x)`, `DOUBLE(x)` β€” convert native C++ values to SQLite string representations via `__sqliteField()` -- `TEXT_LITERAL`, `INTEGER_LITERAL`, `BIGINT_LITERAL`, `UNSIGNED_BIGINT_LITERAL`, `DOUBLE_LITERAL` β€” C++ type aliases for table column literals - -### Enumerations -- `ConstraintOperator` β€” SQL predicate operators (`EQUALS`, `LESS_THAN`, `GLOB`, `REGEXP`, `IN_OP`, etc.) -- `TableAttributes` β€” bitflag enum describing table behavior (`CACHEABLE`, `EVENT_BASED`, `USER_BASED`, `PENDING`, etc.) - -### Structs & Classes -- `Constraint` β€” pairs a `ConstraintOperator` with a string expression -- `ConstraintList` β€” ordered set of `Constraint` objects for a single column; supports `matches()`, `exists()`, `existsAndMatches()`, `notExistsOrMatches()`, and `getAll()` - -### Type Aliases -- `TableColumns` β€” ordered vector of `(name, ColumnType, ColumnOptions)` tuples -- `ConstraintMap` β€” maps column names to their `ConstraintList` -- `ConstraintSet` β€” parsed predicate pairs used to populate a `ConstraintMap` -- `UsedColumns` / `UsedColumnsBitset` β€” tracks which columns are referenced in a query +- **`__sqliteField()`** β€” Overloaded inline helpers that convert various C/C++ types (primitives, char arrays, strings) into `std::string` for SQLite storage +- **Type Affinity Macros** β€” `SQL_TEXT`, `INTEGER`, `BIGINT`, `UNSIGNED_BIGINT`, `DOUBLE` for column value conversion; `TEXT_LITERAL`, `INTEGER_LITERAL`, etc. for C++ type aliases +- **`ConstraintOperator`** β€” Enum of SQL predicate operators (`EQUALS`, `GREATER_THAN`, `LIKE`, `GLOB`, `REGEXP`, etc.) +- **`Constraint`** β€” POD struct pairing an operator with its expression string +- **`ConstraintList`** β€” Non-copyable collection of `Constraint` objects for a single column; provides `matches()`, `exists()`, `existsAndMatches()`, `notExistsOrMatches()`, and `getAll()` query helpers +- **`TableAttributes`** β€” Bitmask enum flagging table behaviors (`CACHEABLE`, `EVENT_BASED`, `USER_BASED`, `PENDING`, etc.) +- **Type Aliases** β€” `TableColumns`, `ConstraintMap`, `ConstraintSet`, `UsedColumns`, `UsedColumnsBitset`, `ColumnAliasSet`, `AliasColumnMap` ## Usage Example ```cpp -// Check if a constraint exists and matches a value -ConstraintList cl; -cl.add(Constraint(EQUALS, "root")); - -if (cl.existsAndMatches(std::string("root"))) { - // Filter results to user "root" +// Checking constraints in a table generator +void MyTablePlugin::generate(QueryContext& context) { + auto& uid_constraints = context.constraints["uid"]; + + // Only filter if a constraint was provided + if (uid_constraints.exists(EQUALS)) { + for (const auto& uid : uid_constraints.getAll(EQUALS)) { + // fetch rows matching uid + } + } + + // Build a row using type affinity macros + Row r; + r["uid"] = INTEGER(1000); + r["name"] = SQL_TEXT("alice"); + r["size"] = BIGINT(file_size); } - -// Get all EQUALS expressions for iteration -for (const auto& expr : cl.getAll(EQUALS)) { - generateRowsForUser(expr); -} - -// Convert a native value to SQLite text representation -std::string val = INTEGER(42); // "42" -std::string txt = SQL_TEXT("hi"); // "hi" ``` \ No newline at end of file diff --git a/osquery/core/.watcher.md b/osquery/core/.watcher.md index 93cd774f62e..ed2feeba5cf 100644 --- a/osquery/core/.watcher.md +++ b/osquery/core/.watcher.md @@ -1,41 +1,39 @@ -Defines the watchdog infrastructure for osquery, managing the lifecycle and performance monitoring of worker processes and autoloaded extension processes. +Thread-safe watchdog system for managing and monitoring osquery worker and extension child processes, enforcing CPU, memory, and respawn performance limits. ## Key Components ### Enums & Structs -- **`WatchdogLimitType`** β€” Categorizes performance limits: `MEMORY_LIMIT`, `UTILIZATION_LIMIT`, `RESPAWN_LIMIT`, `RESPAWN_DELAY`, `LATENCY_LIMIT`, `INTERVAL` -- **`PerformanceState`** β€” Snapshot struct tracking CPU time, memory footprint, respawn timestamps, and sustained latency counters for a monitored process +- **`WatchdogLimitType`** β€” Categorizes performance limit types: `MEMORY_LIMIT`, `UTILIZATION_LIMIT`, `RESPAWN_LIMIT`, `RESPAWN_DELAY`, `LATENCY_LIMIT`, `INTERVAL` +- **`PerformanceState`** β€” Snapshot struct tracking CPU times, respawn timestamp, sustained latency, and initial memory footprint for a monitored process +- **`ExtensionMap`** β€” Alias for `std::map>` ### Classes -- **`Watcher`** β€” Thread-safe, noncopyable singleton managing the worker and extension process map. Tracks performance states, restart counts, and process handles. Only accessible by `WatcherRunner` -- **`WatcherRunner`** β€” `InternalRunnable` thread that spawns, monitors, and respawns worker/extension processes. Enforces resource limits and handles clean/forced shutdowns -- **`WatcherWatcherRunner`** β€” A reverse-watchdog spawned inside the worker process; monitors the parent watcher process and exits if it disappears +- **`Watcher`** β€” Non-copyable, thread-safe state manager for the worker process and autoloaded extensions. Tracks performance states, process handles, restart counts, and fate-binding between watcher and worker. Exposes `hasManagedExtensions()`, `loadExtensions()`, and `getWorkerStatus()` +- **`WatcherRunner`** β€” `InternalRunnable` dispatcher thread that spawns/monitors worker and extension processes. Polls health, enforces limits, and respawns children that violate thresholds. Key methods: `watch()`, `watchExtensions()`, `isChildSane()`, `isWatcherHealthy()`, `createWorker()`, `createExtension()`, `stopChild()` +- **`WatcherWatcherRunner`** β€” `InternalRunnable` spawned inside the worker process to watch the watcher itself, ensuring the worker exits if its parent watchdog dies ### Free Functions -- **`getWorkerLimit(WatchdogLimitType)`** β€” Returns a numeric performance threshold for a given limit category and configured watchdog level - -### Type Aliases - -- **`ExtensionMap`** β€” `std::map>` mapping extension paths to their process handles +- **`getWorkerLimit(WatchdogLimitType)`** β€” Returns the numeric performance limit for a given limit category and configured watchdog level ## Usage Example ```cpp -// Construct and bind a Watcher, then launch the WatcherRunner thread +// Instantiate and bind a Watcher auto watcher = std::make_shared(); watcher->loadExtensions(); +// Spawn the WatcherRunner dispatcher thread auto runner = std::make_shared( - argc, argv, /*use_worker=*/true, watcher); - -// Query a specific performance limit -uint64_t memLimit = osquery::getWorkerLimit( - osquery::WatchdogLimitType::MEMORY_LIMIT); - -// Bind watcher/worker fates (watcher exit kills worker) -watcher->bindFates(); + argc, argv, /*use_worker=*/true, watcher +); +Dispatcher::addService(runner); + +// Query a performance limit threshold +uint64_t mem_limit = osquery::getWorkerLimit( + osquery::WatchdogLimitType::MEMORY_LIMIT +); ``` \ No newline at end of file diff --git a/osquery/core/plugins/.logger.md b/osquery/core/plugins/.logger.md index 201e0d11239..951ab08c16f 100644 --- a/osquery/core/plugins/.logger.md +++ b/osquery/core/plugins/.logger.md @@ -1,28 +1,26 @@ -Defines the pluggable logging interface for osquery, providing the base class and supporting types for implementing custom log destinations (e.g., Flume, Splunk, syslog). +Defines the `LoggerPlugin` base class and supporting types for osquery's pluggable logging system, enabling custom log destinations (Splunk, syslog, Flume, etc.) via a plugin interface. ## Key Components -### Enumerations -- **`LoggerFeatures`** β€” Opt-in feature flags: `LOGGER_FEATURE_BLANK`, `LOGGER_FEATURE_LOGSTATUS`, `LOGGER_FEATURE_LOGEVENT` -- **`StatusLogSeverity`** β€” Severity levels mirroring Glog: `O_INFO`, `O_WARNING`, `O_ERROR`, `O_FATAL` - -### Structs -- **`StatusLogLine`** β€” Intermediate log entry holding severity, filename, line number, message, timestamp, UNIX time, and host identifier - -### Classes -- **`LoggerPlugin`** β€” Abstract base class (extends `Plugin`) for all logger implementations - -### `LoggerPlugin` Virtual Methods - -| Method | Required | Description | -|---|---|---| -| `logString(s)` | βœ… Pure virtual | Core log output handler | -| `init(binary_name, log)` | βœ… Pure virtual | Called once at startup with buffered pre-init logs | -| `logStatus(log)` | Optional | Handles Glog status lines | -| `logSnapshot(s)` | Optional | Handles snapshot query results; defaults to `logString` | -| `logEvent(s)` | Optional | Handles individual forwarded events | -| `logStringBatch(batch)` | Optional | Handles a JSON array batch of events | +**Enumerations** +- `LoggerFeatures` β€” Feature flags (`LOGGER_FEATURE_BLANK`, `LOGGER_FEATURE_LOGSTATUS`, `LOGGER_FEATURE_LOGEVENT`) for opt-in logger capabilities +- `StatusLogSeverity` β€” Severity levels (`O_INFO`, `O_WARNING`, `O_ERROR`, `O_FATAL`) mapped to Glog's `LogSeverity` + +**Structs** +- `StatusLogLine` β€” Intermediate log record holding `severity`, `filename`, `line`, `message`, `calendar_time`, `time`, and `identifier` + +**Class: `LoggerPlugin`** *(extends `Plugin`)* +| Method | Description | +|--------|-------------| +| `logString(s)` | **Pure virtual.** Core logging method all subclasses must implement | +| `init(binary_name, log)` | **Pure virtual.** Called once on startup with buffered pre-init logs | +| `logStatus(log)` | Optional. Handle Glog status lines directly | +| `logSnapshot(s)` | Optional. Separate sink for snapshot query results; defaults to `logString` | +| `logEvent(s)` | Optional. Per-event forwarding, bypassing the database layer | +| `logStringBatch(batch)` | Optional. Batch event processing; iterates JSON array calling `logString` | +| `usesLogStatus()` | Feature flag; return `true` to intercept Glog output | +| `usesLogEvent()` | Feature flag; return `true` to receive direct event forwarding | ## Usage Example @@ -32,25 +30,17 @@ Defines the pluggable logging interface for osquery, providing the base class an class MyLoggerPlugin : public osquery::LoggerPlugin { public: osquery::Status logString(const std::string& s) override { - writeToMyBackend(s); - return osquery::Status::success(); - } - - bool usesLogStatus() override { return true; } - - osquery::Status logStatus( - const std::vector& log) override { - for (const auto& line : log) { - writeStatusToMyBackend(line.severity, line.message); - } + sendToMyBackend(s); return osquery::Status::success(); } protected: void init(const std::string& binary_name, const std::vector& log) override { - setProcessName(binary_name); - logStatus(log); // flush buffered pre-init logs + // Flush buffered pre-init status logs + for (const auto& entry : log) { + sendToMyBackend(entry.message); + } } }; diff --git a/osquery/core/plugins/.plugin.md b/osquery/core/plugins/.plugin.md index 57a9afb5479..9fe6e1b47f0 100644 --- a/osquery/core/plugins/.plugin.md +++ b/osquery/core/plugins/.plugin.md @@ -1,57 +1,50 @@ -Defines the core `Plugin` base class and associated types for the osquery plugin/registry system, providing the interface that all registry items must implement. +Header defining the core plugin interface and type aliases for osquery's registry-based plugin system. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `PluginRequest` | `std::map` | Key-value input map passed to a plugin call, typically includes an `"action"` key | -| `PluginResponse` | `std::vector` | Vector of key-value maps returned from a successful plugin call | -| `Plugin` | Abstract class | Non-copyable base class all plugins must inherit; requires `call()` implementation | -| `PluginRef` | `std::shared_ptr` | Shared pointer alias for registry-managed plugin instances | -| `tableRowsToPluginResponse()` | Free function | Converts a `TableRows` result into a `PluginResponse` | +- **`PluginRequest`** β€” Type alias for `std::map`; carries action keys and parameters into a plugin call +- **`PluginResponse`** β€” Type alias for `std::vector`; carries structured results back from a plugin call +- **`Plugin`** β€” Abstract base class (non-copyable) that all registry plugins must inherit from +- **`PluginRef`** β€” Convenience alias for `std::shared_ptr` +- **`tableRowsToPluginResponse()`** β€” Free function to convert `TableRows` query results into a `PluginResponse` -### `Plugin` Virtual Methods +### `Plugin` Virtual Interface | Method | Required | Description | |--------|----------|-------------| -| `call()` | βœ… Pure virtual | Handles plugin invocation with request/response | -| `setUp()` | ❌ Optional | One-time initialization logic | -| `tearDown()` | ❌ Optional | Cleanup/release logic | -| `configure()` | ❌ Optional | React to configuration changes | -| `routeInfo()` | ❌ Optional | Publish additional routing metadata | -| `addExternal()` | ❌ Static | Hook for when an external plugin is registered | -| `removeExternal()` | ❌ Static | Hook for when an external plugin is removed | +| `call()` | βœ… Pure virtual | Core dispatch method; handles a `PluginRequest` and fills a `PluginResponse` | +| `setUp()` | Optional | One-time initialization on plugin load | +| `tearDown()` | Optional | Cleanup on plugin unload | +| `configure()` | Optional | React to configuration changes | +| `routeInfo()` | Optional | Publish extra routing metadata | +| `addExternal()` | Static | Hook for when an external (extension) plugin is registered | +| `removeExternal()` | Static | Hook for when an external plugin is removed | ## Usage Example ```cpp #include -namespace osquery { - -class MyPlugin : public Plugin { +class MyPlugin : public osquery::Plugin { public: - Status call(const PluginRequest& request, - PluginResponse& response) override { + osquery::Status call(const osquery::PluginRequest& request, + osquery::PluginResponse& response) override { auto action = request.find("action"); if (action == request.end()) { - return Status::failure("Missing action key"); + return osquery::Status::failure("No action specified"); } if (action->second == "query") { - response.push_back({{"result", "hello from MyPlugin"}}); - return Status::success(); + response.push_back({{"result", "some_value"}}); } - return Status::failure("Unknown action: " + action->second); + return osquery::Status::success(); } - Status setUp() override { - // Optional: initialize resources - return Status::success(); + osquery::Status setUp() override { + // Optional one-time init + return osquery::Status::success(); } }; - -} // namespace osquery ``` \ No newline at end of file diff --git a/osquery/core/plugins/.sql.md b/osquery/core/plugins/.sql.md index c0e5554f802..13727b31bfa 100644 --- a/osquery/core/plugins/.sql.md +++ b/osquery/core/plugins/.sql.md @@ -1,49 +1,42 @@ -Defines the abstract `SQLPlugin` interface that enables osquery's pluggable SQL execution layer, decoupling the SQL API from its underlying implementation (e.g., SQLite). +Defines the abstract `SQLPlugin` interface for osquery's pluggable SQL backend, allowing the SQL implementation (e.g., SQLite) to be registered and swapped via the osquery plugin registry without coupling core SDK code to third-party dependencies. ## Key Components -- **`SQLPlugin`** β€” Abstract base class inheriting from `Plugin` that defines the contract for SQL backend implementations registered under the `"sql"` registry key. - -| Method | Description | -|---|---| -| `query()` | Executes a SQL query string and populates a `QueryData` result set | -| `getQueryColumns()` | Parses a query and returns column metadata (`name`, `type`) | -| `getQueryTables()` | Returns the list of virtual tables scanned by a given query | -| `attach()` | Optionally attaches a virtual table at runtime (no-op default) | -| `detach()` | Optionally detaches a virtual table by name (no-op default) | -| `call()` | Dispatches plugin registry requests to the appropriate method | +- **`SQLPlugin`** β€” Abstract plugin class inheriting from `Plugin` that defines the contract for SQL backend implementations: + - `query()` β€” Executes a SQL query string, returning results in a `QueryData` object + - `getQueryColumns()` β€” Parses a query and returns column metadata (`name`, `type`) as `TableColumns` + - `getQueryTables()` β€” Extracts the list of scanned table names from a query + - `attach()` β€” Virtual hook to attach a named virtual table at runtime (default: no-op success) + - `detach()` β€” Virtual hook to detach a named virtual table (default: no-op success) + - `call()` β€” Overrides the base `Plugin::call()` to handle `PluginRequest`/`PluginResponse` dispatch ## Usage Example ```cpp #include -// Implementing a custom SQL backend plugin class MySQLPlugin : public osquery::SQLPlugin { public: osquery::Status query(const std::string& q, osquery::QueryData& results, bool use_cache) const override { - // Execute query against your SQL backend + // Execute query against custom SQL backend return osquery::Status::success(); } osquery::Status getQueryColumns(const std::string& q, - osquery::TableColumns& columns) const override { - // Parse and return column descriptors + osquery::TableColumns& cols) const override { + // Parse and return column metadata return osquery::Status::success(); } osquery::Status getQueryTables(const std::string& q, std::vector& tables) const override { - // Return scanned table names + // Extract referenced table names return osquery::Status::success(); } }; -// Register the plugin under the "sql" registry REGISTER(MySQLPlugin, "sql", "sql"); -``` - -> **Note:** The default `attach()` and `detach()` implementations return `Status::success()` β€” override them only if your backend requires runtime virtual table management (as SQLite does during initialization). \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/core/sql/.column.md b/osquery/core/sql/.column.md index 997711a199b..38955890acc 100644 --- a/osquery/core/sql/.column.md +++ b/osquery/core/sql/.column.md @@ -1,45 +1,40 @@ -Defines column metadata types and enumerations used to describe osquery table schema β€” column options, column types, and related type aliases for table definitions. +Defines column metadata types used by osquery's table plugin system, including column options (flags), column types, and related type aliases for describing table schema. ## Key Components -### `ColumnOptions` (enum class) -Bitmask flags controlling column behavior in SQLite table implementations: +- **`ColumnOptions`** β€” Bitmask enum controlling column behavior: + - `DEFAULT`, `INDEX`, `REQUIRED`, `ADDITIONAL`, `OPTIMIZED`, `HIDDEN` + - Collation options: `COLLATEBINARY`, `COLLATENOCASE`, `COLLATERTRIM`, `COLLATEVERSION`, `COLLATEVERSION_ARCH`, `COLLATEVERSION_DPKG`, `COLLATEVERSION_RHEL` -| Flag | Value | Description | -|------|-------|-------------| -| `DEFAULT` | 0 | No special behavior | -| `INDEX` | 1 | Treat as primary key | -| `REQUIRED` | 2 | Must appear in query predicate | -| `ADDITIONAL` | 4 | Generates extra data when in predicate | -| `OPTIMIZED` | 8 | Enables query-time optimization | -| `HIDDEN` | 16 | Excluded from `SELECT *` results | -| `COLLATEBINARY` / `COLLATE*` | 32–2048 | SQLite collation sequences | +- **`operator|` / `operator&`** β€” Bitwise operators enabling `ColumnOptions` to be combined and tested as flags. -Bitwise `|` and `&` operators are overloaded for flag composition. +- **`ColumnType`** β€” Enum of supported SQLite column types: `TEXT_TYPE`, `INTEGER_TYPE`, `BIGINT_TYPE`, `UNSIGNED_BIGINT_TYPE`, `DOUBLE_TYPE`, `BLOB_TYPE`. -### `ColumnType` (enum) -Supported SQLite column data types: `TEXT_TYPE`, `INTEGER_TYPE`, `BIGINT_TYPE`, `UNSIGNED_BIGINT_TYPE`, `DOUBLE_TYPE`, `BLOB_TYPE`, `UNKNOWN_TYPE`. +- **`kColumnTypeNames`** β€” External map from `ColumnType` to its SQLite string representation (e.g., `TEXT_TYPE` β†’ `"TEXT"`). -### Type Aliases -- `TableName` β€” `std::string` alias for plugin/table names -- `TableColumns` β€” ordered vector of `(name, ColumnType, ColumnOptions)` tuples describing a table's schema -- `ColumnAliasSet` β€” map of column name to a set of alias strings +- **`TableColumns`** β€” Ordered vector of tuples `(column_name, ColumnType, ColumnOptions)` describing a table's schema. -### `kColumnTypeNames` -External map from `ColumnType` to its SQLite string representation (e.g., `TEXT`, `INTEGER`). +- **`ColumnAliasSet`** β€” Map of column names to their alias sets. ## Usage Example -```cpp -#include +```c #include "column.h" -osquery::TableColumns myTableColumns = { - // Column name, type, options - {"uid", osquery::INTEGER_TYPE, osquery::ColumnOptions::INDEX}, - {"name", osquery::TEXT_TYPE, osquery::ColumnOptions::DEFAULT}, - {"token", osquery::TEXT_TYPE, osquery::ColumnOptions::HIDDEN | - osquery::ColumnOptions::REQUIRED}, +using namespace osquery; + +// Define a table's columns with types and options +TableColumns myTableColumns = { + {"pid", INTEGER_TYPE, ColumnOptions::INDEX}, + {"name", TEXT_TYPE, ColumnOptions::DEFAULT}, + {"uid", BIGINT_TYPE, ColumnOptions::ADDITIONAL | ColumnOptions::HIDDEN}, + {"version", TEXT_TYPE, ColumnOptions::OPTIMIZED | ColumnOptions::COLLATEVERSION}, }; + +// Test a column option flag +ColumnOptions opts = ColumnOptions::INDEX | ColumnOptions::REQUIRED; +if (opts & ColumnOptions::REQUIRED) { + // Column requires a WHERE predicate +} ``` \ No newline at end of file diff --git a/osquery/core/sql/.diff_results.md b/osquery/core/sql/.diff_results.md index fdf548bc014..51af93279fa 100644 --- a/osquery/core/sql/.diff_results.md +++ b/osquery/core/sql/.diff_results.md @@ -1,44 +1,44 @@ -Defines the `DiffResults` structure and related utilities for computing and serializing the difference between two query result sets in osquery. +Defines the `DiffResults` structure and related utilities for computing and serializing the difference between two osquery query result sets. ## Key Components ### `DiffResults` (struct) -A move-only struct representing the delta between two `QueryDataTyped` result sets: +A move-only struct holding the delta between two `QueryDataTyped` result sets. | Member | Type | Description | -|---|---|---| +|--------|------|-------------| | `added` | `QueryDataTyped` | Rows present in the new result but not the old | | `removed` | `QueryDataTyped` | Rows present in the old result but not the new | - -**Methods:** -- `hasNoResults()` β€” Returns `true` if both `added` and `removed` are empty -- `operator==` / `operator!=` β€” Equality comparison between two `DiffResults` +| `hasNoResults()` | `bool` | Returns `true` if both `added` and `removed` are empty | +| `operator==` / `operator!=` | `bool` | Value-equality comparisons | ### Free Functions -| Function | Description | -|---|---| -| `diff(old_, new_)` | Computes delta between a `QueryDataSet` and `QueryDataTyped`, returns a `DiffResults` | -| `serializeDiffResults(...)` | Serializes a `DiffResults` into a `rapidjson::Document` object | -| `serializeDiffResultsJSON(...)` | Serializes a `DiffResults` into a JSON string | +| Function | Returns | Description | +|----------|---------|-------------| +| `diff(old_, new_)` | `DiffResults` | Computes the delta between a `QueryDataSet` and a `QueryDataTyped` | +| `serializeDiffResults(d, doc, obj, asNumeric)` | `Status` | Serializes into a `rapidjson::Document` object with `added`/`removed` keys | +| `serializeDiffResultsJSON(d, json, asNumeric)` | `Status` | Serializes into a JSON string | ## Usage Example ```cpp -// Compute diff between old and new query results -QueryDataSet oldResults = getStoredResults(); -QueryDataTyped newResults = runQuery(); +#include -DiffResults delta = osquery::diff(oldResults, newResults); +// Compute diff between previous and current query results +QueryDataSet previous = getPreviousResults(); +QueryDataTyped current = getCurrentResults(); -if (!delta.hasNoResults()) { - std::string jsonOutput; - auto status = osquery::serializeDiffResultsJSON(delta, jsonOutput, false); +DiffResults delta = osquery::diff(previous, current); +if (!delta.hasNoResults()) { + std::string json; + auto status = osquery::serializeDiffResultsJSON(delta, json, /*asNumeric=*/false); if (status.ok()) { - // jsonOutput contains {"added": [...], "removed": [...]} - processChanges(jsonOutput); + processChanges(json); // { "added": [...], "removed": [...] } } } -``` \ No newline at end of file +``` + +> **Note:** `DiffResults` is move-only (`only_movable`). Use `std::move` when passing instances between scopes β€” copy construction and assignment are disabled. \ No newline at end of file diff --git a/osquery/core/sql/.query_data.md b/osquery/core/sql/.query_data.md index 554cf1f10c3..4975f3cbc7c 100644 --- a/osquery/core/sql/.query_data.md +++ b/osquery/core/sql/.query_data.md @@ -1,25 +1,23 @@ -Defines the core result set types and serialization utilities for osquery SQL query results, providing `QueryData`, `QueryDataTyped`, and `QueryDataSet` container types along with JSON serialization/deserialization functions. +Defines the core result set types and serialization utilities for osquery SQL query results, providing `QueryData`, `QueryDataTyped`, and `QueryDataSet` containers along with JSON conversion functions. ## Key Components -### Type Aliases -| Type | Definition | Purpose | -|---|---|---| -| `QueryData` | `std::vector` | Standard untyped SQL result set | -| `QueryDataTyped` | `std::vector` | Typed SQL result set | -| `QueryDataSet` | `std::multiset` | Multiset for fast row lookup | +**Type Aliases** +- `QueryData` β€” `std::vector`; the standard untyped SQL result set +- `QueryDataTyped` β€” `std::vector`; typed SQL result set +- `QueryDataSet` β€” `std::multiset`; set-based result for fast row lookup -### Serialization Functions -- **`serializeQueryData`** β€” Converts `QueryData` or `QueryDataTyped` into a RapidJSON array, with optional numeric value preservation -- **`serializeQueryDataJSON`** β€” Serializes query results directly to a `JSON` document or `std::string` +**Serialization** +- `serializeQueryData()` β€” Serializes `QueryData` or `QueryDataTyped` into a `rapidjson::Document` array; typed variant accepts an `asNumeric` flag to preserve numeric types +- `serializeQueryDataJSON()` β€” Convenience overloads serializing directly to a `JSON` document or `std::string` -### Deserialization Functions -- **`deserializeQueryData`** β€” Overloaded for `QueryData`, `QueryDataTyped`, and `QueryDataSet` targets from a `rapidjson::Value` -- **`deserializeQueryDataJSON`** β€” Overloaded for `QueryData` and `QueryDataSet` targets from a JSON string or `JSON` document +**Deserialization** +- `deserializeQueryData()` β€” Converts a `rapidjson::Value` array back into `QueryData`, `QueryDataTyped`, or `QueryDataSet` +- `deserializeQueryDataJSON()` β€” Converts a `JSON` doc or JSON string back into `QueryData` or `QueryDataSet` -### Utility -- **`addUniqueRowToQueryData`** β€” Appends a `RowTyped` to a `QueryDataTyped` only if it does not already exist (linear scan) +**Utilities** +- `addUniqueRowToQueryData()` β€” Appends a `RowTyped` to a `QueryDataTyped` only if it is not already present (linear scan) ## Usage Example @@ -27,21 +25,20 @@ Defines the core result set types and serialization utilities for osquery SQL qu #include // Serialize query results to JSON string -osquery::QueryData results = runMyQuery(); -std::string jsonOutput; -auto status = osquery::serializeQueryDataJSON(results, jsonOutput); -if (!status.ok()) { - LOG(ERROR) << "Serialization failed: " << status.getMessage(); +osquery::QueryData results = runSomeQuery(); +std::string json; +auto status = osquery::serializeQueryDataJSON(results, json); +if (status.ok()) { + LOG(INFO) << "Results: " << json; } // Deserialize back from JSON osquery::QueryData restored; -status = osquery::deserializeQueryDataJSON(jsonOutput, restored); - -// Add unique rows only -osquery::QueryDataTyped typedResults; -osquery::RowTyped row = buildRow(); -bool added = osquery::addUniqueRowToQueryData(typedResults, row); -``` - -> **Note:** `addUniqueRowToQueryData` performs a linear scan β€” avoid using it in performance-critical loops with large result sets. \ No newline at end of file +osquery::deserializeQueryDataJSON(json, restored); + +// Append unique typed rows +osquery::QueryDataTyped typed; +osquery::RowTyped row = {{"column", {1LL}}}; +osquery::addUniqueRowToQueryData(typed, row); // added +osquery::addUniqueRowToQueryData(typed, row); // skipped (duplicate) +``` \ No newline at end of file diff --git a/osquery/core/sql/.query_performance.md b/osquery/core/sql/.query_performance.md index 2441d260de2..40a0516c480 100644 --- a/osquery/core/sql/.query_performance.md +++ b/osquery/core/sql/.query_performance.md @@ -1,45 +1,49 @@ -Defines the `QueryPerformance` struct within the `osquery` namespace for tracking runtime performance statistics of scheduled queries. +Defines the `QueryPerformance` struct within the `osquery` namespace for tracking per-query execution statistics, including timing, memory, and output metrics. ## Key Components ### `QueryPerformance` (struct) -Aggregates cumulative and per-execution metrics for a single query: +Aggregates performance data collected across query executions: | Field | Type | Description | -|---|---|---| +|-------|------|-------------| | `executions` | `size_t` | Total number of times the query has run | -| `last_executed` | `uint64_t` | UNIX timestamp (seconds) of last successful execution | -| `wall_time` / `wall_time_ms` | `uint64_t` | Cumulative wall clock time (seconds / milliseconds) | +| `last_executed` | `uint64_t` | Last successful execution timestamp (UNIX seconds) | +| `wall_time` / `wall_time_ms` | `uint64_t` | Cumulative wall time (seconds / milliseconds) | | `last_wall_time_ms` | `uint64_t` | Wall time of the most recent execution (ms) | -| `user_time` / `last_user_time` | `uint64_t` | Cumulative and latest user-space CPU time (ms) | -| `system_time` / `last_system_time` | `uint64_t` | Cumulative and latest kernel CPU time (ms) | -| `average_memory` | `uint64_t` | Average resident memory (bytes) after result collection | -| `last_memory` | `uint64_t` | Resident memory of the most recent execution (bytes) | +| `user_time` / `last_user_time` | `uint64_t` | Cumulative and latest user-mode CPU time (ms) | +| `system_time` / `last_system_time` | `uint64_t` | Cumulative and latest kernel-mode CPU time (ms) | +| `average_memory` | `uint64_t` | Average resident memory after result collection (bytes) | +| `last_memory` | `uint64_t` | Resident memory after the latest execution (bytes) | | `output_size` | `uint64_t` | Total bytes produced by the query | **Methods:** -- `QueryPerformance(const std::string& csv)` β€” Deserializes a `QueryPerformance` from a CSV string -- `toCSV() const` β€” Serializes the struct to a CSV string for persistence -- `operator==` β€” Equality comparison between two instances +- `QueryPerformance(const std::string& csv)` β€” Constructs an instance by deserializing a CSV string +- `toCSV() const` β€” Serializes the struct to a CSV string for storage or transport +- `operator==` β€” Equality comparison between two `QueryPerformance` instances ## Usage Example ```cpp #include "query_performance.h" -// Default-initialize and inspect after a query run +// Default construction osquery::QueryPerformance perf; -perf.executions++; -perf.last_wall_time_ms = 42; +perf.executions = 1; +perf.wall_time_ms = 42; +perf.last_memory = 204800; -// Persist to CSV -std::string serialized = perf.toCSV(); +// Serialize to CSV for persistence +std::string csv = perf.toCSV(); // Restore from CSV -osquery::QueryPerformance restored(serialized); +osquery::QueryPerformance restored(csv); -assert(perf == restored); +// Compare +if (perf == restored) { + // Serialization round-trip succeeded +} ``` \ No newline at end of file diff --git a/osquery/core/sql/.row.md b/osquery/core/sql/.row.md index e0e3d010bcc..7fecf377044 100644 --- a/osquery/core/sql/.row.md +++ b/osquery/core/sql/.row.md @@ -1,27 +1,24 @@ -Defines the core data types and serialization utilities for representing single database query rows in osquery, supporting both string-based and typed (SQLite affinity) row formats. +Defines the core data structures and serialization utilities for representing and manipulating single database query rows within the osquery framework. ## Key Components ### Type Aliases - | Type | Definition | -|------|-----------| +|------|------------| | `RowData` | `std::string` β€” raw column value | | `RowDataTyped` | `boost::variant` β€” typed column value | -| `Row` | `std::map` β€” untyped string row | -| `RowTyped` | `std::map` β€” typed variant row | +| `Row` | `std::map` β€” untyped row (column β†’ string) | +| `RowTyped` | `std::map` β€” typed row (column β†’ variant) | | `ColumnNames` | `std::vector` β€” ordered column name list | ### Serialization Functions - -- **`serializeRow`** β€” Serializes a `Row` or `RowTyped` into a `rapidjson::Value` object, with optional column ordering via `ColumnNames` -- **`serializeRowJSON`** β€” Serializes a `Row` or `RowTyped` directly to a JSON string; typed variant accepts an `asNumeric` flag to preserve numeric types +- **`serializeRow`** β€” Serializes a `Row` or `RowTyped` into a `rapidjson::Value` object within a managed `JSON` document +- **`serializeRowJSON`** β€” Serializes a `Row` or `RowTyped` directly to a JSON string; supports optional numeric type preservation via `asNumeric` ### Deserialization Functions - - **`deserializeRow`** β€” Deserializes a `rapidjson::Value` object into a `Row` or `RowTyped` -- **`deserializeRowJSON`** β€” Deserializes a JSON string into a `Row` or `RowTyped` +- **`deserializeRowJSON`** β€” Deserializes a raw JSON string into a `Row` or `RowTyped` All functions return an osquery `Status` object indicating success or failure. @@ -29,29 +26,30 @@ All functions return an osquery `Status` object indicating success or failure. ```c #include -#include +#include -// Serialize a Row to JSON string +// Build a simple row osquery::Row r; -r["pid"] = "1234"; +r["pid"] = "1234"; r["name"] = "my_process"; -std::string json_output; -auto status = osquery::serializeRowJSON(r, json_output); +// Serialize to JSON string +std::string json; +auto status = osquery::serializeRowJSON(r, json); if (status.ok()) { - // json_output -> {"pid":"1234","name":"my_process"} + // json == {"pid":"1234","name":"my_process"} } -// Deserialize back from JSON string -osquery::Row restored; -osquery::deserializeRowJSON(json_output, restored); +// Deserialize back from JSON +osquery::Row r2; +osquery::deserializeRowJSON(json, r2); -// Using typed rows with numeric preservation -osquery::RowTyped typed_row; -typed_row["pid"] = (long long)1234; -typed_row["load"] = 0.75; +// Typed row with numeric preservation +osquery::RowTyped rt; +rt["pid"] = 1234LL; +rt["load"] = 0.75; +rt["name"] = std::string("my_process"); -std::string typed_json; -osquery::serializeRowJSON(typed_row, typed_json, /*asNumeric=*/true); -// typed_json -> {"pid":1234,"load":0.75} +std::string typedJson; +osquery::serializeRowJSON(rt, typedJson, /*asNumeric=*/true); ``` \ No newline at end of file diff --git a/osquery/core/sql/.scheduled_query.md b/osquery/core/sql/.scheduled_query.md index 42d5c135e19..c9dfaf24754 100644 --- a/osquery/core/sql/.scheduled_query.md +++ b/osquery/core/sql/.scheduled_query.md @@ -1,48 +1,49 @@ -Defines the `ScheduledQuery` struct used within osqueryd to represent all relevant parameters and metadata for a scheduled SQL query. +Defines the `ScheduledQuery` struct representing a single scheduled query's configuration within the osqueryd daemon. ## Key Components ### `ScheduledQuery` (struct) -A move-only data structure (inherits `only_movable`) containing: +A move-only data structure (inherits `only_movable`) encapsulating all attributes of a scheduled query: | Field | Type | Description | |---|---|---| | `pack_name` | `std::string` | Name of the containing pack | | `name` | `std::string` | Query identifier | -| `query` | `std::string` | Raw SQL query string | -| `oncall` | `std::string` | Query owner/team | +| `query` | `std::string` | SQL statement | +| `oncall` | `std::string` | Query owner | | `interval` | `uint64_t` | Execution frequency in seconds | -| `splayed_interval` | `uint64_t` | Temporary splayed interval value | -| `startup_priority` | `uint64_t` | Startup execution priority (`UINT64_MAX` = disabled) | -| `denylisted` | `bool` | Whether query is currently denylisted | -| `options` | `std::map` | Key/value query option flags | +| `splayed_interval` | `uint64_t` | Temporary splayed interval | +| `startup_priority` | `uint64_t` | Startup run priority (`UINT64_MAX` = disabled) | +| `denylisted` | `bool` | Whether query is denylisted | +| `options` | `map` | Key-value query options (e.g. `snapshot`, `removed`) | -**Methods:** -- `isSnapshotQuery()` β€” Returns `true` if the `"snapshot"` option is set -- `reportRemovedRows()` β€” Returns `true` if the `"removed"` option is set or absent (default behavior) -- `operator==` / `operator!=` β€” Equality based on `query` and `interval` fields +### Key Methods + +- **`isSnapshotQuery()`** β€” Returns `true` if the `snapshot` option is set to `true` +- **`reportRemovedRows()`** β€” Returns `true` if the `removed` option is set or absent (default behavior) +- **`operator==` / `operator!=`** β€” Equality based on `query` and `interval` fields only ## Usage Example ```cpp #include -// Construct with required fields -osquery::ScheduledQuery sq("my_pack", "active_users", "SELECT * FROM users;"); +// Construct a scheduled query +osquery::ScheduledQuery sq("my_pack", "running_procs", "SELECT * FROM processes;"); sq.interval = 60; sq.options["snapshot"] = false; sq.options["removed"] = true; -// Check query behavior +// Inspect query behavior if (sq.isSnapshotQuery()) { - // handle snapshot mode + // handle snapshot mode } if (sq.reportRemovedRows()) { - // include removed row diffs in results + // emit removed row events } -// Move semantics only β€” copies are disabled +// Move semantics only β€” copy is disabled osquery::ScheduledQuery sq2 = std::move(sq); ``` \ No newline at end of file diff --git a/osquery/core/sql/.table_rows.md b/osquery/core/sql/.table_rows.md index 91963c102ad..f7211a7c353 100644 --- a/osquery/core/sql/.table_rows.md +++ b/osquery/core/sql/.table_rows.md @@ -3,45 +3,33 @@ Defines the `TableRows` type alias and declares serialization/deserialization ut ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `TableRows` | Type alias | `std::vector` β€” an ordered collection of table row objects | -| `serializeTableRows` | Function | Serializes a `TableRows` into a `rapidjson::Document` JSON array | -| `serializeTableRowsJSON` | Function | Serializes a `TableRows` directly into a JSON string | -| `deserializeTableRows` | Function | Converts a `rapidjson::Value` JSON array back into a `TableRows` | -| `deserializeTableRowsJSON` | Function | Converts a raw JSON string back into a `TableRows` | - -All functions return `osquery::Status` to indicate success or failure. +- **`TableRows`** β€” Type alias for `std::vector`, representing a collection of query result rows. +- **`serializeTableRows()`** β€” Serializes a `TableRows` object into a `rapidjson::Document` JSON array. +- **`serializeTableRowsJSON()`** β€” Serializes a `TableRows` object directly into a JSON-formatted `std::string`. +- **`deserializeTableRows()`** β€” Converts a `rapidjson::Value` JSON array back into a `TableRows` object. +- **`deserializeTableRowsJSON()`** β€” Converts a raw JSON string back into a `TableRows` object. ## Usage Example ```c #include "table_rows.h" -// Serialize to JSON string -osquery::TableRows rows = getMyTableRows(); -std::string jsonOutput; +osquery::TableRows rows; +// ... populate rows from a query ... -osquery::Status s = osquery::serializeTableRowsJSON(rows, jsonOutput); -if (!s.ok()) { - LOG(ERROR) << "Serialization failed: " << s.getMessage(); +// Serialize to JSON string +std::string json_output; +osquery::Status status = osquery::serializeTableRowsJSON(rows, json_output); +if (status.ok()) { + // json_output now contains the serialized rows } -// Deserialize from JSON string -osquery::TableRows restored; -osquery::Status ds = osquery::deserializeTableRowsJSON(jsonOutput, restored); +// Deserialize back from JSON string +osquery::TableRows restored_rows; +osquery::Status ds = osquery::deserializeTableRowsJSON(json_output, restored_rows); if (!ds.ok()) { - LOG(ERROR) << "Deserialization failed: " << ds.getMessage(); + // handle error } - -// Serialize into an existing JSON document/array -osquery::JSON doc; -rapidjson::Document arr; -osquery::serializeTableRows(rows, doc, arr); ``` -## Dependencies - -- `osquery/utils/json/json.h` β€” JSON document wrapper -- `query_data.h` β€” underlying query result types -- `table_row.h` β€” `TableRowHolder` definition \ No newline at end of file +All four functions return an `osquery::Status` object β€” check `.ok()` to confirm success or retrieve an error message via `.getMessage()`. \ No newline at end of file diff --git a/osquery/core/windows/.global_users_groups_cache.md b/osquery/core/windows/.global_users_groups_cache.md index 42bc31f30e6..67f5427aa59 100644 --- a/osquery/core/windows/.global_users_groups_cache.md +++ b/osquery/core/windows/.global_users_groups_cache.md @@ -1,34 +1,31 @@ -Declares the `GlobalUsersGroupsCache` class, which provides synchronized global access to Windows user and group cache data initialized by background services. +Provides a thread-safe global accessor for Windows user and group caches, blocking until the respective cache has been initialized by its background service before returning a reference. ## Key Components -### `GlobalUsersGroupsCache` (class) - -A singleton-style accessor class for Windows users and groups cache data, designed to block until cache initialization completes before returning references. +**`GlobalUsersGroupsCache`** β€” Singleton-style class exposing two static accessors backed by `std::shared_future` synchronization. | Member | Type | Description | |--------|------|-------------| | `getUsersCache()` | `static const UsersCache&` | Blocks until the users cache is ready, then returns a const reference | | `getGroupsCache()` | `static const GroupsCache&` | Blocks until the groups cache is ready, then returns a const reference | -| `global_users_cache_future_` | `std::shared_future` | Synchronization primitive tracking users cache initialization | -| `global_groups_cache_future_` | `std::shared_future` | Synchronization primitive tracking groups cache initialization | -| `global_users_cache_` | `std::shared_ptr` | Shared ownership pointer to the users cache instance | -| `global_groups_cache_` | `std::shared_ptr` | Shared ownership pointer to the groups cache instance | +| `global_users_cache_future_` | `std::shared_future` | Signals when the users cache has finished initialization | +| `global_groups_cache_future_` | `std::shared_future` | Signals when the groups cache has finished initialization | +| `global_users_cache_` | `std::shared_ptr` | Owned users cache instance | +| `global_groups_cache_` | `std::shared_ptr` | Owned groups cache instance | -**Friend declarations:** `Initializer`, `initUsersAndGroupsServices`, `deinitUsersAndGroupsServices` β€” only these can populate the private cache state. +**Friend declarations** grant `Initializer`, `initUsersAndGroupsServices`, and `deinitUsersAndGroupsServices` access to private members for lifecycle management. ## Usage Example ```cpp #include -// Blocks until the background service has finished populating the cache, -// then returns a safe const reference for querying. +// Safely retrieve cached users β€” blocks if initialization is still in progress const UsersCache& users = GlobalUsersGroupsCache::getUsersCache(); -const GroupsCache& groups = GlobalUsersGroupsCache::getGroupsCache(); -// Use the caches for lookup operations (Windows only) +// Safely retrieve cached groups β€” blocks if initialization is still in progress +const GroupsCache& groups = GlobalUsersGroupsCache::getGroupsCache(); ``` -> **Note:** Both accessors are blocking calls. They rely on `std::shared_future` to ensure the background initialization service has completed before returning. This header is Windows-specific and wraps `users_groups_cache.h`. \ No newline at end of file +> **Note:** Both accessors use `std::shared_future::get()` semantics internally, making them safe to call from multiple threads simultaneously. Initialization is performed by the background services wired up through `initUsersAndGroupsServices`. \ No newline at end of file diff --git a/osquery/core/windows/.handle.md b/osquery/core/windows/.handle.md index d38911d1f47..d947fa57708 100644 --- a/osquery/core/windows/.handle.md +++ b/osquery/core/windows/.handle.md @@ -1,41 +1,54 @@ -RAII-style wrapper class for Windows kernel object handles (`HANDLE`), providing safe open/close lifecycle management for directory and symbolic link objects. +RAII-style wrapper around a Windows `HANDLE` that provides safe lifecycle management and named object access for directories and symbolic links. ## Key Components -### `Handle` class +### `Handle` Class | Member | Description | -|---|---| +|--------|-------------| | `openDirObj(directory)` | Opens a Windows object directory by name using `OPEN_DIRECTORY` access mask | | `openSymLinkObj(symlink)` | Opens a Windows symbolic link by name using `SYMBOLIC_LINK_QUERY` access mask | | `getAsHandle()` | Returns the underlying `HANDLE` value | -| `close()` | Explicitly releases the handle | +| `close()` | Explicitly closes the handle | | `valid()` | Returns `true` if the handle is currently open and valid | -| `~Handle()` | Destructor automatically closes the handle on scope exit | +| `~Handle()` | Destructor automatically closes the handle (RAII) | ## Usage Example -```c +```cpp #include "handle.h" osquery::Handle h; // Open a Windows object directory -osquery::Status s = h.openDirObj(L"\\BaseNamedObjects"); -if (!s.ok()) { +osquery::Status status = h.openDirObj(L"\\BaseNamedObjects"); +if (!status.ok()) { // handle error } +// Use the raw HANDLE if needed HANDLE raw = h.getAsHandle(); // Check validity before use if (h.valid()) { - // perform operations on raw handle + // perform operations } -// Explicitly close, or destructor will close on scope exit +// Explicit close (also called automatically on destruction) h.close(); + +// Open a symbolic link +osquery::Handle linkHandle; +osquery::Status linkStatus = linkHandle.openSymLinkObj(L"\\DosDevices\\C:"); +if (!linkStatus.ok()) { + // handle error +} +// linkHandle closes automatically when it goes out of scope ``` -> **Note:** This class is Windows-only and relies on `osquery/utils/system/system.h`. The RAII destructor ensures handles are not leaked even if exceptions occur, making it safer than manual `CloseHandle` calls. \ No newline at end of file +## Notes + +- Defined within the `osquery` namespace +- Windows-only; depends on `osquery/utils/system/system.h` +- Returns `osquery::Status` from open methods for consistent error propagation across the osquery framework \ No newline at end of file diff --git a/osquery/core/windows/.wmi.md b/osquery/core/windows/.wmi.md index c5bf928f37f..82b1b82a4ec 100644 --- a/osquery/core/windows/.wmi.md +++ b/osquery/core/windows/.wmi.md @@ -1,50 +1,46 @@ -Windows Management Instrumentation (WMI) query abstraction layer for osquery, providing C++ wrapper classes to execute WMI queries and retrieve typed results on Windows systems. +Windows Management Instrumentation (WMI) wrapper for osquery, providing a C++ interface to query and retrieve Windows system information via WMI queries. ## Key Components ### `WmiMethodArgs` -Constructs and holds arguments for WMI method calls. Used alongside `WmiResultItem::ExecMethod`. +Constructs and holds arguments for WMI method calls. Used with `WmiResultItem::ExecMethod`. - `Put(name, value)` β€” Adds a typed argument to the call - `GetArguments()` β€” Returns the underlying `WmiMethodArgsMap` ### `WmiResultItem` -Wraps a single `IWbemClassObject` result, providing typed getters for WMI properties: -- `GetBool`, `GetString`, `GetLong`, `GetUnsignedInt32`, `GetDateTime`, etc. -- `GetVectorOfStrings`, `GetVectorOfLongs` β€” For multi-value properties -- `PrintType(name)` β€” Debug helper to inspect property VARIANT types +Wraps a single `IWbemClassObject` result from a WMI query. Provides typed getters: +- `GetBool`, `GetString`, `GetLong`, `GetUnsignedInt32`, `GetLongLong`, etc. +- `GetDateTime` β€” Retrieves local/non-local `FILETIME` values +- `GetVectorOfStrings` / `GetVectorOfLongs` β€” Multi-value property support +- All getters return `Status` indicating success or failure ### `WmiRequest` -Main entry point for executing WMI queries against a namespace. -- `CreateWmiRequest(query, nspace)` β€” Static factory; returns `Expected` -- `results()` β€” Returns `std::vector` from the query -- `getStatus()` β€” Retrieves execution status +Primary query class. Manages WMI COM object lifecycle (`IWbemLocator`, `IWbemServices`, `IEnumWbemClassObject`). +- `CreateWmiRequest(query, nspace)` β€” Factory method returning `Expected`; defaults to `ROOT\CIMV2` +- `results()` β€” Returns the vector of `WmiResultItem` results - `ExecMethod(object, method, args, out)` β€” Executes a WMI method on a result object +- `getStatus()` β€” Returns the request execution status ### `impl::WmiObjectDeleter` -Internal RAII deleter for COM `IUnknown` pointers, ensuring `Release()` is called on cleanup. +Internal RAII deleter for COM `IUnknown` pointers, used with `std::unique_ptr`. ## Usage Example ```cpp #include -// Execute a WMI query +// Query all running processes auto request = osquery::WmiRequest::CreateWmiRequest( - "SELECT * FROM Win32_OperatingSystem" -); + "SELECT Name, ProcessId FROM Win32_Process"); if (request) { - for (const auto& item : request->results()) { - std::string caption; - if (item.GetString("Caption", caption).ok()) { - std::cout << "OS: " << caption << std::endl; + for (const auto& item : request->results()) { + std::string name; + unsigned int pid = 0; + item.GetString("Name", name); + item.GetUnsignedInt32("ProcessId", pid); + std::cout << name << " [" << pid << "]\n"; } - - unsigned long long totalMemory = 0; - if (item.GetUnsignedLongLong("TotalVisibleMemorySize", totalMemory).ok()) { - std::cout << "RAM (KB): " << totalMemory << std::endl; - } - } } ``` \ No newline at end of file diff --git a/osquery/database/.database.md b/osquery/database/.database.md index e9e76edce13..fbd16f2e8b8 100644 --- a/osquery/database/.database.md +++ b/osquery/database/.database.md @@ -1,12 +1,12 @@ -Defines the `DatabasePlugin` abstract interface and supporting free functions for osquery's key-value backing storage system, abstracting RocksDB (and SQLite) behind a domain/key API. +Header defining osquery's persistent backing storage abstraction layer, exposing the `DatabasePlugin` interface and free-function wrappers for key/value database operations across named domains. ## Key Components -### Domain Constants +**Domain Constants** | Constant | Purpose | |---|---| -| `kPersistentSettings` | Node UUID, enrollment keys, persisted config | +| `kPersistentSettings` | Node UUID, enrollment keys, cross-run settings | | `kQueries` | Scheduled query results | | `kEvents` | Queued event results | | `kCarves` | File carve query results | @@ -14,34 +14,26 @@ Defines the `DatabasePlugin` abstract interface and supporting free functions fo | `kDistributedQueries` | Distributed query storage | | `kQueryPerformance` | Query performance statistics | -### `DatabasePlugin` (abstract class) -Extends `Plugin`; all backing store implementations must override: -- `get()` β€” retrieve a string or int value by domain + key -- `put()` β€” store a string or int value -- `putBatch()` β€” write multiple key/value pairs atomically -- `remove()` / `removeRange()` β€” delete single or ranged keys -- `scan()` β€” list keys with optional prefix filter -- `call()` β€” registry/extension routing entrypoint - -### Free Functions -- `getDatabaseValue` / `setDatabaseValue` / `setDatabaseBatch` β€” primary read/write API for all osquery components -- `deleteDatabaseValue` / `deleteDatabaseRange` β€” key removal -- `scanDatabaseKeys` β€” enumerate keys in a domain -- `initDatabasePlugin` / `initDatabasePluginForTesting` β€” lifecycle setup -- `shutdownDatabase` / `resetDatabase` β€” teardown and reset -- `upgradeDatabase` β€” migrates legacy ptree JSON to RapidJSON format -- `getOsqueryDatabase` β€” returns the active `IDatabaseInterface` +**`DatabasePlugin`** β€” Abstract base class (extends `Plugin`) implementing the backing store interface. Requires subclasses to implement `get`, `put`, `putBatch`, `remove`, `removeRange`. Provides `call()` for registry/extension routing and `reset()`/`checkDB()` lifecycle helpers. + +**Free Functions** +- `getDatabaseValue` / `setDatabaseValue` / `setDatabaseBatch` β€” Primary read/write wrappers +- `deleteDatabaseValue` / `deleteDatabaseRange` β€” Key removal +- `scanDatabaseKeys` β€” Key enumeration with optional prefix filter +- `initDatabasePlugin` / `shutdownDatabase` / `resetDatabase` β€” Lifecycle management +- `upgradeDatabase` β€” Migrates legacy ptree JSON to RapidJSON format (schema versioned via `kDbCurrentVersion = 2`) +- `getOsqueryDatabase` β€” Returns the registry-routed `IDatabaseInterface` ## Usage Example ```c #include -// Write a persistent value +// Store a persistent value osquery::Status s = osquery::setDatabaseValue( osquery::kPersistentSettings, "node_key", "abc-123"); -// Read it back +// Retrieve it later std::string node_key; s = osquery::getDatabaseValue( osquery::kPersistentSettings, "node_key", node_key); @@ -51,8 +43,6 @@ std::vector keys; osquery::scanDatabaseKeys(osquery::kQueries, keys); // Batch write -osquery::DatabaseStringValueList batch = { - {"key1", "value1"}, - {"key2", "value2"}}; +osquery::DatabaseStringValueList batch = {{"k1","v1"}, {"k2","v2"}}; osquery::setDatabaseBatch(osquery::kLogs, batch); ``` \ No newline at end of file diff --git a/osquery/database/tests/.test_utils.md b/osquery/database/tests/.test_utils.md index f27cb6fbe72..76244676459 100644 --- a/osquery/database/tests/.test_utils.md +++ b/osquery/database/tests/.test_utils.md @@ -1,41 +1,43 @@ -Provides a reusable Google Test base class and macro for testing osquery database plugin implementations. +Provides a reusable base test fixture and macro for writing standardized Google Test suites that validate `osquery` database plugin implementations. ## Key Components ### Macro -- **`CREATE_DATABASE_TESTS(n)`** β€” Expands into a full suite of `TEST_F` cases for a given fixture class `n`, covering all standard database operations. Use this in a `.cpp` file to instantiate tests for a specific plugin. +- **`CREATE_DATABASE_TESTS(n)`** β€” Expands into a full suite of `TEST_F` declarations for a given fixture class `n`, covering all standard database operations. Eliminates boilerplate when testing multiple plugin backends. ### Class: `DatabasePluginTests` -A `testing::Test` subclass within the `osquery` namespace that serves as the base fixture for all database plugin test suites. +Abstract Google Test fixture (`testing::Test`) within the `osquery` namespace. Subclasses must implement `name()` to identify the plugin under test. | Member | Type | Description | |---|---|---| | `path_` | `std::string` | Path to the temporary test database | -| `previous_path_` | `std::string` | Database path saved before `SetUp` runs | -| `name()` | pure virtual | Must return the plugin name under test | -| `SetUp()` / `TearDown()` | lifecycle | Initializes and cleans up the plugin under test | +| `previous_path_` | `std::string` | Database path saved before `SetUp()` | +| `SetUp()` / `TearDown()` | overrides | Initializes and cleans up the plugin under test | +| `name()` | pure virtual | Returns the plugin name for the subclass | -**Protected test methods** (called by the macro-generated test cases): +**Protected test methods:** -`testPluginCheck` Β· `testReset` Β· `testPut` Β· `testPutBatch` Β· `testGet` Β· `testDelete` Β· `testDeleteRange` Β· `testScan` Β· `testScanLimit` +- `testPluginCheck()` β€” Validates plugin registration +- `testReset()` β€” Tests database reset behavior +- `testPut()` / `testPutBatch()` β€” Single and batch write operations +- `testGet()` β€” Key lookup +- `testDelete()` / `testDeleteRange()` β€” Single and range deletion +- `testScan()` / `testScanLimit()` β€” Full and limited key scanning ## Usage Example ```cpp -#include "test_utils.h" - -class RocksDBDatabaseTests : public osquery::DatabasePluginTests { +// Define a fixture for your plugin backend +class RocksDBPluginTests : public DatabasePluginTests { protected: std::string name() override { return "rocksdb"; } }; -// Expands all standard TEST_F cases for RocksDBDatabaseTests -CREATE_DATABASE_TESTS(RocksDBDatabaseTests); -``` - -Adding a new database plugin test suite requires only subclassing `DatabasePluginTests`, implementing `name()`, and invoking `CREATE_DATABASE_TESTS` β€” no boilerplate test bodies needed. \ No newline at end of file +// Expand all standard test cases for this fixture in one line +CREATE_DATABASE_TESTS(RocksDBPluginTests); +``` \ No newline at end of file diff --git a/osquery/dispatcher/.dispatcher.md b/osquery/dispatcher/.dispatcher.md index 6df777c8bb4..057057754bc 100644 --- a/osquery/dispatcher/.dispatcher.md +++ b/osquery/dispatcher/.dispatcher.md @@ -1,74 +1,59 @@ -Manages thread lifecycle and parallel task execution for osquery services, providing a singleton dispatcher that coordinates interruptible background threads. +Manages concurrent thread execution for osquery services, providing a singleton dispatcher that coordinates parallel asynchronous tasks via interruptible runnable abstractions. ## Key Components ### `InterruptibleRunnable` -Base class for any runnable that supports clean interruption. - -| Member | Description | -|---|---| -| `interrupt()` | Signals the runnable to stop (final, non-overridable) | -| `interrupted()` | Checks if an interrupt has been requested | -| `pause(milliseconds)` | Interruptible sleep using a condition variable | -| `stop()` | Pure virtual β€” subclasses define their shutdown logic | -| `runnable_name_` | Thread display name | +Base class for any task that can be paused or stopped mid-execution. +- `interrupt()` β€” signals the runnable to stop +- `interrupted()` β€” checks current interruption state +- `pause(milliseconds)` β€” interruptible sleep using a condition variable +- `stop()` β€” pure virtual; subclasses define their shutdown logic ### `InternalRunnable` -Extends `InterruptibleRunnable` with a thread entrypoint. Non-copyable. - -| Member | Description | -|---|---| -| `run()` | Thread entrypoint called by `Dispatcher` (final) | -| `start()` | Pure virtual β€” subclasses define their work loop | -| `hasRun()` | Returns whether the thread context has started | - -### `Dispatcher` (Singleton) -Manages all `InternalRunnable` service threads. - -| Method | Description | -|---|---| -| `instance()` | Returns the singleton instance | -| `addService(ref)` | Spawns a new service thread (unlimited pool) | -| `joinServices()` | Blocks until all services complete | -| `stopServices()` | Interrupts and destroys all running services | -| `serviceCount()` | Returns the number of active services | +Extends `InterruptibleRunnable` (non-copyable) as the concrete thread entry point used by the `Dispatcher`. +- `run()` β€” thread entrypoint called by the Dispatcher +- `start()` β€” pure virtual; subclasses define their work loop +- `hasRun()` β€” checks whether the thread has started execution ### Type Aliases +- `InternalRunnableRef` β€” `std::shared_ptr` +- `InternalThreadRef` β€” `std::unique_ptr` -```c -using InternalRunnableRef = std::shared_ptr; -using InternalThreadRef = std::unique_ptr; -``` +### `Dispatcher` (Singleton) +Coordinates lifecycle of all osquery background service threads. +- `instance()` β€” returns the singleton instance +- `addService(service)` β€” launches a service on a new thread +- `joinServices()` β€” blocks until all service threads complete +- `stopServices()` β€” signals and tears down all running services +- `serviceCount()` β€” returns the number of active services ## Usage Example -```c +```cpp // Define a custom background service -class MyService : public InternalRunnable { +class MyService : public osquery::InternalRunnable { public: MyService() : InternalRunnable("MyService") {} protected: void start() override { while (!interrupted()) { - // do work - pause(std::chrono::milliseconds(500)); + // Do periodic work + pause(std::chrono::milliseconds(1000)); } } void stop() override { - // cleanup + // Cleanup on shutdown } }; // Register and launch the service -auto svc = std::make_shared(); -osquery::Dispatcher::addService(svc); +auto service = std::make_shared(); +osquery::Dispatcher::addService(service); -// Later, cleanly shut everything down +// Gracefully stop all services on shutdown osquery::Dispatcher::stopServices(); osquery::Dispatcher::joinServices(); -``` - -> **Note:** `Dispatcher` is a singleton β€” never construct it directly. Use `Dispatcher::instance()`. Services are not subject to a fixed thread pool size, unlike the underlying Thrift pool. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/dispatcher/.distributed_runner.md b/osquery/dispatcher/.distributed_runner.md index c52d1a98aff..050c9b0285a 100644 --- a/osquery/dispatcher/.distributed_runner.md +++ b/osquery/dispatcher/.distributed_runner.md @@ -1,32 +1,34 @@ -Declares the `DistributedRunner` service thread responsible for executing distributed queries within the osquery dispatcher framework. +Defines the `DistributedRunner` service thread responsible for executing distributed queries within the osquery dispatcher framework. ## Key Components -- **`DistributedRunner`** β€” A `InternalRunnable` subclass that runs as a long-lived Dispatcher service thread, implementing the distributed query polling/execution loop. - - `start()` β€” Entry point invoked by the Dispatcher when the thread is scheduled. -- **`startDistributed()`** β€” Factory/helper function to initialize and register the `DistributedRunner` with the osquery Dispatcher. +- **`DistributedRunner`** β€” A `InternalRunnable` subclass that runs as a persistent Dispatcher service thread, handling distributed query execution. + - `start()` β€” Thread entry point override; contains the main distributed query processing loop. -## Usage Example +- **`startDistributed()`** β€” Free function that initializes and registers the `DistributedRunner` with the osquery Dispatcher. -```cpp -#include +## Usage Example -// Start the distributed query service (typically called during osquery startup) -Status s = osquery::startDistributed(); +```c +// Start the distributed query runner service +Status s = startDistributed(); if (!s.ok()) { - LOG(ERROR) << "Failed to start distributed runner: " << s.getMessage(); + LOG(ERROR) << "Failed to start distributed runner: " << s.getMessage(); } -``` -Internally, the runner is registered via the Dispatcher: - -```cpp -// Inside startDistributed() β€” conceptual implementation -Status startDistributed() { - Dispatcher::addService(std::make_shared()); - return Status::success(); -} +// The runner is managed by the Dispatcher; to integrate a custom runner: +class MyDistributedRunner : public DistributedRunner { + public: + void start() override { + // Custom distributed query handling logic + DistributedRunner::start(); + } +}; ``` -> **Note:** `DistributedRunner` is intended to be managed exclusively by the osquery `Dispatcher`. Do not instantiate or call `start()` directly β€” use `startDistributed()` instead to ensure proper lifecycle management. \ No newline at end of file +## Notes + +- Inherits lifecycle management (start/stop/join) from `InternalRunnable` via the Dispatcher subsystem. +- Registered under the name `"DistributedRunner"` for identification within the Dispatcher thread registry. +- Intended to be launched once via `startDistributed()` rather than instantiated directly. \ No newline at end of file diff --git a/osquery/dispatcher/.scheduler.md b/osquery/dispatcher/.scheduler.md index 01692c14232..58a16c32e84 100644 --- a/osquery/dispatcher/.scheduler.md +++ b/osquery/dispatcher/.scheduler.md @@ -1,54 +1,45 @@ -Defines the `SchedulerRunner` class and supporting utilities for executing scheduled osquery SQL queries at configurable intervals within the osquery dispatcher framework. +Defines the `SchedulerRunner` class and related utilities for executing scheduled osquery SQL queries at configurable intervals within the dispatcher framework. ## Key Components ### `SchedulerRunner` (class) -A `InternalRunnable` dispatcher service thread that drives osquery's schedule execution loop. +A `InternalRunnable`-derived dispatcher service thread that drives the osquery schedule loop. | Member | Description | -|---|---| -| `SchedulerRunner(timeout, interval, max_time_drift)` | Constructor; configures loop timeout, step interval, and optional drift cap | -| `start()` | Thread entry point; runs the scheduling loop | -| `stop()` | Interrupt/stop hook (no-op implementation) | -| `getCurrentTimeDrift()` | Returns accumulated time drift in milliseconds | - -### Private Helpers -| Method | Description | -|---|---| -| `calculateTimeDriftAndMaybePause()` | Tracks loop step duration and compensates for drift | -| `maybeRunDecorators()` | Fires interval-based query decorators | -| `maybeReloadSchedule()` | Checks and applies config schedule changes | -| `maybeFlushLogs()` | Conditionally flushes buffered status logs | -| `maybeScheduleCarves()` | Triggers file carve requests on schedule | +|--------|-------------| +| `SchedulerRunner(timeout, interval, max_time_drift)` | Constructs runner with step interval, max steps, and optional drift cap | +| `start()` | Thread entry point; runs the schedule loop | +| `stop()` | Interrupt/stop hook (no-op) | +| `getCurrentTimeDrift()` | Returns accumulated millisecond drift | +| `calculateTimeDriftAndMaybePause()` | Adjusts sleep to compensate for loop overrun | +| `maybeRunDecorators()` | Fires interval-based decorator queries | +| `maybeReloadSchedule()` | Checks for config schedule changes | +| `maybeFlushLogs()` | Flushes buffered status logs when due | +| `maybeScheduleCarves()` | Enqueues pending file carve requests | ### Free Functions + | Function | Description | -|---|---| -| `monitor(name, query)` | Executes a named `ScheduledQuery` and returns an `SQLInternal` result | +|----------|-------------| +| `monitor(name, query)` | Executes a named `ScheduledQuery` and returns a `SQLInternal` result | | `startScheduler()` | Starts the scheduler using config-defined settings | -| `startScheduler(timeout, interval)` | Starts the scheduler with explicit timeout and interval (used in tests) | +| `startScheduler(timeout, interval)` | Overload for testing with explicit timeout/interval | ## Usage Example ```cpp -#include "osquery/scheduler/scheduler.h" - -// Start the scheduler using the loaded osquery config schedule +// Production start β€” reads schedule from config osquery::startScheduler(); -// Or start with explicit parameters (e.g., in tests) -osquery::startScheduler( - /*timeout=*/3600UL, // run for 3600 steps - /*interval=*/60 // execute every 60 seconds -); - -// Manually monitor a specific scheduled query -osquery::ScheduledQuery sq; -sq.query = "SELECT * FROM processes;"; -sq.interval = 60; - -osquery::SQLInternal result = osquery::monitor("process_monitor", sq); -``` +// Test/controlled start β€” 60 steps, 1-second interval +osquery::startScheduler(60UL, 1); -> **Note:** `request_shutdown_on_expiration` defaults to `true` in production but is suppressed during unit tests via `FRIEND_TEST(TLSConfigTests, test_runner_and_scheduler)` to prevent premature process exit. \ No newline at end of file +// Construct runner directly with drift cap of 500ms +auto runner = std::make_shared( + 3600UL, // timeout (steps) + 60, // interval (seconds) + std::chrono::milliseconds(500) // max allowed drift +); +Dispatcher::addService(runner); +``` \ No newline at end of file diff --git a/osquery/distributed/.distributed.md b/osquery/distributed/.distributed.md index e9b7444e2a4..ae034c9449d 100644 --- a/osquery/distributed/.distributed.md +++ b/osquery/distributed/.distributed.md @@ -1,27 +1,24 @@ -Defines the interfaces and data structures for osquery's distributed query system, enabling remote query execution, result serialization, and denylist management across endpoints. +Defines the core types, serialization utilities, and classes for osquery's distributed query system, enabling remote query dispatch and result collection. ## Key Components ### Structs -- **`DistributedQueryRequest`** β€” Holds a query string and its unique ID for a pending distributed query -- **`DistributedQueryResult`** β€” Holds the request, result rows, column names, status, and message from a completed query execution +- **`DistributedQueryRequest`** β€” Holds a query string and its associated ID for remote execution +- **`DistributedQueryResult`** β€” Contains the request, result rows, column names, status, and message from a completed query ### Free Functions - -| Function | Description | -|---|---| -| `serializeDistributedQueryRequest[JSON]` | Serialize a request to JSON tree or string | -| `deserializeDistributedQueryRequest[JSON]` | Deserialize a request from JSON tree or string | -| `serializeDistributedQueryResult[JSON]` | Serialize a result to JSON tree or string | -| `deserializeDistributedQueryResult[JSON]` | Deserialize a result from JSON tree or string | -| `hashQuery` | Returns SHA-256 hex digest of a query string | -| `denylistDuration` | Returns configured denylist duration in seconds | -| `denylistedQueryTimestampExpired` | Checks if a denylisted query's timeout has elapsed | +- **`serializeDistributedQueryRequest`** / **`serializeDistributedQueryRequestJSON`** β€” Serialize a request to a rapidjson object or JSON string +- **`deserializeDistributedQueryRequest`** / **`deserializeDistributedQueryRequestJSON`** β€” Deserialize a request from a rapidjson object or JSON string +- **`serializeDistributedQueryResult`** / **`serializeDistributedQueryResultJSON`** β€” Serialize a result to a rapidjson object or JSON string +- **`deserializeDistributedQueryResult`** / **`deserializeDistributedQueryResultJSON`** β€” Deserialize a result from a rapidjson object or JSON string +- **`hashQuery`** β€” Returns the SHA-256 hex digest of a query string (used as a denylist key) +- **`denylistDuration`** β€” Returns the configured denylist duration in seconds +- **`denylistedQueryTimestampExpired`** β€” Checks whether a denylist timestamp has expired ### Classes -- **`DistributedPlugin`** β€” Abstract plugin base; implementors must provide `getQueries()` and `writeResults()` to integrate with a remote query backend -- **`Distributed`** β€” Orchestrates the full distributed query lifecycle: pulling, executing, tracking running state, recording performance, and flushing results +- **`DistributedPlugin`** β€” Abstract plugin base; implementors provide `getQueries()` and `writeResults()` to integrate a remote query backend +- **`Distributed`** β€” Orchestrates the full distributed query lifecycle: pulling work, executing queries, tracking running state, recording performance, and flushing results ## Usage Example @@ -30,22 +27,19 @@ Defines the interfaces and data structures for osquery's distributed query syste auto dist = osquery::Distributed(); while (true) { - // Pull pending queries from remote server - dist.pullUpdates(); - - if (dist.getPendingQueryCount() > 0) { - // Execute all pending queries and queue results - dist.runQueries(); - } + dist.pullUpdates(); - // Serialize and inspect buffered results - std::string json; - dist.serializeResults(json); + if (dist.getPendingQueryCount() > 0) { + dist.runQueries(); + } } -``` -## Notes +// Serialize results before flushing +std::string json; +dist.serializeResults(json); -- Query deduplication is enforced via SHA-256 hashing (`hashQuery`) combined with denylist tracking (`checkAndSetAsRunning`) to prevent re-execution within the configured denylist window -- Discovery queries gate execution: if a discovery query returns no rows, its associated distributed query is skipped -- Performance statistics per query are recorded in `performance_` and reported alongside results \ No newline at end of file +// Serialize a single result to JSON +osquery::DistributedQueryResult result; +std::string resultJson; +osquery::serializeDistributedQueryResultJSON(result, resultJson); +``` \ No newline at end of file diff --git a/osquery/events/.eventer.md b/osquery/events/.eventer.md index 739769904da..8647b7d3acc 100644 --- a/osquery/events/.eventer.md +++ b/osquery/events/.eventer.md @@ -1,40 +1,41 @@ -Defines the base `Eventer` class and `EventState` enum used to track the lifecycle state of event publishers and subscribers within osquery's eventing system. +Defines the base `Eventer` class and `EventState` enum used to track the lifecycle state of osquery EventSubscribers and Publishers during initialization and runtime. ## Key Components ### `EventState` (enum class) -Represents the possible lifecycle states of an event publisher or subscriber: +Represents the possible lifecycle states of a subscriber or publisher: -| Value | Description | +| Value | Meaning | |---|---| -| `EVENT_NONE` | Default uninitialized state | -| `EVENT_SETUP` | Subscriber attached and setup complete | -| `EVENT_RUNNING` | Ready to receive events | +| `EVENT_NONE` | Default, uninitialized state | +| `EVENT_SETUP` | Attached and setup has run | +| `EVENT_RUNNING` | Ready to accept events | | `EVENT_PAUSED` | Initialized but not accepting events | -| `EVENT_FAILED` | Initialization failed or offline | +| `EVENT_FAILED` | Failed to initialize or offline | ### `Eventer` (class) -Base class providing state management for event publishers and subscribers. +Base class providing state management for event subscribers and publishers. | Member | Access | Description | |---|---|---| -| `state() const` | `public` | Returns current `EventState` | +| `state()` | `public` | Returns current `EventState` | | `state(EventState)` | `protected` | Sets the current state | | `state_` | `private` | Internal state storage, defaults to `EVENT_NONE` | - -`EventFactory` is declared a `friend` class, granting it direct access to internal state. +| `EventFactory` | `friend` | Granted direct access for lifecycle management | ## Usage Example ```cpp -#include +#include +// Typically subclassed by an EventSubscriber or EventPublisher class MySubscriber : public Eventer { public: void initialize() { // Transition to running after setup state(EventState::EVENT_SETUP); + if (setupSucceeded()) { state(EventState::EVENT_RUNNING); } else { @@ -44,8 +45,10 @@ class MySubscriber : public Eventer { void checkStatus() { if (state() == EventState::EVENT_RUNNING) { - // Accept and process events + // Accept and process incoming events } } }; -``` \ No newline at end of file +``` + +> `EventFactory` is the only external class with direct access to `Eventer`'s private members, maintaining controlled state transitions across the event subsystem. \ No newline at end of file diff --git a/osquery/events/.eventfactory.md b/osquery/events/.eventfactory.md index c83bdcca4f3..44709b00fe9 100644 --- a/osquery/events/.eventfactory.md +++ b/osquery/events/.eventfactory.md @@ -1,47 +1,67 @@ -Singleton factory class for managing osquery event publishers, subscribers, and subscriptions throughout the application lifecycle. +Singleton factory that manages the full lifecycle of event publishers and subscribers in osquery's eventing system β€” handling registration, subscription wiring, run loop dispatch, and teardown. ## Key Components -| Member | Type | Description | -|--------|------|-------------| -| `getInstance()` | Static method | Returns the singleton `EventFactory` instance | -| `registerEventPublisher()` | Static method | Registers an `EventPublisher` plugin with the factory | -| `registerEventSubscriber()` | Template method | Registers an `EventSubscriber` by type or instance | -| `addSubscription()` | Static method | Binds a `SubscriptionContext` and `EventCallback` to a publisher | -| `deregisterEventPublisher()` | Static method | Halts a publisher's run loop and removes it from the factory | -| `getEventPublisher()` | Static method | Retrieves a registered publisher by name | -| `getEventSubscriber()` | Static method | Retrieves a registered subscriber by name | -| `fire()` | Template method | Fires an `EventContext` into a publisher's subscriber callbacks | -| `getType()` | Template method | Resolves the registry plugin name for a publisher type | -| `delay()` | Static method | Spawns all registered publisher run loop threads | -| `end()` | Static method | Terminates all publisher run loops and optionally joins threads | -| `configUpdate()` | Static method | Notifies publishers/subscribers of configuration changes | +### `EventFactory` (Singleton Class) +The central coordinator for all osquery event infrastructure. Accessed via `EventFactory::getInstance()`. + +**Publisher Management** +- `registerEventPublisher(pub)` β€” registers an `EventPublisher` plugin instance +- `deregisterEventPublisher(pub/type_id)` β€” halts the run loop and cleans up +- `getEventPublisher(pub)` β€” retrieves a registered publisher by name +- `numEventPublishers()` β€” returns count of registered publishers +- `publisherTypes()` β€” returns all registered publisher registry names + +**Subscriber Management** +- `registerEventSubscriber()` β€” template registration; creates and registers a typed subscriber +- `registerEventSubscriber(sub)` β€” registration by plugin instance +- `deregisterEventSubscriber(sub)` β€” removes a subscriber by name +- `getEventSubscriber(sub)` β€” retrieves a registered subscriber +- `subscriberNames()` β€” returns all subscriber registry names +- `exists(sub)` β€” checks if a subscriber is registered + +**Subscription Wiring** +- `addSubscription(type_id, name_id, sc, cb)` β€” creates and attaches a `Subscription` to a publisher +- `addSubscription(type_id, subscription)` β€” attaches a pre-built `Subscription` instance +- `numSubscriptions(type_id)` β€” counts subscriptions for a given publisher + +**Run Loop Control** +- `delay()` β€” spawns all publisher run loop threads +- `run(type_id)` β€” entry point for a single publisher's dispatch thread +- `end(join)` β€” shuts down all publishers; optionally joins threads +- `fire(ec)` β€” fires an `EventContext` into a publisher from a static callback + +**Utilities** +- `getType()` β€” returns the registry name for a publisher type +- `configUpdate()` β€” notifies the factory of configuration changes +- `addForwarder(logger)` / `forwardEvent(event)` β€” routes events to logger plugins ## Usage Example ```cpp -// Register a publisher plugin +// Register a publisher and subscriber EventFactory::registerEventPublisher(pub_plugin_ref); - -// Register a subscriber by type EventFactory::registerEventSubscriber(); -// Add a subscription with context and callback +// Wire a subscription with a callback auto sc = std::make_shared(); -EventFactory::addSubscription("my_publisher", "my_subscriber", sc, - [](const EventContextRef& ec, const SubscriptionContextRef& sc) { - // Handle event - return Status::success(); - }); - -// Fire an event from a static publisher callback -EventContextRef ec = std::make_shared(); -EventFactory::fire(ec); +EventFactory::addSubscription( + "my_publisher", + "my_subscriber", + sc, + [](const EventContextRef& ec, const SubscriptionContextRef& sc) { + // handle event + return Status::success(); + } +); // Start all publisher run loops EventFactory::delay(); -// Shutdown all publishers (with thread join) -EventFactory::end(true); +// Fire an event from a static publisher callback +EventFactory::fire(event_context); + +// Shutdown +EventFactory::end(true /* join threads */); ``` \ No newline at end of file diff --git a/osquery/events/.eventpublisherplugin.md b/osquery/events/.eventpublisherplugin.md index 9903fac46e1..3777ff0336b 100644 --- a/osquery/events/.eventpublisherplugin.md +++ b/osquery/events/.eventpublisherplugin.md @@ -1,37 +1,31 @@ -Defines the `EventPublisherPlugin` base class for osquery's event-driven architecture, providing the lifecycle interface that all event publishers must implement to register OS-level callbacks, manage subscriptions, and fire events to subscribers. +Plugin interface for osquery event publishers, defining the lifecycle and subscription management API for event-driven data collection. ## Key Components -### Class: `EventPublisherPlugin` -Inherits from `Plugin`, `InterruptibleRunnable`, and `Eventer`. Serves as the abstract base for all event publisher implementations. +### `EventPublisherPlugin` Class +Inherits from `Plugin`, `InterruptibleRunnable`, and `Eventer`. Provides the base class for all osquery event publishers. **Lifecycle Methods** -| Method | Description | -|--------|-------------| -| `setUp()` | Initialize handles and register OS API callbacks (called before run loop) | -| `configure()` | Re-optimize subscriptions (e.g., dedup inotify paths) when subscriptions change | -| `tearDown()` | Release handles and clean up resources before exit | -| `run()` | Single iteration of the publisher's run loop; return `FAILED` to exit | -| `stop()` | Signal the run loop to terminate (called by `EventFactory`) | +- `setUp()` β€” Called before the run loop starts; opens handles and registers OS API callbacks +- `configure()` β€” Called when subscriptions change; allows optimization (e.g., deduplicating `inotify` paths) +- `run()` β€” Single step of the publisher's run loop; returning `FAILED` exits the loop +- `tearDown()` β€” Cleans up handles and resources on shutdown; may be called multiple times +- `stop()` β€” Signals the run loop to terminate via `isEnding` **Subscription Management** -| Method | Description | -|--------|-------------| -| `addSubscription(sub)` | Register a new `SubscriptionRef` with this publisher | -| `removeSubscriptions(subscriber)` | Remove all subscriptions for a named subscriber | -| `numSubscriptions()` | Returns active subscription count | +- `addSubscription(subscription)` β€” Registers a new `SubscriptionRef` with this publisher +- `removeSubscriptions(subscriber)` β€” Removes all subscriptions for a named subscriber +- `numSubscriptions()` β€” Returns the active subscription count **State Inspection** -- `isEnding()` / `hasStarted()` β€” get/set publisher lifecycle state -- `numEvents()` β€” total fired event count -- `restartCount()` β€” run loop restart iterations +- `isEnding()` / `hasStarted()` / `restartCount()` β€” Query run loop lifecycle state +- `numEvents()` β€” Returns total `EventContextID` count fired -### Protected Members -- `fire(ec, time)` β€” dispatches an `EventContext` to all matching subscribers -- `fireCallback(sub, ec)` β€” pure virtual; typed publishers implement actual dispatch -- `subscriptions_` β€” the internal `SubscriptionVector` -- `next_ec_id_` β€” atomic event context ID counter +**Protected Internals** +- `fire(ec, time)` β€” Enumerates subscriptions and dispatches event callbacks +- `fireCallback(sub, ec)` β€” Pure virtual; implemented by typed publishers to invoke subscriber callbacks +- `subscriptions_` / `subscription_lock_` β€” Thread-safe subscription storage ## Usage Example @@ -39,24 +33,23 @@ Inherits from `Plugin`, `InterruptibleRunnable`, and `Eventer`. Serves as the ab class MyPublisher : public EventPublisherPlugin { public: Status setUp() override { - // Open OS handle or register callback + // Register OS-level event source return Status::success(); } Status run() override { auto ec = createEventContext(); - fire(ec); // dispatches to all subscribers + // Populate ec with event data + fire(ec); return Status::success(); } - void tearDown() override { - // Close handles - } - protected: void fireCallback(const SubscriptionRef& sub, const EventContextRef& ec) const override { - // Invoke subscriber's typed callback + // Invoke subscriber-specific callback } }; -``` \ No newline at end of file +``` + +> **Note:** `interrupted()` is deprecated β€” use `isEnding()` to check publisher termination state. Copy construction and assignment are explicitly deleted. \ No newline at end of file diff --git a/osquery/events/.events.md b/osquery/events/.events.md index 185e2df9624..9b2e23aaf1f 100644 --- a/osquery/events/.events.md +++ b/osquery/events/.events.md @@ -1,26 +1,29 @@ -Utility header declaring event system lifecycle functions within the `osquery` namespace for managing event publisher run loops and query denylist enforcement. +Provides event system utilities for managing publisher lifecycles and query access control within the osquery event framework. ## Key Components -| Function | Description | -|---|---| -| `attachEvents()` | Iterates the event publisher registry and creates run loops for each publisher via the event factory | -| `enforceEventsDenylist(query)` | Returns `true` if the given query operates exclusively on event-based tables and should be denylisted | +- **`attachEvents()`** β€” Iterates the event publisher registry and creates run loops for each publisher using the event factory +- **`enforceEventsDenylist(const std::string& query)`** β€” Evaluates whether a query exclusively targets event-based tables and should be denied/blocked ## Usage Example ```c #include "events.h" -// Initialize all registered event publishers +// Initialize all registered event publishers at startup osquery::attachEvents(); -// Check if a query should be denied based on event-table usage -std::string query = "SELECT * FROM process_events"; -if (osquery::enforceEventsDenylist(query)) { - // Query is denylist-eligible; skip or block execution +// Before executing a query, check if it should be denied +std::string incomingQuery = "SELECT * FROM process_events"; +if (osquery::enforceEventsDenylist(incomingQuery)) { + // Query operates only on event-based tables β€” block execution + return; } ``` -> **Note:** `attachEvents()` is typically called once during daemon startup to wire up all registered publishers. `enforceEventsDenylist` is used during query scheduling to prevent high-frequency polling on purely event-driven tables. \ No newline at end of file +## Notes + +- Both functions live in the `osquery` namespace +- `attachEvents()` is typically called during daemon initialization to start all event-driven subscribers +- `enforceEventsDenylist()` acts as a guard for queries that would only consume event-table data, preventing potential abuse or excessive resource usage \ No newline at end of file diff --git a/osquery/events/.eventsubscriberplugin.md b/osquery/events/.eventsubscriberplugin.md index 9338289cef4..b904f6a747a 100644 --- a/osquery/events/.eventsubscriberplugin.md +++ b/osquery/events/.eventsubscriberplugin.md @@ -1,29 +1,29 @@ -Defines the `EventSubscriberPlugin` class, the base interface for osquery event subscribers that receive, store, and expose event data as queryable table rows via a backing database store. +Defines the `EventSubscriberPlugin` class, the base interface for osquery event subscribers that receive, store, and expose event data as queryable table rows via a backing store. ## Key Components ### Class: `EventSubscriberPlugin` -Inherits from `Plugin` and `Eventer`. Manages event ingestion, indexed storage, expiration, and query-time row generation. +Inherits from `Plugin` and `Eventer`. Manages the full lifecycle of event subscription, storage, indexing, expiration, and table generation. -### Core Methods +### Key Methods | Method | Description | -|---|---| -| `init()` | Override to register `Subscription`s with an `EventPublisher` | -| `call(PluginRequest, PluginResponse)` | Plugin dispatch entrypoint | -| `addBatch(row_list)` | Store a batch of parsed event rows to the backing store (preferred API) | -| `add(row)` | *(Deprecated)* Store a single event row β€” use `addBatch()` instead | -| `genTable(yield, ctx)` | Table generation entrypoint; retrieves stored events filtered by query time window | -| `generateRows(...)` | Static: enumerate stored events within a `[start_time, end_time]` range | -| `expireEventBatches(...)` | Static: prune old event batches from backing store | -| `removeOverflowingEventBatches(...)` | Static: enforce maximum batch count | -| `getOptimizeData(...)` / `setOptimizeData(...)` | Static: manage query optimization checkpoints | - -### Supporting Structures - -- **`Context`** β€” Holds the database namespace, event index, last query/event timestamps -- **`GenerateRowsResult`** β€” Returns pagination state (`isEnd`, `last_time`, `last_id`) from `generateRows` +|--------|-------------| +| `init()` | Entry point for registering subscriptions with the EventPublisher | +| `addBatch(row_list)` | Stores a batch of parsed event rows in the backing store (preferred API) | +| `add(r)` | *(Deprecated)* Stores a single event row; use `addBatch()` instead | +| `genTable(yield, ctx)` | Main entrypoint for table generation at query time | +| `generateRows(...)` | Retrieves stored events filtered by time window | +| `getEventsExpiry()` | Returns the expiration timeout for this event type | +| `getEventBatchesMax()` | Returns the maximum number of stored event batches | +| `getOptimizeData(...)` | Reads optimization state (last event time/ID) from the database | +| `setOptimizeData(...)` | Writes optimization state to the database | + +### Supporting Types + +- **`Context`** β€” Holds per-subscriber state: database namespace, event index, last query time, and last event ID +- **`GenerateRowsResult`** β€” Return type for `generateRows`, including end-of-range flag and last visited time/ID ## Usage Example @@ -31,19 +31,22 @@ Inherits from `Plugin` and `Eventer`. Manages event ingestion, indexed storage, class MySubscriber : public EventSubscriber { public: Status init() override { - auto sub = createSubscription(); - return EventFactory::addSubscription(getType(), sub); + // Create and add a subscription + auto sc = MyPublisher::createSubscriptionContext(); + sc->path = "/etc/passwd"; + subscribe(&MySubscriber::callback, sc); + return Status::success(); } - Status Callback(const EventContextRef& ec, const SubscriptionContextRef& sc) { + Status callback(const ECRef& ec, const SCRef& sc) { Row r; + r["path"] = ec->path; r["action"] = ec->action; - r["time"] = INTEGER(ec->time); - - std::vector batch = {r}; - return addBatch(batch); // preferred over deprecated add() + // Batch rows and store them + std::vector rows = {r}; + return addBatch(rows); } }; ``` -> **Note:** `add()` is deprecated. Always use `addBatch()` to group events together for efficient indexed storage and reduced write overhead. \ No newline at end of file +> **Note:** Always use `addBatch()` over the deprecated `add()`. Subscriptions must be registered in `init()`, not the constructor, to ensure the EventPublisher is fully initialized. \ No newline at end of file diff --git a/osquery/events/.subscription.md b/osquery/events/.subscription.md index b7de73f51af..861d2126fc6 100644 --- a/osquery/events/.subscription.md +++ b/osquery/events/.subscription.md @@ -1,40 +1,38 @@ -Defines the `Subscription` struct used to configure an `EventPublisher` and bind event callbacks to a `SubscriptionContext` within osquery's event framework. +Defines the `Subscription` struct and related type aliases used to configure event publishers and bind callbacks within osquery's event framework. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `EventCallback` | `using` alias | Function signature `Status(const EventContextRef&, const SubscriptionContextRef&)` invoked when a matching event fires | -| `SubscriptionRef` | `using` alias | `shared_ptr` for shared ownership of a subscription | -| `SubscriptionVector` | `using` alias | `vector` used by publishers to track all active subscriptions | -| `Subscription` | `struct` | Non-copyable aggregate holding a subscriber name, context, and callback | -| `Subscription::create()` | static factory | Two overloads β€” name-only, or name + context + optional callback | +- **`EventCallback`** β€” Type alias for `std::function`; the callback signature fired when a matching event occurs. +- **`SubscriptionRef`** β€” `std::shared_ptr` for safe shared ownership. +- **`SubscriptionVector`** β€” `std::vector` used by publishers to track all active subscriptions. +- **`Subscription`** (struct) β€” Core non-copyable struct holding: + - `subscriber_name` β€” Name of the owning `EventSubscriber`. + - `context` β€” Publisher-specific `SubscriptionContextRef` defining event scope/filters. + - `callback` β€” The `EventCallback` to invoke on matching events. + - `create()` (static, x2) β€” Factory methods to construct a `SubscriptionRef`, with or without a context and callback. ## Usage Example ```cpp #include -// Define a callback matching the EventCallback signature -Status onFileEvent(const EventContextRef& ec, - const SubscriptionContextRef& sc) { - // Handle the event +// Define a callback +EventCallback my_callback = [](const EventContextRef& ec, + const SubscriptionContextRef& sc) -> Status { + // Handle event return Status::success(); -} +}; -// Create a subscription with context and callback +// Create a subscription context (publisher-specific) auto ctx = std::make_shared(); ctx->path = "/etc/passwd"; -SubscriptionRef sub = Subscription::create( - "my_subscriber", // EventSubscriber name - ctx, // Publisher-specific context - onFileEvent // Callback fired on matching events -); +// Create the subscription +auto sub = Subscription::create("my_subscriber", ctx, my_callback); -// Register via EventFactory +// Register with the event publisher via EventFactory EventFactory::addSubscription("MyEventPublisher", sub); ``` -> **Note:** `Subscription` inherits `boost::noncopyable` β€” always pass instances via `SubscriptionRef` (`shared_ptr`). The default constructor is explicitly deleted; use the static `create()` factory methods. \ No newline at end of file +> **Note:** `Subscription` is non-copyable (`boost::noncopyable`). Always use `SubscriptionRef` (shared pointer) when passing or storing subscriptions. The default constructor is deleted β€” use the `create()` factory methods. \ No newline at end of file diff --git a/osquery/events/darwin/.diskarbitration.md b/osquery/events/darwin/.diskarbitration.md index fe2b809a169..8336295e461 100644 --- a/osquery/events/darwin/.diskarbitration.md +++ b/osquery/events/darwin/.diskarbitration.md @@ -1,50 +1,51 @@ -Provides the osquery event publisher interface for monitoring macOS disk mount/unmount events via the DiskArbitration framework. +Provides the macOS Disk Arbitration event publisher interface for osquery, enabling subscription to disk mount/unmount events on macOS systems. ## Key Components ### Macros - -| Macro | Purpose | -|---|---| -| `kIOPropertyProtocolCharacteristicsKey_` | IOKit key for protocol characteristics lookup | -| `kVirtualInterfaceLocation_` | Key identifying virtual disk interface paths | -| `kDAAppearanceTime_` | Key for disk appearance timestamp | +- `kIOPropertyProtocolCharacteristicsKey_` β€” IOKit key for protocol characteristics lookup +- `kVirtualInterfaceLocation_` β€” Key identifying virtual disk interface location paths +- `kDAAppearanceTime_` β€” Key for disk appearance timestamp ### Structs - - **`DiskArbitrationSubscriptionContext`** β€” Subscription filter; `physical_disks` flag limits events to physical vs. virtual (DMG) disks -- **`DiskArbitrationEventContext`** β€” Event payload carrying disk metadata: `action`, `path`, `device_path`, `uuid`, `size`, `filesystem`, `vendor`, `checksum`, and mount capability flags +- **`DiskArbitrationEventContext`** β€” Event payload containing disk metadata: `action`, `path`, `device_path`, `name`, `uuid`, `size`, `ejectable`, `mountable`, `writable`, `filesystem`, `vendor`, `checksum`, and more ### Type Aliases - - `DiskArbitrationEventContextRef` β€” `shared_ptr` - `DiskArbitrationSubscriptionContextRef` β€” `shared_ptr` ### Class: `DiskArbitrationEventPublisher` +Extends `EventPublisher`. -Extends `EventPublisher` to wrap macOS `DASession` callbacks into osquery events. - -| Member | Description | -|---|---| -| `run()` | Starts the `CFRunLoop` and registers DA session callbacks | -| `tearDown()` | Releases DA session and run loop resources | +| Method | Description | +|--------|-------------| +| `run()` | Starts the DA session and CFRunLoop | +| `tearDown()` | Cleans up session resources | | `shouldFire()` | Filters events against subscription context | -| `DiskAppearedCallback()` | Static DA callback for disk mount events | -| `DiskDisappearedCallback()` | Static DA callback for disk unmount events | -| `extractUdifChecksum()` | Reads checksum from UDIF (DMG) disk images | -| `fire()` | Builds and dispatches `DiskArbitrationEventContext` from a `CFDictionaryRef` | +| `DiskAppearedCallback()` | Static DA callback on disk mount | +| `DiskDisappearedCallback()` | Static DA callback on disk unmount | +| `extractUdifChecksum()` | Extracts checksum from DMG (UDIF) files | +| `getProperty()` | Reads a `CFStringRef` value from a `CFDictionaryRef` | + +**Private members:** `DASessionRef session_`, `CFRunLoopRef run_loop_`, `Mutex mutex_` ## Usage Example ```cpp -// Subscribe to physical disk events only +// Subscribe to disk arbitration events (all disks) auto sc = std::make_shared(); -sc->physical_disks = true; +sc->physical_disks = true; // restrict to physical disks only -auto sub = EventFactory::addSubscription( +EventFactory::addSubscription( "diskarbitration", - EventSubscriberPlugin::SubscriptionAction::ADD, - sc, - callback); -``` \ No newline at end of file + Subscription::create("my_subscriber", sc, + [](const DiskArbitrationEventContextRef& ec, + const DiskArbitrationSubscriptionContextRef&) { + LOG(INFO) << ec->action << " " << ec->path + << " (" << ec->filesystem << ")"; + })); +``` + +> **Platform note:** This publisher is macOS-only, depending on `DiskArbitration.framework`, `IOKit.framework`, and `CoreServices.framework`. \ No newline at end of file diff --git a/osquery/events/darwin/.endpointsecurity.md b/osquery/events/darwin/.endpointsecurity.md index 4e48c562096..787613e5224 100644 --- a/osquery/events/darwin/.endpointsecurity.md +++ b/osquery/events/darwin/.endpointsecurity.md @@ -1,26 +1,26 @@ -macOS Endpoint Security event publisher and subscriber definitions for osquery, providing process and file system monitoring via Apple's EndpointSecurity framework (requires macOS 10.15+). +macOS Endpoint Security event publisher and subscriber definitions for osquery, providing process and file system event monitoring via Apple's EndpointSecurity framework (requires macOS 10.15+). ## Key Components -### Context Structs +### Subscription & Event Context Structs | Struct | Purpose | |---|---| -| `EndpointSecuritySubscriptionContext` | Holds ES event type subscriptions and row buffer for process events | -| `EndpointSecurityEventContext` | Full process event payload β€” PID, credentials, code signing, exec args, fork/exit data | -| `EndpointSecurityFileSubscriptionContext` | Holds ES event type subscriptions and row buffer for file events | -| `EndpointSecurityFileEventContext` | Extends process context with `filename` and `dest_filename` for file operations | +| `EndpointSecuritySubscriptionContext` | Holds process event type subscriptions and accumulated rows | +| `EndpointSecurityEventContext` | Rich process event data: PID, credentials, codesigning, exec args, fork/exit info | +| `EndpointSecurityFileSubscriptionContext` | Holds file event type subscriptions and accumulated rows | +| `EndpointSecurityFileEventContext` | Extends process context with source/destination filename fields | ### Publishers -- **`EndpointSecurityPublisher`** β€” Registers as `"endpointsecurity"`, manages an `es_client_s*` handle, dispatches process-level ES events (`exec`, `fork`, `exit`) to subscribers -- **`EndpointSecurityFileEventPublisher`** β€” Registers as `"endpointsecurity_fim"`, manages a separate `es_file_client_s*` handle with configurable path muting/filtering for file integrity monitoring (FIM); ships with a default muted path list covering high-noise system daemons (e.g., `WindowServer`, `tccd`, `securityd`) +- **`EndpointSecurityPublisher`** β€” Manages an `es_client_s` connection, dispatches process-level ES events (`exec`, `fork`, `exit`) to subscribers. Publisher name: `"endpointsecurity"`. +- **`EndpointSecurityFileEventPublisher`** β€” Manages a separate `es_file_client_s` connection for file integrity monitoring (FIM). Supports path muting, prefix exclusions, and ships with a default muted list of high-noise Apple system processes. Publisher name: `"endpointsecurity_fim"`. ### Subscribers -- **`ESProcessEventSubscriber`** β€” Subscribes to `EndpointSecurityPublisher`, populates the `es_process_events` osquery table -- **`ESProcessFileEventSubscriber`** β€” Subscribes to `EndpointSecurityFileEventPublisher`, populates the `es_process_file_events` osquery table +- **`ESProcessEventSubscriber`** β€” Subscribes to `EndpointSecurityPublisher`; populates the `es_process_events` table. +- **`ESProcessFileEventSubscriber`** β€” Subscribes to `EndpointSecurityFileEventPublisher`; populates the `es_process_file_events` table. ## Usage Example @@ -40,10 +40,14 @@ class MyProcessSubscriber Status Callback(const EndpointSecurityEventContextRef& ec, const EndpointSecuritySubscriptionContextRef& sc) { - // Access ec->pid, ec->path, ec->args, ec->signing_id, etc. + Row r; + r["pid"] = INTEGER(ec->pid); + r["path"] = ec->path; + r["args"] = ec->args; + addRow(r); return Status::success(); } }; ``` -> **Platform note:** All publisher and subscriber methods are gated with `API_AVAILABLE(macos(10.15))`. This header has no effect on non-macOS builds. \ No newline at end of file +> **Note:** All publisher and subscriber methods are gated with `API_AVAILABLE(macos(10.15))`. This header is macOS-only and requires the `EndpointSecurity.framework` entitlement at runtime. \ No newline at end of file diff --git a/osquery/events/darwin/.es_utils.md b/osquery/events/darwin/.es_utils.md index ee9049cde42..a038335a34e 100644 --- a/osquery/events/darwin/.es_utils.md +++ b/osquery/events/darwin/.es_utils.md @@ -1,26 +1,32 @@ -Utility header for extracting and formatting process and string properties from macOS Endpoint Security (`es_*`) data structures within the osquery framework. +Utility header providing helper functions for extracting and formatting data from Apple's Endpoint Security framework (`es_*`) types within the osquery event pipeline. ## Key Components | Function | Description | |---|---| -| `getEsNewClientErrorMessage` | Converts an `es_new_client_result_t` error code into a human-readable string | -| `getPath` | Extracts the executable path from an `es_process_t` process struct | +| `getEsNewClientErrorMessage` | Converts `es_new_client_result_t` error codes to human-readable strings | +| `getPath` | Extracts the executable path from an `es_process_t` struct | | `getSigningId` | Retrieves the code-signing identifier from an `es_process_t` | | `getTeamId` | Retrieves the Apple Developer Team ID from an `es_process_t` | -| `getStringFromToken` | Converts a mutable or const `es_string_token_t` to a `std::string` | -| `getCwdPathFromPid` | Resolves the current working directory path for a given PID | -| `getCDHash` | Extracts the code-directory hash from an `es_process_t` | -| `getProcessProperties` | Populates an `EndpointSecurityEventContextRef` with properties from a process | -| `appendQuotedString` | Appends a delimiter-quoted string to an output stream | +| `getStringFromToken` | Converts `es_string_token_t` (mutable and const overloads) to `std::string` | +| `getCwdPathFromPid` | Resolves the current working directory for a given PID | +| `getCDHash` | Extracts the CDHash (code directory hash) from an `es_process_t` | +| `getProcessProperties` | Populates an `EndpointSecurityEventContextRef` with properties from an `es_process_t` | +| `appendQuotedString` | Writes a delimiter-quoted string to an output stream | ## Usage Example -```c +```cpp #include -// Inside an Endpoint Security event handler: +// Handling a new ES client creation failure +es_new_client_result_t result = es_new_client(&client, handler); +if (result != ES_NEW_CLIENT_RESULT_SUCCESS) { + LOG(ERROR) << getEsNewClientErrorMessage(result); +} + +// Extracting process metadata from an ES event void handleEvent(const es_message_t* msg) { const es_process_t* proc = msg->process; @@ -28,13 +34,16 @@ void handleEvent(const es_message_t* msg) { std::string signingId = osquery::getSigningId(proc); std::string teamId = osquery::getTeamId(proc); std::string cdHash = osquery::getCDHash(proc); - std::string cwd = osquery::getCwdPathFromPid(audit_token_to_pid(proc->audit_token)); + std::string cwd = osquery::getCwdPathFromPid( + audit_token_to_pid(proc->audit_token)); + // Populate event context osquery::getProcessProperties(proc, eventContext); - - std::ostringstream out; - osquery::appendQuotedString(out, path, '"'); } ``` -> **Note:** This header is macOS-only and depends on the Apple Endpoint Security framework (`es_process_t`, `es_string_token_t`). It is used internally by osquery's EndpointSecurity event publisher to normalize raw ES API types into standard C++ types before query processing. \ No newline at end of file +## Notes + +- macOS-only; depends on the `EndpointSecurity` framework +- All functions live in the `osquery` namespace +- `getStringFromToken` provides both `const` and non-`const` overloads for `es_string_token_t` \ No newline at end of file diff --git a/osquery/events/darwin/.event_taps.md b/osquery/events/darwin/.event_taps.md index f8b09845338..efecafbc510 100644 --- a/osquery/events/darwin/.event_taps.md +++ b/osquery/events/darwin/.event_taps.md @@ -1,46 +1,45 @@ -macOS event tap publisher that monitors and intercepts system-level input events (keyboard, mouse, etc.) using the CoreGraphics `CGEventTap` API, exposing them to osquery subscribers. +macOS event tap publisher that captures and dispatches system-level input events (keyboard, mouse, etc.) using the Core Graphics Event Services API. ## Key Components | Symbol | Type | Description | -|---|---|---| +|--------|------|-------------| | `EventTappingSubscriptionContext` | Struct | Empty subscription context for event tap subscriptions | -| `EventTappingEventContext` | Struct | Empty event context carrying tapped event data | -| `EventTappingEventPublisher` | Class | Core publisher managing the CGEventTap lifecycle and run loop | -| `EventTappingConsumerRunner` | Class (forward) | Dispatched service that processes published event tap events | +| `EventTappingEventContext` | Struct | Empty event context carrying dispatched tap events | +| `EventTappingEventPublisher` | Class | Publisher that installs a CGEvent tap and fires osquery events | +| `EventTappingConsumerRunner` | Class (fwd) | Dispatched service that processes published tap events | -### `EventTappingEventPublisher` Methods +## Key Methods | Method | Description | -|---|---| -| `setUp()` | Initializes the CGEventTap and run loop source | -| `tearDown()` | Cleans up tap resources | -| `stop()` | Stops the run loop | -| `run()` | Starts the publisher run loop | -| `restart()` | Reinitializes the event tap (e.g., after system re-enables it) | -| `eventCallback()` | Static CGEventTap callback receiving raw `CGEventRef` events | +|--------|-------------| +| `setUp()` | Installs the CGEvent tap and run loop source | +| `tearDown()` / `stop()` | Cleans up the tap, run loop source, and Mach port | +| `run()` | Enters the CFRunLoop to receive events | +| `restart()` | Re-enables the tap if it becomes disabled | +| `eventCallback()` | Static CGEvent tap callback; fires subscriber events | | `shouldFire()` | Filters events against subscription context | ## Usage Example ```cpp -// Subscribing to event tap events in an osquery EventSubscriber +// Subscribing to macOS event tap events class MyEventTapSubscriber : public EventSubscriber { public: Status init() override { auto sc = createSubscriptionContext(); - subscribe(&MyEventTapSubscriber::onEvent, sc); + subscribe(&MyEventTapSubscriber::handleEvent, sc); return Status::success(); } - Status onEvent(const EventTappingEventContextRef& ec, - const EventTappingSubscriptionContextRef& sc) { - // Process intercepted macOS input event + Status handleEvent(const EventTappingEventContextRef& ec, + const EventTappingSubscriptionContextRef& sc) { + // Process captured macOS input events here return Status::success(); } }; ``` -> **Note:** This publisher is macOS-only and requires the `ApplicationServices` framework. The process may need Accessibility permissions (`TCC`) for the event tap to function correctly in macOS 10.15+. \ No newline at end of file +> **Note:** Requires `com.apple.security.temporary-exception.mach-lookup.global-name` entitlement or Accessibility permissions on macOS. The publisher uses a `CFRunLoop` with a `CFMachPortRef`-backed tap; the tap may be automatically disabled by the OS if the callback is too slow. \ No newline at end of file diff --git a/osquery/events/darwin/.fsevents.md b/osquery/events/darwin/.fsevents.md index 68064e595fb..f14636ebfc8 100644 --- a/osquery/events/darwin/.fsevents.md +++ b/osquery/events/darwin/.fsevents.md @@ -1,47 +1,51 @@ -Wraps Apple's FSEvents API as an osquery `EventPublisher`, enabling filesystem change monitoring with path globbing, recursive watching, and event filtering on macOS. +Defines the macOS FSEvents event publisher for osquery, wrapping Apple's `FSEventStreamRef` API to deliver filesystem change notifications to subscribers. ## Key Components ### `FSEventsSubscriptionContext` -Defines what to watch: -- `path` β€” filesystem path to monitor (supports glob wildcards) +Subscription configuration struct containing: +- `path` β€” filesystem path to watch - `mask` β€” optional `FSEventStreamEventFlags` filter -- `recursive` β€” enables recursive directory watching -- `category` β€” config-originated category label +- `recursive` β€” whether the watch is recursive +- `category` β€” config category origin - `requireAction()` β€” appends a required action filter -- Private fields `discovered_` and `recursive_match` used internally by the publisher for path resolution and event matching +- Private fields `discovered_` and `recursive_match` used internally by the publisher for glob/wildcard resolution ### `FSEventsEventContext` -Carries data for a fired event: +Event data struct populated on each filesystem event: - `fsevent_stream`, `fsevent_flags`, `transaction_id` β€” raw FSEvents metadata -- `path`, `action` β€” resolved path and human-readable action string +- `path`, `action` β€” resolved path and action string ### `FSEventsEventPublisher` Core publisher class inheriting `EventPublisher`: -| Method | Purpose | +| Method | Description | |---|---| -| `configure()` | Rebuilds watch paths and exclude sets on config change | -| `run()` | Starts the CFRunLoop-based event loop | -| `tearDown()` / `stop()` | Stops the stream and cleans up | -| `Callback()` | Static FSEvents client callback; fires event contexts | -| `shouldFire()` | Filters events against subscription criteria | +| `configure()` | Rebuilds watched path set on config load/update | +| `run()` | Starts the `CFRunLoop` entrypoint | +| `tearDown()` / `stop()` | Stops stream and run loop | +| `Callback()` | Static FSEvents client callback receiving raw kernel events | +| `shouldFire()` | Filters events against subscription context | | `transformSubscription()` | Resolves globs/wildcards at configure time | -| `buildExcludePathsSet()` | Populates `ExcludePathSet` for suppressed paths | +| `buildExcludePathsSet()` | Populates `ExcludePathSet` from config exclusions | +| `flush()` | Forces flush of kernel-buffered events | + +### Type Aliases +- `FSEventsEventContextRef` β€” `std::shared_ptr` +- `FSEventsSubscriptionContextRef` β€” `std::shared_ptr` +- `ExcludePathSet` β€” `PathSet` for exclusion matching ## Usage Example ```cpp -// Subscribe to all changes under /etc recursively +// Register a subscription to watch /etc recursively auto sc = std::make_shared(); -sc->path = "/etc"; +sc->path = "/etc"; sc->recursive = true; -sc->category = "config_files"; -sc->requireAction("UPDATED"); - -// The publisher matches fired events against this context via shouldFire() -// and delivers FSEventsEventContext to the subscriber callback -``` +sc->category = "fim"; +sc->requireAction("CREATED"); -> **Note:** This publisher is macOS-only and requires `CoreServices.framework`. Path inputs should use filesystem glob syntax (e.g., `**`), not SQL wildcards. \ No newline at end of file +// The publisher resolves globs, starts the CFRunLoop stream, +// and calls shouldFire() before dispatching to subscribers. +``` \ No newline at end of file diff --git a/osquery/events/darwin/.iokit.md b/osquery/events/darwin/.iokit.md index d99f0934fde..9eb46eefaf3 100644 --- a/osquery/events/darwin/.iokit.md +++ b/osquery/events/darwin/.iokit.md @@ -1,57 +1,54 @@ -Provides the `IOKitEventPublisher` class and associated context structures for subscribing to and handling macOS IOKit hardware device attach/detach events within the osquery event framework. +Provides the `IOKitEventPublisher` class and supporting types for monitoring macOS IOKit hardware device attach/detach events within the osquery event framework. ## Key Components -### Structs +**Structs** +- `IOKitSubscriptionContext` β€” Filter criteria for event subscriptions; holds `model_id`, `vendor_id`, and bus `type` (e.g., `"USB"`) +- `IOKitEventContext` β€” Event payload describing a device event; includes `action`, `type`, `vendor`, `model`, `path`, `driver`, `version`, and `serial` +- `DeviceTracker` β€” Internal struct tracking active device detach notifications -| Struct | Description | -|---|---| -| `IOKitSubscriptionContext` | Subscription filter with `model_id`, `vendor_id`, and bus `type` (e.g., `"USB"`) | -| `IOKitEventContext` | Event payload carrying device metadata: `action`, `vendor`, `model`, `path`, `driver`, `version`, `serial` | - -### Enums - -- **`IOKitEventContext::Action`** β€” `DEVICE_ATTACH (0)` or `DEVICE_DETACH` - -### Type Aliases +**Enums** +- `IOKitEventContext::Action` β€” `DEVICE_ATTACH` (`0`) or `DEVICE_DETACH` +**Type Aliases** - `IOKitEventContextRef` β€” `std::shared_ptr` - `IOKitSubscriptionContextRef` β€” `std::shared_ptr` -### Class: `IOKitEventPublisher` - -Extends `EventPublisher`. Manages a `CFRunLoop` and `IONotificationPort` to listen for IOKit device notifications. +**Class: `IOKitEventPublisher`** +Extends `EventPublisher`; manages a `CFRunLoop` and `IONotificationPort` to receive kernel-level device notifications. | Method | Description | |---|---| -| `run()` | Starts the CFRunLoop and notification monitoring | -| `tearDown()` | Cleans up run loop and IOKit resources | -| `shouldFire()` | Filters events against subscription context | -| `deviceAttach()` *(static)* | IOKit callback for device connect events | -| `deviceDetach()` *(static)* | IOKit callback for device disconnect events | +| `run()` | Starts the `CFRunLoop`; blocks until stopped | +| `tearDown()` | Cleans up IOKit resources | +| `shouldFire()` | Filters events against subscription criteria | +| `deviceAttach()` | Static IOKit callback for device connect | +| `deviceDetach()` | Static IOKit callback for device disconnect | | `newEvent()` | Constructs and fires an `IOKitEventContext` | -| `restart()` | Resets the run loop and re-registers notifications | +| `restart()` | Resets the run loop and notification port | ## Usage Example ```cpp // Subscribe to USB device events auto sc = std::make_shared(); -sc->type = "USB"; +sc->type = "USB"; sc->vendor_id = "05ac"; // Apple vendor ID EventFactory::addSubscription( "iokit", - Subscription::create("usb_monitor", sc, - [](const EventContextRef& ec, const void*) { - auto event = std::static_pointer_cast(ec); - if (event->action == IOKitEventContext::DEVICE_ATTACH) { - LOG(INFO) << "Device attached: " << event->model - << " path=" << event->path; - } - }) + Subscription::create("my_usb_table", sc, callback) ); + +// Event callback receives IOKitEventContextRef +void callback(const IOKitEventContextRef& ec, + const IOKitSubscriptionContextRef& sc) { + if (ec->action == IOKitEventContext::DEVICE_ATTACH) { + LOG(INFO) << "Device attached: " << ec->model + << " path=" << ec->path; + } +} ``` -> **Note:** `publisher_started_` is intentionally set `false` during initial iterator seeding to suppress synthetic attach events at startup β€” only real plug/unplug events are emitted after `restart()` completes. \ No newline at end of file +> **Note:** `publisher_started_` is intentionally `false` during the initial IOKit iterator seed walk to suppress spurious attach events at startup. \ No newline at end of file diff --git a/osquery/events/darwin/.openbsm.md b/osquery/events/darwin/.openbsm.md index 4e4167c36be..63af49d76af 100644 --- a/osquery/events/darwin/.openbsm.md +++ b/osquery/events/darwin/.openbsm.md @@ -1,37 +1,49 @@ -Defines the OpenBSM event publisher and associated context structures for subscribing to and processing macOS/BSD audit log events within the osquery eventing framework. +Header file defining the OpenBSM audit event publisher for osquery, enabling subscription-based monitoring of macOS/BSD kernel audit events via the OpenBSM audit pipe. ## Key Components -| Name | Type | Description | -|---|---|---| -| `OpenBSMSubscriptionContext` | Struct | Holds the `event_id` a subscriber wants to monitor (e.g., `23` for `execve`) | -| `OpenBSMEventContext` | Struct | Carries a fired event's `event_id`, parsed `tokenstr_t` tokens, and a shared memory buffer | -| `OpenBSMEventContextRef` | Type alias | `std::shared_ptr` | -| `OpenBSMSubscriptionContextRef` | Type alias | `std::shared_ptr` | -| `OpenBSMEventPublisher` | Class | Core publisher that reads from the audit pipe, filters events by subscribed IDs, and fires matching event contexts | +### Structs -### `OpenBSMEventPublisher` Methods +- **`OpenBSMSubscriptionContext`** β€” Subscription filter containing `event_id` (e.g., `23` for `execve`) to match specific audit event types +- **`OpenBSMEventContext`** β€” Event payload carrying `event_id`, a vector of parsed `tokenstr_t` tokens, and a managed `buffer` pointer to raw OpenBSM memory -| Method | Description | +### Type Aliases + +- **`OpenBSMEventContextRef`** β€” `std::shared_ptr` +- **`OpenBSMSubscriptionContextRef`** β€” `std::shared_ptr` + +### Classes + +- **`OpenBSMEventPublisher`** β€” Core publisher implementing the osquery `EventPublisher` interface. Manages the audit pipe lifecycle and dispatches matching events to subscribers. + +| Method | Purpose | |---|---| -| `setUp()` | Initializes the publisher | +| `setUp()` | Initializes the audit pipe connection | | `configure()` | Rebuilds the set of subscribed event IDs | -| `tearDown()` | Closes the audit pipe and cleans up resources | -| `run()` | Polling loop β€” reads audit records until interrupted | -| `acquireMessages()` | Dequeues and parses raw records from the audit pipe | -| `configureAuditPipe()` | Opens and configures `/dev/auditpipe` | -| `shouldFire()` | Matches incoming events against subscription contexts | +| `tearDown()` | Closes the audit pipe and cleans up | +| `run()` | Polling loop β€” reads from the audit descriptor until interrupted | +| `shouldFire()` | Filters events against active subscriptions | ## Usage Example ```c -// Subscribe to execve events (event_id = 23) +// Define a subscriber for execve events (event_id = 23) auto sub_ctx = std::make_shared(); sub_ctx->event_id = 23; -// Register the subscription β€” the publisher handles routing -EventFactory::addSubscription("openbsm", sub_ctx, callback); +// Subscribe and handle incoming events +EventFactory::addSubscription( + "openbsm", + Subscription::create("my_subscriber", sub_ctx, callback)); + +// In the callback, access parsed tokens +void callback(const OpenBSMEventContextRef& ec, + const OpenBSMSubscriptionContextRef& sc) { + for (const auto& token : ec->tokens) { + // Process each OpenBSM audit token + } +} ``` -> **Note:** `OpenBSMEventPublisher` is a dispatched service β€” `run()` is called by the osquery eventing thread, not directly by subscribers. Access to `audit_pipe_` and `event_ids_` is guarded by dedicated `Mutex` members. \ No newline at end of file +> **Note:** `OpenBSMEventPublisher` runs as a dispatched service (`OpenBSMConsumerRunner`) and is thread-safe via `Mutex` guards on both the audit pipe and event ID set. \ No newline at end of file diff --git a/osquery/events/darwin/.scnetwork.md b/osquery/events/darwin/.scnetwork.md index 4f5dc1b2f1c..5a99fba90d9 100644 --- a/osquery/events/darwin/.scnetwork.md +++ b/osquery/events/darwin/.scnetwork.md @@ -3,44 +3,39 @@ Provides the osquery event publisher interface for monitoring macOS network reac ## Key Components -### Enumerations - -- **`SCNetworkSubscriptionType`** β€” Defines target types: `ADDRESS_TARGET` (IP address) or `NAME_TARGET` (hostname) +### Enums +- **`SCNetworkSubscriptionType`** β€” Distinguishes between `ADDRESS_TARGET` (IP-based) and `NAME_TARGET` (hostname-based) reachability targets. ### Structs - -- **`SCNetworkSubscriptionContext`** β€” Subscription configuration containing target type, hostname/address string, address family, and a reachability flags bitmask filter -- **`SCNetworkEventContext`** β€” Event payload carrying the originating subscription context and observed `SCNetworkReachabilityFlags` +- **`SCNetworkSubscriptionContext`** β€” Subscription configuration holding the target type, hostname/address string, address family, and an optional reachability flags bitmask (`mask`) to filter events. +- **`SCNetworkEventContext`** β€” Event payload carrying the triggering subscription reference and the current `SCNetworkReachabilityFlags` at the time of the callback. ### Class: `SCNetworkEventPublisher` - -An `EventPublisher` subclass registered under the `"scnetwork"` identifier. Manages a CoreFoundation run loop with registered reachability targets. +An `EventPublisher` that wraps the macOS `SCNetworkReachability` run-loop callback model. | Method | Description | |---|---| -| `configure()` | Rebuilds and registers reachability targets | -| `run()` | Starts the CFRunLoop for async callbacks | -| `tearDown()` | Cleans up all registered targets | -| `shouldFire()` | Filters events against subscription flag masks | -| `Callback()` (static) | SCNetwork callback invoked on reachability change | -| `addHostname()` / `addAddress()` | Register name or address targets | -| `clearAll()` | Removes all tracked targets and contexts | +| `configure()` | Rebuilds reachability targets from active subscriptions | +| `setUp()` | Returns a disabled status (publisher is callback-driven, not polled) | +| `tearDown()` | Releases all targets and stops monitoring | +| `run()` | Enters the `CFRunLoop` for the publisher thread | +| `Callback()` *(static)* | SCNetwork callback invoked on reachability flag changes | +| `shouldFire()` | Filters events against the subscription's flag mask | +| `addHostname()` / `addAddress()` | Register hostname or IP targets with SCNetwork | +| `clearAll()` | Releases and removes all registered reachability targets | ## Usage Example ```cpp -// Define a subscription context to monitor a hostname +// Define a subscription watching a hostname for reachability changes auto sc = std::make_shared(); -sc->type = NAME_TARGET; -sc->target = "example.com"; +sc->type = SCNetworkSubscriptionType::NAME_TARGET; +sc->target = "api.example.com"; sc->mask = kSCNetworkReachabilityFlagsReachable; -// Subscribe an event subscriber using this context -auto sub = EventFactory::createSubscription("scnetwork", sc); +// Subscribe an EventSubscriber to the publisher +auto subscription = SCNetworkEventPublisher::createSubscriptionContext(); +EventFactory::addSubscription("scnetwork", subscription); ``` -## Notes - -- macOS-only; requires `SystemConfiguration.framework` -- `setUp()` always returns `Status(1, "Publisher not used")` β€” publisher lifecycle is driven by `configure()` and `run()` -- Internal state is protected by a `Mutex` for thread-safe target management \ No newline at end of file +> **Note:** `setUp()` intentionally returns a failure status β€” this publisher relies entirely on Apple's `SCNetworkReachability` callback mechanism rather than a polling loop, so it is configured via `configure()` and driven by `CFRunLoop`. \ No newline at end of file diff --git a/osquery/events/linux/.auditdnetlink.md b/osquery/events/linux/.auditdnetlink.md index d1fb75c544f..0c935cba3a0 100644 --- a/osquery/events/linux/.auditdnetlink.md +++ b/osquery/events/linux/.auditdnetlink.md @@ -1,43 +1,43 @@ -Header file defining the audit netlink interface for osquery's Linux audit subsystem integration, providing classes and data structures for reading, parsing, and consuming kernel audit events via the Linux netlink socket. +Header defining the audit netlink interface for osquery's Linux audit subsystem integration, providing structures and services for reading, parsing, and accessing kernel audit events via the Netlink socket. ## Key Components ### Enums & Type Aliases -- **`NetlinkStatus`** β€” Enum indicating netlink handle state: `ActiveMutable`, `ActiveImmutable`, `Disabled`, or `Error` -- **`AuditRuleDataObject`** β€” `std::vector` wrapping raw audit rule data -- **`AuditdContextRef`** β€” `shared_ptr` for shared ownership across services +- **`NetlinkStatus`** β€” Enum indicating handle state: `ActiveMutable`, `ActiveImmutable`, `Disabled`, `Error` +- **`AuditRuleDataObject`** β€” `std::vector` wrapping raw `audit_rule_data` +- **`AuditdContextRef`** β€” `shared_ptr` for shared ownership between services ### Structs -- **`AuditEventRecord`** β€” A parsed audit event containing type, timestamp, audit ID, key-value fields, and raw message data -- **`AuditdContext`** β€” Thread-safe shared state between reader and parser services; holds unprocessed/processed queues, mutexes, condition variables, and throttling counters +- **`AuditEventRecord`** β€” A parsed audit record containing `type`, `time`, `audit_id`, `fields` map, and `raw_data` for SELinux/AppArmor records +- **`AuditdContext`** β€” Thread-safe shared state between reader and parser services, holding unprocessed/processed queues, mutexes, condition variables, and throttling counters ### Classes -- **`AuditdNetlinkReader`** β€” `InternalRunnable` service that acquires the netlink handle, configures audit rules, reads raw `audit_reply` records, and manages audit service lifecycle -- **`AuditdNetlinkParser`** β€” `InternalRunnable` service that parses raw `audit_reply` structures into `AuditEventRecord` objects; exposes static `ParseAuditReply()` and `AdjustAuditReply()` utilities -- **`AuditdNetlink`** β€” Top-level consumer interface; calls `getEvents()` to retrieve fully processed `AuditEventRecord` vectors +- **`AuditdNetlinkReader`** (`InternalRunnable`) β€” Background service that acquires the netlink handle, configures audit rules, reads raw `audit_reply` messages, and manages rule lifecycle +- **`AuditdNetlinkParser`** (`InternalRunnable`) β€” Background service that parses raw `audit_reply` structs into `AuditEventRecord` objects; exposes static `ParseAuditReply()` and `AdjustAuditReply()` helpers +- **`AuditdNetlink`** β€” Public-facing, non-copyable consumer class; call `getEvents()` to drain processed records -### Utility -- **`DecodeAuditPathValues()`** β€” Inline function that strips surrounding quotes or hex-decodes audit field values using `boost::algorithm::unhex` +### Inline Utility +- **`DecodeAuditPathValues()`** β€” Strips surrounding quotes or hex-decodes audit field values via Boost ## Usage Example ```cpp #include "auditdnetlink.h" -// Initialize shared context and the top-level netlink interface +// Instantiate the netlink interface (starts reader/parser internally) osquery::AuditdNetlink netlink; -// Poll for new audit events in a publisher loop +// Poll for parsed audit events in a publisher loop while (running) { - std::vector events = netlink.getEvents(); - for (const auto& record : events) { - // Access parsed fields - auto it = record.fields.find("exe"); - if (it != record.fields.end()) { - std::string exe = osquery::DecodeAuditPathValues(it->second); - // process exe path... + auto events = netlink.getEvents(); + for (const auto& record : events) { + // record.type -> AUDIT_SYSCALL, AUDIT_PATH, etc. + // record.fields -> key/value pairs from the audit record + auto path_it = record.fields.find("name"); + if (path_it != record.fields.end()) { + auto decoded = osquery::DecodeAuditPathValues(path_it->second); + } } - } } ``` \ No newline at end of file diff --git a/osquery/events/linux/.auditeventpublisher.md b/osquery/events/linux/.auditeventpublisher.md index b3b36217a6a..3b3a3a080e4 100644 --- a/osquery/events/linux/.auditeventpublisher.md +++ b/osquery/events/linux/.auditeventpublisher.md @@ -1,60 +1,74 @@ -Header file defining the Linux audit event publisher for osquery, providing data structures and interfaces for processing kernel audit events including syscalls, AppArmor, SELinux, and seccomp notifications via the Linux Audit netlink subsystem. +Defines the Linux audit event publisher for osquery, providing data structures and publisher logic to collect, parse, and dispatch kernel audit events (syscalls, user events, AppArmor, SELinux, and Seccomp) from the Linux audit subsystem via netlink. ## Key Components ### Event Data Structures | Struct | Description | -|---|---| +|--------|-------------| | `UserAuditEventData` | Holds a user-space audit event ID | -| `SyscallAuditEventData` | Captures syscall metadata: PID, UID/GID sets, executable path, success status | -| `AppArmorAuditEventData` | Key-value fields for AppArmor audit records (profile, operation, masks, etc.) | -| `SeccompAuditEventData` | Key-value fields for seccomp audit records (arch, syscall, signal, instruction pointer, etc.) | -| `AuditEvent` | Top-level event descriptor wrapping all event types via `boost::variant` with a raw `record_list` | +| `SyscallAuditEventData` | Syscall metadata: PID, UID/GID sets, executable path, success flag | +| `AppArmorAuditEventData` | AppArmor fields (profile, operation, capability, masks, etc.) | +| `SeccompAuditEventData` | Seccomp fields (arch, syscall, signal, IP, compat mode, etc.) | +| `AuditEvent` | Top-level event descriptor holding type enum + variant data + raw record list | -### Publisher Infrastructure +### Publisher & Context Types -| Type | Description | -|---|---| -| `AuditEventPublisher` | Core `EventPublisher` subclass; manages netlink I/O, assembles audit records into events | -| `AuditEventContext` | Carries a `vector` dispatched to subscribers | -| `AuditSubscriptionContext` | Subscription handle (opaque; controlled by the publisher) | -| `AuditTraceContext` | `map` tracking in-flight multi-record audit sequences | +- **`AuditEventPublisher`** β€” Core publisher class; drives the netlink read loop (`run()`), assembles raw records into typed `AuditEvent` objects, and dispatches to subscribers +- **`AuditSubscriptionContext`** / **`AuditEventContext`** β€” Standard osquery pub/sub context types; event context carries a `std::vector` +- **`AuditTraceContext`** β€” `std::map` used to correlate multi-record audit sequences by event ID -### Free Functions +### Key Free Functions | Function | Description | -|---|---| -| `ProcessEvents()` | Aggregates raw `AuditEventRecord` lists into typed `AuditEvent` objects | -| `GetEventRecord()` | Extracts a record of a specific type from an `AuditEvent` | -| `GetStringFieldFromMap()` | Safe string field lookup with default value | -| `GetIntegerFieldFromMap()` | Safe integer field lookup with configurable base and default | -| `CopyFieldFromMap()` | Copies a named field into an osquery `Row` | +|----------|-------------| +| `AuditEventPublisher::ProcessEvents()` | Aggregates raw `AuditEventRecord` lists into complete `AuditEvent` objects | +| `GetEventRecord()` | Retrieves a record by type from an `AuditEvent` | +| `GetStringFieldFromMap()` | Safe string field extraction from audit field maps | +| `GetIntegerFieldFromMap()` | Safe integer field extraction with configurable base/default | +| `CopyFieldFromMap()` | Copies a field directly into an osquery `Row` | | `StripQuotes()` | Removes surrounding quotes from audit field values | -| `parseSeccompEvent()` | Parses a raw seccomp record into `SeccompAuditEventData` | +| `parseSeccompEvent()` | Parses a raw Seccomp record into `SeccompAuditEventData` | ## Usage Example ```cpp -// Subscribing to audit events -auto sub_ctx = AuditEventPublisher::createSubscriptionContext(); -EventFactory::addSubscription("auditeventpublisher", sub_ctx, - [](const AuditEventContextRef& ctx, const AuditSubscriptionContextRef&) { - for (const auto& event : ctx->audit_events) { - if (event.type == AuditEvent::Type::Syscall) { - const auto& data = boost::get(event.data); - LOG(INFO) << "Syscall " << data.syscall_number - << " by PID " << data.process_id - << " exe: " << data.executable_path; +// Subscribing to audit events in an osquery EventSubscriber +void AuditMySubscriber::init() { + auto subscription_context = + createSubscriptionContext(); + + subscribe(&AuditMySubscriber::eventCallback, subscription_context); +} + +Status AuditMySubscriber::eventCallback( + const AuditEventContextRef& context, + const AuditSubscriptionContextRef& /*subscription*/) { + + for (const auto& audit_event : context->audit_events) { + if (audit_event.type == AuditEvent::Type::Syscall) { + const auto& data = + boost::get(audit_event.data); + + Row row; + row["syscall"] = std::to_string(data.syscall_number); + row["pid"] = std::to_string(data.process_id); + row["executable"] = data.executable_path; + row["succeeded"] = data.succeeded ? "1" : "0"; + + // Extract additional fields from raw records + const auto* record = + GetEventRecord(audit_event, AUDIT_SYSCALL); + if (record != nullptr) { + std::string cwd; + GetStringFieldFromMap(cwd, record->fields, "cwd", ""); + row["cwd"] = StripQuotes(cwd); } + + addRow(row); } - return Status::success(); - }); - -// Assembling raw records into typed events (used internally / in tests) -AuditEventContextRef ctx = std::make_shared(); -AuditTraceContext trace; -std::set allowed_failures = {257}; // e.g. openat allowed to fail -AuditEventPublisher::ProcessEvents(ctx, raw_records, trace, allowed_failures); + } + return Status::success(); +} ``` \ No newline at end of file diff --git a/osquery/events/linux/.inotify.md b/osquery/events/linux/.inotify.md index 920966d4189..48edb9f8365 100644 --- a/osquery/events/linux/.inotify.md +++ b/osquery/events/linux/.inotify.md @@ -1,44 +1,57 @@ -Linux `inotify`-based filesystem event publisher header for osquery, defining subscription contexts, event contexts, and the `INotifyEventPublisher` class used to monitor file and directory changes on Linux. +Linux `inotify` filesystem event publisher interface for osquery, defining the subscription context, event context, and publisher class used to monitor file and directory changes via the kernel's inotify API. ## Key Components -### Constants & Type Aliases -- `kMaskActions` β€” maps `inotify` mask bits to human-readable action strings -- `kFileDefaultMasks` / `kFileAccessMasks` β€” predefined `inotify` event mask sets -- `PathDescriptorMap` / `DescriptorPathMap` β€” bidirectional path ↔ watch descriptor lookup maps -- `ExcludePathSet` β€” pattern-aware set of paths excluded from event propagation +### Structs -### `INotifySubscriptionContext` -Subscription configuration struct holding the watched path, optional action mask, recursive flag, category label, and lazy-deletion marker. Provides `requireAction()` to map string action names to `inotify` bitmask values. +- **`INotifySubscriptionContext`** β€” Subscription configuration for a watched path. Holds the target path, event mask, recursive flag, category, and internal descriptor/status-change-time maps. Provides `requireAction()` to map string action names to inotify mask bits. +- **`INotifyEventContext`** β€” Carries a fired inotify event, including the raw `inotify_event` struct, resolved path string, action string, transaction ID, and a back-reference to the originating subscription context. -### `INotifyEventContext` -Event payload struct carrying the raw `inotify_event`, resolved path string, action string, transaction ID, and a reference back to the originating subscription context. +### Type Aliases -### `INotifyEventPublisher` -Core publisher class (inherits `EventPublisher`) managing the `inotify` file descriptor lifecycle, watch descriptor bookkeeping, and event dispatch loop. Key methods: - -| Method | Purpose | +| Alias | Underlying Type | |---|---| -| `setUp()` | Opens the `inotify` handle | -| `configure()` | Applies updated subscriptions from config | -| `run()` | Main event polling loop | -| `addMonitor()` | Registers a path watch, optionally recursive | -| `needMonitoring()` | Guards `addMonitor` using status-change time | -| `removeMonitor()` | Unregisters a watch descriptor | -| `shouldFire()` | Matches an event against a subscription's path/mask | -| `handleOverflow()` | Recovers from `IN_Q_OVERFLOW` by increasing read batch size | +| `PathDescriptorMap` | `std::map` | +| `DescriptorPathMap` | `std::map` | +| `DescriptorINotifySubCtxMap` | `std::map` | +| `ExcludePathSet` | `PathSet` | + +### Class: `INotifyEventPublisher` + +Extends `EventPublisher`. Manages the inotify file descriptor lifecycle, watch descriptor bookkeeping, recursive directory monitoring, and event dispatch. + +**Key public methods:** + +- `setUp()` β€” Initializes the inotify handle +- `configure()` β€” Applies updated config to active watches +- `tearDown()` β€” Releases the inotify handle and cleans up +- `run()` β€” Main event loop; reads and dispatches inotify events +- `addSubscription()` β€” Deduplicates and registers a new subscription +- `removeSubscriptions()` β€” Lazily marks subscriptions for removal + +**Key private methods:** + +- `addMonitor()` / `needMonitoring()` β€” Add inotify watches with optional recursive enumeration +- `monitorSubscription()` β€” Resolves a subscription context to actual watches +- `shouldFire()` β€” Matches an event against a subscription's path and mask +- `buildExcludePathsSet()` β€” Populates the path exclusion set from config +- `handleOverflow()` β€” Recovers from `IN_Q_OVERFLOW` by reading extra events + +### Global Constants + +- `kMaskActions` β€” Maps inotify mask bits to human-readable action strings +- `kFileDefaultMasks` / `kFileAccessMasks` β€” Preset mask combinations for common monitoring scenarios ## Usage Example ```cpp -// Subscribe to file creation events under a directory +// Subscribe to write events on a directory auto sc = std::make_shared(); -sc->path = "/etc/myapp/"; +sc->path = "/etc/"; sc->recursive = true; -sc->requireAction("CREATED"); +sc->requireAction("UPDATED"); // sets IN_CLOSE_WRITE bit in mask -// The publisher dispatches INotifyEventContext to matching subscribers -// ec->path β†’ absolute path of the affected file -// ec->action β†’ "CREATED", "UPDATED", "DELETED", etc. +auto sub = Subscription::create("my_subscriber", sc, callback); +EventFactory::addSubscription("inotify", sub); ``` \ No newline at end of file diff --git a/osquery/events/linux/.socket_events.md b/osquery/events/linux/.socket_events.md index e6e1e3b6741..06d333d269a 100644 --- a/osquery/events/linux/.socket_events.md +++ b/osquery/events/linux/.socket_events.md @@ -1,9 +1,9 @@ -Declares the interface for retrieving the set of system call numbers monitored by the socket events subsystem in osquery. +Declares the interface for retrieving the set of system call identifiers monitored for socket-related events within the osquery event framework. ## Key Components -- **`getSocketEventsSyscalls()`** β€” Returns a `std::set` containing the syscall numbers relevant to socket event monitoring (e.g., `connect`, `bind`, `accept`). +- **`getSocketEventsSyscalls()`** β€” Returns a `std::set` containing the syscall numbers relevant to socket event monitoring (e.g., `socket`, `connect`, `bind`, `accept`). ## Usage Example @@ -11,16 +11,12 @@ Declares the interface for retrieving the set of system call numbers monitored b #include "socket_events.h" #include -// Retrieve all syscalls tracked for socket events +// Retrieve syscalls tracked for socket events std::set syscalls = osquery::getSocketEventsSyscalls(); -for (int syscall_nr : syscalls) { - std::cout << "Monitoring syscall: " << syscall_nr << std::endl; +for (int syscall : syscalls) { + std::cout << "Monitoring syscall: " << syscall << std::endl; } ``` -## Notes - -- Lives within the `osquery` namespace. -- Typically consumed by the audit-based event publisher to register which syscalls to subscribe to for network socket activity tracking. -- The returned set drives kernel-level audit rule registration β€” only syscalls in this set will generate socket events in the osquery event pipeline. \ No newline at end of file +> **Note:** This header is typically used internally by osquery's audit-based event publishers to register the correct syscall filters when setting up socket activity monitoring on Linux systems. \ No newline at end of file diff --git a/osquery/events/linux/.syslog.md b/osquery/events/linux/.syslog.md index cd8b52ae6e8..1bf0f1a58b9 100644 --- a/osquery/events/linux/.syslog.md +++ b/osquery/events/linux/.syslog.md @@ -1,25 +1,29 @@ -Declares the event publisher and supporting types for ingesting syslog entries forwarded from rsyslog via a named pipe, publishing parsed log fields to osquery subscribers. +Defines the event publisher infrastructure for ingesting syslog entries forwarded through rsyslog, including context types, a non-blocking pipe reader, and a CSV tokenizer for rsyslog output. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `SyslogSubscriptionContext` | struct | Empty subscription context; no filtering criteria needed | -| `SyslogEventContext` | struct | Holds a syslog message as a `map` of parsed fields | -| `NonBlockingFStream` | class | Non-blocking pipe reader with internal ring buffer; exposes `openReadOnly`, `getline`, and `close` | -| `SyslogEventPublisher` | class | osquery event publisher; reads JSON syslog lines from a named pipe, parses them, and fires events | -| `RsyslogCsvSeparator` | class | Boost `TokenizerFunction` functor that handles rsyslog's CSV dialect (`""` escaping, no backslash escaping) | +### Structs +- **`SyslogSubscriptionContext`** β€” Empty subscription context for syslog event filtering (reserved for future use) +- **`SyslogEventContext`** β€” Holds a parsed syslog entry as a `map` of field name/value pairs + +### Classes +- **`NonBlockingFStream`** β€” Non-blocking file stream wrapper over a raw file descriptor; buffers reads and exposes a `getline()` that returns immediately without blocking on a pipe. Default buffer size is 2048 bytes. +- **`SyslogEventPublisher`** β€” Core event publisher that reads JSON syslog lines from a named pipe created for rsyslog forwarding. Manages pipe creation, exclusive locking, error thresholding, and event dispatch to subscribers. +- **`RsyslogCsvSeparator`** β€” Boost `TokenizerFunction` functor that correctly parses rsyslog CSV output, handling rsyslog's `""` quote-escaping convention and no backslash escaping. + +### Type Aliases +- `SyslogEventContextRef` β€” `shared_ptr` +- `SyslogSubscriptionContextRef` β€” `shared_ptr` ## Usage Example ```cpp -// Subscribing to syslog events in an EventSubscriber +// Subscribing to syslog events class MySyslogSubscriber : public EventSubscriber { public: Status init() override { - // No filter criteria needed; SyslogSubscriptionContext is empty auto sc = createSubscriptionContext(); subscribe(&MySyslogSubscriber::onEvent, sc); return Status::success(); @@ -30,11 +34,19 @@ class MySyslogSubscriber // Access parsed syslog fields auto it = ec->fields.find("message"); if (it != ec->fields.end()) { - LOG(INFO) << "syslog message: " << it->second; + LOG(INFO) << "Syslog message: " << it->second; } return Status::success(); } }; + +// Non-blocking pipe read (used internally by SyslogEventPublisher) +NonBlockingFStream stream; +stream.openReadOnly("/var/osquery/syslog.pipe"); +std::string line; +if (stream.getline(line).ok()) { + // process complete line +} ``` -> **Note:** `SyslogEventPublisher` requires rsyslog to be configured to write JSON-formatted entries to the named pipe path defined in osquery's configuration. Only the first osquery process to acquire the pipe lock (`lockPipe`) will consume events; subsequent processes (e.g. `osqueryi`) are silently skipped to avoid corrupting `osqueryd` reads. \ No newline at end of file +> **Note:** rsyslog must be configured to forward JSON-formatted syslog entries to the named pipe path that `SyslogEventPublisher` monitors. Only the first osquery process to acquire the pipe lock will be permitted to read, preventing conflicts between `osqueryd` and `osqueryi` instances. \ No newline at end of file diff --git a/osquery/events/linux/.udev.md b/osquery/events/linux/.udev.md index 2dcfc678f43..487922414c6 100644 --- a/osquery/events/linux/.udev.md +++ b/osquery/events/linux/.udev.md @@ -1,49 +1,44 @@ -Defines the Linux `udev` event publisher interface for osquery, enabling subscribers to monitor hardware device events (add, remove, change) via the `libudev` API. +Defines the Linux `udev` event publisher interface for osquery, enabling subscribers to monitor hardware device events (add, remove, change) via the kernel's udev subsystem. ## Key Components -### Enums -- **`udev_event_action`** β€” Device event types: `ADD`, `REMOVE`, `CHANGE`, `UNKNOWN`, and `ALL` (subscriber catch-all) +**Enums** +- `udev_event_action` β€” Device event action types: `ADD`, `REMOVE`, `CHANGE`, `UNKNOWN`, and `ALL` (subscriber catch-all) -### Structs -- **`UdevSubscriptionContext`** β€” Subscription filter criteria; restrict events by `action`, `subsystem`, `devnode`, `devtype`, or `driver` -- **`UdevEventContext`** β€” Fired event payload carrying a `udev_device*` pointer, resolved `action`, and device metadata strings +**Structs** +- `UdevSubscriptionContext` β€” Subscription filter criteria: action, subsystem, devnode, devtype, and driver +- `UdevEventContext` β€” Event payload: raw `udev_device*` pointer, action enum + string, subsystem, devnode, devtype, and driver -### Type Aliases -- **`UdevEventContextRef`** β€” `std::shared_ptr` -- **`UdevSubscriptionContextRef`** β€” `std::shared_ptr` - -### Class: `UdevEventPublisher` -Extends `EventPublisher` and manages the udev socket monitor lifecycle. +**Type Aliases** +- `UdevEventContextRef` β€” `std::shared_ptr` +- `UdevSubscriptionContextRef` β€” `std::shared_ptr` +**Class: `UdevEventPublisher`** | Method | Description | |---|---| | `setUp()` | Initializes the udev handle and monitor | | `tearDown()` | Releases udev resources | -| `run()` | Polls the monitor and dispatches events | -| `getValue(device, property)` | Returns a udev property string | -| `getAttr(device, attr)` | Returns a udev sysattr string | -| `shouldFire(...)` | Filters events against subscription context | +| `run()` | Event loop β€” polls udev monitor socket | +| `getValue(device, property)` | Returns a udev property string value | +| `getAttr(device, attr)` | Returns a udev sysattr string value | +| `shouldFire(...)` | Matches incoming events against subscription filters | | `createEventContextFrom(device)` | Builds an `UdevEventContextRef` from a raw device pointer | ## Usage Example ```cpp -// Create a subscription context filtering USB add events -auto sc = createSubscriptionContext(); +// Define a subscription context to watch USB block device additions +auto sc = std::make_shared(); sc->action = UDEV_EVENT_ACTION_ADD; sc->subsystem = "usb"; - -// Subscribe with a callback -subscribe(&MySubscriber::onUsbAdd, sc); - -// In the callback, inspect event context -void onUsbAdd(const UdevEventContextRef& ec) { - auto node = ec->devnode; // e.g. "/dev/bus/usb/001/002" - auto driver = ec->driver; - auto device = ec->device; // raw udev_device* for deeper inspection - - auto vendor = UdevEventPublisher::getValue(device, "ID_VENDOR"); +sc->devtype = "usb_device"; + +// Retrieve a property from an event +void myCallback(const UdevEventContextRef& ec) { + std::string vendor = UdevEventPublisher::getValue( + ec->device, "ID_VENDOR"); + std::string model = UdevEventPublisher::getAttr( + ec->device, "idProduct"); } ``` \ No newline at end of file diff --git a/osquery/events/linux/bpf/.bpferrorstate.md b/osquery/events/linux/bpf/.bpferrorstate.md index 219a15712c6..1014efe9167 100644 --- a/osquery/events/linux/bpf/.bpferrorstate.md +++ b/osquery/events/linux/bpf/.bpferrorstate.md @@ -1,38 +1,35 @@ -Defines the `BPFErrorState` structure and related utilities for tracking and reporting errors encountered during BPF-based event collection in the osquery BPF publisher. +Defines the `BPFErrorState` structure and utility functions for tracking, reporting, and clearing BPF-related error counters within the osquery BPF event publisher. ## Key Components ### `BPFErrorState` (struct) -A container aggregating all BPF publisher error counters: +A container aggregating all error counters produced during BPF event publishing: | Field | Type | Description | -|---|---|---| -| `perf_error_counters` | `IPerfEventReader::ErrorCounters` | Errors from `perf_output` ring buffer reads | -| `probe_error_counter` | `std::size_t` | Failures from BPF probes (e.g. string/buffer capture failures) | +|-------|------|-------------| +| `perf_error_counters` | `IPerfEventReader::ErrorCounters` | Errors from `perf_output` event reading | +| `probe_error_counter` | `std::size_t` | Failures within loaded BPF probes (e.g., string or buffer capture failures) | | `errored_tracer_list` | `std::unordered_set` | IDs of tracers that produced unprocessable events | -### `updateBpfErrorState()` -Merges incoming `perf_output` error counters into an existing `BPFErrorState` instance. +### Functions -### `reportAndClearBpfErrorState()` -Logs the current error state (typically to the osquery logger) and resets all counters and sets to their zero/empty defaults. +- **`updateBpfErrorState`** β€” Merges incoming `perf_output` error counters into an existing `BPFErrorState` instance. +- **`reportAndClearBpfErrorState`** β€” Logs the current error state and resets all counters/sets to their zero/empty state. ## Usage Example -```c -BPFErrorState error_state{}; +```cpp +#include "bpferrorstate.h" -// After each perf reader poll cycle, merge new counters in -IPerfEventReader::ErrorCounters new_counters = reader->getErrorCounters(); -updateBpfErrorState(error_state, new_counters); +osquery::BPFErrorState error_state{}; -// Periodically flush errors to logs and reset state -reportAndClearBpfErrorState(error_state); -``` +// Update with new perf counters after each read cycle +tob::ebpfpub::IPerfEventReader::ErrorCounters counters = reader->errorCounters(); +osquery::updateBpfErrorState(error_state, counters); -## Notes +// Periodically report and reset accumulated errors +osquery::reportAndClearBpfErrorState(error_state); +``` -- `probe_error_counter` covers soft failures such as truncated strings in `open` syscall captures or malformed `sockaddr_in` buffers β€” not fatal crashes. -- `errored_tracer_list` deduplicates affected tracer IDs, making it useful for identifying persistently failing probes. -- Both functions operate on `BPFErrorState` by reference, making them safe to use with a long-lived state object across polling intervals. \ No newline at end of file +> **Note:** `errored_tracer_list` captures tracers affected by lost events or probe-level failures. A non-empty set after a cycle indicates events may have been silently dropped and warrants investigation. \ No newline at end of file diff --git a/osquery/events/linux/bpf/.bpfeventpublisher.md b/osquery/events/linux/bpf/.bpfeventpublisher.md index 1e344a84b2d..2c0cca1e380 100644 --- a/osquery/events/linux/bpf/.bpfeventpublisher.md +++ b/osquery/events/linux/bpf/.bpfeventpublisher.md @@ -1,54 +1,56 @@ -Header defining the `BPFEventPublisher` class for osquery's Linux BPF-based event publishing system, which captures and processes kernel syscall events via eBPF tracing. +BPF-based event publisher for osquery on Linux that captures and processes kernel-level system call events using eBPF tracing infrastructure. ## Key Components ### Structs -| Name | Base | Description | -|------|------|-------------| -| `BPFEventSC` | `SubscriptionContext` | Subscription context for BPF event subscriptions (opaque to subscribers) | -| `BPFEventEC` | `EventContext` | Event context carrying a list of system state events (`ISystemStateTracker::EventList`) | +| Name | Description | +|------|-------------| +| `BPFEventSC` | Subscription context for BPF events (opaque to subscribers) | +| `BPFEventEC` | Event context carrying a list of tracked system state events | ### Class: `BPFEventPublisher` -Extends `EventPublisher`. Manages the eBPF perf event reader lifecycle and dispatches processed syscall events to subscribers. +Extends `EventPublisher` and manages the full lifecycle of eBPF-based syscall monitoring. -**Lifecycle methods:** +**Lifecycle Methods** | Method | Description | |--------|-------------| -| `setUp()` | Initializes eBPF tracers and perf event reader | +| `setUp()` | Initializes eBPF tracers and perf event readers | | `configure()` | Applies runtime configuration | -| `run()` | Poll loop consuming BPF perf events | +| `run()` | Main loop β€” reads perf events and dispatches to subscribers | | `tearDown()` | Cleans up eBPF resources | -**Static helper β€” `getEventMapValue`** +**Static Syscall Processors** -Extracts a typed value from a `IFunctionTracer::Event::FieldMap` by key using `std::variant`. Returns `false` if the key is missing or the type doesn't match. - -**Static syscall processors** - -One handler per traced syscall, each updating an `ISystemStateTracker` with the parsed event: +Each `process*Event` method handles a specific syscall traced via eBPF, updating the `ISystemStateTracker` accordingly: - **Process lifecycle:** `processForkEvent`, `processVforkEvent`, `processCloneEvent`, `processExecveEvent`, `processExecveatEvent` - **File descriptors:** `processCloseEvent`, `processDupEvent`, `processDup2Event`, `processDup3Event`, `processFcntlEvent` -- **File system:** `processCreatEvent`, `processMknodatEvent`, `processOpenEvent`, `processOpenatEvent`, `processOpenat2Event`, `processChdirEvent`, `processFchdirEvent`, `processNameToHandleAtEvent`, `processOpenByHandleAtEvent` +- **File system:** `processOpenEvent`, `processOpenatEvent`, `processOpenat2Event`, `processCreatEvent`, `processMknodatEvent`, `processChdirEvent`, `processFchdirEvent`, `processNameToHandleAtEvent`, `processOpenByHandleAtEvent` - **Networking:** `processSocketEvent`, `processConnectEvent`, `processAcceptEvent`, `processAccept4Event`, `processBindEvent`, `processListenEvent` +**Helper** + +`getEventMapValue` β€” typed extraction of a field value from an eBPF event's field map. + ## Usage Example ```cpp -// Retrieve a typed field from a BPF event map -uint64_t fd = 0; -bool ok = BPFEventPublisher::getEventMapValue( - fd, event.field_map, "fd"); - -if (ok) { - // Use fd value -} - -// Dispatch a clone syscall event to the state tracker -bool handled = BPFEventPublisher::processCloneEvent( - stateTracker, bpfEvent); +// Extracting a field from a BPF event's field map +uint64_t pid = 0; +bool ok = BPFEventPublisher::getEventMapValue( + pid, + event.field_map, + "pid" +); + +// Handling a fork syscall event against the state tracker +bool handled = BPFEventPublisher::processForkEvent(stateTracker, bpfEvent); + +// Subscribing to BPF events +auto sub_ctx = std::make_shared(); +EventFactory::addSubscription("BPFEventPublisher", sub_ctx, callback); ``` \ No newline at end of file diff --git a/osquery/events/linux/bpf/.filesystem.md b/osquery/events/linux/bpf/.filesystem.md index b295a6a391e..45083a5796a 100644 --- a/osquery/events/linux/bpf/.filesystem.md +++ b/osquery/events/linux/bpf/.filesystem.md @@ -1,48 +1,47 @@ -Concrete implementation of the `IFilesystem` interface for Linux BPF event processing, providing real filesystem operations backed by system calls. +Concrete implementation of the `IFilesystem` interface for Linux BPF event processing, providing real filesystem operations backed by native system calls. ## Key Components **`Filesystem`** β€” Final class implementing `IFilesystem` with the following methods: | Method | Description | -|---|---| -| `open()` | Opens a file at the given path with specified flags, returning a unique fd | -| `openAt()` | Opens a file relative to a directory fd (`dirfd`) | -| `readLinkAt()` | Reads a symbolic link target relative to a directory fd | -| `read()` | Reads up to `max_size` bytes from an open fd into a buffer | +|--------|-------------| +| `open()` | Opens a file at the given path with specified flags, returning a unique FD | +| `openAt()` | Opens a file relative to a directory FD (`dirfd`) | +| `readLinkAt()` | Resolves a symbolic link relative to a directory FD | +| `read()` | Reads up to `max_size` bytes from an open FD into a buffer | | `enumFiles()` | Enumerates files in a directory, invoking a callback per entry | -| `fileExists()` | Checks whether a named file exists within a directory fd | +| `fileExists()` | Checks whether a named file exists within a directory FD | -> Construction is private (`Filesystem() = default`), enforcing instantiation exclusively through the `IFilesystem` factory (friend class pattern). +**Construction** is private β€” instances are created exclusively through `IFilesystem` (factory pattern via `friend class IFilesystem`). ## Usage Example ```cpp #include -// Instantiate via the IFilesystem factory (not directly) +// Obtain instance via the IFilesystem factory auto fs = IFilesystem::create(); // Open a file tob::utils::UniqueFd fd; -if (fs->open(fd, "/proc/self/maps", O_RDONLY)) { +if (fs->open(fd, "/proc/self/status", O_RDONLY)) { std::vector buffer; fs->read(buffer, fd.get(), 4096); } -// Check file existence relative to a directory fd +// Check existence relative to a directory fd bool exists = false; -fs->fileExists(exists, AT_FDCWD, "config.json"); +fs->fileExists(exists, dirfd, "somefile.txt"); // Enumerate directory entries -fs->enumFiles(AT_FDCWD, [](const std::string& name) { - // process each entry +fs->enumFiles(dirfd, [](const std::string& name, bool is_dir) { + // handle each entry }); ``` ## Notes -- Scoped to the `osquery` namespace -- Designed for Linux only (BPF event subsystem) -- All methods return `bool` indicating success/failure; output is via reference parameters \ No newline at end of file +- Instantiation is restricted to `IFilesystem` via the private constructor and `friend` declaration, enforcing use of the factory pattern. +- Designed exclusively for the Linux BPF events subsystem within osquery. \ No newline at end of file diff --git a/osquery/events/linux/bpf/.processcontextfactory.md b/osquery/events/linux/bpf/.processcontextfactory.md index dad39b819ef..d84786ac31f 100644 --- a/osquery/events/linux/bpf/.processcontextfactory.md +++ b/osquery/events/linux/bpf/.processcontextfactory.md @@ -1,51 +1,44 @@ -Concrete implementation of `IProcessContextFactory` for capturing Linux process context data by reading `/proc` filesystem entries via BPF event handling. +Concrete implementation of `IProcessContextFactory` for capturing Linux process context data via the `/proc` filesystem using BPF event infrastructure. ## Key Components -### Class: `ProcessContextFactory` - -Final implementation of `IProcessContextFactory` with both instance and static utility methods. +### `ProcessContextFactory` (class) +Final implementation of `IProcessContextFactory` that reads process metadata from the Linux `/proc` filesystem. **Instance Methods** +- `captureSingleProcess(ProcessContext&, pid_t)` β€” Captures context for a single process by PID +- `captureAllProcesses(ProcessContextMap&)` β€” Captures context for all currently running processes -| Method | Description | -|---|---| -| `captureSingleProcess(ProcessContext&, pid_t)` | Captures context for a single process by PID | -| `captureAllProcesses(ProcessContextMap&)` | Captures context for all running processes | - -**Static Utility Methods** - -| Method | Description | -|---|---| -| `captureSingleProcess(IFilesystem&, ProcessContext&, pid_t)` | Filesystem-injectable single process capture | -| `captureAllProcesses(IFilesystem&, ProcessContextMap&)` | Filesystem-injectable bulk process capture | -| `getArgvFromCmdlineFile(IFilesystem&, vector&, int fd)` | Parses argv from `/proc//cmdline` | -| `getParentPidFromStatFile(IFilesystem&, pid_t&, int fd)` | Extracts parent PID from `/proc//stat` | +**Static Helpers** +- `captureSingleProcess(IFilesystem&, ProcessContext&, pid_t)` β€” Filesystem-injectable variant for single process capture +- `captureAllProcesses(IFilesystem&, ProcessContextMap&)` β€” Filesystem-injectable variant for bulk capture +- `getArgvFromCmdlineFile(IFilesystem&, std::vector&, int fd)` β€” Parses argv from a `/proc//cmdline` file descriptor +- `getParentPidFromStatFile(IFilesystem&, pid_t&, int fd)` β€” Extracts parent PID from a `/proc//stat` file descriptor -**Private Members** -- `IFilesystem::Ref fs` β€” injected filesystem abstraction for testability -- Private constructor (instantiated via `IProcessContextFactory` friend) +**Private** +- `fs` β€” `IFilesystem::Ref` dependency used for `/proc` reads +- Constructor is private; instantiation is controlled via `IProcessContextFactory` (friend) ## Usage Example -```c -// Typically instantiated through IProcessContextFactory -auto factory = IProcessContextFactory::create(fs_ref); +```cpp +// Instantiation is handled via IProcessContextFactory factory method +auto factory = IProcessContextFactory::create(filesystem_ref); -// Capture a single process by PID -ProcessContext ctx; +// Capture a single process +osquery::ProcessContext ctx; if (factory->captureSingleProcess(ctx, 1234)) { - // ctx now contains argv, parent PID, etc. + // ctx now populated with PID 1234's argv, ppid, etc. } -// Capture all running processes -ProcessContextMap all_procs; -factory->captureAllProcesses(all_procs); +// Capture all processes +osquery::ProcessContextMap process_map; +factory->captureAllProcesses(process_map); -// Static form β€” useful in tests with mock filesystem -ProcessContext ctx2; -ProcessContextFactory::captureSingleProcess(mock_fs, ctx2, 1234); +// Using static helpers directly (e.g., in tests) +ProcessContextFactory::getArgvFromCmdlineFile(fs, argv, fd); +ProcessContextFactory::getParentPidFromStatFile(fs, parent_pid, fd); ``` -> **Note:** The private constructor and `friend class IProcessContextFactory` pattern enforces creation through the factory interface, enabling dependency injection of `IFilesystem` for unit testing. \ No newline at end of file +> The static method overloads accept an `IFilesystem&` parameter, enabling dependency injection for unit testing without requiring a real `/proc` filesystem. \ No newline at end of file diff --git a/osquery/events/linux/bpf/.serializers.md b/osquery/events/linux/bpf/.serializers.md index e55d7396631..ed0df53416b 100644 --- a/osquery/events/linux/bpf/.serializers.md +++ b/osquery/events/linux/bpf/.serializers.md @@ -1,28 +1,20 @@ -Declares the `ParameterListMap` type and the `kParameterListMap` lookup table used to map eBPF function tracer names to their corresponding parameter lists. +Defines the `ParameterListMap` type alias and declares the `kParameterListMap` constant used to map syscall/function names to their eBPF parameter list definitions for osquery's eBPF-based event tracing. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `ParameterListMap` | Type alias | `unordered_map` keyed by `std::string`, mapping to `IFunctionTracer::ParameterList` | -| `kParameterListMap` | `extern const` | Global registry of eBPF-traced function names to their parameter definitions | +- **`ParameterListMap`** β€” Type alias for `std::unordered_map`, mapping function/syscall name strings to their corresponding eBPF parameter lists. +- **`kParameterListMap`** β€” Externally declared constant map instance containing the pre-defined parameter list entries used during eBPF function tracing serialization. ## Usage Example ```cpp #include "serializers.h" -// Look up parameters for a known syscall tracer -auto it = osquery::kParameterListMap.find("open"); +// Look up parameter list for a specific syscall +auto it = osquery::kParameterListMap.find("execve"); if (it != osquery::kParameterListMap.end()) { - const auto& params = it->second; - // iterate or pass params to a function tracer + const auto& paramList = it->second; + // Use paramList with IFunctionTracer to configure eBPF probes } -``` - -## Notes - -- Depends on `ebpfpub/ifunctiontracer.h` from the `tob::ebpfpub` namespace. -- `kParameterListMap` is defined externally (in the corresponding `.cpp`); this header only declares it. -- Intended for use within osquery's eBPF event tracing subsystem to drive syscall parameter serialization. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/events/linux/bpf/.setrlimit.md b/osquery/events/linux/bpf/.setrlimit.md index 7af86cfb756..bb7f1bbbc0f 100644 --- a/osquery/events/linux/bpf/.setrlimit.md +++ b/osquery/events/linux/bpf/.setrlimit.md @@ -3,7 +3,7 @@ Declares a utility function for configuring BPF (Berkeley Packet Filter) memory ## Key Components -- **`configureBPFMemoryLimits()`** β€” Sets system resource limits (`rlimit`) required for BPF operations. Returns a `Status` object indicating success or failure. +- **`configureBPFMemoryLimits()`** β€” Sets system resource limits (`rlimit`) required for BPF operations. Returns an osquery `Status` object indicating success or failure. ## Usage Example @@ -13,15 +13,12 @@ Declares a utility function for configuring BPF (Berkeley Packet Filter) memory void initBPF() { osquery::Status status = osquery::configureBPFMemoryLimits(); if (!status.ok()) { - // Handle failure: status.getMessage() provides details - return; + // Handle failure: status.getMessage() provides error details } - // Proceed with BPF initialization } ``` ## Notes -- Part of the `osquery` namespace. -- Relies on `osquery::Status` from `osquery/utils/status/status.h` for error propagation. -- Typically called during BPF subsystem initialization to ensure adequate locked memory (`RLIMIT_MEMLOCK`) is available for eBPF map and program loading. \ No newline at end of file +- Returns `Status::success()` on success or a descriptive error `Status` on failure. +- Typically called during BPF subsystem initialization to ensure sufficient locked memory (`RLIMIT_MEMLOCK`) is available for eBPF map and program loading. \ No newline at end of file diff --git a/osquery/events/linux/bpf/.systemstatetracker.md b/osquery/events/linux/bpf/.systemstatetracker.md index 5c345dc8594..ee63c0f6cc2 100644 --- a/osquery/events/linux/bpf/.systemstatetracker.md +++ b/osquery/events/linux/bpf/.systemstatetracker.md @@ -1,52 +1,54 @@ -Concrete implementation of `ISystemStateTracker` that tracks Linux process and system state via BPF event hooks, maintaining per-process context maps and generating an event list from observed kernel activity. +Concrete implementation of `ISystemStateTracker` that tracks Linux process and file descriptor state using BPF event data within the osquery framework. ## Key Components -### Factory Methods -- `create()` β€” Instantiates tracker with a default `IProcessContextFactory` -- `create(IProcessContextFactory::Ref)` β€” Instantiates tracker with an injected factory (for testing) +### Class: `SystemStateTracker` + +**Factory Methods** +- `create()` β€” instantiates with a default `IProcessContextFactory` +- `create(IProcessContextFactory::Ref)` β€” instantiates with a custom factory (useful for testing) + +**Instance Methods (ISystemStateTracker overrides)** -### Instance Methods (virtual overrides) | Method | Description | |---|---| -| `restart()` | Reinitializes tracker state | -| `createProcess()` | Records a fork/clone event | -| `executeBinary()` | Records an exec event with argv | -| `setWorkingDirectory()` | Updates CWD by `dirfd` or path string | -| `openFile()` / `closeHandle()` / `duplicateHandle()` | Tracks file descriptor lifecycle | -| `createSocket()` / `bind()` / `listen()` / `connect()` / `accept()` | Tracks socket state transitions | -| `nameToHandleAt()` / `openByHandleAt()` | Tracks opaque file handle resolution | -| `eventList()` | Returns accumulated BPF events | - -### Internal Static Helpers -- `getProcessContext()` β€” Looks up or creates a `ProcessContext` for a given PID -- `expireProcessContexts()` β€” Prunes stale process entries -- `parseUnixSockaddr()` / `parseInetSockaddr()` / `parseInet6Sockaddr()` / `parseNetlinkSockaddr()` β€” Deserializes raw sockaddr bytes into address/port strings -- `parseSocketAddress()` β€” Dispatches to the appropriate sockaddr parser -- `saveFileHandle()` / `expireFileHandleEntries()` β€” Manages the file handle index/map - -### Key Data Structures -- `Context` β€” Holds the `ProcessContextMap`, accumulated `EventList`, and file handle index/map -- `FileHandleStruct` β€” Associates a handle with its originating `dfd`, name, and flags -- `PrivateData` β€” Opaque pImpl struct holding internal state +| `createProcess` | Records a new child process fork event | +| `executeBinary` | Tracks `execve`-style binary execution with argv | +| `setWorkingDirectory` | Updates a process's `cwd` by fd or path | +| `openFile` / `closeHandle` / `duplicateHandle` | Maintains per-process file descriptor table | +| `createSocket` / `bind` / `listen` / `connect` / `accept` | Tracks socket lifecycle and addressing | +| `nameToHandleAt` / `openByHandleAt` | Handles file-handle-based open operations | +| `eventList` | Returns accumulated BPF events for consumption | +| `restart` | Resets internal state | + +**Internal Structures** + +- `Context` β€” holds the `ProcessContextMap`, `EventList`, and file handle index/map +- `FileHandleStruct` β€” stores `dfd`, `name`, and `flags` for a pending file handle +- `PrivateData` β€” pimpl-pattern struct hiding implementation details + +**Static Helpers** +- `parseUnixSockaddr` / `parseInetSockaddr` / `parseInet6Sockaddr` / `parseNetlinkSockaddr` β€” parse raw sockaddr bytes into address/port strings +- `parseSocketAddress` β€” dispatches to the correct parser based on socket domain +- `expireProcessContexts` / `expireFileHandleEntries` β€” prune stale entries to bound memory growth ## Usage Example ```cpp -// Default construction +// Create tracker with default factory auto tracker = osquery::SystemStateTracker::create(); -// Process a fork event +// Handle a fork BPF event tracker->createProcess(event_header, parent_pid, child_pid); -// Track a binary execution -tracker->executeBinary(event_header, pid, AT_FDCWD, 0, "/usr/bin/ls", argv); +// Handle an execve BPF event +tracker->executeBinary(event_header, pid, AT_FDCWD, 0, + "/usr/bin/curl", argv); -// Retrieve accumulated events -auto events = tracker->eventList(); +// Handle socket connect +tracker->connect(event_header, pid, sockfd, raw_sockaddr_bytes); -// Inject a custom factory (e.g., in unit tests) -auto mock_factory = std::make_shared(); -auto tracker = osquery::SystemStateTracker::create(mock_factory); +// Drain events for processing +auto events = tracker->eventList(); ``` \ No newline at end of file diff --git a/osquery/events/tests/.mockedosquerydatabase.md b/osquery/events/tests/.mockedosquerydatabase.md index adf5afcc7e9..3d8d721bd3f 100644 --- a/osquery/events/tests/.mockedosquerydatabase.md +++ b/osquery/events/tests/.mockedosquerydatabase.md @@ -1,41 +1,48 @@ -In-memory mock implementation of `IDatabaseInterface` for use in osquery unit tests, providing a simple `std::map`-backed database without requiring a real storage backend. +A test utility header defining an in-memory mock implementation of the osquery database interface, used to isolate unit tests from real database I/O. ## Key Components -- **`MockedOsqueryDatabase`** β€” Final class implementing `IDatabaseInterface` with an in-memory `std::map key_map` as the backing store. -- **`generateEvents()`** β€” Helper to simulate event generation for a given publisher and event name during tests. -- **`getDatabaseValue()`** β€” Overloaded for `std::string` and `int` return types; reads from `key_map`. -- **`setDatabaseValue()`** β€” Overloaded for `std::string` and `int` values; writes to `key_map`. -- **`setDatabaseBatch()`** β€” Bulk insert of string key-value pairs into the mock store. -- **`deleteDatabaseValue()`** / **`deleteDatabaseRange()`** β€” Remove single or range-bounded entries from `key_map`. -- **`scanDatabaseKeys()`** β€” Overloaded to list keys within a domain, with optional prefix filtering and a result size limit. +**`MockedOsqueryDatabase`** β€” A `final` class implementing `IDatabaseInterface` backed by an in-memory `std::map`. + +| Member | Description | +|---|---| +| `key_map` | Mutable in-memory store mapping keys to string values | +| `generateEvents()` | Injects synthetic events for a given publisher/name pair | +| `getDatabaseValue()` | Reads a `string` or `int` value by domain + key | +| `setDatabaseValue()` | Writes a `string` or `int` value by domain + key | +| `setDatabaseBatch()` | Writes multiple string values in one call | +| `deleteDatabaseValue()` | Removes a single key from a domain | +| `deleteDatabaseRange()` | Removes all keys between `low` and `high` in a domain | +| `scanDatabaseKeys()` | Lists keys in a domain, optionally filtered by prefix and capped by `max` | ## Usage Example ```cpp #include "mockedosquerydatabase.h" -using namespace osquery; +TEST(MyFeatureTest, StoresAndRetrievesValue) { + osquery::MockedOsqueryDatabase db; -// Instantiate mock database for a unit test -MockedOsqueryDatabase db; + // Write a value + db.setDatabaseValue("config", "version", std::string("1.0")); -// Write a value -db.setDatabaseValue("config", "refresh_interval", std::string("60")); + // Read it back + std::string value; + auto status = db.getDatabaseValue("config", "version", value); -// Read it back -std::string value; -Status s = db.getDatabaseValue("config", "refresh_interval", value); -assert(s.ok()); -assert(value == "60"); + EXPECT_TRUE(status.ok()); + EXPECT_EQ(value, "1.0"); +} -// Scan keys in a domain -std::vector keys; -db.scanDatabaseKeys("config", keys, 100); +TEST(MyFeatureTest, SimulatesEventPublishing) { + osquery::MockedOsqueryDatabase db; + db.generateEvents("file_events", "file_created"); -// Simulate event generation -db.generateEvents("file_events", "file_created"); + std::vector keys; + db.scanDatabaseKeys("events", keys, /* max */ 100); + EXPECT_FALSE(keys.empty()); +} ``` -> **Note:** `key_map` is `mutable` to allow `const`-qualified read/write methods to modify the in-memory store β€” a deliberate design choice for test convenience. Not intended for production use. \ No newline at end of file +> **Note:** All state lives in `key_map` and is discarded when the object goes out of scope β€” no cleanup or teardown required between tests. \ No newline at end of file diff --git a/osquery/events/tests/linux/bpf/.mockedfilesystem.md b/osquery/events/tests/linux/bpf/.mockedfilesystem.md index 8f8daa9374c..aa8cc42e698 100644 --- a/osquery/events/tests/linux/bpf/.mockedfilesystem.md +++ b/osquery/events/tests/linux/bpf/.mockedfilesystem.md @@ -1,37 +1,40 @@ -A test double implementation of the `IFilesystem` interface that provides a mockable filesystem abstraction for unit testing BPF-related Linux event components without requiring real filesystem access. +Declares `MockedFilesystem`, a test double implementing the `IFilesystem` interface for use in unit tests that require controlled, in-memory filesystem behavior without real I/O operations. ## Key Components -### `MockedFilesystem` class - -Concrete final class inheriting from `IFilesystem`, overriding all virtual methods: +**`MockedFilesystem`** β€” Final class inheriting from `IFilesystem`, overriding all virtual filesystem methods: | Method | Description | |---|---| -| `open()` | Mocked file open by path with flags | -| `openAt()` | Mocked file open relative to a directory fd | -| `readLinkAt()` | Mocked symlink resolution relative to a directory fd | -| `read()` | Mocked buffered file read up to a max size | -| `enumFiles()` | Mocked directory file enumeration via callback | -| `fileExists()` | Mocked existence check relative to a directory fd | +| `open()` | Opens a file descriptor by path and flags | +| `openAt()` | Opens a file descriptor relative to a directory fd | +| `readLinkAt()` | Reads a symbolic link relative to a directory fd | +| `read()` | Reads file contents into a buffer up to `max_size` | +| `enumFiles()` | Enumerates files in a directory via callback | +| `fileExists()` | Checks for file existence relative to a directory fd | ## Usage Example ```cpp #include "mockedfilesystem.h" -// Inject MockedFilesystem into a component under test -osquery::MockedFilesystem mockFs; +using namespace osquery; + +// Instantiate the mock in a test fixture +MockedFilesystem fs; -// Use with a BPF event processor that depends on IFilesystem +// Use with a BPF event processor that accepts IFilesystem& tob::utils::UniqueFd fd; -bool success = mockFs.open(fd, "/proc/self/maps", O_RDONLY); +bool success = fs.open(fd, "/proc/self/status", O_RDONLY); + +// Read contents into a buffer +std::vector buffer; +fs.read(buffer, fd.get(), 4096); -// Typical usage: inject via IFilesystem interface pointer -std::unique_ptr fs = - std::make_unique(); -MyBpfComponent component(std::move(fs)); +// Check file existence relative to a directory fd +bool exists = false; +fs.fileExists(exists, AT_FDCWD, "some_file.txt"); ``` -> **Note:** Method bodies are defined in the accompanying `.cpp` file where mock behavior (e.g., via GoogleMock `MOCK_METHOD` macros or manual stubs) is implemented. This header only declares the override surface. \ No newline at end of file +> **Note:** Method implementations are defined separately (typically in a `.cpp` test fixture file), where test-specific behavior such as returning hardcoded data or simulating errors can be injected without touching the real filesystem. \ No newline at end of file diff --git a/osquery/events/tests/linux/bpf/.mockedprocesscontextfactory.md b/osquery/events/tests/linux/bpf/.mockedprocesscontextfactory.md index 24fb05c7b91..48321d9705b 100644 --- a/osquery/events/tests/linux/bpf/.mockedprocesscontextfactory.md +++ b/osquery/events/tests/linux/bpf/.mockedprocesscontextfactory.md @@ -1,42 +1,41 @@ -Provides a test double implementation of `IProcessContextFactory` for unit testing BPF process context capture logic in osquery's Linux event system. +Mock implementation of `IProcessContextFactory` for use in unit tests within the osquery Linux BPF event system. ## Key Components -| Member | Description | -|---|---| -| `MockedProcessContextFactory` | Concrete mock class implementing `IProcessContextFactory` | -| `invocationCount()` | Returns the number of times capture methods have been called | -| `failNextRequest()` | Configures the mock to return failure on the next capture call | -| `captureSingleProcess()` | Mock override; captures context for a single PID | -| `captureAllProcesses()` | Mock override; captures context for all processes | +- **`MockedProcessContextFactory`** β€” Concrete test double implementing `IProcessContextFactory`, providing controllable behavior for process context capture operations +- **`invocationCount()`** β€” Returns the number of times capture methods have been called, enabling call-count assertions in tests +- **`failNextRequest()`** β€” Arms the mock to return a failure on the next capture call, allowing error-path testing +- **`captureSingleProcess()`** β€” Mock override that captures context for a single process by PID +- **`captureAllProcesses()`** β€” Mock override that captures context for all running processes into a `ProcessContextMap` + +## Private State + +| Member | Type | Purpose | +|--------|------|---------| +| `fail_next_request` | `mutable bool` | Flags the next call to return failure | +| `invocation_count` | `mutable std::size_t` | Tracks total capture invocations | ## Usage Example -```c +```cpp #include "mockedprocesscontextfactory.h" -// Basic happy-path test +// Basic invocation tracking MockedProcessContextFactory factory; ProcessContext ctx; - -bool ok = factory.captureSingleProcess(ctx, 1234); -assert(ok); +factory.captureSingleProcess(ctx, 1234); assert(factory.invocationCount() == 1); -// Simulate capture failure +// Test failure handling in your component factory.failNextRequest(); -bool failed = factory.captureSingleProcess(ctx, 5678); -assert(!failed); +bool result = factory.captureSingleProcess(ctx, 1234); +assert(result == false); // next call returns failure -// Bulk capture +// Verify all-process capture ProcessContextMap process_map; factory.captureAllProcesses(process_map); -assert(factory.invocationCount() == 3); +assert(factory.invocationCount() == 2); ``` -## Notes - -- Both `fail_next_request` and `invocation_count` are `mutable`, allowing state tracking inside `const` method overrides. -- Marked `final` β€” not intended to be subclassed further. -- Depends on `IProcessContextFactory` from `osquery/events/linux/bpf/`. \ No newline at end of file +> **Note:** Both private fields are marked `mutable`, allowing state updates from `const`-qualified methods (`captureSingleProcess` and `captureAllProcesses` are `const` overrides). \ No newline at end of file diff --git a/osquery/events/tests/linux/bpf/.utils.md b/osquery/events/tests/linux/bpf/.utils.md index e03ac74cf11..b7e72707366 100644 --- a/osquery/events/tests/linux/bpf/.utils.md +++ b/osquery/events/tests/linux/bpf/.utils.md @@ -1,53 +1,39 @@ -Utility header for managing and validating file and socket descriptors within BPF process context tracking on Linux. +Utility header for managing and validating file and socket descriptors within BPF process contexts on Linux. Provides helpers used in osquery's BPF event subsystem to register and verify descriptor state for tracked processes. ## Key Components All functions reside in the `osquery` namespace and operate on either a single `ProcessContext` or a `ProcessContextMap` (keyed by `pid_t`). -### Set Functions - | Function | Description | |---|---| -| `setFileDescriptor` | Registers a file descriptor with its path and `close_on_exec` flag into a process context | +| `setFileDescriptor` | Registers a file descriptor (path + close-on-exec flag) into a process context | | `setSocketDescriptor` | Registers a socket descriptor with full network metadata (domain, type, protocol, local/remote address and port) | +| `validateFileDescriptor` | Verifies a file descriptor entry matches the expected path and flags | +| `validateSocketDescriptor` | Verifies a socket descriptor entry matches all expected network attributes | -Both setters are overloaded to accept either a direct `ProcessContext&` or a `ProcessContextMap&` + `pid_t` pair. - -### Validate Functions - -| Function | Description | -|---|---| -| `validateFileDescriptor` | Returns `true` if the given fd matches the expected path and `close_on_exec` state in the context | -| `validateSocketDescriptor` | Returns `true` if the given fd matches all expected socket properties in the context | - -Same overload pattern as the setters. +Each function is overloaded twice β€” once accepting a direct `ProcessContext&` reference, and once accepting a `ProcessContextMap&` plus a `pid_t` to look up the target process. ## Usage Example ```cpp #include "utils.h" +// Register a file descriptor for a tracked process osquery::ProcessContext ctx; -osquery::ProcessContextMap ctx_map; - -// Register a file descriptor on a single context -osquery::setFileDescriptor(ctx, 3, false, "/etc/passwd"); - -// Register a socket on a context map entry for pid 1234 -osquery::setSocketDescriptor(ctx_map, 1234, 5, false, - AF_INET, SOCK_STREAM, IPPROTO_TCP, - "127.0.0.1", 8080, - "10.0.0.1", 443); +osquery::setFileDescriptor(ctx, /*fd=*/3, /*close_on_exec=*/true, "/etc/passwd"); -// Validate the file descriptor -bool ok = osquery::validateFileDescriptor(ctx, 3, false, "/etc/passwd"); +// Validate it was recorded correctly +bool ok = osquery::validateFileDescriptor(ctx, 3, true, "/etc/passwd"); -// Validate the socket via map -bool valid = osquery::validateSocketDescriptor(ctx_map, 1234, 5, false, - AF_INET, SOCK_STREAM, IPPROTO_TCP, - "127.0.0.1", 8080, - "10.0.0.1", 443); -``` - -> Primarily used in BPF event processing tests to populate and assert process context state without requiring live kernel events. \ No newline at end of file +// Register a socket descriptor via process map +osquery::ProcessContextMap ctx_map; +pid_t pid = 1234; +osquery::setSocketDescriptor(ctx_map, pid, /*fd=*/5, false, + AF_INET, SOCK_STREAM, IPPROTO_TCP, + "127.0.0.1", 8080, "10.0.0.1", 443); + +bool valid = osquery::validateSocketDescriptor(ctx_map, pid, 5, false, + AF_INET, SOCK_STREAM, IPPROTO_TCP, + "127.0.0.1", 8080, "10.0.0.1", 443); +``` \ No newline at end of file diff --git a/osquery/events/windows/.evtsubscription.md b/osquery/events/windows/.evtsubscription.md index 524810ccc31..da3cd6f94ae 100644 --- a/osquery/events/windows/.evtsubscription.md +++ b/osquery/events/windows/.evtsubscription.md @@ -1,17 +1,25 @@ -Defines the `EvtSubscription` class and its Windows Event Log callback dispatcher for subscribing to and collecting Windows Event Log entries within osquery. +Windows Event Log subscription wrapper that provides a C++ RAII interface for subscribing to and collecting Windows Event Log channel events via the Win32 `EvtSubscribe` API. ## Key Components -- **`EvtSubscriptionCallbackDispatcher`** β€” Windows callback function (`DWORD WINAPI`) invoked by the WinEvt API on new events; dispatches to the appropriate `EvtSubscription` instance via the `context` pointer. -- **`EvtSubscription`** β€” RAII-style, non-copyable class managing a Windows Event Log channel subscription. - - `Ref` β€” `std::unique_ptr` alias for ownership semantics. - - `Event` / `EventList` β€” Type aliases for `std::wstring` and `std::vector` representing collected events. - - `create(Ref&, channel)` β€” Static factory method; constructs a subscription for the given channel name, returning an osquery `Status`. - - `channel()` β€” Returns the subscribed channel name. - - `getEvents()` β€” Drains and returns all buffered events collected since the last call. - - `processEvent(EVT_HANDLE)` β€” Private handler called by the dispatcher to render and buffer an incoming event. - - `PrivateData` β€” Pimpl struct hiding WinEvt handles and internal state. +### `EvtSubscriptionCallbackDispatcher` +Free function serving as the Win32 callback entry point (`EVT_SUBSCRIBE_CALLBACK`). Dispatches raw `EVT_HANDLE` events to the owning `EvtSubscription` instance. Declared as a `friend` to access private members. + +### `EvtSubscription` +Non-copyable RAII class managing a Windows Event Log subscription. + +| Member | Description | +|---|---| +| `Ref` | `std::unique_ptr` β€” ownership handle | +| `Event` | `std::wstring` β€” rendered XML event string | +| `EventList` | `std::vector` β€” batch of collected events | +| `create(Ref&, channel)` | Factory method; returns `Status`, populates `Ref` on success | +| `channel()` | Returns the subscribed channel name (e.g. `"System"`) | +| `getEvents()` | Drains and returns all buffered events since last call | +| `~EvtSubscription()` | Closes the `EVT_HANDLE` and cleans up subscription resources | +| `processEvent(EVT_HANDLE)` | Private; renders and buffers a single incoming event | +| `PrivateData` | Pimpl struct hiding Win32 handles and the event buffer | ## Usage Example @@ -19,18 +27,19 @@ Defines the `EvtSubscription` class and its Windows Event Log callback dispatche #include "evtsubscription.h" osquery::EvtSubscription::Ref sub; -auto status = osquery::EvtSubscription::create(sub, "System"); +auto status = osquery::EvtSubscription::create(sub, "System"); if (!status.ok()) { - LOG(ERROR) << "Subscription failed: " << status.getMessage(); + LOG(ERROR) << "Failed to subscribe: " << status.getMessage(); return; } -// Later, poll for collected events -auto events = sub->getEvents(); -for (const auto& event : events) { - // process wide-string XML event data +// Poll for new events +osquery::EvtSubscription::EventList events = sub->getEvents(); +for (const auto& evt : events) { + // evt is a std::wstring containing the rendered XML event + processXmlEvent(evt); } ``` -> **Platform note:** This header is Windows-only. It depends on `` and `` and must only be compiled in Windows build targets. \ No newline at end of file +> **Platform:** Windows only β€” requires linking against `wevtapi.lib`. The pimpl (`PrivateData`) pattern keeps Win32 types out of consumer headers. \ No newline at end of file diff --git a/osquery/events/windows/.ntfs_event_publisher.md b/osquery/events/windows/.ntfs_event_publisher.md index bc37d986214..5da9667fb5d 100644 --- a/osquery/events/windows/.ntfs_event_publisher.md +++ b/osquery/events/windows/.ntfs_event_publisher.md @@ -1,54 +1,61 @@ -Declares the NTFS event publisher for osquery on Windows, which consumes raw USN journal records and emits structured file change events to subscribers. +Declares the NTFS event publisher for osquery on Windows, enabling file system change monitoring by consuming raw USN (Update Sequence Number) journal records and emitting structured file change events to subscribers. ## Key Components ### Structs | Name | Purpose | -|---|---| -| `NTFSEventSubscriptionContext` | Subscriber filter: paths/FRNs for write-only or always-include (read+write) monitoring | -| `NTFSEventRecord` | Single normalized NTFS event with path, type, timestamps, FRNs, and drive letter | -| `NTFSEventContext` | Batch event context holding a `vector` emitted to subscribers | -| `VolumeData` | Holds volume/root folder HANDLE and root FRN for a monitored drive | -| `USNJournalReaderInstance` | Tracks a running reader service, its shared context, parent FRN cache, and rename merge map | +|------|---------| +| `NTFSEventSubscriptionContext` | Holds subscriber-defined path filters (write-only paths vs. always-monitored access paths) with FRN and string-based sets | +| `NTFSEventRecord` | Represents a single processed NTFS event including path, old path (renames), timestamps, attributes, and FRN metadata | +| `NTFSEventContext` | Batched event context carrying a `vector` dispatched to subscribers | +| `VolumeData` | Wraps volume and root folder handles alongside the root FRN for journal tree traversal | +| `USNJournalReaderInstance` | Tracks a running reader service, its shared context, a parent FRN cache, and a rename-merge map | -### Type Aliases +### Types / Aliases -- `NTFSEventPublisherConfiguration` β€” `unordered_set` of drive letters to monitor -- `ParentFRNCache` β€” `unordered_map` for resolving parent directory paths +- `NTFSEventSubscriptionContextRef` β€” `shared_ptr` to subscription context +- `NTFSEventContextRef` β€” `shared_ptr` to event context +- `ParentFRNCache` β€” `unordered_map` for FRN-to-directory resolution +- `NTFSEventPublisherConfiguration` β€” `unordered_set` of monitored drive letters ### Class: `NTFSEventPublisher` -Extends `EventPublisher`. Core lifecycle methods: +Inherits `EventPublisher`. Key methods: | Method | Role | -|---|---| -| `setUp()` | One-time initialization | -| `configure()` | Reads drive list from config; called on reload | -| `run()` | Main loop: acquires journal records, resolves paths, fires events | -| `tearDown()` | Releases handles and cleans up resources | +|--------|------| +| `setUp()` | Initializes publisher resources | +| `configure()` | Reads drive configuration on start or config reload | +| `run()` | Main loop β€” acquires USN journal records and fires events | +| `tearDown()` | Releases handles and cleans up | +| `getPathFromReferenceNumber()` | Resolves a full path from a file reference number | +| `getPathFromParentFRN()` | Resolves path using parent FRN + basename | +| `restartJournalReaderServices()` | Spawns/restarts `USNJournalReader` instances per active drive | ### Free Function -- `GetNativeFileIdFromUSNReference()` β€” Converts a `USNFileReferenceNumber` to a `FILE_ID_DESCRIPTOR` for use with the `OpenFileById` Win32 API +`GetNativeFileIdFromUSNReference()` β€” Converts a `USNFileReferenceNumber` to a `FILE_ID_DESCRIPTOR` for use with the `OpenFileById` Windows API. ## Usage Example ```cpp -// Subscribing to NTFS write events on C:\Windows -auto sc = std::make_shared(); -sc->category = "system_files"; -sc->write_paths.insert("C:\\Windows\\System32"); - -// The publisher resolves FRNs to full paths internally -// and emits NTFSEventContext batches containing NTFSEventRecord entries -// Subscribers inspect each record's type, path, and old_path (renames) -for (const auto& record : context->event_list) { - if (record.type == USNJournalEventRecord::Type::FileWrite) { - LOG(INFO) << "Modified: " << record.path; +// Subscribing to NTFS write events on C:\sensitive +auto sc = NTFSEventPublisher::createSubscriptionContext(); +sc->write_paths.insert("C:\\sensitive\\config.db"); + +// Subscriber callback receives NTFSEventContext batches +auto callback = [](const NTFSEventContextRef& ec, + const NTFSEventSubscriptionContextRef& sc) { + for (const auto& record : ec->event_list) { + LOG(INFO) << "Change detected: " << record.path + << " drive: " << record.drive_letter + << " partial: " << record.partial; } -} +}; + +subscribe(callback, sc); ``` -> **Note:** This publisher is Windows-only and depends on the USN change journal. Partial events (`record.partial == true`) indicate only a filename was resolved, not a full path. \ No newline at end of file +> **Windows only.** This publisher relies on the Windows USN Change Journal and Win32 volume handle APIs. It has no effect on non-Windows builds. \ No newline at end of file diff --git a/osquery/events/windows/.usn_journal_reader.md b/osquery/events/windows/.usn_journal_reader.md index 144bfdc2d1e..7d4eddce159 100644 --- a/osquery/events/windows/.usn_journal_reader.md +++ b/osquery/events/windows/.usn_journal_reader.md @@ -1,52 +1,55 @@ -Windows NTFS USN (Update Sequence Number) Journal reader interface for osquery's file change event monitoring on Windows. +Windows NTFS USN (Update Sequence Number) Journal reader interface for osquery, providing types, structures, and service class definitions for asynchronously monitoring file system change events on Windows volumes. ## Key Components ### Structs -| Name | Description | -|------|-------------| -| `USNFileReferenceNumber` | Uniquely identifies a file within a volume; supports 64-bit, 128-bit (`FILE_ID_128`), and GUID formats | -| `USNJournalEventRecord` | Represents a single file system change event, including type, path, timestamps, and attributes | -| `USNJournalReaderContext` | Shared state between `USNJournalReader` service and `FileChangePublisher`; thread-safe via mutex and condition variable | +- **`USNFileReferenceNumber`** β€” Wraps NTFS file reference numbers (64-bit `DWORDLONG`, 128-bit `FILE_ID_128`, or `GUID`) into a portable byte vector; supports comparison, hashing, and string serialization +- **`USNJournalEventRecord`** β€” Represents a single parsed file system event, including type, drive letter, timestamps, attributes, file name, and parent/node reference numbers; defines 32 event types (creation, deletion, rename, write, etc.) +- **`USNJournalReaderContext`** β€” Thread-safe shared state between the reader service and `FileChangePublisher`, holding the processed record queue, mutex, condition variable, and a termination flag -### Classes +### Class -- **`USNJournalReader`** β€” Background service thread (`InternalRunnable`) that asynchronously reads USN journal records from a volume, decompresses compound events, and dispatches them to the publisher. +- **`USNJournalReader`** β€” An `InternalRunnable` (osquery service thread) that asynchronously reads USN journal records from a volume. Key static methods: + - `DecompressRecord()` β€” Expands a compound USN reason bitmask into discrete `USNJournalEventRecord` events + - `ProcessAndAppendUSNRecord()` β€” Parses a raw `USN_RECORD*` and appends typed records to an output vector ### Namespace `USNParsers` -Utility functions for extracting individual fields from raw `USN_RECORD` pointers: +Stateless parsing helpers that extract individual fields from a raw `USN_RECORD*`: `GetUpdateSequenceNumber`, `GetFileReferenceNumber`, `GetParentFileReferenceNumber`, `GetTimeStamp`, `GetAttributes`, `GetReason`, `GetEventType`, `GetEventString` -- `GetUpdateSequenceNumber`, `GetFileReferenceNumber`, `GetParentFileReferenceNumber` -- `GetTimeStamp`, `GetAttributes`, `GetReason`, `GetEventType`, `GetEventString` +### Utility -### Key Type Aliases - -- `USNPerFileLastRecordType` β€” `std::map` used to deduplicate event types per file reference -- `USNJournalReaderRef` / `USNJournalReaderContextRef` β€” `shared_ptr` wrappers for shared ownership +- `kWindowsFileAttributeMap` / `kNTFSEventToStringMap` β€” Lookup maps for human-readable attribute and event descriptions +- `GetNativeFileIdFromUSNReference()` β€” Converts `USNFileReferenceNumber` to Windows `FILE_ID_DESCRIPTOR` +- `std::hash` specialization for `USNFileReferenceNumber` via `boost::hash_range` ## Usage Example ```cpp -// Create a shared context for the reader and publisher -auto context = std::make_shared(); -context->drive_letter = 'C'; +// Create shared context for a volume mounted at drive 'C' +auto ctx = std::make_shared(); +ctx->drive_letter = 'C'; -// Instantiate and start the reader service -auto reader = std::make_shared(context); -reader->start(); +// Launch the reader service +auto reader = std::make_shared(ctx); +Dispatcher::addService(reader); -// Consume processed records (publisher side) +// Consumer side: wait for records and drain the queue { - std::lock_guard lock(context->processed_records_mutex); - for (const auto& record : context->processed_record_list) { - std::cout << record << "\n"; // uses operator<< overload + std::unique_lock lock(ctx->processed_records_mutex); + ctx->processed_records_cv.wait(lock, [&] { + return !ctx->processed_record_list.empty() || ctx->terminate.load(); + }); + + for (const auto& record : ctx->processed_record_list) { + // e.g. log: record.drive_letter, record.name, record.type + std::cout << record << "\n"; } - context->processed_record_list.clear(); + ctx->processed_record_list.clear(); } // Signal shutdown -context->terminate.store(true); +ctx->terminate = true; ``` \ No newline at end of file diff --git a/osquery/events/windows/.windowseventlogparser.md b/osquery/events/windows/.windowseventlogparser.md index 9c48abcf6f1..db1e1744a93 100644 --- a/osquery/events/windows/.windowseventlogparser.md +++ b/osquery/events/windows/.windowseventlogparser.md @@ -1,57 +1,52 @@ -Windows Event Log parser interface for osquery, providing data structures and parsing utilities to convert raw Windows Event Log XML into structured C++ objects. +Header file defining data structures and parsing utilities for Windows Event Log entries within the osquery framework. ## Key Components ### `WELEvent` Struct -A plain data container representing a parsed Windows Event Log entry with the following fields: +A plain data structure representing a parsed Windows Event Log entry with the following fields: | Field | Type | Description | -|---|---|---| +|-------|------|-------------| | `osquery_time` | `std::time_t` | Timestamp when osquery captured the event | -| `datetime` | `std::string` | Human-readable event datetime | +| `datetime` | `std::string` | Human-readable event datetime string | | `source` | `std::string` | Event log source channel | -| `provider_name` | `std::string` | Publisher/provider name | -| `provider_guid` | `std::string` | Provider GUID | +| `provider_name` | `std::string` | Name of the event provider | +| `provider_guid` | `std::string` | GUID of the event provider | | `computer_name` | `std::string` | Originating machine name | | `event_id` | `int64_t` | Windows Event ID | -| `task_id` | `int64_t` | Task category ID | +| `task_id` | `int64_t` | Associated task category ID | | `level` | `int64_t` | Severity level | | `pid` / `tid` | `int64_t` | Process and thread IDs | -| `keywords` | `std::string` | Event keywords bitmask string | -| `data` | `std::string` | Raw event data payload | +| `keywords` | `std::string` | Event keyword flags | +| `data` | `std::string` | Raw event payload/data | -### `parseWindowsEventLogXML()` -Parses a raw Windows Event Log XML string (`std::wstring`) into a Boost `property_tree` object for further processing. +### Functions -### `parseWindowsEventLogPTree()` -Converts a populated Boost `property_tree` into a `WELEvent` struct, extracting all structured fields. +**`parseWindowsEventLogXML`** β€” Parses a raw Windows Event Log XML string (`std::wstring`) into a Boost `property_tree` object for further processing. + +**`parseWindowsEventLogPTree`** β€” Converts a populated Boost `property_tree` into a `WELEvent` struct, extracting all structured fields. ## Usage Example ```cpp #include "windowseventlogparser.h" -// Step 1: Parse raw XML into a property tree -boost::property_tree::ptree event_object; -std::wstring raw_xml = /* ... raw WEL XML ... */; - -osquery::Status parse_status = - osquery::parseWindowsEventLogXML(event_object, raw_xml); +osquery::WELEvent event; +boost::property_tree::ptree event_tree; -if (!parse_status.ok()) { - // handle error +// Step 1: Parse raw XML from Windows Event Log API +osquery::Status s = osquery::parseWindowsEventLogXML(event_tree, xml_event_string); +if (!s.ok()) { + // handle parse error } -// Step 2: Populate a WELEvent struct from the property tree -osquery::WELEvent wel_event; -osquery::Status map_status = - osquery::parseWindowsEventLogPTree(wel_event, event_object); - -if (map_status.ok()) { - // Access structured fields - std::cout << wel_event.provider_name << " - ID: " << wel_event.event_id; +// Step 2: Populate WELEvent struct from the property tree +s = osquery::parseWindowsEventLogPTree(event, event_tree); +if (s.ok()) { + std::cout << "Event ID: " << event.event_id + << " from: " << event.computer_name << "\n"; } ``` -The two-step design separates XML deserialization from field extraction, allowing either stage to be used independently or tested in isolation. \ No newline at end of file +The two-step design allows callers to inspect or manipulate the intermediate `property_tree` before committing to a `WELEvent`. \ No newline at end of file diff --git a/osquery/events/windows/.windowseventlogparserservice.md b/osquery/events/windows/.windowseventlogparserservice.md index 03bf850b8cd..e20407d605e 100644 --- a/osquery/events/windows/.windowseventlogparserservice.md +++ b/osquery/events/windows/.windowseventlogparserservice.md @@ -1,50 +1,44 @@ -Parses raw Windows Event Log subscription data into structured property trees for consumption by osquery event publishers. +Background service that parses raw Windows Event Log subscription data into structured property trees, organized by event channel. ## Key Components -### Class: `WindowsEventLogParserService` - -Extends `InternalRunnable` (osquery dispatcher thread). Runs as a background service that deserializes Windows Event Log entries from active `EvtSubscription` callbacks. +**`WindowsEventLogParserService`** β€” Inherits from `InternalRunnable`; runs as a background dispatcher thread within the osquery framework. **Type Aliases** - -| Type | Definition | -|---|---| -| `PropertyTreeList` | `std::vector` | -| `ChannelEventObjects` | `std::unordered_map` | +- `PropertyTreeList` β€” `std::vector`: a list of parsed event objects +- `ChannelEventObjects` β€” `std::unordered_map`: parsed events keyed by channel name **Public Methods** | Method | Description | -|---|---| -| `start()` | Begins the parser service loop (called by dispatcher) | -| `stop()` | Signals the service loop to terminate | -| `addEventList(channel, event_list)` | Queues raw `EvtSubscription::EventList` entries for a named channel | -| `getChannelEventObjects()` | Returns parsed property trees grouped by channel name, draining the internal buffer | +|--------|-------------| +| `start()` / `stop()` | Lifecycle controls inherited from `InternalRunnable` | +| `addEventList(channel, event_list)` | Enqueues raw `EvtSubscription::EventList` data for a given channel | +| `getChannelEventObjects()` | Returns all accumulated parsed event objects, grouped by channel | -**Implementation Detail** - -Uses the PIMPL idiom (`struct PrivateData` + `std::unique_ptr d_`) to hide internal state (likely mutex, event queue, and parse buffers) from the header. +**`PrivateData` (PIMPL)** β€” Internal implementation details hidden via `std::unique_ptr` to minimize header dependencies and ABI exposure. ## Usage Example ```cpp -// Instantiated and managed by the osquery dispatcher -auto parser = std::make_shared(); +#include + +// Instantiate and register with the dispatcher +auto parser = std::make_shared(); -// From a subscription callback, queue raw events for a channel -parser->addEventList("Security", std::move(event_list)); +// Feed raw events from a subscription callback +parser->addEventList("Security", subscription.drainEvents()); -// From the publisher, drain parsed results -WindowsEventLogParserService::ChannelEventObjects parsed = - parser->getChannelEventObjects(); +// Retrieve structured results after processing +auto channelEvents = parser->getChannelEventObjects(); -for (const auto& [channel, trees] : parsed) { - for (const auto& tree : trees) { - // Process each boost::property_tree representing one event record +for (const auto& [channel, treeList] : channelEvents) { + for (const auto& tree : treeList) { + // Access parsed event fields via Boost property_tree + auto eventId = tree.get("Event.System.EventID", ""); } } ``` -> **Note:** `addEventList` and `getChannelEventObjects` are intended to be called from different threads β€” the subscription callback thread writes, while the publisher thread reads. Internal synchronization is handled inside `PrivateData`. \ No newline at end of file +> **Note:** `addEventList` and `getChannelEventObjects` are intended to be called from the event subscription thread and consumer thread respectively. Thread safety is handled internally via the `PrivateData` PIMPL. \ No newline at end of file diff --git a/osquery/events/windows/.windowseventlogpublisher.md b/osquery/events/windows/.windowseventlogpublisher.md index e9c5eeae94a..b776a46df76 100644 --- a/osquery/events/windows/.windowseventlogpublisher.md +++ b/osquery/events/windows/.windowseventlogpublisher.md @@ -1,49 +1,53 @@ -Windows Event Log event publisher interface for osquery, providing subscription-based access to Windows Event Log channels with channel filtering and character frequency analysis for event matching. +Windows Event Log event publisher for osquery, providing subscription-based access to Windows Event Log channels with optional cosine similarity filtering for event content analysis. ## Key Components ### Structs -| Name | Description | -|------|-------------| -| `WindowsEventLogSubscriptionContext` | Holds subscriber configuration β€” a set of channel names to monitor and a character frequency map used for cosine similarity filtering | -| `WindowsEventLogEC` | Event context payload carrying the source channel name and a list of parsed property tree event objects | +- **`WindowsEventLogSubscriptionContext`** β€” Subscription context holding a set of target channel names (`channel_list`) and an optional character frequency map (`character_frequency_map`) used for similarity-based filtering. +- **`WindowsEventLogEC`** (Event Context) β€” Carries the source `channel` name and a parsed `event_objects` list (`PropertyTreeList`) for each fired event. ### Type Aliases -| Alias | Underlying Type | -|-------|----------------| -| `WindowsEventLogECRef` | `std::shared_ptr` | -| `WindowsEventLogSCRef` | `std::shared_ptr` | +- **`WindowsEventLogECRef`** β€” `shared_ptr` +- **`WindowsEventLogSCRef`** β€” `shared_ptr` ### Class: `WindowsEventLogPublisher` -Extends `EventPublisher`. +Inherits from `EventPublisher`. | Method | Description | -|--------|-------------| -| `shouldFire()` | Determines if an event should be dispatched to a given subscriber based on channel and frequency matching | -| `configure()` | Sets up event log channel subscriptions | -| `tearDown()` | Cleans up active subscriptions and resources | -| `run()` | Main loop β€” polls/receives Windows Event Log entries and fires events | -| `cosineSimilarity()` | Static utility computing cosine similarity between a buffer's character frequency and a reference frequency map | +|---|---| +| `configure()` | Sets up or reconfigures event channel subscriptions | +| `run()` | Main event loop; polls and dispatches Windows Event Log entries | +| `tearDown()` | Cleans up handles and stops the publisher | +| `shouldFire()` | Filters events against a subscription's channel list and frequency map | +| `cosineSimilarity()` | Static utility; computes cosine similarity between a string buffer and a global character frequency vector for anomaly detection | + +Uses a `PrivateData` (PIMPL) struct to hide platform-specific implementation details. ## Usage Example ```cpp -// Define a subscriber with specific channels to monitor +// Define a subscription context targeting specific channels auto sc = std::make_shared(); -sc->channel_list.insert("Security"); -sc->channel_list.insert("System"); - -// Compute similarity between an event buffer and known frequency baseline -std::vector baseline = loadBaselineFrequencies(); -double score = WindowsEventLogPublisher::cosineSimilarity(rawEventBuffer, baseline); - -if (score > threshold) { - // Process suspicious event +sc->channel_list = {"Security", "System", "Application"}; + +// Optionally set a character frequency map for cosine similarity filtering +sc->character_frequency_map = getBaselineFrequencies(); + +// Subscribe to the publisher +EventFactory::addSubscription( + "WindowsEventLogPublisher", + Subscription::create("my_subscriber", sc, eventCallback) +); + +// Standalone similarity check +double score = WindowsEventLogPublisher::cosineSimilarity( + rawEventBuffer, globalFrequencyVector +); +if (score < threshold) { + // Flag as anomalous } -``` - -> **Note:** The `PrivateData` pimpl pattern hides platform-specific WinAPI subscription handles, keeping the public interface clean and ABI-stable. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_controller.md b/osquery/events/windows/etw/.etw_controller.md index a386ff138e5..c85b5c60f45 100644 --- a/osquery/events/windows/etw/.etw_controller.md +++ b/osquery/events/windows/etw/.etw_controller.md @@ -1,49 +1,43 @@ -Singleton controller class for managing Windows Event Tracing (ETW) sessions and event processing pipelines within the osquery framework. +Manages ETW (Event Tracing for Windows) session lifecycle and event processing pipeline within the osquery framework, providing a singleton interface to subscribe to both userspace and kernel ETW providers. ## Key Components | Member | Type | Description | -|---|---|---| +|--------|------|-------------| | `instance()` | Static method | Returns the singleton `EtwController` instance | -| `addProvider()` | Public method | Registers an ETW provider with pre/post-processor callbacks | +| `addProvider()` | Public method | Registers an ETW provider config with pre/post-processor callbacks | | `dispatchETWEvents()` | Public method | Routes captured ETW events into the processing pipeline | -| `startProcessing()` | Private method | Starts ETW session listening and event processing | -| `initialize()` | Private method | Bootstraps ETW sessions and the post-processing engine | - -**Internal Sessions:** - -| Field | Purpose | -|---|---| -| `etwUserSession_` | Manages userspace ETW provider session | -| `etwKernelSession_` | Manages kernel ETW provider session | -| `etwPostProcessingEngine_` | Runs post-processor callbacks on captured events | -| `concurrentQueue_` | Thread-safe event queue bridging capture and processing | +| `startProcessing()` | Private method | Initializes and starts ETW listening sessions | +| `initialize()` | Private method | Sets up ETW sessions and post-processing engine | +| `etwUserSession_` | `UserEtwSessionRunnable` | Handles userspace ETW provider sessions | +| `etwKernelSession_` | `KernelEtwSessionRunnable` | Handles kernel ETW provider sessions | +| `etwPostProcessingEngine_` | `EtwPostProcessorsRunnable` | Runs registered post-processor callbacks | +| `concurrentQueue_` | `ConcurrentEventQueue` | Thread-safe event dispatch queue | ## Usage Example -```cpp -#include - -// Retrieve the singleton instance +```c +// Obtain the singleton controller instance EtwController& controller = EtwController::instance(); -// Build a provider configuration +// Build and register an ETW provider configuration EtwProviderConfig config; // ... populate config with provider GUID, pre/post-processor callbacks ... -// Register the provider β€” starts sessions if not already running Status status = controller.addProvider(config); if (!status.ok()) { - LOG(ERROR) << "Failed to add ETW provider: " << status.getMessage(); + // Handle registration failure } -// Events are automatically dispatched internally via: -// controller.dispatchETWEvents(eventDataRef); +// Dispatch an event from a preprocessor callback into the pipeline +EtwEventDataRef eventData = /* captured ETW event */; +controller.dispatchETWEvents(eventData); ``` ## Notes -- Inherits `boost::noncopyable` β€” copy and assignment are explicitly disabled. -- Thread-safety is enforced via `std::mutex` and `std::atomic initialized_`. -- Session names are fixed: `OsqueryUserETWSession`, `OsqueryKernelETWSession`, and `EtwPostProcessingEngine`. \ No newline at end of file +- Inherits `boost::noncopyable` β€” copy and move construction are disabled by design +- Thread safety is enforced via `std::mutex` on shared state +- Lazy initialization via `initialized_` atomic flag prevents redundant setup +- Session names (`OsqueryUserETWSession`, `OsqueryKernelETWSession`) are fixed string constants used to identify active ETW trace sessions \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_kernel_session.md b/osquery/events/windows/etw/.etw_kernel_session.md index 26982a77877..0d60a8631fe 100644 --- a/osquery/events/windows/etw/.etw_kernel_session.md +++ b/osquery/events/windows/etw/.etw_kernel_session.md @@ -1,42 +1,45 @@ -Manages the ETW (Event Tracing for Windows) kernel-space trace session as a dedicated runnable thread within osquery, wrapping the KrabsETW library to listen for and process kernel-level ETW events. +Manages the ETW (Event Tracing for Windows) kernel-space trace session as a dedicated thread using osquery's `InternalRunnable` abstraction, enabling collection of kernel-level ETW events via the KrabsETW library. ## Key Components -### Class: `KernelEtwSessionRunnable` -Extends `InternalRunnable` to run a kernel ETW trace session on a dedicated thread. +### `KernelEtwSessionRunnable` (class) +A final class extending `InternalRunnable` that owns and controls a single kernel ETW trace session. | Member | Type | Description | |--------|------|-------------| -| `addProvider()` | `Status` | Registers a kernel ETW provider with pre/post-processor callbacks | -| `start()` | `void` | Starts the InternalRunnable thread | -| `stop()` | `void` | Stops the InternalRunnable thread | -| `pause()` / `resume()` | `void` | Suspends or resumes event listening | -| `initKernelTraceSession()` | `void` | Best-effort initialization of the kernel trace session | -| `stopKernelTraceSession()` | `void` | Best-effort teardown of the kernel trace session | -| `kernelTraceSession_` | `shared_ptr` | Underlying KrabsETW kernel trace object | -| `runningProviders_` | `KernelProvidersCollection` | Cache of active kernel-space providers | +| `KernelEtwSessionRunnable()` | Constructor | Initializes the runnable with a named thread | +| `addProvider()` | Public method | Registers a kernel ETW provider with its pre/post-processor callbacks | +| `start()` / `stop()` | Protected overrides | Lifecycle hooks for the managed thread | +| `pause()` / `resume()` | Private methods | Suspends or continues event listening | +| `initKernelTraceSession()` | Private helper | Best-effort initialization of the KrabsETW kernel trace | +| `stopKernelTraceSession()` | Private helper | Best-effort teardown of the kernel trace session | +| `kernelTraceSession_` | `shared_ptr` | Underlying KrabsETW session object | +| `runningProviders_` | `vector>` | Cache of active kernel providers | +| `endTraceSession_` / `traceSessionStopped_` | `atomic` | Thread-safe stop/status flags | +| `condition_` | `condition_variable` | Signals session readiness to resume | ## Usage Example -```c -// Instantiate and start the kernel ETW session runnable -auto kernelSession = std::make_shared("KernelEtwSession"); +```cpp +#include "etw_kernel_session.h" +#include -// Register a kernel provider with its event config -EtwProviderConfig providerConfig = /* ... */; -Status status = kernelSession->addProvider(providerConfig); +// Create and start the kernel ETW session thread +auto session = std::make_shared( + "kernel_etw_session" +); +// Register a kernel provider (e.g., process events) +osquery::EtwProviderConfig config; +// ... configure provider, pre/post-processor callbacks ... +auto status = session->addProvider(config); if (!status.ok()) { - LOG(ERROR) << "Failed to add ETW kernel provider: " << status.getMessage(); + LOG(ERROR) << "Failed to add kernel ETW provider: " << status.getMessage(); } -// Thread lifecycle is managed via InternalRunnable (Dispatcher) -Dispatcher::addService(kernelSession); +// Start the dedicated trace thread +osquery::Dispatcher::addService(session); ``` -## Thread Safety - -- All shared state is protected by `std::mutex mutex_` -- Session lifecycle flags use `std::atomic` (`endTraceSession_`, `traceSessionStopped_`) -- A `std::condition_variable` coordinates pause/resume signaling \ No newline at end of file +> **Note:** Only one `KernelEtwSessionRunnable` instance should be active at a time. Thread safety is enforced internally via `std::mutex`. \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_post_processing_pipeline.md b/osquery/events/windows/etw/.etw_post_processing_pipeline.md index 2b4f5be28aa..d41cdb4ac77 100644 --- a/osquery/events/windows/etw/.etw_post_processing_pipeline.md +++ b/osquery/events/windows/etw/.etw_post_processing_pipeline.md @@ -1,44 +1,44 @@ -Manages the ETW (Event Tracing for Windows) post-processing pipeline by collecting and dispatching ETW events to subscribers via registered callback handlers. +Manages the ETW (Event Tracing for Windows) post-processing pipeline, responsible for consuming events from a shared concurrent queue and dispatching them to registered event subscribers via configurable callbacks. ## Key Components -### `EtwPostProcessorsRunnable` -A thread-safe runnable class (extending `InternalRunnable`) that processes ETW events from a shared concurrent queue and dispatches them to registered event subscribers. +### `EtwPostProcessorsRunnable` (class) +Extends `InternalRunnable` to run as a managed background thread. Maintains a registry of post-processor callbacks mapped by `EtwEventType` and processes events from a shared `ConcurrentEventQueueRef`. | Member | Type | Description | -|---|---|---| -| `addProvider()` | `Status` | Registers a post-processor callback for specific ETW event types | -| `start()` | `void` | Starts the processing thread | -| `stop()` | `void` | Signals the thread to stop | -| `CommonPostProcessing()` | `bool` (private) | Applies common enrichment logic to every ETW event | -| `etwPostProcessors_` | `ProviderProcessors` | Map of ETW event types to their post-processor callbacks | -| `shouldRun_` | `atomic` | Atomic flag controlling thread lifecycle | -| `concurrentQueue_` | `ConcurrentEventQueueRef&` | Reference to the shared concurrent event queue | +|--------|------|-------------| +| `addProvider()` | `Status` | Registers a post-processor callback for one or more ETW event types | +| `start()` | `void` | Starts the background processing thread | +| `stop()` | `void` | Signals the thread to exit via `shouldRun_` flag | +| `CommonPostProcessing()` | `bool` | Applies common enrichment logic to every ETW event before dispatch | +| `etwPostProcessors_` | `ProviderProcessors` | Map of `EtwEventType` β†’ post-processor callback | +| `concurrentQueue_` | `ConcurrentEventQueueRef&` | Shared reference to the concurrent event queue | ## Usage Example ```cpp -// Create a shared concurrent queue -ConcurrentEventQueueRef sharedQueue = std::make_shared(); +// Create the shared concurrent queue +ConcurrentEventQueueRef queue = std::make_shared(); // Instantiate the post-processing pipeline runnable -auto postProcessor = std::make_shared( - "etw_post_processor", sharedQueue); +auto pipeline = std::make_shared( + "ETWPostProcessor", queue); -// Register an ETW provider with its post-processing callback +// Register a provider with its post-processing callback EtwProviderConfig providerConfig; -// ... configure providerConfig with event types and callback ... -Status status = postProcessor->addProvider(providerConfig); +// ... configure providerConfig with event types and callbacks ... +Status status = pipeline->addProvider(providerConfig); -if (status.ok()) { - // Dispatcher manages the thread lifecycle - Dispatcher::addService(postProcessor); +if (!status.ok()) { + LOG(ERROR) << "Failed to register ETW provider: " << status.getMessage(); } + +// Hand off to the dispatcher to manage the thread lifecycle +Dispatcher::addService(pipeline); ``` ## Notes - -- Uses `ProviderProcessors` (`std::map`) to route events to their handlers. -- `CommonPostProcessing()` runs on **every** event before provider-specific callbacks, enabling consistent event enrichment across all ETW sources. -- Thread safety is enforced via `std::atomic` for the run flag and the shared `ConcurrentEventQueueRef`. \ No newline at end of file +- The `shouldRun_` atomic flag ensures safe cross-thread shutdown via `stop()`. +- `CommonPostProcessing()` is called on **every** event before type-specific callbacks, enabling universal enrichment (e.g., timestamps, process metadata). +- Windows-only; part of the `osquery` ETW event subsystem under `osquery/events/windows/etw/`. \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_provider_config.md b/osquery/events/windows/etw/.etw_provider_config.md index 0374d01d841..7e4edf6ed94 100644 --- a/osquery/events/windows/etw/.etw_provider_config.md +++ b/osquery/events/windows/etw/.etw_provider_config.md @@ -1,11 +1,10 @@ -Defines the `EtwProviderConfig` class, which encapsulates the configurable parameters and callback hooks for an ETW (Event Tracing for Windows) provider within osquery's Windows event pipeline. +Defines the `EtwProviderConfig` class, which encapsulates all configurable parameters for an ETW (Event Tracing for Windows) provider, including event filtering options, pre/post-processing callbacks, and kernel/user provider settings. ## Key Components -### `EtwKernelProviderType` (enum) -Enumerates supported kernel provider categories: -- `File`, `ImageLoad`, `Network`, `Process`, `Registry`, `ObjectManager`, `Invalid` +### Enum +- **`EtwKernelProviderType`** β€” Identifies supported kernel providers: `File`, `ImageLoad`, `Network`, `Process`, `Registry`, `ObjectManager`, `Invalid` ### Type Aliases | Alias | Underlying Type | @@ -15,37 +14,35 @@ Enumerates supported kernel provider categories: | `EtwLevel` | `boost::optional` | | `EventProviderPreProcessor` | `krabs::c_provider_callback` | | `EventProviderPostProcessor` | `std::function` | +| `EtwProviderConfigRef` | `std::shared_ptr` | -### Core Methods -- **`setPreProcessor` / `getPreProcessor`** β€” Attach a raw `krabs` callback for low-level event pre-processing -- **`setPostProcessor` / `getPostProcessor`** β€” Attach a `std::function` for structured post-processing of `EtwEventDataRef` -- **`setEventTypes` / `addEventTypeToHandle`** β€” Define which ETW event types this provider handles -- **`isValid()`** β€” Validates the configuration before use -- **Optional flag setters/getters** β€” `AnyBitmask`, `AllBitmask`, `Level`, `TraceFlags`, `Tag` - -### `EtwProviderConfigRef` -Convenience alias: `std::shared_ptr` +### Key Methods +- **`setPreProcessor` / `getPreProcessor`** β€” Manages the raw `krabs` callback for initial event ingestion +- **`setPostProcessor` / `getPostProcessor`** β€” Manages the `std::function` callback for structured post-processing +- **`setEventTypes` / `addEventTypeToHandle`** β€” Configures which `EtwEventType` values this provider handles +- **`isValid()`** β€” Validates the configuration before provider registration +- **`isAnyBitmaskSet`, `isAllBitmaskSet`, `isLevelSet`** β€” Optional flag presence checks ## Usage Example ```cpp auto config = std::make_shared(); -// Configure a user-mode provider config->setName("Microsoft-Windows-Kernel-Process"); -config->setLevel(osquery::EtwProviderConfig::EtwLevel(4)); // Informational -config->setAnyBitmask(osquery::EtwProviderConfig::EtwBitmask(0x10)); +config->setKernelProviderType( + EtwProviderConfig::EtwKernelProviderType::Process); +config->setLevel(boost::optional(4)); +config->setAnyBitmask(boost::optional(0x10)); -// Attach processing callbacks config->setPreProcessor(myKrabsCallback); -config->setPostProcessor([](const osquery::EtwEventDataRef& event) { - // Handle structured event data +config->setPostProcessor([](const EtwEventDataRef& event) { + // structured post-processing logic }); -config->addEventTypeToHandle(osquery::EtwEventType::ProcessStart); +config->addEventTypeToHandle(EtwEventType::ProcessStart); auto status = config->isValid(); if (!status.ok()) { - // Handle invalid configuration + LOG(ERROR) << "Invalid ETW provider config: " << status.getMessage(); } ``` \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_publisher.md b/osquery/events/windows/etw/.etw_publisher.md index c9744ec7b2c..811d685d473 100644 --- a/osquery/events/windows/etw/.etw_publisher.md +++ b/osquery/events/windows/etw/.etw_publisher.md @@ -1,44 +1,59 @@ -Windows ETW (Event Tracing for Windows) publisher base class that abstracts event collection and dispatching infrastructure for osquery's ETW event system. +ETW (Event Tracing for Windows) publisher base class and registration macros for osquery's Windows event collection subsystem. ## Key Components ### Macros | Macro | Purpose | -|---|---| +|-------|---------| | `REGISTER_ETW_PUBLISHER` | Registers a class as an ETW event publisher plugin | | `REGISTER_ETW_SUBSCRIBER` | Registers a class as an ETW event subscriber plugin | -| `getPreProcessorCallback()` | Generates a lambda wrapping `providerPreProcessor` with structured error logging for raw ETW events | +| `getPreProcessorCallback()` | Generates a lambda wrapping `providerPreProcessor` with structured error logging for ETW raw events | ### Class: `EtwPublisherBase` -Abstract base class for all ETW publishers. Concrete publishers must inherit from this and implement `providerPostProcessor`. +Abstract base class that simplifies ETW event collection and dispatching for concrete publisher implementations. -| Member | Type | Description | -|---|---|---| -| `EtwEngine()` | Method | Returns reference to the global `EtwController` instance | -| `run()` | Method | Signals dispatcher that no dedicated running thread is required | -| `getPostProcessorCallback()` | Method | Returns a `std::function` wrapping the publisher's post-processor | -| `updateUserInfo()` | Protected | Resolves and caches username from a SID string | -| `updateHardVolumeWithLogicalDrive()` | Protected | Converts device paths (e.g., `\Device\HarddiskVolume3\`) to logical drive paths (e.g., `C:\`) | -| `providerPostProcessor()` | Private/Pure virtual | Must be implemented by concrete ETW publishers to handle parsed event data | +**Public Methods** + +| Method | Description | +|--------|-------------| +| `EtwPublisherBase(name)` | Constructs publisher with a provider name | +| `EtwEngine()` | Returns reference to the global `EtwController` instance | +| `run()` | Signals the dispatcher that no dedicated thread is required | +| `getPostProcessorCallback()` | Returns a lambda wrapping the publisher's post-processor | + +**Protected Helpers** + +| Method | Description | +|--------|-------------| +| `updateUserInfo(userSid, username)` | Resolves a SID to a username using an internal cache | +| `updateHardVolumeWithLogicalDrive(path)` | Converts device paths (e.g., `\Device\HarddiskVolume3\...`) to logical drive paths (e.g., `C:\...`) | + +**Private** + +- `providerPostProcessor(data)` β€” Pure virtual; must be implemented by each concrete ETW publisher. ## Usage Example ```cpp -class MyEtwPublisher : public EtwPublisherBase { +class MyEtwPublisher : public EtwPublisherBase, + public EventPublisher { public: - MyEtwPublisher() : EtwPublisherBase("my_etw_publisher") {} + MyEtwPublisher() : EtwPublisherBase("MyProvider") {} + + Status setUp() override { + // Register ETW provider config with EtwEngine() + EtwEngine().addProvider(myProviderConfig, getPreProcessorCallback()); + return Status::success(); + } private: void providerPostProcessor(const EtwEventDataRef& data) override { - // Process and dispatch parsed ETW event data - std::string username; - updateUserInfo(data->userSid, username); - - std::string path = data->imagePath; - updateHardVolumeWithLogicalDrive(path); + // Parse and dispatch event data to subscribers + auto ec = createEventContext(); + fire(ec); } }; diff --git a/osquery/events/windows/etw/.etw_publisher_dns.md b/osquery/events/windows/etw/.etw_publisher_dns.md index 99ba0945ddf..72649e7e020 100644 --- a/osquery/events/windows/etw/.etw_publisher_dns.md +++ b/osquery/events/windows/etw/.etw_publisher_dns.md @@ -1,18 +1,18 @@ -ETW publisher header defining the DNS event collection pipeline for Windows ETW-based DNS monitoring in osquery. +ETW publisher header defining the DNS event monitoring component for Windows Event Tracing, enabling collection and dispatch of DNS query/response events via the ETW infrastructure. ## Key Components ### Structs -- **`EtwDNSEventSubContext`** β€” Subscription context for DNS ETW events; friend-scoped to `EtwPublisherDNS` -- **`EtwDNSEventContext`** β€” Event context carrying an `EtwEventDataRef` payload dispatched to subscribers +- **`EtwDNSEventSubContext`** β€” Subscription context for DNS ETW events; friend-access restricted to `EtwPublisherDNS` +- **`EtwDNSEventContext`** β€” Event context carrying an `EtwEventDataRef` payload for each DNS event ### Type Aliases - **`EtwDNSEventContextRef`** β€” `shared_ptr` - **`EtwDNSEventSubContextRef`** β€” `shared_ptr` ### Constants -- **`kEtwDNSPublisherName`** β€” Publisher identifier string `"etw_dns_publisher"` +- **`kEtwDNSPublisherName`** β€” Publisher identifier string: `"etw_dns_publisher"` ### Class: `EtwPublisherDNS` Inherits from `EtwPublisherBase` and `EventPublisher`. @@ -21,31 +21,28 @@ Inherits from `EtwPublisherBase` and `EventPublisher -// Register and start the DNS ETW publisher -auto publisher = std::make_shared(); -publisher->setUp(); - // Subscribe to DNS events -auto sub = std::make_shared(); -auto callback = [](const EtwDNSEventContextRef& ctx, - const EtwDNSEventSubContextRef&) { - // Access enriched DNS event data +auto sub_ctx = std::make_shared(); + +EventFactory::registerEventPublisher(); + +// Access event data in a subscriber callback +void onDNSEvent(const EtwDNSEventContextRef& ctx, + const EtwDNSEventSubContextRef& sub) { auto& data = ctx->data; - // process data... - return Status::success(); -}; + // Process DNS event payload... +} ``` \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_publisher_processes.md b/osquery/events/windows/etw/.etw_publisher_processes.md index 8578b68d793..7a872a1337c 100644 --- a/osquery/events/windows/etw/.etw_publisher_processes.md +++ b/osquery/events/windows/etw/.etw_publisher_processes.md @@ -1,59 +1,48 @@ -ETW publisher header for collecting and dispatching Windows process lifecycle events (start/stop) using the Event Tracing for Windows (ETW) infrastructure within osquery. +ETW publisher header for collecting and dispatching Windows process lifecycle events (start/stop) via the Event Tracing for Windows (ETW) infrastructure within osquery. ## Key Components -### Structs / Type Aliases +### Structs +- **`EtwProcEventSubContext`** β€” Subscription context for process event subscribers; friend-scoped to `EtwPublisherProcesses` +- **`EtwProcessEventContext`** β€” Event context carrying a reference to parsed ETW event data (`EtwEventDataRef`) -| Type | Description | -|---|---| -| `EtwProcEventSubContext` | Subscription context for process event subscribers | -| `EtwProcessEventContext` | Event context carrying `EtwEventDataRef` payload data | -| `EtwProcessEventContextRef` | `shared_ptr` alias for `EtwProcessEventContext` | -| `EtwProcEventSubContextRef` | `shared_ptr` alias for `EtwProcEventSubContext` | +### Type Aliases +- **`EtwProcessEventContextRef`** β€” `shared_ptr` +- **`EtwProcEventSubContextRef`** β€” `shared_ptr` ### Constants - -- `kEtwProcessPublisherName` β€” Publisher identifier string: `"etw_process_publisher"` +- **`kEtwProcessPublisherName`** β€” Publisher identifier string `"etw_process_publisher"` ### Class: `EtwPublisherProcesses` - -Inherits from `EtwPublisherBase` and `EventPublisher`. Listens to kernel and user-mode ETW providers for process start/stop events. +Inherits from `EtwPublisherBase` and `EventPublisher`. | Member | Description | -|---|---| -| `setUp()` | Registers ETW providers, configuration, and callbacks | -| `providerPreProcessor()` | Static C-function callback; lightweight ETW event entry point called per OS event | -| `providerPostProcessor()` | Enriches and aggregates event data before dispatching to subscribers | -| `cleanOldAggregationCacheEntries()` | Purges stale cache entries | -| `updateTokenInfo()` | Resolves token type metadata | -| `updateImagePath()` | Resolves process image paths via composite key lookup | -| `processStartAggregationCache_` | Cache keyed by composed `uint64` key for process start correlation | -| `processImageCache_` | Cache mapping composite keys to image path strings | - -**Supported event versions tracked internally:** -- User process start: `Version0–3` -- User process stop: `Version0–2` -- Kernel process events: `Version3–4` +|--------|-------------| +| `setUp()` | Registers ETW providers and binds processing callbacks | +| `providerPreProcessor()` | Static C-function callback; lightweight entry point called per OS ETW event | +| `providerPostProcessor()` | Enriches/aggregates event data before subscriber dispatch | +| `cleanOldAggregationCacheEntries()` | Evicts stale entries from the process cache | +| `updateTokenInfo()` | Resolves token type to string representation | +| `updateImagePath()` | Resolves composite key to an executable image path | +| `processStartAggregationCache_` | Keyed cache of in-flight process start events | +| `processImageCache_` | Keyed cache of process image paths | + +**Supported ETW event IDs:** kernel default (`0`), process start (`1`), process stop (`2`), across multiple provider schema versions. ## Usage Example -```cpp -// Register and start the ETW process publisher -auto publisher = std::make_shared(); +```c +// Subscribe to process events +auto sub = EtwPublisherProcesses::createSubscriptionContext(); -Status status = publisher->setUp(); -if (!status.ok()) { - LOG(ERROR) << "ETW process publisher setup failed: " << status.getMessage(); -} +EventFactory::addEventSubscriber(); -// Subscribe to process events -auto subscription = EtwProcEventSubContext::create(); -EventFactory::addSubscription(kEtwProcessPublisherName, subscription, - [](const EtwProcessEventContextRef& ctx, - const EtwProcEventSubContextRef& /*sub*/) -> Status { +// Accessing event data inside a subscriber callback +Status MySubscriber::Callback(const EtwProcessEventContextRef& ctx, + const EtwProcEventSubContextRef& /*sub*/) { auto& data = ctx->data; - // Handle process start/stop event data + // Inspect process start/stop payload via data fields return Status::success(); - }); +} ``` \ No newline at end of file diff --git a/osquery/events/windows/etw/.etw_user_session.md b/osquery/events/windows/etw/.etw_user_session.md index 96784ea8cb2..c620f1affff 100644 --- a/osquery/events/windows/etw/.etw_user_session.md +++ b/osquery/events/windows/etw/.etw_user_session.md @@ -1,49 +1,53 @@ -Manages the ETW (Event Tracing for Windows) user-space trace session for osquery, running on a dedicated thread via the `InternalRunnable` abstraction to listen for and process user-space ETW events. +Manages an ETW (Event Tracing for Windows) user-space trace session, running it on a dedicated thread via osquery's `InternalRunnable` abstraction to listen for and process user-space ETW events. ## Key Components ### `UserEtwSessionRunnable` (class) -Inherits from `InternalRunnable`. Owns and manages a `krabs::user_trace` session and a collection of registered ETW providers. - -| Member | Type | Description | -|---|---|---| -| `addProvider()` | `Status` | Registers an ETW user-space provider with pre/post-processor callbacks | -| `start()` | `void` | Starts the dedicated trace session thread | -| `stop()` | `void` | Stops the dedicated trace session thread | -| `pause()` / `resume()` | `void` | Pauses and resumes event listening | -| `initUserTraceSession()` | `void` | Best-effort initialization of the KrabsETW user trace | -| `stopUserTraceSession()` | `void` | Best-effort teardown of the KrabsETW user trace | - -**Internal state:** -- `userTraceSession_` β€” KrabsETW `krabs::user_trace` instance -- `runningProviders_` β€” vector of active `krabs::provider<>` references -- `mutex_` β€” thread-safety guard -- `endTraceSession_` / `traceSessionStopped_` β€” atomic lifecycle flags -- `condition_` β€” condition variable for pause/resume signaling +A thread-safe, final class extending `InternalRunnable` that owns and controls a KrabsETW `user_trace` session. + +**Type Aliases** +| Alias | Underlying Type | +|---|---| +| `EtwUserProvider` | `std::shared_ptr` | +| `UserProviderRef` | `std::shared_ptr>` | +| `UserProvidersCollection` | `std::vector` | + +**Public Methods** +- `UserEtwSessionRunnable(name)` β€” Constructs the runnable with a given thread name +- `addProvider(configData)` β€” Registers a user-space ETW provider along with its pre/post processor callbacks + +**Protected Methods** +- `start()` β€” Starts the underlying `InternalRunnable` thread +- `stop()` β€” Stops the underlying `InternalRunnable` thread + +**Private Helpers** +- `pause()` / `resume()` β€” Suspend or continue event listening +- `initUserTraceSession(name)` β€” Best-effort initialization of the KrabsETW user trace +- `stopUserTraceSession(name)` β€” Best-effort teardown of the trace session + +**Key Members** +- `userTraceSession_` β€” KrabsETW `user_trace` instance +- `runningProviders_` β€” Cache of active provider references +- `endTraceSession_` / `traceSessionStopped_` β€” Atomic flags for lifecycle control +- `mutex_` / `condition_` β€” Thread-safety and pause/resume synchronization ## Usage Example ```cpp -#include +// Instantiate and start the ETW user session thread +auto session = std::make_shared("etw-user-session"); -// Create and start a user ETW session runnable -auto session = std::make_shared("MyUserSession"); +// Register a provider with preprocessing/postprocessing callbacks +osquery::EtwProviderConfig config; // populate with GUID, callbacks, etc. +osquery::Status status = session->addProvider(config); -// Register a provider using its configuration -osquery::EtwProviderConfig config; -// ... populate config with provider GUID, callbacks, etc. - -auto status = session->addProvider(config); if (!status.ok()) { LOG(ERROR) << "Failed to add ETW provider: " << status.getMessage(); } -// Start listening (runs on dedicated internal thread) -session->start(); - -// Stop when done -session->stop(); -``` - -> **Note:** This class is Windows-only and depends on the [KrabsETW](https://github.com/microsoft/krabsetw) library (`krabs::user_trace`). It is `final` and not intended for subclassing. \ No newline at end of file +// Thread lifecycle is managed via InternalRunnable +session->start(); // begins listening for user-space ETW events +// ... +session->stop(); // signals the trace session to terminate +``` \ No newline at end of file diff --git a/osquery/experimental/events_stream/.events_stream.md b/osquery/experimental/events_stream/.events_stream.md index 8e0dc05217d..f650d6cf477 100644 --- a/osquery/experimental/events_stream/.events_stream.md +++ b/osquery/experimental/events_stream/.events_stream.md @@ -1,18 +1,25 @@ -Declares a single function for dispatching serialized events within the osquery events subsystem. +Declares a single event dispatch function within the `osquery::events` namespace for routing serialized event data through the events streaming subsystem. ## Key Components -- **`dispatchSerializedEvent(const std::string& event)`** β€” Accepts a serialized event string and dispatches it through the osquery event pipeline. +| Symbol | Type | Description | +|--------|------|-------------| +| `osquery::events::dispatchSerializedEvent` | Function | Dispatches a pre-serialized event string into the event stream pipeline | ## Usage Example ```c #include "events_stream.h" -// Dispatch a pre-serialized event payload -std::string serialized = serializeMyEvent(data); -osquery::events::dispatchSerializedEvent(serialized); +// Serialize your event payload (e.g., JSON or protobuf string) +std::string serializedPayload = R"({"action":"open","path":"/etc/passwd"})"; + +// Dispatch the event into the osquery event stream +osquery::events::dispatchSerializedEvent(serializedPayload); ``` -> **Note:** The function expects the event to already be serialized before being passed in. Serialization format is determined by the calling subsystem. \ No newline at end of file +## Notes + +- The function accepts events as pre-serialized `std::string` values β€” callers are responsible for serialization before dispatch. +- Designed for internal use within the `osquery` event subsystem; typically called after an event has been captured and encoded by a platform-specific event publisher. \ No newline at end of file diff --git a/osquery/experimental/events_stream/.events_stream_registry.md b/osquery/experimental/events_stream/.events_stream_registry.md index 2f172cfff4c..92485a98a01 100644 --- a/osquery/experimental/events_stream/.events_stream_registry.md +++ b/osquery/experimental/events_stream/.events_stream_registry.md @@ -1,13 +1,10 @@ -Declares the `EventsStreamPlugin` class and supporting utilities for the osquery events stream plugin registry, enabling event data to be dispatched through the plugin framework. +Declares the `EventsStreamPlugin` class and registry utilities for osquery's events streaming subsystem, providing a plugin interface for routing event data through the plugin registry. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `EventsStreamPlugin` | Class | Plugin subclass that handles event stream dispatch via the osquery plugin registry | -| `EventsStreamPlugin::call` | Method | Processes incoming `PluginRequest` and populates a `PluginResponse`; overrides `Plugin::call` | -| `events::streamRegistryName` | Function | Returns the canonical string name used to register the events stream in the plugin registry | +- **`EventsStreamPlugin`** β€” Extends `Plugin` to handle stream-based event dispatch; overrides `call()` to process incoming `PluginRequest` and populate a `PluginResponse` +- **`events::streamRegistryName()`** β€” Returns the string identifier used to register and look up the events stream plugin within osquery's plugin registry ## Usage Example @@ -17,22 +14,26 @@ class MyEventsStreamPlugin : public EventsStreamPlugin { public: Status call(const PluginRequest& request, PluginResponse& response) override { - // Handle the incoming event stream request + // Handle incoming event stream data auto event_data = request.at("data"); response.push_back({{"status", "ok"}}); return Status::success(); } }; -// Register using the canonical registry name -REGISTER(MyEventsStreamPlugin, - events::streamRegistryName(), - "my_events_stream"); +REGISTER(MyEventsStreamPlugin, events::streamRegistryName(), "my_stream"); + +// Dispatching to the events stream registry +PluginRequest req = {{"data", serialized_event}}; +PluginResponse res; +Registry::call(events::streamRegistryName(), "my_stream", req, res); ``` ## Dependencies -- `osquery/core/plugins/plugin.h` β€” base `Plugin` class and `PluginRequest`/`PluginResponse` types -- `osquery/core/query.h` β€” core query types shared across the osquery plugin framework -- `osquery/utils/expected/expected.h` β€” `Expected<>` utility for error-propagating return types -- `osquery/numeric_monitoring/numeric_monitoring.h` β€” numeric metrics instrumentation support \ No newline at end of file +| Header | Purpose | +|--------|---------| +| `plugin.h` | Base `Plugin` class and `PluginRequest`/`PluginResponse` types | +| `query.h` | Core query result structures | +| `expected.h` | Error-handling utility types | +| `numeric_monitoring.h` | Numeric telemetry hooks for stream metrics | \ No newline at end of file diff --git a/osquery/experimental/experiments/linuxevents/src/.bpfprocesseventstable.md b/osquery/experimental/experiments/linuxevents/src/.bpfprocesseventstable.md index 136755aa829..c51ba85dc63 100644 --- a/osquery/experimental/experiments/linuxevents/src/.bpfprocesseventstable.md +++ b/osquery/experimental/experiments/linuxevents/src/.bpfprocesseventstable.md @@ -1,43 +1,50 @@ -Header defining the `BPFProcessEventsTable` plugin, an osquery table that collects Linux process events captured via BPF (Berkeley Packet Filter) tracing. +Header defining the `BPFProcessEventsTable` plugin, an osquery table that collects process events captured via BPF (Berkeley Packet Filter) on Linux systems. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `BPFProcessEventsTable` | Class | osquery `TablePlugin` that exposes BPF process events as a queryable table | -| `ErrorCode` | Enum | Factory error states β€” currently `MemoryAllocationFailure` | -| `Ptr` | Type alias | `std::shared_ptr` for safe ownership | -| `create()` | Static factory | Returns `Expected` β€” preferred construction path | -| `addEvents()` | Method | Ingests a batch of `ILinuxEvents::EventList` events into the table buffer | -| `name()` | Method | Returns the registered osquery table name | -| `columns()` | Override | Declares the table schema to the osquery runtime | -| `generate()` | Override | Produces `TableRows` in response to SQL queries | +**Class: `BPFProcessEventsTable`** +- Extends `TablePlugin` to expose BPF-captured process events as a queryable osquery table +- Uses the pImpl pattern (`PrivateData`) to hide implementation details +- Non-copyable by design (deleted copy constructor and assignment operator) + +**`ErrorCode` enum** +- `MemoryAllocationFailure` β€” returned when table instantiation fails due to memory constraints + +**Key Methods** + +| Method | Description | +|--------|-------------| +| `create()` | Static factory returning `Expected`; preferred over direct construction | +| `name()` | Returns the registered table name string | +| `addEvents(EventList)` | Ingests a list of Linux events from the BPF event source | +| `columns()` | Declares the table schema (overrides `TablePlugin`) | +| `generate(QueryContext&)` | Produces rows for query execution (overrides `TablePlugin`) | ## Usage Example ```cpp #include "bpfprocesseventstable.h" -// Create the table plugin via factory +// Create the table instance safely auto result = osquery::BPFProcessEventsTable::create(); if (result.isError()) { - // Handle MemoryAllocationFailure - return; + // Handle MemoryAllocationFailure + return; } osquery::BPFProcessEventsTable::Ptr table = result.take(); -// Push captured BPF events into the table -tob::linuxevents::ILinuxEvents::EventList events = captureEvents(); +// Feed captured Linux events into the table +tob::linuxevents::ILinuxEvents::EventList events = getEventsFromBPF(); table->addEvents(std::move(events)); -// osquery runtime calls generate() internally when -// a query like `SELECT * FROM bpf_process_events` is executed +// osquery runtime will call generate() internally +// when a query targets this table, e.g.: +// SELECT pid, path, cmdline FROM bpf_process_events; ``` ## Notes - -- Non-copyable by design β€” copy constructor and assignment operator are explicitly deleted -- Uses the **PIMPL idiom** (`PrivateData`) to hide implementation details and reduce compile-time dependencies -- Depends on `tob::linuxevents::ILinuxEvents` from the `linuxevents` library for raw event ingestion \ No newline at end of file +- Depends on `tob::linuxevents::ILinuxEvents` for the BPF event source abstraction +- Linux-only; relies on the `linuxevents` library from the Trail of Bits ecosystem +- Thread safety of `addEvents` depends on the `PrivateData` implementation \ No newline at end of file diff --git a/osquery/experimental/experiments/linuxevents/src/.linuxeventsservice.md b/osquery/experimental/experiments/linuxevents/src/.linuxeventsservice.md index b8afb1daee8..b8475235cc7 100644 --- a/osquery/experimental/experiments/linuxevents/src/.linuxeventsservice.md +++ b/osquery/experimental/experiments/linuxevents/src/.linuxeventsservice.md @@ -1,15 +1,13 @@ -A background service class that runs as an osquery internal thread, consuming BPF-sourced Linux process events and feeding them into the `BPFProcessEventsTable`. +Background service that runs as an osquery dispatcher thread, collecting Linux system events via BPF and feeding them into the `BPFProcessEventsTable`. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `LinuxEventsService` | Class | Final class inheriting from `InternalRunnable`; manages the lifecycle of the BPF event processing loop | -| `LinuxEventsService(BPFProcessEventsTable&)` | Constructor | Binds the service to a specific `BPFProcessEventsTable` instance | -| `start()` | Protected method | Entry point called by the dispatcher to begin event consumption | -| `stop()` | Protected method | Signals the service to halt gracefully | -| `PrivateData` | Private struct (PIMPL) | Opaque implementation detail; hides internal state behind a `unique_ptr` | +- **`LinuxEventsService`** β€” Final class inheriting from `InternalRunnable`; manages the lifecycle of the BPF-based Linux event collection loop + - `LinuxEventsService(BPFProcessEventsTable& table)` β€” Constructor binding the service to a target events table + - `start()` β€” Begins the event collection loop (called by the dispatcher) + - `stop()` β€” Signals the service to halt gracefully + - `PrivateData` β€” Opaque `pImpl` struct hiding implementation details (BPF handles, ring buffers, etc.) ## Usage Example @@ -18,18 +16,19 @@ A background service class that runs as an osquery internal thread, consuming BP #include "bpfprocesseventstable.h" #include -// Instantiate the table that will receive BPF process events -osquery::BPFProcessEventsTable bpfTable; +// Instantiate the events table (registered separately with osquery) +BPFProcessEventsTable eventsTable; -// Create and register the service with the osquery dispatcher -auto service = std::make_shared(bpfTable); +// Create and launch the background service via the osquery dispatcher +auto service = std::make_shared(eventsTable); osquery::Dispatcher::addService(service); -// The dispatcher calls start() internally; stop() is invoked on shutdown +// The dispatcher calls start() internally; +// stop() is invoked on shutdown via the dispatcher teardown path ``` ## Notes -- Uses the **PIMPL idiom** (`PrivateData` + `unique_ptr`) to keep implementation details out of the header and reduce recompilation overhead. -- Marked `final` β€” not intended to be subclassed. -- Depends on `osquery::InternalRunnable` (from `dispatcher/dispatcher.h`) for thread lifecycle management. \ No newline at end of file +- Uses the **pImpl idiom** (`PrivateData`) to keep BPF-specific includes and state out of the public header +- Marked `final` β€” not intended for further subclassing +- Lifecycle is managed entirely by the osquery `Dispatcher`; do not call `start()`/`stop()` directly \ No newline at end of file diff --git a/osquery/extensions/.extensions.md b/osquery/extensions/.extensions.md index d9e0be1f5b4..24bfc67a5db 100644 --- a/osquery/extensions/.extensions.md +++ b/osquery/extensions/.extensions.md @@ -1,52 +1,52 @@ -Defines the public API for osquery's extension system, providing interfaces for loading, managing, and communicating with external extension processes via UNIX domain sockets. +Defines the public interface for osquery's extension system, providing types, utilities, and function declarations for loading, managing, and communicating with external extensions via UNIX domain sockets. ## Key Components ### Flags -| Flag | Type | Description | -|------|------|-------------| +| Flag | Type | Purpose | +|------|------|---------| | `extensions_socket` | string | Path to the extension manager socket | -| `extensions_autoload` | string | Search path for autoloading extensions | +| `extensions_autoload` | string | Search path for auto-loaded extensions | | `extensions_timeout` | string | Global timeout for extension operations | -| `disable_extensions` | bool | Toggle to disable the extension subsystem | +| `disable_extensions` | bool | Toggle extension support entirely | -### Structs & Types -- **`ExtensionInfo`** β€” Metadata container (name, version, sdk_version, min_sdk_version) matching Thrift's `InternalExtensionInfo` -- **`ExtensionList`** β€” `std::map` mapping route IDs to extension metadata +### Types +- **`ExtensionInfo`** β€” Metadata struct holding `name`, `version`, `sdk_version`, and `min_sdk_version` +- **`ExtensionList`** β€” `std::map` mapping route UUIDs to extension metadata -### Classes -- **`ExternalSQLPlugin`** β€” `SQLPlugin` implementation that routes SQL queries through an extension registry, exposing `query()` and `getQueryColumns()` +### `ExternalSQLPlugin` +Implements `SQLPlugin` for extensions, routing `query()` and `getQueryColumns()` calls over the extension socket. ### Key Functions - -| Function | Description | -|----------|-------------| -| `getExtensionSocket()` | Derives a socket path for a given `RouteUUID` | -| `loadExtensions()` | Reads autoload flags and returns valid extension paths | -| `startExtension()` | Launches an `ExtensionRunner` thread for an extension process | -| `startExtensionManager()` | Launches the `ExtensionManagerRunner` thread | -| `startExtensionWatcher()` | Starts a watcher thread monitoring a manager connection | -| `callExtension()` | Invokes a plugin exposed by an extension over a socket | -| `getExtensions()` | Retrieves the list of currently active extensions | -| `pingExtension()` | Health-checks an extension or manager socket | -| `applyExtensionDelay()` | Runs a predicate in a retry loop bounded by the extension timeout | -| `initShellSocket()` | Resolves a unique socket path for `osqueryi` shell sessions | +| Function | Purpose | +|----------|---------| +| `getExtensionSocket()` | Resolves socket path for a given UUID | +| `getExtensions()` | Lists active extensions via manager socket | +| `pingExtension()` | Health-checks an extension or manager | +| `applyExtensionDelay()` | Polls a predicate until timeout or success | +| `loadExtensions()` | Reads autoload paths and returns valid extension paths | +| `callExtension()` | Dispatches a plugin request to an extension over IPC | +| `startExtension()` | Launches an `ExtensionRunner` thread for a named extension | +| `startExtensionWatcher()` | Starts a watchdog thread monitoring manager liveness | +| `startExtensionManager()` | Starts the `ExtensionManagerRunner` accepting extension connections | +| `initShellSocket()` | Computes a unique socket path for `osqueryi` shell sessions | ## Usage Example ```c -// Autoload extensions at startup -std::set paths = osquery::loadExtensions(); - -// Start the extension manager -osquery::startExtensionManager(); - -// Call a plugin exposed by a registered extension -osquery::PluginRequest req = {{"action", "query"}}; -osquery::PluginResponse resp; -osquery::callExtension(uuid, "table", "my_table", req, resp); - -// Start an extension process and connect to the manager -osquery::startExtension("my_extension", "1.0.0", "2.0.0"); +// Start an extension with version and SDK compatibility info +Status s = startExtension("my_extension", "1.0.0", "2.0.0"); +if (!s.ok()) { + LOG(ERROR) << "Failed to start extension: " << s.getMessage(); +} + +// Call a plugin exposed by a remote extension +PluginRequest request = {{"action", "get"}}; +PluginResponse response; +Status call_status = callExtension(uuid, "table", "my_table", request, response); + +// Resolve socket path for a given extension UUID +std::string sock = getExtensionSocket(42); +// Returns: ".42" ``` \ No newline at end of file diff --git a/osquery/extensions/.interface.md b/osquery/extensions/.interface.md index 5b471e23d4f..b0de8f5b00a 100644 --- a/osquery/extensions/.interface.md +++ b/osquery/extensions/.interface.md @@ -1,40 +1,37 @@ -Defines the abstract interfaces, data structures, and concrete classes for osquery's Thrift-based extension IPC system, providing both server-side handlers and client-side accessors for extension/extension-manager communication. +Defines the abstract interfaces and concrete implementations for osquery's Thrift-based extension IPC system, including server handlers, client accessors, and dispatcher runner threads for both extension and extension manager endpoints. ## Key Components -| Class / Type | Role | -|---|---| -| `ExtensionAPI` | Pure abstract base defining `ping()`, `call()`, and `shutdown()` | -| `ExtensionManagerAPI` | Pure abstract base for manager operations: register, deregister, query, options | -| `ExtensionInterface` | Concrete `ExtensionAPI` server implementation with a transient `RouteUUID` | -| `ExtensionManagerInterface` | Full server implementation combining both APIs; manages route table and extension list | -| `ExtensionRunnerInterface` | PIMPL wrapper around the Thrift server setup (`serve`, `connect`, `init`) | -| `ExtensionRunnerCore` | `InternalRunnable` base for dispatcher threads with start/stop lifecycle | -| `ExtensionRunner` | Dispatcher thread serving a single extension handler | -| `ExtensionManagerRunner` | Dispatcher thread serving the extension manager handler | -| `ExtensionClientCore` | Non-copyable PIMPL base for UNIX socket clients | -| `ExtensionClient` | Client to an individual extension (from the manager side) | -| `ExtensionManagerClient` | Client to the extension manager (from an extension side) | -| `Option` | Holds a flag's current value, default, and type string | -| `ExtensionCode` | Enum mirroring Thrift IDL status codes (`EXT_SUCCESS`, `EXT_FAILED`, `EXT_FATAL`) | +- **`Option`** β€” Struct holding flag metadata (current value, default value, type string) +- **`ExtensionCode`** β€” Enum for Thrift response codes (`EXT_SUCCESS`, `EXT_FAILED`, `EXT_FATAL`) +- **`ExtensionAPI`** β€” Pure abstract base defining the extension-side API (`ping`, `call`, `shutdown`) +- **`ExtensionManagerAPI`** β€” Pure abstract base defining the manager-side API (`extensions`, `options`, `registerExtension`, `deregisterExtension`, `query`, `getQueryColumns`) +- **`ExtensionInterface`** β€” Concrete Thrift server handler implementing `ExtensionAPI` for extension processes +- **`ExtensionManagerInterface`** β€” Concrete Thrift server handler implementing both APIs for the osquery core process; manages extension registration, route tracking, and SQL query forwarding +- **`ExtensionRunnerInterface`** β€” PIMPL wrapper around the Thrift server lifecycle (`connect`, `init`, `serve`, `stopServer`) +- **`ExtensionRunnerCore`** β€” `InternalRunnable` base for Dispatcher-managed Thrift server threads +- **`ExtensionRunner`** β€” Dispatcher thread that serves an extension's Thrift endpoint +- **`ExtensionManagerRunner`** β€” Dispatcher thread that serves the extension manager's Thrift endpoint +- **`ExtensionClientCore`** β€” Non-copyable PIMPL base for Thrift client connections (socket path, timeouts) +- **`ExtensionClient`** β€” Client for calling into a remote extension +- **`ExtensionManagerClient`** β€” Client for calling into the extension manager from an extension process ## Usage Example ```cpp // Start an extension manager server -auto manager_runner = std::make_unique( - "/var/osquery/osquery.em"); -Dispatcher::addService(std::move(manager_runner)); +auto runner = std::make_shared("/var/osquery/osquery.em"); +Dispatcher::addService(runner); -// Connect a client from an extension back to the manager +// Connect a client from an extension process ExtensionManagerClient client("/var/osquery/osquery.em"); RouteUUID uuid; ExtensionRegistry registry; Status s = client.registerExtension(info, registry, uuid); -// Call a plugin on a connected extension from the manager -ExtensionClient ext_client("/var/osquery/ext.sock"); +// Forward a plugin call to a registered extension +ExtensionClient ext_client("/var/osquery/ext.12345"); PluginResponse response; -ext_client.call("table", "users", {{"action", "generate"}}, response); +ext_client.call("table", "my_table", {{"action", "generate"}}, response); ``` \ No newline at end of file diff --git a/osquery/extensions/thrift/gen/.Extension.md b/osquery/extensions/thrift/gen/.Extension.md index 9af9061d1fe..cad29fb0181 100644 --- a/osquery/extensions/thrift/gen/.Extension.md +++ b/osquery/extensions/thrift/gen/.Extension.md @@ -1,51 +1,48 @@ -Autogenerated Apache Thrift RPC interface header for the osquery Extension service, defining the client, processor, and service interface for inter-process communication between osquery core and its extensions. +Autogenerated Thrift RPC interface header (Thrift Compiler 0.13.0) defining the `Extension` service within the `osquery::extensions` namespace, enabling IPC communication between osquery core and its extensions. ## Key Components | Class | Description | |---|---| -| `ExtensionIf` | Pure virtual service interface defining the three RPC methods all extensions must implement | -| `ExtensionNull` | No-op implementation of `ExtensionIf`; useful as a stub or placeholder | -| `ExtensionIfFactory` / `ExtensionIfSingletonFactory` | Factory abstractions for creating/releasing handler instances per connection | -| `ExtensionClient` | Thrift-generated RPC client for calling remote extension methods over a protocol transport | -| `ExtensionProcessor` | Server-side dispatcher that routes incoming RPC calls to the appropriate handler method | -| Arg/Result structs | Serialization containers (`Extension_ping_args`, `Extension_call_args`, etc.) for each RPC method's parameters and return values | - -## RPC Methods - -- **`ping`** β€” Health check; returns an `ExtensionStatus` -- **`call`** β€” Invokes a plugin by registry name and item key, passing an `ExtensionPluginRequest`; returns an `ExtensionResponse` -- **`shutdown`** β€” Signals the extension to terminate +| `ExtensionIf` | Pure virtual interface defining the three RPC methods all extensions must implement | +| `ExtensionNull` | No-op implementation of `ExtensionIf`; useful as a stub or default handler | +| `ExtensionIfFactory` / `ExtensionIfSingletonFactory` | Factory classes for creating/releasing handler instances per connection | +| `ExtensionClient` | Thrift-generated client that sends RPC calls over a `TProtocol` transport | +| `ExtensionProcessor` | Server-side dispatcher that routes incoming RPC frames to the correct handler method | +| `Extension_*_args/pargs/result/presult` | Serialization structs for marshalling RPC arguments and return values over the wire | + +### RPC Methods (defined on `ExtensionIf`) + +```c +void ping(ExtensionStatus& _return); +void call(ExtensionResponse& _return, + const std::string& registry, + const std::string& item, + const ExtensionPluginRequest& request); +void shutdown(); +``` ## Usage Example ```cpp -#include "Extension.h" -#include -#include - -// Connect to a running extension over a socket -auto socket = std::make_shared("localhost", 9090); -auto transport = std::make_shared(socket); +// Connect to a running osquery extension socket +auto transport = std::make_shared("/tmp/osquery.em.12345"); auto protocol = std::make_shared(transport); -transport->open(); - osquery::extensions::ExtensionClient client(protocol); +transport->open(); -// Health check +// Health-check the extension osquery::extensions::ExtensionStatus status; client.ping(status); -// Invoke a plugin -osquery::extensions::ExtensionResponse response; +// Invoke a plugin registered in the extension osquery::extensions::ExtensionPluginRequest req; -client.call(response, "table", "processes", req); +osquery::extensions::ExtensionResponse resp; +client.call(resp, "table", "my_table", req); -// Shut down the extension +// Gracefully stop the extension client.shutdown(); transport->close(); -``` - -> **Note:** This file is autogenerated by Thrift Compiler 0.13.0. Do not edit manually β€” regenerate from the `.thrift` IDL definition instead. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/extensions/thrift/gen/.ExtensionManager.md b/osquery/extensions/thrift/gen/.ExtensionManager.md index 39e9729e8f1..7cc47012e64 100644 --- a/osquery/extensions/thrift/gen/.ExtensionManager.md +++ b/osquery/extensions/thrift/gen/.ExtensionManager.md @@ -1,46 +1,45 @@ -Autogenerated Thrift RPC header (Thrift Compiler 0.13.0) defining the `ExtensionManager` service interface for osquery's inter-process extension communication within the `osquery::extensions` namespace. +Autogenerated Thrift RPC interface header defining the `ExtensionManager` service for osquery's extension IPC layer, extending the base `Extension` service with manager-specific operations. ## Key Components -| Class | Description | -|---|---| -| `ExtensionManagerIf` | Pure virtual interface extending `ExtensionIf`; defines all RPC method contracts | -| `ExtensionManagerIfFactory` | Abstract factory for creating/releasing `ExtensionManagerIf` handler instances | -| `ExtensionManagerIfSingletonFactory` | Concrete factory wrapping a shared singleton handler instance | -| `ExtensionManagerNull` | No-op implementation of all interface methods; useful for testing/stubbing | -| `ExtensionManager_*_args / _pargs` | Thrift-generated argument containers for each RPC call | -| `ExtensionManager_*_result / _presult` | Thrift-generated result containers with `__isset` tracking for optional fields | - -### RPC Methods (via `ExtensionManagerIf`) +### Interfaces +- **`ExtensionManagerIf`** β€” Pure virtual interface extending `ExtensionIf`; defines all manager RPC methods that implementations must provide +- **`ExtensionManagerIfFactory`** / **`ExtensionManagerIfSingletonFactory`** β€” Factory classes for creating/releasing handler instances per connection; singleton variant wraps a shared handler +- **`ExtensionManagerNull`** β€” No-op implementation of all interface methods; useful for testing or stub implementations +### RPC Methods (defined on `ExtensionManagerIf`) | Method | Description | -|---|---| -| `extensions()` | Returns list of registered extensions (`InternalExtensionList`) | -| `options()` | Returns available options (`InternalOptionList`) | -| `registerExtension()` | Registers an extension with its info and route registry | +|--------|-------------| +| `extensions()` | Returns list of all registered extensions | +| `options()` | Returns current configuration options | +| `registerExtension()` | Registers a new extension with its route registry | | `deregisterExtension()` | Removes an extension by UUID | -| `query()` | Executes a SQL query via an extension | -| `getQueryColumns()` | Returns column metadata for a SQL query | +| `query()` | Executes a SQL query via the extension channel | +| `getQueryColumns()` | Returns column schema for a SQL query | + +### Thrift-Generated Serialization Structs +Each RPC method has corresponding `_args`, `_pargs`, `_result`, and `_presult` classes handling Thrift protocol serialization (`read`/`write` over `TProtocol`). ## Usage Example ```cpp -#include "ExtensionManager.h" - -// Implement the interface +// Implement the manager interface class MyExtensionManager : virtual public osquery::extensions::ExtensionManagerIf { public: - void extensions(osquery::extensions::InternalExtensionList& _return) override { + void extensions(InternalExtensionList& _return) override { // populate _return with registered extensions } - void query(osquery::extensions::ExtensionResponse& _return, - const std::string& sql) override { - // execute SQL and populate _return + void registerExtension( + ExtensionStatus& _return, + const InternalExtensionInfo& info, + const ExtensionRegistry& registry) override { + // validate and register the extension + _return.code = 0; } - // ... implement remaining pure virtual methods + // implement remaining pure virtual methods... }; // Wrap in singleton factory for Thrift server @@ -49,4 +48,4 @@ auto factory = std::make_shared< osquery::extensions::ExtensionManagerIfSingletonFactory>(handler); ``` -> **Note:** This file is autogenerated β€” do not edit manually. Regenerate using Thrift Compiler 0.13.0 from the corresponding `.thrift` service definition. \ No newline at end of file +> **Note:** This file is autogenerated by Thrift Compiler 0.13.0. Do not edit manually β€” regenerate from the `.thrift` IDL source instead. \ No newline at end of file diff --git a/osquery/extensions/thrift/gen/.osquery_constants.md b/osquery/extensions/thrift/gen/.osquery_constants.md index 2d1f433767c..f411210874b 100644 --- a/osquery/extensions/thrift/gen/.osquery_constants.md +++ b/osquery/extensions/thrift/gen/.osquery_constants.md @@ -1,12 +1,11 @@ -Auto-generated Thrift constants header for the `osquery` extensions namespace. Declares the `osqueryConstants` class and its global instance used across the osquery Thrift service interface. +Auto-generated Thrift compiler constants header for the `osquery` extensions namespace. Defines a constants class and global instance as part of the Thrift-generated osquery IPC interface. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `osqueryConstants` | Class | Thrift-generated constants container for the osquery extensions namespace | -| `g_osquery_constants` | `extern const osqueryConstants` | Global singleton instance of the constants class | +- **`osqueryConstants`** β€” Thrift-generated constants class for the osquery extensions namespace (currently empty; populated if constants are defined in the `.thrift` IDL) +- **`g_osquery_constants`** β€” Global `extern` instance of `osqueryConstants`, providing process-wide access to any defined constants +- **`osquery::extensions` namespace** β€” Encapsulates all Thrift-generated osquery extension types and constants ## Usage Example @@ -14,8 +13,11 @@ Auto-generated Thrift constants header for the `osquery` extensions namespace. D #include "osquery_constants.h" // Access the global constants instance -const osquery::extensions::osqueryConstants& constants = +const osquery::extensions::osqueryConstants& consts = osquery::extensions::g_osquery_constants; + +// Use defined constants (populated if declared in the .thrift source) +// e.g., consts.someDefinedConstant ``` -> **⚠️ Do not manually edit this file.** It is auto-generated by the Thrift Compiler (v0.13.0). Any changes will be overwritten on regeneration. Modify the upstream `.thrift` schema definition instead. \ No newline at end of file +> **Note:** This file is auto-generated by Thrift 0.13.0. Do not edit manually. To add or modify constants, update the upstream `.thrift` IDL file and regenerate. \ No newline at end of file diff --git a/osquery/extensions/thrift/gen/.osquery_types.md b/osquery/extensions/thrift/gen/.osquery_types.md index 117febb4c4b..2ea2f2bc536 100644 --- a/osquery/extensions/thrift/gen/.osquery_types.md +++ b/osquery/extensions/thrift/gen/.osquery_types.md @@ -1,5 +1,5 @@ -Autogenerated Thrift header defining the core data types and structures used for osquery extension IPC communication within the `osquery::extensions` namespace. +Autogenerated Apache Thrift header defining the core data types and structures used for osquery's extension IPC (inter-process communication) protocol within the `osquery::extensions` namespace. ## Key Components @@ -18,34 +18,42 @@ Autogenerated Thrift header defining the core data types and structures used for | `InternalExtensionList` | `map` | ### Classes -- **`InternalOptionInfo`** β€” Holds a config option's `value`, `default_value`, and `type` strings +- **`InternalOptionInfo`** β€” Holds a plugin option's `value`, `default_value`, and `type` strings - **`InternalExtensionInfo`** β€” Describes a registered extension: `name`, `version`, `sdk_version`, `min_sdk_version` - **`ExtensionStatus`** β€” Response status with integer `code`, `message`, and `uuid` -- **`ExtensionResponse`** β€” Combines an `ExtensionStatus` with an `ExtensionPluginResponse` payload -- **`ExtensionException`** β€” Throwable Thrift exception carrying `code`, `message`, and `uuid` +- **`ExtensionResponse`** β€” Wraps an `ExtensionStatus` with an `ExtensionPluginResponse` payload +- **`ExtensionException`** β€” Throwable Thrift exception mirroring `ExtensionStatus` fields -All classes inherit from `apache::thrift::TBase` and support Thrift protocol `read()`/`write()`, equality operators, and `std::ostream` output. +All classes inherit from `apache::thrift::TBase` (or `TException`) and support Thrift serialization via `read()`/`write()`. ## Usage Example ```cpp #include "osquery_types.h" -osquery::extensions::ExtensionStatus status; -status.__set_code(osquery::extensions::ExtensionCode::EXT_SUCCESS); +using namespace osquery::extensions; + +// Build a status response +ExtensionStatus status; +status.__set_code(ExtensionCode::EXT_SUCCESS); status.__set_message("OK"); status.__set_uuid(42); -osquery::extensions::ExtensionResponse resp; +// Wrap in a full response with plugin data +ExtensionResponse resp; resp.__set_status(status); -// Serialize over a Thrift protocol -resp.write(protocol); +ExtensionPluginRequest req; +req["action"] = "query"; +req["sql"] = "SELECT * FROM processes"; -// Check result -if (resp.status.code == osquery::extensions::ExtensionCode::EXT_SUCCESS) { - // handle success +// Raise on failure +if (status.code != ExtensionCode::EXT_SUCCESS) { + ExtensionException ex; + ex.__set_code(status.code); + ex.__set_message(status.message); + throw ex; } ``` -> **Note:** This file is autogenerated by the Thrift 0.13.0 compiler. Do not edit manually β€” regenerate from the `.thrift` IDL source instead. \ No newline at end of file +> **Note:** This file is autogenerated by Thrift Compiler 0.13.0. Do not edit manually β€” regenerate from the `.thrift` IDL source instead. \ No newline at end of file diff --git a/osquery/filesystem/.filesystem.md b/osquery/filesystem/.filesystem.md index 0867794ccee..66e5c9bf4b2 100644 --- a/osquery/filesystem/.filesystem.md +++ b/osquery/filesystem/.filesystem.md @@ -1,39 +1,41 @@ -Filesystem utility header for osquery providing cross-platform file I/O, directory traversal, glob pattern resolution, and Linux-specific `/proc` filesystem helpers. +Provides cross-platform filesystem abstraction utilities for osquery, including file I/O, directory traversal, glob pattern resolution, permission checks, and Linux-specific `/proc` filesystem helpers. ## Key Components ### Enums & Constants - **`GlobLimits`** β€” Bitmask enum controlling glob traversal: `GLOB_FILES`, `GLOB_FOLDERS`, `GLOB_ALL`, `GLOB_NO_CANON` -- **`kSQLGlobWildcard`** / **`kSQLGlobRecursive`** β€” SQL `%` wildcard constants used in path pattern matching +- **`kSQLGlobWildcard`** / **`kSQLGlobRecursive`** β€” SQL-style `%` and `%%` glob tokens ### File I/O -- **`readFile()`** β€” Reads file contents into a string or streams chunks via callback; respects `read_max` flag and handles special files (FIFOs, named pipes) safely -- **`writeTextFile()`** β€” Writes text to disk with configurable permissions and open mode flags -- **`isWritable()` / `isReadable()`** β€” Permission checks with optional effective UID support +- **`readFile(path, content)`** β€” Non-blocking file read, respects `read_max` flag; handles special files (FIFOs, Named Pipes) +- **`readFile(path, predicate)`** β€” Chunked file read with callback +- **`writeTextFile(path, content, permissions, mode)`** β€” Write text to disk with configurable permissions +- **`parseJSON(path, tree)`** β€” Parse a JSON file into a Boost property tree -### Path Utilities -- **`pathExists()`** β€” Checks disk presence; returns `-1` (no input), `0` (not found), or `1` (found) -- **`isDirectory()`** β€” Tests if a path is a directory -- **`removePath()` / `movePath()`** β€” Delete or relocate files and directories -- **`createDirectory()`** β€” Creates directories with optional recursive and ignore-existing flags -- **`safePermissions()`** β€” Validates file ownership is root or current process UID (not world-writable) +### Path & Directory Utilities +- **`pathExists(path)`** β€” Check if a path exists on disk +- **`isDirectory(path)`** β€” Check if a path is a directory +- **`isReadable(path)`** / **`isWritable(path)`** β€” Permission checks (supports effective UID) +- **`listFilesInDirectory(path, results)`** β€” List files, optionally recursive +- **`listDirectoriesInDirectory(path, results)`** β€” List subdirectories, optionally recursive +- **`createDirectory(path)`** β€” Create directory with optional recursive and ignore-exists flags +- **`removePath(path)`** / **`movePath(from, to)`** β€” Remove or move files/directories +- **`getHomeDirectories()`** β€” Returns all system home directory paths -### Glob & Listing -- **`resolveFilePattern()`** β€” Expands SQL-style glob patterns into matching path lists -- **`replaceGlobWildcards()`** β€” Converts SQL `%` wildcards to filesystem `*` with optional path canonicalization -- **`listFilesInDirectory()` / `listDirectoriesInDirectory()`** β€” Enumerate directory contents, optionally recursive +### Glob Resolution +- **`resolveFilePattern(pattern, results)`** β€” Resolve SQL-style glob patterns to matching paths +- **`replaceGlobWildcards(pattern)`** β€” Convert SQL `%` wildcards to filesystem `*` globs, with path canonicalization -### System Helpers -- **`getHomeDirectories()`** β€” Returns all user home directories on the system -- **`osqueryHomeDirectory()`** β€” Returns osquery's protected local storage path -- **`parseJSON()`** β€” Parses a JSON file from disk into a Boost property tree -- **`lsperms()`** β€” Converts numeric permission mode to `ls`-style string +### Security & Misc +- **`safePermissions(dir, path)`** β€” Validate file ownership/permissions against `/tmp`-style attack vectors +- **`osqueryHomeDirectory()`** β€” Returns osquery's protected home directory path +- **`lsperms(mode)`** β€” Convert numeric permission bitmask to `ls`-style string ### Linux-Only (`#ifdef __linux__`) -- **`procProcesses()`** β€” Lists running PIDs from `/proc` -- **`procDescriptors()`** β€” Enumerates open file descriptors for a given PID -- **`procReadDescriptor()`** β€” Resolves a descriptor's virtual symlink path +- **`procProcesses(processes)`** β€” List all PIDs from `/proc` +- **`procDescriptors(process, descriptors)`** β€” List open file descriptors for a PID +- **`procReadDescriptor(process, descriptor, result)`** β€” Resolve a descriptor's virtual path ## Usage Example @@ -43,29 +45,20 @@ Filesystem utility header for osquery providing cross-platform file I/O, directo // Read a file std::string content; auto status = osquery::readFile("/etc/os-release", content); -if (!status.ok()) { - LOG(ERROR) << status.getMessage(); +if (status.ok()) { + // use content } -// Stream large file in chunks +// Glob resolve with SQL-style wildcard +std::vector results; +osquery::resolveFilePattern("/etc/cron.d/%", results, osquery::GLOB_FILES); + +// Chunked read with callback osquery::readFile("/var/log/syslog", [](std::string_view chunk) { - processChunk(chunk); + // process chunk }); -// Resolve glob pattern (SQL-style wildcards) -std::vector matches; -osquery::resolveFilePattern("/home/%/.ssh/%.pub", matches, osquery::GLOB_FILES); - -// Write with explicit permissions -osquery::writeTextFile("/tmp/report.txt", "data", 0644); - -// Linux: iterate /proc -#ifdef __linux__ +// Linux: iterate /proc processes std::set pids; osquery::procProcesses(pids); -for (const auto& pid : pids) { - std::map fds; - osquery::procDescriptors(pid, fds); -} -#endif ``` \ No newline at end of file diff --git a/osquery/filesystem/.mock_file_structure.md b/osquery/filesystem/.mock_file_structure.md index 0ecfcc8c2c9..5c6b4cf2bff 100644 --- a/osquery/filesystem/.mock_file_structure.md +++ b/osquery/filesystem/.mock_file_structure.md @@ -1,34 +1,29 @@ -Provides a test utility for generating a temporary mock directory structure used in osquery filesystem-related unit tests. +Utility header for creating a temporary mock directory structure used in osquery unit tests. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `kTopLevelMockFolderName` | `extern const std::string` | Name constant for the top-level mock folder | -| `createMockFileStructure()` | `boost::filesystem::path` | Creates a small temporary directory tree for testing | +- **`kTopLevelMockFolderName`** β€” Extern constant holding the name of the top-level mock folder. +- **`createMockFileStructure()`** β€” Creates a small temporary directory tree on disk and returns the path to its root. ## Usage Example ```c #include "mock_file_structure.h" -#include -namespace osquery { +using namespace osquery; -void myFilesystemTest() { - // Create a temporary mock directory tree - boost::filesystem::path mockRoot = createMockFileStructure(); +void myTest() { + // Create a temporary mock directory structure for testing + boost::filesystem::path mockRoot = createMockFileStructure(); - // Use mockRoot in test assertions - // e.g., verify files exist under kTopLevelMockFolderName - boost::filesystem::path topLevel = mockRoot / kTopLevelMockFolderName; - - // Clean up after test - boost::filesystem::remove_all(mockRoot); + // Use mockRoot as the base path for file system test assertions + boost::filesystem::path testFile = mockRoot / kTopLevelMockFolderName / "somefile.txt"; } - -} // namespace osquery ``` -> **Note:** This header is intended exclusively for test environments. Do not use `createMockFileStructure()` in production code paths, as it generates temporary filesystem artifacts on the host. \ No newline at end of file +## Notes + +- Intended for **test environments only** β€” not for production use. +- The returned path points to the generated directory tree root. +- Depends on `boost::filesystem` for cross-platform path handling. \ No newline at end of file diff --git a/osquery/filesystem/linux/.mounts.md b/osquery/filesystem/linux/.mounts.md index d3dfe9984dd..f3db560540a 100644 --- a/osquery/filesystem/linux/.mounts.md +++ b/osquery/filesystem/linux/.mounts.md @@ -1,33 +1,32 @@ -Declares data structures and a utility function for querying mounted filesystem information within the osquery framework. +Declares data structures and a query function for retrieving mounted filesystem information within the osquery framework. ## Key Components -### Structs - -- **`MountInformation::StatFsInfo`** β€” Nested struct holding low-level `statfs` metrics: - - `block_size` β€” Optimal transfer block size (`f_bsize`) - - `block_count` β€” Total data blocks (`f_blocks`) - - `free_block_count` β€” Free blocks (`f_bfree`) - - `unprivileged_free_block_count` β€” Free blocks available to unprivileged users (`f_bavail`) - - `inode_count` β€” Total inodes (`f_files`) - - `free_inode_count` β€” Free inodes (`f_ffree`) - -- **`MountInformation`** β€” Describes a single mounted filesystem: - - `type` β€” Filesystem type (e.g., `ext4`, `tmpfs`) - - `device` β€” Raw device path - - `device_alias` β€” Canonicalized device path - - `path` β€” Mount point - - `flags` β€” Mount options - - `optional_statfs_info` β€” Optional `StatFsInfo`; absent if `statfs` failed - -### Type Aliases - -- **`MountedFilesystems`** β€” `std::vector` representing all active mounts - -### Functions - -- **`getMountedFilesystems(MountedFilesystems&)`** β€” Populates the provided vector with information about all currently mounted filesystems; returns an osquery `Status` +### `MountInformation` +A struct describing a single mounted filesystem, containing: +- **`StatFsInfo`** β€” Nested struct wrapping `statfs` metrics: + - `block_size` β€” Optimal transfer block size + - `block_count` β€” Total data blocks + - `free_block_count` β€” Free blocks in filesystem + - `unprivileged_free_block_count` β€” Free blocks available to unprivileged users + - `inode_count` β€” Total file nodes + - `free_inode_count` β€” Free file nodes +- `type` β€” Filesystem type string +- `device` β€” Raw device path +- `device_alias` β€” Canonicalized device path +- `path` β€” Mount point path +- `flags` β€” Mount option flags +- `optional_statfs_info` β€” `boost::optional` (absent if `statfs` call failed) + +### `MountedFilesystems` +Type alias for `std::vector` β€” a collection of all active mount entries. + +### `getMountedFilesystems()` +```c +Status getMountedFilesystems(MountedFilesystems& mounted_fs_info); +``` +Populates `mounted_fs_info` with all currently mounted filesystems. Returns an osquery `Status` indicating success or failure. ## Usage Example @@ -39,13 +38,14 @@ auto status = osquery::getMountedFilesystems(mounts); if (status.ok()) { for (const auto& mount : mounts) { - std::cout << mount.path << " -> " << mount.device - << " [" << mount.type << "]\n"; + std::cout << "Device: " << mount.device + << " -> Path: " << mount.path + << " Type: " << mount.type << "\n"; if (mount.optional_statfs_info) { - auto& fs = *mount.optional_statfs_info; + const auto& fs = *mount.optional_statfs_info; std::cout << " Blocks: " << fs.block_count - << ", Free: " << fs.free_block_count << "\n"; + << " Free: " << fs.free_block_count << "\n"; } } } diff --git a/osquery/filesystem/linux/.proc.md b/osquery/filesystem/linux/.proc.md index b43d18e1a1f..a0c276c2b91 100644 --- a/osquery/filesystem/linux/.proc.md +++ b/osquery/filesystem/linux/.proc.md @@ -1,39 +1,37 @@ -Linux `/proc` filesystem parsing utilities for osquery, providing socket enumeration, process namespace inspection, and file descriptor traversal on Linux systems. +Linux `/proc` filesystem utility header for osquery, providing data structures and functions to enumerate processes, parse socket information, and read file descriptors from the Linux process filesystem. ## Key Components ### Data Structures -| Type | Description | -|------|-------------| -| `SocketInfo` | Holds socket metadata: family, protocol, local/remote address+port, state, and network namespace | -| `SocketProcessInfo` | Maps a socket to its owning process (`pid` + `fd`) | -| `SocketInodeToProcessInfoMap` | `map` keyed by socket inode | -| `ProcessNamespaceList` | `map` of namespace name to inode | +- **`SocketInfo`** β€” Holds socket metadata: family, protocol, local/remote address+port, Unix socket path, network namespace, and connection state +- **`SocketProcessInfo`** β€” Maps a socket inode to its owning process ID and file descriptor +- **`ProcessNamespaceList`** β€” Maps namespace names to inode numbers ### Constants -- `kLinuxProcPath` β€” `/proc` root path -- `kLinuxProtocolNames` β€” maps `IPPROTO_*` values to `/proc/net` filenames -- `tcp_states` β€” ordered TCP state name lookup by index +- `kLinuxProcPath` β€” Base path `/proc` +- `kLinuxProtocolNames` β€” Protocol number β†’ `/proc/net` filename mapping (icmp, tcp, udp, etc.) +- `tcp_states` β€” TCP connection state strings indexed by kernel state number -### Functions +### Free Functions | Function | Description | -|----------|-------------| -| `procGetSocketList` | Builds `SocketInfoList` from `/proc//net` for a given family/protocol | -| `procGetSocketListPacket` | Parses `/proc/net/packet` content into `SocketInfoList` | -| `procGetSocketInodeToProcessInfoMap` | Maps socket inodes to owning process via `/proc//fd` | +|---|---| +| `procGetSocketList` | Builds socket info list from `/proc//net` for a given family/protocol | +| `procGetSocketListPacket` | Parses `/proc/net/packet` file contents into `SocketInfoList` | +| `procGetSocketInodeToProcessInfoMap` | Maps socket inodes to owning process info via `/proc//fd` | | `procGetProcessNamespaces` | Reads namespace inodes from `/proc//ns` | -| `procDecodeAddressFromHex` | Decodes hex-encoded addresses from `/proc/net` files for `AF_INET`/`AF_INET6` | -| `procDecodePortFromHex` | Decodes hex-encoded port numbers | -| `getProcRSS` | Returns RSS memory usage for a process | +| `procGetNamespaceInode` | Parses inode from a namespace symlink | +| `procDecodeAddressFromHex` | Decodes hex-encoded IP address (IPv4/IPv6) from `/proc/net` files | +| `procDecodePortFromHex` | Decodes hex-encoded port number | +| `getProcRSS` | Returns resident set size for a process | -### Templates +### Template Functions -- `procEnumerateProcesses` β€” Iterates all PIDs under `/proc`, invoking a callback per process; stops on first `false` return -- `procEnumerateProcessDescriptors` β€” Iterates `/proc//fd`, resolving symlinks and invoking a callback per descriptor +- **`procEnumerateProcesses`** β€” Iterates all PIDs under `/proc`, invoking a callback per process; stops on first `false` return +- **`procEnumerateProcessDescriptors`** β€” Iterates all open file descriptors for a given PID under `/proc//fd`, resolving symlink targets ## Usage Example @@ -43,15 +41,15 @@ struct MyData { SocketInfoList sockets; }; MyData data; procEnumerateProcesses(data, [](const std::string& pid, MyData& d) -> bool { - procGetSocketList(AF_INET, IPPROTO_TCP, 0, pid, d.sockets); - return true; // continue iteration + procGetSocketList(AF_INET, IPPROTO_TCP, 0, pid, d.sockets); + return true; // continue iteration }); // Decode a hex address from /proc/net/tcp -std::string addr = procDecodeAddressFromHex("0100007F", AF_INET); -// addr == "127.0.0.1" +std::string addr = procDecodeAddressFromHex("0F02000A", AF_INET); +// addr == "10.0.2.15" -// Map socket inodes to processes +// Get socket inode β†’ process mapping SocketInodeToProcessInfoMap inodeMap; procGetSocketInodeToProcessInfoMap("1234", inodeMap); ``` \ No newline at end of file diff --git a/osquery/filesystem/posix/.xattrs.md b/osquery/filesystem/posix/.xattrs.md index e9ae5a43997..858b65df50f 100644 --- a/osquery/filesystem/posix/.xattrs.md +++ b/osquery/filesystem/posix/.xattrs.md @@ -1,38 +1,48 @@ -Provides declarations for reading extended attributes (xattrs) from files on supported filesystems, exposing typed error enums and `Expected`-based result types for safe attribute retrieval. +Provides declarations for reading POSIX extended attributes (xattrs) from files, including error types, result aliases, and core retrieval functions used within the osquery framework. ## Key Components -### Error Enums -| Enum | Values | Description | -|------|--------|-------------| -| `XAttrFileError` | `NoLength`, `List`, `SizeChanged` | Errors when listing attribute names | -| `XAttrValueError` | `NoLength`, `Get`, `SizeChanged` | Errors when reading an attribute value | -| `XAttrGetError` | `GenericError`, `NoFile` | Top-level errors for the combined get operation | +### Error Enumerations +- **`XAttrFileError`** β€” Errors when listing attribute names (`NoLength`, `List`, `SizeChanged`) +- **`XAttrValueError`** β€” Errors when reading an attribute value (`NoLength`, `Get`, `SizeChanged`) +- **`XAttrGetError`** β€” Top-level retrieval errors (`GenericError`, `NoFile`) ### Type Aliases -- **`ExtendedAttributeValue`** β€” `vector` raw byte buffer for an attribute's value -- **`ExtendedAttributeMap`** β€” `unordered_map` mapping attribute names to their values -- **`XAttrGetResult`** / **`XAttrNameListResult`** / **`XAttrValueResult`** β€” `Expected` wrappers for fallible return types +- **`ExtendedAttributeValue`** β€” Raw attribute data as `std::vector` +- **`ExtendedAttributeMap`** β€” Map of attribute name β†’ value (`unordered_map`) +- **`XAttrGetResult`** β€” `Expected` +- **`XAttrNameListResult`** β€” `Expected, XAttrFileError>` +- **`XAttrValueResult`** β€” `Expected` ### Functions -- **`xAttrFileErrorToString`** β€” Converts a `XAttrFileError` to a human-readable string given a file path -- **`xAttrValueErrorToString`** β€” Converts a `XAttrValueError` to a human-readable string given a path and attribute name -- **`getExtendedAttributesNames(int fd)`** β€” Lists all xattr names on an open file descriptor -- **`getExtendedAttributeValue(int fd, name)`** β€” Reads a single xattr value by name from an open file descriptor -- **`getExtendedAttributes(path)`** β€” High-level call: opens the file by path and returns all attributes as a map +| Function | Description | +|---|---| +| `getExtendedAttributes(path)` | Retrieves all xattrs for a file by path | +| `getExtendedAttributesNames(fd)` | Lists all xattr names for an open file descriptor | +| `getExtendedAttributeValue(fd, name)` | Reads a single xattr value by name from an open fd | +| `xAttrFileErrorToString(error, path)` | Converts a `XAttrFileError` to a human-readable string | +| `xAttrValueErrorToString(error, path, name)` | Converts a `XAttrValueError` to a human-readable string | ## Usage Example ```c -auto result = osquery::getExtendedAttributes("/path/to/file"); -if (result.isError()) { +#include "xattrs.h" + +// Retrieve all extended attributes for a file path +auto result = osquery::getExtendedAttributes("/etc/myfile.conf"); +if (result.isValue()) { + const auto& attrs = result.get(); + for (const auto& [name, value] : attrs) { + // process each attribute name/value pair + } +} else { // handle XAttrGetError - return; } -for (const auto& [name, value] : result.get()) { - // name -> std::string (e.g. "user.comment") - // value -> std::vector +// Read a single attribute from an open file descriptor +auto valueResult = osquery::getExtendedAttributeValue(fd, "user.checksum"); +if (valueResult.isValue()) { + const auto& bytes = valueResult.get(); // std::vector } ``` \ No newline at end of file diff --git a/osquery/hashing/.hashing.md b/osquery/hashing/.hashing.md index dc6e457077c..35c98a7dfc5 100644 --- a/osquery/hashing/.hashing.md +++ b/osquery/hashing/.hashing.md @@ -1,49 +1,48 @@ -Provides hashing utilities for osquery, supporting MD5, SHA1, and SHA256 algorithms with hex or Base64 encoding, for buffers and filesystem paths. +Provides hashing utilities for the osquery framework, supporting MD5, SHA1, and SHA256 algorithms with hex or Base64 encoding for buffers and files. ## Key Components ### Enums -- **`HashType`** β€” Algorithm selector: `HASH_TYPE_MD5`, `HASH_TYPE_SHA1`, `HASH_TYPE_SHA256` -- **`HashEncodingType`** β€” Output encoding: `HASH_ENCODING_TYPE_HEX`, `HASH_ENCODING_TYPE_BASE64` +- **`HashType`** β€” Selects the hashing algorithm: `HASH_TYPE_MD5`, `HASH_TYPE_SHA1`, `HASH_TYPE_SHA256` +- **`HashEncodingType`** β€” Selects output encoding: `HASH_ENCODING_TYPE_HEX`, `HASH_ENCODING_TYPE_BASE64` ### Structs -- **`MultiHashes`** β€” Holds results for simultaneous multi-algorithm hashing (`md5`, `sha1`, `sha256` strings + a bitmask) +- **`MultiHashes`** β€” Holds simultaneous results for MD5, SHA1, and SHA256 with a bitmask indicating which were computed ### Class: `Hash` -Non-copyable (move-only) streaming hash context. +Non-copyable, move-enabled class for incremental hashing. | Method | Description | -|---|---| -| `Hash(HashType)` | Init with algorithm, defaults to hex encoding | -| `Hash(HashType, HashEncodingType)` | Init with algorithm and encoding | -| `update(buffer, size)` | Feed data incrementally | -| `digest()` | Finalize and return encoded hash string | +|--------|-------------| +| `Hash(HashType)` | Initialize with algorithm; defaults to hex encoding | +| `Hash(HashType, HashEncodingType)` | Initialize with algorithm and encoding | +| `update(buffer, size)` | Feed data chunks into the hash context | +| `digest()` | Finalize and return the encoded hash string | ### Free Functions - -| Function | Description | -|---|---| -| `hashFromFile(hash_type, path)` | Hash a file, returns hex string | -| `hashMultiFromFile(mask, path)` | Hash a file with multiple algorithms simultaneously | -| `hashFromBuffer(hash_type, buffer, size)` | Hash an in-memory buffer, returns hex string | +- **`hashFromFile(hash_type, path)`** β€” Hash a file's contents, returns hex string +- **`hashMultiFromFile(mask, path)`** β€” Compute multiple hashes from a file in one pass +- **`hashFromBuffer(hash_type, buffer, size)`** β€” Hash an in-memory buffer, returns hex string ## Usage Example ```cpp -// Streaming hash (large file chunks) -osquery::Hash hasher(HASH_TYPE_SHA256, HASH_ENCODING_TYPE_HEX); -hasher.update(chunk_ptr, chunk_size); +// Incremental hashing of a large buffer +osquery::Hash hasher(HASH_TYPE_SHA256); +hasher.update(chunk1, chunk1_size); +hasher.update(chunk2, chunk2_size); std::string result = hasher.digest(); // One-shot file hash std::string md5 = osquery::hashFromFile(HASH_TYPE_MD5, "/etc/hosts"); -// Multiple algorithms in a single file pass +// Compute MD5 + SHA256 simultaneously osquery::MultiHashes hashes = osquery::hashMultiFromFile( - HASH_TYPE_MD5 | HASH_TYPE_SHA256, "/var/log/syslog"); + HASH_TYPE_MD5 | HASH_TYPE_SHA256, "/etc/hosts"); +// hashes.md5, hashes.sha256 -// Buffer hash +// Hash a buffer directly std::string sha1 = osquery::hashFromBuffer( - HASH_TYPE_SHA1, data_ptr, data_size); + HASH_TYPE_SHA1, my_buffer, my_buffer_size); ``` \ No newline at end of file diff --git a/osquery/logger/.logger.md b/osquery/logger/.logger.md index 89cb679344b..46acb0efa8d 100644 --- a/osquery/logger/.logger.md +++ b/osquery/logger/.logger.md @@ -1,37 +1,28 @@ -Thin wrapper around `glog` that provides osquery-specific logging macros for consistent verbosity and log-line referencing across the codebase. +Logging utility header for the `osquery` namespace, providing convenience macros that wrap Google's `glog` library for consistent log output across the codebase. ## Key Components | Macro | Description | |-------|-------------| -| `TLOG` | Alias for `VLOG(1)` β€” verbose logging intended for table-level parsing events and non-critical edge cases | -| `RLOG(n)` | Prepends a formatted reference tag (e.g., `[Ref #42]`) to a log line for external search and triage | +| `TLOG` | Verbose log alias (`VLOG(1)`) intended for table-generated log lines such as parsing edge-cases | +| `RLOG(n)` | Prepends a formatted reference number string (e.g., `[Ref #42]`) to a log message for external lookup | ## Usage Example ```c -#include +#include "logger.h" -// Table-level verbose log (only shown when verbosity >= 1) -TLOG << "Parsed unexpected value in users table, skipping row."; +// Table-level verbose logging (only shown at verbosity level 1+) +TLOG << "Unexpected value encountered while parsing row data"; -// Log with a reference number for documentation/issue tracking -LOG(WARNING) << RLOG(101) << "Unexpected null pointer in process cache."; -``` - -**Output examples:** - -```text -// TLOG output (verbosity level 1): -I0101 12:00:00 table.cpp:42] Parsed unexpected value in users table, skipping row. - -// RLOG output: -W0101 12:00:00 cache.cpp:88] [Ref #101] Unexpected null pointer in process cache. +// Log with a reference number for documentation cross-linking +LOG(WARNING) << RLOG(99) << "Suspicious process path detected"; +// Output: [Ref #99] Suspicious process path detected ``` ## Notes -- `TLOG` is intended for **table developers** β€” use it for routine parse warnings, not critical errors. -- `RLOG(n)` is purely a string prefix; combine it with any standard `glog` severity macro (`LOG(WARNING)`, `LOG(ERROR)`, etc.). -- Verbosity for `TLOG` is controlled at runtime via CLI flags or osquery config, keeping table noise out of production logs by default. \ No newline at end of file +- Requires `glog` (``) as a dependency. +- `TLOG` output is suppressed unless the user enables verbosity via CLI flag or config, keeping default output clean. +- `RLOG(n)` expands at compile time using token stringification (`#n`), so `n` must be a literal token β€” not a variable. \ No newline at end of file diff --git a/osquery/main/.main.md b/osquery/main/.main.md index 9da5b3c6709..83869a0e584 100644 --- a/osquery/main/.main.md +++ b/osquery/main/.main.md @@ -1,32 +1,37 @@ -Platform entry point header that declares service installation/uninstallation functions and the primary osquery startup routine. +Primary entry-point declarations for the osquery daemon and service lifecycle, providing cross-platform initialization and Windows-specific service management interfaces. ## Key Components | Symbol | Type | Description | |--------|------|-------------| -| `installService` | Function | Installs osqueryd as a platform service (currently Windows-only) | -| `uninstallService` | Function | Removes the previously installed osqueryd service | -| `startOsquery` | Function | Platform-agnostic entry point that initializes the shell and daemon | +| `installService` | Function | Installs `osqueryd` as a platform service (Windows only) | +| `uninstallService` | Function | Removes the previously installed `osqueryd` service | +| `startOsquery` | Function | Platform-agnostic entry point that bootstraps the shell and daemon | ## Usage Example ```c #include "main.h" -// Install osqueryd as a Windows service -osquery::Status status = osquery::installService("C:\\osquery\\osqueryd.exe"); +// Windows: install osqueryd as a system service +osquery::Status status = osquery::installService("C:\\Program Files\\osquery\\osqueryd.exe"); if (!status.ok()) { - // handle error + // handle installation failure } -// Start the osquery shell/daemon (called from main()) -int result = osquery::startOsquery(argc, argv); +// Later, remove the service +osquery::Status uninstallStatus = osquery::uninstallService(); + +// Standard daemon/shell startup (called from main()) +int main(int argc, char* argv[]) { + return osquery::startOsquery(argc, argv); +} ``` ## Notes -- `installService` / `uninstallService` are **Windows-only** at runtime. POSIX platforms rely on an external `osqueryctl` companion script to handle daemon registration (launchd, systemd, init). -- Refactoring POSIX install flows into these methods is a known limitation tracked by the osquery authors. -- Both service functions return `osquery::Status` (from `osquery/utils/status/status.h`), enabling structured error handling. -- `startOsquery` accepts standard `argc`/`argv` and serves as the single cross-platform bootstrap call shared by both the shell and daemon binaries. \ No newline at end of file +- `installService` / `uninstallService` are **Windows-only** in practice; POSIX platforms delegate service installation to the companion `osqueryctl` script via launch daemons, init scripts, or systemd units. +- The POSIX install flow divergence is a known limitation β€” future refactoring should unify both paths under `installService` / `uninstallService`. +- All functions live in the `osquery` namespace. +- Depends on `osquery/utils/status/status.h` for the `Status` return type used in service operations. \ No newline at end of file diff --git a/osquery/main/harnesses/.fuzz_utils.md b/osquery/main/harnesses/.fuzz_utils.md index 149e34bba61..c9429b13209 100644 --- a/osquery/main/harnesses/.fuzz_utils.md +++ b/osquery/main/harnesses/.fuzz_utils.md @@ -1,9 +1,9 @@ -Provides a fuzzing utility interface for osquery, exposing an initialization function that disables core stateful features to create a clean environment for LLVM-based fuzz testing. +Provides fuzzing utility declarations for osquery, offering a helper to initialize osquery in a reduced-state mode suitable for fuzz testing. ## Key Components -- **`osqueryFuzzerInitialize(int* argc, char*** argv)`** β€” Disables core osquery features to reduce statefulness. Intended to be called within `LLVMFuzzerInitialize` before fuzz test execution begins. +- **`osqueryFuzzerInitialize(int* argc, char*** argv)`** β€” Disables core osquery features to minimize statefulness during fuzz testing. Intended to be called from within `LLVMFuzzerInitialize`. ## Usage Example @@ -11,13 +11,17 @@ Provides a fuzzing utility interface for osquery, exposing an initialization fun #include "fuzz_utils.h" extern "C" int LLVMFuzzerInitialize(int* argc, char*** argv) { - return osquery::osqueryFuzzerInitialize(argc, argv); + return osquery::osqueryFuzzerInitialize(argc, argv); } extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) { - // Fuzz target logic here - return 0; + // Your fuzz target logic here + return 0; } ``` -> **Note:** Always call `osqueryFuzzerInitialize` before any fuzz input is processed to ensure core subsystems (logging, eventing, etc.) are suppressed and do not introduce side effects during fuzzing runs. \ No newline at end of file +## Notes + +- Must be called before any fuzz input processing to ensure osquery internals are in a clean, low-state configuration. +- Designed for use with **libFuzzer** (`LLVMFuzzerInitialize` entry point). +- Reduces side effects from osquery's logging, eventing, and other subsystems during corpus-driven testing. \ No newline at end of file diff --git a/osquery/numeric_monitoring/.numeric_monitoring.md b/osquery/numeric_monitoring/.numeric_monitoring.md index 04024910d7d..935ba234006 100644 --- a/osquery/numeric_monitoring/.numeric_monitoring.md +++ b/osquery/numeric_monitoring/.numeric_monitoring.md @@ -1,53 +1,60 @@ -Defines the public interface for osquery's numeric monitoring subsystem, providing types, constants, and functions for recording time-series numeric data points to a monitoring backend. +Defines the public interface for osquery's numeric monitoring system, providing types and functions to record numeric data points and flush aggregation buffers via a plugin-based architecture. ## Key Components ### Structs -- **`RecordKeys`** β€” Field name constants for monitoring record attributes (`path`, `value`, `timestamp`, `pre_aggregation`, `sync`) -- **`HostIdentifierKeys`** β€” Field name constants for host identification (`name`, `scheme`) +- **`RecordKeys`** β€” Field name descriptors (`path`, `value`, `timestamp`, `pre_aggregation`, `sync`) used when serializing monitoring records +- **`HostIdentifierKeys`** β€” Descriptor keys (`name`, `scheme`) for host identity metadata ### Type Aliases -- **`Clock`** / **`TimePoint`** β€” `std::chrono::system_clock` aliases for consistent time handling -- **`ValueType`** β€” `long long int`, the numeric type for all monitored values +- **`Clock`** / **`TimePoint`** β€” Aliases for `std::chrono::system_clock` and its `time_point` +- **`ValueType`** β€” `long long int`, the numeric type for all recorded values ### Enum -- **`PreAggregationType`** β€” Aggregation strategy applied before data is sent: `None`, `Sum`, `Min`, `Max`, `Avg`, `Stddev`, percentiles (`P10`, `P50`, `P95`, `P99`) +- **`PreAggregationType`** β€” Aggregation strategy applied before data is sent: `None`, `Sum`, `Min`, `Max`, `Avg`, `Stddev`, `P10`, `P50`, `P95`, `P99` ### Functions -- **`record()`** β€” Submits a new numeric data point with optional pre-aggregation, sync mode, and timestamp -- **`flush()`** β€” Forces immediate flush of the pre-aggregation buffer -- **`hostIdentifierKeys()`** / **`recordKeys()`** β€” Accessors for canonical field name constants -- **`registryName()`** β€” Returns the plugin registry name for this monitoring system - -### Conversions -- **`to()`** β€” Serializes a `PreAggregationType` to string -- **`tryTo()`** β€” Parses a `PreAggregationType` from string, returning `Expected` +| Function | Description | +|---|---| +| `record(path, value, pre_aggregation, sync, time_point)` | Records a numeric data point to the monitoring system | +| `flush()` | Forces a flush of the pre-aggregation buffer | +| `hostIdentifierKeys()` | Returns canonical host identifier field keys | +| `recordKeys()` | Returns canonical record field keys | +| `registryName()` | Returns the plugin registry name for monitoring | + +### Templates +- **`to(PreAggregationType)`** β€” Converts a `PreAggregationType` to its string representation +- **`tryTo(std::string)`** β€” Parses a `PreAggregationType` from a string, returning `Expected` ## Usage Example ```cpp -#include "numeric_monitoring.h" +#include "osquery/utils/monitoring/numeric_monitoring.h" -// Record a simple counter with sum pre-aggregation +// Record a summed metric at the current time osquery::monitoring::record( - "osquery.worker.queries.executed", + "workers.query.execution_ms", 42LL, osquery::monitoring::PreAggregationType::Sum ); -// Record a latency value, forcing immediate delivery +// Record synchronously (blocks until sent) osquery::monitoring::record( - "osquery.worker.query.latency_ms", - 150LL, - osquery::monitoring::PreAggregationType::P95, - true /* sync */ + "workers.query.count", + 1LL, + osquery::monitoring::PreAggregationType::None, + /* sync = */ true ); -// Convert aggregation type to/from string -auto str = osquery::to(osquery::monitoring::PreAggregationType::Sum); -// str == "Sum" +// Convert enum to string for logging +std::string label = osquery::to( + osquery::monitoring::PreAggregationType::P95 +); -auto type = osquery::tryTo("P99"); -// type.get() == PreAggregationType::P99 +// Parse enum from config string +auto result = osquery::tryTo("Sum"); +if (result) { + osquery::monitoring::flush(); +} ``` \ No newline at end of file diff --git a/osquery/numeric_monitoring/.plugin_interface.md b/osquery/numeric_monitoring/.plugin_interface.md index 411756ccf7d..2c2e26f8357 100644 --- a/osquery/numeric_monitoring/.plugin_interface.md +++ b/osquery/numeric_monitoring/.plugin_interface.md @@ -1,39 +1,36 @@ -Defines the `NumericMonitoringPlugin` interface, an abstract base class for plugins that integrate with osquery's numeric monitoring system. +Defines the `NumericMonitoringPlugin` interface, providing a base class for numeric monitoring system plugins within the osquery framework. ## Key Components -- **`NumericMonitoringPlugin`** β€” Inherits from `Plugin`; provides the entry point for numeric monitoring backend implementations (e.g., filesystem-based storage). - - `call(const PluginRequest&, PluginResponse&)` β€” Overrides the base `Plugin::call` to handle incoming monitoring requests and produce responses. +- **`NumericMonitoringPlugin`** β€” Abstract plugin base class inheriting from `Plugin`, used to implement numeric monitoring backends (e.g., filesystem-based monitoring via `NumericMonitoringFilesystemPlugin`) +- **`call()`** β€” Overridden entry point that handles incoming `PluginRequest` and populates a `PluginResponse` ## Usage Example ```c -#include - -namespace osquery { - +// Implement a custom numeric monitoring backend class MyMonitoringPlugin : public NumericMonitoringPlugin { public: - Status call(const PluginRequest& request, PluginResponse& response) override { + Status call(const PluginRequest& request, + PluginResponse& response) override { // Extract monitoring path and value from request auto path = request.at("path"); auto value = request.at("value"); - // Persist or forward the numeric metric + // Write metric to custom backend storeMetric(path, value); return Status::success(); } }; +// Register plugin with osquery registry REGISTER(MyMonitoringPlugin, "numeric_monitoring", "my_backend"); - -} // namespace osquery ``` ## Notes -- Concrete implementations must override `call()` and register via `REGISTER`. -- The reference implementation is `NumericMonitoringFilesystemPlugin` in `osquery/numeric_monitoring/plugins/filesystem.h`. -- Relies on `osquery::numeric_monitoring` types for point values and pre-aggregation semantics. \ No newline at end of file +- Concrete implementations must override `call()` to handle metric ingestion +- The built-in reference implementation is `NumericMonitoringFilesystemPlugin` (`osquery/numeric_monitoring/plugins/filesystem.h`) +- Plugin requests carry monitoring data as key-value pairs aligned with the `numeric_monitoring` subsystem contract \ No newline at end of file diff --git a/osquery/numeric_monitoring/.pre_aggregation_cache.md b/osquery/numeric_monitoring/.pre_aggregation_cache.md index 4220542ae73..aca6adbb783 100644 --- a/osquery/numeric_monitoring/.pre_aggregation_cache.md +++ b/osquery/numeric_monitoring/.pre_aggregation_cache.md @@ -1,42 +1,45 @@ -Pre-aggregation cache for the osquery numeric monitoring system. Provides in-memory buffering and aggregation of monitoring data points before they are flushed to the monitoring backend. +Provides the pre-aggregation cache mechanism for osquery's numeric monitoring system, buffering and collapsing monitoring data points before they are flushed to the backend. ## Key Components -### `Point` -Represents a single monitoring observation unit with: -- `path_` β€” unique identifier/name for the metric sequence -- `value_` β€” the observed numeric value -- `pre_aggregation_type_` β€” aggregation strategy (e.g., sum, min, max) -- `time_point_` β€” timestamp of observation -- `tryToAggregate(const Point&)` β€” merges a new point into itself if `path_` and `pre_aggregation_type_` match; returns `true` on success, `false` otherwise - -### `PreAggregationCache` -Accumulates `Point` objects and handles deduplication via pre-aggregation: -- `addPoint(Point)` β€” inserts or aggregates a point into the internal store -- `takePoints()` β€” drains and returns all buffered points as a `std::vector` -- `size()` β€” returns current number of buffered points +### `Point` (class) +The atomic monitoring unit representing a single observed value. + +| Member | Type | Description | +|--------|------|-------------| +| `path_` | `std::string` | Unique identifier for the metric sequence | +| `value_` | `ValueType` | The observed numeric value | +| `pre_aggregation_type_` | `PreAggregationType` | Aggregation strategy (sum, min, max, etc.) | +| `time_point_` | `TimePoint` | Observation timestamp | + +**Methods:** +- `Point(path, value, pre_aggregation_type, time_point)` β€” constructs a monitoring point +- `tryToAggregate(new_point)` β€” attempts to merge `new_point` into this point; returns `true` if `path_` and `pre_aggregation_type_` match and aggregation was applied, `false` otherwise -Internally uses `points_index_` (path β†’ index map) and `points_` (ordered vector) to efficiently locate existing points for aggregation. +### `PreAggregationCache` (class) +Accumulates `Point` instances, deduplicating by path using in-place aggregation before dispatch. + +**Methods:** +- `addPoint(point)` β€” inserts or aggregates a point into the cache +- `takePoints()` β€” drains and returns all buffered points as a `std::vector` +- `size()` β€” returns the current number of distinct cached points ## Usage Example ```cpp -#include "pre_aggregation_cache.h" - using namespace osquery::monitoring; PreAggregationCache cache; -// Add monitoring points cache.addPoint(Point("cpu.usage", 42.0, PreAggregationType::Sum, now())); cache.addPoint(Point("cpu.usage", 18.0, PreAggregationType::Sum, now())); -// Same path + type: values are aggregated (e.g., stored as 60.0) - -// Flush buffered points to the monitoring backend -auto points = cache.takePoints(); -for (const auto& p : points) { - sendToBackend(p.path_, p.value_); -} -// Cache is now empty -``` \ No newline at end of file +// "cpu.usage" is aggregated in-place (value becomes 60.0) + +cache.addPoint(Point("mem.free", 1024.0, PreAggregationType::Min, now())); + +// Flush all points for dispatch +auto points = cache.takePoints(); // returns 2 points +``` + +> **Note:** `points_index_` maps each `path_` string to its position in `points_`, enabling O(1) lookup during aggregation without scanning the entire vector. \ No newline at end of file diff --git a/osquery/registry/.registry_factory.md b/osquery/registry/.registry_factory.md index 908d83f9544..094eddd184e 100644 --- a/osquery/registry/.registry_factory.md +++ b/osquery/registry/.registry_factory.md @@ -1,41 +1,47 @@ -Defines the `RegistryFactory` singleton and supporting registration infrastructure for managing osquery plugin registries, including static call dispatch, broadcast serialization, and extension route tracking. +Singleton factory that manages all osquery plugin registries, providing a unified interface for registering, looking up, and invoking plugins across both core and extension processes. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `RegistryFactory` | Singleton class | Central manager for all plugin registries; provides add, lookup, call, alias, and broadcast operations | -| `RegistryBroadcast` | Type alias | `std::map` representing all routes from all registries | -| `Registry` | Type alias | Convenience alias for `RegistryFactory` | -| `registries::AR` | Template class | Auto-registers a registry type at startup | -| `registries::AP

` | Template class | Auto-registers a plugin instance into an existing registry | -| `registries::RI` | Template struct | RAII trigger for registry auto-registration | -| `registries::PI

` | Template struct | RAII trigger for plugin auto-registration | -| `CREATE_REGISTRY` | Macro | Registers a registry class under a given name at static init time | -| `CREATE_LAZY_REGISTRY` | Macro | Same as `CREATE_REGISTRY` but marks the registry as lazy (skips `setUp`) | -| `REGISTER` | Macro | Registers a plugin class into a named registry | -| `REGISTER_INTERNAL` | Macro | Same as `REGISTER` but marks the plugin as optional/internal | +**`RegistryFactory`** β€” Non-copyable singleton (aliased as `Registry`) that owns all registry instances and handles: +- `call()` β€” Four overloads to invoke plugins by registry/item name or via the active plugin +- `getBroadcast()` / `addBroadcast()` / `removeBroadcast()` β€” Serialize and sync registry routes with extensions via `RouteUUID` +- `addAlias()` / `getAlias()` β€” Alias support for internal items (only the alias name is broadcast) +- `setActive()` / `getActive()` β€” Manage the active plugin per registry +- `exists()`, `names()`, `count()` β€” Introspection helpers +- `setExternal()` β€” Marks the process as an extension, forwarding internal events to core + +**`registries` namespace** β€” Auto-registration infrastructure: +- `AR` β€” Auto-registers a registry type via `RegistryFactory::get().add()` +- `AP

` β€” Auto-registers a plugin into an existing registry +- `RI` / `PI

` β€” Static initializer structs that trigger registration at program startup + +**Macros** β€” Declarative registration at global scope: + +| Macro | Purpose | +|---|---| +| `CREATE_REGISTRY` | Register a registry type (eager) | +| `CREATE_LAZY_REGISTRY` | Register a registry type (lazy `setUp`) | +| `REGISTER` | Register a plugin into a registry | +| `REGISTER_INTERNAL` | Register an optional/internal plugin | ## Usage Example ```cpp -// Define and register a custom registry -class MyRegistry : public RegistryPlugin { /* ... */ }; -CREATE_REGISTRY(MyRegistry, "my_registry"); +// Define and register a custom registry type +class HashRegistry : public RegistryType {}; +CREATE_REGISTRY(HashRegistry, "hash") -// Register a plugin into that registry -class MyPlugin : public Plugin { /* ... */ }; -REGISTER(MyPlugin, "my_registry", "my_plugin"); +// Register a plugin implementation into that registry +class MD5Plugin : public HashPlugin { /* ... */ }; +REGISTER(MD5Plugin, "hash", "md5") -// Call a registered plugin at runtime -PluginRequest request = {{"action", "run"}}; +// Invoke the plugin at runtime +PluginRequest request = {{"action", "compute"}, {"data", "hello"}}; PluginResponse response; -Status s = RegistryFactory::call("my_registry", "my_plugin", request, response); +Status s = RegistryFactory::call("hash", "md5", request, response); -// Check existence and use the active plugin -if (RegistryFactory::get().exists("my_registry", "my_plugin")) { - RegistryFactory::get().setActive("my_registry", "my_plugin"); - RegistryFactory::call("my_registry", request, response); -} +// Or call via the active plugin +RegistryFactory::get().setActive("hash", "md5"); +RegistryFactory::call("hash", request, response); ``` \ No newline at end of file diff --git a/osquery/registry/.registry_interface.md b/osquery/registry/.registry_interface.md index 5d52f255158..d555fe5cfc4 100644 --- a/osquery/registry/.registry_interface.md +++ b/osquery/registry/.registry_interface.md @@ -1,50 +1,51 @@ -Defines the core plugin registry abstraction layer for osquery, providing interfaces for registering, managing, and routing calls to plugins across both local and external (extension) processes. +Defines the core registry abstraction layer for osquery's plugin system, providing interfaces for registering, managing, and routing calls to named plugins both internally and via external extension processes. ## Key Components ### Type Aliases -- **`RegistryRoutes`** β€” Map of plugin name β†’ `PluginResponse` for broadcasting route info -- **`RouteUUID`** β€” `uint64_t` identifier for extension connections -- **`AddExternalCallback`** / **`RemoveExternalCallback`** β€” Function signatures for extension lifecycle hooks -- **`RegistryInterfaceRef`** β€” `shared_ptr` +- `AddExternalCallback` / `RemoveExternalCallback` β€” Callbacks for managing external plugin lifecycle +- `RegistryRoutes` β€” Map of plugin name to optional `PluginResponse` route info +- `RouteUUID` β€” `uint64_t` identifier for extension registrations ### `RegistryInterface` -Abstract base class (non-copyable) managing plugin lifecycle. Key methods: - -| Method | Description | -|---|---| -| `add()` | Register a plugin (pure virtual) | -| `remove()` | Unregister a plugin by name | -| `call()` | Dispatch a request to a named plugin | -| `addExternal()` / `removeExternal()` | Manage plugins from remote extensions by UUID | -| `getRoutes()` | Broadcast route table for extensions | -| `setUp()` | Initialize all registered plugins | -| `setActive()` | Designate a default plugin for nameless calls | +Abstract base class (non-copyable) for all registry types. Manages: +- Plugin registration (`add`, `remove`, `addPlugin`) +- External extension routing (`addExternal`, `removeExternal`) +- Plugin lookup (`exists`, `plugin`, `names`, `count`) +- Active plugin selection (`setActive`, `getActive`) +- Alias mapping (`addAlias`, `getAlias`) +- Route broadcasting (`getRoutes`) ### `RegistryType` -Templated concrete registry that downcasts `PluginRef` to the specific `PluginType`. Provides type-safe `add()` and `plugin()` accessors, plus trampoline methods into `PluginType::addExternal` / `removeExternal`. +Typed template subclass that: +- Enforces type safety via `dynamic_pointer_cast` on `add()` +- Provides trampoline methods (`addExternalPlugin`, `removeExternalPlugin`) delegating to the concrete `PluginType`'s static callbacks ### `AutoRegisterInterface` -Static registration utility for auto-loading registries and plugins at startup via `registries()` / `plugins()` collections. Consumed by `registryAndPluginInit()`. +Supports compile-time auto-registration of registries and plugins. Static accessors (`registries()`, `plugins()`) and inserters (`autoloadRegistry`, `autoloadPlugin`) power the `registryAndPluginInit()` bootstrap function. ## Usage Example ```cpp // Define a typed registry for a custom plugin type -class MyRegistry : public RegistryType { +class MyPluginType : public Plugin { /* ... */ }; + +class MyRegistry : public RegistryType { public: explicit MyRegistry() : RegistryType("my_registry", true) {} }; -// Add and invoke a plugin +// Add a plugin instance auto registry = std::make_shared(); -registry->add("my_plugin", std::make_shared()); +auto plugin = std::make_shared(); +Status s = registry->add("my_plugin", plugin, /*internal=*/false); -PluginRequest request = {{"action", "query"}}; -PluginResponse response; -auto status = registry->call("my_plugin", request, response); +// Call a plugin by name +PluginRequest req = {{"action", "query"}}; +PluginResponse resp; +registry->call("my_plugin", req, resp); -// Retrieve route info for extension broadcasting +// Inspect registered routes for extension broadcast RegistryRoutes routes = registry->getRoutes(); ``` \ No newline at end of file diff --git a/osquery/remote/.http_client.md b/osquery/remote/.http_client.md index 54fba76226a..c99b4aceed1 100644 --- a/osquery/remote/.http_client.md +++ b/osquery/remote/.http_client.md @@ -1,33 +1,45 @@ -HTTP client interface providing synchronous HTTP/HTTPS request capabilities built on Boost.Beast and OpenSSL, used by osquery's remote communication subsystem. +HTTP client header defining a Boost.Beast-based HTTP/HTTPS client for osquery's remote communication layer, including request/response types, URI parsing, and TLS configuration. ## Key Components ### `Client` -The primary HTTP client class supporting `GET`, `POST`, `PUT`, `HEAD`, and `DELETE` operations over plain or TLS-encrypted connections. +Main HTTP client class supporting both plain and TLS connections via Boost.Beast and OpenSSL. + +**Methods:** +- `put()` / `post()` β€” Send HTTP PUT/POST requests with a body and optional content type +- `get()` / `head()` / `delete_()` β€” Standard HTTP retrieval/inspection/deletion requests +- `setOptions()` β€” Reconfigure client behavior at runtime ### `Client::Options` -A fluent builder for configuring client behavior. Key settings include: +Fluent builder class for configuring client behavior. Returns `*this` for method chaining. -| Option | Description | +| Method | Description | |---|---| -| `ssl_connection(bool)` | Enable TLS wrapping | -| `timeout(int)` | Request timeout in seconds | -| `always_verify_peer(bool)` | Enforce peer certificate validation | -| `follow_redirects(bool)` | Auto-follow HTTP redirects | -| `proxy_hostname(string)` | Route through a proxy | -| `openssl_certificate(string)` | Server certificate for pinning | -| `openssl_ciphers(string)` | Allowed TLS cipher suites | +| `ssl_connection(bool)` | Enable/disable TLS wrapping | +| `timeout(int)` | Set request timeout | +| `always_verify_peer(bool)` | Enforce peer certificate verification | +| `follow_redirects(bool)` | Enable redirect following | +| `openssl_certificate(str)` | Set server certificate | +| `openssl_private_key_file(str)` | Set client private key path | +| `proxy_hostname(str)` | Set proxy host | + +### `HTTP_Request` +Template wrapper over Boost.Beast request types, adding URI parsing via `osquery::Uri`. -### `HTTP_Request` / `HTTP_Response` -Template wrappers extending Boost.Beast request/response types with URI parsing. `HTTP_Request` exposes `remoteHost()`, `remotePort()`, `remotePath()`, and `protocol()` helpers. +- `remoteHost()` β€” Extracts hostname from URL +- `remotePort()` β€” Extracts port from URL +- `remotePath()` β€” Returns path + query + fragment +- `protocol()` β€” Returns scheme (`http` or `https`) ### Type Aliases -- `Request` β†’ `HTTP_Request` -- `Response` β†’ `HTTP_Response` +```cpp +typedef HTTP_Request Request; +typedef HTTP_Response Response; +``` ### Constants -- `kInstanceMetadataAuthority` β€” IP `169.254.169.254` for cloud metadata services (EC2, Azure, etc.) +- `kInstanceMetadataAuthority` β€” Link-local address `169.254.169.254` for cloud metadata services (EC2, Azure) ## Usage Example @@ -45,8 +57,9 @@ osquery::http::Client client(opts); osquery::http::Request req("https://example.com/api/data"); auto response = client.get(req); -osquery::http::Request post_req("https://example.com/api/submit"); -auto result = client.post(post_req, R"({"key":"value"})", "application/json"); +// POST with body +osquery::http::Request postReq("https://example.com/api/ingest"); +auto postResp = client.post(postReq, "{\"key\":\"value\"}", "application/json"); ``` -> **Note:** SSL2, SSL3, and MD5 are explicitly disabled at compile time. On Windows, Boost.ASIO thread cleanup (`set_terminate_threads`) is initialized once via `std::call_once` to prevent resource leaks. \ No newline at end of file +> **Note:** SSL2, SSL3, and MD5 are explicitly disabled via preprocessor defines. On Windows, Boost.ASIO thread cleanup is initialized once via `std::call_once` to prevent handle leaks (fixes #4235, #5341). \ No newline at end of file diff --git a/osquery/remote/.requests.md b/osquery/remote/.requests.md index 9b517e57595..806f8702710 100644 --- a/osquery/remote/.requests.md +++ b/osquery/remote/.requests.md @@ -1,54 +1,44 @@ -Defines the core abstractions for osquery's remote communication layer, providing abstract base classes for transport and serialization mechanisms along with a templated `Request` class that composes them. +Defines the core abstractions for osquery's remote communication layer, providing pluggable transport and serialization interfaces along with a templated `Request` class that composes them. ## Key Components -| Component | Type | Description | -|---|---|---| +| Symbol | Type | Description | +|--------|------|-------------| | `compressString` | Function | GZip-compresses a string; applied post-serialization before transport | -| `Transport` | Abstract class | Base for transport implementations (HTTP, WebSockets, etc.) | -| `Serializer` | Abstract class | Base for serialization formats (JSON, XML, etc.) | -| `Request` | Template class | Composes a transport and serializer to execute remote calls | - -### `Transport` Interface - -| Method | Description | -|---|---| -| `setDestination()` | Sets the remote URI | -| `setSerializer()` | Binds a serializer instance | -| `sendRequest()` | Pure virtual β€” no-param request | -| `sendRequest(params, compress)` | Pure virtual β€” parameterized request with optional GZip | -| `getResponseStatus()` | Returns the response `Status` | -| `getResponseParams()` | Returns response body as `JSON` | -| `setOption(name, value)` | Sets transport-specific options | - -### `Serializer` Interface - -| Method | Description | -|---|---| -| `getContentType()` | Returns HTTP content-type string | -| `serialize(json, out)` | Encodes a `JSON` object to string | -| `deserialize(string, out)` | Decodes a string into a `JSON` object | +| `Transport` | Abstract class | Base for transport backends (e.g., HTTP/TLS); subclass and implement `sendRequest()` | +| `Serializer` | Abstract class | Base for serialization formats (e.g., JSON); subclass and implement `serialize()`/`deserialize()` | +| `Request` | Template class | Composes a transport and serializer to send typed remote requests | ## Usage Example ```cpp -// Instantiate a request with concrete transport and serializer types -osquery::Request req("https://example.com/api"); +#include +#include +#include -// Set options (e.g., enable compression) +using namespace osquery; + +// Instantiate a request with TLS transport and JSON serialization +Request req("https://example.com/api/endpoint"); + +// Optional: enable GZip compression on outgoing payloads req.setOption("compress", true); // Send a parameterized request -osquery::JSON params; +JSON params; params.add("key", "value"); -auto status = req.call(params); +Status s = req.call(params); -// Retrieve the response -osquery::JSON response; -if (req.getResponse(response).ok()) { - // process response.doc() +// Read the response +if (s.ok()) { + JSON response; + Status rs = req.getResponse(response); } ``` -> **Note:** `Request` has a private constructor accepting a custom `TTransport` shared pointer, reserved for unit testing via `FRIEND_TEST` grants to `TLSTransportsTests`. \ No newline at end of file +## Notes + +- `Transport` and `Serializer` are pure-virtual; provide your own subclasses to support new protocols or formats. +- `compressString` is called automatically by `Request::call(const JSON&)` when the `"compress"` option is set to `true`. +- The private `Request(destination, transport)` constructor is test-only, exposed via `FRIEND_TEST` macros for `TLSTransportsTests`. \ No newline at end of file diff --git a/osquery/remote/.uri.md b/osquery/remote/.uri.md index 176886f982c..89132dd9893 100644 --- a/osquery/remote/.uri.md +++ b/osquery/remote/.uri.md @@ -1,25 +1,22 @@ -A URI parsing utility class (derived from folly/uri) that breaks a URI string into its component parts: scheme, authority, host, port, path, query, and fragment. +A URI parsing utility class (adapted from folly/uri) that parses and provides structured access to URI components within the `osquery` namespace. ## Key Components -### `Uri` Class (`osquery` namespace) - -| Member | Description | -|--------|-------------| -| `Uri(const std::string& str)` | Constructor β€” parses a URI string; throws `std::invalid_argument` on failure | -| `scheme()` | Returns the lowercased scheme (e.g. `"http"`) | -| `host()` | Returns host, with square brackets for IPv6 (e.g. `"[::1]"`) | -| `hostname()` | Returns host without brackets β€” safe for `getaddrinfo()` and similar APIs | -| `port()` | Returns the port as `uint16_t` | -| `path()` | Returns the URI path component | -| `query()` | Returns the raw query string | -| `fragment()` | Returns the fragment (anchor) | -| `authority()` | Returns the combined authority string (host + port) | -| `setPort(uint16_t)` | Overrides the port and marks authority as present | -| `getQueryParams()` | Parses and returns query string as `vector>` β€” **not thread-safe on first call** | - -> **Note:** Component parts are **not** percent-decoded. Use `uriUnescape()` manually on authority/path, and `uriUnescape(..., UriEscapeMode::QUERY)` on query parameters. +### `Uri` class +- **Constructor** β€” `Uri(const std::string& str)`: Parses a URI string; throws `std::invalid_argument` on failure +- **`scheme()`** β€” Returns the lowercased scheme (e.g., `"http"`) +- **`host()`** β€” Returns host, including brackets for IPv6 (e.g., `"[::1]"`) +- **`hostname()`** β€” Returns host without IPv6 brackets; suitable for `getaddrinfo()` and similar APIs +- **`port()`** β€” Returns the port as `uint16_t` +- **`path()`** β€” Returns the URI path (e.g., `"/foo/bar"`) +- **`query()`** β€” Returns the raw query string (e.g., `"key=foo"`) +- **`fragment()`** β€” Returns the fragment/anchor (e.g., `"anchor"`) +- **`authority()`** β€” Returns the combined authority (host + port) +- **`getQueryParams()`** β€” Parses and returns query string as `vector>`; lazy, not thread-safe on first call +- **`setPort(uint16_t)`** β€” Overrides the port value + +> **Note:** Component parts are **not** percent-decoded. Call `uriUnescape()` separately as needed. ## Usage Example @@ -27,19 +24,18 @@ A URI parsing utility class (derived from folly/uri) that breaks a URI string in #include "uri.h" try { - osquery::Uri uri("http://user:pass@www.example.com:8080/foo/bar?key=val#anchor"); + osquery::Uri uri("http://www.example.com:8080/foo/bar?key=foo#anchor"); uri.scheme(); // "http" - uri.hostname(); // "www.example.com" (safe for getaddrinfo) - uri.host(); // "www.example.com" + uri.hostname(); // "www.example.com" uri.port(); // 8080 uri.path(); // "/foo/bar" - uri.query(); // "key=val" + uri.query(); // "key=foo" uri.fragment(); // "anchor" // Parse query parameters for (const auto& [key, value] : uri.getQueryParams()) { - // key="key", value="val" + // key="key", value="foo" } } catch (const std::invalid_argument& e) { // Handle malformed URI diff --git a/osquery/remote/enroll/.enroll.md b/osquery/remote/enroll/.enroll.md index 556a54703da..04d0e9ea5ec 100644 --- a/osquery/remote/enroll/.enroll.md +++ b/osquery/remote/enroll/.enroll.md @@ -1,42 +1,49 @@ -Header defining osquery's enrollment plugin interface and utilities for node authentication and secret management in distributed osquery deployments. +Header defining the enrollment plugin interface for osquery's node authentication system, enabling Config and Logger plugins to authenticate with remote backends. ## Key Components | Symbol | Type | Description | -|---|---|---| +|--------|------|-------------| | `FLAGS_disable_enrollment` | Flag | Boolean flag to disable all enrollment features | -| `kEnrollHostDetails` | `const std::set` | Table names used to populate host identification details sent during enrollment | -| `EnrollPlugin` | Abstract class | Base class for all enrollment plugins; extends `Plugin` | +| `kEnrollHostDetails` | `const std::set` | Table names used to populate host identification data sent during enrollment | +| `EnrollPlugin` | Class | Abstract base class for enrollment plugins; extends `Plugin` | | `EnrollPlugin::call()` | Method | Routes incoming `PluginRequest` actions to the appropriate handler | -| `EnrollPlugin::enroll()` | Pure virtual | Implemented by subclasses to return a node secret/key | -| `EnrollPlugin::genHostDetails()` | Protected method | Builds a JSON object from `kEnrollHostDetails` table data | -| `getNodeKey()` | Free function | Retrieves a cached node key from RocksDB or triggers enrollment | -| `clearNodeKey()` | Free function | Removes the existing node key from persistent storage | -| `getEnrollSecret()` | Free function | Reads and returns the deployment enrollment secret from disk | +| `EnrollPlugin::enroll()` | Virtual Method | Override to implement enrollment logic; returns a node secret/key | +| `EnrollPlugin::genHostDetails()` | Method | Builds a JSON object from `kEnrollHostDetails` table query results | +| `getNodeKey()` | Function | Retrieves a cached node key from RocksDB or triggers enrollment | +| `clearNodeKey()` | Function | Deletes the stored node key from persistent storage | +| `getEnrollSecret()` | Function | Reads a shared enrollment secret from disk | ## Usage Example ```cpp -// Implement a custom enroll plugin -class MyEnrollPlugin : public EnrollPlugin { +#include + +// Implement a custom enrollment plugin +class MyEnrollPlugin : public osquery::EnrollPlugin { protected: std::string enroll() override { - JSON host_details; - genHostDetails(host_details); // populate host identity info - // call remote endpoint with host_details... - return "my-node-secret-key"; + osquery::JSON host_details; + genHostDetails(host_details); + + // Exchange host details with your enrollment endpoint + // and return a node secret + return "my-secret-node-key-from-backend"; } }; -// Retrieve (or request) a node key using a named plugin -std::string node_key = getNodeKey("my_enroll_plugin"); +REGISTER(MyEnrollPlugin, "enroll", "my_enroll"); + +// Retrieve or generate a node key using the registered plugin +std::string node_key = osquery::getNodeKey("my_enroll"); -// Clear stored key to force re-enrollment on next run -Status s = clearNodeKey(); +// Re-enroll by clearing the existing key first +osquery::clearNodeKey(); +node_key = osquery::getNodeKey("my_enroll"); -// Read the shared enterprise enrollment secret -const std::string secret = getEnrollSecret(); +// Read a pre-shared enrollment secret from disk +std::string secret = osquery::getEnrollSecret(); ``` -> Enrollment plugins are best used as part of a coordinated **enroll β†’ config β†’ logger** plugin suite. See the [osquery enrollment docs](https://osquery.readthedocs.io/en/stable/deployment/remote/) for full deployment guidance. \ No newline at end of file +> Enrollment plugins are best deployed as part of a suite alongside matching Config and Logger plugins. See the [osquery enrollment documentation](https://osquery.readthedocs.io/en/stable/deployment/remote/) for full integration details. \ No newline at end of file diff --git a/osquery/remote/serializers/.json.md b/osquery/remote/serializers/.json.md index fe919882cd5..d5e168c954d 100644 --- a/osquery/remote/serializers/.json.md +++ b/osquery/remote/serializers/.json.md @@ -1,18 +1,16 @@ -Defines `JSONSerializer`, a concrete implementation of the `Serializer` interface for serializing and deserializing JSON payloads in osquery remote requests. +Defines the `JSONSerializer` class for serializing and deserializing JSON data in osquery's remote request infrastructure. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `JSONSerializer` | Class | Serializes/deserializes `JSON` objects to/from `std::string` | -| `serialize()` | Method | Converts a `JSON` object into a serialized string | -| `deserialize()` | Method | Parses a serialized string back into a `JSON` object | -| `getContentType()` | Method | Returns `"application/json"` as the MIME content type | +- **`JSONSerializer`** β€” Concrete implementation of the `Serializer` interface that handles JSON-formatted data for remote communications. + - `serialize(const JSON& json, std::string& serialized)` β€” Converts a `JSON` object into a string representation. + - `deserialize(const std::string& serialized, JSON& json)` β€” Parses a JSON string back into a `JSON` object. + - `getContentType()` β€” Returns `"application/json"` as the MIME content type. ## Usage Example -```c +```cpp #include osquery::JSONSerializer serializer; @@ -20,20 +18,13 @@ osquery::JSONSerializer serializer; // Serialize osquery::JSON json; std::string output; -auto status = serializer.serialize(json, output); -if (!status.ok()) { - // handle error -} +osquery::Status status = serializer.serialize(json, output); // Deserialize osquery::JSON parsed; -auto status2 = serializer.deserialize(output, parsed); -if (!status2.ok()) { - // handle error -} +osquery::Status status = serializer.deserialize(output, parsed); -// Content type for HTTP headers -std::string ct = serializer.getContentType(); // "application/json" -``` - -> `JSONSerializer` is a lightweight adapter β€” all methods delegate to the base `Serializer` contract defined in `osquery/remote/requests.h`. Wire up this class wherever a remote transport requires a `Serializer` instance with JSON encoding. \ No newline at end of file +// Content-Type header value +std::string contentType = serializer.getContentType(); +// "application/json" +``` \ No newline at end of file diff --git a/osquery/remote/tests/.test_utils.md b/osquery/remote/tests/.test_utils.md index 92a3174fa03..254dea587c5 100644 --- a/osquery/remote/tests/.test_utils.md +++ b/osquery/remote/tests/.test_utils.md @@ -1,48 +1,45 @@ -Provides a singleton test utility class for managing an embedded TLS server process during osquery integration tests, enabling client TLS configuration and lifecycle control. +Provides a singleton test utility class for managing a TLS server subprocess during osquery integration tests, including helpers to configure and tear down TLS client flags. ## Key Components -### `TLSServerRunner` (class) -A non-copyable singleton that spawns and manages a local TLS server subprocess for testing purposes. +### `TLSServerRunner` (class, `osquery` namespace) +Singleton (non-copyable) that spawns and manages a TLS server process for testing. | Member | Description | |--------|-------------| -| `instance()` | Returns the singleton `TLSServerRunner` instance | -| `start(server_cert, verify_client_cert)` | Launches the TLS server process; returns `false` on failure | +| `instance()` | Returns the singleton instance | +| `start(server_cert, verify_client_cert)` | Starts the TLS server subprocess; returns `false` on failure | | `stop()` | Terminates the server process on exit | -| `port()` | Returns the TCP port the server is bound to | -| `setClientConfig()` | Applies osquery flags needed for TLS client tests | +| `port()` | Returns the bound TCP port string | +| `setClientConfig()` | Sets osquery flags for TLS client test configuration | | `unsetClientConfig()` | Restores osquery flags after TLS client tests | -| `getListeningPortPid()` | Resolves the PID listening on a given port (empty if not found) | -| `startAndSetScript()` | Internal helper to spawn the server subprocess | +| `getListeningPortPid()` | (private) Resolves PID of process listening on the given port | +| `startAndSetScript()` | (private) Spawns the server script and assigns `server_` handle | -**Private state:** - -- `server_` β€” shared handle to the spawned `PlatformProcess` -- `port_` β€” bound TCP port string -- Saved flag values: `tls_hostname_`, `enroll_tls_endpoint_`, `tls_server_certs_`, `enroll_secret_path_` +**Private state:** `server_` (`PlatformProcess`), `port_`, and saved flag values (`tls_hostname_`, `enroll_tls_endpoint_`, `tls_server_certs_`, `enroll_secret_path_`). ## Usage Example ```cpp #include "test_utils.h" -// Start TLS server before tests -bool ok = osquery::TLSServerRunner::start("/path/to/server.crt", false); -ASSERT_TRUE(ok); - -// Configure the osquery client flags to point at the test server -osquery::TLSServerRunner::setClientConfig(); +// In a test fixture setup: +void SetUp() { + // Start TLS server with default cert, no client cert verification + ASSERT_TRUE(osquery::TLSServerRunner::start()); -// Access the bound port -std::string port = osquery::TLSServerRunner::port(); + // Configure osquery flags to point at the test server + osquery::TLSServerRunner::setClientConfig(); +} -// Run TLS-dependent tests... +void TearDown() { + osquery::TLSServerRunner::unsetClientConfig(); + osquery::TLSServerRunner::stop(); +} -// Restore flags and shut down -osquery::TLSServerRunner::unsetClientConfig(); -osquery::TLSServerRunner::stop(); +// Access the bound port if needed: +std::string url = "https://localhost:" + osquery::TLSServerRunner::port(); ``` -> **Note:** `TLSServerRunner` is non-copyable (inherits `boost::noncopyable`). Always access it through the `instance()` singleton or the provided static methods. \ No newline at end of file +> **Note:** `start()` is idempotent β€” calling it multiple times will not spawn duplicate server processes. Use `setClientConfig()` / `unsetClientConfig()` to bracket each test that exercises TLS client code. \ No newline at end of file diff --git a/osquery/remote/transports/.tls.md b/osquery/remote/transports/.tls.md index cc1cee8c5ac..3bb49aa053b 100644 --- a/osquery/remote/transports/.tls.md +++ b/osquery/remote/transports/.tls.md @@ -1,50 +1,47 @@ -HTTPS/TLS transport layer for osquery remote communications, providing secure HTTP client functionality with configurable certificate handling, cipher suite enforcement, and peer verification. +HTTPS/TLS transport layer implementation for osquery remote communications, providing secure request handling with configurable client authentication, server certificate pinning, and cipher suite enforcement. ## Key Components ### Constants & Flags -- `kTLSCiphers` β€” Hardcoded restrictive cipher suite string (ECDH/DH/RSA with AES/3DES, excludes weak algorithms) -- `tls_client_key` β€” CLI flag: path to TLS client private key -- `tls_client_cert` β€” CLI flag: path to TLS client certificate (PEM) -- `tls_hostname` β€” CLI flag: TLS server hostname +- `kTLSCiphers` β€” Hardened cipher suite string restricting weak/null ciphers (`!aNULL`, `!MD5`, `!CBC`, `!SHA`) +- `tls_client_key` β€” Flag: path to PEM client private key +- `tls_client_cert` β€” Flag: path to PEM client certificate +- `tls_hostname` β€” Flag: target TLS server hostname ### Enums -- `HTTPVerb` β€” Defines `HTTP_POST` and `HTTP_PUT` verb selectors +- `HTTPVerb` β€” Supported HTTP methods: `HTTP_POST`, `HTTP_PUT` ### Class: `TLSTransport` (extends `Transport`) - | Method | Description | |---|---| -| `sendRequest()` | Sends a parameterless HTTPS request | -| `sendRequest(params, compress)` | Sends a request with serialized parameters, optional gzip compression | -| `getInternalOptions()` | Returns strict options (limited ciphers + client certs) for osquery infrastructure | -| `getOptions()` | Returns general options for AWS or public internet endpoints | -| `decorateRequest(r)` | Applies base modifications to an HTTP request object | -| `disableVerifyPeer()` | Test-only: disables peer certificate verification | -| `setClientCertificate(cert, key)` | Configures mutual TLS client authentication | -| `setPeerCertificate(cert)` | Pins a specific server CA/certificate bundle | +| `sendRequest()` | Parameterless HTTPS request | +| `sendRequest(params, compress)` | HTTPS request with serialized body, optional compression | +| `getInternalOptions()` | Restrictive options (limited ciphers + client certs) for osquery infrastructure | +| `getOptions()` | General options for AWS or public internet endpoints | +| `decorateRequest(r)` | Applies base modifications to a request object | +| `setClientCertificate(cert, key)` | Configures mutual TLS client auth | +| `setPeerCertificate(cert)` | Pins a specific server CA/certificate | +| `disableVerifyPeer()` | Disables peer verification (test use only) | ## Usage Example ```cpp #include -// Basic TLS request with internal (strict) options -auto transport = std::make_shared(); +osquery::TLSTransport transport; -// For osquery infrastructure β€” enforces restrictive cipher suite -auto options = transport->getInternalOptions(); +// Use restrictive cipher suite for osquery infrastructure +auto options = transport.getInternalOptions(); -// For generic/AWS endpoints β€” uses permissive options -auto generalOptions = transport->getOptions(); +// Send a POST request with JSON payload +std::string payload = R"({"key":"value"})"; +auto status = transport.sendRequest(payload, /* compress= */ false); -// Send a POST request with JSON parameters -Status status = transport->sendRequest("{\"key\":\"value\"}", /* compress */ false); if (!status.ok()) { - // status.getCode() == 1: connectivity error - // status.getCode() == 2: TLS-specific error + // Code 1 = connectivity error, Code 2 = TLS-specific error + LOG(ERROR) << "Transport error: " << status.getMessage(); } ``` -> **Return codes:** `sendRequest` returns code `1` for general connectivity failures and code `2` for TLS-specific errors (e.g., handshake failure, certificate mismatch). \ No newline at end of file +> **Note:** Return code `1` signals general connectivity failures; return code `2` indicates TLS-specific errors (handshake, certificate validation, etc.). \ No newline at end of file diff --git a/osquery/sql/.dynamic_table_row.md b/osquery/sql/.dynamic_table_row.md index cce2fa01dbc..2245457ea52 100644 --- a/osquery/sql/.dynamic_table_row.md +++ b/osquery/sql/.dynamic_table_row.md @@ -1,46 +1,37 @@ -Defines the `DynamicTableRow` and `DynamicTableRowHolder` classes, providing a string-map-backed implementation of the `TableRow` interface for use in osquery's virtual table system. +Defines `DynamicTableRow` and `DynamicTableRowHolder`, providing a string-map-backed implementation of the `TableRow` interface for use in osquery's virtual table system. ## Key Components -### `DynamicTableRow` -Concrete `TableRow` subclass backed by a `Row` (string map). Supports: -- Construction from `Row&&`, initializer lists, or default empty state -- `get_rowid()` β€” retrieves the SQLite row ID -- `get_column()` β€” populates a SQLite column value from the internal map -- `serialize()` β€” serializes the row to a JSON object -- `clone()` β€” returns a deep copy wrapped in a `TableRowHolder` -- `operator[]` / `count()` β€” map-style key access - -### `DynamicTableRowHolder` -Convenience wrapper that owns a `DynamicTableRow*` alongside its `TableRowHolder`. Simplifies row construction and assignment before strong typing is fully adopted. Implicitly converts to `TableRowHolder&&`. - -### Factory Functions -- `make_table_row()` β€” creates an empty `DynamicTableRowHolder` -- `make_table_row({...})` β€” creates a pre-populated row from an initializer list - -### Free Functions -- `tableRowsFromQueryData(QueryData&&)` β€” converts legacy `QueryData` to `TableRows` -- `deserializeRow(const rapidjson::Value&, DynamicTableRowHolder&)` β€” populates a row from a JSON object +- **`DynamicTableRow`** β€” Concrete `TableRow` subclass wrapping a `Row` (string map). Implements `get_rowid`, `get_column`, `serialize`, and `clone` for SQLite virtual table integration. +- **`DynamicTableRowHolder`** β€” RAII wrapper around a heap-allocated `DynamicTableRow` with implicit conversion to `TableRowHolder&&`, simplifying ownership transfer. +- **`make_table_row()`** β€” Factory helpers returning a `DynamicTableRowHolder`, either empty or initialized from a key-value list. +- **`tableRowsFromQueryData()`** β€” Converts a `QueryData` struct into `TableRows`; intended for generated table code. +- **`deserializeRow()`** β€” Populates a `DynamicTableRowHolder` from a `rapidjson::Value` object. ## Usage Example -```c -// Construct a row with known columns +```cpp +// Create a row with initial key-value pairs auto row = make_table_row({ {"pid", "1234"}, - {"name", "launchd"}, - {"path", "/sbin/launchd"} + {"name", "my_process"}, + {"path", "/usr/bin/my_process"} }); -// Access or modify a field -row["state"] = "S"; +// Access and mutate fields +row["status"] = "running"; -// Convert to TableRowHolder for use in query results +// Transfer ownership into a TableRows collection TableRows results; -results.push_back(std::move(row)); +results.push_back(std::move((TableRowHolder&&)row)); // Deserialize from JSON DynamicTableRowHolder deserialized; Status s = deserializeRow(jsonValue, deserialized); -``` \ No newline at end of file +if (!s.ok()) { + LOG(ERROR) << "Deserialization failed: " << s.getMessage(); +} +``` + +> **Note:** `DynamicTableRowHolder` is transitional scaffolding β€” it is expected to be replaced once strongly typed row implementations are adopted throughout the codebase. \ No newline at end of file diff --git a/osquery/sql/.sql.md b/osquery/sql/.sql.md index 101d5f03db5..a186601c1fb 100644 --- a/osquery/sql/.sql.md +++ b/osquery/sql/.sql.md @@ -1,35 +1,35 @@ -Defines the core SQL query interface for osquery, providing both a high-level `SQL` class and lower-level utility functions for executing, analyzing, and inspecting SQLite-backed virtual table queries. +Defines the core SQL query interface for osquery, providing both a high-level `SQL` class and lower-level free functions for executing queries against osquery's virtual table system. ## Key Components ### `SQL` Class -The primary interface for executing osquery SQL queries. Inherits `only_movable` (no copy semantics). +The primary interface for executing osquery SQL queries. Wraps query execution with result storage and status reporting. | Member | Description | -|---|---| -| `SQL(query, use_cache)` | Constructor β€” executes query on instantiation | -| `rows()` | Returns `QueryData` results (const and mutable overloads) | -| `columns()` | Returns `ColumnNames` for result columns | +|--------|-------------| +| `SQL(query, use_cache)` | Constructor β€” executes the query immediately | +| `rows()` | Returns `QueryData` with result rows | +| `columns()` | Returns column name/order info | | `ok()` | Returns `true` if query succeeded | -| `getStatus()` | Returns the underlying `Status` object | +| `getStatus()` | Returns the full `Status` object | | `getMessageString()` | Returns human-readable status message | -| `selectAllFrom(table)` | Static: executes `SELECT * FROM ` | -| `selectAllFrom(table, col, op, expr)` | Static: `SELECT *` with a single WHERE constraint | -| `selectFrom(cols, table, col, op, expr)` | Static: `SELECT [cols]` with a single WHERE constraint | +| `selectAllFrom(table)` | Static β€” runs `SELECT * FROM
` | +| `selectAllFrom(table, col, op, expr)` | Static β€” runs `SELECT *` with a single WHERE constraint | +| `selectFrom(cols, table, col, op, expr)` | Static β€” runs `SELECT [cols]` with a WHERE constraint | ### Free Functions | Function | Description | -|---|---| -| `query(q, results, use_cache)` | Lower-level query execution; prefer `SQL` class | -| `getQueryColumns(q, columns)` | Introspects result column names and types via SQLite | +|----------|-------------| +| `query(q, results, use_cache)` | Lower-level query execution; populates a `QueryData` output parameter | +| `getQueryColumns(q, columns)` | Analyzes a query and returns result column names/types via SQLite | | `getQueryTables(q, tables)` | Extracts virtual table names referenced in a query | ## Usage Example -```cpp -// High-level usage via SQL class +```c +// High-level SQL class osquery::SQL sql("SELECT * FROM time"); if (sql.ok()) { for (const auto& row : sql.rows()) { @@ -41,16 +41,15 @@ if (sql.ok()) { LOG(ERROR) << sql.getMessageString(); } -// Static helper with constraint -auto results = osquery::SQL::selectAllFrom( - "processes", "pid", osquery::EQUALS, "1234" -); - -// Lower-level query function +// Lower-level free function osquery::QueryData results; -auto status = osquery::query("SELECT * FROM users;", results); +auto status = osquery::query("SELECT * FROM processes;", results); +if (!status.ok()) { + LOG(ERROR) << status.what(); +} -// Inspect columns before executing -osquery::TableColumns cols; -osquery::getQueryColumns("SELECT pid, name FROM processes", cols); +// Static helper with constraint +auto rows = osquery::SQL::selectAllFrom( + "users", "uid", osquery::EQUALS, "0" +); ``` \ No newline at end of file diff --git a/osquery/sql/.sqlite_util.md b/osquery/sql/.sqlite_util.md index 2d7de88818b..09a74f4ae4d 100644 --- a/osquery/sql/.sqlite_util.md +++ b/osquery/sql/.sqlite_util.md @@ -1,62 +1,48 @@ -Provides SQLite database management utilities for osquery, including RAII database instance wrappers, a singleton manager, query planning, and security authorization controls for safe SQL execution. +Provides SQLite database management utilities for osquery, including RAII wrappers, connection management, security authorization, and query planning infrastructure. ## Key Components -### `sqliteAuthorizer` -Callback function enforcing SQLite action allowlists (`kAllowedSQLiteActionCodes`, `kAllowedSQLitePragmas`). Blocks dangerous operations like `SQLITE_ATTACH`. +### Constants +- `SQLITE_SOFT_HEAP_LIMIT` β€” 5MB soft heap cap for SQLite memory usage +- `kAllowedSQLiteActionCodes` β€” Allowlist of permitted SQLite action codes (denies `SQLITE_ATTACH` to prevent arbitrary file writes) +- `kAllowedSQLitePragmas` β€” Allowlist of permitted SQLite pragmas +- `kSQLOpcodes` β€” Map of SQLite opcodes used for column type inference -### `SQLiteDBInstance` -RAII wrapper around a `sqlite3*` connection. Supports both managed (primary) and transient instances, virtual table cache control, and thread-safe attach locking. +### Functions +- `sqliteAuthorizer()` β€” SQLite authorizer callback enforcing the action/pragma allowlists +- `queryInternal()` β€” Execute a query against a specific `SQLiteDBInstanceRef` -| Method | Description | -|---|---| -| `db()` | Access the raw `sqlite3*` pointer | -| `isPrimary()` | Check if using the shared primary DB | -| `useCache(bool)` | Toggle warm query cache | -| `addAffectedTable()` | Track virtual tables used in a query | -| `attachLock()` | Acquire recursive lock for table attachment | - -### `SQLiteDBManager` -Singleton managing SQLite resource lifecycle. Provides connection pooling β€” returns the primary DB when available, otherwise a transient instance. +### Classes -| Method | Description | +| Class | Purpose | |---|---| -| `get()` | Return a connection (primary or transient) | -| `getUnique()` | Always return a fresh transient connection | -| `resetPrimary()` | Close and reinitialize the primary DB | -| `isDisabled(name)` | Check if a table is disabled via flags | - -### `QueryPlanner` -Lightweight planner using SQLite `EXPLAIN` output to infer column types and identify scanned tables. - -### `queryInternal` -Execute a raw SQL query against a given `SQLiteDBInstanceRef`. +| `SQLiteDBInstance` | RAII wrapper around `sqlite3*`; manages primary vs. transient connections, cache state, and affected virtual tables | +| `SQLiteDBManager` | Singleton resource manager; sole access point for SQLite connections with contention-aware pooling | +| `QueryPlanner` | Lightweight query planner using SQLite `EXPLAIN` to infer column types and table scan order | ## Usage Example ```cpp -// Get a DB connection and run a query +// Obtain a managed database connection SQLiteDBInstanceRef db = SQLiteDBManager::get(); + +// Execute a query using the internal helper QueryDataTyped results; +Status s = queryInternal("SELECT pid, name FROM processes", results, db); +if (!s.ok()) { + LOG(ERROR) << "Query failed: " << s.getMessage(); +} -Status status = queryInternal( - "SELECT pid, name FROM processes WHERE uid = 0", - results, - db -); +// Use the query planner to infer column types +TableColumns columns = { /* ... */ }; +QueryPlanner planner("SELECT pid, name FROM processes", db); +planner.applyTypes(columns); -if (status.ok()) { - for (const auto& row : results) { - // process each row - } +// Check if a table is disabled via --disable_tables flag +if (SQLiteDBManager::isDisabled("processes")) { + // skip } +``` -// Use a unique transient connection (e.g., in tests) -SQLiteDBInstanceRef testDb = SQLiteDBManager::getUnique(); - -// Infer column types via query planner -QueryPlanner planner("SELECT * FROM users", db); -TableColumns columns; -planner.applyTypes(columns); -``` \ No newline at end of file +> **Security Note:** `SQLITE_ATTACH` is intentionally excluded from `kAllowedSQLiteActionCodes` to prevent writing arbitrary files via attached databases. \ No newline at end of file diff --git a/osquery/sql/.virtual_table.md b/osquery/sql/.virtual_table.md index 49405ccb9cc..dd09756240b 100644 --- a/osquery/sql/.virtual_table.md +++ b/osquery/sql/.virtual_table.md @@ -1,44 +1,42 @@ -Defines the SQLite virtual table interface for osquery, providing the core structures and functions needed to expose osquery table plugins as SQLite virtual tables. +Defines the SQLite virtual table interface for osquery, providing the core structures and functions that bridge osquery table plugins with SQLite's virtual table module system. ## Key Components ### Structures -| Name | Description | -|------|-------------| -| `BaseCursor` | Tracks SQLite cursor state per query β€” holds rows, position, generator, and current result | -| `VirtualTable` | Wraps a SQLite `sqlite3_vtab` with osquery metadata: table content and the active DB instance | +- **`BaseCursor`** β€” Represents a SQLite virtual table cursor; tracks row position, buffered `TableRows`, an optional coroutine-based `RowGenerator`, and the current result holder +- **`VirtualTable`** β€” Wraps the SQLite `sqlite3_vtab` struct; holds a `VirtualTableContent` metadata reference and a pointer to the thread-local `SQLiteDBInstance` ### Globals -- **`kAttachMutex`** β€” `RecursiveMutex` guarding concurrent table attach operations, since SQLite attach is not thread-safe +- **`kAttachMutex`** β€” `RecursiveMutex` protecting non-thread-safe table attach operations, particularly relevant when the extensions API accesses SQLite concurrently ### Functions | Function | Description | -|----------|-------------| -| `attachTableInternal()` | Attaches a named table plugin to a SQLite in-memory DB instance | -| `detachTableInternal()` | Drops a previously attached virtual table | -| `attachFunctionInternal()` | Registers a custom scalar function with SQLite | -| `attachVirtualTables()` | Bulk-attaches all registered table plugins to a DB instance | -| `registerForeignTables()` | Registers cross-platform schema-only table plugins from the generated amalgamation (excluded in extension builds) | +|---|---| +| `attachTableInternal` | Attaches a named table plugin to a SQLite in-memory DB instance | +| `detachTableInternal` | Drops a previously attached virtual table | +| `attachFunctionInternal` | Registers a custom scalar function with SQLite | +| `attachVirtualTables` | Bulk-attaches all registered table plugins to a DB instance | +| `registerForeignTables` | *(Core builds only)* Registers cross-platform schema plugins from the generated amalgamation | ## Usage Example -```c -// Attach all tables to a new SQLite instance -SQLiteDBInstanceRef db = SQLiteDBManager::get(); -attachVirtualTables(db); +```cpp +#include + +// Attach all registered table plugins to a new DB instance +auto instance = SQLiteDBManager::getUnique(); +osquery::attachVirtualTables(instance); // Attach a single table by name -Status s = attachTableInternal("processes", db, /*is_extension=*/false); -if (!s.ok()) { - LOG(ERROR) << "Failed to attach: " << s.getMessage(); +auto status = osquery::attachTableInternal("processes", instance, false); +if (!status.ok()) { + LOG(ERROR) << "Failed to attach table: " << status.getMessage(); } -// Detach when done -detachTableInternal("processes", db); -``` - -> **Note:** All attach operations should be performed while holding `kAttachMutex` to prevent race conditions in concurrent query environments. `BaseCursor` and `VirtualTable` are non-copyable and are only used internally by SQLite virtual table module callbacks. \ No newline at end of file +// Detach when no longer needed +osquery::detachTableInternal("processes", instance); +``` \ No newline at end of file diff --git a/osquery/sql/tests/.sql_test_utils.md b/osquery/sql/tests/.sql_test_utils.md index 7aea9eda6d7..0e318097453 100644 --- a/osquery/sql/tests/.sql_test_utils.md +++ b/osquery/sql/tests/.sql_test_utils.md @@ -1,38 +1,42 @@ -Utility header providing test fixtures and serialization helpers for osquery SQL query testing, including pre-built query results, diff results, and query log item data in both structured and JSON formats. +Utility header providing test fixtures and serialization helpers for osquery SQL unit tests, including pre-built query datasets, diff results, and log item pairs for round-trip serialization validation. ## Key Components | Symbol | Type | Description | -|---|---|---| -| `kTestQuery` | `const std::string` | Standard test query: `SELECT * FROM test_table` | -| `getTestDBResultStream()` | Function | Returns query/result pairs representing incremental mutations on the test database | -| `getTestDBExpectedResults()` | Function | Returns the baseline expected results for `kTestQuery` | -| `getSerializedRowColumnNames()` | Function | Returns test column names, optionally unordered and with duplicates | -| `getSerializedRow()` | Function | Returns a `(JSON, RowTyped)` pair for serialization round-trip testing | -| `getSerializedQueryData()` | Function | Returns a `(JSON, QueryDataTyped)` pair for standard query data | -| `getSerializedQueryDataWithColumnOrder()` | Function | Returns query data with repeated/non-alphabetical columns | +|--------|------|-------------| +| `kTestQuery` | `const std::string` | Standard test query string: `SELECT * FROM test_table` | +| `getTestDBResultStream()` | Function | Returns query/result pairs representing incremental mutations on the test dataset | +| `getTestDBExpectedResults()` | Function | Returns expected `QueryDataTyped` results for `kTestQuery` against the initial test DB | +| `getSerializedRowColumnNames()` | Function | Returns column name list; optionally unordered with a repeated column | +| `getSerializedRow()` | Function | Returns a `(JSON, RowTyped)` pair for round-trip serialization testing | +| `getSerializedQueryData()` | Function | Returns a `(JSON, QueryDataTyped)` pair for standard serialization tests | +| `getSerializedQueryDataWithColumnOrder()` | Function | Same as above but with repeated/non-alphabetical columns | | `getSerializedQueryDataJSON()` | Function | Returns a `(string, QueryDataTyped)` pair in raw JSON string form | -| `getSerializedDiffResults()` | Function | Returns a `(JSON, DiffResults)` pair for diff result testing | -| `getSerializedQueryLogItem()` | Function | Returns a `(JSON, QueryLogItem)` pair for log item testing | +| `getSerializedDiffResults()` | Function | Returns a `(JSON, DiffResults)` pair for diff serialization tests | +| `getSerializedDiffResultsJSON()` | Function | Returns a `(string, DiffResults)` pair in raw JSON string form | +| `getSerializedQueryLogItem()` | Function | Returns a `(JSON, QueryLogItem)` pair for log item serialization tests | +| `getSerializedQueryLogItemJSON()` | Function | Returns a `(string, QueryLogItem)` pair in raw JSON string form | ## Usage Example ```cpp #include "sql_test_utils.h" -// Verify expected query results match baseline -QueryDataTyped expected = osquery::getTestDBExpectedResults(); - -// Round-trip serialization test for a single row -auto [json, row] = osquery::getSerializedRow(); -// Serialize row β†’ compare to json, deserialize json β†’ compare to row - -// Test diff result serialization -auto [diffJson, diffResult] = osquery::getSerializedDiffResults(); - -// Iterate over incremental DB mutation stream -for (auto& [query, result] : osquery::getTestDBResultStream()) { - // Apply query, assert result matches expected +TEST(SQLTest, TestRoundTripSerialization) { + // Verify row serializes and deserializes correctly + auto [json, row] = osquery::getSerializedRow(); + // json.first serializes to json.second and vice versa + + // Validate expected query results against test DB + auto expected = osquery::getTestDBExpectedResults(); + auto results = runQuery(osquery::kTestQuery); + EXPECT_EQ(results, expected); + + // Iterate mutation stream for incremental diff testing + for (auto& [query, data] : osquery::getTestDBResultStream()) { + auto actual = runQuery(query); + EXPECT_EQ(actual, data); + } } ``` \ No newline at end of file diff --git a/osquery/system/network/.hostname.md b/osquery/system/network/.hostname.md index 37e6826819d..d75a48fce72 100644 --- a/osquery/system/network/.hostname.md +++ b/osquery/system/network/.hostname.md @@ -1,35 +1,40 @@ -Defines the `HostIdentity` structure used to represent a host machine's identity within the osquery framework, capturing its fully qualified domain name and UUID. +Defines the `HostIdentity` struct for representing a host's network identity within the osquery framework, using the Schemer serialization interface. ## Key Components ### `HostIdentity` (class) -A final class encapsulating host identification data within the `osquery` namespace. +A final class encapsulating host identification data with two public fields: -| Member | Type | Description | -|--------|------|-------------| -| `fqdn` | `std::string` | Fully qualified domain name of the host | +| Field | Type | Description | +|-------|------|-------------| +| `fqdn` | `std::string` | Fully Qualified Domain Name of the host | | `uuid` | `std::string` | Unique identifier of the host | -### `localhost()` (static method) +### `HostIdentity::localhost()` (static method) Factory method that constructs and returns a `HostIdentity` instance populated with the local machine's identity information. -### `discloseSchema()` (static template method) -Integrates with the `schemer` serialization framework to expose the `fqdn` and `uuid` fields for schema-driven serialization/deserialization (e.g., JSON, config persistence). +### `HostIdentity::discloseSchema()` (static template method) +Schemer-compatible schema disclosure method that registers `fqdn` and `uuid` fields for serialization/deserialization via the osquery Schemer archive interface. ## Usage Example -```cpp -#include -#include "hostname.h" +```c +#include -// Retrieve identity of the local host -osquery::HostIdentity host = osquery::HostIdentity::localhost(); +// Retrieve the local host's identity +osquery::HostIdentity identity = osquery::HostIdentity::localhost(); -// Access identity fields -std::string fqdn = host.fqdn; // e.g. "my-machine.example.com" -std::string uuid = host.uuid; // e.g. "550e8400-e29b-41d4-a716-446655440000" +// Access host fields +std::string hostname = identity.fqdn; // e.g. "myhost.example.com" +std::string uid = identity.uuid; // e.g. "550e8400-e29b-41d4-a716-446655440000" -// Schema disclosure is handled automatically by the schemer framework -// when serializing HostIdentity instances -``` \ No newline at end of file +// Schemer serialization (used internally by osquery) +MyArchive archive; +osquery::HostIdentity::discloseSchema(archive, identity); +``` + +## Notes +- The class is marked `final` β€” it is not intended to be subclassed. +- The default constructor is explicitly defined as `default`. +- Schema disclosure enables both serialization and deserialization through osquery's Schemer framework, making `HostIdentity` suitable for use in distributed query results and configuration payloads. \ No newline at end of file diff --git a/osquery/system/usersgroups/windows/.groups_service.md b/osquery/system/usersgroups/windows/.groups_service.md index b38d06ecef0..c92c3060575 100644 --- a/osquery/system/usersgroups/windows/.groups_service.md +++ b/osquery/system/usersgroups/windows/.groups_service.md @@ -1,42 +1,36 @@ -Background service that enumerates and caches local Windows groups by extending the `InternalRunnable` dispatcher interface. +Declares the `GroupsService` class, a background service that enumerates and caches local Windows groups for use in osquery's user/group table queries. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `GroupsService` | Class | Background runnable service that populates and maintains the `GroupsCache` for local Windows groups | -| `GroupsService()` | Constructor | Accepts a `std::promise` (signals cache readiness) and a shared pointer to the `GroupsCache` instance | -| `start()` | Protected method | Entry point called by the dispatcher when the service thread begins execution | -| `processLocalGroups()` | Private method | Iterates over local groups and invokes the provided callback to insert each `Group` into the cache | -| `UpdateGroupFunc` | Type alias | Defines the signature `void(Group)` used for the per-group update callback | +### `GroupsService` (class) +Extends `InternalRunnable` to run as a background dispatcher thread. Responsible for populating a shared `GroupsCache` with local group data. + +| Member | Description | +|--------|-------------| +| `GroupsService(promise, cache)` | Constructor accepting a fulfillment promise and shared cache reference | +| `start()` *(protected)* | Entry point called by the dispatcher to begin group enumeration | +| `processLocalGroups(func)` *(private)* | Iterates local groups, invoking `update_group_func` for each discovered `Group` | +| `groups_cache_promise_` | Signaled once the initial cache population is complete | +| `groups_cache_` | Shared pointer to the `GroupsCache` populated by this service | ## Usage Example ```cpp #include -// Create the shared cache and a promise to signal completion +// Create the shared cache and a promise to signal readiness auto groups_cache = std::make_shared(); -std::promise cache_ready_promise; - -// Retrieve the future before moving the promise -auto cache_ready = cache_ready_promise.get_future(); +std::promise ready_promise; +auto ready_future = ready_promise.get_future(); // Instantiate and dispatch the service -auto groups_svc = std::make_shared( - std::move(cache_ready_promise), groups_cache); - -Dispatcher::addService(groups_svc); +auto service = std::make_shared( + std::move(ready_promise), groups_cache); -// Block until the initial cache population is complete -cache_ready.wait(); - -// GroupsCache is now populated and safe to query +// Wait for initial population to complete before querying +ready_future.wait(); +// groups_cache is now populated with local group data ``` -## Notes - -- Windows-only β€” lives under the `windows/` users/groups subsystem path -- The `std::promise` pattern ensures consumers can block until the first full enumeration completes before querying the cache -- Follows the osquery `InternalRunnable` pattern; do not call `start()` directly β€” use `Dispatcher::addService()` \ No newline at end of file +> **Note:** This header is Windows-specific. It is part of the `osquery` namespace and depends on the dispatcher infrastructure (`InternalRunnable`) and the shared `UsersGroupsCache` subsystem. \ No newline at end of file diff --git a/osquery/system/usersgroups/windows/.users_groups_cache.md b/osquery/system/usersgroups/windows/.users_groups_cache.md index 429efb17b40..eae6fd62a54 100644 --- a/osquery/system/usersgroups/windows/.users_groups_cache.md +++ b/osquery/system/usersgroups/windows/.users_groups_cache.md @@ -1,63 +1,50 @@ -Thread-safe Windows user and group caching layer for osquery, providing indexed lookup of system accounts with generation-based cache invalidation. +Thread-safe cache structures for Windows users and groups within the osquery framework, providing generation-based cache invalidation and multi-index lookup for user/group data. ## Key Components ### Structs +- **`User`** β€” Holds user attributes: `uid`, `gid`, `sid`, `username`, `description`, `type`, `directory`, and a `generation` counter for cache lifecycle tracking +- **`Group`** β€” Holds group attributes: `gid`, `sid`, `groupname`, `comment`, and a `generation` counter -| Struct | Fields | Description | -|--------|--------|-------------| -| `User` | `uid`, `gid`, `sid`, `username`, `description`, `type`, `directory`, `generation` | Represents a Windows user account | -| `Group` | `gid`, `sid`, `groupname`, `comment`, `generation` | Represents a Windows group | - -Both structs implement `operator==` for value comparison (excluding `generation`). - -### Cache Index Types - -```c -using UidCacheIndex = std::unordered_multimap; -using GidCacheIndex = std::unordered_multimap; -using SidCacheIndex = std::unordered_map; -using GroupnameCacheIndex = std::unordered_map; -``` - -Indexes map lookup keys to positions in the backing `std::vector`. +### Type Aliases +- `UidCacheIndex` / `GidCacheIndex` β€” `unordered_multimap` for numeric ID lookups +- `SidCacheIndex` β€” `unordered_map` for SID string lookups +- `GroupnameCacheIndex` β€” `unordered_map` for group name lookups ### `UsersCache` - | Method | Description | -|--------|-------------| +|---|---| | `initializeCache(users)` | Bulk-loads initial user set | -| `updateUser(user)` | Upserts by SID, bumps generation | -| `increaseGeneration()` | Advances cache generation counter | -| `cleanupExpiredUsers()` | Evicts users below current generation | +| `updateUser(user)` | Inserts or updates by SID; bumps generation | +| `increaseGeneration()` | Advances the global generation counter | +| `cleanupExpiredUsers()` | Evicts entries below current generation | | `getUsersByUid(uid)` | Returns all users matching a UID | | `getUserBySid(sid)` | Returns a single user by SID | | `getAllUsers()` | Returns full user list | -| `clear()` | Resets cache state (testing) | ### `GroupsCache` - -Mirrors `UsersCache` with additional `getGroupByName(name)` lookup. Internally maintains three indexes: GID, SID, and group name. +Mirrors `UsersCache` with additional `getGroupByName(name)` lookup. ## Usage Example -```cpp -osquery::UsersCache cache; - -// Bulk initialize -cache.initializeCache(fetchAllSystemUsers()); +```c +// Initialize and populate +osquery::UsersCache users_cache; +users_cache.initializeCache(fetchAllUsersFromSystem()); // Incremental update cycle for (auto& user : getUpdatedUsers()) { - cache.updateUser(user); + users_cache.updateUser(user); } -cache.increaseGeneration(); -cache.cleanupExpiredUsers(); // evicts stale entries +users_cache.increaseGeneration(); +users_cache.cleanupExpiredUsers(); // removes stale entries -// Lookups -auto user = cache.getUserBySid("S-1-5-21-..."); -auto users = cache.getUsersByUid(1001); +// Lookup +auto user = users_cache.getUserBySid("S-1-5-21-..."); +if (user.has_value()) { + std::cout << user->username; +} ``` -> **Generation mechanism:** `updateUser` / `updateGroup` stamps entries with the next generation. After a full refresh cycle, call `increaseGeneration()` then `cleanup*()` to evict any accounts removed from the system since the last scan. \ No newline at end of file +> **Note:** Both caches are mutex-protected (`cache_mutex_`) and safe for concurrent reads and writes across threads. \ No newline at end of file diff --git a/osquery/system/usersgroups/windows/.users_service.md b/osquery/system/usersgroups/windows/.users_service.md index 395cbaeb1d1..dd258dc381d 100644 --- a/osquery/system/usersgroups/windows/.users_service.md +++ b/osquery/system/usersgroups/windows/.users_service.md @@ -1,36 +1,53 @@ -Declares the `UsersService` class, a background service that populates and maintains a Windows user cache by enumerating local accounts and roaming profiles. +Background service that populates and maintains a Windows user cache by enumerating local accounts and roaming profiles. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `UsersService` | Class | Background `InternalRunnable` that builds the `UsersCache` on Windows | -| `UsersService()` | Constructor | Accepts a `std::promise` (signals cache readiness) and a shared `UsersCache` pointer | -| `start()` | Protected method | Entry point called by the dispatcher to begin user enumeration | -| `processLocalAccounts()` | Private method | Enumerates local Windows accounts and invokes a callback per user; tracks processed SIDs | -| `processRoamingProfiles()` | Private method | Enumerates roaming profile accounts not already covered by local account processing | -| `UpdateUserFunc` | Type alias | `void(User user)` β€” callback signature used to update the cache | +### `UsersService` (class) +Extends `InternalRunnable` to run as a background dispatcher thread. Responsible for discovering Windows users and populating a shared `UsersCache` instance. + +**Constructor** +```c +UsersService(std::promise users_cache_promise, + std::shared_ptr users_cache); +``` +Accepts a promise (signaled when the initial cache population completes) and a shared pointer to the cache being populated. + +**Protected Methods** + +| Method | Description | +|--------|-------------| +| `start()` | Entry point called by the dispatcher to begin cache population | + +**Private Methods** + +| Method | Description | +|--------|-------------| +| `processLocalAccounts()` | Enumerates local Windows accounts; tracks processed SIDs to avoid duplicates | +| `processRoamingProfiles()` | Enumerates roaming profile accounts not already covered by local account processing | + +Both methods accept a callback (`UpdateUserFunc`) to decouple discovery from cache-write logic. ## Usage Example ```cpp #include -#include -// Create the shared cache and a promise to signal when it's ready -auto users_cache = std::make_shared(); +// Create shared cache and a promise to await initial population +auto users_cache = std::make_shared(); std::promise cache_ready_promise; -auto cache_ready_future = cache_ready_promise.get_future(); +auto cache_ready = cache_ready_promise.get_future(); -// Instantiate and dispatch the service -auto service = std::make_shared( - std::move(cache_ready_promise), users_cache); +// Dispatch the service as a background thread +Dispatcher::addService( + std::make_shared( + std::move(cache_ready_promise), + users_cache)); -osquery::Dispatcher::addService(service); +// Block until the initial user enumeration is complete +cache_ready.wait(); -// Block until the initial cache population is complete -cache_ready_future.wait(); +// Cache is now ready for queries ``` -> **Note:** This header is Windows-specific. The `std::promise` pattern ensures that consumers can synchronize against the initial cache population before querying user data. \ No newline at end of file +> **Note:** This header is Windows-specific and depends on the `UsersCache` type defined in `users_groups_cache.h`. The `std::promise` pattern ensures consumers can synchronize against the first full cache load before issuing queries. \ No newline at end of file diff --git a/osquery/tables/applications/.jetbrains_plugins.md b/osquery/tables/applications/.jetbrains_plugins.md index 0b2d0d00fb7..e0e1cb84037 100644 --- a/osquery/tables/applications/.jetbrains_plugins.md +++ b/osquery/tables/applications/.jetbrains_plugins.md @@ -1,35 +1,37 @@ -Defines constants and utilities for locating and identifying JetBrains IDE plugin directories across Windows, macOS, and Linux within the osquery tables framework. +Defines constants and utilities for locating and identifying JetBrains IDE plugin directories across Windows, macOS, and Linux platforms within the osquery tables namespace. ## Key Components -- **`JetBrainsProductType`** β€” Enum class enumerating all supported JetBrains IDEs: CLion, DataGrip, GoLand, IntelliJ IDEA (Ultimate + Community), PhpStorm, PyCharm (Ultimate + Community), ReSharper, Rider, RubyMine, RustRover, and WebStorm. -- **`ProductPathMap`** β€” Type alias for `std::vector>` mapping product types to their relative plugin directory paths. -- **`kWindowsPathList`** β€” Platform path map for Windows (`AppData\Roaming\JetBrains\%\plugins`). -- **`kMacOsPathList`** β€” Platform path map for macOS (`Library/Application Support/JetBrains/%/plugins`). -- **`kLinuxPathList`** β€” Platform path map for Linux (`.local/share/JetBrains/%`). Note: Linux paths omit the trailing `/plugins` segment. -- **`kProductTypeToString`** β€” Lookup map converting `JetBrainsProductType` enum values to lowercase string identifiers (e.g., `"intellij_idea"`). +- **`JetBrainsProductType`** β€” Enum class enumerating 13 supported JetBrains IDEs: CLion, DataGrip, GoLand, IntelliJIdea, IntelliJIdeaCommunityEdition, PhpStorm, PyCharm, PyCharmCommunityEdition, ReSharper, Rider, RubyMine, RustRover, WebStorm. + +- **`ProductPathMap`** β€” Type alias for `std::vector>` mapping product types to relative plugin path patterns. The `%` wildcard in paths matches versioned directory suffixes (e.g., `CLion2024.1`). + +- **`kWindowsPathList`** β€” Path map targeting `AppData\Roaming\JetBrains\%\plugins`. + +- **`kMacOsPathList`** β€” Path map targeting `Library/Application Support/JetBrains/%/plugins`. + +- **`kLinuxPathList`** β€” Path map targeting `.local/share/JetBrains/%`. + +- **`kProductTypeToString`** β€” Maps `JetBrainsProductType` enum values to lowercase string identifiers (e.g., `intellij_idea`, `pycharm_community_edition`) used in osquery table output. + - **`getProductName()`** β€” Returns the string name for a given `JetBrainsProductType`. -- **`putMoreLikelyPluginJarsFirst()`** β€” Reorders a list of files in a plugin's `lib` directory, prioritizing the most likely primary JAR. -- **`fileNameIsLikeVersionedLibraryName()`** β€” Predicate that checks whether a filename matches the pattern of a versioned library JAR. + +- **`putMoreLikelyPluginJarsFirst()`** β€” Reorders a list of JAR files so the most likely plugin JARs appear first. + +- **`fileNameIsLikeVersionedLibraryName()`** β€” Checks whether a filename matches the pattern of a versioned library name. ## Usage Example ```c -// Iterate all known Windows JetBrains plugin paths -for (const auto& [product, path] : osquery::tables::kWindowsPathList) { +// Iterate plugin paths on macOS to discover installed plugins +for (const auto& [product, path] : osquery::tables::kMacOsPathList) { std::string name = osquery::tables::getProductName(product); - // e.g., name = "clion", path = "AppData\\Roaming\\JetBrains\\CLion%\\plugins" - std::cout << name << " -> " << path << "\n"; + // Expand % wildcard and enumerate plugins under resolved path + std::cout << name << " -> " << path << std::endl; } -// Sort plugin JARs for a plugin directory -std::vector jars = {"plugin-1.0.jar", "plugin.jar", "util.jar"}; -osquery::tables::putMoreLikelyPluginJarsFirst("my-plugin", jars); - -// Check if a filename looks like a versioned library -bool versioned = osquery::tables::fileNameIsLikeVersionedLibraryName("log4j-2.17.jar"); -// versioned == true -``` - -> **Note:** The `%` wildcard in path strings represents a version suffix glob, allowing matching across multiple installed IDE versions. \ No newline at end of file +// Reorder discovered JARs so primary plugin JARs come first +std::vector jars = getFilesInLibDir(pluginDir); +osquery::tables::putMoreLikelyPluginJarsFirst(pluginDir, jars); +``` \ No newline at end of file diff --git a/osquery/tables/applications/chrome/.utils.md b/osquery/tables/applications/chrome/.utils.md index 6bfd0fe53d0..376c71f9377 100644 --- a/osquery/tables/applications/chrome/.utils.md +++ b/osquery/tables/applications/chrome/.utils.md @@ -1,54 +1,51 @@ -Utility header for Chrome-based browser extension and profile introspection within the osquery tables subsystem. Defines data structures and helper functions for enumerating browser profiles, parsing extension manifests, and extracting metadata across multiple Chromium-derived browsers. +Utility header for osquery's Chrome browser extension and profile inspection tables, providing data structures and helper functions to enumerate and parse Chrome-based browser profiles and extensions. ## Key Components -### Enums -- **`ChromeBrowserType`** β€” Identifies supported Chromium-based browsers (Chrome, Brave, Edge, Opera, Vivaldi, Arc, etc.) -- **`ExtensionKeyError`** β€” Error codes for extension identifier computation failures +### Enumerations +- **`ChromeBrowserType`** β€” Identifies supported Chromium-based browsers: Chrome, Brave, Edge, Opera, Vivaldi, Arc, Yandex, and others -### Structs -- **`ChromeProfileSnapshot`** β€” Raw file contents captured from a browser profile directory, including `Preferences`, `Secure Preferences`, and extension manifests -- **`ChromeProfile`** β€” Parsed, structured representation of a profile with its associated `Extension` list and metadata -- **`ContentScriptsEntry`** β€” A `script`/`match` pair parsed from a manifest's `content_scripts` array +### Data Structures +- **`ChromeProfileSnapshot`** β€” Raw file snapshot of a Chrome profile, holding `Preferences`, `Secure Preferences`, and maps of referenced/unreferenced extensions +- **`ChromeProfile`** β€” Parsed profile representation with a name, UID, browser type, and a list of fully resolved `Extension` objects +- **`ChromeProfile::Extension`** β€” Represents a single extension with manifest JSON, SHA256 hash, computed ID, content script matches, and profile settings ### Key Functions | Function | Purpose | |---|---| | `getChromeProfiles()` | Entry point β€” returns all profiles for all supported browsers | -| `getChromeProfilesFromSnapshotList()` | Converts raw snapshots into structured `ChromeProfile` objects | -| `getExtensionFromSnapshot()` | Parses a single extension snapshot into a `ChromeProfile::Extension` | -| `computeExtensionIdentifier()` | Derives extension ID from the manifest `key` property | +| `getChromeProfilesFromSnapshotList()` | Converts raw snapshots into parsed `ChromeProfile` objects | +| `getExtensionFromSnapshot()` | Parses a raw extension snapshot into a structured `Extension` | +| `computeExtensionIdentifier()` | Derives the extension ID from the manifest `key` property | | `webkitTimeToUnixTimestamp()` | Converts WebKit epoch timestamps to Unix format | -| `getStringLocalization()` | Resolves `__MSG_*` style localized strings from a parsed locale tree | -| `getExtensionProperty()` | Safe accessor for extension manifest properties with optional defaults | +| `getStringLocalization()` | Resolves `__MSG_name__` style localized strings | +| `getExtensionProperty()` | Safe accessor for extension manifest properties | ## Usage Example ```cpp #include "utils.h" -// Retrieve all Chrome-based browser profiles for a query -osquery::QueryContext ctx; -auto profiles = osquery::tables::getChromeProfiles(ctx); +using namespace osquery::tables; + +// Enumerate all Chrome-based browser profiles +QueryContext ctx; +ChromeProfileList profiles = getChromeProfiles(ctx); for (const auto& profile : profiles) { - // Print browser name and profile path - std::cout << getChromeBrowserName(profile.type) - << " - " << profile.path << "\n"; - - for (const auto& ext : profile.extension_list) { - // Resolve computed extension identifier - auto id_result = osquery::tables::computeExtensionIdentifier(ext); - if (id_result) { - std::cout << " Extension ID: " << id_result.get() << "\n"; - } + std::cout << getChromeBrowserName(profile.type) + << " | " << profile.name << "\n"; - // Access a manifest property with a fallback default - std::string version = osquery::tables::getExtensionProperty( - ext, "version", /*optional=*/true, "unknown"); - std::cout << " Version: " << version << "\n"; - } + for (const auto& ext : profile.extension_list) { + auto id = computeExtensionIdentifier(ext); + if (id) { + std::cout << " Extension: " << id.get() << "\n"; + } + + std::string version = getExtensionProperty(ext, "version", false); + std::cout << " Version: " << version << "\n"; + } } ``` \ No newline at end of file diff --git a/osquery/tables/applications/posix/.prometheus_metrics.md b/osquery/tables/applications/posix/.prometheus_metrics.md index d7ea278e74e..b5f80844a61 100644 --- a/osquery/tables/applications/posix/.prometheus_metrics.md +++ b/osquery/tables/applications/posix/.prometheus_metrics.md @@ -1,25 +1,17 @@ -Declares the data structures and core functions for the `prometheus_metrics` osquery table, which scrapes and parses Prometheus-format metrics from HTTP endpoints. +Defines types and declares functions for scraping and parsing Prometheus metrics endpoints within the osquery tables framework. ## Key Components -### Constants +**Constants** +- `kColTargetName`, `kColMetric`, `kColValue`, `kColTimeStamp` β€” Column name strings used when building `QueryData` rows (`target_name`, `metric_name`, `metric_value`, `timestamp_ms`) -| Constant | Value | Description | -|---|---|---| -| `kColTargetName` | `"target_name"` | Column name for the scraped target URL | -| `kColMetric` | `"metric_name"` | Column name for the metric identifier | -| `kColValue` | `"metric_value"` | Column name for the metric value | -| `kColTimeStamp` | `"timestamp_ms"` | Column name for the scrape timestamp | +**Struct** +- `PrometheusResponseData` β€” Holds a scraped target's raw response body (`content`) and the millisecond-precision timestamp of when the scrape occurred (`timestampMS`) -### Struct - -**`PrometheusResponseData`** β€” Holds the raw HTTP response body (`content`) and the millisecond-precision timestamp (`timestampMS`) recorded at scrape time. - -### Functions - -- **`scrapeTargets(scrapeResults, timeoutS)`** β€” Performs HTTP GET requests against all target URLs provided as keys in `scrapeResults`, writing the response body and timestamp into each corresponding `PrometheusResponseData` value. Default timeout is `1` second. -- **`parseScrapeResults(scrapeResults, rows)`** β€” Converts the raw Prometheus text-format payloads from `scrapeResults` into osquery `QueryData` rows, mapping each metric to the four defined columns. +**Functions** +- `scrapeTargets()` β€” Performs HTTP scrapes against one or more Prometheus targets; populates a `PrometheusResponseData` entry per URL; accepts an optional timeout in seconds (default: `1`) +- `parseScrapeResults()` β€” Converts the raw scrape map into osquery `QueryData` rows using the defined column name constants ## Usage Example @@ -28,18 +20,23 @@ Declares the data structures and core functions for the `prometheus_metrics` osq using namespace osquery::tables; -// Define targets to scrape +// 1. Define targets to scrape std::map scrapeResults = { {"http://localhost:9090/metrics", {}}, {"http://localhost:9100/metrics", {}}, }; -// Scrape all targets with a 2-second timeout +// 2. Scrape all targets (2-second timeout) scrapeTargets(scrapeResults, 2); -// Parse results into osquery rows +// 3. Parse raw payloads into QueryData rows QueryData rows; parseScrapeResults(scrapeResults, rows); -// rows now contains columns: target_name, metric_name, metric_value, timestamp_ms +// 4. Each row contains: target_name, metric_name, metric_value, timestamp_ms +for (const auto& row : rows) { + std::cout << row.at(kColTargetName) << " " + << row.at(kColMetric) << " = " + << row.at(kColValue) << "\n"; +} ``` \ No newline at end of file diff --git a/osquery/tables/events/.event_utils.md b/osquery/tables/events/.event_utils.md index c35eed836e3..2f904d4573b 100644 --- a/osquery/tables/events/.event_utils.md +++ b/osquery/tables/events/.event_utils.md @@ -1,25 +1,27 @@ -Utility header providing shared helpers for cross-platform file event handling in osquery. +Utility header providing shared helpers for cross-platform file event handling in osquery's eventing system. ## Key Components -- **`kCommonFileColumns`** β€” Extern constant set of column names shared across all platform file event implementations. -- **`decorateFileEvent()`** β€” Decorator function that populates a `Row` with common file metadata and optional hash data from the `file` table. +- **`kCommonFileColumns`** β€” A constant set of column names shared across file event tables on all platforms, ensuring consistent schema decoration. +- **`decorateFileEvent(path, hash, r)`** β€” Populates a `Row` with common file metadata (path info, optional hash) sourced from the `file` table, abstracting platform-specific `file_events` implementations. ## Usage Example -```c +```cpp #include "event_utils.h" -// Populate a row with file event metadata -osquery::Row r; -osquery::decorateFileEvent("/etc/passwd", /* hash= */ true, r); +void handleFileEvent(const std::string& filePath, Row& eventRow) { + // Decorate the row with common file columns, including hash + decorateFileEvent(filePath, /* hash= */ true, eventRow); -// r now contains common file columns (e.g., size, inode, hash) + // eventRow now contains hashed file metadata + // and all columns defined in kCommonFileColumns +} ``` ## Notes -- Designed to unify file event columns across Linux (inotify), macOS (FSEvents), and Windows platform implementations. -- When `hash` is `true`, the target file is read and its hash is computed and stored in the row. -- Consumers should check `kCommonFileColumns` to determine which columns are guaranteed present after decoration. \ No newline at end of file +- `hash` parameter controls whether the file at `path` is read and checksummed; pass `false` for performance-sensitive paths where hashing is unnecessary. +- Intended to be called from each platform's `file_events` publisher (Linux inotify, macOS FSEvents, Windows USN journal) to normalize output rows before emission. +- Depends on `osquery/core/tables.h` for the `Row` type. \ No newline at end of file diff --git a/osquery/tables/events/linux/.apparmor_events.md b/osquery/tables/events/linux/.apparmor_events.md index e183097e9d6..3677f184dec 100644 --- a/osquery/tables/events/linux/.apparmor_events.md +++ b/osquery/tables/events/linux/.apparmor_events.md @@ -1,46 +1,45 @@ -Brief description of the file's purpose: Declares the `AppArmorEventSubscriber` class, which subscribes to Linux audit events related to AppArmor policy enforcement and processes them into queryable osquery rows. +Defines the `AppArmorEventSubscriber` class, an osquery event subscriber that listens for AppArmor-related Linux audit events and processes them into queryable row data. ## Key Components -### `AppArmorEventSubscriber` -A final class inheriting from `EventSubscriber` with the following members: +**`AppArmorEventSubscriber`** β€” Final class inheriting from `EventSubscriber` with the following members: -| Member | Type | Description | -|--------|------|-------------| -| `init()` | `Status` | Registers the audit event type subscription on startup | -| `callback()` | `Status` | Fires when a matching kernel audit event is received | -| `processEvents()` | `static Status` | Parses a batch of `AuditEvent` objects into `QueryData` rows | -| `getEventSet()` | `static const std::set&` | Returns the set of audit event type IDs this subscriber handles | +| Member | Description | +|--------|-------------| +| `init()` | Registers the audit event type subscription on startup | +| `callback()` | Fired when a matching kernel audit event is received | +| `processEvents()` | Static method that transforms a list of `AuditEvent` objects into `QueryData` rows | +| `getEventSet()` | Static method returning the set of audit event type IDs this subscriber handles | ## Usage Example ```c -// Typical osquery subscriber lifecycle (handled by the framework): +// Typical osquery subscriber lifecycle (managed by the framework) -// 1. Framework calls init() to register subscriptions AppArmorEventSubscriber subscriber; + +// Framework calls init() to register subscriptions subscriber.init(); -// 2. Kernel audit events fire callback() automatically -// subscriber.callback(ec, sc) is invoked by AuditEventPublisher +// When matching audit events arrive, the framework invokes: +subscriber.callback(event_context_ref, subscription_context_ref); -// 3. Manually process a batch of audit events (e.g., in tests) +// Standalone event processing (e.g., for testing) QueryData results; -std::vector events = getAuditEvents(); +std::vector events = { /* populated by publisher */ }; -auto status = AppArmorEventSubscriber::processEvents(results, events); -if (status.ok()) { - for (const auto& row : results) { - // Each row contains AppArmor enforcement data - } +Status s = AppArmorEventSubscriber::processEvents(results, events); +if (s.ok()) { + // results now contains AppArmor audit rows } -// 4. Inspect handled event type IDs -const auto& eventSet = AppArmorEventSubscriber::getEventSet(); -for (int eventType : eventSet) { - // e.g., AUDIT_APPARMOR_ALLOWED, AUDIT_APPARMOR_DENIED, etc. -} +// Inspect which audit event IDs are handled +const std::set& handled = AppArmorEventSubscriber::getEventSet(); ``` -> **Note:** This subscriber is Linux-only and depends on the kernel audit subsystem (`linux/audit.h`). It integrates with osquery's `AuditEventPublisher` to surface AppArmor allow/deny decisions as structured table data. \ No newline at end of file +## Notes + +- Declared inside the `osquery` namespace +- Depends on `AuditEventPublisher` from `osquery/events/linux/auditeventpublisher.h` +- `processEvents` and `getEventSet` are `noexcept` static utilities, safe to call without an instance \ No newline at end of file diff --git a/osquery/tables/events/linux/.bpf_process_events.md b/osquery/tables/events/linux/.bpf_process_events.md index 19e0a9c955f..ead73eaa748 100644 --- a/osquery/tables/events/linux/.bpf_process_events.md +++ b/osquery/tables/events/linux/.bpf_process_events.md @@ -1,45 +1,37 @@ -Header file defining the `BPFProcessEventSubscriber` class, which subscribes to BPF-based process events on Linux and converts them into osquery table rows. +Header defining the `BPFProcessEventSubscriber` class, which subscribes to BPF-based process events on Linux and transforms them into osquery table rows. ## Key Components -### `BPFProcessEventSubscriber` -An `EventSubscriber` specialized for `BPFEventPublisher` that handles process lifecycle events captured via eBPF. - -| Method | Description | -|---|---| -| `init()` | Initializes the subscriber and registers subscriptions | -| `eventCallback(...)` | Handles incoming BPF events from the publisher | -| `generateRow(...)` | Converts a single `ISystemStateTracker::Event` into a table `Row` | -| `generateRowList(...)` | Batch-converts an `EventList` into a `vector` | -| `generateCmdlineColumn(...)` | Formats argv into a space-separated command line string | -| `generateJsonCmdlineColumn(...)` | Formats argv into a JSON-encoded command line string | +- **`BPFProcessEventSubscriber`** β€” Final class extending `EventSubscriber`; handles lifecycle and event processing for BPF process events +- **`init()`** β€” Initializes the subscriber and registers it with the BPF event publisher +- **`eventCallback()`** β€” Invoked per event; receives event and subscription contexts for real-time processing +- **`generateRow()`** β€” Converts a single `ISystemStateTracker::Event` into an osquery `Row` +- **`generateRowList()`** β€” Batch-converts an `EventList` into a `std::vector` +- **`generateCmdlineColumn()`** β€” Serializes an `argv` vector into a plain space-separated command-line string +- **`generateJsonCmdlineColumn()`** β€” Serializes an `argv` vector into a JSON-formatted command-line string ## Usage Example ```c -// Typical osquery subscriber pattern - registered automatically via macro -// Implementation side (bpf_process_events.cpp): - -Status BPFProcessEventSubscriber::init() { - auto subscription = createSubscriptionContext(); - subscribe(&BPFProcessEventSubscriber::eventCallback, subscription); - return Status::success(); -} +// Subscriber is registered automatically via osquery's event framework. +// Static helpers can be used independently for row generation: -// Generating a row from a captured process event: -ISystemStateTracker::Event event = /* from BPF publisher */; +ISystemStateTracker::Event event = /* ... */; Row row; + if (BPFProcessEventSubscriber::generateRow(row, event)) { - // row is populated with pid, path, cmdline, etc. + // row now contains columns: pid, path, cmdline, etc. } -// Formatting command-line arguments: -std::vector argv = {"/usr/bin/ssh", "-l", "user", "host"}; -auto cmdline = BPFProcessEventSubscriber::generateCmdlineColumn(argv); -// β†’ "/usr/bin/ssh -l user host" -auto json_cmdline = BPFProcessEventSubscriber::generateJsonCmdlineColumn(argv); -// β†’ '["/usr/bin/ssh","-l","user","host"]' +// Build command-line columns from argv +std::vector argv = {"/usr/bin/curl", "-s", "https://example.com"}; + +std::string cmdline = BPFProcessEventSubscriber::generateCmdlineColumn(argv); +// β†’ "/usr/bin/curl -s https://example.com" + +std::string jsonCmdline = BPFProcessEventSubscriber::generateJsonCmdlineColumn(argv); +// β†’ ["/usr/bin/curl", "-s", "https://example.com"] ``` -> **Note:** This subscriber is Linux-only and requires eBPF support in the kernel. It feeds the `bpf_process_events` virtual table in osquery. \ No newline at end of file +> **Platform note:** This subscriber is Linux-only and requires BPF support in the kernel. It feeds the `bpf_process_events` virtual table in osquery. \ No newline at end of file diff --git a/osquery/tables/events/linux/.bpf_socket_events.md b/osquery/tables/events/linux/.bpf_socket_events.md index 499edd568be..aff6f51512a 100644 --- a/osquery/tables/events/linux/.bpf_socket_events.md +++ b/osquery/tables/events/linux/.bpf_socket_events.md @@ -1,37 +1,39 @@ -BPF-based socket event subscriber that listens for socket-related system events via the eBPF event publisher and converts them into osquery table rows. +Header file defining the BPF-based socket event subscriber for osquery's Linux event monitoring system. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `BPFSocketEventSubscriber` | Class | Final subscriber class inheriting from `EventSubscriber` | -| `init()` | Method | Initializes the subscriber and registers subscriptions | -| `eventCallback()` | Method | Handles incoming BPF socket events from the publisher | -| `generateRow()` | Static Method | Converts a single `ISystemStateTracker::Event` into an osquery `Row` | -| `generateRowList()` | Static Method | Batch-converts an `EventList` into a `std::vector` | +### `BPFSocketEventSubscriber` +A final class inheriting from `EventSubscriber` that captures and processes socket-related system events via BPF (Berkeley Packet Filter). + +| Method | Description | +|--------|-------------| +| `init()` | Initializes the subscriber and registers subscriptions with the BPF publisher | +| `eventCallback(ECRef, SCRef)` | Handles incoming BPF socket events from the publisher | +| `generateRow(Row&, Event&)` | Converts a single system state tracker event into an osquery table row | +| `generateRowList(EventList&)` | Batch-converts a list of events into multiple table rows | ## Usage Example ```c // Subscriber is registered automatically via osquery's event framework. -// Static helpers can be used independently to convert events to rows: +// generateRow is typically called internally during event processing: -ISystemStateTracker::Event socket_event = /* ... */; -Row row; +ISystemStateTracker::Event socket_event = /* ... populated by BPF publisher ... */; +Row row; if (BPFSocketEventSubscriber::generateRow(row, socket_event)) { - // row is now populated with socket event data + // row now contains socket event data (fd, family, protocol, etc.) + results.push_back(row); } -// Batch conversion from an event list +// Batch processing a list of events: ISystemStateTracker::EventList event_list = /* ... */; -std::vector rows = - BPFSocketEventSubscriber::generateRowList(event_list); +std::vector rows = BPFSocketEventSubscriber::generateRowList(event_list); ``` ## Notes -- Operates on Linux only via the `BPFEventPublisher` pipeline -- The `generateRow` / `generateRowList` static methods are useful for testing row generation independently of the event loop -- Follows osquery's standard subscriber pattern: `init()` β†’ `eventCallback()` β†’ row emission \ No newline at end of file +- Belongs to the `osquery` namespace +- Targets **Linux only** β€” depends on `bpfeventpublisher.h` +- Static helper methods (`generateRow`, `generateRowList`) can be used independently for testing or custom row generation without a live subscriber instance \ No newline at end of file diff --git a/osquery/tables/events/linux/.process_events.md b/osquery/tables/events/linux/.process_events.md index fe10862489c..0903a4d6b7a 100644 --- a/osquery/tables/events/linux/.process_events.md +++ b/osquery/tables/events/linux/.process_events.md @@ -1,38 +1,40 @@ -Defines the `AuditProcessEventSubscriber` class, an osquery event subscriber that listens to Linux audit events for process-related system calls (e.g., `execve`, `clone`). +Declares the `AuditProcessEventSubscriber` class, an osquery event subscriber that listens to Linux audit events for process-related syscalls (e.g., `execve`, `clone`). ## Key Components -| Member | Description | -|---|---| -| `init()` | Registers the subscriber's audit event type subscription | -| `Callback()` | Invoked when kernel audit events matching the subscription fire | -| `ProcessEvents()` | Batch-processes a list of `AuditEvent` objects into `Row` output | +**`AuditProcessEventSubscriber`** β€” Final class extending `EventSubscriber`: + +| Method | Description | +|--------|-------------| +| `init()` | Registers the audit event type subscription | +| `Callback()` | Fires when kernel events match the subscribed type | +| `ProcessEvents()` | Transforms a batch of `AuditEvent` objects into emittable `Row` entries | | `ProcessExecveEventData()` | Extracts fields specific to `execve`/`execveat` syscall events | | `GetProcessIDs()` | Parses `pid` and `ppid` from a syscall audit record | -| `IsThreadClone()` | Checks whether a `clone()` syscall includes the `CLONE_THREAD` flag | +| `IsThreadClone()` | Checks whether a `clone()` call includes the `CLONE_THREAD` flag | | `GetSyscallName()` | Resolves a syscall number to its string name | -| `GetSyscallNameMap()` | Returns the static syscall number-to-name lookup table | +| `GetSyscallNameMap()` | Returns the full static syscall number-to-name map | ## Usage Example ```cpp -// Typical subscriber wiring (handled by osquery internals) -AuditProcessEventSubscriber subscriber; -subscriber.init(); // registers audit event subscription +#include -// Static processing used in unit tests or direct invocation +// Manually invoke event processing (e.g., in tests) std::vector rows; std::vector events = getAuditEvents(); auto status = AuditProcessEventSubscriber::ProcessEvents(rows, events); if (!status.ok()) { - LOG(ERROR) << "Failed to process audit events: " << status.getMessage(); + LOG(ERROR) << "Failed to process audit events: " << status.getMessage(); } -// Check if a clone() call spawned a thread vs. a process -bool is_thread = false; -AuditProcessEventSubscriber::IsThreadClone(is_thread, syscall_nr, record); +// Resolve a syscall number to its name +std::string name; +if (AuditProcessEventSubscriber::GetSyscallName(name, 59 /* execve */)) { + LOG(INFO) << "Syscall: " << name; // "execve" +} ``` -> **Note:** All processing methods are declared `static` and `noexcept`, making them independently testable without a live subscriber instance. This class is Linux-only, depending on `AuditEventPublisher` from `osquery/events/linux/`. \ No newline at end of file +All processing methods are `static` and `noexcept`, making them safe to call directly in unit tests without instantiating a full subscriber. \ No newline at end of file diff --git a/osquery/tables/events/linux/.process_file_events.md b/osquery/tables/events/linux/.process_file_events.md index 0d06d5a5937..d7e7d5ec74b 100644 --- a/osquery/tables/events/linux/.process_file_events.md +++ b/osquery/tables/events/linux/.process_file_events.md @@ -1,48 +1,64 @@ -Header file defining the Linux audit-based File Integrity Monitoring (FIM) subsystem for osquery, tracking file system events via syscall interception through the Linux audit framework. +Header file defining the Linux audit-based File Integrity Monitoring (FIM) system for osquery, tracking file system events via kernel syscall interception. ## Key Components ### Data Structures -| Struct/Class | Purpose | -|---|---| -| `AuditdFimInodeDescriptor` | Describes a file/folder inode with path and type | -| `AuditdFimFdDescriptor` | Tracks an open file descriptor and its last operation | -| `AuditdFimPathRecordItem` | Aggregates `AUDIT_PATH` records per syscall | -| `AuditdFimSrcDestData` | Holds source/destination for rename/link operations | -| `AuditdFimIOData` | Holds target and type for open/read/write/close/unlink operations | -| `AuditdFimSyscallContext` | Full context for a captured syscall event (PIDs, UIDs, paths, return values) | -| `AuditdFimContext` | Top-level FIM state: included paths, process map, inode map | +| Type | Purpose | +|------|---------| +| `AuditdFimInodeDescriptor` | Maps an inode to its path and type (file/folder) | +| `AuditdFimFdDescriptor` | Maps a file descriptor to its inode and last operation (open, read, write, etc.) | +| `AuditdFimPathRecordItem` | Aggregates a single `AUDIT_PATH` kernel record | +| `AuditdFimSrcDestData` | Holds source/destination paths for two-path syscalls (`rename`, `link`) | +| `AuditdFimIOData` | Holds target path and I/O type for single-path syscalls | +| `AuditdFimSyscallContext` | Full context for a syscall event including process credentials, paths, and return value | +| `AuditdFimContext` | Top-level FIM state: included path configuration, process map, and inode map | -### Core Classes +### Classes -- **`AuditdFimInodeMap`** β€” Global inode-to-path registry with get/save/remove operations -- **`AuditdFimFdMap`** β€” Per-process file descriptor map supporting open, dup, and close tracking -- **`AuditdFimProcessMap`** β€” Process-level manager over `AuditdFimFdMap` instances; handles `fork`/`clone` via `clone()` and `dup`/`dup2` via `duplicate()` -- **`ProcessFileEventSubscriber`** β€” osquery `EventSubscriber` that consumes `AuditEventPublisher` events and emits rows on read/write activity +- **`AuditdFimInodeMap`** β€” Global inode-to-path registry; supports get, save, remove, and take-and-remove operations +- **`AuditdFimFdMap`** β€” Per-process file descriptor map; handles `dup`/`dup2`/`dup3` duplication with warning suppression +- **`AuditdFimProcessMap`** β€” Manages fd maps across all tracked processes; supports `fork`/`vfork`/`clone` via `clone()` +- **`ProcessFileEventSubscriber`** β€” `EventSubscriber` that consumes `AuditEventPublisher` events, maintains FIM state, and emits rows on read/write activity -### Supported Syscall Types +### Key Methods -```text -Link Β· Symlink Β· Unlink Β· Rename Β· Open Β· OpenTruncate -Close Β· Dup Β· Read Β· Write Β· Truncate Β· Mmap Β· NameToHandleAt Β· CloneOrFork +```cpp +// Process a batch of audit events and emit rows +static Status ProcessEvents( + std::vector& emitted_row_list, + AuditdFimContext& fim_context, + const std::vector& event_list) noexcept; + +// Get the set of syscall numbers this subscriber handles +static const std::set& GetSyscallSet() noexcept; ``` ## Usage Example ```cpp -// Static entry points on the subscriber +// Subscriber is registered automatically via osquery's event framework. +// ProcessEvents can be tested directly: + std::vector rows; AuditdFimContext ctx; +ctx.included_path_list = {"/etc", "/var/log"}; -// Provide audit events from the publisher Status s = ProcessFileEventSubscriber::ProcessEvents( - rows, ctx, event_list); + rows, ctx, incoming_audit_events); -// Query which syscall numbers this subscriber handles -const std::set& syscalls = - ProcessFileEventSubscriber::GetSyscallSet(); +if (s.ok()) { + for (const auto& row : rows) { + // Each row represents a file read/write/unlink/rename event + } +} ``` -> **Note:** This header is Linux-only (depends on `` and the Linux audit framework). The subscriber integrates with osquery's event pipeline via `AuditEventPublisher`. \ No newline at end of file +## Tracked Syscall Types + +```text +Link Β· Symlink Β· Unlink Β· Rename Β· Open Β· OpenTruncate +Close Β· Dup Β· Read Β· Write Β· Truncate Β· Mmap +NameToHandleAt Β· CloneOrFork +``` \ No newline at end of file diff --git a/osquery/tables/events/linux/.seccomp_events.md b/osquery/tables/events/linux/.seccomp_events.md index 0db6a3818e6..009a9a513f5 100644 --- a/osquery/tables/events/linux/.seccomp_events.md +++ b/osquery/tables/events/linux/.seccomp_events.md @@ -1,51 +1,47 @@ -Defines the `SeccompEventSubscriber` class for capturing and parsing Linux seccomp audit events within the osquery event framework. +Header defining the `SeccompEventSubscriber` class, which subscribes to Linux audit events related to `seccomp` (Secure Computing Mode) syscall filtering activity. ## Key Components -### Preprocessor Definitions -Backward-compatibility constants for kernels older than 4.14: +### Compatibility Macros +Backfill constants introduced in Linux kernel 4.14 if missing from the system's `seccomp.h`: -| Constant | Value | Description | +| Macro | Value | Description | |---|---|---| -| `SECCOMP_RET_KILL_PROCESS` | `0x80000000U` | Kill entire process | -| `SECCOMP_RET_KILL_THREAD` | `SECCOMP_RET_KILL` | Kill calling thread | -| `SECCOMP_RET_LOG` | `0x7ffc0000U` | Allow after logging | +| `SECCOMP_RET_KILL_PROCESS` | `0x80000000U` | Kill the entire process | +| `SECCOMP_RET_KILL_THREAD` | `SECCOMP_RET_KILL` | Kill only the offending thread | +| `SECCOMP_RET_LOG` | `0x7ffc0000U` | Allow syscall but log it | ### `SeccompEventSubscriber` -Subscribes to `AuditEventPublisher` events and translates raw audit records into structured query rows. +Extends `EventSubscriber` to capture and decode seccomp audit events. -**Static Lookup Tables** -- `seccomp_actions_map` β€” Maps seccomp return codes to human-readable action names -- `arch_codes_map` β€” Maps architecture codes (from `audit.h`) to architecture name strings +**Static Lookup Maps** +- `seccomp_actions_map` β€” Maps seccomp return codes (e.g. `SECCOMP_RET_KILL`) to human-readable action names +- `arch_codes_map` β€” Maps audit architecture codes to architecture name strings - `syscall_x86_64_map` β€” Maps x86_64 syscall numbers to syscall name strings **Methods** | Method | Description | |---|---| -| `init()` | Registers the audit event type subscription | -| `Callback(ec, sc)` | Fires when a matching kernel seccomp event is received | -| `processEvents(...)` | Batch-processes a list of `AuditEvent` objects into `QueryData` rows | -| `parseEvent(...)` | Internal parser mapping raw `AuditEvent` fields to a `Row` | +| `init()` | Declares the audit event type subscription | +| `Callback(ec, sc)` | Fires on matching kernel audit events | +| `processEvents(emitted_row_list, event_list)` | Batch-processes audit events into query rows | +| `parseEvent(event, parsed_event)` | Internal parser; decodes a single `AuditEvent` into a `Row` | ## Usage Example ```c -// Subscriber is auto-registered via the osquery event framework. -// Emitted rows can be queried via osquery SQL: -// -// SELECT * FROM seccomp_events; -// -// processEvents can be called directly in tests: +// Subscriber is registered automatically via osquery's event framework. +// processEvents can be called directly for testing: QueryData results; -std::vector events = { /* populated by publisher */ }; +std::vector events = getAuditEvents(); Status s = SeccompEventSubscriber::processEvents(results, events); if (s.ok()) { for (const auto& row : results) { - // row["syscall"], row["action"], row["arch"], etc. + // row contains: syscall name, arch, action, pid, etc. } } ``` \ No newline at end of file diff --git a/osquery/tables/events/linux/.selinux_events.md b/osquery/tables/events/linux/.selinux_events.md index b60e6324b86..1c75d8404e6 100644 --- a/osquery/tables/events/linux/.selinux_events.md +++ b/osquery/tables/events/linux/.selinux_events.md @@ -1,43 +1,40 @@ -Defines the `SELinuxEventSubscriber` class, an osquery event subscriber that listens for SELinux-related audit events on Linux via the `AuditEventPublisher`. +Defines the `SELinuxEventSubscriber` class for capturing and processing SELinux-related audit events on Linux systems via the osquery audit event framework. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `SELinuxEventSubscriber` | Class | Event subscriber for SELinux audit events | -| `init()` | Method | Registers the audit event type subscription on startup | -| `Callback()` | Method | Invoked when kernel audit events matching the subscribed type fire | -| `ProcessEvents()` | Static Method | Parses and transforms a batch of `AuditEvent` objects into osquery `Row` results | -| `GetEventSet()` | Static Method | Returns the set of audit event type IDs this subscriber handles | +### `SELinuxEventSubscriber` +Inherits from `EventSubscriber`. Subscribes to Linux kernel audit events related to SELinux enforcement actions. + +| Method | Description | +|--------|-------------| +| `init()` | Registers the subscriber's audit event type subscription at startup | +| `Callback(ec, sc)` | Invoked when a matching kernel audit event fires; receives event and subscription context refs | +| `ProcessEvents(emitted_row_list, event_list)` | Static method that transforms raw `AuditEvent` objects into osquery `Row` entries; `noexcept` | +| `GetEventSet()` | Static method returning the set of audit event type integers this subscriber handles; `noexcept` | ## Usage Example ```cpp -// ProcessEvents can be called directly for unit testing -// without requiring a live audit publisher +// Subscriber is auto-registered via osquery's EventFactory. +// Manual ProcessEvents usage (e.g., in unit tests): std::vector rows; -std::vector events = GetMockSELinuxEvents(); - -Status status = SELinuxEventSubscriber::ProcessEvents(rows, events); +std::vector raw_events = GetTestAuditEvents(); -if (status.ok()) { +Status s = SELinuxEventSubscriber::ProcessEvents(rows, raw_events); +if (s.ok()) { for (const auto& row : rows) { - // Each row contains SELinux event fields - // e.g., row["avc_decision"], row["pid"], row["scontext"] + // Each row contains SELinux audit fields (e.g., avc, syscall, context) } } -// Inspect which audit event codes this subscriber handles -const std::set& handled = SELinuxEventSubscriber::GetEventSet(); -for (int code : handled) { - LOG(INFO) << "Handles audit event code: " << code; -} +// Retrieve handled event type codes: +const auto& event_types = SELinuxEventSubscriber::GetEventSet(); ``` ## Notes -- Inherits from `EventSubscriber`, binding it to Linux's audit subsystem -- `ProcessEvents` and `GetEventSet` are `static` and `noexcept`, making them safely testable in isolation -- Lives inside the `osquery` namespace alongside other platform event subscribers \ No newline at end of file +- Depends on `AuditEventPublisher` β€” Linux-only, requires audit subsystem access +- `ProcessEvents` and `GetEventSet` are static to allow isolated unit testing without a running event loop +- Part of the `osquery` namespace; registration with `EventFactory` is handled by the osquery core infrastructure \ No newline at end of file diff --git a/osquery/tables/events/linux/.socket_events.md b/osquery/tables/events/linux/.socket_events.md index bc3b6b8956a..84154ae3eb5 100644 --- a/osquery/tables/events/linux/.socket_events.md +++ b/osquery/tables/events/linux/.socket_events.md @@ -1,46 +1,45 @@ -Defines the `SocketEventSubscriber` class for capturing and processing Linux socket-related audit events via the osquery audit event framework. +Defines the `SocketEventSubscriber` class for capturing and processing Linux socket-related system call events via the audit subsystem. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `SocketEventSubscriber` | Class | EventSubscriber for `AuditEventPublisher`; handles socket syscall audit events | -| `init()` | Method | Registers audit event type subscriptions on startup | -| `Callback()` | Method | Fires when kernel socket events match the subscribed type | -| `ProcessEvents()` | Static Method | Transforms raw `AuditEvent` list into osquery `Row` results with configurable filtering | -| `GetSyscallSet()` | Static Method | Returns the set of syscall IDs this subscriber handles | -| `parseSockAddr()` | Static Method | Decodes the `saddr` field from `AUDIT_SOCKADDR` records into row columns | +**`SocketEventSubscriber`** β€” Event subscriber that hooks into `AuditEventPublisher` to monitor socket activity. + +| Member | Description | +|--------|-------------| +| `init()` | Registers audit event type subscriptions for socket syscalls | +| `Callback()` | Fires when a kernel event matches the subscribed event type | +| `ProcessEvents()` | Transforms raw `AuditEvent` list into `Row` records with configurable filters | +| `GetSyscallSet()` | Returns the static set of syscall IDs this subscriber handles | +| `parseSockAddr()` | Decodes the `saddr` field from `AUDIT_SOCKADDR` records into structured row data | ## Usage Example ```cpp -#include +#include "socket_events.h" + +// ProcessEvents is the primary static entry point for unit testing or +// direct event processing without the full subscriber lifecycle. -// ProcessEvents is the primary testable interface std::vector rows; -std::vector events = getAuditEvents(); +std::vector events = /* ... collected audit events ... */; Status status = SocketEventSubscriber::ProcessEvents( rows, events, - /*allow_failed_socket_events=*/false, - /*allow_unix_socket_events=*/true, - /*allow_null_accept_events=*/false, - /*allow_null_accept_socket_events=*/false + /* allow_failed_socket_events */ false, + /* allow_unix_socket_events */ true, + /* allow_null_accept_events */ false, + /* allow_null_accept_socket_events */ false ); -// Inspect handled syscalls -const auto& syscalls = SocketEventSubscriber::GetSyscallSet(); - -// Parse a raw saddr field +// Parse a raw saddr string from an AUDIT_SOCKADDR record Row row; -bool is_unix = false; -bool ok = SocketEventSubscriber::parseSockAddr("0200...", row, is_unix); +bool is_unix_socket = false; +bool ok = SocketEventSubscriber::parseSockAddr("0200...", row, is_unix_socket); ``` ## Notes -- Filtering flags in `ProcessEvents` allow fine-grained control over which socket events are emitted (failed, UNIX domain, null-accept variants) -- `parseSockAddr` sets `unix_socket` to `true` when the address family is `AF_UNIX` -- Part of the osquery Linux audit subsystem; requires `AuditEventPublisher` to be active \ No newline at end of file +- Filtering behavior is controlled at `ProcessEvents` call time via boolean flags, covering failed sockets, Unix domain sockets, and null-accept edge cases. +- `parseSockAddr` sets `unix_socket` by reference, allowing callers to branch on address family after parsing. \ No newline at end of file diff --git a/osquery/tables/events/windows/.dns_lookup_events.md b/osquery/tables/events/windows/.dns_lookup_events.md index d14854b274b..8162a6e7f87 100644 --- a/osquery/tables/events/windows/.dns_lookup_events.md +++ b/osquery/tables/events/windows/.dns_lookup_events.md @@ -1,40 +1,31 @@ -Declares the `EtwDNSEventSubscriber` class, an ETW-based Windows event subscriber that captures and processes DNS lookup events via the osquery event system. +Declares the ETW-based DNS lookup event subscriber for Windows, which hooks into the `EtwPublisherDNS` publisher to capture and process DNS resolution events. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `EtwDNSEventSubscriber` | Class | Final subscriber class templated on `EtwPublisherDNS` | -| `init()` | Method | Initializes the subscriber and registers subscriptions | -| `eventCallback()` | Method | Handles incoming DNS ETW events with event and subscription context | +- **`EtwDNSEventSubscriber`** β€” Final class inheriting from `EventSubscriber`. Subscribes to Windows ETW DNS events and handles their lifecycle. + - `init()` β€” Registers the subscriber and configures its subscription context. + - `eventCallback()` β€” Invoked per DNS event; receives an event context (`ECRef`) and subscription context (`SCRef`) for processing captured DNS lookup data. ## Usage Example ```c -// Registered automatically via osquery's event framework. -// The subscriber hooks into EtwPublisherDNS at init time. - -class EtwDNSEventSubscriber final : public EventSubscriber { - public: - Status init() override { - // Subscribe to DNS ETW events - auto sc = createSubscriptionContext(); - subscribe(&EtwDNSEventSubscriber::eventCallback, sc); - return Status::success(); - } - - Status eventCallback(const ECRef& event_context, - const SCRef& subscription_context) { - // Process DNS lookup event data from ETW - // e.g., extract queried hostname, response IPs, PID - return Status::success(); - } -}; +// Subscriber is auto-registered via osquery's event framework. +// Implement eventCallback to handle incoming DNS events: + +Status EtwDNSEventSubscriber::eventCallback( + const ECRef& event_context, + const SCRef& subscription_context) { + + // Extract DNS query/response fields from event_context + // and emit rows to the dns_lookup_events virtual table. + Row r; + r["query"] = event_context->query_name; + r["type"] = event_context->query_type; + addRow(r); + + return Status::success(); +} ``` -## Notes - -- **Windows-only**: Depends on ETW (Event Tracing for Windows) infrastructure via `etw_publisher_dns.h` -- Follows osquery's pub/sub event model β€” the publisher (`EtwPublisherDNS`) emits raw ETW DNS events; this subscriber consumes and tables them -- Registered into the osquery event framework automatically via `REGISTER` macros (typically in the corresponding `.cpp` file) \ No newline at end of file +> **Note:** This subscriber is Windows-only and depends on the ETW (Event Tracing for Windows) infrastructure. It populates the `dns_lookup_events` osquery table with real-time DNS resolution activity. \ No newline at end of file diff --git a/osquery/tables/events/windows/.etw_process_events.md b/osquery/tables/events/windows/.etw_process_events.md index 4beed37bb1e..c0bf3aac979 100644 --- a/osquery/tables/events/windows/.etw_process_events.md +++ b/osquery/tables/events/windows/.etw_process_events.md @@ -1,35 +1,39 @@ -Defines the `EtwProcessEventSubscriber` class, an ETW (Event Tracing for Windows) event subscriber that listens for process-related events published by `EtwPublisherProcesses`. +Defines the ETW (Event Tracing for Windows) process event subscriber that listens to process-related events published by `EtwPublisherProcesses`. ## Key Components -- **`EtwProcessEventSubscriber`** β€” Final class inheriting from `EventSubscriber`. Subscribes to Windows ETW process events within the osquery event framework. - - **`init()`** β€” Initializes the subscriber and registers its event subscriptions. - - **`eventCallback()`** β€” Called by the event framework when a matching process event fires; receives both the event context (`ECRef`) and subscription context (`SCRef`) for processing. +- **`EtwProcessEventSubscriber`** β€” Final class inheriting from `EventSubscriber`. Subscribes to Windows ETW process events and handles their processing via a callback. + +| Member | Description | +|--------|-------------| +| `init()` | Initializes the subscriber and registers subscriptions | +| `eventCallback(ECRef, SCRef)` | Invoked when a process event is received; handles event data processing | ## Usage Example -```c -// Subscriber is registered automatically via osquery's plugin/event system. -// Typical initialization flow: +```cpp +// Subscriber is registered automatically via osquery's event framework. +// Typical interaction pattern: namespace osquery { Status EtwProcessEventSubscriber::init() { - auto subscription = createSubscriptionContext(); - subscribe(&EtwProcessEventSubscriber::eventCallback, subscription); - return Status::success(); + // Register subscription context filters (e.g., process create/terminate) + auto subscription_context = createSubscriptionContext(); + subscribe(&EtwProcessEventSubscriber::eventCallback, subscription_context); + return Status::success(); } Status EtwProcessEventSubscriber::eventCallback( const ECRef& event_context, const SCRef& subscription_context) { - // Handle incoming ETW process event data - // e.g., process creation, termination events - return Status::success(); + // Process incoming ETW process event data + // e.g., log process creation, extract PID, image path, etc. + return Status::success(); } } // namespace osquery ``` -> **Note:** This subscriber is Windows-only. It integrates with osquery's `EventSubscriber` framework, meaning registration and dispatch are handled by the osquery event publisher/subscriber lifecycle β€” no manual instantiation is required. \ No newline at end of file +> The subscriber follows osquery's pub/sub event model β€” `EtwPublisherProcesses` captures raw ETW events from the Windows kernel, while this subscriber consumes and transforms them into queryable table rows. \ No newline at end of file diff --git a/osquery/tables/events/windows/.ntfs_journal_events.md b/osquery/tables/events/windows/.ntfs_journal_events.md index b7185618434..53f12b92c7e 100644 --- a/osquery/tables/events/windows/.ntfs_journal_events.md +++ b/osquery/tables/events/windows/.ntfs_journal_events.md @@ -1,45 +1,34 @@ -Declares the `NTFSEventSubscriber` class and supporting utilities for subscribing to and processing NTFS file system change events on Windows. +Declares the `NTFSEventSubscriber` class and supporting utilities for subscribing to NTFS file system change events on Windows within the osquery event framework. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `NTFSEventSubscriber` | Class | Event subscriber that receives NTFS file change events from `NTFSEventPublisher` | -| `shouldEmit` | Method | Filters events based on subscription context before emission | -| `generateRowFromEvent` | Method | Converts an `NTFSEventRecord` into an osquery `Row` for table output | -| `init` | Method | Initialization routine called once at startup | -| `configure` | Method | Configuration callback; may be invoked multiple times on config changes | -| `Callback` | Method | Primary entry point for receiving events from the publisher | -| `StringList` | Type alias | Convenience alias for `std::vector` | -| `processConfiguration` | Function | Parses subscriber configuration to populate include/exclude path lists and access categories | +- **`NTFSEventSubscriber`** β€” Event subscriber class (extends `EventSubscriber`) that receives, filters, and records NTFS file change events + - `shouldEmit()` β€” Filters events based on subscription context before emission + - `generateRowFromEvent()` β€” Converts a raw `NTFSEventRecord` into a queryable `Row` + - `init()` β€” One-time initialization routine + - `configure()` β€” Configuration callback, may be invoked multiple times on config changes + - `Callback()` β€” Primary handler invoked by the publisher when events arrive + +- **`StringList`** β€” Type alias for `std::vector`, used for path collections + +- **`processConfiguration()`** β€” Parses subscriber configuration, populating include/exclude path lists and access category filters from a given `NTFSEventSubscriptionContextRef` ## Usage Example ```c -// Typical lifecycle managed by the osquery event framework: - +// Typical osquery subscriber registration and callback flow NTFSEventSubscriber subscriber; - -// Called once at startup subscriber.init(); - -// Called on config load or reload subscriber.configure(); -// Populate include/exclude paths from config -StringList access_categories = {"write", "delete"}; -StringList include_paths, exclude_paths; - -processConfiguration( - subscriptionContext, - access_categories, - include_paths, - exclude_paths -); +// Called automatically by the event publisher on each NTFS event +Status result = subscriber.Callback(ec, sc); -// Events arrive automatically via: -// subscriber.Callback(eventContext, subscriptionContext); +// Standalone configuration processing +StringList include_paths, exclude_paths; +StringList access_categories = {"writes", "deletes"}; +processConfiguration(context, access_categories, include_paths, exclude_paths); ``` -> **Note:** This header is Windows-only and depends on `ntfs_event_publisher.h`. The subscriber integrates with the osquery event framework β€” direct instantiation is typically handled by the framework, not application code. \ No newline at end of file +> **Note:** This header is Windows-only and depends on `ntfs_event_publisher.h`. The subscriber integrates with osquery's eventing pipeline β€” direct instantiation outside the framework is not typical in production use. \ No newline at end of file diff --git a/osquery/tables/events/windows/.powershell_events.md b/osquery/tables/events/windows/.powershell_events.md index d53bf51e52f..cc0591db9fc 100644 --- a/osquery/tables/events/windows/.powershell_events.md +++ b/osquery/tables/events/windows/.powershell_events.md @@ -1,53 +1,37 @@ -Header file defining the `PowershellEventSubscriber` class, which subscribes to Windows Event Log events to capture and process PowerShell script block logging activity within the osquery framework. +Defines the `PowershellEventSubscriber` class for capturing and processing Windows PowerShell script block logging events via the Windows Event Log publisher in osquery. ## Key Components -### `PowershellEventSubscriber` -Extends `EventSubscriber` to collect PowerShell execution telemetry from Windows Event Log. - -- **`init()`** β€” Registers subscriptions with the Windows Event Log publisher -- **`Callback()`** β€” Handles incoming PowerShell event log entries - -### `Context` (nested struct) -Holds stateful tracking data across multi-part script block messages: - -| Field | Purpose | -|---|---| -| `script_state_map` | Maps script block IDs to their partial message fragments | -| `character_frequency_map` | Frequency analysis vector (entropy/obfuscation detection) | -| `row_list` | Accumulated rows ready for osquery table emission | -| `last_event_expiration_time` | Tracks stale event cleanup threshold | -| `invalid_event_count` / `expired_event_count` | Diagnostic counters | - -### Static Processing Methods - -| Method | Purpose | -|---|---| -| `parseScriptMessageEvent()` | Extracts a `ScriptMessage` from a raw ptree event | -| `processEventObject()` | Advances context state with a new event | -| `processEventExpiration()` | Evicts incomplete/stale script block entries | -| `generateRow()` | Assembles a final osquery `Row` from completed message fragments | +- **`PowershellEventSubscriber`** β€” Event subscriber that listens to `WindowsEventLogPublisher` for PowerShell execution events (ETW/WEL script block logs) +- **`Context`** β€” Internal state struct tracking in-flight script messages, character frequency analysis data, accumulated rows, and expiration counters +- **`Context::ScriptMessage`** β€” Represents a single fragment of a multi-part PowerShell script block event, including block ID, message content, timestamps, and script path metadata +- **`generateRow()`** β€” Static method that assembles a finalized osquery `Row` from a complete list of script message fragments and a character frequency map +- **`parseScriptMessageEvent()`** β€” Static method that parses a raw `ptree` event object into a `ScriptMessage` struct +- **`processEventObject()`** β€” Static method that feeds a parsed event into the subscriber's `Context`, accumulating fragments by `ScriptBlockID` +- **`processEventExpiration()`** β€” Static method that purges stale/incomplete script block entries from `Context` to prevent unbounded memory growth ## Usage Example ```cpp -// Subscriber is auto-registered via osquery plugin system. -// Manual interaction with static helpers: +// Subscriber is registered automatically via osquery's event framework. +// Direct static utility usage example: boost::optional msg; -PowershellEventSubscriber::parseScriptMessageEvent(msg, event_ptree); +PowershellEventSubscriber::Context ctx; + +// Parse a raw Windows Event Log ptree node +auto status = PowershellEventSubscriber::parseScriptMessageEvent(msg, event_ptree); -if (msg) { - PowershellEventSubscriber::Context ctx; +if (status.ok() && msg) { + // Accumulate into context state PowershellEventSubscriber::processEventObject(ctx, event_ptree); + + // Flush expired incomplete script blocks periodically PowershellEventSubscriber::processEventExpiration(ctx); + // Generate a row once all fragments are collected Row row; - PowershellEventSubscriber::generateRow( - row, ctx.script_state_map[msg->script_block_id], ctx.character_frequency_map - ); + PowershellEventSubscriber::generateRow(row, ctx.script_state_map[msg->script_block_id], ctx.character_frequency_map); } -``` - -> **Note:** PowerShell script block logging must be enabled via Windows Group Policy (`ScriptBlockLogging`) for events to be captured. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/tables/events/windows/.windows_events.md b/osquery/tables/events/windows/.windows_events.md index 7881c43a704..de495e322c8 100644 --- a/osquery/tables/events/windows/.windows_events.md +++ b/osquery/tables/events/windows/.windows_events.md @@ -1,41 +1,40 @@ -Brief header defining the `WindowsEventSubscriber` class, which subscribes to Windows Event Log events and converts them into osquery table rows. +Defines the `WindowsEventSubscriber` class, an osquery event subscriber that listens to Windows Event Log entries and converts them into queryable table rows. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `WindowsEventSubscriber` | Class | Event subscriber for the Windows Event Log publisher | -| `init()` | Method | Initializes the subscriber and registers subscriptions | -| `Callback()` | Method | Handles incoming Windows Event Log events | -| `generateRow()` | Static Method | Converts a `WELEvent` into an osquery `Row` for table output | +- **`WindowsEventSubscriber`** β€” Inherits from `EventSubscriber`; subscribes to Windows Event Log events and processes them for osquery. +- **`init()`** β€” Initializes the subscriber and registers subscriptions with the publisher. +- **`Callback()`** β€” Invoked per event; receives a Windows Event Log event reference and its subscription context for processing. +- **`generateRow()`** β€” Static utility that maps a `WELEvent` struct into an osquery `Row`, suitable for table population. ## Usage Example -```c -// Subscriber is auto-registered via osquery's event framework. -// Implement callback to process incoming Windows events: - -Status WindowsEventSubscriber::Callback(const ECRef& event, - const SCRef& subscription) { - Row row; - WindowsEventSubscriber::generateRow(row, event->windows_event); - add(row); // Adds the row to the osquery results table - return Status::success(); -} - -// generateRow populates an osquery Row from a parsed WELEvent: -void WindowsEventSubscriber::generateRow(Row& row, - const WELEvent& windows_event) { - row["channel"] = windows_event.channel; - row["datetime"] = windows_event.datetime; - row["eventid"] = BIGINT(windows_event.eventId); - // ... additional fields mapped from WELEvent -} +```cpp +// Subscriber registration is handled automatically via osquery's +// EventFactory. Direct usage follows the EventSubscriber pattern: + +class WindowsEventSubscriber + : public EventSubscriber { + public: + Status init() override { + auto subscription = createSubscriptionContext(); + subscribe(&WindowsEventSubscriber::Callback, subscription); + return Status::success(); + } + + Status Callback(const ECRef& event, const SCRef& subscription) { + Row row; + WindowsEventSubscriber::generateRow(row, event->windows_event); + addRow(row); + return Status::success(); + } +}; + +// Generating a row from a WELEvent directly: +Row r; +WELEvent wel_event = /* ... */; +WindowsEventSubscriber::generateRow(r, wel_event); ``` -## Dependencies - -- **`WindowsEventLogPublisher`** β€” the publisher this subscriber binds to via the `EventSubscriber` template -- **`WindowsEventLogParser`** β€” provides the `WELEvent` struct populated before `Callback` is invoked -- **`boost::property_tree`** β€” available as namespace alias `pt` for XML/JSON event data parsing \ No newline at end of file +> **Note:** This subscriber integrates with `WindowsEventLogPublisher` and `WindowsEventLogParser` β€” ensure both are available in your osquery build environment before use. \ No newline at end of file diff --git a/osquery/tables/networking/linux/.iptc_proxy.md b/osquery/tables/networking/linux/.iptc_proxy.md index 5d0148eb1cd..75685d49d27 100644 --- a/osquery/tables/networking/linux/.iptc_proxy.md +++ b/osquery/tables/networking/linux/.iptc_proxy.md @@ -1,5 +1,5 @@ -A C header providing an abstraction layer over `libiptc` for querying Linux netfilter/iptables rules and chains from osquery table implementations. +A proxy abstraction layer over `libiptc` for querying Linux `iptables` firewall rules, providing an opaque handle-based API to iterate chains and rules within a named filter table. ## Key Components @@ -7,27 +7,27 @@ A C header providing an abstraction layer over `libiptc` for querying Linux netf | Type | Description | |------|-------------| -| `iptcproxy_handle` | Opaque handle for an iptables filter context | -| `iptcproxy_chain` | Represents a chain with its name, policy, and packet/byte counters | -| `iptcproxy_rule` | Represents a firewall rule with target, port match data, and full IP header fields | +| `iptcproxy_handle` | Opaque handle representing an open connection to an iptables filter table | +| `iptcproxy_chain` | Chain descriptor with name, default policy, and packet/byte counters | +| `iptcproxy_rule` | Rule descriptor with target, optional port match data, and full IP header criteria | ### `invflags` Bitmask Constants | Constant | Value | Description | |----------|-------|-------------| -| `IPTC_INV_VIA_IN` | `0x01` | Invert inbound interface match | -| `IPTC_INV_VIA_OUT` | `0x02` | Invert outbound interface match | +| `IPTC_INV_VIA_IN` | `0x01` | Invert ingress interface match | +| `IPTC_INV_VIA_OUT` | `0x02` | Invert egress interface match | | `IPTC_INV_SRCIP` | `0x08` | Invert source IP match | | `IPTC_INV_DSTIP` | `0x10` | Invert destination IP match | | `IPTC_INV_PROTO` | `0x40` | Invert protocol match | -| `IPTC_INV_MASK` | `0x7F` | All flags mask | +| `IPTC_INV_MASK` | `0x7F` | All flags combined | ### Functions -- **`iptcproxy_init(filter)`** β€” Opens a handle for the given iptables table (e.g. `"filter"`) -- **`iptcproxy_free(handle)`** β€” Releases resources for an open handle -- **`iptcproxy_first_chain / iptcproxy_next_chain`** β€” Iterates chains in the table -- **`iptcproxy_first_rule / iptcproxy_next_rule`** β€” Iterates rules within a named chain +- **`iptcproxy_init(filter)`** β€” Opens the named iptables table (e.g. `"filter"`) and returns an opaque handle +- **`iptcproxy_free(handle)`** β€” Releases all resources associated with a handle +- **`iptcproxy_first_chain / iptcproxy_next_chain`** β€” Iterator pair for enumerating chains in the table +- **`iptcproxy_first_rule / iptcproxy_next_rule`** β€” Iterator pair for enumerating rules within a named chain ## Usage Example @@ -35,34 +35,32 @@ A C header providing an abstraction layer over `libiptc` for querying Linux netf #include "iptc_proxy.h" #include -void enumerate_rules(void) { - const iptcproxy_handle* h = iptcproxy_init("filter"); +void dump_rules(const char* table) { + const iptcproxy_handle* h = iptcproxy_init(table); if (!h) return; - const iptcproxy_chain* chain = iptcproxy_first_chain(h); - while (chain) { + for (const iptcproxy_chain* c = iptcproxy_first_chain(h); + c != NULL; + c = iptcproxy_next_chain(h)) { + printf("Chain: %s Policy: %s Packets: %llu\n", - chain->chain, - chain->policy, - (unsigned long long)chain->policy_data.pcnt); - - const iptcproxy_rule* rule = iptcproxy_first_rule(chain->chain, h); - while (rule) { - printf(" Target: %s Proto: %u\n", - rule->target, - rule->ip_data.proto); - /* Check inverted source IP flag */ - if (rule->ip_data.invflags & IPTC_INV_SRCIP) { - printf(" (source IP is inverted)\n"); + c->chain, c->policy, c->policy_data.pcnt); + + for (const iptcproxy_rule* r = iptcproxy_first_rule(c->chain, h); + r != NULL; + r = iptcproxy_next_rule(h)) { + + printf(" -> Target: %s Proto: %u\n", + r->target, r->ip_data.proto); + + if (r->match && r->match_data.valid) { + printf(" src_ports: %u-%u dst_ports: %u-%u\n", + r->match_data.spts[0], r->match_data.spts[1], + r->match_data.dpts[0], r->match_data.dpts[1]); } - rule = iptcproxy_next_rule(h); } - - chain = iptcproxy_next_chain(h); } iptcproxy_free(h); } -``` - -> **Note:** Iteration is stateful per handle. Always call `iptcproxy_first_chain` or `iptcproxy_first_rule` before the corresponding `_next_*` calls. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/tables/networking/posix/.interfaces.md b/osquery/tables/networking/posix/.interfaces.md index c51cb5040e5..9e35d3d8d73 100644 --- a/osquery/tables/networking/posix/.interfaces.md +++ b/osquery/tables/networking/posix/.interfaces.md @@ -3,32 +3,17 @@ Declares the `genInterfaceDetails` table generation function for querying networ ## Key Components -- **`genInterfaceDetails(QueryContext& context)`** β€” Generates rows of network interface data (e.g., name, MAC address, IP configuration) for the `interface_details` virtual table. +- **`genInterfaceDetails(QueryContext& context)`** β€” Generates rows of network interface detail data (e.g., name, IP address, MAC, flags) for the `interface_details` virtual table. ## Usage Example ```c -#include +#include "interfaces.h" -namespace osquery { -namespace tables { +osquery::QueryContext ctx; +osquery::QueryData rows = osquery::tables::genInterfaceDetails(ctx); -QueryData genInterfaceDetails(QueryContext& context) { - QueryData results; - // Enumerate network interfaces and populate rows - Row r; - r["interface"] = "eth0"; - r["mac"] = "00:1a:2b:3c:4d:5e"; - results.push_back(r); - return results; +for (const auto& row : rows) { + // Each row contains interface fields: name, address, mac, mtu, etc. } - -} // namespace tables -} // namespace osquery -``` - -## Notes - -- Depends on `osquery/core/core.h` and `osquery/core/tables.h`. -- Platform-specific implementations (Linux, macOS, Windows) are expected in corresponding `.cpp` source files. -- Registered as a virtual SQL table via osquery's table plugin system, queryable as `SELECT * FROM interface_details`. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/tables/networking/posix/.utils.md b/osquery/tables/networking/posix/.utils.md index 7586a4c1ead..e7fc0b35f26 100644 --- a/osquery/tables/networking/posix/.utils.md +++ b/osquery/tables/networking/posix/.utils.md @@ -1,14 +1,14 @@ -Utility header for network address conversion functions used in osquery's network interface tables. Provides cross-platform helpers to convert raw socket/IP structures into human-readable strings. +Utility header for network interface address formatting within the `osquery::tables` namespace. ## Key Components -- **`AF_INTERFACE` macro** β€” Platform alias for interface address family (`AF_PACKET` on Linux, `AF_LINK` on BSD/macOS) -- **`ipAsString(sockaddr*)`** β€” Converts an IPv4/IPv6 `sockaddr` struct to a string -- **`ipAsString(in_addr*)`** β€” Converts an IPv4 `in_addr` struct to a string -- **`macAsString(ifaddrs*)`** β€” Extracts and formats a MAC address from an `ifaddrs` struct -- **`macAsString(char*)`** β€” Formats a MAC address from a raw byte buffer -- **`netmaskFromIP(sockaddr*)`** β€” Returns the CIDR prefix length from a `sockaddr` netmask +- **`AF_INTERFACE` macro** β€” Platform-agnostic alias for interface address family (`AF_PACKET` on Linux, `AF_LINK` on BSD/macOS) +- **`ipAsString(const struct sockaddr *)`** β€” Converts a generic socket address to a human-readable IPv4/IPv6 string +- **`ipAsString(const struct in_addr *)`** β€” Overload for converting an `in_addr` struct directly to an IP string +- **`macAsString(const struct ifaddrs *)`** β€” Extracts and formats a MAC address from an `ifaddrs` interface entry +- **`macAsString(const char *)`** β€” Overload accepting a raw byte buffer representing a MAC address +- **`netmaskFromIP(const struct sockaddr *)`** β€” Returns the CIDR prefix length derived from a netmask socket address ## Usage Example @@ -17,21 +17,22 @@ Utility header for network address conversion functions used in osquery's networ #include struct ifaddrs *ifap; -if (getifaddrs(&ifap) == 0) { - for (struct ifaddrs *ifa = ifap; ifa != nullptr; ifa = ifa->ifa_next) { - if (ifa->ifa_addr == nullptr) continue; +getifaddrs(&ifap); - // Get IP address string - std::string ip = osquery::tables::ipAsString(ifa->ifa_addr); +for (struct ifaddrs *ifa = ifap; ifa != nullptr; ifa = ifa->ifa_next) { + if (ifa->ifa_addr == nullptr) continue; - // Get subnet prefix length - int prefix = osquery::tables::netmaskFromIP(ifa->ifa_netmask); + if (ifa->ifa_addr->sa_family == AF_INET) { + std::string ip = osquery::tables::ipAsString(ifa->ifa_addr); + int prefix = osquery::tables::netmaskFromIP(ifa->ifa_netmask); + // e.g. ip = "192.168.1.10", prefix = 24 + } - // Get MAC address (on link-layer entries) - if (ifa->ifa_addr->sa_family == AF_INTERFACE) { - std::string mac = osquery::tables::macAsString(ifa); - } + if (ifa->ifa_addr->sa_family == AF_INTERFACE) { + std::string mac = osquery::tables::macAsString(ifa); + // e.g. mac = "aa:bb:cc:dd:ee:ff" } - freeifaddrs(ifap); } + +freeifaddrs(ifap); ``` \ No newline at end of file diff --git a/osquery/tables/networking/windows/.interfaces.md b/osquery/tables/networking/windows/.interfaces.md index 666822b762d..0c2dd2c5a00 100644 --- a/osquery/tables/networking/windows/.interfaces.md +++ b/osquery/tables/networking/windows/.interfaces.md @@ -3,23 +3,28 @@ Declares the `genInterfaceDetails` function for querying network interface detai ## Key Components -- **`genInterfaceDetails(QueryContext& context)`** β€” Table generator function that retrieves detailed network interface information, returning a `QueryData` result set for use in the `interfaces` virtual table. +- **`genInterfaceDetails(QueryContext& context)`** β€” Table generator function that retrieves detailed information about network interfaces on the host system. Returns a `QueryData` result set for use in osquery SQL queries. ## Usage Example ```c #include "interfaces.h" -namespace osquery { -namespace tables { +using namespace osquery; +using namespace osquery::tables; -QueryData genInterfaceDetails(QueryContext& context) { - QueryData results; - // Populate rows with network interface details - // (IP addresses, MAC, MTU, flags, etc.) - return results; +// Called by the osquery table plugin infrastructure +QueryContext context; +QueryData results = genInterfaceDetails(context); + +for (const auto& row : results) { + // Each row contains network interface fields + // e.g., row["interface"], row["address"], row["mac"] } +``` + +## Notes -} // namespace tables -} // namespace osquery -``` \ No newline at end of file +- Part of the `osquery::tables` namespace +- Depends on `osquery/core/core.h` and `osquery/core/tables.h` +- Typically invoked by the osquery SQL engine when querying the `interface_details` virtual table \ No newline at end of file diff --git a/osquery/tables/networking/windows/.windows_firewall_rules.md b/osquery/tables/networking/windows/.windows_firewall_rules.md index f8aedf7a821..80dbd9538f3 100644 --- a/osquery/tables/networking/windows/.windows_firewall_rules.md +++ b/osquery/tables/networking/windows/.windows_firewall_rules.md @@ -1,16 +1,16 @@ -Defines types and interfaces for querying Windows Firewall rules within the osquery `tables` namespace, providing structured representations of firewall rule data and error states. +Defines types and interfaces for querying Windows Firewall rules within the osquery tables framework, providing structured representations of firewall rule data and error states. ## Key Components ### `WindowsFirewallError` (enum class) -Enumerates all possible failure points when retrieving firewall rules via the Windows Firewall API (`netfw.h`), from policy access errors through individual rule property read failures. +Enumerates all possible failure points during firewall rule enumeration, from policy access errors (`NetFwPolicyError`) through individual rule property retrieval failures (`RuleNameError`, `RuleActionError`, etc.). ### `WindowsFirewallRule` (struct) Represents a single Windows Firewall rule with the following fields: | Field | Type | Default | -|---|---|---| +|-------|------|---------| | `name` | `std::string` | β€” | | `appName` | `std::string` | β€” | | `action` | `NET_FW_ACTION` | `NET_FW_ACTION_BLOCK` | @@ -24,28 +24,26 @@ Represents a single Windows Firewall rule with the following fields: | `profileBitmask` | `long` | `0` | | `serviceName` | `std::string` | β€” | -### `WindowsFirewallRules` (type alias) -```cpp -using WindowsFirewallRules = std::vector; -``` +### `WindowsFirewallRules` +Type alias for `std::vector`, representing a collection of firewall rules. ### `renderWindowsFirewallRules()` -Converts a collection of `WindowsFirewallRule` structs into osquery `QueryData` for table output. +Converts a `WindowsFirewallRules` collection into osquery `QueryData` for table output. ## Usage Example -```cpp -#include "windows_firewall_rules.h" - +```c +// Populate rules from COM enumeration, then render for osquery WindowsFirewallRules rules; WindowsFirewallRule rule; -rule.name = "Block Inbound Telnet"; -rule.action = NET_FW_ACTION_BLOCK; +rule.name = "Allow HTTP"; +rule.appName = "C:\\app\\server.exe"; +rule.action = NET_FW_ACTION_ALLOW; rule.enabled = true; rule.direction = NET_FW_RULE_DIR_IN; -rule.protocol = 6; // TCP -rule.localPorts = "23"; +rule.localPorts = "80"; +rule.protocol = NET_FW_IP_PROTOCOL_TCP; rules.push_back(rule); QueryData result = renderWindowsFirewallRules(rules); diff --git a/osquery/tables/system/.system_utils.md b/osquery/tables/system/.system_utils.md index c881c3304aa..bb72978359e 100644 --- a/osquery/tables/system/.system_utils.md +++ b/osquery/tables/system/.system_utils.md @@ -3,45 +3,37 @@ Utility header providing context-aware helper functions for resolving users and ## Key Components -| Function | Description | -|---|---| -| `usersFromContext` | Resolves users from a query context; defaults to current user if none specified | -| `pidsFromContext` | Resolves process IDs from a query context; returns all PIDs by default | +| Symbol | Description | +|--------|-------------| +| `usersFromContext` | Resolves a list of users from a query context; defaults to the current user if none specified | +| `pidsFromContext` | Resolves a list of process IDs from a query context; defaults to all PIDs if none specified | -Both functions return a `QueryData` set of rows suitable for direct use in table implementations. +Both functions live in the `osquery::tables` namespace and return `QueryData` rows suitable for direct use in table implementations. ## Usage Example -```c -#include +```cpp +#include "system_utils.h" namespace osquery { namespace tables { -QueryData genUserFiles(QueryContext& context) { +QueryData genMyTable(QueryContext& context) { QueryData results; - // Resolve users from context (or current user if none provided) + // Resolve users from context (or fall back to current user) QueryData users = usersFromContext(context); + // Resolve PIDs from context (or fall back to all PIDs) + QueryData pids = pidsFromContext(context); + + // Iterate and populate results... for (const auto& user : users) { Row r; r["username"] = user.at("username"); results.push_back(r); } - return results; -} - -QueryData genProcessInfo(QueryContext& context) { - // Returns all PIDs by default, or filtered PIDs if context specifies - QueryData pids = pidsFromContext(context); - QueryData results; - for (const auto& pid : pids) { - Row r; - r["pid"] = pid.at("pid"); - results.push_back(r); - } return results; } @@ -49,8 +41,4 @@ QueryData genProcessInfo(QueryContext& context) { } // namespace osquery ``` -## Notes - -- `usersFromContext` defaults `all = false` β€” returns only the current user when no user constraint is present in the query. -- `pidsFromContext` defaults `all = true` β€” enumerates all running processes unless a specific PID constraint is provided. -- Both helpers simplify writing constraint-aware table generators by abstracting context parsing logic. \ No newline at end of file +> The `all` parameter overrides context constraints β€” pass `true` to force full enumeration regardless of what the query context specifies. This is useful in scheduled or administrative queries where user/process filtering should be bypassed. \ No newline at end of file diff --git a/osquery/tables/system/darwin/.asl_utils.md b/osquery/tables/system/darwin/.asl_utils.md index 3bd518f6117..849f3cb02de 100644 --- a/osquery/tables/system/darwin/.asl_utils.md +++ b/osquery/tables/system/darwin/.asl_utils.md @@ -1,39 +1,44 @@ -Utility header for bridging osquery query constraints with Apple System Log (ASL) API operations on macOS. +Utility header for bridging osquery's query interface with Apple System Log (ASL) API operations on macOS. ## Key Components -### Compatibility Macros -- **`OLD_ASL_API`** β€” Defined when `ASL_API_VERSION` is absent; enables inline wrappers (`asl_release`, `asl_next`) that map deprecated ASL API calls to their modern equivalents +### Compatibility Shims (`OLD_ASL_API`) +Inline wrappers that normalize the deprecated pre-10.12 ASL API to match the modern API surface: +- `asl_release(aslmsg)` β†’ wraps `asl_free()` +- `asl_release(aslresponse)` β†’ wraps `aslresponse_free()` +- `asl_next(aslresponse)` β†’ wraps `aslresponse_next()` ### Functions | Function | Description | |---|---| -| `addQueryOp()` | Appends a single constraint operation (ANDed) to an ASL query, converting osquery `ConstraintOperator` and `ColumnType` to ASL equivalents | -| `createAslQuery()` | Builds a complete `aslmsg` query object from an osquery `QueryContext` | -| `readAslRow()` | Reads fields from an ASL message row and populates an osquery `Row` | -| `convertLikeRegex()` | Translates a SQL `LIKE`-style pattern string into a regex for ASL matching | +| `addQueryOp()` | Appends a constraint operation (ANDed) to an ASL query, converting osquery `ConstraintOperator` and `ColumnType` to ASL equivalents | +| `createAslQuery()` | Builds a complete ASL query object from an osquery `QueryContext` | +| `readAslRow()` | Reads an `aslmsg` row and populates an osquery `Row` | +| `convertLikeRegex()` | Converts an SQL `LIKE` pattern string into a regex string for ASL matching | ## Usage Example ```c -// Build an ASL query from an osquery context and iterate results +// Build and execute an ASL query from an osquery context aslmsg query = createAslQuery(context); -asl_object_t client = asl_open(nullptr, nullptr, 0); -aslresponse response = asl_search(client, query); +// Or manually add a constraint operation +aslmsg query = asl_new(ASL_TYPE_QUERY); +addQueryOp(query, ASL_KEY_FACILITY, "com.example.app", + EQUALS, TEXT_TYPE); +// Iterate results and populate rows aslmsg row; -while ((row = asl_next(response)) != nullptr) { +aslresponse resp = asl_search(nullptr, query); +while ((row = asl_next(resp)) != nullptr) { Row r; readAslRow(row, r); results.push_back(r); } - -asl_release(response); +asl_release(resp); asl_release(query); -asl_close(client); ``` -> **Note:** This header is macOS-only and depends on ``. The `OLD_ASL_API` compatibility shim handles pre-10.12 macOS SDK differences where `asl_free`/`aslresponse_free` were used instead of the unified `asl_release`. \ No newline at end of file +> **Note:** This header targets macOS only. The `OLD_ASL_API` guard is defined when `ASL_API_VERSION` is absent, enabling compatibility with pre-macOS 10.12 system headers. \ No newline at end of file diff --git a/osquery/tables/system/linux/.apt_sources.md b/osquery/tables/system/linux/.apt_sources.md index 190773b5b21..74442a3a109 100644 --- a/osquery/tables/system/linux/.apt_sources.md +++ b/osquery/tables/system/linux/.apt_sources.md @@ -1,41 +1,41 @@ -Defines data structures and parsing utilities for APT (Advanced Package Tool) source list entries used by the `apt_sources` osquery table implementation. +Defines data structures and parsing utilities for APT package source entries within the osquery tables module. ## Key Components ### `AptSource` struct -Represents a single APT repository source with the following fields: - -| Field | Type | Description | -|---|---|---| -| `base_uri` | `std::string` | Repository base URI (e.g., `http://archive.ubuntu.com/ubuntu`) | -| `name` | `std::string` | Distribution/suite name | -| `cache_file` | `std::vector` | Components used to construct the local cache filename | +Represents a single APT source entry with the following fields: +- `base_uri` β€” The base URI of the package repository +- `name` β€” The source name/distribution (e.g., `focal`, `stable`) +- `cache_file` β€” Components used to construct the local APT cache filename ### Functions -- **`parseAptSourceLine`** β€” Parses a single line from a traditional `sources.list` format into an `AptSource` struct -- **`parseDeb822Block`** β€” Parses a multi-line [Deb822-format](https://wiki.debian.org/SourcesList#Deb822_format) stanza block into one or more `AptSource` entries -- **`getCacheFilename`** β€” Reconstructs the APT cache filename from the `cache_file` components vector +| Function | Description | +|---|---| +| `parseAptSourceLine()` | Parses a single line from a traditional `sources.list` format into an `AptSource` | +| `parseDeb822Block()` | Parses a multi-line Deb822-format block (`.sources` files) into a vector of `AptSource` entries | +| `getCacheFilename()` | Reconstructs the APT cache filename from its component parts | ## Usage Example -```cpp +```c #include "apt_sources.h" +// Parse a traditional one-line APT source entry osquery::tables::AptSource source; - -// Parse a classic one-line sources.list entry std::string line = "deb http://archive.ubuntu.com/ubuntu focal main restricted"; -Status s = osquery::tables::parseAptSourceLine(line, source); -if (s.ok()) { +osquery::Status status = osquery::tables::parseAptSourceLine(line, source); +if (status.ok()) { std::string cache = osquery::tables::getCacheFilename(source.cache_file); - // cache -> "archive.ubuntu.com_ubuntu_dists_focal_main_binary-amd64_Packages" + // cache -> "archive.ubuntu.com_ubuntu_dists_focal_main_..." } // Parse a Deb822-format block std::vector sources; -std::string block = "Types: deb\nURIs: http://archive.ubuntu.com/ubuntu\nSuites: focal\nComponents: main"; -Status s2 = osquery::tables::parseDeb822Block(block, sources); +std::string block = "Types: deb\nURIs: http://archive.ubuntu.com/ubuntu\n" + "Suites: focal\nComponents: main"; + +osquery::tables::parseDeb822Block(block, sources); ``` \ No newline at end of file diff --git a/osquery/tables/system/linux/.md_tables.md b/osquery/tables/system/linux/.md_tables.md index c48dff397b1..420755292d7 100644 --- a/osquery/tables/system/linux/.md_tables.md +++ b/osquery/tables/system/linux/.md_tables.md @@ -1,5 +1,5 @@ -Header file defining data structures and interfaces for querying Linux MD (Multiple Device) RAID array information within the osquery tables framework. +Header file defining data structures and interfaces for querying Linux MD (Multiple Device) software RAID information within the osquery tables framework. ## Key Components @@ -10,45 +10,36 @@ Header file defining data structures and interfaces for querying Linux MD (Multi | `MDDrive` | Represents a single drive in an MD array (name, position) | | `MDAction` | Tracks ongoing array operations (progress, finish time, speed) | | `MDBitmap` | Holds bitmap metadata (memory usage, chunk size, external file path) | -| `MDDevice` | Full MD array descriptor including drives, status, RAID level, and active actions | +| `MDDevice` | Full MD array device descriptor including drives, RAID level, status, and active operations | | `MDStat` | Top-level container parsed from `/proc/mdstat` (personalities, devices, unused line) | ### Classes -- **`MDInterface`** β€” Abstract base class defining the contract for MD driver interaction. All methods are pure virtual, enabling dependency injection and testability. -- **`MD`** β€” Concrete implementation of `MDInterface` that performs actual ioctl-based system calls and file parsing. +**`MDInterface`** β€” Abstract base class defining the MD query contract: +- `getDiskInfo()` β€” Fetches per-disk info via MD ioctl +- `getArrayInfo()` β€” Fetches array-level info via MD ioctl +- `parseMDStat()` β€” Parses `/proc/mdstat` text into an `MDStat` struct +- `getPathByDevName()` β€” Resolves device path from short name (e.g., `md0`) +- `getDevName()` β€” Resolves device name from major/minor numbers +- `getSuperblkVersion()` β€” Returns the superblock version of an array -### Key Methods - -| Method | Description | -|--------|-------------| -| `getDiskInfo()` | Retrieves per-disk info via MD ioctl for a named array | -| `getArrayInfo()` | Retrieves array-level info via MD ioctl | -| `parseMDStat()` | Parses `/proc/mdstat` lines into an `MDStat` struct | -| `getPathByDevName()` | Resolves device path from short name (e.g., `md0`) | -| `getDevName()` | Resolves device name from major/minor numbers | -| `getSuperblkVersion()` | Returns the superblock version of an array | +**`MD`** β€” Concrete implementation of `MDInterface` using Linux MD driver syscalls. ### Free Function -- **`getDrivesForArray()`** β€” Populates `QueryData` with all drive information for a given array, using an `MDInterface` instance. +- `getDrivesForArray()` β€” Populates `QueryData` with all drive details for a given array, using an `MDInterface` instance (enables dependency injection for testing). ## Usage Example -```c -// Instantiate the concrete MD implementation +```cpp osquery::tables::MD md; -osquery::QueryData results; +osquery::tables::MDStat stat; -// Parse /proc/mdstat content +// Parse /proc/mdstat lines std::vector lines = { /* mdstat file lines */ }; -osquery::tables::MDStat stat; md.parseMDStat(lines, stat); -// Query drives for a specific array +// Query drive details for a specific array +osquery::QueryData results; osquery::tables::getDrivesForArray("md0", md, results); - -// Retrieve array-level info via ioctl -mdu_array_info_t arrayInfo; -bool ok = md.getArrayInfo("md0", arrayInfo); ``` \ No newline at end of file diff --git a/osquery/tables/system/linux/.pci_devices.md b/osquery/tables/system/linux/.pci_devices.md index a2eb32b7272..55c2e06bd0d 100644 --- a/osquery/tables/system/linux/.pci_devices.md +++ b/osquery/tables/system/linux/.pci_devices.md @@ -1,54 +1,48 @@ -Defines data structures and a database class for parsing and querying PCI device information from the system `pci.ids` database file. +Defines data structures and interfaces for parsing and querying PCI device information from the system `pci.ids` database, used to populate osquery PCI device tables. ## Key Components ### Structs -- **`PciModel`** β€” Holds a PCI device model's ID, description, and a map of subsystem info keyed by `" "` -- **`PciVendor`** β€” Holds a vendor's ID, name, and a map of `PciModel` entries keyed by device model ID +- **`PciModel`** β€” Holds a PCI device model's ID, description, and subsystem info map (keyed by `" "`) +- **`PciVendor`** β€” Holds a vendor's ID, name, and a map of `PciModel` entries keyed by model ID ### Class: `PciDB` -Parses a `pci.ids` filestream and exposes lookup methods: +Parses and queries a `pci.ids` database file stream. | Method | Description | -|---|---| -| `PciDB(istream&)` | Constructor β€” parses the `pci.ids` stream into an internal map | -| `getVendorName()` | Looks up vendor name by vendor ID | -| `getModel()` | Looks up model description by vendor and model ID | -| `getSubsystemInfo()` | Looks up subsystem description by vendor, model, and subsystem IDs | +|--------|-------------| +| `PciDB(std::istream&)` | Constructor; parses the provided `pci.ids` filestream | +| `getVendorName(vendor_id, vendor)` | Looks up vendor name by vendor ID | +| `getModel(vendor_id, model_id, model)` | Looks up model description | +| `getSubsystemInfo(vendor_id, model_id, subsystem_vendor_id, subsystem_device_id, subsystem)` | Looks up subsystem description | -Private parsing helpers (`parseLine`, `parseVendor`, `parseModel`, `parseSubsystem`) drive the line-by-line ingestion of the database file. +Private parsing methods (`parseLine`, `parseVendor`, `parseModel`, `parseSubsystem`) handle line-by-line ingestion of the `pci.ids` format. ### Free Functions -- **`extractVendorModelFromPciDBIfPresent()`** β€” Populates an osquery `Row` with vendor/model info from sysfs attributes using a `PciDB` instance -- **`extractPCIClassIDAttrs()`** β€” Populates an osquery `Row` with PCI class identifier data from a sysfs attribute string +- **`extractVendorModelFromPciDBIfPresent`** β€” Populates an osquery `Row` with vendor/model info from sysfs device and subsystem ID attributes +- **`extractPCIClassIDAttrs`** β€” Populates an osquery `Row` with PCI class identifier data from a sysfs class attribute string ## Usage Example ```cpp -#include "pci_devices.h" -#include +std::ifstream pci_ids_file("/usr/share/misc/pci.ids"); +osquery::tables::PciDB pcidb(pci_ids_file); -// Load and query the PCI database -std::ifstream pci_ids("/usr/share/misc/pci.ids"); -osquery::tables::PciDB pcidb(pci_ids); - -std::string vendor_name, model_desc; - -// Lookup vendor +std::string vendor_name; auto status = pcidb.getVendorName("8086", vendor_name); if (status.ok()) { // vendor_name == "Intel Corporation" } -// Lookup model -status = pcidb.getModel("8086", "1234", model_desc); +std::string model_desc; +pcidb.getModel("8086", "1234", model_desc); -// Populate an osquery row from sysfs +// Populate an osquery row from sysfs attributes osquery::Row row; osquery::tables::extractVendorModelFromPciDBIfPresent( - row, "0x80861234", "0x80865678", pcidb); + row, "8086:1234", "8086:5678", pcidb); ``` \ No newline at end of file diff --git a/osquery/tables/system/linux/.processes.md b/osquery/tables/system/linux/.processes.md index ba08b30411a..9d70f537101 100644 --- a/osquery/tables/system/linux/.processes.md +++ b/osquery/tables/system/linux/.processes.md @@ -1,22 +1,24 @@ -Declares a utility function for parsing Linux process cgroup information within the osquery tables subsystem. +Declares a utility function for parsing Linux process cgroup data from `/proc` filesystem entries within the osquery tables subsystem. ## Key Components -- **`parseProcCGroup(const std::string& content)`** β€” Parses the raw string content of a Linux `/proc//cgroup` file and returns the extracted cgroup information as a formatted string. +- **`parseProcCGroup(const std::string& content)`** β€” Parses raw cgroup content strings (typically read from `/proc//cgroup`) and returns a normalized/extracted string representation of the cgroup information. ## Usage Example ```c #include "processes.h" -std::string cgroupContent = "12:cpu:/user.slice\n11:memory:/system.slice\n"; -std::string parsed = osquery::tables::parseProcCGroup(cgroupContent); -// parsed contains the processed cgroup hierarchy data +// Raw content read from /proc//cgroup +std::string rawCgroup = "12:cpuset:/\n11:memory:/user.slice\n0::/user.slice/user-1000.slice"; + +std::string parsed = osquery::tables::parseProcCGroup(rawCgroup); +// parsed contains the extracted cgroup path or identifier ``` ## Notes -- Lives in the `osquery::tables` namespace, consistent with osquery's table-based query architecture. -- Intended for use in Linux process enumeration tables (e.g., `processes`). -- Input is typically the raw file content read from `/proc//cgroup`. \ No newline at end of file +- Lives within the `osquery::tables` namespace, consistent with osquery's table-based architecture. +- Intended for internal use by Linux-specific process table implementations (e.g., `processes` virtual table). +- Input is typically the raw multi-line string content of `/proc//cgroup`. \ No newline at end of file diff --git a/osquery/tables/system/linux/dbus/.uniquedbusconnection.md b/osquery/tables/system/linux/dbus/.uniquedbusconnection.md index 40514efbe3f..584dec5421c 100644 --- a/osquery/tables/system/linux/dbus/.uniquedbusconnection.md +++ b/osquery/tables/system/linux/dbus/.uniquedbusconnection.md @@ -1,42 +1,35 @@ -RAII wrapper for managing D-Bus connection lifetimes in osquery's Linux system tables, ensuring connections are automatically allocated and released. +Provides a RAII wrapper for managing DBus connection lifetimes in osquery's Linux system tables, ensuring automatic allocation and cleanup of `DBusConnection` resources. ## Key Components -### `UniqueDbusConnectionAllocator` -A policy class that implements allocation and deallocation strategies for `DBusConnection*` resources. +- **`UniqueDbusConnectionAllocator`** β€” Allocator policy class implementing the allocate/deallocate interface required by `UniqueResource`. Manages `DBusConnection*` lifecycle. + - `allocate(ResourceType& connection, bool system)` β€” Opens a DBus connection; the `system` flag selects between the system bus (`true`) or session bus (`false`). + - `deallocate(ResourceType& connection)` β€” Closes and unreferences the DBus connection. -| Member | Description | -|---|---| -| `ResourceType` | Alias for `DBusConnection*` | -| `allocate(connection, system)` | Opens a D-Bus connection; `system=true` connects to the system bus, `false` to the session bus | -| `deallocate(connection)` | Closes and unreferences the D-Bus connection | - -### `UniqueDbusConnection` -A type alias combining `UniqueDbusConnectionAllocator` with `UniqueResource<>`, providing automatic RAII lifecycle management for D-Bus connections. +- **`UniqueDbusConnection`** β€” Type alias for `UniqueResource`. The `bool` parameter passed at construction selects the bus type. ## Usage Example -```c +```cpp #include namespace osquery { -Status queryDbusSystemBus() { - // Allocate a system bus connection (auto-deallocated on scope exit) +Status queryDbusService() { + // Connect to the system bus (pass 'false' for session bus) UniqueDbusConnection connection; - // true = system bus, false = session bus - auto status = connection.allocate(true); - if (!status.ok()) { - return status; + Status s = connection.allocate(/* system */ true); + if (!s.ok()) { + return s; } DBusConnection* raw = connection.get(); - // Use raw D-Bus connection for message passing... + // Use raw DBus connection... + // Automatically deallocated when 'connection' goes out of scope return Status::success(); - // connection automatically deallocated here } } // namespace osquery @@ -44,6 +37,6 @@ Status queryDbusSystemBus() { ## Notes -- Inherits from `UniqueResource<>` β€” see `uniqueresource.h` for the full RAII interface -- The `bool` template parameter maps to the `system` flag passed to `allocate()` -- Part of osquery's Linux-only D-Bus infrastructure under `osquery/tables/system/linux/dbus/` \ No newline at end of file +- Depends on `UniqueResource<>` for the RAII pattern; see `uniqueresource.h` for the base template. +- Requires `libdbus-1` (`dbus/dbus.h`) as a system dependency. +- Only applicable on Linux targets within the `osquery` namespace. \ No newline at end of file diff --git a/osquery/tables/system/linux/dbus/.uniquedbusmessage.md b/osquery/tables/system/linux/dbus/.uniquedbusmessage.md index 0cb0b4a68b4..f7e8485fa80 100644 --- a/osquery/tables/system/linux/dbus/.uniquedbusmessage.md +++ b/osquery/tables/system/linux/dbus/.uniquedbusmessage.md @@ -1,40 +1,39 @@ -RAII wrapper for D-Bus message lifecycle management within the osquery Linux system tables infrastructure. +RAII wrapper for managing `DBusMessage*` lifetimes within the osquery D-Bus abstraction layer, ensuring automatic allocation and deallocation of D-Bus method call messages. ## Key Components -### `UniqueDbusMessageAllocator` -A protected allocator class managing `DBusMessage*` resource lifetime: - -- **`allocate()`** β€” Creates a new `DBusMessage*` for a method call, parameterized by destination service, object path, interface, and method name -- **`deallocate()`** β€” Safely releases the `DBusMessage*` resource - -### `UniqueDbusMessage` (type alias) -A `UniqueResource` specialization that combines `UniqueDbusMessageAllocator` with four `const std::string&` constructor arguments, providing automatic cleanup via RAII semantics. +- **`UniqueDbusMessageAllocator`** β€” Policy class defining allocation/deallocation strategies for `DBusMessage*` resources: + - `allocate()` β€” Creates a new D-Bus method call message targeting a destination, object path, interface, and method name + - `deallocate()` β€” Releases the underlying `DBusMessage*` resource +- **`UniqueDbusMessage`** β€” Type alias instantiating `UniqueResource<>` with `UniqueDbusMessageAllocator` and four `const std::string&` constructor parameters, providing scope-bound ownership of a D-Bus message ## Usage Example ```cpp #include -// Create a scoped D-Bus method call message -UniqueDbusMessage message; -auto status = message.initialize( - "org.freedesktop.NetworkManager", // destination - "/org/freedesktop/NetworkManager", // path - "org.freedesktop.NetworkManager", // interface - "GetDevices" // method +// Automatically allocates a DBusMessage* for the given D-Bus target. +// The message is freed when 'msg' goes out of scope. +UniqueDbusMessage msg; + +auto status = msg.get( + "org.freedesktop.NetworkManager", // destination + "/org/freedesktop/NetworkManager", // object path + "org.freedesktop.NetworkManager", // interface + "GetDevices" // method ); if (!status.ok()) { - // handle error + LOG(ERROR) << "Failed to allocate D-Bus message: " << status.getMessage(); + return; } -// message is automatically freed when it goes out of scope +DBusMessage* raw = msg.resource(); // access the raw pointer when needed ``` ## Notes -- Follows the `UniqueResource` pattern used across osquery's D-Bus table helpers for consistent resource safety -- Intended for **Linux only** β€” part of the `osquery/tables/system/linux/dbus/` subsystem -- Dual-licensed under Apache-2.0 or GPL-2.0-only \ No newline at end of file +- Follows the same allocator-policy pattern used across osquery's D-Bus wrappers (see `uniqueresource.h`) +- The destructor is virtual to support potential subclassing of the allocator policy +- Part of osquery's Linux-specific D-Bus table infrastructure \ No newline at end of file diff --git a/osquery/tables/system/linux/dbus/methods/.getstringproperty.md b/osquery/tables/system/linux/dbus/methods/.getstringproperty.md index d87c29e3df1..2f8213d5343 100644 --- a/osquery/tables/system/linux/dbus/methods/.getstringproperty.md +++ b/osquery/tables/system/linux/dbus/methods/.getstringproperty.md @@ -3,24 +3,21 @@ Defines a D-Bus method handler for retrieving a single string property from a sy ## Key Components -- **`GetStringPropertyMethodHandler`** β€” Handler class implementing the D-Bus `Get` property call against the `org.freedesktop.systemd1` destination. Contains: - - `kDestination` β€” Target D-Bus service (`org.freedesktop.systemd1`) - - `kInterface` β€” Property interface (`org.freedesktop.DBus.Properties`) - - `kMethod` β€” D-Bus method name (`Get`) - - `parseReply()` β€” Parses the D-Bus reply message into a `std::string` output, returning an osquery `Status` +- **`GetStringPropertyMethodHandler`** β€” Handler class implementing the D-Bus property fetch logic. Targets the `org.freedesktop.systemd1` destination via the `org.freedesktop.DBus.Properties` interface, calling the `Get` method. Exposes: + - `kDestination`, `kInterface`, `kMethod` β€” compile-time constants identifying the D-Bus endpoint + - `Output` β€” type alias for `std::string`, representing the fetched property value + - `parseReply()` β€” parses a raw `UniqueDbusMessage` reply and populates the output string -- **`GetStringPropertyMethod`** β€” Type alias instantiating `DbusMethod<>` with the handler and two `const std::string&` parameters (interface name and property name) +- **`GetStringPropertyMethod`** β€” Type alias composing `GetStringPropertyMethodHandler` into a fully typed `DbusMethod` template, accepting two `const std::string&` parameters (interface name and property name) ## Usage Example ```cpp #include -using namespace osquery; - -// Invoke to retrieve a string property from a systemd unit +// Instantiate and call the method with interface and property name arguments GetStringPropertyMethod method; -std::string result; +GetStringPropertyMethodHandler::Output result; auto status = method.call( connection, @@ -30,11 +27,11 @@ auto status = method.call( ); if (status.ok()) { - // result contains the property value, e.g. "active" + std::cout << "Property value: " << result << std::endl; } ``` ## Notes -- Protected constructor enforces use only through the `GetStringPropertyMethod` alias or subclassing -- Designed for use within osquery's Linux system tables that introspect systemd unit properties via D-Bus \ No newline at end of file +- Protected constructor/destructor enforce use via the `DbusMethod` template rather than direct instantiation +- Part of the osquery Linux D-Bus subsystem under `osquery/tables/system/linux/dbus/methods/` \ No newline at end of file diff --git a/osquery/tables/system/linux/dbus/methods/.listunitsmethodhandler.md b/osquery/tables/system/linux/dbus/methods/.listunitsmethodhandler.md index a2697f4e24d..63dd5b787e6 100644 --- a/osquery/tables/system/linux/dbus/methods/.listunitsmethodhandler.md +++ b/osquery/tables/system/linux/dbus/methods/.listunitsmethodhandler.md @@ -1,44 +1,46 @@ -Defines the D-Bus method handler for invoking `ListUnits` on the systemd1 Manager interface, parsing the response into structured unit records for use in osquery's Linux system tables. +Defines the D-Bus method handler for querying systemd's `ListUnits` RPC, mapping the response into structured `Unit` objects for osquery's Linux system tables. ## Key Components -### `ListUnitsMethodHandler` -Core handler class that encodes D-Bus targeting constants and reply parsing logic. +**`ListUnitsMethodHandler`** β€” Core handler class implementing the D-Bus `org.freedesktop.systemd1.Manager.ListUnits` method. | Member | Description | -|---|---| -| `kDestination` | D-Bus destination: `org.freedesktop.systemd1` | +|--------|-------------| +| `kDestination` | D-Bus service name: `org.freedesktop.systemd1` | | `kInterface` | Target interface: `org.freedesktop.systemd1.Manager` | | `kMethod` | Method name: `ListUnits` | -| `Unit` | POD struct holding per-unit fields (id, states, job info, object path) | -| `Output` | Alias for `std::vector` | -| `parseReply()` | Parses a raw `DBusMessage` reply into the `Output` vector | +| `Unit` | POD struct holding a single systemd unit's metadata | +| `Output` | Type alias for `std::vector` | +| `parseReply()` | Deserializes a raw `DBusMessage` reply into an `Output` vector | | `readUnitInformation()` | Reads a single unit's fields from a `DBusMessageIter` | -### `ListUnitsMethod` -Type alias combining `ListUnitsMethodHandler` with the generic `DbusMethod` template β€” the primary type used by callers. +**`ListUnitsMethod`** β€” Convenience alias combining the handler with the generic `DbusMethod<>` template for dispatch. + +**`Unit` struct fields:** + +```text +id, description, load_state, active_state, +sub_state, following, path, job_id, job_type, job_path +``` ## Usage Example ```c -#include - +// Instantiate and invoke the method via the type alias +osquery::ListUnitsMethod method; osquery::ListUnitsMethodHandler::Output units; -// Invoke via the composed method type auto conn = /* acquire DBusConnection */; -osquery::ListUnitsMethod method; auto status = method.call(conn, units); if (status.ok()) { for (const auto& unit : units) { - // Access unit.id, unit.active_state, unit.load_state, etc. + printf("Unit: %s | State: %s\n", + unit.id.c_str(), + unit.active_state.c_str()); } } ``` -## Notes - -- Constructor and destructor are `protected` β€” instantiation is intended only through the `DbusMethod` template. -- `Unit::job_id` defaults to `0U`; a non-zero value indicates a pending systemd job. \ No newline at end of file +> `ListUnitsMethod` is the primary entry point; direct use of `ListUnitsMethodHandler` is discouraged as its constructor is `protected`. \ No newline at end of file diff --git a/osquery/tables/system/posix/.authorized_keys.md b/osquery/tables/system/posix/.authorized_keys.md index 4a9167b2c77..2589c07d8db 100644 --- a/osquery/tables/system/posix/.authorized_keys.md +++ b/osquery/tables/system/posix/.authorized_keys.md @@ -1,39 +1,34 @@ -Header file declaring functions for parsing and retrieving SSH authorized keys for system users within the osquery tables framework. +Declares the interface for querying SSH `authorized_keys` files per user within the osquery tables framework. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `genSSHkeysForUser` | Function | Parses the `~/.ssh/authorized_keys` file for a specific user, appending discovered key entries to the `QueryData` results | -| `getAuthorizedKeys` | Function | Entry point for the `authorized_keys` osquery virtual table; resolves user context and aggregates SSH key data across all eligible users | - -Both functions live inside the `osquery::tables` namespace. +- **`genSSHkeysForUser`** β€” Parses and generates SSH key rows for a specific user, identified by `uid`, `gid`, and home `directory`, appending results to a `QueryData` collection using the provided `Logger`. +- **`getAuthorizedKeys`** β€” Entry point for the osquery virtual table query; accepts a `QueryContext` and returns all discovered authorized SSH keys across users as `QueryData`. ## Usage Example -```c +```cpp #include "authorized_keys.h" -// Populate SSH keys for a single user -osquery::QueryData results; -osquery::GlogLogger logger; - -osquery::tables::genSSHkeysForUser( - "1000", // uid - "1000", // gid - "/home/alice", // home directory - results, - logger -); - -// Or query via the full table interface (used internally by osquery) -osquery::QueryContext ctx; -osquery::QueryData allKeys = osquery::tables::getAuthorizedKeys(ctx); -``` +namespace osquery { +namespace tables { + +// Directly generate keys for a single user +QueryData results; +GlogLogger logger; +genSSHkeysForUser("1001", "1001", "/home/alice", results, logger); -## Notes +// Or invoke via the full table query context +QueryContext ctx; +QueryData allKeys = getAuthorizedKeys(ctx); + +for (const auto& row : allKeys) { + // Each row contains fields such as uid, key, key_type, comment, etc. +} + +} // namespace tables +} // namespace osquery +``` -- `genSSHkeysForUser` is designed to be called in a worker process context, accepting a `Logger&` reference to support osquery's inter-process logging via `GlogLogger`. -- `getAuthorizedKeys` integrates with the osquery table registration system and is typically invoked by the query engine rather than called directly. -- Both functions depend on `osquery/core/tables.h` (`QueryData`, `QueryContext`) and `osquery/logger/logger.h`. \ No newline at end of file +> **Note:** Both functions operate within the `osquery::tables` namespace and rely on osquery's `QueryData` and `QueryContext` primitives. `genSSHkeysForUser` is typically called internally by `getAuthorizedKeys` during a table scan, but is exposed here to support per-user worker process invocation across privilege boundaries. \ No newline at end of file diff --git a/osquery/tables/system/posix/.known_hosts.md b/osquery/tables/system/posix/.known_hosts.md index a5c383e0a86..34d328827d3 100644 --- a/osquery/tables/system/posix/.known_hosts.md +++ b/osquery/tables/system/posix/.known_hosts.md @@ -1,32 +1,28 @@ -Declares the interface for parsing SSH `known_hosts` files and enumerating trusted host keys for system users. +Declares the interface for querying SSH `known_hosts` entries as osquery table data. ## Key Components -- **`getKnownHostsKeys(QueryContext& context)`** β€” Primary table generator that queries all known SSH host keys across users on the system. Integrates with the osquery table dispatch system. -- **`impl::genSSHkeysForHosts(...)`** β€” Internal helper that parses a specific user's `known_hosts` file given their `uid`, `gid`, and home `directory`, appending discovered key entries to the provided `QueryData` results set. +- **`getKnownHostsKeys(QueryContext&)`** β€” Primary table generation function; retrieves all SSH known host key entries for the `known_hosts` osquery virtual table. +- **`impl::genSSHkeysForHosts(...)`** β€” Internal implementation helper that parses a user's `known_hosts` file given a UID, GID, and home directory path, appending results to the provided `QueryData` output vector. ## Usage Example -```c -// Typical osquery table dispatch usage -namespace osquery { -namespace tables { - -// Called by the osquery table infrastructure -QueryData results = getKnownHostsKeys(context); - -// Internal usage: parse known_hosts for a specific user -QueryData rows; -impl::genSSHkeysForHosts("1000", "1000", "/home/alice", rows); - -// rows now contains entries from /home/alice/.ssh/known_hosts -} -} +```cpp +#include "known_hosts.h" + +// Querying via the table interface +osquery::QueryContext context; +osquery::QueryData results = osquery::tables::getKnownHostsKeys(context); + +// Or using the internal helper directly (e.g., when iterating users) +osquery::QueryData rows; +osquery::tables::impl::genSSHkeysForHosts( + "1000", // uid + "1000", // gid + "/home/alice", // home directory + rows // output results +); ``` -## Notes - -- The `impl` namespace separates internal parsing logic from the public table interface, allowing unit-testable decomposition. -- Both functions operate within `osquery::tables`, following standard osquery table module conventions. -- The header depends on `osquery/core/query.h` and `osquery/core/tables.h`, which provide `QueryData` and `QueryContext` respectively. \ No newline at end of file +> **Note:** `genSSHkeysForHosts` lives in the `impl` namespace and is intended for internal use β€” typically called once per system user when enumerating all known host keys across accounts. \ No newline at end of file diff --git a/osquery/tables/system/posix/.last.md b/osquery/tables/system/posix/.last.md index a29cd4c63be..5f70d6ed394 100644 --- a/osquery/tables/system/posix/.last.md +++ b/osquery/tables/system/posix/.last.md @@ -1,31 +1,26 @@ -Declares the interface for querying last login/access records from the system's `utmpx` database on Unix-like systems. +Declares the interface for the `last` table plugin, which queries historical login and session records from the system's `utmpx` database. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `genLastAccess` | Function | Primary table generator that queries all last login/access records | -| `impl::genLastAccessForRow` | Function | Internal helper that processes a single `utmpx` entry into a result row | +- **`genLastAccess(QueryContext& context)`** β€” Primary table generation function; retrieves all last-access/login records and returns them as `QueryData`. +- **`impl::genLastAccessForRow(const utmpx& ut, QueryData& results)`** β€” Internal helper that processes a single `utmpx` struct entry and appends it to the results set. ## Usage Example ```c -#include "last.h" - -// Generate all last login records -osquery::QueryContext context; -osquery::QueryData results = osquery::tables::genLastAccess(context); - -// Or process a single utmpx row directly (internal use) -utmpx entry; -// ... populate entry via getutxent() or similar ... -osquery::QueryData rowResults; -osquery::tables::impl::genLastAccessForRow(entry, rowResults); +// Typical table registration and query flow +QueryContext context; +QueryData results = genLastAccess(context); + +// Internal row processing (used within implementation) +struct utmpx entry; +QueryData rows; +impl::genLastAccessForRow(entry, rows); ``` ## Notes -- Depends on `` β€” available on Linux, macOS, and other POSIX systems; **not available on Windows** -- `genLastAccessForRow` lives in the `impl` namespace, signaling it is an internal implementation detail not intended for direct external use -- Feeds the osquery `last` virtual table, which exposes data equivalent to running the `last` command in a terminal \ No newline at end of file +- Depends on ``, making this POSIX/Unix-specific (Linux/macOS). +- Follows the standard osquery table plugin pattern: a public `gen*` entry point backed by internal `impl::` helpers. +- Links against `osquery/core/query.h` and `osquery/core/tables.h` for the `QueryContext` and `QueryData` types. \ No newline at end of file diff --git a/osquery/tables/system/posix/.openssl_utils.md b/osquery/tables/system/posix/.openssl_utils.md index b38f6f48488..6b86837eac0 100644 --- a/osquery/tables/system/posix/.openssl_utils.md +++ b/osquery/tables/system/posix/.openssl_utils.md @@ -1,45 +1,54 @@ -Utility header for extracting metadata and attributes from OpenSSL X.509 certificate objects within the `osquery::tables` namespace. +Header file providing OpenSSL-based utility functions for extracting and inspecting X.509 certificate attributes within the `osquery::tables` namespace. ## Key Components -All functions accept a raw `X509*` pointer and return either `boost::optional`, `boost::optional`, or void. +All functions accept an `X509*` pointer and return `boost::optional` values (returning empty optional on failure). | Function | Returns | Description | -|---|---|---| -| `generateCertificateSHA1Digest` | `optional` | SHA1 fingerprint of the certificate | -| `getCertificateAttributes` | `void` | Populates CA and self-signed boolean flags | -| `getCertificateKeyUsage` | `optional` | Key usage extension value | -| `getCertificateSerialNumber` | `optional` | Certificate serial number | -| `getCertificateAuthorityKeyID` | `optional` | Authority Key Identifier extension | -| `getCertificateSubjectKeyID` | `optional` | Subject Key Identifier extension | -| `getCertificateIssuerName` | `optional` | Issuer DN (legacy format toggle via flag) | -| `getCertificateSubjectName` | `optional` | Subject DN (legacy format toggle via flag) | -| `getCertificateCommonName` | `optional` | CN field from the subject | -| `getCertificateSigningAlgorithm` | `optional` | Signature algorithm (e.g. `sha256WithRSAEncryption`) | -| `getCertificateKeyAlgorithm` | `optional` | Public key algorithm | -| `getCertificateKeyStregth` | `optional` | Key size/strength (note: typo in original symbol name) | -| `getCertificateNotValidBefore` | `optional` | Certificate validity start timestamp | -| `getCertificateNotValidAfter` | `optional` | Certificate validity end timestamp | +|----------|---------|-------------| +| `generateCertificateSHA1Digest` | `string` | SHA1 fingerprint of the certificate | +| `getCertificateAttributes` | `void` | Populates `is_ca` and `is_self_signed` flags | +| `getCertificateKeyUsage` | `string` | Key usage extension value | +| `getCertificateSerialNumber` | `string` | Certificate serial number | +| `getCertificateAuthorityKeyID` | `string` | Authority Key Identifier extension | +| `getCertificateSubjectKeyID` | `string` | Subject Key Identifier extension | +| `getCertificateIssuerName` | `string` | Issuer distinguished name | +| `getCertificateSubjectName` | `string` | Subject distinguished name | +| `getCertificateCommonName` | `string` | Subject CN field | +| `getCertificateSigningAlgorithm` | `string` | Signature algorithm (e.g. `sha256WithRSAEncryption`) | +| `getCertificateKeyAlgorithm` | `string` | Public key algorithm | +| `getCertificateKeyStregth` | `string` | Key size/strength (note: typo in source β€” `Stregth`) | +| `getCertificateNotValidBefore` | `time_t` | Certificate validity start time | +| `getCertificateNotValidAfter` | `time_t` | Certificate validity end time | ## Usage Example -```cpp +```c #include "openssl_utils.h" +#include -// cert is an X509* obtained from OpenSSL (e.g. via PEM/DER parsing) -X509* cert = /* ... */; +void inspectCert(X509* cert) { + using namespace osquery::tables; -bool is_ca = false, is_self_signed = false; -osquery::tables::getCertificateAttributes(cert, is_ca, is_self_signed); + // Extract basic identity fields + if (auto cn = getCertificateCommonName(cert)) { + printf("CN: %s\n", cn->c_str()); + } -if (auto sha1 = osquery::tables::generateCertificateSHA1Digest(cert)) { - std::cout << "Fingerprint: " << *sha1 << "\n"; -} + if (auto sha1 = generateCertificateSHA1Digest(cert)) { + printf("Fingerprint: %s\n", sha1->c_str()); + } + + // Check certificate role + bool is_ca = false, is_self_signed = false; + getCertificateAttributes(cert, is_ca, is_self_signed); -if (auto expiry = osquery::tables::getCertificateNotValidAfter(cert)) { - std::cout << "Expires: " << std::ctime(&(*expiry)); + // Check validity window + if (auto notAfter = getCertificateNotValidAfter(cert)) { + // Compare with current time for expiry checks + } } ``` -> **Note:** `getCertificateKeyStregth` contains a typo in the original symbol name; use as-is to avoid linker errors. \ No newline at end of file +> **Note:** The function `getCertificateKeyStregth` contains a typo in the original source (`Stregth` instead of `Strength`). Match this spelling exactly when calling it. \ No newline at end of file diff --git a/osquery/tables/system/posix/.shell_history.md b/osquery/tables/system/posix/.shell_history.md index 49b8f1564f9..614e0daa47c 100644 --- a/osquery/tables/system/posix/.shell_history.md +++ b/osquery/tables/system/posix/.shell_history.md @@ -1,42 +1,39 @@ -Declares internal helper functions for generating shell history table rows in osquery's `shell_history` virtual table implementation. +Declares internal helper functions for generating shell history table rows from user shell history files and bash sessions. ## Key Components -### Functions +- **`genShellHistoryFromBashSessions`** β€” Parses shell history entries from Bash session files within a given directory for a specified user ID. Accepts a predicate callback to process each resulting `DynamicTableRowHolder`. -- **`genShellHistoryFromBashSessions`** β€” Parses shell history from Bash session files within a given directory for a specific user ID. Accepts a predicate callback for row emission, enabling testability without direct table coupling. +- **`genShellHistoryForUser`** β€” Parses shell history entries for a specific user (by UID/GID) from a home directory. Also accepts a predicate callback for row handling. -- **`genShellHistoryForUser`** β€” Enumerates shell history entries for a user identified by `uid`/`gid` and home `directory`. Feeds results through a `DynamicTableRowHolder` predicate rather than returning data directly. +Both functions are designed with testability in mind β€” the predicate parameter allows callers to inject custom row-handling logic without depending on a live query context. ## Usage Example -```c +```cpp #include "shell_history.h" -// Collect all history rows for a user +// Collect all shell history rows for a user +osquery::QueryData results; + osquery::tables::genShellHistoryForUser( - "1000", // uid - "1000", // gid + "1000", // uid + "1000", // gid "/home/alice", // home directory - [](osquery::DynamicTableRowHolder& row) { - // Process or assert on each emitted history row - std::cout << row["command"] << std::endl; + [&results](osquery::DynamicTableRowHolder& row) { + results.push_back(row->get()); } ); -// Collect Bash session history specifically +// Collect history from active bash sessions osquery::tables::genShellHistoryFromBashSessions( "1000", "/home/alice", - [](osquery::DynamicTableRowHolder& row) { - // Handle each session history row + [&results](osquery::DynamicTableRowHolder& row) { + results.push_back(row->get()); } ); ``` -## Notes - -- Both functions use a **predicate/callback pattern** (`std::function`) to decouple row generation from table registration, making unit testing straightforward without requiring a live osquery table context. -- Declared within `osquery::tables` namespace, consistent with osquery's virtual table architecture. -- Depends on `osquery/core/tables.h` and `osquery/sql/dynamic_table_row.h`. \ No newline at end of file +Both functions live in the `osquery::tables` namespace and integrate with the osquery virtual table framework via `DynamicTableRowHolder` from `osquery/sql/dynamic_table_row.h`. \ No newline at end of file diff --git a/osquery/tables/system/posix/.sudoers.md b/osquery/tables/system/posix/.sudoers.md index 129383a3026..7c24137ab6e 100644 --- a/osquery/tables/system/posix/.sudoers.md +++ b/osquery/tables/system/posix/.sudoers.md @@ -1,32 +1,34 @@ -Declares the interface for parsing and querying sudoers configuration files within the osquery tables framework. +Declares the interface for parsing and querying `sudoers` configuration files within the osquery tables framework. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `genSudoersFile` | Function | Parses a single sudoers file at the given path, recursively handling `#include`/`@include` directives via the `level` depth parameter, and appends results to `QueryData` | -| `genSudoers` | Function | Entry point for the `sudoers` virtual table; dispatches file parsing and returns the full `QueryData` result set | +### Functions -Both symbols reside in the `osquery::tables` namespace. +- **`genSudoersFile`** β€” Parses a single sudoers file (or included fragment) at a given include depth `level`, appending discovered rules into `results`. Handles recursive `#include` / `@include` directives via the `level` counter. +- **`genSudoers`** β€” Primary osquery table generator; accepts a `QueryContext` and returns a `QueryData` set representing all sudoers entries found on the system. ## Usage Example ```cpp #include "sudoers.h" -// Query all sudoers entries via the osquery table interface -osquery::QueryContext ctx; -osquery::QueryData results = osquery::tables::genSudoers(ctx); +// Direct file parsing +osquery::QueryData results; +osquery::tables::genSudoersFile("/etc/sudoers", 0, results); -// Or parse a specific sudoers file directly -osquery::QueryData fileResults; -osquery::tables::genSudoersFile("/etc/sudoers", 0, fileResults); +// Via osquery table context +osquery::QueryContext ctx; +osquery::QueryData rows = osquery::tables::genSudoers(ctx); -for (const auto& row : fileResults) { - // Each row contains parsed sudoers rule fields - // e.g., row.at("source"), row.at("header"), row.at("rule") +for (const auto& row : rows) { + // Each row contains sudoers rule fields + // e.g., row.at("header"), row.at("source"), row.at("rule") } ``` -> **Note:** The `level` parameter in `genSudoersFile` guards against circular or deeply nested `#include` directives. Pass `0` for the top-level file; recursive calls increment this value automatically. \ No newline at end of file +## Notes + +- Both functions live under the `osquery::tables` namespace. +- The `level` parameter in `genSudoersFile` guards against infinite recursion from circular sudoers includes. +- `genSudoers` is the entry point registered with the osquery table registry; `genSudoersFile` is a lower-level helper intended for internal use. \ No newline at end of file diff --git a/osquery/tables/system/windows/.certificates.md b/osquery/tables/system/windows/.certificates.md index 921161b2584..621ffa365f0 100644 --- a/osquery/tables/system/windows/.certificates.md +++ b/osquery/tables/system/windows/.certificates.md @@ -4,32 +4,41 @@ Header for Windows certificate store parsing utilities within osquery's tables m ## Key Components ### Type Aliases -- **`ServiceNameMap`** β€” `std::unordered_map` mapping service names to SIDs for caching lookups +- **`ServiceNameMap`** β€” `std::unordered_map` mapping service names to their SIDs ### Constants | Constant | Value | Description | |---|---|---| -| `kLocalSystem` | `S-1-5-18` | SID for the Local System account | -| `kLocalService` | `S-1-5-19` | SID for the Local Service account | -| `kNetworkService` | `S-1-5-20` | SID for the Network Service account | +| `kLocalSystem` | `S-1-5-18` | Well-known SID for the Local System account | +| `kLocalService` | `S-1-5-19` | Well-known SID for the Local Service account | +| `kNetworkService` | `S-1-5-20` | Well-known SID for the Network Service account | ### Functions -- **`parseSystemStoreString`** β€” Parses a wide-character Windows certificate store path string, resolving the store location, service name or user ID, SID, and store name from the system store identifier + +**`parseSystemStoreString`** β€” Parses a Windows certificate system store string (`LPCWSTR`) and extracts identity and store metadata. + +| Parameter | Type | Description | +|---|---|---| +| `sysStoreW` | `LPCWSTR` | Wide-char system store identifier from the Windows certificate API | +| `storeLocation` | `const std::string&` | Registry/store location context | +| `service2sidCache` | `ServiceNameMap&` | Cache for resolved service name β†’ SID lookups | +| `serviceNameOrUserId` | `std::string&` | Output: resolved service name or user ID | +| `sid` | `std::string&` | Output: resolved security identifier (SID) | +| `storeName` | `std::string&` | Output: certificate store name | ## Usage Example ```c osquery::tables::ServiceNameMap sidCache; -std::string serviceNameOrUserId, sid, storeName; +std::string serviceOrUser, sid, storeName; osquery::tables::parseSystemStoreString( - L"S-1-5-18\\My", // system store wide string - "LocalMachine", // store location context - sidCache, // service-to-SID cache (mutated) - serviceNameOrUserId, // out: resolved service name or user ID - sid, // out: resolved SID string - storeName // out: certificate store name (e.g. "My") + L"S-1-5-18\\My", + "LocalMachine", + sidCache, + serviceOrUser, + sid, + storeName ); -``` - -> **Note:** This header is Windows-specific. It depends on `LPCWSTR` (Windows API type) and is intended for use within osquery's certificate enumeration table implementation. The `ServiceNameMap` cache avoids redundant service-name-to-SID resolution across multiple store entries. \ No newline at end of file +// sid == "S-1-5-18", storeName == "My" +``` \ No newline at end of file diff --git a/osquery/tables/system/windows/.programs.md b/osquery/tables/system/windows/.programs.md index 91da9487db6..1e6b2da32a1 100644 --- a/osquery/tables/system/windows/.programs.md +++ b/osquery/tables/system/windows/.programs.md @@ -1,36 +1,27 @@ -Header file for Windows MSI program registry utilities within the osquery tables subsystem, providing GUID decoding functionality for installed software enumeration. +Header file declaring a Windows MSI registry GUID decoding utility within the osquery tables namespace. ## Key Components -### `decodeMsiRegistryGuid` -```c -std::string decodeMsiRegistryGuid(const std::string& encoded); -``` -Converts a registry-encoded MSI GUID (compressed/shuffled format) into a standard human-readable GUID format. - -- **Input:** 32-character encoded GUID string (e.g., `"0D8797326E7E4114DAECB3B66B9CD045"`) -- **Output:** Standard GUID string (e.g., `"{237978D0-E7E6-4114-ADCE-3B6BB6C90D54}"`) or empty string on invalid input +- **`decodeMsiRegistryGuid(const std::string& encoded)`** β€” Converts a registry-encoded GUID (32-char compact format) into a standard formatted GUID string. ## Usage Example ```c #include "programs.h" -// Decode a registry-encoded MSI GUID +// Encoded registry GUID (32 chars, no hyphens/braces) std::string encoded = "0D8797326E7E4114DAECB3B66B9CD045"; + std::string guid = osquery::tables::decodeMsiRegistryGuid(encoded); +// Returns: "{237978D0-E7E6-4114-ADCE-3B6BB6C90D54}" -if (!guid.empty()) { - // guid == "{237978D0-E7E6-4114-ADCE-3B6BB6C90D54}" - // Use guid to look up installed program metadata -} else { - // Handle invalid/malformed encoded input -} +// Invalid input returns empty string +std::string invalid = osquery::tables::decodeMsiRegistryGuid("short"); +// Returns: "" ``` ## Notes -- Input must be exactly **32 characters** long; any other length returns an empty string -- Used internally by osquery's `programs` table to enumerate installed Windows software from the MSI registry keys under `HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer` -- Namespace: `osquery::tables` \ No newline at end of file +- Input must be exactly **32 characters** long; shorter or longer strings return an empty string. +- Used internally by the `programs` table to decode MSI product GUIDs stored in the Windows registry under paths like `HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData`. \ No newline at end of file diff --git a/osquery/tables/system/windows/.registry.md b/osquery/tables/system/windows/.registry.md index a7340e9b2dc..ed185639de3 100644 --- a/osquery/tables/system/windows/.registry.md +++ b/osquery/tables/system/windows/.registry.md @@ -1,56 +1,49 @@ -Windows Registry query utilities for osquery's table implementations, providing helpers to enumerate, expand, and parse registry keys, paths, hives, and COM class identifiers. +Header defining Windows Registry query utilities for osquery's table implementation, providing helpers for key enumeration, glob expansion, and COM class resolution. ## Key Components -### Constants +**Constants** +- `kRegSep` β€” Registry path separator (`\`) +- `kDefaultRegName` β€” Default key name string `(Default)` +- `kRegMaxRecursiveDepth` β€” Maximum recursion depth for registry traversal (`32`) -| Name | Value | Purpose | -|---|---|---| -| `kRegSep` | `"\\"` | Registry path separator | -| `kDefaultRegName` | `"(Default)"` | Default registry key name | -| `kRegMaxRecursiveDepth` | `32` | Maximum glob expansion depth | - -### Functions +**Functions** | Function | Description | |---|---| -| `queryKey()` | Queries a single registry key path and populates results | -| `queryMultipleRegistryKeys()` | Batch-queries registry keys using LIKE pattern matching | -| `queryMultipleRegistryPaths()` | Batch-queries full registry paths using LIKE pattern matching | -| `getClassName()` | Resolves a COM Class ID (CLSID) to its human-readable class name | -| `getClassExecutables()` | Returns all executables (`.dll`, `.exe`, etc.) associated with a CLSID | -| `expandRegistryGlobs()` | Expands SQL glob patterns (e.g. `HKLM\%\Microsoft`) into matching key sets | -| `explodeRegistryPath()` | Splits a full registry path into its HIVE and KEY components | -| `getUsernameFromKey()` | Extracts a username from an `HKEY_USERS\\...` path | -| `populateSubkeys()` | Expands a key set with discovered subkeys, optionally replacing the input set | +| `queryKey` | Queries a single registry key path and returns results | +| `queryMultipleRegistryKeys` | Batch-queries multiple keys with LIKE pattern support | +| `queryMultipleRegistryPaths` | Batch-queries multiple full registry paths | +| `getClassName` | Resolves a CLSID (e.g. `{0000002F-...}`) to a human-readable class name | +| `getClassExecutables` | Returns executables (`.dll`, `.exe`) associated with a CLSID | +| `expandRegistryGlobs` | Expands SQL glob patterns (e.g. `HKEY_LOCAL_MACHINE\%\Microsoft`) into matching key sets | +| `explodeRegistryPath` | Splits a full path into HIVE and KEY components | +| `getUsernameFromKey` | Extracts a username from an `HKEY_USERS\\...` path | +| `populateSubkeys` | Expands a key set with discovered subkeys | ## Usage Example -```c +```cpp #include using namespace osquery::tables; // Query a single key QueryData results; -Status s = queryKey("HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft", results); +auto status = queryKey("HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft", results); -// Expand a glob pattern to matching keys +// Expand a glob pattern then batch query std::set keys; -expandRegistryGlobs("HKEY_LOCAL_MACHINE\\%\\Microsoft", keys); +expandRegistryGlobs("HKEY_LOCAL_MACHINE\\%\\Run", keys); +queryMultipleRegistryKeys({keys.begin(), keys.end()}, results); -// Split path into hive + key +// Split a path into hive + key std::string hive, key; explodeRegistryPath("HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft", hive, key); // hive = "HKEY_LOCAL_MACHINE", key = "SOFTWARE\\Microsoft" -// Resolve a CLSID to its class name -std::string className; -getClassName("{0000002F-0000-0000-C000-000000000046}", className); - -// Resolve a username from an HKEY_USERS SID path -std::string username; -getUsernameFromKey("HKEY_USERS\\S-1-5-19\\Software", username); -// username = "LOCAL SERVICE" +// Resolve a CLSID to executables +std::vector execs; +getClassExecutables("{0000002F-0000-0000-C000-000000000046}", execs); ``` \ No newline at end of file diff --git a/osquery/tables/system/windows/.security_profile_info_utils.md b/osquery/tables/system/windows/.security_profile_info_utils.md index a0906e96118..e9bb81d63b2 100644 --- a/osquery/tables/system/windows/.security_profile_info_utils.md +++ b/osquery/tables/system/windows/.security_profile_info_utils.md @@ -4,55 +4,42 @@ Utility header providing Windows Security Configuration Engine (SCE) RPC client ## Key Components ### `SceClientHelper` (Singleton Class) -Manages runtime dynamic linking to `scecli.dll` and exposes the undocumented SCE RPC Client API. +Manages runtime dynamic linking to `scecli.dll` and exposes the undocumented SCE RPC API. | Member | Description | -|---|---| -| `instance()` | Returns the singleton `SceClientHelper` instance | -| `getSceSecurityProfileInfo()` | Calls `SceGetSecurityProfileInfo()` to fetch system security profile data | -| `releaseSceProfileData()` | Calls `SceFreeMemory()` to free SCE-allocated memory | -| `isWow64Process()` | Detects WoW64 (32-bit process on 64-bit Windows) via `IsWow64Process()` | -| `SceProfileInfo` | Raw data structure mapping the SCE RPC protocol response (password policy, audit settings, account settings, log configuration) | +|--------|-------------| +| `instance()` | Returns the singleton instance | +| `getSceSecurityProfileInfo()` | Calls `SceGetSecurityProfileInfo()` to fetch system security policy data | +| `releaseSceProfileData()` | Calls `SceFreeMemory()` to free allocated profile memory | +| `isWow64Process()` | Detects if the current process runs under WoW64 emulation | +| `SceProfileInfo` | Raw data structure mapping the SCE RPC protocol response (password policy, audit settings, account flags, etc.) | ### `SceProfileData` (RAII Wrapper) -Scoped memory manager for `SceProfileInfo` data returned by the SCE RPC server. +Scoped lifetime manager for the `SceProfileInfo` memory block. | Member | Description | -|---|---| -| `getProfileInfo()` | Returns typed pointer to the `SceProfileInfo` buffer | +|--------|-------------| +| `getProfileInfo()` | Returns a typed pointer to the underlying `SceProfileInfo` | | `~SceProfileData()` | Automatically calls `SceFreeMemory()` on destruction | -| `getNormalizedInt()` | Normalizes SCE protocol `DWORD` values β€” maps `(DWORD)-1` to `-1` integer, matching `secedit.exe` behavior | +| `getNormalizedInt()` | Normalizes `DWORD` values where `-1` signals a maximum/unlimited policy | ## Usage Example -```cpp -#include "security_profile_info_utils.h" +```c +// Acquire profile data with automatic cleanup +osquery::tables::SceProfileData profileData; -using namespace osquery::tables; +auto& sce = osquery::tables::SceClientHelper::instance(); -void querySecurityProfile() { - // Scoped RAII wrapper handles memory cleanup automatically - SceProfileData profileData; +PVOID rawData = nullptr; +Status status = sce.getSceSecurityProfileInfo(rawData); - // Fetch system security profile via SCE RPC - PVOID rawData = nullptr; - auto& sce = SceClientHelper::instance(); - Status s = sce.getSceSecurityProfileInfo(rawData); - - if (!s.ok()) { - LOG(ERROR) << "SCE call failed: " << s.getMessage(); - return; - } - - // Access typed profile fields - const auto* info = profileData.getProfileInfo(); - if (info != nullptr) { - int minAge = SceProfileData::getNormalizedInt(info->MinPasswdAge); +if (status.ok()) { + const auto* info = profileData.getProfileInfo(); int maxAge = SceProfileData::getNormalizedInt(info->MaxPasswdAge); - // info->LockoutBadCount, info->AuditLogonEvents, etc. - } - // Memory freed automatically when profileData goes out of scope + int minLen = SceProfileData::getNormalizedInt(info->MinPasswdLen); + // profileData destructor calls SceFreeMemory() automatically } ``` -> **Note:** This header is Windows-only and depends on undocumented `scecli.dll` exports (`SceGetSecurityProfileInfo`, `SceFreeMemory`) whose prototypes have remained stable since Windows 7. \ No newline at end of file +> **Note:** This header is Windows-only and targets the undocumented `scecli.dll` SCE RPC interface. The `SceProfileInfo` layout has been stable since Windows 7. WoW64 detection via `isWow64Process()` should be validated before use in 32-bit processes on 64-bit hosts. \ No newline at end of file diff --git a/osquery/tables/system/windows/.startup_items.md b/osquery/tables/system/windows/.startup_items.md index 9c2967ef01f..4f8589bd9f1 100644 --- a/osquery/tables/system/windows/.startup_items.md +++ b/osquery/tables/system/windows/.startup_items.md @@ -1,9 +1,13 @@ -Declares the `parseStartupPath` utility function for extracting executable paths and arguments from macOS startup item entries. +Declares a utility function for parsing macOS startup item path entries into structured table rows for osquery's `startup_items` table implementation. ## Key Components -- **`parseStartupPath(const std::string& entry, Row& r)`** β€” Parses a raw startup path string, splitting it into `path` and `args` fields stored in the provided `Row`. Returns `true` on success, `false` if parsing fails. +### `parseStartupPath` +```c +bool parseStartupPath(const std::string& entry, Row& r); +``` +Parses a raw startup path string, extracting the executable `path` and `args` fields into a `Row` object. Returns `true` on successful parse, `false` otherwise. ## Usage Example @@ -11,18 +15,15 @@ Declares the `parseStartupPath` utility function for extracting executable paths #include "startup_items.h" osquery::Row r; -std::string entry = "/usr/bin/myapp --config /etc/myapp.conf"; +std::string entry = "/usr/bin/myapp --flag value"; if (osquery::tables::parseStartupPath(entry, r)) { // r["path"] == "/usr/bin/myapp" - // r["args"] == "--config /etc/myapp.conf" -} else { - // Handle malformed startup entry + // r["args"] == "--flag value" } ``` ## Notes - -- Lives within the `osquery::tables` namespace, consistent with osquery's table implementation conventions. -- Intended as a helper for the `startup_items` table implementation, isolating path-parsing logic for testability. -- The `Row` type is defined in `osquery/core/tables.h` and maps column names to string values. \ No newline at end of file +- Lives within the `osquery::tables` namespace +- Depends on `osquery/core/tables.h` for the `Row` type +- Intended as an internal helper for the startup items table implementation; not a public API \ No newline at end of file diff --git a/osquery/tables/system/windows/.windows_eventlog.md b/osquery/tables/system/windows/.windows_eventlog.md index b4ccb613a64..21be18ceccb 100644 --- a/osquery/tables/system/windows/.windows_eventlog.md +++ b/osquery/tables/system/windows/.windows_eventlog.md @@ -1,12 +1,19 @@ -Declares two helper functions for querying and parsing Windows Event Log data within the osquery tables framework. +Header file declaring two helper utilities for querying and parsing Windows Event Log data within the osquery tables framework. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `parseWelXml` | Function | Parses a Windows Event Log entry rendered as XML into an osquery `Row` | -| `genXfilterFromConstraints` | Function | Builds an XPath filter string from a `QueryContext` for use with the Windows `EvtQuery` API | +### `parseWelXml` +```text +Status parseWelXml(QueryContext& context, std::wstring& xml_event, Row& row) +``` +Parses a Windows Event Log entry rendered in XML format (`xml_event`) and populates an osquery table `Row`. Returns a `Status` indicating parse success or failure. + +### `genXfilterFromConstraints` +```text +void genXfilterFromConstraints(QueryContext& context, std::string& xfilter) +``` +Derives an XPath filter string (`xfilter`) from the active `QueryContext` constraints. The resulting string is compatible with the Windows `EvtQuery` API for selective event filtering. ## Usage Example @@ -16,19 +23,20 @@ Declares two helper functions for querying and parsing Windows Event Log data wi namespace osquery { namespace tables { -QueryRows genWindowsEventLog(QueryContext& context) { - QueryRows results; - std::string xfilter; +QueryData genWindowsEventLog(QueryContext& context) { + QueryData results; - // Build an XPath filter from the query constraints (e.g., channel, event ID) + // Build XPath filter from query constraints (e.g., WHERE channel='System') + std::string xfilter; genXfilterFromConstraints(context, xfilter); - // ... open event log channel via EvtQuery using xfilter ... + // ... open event log handle using xfilter with EvtQuery ... - std::wstring xml_event = /* rendered XML from EvtRender */; + // For each raw XML event string retrieved: + std::wstring xml_event = /* result from EvtRender */; Row row; - Status status = parseWelXml(context, xml_event, row); - if (status.ok()) { + Status s = parseWelXml(context, xml_event, row); + if (s.ok()) { results.push_back(row); } @@ -41,6 +49,6 @@ QueryRows genWindowsEventLog(QueryContext& context) { ## Notes -- Both functions live in the `osquery::tables` namespace and depend on `osquery/core/core.h` and `osquery/core/tables.h` -- `parseWelXml` accepts a `std::wstring` to handle the wide-character encoding used by the Windows Event Log XML renderer (`EvtRender`) -- `genXfilterFromConstraints` produces an XPath-compatible string, allowing osquery to push SQL `WHERE` clause predicates (channel, event ID, time range, etc.) down to the native Windows API, reducing unnecessary data transfer \ No newline at end of file +- Both functions operate within the `osquery::tables` namespace. +- `parseWelXml` accepts a **wide string** (`std::wstring`) reflecting the UTF-16 encoding used by the Windows Event Log API. +- `xfilter` output from `genXfilterFromConstraints` is intended for direct use with `EvtQuery` from the Windows `winevt.h` API. \ No newline at end of file diff --git a/osquery/tables/system/windows/.windows_update_history.md b/osquery/tables/system/windows/.windows_update_history.md index 4abe6ae1aca..b92d54770cd 100644 --- a/osquery/tables/system/windows/.windows_update_history.md +++ b/osquery/tables/system/windows/.windows_update_history.md @@ -1,47 +1,49 @@ -Header defining types and interfaces for querying Windows Update installation history via the Windows Update Agent (WUA) API. +Declares types and interfaces for querying and rendering Windows Update installation history via the Windows Update Agent (WUA) API. ## Key Components -### `WindowsUpdateHistoryError` (enum class) -Enumerates error conditions that can occur when retrieving update history, covering failures at each stage of the WUA query pipeline β€” from initializing the update searcher through extracting individual entry fields (title, date, description, HResult, etc.). +### Enum: `WindowsUpdateHistoryError` +Enumerates all possible failure points when retrieving update history, including searcher initialization, query execution, and field-level parsing errors (e.g., `TitleError`, `DateError`, `HResultError`). -### `WindowsUpdateHistoryEntry` (struct) -Represents a single Windows Update history record with fields mapped directly from the WUA `IUpdateHistoryEntry` COM interface: +### Struct: `WindowsUpdateHistoryEntry` +Represents a single Windows Update history record with the following fields: | Field | Type | Description | |---|---|---| | `clientAppID` | `std::string` | Application that initiated the update | -| `date` | `LONGLONG` | Install/uninstall timestamp | -| `description` | `std::string` | Update description text | -| `hresult` | `LONG` | Win32 result code | -| `updateOp` | `UpdateOperation` | Install or uninstall operation | +| `date` | `LONGLONG` | Timestamp of the operation | +| `description` | `std::string` | Update description | +| `hresult` | `LONG` | Result code from the operation | +| `updateOp` | `UpdateOperation` | Install or uninstall | | `resultCode` | `OperationResultCode` | Success/failure outcome | -| `serverSelection` | `ServerSelection` | Source server (WSUS, Windows Update, etc.) | -| `updateID` | `std::string` | Unique update GUID | +| `serverSelection` | `ServerSelection` | Update server used | +| `updateID` | `std::string` | Unique update identifier | | `updateRevision` | `LONG` | Revision number of the update | -### `WindowsUpdateHistory` (type alias) -`std::vector` β€” a collection of update history entries. +### Type Alias: `WindowsUpdateHistory` +```cpp +using WindowsUpdateHistory = std::vector; +``` +A collection of update history entries. -### `renderWindowsUpdateHistory()` -Converts a `WindowsUpdateHistory` collection into osquery's `QueryData` format for table output. +### Function: `renderWindowsUpdateHistory` +```cpp +QueryData renderWindowsUpdateHistory(const WindowsUpdateHistory& history); +``` +Converts a `WindowsUpdateHistory` collection into osquery `QueryData` rows suitable for table output. ## Usage Example ```cpp -#include "windows_update_history.h" - -// After populating entries from the WUA COM API: -WindowsUpdateHistory history; -WindowsUpdateHistoryEntry entry; -entry.title = "Security Update KB5012345"; -entry.updateID = "a1b2c3d4-..."; -entry.hresult = S_OK; -entry.updateOp = uoInstallation; -entry.resultCode = orcSucceeded; -history.push_back(entry); - -// Render for osquery table response -QueryData rows = renderWindowsUpdateHistory(history); -``` \ No newline at end of file +WindowsUpdateHistory history = getUpdateHistory(); // populate via WUA API + +QueryData results = renderWindowsUpdateHistory(history); + +for (const auto& row : results) { + // Each row maps column names to string values + // e.g., row["title"], row["hresult"], row["update_id"] +} +``` + +> **Platform Note:** This header is Windows-only. It depends on `` and the Windows Update Agent COM interfaces, and will not compile on non-Windows targets. \ No newline at end of file diff --git a/osquery/tables/yara/.yara_utils.md b/osquery/tables/yara/.yara_utils.md index e77dd1ea1f2..a96c0aeb16c 100644 --- a/osquery/tables/yara/.yara_utils.md +++ b/osquery/tables/yara/.yara_utils.md @@ -1,45 +1,55 @@ -Header file providing YARA integration utilities for osquery, enabling rule compilation, file scanning, and configuration parsing for threat detection workflows. +Header file providing YARA integration utilities for osquery, including rule compilation, scanning callbacks, and configuration parsing for YARA-based file scanning. ## Key Components ### `YaraRulesHandle` -RAII wrapper for `YR_RULES*` that ensures proper cleanup via `yr_rules_destroy()` on destruction. Non-copyable, move-only semantics prevent double-free issues. +RAII wrapper around `YR_RULES*` that automatically calls `yr_rules_destroy()` on destruction. Move-only (copy disabled). -### `YaraCompilerError` / `YaraCompilerResult` -Error enum and `Expected` alias for typed compiler result handling. +- `get()` β€” Returns the underlying `YR_RULES*` pointer -### `YARAConfigParserPlugin` -A `ConfigParserPlugin` that registers a `"yara"` top-level config key. Stores compiled rules grouped by category and maintains a URL allowlist for remote rule sources. +### `YaraCompilerError` / `YaraCompilerResult` +Error enum and `Expected` result type for compiler operations. -### Key Functions +### Free Functions | Function | Description | -|---|---| -| `yaraInitialize()` / `yaraFinalize()` | Lifecycle management for the YARA library | -| `compileSingleFile(file)` | Compiles a YARA rule from a file path | -| `compileFromString(buffer)` | Compiles YARA rules from an in-memory string | -| `handleRuleFiles(category, rule_files, rules)` | Processes a ptree of rule files into a compiled rules map | -| `yaraShouldSkipFile(path, st_mode)` | Guards against scanning files that may cause hangs | -| `YARACallback(...)` | Scan result callback invoked per YARA match event | -| `YARACompilerCallback(...)` | Compiler diagnostic callback for error/warning reporting | +|----------|-------------| +| `yaraInitialize()` | Initializes the YARA library | +| `yaraFinalize()` | Shuts down the YARA library | +| `compileSingleFile(file)` | Compiles YARA rules from a file path | +| `compileFromString(buffer)` | Compiles YARA rules from a string buffer | +| `handleRuleFiles(category, rule_files, rules)` | Loads and compiles rule files from a config subtree | +| `yaraShouldSkipFile(path, st_mode)` | Returns `true` for files that should be skipped to avoid hangs | +| `YARACallback(...)` | Scan result callback invoked by the YARA engine | +| `YARACompilerCallback(...)` | Compiler error/warning callback | + +### `YARAConfigParserPlugin` +A `ConfigParserPlugin` that parses the top-level `"yara"` key from osquery config. + +- `rules()` β€” Returns compiled rules map (`group β†’ YaraRulesHandle`) +- `url_allow_set()` β€” Returns the set of allowed URLs for remote rule fetching +- `setUp()` / `update()` β€” Lifecycle hooks for loading and recompiling rules ## Usage Example ```c -// Initialize YARA, compile a rule file, and scan -if (yaraInitialize().ok()) { - auto result = compileSingleFile("/etc/osquery/yara/malware.yar"); - if (result) { - YaraRulesHandle handle = std::move(result.take()); - yr_rules_scan_file(handle.get(), "/tmp/suspect", 0, YARACallback, &row, 0); - } - yaraFinalize(); +// Initialize YARA and compile rules from a file +yaraInitialize(); + +YaraCompilerResult result = compileSingleFile("/etc/osquery/rules/malware.yar"); +if (result) { + YaraRulesHandle handle = std::move(result.take()); + // Use handle.get() to pass YR_RULES* to YARA scan APIs } -// Access parsed config rules via plugin +yaraFinalize(); + +// Access parsed config rules via the plugin auto parser = Config::getParser("yara"); -auto& compiledRules = parser->rules(); // map +auto& plugin = dynamic_cast(*parser); +auto& rules = plugin.rules(); // map ``` -> **Note:** `kYARAHome` defaults to `OSQUERY_HOME/yara/` as the base directory for local rule files. \ No newline at end of file +**Constants:** +- `kYARAHome` β€” Default YARA rules directory: `/yara/` \ No newline at end of file diff --git a/osquery/utils/.base64.md b/osquery/utils/.base64.md index 586f4812aeb..79053748375 100644 --- a/osquery/utils/.base64.md +++ b/osquery/utils/.base64.md @@ -6,7 +6,7 @@ Provides Base64 encoding and decoding utilities within the `osquery::base64` nam | Function | Signature | Description | |----------|-----------|-------------| | `decode` | `std::string decode(std::string encoded)` | Decodes a Base64-encoded string and returns the original plaintext | -| `encode` | `std::string encode(const std::string_view unencoded)` | Encodes a raw string into its Base64 representation | +| `encode` | `std::string encode(const std::string_view unencoded)` | Encodes a raw string into Base64 format | ## Usage Example @@ -22,4 +22,4 @@ std::string decoded = osquery::base64::decode(encoded); // decoded == "hello world" ``` -> Both functions operate on standard C++ `std::string` / `std::string_view` types, requiring no external dependencies beyond the STL. \ No newline at end of file +> **Note:** `encode` accepts a `std::string_view` for zero-copy input, while `decode` takes a `std::string` by value, allowing the implementation to modify the buffer in place if needed. \ No newline at end of file diff --git a/osquery/utils/.chars.md b/osquery/utils/.chars.md index 22888dbe309..41196b10bce 100644 --- a/osquery/utils/.chars.md +++ b/osquery/utils/.chars.md @@ -4,38 +4,35 @@ Utility header providing character and string encoding helpers within the `osque ## Key Components | Symbol | Type | Description | -|---|---|---| -| `isPrintable` | Function | Checks whether all characters in a `string_view` are ASCII printable | -| `incUtf8StringIterator` | Template Function | Advances an iterator past the current UTF-8 multi-byte sequence, returning bytes consumed | -| `utf8StringSize` | Function | Returns the logical character count (not byte length) of a UTF-8 encoded string | -| `unescapeUnicode` | Function | Converts unicode escape sequences (e.g. `\uXXXX`) in a string to their UTF-8 representation | +|--------|------|-------------| +| `isPrintable` | `bool(string_view)` | Returns `true` if every character in the string is ASCII printable | +| `incUtf8StringIterator` | Template function | Advances an iterator past a single UTF-8 multi-byte sequence; returns bytes consumed | +| `utf8StringSize` | `size_t(string&)` | Returns the number of UTF-8 characters (not raw bytes) in a string | +| `unescapeUnicode` | `string(string&)` | Converts Unicode-escaped sequences (e.g. `\uXXXX`) into their UTF-8 byte representations | ## Usage Example -```cpp +```c #include +#include -// Check if a query result value is safe to display -std::string value = "Hello, World!"; -if (osquery::isPrintable(value)) { - // safe to display +// Check if a value is safe to display +std::string raw = "Hello\x01World"; +if (!osquery::isPrintable(raw)) { + std::cout << "Non-printable characters detected\n"; } -// Get the display length of a UTF-8 string (character count, not bytes) -std::string utf8str = "caf\xC3\xA9"; // "cafΓ©" -size_t len = osquery::utf8StringSize(utf8str); // returns 4, not 5 +// Get true character count of a UTF-8 string +std::string utf8 = "caf\xC3\xA9"; // "cafΓ©" +size_t len = osquery::utf8StringSize(utf8); // returns 4, not 5 -// Decode a unicode escape sequence -std::string escaped = "\\u0041"; // escaped 'A' -std::string decoded = osquery::unescapeUnicode(escaped); // returns "A" +// Unescape a Unicode sequence +std::string escaped = "\\u0041"; // \u0041 = 'A' +std::string result = osquery::unescapeUnicode(escaped); // returns "A" -// Manually iterate UTF-8 characters using the inline helper -auto it = utf8str.begin(); -auto end = utf8str.end(); -while (it != end) { - size_t bytes = osquery::incUtf8StringIterator(it, end); - // bytes = number of bytes in the current character -} +// Manually advance a UTF-8 iterator +auto it = utf8.begin(); +size_t bytes = osquery::incUtf8StringIterator(it, utf8.end()); ``` -> **Note:** `utf8StringSize` counts Unicode code points, not bytes. For ASCII-only strings both values are equal, but multi-byte sequences will differ. \ No newline at end of file +> **Note:** `incUtf8StringIterator` is a low-level helper designed for internal use by `utf8StringSize`. Prefer `utf8StringSize` for character counting over direct iterator manipulation. \ No newline at end of file diff --git a/osquery/utils/.only_movable.md b/osquery/utils/.only_movable.md index bd15d7cdf85..d95183c3e3e 100644 --- a/osquery/utils/.only_movable.md +++ b/osquery/utils/.only_movable.md @@ -1,47 +1,44 @@ -Defines a base class mixin within the `osquery` namespace that enforces move-only semantics, preventing accidental copying of derived objects while explicitly permitting move construction and assignment. +A base class utility that enforces move-only semantics for derived classes, preventing accidental copying while explicitly permitting move operations. ## Key Components -### `only_movable` (class) +**`only_movable` (class)** +A non-copyable, move-enabled abstract base class within the `osquery` namespace. Similar in purpose to `boost::noncopyable`, but extends the pattern by explicitly enabling move semantics. -A non-copyable, move-enabled abstract base class analogous to `boost::noncopyable`. Designed to be inherited by classes that manage exclusive ownership of resources (e.g., file handles, sockets, threads). - -| Member | Type | Description | -|---|---|---| -| `only_movable()` | `protected default` | Default constructor | -| `~only_movable()` | `protected default` | Default destructor | -| `only_movable(only_movable&&)` | `protected default` | Move constructor | -| `operator=(only_movable&&)` | `protected default` | Move assignment | -| `only_movable(const only_movable&)` | `public delete` | Copy constructor β€” **disabled** | -| `operator=(const only_movable&)` | `public delete` | Copy assignment β€” **disabled** | +| Member | Visibility | Description | +|--------|------------|-------------| +| Default constructor | `protected` | Allows subclass instantiation | +| Destructor | `protected` | Standard cleanup | +| Move constructor | `protected` | Enables subclass move construction | +| Move assignment | `protected` | Enables subclass move assignment | +| Copy constructor | `public` (deleted) | Prevents copying | +| Copy assignment | `public` (deleted) | Prevents copy assignment | ## Usage Example ```cpp -#include - -namespace osquery { +#include "only_movable.h" -// Derived class inherits move-only semantics automatically -class ResourceHandle : public only_movable { - public: - ResourceHandle() = default; +class ResourceHandle : public osquery::only_movable { +public: + ResourceHandle() = default; - // Explicitly default move operations from the base - ResourceHandle(ResourceHandle&&) noexcept = default; - ResourceHandle& operator=(ResourceHandle&&) = default; - - // Copy is implicitly deleted via the base class + // Opt into move semantics from the protected base + ResourceHandle(ResourceHandle&&) noexcept = default; + ResourceHandle& operator=(ResourceHandle&&) = default; }; -void example() { - ResourceHandle a; - ResourceHandle b = std::move(a); // OK: move allowed - // ResourceHandle c = b; // ERROR: copy deleted -} +// OK β€” move is allowed +ResourceHandle a; +ResourceHandle b = std::move(a); -} // namespace osquery +// Compile error β€” copy is deleted +// ResourceHandle c = b; ``` -> **Design note:** Deleting the copy members in `public` scope (rather than `private`) produces clearer compiler diagnostics β€” the error explicitly states the function is deleted rather than inaccessible. \ No newline at end of file +## Notes + +- Deleted copy members are declared `public` to produce a clear, explicit compiler error at the call site rather than a less informative access-violation error. +- Derived classes must explicitly `= default` their own move constructor/assignment to expose move capability, since protected members are not implicitly inherited as public. +- Prefer this base class over manual `= delete` boilerplate when a type owns non-copyable resources (file handles, sockets, locks, etc.). \ No newline at end of file diff --git a/osquery/utils/.rot13.md b/osquery/utils/.rot13.md index b811ecddd3f..ed9e43d820a 100644 --- a/osquery/utils/.rot13.md +++ b/osquery/utils/.rot13.md @@ -3,7 +3,7 @@ Provides a ROT13 string decoding utility within the `osquery` namespace. ## Key Components -- **`rotDecode(const std::string& rot_string)`** β€” Decodes a ROT13-encoded string and returns the plaintext result. +- **`rotDecode(const std::string& rot_string)`** β€” Decodes a ROT13-encoded string, shifting each alphabetic character back by 13 positions. Returns the decoded `std::string`. ## Usage Example @@ -16,4 +16,4 @@ std::string decoded = osquery::rotDecode(encoded); // decoded == "Hello, World!" ``` -> **Note:** ROT13 is a symmetric cipher β€” encoding and decoding use the same operation. Calling `rotDecode` on a plaintext string will also produce the ROT13-encoded output. \ No newline at end of file +> **Note:** ROT13 is a symmetric cipher β€” encoding and decoding are the same operation. `rotDecode` can also be used to *encode* plain text strings. \ No newline at end of file diff --git a/osquery/utils/aws/.aws_util.md b/osquery/utils/aws/.aws_util.md index 239572a168a..a8465304856 100644 --- a/osquery/utils/aws/.aws_util.md +++ b/osquery/utils/aws/.aws_util.md @@ -1,52 +1,60 @@ -Utility header for integrating the AWS SDK into osquery, providing HTTP clients, credential providers, and helpers for instantiating AWS service clients with osquery-specific configuration. +Utility header providing AWS SDK integration for osquery, including custom HTTP client implementations, credentials providers, region management, and client factory helpers for EC2, Kinesis, Firehose, and STS services. ## Key Components ### Constants -| Constant | Description | -|---|---| -| `kEc2MetadataUrl` | EC2 instance metadata service URL | -| `kImdsTokenResource` | URL resource for IMDSv2 token requests | -| `kImdsTokenHeader` / `kImdsTokenTtlHeader` | HTTP headers for IMDSv2 API calls | +- `kEc2MetadataUrl` β€” EC2 instance metadata endpoint URL +- `kImdsTokenResource`, `kImdsTokenHeader`, `kImdsTokenTtlHeader`, `kImdsTokenTtlDefaultValue` β€” IMDSv2 token request configuration + +### Enumerations +- `AWSServiceType` β€” Identifies supported services: `EC2`, `Firehose`, `Kinesis`, `STS` +- `AWSRegionError` β€” Region validation error kinds: `Generic`, `NotFIPSCompliant` ### Classes -- **`AWSRegion`** β€” Validated wrapper around an AWS region string; constructed via `AWSRegion::make()` -- **`OsqueryHttpClientFactory`** / **`OsqueryHttpClient`** β€” Drop-in AWS SDK HTTP client backed by osquery's HTTP layer instead of libcurl -- **`OsqueryFlagsAWSCredentialsProvider`** β€” Reads credentials from osquery config flags (`aws_access_key_id`, `aws_secret_access_key`) -- **`OsquerySTSAWSCredentialsProvider`** β€” Obtains temporary credentials via AWS STS assume-role -- **`OsqueryAWSCredentialsProviderChain`** β€” Ordered credential resolution: STS β†’ flags β†’ profile β†’ env vars β†’ default profile β†’ EC2 IMDS - -### Free Functions -- **`initAwsSdk()`** β€” One-time SDK initialization using `OsqueryHttpClientFactory` -- **`makeAWSClient()`** β€” Template factory that instantiates Kinesis, Firehose, STS, or EC2 clients with resolved region and credentials -- **`getIMDSToken()`** β€” Fetches an IMDSv2 session token with retry logic -- **`getInstanceIDAndRegion()`** β€” Returns EC2 instance ID and region from metadata service -- **`setAWSProxy()`** β€” Applies proxy config from osquery flags to a `ClientConfiguration` -- **`appendLogTypeToJson()`** β€” Injects a `log_type` key into a JSON log string -- **`enableFIPSInClientConfig()`** β€” Enables FIPS-compliant endpoints for a given service +- `AWSRegion` β€” Validated AWS region wrapper; constructed via `AWSRegion::make()` +- `OsqueryHttpClientFactory` β€” Custom `HttpClientFactory` replacing libcurl with the osquery HTTP client +- `OsqueryHttpClient` β€” AWS-compatible HTTP client backed by osquery's native HTTP stack +- `OsqueryFlagsAWSCredentialsProvider` β€” Reads credentials from `aws_access_key_id` / `aws_secret_access_key` config flags +- `OsquerySTSAWSCredentialsProvider` β€” Acquires temporary credentials via STS AssumeRole +- `OsqueryAWSCredentialsProviderChain` β€” Ordered credential resolution: STS β†’ flags β†’ profile β†’ env vars β†’ default profile β†’ EC2 IMDS + +### Functions +| Function | Description | +|---|---| +| `initAwsSdk()` | One-time SDK initialization using the custom HTTP factory | +| `getIMDSToken()` | Retrieves an IMDSv2 session token via HTTP PUT | +| `getInstanceIDAndRegion()` | Returns EC2 instance ID and region from metadata service | +| `setAWSProxy()` | Applies proxy settings to a `ClientConfiguration` from flags | +| `setAwsClientConfig()` | Configures a `ClientConfiguration` for a given region and service | +| `makeAWSClient()` | Template factory; instantiates a typed AWS client with full osquery config | +| `appendLogTypeToJson()` | Appends a `log_type` field to a JSON log string | +| `enableFIPSInClientConfig()` | Enables FIPS-compliant endpoints in a `ClientConfiguration` | ## Usage Example ```cpp -// Initialize the SDK once at plugin startup +#include + +// Initialize the AWS SDK once at plugin startup osquery::initAwsSdk(); -// Build a validated region +// Create a validated region auto region_result = osquery::AWSRegion::make("us-east-1"); if (!region_result) { - return Status::failure("Invalid region"); + // handle error } -// Instantiate a Kinesis client using osquery credentials + config -std::shared_ptr client; -Status s = osquery::makeAWSClient(client, region_result.get()); +// Instantiate a Kinesis client using osquery credentials and config +std::shared_ptr kinesis_client; +osquery::Status s = osquery::makeAWSClient(kinesis_client, region_result.get()); if (!s.ok()) { - return s; + LOG(ERROR) << "Failed to create Kinesis client: " << s.getMessage(); } -// Append log type metadata before sending to AWS -std::string log = R"({"host":"web01","level":"info"})"; -osquery::appendLogTypeToJson("result", log); -// log β†’ {"host":"web01","level":"info","log_type":"result"} +// Retrieve EC2 instance metadata +auto meta = osquery::getInstanceIDAndRegion(); +if (meta) { + auto [instance_id, region] = *meta; +} ``` \ No newline at end of file diff --git a/osquery/utils/azure/.azure_util.md b/osquery/utils/azure/.azure_util.md index 9978ff40a42..1e458613a48 100644 --- a/osquery/utils/azure/.azure_util.md +++ b/osquery/utils/azure/.azure_util.md @@ -1,42 +1,27 @@ -Utility header for fetching and parsing Azure Instance Metadata Service (IMDS) responses within the osquery framework. +Utility header for fetching and parsing Azure instance metadata within the osquery framework. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `getAzureKey` | Function | Extracts a string value by key from a parsed Azure metadata JSON document | -| `fetchAzureMetadata` | Function | Queries the Azure IMDS endpoint and populates a `JSON` document with the response | +- **`getAzureKey(JSON& doc, const std::string& key)`** β€” Extracts a string value from a parsed Azure metadata JSON document by key name. +- **`fetchAzureMetadata(JSON& doc)`** β€” Queries the Azure Instance Metadata Service (IMDS) endpoint and populates the provided JSON document with the response. Returns a `Status` indicating success or failure. ## Usage Example ```c #include -namespace osquery { +osquery::JSON doc; +osquery::Status status = osquery::fetchAzureMetadata(doc); -void readAzureInstanceInfo() { - JSON doc; - - // Fetch metadata from Azure IMDS endpoint - Status status = fetchAzureMetadata(doc); - if (!status.ok()) { - // Handle fetch error - return; - } - - // Extract individual fields from the metadata document - std::string vmId = getAzureKey(doc, "vmId"); - std::string region = getAzureKey(doc, "location"); - std::string vmSize = getAzureKey(doc, "vmSize"); +if (status.ok()) { + std::string vmId = osquery::getAzureKey(doc, "vmId"); + std::string location = osquery::getAzureKey(doc, "location"); } - -} // namespace osquery ``` ## Notes -- Both functions live in the `osquery` namespace. -- `fetchAzureMetadata` returns an `osquery::Status`, allowing callers to distinguish between network failures, parse errors, and missing data. -- `getAzureKey` returns an empty `std::string` when the requested key is absent in the document. -- Depends on `osquery/utils/json/json.h` for the `JSON` wrapper type and `osquery/utils/status/status.h` for structured error reporting. \ No newline at end of file +- Both functions operate within the `osquery` namespace. +- `fetchAzureMetadata` is typically called once to populate the `JSON` document before using `getAzureKey` to read individual fields. +- Designed for use on Azure-hosted virtual machines where the IMDS endpoint (`169.254.169.254`) is accessible. \ No newline at end of file diff --git a/osquery/utils/config/.default_paths.md b/osquery/utils/config/.default_paths.md index 31adc838811..abb72948362 100644 --- a/osquery/utils/config/.default_paths.md +++ b/osquery/utils/config/.default_paths.md @@ -1,40 +1,39 @@ -Defines platform-specific default filesystem paths used by osquery for configuration, storage, logging, and runtime files across Linux, Windows, FreeBSD, and other Unix-like systems. +Defines platform-specific default directory paths used by osquery across Linux, Windows, FreeBSD, and other POSIX systems. ## Key Components | Macro | Purpose | |---|---| -| `OSQUERY_HOME` | Base directory for config, flagfile, extensions, and module autoload | +| `OSQUERY_HOME` | Base directory for configuration, flagfiles, extensions, and module autoload | | `OSQUERY_DB_HOME` | RocksDB persistent storage location | -| `OSQUERY_SOCKET` | Unix domain socket (or Windows named pipe) path | +| `OSQUERY_SOCKET` | IPC socket or named pipe path | | `OSQUERY_PIDFILE` | Process ID file directory | | `OSQUERY_LOG_HOME` | Log output directory (used by the filesystem logger plugin) | -| `OSQUERY_CERTS_HOME` | TLS/SSL certificate bundle directory | +| `OSQUERY_CERTS_HOME` | TLS certificate store location | ## Platform Defaults | Macro | Linux | Windows | FreeBSD | Other | |---|---|---|---|---| | `OSQUERY_HOME` | `/etc/osquery/` | `\Program Files\osquery\` | `/var/db/osquery/` | `/var/osquery/` | -| `OSQUERY_DB_HOME` | `/var/osquery/` | *(same as HOME)* | *(same as HOME)* | *(same as HOME)* | -| `OSQUERY_SOCKET` | `/var/osquery/` | `\\.\pipe\` | `/var/run/` | *(same as DB_HOME)* | -| `OSQUERY_LOG_HOME` | `/var/log/osquery/` | `...\log\` | `/var/log/osquery/` | `/var/log/osquery/` | -| `OSQUERY_CERTS_HOME` | `/opt/osquery/share/osquery/certs/` | `...\certs\` | `/etc/ssl/` | `...certs/` | +| `OSQUERY_DB_HOME` | `/var/osquery/` | _(same as HOME)_ | _(same as HOME)_ | _(same as HOME)_ | +| `OSQUERY_LOG_HOME` | `/var/log/osquery/` | `%HOME%\log\` | `/var/log/osquery/` | `/var/log/osquery/` | +| `OSQUERY_CERTS_HOME` | `/opt/osquery/share/osquery/certs/` | `%HOME%\certs\` | `/etc/ssl/` | `%HOME%certs/` | ## Usage Example ```c #include "default_paths.h" -#include - -int main(void) { - printf("Config home: %s\n", OSQUERY_HOME); - printf("Database home: %s\n", OSQUERY_DB_HOME); - printf("Log directory: %s\n", OSQUERY_LOG_HOME); - printf("Certificates: %s\n", OSQUERY_CERTS_HOME); - return 0; -} + +// Reference a compiled-in default path at runtime +const char *config_dir = OSQUERY_HOME; // e.g. "/etc/osquery/" on Linux +const char *db_dir = OSQUERY_DB_HOME; // e.g. "/var/osquery/" on Linux +const char *log_dir = OSQUERY_LOG_HOME; // e.g. "/var/log/osquery/" on Linux +const char *certs_dir = OSQUERY_CERTS_HOME; // e.g. "/opt/osquery/share/osquery/certs/" + +// Compose a full path using string concatenation +const char *db_path = OSQUERY_DB_HOME "osquery.db"; ``` -> These macros serve as compile-time defaults and are typically overridden at runtime via CLI flags or configuration files. \ No newline at end of file +> These macros are compile-time constants. Runtime path overrides should be applied via flagfiles or configuration rather than modifying this header. \ No newline at end of file diff --git a/osquery/utils/conversions/.split.md b/osquery/utils/conversions/.split.md index 80799ce338d..80de978ca0d 100644 --- a/osquery/utils/conversions/.split.md +++ b/osquery/utils/conversions/.split.md @@ -3,33 +3,33 @@ Utility header providing string splitting functions within the `osquery` namespa ## Key Components -| Function | Signature | Description | -|---|---|---| -| `split` | `(string, delim="\t ")` | Splits a string by a delimiter; defaults to whitespace/tab | -| `split` | `(string, char, occurrences)` | Splits a string by a char delimiter up to N times | -| `vsplit` | `(string_view, char)` | High-performance split returning `string_view` tokens (2x+ faster) | +| Function | Description | +|----------|-------------| +| `split(s, delim)` | Splits a string by a delimiter string (defaults to tab/space whitespace) | +| `split(s, delim, occurrences)` | Splits a string by a char delimiter up to a maximum number of splits | +| `vsplit(source, delimiter)` | High-performance split returning `string_view` slices β€” at least 2Γ— faster than `split` | ## Usage Example ```cpp -#include +#include "split.h" // Whitespace split (default) -auto tokens = osquery::split("foo bar baz"); -// β†’ ["foo", "bar", "baz"] +auto tokens = osquery::split("foo bar\tbaz"); +// tokens β†’ ["foo", "bar", "baz"] -// Custom string delimiter +// Delimiter string split auto parts = osquery::split("a,b,c", ","); -// β†’ ["a", "b", "c"] +// parts β†’ ["a", "b", "c"] -// Limited occurrences +// Limited occurrences split auto limited = osquery::split("a:b:c:d", ':', 2); -// β†’ ["a", "b", "c:d"] +// limited β†’ ["a", "b", "c:d"] -// High-performance string_view split (no heap allocations per token) +// Fast string_view split (no allocations per element) std::string_view line = "col1|col2|col3"; auto views = osquery::vsplit(line, '|'); -// β†’ ["col1", "col2", "col3"] as string_views +// views β†’ ["col1", "col2", "col3"] (string_views into original buffer) ``` -> **Performance note:** Prefer `vsplit` over `split` when processing large strings or log lines where you need fast tokenization without unnecessary heap allocations. Since it returns `std::string_view` tokens, ensure the source string outlives the result vector. \ No newline at end of file +> **Note:** Prefer `vsplit` for performance-critical paths where only a subset of elements is needed, as it avoids string copies by returning `std::string_view` slices into the original buffer. Ensure the source string outlives the returned views. \ No newline at end of file diff --git a/osquery/utils/conversions/.trim.md b/osquery/utils/conversions/.trim.md index a04e925d550..31e101a98f8 100644 --- a/osquery/utils/conversions/.trim.md +++ b/osquery/utils/conversions/.trim.md @@ -1,9 +1,9 @@ -Provides a utility function for trimming leading and trailing whitespace from a `std::string_view` within the `osquery` namespace. +Provides a utility function for trimming leading and trailing whitespace from string views within the `osquery` namespace. ## Key Components -- **`osquery::trim(std::string_view input)`** β€” Returns a `std::string_view` with all leading and trailing spaces removed from the input. Non-owning; no heap allocation. +- **`trim(std::string_view input)`** β€” Adjusts a `std::string_view` to exclude any leading and trailing whitespace characters, returning the trimmed view without allocating a new string. ## Usage Example @@ -14,8 +14,8 @@ Provides a utility function for trimming leading and trailing whitespace from a int main() { std::string_view raw = " hello world "; std::string_view trimmed = osquery::trim(raw); - std::cout << trimmed; // "hello world" + std::cout << trimmed; // Output: "hello world" } ``` -> **Note:** The returned `std::string_view` references the original string's memory. Ensure the source string outlives the returned view. \ No newline at end of file +> **Note:** Since the function returns a `std::string_view`, no heap allocation occurs. The returned view points into the original string's memory β€” ensure the source string outlives the view. \ No newline at end of file diff --git a/osquery/utils/conversions/.tryto.md b/osquery/utils/conversions/.tryto.md index 23304fc2f0e..c400dd63d6f 100644 --- a/osquery/utils/conversions/.tryto.md +++ b/osquery/utils/conversions/.tryto.md @@ -1,47 +1,44 @@ -Provides type-safe conversion utilities for transforming between C++ types (string-to-integer, string-to-bool, and identity conversions) using the `Expected` error-handling pattern instead of exceptions. +Provides type-safe conversion utilities for transforming values between types (same-type passthrough, string-to-integer, and string-to-boolean) using the `Expected` error-handling pattern to avoid exceptions in calling code. ## Key Components -### `ConversionError` Enum -Error codes returned on failed conversions: -- `InvalidArgument` β€” input string cannot be parsed -- `OutOfRange` β€” parsed value exceeds the target type's range -- `Unknown` β€” unexpected error during conversion - -### `tryTo(from, base)` β€” Primary API -Overloaded template function with three specializations: -- **Identity conversion** β€” when `ToType` matches `FromType`, forwards the value directly -- **String β†’ Integer** β€” converts `std::string`/`std::wstring` to any integer type (`int`, `long`, `unsigned long long`, etc.) with configurable base (default: 10) -- **String β†’ Bool** β€” parses human-readable boolean strings (`"1"`, `"yes"`, `"y"`, `"true"`, `"0"`, `"no"`, `"n"`, `"false"`, etc.) - -### `impl::throwingStringToInt(from, base)` -Internal dispatch layer that selects the correct STL conversion function (`stoi`, `stol`, `stoll`, `stoul`, `stoull`) based on the target integer type. - -### `operator"" _sz` -User-defined literal converting `unsigned long long` to `uint64_t` for size expressions. +- **`ConversionError` enum** β€” Error codes returned on failed conversions: `InvalidArgument`, `OutOfRange`, `Unknown` +- **`tryTo(from)`** β€” Primary conversion template with three specializations: + - **Same-type passthrough** β€” Returns the value directly via perfect forwarding when source and target types match + - **String β†’ Integer** β€” Converts `std::string`/`std::wstring` to any integer type (`int`, `long`, `long long`, `unsigned` variants) with optional numeric base; wraps `std::stoi`/`stol`/`stoll`/`stoul`/`stoull` + - **String β†’ Bool** β€” Parses common boolean representations (`"1"`, `"0"`, `"y"`, `"yes"`, `"n"`, `"no"`, etc.) via `impl::stringToBool` +- **`impl::IsStlString`** β€” Trait detecting `std::string` or `std::wstring` +- **`impl::IsInteger`** β€” Trait detecting integral types excluding `bool` +- **`impl::throwingStringToInt`** β€” Internal dispatcher selecting the correct `std::stox` overload via SFINAE +- **`operator"" _sz`** β€” User-defined literal converting `unsigned long long` to `uint64_t` ## Usage Example ```cpp -// String to integer +#include + +// String to integer (base 10) auto result = osquery::tryTo("42"); if (result) { - int value = result.get(); // 42 + int value = result.get(); // 42 } -// String to integer with base -auto hex = osquery::tryTo("FF", 16); +// String to integer (hex) +auto hexResult = osquery::tryTo("FF", 16); // String to bool -auto flag = osquery::tryTo("yes"); -if (flag) { - bool val = flag.get(); // true +auto boolResult = osquery::tryTo("yes"); +if (boolResult) { + bool flag = boolResult.get(); // true } -// Error handling +// Handle errors auto bad = osquery::tryTo("not_a_number"); -if (bad.isError()) { - // bad.getError() == ConversionError::InvalidArgument +if (!bad) { + // bad.getError() == ConversionError::InvalidArgument } + +// Same-type passthrough +auto passthrough = osquery::tryTo(std::string("hello")); ``` \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.cfdata.md b/osquery/utils/conversions/darwin/.cfdata.md index 41967c69b79..14e9b5cf183 100644 --- a/osquery/utils/conversions/darwin/.cfdata.md +++ b/osquery/utils/conversions/darwin/.cfdata.md @@ -1,13 +1,9 @@ -Utility header for converting Core Foundation `CFDataRef` objects to C++ `std::string` values on macOS/Darwin platforms. +Utility header for converting CoreFoundation `CFDataRef` objects to C++ `std::string` on macOS/Apple platforms. ## Key Components -### Functions - -| Function | Description | -|---|---| -| `stringFromCFData(const CFDataRef& cf_data)` | Converts a Core Foundation `CFDataRef` to a `std::string` | +- **`stringFromCFData(const CFDataRef& cf_data)`** β€” Converts a CoreFoundation `CFDataRef` binary data reference into a `std::string`, enabling interoperability between Apple's CoreFoundation C APIs and standard C++ string handling within the `osquery` namespace. ## Usage Example @@ -15,13 +11,14 @@ Utility header for converting Core Foundation `CFDataRef` objects to C++ `std::s #include "cfdata.h" #include -// Convert a CFDataRef to std::string -CFDataRef rawData = /* obtained from a CF API */; -std::string result = osquery::stringFromCFData(rawData); -``` +// Example: converting raw CFData to std::string +const char* raw = "hello"; +CFDataRef cfData = CFDataCreate(kCFAllocatorDefault, + (const UInt8*)raw, + strlen(raw)); -## Notes +std::string result = osquery::stringFromCFData(cfData); +CFRelease(cfData); +``` -- macOS/Darwin only β€” depends on the `CoreFoundation` framework -- Lives in the `osquery` namespace -- Typically used when consuming binary or raw byte data from macOS system APIs that return `CFDataRef` handles \ No newline at end of file +> **Note:** This header is macOS-only due to its dependency on ``. It is typically guarded by platform preprocessor checks in cross-platform builds. \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.cfdictionary.md b/osquery/utils/conversions/darwin/.cfdictionary.md index a367642cbe1..68f7fffc127 100644 --- a/osquery/utils/conversions/darwin/.cfdictionary.md +++ b/osquery/utils/conversions/darwin/.cfdictionary.md @@ -1,37 +1,33 @@ -Utility header for extracting typed values from Core Foundation `CFDictionaryRef` objects within the osquery macOS/Darwin data collection framework. +Provides a utility function for extracting string values from Apple's `CFDictionaryRef` type within the osquery framework. ## Key Components -### Functions +**Namespace:** `osquery` -- **`getPropertiesFromDictionary(dict, key)`** β€” Looks up a key in a `CFDictionaryRef` and returns the corresponding value as a `std::string`. Relies on the sibling CF conversion helpers (`cfdata.h`, `cfnumber.h`, `cfstring.h`) to handle type coercion. +| Symbol | Type | Description | +|--------|------|-------------| +| `getPropertiesFromDictionary` | `function` | Retrieves a value from a `CFDictionaryRef` by key and returns it as a `std::string` | -### Dependencies - -| Header | Purpose | -|---|---| -| `cfdata.h` | CF `Data` β†’ `std::string` conversion | -| `cfnumber.h` | CF `Number` β†’ `std::string` conversion | -| `cfstring.h` | CF `String` β†’ `std::string` conversion | -| `CoreFoundation/CoreFoundation.h` | Native CF types (`CFDictionaryRef`, etc.) | +**Dependencies:** +- `cfdata.h`, `cfnumber.h`, `cfstring.h` β€” sibling CF type utilities +- `CoreFoundation/CoreFoundation.h` β€” macOS/Apple platform framework ## Usage Example -```cpp +```c #include "cfdictionary.h" #include -// Assume `plistDict` is a CFDictionaryRef obtained from a parsed plist +// Assume `plistDict` is a CFDictionaryRef from a parsed plist or system API CFDictionaryRef plistDict = /* ... */; -std::string version = osquery::getPropertiesFromDictionary(plistDict, "CFBundleVersion"); -std::string bundleId = osquery::getPropertiesFromDictionary(plistDict, "CFBundleIdentifier"); +std::string value = osquery::getPropertiesFromDictionary(plistDict, "DeviceName"); -// Returns an empty string if the key is missing or conversion fails -if (!version.empty()) { - // use version string +if (!value.empty()) { + // Use the extracted string value + LOG(INFO) << "Device name: " << value; } ``` -> **Platform note:** This header is macOS/Darwin-only. It depends on `CoreFoundation` and is compiled exclusively for Apple platform targets within osquery. \ No newline at end of file +> **Note:** This header is macOS-only due to its dependency on `CoreFoundation`. It is typically used in Darwin-specific osquery table implementations to parse system property dictionaries (e.g., IOKit, plist data, network configuration). \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.cfnumber.md b/osquery/utils/conversions/darwin/.cfnumber.md index b80a7db02a0..a930ca07d80 100644 --- a/osquery/utils/conversions/darwin/.cfnumber.md +++ b/osquery/utils/conversions/darwin/.cfnumber.md @@ -1,32 +1,29 @@ -Utility header for converting Apple CoreFoundation `CFNumberRef` values to C++ `std::string` objects within the osquery framework. +Provides utility functions for converting Core Foundation `CFNumberRef` values to `std::string` on macOS. ## Key Components ### Functions -| Function | Description | -|---|---| -| `stringFromCFNumber(const CFDataRef&)` | Converts a `CFNumberRef` to a `std::string`, auto-detecting the number type | -| `stringFromCFNumber(const CFDataRef&, CFNumberType)` | Converts a `CFNumberRef` to a `std::string` using an explicitly specified `CFNumberType` | +- **`stringFromCFNumber(const CFDataRef& cf_number)`** β€” Converts a `CFDataRef` (representing a `CFNumber`) to a `std::string`, automatically inferring the numeric type. +- **`stringFromCFNumber(const CFDataRef& cf_number, CFNumberType type)`** β€” Converts a `CFDataRef` to a `std::string` using an explicitly specified `CFNumberType` (e.g., `kCFNumberIntType`, `kCFNumberDoubleType`). ## Usage Example -```cpp -#include -#include +```c +#include "cfnumber.h" -// Auto-detect number type -CFNumberRef cf_num = /* obtained from a CF API */; -std::string value = osquery::stringFromCFNumber( - reinterpret_cast(cf_num) -); +// Auto-infer numeric type +CFNumberRef num = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &myInt); +std::string result = osquery::stringFromCFNumber((CFDataRef)num); -// Explicit type conversion -std::string int_value = osquery::stringFromCFNumber( - reinterpret_cast(cf_num), +// Explicit numeric type +std::string result2 = osquery::stringFromCFNumber( + (CFDataRef)num, kCFNumberIntType ); + +CFRelease(num); ``` -> **Note:** This header is macOS-only, as it depends on the CoreFoundation framework. It is intended for use in osquery Darwin-specific table implementations that parse CF property list or registry data structures. \ No newline at end of file +> **Note:** This header is macOS-only and requires linking against the `CoreFoundation` framework. It is part of osquery's platform abstraction layer for parsing Apple property lists and system data structures. \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.cfstring.md b/osquery/utils/conversions/darwin/.cfstring.md index 2cc21378fec..4d86b89140b 100644 --- a/osquery/utils/conversions/darwin/.cfstring.md +++ b/osquery/utils/conversions/darwin/.cfstring.md @@ -1,9 +1,9 @@ -Utility header for converting Apple CoreFoundation `CFStringRef` objects to standard C++ `std::string` values within the osquery namespace. +Utility header for converting macOS CoreFoundation `CFStringRef` objects to standard C++ `std::string` types within the osquery namespace. ## Key Components -- **`stringFromCFString(const CFStringRef& cf_string)`** β€” Converts a CoreFoundation `CFStringRef` to a `std::string`. Useful when working with macOS system APIs that return CF types. +- **`stringFromCFString(const CFStringRef& cf_string)`** β€” Converts a CoreFoundation `CFStringRef` to a `std::string`. Declared within the `osquery` namespace. ## Usage Example @@ -11,15 +11,13 @@ Utility header for converting Apple CoreFoundation `CFStringRef` objects to stan #include "cfstring.h" #include -// Convert a CFStringRef returned by a macOS API to std::string -CFStringRef cf_name = CFStringCreateWithCString( - kCFAllocatorDefault, "example", kCFStringEncodingUTF8); - -std::string name = osquery::stringFromCFString(cf_name); -CFRelease(cf_name); +// Convert a CFStringRef to std::string +CFStringRef cf_str = CFSTR("Hello from CoreFoundation"); +std::string result = osquery::stringFromCFString(cf_str); +// result == "Hello from CoreFoundation" ``` ## Notes -- macOS only β€” depends on `CoreFoundation`, which is not available on Linux or Windows. -- Caller is responsible for memory management of the `CFStringRef` (follow standard CF ownership rules). \ No newline at end of file +- macOS-only; depends on `CoreFoundation` framework. +- Header-guarded via `#pragma once`. \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.cftime.md b/osquery/utils/conversions/darwin/.cftime.md index bf5ea0812dd..c0342dbf256 100644 --- a/osquery/utils/conversions/darwin/.cftime.md +++ b/osquery/utils/conversions/darwin/.cftime.md @@ -1,5 +1,5 @@ -Utility header for converting CoreFoundation `CFAbsoluteTime` values to standard C++ strings on macOS/Apple platforms. +Utility header for converting CoreFoundation `CFAbsoluteTime` values to standard C++ strings within the osquery macOS platform layer. ## Key Components @@ -9,17 +9,16 @@ Utility header for converting CoreFoundation `CFAbsoluteTime` values to standard ```c #include "cftime.h" -#include -// Assume cf_data is a CFDataRef containing a CFAbsoluteTime -CFDataRef cf_data = /* ... obtain from system API ... */; +// Given a CFDataRef containing a CFAbsoluteTime value: +CFDataRef cf_time_data = /* ... obtained from CoreFoundation API ... */; -std::string timeStr = osquery::stringFromCFAbsoluteTime(cf_data); -// timeStr now contains a formatted time string +std::string time_str = osquery::stringFromCFAbsoluteTime(cf_time_data); +// time_str now holds a human-readable timestamp string ``` ## Notes -- macOS/Apple only β€” depends on `CoreFoundation` framework. -- Lives within the `osquery` namespace. -- Typical use case: parsing macOS plist or system data where timestamps are stored as `CFAbsoluteTime` (seconds since Jan 1, 2001). \ No newline at end of file +- macOS/Darwin only β€” depends on `CoreFoundation/CoreFoundation.h` +- `CFAbsoluteTime` measures seconds since January 1, 2001 00:00:00 UTC; this utility handles that epoch conversion automatically +- Lives in the `osquery` namespace \ No newline at end of file diff --git a/osquery/utils/conversions/darwin/.iokit.md b/osquery/utils/conversions/darwin/.iokit.md index 2dfd6cc9bab..90644d47446 100644 --- a/osquery/utils/conversions/darwin/.iokit.md +++ b/osquery/utils/conversions/darwin/.iokit.md @@ -1,51 +1,45 @@ -Provides IOKit utility functions and constants for querying macOS hardware device information via Apple's IOKit framework. +Provides IOKit utility declarations for querying macOS hardware device properties via Apple's IOKit framework, used within osquery's device enumeration subsystem. ## Key Components -### Device Class Name Constants +**Device Class Name Constants** -String constants identifying IOKit device class types: - -| Constant | Device Type | +| Constant | Description | |---|---| -| `kIOUSBDeviceClassName_` | USB devices | -| `kIOPCIDeviceClassName_` | PCI devices | -| `kIOPlatformExpertDeviceClassName_` | Platform expert devices | -| `kIOACPIPlatformDeviceClassName_` | ACPI platform devices | -| `kIOPlatformDeviceClassName_` | Generic platform devices | -| `kAppleARMIODeviceClassName_` | Apple ARM I/O devices | - -### `IOKitPCIProperties` Struct +| `kIOUSBDeviceClassName_` | USB device class identifier | +| `kIOPCIDeviceClassName_` | PCI device class identifier | +| `kIOPlatformExpertDeviceClassName_` | Platform expert device class | +| `kIOACPIPlatformDeviceClassName_` | ACPI platform device class | +| `kIOPlatformDeviceClassName_` | Generic platform device class | +| `kAppleARMIODeviceClassName_` | Apple ARM I/O device class | -Holds parsed PCI device properties extracted from the IOKit `"compatible"` property string: +**`IOKitPCIProperties` struct** -- `vendor_id` β€” PCI vendor identifier -- `model_id` β€” PCI model/device identifier -- `pci_class` β€” PCI device class -- `driver` β€” Associated driver name +Parses and holds PCI device metadata extracted from the IOKit `"compatible"` property string. Fields include `vendor_id`, `model_id`, `pci_class`, and `driver`. -Constructed via `IOKitPCIProperties(const std::string& compatible)` which parses the raw compatible string automatically. +**Helper Functions** -### Functions - -- **`getIOKitProperty`** β€” Retrieves a string property value from a `CFMutableDictionaryRef` by key -- **`getNumIOKitProperty`** β€” Retrieves a numeric (`long long int`) property value from a `CFMutableDictionaryRef` by key -- **`idToHex`** β€” Converts a device ID string to its hexadecimal representation in-place +- `getIOKitProperty()` β€” Retrieves a string property value from a `CFMutableDictionaryRef` by key. +- `getNumIOKitProperty()` β€” Retrieves a numeric (`long long int`) property value from a `CFMutableDictionaryRef` by key. +- `idToHex()` β€” Converts a device ID string to its hexadecimal representation in-place. ## Usage Example ```c -CFMutableDictionaryRef details = getDeviceDetails(); // IOKit registry entry +CFMutableDictionaryRef details = /* IOKit service properties */; + +// Read a string property +std::string model = osquery::getIOKitProperty(details, "model"); -// Read string and numeric properties -std::string vendor = osquery::getIOKitProperty(details, "vendor-id"); -long long int deviceClass = osquery::getNumIOKitProperty(details, "class-code"); +// Read a numeric property +long long int revision = osquery::getNumIOKitProperty(details, "revision-id"); // Parse PCI compatible string -osquery::IOKitPCIProperties pci("pci8086,1502"); -// pci.vendor_id == "8086", pci.model_id == "1502" +osquery::IOKitPCIProperties pci("pci8086,1234 pci8086 pciclass,020000"); +// pci.vendor_id == "8086", pci.model_id == "1234", pci.pci_class == "020000" -// Convert ID to hex representation -osquery::idToHex(vendor); +// Convert vendor ID to hex +std::string vid = "32902"; +osquery::idToHex(vid); // vid == "0x8086" ``` \ No newline at end of file diff --git a/osquery/utils/conversions/windows/.strings.md b/osquery/utils/conversions/windows/.strings.md index a1181fe9b65..fd2d1186c30 100644 --- a/osquery/utils/conversions/windows/.strings.md +++ b/osquery/utils/conversions/windows/.strings.md @@ -1,39 +1,39 @@ -Windows-specific string utility header providing conversion functions between narrow/wide strings and Windows-native types within the osquery framework. +Windows-specific string utility header providing conversion functions between narrow/wide strings, Windows-native types, and common data formats used throughout the osquery Windows platform layer. ## Key Components | Function | Description | -|---|---| -| `stringToWstring` | Converts `std::string` β†’ `std::wstring` | +|----------|-------------| +| `stringToWstring()` | Converts `std::string` β†’ `std::wstring` | | `wstringToString(const std::wstring&)` | Converts `std::wstring` β†’ `std::string` | | `wstringToString(const wchar_t*)` | Converts wide C-string β†’ `std::string` | -| `cimDatetimeToUnixtime` | Converts WMI CIM datetime string β†’ Unix timestamp (`LONGLONG`) | -| `bstrToString` | Converts Windows `BSTR` β†’ `std::string` | -| `swapEndianess` | Swaps byte order of a string (little ↔ big endian) | -| `errorDwordToString` | Converts a Windows `DWORD` error code β†’ human-readable `std::string` | - -All functions reside in the `osquery` namespace and are Windows-only (depends on `comutil.h`). +| `cimDatetimeToUnixtime()` | Converts WMI CIM datetime string β†’ Unix timestamp (`LONGLONG`) | +| `bstrToString()` | Converts Windows `BSTR` type β†’ `std::string` | +| `swapEndianess()` | Swaps byte order of a string (little ↔ big endian) | +| `errorDwordToString()` | Converts a Windows `DWORD` error code β†’ human-readable `std::string` | ## Usage Example -```c -#include "strings.h" +```cpp +#include // Wide/narrow string conversion std::wstring wide = osquery::stringToWstring("Hello, Windows"); std::string narrow = osquery::wstringToString(wide); -// WMI CIM datetime to Unix timestamp -LONGLONG ts = osquery::cimDatetimeToUnixtime("20240101120000.000000+000"); +// WMI query result handling +std::string cimTime = "20230101120000.000000+000"; +LONGLONG unixTs = osquery::cimDatetimeToUnixtime(cimTime); -// Convert BSTR from WMI query result -BSTR bstr = SysAllocString(L"WMI Result"); -std::string result = osquery::bstrToString(bstr); +// Windows BSTR from WMI COM result +BSTR bstrVal = SysAllocString(L"ExampleValue"); +std::string result = osquery::bstrToString(bstrVal); +SysFreeString(bstrVal); -// Decode a Windows error code -std::string errMsg = osquery::errorDwordToString(GetLastError()); +// Windows error code to string +DWORD err = GetLastError(); +std::string errMsg = osquery::errorDwordToString(err); +``` -// Swap endianness (e.g., for GUID/UUID processing) -std::string swapped = osquery::swapEndianess(rawBytes); -``` \ No newline at end of file +> **Note:** This header is Windows-only and depends on ``. It is not available on Linux or macOS builds. \ No newline at end of file diff --git a/osquery/utils/conversions/windows/.windows_time.md b/osquery/utils/conversions/windows/.windows_time.md index dcc19b84842..9d7827eb904 100644 --- a/osquery/utils/conversions/windows/.windows_time.md +++ b/osquery/utils/conversions/windows/.windows_time.md @@ -1,16 +1,16 @@ -Windows-specific utility header providing timestamp conversion functions that normalize various Windows time formats into Unix epoch values for consistent cross-platform time handling within osquery. +Windows-specific utility header providing time format conversion functions that translate various Windows timestamp representations into Unix epoch values. ## Key Components | Function | Input | Description | -|---|---|---| -| `filetimeToUnixtime` | `const FILETIME&` | Converts Windows `FILETIME` structure to Unix epoch | -| `longIntToUnixtime` | `LARGE_INTEGER&` | Converts Windows `LARGE_INTEGER` value to Unix epoch | -| `littleEndianToUnixTime` | `const std::string&` | Converts little-endian encoded `FILETIME` (as used in the Windows Registry) to Unix epoch | -| `parseFatTime` | `const std::string&` | Parses and converts FAT/DOS timestamp format to Unix epoch | +|----------|-------|-------------| +| `filetimeToUnixtime` | `const FILETIME&` | Converts a Windows `FILETIME` struct to Unix epoch | +| `longIntToUnixtime` | `LARGE_INTEGER&` | Converts a Windows `LARGE_INTEGER` timestamp to Unix epoch | +| `littleEndianToUnixTime` | `const std::string&` | Converts a little-endian encoded `FILETIME` (as used in the Windows Registry) to Unix epoch | +| `parseFatTime` | `const std::string&` | Parses and converts FAT/DOS filesystem timestamps to Unix epoch | -All functions return `LONGLONG` representing seconds since the Unix epoch. +All functions return `LONGLONG` representing the Unix epoch value. ## Usage Example @@ -19,20 +19,25 @@ All functions return `LONGLONG` representing seconds since the Unix epoch. namespace osquery { -// Convert a FILETIME (e.g., from file metadata) to Unix epoch +// Convert a standard FILETIME (e.g., from Win32 API) FILETIME ft; GetSystemTimeAsFileTime(&ft); LONGLONG unixTime = filetimeToUnixtime(ft); -// Convert a registry-stored little-endian FILETIME string -std::string regTimeData = getRegistryTimeValue(); -LONGLONG regUnixTime = littleEndianToUnixTime(regTimeData); +// Convert a LARGE_INTEGER timestamp (e.g., from NtQuerySystemInformation) +LARGE_INTEGER li; +li.QuadPart = someTimestampValue; +LONGLONG unixTime2 = longIntToUnixtime(li); -// Convert a FAT/DOS timestamp from a filesystem entry -std::string dosTime = getFatTimestamp(); -LONGLONG fatUnixTime = parseFatTime(dosTime); +// Convert a little-endian FILETIME string from the Windows Registry +std::string regTimeData = readRegistryBinaryValue(); +LONGLONG unixTime3 = littleEndianToUnixTime(regTimeData); + +// Convert a FAT/DOS timestamp (e.g., from ZIP or FAT filesystem metadata) +std::string dosTimeData = readDosTimestamp(); +LONGLONG fatTime = parseFatTime(dosTimeData); } // namespace osquery ``` -> **Note:** This header is Windows-only and guarded by `#pragma once`. It depends on `osquery/utils/system/system.h` for Windows type availability (`LONGLONG`, `FILETIME`, `LARGE_INTEGER`). Use these helpers whenever normalizing Windows-native timestamps to ensure consistent Unix epoch output across osquery tables. \ No newline at end of file +> **Note:** This header is Windows-only and depends on `osquery/utils/system/system.h` for Windows type definitions. It will not compile on non-Windows platforms. \ No newline at end of file diff --git a/osquery/utils/info/.platform_type.md b/osquery/utils/info/.platform_type.md index 1c36552b97b..1359430dd1f 100644 --- a/osquery/utils/info/.platform_type.md +++ b/osquery/utils/info/.platform_type.md @@ -1,18 +1,34 @@ -Defines platform detection types and compile-time constants for identifying the target OS at both build time and runtime within the osquery framework. +Defines platform detection utilities for osquery, providing both compile-time and runtime mechanisms to identify the operating system and platform configuration. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `PlatformType` | `enum class` | Bitmask enum for platform identification (`TYPE_POSIX`, `TYPE_WINDOWS`, `TYPE_BSD`, `TYPE_LINUX`, `TYPE_OSX`, `TYPE_FREEBSD`) | -| `kPlatformType` | `constexpr` | Compile-time bitmask constant built from preprocessor defines (`POSIX`, `WINDOWS`, `BSD`, `LINUX`, `DARWIN`, `FREEBSD`) | -| `isPlatform()` | Function | Runtime check whether a given `PlatformType` flag is active | -| `operator\|` | Operator | Bitwise OR for combining `PlatformType` values | -| `kSDKPlatform` | `extern const std::string` | String identifier of the build platform, mirrors `OSQUERY_BUILD_PLATFORM` | -| `OSQUERY_PLATFORM` | `#define` | Macro alias for `OSQUERY_BUILD_PLATFORM` | +### `PlatformType` Enum +Bitmask enumeration of supported platform types: -> **Note:** `OSQUERY_BUILD_PLATFORM` and `OSQUERY_BUILD_DISTRO` **must** be defined by the build system β€” the header enforces this with `#error` directives. +| Value | Mask | +|---|---| +| `TYPE_POSIX` | `0x01` | +| `TYPE_WINDOWS` | `0x02` | +| `TYPE_BSD` | `0x04` | +| `TYPE_LINUX` | `0x08` | +| `TYPE_OSX` | `0x10` | +| `TYPE_FREEBSD` | `0x20` | + +### `kPlatformType` (constexpr) +Compile-time constant built by OR-ing `PlatformType` flags based on preprocessor defines (`POSIX`, `WINDOWS`, `BSD`, `LINUX`, `DARWIN`, `FREEBSD`) set by CMake. + +### `isPlatform()` +Runtime check that tests whether a given `PlatformType` flag is active in the current build mask. + +### `operator|` +Bitwise OR operator enabling combination of `PlatformType` values. + +### `kSDKPlatform` +Extern string exposing `OSQUERY_BUILD_PLATFORM` at runtime (e.g., `"linux"`, `"darwin"`, `"windows"`). + +### Required Build Defines +The build system **must** define both `OSQUERY_BUILD_PLATFORM` and `OSQUERY_BUILD_DISTRO` β€” a `#error` is raised at compile time if either is missing. ## Usage Example @@ -21,17 +37,19 @@ Defines platform detection types and compile-time constants for identifying the // Runtime platform check if (osquery::isPlatform(osquery::PlatformType::TYPE_LINUX)) { - // Linux-specific logic + // Linux-specific logic } -// Combining platform flags -osquery::PlatformType combined = - osquery::PlatformType::TYPE_POSIX | osquery::PlatformType::TYPE_LINUX; +// Combine platform flags for multi-platform checks +auto bsd_or_posix = osquery::PlatformType::TYPE_BSD + | osquery::PlatformType::TYPE_POSIX; -if (osquery::isPlatform(osquery::PlatformType::TYPE_POSIX, combined)) { - // Runs if combined mask includes POSIX +if (osquery::isPlatform(bsd_or_posix)) { + // Runs on BSD or POSIX platforms } -// Accessing build-time platform string -std::cout << osquery::kSDKPlatform; // e.g. "linux", "darwin", "windows" -``` \ No newline at end of file +// Access platform string (exposed in osquery_info table) +std::string platform = osquery::kSDKPlatform; // e.g. "darwin" +``` + +> **Tip:** Prefer `isPlatform()` runtime checks over `#ifdef` preprocessor guards where possible, as recommended by the osquery style guidelines. \ No newline at end of file diff --git a/osquery/utils/info/.tool_type.md b/osquery/utils/info/.tool_type.md index 3e08fd43d30..04eb375911e 100644 --- a/osquery/utils/info/.tool_type.md +++ b/osquery/utils/info/.tool_type.md @@ -1,45 +1,44 @@ -Defines the `ToolType` enumeration and related utility functions for identifying the running osquery process type at runtime, enabling conditional behavior across daemons, shells, extensions, and tests. +Defines the `ToolType` enumeration and related utility functions for identifying the running osquery process type at runtime, enabling conditional behavior across logging, help output, and debugging. ## Key Components ### Enum: `ToolType` - | Value | Description | -|---|---| +|-------|-------------| | `UNKNOWN` | Default/undetected tool type | | `SHELL` | Interactive osquery shell | | `DAEMON` | Background osquery daemon | -| `TEST` | Test harness process | +| `TEST` | Test runner context | | `EXTENSION` | osquery extension process | | `SHELL_DAEMON` | Combined shell/daemon mode | ### Functions -- **`setToolType(ToolType tool)`** β€” Sets the global tool type, typically called during initialization by the `Initializer` class. -- **`getToolType()`** β€” Returns the currently active `ToolType` for runtime branching. -- **`isDaemon()`** β€” Convenience check; returns `true` if the current tool type is `DAEMON`. -- **`isShell()`** β€” Convenience check; returns `true` if the current tool type is `SHELL`. +- **`setToolType(ToolType tool)`** β€” Sets the active tool type at process initialization +- **`getToolType()`** β€” Returns the currently configured `ToolType` +- **`isDaemon()`** β€” Convenience check; returns `true` if running as daemon +- **`isShell()`** β€” Convenience check; returns `true` if running as shell ## Usage Example ```c #include -// Set tool type during startup +// Set during initialization (typically done by the Initializer class) osquery::setToolType(osquery::ToolType::DAEMON); -// Conditional behavior based on tool type +// Conditional logic based on tool type if (osquery::isDaemon()) { - // daemon-specific initialization + // Daemon-specific behavior (e.g., background scheduling) } else if (osquery::isShell()) { - // interactive shell setup + // Shell-specific behavior (e.g., interactive output formatting) } // Direct enum comparison if (osquery::getToolType() == osquery::ToolType::EXTENSION) { - // extension-specific logic + // Extension-specific initialization } ``` -> **Note:** The `ToolType` is typically auto-detected by the `Initializer` class using the process name and compile-time flags β€” manual calls to `setToolType` are generally reserved for tests or special bootstrapping scenarios. \ No newline at end of file +> **Note:** The `ToolType` is normally set automatically by the `Initializer` class using the binary name and compile-time options. Manual calls to `setToolType()` are typically only needed in test contexts or specialized entry points. \ No newline at end of file diff --git a/osquery/utils/info/.version.md b/osquery/utils/info/.version.md index f1e8e244994..faba795e62c 100644 --- a/osquery/utils/info/.version.md +++ b/osquery/utils/info/.version.md @@ -1,40 +1,41 @@ -Defines version constants and comparison utilities for the osquery runtime and SDK, enforcing that required version macros are set at build time. +Defines version constants and comparison utilities for the osquery runtime, enforcing that required build-time version macros are present at compile time. ## Key Components | Symbol | Type | Description | -|---|---|---| -| `kVersion` | `extern const std::string` | The current osquery runtime version string | -| `kSDKVersion` | `extern const std::string` | The osquery SDK version string | -| `OSQUERY_SDK_VERSION` | Macro | Stringified form of `OSQUERY_BUILD_SDK_VERSION` | -| `versionAtLeast()` | Function | Compares a version string against the SDK/core version | +|--------|------|-------------| +| `kVersion` | `const std::string` | The runtime osquery version string | +| `kSDKVersion` | `const std::string` | The runtime osquery SDK version string | +| `OSQUERY_SDK_VERSION` | Macro | Stringified compile-time SDK version | +| `versionAtLeast()` | Function | Compares a version string against the SDK or a provided version | -### Build-time Requirements - -| Macro | Behavior if missing | -|---|---| -| `OSQUERY_VERSION` | **Compile error** β€” must be defined | -| `OSQUERY_BUILD_SDK_VERSION` | **Compile error** β€” must be defined | -| `OSQUERY_BUILD_VERSION` | **Warning** β€” falls back to `1.0.0-unknown` | +**Required build defines:** +- `OSQUERY_VERSION` β€” must be set; compile error if missing +- `OSQUERY_BUILD_SDK_VERSION` β€” must be set; compile error if missing +- `OSQUERY_BUILD_VERSION` β€” recommended; defaults to `1.0.0-unknown` with a warning ## Usage Example ```c #include +#include -// Check if a loaded extension meets the minimum version requirement -void checkExtensionCompatibility(const std::string& extensionVersion) { - if (!osquery::versionAtLeast(extensionVersion)) { - LOG(WARNING) << "Extension version " << extensionVersion - << " is older than core version " << osquery::kVersion; - } -} +namespace osquery { -// Check against a specific version instead of kVersion -bool isAtLeast_5_0(const std::string& v) { - return osquery::versionAtLeast(v, "5.0.0"); +void checkCompatibility(const std::string& extensionVersion) { + // Check if the extension meets the minimum SDK version + if (!versionAtLeast(extensionVersion, kSDKVersion)) { + std::cerr << "Extension version " << extensionVersion + << " is below required SDK version " << kSDKVersion + << std::endl; + return; + } + + std::cout << "Running osquery v" << kVersion << std::endl; } + +} // namespace osquery ``` -> Version strings follow `major.minor.patch-commit-hash` format. `versionAtLeast(v)` returns `true` if `v` is equal to or newer than the current osquery version. \ No newline at end of file +> **Note:** Version strings follow `major.minor.patch-commit-hash` format. `versionAtLeast(v)` returns `true` if `v` is greater than or equal to `kVersion` (or the optionally supplied version string). \ No newline at end of file diff --git a/osquery/utils/json/.json.md b/osquery/utils/json/.json.md index 5b88e19c6f2..327d92efb45 100644 --- a/osquery/utils/json/.json.md +++ b/osquery/utils/json/.json.md @@ -1,50 +1,52 @@ -A wrapper class around RapidJSON that provides safe, memory-leak-resistant construction and manipulation of JSON objects and arrays within the osquery framework. +A thin RAII/safety wrapper around RapidJSON that simplifies construction, mutation, serialization, and parsing of JSON documents within the osquery framework, preventing common allocator misuse and memory leaks. ## Key Components -### `JSON` class -A move-only (`only_movable`) wrapper around `rapidjson::Document` with the following groups of methods: +**`JSON` class** β€” move-only wrapper (`only_movable`) around a `rapidjson::Document` with a shared allocator. | Method Group | Description | |---|---| | `newObject()` / `newArray()` / `newFromValue()` | Static factory constructors | -| `add(key, value, ...)` | Overloaded adders for `string`, `char*`, `int`, `long`, `long long`, `unsigned` variants, `double`, `bool`, `rapidjson::Value` | -| `addCopy()` / `addRef()` | String-specific adders: copy vs. reference semantics | -| `push(value, ...)` | Append values to JSON arrays | -| `pushCopy()` | Append string copies to arrays | -| `copyFrom()` | Deep-copy a JSON value into the document | -| `mergeObject()` / `mergeArray()` | Merge two JSON objects or arrays | -| `toString()` / `toPrettyString()` | Serialize document to string | -| `fromString()` | Parse JSON string (iterative or recursive mode) | -| `getObject()` / `getArray()` | Create new empty sub-documents | -| `doc()` | Direct access to the underlying `rapidjson::Document` | -| `valueToSize()` | Static helper to extract `uint64_t` from a value | - -### `ParseMode` enum -Controls RapidJSON parsing strategy: -- `Iterative` β€” stack-safe, avoids overflow on deep nesting -- `Recursive` β€” default RapidJSON behavior +| `getObject()` / `getArray()` | Create child documents | +| `add(key, value[, obj])` | Add typed values (`string`, `char*`, `int`, `long`, `long long`, `unsigned` variants, `double`, `bool`, `rapidjson::Value`) to an object | +| `addCopy()` / `addRef()` | Explicit copy-vs-reference string insertion | +| `push(value[, arr])` | Append values to a JSON array | +| `pushCopy()` | Append string copies to an array | +| `copyFrom()` | Deep-copy a value into the document or a target node | +| `toString()` / `toPrettyString()` | Serialize to compact or indented JSON string | +| `fromString()` | Parse a JSON string; supports `Iterative` or `Recursive` parse modes | +| `mergeObject()` / `mergeArray()` | Merge members from one node into another | +| `doc()` | Access the underlying `rapidjson::Document` | +| `valueToSize()` | Static helper β€” extract a `uint64_t` size from a value | + +> **Note:** `RAPIDJSON_PARSE_DEFAULT_FLAGS` is set to `kParseIterativeFlag` globally to prevent stack overflows during recursive parsing of deeply nested documents. ## Usage Example ```c -// Create a JSON object and serialize it -osquery::JSON json = osquery::JSON::newObject(); -json.add("hostname", "server01"); +#include + +// Build a JSON object +auto json = osquery::JSON::newObject(); +json.add("hostname", "web-01"); json.add("port", 8080); json.add("active", true); +// Serialize std::string output; -auto status = json.toString(output); +json.toString(output); +// output β†’ {"hostname":"web-01","port":8080,"active":true} + +// Parse from string +auto parsed = osquery::JSON::newObject(); +auto status = parsed.fromString(R"({"key":"value"})"); if (status.ok()) { - // output == {"hostname":"server01","port":8080,"active":true} + // use parsed.doc()["key"].GetString() } -// Parse JSON from a string -osquery::JSON parsed; -auto s = parsed.fromString( - "{\"key\":\"value\"}", - osquery::JSON::ParseMode::Iterative -); +// Nested array +auto arr = osquery::JSON::newArray(); +arr.push(std::size_t{42}); +arr.pushCopy(std::string{"item"}); ``` \ No newline at end of file diff --git a/osquery/utils/linux/dpkg/.dpkgquery.md b/osquery/utils/linux/dpkg/.dpkgquery.md index 43012c91fdf..61b49e6cba8 100644 --- a/osquery/utils/linux/dpkg/.dpkgquery.md +++ b/osquery/utils/linux/dpkg/.dpkgquery.md @@ -1,28 +1,19 @@ -Concrete implementation of the `IDpkgQuery` interface that queries the dpkg package database on Linux systems to retrieve installed package information. +Concrete implementation of the `IDpkgQuery` interface that queries installed Debian packages via `libdpkg`. ## Key Components -### Class: `DpkgQuery` - -A `final` class inheriting from `IDpkgQuery`, providing the concrete dpkg querying logic. +**`DpkgQuery`** β€” Final class implementing `IDpkgQuery`: | Member | Type | Description | -|---|---|---| -| `getPackageList()` | Public | Returns an `Expected` with all installed dpkg packages | -| `~DpkgQuery()` | Public | Destructor; cleans up dpkg library state | -| `DpkgQuery(admindir)` | Private | Constructor accepting a custom dpkg admin directory path | -| `validateAdminDir(path)` | Private | Validates the provided admin directory exists and is accessible | -| `packageIteratorCallback(...)` | Private Static | Callback invoked per-package during dpkg array iteration; populates `PackageList` | - -### Notable Macros - -- `LIBDPKG_VOLATILE_API` β€” Required define to acknowledge the unstable dpkg C API before including dpkg headers +|--------|------|-------------| +| `getPackageList()` | `Expected` | Queries libdpkg and returns all installed packages | +| `~DpkgQuery()` | Destructor | Cleans up libdpkg resources | +| `DpkgQuery(dpkg_admindir)` | Private constructor | Initializes with a custom dpkg admin directory (e.g. `/var/lib/dpkg`) | +| `validateAdminDir(path)` | Private method | Validates the admin directory exists and is accessible | +| `packageIteratorCallback(...)` | Static callback | Iterates over a `pkg_array`, populating the `PackageList` result | -### Dependencies - -- `` β€” dpkg C library for package array iteration -- `osquery/utils/linux/idpkgquery.h` β€” Abstract interface defining `IDpkgQuery`, `PackageList`, and `ErrorCode` +**`LIBDPKG_VOLATILE_API`** β€” Required preprocessor define to acknowledge the unstable libdpkg API before including its headers. ## Usage Example @@ -31,13 +22,13 @@ A `final` class inheriting from `IDpkgQuery`, providing the concrete dpkg queryi auto query = IDpkgQuery::create("/var/lib/dpkg"); auto result = query->getPackageList(); -if (result.isValue()) { +if (result.isError()) { + LOG(ERROR) << "Failed to query dpkg: " << result.getError(); +} else { for (const auto& pkg : result.get()) { - // Process each installed package entry + LOG(INFO) << pkg.name << " " << pkg.version; } -} else { - // Handle ErrorCode from result.getError() } ``` -> **Note:** `DpkgQuery` cannot be instantiated directly β€” the constructor is private and only accessible via the `IDpkgQuery` factory pattern (declared as `friend class IDpkgQuery`). \ No newline at end of file +> **Note:** `DpkgQuery` cannot be instantiated directly β€” construction is delegated to `IDpkgQuery` via the `friend` relationship, enforcing use of the factory pattern. \ No newline at end of file diff --git a/osquery/utils/linux/dpkg/.idpkgquery.md b/osquery/utils/linux/dpkg/.idpkgquery.md index bf8f83483dc..0aff8bf753b 100644 --- a/osquery/utils/linux/dpkg/.idpkgquery.md +++ b/osquery/utils/linux/dpkg/.idpkgquery.md @@ -1,37 +1,34 @@ -Interface definition for querying Debian package (dpkg) information, providing an abstraction layer for accessing installed package metadata from a dpkg admin directory. +Interface definition for querying dpkg (Debian package manager) package information within the osquery framework. ## Key Components ### `IDpkgQuery` (Abstract Class) -The primary interface for dpkg package queries within the `osquery` namespace. +The primary interface for dpkg package queries. Non-copyable, uses unique_ptr ownership semantics. **`ErrorCode` Enum** -Represents failure states returned via `Expected<>`: - -| Code | Description | -|---|---| +| Value | Description | +|-------|-------------| | `MemoryAllocationFailure` | Failed to allocate required memory | -| `InvalidAdminDirPath` | Provided path is not valid | +| `InvalidAdminDirPath` | Provided path is invalid | | `NotADpkgAdminDir` | Path exists but is not a dpkg admin directory | -| `NoPackagesFound` | Admin dir is valid but contains no packages | -| `PermissionError` | Insufficient permissions to read the directory | +| `NoPackagesFound` | Directory contains no package data | +| `PermissionError` | Insufficient permissions to read directory | **`Package` Struct** Holds metadata for a single installed package: -| Field | Type | -|---|---| -| `name`, `version`, `source` | `std::string` | -| `size`, `arch`, `revision` | `std::string` | -| `status`, `status_string` | `std::string` | -| `maintainer`, `section`, `priority` | `std::string` | +```text +name Β· version Β· source Β· size Β· arch +revision Β· status Β· status_string Β· maintainer Β· section Β· priority +``` -**Key Methods** +**Static Methods** +- `create(dpkg_admindir)` β€” Factory method; returns `Expected` +- `getErrorCodeDescription(error_code)` β€” Returns human-readable error string -- `create(dpkg_admindir)` β€” Static factory returning `Expected` -- `getPackageList()` β€” Pure virtual; returns all installed packages -- `getErrorCodeDescription(error_code)` β€” Human-readable error message +**Virtual Methods** +- `getPackageList()` β€” Returns `Expected` of all installed packages ## Usage Example @@ -41,21 +38,20 @@ Holds metadata for a single installed package: auto result = osquery::IDpkgQuery::create("/var/lib/dpkg"); if (result.isError()) { - auto msg = osquery::IDpkgQuery::getErrorCodeDescription( - result.getError() + auto desc = osquery::IDpkgQuery::getErrorCodeDescription( + result.getErrorCode() ); - LOG(ERROR) << "Failed to create dpkg query: " << msg; + LOG(ERROR) << "Failed to initialize dpkg query: " << desc; return; } auto query = std::move(result.get()); auto packages = query->getPackageList(); -if (!packages.isError()) { +if (packages.isValue()) { for (const auto& pkg : packages.get()) { - LOG(INFO) << pkg.name << " @ " << pkg.version; + LOG(INFO) << pkg.name << " @ " << pkg.version + << " [" << pkg.arch << "]"; } } -``` - -> Non-copyable by design β€” use `IDpkgQuery::Ptr` (`std::unique_ptr`) for ownership transfer. \ No newline at end of file +``` \ No newline at end of file diff --git a/osquery/utils/pidfile/.pidfile.md b/osquery/utils/pidfile/.pidfile.md index d92e72d1534..8a30963caac 100644 --- a/osquery/utils/pidfile/.pidfile.md +++ b/osquery/utils/pidfile/.pidfile.md @@ -1,52 +1,54 @@ -Provides the `Pidfile` class interface for creating, locking, reading, and managing PID files to ensure single-instance process execution within the osquery framework. +Provides the `Pidfile` class interface for creating, locking, reading, and managing PID files to ensure single-instance process control in osquery. ## Key Components ### `Pidfile` Class -A non-copyable, move-only RAII class managing the lifecycle of a PID file. - -### Error Enum -Defines failure states returned via `Expected`: +A move-only, non-copyable RAII class that manages the lifecycle of a PID file. +**Error Enum** | Value | Description | |---|---| +| `Unknown` | Unclassified error | | `Busy` | Another instance is already running | -| `NotRunning` | No active process found | -| `AccessDenied` | Insufficient file permissions | +| `NotRunning` | Process referenced in PID file is not active | +| `AccessDenied` | Insufficient permissions | | `MemoryAllocationFailure` | Heap allocation failed | -| `IOError` | Filesystem read/write failure | -| `InvalidProcessID` | PID value is malformed or invalid | -| `Unknown` | Unclassified error | +| `IOError` | File I/O operation failed | +| `InvalidProcessID` | PID value is malformed or out of range | -### Static Methods +**Static Methods** +- `create(path)` β€” Creates and locks a PID file, writing the current PID; returns an `Expected` +- `read(path)` β€” Reads and returns the PID stored in an existing PID file -| Method | Description | -|---|---| -| `create(path)` | Creates and locks a PID file, writing the current PID | -| `read(path)` | Reads and returns the PID stored in an existing file | -| `createFile(path)` | Opens/creates the underlying file handle | -| `lockFile(handle)` | Acquires an exclusive lock on the file | -| `readFile(handle)` | Reads raw content from the file handle | -| `writeFile(handle)` | Writes the current process ID to the file | -| `closeFile(handle)` | Releases the file handle | -| `destroyFile(handle, path)` | Closes and deletes the PID file | +**Low-Level File Operations** *(public, platform-specific use)* +- `createFile(path)` β€” Opens/creates the file, returns a `FileHandle` +- `lockFile(handle)` β€” Acquires an exclusive lock on the file +- `readFile(handle)` β€” Reads raw content from the file +- `writeFile(handle)` β€” Writes the current process ID to the file +- `closeFile(handle)` β€” Closes the file handle +- `destroyFile(handle, path)` β€” Releases the lock, closes, and removes the file + +**Stream Operator** +- `operator<<` β€” Formats a `Pidfile::Error` value to an output stream for logging/debugging ## Usage Example ```cpp #include -// Create and lock a PID file for single-instance enforcement +// Create and hold a PID file for the duration of the process auto pidfile = osquery::Pidfile::create("/var/run/osqueryd.pid"); if (pidfile.isError()) { if (pidfile.getError() == osquery::Pidfile::Error::Busy) { std::cerr << "Another instance is already running.\n"; + } else { + std::cerr << "PID file error: " << pidfile.getError() << "\n"; } return 1; } -// Read an existing PID file +// Read an existing PID file without acquiring a lock auto pid = osquery::Pidfile::read("/var/run/osqueryd.pid"); if (pid.isValue()) { std::cout << "Running PID: " << pid.get() << "\n"; diff --git a/osquery/utils/status/.status.md b/osquery/utils/status/.status.md index e754c3118db..fc209a5c079 100644 --- a/osquery/utils/status/.status.md +++ b/osquery/utils/status/.status.md @@ -1,47 +1,51 @@ -Defines the `osquery::Status` class, a lightweight utility for expressing operation outcomes with a numeric code and descriptive message β€” analogous to a simple result type used throughout the osquery codebase. +Defines the `osquery::Status` class, a lightweight result type used throughout osquery to express success or failure of operations, along with a template conversion utility from `Expected` types. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `Status` | Class | Core result type wrapping an integer code and string message | -| `kSuccessCode` | `static constexpr int` | Success sentinel value (`0`) | -| `ok()` | Method | Returns `true` if code equals `kSuccessCode` | -| `getCode()` | Method | Returns the raw integer status code | -| `getMessage()` / `toString()` / `what()` | Methods | Returns the descriptive message string | -| `Status::success()` | Static factory | Creates a success `Status` | -| `Status::failure(...)` | Static factory | Creates a failure `Status` with optional code and message | -| `to(Expected)` | Template function | Converts an `Expected` to a `Status` | +**`Status` class** +- `Status(int c, std::string m)` β€” constructs with a numeric code and message; code `0` = success +- `Status(const ErrorBase&)` β€” constructs a failure status directly from an error object +- `ok()` β€” returns `true` when code equals `kSuccessCode` (0) +- `getCode()` β€” returns the integer status code +- `getMessage()` / `toString()` / `what()` β€” return the status message string +- `operator bool()` β€” allows direct use in `if` statements +- `Status::success()` β€” static factory for a successful status +- `Status::failure(message)` / `Status::failure(code, message)` β€” static factories for failure statuses +- `operator==` / `operator!=` / `operator<<` β€” comparison and stream operators for gtest compatibility + +**`to(Expected)` template** +Converts an `Expected` into a `Status`, mapping success to `Status::success()` and failure to `Status::failure(message)`. ## Usage Example ```cpp #include -#include +#include -// Return status from a function osquery::Status doWork() { - if (someCondition()) { + if (someCondition) { return osquery::Status::success(); } - return osquery::Status::failure("Something went wrong"); + return osquery::Status::failure("something went wrong"); } -// Check result with ok() -auto s = doWork(); -if (s.ok()) { - LOG(INFO) << "Success"; -} else { - LOG(ERROR) << s.getMessage(); // or s.toString() / s.what() -} +void caller() { + auto s = doWork(); -// Implicit bool conversion -if (doWork()) { - LOG(INFO) << "Also works!"; -} + // Boolean check via operator bool() + if (!s) { + LOG(ERROR) << "Failed: " << s.getMessage(); + return; + } -// Convert from Expected<> to Status -Expected result = getExpectedValue(); -osquery::Status status = osquery::to(result); + // Or explicit ok() check + if (s.ok()) { + LOG(INFO) << "Success"; + } + + // Converting an Expected<> to Status + Expected result = getExpected(); + osquery::Status converted = osquery::to(result); +} ``` \ No newline at end of file diff --git a/osquery/utils/system/.time.md b/osquery/utils/system/.time.md index 814e95af8ee..504d1073256 100644 --- a/osquery/utils/system/.time.md +++ b/osquery/utils/system/.time.md @@ -1,38 +1,37 @@ -Cross-platform time utility header providing UNIX epoch and human-readable time conversion functions within the `osquery` namespace. +Cross-platform time utility header for the `osquery` namespace, providing helpers to retrieve, convert, and format time values as UNIX epochs or human-readable ASCII strings. ## Key Components | Function | Description | |---|---| -| `platformAsctime()` | Converts a `struct tm*` to a C++ `std::string` using the platform's `asctime` equivalent | -| `toUnixTime()` | Converts a `struct tm*` to a `uint64_t` UNIX epoch timestamp | -| `getUnixTime()` | Returns the current time as a `uint64_t` UNIX epoch timestamp | -| `toAsciiTime()` | Converts a UTC `struct tm*` to a human-readable string (`"Wed Sep 21 10:27:52 2011"`) | -| `toAsciiTimeUTC()` | Converts a local `struct tm*` to a UTC human-readable string via epoch + `gmtime()` | -| `getAsciiTime()` | Returns the current date/time as a human-readable string | +| `platformAsctime` | Converts a `struct tm*` to a C++ `std::string` using the platform's `asctime` equivalent | +| `toUnixTime` | Converts a `struct tm*` to a `uint64_t` UNIX epoch timestamp | +| `getUnixTime` | Returns the current time as a `uint64_t` UNIX epoch (seconds since epoch) | +| `toAsciiTime` | Converts a UTC `struct tm*` to a human-readable string (e.g. `"Wed Sep 21 10:27:52 2011"`) | +| `toAsciiTimeUTC` | Converts a local `struct tm*` to UTC, then formats it as the same human-readable string | +| `getAsciiTime` | Returns the current UTC time as a human-readable string | ## Usage Example -```cpp +```c #include #include // Get current time as UNIX epoch uint64_t epoch = osquery::getUnixTime(); -std::cout << "Epoch: " << epoch << std::endl; // Get current time as readable string std::string now = osquery::getAsciiTime(); -std::cout << "Now: " << now << std::endl; // "Wed Sep 21 10:27:52 2011" +// now == "Wed Sep 21 10:27:52 2011" // Convert a struct tm to UTC ASCII struct tm tm_time = {}; -gmtime_r(&some_time_t, &tm_time); -std::string utc = osquery::toAsciiTimeUTC(&tm_time); +// ... populate tm_time ... +std::string utcStr = osquery::toAsciiTimeUTC(&tm_time); -// Convert struct tm to epoch +// Convert a struct tm to UNIX epoch uint64_t ts = osquery::toUnixTime(&tm_time); ``` -> **Note:** `toAsciiTime()` expects the `struct tm` to already be in UTC. Use `toAsciiTimeUTC()` when starting from local time, as it internally normalizes via epoch conversion and `gmtime()`. \ No newline at end of file +> **Note:** `toAsciiTime` expects the `struct tm` to already be in UTC. Use `toAsciiTimeUTC` when starting from local time, as it internally applies `gmtime()` conversion. \ No newline at end of file diff --git a/osquery/utils/system/.uptime.md b/osquery/utils/system/.uptime.md index 79f63b62960..396dc49eeda 100644 --- a/osquery/utils/system/.uptime.md +++ b/osquery/utils/system/.uptime.md @@ -1,15 +1,15 @@ -Provides a utility interface for retrieving system uptime within the osquery framework. +Provides a single utility function for retrieving system uptime within the osquery framework. ## Key Components -- **`getUptime()`** β€” Returns the system uptime as a `long` value (typically in seconds) within the `osquery` namespace. +- **`getUptime()`** β€” Returns the system uptime as a `long` value (in seconds) within the `osquery` namespace. ## Usage Example ```c #include "uptime.h" -// Retrieve current system uptime +// Retrieve current system uptime in seconds long uptimeSeconds = osquery::getUptime(); ``` \ No newline at end of file diff --git a/osquery/utils/system/linux/.cpu.md b/osquery/utils/system/linux/.cpu.md index 0a76034b4da..da2f82dcec4 100644 --- a/osquery/utils/system/linux/.cpu.md +++ b/osquery/utils/system/linux/.cpu.md @@ -1,46 +1,46 @@ -Provides CPU topology utilities for Linux systems, exposing sysfs-based CPU state information as bitmask representations. +Header for Linux CPU topology utilities, providing functions to query and decode CPU state masks from sysfs. ## Key Components **Types & Constants** -- `Error` β€” enum with `IOError` and `IncorrectRange` variants for error handling -- `Mask` β€” alias for `std::bitset<128>`, representing up to 128 CPU cores (aligned with Linux `NR_CPUS`) -- `kMaskSize` β€” constant defining the bitmask width (`128`) +- `Error` β€” enum with `IOError` and `IncorrectRange` error codes +- `Mask` β€” `std::bitset<128>` representing up to 128 CPUs (aligned with `NR_CPUS`) +- `kMaskSize` β€” compile-time constant of `128` **Functions** -| Function | Description | -|---|---| -| `decodeMaskFromString(encoded_str)` | Parses a sysfs CPU mask string into a `Mask` bitset | -| `getOfflineRaw()` / `getOffline()` | Returns hotplugged-off or kernel-limited CPUs | -| `getOnlineRaw()` / `getOnline()` | Returns CPUs currently online and scheduled | -| `getPossibleRaw()` / `getPossible()` | Returns CPUs allocated and bringable online | -| `getPresentRaw()` / `getPresent()` | Returns CPUs physically present in the system | +| Function | Returns | Description | +|---|---|---| +| `decodeMaskFromString` | `Expected` | Decodes a sysfs CPU mask string into a `Mask` bitset | +| `getOffline` / `getOfflineRaw` | `Expected` | CPUs hotplugged off or exceeding kernel config limit | +| `getOnline` / `getOnlineRaw` | `Expected` | CPUs currently online and being scheduled | +| `getPossible` / `getPossibleRaw` | `Expected` | CPUs allocated resources, bringable online if present | +| `getPresent` / `getPresentRaw` | `Expected` | CPUs physically identified in the system | -Each query is available in two forms: raw string (`*Raw`) as read from sysfs, or decoded `Mask` bitset. +Each query has both a raw string variant (sysfs content) and a decoded `Mask` variant. ## Usage Example ```cpp -#include +#include -// Get online CPUs as a bitmask -auto result = osquery::cpu::getOnline(); -if (result) { - osquery::cpu::Mask onlineMask = result.get(); +// Get online CPUs as a bitset +auto online = osquery::cpu::getOnline(); +if (online) { + osquery::cpu::Mask mask = online.get(); for (std::size_t i = 0; i < osquery::cpu::kMaskSize; ++i) { - if (onlineMask.test(i)) { + if (mask.test(i)) { // CPU i is online } } -} else { - auto err = result.getError(); - // handle Error::IOError or Error::IncorrectRange } -// Decode a mask string directly -auto mask = osquery::cpu::decodeMaskFromString("0000,0000003f"); +// Decode a mask string manually +auto decoded = osquery::cpu::decodeMaskFromString("0,2-4,7"); +if (!decoded) { + // handle decoded.getError() == Error::IncorrectRange +} ``` -> All functions return `Expected`, following osquery's error-handling pattern β€” always check for errors before accessing the value. \ No newline at end of file +> **Platform:** Linux only β€” reads from `/sys/devices/system/cpu/` sysfs entries as documented in the kernel's `Documentation/cputopology.txt`. \ No newline at end of file diff --git a/osquery/utils/system/linux/proc/.proc.md b/osquery/utils/system/linux/proc/.proc.md index 4e5d984393f..188cc109699 100644 --- a/osquery/utils/system/linux/proc/.proc.md +++ b/osquery/utils/system/linux/proc/.proc.md @@ -1,18 +1,26 @@ -Declares a utility function for retrieving the command-line string of a running process by PID on Linux/Unix systems. +Provides a utility interface for retrieving process information from the Linux `/proc` filesystem. ## Key Components -- **`osquery::proc::cmdline(pid_t pid)`** β€” Reads and returns the full command-line string for the given process ID, typically sourced from `/proc//cmdline`. +- **`osquery::proc::cmdline(pid_t pid)`** β€” Returns the full command-line string for the given process ID by reading from `/proc//cmdline`. ## Usage Example -```c +```cpp #include "proc.h" #include -// Retrieve the command line for a known PID -pid_t target = 1234; -std::string cmd = osquery::proc::cmdline(target); -std::cout << "Command: " << cmd << std::endl; -``` \ No newline at end of file +void printProcessCmdline(pid_t pid) { + std::string cmd = osquery::proc::cmdline(pid); + if (!cmd.empty()) { + std::cout << "Command line for PID " << pid << ": " << cmd << std::endl; + } +} +``` + +## Notes + +- Scoped under the `osquery::proc` namespace to avoid collisions with system-level proc utilities. +- Depends on `boost::filesystem::path` (included for potential path resolution of `/proc` entries). +- Returns an empty string if the PID does not exist or the process command line is unavailable. \ No newline at end of file diff --git a/osquery/utils/system/posix/.errno.md b/osquery/utils/system/posix/.errno.md index 675483fc3a4..e5a9dfa3b85 100644 --- a/osquery/utils/system/posix/.errno.md +++ b/osquery/utils/system/posix/.errno.md @@ -1,36 +1,43 @@ -Provides a type-safe C++ wrapper around POSIX `errno` values within the `osquery` namespace, mapping raw integer error codes to a strongly-typed enum. +Provides a type-safe C++ wrapper around POSIX `errno` values within the osquery namespace, mapping raw integer error codes to a strongly-typed enum. ## Key Components -- **`PosixError` (enum class)** β€” Strongly-typed enumeration of standard POSIX error codes (e.g., `PERM`, `NOENT`, `NOMEM`), mapped directly from system `errno` macros. Includes an `Unknown = 0` sentinel for unrecognized errors. +### `PosixError` (enum class) +A scoped enum mapping standard POSIX error constants (e.g., `EPERM`, `ENOENT`, `ENOMEM`) to named enumerators. Includes an `Unknown = 0` sentinel for unrecognized errors. -- **`impl::toPosixSystemError(int)`** β€” Internal implementation function that converts a raw `errno` integer to its corresponding `PosixError` enum value. +### `impl::toPosixSystemError(int from_errno)` +Internal conversion function that translates a raw `errno` integer value into the corresponding `PosixError` enumerator. -- **`to(int from_errno)`** β€” Template conversion function, enabled only when `ToType` is `PosixError` (via `std::enable_if`). Provides a clean, generic interface for converting raw errno integers to `PosixError`. +### `to(int from_errno)` +A templated conversion helper, enabled only when `ToType` is `PosixError` (via `std::enable_if`). Provides a clean, generic call-site interface for converting raw errno integers. ## Usage Example ```cpp -#include - -// Direct conversion from a raw errno value -int result = open("/nonexistent/path", O_RDONLY); -if (result < 0) { - PosixError err = osquery::to(errno); - - switch (err) { - case osquery::PosixError::NOENT: - // Handle file not found - break; - case osquery::PosixError::ACCES: - // Handle permission denied - break; - default: - // Handle unknown/other errors - break; +#include "errno.h" // osquery errno wrapper + +#include + +void openFile(const char* path) { + int fd = open(path, O_RDONLY); + if (fd < 0) { + // Convert raw errno to type-safe PosixError + osquery::PosixError err = osquery::to(errno); + + switch (err) { + case osquery::PosixError::NOENT: + // Handle file not found + break; + case osquery::PosixError::ACCES: + // Handle permission denied + break; + default: + // Handle unknown/other errors + break; + } } } ``` -> **Note:** The `to()` template is SFINAE-constrained, so calling it with a non-`PosixError` type will result in a compile-time error, preventing accidental misuse. \ No newline at end of file +> **Note:** The `to<>()` template is SFINAE-constrained β€” passing any type other than `PosixError` will result in a compile-time error, preventing accidental misuse. \ No newline at end of file diff --git a/osquery/utils/system/posix/.system.md b/osquery/utils/system/posix/.system.md index 76ea3ea991a..1881074ac12 100644 --- a/osquery/utils/system/posix/.system.md +++ b/osquery/utils/system/posix/.system.md @@ -1,49 +1,46 @@ -Provides privilege dropping and restoration utilities for osquery, enabling temporary reduction of process permissions to a specified user, group, or path owner. +Provides the `DropPrivileges` RAII class for managing Unix process privilege dropping and restoration within the osquery framework. ## Key Components ### Types -- **`PlatformPidType`** β€” Platform-agnostic alias for `pid_t`, representing a process identifier. +- **`PlatformPidType`** β€” Platform-agnostic alias for `pid_t` process identifiers ### Class: `DropPrivileges` -A non-copyable RAII-style class that temporarily lowers effective process privileges and automatically restores them on destruction. +A noncopyable RAII class that temporarily lowers effective UID/GID and automatically restores them on destruction. | Member | Description | |---|---| -| `static get()` | Factory method returning a `DropPrivilegesRef` (`shared_ptr`) instance | -| `dropToParent(path)` | Drops privileges to the owner of the parent directory of the given path | +| `get()` | Static factory β€” returns a `DropPrivilegesRef` (`shared_ptr`) instance | +| `dropToParent(path)` | Drops privileges to match the owner of the parent directory of `path` | | `dropTo(uid, gid)` | Drops to explicit numeric UID/GID | -| `dropTo(uid_str, gid_str)` | Drops to string-specified UID/GID | | `dropTo(user)` | Drops to a named user's UID/GID | -| `dropped()` | Returns `true` if effective privileges differ from real privileges | -| `~DropPrivileges()` | Destructor automatically restores original effective permissions | +| `dropTo(uid_str, gid_str)` | Drops to string-encoded UID/GID values | +| `dropped()` | Returns `true` if effective IDs differ from real IDs | +| `~DropPrivileges()` | Restores original effective privileges on scope exit | + +**`DropPrivilegesRef`** β€” `std::shared_ptr` convenience alias. ## Usage Example -```cpp +```c #include -#include -// Drop privileges to the owner of a file's parent directory -{ +void accessRestrictedFile(const boost::filesystem::path& filePath) { + // Acquire a privilege-drop handle scoped to this block auto dropper = osquery::DropPrivileges::get(); - if (dropper->dropToParent("/etc/osquery/osquery.conf")) { - // Executing with reduced privileges here - readSensitiveFile(); + + // Drop to the owning user of the parent directory + if (!dropper->dropToParent(filePath)) { + // Handle failure β€” privileges could not be dropped + return; } - // Privileges automatically restored when 'dropper' goes out of scope -} -// Drop to a specific user -{ - auto dropper = osquery::DropPrivileges::get(); - dropper->dropTo("nobody"); + // Effective UID/GID are now lowered + // ... perform restricted operations ... - if (dropper->dropped()) { - // Confirms effective UID/GID differ from real - } + // Privileges are automatically restored when 'dropper' goes out of scope } ``` -> **Note:** Only one active privilege drop should exist at a time. Attempting a second drop while one is still active will return `false`. The destructor guarantees privilege restoration, making this safe for use in scoped blocks. \ No newline at end of file +> **Note:** Only one active privilege drop is permitted at a time. Calling any `dropTo` variant while a drop is already active will return `false`. The destructor calls `restoreGroups()` internally to clean up supplemental group state. \ No newline at end of file diff --git a/osquery/utils/system/windows/.etw_helpers.md b/osquery/utils/system/windows/.etw_helpers.md index 38423f79748..30152bbc12f 100644 --- a/osquery/utils/system/windows/.etw_helpers.md +++ b/osquery/utils/system/windows/.etw_helpers.md @@ -5,34 +5,25 @@ Windows ETW (Event Tracing for Windows) utility header providing helper function ### Functions -| Function | Signature | Description | -|---|---|---| -| `sidStringFromEtwRecord` | `std::string(const EVENT_RECORD&)` | Extracts and returns the user SID string from the extended header of an ETW event record | -| `processImagePathFromProcessId` | `std::string(uint32_t processId)` | Resolves and returns the process image file path from a given process ID | +| Function | Description | +|----------|-------------| +| `sidStringFromEtwRecord` | Extracts and returns the user SID string from an ETW `EVENT_RECORD`'s extended header data | +| `processImagePathFromProcessId` | Resolves and returns the executable image file path for a given process ID | ## Usage Example ```c #include "etw_helpers.h" -// Extract user SID from an ETW event record -VOID WINAPI MyEventCallback(PEVENT_RECORD pRecord) { - // Get the SID of the user associated with the event - std::string userSid = osquery::sidStringFromEtwRecord(*pRecord); +// ETW event callback context +void WINAPI onEventRecord(PEVENT_RECORD record) { + // Extract the SID of the user associated with the ETW event + std::string userSid = osquery::sidStringFromEtwRecord(*record); - // Resolve the image path of the process that generated the event - uint32_t pid = pRecord->EventHeader.ProcessId; + // Resolve the process image path from a known PID + uint32_t pid = static_cast(record->EventHeader.ProcessId); std::string imagePath = osquery::processImagePathFromProcessId(pid); - - // Use the extracted metadata - LOG(INFO) << "Event from PID " << pid - << " | SID: " << userSid - << " | Path: " << imagePath; } ``` -## Notes - -- **Windows-only**: Depends on `` and `osquery/utils/system/system.h`, scoped to Windows ETW infrastructure. -- All symbols are declared within the `osquery` namespace. -- SID extraction relies on the **extended data** section of the `EVENT_RECORD` structure; ensure the ETW session is configured to include user SID data. \ No newline at end of file +> **Note:** This header is Windows-only and depends on `evntcons.h` from the Windows SDK. It is intended for use within osquery's ETW-based event subscribers and process monitoring components. \ No newline at end of file diff --git a/osquery/utils/system/windows/.system.md b/osquery/utils/system/windows/.system.md index dea479f326f..767ee2637f0 100644 --- a/osquery/utils/system/windows/.system.md +++ b/osquery/utils/system/windows/.system.md @@ -1,19 +1,26 @@ -Windows platform compatibility header that provides POSIX-to-Windows shim definitions and type aliases for the osquery framework, ensuring consistent Windows API configuration across compilation units. +Windows platform compatibility header that bridges POSIX/Unix APIs and definitions to their Windows equivalents for the osquery framework. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `SIGHUP` | Macro | Defined as `1`, mirroring POSIX convention | -| `SIGUSR1` | Macro | Mapped to `SIGILL` (Windows has no native `SIGUSR1`) | -| `SIGALRM` | Macro | Aliased to `SIGUSR1` | -| `GetObject` / `GetMessage` | `#undef` | Removes conflicting Win32 global macros that break RapidJSON | -| `gmtime_r` | Function | Thread-safe `gmtime` wrapper matching POSIX signature | -| `localtime_r` | Function | Thread-safe `localtime` wrapper matching POSIX signature | -| `alarm` | Function | No-op stub satisfying POSIX `alarm()` call sites | -| `pid_t` | Type alias | Defined as `unsigned long` (equivalent to Win32 `DWORD`) | -| `PlatformPidType` | Type alias | Defined as `void*` for Windows process handle abstraction | +**Macro Definitions** +- `SIGHUP` β€” Defined as `1` to mirror POSIX behavior +- `SIGUSR1` β€” Mapped to `SIGILL` (Windows does not generate `SIGUSR1`) +- `SIGALRM` β€” Aliased to `SIGUSR1` +- `NOMINMAX`, `WIN32_LEAN_AND_MEAN`, `_WIN32_DCOM` β€” Standard Windows build guards + +**Macro Undefs** +- `#undef GetObject` / `#undef GetMessage` β€” Removes conflicting Windows global macros that break RapidJSON and other libraries + +**`osquery` Namespace Exports** + +| Symbol | Description | +|--------|-------------| +| `gmtime_r()` | POSIX-compatible `gmtime_r` wrapper (reversed parameter order vs. Windows `gmtime_s`) | +| `localtime_r()` | POSIX-compatible `localtime_r` wrapper | +| `alarm()` | No-op stub satisfying POSIX `alarm()` call sites | +| `pid_t` | Typedef aliasing `unsigned long` (Windows `DWORD`) | +| `PlatformPidType` | Typedef aliasing `void*` for Windows process handle abstraction | ## Usage Example @@ -25,15 +32,13 @@ time_t now = time(nullptr); struct tm result; osquery::gmtime_r(&now, &result); -// Use portable pid type +// Use portable pid_t osquery::pid_t pid = GetCurrentProcessId(); // Signal constants work cross-platform -int sig = SIGHUP; // = 1 +if (sig == SIGHUP) { + // handle reload +} ``` -## Notes - -- Requires `NTDDI_VERSION` and `_WIN32_WINNT` to be defined at build time (enforced via `#error`); see `tools/build_defs/oss/osquery/cxx.bzl`. -- `WIN32_LEAN_AND_MEAN` and `_WIN32_DCOM` are set here to guarantee consistent Windows API surface across all translation units. -- Warning `C4067` is suppressed locally around `sddl.h` to avoid spurious preprocessor diagnostic noise from the Windows SDK. \ No newline at end of file +> **Note:** `NTDDI_VERSION` and `_WIN32_WINNT` **must** be defined before including this header β€” typically via `tools/build_defs/oss/osquery/cxx.bzl`. A missing definition triggers a compile-time `#error`. \ No newline at end of file diff --git a/osquery/utils/system/windows/.users_groups_helpers.md b/osquery/utils/system/windows/.users_groups_helpers.md index 621b0330895..6404c63c6b0 100644 --- a/osquery/utils/system/windows/.users_groups_helpers.md +++ b/osquery/utils/system/windows/.users_groups_helpers.md @@ -1,56 +1,60 @@ -Windows-specific utility header providing RAII wrappers and helper functions for Windows Net API objects, SID manipulation, and user/group account lookups within the osquery framework. +Windows-specific RAII wrapper and utility header for managing Windows Net API objects and performing SID/user/group lookups within the osquery framework. ## Key Components ### `NetApiObjectPtr` -A move-only RAII smart pointer template that manages Windows Net API buffer lifetimes, automatically calling `NetApiBufferFree()` on destruction. - -| Method | Description | -|--------|-------------| -| `get_new_ptr()` | Returns a raw double pointer for use with Net API calls; frees existing buffer first | -| `get()` | Returns a const raw pointer to the managed buffer | -| `operator->()` | Direct member access to the managed object | +A move-only RAII smart pointer template that manages Windows Net API buffer lifetimes, automatically calling `NetApiBufferFree()` on destruction. Provides: +- `get_new_ptr()` β€” safely acquires a new raw pointer, freeing any existing allocation first +- `get()` β€” const access to the underlying pointer +- Standard comparison operators against raw pointers ### Type Aliases +Convenience aliases for common Net API info structures: -```c -user_info_0_ptr // NetApiObjectPtr -user_info_2_ptr // NetApiObjectPtr -user_info_3_ptr // NetApiObjectPtr -user_info_4_ptr // NetApiObjectPtr -localgroup_users_info_0_ptr // NetApiObjectPtr -localgroup_info_1_ptr // NetApiObjectPtr -``` +| Alias | Type | +|---|---| +| `user_info_0_ptr` | `NetApiObjectPtr` | +| `user_info_2_ptr` | `NetApiObjectPtr` | +| `user_info_3_ptr` | `NetApiObjectPtr` | +| `user_info_4_ptr` | `NetApiObjectPtr` | +| `localgroup_users_info_0_ptr` | `NetApiObjectPtr` | +| `localgroup_info_1_ptr` | `NetApiObjectPtr` | ### Free Functions | Function | Description | -|----------|-------------| +|---|---| | `psidToString(PSID)` | Converts a binary SID struct to its string representation | -| `getSidFromAccountName(...)` | Looks up a SID by account name (wide string or `LPCWSTR`) | +| `getSidFromAccountName(...)` | Looks up a SID by username (overloads for `std::wstring` and `LPCWSTR`) | | `getRidFromSid(PSID)` | Extracts the Relative Identifier (RID) from a SID | -| `getGidFromUserSid(PSID)` | Returns optional GID from a user's SID | -| `getGidFromUsername(LPCWSTR)` | Returns optional GID from a username | -| `getGroupSidFromUserSid(PSID)` | Returns group SID string from a user SID | -| `getGroupSidFromUsername(...)` | Returns group SID string from a username | +| `getGidFromUserSid(PSID)` | Returns the primary group ID for a user SID | +| `getGidFromUsername(LPCWSTR)` | Returns the primary group ID by username | +| `getGroupSidFromUserSid(PSID)` | Returns the group SID string for a user SID | +| `getGroupSidFromUsername(...)` | Returns the group SID string by username | | `getUserHomeDir(string)` | Returns the home directory path for a given SID string | ## Usage Example -```c -// RAII-managed Net API buffer usage +```cpp +#include "users_groups_helpers.h" + +// RAII-managed Net API buffer osquery::user_info_4_ptr userInfo; -NetUserGetInfo(nullptr, L"john.doe", 4, +NetUserGetInfo(nullptr, L"jdoe", 4, reinterpret_cast(userInfo.get_new_ptr())); -// SID conversions -std::string sidStr = osquery::psidToString(userInfo->usri4_user_sid); -DWORD rid = osquery::getRidFromSid(userInfo->usri4_user_sid); +// Convert SID to string +if (userInfo != nullptr) { + std::string sidStr = osquery::psidToString(userInfo->usri4_user_sid); + + // Lookup group GID + auto gid = osquery::getGidFromUserSid(userInfo->usri4_user_sid); + if (gid.has_value()) { + std::cout << "GID: " << gid.value() << "\n"; + } -// Group lookup -auto gid = osquery::getGidFromUsername(L"john.doe"); -if (gid.has_value()) { - // use gid.value() + // Get home directory + std::string homeDir = osquery::getUserHomeDir(sidStr); } ``` \ No newline at end of file diff --git a/osquery/utils/windows/.lzxpress.md b/osquery/utils/windows/.lzxpress.md index bcc5e25a6b0..6faa65b2101 100644 --- a/osquery/utils/windows/.lzxpress.md +++ b/osquery/utils/windows/.lzxpress.md @@ -1,34 +1,28 @@ -Header file exposing a Windows LZ Xpress Huffman decompression utility within the osquery framework. +Header file providing an interface for decompressing LZ Xpress Huffman-compressed data within the osquery framework. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `ExpectedDecompressData` | Type alias | `Expected, ConversionError>` β€” wraps decompressed bytes or a conversion error | -| `decompressLZxpress` | Function | Decompresses LZ Xpress Huffman-encoded data, returning a byte vector or error | +- **`ExpectedDecompressData`** β€” Type alias for `Expected, ConversionError>`, representing either the decompressed byte vector or a conversion error. +- **`decompressLZxpress()`** β€” Decompresses LZ Xpress Huffman-encoded data, returning the raw bytes or an error on failure. ## Usage Example ```c -#include +#include -std::vector compressed = /* raw compressed bytes from registry/EVTX/etc. */; -unsigned long uncompressed_size = 4096; // known original size +std::vector compressed = getCompressedBuffer(); // your compressed input +unsigned long uncompressedSize = 4096; // known uncompressed size -auto result = osquery::decompressLZxpress(compressed, uncompressed_size); +ExpectedDecompressData result = osquery::decompressLZxpress(compressed, uncompressedSize); -if (result.isError()) { - // Handle ConversionError - LOG(ERROR) << "Decompression failed: " << result.getError().getMessage(); -} else { +if (result) { std::vector data = result.take(); - // Process decompressed bytes... + // use decompressed data... +} else { + // handle ConversionError + LOG(ERROR) << "Decompression failed"; } ``` -## Notes - -- **Windows-only**: Uses `UCHAR` from ``, which maps to Windows platform types. -- **Caller-provided size**: The `size` parameter must reflect the **total uncompressed output size** in bytes β€” typically sourced from metadata (e.g., registry hive headers or event log records). -- **Error handling**: Returns an `Expected` type (similar to `std::expected`) β€” always check `isError()` before consuming the result to avoid undefined behavior. \ No newline at end of file +> **Note:** The caller must supply the expected uncompressed size in bytes. LZ Xpress Huffman is commonly encountered in Windows artifacts (e.g., registry hives, prefetch files), making this utility relevant to Windows-targeted osquery tables. \ No newline at end of file diff --git a/osquery/utils/windows/.shellitem.md b/osquery/utils/windows/.shellitem.md index 3a9de17da96..9f8431114cd 100644 --- a/osquery/utils/windows/.shellitem.md +++ b/osquery/utils/windows/.shellitem.md @@ -1,56 +1,51 @@ -Windows Shell Item parsing utilities for osquery, providing helper functions to extract structured data from various Windows Shell Item binary formats (LNK files, jump lists, shellbags, etc.). +Windows Shell Item parsing utilities for osquery, providing helper functions to extract structured data from various Windows Shell Link (LNK) and Jump List binary formats. ## Key Components -### `ShellFileEntryData` Struct +### `ShellFileEntryData` struct Holds parsed file entry metadata extracted from shell items: | Field | Type | Description | |---|---|---| | `path` | `string` | File system path | -| `dos_created/accessed/modified` | `long long` | DOS timestamps | +| `dos_created` / `dos_accessed` / `dos_modified` | `long long` | DOS timestamps | | `mft_entry` / `mft_sequence` | `long long` / `int` | NTFS MFT reference | -| `extension_sig` / `identifier` | `string` | Shell extension data | -| `version` / `string_size` | `int` | Format metadata | +| `extension_sig` / `identifier` | `string` | Shell extension metadata | +| `version` / `string_size` | `int` | Format versioning | ### Parsing Functions | Function | Returns | |---|---| -| `fileEntry()` | `ShellFileEntryData` struct with full file metadata | -| `propertyStore()` | Windows Property List GUID name or value | -| `networkShareItem()` | Network share UNC name | -| `zipContentItem()` | Zip archive entry name | -| `rootFolderItem()` | Root folder (e.g., Desktop, My Computer) name | -| `driveLetterItem()` | Drive letter string | -| `controlPanelItem()` / `controlPanelCategoryItem()` | Control panel entry name | -| `ftpItem()` | Vector of FTP hostnames/paths | -| `guidParse()` | Properly-ordered GUID string from little-endian bytes | -| `propertyViewDrive()` | Drive name from property view data | -| `variableGuid()` / `variableFtp()` | GUID name or FTP string from variable-length items | -| `mtpDevice()` / `mtpFolder()` / `mtpRoot()` | MTP (Media Transfer Protocol) device/folder/root names | +| `fileEntry()` | `ShellFileEntryData` β€” full file entry struct | +| `propertyStore()` | Property List GUID name or value | +| `networkShareItem()` | Network share name | +| `zipContentItem()` | Zip content name | +| `rootFolderItem()` | Root folder name | +| `driveLetterItem()` | Drive letter/name | +| `controlPanelCategoryItem()` | Control panel category | +| `controlPanelItem()` | Control panel name | +| `ftpItem()` | `vector` of FTP hostnames | +| `guidParse()` | Normalized GUID string from little-endian bytes | +| `propertyViewDrive()` | User property drive name | +| `variableGuid()` | Variable GUID name or raw GUID | +| `variableFtp()` | Variable FTP string | +| `mtpDevice()` / `mtpFolder()` / `mtpRoot()` | MTP device/folder/root names | ## Usage Example ```c #include "shellitem.h" -// Parse a raw shell item binary blob -std::string raw_shell_data = /* binary data from shellbag registry key */; +// Parse a file entry from raw shell item binary data +osquery::ShellFileEntryData entry = osquery::fileEntry(raw_shell_bytes); +std::cout << entry.path << " (MFT: " << entry.mft_entry << ")\n"; -// Extract file entry metadata -osquery::ShellFileEntryData entry = osquery::fileEntry(raw_shell_data); -std::cout << "Path: " << entry.path << "\n"; -std::cout << "Created: " << entry.dos_created << "\n"; -std::cout << "MFT Entry: " << entry.mft_entry << "\n"; +// Parse a GUID from little-endian binary +std::string guid = osquery::guidParse(raw_guid_bytes); -// Parse a GUID from little-endian bytes -std::string guid = osquery::guidParse(raw_shell_data); - -// Resolve a network share item -std::string share = osquery::networkShareItem(raw_shell_data); - -// Enumerate FTP entries -std::vector ftp_paths = osquery::ftpItem(raw_shell_data); +// Resolve a Windows Property Store entry +std::vector offsets = {0x20, 0x48}; +std::string prop = osquery::propertyStore(raw_shell_bytes, offsets); ``` \ No newline at end of file diff --git a/osquery/utils/ycloud/.ycloud_util.md b/osquery/utils/ycloud/.ycloud_util.md index fd87fea4287..8e4774200c5 100644 --- a/osquery/utils/ycloud/.ycloud_util.md +++ b/osquery/utils/ycloud/.ycloud_util.md @@ -1,36 +1,38 @@ -Utility header for retrieving Yandex Cloud (YCloud) instance metadata within the osquery framework. +Utility header for interacting with Yandex Cloud (YCloud) instance metadata, providing functions to extract and fetch cloud-specific configuration values from JSON metadata documents. ## Key Components | Function | Description | |---|---| | `getYCloudKey` | Extracts a named key from a YCloud metadata JSON document | -| `getVendorKey` | Extracts a vendor-specific key from a YCloud metadata JSON document | -| `getYCloudSshKey` | Retrieves the SSH public key from a YCloud metadata document | -| `getSerialPortEnabled` | Reads the serial port enabled flag from a YCloud metadata document | -| `getZoneId` | Parses and returns the zone identifier from a raw zone string | -| `fetchYCloudMetadata` | Fetches YCloud instance metadata from a given endpoint and populates a JSON document | - -All functions reside in the `osquery` namespace and operate on `JSON` document objects (`osquery::JSON`). +| `getVendorKey` | Extracts a named key from the vendor-specific section of a JSON document | +| `getYCloudSshKey` | Retrieves the SSH public key from YCloud metadata | +| `getSerialPortEnabled` | Returns whether the serial port is enabled on the instance | +| `getZoneId` | Parses and returns the availability zone ID from a zone string | +| `fetchYCloudMetadata` | Fetches the YCloud instance metadata from the given endpoint and populates a JSON document | ## Usage Example ```c #include "ycloud_util.h" +#include +#include + +using namespace osquery; + +JSON doc; +const std::string endpoint = "http://169.254.169.254/latest/meta-data/"; + +// Fetch metadata from the YCloud IMDS endpoint +Status s = fetchYCloudMetadata(doc, endpoint); -osquery::JSON doc; -osquery::Status status = osquery::fetchYCloudMetadata( - doc, "http://169.254.169.254/latest/meta-data/" -); - -if (status.ok()) { - std::string instanceId = osquery::getYCloudKey(doc, "id"); - std::string vendor = osquery::getVendorKey(doc, "vendor"); - std::string sshKey = osquery::getYCloudSshKey(doc); - std::string serialPort = osquery::getSerialPortEnabled(doc); - std::string zone = osquery::getZoneId("ru-central1-a"); +if (s.ok()) { + std::string instanceId = getYCloudKey(doc, "id"); + std::string sshKey = getYCloudSshKey(doc); + std::string zone = getZoneId(getYCloudKey(doc, "zone")); + std::string serialPort = getSerialPortEnabled(doc); } ``` -> **Note:** `fetchYCloudMetadata` returns an `osquery::Status` object β€” always check `status.ok()` before consuming the populated `doc` to handle network or parse failures gracefully. \ No newline at end of file +> **Note:** `fetchYCloudMetadata` returns an osquery `Status` object. Always check `s.ok()` before accessing parsed fields to handle network or parsing failures gracefully. \ No newline at end of file diff --git a/osquery/worker/ipc/.table_ipc_json_converter.md b/osquery/worker/ipc/.table_ipc_json_converter.md index 09a49b32dfd..cb216e91eb1 100644 --- a/osquery/worker/ipc/.table_ipc_json_converter.md +++ b/osquery/worker/ipc/.table_ipc_json_converter.md @@ -1,48 +1,42 @@ -Defines the `TableIPCJSONConverter` utility class for bidirectional serialization between osquery's internal data structures and JSON messages exchanged over IPC channels. +Defines the `TableIPCJSONConverter` utility class for serializing and deserializing IPC messages between JSON and osquery's internal data types (query results and log messages) used in inter-process communication for table extensions. ## Key Components -### Enum: `JSONMessageType` -Identifies the type of JSON message being transmitted over IPC: +**Enum** +- `JSONMessageType` β€” Discriminates IPC message kinds: `None`, `QueryData`, `Log`, `Job` -| Value | Description | -|---|---| -| `None` | Unrecognized or unset message type | -| `QueryData` | Contains SQL query result rows | -| `Log` | Contains a log message with priority and type | -| `Job` | Represents a job/task message | - -### Class: `TableIPCJSONConverter` - -All methods are static β€” no instantiation required. +**Class: `TableIPCJSONConverter`** (all methods static) | Method | Direction | Description | |---|---|---| -| `JSONToQueryData` | JSON β†’ QueryData | Deserializes JSON into osquery row results | -| `queryDataToJSON` | QueryData β†’ JSON | Serializes query results into JSON | -| `JSONToLogMessage` | JSON β†’ Log fields | Extracts priority, log type, and message string | -| `logMessageToJSON` | Log fields β†’ JSON | Serializes a log entry into JSON | -| `JSONTypeToMessageType` | JSON β†’ enum | Inspects a JSON message and resolves its `JSONMessageType` | +| `JSONToQueryData` | JSON β†’ `QueryData` | Deserializes a JSON payload into query result rows | +| `queryDataToJSON` | `QueryData` β†’ JSON | Serializes query result rows into a JSON payload | +| `JSONToLogMessage` | JSON β†’ log fields | Extracts `priority`, `log_type`, and `message` string from JSON | +| `logMessageToJSON` | log fields β†’ JSON | Packs log fields into a JSON payload | +| `JSONTypeToMessageType` | JSON β†’ enum | Reads the message type discriminator from a JSON envelope | ## Usage Example -```cpp -#include +```c +#include -// Serialize query results for IPC transmission -QueryData results = getQueryResults(); +// Serialize query results for IPC transport +QueryData results = getTableRows(); JSON json_helper; Status s = TableIPCJSONConverter::queryDataToJSON(results, json_helper); -if (!s.ok()) { /* handle error */ } -// Deserialize received JSON back into QueryData -JSON received_json = receiveFromIPC(); +// On the receiving end, detect message type then deserialize JSONMessageType msg_type; -TableIPCJSONConverter::JSONTypeToMessageType(received_json, msg_type); +TableIPCJSONConverter::JSONTypeToMessageType(json_helper, msg_type); if (msg_type == JSONMessageType::QueryData) { - QueryData data; - TableIPCJSONConverter::JSONToQueryData(received_json, data); + QueryData received; + TableIPCJSONConverter::JSONToQueryData(json_helper, received); } + +// Serialize a log message for IPC transport +JSON log_json; +TableIPCJSONConverter::logMessageToJSON( + /*priority=*/2, /*log_type=*/1, "Extension loaded", log_json); ``` \ No newline at end of file diff --git a/osquery/worker/ipc/linux/.linux_table_container_ipc.md b/osquery/worker/ipc/linux/.linux_table_container_ipc.md index 5dccd728bf4..a9a7d949ec4 100644 --- a/osquery/worker/ipc/linux/.linux_table_container_ipc.md +++ b/osquery/worker/ipc/linux/.linux_table_container_ipc.md @@ -1,35 +1,32 @@ -Manages IPC communication between the osquery host process and container worker processes on Linux, orchestrating connection lifecycle, query dispatch, and result retrieval across namespace boundaries. +Manages IPC communication between osquery and Linux container workers for executing table queries within container namespaces. ## Key Components -### `LinuxTableContainerIPC` Class -Main class driving container worker lifecycle and query execution over pipe-based IPC. +### `LinuxTableContainerIPC` (class) +Drives the full lifecycle of container-based table queries β€” connecting to a container worker process, sending query jobs, and retrieving results via pipe-based IPC. -| Member | Description | -|---|---| -| `connectToContainer()` | Spawns/connects to a container worker process for a given table | +| Method | Description | +|--------|-------------| +| `connectToContainer()` | Spawns/connects to a container worker process for a named table | | `retrieveQueryDataFromContainer()` | Sends a `QueryContext` and collects `QueryData` results | -| `executeQueryJobs()` | Worker-side loop that processes incoming query jobs (no-return) | -| `stopContainerWorker()` | Terminates the container worker process and cleans up | -| `handleLog()` | Routes GLOG log messages received from the worker | -| `handleJob()` | Executes a table generate function inside the container namespace | +| `executeQueryJobs()` | Runs the worker-side job loop (non-returning) | +| `stopContainerWorker()` | Terminates the container worker and cleans up resources | +| `handleLog()` | IPC message handler for log messages from the worker | +| `handleJob()` | IPC message handler for incoming query jobs | -### `CleanupWorkerOnError` (RAII Guard) -Inner class that automatically calls `stopContainerWorker()` on scope exit unless `dismiss()` is called β€” ensures cleanup on error paths. +### `CleanupWorkerOnError` (RAII guard, private inner class) +Automatically calls `stopContainerWorker()` on scope exit unless `dismiss()` is called β€” ensures cleanup on error paths. ### Free Functions | Function | Description | -|---|---| -| `hasNamespaceConstraint()` | Returns `true` if `QueryContext` includes a `pid_with_namespace` constraint | -| `generateInNamespace()` | Entry point to run a table generator inside a container namespace via IPC | +|----------|-------------| +| `hasNamespaceConstraint()` | Returns `true` if the query context includes a `pid_with_namespace` constraint | +| `generateInNamespace()` | Entry point to execute a table generator function inside a target container namespace | -### Type Alias -```c -using TableGeneratePtr = QueryData (*)(QueryContext&, Logger&); -``` -Function pointer type for table generator callbacks passed into the container worker. +### `TableGeneratePtr` (type alias) +Function pointer type for table generator functions: `QueryData(*)(QueryContext&, Logger&)`. ## Usage Example @@ -37,17 +34,15 @@ Function pointer type for table generator callbacks passed into the container wo PipeChannelFactory factory; LinuxTableContainerIPC container_ipc(factory); +// Connect to a container worker for the target table Status s = container_ipc.connectToContainer( - "process_open_files", /*keep_process_open=*/true, &myTableGenerate); + "processes", /* keep_process_open */ true, &generateProcesses); if (s.ok()) { QueryData results; container_ipc.retrieveQueryDataFromContainer(context, results); } -``` -For namespace-aware table generation, prefer the free function: - -```cpp +// Or use the high-level helper for namespace-aware generation QueryData rows = generateInNamespace(context, "processes", &generateProcesses); ``` \ No newline at end of file diff --git a/osquery/worker/ipc/linux/.linux_table_ipc.md b/osquery/worker/ipc/linux/.linux_table_ipc.md index b2371264836..69204ea8731 100644 --- a/osquery/worker/ipc/linux/.linux_table_ipc.md +++ b/osquery/worker/ipc/linux/.linux_table_ipc.md @@ -4,41 +4,46 @@ Inter-process communication layer for Linux table workers in osquery, providing ## Key Components ### `LinuxTableIPC` -Extends `TableIPCBase` (CRTP) to manage blocking pipe-based IPC channels between worker processes. +Extends `TableIPCBase` (CRTP pattern) to manage blocking pipe channels between processes. | Method | Description | -|---|---| -| `sendJSONString` / `recvJSONString` | Low-level JSON string transport over the active pipe channel | -| `processLogMessage` | Deserializes and handles an incoming log message | -| `processJobMessage` | Deserializes and dispatches an incoming query job | -| `processQueryDataMessage` | Deserializes query results into a `QueryData` output param | -| `createChannelTicket` | Allocates a new `PipeChannelTicket` via the factory | -| `connectToChild` / `connectToParent` | Establishes the active channel from either side of the fork | -| `setActiveChannelIfOpen` | Activates an existing channel by table name if available | -| `closeActiveChannel` | Tears down the current pipe connection | -| `getTableNameFromPid` | Reverse-lookup of table name by remote PID | -| `isChannelOpen` / `getRemotePid` / `getTableName` | Channel state inspection helpers | +|--------|-------------| +| `sendJSONString` / `recvJSONString` | Low-level JSON message transport over pipes | +| `processLogMessage` | Handles incoming log messages from child | +| `processJobMessage` | Handles incoming query job dispatch | +| `processQueryDataMessage` | Receives and populates `QueryData` results | +| `createChannelTicket` | Allocates a new `PipeChannelTicket` via factory | +| `connectToChild` / `connectToParent` | Establishes pipe channel from either end | +| `setActiveChannelIfOpen` | Activates an existing open channel by table name | +| `closeActiveChannel` | Tears down the current pipe channel | +| `getTableNameFromPid` | Resolves a table name from a child PID | +| `isChannelOpen` / `getRemotePid` / `getTableName` | Channel state accessors | ### `LinuxTableIPCLogger` -Implements the `Logger` interface, forwarding `log()` and `vlog()` calls as serialized log messages over the IPC channel. +Implements the `Logger` interface, forwarding `log()` and `vlog()` calls as IPC log messages through a `LinuxTableIPC` instance using `GLOGLogType`. ## Usage Example ```cpp PipeChannelFactory factory; TableIPCMessageHandler handler; -LinuxTableIPC ipc(factory, handler); - -// Parent side: establish connection to a spawned child -auto ticket = ipc.createChannelTicket(); -ipc.connectToParent("some_table", ticket); -// Send/receive JSON payloads -std::string response; -ipc.sendJSONString(R"({"action":"query"})"); -ipc.recvJSONString(response); +LinuxTableIPC ipc(factory, handler); -// Attach logger to route GLOG output through the pipe +// Parent side: create ticket and connect after fork +PipeChannelTicket ticket = ipc.createChannelTicket(); +pid_t child = fork(); +if (child == 0) { + ipc.connectToParent("custom_table", ticket); + std::string json; + ipc.recvJSONString(json); + ipc.processJobMessage(/* parsed JSON */); +} else { + ipc.connectToChild("custom_table", ticket, child); + ipc.sendJSONString("{\"action\":\"query\"}"); +} + +// Attach logger in child worker LinuxTableIPCLogger logger(ipc); -logger.log(google::INFO, "Worker ready"); +logger.log(1, "Worker initialized"); ``` \ No newline at end of file diff --git a/osquery/worker/ipc/posix/.pipe_channel.md b/osquery/worker/ipc/posix/.pipe_channel.md index 75073f9eae1..b1fc21a9014 100644 --- a/osquery/worker/ipc/posix/.pipe_channel.md +++ b/osquery/worker/ipc/posix/.pipe_channel.md @@ -1,36 +1,41 @@ -Implements a pipe-based IPC channel for inter-process communication between osquery worker processes, using file descriptors for bidirectional message passing. +Implements a unidirectional IPC pipe channel for inter-process communication between osquery worker processes and table query handlers. ## Key Components -- **`PipeChannel`** β€” Concrete channel class inheriting from `TableChannelBase` (CRTP pattern) that wraps a pair of Unix pipe file descriptors for sending and receiving string messages between processes. +**`PipeChannel`** β€” Concrete channel class inheriting from `TableChannelBase` (CRTP pattern) that wraps a pair of Unix file descriptors for bidirectional string message passing. | Member | Description | |---|---| -| `PipeChannel(table_name, read_pipe_fd, write_pipe_fd, remote_pid)` | Constructs a channel with dedicated read/write pipe FDs and the remote process PID | -| `~PipeChannel()` | Automatically closes both pipe FDs on destruction | +| `PipeChannel(table_name, read_pipe_fd, write_pipe_fd, remote_pid)` | Constructs a channel from pre-opened pipe FDs and the remote process PID | +| `~PipeChannel()` | Closes both pipe file descriptors on destruction | | `getRemotePid()` | Returns the PID of the remote process on the other end of the pipe | -| `sendStringMessageImpl()` | CRTP-dispatched implementation for writing a string message to the write pipe | -| `recvStringMessageImpl()` | CRTP-dispatched implementation for reading a string message from the read pipe | -| `blockSIGPIPE()` / `restoreSIGPIPE()` | Signal mask helpers that suppress `SIGPIPE` during writes to detect broken pipes gracefully via return codes | +| `sendStringMessageImpl()` | CRTP-dispatched implementation for writing a string over the write FD | +| `recvStringMessageImpl()` | CRTP-dispatched implementation for reading a string from the read FD | +| `blockSIGPIPE()` / `restoreSIGPIPE()` | Guards writes against `SIGPIPE` signals when the remote end closes | -- **`PipeChannelFactory`** β€” Forward-declared factory (defined elsewhere) responsible for constructing `PipeChannel` instances. +**`PipeChannelFactory`** β€” Forward-declared factory class responsible for constructing `PipeChannel` instances. ## Usage Example -```cpp +```c // Typically constructed via PipeChannelFactory, not directly. -// Conceptual usage after channel setup: -PipeChannel channel("processes_table", read_fd, write_fd, child_pid); +// Example of direct construction (e.g., in worker process setup): -std::string response; -Status recv_status = channel.recvStringMessage(response); +int read_fd = pipe_fds[0]; +int write_fd = pipe_fds[1]; +pid_t worker_pid = fork(); + +PipeChannel channel("my_table", read_fd, write_fd, worker_pid); -if (!recv_status.ok()) { - LOG(ERROR) << "IPC recv failed: " << recv_status.getMessage(); -} +// Send a serialized message to the remote process +Status s = channel.sendStringMessage("{\"query\": \"SELECT * FROM users\"}"); + +// Receive a response +std::string response; +Status r = channel.recvStringMessage(response); -// Channel FDs are closed automatically when channel goes out of scope +pid_t remote = channel.getRemotePid(); // retrieve remote PID if needed ``` -> **Note:** `PipeChannel` is non-default-constructible by design. All instances must be created with explicit pipe FDs and a remote PID, enforcing valid channel state at construction time. \ No newline at end of file +> **Note:** Default construction is disabled (`= delete`). Channels must be created with explicit pipe FDs and a valid remote PID. \ No newline at end of file diff --git a/osquery/worker/ipc/posix/.pipe_channel_factory.md b/osquery/worker/ipc/posix/.pipe_channel_factory.md index a827d7dccdf..dfec997bebd 100644 --- a/osquery/worker/ipc/posix/.pipe_channel_factory.md +++ b/osquery/worker/ipc/posix/.pipe_channel_factory.md @@ -1,46 +1,51 @@ -Manages creation and lifecycle of POSIX pipe-based IPC channels used for inter-process communication between parent and child worker processes in osquery's table extension system. +Factory and ticket management classes for creating POSIX pipe-based IPC channels between parent and child worker processes in osquery's table extension system. ## Key Components ### `PipeChannelTicket` -A move-only RAII wrapper that holds pre-allocated pipe file descriptors before a channel is fully constructed. Ensures proper cleanup of unused file descriptors and prevents double-use via a `used_` flag. +A move-only RAII wrapper that holds pre-created pipe file descriptors before they are consumed by a channel. Ensures safe ownership transfer of raw pipe FDs across process boundaries. -| Method | Description | -|---|---| -| `getAndUseReadFd(index)` | Transfers ownership of a read-end fd (marks it consumed) | -| `getAndUseWriteFd(index)` | Transfers ownership of a write-end fd (marks it consumed) | -| `~PipeChannelTicket()` | Closes any unclaimed file descriptors | +- `getAndUseReadFd(index)` β€” Transfers ownership of a read-end FD (index `0` or `1`); returns `-1` if already consumed +- `getAndUseWriteFd(index)` β€” Transfers ownership of a write-end FD; same single-use semantics +- Destructor auto-closes any unconsumed FDs to prevent leaks +- Non-copyable; move operations throw `std::logic_error` if the ticket was already used ### `PipeChannelFactory` -Concrete factory extending `TableChannelFactoryBase` that creates `PipeChannel` instances for parent/child process pairs. +Concrete factory extending `TableChannelFactoryBase` (CRTP) that creates `PipeChannel` instances for parent/child IPC. | Method | Description | -|---|---| -| `createChannelTicket()` | Allocates pipe fd pairs; returns a `PipeChannelTicket` | -| `createChildChannel(name, ticket)` | Builds the child-side `PipeChannel` from a ticket | -| `createParentChannel(name, ticket, pid)` | Builds the parent-side `PipeChannel`, tracking child PID | -| `getTableNameFromPid(pid)` | Looks up the table name associated with a child PID | +|--------|-------------| +| `createChannelTicket()` | Allocates two pipe pairs and returns a `PipeChannelTicket` | +| `createChildChannel(name, ticket)` | Builds a `PipeChannel` from the child process perspective | +| `createParentChannel(name, ticket, pid)` | Builds a `PipeChannel` from the parent process perspective, associating the child PID | +| `getTableNameFromPid(pid)` | Reverse-lookup of a table name by child process PID | -### `GetChannelType` (template specialization) -Maps `PipeChannelFactory` β†’ `PipeChannel` for compile-time type resolution in the factory base. +### Template Specialization +```cpp +template <> +struct GetChannelType { + using Channel = PipeChannel; +}; +``` +Wires `PipeChannelFactory` into the CRTP factory type system. ## Usage Example ```cpp PipeChannelFactory factory; -// Before fork: create the ticket (pre-allocates pipe fds) -auto ticket = factory.createChannelTicket(); +// Before fork(): create a ticket holding both pipe pairs +PipeChannelTicket ticket = factory.createChannelTicket(); -pid_t child_pid = fork(); -if (child_pid == 0) { - // Child process - factory.createChildChannel("processes", std::move(ticket)); +pid_t pid = fork(); +if (pid == 0) { + // Child process + PipeChannel& child_ch = factory.createChildChannel("processes", std::move(ticket)); } else { - // Parent process - factory.createParentChannel("processes", std::move(ticket), child_pid); + // Parent process + PipeChannel& parent_ch = factory.createParentChannel("processes", std::move(ticket), pid); } ``` -> **Note:** `PipeChannelTicket` is move-only and cannot be reused after channel creation. Attempting to move a `used_` ticket throws `std::logic_error`. \ No newline at end of file +> **Note:** A `PipeChannelTicket` is single-use β€” consuming it twice via `getAndUse*Fd()` returns `-1`. Moving a used ticket throws `std::logic_error`. \ No newline at end of file diff --git a/osquery/worker/logging/glog/.glog_logger.md b/osquery/worker/logging/glog/.glog_logger.md index b0533b1b301..72c9a7dbcaf 100644 --- a/osquery/worker/logging/glog/.glog_logger.md +++ b/osquery/worker/logging/glog/.glog_logger.md @@ -1,14 +1,12 @@ -Defines `GLOGLogger`, a singleton logger implementation that bridges osquery's worker logging interface to the GLOG (Google Logging Library) backend. +Defines the `GLOGLogger` class, a singleton logger implementation that bridges osquery's worker logging interface to the GLOG (Google Logging Library) backend. ## Key Components -| Symbol | Type | Description | -|---|---|---| -| `GLOGLogger` | `class` | Final singleton class implementing the `Logger` interface using GLOG | -| `instance()` | `static method` | Returns the singleton `GLOGLogger` instance | -| `log()` | `method` | Logs a message at the specified severity level via GLOG | -| `vlog()` | `method` | Logs a verbose message at the specified severity level via GLOG | +- **`GLOGLogger`** β€” Final class inheriting from `Logger`; implements GLOG-backed logging for worker processes +- **`instance()`** β€” Static method returning the singleton `GLOGLogger` instance +- **`log(int severity, const std::string& message)`** β€” Logs a message at the specified severity level via GLOG +- **`vlog(int severity, const std::string& message)`** β€” Logs a verbose message at the specified verbosity level via GLOG ## Usage Example @@ -16,18 +14,18 @@ Defines `GLOGLogger`, a singleton logger implementation that bridges osquery's w #include // Retrieve the singleton instance and log a message -osquery::GLOGLogger& logger = osquery::GLOGLogger::instance(); +auto& logger = osquery::GLOGLogger::instance(); // Standard severity log (e.g., INFO=0, WARNING=1, ERROR=2) logger.log(0, "Worker process started successfully"); -// Verbose log for debug-level output -logger.vlog(1, "Processing query batch"); +// Verbose log with verbosity level +logger.vlog(1, "Detailed diagnostic message"); ``` ## Notes -- Inherits from `osquery::Logger` (defined in `osquery/worker/logging/logger.h`) -- Marked `final` β€” not intended for further subclassing -- Singleton pattern ensures a single shared GLOG logging instance across worker processes -- Severity integers map to standard GLOG severity levels \ No newline at end of file +- Declared `final` β€” not intended for further subclassing +- Follows the singleton pattern; do not instantiate directly +- Severity levels follow GLOG conventions (`google::INFO`, `google::WARNING`, etc.) +- Part of the `osquery` namespace within the worker logging subsystem \ No newline at end of file diff --git a/plugins/config/.tls_config.md b/plugins/config/.tls_config.md index a6805e17c60..cf607177bce 100644 --- a/plugins/config/.tls_config.md +++ b/plugins/config/.tls_config.md @@ -1,39 +1,39 @@ -Declares the `TLSConfigPlugin` class, which retrieves osquery configuration over TLS from a remote endpoint. +Defines the `TLSConfigPlugin` class, which retrieves osquery configuration over a TLS-secured HTTP connection from a remote endpoint. ## Key Components ### `TLSConfigPlugin` -Inherits from `ConfigPlugin` and `std::enable_shared_from_this`. Implements TLS-based remote config fetching. +Inherits from `ConfigPlugin` and `std::enable_shared_from_this`. Implements the config plugin interface for TLS-based remote configuration retrieval. | Member | Type | Description | |--------|------|-------------| -| `setUp()` | `Status` | Initializes the plugin, validates and caches the remote TLS config URL | -| `genConfig()` | `Status` | Fetches and populates the config map from the remote TLS endpoint | -| `uri_` | `std::string` (protected) | Cached URL computed once during setup | +| `setUp()` | `Status` | Initializes the plugin, validates and caches the remote endpoint URI | +| `genConfig()` | `Status` | Fetches configuration from the TLS endpoint and populates the config map | +| `uri_` | `std::string` (protected) | Cached remote endpoint URL, computed once during `setUp()` | ## Usage Example -```cpp -// Register and use TLSConfigPlugin via the osquery plugin system -auto plugin = std::make_shared(); +```c +// Plugin is registered and invoked via the osquery config subsystem. +// Direct instantiation is typically handled by the plugin registry. + +auto plugin = std::make_shared(); Status s = plugin->setUp(); if (!s.ok()) { - LOG(ERROR) << "TLS config setup failed: " << s.getMessage(); + LOG(ERROR) << "TLS config setup failed: " << s.getMessage(); } std::map config; Status result = plugin->genConfig(config); if (result.ok()) { - for (const auto& [source, json] : config) { - // Process each config source JSON blob - } + // config map now contains key/value pairs from the remote TLS endpoint } ``` ## Notes -- The `uri_` field is computed once in `setUp()` and reused across `genConfig()` calls to avoid redundant URL resolution. -- `TLSConfigTests` is declared a `friend` class for unit test access to internals. -- Typically registered via osquery's plugin registry rather than instantiated directly. \ No newline at end of file +- The `uri_` field is computed and cached during `setUp()` to avoid redundant URL resolution on repeated `genConfig()` calls. +- `TLSConfigTests` is declared as a `friend` class to allow white-box unit testing of private internals. +- Defined within the `osquery` namespace; register via osquery's plugin registry rather than instantiating directly. \ No newline at end of file diff --git a/plugins/config/parsers/.auto_constructed_tables.md b/plugins/config/parsers/.auto_constructed_tables.md index b338c64b10c..cec1f3a9427 100644 --- a/plugins/config/parsers/.auto_constructed_tables.md +++ b/plugins/config/parsers/.auto_constructed_tables.md @@ -1,45 +1,50 @@ -Defines the plugin classes for osquery's Auto Table Construction (ATC) system, which dynamically generates queryable SQL tables from user-defined configuration. +Declares the ATC (Auto Table Construction) plugin classes that enable dynamic osquery table creation from SQLite database files via configuration. ## Key Components ### `ATCPlugin` (extends `TablePlugin`) -Represents a dynamically constructed table backed by a SQLite query against an external database path. +Represents a dynamically constructed table backed by an external SQLite database. | Member | Description | -|---|---| -| `ATCPlugin(path, tc_columns, sqlite_query)` | Constructor; initializes table in `PENDING` state | +|--------|-------------| +| `ATCPlugin(path, tc_columns, sqlite_query)` | Constructs plugin with DB path, column schema, and query | | `generate(context)` | Executes the configured SQLite query and returns rows | -| `setActive()` | Transitions the table out of `PENDING` state, marking it ready for queries | -| `attributes()` | Returns current `TableAttributes` (e.g., `PENDING` vs active) | +| `setActive()` | Transitions table out of `PENDING` state, signaling it is ready for queries | +| `attributes()` | Returns current `TableAttributes` (e.g., `PENDING` or active) | +| `columnDefinition()` | Returns SQL column definition string | -> **Note:** The `PENDING` state guards against race conditions when the SQL database is reinitialized between table registration and attachment. +> The `PENDING` state prevents race conditions when the SQL database is reinitialized between table registration and attachment. ### `ATCConfigParserPlugin` (extends `ConfigParserPlugin`) -Parses the `auto_table_construction` section of the osquery config, manages registration and teardown of ATC tables. +Parses the `auto_table_construction` config section and manages the lifecycle of registered ATC tables. | Member | Description | -|---|---| -| `keys()` | Returns `{"auto_table_construction"}` as the config key | +|--------|-------------| +| `keys()` | Returns `{"auto_table_construction"}` as the handled config key | | `setUp()` | Initializes the parser and restores previously registered ATC tables | -| `update(source, config)` | Processes config changes β€” registers new tables and removes stale ones | -| `removeATCTables(tables)` | Deregisters a set of ATC tables | -| `registeredATCTables()` | Returns the set of currently active ATC table names from the database | +| `update(source, config)` | Diffs config changes, removing stale tables and registering new ones | +| `removeATCTables(tables)` | Unregisters a set of ATC tables by name | +| `registeredATCTables()` | Returns the set of currently registered ATC table names (from DB with prefix `atc.`) | ## Usage Example -ATC tables are declared in the osquery config: - -```json -{ - "auto_table_construction": { - "my_custom_table": { - "query": "SELECT * FROM my_db_table", - "path": "/path/to/database.db", - "columns": ["col1", "col2"] - } - } -} -``` - -At runtime, `ATCConfigParserPlugin` parses this config and instantiates an `ATCPlugin` per entry, which is then registered and attached as a queryable osquery table. \ No newline at end of file +```c +// ATC tables are driven by osquery config β€” example config JSON: +// { +// "auto_table_construction": { +// "my_custom_table": { +// "query": "SELECT name, value FROM settings", +// "path": "/var/db/myapp.db", +// "columns": ["name", "value"] +// } +// } +// } + +// ATCConfigParserPlugin picks up the config and registers ATCPlugin instances: +ATCPlugin plugin("/var/db/myapp.db", columns, "SELECT name, value FROM settings"); +plugin.setActive(); // Mark ready after SQL attachment + +// Query via standard osquery SQL: +// SELECT * FROM my_custom_table; +``` \ No newline at end of file diff --git a/plugins/config/parsers/.decorators.md b/plugins/config/parsers/.decorators.md index e6398d8ce3b..b06f4362b1f 100644 --- a/plugins/config/parsers/.decorators.md +++ b/plugins/config/parsers/.decorators.md @@ -1,46 +1,40 @@ -Defines the decorator subsystem for osquery, which enriches log items with additional key-value metadata before they are dispatched to downstream logging APIs. +Defines the decoration system for osquery, which attaches key-value metadata to log items before they are dispatched to downstream logging APIs. ## Key Components -### `DecorationPoint` Enum -Specifies when decorators should execute: +- **`DecorationPoint`** β€” Enum specifying when decorators execute: + - `DECORATE_LOAD` β€” once at config load + - `DECORATE_ALWAYS` β€” on every query run + - `DECORATE_INTERVAL` β€” on a timed interval -| Value | Description | -|---|---| -| `DECORATE_LOAD` | Run once on configuration load | -| `DECORATE_ALWAYS` | Run on every log event | -| `DECORATE_INTERVAL` | Run at a specified time interval | +- **`kDecorationPointKeys`** β€” Constant map linking each `DecorationPoint` to its configuration key string. -### `kDecorationPointKeys` -A constant map linking each `DecorationPoint` enum value to its corresponding configuration key string. +- **`runDecorators()`** β€” Iterates all configured decorators for a given decoration point. Accepts an optional timestamp (for interval-based points) and an optional source filter to restrict execution to a specific config source. -### Functions +- **`getDecorations()`** β€” Populates an output map with the current set of accumulated decoration column/value pairs, ready to be merged into log entries. -| Function | Description | -|---|---| -| `runDecorators(point, time, source)` | Iterates and executes all discovered decorators for the given `DecorationPoint`. Accepts an optional timestamp (for interval-based points) and an optional source filter. | -| `getDecorations(results)` | Populates `results` with the current set of accumulated decoration key-value pairs. | -| `clearDecorations(source)` | Removes all stored decorations associated with a given config source, typically called when that source updates. | +- **`clearDecorations()`** β€” Removes stored decorations for a given config source, typically called when that source is updated. ## Usage Example ```c -#include +#include -// Execute all "always" decorators +// Run decorators that should execute on every query osquery::runDecorators(osquery::DECORATE_ALWAYS); -// Execute interval decorators at a specific unix timestamp -osquery::runDecorators(osquery::DECORATE_INTERVAL, 1700000000ULL); +// Run interval decorators with a specific timestamp and source +osquery::runDecorators(osquery::DECORATE_INTERVAL, currentTime, "config_source"); -// Retrieve accumulated decorations and apply to a log record +// Retrieve accumulated decorations and merge into a log record std::map decorations; osquery::getDecorations(decorations); + for (const auto& kv : decorations) { logRecord[kv.first] = kv.second; } // Clear decorations when a config source is refreshed -osquery::clearDecorations("filesystem"); +osquery::clearDecorations("config_source"); ``` \ No newline at end of file diff --git a/plugins/config/parsers/.feature_vectors.md b/plugins/config/parsers/.feature_vectors.md index c3dcda207a4..83772ec4aeb 100644 --- a/plugins/config/parsers/.feature_vectors.md +++ b/plugins/config/parsers/.feature_vectors.md @@ -1,27 +1,31 @@ -A `ConfigParserPlugin` implementation that parses and stores feature vector configuration keys from the osquery config system. +Parses and stores feature vector configuration keys from osquery's config system, providing a plugin interface for feature vector dictionary management. ## Key Components -- **`kFeatureVectorsRootKey`** β€” External constant defining the root key used to locate feature vector data within the config tree. -- **`FeatureVectorsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass responsible for extracting feature vector dictionary keys from the osquery configuration. - - **`keys()`** β€” Returns the list of top-level config keys this parser handles. - - **`update()`** β€” Called when the configuration is loaded or refreshed; processes the parsed config data for the given source. +- **`kFeatureVectorsRootKey`** β€” External constant string defining the root configuration key for feature vectors +- **`FeatureVectorsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass that handles feature vector config sections + +### Methods + +| Method | Description | +|--------|-------------| +| `keys()` | Returns the list of configuration keys this parser handles | +| `update(source, config)` | Called when config is updated; processes feature vector entries from the given source | ## Usage Example -```cpp -// Typically registered automatically via osquery's plugin system. -// Access parsed feature vector data through the Config interface: - -auto parser = Config::getParser("feature_vectors"); -if (parser != nullptr) { - const auto& doc = parser->getData(); - auto it = doc.doc().FindMember(kFeatureVectorsRootKey.c_str()); - if (it != doc.doc().MemberEnd()) { - // Iterate over feature vector keys - } -} +```c +// Plugin is typically registered and invoked automatically by the config system. +// Direct usage example: + +auto parser = std::make_shared(); + +// Retrieve the keys this parser manages +auto managed_keys = parser->keys(); + +// Access parsed data via the root key +auto& data = Config::get().getParser(kFeatureVectorsRootKey); ``` -> **Note:** This parser is registered as part of osquery's config subsystem and is invoked automatically during config load/refresh cycles. Direct instantiation is not typically required β€” rely on `Config::getParser()` to retrieve the active instance. \ No newline at end of file +> The plugin integrates with osquery's `ConfigParserPlugin` framework β€” it is registered automatically and invoked during config load/reload cycles. Direct instantiation is generally unnecessary outside of testing. \ No newline at end of file diff --git a/plugins/config/parsers/.kafka_topics.md b/plugins/config/parsers/.kafka_topics.md index f5fc4d378e7..5b3a9392b47 100644 --- a/plugins/config/parsers/.kafka_topics.md +++ b/plugins/config/parsers/.kafka_topics.md @@ -1,12 +1,12 @@ -Declares the `KafkaTopicsConfigParserPlugin` class and associated constants for parsing Kafka topic configurations from osquery's configuration system. +Declares the `KafkaTopicsConfigParserPlugin` class and its associated root key constant for parsing Kafka topic configurations from the osquery configuration system. ## Key Components -- **`kKafkaTopicParserRootKey`** β€” Extern constant string defining the root key used to retrieve Kafka topic configurations from the osquery config. -- **`KafkaTopicsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass responsible for extracting and updating Kafka topic configurations. - - `keys()` β€” Returns the list of configuration keys this parser handles. - - `update(source, config)` β€” Processes and applies updated Kafka topic configuration from a given source. +- **`kKafkaTopicParserRootKey`** β€” Extern constant string defining the root JSON key used to look up Kafka topic configuration entries. +- **`KafkaTopicsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass responsible for extracting and updating Kafka topic settings from osquery's config pipeline. + - `keys()` β€” Returns the list of config keys this parser handles. + - `update()` β€” Called by the config system when a config source is updated, allowing the plugin to re-parse Kafka topic data. ## Usage Example @@ -15,20 +15,21 @@ Declares the `KafkaTopicsConfigParserPlugin` class and associated constants for namespace osquery { -// Access the root key for Kafka topic config lookup +// Access the root key used for Kafka topic config lookup std::cout << kKafkaTopicParserRootKey << std::endl; -// Plugin is registered via osquery's config plugin system -// and invoked automatically on config updates. -// To retrieve parsed config data: -auto& config = Config::get(); -auto parser = config.getParser(kKafkaTopicParserRootKey); +// The plugin is typically registered automatically via the config system. +// To retrieve parsed Kafka topics after config load: +auto parser = Config::getParser(kKafkaTopicParserRootKey); if (parser != nullptr) { - const auto& doc = parser->getData(); - // Access parsed Kafka topic entries from doc + const auto& data = parser->getData(); + // Iterate over parsed Kafka topic entries + for (const auto& topic : data.get_child(kKafkaTopicParserRootKey)) { + std::cout << "Topic: " << topic.second.get_value() << std::endl; + } } } // namespace osquery ``` -> **Note:** `KafkaTopicsConfigParserPlugin` is intended to be registered with osquery's plugin registry. Direct instantiation is typically handled by the framework during config initialization. \ No newline at end of file +> **Note:** `KafkaTopicsConfigParserPlugin` integrates with osquery's plugin registration system. It is not typically instantiated directly β€” register it via `REGISTER_INTERNAL` and the config subsystem will invoke `update()` automatically when configurations change. \ No newline at end of file diff --git a/plugins/config/parsers/.logger.md b/plugins/config/parsers/.logger.md index b60609a6265..18982a80244 100644 --- a/plugins/config/parsers/.logger.md +++ b/plugins/config/parsers/.logger.md @@ -1,29 +1,28 @@ -Config parser plugin for osquery logger settings, exposing a `ConfigParserPlugin` implementation that reads and updates logger-related configuration keys. +A configuration parser plugin for handling logger-related settings within the osquery configuration system. ## Key Components -- **`LoggerConfigParserPlugin`** β€” Extends `ConfigParserPlugin` to handle the logger configuration section. - - **`keys()`** β€” Returns the list of config keys this parser handles (driven by `kLoggerKey`). - - **`update()`** β€” Called when the configuration source changes; applies updated logger settings. - - **`kLoggerKey`** *(private static)* β€” The string key identifying the logger section in the osquery config. +- **`LoggerConfigParserPlugin`** β€” Extends `ConfigParserPlugin` to parse and process the logger configuration section from osquery config sources. + - **`keys()`** β€” Returns the configuration keys this parser handles, specifically the logger key (`kLoggerKey`). + - **`update()`** β€” Called when configuration is loaded or refreshed; processes the logger config from the given source. + - **`kLoggerKey`** *(private static)* β€” The string key identifying the logger section in the config. ## Usage Example ```c -// The plugin is registered with osquery's config system and invoked automatically. -// When a config update is received containing the logger key, update() is triggered: +// The plugin is registered and used automatically by osquery's config system. +// When config is loaded, the update() method is invoked with logger settings: Status LoggerConfigParserPlugin::update(const std::string& source, const ParserConfig& config) { - // Access logger config section via kLoggerKey + // Access logger config entries via kLoggerKey auto logger_config = config.find(kLoggerKey); if (logger_config != config.end()) { - // Apply logger settings (e.g., log paths, verbosity) - data_[kLoggerKey] = logger_config->second; + // Apply logger settings from the parsed configuration } return Status::success(); } ``` -> **Note:** This parser is automatically discovered and invoked by osquery's `Config` subsystem β€” no manual instantiation is required. Register it via the standard plugin registration macros if extending. \ No newline at end of file +> This plugin integrates with osquery's `ConfigParserPlugin` registration system and is invoked automatically when configuration sources are loaded or updated β€” no manual instantiation is required. \ No newline at end of file diff --git a/plugins/config/parsers/.prometheus_targets.md b/plugins/config/parsers/.prometheus_targets.md index 098a50d8bb8..4e17f4e5a13 100644 --- a/plugins/config/parsers/.prometheus_targets.md +++ b/plugins/config/parsers/.prometheus_targets.md @@ -1,29 +1,33 @@ -Defines the `PrometheusMetricsConfigParserPlugin` class and associated constants for parsing Prometheus target configurations within the osquery config system. +Defines the `PrometheusMetricsConfigParserPlugin` class and associated constants for parsing Prometheus target configuration from osquery's config system. ## Key Components -- **`kPrometheusParserRootKey`** β€” External constant string defining the root configuration key used to identify Prometheus metrics entries in the osquery config. -- **`PrometheusMetricsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass responsible for extracting and updating Prometheus scrape target configuration from osquery's config sources. - - **`keys()`** β€” Returns the list of config keys this parser handles (typically wrapping `kPrometheusParserRootKey`). - - **`update()`** β€” Called by the config subsystem when a config source changes; processes the parsed Prometheus target data and stores it for use by the Prometheus metrics table. +- **`kPrometheusParserRootKey`** β€” External string constant representing the root config key used to identify Prometheus metrics configuration blocks +- **`PrometheusMetricsConfigParserPlugin`** β€” A `ConfigParserPlugin` subclass responsible for extracting and updating Prometheus scrape target settings from the osquery config + +### Methods + +| Method | Description | +|--------|-------------| +| `keys()` | Returns the list of config keys this parser handles | +| `update(source, config)` | Called by the config system when relevant config sections change | ## Usage Example ```cpp -// Plugin is registered automatically via osquery's plugin system. -// Access parsed Prometheus targets through the config after registration: - -auto& config = Config::get(); -config.parsers(); // triggers update() on all registered ConfigParserPlugins +// The plugin is typically registered via osquery's plugin registry, +// then invoked automatically during config updates. -// Retrieve parsed Prometheus config data using the root key: +// Retrieve parsed Prometheus config via the config parser interface: auto parser = Config::getParser(kPrometheusParserRootKey); if (parser != nullptr) { - const auto& data = parser->getData(); - // data contains the parsed Prometheus scrape targets - auto targets = data.get(kPrometheusParserRootKey, {}); + const auto& root = parser->getData().get_child(kPrometheusParserRootKey); + for (const auto& target : root) { + std::string url = target.second.get("url", ""); + // process each Prometheus target URL + } } ``` -> **Note:** This header is part of osquery's Prometheus metrics integration. The plugin is typically auto-registered at startup and interacts with the config subsystem rather than being instantiated directly by consumers. \ No newline at end of file +> **Note:** This header is part of the osquery Prometheus metrics integration. The plugin registers itself with the config subsystem and is invoked whenever the osquery configuration is loaded or refreshed β€” no manual instantiation is required in typical usage. \ No newline at end of file diff --git a/plugins/database/.rocksdb.md b/plugins/database/.rocksdb.md index 54d3288da3d..f8e2540c5a8 100644 --- a/plugins/database/.rocksdb.md +++ b/plugins/database/.rocksdb.md @@ -1,59 +1,50 @@ -RocksDB database plugin implementation for osquery, providing persistent key-value storage backed by RocksDB with column family support, corruption detection, and integrated logging via Glog. +RocksDB-backed database plugin implementation for osquery's persistent key-value storage layer, providing CRUD operations organized by named column family domains. ## Key Components ### `GlogRocksDBLogger` -Extends `rocksdb::Logger` to intercept internal RocksDB log events, forwarding warnings and errors to Glog rather than letting them fall through to stderr. +Custom RocksDB logger that intercepts internal RocksDB log events and forwards them to Glog instead of stderr, enabling corruption detection and unified log management. ### `RocksDBDatabasePlugin` -Implements the `DatabasePlugin` interface using RocksDB as the storage backend. Key method groups: - -| Category | Methods | -|---|---| -| **Lifecycle** | `setUp()`, `tearDown()`, `close()`, `flush()` | -| **Read** | `get()` (string & int overloads), `scan()` | -| **Write** | `put()` (string & int overloads), `putBatch()` | -| **Delete** | `remove()`, `removeRange()` | -| **Maintenance** | `compactFiles()`, `repairDB()` | -| **Corruption** | `setCorrupted()`, `isCorrupted()` | - -**Private members of note:** -- `db_` β€” raw RocksDB database handle -- `handles_` β€” column family handle pointers per domain -- `close_mutex_` β€” guards teardown against concurrent access -- `options_` β€” RocksDB connection/tuning options +Concrete implementation of `DatabasePlugin` backed by RocksDB. Manages the full database lifecycle including column families, compaction, and corruption recovery. -## Usage Example +**Public Methods:** -```cpp -#include "rocksdb.h" +| Method | Description | +|--------|-------------| +| `get(domain, key, value)` | Retrieve string or int value by domain + key | +| `put(domain, key, value)` | Store string or int value | +| `putBatch(domain, data)` | Bulk insert a list of string key-value pairs | +| `remove(domain, key)` | Delete a single key | +| `removeRange(domain, low, high)` | Delete all keys within a range | +| `scan(domain, results, prefix, max)` | List keys, optionally filtered by prefix | +| `setUp()` / `tearDown()` | Open and close the database lifecycle | + +**Private Helpers:** -osquery::RocksDBDatabasePlugin plugin; +- `getHandleForColumnFamily(cf)` β€” resolves a column family name to its RocksDB handle +- `compactFiles(domain)` β€” triggers per-domain compaction +- `repairDB()` β€” best-effort corruption repair +- `flush()` β€” flushes memtables and triggers compaction +- `setCorrupted()` / `isCorrupted()` β€” global corruption state flag -// Initialize the database -auto status = plugin.setUp(); -if (!status.ok()) { - LOG(ERROR) << "DB init failed: " << status.getMessage(); -} +## Usage Example -// Store a value -plugin.put("config", "refresh_interval", "60"); +```cpp +// Typical lifecycle managed via the plugin registry: +RocksDBDatabasePlugin db; +db.setUp(); -// Retrieve a value std::string value; -plugin.get("config", "refresh_interval", value); +db.put("config_data", "my_key", "my_value"); +db.get("config_data", "my_key", value); // value == "my_value" -// Scan keys with a prefix std::vector keys; -plugin.scan("config", keys, "refresh_", 100); - -// Batch write -osquery::DatabaseStringValueList batch = {{"key1", "val1"}, {"key2", "val2"}}; -plugin.putBatch("config", batch); +db.scan("config_data", keys, "", 100); -// Cleanup -plugin.tearDown(); +db.remove("config_data", "my_key"); +db.tearDown(); ``` -> **Note:** Corruption is tracked via a global indicator (`setCorrupted`/`isCorrupted`). If detected, `repairDB()` is invoked automatically during `setUp()` as a best-effort recovery step. \ No newline at end of file +> **Note:** Direct instantiation is uncommon. This plugin is typically registered with and invoked through osquery's database plugin registry, which abstracts the backing store. \ No newline at end of file diff --git a/plugins/database/.sqlite.md b/plugins/database/.sqlite.md index 50a8a9c00c3..b81fb877d5e 100644 --- a/plugins/database/.sqlite.md +++ b/plugins/database/.sqlite.md @@ -1,50 +1,51 @@ -Defines the `SQLiteDatabasePlugin` class, providing a SQLite-backed implementation of osquery's `DatabasePlugin` interface for persistent key-value storage. +SQLite-backed implementation of osquery's `DatabasePlugin` interface, providing persistent key-value storage for osquery's internal/core data using an embedded SQLite3 database. ## Key Components ### `SQLiteDatabasePlugin` -Concrete plugin class inheriting from `DatabasePlugin` that wraps a long-lived `sqlite3*` handle. Registered internally as the `"sqlite"` database provider. +Concrete class extending `DatabasePlugin` with full CRUD and lifecycle management: | Method | Description | -|---|---| -| `get()` | Retrieves a `string` or `int` value by domain/key | -| `put()` | Stores a `string` or `int` value by domain/key | -| `putBatch()` | Bulk-inserts a list of string key-value pairs | -| `remove()` | Deletes a single key from a domain | -| `removeRange()` | Deletes all keys within a lexicographic range | -| `scan()` | Lists keys in a domain, optionally filtered by prefix and count | -| `setUp()` | Opens and initializes the SQLite database file | -| `tearDown()` / `~SQLiteDatabasePlugin()` | Closes the database safely | +|--------|-------------| +| `get()` | Retrieve `string` or `int` value by domain + key | +| `put()` | Store `string` or `int` value by domain + key | +| `putBatch()` | Bulk-insert a list of string key-value pairs | +| `remove()` | Delete a single key from a domain | +| `removeRange()` | Delete all keys within a lexicographic range | +| `scan()` | List keys in a domain, with optional prefix filter and result cap | +| `setUp()` | Opens/initializes the SQLite database file | +| `tearDown()` / `~SQLiteDatabasePlugin()` | Closes and cleans up the database handle | ### Private Members -- `db_` β€” raw `sqlite3*` handle to the open database -- `close_mutex_` β€” `Mutex` guarding safe concurrent teardown -- `close()` β€” internal cleanup helper called by both destructor and `tearDown()` +- `sqlite3* db_` β€” Raw SQLite3 database handle (initialized to `nullptr`) +- `Mutex close_mutex_` β€” Guards safe concurrent teardown -### Registration +### Plugin Registration ```cpp REGISTER_INTERNAL(SQLiteDatabasePlugin, "database", "sqlite"); ``` -Registers the plugin as the internal `"sqlite"` database backend, activated via the `database_path` flag. +Registers the plugin as the internal `sqlite` provider under the `database` category. ## Usage Example ```cpp -// Instantiated and managed by the osquery plugin registry. -// Direct interaction is via the DatabasePlugin interface: +// Plugin is auto-registered; typical usage goes through the DatabasePlugin API: +auto db = RegistryFactory::get().plugin("database", "sqlite"); -std::string value; -auto s = db->get("config", "my_key", value); -if (s.ok()) { - // use value -} +// Store a value +db->put("configs", "refresh_interval", "60"); -db->put("config", "my_key", "my_value"); +// Retrieve a value +std::string value; +db->get("configs", "refresh_interval", value); +// Scan keys with prefix std::vector keys; -db->scan("config", keys, "prefix_", 100); +db->scan("configs", keys, "refresh_", 100); + +// Remove a key +db->remove("configs", "refresh_interval"); +``` -db->remove("config", "my_key"); -db->removeRange("config", "a", "z"); -``` \ No newline at end of file +> The database file path is controlled by the `--database_path` flag declared via `DECLARE_string(database_path)`. \ No newline at end of file diff --git a/plugins/logger/.aws_firehose.md b/plugins/logger/.aws_firehose.md index 75d5cf30a80..59933240cd4 100644 --- a/plugins/logger/.aws_firehose.md +++ b/plugins/logger/.aws_firehose.md @@ -1,49 +1,37 @@ -Header file defining the AWS Kinesis Firehose logger plugin and log forwarder for osquery, enabling buffered log streaming to AWS Firehose delivery streams. +Header defining the AWS Kinesis Firehose log forwarding plugin for osquery, providing classes to batch and deliver log records to a Firehose delivery stream. ## Key Components -### Type Alias: `IFirehoseLogForwarder` -Template instantiation of `AwsLogForwarder` parameterized with Firehose-specific AWS SDK types (`Record`, `FirehoseClient`, `PutRecordBatchOutcome`, `PutRecordBatchResponseEntry`). +### Type Alias +- **`IFirehoseLogForwarder`** β€” Templated alias for `AwsLogForwarder` specialised with Firehose-specific AWS SDK types (`Record`, `FirehoseClient`, `PutRecordBatchOutcome`, `PutRecordBatchResponseEntry`) -### Class: `FirehoseLogForwarder` -Concrete implementation of `IFirehoseLogForwarder` that handles the low-level Firehose batching and transmission logic. +### Classes -| Method | Purpose | -|---|---| -| `internalSetup()` | Initializes the Firehose client connection | -| `internalSend(batch)` | Transmits a batch of records to Firehose | -| `initializeRecord()` | Populates a Firehose `Record` from a byte buffer | -| `getMax*()` methods | Returns Firehose-specific limits (batch size, record count, retry config) | -| `appendNewlineSeparators()` | Controls newline insertion between log entries | -| `getFailedRecordCount()` / `getResult()` | Parses batch response for failures | +- **`FirehoseLogForwarder`** β€” Concrete forwarder implementing the Firehose-specific batching and send logic: + - `internalSetup()` β€” Initialises the Firehose client + - `internalSend(batch)` β€” Submits a batch via `PutRecordBatch` + - `initializeRecord()` β€” Populates a Firehose `Record` from a byte buffer + - Overrides for batch/record size limits, retry policy, and newline separator behaviour -### Class: `FirehoseLoggerPlugin` -osquery `LoggerPlugin` integration layer that owns a `FirehoseLogForwarder` instance and bridges osquery's logging API to Firehose. +- **`FirehoseLoggerPlugin`** β€” osquery `LoggerPlugin` entry point that owns a `FirehoseLogForwarder` and routes log strings and status lines into it: + - `setUp()` β€” Configures the plugin and spawns the forwarder + - `logString()` / `logStatus()` β€” Feed individual log lines and status entries to the forwarder + - `usesLogStatus()` β€” Returns `true`, enabling status log routing -| Method | Purpose | -|---|---| -| `setUp()` | Configures the plugin and forwarder | -| `logString()` | Forwards string log entries | -| `logStatus()` | Forwards structured status log lines | -| `usesLogStatus()` | Declares status log support (`true`) | - -### Flag: `aws_firehose_period` -Controls the flush interval (in seconds) for the log forwarder background thread. +### Flag +- **`aws_firehose_period`** β€” gflags `uint64` controlling the flush interval of the forwarder ## Usage Example -```cpp -// Registration is handled by osquery's plugin registry. -// Configure via osquery flags at startup: -// -// --logger_plugin=aws_firehose -// --aws_firehose_stream=my-delivery-stream -// --aws_firehose_period=10 -// --aws_region=us-east-1 - -// The plugin is instantiated internally; direct usage follows the LoggerPlugin interface: +```c +// Register and use via osquery's logger registry auto plugin = std::make_shared(); plugin->setUp(); -plugin->logString("{\"name\":\"osquery\",\"result\":{...}}"); + +// Log a raw string to the Firehose stream +plugin->logString(R"({"query":"users","host":"node-01"})"); + +// Log status messages (INFO/WARNING/ERROR) +plugin->logStatus(statusLogLines); ``` \ No newline at end of file diff --git a/plugins/logger/.aws_kinesis.md b/plugins/logger/.aws_kinesis.md index ecaf9dcd578..f47f83d904e 100644 --- a/plugins/logger/.aws_kinesis.md +++ b/plugins/logger/.aws_kinesis.md @@ -1,47 +1,43 @@ -Header file defining the AWS Kinesis log forwarding integration for osquery, providing classes to batch and send log records to an Amazon Kinesis stream. +Defines the AWS Kinesis log forwarding plugin for osquery, providing classes to batch and stream log records to an Amazon Kinesis stream. ## Key Components ### Type Alias -- **`IKinesisLogForwarder`** β€” Specialization of the generic `AwsLogForwarder` template, configured with Kinesis-specific types (`PutRecordsRequestEntry`, `KinesisClient`, `PutRecordsOutcome`, `PutRecordsResultEntry`) +- **`IKinesisLogForwarder`** β€” Specialization of the generic `AwsLogForwarder` template, parameterized with Kinesis-specific AWS SDK types (`PutRecordsRequestEntry`, `KinesisClient`, `PutRecordsOutcome`, `PutRecordsResultEntry`). -### `KinesisLogForwarder` -Concrete forwarder class that implements batched log delivery to Kinesis. Key overrides: +### Classes -| Method | Purpose | -|---|---| -| `internalSetup()` | Initializes the Kinesis client and stream configuration | -| `internalSend(batch)` | Sends a batch of records via `PutRecords` API | -| `initializeRecord()` | Populates a `PutRecordsRequestEntry` with a byte buffer | -| `getMax*()` methods | Enforce Kinesis service limits (batch size, record size, retry policy) | -| `getFailedRecordCount()` | Parses partial failures from the outcome | -| `appendNewlineSeparators()` | Controls newline formatting between records | +- **`KinesisLogForwarder`** β€” Concrete log forwarder that implements batching and sending logic for Kinesis. Key overrides: + - `internalSetup()` β€” Initializes the Kinesis client and partition key. + - `internalSend(batch)` β€” Submits a batch of records via `PutRecords`. + - `initializeRecord()` β€” Populates a `PutRecordsRequestEntry` with a byte buffer. + - Batch constraint methods: `getMaxBytesPerRecord()`, `getMaxRecordsPerBatch()`, `getMaxBytesPerBatch()`, `getMaxRetryCount()`, `getInitialRetryDelay()`. + - Holds a private `partition_key_` string (overridden by `aws_kinesis_random_partition_key` flag). -**Private member:** `partition_key_` β€” used for Kinesis stream sharding unless `aws_kinesis_random_partition_key` flag is set. - -### `KinesisLoggerPlugin` -osquery `LoggerPlugin` implementation that owns a `KinesisLogForwarder` and integrates with the plugin system. - -| Method | Purpose | -|---|---| -| `setUp()` | Bootstraps the forwarder | -| `logString()` | Forwards a raw string log entry | -| `logStatus()` | Forwards ERROR/WARNING/INFO status lines | -| `usesLogStatus()` | Declares status log support | +- **`KinesisLoggerPlugin`** β€” osquery `LoggerPlugin` that owns a `KinesisLogForwarder` instance and bridges the osquery logging API to Kinesis: + - `setUp()` β€” Instantiates and configures the forwarder. + - `logString()` β€” Forwards a raw log string. + - `logStatus()` β€” Forwards ERROR/WARNING/INFO status log lines. + - `usesLogStatus()` β€” Signals that status logs are supported. ### Flag -- **`aws_kinesis_period`** β€” `uint64` gflag controlling the forwarding interval +- **`aws_kinesis_period`** β€” Declared `uint64` gflag controlling the flush interval of the forwarder. ## Usage Example ```cpp -// Instantiated internally by the plugin system via KinesisLoggerPlugin::setUp() -auto forwarder = std::make_shared( - "kinesis-forwarder", - /*log_period=*/10, - /*max_lines=*/500, - /*endpoint_override=*/"", - region +// Registration happens via osquery plugin registry; direct usage example: +KinesisLogForwarder forwarder( + "my-stream", // Kinesis stream name + 10, // log_period (seconds) + 1000, // max_lines per batch + "", // no endpoint override + AWSRegion::US_EAST_1 ); + +Status s = forwarder.internalSetup(); +if (!s.ok()) { + LOG(ERROR) << "Kinesis setup failed: " << s.getMessage(); +} ``` \ No newline at end of file diff --git a/plugins/logger/.buffered.md b/plugins/logger/.buffered.md index b43f2776c88..00bf0f76b4d 100644 --- a/plugins/logger/.buffered.md +++ b/plugins/logger/.buffered.md @@ -1,31 +1,26 @@ -Defines the `BufferedLogForwarder` base class and `iterate` utility for osquery's buffered log forwarding system, providing reliable database-backed buffering of status and result logs with periodic flushing, backoff support, and utilization-aware iteration. +Provides a thread-safe, database-backed log buffering and forwarding base class for osquery logger plugins, along with a utility for rate-limited iteration over string collections. ## Key Components -### `iterate()` +### `iterate()` (inline function) +Iterates over a `std::vector`, applying a mutable predicate to each element. Automatically sleeps the thread for 20ms every 100 iterations to prevent CPU thrash. -A free function that iterates a `std::vector`, applying a predicate to each element. Automatically yields the thread (`20ms` sleep) every 100 iterations to prevent CPU thrash. +### `BufferedLogForwarder` (class) +An abstract `InternalRunnable` subclass that buffers result and status logs to a backing store (database), then periodically flushes them via a background thread. Supports exponential backoff on send failures. -### `BufferedLogForwarder` +**Key methods:** -An abstract `InternalRunnable` subclass managing the full lifecycle of buffered log forwarding: - -| Member | Description | -|---|---| -| `start()` | Runnable entry point; waits and flushes on schedule | -| `setUp()` | Initializes buffer count β€” **must** be called by subclasses | -| `logString()` | Writes a result string to the backing store | -| `logStatus()` | Decorates and writes status lines to the backing store | -| `send()` | **Pure virtual** β€” subclasses implement actual log delivery | -| `check()` | Scans buffered logs, sorts by type, forwards via `send()`, then purges | -| `purge()` | Drops oldest entries when buffer exceeds `buffered_log_max` | -| `backoffTick()` | Reduces exponential backoff period on each successful send | - -**Key constants:** - -- `kLogPeriod` β€” Default flush interval (seconds) -- `kMaxLogLines` β€” Maximum lines flushed per `check()` cycle +| Method | Description | +|--------|-------------| +| `start()` | Runs the flush loop at the configured `log_period_` interval | +| `setUp()` | Initializes buffer count; **must** be called by subclasses that override it | +| `logString()` | Buffers a result log string to the backing store | +| `logStatus()` | Decorates and buffers a vector of status log lines | +| `send()` *(pure virtual)* | Subclasses implement actual log delivery | +| `check()` | Scans buffered logs up to `max_log_lines_`, dispatches to `send()`, then calls `purge()` | +| `purge()` | Removes oldest buffered logs when the count exceeds `buffered_log_max` | +| `backoffTick()` | Decrements exponential backoff timers | ## Usage Example @@ -36,21 +31,23 @@ class MyLogForwarder : public BufferedLogForwarder { : BufferedLogForwarder("my_service", "mylogger", std::chrono::seconds(30), 500) {} - // Must call base setUp() Status setUp() override { - return BufferedLogForwarder::setUp(); + // Custom init (e.g., open remote connection) + return BufferedLogForwarder::setUp(); // MUST call base } protected: - // Implement actual delivery (e.g., HTTP, file, socket) Status send(std::vector& log_data, const std::string& log_type) override { - for (auto& line : log_data) { - sendToRemote(log_type, line); + for (auto& entry : log_data) { + // Forward entry to remote endpoint } return Status::success(); } }; + +// Dispatch the forwarder as a background service +Dispatcher::addService(std::make_shared()); ``` -> **Note:** Subclasses overriding `setUp()` **must** call `BufferedLogForwarder::setUp()` to correctly initialize the internal buffer count. \ No newline at end of file +> **Note:** Subclasses must implement `send()` and must call `BufferedLogForwarder::setUp()` if they override `setUp()`, otherwise buffer count initialization will be skipped. \ No newline at end of file diff --git a/plugins/logger/.filesystem_logger.md b/plugins/logger/.filesystem_logger.md index 8ef7b1db21c..73e878abbb5 100644 --- a/plugins/logger/.filesystem_logger.md +++ b/plugins/logger/.filesystem_logger.md @@ -1,37 +1,41 @@ -A logger plugin that writes osquery query results, snapshots, and status logs to the local filesystem using separate file paths for each log type. +Header file defining the `FilesystemLoggerPlugin` class, which implements filesystem-based logging for osquery by writing query results, snapshots, and status logs to separate files on disk. ## Key Components -| Component | Description | -|---|---| -| `FilesystemLoggerPlugin` | Main logger plugin class extending `LoggerPlugin` | -| `setUp()` | Initializes filesystem paths and prepares log destinations | -| `logString()` | Writes differential query results to a dedicated results file | -| `logSnapshot()` | Writes snapshot query results to a dedicated snapshot file | -| `logStatus()` | Forwards status/health logs to Glog | -| `init()` | Post-startup initialization; may return errors to allow Glog direct file writes | -| `logStringToFile()` | Internal helper that performs the actual file write operation | -| `pimpl_` | Pointer-to-implementation (PIMPL) pattern for private state encapsulation | +**`FilesystemLoggerPlugin`** β€” Extends `LoggerPlugin` with the following interface: + +| Method | Description | +|--------|-------------| +| `setUp()` | Initializes the plugin and opens log file handles | +| `logString(s)` | Writes differential query results to a dedicated results log path | +| `logSnapshot(s)` | Writes snapshot query results to a separate snapshot log path | +| `init(name, log)` | Post-startup initialization; uniquely may return an error to redirect Glog output to files | +| `logStatus(log)` | Forwards status/health logs to Glog | +| `logStringToFile(s, filename, empty)` | Internal helper that performs the actual file write | + +The class uses a **Pimpl idiom** (`struct impl` / `std::unique_ptr pimpl_`) to hide file handle state and reduce compile-time dependencies. + +The `REGISTER` macro at the bottom registers the plugin under the `"logger"` category with the key `"filesystem"`. ## Usage Example ```c -// Plugin is auto-registered via the REGISTER macro at load time: -// REGISTER(FilesystemLoggerPlugin, "logger", "filesystem"); +// Plugin is auto-registered via the REGISTER macro. +// Enable via osquery flag at runtime: +// --logger_plugin=filesystem +// --logger_path=/var/log/osquery -// Configure via osquery flags (e.g., osquery.flags or CLI): -// --logger_plugin=filesystem -// --logger_path=/var/log/osquery +// Internally, the plugin dispatches like this: +FilesystemLoggerPlugin plugin; +plugin.setUp(); -// The plugin then writes to separate files: -// /var/log/osquery/osqueryd.results.log <- logString() -// /var/log/osquery/osqueryd.snapshots.log <- logSnapshot() -// /var/log/osquery/osqueryd.INFO <- logStatus() via Glog -``` +// Differential query result +plugin.logString("{\"action\":\"added\",\"columns\":{\"pid\":\"1234\"}}"); -## Notes +// Snapshot result written to a separate file +plugin.logSnapshot("{\"snapshot\":[{\"pid\":\"1234\"}]}"); -- Registered globally under the `"logger"` registry with the key `"filesystem"` β€” no manual instantiation required. -- Uses the **PIMPL idiom** (`struct impl`) to hide implementation details and reduce compile-time dependencies. -- Unique among osquery loggers in that `init()` can intentionally return an error, delegating status log file management directly to Glog. \ No newline at end of file +// Status/health messages forwarded to Glog +plugin.logStatus({{StatusLogLine::WARNING, "low disk space", 42, "disk"}}); +``` \ No newline at end of file diff --git a/plugins/logger/.kafka_producer.md b/plugins/logger/.kafka_producer.md index 79718d53add..1b147f669f8 100644 --- a/plugins/logger/.kafka_producer.md +++ b/plugins/logger/.kafka_producer.md @@ -1,40 +1,50 @@ -Declares the `KafkaProducerPlugin` class, which integrates osquery's logging pipeline with Apache Kafka by implementing both `LoggerPlugin` and `InternalRunnable` interfaces via librdkafka. +A logger plugin that integrates osquery with Apache Kafka, enabling log payloads to be published to Kafka topics via librdkafka. ## Key Components -| Symbol | Type | Description | -|--------|------|-------------| -| `kKafkaBaseTopic` | `extern const std::string` | Default Kafka topic used when a payload's `name` field has no configured mapping | -| `getMsgName()` | Free function | Extracts the `"name"` field from a JSON log payload string | -| `KafkaProducerPlugin` | Class | Core plugin β€” produces log messages to Kafka brokers, manages topic routing, and runs a background poll loop | -| `logString()` | Method | Serializes a log string and produces it to the appropriate Kafka topic | -| `init()` | Method | Configures librdkafka producer, registers the OS hostname as `client.id` | -| `start()` / `stop()` | Methods | `InternalRunnable` lifecycle β€” poll loop entry point and graceful flush/shutdown | -| `publishMsg()` | Protected virtual | Low-level publish to a specific `rd_kafka_topic_t*` | -| `flushMessages()` | Protected virtual | Mutex-guarded `rd_kafka_flush` with a 3-second timeout | -| `pollKafka()` | Protected virtual | Mutex-guarded `rd_kafka_poll` to drive delivery callbacks | -| `queryToTopics_` | `std::map` | Routes query names to their corresponding Kafka topic handles | +**Free Functions** +- `getMsgName(payload)` β€” Extracts the `"name"` field from a log payload string, used to route messages to the correct topic. + +**Constants** +- `kKafkaBaseTopic` β€” Default Kafka topic used when no matching topic is found for a payload name. + +**`KafkaProducerPlugin`** β€” Inherits from both `LoggerPlugin` and `InternalRunnable`. + +| Member | Description | +|---|---| +| `logString(s)` | Publishes a log string to Kafka and polls for delivery callbacks | +| `init(name, log)` | Configures the producer, sets hostname as `client.id`, and registers topics | +| `start()` | Background runnable loop; polls Kafka until interrupted | +| `stop()` | Flushes remaining messages with a 3-second timeout on shutdown | +| `publishMsg(topic, payload)` | Protected; sends a message to a specific `rd_kafka_topic_t` | +| `flushMessages()` | Protected; mutex-guarded wrapper around `rd_kafka_flush` | +| `pollKafka()` | Protected; mutex-guarded wrapper around `rd_kafka_poll` | +| `running_` | Atomic boolean tracking producer state | +| `queryToTopics_` | Maps query names to their corresponding Kafka topics | ## Usage Example ```cpp -// Registration via osquery plugin system (typical CMake/registry wiring) -REGISTER(KafkaProducerPlugin, "logger", "kafka_producer"); +#include "kafka_producer.h" + +// Plugin is registered and managed by osquery's plugin system. +// Direct instantiation example: +osquery::KafkaProducerPlugin producer; -// Retrieve and initialize the plugin (osquery runtime handles this internally) -auto plugin = std::make_shared(); -plugin->init("kafka_producer", {}); +// Initialization is called by the logger registry with configured flags: +// --logger_kafka_brokers, --logger_kafka_topic +producer.init("kafka_producer", {}); -// Log a JSON payload β€” topic is resolved from the payload's "name" field; -// falls back to kKafkaBaseTopic if no matching topic is configured. -Status s = plugin->logString(R"({"name":"pack_query","result":"..."})"); +// Dispatched as an InternalRunnable β€” polling runs in background thread. +// Log a query result payload: +osquery::Status s = producer.logString(R"({"name":"process_events","action":"exec"})"); if (!s.ok()) { - LOG(ERROR) << "Kafka publish failed: " << s.getMessage(); + LOG(ERROR) << "Kafka publish failed: " << s.getMessage(); } -// Background dispatcher thread calls start(); shutdown triggers stop() -// which flushes up to 3 seconds of buffered messages before exit. +// On shutdown: +producer.stop(); // flushes buffered messages (max 3s wait) ``` -> **Note:** `KafkaProducerPlugin` is non-copyable by design. Topic-to-query routing is configured via osquery flags consumed inside `configureTopics()` at `init()` time. \ No newline at end of file +> **Note:** `KafkaProducerPlugin` is non-copyable. Topic routing relies on `getMsgName()` matching keys in `queryToTopics_`; unmatched payloads fall back to `kKafkaBaseTopic`. \ No newline at end of file diff --git a/plugins/logger/.stdout.md b/plugins/logger/.stdout.md index 428dc446142..7f51d4fc92a 100644 --- a/plugins/logger/.stdout.md +++ b/plugins/logger/.stdout.md @@ -1,40 +1,29 @@ -Defines the `StdoutLoggerPlugin` class, an osquery logger plugin that outputs log messages directly to standard output (stdout). +Defines the `StdoutLoggerPlugin` class, an osquery logger plugin that writes log output directly to standard output (stdout). ## Key Components -### `StdoutLoggerPlugin` -A concrete implementation of `LoggerPlugin` that writes log output to stdout. - -| Member | Type | Description | -|---|---|---| -| `usesLogStatus()` | `bool` | Returns `true`, enabling status log handling | -| `logString()` | `Status` | Writes a plain string message to stdout | -| `init()` | `void` | Initializes the plugin with a name and buffered startup log lines | -| `logStatus()` | `Status` | Writes a batch of structured `StatusLogLine` entries to stdout | - -### Plugin Registration -```c -REGISTER(StdoutLoggerPlugin, "logger", "stdout"); -``` -Registers the plugin under the `"logger"` registry with the key `"stdout"`, making it selectable via osquery configuration. +- **`StdoutLoggerPlugin`** β€” Concrete implementation of `LoggerPlugin` that routes osquery log data to stdout +- **`usesLogStatus()`** β€” Returns `true`, indicating this plugin handles status log lines +- **`logString()`** β€” Writes a plain string log entry to stdout +- **`init()`** β€” Initializes the plugin with a name and any buffered startup log lines +- **`logStatus()`** β€” Writes structured `StatusLogLine` entries to stdout +- **`REGISTER` macro** β€” Registers the plugin under the `"logger"` category with the name `"stdout"` in the osquery plugin registry ## Usage Example -Enable the stdout logger in osquery configuration or via the `--logger_plugin` flag: - ```bash -osqueryi --logger_plugin=stdout +# Enable the stdout logger via osquery flag +osqueryd --logger_plugin=stdout ``` -Or in `osquery.conf`: +```cpp +// The plugin is auto-registered via the REGISTER macro. +// To resolve it manually from the registry: +auto plugin = RegistryFactory::get().plugin("logger", "stdout"); -```json -{ - "options": { - "logger_plugin": "stdout" - } -} +// Or configure via osquery flags at startup: +// --logger_plugin=stdout ``` -Log output will be streamed directly to stdout, useful for containerized environments or debugging where log forwarding is handled externally (e.g., Docker log drivers, systemd journal capture). \ No newline at end of file +> The `REGISTER` macro handles plugin registration at static initialization time, so no manual instantiation is needed. The plugin is available as `"stdout"` within the `"logger"` registry namespace. \ No newline at end of file diff --git a/plugins/logger/.syslog_logger.md b/plugins/logger/.syslog_logger.md index 9ccfb0a9dfd..b6c04fd098b 100644 --- a/plugins/logger/.syslog_logger.md +++ b/plugins/logger/.syslog_logger.md @@ -1,40 +1,39 @@ -Header defining the `SyslogLoggerPlugin` class, which implements osquery's logger plugin interface to route log output to the system syslog facility. +Defines the `SyslogLoggerPlugin` class, an osquery logger plugin that forwards query results and status messages to the system syslog facility. ## Key Components -### `SyslogLoggerPlugin` -Extends `LoggerPlugin` to provide syslog-based logging for osquery. +- **`SyslogLoggerPlugin`** β€” Extends `LoggerPlugin` to route osquery output through the POSIX `syslog` interface + - `usesLogStatus()` β€” Returns `true`, enabling status log forwarding to syslog + - `logString(const std::string& s)` β€” Writes a single log string entry to syslog + - `init(const std::string& name, const std::vector& log)` β€” Initializes the plugin and flushes any buffered startup logs + - `logStatus(const std::vector& log)` β€” Forwards a batch of status log lines to syslog -| Method | Description | -|---|---| -| `usesLogStatus()` | Returns `true`, enabling status log routing through this plugin | -| `logString(const std::string& s)` | Writes a single log string entry to syslog | -| `init(const std::string& name, const std::vector& log)` | Initializes the plugin and flushes any buffered startup logs | -| `logStatus(const std::vector& log)` | Writes a batch of status log lines to syslog | - -### Plugin Registration -```c -REGISTER(SyslogLoggerPlugin, "logger", "syslog"); -``` -Registers the plugin under the `"logger"` category with the key `"syslog"`, making it selectable via osquery configuration. +- **`REGISTER` macro** β€” Registers `SyslogLoggerPlugin` under the `"logger"` category with the name `"syslog"` in the osquery plugin registry ## Usage Example -Enable the syslog logger by setting the `logger_plugin` flag in your osquery configuration: +Enable the syslog logger by setting the `--logger_plugin` flag at osquery startup: ```bash osqueryd --logger_plugin=syslog ``` -Or in a config file: +To use programmatically within an osquery plugin context: + +```c +// The plugin is auto-registered via the REGISTER macro. +// To retrieve it from the registry at runtime: +auto plugin = Registry::getPlugin("logger", "syslog"); + +// Direct instantiation (typically handled by the registry): +SyslogLoggerPlugin logger; +logger.init("osqueryd", bufferedLogs); -```json -{ - "options": { - "logger_plugin": "syslog" - } +Status result = logger.logString("{\"name\":\"processes\",\"action\":\"added\"}"); +if (!result.ok()) { + // handle error } ``` -Once active, osquery query results and status messages are forwarded to the system syslog daemon (typically viewable via `journalctl` or `/var/log/syslog`), using the standard `` interface for log level mapping. \ No newline at end of file +Log output will appear in the system journal (e.g., `/var/log/syslog` or `journalctl`) under the osquery process name. \ No newline at end of file diff --git a/plugins/logger/.tls_logger.md b/plugins/logger/.tls_logger.md index 814e7188e70..39fffb4d4c8 100644 --- a/plugins/logger/.tls_logger.md +++ b/plugins/logger/.tls_logger.md @@ -1,51 +1,48 @@ -TLS logger plugin and forwarder declarations for buffering and transmitting osquery logs to a remote TLS endpoint via a background dispatcher thread. +TLS logger plugin and log forwarder declarations for osquery's buffered, TLS-based log shipping to a remote endpoint. ## Key Components -### `TLSLogForwarder` -Extends `BufferedLogForwarder` to flush database-buffered result and status logs to a TLS endpoint. Runs as a `Dispatcher` service when an enrollment key is present. +### `TLSLogForwarder` (extends `BufferedLogForwarder`) +A dispatcher service thread that flushes buffered result and status logs to a TLS endpoint. Started when an enrollment key is present at startup. | Member | Description | |---|---| -| `configuration_mutex` | Guards concurrent reads/writes to configuration state | -| `configuration_updated` | Flag signaling pending config changes | -| `updated_uri`, `updated_log_period`, etc. | Staged config values applied on next `applyNewConfiguration()` | -| `send()` | Transmits a batch of log strings to the TLS endpoint | -| `applyNewConfiguration()` | Applies staged configuration changes atomically | +| `configuration_mutex` | Guards concurrent configuration reads/writes | +| `configuration_updated` | Flag indicating pending config changes | +| `updated_uri`, `updated_log_period`, etc. | Staged config values applied on next cycle | +| `send()` | Transmits a batch of log lines to the TLS endpoint | +| `applyNewConfiguration()` | Applies staged configuration updates | | `uri_` | Active endpoint URI | -### `TLSLoggerPlugin` -Implements `LoggerPlugin` to receive, buffer, and forward osquery logs over TLS. +### `TLSLoggerPlugin` (extends `LoggerPlugin`) +The osquery logger plugin that bridges the osquery logging API to the TLS forwarder. | Method | Description | |---|---| -| `init()` | Buffers early status logs captured before logger initialization | -| `setUp()` | Validates node key and starts the forwarder dispatcher thread | -| `configure()` | Reacts to runtime configuration updates | -| `logString()` | Handles snapshot and event result logs | -| `logStatus()` | Handles `ERROR`/`WARNING`/`INFO` status logs | -| `usesLogStatus()` | Returns `true` β€” plugin consumes status log lines | +| `init()` | Receives early-startup buffered status logs and queues them | +| `setUp()` | Validates node key and spawns the `TLSLogForwarder` dispatcher thread | +| `configure()` | Reacts to runtime configuration changes | +| `logString()` | Forwards snapshot/event result strings to the buffer | +| `logStatus()` | Forwards ERROR/WARNING/INFO status lines to the buffer | +| `usesLogStatus()` | Returns `true`, enabling status log ingestion | ## Usage Example -```cpp -// Plugin is registered and managed by the osquery plugin system. -// Direct instantiation is typically handled by the registry: - -auto plugin = std::make_shared(); - -// Called by the framework at startup with early-buffered status logs: -plugin->init("tls", early_status_logs); - -// Called after node enrollment to start the forwarder thread: -Status s = plugin->setUp(); -if (!s.ok()) { - LOG(ERROR) << "TLS logger setup failed: " << s.getMessage(); +```c +// Registered automatically as an osquery logger plugin. +// Configured via osquery flags, e.g.: +// --logger_plugin=tls +// --logger_tls_endpoint=/api/v1/log +// --logger_tls_period=4 (flush interval in seconds) +// --logger_tls_max_lines=1024 (max lines per batch) + +// Runtime config update (e.g., from a config plugin callback): +{ + std::lock_guard lock(forwarder->configuration_mutex); + forwarder->updated_uri = "https://tls.example.com/log"; + forwarder->updated_log_period = std::chrono::seconds(10); + forwarder->configuration_updated = true; } - -// React to a config change (e.g., updated TLS endpoint URI): -plugin->configure(); -``` - -> **Note:** `TLSLogForwarder` configuration fields (`updated_uri`, `updated_log_period`, etc.) should be written under `configuration_mutex` before setting `configuration_updated = true` to safely stage changes for the next flush cycle. \ No newline at end of file +// TLSLogForwarder picks up changes on its next applyNewConfiguration() call. +``` \ No newline at end of file diff --git a/plugins/logger/.windows_event_log.md b/plugins/logger/.windows_event_log.md index ab21a48c8ad..a312dce80cf 100644 --- a/plugins/logger/.windows_event_log.md +++ b/plugins/logger/.windows_event_log.md @@ -1,48 +1,46 @@ -Windows Event Log logger plugin interface for osquery, providing integration with the Windows Event Log system via ETW (Event Tracing for Windows). +Windows Event Log logger plugin header for osquery, defining the `WindowsEventLoggerPlugin` class that routes osquery log output to the Windows Event Log system via ETW (Event Tracing for Windows). ## Key Components -### `WindowsEventLoggerPlugin` -Extends `LoggerPlugin` to route osquery logs to the Windows Event Log subsystem. +**`WindowsEventLoggerPlugin`** β€” Extends `LoggerPlugin` to implement Windows Event Log integration: | Member | Description | -|---|---| -| `usesLogStatus()` | Returns `true` β€” enables status log routing | -| `logString()` | Writes a plain string message to the event log | -| `init()` | Initializes the plugin with a name and initial log lines | +|--------|-------------| +| `usesLogStatus()` | Returns `true` to enable status log routing | +| `logString()` | Writes a raw string entry to the event log | +| `init()` | Initializes the plugin with a name and buffered startup logs | | `logStatus()` | Writes structured `StatusLogLine` entries to the event log | -| `acquireHandle()` | Registers the process as a Windows Event Log provider | -| `releaseHandle()` | Releases the ETW `REGHANDLE` provider registration | -| `emitLogRecord()` | Emits a single log record with severity, source file, and line number | +| `acquireHandle()` | Registers the process as a Windows ETW provider, populates a `REGHANDLE` | +| `releaseHandle()` | Unregisters the ETW provider and releases the handle | +| `emitLogRecord()` | Emits a single log event with severity, source file, and line number metadata | -**Private member:** `registration_handle_` β€” the `REGHANDLE` used for all ETW provider operations. +**`registration_handle_`** β€” Private `REGHANDLE` storing the active ETW provider registration. ## Usage Example ```cpp #include "windows_event_log.h" -osquery::WindowsEventLoggerPlugin plugin; +REGHANDLE handle{0}; -// Acquire an ETW provider handle -REGHANDLE handle; +// Register as an ETW provider auto status = osquery::WindowsEventLoggerPlugin::acquireHandle(handle); if (!status.ok()) { - // handle error + // handle registration failure } -// Emit a log record manually +// Emit a log record osquery::WindowsEventLoggerPlugin::emitLogRecord( handle, - "osquery service started", + "Scheduled task executed successfully", osquery::O_INFO, __FILE__, __LINE__ ); -// Release the handle when done +// Release when done osquery::WindowsEventLoggerPlugin::releaseHandle(handle); ``` -> **Note:** This header is Windows-only and depends on `evntprov.h` (Windows SDK). The static `acquireHandle` / `releaseHandle` / `emitLogRecord` methods can be used independently of the plugin lifecycle for direct ETW emission. \ No newline at end of file +> **Note:** This header is Windows-only. It depends on `` (ETW provider API) and is guarded accordingly via ``. \ No newline at end of file diff --git a/plugins/numeric_monitoring/.filesystem.md b/plugins/numeric_monitoring/.filesystem.md index 80f740c370c..4513c2b621d 100644 --- a/plugins/numeric_monitoring/.filesystem.md +++ b/plugins/numeric_monitoring/.filesystem.md @@ -1,45 +1,42 @@ -Filesystem-backed implementation of the `NumericMonitoringPlugin` interface that writes numeric monitoring data points as delimited text lines to a log file. +Filesystem-backed implementation of the `NumericMonitoringPlugin` interface that writes numeric monitoring data as delimited text lines to a log file. ## Key Components -### `NumericMonitoringFilesystemPlugin` -Extends `NumericMonitoringPlugin` to persist monitoring metrics to disk. +**`NumericMonitoringFilesystemPlugin`** β€” Concrete plugin class inheriting from `NumericMonitoringPlugin`. | Member | Description | |---|---| | `NumericMonitoringFilesystemPlugin()` | Default constructor using configured log file path | | `NumericMonitoringFilesystemPlugin(path)` | Constructor with explicit log file path override | -| `call(request, response)` | Handles incoming plugin requests and writes formatted metric lines | -| `setUp()` | Initializes the output file stream; must be called before `call()` | -| `isSetUp()` | Returns whether the plugin has been successfully initialized | -| `formTheLine(line, request)` | Private helper that formats a metric entry using `line_format_` and `separator_` | +| `call(request, response)` | Handles plugin dispatch; formats and writes a monitoring record | +| `setUp()` | Opens the output file stream; must succeed before `call()` is used | +| `isSetUp()` | Returns whether the file stream was successfully initialized | +| `formTheLine(line, request)` *(private)* | Formats a `PluginRequest` into a delimited string using `line_format_` and `separator_` | **Private fields:** -- `line_format_` β€” ordered list of fields to include per line -- `separator_` β€” single character delimiter between fields -- `log_file_path_` β€” resolved path to the output log file -- `output_file_stream_` β€” open file stream for writing -- `output_file_mutex_` β€” mutex guarding concurrent writes + +- `line_format_` β€” Ordered list of field keys to extract from the request +- `separator_` β€” Character delimiter between fields in each output line +- `log_file_path_` β€” Path to the target log file +- `output_file_stream_` β€” File stream for appending records +- `output_file_mutex_` β€” Mutex guarding concurrent writes ## Usage Example ```cpp -#include +#include "filesystem.h" -// Use explicit path (e.g., in tests or custom setups) -osquery::NumericMonitoringFilesystemPlugin plugin( - boost::filesystem::path("/var/log/osquery/numeric_monitoring.log")); +// Default path from osquery config +osquery::NumericMonitoringFilesystemPlugin plugin; -auto status = plugin.setUp(); -if (!plugin.isSetUp()) { - // handle initialization failure +osquery::Status status = plugin.setUp(); +if (!status.ok() || !plugin.isSetUp()) { + LOG(ERROR) << "Failed to initialize plugin: " << status.getMessage(); } -// Plugin is invoked via the registry; direct call example: -osquery::PluginRequest req = {{"path", "cpu.usage"}, {"value", "42.5"}}; -osquery::PluginResponse resp; -plugin.call(req, resp); -``` - -> **Note:** `setUp()` must succeed before invoking `call()`. The plugin is thread-safe for concurrent metric writes via `output_file_mutex_`. \ No newline at end of file +// Dispatch a monitoring record +osquery::PluginRequest request = {{"path", "cpu.usage"}, {"value", "42.5"}, {"unix_time", "1700000000"}}; +osquery::PluginResponse response; +plugin.call(request, response); +``` \ No newline at end of file diff --git a/plugins/remote/enroll/.tls_enroll.md b/plugins/remote/enroll/.tls_enroll.md index 8d8875c6c5a..72bf8c4a0ce 100644 --- a/plugins/remote/enroll/.tls_enroll.md +++ b/plugins/remote/enroll/.tls_enroll.md @@ -1,31 +1,25 @@ -TLS enrollment plugin that extends the base `EnrollPlugin` to handle node key acquisition and caching via TLS endpoints. +Declares the `TLSEnrollPlugin` class, which implements TLS-based node enrollment for osquery's remote enrollment system by requesting and caching enrollment keys from a TLS endpoint. ## Key Components -- **`TLSEnrollPlugin`** β€” Concrete implementation of `EnrollPlugin` for TLS-based enrollment - - `enroll()` β€” Returns a cached node key if available, otherwise triggers a new key request - - `requestKey(uri, node_key)` β€” Contacts the configured TLS endpoint to obtain a new enrollment key - - `TLSEnrollTests` β€” Declared as a friend class to allow unit test access to private members +- **`TLSEnrollPlugin`** β€” Extends `EnrollPlugin` to provide TLS-specific enrollment logic. + - **`enroll()`** β€” Returns a cached enrollment key if available; otherwise triggers a new key request. + - **`requestKey(uri, node_key)`** β€” Performs the actual HTTP/TLS call to the enrollment endpoint and populates the returned node key. + - **`TLSEnrollTests`** β€” Declared as a `friend` class to allow direct unit test access to private members. ## Usage Example -```c -// Plugin is registered and invoked through the osquery enrollment system. -// Direct instantiation is managed by the plugin registry. +```cpp +#include -// Enrollment is triggered automatically when the agent starts: -// 1. enroll() checks for a cached node_key -// 2. If none exists, requestKey() contacts the TLS endpoint -// 3. The returned key is cached for subsequent calls +// Register the plugin with the osquery registry +REGISTER(TLSEnrollPlugin, "enroll", "tls"); -// Configured via osquery flags, e.g.: -// --enroll_tls_endpoint=/api/v1/enroll -// --tls_hostname=fleet.example.com +// The plugin is invoked internally by the enrollment system. +// On first call, enroll() contacts the TLS endpoint: +// POST https:///enroll +// On subsequent calls, the cached node_key is returned directly. ``` -## Notes - -- Inherits enrollment caching logic from `EnrollPlugin` base class -- `requestKey()` returns a `Status` object β€” callers should check for success before using the populated `node_key` -- All members are private; interaction is through the plugin registry interface defined in `enroll.h` \ No newline at end of file +The plugin integrates with osquery's plugin registry under the `"enroll"` category. The `requestKey()` method handles the network round-trip, while `enroll()` acts as the cache-aware entry point β€” minimizing redundant enrollment requests across the agent lifecycle. \ No newline at end of file diff --git a/tests/.test_util.md b/tests/.test_util.md index f5eb91f435b..1c50caa5201 100644 --- a/tests/.test_util.md +++ b/tests/.test_util.md @@ -1,32 +1,33 @@ -Utility header providing shared test infrastructure for osquery unit tests and benchmarks, including process exit codes, path variables, and helper functions used across the test suite. +Shared test utilities and constants for osquery's test and benchmark infrastructure, providing initialization routines, path globals, process validation codes, and helper functions used across the test suite. ## Key Components -### Constants & Exit Codes +**Macros β€” Exit & Error Codes** -| Constant | Value | Purpose | +| Macro | Value | Purpose | |---|---|---| | `EXTENSION_SUCCESS_CODE` | `0x45` | Expected exit code for successful extension child process | | `WORKER_SUCCESS_CODE` | `0x57` | Expected exit code for successful worker child process | -| `ERROR_COMPARE_ARGUMENT` | `-1` | Child process argument comparison failure | +| `ERROR_COMPARE_ARGUMENT` | `-1` | Argument comparison failure | | `ERROR_LAUNCHER_PROCESS` | `-2` | Launcher process error | | `ERROR_LAUNCHER_MISMATCH` | `-5` | Launcher identity mismatch | -### Global Path Variables +**Global Variables** -- `kTestDataPath` β€” resolved path to test data (source or build directory) -- `kTestWorkingDirectory` β€” working directory for config, logs, and caching during tests -- `kFakeDirectory` β€” root of the fake filesystem tree (`fstree`) used in iterator tests -- `kProcessTestExecPath` β€” path to the currently executing test binary +- `kTestDataPath` β€” Resolved path to test data (source or build directory) +- `kTestWorkingDirectory` β€” Working directory for config, logs, and cache during tests +- `kFakeDirectory` β€” Root of the fake filesystem tree used in iterator tests +- `kProcessTestExecPath` β€” Path of the currently running test executable +- `kExpectedWorkerArgs` / `kExpectedExtensionArgs` β€” Expected argument arrays for child process validation -### Functions +**Functions** -- `initTesting()` β€” initializes osquery runtime for tests and benchmarks -- `shutdownTesting()` β€” tears down the osquery runtime after tests -- `getOsqueryScheduledQuery()` β€” returns a sample `ScheduledQuery` for test use -- `genRows(EventSubscriberPlugin*)` β€” materializes all rows from a generator-based event subscriber table -- `initUsersAndGroupsServices()` / `deinitUsersAndGroupsServices()` *(Windows only)* β€” manages Windows user/group service lifecycle in tests +- `initTesting()` β€” Bootstraps the test/benchmark environment +- `shutdownTesting()` β€” Tears down the test/benchmark environment +- `getOsqueryScheduledQuery()` β€” Returns a sample `ScheduledQuery` for use in tests +- `genRows(EventSubscriberPlugin*)` β€” Materializes all rows from a generator-based table subscriber +- `initUsersAndGroupsServices()` / `deinitUsersAndGroupsServices()` *(Windows only)* β€” Manages Windows user/group service lifecycle in tests ## Usage Example @@ -37,10 +38,10 @@ int main() { osquery::initTesting(); // Access resolved test data path - std::string dataPath = osquery::kTestDataPath; + std::string dataFile = osquery::kTestDataPath + "/sample.json"; - // Generate rows from an event subscriber - auto rows = osquery::genRows(mySubscriberPlugin); + // Validate a scheduled query helper + auto query = osquery::getOsqueryScheduledQuery(); osquery::shutdownTesting(); return 0; diff --git a/tests/integration/tables/.helper.md b/tests/integration/tables/.helper.md index 79409be36c4..4b0a416c21b 100644 --- a/tests/integration/tables/.helper.md +++ b/tests/integration/tables/.helper.md @@ -1,46 +1,41 @@ -Utility header for osquery table-level integration tests, providing validation primitives, type-flag constants, and helper functions used across SQL table test suites. +Utility header for osquery table tests, providing validation helpers, type-check flags, and query execution tools used across SQL table unit tests. ## Key Components ### Validator Classes - -| Class | Purpose | -|---|---| -| `IntMinMaxCheck` | Validates a string-encoded integer falls within `[min, max]` | -| `SpecificValuesCheck` | Validates a string belongs to a predefined allowlist | -| `CronValuesCheck` | Validates cron-style values β€” numeric range or special strings (e.g., `*`, `@reboot`) | +- **`IntMinMaxCheck`** β€” Checks if a string value parses as an integer within `[min, max]` +- **`SpecificValuesCheck`** β€” Verifies a string matches one of a fixed set of allowed values +- **`CronValuesCheck`** β€” Validates cron-style fields: numeric range or symbolic values (e.g. `"*"`, `"@reboot"`) ### Standalone Validators - -- `verifyIpAddress` β€” checks IPv4/IPv6 format -- `verifyEmptyStringOrIpAddress` β€” allows empty string or valid IP -- `verifyMacAddress` β€” validates MAC address format -- `verifyUidGid` β€” validates UNIX UID/GID values -- `is_valid_hex` β€” checks hexadecimal string format - -### Type Flag Enum - -Bitmask flags for column-level validation rules: - -```text -NormalType Β· IntType Β· NonEmpty Β· NonNull Β· NonZero -FileOnDisk Β· DirectoryOnDisk Β· ValidUUID Β· MD5 Β· SHA256 -SHA1 Β· Bool Β· EmptyOk Β· NullOk -Composites: NonNegativeInt Β· NonNegativeOrErrorInt Β· NonEmptyString Β· IntOrEmpty -``` - -### Core Test Helpers - -- `execute_query(query)` β€” runs a raw SQL query against the osquery SQLite engine -- `validate_row(row, map)` β€” validates a single result row against a `ValidationMap` -- `validate_rows(rows, map)` β€” validates all rows in a result set -- `validate_value_using_flags(value, flags)` β€” applies bitmask flag rules to a single value -- `validate_container_rows(table, map, constraints)` β€” validates a full table query with optional SQL constraints -- `setUpEnvironment()` β€” initializes the test environment before test execution - -### Type Aliases - +- **`verifyIpAddress`** β€” Returns `true` if value is a valid IPv4/IPv6 address +- **`verifyEmptyStringOrIpAddress`** β€” Accepts empty string or valid IP +- **`verifyMacAddress`** β€” Validates MAC address format +- **`verifyUidGid`** β€” Validates Unix UID/GID values + +### Validation Flags (bitmask enum) +Composable flags for common column constraints: + +| Flag | Meaning | +|------|---------| +| `IntType` | Must parse as integer | +| `NonEmpty` | Must not be empty string | +| `NonNull` | Must not be SQL NULL | +| `FileOnDisk` | Path must exist as a file | +| `ValidUUID` | Must be a valid UUID | +| `MD5` / `SHA256` / `SHA1` | Must match hash format | +| `NonNegativeInt` | Composite: int, non-empty, non-null | + +### Core Functions +- **`execute_query`** β€” Runs a SQL query against the osquery SQLite engine +- **`validate_row`** / **`validate_rows`** β€” Assert all columns in a row match their `ValidationMap` entries +- **`validate_value_using_flags`** β€” Applies bitmask flags to a single string value +- **`validate_container_rows`** β€” Validates a full table with optional SQL constraints +- **`is_valid_hex`** β€” Checks if a string is valid hexadecimal +- **`setUpEnvironment`** β€” Initializes the test environment + +### Types ```cpp using CustomCheckerType = std::function; using ValidationDataType = boost::variant; @@ -52,19 +47,18 @@ using ValidationMap = std::unordered_map; ```cpp #include "helper.h" -using namespace osquery::table_tests; +TEST_F(ProcessesTableTest, test_select_all) { + auto rows = execute_query("SELECT * FROM processes"); -TEST_F(MyTableTest, ValidateColumns) { ValidationMap row_map = { - {"pid", NonNegativeInt}, - {"name", NonEmptyString}, - {"path", FileOnDisk}, - {"checksum", SHA256}, - {"status", SpecificValuesCheck({"running", "stopped", "idle"})}, - {"port", IntMinMaxCheck(0, 65535)}, + {"pid", NonNegativeInt}, + {"name", NonEmptyString}, + {"path", FileOnDisk}, + {"state", SpecificValuesCheck({"R", "S", "D", "Z", "T"})}, + {"uid", verifyUidGid}, // CustomCheckerType + {"cpu_time", IntMinMaxCheck(0, INT64_MAX)}, }; - auto rows = execute_query("SELECT * FROM my_table"); validate_rows(rows, row_map); } ``` \ No newline at end of file