From 282125660e3356bdb9e73d216f888b7b9ec74904 Mon Sep 17 00:00:00 2001 From: Matteo Renoldi Date: Wed, 4 Feb 2026 18:24:10 +0100 Subject: [PATCH 1/3] docs: adjust the logo --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 4a3a69d..e84a61f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,7 +3,7 @@ Regression testing tool for modern data warehouses, powered by duck-db -![quack-diff logo](docs/images/quack_diff.jpg) +![quack-diff logo](images/quack_diff.jpg) ## Features From fb0ee21069e550889e8392af070a20e251fbb6cd Mon Sep 17 00:00:00 2001 From: Matteo Renoldi Date: Wed, 4 Feb 2026 18:26:12 +0100 Subject: [PATCH 2/3] docs: cleanup --- docs/api/sql_utils.md | 303 ------------------------------------------ docs/security.md | 265 ------------------------------------ 2 files changed, 568 deletions(-) delete mode 100644 docs/api/sql_utils.md delete mode 100644 docs/security.md diff --git a/docs/api/sql_utils.md b/docs/api/sql_utils.md deleted file mode 100644 index 0e95094..0000000 --- a/docs/api/sql_utils.md +++ /dev/null @@ -1,303 +0,0 @@ -# SQL Utilities API Reference - -## Module: `quack_diff.core.sql_utils` - -Provides SQL sanitization and security utilities to prevent SQL injection attacks. - -## Functions - -### `sanitize_identifier(identifier: str, max_length: int = 255) -> str` - -Sanitize a SQL identifier (table name, column name, database name, etc.). - -**Parameters:** - -- `identifier` (str): The identifier to sanitize -- `max_length` (int, optional): Maximum allowed length. Default: 255 - -**Returns:** - -- str: The validated identifier - -**Raises:** - -- `SQLInjectionError`: If the identifier contains unsafe characters or patterns -- `ValueError`: If the identifier is empty or too long - -**Example:** - -```python -from quack_diff.core.sql_utils import sanitize_identifier - -# Valid identifiers -sanitize_identifier("users") # Returns: "users" -sanitize_identifier("my_schema.my_table") # Returns: "my_schema.my_table" - -# Invalid identifiers (raises SQLInjectionError) -sanitize_identifier("users; DROP TABLE") # Raises: SQLInjectionError -``` - ---- - -### `quote_identifier(identifier: str) -> str` - -Quote a SQL identifier for safe use in queries. - -**Parameters:** - -- `identifier` (str): The identifier to quote - -**Returns:** - -- str: The quoted identifier - -**Raises:** - -- `SQLInjectionError`: If the identifier is unsafe - -**Example:** - -```python -from quack_diff.core.sql_utils import quote_identifier - -quote_identifier("table") # Returns: '"table"' -quote_identifier("schema.table") # Returns: '"schema"."table"' -``` - ---- - -### `sanitize_path(path: str) -> str` - -Sanitize a file path for use in SQL ATTACH statements. - -**Parameters:** - -- `path` (str): The file path to sanitize - -**Returns:** - -- str: The validated path - -**Raises:** - -- `SQLInjectionError`: If the path contains unsafe patterns -- `ValueError`: If the path is empty - -**Example:** - -```python -from quack_diff.core.sql_utils import sanitize_path - -sanitize_path("/path/to/database.duckdb") # Returns: "/path/to/database.duckdb" -sanitize_path("./data/mydb.duckdb") # Returns: "./data/mydb.duckdb" - -# Invalid paths (raises SQLInjectionError) -sanitize_path("/path; DROP TABLE") # Raises: SQLInjectionError -``` - ---- - -### `validate_limit(limit: int | None) -> int | None` - -Validate a LIMIT value for SQL queries. - -**Parameters:** - -- `limit` (int | None): The limit value to validate - -**Returns:** - -- int | None: The validated limit value - -**Raises:** - -- `ValueError`: If the limit is invalid (negative or zero) - -**Example:** - -```python -from quack_diff.core.sql_utils import validate_limit - -validate_limit(10) # Returns: 10 -validate_limit(None) # Returns: None -validate_limit(-1) # Raises: ValueError -``` - ---- - -### `build_parameterized_in_clause(values: list[Any]) -> tuple[str, list[Any]]` - -Build a parameterized IN clause for safe SQL queries. - -**Parameters:** - -- `values` (list): List of values for the IN clause - -**Returns:** - -- tuple[str, list]: Tuple of (placeholders_string, values_list) - -**Raises:** - -- `ValueError`: If values list is empty - -**Example:** - -```python -from quack_diff.core.sql_utils import build_parameterized_in_clause - -placeholders, params = build_parameterized_in_clause([1, 2, 3]) -# placeholders: "?, ?, ?" -# params: [1, 2, 3] - -query = f"SELECT * FROM users WHERE id IN ({placeholders})" -result = connector.execute(query, params) -``` - ---- - -### `escape_like_pattern(pattern: str) -> str` - -Escape special characters in a LIKE pattern. - -**Parameters:** - -- `pattern` (str): The pattern to escape - -**Returns:** - -- str: The escaped pattern - -**Example:** - -```python -from quack_diff.core.sql_utils import escape_like_pattern - -escape_like_pattern("test_pattern%") # Returns: "test\\_pattern\\%" -``` - ---- - -## Exception - -### `class SQLInjectionError(ValueError)` - -Raised when a potential SQL injection attempt is detected. - -**Inheritance:** `ValueError` → `SQLInjectionError` - -**Example:** - -```python -from quack_diff import SQLInjectionError -from quack_diff.core.sql_utils import sanitize_identifier - -try: - sanitize_identifier("users; DROP TABLE students") -except SQLInjectionError as e: - print(f"Security error: {e}") - # Output: Security error: Invalid identifier '...': contains SQL injection pattern -``` - ---- - -## Security Patterns Detected - -The sanitization functions detect and block common SQL injection patterns: - -### Statement Separators - -- `;` - Multiple statement execution - -### SQL Comments - -- `--` - Single-line comment -- `/* */` - Multi-line comment - -### SQL Keywords - -- `DROP`, `DELETE`, `INSERT`, `UPDATE` - Data modification -- `UNION` - Union injection -- `SELECT ... FROM` - Nested queries -- `EXEC`, `CREATE` - Administrative commands - -### Boolean Injection - -- `OR ... = ...` - OR-based injection -- `AND ... = ...` - AND-based injection - -### Invalid Characters - -- Special characters: `!@#$%^&*()` -- Quotes in unsafe contexts - ---- - -## Usage Examples - -### Basic Sanitization - -```python -from quack_diff import DuckDBConnector, SQLInjectionError - -connector = DuckDBConnector() - -# Safe: Sanitization happens automatically -try: - connector.attach_duckdb( - name="my_db", - path="/path/to/database.duckdb" - ) -except SQLInjectionError as e: - print(f"Invalid input: {e}") -``` - -### Query Building - -```python -from quack_diff.core.query_builder import QueryBuilder -from quack_diff import SQLInjectionError - -builder = QueryBuilder() - -try: - query = builder.build_hash_query( - table="my_schema.my_table", - columns=["id", "name", "email"], - key_column="id" - ) -except SQLInjectionError as e: - print(f"Invalid identifier: {e}") -``` - -### Manual Sanitization - -```python -from quack_diff.core.sql_utils import sanitize_identifier, sanitize_path - -# Validate user input before use -user_table_name = input("Enter table name: ") - -try: - safe_table = sanitize_identifier(user_table_name) - print(f"Using table: {safe_table}") -except SQLInjectionError as e: - print(f"Invalid table name: {e}") -``` - ---- - -## Best Practices - -1. **Always sanitize user input** - Never trust user-provided identifiers or paths -2. **Use parameterized queries** - When dealing with values, use parameterized queries -3. **Validate early** - Sanitize at the entry point of your application -4. **Handle errors gracefully** - Catch `SQLInjectionError` and provide clear feedback -5. **Use automatic sanitization** - The core modules handle sanitization automatically - ---- - -## See Also - -- [Security Guide](../security.md) - Comprehensive security documentation -- [API Reference](../index.md) - Main documentation index diff --git a/docs/security.md b/docs/security.md deleted file mode 100644 index 248f634..0000000 --- a/docs/security.md +++ /dev/null @@ -1,265 +0,0 @@ -# Security - -## SQL Injection Prevention - -quack-diff implements comprehensive SQL injection prevention mechanisms to ensure safe handling of user-provided input in SQL queries. - -### Overview - -SQL injection is a serious security vulnerability where malicious users can manipulate SQL queries by injecting their own SQL code through user input. quack-diff addresses this through: - -1. **Input Sanitization** - All SQL identifiers (table names, column names, database names) are validated -2. **Path Validation** - File paths used in `ATTACH` statements are validated -3. **Parameterized Queries** - Where possible, parameterized queries are used -4. **Value Escaping** - String values in IN clauses are properly escaped - -### Sanitization Functions - -All sanitization functions are available in `quack_diff.core.sql_utils`: - -#### `sanitize_identifier(identifier: str) -> str` - -Validates and sanitizes SQL identifiers (table names, column names, database aliases). - -**Allowed characters:** - -- Alphanumeric: `a-z`, `A-Z`, `0-9` -- Underscore: `_` -- Hyphen: `-` -- Dot: `.` (for qualified names like `schema.table`) - -**Blocked patterns:** - -- SQL injection keywords: `DROP`, `DELETE`, `UNION`, `SELECT ... FROM`, etc. -- SQL comments: `--`, `/* */` -- Statement separators: `;` -- Boolean injection patterns: `OR 1=1`, `AND 1=1` - -**Examples:** - -```python -from quack_diff.core.sql_utils import sanitize_identifier, SQLInjectionError - -# Valid identifiers -sanitize_identifier("users") # ✓ Returns: "users" -sanitize_identifier("my_schema.my_table") # ✓ Returns: "my_schema.my_table" -sanitize_identifier("db.schema.table") # ✓ Returns: "db.schema.table" - -# Invalid identifiers (raise SQLInjectionError) -sanitize_identifier("users; DROP TABLE students") # ✗ SQL injection -sanitize_identifier("users-- comment") # ✗ SQL comment -sanitize_identifier("users OR 1=1") # ✗ Boolean injection -sanitize_identifier("table!@#") # ✗ Invalid characters -``` - -#### `sanitize_path(path: str) -> str` - -Validates file paths used in `ATTACH` statements. - -**Allowed characters:** - -- Alphanumeric: `a-z`, `A-Z`, `0-9` -- Path separators: `/`, `\` -- Special path chars: `.`, `-`, `_`, `:`, `~`, `()`, `[]`, spaces - -**Blocked patterns:** - -- SQL injection: `;`, `--`, `/* */` -- Command injection attempts - -**Examples:** - -```python -from quack_diff.core.sql_utils import sanitize_path - -# Valid paths -sanitize_path("/path/to/database.duckdb") # ✓ -sanitize_path("C:/Users/data.db") # ✓ -sanitize_path("./relative/path.db") # ✓ - -# Invalid paths (raise SQLInjectionError) -sanitize_path("/path/to/db.duckdb; DROP TABLE") # ✗ SQL injection -sanitize_path("/path/to/db.duckdb-- comment") # ✗ SQL comment -``` - -#### `quote_identifier(identifier: str) -> str` - -Sanitizes and quotes an identifier for safe use in SQL queries. - -```python -from quack_diff.core.sql_utils import quote_identifier - -quote_identifier("table") # Returns: '"table"' -quote_identifier("my_schema.my_table") # Returns: '"my_schema"."my_table"' -``` - -#### `validate_limit(limit: int | None) -> int | None` - -Validates `LIMIT` clause values. - -```python -from quack_diff.core.sql_utils import validate_limit - -validate_limit(10) # ✓ Returns: 10 -validate_limit(None) # ✓ Returns: None -validate_limit(-1) # ✗ Raises ValueError -validate_limit(0) # ✗ Raises ValueError -``` - -### Implementation Details - -#### Connector Layer - -The `DuckDBConnector` class sanitizes all user input: - -```python -from quack_diff.core.connector import DuckDBConnector - -connector = DuckDBConnector() - -# Sanitized automatically -connector.attach_duckdb( - name="other_db", # Sanitized - path="/path/to/db.duckdb" # Sanitized -) - -# These would raise SQLInjectionError: -# connector.attach_duckdb("db; DROP TABLE users", "path.db") -# connector.attach_duckdb("db", "/path; DELETE FROM users") -``` - -#### Query Builder Layer - -The `QueryBuilder` class sanitizes all identifiers used in query construction: - -```python -from quack_diff.core.query_builder import QueryBuilder - -builder = QueryBuilder() - -# All identifiers are sanitized -query = builder.build_hash_query( - table="my_schema.my_table", # Sanitized - columns=["id", "name", "email"], # Each sanitized - key_column="id" # Sanitized -) -``` - -### Remaining Considerations - -#### Value Escaping in IN Clauses - -Currently, the `build_sample_query` method uses string interpolation for key values in `IN` clauses. To prevent SQL injection, single quotes in values are escaped (`'` → `''`): - -```python -# Current implementation -keys = ["value1", "value2"] -escaped_keys = [str(k).replace("'", "''") for k in keys] -formatted_keys = ", ".join(f"'{k}'" for k in escaped_keys) -``` - -**Future improvement:** Consider using parameterized queries with placeholders for better security. - -#### Time-Travel Timestamp Values - -Timestamp values in Snowflake time-travel queries are interpolated as strings. These values come from trusted sources (user configuration) but should be validated if exposed directly to end-users. - -### Best Practices - -1. **Never concatenate user input directly into SQL** - - ```python - # BAD - Don't do this - query = f"SELECT * FROM {user_input}" - - # GOOD - Use sanitization - from quack_diff.core.sql_utils import sanitize_identifier - safe_table = sanitize_identifier(user_input) - query = f"SELECT * FROM {safe_table}" - ``` - -2. **Use parameterized queries where possible** - - ```python - # GOOD - DuckDB supports parameterized queries - connector.execute("SELECT * FROM users WHERE id = ?", [user_id]) - ``` - -3. **Validate all inputs at the entry point** - - ```python - # Validate early in your application - try: - safe_name = sanitize_identifier(user_table_name) - except SQLInjectionError as e: - return error_response(f"Invalid table name: {e}") - ``` - -4. **Be especially careful with dynamic identifiers** - - Table names from user input - - Column names from user input - - Database names from user input - - File paths from user input - -### Error Handling - -SQL injection attempts raise `SQLInjectionError`, a subclass of `ValueError`: - -```python -from quack_diff.core.sql_utils import sanitize_identifier, SQLInjectionError - -try: - safe_name = sanitize_identifier(user_input) -except SQLInjectionError as e: - print(f"Security error: {e}") -except ValueError as e: - print(f"Validation error: {e}") -``` - -### Testing - -Comprehensive tests for all sanitization functions are available in `tests/core/test_sql_utils.py`. The tests cover: - -- Valid identifier formats -- SQL injection patterns -- Invalid characters -- Edge cases (empty, too long, malformed) -- Path validation -- Value escaping - -Run the security tests: - -```bash -uv run pytest tests/core/test_sql_utils.py -v -``` - -### Reporting Security Issues - -If you discover a security vulnerability in quack-diff, please report it privately to the maintainers. Do not open a public issue. - -## Additional Security Considerations - -### File System Access - -When attaching DuckDB databases, ensure: - -- File paths are validated -- Appropriate file system permissions are set -- Database files are from trusted sources - -### Snowflake Credentials - -When using Snowflake integration: - -- Store credentials securely (use environment variables or `~/.snowflake/connections.toml`) -- Never hardcode credentials in source code -- Use read-only roles when possible -- Limit warehouse access appropriately - -### Network Security - -For Snowflake connections: - -- Connections use TLS by default -- Consider using network policies in Snowflake -- Use IP allowlists where appropriate From 6d085dc48de5f9746b4fb828b0f47370dbbe38df Mon Sep 17 00:00:00 2001 From: Matteo Renoldi Date: Wed, 4 Feb 2026 18:32:47 +0100 Subject: [PATCH 3/3] docs: update index --- docs/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index e84a61f..077cd51 100644 --- a/docs/index.md +++ b/docs/index.md @@ -55,4 +55,3 @@ quack-diff leverages DuckDB's extension system to connect to external databases: - [Installation](installation.md) — Install quack-diff with uv or pip - [Configuration](configuration.md) — Environment variables and config file -- [Security](security.md) — SQL injection prevention and security best practices