From 3ab9c8e4e98ebef479e401e63bf35283e7f7e126 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 24 Sep 2025 15:57:20 +0100 Subject: [PATCH 1/6] Updated installation instructions --- site/content/docs/overview/getting-started.md | 34 +++++++++++++++---- 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/site/content/docs/overview/getting-started.md b/site/content/docs/overview/getting-started.md index e36bd3db..2d9e0513 100644 --- a/site/content/docs/overview/getting-started.md +++ b/site/content/docs/overview/getting-started.md @@ -7,13 +7,35 @@ sidebar: This guide will help you get started with dpkit. If you are new to the core framework's tecnhologies, please take a look at the [Data Package standard](https://datapackage.org/) and [Polars DataFrames](https://pola.rs/) documentation. -:::caution -This article is a work in progress. -::: +## Prerequisites -:::tip -CLI binary distributions are in under development. Soon you will be able to install the CLI without using Node and its package manager. -::: +Supported operating systems: + +- **Linux** (x64/arm64) +- **macOS** (x64/arm64) +- **Windows** (x64) + +## Installation + +dpkit is a command-line tool that can be downloaded as a binary executable or installed using Node.js and its package managers. + +### Binary + +You can download the latest binary from the [releases page](https://github.com/datisthq/dpkit/releases) or use the following command (for POSIX-compatible shells including Git for Windows): + +```sh +curl -fsSL https://dpkit.dev/install.sh | sh +``` + +After downloading, you can verify the binary using the following command: + +```sh +./dp --version +``` + +We recommend adding the binary to your PATH environment variable to make it easier to use. + +### Node You can install the CLI using this command: From 0fcf19adcfe22ddd4b2c25d64cef78200a8a214a Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 24 Sep 2025 16:01:35 +0100 Subject: [PATCH 2/6] Updated commands guide --- site/content/docs/guides/commands.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/site/content/docs/guides/commands.md b/site/content/docs/guides/commands.md index 2df1ab49..e9ca6240 100644 --- a/site/content/docs/guides/commands.md +++ b/site/content/docs/guides/commands.md @@ -4,9 +4,7 @@ sidebar: order: 2 --- -:::note -After instalation, the dpkit's command-line tool is available in your terminal under name `dp` -::: +After instalation, the dpkit's command-line tool is available in your terminal under name `dp`. Please read the [Getting Started](../../overview/getting-started) guide for more information. ## Command categories @@ -17,21 +15,18 @@ The CLI commands are organized into categories named after the main objects they - dialect - schema - table +- file Each category has its own commands for example the `table` category: - `dp table convert` - `dp table describe` - `dp table explore` -- `dp table query` +- `dp table script` - `dp table validate` ## Working with data packages -:::caution -This functionality is under development. -::: - Usually non-package command support the `-p/--package` and `-r/--resource` options to specify the datapackage file path and a resource name to access an object inside a data package. For example, we can explore a table using this command: From 2dc77e3eeb36940b68c85a0e4a0824c1a27161fd Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 24 Sep 2025 16:06:27 +0100 Subject: [PATCH 3/6] Added package guide --- site/content/docs/guides/package.md | 263 +++++++++++++++++++++++++++- 1 file changed, 260 insertions(+), 3 deletions(-) diff --git a/site/content/docs/guides/package.md b/site/content/docs/guides/package.md index 1eea7ba2..bdb04fc6 100644 --- a/site/content/docs/guides/package.md +++ b/site/content/docs/guides/package.md @@ -4,6 +4,263 @@ sidebar: order: 3 --- -:::caution -This article is under development. -::: +# Package Commands + +Data Package commands help you work with Data Packages - collections of data files along with their metadata. These commands allow you to create, validate, explore, and publish data packages across various platforms. + +## Available Commands + +### `dp package copy` + +Copy a local or remote Data Package to a local folder, a ZIP archive or a database. + +```bash +dp package copy --to-path +``` + +**Options:** +- `--to-path` (required): Target destination for the copy +- `--with-remote`: Include remote resources in the copy +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Copy package to local directory +dp package copy datapackage.json --to-path ./output + +# Copy package to ZIP archive +dp package copy datapackage.json --to-path package.zip + +# Copy remote package including remote resources +dp package copy https://example.com/datapackage.json --to-path ./local --with-remote +``` + +### `dp package infer` + +Infer a data package from local or remote file paths. This command analyzes data files and automatically generates metadata including schema information. + +```bash +dp package infer +``` + +**Table Dialect Options:** +- `--delimiter`: Field delimiter character +- `--header`: Whether files have headers +- `--header-rows`: Number of header rows +- `--header-join`: Join character for multi-row headers +- `--comment-rows`: Number of comment rows to skip +- `--comment-char`: Comment character +- `--quote-char`: Quote character for fields +- `--double-quote`: Whether quotes are doubled for escaping +- `--escape-char`: Escape character +- `--null-sequence`: Sequence representing null values +- `--skip-initial-space`: Skip initial whitespace +- `--property`: JSON property path for nested data +- `--item-type`: Type of items in arrays +- `--item-keys`: Keys for object items +- `--sheet-number`: Excel sheet number +- `--sheet-name`: Excel sheet name +- `--table`: Database table name +- `--sample-bytes`: Bytes to sample for inference + +**Table Schema Options:** +- `--field-names`: Override field names +- `--field-types`: Override field types +- `--missing-values`: Values to treat as missing +- `--string-format`: String format specification +- `--decimal-char`: Decimal separator character +- `--group-char`: Thousands separator character +- `--bare-number`: Allow bare numbers +- `--true-values`: Values to treat as true +- `--false-values`: Values to treat as false +- `--datetime-format`: DateTime format string +- `--date-format`: Date format string +- `--time-format`: Time format string +- `--array-type`: Type of array elements +- `--list-delimiter`: List item delimiter +- `--list-item-type`: Type of list items +- `--geopoint-format`: Geopoint format specification +- `--geojson-format`: GeoJSON format specification +- `--sample-rows`: Rows to sample for inference +- `--confidence`: Confidence threshold for type inference +- `--comma-decimal`: Use comma as decimal separator +- `--month-first`: Parse dates with month first +- `--keep-strings`: Keep string types when possible + +**Examples:** +```bash +# Infer package from CSV files +dp package infer data1.csv data2.csv + +# Infer with custom delimiter +dp package infer data.csv --delimiter ";" + +# Infer from remote files +dp package infer https://example.com/data.csv +``` + +### `dp package explore` + +Explore a Data Package descriptor to view its structure and metadata in an interactive format. + +```bash +dp package explore +``` + +**Options:** +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Explore local package +dp package explore datapackage.json + +# Explore remote package +dp package explore https://example.com/datapackage.json + +# Export structure as JSON +dp package explore datapackage.json --json +``` + +### `dp package validate` + +Validate a data package from a local or remote path against the Data Package specification. + +```bash +dp package validate +``` + +**Options:** +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate local package +dp package validate datapackage.json + +# Validate remote package +dp package validate https://example.com/datapackage.json + +# Get validation results as JSON +dp package validate datapackage.json --json +``` + +### `dp package script` + +Open an interactive scripting session with a loaded Data Package. This provides a REPL environment where you can programmatically interact with the package data. + +```bash +dp package script +``` + +**Available Variables:** +- `dpkit`: The dpkit library object +- `dataPackage`: The loaded data package object + +**Examples:** +```bash +# Start scripting session +dp package script datapackage.json + +# In the REPL session: +dp> dataPackage.resources.length +dp> dataPackage.resources[0].schema.fields +``` + +### `dp package publish` + +Publish data packages to various platforms. This is a parent command with platform-specific subcommands. + +#### `dp package publish ckan` + +Publish a data package to a CKAN instance. + +```bash +dp package publish ckan +``` + +**CKAN Options:** +- `--to-ckan-api-key`: CKAN API key for authentication +- `--to-ckan-url`: CKAN instance URL +- `--to-ckan-owner-org`: Organization to publish under +- `--to-ckan-dataset-name`: Name for the dataset + +**Examples:** +```bash +# Publish to CKAN +dp package publish ckan datapackage.json \ + --to-ckan-url https://demo.ckan.org \ + --to-ckan-api-key your-api-key \ + --to-ckan-owner-org your-org +``` + +#### `dp package publish github` + +Publish a data package to GitHub as releases or repository files. + +```bash +dp package publish github +``` + +#### `dp package publish zenodo` + +Publish a data package to Zenodo for academic archiving. + +```bash +dp package publish zenodo +``` + +## Common Workflows + +### Creating a New Package + +1. **Infer from data files:** + ```bash + dp package infer *.csv --json > datapackage.json + ``` + +2. **Validate the generated package:** + ```bash + dp package validate datapackage.json + ``` + +3. **Explore the package structure:** + ```bash + dp package explore datapackage.json + ``` + +### Working with Remote Packages + +```bash +# Explore remote package +dp package explore https://example.com/datapackage.json + +# Copy remote package locally +dp package copy https://example.com/datapackage.json --to-path ./local-copy + +# Validate remote package +dp package validate https://example.com/datapackage.json +``` + +### Publishing Workflow + +```bash +# Validate before publishing +dp package validate datapackage.json + +# Publish to CKAN +dp package publish ckan datapackage.json \ + --to-ckan-url https://your-ckan-instance.org \ + --to-ckan-api-key $CKAN_API_KEY +``` + +## Output Formats + +Most commands support multiple output formats: + +- **Interactive Display**: Default rich terminal interface +- **JSON**: Use `--json` flag for machine-readable output +- **Debug Mode**: Use `--debug` for detailed operation logs From ba0d1de0aac42a673882c7eea43aeda098cc0d24 Mon Sep 17 00:00:00 2001 From: roll Date: Wed, 24 Sep 2025 16:29:35 +0100 Subject: [PATCH 4/6] Added other guides --- site/content/docs/guides/dialect.md | 314 ++++++++++++++++++++++- site/content/docs/guides/file.md | 305 +++++++++++++++++++++++ site/content/docs/guides/resource.md | 269 +++++++++++++++++++- site/content/docs/guides/schema.md | 306 ++++++++++++++++++++++- site/content/docs/guides/table.md | 357 ++++++++++++++++++++++++++- 5 files changed, 1539 insertions(+), 12 deletions(-) create mode 100644 site/content/docs/guides/file.md diff --git a/site/content/docs/guides/dialect.md b/site/content/docs/guides/dialect.md index 027e36c4..ec84bf47 100644 --- a/site/content/docs/guides/dialect.md +++ b/site/content/docs/guides/dialect.md @@ -4,6 +4,314 @@ sidebar: order: 5 --- -:::caution -This article is under development. -::: +# Dialect Commands + +Table Dialect commands help you work with CSV dialects - metadata that describes how to parse CSV and similar tabular text files. These commands allow you to infer parsing parameters from files, validate dialect definitions, and explore dialect properties. + +## Available Commands + +### `dp dialect infer` + +Infer a table dialect from a table by analyzing its structure and determining the best parsing parameters such as delimiter, quote character, and header configuration. + +```bash +dp dialect infer +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode +- `--sample-bytes`: Number of bytes to sample for dialect inference + +**Examples:** +```bash +# Infer dialect from CSV file +dp dialect infer data.csv + +# Infer from remote file +dp dialect infer https://example.com/data.csv + +# Infer from resource in package +dp dialect infer --from-package datapackage.json --from-resource "users" + +# Export dialect as JSON +dp dialect infer data.csv --json > dialect.json + +# Use larger sample for complex files +dp dialect infer complex_data.csv --sample-bytes 8192 +``` + +### `dp dialect explore` + +Explore a table dialect from a local or remote path to view its parsing configuration in an interactive format. + +```bash +dp dialect explore +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Explore dialect descriptor +dp dialect explore dialect.json + +# Explore remote dialect +dp dialect explore https://example.com/dialect.json + +# Explore dialect from package resource +dp dialect explore --from-package datapackage.json --from-resource "users" + +# Export dialect structure as JSON +dp dialect explore dialect.json --json +``` + +### `dp dialect validate` + +Validate a table dialect from a local or remote path against the CSV Dialect specification. + +```bash +dp dialect validate +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate dialect descriptor +dp dialect validate dialect.json + +# Validate remote dialect +dp dialect validate https://example.com/dialect.json + +# Validate dialect from package resource +dp dialect validate --from-package datapackage.json --from-resource "users" + +# Get validation results as JSON +dp dialect validate dialect.json --json + +# Interactive selection when no path provided +dp dialect validate --from-package datapackage.json +``` + +### `dp dialect script` + +Open an interactive scripting session with a loaded table dialect. This provides a REPL environment where you can programmatically interact with the dialect definition. + +```bash +dp dialect script +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Available Variables:** +- `dpkit`: The dpkit library object +- `dialect`: The loaded dialect object + +**Examples:** +```bash +# Start scripting session with dialect +dp dialect script dialect.json + +# Script dialect from package resource +dp dialect script --from-package datapackage.json --from-resource "users" + +# In the REPL session: +dp> dialect.delimiter +dp> dialect.quoteChar +dp> dialect.header +dp> dialect.skipInitialSpace +``` + +## Common Workflows + +### Creating Dialect Definitions + +1. **Infer from data file:** + ```bash + dp dialect infer data.csv --json > dialect.json + ``` + +2. **Validate the generated dialect:** + ```bash + dp dialect validate dialect.json + ``` + +3. **Explore the dialect configuration:** + ```bash + dp dialect explore dialect.json + ``` + +### Dialect Analysis for Complex Files + +```bash +# Infer dialect with larger sample for better accuracy +dp dialect infer complex_file.csv --sample-bytes 16384 + +# Validate and explore for verification +dp dialect validate dialect.json +dp dialect explore dialect.json + +# Script for custom dialect analysis +dp dialect script dialect.json +``` + +### Working with Package Dialects + +```bash +# Validate all dialects in a package interactively +dp dialect validate --from-package datapackage.json + +# Infer improved dialect for specific resource +dp dialect infer --from-package datapackage.json --from-resource "transactions" + +# Compare dialects using scripting +dp dialect script --from-package datapackage.json --from-resource "users" +``` + +### Remote Dialect Handling + +```bash +# Work with remote dialects +dp dialect explore https://example.com/dialect.json +dp dialect validate https://example.com/dialect.json +dp dialect infer https://example.com/data.csv +``` + +## Dialect Properties + +CSV Dialect specifications typically include: + +### Core Properties +- **delimiter**: Field separator character (e.g., `,`, `;`, `\t`) +- **quoteChar**: Character used to quote fields (e.g., `"`, `'`) +- **escapeChar**: Character used to escape quotes within fields +- **doubleQuote**: Whether quotes are escaped by doubling them + +### Header Configuration +- **header**: Whether the first row contains headers +- **headerRows**: Number of header rows +- **headerJoin**: Character used to join multi-row headers + +### Whitespace Handling +- **skipInitialSpace**: Whether to skip whitespace after delimiters +- **nullSequence**: Sequence representing null values + +### Comment Handling +- **commentRows**: Number of comment rows to skip +- **commentChar**: Character indicating comment lines + +## Common Dialect Patterns + +### Standard CSV +```json +{ + "delimiter": ",", + "quoteChar": "\"", + "doubleQuote": true, + "header": true +} +``` + +### European CSV (semicolon-separated) +```json +{ + "delimiter": ";", + "quoteChar": "\"", + "doubleQuote": true, + "header": true +} +``` + +### Tab-separated values +```json +{ + "delimiter": "\t", + "quoteChar": "\"", + "doubleQuote": true, + "header": true +} +``` + +### Custom formats with comments +```json +{ + "delimiter": "|", + "quoteChar": "'", + "header": true, + "commentRows": 3, + "commentChar": "#" +} +``` + +## Troubleshooting Dialect Inference + +### For files with unusual formatting: +```bash +# Use larger sample size +dp dialect infer unusual_file.csv --sample-bytes 32768 + +# Check inferred dialect +dp dialect explore dialect.json + +# Manually verify with table commands +dp table explore unusual_file.csv --dialect dialect.json +``` + +### For files with multiple header rows: +```bash +# The dialect inference will detect headerRows automatically +dp dialect infer multi_header.csv --json + +# Verify the header configuration +dp dialect script dialect.json +# Then in REPL: dialect.headerRows +``` + +## Output Formats + +All dialect commands support multiple output formats: + +- **Interactive Display**: Default rich terminal interface showing dialect properties +- **JSON**: Use `--json` flag for machine-readable output +- **Debug Mode**: Use `--debug` for detailed operation logs + +## Integration with Other Commands + +Dialect commands work seamlessly with other dpkit commands: + +```bash +# Create dialect, then use it for table operations +dp dialect infer data.csv --json > dialect.json +dp table validate data.csv --dialect dialect.json + +# Work within package context +dp package infer *.csv --json > datapackage.json +dp dialect validate --from-package datapackage.json --from-resource "data" + +# Use inferred dialect for schema inference +dp dialect infer data.csv --json > dialect.json +dp schema infer data.csv --delimiter ";" --header-rows 2 +``` + +## Best Practices + +1. **Sample Size**: Use larger `--sample-bytes` for files with complex or inconsistent formatting +2. **Validation**: Always validate inferred dialects before using them in production +3. **Testing**: Test dialect definitions with actual data using table commands +4. **Documentation**: Include dialect files alongside data files for reproducibility diff --git a/site/content/docs/guides/file.md b/site/content/docs/guides/file.md new file mode 100644 index 00000000..2a630bad --- /dev/null +++ b/site/content/docs/guides/file.md @@ -0,0 +1,305 @@ +--- +title: File +sidebar: + order: 8 +--- + +# File Commands + +File commands help you work with individual files, providing utilities for copying, describing, and validating files of various formats. These commands are useful for file-level operations and diagnostics. + +## Available Commands + +### `dp file copy` + +Copy a file from one location to another with support for local and remote sources and destinations. + +```bash +dp file copy +``` + +**Options:** +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Copy local file +dp file copy data.csv backup.csv + +# Copy remote file to local +dp file copy https://example.com/data.csv local_data.csv + +# Copy to different directory +dp file copy data.csv ./backup/data_backup.csv +``` + +### `dp file describe` + +Describe a file's properties including size, format, encoding, and basic metadata information. + +```bash +dp file describe +``` + +**Options:** +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Describe local file +dp file describe data.csv + +# Describe remote file +dp file describe https://example.com/data.csv + +# Get description as JSON +dp file describe data.csv --json + +# Describe various file types +dp file describe document.pdf +dp file describe image.png +dp file describe archive.zip +``` + +### `dp file validate` + +Validate a file's integrity, format compliance, and accessibility. + +```bash +dp file validate +``` + +**Options:** +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate local file +dp file validate data.csv + +# Validate remote file +dp file validate https://example.com/data.csv + +# Get validation results as JSON +dp file validate data.csv --json + +# Validate multiple file types +dp file validate document.json +dp file validate image.jpg +dp file validate data.parquet +``` + +## Common Workflows + +### File Backup and Migration + +```bash +# Create backup copy +dp file copy important_data.csv backup/important_data_$(date +%Y%m%d).csv + +# Validate backup integrity +dp file validate backup/important_data_20240101.csv + +# Describe backup properties +dp file describe backup/important_data_20240101.csv +``` + +### Remote File Handling + +```bash +# Download and validate remote file +dp file copy https://example.com/dataset.csv local_dataset.csv +dp file validate local_dataset.csv + +# Describe remote file without downloading +dp file describe https://example.com/dataset.csv +``` + +### File Diagnostics + +```bash +# Check file properties +dp file describe suspicious_file.csv + +# Validate file integrity +dp file validate suspicious_file.csv + +# Get detailed diagnostics as JSON +dp file describe problematic_file.csv --json +dp file validate problematic_file.csv --json +``` + +### Batch File Operations + +```bash +# Describe multiple files +for file in *.csv; do + echo "Describing $file:" + dp file describe "$file" + echo "---" +done + +# Validate all files in directory +for file in data/*.json; do + dp file validate "$file" --json >> validation_report.json +done +``` + +## File Type Support + +File commands work with various file formats: + +### Data Formats +- **CSV/TSV**: Comma and tab-separated values +- **JSON**: JavaScript Object Notation +- **Excel**: .xlsx and .xls files +- **Parquet**: Apache Parquet files +- **Arrow**: Apache Arrow files +- **ODS**: OpenDocument Spreadsheet + +### Archive Formats +- **ZIP**: Compressed archives +- **TAR**: Tape archives +- **GZ**: Gzip compressed files + +### Document Formats +- **PDF**: Portable Document Format +- **XML**: Extensible Markup Language +- **YAML**: YAML Ain't Markup Language + +### Image Formats +- **PNG**: Portable Network Graphics +- **JPEG/JPG**: Joint Photographic Experts Group +- **SVG**: Scalable Vector Graphics + +## File Information Retrieved + +### Basic Properties +- **Size**: File size in bytes +- **Format**: Detected file format and MIME type +- **Encoding**: Text encoding (for text files) +- **Permissions**: File access permissions (local files) + +### Content Analysis +- **Structure**: Basic structure analysis for supported formats +- **Validity**: Format compliance checking +- **Metadata**: Embedded metadata extraction + +### Remote File Properties +- **Accessibility**: Whether the remote file is accessible +- **Headers**: HTTP headers for remote files +- **Redirects**: Information about URL redirections + +## Error Handling + +### Common Issues and Solutions + +#### File Not Found +```bash +dp file describe missing_file.csv +# Error: File not found +# Solution: Check file path and permissions +``` + +#### Network Issues (Remote Files) +```bash +dp file copy https://unreachable.com/data.csv local.csv +# Error: Network timeout +# Solution: Check URL and network connectivity +``` + +#### Format Recognition +```bash +dp file describe unknown_format.dat +# May show limited information for unknown formats +# Solution: Use --debug for more details +``` + +#### Permission Issues +```bash +dp file copy protected_file.csv backup.csv +# Error: Permission denied +# Solution: Check file permissions +``` + +## Advanced Usage + +### Scripting and Automation +```bash +#!/bin/bash +# File processing script + +FILES="*.csv" +for file in $FILES; do + echo "Processing $file" + + # Validate file + if dp file validate "$file" --json | jq -r '.valid' | grep -q "true"; then + echo "✓ $file is valid" + + # Create backup + dp file copy "$file" "backup/${file%.csv}_$(date +%Y%m%d).csv" + + # Get file info + dp file describe "$file" --json > "info/${file%.csv}_info.json" + else + echo "✗ $file is invalid" + fi +done +``` + +### Integration with Other Commands +```bash +# Validate file before processing with table commands +dp file validate data.csv && dp table explore data.csv + +# Describe file and then infer schema +dp file describe data.csv +dp schema infer data.csv --json > schema.json + +# Copy and then create package +dp file copy remote_data.csv local_data.csv +dp package infer local_data.csv --json > datapackage.json +``` + +### Monitoring and Logging +```bash +# Create validation log +dp file validate data.csv --json | jq '{file: "data.csv", valid: .valid, timestamp: now}' >> validation.log + +# Monitor file changes +while true; do + dp file describe changing_file.csv --json > current_state.json + if ! cmp -s current_state.json previous_state.json; then + echo "File changed at $(date)" + cp current_state.json previous_state.json + fi + sleep 60 +done +``` + +## Output Formats + +File commands support multiple output formats: + +- **Human-readable**: Default formatted output for terminal viewing +- **JSON**: Machine-readable structured output with `--json` flag +- **Debug**: Detailed diagnostic information with `--debug` flag + +## Best Practices + +1. **Validation First**: Always validate files before processing them with other commands +2. **Backup Important Files**: Use `copy` command to create backups before modifications +3. **Remote File Handling**: Describe remote files before downloading to check size and format +4. **Error Checking**: Use JSON output for programmatic error checking and handling +5. **Documentation**: Use `describe` to document file properties for reproducibility + +## Security Considerations + +- Remote file operations follow URL protocols and security restrictions +- Local file operations respect system file permissions +- Validation helps identify potentially corrupted or malicious files +- Debug mode may expose sensitive file system information \ No newline at end of file diff --git a/site/content/docs/guides/resource.md b/site/content/docs/guides/resource.md index 9f66f3b9..4867cd06 100644 --- a/site/content/docs/guides/resource.md +++ b/site/content/docs/guides/resource.md @@ -4,6 +4,269 @@ sidebar: order: 4 --- -:::caution -This article is under development. -::: +# Resource Commands + +Data Resource commands help you work with individual data resources - the building blocks of Data Packages. These commands allow you to infer metadata from data files, validate resource structure, and explore resource properties. + +## Available Commands + +### `dp resource infer` + +Infer a data resource from a table by analyzing its structure and generating metadata including schema information. + +```bash +dp resource infer +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Table Dialect Options:** +- `--delimiter`: Field delimiter character +- `--header`: Whether files have headers +- `--header-rows`: Number of header rows +- `--header-join`: Join character for multi-row headers +- `--comment-rows`: Number of comment rows to skip +- `--comment-char`: Comment character +- `--quote-char`: Quote character for fields +- `--double-quote`: Whether quotes are doubled for escaping +- `--escape-char`: Escape character +- `--null-sequence`: Sequence representing null values +- `--skip-initial-space`: Skip initial whitespace +- `--property`: JSON property path for nested data +- `--item-type`: Type of items in arrays +- `--item-keys`: Keys for object items +- `--sheet-number`: Excel sheet number +- `--sheet-name`: Excel sheet name +- `--table`: Database table name +- `--sample-bytes`: Bytes to sample for inference + +**Table Schema Options:** +- `--field-names`: Override field names +- `--field-types`: Override field types +- `--missing-values`: Values to treat as missing +- `--string-format`: String format specification +- `--decimal-char`: Decimal separator character +- `--group-char`: Thousands separator character +- `--bare-number`: Allow bare numbers +- `--true-values`: Values to treat as true +- `--false-values`: Values to treat as false +- `--datetime-format`: DateTime format string +- `--date-format`: Date format string +- `--time-format`: Time format string +- `--array-type`: Type of array elements +- `--list-delimiter`: List item delimiter +- `--list-item-type`: Type of list items +- `--geopoint-format`: Geopoint format specification +- `--geojson-format`: GeoJSON format specification +- `--sample-rows`: Rows to sample for inference +- `--confidence`: Confidence threshold for type inference +- `--comma-decimal`: Use comma as decimal separator +- `--month-first`: Parse dates with month first +- `--keep-strings`: Keep string types when possible + +**Examples:** +```bash +# Infer resource from CSV file +dp resource infer data.csv + +# Infer with custom delimiter +dp resource infer data.csv --delimiter ";" + +# Infer from remote file +dp resource infer https://example.com/data.csv + +# Infer from resource in package +dp resource infer --from-package datapackage.json --from-resource "users" + +# Export as JSON +dp resource infer data.csv --json +``` + +### `dp resource explore` + +Explore a data resource from a local or remote path to view its structure and metadata in an interactive format. + +```bash +dp resource explore +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Explore resource descriptor +dp resource explore resource.json + +# Explore remote resource +dp resource explore https://example.com/resource.json + +# Explore resource from package +dp resource explore --from-package datapackage.json --from-resource "users" + +# Export structure as JSON +dp resource explore resource.json --json +``` + +### `dp resource validate` + +Validate a data resource from a local or remote path against the Data Resource specification. + +```bash +dp resource validate [descriptor-path] +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate resource descriptor +dp resource validate resource.json + +# Validate remote resource +dp resource validate https://example.com/resource.json + +# Validate resource from package +dp resource validate --from-package datapackage.json --from-resource "users" + +# Get validation results as JSON +dp resource validate resource.json --json + +# Interactive selection when no path provided +dp resource validate --from-package datapackage.json +``` + +### `dp resource script` + +Open an interactive scripting session with a loaded data resource. This provides a REPL environment where you can programmatically interact with the resource metadata. + +```bash +dp resource script +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Available Variables:** +- `dpkit`: The dpkit library object +- `resource`: The loaded resource object + +**Examples:** +```bash +# Start scripting session with resource +dp resource script resource.json + +# Script resource from package +dp resource script --from-package datapackage.json --from-resource "users" + +# In the REPL session: +dp> resource.schema.fields.length +dp> resource.schema.fields[0].type +dp> resource.path +``` + +## Common Workflows + +### Creating Resource Metadata + +1. **Infer from data file:** + ```bash + dp resource infer data.csv --json > resource.json + ``` + +2. **Validate the generated resource:** + ```bash + dp resource validate resource.json + ``` + +3. **Explore the resource structure:** + ```bash + dp resource explore resource.json + ``` + +### Working with Package Resources + +```bash +# Explore all resources in a package interactively +dp resource validate --from-package datapackage.json + +# Infer metadata for specific resource +dp resource infer --from-package datapackage.json --from-resource "users" + +# Script specific resource from package +dp resource script --from-package datapackage.json --from-resource "transactions" +``` + +### Resource Analysis Workflow + +```bash +# Infer resource with custom options +dp resource infer data.csv \ + --delimiter ";" \ + --header-rows 2 \ + --sample-rows 1000 + +# Validate the inferred resource +dp resource validate resource.json + +# Explore interactively to verify structure +dp resource explore resource.json +``` + +### Remote Resource Handling + +```bash +# Work with remote resources +dp resource explore https://example.com/resource.json +dp resource validate https://example.com/resource.json +dp resource infer https://example.com/data.csv +``` + +## Resource Selection + +When working with resources from packages, you can either: + +1. **Specify explicitly:** + ```bash + dp resource explore --from-package datapackage.json --from-resource "users" + ``` + +2. **Interactive selection:** + ```bash + dp resource validate --from-package datapackage.json + # Will prompt to select from available resources + ``` + +## Output Formats + +All resource commands support multiple output formats: + +- **Interactive Display**: Default rich terminal interface showing resource structure +- **JSON**: Use `--json` flag for machine-readable output +- **Debug Mode**: Use `--debug` for detailed operation logs + +## Integration with Package Commands + +Resource commands work seamlessly with package commands: + +```bash +# Create package, then work with individual resources +dp package infer *.csv --json > datapackage.json +dp resource validate --from-package datapackage.json --from-resource "data" +dp resource explore --from-package datapackage.json --from-resource "users" +``` diff --git a/site/content/docs/guides/schema.md b/site/content/docs/guides/schema.md index eff29a19..ec2234cc 100644 --- a/site/content/docs/guides/schema.md +++ b/site/content/docs/guides/schema.md @@ -4,6 +4,306 @@ sidebar: order: 6 --- -:::caution -This article is under development. -::: +# Schema Commands + +Table Schema commands help you work with table schemas - metadata that describes the structure, types, and constraints of tabular data. These commands allow you to infer schema from data, validate schema definitions, and explore schema properties. + +## Available Commands + +### `dp schema infer` + +Infer a table schema from a table by analyzing its data and generating field definitions including types, constraints, and formats. + +```bash +dp schema infer +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Table Dialect Options:** +- `--delimiter`: Field delimiter character +- `--header`: Whether files have headers +- `--header-rows`: Number of header rows +- `--header-join`: Join character for multi-row headers +- `--comment-rows`: Number of comment rows to skip +- `--comment-char`: Comment character +- `--quote-char`: Quote character for fields +- `--double-quote`: Whether quotes are doubled for escaping +- `--escape-char`: Escape character +- `--null-sequence`: Sequence representing null values +- `--skip-initial-space`: Skip initial whitespace +- `--property`: JSON property path for nested data +- `--item-type`: Type of items in arrays +- `--item-keys`: Keys for object items +- `--sheet-number`: Excel sheet number +- `--sheet-name`: Excel sheet name +- `--table`: Database table name +- `--sample-bytes`: Bytes to sample for inference + +**Table Schema Options:** +- `--field-names`: Override field names +- `--field-types`: Override field types +- `--missing-values`: Values to treat as missing +- `--string-format`: String format specification +- `--decimal-char`: Decimal separator character +- `--group-char`: Thousands separator character +- `--bare-number`: Allow bare numbers +- `--true-values`: Values to treat as true +- `--false-values`: Values to treat as false +- `--datetime-format`: DateTime format string +- `--date-format`: Date format string +- `--time-format`: Time format string +- `--array-type`: Type of array elements +- `--list-delimiter`: List item delimiter +- `--list-item-type`: Type of list items +- `--geopoint-format`: Geopoint format specification +- `--geojson-format`: GeoJSON format specification +- `--sample-rows`: Rows to sample for inference +- `--confidence`: Confidence threshold for type inference +- `--comma-decimal`: Use comma as decimal separator +- `--month-first`: Parse dates with month first +- `--keep-strings`: Keep string types when possible + +**Examples:** +```bash +# Infer schema from CSV file +dp schema infer data.csv + +# Infer with custom delimiter and date format +dp schema infer data.csv --delimiter ";" --date-format "%d/%m/%Y" + +# Infer from remote file +dp schema infer https://example.com/data.csv + +# Infer from resource in package +dp schema infer --from-package datapackage.json --from-resource "users" + +# Export schema as JSON +dp schema infer data.csv --json > schema.json +``` + +### `dp schema explore` + +Explore a table schema from a local or remote path to view its field definitions and constraints in an interactive format. + +```bash +dp schema explore +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Explore schema descriptor +dp schema explore schema.json + +# Explore remote schema +dp schema explore https://example.com/schema.json + +# Explore schema from package resource +dp schema explore --from-package datapackage.json --from-resource "users" + +# Export schema structure as JSON +dp schema explore schema.json --json +``` + +### `dp schema validate` + +Validate a table schema from a local or remote path against the Table Schema specification. + +```bash +dp schema validate +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate schema descriptor +dp schema validate schema.json + +# Validate remote schema +dp schema validate https://example.com/schema.json + +# Validate schema from package resource +dp schema validate --from-package datapackage.json --from-resource "users" + +# Get validation results as JSON +dp schema validate schema.json --json + +# Interactive selection when no path provided +dp schema validate --from-package datapackage.json +``` + +### `dp schema script` + +Open an interactive scripting session with a loaded table schema. This provides a REPL environment where you can programmatically interact with the schema definition. + +```bash +dp schema script +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Available Variables:** +- `dpkit`: The dpkit library object +- `schema`: The loaded schema object + +**Examples:** +```bash +# Start scripting session with schema +dp schema script schema.json + +# Script schema from package resource +dp schema script --from-package datapackage.json --from-resource "users" + +# In the REPL session: +dp> schema.fields.length +dp> schema.fields[0].name +dp> schema.fields.filter(f => f.type === 'integer') +dp> schema.primaryKey +``` + +## Common Workflows + +### Creating Schema Definitions + +1. **Infer from data file:** + ```bash + dp schema infer data.csv --json > schema.json + ``` + +2. **Validate the generated schema:** + ```bash + dp schema validate schema.json + ``` + +3. **Explore the schema structure:** + ```bash + dp schema explore schema.json + ``` + +### Schema Analysis and Refinement + +```bash +# Infer schema with high confidence threshold +dp schema infer data.csv --confidence 0.8 --sample-rows 10000 + +# Validate and explore for refinement +dp schema validate schema.json +dp schema explore schema.json + +# Script for custom analysis +dp schema script schema.json +``` + +### Working with Package Schemas + +```bash +# Validate all schemas in a package interactively +dp schema validate --from-package datapackage.json + +# Infer improved schema for specific resource +dp schema infer --from-package datapackage.json --from-resource "transactions" + +# Compare schemas using scripting +dp schema script --from-package datapackage.json --from-resource "users" +``` + +### Custom Type Inference + +```bash +# Configure specific data types and formats +dp schema infer data.csv \ + --datetime-format "%Y-%m-%d %H:%M:%S" \ + --true-values "Yes,True,1" \ + --false-values "No,False,0" \ + --decimal-char "," \ + --missing-values "NULL,N/A,," +``` + +### Remote Schema Handling + +```bash +# Work with remote schemas +dp schema explore https://example.com/schema.json +dp schema validate https://example.com/schema.json +dp schema infer https://example.com/data.csv +``` + +## Schema Field Types + +The schema inference supports various field types: + +- **Basic Types**: `string`, `integer`, `number`, `boolean` +- **Date/Time Types**: `date`, `datetime`, `time`, `year`, `yearmonth`, `duration` +- **Structured Types**: `array`, `object`, `list` +- **Geographic Types**: `geopoint`, `geojson` + +## Advanced Inference Options + +### Confidence Tuning +```bash +# High confidence for clean data +dp schema infer data.csv --confidence 0.9 + +# Lower confidence for messy data +dp schema infer data.csv --confidence 0.6 +``` + +### Sample Size Control +```bash +# Large sample for better inference +dp schema infer large_data.csv --sample-rows 50000 + +# Quick inference with small sample +dp schema infer data.csv --sample-rows 100 +``` + +### Format Specifications +```bash +# European date format +dp schema infer data.csv --date-format "%d.%m.%Y" + +# Custom boolean values +dp schema infer data.csv --true-values "Ja,Oui,Sí" --false-values "Nein,Non,No" +``` + +## Output Formats + +All schema commands support multiple output formats: + +- **Interactive Display**: Default rich terminal interface showing field definitions +- **JSON**: Use `--json` flag for machine-readable output +- **Debug Mode**: Use `--debug` for detailed operation logs + +## Integration with Other Commands + +Schema commands work seamlessly with other dpkit commands: + +```bash +# Create schema, then use it for validation +dp schema infer data.csv --json > schema.json +dp table validate data.csv --schema schema.json + +# Work within package context +dp package infer *.csv --json > datapackage.json +dp schema validate --from-package datapackage.json --from-resource "data" +``` diff --git a/site/content/docs/guides/table.md b/site/content/docs/guides/table.md index 95e7eaeb..572fcfc6 100644 --- a/site/content/docs/guides/table.md +++ b/site/content/docs/guides/table.md @@ -4,6 +4,357 @@ sidebar: order: 7 --- -:::caution -This article is under development. -::: +# Table Commands + +Table commands help you work directly with tabular data files. These commands allow you to explore, validate, convert, and analyze data tables with support for various formats including CSV, Excel, JSON, and more. + +## Available Commands + +### `dp table convert` + +Convert a table from one format to another with support for various input and output formats. + +```bash +dp table convert +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--debug`: Enable debug mode + +**Supported Formats:** +- **Input**: CSV, TSV, Excel (.xlsx, .xls), JSON, Parquet, Arrow, ODS +- **Output**: CSV, TSV, Excel (.xlsx), JSON, Parquet, Arrow, ODS + +**Examples:** +```bash +# Convert CSV to Excel +dp table convert data.csv data.xlsx + +# Convert Excel to JSON +dp table convert data.xlsx data.json + +# Convert from package resource +dp table convert --from-package datapackage.json --from-resource "users" users.xlsx + +# Convert Parquet to CSV +dp table convert data.parquet data.csv +``` + +### `dp table describe` + +Generate statistical descriptions and summaries of table data including column statistics, data types, and quality metrics. + +```bash +dp table describe +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Describe CSV file +dp table describe data.csv + +# Describe with JSON output +dp table describe data.csv --json + +# Describe resource from package +dp table describe --from-package datapackage.json --from-resource "sales" + +# Describe remote table +dp table describe https://example.com/data.csv +``` + +### `dp table explore` + +Explore a table interactively, viewing data samples, column information, and basic statistics in a rich terminal interface. + +```bash +dp table explore +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--schema`: Path to schema file for validation during exploration +- `--dialect`: Path to dialect file for parsing configuration +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Examples:** +```bash +# Explore CSV file +dp table explore data.csv + +# Explore with schema validation +dp table explore data.csv --schema schema.json + +# Explore with custom dialect +dp table explore data.csv --dialect dialect.json + +# Explore resource from package +dp table explore --from-package datapackage.json --from-resource "users" + +# Explore remote table +dp table explore https://example.com/data.csv +``` + +### `dp table validate` + +Validate table data against a schema, checking data types, constraints, and data quality issues. + +```bash +dp table validate +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--schema`: Path to schema file for validation +- `--dialect`: Path to dialect file for parsing +- `--json`: Output validation results as JSON +- `--debug`: Enable debug mode +- `--quit`: Exit immediately after validation (don't prompt for error filtering) + +**Examples:** +```bash +# Validate with schema +dp table validate data.csv --schema schema.json + +# Validate with custom dialect and schema +dp table validate data.csv --dialect dialect.json --schema schema.json + +# Validate resource from package +dp table validate --from-package datapackage.json --from-resource "users" + +# Get validation results as JSON +dp table validate data.csv --schema schema.json --json + +# Validate remote table +dp table validate https://example.com/data.csv --schema https://example.com/schema.json +``` + +### `dp table script` + +Open an interactive scripting session with a loaded table. This provides a REPL environment where you can programmatically analyze and manipulate table data. + +```bash +dp table script +``` + +**Options:** +- `--from-package`: Path to package containing the resource +- `--from-resource`: Name of resource within package +- `--schema`: Path to schema file +- `--dialect`: Path to dialect file +- `--json`: Output as JSON +- `--debug`: Enable debug mode + +**Available Variables:** +- `dpkit`: The dpkit library object +- `table`: The loaded table object + +**Examples:** +```bash +# Start scripting session with table +dp table script data.csv + +# Script with schema and dialect +dp table script data.csv --schema schema.json --dialect dialect.json + +# Script resource from package +dp table script --from-package datapackage.json --from-resource "sales" + +# In the REPL session: +dp> table.rows.length +dp> table.columns +dp> table.rows[0] +dp> table.schema.fields.map(f => f.name) +``` + +## Common Workflows + +### Data Exploration and Analysis + +1. **Quick exploration:** + ```bash + dp table explore data.csv + ``` + +2. **Generate statistical summary:** + ```bash + dp table describe data.csv + ``` + +3. **Interactive analysis:** + ```bash + dp table script data.csv + ``` + +### Data Validation Workflow + +```bash +# Infer schema from table +dp schema infer data.csv --json > schema.json + +# Validate table against schema +dp table validate data.csv --schema schema.json + +# Explore validation issues +dp table explore data.csv --schema schema.json +``` + +### Format Conversion Pipeline + +```bash +# Convert Excel to CSV for processing +dp table convert input.xlsx temp.csv + +# Process and validate +dp table validate temp.csv --schema schema.json + +# Convert to final format +dp table convert temp.csv output.json +``` + +### Package Integration Workflow + +```bash +# Create package with tables +dp package infer *.csv --json > datapackage.json + +# Validate individual tables +dp table validate --from-package datapackage.json --from-resource "users" + +# Describe tables for documentation +dp table describe --from-package datapackage.json --from-resource "sales" --json +``` + +### Remote Table Processing + +```bash +# Explore remote table +dp table explore https://example.com/data.csv + +# Convert remote to local format +dp table convert https://example.com/data.csv local_data.xlsx + +# Validate remote table with local schema +dp table validate https://example.com/data.csv --schema local_schema.json +``` + +## Supported File Formats + +### Input Formats +- **CSV/TSV**: Comma/tab-separated values with dialect support +- **Excel**: .xlsx and .xls files with sheet selection +- **JSON**: Various JSON table formats +- **Parquet**: Apache Parquet columnar format +- **Arrow**: Apache Arrow format +- **ODS**: OpenDocument Spreadsheet format + +### Output Formats +- **CSV**: Standard comma-separated values +- **Excel**: .xlsx format +- **JSON**: Various JSON structures +- **Parquet**: Efficient columnar storage +- **Arrow**: High-performance analytics +- **ODS**: Cross-platform spreadsheets + +## Advanced Options + +### Schema-Aware Operations +```bash +# Validate with type checking +dp table validate data.csv --schema schema.json + +# Explore with schema hints +dp table explore data.csv --schema schema.json +``` + +### Custom Parsing +```bash +# Use custom dialect for parsing +dp table explore data.csv --dialect custom_dialect.json + +# Convert with parsing options +dp table convert complex_data.csv output.xlsx --dialect dialect.json +``` + +### Statistical Analysis +```bash +# Generate comprehensive statistics +dp table describe large_dataset.csv --json > stats.json + +# Interactive statistical exploration +dp table script data.csv +# In REPL: analyze column distributions, correlations, etc. +``` + +## Error Handling and Debugging + +### Validation Issues +```bash +# Get detailed validation report +dp table validate data.csv --schema schema.json --json + +# Interactive error exploration (don't quit on errors) +dp table validate data.csv --schema schema.json +``` + +### Parsing Problems +```bash +# Debug parsing issues +dp table explore problematic.csv --debug + +# Infer and test dialect +dp dialect infer problematic.csv --json > dialect.json +dp table explore problematic.csv --dialect dialect.json +``` + +### Performance Optimization +```bash +# For large files, use sampling +dp table describe huge_file.csv --sample-rows 10000 + +# Convert to efficient formats for repeated analysis +dp table convert large_data.csv data.parquet +``` + +## Output Formats + +Table commands support multiple output formats: + +- **Interactive Display**: Rich terminal interface with data previews +- **JSON**: Machine-readable structured output +- **Statistical Reports**: Comprehensive data summaries +- **Debug Mode**: Detailed operation logs and diagnostics + +## Integration Examples + +### With Package Commands +```bash +# Create and validate package +dp package infer *.csv --json > datapackage.json +dp table validate --from-package datapackage.json --from-resource "main" +``` + +### With Schema Commands +```bash +# Infer schema and validate table +dp schema infer data.csv --json > schema.json +dp table validate data.csv --schema schema.json +``` + +### With Dialect Commands +```bash +# Infer dialect and use for table operations +dp dialect infer data.csv --json > dialect.json +dp table explore data.csv --dialect dialect.json +``` From 566287c92593451d76e7144da0aa73c3deb0e6ff Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 25 Sep 2025 10:16:55 +0100 Subject: [PATCH 5/6] Removed header --- site/content/docs/guides/dialect.md | 2 -- site/content/docs/guides/file.md | 4 +--- site/content/docs/guides/package.md | 2 -- site/content/docs/guides/resource.md | 2 -- site/content/docs/guides/schema.md | 2 -- site/content/docs/guides/table.md | 2 -- 6 files changed, 1 insertion(+), 13 deletions(-) diff --git a/site/content/docs/guides/dialect.md b/site/content/docs/guides/dialect.md index ec84bf47..26781e5f 100644 --- a/site/content/docs/guides/dialect.md +++ b/site/content/docs/guides/dialect.md @@ -4,8 +4,6 @@ sidebar: order: 5 --- -# Dialect Commands - Table Dialect commands help you work with CSV dialects - metadata that describes how to parse CSV and similar tabular text files. These commands allow you to infer parsing parameters from files, validate dialect definitions, and explore dialect properties. ## Available Commands diff --git a/site/content/docs/guides/file.md b/site/content/docs/guides/file.md index 2a630bad..1b49ac75 100644 --- a/site/content/docs/guides/file.md +++ b/site/content/docs/guides/file.md @@ -4,8 +4,6 @@ sidebar: order: 8 --- -# File Commands - File commands help you work with individual files, providing utilities for copying, describing, and validating files of various formats. These commands are useful for file-level operations and diagnostics. ## Available Commands @@ -302,4 +300,4 @@ File commands support multiple output formats: - Remote file operations follow URL protocols and security restrictions - Local file operations respect system file permissions - Validation helps identify potentially corrupted or malicious files -- Debug mode may expose sensitive file system information \ No newline at end of file +- Debug mode may expose sensitive file system information diff --git a/site/content/docs/guides/package.md b/site/content/docs/guides/package.md index bdb04fc6..e1bb4941 100644 --- a/site/content/docs/guides/package.md +++ b/site/content/docs/guides/package.md @@ -4,8 +4,6 @@ sidebar: order: 3 --- -# Package Commands - Data Package commands help you work with Data Packages - collections of data files along with their metadata. These commands allow you to create, validate, explore, and publish data packages across various platforms. ## Available Commands diff --git a/site/content/docs/guides/resource.md b/site/content/docs/guides/resource.md index 4867cd06..6292cc1b 100644 --- a/site/content/docs/guides/resource.md +++ b/site/content/docs/guides/resource.md @@ -4,8 +4,6 @@ sidebar: order: 4 --- -# Resource Commands - Data Resource commands help you work with individual data resources - the building blocks of Data Packages. These commands allow you to infer metadata from data files, validate resource structure, and explore resource properties. ## Available Commands diff --git a/site/content/docs/guides/schema.md b/site/content/docs/guides/schema.md index ec2234cc..d510c7b7 100644 --- a/site/content/docs/guides/schema.md +++ b/site/content/docs/guides/schema.md @@ -4,8 +4,6 @@ sidebar: order: 6 --- -# Schema Commands - Table Schema commands help you work with table schemas - metadata that describes the structure, types, and constraints of tabular data. These commands allow you to infer schema from data, validate schema definitions, and explore schema properties. ## Available Commands diff --git a/site/content/docs/guides/table.md b/site/content/docs/guides/table.md index 572fcfc6..ea9467f8 100644 --- a/site/content/docs/guides/table.md +++ b/site/content/docs/guides/table.md @@ -4,8 +4,6 @@ sidebar: order: 7 --- -# Table Commands - Table commands help you work directly with tabular data files. These commands allow you to explore, validate, convert, and analyze data tables with support for various formats including CSV, Excel, JSON, and more. ## Available Commands From c0effa16b4fffc93389418edbcb4157a5a6f5b9e Mon Sep 17 00:00:00 2001 From: roll Date: Thu, 25 Sep 2025 10:28:38 +0100 Subject: [PATCH 6/6] Added shortcuts --- site/content/docs/guides/dialect.md | 34 ++++++++++++++-------------- site/content/docs/guides/file.md | 10 ++++---- site/content/docs/guides/package.md | 10 ++++---- site/content/docs/guides/resource.md | 26 ++++++++++----------- site/content/docs/guides/schema.md | 26 ++++++++++----------- site/content/docs/guides/table.md | 16 ++++++------- 6 files changed, 61 insertions(+), 61 deletions(-) diff --git a/site/content/docs/guides/dialect.md b/site/content/docs/guides/dialect.md index 26781e5f..22ae0304 100644 --- a/site/content/docs/guides/dialect.md +++ b/site/content/docs/guides/dialect.md @@ -17,10 +17,10 @@ dp dialect infer ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode - `--sample-bytes`: Number of bytes to sample for dialect inference **Examples:** @@ -50,10 +50,10 @@ dp dialect explore ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -79,11 +79,11 @@ dp dialect validate ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output validation results as JSON -- `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output validation results as JSON +- `-d, --debug`: Enable debug mode +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash @@ -112,10 +112,10 @@ dp dialect script ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Available Variables:** - `dpkit`: The dpkit library object diff --git a/site/content/docs/guides/file.md b/site/content/docs/guides/file.md index 1b49ac75..8ae27d38 100644 --- a/site/content/docs/guides/file.md +++ b/site/content/docs/guides/file.md @@ -17,7 +17,7 @@ dp file copy ``` **Options:** -- `--debug`: Enable debug mode +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -40,8 +40,8 @@ dp file describe ``` **Options:** -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -70,8 +70,8 @@ dp file validate **Options:** - `--json`: Output validation results as JSON -- `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-d, --debug`: Enable debug mode +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash diff --git a/site/content/docs/guides/package.md b/site/content/docs/guides/package.md index e1bb4941..d0870bf6 100644 --- a/site/content/docs/guides/package.md +++ b/site/content/docs/guides/package.md @@ -19,7 +19,7 @@ dp package copy --to-path **Options:** - `--to-path` (required): Target destination for the copy - `--with-remote`: Include remote resources in the copy -- `--debug`: Enable debug mode +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -106,8 +106,8 @@ dp package explore ``` **Options:** -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -131,8 +131,8 @@ dp package validate **Options:** - `--json`: Output validation results as JSON -- `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-d, --debug`: Enable debug mode +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash diff --git a/site/content/docs/guides/resource.md b/site/content/docs/guides/resource.md index 6292cc1b..537e0e1f 100644 --- a/site/content/docs/guides/resource.md +++ b/site/content/docs/guides/resource.md @@ -17,10 +17,10 @@ dp resource infer ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Table Dialect Options:** - `--delimiter`: Field delimiter character @@ -93,10 +93,10 @@ dp resource explore ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -126,7 +126,7 @@ dp resource validate [descriptor-path] - `--from-resource`: Name of resource within package - `--json`: Output validation results as JSON - `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash @@ -155,10 +155,10 @@ dp resource script ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Available Variables:** - `dpkit`: The dpkit library object diff --git a/site/content/docs/guides/schema.md b/site/content/docs/guides/schema.md index d510c7b7..26e9186d 100644 --- a/site/content/docs/guides/schema.md +++ b/site/content/docs/guides/schema.md @@ -17,10 +17,10 @@ dp schema infer ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Table Dialect Options:** - `--delimiter`: Field delimiter character @@ -93,10 +93,10 @@ dp schema explore ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -126,7 +126,7 @@ dp schema validate - `--from-resource`: Name of resource within package - `--json`: Output validation results as JSON - `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash @@ -155,10 +155,10 @@ dp schema script ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Available Variables:** - `dpkit`: The dpkit library object diff --git a/site/content/docs/guides/table.md b/site/content/docs/guides/table.md index ea9467f8..a493b3cf 100644 --- a/site/content/docs/guides/table.md +++ b/site/content/docs/guides/table.md @@ -17,9 +17,9 @@ dp table convert ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-d, --debug`: Enable debug mode **Supported Formats:** - **Input**: CSV, TSV, Excel (.xlsx, .xls), JSON, Parquet, Arrow, ODS @@ -49,10 +49,10 @@ dp table describe ``` **Options:** -- `--from-package`: Path to package containing the resource -- `--from-resource`: Name of resource within package -- `--json`: Output as JSON -- `--debug`: Enable debug mode +- `-p, --from-package`: Path to package containing the resource +- `-r, --from-resource`: Name of resource within package +- `-j, --json`: Output as JSON +- `-d, --debug`: Enable debug mode **Examples:** ```bash @@ -118,7 +118,7 @@ dp table validate - `--dialect`: Path to dialect file for parsing - `--json`: Output validation results as JSON - `--debug`: Enable debug mode -- `--quit`: Exit immediately after validation (don't prompt for error filtering) +- `-q, --quit`: Exit immediately after validation (don't prompt for error filtering) **Examples:** ```bash