Full refactoring of store documentation.

.github/workflows/docs.yml

@@ -28,7 +28,7 @@ jobs:
 
       # Build docs
       - name: Build docs
-        run: catchsegv sphinx-build docs/source docs/build
+        run: sphinx-build docs/source docs/build
 
       # Add nojeckyll
       - name: Add nojeckyll

.pre-commit-config.yaml

@@ -1,13 +1,4 @@
 repos:
-  # Commitizen
-  - repo: https://github.com/commitizen-tools/commitizen
-    rev: v4.8.0
-    hooks:
-      - id: commitizen
-        stages: [commit-msg]
-      - id: commitizen-branch
-        stages: [push]
-
   # Pre-commit hooks
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0

docs/source/api.rst

@@ -1,22 +1,154 @@
-API reference
+API Reference
 =============
 
-.. currentmodule:: oups
+This section provides detailed API documentation for the main store components.
 
-.. autosummary::
-    toplevel
-    sublevel
-    ParquetSet
-    ParquetSet.__setitem__
-    ParquetSet.__getitem__
-    oups.router.ParquetHandle.pf
-    oups.router.ParquetHandle.pdf
-    oups.router.ParquetHandle.vdf
+Indexer Functions
+-----------------
 
-.. autofunction:: oups.indexer.toplevel
+.. autofunction:: oups.store.toplevel
 
-.. autofunction:: oups.indexer.sublevel
+.. autofunction:: oups.store.sublevel
 
-.. autoclass:: oups.collection.ParquetSet
+.. autofunction:: oups.store.is_toplevel
 
-.. autofunction:: oups.writer.write
+Core Classes
+------------
+
+OrderedParquetDataset
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: oups.store.OrderedParquetDataset
+   :members:
+   :show-inheritance:
+
+Store
+~~~~~
+
+.. autoclass:: oups.store.Store
+   :members:
+   :show-inheritance:
+
+Write Operations
+----------------
+
+.. autofunction:: oups.store.write
+
+Utility Functions
+-----------------
+
+.. autofunction:: oups.store.check_cmidx
+
+.. autofunction:: oups.store.conform_cmidx
+
+Type Definitions
+----------------
+
+The following are important type definitions used throughout the store module:
+
+**Index Types**
+
+Indexer classes are dataclasses decorated with ``@toplevel`` that define the schema for organizing datasets.
+
+**Ordered Column Types**
+
+The ``ordered_on`` parameter accepts:
+
+- ``str``: Single column name
+- ``Tuple[str]``: Multi-index column name (for hierarchical columns)
+
+**Row Group Target Size Types**
+
+The ``row_group_target_size`` parameter accepts:
+
+- ``int``: Target number of rows per row group
+- ``str``: Pandas frequency string (e.g., "1D", "1H") for time-based grouping
+
+**Key-Value Metadata**
+
+Custom metadata stored as ``Dict[str, str]`` alongside parquet files.
+
+Examples
+--------
+
+Basic Usage
+~~~~~~~~~~~
+
+.. code-block:: python
+
+    from oups.store import toplevel, Store, OrderedParquetDataset
+    import pandas as pd
+
+    # Define indexer schema
+    @toplevel
+    class MyIndex:
+        category: str
+        subcategory: str
+
+    # Create store
+    store = Store("/path/to/data", MyIndex)
+
+    # Create sample data
+    df = pd.DataFrame({
+        "timestamp": pd.date_range("2023-01-01", periods=1000),
+        "value": range(1000)
+    })
+
+    # Access dataset and write data
+    key = MyIndex("stocks", "tech")
+    dataset = store[key]
+    dataset.write(df=df, ordered_on="timestamp")
+
+Advanced Write Options
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    from oups.store import write
+
+    # Time-based row groups with duplicate handling
+    write(
+        "/path/to/dataset",
+        ordered_on="timestamp",
+        df=df,
+        row_group_target_size="1D",  # Daily row groups
+        duplicates_on=["timestamp", "symbol"],  # Drop duplicates
+        max_n_off_target_rgs=2,  # Coalesce small row groups
+        key_value_metadata={"source": "bloomberg", "version": "1.0"}
+    )
+
+Cross-Dataset Queries
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    # Query multiple datasets simultaneously
+    keys = [MyIndex("stocks", "tech"), MyIndex("stocks", "finance")]
+
+    for intersection in store.iter_intersections(
+        keys,
+        start=pd.Timestamp("2023-01-01"),
+        end_excl=pd.Timestamp("2023-02-01")
+    ):
+        for key, df in intersection.items():
+            print(f"Processing {key}: {len(df)} rows")
+
+Hierarchical Indexing
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    from oups.store import toplevel, sublevel
+
+    @sublevel
+    class DateInfo:
+        year: str
+        month: str
+
+    @toplevel
+    class HierarchicalIndex:
+        symbol: str
+        date_info: DateInfo
+
+    # This creates paths like: AAPL/2023-01/
+    key = HierarchicalIndex("AAPL", DateInfo("2023", "01"))

docs/source/index.rst

@@ -6,20 +6,35 @@
 oups
 ====
 
-*oups* stands for Ordered Updatable Parquet Store.
+*oups* stands for **Ordered Updatable Parquet Store**.
 
-*oups* Python library provides convenience functions and class to manage a collection of parquet datasets. This includes mostly collection indexing, and ability to update ordered datasets.
+*oups* is a Python library that provides powerful tools for managing collections of ordered parquet datasets. It enables efficient storage, indexing, and querying of time-series data with validated ordering and good performance.
 
-Index
------
+Key Features
+------------
+
+- **Ordered Storage**: Validates data ordering within datasets
+- **Schema-based Indexing**: Hierarchical organization using dataclass schemas
+- **Incremental Updates**: Efficiently merge new data with existing datasets
+- **Row Group Management**: Optimizing storage layout
+- **Duplicate Handling**: Configurable duplicate detection and removal
+- **Lock-based Concurrency**: Safe concurrent access to datasets
+- **Cross-dataset Queries**: Query multiple datasets simultaneously
+
+Documentation
+-------------
 
 .. toctree::
-    install
-    quickstart
-    purpose
-    indexing
-    parquetset
-    api
+   :maxdepth: 2
+
+   install
+   quickstart
+   purpose
+   store
+   api
+
+Indices and Tables
+------------------
 
 * :ref:`genindex`