Skip to content

README.md

Latest commit

 

History

History
299 lines (216 loc) · 16.6 KB

README.md

File metadata and controls

299 lines (216 loc) · 16.6 KB
Raw

Apache Ossie Converters

Overview

An Ossie Converter translates between the Ossie semantic model format and a specific vendor's semantic implementation. This enables teams to author a semantic model once in the Ossie standard and then generate the corresponding vendor-specific representation automatically.

Hub-and-Spoke Model

Ossie converters follow a hub-and-spoke architecture:

  • Hub: The Ossie core specification acts as the central, vendor-neutral format.
  • Spokes: Each converter handles translation to or from a specific vendor format.
                  ┌─────────────┐
                  │  Snowflake  │
                  └──────┬──────┘
                         │
┌─────────────┐    ┌─────┴─────┐    ┌─────────────┐
│     dbt     ├────┤   Ossie   ├────┤  Salesforce │
└─────────────┘    └─────┬─────┘    └─────────────┘
                         │
                  ┌──────┴──────┐
                  │  Databricks │
                  └─────────────┘

This approach avoids the need for point-to-point converters between every pair of vendors. With N vendors, a point-to-point strategy would require N*(N-1) converters. With Ossie as the hub, only 2*N converters are needed (one import and one export per vendor), and interoperability with all other vendors comes for free.

Converter Responsibilities

A converter handles two directions of translation:

Export (Apache Ossie → Vendor)

Read an Ossie semantic model and produce the equivalent vendor-specific representation. For example:

  • Ossie → Snowflake semantic model definition
  • Ossie → dbt semantic_models YAML
  • Ossie → Tableau data source / Salesforce semantic layer
  • Ossie → Databricks semantic layer definition

Import (Vendor → Apache Ossie)

Read a vendor-specific semantic model and produce a valid Ossie model, including mapping vendor-specific metadata into custom_extensions.

Supported Vendors

The Ossie specification currently defines extensions for the following vendors:

Vendor Description
SNOWFLAKE Snowflake semantic model
SALESFORCE Salesforce / Tableau semantic layer
DBT dbt semantic models
DATABRICKS Databricks semantic layer
OMNI Omni semantic model
WISDOM WisdomAI domain
NVIDIA_GSF NVIDIA Generative Semantic Fabric standalone YAML

Each vendor may define custom extensions (via the custom_extensions field in the Ossie spec) to carry vendor-specific metadata that does not have an equivalent in the core specification.

Mapping Core Constructs

The following table shows how each Ossie construct maps conceptually to vendor equivalents. A converter must handle each of these:

Semantic Model

The top-level container. Maps to the root object in each vendor's format.

Ossie Field Description Converter Consideration
name Model identifier Map to vendor's model/project name
description Human-readable description Most vendors support a description field
ai_context Instructions, synonyms for AI tools Map if vendor supports AI/LLM annotations
datasets Logical datasets (fact/dimension tables) See dataset mapping below
relationships Foreign key connections See relationship mapping below
metrics Aggregate measures See metric mapping below
custom_extensions Vendor-specific metadata Extract extensions matching the target vendor

Datasets

Datasets represent logical tables (fact or dimension tables). They contain fields and define the structure of the data.

Ossie Field Description Converter Consideration
name Dataset identifier Map to table/entity/model name
source Physical table reference (e.g., database.schema.table) Parse into vendor-specific catalog/schema/table components
primary_key Single or composite primary key Map to vendor's PK syntax; note composite keys use arrays like [order_id, line_number]
unique_keys Alternative unique identifiers Map if vendor supports unique constraints
fields Row-level attributes (columns) See field mapping below
ai_context Synonyms and context for AI Map if vendor supports semantic annotations
custom_extensions Vendor-specific metadata Extract extensions matching the target vendor

Fields

Fields represent row-level attributes. They can be simple column references or computed expressions.

Note: datatype (on Field and Metric) declares a field's logical data type; dimension.is_time is an independent temporal-role marker. A field may carry both, either, or neither. Use datatype for data-type questions (casting, serialization); use is_time for role questions (classifying time dimensions). When is_time is unset it defaults to true if datatype is one of Date, Time, DateTime, DateTimeTz, and false otherwise. Explicit is_time always wins.

Ossie Field Description Converter Consideration
name Field identifier Map to column/attribute name
expression.dialects Multi-dialect SQL expressions Select the dialect matching the target vendor; fall back to ANSI_SQL
datatype Logical data type of the field (one of String, Integer, Decimal, Float, Boolean, Date, Time, DateTime, DateTimeTz, Opaque). Converters SHOULD consult datatype for the field's logical data type. Decimal is exact base-10 with unspecified precision and scale; Float is approximate. Prefer the temporal members (Date, Time, DateTime, DateTimeTz) to classify time dimensions. Omit datatype when unknown; use Opaque + custom_extensions for a known type outside the portable vocabulary.
dimension.is_time Temporal-role marker. When true, the field should be treated as a time dimension regardless of its datatype (e.g. an Integer year grain, a String month name, or a Date column). When unset, defaults to true for temporal datatypes (Date, Time, DateTime, DateTimeTz) and false otherwise. Map to vendor-specific time dimension markers. Converters SHOULD classify as a time dimension when is_time resolves to true (either explicit or defaulted from a temporal datatype). An explicit is_time: false suppresses the time-dimension classification even on temporal-typed columns.
label Categorization label Map if vendor supports field labels/tags
description Human-readable description Most vendors support field descriptions
ai_context Synonyms and business context Map if vendor supports semantic annotations

Expression dialect selection: A field may provide expressions in multiple dialects. For example:

expression:
  dialects:
    - dialect: ANSI_SQL
      expression: LOWER(email)
    - dialect: SNOWFLAKE
      expression: LOWER(email)::VARCHAR

A Snowflake converter should pick the SNOWFLAKE dialect when available, and fall back to ANSI_SQL otherwise.

Relationships

Relationships define foreign key connections between datasets. They support both simple and composite keys.

Ossie Field Description Converter Consideration
name Relationship identifier Map to vendor's join/relationship name
from Many-side dataset Map to the referencing table
to One-side dataset Map to the referenced table
from_columns Foreign key columns (array) Positional correspondence with to_columns; handle composite keys
to_columns Primary/unique key columns (array) Must have same cardinality as from_columns

Composite key example: A composite relationship like:

from_columns: [product_id, variant_id]
to_columns: [id, variant_id]

means from.product_id = to.id AND from.variant_id = to.variant_id. The converter must generate the equivalent multi-column join in the target vendor format.

Metrics

Metrics are aggregate measures defined at the semantic model level. They can span multiple datasets via relationships.

Ossie Field Description Converter Consideration
name Metric identifier Map to vendor's measure/KPI name
expression.dialects Multi-dialect aggregate expressions Select the appropriate dialect; fall back to ANSI_SQL
datatype Logical data type of the metric result (one of String, Integer, Decimal, Float, Boolean, Date, Time, DateTime, DateTimeTz, Opaque). Converters SHOULD consult datatype to declare the result type of the aggregation. Numeric measures should distinguish exact Decimal or Integer results from approximate Float results. Omit datatype when unknown; use Opaque + custom_extensions for a known type outside the portable vocabulary.
description What the metric measures Most vendors support descriptions
ai_context Synonyms and business context Map if vendor supports semantic annotations

Cross-dataset metric example:

- name: customer_lifetime_value
  expression:
    dialects:
      - dialect: ANSI_SQL
        expression: SUM(store_sales.ss_ext_sales_price) / COUNT(DISTINCT customer.c_customer_sk)

The converter must ensure that the dataset references (store_sales, customer) are resolved correctly in the target vendor's format and that any required joins are established.

Custom Extensions

Custom extensions carry vendor-specific metadata as a JSON string. A converter should:

  1. On export (Ossie → Vendor): Extract extensions where vendor_name matches the target vendor, parse the data JSON, and apply the vendor-specific settings.
  2. On import (Vendor → Ossie): Capture vendor-specific settings that have no Ossie core equivalent and store them as a custom_extension entry.

Example: A Snowflake export converter would extract:

custom_extensions:
  - vendor_name: SNOWFLAKE
    data: '{"warehouse": "ANALYTICS_WH", "database": "PROD", "schema": "PUBLIC"}'

and apply warehouse, database, and schema to the appropriate places in the Snowflake model definition.

AI Context

The ai_context field appears at every level (model, dataset, field, relationship, metric). It can be a simple string or a structured object:

# Simple string
ai_context: "orders, purchases, sales"

# Structured object
ai_context:
  instructions: "Use this for sales analysis"
  synonyms:
    - "orders"
    - "purchases"
  examples:
    - "Show total sales last month"

A converter should map ai_context when the target vendor supports equivalent constructs (e.g., LLM instructions, column descriptions with business terms). When the target vendor does not support AI annotations natively, the converter may encode them in custom_extensions on import to avoid data loss.

Writing a Converter

Step-by-Step Guide

  1. Validate input: Use the Ossie JSON Schema and the validation script to ensure the source Ossie model is valid before conversion.

  2. Parse the Ossie model: Load the YAML file and iterate over the top-level semantic_model entries.

  3. Map datasets: For each dataset, translate the name, source, primary_key, unique_keys, and fields to the vendor's format. Parse the source string (typically database.schema.table) into the vendor's catalog structure.

  4. Map fields with dialect selection: For each field, select the expression dialect that matches the target vendor. Implement a fallback chain:

    • Prefer the vendor-specific dialect (e.g., SNOWFLAKE for a Snowflake converter)
    • Fall back to ANSI_SQL if the vendor dialect is not available
    • Raise a warning or error if neither is available

  5. Map relationships: Translate relationship definitions into the vendor's join syntax. Ensure composite key column ordering is preserved.

  6. Map metrics: Translate metric expressions using the same dialect selection logic as fields. Resolve dataset references (e.g., store_sales.ss_ext_sales_price) into the vendor's qualified column format.

  7. Apply custom extensions: Extract custom_extensions entries matching the target vendor_name and apply vendor-specific settings to the output.

  8. Preserve AI context: Map ai_context to vendor-equivalent annotations where supported.

  9. Validate output: Verify the generated vendor model is valid according to the vendor's own schema or tooling.

Handling Edge Cases

Scenario Recommended Approach
Missing vendor-specific dialect for a field/metric Fall back to ANSI_SQL; log a warning
Computed fields with vendor-specific SQL syntax Require the vendor dialect in the source Ossie model; error if neither vendor nor ANSI dialect is available
Composite primary keys Ensure the vendor format supports composite keys; if not, flatten or document the limitation
Cross-dataset metrics referencing multiple tables Ensure all referenced datasets exist and relationships are defined; resolve qualified column names
Custom extensions with unknown vendor Ignore (do not discard) — preserve for round-tripping
ai_context on a vendor that doesn't support it Store in a custom_extension with vendor_name: COMMON to preserve for round-tripping

Round-Trip Fidelity

A well-implemented converter pair (import + export) should preserve as much information as possible during round-tripping:

Vendor A model → [Import] → Ossie model → [Export] → Vendor A model

To achieve this:

  • Never discard information silently. If a vendor-specific attribute has no Ossie core equivalent, store it in custom_extensions.
  • Preserve field ordering where possible, as some vendors are sensitive to declaration order.
  • Preserve custom_extensions for all vendors, not just the target vendor. This allows a model to carry metadata for multiple vendors simultaneously (e.g., a single Ossie model with both Snowflake and dbt extensions).

Example: Conceptual Conversion Flow

Given the TPC-DS example included in this repository, a Snowflake export converter would:

  1. Read the tpcds_retail_model semantic model.
  2. Create a Snowflake semantic model with name tpcds_retail_model.
  3. Map each dataset (store_sales, date_dim, customer, item, store) to Snowflake table references, parsing source values like tpcds.public.store_sales into Snowflake's database.schema.table format.
  4. For each field, select the ANSI_SQL dialect expression (since no SNOWFLAKE dialect is provided in this example).
  5. Translate relationships into Snowflake join definitions, mapping from_columns and to_columns pairs.
  6. Translate metrics like total_sales (SUM(store_sales.ss_ext_sales_price)) into Snowflake metric definitions.
  7. Extract the SALESFORCE and DBT custom extensions — these would not be applied to the Snowflake output but should be preserved if the model is later imported back to Ossie.

Contributing a New Converter

To add support for a new vendor:

  1. Use a stable vendor_name string in each custom extension emitted by the converter.
  2. Define the custom extension schema for the vendor (what vendor-specific metadata fields are supported in the data JSON).
  3. Implement the export converter (Ossie → Vendor).
  4. Implement the import converter (Vendor → Ossie).
  5. Add tests using the TPC-DS example model as a baseline.
  6. Document any limitations or unsupported constructs.