diff --git a/roles/telemetry_chargeback/README.md b/roles/telemetry_chargeback/README.md index fde62542e..ae93799bb 100644 --- a/roles/telemetry_chargeback/README.md +++ b/roles/telemetry_chargeback/README.md @@ -64,6 +64,24 @@ These variables can be overridden when importing the role or set at the play lev ### Synthetic Data Scripts +These variables are used internally by the role and should not be modified. They use `role_path` for internal file/script references and define internal file naming conventions. + +| Variable | Default Value | Description | +|----------|---------------|-------------| +| `cloudkitty_scenario_dir` | `{{ role_path }}/files` | Directory containing scenario files (`test_*.yml`). | +| `cloudkitty_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. | +| `cloudkitty_data_template` | `{{ role_path }}/templates/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. | +| `cloudkitty_summary_script` | `{{ role_path }}/files/gen_db_summary.py` | Path to the summary script (gen_db_summary.py). | +| `cloudkitty_synth_data_suffix` | `-synth_data.json` | Suffix for generated synthetic data files. | +| `cloudkitty_loki_data_suffix` | `-loki_data.json` | Suffix for Loki query result JSON files. | +| `cloudkitty_synth_totals_metrics_suffix` | `-synth_metrics_summary.yml` | Suffix for generated metric totals files (from synthetic data). | +| `cloudkitty_loki_totals_metrics_suffix` | `-loki_metrics_summary.yml` | Suffix for metric totals computed from Loki-retrieved JSON (retrieve_loki_data task). | +| `cloudkitty_loki_totals_suffix` | `-rating.yml` | Suffix for CloudKitty rating summary output files (from loki_rate task). | + +**Note:** Loki push/query URLs are set dynamically in `setup_loki_env.yml` from the Cloudkitty Loki route. + +### Synthetic Data Scripts + **gen_synth_loki_data.py** — Generates Loki-format JSON from a scenario YAML and template. The role invokes it with `-r` so that timestamps in the output are in **reverse** order (youngest first, oldest last). When run manually you can omit `-r` for chronological order (oldest first, youngest last). | Option | Description | @@ -90,23 +108,6 @@ Output YAML structure: * **data_log** — `total_timesteps`, `metrics_per_step`, `log_count`. * **rate** — `by_types` (per-type `Rate` calculated as `Σ((qty_mutated * factor + offset) * price)`) and `total.Rating` (sum of all rates). -### Dynamically Set Variables - -Set in **main.yml** from the OpenStack CLI (`openstack project show admin` / `openstack user show admin`): - -| Variable | Description | -|----------|-------------| -| `cloudkitty_project_id` | ID of the OpenStack project named `admin` (empty string if not found). Passed as `-p` to the synthetic data generator when non-empty. | -| `cloudkitty_user_id` | ID of the OpenStack user named `admin` (empty string if not found). Passed as `-u` to the synthetic data generator when non-empty. | - -Set in **gen_synth_loki_data.yml** for each scenario file during the loop: - -| Variable | Description | -|----------|-------------| -| `cloudkitty_data_file` | Local path for generated JSON data (`{{ artifacts_dir_zuul }}/{{ scenario_name }}-synth_data.json`) | -| `cloudkitty_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}-synth_metrics_summary.yml`) | -| `cloudkitty_test_file` | Path to the scenario configuration file (`{{ role_path }}/files/{{ scenario_name }}.yml`) | - Scenario Configuration ---------------------- The synthetic data generation is controlled by YAML configuration files in the `files/` directory. Any file matching `test_*.yml` will be automatically discovered and processed. Files whose names start with an underscore (e.g. `_test_*.yml`) are **not** discovered by the role; they can be used as reference or for manual runs. diff --git a/roles/telemetry_chargeback/files/gen_synth_loki_data.py b/roles/telemetry_chargeback/files/gen_synth_loki_data.py index 263554dc6..e3e0f5e12 100755 --- a/roles/telemetry_chargeback/files/gen_synth_loki_data.py +++ b/roles/telemetry_chargeback/files/gen_synth_loki_data.py @@ -1,3 +1,4 @@ +### Generated by Cursor """Generate synthetic Loki log data from a Jinja2 template.""" import logging import argparse @@ -263,7 +264,10 @@ def generate_loki_data( "qty": qty_list, "price": price_list, "groupby": groupby.copy(), - "metadata": log_type_config.get("metadata", {}) + "metadata": log_type_config.get("metadata", {}), + "factor": log_type_config.get("factor", 1), + "offset": log_type_config.get("offset", 0), + "mutate": log_type_config.get("mutate", "NONE") } # --- Step 3: Load template and render --- diff --git a/roles/telemetry_chargeback/tasks/loki_rate.yml b/roles/telemetry_chargeback/tasks/loki_rate.yml index 822585336..d96a4564a 100644 --- a/roles/telemetry_chargeback/tasks/loki_rate.yml +++ b/roles/telemetry_chargeback/tasks/loki_rate.yml @@ -18,7 +18,7 @@ - name: "TEST Get Rate and Qty Summary from CloudKitty {{ item }}" ansible.builtin.command: - cmd: "{{ openstack_cmd }} --rating-api-version 2 rating summary get -f yaml" + cmd: "{{ openstack_cmd }} --rating-api-version 2 rating summary get -f yaml -b \"{{ synth_data_rates.time.begin_step.begin }}\" -e \"{{ synth_data_rates.time.end_step.end }}\"" register: cost_totals_summary changed_when: false failed_when: cost_totals_summary.rc != 0 diff --git a/roles/telemetry_chargeback/tasks/retrieve_loki_data.yml b/roles/telemetry_chargeback/tasks/retrieve_loki_data.yml index 2f130e711..adaa2b34a 100644 --- a/roles/telemetry_chargeback/tasks/retrieve_loki_data.yml +++ b/roles/telemetry_chargeback/tasks/retrieve_loki_data.yml @@ -67,5 +67,5 @@ -j "{{ artifacts_dir_zuul }}/{{ item }}{{ cloudkitty_loki_data_suffix }}" -o "{{ artifacts_dir_zuul }}/{{ item }}{{ cloudkitty_loki_totals_metrics_suffix }}" --debug "{{ cloudkitty_debug_dir }}" - register: synth_rating_info - changed_when: synth_rating_info.rc == 0 + register: loki_retrieved_summary_info + changed_when: loki_retrieved_summary_info.rc == 0 diff --git a/roles/telemetry_chargeback/templates/loki_data_templ.j2 b/roles/telemetry_chargeback/templates/loki_data_templ.j2 index b676f3013..c7f96f0c0 100644 --- a/roles/telemetry_chargeback/templates/loki_data_templ.j2 +++ b/roles/telemetry_chargeback/templates/loki_data_templ.j2 @@ -13,7 +13,10 @@ "qty": entry_data.qty, "price": entry_data.price, "groupby": entry_data.groupby, - "metadata": entry_data.metadata + "metadata": entry_data.metadata, + "factor": entry_data.factor, + "offset": entry_data.offset, + "mutate": entry_data.mutate } -%} [ "{{ item.nanoseconds }}", diff --git a/roles/telemetry_chargeback/vars/main.yml b/roles/telemetry_chargeback/vars/main.yml index 27010b5dc..f694c84c6 100644 --- a/roles/telemetry_chargeback/vars/main.yml +++ b/roles/telemetry_chargeback/vars/main.yml @@ -5,6 +5,16 @@ cloudkitty_synth_script: "{{ role_path }}/files/gen_synth_loki_data.py" cloudkitty_data_template: "{{ role_path }}/templates/loki_data_templ.j2" cloudkitty_summary_script: "{{ role_path }}/files/gen_db_summary.py" +cloudkitty_synth_script: "{{ role_path }}/files/gen_synth_loki_data.py" +cloudkitty_data_template: "{{ role_path }}/templates/loki_data_templ.j2" +ck_data_config: "{{ role_path }}/files/test_static.yml" +ck_output_file_local: "{{ artifacts_dir_zuul }}/loki_synth_data.json" +ck_output_file_remote: "{{ logs_dir_zuul }}/gen_loki_synth_data.log" + +# Scenario and script paths (using role_path) +cloudkitty_scenario_dir: "{{ role_path }}/files" +cloudkitty_summary_script: "{{ role_path }}/files/gen_db_summary.py" + # File naming conventions (internal standardization) cloudkitty_synth_data_suffix: "-synth_data.json" cloudkitty_loki_data_suffix: "-loki_data.json"