[telemetry_chargeback] Add chargeback rating testing for Cloudops FVT (TEMP)

roles/telemetry_chargeback/README.md

@@ -64,6 +64,24 @@ These variables can be overridden when importing the role or set at the play lev
 
 ### Synthetic Data Scripts
 
+These variables are used internally by the role and should not be modified. They use `role_path` for internal file/script references and define internal file naming conventions.
+
+| Variable | Default Value | Description |
+|----------|---------------|-------------|
+| `cloudkitty_scenario_dir` | `{{ role_path }}/files` | Directory containing scenario files (`test_*.yml`). |
+| `cloudkitty_synth_script` | `{{ role_path }}/files/gen_synth_loki_data.py` | Path to the synthetic data generation script. |
+| `cloudkitty_data_template` | `{{ role_path }}/templates/loki_data_templ.j2` | Path to the Jinja2 template for Loki data format. |
+| `cloudkitty_summary_script` | `{{ role_path }}/files/gen_db_summary.py` | Path to the summary script (gen_db_summary.py). |
+| `cloudkitty_synth_data_suffix` | `-synth_data.json` | Suffix for generated synthetic data files. |
+| `cloudkitty_loki_data_suffix` | `-loki_data.json` | Suffix for Loki query result JSON files. |
+| `cloudkitty_synth_totals_metrics_suffix` | `-synth_metrics_summary.yml` | Suffix for generated metric totals files (from synthetic data). |
+| `cloudkitty_loki_totals_metrics_suffix` | `-loki_metrics_summary.yml` | Suffix for metric totals computed from Loki-retrieved JSON (retrieve_loki_data task). |
+| `cloudkitty_loki_totals_suffix` | `-rating.yml` | Suffix for CloudKitty rating summary output files (from loki_rate task). |
+
+**Note:** Loki push/query URLs are set dynamically in `setup_loki_env.yml` from the Cloudkitty Loki route.
+
+### Synthetic Data Scripts
+
 **gen_synth_loki_data.py** — Generates Loki-format JSON from a scenario YAML and template. The role invokes it with `-r` so that timestamps in the output are in **reverse** order (youngest first, oldest last). When run manually you can omit `-r` for chronological order (oldest first, youngest last).
 
 | Option | Description |
@@ -90,23 +108,6 @@ Output YAML structure:
 * **data_log** — `total_timesteps`, `metrics_per_step`, `log_count`.
 * **rate** — `by_types` (per-type `Rate` calculated as `Σ((qty_mutated * factor + offset) * price)`) and `total.Rating` (sum of all rates).
 
-### Dynamically Set Variables
-
-Set in **main.yml** from the OpenStack CLI (`openstack project show admin` / `openstack user show admin`):
-
-| Variable | Description |
-|----------|-------------|
-| `cloudkitty_project_id` | ID of the OpenStack project named `admin` (empty string if not found). Passed as `-p` to the synthetic data generator when non-empty. |
-| `cloudkitty_user_id` | ID of the OpenStack user named `admin` (empty string if not found). Passed as `-u` to the synthetic data generator when non-empty. |
-
-Set in **gen_synth_loki_data.yml** for each scenario file during the loop:
-
-| Variable | Description |
-|----------|-------------|
-| `cloudkitty_data_file` | Local path for generated JSON data (`{{ artifacts_dir_zuul }}/{{ scenario_name }}-synth_data.json`) |
-| `cloudkitty_synth_totals_file` | Local path for calculated metric totals (`{{ artifacts_dir_zuul }}/{{ scenario_name }}-synth_metrics_summary.yml`) |
-| `cloudkitty_test_file` | Path to the scenario configuration file (`{{ role_path }}/files/{{ scenario_name }}.yml`) |
-
 Scenario Configuration
 ----------------------
 The synthetic data generation is controlled by YAML configuration files in the `files/` directory. Any file matching `test_*.yml` will be automatically discovered and processed. Files whose names start with an underscore (e.g. `_test_*.yml`) are **not** discovered by the role; they can be used as reference or for manual runs.

roles/telemetry_chargeback/files/gen_synth_loki_data.py

@@ -1,3 +1,4 @@
+### Generated by Cursor
 """Generate synthetic Loki log data from a Jinja2 template."""
 import logging
 import argparse
@@ -263,7 +264,10 @@ def generate_loki_data(
             "qty": qty_list,
             "price": price_list,
             "groupby": groupby.copy(),
-            "metadata": log_type_config.get("metadata", {})
+            "metadata": log_type_config.get("metadata", {}),
+            "factor": log_type_config.get("factor", 1),
+            "offset": log_type_config.get("offset", 0),
+            "mutate": log_type_config.get("mutate", "NONE")
         }
 
     # --- Step 3: Load template and render ---

roles/telemetry_chargeback/tasks/loki_rate.yml

@@ -18,7 +18,7 @@
 
 - name: "TEST Get Rate and Qty Summary from CloudKitty {{ item }}"
   ansible.builtin.command:
-    cmd: "{{ openstack_cmd }} --rating-api-version 2 rating summary get -f yaml"
+    cmd: "{{ openstack_cmd }} --rating-api-version 2 rating summary get -f yaml -b \"{{ synth_data_rates.time.begin_step.begin }}\" -e \"{{ synth_data_rates.time.end_step.end }}\""
   register: cost_totals_summary
   changed_when: false
   failed_when: cost_totals_summary.rc != 0

roles/telemetry_chargeback/tasks/retrieve_loki_data.yml

@@ -67,5 +67,5 @@
       -j "{{ artifacts_dir_zuul }}/{{ item }}{{ cloudkitty_loki_data_suffix }}"
       -o "{{ artifacts_dir_zuul }}/{{ item }}{{ cloudkitty_loki_totals_metrics_suffix }}"
       --debug "{{ cloudkitty_debug_dir }}"
-  register: synth_rating_info
-  changed_when: synth_rating_info.rc == 0
+  register: loki_retrieved_summary_info
+  changed_when: loki_retrieved_summary_info.rc == 0

roles/telemetry_chargeback/templates/loki_data_templ.j2

@@ -13,7 +13,10 @@
     "qty": entry_data.qty,
     "price": entry_data.price,
     "groupby": entry_data.groupby,
-    "metadata": entry_data.metadata
+    "metadata": entry_data.metadata,
+    "factor": entry_data.factor,
+    "offset": entry_data.offset,
+    "mutate": entry_data.mutate
 } -%}
 [
 "{{ item.nanoseconds }}",

roles/telemetry_chargeback/vars/main.yml

@@ -5,6 +5,16 @@ cloudkitty_synth_script: "{{ role_path }}/files/gen_synth_loki_data.py"
 cloudkitty_data_template: "{{ role_path }}/templates/loki_data_templ.j2"
 cloudkitty_summary_script: "{{ role_path }}/files/gen_db_summary.py"
 
+cloudkitty_synth_script: "{{ role_path }}/files/gen_synth_loki_data.py"
+cloudkitty_data_template: "{{ role_path }}/templates/loki_data_templ.j2"
+ck_data_config: "{{ role_path }}/files/test_static.yml"
+ck_output_file_local: "{{ artifacts_dir_zuul }}/loki_synth_data.json"
+ck_output_file_remote: "{{ logs_dir_zuul }}/gen_loki_synth_data.log"
+
+# Scenario and script paths (using role_path)
+cloudkitty_scenario_dir: "{{ role_path }}/files"
+cloudkitty_summary_script: "{{ role_path }}/files/gen_db_summary.py"
+
 # File naming conventions (internal standardization)
 cloudkitty_synth_data_suffix: "-synth_data.json"
 cloudkitty_loki_data_suffix: "-loki_data.json"