Skip to content

Group by fixes action files#2248

Open
kylershih0614 wants to merge 5 commits into
ubccr:mainfrom
kylershih0614:group_by_fixes_action_files
Open

Group by fixes action files#2248
kylershih0614 wants to merge 5 commits into
ubccr:mainfrom
kylershih0614:group_by_fixes_action_files

Conversation

@kylershih0614

Copy link
Copy Markdown
Collaborator

Description

Edited configuration/etl/etl_action_defs.d folder to abide by SQL strict mode.
Added "processorbucket_id", "gpubucket_id" fields in configuration/etl/etl_action_defs.d/jobs/hpc-aggregation-day.json.
Added "sp.organization_id", "act.account_id" fields in configuration/etl/etl_action_defs.d/cloud_generic/instance.json.
Wrapped date and times in MIN() inside of configuration/etl/etl_action_defs.d/cloud_generic/instance_type.json and added "description" as a field in the group by.
Added "raw.project_name" field in configuration/etl/etl_action_defs.d/cloud_openstack/account.json.
Added "act.account_id", "sp.organization_id" fields in configuration/etl/etl_action_defs.d/cloud_openstack/instance.json.
Wrapped the denominator inside of configuration/etl/etl_action_defs.d/jobs/hpc-aggregation-day.json with a NULLIF().
Wrapped the denominator inside of configuration/etl/etl_action_defs.d/jobs/hpc-aggregation.json with a NULLIF() and added "processorbucket_id", "gpubucket_id" fields.
Added rs.memory_mb, rs.vcpus fields to the SQL group by in configuration/etl/etl_sql.d/cloud_common/resource_specification_end_day.sql.

Motivation and Context

This change was addressing the ONLY_FULL_GROUP_BY error when running individual tests on these files. This solves the issue of these action files failing when tested with FULL_GROUP_BY turned on. The records fields were not all in the group by sections and they need to be added to abide by ONLY_FULL_GROUP_BY.

Tests performed

When running the automated test suite with ONLY_FULL_GROUP_BY turned on, the tests fail.

I tested my changes using this individual test php /usr/share/xdmod/tools/etl/etl_overseer.php -a xdmod.<pipeline_name> .<name_of_action> -v debug

I tested every single one of the files that had a group by statement in configuration/etl/etl_action_defs.d with the aforementioned command and all of them passed without issue.

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@kylershih0614 kylershih0614 added this to the 11.5.0 milestone Jun 18, 2026
@kylershih0614 kylershih0614 self-assigned this Jun 18, 2026
@kylershih0614 kylershih0614 added enhancement Enhancement of the functionality of an existing feature Category:ETL Extract Transform Load Category:Cloud Cloud Realm labels Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Category:Cloud Cloud Realm Category:ETL Extract Transform Load enhancement Enhancement of the functionality of an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant