Templating args passed to klio-cli are not transmitted to klioexec

[pomdtr]

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When running the klio job run command with template args:

klio job run --config-file config/streaming/klio-job-stage.yaml \
  --template dailymotion_api_key=$DAILYMOTION_API_KEY \
  --template dailymotion_api_secret=$DAILYMOTION_API_SECRET \
  --direct-runner --force-build

The template args are not passed to klioexec entrypoint:

'command': ['run',
             '--image-tag',
             '1c6edd0a-dirty',
             '--direct-runner',
             '--config-file',
             '/usr/src/config/materialized_config.yaml']

Causing the following exception

klio_core.exceptions.KlioConfigTemplatingException: 'dailymotion_api_key' missing in key template overrides.

What you expected to happen

Template args should be passed to the klioexec command

How to reproduce (as minimally and precisely as possible)

Run any command involving the template args.

[pomdtr]

Ok looking at it further it looks like /usr/src/config/materialized_config.yaml correctly resolve the secrets. But why is the preprocessing step running again in klioexec ?

[pomdtr]

Klioexec is trying to process the build time config in order to compare it with the run time config. However it is not able to process it due to the missing template args.

[riordan]

Great catch; thank you! We'll dig into this.

Is this a blocking issue for you or is there a workaround in the interim?

[pomdtr]

We are not sure we want to use the template feature to pass secrets to the pipelines. I currently use a wrapper to use directly klioexec instead, and pass the the secrets through the --env-file docker parameter.

However templating would still be useful to handle our different environment, so this issue is a bit problematic for us.

[DanSimon]

Hey @pomdtr, can you include the versions of klio-exec, klio, and klio-core you're using in your image? Can you also include whether your project is using setup.py or FnApi for its deployment.

Also to verify, is the KlioConfigTemplatingException happening as soon as klioexec starts up, or is the error coming from a worker after the pipeline has started?

[pomdtr]

• klio-cli -> v1.0.5
• klio-exec -> 0.2.2
• klio-core -> 0.2.2
• klio -> 0.2.4

Our project is using the worker_harness_container_image beam option, we are not defining any setup.py.

The error is triggered as soon as the klioexec command starts up.

[pomdtr]

Full traceback:

INFO:root:Found worker image: gcr.io/dailymotion-data-backbone-stg/klio-frames-worker:1c6edd0a-dirty
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 159, in _apply_templates
    return template.substitute(**templates)
  File "/usr/local/lib/python3.6/string.py", line 130, in substitute
    return self.pattern.sub(convert, self.template)
  File "/usr/local/lib/python3.6/string.py", line 123, in convert
    return str(mapping[named])
KeyError: 'dailymotion_api_key'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/klioexec", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/klio_core/utils.py", line 238, in wrapper
    func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/klio_exec/cli.py", line 80, in run_pipeline
    if _compare_runtime_to_buildtime_config(klio_config) is False:
  File "/usr/local/lib/python3.6/site-packages/klio_exec/cli.py", line 62, in _compare_runtime_to_buildtime_config
    buildtime_config = config.KlioConfig(_get_config(buildtime_config_path))
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/core.py", line 101, in __init__
    raw_override_list=raw_overrides or [],
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 241, in process
    raw_config_data, template_dict
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 161, in _apply_templates
    raise exceptions.KlioConfigTemplatingException(e)
klio_core.exceptions.KlioConfigTemplatingException: 'dailymotion_api_key' missing in key template overrides.

[fallonchen]

@pomdtr The fix for this specific issue should be included in our latest release candidate. Would you mind giving them a try? You can get them by setting your klio-* libraries to version 21.2.0rc1.

Note that we uncovered another issue while investigating this one - basically, all klio subcommands won't work with templated values unless you pass along the templated argument. So when running something like klio message publish, you will probably encounter another KlioConfigTemplatingException. We are working on a fix now, but for now you can work around it by passing in the template argument klio message publish --template templated_value=whatever.

[pomdtr]

Thanks a lot! I will check it as soon as possible :)

[alecgarza96]

Is this still an issue?