Users would like to be able to unit test code in the workflows.
Testing adaptor logic is super hard because those require system integrations - those are integration tests.
But the pure functions within a workflow can and should be unit testable
The Problem
Users cannot run unit tests on job code right now without effort. Job expressions are not executable javascript: try to import them into mocha or jest and they'll throw exceptions.
Compiling the code (using the CLI for example) gets you a lot closer: now you have import statements and the code can be safely imported into another JS file.
But you can only compile a single workflow or expression at a time, and the devx of this is generally not very good.
Solutions
These can be done in any order I think?
1. Compile whole projects
The openfn compile command can right now only compile a workflow or expression.
We want to be able to just run openfn compile and it'll automatically compile all workflows the current repo and write them to disk.
Compiled js files should be emitted to a folder like compiled or dist at the top of the project. Optionally uses can provide an output path (usually the CLI uses -o for output, so make sure we can accept a folder here. We might also add an option to openfn.yaml for the default compile path.
Refer to how cli execute works as a guide. The projects package knows how to parse and read a project file system and return the active workflows. Use that.
2. Add a compiler flag to emit only pure JS
It may be desirable to strip operation code, and maybe run other optimisations, while compiling code for testing. Perhaps we only want to compile exported value. This would help skip large data structures from compilation and keep the compiled code generally tidier.
To do this, a new flag like --test (or something else, if that's confusing) needs to be added to the command. This should work when compiling steps, workflows or projects.
I would recommend that import statements are still added to the pure JS files. For example if using the exported dateFns function of adaptor, the compiled code should still import { datefns } from '@openfn/language-common'
3. Add --watch command to compile
Compiler should listen for changes on step and workflow files and re-compile when they are modified.
4. Document
Once verified this approach should be documented somewhere for other users to follow.
Consider adding to the project template.
Users would like to be able to unit test code in the workflows.
Testing adaptor logic is super hard because those require system integrations - those are integration tests.
But the pure functions within a workflow can and should be unit testable
The Problem
Users cannot run unit tests on job code right now without effort. Job expressions are not executable javascript: try to import them into mocha or jest and they'll throw exceptions.
Compiling the code (using the CLI for example) gets you a lot closer: now you have import statements and the code can be safely imported into another JS file.
But you can only compile a single workflow or expression at a time, and the devx of this is generally not very good.
Solutions
These can be done in any order I think?
1. Compile whole projects
The
openfn compilecommand can right now only compile a workflow or expression.We want to be able to just run
openfn compileand it'll automatically compile all workflows the current repo and write them to disk.Compiled js files should be emitted to a folder like
compiledordistat the top of the project. Optionally uses can provide an output path (usually the CLI uses-ofor output, so make sure we can accept a folder here. We might also add an option toopenfn.yamlfor the default compile path.Refer to how
cli executeworks as a guide. Theprojectspackage knows how to parse and read a project file system and return the active workflows. Use that.2. Add a compiler flag to emit only pure JS
It may be desirable to strip operation code, and maybe run other optimisations, while compiling code for testing. Perhaps we only want to compile exported value. This would help skip large data structures from compilation and keep the compiled code generally tidier.
To do this, a new flag like
--test(or something else, if that's confusing) needs to be added to the command. This should work when compiling steps, workflows or projects.I would recommend that import statements are still added to the pure JS files. For example if using the exported
dateFnsfunction of adaptor, the compiled code should stillimport { datefns } from '@openfn/language-common'3. Add
--watchcommand to compileCompiler should listen for changes on step and workflow files and re-compile when they are modified.
4. Document
Once verified this approach should be documented somewhere for other users to follow.
Consider adding to the project template.