Functional
Though importing per row works fine for simple imports, we sometimes have things that have to go into a certain order:
- products need to be created
- these products sometimes from kit-products (using earlier created products)
- products need to be published once step 1 and 2 are done
Technical
Building on top of #7, where we built in a delay and retry logic, we need something more sophisticated.
We can use a pattern described here: https://medium.com/aaron-renner/building-workflows-with-sidekiq-batches-903937abccb0
Every importer can optionally define a workflow. The default workflow is one existing of one task: build (the method in the importer). Workflows can be defined as follows:
orders_importer.rb:
products_importer.rb:
workflow %i[build build_kits event]
Each entry in the workflow describes the action that will run in a batch next. The first entry will run immediately in a batch, the second entry when the first batch completes, the third when the second batch completes, etc.
Relevant code will be the import! and the process_data_row methods of the Importable concern.
For each workflow step the relevant method needs to be called in the importer, taking the same parameters as the build method. It's up to the method to decide if it needs to do something. We also need a around_save 'callback' method.
The delay and Importo::RetryError logic should go.
The total/count (for signum) logic should be changed a bit: the total should match the number of workflow steps x number-of-rows. Every call to a workflow step (build/event) should increate the count.
See:
https://github.com/sidekiq/sidekiq/wiki/Really-Complex-Workflows-with-Batches
and
https://github.com/bensheldon/good_job?tab=readme-ov-file#complex-batches
Functional
Though importing per row works fine for simple imports, we sometimes have things that have to go into a certain order:
Technical
Building on top of #7, where we built in a delay and retry logic, we need something more sophisticated.
We can use a pattern described here: https://medium.com/aaron-renner/building-workflows-with-sidekiq-batches-903937abccb0
Every importer can optionally define a workflow. The default workflow is one existing of one task: build (the method in the importer). Workflows can be defined as follows:
orders_importer.rb:
products_importer.rb:
Each entry in the workflow describes the action that will run in a batch next. The first entry will run immediately in a batch, the second entry when the first batch completes, the third when the second batch completes, etc.
Relevant code will be the
import!and theprocess_data_rowmethods of the Importable concern.For each workflow step the relevant method needs to be called in the importer, taking the same parameters as the
buildmethod. It's up to the method to decide if it needs to do something. We also need a around_save 'callback' method.The delay and
Importo::RetryErrorlogic should go.The total/count (for signum) logic should be changed a bit: the total should match the number of workflow steps x number-of-rows. Every call to a workflow step (build/event) should increate the count.
See:
https://github.com/sidekiq/sidekiq/wiki/Really-Complex-Workflows-with-Batches
and
https://github.com/bensheldon/good_job?tab=readme-ov-file#complex-batches