Trigger Materialization after uploading a synapse table

[Mohinta2892]

Hi @jakobtroidl,

I have updated a synapse table and the same can now be found and queried via client.annotation.get_* methods.

However, I am unable to trigger a materialization of the same and create a version.

This is perhaps an extremely naive question, but how do we successfully trigger a materialization?

Should this be enough to do so:

client.materialize.ingest_annotation_table(destination_table_name)

BW.
Samia

[jakobtroidl]

Hi Samia, am I understanding you correctly that you want to trigger a materialization for a new synapse table, that you created ? @jinhan has been working on materializing synapse tables for a while (it's not trivial bc the tables are large). If that's what you want to do, I am afraid that iirc there's no simple solution right now. The best person to ask would be some of the CAVE developers, such as Forrest Collman.

[Mohinta2892]

Hi @jakobtroidl,

Thanks. Yes, your understanding is correct. I can see that the table ingests correctly, at least looks correct in the AnnotationEngine.
However, there are no materialized views of the same. Since there are no error logs on my end, I was wondering if there are any known points of failure.

[jinhan]

Hi Samia, how many rows are in your synapse table?
After you upload the annotation table, you need to run the complete_workflow API (api/v2/materialize/run/complete_workflow/datastack/{datastack_name}) from the Materialization Engine API to trigger materialization.
If the table has over 1 million rows, complete_workflow will skip the table. In that case, you will need to run the spatial_lookup workflow (/api/v2/run/spatial_lookup/datastack/{datastack_name}/{table_name}).

[Mohinta2892]

Hi @jinhan,

My table has ~3.5M rows. Thank you for the details, I shall try to run the spatial_lookup workflow.