Project sync fails when uploading a big file

[constantinneagu]

When uploading a big file (> 500 MB) the last step of the project sync fails with a "502 Bad Gateway" error.

Looks like something is timing out.

[tomasMizera]

Hi @constantinneagu, if possible try to split your upload to multiple steps (different syncs). We are currently working on a more robust sync, see MerginMaps/python-api-client#262. I'll close the ticket in favor of the upcoming upgrades, feel free to reach out to our support at support@merginmaps.com if you still have issues. Thank you for the report!

[constantinneagu]

Hi @tomasMizera and thanks for your answer. Unfortunately we are dealing with a single big file. Logically it could be split in multiple gpkg's, but nobody has time for this right now. Let's hope that the upcoming changes will fix the issue :)

[tomasMizera]

Is this a new file that you are trying to upload @constantinneagu?

[constantinneagu]

No, it is not a new file. I guess that the original file was smaller.
We are trying to automate some processes and this file contains the end result. It is an export from a MongoDB instance and it is changed every time we get new data.
There where no issues for the first 2-3 syncs but now we get the above error message at the very end (progress bar hits 100%). We have other, smaller files that are generated and I can sync them with no problem (after discarding the changes to the big file).
I tried starting from a clean copy of the project but to no avail.

[tomasMizera]

Let me add my colleague @varmar05 here who will further investigate what is going on here.

[tomasMizera]

Thanks for the details @constantinneagu

[varmar05]

@constantinneagu ideally we would need to know some details of what you are trying to update. From what you describe I think it is a known limitation (which we are going to address) of pushing either too many or too large updates. Not new files, but updates to .gpkg files. A large update can be either small data added to the large file (hundreds-thousands MB) or rather bigger data (tens-hundreds MB) added to any file.

Is that your case? Currently, the only workaround would be to do more but smaller uploads, possibly even splitting into multiple files. But I understand it requires manual intervention though.

[constantinneagu]

@varmar05 So we have a data processing pipeline that basically reconstructs this > 500 MB gpkg from scratch. One potential problem is that although the schema is more or less stable, the column order can change (very bad design I know) from one run to the other. From the perspective of mergin, everything changes.

We could split the file, but we can't do it right now.

I was kinda hoping that one can just increase the time interval for which that link is valid. At least for larger files.

Thanks for answering :)

[constantinneagu]

@varmar05 Following your explanations I tried:
delete the big file from the project => sync => put the big file back => sync => Success !
Like this there where no issues. :)
Thanks for your support :)

[tomasMizera]

Great! Let me close the ticket as this is a known limitation of the system and we are working on a new version.