Abstract perform_https_request so that it can be used in Wasm context

[carlopi]

This is something I touched on recently while looking at what would it take to make possible for https://github.com/duckdb/uc_catalog to work in a Web context.

Problem is basically how to wrap a minimal API that allows to make requests either natively OR from WebAssembly (there Web and Node might be two slightly different challenges).

I think I know what's the way forward, and implementing a curl wrapper (say as a header) would allow to solve the problems in:

• duckdb/uc_catalog
• evidence-dev/duckdb_gsheets
  and possibly elsewhere.

I think it's likely on me, I can ping when I have a solution.

Given it's currently not functional, I opened a PR to disable Wasm, @archiewood, WDYT? (duckdb/community-extensions#169)

[archiewood]

Agreed, I would love to bring this to WASM.

I'm not particularly happy with the implementation of perform_http_request() using OpenSSL as it is. It feels pretty dirty.

I was experimenting with libcurl but couldn't get it to work

[pspeter3]

WASM support for GSheets would be awesome!

[kellington]

Agree. Allowing SQL on a Google Sheet via shell. would be fantastic

[carlopi]

Hello!
I might have something to share, I have a fork of duckdb_gsheets that is compatible with the WasmHttpLib, I do need to rework a bit but will send a PR soon.

With that in, and disabling CORS, I get:

(I provided the token manually, possibly the token generation also could be adapted to be browser-aware, but that's just convenience).

Now the main problem: what would be a reasonable setup for a CORS-proxy? Or even better, are there alternatives to setting up a proxy for https://sheets.googleapis.com?

[carlopi]

Branch is at https://github.com/carlopi/duckdb_gsheets/tree/http_util, some more tweaks needed but almost there

[archiewood]

Nice.

As a side question, what formatter do you use? I'm such a C novice that I have no idea how this works.

In js i would just have a prettier.config.js file and npm run format, but is there an equivalent that I should add for C++?

When it was just me working on this code, formatting seemed not an issue, but now every PR has gnarly diffs

I think i figured this out from your commits, install clang-format, cmake format, then

make format

[archiewood]

Now the main problem: what would be a reasonable setup for a CORS-proxy? Or even better, are there alternatives to setting up a proxy for https://sheets.googleapis.com/?

As I understand it this would only be an issue for POST requests to the google sheets API, so reading would work, but writing would be blocked without a CORS solution?

This might be an acceptable outcome.

I suspect the challenge with a CORS proxy is who would host it. I don't think users would want to push requests through my server, and I wouldn't really want to run such a server. We could include minimal examples on how to run one, but that makes this hard to use.

[carlopi]

Now the main problem: what would be a reasonable setup for a CORS-proxy? Or even better, are there alternatives to setting up a proxy for https://sheets.googleapis.com/?

As I understand it this would only be an issue for POST requests to the google sheets API, so reading would work, but writing would be blocked without a CORS solution?

No, as far as I am aware this is an issue for all requests, both read, write and even accessing metadata.

There might be a path around CORS requirements, but that requires skipping Authorization header, that might be theoretically OK for public sheets BUT when I tried I still got: "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API" from the sheets API.

I think reading publicly available sheets might be workable without fixing CORS by changing the API calls.

I suspect the challenge with a CORS proxy is who would host it. I don't think users would want to push requests through my server, and I wouldn't really want to run such a server. We could include minimal examples on how to run one, but that makes this hard to use.

Yes, I think this is also the main problem, I was thinking we might be able to setup such a server at duckdb, even just to demo this, but then it would need clear opt-in from users and comes with extra surface.

[pspeter3]

For private sheets, do you need CORS? Google Drive / Sheets already has JS API support.

[mharrisb1]

The abstraction piece of this was implemented in #100 but actual WASM support is still blocked on HTTPUtil implementing the missing HTTP verbs.

Closing for now but feel free to reopen. I'll create a separate issue specifically for WASM support