CGO dependency as adoption blocker for C++ engines

[metegenez]

Hey folks,
I've been working on integrating ADBC into a StarRocks. The ADBC C API and driver manager are great, clean, embeddable, easy to work with.
The problem is that the moment you need an actual driver (Flight SQL, Snowflake, etc.), you hit a CGO dependency. For C++ query engines, this is a real friction point:

• Extra Go toolchain in an already complex C++ build system
• Go runtime (GC, goroutine scheduler) embedded inside a latency-sensitive C++ process
• Cross-compilation and static linking headaches

I know there was an earlier C++ Flight SQL driver attempt (apache/arrow#14082) that was dropped in favor of the Go implementation. Since then the c/driver/framework has matured a lot, and Arrow C++ Flight libraries are solid.
Would the community be open to pure C++ driver implementations, starting with Flight SQL? I'm an Arrow committer and happy to help the work, just want to check if this direction makes sense before diving in.

I never worked on this library before, so there are a lot of experienced people here to guide me at least.

[lidavidm]

The C++ implementation hit unsolvable dependency issues, unfortunately. If you ever plan on using Protobuf, gRPC, or possibly arrow-cpp inside your application, it's possible you'll run into the same issues.

Could you expand on the latency concern? I'm not sure how the Go runtime would add meaningful latency compared to, say, sending a query across the network to the cloud, waiting for it, and fetching the result. (If you're worried about the Go runtime taking resources from the host, we can explore ways to control that / gRPC-C++ itself effectively has a runtime with its own thread pools, too; and I'm not sure a driver built with either would be actually unloadable/dlclose-able due to global state.)

That said, if there's enough contributors willing to maintain a C++ version of the Flight SQL driver, I don't think I'd say there's no possibility; I'd be curious if you plan on porting all the drivers, though, and what you envision for build/test/distribution.

[metegenez]

Thanks for the context on the protobuf/gRPC issues, that makes sense, I'll avoid the C++ driver path than. StarRocks/starrocks#70297 I tried to push ADBC into Starrocks, but i feel there is a pushback. What do you think on this? Maybe I should improve how I am loading the library? I guess I did a bit wrong on static linking here? https://github.com/StarRocks/starrocks/pull/70297/changes#diff-803104e5c983ecaeacc02d30e00b0406b63c3179f268bc67228e348e7cf9779bR103 , I couldnt make dlopen work.

[metegenez]

Did I feel the pushback unnecessarily, or is there a real reason behind it? I'd to use ADBC on starrocks but I am having hard time to address the concern.

[zeroshade]

• Extra Go toolchain in an already complex C++ build system
• Cross-compilation and static linking headaches

Another question I have is that the point of using ADBC, like ODBC before it, is so that you don't have to statically link the driver in. Instead you would use the AdbcDriverManager and then just load the driver on the fly so that you can work with any driver instead of being tied to only a single one. Is there a reason you're trying to include the Go Snowflake ADBC driver in your build process rather than just treating the .so as a black box to dynamically load by pulling it from a release? (either by dbc or by a Github release, etc.)

[metegenez]

You're right and that's exactly the approach I'm planning to take. Build-time is solved with pre-built .so + driver manager, no argument there. How about runtime? What trade-offs should i know by using cgo underhood?

[zeroshade]

For the purposes of integrating ADBC, the fact that a particular driver might be using cgo under the hood is honestly irrelevant. Either the driver is performant enough or not, but that has no bearing on the overall utility of ADBC as a whole (plenty of drivers available, only a subset of them use Go/cgo).

The entire point of using the pre-built .so files and driver manager at runtime is that it doesn't matter what the driver is actually written using.