Upgrading to use Databricks API v2.0

[the-jackalope]

Hey @brkyvz !

I know the Databricks 2.0 API doesn't support all of the functionality of the 1.2 API, but would a PR that implemented some 2.0 functionality be welcome? Specifically the better cluster management tools that 2.0 supports would be very useful for our team.

I can't promise we'll put a PR in extremely soon, but I wanted to gauge whether that's something you'd be interested in merging into the project?

Thanks

[justinmills]

According to this, the library api is deprecated as of 2018, which correlates with my experience of being unable to run dbcUpload commands as part of our release process this year.

Is there any plans to upgrade this plugin to use the 2.0 api or is there a preferred workaround to get build artifacts into the Databricks ecosystem?

[jpsimen]

I agree. I would love to use the sbt tool for deployment, but because of the deprecated REST API it's not possible.
Can someone point out, what needs to be done? If it's not too much, I can invest some time to work on a pull request.

[justinmills]

I'm not sure this is possible in the 2.0 api. I've been looking over the 2.0 docs and cannot find where you can upload a jar anymore.

You might be able to use the DBFS api to upload the jar to DBFS and then attach it from there, but I can only find how to attach a jar from DBFS to a job, not an interactive cluster.

[brkyvz]

Yeah, the biggest reason that I hadn't implemented a v2 version was because of a behavior change in the way libraries work through the REST API.

I would really appreciate it if you could submit a PR for it! It would have to follow the following pattern:

1. Require you to upload the jar to DBFS
2. Use that DBFS path to attach the jar to the cluster.

It will no longer show up in the Workspace.

[assiotis]

I submitted a PR (#45) to get dbcUpload working again under the 1.2 API. I verified that at least libraries/list and libraries/delete still work but not the other calls