feat!: v2#166
Merged
Merged
Conversation
Adds the binary compatibility validator plugin to make sure we are aware of any public facing api changes within the library. Also adds the `apiCheck` task to CI/CD. Fixes MOBILE-121
Updated the public api surface of koci to be concise and only expose
what needs to be exposed. Updated api dump to reflect this.
- data classes converted to pure classes and override
hashcode/equals/tostring when needed
- public facing classes are within the api package now
- consumers must create a koci object to create a registry(s) and
repo(s)
- removed ktor exposed classes from API, users must now configure things
through koci config objects, abstracted away from ktor
- internal package created for library needed items that consumers do
not need to know about
Below is a comparison between v1 and v2 creation of objects.
**v1**
```kotlin
val httpClient = HttpClient(CIO) {
install(OCIAuthPlugin) {
cred = Credential("user", "pass", "", "")
}
install(HttpTimeout) {
requestTimeoutMillis = 30_000
}
}
val storage = Layout.create("/tmp/store").getOrThrow()
val registry = Registry("https://ghcr.io", client = httpClient)
try {
// Pull an image
registry.pull(
repository = "library/gradle",
reference = "latest",
storage = storage,
).collect { println("$it%") }
// List tags
registry.extensions.list(n = 100).collect { resp ->
resp.tags.forEach(::println)
}
// Resolve tag to descriptor
val descriptor = registry.resolve("dos-games", "1.1.0").getOrThrow()
// Anonymous registry — new HttpClient
val docker = Registry("https://registry-1.docker.io")
// Tune concurrency / backoff / chunk size — no first-class API.
// Install plugins on HttpClient manually; thread concurrency through
// Flow operators at each call site.
} finally {
httpClient.close()
}
```
**v2**
```kotlin
Koci(root = "/tmp/store").use { koci ->
val registry = koci.registry(
url = "https://ghcr.io",
auth = AuthConfig.Basic("user", "pass"),
timeouts = TimeoutConfig(requestTimeout = 30.seconds),
backOffPolicy = BackOffPolicy.Exponential(factor = 2.0),
pull = PullConfig(concurrency = 8),
push = PushConfig(concurrency = 2, minChunkSize = 5_000_000),
)
// Pull an image
registry.repo("library/gradle")
.pull("latest")
.collect { println("$it%") }
// List tags
registry.repo("library/gradle")
.tags()
.getOrThrow()
.forEach(::println)
// Resolve tag to descriptor
val descriptor = registry.repo("dos-games").resolve("1.1.0").getOrThrow()
// Anonymous registry — shares koci's engine and connection pool
val docker = koci.registry("https://registry-1.docker.io")
// Reuse config across registries
val slow = TimeoutConfig(requestTimeout = 60.seconds)
val harbor1 = koci.registry("https://harbor-1.corp", timeouts = slow)
val harbor2 = koci.registry("https://harbor-2.corp", timeouts = slow)
}
```
| Concern | v1 | v2 |
|---|---|---|
| Entry | HttpClient + Layout.create + Registry (3 separate
constructions) | `KociV2(root = ...)` |
| Auth | Install OCIAuthPlugin on HttpClient manually | `auth =
AuthConfigV2.Basic(...)` on `registry()` |
| Timeouts | Install HttpTimeout manually | `timeouts =
TimeoutConfigV2(...)` on `registry()` |
| Storage | Passed to every `registry.pull(...)` call | Owned by KociV2,
invisible to callers |
| Lifecycle | Remember to `httpClient.close()` | `koci.use { ... }` or
DI-managed |
| Config reuse | Not possible, threaded through plugin installs |
Configs are values and are passed to N registries |
| Repo hierarchy | Extension functions on Registry | Member methods:
`registry.repo(name).pull(tag)` |
Ownership was placed in the correct spots and functions, classes, etc.
that did not need to be exposed to consumers were synched down. Storage
on disk is also managed by the koci object instead of being created
separately and then passed back in. Removes potential bugs.
These changes also allow a lot of customizability through constructor
injection. So different registries can have different backoff policies
or concurrency limits based on their priorities. All things that were
baked into v1 that were not configurable before.
No functionality as to how koci works has been changed. All of these
changes are public api changes. Which touches the whole library
essentially. This did mean however that we needed to move the `Registry`
and `Layout` tests. This is because their public api has changed and for
the work in this PR, it does not fit. Once we work on those classes in
the next set of PRs, tests will be added back testing against the new
api.
NOTE: The library is not expected to be used now and is in a broken
state before we get to hooking things back up again through the correct
pipelines.
Fixes MOBILE-193
## What
Replaces `kotlin.Result<T>`-wrapped returns and the `OCIException`
hierarchy with primitives `(Boolean, T?, List<T>)` and one small sealed
type for pull progress.
Internal-only: HTTP-error parsing, range-resume coordination, push
events. Range-resume per the OCI distribution spec is preserved.
## Not yet shippable
Pull progress reporting is intentionally inconsistent in this PR.
MOBILE-210 will fix this in a subsequent PR.
## Why
Consumers were forced into try/catch + .getOrThrow() because the public
API mixed Result<T> wrappers with thrown OCIException subclasses (digest
mismatch, platform not found, manifest unsupported, etc.).
## Before:
```kotlin
try {
val resolved = repo.resolve("v1.0").getOrThrow()
val pulled = repo.pull(resolved).collect { progress -> bar.update(progress) }
} catch (e: OCIException.PlatformNotFound) { ... }
catch (e: OCIException.ManifestNotSupported) { ... }
catch (e: OCIException.DigestMismatch) { ... }
catch (e: OCIException.IncompletePull) { ... }
catch (e: IllegalArgumentException) { ... }
```
## After:
```kotlin
val descriptor = repo.resolve("v1.0") ?: return showError()
repo.pull(descriptor).collect { event ->
when (event) {
is PullEvent.Progress -> bar.update(event.percent) // semantics fixed in MOBILE-210
is PullEvent.Completed -> done()
is PullEvent.Failed -> showError()
}
}
```
## Shape
| Signature | Before | After |
|---|---|---|
| `Repository.tags()` | `Result<List<String>>` | `List<String>` (failure
→ `emptyList()` + log) |
| `Repository.resolve(...)` | `Result<Descriptor>` (throws) |
`Descriptor?` |
| `Repository.pull(...)` | `Flow<Int>` (throws mid-flow) |
`Flow<PullEvent>` (3 variants) |
| `Repository.manifest/index(...)` | `Result<T>` | `T?` |
| `Repository.fetch<T>(...)` | `T` (throws) | `T?` |
| `Registry.catalog()` | `Result<List<Repository>>` | `List<Repository>`
(failure → `emptyList()` + log) |
| `Reference.parse(...)` | `Result<Reference>` | `Reference?` |
| `Reference.validate()` | `Unit` (throws) | `Boolean` |
| `Reference.digest()` | `Digest` (throws) | `Digest?` |
| `Digest(content: String)` ctor | public, throws | removed; use
`Digest.parse(content): Digest?` |
| `Descriptor.digest` | `Digest` | `Digest?` (malformed wire input is
recoverable) |
`koci` is not a HTTP library — a 4xx/5xx isn't something a consumer can
usefully branch on. Specific failure causes (digest/size mismatch,
registry error code, unsupported content type) are log details
(MOBILE-198), not consumer `when` branches.
## Internal changes
- `Layout.exists()` -> range-resume offset exposed via separate internal
helper partialBytesOnDisk()
- `Layout.remove()` -> boolean now and "still referenced" case logs the
descriptor.
- `Layout.push()` -> collapsed to 3 variants matching PullEvent (still
emits raw bytes — MOBILE-210).
- `failureResponseOrNull()` replaced with `HttpResponse.succeeded()`
- Koci's ktor client drops expectSuccess = true and
HttpResponseValidator; non-success status flows back as a normal
response and succeeded() decides the branch.
- Auth-plugin token fetchers (fetchOAuth2Token, fetchDistributionToken)
return String? instead of throwing; null = skip the bearer-auth attempt.
## Notes
- No throw / require / check / error / checkNotNull in our code.
- Logging is deferred - MOBILE-198 - Log markers placed at every failure
point for the eventual logging framework.
- Sets up MOBILE-210 — committing to PullEvent.Progress(val percent:
Int) forces every streaming op (currently pull, Layout.push) to converge
on percent semantics; today's mix of byte counts and real percents
flowing through that field is the bug MOBILE-210 closes. Without this
PR, MOBILE-210 would also have to undo Flow<Int>/Flow<Long> and
exception-driven termination first.
- Breaking change — koci v2 is a rewrite, expected.
Fixes MOBILE-197
Cleaned up the gradle project to allow us to expand into modularized projects. Created convention plugins to apply to gradle sub projects so they all follow the same linting and publishing rules. Also added a samples project that will be used in the future to show how koci works in different scenarios.
This work encompasses a couple things, mainly perf work. From using okio and okhttp underneath, to cleaning up how we calculate progress, this is a chunky PR. But it was all necessary and hard to split into multiple PRs. - Added a logger that can be configured - Used okio whenever possible (manipulating files, getting hashes, downloading/uploading files from disk) - OkHttp is the engine that runs within ktor and is configurable by consumers - Retry logic added to remove the burden from consumers to figure out how to wrap koci - Cleaned up auth flow logic to be leaner and quicker - Removed public functions that were either redundant or did not make sense to expose - Moved files between `api` and `internal` that were required for library consumers (we use it in an internal app) - Progress tracking is now `0-100` rather than bytes to make it easier for consumers to know where they are in the process of uploading/downloading - Cleaned up docs from being overtly verbose Below are benchmarks of v1 vs v2 vs oras: ## OCI Client Benchmark: koci v1 vs koci current vs oras-go These tests were ran on a M4 Pro MBP on a wired 2.5gbps connection on a layer 3 switch. **Date**: 2026-05-21 19:34 **Machine**: macOS-26.4.1-arm64-arm-64bit-Mach-O **JVM**: openjdk version "21.0.11" 2026-04-21 LTS **Go**: go version go1.26.3 darwin/arm64 **Iterations**: 31 ### Per-Request Overhead | Operation | oras-go mean | v1 mean | current mean | v1 vs oras | current vs oras | current vs v1 | |----------------|--------------|--------------|----------------|----------------|----------------------|------------------| | Ping | 0.6ms | 1.9ms | 1.3ms | 3.1x slower | 2.1x slower | 1.5x faster | | Catalog | 1.2ms | 2.3ms | 1.8ms | 2.0x slower | 1.5x slower | 1.3x faster | | Resolve | 0.9ms | 3.8ms | 1.0ms | 4.0x slower | ~same | 3.9x faster | | Tags | 0.8ms | 1.5ms | 1.0ms | 2.0x slower | 1.3x slower | 1.5x faster | ### Pull | Size | oras-go mean | v1 mean | current mean | v1 vs oras | current vs oras | current vs v1 | |------------------------|--------------|--------------|----------------|----------------|----------------------|------------------| | 4MB | 31.6ms | 37.5ms | 32.8ms | 1.2x slower | ~same | 1.1x faster | | 52MB | 256.4ms | 260.4ms | 257.3ms | ~same | ~same | ~same | | 474MB | 2.25s | 2.25s | 2.25s | ~same | ~same | ~same | | 1.1GB | 5.45s | 5.45s | 5.47s | ~same | ~same | ~same | | 181MB | 868.0ms | 870.0ms | 869.4ms | ~same | ~same | ~same | ### Push | Size | oras-go mean | v1 mean | current mean | v1 vs oras | current vs oras | current vs v1 | |------------------------|--------------|--------------|----------------|----------------|----------------------|------------------| | 5MB | 64.9ms | 68.7ms | 70.0ms | 1.1x slower | 1.1x slower | ~same | | 50MB | 280.5ms | 540.3ms | 529.5ms | 1.9x slower | 1.9x slower | ~same | | 500MB | 2.42s | 4.98s | 4.91s | 2.1x slower | 2.0x slower | ~same | | 1000MB | 4.81s | 9.72s | 9.61s | 2.0x slower | 2.0x slower | ~same | ### Summary | Operation | v1 vs oras-go (avg) | current vs oras-go (avg) | current vs v1 (avg) | |----------------|------------------------|----------------------------|------------------------| | Ping | 3.1x slower | 2.1x slower | 1.5x faster | | Catalog | 2.0x slower | 1.5x slower | 1.3x faster | | Resolve | 4.0x slower | ~same | 3.9x faster | | Tags | 2.0x slower | 1.3x slower | 1.5x faster | | Pull | ~same | ~same | ~same | | Push | 1.8x slower | 1.7x slower | ~same | ### Methodology - All clients tested against the same registry, run sequentially (no concurrent load) - koci v1 = published release 0.4.3 - koci current = latest dev build - Warm-up iterations discarded to account for JVM JIT and connection pool warm-up - Each pull uses a fresh temp directory (no blob cache between iterations) Fixes MOBILE-194 Fixes MOBILE-195 Fixes MOBILE-198 Fixes MOBILE-201 Fixes MOBILE-210
Benchmarks for running koci v1 vs v2 vs oras. README added to show how to get the test env setup. Fixes MOBILE-234
Prior to this PR, we were writing all blobs to their final destination. If a pull stopped mid way through, push code would see the blob existing even though it was corrupted. We now added an extra step to our pull code: temp -> final folder move. This will allow us to make sure any blobs put in the final folder by koci have been SHA + size checked. Any mid progress blobs will be separate, thus operations like push will not see them unless properly verified. This speeds up our existence check in the final folder by only doing a size check instead of a full hash check like we were doing prior. Added tests. Benchmarks below show that this change did not hinder performance. These tests were ran on a M4 Pro MBP on a wired 2.5gbps connection on a layer 3 switch. **Date**: 2026-05-26 13:30 **Machine**: macOS-26.4.1-arm64-arm-64bit-Mach-O **JVM**: openjdk version "21.0.11" 2026-04-21 LTS **Go**: go version go1.26.3 darwin/arm64 **Iterations**: 31 ### Pull | Size | oras-go mean | v1 mean | current mean | v1 vs oras | current vs oras | current vs v1 | |------------------------|--------------|--------------|----------------|----------------|----------------------|------------------| | 4MB | 32.9ms | 35.3ms | 33.9ms | 1.1x slower | ~same | ~same | | 52MB | 258.6ms | 260.7ms | 258.5ms | ~same | ~same | ~same | | 474MB | 2.28s | 2.25s | 2.25s | ~same | ~same | ~same | | 1.1GB | 5.46s | 5.44s | 5.44s | ~same | ~same | ~same | | 181MB | 863.6ms | 868.0ms | 865.5ms | ~same | ~same | ~same |
brianroper
approved these changes
Jun 3, 2026
LandonPatmore
pushed a commit
that referenced
this pull request
Jun 4, 2026
🤖 I have created a release *beep* *boop* --- ## 0.5.0 (2026-06-03) ## What's Changed * chore: removed workflows only running on main by @LandonPatmore in #152 * chore: update CODEOWNERS to uds-fleet and add SECURITY.md by @LandonPatmore in #165 * feat!: v2 by @LandonPatmore in #166 **Full Changelog**: v0.4.3...v0.5.0 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings in all changes for v2. Additional commit for updating the readme with a migration guide.