From a2f7f44bbd5ae1f8bb49494f6f272b5390a017f0 Mon Sep 17 00:00:00 2001 From: JacobLinCool Date: Wed, 5 Aug 2026 01:31:30 +0800 Subject: [PATCH 1/4] Implement the Protocol v1 P0-P2 participant stack Redefine the pre-1.0 Protocol v1 wire contract and add shared closed-world conformance coverage across Kotlin, TypeScript, and Python tooling. Add the durable Android upload outbox, signed join import, randomized local EMA windows, five bounded collectors, typed catalog enforcement, and matching researcher Web authoring. --- .gitignore | 6 +- app/build.gradle.kts | 10 + .../AndroidConfigurationImportTest.kt | 40 + .../linc/androiddatacollector/CoreFlowTest.kt | 13 +- .../P2CollectorEmulatorTest.kt | 268 ++++ app/src/debug/res/raw/demo_study_envelope.txt | 2 +- app/src/main/AndroidManifest.xml | 9 + .../BootRecoveryReceiver.kt | 9 +- .../CollectorApplication.kt | 51 +- .../CollectorDashboard.kt | 12 +- .../androiddatacollector/CollectorSummary.kt | 80 +- .../linc/androiddatacollector/MainActivity.kt | 52 +- .../androiddatacollector/StudyViewModel.kt | 10 +- .../linc/androiddatacollector/UploadWorker.kt | 29 +- .../platform/AndroidStudyPlatform.kt | 260 ++- .../platform/FileUploadOutbox.kt | 323 ++++ .../platform/JoinArtifactDownloader.kt | 100 ++ .../platform/OkHttpStudyUploader.kt | 356 +++-- app/src/main/res/values-zh-rTW/strings.xml | 22 + app/src/main/res/values/strings.xml | 23 + .../CollectorSummaryTest.kt | 14 + .../DemoStudyAssetTest.kt | 29 + .../platform/FileUploadOutboxTest.kt | 199 +++ .../platform/InterventionWorkPolicyTest.kt | 169 ++ .../platform/JoinArtifactDownloaderTest.kt | 91 ++ .../platform/OkHttpStudyUploaderTest.kt | 359 +++++ .../platform/UploadIdentityTest.kt | 41 +- assurance/collector-policy.json | 157 ++ collector/accelerometer/build.gradle.kts | 2 +- .../accelerometer/AccelerometerCollector.kt | 98 +- collector/ambient-light/build.gradle.kts | 27 + .../ambientlight/AmbientLightCollector.kt | 165 ++ .../ambientlight/AmbientLightCollectorTest.kt | 83 + .../applifecycle/AppLifecycleCollector.kt | 12 +- collector/battery-state/build.gradle.kts | 27 + .../batterystate/BatteryStateCollector.kt | 235 +++ .../batterystate/BatteryStateCollectorTest.kt | 79 + collector/gyroscope/build.gradle.kts | 27 + .../collector/gyroscope/GyroscopeCollector.kt | 90 ++ .../gyroscope/GyroscopeCollectorTest.kt | 36 + collector/keyboard-ime/build.gradle.kts | 1 + .../keyboardime/ImeObservationBridge.kt | 33 +- .../keyboardime/KeyboardTouchCollector.kt | 12 +- .../keyboardime/ImeObservationBridgeTest.kt | 76 + .../collector/location/LocationCollector.kt | 59 +- .../networkstate/NetworkStateCollector.kt | 51 +- .../networkusage/NetworkUsageCollector.kt | 17 +- collector/proximity/build.gradle.kts | 27 + .../collector/proximity/ProximityCollector.kt | 182 +++ .../proximity/ProximityCollectorTest.kt | 83 + collector/sensor-common/build.gradle.kts | 25 + .../sensorcommon/AndroidSensorCollector.kt | 112 ++ .../sensorcommon/SensorSourceLifecycleTest.kt | 122 ++ collector/temporal-context/build.gradle.kts | 27 + .../TemporalContextCollector.kt | 226 +++ .../TemporalContextCollectorTest.kt | 91 ++ .../usageevents/UsageEventsCollector.kt | 19 +- .../core/access/AccessManager.kt | 12 +- core/collector-api/build.gradle.kts | 1 + .../core/collector/CollectorContracts.kt | 197 ++- .../core/collector/LatestValueRateGate.kt | 84 + .../core/collector/ProtocolEventContracts.kt | 568 +++++++ .../collector/SerializedCallbackCollector.kt | 89 +- .../core/collector/SourceLifecycle.kt | 96 ++ .../core/collector/EventFieldContractTest.kt | 32 + .../core/collector/LatestValueRateGateTest.kt | 79 + .../core/collector/ProtocolEventSizeTest.kt | 42 + .../SerializedCallbackCollectorTest.kt | 108 +- .../core/collector/SourceLifecycleTest.kt | 69 + .../core/crypto/Ed25519Crypto.kt | 25 + .../core/crypto/HpkeCrypto.kt | 123 +- .../core/crypto/Ed25519CryptoTest.kt | 38 + .../core/crypto/HpkeCryptoTest.kt | 66 +- .../core/runtime/ExperimentRuntime.kt | 134 +- .../core/runtime/ExperimentRuntimeTest.kt | 386 ++++- core/export/build.gradle.kts | 2 + .../core/export/CanonicalJsonWriter.kt | 167 ++ .../core/export/ResearchBundleVerifier.kt | 532 +++++++ .../core/export/ResearchExport.kt | 403 +++-- .../core/export/UploadReceiptCodec.kt | 78 + .../core/export/ResearchExportTest.kt | 486 +++--- .../core/model/StudyData.kt | 6 +- core/protocol/build.gradle.kts | 1 + .../core/protocol/JoinLink.kt | 138 ++ .../core/protocol/SignedConfiguration.kt | 168 +- .../protocol/ConfigurationProtocolTest.kt | 340 ++-- .../core/protocol/JoinLinkTest.kt | 72 + .../storage/EncryptedExperimentStoreTest.kt | 15 + .../core/storage/AppendTransactionRecovery.kt | 97 ++ .../core/storage/EncryptedActiveStudyStore.kt | 74 +- .../core/storage/EncryptedExperimentStore.kt | 82 +- .../core/storage/StudyDataJsonCodec.kt | 76 +- .../storage/AppendTransactionRecoveryTest.kt | 121 ++ .../storage/StudyDataReconciliationTest.kt | 54 + .../InterventionSchedulePlanner.kt | 137 +- .../core/application/StudyApplication.kt | 428 ++++- .../InterventionSchedulePlannerTest.kt | 296 +++- .../application/StudySessionManagerTest.kt | 513 +++++- .../core/definition/ProtocolBase64Url.kt | 23 + .../core/definition/ProtocolCanonicalJson.kt | 131 ++ .../core/definition/StudyConfiguration.kt | 164 +- .../definition/StudyConfigurationCodec.kt | 148 +- .../core/definition/P2ConfigurationTest.kt | 193 +++ protocol/v1/README.md | 303 ++++ protocol/v1/collector-catalog.json | 1411 +++++++++++++++++ protocol/v1/conformance-vectors.json | 430 +++++ protocol/v1/join-link-vectors.json | 48 + researcher-tools/build.gradle.kts | 4 + .../examples/INSECURE-demo-hpke-private.json | 1 - .../examples/INSECURE-demo-hpke-private.key | 1 + .../INSECURE-demo-signing-private.key | 2 +- researcher-tools/examples/README.md | 9 +- researcher-tools/examples/demo-study.json | 23 +- .../androiddatacollector/researcher/Main.kt | 88 +- .../researcher/DecryptCommandTest.kt | 95 ++ settings.gradle.kts | 6 + tools/__init__.py | 1 + tools/catalog.py | 504 ++++++ tools/catalog_parity.py | 372 +++++ tools/collector_assurance.py | 301 ++++ .../kotlin/ProtocolConformanceTest.kt | 143 ++ .../typescript/catalog-parity.spec.ts | 167 ++ .../typescript/protocol-vectors.spec.ts | 109 ++ tools/conformance/vitest.config.ts | 16 + tools/generate_protocol_vectors.mjs | 469 ++++++ tools/protocol-conformance.init.gradle | 14 + tools/tests/__init__.py | 1 + tools/tests/test_catalog.py | 80 + tools/tests/test_catalog_parity.py | 26 + tools/tests/test_collector_assurance.py | 136 ++ tools/validate_protocol_vectors.py | 238 +++ web/CONTRACT.md | 240 ++- web/e2e/researcher-flow.mjs | 102 +- web/e2e/units.mjs | 6 +- web/package.json | 9 +- web/pnpm-lock.yaml | 308 +++- web/src/lib/adc/bundle.ts | 734 +++++---- web/src/lib/adc/canonical.ts | 443 ++---- web/src/lib/adc/crypto.ts | 141 +- web/src/lib/adc/envelope.ts | 70 +- web/src/lib/adc/ids.ts | 38 +- web/src/lib/adc/join.ts | 121 ++ web/src/lib/adc/schema.ts | 213 ++- web/src/lib/adc/tink.ts | 301 ---- web/src/lib/adc/types.ts | 76 +- web/src/lib/i18n/en.ts | 68 +- web/src/lib/i18n/types.ts | 35 +- web/src/lib/i18n/zh-TW.ts | 65 +- web/src/lib/participant/SourceGrid.svelte | 2 +- web/src/lib/participant/content.ts | 12 +- web/src/lib/participant/copy.ts | 53 +- web/src/lib/ui/DownloadTile.svelte | 2 +- web/src/lib/ui/bytes.ts | 11 +- web/src/lib/ui/format.ts | 7 +- web/src/lib/ui/types.ts | 2 +- web/src/routes/researcher/+page.svelte | 7 +- .../routes/researcher/CollectorCard.svelte | 54 +- .../researcher/InterventionEditor.svelte | 147 +- .../routes/researcher/JoinLinkPanel.svelte | 136 ++ web/src/routes/researcher/RateBar.svelte | 2 +- web/src/routes/researcher/StepFiles.svelte | 15 +- web/src/routes/researcher/StepKeys.svelte | 9 +- web/src/routes/researcher/StepRead.svelte | 71 +- web/src/routes/researcher/StepStudy.svelte | 4 +- web/src/routes/researcher/artifacts.ts | 8 +- web/src/routes/researcher/draft.svelte.ts | 59 +- web/src/routes/researcher/estimate.ts | 17 +- web/src/routes/researcher/keys.ts | 60 +- web/src/routes/researcher/labels.ts | 13 +- web/src/routes/researcher/parse.ts | 171 +- web/src/routes/researcher/presets.ts | 6 +- web/src/routes/researcher/random-window.ts | 24 + web/src/routes/researcher/scales.ts | 90 +- web/src/routes/researcher/steps.ts | 2 +- web/src/routes/researcher/units.ts | 9 +- web/tests/bundle.spec.ts | 417 +++-- web/tests/canonical.spec.ts | 405 +---- web/tests/compat.spec.ts | 906 +---------- web/tests/crypto.spec.ts | 154 +- web/tests/fixture.ts | 52 + web/tests/hostile.spec.ts | 302 +--- web/tests/i18n.spec.ts | 1 + web/tests/ids.spec.ts | Bin 12661 -> 2708 bytes web/tests/join.spec.ts | 85 + web/tests/node.d.ts | 46 - web/tests/p2.spec.ts | 157 ++ web/tests/participant-copy.spec.ts | 11 + web/tests/researcher-draft.spec.ts | 6 +- web/tests/researcher.spec.ts | 210 +-- web/tests/scales.spec.ts | 43 +- web/tests/seal.ts | 333 ++-- 191 files changed, 18777 insertions(+), 5184 deletions(-) create mode 100644 app/src/androidTest/kotlin/cool/linc/androiddatacollector/AndroidConfigurationImportTest.kt create mode 100644 app/src/androidTest/kotlin/cool/linc/androiddatacollector/P2CollectorEmulatorTest.kt create mode 100644 app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt create mode 100644 app/src/main/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloader.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/CollectorSummaryTest.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/DemoStudyAssetTest.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/platform/InterventionWorkPolicyTest.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloaderTest.kt create mode 100644 app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt create mode 100644 assurance/collector-policy.json create mode 100644 collector/ambient-light/build.gradle.kts create mode 100644 collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt create mode 100644 collector/ambient-light/src/test/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollectorTest.kt create mode 100644 collector/battery-state/build.gradle.kts create mode 100644 collector/battery-state/src/main/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollector.kt create mode 100644 collector/battery-state/src/test/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollectorTest.kt create mode 100644 collector/gyroscope/build.gradle.kts create mode 100644 collector/gyroscope/src/main/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollector.kt create mode 100644 collector/gyroscope/src/test/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollectorTest.kt create mode 100644 collector/keyboard-ime/src/test/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridgeTest.kt create mode 100644 collector/proximity/build.gradle.kts create mode 100644 collector/proximity/src/main/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollector.kt create mode 100644 collector/proximity/src/test/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollectorTest.kt create mode 100644 collector/sensor-common/build.gradle.kts create mode 100644 collector/sensor-common/src/main/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/AndroidSensorCollector.kt create mode 100644 collector/sensor-common/src/test/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/SensorSourceLifecycleTest.kt create mode 100644 collector/temporal-context/build.gradle.kts create mode 100644 collector/temporal-context/src/main/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollector.kt create mode 100644 collector/temporal-context/src/test/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollectorTest.kt create mode 100644 core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGate.kt create mode 100644 core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt create mode 100644 core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycle.kt create mode 100644 core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/EventFieldContractTest.kt create mode 100644 core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGateTest.kt create mode 100644 core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventSizeTest.kt create mode 100644 core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycleTest.kt create mode 100644 core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519Crypto.kt create mode 100644 core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519CryptoTest.kt create mode 100644 core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/CanonicalJsonWriter.kt create mode 100644 core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt create mode 100644 core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/UploadReceiptCodec.kt create mode 100644 core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLink.kt create mode 100644 core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLinkTest.kt create mode 100644 core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecovery.kt create mode 100644 core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecoveryTest.kt create mode 100644 core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataReconciliationTest.kt create mode 100644 core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolBase64Url.kt create mode 100644 core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolCanonicalJson.kt create mode 100644 core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/P2ConfigurationTest.kt create mode 100644 protocol/v1/README.md create mode 100644 protocol/v1/collector-catalog.json create mode 100644 protocol/v1/conformance-vectors.json create mode 100644 protocol/v1/join-link-vectors.json delete mode 100644 researcher-tools/examples/INSECURE-demo-hpke-private.json create mode 100644 researcher-tools/examples/INSECURE-demo-hpke-private.key create mode 100644 researcher-tools/src/test/kotlin/cool/linc/androiddatacollector/researcher/DecryptCommandTest.kt create mode 100644 tools/__init__.py create mode 100644 tools/catalog.py create mode 100644 tools/catalog_parity.py create mode 100644 tools/collector_assurance.py create mode 100644 tools/conformance/kotlin/ProtocolConformanceTest.kt create mode 100644 tools/conformance/typescript/catalog-parity.spec.ts create mode 100644 tools/conformance/typescript/protocol-vectors.spec.ts create mode 100644 tools/conformance/vitest.config.ts create mode 100644 tools/generate_protocol_vectors.mjs create mode 100644 tools/protocol-conformance.init.gradle create mode 100644 tools/tests/__init__.py create mode 100644 tools/tests/test_catalog.py create mode 100644 tools/tests/test_catalog_parity.py create mode 100644 tools/tests/test_collector_assurance.py create mode 100644 tools/validate_protocol_vectors.py create mode 100644 web/src/lib/adc/join.ts delete mode 100644 web/src/lib/adc/tink.ts create mode 100644 web/src/routes/researcher/JoinLinkPanel.svelte create mode 100644 web/src/routes/researcher/random-window.ts create mode 100644 web/tests/fixture.ts create mode 100644 web/tests/join.spec.ts delete mode 100644 web/tests/node.d.ts create mode 100644 web/tests/p2.spec.ts create mode 100644 web/tests/participant-copy.spec.ts diff --git a/.gitignore b/.gitignore index 9b933de..ce4c05b 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,10 @@ hs_err_pid* replay_pid* +# Python bytecode and interpreter caches +__pycache__/ +*.py[cod] + # Kotlin Gradle plugin data, see https://kotlinlang.org/docs/whatsnew20.html#new-directory-for-kotlin-data-in-gradle-projects .kotlin/ @@ -55,4 +59,4 @@ bin/ *.keystore *.adcexp !researcher-tools/examples/INSECURE-demo-signing-private.key -!researcher-tools/examples/INSECURE-demo-hpke-private.json +!researcher-tools/examples/INSECURE-demo-hpke-private.key diff --git a/app/build.gradle.kts b/app/build.gradle.kts index 306b048..8d2623e 100644 --- a/app/build.gradle.kts +++ b/app/build.gradle.kts @@ -1,4 +1,5 @@ import java.util.Properties +import org.gradle.api.tasks.testing.Test plugins { alias(libs.plugins.android.application) @@ -74,13 +75,22 @@ kotlin { } } +tasks.withType().configureEach { + systemProperty("adc.appProjectDir", projectDir.absolutePath) +} + dependencies { implementation(project(":collector:accelerometer")) implementation(project(":collector:app-lifecycle")) + implementation(project(":collector:ambient-light")) + implementation(project(":collector:battery-state")) + implementation(project(":collector:gyroscope")) implementation(project(":collector:keyboard-ime")) implementation(project(":collector:location")) implementation(project(":collector:network-state")) implementation(project(":collector:network-usage")) + implementation(project(":collector:proximity")) + implementation(project(":collector:temporal-context")) implementation(project(":collector:usage-events")) implementation(project(":core:access")) implementation(project(":core:collector-api")) diff --git a/app/src/androidTest/kotlin/cool/linc/androiddatacollector/AndroidConfigurationImportTest.kt b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/AndroidConfigurationImportTest.kt new file mode 100644 index 0000000..157c05d --- /dev/null +++ b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/AndroidConfigurationImportTest.kt @@ -0,0 +1,40 @@ +package cool.linc.androiddatacollector + +import androidx.test.core.app.ApplicationProvider +import androidx.test.ext.junit.runners.AndroidJUnit4 +import cool.linc.androiddatacollector.core.model.ExperimentState +import kotlinx.coroutines.flow.first +import kotlinx.coroutines.runBlocking +import kotlinx.coroutines.withTimeout +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Test +import org.junit.runner.RunWith + +/** Regression for Protocol v1 raw-key signature verification on Android's provider set. */ +@RunWith(AndroidJUnit4::class) +class AndroidConfigurationImportTest { + @Test + fun debugDemoImportsIntoTheAndroidSession() = runBlocking { + val application = ApplicationProvider.getApplicationContext() + val session = application.session + withTimeout(TIMEOUT_MILLIS) { session.snapshot.first { it.initialized } } + assertNull("test requires a clean study session", session.snapshot.value.configuration) + + try { + val loadDemo = requireNotNull(DemoStudy.load) + session.importSignedConfiguration(loadDemo(application.resources)) + + assertEquals(ExperimentState.IMPORTED, session.snapshot.value.runtime.metadata?.state) + } finally { + if (session.snapshot.value.configuration != null) { + session.withdraw() + session.deleteLocalData() + } + } + } + + private companion object { + const val TIMEOUT_MILLIS = 20_000L + } +} diff --git a/app/src/androidTest/kotlin/cool/linc/androiddatacollector/CoreFlowTest.kt b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/CoreFlowTest.kt index 7ec4227..8b03bd3 100644 --- a/app/src/androidTest/kotlin/cool/linc/androiddatacollector/CoreFlowTest.kt +++ b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/CoreFlowTest.kt @@ -10,8 +10,9 @@ import androidx.compose.ui.test.performScrollTo import androidx.lifecycle.Lifecycle import androidx.test.ext.junit.runners.AndroidJUnit4 import androidx.test.platform.app.InstrumentationRegistry -import cool.linc.androiddatacollector.core.model.ExperimentState import cool.linc.androiddatacollector.core.collector.CollectorStatus +import cool.linc.androiddatacollector.core.model.ExperimentState +import kotlinx.coroutines.runBlocking import org.junit.Assert.assertEquals import org.junit.Assert.assertTrue import org.junit.Rule @@ -29,8 +30,15 @@ class CoreFlowTest { composeRule.waitUntil(TIMEOUT_MILLIS) { session.snapshot.value.initialized } composeRule.onNodeWithTag(UiTags.IMPORT_DEMO).performScrollTo().performClick() composeRule.waitUntil(TIMEOUT_MILLIS) { - session.snapshot.value.runtime.metadata?.state == ExperimentState.IMPORTED + val snapshot = session.snapshot.value + snapshot.runtime.metadata?.state == ExperimentState.IMPORTED || snapshot.incidentCode != null } + val imported = session.snapshot.value + assertEquals( + "Demo import failed: ${imported.incidentCode}", + ExperimentState.IMPORTED, + imported.runtime.metadata?.state, + ) // Setup shows a position rather than a state name, so the assertion is that the first // step's control is the one on screen. composeRule.onNodeWithTag(UiTags.REVIEW).performScrollTo().performClick() @@ -98,6 +106,7 @@ class CoreFlowTest { composeRule.onNodeWithTag(UiTags.STATE) .assertTextEquals(composeRule.activity.getString(R.string.state_completed)) composeRule.onNodeWithTag(UiTags.EXPORT).performScrollTo() + runBlocking { session.deleteLocalData() } } private companion object { diff --git a/app/src/androidTest/kotlin/cool/linc/androiddatacollector/P2CollectorEmulatorTest.kt b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/P2CollectorEmulatorTest.kt new file mode 100644 index 0000000..ef98f88 --- /dev/null +++ b/app/src/androidTest/kotlin/cool/linc/androiddatacollector/P2CollectorEmulatorTest.kt @@ -0,0 +1,268 @@ +package cool.linc.androiddatacollector + +import android.content.Context +import android.hardware.Sensor +import android.hardware.SensorManager +import androidx.test.core.app.ApplicationProvider +import androidx.test.ext.junit.runners.AndroidJUnit4 +import androidx.test.platform.app.InstrumentationRegistry +import cool.linc.androiddatacollector.collector.ambientlight.AmbientLightCollectorPlugin +import cool.linc.androiddatacollector.collector.batterystate.BatteryStateCollectorPlugin +import cool.linc.androiddatacollector.collector.gyroscope.GyroscopeCollectorPlugin +import cool.linc.androiddatacollector.collector.proximity.ProximityCollectorPlugin +import cool.linc.androiddatacollector.collector.temporalcontext.TemporalContextCollectorPlugin +import cool.linc.androiddatacollector.core.collector.AdmissionToken +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.CollectorStatus +import cool.linc.androiddatacollector.core.collector.EmitResult +import cool.linc.androiddatacollector.core.collector.EventSink +import cool.linc.androiddatacollector.core.definition.AmbientLightConfiguration +import cool.linc.androiddatacollector.core.definition.BatteryStateConfiguration +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.definition.GyroscopeConfiguration +import cool.linc.androiddatacollector.core.definition.ProximityConfiguration +import cool.linc.androiddatacollector.core.definition.TemporalContextConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.platform.AndroidResearchClocks +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.cancel +import kotlinx.coroutines.delay +import kotlinx.coroutines.runBlocking +import kotlinx.coroutines.withTimeout +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Assume.assumeTrue +import org.junit.Test +import org.junit.runner.RunWith + +/** + * Exercises the five P2 collectors against Android's real broadcast and SensorManager surfaces. + * Sensor-less devices skip this controlled-fixture test. `p2SyntheticInputs=true` instead requires + * the host to inject the documented fixed values and fails if the fixture is incomplete. + */ +@RunWith(AndroidJUnit4::class) +class P2CollectorEmulatorTest { + @Test + fun p2CollectorsCaptureTypedEventsAndHonorLifecycleBoundaries() = runBlocking { + val context = ApplicationProvider.getApplicationContext() + val syntheticInputsExpected = syntheticInputsExpected() + requireSensorFixture(context, syntheticInputsExpected) + val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default) + val sink = RecordingEventSink() + val collectorContext = CollectorContext( + scope = scope, + eventSink = sink, + clocks = AndroidResearchClocks(context, "p2-emulator-test"), + ) + val fixtures = fixtures(context) + val collectors = fixtures.map { (plugin, configuration) -> + plugin.create(configuration, collectorContext) + } + + try { + collectors.forEach { it.start() } + waitForCollectors(sink) + + collectors.forEach { collector -> + assertEquals(CollectorStatus.ACTIVE, collector.health.value.status) + } + fixtures.forEach { (plugin, _) -> + val captured = sink.latestCaptured(plugin.descriptor.id) + assertTrue( + "${plugin.descriptor.id} emitted an event outside its Protocol v1 contract", + plugin.descriptor.eventContract.accepts(captured.event, captured.sequenceNumber), + ) + } + assertPayloadSemantics(sink) + if (syntheticInputsExpected) assertSyntheticInputs(sink) + + collectors.forEach { it.pause() } + collectors.forEach { collector -> + assertEquals(CollectorStatus.PAUSED, collector.health.value.status) + } + val countAtPause = sink.size + delay(PAUSE_SETTLE_MILLIS) + assertEquals(countAtPause, sink.size) + + collectors.forEach { it.resume() } + collectors.forEach { collector -> + assertEquals(CollectorStatus.ACTIVE, collector.health.value.status) + } + withTimeout(EVENT_TIMEOUT_MILLIS) { + while (sink.count(GyroscopeConfiguration.ID) < 2) delay(POLL_MILLIS) + } + + collectors.asReversed().forEach { it.stop() } + collectors.forEach { collector -> + assertEquals(CollectorStatus.STOPPED, collector.health.value.status) + assertFalse(collector.requiresStop) + } + } finally { + collectors.asReversed().filter(Collector::requiresStop).forEach { collector -> + runCatching { collector.stop() } + } + scope.cancel() + } + } + + private fun fixtures(context: Context): List> = listOf( + BatteryStateCollectorPlugin(context) to BatteryStateConfiguration(required = true), + TemporalContextCollectorPlugin(context) to TemporalContextConfiguration(required = true), + GyroscopeCollectorPlugin(context) to GyroscopeConfiguration( + required = true, + samplingPeriodUs = 20_000, + maximumReportLatencyUs = 0, + ), + AmbientLightCollectorPlugin(context) to AmbientLightConfiguration( + required = true, + samplingPeriodUs = 200_000, + changeThresholdMillilux = 0, + ), + ProximityCollectorPlugin(context) to ProximityConfiguration( + required = true, + minimumEventIntervalMs = 100, + changeThresholdMillimeters = 0, + ), + ) + + private fun requireSensorFixture(context: Context, syntheticInputsExpected: Boolean) { + val sensorManager = context.getSystemService(SensorManager::class.java) + val missing = REQUIRED_SENSOR_TYPES.filter { sensorManager.getDefaultSensor(it) == null } + if (syntheticInputsExpected) { + assertTrue("Controlled emulator is missing sensor types $missing", missing.isEmpty()) + } else { + assumeTrue("P2 sensor integration fixture is unavailable: $missing", missing.isEmpty()) + } + } + + private suspend fun waitForCollectors(sink: RecordingEventSink) { + withTimeout(EVENT_TIMEOUT_MILLIS) { + while (!P2_COLLECTOR_IDS.all { sink.count(it) > 0 }) delay(POLL_MILLIS) + } + } + + private fun assertPayloadSemantics(sink: RecordingEventSink) { + val battery = sink.latestEventDraft(BatteryStateConfiguration.ID) + assertTrue(requireNotNull(battery.fields["percentage"]).toInt() in 0..100) + + val temporal = sink.latestEventDraft(TemporalContextConfiguration.ID) + assertEquals("STUDY_STARTED", temporal.fields["change_reason"]) + assertTrue(requireNotNull(temporal.fields["timezone_id"]).isNotBlank()) + + val gyroscope = sink.latestEventDraft(GyroscopeConfiguration.ID) + GYROSCOPE_FIELDS.forEach { field -> + assertTrue(requireNotNull(gyroscope.fields[field]).toFloat().isFinite()) + } + + val light = sink.latestEventDraft(AmbientLightConfiguration.ID) + assertTrue(requireNotNull(light.fields["illuminance_lux"]).toFloat() >= 0f) + + val proximity = sink.latestEventDraft(ProximityConfiguration.ID) + val distance = requireNotNull(proximity.fields["distance_centimeters"]).toFloat() + val maximumRange = requireNotNull(proximity.fields["maximum_range_centimeters"]).toFloat() + assertEquals((distance < maximumRange).toString(), proximity.fields["near"]) + } + + private fun assertSyntheticInputs(sink: RecordingEventSink) { + val battery = sink.latestEventDraft(BatteryStateConfiguration.ID) + assertEquals("73", battery.fields["percentage"]) + assertEquals("CHARGING", battery.fields["charging_state"]) + assertEquals("AC", battery.fields["charging_source"]) + + val gyroscope = sink.latestEventDraft(GyroscopeConfiguration.ID) + assertEquals(1.25f, requireNotNull(gyroscope.fields["x_radians_per_second"]).toFloat(), FLOAT_TOLERANCE) + assertEquals(-2.5f, requireNotNull(gyroscope.fields["y_radians_per_second"]).toFloat(), FLOAT_TOLERANCE) + assertEquals(0.5f, requireNotNull(gyroscope.fields["z_radians_per_second"]).toFloat(), FLOAT_TOLERANCE) + + val light = sink.latestEventDraft(AmbientLightConfiguration.ID) + assertEquals(123f, requireNotNull(light.fields["illuminance_lux"]).toFloat(), FLOAT_TOLERANCE) + + val proximity = sink.latestEventDraft(ProximityConfiguration.ID) + assertEquals(1f, requireNotNull(proximity.fields["distance_centimeters"]).toFloat(), FLOAT_TOLERANCE) + } + + private fun syntheticInputsExpected(): Boolean = when ( + val value = InstrumentationRegistry.getArguments().getString(SYNTHETIC_INPUTS_ARGUMENT) + ) { + null, "false" -> false + "true" -> true + else -> error("$SYNTHETIC_INPUTS_ARGUMENT must be true or false") + } + + private data class CapturedEvent( + val sequenceNumber: Long, + val event: EventDraft, + ) + + private class RecordingEventSink : EventSink { + private val token = object : AdmissionToken {} + private val events = mutableListOf() + + val size: Int + get() = synchronized(events) { events.size } + + override fun captureToken(): AdmissionToken = token + + override suspend fun emit(token: AdmissionToken, event: EventDraft): EmitResult = synchronized(events) { + check(token === this.token) { "Unexpected admission token" } + val captured = CapturedEvent(events.size.toLong() + 1, event) + events += captured + EmitResult.Accepted(captured.sequenceNumber) + } + + override suspend fun latestEvent(collectorId: String): RecordedEvent? = + synchronized(events) { + events.lastOrNull { it.event.collectorId == collectorId }?.toRecordedEvent() + } + + fun count(collectorId: String): Int = synchronized(events) { + events.count { it.event.collectorId == collectorId } + } + + fun latestCaptured(collectorId: String): CapturedEvent = synchronized(events) { + requireNotNull(events.lastOrNull { it.event.collectorId == collectorId }) + } + + fun latestEventDraft(collectorId: String): EventDraft = latestCaptured(collectorId).event + + private fun CapturedEvent.toRecordedEvent() = RecordedEvent( + sequenceNumber = sequenceNumber, + collectorId = event.collectorId, + payloadSchemaVersion = event.payloadSchemaVersion, + observedTime = event.observedTime, + payloadType = event.payloadType, + fields = event.fields, + ) + } + + private companion object { + const val EVENT_TIMEOUT_MILLIS = 10_000L + const val PAUSE_SETTLE_MILLIS = 750L + const val POLL_MILLIS = 25L + const val FLOAT_TOLERANCE = 0.001f + const val SYNTHETIC_INPUTS_ARGUMENT = "p2SyntheticInputs" + val P2_COLLECTOR_IDS = setOf( + BatteryStateConfiguration.ID, + TemporalContextConfiguration.ID, + GyroscopeConfiguration.ID, + AmbientLightConfiguration.ID, + ProximityConfiguration.ID, + ) + val GYROSCOPE_FIELDS = setOf( + "x_radians_per_second", + "y_radians_per_second", + "z_radians_per_second", + ) + val REQUIRED_SENSOR_TYPES = setOf( + Sensor.TYPE_GYROSCOPE, + Sensor.TYPE_LIGHT, + Sensor.TYPE_PROXIMITY, + ) + } +} diff --git a/app/src/debug/res/raw/demo_study_envelope.txt b/app/src/debug/res/raw/demo_study_envelope.txt index 9e4f9cb..c1549ec 100644 --- a/app/src/debug/res/raw/demo_study_envelope.txt +++ b/app/src/debug/res/raw/demo_study_envelope.txt @@ -1 +1 @@ -QURDQ0ZHMDEAEAAACyYAQGRlbW8tc2lnbmVyLTIwMjZ7InNjaGVtYV92ZXJzaW9uIjoxLCJleHBlcmltZW50X2lkIjoibW9kdWxhci1zZW5zaW5nLWRlbW8iLCJjb25maWd1cmF0aW9uX2lkIjoiZGVtby1jb25maWctMjAyNiIsImFzc2lnbmVkX3BhcnRpY2lwYW50X2lkIjpudWxsLCJpc3N1ZWRfYXQiOiIyMDI2LTAxLTAxVDAwOjAwOjAwWiIsImV4cGlyZXNfYXQiOiIyMDM1LTAxLTAxVDAwOjAwOjAwWiIsIm1pbmltdW1fYXBwX3ZlcnNpb24iOjEsInRpdGxlIjoiTW9kdWxhciBzZW5zaW5nIGRlbW9uc3RyYXRpb24iLCJyZXNlYXJjaGVyIjp7Im5hbWUiOiJBbmRyb2lkIERhdGEgQ29sbGVjdG9yIG1haW50YWluZXJzIiwiY29udGFjdCI6InJlc2VhcmNoQGV4YW1wbGUuaW52YWxpZCJ9LCJwdXJwb3NlIjoiVmVyaWZ5IHRoZSBjb21wbGV0ZSBvbi1kZXZpY2UgY29sbGVjdGlvbiwgcGF1c2UsIGV4cG9ydCwgYW5kIHJlc2VhcmNoZXItZGVjcnlwdGlvbiBsb29wLiIsImR1cmF0aW9uX2hvdXJzIjoyNCwiY29uc2VudCI6eyJkb2N1bWVudF92ZXJzaW9uIjoiZGVtby0xIiwic3VtbWFyeSI6IlRoaXMgZGVtb25zdHJhdGlvbiBjYW4gY29sbGVjdCBwcmVjaXNlIGxvY2F0aW9uLCBtb3Rpb24sIG5ldHdvcmsgc3RhdGUsIGFnZ3JlZ2F0ZSBXaS1GaS9tb2JpbGUgdXNhZ2UsIGFwcCBhbmQgc2NyZWVuIHVzYWdlIGV2ZW50cywgdGhpcyBhcHAncyBsaWZlY3ljbGUsIGFuZCB0b3VjaCBkeW5hbWljcyBtYWRlIGluc2lkZSB0aGUgb3B0aW9uYWwgcmVzZWFyY2gga2V5Ym9hcmQuIEl0IG5ldmVyIHJlY29yZHMga2V5Ym9hcmQgdGV4dC4gRGF0YSBzdGF5cyBlbmNyeXB0ZWQgb24gdGhpcyBkZXZpY2UgdW50aWwgeW91IGNob29zZSBFeHBvcnQuIFlvdSBjYW4gcGF1c2UsIHdpdGhkcmF3LCBleHBvcnQgcmVwZWF0ZWRseSwgb3IgZGVsZXRlIGxvY2FsIGRhdGEuIn0sImNvbGxlY3RvcnMiOlt7ImlkIjoiYXBwX2xpZmVjeWNsZS52MSIsInJlcXVpcmVkIjp0cnVlLCJjb25maWciOnt9fSx7ImlkIjoiYWNjZWxlcm9tZXRlci52MSIsInJlcXVpcmVkIjp0cnVlLCJjb25maWciOnsic2FtcGxpbmdfcGVyaW9kX3VzIjoxMDAwMDAsIm1heGltdW1fcmVwb3J0X2xhdGVuY3lfdXMiOjEwMDAwMDB9fSx7ImlkIjoibmV0d29ya19zdGF0ZS52MSIsInJlcXVpcmVkIjp0cnVlLCJjb25maWciOnsiaW5jbHVkZV9iYW5kd2lkdGhfZXN0aW1hdGVzIjp0cnVlfX0seyJpZCI6Im5ldHdvcmtfdXNhZ2UudjEiLCJyZXF1aXJlZCI6ZmFsc2UsImNvbmZpZyI6eyJ0cmFuc3BvcnRzIjpbIm1vYmlsZSIsIndpZmkiXSwicG9sbF9pbnRlcnZhbF9taW51dGVzIjo1fX0seyJpZCI6InVzYWdlX2V2ZW50cy52MSIsInJlcXVpcmVkIjpmYWxzZSwiY29uZmlnIjp7InBvbGxfaW50ZXJ2YWxfbWludXRlcyI6MTV9fSx7ImlkIjoibG9jYXRpb24udjEiLCJyZXF1aXJlZCI6ZmFsc2UsImNvbmZpZyI6eyJpbnRlcnZhbF9taWxsaXMiOjEwMDAwLCJtaW5pbXVtX2ludGVydmFsX21pbGxpcyI6NTAwMCwibWF4aW11bV9iYXRjaF9kZWxheV9taWxsaXMiOjMwMDAwLCJtaW5pbXVtX2Rpc3BsYWNlbWVudF9tZXRlcnMiOjUuMCwicHJpb3JpdHkiOiJCQUxBTkNFRCJ9fSx7ImlkIjoia2V5Ym9hcmRfdG91Y2gudjEiLCJyZXF1aXJlZCI6ZmFsc2UsImNvbmZpZyI6eyJ0cmFqZWN0b3J5X3NhbXBsaW5nX2h6Ijo2MH19XSwic3VydmV5cyI6W3siaWQiOiJkZW1vLXN1cnZleSIsInRpdGxlIjp7ImRlZmF1bHQiOiJTdHVkeSBjaGVjay1pbiIsInRyYW5zbGF0aW9ucyI6eyJ6aC1UVyI6IueglOeptueiuuiqjSJ9fSwiZGVzY3JpcHRpb24iOnsiZGVmYXVsdCI6IlRlbGwgdXMgaG93IHRoZSBzdHVkeSBpcyBnb2luZy4iLCJ0cmFuc2xhdGlvbnMiOnt9fSwicXVlc3Rpb25zIjpbeyJ0eXBlIjoic2hvcnRfdGV4dCIsImlkIjoic3RhdHVzLW5vdGUiLCJwcm9tcHQiOnsiZGVmYXVsdCI6IkhvdyBpcyBpdCBnb2luZz8iLCJ0cmFuc2xhdGlvbnMiOnt9fSwicmVxdWlyZWQiOmZhbHNlLCJtYXhpbXVtX2xlbmd0aCI6NTAwfV19XSwiaW50ZXJ2ZW50aW9ucyI6W3siaWQiOiJkZW1vLWNoZWNrLWluIiwiYWN0aW9uIjp7InR5cGUiOiJzdXJ2ZXkiLCJub3RpZmljYXRpb25fdGl0bGUiOiJTdHVkeSBjaGVjay1pbiIsIm5vdGlmaWNhdGlvbl9tZXNzYWdlIjoiUGxlYXNlIGNvbXBsZXRlIHRoZSBzdHVkeSBjaGVjay1pbi4iLCJzdXJ2ZXlfaWQiOiJkZW1vLXN1cnZleSJ9LCJ0cmlnZ2VycyI6W3siaWQiOiJhZnRlci1vbmUtaG91ciIsInNjaGVkdWxlIjp7InR5cGUiOiJvbmVfdGltZSIsIm9mZnNldF9taW51dGVzIjo2MCwiY2xvY2siOiJBQ1RJVkVfUlVOTklOR19USU1FIn0sImF2YWlsYWJpbGl0eV9taW51dGVzIjoxNDQwfV19XSwic3RvcmFnZSI6eyJtYXhpbXVtX2xvY2FsX2J5dGVzIjoxNjc3NzIxNn0sInNpZ25lciI6eyJrZXlfaWQiOiJkZW1vLXNpZ25lci0yMDI2IiwicHVibGljX2tleSI6Ik1Db3dCUVlESzJWd0F5RUFzUlNhVHBabVRTQkw3ZU42blMvSEJzTm1MTThuMWhkUm1JdDF2dExac0MwPSJ9LCJleHBvcnQiOnsicmVzZWFyY2hlcl9rZXlfaWQiOiJkZW1vLWhwa2UtMjAyNiIsInRpbmtfaHBrZV9wdWJsaWNfa2V5c2V0Ijp7InByaW1hcnlLZXlJZCI6MjE4OTkyNzI3LCJrZXkiOlt7ImtleURhdGEiOnsidHlwZVVybCI6InR5cGUuZ29vZ2xlYXBpcy5jb20vZ29vZ2xlLmNyeXB0by50aW5rLkhwa2VQdWJsaWNLZXkiLCJ2YWx1ZSI6IkVnWUlBUkFCR0FJYUlCcHlRM3c0ZkZ4OVhnRVV4NWt5elphSVBYTHE3YVlVNlJKK3k5K3JHTkVBIiwia2V5TWF0ZXJpYWxUeXBlIjoiQVNZTU1FVFJJQ19QVUJMSUMifSwic3RhdHVzIjoiRU5BQkxFRCIsImtleUlkIjoyMTg5OTI3MjcsIm91dHB1dFByZWZpeFR5cGUiOiJUSU5LIn1dfX0sInVwbG9hZCI6e319eQt/MbwbgKiep/lySOSuib/oEqqRxyR9LuR7lffn30Tz7G7jz1ue7a3mV+1zqhtEbdYAoLnhleIZRa0GmcvcCA== +QURDQ0ZHMDEAEAAACkFkZW1vLXNpZ25lci0yMDI2eyJhc3NpZ25lZF9wYXJ0aWNpcGFudF9pZCI6bnVsbCwiY29sbGVjdG9ycyI6W3siY29uZmlnIjp7fSwiaWQiOiJhcHBfbGlmZWN5Y2xlLnYxIiwicmVxdWlyZWQiOnRydWV9LHsiY29uZmlnIjp7Im1heGltdW1fcmVwb3J0X2xhdGVuY3lfdXMiOjEwMDAwMDAsInNhbXBsaW5nX3BlcmlvZF91cyI6MTAwMDAwfSwiaWQiOiJhY2NlbGVyb21ldGVyLnYxIiwicmVxdWlyZWQiOnRydWV9LHsiY29uZmlnIjp7ImluY2x1ZGVfYmFuZHdpZHRoX2VzdGltYXRlcyI6dHJ1ZX0sImlkIjoibmV0d29ya19zdGF0ZS52MSIsInJlcXVpcmVkIjp0cnVlfSx7ImNvbmZpZyI6eyJwb2xsX2ludGVydmFsX21pbnV0ZXMiOjUsInRyYW5zcG9ydHMiOlsibW9iaWxlIiwid2lmaSJdfSwiaWQiOiJuZXR3b3JrX3VzYWdlLnYxIiwicmVxdWlyZWQiOmZhbHNlfSx7ImNvbmZpZyI6eyJwb2xsX2ludGVydmFsX21pbnV0ZXMiOjE1fSwiaWQiOiJ1c2FnZV9ldmVudHMudjEiLCJyZXF1aXJlZCI6ZmFsc2V9LHsiY29uZmlnIjp7ImludGVydmFsX21pbGxpcyI6MTAwMDAsIm1heGltdW1fYmF0Y2hfZGVsYXlfbWlsbGlzIjozMDAwMCwibWluaW11bV9kaXNwbGFjZW1lbnRfbWlsbGltZXRlcnMiOjUwMDAsIm1pbmltdW1faW50ZXJ2YWxfbWlsbGlzIjo1MDAwLCJwcmlvcml0eSI6IkJBTEFOQ0VEIn0sImlkIjoibG9jYXRpb24udjEiLCJyZXF1aXJlZCI6ZmFsc2V9LHsiY29uZmlnIjp7InRyYWplY3Rvcnlfc2FtcGxpbmdfaHoiOjYwfSwiaWQiOiJrZXlib2FyZF90b3VjaC52MSIsInJlcXVpcmVkIjpmYWxzZX1dLCJjb25maWd1cmF0aW9uX2lkIjoiZGVtby1jb25maWctMjAyNiIsImNvbnNlbnQiOnsiZG9jdW1lbnRfdmVyc2lvbiI6ImRlbW8tMSIsInN1bW1hcnkiOiJUaGlzIGRlbW9uc3RyYXRpb24gY2FuIGNvbGxlY3QgcHJlY2lzZSBsb2NhdGlvbiwgbW90aW9uLCBuZXR3b3JrIHN0YXRlLCBhZ2dyZWdhdGUgV2ktRmkvbW9iaWxlIHVzYWdlLCBhcHAgYW5kIHNjcmVlbiB1c2FnZSBldmVudHMsIHRoaXMgYXBwJ3MgbGlmZWN5Y2xlLCBhbmQgdG91Y2ggZHluYW1pY3MgbWFkZSBpbnNpZGUgdGhlIG9wdGlvbmFsIHJlc2VhcmNoIGtleWJvYXJkLiBJdCBuZXZlciByZWNvcmRzIGtleWJvYXJkIHRleHQuIERhdGEgc3RheXMgZW5jcnlwdGVkIG9uIHRoaXMgZGV2aWNlIHVudGlsIHlvdSBjaG9vc2UgRXhwb3J0LiBZb3UgY2FuIHBhdXNlLCB3aXRoZHJhdywgZXhwb3J0IHJlcGVhdGVkbHksIG9yIGRlbGV0ZSBsb2NhbCBkYXRhLiJ9LCJkdXJhdGlvbl9ob3VycyI6MjQsImV4cGVyaW1lbnRfaWQiOiJtb2R1bGFyLXNlbnNpbmctZGVtbyIsImV4cGlyZXNfYXQiOiIyMDM1LTAxLTAxVDAwOjAwOjAwWiIsImV4cG9ydCI6eyJocGtlX3B1YmxpY19rZXkiOiJHbkpEZkRoOFhIMWVBUlRIbVRMTmxvZzljdXJ0cGhUcEVuN0wzNnNZMFFBIiwicmVzZWFyY2hlcl9rZXlfaWQiOiJkZW1vLWhwa2UtMjAyNiJ9LCJpbnRlcnZlbnRpb25zIjpbeyJhY3Rpb24iOnsibm90aWZpY2F0aW9uX21lc3NhZ2UiOiJQbGVhc2UgY29tcGxldGUgdGhlIHN0dWR5IGNoZWNrLWluLiIsIm5vdGlmaWNhdGlvbl90aXRsZSI6IlN0dWR5IGNoZWNrLWluIiwic3VydmV5X2lkIjoiZGVtby1zdXJ2ZXkiLCJ0eXBlIjoic3VydmV5In0sImlkIjoiZGVtby1jaGVjay1pbiIsInRyaWdnZXJzIjpbeyJhdmFpbGFiaWxpdHlfbWludXRlcyI6MTQ0MCwiaWQiOiJhZnRlci1vbmUtaG91ciIsInNjaGVkdWxlIjp7ImNsb2NrIjoiQUNUSVZFX1JVTk5JTkdfVElNRSIsIm9mZnNldF9taW51dGVzIjo2MCwidHlwZSI6Im9uZV90aW1lIn19XX1dLCJpc3N1ZWRfYXQiOiIyMDI2LTAxLTAxVDAwOjAwOjAwWiIsIm1pbmltdW1fY2xpZW50X3ZlcnNpb24iOiIxIiwicGxhdGZvcm0iOiJhbmRyb2lkIiwicHVycG9zZSI6IlZlcmlmeSB0aGUgY29tcGxldGUgb24tZGV2aWNlIGNvbGxlY3Rpb24sIHBhdXNlLCBleHBvcnQsIGFuZCByZXNlYXJjaGVyLWRlY3J5cHRpb24gbG9vcC4iLCJyZXNlYXJjaGVyIjp7ImNvbnRhY3QiOiJyZXNlYXJjaEBleGFtcGxlLmludmFsaWQiLCJuYW1lIjoiQW5kcm9pZCBEYXRhIENvbGxlY3RvciBtYWludGFpbmVycyJ9LCJzY2hlbWFfdmVyc2lvbiI6MSwic2lnbmVyIjp7ImtleV9pZCI6ImRlbW8tc2lnbmVyLTIwMjYiLCJwdWJsaWNfa2V5Ijoic1JTYVRwWm1UU0JMN2VONm5TX0hCc05tTE04bjFoZFJtSXQxdnRMWnNDMCJ9LCJzdG9yYWdlIjp7Im1heGltdW1fbG9jYWxfYnl0ZXMiOjE2Nzc3MjE2fSwic3VydmV5cyI6W3siZGVzY3JpcHRpb24iOnsiZGVmYXVsdCI6IlRlbGwgdXMgaG93IHRoZSBzdHVkeSBpcyBnb2luZy4iLCJ0cmFuc2xhdGlvbnMiOnt9fSwiaWQiOiJkZW1vLXN1cnZleSIsInF1ZXN0aW9ucyI6W3siaWQiOiJzdGF0dXMtbm90ZSIsIm1heGltdW1fbGVuZ3RoIjo1MDAsInByb21wdCI6eyJkZWZhdWx0IjoiSG93IGlzIGl0IGdvaW5nPyIsInRyYW5zbGF0aW9ucyI6e319LCJyZXF1aXJlZCI6ZmFsc2UsInR5cGUiOiJzaG9ydF90ZXh0In1dLCJ0aXRsZSI6eyJkZWZhdWx0IjoiU3R1ZHkgY2hlY2staW4iLCJ0cmFuc2xhdGlvbnMiOnsiemgtVFciOiLnoJTnqbbnorroqo0ifX19XSwidGl0bGUiOiJNb2R1bGFyIHNlbnNpbmcgZGVtb25zdHJhdGlvbiIsInVwbG9hZCI6e319UmqpwSmoyhlVTYn/JzgrNbEBiCTEQhDUD6j/6iTkZLDj21a0YnS6JAFQaKSyvS3N/xZDfJnjsNdas/z0oLZ3DQ== diff --git a/app/src/main/AndroidManifest.xml b/app/src/main/AndroidManifest.xml index a7f151b..195cce8 100644 --- a/app/src/main/AndroidManifest.xml +++ b/app/src/main/AndroidManifest.xml @@ -36,6 +36,15 @@ + + + + + + configurationVerifier().verify(bytes).also { ResearchExport.validate(it.configuration) } }, - storeFactory = StudyStoreFactory { configuration -> - EncryptedExperimentStore(this, configuration.experimentId, configuration.maximumLocalBytes) + storeFactory = StudyStoreFactory { experimentId, maximumLocalBytes -> + EncryptedExperimentStore(this, experimentId, maximumLocalBytes) }, runtimeFactory = ExperimentRuntimeFactory { configuration, store, availableAccess -> ExperimentRuntime( @@ -78,22 +102,27 @@ class CollectorApplication : Application() { accessGateway = accessManager, collectionHost = AndroidStudyCollectionHost(this), workScheduler = workScheduler, - exporter = StudyExporter { configuration, metadata, events, destination -> + exporter = StudyExporter { verified, metadata, events, destination -> ResearchExport.encrypt( // Starts at the retained floor, not at 1: anything below it was delivered to // the study's endpoint and reclaimed, and the bundle says so in // first_sequence_number rather than appearing to be a complete history. ExportSnapshot( - configuration, - metadata, - Instant.now().toEpochMilli(), + verifiedConfiguration = verified, + metadata = metadata, + producer = producer, + bundleKind = BundleKind.MANUAL_EXPORT, + exportedAtUtcMillis = Instant.now().toEpochMilli(), fromSequence = metadata.retainedFromSequence, ), events, destination, ) }, - uploader = OkHttpStudyUploader(), + uploader = OkHttpStudyUploader( + outbox = FileUploadOutbox(noBackupFilesDir.resolve("upload-outbox")), + producer = producer, + ), accessPolicy = StudyAccessPolicy(), scope = applicationScope, ) @@ -102,18 +131,20 @@ class CollectorApplication : Application() { // The delivery chain is one-time work, so it has no platform-side repetition to fall // back on. Re-establishing it here covers a link lost to a crash or a force stop. session.snapshot.value.configuration?.let(workScheduler::reschedulePendingWork) - session.rescheduleInterventions() + InterventionDeliveryCoordinator.recoverStalePosting { + session.rescheduleInterventions(recoverStalePosting = true) + } } } private fun configurationVerifier(): ConfigurationVerifier = ConfigurationVerifier( trustedSigningKeys = TRUSTED_SIGNING_KEYS, - appVersionCode = packageManager.getPackageInfo(packageName, 0).longVersionCode.toInt(), + clientVersion = packageManager.getPackageInfo(packageName, 0).longVersionCode, ) private companion object { /** - * Signers this build pins, as key ID to Base64 X.509 Ed25519 public key. + * Signers this build pins, as key ID to unpadded-base64url raw Ed25519 public key. * * Empty on purpose: a study configuration carries its own signing key, so this build runs * any correctly signed study and tells the participant the publisher is unverified. An diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorDashboard.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorDashboard.kt index 26a81b0..f877294 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorDashboard.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorDashboard.kt @@ -565,7 +565,7 @@ private fun AccessPanel(checks: List, actions: StudyUiActions, bus val requiredReady = checks.none { it.requirement.required && !it.granted } Column(verticalArrangement = Arrangement.spacedBy(4.dp)) { checks.forEach { check -> - val actionable = !check.granted && check.requirement.kind != AccessKind.ACCELEROMETER_HARDWARE + val actionable = !check.granted && check.requirement.kind !in HARDWARE_ACCESS Row( modifier = Modifier .fillMaxWidth() @@ -997,8 +997,18 @@ private fun AccessKind.labelRes(): Int = when (this) { AccessKind.RESEARCH_KEYBOARD_ENABLED -> R.string.access_research_keyboard_enabled AccessKind.RESEARCH_KEYBOARD_SELECTED -> R.string.access_research_keyboard_selected AccessKind.ACCELEROMETER_HARDWARE -> R.string.access_accelerometer_hardware + AccessKind.GYROSCOPE_HARDWARE -> R.string.access_gyroscope_hardware + AccessKind.AMBIENT_LIGHT_HARDWARE -> R.string.access_ambient_light_hardware + AccessKind.PROXIMITY_HARDWARE -> R.string.access_proximity_hardware } +private val HARDWARE_ACCESS = setOf( + AccessKind.ACCELEROMETER_HARDWARE, + AccessKind.GYROSCOPE_HARDWARE, + AccessKind.AMBIENT_LIGHT_HARDWARE, + AccessKind.PROXIMITY_HARDWARE, +) + private val TERMINAL_STATES = setOf(ExperimentState.COMPLETED, ExperimentState.WITHDRAWN) private const val TICK_MILLIS = 30_000L diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorSummary.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorSummary.kt index 269edbf..7d6a82a 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorSummary.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/CollectorSummary.kt @@ -4,12 +4,17 @@ import androidx.compose.runtime.Composable import androidx.compose.ui.res.pluralStringResource import androidx.compose.ui.res.stringResource import cool.linc.androiddatacollector.core.definition.AccelerometerConfiguration +import cool.linc.androiddatacollector.core.definition.AmbientLightConfiguration import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration +import cool.linc.androiddatacollector.core.definition.BatteryStateConfiguration import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.KeyboardTouchConfiguration +import cool.linc.androiddatacollector.core.definition.GyroscopeConfiguration import cool.linc.androiddatacollector.core.definition.LocationConfiguration import cool.linc.androiddatacollector.core.definition.NetworkStateConfiguration import cool.linc.androiddatacollector.core.definition.NetworkUsageConfiguration +import cool.linc.androiddatacollector.core.definition.ProximityConfiguration +import cool.linc.androiddatacollector.core.definition.TemporalContextConfiguration import cool.linc.androiddatacollector.core.definition.UsageEventsConfiguration import cool.linc.androiddatacollector.core.definition.UploadConfiguration @@ -50,6 +55,51 @@ fun CollectorConfiguration.summarize(): CollectorSummary = when (this) { optional = !required, ) + is BatteryStateConfiguration -> CollectorSummary( + glyph = Glyph.DATA_VOLUME, + name = stringResource(R.string.collector_battery_state_name), + detail = stringResource(R.string.collector_battery_state_detail), + optional = !required, + ) + + is TemporalContextConfiguration -> CollectorSummary( + glyph = Glyph.CLOCK, + name = stringResource(R.string.collector_temporal_context_name), + detail = stringResource(R.string.collector_temporal_context_detail), + optional = !required, + ) + + is GyroscopeConfiguration -> CollectorSummary( + glyph = Glyph.MOTION, + name = stringResource(R.string.collector_gyroscope_name), + detail = (1_000_000.0 / samplingPeriodUs).toInt().coerceAtLeast(1).let { hz -> + pluralStringResource(R.plurals.collector_gyroscope_detail, hz, hz) + }, + optional = !required, + ) + + is AmbientLightConfiguration -> CollectorSummary( + glyph = Glyph.APP, + name = stringResource(R.string.collector_ambient_light_name), + detail = stringResource( + R.string.collector_ambient_light_detail, + microsLabel(samplingPeriodUs.toLong()), + changeThresholdMillilux, + ), + optional = !required, + ) + + is ProximityConfiguration -> CollectorSummary( + glyph = Glyph.CONNECTION, + name = stringResource(R.string.collector_proximity_name), + detail = stringResource( + R.string.collector_proximity_detail, + millisLabel(minimumEventIntervalMs.toLong()), + changeThresholdMillimeters, + ), + optional = !required, + ) + is NetworkStateConfiguration -> CollectorSummary( glyph = Glyph.CONNECTION, name = stringResource(R.string.collector_network_state_name), @@ -77,7 +127,7 @@ fun CollectorConfiguration.summarize(): CollectorSummary = when (this) { detail = stringResource( R.string.collector_location_detail, millisLabel(intervalMillis), - stringResource(R.string.unit_metres, minimumDisplacementMeters.toInt()), + stringResource(R.string.unit_metres, minimumDisplacementMillimeters / 1_000), ), optional = !required, ) @@ -99,9 +149,31 @@ fun minutesLabel(minutes: Int): String = when { } @Composable -fun millisLabel(millis: Long): String = when { - millis % 60_000L == 0L -> minutesLabel((millis / 60_000L).toInt()) - else -> stringResource(R.string.unit_seconds, (millis / 1_000L).toInt()) +fun millisLabel(millis: Long): String = microsLabel(Math.multiplyExact(millis, 1_000L)) + +@Composable +fun microsLabel(micros: Long): String = when (val duration = exactDuration(micros)) { + is ExactDuration.Microseconds -> stringResource(R.string.unit_microseconds, duration.value) + is ExactDuration.Milliseconds -> stringResource(R.string.unit_milliseconds, duration.value) + is ExactDuration.Seconds -> stringResource(R.string.unit_seconds, duration.value) + is ExactDuration.Minutes -> minutesLabel(duration.value.toInt()) +} + +internal sealed interface ExactDuration { + val value: Long + + data class Microseconds(override val value: Long) : ExactDuration + data class Milliseconds(override val value: Long) : ExactDuration + data class Seconds(override val value: Long) : ExactDuration + data class Minutes(override val value: Long) : ExactDuration +} + +/** Chooses the coarsest integral unit without discarding any signed microseconds. */ +internal fun exactDuration(microseconds: Long): ExactDuration = when { + microseconds % 60_000_000L == 0L -> ExactDuration.Minutes(microseconds / 60_000_000L) + microseconds % 1_000_000L == 0L -> ExactDuration.Seconds(microseconds / 1_000_000L) + microseconds % 1_000L == 0L -> ExactDuration.Milliseconds(microseconds / 1_000L) + else -> ExactDuration.Microseconds(microseconds) } @Composable diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/MainActivity.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/MainActivity.kt index 67d597c..4f372ba 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/MainActivity.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/MainActivity.kt @@ -12,6 +12,8 @@ import androidx.lifecycle.compose.collectAsStateWithLifecycle import androidx.lifecycle.lifecycleScope import cool.linc.androiddatacollector.platform.InterventionWorker import cool.linc.androiddatacollector.core.collector.AccessKind +import cool.linc.androiddatacollector.core.protocol.JoinLink +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec import java.time.Instant import kotlinx.coroutines.flow.first import kotlinx.coroutines.launch @@ -27,7 +29,7 @@ class MainActivity : ComponentActivity() { if (uri == null) return@registerForActivityResult viewModel.importSignedConfiguration { requireNotNull(contentResolver.openInputStream(uri)) { "Cannot open signed configuration" } - .use { it.readNBytes(MAXIMUM_CONFIGURATION_ENVELOPE_BYTES + 1) } + .use { it.readNBytes(SignedConfigurationCodec.MAXIMUM_ENVELOPE_BYTES + 1) } } } @@ -46,7 +48,7 @@ class MainActivity : ComponentActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) - openOccurrence(intent) + handleIntent(intent) enableEdgeToEdge() setContent { val state = viewModel.state.collectAsStateWithLifecycle().value @@ -82,15 +84,39 @@ class MainActivity : ComponentActivity() { override fun onNewIntent(intent: Intent) { super.onNewIntent(intent) setIntent(intent) - openOccurrence(intent) + handleIntent(intent) } - private fun openOccurrence(intent: Intent) { - if (intent.action != InterventionWorker.ACTION_OPEN_OCCURRENCE) return - val occurrenceId = intent.getStringExtra(InterventionWorker.KEY_OCCURRENCE_ID) ?: return - lifecycleScope.launch { - val ready = collectorApplication.session.snapshot.first { it.initialized } - if (ready.configuration != null) collectorApplication.session.openOccurrence(occurrenceId) + private fun handleIntent(intent: Intent) { + when (intent.action) { + InterventionWorker.ACTION_OPEN_OCCURRENCE -> { + val occurrenceId = intent.getStringExtra(InterventionWorker.KEY_OCCURRENCE_ID) ?: return + lifecycleScope.launch { + val ready = collectorApplication.session.snapshot.first { it.initialized } + if (ready.configuration != null) collectorApplication.session.openOccurrence(occurrenceId) + } + } + Intent.ACTION_VIEW -> { + val encoded = intent.dataString ?: return + // Prevent an Activity recreation from starting a second download for the same URI. + intent.data = null + val link = try { + JoinLink.parse(encoded) + } catch (_: IllegalArgumentException) { + viewModel.reportMessage("JOIN_LINK_INVALID") + return + } + lifecycleScope.launch { + val ready = collectorApplication.session.snapshot.first { it.initialized } + if (ready.configuration != null || ready.deletionPending) { + viewModel.reportMessage("JOIN_ACTIVE_STUDY") + } else { + viewModel.importJoin(link) { + collectorApplication.joinArtifactDownloader.download(link) + } + } + } + } } } @@ -117,11 +143,11 @@ class MainActivity : ComponentActivity() { AccessKind.USAGE_ACCESS, AccessKind.RESEARCH_KEYBOARD_ENABLED -> collectorApplication.accessManager.settingsIntent(kind)?.let(::startActivity) ?: viewModel.reportMessage("ACCESS_SETTINGS_UNAVAILABLE") - AccessKind.ACCELEROMETER_HARDWARE -> Unit + AccessKind.ACCELEROMETER_HARDWARE, + AccessKind.GYROSCOPE_HARDWARE, + AccessKind.AMBIENT_LIGHT_HARDWARE, + AccessKind.PROXIMITY_HARDWARE -> Unit } } - private companion object { - const val MAXIMUM_CONFIGURATION_ENVELOPE_BYTES = 1_100_000 - } } diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/StudyViewModel.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/StudyViewModel.kt index 0ebf22c..61b66c8 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/StudyViewModel.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/StudyViewModel.kt @@ -10,6 +10,8 @@ import cool.linc.androiddatacollector.core.collector.CollectorHealth import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.export.ExportReceipt import cool.linc.androiddatacollector.core.model.StudyMetadata +import cool.linc.androiddatacollector.core.protocol.JoinLink +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec import cool.linc.androiddatacollector.core.runtime.CommandResult import java.io.OutputStream import kotlinx.coroutines.CancellationException @@ -81,10 +83,14 @@ class StudyViewModel( fun importSignedConfiguration(load: () -> ByteArray) = operation(INCIDENT_IMPORT_FAILED) { val bytes = withContext(Dispatchers.IO) { load() } - require(bytes.size <= MAXIMUM_CONFIGURATION_ENVELOPE_BYTES) { "Configuration is too large" } + require(bytes.size <= SignedConfigurationCodec.MAXIMUM_ENVELOPE_BYTES) { "Configuration is too large" } session.importSignedConfiguration(bytes) } + fun importJoin(link: JoinLink, load: suspend () -> ByteArray) = operation(INCIDENT_JOIN_FAILED) { + session.importSignedConfiguration(load(), link) + } + fun reviewStudy() = command(session::reviewStudy) fun acceptConsent() = command(session::acceptConsent) fun completeAccessSetup() = command(session::completeAccessSetup) @@ -145,8 +151,8 @@ class StudyViewModel( } private companion object { - const val MAXIMUM_CONFIGURATION_ENVELOPE_BYTES = 1_100_000 const val INCIDENT_IMPORT_FAILED = "CONFIGURATION_IMPORT_FAILED" + const val INCIDENT_JOIN_FAILED = "JOIN_IMPORT_FAILED" const val INCIDENT_EXPORT_FAILED = "EXPORT_FAILED" const val INCIDENT_DELETE_FAILED = "LOCAL_DATA_DELETE_FAILED" const val INCIDENT_COMMAND_FAILED = "COMMAND_FAILED" diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/UploadWorker.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/UploadWorker.kt index ffc1497..6eb54bd 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/UploadWorker.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/UploadWorker.kt @@ -4,7 +4,7 @@ import android.content.Context import androidx.work.CoroutineWorker import androidx.work.ExistingWorkPolicy import androidx.work.WorkerParameters -import cool.linc.androiddatacollector.core.runtime.CommandResult +import cool.linc.androiddatacollector.core.application.UploadAttemptResult import cool.linc.androiddatacollector.platform.AndroidStudyWorkScheduler import kotlinx.coroutines.flow.first @@ -21,18 +21,27 @@ class UploadWorker( ) : CoroutineWorker(context, parameters) { override suspend fun doWork(): Result { val expectedExperimentId = inputData.getString(KEY_EXPERIMENT_ID) ?: return Result.failure() + val expectedConfigurationId = inputData.getString(KEY_CONFIGURATION_ID) ?: return Result.failure() val session = (applicationContext as CollectorApplication).session val snapshot = session.snapshot.first { it.initialized } - // A different study was imported since this work was scheduled; the old job is obsolete. - if (snapshot.configuration?.experimentId != expectedExperimentId) return Result.success() - val upload = snapshot.configuration?.upload ?: return Result.success() + if (snapshot.deletionPending) return Result.success() + val configuration = snapshot.configuration ?: return Result.success() + // Configurations may share an experiment ID; both identities must match the scheduled job. + if (configuration.experimentId != expectedExperimentId || + configuration.configurationId != expectedConfigurationId + ) return Result.success() + val upload = configuration.upload ?: return Result.success() // uploadPending is a no-op before collection starts, so no state check is needed here. - if (session.uploadPending() != CommandResult.Success) { - // Retry rather than fail: the usual cause is a network or endpoint problem that - // resolves on its own, and the events are still safe on the device meanwhile. Retrying - // keeps this link of the chain alive, so no successor is enqueued here. - return Result.retry() + when (val result = session.uploadPending()) { + is UploadAttemptResult.Failed -> { + // Only failures explicitly classified by the transport are retried. A malformed + // receipt, redirect, or other terminal 4xx must not become an endless request loop. + return if (result.retryable) Result.retry() else Result.failure() + } + UploadAttemptResult.NoWork, + is UploadAttemptResult.Confirmed, + -> Unit } // A study that has ended keeps delivering so its backlog still reaches the researcher. @@ -41,6 +50,7 @@ class UploadWorker( if (!session.uploadDrained()) { AndroidStudyWorkScheduler(applicationContext).scheduleUpload( expectedExperimentId, + expectedConfigurationId, upload, ExistingWorkPolicy.REPLACE, ) @@ -50,5 +60,6 @@ class UploadWorker( companion object { const val KEY_EXPERIMENT_ID = "experiment_id" + const val KEY_CONFIGURATION_ID = "configuration_id" } } diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/AndroidStudyPlatform.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/AndroidStudyPlatform.kt index f8a3bf7..a5d1662 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/AndroidStudyPlatform.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/AndroidStudyPlatform.kt @@ -29,8 +29,13 @@ import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.definition.SurveyAction import cool.linc.androiddatacollector.core.definition.UploadConfiguration import cool.linc.androiddatacollector.core.model.InterventionOccurrence +import cool.linc.androiddatacollector.core.runtime.OccurrenceClaimResult +import cool.linc.androiddatacollector.core.runtime.OccurrenceExpiryResult import java.util.concurrent.TimeUnit +import kotlinx.coroutines.CancellationException import kotlinx.coroutines.flow.first +import kotlinx.coroutines.sync.Mutex +import kotlinx.coroutines.sync.withLock class AndroidStudyCollectionHost( private val context: Context, @@ -48,6 +53,7 @@ class AndroidStudyWorkScheduler( context: Context, ) : StudyWorkScheduler { private val workManager = WorkManager.getInstance(context.applicationContext) + private val notificationManager = context.getSystemService(NotificationManager::class.java) override fun schedule(configuration: StudyConfiguration) { val deadline = OneTimeWorkRequestBuilder() @@ -63,15 +69,25 @@ class AndroidStudyWorkScheduler( ExistingWorkPolicy.REPLACE, deadline, ) - configuration.upload?.let { scheduleUpload(configuration.experimentId, it, ExistingWorkPolicy.REPLACE) } + configuration.upload?.let { + scheduleUpload( + configuration.experimentId, + configuration.configurationId, + it, + ExistingWorkPolicy.REPLACE, + ) + } } override fun replaceInterventionWork( configuration: StudyConfiguration, - occurrences: List, + deliveries: List, + expiries: List, ) { - workManager.cancelAllWorkByTag(interventionTag(configuration.experimentId)) - occurrences.forEach { enqueueOccurrence(configuration, it, ExistingWorkPolicy.REPLACE) } + workManager.cancelAllWorkByTag(InterventionWorkIdentity.deliveryTag(configuration.experimentId)) + workManager.cancelAllWorkByTag(InterventionWorkIdentity.expiryTag(configuration.experimentId)) + deliveries.forEach { enqueueDelivery(configuration, it, ExistingWorkPolicy.REPLACE) } + expiries.forEach { enqueueExpiry(configuration, it, ExistingWorkPolicy.REPLACE) } } override fun enqueueOccurrence(configuration: StudyConfiguration, occurrence: InterventionOccurrence) { @@ -82,26 +98,43 @@ class AndroidStudyWorkScheduler( configuration: StudyConfiguration, occurrence: InterventionOccurrence, policy: ExistingWorkPolicy, + ) { + enqueueDelivery(configuration, occurrence, policy) + enqueueExpiry(configuration, occurrence, policy) + } + + private fun enqueueDelivery( + configuration: StudyConfiguration, + occurrence: InterventionOccurrence, + policy: ExistingWorkPolicy, ) { val now = System.currentTimeMillis() val delay = (occurrence.scheduledFor.wallTimeUtcMillis - now).coerceAtLeast(0) val request = OneTimeWorkRequestBuilder() .setInitialDelay(delay, TimeUnit.MILLISECONDS) .setInputData(Data.Builder().putString(InterventionWorker.KEY_OCCURRENCE_ID, occurrence.occurrenceId).build()) - .addTag(interventionTag(configuration.experimentId)) + .addTag(InterventionWorkIdentity.deliveryTag(configuration.experimentId)) .build() workManager.enqueueUniqueWork( - occurrenceWorkName(configuration.experimentId, occurrence.occurrenceId), + InterventionWorkIdentity.deliveryName(configuration.experimentId, occurrence.occurrenceId), policy, request, ) + } + + private fun enqueueExpiry( + configuration: StudyConfiguration, + occurrence: InterventionOccurrence, + policy: ExistingWorkPolicy, + ) { + val now = System.currentTimeMillis() val expiry = OneTimeWorkRequestBuilder() .setInitialDelay((occurrence.expiresAtUtcMillis - now).coerceAtLeast(0), TimeUnit.MILLISECONDS) .setInputData(Data.Builder().putString(InterventionWorker.KEY_OCCURRENCE_ID, occurrence.occurrenceId).build()) - .addTag(interventionTag(configuration.experimentId)) + .addTag(InterventionWorkIdentity.expiryTag(configuration.experimentId)) .build() workManager.enqueueUniqueWork( - "${occurrenceWorkName(configuration.experimentId, occurrence.occurrenceId)}-expiry", + InterventionWorkIdentity.expiryName(configuration.experimentId, occurrence.occurrenceId), policy, expiry, ) @@ -119,6 +152,7 @@ class AndroidStudyWorkScheduler( */ fun scheduleUpload( experimentId: String, + configurationId: String, upload: UploadConfiguration, policy: ExistingWorkPolicy, ) { @@ -137,10 +171,12 @@ class AndroidStudyWorkScheduler( .setInputData( Data.Builder() .putString(UploadWorker.KEY_EXPERIMENT_ID, experimentId) + .putString(UploadWorker.KEY_CONFIGURATION_ID, configurationId) .build(), ) + .addTag(uploadTag(experimentId)) .build() - workManager.enqueueUniqueWork(uploadWorkName(experimentId), policy, request) + workManager.enqueueUniqueWork(uploadWorkName(experimentId, configurationId), policy, request) } /** @@ -149,37 +185,139 @@ class AndroidStudyWorkScheduler( */ fun reschedulePendingWork(configuration: StudyConfiguration) { configuration.upload?.let { - scheduleUpload(configuration.experimentId, it, ExistingWorkPolicy.KEEP) + scheduleUpload( + configuration.experimentId, + configuration.configurationId, + it, + ExistingWorkPolicy.KEEP, + ) } } - override fun cancelCollectionWork(experimentId: String) { - workManager.cancelAllWorkByTag(interventionTag(experimentId)) + override fun cancelInterventionWork(experimentId: String, occurrenceIds: Set) { + workManager.cancelAllWorkByTag(InterventionWorkIdentity.deliveryTag(experimentId)) + workManager.cancelAllWorkByTag(InterventionWorkIdentity.expiryTag(experimentId)) + cancelInterventionNotifications(occurrenceIds) + } + + override fun cancelInterventionNotifications(occurrenceIds: Set) { + occurrenceIds.forEach { notificationManager.cancel(it, 0) } + } + + override fun cancelCollectionWork(experimentId: String, occurrenceIds: Set) { + cancelInterventionWork(experimentId, occurrenceIds) workManager.cancelUniqueWork(deadlineWorkName(experimentId)) } override fun cancel(experimentId: String) { - cancelCollectionWork(experimentId) - workManager.cancelUniqueWork(uploadWorkName(experimentId)) + cancelCollectionWork(experimentId, emptySet()) + workManager.cancelAllWorkByTag(uploadTag(experimentId)) } - private fun interventionTag(experimentId: String) = "adc-intervention-$experimentId" - private fun occurrenceWorkName(experimentId: String, occurrenceId: String) = "adc-intervention-$experimentId-$occurrenceId" private fun deadlineWorkName(experimentId: String) = "adc-deadline-$experimentId" + private fun uploadTag(experimentId: String) = "adc-upload-$experimentId" companion object { - fun uploadWorkName(experimentId: String) = "adc-upload-$experimentId" + fun uploadWorkName(experimentId: String, configurationId: String) = + "adc-upload-$experimentId-$configurationId" } } +internal object InterventionWorkIdentity { + fun deliveryTag(experimentId: String) = "adc-intervention-delivery-$experimentId" + fun expiryTag(experimentId: String) = "adc-intervention-expiry-$experimentId" + fun deliveryName(experimentId: String, occurrenceId: String) = "adc-intervention-$experimentId-$occurrenceId" + fun expiryName(experimentId: String, occurrenceId: String) = "${deliveryName(experimentId, occurrenceId)}-expiry" +} + +internal enum class ExpiryWorkerDirective { RETRY, COMPLETE, COMPLETE_AND_RECOVER } + +internal enum class DeliveryWorkerDirective { DELIVER, RETRY, COMPLETE, RECOVER_SUCCESSOR } + +internal fun deliveryWorkerDirective(result: OccurrenceClaimResult): DeliveryWorkerDirective = when (result) { + is OccurrenceClaimResult.Due -> DeliveryWorkerDirective.DELIVER + is OccurrenceClaimResult.NotDue -> DeliveryWorkerDirective.RETRY + OccurrenceClaimResult.Expired, + OccurrenceClaimResult.Terminal, + -> DeliveryWorkerDirective.RECOVER_SUCCESSOR + OccurrenceClaimResult.InactiveStudy, + OccurrenceClaimResult.Missing, + -> DeliveryWorkerDirective.COMPLETE +} + +internal fun expiryWorkerDirective(result: OccurrenceExpiryResult): ExpiryWorkerDirective = when (result) { + is OccurrenceExpiryResult.NotDue -> ExpiryWorkerDirective.RETRY + OccurrenceExpiryResult.Expired, + OccurrenceExpiryResult.Terminal, + -> ExpiryWorkerDirective.COMPLETE_AND_RECOVER + OccurrenceExpiryResult.InactiveStudy, + OccurrenceExpiryResult.Missing, + -> ExpiryWorkerDirective.COMPLETE +} + +/** Cancels Android's external side effect unless durable POSTING -> POSTED finalization succeeds. */ +internal suspend fun finalizePostedNotification( + finalize: suspend () -> Boolean, + cancel: () -> Unit, +): Boolean { + val finalized = try { + finalize() + } catch (failure: Throwable) { + try { + cancel() + } catch (cleanupFailure: Throwable) { + failure.addSuppressed(cleanupFailure) + } + throw failure + } + if (!finalized) cancel() + return finalized +} + +/** + * One process-local owner for notification delivery and stale-POSTING recovery. + * + * Recovery may cancel a notification that has not reached durable POSTED state, so it must never + * interleave with claim -> notify -> finalize. + */ +internal object InterventionDeliveryCoordinator { + private val mutex = Mutex() + + suspend fun run(operation: suspend () -> T): T = mutex.withLock { operation() } + + suspend fun recoverStalePosting(operation: suspend () -> T): T = mutex.withLock { operation() } +} + class InterventionWorker( context: Context, parameters: WorkerParameters, ) : CoroutineWorker(context, parameters) { - override suspend fun doWork(): Result { + override suspend fun doWork(): Result = InterventionDeliveryCoordinator.run { + try { + // Claim happens after acquiring the coordinator, so a worker never acts on a stale + // POSTING snapshot left by another in-process delivery attempt. + deliver() + } catch (failure: CancellationException) { + throw failure + } catch (_: Exception) { + Result.retry() + } + } + + private suspend fun deliver(): Result { val occurrenceId = inputData.getString(KEY_OCCURRENCE_ID) ?: return Result.failure() val application = applicationContext as cool.linc.androiddatacollector.CollectorApplication if (application.session.snapshot.first { it.initialized }.configuration == null) return Result.success() - val dispatch = application.session.claimOccurrence(occurrenceId) ?: return Result.success() + val claim = application.session.claimOccurrenceIfDue(occurrenceId) + when (deliveryWorkerDirective(claim)) { + DeliveryWorkerDirective.RETRY -> return Result.retry() + DeliveryWorkerDirective.COMPLETE -> return Result.success() + DeliveryWorkerDirective.RECOVER_SUCCESSOR -> { + application.session.scheduleSuccessor(occurrenceId) + return Result.success() + } + DeliveryWorkerDirective.DELIVER -> Unit + } + val dispatch = (claim as OccurrenceClaimResult.Due).dispatch if (applicationContext.checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED ) { @@ -199,27 +337,47 @@ class InterventionWorker( PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE, ) val manager = applicationContext.getSystemService(NotificationManager::class.java) - manager.createNotificationChannel( - NotificationChannel( - CHANNEL_ID, - applicationContext.getString(R.string.intervention_channel), - NotificationManager.IMPORTANCE_DEFAULT, - ), - ) - manager.notify( - occurrenceId, - 0, - android.app.Notification.Builder(applicationContext, CHANNEL_ID) - .setSmallIcon(android.R.drawable.ic_dialog_info) - .setContentTitle(dispatch.action.notificationTitle) - .setContentText(dispatch.action.notificationMessage) - .setStyle(android.app.Notification.BigTextStyle().bigText(dispatch.action.notificationMessage)) - .setContentIntent(pendingIntent) - .setAutoCancel(true) - .setTimeoutAfter((dispatch.occurrence.expiresAtUtcMillis - System.currentTimeMillis()).coerceAtLeast(1)) - .build(), - ) - application.session.markNotificationPosted(occurrenceId) + var finalized = false + try { + manager.createNotificationChannel( + NotificationChannel( + CHANNEL_ID, + applicationContext.getString(R.string.intervention_channel), + NotificationManager.IMPORTANCE_DEFAULT, + ), + ) + manager.notify( + occurrenceId, + 0, + android.app.Notification.Builder(applicationContext, CHANNEL_ID) + .setSmallIcon(android.R.drawable.ic_dialog_info) + .setContentTitle(dispatch.action.notificationTitle) + .setContentText(dispatch.action.notificationMessage) + .setStyle(android.app.Notification.BigTextStyle().bigText(dispatch.action.notificationMessage)) + .setContentIntent(pendingIntent) + .setAutoCancel(true) + .setTimeoutAfter((dispatch.occurrence.expiresAtUtcMillis - System.currentTimeMillis()).coerceAtLeast(1)) + .build(), + ) + // Expiry or a storage failure can win between the durable claim and Android's external + // notify() side effect. Finalization is authoritative; false and exceptions clean up. + finalized = finalizePostedNotification( + finalize = { application.session.markNotificationPosted(occurrenceId) }, + cancel = { manager.cancel(occurrenceId, 0) }, + ) + // Deliberately outside durable finalization: a successor enqueue failure must retry + // without retracting a notification whose POSTED state already committed. + application.session.scheduleSuccessor(occurrenceId) + } catch (failure: Throwable) { + if (!finalized) { + try { + manager.cancel(occurrenceId, 0) + } catch (cleanupFailure: Throwable) { + failure.addSuppressed(cleanupFailure) + } + } + throw failure + } return Result.success() } @@ -235,12 +393,26 @@ class InterventionExpiryWorker( context: Context, parameters: WorkerParameters, ) : CoroutineWorker(context, parameters) { - override suspend fun doWork(): Result { + override suspend fun doWork(): Result = try { + expire() + } catch (failure: CancellationException) { + throw failure + } catch (_: Exception) { + Result.retry() + } + + private suspend fun expire(): Result { val occurrenceId = inputData.getString(InterventionWorker.KEY_OCCURRENCE_ID) ?: return Result.failure() val application = applicationContext as cool.linc.androiddatacollector.CollectorApplication if (application.session.snapshot.first { it.initialized }.configuration == null) return Result.success() - application.session.claimOccurrence(occurrenceId) - applicationContext.getSystemService(NotificationManager::class.java).cancel(occurrenceId, 0) - return Result.success() + return when (expiryWorkerDirective(application.session.expireOccurrenceIfDue(occurrenceId))) { + ExpiryWorkerDirective.RETRY -> Result.retry() + ExpiryWorkerDirective.COMPLETE -> Result.success() + ExpiryWorkerDirective.COMPLETE_AND_RECOVER -> { + applicationContext.getSystemService(NotificationManager::class.java).cancel(occurrenceId, 0) + application.session.scheduleSuccessor(occurrenceId) + Result.success() + } + } } } diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt new file mode 100644 index 0000000..dc8fc87 --- /dev/null +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt @@ -0,0 +1,323 @@ +package cool.linc.androiddatacollector.platform + +import android.system.Os +import android.system.OsConstants +import cool.linc.androiddatacollector.core.application.StudyUploadException +import cool.linc.androiddatacollector.core.export.ExportReceipt +import java.io.BufferedInputStream +import java.io.DataInputStream +import java.io.DataOutputStream +import java.io.File +import java.io.FileInputStream +import java.io.FileOutputStream +import java.io.IOException +import java.io.OutputStream +import java.nio.ByteBuffer +import java.nio.file.Files +import java.nio.file.StandardCopyOption +import java.security.MessageDigest +import java.util.UUID + +/** One immutable ciphertext bundle durably staged before any HTTP request starts. */ +internal data class StagedUpload( + val body: File, + val receipt: ExportReceipt, + val terminalFailureCode: String?, +) + +/** + * A single-entry upload outbox. + * + * The manifest is authoritative and is published only after the body has been flushed, synced and + * atomically renamed. HTTP never sees a temporary or newly regenerated body. Removing a completed + * stage reverses the order: manifest first, then the harmless orphan body. + */ +internal class FileUploadOutbox( + private val directory: File, + private val directorySync: (File) -> Unit = ::syncDirectory, +) { + @Synchronized + fun recover( + configurationSha256: String, + uploadedThroughSequence: Long, + ): StagedUpload? { + ensureDirectory() + deleteIfExists(temporaryBody, sync = false) + deleteIfExists(temporaryManifest, sync = false) + if (!manifest.exists()) { + deleteIfExists(body, sync = true) + return null + } + + val staged = readManifest() + require(staged.receipt.configurationSha256 == configurationSha256) { + "Staged upload configuration digest mismatch" + } + if (staged.receipt.lastSequence <= uploadedThroughSequence) { + acknowledge(staged.receipt.bundleId) + return null + } + verifyBody(staged) + staged.terminalFailureCode?.let { code -> + throw StudyUploadException(code, retryable = false) + } + return staged + } + + suspend fun stage( + writeBundle: suspend (OutputStream) -> ExportReceipt, + ): StagedUpload { + ensureDirectory() + check(!manifest.exists() && !body.exists()) { "Upload outbox already contains a stage" } + deleteIfExists(temporaryBody, sync = false) + deleteIfExists(temporaryManifest, sync = false) + + val receipt = try { + FileOutputStream(temporaryBody).use { output -> + val bounded = BoundedDigestingOutputStream(output, MAXIMUM_BODY_BYTES) + val written = writeBundle(bounded) + bounded.flush() + require(written.byteCount == bounded.count) { "Staged upload length mismatch" } + require(written.sha256 == bounded.digestHex()) { "Staged upload digest mismatch" } + output.fd.sync() + written + } + } catch (failure: Exception) { + suppressCleanupFailure(failure) { Files.deleteIfExists(temporaryBody.toPath()) } + throw failure + } + val staged = StagedUpload( + body = body, + receipt = receipt, + terminalFailureCode = null, + ) + verifyReceipt(staged.receipt, temporaryBody) + moveAtomically(temporaryBody, body) + directorySync(directory) + writeManifest(staged) + return staged + } + + @Synchronized + fun markTerminal(bundleId: UUID, reasonCode: String) { + val staged = readManifest() + require(staged.receipt.bundleId == bundleId) { "Upload outbox bundle mismatch" } + writeManifest(staged.copy(terminalFailureCode = reasonCode)) + } + + @Synchronized + fun acknowledge(bundleId: UUID) { + ensureDirectory() + if (!manifest.exists()) { + deleteIfExists(body, sync = true) + return + } + val staged = readManifest() + require(staged.receipt.bundleId == bundleId) { "Upload outbox bundle mismatch" } + Files.delete(manifest.toPath()) + directorySync(directory) + deleteIfExists(body, sync = true) + } + + @Synchronized + fun clear() { + ensureDirectory() + listOf(manifest, body, temporaryManifest, temporaryBody) + .forEach { Files.deleteIfExists(it.toPath()) } + directorySync(directory) + } + + private fun verifyBody(staged: StagedUpload) { + val receipt = staged.receipt + verifyReceipt(receipt, staged.body) + val digest = MessageDigest.getInstance("SHA-256") + BufferedInputStream(FileInputStream(staged.body)).use { input -> + val buffer = ByteArray(DIGEST_BUFFER_BYTES) + while (true) { + val read = input.read(buffer) + if (read < 0) break + digest.update(buffer, 0, read) + } + } + require(digest.digest().toHex() == receipt.sha256) { "Staged upload digest mismatch" } + } + + private fun verifyReceipt(receipt: ExportReceipt, file: File) { + require(receipt.byteCount in 1..MAXIMUM_BODY_BYTES) { "Staged upload size is out of bounds" } + require(file.isFile && file.length() == receipt.byteCount) { "Staged upload length mismatch" } + require(SHA256_HEX.matches(receipt.sha256)) { "Staged upload digest is malformed" } + } + + private fun readManifest(): StagedUpload = try { + DataInputStream(FileInputStream(manifest).buffered()).use { input -> + val magic = ByteArray(MANIFEST_MAGIC.size).also(input::readFully) + require(magic.contentEquals(MANIFEST_MAGIC)) { "Unsupported upload outbox manifest" } + val bundleId = UUID(input.readLong(), input.readLong()) + val configurationDigest = ByteArray(SHA256_BYTES).also(input::readFully).toHex() + val first = input.readLong() + val last = input.readLong() + val count = input.readLong() + val byteCount = input.readLong() + val bodyDigest = ByteArray(SHA256_BYTES).also(input::readFully).toHex() + val terminalCode = input.readAscii(MAXIMUM_REASON_CODE_BYTES).ifEmpty { null } + require(input.read() == -1) { "Trailing upload outbox manifest bytes" } + StagedUpload( + body = body, + receipt = ExportReceipt( + bundleId = bundleId, + configurationSha256 = configurationDigest, + firstSequence = first, + lastSequence = last, + eventCount = count, + sha256 = bodyDigest, + byteCount = byteCount, + ), + terminalFailureCode = terminalCode, + ) + } + } catch (failure: Exception) { + if (failure is kotlinx.coroutines.CancellationException) throw failure + throw StudyUploadException("UPLOAD_OUTBOX_CORRUPT", retryable = false, cause = failure) + } + + private fun writeManifest(staged: StagedUpload) { + try { + FileOutputStream(temporaryManifest).use { fileOutput -> + DataOutputStream(fileOutput.buffered()).use { output -> + output.write(MANIFEST_MAGIC) + output.writeLong(staged.receipt.bundleId.mostSignificantBits) + output.writeLong(staged.receipt.bundleId.leastSignificantBits) + output.write(staged.receipt.configurationSha256.hexToBytes()) + output.writeLong(staged.receipt.firstSequence) + output.writeLong(staged.receipt.lastSequence) + output.writeLong(staged.receipt.eventCount) + output.writeLong(staged.receipt.byteCount) + output.write(staged.receipt.sha256.hexToBytes()) + output.writeAscii(staged.terminalFailureCode.orEmpty(), MAXIMUM_REASON_CODE_BYTES) + output.flush() + fileOutput.fd.sync() + } + } + moveAtomically(temporaryManifest, manifest) + directorySync(directory) + } catch (failure: Exception) { + suppressCleanupFailure(failure) { Files.deleteIfExists(temporaryManifest.toPath()) } + throw failure + } + } + + private fun ensureDirectory() { + if (!directory.isDirectory && !directory.mkdirs()) { + throw IOException("Cannot create upload outbox") + } + } + + private fun moveAtomically(source: File, destination: File) { + Files.move( + source.toPath(), + destination.toPath(), + StandardCopyOption.ATOMIC_MOVE, + StandardCopyOption.REPLACE_EXISTING, + ) + } + + private fun deleteIfExists(file: File, sync: Boolean) { + if (Files.deleteIfExists(file.toPath()) && sync) directorySync(directory) + } + + private inline fun suppressCleanupFailure(primary: Exception, cleanup: () -> Unit) { + try { + cleanup() + } catch (failure: Exception) { + primary.addSuppressed(failure) + } + } + + private val manifest get() = directory.resolve("stage.manifest") + private val body get() = directory.resolve("stage.adcexp") + private val temporaryManifest get() = directory.resolve("stage.manifest.tmp") + private val temporaryBody get() = directory.resolve("stage.adcexp.tmp") + + private companion object { + val MANIFEST_MAGIC = "ADCOUT01".toByteArray(Charsets.US_ASCII) + val SHA256_HEX = Regex("[0-9a-f]{64}") + const val SHA256_BYTES = 32 + const val DIGEST_BUFFER_BYTES = 64 * 1024 + const val MAXIMUM_BODY_BYTES = 32L * 1024 * 1024 + const val MAXIMUM_REASON_CODE_BYTES = 64 + } +} + +private fun DataOutputStream.writeAscii(value: String, maximumBytes: Int) { + val encoded = value.toByteArray(Charsets.US_ASCII) + require(encoded.size <= maximumBytes && value == encoded.toString(Charsets.US_ASCII)) { + "Upload outbox value is not bounded ASCII" + } + writeByte(encoded.size) + write(encoded) +} + +private fun DataInputStream.readAscii(maximumBytes: Int): String { + val length = readUnsignedByte() + require(length <= maximumBytes) { "Upload outbox value exceeds its bound" } + return ByteArray(length).also(::readFully).toString(Charsets.US_ASCII) +} + +private fun String.hexToBytes(): ByteArray { + require(length == 64 && all { it in '0'..'9' || it in 'a'..'f' }) { "Invalid SHA-256" } + return ByteBuffer.allocate(length / 2).run { + chunked(2).forEach { put(it.toInt(16).toByte()) } + array() + } +} + +private fun ByteArray.toHex(): String = joinToString("") { "%02x".format(it) } + +private fun syncDirectory(directory: File) { + try { + val descriptor = Os.open( + directory.absolutePath, + OsConstants.O_RDONLY or OsConstants.O_CLOEXEC, + 0, + ) + try { + Os.fsync(descriptor) + } finally { + Os.close(descriptor) + } + } catch (failure: android.system.ErrnoException) { + throw IOException("Cannot sync upload outbox directory", failure) + } +} + +private class BoundedDigestingOutputStream( + private val destination: OutputStream, + private val maximumBytes: Long, +) : OutputStream() { + private val digest = MessageDigest.getInstance("SHA-256") + var count = 0L + private set + + override fun write(value: Int) { + requireCapacity(1) + destination.write(value) + digest.update(value.toByte()) + count++ + } + + override fun write(bytes: ByteArray, offset: Int, length: Int) { + require(offset >= 0 && length >= 0 && offset <= bytes.size - length) { "Invalid upload write range" } + requireCapacity(length) + destination.write(bytes, offset, length) + digest.update(bytes, offset, length) + count += length + } + + override fun flush() = destination.flush() + + fun digestHex(): String = digest.digest().toHex() + + private fun requireCapacity(additionalBytes: Int) { + require(count <= maximumBytes - additionalBytes) { "Upload bundle exceeds 32 MiB" } + } +} diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloader.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloader.kt new file mode 100644 index 0000000..2e6e66b --- /dev/null +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloader.kt @@ -0,0 +1,100 @@ +package cool.linc.androiddatacollector.platform + +import cool.linc.androiddatacollector.core.protocol.JoinLink +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec +import java.io.File +import java.io.FileOutputStream +import java.io.IOException +import java.nio.file.Files +import java.security.MessageDigest +import java.util.concurrent.TimeUnit +import kotlinx.coroutines.sync.Mutex +import kotlinx.coroutines.sync.withLock +import okhttp3.OkHttpClient +import okhttp3.Request + +/** Downloads one immutable join artifact into no-backup staging, then removes every staged byte. */ +class JoinArtifactDownloader internal constructor( + private val directory: File, + private val client: OkHttpClient = defaultClient(), +) { + private val mutex = Mutex() + + init { + ensureDirectory() + deleteStaging(directory.resolve(STAGING_FILE)) + } + + suspend fun download(link: JoinLink): ByteArray = mutex.withLock { + val staging = directory.resolve(STAGING_FILE) + deleteStaging(staging) + try { + val request = Request.Builder() + .url(link.artifactUrl.toASCIIString()) + .get() + .build() + val response = client.newCall(request).awaitResponse() + response.use { + require(it.code == 200) { "Join artifact HTTP ${it.code}" } + val body = requireNotNull(it.body) { "Join artifact body is missing" } + val declared = body.contentLength() + require( + declared == -1L || + declared in 1L..SignedConfigurationCodec.MAXIMUM_ENVELOPE_BYTES.toLong(), + ) { + "Join artifact size is out of bounds" + } + val digest = MessageDigest.getInstance("SHA-256") + var count = 0L + FileOutputStream(staging).use { output -> + body.byteStream().use { input -> + val buffer = ByteArray(BUFFER_BYTES) + while (true) { + val read = input.read(buffer) + if (read < 0) break + count += read + require(count <= SignedConfigurationCodec.MAXIMUM_ENVELOPE_BYTES.toLong()) { + "Join artifact is too large" + } + digest.update(buffer, 0, read) + output.write(buffer, 0, read) + } + } + output.fd.sync() + } + require(count > 0 && (declared == -1L || count == declared)) { + "Join artifact length mismatch" + } + require(digest.digest().toHex() == link.artifactSha256) { + "Join artifact digest mismatch" + } + } + staging.readBytes() + } finally { + deleteStaging(staging) + } + } + + private fun ensureDirectory() { + if (!directory.isDirectory && !directory.mkdirs()) throw IOException("Cannot create join staging") + } + + private fun deleteStaging(staging: File) { + Files.deleteIfExists(staging.toPath()) + } + + companion object { + private const val STAGING_FILE = "artifact.adccfg.tmp" + private const val BUFFER_BYTES = 64 * 1024 + + fun defaultClient(): OkHttpClient = OkHttpClient.Builder() + .connectTimeout(30, TimeUnit.SECONDS) + .readTimeout(60, TimeUnit.SECONDS) + .followRedirects(false) + .followSslRedirects(false) + .retryOnConnectionFailure(false) + .build() + } +} + +private fun ByteArray.toHex(): String = joinToString("") { "%02x".format(it) } diff --git a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt index 0587e41..e3ef3a7 100644 --- a/app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt +++ b/app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt @@ -2,151 +2,291 @@ package cool.linc.androiddatacollector.platform import cool.linc.androiddatacollector.core.application.StudyUploadException import cool.linc.androiddatacollector.core.application.StudyUploader -import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.export.BundleKind +import cool.linc.androiddatacollector.core.export.BundleProducer import cool.linc.androiddatacollector.core.export.ExportReceipt import cool.linc.androiddatacollector.core.export.ExportSnapshot import cool.linc.androiddatacollector.core.export.ResearchExport +import cool.linc.androiddatacollector.core.export.UploadReceiptCodec import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import java.io.IOException +import java.util.Base64 +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicReference import java.util.concurrent.TimeUnit -import kotlinx.coroutines.runBlocking +import kotlin.coroutines.resume +import kotlin.coroutines.resumeWithException +import kotlinx.coroutines.CancellationException +import kotlinx.coroutines.suspendCancellableCoroutine +import kotlinx.coroutines.sync.Mutex +import kotlinx.coroutines.sync.withLock +import okhttp3.Call +import okhttp3.Callback import okhttp3.MediaType.Companion.toMediaType import okhttp3.OkHttpClient import okhttp3.Request -import okhttp3.RequestBody -import okio.BufferedSink - -/** - * Posts one encrypted bundle per call to the study's endpoint. - * - * The body is the same HPKE-encrypted bundle a participant would export by hand, so the endpoint - * receives ciphertext it cannot read. Everything the server needs in order to file and deduplicate - * a chunk travels in headers, because reading any of it out of the body would require the - * researcher's private key. - */ -class OkHttpStudyUploader( +import okhttp3.RequestBody.Companion.asRequestBody +import okhttp3.Response + +/** Stages one immutable encrypted bundle, sends it once, and accepts only its exact receipt. */ +class OkHttpStudyUploader internal constructor( + private val outbox: FileUploadOutbox, + private val producer: BundleProducer, private val client: OkHttpClient = defaultClient(), + private val nowUtcMillis: () -> Long = System::currentTimeMillis, ) : StudyUploader { + private val mutex = Mutex() + private val deletionRequested = AtomicBoolean(false) + private val activeCall = AtomicReference(null) + + override suspend fun reconcile(configuration: VerifiedConfiguration, metadata: StudyMetadata) { + mutex.withLock { + requireUploadAllowed() + recover(configuration, metadata) + } + } override suspend fun upload( - configuration: StudyConfiguration, + configuration: VerifiedConfiguration, metadata: StudyMetadata, events: StudyStore, fromSequence: Long, toSequence: Long, - ): ExportReceipt { - val upload = requireNotNull(configuration.upload) { "Study does not define an upload endpoint" } - var receipt: ExportReceipt? = null - - val body = object : RequestBody() { - override fun contentType() = "application/octet-stream".toMediaType() - - /** - * The bundle is generated as it is written, so its length is not known up front and - * OkHttp falls back to chunked transfer encoding. - */ - override fun contentLength() = -1L - - /** The stream is built once from the store; OkHttp must not silently replay it. */ - override fun isOneShot() = true - - override fun writeTo(sink: BufferedSink) { - receipt = runBlocking { - ResearchExport.encrypt( - ExportSnapshot( - configuration = configuration, - metadata = metadata, - exportedAtUtcMillis = System.currentTimeMillis(), - fromSequence = fromSequence, - toSequence = toSequence, - maximumPlaintextBytes = BUDGET_BYTES, - ), - events, - sink.outputStream(), - ) + ): ExportReceipt = mutex.withLock { + requireUploadAllowed() + val upload = requireNotNull(configuration.configuration.upload) { + "Study does not define an upload endpoint" + } + val staged = recover(configuration, metadata) ?: stage( + configuration, + metadata, + events, + fromSequence, + toSequence, + ) + require(staged.receipt.firstSequence == fromSequence) { "Staged upload range start mismatch" } + require(staged.receipt.lastSequence in fromSequence..toSequence) { "Staged upload range end mismatch" } + requireUploadAllowed() + + val request = Request.Builder() + .url(upload.endpoint) + .apply { + uploadHeaders(configuration, staged.receipt).forEach { (name, value) -> + header(name, value) } } - } + .post(staged.body.asRequestBody(BUNDLE_MEDIA_TYPE)) + .build() - val requestBuilder = Request.Builder().url(upload.endpoint) - uploadHeaders(configuration, metadata, fromSequence, toSequence).forEach { (name, value) -> - requestBuilder.header(name, value) + val call = client.newCall(request) + check(activeCall.compareAndSet(null, call)) { "Only one upload call may be active" } + if (deletionRequested.get()) call.cancel() + val response = try { + call.awaitResponse() + } catch (failure: Exception) { + failure.rethrowCancellation() + if (deletionRequested.get()) throw deletionFailure(failure) + throw failure.asUploadFailure() + } finally { + activeCall.compareAndSet(call, null) } - val request = requestBuilder - // Named "at most" because headers are sent before the body is generated, and a budget - // can stop the bundle at any earlier event boundary. The endpoint cannot learn the true - // upper bound — that is inside the ciphertext — so it must not file by this value. - // `X-ADC-Sequence-From` is exact and strictly increasing per participant, which is what - // makes a usable deduplication key. - .post(body) - .build() + response.use { + if (it.code !in ACCEPTED_STATUS_CODES) { + val reason = "UPLOAD_HTTP_${it.code}" + if (it.code.isRetryableStatus()) { + throw StudyUploadException(reason, retryable = true) + } + markTerminal(staged, reason) + } - try { - client.newCall(request).execute().use { response -> - if (!response.isSuccessful) { - throw StudyUploadException("UPLOAD_HTTP_${response.code}") + val receipt = try { + val body = requireNotNull(it.body) { "Upload receipt body is missing" } + require(body.contentType()?.let { type -> type.type == "application" && type.subtype == "json" } == true) { + "Upload receipt media type is invalid" } + val declaredLength = body.contentLength() + require(declaredLength == -1L || declaredLength <= MAXIMUM_RECEIPT_BYTES) { + "Upload receipt body is too large" + } + val bytes = body.byteStream().use { input -> + input.readNBytes(MAXIMUM_RECEIPT_BYTES + 1) + } + require(bytes.size <= MAXIMUM_RECEIPT_BYTES) { "Upload receipt body is too large" } + UploadReceiptCodec.decode(bytes) + } catch (failure: Exception) { + failure.rethrowCancellation() + markTerminal(staged, "UPLOAD_RECEIPT_INVALID", failure) + } + if (receipt != staged.receipt) { + markTerminal(staged, "UPLOAD_RECEIPT_MISMATCH") } - } catch (failure: StudyUploadException) { - throw failure - } catch (failure: Throwable) { - if (failure is kotlinx.coroutines.CancellationException) throw failure - // The classified code is what reaches the participant's screen. The transport exception - // is logged as well because a delivery problem is otherwise undiagnosable from a device, - // and a network error carries no study data. - android.util.Log.w("AdcUpload", "Upload failed", failure) - throw StudyUploadException(failure.reasonCode(), failure) + receipt + } + } + + override suspend fun acknowledge(bundleId: java.util.UUID) = mutex.withLock { + outboxCall { outbox.acknowledge(bundleId) } + } + + override suspend fun prepareDeletion() { + deletionRequested.set(true) + activeCall.get()?.cancel() + // Wait for staging/request teardown. A stage created concurrently observes the flag + // before opening HTTP; an active call is cancelled above. + mutex.withLock { activeCall.get()?.cancel() } + } + + override suspend fun clear() = mutex.withLock { + outboxCall { outbox.clear() } + deletionRequested.set(false) + } + + private suspend fun stage( + configuration: VerifiedConfiguration, + metadata: StudyMetadata, + events: StudyStore, + fromSequence: Long, + toSequence: Long, + ): StagedUpload = try { + outbox.stage { destination -> + ResearchExport.encrypt( + ExportSnapshot( + verifiedConfiguration = configuration, + metadata = metadata, + producer = producer, + bundleKind = BundleKind.AUTOMATIC_UPLOAD, + exportedAtUtcMillis = nowUtcMillis(), + fromSequence = fromSequence, + toSequence = toSequence, + maximumPlaintextBytes = TARGET_PLAINTEXT_BYTES, + ), + events, + destination, + ) } - return requireNotNull(receipt) { "Upload completed without producing a receipt" } + } catch (failure: Exception) { + failure.rethrowCancellation() + throw failure.asOutboxFailure() + } + + private fun recover( + configuration: VerifiedConfiguration, + metadata: StudyMetadata, + ): StagedUpload? = outboxCall { + outbox.recover( + configurationSha256 = configuration.configurationSha256, + uploadedThroughSequence = metadata.uploadedThroughSequence, + ) + } + + private fun markTerminal(staged: StagedUpload, reasonCode: String, cause: Throwable? = null): Nothing { + outboxCall { outbox.markTerminal(staged.receipt.bundleId, reasonCode) } + throw StudyUploadException(reasonCode, retryable = false, cause = cause) + } + + private fun requireUploadAllowed() { + if (deletionRequested.get()) throw deletionFailure() + } + + private fun deletionFailure(cause: Throwable? = null) = + StudyUploadException("UPLOAD_CANCELLED_FOR_DELETION", retryable = false, cause = cause) + + private fun outboxCall(block: () -> T): T = try { + block() + } catch (failure: Exception) { + failure.rethrowCancellation() + if (failure is StudyUploadException) throw failure + throw failure.asOutboxFailure() } companion object { - /** - * Plaintext budget for one request. The study's configured cadence is what paces delivery; - * this only keeps a single request to a size a phone on a real network can finish, and only - * binds while a backlog is being worked off. A bundle stops at the first event boundary past - * it and the next run picks up from there. - */ - const val BUDGET_BYTES = 16L * 1024 * 1024 + const val TARGET_PLAINTEXT_BYTES = 16L * 1024 * 1024 + const val MEDIA_TYPE = "application/vnd.adc.research-bundle" + private const val MAXIMUM_RECEIPT_BYTES = 2_048 + private val ACCEPTED_STATUS_CODES = setOf(200, 201) + private val BUNDLE_MEDIA_TYPE = MEDIA_TYPE.toMediaType() fun defaultClient(): OkHttpClient = OkHttpClient.Builder() .connectTimeout(30, TimeUnit.SECONDS) .writeTimeout(5, TimeUnit.MINUTES) .readTimeout(60, TimeUnit.SECONDS) - // Left on, which is the default. OkHttp already refuses to replay a one-shot body once - // it has started sending, so this only recovers the case where a pooled connection turns - // out to be dead before anything was written — a hard failure otherwise, and a common - // one between widely spaced uploads. - .retryOnConnectionFailure(true) + .followRedirects(false) + .followSslRedirects(false) + .retryOnConnectionFailure(false) .build() } } -/** The complete unencrypted request surface. Assigned participant IDs are deliberately absent. */ +/** Complete unencrypted request surface. Assigned participant IDs are deliberately absent. */ internal fun uploadHeaders( - configuration: StudyConfiguration, - metadata: StudyMetadata, - fromSequence: Long, - toSequence: Long, -): Map = mapOf( - "Content-Type" to "application/octet-stream", - "X-ADC-Bundle-Format" to ResearchExport.BUNDLE_FORMAT, - "X-ADC-Experiment-Id" to configuration.experimentId, - "X-ADC-Configuration-Id" to configuration.configurationId, - "X-ADC-Participant-Instance" to metadata.participantInstanceId, - "X-ADC-Sequence-From" to fromSequence.toString(), - "X-ADC-Sequence-To-At-Most" to toSequence.toString(), -) - -/** Maps a transport failure onto a fixed code, so what reaches a screen or a log is never data. */ -private fun Throwable.reasonCode(): String = when (this) { - is java.net.SocketTimeoutException -> "UPLOAD_TIMEOUT" - is java.net.UnknownHostException -> "UPLOAD_HOST_UNRESOLVED" - is java.net.ConnectException -> "UPLOAD_CONNECT_REFUSED" - is javax.net.ssl.SSLHandshakeException -> "UPLOAD_TLS_HANDSHAKE_FAILED" - is javax.net.ssl.SSLException -> "UPLOAD_TLS_FAILED" - is java.io.InterruptedIOException -> "UPLOAD_INTERRUPTED" - is IOException -> "UPLOAD_IO_FAILED" - else -> "UPLOAD_FAILED" + configuration: VerifiedConfiguration, + receipt: ExportReceipt, +): Map { + require(receipt.configurationSha256 == configuration.configurationSha256) { + "Upload receipt configuration digest mismatch" + } + return mapOf( + "Content-Type" to OkHttpStudyUploader.MEDIA_TYPE, + "Content-Length" to receipt.byteCount.toString(), + "Content-Digest" to "sha-256=:${receipt.sha256.hexDigestBase64()}:", + "X-ADC-Bundle-Id" to receipt.bundleId.toString(), + "X-ADC-Bundle-Format" to ResearchExport.BUNDLE_FORMAT, + "X-ADC-Configuration-SHA256" to receipt.configurationSha256, + "X-ADC-Researcher-Key-Id" to configuration.configuration.export.researcherKeyId, + "X-ADC-Sequence-From" to receipt.firstSequence.toString(), + "X-ADC-Sequence-To" to receipt.lastSequence.toString(), + "X-ADC-Event-Count" to receipt.eventCount.toString(), + ) +} + +private fun Int.isRetryableStatus(): Boolean = this in setOf(408, 425, 429) || this in 500..599 + +private fun Throwable.asUploadFailure(): StudyUploadException = when (this) { + is StudyUploadException -> this + is java.net.SocketTimeoutException -> StudyUploadException("UPLOAD_TIMEOUT", retryable = true, cause = this) + is java.net.UnknownHostException -> StudyUploadException("UPLOAD_HOST_UNRESOLVED", retryable = true, cause = this) + is java.net.ConnectException -> StudyUploadException("UPLOAD_CONNECT_REFUSED", retryable = true, cause = this) + is javax.net.ssl.SSLHandshakeException -> + StudyUploadException("UPLOAD_TLS_HANDSHAKE_FAILED", retryable = true, cause = this) + is javax.net.ssl.SSLException -> StudyUploadException("UPLOAD_TLS_FAILED", retryable = true, cause = this) + is java.io.InterruptedIOException -> StudyUploadException("UPLOAD_INTERRUPTED", retryable = true, cause = this) + is IOException -> StudyUploadException("UPLOAD_IO_FAILED", retryable = true, cause = this) + else -> StudyUploadException("UPLOAD_FAILED", retryable = false, cause = this) +} + +private fun Throwable.asOutboxFailure(): StudyUploadException = when (this) { + is StudyUploadException -> this + is IOException -> StudyUploadException("UPLOAD_OUTBOX_IO", retryable = true, cause = this) + else -> StudyUploadException("UPLOAD_OUTBOX_CORRUPT", retryable = false, cause = this) +} + +private fun Throwable.rethrowCancellation() { + if (this is CancellationException) throw this +} + +internal suspend fun Call.awaitResponse(): Response = suspendCancellableCoroutine { continuation -> + continuation.invokeOnCancellation { cancel() } + enqueue( + object : Callback { + override fun onFailure(call: Call, e: IOException) { + if (continuation.isActive) continuation.resumeWithException(e) + } + + override fun onResponse(call: Call, response: Response) { + if (continuation.isActive) { + continuation.resume(response) { _, value, _ -> value.close() } + } else { + response.close() + } + } + }, + ) +} + +private fun String.hexDigestBase64(): String { + require(length == 64 && all { it in '0'..'9' || it in 'a'..'f' }) { "Invalid SHA-256" } + val bytes = ByteArray(32) { index -> substring(index * 2, index * 2 + 2).toInt(16).toByte() } + return Base64.getEncoder().encodeToString(bytes) } diff --git a/app/src/main/res/values-zh-rTW/strings.xml b/app/src/main/res/values-zh-rTW/strings.xml index 3eefd9e..9c8c16d 100644 --- a/app/src/main/res/values-zh-rTW/strings.xml +++ b/app/src/main/res/values-zh-rTW/strings.xml @@ -71,6 +71,23 @@ 手機的移動狀況，每秒約記錄 %1$d 次以上 + 電池情境 + 整數電量百分比、充電狀態與來源，以及省電模式；不收集電池健康、溫度或硬體識別碼 + + 時間情境 + 時區設定、UTC 偏移、日光節約時間狀態與時鐘變更；這是裝置設定，不代表位置或旅行 + + 手機旋轉 + + 手機的原始旋轉速度，每秒約記錄 %1$d 次以上；不推論方向或活動 + + + 環境光線 + 光線變化至少 %2$d millilux 時，最多每隔 %1$s記錄原始照度；不會記錄環境內容 + + 接近感測器 + 距離變化至少 %2$d mm 或遠近狀態切換時，最多每隔 %1$s記錄原始接近或距離狀態。許多手機只能回報近或遠，不能據此判定有人在場 + 連線類型 目前使用 Wi-Fi 或行動網路，以及連線是否按流量計費 @@ -87,6 +104,8 @@ 你在研究鍵盤內的觸控方式，包括觸碰位置、時間與力道 %1$d 秒 + %1$d 毫秒 + %1$d 微秒 %1$d 分鐘 %1$d 小時 @@ -134,6 +153,9 @@ 啟用研究鍵盤 選擇研究鍵盤 加速度感測器 + 陀螺儀 + 環境光感測器 + 接近感測器 %1$s 筆 diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml index 43437ec..cd81cfb 100644 --- a/app/src/main/res/values/strings.xml +++ b/app/src/main/res/values/strings.xml @@ -85,6 +85,24 @@ Movement of the phone, about %1$d times per second or more + Battery context + Whole battery percentage, charging state and source, and power-save mode — not battery health, temperature, or hardware identity + + Time context + Time-zone setting, UTC offset, daylight-saving state, and clock changes — a setting, not a location or travel claim + + Phone rotation + + Raw rotation of the phone, about once a second or more; no orientation or activity is inferred + Raw rotation of the phone, about %1$d times per second or more; no orientation or activity is inferred + + + Ambient light + Raw light level at most every %1$s after a change of %2$d millilux; it does not record environmental content + + Proximity sensor + Raw near/distance state at most every %1$s after a %2$d mm change or any near/far transition. Many phones report only near or far; this is not a presence claim + Connection type Whether you are on Wi-Fi or mobile data, and whether it is metered @@ -102,6 +120,8 @@ %1$d s + %1$d ms + %1$d µs %1$d min %1$d h @@ -155,6 +175,9 @@ Enable the research keyboard Select the research keyboard Motion sensor + Gyroscope + Ambient-light sensor + Proximity sensor diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/CollectorSummaryTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/CollectorSummaryTest.kt new file mode 100644 index 0000000..99058c5 --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/CollectorSummaryTest.kt @@ -0,0 +1,14 @@ +package cool.linc.androiddatacollector + +import org.junit.Assert.assertEquals +import org.junit.Test + +class CollectorSummaryTest { + @Test + fun durationUnitNeverTruncatesOffLadderConfigurationValues() { + assertEquals(ExactDuration.Microseconds(1_500_500), exactDuration(1_500_500)) + assertEquals(ExactDuration.Milliseconds(1_500), exactDuration(1_500_000)) + assertEquals(ExactDuration.Seconds(90), exactDuration(90_000_000)) + assertEquals(ExactDuration.Minutes(2), exactDuration(120_000_000)) + } +} diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/DemoStudyAssetTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/DemoStudyAssetTest.kt new file mode 100644 index 0000000..066a62c --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/DemoStudyAssetTest.kt @@ -0,0 +1,29 @@ +package cool.linc.androiddatacollector + +import cool.linc.androiddatacollector.core.protocol.ConfigurationVerifier +import java.nio.file.Files +import java.nio.file.Path +import java.time.Instant +import java.util.Base64 +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Test + +class DemoStudyAssetTest { + @Test + fun debugDemoIsAValidCurrentProtocolV1Artifact() { + val projectDirectory = Path.of(requireNotNull(System.getProperty("adc.appProjectDir"))) + val encoded = Files.readString( + projectDirectory.resolve("src/debug/res/raw/demo_study_envelope.txt"), + ) + val verified = ConfigurationVerifier( + trustedSigningKeys = emptyMap(), + clientVersion = 1, + now = { Instant.parse("2026-08-04T00:00:00Z") }, + ).verify(Base64.getDecoder().decode(encoded.trim())) + + assertEquals("modular-sensing-demo", verified.configuration.experimentId) + assertEquals("demo-config-2026", verified.configuration.configurationId) + assertFalse(verified.signerAnchored) + } +} diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt new file mode 100644 index 0000000..3234ba2 --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt @@ -0,0 +1,199 @@ +package cool.linc.androiddatacollector.platform + +import cool.linc.androiddatacollector.core.application.StudyUploadException +import cool.linc.androiddatacollector.core.export.ExportReceipt +import java.io.IOException +import java.security.MessageDigest +import java.util.UUID +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder + +class FileUploadOutboxTest { + @get:Rule + val temporaryFolder = TemporaryFolder() + + @Test + fun stageIsRecoveredWithoutRegeneratingBytes() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("outbox") + val outbox = FileUploadOutbox(directory) {} + val bytes = "immutable ciphertext".toByteArray() + var writes = 0 + val staged = outbox.stage { output -> + writes++ + output.write(bytes) + receipt(bytes) + } + + val recovered = outbox.recover( + CONFIGURATION_DIGEST, + uploadedThroughSequence = 0, + ) + + assertEquals(1, writes) + assertEquals(staged.receipt, recovered?.receipt) + assertArrayEquals(bytes, recovered?.body?.readBytes()) + } + + @Test + fun persistedWatermarkRemovesAlreadyAcknowledgedStageDuringRecovery() = + kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("covered") + val outbox = FileUploadOutbox(directory) {} + val bytes = byteArrayOf(1, 2, 3) + outbox.stage { output -> + output.write(bytes) + receipt(bytes, first = 4, last = 6) + } + + assertNull( + outbox.recover( + CONFIGURATION_DIGEST, + uploadedThroughSequence = 6, + ), + ) + assertTrue(directory.listFiles().orEmpty().isEmpty()) + } + + @Test + fun corruptedBodyFailsClosed() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("corrupt") + val outbox = FileUploadOutbox(directory) {} + val bytes = byteArrayOf(1, 2, 3) + val staged = outbox.stage { output -> + output.write(bytes) + receipt(bytes) + } + staged.body.writeBytes(byteArrayOf(9, 9, 9)) + + val failure = runCatching { + outbox.recover( + CONFIGURATION_DIGEST, + uploadedThroughSequence = 0, + ) + }.exceptionOrNull() + + assertTrue(failure is IllegalArgumentException) + assertTrue(staged.body.exists()) + } + + @Test + fun terminalFailureSurvivesProcessRecreationUntilDeletion() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("terminal") + val outbox = FileUploadOutbox(directory) {} + val bytes = byteArrayOf(1, 2, 3) + val staged = outbox.stage { output -> + output.write(bytes) + receipt(bytes) + } + outbox.markTerminal(staged.receipt.bundleId, "UPLOAD_HTTP_400") + + val failure = runCatching { + FileUploadOutbox(directory) {}.recover( + CONFIGURATION_DIGEST, + uploadedThroughSequence = 0, + ) + }.exceptionOrNull() + + assertTrue(failure is StudyUploadException) + assertEquals("UPLOAD_HTTP_400", (failure as StudyUploadException).reasonCode) + assertFalse(failure.retryable) + outbox.clear() + assertTrue(directory.listFiles().orEmpty().isEmpty()) + } + + @Test + fun stagingStopsAtTheHardBodyLimitAndPublishesNothing() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("bounded") + val outbox = FileUploadOutbox(directory) {} + val chunk = ByteArray(64 * 1024) + + val failure = runCatching { + outbox.stage { output -> + repeat((MAXIMUM_BODY_BYTES / chunk.size).toInt() + 1) { output.write(chunk) } + error("The bounded stream must reject this body") + } + }.exceptionOrNull() + + assertTrue(failure is IllegalArgumentException) + assertTrue(directory.listFiles().orEmpty().isEmpty()) + } + + @Test + fun writerReceiptMustMatchTheBytesActuallyStaged() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("receipt-mismatch") + val outbox = FileUploadOutbox(directory) {} + val bytes = byteArrayOf(1, 2, 3) + + val failure = runCatching { + outbox.stage { output -> + output.write(bytes) + receipt(bytes).copy(sha256 = "f".repeat(64)) + } + }.exceptionOrNull() + + assertTrue(failure is IllegalArgumentException) + assertTrue(directory.listFiles().orEmpty().isEmpty()) + } + + @Test + fun orphanBodyFromDirectorySyncFailureIsRemovedOnRecovery() = kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("orphan-body") + val bytes = byteArrayOf(1, 2, 3) + val outbox = FileUploadOutbox(directory) { throw IOException("sync failed") } + + assertTrue(runCatching { + outbox.stage { output -> + output.write(bytes) + receipt(bytes) + } + }.isFailure) + assertTrue(directory.resolve("stage.adcexp").exists()) + + assertNull(FileUploadOutbox(directory) {}.recover(CONFIGURATION_DIGEST, 0)) + assertTrue(directory.listFiles().orEmpty().isEmpty()) + } + + @Test + fun publishedManifestRemainsRecoverableIfItsDirectorySyncReportsFailure() = + kotlinx.coroutines.runBlocking { + val directory = temporaryFolder.newFolder("published-manifest") + val bytes = byteArrayOf(1, 2, 3) + var syncCount = 0 + val outbox = FileUploadOutbox(directory) { + syncCount += 1 + if (syncCount == 2) throw IOException("manifest sync failed") + } + + assertTrue(runCatching { + outbox.stage { output -> + output.write(bytes) + receipt(bytes) + } + }.isFailure) + + val recovered = FileUploadOutbox(directory) {}.recover(CONFIGURATION_DIGEST, 0) + assertArrayEquals(bytes, recovered?.body?.readBytes()) + } + + private fun receipt(bytes: ByteArray, first: Long = 1, last: Long = 1): ExportReceipt = ExportReceipt( + bundleId = UUID.fromString("00000000-0000-4000-8000-000000000001"), + configurationSha256 = CONFIGURATION_DIGEST, + firstSequence = first, + lastSequence = last, + eventCount = last - first + 1, + sha256 = MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }, + byteCount = bytes.size.toLong(), + ) + + private companion object { + const val CONFIGURATION_DIGEST = + "0000000000000000000000000000000000000000000000000000000000000000" + const val MAXIMUM_BODY_BYTES = 32L * 1024 * 1024 + } +} diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/InterventionWorkPolicyTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/InterventionWorkPolicyTest.kt new file mode 100644 index 0000000..97d41fc --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/InterventionWorkPolicyTest.kt @@ -0,0 +1,169 @@ +package cool.linc.androiddatacollector.platform + +import cool.linc.androiddatacollector.core.definition.NotificationAction +import cool.linc.androiddatacollector.core.model.InterventionOccurrence +import cool.linc.androiddatacollector.core.model.OccurrenceState +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.runtime.OccurrenceClaimResult +import cool.linc.androiddatacollector.core.runtime.OccurrenceDispatch +import cool.linc.androiddatacollector.core.runtime.OccurrenceExpiryResult +import kotlinx.coroutines.CompletableDeferred +import kotlinx.coroutines.async +import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertSame +import org.junit.Test + +class InterventionWorkPolicyTest { + @Test + fun deliveryAndExpiryUseIndependentUniqueNamesAndCancellationTags() { + val experimentId = "experiment-one" + val occurrenceId = "a".repeat(64) + + assertNotEquals( + InterventionWorkIdentity.deliveryName(experimentId, occurrenceId), + InterventionWorkIdentity.expiryName(experimentId, occurrenceId), + ) + assertNotEquals( + InterventionWorkIdentity.deliveryTag(experimentId), + InterventionWorkIdentity.expiryTag(experimentId), + ) + assertEquals( + "${InterventionWorkIdentity.deliveryName(experimentId, occurrenceId)}-expiry", + InterventionWorkIdentity.expiryName(experimentId, occurrenceId), + ) + } + + @Test + fun earlyWorkersRetryWhileDueOrTerminalResultsComplete() { + val dispatch = OccurrenceDispatch( + InterventionOccurrence( + occurrenceId = "b".repeat(64), + interventionId = "notice-one", + triggerId = "after-minute", + scheduleKey = "relative:1", + scheduledFor = ResearchTime(1_000, 1_000, "boot-test"), + expiresAtUtcMillis = 2_000, + state = OccurrenceState.POSTING, + ), + NotificationAction("Check in", "Open the study."), + ) + + assertEquals( + DeliveryWorkerDirective.DELIVER, + deliveryWorkerDirective(OccurrenceClaimResult.Due(dispatch)), + ) + assertEquals( + DeliveryWorkerDirective.RETRY, + deliveryWorkerDirective(OccurrenceClaimResult.NotDue(1)), + ) + assertEquals( + DeliveryWorkerDirective.RECOVER_SUCCESSOR, + deliveryWorkerDirective(OccurrenceClaimResult.Expired), + ) + assertEquals( + ExpiryWorkerDirective.RETRY, + expiryWorkerDirective(OccurrenceExpiryResult.NotDue(1)), + ) + assertEquals( + ExpiryWorkerDirective.COMPLETE_AND_RECOVER, + expiryWorkerDirective(OccurrenceExpiryResult.Expired), + ) + assertEquals( + ExpiryWorkerDirective.COMPLETE_AND_RECOVER, + expiryWorkerDirective(OccurrenceExpiryResult.Terminal), + ) + assertEquals( + ExpiryWorkerDirective.COMPLETE, + expiryWorkerDirective(OccurrenceExpiryResult.InactiveStudy), + ) + } + + @Test + fun notificationFinalizationCancelsOnRejectedOrFailedDurableCommit() = runBlocking { + var cancellations = 0 + finalizePostedNotification(finalize = { true }, cancel = { cancellations += 1 }) + assertEquals(0, cancellations) + + finalizePostedNotification(finalize = { false }, cancel = { cancellations += 1 }) + assertEquals(1, cancellations) + + val expected = IllegalStateException("storage failed") + var caught: Throwable? = null + try { + finalizePostedNotification( + finalize = { throw expected }, + cancel = { cancellations += 1 }, + ) + } catch (failure: Throwable) { + caught = failure + } + assertSame(expected, caught) + assertEquals(2, cancellations) + } + + @Test + fun failedDeliveryAttemptReleasesTheCoordinatorBeforeTheNextAttemptRuns() = runBlocking { + val firstEntered = CompletableDeferred() + val releaseFailure = CompletableDeferred() + val order = mutableListOf() + + coroutineScope { + val first = async { + runCatching { + InterventionDeliveryCoordinator.run { + order += "first-entered" + firstEntered.complete(Unit) + releaseFailure.await() + throw IllegalStateException("durable finalize failed") + } + } + } + firstEntered.await() + val second = async { + InterventionDeliveryCoordinator.run { order += "second-entered" } + } + order += "second-waiting" + releaseFailure.complete(Unit) + first.await() + second.await() + } + + assertEquals(listOf("first-entered", "second-waiting", "second-entered"), order) + } + + @Test + fun stalePostingRecoveryCannotInterleaveNotificationFinalization() = runBlocking { + val notificationPosted = CompletableDeferred() + val allowFinalization = CompletableDeferred() + val order = mutableListOf() + + coroutineScope { + val delivery = async { + InterventionDeliveryCoordinator.run { + order += "notified" + notificationPosted.complete(Unit) + allowFinalization.await() + order += "finalized" + } + } + notificationPosted.await() + val recovery = async { + InterventionDeliveryCoordinator.recoverStalePosting { + order += "recovered" + } + } + order += "recovery-waiting" + allowFinalization.complete(Unit) + delivery.await() + recovery.await() + } + + assertEquals( + listOf("notified", "recovery-waiting", "finalized", "recovered"), + order, + ) + } +} diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloaderTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloaderTest.kt new file mode 100644 index 0000000..1110311 --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloaderTest.kt @@ -0,0 +1,91 @@ +package cool.linc.androiddatacollector.platform + +import cool.linc.androiddatacollector.core.protocol.JoinLink +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec +import java.net.URI +import java.nio.file.Files +import java.security.MessageDigest +import kotlinx.coroutines.runBlocking +import okhttp3.OkHttpClient +import okhttp3.Protocol +import okhttp3.Response +import okhttp3.ResponseBody.Companion.toResponseBody +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder + +class JoinArtifactDownloaderTest { + @get:Rule + val temporaryFolder = TemporaryFolder() + + @Test + fun exactArtifactIsReturnedAfterOneRequestAndStagingIsRemoved() = runBlocking { + val bytes = "signed configuration".toByteArray() + var calls = 0 + val downloader = downloader(bytes) { calls += 1 } + + val loaded = downloader.download(link(bytes)) + + assertArrayEquals(bytes, loaded) + assertTrue(temporaryFolder.root.walkTopDown().none { it.isFile }) + assertTrue(calls == 1) + } + + @Test + fun digestStatusAndSizeFailuresPublishNoStagedBytes() = runBlocking { + val bytes = byteArrayOf(1, 2, 3) + val wrongDigest = link(bytes).copy(artifactSha256 = "f".repeat(64)) + assertTrue(runCatching { downloader(bytes).download(wrongDigest) }.isFailure) + assertTrue(runCatching { downloader(bytes, status = 302).download(link(bytes)) }.isFailure) + val oversized = ByteArray(SignedConfigurationCodec.MAXIMUM_ENVELOPE_BYTES + 1) + assertTrue(runCatching { downloader(oversized).download(link(oversized)) }.isFailure) + assertTrue(temporaryFolder.root.walkTopDown().none { it.isFile }) + } + + @Test + fun defaultClientForbidsRedirectsAndImplicitRetries() { + val client = JoinArtifactDownloader.defaultClient() + assertFalse(client.followRedirects) + assertFalse(client.followSslRedirects) + assertFalse(client.retryOnConnectionFailure) + } + + @Test + fun initializationRemovesAStalePartialArtifactBeforeAnyJoinRequest() { + val directory = temporaryFolder.root.resolve("join") + assertTrue(directory.mkdirs()) + val stale = directory.resolve("artifact.adccfg.tmp") + Files.write(stale.toPath(), byteArrayOf(1, 2, 3)) + + JoinArtifactDownloader(directory, OkHttpClient()) + + assertFalse(stale.exists()) + } + + private fun downloader( + body: ByteArray, + status: Int = 200, + onRequest: () -> Unit = {}, + ): JoinArtifactDownloader { + val client = OkHttpClient.Builder().addInterceptor { chain -> + onRequest() + Response.Builder() + .request(chain.request()) + .protocol(Protocol.HTTP_1_1) + .code(status) + .message("test") + .body(body.toResponseBody()) + .build() + }.build() + return JoinArtifactDownloader(temporaryFolder.root.resolve("join"), client) + } + + private fun link(bytes: ByteArray) = JoinLink( + URI("https://artifacts.example.invalid/opaque-token"), + MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }, + "A".repeat(32), + ) +} diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt new file mode 100644 index 0000000..c62428a --- /dev/null +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt @@ -0,0 +1,359 @@ +package cool.linc.androiddatacollector.platform + +import cool.linc.androiddatacollector.core.application.StudyUploadException +import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration +import cool.linc.androiddatacollector.core.definition.ExportConfiguration +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url +import cool.linc.androiddatacollector.core.definition.SignerIdentity +import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import cool.linc.androiddatacollector.core.definition.UploadConfiguration +import cool.linc.androiddatacollector.core.export.BundleProducer +import cool.linc.androiddatacollector.core.export.ExportReceipt +import cool.linc.androiddatacollector.core.export.ResearchExport +import cool.linc.androiddatacollector.core.export.UploadReceiptCodec +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.model.StorageUsage +import cool.linc.androiddatacollector.core.model.StudyMetadata +import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration +import java.io.IOException +import java.security.MessageDigest +import java.time.Instant +import java.util.UUID +import kotlinx.coroutines.runBlocking +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.OkHttpClient +import okhttp3.Protocol +import okhttp3.Response +import okhttp3.ResponseBody.Companion.toResponseBody +import okio.Buffer +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder + +class OkHttpStudyUploaderTest { + @get:Rule + val temporaryFolder = TemporaryFolder() + + @Test + fun responseLossRetriesTheIdenticalFixedLengthBundleAndAcceptsExactReplay() = runBlocking { + val directory = temporaryFolder.newFolder("replay") + val outbox = FileUploadOutbox(directory) {} + val body = "ciphertext bytes".toByteArray() + val expected = stage(outbox, body) + val sentBodies = mutableListOf() + var calls = 0 + val client = client { request -> + sentBodies += Buffer().also { request.body!!.writeTo(it) }.readByteArray() + calls++ + if (calls == 1) throw IOException("response lost") + response(request, 200, UploadReceiptCodec.encode(expected)) + } + val uploader = uploader(outbox, client) + + val first = runCatching { uploader.upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) }.exceptionOrNull() + assertTrue(first is StudyUploadException) + assertTrue((first as StudyUploadException).retryable) + + assertEquals(expected, uploader.upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1)) + assertEquals(2, sentBodies.size) + assertArrayEquals(body, sentBodies[0]) + assertArrayEquals(sentBodies[0], sentBodies[1]) + } + + @Test + fun requestCarriesOnlyBoundedReceiverMetadataAndAnRfcContentDigest() = runBlocking { + val directory = temporaryFolder.newFolder("headers") + val outbox = FileUploadOutbox(directory) {} + val expected = stage(outbox, byteArrayOf(1, 2, 3)) + var headers: okhttp3.Headers? = null + val client = client { request -> + headers = request.headers + response(request, 201, UploadReceiptCodec.encode(expected)) + } + + uploader(outbox, client).upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) + + val sent = requireNotNull(headers) + assertEquals(expected.byteCount.toString(), sent["Content-Length"]) + assertTrue(requireNotNull(sent["Content-Digest"]).matches(Regex("sha-256=:[A-Za-z0-9+/]+={0,2}:"))) + assertEquals(OkHttpStudyUploader.MEDIA_TYPE, sent["Content-Type"]) + assertEquals(CONFIGURATION_DIGEST, sent["X-ADC-Configuration-SHA256"]) + assertTrue(sent.names().none { it.contains("Participant", ignoreCase = true) }) + assertTrue(sent.values("X-ADC-Participant-Instance").none { it == METADATA.participantInstanceId }) + } + + @Test + fun nonRetryableResponseIsPersistedAsTerminalWithoutAdvancingToAnotherRequest() = runBlocking { + val directory = temporaryFolder.newFolder("terminal-http") + val outbox = FileUploadOutbox(directory) {} + stage(outbox, byteArrayOf(1, 2, 3)) + var calls = 0 + val client = client { request -> + calls++ + response(request, 400, ByteArray(0), mediaType = null) + } + val uploader = uploader(outbox, client) + + val first = runCatching { uploader.upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) }.exceptionOrNull() + val recovered = runCatching { uploader.reconcile(VERIFIED, METADATA) }.exceptionOrNull() + + assertEquals("UPLOAD_HTTP_400", (first as StudyUploadException).reasonCode) + assertFalse(first.retryable) + assertEquals("UPLOAD_HTTP_400", (recovered as StudyUploadException).reasonCode) + assertEquals(1, calls) + } + + @Test + fun onlySpecifiedHttpFailuresAreRetryable() = runBlocking { + listOf(408, 425, 429, 500, 503).forEach { status -> + val outbox = FileUploadOutbox(temporaryFolder.newFolder("retry-$status")) {} + stage(outbox, byteArrayOf(status.toByte())) + val client = client { request -> response(request, status, ByteArray(0), mediaType = null) } + + val failure = runCatching { + uploader(outbox, client).upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) + }.exceptionOrNull() as StudyUploadException + + assertTrue("HTTP $status", failure.retryable) + } + } + + @Test + fun redirectsAcceptedButUndurableAndOversizedRequestsAreTerminal() = runBlocking { + listOf(202, 301, 400, 413).forEach { status -> + val outbox = FileUploadOutbox(temporaryFolder.newFolder("terminal-$status")) {} + stage(outbox, byteArrayOf(status.toByte())) + val client = client { request -> response(request, status, ByteArray(0), mediaType = null) } + + val failure = runCatching { + uploader(outbox, client).upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) + }.exceptionOrNull() as StudyUploadException + + assertEquals("UPLOAD_HTTP_$status", failure.reasonCode) + assertFalse("HTTP $status", failure.retryable) + } + } + + @Test + fun mismatchedReceiptPermanentlyStopsThatStage() = runBlocking { + val outbox = FileUploadOutbox(temporaryFolder.newFolder("receipt-mismatch")) {} + val expected = stage(outbox, byteArrayOf(1, 2, 3)) + val wrong = expected.copy(sha256 = "f".repeat(64)) + val client = client { request -> response(request, 201, UploadReceiptCodec.encode(wrong)) } + val uploader = uploader(outbox, client) + + val failure = runCatching { + uploader.upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) + }.exceptionOrNull() as StudyUploadException + + assertEquals("UPLOAD_RECEIPT_MISMATCH", failure.reasonCode) + assertFalse(failure.retryable) + } + + @Test + fun invalidReceiptMediaTypeAndSizePermanentlyStopTheirStages() = runBlocking { + listOf( + "text" to { request: okhttp3.Request -> + response(request, 201, "{}".toByteArray(), mediaType = "text/plain") + }, + "oversize" to { request: okhttp3.Request -> + response(request, 201, ByteArray(2_049), mediaType = "application/json") + }, + ).forEach { (name, responder) -> + val outbox = FileUploadOutbox(temporaryFolder.newFolder("invalid-receipt-$name")) {} + stage(outbox, byteArrayOf(1, 2, 3)) + val failure = runCatching { + uploader(outbox, client(responder)).upload(VERIFIED, METADATA, UNUSED_STORE, 1, 1) + }.exceptionOrNull() as StudyUploadException + + assertEquals("UPLOAD_RECEIPT_INVALID", failure.reasonCode) + assertFalse(failure.retryable) + } + } + + @Test + fun firstUploadStagesARealEncryptedBundleBeforeSendingIt() = runBlocking { + val privateKey = ProtocolBase64Url.decodeExact(RESEARCHER_PRIVATE_KEY, 32, "test private key") + val configuration = CONFIGURATION.copy( + export = ExportConfiguration("export-key", RESEARCHER_PUBLIC_KEY), + ) + val canonical = StudyConfigurationCodec.encode(configuration) + val digest = MessageDigest.getInstance("SHA-256").digest(canonical).hex() + val verified = VerifiedConfiguration( + configuration, + canonical, + configuration.signer.keyId, + ByteArray(64), + digest, + false, + ) + val metadata = StudyMetadata.initial( + configuration.experimentId, + configuration.configurationId, + configuration.assignedParticipantId, + PARTICIPANT_ID, + ).copy(eventCount = 1, nextSequenceNumber = 2) + val event = RecordedEvent( + sequenceNumber = 1, + collectorId = AppLifecycleConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = ResearchTime(1_000, 2_000, "boot-test"), + payloadType = "ACTIVITY_RESUMED", + fields = mapOf("activity_class" to "test.Activity"), + ) + val store = storeOf(event) + var sent: ByteArray? = null + val client = client { request -> + val bytes = Buffer().also { request.body!!.writeTo(it) }.readByteArray() + sent = bytes + val receipt = ExportReceipt( + bundleId = UUID.fromString(requireNotNull(request.header("X-ADC-Bundle-Id"))), + configurationSha256 = requireNotNull(request.header("X-ADC-Configuration-SHA256")), + firstSequence = requireNotNull(request.header("X-ADC-Sequence-From")).toLong(), + lastSequence = requireNotNull(request.header("X-ADC-Sequence-To")).toLong(), + eventCount = requireNotNull(request.header("X-ADC-Event-Count")).toLong(), + sha256 = MessageDigest.getInstance("SHA-256").digest(bytes).hex(), + byteCount = bytes.size.toLong(), + ) + response(request, 201, UploadReceiptCodec.encode(receipt)) + } + val outbox = FileUploadOutbox(temporaryFolder.newFolder("real-stage")) {} + + val receipt = uploader(outbox, client).upload(verified, metadata, store, 1, 1) + val encrypted = requireNotNull(sent) + val plaintext = ResearchExport.decrypt(encrypted, privateKey, configuration).toString(Charsets.UTF_8) + + assertEquals(encrypted.size.toLong(), receipt.byteCount) + assertTrue(plaintext.contains("\"bundle_kind\":\"automatic_upload\"")) + assertTrue(plaintext.contains("\"sequence_number\":\"1\"")) + assertTrue(plaintext.contains("test.Activity")) + } + + private suspend fun stage(outbox: FileUploadOutbox, bytes: ByteArray): ExportReceipt = + outbox.stage { output -> + output.write(bytes) + ExportReceipt( + bundleId = UUID.fromString("00000000-0000-4000-8000-000000000001"), + configurationSha256 = CONFIGURATION_DIGEST, + firstSequence = 1, + lastSequence = 1, + eventCount = 1, + sha256 = MessageDigest.getInstance("SHA-256").digest(bytes) + .joinToString("") { "%02x".format(it) }, + byteCount = bytes.size.toLong(), + ) + }.receipt + + private fun uploader(outbox: FileUploadOutbox, client: OkHttpClient) = OkHttpStudyUploader( + outbox = outbox, + producer = BundleProducer(StudyConfiguration.ANDROID_PLATFORM, "1"), + client = client, + nowUtcMillis = { 1 }, + ) + + private fun client(intercept: (okhttp3.Request) -> Response): OkHttpClient = + OkHttpClient.Builder().addInterceptor { chain -> intercept(chain.request()) }.build() + + private fun response( + request: okhttp3.Request, + status: Int, + body: ByteArray, + mediaType: String? = "application/json", + ) = Response.Builder() + .request(request) + .protocol(Protocol.HTTP_1_1) + .code(status) + .message("test") + .body(body.toResponseBody(mediaType?.toMediaType())) + .build() + + private fun storeOf(vararg events: RecordedEvent): StudyStore = object : StudyStore { + override suspend fun loadMetadata(): StudyMetadata? = error("unused") + override suspend fun initialize(metadata: StudyMetadata) = error("unused") + override suspend fun saveMetadata(metadata: StudyMetadata) = error("unused") + override suspend fun appendEvent(event: RecordedEvent) = error("unused") + override suspend fun appendEventAtomically(event: RecordedEvent, metadata: StudyMetadata) = error("unused") + override suspend fun readEvents( + fromSequenceInclusive: Long, + upToSequenceInclusive: Long, + consume: (RecordedEvent) -> Unit, + ) = events.filter { it.sequenceNumber in fromSequenceInclusive..upToSequenceInclusive }.forEach(consume) + override suspend fun storageUsage(): StorageUsage = error("unused") + override suspend fun evictThrough(metadata: StudyMetadata, targetBytes: Long): StudyMetadata = error("unused") + override suspend fun clear() = error("unused") + } + + private companion object { + const val EXPERIMENT_ID = "upload-test" + const val CONFIGURATION_ID = "upload-config" + const val PARTICIPANT_ID = "00000000-0000-0000-0000-000000000001" + const val RAW_PUBLIC_KEY = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + const val RESEARCHER_PRIVATE_KEY = "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVpbXF1eX2A" + const val RESEARCHER_PUBLIC_KEY = "ZLEBsdC-WocEvQePmJUAH8A-jp-VIvGI3RKNmEbUhGY" + const val CONFIGURATION_DIGEST = + "0000000000000000000000000000000000000000000000000000000000000000" + + val CONFIGURATION = StudyConfiguration( + schemaVersion = 1, + experimentId = EXPERIMENT_ID, + configurationId = CONFIGURATION_ID, + issuedAt = Instant.parse("2026-01-01T00:00:00Z"), + expiresAt = Instant.parse("2030-01-01T00:00:00Z"), + platform = StudyConfiguration.ANDROID_PLATFORM, + minimumClientVersion = 1, + title = "Upload test", + researcherName = "Researcher", + researcherContact = "research@example.invalid", + purpose = "Verify the replay-safe upload transaction.", + durationHours = 1, + consentDocumentVersion = "v1", + consentSummary = "Test consent.", + assignedParticipantId = "assigned-secret", + collectors = listOf(AppLifecycleConfiguration(true)), + surveys = emptyList(), + interventions = emptyList(), + maximumLocalBytes = 16_777_216, + signer = SignerIdentity("test-signer", RAW_PUBLIC_KEY), + export = ExportConfiguration("export-key", RAW_PUBLIC_KEY), + upload = UploadConfiguration("https://example.invalid/v1", 60, false), + ) + val VERIFIED = VerifiedConfiguration( + CONFIGURATION, + byteArrayOf(1), + CONFIGURATION.signer.keyId, + ByteArray(64), + CONFIGURATION_DIGEST, + false, + ) + val METADATA = StudyMetadata.initial( + EXPERIMENT_ID, + CONFIGURATION_ID, + CONFIGURATION.assignedParticipantId, + PARTICIPANT_ID, + ) + val UNUSED_STORE = object : StudyStore { + override suspend fun loadMetadata(): StudyMetadata? = error("unused") + override suspend fun initialize(metadata: StudyMetadata) = error("unused") + override suspend fun saveMetadata(metadata: StudyMetadata) = error("unused") + override suspend fun appendEvent(event: RecordedEvent) = error("unused") + override suspend fun appendEventAtomically(event: RecordedEvent, metadata: StudyMetadata) = error("unused") + override suspend fun readEvents( + fromSequenceInclusive: Long, + upToSequenceInclusive: Long, + consume: (RecordedEvent) -> Unit, + ) = error("unused") + override suspend fun storageUsage(): StorageUsage = error("unused") + override suspend fun evictThrough(metadata: StudyMetadata, targetBytes: Long): StudyMetadata = error("unused") + override suspend fun clear() = error("unused") + } + } +} + +private fun ByteArray.hex(): String = joinToString("") { "%02x".format(it) } diff --git a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/UploadIdentityTest.kt b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/UploadIdentityTest.kt index 2cb681a..8cd8a7f 100644 --- a/app/src/test/kotlin/cool/linc/androiddatacollector/platform/UploadIdentityTest.kt +++ b/app/src/test/kotlin/cool/linc/androiddatacollector/platform/UploadIdentityTest.kt @@ -5,8 +5,11 @@ import cool.linc.androiddatacollector.core.definition.ExportConfiguration import cool.linc.androiddatacollector.core.definition.SignerIdentity import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.definition.UploadConfiguration +import cool.linc.androiddatacollector.core.export.ExportReceipt import cool.linc.androiddatacollector.core.model.StudyMetadata +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import java.time.Instant +import java.util.UUID import org.junit.Assert.assertEquals import org.junit.Assert.assertNotEquals import org.junit.Assert.assertTrue @@ -14,15 +17,36 @@ import org.junit.Test class UploadIdentityTest { @Test - fun importsAlwaysMintDistinctInstancesAndHeadersNeverExposeAssignedCode() { + fun importsMintDistinctInstancesAndUploadHeadersExposeNoParticipantIdentity() { val first = StudyMetadata.initial("identity-test", "identity-config", "assigned-secret") val second = StudyMetadata.initial("identity-test", "identity-config", "assigned-secret") assertNotEquals(first.participantInstanceId, second.participantInstanceId) assertEquals("assigned-secret", first.assignedParticipantId) - val headers = uploadHeaders(configuration(), first, 1, 9) - assertEquals(first.participantInstanceId, headers["X-ADC-Participant-Instance"]) + val configuration = configuration() + val verified = VerifiedConfiguration( + configuration, + byteArrayOf(1), + configuration.signer.keyId, + ByteArray(64), + "0".repeat(64), + false, + ) + val headers = uploadHeaders( + verified, + ExportReceipt( + UUID.fromString("00000000-0000-4000-8000-000000000001"), + verified.configurationSha256, + 1, + 9, + 9, + "1".repeat(64), + 10, + ), + ) + assertTrue(headers.keys.none { it.contains("Participant", ignoreCase = true) }) assertTrue(headers.keys.none { it.contains("Assigned", ignoreCase = true) }) + assertTrue(headers.values.none { it == first.participantInstanceId }) assertTrue(headers.values.none { it == "assigned-secret" }) } @@ -32,7 +56,8 @@ class UploadIdentityTest { configurationId = "identity-config", issuedAt = Instant.parse("2026-01-01T00:00:00Z"), expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, + platform = StudyConfiguration.ANDROID_PLATFORM, + minimumClientVersion = 1, title = "Identity test", researcherName = "Researcher", researcherContact = "research@example.invalid", @@ -45,8 +70,12 @@ class UploadIdentityTest { surveys = emptyList(), interventions = emptyList(), maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", "x".repeat(32)), - export = ExportConfiguration("export-key", "x".repeat(32)), + signer = SignerIdentity("test-signer", RAW_PUBLIC_KEY), + export = ExportConfiguration("export-key", RAW_PUBLIC_KEY), upload = UploadConfiguration("https://example.invalid/v1", 60, false), ) + + private companion object { + const val RAW_PUBLIC_KEY = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + } } diff --git a/assurance/collector-policy.json b/assurance/collector-policy.json new file mode 100644 index 0000000..8665e2b --- /dev/null +++ b/assurance/collector-policy.json @@ -0,0 +1,157 @@ +{ + "allowed_catalog_dependencies": [ + "libs.coroutines.android", + "libs.coroutines.play.services", + "libs.play.services.location" + ], + "allowed_project_dependencies": [ + ":collector:sensor-common", + ":core:collector-api", + ":core:study-definition" + ], + "forbidden_class_prefixes": [ + "android/database/", + "android/provider/", + "android/webkit/", + "androidx/datastore/", + "androidx/preference/", + "androidx/room/", + "cool/linc/androiddatacollector/core/crypto/", + "cool/linc/androiddatacollector/core/export/", + "cool/linc/androiddatacollector/core/protocol/", + "cool/linc/androiddatacollector/core/storage/", + "dalvik/system/", + "java/io/", + "java/lang/reflect/", + "java/net/", + "java/nio/file/", + "java/security/", + "java/sql/", + "java/util/logging/", + "javax/crypto/", + "javax/net/", + "kotlin/io/path/", + "kotlin/io/FilesKt", + "kotlin/reflect/", + "okhttp3/", + "org/apache/http/", + "org/apache/logging/", + "org/slf4j/", + "retrofit2/", + "timber/log/" + ], + "forbidden_classes": [ + "android/app/DownloadManager", + "android/app/PendingIntent", + "android/content/ContentResolver", + "android/content/SharedPreferences", + "android/util/Log", + "android/util/EventLog", + "java/lang/ClassLoader", + "java/lang/ProcessBuilder", + "java/lang/Runtime", + "java/lang/reflect/AccessibleObject", + "java/util/ServiceLoader", + "kotlin/io/ConsoleKt" + ], + "forbidden_import_prefixes": [ + "android.database.", + "android.provider.", + "android.webkit.", + "androidx.datastore.", + "androidx.preference.", + "androidx.room.", + "cool.linc.androiddatacollector.core.crypto.", + "cool.linc.androiddatacollector.core.export.", + "cool.linc.androiddatacollector.core.protocol.", + "cool.linc.androiddatacollector.core.storage.", + "dalvik.system.", + "java.io.", + "java.lang.ClassLoader", + "java.lang.reflect.", + "java.net.", + "java.nio.file.", + "java.security.", + "java.sql.", + "java.util.logging.", + "java.util.ServiceLoader", + "javax.crypto.", + "javax.net.", + "kotlin.io.path.", + "kotlin.reflect.", + "okhttp3.", + "org.apache.http.", + "org.apache.logging.", + "org.slf4j.", + "retrofit2.", + "timber.log." + ], + "forbidden_methods": [ + { + "name": "forName", + "owner": "java/lang/Class" + }, + { + "name": "load", + "owner": "java/lang/System" + }, + { + "name": "loadLibrary", + "owner": "java/lang/System" + }, + { + "name": "sendBroadcast", + "owner": "android/content/Context" + }, + { + "name": "sendBroadcast", + "owner": "android/content/ContextWrapper" + }, + { + "name": "startActivities", + "owner": "android/content/Context" + }, + { + "name": "startActivity", + "owner": "android/content/Context" + }, + { + "name": "startActivity", + "owner": "android/content/ContextWrapper" + }, + { + "name": "startActivity", + "owner": "android/app/Activity" + }, + { + "name": "startForegroundService", + "owner": "android/content/Context" + }, + { + "name": "startActivities", + "owner": "androidx/core/content/ContextCompat" + }, + { + "name": "startActivity", + "owner": "androidx/core/content/ContextCompat" + }, + { + "name": "startService", + "owner": "android/content/Context" + } + ], + "forbidden_method_names": [ + "databaseList", + "deleteDatabase", + "deleteFile", + "deleteSharedPreferences", + "fileList", + "getSharedPreferences", + "moveDatabaseFrom", + "moveSharedPreferencesFrom", + "openFileInput", + "openFileOutput", + "openOrCreateDatabase", + "writeEvent" + ] +} diff --git a/collector/accelerometer/build.gradle.kts b/collector/accelerometer/build.gradle.kts index 5a0736e..1e6ed2c 100644 --- a/collector/accelerometer/build.gradle.kts +++ b/collector/accelerometer/build.gradle.kts @@ -20,7 +20,7 @@ kotlin { } dependencies { + implementation(project(":collector:sensor-common")) implementation(project(":core:collector-api")) implementation(project(":core:study-definition")) - implementation(libs.coroutines.android) } diff --git a/collector/accelerometer/src/main/kotlin/cool/linc/androiddatacollector/collector/accelerometer/AccelerometerCollector.kt b/collector/accelerometer/src/main/kotlin/cool/linc/androiddatacollector/collector/accelerometer/AccelerometerCollector.kt index 9ca697a..9075e38 100644 --- a/collector/accelerometer/src/main/kotlin/cool/linc/androiddatacollector/collector/accelerometer/AccelerometerCollector.kt +++ b/collector/accelerometer/src/main/kotlin/cool/linc/androiddatacollector/collector/accelerometer/AccelerometerCollector.kt @@ -3,21 +3,18 @@ package cool.linc.androiddatacollector.collector.accelerometer import android.content.Context import android.hardware.Sensor import android.hardware.SensorEvent -import android.hardware.SensorEventListener -import android.hardware.SensorManager -import android.os.Handler -import android.os.HandlerThread +import cool.linc.androiddatacollector.collector.sensorcommon.AndroidSensorCollector import cool.linc.androiddatacollector.core.model.EventDraft import cool.linc.androiddatacollector.core.definition.AccelerometerConfiguration import cool.linc.androiddatacollector.core.collector.AccessKind import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorPlugin -import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector class AccelerometerCollectorPlugin( context: Context, @@ -26,10 +23,9 @@ class AccelerometerCollectorPlugin( override val descriptor = CollectorDescriptor( id = AccelerometerConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Accelerometer", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 2_048, + eventContract = requireNotNull(ProtocolEventContracts[AccelerometerConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -51,64 +47,38 @@ class AccelerometerCollectorPlugin( private class AccelerometerCollector( androidContext: Context, - private val configuration: AccelerometerConfiguration, + configuration: AccelerometerConfiguration, collectorContext: CollectorContext, -) : SerializedCallbackCollector(collectorContext, CHANNEL_CAPACITY), - SensorEventListener { - private val sensorManager = androidContext.getSystemService(SensorManager::class.java) - private val sensor by lazy { - sensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER) - ?: throw IllegalStateException("Accelerometer hardware is unavailable") - } - private var handlerThread: HandlerThread? = null - - override fun onSensorChanged(event: SensorEvent) { - if (event.sensor.type != Sensor.TYPE_ACCELEROMETER || event.values.size < VECTOR_SIZE) return - capture { - EventDraft( - collectorId = AccelerometerConfiguration.ID, - payloadSchemaVersion = 1, - observedTime = context.clocks.now(), - payloadType = "ACCELEROMETER_SAMPLE", - fields = mapOf( - "source_elapsed_realtime_nanos" to event.timestamp.toString(), - "x_meters_per_second_squared" to event.values[0].toString(), - "y_meters_per_second_squared" to event.values[1].toString(), - "z_meters_per_second_squared" to event.values[2].toString(), - "accuracy" to event.accuracy.toString(), - ), - ) - } - } - - override fun onAccuracyChanged( - sensor: Sensor?, - accuracy: Int, - ) = Unit - - override suspend fun registerSource() { - val thread = HandlerThread("adc-accelerometer").also { it.start() } - try { - check( - sensorManager.registerListener( - this, - sensor, - configuration.samplingPeriodUs, - configuration.maximumReportLatencyUs, - Handler(thread.looper), - ), - ) { "Android rejected the accelerometer listener" } - handlerThread = thread - } catch (failure: Throwable) { - thread.quitSafely() - throw failure - } - } - - override suspend fun unregisterSource() { - sensorManager.unregisterListener(this, sensor) - handlerThread?.quitSafely() - handlerThread = null +) : AndroidSensorCollector( + androidContext = androidContext, + collectorContext = collectorContext, + sensorType = Sensor.TYPE_ACCELEROMETER, + samplingPeriodUs = configuration.samplingPeriodUs, + maximumReportLatencyUs = configuration.maximumReportLatencyUs, + threadName = "adc-accelerometer", + queueCapacity = CHANNEL_CAPACITY, +) { + override fun eventDraft(event: SensorEvent): EventDraft? { + if ( + event.timestamp < 0 || + event.values.size < VECTOR_SIZE || + !event.values[0].isFinite() || + !event.values[1].isFinite() || + !event.values[2].isFinite() + ) return null + return EventDraft( + collectorId = AccelerometerConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = context.clocks.now(), + payloadType = "ACCELEROMETER_SAMPLE", + fields = mapOf( + "source_elapsed_realtime_nanos" to event.timestamp.toString(), + "x_meters_per_second_squared" to event.values[0].toString(), + "y_meters_per_second_squared" to event.values[1].toString(), + "z_meters_per_second_squared" to event.values[2].toString(), + "accuracy" to event.accuracy.toString(), + ), + ) } private companion object { diff --git a/collector/ambient-light/build.gradle.kts b/collector/ambient-light/build.gradle.kts new file mode 100644 index 0000000..3007c80 --- /dev/null +++ b/collector/ambient-light/build.gradle.kts @@ -0,0 +1,27 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.ambientlight" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + implementation(project(":collector:sensor-common")) + implementation(project(":core:collector-api")) + implementation(project(":core:study-definition")) + testImplementation(libs.junit4) +} diff --git a/collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt b/collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt new file mode 100644 index 0000000..1b4e7ee --- /dev/null +++ b/collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt @@ -0,0 +1,165 @@ +package cool.linc.androiddatacollector.collector.ambientlight + +import android.content.Context +import android.hardware.Sensor +import android.hardware.SensorEvent +import android.os.SystemClock +import cool.linc.androiddatacollector.collector.sensorcommon.AndroidSensorCollector +import cool.linc.androiddatacollector.core.collector.AccessKind +import cool.linc.androiddatacollector.core.collector.AccessRequirement +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.AmbientLightConfiguration +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import kotlin.math.abs + +class AmbientLightCollectorPlugin(context: Context) : CollectorPlugin { + private val applicationContext = context.applicationContext + + override val descriptor = CollectorDescriptor( + id = AmbientLightConfiguration.ID, + displayName = "Ambient light", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[AmbientLightConfiguration.ID]), + ) + + override fun accessRequirements(configuration: CollectorConfiguration): Set { + val typed = configuration as? AmbientLightConfiguration + ?: throw IllegalArgumentException("Invalid ambient-light configuration") + return setOf(AccessRequirement(AccessKind.AMBIENT_LIGHT_HARDWARE, typed.required)) + } + + override fun create(configuration: CollectorConfiguration, context: CollectorContext): Collector { + val typed = configuration as? AmbientLightConfiguration + ?: throw IllegalArgumentException("Invalid ambient-light configuration") + return AmbientLightCollector(applicationContext, typed, context) + } +} + +private class AmbientLightCollector( + androidContext: Context, + private val configuration: AmbientLightConfiguration, + collectorContext: CollectorContext, +) : AndroidSensorCollector( + androidContext = androidContext, + collectorContext = collectorContext, + sensorType = Sensor.TYPE_LIGHT, + samplingPeriodUs = configuration.samplingPeriodUs, + maximumReportLatencyUs = 0, + threadName = "adc-ambient-light", + queueCapacity = 256, +) { + private val rateGate = LatestValueRateGate( + (configuration.samplingPeriodUs + 999L) / 1_000L, + ) { previous, current -> + sameAmbientLightSample(previous, current, configuration.changeThresholdMillilux) + } + private var pendingScheduled = false + private val pendingRunnable = Runnable { sourceCallback(::publishPending) } + + override suspend fun onSourceRegistering() { + val latest = context.eventSink.latestEvent(AmbientLightConfiguration.ID) ?: return + rateGate.restoreLastEmission( + value = latest.ambientLightSampleOrNull(), + currentElapsedMillis = SystemClock.elapsedRealtime(), + ) + } + + override fun onSensorEvent(event: SensorEvent) { + val sample = ambientLightSample( + lux = event.values.firstOrNull() ?: return, + sourceTimestampNanos = event.timestamp, + accuracy = event.accuracy, + observedTime = context.clocks.now(), + ) ?: return + handle(rateGate.offer(sample, SystemClock.elapsedRealtime())) + } + + override fun onSourceUnregistering() { + sourceHandler?.removeCallbacks(pendingRunnable) + rateGate.clearPending() + pendingScheduled = false + } + + private fun publishPending() { + pendingScheduled = false + handle(rateGate.poll(SystemClock.elapsedRealtime())) + } + + private fun handle(decision: LatestValueRateGate.Decision) { + when (decision) { + is LatestValueRateGate.Decision.Emit -> { + sourceHandler?.removeCallbacks(pendingRunnable) + pendingScheduled = false + emit(decision.value.eventDraft()) + } + is LatestValueRateGate.Decision.Defer -> if (!pendingScheduled) { + pendingScheduled = true + sourceHandler?.postDelayed(pendingRunnable, decision.delayMillis) + } + LatestValueRateGate.Decision.Suppress -> { + sourceHandler?.removeCallbacks(pendingRunnable) + pendingScheduled = false + } + } + } +} + +internal data class AmbientLightSample( + val observedTime: ResearchTime, + val sourceTimestampNanos: Long, + val illuminanceLux: Float, + val accuracy: Int, +) + +internal fun ambientLightSample( + lux: Float, + sourceTimestampNanos: Long, + accuracy: Int, + observedTime: ResearchTime, +): AmbientLightSample? = if (!lux.isFinite() || lux < 0 || sourceTimestampNanos < 0) { + null +} else { + AmbientLightSample(observedTime, sourceTimestampNanos, lux, accuracy) +} + +internal fun sameAmbientLightSample( + previous: AmbientLightSample, + current: AmbientLightSample, + changeThresholdMillilux: Int, +): Boolean = previous.illuminanceLux == current.illuminanceLux || + abs(previous.illuminanceLux - current.illuminanceLux) * 1_000 < changeThresholdMillilux + +internal fun AmbientLightSample.eventDraft() = EventDraft( + collectorId = AmbientLightConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = observedTime, + payloadType = "AMBIENT_LIGHT_SAMPLE", + fields = mapOf( + "source_elapsed_realtime_nanos" to sourceTimestampNanos.toString(), + "illuminance_lux" to illuminanceLux.toString(), + "accuracy" to accuracy.toString(), + ), +) + +internal fun RecordedEvent.ambientLightSampleOrNull(): AmbientLightSample? { + if ( + collectorId != AmbientLightConfiguration.ID || + payloadSchemaVersion != 1 || + payloadType != "AMBIENT_LIGHT_SAMPLE" + ) return null + return ambientLightSample( + lux = fields["illuminance_lux"]?.toFloatOrNull() ?: return null, + sourceTimestampNanos = fields["source_elapsed_realtime_nanos"]?.toLongOrNull() ?: return null, + accuracy = fields["accuracy"]?.toIntOrNull() ?: return null, + observedTime = observedTime, + ) +} diff --git a/collector/ambient-light/src/test/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollectorTest.kt b/collector/ambient-light/src/test/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollectorTest.kt new file mode 100644 index 0000000..0801e99 --- /dev/null +++ b/collector/ambient-light/src/test/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollectorTest.kt @@ -0,0 +1,83 @@ +package cool.linc.androiddatacollector.collector.ambientlight + +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.AmbientLightConfiguration +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +class AmbientLightCollectorTest { + @Test + fun changedOnChangeReadingInsideIntervalIsDeferredWithItsCaptureTime() { + val gate = LatestValueRateGate(200) { previous, current -> + sameAmbientLightSample(previous, current, 1_000) + } + val first = requireNotNull(ambientLightSample(10f, 1_000_000_000, 3, time(10))) + val changed = requireNotNull(ambientLightSample(20f, 1_100_000_000, 3, time(20))) + + assertTrue(gate.offer(first, 1_000) is LatestValueRateGate.Decision.Emit) + val deferred = gate.offer(changed, 1_100) + assertTrue(deferred is LatestValueRateGate.Decision.Defer) + val emitted = gate.poll(1_200) + + require(emitted is LatestValueRateGate.Decision.Emit) + val event = emitted.value.eventDraft() + assertEquals(time(20), event.observedTime) + assertEquals("20.0", event.fields["illuminance_lux"]) + assertEquals("1100000000", event.fields["source_elapsed_realtime_nanos"]) + assertEquals(AmbientLightConfiguration.ID, event.collectorId) + assertTrue(requireNotNull(ProtocolEventContracts[AmbientLightConfiguration.ID]).accepts(event, 1)) + } + + @Test + fun thresholdUsesIlluminanceAndAccuracyRidesOnEmittedSamples() { + val previous = requireNotNull(ambientLightSample(10f, 1, 3, time(1))) + val belowThreshold = requireNotNull(ambientLightSample(10.5f, 2, 3, time(2))) + val changedAccuracy = belowThreshold.copy(accuracy = 2) + + assertTrue(sameAmbientLightSample(previous, belowThreshold, 1_000)) + assertTrue(sameAmbientLightSample(previous, changedAccuracy, 1_000)) + } + + @Test + fun rejectsInvalidPhysicalSamples() { + assertNull(ambientLightSample(Float.NaN, 1, 0, time(1))) + assertNull(ambientLightSample(-1f, 1, 0, time(1))) + assertNull(ambientLightSample(1f, -1, 0, time(1))) + } + + @Test + fun processRestartRestoresAmbientValueAndSamplingDeadline() { + val previous = requireNotNull( + ambientLightSample(10f, 9_000_000_000, 3, ResearchTime(10_000, 10_000_000_000, "boot-a")), + ) + val draft = previous.eventDraft() + val recorded = RecordedEvent( + 1, + draft.collectorId, + draft.payloadSchemaVersion, + draft.observedTime, + draft.payloadType, + draft.fields, + ) + val gate = LatestValueRateGate(200) { old, current -> + sameAmbientLightSample(old, current, 1_000) + } + gate.restoreLastEmission( + recorded.ambientLightSampleOrNull(), + 10_100, + ) + + assertEquals(LatestValueRateGate.Decision.Suppress, gate.offer(previous, 10_100)) + assertEquals( + LatestValueRateGate.Decision.Defer(200), + gate.offer(previous.copy(illuminanceLux = 20f), 10_100), + ) + } + + private fun time(value: Long) = ResearchTime(value, value, "boot-a") +} diff --git a/collector/app-lifecycle/src/main/kotlin/cool/linc/androiddatacollector/collector/applifecycle/AppLifecycleCollector.kt b/collector/app-lifecycle/src/main/kotlin/cool/linc/androiddatacollector/collector/applifecycle/AppLifecycleCollector.kt index 3518a5e..3156683 100644 --- a/collector/app-lifecycle/src/main/kotlin/cool/linc/androiddatacollector/collector/applifecycle/AppLifecycleCollector.kt +++ b/collector/app-lifecycle/src/main/kotlin/cool/linc/androiddatacollector/collector/applifecycle/AppLifecycleCollector.kt @@ -10,9 +10,12 @@ import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.withContext @@ -21,10 +24,9 @@ class AppLifecycleCollectorPlugin( ) : CollectorPlugin { override val descriptor = CollectorDescriptor( id = COLLECTOR_ID, - payloadSchemaVersion = PAYLOAD_SCHEMA_VERSION, displayName = "Own-app lifecycle", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 2_048, + eventContract = requireNotNull(ProtocolEventContracts[COLLECTOR_ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -86,12 +88,14 @@ private class AppLifecycleCollector( } } - override suspend fun registerSource() = withContext(Dispatchers.Main.immediate) { + override suspend fun registerSource(): SourceRegistrationResult = withContext(Dispatchers.Main.immediate) { application.registerActivityLifecycleCallbacks(this@AppLifecycleCollector) + SourceRegistrationResult.Registered } - override suspend fun unregisterSource() = withContext(Dispatchers.Main.immediate) { + override suspend fun unregisterSource(): SourceTeardownResult = withContext(Dispatchers.Main.immediate) { application.unregisterActivityLifecycleCallbacks(this@AppLifecycleCollector) + SourceTeardownResult.Released } private companion object { diff --git a/collector/battery-state/build.gradle.kts b/collector/battery-state/build.gradle.kts new file mode 100644 index 0000000..b7515c0 --- /dev/null +++ b/collector/battery-state/build.gradle.kts @@ -0,0 +1,27 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.batterystate" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + implementation(project(":core:collector-api")) + implementation(project(":core:study-definition")) + implementation(libs.coroutines.android) + testImplementation(libs.junit4) +} diff --git a/collector/battery-state/src/main/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollector.kt b/collector/battery-state/src/main/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollector.kt new file mode 100644 index 0000000..ad5621b --- /dev/null +++ b/collector/battery-state/src/main/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollector.kt @@ -0,0 +1,235 @@ +package cool.linc.androiddatacollector.collector.batterystate + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.content.IntentFilter +import android.os.BatteryManager +import android.os.Handler +import android.os.Looper +import android.os.PowerManager +import android.os.SystemClock +import cool.linc.androiddatacollector.core.collector.AccessRequirement +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback +import cool.linc.androiddatacollector.core.definition.BatteryStateConfiguration +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext + +class BatteryStateCollectorPlugin(context: Context) : CollectorPlugin { + private val applicationContext = context.applicationContext + + override val descriptor = CollectorDescriptor( + id = BatteryStateConfiguration.ID, + displayName = "Battery state", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[BatteryStateConfiguration.ID]), + ) + + override fun accessRequirements(configuration: CollectorConfiguration): Set { + require(configuration is BatteryStateConfiguration) { "Invalid battery-state configuration" } + return emptySet() + } + + override fun create(configuration: CollectorConfiguration, context: CollectorContext): Collector { + require(configuration is BatteryStateConfiguration) { "Invalid battery-state configuration" } + return BatteryStateCollector(applicationContext, context) + } +} + +private class BatteryStateCollector( + private val applicationContext: Context, + collectorContext: CollectorContext, +) : SerializedCallbackCollector(collectorContext, 64) { + private val mainHandler = Handler(Looper.getMainLooper()) + private val powerManager = applicationContext.getSystemService(PowerManager::class.java) + private val rateGate = LatestValueRateGate( + MINIMUM_INTERVAL_MILLIS, + ::sameBatteryState, + ) + private val pendingRunnable = Runnable(::publishPending) + private var pendingScheduled = false + private val receiver = object : BroadcastReceiver() { + override fun onReceive(context: Context?, intent: Intent?) { + val battery = if (intent?.action == Intent.ACTION_BATTERY_CHANGED) intent else batteryIntent() + battery?.let(::snapshot)?.let(::offer) + } + } + + override suspend fun registerSource(): SourceRegistrationResult = withContext(Dispatchers.Main.immediate) { + restoreRateGate() + val filter = IntentFilter().apply { + addAction(Intent.ACTION_BATTERY_CHANGED) + addAction(PowerManager.ACTION_POWER_SAVE_MODE_CHANGED) + } + var receiverRegistered = false + registerSourceWithRollback( + register = { + val sticky = applicationContext.registerReceiver(receiver, filter, Context.RECEIVER_NOT_EXPORTED) + receiverRegistered = true + sticky?.let(::snapshot)?.let(::offer) + }, + rollback = { + completeSourceTeardown( + { if (receiverRegistered) applicationContext.unregisterReceiver(receiver) }, + ::clearPending, + ) + }, + ) + } + + private suspend fun restoreRateGate() { + val latest = context.eventSink.latestEvent(BatteryStateConfiguration.ID) ?: return + rateGate.restoreLastEmission( + value = latest.batterySnapshotOrNull(), + currentElapsedMillis = SystemClock.elapsedRealtime(), + ) + } + + override suspend fun unregisterSource(): SourceTeardownResult = withContext(Dispatchers.Main.immediate) { + completeSourceTeardown( + { applicationContext.unregisterReceiver(receiver) }, + ::clearPending, + ) + SourceTeardownResult.Released + } + + private fun clearPending() { + mainHandler.removeCallbacks(pendingRunnable) + rateGate.clearPending() + pendingScheduled = false + } + + private fun batteryIntent(): Intent? = applicationContext.registerReceiver( + null, + IntentFilter(Intent.ACTION_BATTERY_CHANGED), + ) + + private fun snapshot(intent: Intent): BatterySnapshot? { + val level = intent.getIntExtra(BatteryManager.EXTRA_LEVEL, -1) + val scale = intent.getIntExtra(BatteryManager.EXTRA_SCALE, -1) + val percentage = batteryPercentage(level, scale) ?: return null + return BatterySnapshot( + observedTime = context.clocks.now(), + percentage = percentage, + chargingState = chargingState(intent.getIntExtra(BatteryManager.EXTRA_STATUS, -1)), + chargingSource = chargingSource(intent.getIntExtra(BatteryManager.EXTRA_PLUGGED, 0)), + powerSaveEnabled = powerManager.isPowerSaveMode, + ) + } + + private fun offer(snapshot: BatterySnapshot) { + handle(rateGate.offer(snapshot, SystemClock.elapsedRealtime())) + } + + private fun publishPending() { + pendingScheduled = false + handle(rateGate.poll(SystemClock.elapsedRealtime())) + } + + private fun handle(decision: LatestValueRateGate.Decision) { + when (decision) { + is LatestValueRateGate.Decision.Emit -> { + mainHandler.removeCallbacks(pendingRunnable) + pendingScheduled = false + publish(decision.value) + } + is LatestValueRateGate.Decision.Defer -> if (!pendingScheduled) { + pendingScheduled = true + mainHandler.postDelayed(pendingRunnable, decision.delayMillis) + } + LatestValueRateGate.Decision.Suppress -> { + mainHandler.removeCallbacks(pendingRunnable) + pendingScheduled = false + } + } + } + + private fun publish(snapshot: BatterySnapshot) { + capture(snapshot::eventDraft) + } + + private companion object { const val MINIMUM_INTERVAL_MILLIS = 60_000L } +} + +internal data class BatterySnapshot( + val observedTime: ResearchTime, + val percentage: Int, + val chargingState: String, + val chargingSource: String, + val powerSaveEnabled: Boolean, +) + +internal fun sameBatteryState(previous: BatterySnapshot, current: BatterySnapshot): Boolean = + previous.percentage == current.percentage && + previous.chargingState == current.chargingState && + previous.chargingSource == current.chargingSource && + previous.powerSaveEnabled == current.powerSaveEnabled + +internal fun batteryPercentage(level: Int, scale: Int): Int? = + if (level < 0 || scale <= 0) null else ((level.toLong() * 100) / scale).toInt().coerceIn(0, 100) + +internal fun chargingState(status: Int): String = when (status) { + BatteryManager.BATTERY_STATUS_CHARGING -> "CHARGING" + BatteryManager.BATTERY_STATUS_DISCHARGING -> "DISCHARGING" + BatteryManager.BATTERY_STATUS_FULL -> "FULL" + BatteryManager.BATTERY_STATUS_NOT_CHARGING -> "NOT_CHARGING" + else -> "UNKNOWN" +} + +internal fun chargingSource(bits: Int): String { + val sources = buildList { + if (bits and BatteryManager.BATTERY_PLUGGED_AC != 0) add("AC") + if (bits and BatteryManager.BATTERY_PLUGGED_USB != 0) add("USB") + if (bits and BatteryManager.BATTERY_PLUGGED_WIRELESS != 0) add("WIRELESS") + if (bits and BatteryManager.BATTERY_PLUGGED_DOCK != 0) add("DOCK") + } + return when (sources.size) { + 0 -> if (bits == 0) "NONE" else "UNKNOWN" + 1 -> sources.single() + else -> "MULTIPLE" + } +} + +internal fun BatterySnapshot.eventDraft() = EventDraft( + collectorId = BatteryStateConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = observedTime, + payloadType = "BATTERY_STATE", + fields = mapOf( + "percentage" to percentage.toString(), + "charging_state" to chargingState, + "charging_source" to chargingSource, + "power_save_enabled" to powerSaveEnabled.toString(), + ), +) + +internal fun RecordedEvent.batterySnapshotOrNull(): BatterySnapshot? { + if ( + collectorId != BatteryStateConfiguration.ID || + payloadSchemaVersion != 1 || + payloadType != "BATTERY_STATE" + ) return null + val percentage = fields["percentage"]?.toIntOrNull()?.takeIf { it in 0..100 } ?: return null + val chargingState = fields["charging_state"]?.takeIf(BATTERY_CHARGING_STATES::contains) ?: return null + val chargingSource = fields["charging_source"]?.takeIf(BATTERY_CHARGING_SOURCES::contains) ?: return null + val powerSaveEnabled = fields["power_save_enabled"]?.toBooleanStrictOrNull() ?: return null + return BatterySnapshot(observedTime, percentage, chargingState, chargingSource, powerSaveEnabled) +} + +private val BATTERY_CHARGING_STATES = setOf("CHARGING", "DISCHARGING", "FULL", "NOT_CHARGING", "UNKNOWN") +private val BATTERY_CHARGING_SOURCES = setOf("AC", "DOCK", "MULTIPLE", "NONE", "UNKNOWN", "USB", "WIRELESS") diff --git a/collector/battery-state/src/test/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollectorTest.kt b/collector/battery-state/src/test/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollectorTest.kt new file mode 100644 index 0000000..7253215 --- /dev/null +++ b/collector/battery-state/src/test/kotlin/cool/linc/androiddatacollector/collector/batterystate/BatteryStateCollectorTest.kt @@ -0,0 +1,79 @@ +package cool.linc.androiddatacollector.collector.batterystate + +import android.os.BatteryManager +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.BatteryStateConfiguration +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +class BatteryStateCollectorTest { + @Test + fun mapsBoundedBatteryStateWithoutHardwareIdentity() { + assertEquals(50, batteryPercentage(1, 2)) + assertNull(batteryPercentage(-1, 100)) + assertNull(batteryPercentage(1, 0)) + assertEquals("CHARGING", chargingState(BatteryManager.BATTERY_STATUS_CHARGING)) + assertEquals("UNKNOWN", chargingState(-1)) + assertEquals("NONE", chargingSource(0)) + assertEquals("USB", chargingSource(BatteryManager.BATTERY_PLUGGED_USB)) + assertEquals( + "MULTIPLE", + chargingSource(BatteryManager.BATTERY_PLUGGED_AC or BatteryManager.BATTERY_PLUGGED_USB), + ) + } + + @Test + fun deferredDraftKeepsCaptureTimeWhileDeduplicationIgnoresOnlyThatTime() { + val captured = snapshot(ResearchTime(10, 20, "boot-a")) + val later = snapshot(ResearchTime(30, 40, "boot-a")) + val event = captured.eventDraft() + + assertTrue(sameBatteryState(captured, later)) + assertEquals(captured.observedTime, event.observedTime) + assertEquals(BatteryStateConfiguration.ID, event.collectorId) + assertTrue(requireNotNull(ProtocolEventContracts[BatteryStateConfiguration.ID]).accepts(event, 1)) + assertFalse(sameBatteryState(captured, later.copy(percentage = 49))) + } + + @Test + fun processRestartRestoresTheLastBatteryValueAndRateWatermark() { + val previous = snapshot(ResearchTime(10_000, 10_000_000_000, "boot-a")) + val draft = previous.eventDraft() + val recorded = RecordedEvent( + sequenceNumber = 1, + collectorId = draft.collectorId, + payloadSchemaVersion = draft.payloadSchemaVersion, + observedTime = draft.observedTime, + payloadType = draft.payloadType, + fields = draft.fields, + ) + val gate = LatestValueRateGate(60_000L, ::sameBatteryState) + gate.restoreLastEmission( + value = recorded.batterySnapshotOrNull(), + currentElapsedMillis = 10_100, + ) + + assertEquals( + LatestValueRateGate.Decision.Suppress, + gate.offer(previous.copy(observedTime = ResearchTime(10_100, 10_100_000_000, "boot-a")), 10_100), + ) + assertEquals( + LatestValueRateGate.Decision.Defer(60_000), + gate.offer(previous.copy(percentage = 49), 10_100), + ) + } + + private fun snapshot(time: ResearchTime) = BatterySnapshot( + observedTime = time, + percentage = 50, + chargingState = "DISCHARGING", + chargingSource = "NONE", + powerSaveEnabled = false, + ) +} diff --git a/collector/gyroscope/build.gradle.kts b/collector/gyroscope/build.gradle.kts new file mode 100644 index 0000000..c99b1b4 --- /dev/null +++ b/collector/gyroscope/build.gradle.kts @@ -0,0 +1,27 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.gyroscope" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + implementation(project(":collector:sensor-common")) + implementation(project(":core:collector-api")) + implementation(project(":core:study-definition")) + testImplementation(libs.junit4) +} diff --git a/collector/gyroscope/src/main/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollector.kt b/collector/gyroscope/src/main/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollector.kt new file mode 100644 index 0000000..4017d35 --- /dev/null +++ b/collector/gyroscope/src/main/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollector.kt @@ -0,0 +1,90 @@ +package cool.linc.androiddatacollector.collector.gyroscope + +import android.content.Context +import android.hardware.Sensor +import android.hardware.SensorEvent +import cool.linc.androiddatacollector.collector.sensorcommon.AndroidSensorCollector +import cool.linc.androiddatacollector.core.collector.AccessKind +import cool.linc.androiddatacollector.core.collector.AccessRequirement +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.definition.GyroscopeConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.ResearchTime + +class GyroscopeCollectorPlugin(context: Context) : CollectorPlugin { + private val applicationContext = context.applicationContext + + override val descriptor = CollectorDescriptor( + id = GyroscopeConfiguration.ID, + displayName = "Gyroscope", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[GyroscopeConfiguration.ID]), + ) + + override fun accessRequirements(configuration: CollectorConfiguration): Set { + val typed = configuration as? GyroscopeConfiguration + ?: throw IllegalArgumentException("Invalid gyroscope configuration") + return setOf(AccessRequirement(AccessKind.GYROSCOPE_HARDWARE, typed.required)) + } + + override fun create(configuration: CollectorConfiguration, context: CollectorContext): Collector { + val typed = configuration as? GyroscopeConfiguration + ?: throw IllegalArgumentException("Invalid gyroscope configuration") + return GyroscopeCollector(applicationContext, typed, context) + } +} + +private class GyroscopeCollector( + androidContext: Context, + configuration: GyroscopeConfiguration, + collectorContext: CollectorContext, +) : AndroidSensorCollector( + androidContext = androidContext, + collectorContext = collectorContext, + sensorType = Sensor.TYPE_GYROSCOPE, + samplingPeriodUs = configuration.samplingPeriodUs, + maximumReportLatencyUs = configuration.maximumReportLatencyUs, + threadName = "adc-gyroscope", + queueCapacity = 2_048, +) { + override fun eventDraft(event: SensorEvent): EventDraft? = gyroscopeEvent( + values = event.values, + timestampNanos = event.timestamp, + accuracy = event.accuracy, + observedTime = context.clocks.now(), + ) +} + +internal fun gyroscopeEvent( + values: FloatArray, + timestampNanos: Long, + accuracy: Int, + observedTime: ResearchTime, +): EventDraft? { + if ( + timestampNanos < 0 || + values.size < 3 || + !values[0].isFinite() || + !values[1].isFinite() || + !values[2].isFinite() + ) return null + return EventDraft( + collectorId = GyroscopeConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = observedTime, + payloadType = "GYROSCOPE_SAMPLE", + fields = mapOf( + "source_elapsed_realtime_nanos" to timestampNanos.toString(), + "x_radians_per_second" to values[0].toString(), + "y_radians_per_second" to values[1].toString(), + "z_radians_per_second" to values[2].toString(), + "accuracy" to accuracy.toString(), + ) + ) +} diff --git a/collector/gyroscope/src/test/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollectorTest.kt b/collector/gyroscope/src/test/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollectorTest.kt new file mode 100644 index 0000000..deceddf --- /dev/null +++ b/collector/gyroscope/src/test/kotlin/cool/linc/androiddatacollector/collector/gyroscope/GyroscopeCollectorTest.kt @@ -0,0 +1,36 @@ +package cool.linc.androiddatacollector.collector.gyroscope + +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.GyroscopeConfiguration +import cool.linc.androiddatacollector.core.model.ResearchTime +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +class GyroscopeCollectorTest { + @Test + fun emitsRawAxesWithHardwareAndCaptureTimes() { + val observed = ResearchTime(10, 20, "boot-a") + val event = gyroscopeEvent(floatArrayOf(1.25f, -2.5f, 0.0f), 123, 3, observed) + + requireNotNull(event) + assertEquals(observed, event.observedTime) + assertEquals("123", event.fields["source_elapsed_realtime_nanos"]) + assertEquals("1.25", event.fields["x_radians_per_second"]) + assertEquals("-2.5", event.fields["y_radians_per_second"]) + assertEquals("0.0", event.fields["z_radians_per_second"]) + assertEquals("3", event.fields["accuracy"]) + assertEquals(GyroscopeConfiguration.ID, event.collectorId) + assertTrue(requireNotNull(ProtocolEventContracts[GyroscopeConfiguration.ID]).accepts(event, 1)) + } + + @Test + fun rejectsIncompleteNonFiniteOrNegativeTimestampSamples() { + val observed = ResearchTime(10, 20, "boot-a") + + assertNull(gyroscopeEvent(floatArrayOf(1f, 2f), 1, 0, observed)) + assertNull(gyroscopeEvent(floatArrayOf(1f, Float.NaN, 3f), 1, 0, observed)) + assertNull(gyroscopeEvent(floatArrayOf(1f, 2f, 3f), -1, 0, observed)) + } +} diff --git a/collector/keyboard-ime/build.gradle.kts b/collector/keyboard-ime/build.gradle.kts index 7ba27f5..4834eec 100644 --- a/collector/keyboard-ime/build.gradle.kts +++ b/collector/keyboard-ime/build.gradle.kts @@ -23,4 +23,5 @@ dependencies { implementation(project(":core:collector-api")) implementation(project(":core:study-definition")) implementation(libs.coroutines.android) + testImplementation(libs.junit4) } diff --git a/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridge.kt b/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridge.kt index 1a59b8d..2c5d973 100644 --- a/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridge.kt +++ b/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridge.kt @@ -1,6 +1,7 @@ package cool.linc.androiddatacollector.collector.keyboardime import android.view.MotionEvent +import cool.linc.androiddatacollector.core.collector.SourceCallbackBoundary internal data class ImeTouchObservation( val action: String, @@ -17,7 +18,7 @@ internal data class ImeTouchObservation( ) internal object ImeObservationBridge { - private val lock = Any() + private val callbackBoundary = SourceCallbackBoundary() private var listener: ((ImeTouchObservation) -> Unit)? = null private var minimumMoveIntervalMillis = Long.MAX_VALUE private var lastMoveUptimeMillis = Long.MIN_VALUE @@ -25,14 +26,14 @@ internal object ImeObservationBridge { fun install( samplingHz: Int, listener: (ImeTouchObservation) -> Unit, - ) = synchronized(lock) { + ) = callbackBoundary.activate { check(this.listener == null) { "IME observation listener is already installed" } minimumMoveIntervalMillis = (1_000L / samplingHz).coerceAtLeast(1L) lastMoveUptimeMillis = Long.MIN_VALUE this.listener = listener } - fun uninstall() = synchronized(lock) { + fun uninstall() = callbackBoundary.deactivate { listener = null minimumMoveIntervalMillis = Long.MAX_VALUE lastMoveUptimeMillis = Long.MIN_VALUE @@ -46,15 +47,7 @@ internal object ImeObservationBridge { keyCategory: String, ) { val action = event.actionMasked.toActionName() ?: return - val destination = synchronized(lock) { - val current = listener ?: return - if (event.actionMasked == MotionEvent.ACTION_MOVE) { - if (event.eventTime - lastMoveUptimeMillis < minimumMoveIntervalMillis) return - lastMoveUptimeMillis = event.eventTime - } - current - } - destination( + publishObservation( ImeTouchObservation( action = action, eventUptimeMillis = event.eventTime, @@ -71,6 +64,22 @@ internal object ImeObservationBridge { ) } + internal fun publishObservation(observation: ImeTouchObservation) { + callbackBoundary.runIfActive { + val destination = checkNotNull(listener) { "Active IME callback has no listener" } + if (observation.action == "MOVE") { + if ( + lastMoveUptimeMillis != Long.MIN_VALUE && + observation.eventUptimeMillis - lastMoveUptimeMillis < minimumMoveIntervalMillis + ) { + return@runIfActive + } + lastMoveUptimeMillis = observation.eventUptimeMillis + } + destination(observation) + } + } + private fun Int.toActionName(): String? = when (this) { MotionEvent.ACTION_DOWN -> "DOWN" MotionEvent.ACTION_MOVE -> "MOVE" diff --git a/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/KeyboardTouchCollector.kt b/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/KeyboardTouchCollector.kt index 4eb555e..1ca4890 100644 --- a/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/KeyboardTouchCollector.kt +++ b/collector/keyboard-ime/src/main/kotlin/cool/linc/androiddatacollector/collector/keyboardime/KeyboardTouchCollector.kt @@ -6,19 +6,21 @@ import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.KeyboardTouchConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult class KeyboardTouchCollectorPlugin : CollectorPlugin { override val descriptor = CollectorDescriptor( id = KeyboardTouchConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Research keyboard touch", privacyClass = PrivacyClass.RESTRICTED, - maximumEncodedEventBytes = 4_096, + eventContract = requireNotNull(ProtocolEventContracts[KeyboardTouchConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -44,12 +46,14 @@ private class KeyboardTouchCollector( private val configuration: KeyboardTouchConfiguration, collectorContext: CollectorContext, ) : SerializedCallbackCollector(collectorContext, CHANNEL_CAPACITY) { - override suspend fun registerSource() { + override suspend fun registerSource(): SourceRegistrationResult { ImeObservationBridge.install(configuration.trajectorySamplingHz, ::capture) + return SourceRegistrationResult.Registered } - override suspend fun unregisterSource() { + override suspend fun unregisterSource(): SourceTeardownResult { ImeObservationBridge.uninstall() + return SourceTeardownResult.Released } private fun capture(observation: ImeTouchObservation) { diff --git a/collector/keyboard-ime/src/test/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridgeTest.kt b/collector/keyboard-ime/src/test/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridgeTest.kt new file mode 100644 index 0000000..a92752f --- /dev/null +++ b/collector/keyboard-ime/src/test/kotlin/cool/linc/androiddatacollector/collector/keyboardime/ImeObservationBridgeTest.kt @@ -0,0 +1,76 @@ +package cool.linc.androiddatacollector.collector.keyboardime + +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +class ImeObservationBridgeTest { + @Test + fun uninstallWaitsForInFlightDeliveryAndRejectsLaterObservations() { + val entered = CountDownLatch(1) + val release = CountDownLatch(1) + val uninstallStarted = CountDownLatch(1) + val uninstallFinished = CountDownLatch(1) + var deliveries = 0 + ImeObservationBridge.install(samplingHz = 10) { + entered.countDown() + check(release.await(1, TimeUnit.SECONDS)) + deliveries++ + } + val publisher = Thread { ImeObservationBridge.publishObservation(observation("DOWN", 100)) } + val uninstaller = Thread { + uninstallStarted.countDown() + ImeObservationBridge.uninstall() + uninstallFinished.countDown() + } + + try { + publisher.start() + assertTrue(entered.await(1, TimeUnit.SECONDS)) + uninstaller.start() + assertTrue(uninstallStarted.await(1, TimeUnit.SECONDS)) + assertFalse(uninstallFinished.await(50, TimeUnit.MILLISECONDS)) + release.countDown() + assertTrue(uninstallFinished.await(1, TimeUnit.SECONDS)) + ImeObservationBridge.publishObservation(observation("DOWN", 200)) + assertEquals(1, deliveries) + } finally { + release.countDown() + publisher.join(1_000) + uninstaller.join(1_000) + ImeObservationBridge.uninstall() + } + } + + @Test + fun firstMoveIsDeliveredAndLaterMovesRespectTheConfiguredRate() { + val deliveryTimes = mutableListOf() + ImeObservationBridge.install(samplingHz = 10) { deliveryTimes += it.eventUptimeMillis } + try { + ImeObservationBridge.publishObservation(observation("MOVE", 100)) + ImeObservationBridge.publishObservation(observation("MOVE", 150)) + ImeObservationBridge.publishObservation(observation("MOVE", 200)) + } finally { + ImeObservationBridge.uninstall() + } + + assertEquals(listOf(100L, 200L), deliveryTimes) + } + + private fun observation(action: String, eventUptimeMillis: Long) = ImeTouchObservation( + action = action, + eventUptimeMillis = eventUptimeMillis, + downUptimeMillis = eventUptimeMillis, + pointerId = 0, + relativeX = 0.5f, + relativeY = 0.5f, + pressure = 1f, + size = 1f, + orientationRadians = 0f, + toolType = 1, + keyCategory = "letter", + ) +} diff --git a/collector/location/src/main/kotlin/cool/linc/androiddatacollector/collector/location/LocationCollector.kt b/collector/location/src/main/kotlin/cool/linc/androiddatacollector/collector/location/LocationCollector.kt index 763d4ed..a9b53a7 100644 --- a/collector/location/src/main/kotlin/cool/linc/androiddatacollector/collector/location/LocationCollector.kt +++ b/collector/location/src/main/kotlin/cool/linc/androiddatacollector/collector/location/LocationCollector.kt @@ -12,18 +12,26 @@ import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.LocationConfiguration import cool.linc.androiddatacollector.core.definition.LocationPriority import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceCallbackBoundary +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback import com.google.android.gms.location.FusedLocationProviderClient import com.google.android.gms.location.LocationCallback import com.google.android.gms.location.LocationRequest import com.google.android.gms.location.LocationResult import com.google.android.gms.location.LocationServices import com.google.android.gms.location.Priority +import kotlinx.coroutines.NonCancellable import kotlinx.coroutines.tasks.await +import kotlinx.coroutines.withContext class LocationCollectorPlugin( context: Context, @@ -32,10 +40,9 @@ class LocationCollectorPlugin( override val descriptor = CollectorDescriptor( id = LocationConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Location", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 4_096, + eventContract = requireNotNull(ProtocolEventContracts[LocationConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -65,14 +72,15 @@ private class LocationCollector( ) : SerializedCallbackCollector(collectorContext, CHANNEL_CAPACITY) { private val client: FusedLocationProviderClient = LocationServices.getFusedLocationProviderClient(applicationContext) + private val callbackBoundary = SourceCallbackBoundary() private val callback = object : LocationCallback() { override fun onLocationResult(result: LocationResult) { - result.locations.forEach(::capture) + callbackBoundary.runIfActive { result.locations.forEach(::capture) } } } private var handlerThread: HandlerThread? = null - override suspend fun registerSource() { + override suspend fun registerSource(): SourceRegistrationResult { if (applicationContext.checkSelfPermission(Manifest.permission.ACCESS_FINE_LOCATION) != PackageManager.PERMISSION_GRANTED ) { @@ -82,21 +90,40 @@ private class LocationCollector( val request = LocationRequest.Builder(configuration.priority.toPlayServicesPriority(), configuration.intervalMillis) .setMinUpdateIntervalMillis(configuration.minimumIntervalMillis) .setMaxUpdateDelayMillis(configuration.maximumBatchDelayMillis) - .setMinUpdateDistanceMeters(configuration.minimumDisplacementMeters) + .setMinUpdateDistanceMeters(configuration.minimumDisplacementMillimeters / 1_000f) .build() - try { - client.requestLocationUpdates(request, callback, thread.looper).await() - } catch (failure: Throwable) { - thread.quitSafely() - throw failure - } - handlerThread = thread + callbackBoundary.activate() + var updatesRegistered = false + val result = registerSourceWithRollback( + register = { + withContext(NonCancellable) { + client.requestLocationUpdates(request, callback, thread.looper).await() + updatesRegistered = true + } + }, + rollback = { + completeSourceTeardown( + { if (updatesRegistered) client.removeLocationUpdates(callback).await() }, + { callbackBoundary.deactivate() }, + { thread.quitSafely() }, + ) + }, + ) + if (result == SourceRegistrationResult.Registered) handlerThread = thread + return result } - override suspend fun unregisterSource() { - client.removeLocationUpdates(callback).await() - handlerThread?.quitSafely() - handlerThread = null + override suspend fun unregisterSource(): SourceTeardownResult { + val thread = handlerThread + completeSourceTeardown( + { client.removeLocationUpdates(callback).await() }, + { callbackBoundary.deactivate() }, + { + thread?.quitSafely() + handlerThread = null + }, + ) + return SourceTeardownResult.Released } private fun capture(location: Location) { diff --git a/collector/network-state/src/main/kotlin/cool/linc/androiddatacollector/collector/networkstate/NetworkStateCollector.kt b/collector/network-state/src/main/kotlin/cool/linc/androiddatacollector/collector/networkstate/NetworkStateCollector.kt index 06487f1..6827e0c 100644 --- a/collector/network-state/src/main/kotlin/cool/linc/androiddatacollector/collector/networkstate/NetworkStateCollector.kt +++ b/collector/network-state/src/main/kotlin/cool/linc/androiddatacollector/collector/networkstate/NetworkStateCollector.kt @@ -9,11 +9,17 @@ import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.NetworkStateConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceCallbackBoundary +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback class NetworkStateCollectorPlugin( context: Context, @@ -22,10 +28,9 @@ class NetworkStateCollectorPlugin( override val descriptor = CollectorDescriptor( id = NetworkStateConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Network connection state", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 4_096, + eventContract = requireNotNull(ProtocolEventContracts[NetworkStateConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -50,24 +55,50 @@ private class NetworkStateCollector( collectorContext: CollectorContext, ) : SerializedCallbackCollector(collectorContext, CHANNEL_CAPACITY) { private val connectivityManager = androidContext.getSystemService(ConnectivityManager::class.java) + private val callbackBoundary = SourceCallbackBoundary() private val callback = object : ConnectivityManager.NetworkCallback() { - override fun onAvailable(network: Network) = capture("NETWORK_AVAILABLE", emptyMap()) + override fun onAvailable(network: Network) { + callbackBoundary.runIfActive { capture("NETWORK_AVAILABLE", emptyMap()) } + } - override fun onLost(network: Network) = capture("NETWORK_LOST", emptyMap()) + override fun onLost(network: Network) { + callbackBoundary.runIfActive { capture("NETWORK_LOST", emptyMap()) } + } override fun onCapabilitiesChanged( network: Network, networkCapabilities: NetworkCapabilities, - ) = capture("NETWORK_CAPABILITIES", encodeCapabilities(networkCapabilities)) + ) { + callbackBoundary.runIfActive { + capture("NETWORK_CAPABILITIES", encodeCapabilities(networkCapabilities)) + } + } } - override suspend fun registerSource() { - connectivityManager.registerDefaultNetworkCallback(callback) - captureCurrentState() + override suspend fun registerSource(): SourceRegistrationResult { + callbackBoundary.activate() + var callbackRegistered = false + return registerSourceWithRollback( + register = { + connectivityManager.registerDefaultNetworkCallback(callback) + callbackRegistered = true + captureCurrentState() + }, + rollback = { + completeSourceTeardown( + { if (callbackRegistered) connectivityManager.unregisterNetworkCallback(callback) }, + { callbackBoundary.deactivate() }, + ) + }, + ) } - override suspend fun unregisterSource() { - connectivityManager.unregisterNetworkCallback(callback) + override suspend fun unregisterSource(): SourceTeardownResult { + completeSourceTeardown( + { connectivityManager.unregisterNetworkCallback(callback) }, + { callbackBoundary.deactivate() }, + ) + return SourceTeardownResult.Released } private fun captureCurrentState() { diff --git a/collector/network-usage/src/main/kotlin/cool/linc/androiddatacollector/collector/networkusage/NetworkUsageCollector.kt b/collector/network-usage/src/main/kotlin/cool/linc/androiddatacollector/collector/networkusage/NetworkUsageCollector.kt index 5c5941c..2cb29cd 100644 --- a/collector/network-usage/src/main/kotlin/cool/linc/androiddatacollector/collector/networkusage/NetworkUsageCollector.kt +++ b/collector/network-usage/src/main/kotlin/cool/linc/androiddatacollector/collector/networkusage/NetworkUsageCollector.kt @@ -9,6 +9,7 @@ import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.NetworkTransport import cool.linc.androiddatacollector.core.definition.NetworkUsageConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor @@ -35,10 +36,9 @@ class NetworkUsageCollectorPlugin( override val descriptor = CollectorDescriptor( id = NetworkUsageConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Aggregate network usage", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 2_048, + eventContract = requireNotNull(ProtocolEventContracts[NetworkUsageConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -152,9 +152,16 @@ private class NetworkUsageCollector( ), ), ) - if (result == EmitResult.StorageFailure) { - mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "STORAGE_WRITE_FAILED") - return + when (result) { + EmitResult.ContractViolation -> { + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "EVENT_CONTRACT_VIOLATION") + return + } + EmitResult.StorageFailure -> { + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "STORAGE_WRITE_FAILED") + return + } + else -> Unit } } coverageStartUtcMillis = end diff --git a/collector/proximity/build.gradle.kts b/collector/proximity/build.gradle.kts new file mode 100644 index 0000000..2edfcd9 --- /dev/null +++ b/collector/proximity/build.gradle.kts @@ -0,0 +1,27 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.proximity" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + implementation(project(":collector:sensor-common")) + implementation(project(":core:collector-api")) + implementation(project(":core:study-definition")) + testImplementation(libs.junit4) +} diff --git a/collector/proximity/src/main/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollector.kt b/collector/proximity/src/main/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollector.kt new file mode 100644 index 0000000..0c9271d --- /dev/null +++ b/collector/proximity/src/main/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollector.kt @@ -0,0 +1,182 @@ +package cool.linc.androiddatacollector.collector.proximity + +import android.content.Context +import android.hardware.Sensor +import android.hardware.SensorEvent +import android.os.SystemClock +import cool.linc.androiddatacollector.collector.sensorcommon.AndroidSensorCollector +import cool.linc.androiddatacollector.core.collector.AccessKind +import cool.linc.androiddatacollector.core.collector.AccessRequirement +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.definition.ProximityConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import kotlin.math.abs + +class ProximityCollectorPlugin(context: Context) : CollectorPlugin { + private val applicationContext = context.applicationContext + + override val descriptor = CollectorDescriptor( + id = ProximityConfiguration.ID, + displayName = "Proximity", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[ProximityConfiguration.ID]), + ) + + override fun accessRequirements(configuration: CollectorConfiguration): Set { + val typed = configuration as? ProximityConfiguration + ?: throw IllegalArgumentException("Invalid proximity configuration") + return setOf(AccessRequirement(AccessKind.PROXIMITY_HARDWARE, typed.required)) + } + + override fun create(configuration: CollectorConfiguration, context: CollectorContext): Collector { + val typed = configuration as? ProximityConfiguration + ?: throw IllegalArgumentException("Invalid proximity configuration") + return ProximityCollector(applicationContext, typed, context) + } +} + +private class ProximityCollector( + androidContext: Context, + private val configuration: ProximityConfiguration, + collectorContext: CollectorContext, +) : AndroidSensorCollector( + androidContext = androidContext, + collectorContext = collectorContext, + sensorType = Sensor.TYPE_PROXIMITY, + samplingPeriodUs = configuration.minimumEventIntervalMs * 1_000, + maximumReportLatencyUs = 0, + threadName = "adc-proximity", + queueCapacity = 256, +) { + private val rateGate = LatestValueRateGate( + configuration.minimumEventIntervalMs.toLong(), + ) { previous, current -> + sameProximitySample(previous, current, configuration.changeThresholdMillimeters) + } + private var pendingScheduled = false + private val pendingRunnable = Runnable { sourceCallback(::publishPending) } + + override suspend fun onSourceRegistering() { + val latest = context.eventSink.latestEvent(ProximityConfiguration.ID) ?: return + rateGate.restoreLastEmission( + value = latest.proximitySampleOrNull(), + currentElapsedMillis = SystemClock.elapsedRealtime(), + ) + } + + override fun onSensorEvent(event: SensorEvent) { + val distance = event.values.firstOrNull() ?: return + val maximumRange = event.sensor.maximumRange + val sample = proximitySample(distance, maximumRange, event.timestamp, context.clocks.now()) ?: return + handle(rateGate.offer(sample, SystemClock.elapsedRealtime())) + } + + override fun onSourceUnregistering() { + sourceHandler?.removeCallbacks(pendingRunnable) + rateGate.clearPending() + pendingScheduled = false + } + + private fun publishPending() { + pendingScheduled = false + handle(rateGate.poll(SystemClock.elapsedRealtime())) + } + + private fun handle(decision: LatestValueRateGate.Decision) { + when (decision) { + is LatestValueRateGate.Decision.Emit -> { + sourceHandler?.removeCallbacks(pendingRunnable) + pendingScheduled = false + publish(decision.value) + } + is LatestValueRateGate.Decision.Defer -> if (!pendingScheduled) { + pendingScheduled = true + sourceHandler?.postDelayed(pendingRunnable, decision.delayMillis) + } + LatestValueRateGate.Decision.Suppress -> { + sourceHandler?.removeCallbacks(pendingRunnable) + pendingScheduled = false + } + } + } + + private fun publish(sample: ProximitySample) = emit(sample.eventDraft()) +} + +internal data class ProximitySample( + val observedTime: ResearchTime, + val sourceTimestampNanos: Long, + val distanceCentimeters: Float, + val maximumRangeCentimeters: Float, + val near: Boolean, +) + +internal fun proximitySample( + distanceCentimeters: Float, + maximumRangeCentimeters: Float, + sourceTimestampNanos: Long, + observedTime: ResearchTime, +): ProximitySample? { + if ( + !distanceCentimeters.isFinite() || + !maximumRangeCentimeters.isFinite() || + distanceCentimeters < 0 || + maximumRangeCentimeters < 0 || + sourceTimestampNanos < 0 + ) return null + return ProximitySample( + observedTime = observedTime, + sourceTimestampNanos = sourceTimestampNanos, + distanceCentimeters = distanceCentimeters, + maximumRangeCentimeters = maximumRangeCentimeters, + near = distanceCentimeters < maximumRangeCentimeters, + ) +} + +internal fun sameProximitySample( + previous: ProximitySample, + current: ProximitySample, + changeThresholdMillimeters: Int, +): Boolean = previous.near == current.near && + previous.maximumRangeCentimeters == current.maximumRangeCentimeters && + (previous.distanceCentimeters == current.distanceCentimeters || + abs(previous.distanceCentimeters - current.distanceCentimeters) * 10 < + changeThresholdMillimeters) + +internal fun ProximitySample.eventDraft() = EventDraft( + collectorId = ProximityConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = observedTime, + payloadType = "PROXIMITY_SAMPLE", + fields = mapOf( + "source_elapsed_realtime_nanos" to sourceTimestampNanos.toString(), + "distance_centimeters" to distanceCentimeters.toString(), + "maximum_range_centimeters" to maximumRangeCentimeters.toString(), + "near" to near.toString(), + ), +) + +internal fun RecordedEvent.proximitySampleOrNull(): ProximitySample? { + if ( + collectorId != ProximityConfiguration.ID || + payloadSchemaVersion != 1 || + payloadType != "PROXIMITY_SAMPLE" + ) return null + val sample = proximitySample( + distanceCentimeters = fields["distance_centimeters"]?.toFloatOrNull() ?: return null, + maximumRangeCentimeters = fields["maximum_range_centimeters"]?.toFloatOrNull() ?: return null, + sourceTimestampNanos = fields["source_elapsed_realtime_nanos"]?.toLongOrNull() ?: return null, + observedTime = observedTime, + ) ?: return null + val recordedNear = fields["near"]?.toBooleanStrictOrNull() ?: return null + return sample.takeIf { it.near == recordedNear } +} diff --git a/collector/proximity/src/test/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollectorTest.kt b/collector/proximity/src/test/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollectorTest.kt new file mode 100644 index 0000000..a43f3ae --- /dev/null +++ b/collector/proximity/src/test/kotlin/cool/linc/androiddatacollector/collector/proximity/ProximityCollectorTest.kt @@ -0,0 +1,83 @@ +package cool.linc.androiddatacollector.collector.proximity + +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.ProximityConfiguration +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +class ProximityCollectorTest { + @Test + fun mapsRawDistanceAndKeepsItsCaptureTime() { + val observed = time(10) + val near = proximitySample(1f, 5f, 123, observed) + val far = proximitySample(5f, 5f, 124, time(20)) + + requireNotNull(near) + requireNotNull(far) + val event = near.eventDraft() + assertTrue(near.near) + assertFalse(far.near) + assertEquals(observed, event.observedTime) + assertEquals("1.0", event.fields["distance_centimeters"]) + assertEquals("5.0", event.fields["maximum_range_centimeters"]) + assertEquals("true", event.fields["near"]) + assertEquals(ProximityConfiguration.ID, event.collectorId) + assertTrue(requireNotNull(ProtocolEventContracts[ProximityConfiguration.ID]).accepts(event, 1)) + } + + @Test + fun changeGateIgnoresCaptureMetadataButNotPhysicalState() { + val previous = requireNotNull(proximitySample(1f, 5f, 100, time(10))) + val belowThreshold = requireNotNull(proximitySample(1.09f, 5f, 200, time(20))) + val atThreshold = requireNotNull(proximitySample(1.1f, 5f, 200, time(20))) + + assertTrue(sameProximitySample(previous, belowThreshold, 1)) + assertFalse(sameProximitySample(previous, atThreshold, 1)) + assertFalse(sameProximitySample(previous, previous.copy(maximumRangeCentimeters = 6f), 1)) + } + + @Test + fun rejectsInvalidPhysicalSamples() { + assertNull(proximitySample(Float.NaN, 5f, 1, time(1))) + assertNull(proximitySample(1f, Float.POSITIVE_INFINITY, 1, time(1))) + assertNull(proximitySample(-1f, 5f, 1, time(1))) + assertNull(proximitySample(1f, 5f, -1, time(1))) + } + + @Test + fun processRestartRestoresProximityValueAndMinimumInterval() { + val previous = requireNotNull( + proximitySample(1f, 5f, 9_000_000_000, ResearchTime(10_000, 10_000_000_000, "boot-a")), + ) + val draft = previous.eventDraft() + val recorded = RecordedEvent( + 1, + draft.collectorId, + draft.payloadSchemaVersion, + draft.observedTime, + draft.payloadType, + draft.fields, + ) + val gate = LatestValueRateGate(100) { old, current -> + sameProximitySample(old, current, 1) + } + gate.restoreLastEmission( + recorded.proximitySampleOrNull(), + 10_050, + ) + + assertEquals(LatestValueRateGate.Decision.Suppress, gate.offer(previous, 10_050)) + assertEquals( + LatestValueRateGate.Decision.Defer(100), + gate.offer(previous.copy(distanceCentimeters = 2f), 10_050), + ) + } + + private fun time(value: Long) = ResearchTime(value, value, "boot-a") +} diff --git a/collector/sensor-common/build.gradle.kts b/collector/sensor-common/build.gradle.kts new file mode 100644 index 0000000..ff02ae3 --- /dev/null +++ b/collector/sensor-common/build.gradle.kts @@ -0,0 +1,25 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.sensorcommon" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + api(project(":core:collector-api")) + testImplementation(libs.junit4) +} diff --git a/collector/sensor-common/src/main/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/AndroidSensorCollector.kt b/collector/sensor-common/src/main/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/AndroidSensorCollector.kt new file mode 100644 index 0000000..26a80db --- /dev/null +++ b/collector/sensor-common/src/main/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/AndroidSensorCollector.kt @@ -0,0 +1,112 @@ +package cool.linc.androiddatacollector.collector.sensorcommon + +import android.content.Context +import android.hardware.Sensor +import android.hardware.SensorEvent +import android.hardware.SensorEventListener +import android.hardware.SensorManager +import android.os.Handler +import android.os.HandlerThread +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceCallbackBoundary +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback +import cool.linc.androiddatacollector.core.model.EventDraft + +/** Common listener-thread ownership for raw Android sensor collectors. */ +abstract class AndroidSensorCollector( + androidContext: Context, + collectorContext: CollectorContext, + private val sensorType: Int, + private val samplingPeriodUs: Int, + private val maximumReportLatencyUs: Int, + private val threadName: String, + queueCapacity: Int, +) : SerializedCallbackCollector(collectorContext, queueCapacity), SensorEventListener { + private val sensorManager = androidContext.getSystemService(SensorManager::class.java) + private val sensor by lazy { + sensorManager.getDefaultSensor(sensorType) + ?: throw IllegalStateException("Required sensor hardware is unavailable") + } + private val callbackBoundary = SourceCallbackBoundary() + private var sourceThread: HandlerThread? = null + protected var sourceHandler: Handler? = null + private set + + final override fun onSensorChanged(event: SensorEvent) { + sourceCallback { + if (event.sensor.type == sensorType) onSensorEvent(event) + } + } + + final override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit + + protected open fun onSensorEvent(event: SensorEvent) { + eventDraft(event)?.let(::emit) + } + + protected open fun eventDraft(event: SensorEvent): EventDraft? = null + + protected fun emit(event: EventDraft) = capture { event } + + override suspend fun registerSource(): SourceRegistrationResult { + onSourceRegistering() + val thread = HandlerThread(threadName).also { it.start() } + val handler = Handler(thread.looper) + sourceThread = thread + sourceHandler = handler + callbackBoundary.activate() + var listenerRegistered = false + return registerSourceWithRollback( + register = { + check( + sensorManager.registerListener( + this, + sensor, + samplingPeriodUs, + maximumReportLatencyUs, + handler, + ), + ) { "Android rejected the sensor listener" } + listenerRegistered = true + }, + rollback = { + completeSourceTeardown( + { if (listenerRegistered) sensorManager.unregisterListener(this, sensor) }, + { callbackBoundary.deactivate(::onSourceUnregistering) }, + { releaseSource(thread, handler) }, + ) + }, + ) + } + + override suspend fun unregisterSource(): SourceTeardownResult { + val handler = sourceHandler + val thread = sourceThread + completeSourceTeardown( + { sensorManager.unregisterListener(this, sensor) }, + { callbackBoundary.deactivate(::onSourceUnregistering) }, + { releaseSource(thread, handler) }, + ) + return SourceTeardownResult.Released + } + + private fun releaseSource(thread: HandlerThread?, handler: Handler?) { + handler?.removeCallbacksAndMessages(null) + sourceHandler = null + thread?.quitSafely() + sourceThread = null + } + + protected open suspend fun onSourceRegistering() = Unit + + protected open fun onSourceUnregistering() = Unit + + /** Serializes sensor and handler callbacks against teardown of collector-specific state. */ + protected fun sourceCallback(block: () -> Unit) { + callbackBoundary.runIfActive(block) + } +} diff --git a/collector/sensor-common/src/test/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/SensorSourceLifecycleTest.kt b/collector/sensor-common/src/test/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/SensorSourceLifecycleTest.kt new file mode 100644 index 0000000..19d8f61 --- /dev/null +++ b/collector/sensor-common/src/test/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/SensorSourceLifecycleTest.kt @@ -0,0 +1,122 @@ +package cool.linc.androiddatacollector.collector.sensorcommon + +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import cool.linc.androiddatacollector.core.collector.SourceCallbackBoundary +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertSame +import org.junit.Assert.assertTrue +import org.junit.Test + +class SensorSourceLifecycleTest { + @Test + fun inFlightCallbackFinishesBeforeTeardownAndNoLaterCallbackCanMutateState() { + val boundary = SourceCallbackBoundary() + val entered = CountDownLatch(1) + val release = CountDownLatch(1) + val completed = CountDownLatch(1) + val mutations = mutableListOf() + boundary.activate() + val callback = Thread { + boundary.runIfActive { + entered.countDown() + check(release.await(1, TimeUnit.SECONDS)) + mutations += "callback" + } + completed.countDown() + } + callback.start() + assertTrue(entered.await(1, TimeUnit.SECONDS)) + val teardown = Thread { + boundary.deactivate { mutations += "teardown" } + } + teardown.start() + + assertFalse(completed.await(50, TimeUnit.MILLISECONDS)) + release.countDown() + assertTrue(completed.await(1, TimeUnit.SECONDS)) + callback.join(1_000) + teardown.join(1_000) + assertEquals(listOf("callback", "teardown"), mutations) + assertFalse(boundary.runIfActive { mutations += "late-callback" }) + assertEquals(listOf("callback", "teardown"), mutations) + } + + @Test + fun failedSensorManagerRegistrationRollsBackTheSource() = runBlocking { + val calls = mutableListOf() + val registrationFailure = IllegalStateException("registration") + + val result = registerSourceWithRollback( + register = { + calls += "register" + throw registrationFailure + }, + rollback = { + completeSourceTeardown( + { calls += "sensor-manager-unregister" }, + { calls += "collector-pending-clear" }, + { calls += "handler-callbacks-and-thread-release" }, + ) + }, + ) + + assertTrue(result is SourceRegistrationResult.Released) + assertSame(registrationFailure, (result as SourceRegistrationResult.Released).failure) + assertEquals( + listOf( + "register", + "sensor-manager-unregister", + "collector-pending-clear", + "handler-callbacks-and-thread-release", + ), + calls, + ) + } + + @Test + fun queuedCallbackUsesTheSameBoundaryAsLiveSensorCallbacks() { + val boundary = SourceCallbackBoundary() + var publications = 0 + val queuedCallback = Runnable { boundary.runIfActive { publications++ } } + boundary.activate() + + queuedCallback.run() + boundary.deactivate {} + queuedCallback.run() + + assertEquals(1, publications) + } + + @Test + fun unregisterFailureStillClearsCollectorPendingWorkAndHandlerCallbacks() = runBlocking { + val calls = mutableListOf() + val unregisterFailure = IllegalStateException("unregister") + + val thrown = runCatching { + completeSourceTeardown( + { + calls += "sensor-manager-unregister" + throw unregisterFailure + }, + { calls += "collector-pending-clear" }, + { calls += "handler-callbacks-and-thread-release" }, + ) + }.exceptionOrNull() + + assertSame(unregisterFailure, thrown) + assertEquals( + listOf( + "sensor-manager-unregister", + "collector-pending-clear", + "handler-callbacks-and-thread-release", + ), + calls, + ) + } +} diff --git a/collector/temporal-context/build.gradle.kts b/collector/temporal-context/build.gradle.kts new file mode 100644 index 0000000..bf4f63a --- /dev/null +++ b/collector/temporal-context/build.gradle.kts @@ -0,0 +1,27 @@ +plugins { + alias(libs.plugins.android.library) +} + +android { + namespace = "cool.linc.androiddatacollector.collector.temporalcontext" + compileSdk = 37 + defaultConfig { minSdk = 34 } + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } +} + +kotlin { + compilerOptions { + jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 + allWarningsAsErrors = true + } +} + +dependencies { + implementation(project(":core:collector-api")) + implementation(project(":core:study-definition")) + implementation(libs.coroutines.android) + testImplementation(libs.junit4) +} diff --git a/collector/temporal-context/src/main/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollector.kt b/collector/temporal-context/src/main/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollector.kt new file mode 100644 index 0000000..f80f46a --- /dev/null +++ b/collector/temporal-context/src/main/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollector.kt @@ -0,0 +1,226 @@ +package cool.linc.androiddatacollector.collector.temporalcontext + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.content.IntentFilter +import android.os.Handler +import android.os.Looper +import android.os.SystemClock +import cool.linc.androiddatacollector.core.collector.AccessRequirement +import cool.linc.androiddatacollector.core.collector.Collector +import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorPlugin +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector +import cool.linc.androiddatacollector.core.collector.SourceRegistrationResult +import cool.linc.androiddatacollector.core.collector.SourceTeardownResult +import cool.linc.androiddatacollector.core.collector.completeSourceTeardown +import cool.linc.androiddatacollector.core.collector.registerSourceWithRollback +import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import cool.linc.androiddatacollector.core.definition.TemporalContextConfiguration +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import java.time.Instant +import java.time.ZoneId +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext + +class TemporalContextCollectorPlugin(context: Context) : CollectorPlugin { + private val applicationContext = context.applicationContext + + override val descriptor = CollectorDescriptor( + id = TemporalContextConfiguration.ID, + displayName = "Temporal context", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[TemporalContextConfiguration.ID]), + ) + + override fun accessRequirements(configuration: CollectorConfiguration): Set { + require(configuration is TemporalContextConfiguration) { "Invalid temporal-context configuration" } + return emptySet() + } + + override fun create(configuration: CollectorConfiguration, context: CollectorContext): Collector { + require(configuration is TemporalContextConfiguration) { "Invalid temporal-context configuration" } + return TemporalContextCollector(applicationContext, context) + } +} + +private class TemporalContextCollector( + private val applicationContext: Context, + collectorContext: CollectorContext, +) : SerializedCallbackCollector(collectorContext, 64) { + private val mainHandler = Handler(Looper.getMainLooper()) + private val rateGate = LatestValueRateGate( + MINIMUM_INTERVAL_MILLIS, + ::sameTemporalEvent, + ) + private val pendingRunnable = Runnable(::publishPending) + private var observedContext: TemporalSnapshot? = null + private var pendingScheduled = false + private val receiver = object : BroadcastReceiver() { + override fun onReceive(context: Context?, intent: Intent?) { + val observedTime = this@TemporalContextCollector.context.clocks.now() + val snapshot = snapshot(observedTime) + val reason = when (intent?.action) { + Intent.ACTION_TIMEZONE_CHANGED -> "TIMEZONE_CHANGED" + Intent.ACTION_TIME_CHANGED -> "TIME_SET" + Intent.ACTION_TIME_TICK -> if (snapshot != observedContext) "UTC_OFFSET_CHANGED" else null + else -> null + } + observedContext = snapshot + reason?.let { offer(TemporalEvent(it, snapshot, observedTime)) } + } + } + + override suspend fun registerSource(): SourceRegistrationResult = withContext(Dispatchers.Main.immediate) { + val latest = context.eventSink.latestEvent(TemporalContextConfiguration.ID) + if (latest != null) { + rateGate.restoreLastEmission( + value = latest.temporalEventOrNull(), + currentElapsedMillis = SystemClock.elapsedRealtime(), + ) + } + val hasPriorEvent = latest != null + val filter = IntentFilter().apply { + addAction(Intent.ACTION_TIME_TICK) + addAction(Intent.ACTION_TIME_CHANGED) + addAction(Intent.ACTION_TIMEZONE_CHANGED) + } + var receiverRegistered = false + registerSourceWithRollback( + register = { + applicationContext.registerReceiver(receiver, filter, Context.RECEIVER_NOT_EXPORTED) + receiverRegistered = true + val observedTime = context.clocks.now() + val current = snapshot(observedTime) + observedContext = current + offer(TemporalEvent(if (hasPriorEvent) "RECONCILED" else "STUDY_STARTED", current, observedTime)) + }, + rollback = { + completeSourceTeardown( + { if (receiverRegistered) applicationContext.unregisterReceiver(receiver) }, + ::clearPending, + ) + }, + ) + } + + override suspend fun unregisterSource(): SourceTeardownResult = withContext(Dispatchers.Main.immediate) { + completeSourceTeardown( + { applicationContext.unregisterReceiver(receiver) }, + ::clearPending, + ) + SourceTeardownResult.Released + } + + private fun clearPending() { + mainHandler.removeCallbacks(pendingRunnable) + rateGate.clearPending() + pendingScheduled = false + } + + private fun snapshot(observedTime: ResearchTime): TemporalSnapshot { + return temporalSnapshot(ZoneId.systemDefault(), observedTime) + } + + private fun offer(event: TemporalEvent) { + handle(rateGate.offer(event, SystemClock.elapsedRealtime())) + } + + private fun publishPending() { + pendingScheduled = false + handle(rateGate.poll(SystemClock.elapsedRealtime())) + } + + private fun handle(decision: LatestValueRateGate.Decision) { + when (decision) { + is LatestValueRateGate.Decision.Emit -> { + mainHandler.removeCallbacks(pendingRunnable) + pendingScheduled = false + publish(decision.value) + } + is LatestValueRateGate.Decision.Defer -> if (!pendingScheduled) { + pendingScheduled = true + mainHandler.postDelayed(pendingRunnable, decision.delayMillis) + } + LatestValueRateGate.Decision.Suppress -> { + mainHandler.removeCallbacks(pendingRunnable) + pendingScheduled = false + } + } + } + + private fun publish(event: TemporalEvent) { + capture(event::eventDraft) + } + + private companion object { const val MINIMUM_INTERVAL_MILLIS = 60_000L } +} + +internal data class TemporalEvent( + val reason: String, + val snapshot: TemporalSnapshot, + val observedTime: ResearchTime, +) + +internal data class TemporalSnapshot( + val timezoneId: String, + val utcOffsetSeconds: Int, + val daylightSavingTime: Boolean, +) + +internal fun sameTemporalEvent(previous: TemporalEvent, current: TemporalEvent): Boolean = + current.reason != "TIME_SET" && previous.reason == current.reason && previous.snapshot == current.snapshot + +internal fun temporalSnapshot(zone: ZoneId, observedTime: ResearchTime): TemporalSnapshot { + val instant = Instant.ofEpochMilli(observedTime.wallTimeUtcMillis) + return TemporalSnapshot( + timezoneId = zone.id, + utcOffsetSeconds = zone.rules.getOffset(instant).totalSeconds, + daylightSavingTime = zone.rules.isDaylightSavings(instant), + ) +} + +internal fun TemporalEvent.eventDraft() = EventDraft( + collectorId = TemporalContextConfiguration.ID, + payloadSchemaVersion = 1, + observedTime = observedTime, + payloadType = "TEMPORAL_CONTEXT", + fields = mapOf( + "change_reason" to reason, + "timezone_id" to snapshot.timezoneId, + "utc_offset_seconds" to snapshot.utcOffsetSeconds.toString(), + "daylight_saving_time" to snapshot.daylightSavingTime.toString(), + ), +) + +internal fun RecordedEvent.temporalEventOrNull(): TemporalEvent? { + if ( + collectorId != TemporalContextConfiguration.ID || + payloadSchemaVersion != 1 || + payloadType != "TEMPORAL_CONTEXT" + ) return null + val reason = fields["change_reason"]?.takeIf(TEMPORAL_REASONS::contains) ?: return null + val timezoneId = fields["timezone_id"] ?: return null + val utcOffsetSeconds = fields["utc_offset_seconds"]?.toIntOrNull() ?: return null + val daylightSavingTime = fields["daylight_saving_time"]?.toBooleanStrictOrNull() ?: return null + return TemporalEvent( + reason, + TemporalSnapshot(timezoneId, utcOffsetSeconds, daylightSavingTime), + observedTime, + ) +} + +private val TEMPORAL_REASONS = setOf( + "RECONCILED", + "STUDY_STARTED", + "TIMEZONE_CHANGED", + "TIME_SET", + "UTC_OFFSET_CHANGED", +) diff --git a/collector/temporal-context/src/test/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollectorTest.kt b/collector/temporal-context/src/test/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollectorTest.kt new file mode 100644 index 0000000..ca9390c --- /dev/null +++ b/collector/temporal-context/src/test/kotlin/cool/linc/androiddatacollector/collector/temporalcontext/TemporalContextCollectorTest.kt @@ -0,0 +1,91 @@ +package cool.linc.androiddatacollector.collector.temporalcontext + +import cool.linc.androiddatacollector.core.collector.LatestValueRateGate +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.TemporalContextConfiguration +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import java.time.Instant +import java.time.ZoneId +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +class TemporalContextCollectorTest { + @Test + fun derivesOffsetAndDaylightSavingFromTheCapturedInstant() { + val winter = time(Instant.parse("2026-01-15T12:00:00Z").toEpochMilli()) + val summer = time(Instant.parse("2026-07-15T12:00:00Z").toEpochMilli()) + val zone = ZoneId.of("America/New_York") + + assertEquals(-18_000, temporalSnapshot(zone, winter).utcOffsetSeconds) + assertFalse(temporalSnapshot(zone, winter).daylightSavingTime) + assertEquals(-14_400, temporalSnapshot(zone, summer).utcOffsetSeconds) + assertTrue(temporalSnapshot(zone, summer).daylightSavingTime) + } + + @Test + fun deferredDraftKeepsCaptureTimeWhileDeduplicationIgnoresOnlyThatTime() { + val snapshot = TemporalSnapshot("UTC", 0, false) + val captured = TemporalEvent("TIME_SET", snapshot, time(10)) + val later = TemporalEvent("TIME_SET", snapshot, time(30)) + val event = captured.eventDraft() + + assertFalse(sameTemporalEvent(captured, later)) + assertEquals(captured.observedTime, event.observedTime) + assertEquals(TemporalContextConfiguration.ID, event.collectorId) + assertTrue(requireNotNull(ProtocolEventContracts[TemporalContextConfiguration.ID]).accepts(event, 1)) + assertFalse(sameTemporalEvent(captured, later.copy(reason = "TIMEZONE_CHANGED"))) + assertFalse(sameTemporalEvent(captured, later.copy(snapshot = snapshot.copy(utcOffsetSeconds = 60)))) + assertTrue( + sameTemporalEvent( + captured.copy(reason = "RECONCILED"), + later.copy(reason = "RECONCILED"), + ), + ) + } + + @Test + fun repeatedClockSetBroadcastsCoalesceWithinTheBoundButEmitAgainAfterIt() { + val snapshot = TemporalSnapshot("UTC", 0, false) + val first = TemporalEvent("TIME_SET", snapshot, time(10)) + val second = TemporalEvent("TIME_SET", snapshot, time(20)) + val gate = LatestValueRateGate(60_000L, ::sameTemporalEvent) + + assertEquals(LatestValueRateGate.Decision.Emit(first), gate.offer(first, 1_000)) + assertEquals(LatestValueRateGate.Decision.Defer(59_000), gate.offer(second, 2_000)) + assertEquals(LatestValueRateGate.Decision.Emit(second), gate.poll(61_000)) + val third = TemporalEvent("TIME_SET", snapshot, time(30)) + assertEquals(LatestValueRateGate.Decision.Emit(third), gate.offer(third, 121_000)) + } + + @Test + fun processRestartRestoresClockSetRateWatermarkWithoutSuppressingItForever() { + val previous = TemporalEvent( + "TIME_SET", + TemporalSnapshot("UTC", 0, false), + ResearchTime(10_000, 10_000_000_000, "boot-a"), + ) + val draft = previous.eventDraft() + val recorded = RecordedEvent( + 1, + draft.collectorId, + draft.payloadSchemaVersion, + draft.observedTime, + draft.payloadType, + draft.fields, + ) + val gate = LatestValueRateGate(60_000L, ::sameTemporalEvent) + gate.restoreLastEmission( + recorded.temporalEventOrNull(), + 10_100, + ) + + val current = previous.copy(observedTime = ResearchTime(10_100, 10_100_000_000, "boot-a")) + assertEquals(LatestValueRateGate.Decision.Defer(60_000), gate.offer(current, 10_100)) + assertEquals(LatestValueRateGate.Decision.Emit(current), gate.poll(70_100)) + } + + private fun time(wallMillis: Long) = ResearchTime(wallMillis, wallMillis, "boot-a") +} diff --git a/collector/usage-events/src/main/kotlin/cool/linc/androiddatacollector/collector/usageevents/UsageEventsCollector.kt b/collector/usage-events/src/main/kotlin/cool/linc/androiddatacollector/collector/usageevents/UsageEventsCollector.kt index 63e7725..074062f 100644 --- a/collector/usage-events/src/main/kotlin/cool/linc/androiddatacollector/collector/usageevents/UsageEventsCollector.kt +++ b/collector/usage-events/src/main/kotlin/cool/linc/androiddatacollector/collector/usageevents/UsageEventsCollector.kt @@ -8,6 +8,7 @@ import cool.linc.androiddatacollector.core.collector.AccessKind import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.definition.UsageEventsConfiguration import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext @@ -35,10 +36,9 @@ class UsageEventsCollectorPlugin( override val descriptor = CollectorDescriptor( id = UsageEventsConfiguration.ID, - payloadSchemaVersion = 1, displayName = "App and screen usage events", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 4_096, + eventContract = requireNotNull(ProtocolEventContracts[UsageEventsConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set { @@ -143,7 +143,7 @@ private class UsageEventsCollector( put("source_time_utc_millis", source.timestamp.toString()) source.packageName?.takeIf(String::isNotBlank)?.let { put("package_name", it) } } - if ( + when ( collectorContext.eventSink.emit( token, EventDraft( @@ -153,10 +153,17 @@ private class UsageEventsCollector( payloadType = source.type, fields = fields, ), - ) == EmitResult.StorageFailure + ) ) { - mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "STORAGE_WRITE_FAILED") - return + EmitResult.ContractViolation -> { + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "EVENT_CONTRACT_VIOLATION") + return + } + EmitResult.StorageFailure -> { + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "STORAGE_WRITE_FAILED") + return + } + else -> Unit } } queryStartUtcMillis = end diff --git a/core/access/src/main/kotlin/cool/linc/androiddatacollector/core/access/AccessManager.kt b/core/access/src/main/kotlin/cool/linc/androiddatacollector/core/access/AccessManager.kt index daefa68..c9e04e6 100644 --- a/core/access/src/main/kotlin/cool/linc/androiddatacollector/core/access/AccessManager.kt +++ b/core/access/src/main/kotlin/cool/linc/androiddatacollector/core/access/AccessManager.kt @@ -41,7 +41,10 @@ class AccessManager( AccessKind.FINE_LOCATION, AccessKind.NOTIFICATIONS, AccessKind.RESEARCH_KEYBOARD_SELECTED, - AccessKind.ACCELEROMETER_HARDWARE -> null + AccessKind.ACCELEROMETER_HARDWARE, + AccessKind.GYROSCOPE_HARDWARE, + AccessKind.AMBIENT_LIGHT_HARDWARE, + AccessKind.PROXIMITY_HARDWARE -> null } fun showInputMethodPicker() { @@ -58,8 +61,15 @@ class AccessManager( AccessKind.ACCELEROMETER_HARDWARE -> applicationContext .getSystemService(SensorManager::class.java) .getDefaultSensor(Sensor.TYPE_ACCELEROMETER) != null + AccessKind.GYROSCOPE_HARDWARE -> hasSensor(Sensor.TYPE_GYROSCOPE) + AccessKind.AMBIENT_LIGHT_HARDWARE -> hasSensor(Sensor.TYPE_LIGHT) + AccessKind.PROXIMITY_HARDWARE -> hasSensor(Sensor.TYPE_PROXIMITY) } + private fun hasSensor(type: Int): Boolean = applicationContext + .getSystemService(SensorManager::class.java) + .getDefaultSensor(type) != null + private fun permissionGranted(permission: String): Boolean = applicationContext.checkSelfPermission(permission) == PackageManager.PERMISSION_GRANTED diff --git a/core/collector-api/build.gradle.kts b/core/collector-api/build.gradle.kts index f8673ba..603c8e0 100644 --- a/core/collector-api/build.gradle.kts +++ b/core/collector-api/build.gradle.kts @@ -16,6 +16,7 @@ dependencies { api(project(":core:model")) api(project(":core:study-definition")) api(libs.coroutines.core) + implementation(libs.gson) testImplementation(libs.coroutines.test) testImplementation(libs.junit4) } diff --git a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt index be36f65..e6ee4df 100644 --- a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt +++ b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt @@ -3,6 +3,12 @@ package cool.linc.androiddatacollector.core.collector import cool.linc.androiddatacollector.core.model.EventDraft import cool.linc.androiddatacollector.core.model.RecordedEvent import cool.linc.androiddatacollector.core.definition.CollectorConfiguration +import com.google.gson.JsonParser +import com.google.gson.JsonParseException +import com.google.gson.Strictness +import com.google.gson.stream.JsonReader +import com.google.gson.stream.JsonToken +import java.io.StringReader import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.flow.StateFlow @@ -19,6 +25,9 @@ enum class AccessKind { RESEARCH_KEYBOARD_ENABLED, RESEARCH_KEYBOARD_SELECTED, ACCELEROMETER_HARDWARE, + GYROSCOPE_HARDWARE, + AMBIENT_LIGHT_HARDWARE, + PROXIMITY_HARDWARE, } data class AccessRequirement( @@ -37,16 +46,16 @@ interface StudyAccessGateway { data class CollectorDescriptor( val id: String, - val payloadSchemaVersion: Int, val displayName: String, val privacyClass: PrivacyClass, - val maximumEncodedEventBytes: Int, + val eventContract: CollectorEventContract, ) { + val payloadSchemaVersion get() = eventContract.payloadSchemaVersion + val maximumEncodedEventBytes get() = eventContract.maximumEncodedEventBytes + init { require(ID_PATTERN.matches(id)) { "Invalid collector ID" } - require(payloadSchemaVersion > 0) { "Payload schema version must be positive" } require(displayName.isNotBlank()) { "Collector display name must not be blank" } - require(maximumEncodedEventBytes in 128..65_536) { "Invalid maximum event size" } } private companion object { @@ -54,6 +63,114 @@ data class CollectorDescriptor( } } +enum class EventFieldType { + BOOLEAN, + DECIMAL_STRING, + ENUM, + FLOAT32, + FLOAT64, + INT32, + JSON_STRING, + STRING, +} + +data class EventFieldContract( + val type: EventFieldType, + val required: Boolean, + val enumValues: Set = emptySet(), + val minimum: Double? = null, + val maximum: Double? = null, + val maximumLength: Int? = null, +) { + init { + require((type == EventFieldType.ENUM) == enumValues.isNotEmpty()) { "Invalid event enum contract" } + require(minimum == null || minimum.isFinite()) { "Invalid event field minimum" } + require(maximum == null || maximum.isFinite()) { "Invalid event field maximum" } + require(minimum == null || maximum == null || minimum <= maximum) { "Invalid event field range" } + require(maximumLength == null || maximumLength > 0) { "Invalid event field length" } + } + + internal fun accepts(value: String): Boolean { + if (maximumLength != null && value.length > maximumLength) return false + return when (type) { + EventFieldType.BOOLEAN -> value == "true" || value == "false" + EventFieldType.DECIMAL_STRING -> UNSIGNED_DECIMAL.matches(value) && value.toLongOrNull() != null + EventFieldType.ENUM -> value in enumValues + EventFieldType.FLOAT32 -> FLOAT_DECIMAL.matches(value) && + value.toFloatOrNull()?.let { it.isFinite() && inRange(it.toDouble()) } == true + EventFieldType.FLOAT64 -> FLOAT_DECIMAL.matches(value) && + value.toDoubleOrNull()?.let { it.isFinite() && inRange(it) } == true + EventFieldType.INT32 -> SIGNED_INTEGER.matches(value) && + value.toIntOrNull()?.let { inRange(it.toDouble()) } == true + EventFieldType.JSON_STRING -> isStrictJson(value) + EventFieldType.STRING -> true + } + } + + private fun inRange(value: Double): Boolean = + (minimum == null || value >= minimum) && (maximum == null || value <= maximum) + + private companion object { + val UNSIGNED_DECIMAL = Regex("0|[1-9][0-9]*") + val SIGNED_INTEGER = Regex("0|-?[1-9][0-9]*") + val FLOAT_DECIMAL = Regex("[+-]?(?:(?:[0-9]+(?:\\.[0-9]*)?|\\.[0-9]+)(?:[eE][+-]?[0-9]+)?)") + + fun isStrictJson(value: String): Boolean = try { + val reader = JsonReader(StringReader(value)).apply { strictness = Strictness.STRICT } + JsonParser.parseReader(reader) + reader.peek() == JsonToken.END_DOCUMENT + } catch (_: JsonParseException) { + false + } catch (_: java.io.IOException) { + false + } + } +} + +data class EventPayloadContract( + val fields: Map, +) { + init { + require(fields.keys.all(FIELD_NAME::matches)) { "Invalid event field name" } + } + + internal fun accepts(values: Map): Boolean = + values.keys.all(fields::containsKey) && + fields.all { (name, contract) -> + val value = values[name] + if (value == null) !contract.required else contract.accepts(value) + } + + private companion object { val FIELD_NAME = Regex("[a-z][a-z0-9_]{1,63}") } +} + +data class CollectorEventContract( + val payloadSchemaVersion: Int, + val maximumEncodedEventBytes: Int, + val payloads: Map, +) { + init { + require(payloadSchemaVersion > 0) { "Payload schema version must be positive" } + require(maximumEncodedEventBytes in 128..65_536) { "Invalid maximum event size" } + require(payloads.isNotEmpty()) { "Collector must declare at least one payload" } + require(payloads.keys.all(PAYLOAD_TYPE::matches)) { "Invalid payload type" } + } + + fun accepts(event: EventDraft, sequenceNumber: Long): Boolean { + if (event.payloadSchemaVersion != payloadSchemaVersion) return false + val payload = payloads[event.payloadType] ?: return false + if (!payload.accepts(event.fields)) return false + val encodedBytes = try { + event.protocolEncodedBytes(sequenceNumber) + } catch (_: IllegalArgumentException) { + return false + } + return encodedBytes <= maximumEncodedEventBytes + } + + private companion object { val PAYLOAD_TYPE = Regex("[A-Z][A-Z0-9_]{1,63}") } +} + enum class CollectorStatus { STOPPED, ACTIVE, @@ -87,9 +204,78 @@ sealed interface EmitResult { data object RejectedByAdmissionGate : EmitResult + /** The collector crossed its declared ID, schema, or maximum encoded-size boundary. */ + data object ContractViolation : EmitResult + data object StorageFailure : EmitResult } +/** Exact byte count of this event in the authenticated Protocol v1 event representation. */ +fun EventDraft.protocolEncodedBytes(sequenceNumber: Long): Int { + require(sequenceNumber > 0) { "Sequence number must be positive" } + var size = EVENT_JSON_PUNCTUATION_BYTES + size += collectorId.quotedJsonBytes() + size += fields.entries.sumOf { (key, value) -> + key.quotedJsonBytes() + JSON_NAME_SEPARATOR_BYTES + value.quotedJsonBytes() + } + size += (fields.size - 1).coerceAtLeast(0) * JSON_VALUE_SEPARATOR_BYTES + size += observedTime.bootSessionId.quotedJsonBytes() + size += observedTime.elapsedRealtimeNanos.toString().quotedJsonBytes() + size += observedTime.wallTimeUtcMillis.toString().quotedJsonBytes() + size += payloadSchemaVersion.toString().length + size += payloadType.quotedJsonBytes() + size += sequenceNumber.toString().quotedJsonBytes() + return size +} + +private fun String.quotedJsonBytes(): Int { + var bytes = JSON_QUOTE_BYTES * 2 + var index = 0 + while (index < length) { + val character = this[index] + bytes += when { + character == '"' || character == '\\' -> 2 + character in JSON_NAMED_ESCAPES -> 2 + character.code < 0x20 -> 6 + character.code < 0x80 -> 1 + character.code < 0x800 -> 2 + character.isHighSurrogate() -> { + require(index + 1 < length && this[index + 1].isLowSurrogate()) { + "Event text contains an unpaired surrogate" + } + index++ + 4 + } + character.isLowSurrogate() -> throw IllegalArgumentException( + "Event text contains an unpaired surrogate", + ) + else -> 3 + } + index++ + } + return bytes +} + +private const val JSON_QUOTE_BYTES = 1 +private const val JSON_NAME_SEPARATOR_BYTES = 1 +private const val JSON_VALUE_SEPARATOR_BYTES = 1 +private val JSON_NAMED_ESCAPES = setOf('\b', '\t', '\n', '\u000C', '\r') + +/** + * UTF-8 bytes in the fixed JCS member names, braces, separators, and `fields` object. Dynamic + * values are counted separately above. Keeping the literal here makes protocol changes visible. + */ +private val EVENT_JSON_PUNCTUATION_BYTES = ( + "{\"collector_id\":" + + ",\"fields\":{" + "}" + + ",\"observed_time\":{\"boot_session_id\":" + + ",\"monotonic_time_nanos\":" + + ",\"wall_time_utc_millis\":" + "}" + + ",\"payload_schema_version\":" + + ",\"payload_type\":" + + ",\"sequence_number\":" + "}" + ).toByteArray(Charsets.UTF_8).size + interface EventSink { fun captureToken(): AdmissionToken? @@ -119,6 +305,9 @@ interface CollectorPlugin { interface Collector { val health: StateFlow + /** True while the owner must keep this instance and call [stop] to release process resources. */ + val requiresStop: Boolean get() = false + suspend fun start() suspend fun pause() diff --git a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGate.kt b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGate.kt new file mode 100644 index 0000000..119ba50 --- /dev/null +++ b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGate.kt @@ -0,0 +1,84 @@ +package cool.linc.androiddatacollector.core.collector + +/** + * Bounds an event-driven source while retaining its newest meaningful value. + * + * Time is caller-supplied monotonic milliseconds, keeping this state machine independent of + * Android handlers and making its scheduling contract directly testable. + */ +class LatestValueRateGate( + private val minimumIntervalMillis: Long, + private val equivalent: (previous: T, current: T) -> Boolean = { previous, current -> previous == current }, +) { + init { require(minimumIntervalMillis > 0) } + + private var lastEmitted: T? = null + private var lastEmittedAtMillis = 0L + private var hasEmitted = false + private var pending: T? = null + + fun offer(value: T, elapsedMillis: Long): Decision { + val previous = lastEmitted + if (previous != null && equivalent(previous, value)) { + pending = null + return Decision.Suppress + } + if (!hasEmitted || elapsedMillis - lastEmittedAtMillis >= minimumIntervalMillis) { + pending = null + lastEmitted = value + lastEmittedAtMillis = elapsedMillis + hasEmitted = true + return Decision.Emit(value) + } + pending = value + return Decision.Defer(minimumIntervalMillis - (elapsedMillis - lastEmittedAtMillis)) + } + + fun poll(elapsedMillis: Long): Decision { + val value = pending ?: return Decision.Suppress + val previous = lastEmitted + if (previous != null && equivalent(previous, value)) { + pending = null + return Decision.Suppress + } + val remaining = minimumIntervalMillis - (elapsedMillis - lastEmittedAtMillis) + if (remaining > 0) return Decision.Defer(remaining) + pending = null + lastEmitted = value + lastEmittedAtMillis = elapsedMillis + hasEmitted = true + return Decision.Emit(value) + } + + /** + * Restores the durable rate watermark after process recreation. + * + * Coalesced events preserve capture time rather than their later publication time, so the + * durable event cannot reconstruct the exact prior deadline. Recovery therefore keeps the last + * value for deduplication but conservatively fences one full interval from + * [currentElapsedMillis]. Once this instance has a watermark, later resume calls are no-ops. + * A null [value] still restores the hard rate bound when an old payload cannot be decoded. + */ + fun restoreLastEmission( + value: T?, + currentElapsedMillis: Long, + ) { + require(currentElapsedMillis >= 0) { "Current elapsed time must be non-negative" } + if (hasEmitted) return + lastEmittedAtMillis = currentElapsedMillis + lastEmitted = value + hasEmitted = true + pending = null + } + + fun clearPending() { + pending = null + } + + sealed interface Decision { + data class Emit(val value: T) : Decision + data class Defer(val delayMillis: Long) : Decision + data object Suppress : Decision + } + +} diff --git a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt new file mode 100644 index 0000000..3713b70 --- /dev/null +++ b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt @@ -0,0 +1,568 @@ +// Generated by tools/catalog.py from protocol/v1/collector-catalog.json. Do not edit. +package cool.linc.androiddatacollector.core.collector + +object ProtocolEventContracts { + val contracts: Map = mapOf( + "accelerometer.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 2_048, + payloads = payloads( + listOf("ACCELEROMETER_SAMPLE") to EventPayloadContract( + fields = mapOf( + "accuracy" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + ), + "source_elapsed_realtime_nanos" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "x_meters_per_second_squared" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "y_meters_per_second_squared" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "z_meters_per_second_squared" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + ), + ), + ), + ), + "ambient_light.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 1_024, + payloads = payloads( + listOf("AMBIENT_LIGHT_SAMPLE") to EventPayloadContract( + fields = mapOf( + "accuracy" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + ), + "illuminance_lux" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + ), + "source_elapsed_realtime_nanos" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + ), + ), + ), + ), + "app_lifecycle.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 2_048, + payloads = payloads( + listOf("ACTIVITY_CREATED", "ACTIVITY_DESTROYED", "ACTIVITY_INSTANCE_STATE_SAVED", "ACTIVITY_PAUSED", "ACTIVITY_RESUMED", "ACTIVITY_STARTED", "ACTIVITY_STOPPED") to EventPayloadContract( + fields = mapOf( + "activity_class" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 512, + ), + ), + ), + ), + ), + "battery_state.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 1_024, + payloads = payloads( + listOf("BATTERY_STATE") to EventPayloadContract( + fields = mapOf( + "charging_source" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("AC", "DOCK", "MULTIPLE", "NONE", "UNKNOWN", "USB", "WIRELESS"), + ), + "charging_state" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("CHARGING", "DISCHARGING", "FULL", "NOT_CHARGING", "UNKNOWN"), + ), + "percentage" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + minimum = 0.0, + maximum = 100.0, + ), + "power_save_enabled" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + ), + ), + ), + ), + "gyroscope.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 2_048, + payloads = payloads( + listOf("GYROSCOPE_SAMPLE") to EventPayloadContract( + fields = mapOf( + "accuracy" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + ), + "source_elapsed_realtime_nanos" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "x_radians_per_second" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "y_radians_per_second" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "z_radians_per_second" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + ), + ), + ), + ), + "interventions.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 65_536, + payloads = payloads( + listOf("INTERVENTION_EXPIRED", "INTERVENTION_OPENED", "INTERVENTION_RESCHEDULED", "INTERVENTION_SCHEDULED", "NOTIFICATION_POSTED", "SURVEY_EXPIRED", "SURVEY_OPENED") to EventPayloadContract( + fields = mapOf( + "intervention_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + "occurrence_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + "scheduled_for_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "trigger_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + ), + ), + listOf("SURVEY_SUBMITTED") to EventPayloadContract( + fields = mapOf( + "answers_json" to EventFieldContract( + type = EventFieldType.JSON_STRING, + required = true, + maximumLength = 61440, + ), + "intervention_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + "occurrence_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + "opened_time" to EventFieldContract( + type = EventFieldType.JSON_STRING, + required = true, + ), + "scheduled_for_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "scheduled_time" to EventFieldContract( + type = EventFieldType.JSON_STRING, + required = true, + ), + "submitted_time" to EventFieldContract( + type = EventFieldType.JSON_STRING, + required = true, + ), + "survey_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + "trigger_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 64, + ), + ), + ), + ), + ), + "keyboard_touch.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 4_096, + payloads = payloads( + listOf("KEYBOARD_TOUCH") to EventPayloadContract( + fields = mapOf( + "action" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("CANCEL", "DOWN", "MOVE", "UP"), + ), + "down_uptime_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "event_uptime_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "geometry_version" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("qwerty-v1"), + ), + "key_category" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("BACKSPACE", "ENTER", "LETTER", "SPACE"), + ), + "orientation_radians" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "pointer_id" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + minimum = 0.0, + ), + "pressure" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "relative_x" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + maximum = 1.0, + ), + "relative_y" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + maximum = 1.0, + ), + "size" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + ), + "tool_type" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + ), + ), + ), + ), + ), + "location.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 4_096, + payloads = payloads( + listOf("LOCATION_FIX") to EventPayloadContract( + fields = mapOf( + "altitude_meters" to EventFieldContract( + type = EventFieldType.FLOAT64, + required = false, + ), + "bearing_accuracy_degrees" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = false, + minimum = 0.0, + ), + "bearing_degrees" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = false, + minimum = 0.0, + maximum = 360.0, + ), + "horizontal_accuracy_meters" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + ), + "latitude_degrees" to EventFieldContract( + type = EventFieldType.FLOAT64, + required = true, + minimum = -90.0, + maximum = 90.0, + ), + "longitude_degrees" to EventFieldContract( + type = EventFieldType.FLOAT64, + required = true, + minimum = -180.0, + maximum = 180.0, + ), + "mock" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "source_elapsed_realtime_nanos" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "source_time_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "speed_accuracy_meters_per_second" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = false, + minimum = 0.0, + ), + "speed_meters_per_second" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = false, + minimum = 0.0, + ), + "vertical_accuracy_meters" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = false, + minimum = 0.0, + ), + ), + ), + ), + ), + "network_state.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 4_096, + payloads = payloads( + listOf("NETWORK_AVAILABLE", "NETWORK_LOST") to EventPayloadContract( + fields = mapOf( + ), + ), + listOf("NETWORK_CAPABILITIES") to EventPayloadContract( + fields = mapOf( + "downstream_kbps" to EventFieldContract( + type = EventFieldType.INT32, + required = false, + minimum = 0.0, + ), + "ethernet" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "metered" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "mobile" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "roaming" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "upstream_kbps" to EventFieldContract( + type = EventFieldType.INT32, + required = false, + minimum = 0.0, + ), + "validated" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "vpn" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "wifi" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + ), + ), + listOf("NETWORK_SNAPSHOT") to EventPayloadContract( + fields = mapOf( + "connected" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "downstream_kbps" to EventFieldContract( + type = EventFieldType.INT32, + required = false, + minimum = 0.0, + ), + "ethernet" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "metered" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "mobile" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "roaming" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "upstream_kbps" to EventFieldContract( + type = EventFieldType.INT32, + required = false, + minimum = 0.0, + ), + "validated" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "vpn" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + "wifi" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = false, + ), + ), + ), + ), + ), + "network_usage.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 2_048, + payloads = payloads( + listOf("NETWORK_USAGE_AGGREGATE") to EventPayloadContract( + fields = mapOf( + "coverage_end_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "coverage_start_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "rx_bytes" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "rx_packets" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "transport" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("MOBILE", "WIFI"), + ), + "tx_bytes" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + "tx_packets" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + ), + ), + ), + ), + "proximity.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 1_024, + payloads = payloads( + listOf("PROXIMITY_SAMPLE") to EventPayloadContract( + fields = mapOf( + "distance_centimeters" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + ), + "maximum_range_centimeters" to EventFieldContract( + type = EventFieldType.FLOAT32, + required = true, + minimum = 0.0, + ), + "near" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "source_elapsed_realtime_nanos" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + ), + ), + ), + ), + "temporal_context.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 1_024, + payloads = payloads( + listOf("TEMPORAL_CONTEXT") to EventPayloadContract( + fields = mapOf( + "change_reason" to EventFieldContract( + type = EventFieldType.ENUM, + required = true, + enumValues = setOf("RECONCILED", "STUDY_STARTED", "TIMEZONE_CHANGED", "TIME_SET", "UTC_OFFSET_CHANGED"), + ), + "daylight_saving_time" to EventFieldContract( + type = EventFieldType.BOOLEAN, + required = true, + ), + "timezone_id" to EventFieldContract( + type = EventFieldType.STRING, + required = true, + maximumLength = 128, + ), + "utc_offset_seconds" to EventFieldContract( + type = EventFieldType.INT32, + required = true, + minimum = -64800.0, + maximum = 64800.0, + ), + ), + ), + ), + ), + "usage_events.v1" to CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 4_096, + payloads = payloads( + listOf("ACTIVITY_PAUSED", "ACTIVITY_RESUMED", "ACTIVITY_STOPPED", "DEVICE_SHUTDOWN", "DEVICE_STARTUP", "KEYGUARD_HIDDEN", "KEYGUARD_SHOWN", "SCREEN_INTERACTIVE", "SCREEN_NON_INTERACTIVE") to EventPayloadContract( + fields = mapOf( + "package_name" to EventFieldContract( + type = EventFieldType.STRING, + required = false, + maximumLength = 255, + ), + "source_time_utc_millis" to EventFieldContract( + type = EventFieldType.DECIMAL_STRING, + required = true, + ), + ), + ), + ), + ), + ) + + operator fun get(collectorId: String): CollectorEventContract? = contracts[collectorId] + + private fun payloads( + vararg groups: Pair, EventPayloadContract>, + ): Map = buildMap { + groups.forEach { (types, contract) -> + types.forEach { type -> check(put(type, contract) == null) { "Duplicate payload type" } } + } + } +} diff --git a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollector.kt b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollector.kt index 8e5fc43..92b8fb6 100644 --- a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollector.kt +++ b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollector.kt @@ -19,13 +19,17 @@ abstract class SerializedCallbackCollector( private val messages = Channel(queueCapacity) private val mutableHealth = MutableStateFlow(CollectorHealth(CollectorStatus.STOPPED)) private var consumerJob: Job? = null - private var sourceRegistered = false + private var sourceState = SourceState.RELEASED final override val health: StateFlow get() = mutableHealth.asStateFlow() + final override val requiresStop: Boolean + get() = consumerJob != null + final override suspend fun start() { check(consumerJob == null) { "Collector is already started" } + check(sourceState == SourceState.RELEASED) { "Collector source is not released" } check(mutableHealth.value.status in setOf(CollectorStatus.STOPPED, CollectorStatus.FAILED)) { "Collector cannot be started" } @@ -35,9 +39,7 @@ abstract class SerializedCallbackCollector( register() mutableHealth.value = CollectorHealth(CollectorStatus.ACTIVE) } catch (failure: Throwable) { - messages.send(Message.Stop) - job.join() - consumerJob = null + if (sourceState == SourceState.RELEASED) stopConsumer(job) fail("SOURCE_REGISTRATION_FAILED") throw failure } @@ -45,13 +47,14 @@ abstract class SerializedCallbackCollector( final override suspend fun pause() { checkNotNull(consumerJob) { "Collector is not started" } - try { - unregister() - } catch (failure: Throwable) { + val failure = runCatching { unregister() }.exceptionOrNull() + // unregisterSource owns physical teardown and must finish it before reporting failure. Drain + // every event admitted before that boundary even when Android reports a cleanup error. + flush() + if (failure != null) { fail("SOURCE_UNREGISTRATION_FAILED") throw failure } - flush() if (mutableHealth.value.status != CollectorStatus.FAILED) { mutableHealth.value = CollectorHealth(CollectorStatus.PAUSED) } @@ -73,13 +76,25 @@ abstract class SerializedCallbackCollector( final override suspend fun stop() { val job = consumerJob ?: return - val failure = runCatching { unregister() }.exceptionOrNull() + var failure = runCatching { unregister() }.exceptionOrNull() + if (sourceState == SourceState.UNCERTAIN) { + try { + flush() + } catch (flushFailure: Throwable) { + val first = failure + if (first == null) { + failure = flushFailure + } else if (first !== flushFailure) { + first.addSuppressed(flushFailure) + } + } + fail("SOURCE_UNREGISTRATION_FAILED") + throw checkNotNull(failure) { "Uncertain source teardown did not report a failure" } + } try { flush() } finally { - messages.send(Message.Stop) - job.join() - consumerJob = null + stopConsumer(job) } if (failure != null) { fail("SOURCE_UNREGISTRATION_FAILED") @@ -99,19 +114,45 @@ abstract class SerializedCallbackCollector( mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, reasonCode) } - protected abstract suspend fun registerSource() - protected abstract suspend fun unregisterSource() + protected abstract suspend fun registerSource(): SourceRegistrationResult + /** + * Returns only when a fresh source generation is safe. An exception means physical teardown is + * uncertain, so the base class deliberately keeps the logical registration and blocks resume. + */ + protected abstract suspend fun unregisterSource(): SourceTeardownResult private suspend fun register() { - check(!sourceRegistered) { "Collector source is already registered" } - registerSource() - sourceRegistered = true + check(sourceState == SourceState.RELEASED) { "Collector source is not released" } + when (val result = registerSource()) { + SourceRegistrationResult.Registered -> sourceState = SourceState.REGISTERED + is SourceRegistrationResult.Released -> throw result.failure + is SourceRegistrationResult.Uncertain -> { + sourceState = SourceState.UNCERTAIN + throw result.failure + } + } } private suspend fun unregister() { - if (!sourceRegistered) return - unregisterSource() - sourceRegistered = false + if (sourceState == SourceState.RELEASED) return + try { + when (val result = unregisterSource()) { + SourceTeardownResult.Released -> sourceState = SourceState.RELEASED + is SourceTeardownResult.ReleasedWithFailure -> { + sourceState = SourceState.RELEASED + throw result.failure + } + } + } catch (failure: Throwable) { + if (sourceState != SourceState.RELEASED) sourceState = SourceState.UNCERTAIN + throw failure + } + } + + private suspend fun stopConsumer(job: Job) { + messages.send(Message.Stop) + job.join() + consumerJob = null } private suspend fun flush() { @@ -131,7 +172,11 @@ abstract class SerializedCallbackCollector( } catch (_: Throwable) { EmitResult.StorageFailure } - if (result == EmitResult.StorageFailure) fail("STORAGE_WRITE_FAILED") + when (result) { + EmitResult.ContractViolation -> fail("EVENT_CONTRACT_VIOLATION") + EmitResult.StorageFailure -> fail("STORAGE_WRITE_FAILED") + else -> Unit + } } is Message.Barrier -> message.completion.complete(Unit) Message.Stop -> return @@ -149,4 +194,6 @@ abstract class SerializedCallbackCollector( data object Stop : Message } + + private enum class SourceState { RELEASED, REGISTERED, UNCERTAIN } } diff --git a/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycle.kt b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycle.kt new file mode 100644 index 0000000..3f48616 --- /dev/null +++ b/core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycle.kt @@ -0,0 +1,96 @@ +package cool.linc.androiddatacollector.core.collector + +import kotlinx.coroutines.NonCancellable +import kotlinx.coroutines.withContext + +/** + * Explicit outcome of source registration. + * + * A thrown exception before returning means registration made no externally visible change. + * Multi-step registrations must use [registerSourceWithRollback] so a failed rollback is reported + * as [SourceRegistrationResult.Uncertain] and the collector will refuse to register another source + * over it. + */ +sealed interface SourceRegistrationResult { + data object Registered : SourceRegistrationResult + + data class Released(val failure: Throwable) : SourceRegistrationResult + + data class Uncertain(val failure: Throwable) : SourceRegistrationResult +} + +/** + * Explicit outcome of source teardown. + * + * Returning either outcome promises that callbacks are physically released or independently + * isolated, so registering a fresh source generation is safe. Throwing instead leaves physical + * state uncertain; [SerializedCallbackCollector] then refuses to register over that source. + */ +sealed interface SourceTeardownResult { + data object Released : SourceTeardownResult + + /** Cleanup completed safely but still has a diagnostic failure for the caller to surface. */ + data class ReleasedWithFailure(val failure: Throwable) : SourceTeardownResult +} + +/** One callback generation, serialized against its registration and teardown boundaries. */ +class SourceCallbackBoundary { + private val lock = Any() + private var active = false + + fun activate(beforeFirstCallback: () -> Unit = {}) = synchronized(lock) { + check(!active) { "Source callback generation is already active" } + beforeFirstCallback() + active = true + } + + fun runIfActive(block: () -> Unit): Boolean = synchronized(lock) { + if (!active) return@synchronized false + block() + true + } + + fun deactivate(afterLastCallback: () -> Unit = {}) = synchronized(lock) { + if (!active) return@synchronized + active = false + afterLastCallback() + } +} + +/** Completes rollback in a non-cancellable context and reports whether release is proven. */ +suspend fun registerSourceWithRollback( + register: suspend () -> Unit, + rollback: suspend () -> Unit, +): SourceRegistrationResult = try { + register() + SourceRegistrationResult.Registered +} catch (failure: Throwable) { + try { + withContext(NonCancellable) { rollback() } + SourceRegistrationResult.Released(failure) + } catch (rollbackFailure: Throwable) { + if (rollbackFailure !== failure) failure.addSuppressed(rollbackFailure) + SourceRegistrationResult.Uncertain(failure) + } +} + +/** Runs every teardown operation before rethrowing the first failure with later failures attached. */ +suspend fun completeSourceTeardown(vararg operations: suspend () -> Unit) { + require(operations.isNotEmpty()) { "Source teardown needs at least one operation" } + var failure: Throwable? = null + withContext(NonCancellable) { + operations.forEach { operation -> + try { + operation() + } catch (next: Throwable) { + val first = failure + if (first == null) { + failure = next + } else if (first !== next) { + first.addSuppressed(next) + } + } + } + } + failure?.let { throw it } +} diff --git a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/EventFieldContractTest.kt b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/EventFieldContractTest.kt new file mode 100644 index 0000000..0a4b2cd --- /dev/null +++ b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/EventFieldContractTest.kt @@ -0,0 +1,32 @@ +package cool.linc.androiddatacollector.core.collector + +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +class EventFieldContractTest { + @Test + fun int32UsesOneCanonicalSignedDecimalSpellingAndDescriptorBounds() { + val percentage = EventFieldContract( + type = EventFieldType.INT32, + required = true, + minimum = 0.0, + maximum = 100.0, + ) + + listOf("0", "1", "100").forEach { assertTrue(it, percentage.accepts(it)) } + listOf("-1", "-0", "+1", "01", "101").forEach { assertFalse(it, percentage.accepts(it)) } + } + + @Test + fun floatsUseOnlyTheProtocolDecimalGrammar() { + val value = EventFieldContract(EventFieldType.FLOAT64, required = true) + + listOf("0", "-0", "+1", "01", ".5", "1.", "1e-3", "1E+3").forEach { + assertTrue(it, value.accepts(it)) + } + listOf("", " ", "0x10", "0b10", "NaN", "Infinity", "1_0").forEach { + assertFalse(it, value.accepts(it)) + } + } +} diff --git a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGateTest.kt b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGateTest.kt new file mode 100644 index 0000000..4bb139a --- /dev/null +++ b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/LatestValueRateGateTest.kt @@ -0,0 +1,79 @@ +package cool.linc.androiddatacollector.core.collector + +import org.junit.Assert.assertEquals +import org.junit.Test + +class LatestValueRateGateTest { + @Test + fun emitsFirstValueThenCoalescesToLatestAtDeadline() { + val gate = LatestValueRateGate(1_000) + + assertEquals(emit("first"), gate.offer("first", 10_000)) + assertEquals(defer(800), gate.offer("second", 10_200)) + assertEquals(defer(600), gate.offer("latest", 10_400)) + assertEquals(defer(1), gate.poll(10_999)) + assertEquals(emit("latest"), gate.poll(11_000)) + assertEquals(LatestValueRateGate.Decision.Suppress, gate.poll(12_000)) + } + + @Test + fun duplicateOfLastEmissionCancelsPendingValue() { + val gate = LatestValueRateGate(1_000) + + gate.offer("stable", 0) + gate.offer("changed", 100) + assertEquals(LatestValueRateGate.Decision.Suppress, gate.offer("stable", 200)) + assertEquals(LatestValueRateGate.Decision.Suppress, gate.poll(1_000)) + } + + @Test + fun callerSuppliedEquivalenceDefinesAMeaningfulChange() { + val gate = LatestValueRateGate(1_000) { previous, current -> + kotlin.math.abs(previous - current) < 5 + } + + assertEquals(emit(100), gate.offer(100, 0)) + assertEquals(LatestValueRateGate.Decision.Suppress, gate.offer(104, 1_000)) + assertEquals(emit(105), gate.offer(105, 1_000)) + } + + @Test + fun clearingPendingDoesNotRewriteTheEmissionWatermark() { + val gate = LatestValueRateGate(1_000) + + gate.offer("first", 5_000) + gate.offer("discarded", 5_100) + gate.clearPending() + assertEquals(LatestValueRateGate.Decision.Suppress, gate.poll(6_000)) + assertEquals(emit("next"), gate.offer("next", 6_000)) + } + + @Test + fun processRestartRestoresValueAndUsesAConservativeFullIntervalFence() { + val gate = LatestValueRateGate(1_000) + gate.restoreLastEmission( + value = "stable", + currentElapsedMillis = 10_200, + ) + + assertEquals(LatestValueRateGate.Decision.Suppress, gate.offer("stable", 10_200)) + assertEquals(defer(1_000), gate.offer("changed", 10_200)) + assertEquals(emit("changed"), gate.poll(11_200)) + } + + @Test + fun undecodableValueStillRestoresTheHardBoundAndResumeDoesNotMoveIt() { + val gate = LatestValueRateGate(1_000) + gate.restoreLastEmission( + value = null, + currentElapsedMillis = 200, + ) + gate.restoreLastEmission(value = "later-resume", currentElapsedMillis = 500) + + assertEquals(defer(1_000), gate.offer("first-after-reboot", 200)) + assertEquals(emit("first-after-reboot"), gate.poll(1_200)) + } + + private fun emit(value: T) = LatestValueRateGate.Decision.Emit(value) + private fun defer(delayMillis: Long) = LatestValueRateGate.Decision.Defer(delayMillis) +} diff --git a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventSizeTest.kt b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventSizeTest.kt new file mode 100644 index 0000000..61ddac2 --- /dev/null +++ b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventSizeTest.kt @@ -0,0 +1,42 @@ +package cool.linc.androiddatacollector.core.collector + +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.ResearchTime +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Test + +class ProtocolEventSizeTest { + @Test + fun countsTheExactCanonicalProtocolEventBytes() { + val event = EventDraft( + collectorId = "test.v1", + payloadSchemaVersion = 1, + observedTime = ResearchTime(12, 34, "boot-one"), + payloadType = "TEST_EVENT", + fields = mapOf("note" to "line\n雪"), + ) + val expected = ( + "{\"collector_id\":\"test.v1\",\"fields\":{\"note\":\"line\\n雪\"}," + + "\"observed_time\":{\"boot_session_id\":\"boot-one\"," + + "\"monotonic_time_nanos\":\"34\",\"wall_time_utc_millis\":\"12\"}," + + "\"payload_schema_version\":1,\"payload_type\":\"TEST_EVENT\"," + + "\"sequence_number\":\"99\"}" + ).toByteArray(Charsets.UTF_8).size + + assertEquals(expected, event.protocolEncodedBytes(99)) + } + + @Test + fun refusesTextThatCannotBeRepresentedAsUnicodeJcs() { + val event = EventDraft( + collectorId = "test.v1", + payloadSchemaVersion = 1, + observedTime = ResearchTime(12, 34, "boot-one"), + payloadType = "TEST_EVENT", + fields = mapOf("note" to "\uD800"), + ) + + assertThrows(IllegalArgumentException::class.java) { event.protocolEncodedBytes(1) } + } +} diff --git a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollectorTest.kt b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollectorTest.kt index f0e6370..4cdde0b 100644 --- a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollectorTest.kt +++ b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollectorTest.kt @@ -91,6 +91,90 @@ class SerializedCallbackCollectorTest { collector.stop() } + @Test + fun uncertainStopKeepsConsumerAliveUntilTeardownCanBeRetried() = runTest { + val sink = FakeSink() + val collector = TestCollector(context(sink), queueCapacity = 1) + collector.start() + collector.leaveNextUnregistrationUncertain = true + + val firstFailure = runCatching { collector.stop() }.exceptionOrNull() + assertTrue(firstFailure is IllegalStateException) + assertEquals(CollectorHealth(CollectorStatus.FAILED, "SOURCE_UNREGISTRATION_FAILED"), collector.health.value) + + collector.trigger() + collector.stop() + + assertEquals(1, collector.registerCount) + assertEquals(2, collector.unregisterCount) + assertEquals(1, sink.events.size) + assertEquals(CollectorStatus.STOPPED, collector.health.value.status) + } + + @Test + fun uncertainRegistrationBlocksAnotherGenerationUntilStopReleasesIt() = runTest { + val collector = TestCollector(context(FakeSink()), queueCapacity = 1) + collector.leaveNextRegistrationUncertain = true + + val startFailure = runCatching { collector.start() }.exceptionOrNull() + assertTrue(startFailure is IllegalStateException) + + val resumeFailure = runCatching { collector.resume() }.exceptionOrNull() + assertTrue(resumeFailure is IllegalStateException) + assertEquals(1, collector.registerCount) + + collector.stop() + assertEquals(1, collector.unregisterCount) + assertEquals(CollectorStatus.STOPPED, collector.health.value.status) + } + + @Test + fun failedPauseTeardownStillDrainsAndCanResumeWithAFreshSource() = runTest { + val sink = FakeSink() + val collector = TestCollector(context(sink), queueCapacity = 1) + collector.start() + collector.trigger() + collector.failNextUnregistration = true + + val failure = runCatching { collector.pause() }.exceptionOrNull() + + assertTrue(failure is IllegalStateException) + assertEquals(1, sink.events.size) + assertEquals(CollectorHealth(CollectorStatus.FAILED, "SOURCE_UNREGISTRATION_FAILED"), collector.health.value) + + collector.resume() + collector.trigger() + collector.stop() + + assertEquals(2, collector.registerCount) + assertEquals(2, collector.unregisterCount) + assertEquals(2, sink.events.size) + assertEquals(CollectorStatus.STOPPED, collector.health.value.status) + } + + @Test + fun uncertainPauseTeardownDrainsButRefusesToRegisterOverTheSource() = runTest { + val sink = FakeSink() + val collector = TestCollector(context(sink), queueCapacity = 1) + collector.start() + collector.trigger() + collector.leaveNextUnregistrationUncertain = true + + val pauseFailure = runCatching { collector.pause() }.exceptionOrNull() + + assertTrue(pauseFailure is IllegalStateException) + assertEquals(1, sink.events.size) + assertEquals(CollectorHealth(CollectorStatus.FAILED, "SOURCE_UNREGISTRATION_FAILED"), collector.health.value) + + val resumeFailure = runCatching { collector.resume() }.exceptionOrNull() + assertTrue(resumeFailure is IllegalStateException) + assertEquals(1, collector.registerCount) + + // A later teardown retry may establish a known released state before final shutdown. + collector.stop() + assertEquals(2, collector.unregisterCount) + } + private fun kotlinx.coroutines.test.TestScope.context(sink: FakeSink) = CollectorContext( scope = backgroundScope, eventSink = sink, @@ -107,27 +191,43 @@ class SerializedCallbackCollectorTest { var unregisterCount = 0 var draftConstructed = false var failNextRegistration = false + var leaveNextRegistrationUncertain = false var failNextUnregistration = false + var leaveNextUnregistrationUncertain = false fun trigger() = capture { draftConstructed = true EventDraft("test_collector.v1", 1, context.clocks.now(), "TEST", emptyMap()) } - override suspend fun registerSource() { + override suspend fun registerSource(): SourceRegistrationResult { registerCount += 1 + if (leaveNextRegistrationUncertain) { + leaveNextRegistrationUncertain = false + return SourceRegistrationResult.Uncertain( + IllegalStateException("Registration rollback state is uncertain"), + ) + } if (failNextRegistration) { failNextRegistration = false - error("Registration failed") + return SourceRegistrationResult.Released(IllegalStateException("Registration failed")) } + return SourceRegistrationResult.Registered } - override suspend fun unregisterSource() { + override suspend fun unregisterSource(): SourceTeardownResult { unregisterCount += 1 + if (leaveNextUnregistrationUncertain) { + leaveNextUnregistrationUncertain = false + error("Unregistration state is uncertain") + } if (failNextUnregistration) { failNextUnregistration = false - error("Unregistration failed") + return SourceTeardownResult.ReleasedWithFailure( + IllegalStateException("Unregistration reported a failure after release"), + ) } + return SourceTeardownResult.Released } } diff --git a/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycleTest.kt b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycleTest.kt new file mode 100644 index 0000000..52a88fb --- /dev/null +++ b/core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SourceLifecycleTest.kt @@ -0,0 +1,69 @@ +package cool.linc.androiddatacollector.core.collector + +import kotlinx.coroutines.test.runTest +import org.junit.Assert.assertEquals +import org.junit.Assert.assertSame +import org.junit.Test + +class SourceLifecycleTest { + @Test + fun teardownAttemptsEveryOperationAndPreservesFailureOrder() = runTest { + val calls = mutableListOf() + val first = IllegalStateException("first") + val second = IllegalArgumentException("second") + + val thrown = runCatching { + completeSourceTeardown( + { + calls += "first" + throw first + }, + { calls += "middle" }, + { + calls += "last" + throw second + }, + ) + }.exceptionOrNull() + + assertSame(first, thrown) + assertEquals(listOf(second), first.suppressed.toList()) + assertEquals(listOf("first", "middle", "last"), calls) + } + + @Test + fun failedRegistrationReportsUncertainWhenRollbackAlsoFails() = runTest { + val calls = mutableListOf() + val registration = IllegalStateException("registration") + val rollback = IllegalArgumentException("rollback") + + val result = registerSourceWithRollback( + register = { + calls += "register" + throw registration + }, + rollback = { + calls += "rollback" + throw rollback + }, + ) + + assertEquals(SourceRegistrationResult.Uncertain(registration), result) + assertEquals(1, registration.suppressed.size) + assertEquals(rollback::class.java, registration.suppressed.single()::class.java) + assertEquals(rollback.message, registration.suppressed.single().message) + assertEquals(listOf("register", "rollback"), calls) + } + + @Test + fun failedRegistrationReportsReleasedWhenRollbackCompletes() = runTest { + val registration = IllegalStateException("registration") + + val result = registerSourceWithRollback( + register = { throw registration }, + rollback = {}, + ) + + assertEquals(SourceRegistrationResult.Released(registration), result) + } +} diff --git a/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519Crypto.kt b/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519Crypto.kt new file mode 100644 index 0000000..fbc0427 --- /dev/null +++ b/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519Crypto.kt @@ -0,0 +1,25 @@ +package cool.linc.androiddatacollector.core.crypto + +import com.google.crypto.tink.subtle.Ed25519Verify +import java.security.GeneralSecurityException + +/** Provider-independent Ed25519 verification for Protocol v1 raw public keys. */ +object Ed25519Crypto { + fun verify( + publicKey: ByteArray, + message: ByteArray, + signature: ByteArray, + ): Boolean { + require(publicKey.size == PUBLIC_KEY_BYTES) { "Ed25519 public key must be 32 bytes" } + require(signature.size == SIGNATURE_BYTES) { "Ed25519 signature must be 64 bytes" } + return try { + Ed25519Verify(publicKey).verify(signature, message) + true + } catch (_: GeneralSecurityException) { + false + } + } + + const val PUBLIC_KEY_BYTES = 32 + const val SIGNATURE_BYTES = 64 +} diff --git a/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCrypto.kt b/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCrypto.kt index 8396448..a6b2292 100644 --- a/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCrypto.kt +++ b/core/crypto/src/main/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCrypto.kt @@ -1,102 +1,95 @@ package cool.linc.androiddatacollector.core.crypto +import com.google.crypto.tink.AccessesPartialKey import com.google.crypto.tink.HybridDecrypt import com.google.crypto.tink.HybridEncrypt import com.google.crypto.tink.InsecureSecretKeyAccess -import com.google.crypto.tink.KeyStatus -import com.google.crypto.tink.KeyTemplates import com.google.crypto.tink.KeysetHandle import com.google.crypto.tink.RegistryConfiguration -import com.google.crypto.tink.TinkJsonProtoKeysetFormat import com.google.crypto.tink.hybrid.HybridConfig import com.google.crypto.tink.hybrid.HpkeParameters import com.google.crypto.tink.hybrid.HpkePrivateKey import com.google.crypto.tink.hybrid.HpkePublicKey +import com.google.crypto.tink.subtle.X25519 +import com.google.crypto.tink.util.Bytes +import com.google.crypto.tink.util.SecretBytes -data class HpkeKeysetPair( - val privateKeysetJson: String, - val publicKeysetJson: String, +data class HpkeKeyPair( + val privateKey: ByteArray, + val publicKey: ByteArray, ) +/** RFC 9180 base-mode X25519/HKDF-SHA256/AES-256-GCM with raw, prefix-free keys. */ +@AccessesPartialKey object HpkeCrypto { - private const val TEMPLATE = "DHKEM_X25519_HKDF_SHA256_HKDF_SHA256_AES_256_GCM" + private val PARAMETERS = HpkeParameters.builder() + .setKemId(HpkeParameters.KemId.DHKEM_X25519_HKDF_SHA256) + .setKdfId(HpkeParameters.KdfId.HKDF_SHA256) + .setAeadId(HpkeParameters.AeadId.AES_256_GCM) + .setVariant(HpkeParameters.Variant.NO_PREFIX) + .build() init { HybridConfig.register() } - fun generateKeyset(): HpkeKeysetPair { - val privateHandle = KeysetHandle.generateNew(KeyTemplates.get(TEMPLATE)) - return HpkeKeysetPair( - privateKeysetJson = TinkJsonProtoKeysetFormat.serializeKeyset( - privateHandle, - InsecureSecretKeyAccess.get(), - ), - publicKeysetJson = TinkJsonProtoKeysetFormat.serializeKeysetWithoutSecret( - privateHandle.publicKeysetHandle, - ), - ) + fun generateKeyPair(): HpkeKeyPair { + val privateKey = X25519.generatePrivateKey() + return HpkeKeyPair(privateKey, X25519.publicFromPrivate(privateKey)) + } + + fun validatePublicKey(publicKey: ByteArray) { + publicHandle(publicKey) + .getPrimitive(RegistryConfiguration.get(), HybridEncrypt::class.java) + .encrypt(ByteArray(0), VALIDATION_CONTEXT) } fun encrypt( - publicKeysetJson: String, + publicKey: ByteArray, plaintext: ByteArray, contextInfo: ByteArray, - ): ByteArray { - val handle = TinkJsonProtoKeysetFormat.parseKeysetWithoutSecret(publicKeysetJson) - validatePublicHandle(handle) - val primitive = handle.getPrimitive(RegistryConfiguration.get(), HybridEncrypt::class.java) - return primitive.encrypt(plaintext, contextInfo) - } - - fun validatePublicKeyset(publicKeysetJson: String) { - val handle = TinkJsonProtoKeysetFormat.parseKeysetWithoutSecret(publicKeysetJson) - validatePublicHandle(handle) - handle.getPrimitive(RegistryConfiguration.get(), HybridEncrypt::class.java) - } + ): ByteArray = publicHandle(publicKey) + .getPrimitive(RegistryConfiguration.get(), HybridEncrypt::class.java) + .encrypt(plaintext, contextInfo) + .also { require(it.size == plaintext.size + ENCAPSULATED_KEY_BYTES + TAG_BYTES) { "Unexpected HPKE output size" } } fun decrypt( - privateKeysetJson: String, + privateKey: ByteArray, ciphertext: ByteArray, contextInfo: ByteArray, ): ByteArray { - val handle = TinkJsonProtoKeysetFormat.parseKeyset( - privateKeysetJson, - InsecureSecretKeyAccess.get(), - ) - validatePrivateHandle(handle) - val primitive = handle.getPrimitive(RegistryConfiguration.get(), HybridDecrypt::class.java) - return primitive.decrypt(ciphertext, contextInfo) + require(ciphertext.size >= ENCAPSULATED_KEY_BYTES + TAG_BYTES) { "Truncated HPKE ciphertext" } + return privateHandle(privateKey) + .getPrimitive(RegistryConfiguration.get(), HybridDecrypt::class.java) + .decrypt(ciphertext, contextInfo) } - private fun validatePublicHandle(handle: KeysetHandle) { - require(handle.size() == 1) { "HPKE public keyset must contain exactly one key" } - val entry = handle.getAt(0) - require(entry.isPrimary && entry.status == KeyStatus.ENABLED) { "HPKE public key must be primary and enabled" } - val key = entry.key as? HpkePublicKey - ?: throw IllegalArgumentException("Export key must be an HPKE public key") - validateParameters(key.parameters) + private fun publicHandle(raw: ByteArray): KeysetHandle { + require(raw.size == RAW_KEY_BYTES) { "X25519 public key must be 32 bytes" } + val key = HpkePublicKey.create(PARAMETERS, Bytes.copyFrom(raw), null) + return handle(key) } - private fun validatePrivateHandle(handle: KeysetHandle) { - require(handle.size() == 1) { "HPKE private keyset must contain exactly one key" } - val entry = handle.getAt(0) - require(entry.isPrimary && entry.status == KeyStatus.ENABLED) { "HPKE private key must be primary and enabled" } - val key = entry.key as? HpkePrivateKey - ?: throw IllegalArgumentException("Export decryption key must be an HPKE private key") - validateParameters(key.parameters) + private fun privateHandle(raw: ByteArray): KeysetHandle { + require(raw.size == RAW_KEY_BYTES) { "X25519 private key must be 32 bytes" } + val publicKey = HpkePublicKey.create( + PARAMETERS, + Bytes.copyFrom(X25519.publicFromPrivate(raw)), + null, + ) + val privateKey = HpkePrivateKey.create( + publicKey, + SecretBytes.copyFrom(raw, InsecureSecretKeyAccess.get()), + ) + return handle(privateKey) } - private fun validateParameters(parameters: HpkeParameters) { - require(parameters.kemId == HpkeParameters.KemId.DHKEM_X25519_HKDF_SHA256) { - "Export HPKE KEM must be X25519/HKDF-SHA256" - } - require(parameters.kdfId == HpkeParameters.KdfId.HKDF_SHA256) { - "Export HPKE KDF must be HKDF-SHA256" - } - require(parameters.aeadId == HpkeParameters.AeadId.AES_256_GCM) { - "Export HPKE AEAD must be AES-256-GCM" - } - require(parameters.variant == HpkeParameters.Variant.TINK) { "Export HPKE key must use the TINK variant" } - } + private fun handle(key: com.google.crypto.tink.Key): KeysetHandle = KeysetHandle.newBuilder() + .addEntry(KeysetHandle.importKey(key).withRandomId().makePrimary()) + .build() + + const val RAW_KEY_BYTES = 32 + const val ENCAPSULATED_KEY_BYTES = 32 + const val TAG_BYTES = 16 + private val VALIDATION_CONTEXT = "adc-hpke-public-key-validation".toByteArray(Charsets.US_ASCII) } diff --git a/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519CryptoTest.kt b/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519CryptoTest.kt new file mode 100644 index 0000000..97e2c93 --- /dev/null +++ b/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/Ed25519CryptoTest.kt @@ -0,0 +1,38 @@ +package cool.linc.androiddatacollector.core.crypto + +import java.security.KeyPairGenerator +import java.security.Signature +import org.junit.Assert.assertFalse +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Test + +class Ed25519CryptoTest { + @Test + fun verifiesRawPublicKeysWithoutAProviderSpecificKeyFactory() { + val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() + val publicKey = pair.public.encoded.copyOfRange( + pair.public.encoded.size - Ed25519Crypto.PUBLIC_KEY_BYTES, + pair.public.encoded.size, + ) + val message = "protocol-v1".toByteArray() + val signature = Signature.getInstance("Ed25519").run { + initSign(pair.private) + update(message) + sign() + } + + assertTrue(Ed25519Crypto.verify(publicKey, message, signature)) + assertFalse(Ed25519Crypto.verify(publicKey, message + 0, signature)) + } + + @Test + fun rejectsInvalidWireLengths() { + assertThrows(IllegalArgumentException::class.java) { + Ed25519Crypto.verify(ByteArray(31), ByteArray(0), ByteArray(64)) + } + assertThrows(IllegalArgumentException::class.java) { + Ed25519Crypto.verify(ByteArray(32), ByteArray(0), ByteArray(63)) + } + } +} diff --git a/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCryptoTest.kt b/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCryptoTest.kt index 889de27..ee2592b 100644 --- a/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCryptoTest.kt +++ b/core/crypto/src/test/kotlin/cool/linc/androiddatacollector/core/crypto/HpkeCryptoTest.kt @@ -1,47 +1,59 @@ package cool.linc.androiddatacollector.core.crypto -import com.google.crypto.tink.KeysetHandle -import com.google.crypto.tink.TinkJsonProtoKeysetFormat -import com.google.crypto.tink.hybrid.HybridConfig -import com.google.crypto.tink.hybrid.HpkeParameters import java.security.GeneralSecurityException import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertThrows import org.junit.Test class HpkeCryptoTest { @Test - fun generatedHpkeKeysetEncryptsOnlyForMatchingPrivateKeyAndContext() { - val keys = HpkeCrypto.generateKeyset() - val plaintext = "research content key".toByteArray() - val context = "experiment:config:key".toByteArray() + fun rawNoPrefixHpkeRoundTripsWithTheExactProtocolSuite() { + val keys = HpkeCrypto.generateKeyPair() + val plaintext = ByteArray(32) { it.toByte() } + val context = "adc protocol context".toByteArray() - HpkeCrypto.validatePublicKeyset(keys.publicKeysetJson) - val ciphertext = HpkeCrypto.encrypt(keys.publicKeysetJson, plaintext, context) + HpkeCrypto.validatePublicKey(keys.publicKey) + val ciphertext = HpkeCrypto.encrypt(keys.publicKey, plaintext, context) + + assertEquals(32, keys.privateKey.size) + assertEquals(32, keys.publicKey.size) + assertEquals(80, ciphertext.size) + assertArrayEquals(plaintext, HpkeCrypto.decrypt(keys.privateKey, ciphertext, context)) + } + + @Test + fun hpkeFailsClosedForWrongKeyContextAndCiphertext() { + val keys = HpkeCrypto.generateKeyPair() + val other = HpkeCrypto.generateKeyPair() + val context = "correct".toByteArray() + val ciphertext = HpkeCrypto.encrypt(keys.publicKey, ByteArray(32) { 7 }, context) - assertArrayEquals(plaintext, HpkeCrypto.decrypt(keys.privateKeysetJson, ciphertext, context)) assertThrows(GeneralSecurityException::class.java) { - HpkeCrypto.decrypt(keys.privateKeysetJson, ciphertext, "wrong-context".toByteArray()) + HpkeCrypto.decrypt(keys.privateKey, ciphertext, "wrong".toByteArray()) + } + assertThrows(GeneralSecurityException::class.java) { + HpkeCrypto.decrypt(other.privateKey, ciphertext, context) + } + assertThrows(GeneralSecurityException::class.java) { + HpkeCrypto.decrypt(keys.privateKey, ciphertext.copyOf().also { it[it.lastIndex]++ }, context) } } @Test - fun publicKeyValidationRejectsAValidHpkeKeyWithTheWrongAeadSuite() { - HybridConfig.register() - val privateHandle = KeysetHandle.generateNew( - HpkeParameters.builder() - .setKemId(HpkeParameters.KemId.DHKEM_X25519_HKDF_SHA256) - .setKdfId(HpkeParameters.KdfId.HKDF_SHA256) - .setAeadId(HpkeParameters.AeadId.AES_128_GCM) - .setVariant(HpkeParameters.Variant.TINK) - .build(), - ) - val publicJson = TinkJsonProtoKeysetFormat.serializeKeysetWithoutSecret( - privateHandle.publicKeysetHandle, - ) - + fun rejectsLegacyKeysetsAndInvalidRawLengths() { + val legacyJson = "{\"primaryKeyId\":123,\"key\":[]}".toByteArray() + assertFalse(legacyJson.size == HpkeCrypto.RAW_KEY_BYTES) + assertThrows(IllegalArgumentException::class.java) { HpkeCrypto.validatePublicKey(legacyJson) } + assertThrows(IllegalArgumentException::class.java) { + HpkeCrypto.encrypt(ByteArray(31), ByteArray(32), ByteArray(0)) + } assertThrows(IllegalArgumentException::class.java) { - HpkeCrypto.validatePublicKeyset(publicJson) + HpkeCrypto.decrypt(ByteArray(33), ByteArray(80), ByteArray(0)) + } + assertThrows(GeneralSecurityException::class.java) { + HpkeCrypto.validatePublicKey(ByteArray(32)) } } } diff --git a/core/experiment-runtime/src/main/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntime.kt b/core/experiment-runtime/src/main/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntime.kt index 9fe04c0..3d00000 100644 --- a/core/experiment-runtime/src/main/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntime.kt +++ b/core/experiment-runtime/src/main/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntime.kt @@ -4,6 +4,7 @@ import cool.linc.androiddatacollector.core.collector.AccessKind import cool.linc.androiddatacollector.core.collector.AdmissionToken import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext +import cool.linc.androiddatacollector.core.collector.CollectorDescriptor import cool.linc.androiddatacollector.core.collector.CollectorHealth import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.CollectorRegistry @@ -11,6 +12,7 @@ import cool.linc.androiddatacollector.core.collector.CollectorStatus import cool.linc.androiddatacollector.core.collector.EmitResult import cool.linc.androiddatacollector.core.collector.EventSink import cool.linc.androiddatacollector.core.collector.ResearchClocks +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.definition.InterventionAction import cool.linc.androiddatacollector.core.definition.MultipleChoiceQuestion @@ -57,6 +59,33 @@ data class OccurrenceDispatch( val action: InterventionAction, ) +/** Atomic result of claiming one occurrence at its immutable scheduled wall instant. */ +sealed interface OccurrenceClaimResult { + data class Due(val dispatch: OccurrenceDispatch) : OccurrenceClaimResult + data class NotDue(val remainingDelayMillis: Long) : OccurrenceClaimResult { + init { + require(remainingDelayMillis > 0) { "An early delivery must retain a positive delay" } + } + } + data object Expired : OccurrenceClaimResult + data object Terminal : OccurrenceClaimResult + data object Missing : OccurrenceClaimResult + data object InactiveStudy : OccurrenceClaimResult +} + +/** Atomic result of checking one durable occurrence against its signed expiry instant. */ +sealed interface OccurrenceExpiryResult { + data object Expired : OccurrenceExpiryResult + data class NotDue(val remainingDelayMillis: Long) : OccurrenceExpiryResult { + init { + require(remainingDelayMillis > 0) { "An early expiry must retain a positive delay" } + } + } + data object Terminal : OccurrenceExpiryResult + data object Missing : OccurrenceExpiryResult + data object InactiveStudy : OccurrenceExpiryResult +} + sealed interface SurveyAnswer { data class Text(val value: String) : SurveyAnswer data class Integer(val value: Int) : SurveyAnswer @@ -235,15 +264,20 @@ class ExperimentRuntime( planned } - suspend fun claimOccurrence(occurrenceId: String): OccurrenceDispatch? = metadataMutex.withLock { + suspend fun claimOccurrenceIfDue(occurrenceId: String): OccurrenceClaimResult = metadataMutex.withLock { val metadata = requireMetadata() - val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock null + if (metadata.state != ExperimentState.RUNNING) return@withLock OccurrenceClaimResult.InactiveStudy + val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock OccurrenceClaimResult.Missing + if (occurrence.state !in setOf(OccurrenceState.SCHEDULED, OccurrenceState.POSTING)) { + return@withLock OccurrenceClaimResult.Terminal + } val now = clocks.now() if (now.wallTimeUtcMillis >= occurrence.expiresAtUtcMillis) { expireOccurrence(metadata, occurrence, now) - return@withLock null + return@withLock OccurrenceClaimResult.Expired } - if (occurrence.state !in setOf(OccurrenceState.SCHEDULED, OccurrenceState.POSTING)) return@withLock null + val remaining = occurrence.scheduledFor.wallTimeUtcMillis - now.wallTimeUtcMillis + if (remaining > 0) return@withLock OccurrenceClaimResult.NotDue(remaining) val claimed = if (occurrence.state == OccurrenceState.SCHEDULED) { occurrence.copy(state = OccurrenceState.POSTING).also { next -> val updated = metadata.copy(occurrences = metadata.occurrences + (occurrenceId to next)) @@ -254,24 +288,60 @@ class ExperimentRuntime( } else { occurrence } - OccurrenceDispatch(claimed, intervention(claimed).action) + OccurrenceClaimResult.Due(OccurrenceDispatch(claimed, intervention(claimed).action)) + } + + /** + * Expires one occurrence without ever claiming delivery. + * + * WorkManager may wake early after a wall-clock change. Returning [OccurrenceExpiryResult.NotDue] + * leaves the durable lifecycle untouched so the adapter can retry instead of consuming the only + * expiry job. Notification-only occurrences that were already opened are terminal; surveys stay + * open until submitted or expired. + */ + suspend fun expireOccurrenceIfDue(occurrenceId: String): OccurrenceExpiryResult = metadataMutex.withLock { + val metadata = requireMetadata() + if (metadata.state != ExperimentState.RUNNING) return@withLock OccurrenceExpiryResult.InactiveStudy + val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock OccurrenceExpiryResult.Missing + if ( + occurrence.state in setOf(OccurrenceState.EXPIRED, OccurrenceState.SURVEY_SUBMITTED) || + (occurrence.state == OccurrenceState.OPENED && intervention(occurrence).action !is SurveyAction) + ) { + return@withLock OccurrenceExpiryResult.Terminal + } + val now = clocks.now() + val remaining = occurrence.expiresAtUtcMillis - now.wallTimeUtcMillis + if (remaining > 0) return@withLock OccurrenceExpiryResult.NotDue(remaining) + expireOccurrence(metadata, occurrence, now) + OccurrenceExpiryResult.Expired } - suspend fun markNotificationPosted(occurrenceId: String) = metadataMutex.withLock { + suspend fun markNotificationPosted(occurrenceId: String): Boolean = metadataMutex.withLock { val metadata = requireMetadata() - val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock - if (occurrence.state != OccurrenceState.POSTING) return@withLock + if (metadata.state != ExperimentState.RUNNING) return@withLock false + val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock false + if (occurrence.state !in setOf(OccurrenceState.POSTING, OccurrenceState.NOTIFICATION_POSTED)) { + return@withLock false + } + val now = clocks.now() + if (now.wallTimeUtcMillis >= occurrence.expiresAtUtcMillis) { + expireOccurrence(metadata, occurrence, now) + return@withLock false + } + if (occurrence.state == OccurrenceState.NOTIFICATION_POSTED) return@withLock true val posted = occurrence.copy(state = OccurrenceState.NOTIFICATION_POSTED) appendOccurrenceEvent( metadata.copy(occurrences = metadata.occurrences + (occurrenceId to posted)), posted, "NOTIFICATION_POSTED", - clocks.now(), + now, ) + true } suspend fun openOccurrence(occurrenceId: String): OccurrenceDispatch? = metadataMutex.withLock { val metadata = requireMetadata() + if (metadata.state != ExperimentState.RUNNING) return@withLock null val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock null val now = clocks.now() if (now.wallTimeUtcMillis >= occurrence.expiresAtUtcMillis && occurrence.state != OccurrenceState.SURVEY_SUBMITTED) { @@ -299,6 +369,7 @@ class ExperimentRuntime( answers: Map, ): SurveySubmissionResult = metadataMutex.withLock { val metadata = requireMetadata() + if (metadata.state != ExperimentState.RUNNING) return@withLock SurveySubmissionResult.INVALID val occurrence = metadata.occurrences[occurrenceId] ?: return@withLock SurveySubmissionResult.INVALID if (occurrence.state == OccurrenceState.SURVEY_SUBMITTED) return@withLock SurveySubmissionResult.ALREADY_SUBMITTED val now = clocks.now() @@ -408,6 +479,9 @@ class ExperimentRuntime( token: AdmissionToken, event: EventDraft, ): EmitResult { + val descriptor = collectorEntries[event.collectorId]?.plugin?.descriptor + ?: return EmitResult.ContractViolation + if (!descriptor.eventContract.accepts(event, Long.MAX_VALUE)) return EmitResult.ContractViolation if (!admissionGate.accepts(token, event.observedTime.elapsedRealtimeNanos)) { return EmitResult.RejectedByAdmissionGate } @@ -479,7 +553,7 @@ class ExperimentRuntime( configuration, CollectorContext( scope = scope, - eventSink = this, + eventSink = CollectorEventSink(plugin.descriptor), clocks = clocks, ), ) @@ -519,6 +593,7 @@ class ExperimentRuntime( entry.hasStarted = true } } catch (failure: Throwable) { + entry.hasStarted = entry.hasStarted || entry.collector.requiresStop failure.rethrowIfCancellation() updateCollectorHealth(id, CollectorHealth(CollectorStatus.FAILED, "COLLECTOR_START_FAILED")) } @@ -546,7 +621,7 @@ class ExperimentRuntime( failure.rethrowIfCancellation() updateCollectorHealth(id, CollectorHealth(CollectorStatus.FAILED, "COLLECTOR_STOP_FAILED")) } finally { - entry.hasStarted = false + entry.hasStarted = entry.collector.requiresStop } } } @@ -631,9 +706,8 @@ class ExperimentRuntime( observedAt: cool.linc.androiddatacollector.core.model.ResearchTime, additionalFields: Map = emptyMap(), ) { - val event = RecordedEvent( - sequenceNumber = metadataAfterState.nextSequenceNumber, - collectorId = "interventions.v1", + val draft = EventDraft( + collectorId = INTERVENTION_COLLECTOR_ID, payloadSchemaVersion = 1, observedTime = observedAt, payloadType = payloadType, @@ -644,6 +718,18 @@ class ExperimentRuntime( "scheduled_for_utc_millis" to occurrence.scheduledFor.wallTimeUtcMillis.toString(), ) + additionalFields, ) + check(requireNotNull(ProtocolEventContracts[INTERVENTION_COLLECTOR_ID]).accepts( + draft, + metadataAfterState.nextSequenceNumber, + )) { "Runtime intervention event violates Protocol v1" } + val event = RecordedEvent( + sequenceNumber = metadataAfterState.nextSequenceNumber, + collectorId = draft.collectorId, + payloadSchemaVersion = draft.payloadSchemaVersion, + observedTime = draft.observedTime, + payloadType = draft.payloadType, + fields = draft.fields.toSortedMap(), + ) val updated = metadataAfterState.copy( eventCount = event.sequenceNumber, nextSequenceNumber = event.sequenceNumber + 1, @@ -726,6 +812,25 @@ class ExperimentRuntime( var hasStarted: Boolean = false, ) + /** Binds a collector's shared admission capability to its own declared event contract. */ + private inner class CollectorEventSink( + private val descriptor: CollectorDescriptor, + ) : EventSink { + override fun captureToken(): AdmissionToken? = this@ExperimentRuntime.captureToken() + + override suspend fun emit(token: AdmissionToken, event: EventDraft): EmitResult = + if (event.collectorId != descriptor.id) { + EmitResult.ContractViolation + } else { + this@ExperimentRuntime.emit(token, event) + } + + override suspend fun latestEvent(collectorId: String): RecordedEvent? { + require(collectorId == descriptor.id) { "Collector cannot inspect another collector's event" } + return this@ExperimentRuntime.latestEvent(collectorId) + } + } + private companion object { val EXPORTABLE_STATES = setOf( ExperimentState.RUNNING, @@ -745,6 +850,7 @@ class ExperimentRuntime( const val INCIDENT_STORAGE_WRITE_FAILED = "STORAGE_WRITE_FAILED" const val INCIDENT_PAUSE_PERSISTENCE_FAILED = "PAUSE_PERSISTENCE_FAILED" const val MAXIMUM_SURVEY_ANSWERS_BYTES = 60 * 1024 + const val INTERVENTION_COLLECTOR_ID = "interventions.v1" } } diff --git a/core/experiment-runtime/src/test/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntimeTest.kt b/core/experiment-runtime/src/test/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntimeTest.kt index 3735e97..b86967a 100644 --- a/core/experiment-runtime/src/test/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntimeTest.kt +++ b/core/experiment-runtime/src/test/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntimeTest.kt @@ -9,10 +9,15 @@ import cool.linc.androiddatacollector.core.model.ResearchTime import cool.linc.androiddatacollector.core.model.StorageUsage import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.collector.AccessKind import cool.linc.androiddatacollector.core.collector.AccessRequirement import cool.linc.androiddatacollector.core.collector.Collector import cool.linc.androiddatacollector.core.collector.CollectorContext import cool.linc.androiddatacollector.core.collector.CollectorDescriptor +import cool.linc.androiddatacollector.core.collector.CollectorEventContract +import cool.linc.androiddatacollector.core.collector.EventFieldContract +import cool.linc.androiddatacollector.core.collector.EventFieldType +import cool.linc.androiddatacollector.core.collector.EventPayloadContract import cool.linc.androiddatacollector.core.collector.CollectorHealth import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.CollectorRegistry @@ -45,12 +50,72 @@ import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.StateFlow import kotlinx.coroutines.test.runTest import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertNull import org.junit.Assert.assertTrue import org.junit.Test @OptIn(ExperimentalCoroutinesApi::class) class ExperimentRuntimeTest { + @Test + fun requiredMissingHardwareBlocksEnrollment() = runTest { + val clocks = FakeClocks() + val plugin = FakeCollectorPlugin( + clocks, + AccessRequirement(AccessKind.GYROSCOPE_HARDWARE, required = true), + ) + val runtime = ExperimentRuntime( + configuration = configuration(), + store = InMemoryStudyStore(), + collectorRegistry = CollectorRegistry(listOf(plugin)), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + + assertEquals(CommandResult.Success, runtime.initialize()) + assertEquals(CommandResult.Success, runtime.reviewStudy()) + assertEquals(CommandResult.Success, runtime.acceptConsent()) + assertEquals(CommandResult.Failed("COMMAND_REJECTED"), runtime.completeAccessSetup(emptySet())) + assertEquals(ExperimentState.ACCESS_SETUP, runtime.snapshot.value.metadata?.state) + assertEquals(0, plugin.collector.startCount) + } + + @Test + fun optionalMissingHardwareBlocksOnlyItsCollectorAndStartsWhenAccessAppears() = runTest { + val clocks = FakeClocks() + val plugin = FakeCollectorPlugin( + clocks, + AccessRequirement(AccessKind.GYROSCOPE_HARDWARE, required = false), + ) + var available = emptySet() + val runtime = ExperimentRuntime( + configuration = configuration(), + store = InMemoryStudyStore(), + collectorRegistry = CollectorRegistry(listOf(plugin)), + clocks = clocks, + scope = backgroundScope, + availableAccess = { available }, + ) + + assertEquals(CommandResult.Success, runtime.initialize()) + assertEquals(CommandResult.Success, runtime.reviewStudy()) + assertEquals(CommandResult.Success, runtime.acceptConsent()) + assertEquals(CommandResult.Success, runtime.completeAccessSetup(emptySet())) + assertEquals(CommandResult.Success, runtime.start()) + assertEquals(0, plugin.collector.startCount) + assertEquals( + CollectorHealth(CollectorStatus.BLOCKED_ACCESS, "ACCESS_UNAVAILABLE"), + runtime.snapshot.value.collectorHealth[AppLifecycleConfiguration.ID], + ) + + assertEquals(CommandResult.Success, runtime.pause()) + available = setOf(AccessKind.GYROSCOPE_HARDWARE) + assertEquals(CommandResult.Success, runtime.resume()) + assertEquals(1, plugin.collector.startCount) + assertEquals(CollectorStatus.ACTIVE, plugin.collector.health.value.status) + } + @Test fun participantCommandsGateAndPersistCollectorEvents() = runTest { val store = InMemoryStudyStore() @@ -122,6 +187,82 @@ class ExperimentRuntimeTest { assertTrue(plugin.emit("ACTIVITY_RESUMED") is EmitResult.Accepted) } + @Test + fun failedInitialStartRetainsCollectorOwnershipUntilShutdownReleasesIt() = runTest { + val clocks = FakeClocks() + val plugin = FakeCollectorPlugin(clocks) + val runtime = ExperimentRuntime( + configuration = configuration(), + store = InMemoryStudyStore(), + collectorRegistry = CollectorRegistry(listOf(plugin)), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + assertEquals(CommandResult.Success, runtime.initialize()) + plugin.collector.failNextStartWithOwnedResources = true + assertEquals(CommandResult.Success, runtime.reviewStudy()) + assertEquals(CommandResult.Success, runtime.acceptConsent()) + assertEquals(CommandResult.Success, runtime.completeAccessSetup(emptySet())) + + assertEquals(CommandResult.Success, runtime.start()) + assertTrue(plugin.collector.requiresStop) + assertEquals(CollectorStatus.FAILED, runtime.snapshot.value.collectorHealth[AppLifecycleConfiguration.ID]?.status) + + runtime.shutdown() + assertEquals(1, plugin.collector.stopCount) + assertFalse(plugin.collector.requiresStop) + } + + @Test + fun failedTerminalStopRemainsOwnedAndShutdownRetriesIt() = runTest { + val clocks = FakeClocks() + val plugin = FakeCollectorPlugin(clocks) + val runtime = ExperimentRuntime( + configuration = configuration(), + store = InMemoryStudyStore(), + collectorRegistry = CollectorRegistry(listOf(plugin)), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + plugin.collector.failNextStopWithOwnedResources = true + + assertEquals(CommandResult.Success, runtime.finishEarly()) + assertEquals(1, plugin.collector.stopCount) + assertTrue(plugin.collector.requiresStop) + + runtime.shutdown() + assertEquals(2, plugin.collector.stopCount) + assertFalse(plugin.collector.requiresStop) + } + + @Test + fun collectorEventContractIsEnforcedBeforePersistence() = runTest { + val store = InMemoryStudyStore() + val clocks = FakeClocks() + val plugin = FakeCollectorPlugin(clocks) + val runtime = ExperimentRuntime( + configuration = configuration(), + store = store, + collectorRegistry = CollectorRegistry(listOf(plugin)), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + + assertEquals(EmitResult.ContractViolation, plugin.emit("ACTIVITY_RESUMED", schemaVersion = 2)) + assertEquals(EmitResult.ContractViolation, plugin.emit("ACTIVITY_RESUMED", collectorId = "other.v1")) + assertEquals( + EmitResult.ContractViolation, + plugin.emit("ACTIVITY_RESUMED", fields = mapOf("source" to "x".repeat(2_000))), + ) + assertTrue(store.events.isEmpty()) + assertEquals(0L, runtime.snapshot.value.metadata?.eventCount) + } + @Test fun surveySubmissionValidatesEveryQuestionTypeAndCommitsExactlyOnce() = runTest { val store = InMemoryStudyStore() @@ -145,7 +286,8 @@ class ExperimentRuntimeTest { state = OccurrenceState.SCHEDULED, ) runtime.ensureOccurrence(occurrence) - assertTrue(runtime.claimOccurrence(occurrence.occurrenceId)?.action is SurveyAction) + val claim = runtime.claimOccurrenceIfDue(occurrence.occurrenceId) as OccurrenceClaimResult.Due + assertTrue(claim.dispatch.action is SurveyAction) runtime.markNotificationPosted(occurrence.occurrenceId) assertEquals(OccurrenceState.OPENED, runtime.openOccurrence(occurrence.occurrenceId)?.occurrence?.state) @@ -198,12 +340,165 @@ class ExperimentRuntimeTest { state = OccurrenceState.SCHEDULED, ) runtime.ensureOccurrence(occurrence) - assertNull(runtime.claimOccurrence(occurrence.occurrenceId)) + assertEquals(OccurrenceClaimResult.Expired, runtime.claimOccurrenceIfDue(occurrence.occurrenceId)) assertNull(runtime.openOccurrence(occurrence.occurrenceId)) assertEquals(OccurrenceState.EXPIRED, runtime.snapshot.value.metadata?.occurrences?.get(occurrence.occurrenceId)?.state) assertEquals(listOf("INTERVENTION_SCHEDULED", "SURVEY_EXPIRED"), store.events.map { it.payloadType }) } + @Test + fun dedicatedExpiryCheckLeavesEarlyScheduledWorkUntouchedAndExpiresOnce() = runTest { + val store = InMemoryStudyStore() + val clocks = MutableClocks(1_000) + val runtime = ExperimentRuntime( + configuration = configuration(surveys = listOf(survey()), interventions = listOf(surveyIntervention())), + store = store, + collectorRegistry = CollectorRegistry(listOf(FakeCollectorPlugin(clocks))), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + val occurrence = surveyOccurrence("c", expiresAtUtcMillis = 5_000) + runtime.ensureOccurrence(occurrence) + + clocks.wallTimeUtcMillis = 4_250 + assertEquals(OccurrenceExpiryResult.NotDue(750), runtime.expireOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals( + OccurrenceState.SCHEDULED, + runtime.snapshot.value.metadata?.occurrences?.get(occurrence.occurrenceId)?.state, + ) + + clocks.wallTimeUtcMillis = 5_000 + assertEquals(OccurrenceExpiryResult.Expired, runtime.expireOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals(OccurrenceExpiryResult.Terminal, runtime.expireOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals(OccurrenceClaimResult.Terminal, runtime.claimOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals( + 1, + store.events.count { it.payloadType == "SURVEY_EXPIRED" && it.fields["occurrence_id"] == occurrence.occurrenceId }, + ) + } + + @Test + fun deliveryClaimWaitsForItsWallInstantAndARecoveredPostingClaimIsIdempotent() = runTest { + val clocks = MutableClocks(1_000) + val runtime = ExperimentRuntime( + configuration = configuration(surveys = listOf(survey()), interventions = listOf(surveyIntervention())), + store = InMemoryStudyStore(), + collectorRegistry = CollectorRegistry(listOf(FakeCollectorPlugin(clocks))), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + val occurrence = surveyOccurrence("9", scheduledAtUtcMillis = 3_000, expiresAtUtcMillis = 5_000) + runtime.ensureOccurrence(occurrence) + + clocks.wallTimeUtcMillis = 2_500 + assertEquals(OccurrenceClaimResult.NotDue(500), runtime.claimOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals( + OccurrenceState.SCHEDULED, + runtime.snapshot.value.metadata?.occurrences?.get(occurrence.occurrenceId)?.state, + ) + + clocks.wallTimeUtcMillis = 3_000 + val first = runtime.claimOccurrenceIfDue(occurrence.occurrenceId) as OccurrenceClaimResult.Due + val recovered = runtime.claimOccurrenceIfDue(occurrence.occurrenceId) as OccurrenceClaimResult.Due + assertEquals(OccurrenceState.POSTING, first.dispatch.occurrence.state) + assertEquals(first, recovered) + assertTrue(runtime.markNotificationPosted(occurrence.occurrenceId)) + assertTrue(runtime.markNotificationPosted(occurrence.occurrenceId)) + + clocks.wallTimeUtcMillis = 5_000 + assertFalse(runtime.markNotificationPosted(occurrence.occurrenceId)) + assertEquals( + OccurrenceState.EXPIRED, + runtime.snapshot.value.metadata?.occurrences?.get(occurrence.occurrenceId)?.state, + ) + } + + @Test + fun dedicatedExpiryCheckExpiresPostingPostedAndOpenedSurveyStates() = runTest { + val store = InMemoryStudyStore() + val clocks = MutableClocks(1_000) + val runtime = ExperimentRuntime( + configuration = configuration(surveys = listOf(survey()), interventions = listOf(surveyIntervention())), + store = store, + collectorRegistry = CollectorRegistry(listOf(FakeCollectorPlugin(clocks))), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + val occurrences = listOf("d", "e", "f").mapIndexed { index, prefix -> + surveyOccurrence(prefix, expiresAtUtcMillis = 5_000 + index * 1_000L).also { + runtime.ensureOccurrence(it) + } + } + runtime.claimOccurrenceIfDue(occurrences[0].occurrenceId) + runtime.claimOccurrenceIfDue(occurrences[1].occurrenceId) + runtime.markNotificationPosted(occurrences[1].occurrenceId) + runtime.claimOccurrenceIfDue(occurrences[2].occurrenceId) + runtime.markNotificationPosted(occurrences[2].occurrenceId) + runtime.openOccurrence(occurrences[2].occurrenceId) + + clocks.wallTimeUtcMillis = 10_000 + occurrences.forEach { occurrence -> + assertEquals(OccurrenceExpiryResult.Expired, runtime.expireOccurrenceIfDue(occurrence.occurrenceId)) + assertEquals( + OccurrenceState.EXPIRED, + runtime.snapshot.value.metadata?.occurrences?.get(occurrence.occurrenceId)?.state, + ) + } + assertEquals(3, store.events.count { it.payloadType == "SURVEY_EXPIRED" }) + assertEquals(OccurrenceExpiryResult.Missing, runtime.expireOccurrenceIfDue("0".repeat(64))) + } + + @Test + fun pausedFinishedAndWithdrawnStudiesRejectEveryInterventionMutation() = runTest { + val lifecycleCases = listOf CommandResult>>( + "pause" to { it.pause() }, + "finish" to { it.finishEarly() }, + "withdraw" to { it.withdraw() }, + ) + lifecycleCases.forEach { (name, transition) -> + val store = InMemoryStudyStore() + val clocks = MutableClocks(1_000) + val runtime = ExperimentRuntime( + configuration = configuration(surveys = listOf(survey()), interventions = listOf(surveyIntervention())), + store = store, + collectorRegistry = CollectorRegistry(listOf(FakeCollectorPlugin(clocks))), + clocks = clocks, + scope = backgroundScope, + availableAccess = { emptySet() }, + ) + start(runtime) + val posted = surveyOccurrence("1", expiresAtUtcMillis = 60_000) + val opened = surveyOccurrence("2", expiresAtUtcMillis = 60_000) + val scheduled = surveyOccurrence("3", expiresAtUtcMillis = 60_000) + val posting = surveyOccurrence("4", expiresAtUtcMillis = 60_000) + listOf(posted, opened, scheduled, posting).forEach { runtime.ensureOccurrence(it) } + runtime.claimOccurrenceIfDue(posted.occurrenceId) + assertTrue(runtime.markNotificationPosted(posted.occurrenceId)) + runtime.claimOccurrenceIfDue(opened.occurrenceId) + assertTrue(runtime.markNotificationPosted(opened.occurrenceId)) + assertTrue(runtime.openOccurrence(opened.occurrenceId)?.action is SurveyAction) + runtime.claimOccurrenceIfDue(posting.occurrenceId) + assertEquals(CommandResult.Success, transition(runtime)) + val eventCount = store.events.size + + assertEquals(OccurrenceClaimResult.InactiveStudy, runtime.claimOccurrenceIfDue(scheduled.occurrenceId)) + assertEquals(OccurrenceExpiryResult.InactiveStudy, runtime.expireOccurrenceIfDue(posted.occurrenceId)) + assertFalse(runtime.markNotificationPosted(posting.occurrenceId)) + assertNull(runtime.openOccurrence(posted.occurrenceId)) + assertEquals( + SurveySubmissionResult.INVALID, + runtime.submitSurvey(opened.occurrenceId, validSurveyAnswers()), + ) + assertEquals("$name must not append intervention events", eventCount, store.events.size) + } + } + @Test fun illegalCommandFailsWithoutMutatingDurableState() = runTest { val store = InMemoryStudyStore() @@ -338,22 +633,43 @@ class ExperimentRuntimeTest { ) } + private class MutableClocks( + var wallTimeUtcMillis: Long, + ) : ResearchClocks { + private var elapsedRealtimeNanos = 0L + + override fun now(): ResearchTime = ResearchTime( + wallTimeUtcMillis = wallTimeUtcMillis, + elapsedRealtimeNanos = ++elapsedRealtimeNanos, + bootSessionId = "boot-test", + ) + } + private class FakeCollectorPlugin( private val clocks: ResearchClocks, + private val accessRequirement: AccessRequirement? = null, ) : CollectorPlugin { override val descriptor = CollectorDescriptor( id = AppLifecycleConfiguration.ID, - payloadSchemaVersion = 1, displayName = "Fake collector", privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 1_024, + eventContract = CollectorEventContract( + payloadSchemaVersion = 1, + maximumEncodedEventBytes = 512, + payloads = listOf("ACTIVITY_RESUMED", "ACTIVITY_STOPPED", "ACTIVITY_STARTED") + .associateWith { + EventPayloadContract( + mapOf("source" to EventFieldContract(EventFieldType.STRING, required = true)), + ) + }, + ), ) lateinit var context: CollectorContext lateinit var collector: FakeCollector override fun accessRequirements(configuration: CollectorConfiguration): Set { require(configuration is AppLifecycleConfiguration) - return emptySet() + return setOfNotNull(accessRequirement) } override fun create( @@ -366,16 +682,21 @@ class ExperimentRuntimeTest { return collector } - suspend fun emit(type: String): EmitResult { + suspend fun emit( + type: String, + collectorId: String = descriptor.id, + schemaVersion: Int = descriptor.payloadSchemaVersion, + fields: Map = mapOf("source" to "test"), + ): EmitResult { val token = context.eventSink.captureToken() ?: return EmitResult.RejectedByAdmissionGate return context.eventSink.emit( token, EventDraft( - collectorId = descriptor.id, - payloadSchemaVersion = descriptor.payloadSchemaVersion, + collectorId = collectorId, + payloadSchemaVersion = schemaVersion, observedTime = clocks.now(), payloadType = type, - fields = mapOf("source" to "test"), + fields = fields, ), ) } @@ -384,13 +705,23 @@ class ExperimentRuntimeTest { private class FakeCollector : Collector { private val mutableHealth = MutableStateFlow(CollectorHealth(CollectorStatus.STOPPED)) override val health: StateFlow = mutableHealth + override var requiresStop = false + private set var startCount = 0 var pauseCount = 0 var resumeCount = 0 var stopCount = 0 + var failNextStartWithOwnedResources = false + var failNextStopWithOwnedResources = false override suspend fun start() { startCount += 1 + requiresStop = true + if (failNextStartWithOwnedResources) { + failNextStartWithOwnedResources = false + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "SOURCE_REGISTRATION_FAILED") + error("Start left collector resources requiring cleanup") + } mutableHealth.value = CollectorHealth(CollectorStatus.ACTIVE) } @@ -406,6 +737,12 @@ class ExperimentRuntimeTest { override suspend fun stop() { stopCount += 1 + if (failNextStopWithOwnedResources) { + failNextStopWithOwnedResources = false + mutableHealth.value = CollectorHealth(CollectorStatus.FAILED, "SOURCE_UNREGISTRATION_FAILED") + error("Stop left collector resources requiring cleanup") + } + requiresStop = false mutableHealth.value = CollectorHealth(CollectorStatus.STOPPED) } } @@ -425,7 +762,8 @@ class ExperimentRuntimeTest { assignedParticipantId = null, issuedAt = Instant.parse("2026-01-01T00:00:00Z"), expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, + platform = StudyConfiguration.ANDROID_PLATFORM, + minimumClientVersion = 1, title = "Runtime test", researcherName = "Test researcher", researcherContact = "test@example.invalid", @@ -437,10 +775,10 @@ class ExperimentRuntimeTest { surveys = surveys, interventions = interventions, maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", TEST_SIGNER_PUBLIC_KEY), + signer = SignerIdentity("test-signer", RAW_PUBLIC_KEY), export = ExportConfiguration( researcherKeyId = "test-key", - tinkHpkePublicKeysetJson = "{\"placeholder\":\"not-used-in-runtime-tests\"}", + hpkePublicKey = RAW_PUBLIC_KEY, ), upload = null, ) @@ -486,8 +824,28 @@ class ExperimentRuntimeTest { SurveyAction("Daily survey", "Your survey is ready.", "daily-survey"), listOf(InterventionTrigger("after-minute", OneTimeSchedule(1, RelativeClock.CALENDAR_TIME), 60)), ) + + fun validSurveyAnswers(): Map = mapOf( + "daily-note" to SurveyAnswer.Text("complete"), + "mood-scale" to SurveyAnswer.Integer(4), + "primary-place" to SurveyAnswer.Choices(listOf("place-home")), + "symptoms" to SurveyAnswer.Choices(listOf("symptom-none")), + ) + + fun surveyOccurrence( + prefix: String, + scheduledAtUtcMillis: Long = 100, + expiresAtUtcMillis: Long, + ) = InterventionOccurrence( + occurrenceId = prefix.repeat(64), + interventionId = "survey-notice", + triggerId = "after-minute", + scheduleKey = "relative:$prefix", + scheduledFor = ResearchTime(scheduledAtUtcMillis, scheduledAtUtcMillis, "boot-test"), + expiresAtUtcMillis = expiresAtUtcMillis, + state = OccurrenceState.SCHEDULED, + ) } } -private const val TEST_SIGNER_PUBLIC_KEY = - "MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0=" +private const val RAW_PUBLIC_KEY = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" diff --git a/core/export/build.gradle.kts b/core/export/build.gradle.kts index b6431c6..0eb343c 100644 --- a/core/export/build.gradle.kts +++ b/core/export/build.gradle.kts @@ -15,7 +15,9 @@ kotlin { } dependencies { + implementation(project(":core:collector-api")) api(project(":core:model")) + api(project(":core:protocol")) api(project(":core:study-definition")) implementation(project(":core:crypto")) implementation(libs.gson) diff --git a/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/CanonicalJsonWriter.kt b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/CanonicalJsonWriter.kt new file mode 100644 index 0000000..40135f5 --- /dev/null +++ b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/CanonicalJsonWriter.kt @@ -0,0 +1,167 @@ +package cool.linc.androiddatacollector.core.export + +import java.io.Closeable +import java.io.OutputStream +import java.io.OutputStreamWriter +import java.io.Writer + +/** Small RFC 8785 writer that enforces lexical member order while streaming bundle events. */ +internal class CanonicalJsonWriter(output: OutputStream) : Closeable { + private val output: Writer = OutputStreamWriter(output, Charsets.UTF_8) + private val scopes = ArrayDeque() + private var rootWritten = false + + fun beginObject(): CanonicalJsonWriter { + beforeValue() + output.write("{") + scopes.addLast(Scope.Object()) + return this + } + + fun endObject(): CanonicalJsonWriter { + val scope = scopes.removeLastOrNull() as? Scope.Object + ?: throw IllegalStateException("Not inside an object") + require(!scope.awaitingValue) { "Object member is missing a value" } + output.write("}") + return this + } + + fun beginArray(): CanonicalJsonWriter { + beforeValue() + output.write("[") + scopes.addLast(Scope.Array()) + return this + } + + fun endArray(): CanonicalJsonWriter { + require(scopes.removeLastOrNull() is Scope.Array) { "Not inside an array" } + output.write("]") + return this + } + + fun name(name: String): CanonicalJsonWriter { + val scope = scopes.lastOrNull() as? Scope.Object + ?: throw IllegalStateException("A name requires an object") + require(!scope.awaitingValue) { "Previous object member is missing a value" } + require(scope.lastName == null || scope.lastName!! < name) { "Object members are not in canonical order" } + if (!scope.first) output.write(",") + writeString(name) + output.write(":") + scope.first = false + scope.lastName = name + scope.awaitingValue = true + return this + } + + fun value(value: String?): CanonicalJsonWriter { + beforeValue() + if (value == null) output.write("null") else writeString(value) + return this + } + + fun value(value: Int): CanonicalJsonWriter { + beforeValue() + output.write(value.toString()) + return this + } + + fun value(value: Boolean): CanonicalJsonWriter { + beforeValue() + output.write(value.toString()) + return this + } + + fun nullValue(): CanonicalJsonWriter { + beforeValue() + output.write("null") + return this + } + + fun valueCanonicalInteger(value: String): CanonicalJsonWriter { + require(CANONICAL_INTEGER.matches(value)) { "Protocol JSON integer is not canonical" } + beforeValue() + output.write(value) + return this + } + + fun valueDecimal(value: Long): CanonicalJsonWriter { + require(value >= 0) { "Protocol decimal values must be non-negative" } + return value(value.toString()) + } + + fun rawCanonicalJson(value: ByteArray): CanonicalJsonWriter { + beforeValue() + output.write(value.toString(Charsets.UTF_8)) + return this + } + + fun flush() = output.flush() + + override fun close() { + require(scopes.isEmpty()) { "Incomplete JSON document" } + require(rootWritten) { "Empty JSON document" } + output.close() + } + + private fun beforeValue() { + when (val scope = scopes.lastOrNull()) { + null -> { + require(!rootWritten) { "Multiple JSON roots" } + rootWritten = true + } + is Scope.Object -> { + require(scope.awaitingValue) { "Object value requires a member name" } + scope.awaitingValue = false + } + is Scope.Array -> { + if (!scope.first) output.write(",") + scope.first = false + } + } + } + + private fun writeString(value: String) { + output.write("\"") + var index = 0 + while (index < value.length) { + val character = value[index] + when (character) { + '"' -> output.write("\\\"") + '\\' -> output.write("\\\\") + '\b' -> output.write("\\b") + '\t' -> output.write("\\t") + '\n' -> output.write("\\n") + '\u000c' -> output.write("\\f") + '\r' -> output.write("\\r") + else -> when { + character < ' ' -> output.write("\\u%04x".format(character.code)) + character.isHighSurrogate() -> { + require(index + 1 < value.length && value[index + 1].isLowSurrogate()) { + "Invalid Unicode surrogate" + } + output.write(character.code) + output.write(value[++index].code) + } + character.isLowSurrogate() -> throw IllegalArgumentException("Invalid Unicode surrogate") + else -> output.write(character.code) + } + } + index++ + } + output.write("\"") + } + + private sealed interface Scope { + data class Object( + var first: Boolean = true, + var lastName: String? = null, + var awaitingValue: Boolean = false, + ) : Scope + + data class Array(var first: Boolean = true) : Scope + } + + private companion object { + val CANONICAL_INTEGER = Regex("-?(0|[1-9][0-9]*)") + } +} diff --git a/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt new file mode 100644 index 0000000..dbacd83 --- /dev/null +++ b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt @@ -0,0 +1,532 @@ +package cool.linc.androiddatacollector.core.export + +import com.google.gson.Strictness +import com.google.gson.stream.JsonReader +import com.google.gson.stream.JsonToken +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url +import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import cool.linc.androiddatacollector.core.model.EventDraft +import cool.linc.androiddatacollector.core.model.ExperimentState +import cool.linc.androiddatacollector.core.model.ExperimentStateMachine +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.model.TransitionReason +import cool.linc.androiddatacollector.core.protocol.ConfigurationVerifier +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationEnvelope +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration +import java.io.ByteArrayInputStream +import java.io.InputStream +import java.io.InputStreamReader +import java.io.OutputStream +import java.security.MessageDigest +import java.util.UUID + +/** Typed result published only after the complete authenticated document passes Protocol v1. */ +data class VerifiedResearchBundle( + val header: AuthenticatedBundleHeader, + val kind: BundleKind, + val configuration: VerifiedConfiguration, + val producer: BundleProducer, + val exportedAtUtcMillis: Long, + val experiment: VerifiedExperimentSnapshot, +) + +data class VerifiedExperimentSnapshot( + val experimentId: String, + val configurationId: String, + val participantInstanceId: String, + val assignedParticipantId: String?, + val state: ExperimentState, + val nextSequenceNumber: Long, + val retainedFromSequence: Long, + val durableThroughSequence: Long, + val uploadedThroughSequence: Long, + val firstSequenceNumber: Long, + val lastSequenceNumber: Long, + val eventCount: Long, + val transitionCount: Long, +) + +/** + * The sole closed-world reader for authenticated `research-bundle-v1` plaintext. + * + * The caller must keep [plaintext] private and unpublished until this method returns. Validation is + * streaming: even a manual export near the local-storage quota does not become one in-memory DOM. + */ +object ResearchBundleVerifier { + fun verify( + plaintext: InputStream, + header: AuthenticatedBundleHeader, + expectedConfiguration: StudyConfiguration, + ): VerifiedResearchBundle { + val source = DigestingCountingInputStream(plaintext) + val canonical = DigestingCountingOutputStream() + val reader = JsonReader( + InputStreamReader( + source, + Charsets.UTF_8.newDecoder() + .onMalformedInput(java.nio.charset.CodingErrorAction.REPORT) + .onUnmappableCharacter(java.nio.charset.CodingErrorAction.REPORT), + ), + ).apply { strictness = Strictness.STRICT } + val writer = CanonicalJsonWriter(canonical) + val parsed = Parser(reader, writer, header, expectedConfiguration).parse() + require(reader.peek() == JsonToken.END_DOCUMENT) { "Trailing bundle JSON content" } + writer.close() + require(source.count == canonical.count && source.digest().contentEquals(canonical.digest())) { + "Bundle JSON is not canonical" + } + return parsed + } + + fun verify( + plaintext: ByteArray, + header: AuthenticatedBundleHeader, + expectedConfiguration: StudyConfiguration, + ): VerifiedResearchBundle = verify(ByteArrayInputStream(plaintext), header, expectedConfiguration) + + private class Parser( + private val reader: JsonReader, + private val writer: CanonicalJsonWriter, + private val header: AuthenticatedBundleHeader, + private val expectedConfiguration: StudyConfiguration, + ) { + private val expectedConfigurationBytes = StudyConfigurationCodec.encode(expectedConfiguration) + private val permittedCollectors = buildSet { + expectedConfiguration.collectors.forEach { add(it.id) } + if (expectedConfiguration.interventions.isNotEmpty()) add(INTERVENTION_COLLECTOR_ID) + } + + fun parse(): VerifiedResearchBundle { + require(expectedConfigurationBytes.sha256Hex() == header.configurationSha256) { + "Outer configuration digest mismatch" + } + beginObject() + member("bundle_id") + val bundleId = canonicalUuid(string("bundle_id"), "bundle ID") + require(bundleId == header.bundleId) { "Inner bundle ID mismatch" } + + member("bundle_kind") + val kind = when (string("bundle_kind")) { + BundleKind.MANUAL_EXPORT.wireValue -> BundleKind.MANUAL_EXPORT + BundleKind.AUTOMATIC_UPLOAD.wireValue -> BundleKind.AUTOMATIC_UPLOAD + else -> throw IllegalArgumentException("Unknown bundle kind") + } + + member("configuration") + compareExpectedConfiguration() + + member("configuration_sha256") + require(string("configuration_sha256") == header.configurationSha256) { + "Inner configuration digest mismatch" + } + + member("configuration_signature") + val provenance = configurationSignature() + + member("experiment") + val experiment = experiment(kind) + + member("exported_at_utc_millis") + val exportedAt = decimalLong("exported_at_utc_millis") + + member("format") + require(string("format") == ResearchExport.BUNDLE_FORMAT) { "Unknown bundle format" } + + member("producer") + val producer = producer() + endObject() + + val verified = ConfigurationVerifier( + trustedSigningKeys = emptyMap(), + clientVersion = producer.clientVersion.toLong(), + // Historical analysis verifies the issuance-time contract, not whether it is still + // enrollable today. StudyConfiguration already proves this instant precedes expiry. + now = { expectedConfiguration.issuedAt }, + ).verify( + SignedConfigurationCodec.encode( + SignedConfigurationEnvelope( + signerKeyId = provenance.signerKeyId, + configurationBytes = expectedConfigurationBytes, + signature = provenance.signature, + ), + ), + ) + require(verified.configuration == expectedConfiguration) { "Embedded configuration mismatch" } + require(producer.platform == expectedConfiguration.platform) { "Producer platform mismatch" } + return VerifiedResearchBundle(header, kind, verified, producer, exportedAt, experiment) + } + + private fun compareExpectedConfiguration() { + val expected = JsonReader( + InputStreamReader(ByteArrayInputStream(expectedConfigurationBytes), Charsets.UTF_8), + ).apply { strictness = Strictness.STRICT } + compareValue(reader, expected) + require(expected.peek() == JsonToken.END_DOCUMENT) { "Expected configuration comparison is incomplete" } + writer.rawCanonicalJson(expectedConfigurationBytes) + } + + private fun compareValue(actual: JsonReader, expected: JsonReader) { + require(actual.peek() == expected.peek()) { "Embedded configuration value type mismatch" } + when (expected.peek()) { + JsonToken.BEGIN_OBJECT -> { + actual.beginObject() + expected.beginObject() + while (expected.hasNext()) { + require(actual.hasNext()) { "Embedded configuration member is missing" } + require(actual.nextName() == expected.nextName()) { "Embedded configuration member mismatch" } + compareValue(actual, expected) + } + require(!actual.hasNext()) { "Embedded configuration has an unknown member" } + actual.endObject() + expected.endObject() + } + JsonToken.BEGIN_ARRAY -> { + actual.beginArray() + expected.beginArray() + while (expected.hasNext()) { + require(actual.hasNext()) { "Embedded configuration array entry is missing" } + compareValue(actual, expected) + } + require(!actual.hasNext()) { "Embedded configuration has an extra array entry" } + actual.endArray() + expected.endArray() + } + JsonToken.STRING, JsonToken.NUMBER -> require(actual.nextString() == expected.nextString()) { + "Embedded configuration value mismatch" + } + JsonToken.BOOLEAN -> require(actual.nextBoolean() == expected.nextBoolean()) { + "Embedded configuration value mismatch" + } + JsonToken.NULL -> { + actual.nextNull() + expected.nextNull() + } + else -> throw IllegalArgumentException("Invalid embedded configuration token") + } + } + + private fun configurationSignature(): SignatureProvenance { + beginObject() + member("signature") + val signature = ProtocolBase64Url.decodeExact(string("signature"), SIGNATURE_BYTES, "Ed25519 signature") + member("signer_key_id") + val signerKeyId = string("signer_key_id") + require(signerKeyId == expectedConfiguration.signer.keyId) { "Configuration signer key ID mismatch" } + endObject() + return SignatureProvenance(signerKeyId, signature) + } + + private fun producer(): BundleProducer { + beginObject() + member("client_version") + val clientVersion = decimalLongText("client_version") + require(clientVersion.toLong() > 0) { "Producer client version must be positive" } + member("platform") + val platform = string("platform") + endObject() + return BundleProducer(platform, clientVersion) + } + + private fun experiment(kind: BundleKind): VerifiedExperimentSnapshot { + beginObject() + member("assigned_participant_id") + val assignedParticipantId = nullableString("assigned_participant_id") + require(assignedParticipantId == expectedConfiguration.assignedParticipantId) { + "Assigned participant ID mismatch" + } + member("configuration_id") + val configurationId = string("configuration_id") + require(configurationId == expectedConfiguration.configurationId) { "Configuration ID mismatch" } + member("durable_through_sequence") + val durableThrough = decimalLong("durable_through_sequence") + member("event_count") + val declaredEventCount = decimalLong("event_count") + member("events") + val events = events(declaredEventCount) + member("experiment_id") + val experimentId = string("experiment_id") + require(experimentId == expectedConfiguration.experimentId) { "Experiment ID mismatch" } + member("first_sequence_number") + val firstSequence = decimalLong("first_sequence_number") + require(firstSequence > 0) { "First sequence must be positive" } + member("last_sequence_number") + val lastSequence = decimalLong("last_sequence_number") + member("next_sequence_number") + val nextSequence = decimalLong("next_sequence_number") + require(nextSequence > 0 && durableThrough == nextSequence - 1) { "Durable sequence boundary mismatch" } + member("participant_instance_id") + val participantInstanceId = string("participant_instance_id") + require(PARTICIPANT_INSTANCE_ID.matches(participantInstanceId)) { "Invalid participant instance ID" } + member("retained_from_sequence") + val retainedFrom = decimalLong("retained_from_sequence") + member("state") + val state = enumValue(string("state"), "experiment state") + member("transitions") + val transitions = transitions(state) + member("uploaded_through_sequence") + val uploadedThrough = decimalLong("uploaded_through_sequence") + endObject() + + require(retainedFrom in 1..nextSequence) { "Retained range start is invalid" } + require(uploadedThrough in 0..durableThrough) { "Upload watermark is invalid" } + require(retainedFrom <= uploadedThrough + 1) { "Retained range exceeds the upload watermark" } + require(firstSequence in retainedFrom..nextSequence) { "Bundle starts outside retained data" } + require(lastSequence <= durableThrough) { "Bundle range exceeds durable data" } + val expectedLast = if (declaredEventCount == 0L) { + firstSequence - 1 + } else { + require(firstSequence <= Long.MAX_VALUE - declaredEventCount + 1) { "Bundle range overflows" } + firstSequence + declaredEventCount - 1 + } + require(lastSequence == expectedLast) { "Bundle range/count mismatch" } + require(events.count == declaredEventCount) { "Bundle event count mismatch" } + if (declaredEventCount > 0) { + require(events.firstSequence == firstSequence && events.lastSequence == lastSequence) { + "Bundle events do not cover the declared range" + } + } + if (kind == BundleKind.AUTOMATIC_UPLOAD) { + require(declaredEventCount > 0) { "Automatic upload cannot be empty" } + require(firstSequence == uploadedThrough + 1) { "Automatic upload does not start after its watermark" } + } + return VerifiedExperimentSnapshot( + experimentId, + configurationId, + participantInstanceId, + assignedParticipantId, + state, + nextSequence, + retainedFrom, + durableThrough, + uploadedThrough, + firstSequence, + lastSequence, + declaredEventCount, + transitions, + ) + } + + private fun events(declaredCount: Long): EventSummary { + beginArray() + var count = 0L + var first: Long? = null + var previous: Long? = null + while (reader.hasNext()) { + require(count < declaredCount) { "Bundle contains more events than declared" } + val sequence = event() + previous?.let { require(it < Long.MAX_VALUE && sequence == it + 1) { "Non-contiguous event sequence" } } + if (first == null) first = sequence + previous = sequence + count++ + } + endArray() + return EventSummary(count, first, previous) + } + + private fun event(): Long { + beginObject() + member("collector_id") + val collectorId = string("collector_id") + require(collectorId in permittedCollectors) { "Collector was not enabled by the configuration" } + val contract = requireNotNull(ProtocolEventContracts[collectorId]) { "Unknown collector" } + member("fields") + val fields = fields(contract.payloads.values.maxOf { it.fields.size }) + member("observed_time") + val observedTime = time() + member("payload_schema_version") + val schemaVersion = integer("payload_schema_version") + member("payload_type") + val payloadType = string("payload_type") + member("sequence_number") + val sequenceNumber = decimalLong("sequence_number") + require(sequenceNumber > 0) { "Event sequence must be positive" } + endObject() + + val draft = EventDraft(collectorId, schemaVersion, observedTime, payloadType, fields) + require(contract.accepts(draft, sequenceNumber)) { "Event violates its catalog contract" } + return sequenceNumber + } + + private fun fields(maximumFieldCount: Int): Map { + beginObject() + val fields = linkedMapOf() + var previous: String? = null + while (reader.hasNext()) { + require(fields.size < maximumFieldCount) { "Event has too many fields for its catalog contract" } + val name = reader.nextName() + require(previous == null || previous < name) { "Event fields are not in canonical order" } + writer.name(name) + require(fields.put(name, string(name)) == null) { "Duplicate event field" } + previous = name + } + endObject() + return fields + } + + private fun transitions(finalState: ExperimentState): Long { + beginArray() + val stateMachine = ExperimentStateMachine() + var state = ExperimentState.IMPORTED + var count = 0L + while (reader.hasNext()) { + beginObject() + member("from") + val from = enumValue(string("from"), "transition source") + member("reason") + val reason = enumValue(string("reason"), "transition reason") + member("time") + time() + member("to") + val to = enumValue(string("to"), "transition destination") + endObject() + require(from == state && reason.destination == to && stateMachine.canTransition(from, to)) { + "Invalid experiment transition" + } + state = to + require(count < Long.MAX_VALUE) { "Transition count overflows" } + count++ + } + endArray() + require((count == 0L && finalState == ExperimentState.IMPORTED) || (count > 0 && state == finalState)) { + "Experiment state does not match its transition history" + } + return count + } + + private fun time(): ResearchTime { + beginObject() + member("boot_session_id") + val bootSessionId = string("boot_session_id") + require(bootSessionId.toByteArray(Charsets.UTF_8).size in 1..MAXIMUM_BOOT_SESSION_ID_BYTES) { + "Invalid boot session ID" + } + member("monotonic_time_nanos") + val monotonic = decimalLong("monotonic_time_nanos") + member("wall_time_utc_millis") + val wall = decimalLong("wall_time_utc_millis") + endObject() + return ResearchTime(wall, monotonic, bootSessionId) + } + + private fun beginObject() { + require(reader.peek() == JsonToken.BEGIN_OBJECT) { "Expected JSON object" } + reader.beginObject() + writer.beginObject() + } + + private fun endObject() { + require(!reader.hasNext()) { "JSON object has an unknown member" } + reader.endObject() + writer.endObject() + } + + private fun beginArray() { + require(reader.peek() == JsonToken.BEGIN_ARRAY) { "Expected JSON array" } + reader.beginArray() + writer.beginArray() + } + + private fun endArray() { + reader.endArray() + writer.endArray() + } + + private fun member(expected: String) { + require(reader.hasNext() && reader.nextName() == expected) { "Expected JSON member $expected" } + writer.name(expected) + } + + private fun string(label: String): String { + require(reader.peek() == JsonToken.STRING) { "$label must be a string" } + return reader.nextString().also(writer::value) + } + + private fun nullableString(label: String): String? = if (reader.peek() == JsonToken.NULL) { + reader.nextNull() + writer.nullValue() + null + } else { + string(label) + } + + private fun integer(label: String): Int { + require(reader.peek() == JsonToken.NUMBER) { "$label must be an integer" } + val raw = reader.nextString() + writer.valueCanonicalInteger(raw) + return raw.toIntOrNull() ?: throw IllegalArgumentException("$label is outside Int range") + } + + private fun decimalLongText(label: String): String { + val value = string(label) + require(UNSIGNED_DECIMAL.matches(value) && value.toLongOrNull() != null) { + "$label must be a bounded canonical decimal string" + } + return value + } + + private fun decimalLong(label: String): Long = decimalLongText(label).toLong() + + private data class SignatureProvenance(val signerKeyId: String, val signature: ByteArray) + private data class EventSummary(val count: Long, val firstSequence: Long?, val lastSequence: Long?) + } + + private class DigestingCountingInputStream(private val source: InputStream) : InputStream() { + private val messageDigest = MessageDigest.getInstance("SHA-256") + var count = 0L + private set + + override fun read(): Int = source.read().also { value -> + if (value >= 0) { + messageDigest.update(value.toByte()) + count++ + } + } + + override fun read(bytes: ByteArray, offset: Int, length: Int): Int = + source.read(bytes, offset, length).also { read -> + if (read > 0) { + messageDigest.update(bytes, offset, read) + count += read + } + } + + fun digest(): ByteArray = messageDigest.digest() + } + + private class DigestingCountingOutputStream : OutputStream() { + private val messageDigest = MessageDigest.getInstance("SHA-256") + var count = 0L + private set + + override fun write(value: Int) { + messageDigest.update(value.toByte()) + count++ + } + + override fun write(bytes: ByteArray, offset: Int, length: Int) { + messageDigest.update(bytes, offset, length) + count += length + } + + fun digest(): ByteArray = messageDigest.digest() + } + + private fun canonicalUuid(value: String, label: String): UUID = runCatching { UUID.fromString(value) } + .getOrElse { throw IllegalArgumentException("Invalid $label", it) } + .also { uuid -> + require(uuid.toString() == value && uuid.version() == 4 && uuid.variant() == 2) { "Invalid $label" } + } + + private inline fun > enumValue(value: String, label: String): T = + runCatching { enumValueOf(value) } + .getOrElse { throw IllegalArgumentException("Unknown $label", it) } + + private const val SIGNATURE_BYTES = 64 + private const val MAXIMUM_BOOT_SESSION_ID_BYTES = 128 + private const val INTERVENTION_COLLECTOR_ID = "interventions.v1" + private val UNSIGNED_DECIMAL = Regex("0|[1-9][0-9]*") + private val PARTICIPANT_INSTANCE_ID = Regex("[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}") +} diff --git a/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchExport.kt b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchExport.kt index 865c6ad..cdbb9c8 100644 --- a/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchExport.kt +++ b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchExport.kt @@ -1,19 +1,22 @@ package cool.linc.androiddatacollector.core.export -import com.google.gson.stream.JsonWriter import cool.linc.androiddatacollector.core.crypto.HpkeCrypto +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec import cool.linc.androiddatacollector.core.model.RecordedEvent import cool.linc.androiddatacollector.core.model.ResearchTime import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import java.io.FilterOutputStream +import java.io.InputStream import java.io.OutputStream -import java.io.OutputStreamWriter import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction import java.security.MessageDigest import java.security.SecureRandom +import java.util.UUID import javax.crypto.Cipher import javax.crypto.CipherOutputStream import javax.crypto.KeyGenerator @@ -22,44 +25,106 @@ import javax.crypto.spec.SecretKeySpec import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.withContext +enum class BundleKind(val wireValue: String) { + MANUAL_EXPORT("manual_export"), + AUTOMATIC_UPLOAD("automatic_upload"), +} + +data class BundleProducer( + val platform: String, + val clientVersion: String, +) { + init { + require(platform == StudyConfiguration.ANDROID_PLATFORM) { "Unsupported producer platform" } + require(UNSIGNED_DECIMAL.matches(clientVersion) && clientVersion.toLongOrNull() != null) { + "Invalid producer client version" + } + } + + private companion object { val UNSIGNED_DECIMAL = Regex("0|[1-9][0-9]*") } +} + +/** Outer identities authenticated by both HPKE context and document AEAD. */ +data class AuthenticatedBundleHeader( + val bundleId: UUID, + val configurationSha256: String, + val researcherKeyId: String, +) { + init { + require(bundleId.version() == 4 && bundleId.variant() == 2) { "Bundle ID must be a random UUID" } + require(SHA256.matches(configurationSha256)) { "Invalid configuration digest" } + require(StudyConfiguration.ID.matches(researcherKeyId)) { "Invalid researcher key ID" } + } + + private companion object { val SHA256 = Regex("[0-9a-f]{64}") } +} + data class ExportSnapshot( - val configuration: StudyConfiguration, + val verifiedConfiguration: VerifiedConfiguration, val metadata: StudyMetadata, + val producer: BundleProducer, + val bundleKind: BundleKind, val exportedAtUtcMillis: Long, - /** - * First sequence to include. A participant export starts at whatever the device still holds; - * a scheduled upload sends the window after the last one an endpoint confirmed, so bundles do - * not repeat the whole history. - */ + val bundleId: UUID = UUID.randomUUID(), val fromSequence: Long = 1, - /** - * Last sequence to include, or null for everything durable. - */ val toSequence: Long? = null, - /** - * Soft ceiling on the bundle's plaintext, or null for no ceiling. - * - * A bundle stops at the first event boundary past the budget and reports where it stopped, so a - * caller asks for everything and finds out how much fit. Scheduled delivery uses this to keep a - * single request a sane size while still draining as much of a backlog as it can; a participant - * export passes null, because their copy should be complete. - */ + /** Soft plaintext budget; selection stops only at an event boundary. */ val maximumPlaintextBytes: Long? = null, -) +) { + init { + val configuration = verifiedConfiguration.configuration + require(bundleId.version() == 4 && bundleId.variant() == 2) { "Bundle ID must be a random UUID" } + require(exportedAtUtcMillis >= 0) { "Invalid export time" } + require(metadata.experimentId == configuration.experimentId) { "Experiment ID mismatch" } + require(metadata.configurationId == configuration.configurationId) { "Configuration ID mismatch" } + require(metadata.assignedParticipantId == configuration.assignedParticipantId) { "Assigned participant ID mismatch" } + require(producer.platform == configuration.platform) { "Producer platform mismatch" } + require(producer.clientVersion.toLong() >= configuration.minimumClientVersion) { "Producer client version is too old" } + require(verifiedConfiguration.signerKeyId == configuration.signer.keyId) { "Signer provenance mismatch" } + require(verifiedConfiguration.signature.size == 64) { "Invalid Ed25519 signature provenance" } + val canonical = StudyConfigurationCodec.encode(configuration) + require(canonical.contentEquals(verifiedConfiguration.canonicalConfigurationBytes)) { + "Configuration provenance is not canonical" + } + require(canonical.sha256Hex() == verifiedConfiguration.configurationSha256) { + "Configuration digest provenance mismatch" + } + maximumPlaintextBytes?.let { require(it > 0) { "Plaintext budget must be positive" } } + } +} data class ExportReceipt( - val researcherKeyId: String, + val bundleId: UUID, + val configurationSha256: String, val firstSequence: Long, - /** Last sequence actually written, which is where a budgeted bundle stopped. */ - val sequenceBoundary: Long, + val lastSequence: Long, val eventCount: Long, val sha256: String, val byteCount: Long, -) +) { + init { + require(bundleId.version() == 4 && bundleId.variant() == 2) { "Bundle ID must be a random UUID" } + require(SHA256.matches(configurationSha256)) { "Invalid configuration digest" } + require(firstSequence > 0) { "Invalid first sequence" } + require(lastSequence >= 0) { "Invalid last sequence" } + require(eventCount >= 0) { "Invalid event count" } + val expectedLast = if (eventCount == 0L) { + firstSequence - 1 + } else { + require(firstSequence <= Long.MAX_VALUE - eventCount + 1) { "Receipt sequence range overflows" } + firstSequence + eventCount - 1 + } + require(lastSequence == expectedLast) { "Receipt sequence range is inconsistent" } + require(SHA256.matches(sha256)) { "Invalid bundle digest" } + require(byteCount > 0) { "Invalid bundle byte count" } + } + + private companion object { val SHA256 = Regex("[0-9a-f]{64}") } +} object ResearchExport { fun validate(configuration: StudyConfiguration) { - HpkeCrypto.validatePublicKeyset(configuration.export.tinkHpkePublicKeysetJson) + HpkeCrypto.validatePublicKey(configuration.export.hpkePublicKeyBytes()) } suspend fun encrypt( @@ -67,45 +132,54 @@ object ResearchExport { events: StudyStore, destination: OutputStream, ): ExportReceipt = withContext(Dispatchers.IO) { + val configuration = snapshot.verifiedConfiguration.configuration + validate(configuration) val durable = snapshot.metadata.nextSequenceNumber - 1 require(durable == snapshot.metadata.eventCount) { "Export metadata boundary is inconsistent" } - val boundary = snapshot.toSequence ?: durable - require(boundary in 0..durable) { "Export boundary exceeds the durable event count" } - require(snapshot.fromSequence in 1..(boundary + 1)) { "Export range start is out of bounds" } - val context = contextInfo(snapshot.configuration) - val contentKey = KeyGenerator.getInstance("AES").apply { init(KEY_BITS) }.generateKey() - val wrappedKey = HpkeCrypto.encrypt( - snapshot.configuration.export.tinkHpkePublicKeysetJson, - contentKey.encoded, - context, + require(snapshot.fromSequence >= snapshot.metadata.retainedFromSequence) { "Export starts below retained data" } + val requestedBoundary = snapshot.toSequence ?: durable + require(requestedBoundary in 0..durable) { "Export boundary exceeds durable data" } + require(snapshot.fromSequence in 1..(requestedBoundary + 1)) { "Export range start is out of bounds" } + if (snapshot.bundleKind == BundleKind.AUTOMATIC_UPLOAD) { + require(snapshot.fromSequence <= requestedBoundary) { "Automatic upload cannot be empty" } + } + val boundary = selectBoundary(snapshot, events, requestedBoundary) + val eventCount = (boundary - snapshot.fromSequence + 1).coerceAtLeast(0) + val context = contextInfo( + snapshot.bundleId, + snapshot.verifiedConfiguration.configurationSha256, + configuration.export.researcherKeyId, ) + val contentKey = KeyGenerator.getInstance("AES").apply { init(KEY_BITS) }.generateKey() + val wrappedKey = HpkeCrypto.encrypt(configuration.export.hpkePublicKeyBytes(), contentKey.encoded, context) + require(wrappedKey.size == WRAPPED_KEY_BYTES) { "Unexpected wrapped-key size" } val nonce = ByteArray(NONCE_BYTES).also(SecureRandom()::nextBytes) val cipher = Cipher.getInstance("AES/GCM/NoPadding").apply { init(Cipher.ENCRYPT_MODE, contentKey, GCMParameterSpec(TAG_BITS, nonce)) updateAAD(context) } - val keyId = snapshot.configuration.export.researcherKeyId.toByteArray(Charsets.UTF_8) + val keyId = configuration.export.researcherKeyId.toByteArray(Charsets.UTF_8) val digesting = DigestingOutputStream(destination) digesting.write( - ByteBuffer.allocate(MAGIC.size + Short.SIZE_BYTES + Int.SIZE_BYTES + NONCE_BYTES + keyId.size + wrappedKey.size) + ByteBuffer.allocate(FIXED_HEADER_BYTES + keyId.size + WRAPPED_KEY_BYTES) .put(MAGIC) + .putUuid(snapshot.bundleId) + .put(snapshot.verifiedConfiguration.configurationSha256.hexBytes()) .putShort(keyId.size.toShort()) - .putInt(wrappedKey.size) .put(nonce) .put(keyId) .put(wrappedKey) .array(), ) - val counting = CountingOutputStream(CipherOutputStream(digesting, cipher)) - var written = boundary - JsonWriter(OutputStreamWriter(counting, Charsets.UTF_8)).use { writer -> - written = writeSnapshot(writer, snapshot, events, boundary, counting) + CanonicalJsonWriter(CipherOutputStream(digesting, cipher)).use { writer -> + writeSnapshot(writer, snapshot, events, boundary, eventCount) } ExportReceipt( - researcherKeyId = snapshot.configuration.export.researcherKeyId, + bundleId = snapshot.bundleId, + configurationSha256 = snapshot.verifiedConfiguration.configurationSha256, firstSequence = snapshot.fromSequence, - sequenceBoundary = written, - eventCount = (written - snapshot.fromSequence + 1).coerceAtLeast(0), + lastSequence = boundary, + eventCount = eventCount, sha256 = digesting.digest().toHex(), byteCount = digesting.count, ) @@ -113,59 +187,46 @@ object ResearchExport { fun decrypt( encoded: ByteArray, - privateKeysetJson: String, + privateKey: ByteArray, configuration: StudyConfiguration, ): ByteArray = java.io.ByteArrayOutputStream().also { output -> - decrypt(encoded.inputStream(), output, privateKeysetJson, configuration) + decrypt(encoded.inputStream(), output, privateKey, configuration) }.toByteArray() - /** - * Streams a bundle's plaintext into [output]. - * - * Streaming because a bundle is now bounded by the study's own storage quota rather than by a - * fixed ceiling, so a long study can produce a file far larger than a researcher's machine - * would want to hold in memory. - * - * The AES-GCM tag is verified only once the last byte has been read, so [output] holds - * unauthenticated bytes until this returns. It throws on a bad tag, and a caller that cannot - * tolerate partial output should stage the result and only publish it after this returns — - * `researcher-tools decrypt` writes to a temporary file for exactly that reason. - */ + /** [output] remains unauthenticated staging until this method returns successfully. */ fun decrypt( - input: java.io.InputStream, + input: InputStream, output: OutputStream, - privateKeysetJson: String, + privateKey: ByteArray, configuration: StudyConfiguration, - ) { - val header = ByteArray(MAGIC.size + Short.SIZE_BYTES + Int.SIZE_BYTES + NONCE_BYTES) - require(input.readNBytes(header, 0, header.size) == header.size) { "Truncated export" } - val buffer = ByteBuffer.wrap(header) - val magic = ByteArray(MAGIC.size).also(buffer::get) + ): AuthenticatedBundleHeader { + require(privateKey.size == HpkeCrypto.RAW_KEY_BYTES) { "X25519 private key must be 32 bytes" } + val fixed = input.readNBytes(FIXED_HEADER_BYTES) + require(fixed.size == FIXED_HEADER_BYTES) { "Truncated export" } + val header = ByteBuffer.wrap(fixed) + val magic = ByteArray(MAGIC.size).also(header::get) require(magic.contentEquals(MAGIC)) { "Unsupported export format" } - val keyIdLength = buffer.short.toInt() and 0xffff - val wrappedLength = buffer.int - val nonce = ByteArray(NONCE_BYTES).also(buffer::get) - require(keyIdLength in 3..64) { "Invalid export key ID length" } - require(wrappedLength in 32..16_384) { "Invalid wrapped key length" } - - val keyId = input.readNBytes(keyIdLength).also { - require(it.size == keyIdLength) { "Truncated export" } - }.toString(Charsets.UTF_8) + val bundleId = header.getUuid() + val configurationDigest = ByteArray(SHA256_BYTES).also(header::get).toHex() + val keyIdLength = header.short.toInt() and 0xffff + val nonce = ByteArray(NONCE_BYTES).also(header::get) + require(keyIdLength in 3..64) { "Invalid researcher key ID length" } + val expectedDigest = StudyConfigurationCodec.encode(configuration).sha256Hex() + require(configurationDigest == expectedDigest) { "Configuration digest mismatch" } + val keyIdBytes = input.readNBytes(keyIdLength) + require(keyIdBytes.size == keyIdLength) { "Truncated export" } + val keyId = keyIdBytes.strictUtf8("researcher key ID") require(keyId == configuration.export.researcherKeyId) { "Researcher key ID mismatch" } - val wrappedKey = input.readNBytes(wrappedLength).also { - require(it.size == wrappedLength) { "Truncated export" } - } - - val context = contextInfo(configuration) - val contentKey = HpkeCrypto.decrypt(privateKeysetJson, wrappedKey, context) + val authenticatedHeader = AuthenticatedBundleHeader(bundleId, configurationDigest, keyId) + val wrappedKey = input.readNBytes(WRAPPED_KEY_BYTES) + require(wrappedKey.size == WRAPPED_KEY_BYTES) { "Truncated export" } + val context = contextInfo(bundleId, configurationDigest, keyId) + val contentKey = HpkeCrypto.decrypt(privateKey, wrappedKey, context) require(contentKey.size == KEY_BITS / 8) { "Invalid unwrapped content key" } val cipher = Cipher.getInstance("AES/GCM/NoPadding").apply { init(Cipher.DECRYPT_MODE, SecretKeySpec(contentKey, "AES"), GCMParameterSpec(TAG_BITS, nonce)) updateAAD(context) } - // CipherInputStream swallows an AEAD failure as a plain end-of-stream, which would turn a - // tampered bundle into a silently short file. Driving the cipher directly keeps the tag - // failure an exception. val chunk = ByteArray(DECRYPT_CHUNK_BYTES) var total = 0L while (true) { @@ -176,113 +237,162 @@ object ResearchExport { } require(total > TAG_BYTES) { "Truncated export" } cipher.doFinal()?.let(output::write) + return authenticatedHeader + } + + private suspend fun selectBoundary( + snapshot: ExportSnapshot, + events: StudyStore, + requestedBoundary: Long, + ): Long { + val budget = snapshot.maximumPlaintextBytes ?: return requestedBoundary + if (snapshot.fromSequence > requestedBoundary) return snapshot.fromSequence - 1 + val counter = CountingOutputStream(DiscardingOutputStream) + var boundary = snapshot.fromSequence - 1 + var expected = snapshot.fromSequence + var stopped = false + CanonicalJsonWriter(counter).use { writer -> + writer.beginArray() + events.readEvents(snapshot.fromSequence, requestedBoundary) { event -> + if (!stopped) { + require(event.sequenceNumber == expected) { "Non-contiguous export event range" } + writer.writeEvent(event) + writer.flush() + boundary = event.sequenceNumber + expected++ + stopped = counter.count >= budget + } + } + writer.endArray() + } + return boundary } - /** Returns the last sequence written, which is [boundary] unless the budget stopped it sooner. */ private suspend fun writeSnapshot( - writer: JsonWriter, + writer: CanonicalJsonWriter, snapshot: ExportSnapshot, events: StudyStore, boundary: Long, - counting: CountingOutputStream, - ): Long { + eventCount: Long, + ) { + val verified = snapshot.verifiedConfiguration val metadata = snapshot.metadata writer.beginObject() - writer.name("format").value(BUNDLE_FORMAT) - writer.name("exported_at_utc_millis").value(snapshot.exportedAtUtcMillis) - writer.name("configuration").jsonValue( - StudyConfigurationCodec.encode(snapshot.configuration).toString(Charsets.UTF_8), - ) + writer.name("bundle_id").value(snapshot.bundleId.toString()) + writer.name("bundle_kind").value(snapshot.bundleKind.wireValue) + writer.name("configuration").rawCanonicalJson(verified.canonicalConfigurationBytes) + writer.name("configuration_sha256").value(verified.configurationSha256) + writer.name("configuration_signature").beginObject() + writer.name("signature").value(ProtocolBase64Url.encode(verified.signature)) + writer.name("signer_key_id").value(verified.signerKeyId) + writer.endObject() writer.name("experiment").beginObject() - writer.name("experiment_id").value(metadata.experimentId) + writer.name("assigned_participant_id").value(metadata.assignedParticipantId) writer.name("configuration_id").value(metadata.configurationId) + writer.name("durable_through_sequence").valueDecimal(metadata.nextSequenceNumber - 1) + writer.name("event_count").valueDecimal(eventCount) + writer.name("events").beginArray() + var expected = snapshot.fromSequence + if (eventCount > 0) { + events.readEvents(snapshot.fromSequence, boundary) { event -> + require(event.sequenceNumber == expected) { "Non-contiguous export event range" } + writer.writeEvent(event) + expected++ + } + } + require(expected == boundary + 1) { "Export event count mismatch" } + writer.endArray() + writer.name("experiment_id").value(metadata.experimentId) + writer.name("first_sequence_number").valueDecimal(snapshot.fromSequence) + writer.name("last_sequence_number").valueDecimal(boundary) + writer.name("next_sequence_number").valueDecimal(metadata.nextSequenceNumber) writer.name("participant_instance_id").value(metadata.participantInstanceId) - metadata.assignedParticipantId?.let { writer.name("assigned_participant_id").value(it) } + writer.name("retained_from_sequence").valueDecimal(metadata.retainedFromSequence) writer.name("state").value(metadata.state.name) - writer.name("next_sequence_number").value(metadata.nextSequenceNumber) writer.name("transitions").beginArray() metadata.transitions.forEach { transition -> writer.beginObject() writer.name("from").value(transition.from.name) - writer.name("to").value(transition.to.name) writer.name("reason").value(transition.reason.name) writer.name("time").writeTime(transition.time) + writer.name("to").value(transition.to.name) writer.endObject() } writer.endArray() - writer.name("events").beginArray() - var written = snapshot.fromSequence - 1 - var stopped = false - val budget = snapshot.maximumPlaintextBytes - events.readEvents(snapshot.fromSequence, boundary) { event -> - if (!stopped) { - writer.writeEvent(event) - written = event.sequenceNumber - // Always take at least one event, so a bundle can never make zero progress, and - // check on a stride so the budget costs one flush per batch rather than per event. - if (budget != null && written % BUDGET_CHECK_STRIDE == 0L) { - writer.flush() - if (counting.count >= budget) stopped = true - } - } - } - writer.endArray() - // Written after the events, because a budget decides where the bundle stops while it - // streams. Declaring the window up front would have let a bundle claim a range it does not - // contain, which is worse than not declaring one at all. - writer.name("first_sequence_number").value(snapshot.fromSequence) - writer.name("last_sequence_number").value(written) + writer.name("uploaded_through_sequence").valueDecimal(metadata.uploadedThroughSequence) + writer.endObject() + writer.name("exported_at_utc_millis").valueDecimal(snapshot.exportedAtUtcMillis) + writer.name("format").value(BUNDLE_FORMAT) + writer.name("producer").beginObject() + writer.name("client_version").value(snapshot.producer.clientVersion) + writer.name("platform").value(snapshot.producer.platform) writer.endObject() writer.endObject() - return written } - private fun JsonWriter.writeEvent(event: RecordedEvent) { + private fun CanonicalJsonWriter.writeEvent(event: RecordedEvent) { beginObject() - name("sequence_number").value(event.sequenceNumber) name("collector_id").value(event.collectorId) - name("payload_schema_version").value(event.payloadSchemaVersion) - name("observed_time").writeTime(event.observedTime) - name("payload_type").value(event.payloadType) name("fields").beginObject() event.fields.toSortedMap().forEach { (key, value) -> name(key).value(value) } endObject() + name("observed_time").writeTime(event.observedTime) + name("payload_schema_version").value(event.payloadSchemaVersion) + name("payload_type").value(event.payloadType) + name("sequence_number").valueDecimal(event.sequenceNumber) endObject() } - private fun JsonWriter.writeTime(time: ResearchTime) { + private fun CanonicalJsonWriter.writeTime(time: ResearchTime) { beginObject() - name("wall_time_utc_millis").value(time.wallTimeUtcMillis) - name("elapsed_realtime_nanos").value(time.elapsedRealtimeNanos) name("boot_session_id").value(time.bootSessionId) + name("monotonic_time_nanos").valueDecimal(time.elapsedRealtimeNanos) + name("wall_time_utc_millis").valueDecimal(time.wallTimeUtcMillis) endObject() } - private fun contextInfo(configuration: StudyConfiguration): ByteArray = - "$BUNDLE_FORMAT:${configuration.experimentId}:${configuration.configurationId}:${configuration.export.researcherKeyId}" - .toByteArray(Charsets.UTF_8) + private fun contextInfo(bundleId: UUID, configurationSha256: String, researcherKeyId: String): ByteArray = + ("{\"bundle_format\":\"$BUNDLE_FORMAT\",\"bundle_id\":\"$bundleId\"," + + "\"configuration_sha256\":\"$configurationSha256\",\"researcher_key_id\":\"$researcherKeyId\"}" + ).toByteArray(Charsets.UTF_8) - private fun ByteArray.toHex(): String = joinToString(separator = "") { byte -> "%02x".format(byte) } + private fun cool.linc.androiddatacollector.core.definition.ExportConfiguration.hpkePublicKeyBytes(): ByteArray = + ProtocolBase64Url.decodeExact(hpkePublicKey, HpkeCrypto.RAW_KEY_BYTES, "X25519 public key") - private val MAGIC = "ADCEXP01".toByteArray(Charsets.US_ASCII) + private fun ByteBuffer.putUuid(uuid: UUID): ByteBuffer = putLong(uuid.mostSignificantBits) + .putLong(uuid.leastSignificantBits) + + private fun ByteBuffer.getUuid(): UUID = UUID(getLong(), getLong()) - /** - * Part of the HPKE and AES-GCM associated data as well as the bundle's own `format` field, so a - * reader built for a different version cannot silently accept this one: the tag fails first. - */ + private fun String.hexBytes(): ByteArray { + require(length == SHA256_BYTES * 2 && all { it in '0'..'9' || it in 'a'..'f' }) { "Invalid SHA-256" } + return ByteArray(SHA256_BYTES) { index -> substring(index * 2, index * 2 + 2).toInt(16).toByte() } + } + + private fun ByteArray.strictUtf8(label: String): String = runCatching { + Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(this)) + .toString() + }.getOrElse { throw IllegalArgumentException("$label is not valid UTF-8", it) } + .also { require(StudyConfiguration.ID.matches(it)) { "Invalid $label" } } + + private val MAGIC = "ADCEXP01".toByteArray(Charsets.US_ASCII) const val BUNDLE_FORMAT = "research-bundle-v1" private const val KEY_BITS = 256 private const val NONCE_BYTES = 12 private const val TAG_BITS = 128 private const val TAG_BYTES = TAG_BITS / 8 - private const val MINIMUM_EXPORT_BYTES = 8 + 2 + 4 + NONCE_BYTES + 3 + 32 + TAG_BYTES + 1 - - /** How often the budget is checked, in events. Bounds overshoot to one stride of data. */ - private const val BUDGET_CHECK_STRIDE = 256L + private const val SHA256_BYTES = 32 + private const val WRAPPED_KEY_BYTES = 80 + private const val FIXED_HEADER_BYTES = 8 + 16 + SHA256_BYTES + Short.SIZE_BYTES + NONCE_BYTES private const val DECRYPT_CHUNK_BYTES = 64 * 1024 } -/** Tracks plaintext written so the budget can stop a bundle at an event boundary. */ +internal fun ByteArray.sha256Hex(): String = MessageDigest.getInstance("SHA-256").digest(this).toHex() +internal fun ByteArray.toHex(): String = joinToString("") { "%02x".format(it) } + private class CountingOutputStream(output: OutputStream) : FilterOutputStream(output) { var count = 0L private set @@ -298,9 +408,12 @@ private class CountingOutputStream(output: OutputStream) : FilterOutputStream(ou } } -private class DigestingOutputStream( - private val destination: OutputStream, -) : OutputStream() { +private object DiscardingOutputStream : OutputStream() { + override fun write(value: Int) = Unit + override fun write(bytes: ByteArray, offset: Int, length: Int) = Unit +} + +private class DigestingOutputStream(private val destination: OutputStream) : OutputStream() { private val messageDigest = MessageDigest.getInstance("SHA-256") var count = 0L private set @@ -308,7 +421,7 @@ private class DigestingOutputStream( override fun write(value: Int) { destination.write(value) messageDigest.update(value.toByte()) - count += 1 + count++ } override fun write(bytes: ByteArray, offset: Int, length: Int) { diff --git a/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/UploadReceiptCodec.kt b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/UploadReceiptCodec.kt new file mode 100644 index 0000000..dc8df4e --- /dev/null +++ b/core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/UploadReceiptCodec.kt @@ -0,0 +1,78 @@ +package cool.linc.androiddatacollector.core.export + +import com.google.gson.JsonElement +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction +import java.util.UUID + +/** Exact canonical JSON receipt used by both new-upload and exact-replay responses. */ +object UploadReceiptCodec { + private val KEYS = setOf( + "bundle_id", + "byte_count", + "configuration_sha256", + "event_count", + "first_sequence_number", + "last_sequence_number", + "sha256", + ) + + fun encode(receipt: ExportReceipt): ByteArray = ( + "{\"bundle_id\":\"${receipt.bundleId}\"," + + "\"byte_count\":\"${receipt.byteCount}\"," + + "\"configuration_sha256\":\"${receipt.configurationSha256}\"," + + "\"event_count\":\"${receipt.eventCount}\"," + + "\"first_sequence_number\":\"${receipt.firstSequence}\"," + + "\"last_sequence_number\":\"${receipt.lastSequence}\"," + + "\"sha256\":\"${receipt.sha256}\"}" + ).toByteArray(Charsets.UTF_8) + + fun decode(bytes: ByteArray): ExportReceipt { + require(bytes.size in 2..MAX_RECEIPT_BYTES) { "Invalid upload receipt size" } + val text = runCatching { + Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes)) + .toString() + }.getOrElse { throw IllegalArgumentException("Upload receipt is not valid UTF-8", it) } + val root = runCatching { JsonParser.parseString(text) } + .getOrElse { throw IllegalArgumentException("Invalid upload receipt JSON", it) } + .requireObject() + require(root.keySet() == KEYS) { "Unexpected upload receipt keys" } + val receipt = ExportReceipt( + bundleId = runCatching { UUID.fromString(root.string("bundle_id")) } + .getOrElse { throw IllegalArgumentException("Invalid bundle ID", it) }, + configurationSha256 = root.string("configuration_sha256"), + firstSequence = root.decimalLong("first_sequence_number"), + lastSequence = root.decimalLong("last_sequence_number"), + eventCount = root.decimalLong("event_count"), + sha256 = root.string("sha256"), + byteCount = root.decimalLong("byte_count"), + ) + require(encode(receipt).contentEquals(bytes)) { "Upload receipt JSON is not canonical" } + return receipt + } + + private fun JsonElement.requireObject(): JsonObject { + require(isJsonObject) { "Upload receipt must be an object" } + return asJsonObject + } + + private fun JsonObject.string(name: String): String { + val value = get(name) + require(value != null && value.isJsonPrimitive && value.asJsonPrimitive.isString) { "$name must be a string" } + return value.asString + } + + private fun JsonObject.decimalLong(name: String): Long { + val raw = string(name) + require(UNSIGNED_DECIMAL.matches(raw)) { "$name must be a canonical unsigned decimal string" } + return raw.toLongOrNull() ?: throw IllegalArgumentException("$name is outside Long range") + } + + private const val MAX_RECEIPT_BYTES = 2_048 + private val UNSIGNED_DECIMAL = Regex("0|[1-9][0-9]*") +} diff --git a/core/export/src/test/kotlin/cool/linc/androiddatacollector/core/export/ResearchExportTest.kt b/core/export/src/test/kotlin/cool/linc/androiddatacollector/core/export/ResearchExportTest.kt index 4bf194a..44eb757 100644 --- a/core/export/src/test/kotlin/cool/linc/androiddatacollector/core/export/ResearchExportTest.kt +++ b/core/export/src/test/kotlin/cool/linc/androiddatacollector/core/export/ResearchExportTest.kt @@ -1,19 +1,27 @@ package cool.linc.androiddatacollector.core.export +import com.google.gson.JsonParser import cool.linc.androiddatacollector.core.crypto.HpkeCrypto -import cool.linc.androiddatacollector.core.model.StorageUsage -import cool.linc.androiddatacollector.core.model.StudyMetadata -import cool.linc.androiddatacollector.core.model.ExperimentState -import cool.linc.androiddatacollector.core.model.RecordedEvent -import cool.linc.androiddatacollector.core.model.ResearchTime -import cool.linc.androiddatacollector.core.model.StudyStore import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration import cool.linc.androiddatacollector.core.definition.ExportConfiguration +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url import cool.linc.androiddatacollector.core.definition.SignerIdentity import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import cool.linc.androiddatacollector.core.model.ExperimentState +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.model.StorageUsage +import cool.linc.androiddatacollector.core.model.StudyMetadata +import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import java.io.ByteArrayOutputStream +import java.nio.ByteBuffer import java.time.Instant +import java.util.UUID +import javax.crypto.AEADBadTagException import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertArrayEquals import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse import org.junit.Assert.assertNotEquals @@ -23,153 +31,101 @@ import org.junit.Test class ResearchExportTest { @Test - fun personalizedExportEncryptsAssignedAndInstanceIdentifiersTogether() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson).copy(assignedParticipantId = "arm-a-017") - val time = ResearchTime(1_000, 2_000, "boot-test") - val event = RecordedEvent(1, "app_lifecycle.v1", 1, time, "ACTIVITY_RESUMED", emptyMap()) - val metadata = StudyMetadata.initial( - "export-test", - "export-config", - assignedParticipantId = "arm-a-017", - participantInstanceId = "00000000-0000-4000-8000-000000000017", - ).copy( - state = ExperimentState.RUNNING, - eventCount = 1, - nextSequenceNumber = 2, - lastEvents = mapOf(event.collectorId to event), - ) - val encrypted = ByteArrayOutputStream() - ResearchExport.encrypt(ExportSnapshot(configuration, metadata, 10_000), SnapshotStore(metadata, listOf(event)), encrypted) - - val plaintext = ResearchExport.decrypt(encrypted.toByteArray(), keys.privateKeysetJson, configuration) - .toString(Charsets.UTF_8) - assertTrue(plaintext.contains("\"assigned_participant_id\":\"arm-a-017\"")) - assertTrue(plaintext.contains("\"participant_instance_id\":\"${metadata.participantInstanceId}\"")) - assertFalse(encrypted.toByteArray().toString(Charsets.ISO_8859_1).contains("arm-a-017")) - } - - @Test - fun repeatedExportsAreIndependentDecryptableSnapshotsWithoutAStateTransition() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val event = RecordedEvent(1, "app_lifecycle.v1", 1, time, "ACTIVITY_RESUMED", mapOf("source" to "test")) - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 1, - nextSequenceNumber = 2, - lastEvents = mapOf(event.collectorId to event), - ) - val snapshot = ExportSnapshot(configuration, metadata, 10_000) - val events = SnapshotStore(metadata, listOf(event)) - - val firstBytes = ByteArrayOutputStream() - val first = ResearchExport.encrypt(snapshot, events, firstBytes) - val secondBytes = ByteArrayOutputStream() - val second = ResearchExport.encrypt(snapshot.copy(exportedAtUtcMillis = 11_000), events, secondBytes) - - assertNotEquals(first.sha256, second.sha256) - assertTrue(ResearchExport.decrypt(firstBytes.toByteArray(), keys.privateKeysetJson, configuration).toString(Charsets.UTF_8) - .contains("\"state\":\"RUNNING\"")) - assertTrue(ResearchExport.decrypt(secondBytes.toByteArray(), keys.privateKeysetJson, configuration).toString(Charsets.UTF_8) - .contains("\"sequence_number\":1")) - assertEquals(ExperimentState.RUNNING, metadata.state) - assertFalse(ExperimentState.entries.any { it.name == "EXPORTED" }) - } - - @Test - fun exportReadsOnlyTheMetadataBoundaryWhenEventsAppendDuringStreaming() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val first = RecordedEvent(1, "app_lifecycle.v1", 1, time, "FIRST", emptyMap()) - val second = RecordedEvent(2, "app_lifecycle.v1", 1, time, "SECOND", emptyMap()) - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 1, - nextSequenceNumber = 2, - lastEvents = mapOf(first.collectorId to first), - ) + fun adcExpFrameAndAuthenticatedDocumentCarryExactProvenance() = runBlocking { + val fixture = fixture(assignedParticipantId = "arm-a-017") + val events = events(3) + val metadata = metadata(events, assignedParticipantId = "arm-a-017") + val bundleId = UUID.fromString("00000000-0000-4000-8000-000000000099") val destination = ByteArrayOutputStream() - val receipt = ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000), - SnapshotStore(metadata, listOf(first), appendDuringRead = second), + snapshot(fixture.verified, metadata, bundleId = bundleId), + SnapshotStore(metadata, events), destination, ) + val encrypted = destination.toByteArray() + val header = ByteBuffer.wrap(encrypted) + assertEquals("ADCEXP01", ByteArray(8).also(header::get).toString(Charsets.US_ASCII)) + assertEquals(bundleId.mostSignificantBits, header.long) + assertEquals(bundleId.leastSignificantBits, header.long) + assertEquals(fixture.verified.configurationSha256, ByteArray(32).also(header::get).toHex()) + assertEquals("export-key".length, header.short.toInt()) + header.position(header.position() + 12) + assertEquals("export-key", ByteArray("export-key".length).also(header::get).toString(Charsets.UTF_8)) + header.position(header.position() + 80) + assertTrue(header.hasRemaining()) - val plaintext = ResearchExport.decrypt( - destination.toByteArray(), - keys.privateKeysetJson, - configuration, - ).toString(Charsets.UTF_8) - assertEquals(1L, receipt.sequenceBoundary) - assertTrue(plaintext.contains("\"payload_type\":\"FIRST\"")) - assertFalse(plaintext.contains("\"payload_type\":\"SECOND\"")) + val plaintext = ResearchExport.decrypt(encrypted, fixture.hpke.privateKey, fixture.configuration) + val text = plaintext.toString(Charsets.UTF_8) + val root = JsonParser.parseString(text).asJsonObject + val experiment = root.getAsJsonObject("experiment") + assertTrue(text.startsWith("{\"bundle_id\":\"$bundleId\",\"bundle_kind\":\"manual_export\"")) + assertEquals(BUNDLE_KEYS, root.keySet()) + assertEquals(EXPERIMENT_KEYS, experiment.keySet()) + assertEquals(fixture.verified.configurationSha256, root.get("configuration_sha256").asString) + assertEquals(ProtocolBase64Url.encode(fixture.verified.signature), + root.getAsJsonObject("configuration_signature").get("signature").asString) + assertEquals("3", experiment.get("event_count").asString) + assertEquals("3", experiment.get("last_sequence_number").asString) + assertEquals("1", experiment.getAsJsonArray("events")[0].asJsonObject.get("sequence_number").asString) + assertEquals("2000", experiment.getAsJsonArray("events")[0].asJsonObject + .getAsJsonObject("observed_time").get("monotonic_time_nanos").asString) + assertEquals(bundleId, receipt.bundleId) + assertEquals(3L, receipt.eventCount) + assertEquals(encrypted.size.toLong(), receipt.byteCount) + assertFalse(encrypted.toString(Charsets.ISO_8859_1).contains("arm-a-017")) } @Test - fun rangedBundleCarriesOnlyItsWindowAndDeclaresIt() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val events = (1..5).map { - RecordedEvent(it.toLong(), "app_lifecycle.v1", 1, time, "EVENT_$it", emptyMap()) + fun rangedAndBudgetedBundlesDeclareOnlyRowsActuallyWritten() = runBlocking { + val fixture = fixture() + val events = (1..2_000).map { + event(it.toLong(), fields = mapOf("value" to "x".repeat(200))) } - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 5, - nextSequenceNumber = 6, - lastEvents = mapOf(events.last().collectorId to events.last()), - ) + val metadata = metadata(events) val destination = ByteArrayOutputStream() - val receipt = ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000, fromSequence = 3, toSequence = 4), + snapshot( + fixture.verified, + metadata, + fromSequence = 501, + toSequence = 2_000, + maximumPlaintextBytes = 64 * 1_024, + kind = BundleKind.AUTOMATIC_UPLOAD, + ), SnapshotStore(metadata, events), destination, ) - val plaintext = ResearchExport.decrypt( - destination.toByteArray(), - keys.privateKeysetJson, - configuration, - ).toString(Charsets.UTF_8) - - assertEquals(3L, receipt.firstSequence) - assertEquals(4L, receipt.sequenceBoundary) - assertEquals(2L, receipt.eventCount) - assertTrue(plaintext.contains("\"format\":\"research-bundle-v1\"")) - assertTrue(plaintext.contains("\"first_sequence_number\":3")) - assertTrue(plaintext.contains("\"last_sequence_number\":4")) - // A chunk must identify which install it came from, or a study that uploads cannot tell - // one participant's events from another's. - assertTrue(plaintext.contains("\"participant_instance_id\":\"${metadata.participantInstanceId}\"")) - assertTrue(plaintext.contains("\"payload_type\":\"EVENT_3\"")) - assertTrue(plaintext.contains("\"payload_type\":\"EVENT_4\"")) - assertFalse(plaintext.contains("\"payload_type\":\"EVENT_2\"")) - assertFalse(plaintext.contains("\"payload_type\":\"EVENT_5\"")) + assertEquals(501L, receipt.firstSequence) + assertTrue(receipt.lastSequence in 501..<2_000) + assertEquals(receipt.lastSequence - 500, receipt.eventCount) + val text = ResearchExport.decrypt(destination.toByteArray(), fixture.hpke.privateKey, fixture.configuration) + .toString(Charsets.UTF_8) + assertTrue(text.contains("\"bundle_kind\":\"automatic_upload\"")) + assertTrue(text.contains("\"last_sequence_number\":\"${receipt.lastSequence}\"")) + assertTrue(text.contains("\"sequence_number\":\"${receipt.lastSequence}\"")) + assertFalse(text.contains("\"sequence_number\":\"${receipt.lastSequence + 1}\"")) } @Test - fun rangedBundleRejectsAWindowBeyondTheDurableEventCount() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val event = RecordedEvent(1, "app_lifecycle.v1", 1, time, "ONLY", emptyMap()) - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 1, - nextSequenceNumber = 2, - lastEvents = mapOf(event.collectorId to event), + fun manualEmptyBundleIsValidButAutomaticUploadIsNot() = runBlocking { + val fixture = fixture() + val metadata = StudyMetadata.initial("export-test", "export-config") + val destination = ByteArrayOutputStream() + val receipt = ResearchExport.encrypt( + snapshot(fixture.verified, metadata), + SnapshotStore(metadata, emptyList()), + destination, ) + assertEquals(1L, receipt.firstSequence) + assertEquals(0L, receipt.lastSequence) + assertEquals(0L, receipt.eventCount) assertThrows(IllegalArgumentException::class.java) { runBlocking { ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000, fromSequence = 1, toSequence = 9), - SnapshotStore(metadata, listOf(event)), + snapshot(fixture.verified, metadata, kind = BundleKind.AUTOMATIC_UPLOAD), + SnapshotStore(metadata, emptyList()), ByteArrayOutputStream(), ) } @@ -178,133 +134,127 @@ class ResearchExportTest { } @Test - fun aBudgetStopsAtAnEventBoundaryAndTheReceiptSaysWhere() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val events = (1..5_000).map { - RecordedEvent(it.toLong(), "app_lifecycle.v1", 1, time, "EVENT", mapOf("activity_class" to "x".repeat(200))) - } - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 5_000, - nextSequenceNumber = 5_001, - lastEvents = mapOf(events.last().collectorId to events.last()), - ) + fun wrongContextTamperingAndOldV1FramingFailClosed() = runBlocking { + val fixture = fixture() + val stored = events(1) + val metadata = metadata(stored) val destination = ByteArrayOutputStream() + ResearchExport.encrypt(snapshot(fixture.verified, metadata), SnapshotStore(metadata, stored), destination) + val encoded = destination.toByteArray() - // Ask for everything, with a budget far below what everything would need. - val receipt = ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000, maximumPlaintextBytes = 64 * 1024), - SnapshotStore(metadata, events), - destination, - ) - - assertTrue("expected the budget to stop it short", receipt.sequenceBoundary < 5_000) - assertTrue("expected real progress", receipt.sequenceBoundary > 0) - // Stopping short must still produce a complete, decryptable bundle rather than a truncated - // one, and it must declare the window it actually holds. - val plaintext = ResearchExport.decrypt(destination.toByteArray(), keys.privateKeysetJson, configuration) - .toString(Charsets.UTF_8) - assertTrue(plaintext.contains("\"last_sequence_number\":${receipt.sequenceBoundary}")) - assertTrue(plaintext.contains("\"sequence_number\":${receipt.sequenceBoundary}")) - assertFalse(plaintext.contains("\"sequence_number\":${receipt.sequenceBoundary + 1}")) + assertThrows(Exception::class.java) { + ResearchExport.decrypt(encoded, HpkeCrypto.generateKeyPair().privateKey, fixture.configuration) + } + assertThrows(Exception::class.java) { + ResearchExport.decrypt(encoded.copyOf().also { it[it.lastIndex]++ }, fixture.hpke.privateKey, fixture.configuration) + } + assertThrows(IllegalArgumentException::class.java) { + ResearchExport.decrypt(encoded.copyOf().also { it[24]++ }, fixture.hpke.privateKey, fixture.configuration) + } + val oldV1 = "ADCEXP01".toByteArray() + ByteArray(128) + assertThrows(IllegalArgumentException::class.java) { + ResearchExport.decrypt(oldV1, fixture.hpke.privateKey, fixture.configuration) + } + Unit } @Test - fun noBudgetSendsEverythingAsked() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val events = (1..2_000).map { - RecordedEvent(it.toLong(), "app_lifecycle.v1", 1, time, "EVENT", mapOf("activity_class" to "x".repeat(200))) - } - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 2_000, - nextSequenceNumber = 2_001, - lastEvents = mapOf(events.last().collectorId to events.last()), - ) - - // A participant export passes no budget, so their copy is complete however large it gets. - val receipt = ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000), - SnapshotStore(metadata, events), - ByteArrayOutputStream(), + fun uploadReceiptCodecIsExactCanonicalAndSelfConsistent() { + val receipt = ExportReceipt( + UUID.fromString("00000000-0000-4000-8000-000000000099"), + "11".repeat(32), + 501, + 750, + 250, + "22".repeat(32), + 16_777_216, ) + val encoded = UploadReceiptCodec.encode(receipt) - assertEquals(2_000L, receipt.sequenceBoundary) - assertEquals(2_000L, receipt.eventCount) + assertEquals(receipt, UploadReceiptCodec.decode(encoded)) + assertTrue(encoded.toString(Charsets.UTF_8).contains("\"byte_count\":\"16777216\"")) + assertThrows(IllegalArgumentException::class.java) { + UploadReceiptCodec.decode((" " + encoded.toString(Charsets.UTF_8)).toByteArray()) + } + assertThrows(IllegalArgumentException::class.java) { + UploadReceiptCodec.decode(encoded.toString(Charsets.UTF_8) + .replace("\"501\"", "\"0501\"").toByteArray()) + } + assertThrows(IllegalArgumentException::class.java) { + UploadReceiptCodec.decode(encoded.toString(Charsets.UTF_8) + .replace("\"event_count\":\"250\"", "\"event_count\":250").toByteArray()) + } } @Test - fun streamingDecryptStillRefusesATamperedBundle() = runBlocking { - val keys = HpkeCrypto.generateKeyset() - val configuration = configuration(keys.publicKeysetJson) - val time = ResearchTime(1_000, 2_000, "boot-test") - val event = RecordedEvent(1, "app_lifecycle.v1", 1, time, "ONLY", emptyMap()) - val metadata = StudyMetadata.initial("export-test", "export-config").copy( - state = ExperimentState.RUNNING, - eventCount = 1, - nextSequenceNumber = 2, - lastEvents = mapOf(event.collectorId to event), - ) - val destination = ByteArrayOutputStream() - ResearchExport.encrypt( - ExportSnapshot(configuration, metadata, 10_000), - SnapshotStore(metadata, listOf(event)), - destination, - ) - // Reading in chunks must not turn an authentication failure into a silently short file. - val tampered = destination.toByteArray().also { it[it.lastIndex] = (it.last() + 1).toByte() } - - assertThrows(javax.crypto.AEADBadTagException::class.java) { - ResearchExport.decrypt(tampered, keys.privateKeysetJson, configuration) + fun snapshotRejectsConfigurationProvenanceDrift() { + val fixture = fixture() + val metadata = StudyMetadata.initial("export-test", "export-config") + assertThrows(IllegalArgumentException::class.java) { + snapshot( + fixture.verified.copy(configurationSha256 = "00".repeat(32)), + metadata, + ) + } + assertThrows(IllegalArgumentException::class.java) { + snapshot( + fixture.verified.copy(canonicalConfigurationBytes = fixture.verified.canonicalConfigurationBytes + 0), + metadata, + ) } - Unit } - private class SnapshotStore( - private var metadata: StudyMetadata, - events: List, - private val appendDuringRead: RecordedEvent? = null, - ) : StudyStore { - private val storedEvents = events.toMutableList() + private fun snapshot( + verified: VerifiedConfiguration, + metadata: StudyMetadata, + bundleId: UUID = UUID.fromString("00000000-0000-4000-8000-000000000099"), + fromSequence: Long = metadata.retainedFromSequence, + toSequence: Long? = null, + maximumPlaintextBytes: Long? = null, + kind: BundleKind = BundleKind.MANUAL_EXPORT, + ) = ExportSnapshot( + verified, + metadata, + BundleProducer("android", "42"), + kind, + 10_000, + bundleId, + fromSequence, + toSequence, + maximumPlaintextBytes, + ) - override suspend fun loadMetadata(): StudyMetadata = metadata - override suspend fun initialize(metadata: StudyMetadata) { this.metadata = metadata } - override suspend fun saveMetadata(metadata: StudyMetadata) { this.metadata = metadata } - override suspend fun appendEvent(event: RecordedEvent) { storedEvents += event } - override suspend fun appendEventAtomically(event: RecordedEvent, metadata: StudyMetadata) { storedEvents += event } - override suspend fun readEvents( - fromSequenceInclusive: Long, - upToSequenceInclusive: Long, - consume: (RecordedEvent) -> Unit, - ) { - storedEvents.takeWhile { it.sequenceNumber <= upToSequenceInclusive } - .filter { it.sequenceNumber >= fromSequenceInclusive } - .forEachIndexed { index, event -> - consume(event) - if (index == 0) appendDuringRead?.let(storedEvents::add) - } - } - override suspend fun storageUsage() = StorageUsage(storedEvents.size.toLong(), 16_777_216) - override suspend fun evictThrough(metadata: StudyMetadata, targetBytes: Long) = metadata - override suspend fun clear() { storedEvents.clear() } + private fun fixture(assignedParticipantId: String? = null): Fixture { + val hpke = HpkeCrypto.generateKeyPair() + val configuration = configuration(hpke.publicKey, assignedParticipantId) + val bytes = StudyConfigurationCodec.encode(configuration) + return Fixture( + hpke, + configuration, + VerifiedConfiguration( + configuration, + bytes, + configuration.signer.keyId, + ByteArray(64) { it.toByte() }, + bytes.sha256Hex(), + false, + ), + ) } - private fun configuration(publicKeyset: String) = StudyConfiguration( - schemaVersion = StudyConfiguration.CURRENT_SCHEMA_VERSION, + private fun configuration(publicKey: ByteArray, assignedParticipantId: String?) = StudyConfiguration( + schemaVersion = 1, experimentId = "export-test", configurationId = "export-config", - assignedParticipantId = null, + assignedParticipantId = assignedParticipantId, issuedAt = Instant.parse("2026-01-01T00:00:00Z"), expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, + platform = "android", + minimumClientVersion = 1, title = "Export test", researcherName = "Export researcher", researcherContact = "export@example.invalid", - purpose = "Test export encryption.", + purpose = "Test Protocol v1 export encryption.", durationHours = 1, consentDocumentVersion = "v1", consentSummary = "Export test consent.", @@ -312,11 +262,69 @@ class ResearchExportTest { surveys = emptyList(), interventions = emptyList(), maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", TEST_SIGNER_PUBLIC_KEY), - export = ExportConfiguration("export-key", publicKeyset), + signer = SignerIdentity("test-signer", ProtocolBase64Url.encode(ByteArray(32) { 3 })), + export = ExportConfiguration("export-key", ProtocolBase64Url.encode(publicKey)), upload = null, ) -} -private const val TEST_SIGNER_PUBLIC_KEY = - "MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0=" + private fun events(count: Int) = (1..count).map { event(it.toLong()) } + + private fun event(sequence: Long, fields: Map = emptyMap()) = RecordedEvent( + sequence, + "app_lifecycle.v1", + 1, + ResearchTime(1_000, 2_000, "boot-test"), + "EVENT", + fields, + ) + + private fun metadata(events: List, assignedParticipantId: String? = null) = + StudyMetadata.initial( + "export-test", + "export-config", + assignedParticipantId, + "00000000-0000-4000-8000-000000000017", + ).copy( + state = ExperimentState.RUNNING, + eventCount = events.size.toLong(), + nextSequenceNumber = events.size + 1L, + lastEvents = events.lastOrNull()?.let { mapOf(it.collectorId to it) } ?: emptyMap(), + ) + + private data class Fixture( + val hpke: cool.linc.androiddatacollector.core.crypto.HpkeKeyPair, + val configuration: StudyConfiguration, + val verified: VerifiedConfiguration, + ) + + private class SnapshotStore( + private var metadata: StudyMetadata, + private val events: List, + ) : StudyStore { + override suspend fun loadMetadata() = metadata + override suspend fun initialize(metadata: StudyMetadata) { this.metadata = metadata } + override suspend fun saveMetadata(metadata: StudyMetadata) { this.metadata = metadata } + override suspend fun appendEvent(event: RecordedEvent) = error("Not supported") + override suspend fun appendEventAtomically(event: RecordedEvent, metadata: StudyMetadata) = error("Not supported") + override suspend fun readEvents(fromSequenceInclusive: Long, upToSequenceInclusive: Long, + consume: (RecordedEvent) -> Unit) { + events.filter { it.sequenceNumber in fromSequenceInclusive..upToSequenceInclusive }.forEach(consume) + } + override suspend fun storageUsage() = StorageUsage(events.size.toLong(), 16_777_216) + override suspend fun evictThrough(metadata: StudyMetadata, targetBytes: Long) = metadata + override suspend fun clear() = Unit + } + + private companion object { + val BUNDLE_KEYS = setOf( + "bundle_id", "bundle_kind", "configuration", "configuration_sha256", + "configuration_signature", "experiment", "exported_at_utc_millis", "format", "producer", + ) + val EXPERIMENT_KEYS = setOf( + "assigned_participant_id", "configuration_id", "durable_through_sequence", "event_count", + "events", "experiment_id", "first_sequence_number", "last_sequence_number", + "next_sequence_number", "participant_instance_id", "retained_from_sequence", "state", + "transitions", "uploaded_through_sequence", + ) + } +} diff --git a/core/model/src/main/kotlin/cool/linc/androiddatacollector/core/model/StudyData.kt b/core/model/src/main/kotlin/cool/linc/androiddatacollector/core/model/StudyData.kt index b8700c9..37b386a 100644 --- a/core/model/src/main/kotlin/cool/linc/androiddatacollector/core/model/StudyData.kt +++ b/core/model/src/main/kotlin/cool/linc/androiddatacollector/core/model/StudyData.kt @@ -11,9 +11,9 @@ data class StudyMetadata( val nextSequenceNumber: Long, val lastEvents: Map, /** - * Pseudonymous per-install identifier. A study that uploads has no other way to tell one - * participant's events from another's, because a manual export carries that information - * out of band. Unlike the assigned code, this UUID is also the cleartext upload routing key. + * Pseudonymous per-install identifier used only after authentication and decryption to keep + * event streams distinct. It remains inside ciphertext; upload URLs and headers carry no + * participant identifier. */ val participantInstanceId: String, /** Optional researcher-assigned code; protected inside encrypted metadata and exports. */ diff --git a/core/protocol/build.gradle.kts b/core/protocol/build.gradle.kts index f7561ee..428744b 100644 --- a/core/protocol/build.gradle.kts +++ b/core/protocol/build.gradle.kts @@ -16,5 +16,6 @@ kotlin { dependencies { api(project(":core:study-definition")) + implementation(project(":core:crypto")) testImplementation(libs.junit4) } diff --git a/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLink.kt b/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLink.kt new file mode 100644 index 0000000..a8573b3 --- /dev/null +++ b/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLink.kt @@ -0,0 +1,138 @@ +package cool.linc.androiddatacollector.core.protocol + +import java.io.ByteArrayOutputStream +import java.net.URI +import java.net.URISyntaxException +import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction + +/** Immutable, closed-world transport pointer for one signed configuration artifact. */ +data class JoinLink( + val artifactUrl: URI, + val artifactSha256: String, + /** SHA-256/128 signer fingerprint as 32 uppercase hexadecimal characters, without spaces. */ + val signerFingerprint: String, +) { + init { + requireCanonicalArtifactUrl(artifactUrl) + require(SHA256.matches(artifactSha256)) { "Invalid join artifact SHA-256" } + require(FINGERPRINT.matches(signerFingerprint)) { "Invalid join signer fingerprint" } + } + + fun encode(): String = buildString { + append(PREFIX) + append("artifact=") + append(percentEncode(artifactUrl.toASCIIString())) + append("&sha256=") + append(artifactSha256) + append("&signer_fingerprint=") + append(signerFingerprint) + }.also { + require(it.length <= MAXIMUM_JOIN_LINK_BYTES) { "ADC join link is too long" } + } + + fun displayFingerprint(): String = signerFingerprint.chunked(4).joinToString(" ") + + companion object { + private const val PREFIX = "adc://join/v1?" + private const val MAXIMUM_JOIN_LINK_BYTES = 4_096 + private const val MAXIMUM_ARTIFACT_URL_BYTES = 2_048 + private val SHA256 = Regex("[0-9a-f]{64}") + private val FINGERPRINT = Regex("[0-9A-F]{32}") + private val QUERY_KEYS = listOf("artifact", "sha256", "signer_fingerprint") + private val ARTIFACT_URL = Regex("https://([^/:?#]+)(?::([0-9]+))?(/[A-Za-z0-9._~/-]+)") + private val HOST_LABEL = Regex("(?:[a-z0-9]|[a-z0-9][a-z0-9-]{0,61}[a-z0-9])") + private val CANONICAL_PORT = Regex("[1-9][0-9]{0,4}") + private val UNRESERVED = + ('a'..'z').toSet() + ('A'..'Z').toSet() + ('0'..'9').toSet() + setOf('-', '.', '_', '~') + + fun parse(encoded: String): JoinLink { + require(encoded.length <= MAXIMUM_JOIN_LINK_BYTES && encoded.startsWith(PREFIX)) { + "Invalid ADC join link" + } + val rawParts = encoded.removePrefix(PREFIX).split('&') + require(rawParts.size == QUERY_KEYS.size) { "Invalid ADC join query" } + val values = rawParts.mapIndexed { index, part -> + val separator = part.indexOf('=') + require(separator > 0 && part.indexOf('=', separator + 1) < 0) { "Invalid ADC join query" } + require(part.substring(0, separator) == QUERY_KEYS[index]) { "Invalid ADC join query" } + percentDecode(part.substring(separator + 1)) + } + return JoinLink( + artifactUrl = try { + URI(values[0]) + } catch (failure: URISyntaxException) { + throw IllegalArgumentException("Invalid join artifact URL", failure) + }, + artifactSha256 = values[1], + signerFingerprint = values[2], + ).also { require(it.encode() == encoded) { "ADC join link is not canonical" } } + } + + private fun requireCanonicalArtifactUrl(uri: URI) { + val value = uri.toASCIIString() + require(value.length <= MAXIMUM_ARTIFACT_URL_BYTES) { "Join artifact URL is too long" } + val match = requireNotNull(ARTIFACT_URL.matchEntire(value)) { + "Join artifact URL is outside the canonical HTTPS profile" + } + val host = match.groupValues[1] + require(host.length <= 253 && host.any(Char::isLetter) && host.split('.').all(HOST_LABEL::matches)) { + "Join artifact URL has a noncanonical host" + } + match.groupValues[2].takeIf(String::isNotEmpty)?.let { port -> + require(CANONICAL_PORT.matches(port) && port.toInt() in 1..65_535 && port != "443") { + "Join artifact URL has a noncanonical port" + } + } + require( + match.groupValues[3].removePrefix("/").split('/').all { segment -> + segment.isNotEmpty() && segment != "." && segment != ".." + }, + ) { "Join artifact URL has a noncanonical path" } + } + + private fun percentEncode(value: String): String = buildString { + value.toByteArray(Charsets.UTF_8).forEach { byte -> + val unsigned = byte.toInt() and 0xff + val character = unsigned.toChar() + if (character in UNRESERVED) { + append(character) + } else { + append('%') + append(HEX[unsigned ushr 4]) + append(HEX[unsigned and 0x0f]) + } + } + } + + private fun percentDecode(value: String): String { + val output = ByteArrayOutputStream(value.length) + var index = 0 + while (index < value.length) { + val character = value[index] + when { + character == '%' -> { + require(index + 2 < value.length) { "Invalid ADC join escaping" } + val high = value[index + 1].digitToIntOrNull(16) + val low = value[index + 2].digitToIntOrNull(16) + require(high != null && low != null) { "Invalid ADC join escaping" } + output.write((high shl 4) or low) + index += 3 + } + character.code < 0x80 -> { + output.write(character.code) + index++ + } + else -> throw IllegalArgumentException("ADC join query must be ASCII") + } + } + return Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(output.toByteArray())) + .toString() + } + + private const val HEX = "0123456789ABCDEF" + } +} diff --git a/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/SignedConfiguration.kt b/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/SignedConfiguration.kt index 4e2c8bb..abd0b02 100644 --- a/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/SignedConfiguration.kt +++ b/core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/SignedConfiguration.kt @@ -1,15 +1,13 @@ package cool.linc.androiddatacollector.core.protocol +import cool.linc.androiddatacollector.core.crypto.Ed25519Crypto +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec import java.nio.ByteBuffer -import java.security.KeyFactory -import java.security.PublicKey -import java.security.Signature -import java.security.SignatureException -import java.security.spec.X509EncodedKeySpec +import java.nio.charset.CodingErrorAction +import java.security.MessageDigest import java.time.Instant -import java.util.Base64 data class SignedConfigurationEnvelope( val signerKeyId: String, @@ -17,20 +15,21 @@ data class SignedConfigurationEnvelope( val signature: ByteArray, ) +/** The sole Protocol v1 signed-configuration framing. Older v1 bytes are intentionally rejected. */ object SignedConfigurationCodec { fun encode(envelope: SignedConfigurationEnvelope): ByteArray { + require(StudyConfiguration.ID.matches(envelope.signerKeyId)) { "Invalid signer key ID" } val keyId = envelope.signerKeyId.toByteArray(Charsets.UTF_8) - require(keyId.size in 3..64) { "Invalid signer key ID" } + require(keyId.size in MINIMUM_KEY_ID_BYTES..MAXIMUM_KEY_ID_BYTES) { "Invalid signer key ID" } require(envelope.configurationBytes.size in 2..MAX_CONFIGURATION_BYTES) { "Invalid configuration size" } - require(envelope.signature.size in 32..128) { "Invalid signature size" } + require(envelope.signature.size == SIGNATURE_BYTES) { "Ed25519 signature must be 64 bytes" } return ByteBuffer.allocate( - MAGIC.size + Short.SIZE_BYTES + Int.SIZE_BYTES + Short.SIZE_BYTES + - keyId.size + envelope.configurationBytes.size + envelope.signature.size, + MAGIC.size + Short.SIZE_BYTES + Int.SIZE_BYTES + keyId.size + + envelope.configurationBytes.size + SIGNATURE_BYTES, ) .put(MAGIC) .putShort(keyId.size.toShort()) .putInt(envelope.configurationBytes.size) - .putShort(envelope.signature.size.toShort()) .put(keyId) .put(envelope.configurationBytes) .put(envelope.signature) @@ -44,108 +43,135 @@ object SignedConfigurationCodec { require(magic.contentEquals(MAGIC)) { "Unsupported signed-configuration format" } val keyIdLength = buffer.short.toInt() and 0xffff val configurationLength = buffer.int - val signatureLength = buffer.short.toInt() and 0xffff - require(keyIdLength in 3..64) { "Invalid signer key ID length" } + require(keyIdLength in MINIMUM_KEY_ID_BYTES..MAXIMUM_KEY_ID_BYTES) { "Invalid signer key ID length" } require(configurationLength in 2..MAX_CONFIGURATION_BYTES) { "Invalid configuration length" } - require(signatureLength in 32..128) { "Invalid signature length" } - require(buffer.remaining() == keyIdLength + configurationLength + signatureLength) { "Envelope length mismatch" } + require(buffer.remaining() == keyIdLength + configurationLength + SIGNATURE_BYTES) { + "Envelope length mismatch" + } + val keyIdBytes = ByteArray(keyIdLength).also(buffer::get) + val keyId = runCatching { + Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(keyIdBytes)) + .toString() + }.getOrElse { throw IllegalArgumentException("Signer key ID is not valid UTF-8", it) } + require(StudyConfiguration.ID.matches(keyId)) { "Invalid signer key ID" } return SignedConfigurationEnvelope( - signerKeyId = ByteArray(keyIdLength).also(buffer::get).toString(Charsets.UTF_8), + signerKeyId = keyId, configurationBytes = ByteArray(configurationLength).also(buffer::get), - signature = ByteArray(signatureLength).also(buffer::get), + signature = ByteArray(SIGNATURE_BYTES).also(buffer::get), ) } private val MAGIC = "ADCCFG01".toByteArray(Charsets.US_ASCII) + private const val SIGNATURE_BYTES = 64 + private const val MINIMUM_KEY_ID_BYTES = 3 + private const val MAXIMUM_KEY_ID_BYTES = 64 private const val MAX_CONFIGURATION_BYTES = 1_048_576 - private const val MINIMUM_ENVELOPE_BYTES = 8 + 2 + 4 + 2 + 3 + 2 + 32 - private const val MAXIMUM_ENVELOPE_BYTES = 8 + 2 + 4 + 2 + 64 + MAX_CONFIGURATION_BYTES + 128 + private const val MINIMUM_ENVELOPE_BYTES = 8 + 2 + 4 + MINIMUM_KEY_ID_BYTES + 2 + SIGNATURE_BYTES + const val MAXIMUM_ENVELOPE_BYTES = + 8 + 2 + 4 + MAXIMUM_KEY_ID_BYTES + MAX_CONFIGURATION_BYTES + SIGNATURE_BYTES } -/** A configuration that passed verification, and whether its signer was one the build pins. */ +/** Immutable provenance needed by every authenticated research bundle. */ data class VerifiedConfiguration( val configuration: StudyConfiguration, - /** - * True when [trustedSigningKeys] listed this signer. False means the configuration certified - * itself: the signature proves it is unchanged since signing, not who wrote it. The consent - * screen has to say so. - */ + val canonicalConfigurationBytes: ByteArray, + val signerKeyId: String, + val signature: ByteArray, + /** Lowercase hexadecimal SHA-256 of [canonicalConfigurationBytes]. */ + val configurationSha256: String, val signerAnchored: Boolean, ) -/** - * Verifies a signed study configuration. - * - * The signing public key travels inside the signed bytes, so any configuration can be checked with - * nothing but the file. That is what lets one published app run any researcher's study. - * - * [trustedSigningKeys] is optional hardening rather than the basis of verification. Leave it empty - * and the app accepts any correctly signed configuration, flagging the publisher as unverified. - * Populate it — as an institution shipping its own build would — and only those signers are - * accepted, with the pinned key overriding whatever the configuration declares. - */ class ConfigurationVerifier( trustedSigningKeys: Map, - private val appVersionCode: Int, + private val clientVersion: Long, private val now: () -> Instant = Instant::now, ) { - private val keys: Map = trustedSigningKeys.mapValues { (_, encoded) -> + init { + require(clientVersion > 0) { "Client version must be positive" } + } + + private val keys: Map = trustedSigningKeys.mapValues { (_, encoded) -> decodeSigningKey(encoded) } fun verify(envelopeBytes: ByteArray): VerifiedConfiguration { val envelope = SignedConfigurationCodec.decode(envelopeBytes) - // Decode before verifying so the declared key is available, then verify over the same bytes - // that were decoded. Nothing is trusted until the signature checks out. val configuration = StudyConfigurationCodec.decode(envelope.configurationBytes) require(configuration.signer.keyId == envelope.signerKeyId) { "Envelope signer does not match the configuration" } val anchored = keys.containsKey(envelope.signerKeyId) - if (keys.isNotEmpty()) { - require(anchored) { "Untrusted configuration signer" } - } - // A pinned key wins over the declared one, so pinning cannot be sidestepped by shipping a - // configuration that names a pinned key ID but carries a different public key. - val key = keys[envelope.signerKeyId] ?: decodeSigningKey(configuration.signer.publicKey) + if (keys.isNotEmpty()) require(anchored) { "Untrusted configuration signer" } + val declaredKey = decodeSigningKey(configuration.signer.publicKey) + val verificationKey = keys[envelope.signerKeyId] ?: declaredKey if (anchored) { - require( - keys.getValue(envelope.signerKeyId).encoded - .contentEquals(decodeSigningKey(configuration.signer.publicKey).encoded), - ) { "Configuration signer key does not match the pinned key" } - } - - // Every rejection here is an IllegalArgumentException, and a bad signature has to be one - // too. `Signature.verify` does not report them uniformly: the JDK's Ed25519 returns false - // for a signature that simply does not match, but throws when the S component is out of - // range, which flipping one byte of a real signature produces about 6% of the time. Which - // byte a corrupt file happened to lose should not decide what the caller catches. - val valid = try { - Signature.getInstance("Ed25519").run { - initVerify(key) - update(envelope.configurationBytes) - verify(envelope.signature) + require(verificationKey.contentEquals(declaredKey)) { + "Configuration signer key does not match the pinned key" } - } catch (malformed: SignatureException) { - false } + + val valid = Ed25519Crypto.verify( + publicKey = verificationKey, + message = envelope.configurationBytes, + signature = envelope.signature, + ) require(valid) { "Invalid configuration signature" } val instant = now() require(!instant.isBefore(configuration.issuedAt)) { "Configuration is not active yet" } require(instant.isBefore(configuration.expiresAt)) { "Configuration has expired" } - require(appVersionCode >= configuration.minimumAppVersion) { "App version is too old" } - return VerifiedConfiguration(configuration, anchored) + require(configuration.platform == StudyConfiguration.ANDROID_PLATFORM) { "Configuration targets another platform" } + require(clientVersion >= configuration.minimumClientVersion) { "Client version is too old" } + return VerifiedConfiguration( + configuration = configuration, + canonicalConfigurationBytes = envelope.configurationBytes.copyOf(), + signerKeyId = envelope.signerKeyId, + signature = envelope.signature.copyOf(), + configurationSha256 = MessageDigest.getInstance("SHA-256") + .digest(envelope.configurationBytes) + .toHex(), + signerAnchored = anchored, + ) } - private fun decodeSigningKey(encoded: String): PublicKey = - KeyFactory.getInstance("Ed25519").generatePublic( - X509EncodedKeySpec(Base64.getDecoder().decode(encoded)), - ) + private fun decodeSigningKey(encoded: String): ByteArray = ProtocolBase64Url.decodeExact( + encoded, + Ed25519Crypto.PUBLIC_KEY_BYTES, + "Ed25519 public key", + ) + + private fun ByteArray.toHex(): String = joinToString("") { "%02x".format(it) } +} + +sealed interface ActiveStudyRecord { + data class Active(val envelopeBytes: ByteArray) : ActiveStudyRecord + + /** + * Durable point of no return for participant-requested deletion. + * + * The signed envelope is deliberately absent: once this record is persisted, startup has + * enough information only to finish erasing local state, never to reactivate or upload it. + */ + data class DeletionPending( + val experimentId: String, + val maximumLocalBytes: Long, + ) : ActiveStudyRecord { + init { + require(StudyConfiguration.ID.matches(experimentId)) { "Invalid deletion experiment ID" } + require(maximumLocalBytes in StudyConfiguration.MINIMUM_LOCAL_BYTES..StudyConfiguration.MAXIMUM_LOCAL_BYTES) { + "Invalid deletion storage quota" + } + } + } } interface ActiveStudyStore { - suspend fun load(): ByteArray? + suspend fun load(): ActiveStudyRecord? suspend fun save(envelopeBytes: ByteArray) + suspend fun markDeletionPending(experimentId: String, maximumLocalBytes: Long) suspend fun clear() } diff --git a/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/ConfigurationProtocolTest.kt b/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/ConfigurationProtocolTest.kt index 95b4ecc..e5a13b5 100644 --- a/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/ConfigurationProtocolTest.kt +++ b/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/ConfigurationProtocolTest.kt @@ -1,265 +1,199 @@ package cool.linc.androiddatacollector.core.protocol -import cool.linc.androiddatacollector.core.definition.* +import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration +import cool.linc.androiddatacollector.core.definition.ExportConfiguration +import cool.linc.androiddatacollector.core.definition.LocationConfiguration +import cool.linc.androiddatacollector.core.definition.LocationPriority +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url +import cool.linc.androiddatacollector.core.definition.SignerIdentity +import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import cool.linc.androiddatacollector.core.definition.UploadConfiguration +import java.nio.ByteBuffer +import java.security.KeyPair import java.security.KeyPairGenerator import java.security.Signature import java.time.Instant -import java.util.Base64 import org.junit.Assert.assertArrayEquals import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue import org.junit.Test class ConfigurationProtocolTest { @Test - fun canonicalCodecRoundTripsEveryCollector() { + fun canonicalCodecUsesClosedWorldJcsAndIntegerPhysicalUnits() { val configuration = configuration() val canonical = StudyConfigurationCodec.encode(configuration) + val text = canonical.toString(Charsets.UTF_8) assertEquals(configuration, StudyConfigurationCodec.decode(canonical)) - assertArrayEquals(canonical, StudyConfigurationCodec.canonicalize(pretty(configuration))) + assertArrayEquals(canonical, StudyConfigurationCodec.canonicalize(pretty(canonical))) + assertTrue(text.startsWith("{\"assigned_participant_id\":")) + assertTrue(text.contains("\"minimum_client_version\":\"7\"")) + assertTrue(text.contains("\"minimum_displacement_millimeters\":5000")) + assertFalse(text.contains("minimum_app_version")) + assertFalse(text.contains("tink")) } @Test - fun strictCodecRejectsNonCanonicalAndUnknownFields() { + fun canonicalizerNormalizesIntegralSpellingsButDecoderRequiresCanonicalBytes() { val canonical = StudyConfigurationCodec.encode(configuration()) - val withWhitespace = canonical.toString(Charsets.UTF_8).replaceFirst("{", "{\n") - val withUnknown = canonical.toString(Charsets.UTF_8).replaceFirst("{", "{\"unknown\":true,") - - assertThrows(IllegalArgumentException::class.java) { - StudyConfigurationCodec.decode(withWhitespace.toByteArray()) - } - assertThrows(IllegalArgumentException::class.java) { - StudyConfigurationCodec.decode(withUnknown.toByteArray()) - } - } - - @Test - fun uploadBlockRoundTripsWhenPresentAndAbsent() { - val enabled = configuration(UploadConfiguration("https://intake.example.invalid/v1", 360, false)) - val canonical = StudyConfigurationCodec.encode(enabled) - - assertEquals(enabled, StudyConfigurationCodec.decode(canonical)) - assertArrayEquals(canonical, StudyConfigurationCodec.canonicalize(pretty(enabled))) - - // An absent upload block is encoded as an empty object, so the root key set stays fixed. - val disabled = StudyConfigurationCodec.encode(configuration()) - assertEquals(null, StudyConfigurationCodec.decode(disabled).upload) - assertEquals(true, disabled.toString(Charsets.UTF_8).contains("\"upload\":{}")) - } - - @Test - fun uploadBlockRejectsPartialAndInsecureEndpoints() { - val canonical = StudyConfigurationCodec.encode( - configuration(UploadConfiguration("https://intake.example.invalid/v1", 360, false)), - ).toString(Charsets.UTF_8) + val exponent = canonical.toString(Charsets.UTF_8) + .replace("\"duration_hours\":24", "\"duration_hours\":2.4e1") + .toByteArray() - // Cleartext must be refused in the schema, not left to the platform to block later. + assertThrows(IllegalArgumentException::class.java) { StudyConfigurationCodec.decode(exponent) } + assertArrayEquals(canonical, StudyConfigurationCodec.canonicalize(exponent)) assertThrows(IllegalArgumentException::class.java) { - StudyConfigurationCodec.decode( - canonical.replace("https://intake", "http://intake").toByteArray(), - ) - } - // A half-declared block must not silently inherit a default cadence. - assertThrows(IllegalArgumentException::class.java) { - StudyConfigurationCodec.decode( - canonical.replace(",\"interval_minutes\":360", "").toByteArray(), - ) - } - assertThrows(IllegalArgumentException::class.java) { - StudyConfigurationCodec.decode( - canonical.replace("\"interval_minutes\":360", "\"interval_minutes\":0").toByteArray(), + StudyConfigurationCodec.canonicalize( + exponent.toString(Charsets.UTF_8).replace("2.4e1", "24.5").toByteArray(), ) } } @Test - fun v1IdentityLocalizationAndOccurrenceBoundsAreStrict() { - assertEquals("每日確認", LocalizedText("Default", mapOf("zh-TW" to "每日確認")).resolve("zh-Hant-TW")) - assertThrows(IllegalArgumentException::class.java) { - configuration().copy(assignedParticipantId = "contains space") - } - assertThrows(IllegalArgumentException::class.java) { - configuration().copy( - interventions = listOf( - InterventionConfiguration( - "too-frequent", - NotificationAction("Check-in", "Check in now."), - listOf(InterventionTrigger("every-minute", IntervalSchedule(0, 1, RelativeClock.CALENDAR_TIME), 5)), - ), - ), - ) + fun hostileJsonAndLegacyV1AreRejected() { + val canonical = StudyConfigurationCodec.encode(configuration()) + val text = canonical.toString(Charsets.UTF_8) + val duplicate = text.replaceFirst("{", "{\"assigned_participant_id\":null,").toByteArray() + val unknown = text.replaceFirst("{", "{\"unknown\":true,").toByteArray() + val legacy = text + .replace("\"minimum_client_version\":\"7\",", "\"minimum_app_version\":7,") + .replace("\"platform\":\"android\",", "") + .toByteArray() + val leadingZero = text.replace("\"minimum_client_version\":\"7\"", "\"minimum_client_version\":\"07\"") + .toByteArray() + val loneSurrogate = text.replace("Protocol test", "\\ud800").toByteArray() + val malformedUtf8 = canonical.copyOf().also { it[text.indexOf("Protocol test")] = 0x80.toByte() } + val extremeExponent = text.replace("\"duration_hours\":24", "\"duration_hours\":1e999999999") + .toByteArray() + + listOf(duplicate, unknown, legacy, leadingZero, loneSurrogate, malformedUtf8, extremeExponent).forEach { hostile -> + assertThrows(IllegalArgumentException::class.java) { StudyConfigurationCodec.canonicalize(hostile) } } } @Test - fun verifierAuthenticatesSignerAndValidityWindow() { - val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - val envelope = sign(keyPair) - val verifier = verifier(mapOf("test-signer" to encoded(keyPair))) - - val verified = verifier.verify(envelope) - assertEquals("protocol-test", verified.configuration.experimentId) - assertEquals(true, verified.signerAnchored) - val tampered = envelope.copyOf().also { it[it.lastIndex] = (it.last() + 1).toByte() } - assertThrows(IllegalArgumentException::class.java) { verifier.verify(tampered) } + fun adccfg01HasFixedEd25519TailAndRejectsOldSignatureLengthFraming() { + val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() + val configurationBytes = StudyConfigurationCodec.encode(configuration(publicKey(pair))) + val signature = sign(pair, configurationBytes) + val envelope = SignedConfigurationCodec.encode( + SignedConfigurationEnvelope("test-signer", configurationBytes, signature), + ) + val decoded = SignedConfigurationCodec.decode(envelope) + + assertEquals("ADCCFG01", envelope.copyOfRange(0, 8).toString(Charsets.US_ASCII)) + assertEquals("test-signer".length, ByteBuffer.wrap(envelope, 8, 2).short.toInt()) + assertEquals(configurationBytes.size, ByteBuffer.wrap(envelope, 10, 4).int) + assertArrayEquals(signature, envelope.copyOfRange(envelope.size - 64, envelope.size)) + assertArrayEquals(configurationBytes, decoded.configurationBytes) + + val legacy = ByteBuffer.allocate(envelope.size + 2) + .put(envelope, 0, 14) + .putShort(64) + .put(envelope, 14, envelope.size - 14) + .array() + assertThrows(IllegalArgumentException::class.java) { SignedConfigurationCodec.decode(legacy) } } @Test - fun aConfigurationCertifiesItselfWhenNoSignerIsPinned() { - // What lets one published app run any researcher's study: the signing key travels inside - // the signed bytes, so verification needs nothing but the file. - val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - val envelope = sign(keyPair) + fun verifierReturnsAuthenticatedConfigurationProvenance() { + val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() + val configurationBytes = StudyConfigurationCodec.encode(configuration(publicKey(pair))) + val signature = sign(pair, configurationBytes) + val envelope = SignedConfigurationCodec.encode( + SignedConfigurationEnvelope("test-signer", configurationBytes, signature), + ) + val verified = verifier(mapOf("test-signer" to publicKey(pair))).verify(envelope) - val verified = verifier(emptyMap()).verify(envelope) + assertTrue(verified.signerAnchored) + assertEquals("test-signer", verified.signerKeyId) + assertArrayEquals(configurationBytes, verified.canonicalConfigurationBytes) + assertArrayEquals(signature, verified.signature) + assertTrue(Regex("[0-9a-f]{64}").matches(verified.configurationSha256)) - assertEquals("protocol-test", verified.configuration.experimentId) - // The signature proves the file is unchanged, not who wrote it, and the app must say so. - assertEquals(false, verified.signerAnchored) val tampered = envelope.copyOf().also { it[it.lastIndex] = (it.last() + 1).toByte() } assertThrows(IllegalArgumentException::class.java) { verifier(emptyMap()).verify(tampered) } + assertThrows(IllegalArgumentException::class.java) { + verifier(mapOf("other-signer" to publicKey(pair))).verify(envelope) + } } @Test - fun pinningRefusesAnyOtherSignerAndAnySubstitutedKey() { - val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - val other = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - - // A build that pins signers accepts only those, whatever the configuration declares. + fun rawKeyEncodingRejectsPaddingAndDerWireKeys() { + val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() assertThrows(IllegalArgumentException::class.java) { - verifier(mapOf("someone-else" to encoded(other))).verify(sign(keyPair)) + configuration(publicKey(pair) + "=") } - - // And a configuration cannot claim a pinned key ID while carrying a different key: the - // pinned key wins, so the signature made with the impostor's key fails. assertThrows(IllegalArgumentException::class.java) { - verifier(mapOf("test-signer" to encoded(other))).verify(sign(keyPair)) + configuration(java.util.Base64.getEncoder().encodeToString(pair.public.encoded)) } } @Test - fun envelopeSignerMustMatchTheConfiguration() { - val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - val configurationBytes = StudyConfigurationCodec.encode(configuration(signerPublicKey = encoded(keyPair))) - val signature = Signature.getInstance("Ed25519").run { - initSign(keyPair.private) - update(configurationBytes) - sign() - } - // The envelope label sits outside the signature, so a mismatch has to be caught explicitly. - val relabelled = SignedConfigurationCodec.encode( - SignedConfigurationEnvelope("other-label", configurationBytes, signature), + fun verifierEnforcesActivationAndDecimalClientVersion() { + val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() + val bytes = StudyConfigurationCodec.encode(configuration(publicKey(pair))) + val envelope = SignedConfigurationCodec.encode( + SignedConfigurationEnvelope("test-signer", bytes, sign(pair, bytes)), ) - assertThrows(IllegalArgumentException::class.java) { verifier(emptyMap()).verify(relabelled) } - } - - @Test - fun aMalformedSignatureIsRejectedLikeAnyOtherBadOne() { - // The two tests above flip the last byte, which lands here about 6% of the time and used - // to fail them at that rate: an Ed25519 signature is (R, S) with S little-endian, so the - // last byte is its most significant, and the JDK throws SignatureException rather than - // returning false once S passes the group order. 0xFF is over it every time, so this - // holds the rejection type still instead of leaving it to which byte a corrupt file lost. - val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - val malformed = sign(keyPair).also { it[it.lastIndex] = 0xFF.toByte() } - - assertThrows(IllegalArgumentException::class.java) { verifier(emptyMap()).verify(malformed) } assertThrows(IllegalArgumentException::class.java) { - verifier(mapOf("test-signer" to encoded(keyPair))).verify(malformed) + ConfigurationVerifier(emptyMap(), 6, now = { Instant.parse("2027-01-01T00:00:00Z") }).verify(envelope) } - } - - private fun encoded(keyPair: java.security.KeyPair) = - Base64.getEncoder().encodeToString(keyPair.public.encoded) - - private fun sign(keyPair: java.security.KeyPair): ByteArray { - val configurationBytes = StudyConfigurationCodec.encode( - configuration(signerPublicKey = encoded(keyPair)), - ) - val signature = Signature.getInstance("Ed25519").run { - initSign(keyPair.private) - update(configurationBytes) - sign() + assertThrows(IllegalArgumentException::class.java) { + ConfigurationVerifier(emptyMap(), 7, now = { Instant.parse("2031-01-01T00:00:00Z") }).verify(envelope) } - return SignedConfigurationCodec.encode( - SignedConfigurationEnvelope("test-signer", configurationBytes, signature), - ) } private fun verifier(pinned: Map) = ConfigurationVerifier( pinned, - appVersionCode = 1, - now = { Instant.parse("2026-07-31T00:00:00Z") }, + clientVersion = 7, + now = { Instant.parse("2027-01-01T00:00:00Z") }, ) - private fun configuration( - upload: UploadConfiguration? = null, - signerPublicKey: String = TEST_SIGNER_PUBLIC_KEY, - ) = StudyConfiguration( - schemaVersion = StudyConfiguration.CURRENT_SCHEMA_VERSION, - experimentId = "protocol-test", - configurationId = "protocol-config", - assignedParticipantId = "arm-a-017", - issuedAt = Instant.parse("2026-01-01T00:00:00Z"), - expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, - title = "Protocol test", - researcherName = "Protocol researcher", - researcherContact = "protocol@example.invalid", - purpose = "Exercise strict signed configuration encoding.", - durationHours = 24, - consentDocumentVersion = "v1", - consentSummary = "Protocol test consent summary.", - collectors = listOf( - AppLifecycleConfiguration(true), - AccelerometerConfiguration(true, 100_000, 1_000_000), - NetworkStateConfiguration(true, true), - NetworkUsageConfiguration(false, setOf(NetworkTransport.WIFI, NetworkTransport.MOBILE), 5), - UsageEventsConfiguration(false, 30), - LocationConfiguration(false, 10_000, 5_000, 30_000, 5f, LocationPriority.BALANCED), - KeyboardTouchConfiguration(false, 60), - ), - surveys = listOf( - SurveyDefinition( - id = "daily-survey", - title = LocalizedText("Daily check-in", mapOf("zh-TW" to "每日確認")), - description = LocalizedText("Tell us how today went."), - questions = listOf( - ShortTextQuestion("daily-note", LocalizedText("Anything to share?"), false, 500), - ), - ), - ), - interventions = listOf( - InterventionConfiguration( - id = "daily-check", - action = SurveyAction("Daily check-in", "A short survey is ready.", "daily-survey"), - triggers = listOf( - InterventionTrigger( - "after-hour", - OneTimeSchedule(60, RelativeClock.ACTIVE_RUNNING_TIME), - 1_440, - ), - ), + private fun sign(pair: KeyPair, bytes: ByteArray): ByteArray = Signature.getInstance("Ed25519").run { + initSign(pair.private) + update(bytes) + sign() + } + + private fun publicKey(pair: KeyPair): String = ProtocolBase64Url.encode(pair.public.encoded.copyOfRange(12, 44)) + + private fun configuration(signerPublicKey: String = ProtocolBase64Url.encode(ByteArray(32) { 1 })) = + StudyConfiguration( + schemaVersion = 1, + experimentId = "protocol-test", + configurationId = "protocol-config", + assignedParticipantId = null, + issuedAt = Instant.parse("2026-01-01T00:00:00Z"), + expiresAt = Instant.parse("2030-01-01T00:00:00Z"), + platform = "android", + minimumClientVersion = 7, + title = "Protocol test", + researcherName = "Protocol researcher", + researcherContact = "protocol@example.invalid", + purpose = "Exercise the destructive Protocol v1 contract.", + durationHours = 24, + consentDocumentVersion = "v1", + consentSummary = "Protocol test consent summary.", + collectors = listOf( + AppLifecycleConfiguration(true), + LocationConfiguration(false, 10_000, 5_000, 30_000, 5_000, LocationPriority.BALANCED), ), - ), - maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", signerPublicKey), - export = ExportConfiguration( - "test-hpke", - "{\"primaryKeyId\":123456,\"key\":[]}", - ), - upload = upload, - ) + surveys = emptyList(), + interventions = emptyList(), + maximumLocalBytes = 16_777_216, + signer = SignerIdentity("test-signer", signerPublicKey), + export = ExportConfiguration("test-hpke", ProtocolBase64Url.encode(ByteArray(32) { 2 })), + upload = UploadConfiguration("https://intake.example.invalid/v1", 60, false), + ) - private fun pretty(configuration: StudyConfiguration): ByteArray = - StudyConfigurationCodec.encode(configuration) - .toString(Charsets.UTF_8) - .replace("{", "{\n") - .replace(",", ",\n") - .toByteArray() + private fun pretty(canonical: ByteArray): ByteArray = canonical.toString(Charsets.UTF_8) + .replace("{", "{\n") + .replace(",", ",\n") + .toByteArray() } - -private const val TEST_SIGNER_PUBLIC_KEY = - "MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0=" diff --git a/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLinkTest.kt b/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLinkTest.kt new file mode 100644 index 0000000..205c0e1 --- /dev/null +++ b/core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLinkTest.kt @@ -0,0 +1,72 @@ +package cool.linc.androiddatacollector.core.protocol + +import java.net.URI +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Test + +class JoinLinkTest { + @Test + fun canonicalJoinLinkRoundTripsAnOpaqueHttpsArtifact() { + val link = JoinLink( + URI("https://artifacts.example.invalid/join/dGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4"), + "0".repeat(64), + "0123456789ABCDEFFEDCBA9876543210", + ) + + assertEquals(link, JoinLink.parse(link.encode())) + assertEquals("0123 4567 89AB CDEF FEDC BA98 7654 3210", link.displayFingerprint()) + } + + @Test + fun artifactUrlProfileRejectsValuesThatPlatformParsersWouldNormalizeDifferently() { + listOf( + "https://EXAMPLE.invalid:443/a/../config.adccfg", + "https://artifacts.example.invalid/config.adccfg?download=1", + "https://artifacts.example.invalid/a//config.adccfg", + "https://artifacts.example.invalid/a/%63onfig.adccfg", + "https://127.0.0.1/config.adccfg", + ).forEach { artifact -> + assertThrows(artifact, IllegalArgumentException::class.java) { + JoinLink(URI(artifact), "0".repeat(64), "A".repeat(32)) + } + } + } + + @Test + fun parserRejectsMutableOrAmbiguousJoinLinks() { + val valid = JoinLink( + URI("https://artifacts.example.invalid/config.adccfg"), + "0".repeat(64), + "A".repeat(32), + ).encode() + val encodedArtifact = "https%3A%2F%2Fartifacts.example.invalid%2Fconfig.adccfg" + + listOf( + valid.replace("adc://", "https://"), + valid.replace("artifact=", "unknown=x&artifact="), + valid.replace("&sha256=", "&sha256=${"0".repeat(64)}&sha256="), + valid.replace("https%3A", "http%3A"), + valid.replace( + encodedArtifact, + "https%3A%2F%2Fuser%40artifacts.example.invalid%2Fconfig.adccfg", + ), + valid.replace(encodedArtifact, "$encodedArtifact%23mutable"), + valid.replace("%2F", "%2f"), + "$valid&extra=1", + ).forEach { hostile -> + assertThrows(hostile, IllegalArgumentException::class.java) { JoinLink.parse(hostile) } + } + } + + @Test + fun encoderRejectsAnArtifactWhoseEscapedJoinLinkExceedsTheWireBound() { + val link = JoinLink( + URI("https://artifacts.example.invalid/${"a/".repeat(1_000)}config.adccfg"), + "0".repeat(64), + "A".repeat(32), + ) + + assertThrows(IllegalArgumentException::class.java) { link.encode() } + } +} diff --git a/core/storage/src/androidTest/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStoreTest.kt b/core/storage/src/androidTest/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStoreTest.kt index 6502db3..4553b10 100644 --- a/core/storage/src/androidTest/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStoreTest.kt +++ b/core/storage/src/androidTest/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStoreTest.kt @@ -193,6 +193,21 @@ class EncryptedExperimentStoreTest { Unit } + @Test + fun corruptAppendJournalFailsClosedAndIsNotDeleted() = runBlocking { + store.initialize(runningMetadata()) + val metadata = storageFiles().single { it.name.endsWith(".metadata.adc") } + val transaction = requireNotNull(metadata.parentFile).resolve( + metadata.name.replace(".metadata.adc", ".transaction.adc"), + ) + transaction.writeBytes(byteArrayOf(0x01)) + + assertThrows(IllegalArgumentException::class.java) { + runBlocking { store.loadMetadata() } + } + assertTrue("a corrupt journal must remain available for diagnosis", transaction.exists()) + } + private fun runningMetadata() = StudyMetadata.initial(experimentId, "encrypted-store-config") .copy(state = ExperimentState.RUNNING) diff --git a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecovery.kt b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecovery.kt new file mode 100644 index 0000000..d692e7d --- /dev/null +++ b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecovery.kt @@ -0,0 +1,97 @@ +package cool.linc.androiddatacollector.core.storage + +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.StudyMetadata + +internal data class AppendRecoveryResult( + val metadata: StudyMetadata, + val rewriteMetadata: Boolean, +) + +/** Resolves the one-event write-ahead transaction against the durable event tail. */ +internal object AppendTransactionRecovery { + fun recover( + main: StudyMetadata, + transaction: StudyMetadata?, + durableLastSequence: Long, + durableTail: RecordedEvent?, + ): AppendRecoveryResult { + require(durableLastSequence >= 0) { "Invalid durable event boundary" } + validateLastEvents(main) + if (transaction == null) { + require(main.eventCount == durableLastSequence) { + "Durable event tail has no matching append transaction" + } + require(durableTail == null) { "Unexpected recovery tail without a transaction" } + return AppendRecoveryResult(main, rewriteMetadata = false) + } + + validateLastEvents(transaction) + if (transaction.eventCount == main.eventCount) { + validateImmutableIdentity(main, transaction) + require(durableLastSequence == main.eventCount) { "Committed transaction boundary mismatch" } + require(durableTail == null) { "Same-boundary transaction must not carry a recovery tail" } + return AppendRecoveryResult(main, rewriteMetadata = false) + } + + require(transaction.eventCount == main.eventCount + 1) { + "Append transaction is not a one-event successor" + } + validateStableMetadata(main, transaction) + val appended = transaction.lastEvents.values.singleOrNull { + it.sequenceNumber == transaction.eventCount + } ?: error("Append transaction does not identify exactly one newest event") + require(transaction.lastEvents == main.lastEvents + (appended.collectorId to appended)) { + "Append transaction rewrites unrelated latest events" + } + + return when (durableLastSequence) { + main.eventCount -> { + require(durableTail == null) { "Non-durable transaction has an event tail" } + AppendRecoveryResult(main, rewriteMetadata = false) + } + transaction.eventCount -> { + require(durableTail == appended) { "Append transaction does not match the durable event tail" } + AppendRecoveryResult(transaction, rewriteMetadata = true) + } + else -> error("Durable event tail is outside the append transaction boundary") + } + } + + private fun validateLastEvents(metadata: StudyMetadata) { + require(metadata.lastEvents.all { (collectorId, event) -> collectorId == event.collectorId }) { + "Latest-event collector key mismatch" + } + if (metadata.eventCount == 0L) { + require(metadata.lastEvents.isEmpty()) { "Empty metadata has latest events" } + } else { + require(metadata.lastEvents.values.count { it.sequenceNumber == metadata.eventCount } == 1) { + "Metadata does not identify its newest event" + } + } + } + + private fun validateStableMetadata(main: StudyMetadata, transaction: StudyMetadata) { + validateImmutableIdentity(main, transaction) + require(transaction.state == main.state) { "Append transaction study state changed" } + require(transaction.transitions == main.transitions) { "Append transaction transitions changed" } + require(transaction.uploadedThroughSequence == main.uploadedThroughSequence) { + "Append transaction upload watermark changed" + } + require(transaction.retainedFromSequence == main.retainedFromSequence) { + "Append transaction retained floor changed" + } + } + + private fun validateImmutableIdentity(main: StudyMetadata, transaction: StudyMetadata) { + require(transaction.experimentId == main.experimentId) { "Append transaction experiment changed" } + require(transaction.configurationId == main.configurationId) { "Append transaction configuration changed" } + require(transaction.participantInstanceId == main.participantInstanceId) { + "Append transaction participant changed" + } + require(transaction.assignedParticipantId == main.assignedParticipantId) { + "Append transaction assignment changed" + } + } + +} diff --git a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedActiveStudyStore.kt b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedActiveStudyStore.kt index f8e53d3..7c42742 100644 --- a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedActiveStudyStore.kt +++ b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedActiveStudyStore.kt @@ -4,6 +4,7 @@ import android.content.Context import android.security.keystore.KeyGenParameterSpec import android.security.keystore.KeyProperties import android.util.AtomicFile +import cool.linc.androiddatacollector.core.protocol.ActiveStudyRecord import cool.linc.androiddatacollector.core.protocol.ActiveStudyStore import java.nio.ByteBuffer import java.security.KeyStore @@ -23,7 +24,7 @@ class EncryptedActiveStudyStore( private val keyStore = KeyStore.getInstance(ANDROID_KEYSTORE).apply { load(null) } private val atomicFile = AtomicFile(context.noBackupFilesDir.resolve(FILE_NAME)) - override suspend fun load(): ByteArray? = withContext(Dispatchers.IO) { + override suspend fun load(): ActiveStudyRecord? = withContext(Dispatchers.IO) { mutex.withLock { if (!atomicFile.baseFile.exists()) return@withLock null val bytes = atomicFile.readFully() @@ -32,22 +33,34 @@ class EncryptedActiveStudyStore( } val key = keyStore.getKey(KEY_ALIAS, null) as? SecretKey ?: error("Active-study encryption key is unavailable") - decrypt(bytes, key) + decodeRecord(decrypt(bytes, key)) } } override suspend fun save(envelopeBytes: ByteArray) = withContext(Dispatchers.IO) { require(envelopeBytes.size in 1..MAXIMUM_PLAINTEXT_BYTES) { "Invalid signed study size" } mutex.withLock { - val encrypted = encrypt(envelopeBytes, getOrCreateKey()) - val output = atomicFile.startWrite() - try { - output.write(encrypted) - atomicFile.finishWrite(output) - } catch (failure: Throwable) { - atomicFile.failWrite(output) - throw failure - } + writeRecord(byteArrayOf(ACTIVE_RECORD) + envelopeBytes, getOrCreateKey()) + } + } + + override suspend fun markDeletionPending( + experimentId: String, + maximumLocalBytes: Long, + ) = withContext(Dispatchers.IO) { + val deletion = ActiveStudyRecord.DeletionPending(experimentId, maximumLocalBytes) + mutex.withLock { + require(atomicFile.baseFile.exists()) { "No active study to delete" } + val key = keyStore.getKey(KEY_ALIAS, null) as? SecretKey + ?: error("Active-study encryption key is unavailable") + val id = deletion.experimentId.toByteArray(Charsets.US_ASCII) + val plaintext = ByteBuffer.allocate(1 + 1 + id.size + Long.SIZE_BYTES) + .put(DELETION_RECORD) + .put(id.size.toByte()) + .put(id) + .putLong(deletion.maximumLocalBytes) + .array() + writeRecord(plaintext, key) } } @@ -75,6 +88,39 @@ class EncryptedActiveStudyStore( .array() } + private fun writeRecord(plaintext: ByteArray, key: SecretKey) { + val encrypted = encrypt(plaintext, key) + val output = atomicFile.startWrite() + try { + output.write(encrypted) + atomicFile.finishWrite(output) + } catch (failure: Throwable) { + atomicFile.failWrite(output) + throw failure + } + } + + private fun decodeRecord(plaintext: ByteArray): ActiveStudyRecord { + require(plaintext.isNotEmpty()) { "Active-study record is empty" } + val buffer = ByteBuffer.wrap(plaintext) + return when (buffer.get()) { + ACTIVE_RECORD -> { + require(buffer.hasRemaining()) { "Active-study envelope is empty" } + ActiveStudyRecord.Active(ByteArray(buffer.remaining()).also(buffer::get)) + } + DELETION_RECORD -> { + require(buffer.remaining() >= 1 + Long.SIZE_BYTES) { "Deletion record is truncated" } + val idLength = buffer.get().toInt() and 0xff + require(idLength in 3..64 && buffer.remaining() == idLength + Long.SIZE_BYTES) { + "Deletion record length is invalid" + } + val experimentId = ByteArray(idLength).also(buffer::get).toString(Charsets.US_ASCII) + ActiveStudyRecord.DeletionPending(experimentId, buffer.long) + } + else -> throw IllegalArgumentException("Unsupported active-study record") + } + } + private fun decrypt( encoded: ByteArray, key: SecretKey, @@ -116,8 +162,10 @@ class EncryptedActiveStudyStore( const val IV_BYTES = 12 const val TAG_BITS = 128 const val MAXIMUM_PLAINTEXT_BYTES = 1_100_000 - const val MAXIMUM_ENCODED_BYTES = MAXIMUM_PLAINTEXT_BYTES + 64 + const val MAXIMUM_ENCODED_BYTES = MAXIMUM_PLAINTEXT_BYTES + 65 val HEADER = "ADCACT01".toByteArray(Charsets.US_ASCII) - val MINIMUM_ENCODED_BYTES = HEADER.size + IV_BYTES + TAG_BITS / 8 + 1 + val MINIMUM_ENCODED_BYTES = HEADER.size + IV_BYTES + TAG_BITS / 8 + 2 + const val ACTIVE_RECORD: Byte = 0 + const val DELETION_RECORD: Byte = 1 } } diff --git a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStore.kt b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStore.kt index 44d2507..a3c9dad 100644 --- a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStore.kt +++ b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/EncryptedExperimentStore.kt @@ -52,9 +52,9 @@ class EncryptedExperimentStore( return@withLock null } val key = existingKey() ?: error("Encrypted experiment key is unavailable") - // Framing and sequence contiguity are checked from the plaintext frame headers, so - // opening a study is linear in frames rather than in bytes decrypted. Event payloads - // are authenticated when they are actually read, in readEvents. + // Framing and contiguity come from plaintext frame headers. Normal opening decrypts no + // event payload; the unique journal+durable-tail recovery state authenticates only the + // last one through readDurableTail below. val scan = scanEvents( key, fromSequenceInclusive = 1, @@ -62,26 +62,41 @@ class EncryptedExperimentStore( recoverTail = true, decryptPayloads = false, ) - val encodedMetadata = decryptMetadata(metadataFile.readFully(), key) - var metadata = StudyDataJsonCodec.decodeMetadata( - encodedMetadata, + val mainMetadata = StudyDataJsonCodec.decodeMetadata( + decryptMetadata(metadataFile.readFully(), key), + ) + val hasTransaction = transactionFile.baseFile.exists() + val transactionMetadata = if (hasTransaction) { + StudyDataJsonCodec.decodeMetadata( + decryptDocument(transactionFile.readFully(), key, TRANSACTION_HEADER), + ) + } else { + null + } + val durableTail = if ( + transactionMetadata != null && + transactionMetadata.eventCount == mainMetadata.eventCount + 1 && + transactionMetadata.eventCount == scan.lastSequence + ) { + readDurableTail(key, scan.lastSequence) + } else { + null + } + val recovery = AppendTransactionRecovery.recover( + main = mainMetadata, + transaction = transactionMetadata, + durableLastSequence = scan.lastSequence, + durableTail = durableTail, + ) + val metadata = StudyDataJsonCodec.reconcileMetadata( + recovery.metadata, scan.firstSequence, scan.lastSequence, ) - if (transactionFile.baseFile.exists()) { - val recovered = runCatching { - StudyDataJsonCodec.decodeMetadata( - decryptDocument(transactionFile.readFully(), key, TRANSACTION_HEADER), - scan.firstSequence, - scan.lastSequence, - ) - }.getOrNull() - if (recovered != null && recovered.eventCount == scan.lastSequence && - recovered.eventCount == metadata.eventCount + 1 - ) { - metadata = recovered - writeMetadata(encryptDocument(StudyDataJsonCodec.encodeMetadata(metadata), key, METADATA_HEADER)) - } + if (recovery.rewriteMetadata || metadata != recovery.metadata) { + writeMetadata(encryptDocument(StudyDataJsonCodec.encodeMetadata(metadata), key, METADATA_HEADER)) + } + if (hasTransaction) { transactionFile.delete() } require(metadata.experimentId == experimentId) { "Encrypted experiment ID mismatch" } @@ -133,12 +148,21 @@ class EncryptedExperimentStore( withContext(Dispatchers.IO) { mutex.withLock { appendTransaction(event, metadata) } } private fun appendTransaction(event: RecordedEvent, metadata: StudyMetadata) { - requireNotNull(persistedMetadata) { "Study storage is not initialized" } + val current = requireNotNull(persistedMetadata) { "Study storage is not initialized" } require(event.sequenceNumber == persistedSequenceBoundary + 1) { "Non-contiguous event append" } require(metadata.eventCount == event.sequenceNumber && metadata.nextSequenceNumber == event.sequenceNumber + 1) { "Atomic metadata boundary mismatch" } require(metadata.experimentId == experimentId) { "Experiment ID mismatch" } + val validated = AppendTransactionRecovery.recover( + main = current, + transaction = metadata, + durableLastSequence = event.sequenceNumber, + durableTail = event, + ) + check(validated.rewriteMetadata && validated.metadata == metadata) { + "Atomic append metadata is not a valid one-event successor" + } val key = existingKey() ?: error("Encrypted experiment key is unavailable") val encoded = StudyDataJsonCodec.encodeMetadata(metadata) require(encoded.size <= MAXIMUM_METADATA_BYTES) { "Experiment metadata quota exceeded" } @@ -356,6 +380,22 @@ class EncryptedExperimentStore( return EventScan(firstSequence, lastSequence) } + /** Reuses the range reader so recovery authenticates one event and adds no second framing path. */ + private fun readDurableTail(key: SecretKey, sequenceNumber: Long): RecordedEvent { + var durableTail: RecordedEvent? = null + val scan = scanEvents( + key = key, + fromSequenceInclusive = sequenceNumber, + upToSequenceInclusive = sequenceNumber, + recoverTail = false, + ) { event -> + check(durableTail == null) { "Durable event boundary is not unique" } + durableTail = event + } + require(scan.lastSequence == sequenceNumber) { "Durable event tail is unavailable" } + return requireNotNull(durableTail) { "Durable event tail was not decoded" } + } + private fun recoverTrailingPartialFrame( file: RandomAccessFile, lastCompleteOffset: Long, diff --git a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataJsonCodec.kt b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataJsonCodec.kt index 8d82d3f..01215c8 100644 --- a/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataJsonCodec.kt +++ b/core/storage/src/main/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataJsonCodec.kt @@ -73,54 +73,18 @@ internal object StudyDataJsonCodec { .toString() .toByteArray(Charsets.UTF_8) - /** - * Reconciles the persisted metadata against what is actually on disk. - * - * [scanFirstSequence] and [scanLastSequence] describe the surviving events, and are both 0 when - * no segment exists. The stored counter is authoritative for how far the study has counted, - * because reclaimed events are gone from disk but their sequence numbers must never be reissued. - * The scan is authoritative for the tail, because an event is fsynced before the metadata naming - * it and a crash in between leaves the counter one behind the durable truth. - */ - fun decodeMetadata( - bytes: ByteArray, - scanFirstSequence: Long, - scanLastSequence: Long, - ): StudyMetadata { + /** Decodes exactly the persisted boundary; append recovery reconciles it with the event log. */ + fun decodeMetadata(bytes: ByteArray): StudyMetadata { val root = JSONObject(bytes.toString(Charsets.UTF_8)) root.requireExactKeys(METADATA_KEYS) val transitionsJson = root.getJSONArray("transitions") val storedNextSequence = root.getLong("next_sequence_number") val storedRetainedFrom = root.getLong("retained_from_sequence") val uploadedThrough = root.getLong("uploaded_through_sequence") - - val nextSequenceNumber: Long - val retainedFrom: Long - if (scanFirstSequence == 0L) { - // Reclaiming never removes the newest segment, so an empty store means no event was - // ever written. A populated counter here would mean the log was lost, not reclaimed. - require(storedNextSequence == 1L) { - "Experiment metadata references events that are not durable" - } - nextSequenceNumber = 1L - retainedFrom = 1L - } else { - // The floor is persisted before the segments below it are unlinked, so finding more on - // disk than the floor claims just means an interrupted reclaim; the next pass finishes - // it. Finding *less* means a prefix disappeared without being reclaimed, which is - // indistinguishable from tampering and must not be opened. - require(scanFirstSequence <= storedRetainedFrom) { - "Event segments below the retained floor are missing" - } - require(storedNextSequence <= scanLastSequence + 1) { - "Experiment metadata references an event that is not durable" - } - nextSequenceNumber = maxOf(storedNextSequence, scanLastSequence + 1) - retainedFrom = scanFirstSequence - } + require(storedNextSequence > 0) { "Invalid persisted event boundary" } // Claiming an endpoint received something that was never durable would let it be reclaimed // before it was ever sent. - require(uploadedThrough in 0 until nextSequenceNumber) { + require(uploadedThrough in 0 until storedNextSequence) { "Experiment metadata claims an upload beyond the lifetime event count" } @@ -131,11 +95,13 @@ internal object StudyDataJsonCodec { transitions = List(transitionsJson.length()) { index -> decodeTransition(transitionsJson.getJSONObject(index)) }, - eventCount = nextSequenceNumber - 1, - nextSequenceNumber = nextSequenceNumber, + eventCount = storedNextSequence - 1, + nextSequenceNumber = storedNextSequence, lastEvents = root.getJSONObject("last_events").let { events -> events.keys().asSequence().associateWith { collectorId -> - decodeEvent(events.getJSONObject(collectorId).toString().toByteArray(Charsets.UTF_8)) + decodeEvent(events.getJSONObject(collectorId).toString().toByteArray(Charsets.UTF_8)).also { + require(it.collectorId == collectorId) { "Latest-event collector key mismatch" } + } } }, participantInstanceId = root.getString("participant_instance_id"), @@ -144,10 +110,32 @@ internal object StudyDataJsonCodec { occurrences.keys().asSequence().associateWith { id -> decodeOccurrence(occurrences.getJSONObject(id)) } }, uploadedThroughSequence = uploadedThrough, - retainedFromSequence = retainedFrom, + retainedFromSequence = storedRetainedFrom, ) } + /** Validates the recovered metadata boundary and reconciles an interrupted prefix eviction. */ + fun reconcileMetadata( + metadata: StudyMetadata, + scanFirstSequence: Long, + scanLastSequence: Long, + ): StudyMetadata { + if (scanFirstSequence == 0L) { + require(scanLastSequence == 0L && metadata.eventCount == 0L) { + "Experiment metadata references events that are not durable" + } + return metadata + } + require(scanFirstSequence in 1..scanLastSequence) { "Invalid durable event range" } + require(metadata.eventCount == scanLastSequence) { "Metadata does not name the durable event tail" } + // The floor is persisted before old segments are unlinked. Extra prefix segments therefore + // mean eviction was interrupted; a missing segment below the stored floor is corruption. + require(scanFirstSequence <= metadata.retainedFromSequence) { + "Event segments below the retained floor are missing" + } + return metadata.copy(retainedFromSequence = scanFirstSequence) + } + fun encodeEvent(event: RecordedEvent): ByteArray = JSONObject() .put("sequence_number", event.sequenceNumber) .put("collector_id", event.collectorId) diff --git a/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecoveryTest.kt b/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecoveryTest.kt new file mode 100644 index 0000000..853d027 --- /dev/null +++ b/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/AppendTransactionRecoveryTest.kt @@ -0,0 +1,121 @@ +package cool.linc.androiddatacollector.core.storage + +import cool.linc.androiddatacollector.core.model.InterventionOccurrence +import cool.linc.androiddatacollector.core.model.OccurrenceState +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.model.StudyMetadata +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Test + +class AppendTransactionRecoveryTest { + @Test + fun crashAfterEventFsyncRecoversTheEventAndAtomicOccurrenceSideEffect() { + val main = initial() + val event = event(1, "battery_state.v1") + val occurrence = occurrence() + val transaction = main.withEvent(event).copy( + occurrences = mapOf(occurrence.occurrenceId to occurrence), + ) + + val result = AppendTransactionRecovery.recover(main, transaction, 1, event) + + assertEquals(transaction, result.metadata) + assertTrue(result.rewriteMetadata) + } + + @Test + fun crashAfterJournalWriteDiscardsThePreparedTransaction() { + val main = initial() + val transaction = main.withEvent(event(1, "battery_state.v1")) + + val result = AppendTransactionRecovery.recover(main, transaction, 0, null) + + assertEquals(main, result.metadata) + assertFalse(result.rewriteMetadata) + } + + @Test + fun crashAfterMainCommitKeepsMainAndDiscardsTheLeftoverJournal() { + val committed = initial().withEvent(event(1, "battery_state.v1")) + val staleJournal = committed.copy(occurrences = emptyMap()) + val occurrence = occurrence() + val newerMain = committed.copy(occurrences = mapOf(occurrence.occurrenceId to occurrence)) + + val result = AppendTransactionRecovery.recover( + newerMain, + staleJournal, + newerMain.eventCount, + null, + ) + + assertEquals(newerMain, result.metadata) + assertFalse(result.rewriteMetadata) + } + + @Test + fun sameBoundaryJournalMustStillBelongToTheSameStudyIdentity() { + val main = initial().withEvent(event(1, "battery_state.v1")) + val wrongConfiguration = main.copy(configurationId = "another-config") + + assertThrows(IllegalArgumentException::class.java) { + AppendTransactionRecovery.recover(main, wrongConfiguration, main.eventCount, null) + } + } + + @Test + fun refusesTailWithoutJournalBecauseMetadataSideEffectsCannotBeReconstructed() { + assertThrows(IllegalArgumentException::class.java) { + AppendTransactionRecovery.recover(initial(), null, 1, null) + } + } + + @Test + fun refusesStaleOrMismatchedLatestEventMaps() { + val main = initial().withEvent(event(1, "battery_state.v1")) + val appended = event(2, "temporal_context.v1") + val staleTransaction = main.copy( + eventCount = 2, + nextSequenceNumber = 3, + lastEvents = main.lastEvents + ("temporal_context.v1" to appended.copy(sequenceNumber = 1)), + ) + assertThrows(IllegalArgumentException::class.java) { + AppendTransactionRecovery.recover(main, staleTransaction, 2, appended) + } + + val wrongTail = appended.copy(payloadType = "OTHER") + assertThrows(IllegalArgumentException::class.java) { + AppendTransactionRecovery.recover(main, main.withEvent(appended), 2, wrongTail) + } + } + + private fun initial() = StudyMetadata.initial("recovery-test", "recovery-config") + + private fun StudyMetadata.withEvent(event: RecordedEvent) = copy( + eventCount = event.sequenceNumber, + nextSequenceNumber = event.sequenceNumber + 1, + lastEvents = lastEvents + (event.collectorId to event), + ) + + private fun event(sequence: Long, collectorId: String) = RecordedEvent( + sequenceNumber = sequence, + collectorId = collectorId, + payloadSchemaVersion = 1, + observedTime = ResearchTime(sequence, sequence, "boot-test"), + payloadType = "TEST_EVENT", + fields = emptyMap(), + ) + + private fun occurrence() = InterventionOccurrence( + occurrenceId = "a".repeat(64), + interventionId = "daily-ema", + triggerId = "random-window", + scheduleKey = "random:2026-08-04:morning", + scheduledFor = ResearchTime(1_000, 1_000, "boot-test"), + expiresAtUtcMillis = 2_000, + state = OccurrenceState.SCHEDULED, + ) +} diff --git a/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataReconciliationTest.kt b/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataReconciliationTest.kt new file mode 100644 index 0000000..92d5ff5 --- /dev/null +++ b/core/storage/src/test/kotlin/cool/linc/androiddatacollector/core/storage/StudyDataReconciliationTest.kt @@ -0,0 +1,54 @@ +package cool.linc.androiddatacollector.core.storage + +import cool.linc.androiddatacollector.core.model.RecordedEvent +import cool.linc.androiddatacollector.core.model.ResearchTime +import cool.linc.androiddatacollector.core.model.StudyMetadata +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Test + +class StudyDataReconciliationTest { + @Test + fun interruptedEvictionFloorIsAdoptedBeforeTheNextAppend() { + val first = event(1) + val stored = StudyMetadata.initial("reconcile-test", "reconcile-config") + .withEvent(first) + .copy(uploadedThroughSequence = 1, retainedFromSequence = 2) + + val reconciled = StudyDataJsonCodec.reconcileMetadata(stored, 1, 1) + assertEquals(1L, reconciled.retainedFromSequence) + + val second = event(2) + val successor = reconciled.withEvent(second) + val recovered = AppendTransactionRecovery.recover(reconciled, successor, 2, second) + assertEquals(successor, recovered.metadata) + assertTrue(recovered.rewriteMetadata) + } + + @Test + fun missingPrefixBelowThePersistedFloorFailsClosed() { + val stored = StudyMetadata.initial("reconcile-test", "reconcile-config") + .withEvent(event(1)) + .withEvent(event(2)) + + assertThrows(IllegalArgumentException::class.java) { + StudyDataJsonCodec.reconcileMetadata(stored, 2, 2) + } + } + + private fun StudyMetadata.withEvent(event: RecordedEvent) = copy( + eventCount = event.sequenceNumber, + nextSequenceNumber = event.sequenceNumber + 1, + lastEvents = lastEvents + (event.collectorId to event), + ) + + private fun event(sequence: Long) = RecordedEvent( + sequenceNumber = sequence, + collectorId = "battery_state.v1", + payloadSchemaVersion = 1, + observedTime = ResearchTime(sequence, sequence, "boot-test"), + payloadType = "BATTERY_STATE", + fields = emptyMap(), + ) +} diff --git a/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlanner.kt b/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlanner.kt index 48f2ae8..5c710f6 100644 --- a/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlanner.kt +++ b/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlanner.kt @@ -3,8 +3,10 @@ package cool.linc.androiddatacollector.core.application import cool.linc.androiddatacollector.core.definition.DailyLocalSchedule import cool.linc.androiddatacollector.core.definition.IntervalSchedule import cool.linc.androiddatacollector.core.definition.OneTimeSchedule +import cool.linc.androiddatacollector.core.definition.RandomWindowSchedule import cool.linc.androiddatacollector.core.definition.RelativeClock import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.InterventionTrigger import cool.linc.androiddatacollector.core.model.ExperimentState import cool.linc.androiddatacollector.core.model.InterventionOccurrence import cool.linc.androiddatacollector.core.model.OccurrenceState @@ -13,12 +15,17 @@ import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.TransitionReason import java.nio.charset.StandardCharsets import java.security.MessageDigest +import java.security.SecureRandom import java.time.Instant +import java.time.LocalDate import java.time.LocalTime import java.time.ZoneId +import kotlin.math.abs -/** Pure, deterministic planner. Android owns only the final best-effort WorkManager delay. */ -class InterventionSchedulePlanner { +/** Local planner; randomized times become durable occurrences before Android schedules work. */ +class InterventionSchedulePlanner( + private val randomIndex: (Int) -> Int = SecureRandom()::nextInt, +) { fun next( configuration: StudyConfiguration, metadata: StudyMetadata, @@ -26,13 +33,26 @@ class InterventionSchedulePlanner { zoneId: ZoneId, triggerId: String? = null, ): List { - if (metadata.state in TERMINAL_STATES) return emptyList() + if (metadata.state != ExperimentState.RUNNING) return emptyList() val firstStart = metadata.transitions.firstOrNull { it.reason == TransitionReason.PARTICIPANT_STARTED }?.time ?: return emptyList() - val lifetimeEnd = now.wallTimeUtcMillis - elapsedMillis(firstStart, now) + + val effectiveStartWallUtcMillis = now.wallTimeUtcMillis - elapsedMillis(firstStart, now) + val lifetimeEnd = effectiveStartWallUtcMillis + configuration.durationHours * HOUR_MILLIS return configuration.interventions.flatMap { intervention -> intervention.triggers.filter { triggerId == null || it.id == triggerId }.mapNotNull { trigger -> + if (trigger.schedule is RandomWindowSchedule) { + return@mapNotNull nextRandomOccurrence( + configuration, + metadata, + intervention.id, + trigger, + effectiveStartWallUtcMillis, + lifetimeEnd, + now, + zoneId, + ) + } val candidates = when (val schedule = trigger.schedule) { is OneTimeSchedule -> sequenceOf( relativeCandidate(schedule.offsetMinutes.toLong(), schedule.clock, metadata, firstStart, now) to @@ -46,6 +66,7 @@ class InterventionSchedulePlanner { } is DailyLocalSchedule -> dailyCandidates(firstStart, lifetimeEnd, schedule.localTime, zoneId) .mapIndexed { index, scheduled -> scheduled to "daily:$index" } + is RandomWindowSchedule -> error("Handled above") } candidates.takeWhile { it.first.wallTimeUtcMillis < lifetimeEnd } .map { (scheduled, key) -> @@ -71,6 +92,100 @@ class InterventionSchedulePlanner { } } + private fun nextRandomOccurrence( + configuration: StudyConfiguration, + metadata: StudyMetadata, + interventionId: String, + trigger: InterventionTrigger, + effectiveStartWallUtcMillis: Long, + lifetimeEnd: Long, + now: ResearchTime, + zoneId: ZoneId, + ): InterventionOccurrence? { + val schedule = trigger.schedule as RandomWindowSchedule + val existing = metadata.occurrences.values.filter { + it.interventionId == interventionId && it.triggerId == trigger.id && + it.scheduleKey.startsWith(RANDOM_KEY_PREFIX) + } + existing.filter { it.state in PENDING_STATES } + .minByOrNull { it.scheduledFor.wallTimeUtcMillis } + ?.let { return it } + if (existing.size >= schedule.maximumOccurrencesTotal) return null + val materializedKeys = existing.mapTo(mutableSetOf()) { it.scheduleKey } + // Exact keys prevent a repeated local date from duplicating work. The chronological floor + // prevents wall-clock rollback from reopening elapsed time without treating local-date + // ordering as chronology when the participant crosses the date line. + val chronologicalFloor = existing.maxOfOrNull { it.scheduledFor.wallTimeUtcMillis } + + val firstDate = Instant.ofEpochMilli(effectiveStartWallUtcMillis).atZone(zoneId).toLocalDate() + val finalDate = Instant.ofEpochMilli(lifetimeEnd - 1).atZone(zoneId).toLocalDate() + var date = firstDate + while (!date.isAfter(finalDate)) { + val datePrefix = "$RANDOM_KEY_PREFIX$date:" + val remainingDailyCapacity = schedule.maximumOccurrencesPerDay - + existing.count { it.scheduleKey.startsWith(datePrefix) } + val remainingTotalCapacity = schedule.maximumOccurrencesTotal - existing.size + if (remainingDailyCapacity > 0 && remainingTotalCapacity > 0) { + schedule.localWindows.forEachIndexed { windowIndex, window -> + repeat(schedule.occurrencesPerWindow) { ordinal -> + val key = "$RANDOM_KEY_PREFIX$date:$windowIndex:$ordinal" + if (key in materializedKeys) return@repeat + val previousKey = "$RANDOM_KEY_PREFIX$date:$windowIndex:${ordinal - 1}" + val notBefore = existing.firstOrNull { it.scheduleKey == previousKey } + ?.scheduledFor + ?.wallTimeUtcMillis + ?.plus(schedule.minimumSeparationMinutes * MINUTE_MILLIS) + ?: Long.MIN_VALUE + val remainingUnmaterializedInWindow = + (ordinal + 1 until schedule.occurrencesPerWindow).count { later -> + "$RANDOM_KEY_PREFIX$date:$windowIndex:$later" !in materializedKeys + } + val remainingInWindow = minOf( + remainingUnmaterializedInWindow, + remainingDailyCapacity - 1, + remainingTotalCapacity - 1, + ) + val latestMinute = window.endMinute - 1 - + remainingInWindow * schedule.minimumSeparationMinutes + val eligible = (window.startMinute..latestMinute).mapNotNull { minute -> + val wallMillis = localMinuteInstant(date, minute, zoneId) + ?: return@mapNotNull null + if ( + wallMillis < effectiveStartWallUtcMillis || + wallMillis < now.wallTimeUtcMillis || + (chronologicalFloor != null && wallMillis <= chronologicalFloor) || + wallMillis < notBefore || + wallMillis >= lifetimeEnd + ) return@mapNotNull null + val separated = existing.all { occurrence -> + abs(occurrence.scheduledFor.wallTimeUtcMillis - wallMillis) >= + schedule.minimumSeparationMinutes * MINUTE_MILLIS + } + wallMillis.takeIf { separated } + } + if (eligible.isEmpty()) return@repeat + val wallMillis = eligible[randomIndex(eligible.size)] + val scheduled = estimateResearchTime(wallMillis, now) + return InterventionOccurrence( + occurrenceId = occurrenceId(configuration, interventionId, trigger.id, key), + interventionId = interventionId, + triggerId = trigger.id, + scheduleKey = key, + scheduledFor = scheduled, + expiresAtUtcMillis = minOf( + wallMillis + trigger.availabilityMinutes * MINUTE_MILLIS, + lifetimeEnd, + ), + state = OccurrenceState.SCHEDULED, + ) + } + } + } + date = date.plusDays(1) + } + return null + } + private fun relativeCandidate( offsetMinutes: Long, clock: RelativeClock, @@ -123,7 +238,7 @@ class InterventionSchedulePlanner { val firstDate = Instant.ofEpochMilli(firstStart.wallTimeUtcMillis).atZone(zoneId).toLocalDate() val time = LocalTime.parse(localTime) return generateSequence(firstDate) { it.plusDays(1) } - .map { date -> date.atTime(time).atZone(zoneId).toInstant().toEpochMilli() } + .mapNotNull { date -> localMinuteInstant(date, time.hour * 60 + time.minute, zoneId) } .filter { it >= firstStart.wallTimeUtcMillis } .takeWhile { it < lifetimeEnd } .map { estimateResearchTime(it, firstStart) } @@ -159,7 +274,17 @@ class InterventionSchedulePlanner { private companion object { const val MINUTE_MILLIS = 60_000L const val HOUR_MILLIS = 60 * MINUTE_MILLIS - val TERMINAL_STATES = setOf(ExperimentState.COMPLETED, ExperimentState.WITHDRAWN) + const val RANDOM_KEY_PREFIX = "random:" val PENDING_STATES = setOf(OccurrenceState.SCHEDULED, OccurrenceState.POSTING) } } + +/** + * Resolves a signed local minute without silently moving it outside its window. Gap minutes do not + * exist and are skipped. During an overlap, the first chronological occurrence is chosen. + */ +internal fun localMinuteInstant(date: LocalDate, minute: Int, zoneId: ZoneId): Long? { + val local = date.atTime(LocalTime.of(minute / 60, minute % 60)) + return zoneId.rules.getValidOffsets(local) + .minOfOrNull { offset -> local.atOffset(offset).toInstant().toEpochMilli() } +} diff --git a/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt b/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt index ccf14f0..c3eeecf 100644 --- a/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt +++ b/core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt @@ -6,17 +6,23 @@ import cool.linc.androiddatacollector.core.collector.AccessStatus import cool.linc.androiddatacollector.core.collector.CollectorRegistry import cool.linc.androiddatacollector.core.collector.StudyAccessGateway import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.SurveyAction import cool.linc.androiddatacollector.core.export.ExportReceipt import cool.linc.androiddatacollector.core.model.ExperimentState import cool.linc.androiddatacollector.core.model.InterventionOccurrence +import cool.linc.androiddatacollector.core.model.OccurrenceState import cool.linc.androiddatacollector.core.model.ResearchTime import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.StudyStore import cool.linc.androiddatacollector.core.protocol.ActiveStudyStore +import cool.linc.androiddatacollector.core.protocol.ActiveStudyRecord +import cool.linc.androiddatacollector.core.protocol.JoinLink import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import cool.linc.androiddatacollector.core.runtime.CommandResult import cool.linc.androiddatacollector.core.runtime.ExperimentRuntime +import cool.linc.androiddatacollector.core.runtime.OccurrenceClaimResult import cool.linc.androiddatacollector.core.runtime.OccurrenceDispatch +import cool.linc.androiddatacollector.core.runtime.OccurrenceExpiryResult import cool.linc.androiddatacollector.core.runtime.RuntimeSnapshot import cool.linc.androiddatacollector.core.runtime.SurveyAnswer import cool.linc.androiddatacollector.core.runtime.SurveySubmissionResult @@ -35,7 +41,9 @@ import kotlinx.coroutines.sync.withLock fun interface StudyVerifier { fun verify(envelopeBytes: ByteArray): VerifiedConfiguration } -fun interface StudyStoreFactory { fun create(configuration: StudyConfiguration): StudyStore } +fun interface StudyStoreFactory { + fun create(experimentId: String, maximumLocalBytes: Long): StudyStore +} fun interface ExperimentRuntimeFactory { fun create(configuration: StudyConfiguration, store: StudyStore, availableAccess: () -> Set): ExperimentRuntime @@ -49,12 +57,22 @@ interface StudyCollectionHost { interface StudyWorkScheduler { fun schedule(configuration: StudyConfiguration) - /** Replaces all pending occurrence work after state, clock, reboot, or time-zone recovery. */ - fun replaceInterventionWork(configuration: StudyConfiguration, occurrences: List) + /** Rebuilds delivery and durable expiry work after state, clock, reboot, or time-zone recovery. */ + fun replaceInterventionWork( + configuration: StudyConfiguration, + deliveries: List, + expiries: List, + ) /** Adds the successor of a completed trigger without disturbing unrelated work. */ fun enqueueOccurrence(configuration: StudyConfiguration, occurrence: InterventionOccurrence) + /** Cancels delivery/expiry work and visible notifications while a study is paused. */ + fun cancelInterventionWork(experimentId: String, occurrenceIds: Set) + + /** Idempotently removes notifications that durable occurrence state proves are no longer visible. */ + fun cancelInterventionNotifications(occurrenceIds: Set) + /** * Cancels reminders and the study deadline, leaving scheduled delivery in place. * @@ -62,14 +80,19 @@ interface StudyWorkScheduler { * are still owed to the researcher — stranding them here would defeat the point of uploading at * all, since the participant may never perform a manual export. */ - fun cancelCollectionWork(experimentId: String) + fun cancelCollectionWork(experimentId: String, occurrenceIds: Set) /** Cancels everything, including undelivered work. Used when the data itself is going away. */ fun cancel(experimentId: String) } fun interface StudyExporter { - suspend fun export(configuration: StudyConfiguration, metadata: StudyMetadata, events: StudyStore, destination: OutputStream): ExportReceipt + suspend fun export( + configuration: VerifiedConfiguration, + metadata: StudyMetadata, + events: StudyStore, + destination: OutputStream, + ): ExportReceipt } /** @@ -79,14 +102,30 @@ fun interface StudyExporter { * the endpoint stores ciphertext it cannot read. Returning normally means the endpoint confirmed * receipt and the watermark may advance; anything else must throw. */ -fun interface StudyUploader { +interface StudyUploader { + /** + * Recovers the one durable staged bundle before deciding whether another upload is needed. + * A stage already covered by [StudyMetadata.uploadedThroughSequence] is safe to remove; every + * other stage must remain byte-for-byte identical for its next request. + */ + suspend fun reconcile(configuration: VerifiedConfiguration, metadata: StudyMetadata) + suspend fun upload( - configuration: StudyConfiguration, + configuration: VerifiedConfiguration, metadata: StudyMetadata, events: StudyStore, fromSequence: Long, toSequence: Long, ): ExportReceipt + + /** Prevents a staged-but-not-started request and cancels any request already in flight. */ + suspend fun prepareDeletion() + + /** Removes the durable stage only after the matching upload watermark was persisted. */ + suspend fun acknowledge(bundleId: java.util.UUID) + + /** Removes every staged upload when the participant deletes the study. */ + suspend fun clear() } /** @@ -97,6 +136,7 @@ fun interface StudyUploader { */ class StudyUploadException( val reasonCode: String, + val retryable: Boolean, cause: Throwable? = null, ) : Exception(reasonCode, cause) { init { @@ -114,8 +154,15 @@ data class UploadStatus( val pendingCount: Long, val lastSuccessAtUtcMillis: Long? = null, val lastFailureCode: String? = null, + val lastFailureRetryable: Boolean? = null, ) +sealed interface UploadAttemptResult { + data object NoWork : UploadAttemptResult + data class Confirmed(val receipt: ExportReceipt) : UploadAttemptResult + data class Failed(val reasonCode: String, val retryable: Boolean) : UploadAttemptResult +} + data class StudySessionSnapshot( val initialized: Boolean = false, val configuration: StudyConfiguration? = null, @@ -130,6 +177,8 @@ data class StudySessionSnapshot( /** Kept separate from [lastExport] so a background upload never overwrites what the * participant sees for their own export. */ val upload: UploadStatus? = null, + /** A durable deletion tombstone exists; collection and upload must never resume. */ + val deletionPending: Boolean = false, val incidentCode: String? = null, ) @@ -166,9 +215,9 @@ class StudySessionManager( private val sessionMutex = Mutex() /** - * Serialises uploads against each other without blocking participant actions. Network I/O must - * never run under [sessionMutex]: a stalled request would freeze pause, withdraw and delete for - * as long as the connection hangs. + * Serialises uploads against each other without blocking pause or withdraw. Network I/O never + * runs under [sessionMutex]. Deletion first persists its tombstone, asks the uploader to cancel, + * then waits here so it cannot erase a stage while request teardown is still using that file. */ private val uploadMutex = Mutex() private val mutableSnapshot = MutableStateFlow(StudySessionSnapshot()) @@ -176,29 +225,41 @@ class StudySessionManager( private var runtime: ExperimentRuntime? = null private var studyStore: StudyStore? = null + private var verifiedConfiguration: VerifiedConfiguration? = null private var runtimeObservation: Job? = null + private var deletionPending = false suspend fun initialize() = sessionMutex.withLock { check(!mutableSnapshot.value.initialized) { "Study session is already initialized" } try { - val saved = activeStudyStore.load() - if (saved == null) { - mutableSnapshot.value = StudySessionSnapshot(initialized = true) - } else { - activate(saved, persistEnvelope = false) + when (val saved = activeStudyStore.load()) { + null -> mutableSnapshot.value = StudySessionSnapshot(initialized = true) + is ActiveStudyRecord.Active -> activate(saved.envelopeBytes, persistEnvelope = false, joinLink = null) + is ActiveStudyRecord.DeletionPending -> { + deletionPending = true + mutableSnapshot.value = StudySessionSnapshot(deletionPending = true) + completePendingDeletion(saved) + mutableSnapshot.value = StudySessionSnapshot(initialized = true) + } } } catch (failure: Throwable) { failure.rethrowCancellation() mutableSnapshot.update { - it.copy(initialized = true, incidentCode = INCIDENT_STUDY_RECOVERY_FAILED) + it.copy( + initialized = true, + deletionPending = deletionPending, + incidentCode = INCIDENT_STUDY_RECOVERY_FAILED, + ) } } } - suspend fun importSignedConfiguration(bytes: ByteArray) = sessionMutex.withLock { - check(runtime == null) { "Delete the current study before importing another" } + suspend fun importSignedConfiguration(bytes: ByteArray, joinLink: JoinLink? = null) = sessionMutex.withLock { + check(runtime == null && !deletionPending) { + "Finish pending study deletion before importing another" + } try { - activate(bytes, persistEnvelope = true) + activate(bytes, persistEnvelope = true, joinLink = joinLink) } catch (failure: Throwable) { failure.rethrowCancellation() mutableSnapshot.update { it.copy(incidentCode = INCIDENT_STUDY_IMPORT_FAILED) } @@ -235,7 +296,7 @@ class StudySessionManager( } try { workScheduler.schedule(current.configuration) - syncInterventionsLocked(current, replace = true) + syncInterventionsLocked(current) publish(result) } catch (failure: Throwable) { failure.rethrowCancellation() @@ -251,7 +312,10 @@ class StudySessionManager( val result = current.pause() if (result == CommandResult.Success) { collectionHost.stop() - syncInterventionsLocked(current, replace = true) + workScheduler.cancelInterventionWork( + current.configuration.experimentId, + occurrenceIds(current), + ) } publish(result) } @@ -268,25 +332,42 @@ class StudySessionManager( if (result != CommandResult.Success) { collectionHost.stop() } else { - syncInterventionsLocked(current, replace = true) + syncInterventionsLocked(current) } publish(result) } - suspend fun rescheduleInterventions() = sessionMutex.withLock { - runtime?.let { syncInterventionsLocked(it, replace = true) } + suspend fun rescheduleInterventions(recoverStalePosting: Boolean = false) = sessionMutex.withLock { + runtime?.let { current -> + if (current.snapshot.value.metadata?.state == ExperimentState.RUNNING) { + syncInterventionsLocked(current, recoverStalePosting) + } else { + workScheduler.cancelInterventionWork( + current.configuration.experimentId, + occurrenceIds(current), + ) + } + } + } + + suspend fun claimOccurrenceIfDue(occurrenceId: String): OccurrenceClaimResult = sessionMutex.withLock { + requireRuntime().claimOccurrenceIfDue(occurrenceId) } - suspend fun claimOccurrence(occurrenceId: String): OccurrenceDispatch? = sessionMutex.withLock { - val current = requireRuntime() - current.claimOccurrence(occurrenceId).also { dispatch -> - if (dispatch == null) scheduleNextLocked(current, occurrenceId) - } + suspend fun expireOccurrenceIfDue(occurrenceId: String): OccurrenceExpiryResult = sessionMutex.withLock { + requireRuntime().expireOccurrenceIfDue(occurrenceId) } - suspend fun markNotificationPosted(occurrenceId: String) = sessionMutex.withLock { + suspend fun markNotificationPosted(occurrenceId: String): Boolean = sessionMutex.withLock { requireRuntime().markNotificationPosted(occurrenceId) - scheduleNextLocked(requireRuntime(), occurrenceId) + } + + /** Idempotently restores the trigger chain after an occurrence reaches a durable lifecycle state. */ + suspend fun scheduleSuccessor(occurrenceId: String) = sessionMutex.withLock { + val current = requireRuntime() + if (current.snapshot.value.metadata?.state == ExperimentState.RUNNING) { + scheduleNextLocked(current, occurrenceId) + } } suspend fun openOccurrence(occurrenceId: String): OccurrenceDispatch? = @@ -310,7 +391,7 @@ class StudySessionManager( val current = requireRuntime() val receipt = destination.use { exporter.export( - current.configuration, + requireNotNull(verifiedConfiguration), current.metadataForExport(), requireNotNull(studyStore), it, @@ -328,27 +409,42 @@ class StudySessionManager( * them under [uploadMutex] only. Holding [sessionMutex] across an HTTP request would block the * participant from pausing or withdrawing for as long as the network is unresponsive. */ - suspend fun uploadPending(): CommandResult = uploadMutex.withLock { - val plan = planUpload() ?: return@withLock CommandResult.Success - + suspend fun uploadPending(): UploadAttemptResult = uploadMutex.withLock { + val context = uploadContext() ?: return@withLock UploadAttemptResult.NoWork + val plan = planUpload() ?: run { + try { + // No request will run, but a crash may have left an already-committed stage whose + // manifest still needs removing. + uploader.reconcile(context.configuration, context.metadata) + } catch (failure: Throwable) { + failure.rethrowCancellation() + return@withLock publishUploadFailure(context.metadata, failure) + } + reclaimConfirmedSpace() + return@withLock UploadAttemptResult.NoWork + } val receipt = try { uploader.upload(plan.configuration, plan.metadata, plan.store, plan.from, plan.to) + .also { validateReceipt(plan, it) } } catch (failure: Throwable) { failure.rethrowCancellation() - val reason = (failure as? StudyUploadException)?.reasonCode ?: INCIDENT_UPLOAD_FAILED - // Leave incidentCode alone: a transient upload failure is not a collection incident, - // and overwriting it would bury a storage or access problem the participant must act on. - mutableSnapshot.update { - it.copy( - upload = it.upload?.copy(lastFailureCode = reason) - ?: UploadStatus(plan.from - 1, 0, lastFailureCode = reason), - ) - } - return@withLock CommandResult.Failed(reason) + return@withLock publishUploadFailure(plan.metadata, failure) } - commitUpload(plan, receipt.sequenceBoundary) - CommandResult.Success + try { + commitUploadWatermark(plan, receipt) + uploader.acknowledge(receipt.bundleId) + } catch (failure: Throwable) { + failure.rethrowCancellation() + return@withLock publishUploadFailure( + plan.metadata, + failure, + defaultCode = INCIDENT_UPLOAD_COMMIT_FAILED, + defaultRetryable = true, + ) + } + reclaimConfirmedSpace() + UploadAttemptResult.Confirmed(receipt) } /** @@ -361,6 +457,14 @@ class StudySessionManager( metadata.uploadedThroughSequence >= metadata.eventCount } + private suspend fun uploadContext(): UploadContext? = sessionMutex.withLock { + if (deletionPending) return@withLock null + val current = runtime ?: return@withLock null + if (current.configuration.upload == null) return@withLock null + val metadata = current.snapshot.value.metadata ?: return@withLock null + UploadContext(requireNotNull(verifiedConfiguration), metadata) + } + /** Null when the study does not upload, has no active runtime, or has nothing undelivered. */ private suspend fun planUpload(): UploadPlan? = sessionMutex.withLock { val current = runtime ?: return@withLock null @@ -374,7 +478,7 @@ class StudySessionManager( // Ask for everything outstanding. How much actually fits is decided while the bundle // streams, and comes back in the receipt. UploadPlan( - configuration = current.configuration, + configuration = requireNotNull(verifiedConfiguration), metadata = metadata, store = requireNotNull(studyStore), from = from, @@ -382,16 +486,28 @@ class StudySessionManager( ) } - private suspend fun commitUpload(plan: UploadPlan, deliveredThrough: Long) = sessionMutex.withLock { + private fun validateReceipt(plan: UploadPlan, receipt: ExportReceipt) { + require(receipt.configurationSha256 == plan.configuration.configurationSha256) { + "Upload receipt configuration digest mismatch" + } + require(receipt.firstSequence == plan.from) { "Upload receipt range start mismatch" } + require(receipt.lastSequence in plan.from..plan.to) { "Upload receipt range end mismatch" } + require(receipt.eventCount == receipt.lastSequence - receipt.firstSequence + 1) { + "Upload receipt event count mismatch" + } + require(receipt.byteCount in 1..MAXIMUM_UPLOAD_BYTES) { "Upload receipt byte count is out of bounds" } + require(SHA256_HEX.matches(receipt.sha256)) { "Upload receipt digest is invalid" } + } + + private suspend fun commitUploadWatermark(plan: UploadPlan, receipt: ExportReceipt) = sessionMutex.withLock { // The study may have been withdrawn, deleted or replaced while the request was in flight. - val current = runtime ?: return@withLock - if (current.configuration.experimentId != plan.configuration.experimentId) return@withLock + val current = checkNotNull(runtime) { "Study was deleted during upload" } + check(requireNotNull(verifiedConfiguration).configurationSha256 == plan.configuration.configurationSha256) { + "Study changed during upload" + } // The receipt, not the plan: a budgeted bundle may have stopped short, and the rest goes // out on the next run. - current.confirmUploaded(deliveredThrough) - // Confirmed delivery is the only thing that makes local data reclaimable, so this is the - // one point where reclaiming can make progress. - val updated = current.reclaimLocalSpace() + val updated = current.confirmUploaded(receipt.lastSequence) mutableSnapshot.update { it.copy( upload = UploadStatus( @@ -403,28 +519,134 @@ class StudySessionManager( } } + private suspend fun reclaimConfirmedSpace() = sessionMutex.withLock { + val current = runtime ?: return@withLock + try { + val updated = current.reclaimLocalSpace() + mutableSnapshot.update { snapshot -> + val upload = snapshot.upload ?: return@update snapshot + snapshot.copy( + upload = upload.copy( + uploadedThroughSequence = updated.uploadedThroughSequence, + pendingCount = updated.eventCount - updated.uploadedThroughSequence, + ), + ) + } + } catch (failure: Throwable) { + failure.rethrowCancellation() + mutableSnapshot.update { it.copy(incidentCode = INCIDENT_RECLAIM_FAILED) } + } + } + + private fun publishUploadFailure( + metadata: StudyMetadata, + failure: Throwable, + defaultCode: String = INCIDENT_UPLOAD_FAILED, + defaultRetryable: Boolean = false, + ): UploadAttemptResult.Failed { + val classified = failure as? StudyUploadException + val reason = classified?.reasonCode ?: defaultCode + val retryable = classified?.retryable ?: defaultRetryable + // An upload failure is not a collection incident. Keep any storage/access incident visible. + mutableSnapshot.update { + it.copy( + upload = it.upload?.copy( + lastFailureCode = reason, + lastFailureRetryable = retryable, + ) ?: UploadStatus( + uploadedThroughSequence = metadata.uploadedThroughSequence, + pendingCount = metadata.eventCount - metadata.uploadedThroughSequence, + lastFailureCode = reason, + lastFailureRetryable = retryable, + ), + ) + } + return UploadAttemptResult.Failed(reason, retryable) + } + + private data class UploadContext( + val configuration: VerifiedConfiguration, + val metadata: StudyMetadata, + ) + private class UploadPlan( - val configuration: StudyConfiguration, + val configuration: VerifiedConfiguration, val metadata: StudyMetadata, val store: StudyStore, val from: Long, val to: Long, ) - suspend fun deleteLocalData() = sessionMutex.withLock { - val current = requireRuntime() - require(current.snapshot.value.metadata?.state in TERMINAL_STATES) { - "Withdraw or complete the study before deleting its data" + suspend fun deleteLocalData() { + val deletion = sessionMutex.withLock { + val current = requireRuntime() + require(current.snapshot.value.metadata?.state in TERMINAL_STATES) { + "Withdraw or complete the study before deleting its data" + } + val target = ActiveStudyRecord.DeletionPending( + current.configuration.experimentId, + current.configuration.maximumLocalBytes, + ) + activeStudyStore.markDeletionPending(target.experimentId, target.maximumLocalBytes) + deletionPending = true + mutableSnapshot.update { it.copy(deletionPending = true) } + runtimeObservation?.cancel() + DeletionContext(target, requireNotNull(studyStore)) } - runtimeObservation?.cancel() - requireNotNull(studyStore).clear() - activeStudyStore.clear() - workScheduler.cancel(current.configuration.experimentId) - collectionHost.stop() - runtime = null - studyStore = null - runtimeObservation = null - mutableSnapshot.value = StudySessionSnapshot(initialized = true) + + // The tombstone is already durable. Quiesce a request without waiting for its full + // network timeout, then take the session upload lock so watermark handling has finished. + uploader.prepareDeletion() + uploadMutex.withLock { + sessionMutex.withLock { + completeDeletion(deletion.target, deletion.store) + runtime = null + studyStore = null + verifiedConfiguration = null + runtimeObservation = null + deletionPending = false + mutableSnapshot.value = StudySessionSnapshot(initialized = true) + } + } + } + + private data class DeletionContext( + val target: ActiveStudyRecord.DeletionPending, + val store: StudyStore, + ) + + private suspend fun completePendingDeletion(deletion: ActiveStudyRecord.DeletionPending) { + completeDeletion( + deletion, + storeFactory.create(deletion.experimentId, deletion.maximumLocalBytes), + ) + deletionPending = false + } + + /** + * Best-effort all cleanup, while keeping the tombstone unless every step succeeds. + * This makes each crash/failure point retryable without ever restoring upload capability. + */ + private suspend fun completeDeletion( + deletion: ActiveStudyRecord.DeletionPending, + store: StudyStore, + ) { + var firstFailure: Exception? = null + suspend fun attempt(block: suspend () -> Unit) { + try { + block() + } catch (failure: Exception) { + failure.rethrowCancellation() + if (firstFailure == null) firstFailure = failure else firstFailure.addSuppressed(failure) + } + } + + attempt { collectionHost.stop() } + attempt { workScheduler.cancel(deletion.experimentId) } + attempt { uploader.clear() } + attempt { store.clear() } + if (firstFailure == null) attempt { activeStudyStore.clear() } + firstFailure?.let { throw it } } fun refreshAccess() { @@ -437,13 +659,25 @@ class StudySessionManager( private suspend fun activate( envelopeBytes: ByteArray, persistEnvelope: Boolean, + joinLink: JoinLink?, ) { + joinLink?.let { expected -> + val actual = java.security.MessageDigest.getInstance("SHA-256") + .digest(envelopeBytes) + .joinToString("") { "%02x".format(it) } + require(actual == expected.artifactSha256) { "Join artifact digest mismatch" } + } val verified = verifier.verify(envelopeBytes) + joinLink?.let { expected -> + require(verified.configuration.signer.fingerprint == expected.displayFingerprint()) { + "Join signer fingerprint mismatch" + } + } val configuration = verified.configuration configuration.collectors.forEach(collectorRegistry::pluginFor) val requirements = requirements(configuration) val access = accessGateway.inspect(requirements) - val createdStore = storeFactory.create(configuration) + val createdStore = storeFactory.create(configuration.experimentId, configuration.maximumLocalBytes) var createdRuntime: ExperimentRuntime? = null try { val created = runtimeFactory.create( @@ -462,6 +696,7 @@ class StudySessionManager( } runtime = created studyStore = createdStore + verifiedConfiguration = verified mutableSnapshot.value = StudySessionSnapshot( initialized = true, configuration = configuration, @@ -497,16 +732,38 @@ class StudySessionManager( publish(result) } - private suspend fun syncInterventionsLocked(current: ExperimentRuntime, replace: Boolean) { + private suspend fun syncInterventionsLocked( + current: ExperimentRuntime, + recoverStalePosting: Boolean = false, + ) { val metadata = current.snapshot.value.metadata ?: return - val plans = schedulePlanner.next( + if (metadata.state != ExperimentState.RUNNING) return + // External side-effect cleanup comes before planning or durable writes. A quota/storage + // failure while ensuring another occurrence must not leave a crash-stale notification. + workScheduler.cancelInterventionNotifications( + metadata.occurrences.values + .filter { + it.state in NON_VISIBLE_OCCURRENCE_STATES || + (recoverStalePosting && it.state == OccurrenceState.POSTING) + } + .mapTo(mutableSetOf()) { it.occurrenceId }, + ) + val deliveries = schedulePlanner.next( current.configuration, metadata, current.now(), java.time.ZoneId.systemDefault(), ).map { current.ensureOccurrence(it) } - if (replace) workScheduler.replaceInterventionWork(current.configuration, plans) - else plans.forEach { workScheduler.enqueueOccurrence(current.configuration, it) } + val surveyInterventionIds = current.configuration.interventions + .filter { it.action is SurveyAction } + .mapTo(mutableSetOf()) { it.id } + val expiries = current.snapshot.value.metadata?.occurrences?.values + ?.filter { occurrence -> + occurrence.state in EXPIRABLE_UNOPENED_OCCURRENCE_STATES || + (occurrence.state == OccurrenceState.OPENED && occurrence.interventionId in surveyInterventionIds) + } + .orEmpty() + workScheduler.replaceInterventionWork(current.configuration, deliveries, expiries) } private suspend fun scheduleNextLocked(current: ExperimentRuntime, completedOccurrenceId: String) { @@ -530,11 +787,17 @@ class StudySessionManager( if (result == CommandResult.Success) { collectionHost.stop() // Not cancel(): the study is over, but its undelivered tail is not. - workScheduler.cancelCollectionWork(current.configuration.experimentId) + workScheduler.cancelCollectionWork( + current.configuration.experimentId, + occurrenceIds(current), + ) } publish(result) } + private fun occurrenceIds(current: ExperimentRuntime): Set = + current.snapshot.value.metadata?.occurrences?.keys.orEmpty() + private fun requirements(configuration: StudyConfiguration): Set = accessPolicy.requirements( configuration, @@ -576,6 +839,17 @@ class StudySessionManager( private companion object { val TERMINAL_STATES = setOf(ExperimentState.COMPLETED, ExperimentState.WITHDRAWN) + val EXPIRABLE_UNOPENED_OCCURRENCE_STATES = setOf( + OccurrenceState.SCHEDULED, + OccurrenceState.POSTING, + OccurrenceState.NOTIFICATION_POSTED, + ) + val NON_VISIBLE_OCCURRENCE_STATES = setOf( + OccurrenceState.SCHEDULED, + OccurrenceState.OPENED, + OccurrenceState.SURVEY_SUBMITTED, + OccurrenceState.EXPIRED, + ) /** * States that can hold deliverable events. A study that ended still uploads, so its tail @@ -593,6 +867,10 @@ class StudySessionManager( const val INCIDENT_COLLECTION_HOST_FAILED = "COLLECTION_HOST_FAILED" const val INCIDENT_WORK_SCHEDULING_FAILED = "WORK_SCHEDULING_FAILED" const val INCIDENT_UPLOAD_FAILED = "UPLOAD_FAILED" + const val INCIDENT_UPLOAD_COMMIT_FAILED = "UPLOAD_COMMIT_FAILED" + const val INCIDENT_RECLAIM_FAILED = "LOCAL_RECLAIM_FAILED" + const val MAXIMUM_UPLOAD_BYTES = 32L * 1024 * 1024 + val SHA256_HEX = Regex("[0-9a-f]{64}") } } diff --git a/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlannerTest.kt b/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlannerTest.kt index cc26208..3299a60 100644 --- a/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlannerTest.kt +++ b/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlannerTest.kt @@ -10,6 +10,8 @@ import cool.linc.androiddatacollector.core.definition.IntervalSchedule import cool.linc.androiddatacollector.core.definition.NotificationAction import cool.linc.androiddatacollector.core.definition.OneTimeSchedule import cool.linc.androiddatacollector.core.definition.RelativeClock +import cool.linc.androiddatacollector.core.definition.RandomLocalWindow +import cool.linc.androiddatacollector.core.definition.RandomWindowSchedule import cool.linc.androiddatacollector.core.definition.SignerIdentity import cool.linc.androiddatacollector.core.definition.StudyConfiguration import cool.linc.androiddatacollector.core.model.ExperimentState @@ -19,9 +21,11 @@ import cool.linc.androiddatacollector.core.model.ResearchTime import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.TransitionReason import java.time.Instant +import java.time.LocalDate import java.time.ZoneId import org.junit.Assert.assertEquals import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertNull import org.junit.Assert.assertTrue import org.junit.Test @@ -134,13 +138,283 @@ class InterventionSchedulePlannerTest { ) } + @Test + fun randomWindowUsesCsprngChoiceThenReusesTheDurableOccurrenceExactly() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("09:00", "12:00")), + occurrencesPerWindow = 2, + maximumOccurrencesPerDay = 2, + maximumOccurrencesTotal = 4, + minimumSeparationMinutes = 30, + ) + val configuration = configuration(schedule) + val metadata = runningMetadata() + val chooseLatest = InterventionSchedulePlanner { bound -> bound - 1 } + val first = chooseLatest.next(configuration, metadata, at(1), ZoneId.of("UTC")).single() + + assertEquals("random:2026-01-01:0:0", first.scheduleKey) + assertEquals(at(11 * 60 + 29).wallTimeUtcMillis, first.scheduledFor.wallTimeUtcMillis) + + val persisted = metadata.copy(occurrences = mapOf(first.occurrenceId to first)) + val afterProcessDeath = InterventionSchedulePlanner { 0 } + .next(configuration, persisted, at(5, "new-boot"), ZoneId.of("Pacific/Kiritimati")) + .single() + assertEquals(first, afterProcessDeath) + + val posted = persisted.copy( + occurrences = mapOf(first.occurrenceId to first.copy(state = OccurrenceState.NOTIFICATION_POSTED)), + ) + val second = InterventionSchedulePlanner { 0 } + .next(configuration, posted, at(5), ZoneId.of("UTC")) + .single() + assertEquals("random:2026-01-01:0:1", second.scheduleKey) + assertEquals(at(11 * 60 + 59).wallTimeUtcMillis, second.scheduledFor.wallTimeUtcMillis) + } + + @Test + fun randomWindowHonorsTheSignedDailyAndTotalCaps() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("09:00", "12:00")), + occurrencesPerWindow = 2, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 1, + minimumSeparationMinutes = 30, + ) + val configuration = configuration(schedule) + val metadata = runningMetadata() + val first = InterventionSchedulePlanner { 0 } + .next(configuration, metadata, at(1), ZoneId.of("UTC")) + .single() + val completed = metadata.copy( + occurrences = mapOf(first.occurrenceId to first.copy(state = OccurrenceState.NOTIFICATION_POSTED)), + ) + + assertTrue( + InterventionSchedulePlanner { 0 } + .next(configuration, completed, at(10), ZoneId.of("UTC")) + .isEmpty(), + ) + } + + @Test + fun randomWindowDoesNotReopenACompletedEarlierDateAfterClockRollback() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("09:00", "10:00")), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 2, + minimumSeparationMinutes = 30, + ) + val configuration = configuration(schedule) + val metadata = runningMetadata() + val tomorrow = InterventionSchedulePlanner { 0 } + .next(configuration, metadata, at(13 * 60), ZoneId.of("UTC")) + .single() + assertEquals("random:2026-01-02:0:0", tomorrow.scheduleKey) + val completed = metadata.copy( + occurrences = mapOf( + tomorrow.occurrenceId to tomorrow.copy(state = OccurrenceState.NOTIFICATION_POSTED), + ), + ) + + assertTrue( + InterventionSchedulePlanner { 0 } + .next(configuration, completed, at(8 * 60), ZoneId.of("UTC")) + .isEmpty(), + ) + } + + @Test + fun randomWindowUsesTheMonotonicStudyAnchorAfterWallClockRollback() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("09:00", "10:00")), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 2, + minimumSeparationMinutes = 30, + ) + val start = ResearchTime( + Instant.parse("2026-01-02T00:00:00Z").toEpochMilli(), + 1_000_000_000, + "boot-rollback", + ) + val afterRollback = ResearchTime( + Instant.parse("2026-01-01T01:00:00Z").toEpochMilli(), + 3_601_000_000_000, + "boot-rollback", + ) + + val occurrence = InterventionSchedulePlanner { 0 } + .next(configuration(schedule), runningMetadata(start), afterRollback, ZoneId.of("UTC")) + .single() + + assertEquals("random:2026-01-01:0:0", occurrence.scheduleKey) + assertEquals( + Instant.parse("2026-01-01T09:00:00Z").toEpochMilli(), + occurrence.scheduledFor.wallTimeUtcMillis, + ) + } + + @Test + fun randomWindowUsesChronologyNotLocalDateOrderingAfterCrossingTheDateLine() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("09:00", "10:00")), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 2, + minimumSeparationMinutes = 30, + ) + val configuration = configuration(schedule) + val metadata = runningMetadata() + val first = InterventionSchedulePlanner { 0 } + .next(configuration, metadata, at(13 * 60), ZoneId.of("Pacific/Kiritimati")) + .single() + assertEquals("random:2026-01-02:0:0", first.scheduleKey) + assertEquals(at(19 * 60).wallTimeUtcMillis, first.scheduledFor.wallTimeUtcMillis) + val completed = metadata.copy( + occurrences = mapOf(first.occurrenceId to first.copy(state = OccurrenceState.NOTIFICATION_POSTED)), + ) + + val chronologicallyLater = InterventionSchedulePlanner { 0 } + .next(configuration, completed, at(20 * 60), ZoneId.of("Etc/GMT+12")) + .single() + + assertEquals("random:2026-01-01:0:0", chronologicallyLater.scheduleKey) + assertEquals(at(21 * 60).wallTimeUtcMillis, chronologicallyLater.scheduledFor.wallTimeUtcMillis) + assertTrue( + chronologicallyLater.scheduledFor.wallTimeUtcMillis > first.scheduledFor.wallTimeUtcMillis, + ) + } + + @Test + fun localMinuteResolutionSkipsDstGapsAndChoosesTheFirstOverlapOccurrence() { + val newYork = ZoneId.of("America/New_York") + + assertNull(localMinuteInstant(LocalDate.of(2026, 3, 8), 2 * 60 + 30, newYork)) + assertEquals( + Instant.parse("2026-11-01T05:30:00Z").toEpochMilli(), + localMinuteInstant(LocalDate.of(2026, 11, 1), 1 * 60 + 30, newYork), + ) + } + + @Test + fun randomDailyCapCountsOnlyMaterializedOccurrencesSoEveningRemainsEligible() { + val schedule = RandomWindowSchedule( + localWindows = listOf( + RandomLocalWindow("09:00", "10:00"), + RandomLocalWindow("18:00", "19:00"), + ), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 1, + minimumSeparationMinutes = 30, + ) + + val occurrence = InterventionSchedulePlanner { 0 } + .next(configuration(schedule), runningMetadata(), at(12 * 60), ZoneId.of("UTC")) + .single() + + assertEquals("random:2026-01-01:1:0", occurrence.scheduleKey) + assertEquals(at(18 * 60).wallTimeUtcMillis, occurrence.scheduledFor.wallTimeUtcMillis) + } + + @Test + fun randomDailyCapTruncatesInSignedWindowOrderBeforeMinuteRandomization() { + val schedule = RandomWindowSchedule( + localWindows = listOf( + RandomLocalWindow("09:00", "10:00"), + RandomLocalWindow("18:00", "19:00"), + ), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 2, + minimumSeparationMinutes = 30, + ) + + val occurrence = InterventionSchedulePlanner { bound -> bound - 1 } + .next(configuration(schedule), runningMetadata(), at(1), ZoneId.of("UTC")) + .single() + + assertEquals("random:2026-01-01:0:0", occurrence.scheduleKey) + assertEquals(at(9 * 60 + 59).wallTimeUtcMillis, occurrence.scheduledFor.wallTimeUtcMillis) + } + + @Test + fun dstGapWindowDoesNotConsumeTheDailyCapBeforeAValidWindow() { + val schedule = RandomWindowSchedule( + localWindows = listOf( + RandomLocalWindow("02:00", "03:00"), + RandomLocalWindow("04:00", "05:00"), + ), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 1, + minimumSeparationMinutes = 30, + ) + val start = researchTime(Instant.parse("2026-03-08T05:00:00Z")) + + val occurrence = InterventionSchedulePlanner { 0 } + .next(configuration(schedule), runningMetadata(start), start, ZoneId.of("America/New_York")) + .single() + + assertEquals("random:2026-03-08:1:0", occurrence.scheduleKey) + assertEquals( + Instant.parse("2026-03-08T08:00:00Z").toEpochMilli(), + occurrence.scheduledFor.wallTimeUtcMillis, + ) + } + + @Test + fun onePromptCapsCanChooseAcrossTheEntireSignedWindow() { + val schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("08:00", "12:00")), + occurrencesPerWindow = 8, + maximumOccurrencesPerDay = 1, + maximumOccurrencesTotal = 1, + minimumSeparationMinutes = 30, + ) + + val occurrence = InterventionSchedulePlanner { bound -> bound - 1 } + .next(configuration(schedule), runningMetadata(), at(1), ZoneId.of("UTC")) + .single() + + assertEquals(at(11 * 60 + 59).wallTimeUtcMillis, occurrence.scheduledFor.wallTimeUtcMillis) + } + + @Test + fun dailyLocalAlsoSkipsDstGapInsteadOfShiftingOutsideTheSignedTime() { + val zone = ZoneId.of("America/New_York") + val start = researchTime(Instant.parse("2026-03-07T05:00:00Z")) + val metadata = runningMetadata(start) + val configuration = configuration(DailyLocalSchedule("02:30"), durationHours = 72) + val first = planner.next(configuration, metadata, start, zone).single() + assertEquals(Instant.parse("2026-03-07T07:30:00Z").toEpochMilli(), first.scheduledFor.wallTimeUtcMillis) + val completed = metadata.copy( + occurrences = mapOf(first.occurrenceId to first.copy(state = OccurrenceState.NOTIFICATION_POSTED)), + ) + + val next = planner.next( + configuration, + completed, + researchTime(Instant.parse("2026-03-07T08:00:00Z")), + zone, + ).single() + + assertEquals(Instant.parse("2026-03-09T06:30:00Z").toEpochMilli(), next.scheduledFor.wallTimeUtcMillis) + } + private fun plan(schedule: InterventionSchedule, metadata: StudyMetadata, nowMinutes: Long) = planner.next(configuration(schedule), metadata, at(nowMinutes), ZoneId.of("UTC")).single() - private fun runningMetadata() = StudyMetadata.initial("schedule-test", "schedule-config").copy( + private fun runningMetadata(start: ResearchTime = at(0)) = StudyMetadata.initial("schedule-test", "schedule-config").copy( state = ExperimentState.RUNNING, transitions = listOf( - transition(ExperimentState.READY, ExperimentState.RUNNING, TransitionReason.PARTICIPANT_STARTED, 0), + ExperimentTransition( + ExperimentState.READY, + ExperimentState.RUNNING, + TransitionReason.PARTICIPANT_STARTED, + start, + ), ), ) @@ -151,18 +425,19 @@ class InterventionSchedulePlannerTest { minutes: Long, ) = ExperimentTransition(from, to, reason, at(minutes)) - private fun configuration(schedule: InterventionSchedule) = StudyConfiguration( + private fun configuration(schedule: InterventionSchedule, durationHours: Int = 48) = StudyConfiguration( schemaVersion = 1, experimentId = "schedule-test", configurationId = "schedule-config", issuedAt = Instant.parse("2025-01-01T00:00:00Z"), expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, + platform = StudyConfiguration.ANDROID_PLATFORM, + minimumClientVersion = 1, title = "Schedule test", researcherName = "Researcher", researcherContact = "research@example.invalid", purpose = "Test deterministic intervention scheduling.", - durationHours = 48, + durationHours = durationHours, consentDocumentVersion = "v1", consentSummary = "Test consent.", assignedParticipantId = null, @@ -176,8 +451,8 @@ class InterventionSchedulePlannerTest { ), ), maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", "x".repeat(32)), - export = ExportConfiguration("export-key", "x".repeat(32)), + signer = SignerIdentity("test-signer", RAW_PUBLIC_KEY), + export = ExportConfiguration("export-key", RAW_PUBLIC_KEY), upload = null, ) @@ -187,7 +462,14 @@ class InterventionSchedulePlannerTest { bootSessionId = boot, ) + private fun researchTime(instant: Instant) = ResearchTime( + wallTimeUtcMillis = instant.toEpochMilli(), + elapsedRealtimeNanos = 1_000_000_000, + bootSessionId = "boot-dst", + ) + private companion object { val BASE_UTC_MILLIS: Long = Instant.parse("2026-01-01T00:00:00Z").toEpochMilli() + const val RAW_PUBLIC_KEY = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" } } diff --git a/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/StudySessionManagerTest.kt b/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/StudySessionManagerTest.kt index 213b90a..ba75a8a 100644 --- a/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/StudySessionManagerTest.kt +++ b/core/study-application/src/test/kotlin/cool/linc/androiddatacollector/core/application/StudySessionManagerTest.kt @@ -11,6 +11,7 @@ import cool.linc.androiddatacollector.core.collector.CollectorPlugin import cool.linc.androiddatacollector.core.collector.CollectorRegistry import cool.linc.androiddatacollector.core.collector.CollectorStatus import cool.linc.androiddatacollector.core.collector.PrivacyClass +import cool.linc.androiddatacollector.core.collector.ProtocolEventContracts import cool.linc.androiddatacollector.core.collector.ResearchClocks import cool.linc.androiddatacollector.core.collector.StudyAccessGateway import cool.linc.androiddatacollector.core.definition.AppLifecycleConfiguration @@ -18,27 +19,42 @@ import cool.linc.androiddatacollector.core.definition.CollectorConfiguration import cool.linc.androiddatacollector.core.definition.ExportConfiguration import cool.linc.androiddatacollector.core.definition.InterventionConfiguration import cool.linc.androiddatacollector.core.definition.InterventionTrigger +import cool.linc.androiddatacollector.core.definition.IntervalSchedule +import cool.linc.androiddatacollector.core.definition.LocalizedText import cool.linc.androiddatacollector.core.definition.NotificationAction import cool.linc.androiddatacollector.core.definition.OneTimeSchedule import cool.linc.androiddatacollector.core.definition.RelativeClock import cool.linc.androiddatacollector.core.definition.SignerIdentity +import cool.linc.androiddatacollector.core.definition.ShortTextQuestion import cool.linc.androiddatacollector.core.definition.StudyConfiguration +import cool.linc.androiddatacollector.core.definition.SurveyAction +import cool.linc.androiddatacollector.core.definition.SurveyDefinition import cool.linc.androiddatacollector.core.definition.UploadConfiguration import cool.linc.androiddatacollector.core.export.ExportReceipt import cool.linc.androiddatacollector.core.model.EventDraft import cool.linc.androiddatacollector.core.model.ExperimentState +import cool.linc.androiddatacollector.core.model.ExperimentTransition import cool.linc.androiddatacollector.core.model.InterventionOccurrence +import cool.linc.androiddatacollector.core.model.OccurrenceState import cool.linc.androiddatacollector.core.model.RecordedEvent import cool.linc.androiddatacollector.core.model.ResearchTime import cool.linc.androiddatacollector.core.model.StorageUsage import cool.linc.androiddatacollector.core.model.StudyMetadata import cool.linc.androiddatacollector.core.model.StudyStore +import cool.linc.androiddatacollector.core.model.TransitionReason import cool.linc.androiddatacollector.core.runtime.CommandResult import cool.linc.androiddatacollector.core.runtime.ExperimentRuntime +import cool.linc.androiddatacollector.core.runtime.OccurrenceClaimResult +import cool.linc.androiddatacollector.core.runtime.OccurrenceExpiryResult import cool.linc.androiddatacollector.core.protocol.ActiveStudyStore +import cool.linc.androiddatacollector.core.protocol.ActiveStudyRecord +import cool.linc.androiddatacollector.core.protocol.JoinLink import cool.linc.androiddatacollector.core.protocol.VerifiedConfiguration import java.io.OutputStream +import java.net.URI +import java.security.MessageDigest import java.time.Instant +import java.util.UUID import kotlinx.coroutines.ExperimentalCoroutinesApi import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.StateFlow @@ -59,7 +75,10 @@ class StudySessionManagerTest { manager.initialize() manager.importSignedConfiguration(byteArrayOf(1, 2, 3)) - assertTrue(fixture.active.saved!!.contentEquals(byteArrayOf(1, 2, 3))) + assertTrue( + (fixture.active.record as ActiveStudyRecord.Active).envelopeBytes + .contentEquals(byteArrayOf(1, 2, 3)), + ) assertEquals(CommandResult.Success, manager.reviewStudy()) assertEquals(CommandResult.Success, manager.acceptConsent()) assertEquals(CommandResult.Success, manager.completeAccessSetup()) @@ -78,6 +97,7 @@ class StudySessionManagerTest { assertEquals(2, fixture.host.stopCount) // Finishing retires reminders and the deadline but leaves delivery scheduled, so a study // that ends with an undelivered backlog still gets it to the researcher. + assertEquals(1, fixture.work.cancelInterventionCount) assertEquals(1, fixture.work.cancelCollectionCount) assertEquals(0, fixture.work.cancelCount) assertEquals(1, fixture.collector.startCount) @@ -92,10 +112,117 @@ class StudySessionManagerTest { manager.deleteLocalData() assertTrue(fixture.store.cleared) - assertNull(fixture.active.saved) + assertNull(fixture.active.record) assertNull(manager.snapshot.value.configuration) // Deleting the data is the point at which delivery has nothing left to deliver. assertEquals(1, fixture.work.cancelCount) + assertTrue((fixture.uploader as FakeUploader).cleared) + } + + @Test + fun deletionTombstoneBlocksUploadAndImportUntilCleanupCanFinish() = runTest { + val uploader = FakeUploader() + val fixture = fixture( + configuration(upload = UploadConfiguration("https://intake.example.invalid/v1", 60, false)), + uploader = uploader, + ) + val manager = fixture.manager + manager.initialize() + manager.importSignedConfiguration(byteArrayOf(1)) + manager.reviewStudy() + manager.acceptConsent() + manager.completeAccessSetup() + manager.start() + manager.finish() + uploader.clearFailure = IllegalStateException("outbox unavailable") + + val failure = runCatching { manager.deleteLocalData() }.exceptionOrNull() + + assertEquals("outbox unavailable", failure?.message) + assertTrue(fixture.active.record is ActiveStudyRecord.DeletionPending) + assertTrue(manager.snapshot.value.deletionPending) + assertTrue(uploader.deletionPrepared) + assertEquals(UploadAttemptResult.NoWork, manager.uploadPending()) + assertTrue(runCatching { manager.importSignedConfiguration(byteArrayOf(2)) }.isFailure) + + uploader.clearFailure = null + manager.deleteLocalData() + assertNull(fixture.active.record) + assertNull(manager.snapshot.value.configuration) + } + + @Test + fun initializationCompletesADeletionTombstoneWithoutReactivatingTheStudy() = runTest { + val deletion = ActiveStudyRecord.DeletionPending("session-test", 16_777_216) + val fixture = fixture(configuration(), activeRecord = deletion) + + fixture.manager.initialize() + + assertNull(fixture.active.record) + assertTrue(fixture.store.cleared) + assertTrue((fixture.uploader as FakeUploader).cleared) + assertEquals(0, fixture.host.startCount) + assertNull(fixture.manager.snapshot.value.configuration) + assertTrue(fixture.manager.snapshot.value.initialized) + } + + @Test + fun joinImportBindsTheExactArtifactAndSignerBeforePersistingIt() = runTest { + val bytes = byteArrayOf(1, 2, 3) + val configuration = configuration() + val valid = JoinLink( + URI("https://artifacts.example.invalid/opaque-token"), + MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }, + configuration.signer.fingerprint.replace(" ", ""), + ) + val accepted = fixture(configuration) + accepted.manager.initialize() + + accepted.manager.importSignedConfiguration(bytes, valid) + + assertTrue((accepted.active.record as ActiveStudyRecord.Active).envelopeBytes.contentEquals(bytes)) + + listOf( + valid.copy(artifactSha256 = "f".repeat(64)), + valid.copy(signerFingerprint = "F".repeat(32)), + ).forEach { hostile -> + val rejected = fixture(configuration) + rejected.manager.initialize() + assertTrue(runCatching { + rejected.manager.importSignedConfiguration(bytes, hostile) + }.isFailure) + assertNull(rejected.active.record) + assertNull(rejected.manager.snapshot.value.configuration) + } + } + + @Test + fun everyDeletionCleanupStepIsAttemptedAndAnyFailureRetainsTheTombstone() = runTest { + listOf("host", "work", "uploader", "store", "active").forEach { failingStep -> + val uploader = FakeUploader() + val fixture = fixture(configuration(), uploader = uploader) + fixture.manager.initialize() + fixture.manager.importSignedConfiguration(byteArrayOf(1)) + fixture.manager.reviewStudy() + fixture.manager.acceptConsent() + fixture.manager.completeAccessSetup() + fixture.manager.start() + fixture.manager.finish() + when (failingStep) { + "host" -> fixture.host.stopFailure = IllegalStateException(failingStep) + "work" -> fixture.work.cancelFailure = IllegalStateException(failingStep) + "uploader" -> uploader.clearFailure = IllegalStateException(failingStep) + "store" -> fixture.store.clearFailure = IllegalStateException(failingStep) + "active" -> fixture.active.clearFailure = IllegalStateException(failingStep) + } + + assertTrue(runCatching { fixture.manager.deleteLocalData() }.isFailure) + assertTrue(fixture.active.record is ActiveStudyRecord.DeletionPending) + assertTrue(fixture.host.stopCount >= 2) + assertTrue(fixture.work.cancelCount >= 1) + assertTrue(uploader.clearAttempts >= 1) + assertTrue(fixture.store.clearAttempts >= 1) + } } @Test @@ -151,6 +278,195 @@ class StudySessionManagerTest { assertEquals(1, fixture.collector.startCount) } + @Test + fun bootOrTimezoneReconciliationRestoresPostedAndOpenedExpiryWork() = runTest { + val notice = InterventionConfiguration( + "notice-one", + NotificationAction("Study check-in", "Check in"), + listOf(InterventionTrigger("after-minute", OneTimeSchedule(1, RelativeClock.CALENDAR_TIME), 60)), + ) + val survey = SurveyDefinition( + "survey-one", + LocalizedText("Survey"), + LocalizedText("One question"), + listOf(ShortTextQuestion("answer-one", LocalizedText("Answer"), false, 40)), + ) + val surveyNotice = InterventionConfiguration( + "survey-notice", + SurveyAction("Study survey", "Answer now", survey.id), + listOf(InterventionTrigger("survey-minute", OneTimeSchedule(1, RelativeClock.CALENDAR_TIME), 60)), + ) + val configuration = configuration(interventions = listOf(notice, surveyNotice), surveys = listOf(survey)) + val posted = occurrence("a".repeat(64), OccurrenceState.NOTIFICATION_POSTED) + val openedNotice = occurrence("b".repeat(64), OccurrenceState.OPENED).copy( + openedAt = ResearchTime(200, 200, "boot-before-recovery"), + ) + val openedSurvey = occurrence( + "c".repeat(64), + OccurrenceState.OPENED, + interventionId = surveyNotice.id, + triggerId = "survey-minute", + ).copy(openedAt = ResearchTime(200, 200, "boot-before-recovery")) + val posting = occurrence("d".repeat(64), OccurrenceState.POSTING) + val expired = occurrence("e".repeat(64), OccurrenceState.EXPIRED) + val metadata = StudyMetadata.initial(configuration.experimentId, configuration.configurationId).copy( + state = ExperimentState.RUNNING, + occurrences = listOf(posted, openedNotice, openedSurvey, posting, expired).associateBy { it.occurrenceId }, + ) + val fixture = fixture( + configuration, + activeEnvelope = byteArrayOf(9), + initialMetadata = metadata, + ) + + fixture.manager.initialize() + fixture.manager.rescheduleInterventions(recoverStalePosting = true) + + assertTrue(fixture.work.replacementDeliveries.isEmpty()) + assertEquals( + setOf(posted.occurrenceId, openedSurvey.occurrenceId, posting.occurrenceId), + fixture.work.replacementExpiries.mapTo(mutableSetOf()) { it.occurrenceId }, + ) + assertEquals( + setOf(openedNotice.occurrenceId, openedSurvey.occurrenceId, posting.occurrenceId, expired.occurrenceId), + fixture.work.cancelledNotificationIds, + ) + assertTrue(posted.occurrenceId !in fixture.work.cancelledNotificationIds) + + // These are the same atomic checks each restored InterventionExpiryWorker performs. + assertEquals(OccurrenceExpiryResult.Expired, fixture.manager.expireOccurrenceIfDue(posted.occurrenceId)) + assertEquals(OccurrenceExpiryResult.Terminal, fixture.manager.expireOccurrenceIfDue(openedNotice.occurrenceId)) + assertEquals(OccurrenceExpiryResult.Expired, fixture.manager.expireOccurrenceIfDue(openedSurvey.occurrenceId)) + runCurrent() + assertEquals( + OccurrenceState.EXPIRED, + fixture.manager.snapshot.value.runtime.metadata?.occurrences?.get(posted.occurrenceId)?.state, + ) + assertEquals( + OccurrenceState.OPENED, + fixture.manager.snapshot.value.runtime.metadata?.occurrences?.get(openedNotice.occurrenceId)?.state, + ) + assertEquals( + OccurrenceState.EXPIRED, + fixture.manager.snapshot.value.runtime.metadata?.occurrences?.get(openedSurvey.occurrenceId)?.state, + ) + } + + @Test + fun recoveryCancelsStaleNotificationBeforeAPlanningWriteCanFail() = runTest { + val intervention = InterventionConfiguration( + "notice-one", + NotificationAction("Study check-in", "Check in"), + listOf(InterventionTrigger("after-minute", OneTimeSchedule(1, RelativeClock.CALENDAR_TIME), 60)), + ) + val configuration = configuration(interventions = listOf(intervention)) + val stalePosting = occurrence("9".repeat(64), OccurrenceState.POSTING) + val start = ResearchTime(100, 100, "boot-before-recovery") + val metadata = StudyMetadata.initial(configuration.experimentId, configuration.configurationId).copy( + state = ExperimentState.RUNNING, + transitions = listOf( + ExperimentTransition( + ExperimentState.READY, + ExperimentState.RUNNING, + TransitionReason.PARTICIPANT_STARTED, + start, + ), + ), + occurrences = mapOf(stalePosting.occurrenceId to stalePosting), + ) + val fixture = fixture(configuration, activeEnvelope = byteArrayOf(9), initialMetadata = metadata) + fixture.manager.initialize() + fixture.store.appendFailure = IllegalStateException("storage unavailable") + + val failure = runCatching { + fixture.manager.rescheduleInterventions(recoverStalePosting = true) + }.exceptionOrNull() + + assertTrue(failure is IllegalStateException) + assertEquals(setOf(stalePosting.occurrenceId), fixture.work.cancelledNotificationIds) + assertTrue(fixture.work.replacementDeliveries.isEmpty()) + } + + @Test + fun postedCommitSurvivesSuccessorSchedulingFailureAndRetry() = runTest { + val intervention = InterventionConfiguration( + "notice-one", + NotificationAction("Study check-in", "Check in"), + listOf( + InterventionTrigger( + "interval-trigger", + IntervalSchedule(0, 10, RelativeClock.CALENDAR_TIME), + 60, + ), + ), + ) + val fixture = fixture(configuration(interventions = listOf(intervention))) + fixture.manager.initialize() + fixture.manager.importSignedConfiguration(byteArrayOf(1)) + fixture.manager.reviewStudy() + fixture.manager.acceptConsent() + fixture.access.granted += AccessKind.NOTIFICATIONS + fixture.manager.completeAccessSetup() + fixture.manager.start() + runCurrent() + val first = requireNotNull( + fixture.manager.snapshot.value.runtime.metadata?.occurrences?.values?.single(), + ) + assertTrue(fixture.manager.claimOccurrenceIfDue(first.occurrenceId) is OccurrenceClaimResult.Due) + assertTrue(fixture.manager.markNotificationPosted(first.occurrenceId)) + + fixture.work.failEnqueue = true + val failure = runCatching { fixture.manager.scheduleSuccessor(first.occurrenceId) }.exceptionOrNull() + runCurrent() + assertTrue(failure is IllegalStateException) + assertEquals( + OccurrenceState.NOTIFICATION_POSTED, + fixture.manager.snapshot.value.runtime.metadata?.occurrences?.get(first.occurrenceId)?.state, + ) + + fixture.work.failEnqueue = false + val enqueuedBeforeRetry = fixture.work.enqueuedOccurrences.size + fixture.manager.scheduleSuccessor(first.occurrenceId) + assertEquals(enqueuedBeforeRetry + 1, fixture.work.enqueuedOccurrences.size) + assertTrue(fixture.work.enqueuedOccurrences.last().occurrenceId != first.occurrenceId) + } + + @Test + fun pauseFinishAndWithdrawCancelVisibleOccurrenceNotifications() = runTest { + val intervention = InterventionConfiguration( + "notice-one", + NotificationAction("Study check-in", "Check in"), + listOf(InterventionTrigger("after-minute", OneTimeSchedule(1, RelativeClock.CALENDAR_TIME), 60)), + ) + val configuration = configuration(interventions = listOf(intervention)) + suspend fun fixtureWithPostedOccurrence(): Fixture { + val posted = occurrence("f".repeat(64), OccurrenceState.NOTIFICATION_POSTED) + return fixture( + configuration, + activeEnvelope = byteArrayOf(9), + initialMetadata = StudyMetadata.initial(configuration.experimentId, configuration.configurationId).copy( + state = ExperimentState.RUNNING, + occurrences = mapOf(posted.occurrenceId to posted), + ), + ).also { it.manager.initialize() } + } + + val paused = fixtureWithPostedOccurrence() + assertEquals(CommandResult.Success, paused.manager.pause()) + assertEquals(setOf("f".repeat(64)), paused.work.cancelledNotificationIds) + assertEquals(1, paused.work.cancelInterventionCount) + + listOf CommandResult>( + { it.finish() }, + { it.withdraw() }, + ).forEach { terminate -> + val fixture = fixtureWithPostedOccurrence() + assertEquals(CommandResult.Success, terminate(fixture.manager)) + assertEquals(setOf("f".repeat(64)), fixture.work.cancelledNotificationIds) + assertEquals(1, fixture.work.cancelCollectionCount) + } + } + @Test fun schedulingFailureCompensatesToPausedAndStopsForegroundHost() = runTest { val fixture = fixture(configuration()) @@ -185,21 +501,22 @@ class StudySessionManagerTest { fixture.collector.emit(3) runCurrent() - assertEquals(CommandResult.Success, manager.uploadPending()) + assertTrue(manager.uploadPending() is UploadAttemptResult.Confirmed) runCurrent() assertEquals(listOf(1L to 3L), uploader.ranges) + assertEquals(1, uploader.acknowledged.size) assertEquals(3L, manager.snapshot.value.upload?.uploadedThroughSequence) assertEquals(0L, manager.snapshot.value.upload?.pendingCount) // A second call with nothing new must not re-send the same events. - assertEquals(CommandResult.Success, manager.uploadPending()) + assertEquals(UploadAttemptResult.NoWork, manager.uploadPending()) assertEquals(listOf(1L to 3L), uploader.ranges) // Only the events collected since the last confirmation go out next. fixture.collector.emit(2) runCurrent() - assertEquals(CommandResult.Success, manager.uploadPending()) + assertTrue(manager.uploadPending() is UploadAttemptResult.Confirmed) runCurrent() assertEquals(listOf(1L to 3L, 4L to 5L), uploader.ranges) assertEquals(5L, manager.snapshot.value.upload?.uploadedThroughSequence) @@ -220,7 +537,7 @@ class StudySessionManagerTest { fixture.collector.emit(2) runCurrent() - assertEquals(CommandResult.Failed("UPLOAD_FAILED"), manager.uploadPending()) + assertEquals(UploadAttemptResult.Failed("UPLOAD_FAILED", retryable = false), manager.uploadPending()) runCurrent() assertEquals(0L, manager.snapshot.value.upload?.uploadedThroughSequence) @@ -252,7 +569,7 @@ class StudySessionManagerTest { assertTrue(!manager.uploadDrained()) // Delivery is still scheduled, so the backlog goes out after the study is over. - assertEquals(CommandResult.Success, manager.uploadPending()) + assertTrue(manager.uploadPending() is UploadAttemptResult.Confirmed) runCurrent() assertEquals(listOf(1L to 2L), uploader.ranges) @@ -276,7 +593,7 @@ class StudySessionManagerTest { // Comfortably inside the quota: full local retention is the norm, not an optimisation. fixture.store.usedBytes = 100 fixture.store.quotaBytes = 1_000 - assertEquals(CommandResult.Success, manager.uploadPending()) + assertTrue(manager.uploadPending() is UploadAttemptResult.Confirmed) runCurrent() assertEquals(4L, manager.snapshot.value.upload?.uploadedThroughSequence) @@ -301,7 +618,7 @@ class StudySessionManagerTest { // Past the 80% mark, so delivered events become reclaimable. fixture.store.usedBytes = 900 fixture.store.quotaBytes = 1_000 - assertEquals(CommandResult.Success, manager.uploadPending()) + assertTrue(manager.uploadPending() is UploadAttemptResult.Confirmed) runCurrent() assertEquals(1, fixture.store.evictionCount) @@ -334,7 +651,7 @@ class StudySessionManagerTest { fixture.store.usedBytes = 990 fixture.store.quotaBytes = 1_000 - assertEquals(CommandResult.Failed("UPLOAD_FAILED"), manager.uploadPending()) + assertEquals(UploadAttemptResult.Failed("UPLOAD_FAILED", retryable = false), manager.uploadPending()) runCurrent() assertEquals(0, fixture.store.evictionCount) @@ -351,7 +668,7 @@ class StudySessionManagerTest { manager.importSignedConfiguration(byteArrayOf(1)) // A periodic worker can fire before the participant has started the study. - assertEquals(CommandResult.Success, manager.uploadPending()) + assertEquals(UploadAttemptResult.NoWork, manager.uploadPending()) assertTrue(uploader.ranges.isEmpty()) assertNull(manager.snapshot.value.upload) @@ -371,7 +688,7 @@ class StudySessionManagerTest { fixture.collector.emit(2) runCurrent() - assertEquals(CommandResult.Success, manager.uploadPending()) + assertEquals(UploadAttemptResult.NoWork, manager.uploadPending()) assertTrue(uploader.ranges.isEmpty()) assertNull(manager.snapshot.value.upload) @@ -380,10 +697,11 @@ class StudySessionManagerTest { private fun TestScope.fixture( configuration: StudyConfiguration, activeEnvelope: ByteArray? = null, + activeRecord: ActiveStudyRecord? = activeEnvelope?.let(ActiveStudyRecord::Active), initialMetadata: StudyMetadata? = null, uploader: StudyUploader = FakeUploader(), ): Fixture { - val active = FakeActiveStudyStore(activeEnvelope) + val active = FakeActiveStudyStore(activeRecord) val store = FakeStudyStore(initialMetadata) val collector = FakeCollector() val registry = CollectorRegistry(listOf(FakePlugin(collector))) @@ -392,8 +710,8 @@ class StudySessionManagerTest { val work = FakeWorkScheduler() val manager = StudySessionManager( activeStudyStore = active, - verifier = StudyVerifier { VerifiedConfiguration(configuration, signerAnchored = false) }, - storeFactory = StudyStoreFactory { store }, + verifier = StudyVerifier { verified(configuration) }, + storeFactory = StudyStoreFactory { _, _ -> store }, runtimeFactory = ExperimentRuntimeFactory { verified, createdStore, availableAccess -> ExperimentRuntime( verified, @@ -413,7 +731,7 @@ class StudySessionManagerTest { accessPolicy = StudyAccessPolicy(), scope = backgroundScope, ) - return Fixture(manager, active, store, collector, host, work, uploader) + return Fixture(manager, active, store, collector, host, work, uploader, access) } private data class Fixture( @@ -424,13 +742,25 @@ class StudySessionManagerTest { val host: FakeHost, val work: FakeWorkScheduler, val uploader: StudyUploader, + val access: FakeAccessGateway, ) - private class FakeActiveStudyStore(initial: ByteArray?) : ActiveStudyStore { - var saved = initial - override suspend fun load(): ByteArray? = saved - override suspend fun save(envelopeBytes: ByteArray) { saved = envelopeBytes } - override suspend fun clear() { saved = null } + private class FakeActiveStudyStore(initial: ActiveStudyRecord?) : ActiveStudyStore { + var record: ActiveStudyRecord? = initial + var clearFailure: Exception? = null + override suspend fun load(): ActiveStudyRecord? = record + override suspend fun save(envelopeBytes: ByteArray) { + record = ActiveStudyRecord.Active(envelopeBytes) + } + override suspend fun markDeletionPending(experimentId: String, maximumLocalBytes: Long) { + val deletion = ActiveStudyRecord.DeletionPending(experimentId, maximumLocalBytes) + check(record is ActiveStudyRecord.Active || record == deletion) + record = deletion + } + override suspend fun clear() { + clearFailure?.let { throw it } + record = null + } } private class FakeStudyStore(initial: StudyMetadata?) : StudyStore { @@ -440,6 +770,9 @@ class StudySessionManagerTest { var usedBytes = 0L var quotaBytes = 16_777_216L var evictionCount = 0 + var clearAttempts = 0 + var clearFailure: Exception? = null + var appendFailure: Exception? = null override suspend fun storageUsage() = StorageUsage(usedBytes, quotaBytes) @@ -462,6 +795,7 @@ class StudySessionManagerTest { override suspend fun saveMetadata(metadata: StudyMetadata) { this.metadata = metadata } override suspend fun appendEvent(event: RecordedEvent) { events += event } override suspend fun appendEventAtomically(event: RecordedEvent, metadata: StudyMetadata) { + appendFailure?.let { throw it } events += event this.metadata = metadata } @@ -475,6 +809,8 @@ class StudySessionManagerTest { .forEach(consume) } override suspend fun clear() { + clearAttempts += 1 + clearFailure?.let { throw it } metadata = null events.clear() cleared = true @@ -485,11 +821,10 @@ class StudySessionManagerTest { private val collector: FakeCollector, ) : CollectorPlugin { override val descriptor = CollectorDescriptor( - AppLifecycleConfiguration.ID, - 1, - "Test collector", - PrivacyClass.SENSITIVE, - 1_024, + id = AppLifecycleConfiguration.ID, + displayName = "Test collector", + privacyClass = PrivacyClass.SENSITIVE, + eventContract = requireNotNull(ProtocolEventContracts[AppLifecycleConfiguration.ID]), ) override fun accessRequirements(configuration: CollectorConfiguration): Set = emptySet() @@ -523,7 +858,7 @@ class StudySessionManagerTest { payloadSchemaVersion = 1, observedTime = checkNotNull(context).clocks.now(), payloadType = "ACTIVITY_RESUMED", - fields = emptyMap(), + fields = mapOf("activity_class" to "test.Activity"), ), ) } @@ -547,38 +882,76 @@ class StudySessionManagerTest { private class FakeHost : StudyCollectionHost { var startCount = 0 var stopCount = 0 + var stopFailure: Exception? = null override fun start(studyTitle: String, usesLocation: Boolean) { startCount += 1 } - override fun stop() { stopCount += 1 } + override fun stop() { + stopCount += 1 + stopFailure?.let { throw it } + } } private class FakeWorkScheduler : StudyWorkScheduler { var scheduleCount = 0 var cancelCount = 0 var cancelCollectionCount = 0 + var cancelInterventionCount = 0 var failSchedule = false + var failEnqueue = false + var cancelFailure: Exception? = null + var replacementDeliveries = emptyList() + var replacementExpiries = emptyList() + val cancelledNotificationIds = mutableSetOf() + val enqueuedOccurrences = mutableListOf() override fun schedule(configuration: StudyConfiguration) { scheduleCount += 1 if (failSchedule) error("Scheduling failed") } override fun replaceInterventionWork( configuration: StudyConfiguration, - occurrences: List, - ) = Unit + deliveries: List, + expiries: List, + ) { + replacementDeliveries = deliveries + replacementExpiries = expiries + } override fun enqueueOccurrence( configuration: StudyConfiguration, occurrence: InterventionOccurrence, - ) = Unit - override fun cancelCollectionWork(experimentId: String) { cancelCollectionCount += 1 } - override fun cancel(experimentId: String) { cancelCount += 1 } + ) { + if (failEnqueue) error("Enqueue failed") + enqueuedOccurrences += occurrence + } + override fun cancelInterventionWork(experimentId: String, occurrenceIds: Set) { + cancelInterventionCount += 1 + cancelledNotificationIds += occurrenceIds + } + override fun cancelInterventionNotifications(occurrenceIds: Set) { + cancelledNotificationIds += occurrenceIds + } + override fun cancelCollectionWork(experimentId: String, occurrenceIds: Set) { + cancelCollectionCount += 1 + cancelledNotificationIds += occurrenceIds + } + override fun cancel(experimentId: String) { + cancelCount += 1 + cancelFailure?.let { throw it } + } } private class FakeUploader( - private val failure: Throwable? = null, + var failure: Throwable? = null, ) : StudyUploader { val ranges = mutableListOf>() + val acknowledged = mutableListOf() + var cleared = false + var deletionPrepared = false + var clearAttempts = 0 + var clearFailure: Exception? = null + + override suspend fun reconcile(configuration: VerifiedConfiguration, metadata: StudyMetadata) = Unit override suspend fun upload( - configuration: StudyConfiguration, + configuration: VerifiedConfiguration, metadata: StudyMetadata, events: StudyStore, fromSequence: Long, @@ -587,25 +960,42 @@ class StudySessionManagerTest { ranges += fromSequence to toSequence failure?.let { throw it } return ExportReceipt( - configuration.export.researcherKeyId, - fromSequence, - toSequence, - toSequence - fromSequence + 1, - "hash", - 1, + bundleId = UUID.fromString("00000000-0000-4000-8000-000000000001"), + configurationSha256 = configuration.configurationSha256, + firstSequence = fromSequence, + lastSequence = toSequence, + eventCount = toSequence - fromSequence + 1, + sha256 = "1".repeat(64), + byteCount = 1, ) } + + override suspend fun acknowledge(bundleId: UUID) { acknowledged += bundleId } + override suspend fun prepareDeletion() { deletionPrepared = true } + override suspend fun clear() { + clearAttempts += 1 + clearFailure?.let { throw it } + cleared = true + } } private class FakeExporter : StudyExporter { override suspend fun export( - configuration: StudyConfiguration, + configuration: VerifiedConfiguration, metadata: StudyMetadata, events: StudyStore, destination: OutputStream, ): ExportReceipt { destination.write(1) - return ExportReceipt(configuration.export.researcherKeyId, 1, metadata.eventCount, metadata.eventCount, "hash", 1) + return ExportReceipt( + bundleId = UUID.fromString("00000000-0000-4000-8000-000000000002"), + configurationSha256 = configuration.configurationSha256, + firstSequence = 1, + lastSequence = metadata.eventCount, + eventCount = metadata.eventCount, + sha256 = "1".repeat(64), + byteCount = 1, + ) } } @@ -617,6 +1007,7 @@ class StudySessionManagerTest { private fun configuration( interventions: List = emptyList(), + surveys: List = emptyList(), upload: UploadConfiguration? = null, ) = StudyConfiguration( schemaVersion = StudyConfiguration.CURRENT_SCHEMA_VERSION, @@ -625,7 +1016,8 @@ class StudySessionManagerTest { assignedParticipantId = null, issuedAt = Instant.parse("2026-01-01T00:00:00Z"), expiresAt = Instant.parse("2030-01-01T00:00:00Z"), - minimumAppVersion = 1, + platform = StudyConfiguration.ANDROID_PLATFORM, + minimumClientVersion = 1, title = "Session test", researcherName = "Test researcher", researcherContact = "test@example.invalid", @@ -634,14 +1026,37 @@ class StudySessionManagerTest { consentDocumentVersion = "v1", consentSummary = "Test consent", collectors = listOf(AppLifecycleConfiguration(required = true)), - surveys = emptyList(), + surveys = surveys, interventions = interventions, maximumLocalBytes = 16_777_216, - signer = SignerIdentity("test-signer", TEST_SIGNER_PUBLIC_KEY), - export = ExportConfiguration("export-key", "x".repeat(32)), + signer = SignerIdentity("test-signer", RAW_PUBLIC_KEY), + export = ExportConfiguration("export-key", RAW_PUBLIC_KEY), upload = upload, ) + + private fun verified(configuration: StudyConfiguration) = VerifiedConfiguration( + configuration = configuration, + canonicalConfigurationBytes = byteArrayOf(1), + signerKeyId = configuration.signer.keyId, + signature = ByteArray(64), + configurationSha256 = "0".repeat(64), + signerAnchored = false, + ) + + private fun occurrence( + id: String, + state: OccurrenceState, + interventionId: String = "notice-one", + triggerId: String = "after-minute", + ) = InterventionOccurrence( + occurrenceId = id, + interventionId = interventionId, + triggerId = triggerId, + scheduleKey = "relative:1", + scheduledFor = ResearchTime(100, 100, "boot-before-recovery"), + expiresAtUtcMillis = 500, + state = state, + ) } -private const val TEST_SIGNER_PUBLIC_KEY = - "MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0=" +private const val RAW_PUBLIC_KEY = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" diff --git a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolBase64Url.kt b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolBase64Url.kt new file mode 100644 index 0000000..521077d --- /dev/null +++ b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolBase64Url.kt @@ -0,0 +1,23 @@ +package cool.linc.androiddatacollector.core.definition + +import java.util.Base64 + +/** Strict unpadded base64url used for all Protocol v1 raw key and signature bytes. */ +object ProtocolBase64Url { + private val encoder = Base64.getUrlEncoder().withoutPadding() + private val decoder = Base64.getUrlDecoder() + private val textPattern = Regex("[A-Za-z0-9_-]+") + + fun encode(bytes: ByteArray): String = encoder.encodeToString(bytes) + + fun decode(text: String, label: String): ByteArray { + require(textPattern.matches(text) && '=' !in text) { "Invalid $label encoding" } + val decoded = runCatching { decoder.decode(text) } + .getOrElse { throw IllegalArgumentException("Invalid $label encoding", it) } + require(encode(decoded) == text) { "Invalid $label encoding" } + return decoded + } + + fun decodeExact(text: String, size: Int, label: String): ByteArray = + decode(text, label).also { require(it.size == size) { "Invalid $label length" } } +} diff --git a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolCanonicalJson.kt b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolCanonicalJson.kt new file mode 100644 index 0000000..597264a --- /dev/null +++ b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/ProtocolCanonicalJson.kt @@ -0,0 +1,131 @@ +package cool.linc.androiddatacollector.core.definition + +import com.google.gson.JsonArray +import com.google.gson.JsonElement +import com.google.gson.JsonNull +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import com.google.gson.JsonPrimitive +import com.google.gson.Strictness +import com.google.gson.stream.JsonReader +import com.google.gson.stream.JsonToken +import java.io.StringReader +import java.math.BigDecimal +import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction + +/** The one strict, integral-only RFC 8785 implementation shared by Protocol v1 readers. */ +object ProtocolCanonicalJson { + fun requireCanonical(bytes: ByteArray, maximumBytes: Int): JsonElement { + val decoded = parse(bytes, maximumBytes) + require(encode(decoded).contentEquals(bytes)) { "JSON is not canonical" } + return decoded + } + + fun canonicalize(bytes: ByteArray, maximumBytes: Int): ByteArray = + encode(parse(bytes, maximumBytes)) + + fun parse(bytes: ByteArray, maximumBytes: Int): JsonElement { + require(maximumBytes > 0 && bytes.size in 1..maximumBytes) { "Invalid JSON size" } + val text = runCatching { + Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes)) + .toString() + }.getOrElse { throw IllegalArgumentException("JSON is not valid UTF-8", it) } + val reader = JsonReader(StringReader(text)).apply { strictness = Strictness.STRICT } + return readStrict(reader).also { + require(reader.peek() == JsonToken.END_DOCUMENT) { "Trailing JSON content" } + } + } + + fun encode(element: JsonElement): ByteArray = buildString { + appendCanonical(element) + }.toByteArray(Charsets.UTF_8) + + private fun readStrict(reader: JsonReader): JsonElement = when (reader.peek()) { + JsonToken.BEGIN_OBJECT -> JsonObject().also { result -> + reader.beginObject() + val names = mutableSetOf() + while (reader.hasNext()) { + val name = reader.nextName() + require(names.add(name)) { "Duplicate JSON member: $name" } + result.add(name, readStrict(reader)) + } + reader.endObject() + } + JsonToken.BEGIN_ARRAY -> JsonArray().also { result -> + reader.beginArray() + while (reader.hasNext()) result.add(readStrict(reader)) + reader.endArray() + } + JsonToken.STRING -> JsonPrimitive(reader.nextString()) + JsonToken.NUMBER -> JsonParser.parseString(reader.nextString()) + JsonToken.BOOLEAN -> JsonPrimitive(reader.nextBoolean()) + JsonToken.NULL -> JsonNull.INSTANCE.also { reader.nextNull() } + else -> throw IllegalArgumentException("Invalid JSON value") + } + + /** Protocol v1 permits only integral JSON numbers, so no floating-point JCS branch exists. */ + private fun StringBuilder.appendCanonical(element: JsonElement) { + when { + element.isJsonNull -> append("null") + element.isJsonArray -> { + append('[') + element.asJsonArray.forEachIndexed { index, item -> + if (index > 0) append(',') + appendCanonical(item) + } + append(']') + } + element.isJsonObject -> { + append('{') + element.asJsonObject.keySet().sorted().forEachIndexed { index, name -> + if (index > 0) append(',') + appendJsonString(name) + append(':') + appendCanonical(element.asJsonObject.get(name)) + } + append('}') + } + element.asJsonPrimitive.isString -> appendJsonString(element.asString) + element.asJsonPrimitive.isBoolean -> append(element.asBoolean) + else -> { + val integer = runCatching { BigDecimal(element.asString).toBigIntegerExact() }.getOrNull() + require(integer != null) { "Protocol JSON numbers must be integral" } + append(integer) + } + } + } + + private fun StringBuilder.appendJsonString(value: String) { + append('"') + var index = 0 + while (index < value.length) { + val character = value[index] + when (character) { + '"' -> append("\\\"") + '\\' -> append("\\\\") + '\b' -> append("\\b") + '\t' -> append("\\t") + '\n' -> append("\\n") + '\u000c' -> append("\\f") + '\r' -> append("\\r") + else -> when { + character < ' ' -> append("\\u%04x".format(character.code)) + character.isHighSurrogate() -> { + require(index + 1 < value.length && value[index + 1].isLowSurrogate()) { + "Invalid Unicode surrogate" + } + append(character).append(value[++index]) + } + character.isLowSurrogate() -> throw IllegalArgumentException("Invalid Unicode surrogate") + else -> append(character) + } + } + index++ + } + append('"') + } +} diff --git a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt index be99bf7..8d9695b 100644 --- a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt +++ b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt @@ -8,7 +8,8 @@ data class StudyConfiguration( val configurationId: String, val issuedAt: Instant, val expiresAt: Instant, - val minimumAppVersion: Int, + val platform: String, + val minimumClientVersion: Long, val title: String, val researcherName: String, val researcherContact: String, @@ -31,7 +32,8 @@ data class StudyConfiguration( require(ID.matches(experimentId)) { "Invalid experiment ID" } require(ID.matches(configurationId)) { "Invalid configuration ID" } require(issuedAt < expiresAt) { "Configuration expiry must follow issue time" } - require(minimumAppVersion > 0) { "Minimum app version must be positive" } + require(platform == ANDROID_PLATFORM) { "Unsupported target platform" } + require(minimumClientVersion > 0) { "Minimum client version must be positive" } require(title.length in 1..120) { "Invalid study title" } require(researcherName.length in 1..120) { "Invalid researcher name" } require(researcherContact.length in 3..240) { "Invalid researcher contact" } @@ -67,12 +69,12 @@ data class StudyConfiguration( companion object { /** - * The only accepted schema. There is no fallback reader and no migration: a configuration - * either matches this exactly or is refused, which is what keeps the closed-world key - * checks meaningful. Adding a root key is a version bump that invalidates every existing - * file, so the number moves only when the format really changes in the field. + * The only accepted pre-1.0 Protocol v1 schema. Protocol v1 is replaced in place while it + * is pre-release: there is no legacy reader, fallback, or migration. An artifact either + * matches the current closed-world contract exactly or is refused. */ const val CURRENT_SCHEMA_VERSION = 1 + const val ANDROID_PLATFORM = "android" /** * Local budget a study may claim, 8 MiB to 8 GiB. The floor leaves room for the metadata * reserve; the ceiling is generous because high-rate collectors fill space quickly — an @@ -115,6 +117,67 @@ data class AccelerometerConfiguration( companion object { const val ID = "accelerometer.v1" } } +data class BatteryStateConfiguration( + override val required: Boolean, +) : CollectorConfiguration { + override val id: String = ID + + companion object { const val ID = "battery_state.v1" } +} + +data class TemporalContextConfiguration( + override val required: Boolean, +) : CollectorConfiguration { + override val id: String = ID + + companion object { const val ID = "temporal_context.v1" } +} + +data class GyroscopeConfiguration( + override val required: Boolean, + val samplingPeriodUs: Int, + val maximumReportLatencyUs: Int, +) : CollectorConfiguration { + override val id: String = ID + + init { + require(samplingPeriodUs in 5_000..1_000_000) { "Invalid gyroscope sampling period" } + require(maximumReportLatencyUs in 0..60_000_000) { "Invalid gyroscope report latency" } + } + + companion object { const val ID = "gyroscope.v1" } +} + +data class AmbientLightConfiguration( + override val required: Boolean, + val samplingPeriodUs: Int, + val changeThresholdMillilux: Int, +) : CollectorConfiguration { + override val id: String = ID + + init { + require(samplingPeriodUs in 200_000..10_000_000) { "Invalid ambient-light sampling period" } + require(changeThresholdMillilux in 0..100_000_000) { "Invalid ambient-light change threshold" } + } + + companion object { const val ID = "ambient_light.v1" } +} + +data class ProximityConfiguration( + override val required: Boolean, + val minimumEventIntervalMs: Int, + val changeThresholdMillimeters: Int, +) : CollectorConfiguration { + override val id: String = ID + + init { + require(minimumEventIntervalMs in 100..60_000) { "Invalid proximity event interval" } + require(changeThresholdMillimeters in 0..10_000) { "Invalid proximity change threshold" } + } + + companion object { const val ID = "proximity.v1" } +} + data class NetworkStateConfiguration( override val required: Boolean, val includeBandwidthEstimates: Boolean, @@ -161,7 +224,7 @@ data class LocationConfiguration( val intervalMillis: Long, val minimumIntervalMillis: Long, val maximumBatchDelayMillis: Long, - val minimumDisplacementMeters: Float, + val minimumDisplacementMillimeters: Int, val priority: LocationPriority, ) : CollectorConfiguration { override val id: String = ID @@ -170,7 +233,7 @@ data class LocationConfiguration( require(intervalMillis in 1_000..3_600_000) { "Invalid location interval" } require(minimumIntervalMillis in 500..intervalMillis) { "Invalid location minimum interval" } require(maximumBatchDelayMillis in 0..86_400_000) { "Invalid location batch delay" } - require(minimumDisplacementMeters in 0f..10_000f) { "Invalid location displacement" } + require(minimumDisplacementMillimeters in 0..10_000_000) { "Invalid location displacement" } } companion object { const val ID = "location.v1" } @@ -245,6 +308,16 @@ sealed interface InterventionSchedule { fun maximumOccurrences(studyMinutes: Int): Long } +/** + * Conservative count of local dates reachable while Android's zone can move between its legal + * fixed-offset extremes (UTC-18 through UTC+18). The extra partial dates matter to the global + * durable-occurrence bound even for a study shorter than one day. + */ +internal fun maximumReachableLocalDates(studyMinutes: Int): Long = + (studyMinutes + MAXIMUM_ZONE_OFFSET_SPAN_MINUTES + 1_439L) / 1_440L + 1 + +private const val MAXIMUM_ZONE_OFFSET_SPAN_MINUTES = 36 * 60 + enum class RelativeClock { CALENDAR_TIME, ACTIVE_RUNNING_TIME } data class OneTimeSchedule( @@ -280,11 +353,68 @@ data class DailyLocalSchedule( ) : InterventionSchedule { init { require(LOCAL_TIME.matches(localTime)) { "Invalid daily local time" } } override fun requireWithin(studyMinutes: Int) = Unit - override fun maximumOccurrences(studyMinutes: Int): Long = (studyMinutes + 1_439L) / 1_440L + 1 + override fun maximumOccurrences(studyMinutes: Int): Long = maximumReachableLocalDates(studyMinutes) companion object { private val LOCAL_TIME = Regex("(?:[01][0-9]|2[0-3]):[0-5][0-9]") } } +data class RandomLocalWindow( + /** Inclusive local wall-clock minute, `HH:mm`. */ + val startLocalTime: String, + /** Exclusive local wall-clock minute on the same local date, `HH:mm`. */ + val endLocalTime: String, +) { + val startMinute: Int = parseLocalMinute(startLocalTime) + val endMinute: Int = parseLocalMinute(endLocalTime) + + init { require(startMinute < endMinute) { "Random window must end after it starts" } } +} + +data class RandomWindowSchedule( + val localWindows: List, + val occurrencesPerWindow: Int, + val maximumOccurrencesPerDay: Int, + val maximumOccurrencesTotal: Int, + val minimumSeparationMinutes: Int, +) : InterventionSchedule { + init { + require(localWindows.size in 1..8) { "Invalid random-window count" } + require(localWindows.zipWithNext().all { (first, second) -> first.endMinute <= second.startMinute }) { + "Random windows must be sorted and non-overlapping" + } + require(occurrencesPerWindow in 1..8) { "Invalid occurrences per random window" } + require(maximumOccurrencesPerDay in 1..64) { "Invalid daily random occurrence limit" } + require(maximumOccurrencesPerDay <= localWindows.size * occurrencesPerWindow) { + "Daily random occurrence limit exceeds window capacity" + } + require(maximumOccurrencesTotal in 1..512) { "Invalid total random occurrence limit" } + require(minimumSeparationMinutes in 1..1_440) { "Invalid random occurrence separation" } + require(localWindows.all { window -> + window.endMinute - window.startMinute >= + 1 + (occurrencesPerWindow - 1) * minimumSeparationMinutes + }) { "A random window cannot fit its configured occurrences" } + require(localWindows.indices.all { index -> + val current = localWindows[index] + val next = localWindows[(index + 1) % localWindows.size] + val nextStart = next.startMinute + if (index == localWindows.lastIndex) 1_440 else 0 + nextStart - (current.endMinute - 1) >= minimumSeparationMinutes + }) { "Random windows are too close for the configured separation" } + } + + override fun requireWithin(studyMinutes: Int) = Unit + + // Wall-clock edits can expose arbitrarily many local dates inside a short monotonic study. + // The signed lifetime cap is therefore the only safe contribution to the global 512 bound. + override fun maximumOccurrences(studyMinutes: Int): Long = maximumOccurrencesTotal.toLong() +} + +private fun parseLocalMinute(value: String): Int { + require(LOCAL_MINUTE.matches(value)) { "Invalid local time" } + return value.substring(0, 2).toInt() * 60 + value.substring(3).toInt() +} + +private val LOCAL_MINUTE = Regex("(?:[01][0-9]|2[0-3]):[0-5][0-9]") + data class SurveyDefinition( val id: String, val title: LocalizedText, @@ -413,12 +543,12 @@ private fun validateOptions(options: List) { */ data class SignerIdentity( val keyId: String, - /** Base64 X.509 SubjectPublicKeyInfo for an Ed25519 key. */ + /** Unpadded base64url raw 32-byte Ed25519 public key. */ val publicKey: String, ) { init { require(StudyConfiguration.ID.matches(keyId)) { "Invalid signer key ID" } - require(publicKey.length in 32..1_024) { "Invalid signer public key" } + ProtocolBase64Url.decodeExact(publicKey, RAW_PUBLIC_KEY_BYTES, "signer public key") } /** @@ -427,22 +557,28 @@ data class SignerIdentity( */ val fingerprint: String by lazy { java.security.MessageDigest.getInstance("SHA-256") - .digest(java.util.Base64.getDecoder().decode(publicKey)) + .digest(ProtocolBase64Url.decodeExact(publicKey, RAW_PUBLIC_KEY_BYTES, "signer public key")) .take(16) .joinToString("") { "%02X".format(it) } .chunked(4) .joinToString(" ") } + + companion object { const val RAW_PUBLIC_KEY_BYTES = 32 } } data class ExportConfiguration( val researcherKeyId: String, - val tinkHpkePublicKeysetJson: String, + /** Unpadded base64url raw 32-byte X25519 public key. */ + val hpkePublicKey: String, ) { init { require(StudyConfiguration.ID.matches(researcherKeyId)) { "Invalid researcher key ID" } - require(tinkHpkePublicKeysetJson.length in 32..16_384) { "Invalid researcher public keyset" } + ProtocolBase64Url.decodeExact(hpkePublicKey, RAW_PUBLIC_KEY_BYTES, "researcher public key") } + + + companion object { const val RAW_PUBLIC_KEY_BYTES = 32 } } /** diff --git a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt index e6b9652..f776f71 100644 --- a/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt +++ b/core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt @@ -6,6 +6,7 @@ import com.google.gson.JsonObject import com.google.gson.JsonParser import com.google.gson.stream.JsonWriter import java.io.StringWriter +import java.math.BigDecimal import java.time.Instant object StudyConfigurationCodec { @@ -16,7 +17,8 @@ object StudyConfigurationCodec { "assigned_participant_id", "issued_at", "expires_at", - "minimum_app_version", + "platform", + "minimum_client_version", "title", "researcher", "purpose", @@ -44,7 +46,7 @@ object StudyConfigurationCodec { private fun decodeStructure(bytes: ByteArray): StudyConfiguration { require(bytes.size in 2..MAX_CONFIGURATION_BYTES) { "Invalid configuration size" } - val root = JsonParser.parseString(bytes.toString(Charsets.UTF_8)).requireObject("root") + val root = ProtocolCanonicalJson.parse(bytes, MAX_CONFIGURATION_BYTES).requireObject("root") root.requireExactKeys(ROOT_KEYS) return StudyConfiguration( schemaVersion = root.requireInt("schema_version"), @@ -53,7 +55,8 @@ object StudyConfigurationCodec { assignedParticipantId = root.requireNullableString("assigned_participant_id"), issuedAt = Instant.parse(root.requireString("issued_at")), expiresAt = Instant.parse(root.requireString("expires_at")), - minimumAppVersion = root.requireInt("minimum_app_version"), + platform = root.requireString("platform"), + minimumClientVersion = root.requireDecimalLongString("minimum_client_version"), title = root.requireString("title"), researcherName = root.requireObject("researcher").also { it.requireExactKeys(setOf("name", "contact")) @@ -90,7 +93,8 @@ object StudyConfigurationCodec { writer.name("assigned_participant_id").value(configuration.assignedParticipantId) writer.name("issued_at").value(configuration.issuedAt.toString()) writer.name("expires_at").value(configuration.expiresAt.toString()) - writer.name("minimum_app_version").value(configuration.minimumAppVersion) + writer.name("platform").value(configuration.platform) + writer.name("minimum_client_version").value(configuration.minimumClientVersion.toString()) writer.name("title").value(configuration.title) writer.name("researcher").beginObject() writer.name("name").value(configuration.researcherName) @@ -120,7 +124,7 @@ object StudyConfigurationCodec { writer.endObject() writer.name("export").beginObject() writer.name("researcher_key_id").value(configuration.export.researcherKeyId) - writer.name("tink_hpke_public_keyset").jsonValue(configuration.export.tinkHpkePublicKeysetJson) + writer.name("hpke_public_key").value(configuration.export.hpkePublicKey) writer.endObject() writer.name("upload").beginObject() configuration.upload?.let { upload -> @@ -131,7 +135,7 @@ object StudyConfigurationCodec { writer.endObject() writer.endObject() } - return output.toString().toByteArray(Charsets.UTF_8) + return ProtocolCanonicalJson.encode(JsonParser.parseString(output.toString())) } private fun decodeCollector(element: JsonElement): CollectorConfiguration { @@ -152,6 +156,38 @@ object StudyConfigurationCodec { config.requireInt("maximum_report_latency_us"), ) } + BatteryStateConfiguration.ID -> { + config.requireExactKeys(emptySet()) + BatteryStateConfiguration(required) + } + TemporalContextConfiguration.ID -> { + config.requireExactKeys(emptySet()) + TemporalContextConfiguration(required) + } + GyroscopeConfiguration.ID -> { + config.requireExactKeys(setOf("sampling_period_us", "maximum_report_latency_us")) + GyroscopeConfiguration( + required, + config.requireInt("sampling_period_us"), + config.requireInt("maximum_report_latency_us"), + ) + } + AmbientLightConfiguration.ID -> { + config.requireExactKeys(setOf("sampling_period_us", "change_threshold_millilux")) + AmbientLightConfiguration( + required, + config.requireInt("sampling_period_us"), + config.requireInt("change_threshold_millilux"), + ) + } + ProximityConfiguration.ID -> { + config.requireExactKeys(setOf("minimum_event_interval_ms", "change_threshold_millimeters")) + ProximityConfiguration( + required, + config.requireInt("minimum_event_interval_ms"), + config.requireInt("change_threshold_millimeters"), + ) + } NetworkStateConfiguration.ID -> { config.requireExactKeys(setOf("include_bandwidth_estimates")) NetworkStateConfiguration(required, config.requireBoolean("include_bandwidth_estimates")) @@ -173,7 +209,7 @@ object StudyConfigurationCodec { "interval_millis", "minimum_interval_millis", "maximum_batch_delay_millis", - "minimum_displacement_meters", + "minimum_displacement_millimeters", "priority", ), ) @@ -182,7 +218,7 @@ object StudyConfigurationCodec { config.requireLong("interval_millis"), config.requireLong("minimum_interval_millis"), config.requireLong("maximum_batch_delay_millis"), - config.requireFloat("minimum_displacement_meters"), + config.requireInt("minimum_displacement_millimeters"), LocationPriority.valueOf(config.requireString("priority")), ) } @@ -205,6 +241,19 @@ object StudyConfigurationCodec { writer.name("sampling_period_us").value(collector.samplingPeriodUs) writer.name("maximum_report_latency_us").value(collector.maximumReportLatencyUs) } + is BatteryStateConfiguration, is TemporalContextConfiguration -> Unit + is GyroscopeConfiguration -> { + writer.name("sampling_period_us").value(collector.samplingPeriodUs) + writer.name("maximum_report_latency_us").value(collector.maximumReportLatencyUs) + } + is AmbientLightConfiguration -> { + writer.name("sampling_period_us").value(collector.samplingPeriodUs) + writer.name("change_threshold_millilux").value(collector.changeThresholdMillilux) + } + is ProximityConfiguration -> { + writer.name("minimum_event_interval_ms").value(collector.minimumEventIntervalMs) + writer.name("change_threshold_millimeters").value(collector.changeThresholdMillimeters) + } is NetworkStateConfiguration -> writer.name("include_bandwidth_estimates").value(collector.includeBandwidthEstimates) is NetworkUsageConfiguration -> { @@ -218,7 +267,7 @@ object StudyConfigurationCodec { writer.name("interval_millis").value(collector.intervalMillis) writer.name("minimum_interval_millis").value(collector.minimumIntervalMillis) writer.name("maximum_batch_delay_millis").value(collector.maximumBatchDelayMillis) - writer.name("minimum_displacement_meters").value(collector.minimumDisplacementMeters) + writer.name("minimum_displacement_millimeters").value(collector.minimumDisplacementMillimeters) writer.name("priority").value(collector.priority.name) } is KeyboardTouchConfiguration -> @@ -277,6 +326,32 @@ object StudyConfigurationCodec { schedule.requireExactKeys(setOf("type", "local_time")) DailyLocalSchedule(schedule.requireString("local_time")) } + "random_window" -> { + schedule.requireExactKeys( + setOf( + "type", + "local_windows", + "occurrences_per_window", + "maximum_occurrences_per_day", + "maximum_occurrences_total", + "minimum_separation_minutes", + ), + ) + RandomWindowSchedule( + localWindows = schedule.requireArray("local_windows").mapElements { item -> + val window = item.requireObject("random local window") + window.requireExactKeys(setOf("start_local_time", "end_local_time")) + RandomLocalWindow( + window.requireString("start_local_time"), + window.requireString("end_local_time"), + ) + }, + occurrencesPerWindow = schedule.requireInt("occurrences_per_window"), + maximumOccurrencesPerDay = schedule.requireInt("maximum_occurrences_per_day"), + maximumOccurrencesTotal = schedule.requireInt("maximum_occurrences_total"), + minimumSeparationMinutes = schedule.requireInt("minimum_separation_minutes"), + ) + } else -> throw IllegalArgumentException("Unknown intervention schedule") }, root.requireInt("availability_minutes"), @@ -387,6 +462,21 @@ object StudyConfigurationCodec { writer.name("type").value("daily_local") writer.name("local_time").value(schedule.localTime) } + is RandomWindowSchedule -> { + writer.name("type").value("random_window") + writer.name("local_windows").beginArray() + schedule.localWindows.forEach { window -> + writer.beginObject() + writer.name("start_local_time").value(window.startLocalTime) + writer.name("end_local_time").value(window.endLocalTime) + writer.endObject() + } + writer.endArray() + writer.name("occurrences_per_window").value(schedule.occurrencesPerWindow) + writer.name("maximum_occurrences_per_day").value(schedule.maximumOccurrencesPerDay) + writer.name("maximum_occurrences_total").value(schedule.maximumOccurrencesTotal) + writer.name("minimum_separation_minutes").value(schedule.minimumSeparationMinutes) + } } writer.endObject() writer.name("availability_minutes").value(trigger.availabilityMinutes) @@ -459,10 +549,8 @@ object StudyConfigurationCodec { } private fun decodeExport(root: JsonObject): ExportConfiguration { - root.requireExactKeys(setOf("researcher_key_id", "tink_hpke_public_keyset")) - val keyset = root.get("tink_hpke_public_keyset") - require(keyset != null && keyset.isJsonObject) { "Public keyset must be an object" } - return ExportConfiguration(root.requireString("researcher_key_id"), keyset.toString()) + root.requireExactKeys(setOf("researcher_key_id", "hpke_public_key")) + return ExportConfiguration(root.requireString("researcher_key_id"), root.requireString("hpke_public_key")) } /** @@ -511,20 +599,25 @@ object StudyConfigurationCodec { return raw.toLongOrNull() ?: throw IllegalArgumentException("$name is outside Long range") } - private fun JsonObject.requireFloat(name: String): Float { - val value = requireNotNull(get(name)) - require(value.isJsonPrimitive && value.asJsonPrimitive.isNumber) { "$name must be numeric" } - val parsed = value.asString.toFloatOrNull() - require(parsed != null && parsed.isFinite()) { "$name must be finite" } - return parsed - } - private fun JsonObject.requireIntegerLiteral(name: String): String { val value = requireNotNull(get(name)) require(value.isJsonPrimitive && value.asJsonPrimitive.isNumber) { "$name must be an integer" } val raw = value.asString - require(INTEGER.matches(raw)) { "$name must be an integer literal" } - return raw + require(raw.length <= MAXIMUM_NUMBER_CHARACTERS) { "$name is outside the supported numeric range" } + raw.substringAfterAny('e', 'E')?.let { exponent -> + require(exponent.toLongOrNull()?.let { kotlin.math.abs(it) <= MAXIMUM_DECIMAL_EXPONENT } == true) { + "$name is outside the supported numeric range" + } + } + val integer = runCatching { BigDecimal(raw).toBigIntegerExact() }.getOrNull() + require(integer != null) { "$name must be an integer" } + return integer.toString() + } + + private fun JsonObject.requireDecimalLongString(name: String): Long { + val raw = requireString(name) + require(UNSIGNED_DECIMAL.matches(raw)) { "$name must be a canonical unsigned decimal string" } + return raw.toLongOrNull() ?: throw IllegalArgumentException("$name is outside Long range") } private fun JsonObject.requireObject(name: String): JsonObject = @@ -544,6 +637,13 @@ object StudyConfigurationCodec { private fun JsonArray.mapElements(transform: (JsonElement) -> T): List = map(transform) + private fun String.substringAfterAny(first: Char, second: Char): String? { + val index = indexOfAny(charArrayOf(first, second)) + return if (index < 0) null else substring(index + 1) + } + private const val MAX_CONFIGURATION_BYTES = 1_048_576 - private val INTEGER = Regex("-?(0|[1-9][0-9]*)") + private const val MAXIMUM_NUMBER_CHARACTERS = 64 + private const val MAXIMUM_DECIMAL_EXPONENT = 64L + private val UNSIGNED_DECIMAL = Regex("0|[1-9][0-9]*") } diff --git a/core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/P2ConfigurationTest.kt b/core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/P2ConfigurationTest.kt new file mode 100644 index 0000000..90e2f6f --- /dev/null +++ b/core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/P2ConfigurationTest.kt @@ -0,0 +1,193 @@ +package cool.linc.androiddatacollector.core.definition + +import java.time.Instant +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Test + +class P2ConfigurationTest { + @Test + fun newCollectorsAndRandomWindowsRoundTripThroughTheClosedWorldCodec() { + val configuration = baseConfiguration( + collectors = listOf( + BatteryStateConfiguration(required = false), + TemporalContextConfiguration(required = false), + GyroscopeConfiguration(false, 20_000, 1_000_000), + AmbientLightConfiguration(false, 500_000, 2_000), + ProximityConfiguration(false, 250, 10), + ), + interventions = listOf( + InterventionConfiguration( + id = "random-ema", + action = NotificationAction("Check in", "Please answer now."), + triggers = listOf( + InterventionTrigger( + id = "random-window", + schedule = RandomWindowSchedule( + localWindows = listOf( + RandomLocalWindow("09:00", "12:00"), + RandomLocalWindow("14:00", "18:00"), + ), + occurrencesPerWindow = 2, + maximumOccurrencesPerDay = 3, + maximumOccurrencesTotal = 20, + minimumSeparationMinutes = 30, + ), + availabilityMinutes = 20, + ), + ), + ), + ), + ) + + val encoded = StudyConfigurationCodec.encode(configuration) + + assertEquals(configuration, StudyConfigurationCodec.decode(encoded)) + } + + @Test + fun p2IntegerFieldsEnforceBothBoundsAndStrictCodecShapes() { + data class IntegerFieldCase( + val field: String, + val minimum: Int, + val maximum: Int, + val collector: (Int) -> CollectorConfiguration, + ) + val fields = listOf( + IntegerFieldCase("sampling_period_us", 5_000, 1_000_000) { + GyroscopeConfiguration(false, it, 1_000_000) + }, + IntegerFieldCase("maximum_report_latency_us", 0, 60_000_000) { + GyroscopeConfiguration(false, 20_000, it) + }, + IntegerFieldCase("sampling_period_us", 200_000, 10_000_000) { + AmbientLightConfiguration(false, it, 2_000) + }, + IntegerFieldCase("change_threshold_millilux", 0, 100_000_000) { + AmbientLightConfiguration(false, 500_000, it) + }, + IntegerFieldCase("minimum_event_interval_ms", 100, 60_000) { + ProximityConfiguration(false, it, 10) + }, + IntegerFieldCase("change_threshold_millimeters", 0, 10_000) { + ProximityConfiguration(false, 250, it) + }, + ) + + fields.forEach { case -> + listOf(case.minimum, case.maximum).forEach { boundary -> + val configuration = baseConfiguration(listOf(case.collector(boundary)), emptyList()) + assertEquals(configuration, StudyConfigurationCodec.decode(StudyConfigurationCodec.encode(configuration))) + } + assertThrows(IllegalArgumentException::class.java) { case.collector(case.minimum - 1) } + assertThrows(IllegalArgumentException::class.java) { case.collector(case.maximum + 1) } + + val canonical = StudyConfigurationCodec.encode( + baseConfiguration(listOf(case.collector(case.minimum)), emptyList()), + ).toString(Charsets.UTF_8) + val member = "\"${case.field}\":${case.minimum}" + val withoutMember = canonical + .replace("$member,", "") + .replace(",$member", "") + val withUnknownMember = canonical.replace("\"config\":{", "\"config\":{\"unknown\":0,") + val withWrongType = canonical.replace(member, "\"${case.field}\":\"${case.minimum}\"") + listOf(withoutMember, withUnknownMember, withWrongType).forEach { hostile -> + assertThrows(IllegalArgumentException::class.java) { + StudyConfigurationCodec.decode(hostile.toByteArray(Charsets.UTF_8)) + } + } + } + } + + @Test + fun randomWindowsRejectAmbiguousOrImpossibleBounds() { + assertThrows(IllegalArgumentException::class.java) { + RandomWindowSchedule( + listOf(RandomLocalWindow("12:00", "13:00"), RandomLocalWindow("09:00", "10:00")), + 1, + 1, + 1, + 10, + ) + } + assertThrows(IllegalArgumentException::class.java) { + RandomWindowSchedule( + listOf(RandomLocalWindow("09:00", "09:30")), + occurrencesPerWindow = 2, + maximumOccurrencesPerDay = 2, + maximumOccurrencesTotal = 10, + minimumSeparationMinutes = 30, + ) + } + assertThrows(IllegalArgumentException::class.java) { + RandomWindowSchedule( + listOf(RandomLocalWindow("09:00", "10:00")), + occurrencesPerWindow = 1, + maximumOccurrencesPerDay = 2, + maximumOccurrencesTotal = 10, + minimumSeparationMinutes = 5, + ) + } + } + + @Test + fun randomOccurrenceBoundUsesTheSignedTotalUnderArbitraryWallClockEdits() { + assertEquals(3L, maximumReachableLocalDates(studyMinutes = 60)) + val triggers = (1..2).map { index -> + InterventionTrigger( + id = "random-trigger-$index", + schedule = RandomWindowSchedule( + localWindows = listOf(RandomLocalWindow("08:00", "09:00")), + occurrencesPerWindow = 8, + maximumOccurrencesPerDay = 8, + maximumOccurrencesTotal = 512, + minimumSeparationMinutes = 1, + ), + availabilityMinutes = 20, + ) + } + fun configuration(triggerCount: Int) = baseConfiguration( + collectors = listOf(BatteryStateConfiguration(required = false)), + interventions = listOf( + InterventionConfiguration( + id = "random-ema", + action = NotificationAction("Check in", "Please answer now."), + triggers = triggers.take(triggerCount), + ), + ), + durationHours = 1, + ) + + assertEquals(1, configuration(1).interventions.single().triggers.size) + assertThrows(IllegalArgumentException::class.java) { configuration(2) } + } + + private fun baseConfiguration( + collectors: List, + interventions: List, + durationHours: Int = 24, + ) = StudyConfiguration( + schemaVersion = 1, + experimentId = "p2-test", + configurationId = "p2-config", + issuedAt = Instant.parse("2026-01-01T00:00:00Z"), + expiresAt = Instant.parse("2030-01-01T00:00:00Z"), + platform = "android", + minimumClientVersion = 1, + title = "P2 test", + researcherName = "Researcher", + researcherContact = "researcher@example.invalid", + purpose = "Exercise P2 configuration contracts.", + durationHours = durationHours, + consentDocumentVersion = "v1", + consentSummary = "Consent summary.", + assignedParticipantId = null, + collectors = collectors, + surveys = emptyList(), + interventions = interventions, + maximumLocalBytes = StudyConfiguration.MINIMUM_LOCAL_BYTES, + signer = SignerIdentity("signer-key", ProtocolBase64Url.encode(ByteArray(32) { 1 })), + export = ExportConfiguration("export-key", ProtocolBase64Url.encode(ByteArray(32) { 2 })), + upload = null, + ) +} diff --git a/protocol/v1/README.md b/protocol/v1/README.md new file mode 100644 index 0000000..d2eeeb8 --- /dev/null +++ b/protocol/v1/README.md @@ -0,0 +1,303 @@ +# ADC Protocol v1 + +This directory is the normative, language-neutral contract for ADC Protocol v1. Kotlin, +TypeScript, Python, and future Swift implementations are conforming implementations; none of +them is the specification. + +Protocol v1 is a destructive pre-1.0 replacement. The names `schema_version: 1`, `ADCCFG01`, +`ADCEXP01`, and `research-bundle-v1` remain, but artifacts made by the former implementation are +invalid. Readers MUST NOT retain an old-v1 parser, migration path, dual interpretation, or +fallback. + +The companion [`collector-catalog.json`](collector-catalog.json) is the closed-world collector and +event schema. [`conformance-vectors.json`](conformance-vectors.json) and +[`join-link-vectors.json`](join-link-vectors.json) are the executable valid and hostile corpora. +Start with these files; platform code must not define a second contract. + +The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, and MAY are to be +interpreted as described by RFC 2119 and RFC 8174. + +## Common encoding rules + +- All binary integers are unsigned, big-endian, and use the exact width stated below. +- JSON is UTF-8 RFC 8785 JSON Canonicalization Scheme (JCS). Duplicate object members, + noncanonical bytes, malformed UTF-8, non-integral JSON numbers, and trailing bytes are invalid. +- JSON numbers are permitted only where the schema supplies bounded integral minimum and maximum + values. Sequence numbers, byte counts, Unix times, monotonic times, and client build numbers are + canonical decimal strings matching `0|[1-9][0-9]*`. +- UUIDs use lowercase RFC 4122 text in JSON and headers, and their 16 network-order bytes in binary + framing. Producers generate cryptographically random version-4 bundle IDs. +- SHA-256 values use 64 lowercase hexadecimal characters in JSON and ADC headers. +- Ed25519 and X25519 keys are raw 32-byte values encoded as unpadded base64url. Signatures are raw + 64-byte Ed25519 signatures encoded the same way in JSON. Tink JSON, protobuf keysets, X.509, + PKCS#8, padded base64, and standard-base64 wire keys are invalid. +- Every decoder is closed-world. An unknown member, enum, collector, payload type, platform, key + context, or framing byte fails the whole artifact. +- Implementations MUST reject values before allocating from a claimed length. A complete + `ADCEXP01` upload body is limited to 33,554,432 bytes (32 MiB). + +## Signed configuration (`ADCCFG01`) + +The configuration is an Android-targeted, closed-world JCS object with `schema_version` equal to +the JSON number `1`, `platform` equal to `"android"`, and `minimum_client_version` encoded as a +canonical decimal string. Android and future iOS configurations may share `experiment_id`; they +MUST use different `configuration_id` values and signatures. The Android client rejects every +other platform. + +All configuration quantities are integral physical units. In particular, +`location.v1.minimum_displacement_millimeters` replaces the former floating-point metre value. +The collector portion of the configuration is defined by +[`collector-catalog.json`](collector-catalog.json). + +The envelope is exactly: + +```text +offset size value +0 8 ASCII "ADCCFG01" +8 2 signer_key_id_length (u16) +10 4 configuration_length (u32) +14 K signer_key_id UTF-8 +14+K N configuration_jcs +14+K+N 64 Ed25519 signature +``` + +`signer_key_id_length` is in `3..64`, and its strict UTF-8 value matches +`[a-z0-9][a-z0-9-]{2,63}`. `configuration_length` is in `2..1,048,576`. The envelope ends after the +signature. The signed message is exactly `configuration_jcs`, without a framing prefix. The key ID +must equal `configuration.signer.key_id`. The verifier obtains the raw Ed25519 public key from +`configuration.signer.public_key`, verifies the fixed 64-byte signature, then applies signer +pinning policy. A valid self-contained signature proves integrity, not publisher identity. + +The configuration SHA-256 used everywhere below is SHA-256 over `configuration_jcs`, not over the +envelope. + +## Immutable join link (`adc://join/v1`) + +A join link is a transport pointer to one immutable `ADCCFG01` artifact. Its exact ASCII form is: + +```text +adc://join/v1?artifact=&sha256=<64-lowercase-hex>&signer_fingerprint=<32-uppercase-hex> +``` + +The query order is fixed. RFC 3986 unreserved bytes are literal; every other artifact-URL byte is +percent encoded with uppercase hexadecimal. Duplicate, missing, reordered, or unknown query +members, lowercase escapes, decoded non-ASCII, and links longer than 4,096 bytes are invalid. +`signer_fingerprint` is the first 16 bytes of SHA-256 over the raw Ed25519 public key. It has no +spaces on the wire. + +To prevent Java `URI` and WHATWG `URL` from silently accepting different text for one locator, the +decoded artifact URL uses this deliberately narrow canonical HTTPS profile: + +- at most 2,048 ASCII bytes and exactly lowercase `https://`; +- a lowercase DNS-style host of labels in `[a-z0-9-]`, each 1–63 bytes, with at least one ASCII + letter overall; no user information, IP literal, trailing dot, or internationalized host; +- no port, or a canonical decimal port in `1..65535` other than the redundant default `443`; +- one or more non-empty path segments containing only `[A-Za-z0-9._~-]`; `.` and `..` segments, + repeated slashes, percent escapes, a trailing slash, query, and fragment are invalid. + +This profile is intentionally sufficient for an immutable filename or opaque path token, not a +general browser URL. A personalized artifact MUST use at least 128 bits of random opaque path +material (the authoring tools require a final base64url segment of at least 22 characters), and +MUST NOT put an assigned participant ID in the URL or link. + +The Android app rejects a join while any active study or pending deletion exists. It performs one +bounded GET with redirects and implicit retries disabled, stages under no-backup storage, checks +the complete artifact SHA-256, then executes the ordinary Ed25519 verification and fingerprint / +consent flow. The host cannot replace accepted bytes: digest mismatch fails before signature +verification. Staging is cleared on process startup, before each attempt, and after success or +failure. There is no polling, refresh, replacement, background update, or assigned participant ID +in the join URI. + +## Encrypted bundle (`ADCEXP01`) + +The only cryptographic suite is RFC 9180 base mode (`mode = 0x00`) with: + +| Parameter | Value | +| --- | --- | +| KEM | DHKEM(X25519, HKDF-SHA256), `0x0020` | +| KDF | HKDF-SHA256, `0x0001` | +| AEAD | AES-256-GCM, `0x0002` | + +Each bundle creates an independent random 32-byte AES-256-GCM content key and a fresh random +12-byte content nonce. HPKE seals the 32-byte content key to the configuration's researcher X25519 +public key. With the fixed suite, `enc` is 32 bytes and the sealed content-key ciphertext is 48 +bytes including its tag. + +The container is exactly; document ciphertext consumes the remainder of the file: + +```text +offset size value +0 8 ASCII "ADCEXP01" +8 16 bundle_id UUID bytes +24 32 configuration_sha256 +56 2 researcher_key_id_length (u16) +58 12 AES-256-GCM content nonce +70 K researcher_key_id UTF-8 +70+K 80 HPKE wrapped content key: enc[32] || sealed_key[48] +150+K C encrypted document and 16-byte GCM tag, to end of file +``` + +`researcher_key_id_length` is in `3..64`; the decoded value matches +`[a-z0-9][a-z0-9-]{2,63}`. `C` is greater than the 16-byte GCM tag. An automatic-upload container is +at most 32 MiB; a manual export has no 32 MiB wire limit and is instead bounded by the signed local +storage quota, so manual-export readers stream it. There is no ciphertext-length field: the file or +HTTP body ends the container, and truncation or appended bytes fail authentication or JCS +validation. + +### Cryptographic context + +The following exact JCS bytes bind both cryptographic layers (`bundle_id` is lowercase UUID text): + +```text +context = UTF8({"bundle_format":"research-bundle-v1","bundle_id":"","configuration_sha256":"","researcher_key_id":""}) +``` + +The context is RFC 9180 `info` when sealing the content key; HPKE base-mode `aad` is empty. The +context bytes are separately used as AES-256-GCM associated data for the document. A wrong bundle format, bundle ID, +configuration digest, researcher key ID, HPKE suite, `enc`, sealed key, or content nonce fails +authentication. + +### Authenticated document + +The decrypted bytes are one JCS `research-bundle-v1` object with exactly these root members: + +| Member | Type and rule | +| --- | --- | +| `format` | exactly `"research-bundle-v1"` | +| `bundle_id` | the outer UUID as lowercase text | +| `bundle_kind` | `"manual_export"` or `"automatic_upload"` | +| `configuration_sha256` | the outer digest | +| `configuration` | the exact signed configuration object; its JCS digest must match | +| `configuration_signature` | exact `{signer_key_id, signature}` object; signature is unpadded base64url raw Ed25519[64] | +| `producer` | exact `{client_version, platform}` object; version is a positive decimal string and platform matches configuration | +| `exported_at_utc_millis` | canonical decimal string | +| `experiment` | exact experiment snapshot object described below | + +The `experiment` object has exactly `assigned_participant_id`, `configuration_id`, +`durable_through_sequence`, `event_count`, `events`, `experiment_id`, `first_sequence_number`, +`last_sequence_number`, `next_sequence_number`, `participant_instance_id`, +`retained_from_sequence`, `state`, `transitions`, and `uploaded_through_sequence`. All sequence and +count values are canonical decimal strings. For a non-empty document, +`event_count = last_sequence_number - first_sequence_number + 1`; event sequence numbers are +strictly contiguous and cover that exact range. An automatic upload is never empty. For an empty +manual export, `event_count` is `"0"`, `last_sequence_number = first_sequence_number - 1`, and +`events` is empty. + +Each event has exactly `sequence_number`, `collector_id`, `payload_schema_version`, +`observed_time`, `payload_type`, and `fields`. Sequence, `wall_time_utc_millis`, and +`monotonic_time_nanos` are canonical decimal strings; `boot_session_id` is 1–128 UTF-8 bytes. The +event's collector, schema version, payload type, field set, values, and encoded size must +validate against the catalog. Unknown payloads do not become generic rows. + +Catalog `int32` payload values remain JSON strings and have one signed decimal spelling: +`0|-?[1-9][0-9]*`. Leading `+`, leading zeroes, and `-0` are invalid; the parsed value must also fit +signed 32-bit range and the field's catalog minimum / maximum. + +Catalog `float32` and `float64` payload values are also JSON strings. Their exact decimal grammar +is `[+-]?(?:(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?)`. Empty or whitespace-padded +values, hexadecimal/binary spellings, separators, `NaN`, and infinities are invalid. The parsed +value must be finite, fit its declared precision, and satisfy the catalog bounds. + +The reader verifies, in order: framing and bounds; HPKE; content AEAD; JCS bytes; repeated outer +identities; embedded configuration digest and Ed25519 signature; platform/client requirements; +range/count contiguity; and every catalog payload. It publishes no plaintext-derived record before +all checks succeed. + +## Automatic upload request + +The receiver exposes one endpoint chosen at deployment. The request is `POST`; redirects are +forbidden. It has a fixed `Content-Length`, no `Transfer-Encoding`, and an immutable `ADCEXP01` +body staged before HTTP starts. The exact request headers are: + +| Header | Value | +| --- | --- | +| `Content-Type` | `application/vnd.adc.research-bundle` | +| `Content-Length` | canonical decimal body byte count, at most 33,554,432 | +| `Content-Digest` | RFC 9530 `sha-256=::` | +| `X-ADC-Bundle-Format` | `research-bundle-v1` | +| `X-ADC-Bundle-Id` | lowercase bundle UUID | +| `X-ADC-Configuration-SHA256` | 64 lowercase hex characters | +| `X-ADC-Researcher-Key-Id` | researcher key ID | +| `X-ADC-Sequence-From` | canonical decimal exact first sequence | +| `X-ADC-Sequence-To` | canonical decimal exact last sequence | +| `X-ADC-Event-Count` | canonical decimal count | + +The routing headers are untrusted claims. The receiver checks their syntax, internal range/count +arithmetic, body length/digest, and equality to the parseable outer bundle ID, configuration +digest, and researcher key ID. It cannot authenticate the encrypted participant or sequence claims +and MUST NOT describe them as authenticated. + +The body and every header are fixed for all attempts of one staged bundle. Clients disable +automatic redirects and transport-library request replay. Only I/O failure, 408, 425, 429, and 5xx +are retryable. `202 Accepted`, redirects, every other 4xx, malformed receipts, and receipt mismatch +are terminal delivery failures; they do not stop collection. + +## Receiver write and receipt + +The receiver streams the bounded body directly into a new R2 object whose key is the lowercase +bundle UUID. It verifies SHA-256 during the write, uses a create-only conditional write, and returns +success only after R2 durability is confirmed. + +- New immutable object: `201 Created`. +- Existing object with identical byte count, content digest, configuration digest, key ID, and + claimed range/count metadata: `200 OK`, with the original receipt bytes. +- Existing bundle ID with any mismatch: `409 Conflict`; it is never overwritten. + +A success body is JCS JSON with `Content-Type: application/json` and exactly the following members +(shown expanded for readability; response bytes are compact JCS): + +```json +{ + "bundle_id": "550e8400-e29b-41d4-a716-446655440000", + "byte_count": "1234", + "configuration_sha256": "64 lowercase hex characters", + "event_count": "1", + "first_sequence_number": "1", + "last_sequence_number": "1", + "sha256": "64 lowercase hex characters" +} +``` + +Both `201 Created` and exact-replay `200 OK` return this same canonical seven-member receipt. Before +advancing its watermark, the client requires every receipt value to match its durable outbox +manifest exactly. Receive time, researcher key ID, and claimed range may additionally be retained +as untrusted R2 custom metadata; they are not added to the receipt JSON. + +The receiver has no list, download, delete, administration, decryption, private-key, D1, Queue, KV, +Durable Object, dashboard, or runtime-configuration path. Deployment-time allowlists, WAF/rate +limits, R2 lifecycle rules, and minimal S3 read credentials are operational controls, not protocol +extensions. + +## Conformance + +Every implementation must consume the shared valid and hostile corpus in this directory. The +corpus must cover Unicode JCS ordering, integral bounds, raw-key encodings, signature input, HPKE +labels and wrong contexts, malformed lengths, wrong outer/inner identities, body tampering, +range/count mismatch, old-v1 rejection, unknown fields and payloads, non-finite sensor values, and +trailing bytes. Absence of a vector is not permission to accept an unspecified encoding. + +The join-link corpus is consumed by Kotlin and TypeScript, the two implementations that create or +open join links. Python analysis has no join-link entrypoint and does not interpret that corpus. + +Validate the checked-in sources with: + +```sh +python3 tools/catalog.py check +python3 tools/validate_protocol_vectors.py +``` + +After deliberately changing a wire rule, regenerate the deterministic corpus with +`node tools/generate_protocol_vectors.mjs` and make every language consumer pass the new bytes in +the same change. + +## Implementation map + +For the join path, Web authoring is in `web/src/lib/adc/join.ts` and +`web/src/routes/researcher/JoinLinkPanel.svelte`; the shared parser is +`core/protocol/.../JoinLink.kt`; Android staging is +`app/.../platform/JoinArtifactDownloader.kt`; the `adc://join/v1` intent enters through +`app/.../MainActivity.kt`; and digest → signature → fingerprint binding is enforced by +`core/study-application/.../StudyApplication.kt`. The adjacent tests and shared +`join-link-vectors.json` are the executable map. Automatic upload instead follows the outbox and +HTTP adapter named in the repository README; receiver and offline analysis each have their own +README code map. diff --git a/protocol/v1/collector-catalog.json b/protocol/v1/collector-catalog.json new file mode 100644 index 0000000..61a0dd1 --- /dev/null +++ b/protocol/v1/collector-catalog.json @@ -0,0 +1,1411 @@ +{ + "catalog_format": "adc-collector-catalog-v1", + "catalog_version": 1, + "collectors": [ + { + "access": [ + { + "kind": "ACCELEROMETER_HARDWARE", + "mode": "hardware" + } + ], + "configuration": { + "fields": { + "maximum_report_latency_us": { + "maximum": 60000000, + "meaning": "Maximum Android hardware FIFO batching latency.", + "minimum": 0, + "type": "integer", + "unit": "microsecond" + }, + "sampling_period_us": { + "maximum": 1000000, + "meaning": "Requested Android sensor sampling period; this is a hint, not an achieved-rate guarantee.", + "minimum": 5000, + "type": "integer", + "unit": "microsecond" + } + }, + "required": [ + "maximum_report_latency_us", + "sampling_period_us" + ] + }, + "id": "accelerometer.v1", + "implementation": { + "android_module": ":collector:accelerometer", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 2048, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "accuracy": { + "clock_basis": null, + "meaning": "Raw Android SensorEvent accuracy code.", + "required": true, + "type": "int32", + "unit": "android_sensor_accuracy" + }, + "source_elapsed_realtime_nanos": { + "clock_basis": "continuous_monotonic_since_boot", + "meaning": "SensorEvent hardware sample timestamp.", + "required": true, + "type": "decimal_string", + "unit": "nanosecond" + }, + "x_meters_per_second_squared": { + "clock_basis": null, + "meaning": "Raw acceleration including gravity on the device X axis.", + "required": true, + "type": "float32", + "unit": "meter_per_second_squared" + }, + "y_meters_per_second_squared": { + "clock_basis": null, + "meaning": "Raw acceleration including gravity on the device Y axis.", + "required": true, + "type": "float32", + "unit": "meter_per_second_squared" + }, + "z_meters_per_second_squared": { + "clock_basis": null, + "meaning": "Raw acceleration including gravity on the device Z axis.", + "required": true, + "type": "float32", + "unit": "meter_per_second_squared" + } + }, + "types": [ + "ACCELEROMETER_SAMPLE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "android_platform_without_high-rate-permission", + "kind": "android_platform_cap", + "maximum_events_per_hour": 720000 + }, + "selectable": true + }, + { + "access": [ + { + "kind": "AMBIENT_LIGHT_HARDWARE", + "mode": "hardware" + } + ], + "configuration": { + "fields": { + "change_threshold_millilux": { + "maximum": 100000000, + "meaning": "Minimum illuminance change from the last emitted sample. The newest changed on-change sample inside the rate interval is retained and emitted later with its original observation and hardware times.", + "minimum": 0, + "type": "integer", + "unit": "millilux" + }, + "sampling_period_us": { + "maximum": 10000000, + "meaning": "Minimum emitted-sample period and Android sensor request period; rate limiting coalesces to the newest meaningful sample rather than dropping it.", + "minimum": 200000, + "type": "integer", + "unit": "microsecond" + } + }, + "required": [ + "change_threshold_millilux", + "sampling_period_us" + ] + }, + "id": "ambient_light.v1", + "implementation": { + "android_module": ":collector:ambient-light", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 1024, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "accuracy": { + "clock_basis": null, + "meaning": "Raw Android SensorEvent accuracy code.", + "required": true, + "type": "int32", + "unit": "android_sensor_accuracy" + }, + "illuminance_lux": { + "clock_basis": null, + "meaning": "Raw ambient illuminance reported by the light sensor.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "lux" + }, + "source_elapsed_realtime_nanos": { + "clock_basis": "continuous_monotonic_since_boot", + "meaning": "SensorEvent hardware sample timestamp.", + "required": true, + "type": "decimal_string", + "unit": "nanosecond" + } + }, + "types": [ + "AMBIENT_LIGHT_SAMPLE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "collector-monotonic-rate-gate", + "kind": "collector_rate_limit", + "maximum_events_per_hour": 18000 + }, + "selectable": true + }, + { + "access": [], + "configuration": { + "fields": {}, + "required": [] + }, + "id": "app_lifecycle.v1", + "implementation": { + "android_module": ":collector:app-lifecycle", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 2048, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "activity_class": { + "clock_basis": null, + "maximum_length": 512, + "meaning": "Fully-qualified class name of this app's Activity.", + "required": true, + "type": "string", + "unit": "none" + } + }, + "types": [ + "ACTIVITY_CREATED", + "ACTIVITY_DESTROYED", + "ACTIVITY_INSTANCE_STATE_SAVED", + "ACTIVITY_PAUSED", + "ACTIVITY_RESUMED", + "ACTIVITY_STARTED", + "ACTIVITY_STOPPED" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "none", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": true + }, + { + "access": [], + "configuration": { + "fields": {}, + "required": [] + }, + "id": "battery_state.v1", + "implementation": { + "android_module": ":collector:battery-state", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 1024, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "charging_source": { + "clock_basis": null, + "enum": [ + "AC", + "DOCK", + "MULTIPLE", + "NONE", + "UNKNOWN", + "USB", + "WIRELESS" + ], + "meaning": "Mapped Android charging source without hardware identity.", + "required": true, + "type": "enum", + "unit": "none" + }, + "charging_state": { + "clock_basis": null, + "enum": [ + "CHARGING", + "DISCHARGING", + "FULL", + "NOT_CHARGING", + "UNKNOWN" + ], + "meaning": "Mapped Android battery charging status.", + "required": true, + "type": "enum", + "unit": "none" + }, + "percentage": { + "clock_basis": null, + "maximum": 100, + "meaning": "Whole remaining-battery percentage.", + "minimum": 0, + "required": true, + "type": "int32", + "unit": "percent" + }, + "power_save_enabled": { + "clock_basis": null, + "meaning": "Whether Android power-save mode is enabled.", + "required": true, + "type": "boolean", + "unit": "none" + } + }, + "types": [ + "BATTERY_STATE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "collector-coalescing-rate-gate", + "kind": "collector_rate_limit", + "maximum_events_per_hour": 60 + }, + "selectable": true + }, + { + "access": [ + { + "kind": "GYROSCOPE_HARDWARE", + "mode": "hardware" + } + ], + "configuration": { + "fields": { + "maximum_report_latency_us": { + "maximum": 60000000, + "meaning": "Maximum Android hardware FIFO batching latency.", + "minimum": 0, + "type": "integer", + "unit": "microsecond" + }, + "sampling_period_us": { + "maximum": 1000000, + "meaning": "Requested Android sensor sampling period; this is a hint, not an achieved-rate guarantee.", + "minimum": 5000, + "type": "integer", + "unit": "microsecond" + } + }, + "required": [ + "maximum_report_latency_us", + "sampling_period_us" + ] + }, + "id": "gyroscope.v1", + "implementation": { + "android_module": ":collector:gyroscope", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 2048, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "accuracy": { + "clock_basis": null, + "meaning": "Raw Android SensorEvent accuracy code.", + "required": true, + "type": "int32", + "unit": "android_sensor_accuracy" + }, + "source_elapsed_realtime_nanos": { + "clock_basis": "continuous_monotonic_since_boot", + "meaning": "SensorEvent hardware sample timestamp.", + "required": true, + "type": "decimal_string", + "unit": "nanosecond" + }, + "x_radians_per_second": { + "clock_basis": null, + "meaning": "Raw angular velocity around the device X axis.", + "required": true, + "type": "float32", + "unit": "radian_per_second" + }, + "y_radians_per_second": { + "clock_basis": null, + "meaning": "Raw angular velocity around the device Y axis.", + "required": true, + "type": "float32", + "unit": "radian_per_second" + }, + "z_radians_per_second": { + "clock_basis": null, + "meaning": "Raw angular velocity around the device Z axis.", + "required": true, + "type": "float32", + "unit": "radian_per_second" + } + }, + "types": [ + "GYROSCOPE_SAMPLE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "android-platform-without-high-rate-permission", + "kind": "android_platform_cap", + "maximum_events_per_hour": 720000 + }, + "selectable": true + }, + { + "access": [], + "configuration": null, + "id": "interventions.v1", + "implementation": { + "android_module": ":core:experiment-runtime", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 65536, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "intervention_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "Stable signed intervention identifier.", + "required": true, + "type": "string", + "unit": "none" + }, + "occurrence_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "SHA-256 identity of the logical occurrence.", + "required": true, + "type": "string", + "unit": "none" + }, + "scheduled_for_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Materialized scheduled instant.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "trigger_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "Stable signed trigger identifier.", + "required": true, + "type": "string", + "unit": "none" + } + }, + "types": [ + "INTERVENTION_EXPIRED", + "INTERVENTION_OPENED", + "INTERVENTION_RESCHEDULED", + "INTERVENTION_SCHEDULED", + "NOTIFICATION_POSTED", + "SURVEY_EXPIRED", + "SURVEY_OPENED" + ] + }, + { + "fields": { + "answers_json": { + "clock_basis": null, + "maximum_length": 61440, + "meaning": "Validated final survey answers as compact JSON keyed by stable question IDs.", + "required": true, + "type": "json_string", + "unit": "none" + }, + "intervention_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "Stable signed intervention identifier.", + "required": true, + "type": "string", + "unit": "none" + }, + "occurrence_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "SHA-256 identity of the logical occurrence.", + "required": true, + "type": "string", + "unit": "none" + }, + "opened_time": { + "clock_basis": "research_time_object", + "meaning": "Compact canonical ResearchTime JSON for the open action.", + "required": true, + "type": "json_string", + "unit": "none" + }, + "scheduled_for_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Materialized scheduled instant.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "scheduled_time": { + "clock_basis": "research_time_object", + "meaning": "Compact canonical ResearchTime JSON for the scheduled instant.", + "required": true, + "type": "json_string", + "unit": "none" + }, + "submitted_time": { + "clock_basis": "research_time_object", + "meaning": "Compact canonical ResearchTime JSON for submission.", + "required": true, + "type": "json_string", + "unit": "none" + }, + "survey_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "Stable signed survey identifier.", + "required": true, + "type": "string", + "unit": "none" + }, + "trigger_id": { + "clock_basis": null, + "maximum_length": 64, + "meaning": "Stable signed trigger identifier.", + "required": true, + "type": "string", + "unit": "none" + } + }, + "types": [ + "SURVEY_SUBMITTED" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "none-for-lifecycle-reschedules", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": false + }, + { + "access": [ + { + "kind": "RESEARCH_KEYBOARD_ENABLED", + "mode": "participant_setting" + }, + { + "kind": "RESEARCH_KEYBOARD_SELECTED", + "mode": "participant_setting" + } + ], + "configuration": { + "fields": { + "trajectory_sampling_hz": { + "maximum": 120, + "meaning": "Maximum MOVE-event sampling rate; boundary actions are not rate limited.", + "minimum": 1, + "type": "integer", + "unit": "hertz" + } + }, + "required": [ + "trajectory_sampling_hz" + ] + }, + "id": "keyboard_touch.v1", + "implementation": { + "android_module": ":collector:keyboard-ime", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 4096, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "action": { + "clock_basis": null, + "enum": [ + "CANCEL", + "DOWN", + "MOVE", + "UP" + ], + "meaning": "Touch lifecycle action.", + "required": true, + "type": "enum", + "unit": "none" + }, + "down_uptime_millis": { + "clock_basis": "uptime_since_boot_excluding_deep_sleep", + "meaning": "Uptime when the gesture began.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "event_uptime_millis": { + "clock_basis": "uptime_since_boot_excluding_deep_sleep", + "meaning": "Uptime of this touch event.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "geometry_version": { + "clock_basis": null, + "enum": [ + "qwerty-v1" + ], + "meaning": "Fixed research-keyboard geometry identity.", + "required": true, + "type": "enum", + "unit": "none" + }, + "key_category": { + "clock_basis": null, + "enum": [ + "BACKSPACE", + "ENTER", + "LETTER", + "SPACE" + ], + "meaning": "Key category only; never key identity or text.", + "required": true, + "type": "enum", + "unit": "none" + }, + "orientation_radians": { + "clock_basis": null, + "meaning": "Raw MotionEvent touch-major-axis orientation.", + "required": true, + "type": "float32", + "unit": "radian" + }, + "pointer_id": { + "clock_basis": null, + "meaning": "Pointer identity within the current gesture.", + "minimum": 0, + "required": true, + "type": "int32", + "unit": "none" + }, + "pressure": { + "clock_basis": null, + "meaning": "Uncalibrated device-relative MotionEvent pressure.", + "required": true, + "type": "float32", + "unit": "device_relative" + }, + "relative_x": { + "clock_basis": null, + "maximum": 1, + "meaning": "Clamped horizontal position within the touched key.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "fraction" + }, + "relative_y": { + "clock_basis": null, + "maximum": 1, + "meaning": "Clamped vertical position within the touched key.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "fraction" + }, + "size": { + "clock_basis": null, + "meaning": "Uncalibrated device-relative contact size.", + "required": true, + "type": "float32", + "unit": "device_relative" + }, + "tool_type": { + "clock_basis": null, + "meaning": "Raw Android MotionEvent tool-type code.", + "required": true, + "type": "int32", + "unit": "android_tool_type" + } + }, + "types": [ + "KEYBOARD_TOUCH" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "RESTRICTED", + "rate_bound": { + "enforced_by": "move-events-only", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": true + }, + { + "access": [ + { + "kind": "BACKGROUND_LOCATION", + "mode": "runtime_permission" + }, + { + "kind": "FINE_LOCATION", + "mode": "runtime_permission" + } + ], + "configuration": { + "fields": { + "interval_millis": { + "maximum": 3600000, + "meaning": "Requested fused-location interval.", + "minimum": 1000, + "type": "integer", + "unit": "millisecond" + }, + "maximum_batch_delay_millis": { + "maximum": 86400000, + "meaning": "Maximum fused-location batching delay.", + "minimum": 0, + "type": "integer", + "unit": "millisecond" + }, + "minimum_displacement_millimeters": { + "maximum": 10000000, + "meaning": "Requested minimum displacement, represented without a floating-point configuration value.", + "minimum": 0, + "type": "integer", + "unit": "millimeter" + }, + "minimum_interval_millis": { + "maximum": 3600000, + "maximum_field": "interval_millis", + "meaning": "Fastest requested fused-location interval; additionally bounded by interval_millis.", + "minimum": 500, + "type": "integer", + "unit": "millisecond" + }, + "priority": { + "enum": [ + "BALANCED", + "HIGH_ACCURACY" + ], + "meaning": "Fused-location power/accuracy priority within precise location.", + "type": "enum", + "unit": "none" + } + }, + "required": [ + "interval_millis", + "maximum_batch_delay_millis", + "minimum_displacement_millimeters", + "minimum_interval_millis", + "priority" + ] + }, + "id": "location.v1", + "implementation": { + "android_module": ":collector:location", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 4096, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "altitude_meters": { + "clock_basis": null, + "meaning": "WGS84 ellipsoid altitude when supplied.", + "required": false, + "type": "float64", + "unit": "meter" + }, + "bearing_accuracy_degrees": { + "clock_basis": null, + "meaning": "Bearing accuracy when supplied.", + "minimum": 0, + "required": false, + "type": "float32", + "unit": "degree" + }, + "bearing_degrees": { + "clock_basis": null, + "maximum": 360, + "meaning": "Bearing from true north when supplied.", + "minimum": 0, + "required": false, + "type": "float32", + "unit": "degree" + }, + "horizontal_accuracy_meters": { + "clock_basis": null, + "meaning": "Horizontal 68-percent confidence radius.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "meter" + }, + "latitude_degrees": { + "clock_basis": null, + "maximum": 90, + "meaning": "WGS84 latitude.", + "minimum": -90, + "required": true, + "type": "float64", + "unit": "degree" + }, + "longitude_degrees": { + "clock_basis": null, + "maximum": 180, + "meaning": "WGS84 longitude.", + "minimum": -180, + "required": true, + "type": "float64", + "unit": "degree" + }, + "mock": { + "clock_basis": null, + "meaning": "Whether Android marks this location as mock.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "source_elapsed_realtime_nanos": { + "clock_basis": "continuous_monotonic_since_boot", + "meaning": "Provider monotonic fix timestamp.", + "required": true, + "type": "decimal_string", + "unit": "nanosecond" + }, + "source_time_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Provider wall-clock fix timestamp.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "speed_accuracy_meters_per_second": { + "clock_basis": null, + "meaning": "Speed accuracy when supplied.", + "minimum": 0, + "required": false, + "type": "float32", + "unit": "meter_per_second" + }, + "speed_meters_per_second": { + "clock_basis": null, + "meaning": "Provider speed when supplied.", + "minimum": 0, + "required": false, + "type": "float32", + "unit": "meter_per_second" + }, + "vertical_accuracy_meters": { + "clock_basis": null, + "meaning": "Vertical accuracy when supplied.", + "minimum": 0, + "required": false, + "type": "float32", + "unit": "meter" + } + }, + "types": [ + "LOCATION_FIX" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "none-android-request-is-a-hint", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": true + }, + { + "access": [ + { + "kind": "ACCESS_NETWORK_STATE", + "mode": "install_permission" + } + ], + "configuration": { + "fields": { + "include_bandwidth_estimates": { + "meaning": "Whether capability events include Android link-bandwidth estimates.", + "type": "boolean", + "unit": "none" + } + }, + "required": [ + "include_bandwidth_estimates" + ] + }, + "id": "network_state.v1", + "implementation": { + "android_module": ":collector:network-state", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 4096, + "payload_schema_version": 1, + "payloads": [ + { + "fields": {}, + "types": [ + "NETWORK_AVAILABLE", + "NETWORK_LOST" + ] + }, + { + "fields": { + "downstream_kbps": { + "clock_basis": null, + "meaning": "Android downstream link-bandwidth estimate when enabled.", + "minimum": 0, + "required": false, + "type": "int32", + "unit": "kilobit_per_second" + }, + "ethernet": { + "clock_basis": null, + "meaning": "Whether the default network has ethernet transport.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "metered": { + "clock_basis": null, + "meaning": "Whether Android reports the default network as metered.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "mobile": { + "clock_basis": null, + "meaning": "Whether the default network has cellular transport.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "roaming": { + "clock_basis": null, + "meaning": "Whether Android reports the default network as roaming.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "upstream_kbps": { + "clock_basis": null, + "meaning": "Android upstream link-bandwidth estimate when enabled.", + "minimum": 0, + "required": false, + "type": "int32", + "unit": "kilobit_per_second" + }, + "validated": { + "clock_basis": null, + "meaning": "Whether Android validated internet connectivity.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "vpn": { + "clock_basis": null, + "meaning": "Whether the default network has VPN transport.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "wifi": { + "clock_basis": null, + "meaning": "Whether the default network has Wi-Fi transport.", + "required": true, + "type": "boolean", + "unit": "none" + } + }, + "types": [ + "NETWORK_CAPABILITIES" + ] + }, + { + "fields": { + "connected": { + "clock_basis": null, + "meaning": "Whether a default network was available at snapshot time.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "downstream_kbps": { + "clock_basis": null, + "meaning": "Android downstream link-bandwidth estimate when connected and enabled.", + "minimum": 0, + "required": false, + "type": "int32", + "unit": "kilobit_per_second" + }, + "ethernet": { + "clock_basis": null, + "meaning": "Ethernet transport; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "metered": { + "clock_basis": null, + "meaning": "Metered state; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "mobile": { + "clock_basis": null, + "meaning": "Cellular transport; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "roaming": { + "clock_basis": null, + "meaning": "Roaming state; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "upstream_kbps": { + "clock_basis": null, + "meaning": "Android upstream link-bandwidth estimate when connected and enabled.", + "minimum": 0, + "required": false, + "type": "int32", + "unit": "kilobit_per_second" + }, + "validated": { + "clock_basis": null, + "meaning": "Validated state; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "vpn": { + "clock_basis": null, + "meaning": "VPN transport; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + }, + "wifi": { + "clock_basis": null, + "meaning": "Wi-Fi transport; omitted when disconnected.", + "required": false, + "type": "boolean", + "unit": "none" + } + }, + "types": [ + "NETWORK_SNAPSHOT" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "none", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": true + }, + { + "access": [ + { + "kind": "USAGE_ACCESS", + "mode": "special_access" + } + ], + "configuration": { + "fields": { + "poll_interval_minutes": { + "maximum": 1440, + "meaning": "Polling interval for Android device-total counters.", + "minimum": 1, + "type": "integer", + "unit": "minute" + }, + "transports": { + "items_enum": [ + "mobile", + "wifi" + ], + "maximum_items": 2, + "meaning": "Device-total transports to query.", + "minimum_items": 1, + "type": "enum_array", + "unit": "none" + } + }, + "required": [ + "poll_interval_minutes", + "transports" + ] + }, + "id": "network_usage.v1", + "implementation": { + "android_module": ":collector:network-usage", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 2048, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "coverage_end_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Exclusive end of the queried accounting window.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "coverage_start_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Inclusive start of the queried accounting window.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + }, + "rx_bytes": { + "clock_basis": null, + "meaning": "Device-total received bytes in the window.", + "required": true, + "type": "decimal_string", + "unit": "byte" + }, + "rx_packets": { + "clock_basis": null, + "meaning": "Device-total received packets in the window.", + "required": true, + "type": "decimal_string", + "unit": "packet" + }, + "transport": { + "clock_basis": null, + "enum": [ + "MOBILE", + "WIFI" + ], + "meaning": "Transport whose device-total counters were queried.", + "required": true, + "type": "enum", + "unit": "none" + }, + "tx_bytes": { + "clock_basis": null, + "meaning": "Device-total transmitted bytes in the window.", + "required": true, + "type": "decimal_string", + "unit": "byte" + }, + "tx_packets": { + "clock_basis": null, + "meaning": "Device-total transmitted packets in the window.", + "required": true, + "type": "decimal_string", + "unit": "packet" + } + }, + "types": [ + "NETWORK_USAGE_AGGREGATE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "poll-interval-and-two-transport-maximum", + "kind": "poll_configuration", + "maximum_events_per_hour": 120 + }, + "selectable": true + }, + { + "access": [ + { + "kind": "PROXIMITY_HARDWARE", + "mode": "hardware" + } + ], + "configuration": { + "fields": { + "change_threshold_millimeters": { + "maximum": 10000, + "meaning": "Minimum raw-distance change before another same-state event is emitted.", + "minimum": 0, + "type": "integer", + "unit": "millimeter" + }, + "minimum_event_interval_ms": { + "maximum": 60000, + "meaning": "Minimum interval between emitted events; the latest pending state is retained.", + "minimum": 100, + "type": "integer", + "unit": "millisecond" + } + }, + "required": [ + "change_threshold_millimeters", + "minimum_event_interval_ms" + ] + }, + "id": "proximity.v1", + "implementation": { + "android_module": ":collector:proximity", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 1024, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "distance_centimeters": { + "clock_basis": null, + "meaning": "Raw Android proximity distance; binary sensors report only near and maximum range.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "centimeter" + }, + "maximum_range_centimeters": { + "clock_basis": null, + "meaning": "Maximum range declared by this device's sensor.", + "minimum": 0, + "required": true, + "type": "float32", + "unit": "centimeter" + }, + "near": { + "clock_basis": null, + "meaning": "True when raw distance is less than the sensor maximum range.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "source_elapsed_realtime_nanos": { + "clock_basis": "continuous_monotonic_since_boot", + "meaning": "SensorEvent hardware sample timestamp.", + "required": true, + "type": "decimal_string", + "unit": "nanosecond" + } + }, + "types": [ + "PROXIMITY_SAMPLE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "collector-coalescing-rate-gate", + "kind": "collector_rate_limit", + "maximum_events_per_hour": 36000 + }, + "selectable": true + }, + { + "access": [], + "configuration": { + "fields": {}, + "required": [] + }, + "id": "temporal_context.v1", + "implementation": { + "android_module": ":collector:temporal-context", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 1024, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "change_reason": { + "clock_basis": null, + "enum": [ + "RECONCILED", + "STUDY_STARTED", + "TIMEZONE_CHANGED", + "TIME_SET", + "UTC_OFFSET_CHANGED" + ], + "meaning": "Reason the temporal-context snapshot was emitted.", + "required": true, + "type": "enum", + "unit": "none" + }, + "daylight_saving_time": { + "clock_basis": null, + "meaning": "Whether the selected zone rules are in daylight-saving time at observation.", + "required": true, + "type": "boolean", + "unit": "none" + }, + "timezone_id": { + "clock_basis": null, + "maximum_length": 128, + "meaning": "Current device ZoneId; this is a setting, not a location claim.", + "required": true, + "type": "string", + "unit": "none" + }, + "utc_offset_seconds": { + "clock_basis": null, + "maximum": 64800, + "meaning": "Zone-rule UTC offset at the observation instant.", + "minimum": -64800, + "required": true, + "type": "int32", + "unit": "second" + } + }, + "types": [ + "TEMPORAL_CONTEXT" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "distinct-minute-context-gate", + "kind": "collector_rate_limit", + "maximum_events_per_hour": 60 + }, + "selectable": true + }, + { + "access": [ + { + "kind": "USAGE_ACCESS", + "mode": "special_access" + } + ], + "configuration": { + "fields": { + "poll_interval_minutes": { + "maximum": 1440, + "meaning": "Interval between UsageStats queries; it does not improve source completeness.", + "minimum": 1, + "type": "integer", + "unit": "minute" + } + }, + "required": [ + "poll_interval_minutes" + ] + }, + "id": "usage_events.v1", + "implementation": { + "android_module": ":collector:usage-events", + "status": "implemented" + }, + "maximum_encoded_event_bytes": 4096, + "payload_schema_version": 1, + "payloads": [ + { + "fields": { + "package_name": { + "clock_basis": null, + "maximum_length": 255, + "meaning": "Reporting package when Android supplies a nonblank value.", + "required": false, + "type": "string", + "unit": "none" + }, + "source_time_utc_millis": { + "clock_basis": "wall_utc", + "meaning": "Android UsageEvents source timestamp.", + "required": true, + "type": "decimal_string", + "unit": "millisecond" + } + }, + "types": [ + "ACTIVITY_PAUSED", + "ACTIVITY_RESUMED", + "ACTIVITY_STOPPED", + "DEVICE_SHUTDOWN", + "DEVICE_STARTUP", + "KEYGUARD_HIDDEN", + "KEYGUARD_SHOWN", + "SCREEN_INTERACTIVE", + "SCREEN_NON_INTERACTIVE" + ] + } + ], + "platforms": [ + "android" + ], + "privacy_class": "SENSITIVE", + "rate_bound": { + "enforced_by": "none-source-query-can-return-an-unbounded-count", + "kind": "not_enforced", + "maximum_events_per_hour": null + }, + "selectable": true + } + ], + "protocol_schema_version": 1 +} diff --git a/protocol/v1/conformance-vectors.json b/protocol/v1/conformance-vectors.json new file mode 100644 index 0000000..bac0986 --- /dev/null +++ b/protocol/v1/conformance-vectors.json @@ -0,0 +1,430 @@ +{ + "corpus_format": "adc-protocol-conformance-v1", + "hostile": [ + { + "category": "unicode_jcs", + "entrypoint": "canonical_json", + "expected_failure": "utf16_key_order", + "id": "jcs-wrong-utf16-key-order", + "input_hex": "7b22ee8080223a302c22f0908080223a317d" + }, + { + "category": "unicode_jcs", + "entrypoint": "canonical_json", + "expected_failure": "malformed_utf8", + "id": "jcs-malformed-utf8", + "input_hex": "7b2278223a22c328227d" + }, + { + "category": "unicode_jcs", + "entrypoint": "canonical_json", + "expected_failure": "unpaired_surrogate", + "id": "jcs-unpaired-surrogate", + "input_hex": "7b2278223a225c7564383030227d" + }, + { + "category": "unicode_jcs", + "entrypoint": "canonical_json", + "expected_failure": "noncanonical_escape", + "id": "jcs-noncanonical-unicode-escape", + "input_hex": "7b2278223a225c7530303631227d" + }, + { + "category": "integral_bounds", + "entrypoint": "canonical_json", + "expected_failure": "negative_zero", + "id": "jcs-negative-zero", + "input_hex": "7b226e223a2d307d" + }, + { + "category": "trailing_bytes", + "entrypoint": "canonical_json", + "expected_failure": "trailing_whitespace", + "id": "jcs-trailing-whitespace", + "input_hex": "7b225c72223a2243617272696167652052657475726e222c2231223a224f6e65222c22c280223a22436f6e74726f6c222c22c3b6223a224c6174696e20536d616c6c204c6574746572204f205769746820446961657265736973222c22e282ac223a224575726f205369676e222c22f09f9880223a22456d6f6a693a204772696e6e696e672046616365222c22efacb3223a22486562726577204c65747465722044616c6574205769746820446167657368227d0a" + }, + { + "category": "unknown_field", + "entrypoint": "configuration_jcs", + "expected_failure": "duplicate_member", + "id": "config-duplicate-member", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "old_v1", + "entrypoint": "configuration_jcs", + "expected_failure": "legacy_field", + "id": "config-old-v1-field", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f6170705f76657273696f6e223a372c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "integral_bounds", + "entrypoint": "configuration_jcs", + "expected_failure": "nonintegral_number", + "id": "config-nonintegral-duration", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a312e352c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "integral_bounds", + "entrypoint": "configuration_jcs", + "expected_failure": "int64_overflow", + "id": "config-client-version-overflow", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2239323233333732303336383534373735383038222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "integral_bounds", + "entrypoint": "configuration_jcs", + "expected_failure": "physical_bound", + "id": "config-zero-duration", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a302c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "raw_key_encoding", + "entrypoint": "configuration_jcs", + "expected_failure": "padded_base64url", + "id": "config-padded-signing-key", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d513d227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "raw_key_encoding", + "entrypoint": "configuration_jcs", + "expected_failure": "wrong_key_length", + "id": "config-short-hpke-key", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d4562556841222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "raw_key_encoding", + "entrypoint": "configuration_jcs", + "expected_failure": "legacy_tink_keyset", + "id": "config-tink-hpke-keyset", + "input_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a227b5c227072696d6172794b657949645c223a312c5c226b65795c223a5b5d7d222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "trailing_bytes", + "entrypoint": "configuration_jcs", + "expected_failure": "noncanonical_json", + "id": "config-leading-whitespace", + "input_hex": "207b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d" + }, + { + "category": "old_v1", + "entrypoint": "signed_configuration", + "expected_failure": "old_v1_framing", + "id": "adccfg-old-signature-length", + "input_hex": "4144434346473031000d000003c90040766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f105" + }, + { + "category": "malformed_length", + "entrypoint": "signed_configuration", + "expected_failure": "zero_key_length", + "id": "adccfg-zero-key-length", + "input_hex": "41444343464730310000000003c9766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f105" + }, + { + "category": "malformed_length", + "entrypoint": "signed_configuration", + "expected_failure": "truncated", + "id": "adccfg-truncated", + "input_hex": "4144434346473031000d000003c9766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f1" + }, + { + "category": "trailing_bytes", + "entrypoint": "signed_configuration", + "expected_failure": "trailing_byte", + "id": "adccfg-trailing-byte", + "input_hex": "4144434346473031000d000003c9766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f10500" + }, + { + "category": "signature_input", + "entrypoint": "signed_configuration", + "expected_failure": "signature_payload_mismatch", + "id": "adccfg-wrong-signature-input", + "input_hex": "4144434346473031000d000003cf766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2257726f6e67207369676e617475726520696e707574222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f105" + }, + { + "category": "signature_input", + "entrypoint": "signed_configuration", + "expected_failure": "tampered_signature", + "id": "adccfg-tampered-signature", + "input_hex": "4144434346473031000d000003c9766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f104" + }, + { + "category": "old_v1", + "entrypoint": "bundle", + "expected_failure": "old_v1_framing", + "id": "bundle-old-zero-header", + "input_hex": "41444345585030310000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + }, + { + "category": "outer_inner_identity", + "entrypoint": "bundle", + "expected_failure": "configuration_digest_mismatch", + "id": "bundle-wrong-configuration-digest", + "input_hex": "414443455850303100000000000040008000000000000099fa2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca63" + }, + { + "category": "body_tampering", + "entrypoint": "bundle", + "expected_failure": "aead_authentication", + "id": "bundle-tampered-tag", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca62" + }, + { + "category": "malformed_length", + "entrypoint": "bundle", + "expected_failure": "truncated", + "id": "bundle-truncated", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca" + }, + { + "category": "malformed_length", + "entrypoint": "bundle", + "expected_failure": "zero_key_length", + "id": "bundle-zero-key-length", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c70000101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca63" + }, + { + "category": "trailing_bytes", + "entrypoint": "bundle", + "expected_failure": "aead_authentication", + "id": "bundle-trailing-byte", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca6300" + }, + { + "category": "outer_inner_identity", + "entrypoint": "bundle", + "expected_failure": "inner_bundle_id", + "id": "bundle-inner-id-mismatch", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1c766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b947f780ca16af2d5f73fcb8b053a3ae1c3dea27994dc1a9cfbddd183d0d75658aeddf4db957813d089ad81abb16074425edf4cc4abeb769031a86e33ad0569d3421d3b2cc2039fd7191025939178a3f566dc86b4c6ec4ae375280370a7cd09a3fa13bcbffe19e6a428cf718c8a819dae6a8bc6faeb24a4168f28b3c1326365ad92231fd86ce0aa478793eea5df7726b164984a090fa099aa6c6964a3206532d0f9ed18bcf781b1f8e341058ac26941c99d6cd9d41cfbcea28fafd6436760c368f7eb33e5efde41a0c6e22e4cfcde60311256dcb52ac48a08e4471449df2b9bdfe4be7ef321376bcfaaf5dce4e98c41d5bfa147e62d23968ef54222afb59d5b9f021bd72c57dad8b97e2fe398e0e58807cdcb5ad2d6cabfcb4e8ad9f44292b855c2ff82f8c63d8fc5d3112c5ff37a4194e198dcb4cf408a8a44dab91f9c908d411df2f34426682f592ff8c72cc42b59161ca7fe6c8ae0557294dc2f87c26c7e8a2772d051d0e6936c32687450621d7949a4a43f84310b0e8761375367a84d63ca487465e651307b919b663a2207f4f04b8f80f544d1e45f4ad51f12dffc56d0dc0e0f8ca2c79170a0b2e1024611f46f14d869f7a4cf98333cbbf1155e446709c93df67cc4bdd7d6a0ef89feb9cd2e57d34acbe9e4bdeddc7f820bb5964cbe8640e062e1b354d766018853f015aa0aa45b41f3df4910f467990f48d8d149f623bf2317f6b49c1599796ac77d1b4a31be75ffe426bdc0f3f11d9373234be13c535451aa844599d52f5aeee313075f07da69897d97c534510cad10180131ebe8b9b354632207a4f8a8cc2704cecd61a27d93696cfbc29f0eadedbdd853b7f3a62a83d6128dbcb581ac0407bdc2e78bc0ba2ac04cf3ebd8a7f2ac4adccabeae94474c9eab21884442332cc70dcb2b3e06b6d423fee125655c38935850c7d64a35a1d17bab82405fff3c7f0b0fbf5ad721e068b7f179253f37395850a85ba7f767d80e23f198e73d5a2b72ea152e00194ab728016fa7e3a4c31f14470e5a61b95d6d9c3204b3886359cc39adf6aa2634c8c94962c73aa8546d1330b5500e150773a36e61d9cdd8ae8e2c539b27d591efb0ab45804c2d3dce2784774e8c62118740975455a5a64723cfd4f08b007c141d8d8b28e27344f558877dbdd9ece7cb5c8433e070a5504ab6760256a4450cb0dc107e5359b213a0cd285ad338b825ee22feb46f5ce03f8629799e93a0d12425138c172c8fad45df53c90c2d391af97bf602b9a50c8e872b00e3df6af1bf8a9e7fa845511c969f67f6245411a55caa0911b6fa5a677332575a145f5269c06fc2159f86999a43033ed1664ccf41c1a301bb29edf4466c8c5d736ae848ba329fdbfec7761b83262bd742086252ac23c49f82967c7ebc5983af5d94b79d46ed3a9f24171f58b949c6641ad3265cdf6df6037cfb4e84c91122b2619306217deb21fd25ba23af9af66f13f581c8a2970d204d69b84798e4ccbc0852b45b24ef9b7a01372c91b0584ce6c980288ee4b6052a7ecdf5e21b29d1f7b6ca263721735197af239ca467bc50dda17df9fad301cb958b34ea551e2b63a7ebfb657f92bc366aa26eb44e3b62dd489463bb9b87baecee49afb8286d1efd4fe256ebdb98976ae587d65884c9c527fa692298bf68c5b5cb6e9e4333bed08cd7a14810c296aafdbc469881d2de6a2aa4e4e692aafa516cc7f1b7a465f0cf6501325d7e3c46d19fdd10fcc23c43abd8e43dfe993ce7ddeb8f5e6c8913f6e3582346085e251fbb2ab620801d3893c717f70102df0f7f48eb8174a2ad9dc0acb50b24a4b85e2cbb54927357ca500c98fa2f4f849fbc55aae90aa4b18fbe0b2acd6366bf91bafac7f92c40d605acc467ba45da0ebff750b06893253ac7ec8126813875d7f329fadb8957de36dadaf925b199ae76a2ac7c78416ec9063e492d74d32e375477e7f3437ad7d9ac531d117cd3a99e3e6419b65764b982c4e7a95a6e409a45a3a2bf456d337ee645c9b515d22ce3d94504ddd7b660b2aafe078eb94820822c8239535a8481d672e1976945706524c33a46bb053b1ab88e2221f2f98c88713706cd9454435bbab574e46cdf74ae6fb9973e832e01327859ae264d355612c82de26133722babc5d4d4325f8f1f806aa0f32709565cbe335d92641388848bb4bca58498d2ebce3200729fe1807b0643959498915a50fb52592319428a11ff74912910cae5add7fc80784c2a4d0c5702f3983828507c16173b276c2025c0fdc97362015677104f80178cc87b08d2f60bd4a0bc01fb46d96f6ff653efad9d1fbbbc20218ad99042d9f59ab08b8d4eb8c01a58a9a2a5aab9656e96b164a3a138e86755a171276ed714476a61947bda517861c941f7597a3c5ad44eaa0784e7e53ea3609b6345b1cc0793ec6de8abf46038ca1c9f708aaeeaa2017575d7bb37def63b76788698ddeb8064959a8188355a53f4b1c28175d9adefbcbae4a303ff65f731f0c915e7fc8662ef0e81434aaf4759cb07d43bc40ab7e6f6dfe507bfd539bb07df380ad642238bab029a477765b5648441b83aa3a279661871b5dac5581f32b1e56421687b772c72bb365bc9fa9cfaa317adf359eb29f6acb9962c6775f28dfa667fa243aee75ec9707a18cbc418ce9379de640787b633ca8176bf871fe7d32ac8fce33574521873a8c9fac6c60d26472b7d50addccebf2ef56f171d6e306d2a32299ee3f3809aedaf230e7bde25c80f845f6e2adaca2883bdb28e190e58d8855b607b3f03b69a018e6eb60aa45c7b2beeb6b02650399245cd695766d32c7b007123f1fd6e6f8ffd017e627abe155ea64f06676e3924641d7fcfbab501281e11f70db8d41aafd0780c62be8a3a52c44fabc66d0c5bab25062ba86031f9c0ac7dbbd0149c243ad78a3c58332e35e9da7e0070f6e33cca6dfb25e566b07b770ea4ce9fff65c26146297862ad878ff4a18db23a85535ee62acbda025624c1fb46ddde8ea8944793405aa7570d163dd53dab79f78af70a8ce50567caa87d7cfd6a7d2e977f51909ad3f488453bfa07bac2e1a64c26b69701c1ee0c8951593200129630740e0f137e4d9451532f49d91e03cafdfe32df26d25eb14a2035ca5301cd21b74238e926cd701e26bd150fd523f61998b5caae182c06d7973da4ff43e6f73ceb19627b160ae1ef7f3385e9425aa0a8783bab07bf3b1109e51d395eb64a8ba956a2b6af02b6c0f3c6c730a1d46169b1b50d866eb364a472287d5d419e6cd67da378a0d8cd9b25af1fc7973132a1fbc85b3a6cbb80a2e4968d660ef53d2d28f5acbc983f8b2ec0a8d99d0c7efd14e2cc6dadf496cb92db824792525d62464ae1939df90652b8c864075c4a1530b45bc6c7869bf35c65afb241110c92775ce3a9c92392c5f99fafdcaf293ea5e44f24b82edae783925b13289de7958c0a2a8353136b1297f7aa7cd74575c200f36673843412c786e5b99c21b7501d1a4485ece82530f54c4857609adde3abf70bb60b22a97dc914b8d85b659267e3283e835300fd6144739510f8e37f1c8fda7bf1c772723a9d2b603b3ebefa13edeaf802735ab07bd6908434a40ca56feb0600400d29205bfe741b2eacbb135fe6ccb45be474b255b74f06b81892cb51825af535954d17cf92518dfef189b1b0aec38ea8e962b8e76aa852499b39396d9618ef96fa6fa5900e728a695cf76f4db002761f211b608c4081dc8c5a5de0f85d231ea01cf640ad218eb10814ea1cb8c49fd3cdcaa8fd1f87e66fd42a71140ead1b91cb0ec4528f8973202ed6acae4654cf08e2d5609467cb7a4d4948e977b2b83cbe2e8ef8056af3dba74f216a2b105956cb08aa86f63607c449f5c729c4bd5535012f31765880460127b685587e1fe27921e4aaa93cc55ee947b1416128a9d5d41c32ac30338971ac452dca738bce904fb52b1c5e849f33ffc748d8515b590f150471c34c16735f73d3eba835adf70d5625c32ae24e3512942cd857c6d3c34dfd7901743444dcffb376779a63374381f290fb92454be38d905b2a0084ae5389a5f49daa66e62423cbcb9f965b08de2b283d26d3376a3bfebd2eb513c0709a80e6ca016cf0daf6916b87f7e26e374ced64f9581010b6e41c12b8e19c9c4f7ea2dc70bf9cc1ebd6e02d97b5e092f5ea1e696ec1701db4787a9b2d58b8861a3042cd15b84b82ed6e0ce9dc2fe27f87abea1d9269b3fb62469472d411f2d20569ab8946502294ba96f271a115bc3287d6bebd97b45aa3ea5f9af3a235d791" + }, + { + "category": "outer_inner_identity", + "entrypoint": "bundle", + "expected_failure": "embedded_configuration", + "id": "bundle-embedded-configuration-member", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1d766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b83be6b196e83aca2d0c05a846800f01341c8bbe0f06c9a80a56110b675488df562a60659acec0a8983166f056bc63ceae2a319b27f5c70bce65bf4bc93f0e1c739ea85201cdd924ef6844eda072df05f2fcab34b03313b20bfe716d5d33d4716881dca9ed3bda0925a31e895f804ac77b83dfcee85846dac9f0dc8611570b68810a451f4b8241d81d07eb5e5261145d0700799385ab01163571c2b98f46c9eb1ca3521b026956663cb29b4daff427562645cd52c1b82b5f93d949a3523e116e5a540a478f4bc22cea75fa259c22a53ff3c28bb4522a34f60b1150d0b4cd6dd51af4755805c1d54c50acb3e79a8e8beaf273d2d924cfef8f2a2da5e12db4acad98e87580096d630203661a233efdd9c5af0927b3171ffbd022dedca10639d7eeadd74e69dd76e9910e59878153aed69db8b998d594bb9c74e4394a6bc8235dd0370b34bb1f7508ac125bd7d5d4d4157a0e8af75098d4b7d052e47bff12d106df15c78214ba3bfcd142b2d0cd4bc17d1844bded96111916e03e1a84006f5fb09e1f47c8d85e8ad23d4faad722f318ddfb6a78dad36f7ded67a6022a04ce91b262adef5222983dc0e853e8dd6ebcc9130df63b7668fcfeb0494a5a40d98138f407bc63a7a816ced5031d1de7655933f34b1b66242f6baae86f32c276ba92e5c0d102077db7d57329ab6f3363f6f3dd4dc1cd44ebeb94836ec0ad9b5c4297cfb60d005b3de87cd181621f255607d6da0a27ccf9eec3fb568983ec5364e525f304d1c4c6e078ca461de367fc318ff27188c897a1822bb975db748bc982939034a1cb41414145b858cceca6486b9c8f58bc7eef43757ecca7861e28017bd7aea4632edd7c77321f7f1111b91358ede5ef7bcbbf47668deac20b9def81226eef4c95cd049c625849d5ba21c8c46fe9b8ad69541943f38be7682fa6a7fed9db9269ec3b111d4d62a4b7a73226e841166eda6f4b4b54ca84191ea7380995e3607b452c26122ec368269448323ccc062296076bb96cb50d3cc688f0093e103a2b0af7bc920441393f1649bd49fe5f83c4bbdbcfd2ddcf8b378b98d97f976fd694feecdcf717f423d0ea0e1b566ba4211f517246b7ac4f2e91ecd869c56bb2c7828c46adbccee58f125eeb72651551bbaaba21a5fd6c0b09e437ab037329f07eceecf5e2933909ba4fd864dbede0590e9f6978519125941be20c677cb6073c74f2d49b19b608719ad5e20263252a57c54744f1aa74befe6dc095bd4628668ad865eda1e6c9806fea5b953eeaf16d86aec4ffa7973337c559d5a3f363de26a9d8bb9b3c6a8820c32d63aac6ea9e75627a6154f5b6e78c1410f7f78b7b0605720e36562748acf4144e6e5f01bbf3b1469861bd6f48bc444ac9725cdc8cf627c2b9f46f9ba8761356a58e8e0fa867420ca6823d623a17b778fb185c44665a01fb25032edc6253c58a014842e38cefdf9ada3a834352ac8961178a0dfab10303d44c74c660768d220a1c70098742cd2d944653d4b1a47d0419703cf9483742959dcbb4bb1594ce48c432c916372d0d9cbf7c611a8475c090be16a2ed8fa7a121b2ace9cedc7ee2245f71328773b6cd4f7e3de414e31371506e081b2139eeffd680fe9ca5945d6af67f8d002d03aa7f1ec568b57023ee847418d208bd3d86b3a68d5b3566a05ecd5752860b56acc8f870f75eac9d5cc16ad568da446c36da80a585f797b6ee372a778176677118be43498e236d150eada6d38747c6d72fc510332e25d406e116a316328561492ed5fed014be60d45703baa79cd329fc865e36a7a5bffe38f0595a8c87f8ec0a54e2951562a0b9817ce81218efdd1913ba7451e77ae1313e844374b1f1c29080cf342e13f5be02f694d0f9c78ee25b0a924368d73ca2ef8360e5393fb1f211f73c2981207d23aa9460e16a6043bea6a2362f0aa4843de3c6a5e0063134fc975b2716d97c8a4824b0303f052c3b26e499d7aed0cd8e145516a2020c1228884365226004fd65887f2e92c42f60b79b73e88227d4994d555c1068fe5e1b2f34e0566d0a84f58f3262219755e657d283a22b546aff12f54a10760186fe61b297ef7c09bfc79c2d513f848b759793cb4fd51273e01efcdb55ad7dbe36d911eaf2b15ef2a4c2c246271d3a14f0ccb42a073019c272c99d69b47af957549c82937cc1fb887602e54f6c65c473270ec602ffb828b74c42cd838ff1e1b433da5045a742d8535d5cb2117d3ab675b4345abad5f952d3d1928d3c83c798d4483102ade7f22438df5daaa8f998bb605d5e1064dab130031dbc924426e04d3bdf1aea9345fd5e3fac6f71df4111ee61d7f3635316d3a2023d3428e7441598d5baa63583a7eadaf219b5ef0baf1e67d2adef6dbb72338e61dbf746117ad0075c838b99e55d30faeeef4736667f413187a6846093311e7796d5e12dad7b52fa9f3ab2c7dbfb626555b7ee1dc09bb89705b8b52da06738c84bf4cf747dc832ec223f09dc825a1ba87e362ca370f1eec76136436e61b07cf6541a4c4b0e5fc98633474d67e075f487b4f7a27d313fc28481bbc928e8e3f37972d4a26aa7769384a0865274ccaf08d973ed8aff56ad470d5bde5aa43fd0952c85588a16660736e9d5a66af9ef9ea8b0f0e7c5990b3f3d1d93022499171d91642ee8196856b50405c7f3902839c373632be1a305ebd6b3b6dc439299508e034c9c0340b56b562a59dd9949f396fff0756b9271c787df101bb1f2409d9eb4cad8736e24777622f2d2eb70e62bc94ecc2b78d43cbe37541ada90f464f4e954c8c126264926e07240a2084c073ccfc7077ed9e65c8e6db9e003593ee2499cc458b2817642123e5e1271b1743faa02275c7e1656d9e9c89dd55e20472b363b13ef337d371d695e984249faf07934b959ce6257c9a7fc56df0e55fa02d7d782188911ec6914efda56cac32e033de0f184f6a5c6072f39745b2850d596210807e55b3dd1792d960f6fff08caa0e0575ec7de16d5007b8a81672278be25bd9c70cc57f4cdb65b6d3630c6859ed983de895c93d7e732310b70f022ff38bd1d59b3521ba55da614b267161cab414d28ef55122807c001e8c44f995ea8e00e02fc1f2f0b52cb6f4b6f43c9667fe348cd8ee1d27be7791f50154240ea67249ea1f8bc5af911289b75a3c17b6d44a685cb2ca53977829addb1e27d03eebd50945be1733cb50e875ef53ab9e07148ce57586dc796366ec14636733e98111a2b0d438a8c23eb4a997416296cec4da955544ae432a85fcc44e6bc3d4f2541af7453d1ad7d5ed5fe5662729dc2cc630f5cc122391c57bbf5bded0c8fff15fe9fc23bf12c7fcc6df7b6ba29bf961895db59ca67f0537637f040a4436e9b56c9c00aeb9fb266fa1febe8d3e55dc7697558bba98a275754eb750f8f1f883b60a5894681f60c6b6dee73b984a703e2f41f49011419cde985e2e733367b5fd8b50b15eccf4a4ea19b335d99d3eee7fc0fe01b361dc514334b3d65d44a062fa92fb019b5bfa076877810f20f7107ee0d6747a64c31d5cb66d9c36b5f785f83eda4b5cdd5ae32188254cd40d062a057efb990b46fa1c8cb34c9c1046d42c05eccd2f01c3cbb26239e2cb97bbb97928714e76e3d946c8ea66588f1045cde523e0ca41f77a68a690516ac30d09361db7e571ec3a7c96252c5a726f0e830218778adf89acf00372b56d37b739e641f5d0fc0cf5594d66c85e93eb47dd3770cab21e05d6b89b86f0f15f391d16ed87030e13c213a34869a61eb0758bccb679d0815887d341effce9c91a5d23af69cea0d438dbb2a5e1d6daf4a3a059a10b35139a6233d9aac71521ddeba23891afb0ed6cc9649fe9f0a4bcd8f37db5b39c3e1f6c2ba347f791ac2eb4401014f2eb78826d8b960a1cd39686ac8f3de9d60a4da34bb8eb249ef463dbfccbd8496c23851aee9b070bf54ad5809de70e04d6c059eb36f7e28aa60e56883e6c852ef5d605f56af4ac85cc4259f37421339c6b5042a31176cb33dab9846bb51bbb5f23b7af1944e5d76845a787f121cd6e1f0354b1a79120bab5db1349166b99e1776ba7b6bdfc11974ab153123aba67a2bd1c99dc2d02d0751b738fbad99e5eaaaffa8dec431abafcc32fd7f2513090dc6674a6ff1d4dc02ad198a0cd0392f8044a72bd626defca1bf44698eecd20b49adce4b4365810717886bf70e2e755aaa6b5e249e05a2ec461da38fee4e789d472fb06c07c39293b3d48f6cc0e4fe74b09fdbdd9a742254fe10fee205a05f55a3cb8fd2f0bcc20efe9010004f9ace95575fa622792b4957cbd277ead67" + }, + { + "category": "outer_inner_identity", + "entrypoint": "bundle", + "expected_failure": "inner_configuration_digest", + "id": "bundle-inner-configuration-digest", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1e766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bb1a9c24c9e3978edec3b0de59ff2508ba52a8ffaff65fd7fe81db186e333af5ffe5c7bbd2ec5c031a539ec3e26e76c3bce7e04f3be37c33d5ab190801f683ee3d8a93686dcc3700e8b12619d19f815df83932ffb89c1ee740e3a793512d82ed65cba7071ec753c8e902b9ba27bcdcbe9510adb6a9e5c88fb35c993f5dc0da75b957e13d4274e63d0ae69b7b548bdffe1a0ba175266afbc28d0e879032a78dd8bfdb94c739141f0c68f0ad1bf13a868b4d660712988b37d28035287df065a43436e65a22b8acfb90588ee8e02ae795a9a7a942b9aaa27fee9f3114be66e1a26bc7e3b7f96bec954e4557826632fb206463e994e55308cabc38ce222a2a7356da440f65cd98ccdf72ca8ba9c601e14881973b93ae5e34a7e8ce669685d46a777e3b51cd83f679d4766f037e96f33cccb36f70f6400a9e3e0d6287afe35dfac600ee98703492f42faa80e1762a4e48b3dea8fad491cd2fa0a3b96ce7095259cb303bd98649278ddc0787465399ec16c83fcfd7ac0c56196c9b44bd0bf55799a5033200d65d73d853797e201bf293add5b9c604774c81e0f070325b5bc8056e4d1b2f9d1140f26aeb4a268a077201db50e529f220c00a35080b6ecea81ba6f1ce84aacbb18505dc0a70664c4251017ab5364aba6abb7e614bd1929d8ef1b57a4da880f1d615ad3b521e77af7a78b8c36df518c3c3c56eae306929ac94a5c62e2958c44d73b1dcc0caf464af8efa14cda0c33e61c0d1f7f290bd734a6bd8f6bce23bfac730130d84582271befd9258787795f66e0a1e34837bf9cd63790cd29d025d5db2ba4b85c156baf2ecd6ceaab2e8dafc5be680d6b9fe083be0d07a31acbe22aa93a65d4c8245a15ac7e62d6d9599d9482de458f319f57aa23fb6d4b5ebd0230da84a4458138f5b296ee6a6fe868df0031379e9d579b0a0f2feb0f17ba757fca66e0824c9215454ff3e45aeb03879db899be221e521d1d56c6a3d760397d3a6f53ed76e4dd5693dd4367d1015b420fbe0076d2d21f4addfa449a10286497430d1a5a6cd9ca4ca51c6504e00cbdcc2870da6ce4e5c16e2fa688fe95218f45f620a1cee3e12bb1a10a777ab06167d0ae1c047ab76f8dfddc62c5bf2de730287d0218ddedb3f76edab0ff4327f7e3c6ff85c73867b29011e75a167402bc6fde64d6c8cb0cae0feb598edf082d868dc07e7a7d54e1d81c9bc05de33e69e6dc9a212b38681430d79733669115cf7ffe389c2926a4031c49bddce6526c125663f1443daeee51377e5f50c890f886eec3deb8dd36b95e080d1f5158ee03d35d41092e79dbf63ff31e6312e80e11d83f17d3f49f1ffe74de6bf3ec4b4b8922108240ec3249f9282a9bb7d208c292fe5a0aef6a10d7c6ef2bcd7ae14110a51898998022a7472a5ededbdbd720765d9b76b15a0549c1910ceab80b124bc5623e00337de0ecbe50453474b9c31c66574d1f50ad89a37bc48e6d3afd59844f306dc54a69a5be1b13f8cbc0b9732d730462637e2d6f876cfa6a116e0c2d387c1565a0f027dc76f5db582c05e3b230a6a7d64478ed1223da2136ff026030d2298c7499c10db97c5ec33dcacaa5aad6480d99dfe0c3569d21fddf992726ca036130c4715f9b051e612b99135613b22f756c2067b8acb2bb67f6288944babfdb2672ebebcbaa960a05a880a42a64917934b2ad2cbaa31d72e6c7584619900eec05eae42ca68982c309c74d498800621ad4081ffb94c8e555f959d48e4913c643223e656f41480533b5f56ed038e346cf06fefc121ccc7d3c4e0d5288374182ec28866df18bf4cf29fdf4840c06062afeb8587f204ec23a41bf2fb5b607b664ac80c9b374a4a476e288bd30d6dbbe46f4180301e91b691045f159289c4632d834166596edd4010a74fe4032fcc7e665ec1c1a21017dfaa247993dd47a97532278c9ac98bef4635915e196ba87c9f11eff77f80e45d6aef5196dc7fb9e835de43d7b5508239ea7c166fcfc61b66d66433f10696aae7b82287a5eb35fb1ef1cdcbb2bdcc1cb052a2ebed165325b874d7c6620afc33730b40ac34decb6242818bec445e93ed31c2ffc2aed904ea52b6ae16406e74cff51fc65eb7c680b82c50484c44c794e9bf6a6582babed8622d27e29e8beeb6d6be3e49f9293781b05f84d760d1c65da7b880785110c4af0004a4c2e436bd6affe8381695e48e88447ac4a43a78aa10168eb52ba63e646937634e082c31cfb59b2f1fb02c67b3bd4ebba9fd340d361f9b13bf57fca756ae61d269a9be287e73dfa1271cd19f1b8d0262fa3612beb655ca26f94b57c2be0d8c9d8f16997cd2b0f7802de77fd76d0850715810d3d10ff29c7f718f2c5158c947bf7f182c87d07c4eb5ffc280435fa38fcde806cb25414351ea829eee08ff2560c57ece4badc5e275f36696ec6feef41960bcf4f7cfd34234c92ca5ae74b8e82603263836d832cb081383261a11429f599848c760339d2917c5ba326a1e53682a92d6e8037e1c23af56dcabf899438f91278fbc62a1dd624d04791c2d5f3bf34c77ee80ca8714c9fbacb7ded92deb4eec15173368432515594658aaec3de118e831f1329a41a8786d36ab526f1d34253eee28879bf066e527fc331adf63cf284c1411359dc4e3cbd9cf9af1f2b51bb9a9efa97b30fd5fe1f0f5244c1715e24fc18554d608e11864702cf27d7c4541f9d39b4f62d1c85afc3517b90be3eacf7c1cd6711be9fbae5bc4de8d781062f18065fd8bb3e4b14c66d3b50e4d15681e765226f44f99a79995d0ac5b68510dc0219daa92474ac732e67210b5b387301977d7126a9aed9be7103c4ca35221f24220500eecefc92e22ba26e0f56ff5eed8b3677d6db58991af78a7734e292e21a09d50ed58d5d420b46e16f4c117030fd2acbb9d1f0fa53ec100b9a29f0773f56ec9a35b895e3e2ca1ab892efcc5f5d05e14827eb349e9fdc0dc4c9dc1ae39d9ed9693c449ff2b58ead8b662cc3153faf0f8c4e582ffb85e5057fe26a71c4e9d4d6ea3996f5aec891202bc640902077517986eb612fe614e0c2e1daf63f2ac24fd286695308c621d049b492db54908a3d22adf6b40ae5e342debb19d330fb52a03afdcda073eed1b983bfced32fce8e9d2748ddbcd2c472be1d8afe9e92eb3228b06e60a8d536e5b4e74871a790a11db99d1dd1901c2823ab2c44f0f74cf2033679f53a5016b1eb4bb27a2a1030004bf2a426921e75b27df868d30840eef475e65f1dc0830c952f00353fd4ceda887fe3a2932b30df7be23c11e83fb6f7cf2b9766966e19f4ef1625351610283d6f4ae9ed252957f7632ba6d653475ef3dff4e1dd1f232dd99f88d3a190ebedbe28d9efa90e20c2ea0fc1b4259da2c0eddae73d6f3f37664399e73b91b9356828c3b27010f8e27158327ca215243a3fface151ef51e4853d6e50be5c56dd70b72198d968fc54ffe76688406554b30ec32c45e52334e03e976feb70036fb1e049fe226bd1c0c0728fd6f4c53b9b9e4a260a2a565a61dd795d39bd1e413df9ba6a7be56f3dd244deb7f7c8093e585b715f669ac88ed28efb2d60b219a00c42e07109638dcf5bf9842fd55043fb8d0a8af41200aee34073f6fe130c1a258caa21c99b5e9dbbade6bb697d10bbddbef2cf8e1409f540aae9ab33b3fc340974a5bd7b0b6bd3d447b5bee133e5f9d0672fa984b288f34d7030d854fc568a636513807bb92c6a2cd8ea704170905633ee8f9f9ad65adb38ad4fff358a191424331b271475a1a5ec875a96dcd27357a6aa7f685896ed47e45fcb3ec321cc773dc2497f39f406e5d378079ad976ea23380b49cef1f3c35c37d31bcde8468fd48746e3097130f07cda9339147867acd1f41860e2451beb8a59e23c0ad070ccad6be04a3a863baa04078be40d968f21b74d5c40e0560975809a9e88c4d912d7dcf5287664d714308b17127277348fe3f0866610bb302310a79a1ebfc735654faa311282e3eb0ee7b79e22c9a7271c6f789622fc0b6ea29af622b7f0f8d92fc211412ae21b53947fee2f5300ab811690ffb49abe86031f377865bc51553fde9614cc7c2954db3ce9cad98b5a70eea8866e0ff55bd2918420214a0270c0646d6401016716d9587d1dc3b914377e22a3ad8d1497ca643582c2084af0f2f3f43a1d7ba55e991d9fb923c8eff03a7330944de0dfab3ba96804a19311db34e4bb4d5adb95c00b77c8102ee8f7833ef7fcb4295450738fb2bdb498d1c766a7f08d18aa1d04c88c989120899a2b3c5d9cf6347ec36bc42d2" + }, + { + "category": "signature_input", + "entrypoint": "bundle", + "expected_failure": "embedded_signature", + "id": "bundle-invalid-embedded-signature", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1f766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b1a0c97be426c97f63ffc2adcf8f881e1edc102c7e5c4fb470191dc898081e2ece826bb2d6271b28c6c966bfabb09d47826f8e92d4da19cc4ef659182f69631604d216a3a795c1d1e7fce14414c4e57df1259aab00423f9091ad353d6f7e33024bab797fc7fff9cde3a6ff1ea53f3bc18fa27b1bd808ab123c1c0d440f41f405c8db42b2781e0b725146c90c6b3ffd8afd6d5403ff5388d3a3504b7906595b7c943ec5fd8239bc20f04ca20ad6e5c223da64d24763105c780336d02d671978659c010560d618b56756ed7949ebf85fc5ceedf93521c0b70cf015d264d3d1b617062cedc1223908c37203d5f12b2a2e44f4e99ba15bc9ba94abf8f1a20045ae445cc9d9f2b6765481c727f9c792fc908b0c6c97bb81acdaa7a4366d7fc30adbab3a8e696ab28492963b5046e2c8b21d7c486b7c1c3fa0385297aad4ac20faf5bca56b369e4cdc5279b1f34ac4037a700e4bab265375a01b9e3c6390d8177ef77c5c21842a6c7845a9fb8c34f4aab7a04120f5fa79ca207cca1af8d3ac8bd81c41702778f108a24d15f6fb411bb64e6d54924a519c665296d277dfc51bb5604d9c89404b89823310ad11453aba37236bd4336b801f6cd31ca90c7368012c4395747ae32192e5ea0ae8db8096af17c36373b06d83dbec4650a958ed6b49528d0e6eaede202471547da23f7f2ad29767e7cf2e4fc52ea25548e6ea9375339bd832d80b99fc5dec486dbc33cd12b8483795a1b0413959bf7a0a69ac1f3fc46c7bf86e947459a080226537798c4473bb85c8eb2d4c2aa50318c68165a1f79993e9ec381abd33c9c7bcc2543ccfd6c56db15d40167912c679f5e6abf496dac67b1f7568ec2553be95f872dcf609d5d14a5859829c9213fb6ce4aa6d3bfe8bcc6e16b12bf5f7c383a8965e9d055f5716d297fe2177edb226cc9076e95ccc9c436961fb0dc1f1a649ce04400e32a74c5a4b65f55a302bfc96a057fb9a10a41425916967a863155ac8e3b98b5feefccfe0691968ac8e08b723fb279e4755cd3fcbc188df1d2038084f3054ccd7f8e303ddce3368a3d93e1d282f37eebf63b919aa52458ea4a83d82895532a4b99f14bf442e88405f95dfa39ec0c9c2ef876cafdc17f60a26c805e214ec26a81c0357613e04de6745f810d5dd06fe13728854833d31db65b5b10be53da251e8b70e377f5e1e427f12e00e1cf207de971874ef6d21965a7c299e10fbc15e052f87b0d9c5db0cdf657edc07232a583d91f3529039ba43bbd36ed69ab805c160108071a1ad457e906da04054ad1147af16e6bcd77868669ad8641ee2b7b483dda8d3b1618189a9bae4c9493fdc30f36f070d6657aa7f8b22d0b5c362025a5d01fb61e37e883bd452cc022b5fd5bf10ed73b8eb21e448319e101a3396bf34bd2f1983a446b0e2e14bede5bafaca0cbb63be4967d6f012035837552c1f04f66e5e64e795ee1249b85a0e7a5b6ecdb322690da05065655d895110a6424f492d72fe2fa40d7f76b529006005f970d8411712aa428645b0faa2425bff06cfdfc2821e8f27bb44bad0e86c879ac0b8b39835e6b82a39ed43896c561d2351b7632eaeb5aac9dff25c6ef7800feef3778da7b8386645e0b09de222f24cad77477cf9db5995f192551fcce21a4b01732fc4928a935e11aca78a3211bd5bf62c9d658523b1086e20405e8666aeea8182d9060b605ffc36c6d0133d49dab228ffd7a661a630c9bf133ec0024e33acdac4e8c0908ed4ca683aa133d658968f4006e8b02fb74721ab50964a710d34caa978533c3231d91d0306340d96761a92b4054cf124e213bc434ca06c7d88d3bb84dbbedc84ac1563497f0091a1deb7a49a9ce95f1ae7d10873b4338233ba93d176dd6dd5adc1a483bdb2fd8ddc133a92fb9181575cabc14843f32e3c1f7b0f7cbe52bf6d44be775c92dd2b5d135def99fc5f1aab57d50317d7144948a66a115f788d45699220ada1e409db45f2ff0aeb78e08739d5b964505bbc8040ee454bc118d8afa08404d7571912b3ecc93e8f51981895d03920b152a21780ce7d88a899794e9f3715c8fea88aa083c2b0b7a5f9cbd7b37883fa0c1d6237dc9674c8716c634df473ebbcba7ad071793de46409cac83519e2fb6286fe48f1180ce02f5b6c98b172292e97d0e457775a17d757db7c6ce3f1334cbe9d3292e42abeafbb8fbe90451038f94d8e74ddee74800c7210a1a5f5617afbd78a30fce682ec3f8db3c73e64e0efcb7a50b6588a088bc6f88763b91f9852efef959876894dab362d345c87c8fb9a220fa91ee439bbd765b69beb13e79e17174bce694742e8e9fff06095613bd5af1d19e218674fcfb5572d9daf3a554e1c6e1d8bf5d7e5cbd776c2dc1da9cb772d92026b6cc0d2ab3c71fdac6c1c8cdb83833b93da346d57430676996f81f2abae44af4faf0eb31770c7ca8106f51779e05e3f4d33e2ff52bb5d98867df547cf051454c50ab5c0f71f3418b13050f03cebe5cc96450b6c7fc0c42640013a75d7a1a7522aab861add4a3daf2742ee3431dbbf6aa76d757cfcbefcadef8ff663ec4bc65b356b62ef0a13388fb275d5b2c002a0ccfc9c3556d02eb2e3838b955dca415cab1a869580e5a89f449acb5e0ee70aa813baca88dd6f0ca060a6a8b6e5e75bae3317c19d751f27232afeb47331fcddffb89a36141744df74aca40537c7568112da012e66957d9d7cbb1313ea4b76516e27d9258a05a292c09a2a813895656befcc9a4ea30f5e43e0e2e44ecc60e56e4b5819ed2f9176a56f12eb3f0573a532521e0bce71b9c5a42109064b52a9eb5c8a133cdf2431e4fe175aa6df96d2adfefda311397cb05217e06c4d4304ae47b5e8bc0911baf77eec2114b3e6e5ce9025c516c7984ece0dc0c4ae280ad897c8daf0b12b00d4142ceb721e8a8a6ef36c637a31ca613495ec4e50e7e70989be015e19df9abc012267eb2b0e4dacdc25d21957f3e90bba32dffd57e50b06d498409809bb40a57d7ba4353939bbefea336c83ccf888960efc398fe381e9536c3c0c60827fe2dc16ed1f5a891957083e0a46a38ab5897a6daa0bfa8fa4398bba36dcb3398bc5177ffc918f80af727cbaea359c8ad39edc2ba87f1f67dd36495ab9226f706e834677972d6d154612412323f01e3cf028ed966a41dc04384dce514990518b503cd4e0276d311887228d8618c35609192b5e7fd4b8ed02ec0b9f2e4b9836e8c88de75e8903a00a72e40a8f2e6cce99550a382941b9edea375382414652eb1efdd2f4db9f13bc51806472e67ce17f34c18fceee24cfc5b885cfda69feea5918cc6617c1cabafc5f9e82a3186f9830f78c55ba4573837f36602e57660ddfa864ee575d8a591cc3b971fca1b3a41242df3c0abf149d8850231e90ab5785041f7045985dbaea3c6fe5c3df14116c1bf64c78c9ed7eb6f7302f49b5bf26533cb0223f761832040410fd2f8b31a2fa2296892872e8355eb0e8474029e7ca08485cd79c1c523a10f9ef173e2d7df3ca9cff8975b87a4cb64407b566fcd74a05333e0cb9732d9380a75d8a78968e62eb615273054513f9c93ab7dc7cd9784a74744c4bf1b49b2809864abc89436a81e43e3b66c62f07f54e86c32074ff5f3b21424a315e8ad53ca482f0516199391015808ec234f2e53bc43da19212475d61adad8936ab22c5326e60e25ac316b4b7b5bdd41bcd3c20f7688c6719e5547c235fd7628b6208bb47656b24cc429aa4465cc5e1df86d8059b23539621a2a19f4cc9fd6da8fa9460c899e50a7c0f40e37b09a77d5b0833a7a5e6737f869adab50cd0ddba01b5892fe8e5e00e9f17c90cb8e0fc49056f9ede291adb986189c1519f0827451b583aa0143b3a2f19e0390d11329453ed59fc662ab4a032a674838b954ac9690fce82be2e8a1ee000370d75e139ab0fbc15282fa3cfb943509885aca7973e1addcdd33e6b0bc21f2b580fdc4ae798b4ebb74c11903f57c28e6045fbeb43ea1fecca26bd1f10df625ca896a980cc6a9ff67a475a6892bcd7d4fe4d0f8f42778899e4e726afdee4af503628973c234f812b5b181fa9f388dd032521c104839dddd2a0d59a0af9f9ce576b52e75ce02df9284fdab305ab6b991f794a47477fac11f181a597d911de22335ff73dc438837917d63e1ff74a2c63857f1bb38525d62d5ae89ab0ab64f957daa7ae63a3781da592a482a0b1078b1bd0a8309a75e994fe59dfdc6e2d63dd1ff00bc6ddbbaa87932cbca4f683f4840d650a371929dca7c469b39c9f80fd93cdb4d" + }, + { + "category": "integral_bounds", + "entrypoint": "bundle", + "expected_failure": "minimum_client_version", + "id": "bundle-old-producer", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a20766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b55b981e46e06c8fc17b9572e752decc01327cd4452b05d3cbc3c129b97cbc05d80c0cb4e2ea854ed43f6d99abb7aabb9e5029fba6c12ed657af80ba0edc3bce371c0c077c0b8274622b73842743abe53c8e5a644ace240e546cf512e68115559b1f05f6b4ce19c0a9e54100fb3720ed175e80f2926df454d9bd9a61d8aae9dde28a80ad8fc2a62496bfd367672c851b8a480dc067c9c767a85a7e24b97d9fcfba5f938739d4e63d6f8e48b457d312b5f61ba46fd068fd9a67483bb4f7f73c5a2138687a34f19fa8329bcb1a54463f4e2f0d46a18300d49a262e66a6d36f3b5d29c4b290aaf64d1e9c487291b48b08d3497548d33ac887a034b1e7aaec7d567e576329607546eb4497f4af012d42af8a2033f3f8f769f40ef58090efd53c81b3aa56e4e4650dbc747f0d63c8b9c01c00f21b4d2581058af226bb7f4808110b18946333685b5406070188ee3e6a15f5a09c6e90cbe456b8823dedec13a33ff96b1c208d43845ed324d0081044e4a3138f92aa96dc6de7c4442970d5ab6f2a36cbda90e62db20e91ebac3ef682d2d797037d1706fa42d6bcb2b527f718f159cf7061350382b4d9e2e73ef8e6da312916f30f75911f3872780971ed56f548e17fad8c17dbb0449f36909bd150bc90835f5290eba9d2935b234581c2764be57f81785eb8bab6152cdca074ee16bf7d695e21e27448c2d84d62bf96fa51e35e8ff95847c801e9f374303ed0be2aca51beaf3e333d2c51428af2556fa16acecb045133873c1c21e46e4d8bd11034e900b045a00a5f9cafce2fd27ed92899c31f723c9c50fcf0afd7a3ec0a01916e52908c1cb3e9a9757e89458815ff6b86b98ba7d4cff22c38eae9b720b6c02b29c26544a68727c0b7eaa561070731166a8aa1b138a32bcb12bb54071700e9f074e0cc8c0bc8b6dfe659ccf4cad6d970dd20b563a0f1e89e4c9670857bd3b7fcdd8f8bfb0905ac72bc89ce09fca52d5e7be23772383774d12fa8184911ef6fc746f004493080a54894f3b054df3a8eeb71a39d3448e1436366717b60cb022f89b71dd78cbf1c3b8903c4a3295830f1ffd99d2a68ea9273ede80b699c33cd23e025885283add1f79b004c81daa6b7efb4d6228ad405e00b5b0e2d1bad9aafee1320e907ba5e07bbdfed1f8ad800f65f56ef3ca13c03f4a47f5ba41db686504e03114d52a3c63d418d2f423b6d8e617a6fcbced4042b76a5834adf9e96cc009cb18ea3d157332958f32c15c20396b038b23b5dcac8135f0f0fb0ba82ea6283da098f5ae80c0a12002328ebbd24e0b82bad1d8329c692e51a86cad7570a24eb34b014fe35bfccedddb3d83938a203a00dec70bbb63142ce0adc4f5ce19dc08c388ca95031b4db28331f41c78a37092d8f8001c2e32ef63a5d1bc52cdd01a66f3458a36b3371a783e72f4516d93708d0637f0fca4f9fcee28242bf85b76781f60a1445fe8743370b3f3891d63c72fd6b0b5d58321f84d9227242d1a4aeab9d27856e2d1654d136b424ea781da5952e47ade11669241d938b01cea6923b87335d97690df0499d6b55a52f6c2f5baef497904da7e868d574b4c139944ddb7ac9aaadd5db01cc903c32027ba77eed056f6473fa004ebe5fbfb0be97dcda625b75ffc6230e41713a99e9a342bd6b27a7115f950afacf33bba2e43c8d75940c2bfaff6dbe4a0f7e43a19221027d182c4df2613ba3935063557ade1fd22433219ece866598860dd06cf91bf2cf1c007c4e305d1c58c8728106e1491293b95ccafbeae9fd05c315312549caad07e8bce33fea0c4c3fee4ed8d101bbb67f71266048b6925366dc41b3f7631d5edeb2a283d0b3035f3eadb879bc4c523ba2cce69f71020f88afa394459fc8bc93f89f8d646a97161dde6511e277ebcc7ce982c7ed758f79c2f3386a5d0343aceee1c0fc1b872026b91e00b776f0cd418dabe501115f596b2cfbc627304c717beefbd6b22d7d75048a02b03a5d03a0ec115b72b0c0a4138653196366a3655dc2f97eb1ad05257460a34e9f90af031f1ede99d1aaf7cad79a1f0a0dd7c00fbad9d5f0507f56d09bcb8342350f2ed11db9a2d76061b74ec6db31bba5f5ab86069df2cb35ed422fc4b4f34cfa5f30a02cf13ea098ddba3e7d0bea0ccf67dd594bdaf2394341f5be23300d31aabf5969852d8d844a4c809b22d4cc5a9d0511681be50fbe3f5066f80a3f10a00dd84697d81ca8ab9f8ac6007824ff24c259e7236b5cff2e6aabf94cc20b66e9ea349ecaa175e4a0ad1756853a3d6dddf8a1bf17ff1ea29cbcf57beedc03b7a8f1bf0cbada11b6ae0c6ccf7669da3a2174aad45ae3a1619f505eb949d231a95f6bb5469555af1de6a5eb2023d2361a5533001d5e0afb75e2f4c8c90ea6266b91baaa8c24fe8a69576b3bb71a2ccbd61f640182eb2447cce3d777c7bc9cf42783f6b8e08107d5b370aed26a241ba0667515f4615be2515363184d718ad4ae93c1f1f5f3dee5bb380cfd5e81ee74f07824a1e79831c31a0a6d7b70dbfb4511ff703399adb389a031104a694e22e29c423f3d0ddc30a1964e954034eb0f742f1dc839bebc05efa7b6b10f4ed5d1693e870f0733fd1c0641a05b074b595b08d748fdb83a58a806eddac4a8bda416b7e1360778a381d13c0d059daa7d6da1c873313f91e693385d928c2e46505ed943d4f57a049b82709075c782916ef75825096c4d6e9289f561e65f5b1cbe85ce6ac4da914df8040c095ed0deacac6f1236b42b574595fba36adf6c96ecb10d1a42be6dec80864fa8d57a3ab0c2381672afc38ef14627fe619ece92745c23e1375799ed09707d15e693d4ecc71c6efe92f3c660a62db196d418aba8eb74ef4413c944281fe22b9a75452c4be449232a2e227519716fffa69304748daedd9a61084b38866e0f7dbc64058506c7d80e947ae25e280fa8812d8c9705e300c5f9c363d936c4065c7cf41552952e4d02ca62cbd7bd426c170861801558ec78eaaae9e6c3bad2bbd64d182d5d809efc3142ee996e35f6fe500cc5ab3d9b487935d86e75465a68f00fb68e416d3d0b1860a1519539f218117e2a99c19104ca454b6c7a95970202681c3127767d1da054719d94b896be48ef4517e866a75d660564c5eb25b5cb5be8ebe489d0b248acdc5913e41f677e8c4e68933d86c0c279f9a023747132c237730434dda2e0a5cbe6ab0ccd87df2967fd5010ec68b780a4f90da9a8d0e8d04410362803b05cedc830a953bd4c3279ad116f758726edefe083b356fb4827556661eac3a0436343def00ca3384aba9e6af9d5f3bf22dad63d01e2b428774c09ec4598872a409469cd02bf13fd5e2c6fcc1b367db454ee55c176032820f96f879f1040c44829e5d0e4d49f92f18a9544f43e23a136748cecff491ff18da0be2f998175a7ca431768ec4ed4d765ace7c755dc59335d5125133f880e168747dac7e5529ddb36e95277eaef27230819eab95b17f00157abf745c81c5ebfa9cdfa4379dc2d96e2f4b5111ec24ab23f8785ac2371e351de462da44bb27c750c5c6e67eaf40fae27c960ef810f7a918da06239f17f6cfcfaa445fcba527b8ed17e68a4b680a3bcba59c5a14621681d96110510bc0c3fa62e40a4870cfb1735fb39bdfcfb51d34c7f9f2ee74a49ef8a3e8e1e7b4c5318161e4e4f5f75f5e3d7fceca5f0b5aa492eddf46a779b2414f239257ba67e955bbf6a6f05159162ea57c0307ae8c8f344aabac238ff35dfc1b0f96d15d3597da3c2e7bf3453b2c26d4383295cada7e0bac00831a2bec0dba3eb97aca0ce6aef115e391ace22cc6074af0040fb963a48c36b68e107e88328ad824761a6128d2828b318cf3ea83dc919d07c6950fd045f3b44377ed16aba1d62f7fcc22a99e878c0e9522b7dd66f7528d00634150ce83dbb5073a52885e1d47cda820571c8a0204948f4acef220d2f3a703293d19ad1ebed060d21897e9aea5f16f2a30781dcef8f72401f95664d7d5c0a175cd64c1ba9567e4a685eeee3dcb42eab7f6c81648fc976a472b0b16819e5221ab762369007986ff2dde11a9fdc8b80fd9c3405570c50f6723e8967bd630cad360bdf207c8532dee18dac1787b8b33c92583bd433572140f3524246dfa58c8a2bd4ab98787938291012eab1d080aa55fda46b06aa56896375c865734567ba9f245a4eddae7318092ab560ada8693b4a59e0bf3c3ef63d97f278064eaa407b566474017695ba9806cdd1eb8af037c1d4202b8b35beb002fd50080311d4334b60e41153c3fd08c133c" + }, + { + "category": "unknown_field", + "entrypoint": "bundle", + "expected_failure": "unknown_root_member", + "id": "bundle-unknown-root-member", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a21766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b2e8827676f43e9e505993293c25744defb620cda44f1f5218cf1ac80e2fe27d7abd6afc884096126c0f83c3f4a642188f99e259292b75655f69def04ec122036d9306e26b9cefe43c8f76b379671ff7c6efee49c077c989c17c4296dc188f70e71d792edd5d906129cf6fb712e1cd3c4f26a4889e66c95b4bb6c4c309b340a56c1d77978df1a7c8b07d37a382d1bd367e634ec73f5acb0090b125cbad0a729b36446efa3f307ecfda6c666342128c99d0b2388a3caa80998bed7bfe1b68488fb832bc3e6ab9ff65159566a9ad71f0435c3c40655f3397fb3e5b4281131afd4586b3c6be9dac2780c1b6e5ec1864face95d1a767385b581135b71d04bb0d494acc7911085865c8802bce02d0719bc460d8011d0d1a6420363e11396603ed6fbb045250c0e8e3900c7d70a07dbf870aef0c249b535a019d07b54bfc4c4d6b3c57ca6ecb6026e1eeaf768f515b01114783d3eed469192644e2758f7d9d23c2b1558f4e1a6200e2694d799eb5eebd14dde9ddb3d37a51cdc312c7c37550dfd57a4a9d8bc1ec00512588cea9ff774b557766a119ace35a75ed76eb27ddb2b911caf8dccb73ff525709a0d740379f963319f2063dbd5d47a7f417d969cd37a6722c979b412f9f266eae36e5087bd83eabb8c3c7562edf80739faddb71c53124fd5ab67f475ac8a825b707ab93022320f924bd06a4975714962fa403b659ecb1b27d576e14d6fca7012a87a571d0a8c18637bba1efd17ec3fa4bba26fdcaa68b691e8376698e6febfd3b949c4c7c967ad1462339254b7c76832676b433cecc591bc00a45a5b9f4399b311a32144c3635f03bb15ebd04c9b18385e591d115061f426458b0405d6c5b94527b32a8015ee9945701ad382ef069e9d289723e9a3d57d4c90fbbe907a411e8330ba7c1352584048f9e51bdc1ec85e02d7fd59c7e0c16f32a4b4a7015e60b6f4fe29833aadb3311f83a5734b802207a4bc06429d055ab124d296c410314a9010b381afad7880969df42f1953df8a4b208a80bca5a96302732d765a106217c368e0cea263c5e196a1845b04010553d22eac782bb8d61551f096536e9eab01f5bbb9046330626c1774b5c948680ca284d421d591450a1fe517e3725d41e84b5eae671d0e3e85c6f08dba709870e33f74de3603533a0787023474ce665d45cbb08ccf3516479675b88804991cf6799ddf1c8922db1c461e756d4f1a903a8ea21360a411c210341715c883330010d77bf42e886ddf2b396c80aa7ca8b56a1676a13a6e7503fef59e0d6d08ffb1dda9e1d356800b60dd04913d27f68989d78278153733fe0b1f38a0113c34ef03809e32926ee507e3fc73978faae4642d7de6f463b1b256a2f149acf3441e45a6ff50ad10459604f295f8b923f285fe35c193f161c286bbdc010af0e84aaf22c1983607bcef7d3572f15d7d52ee4bc01c49c3c2c354ccbdfcdaa9dd1d48d41749b98e48d663334699aadd7f8f7749eca0d5faf45251c6c05dea633227c1814f78259387ff6ab15111260ee103ec0be9a206990fbd5dfb682d650743bd727cb4c640045d962afad24d3374fc4aae845cbbeb31460396d64a313dfbf29c3e36ee733d195245065f6013ed7cc2d52e691cbcc54a285c5082f199fa3f9c762a05d15c282c62ece02ad67a796b64bc1220dc4f8a926d36e291fdec145cf3ff7b85d1970e37f8bfd18037e22b53e4f221ec41825f3055e24ba86718348534da1f0b530d1360750fa5dcb207660a972a30b5af0f1f47c0225e5a256dc14a6bcfadcd2e78345a414ba47d10e60f69eba8de80de74e77177276f1d9a74f58928a2a8e0c9956ef315a5c598f67b024c7d3b62ace5fbf3854b96378b006b368934acb08b6a2089dd56057de8ebfcde1e449a5522aabc34413ea6c8e94ec20c69b2659fbf55f86f245a35035fdf1eec87edff16eaa9c99b3f4223296adbe7fa10d68ed9eb5533c25339bca3c93ac08354d2df6c58a88397fe05fc234667b7f042f6e2a5bcd2e46dcf073818116e21927493f7a746d0bc8d084051f44fa0923927ee2ae119965165ea288ab84b56004fedc697a7d4469e14a72dff14f7f77888fd2311925bcf30c53b7c0e9a8a0e53f4770f74870e0c0e16493c96a7f8a7b8f205e7851a78d952e69b409df249bedc167de8613a36733417748a877eaf1957a1d0fb4cf62b8fdb825bc17eb015ff2a95e8c366c71ab6bc5bd983ecc4f182cb89d1a94fe2ce24a9db3dd7b6489f46c6f36bcfb85a6589a531e9cb271352f24080392b24a6726b74a4928fd1489c081780c89fcc14a71b8dcdb92d68c74acb9330aa14bd0e1ed0494525bc559441e289e9ffad7c2cae64145eecb1023339ed43c14a2d6ebfbe3a8b14f45711edf6ff67fa42b3849fcc42a98ef1f24d85f8f72ed421e56e4dafc4586e17b84ac8ddfcdbe760285ccce48ae5ce628502bd0193d7ce3060fc4cdaf460158a794fde023e7cdae420468e4d7593bf98e2da1dd12e1a6a83d8dba4d9c4694a61ea00589f0b7ed3ad3cf418c7163765e00709c0fe9476956323a22f9e61b2fc8d78093fc09addeab510452605530b8e3b4bf5e077839a1cb19ef339a73460153a763ef87442df6defe0bf5ce9338f6ff2ebfad9bf90ca3fa6e26380137d074d2906d7334c931262fc97cabe07bd02121f9d519bca1af457922ba528549cd0dd11d087ca4fe07807777b3e0b70337aa9af87abdba7f3d365ae3079e66195d32315e8a3cc09cc4c38e2f98f405b589dfea5389ccb96fd1ed01e0b8c3f09890a431bc2e6cae8b005393451db76fcc3c643582bd135a52526ee09c7daff0376c9fc060c40ed11917edaf967eb7f9a46c45b8212afc479f69019d73c7038b40fc3d647f8b6457b7caed70c383f4e690739d7f75b7bda5259071421ddb232d929a34dca0d7d149a36fc2ffff2b92e619d13f0d253b3788053f57d1350bbe192a1ea8505f2ae51caac26d26078ab963ecc7da4d98649fcd64dde2ade13467c7d3da46d180bd7d1a7a005830ce5bba54cccc13a69b2142e7d86f2be38d065af400965bbbf94177bb9a6d93838ba8c3623efd778517166e070750c2ed21fdf1b01242a5283f25d19dc727ebf62b1a6c3c5533ab8285a874692eb84d3a9cacb86d13b93703abe90a444e24e51f8b1f9d1fe26db67e0acfcc0c2a84f19cff7b6978036dac90bb059efc858ded0567a88b1564f789408817d558c58cea50053eb5b9da7db9cd710475d6cfe40bbb8b8c69fe4a15bedf498d84bfc736729c485c03093ab11c64a6e2df9fde70e51635a1e21d359d5b037e156982d6ced67a5798d5683d8e1139650a29c70dad397fee241035eb0dcb6b3ea08e2f78dd49fcc98bd1872a8f4de6a60be0c9f0bfa9a01b8ebc93b1eb56ff6633c12146d61b9679c0e3d66f34bad137c05f179fc6f155f1ddf295964d1f9e859b0a5ea931f48bb8d77c1c7e189f58c6eb5e50a229993e5384990ee404552a4c40cc06eed5ca6f4e3ca2c7356797c8231b453baa9d0dc48dffd13d6465b54776ff035c40cd30302720093942d0353da92b39a261a5b06d05740816182036de8887bdabbfa6ae3c267f0a65229322bb025d2ae6dde3b7fddfc3493d41316f51b9fb75c0c3f83da0036996e84b45c2b95b0b814714004d1febfec7ba1ccaa2592cd8be776fe3f9a6fccb9b8fd4a6f7695a4243869372490bd2b8d768e4a482a08f135396b70988a160bc60b11c3cd5a00f46157c71756f7056247324cd859eb0ce0453eaf2f977fc9b80ac336cc91d9861ee0dc7f2364962c5c926b0b6d9726450a75a433d4e5f06ad05adadad0cd7c11fec6fef909a56dcbccb489920d721d82ef13a9a7ed1b48a3f3b81ff21b85cfa8854dc41ce95a4bf5a11d17ead3fa5227c98222a251ec47490856d3ab9672f5a1c79ae7aa961beaf98c27dd9be3045ce01f8202f82286ff2e1b53149c8c7a258bcb8dce2bd1ff5e54e0855ced588d9ed8865fe896c28484a3c531fe2bc6d94270f8909429f6b6f59e75511345b4cbc94f33930d204a9213443f16501dddb2a5fae55fe3d3957ad730a066f321abe320e358e575d30e2fb7bb4bec054d495e506be7129271062bf1770295ca8837b3b969395f775e0cd0904f0f88e4b81b4a5ac80a9e5a67c47f49b53d0ad3116924a9482a41fd80fe412b83906c9b99411bb0fd073638857f0497179ffe058f52744ae4a5d8262fc8eb87ba253f5994f43637c804a53a7a3016c6368d4a8c7169c027efac7fd0a6f6d816edfdff01789d95830c06989577d1c9c92a4f5237" + }, + { + "category": "range_count", + "entrypoint": "bundle", + "expected_failure": "event_count", + "id": "bundle-range-count-mismatch", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a22766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b86f0f3cb589dd7c867f411da54a4984ee76a418bd81687e3f0398d5932ce61988214b2fc3bf577876263ccb3e01f7ae558b618a734e919ab1e9f775ea78cc3078f518d2137292bdb2345705c3112ccb32da2c96016faa111d419bc71d1f1df58073f9843bcfc8b0553164af0003b7792689b6d3c7b76e25ea5cfb891476e28c8256fef44cd1f5f4c27e0877664ac2b43b241d208790fd4310a452995967c045a3033d7c920948ba19f43ab3fb5252e9ea9699b5da3d14000df24b3b0025e06f1b34729cc7fa418aaaf9d6da4a09d299d454262e3b64b70b8a49dd80881ee7eca94b689f832183ac3a72397d96151cd226e513f8a660a6729bc2932c4ff9b7e39f9138f43892b8063bc3b772d632d634c7c47e0d7ff4460090e7fd6c5be2303a2a345b1fca2b06fb5b3f07ff71a91353013c0e4c67f6d93160b26d4fae17216f973267e0687bfdb9276c24124900369240aaed7568fc818e6740648c7317f0eef66b6c820dbc58b2c57aa72a7e4fd876f4fffc6527a6b894803dcf3c6f9e39fb7bebb80d3acdda4d8a86007dd19755e482b842368de56d326b1a39ee4e81db64d919e39a0713ff5557afbe47a1c84a2737ca25ea69e409ead63be3c1051ec9539b7914c96323910443afe935b55c22b7e7e70fb84a3ab767bb7fb3251455770203d7c0dbbf3c27540cd2521eff7050bb0bf6d8d4a0a32bcf2cdf4ea3f06dcf693e150df8d357b95a923a00bc7846eaf9e21785ea949316079f224ccb18c764f004d5af9827f8e463e0e9190d30480c8b9cce90079c58d1ca5e6591c327f9c631a16b43961d41246bcc4ba07734606a84901f64160b6312439dece5f3a166d68c4139aa49a8226b2b4a806013f000c134bc37a65274ac7f9b7e9fad3f96d547679693aa13d9540e7622631ad466ef0733b2b00d29d1ab4cbd92e1e141174594aa9887e4004ffefe11b0f35f98ce947beb68de8ce4cf4a6d520a6a76b78f16c3812f2b5e50b4d74e34704fc7dba747f9826a62092c286bd28abc5d0892e0d988ce5e50ec3ccbbbe9a52961e8c5225196ebe4688dcbcbc93c7cd6573a8755f8134bc35b7ce031b5ed3c752aab47b000d75ee8f167ef0250baff59849e9379819d127a75635ce5de626b0963190f8e7482966ea6789366ff5b41bc7e2660a7271888352941da965b8973d0ca2ec89b04401766b39ac6eb894e605a7433ba1a27da0506a0326de071adefd69ce3b78619555b006ba67c73ade6223c6346b7c67045df421bdc863078b4cb0aa7d8bb817fb794fba12417a9cc70dc62f033ab771c09b1c73c48eda2558c0a644e5f3a79d645aac113c760edb246ffbc0135e4d50e12879c06319233f87f36e7195e3bbc9f900474736e6114a05e9780e53df359e488dcd23c415c3b0a8cc219b6531b97ab9ae1f662166965d7bae4ad85e98a2c621a8fb49d9aa2795a71751a05a002aeb2a31366bb821ded930a54332fdfd3354ff08b46d89154771fbdb568c4462273b9ee1911d1fb3906a165498babe7f31d25bcf0a3a4e35c15436f6b8c673575aea2096ecf51fd77a9a50ba2d6774bef0e96f251d770ef2818d591e01094953a80d5cfbc1c5f4eeeb1812e57fd2fc4f84c70a7d02e62bfab16b439a54c4029f3fe05eba9f5619adf2ef9de8e4de0d4e632fedf0178b185596236d38709e45632a13c93d171db46738a40963a98456d20c522a86dd351c62b56c87b7856e164312279caa5909c7079653083b5b41f349e5923b92314844b6a27b69ce19e2616ae35d7b5cc0318e080a59a045c996d9939632c9c888a5c0431684dcd2179297c43211a398719aefcae837078add8272c625843bca9a0f39b287f0d018775b28955154820a0a2d7d772b4ef7c79d45ce4edc22452ba9f2cdad0c3131715ee7e4734a34e5de449965b89254d009fa9424941102a8adf22e027aa7c608defc6ef6c7a6bf45fce378995138e4eef7dd58845b7c93c76461f6d637fe9939e01ef3ef2455fa25c3b5ba061daf3650e8cec17957116647036ac849e1063cff5503e8f7e45ed5c8b08560ce22db9231618317b876e0f5ed55c43cbdbf9643eaf6b13792c1ee910a22ddffd4c01d7cffd236417539274f3d8451a331711f16afd7aac7f206aa836cf9b9a8b917cbee81481fe04c1e61e829bbfb2ef5dba32199fbb2a78374605f53014a558caf782daa785c54388646343bbc553361442ff9909e3e39ded0b340c02e2bcc0cf5443fdbe7463bdb415b6a3baf806cfd7fa81135dd037fb1b3dd5b762341f0cd81d0b735c793fe1311fbc7183fae12a35d5bb3df20ae059edeb5a783f9c6aa0009395295859223e542e5aec13af600d22eb1069f2222cb30500338c1478ca38543dbfda2b60907e5e4db6ba947d5986353e9eb744f916f38db5550b1f2a7efe1c89ee8f86f59d0e5c1a6d7e8861745fa26b30dc7b220b79c38b29dbf13a9c41a1b812424a4f1cf44da6cf0ba20c4aab38c2590f34cadae7c56080278798040256540fd8e454126f3241a8d82f704b3d9c2d90c6d1539e8851e3939c315d79a3c5a740ae0fb049462f8b9b5dc67b72184faab973bbb2e02212062d981115e30c0f042628fd0be63abbca822e3b2ebf15772294e0e5df60303a7b17362600e11d674b37b5a7590aa5bd500248bdc46a86c1631a7d3659f573687e51de3d6e5d03427c41424348bd50780d56c0be61208b790ab2bef274c67c680a7a6b7b2fdea4dafc9a9baf8d85b5bfdfadcba2f788127c2ea726d130f68955c2eb6d3c7b157f69795f4dedb497a50714824bde7b469aad8e0ca538b79fba7650555bd5fae868759fff54fef1b5eb4a3f5efcd96d9b3434683ccb633e2ed6a3f2445ea3a9c01e47093a7943b5fe310772ecda40782dc73050b4aa76d3ad66ee9d3141a8ec93ae802b1b15616f78e4f6de62ba0de9812a87b10a55a90ee9de373b259066ec594caee21881b0b80a3d8b95f40e554f0d600097d1ec58eae176da80501f5818213e65078cdfe5206724489c368fbcd8af6a5e07aed6c80ea1b8e31d73e1197a8ef5b391372c13bfe1ecb762b200ab9a6ce87a3fc037288385558a8ad2f7f6691b070b46afb0886291b66daacb0b7dd57eff4c54e10fe9bacfefa0f9b8949d26f50ad2901bd67bd267e46634b08e1ddb660b00f6186fb2ca34fa61115cd4da30fe24b5904d0c04e9676d485e0926e3badec9920129389cf8e85683b9d93ddfc4c1eb41f234c8f845cb8463a44e634f5f8e9be2fb23cd5b46b6891f69089accb877b9c34b9d5254204107c3bbbf5cebee13a03150fca895c2b361cbfd5fc92736e14cdc247d5629bd18022b14ddd29e635397892a125210eb2f5026147a6cd676cfd6d3163e50df17db7e9234c4e5a8eb8bb47b6fc2c17805bd222d278c07b95787553bc367272170fe2ab52b12cf5daa2fddb88f1b2620c3c8c9013e44783371467588ce4b50b7f3037b0ead8cf9d15ea4f2b6107a83b903b26494b6a8e0fd9c737115969884f79cdcc4e08e93c03f93a3d00285305b0c8514bbac39f0ee9a9b92564f3d6873b5a88bc1ca02fa3b92a7ac276705e3e6ab9a675172e47b2ced0bc90b9cb6217999b9f0b80f86ba4ae58cee6bd9e66332184deb436ba4478b217f8316adfde387fd758d40fad83f07396852a8f1cfcb18cfdb5dceccb684a7bc788532994a1c430ea4ab1a17a9757456caebb8e738672d847934ca52e2313284e52c7e3c0068429dab590233a812d164d54848abc94e668f4c99fc612eff59cca7364c54cb45f227504ee271e590bfc21f97b6ab22c9d0aba9e55ed9e58d37a162118df6ea930db3532229185dd7c67a474dc382be21771e47234f07acd2e5f0a7fb1ccaf686ab1e2179445a065189999d6730db4b4b4be4c2036ae12b099001500a1f76332f1c6269022bd40bccbaddc0bda1c54a6bd365221f44d7d5447557cfabb90e895320d9697685e0a0aca601e69139cfca66ac54dd276ffaef0f3c1df7e295c1e62fc49b39b5e89e383d7c1d77b9c9edfbb10adce8169a0af08863c81de45751a420ab28d0a66b34abe907b30afcd00b9c9853cf49df17a505f69162a6f644734338f64f4f98c7a11b286cd6002bc4bfd7431078e6570b95ebe3a00cfc58a6a0d9938be42d4e871898b418731cc78ee65cd105993068a22328e5b6781e015422338b2384d2c9ecadd805516b41728f97291984973b1f0fe1e690b256ff1e222d48a4539a4a009051c13487a5bcaab1178731b6ba85df6c3f7ce" + }, + { + "category": "range_count", + "entrypoint": "bundle", + "expected_failure": "noncontiguous_sequence", + "id": "bundle-sequence-gap", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a23766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b59fed92bcf35daa17dca7d5dc3462317032fca363f26c2433ad87658163f084628f33d7c46f67520e963f5af988f122429259cc1992d88ea4deb7ef786ee5cc0d33e0d751d645d021301e020811a771d0ecde1b145655f5f93388e12b0e60ed08de2478ef279fdec9f513703b525ca37431d74f57ce7ee6767b240fef8ed42672bb7bf20436811a19255282d9bc573ce98a296798030f2760959b9972a1eb07cca601509135b40acfbd3fa9769ec1efe3942f72c4cd94b20b67789be7ccc80fa12dc0d64811fe1984c0daa860e76017fd6436e04597d8682c99d3f1ed06c587457787c4c9c392206c09383d08c96b43935c08f0ceda52bd0df414d57641f287e071468ada27d8c7b659ebdfc6a4eff74b4ebe6c7e436a9b9fba5e32eb2985d34529150cca3e068a0aeab3f7894f85fb562b73a6026ebe7355e569642299cecadf9b11e63443b698af0dcbf96ead4db35f421e0866ad191103334974a83116e1ddf34ef0ed2f8b9f247168267288282c36d87022fd952c5acc56c473cde4234d386a615ae5493fd2e049eb408c63baea4c129831851a8ab2350c41af9a2dc5678845b2a4ab5931ddf11e640f7f6008facc42cc4b0e2699c09797d32e0cbd8eec7230913cc4407ecc6e4aef7030217aff0ab20c873d0178a54ca091acb90abc9114f6210ab71b94eaec2715b21c573778595debb2c3544c3f9091de270870f6fba69c9fae2c293256b52543730afa9fbb5d4a2ff54c9b0b52b50b90a53b7e83c6324a55df6b0ed9693195b49a7045a1c853cdc6152ed88e34b22cf7e83d6be47842bcee8f316e06874ab8bca2bff8e2ca0c204480e09f2fb9d1fd942f394ef0e9b7e8c9eb7381b982bb555f5b29bae8a20e2e81d0cf9e1112575842e96e3100d666fec55954e9751dad1ed472ec601347b2db4996c62077ce93afd14b5c7232ba6c7959c20ee066943cb355a5844fbf92d03edf66a80cc3345f335d8d88dee17a516cd50f270a39d2860b950099e15d0470ce4ada96b4c22348ace8de9cbbb0bee535fe6faca66eefc05021fdea502fc26ef991c817eb71e67a3daebab0b8fe604265461973d191bffa448582567086f11728fd05880cf2507c9e11da3639c6b9cb3611faee86787f39c67b2e2b1fb604be6cac0b6ec3bfde3f7b14b2d2703db86a4c56494d1ff5852c4af7b1a687ae8c847812e3768b56f67fcecb8934874c0c4dec1063b475dfbfea6e80046da871c77d3ec8249494e0b874b719ede55a081c8e30601b2ae968f813a577b2495ce72e68ece80a58ed2a7163c7c05c078d06a8ffdc979b095a57d470ccd549c8bc7c11ee615a8c71983b3b38eba46f7ca22598e5532162453c00f0da037f078d1f9bbc2e8a789aaa758a816e4885c890f91d7f5d5e3ec261db38a7c3c4bd888acd66db6e5560e9fcb45ced4d8d8d2329b53672114fe7f5995a68f40584812edf27f32efc3ea531042376a4759a267999a89140e605c3797b290423d472b6487d7ade10d604fdce71fef533faf58126812de2bd882f78d5bef88a8f9e7483d1a3c4c3ebe7a7f05a1a6c0b679a0a71cdaea3d9143a41f7a2a48fa805f784001028f40be5dda2a63e1bc29bb0d5cf69014e0f1e7c7fc128259b2d629973f62de40fc04e443074f7d8db6e7e4e6f8d808a15f3cc0bdf56b87162f63007c47efba7e59612ac2386d46ab92599423df0d6fd5cf15aac417b3c8424f6f9f0471b8c0e1d3fbbdf4246e7c3f9cb448e09f44018d0e3bd240ed4eef4bdc9c9c76ddf00b89373bb809892556ab570f0a26f15003bddcedb0b3a65c84f4d9890cc817bdd7cd8dfdaacabbb25b14380804bdf127f7e4ffed3dac0c59fac4d5f42bebf31d1936f03e035fcfc3b0741a7ef0b728fcef508ec746204dbf402b95be31050ead8852a0023c70da70492e06c56c41b47103c5506e7df3e85d8375088cb4fe771d623dc06dea26945778c4e9e4f3e5c5bc73ab24c083066203c168e22ea722db71ee5cb60e3f1d9db05230b53276dbc3cdfcadc33ffc35e5857038ff5d9cd3609daedb262017f85268a30ae37875cc59a18c2677fe679137ad2776d0db4eee9385d33e1340a4391954dd134b06f0e030eb16171e1e09dfc928b9b4ad713a43038851531466da10fad959d48aa112c853bf72e14276232fbbb7da2298000ed5f80403270886d7fc22c0c557adfe2bb32c0ef8eb6994fa0ed9fe63184212e802a510afccae304eb093c0897790706c89ec8c0d9746423694263e5b690ab2bdab8bd40d914dd63397f5e56b43aacee54dbe06ab5a371fe49a04fa1928726adf860ee2b26e759a3f38c82a9251780bc38989d50e3fc70f5b9300650988af231400969e04c6f3cc71069554ed73acb310cb16ca1b4b8da67c3f2ca03d4a07008c0bdaf9288f2fde1c3aedbbac5322f051bc50c0618811f999a93b45bc060968f1a3fd8c7e83954dbf5a06006fd79a6f1fdafa841afeab6bcecb82857f3b560586da7f1db81f987be1276abf0a8e63b1c5453e03181625f7882d354bd5fd1507e476c608b8eef9993eadc4d5c422aa16b37cfc3e7f05359f426d0867f1af63737da82b78d57370917ea6095baf0c8fd482c40943b93157bb5cbc04dd437b7334284023bab2f89d8a34914647b5c3045aca00e52b9511b1630a43324a6ceb28bc59275b95b7987413792100e6a44cdffc1c3c06b0bac8b0ab5f0bb348a81fcd34dd254470e156a4e26331c09f678362200a4cd8f86e18d68a201fa2204d48436c5c344da89ce01d418da39b39a23cf247646bbb831e85673d4cfe4b1c1fdda2a0da8007d82e5624fc77308f0266ba8e854913e0a5f22a56e72ecc59b5470e53bd73984c0204ad11c4a0d6b5dcf8787b3d503042358c71e25f9164a01fbe475aca4df92ffb5879a7bbc2ac4abf11e06c6f8f1d69fdfacc295ee65235ce3d865dae077dd3dbf965d9bebfc7dd3b1bba0027f0cb99b365089e288561516d77f8af6f7f07fca9b1cdb53428a9d576309e672afbdbefc190f1cc0ffb61a407969965aea4cfe86d6d398fcd73c860d6c34705a51feb8fd3d99c4f2aae299d658fe69b4b0f4da772bc42fb2fa186fde74b1e39cfbbb5ddd45069a8392a9eca2a1270e6b6df498ef92faa243deedd94f24434047cca9c047d9675d38bc0b63c7004ecdfc2883c02a948269d15cbfe8be11cf7e0a29947edf0fc996ebf877e35b09b05b7944bb062b3eb880d225e5e43edb91b31fca3b16c82c4eba8d8525d36647f426912a4bdb563f3b70fdbb28f52cd76cd072509aedf23276d0d6223a4f8f40db02db02c9267bdea977080369dba249a3b758276c215110307bdb608977e4ff2edd66b6734f373b958afa2071d29c268b242afc36d2b9c99b9e20bf4d662fe7a6339034df6fb0b66714eb0e88c39d075d58012c9a44063befa6cfc41fe94ca43ab711491e1fdf613e400dea78cfd11287d0c801bf257df51df1c512b15f6f8573c45ea89f51e273ef68f3544c001768c02e118d9f9ff1111073d1ccbd72d088fbdd6904f20ca075aa368a7d848e7a762b689a5f7cd98bd764d7be34b28bca832d29a5bce46dad8e2715db02527b3f5cd46250fa4770c57a707a544243d7b9abdeab849f223203bb0eed9fd7a2d9a19170ff17b4acfc12dc04b3320d5621d295e2864453336c5640dc7d86f533925bc7fd3585cb39309ed7d91938f38715cbd61f75e0a251cb4e22956b1410940f287798777028c56bc4d47d7ef78026bed0082289b01c8be7bee035d59f5ed8a1b7100695c8a3693bdaa6f41675b3b40782bc236b4d41a7fb8a2468e710e6bfaee3b0a3ce280742e208cd56daa12aeed51dc99c7f5e5b0e0485afe90ffc0057621aec2c7c36e8b142fefb925d93631be915648bc284a5ee5a6196173b837efc8301df5c943751426ad30e32aebf478d1160997b87d870ad15454f6e1bb3605e165a70b32b5a32e442c40c0fd49099e95e60886c1f7c1e6221364b7e20d8fab6eb188fa18cb655e1930fe0c1a8b25f093a81aed211ab3cb348f38defb0da993ef1c14495655063128cf761781942cd336ab05761470a4ef85ddce43b1e71c58a58a4b1a56869f3045e943f1338136add6fc8f33c06341c6051cd085311571725663d1bd941313da9c58002617f6c85400a86a88e940aea1ce46774b0919d973418512a1d00d3eea961d9758a0e031a66fe5c459fa3a56f94467396cf6e89a2604503ee3c4b96574e3bc2300ab644a8f8f778681858656d44998eb96787bdcad91580d60d39b351659e42b6f46a5e002cf4a261db4ffebfc472e41a83beb062ca56935e9552a83c095584771c72812e2269647fa4bc312740bb5c4d5d9eb7592026d2edc499873a2764abf4054b88747073fdbc4fe67211e9395ef1ace90448e63c5a65b7d9d804061110f1495f17c41284b28003ed024e95c6e475c50a3625a4db4e9d15fd173b2379214a508f0a955cb7752afbff86e8f50d3c2d2faf331a9e9f1de640477ef931fd684a95d2fe7bad8f4081d4d0d6d3618af0d3a6274d55bc0bce1f2655c5ccb52e12048e8367460dc2a2eb0ae10dd661a199662f3d75803f1aa6a2b363b9ceb9979e718c3b10c041017634b8ca9b1bb4e4d1ac23cccdeddc82a2b77796f917f9cfabe7912184d94" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "unknown_collector", + "id": "bundle-unknown-collector", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a24766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b7265cc4c59fd270077a5f6de0c36a166fe4aada8188c8f141c5ff7c2cd29cbe898664343771365f86b22525bc02a2611faf0b1cbaac8de6d064109d111d241bd2bcecb7ae22f368e2c55e550f73e4cecd2e75a5c88bba32f604c54907a3c99dab53951eafd22e8e81c84801873b2f64d9a4f701aca60efa607b52612ac2e35f93d72f311ada81dcd4761b3adadb7aa32d6dcaef85fd1f2418284428d875a83bb40063a45d8428ad4e66c212738ede1a55ae50d4b5d877e0e74e323cd8ea71e88e46330d36d98c38615d37291a178ec43be46a7ad5afe326be8c6c7dd9400d5b416f014bd33070e28d582d0526833356cc3b131f7843418d8767b57630807ee2f93df999f849e0f1b7a814f2d69df62aacadfc901852adeedf690ed439835d03a0295d68acce3a5b73dc67928d87f78cb5547dba39358643bcbc5461afb2209eb78bdc616e378483c1a4cb924ee5dc5fdac628a94e0fc42f6ac21b7b221e255fe46f3d12580a8e4d73c7e6c5f98c1eeac64d6e90d5959d3be5824811d3834a32793b655d1f3d4efa7322a6f0505fc7678d4e3aa782854c02bb9236b114ea5d56dbd3c2f213b71054401e511b3f6bcc527a4173372c5e7f453577158f8b6727843be78874a3d820f0bb59f1ce84de513706d3d739d5f18294b8e85868011764f86d8c822f05b2a8aef6ee1ff6a3887c2dee8769e6aa9f3ca134b9c6eb24f4c3a6e1f27116c64f0cba8203337a1fb2ed9520c878635f050a78a106f4d4b6f89f9b949cd91f58c214fb2f43dfbe75d2addb446da21c4dde2d4c3ec70c3986226cacb08f0a8320bb154f957f128d77ed69bab4b7fc29614ee58a4126897ae6dcef02b2f407bc5b69fa4cac05e28de0dc8095b15495e4d02f1143ea527b8c72240d09c74b41c18ae4183d925d4ccdf5f98f80fbb05c403cfbfc6dfbb3ba434be4c04fa5f78a9715c9191feb565694864fdb7208a5751670e811e343986179a6bb0b2d9e6304eeb5e67a249e0d72b7b8b37d54fbd2df49bc4269c2c1d0791a6f2ef357ede1ef8aa8ed64cf4b76e2ea24068ffc64f1ccb7e85aedc2054f3e6f8205e104b2f3c2b4a5b62bae72313ea9d5952e28a28a84a7cf060dac5ce8bf6dc3b7abda7f657be4f9b5f361ceec578958c14e0ea99acea74007fc98ffd9747408ba18f4b4c3d0ca878963b071d34a82a52382c9bdadaf23d92f88cd6d00ddebda130e8146b06a512021f9ed509d2018d2ec7856b5b7db4c5b34b6a7fca9132eb1d20852c92a49f3431532b78e40ee38f83c2fd494f9136809e656cbb6f38a663acab67c32e4898c401e243224fad4265144010efe5b3aed6ca4353f4004b822335a8b4c25bd12bfeb1badbd4d7a8ee58922223c41ab766c50b016d7cee49d7e0d09dba4727f725e46ec9e7691de6ab299b831e9222f8da442704d963227841bb30f884af5335bc90f9e39a374ecd0512f5321093c9baf5ca758f50c477e104e2885c21a2e3586d422466574ca3bb81afa9524c9c0dc53278a9d4b3769fca5a0b2493cd961ff3898211ecbc02dfc5785745dc0110daaba980878a7bb77cf959a6f76c84a8c7476c5e88949465fe3906e06594c9f9460f72acf9eeccf6b55148783eca5b4349fd8a03c724184fc2867f33d6f7389a1952b4d11cb9817b424ed8d08c87f24fd603e6e666475e91486e9b38e71390ee6383e9fee304678f2237f0b13438696ed6fc10033b195f3a4a50efbf5ddabdafcaca174ad4a63377e52edb15b64ef1213ffc1a2db18083639124a3f6f0e035eac668548a4a5707c3a28d1c1abd91708238e65b319ac819f866be37f88bde4e72c5847024975a1bafdc8d5e9cd7d05c5c684e55f2fc95bed9ef0ad8dbc67d4b230bdd7d2ed668adbce93a5504e17e44b1f66bcaa8097d5ee99c2f222ab86b45f86f05db2c63a63e38c98d341ef43400b12fd5fb7b65bcbd30398d12bc2d042a292891f8662ea5d9ebcfb335af434109a5e5ff9edbeccafed74ded1809d0bd1ca5c5cf74918665941beed46c5d8a16f97b95fb842910c1e3a0263c124002ae6541e2fe4b52797b08edeac24ba12629ebc382b56553a9b7273c458ec03d6e0a61341bc613c93ef07589efb2c1281750295875cddd267852986cd2eab90aa7017ecad6befdc72da54d4ccfbbc88c9ee6d5e2f0af5f8a067fd54ae49bc89550ec4db5645dac5853ba0f0249b2931e17f3486aa771886c03b1ba7e891909d64291e3a9c1b4d8362138e1642ed556ea76bbc14e4f45ab177b433fe0025d43050a176dd4ce47b038016236a5db75f0ca356c437febc2d12e6b0c61e6c99aef59207447b68b1586ccefdeec0f21c58c6d3d1493ee69fc888e557736203cad5f0009df9f932564c567636c02901f9446f214bed56075649649866151fb299fc2f55939e2a67ba106517e7740a4968a893530c7ad72a001cd6e535becde321d000b0d724cb06b5b2b0f66b242180030d1e7ada080cf737f631823ca9c6f6a9fcd4d1be8178ece41c60d4d706fd61d196b1f88b308812cfa68cbd98747e828a41475232e8accf0dc42768e3a7fcf32c99282025dd9b468ea7e4d6908d79d42036ddffcb7b1c9f04c5916622395498afcd667f05f89a2bd58d6db896580387b386375c3332a649cd242d039e8d65acd1cfcb3045510fb53a1cf3fe3b0b540cf4783451dfc50cd58d3b12f43f0c6ecfa4c86ff84d02d399e106462fe53173b4c6cbf9aabadb88e9902782c003e09b20e03d943ee7f155189127d925639fadb409d9b13fd559894efde8ccefad714e4966ee17828758627078fa8fc432098ff7b34ef40e7e42169ede8b376a716ff62cfd2499b569b9c927bf983bd0820430ba8e3adca3ec34f59f8f1a4fff6d003628d95f50e37836b52dac8cd0a3ecdf041d26e15704d2c08b01f099ffab429298c1abb291a43ff07b84236470d52db0e2fcc2570e7147c3531230b9c66df461b28e8e6bb93d470b3f54b2d5d27598ba9e8b43ab3c9d646c1a4ed12c8d512cda33855586794696c5a7410816d78752d87058d78b544b0e07eff64a078b7d86b72deb305b84cff6bccf489cf95a08ec82ad04e77e4daea9061f257f967c0af779e79159c236b6e27ed5ad4409990cab9e93ad69f8ecd0cadeaf4f8f68255d06629ea7bf58e9a5cbac44d375427c3a2c2884a9486d9ff4d94ce42be10adf1af346496dbf1888ec09aafcf653df26d2da163b1f5a7bc513fa7ac483d5b16b20974ba1a21e59d6d3cfb1003eda4960504f31dee4577a028ba567362bab396d97d39d2f117a4dc5edb10d74efd35078c9df2dc9e87e82f8d8aa4b8ffe1306379270e0ba4eb84d20fce36524878c33374aa5442b8a3aa518079eafd212b97fc1dc4022bd6534b31ea5f4af5d9dcbd5b31191b89931099f3bfbd3244d4285691953d354fe8dfca7c84d142c396d641289fe1c3efec2bc05b9f7428c9b73d3dc6717036e4adf2659f6e765184068ff29175d7cb04fb1550ec9e4b7e7cc5736850d2bc02759e8ab137b3fa849b1b343a9b7a8b915218357e8903da61e562ee616126b12a73c571e3f71969863dbc1f5c35c89cedc0d065bd98b454b0eee713c70c7657a8be5ed68fb4131c6ac364491140d71e9de5bdbb438105b17e95ac05f8cdba91b0a9fb3aef0c2b05d87443d2cc47ebc01c674fee28a0bb1c5cf1d1b3bcf3db61897d5b3fcd4d43c17a8301a8353e828af425e70866d0e1620ff1c4a0004d31ea075a62e454c8d96be45720c9b87d0584d8ee8d43ad2f7dd0f1b32efa40e07f948884d32e02df5a8f5059f17f31a54f9758240e563172af4a8c2b800103d21cb59e1677a5708c3cc01937cd663e1a5367dc65cdceb6fd4a3bfaaf94984ef4908c4df8a6dbae2b12b947d38194bcebcc8e69fc91b4011c2cf751d4acef17f082464e961c4510b46bef003b4c8aae92dab7096be25907840624ece7a55aa6792f7d049dd6eb003619b2e950a4f62d59b1ad98e53a929c5d6aa19bdc1717c2ccf60d0f692a3fea1431d152e15785233fb0166f4cdc35221e9d3ca620bc288dea588c2d70cdd96196472308be48c38a89f51080015f60f9a14c15d0c187a6523cfea108fe9790da8652910c48a8281b1ef0d8def13cb4ebf02c85aff828c4e67d5bee3331100fb29fe1ddcc03ef796662f29f10793b3beeac7f9e1b32f2d045dcb612ecc23dfd3950c81132c17d4e15aa3781f54ff0dd7829dcf2e09edb2480c3dd1289302fa571c784a7f260f52b2f98542a42" + }, + { + "category": "unknown_payload", + "entrypoint": "bundle", + "expected_failure": "unknown_payload", + "id": "bundle-unknown-payload", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a25766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b78464c7153c7c48c3856b8fd320e3d5a8ee49bdeaae658e489f659905b029bac3ef90239df8e881680e8dc6d07b1d48a21f5af4641df79d598cb5ad33596169d1588cd687dfb3b31c6945bad09f76ed14afb54f89c8f2c2fbc34124529f5f5ce1c810d65d8f52b38f9befb1f538bce34529e58b94a6344cb2a89cc0e8acbb3abe280b696ca457104e2afcd960624984c42ae0802a51042984eb198311a8c57ab79c6fa290f710ab98641d71be13eb63d4491580680f37d1159b9fc596f7efe42ace1966d47f22b6dd8f171df445ce02d485992d186b12518b98bbd423898af72266131d46402c04d9c312c9f624959fb8d1b5efea022a96be64c1dfdafb916db81d03d1ef13a0b1a7db4b15e634eb560e3fb85bdec4177c7bc84ecd7c2c01d061028544bb85f75d20d4d680e425f05dd2006e569e69db477737bbc4f6a43ebf7514aa220d92751ad9c3674aa444d72f625179c346f613f63b133785bc619bd46656a1b2e5d8d3f3d02aee0c1431995d7ed8624b8563f2f4af617d73cf27f4e62a83b6eb574a23b8735ee035fbbe90c9d80e383f03b8ec789ac95ef2ff2fc98b974dd411a6ba2b37b09bc01203b4eae7d69e87e85aeb6b09889ad80ca4f0d2769311f297318a339f003f2f1fc098d3af3c7aa7a9a329129e288bbbd293006558bf5bfbf49e4d15347005afda04a6205b63313653b6f035dd0207638145fe6e9642b5a0eaff0e4c57f0b0700d823dfb5cc2b8c13869d7e9b49d5ccad4f2bff56d631a1d5c28bf92b2fe286c878f1d9a8355f9ef5bfa90433bf901500c2926e09e5b73fd99a3d2e7f5582c0c8503c35276a0bac26e3b8fa4a0b95ec9b311e1a7e54e7cdafbd0b450e2a55b3923878654e135d71f620c42fe7adf211c743a095866f328196b8b49eacbe944ce122a258558fff7e6be14e8e7540068fc7ac24efffcb06b5b7be10adf5968ec2da9e51a26934e2f7dd753b225544bc22cdd5b5ee5b6c028f8547028e3bb5045080743f045c3c250328e0a0a027386e11adbe70e724542afd125df7cdb602059548596e2e69f279f038f7a25754aefddf2f09f5055902c21074bbdd4ff973ad252806488f7ac2681d9496f49c4dc0a9ba7af3d044b8f28bafc7e9fa2fe3729af4123f2d59cda0674b5f533bdc83a2ac3040c29f0be70641bbc4a8832052193f64149c33c7cdeab78dbbcc5d00ba6a65d2a0ea55092bd4833c2a5a42dd6046e9431d4878da14ab9d27c092f18442b3aef76ccc293e01db028c27d7948b7d5d27e5a2b4cd1b1b5f26b2690f0c2c7bf874ca078fe5b28b3a671adaec78f3b7b2599543428f7da09d1ff0d1c9d65f60a63d53c0dd98672dd1f4dba5129d5d5e07e40b0782049f97cc1d8dd1256cd208cb323bab4811e00ea3184b7b401f4afc45bedf788b4979dadce3a06f01eecb8a2884f6427f67a03a9e8512864349aa9c39e2e08ff2079f00b0cd5527f26fb0303d4c75abc953c8a508d0db6ff2f384b0964ccf4ec4eba595040195a5cb0df7ac0f4f82508c46c071204d9b8d53d4b190e16b308a6021476a8f752b5e95e1aea7610a9f1c167b0779c5f0a4f1f5170ac58829a75a8724edce84d30e5d5ae342937235b2557ff00ecacfe7ddc8bd8b3e534c03bfd168807b0d2cb5f02086334441ac0dd39653eddd0c468f42e3074a7dda71f06dd51f9f32216f35ca4141201ffa4da876f1ab9b6cdeb4c8903dc7eadca64cf42ac1b5e4884a13121f79fa03e142b3431b5b9184ba5bbea1e442f187bdd9064dde476c17410f6fc8f7f5627a0073dd5dc3dae8edfd109da72445047f98076790bcd4862fdf61573f66d9ce895ca545fc6ce0de4091eff008c82884627abf6b16b505efd4b0c1007505f17ce094f6ff78e5eb728b858362f216e3dd9ad2ded2f5552e2c264e23cb3f99e5b93eac7a886c2c0fa53bbbbbc2cd826f9d6538d4a54b4cbd4e3e7ee70b6d1843273a860e3a09d197a350ad806eeb59c4211bcaa4f3f8c72a19abe660797c25f849da32c265052ea5eab9723461cb737ecbff311741473cd947707b0c7caf8cfeec376e5d65ac1d4debc3523913b714af677b273c1e267898b8a68246c134844ed2098c85dc5a8966d86efdaf0fe32407d58d8b4cc9f69ed66880a147d802f8c67a20f03c25f8920357495eacf6036d8bf0bd9d49ec3b91233536699d95304edd0cb3675fcbbe8b5a632a4ed13ec4c2399134a52066f73689afe7b11af08bccc58e000bb1a17b12684e0b48862efb7c364ac88aee12d0638cc4ca05ae0f50b4c2307fd68364ba081337aed9f8411dad73dcc7e2ebb0ca2eda387ae7cc78630e25e06d3c42add934704e3ccedbb475b53120e02f990773e1813c31899159cdc328cf771e6181d25a86dfec57d41cce2ba783a142b3a3c0b74b8545c08f2bf54a7baf52f6c6f520cb73cb93ad6afde202ed1d6c22e2a7d02628074de49d49a52c33605a88fb4b710def0c8f9dc1a20136ded08110aa3a67a7040080382ca52fe2bd1be8a5be7d29b0b1a874044702e6fc0c3e1b3d12cf98da9139a3e07d1eaf72d25f4265fb847137e4cd71c118c062c66b16bc9acbabab2c7853d8d7863dfc8d39d52a322d039a1e568626e9fb892bdf95deb95ea34c994193a58e99db1a525a18c287c9ae77689276e4fbc0f2f689c835be6266e3738d21469909e1e88b6d194deb738e0a6f711afff7067a3a6746ecd05cadbcca7d47ed6ace24ad69cc3070c167448ec3e8463862079d5c29034e15ba1e91b74b760fae450236ec6342f0fb08679b600cc2b66565a33ce74bba21b0e321514b7545a57a49d34083763427bc296d88993bb3b99471ea3430093216ffa3dc0830626dca70ce6c6dd3bc7559567a33aa87db4deb509a8c8093abadf1db8bd48075ea071170f64d970a0a9693edc51a58b7f2d5bfaac0fc7d17e8328500ae11e8fec74754feb52652b146c3cc9586901e9f27f0bf2399e399ae9ca9104663e87b212e06236d369b7248f96ac0b0219df055b9a2b79d9deb845d5d50a7e5dd9d66dadb9a859d72d18ecadc31bc0c495210e2748165f79a0c65e27d2fe185df2afb672f85e5bc20db5b4d6c42468ddee1712bf2a937441f462ac759cce281619744990273fcb7f81587c29f192ffbd0f3f27d290400f1d7c05943660e59d3b74a79cd381209b119170bb0b71524b33f29a1451041c9cdb195b3edec6e2292b57c5bdbde275941cb748890838fb467edb084a5282e127aaadc09aa1cd1b3aa00b9d282b60e3e0520d89b511bba2db67d03c1449dce01837a368f0b8b3d126d187f7646239ccbb9433c93a2532a63c9228c0dfcb4b7d7e4b7b68041fae3a794d60fb0648f348d19e25ea4b9162a274bb64289b92506ca87c55f4fb8cc70ea03c6a30c7d195ab770cda49cba91759f56739ea05195ad45b1b70d59b07060cbf954cfa016ae33cd904880dd597bcd0247d55db686e91522dfd8aaada82827148b79cbcce501cca65fa2fe9c0a954b448d6ff5f744cdb5e330c585ca5e039c4e9362d3a1840a8d8d525073460f5fc94c7641ad77fe7f9aa3abb7806075c4a931fa8102f3736d9bbcc38316dad478025ff546c6fcd8116b6ca7b8259efe178aba304e3f16244ca4ab238d4cdb3f326ea1ffc56578f64d66d90b3b0d081132ea6f917226fd2ea0594473811763197ae6b9abe663beefba7963e74da91114691e8de035041b3fb5e71e8e57010a14baf844b05c165539dfde0a550e49c8a78c21a9b0bb73cbd0b5a8f927b9b9e9456c940c010e2468f315eb603a3dacd28cade49afcbc3f9d0247401024d2a0371e725e16b987662dcc2570bac16304abfa8208f6172737f7bc043b288cd47572ca63d3441d69c746d0b7fd7f4382723cc9e28af4ff9cbdaf744497121d9c6ef6610d29cc0f2a5d975c76523a1cc27783ec195aea639762c2801822c8be175be2692b1b51fd75f6fb7b4c849dcc25586defe97282d5efa9a330e41f3665447942f5eaf6e2010c729ae083f3fa6cca4ee835c265f1a7871163b78c8618538ff9c26ceafc5826a471c4e6c924998cf9c22d93d39307400d2e9298e3b8e313a58feeab415c3288dbf628c6a7b180558fb82a023c5ba5f0782e458cdcc31a20fdc4419c0d292176bd34f9b0edabcd017b05a4ae54d1b162c5ccf30a23edebd6abfd1a08dbae5ded1795dd69bd3a7d12fee59c16e3fc45491962c94e60e8ae9a230c72cbe4b01cb4ba789cb61eabfcdb4eb42eb24b3f222b2d00b556a3dc1e15d0" + }, + { + "category": "unknown_payload", + "entrypoint": "bundle", + "expected_failure": "unknown_payload_schema", + "id": "bundle-unknown-payload-schema", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a26766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b46535ae3f3a47b7f945ce019695439b27dec63b7ee3f62d73a072bb5810d2a1b2b0a08cb0a926afb6bef240fd3bfb13f2bdcbc49ba51e8ab2409f3a5112e7e020ae3ec95b2de6f5c85e0c463b7cc4c2b9c0f1bb119b43218be721bf06f5fee4b3dfb9d66160eedcec69e55cbbcf2ca49c50087ada866dee44910bd9af8af22df526ccae37077cfdffa95b00494b27503d2ee4c534ec9bce495de8bcb13f59be97e8f112799c896ae25ec335dd15acd3f1286e9608d41746276bce1e1194680cc6cb08989e4fdbd422eb366e7ffee4e0f322387889b196dae8e9e272af02face09368027e70d938f3a2fd5f9f571c15a6b47d078bbc0b7fc2db16fb36c92ffe2ee1df04a9e32e08e60de2d6fcfbf0a75ab9c39f37b79ee7a36154c8611b1581105c7f64eab7102f7044acbe3d9c93554881ecd4cac6c262f342bfb06ceceb502e79091dc4dc258e92a97ebf7044c935cd9da7d10da0448125f9f4778176c7e100cfff02e68e66bc0c637bc77410a0c95ec0304542f68fb2e99a901f252dd57de509ceea43703e6c9ac75ca9dc4fe53f9a43e1e9a5c76d70f11f0497a1d8cec720d2b1f1fe0a87aee07227afbce01792d84826ab1e89bda6bac1086c61abf551400a50ea8bcdd77bd1585fe25bbabcab6cdcec71b9208f342ecfdc5962a0105dd37c6c93195e7a4d094e48012f857a1f46fa3b084df4fbc84eba936e576d3ee46ad8bcbd4d88afe858a027a2e2e88e83f1eb461d5f75bd85229116fe0de838ccc041e601bdee5b0d27a363fb71e6b32d38ae8d91a5b3b5bbdc680bda0212e6a96e1ae56195a1e4437607a0a30b4a846658688113992e6477937df492cbc2725561002085873ac064309f52da09b1d731d126e16c3f4a541003cdd6189aa8a8b02e9a9d7e4fe7225fd66df0561137dfdbb9b5f1af9d0ead13a517f70ab13aca83e23302d81fcf6fe8f6f7cf5a0853882d2db4b51038b00b05a9ad1d937d28717dcea9fc20fe1d78b642360c9364b437cfa40fd29cc9664695ccbe21de102ffac43f2b823f2cba2a897ced75b22768d3d27068befd1e641e36d25677c9f717a8c5867e63861cfc08660963525650558aa8338540f3dc09c5c0b1bd419da262949d5a6e4bd6936459ae85a7194052633ae4951bcb2fe8ccf90b47376222ee1a48ea026303b9e0eadbd008dcd81588363eadcf67d6f7cabedec1a24aa9266dc0fbfa47709a7d8536ec137026bcfbd9af441606f7184cda8f37940edc3a9f530602940af1f1f60b9b436f8ae6bb3bfe9e9934b1271322e43f4d4d2b7d01da105174e583052f00f7f06ad2d5bb85832c4e0be495dd3830e482d696a8f6a66d5bbeb39a7aa6fde9feeb70274538712f948963e0a60f6cf9c642e51f5f75f1d29e092812c7eb87414319a89b085af19510fab237d97d10f0aece0fda403eb73ac10e740b34b0182c61b140f9aa8ffeb9e7460a3c230f8e28495543f372c29d4f3273a34e3c26ac7c79c02f62ff045b2b531a22c2df9705a722020d48bb72754677fdb2ed61db598a129f033166e201fc2c9a81ad177fe28616eb8698b61788f0fa6a369a6e5e5769dc5d515132d86549cab8cdd4be489a3e6cf35431b6e3355bad92e83df2bb8dcc28d96c4bedcd7b05f067be6c27e515f7f24808565f419d8e8fb28df6400851f3592bf32201988f34f5bf8e78734c75488a5a10d49a27892188da7734d317d8b07e10d10108f1113b3fdd7b2c3632624968dae745df194f86bea1ad7476148f3c223419d60932911d18e80732d9a09d271a5e3927b5c529e76b6f64a5510dfbd23d6be0303699c57f3ce69650a8fb1d6d6ee498a3ebae9b6d48f542fbd39ee186eada0c23ec5d53663988237b588b3f2a0aa607ce7ef7aa59f8771b39abc71b119fb831859a05a1b9b31ebb0a213cae01c08b2def4d537c7f7c3a1ac9f361cf1fe387618d1147cd164af9243e9c8c7681a55ec8f0a5e2f12e82fedabeb52f76b267fe18af96bea76b99398508db6e38a54dbdef1213862a2b9e90b0946dd7f5c8c865b6fee0d55970915ebdd2749d870ea64b0266828a56e224279e9bf20115f86392b02cec9275002bb7473bac4fa3c77a01e4857aaebfe44cb9ce8173cdd80f7adb6605c2ec42b8ec3c87e2fdb99193e237765fa3c5f63f1a1a27d524ffab16e1d166ce3e9b7a853381ccece84094458efc29e298726bb704add67b970ebf3a33c02bd245d9c226a31bbc325f567f8481d090ba65e2cfe5ef14d1f7ea3d6336149ac727510c8084755aea86f84b3ce152048fac40f9c7eafeeb2b747222bcc59d5f21b8c7993f8faf3d5bb275a743cd64f854f7353fd9dbcaa39884206966491db363b4f752a3fa17687d33979dc8634cbdb0645390a4d8a855991b1ad9ea43b5f0633859e945ca71980de01cb2a6750ec951ff0009ac19db75c13b4fa7636180d4508b48b11f6cffcf8ea810df96298fc716e83d6978b3beb7b84c0d400f4237697d3734ffe3aa236ffc8899bcf75800dbd07a31bdba2c413900aa759e992f0454ca114c74fd201213fb9b8213806e06c7aaf065c08d051aa05216952019d48f69c23924833bbe17608a7caa07f3a3e08f6329fd7f53d4a6a66b9e76b5fb2f020f840b92bca59352517f936dc8309c20fe2c09d755726a843c2c7ac31caa0aeb058626e768113aeea44692652f82447a763f81e08cffb46cb1921bf631ffb595e6283938a0ac0ffe74bedd7d62564135c5606aba781f11ec6a3ed7a35c4c9c99946b8f23e7d48ece766d34b4fdb36a8d05a51f6f2578af0b045feabf70af86f56ef973c1e0a254db720c5a863b4c92c97f950557259d48f41b7eb823e0a55d98234c594e93978cf436cf107f66ef9fe9223b7f7f196fec49148a995431b1e4c9a39cc5271f50c93d41e3cf7c14c6f27bd10e34e98addafb0559db7caab5a3aaa3a173ad030238fd346e15d2b57773bfa5cbfc2002b3005b3a37552291f10d93bf0e12625de08a82b18c9b6191840ffd38985f12de7b228cbbdd184024151b207e56c419738bbedd764c2eb52306caa9f2e7bbb357751a49dbb5d0d3e205086fe28df94373909329827d2ba8be752cbb89b90be48481a559cdcaa9512ff484278fcd72a02eee35d46489464815c56572f7933281431c1c202e9c6a7b65538cd1a2a589359a8964b06a3e1e71bc34a46a1e6be8c4a599105374cba733aa02c5df14b12ea610bc9f1b38f7809abe874613eb43129a6f18340f7ec5bc60a2b440a29928963893a8d7437e69e2054a181977a8110a4faca63692ae04b0361c30164232d51fe3e92ee4a1623a76d94d2d32d3fa87db217523ab7c7eeb4f32e95c288fb2896105f48f575084840d732ed0e5abcfb0167cd50f02e3be978d335dac03831a7426ba5ac8cd778c45a0a07a1c3b7998736c54927dcfc7a2dd58cc3435e02966fb666891b801ad95582c805f4d5ba0f3fe771b28fdac50be90ac81d833866f7577a5db0aaef1a3d27a9ab82cac0311e735b84527d6bb3e9d0a5d30a3a0b48e600505e875899c911a01b69987fe30dc58af86d8da9ec67b90477d43f1ce028d44e22ef7e680746f48a22b198b8e87ff301ca3726d49cff4097577167daef0ecf7f6d6ce81ddf83b82432ffd711a5755c74414e146cdf759ba09c7b2be2209e1be73687a36dc53d84071833ed6d3bab4ee7fbc9b10b7bc1e5cf9319f102547f6aec79c9b6a19bed9a8f5e0f2a411bfefd5212bf3be59b2034e9953fcb163fcda109e8c4d7ad24bfd0cfd5c1af48cda996138315e14c8a9d42a8b115beef6684f3384b6af597f6dc7497a6f7a81e904a5487765df9372ad2ad6ba9b40a3c2fb6ca65406ded2ecf98c471685621fa4eeff5fedc029c8cd032b3544ecc565413d50e7dcef8d54cfb37e5c645399e48913092c44e3888515299189f89e1d960759cbb7f436c84f886c410c2dc728edf093bca03bb0d6eb0029af1ed3cba978f659274ef0fea1492d65e0bbfbfe0c5d14a39915718e1daa4bae6c91524fc85f85d38ff41c1c65d1a424d99e51d60c8d244df1884f90a5616156e2b67af181c28169e8617389ab93844989dd720c2f3379ff28bff86ede8978f1ba7d541cbdced3ce98d8ee294c53a88b92895ff1de0494465509282e4fbb17380fc5b5fbace2f0ff062db1feb38cf8925cbd83f1e74a4a9b9c08a4a30512b667a8d042c3f4ecf3e6440b0c4110342a62c510370ae38a1b177478fdf8c3fa898496d5cfe075b7e37f291bcf28d54b8a6c0ad" + }, + { + "category": "unknown_field", + "entrypoint": "bundle", + "expected_failure": "unknown_event_field", + "id": "bundle-unknown-event-field", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a27766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b01181a0ddd90dbbc6a039e6ef0feb4f57302f8e7caf04887eccded9a667962ebec8aa0444a155dce64d6a30c343711ea26a75b037b458fd01830174789fdb0271ab58b6b767057020b938833723ce23813eed955e4913494cf20d2a5d11f96a7960a165e4b10644139b86b8652aae8b00fdc63822dd79c1c79f59307b0fdb6579c5ab9b74044fccc8ef1421bd97b2c85fad83f61662a9dfa9d13773c36288b0a45a894f240fdf8cd4a2a2cb84aba625d6d8742c8687aa4688f0abf90b5cb7e7ed65e4fee19fa9d8f9365dfe1feba13a1911845fff5f6fdccec10903948b144bc320e27d0c7ce1406454bf1ea5565c360e484ec0d5f8f04e15222ed922c289fc332ea433f7aee13b99f6c58b770c13d6a0892fbb3413a20adb553fb1aacf8dab263e4220fc9e3f90d01515cdb7a72d26017302b3f24346ad2627cd5ea12d2cf3b0ba3c78f5de193a801bc581a89e3a0c25eea4aae4494d0bfa4c8f4a33607be24b22c3cc2e2ab4f0a635c7370dd8e33d4b87ce013e3748c57f9cab051cfd4a0d202230a153e7393242d096881691f9e1230cd004714abc05cc5c114bdacd39977848c2a082b06f9823ef859436e9322afdc79cf60bb0ddbaec62f65adbb4171aebd1f6d60d1302d64a6abb6a84f54daa177e18461537a58d7dc5d10b99295c8d6b6e8d9f808618c2369cab77c6bee039b19a9f6dc295e5d6f463a47f380f167c20d13c4945c685796446dad695c5ba5c6f54375891ab4feb5aa8fc9ad75104ffa95042ac159cba0db5bb0d719299267d2954945b8914f47a489409292bfb4b4e338e8575c88824feec6f470264895e1e1973a226b1ccb1c35714476096740699158c91360377161bd9afb3d3c98b00b6930103d1c186678aa9b05c2e3009b2a4ba897cebe9f6de284c26a29b24a071a216fbe990f7fc3609fdd90b36f233fafa9eec905d4da4f6450921b3392e1d73da22d9f40ddbe7440e938aaf3b4cb499d787c14843d3a0170456569380feb7e3da91f0eb889601bbddef0a56d168a607ca832dd188fc07b49008ab33f9f284264b69378d886d56ab2c101a18222866769f4b434a1d233bb7bc9a3cc77d820e93012e04e72987808a2c8ab1467ce4c46b4216ee1ae3e7081943a0d7c30aa482a63f533c8c93face8cafc9fb711d9a5b3ae38907f5a354d92a64ec98fa79c1af1b5b6ff561c7817b595b6843812d922cc49ecf5f49eb16b41cc385a05ce04acfab668f62f43f213c0007b676989fe514e94c794d6545e4d99dc0d475526d925a602516ae31f3479f87f6e8d5b6b4b224df20a05b23fdd30ca8595fc04049b686e284c0203c10ac461ae22c0edf2f5ddd271be066cf40ca2d0fb76795061f15210712606660b77fefe92eaf51b283eb2a32c6d3c6fb12fb15f2d97277f7da293cec7b1613877868fcde8cb77d78118ce7179cc06bcad03f8fd42870f46106194752d9c3fb55e51693e924c6bb413d20cca027b718dae6e328d3d10528ae0612b0a420d7fedbf29d30c604b6fa9ad8c398aca3b47134c939ed4e74905d082aff15b6efb6cf1e9b267838852e6bfe7dd8d22cdaa1160d578364f4b29cb8f3beea711110abe778efdefd431f639d9e35f8ed749839f5b974ecd9fb9fe5075d3cfd1a2e71332e54ffe4acba9bb2e5b42cb63b26c5b93a5f26ea8dd2befea36340ba7ad5219ff46b7d80d962c6a28668f96b320940bfab258e85ec68bbff6a739567459bc96f401365883fd1241b765248516df6db9ffe06529fa70cb58427ed71ecba2f206a1a3c7ae25770d8cfdfca6613aec3134f1c2cd078599872acdb7e6a4ea8417df1755d6981f5605c03613cdc3d05ea661a29890a88c4d6db0c6917718436954dd445bc3fc46560b3af522cbc921b002dcf77a2e86ed89704276f4040634dbdaf253648eb9ebf41cb5a116b598d133e6b99f59bf4f03b20bae6156d981f3c93985bd8e8d7d4aa1a468afed655140d70adc14a43c68acb5ea465d0e7b73b15b23b720f6505f962ccb395f064d1e1bc29559b0c153b506f8174592432019e898e95156642f6f79698be1fb0b57763b8f37793d8a9f69fac9e67dcba58956a3daa7a1f3afc92738c2264f76c9b3119ebaa18b938afedf1d100c6ecdcd42f06e4e3aa3ddc43c4eab6d49b44edda939eeb53830a928ac106d8410c66b3124f8f5ed9250c785012328dbceb1f5e40430ca40c58f0af0c5460ad7cf03a01615259d4bd2538907c4c6a7bcfd0c3c2017256b6a0e6fb218a5f9a4222afcbadd0e635e996cb174d25e17888f0a5ee6337dc476a237441597134416aea46cd683ebcb32bb1f44e148f73333540f0082640386e7d8fc74fb97d43fb70d182d08dee907170e7ab36f11aa6cef35a3e99666c7ba50643c3688b736e8cfc1d75e5811d6136e86ace8aaa5c189e9c3f6a7357cc47a10b3babd542679c4d35fb4d2a33cd72c5bfbed77a6a98f78ca6711136552aeef65ba9493b7b819d8fec5c076ba973163604d841ca7774646fe9ec690937cec0897004c402665fcd21ef658d6fb29d28742053df853de506dfb0c87850ba2ccde3a7bf56002f9935dc9c1046348423701314e585d33c9d83f466a44d8b97147a912df9f9e9728bc309d03faf5a78232e65eb568360fb0a33f90b70f6a6f655f5f367b3eecf52be7820a0981578b0f286e6f333fca4490b276d18313c2358a381a9553088b1bbadaade0f3f3b2cb8ceab30a8a3754e160c49f4fac930ee1e1e6d233c4cc6eebee221d68d41f254243e94e7bb808c570fb50a62587f5e81483045ee1e8d68212b16b0d058b2f42e0c36407e689085e79cbc77c3c797544cba908230239d7c0603ab4ca31abe06e7e0c8175caafe7d1d505101ce870b1f016eee7e7961b71c60c54eb9b64eb5faf66e2758974348868994d48174246e02b71b62146d46cec7dc62eaf7a67fb538f38b2d666095f3f9824529935e0327a62fb63429ceff516c4fa085142f991994887e92b11ebbb9def870e2f0ef9fd6cbd87d5253bdfea598d22aac9365694bc638908225f522ebd620b797d2809104416e8d0a7c32fd58ef85afeb7cc83d274d595e535b821d778a35eefe152bd62c54ad42f1b343249e0ff0f14339938eae3ea9106e0f194c1e67e9ed389b725a0abd774c36cb2fce195eb9b3dc5d5045c3e422ffb47f8a4a94ff7f65b8733fd878584e40709199930f48cf0633718910815de75544f617ba6181aa6dd6acb333189df1254180a49cbe8f45d98fc211621ea7cd743f805a75150a38b6650f00d891430d9339613453aff5d7e080786b946ddaac0b92d9fb41f986fcd2d23f0f273f2334ef42babf4b8942fe5c2265359c5d55c3be37af983ec507ba50655c82feef18c02d4b93bf90ce124181dc726da1bd390fde4d9ee0b425f45b5961ee6fa16ab556bd3e178768a16d1ff6506ddd485cdae3c39e7bece4c8bbdf80920b16380072c007c094bd5e20a1284712a16ba36d1ba6dfc7cc56c20f0acc12c9f1938bdf5139680d975d5b6eb120297ebf7527305603bf3333ed3941b6ac33a3a38e286807dc4f10c00985cc1eea4999d8b966d3110461a3114ecee0cb3463588c0d308b81486f9841412a1bb97394676914183e19d7576b8af3d208eb6873e85694cff281e4edd125d53fec9e4390d1b1605bc8bb7bac9ae7b26c13f2391e99ef60c0bae564d7da1e8ad94cd0a5636457e8cf574656d6ee5fcda32854551bba172e19cae40229f3deffda8c8d7fba02b7293874d54f9003947a184b891f17f8dee80b5adc0feea915602f0b12e0d91377a9a8cd47f00045ef459d1c9ce0e4664a9f69501b83b25c4b70405e8d7211a88280eb1214755800cb27a2cd03a6647ba3c15e0b240cb9a7646591f0b3ad8ab36bad5360548d73cca335d2e726d0a02f794b7856a9fef2a98469e2da2721d0ec8d750e6779584e88920070003771e074f53710911fe83059228a5a099720279eb96009c78ef715f414b88387e59112d1c8c4cf9da02e10f1552a81f919d3f2cb90d1c594172f619b74d60c9106d16de9d8b46c1e35ae048e27f8bd6dc1615a18a17f87e977638790d2d9f26b2a0a32bd1f281a229c447c966ef67060f9165b253649bb5cb761941c91399fdb19573807c6a46f577ab13105322e919a7157bc1b5cbb21aceb8510dbcaa01d8e16f9f07e900478f8fe67e2e51e1553949a83a916aeaf19b40607651b2a0caa3bc6d500a80e0984fde50c764955a3c6b4489230951e9918cf588f02f0889c56e40b6c3bc21b1b9d0432cc26acc06086a70642f" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "field_bound", + "id": "bundle-oversized-event-field", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a28766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b56876b68b1bf2bc796234e431b8dd276d4b7afaa3281b05fa93c73525e8b9abe1bcf49b0e3524f5040d51a2c1c6524bb6e3e5411b5b75f346f8bfc61e57ee1aa3c3f91f8500b3939157b7d332443523a77a86866518397f5a6266f446034a762cbe82114ba2c20c70f85d17d930c05c3505868e64b63a0276797cf3674691e54d1646c423f51bc8770b0fcc2d548d98b528ac5d915cc5378b8b3d84e347cad3cef36b7c7bddbfce2d4621730ce7d218bb654ce570e164d142ab1a6151cc3aa42221eb58fd6dada1c90ace4d38d26a9baa6101440e13b007ff6076cb0d7586063e77c2693e5e3dbe7da0f42841dfe4744e2891f7cb2c1b30a3bfbcaa76c7408130ff7daa7bdc8d65876672255fb614ac6846835eae7b0a566a20a9a34c45254cccdf18e0d7270cb694274401f3ec5444740659e7db10a862dd20b973c8e49f0ea88e237dffd6536390361cdc59faafdc123e659e6905085e0711fb0a3a31bd74cf733502df42470f6cc9a9af243b32c100338eb01d68a8cb653d6beb1984142e2893a5d77e782492f5644ecf546579bb7d022307e938d05fca679084a0488501e76e444df3f9116ad699c0fdf791e3b6162d5a9d621b1cc95672bee7d43e71ee98c0fbb7476ff2f0255a94255725f86a4192c1174de059eabe48e5ef361e0e74456e4cb2a0d67847a445a697a109161b8ed5b8b1dbc02da8b7856d2015de7ab0d358e14c24d9816756d369a86992d94529b7e24747ddc7de89f8dc853cb1454c34897fb2fcce3ffc8c169bdd626b47976d62771e02a2643ae1100a876118d5ded86ae976fe93bdfc0bc8217917ca9217cef2a2ddebbfd97eab53ab440a35e436442eb70a82a242b799000a5741769349aa4bd3c061fafa62a9ff23d229a02882d53cc15131b2f07c583c1e9883d257c49f3ad58db230ea63558552df0ae86c265f6f9778009c0fa4581074dcbbdecd38b5e4a42cf86151e187ea3be7a7be78c435905054ca5ba72f0a49ea0ccd991ba9f5cda5ee6e74b38e5ed6b6a1e7aeec589d7a3c4c0be08c659731d8387709254e1d214bb362330e0398f0b7fa931c3b08dd5768d5df213cd63cc45d7b3386063a53fec5a120a1258fa8133777398e381e0456ba84a4d8b30f2e6df77009b8dc91d08759a965c898b2176be6cd8997b3dc00de1cdd517ed8efbd859488cd27426059f7362eaf590201c82c88f4845f35dbf670379d62eff10b95537ec577cbe43fd7c8efa49a8bd5c0ef84c32b43fe177c4369b9f537eb3175cc40f23e81824f9d02cb49f0c537deb37dba02d80b81b6d9110e7689517d3a5111eeaf85eb37b37ffde8d9d56af167fcc11f8f2558d79c28fd30e131280b27ed8a513051c68d4b64ee911d94d3bd68e4556dd4bce6e3738a70df653a9d36faeea1ffc62d146957bc3c1cfa62bc51fc763f92669615e862a1d23e53cb686968cb3eeeff0fab3772420661e6b01c8358271f332e95ee7b4be5464847f033264433c650ab04e5c9a4b4c05316a4dfc75d98d37a526d96ebe2ce1628ee21269d11a55611b873e98e95cf4a249cee39d829051db7d0c38c792f12b55746449b83e2eabdd2682f4f3aac9bdf3ef9096a2bea76557362d8cd172adebf8858175fcc97b00cc1ffcca59e30e6fc84c627a71d6fefe09de5f95778624cfcab1195191a1ce741c56ca6c85cc43ff49716124801926dc1a37424e27329f4eb231371f33e7c78eaef4535c5d24d4603a6018ffc1145005a2dd54e2c3e626218645b02a119cc4a14f294893feeec88d13c7ceb99ab122ba879e9955cab169caed6c48db668a4d51eaa6fcb54b0d78b5b1d6044cf90d16efdc4834f711ed87698689ee0bed6406373b0e3fc9cf4b76d9decf013359e87254028c040b19426db6e29b612e03a1dbb41a508c4c913667a5442971c62d6db85bb5be7fdae006184f84a928ff00ab4cd3ff28b10bc1e1544c602d74909194a22c70f059891310f1abe29473d34db10ed614295ca58c8c34c13c0501d6c7f15bcf4aef8fc41eb22010ab38e1c29a6d2c91bdb6cddf82a2683475d42d0ff0a2f03f00039457508d95daf8f187f02a736c0610612e764a8069cf8dcaa6c0eee11628107da85ae0750e82a328ef2e73fdbe2fa2095cadc11a8cb9d79bb616c0268247d7102120ba5ff3b5dc8df99590a792a93c6c4ecc63dc60e37fee6460511a8acc90999274288c698b612d705eb03b138d448ecac9548c2abe1346bf97da5ba361e8d19f4d65970e3fa157c96f36a17aa7c0e0582da24bd2e071bf8f5740bdd6c672e4cfde07c41c62babf35fe601a5c5321e0f4461ecc56a040ee0d8b211d230c867656ed222a6ba352e1153c2cde0a5b3ab585ec937ee2b20a37f33b61ab8d2ab447a161faa24727315c9855bfbefafdec3cf8bb8dacbe8c035a550dc2a278834496d63382c18c502d81c3bc5967089bf4ba5e57825503fb730177c0269363989ada5bcf03119f7eafb046faa8f9aa5fcc439eb7831d320fb8871fe4cb6e22203bc02aaae67c9d53ea343dfc4d9c3057aa8b1a653d9dd606e51a7d3e3185704db742b6779b98173d510690dd382e29f8ce8ce63a7512fb2295ea269373e5bf22206e8eb143adcd9d07b38790463509d2ff4cdc93e1c5bb2ec5129e700e496a4a80941e4592a231535cc717364787a3e7d3787791b103cb5b16b3a6a921490b9d4247444d55846bc1d76d92012747be5cea2d4aec1533437afd7dcda96b61917ce2e75204e9c05381a113ae0bd1921d09db51ead81ad121c5890451e83ba3dc29165e19a4e5f77e767f9d10af92c9dae7ac4db28e67584a903917604ebaa38a6d9e0e953162029a2137ae8937423674a75dd1a98db5867c4b8c58875112e770b288e6cbdaaa11cdf6579fa1296e612fd753b00595e52bda6dd3bab04390e843ed015fb97c7661468c9094f0306c0d3b0ca512660ed70e40e82e2fa95eca0103fba89eb442d087d6e27a4a4bf212c4fadc1e1096b01cc3926fccb501964fdd814b4322b413dedc67b13f10100470ca14e96d31f8231f68c70a1088a53739c611b64a9b3387a746ce2a70dcbc90d9de858f69d33bf8fb9029dbc80b15a3c799b960a8ca2752e8d79ba4e3076b2ff91925a350323b4992fb8e0e52f91b8c48f5a63209df1603d35a98a6cba8359c43057da00771d56cceb6032d60f1d915e4a4d02e0fbb9b08d4dc1c96a8fe4a56209e38db5efd07c8ff9d17e5b26f239853d540173b818121223d6ef13ced1d79922e4f868521e7763069f57c708add8b5885c801b53d593388015eeb912e60e4a6ad1a44f58dd76fdbe24f29d192bb829794a30fefce67f5f148de8b3d285745b23d29573b145e5a8ff6a9270463fe2979be4ccbca0678bb0354524509b67b8300ccfca7ba1ca0194a6ff5a2280bc7590e9b9f865f443e6225d0858ffe0db5a2fa3298354547e120834d44a6d675db914d36671b7096132dfd43ba4ae27135da83706194e56d96520dc4b03ff6d9a2beaa58d892a4c8acab676d5bd01af7339347f352efcdfd2494a6c1c2cfa49cc5911e3a187554857e667aa019115eb7425fe12e8ab29f2a28b675cf7cf33d7b3ce22bd0cbb7617be457655427a4a180709a4822a531611eb5d019164acbf3614d18b0c47bcf50094f123817cbc5f2d86e0b16cffefa2440db9575863c11cff54f6869d8523b7aa12ebdb041ffe386c393e8237edfd59e2f66a42a6f196ba2a3f94dffcedb1d09da176ca2a33da8dbfed7a473f70849f281a67634dab690920164acc10ca62aa0763f1f85db603d6d375653aa4406584b4a48dedbf11f88b262ec6742dc9501dee6fa3e6f85cdc60a41a8753eae811e881b7e82dce1e4bbb4d35734695ea7cacf4f0031d3a4b0fb5d7be865a509087291183cde5f10257109d72abe81f2d1299ad1d098ed001518487ba954fda1cfd7ad2525989ffe109b24e888a1a4f7c32335fb7d9a11fea02e53e802035818ff38eba39a89f99b9eda0f53afacd9c8976ff18ff8504de3d46f23331decb03d0f331cf8f453d024d2eae2619965bf67d823d24b84dd357da9f814488bdb5c07124bdf66a3a54d3a8fea67d8857bf1f2140fdae48d7561fb4bcfc4c179af732617403e6e2036a7862af33e71da5ffca6859d5177c4d250215dc167d86295d606cd1977d23a3e2a33775ade9262c53a344c44cdcba882e079d922869419e572c64c8036f6fe4caf84e33c9b7435a6a3f370aad62b78799f5ee250054980156f03fce12ddca57fd2866cef7201ce0e178116c9b465d4e3687ba8140668f407b10a13532a8a9487f2fb8629d94a45e802ff024949d1d22d026c5805ead980aa82dcb10b3a89cd8952666ef25c4990a4e2f89c3b9b99de7bb97e9c09625ccc9587d9b72b84c0f95bc080c90b0beecd1fbe439aff5635f6fede7222def73ab4d68be4b71079b25e56f26a7b1a755de896f691c078a66677aaa974321f3b60740717d641e81f6a17d95435740e07105ef29514c9a5b9a4531e1d7d9c592573c8cebe4fc87964265edf212f2400d1d298261bee9ebe6533e4a55266774290836ca5a8abf000b047ab5a391ab0651469cdb1d36d6932ae1f6cb529df5d5fad8a64224f313b0d64103ae4977c8087f1c20cf191ba57dd12a7ba7028607cf443864421561dee399d1e39d01db8511bf145d8a9ab059c3e002f950ff909ead5b0f58de48811e574ff5dd14b9a51de8dee75d8b10a7c4650db9aa9361f846ec526052a5b303cb768151afb73c0627272994ee41a57b751862265b022ace9617bdcdb6b3481d5a479451dc2a627f1ff22b53c2f784a7553414eb565edf7ec59ef0a8c5aa6b925420ee359e9bdd292b927fc415aa8be9c2fd741942a127a7f96ad83c1963904c6594048c79e16932a14aa644dec1c787188a0f05fa43bfe083347cebb6a50a80614fab1f3ecbd772b2911d67dd4d9a3d9f702a79dc4649442" + }, + { + "category": "nonfinite_sensor", + "entrypoint": "bundle", + "expected_failure": "nonfinite_float", + "id": "bundle-nonfinite-sensor", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a29766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52be37e43dab920446abfc985b78bb56c0ff753d69b56633f9e111a7d15aa093aed1c82c85c0d011d5deee5caac115f4e041f6462b8e597b6ee7f03d3bfcb33a9a6d884e64b4dd470ec975d3d2a5761d234f94ca2548008b31a743c289ecf4ba4c5150fb82c2f7c0d5063922a384299704af0f7d474b92301b78c4ebb99caab99d85455fc6a6a57d855da1fd01ad6a1fe2c191c4f0bffe320196cdd98821f4a9d682cfe43fdf2e8455865e53c27dd8bd7d64f4cb054604b8999d7bdc172cad84f66603b33fa68bf27e4aafccaf152b623bd43ae7357ca589aee7df1a0c176407eadd92477fdd0c02cb1c8340bc2f1c1b662270d96ffa7d876538debfd89a4a5f30a3fb6bb0b7b5e91245686f54c82e57dfd4e2c229285fef9d1fb5c93f3143268fa0e0ad8ed8195bf1ce7bfede4e1d136ec8b8aa8dabc6adcadb58e1f507aef6f02e487ce893a5c21b7992a4d3dd0304a2558775273c76256e4f6f1d657eabfdb8e5cab00b325c85b6102315fcd079b4af2cb0ec5cb90b4333211071180e1ea49c9771a3a4e7cf8a304c06a2c603327b75665aa302f2480b0d887005da998e8f3fc9cf65ef1fbc893ac9ca761d0e6aad7e1d824af368ff29c5ce31b72bd54a82498d39e28e8801370bc4883537ebebc7da76ec442b32c321c99eaea879a86a16a9f8d1b62e8c9c40331e5a0e516a0c730ac19ed0ba4aa063f230dd5e7e9273736b33a2a341eff9ab269b4e36d20edc6931852238348d2ac5244a23fa6faeb22035b839289753f278134d5ae23c3c5b01d7416cbfbd8a828ac9af90c5c38b8285614182a6f6fa599efb2820b336d57ba14bb51e4bdf8585aa922d65efa3d92f41175549eaa1c1ab58b5aef32f6cbb2af5302f7d9723ebe72b2c1924d7917b0a1dae1fbb89edd2acb569ffc08f4435d186a446d46d71c87dfa6b8d31c015463cbbcb5909c19c22448651eb04c00f9e637f9bb60fdc7fea490c65505ea357da7f8c81cb937525dc8c008121c395c7f2c275287a34fc477635f8c46c116f345c82d8737de3f177eea3c8eb3e8464446dbbe424939f96c63c5bc0a47f49abd470dab155a5333aed0082afdf0cde1ef0b003dcaee6343c4c26dfe9d70d9d83a4c773d6b5c21968620fbb41aeca1717cc9924906abae62872b0de93e505ba65e07d94012893cdaa23f91dc2d6236b4f6df4f7a7dc90b7c11f68b8bf6370692b34b978d710078195e4376e3ae44866a90b59e9085dbc9c1e848b215ffc5af46681447bc1c04e7698d6d0ae0a8f55754e2406bd00cb86853bae58f5b8a4481384c6549eae0aea15a44d8f5245e0cbca5f86c12436c3c52af8d769818c7cf0590d807b139e5debdbb6e5d01c5293da8d0c82968a9a13191befbe57d7a5ab350bc2bff9536f529f9a8dc670f4f4994772abd2e6db7985df07adcc3c9772aafa7a1bd2cc011b3740e88e573e5fbbf83de119abaae8cbf6114130cb0a2f285aa93b3fc70067d73e53ca72cd9117794fc70bd5d4e0c1e8ea8bb7e2672a419ae09dc253ddfe10cbd76d7e7daa425f341d104b1c1ef67278b2b61e4e9d5d1300113e800680548676022ee100b1d0f24ae87121e390990030cb8cfd1d12bfc7951a7d0a296c6e8a09312d68240731b398a508ebdc0e71bd2a08527a951f21e45459c86837bcaa4985da02b5ed7302c4a02ab165b4735e5343d5a989ccaa12969ea13f62268e801984fa9faa61a6178965c2b11b0e69810292ddab96c6f7cf361875ea94af5afe2d769c58a245972d75a68aa473d8f64ab09c04ad3f29c4284c4be192214993a3c0331254db6df0394fb49a677556d915538720b5f7381a5d334cc2086d8c878492db3d3b71b8cb098cb868b11127dfa503363f36d9c457cb27eb18982db0bba8736758f674f251c1959e66ca15d8c4c6465181a170b5185575af2bd26bd8a67c4409c30e92d34e9f2f2259c04f5f8a12f76dde86a3e3b53b26be2d3b8fbae16891dd44da865109cd71e250d22eef5cfcf640a3c7fc47a6e078c2e40d55a94e918d8c6367c24d8e8fea1c8fb52d5f20e24860011ce59f51e3520a3a16bebdbfbf7874ee3151ea8437766e1cb543cc53889c06e9b8e61447b89258ce47306919cd6418ec1bb7973871390e8a8f4b03d2b3f1e49a2ca347ad8bb30f263ee75fe85f5039cd050400c6d662b25c97967ebef3de5a2735e49558c5dc8df1f02ebae77396fca2f4c49a8601e59e3bf8fcd35d3c970c68cc1d84b5c13fc8dcef8eec7b0d0bf9e08b7485b24c826c18a8ce621eb0873dccd13d403cc45c5aa4448403b53be024326c65789c187ab85d55ce99dcf5cd5f0cd55fa1f44417c24b9174e2e91b1ed02e8d86a6aba7a1448ffc8c944d3554d1a5aff2de4eab55afd5b27cb5abb4dcbd5ff08a570f8021839ace563495cf05aa655fed215e1948b79c2f03d0b56c05ed5efb30ce1077a1929b942ef20f09c7a8027776470c2f678987be46866eaa764b89ea37ea2fbd79d7eaaf2109aad2a09ee2bc14781f3906b55917c43978796eaeb976994dfdc61da7fb6f51cb9ae14a44e7c8bbc6df44d6432caa47a9226fe0eec522e85bc98c29d0eee4b9de8db2f69b206946aa5622667e1a27ec53b8bff52b007747b47f4f929270f886b48032e471042915be89a1e2219a23e297c3590edb78b134c08f732e8683f17438084beb0b2a05a80e7cde02c961cd246eb014e184eb3955a54349ba652ae01f40b136c7d726260e75503141bf012a9b3a7aef59a9f06872e90402e8c18323669541119b5ee819774b85dbd27e5e45f31f285d2a9d59e6ba4d19f16152680bfae5c5118d7cf70ca33f9d25fe125fa897bc76778ca0660ec1546516b1fff59ff5d8d20023b85f589160e3c138b83ea6ff0fe57312cdd0fbbc9351f6e1b04654c2bcc418c8ff9283c719f2ba18f8338c342d12a3469ba1615e088b15c464cec3f7215b4ebdbb5a08d43d120b7f0383cb4e83fe11784c4f4ccb6b6c06a90138b4276c986966068ae868204c024d6e52e02e7c6b35a140838521f3b52a328caa8c9895db8b11dbbcd0b0479152d934677b50a44ef6b13200499092f174eadd893bb22d52166093327b99edf0ed5e5628d85373cf56801f5adeec24e1ba0df49ded27ceee4f21595108ea2fdbdb0050aa622f910310776b8f2b5f3e47bff9700b5b807719fca3c4ffa5bd1a0bbe35b5149a83f0dacf77a078f76bc57d7eef1a32499e646d4acda090122085d728dd37b4b098d9535e9e529a3e5707533ed3599ab7a06adc68137003cc8cac671018040aebfd81c25d0332b22f5196b9548ccb46dfa95c9a217e09da7ec6612edfb8eb1ed82c340e41037fc4705e2944b5c12a53c3b8d81ac832e8cfb8a88717ce70a77fae2b5c379fe21d7fbffc18edab030ce5dcb27a0678f26c2cffcac654f660f2bbb36e826f0fe24f6bd4926909b450f010c3c6a66cb25e9c51f8c46f99e8c3b16aaaeb18774a71dcf2279fe098fa4ecc17fdccdfff9cd68a554213ab5237b7f4837bfb241219c2783ce2d2bf0ec6769fb1c8865293f5e2e22365805849c61c1157f3688a9f48a962570bb7c34d3e4507abce5243f8e4ebe47e334ba0a30f114e8d762ab8db781527cbed10bedd281ba0ae1b0b37b2f9fd44d0097bb782c9fc69efeef8d3a325f7ec48c6273b2b8a2f91e532622937e4c4ef7565c646055cd9181b91b8c8edbc22af52af20d7e81198db8a7230a48459d700e60f8c1c03104aa4f97678d11d92f488ab26f93777816ae50b27a8af849cbabd19beb78d3a0063a50f09883f72fe6a367d38369ff1786f9f0e15e8ff8dbab64690f817174fa71d9120af25813f67567c696cc1e9ffaa410ca0bf0f79f33b128209ac3313593c9a59ece96cb876f02ee5ac10ef23e88fca98923b3efb2ec2ef1d7ab976b39d254dc1875dd2c636a28861fda1c4bba66040832293691d7540f69df4c42e2a6e97fb1ec445adca6992ea5d3d03e8ab4e28b3faec860f0a6eadfcdf8f4500bad88861a129e91b7810d574078af924a7be8cf4f8f49aa80f568a3d1a283496bce0965cb3c43fc97b1c45d747e28bb3a064b992b01a4b6eac242a9e272e303c33e7e7850110b1cf1b1ec6920ce4da236208df1a794cae9d9daf94e1aedaecbb3657cc00807573146a0e673ca0b3ce3526c69dd22e9b00493cb387de32d35a0a7c50e967ad6d872a107f7ff9cd266b0ce8401cec7c5143265a286f1e4a4aa06be4106cf062e572e6126a088bdfc291dae864b238ff6200791caf71c55ef706ac35729c9e0342222faebc488da7797d87668ba2d421d1fd9f008ac051a23b3131ecbeb1ed712990033763fa141f1979f3dc81af61e19be973f5b9ddfd1ee99d81ea8891463c7765840e9667b5cb0026f29723b0f094528244ac695ce3ec805ee9c7386d33c7fb7c3ef47e89b78d8214a969ab782e4da7595a9eb2fc2f6" + }, + { + "category": "float_grammar", + "entrypoint": "bundle", + "expected_failure": "empty_float", + "id": "bundle-empty-sensor-float", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2a766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bcec22dfa106594c46603eb3a65da04b992c665cefd1500be5081e143fb955540ca6bd163ef468dbfd0c45c286d933f25bd23555797652210916a4bde18954938b134ccbe2ca2ba17fde5ab18797b73c61606e984c24a1231f959de9939ed17226c807074747390afc8cfbedd82ec4d2f6355dbf3131f06fb24156ecc0d25b34bfe442d3187ef552c1fa32e5256694981a18a1dfc6c3c7dec783e39bbba03bfb4d082ac445b99197e01cea24e717d01fbd8b594c8dfa4890ea1930f2cded1fbd94435390ebf52b1904054fcb0fb41d15d640868fedfca807a3939ef2030908bdb805379401b4da2171922b20cfc2b9e842f8621a51b6b7efad626bedef624e70658f3ffc8046fb5d50f71ef3b92877b130f47ba414dc04304920de9752f306419600bad70693d6ece99272614c6786287c0f7e9fbaea85dd24de97ef6e1f1c780b518ea198c40025196a4d16dfacc8febd21ba3c5b6d46efa795cbefea61e416105cf73ae18235237535b16cc11a17d1a92b553d4e9052cf26dbeb2c6018a70b264c7033b5f35f9057d311608357123356ac00d0ca98bf7a3933b47b60578189a14edb5172d6bc9fa6ab9b910f42548109b3b278e1849585c0f9cfa878ca459a9d2bed9a9a37416f8562465f8991ac6df6b3978f120fc1bfacdc07d37bec34312c8df03403ed95de38a29d9d5bed2f8f2fd5183c62a47f1e773a6ab17a293ff81d65fad4067ed0a010fc6d96ba2cb3ba98759fc65bf1cd52731256e9da5281269f7584d8caa22507e0885e1e6da8a6cd0ade2137f3e69e972d23a9b4aa835ec685fe6c60c1b6826d32a7026205ee68550357946b9b94bdcaeeeece2c78f8b99f674dd0ea4068ac87e942e0f4758413fbc04cd44a4df99673cae7b3b04bcac66f5b134a69829db7022b5cefe3efddde477ef85b78df88b6ead21b123d830a770eeb43d3216ba03cc722be6a8c129e48d43dbda2df0d0aa2659969f2ec402d2e5d744027bb60c5d111f7ed4970177db30434058820882fa4c5aaedf6f4841f85ccd04176d929b6222f99fbdfd5b7e4a0254dffb2597bf3e43a68c115c9b9edc8bee449d6dd862e46697482049faf489bd04926d9b7416eabc3bbd2698b0f723cc06def06516fa2aff721560ff12cf9c05db34d46d42e669182eafe82974c4237b0b7622c0aeeac430aea603638fa1e28d9fd15abd0d89032c0f66032f574fdf687e34b85d5f3e762aa49cd77cb744b3bf06edd2fb3af4fa4831a5a8dae3a369c9f60f27c1bc9a3c370ce292a64b774d3589ca374ffbb4858ee27d4e0150af3af648620dbed446ee2d649045a716269cea7e16dff9061aad5b9d980a859742977844317a16e418e32ee57c6aef2c00a1346cf6bb6b32b248cc8d169f844f79c4e12b9e7c5745f97a71ee2e0e6780143637adf5b64d9e1cae0791f4cddbdc3ace72989da7b365e963568e8a65018b19ea13a700fe92379f6aac465262ec2ff7d9d25485d1fca0613a941d9cbae79a4cb07f732d2fb90b3c1943fa689e8aa404375bc145122798486902e4b9e3b66bcb5639b865819ecf1c4ef2e2565bf9072e53b4cddfebe1e8060ff60cbee9d2eada49ed159256f941df81216a05b072e60a5e2a74113277a9b790411c9e6d83a5831367d656f9156c4362665227bf9fb853c6f7a7677860a2848d55803882284f66212284aa72f29c9a8b32ef13846998decf020a630a64fdc888ed8443f87ea69190b457ad973dc8903de7573c05b1b499048048b0e9d02ceb0291b10bbd97d417d68f5b1e4177cb6b3ad4e0d34223402d4e29b8dd21c9c0fd8586f8b7613cb51ea2510fde369aa0e7d5ebf90bb532ed04eb5cefc1fb7148d4b4074336ec39c31a7a73231166db81e0b9f471c7c86263883ff573143781ac67cb58b43ff979d09f967b733655ea64fa9b1881aefe32d9157b6815c6fecc6d4dfd182d37ae974110093b24114d161e66b18812e73030384b7520395dc66db593b083fb4d5fa4af52a4db6fffc6a84d20fcab65d9b565419b332a4951afaf146e87982fc8c2ee41f5743434147eefda3930aa5478ecb9a3537462a90355fb04f060c4db006acee45bd23468a653cac3c3cdbe5f7f6f1c7d930c5e431cbe6128ae14c506d0fa04707c6436419817ce950861dbf2372c5dfb929ff8bdee7f822e48bc8b5d5946609546e1ce9d4af4f542eb3919f9fe8866310880280b0200fdcb2108e1b3af9074418d1baa1977631527285ae3e2568d6a425d2ff13ffcc0d0b0bfbdb539a5738bdec84a69ea4f4367af261e5803adf878886c9d8894cc3b5b0e0b34d4efc2e3dd692f76888896d28fa9b90f1421868de85cf3cbe757eb3367ec7405c93c525b0facb39abd227f9aa72b6db252b3915042d8ed64cc77c74dd290ac7d567d9f922066c67af9f250fd39abd3a80ef7ca634a17ec604d2038599a43825f6d478a8005c46e4b737368add3f04df9cc0708e3c5c25e2d5d8d9a02b19d1f8d0c376b60c6eb5feb7ff5a9c4c766be7d6f0086e05e901a957c70c01b462ce093d0780421b4c49d924729a3f4c2c31ab69b33150bba51677da8ff3eee2aa6fc38f4e931411cafebef0656c442f739868639c913d6df98143875ea31cd6a0d1b19f976254f1877695c5d28a33abfa8f7b76a6a68b65ab200294a9d9a03f398f96027705866d5ece7b5c873eea5d51821fa11f5b1fef57cdc9d7474864c2901971dd69710952bc76996092579d294ec1a1705cd39de645fd5c9a27ae50f23d3aa42e568c5e5a0690a5247d1fc4bd0b23f80c21d9c990446271d06357c1bf0e93381fc07db84b76ba19982460cc0020cc855b3fef77c7b86cc52b21dc5f64028384843a9c03c52c93f1c392074d1879010707c6fcfa761c1ae05872757d641b824224afe98c6dbc5519526822425204df648fd7d026791bc803db344c8e1b9b3f9ac69ad6a8f604165db077218025b7098502ce9b0f7185e38dcb261eaf363cf2ffc81563c21c08aadd130c12a92f5fe965cc1f0a01f856780ff8633c7fe22938332875c6c929917a6315e8409c03880845b19352c7f0bf54acf8153518622d8f2e33f9ec4cd9d4304defe1f97154cc9889ef9172a0cdafb3c97bc67d31c3df96481070f520da2c6b83d0df4c7921d5cc73fc14f0f4c7c7bb2ee7e847c9ee9e71aed56803ef645de4b8effeeee45437e416c1082ad7e2c50cd8c7843cd8847326936c8a25555fc1a1ff38380c11a39affdf74d355ce0e29898ee8821bfb70034f5143c7887d932efe5ff334002c7a6677f0519e1234214557323dbf119bf07f45292f1cd0dc296785152e7f48e0d706cbf5ee8fd0b69d6650a1214f74fead208fcd0eee7681a544e9449126b117d24ea6205fca597b64f3ca37c72e04e878c8424cefda8a008e0b43c6560f388f8f7d13db0285943c8d1a1be47bfe4df5b8bb3477263960d728e372d1bfb776b76ac35f7517dc40be334ad61f430485a5011eb5e7fa62e5aae352632cfefd265b3842322e58a684623d973b51bd25b42684c0fd1cfd7bec2e52ec60c3eeac070be386958203c187c7fce22494c9e15d8a4e3c2384c7022532bbe2b2176449f33f120a795ec799776e8fa0d788d6147ac656f6ba878e275d50dcc8654d95f59ac4d9e9790289bd18bb127068feb94bf2e5e8b23aba5de2c24854c16b0e8964c75a37bf4f763ab559c72723785a91bcf30c029f7070e4a226f6eda8669d2710fcc634f551279228405cdc1ef4ac0ce42105a7150d1d30486756280f45495f406b1c0e969c26a0a727d6ce3f33ab01d910517a462936b828bae70bc2ce975150e9a666e23b594e873696d92abaf09bf9ea74f27ff822995bc26b2d928a0915a0bee651d284bcc8a4f9e9b75171ae34e13660bc27da83cca2d78693b85b8685f36c6a9b390a7ad07341af1ae2605150dd01b930de2150e21d2f5a1fc4f758f289680c0c4724aa0012a4c7d4dc8dcaecc7bc9446d5673b4f6775fa7b007becbe2d642b64a27d4b13e913ecc84374a368e36b39108247ec19a173846711f55a036bdbdb0658d137419a80440b8f6e64881be243d587c690a92d312cb9ce7cc9b8573a951765a0a72d693e2b539746ffcb55e625f7b4471d8b24aedae1b63af58fd7450df13d1e04ce2fe05394cfc4a07bf8f27b35b4b9f706a81193a95cab5a9cf6b6f0277b506d7d313d8966e4ffeb58e3cffda7a3e6e3f8cc0f697054a957115e6b6f57fec309ff245cf7623849160f639edde2a767bb2034028f9a68e06ee6d26c83108462219301ba02261e9d3542dc755e2f71798732b3c3e3064f08a037cb1895adeecf72ac7a55bf4302ada80b40bb66bcef2b608e720fd37beb2322cfc6efe9d6f5bbb600f725f79a5af7f352570c7c420219eff8a58021832fb51ead4e0157012c604a6424fb637cc6a0c71c9087bd451ae6e1d6fb806a8de" + }, + { + "category": "float_grammar", + "entrypoint": "bundle", + "expected_failure": "whitespace_float", + "id": "bundle-whitespace-sensor-float", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b1ef015ed973a794093ae8bac525ac0657f406c83ea17bd3704420c7ccd68e5f7671f90f5cb9c7e1b5556366164717a6610c8d54d8477cd9d599ac92f59c1cd0db5ec3afa4d4c3730b1a94a7d5d2b1856ddae39b0191456bd359da4719fcbc02e2d1ab54cf0df586e6a87476a092d61abf08cbdd9104259c1eb9301e53c80dcb487e71cae70145fbb8b248f73425bf70cf1a31888572cecf7da25c789ef9bd558ddb7ea937a188f7684ca6cbb9ef13232f6c4563ff1ab50536d482a5007b4452a8072e2861e3820e2438914e0c855232c80ca29cb7220b8ab8c5cb65e68998a23ee19b789e6a5485681ec8b5afefa769b6ff6a3a9a3adc5a02e090890305c2da8dab1825903718352247a01a7094ee5845ec2ce1efe865752b6c7c3c5a679f1feb1b7ee2638a34882e76075733a97f5b45184aeb22d15b7292837469d82bcafe425bcf0a284a1fe827be358679f7a528f0ef8a8bb65bab6784768b8c727f1432ef4fc3e44d1232d27b3337234df3d75e0dc66b3f2b885dbd4b3e2ad6ed8514fd0856cc264ba0a35e28660dba9c6f3c04892001654af2642cb7c27239b403324aefacd54a64194c8a25a71111b33f50ce46a291085fb2003287651af33ec13d2986234645742f1a5f137ad4b273196a025cdd086b3dcae46919f1d2bc14692503730f404d87ed3fb5de2af233fb47ede99443af33bbbba9aee644012c2e92fbfd0e4843854eaeff3d88eb41ef4dc7eb203cc295fad8484d2f9ee0f9deefa5d9ad5a6928a4d0a040776c5513bcb914c626a15b0f0c37d1b876e79f98edc5ed21cea23800037fd2e0630c080b6bb6fc7c1adec31b0b6ccd34965356746e4f6770c24621060545eae2f9ccf1a7388fda0ff76f1fcda5e35e53edfdd56d4a3eac984a5075bb86f331577a551dc91eabfe5cd1aefd9423786bd4d3d00360bbfe798b70a9a15bbf6d9922a599be07b902f10fa8c05bd3bfd3cfbbe7cf9973e52e180ed480ee276e7cb7022eeb0a281ff8338ae7e51c417a1046a06c702ebc467ddff1afb91ad4476ffefd8b3d6199b3fbec72cafc7a97b6a20b490de5f1e35e06fb62db369df2f0e2b435c7d917091418bae71e340396db355ca16fb8c615b7ef4b4965e66b738694894928cfab6faa68e7536617c1c8e2c8344d69fb3a5c8af33557702ef68e58cca26d2f6c5f843f736a129979333ecc546eef2bbbe8e0aa0f44ffb3d3ca1c712e2d7e4f0737a92208c825af29a1b94c9b71b846ecb467b6ed1362c76c5d26c8c94fc6dec289fa47de680a276d125ad8559c82379b737e966e0fa4fb44efa99a0588b9c02d10cae39ff8bae9e2471cc0df19f4fd9941b64b8af91ba1bdac7cf7585e61a4abe0a30c43c1dbd0781f534fd95c559b4f9dbe490dac48daf2997dec5eabd87d2c93726e89aae5ef7b520592583805252431a870186ccb1109d97ab8fd34ab4f6b9a03c27a94a625f251ec62a1b00b28dec6d74e1f8ec26a0aedcb4f78091cdd2eacda9240c2b90d1a30f12f03d7e15ea83fe5495de461c21168c2f5c677f2fe3929bba842c3213f61c3735c4344119ec53633e14beee085b41bd3a6c47d31c8c2957336be4498f18a3e198adb81949d04c6344a75d1ee0f2151de175bc0bc74fb3b6051485877458c1088cd4ff6c46aa119b370928fb2584ba218dd5084aa0df1fb8ce901e02f5dc72774715fecb87a87d0c9cc87f0b64b2cad7eaa3fc6802e536fa7eb928eb75b07d9c007ec3f657fe874d5b6ff4ed1263834e79821fc7ff79e7a6ad351f4f6807ae6c4a183502fb092b821e05bb0ac4c1b4f9a56676e76f3de146ae8d999128d554a0e2177a17fa0132637b2a7db3079b13f678a516112ea2f08c64223233cb3a9fec61e57cafb2af9010b613d3ba0dcb7fa9a535c82b699ab37ad02f0ea2a8becc152e38f92c4a88d81a3336aadb1532b9ecec7319cff64476ac1edff20348c516c8071d2e6c28ce3645e30c0c0a62c6ce970ee590896e78e86892d7e611ba2a7de1ce7dc9d5b8fa88c4ebd81ad97cbcc27739512e576bcdea8caafe64ade2b5bffc85ad625cbb19d5ded1d1279ad8d6fe43624aff72980704ba1b44c7472aa1b9290ce85afe4ccf7ad8bcabaccfb77ef2ad90563d2c9c539c4a6c2d2cfe3f240982aad3055ae4e3d91fccc36a61683607e843f60016ebc7694076386877dd00e67102b5b95fe16d11948a5a0c6280c88c11e6132987cf52af0b1330e64f3dfd1fbca81557658a4d7b851298651036a85c8bb76b37062d6a7897ff8ca5f1cc808e40fa2ba53bccdd3cbef14a8768052de27e95fe1b361c109d3b6c85167307fe6bd321b7d1fab90b3e1e64722acc37caa473ee95a43f097f5a1f8263e3da35a1cbdc8f9bd6cc7239bf6a16fd8968fb74372ffc9c769b2d2b67acd9bf7a04ed918d395e65b0f1f4b0d06b0181baad9423440378c5deb34f0ba8800877ef840164050eb7ea07d3b6a304299a73fe8a3f5b8cf635aa0276b61f9280617e2c71b5ca9e6a848526940cacad754144927c274ea7f0a377c5f974ab885785aeee4104511151a5b8f263cb68ccf722117a389fadac0f03d770753e9be41e9340e37731a9532d7d92627c6300980b58f13438292049493f94f7f130e5b27053023a3b9d8a4848b13d810475944ba0e0d03dc8699058e7ebda134c2209e6a0d4ed1743c9a04502236ab3bd63a05f11976f4e49fe793d97721e1370fbef0193b7c7f69389fede535abebfa13fdd0e0afd8e361cf9b1d111c919e5a31295e2dfdd73ea7d8af854a99ab356cce10d4b407c0b9b3464fa4132839b00363b8bef8b6842ebacf2e8647c8016659f63cec424d15bc918db89aea146c3aba22acaaaa022726397e6c653f0f42645faf0c3109555306a6eb23ec134f9e3e82cdcb062dfd154bc80a4931d0f204060fbe7905633be7a17eb71f5b122d6deb3123b232fe041143d469b2f1de9d1f2ab47beac9266223af1f35bb1ba950e90b3427f5b4d0f075cdd9c4b2e5cf6fb2bf120bb14e743b658c2acbe4ae503727a4fa0b2524d279726aa76187f18c1b1cf7859c4c3c0e19daab566399ef87314de2613349ea9762f879578033fd0c6b2334ba94120ea95cc4aeda934dd0b1bbee39d23e8629c2be689dd59b3a11faf5a101473708298a70c9ec3a7584ae2023b4d9191c501c441138f834566676ae0af0f3ddaf56e40becf192286f0c37dce669029600ae3591780b1e57ab4608353703d2babeff417d54700fd5fecebe119a8ddadacba509dc1c0f37e44d89a17fd6b2549ec38ac78497d325c9cb12003f493c69f4fc58c28c179e560d8d4093d46a7f6b3d5f701fe0aa313370b08cfaa031faaada9034bbd2d08141d3955ebd2648c6ccc1b514b89eb988cb08432dbfd7c7baf2a71c73a9fb954c555f0cb80cd38372c6453c1cf1edceee0b3a773b74da5defa4426980ca00b1b604087449beb780b653768ce0ff1231a93d673e04aa29fc1c33b73a3535ce331684483df7109ba1e9e0faa52eaab95aefd6d4ca5afc00b917e4d63ecd33f013af43f493fbed4aa9192c41f5551a1ee314e751ff2f7a9d4a32f554f0fbc6b29db66298c71ff47cdad2890a2c65dc10abe4f3cc8a76f0a9dfc9948ba8575395192da0cb7fa2bc24a4422aedbc6222baa29838f354b15f4d9c63af347c9e7bac3911b82a27feed046e4d535f26bf71d004175d577c76c15d682bd29734301bd8df8e8e924132a660ecf7e43d6bc16ae383dafaa3b57330c5e27548b187a12241ae72ba058a0a1f4fd8f97e682bbb7929f99dc22710efaccc5bfadaa8d05ce61f6be11ca30e4bd0dcf8d21ba3af040283a465e0e1e1fc235316dca04b4e12cb4d68a41fa3ee62f1c17d38933c914508b2f9836b3f9e97252bedf6ea2e91c95d026293c0df555cbd17c477fc7e01d2db147a452716bd5479175d0d9eed407a8d5172bb9353d664036796839c55483fe3ec9e32bb4dd061ad3bdf7fff6da28ee01d14091975c456b11a1353c8171b19971befde901bdfa2078e7978cf2f7f75644e0f921b0256a3a8a59efa046f5f8207b9ce003378e9803b41949abd49597570afbdf7c9295a9a00017c769710d2951a1faba3a4a0e0a5b977207c6653d718dc45a1b234f34c05703f1cabd40fb0d0ee51a4c9738f0d2493a0ababbb332132f19c697c9cb44c88fadbb5e724468276210b5a0ebb12dc51ad6350eb72e9c0a1f322fd63c636971326f993dd7ecd51795a0bf2d004d94ddd9052bc30fb8a03925f93fd4c53d51d0efa119d22fd4e96867a16d35c5ba5e18b1f559c22decfddafe71f63ab4379dc8875545285bcea9d9872328b3367c4c059866d48948a0e886ec0e55ed5083c87dd4f62b86c0a0e8e639ce1c53965a29d418aeee39eb009511b6ff8b211a054bcbbdfe5e754be18212895c6f3f7df83d6eeab79b8bb9c2bd634bbfe5e0cd" + }, + { + "category": "float_grammar", + "entrypoint": "bundle", + "expected_failure": "hex_float", + "id": "bundle-hex-sensor-float", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2c766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bb04392690f48e647e3112886ad131a1b763701a6df4bc1aebcf488240d173fe32e544b8a4f8d173eaf21cb52596a533176334a10f20f7b3183304248779b848c381f65ad83b8768f72a1ccb9f8231c15eb575464f1e8f4ff1a2d3ac4ee551b166aa13c4cb9a97e2e599b0d06dbde2fdae483107b5efa7c998fd9f8d15d75d5ed92416c14d7523f351650eb6b75f245753ef8ea21ec6ab288884f48f58906e3b875c16c52ebbd205738f3a0f55939b46b55c022ec9e12378bb742e0687b17ff4cdc1fb806a4d00aeff0cbf09d1d39371296ab82c329fec5ff89cd5bad33cd4967c4b6b9861b5f2682e40d1fc4db5cf16ceddec1cea67c75c033612f71f6fbaba9a91f8184ee41aa5db6bc7b270dbc08c24e2de0da4441ecb91a458b277eb4c892d00956a0c6e6d09b063ffe9de1f5611943f0810e9ec3c9f52c7910331f3dfc0e5ac2df7365e7c589e9f29a41db5d07249dcb8e3ea1dd84d5b9bdaf711b86e40852ac6eaf17a70ec459336fd624f1aa20da49b0feec0b5602301d443e19084bf62d9aba6b532d787cb864a033f067efaf2fc6c5bc635109249ffb9e0746b7614e21929194f4dbc02a5a11b2f790edbfe26a477c834def349f9cd5c3ee13acb4a764a1f74f9665e14a32a609038da6630718fbe94e7204ecd35760404a25bc59ac536d5996a08b23dd3e607a08f5897c4188eb3cded7a43890e8e54473559433d0c1298a7a4df31a61386b081072f9c63a12c9d5b8bf0d97b546f8e82e30382753969218df98787222da9bc91c324fa80ef81e6a2eca466492927468ff7f74efe165cb9ed84c183af185bfdcec51427fdc5b0ce38bdc490472d49d0312bd2cbaca0abe66d50564dde4145f6d2f495926650d52616f58bc24c36f02348b83dc1fe78b3bc879d6501ab29cf52a443cb2b2579ce574d2eca49b2dfbf4e33bcc6fee7a1225412b653973727013476e7af4b0f58b611a4a0a8f3d7da86c67386e0fe945392714e051c3ba3429de84547e511e1460d6bf7b7d7c71077782dead772d17342c622482df7f38317f2897b74a48aa7195d3029d8e6b85d6771f77bd673e4862574fff8fc0ce477661be619293f3fee27dc235e97d795dc5e4a7efa654c05746b76f10c974529191ca5578f3015bbbcddfded745c10c90ddaeef3a4e7b3e28f5910aa6854147c1cf94526e00c7d5467e02cc436772b583952cd56106fdf036613b761e45102c9d43fa9b6b75544f96b579b3d5becb5c44a3123b0cb778ee726baa4b917c5bfc088120b48d6a4aeee921a7ca0fdfed820b31025b8dba8bcdcd668ac9d077cb2b426b648813280aec920b3c3d357ab343a47d5216228d744ea86524ca89553a18ed5d6c75f60f1d8ba78e3633bd06d24ce2e0ca1b49452fa67dd56ade7d220e03dfcfd7f9d0e44371636795e7c6e494b5ff41cefaba3d4e7155c54227c267d5a6a6f9594ace09644cad6e65db7aff9062de53d7b1abae80cb6f58e66be6ba17cd68c18f778a69c61a56e0e52aae4aeb02c7ded0d6a038326fe2a36e8ed20eebd09ae8dc0eca988d0e6a60335829440a0f8d670022752dda7f4c6ee129fe11ddeeb30779706808264a656538f4aceb28f8b466e23f1006323c09170ce362580506020216caa90e6547b634b2d3de162c3559d4babeb6379dc05dfa2502737f1d9e86cd343e7661d918927a773c10d58c795126d13b430e303378ca69d33baeeebf4a9d4c5d000e66894aa22029f607563a193bd2ed5d9527da40c16d9e355b08a371710d37967c230edea56f4a1c50ed5305fa5796fd6c329563f6fab3b9366e10749c0b969c1b93f01422aba215ffc30a46197f0263d02fe0e01bf9ec63b59bb21ddfddd70f4a1ca513c9d09df5710c222670566ecb1bc4d7454760107621a22a20e59465d11ee3ffa83fd097952c7593fef3b8552914d708eb454f5cf0e56fc130b6016992a8b0e78183ff1fa940b7052c7b6603c637cd8dd3aff4553bb99fa79a087380cb19cdeb38d9ee7996be4ef193099c4b2e97b6db2a8c6c4d5cc60101d6d49e0ae4694a498c4b31352ae485cd9efca92d2df543f5d6d73df138b26fcc03b1c386f608cf3efdc504aa2079f897ca27584cf90aa1c155bf5eac90897341a07d6cdef75bc2a7f58fe8d18cf86470fe469aaf29b2b4e4b22b411592c9fe6d60a401c9e4783f1df53864ce5eb6c5bea01a664bff8ec27b9af3112d4e8bd707986b6310fdecef08da97ab022254c9f2a49a808155b198c71ee613c4a6e2a68d4db50a5fa4f4db4ce244c6a59fe0e2c2dbba471db3376aca807933197dd8d4dd62fd28d61f1dcb1feb236ad5fb1a47773a259750ff7b8463e333f7335d6c9107f7a6fb969c85f1728743ae2732481ccdb5bf36d7607946c4ad1b76f6405b2dd0a3e79c9708701c9a84151e6fd265764c0aaa4da7a996b29a64bdbf960ec958d2a7c7eadcbf64b555a875982e00226cf74faf459fc2cc57e12a54e7e72dc94573fbad675c6c2ea22d01edd9a5a5ed2b358a576d7e3386dbce6990f13ebfd550f5a55043f5e04c01c4cc4fcf1f3917a8d1a379ec4963c76ee6a1bf4aa1fb3ede7d1c642f724ec4333d22ee9535553f1cb98dc9aec71ded7245edd6eb8fa1ec7be5be106d995cbf8d03ffe5e64d18ac602f7149421eaadebc82d09484d27917243e8ded22ff65a34e6dcedeb9f6e888d25fc35d2791ffa41f8ea5ae0a0de4c992cf23354e3f5029774175f55dd6e64c75252baae1d069c624432dce6f36c005ec38da6f11946c74915c9bf773f850c1dfe2a144d814720414ac91f9cf2e4f26313008a5c7414fe2c268ad11ef852a9e3a350e20c201439fa22483d6a8d354635b045da20812e94edeaed14ea3433d9eb31547b5eafe2292943d472533de17cd682f9cce4f4f4ceeca72a0cd03a978dbe1b59ef7998157b7b7f8c1ae5eee5b59d324981f1c4193977353c55de6322d0644ee85363487491c9bfc959b5fe0f856612d870c9e3d3368d2edf35ab4e2fbf108d76766db103aed5df9513987765f1cde0f433368b0339837bcd9440aa79fd5ede7df81b65b7024e04e17c7b64e924c227976b7ac5594e647b03f0131475c9b412a602881e138b4bf1f00becd77cf4e960e5393d3185de6c44c50cc99579b9f4875f4c300603a614799f20b6b9bb134483d3c132e74735d40d38056e86860dd2e204bd506e4faba396f8889bc42b82688677f2a88b31503781ae9b2b1a16e8549670b1637d2181bcfcdfca9f761ee2bd81472a035f302b3f85a130d7f796d688c85fb4357350410a86ed5a4728632e7fe966c6d94cc33d48113ba106fb662260dde93fec727538019cd859d5ed1ffdea570e317bd1f284a4f868b582b758a678bbc1fc46514ec58bfaf0379c46ce324f5d57b5c1db513622b4ecc26380445b2005257bffc70dca4ab96e6111c08d309dc33b3a79cfadaee09f94e8866a551e151978b36049ccee046b1e67eee96928a52ef6175e7e87b088e64749b5305f255e3594c842cbbe90ef36a13aed7991284e74013d234d92450b2a16ed80921a04e7abda3d872b22fa654698ee51758b2497f91803e6a9ead0c5b519069fa04451927cfa7a25e9c040c249f939b2efa4f3cbb26b3dcf0d957f8cac6a8aca9d192eeae8f900011b20b66cfbdd52714c1704832f5dbd078dd4a673b57e444e4b298d5dcf546335e53a994ffd7f78d8e4207b8573973cad73fcd00758b7f4f170e06c387469c913c6837753dc1ec4f45eb3a218c288f502f3f486453701f113c42197f83eb2462d816cb02f5762f41f566d22fdc83bd196b1a1115ddb8984bd983508852b76892bbed5d405221517ea14bc7e9e812df126c61724550c6e53eac1bf99ad3f318a4f6087d6316642194fad78d6db506ee321dd44751742e0b35ffbca58be9a857c1e32adcad8ee224b0e689cca699efcbecfe0a411e7120ab61bd3f798e581c93414c30c876e4335c53929140138bb35c6ee831c8e2b32366584f94c5795137b5a0cdcf7b0677c45c374848504fd22bacb81b29253f0ff6755b460c64ba5922365a04b111217d91a106868a959d839c328d225c0c432750dafdafec68c9364ac9a25c98c8a3a8b0b0aa34a74239d39d4200606189a32b02e6550d723aa454da87bd9affcd6ef9e7c04f6d4e4e1bdd6618d4f83e957e49e7b00a2c4e4df4ec324cfc10215b2fc380f781ce554fcc8c3bf216b1cc458dbfdf402291725a1b2875b0d2cf1ed75f8af97bbd73952e7b1d425329afdaef73d44775da74abb46403fbfcf78833c1e1eb4018d84ad08b47352386c29e25771474238481ab0281a0e4eaa0f37f47c2ee1ccf029bfee27f19f49e8d498114b68062054b071cff11f8efc14b96f6486a98d72ecb7a4ffe80724be4ab897c0b98567b5e2353b1fb0dda77b7905a0b9377c05bdfb6b664385d2a93363dc5200dd2e666caf2" + }, + { + "category": "float_grammar", + "entrypoint": "bundle", + "expected_failure": "binary_float", + "id": "bundle-binary-sensor-float", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2d766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bde4b888df257f955b76a55a127cbf23892f063883b8f8b9aa084ec77aca31802c70c2fa559e187e89fed50ffb10249684c8fa5fe3bde493ebed6b646c06332d07059eae32a693bc9788517f00c3cf53f4764ab8f421ad0ccdc8f127165ed52fcb31ef0db852bf968fd71a68c6f961532a28884450935c91ccfe4f4f30179625a2ad7a43cfc33d2b8efc47e21d1c5eeb9a15b2d2418a343cdfa32e5fbd2c2ec1c87d021392e6a2c7070fc1601b1775ae5d137d7b9ffb97eb0027c10d6ee6427d6dbcb96ebed482760446f028ae66a158ac2613a34365d450461b90d4e76a154938b2c6287109ce55a4d0c875a123a00f4614767a4031f2186898bbca4117d03ad9543890e37ca46e7ad3bad525945648f21cba3bd957c112ef78cbd04110d65b5edabe15450bd1bc0e242fc83251272e9dbddcc25962f577459ec85c82107a73531288714ec2a37ec8d5eafb5378e81730dcd864b03691ec6e6d70119527a9c22d6175986e0dc2b8b8a267054b30d16e5e1a47259d0ec38b7b6e58993876c18487c79589f36b976c48a975152b8d3490092d978585ff5c088f82bed30aa1bbaef2be0848bd875174721b557894be427cde1536a59c62fc3fe25df2531b62c62b521d483e3859087191995f719cb62c6245c806323da24605c71a0dcc027b81983f4f060eecef78a21c03861914327bb2cfb2e70413db1d6a43e48aa495e19d27d801a6e9127d124b0c0e4ded20b1c07a37f1c8876c496cca54b19c48d9648276a2bd29f8770f35d4c4a5be023bfcc18e182e9c1c2e9e04ae25df5f3ac0e204369cd45fe8fb3818eecdb4b2e5234b71fe55791d08f84e834c2518db5f35b2bd6c942d80aa77f324976240f54dc180ed86c167d8341d9579e60207584e37bd5691d345e91f827fe3be238402112044c9d1b73ddf1043e723693f91f3be4fc04684af61f96e177da6173bb89717d8e8c3b26bd7202fb037270032fdfa04131c39dec8d89648e51d8084ed8b6ed3991437af8e305e10e6b6f07ae2be586a698f9b2fbcc83401c406f9d726a34651fe3c27341432372b5f0cae3b285f7d17c4af19dc7c320f47151cb7c22ea9f9d09c0ade0eb2466894c5927b9628fb8365cd5528c792fcbd23ee796e8a5abbcceaeb1b3818b918d243761c7ec1a23870d74d36a16f1387b6fd66bedf69fabbdc512068e3179fdc7c54588d1aa22328121542cd95998b37ec0ddcdd28188004d853a60586265f7b6f3f26033f869d8218830248e22ee2021ba0cc2ac00e15539ab210073919af8991916761cfbb7876d142e734ec6e4810860bc358615f38c75bd9335762e41bd3f4e10a9d30d7bcd4d6ed5227865de39cbf49f72cec34709f069f964130658a0467b2b8f9ab4ddfaac0b9cc9adf88fbb435397c3fbe5075fe0a4e3de93fd84a988c4679e1ead841b82fae972278adfc62fcff0534697414c9e472d6cab3eaeb72464b10929e717ebf1d51c8473d8c5e6c02d87101b6ea3f7412f45d516159b69d76d973635ee56b65e80ef61e83a3491c500f68bde6de17ade3836aa1d42880f0a69b566520f6cb126167759d76091e25f71f0ea5cc60857fa7a4e7fbeae707b0cd208b66e4c442e5a2fecd9b39393dd16d72d640166b355fa75fa812b9df5b2c47db79b06d46d9690a0db0eae07293d446a4dedac5f78a6607ec2eed04c61a726d0819ca1a176e1557274ca682814e648055448ddcbe73f5a5088f98b991bd5fb3955d78cfd111b136a1db2c3caa4c292049116af56c18570cd9ac2a0a33d8bd3474b0726a97cd7d57def3f1507e4c63af826e93ff3742b4705f84409f71a890ac14a88236e71045015e1102e7fdc3355cae8e6a31029dfe2537fed3ae79d07f507161b0931b006c9205d3df5ccb7f9c6f4b142c003d3a3d2df5896a4397fd8b45b37fc7ba10cb0ef7254206dc9fb6aa6f7dd740b0538af377b66de6b6d850a5a24499eb3a860d3ed9ca91f48e4c6915bdd11862303d813c9d9b8d0535f5a563e9ffcfe93beee59957c965cf40b640996ecc4f6af7d381f6bda173d611a0f3fe42f4ad6f6e1ba0e03a99a042a8b51a87ccdf29e7d0e20802a5a69ee6d8ac3de9a3db484682da23361dfd0c32567a3c2fcc43124f1668b62c9ca6e4f4861a47a2a022b2aa74c122c52bc4bdacb78945d5c4b2ee3fad1b2b0462f176731fafcb5cbbd2832f7924f6a6aabebe432447c1b43ce38633d6551735f798b05b91bed6f79e1f43db668df2025575318cbe259d0ce554668a50c3ca07890ea973825ed1b555655fffd3001c3981a6c06553947007fb83b6631a60d1356a8ad6a2af39fe5c14f0a932935440aac910d7b9ad3bc08a3bc76e7a3a523ea1e97e1b4518f7831f0dbc016e91562ab2279450c25b85229fa75638576c258db026a2a617599fa9558e54edeaa127052dbbb569f70bbae36170ea9ad35847821132a536ae1bc27ba858dd56f6fef5e92874c7fecb56dc235e65b92712786dac0600848b1e9c1c29149ac01a7dc342d31b8978af7d7623d957d4cb5b619270b235c9d62547bcac60daedd5cfcd7a43646dfafcc137db80db47321d68fa18e1326674d86224d11bec2f4fa204427a53f93a5c9b6b6bbfac4da9a0ecfedc6bd2616b45da10d469c43deafdaeb05a2327e71c3e135f76b6bb12bb7fa6316e40968d69d801f0634f26d9f4195de627aa86c2404609b3a80b10ce874ec758c951a11629ba32c582400401d3cac113e219eaea4f083fd73d1797546364dcc321e189b6668e12b73a8ad7e0bede2c646efa5a645db4d59c7bc11c75fc3d53690df84b1d7b565296c5128541205128bb8f3e0132438f4f8640bdd6c9f48cd40863fe10a051721b468e280b7acce9d02545c3c444a2880d8b391b76c98d2620d14101fb0fbf0123dd8af102640858f1b365d0fd59144d2c32abe6b866c474cd8159959e906ff81499e1bc64b387c07ad08fc248a2dcf6f5e783911e7cbdd54ae21d18564d3dc935c36179db6c48927b79d7a6dff09e71e60aa9389e50c2df13ec0b7e39b261ce7d2dd55c4f8f2adae54f3c5c43c50753b6ae0ca7c89e31edf579aa567065c7f0fbc1ac269c0f2c18e19b8d1426b27f5936ec19d05212ff239633af2572302d56f818976d015d2fa539facc59d48d847e866aec097b24806d66b5a1e1b1e56c77c03528be1748ab45f4548037ce5e454ea95eb3c5b6243c85ed251073f8554e41e36f291ce516e7b2fa161bf66f2e93690945b1fb6273327b023f9095a0493b8eac781ee6fa99a3f89e0eb338a0d7ea817ca6f9fba9175df55e7215b45349d8c871b6744678f6d4b1fce10e1b597d6dff679d5b19a0cc164525ebec993d0094300628ed96c62b331c11b2ed1313eece99b4ce8747a89a2c0a3904115d47beab6eb5edb7c04c47f73061b2441728cec48f2c7d6da008090ea0aeac92db9411bc3ae98e6622a3fb9b13174cc805733ffd7cf1c2faef457a9dcb19944718f1bd1fc15b4f6b7fdf675d4bbabf776d87b075c79f04b0c4954b3e3155cabd92db31c546f8aa95989b62b57aaad85d6019a173c678184117691a7dfe97ffd4287ea0662cd4041af5a38d64740dbade5a22f3769316c2d7a30c3cc259c13183a55fcd86f96dc6df2f21952ea9f4212780778d0719a46ab042d74ef94aa2348a29ace8d9c049a44776e3905a7fdd73fb5a621467399dc105bef303e613798ac7ef6b77d1a127ed7fb3afdb74e8475d3cd0041b44bebaf4f34e56bd43dc7367f0aa577ddde756576734a9f56b578cd72d4210d6f80e161444eecf4b97cb3e4ff8c263d83a87bfc661dfa8595da66e7032e2d7778b740fee86790f21fe9557df0a19c2bc84ed7d3c9a5ed12b5dcd88bf734d7a6e1e08e355fc13c48b19d4df39264eb290830e22c688244729cb8c3f6c3de4f444664b5e311f023020989af326cd8b717848e4b2f28398d8bf48bddc0f3123d2d16d61e167917af0748c3885c6d28a382ead8cba02f2e576222f528cc11c30ff9c98964d3bec344e1cb833a9ab8ee830ae174726baf4adbc55b5b5620c0c7c75b0efaefd8c30cacb9a8dea8bd3435d8cc8ef02b9d90af04ea84abce55b0c22c2188c43f4c4a4cf4df7b6ec8f592f7debc9bcbad32271baf3754e18335cbd0ff2dee6513f150fcededd092ccc3e08eeeba6556158620e36b71267f7edf283210663b4208bb40d425656e811cdd3414bd169d04a9f8d562934d38d366eac2b30266ff5731a3cd96ead5fbeb9bd68e6f8091503635a88d60e5b18cd5a0613dc0e2965ad9e8d3a4f17079c54157a52d7940eeecc61140503b3a0a57a9bb08067eb7912c959de0d36481080b5b373e243f20bcb59230f8f386c6c72e476436cfd265c08b4e8886a7060f99f72948301aa07d81168b38434950577d50d260705af7388ecd2f3aed5dc81e5c1e4ea1296ea253adc15f2b41bcdc6e" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "noncanonical_int32", + "id": "bundle-noncanonical-int32", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2e766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b1ae75d52cdefccc51a046bcb4508b80f3847c5b7e94d9eb0c6886afa3058d1988fb2f5297d64ecdf7f26b526dd8b2eed1ef12e95d9872e7a357fd3996785d470c4ea779d8a192feaf3a18263575f6b6f810273b27e12727b6039e879444d9a9b09e694eb6b11d11e8245889cb59b0450892ad29ea86e0d06f3876d67d75c7ea2e8e33d69f03e6a245146f3f680a3375ca1a330729e86499fbb85cf10bbff42ffda6c62886daadb991d03f0e013719b7815adb57c7ef777095d291991533b72686d45c14aa06dc7eb5ee8438392a448aeefe8cad4ffd90d65ed41b34783fcb3e0144bce0a2d145b8a9acee498f3f43608079a4c2af6cb026660ff1fd505acd4ba41572dc2d875eca29a03bb1400c8401d13502ff08373024a6068e97a677fcdd615956e1517a44321cc5804abf2536157825d1df20269b33cb91d1667ba4aa5af1d4ffa0374b074e8ec03a023998bc995319901a4223b79b6e7a801e6ef6719fb63173f4c4ef3fe8ffa0cc1a38fb6773fd7532bc3487fc1b1d73d777dabf4d9a6f16a275d604e71cd4c25065993a5306a79857598b7b255640d3caca3412c52e675f19761ea2a388be9ef295665829d9cc3f854df03ef736cd9a6810162067e2677c7ed9f31050fa6a7aee8269afd2a22ccec0cf495f3083f3c6ff1f61cc29dfd5d2928b2c7c186aacdb283b497d141705d9b723390a3280ee9d275d1f92cd25b09c65068faff507407f770364ed597aeb21e08b8a54193aa8e97a0eb86e2f0ee39f1e5497895f68a41bcd45b687f3c360abd107b0dc9cb6623a9165310cdfec6f124899c7e5bd2b1c495be969ade1be434691ff90309f41e717f3f7b891df1dae1f215deeb306703bb2f3ea5d6aafb62ebc07419df14e25ab17d095b7350b1413ef198b26dd8df319b4b064cf951cf5ec184e32778e6075b3ec246bc761c3b344eb8be6a83a47374a8f7b8e2faec0c522feeb3486e60f8f4cc20c74a7211608d4ff10c2ae17cb13a9f9f452488e372d3a993ff2475add74afa172968bba2699a70e84b5a274225951982ab119199611c9b64eb3facaf5490a7408ff20b2a90cf1a227597fede104d2af25db767147c31db248c025fc3013da6fac8a88341c99a5c1f1141d10f98571d6683574ce8f82a5d76e1b33d1cf76602dfbd72d203e4f9159fe07779cf61fe1fe0bc6a8ae590b12710eacdb34989b22d29a8ab9049d72dd4e48a7e0c75d2bda5cb39c176bf502b7354ac7548c38ad518b82a0d3c4ac25f7f34bd1e6e8f2da8337baec23a2f942b511b3b1e754af1a0d03b3127024a3722c8414a47d6d9905326709f70326a39135817ae01fabd5c60564a3dfb8c4506134b3987e05f68a5732523d0f53f11c478819822c3f52f611d9a5ba4030dba3a13460c0ac6d90c43a9309b0988dbb6d7d58459c1ff54ce3ffa3f0eb5b58a86a41773790016bbba991bcd5bfa34f311e102ed03c87b8c8a08f9ae76afeee1a3d9e4d88bf4616eed969c96af03d18e4f28a6e78a3c194eb5489cd60be32e6a35eb568661a1c1aa104e3d10dede3dd2124735fffa73652d8a94527a39c625e0fb15d7d20ea3926a0d94527a7f9567310e55b1a90f46d740b9415445da24eea93fba1508e8142a5f48cf88fc038e68b5e3413c38befe4f551a9fa8e95eab126a3a5986cc647c8fd692c8cfa15b5ceb32e487d4332fa6bd5560bf25fda74ce57ae1e40e863411d53ecb2e383352a790a564dbc472fef099cfe49a870f827c96e8f5ac1d2b9ec8f50cbc545abbaf5286c3f414ab596e03e1679b07bf4ea54e8e36997aa5541ec5858b097c7e167f67c79845f525b513bf8143eb2ca6631cde16cf53c5a818873d01604f67a577cdb5f5ee061b6628048af3ae7f01677d4cbc7cf556297c427bb9b15cc1d90c63bfec43c753bef950fdacb7d8bd4fc93d2ba7e29054af8f86aa0083a9855c2b3e27600e27db7442d4c2610ace438c95333e829ce2f02af97826eafff5fc1ad1e6aee0dc23fead07db79d8fdd21f8fdcfcaeca675de392e94dc3fac09e4d3e40119a23aeda471cc543f6a4e8351331b914ccd850972d61782f21fc22132a115ac1b23a4b152cb3fd3250c5e32cd52be36edc53c946af1256151e980e9bce9a82f4c6d120ee9fd511872942043732556ed887fc171043b52757c186d7bc756d9d2a03e1f6063d8f61c96bfb211cb765a46a0831a5ee3450d69f482001f24e23c674ffa611abb64d0cdf9e3b366faf90ff2fe0f159ead6fce2768c50721d376672eabc941e4d049c975e612a01d219b4554624d97f74d53c301ebf8fcfc854df882acfe8836021f94fde0f4b7cae08bf3189a47f9c54b35e5325f5a9f0c966a528e19dec88475b7383435dcdcfd76a2b67cee6a1657c2ac4267f23ba1b156028e73f94548d6464a7ae2a2952e3189b0830689528eb4a5cc45d988ab22f7c86128762212bfb91db5c7caada260e7073358704db202197d89c49956fa58219295a6d906ca27d1f0bfe2420c4c31d67031a6ed0cd578da0ff14d0058ec83bfe5349ea2465cc00c3a365f440b4c4d1ce706a00acebfa8d1a15a409f6dd3573fffcaf55b896c1e3fd05ab11e16d285f6ba558a22e6d6d2c0bd4854754e83b76b2953cb95bca2538c8f039a5e8120498586896135099149fd2d55d7ef3e059c335513df4facde741c0ac4d7c2658ee5a1159900784401e754dac56c4d9e6f1717dfb8e882a8544ece92d237fecc2fe12ee9253caa3177cb593967f1d0636d41deb8fa331a597aa26934d34eb3e708153e4c9ddf8708e6c3d95a7ba8cdcb83511879373864969cd5fc23f0a93af249a91706e7962219ea7340d098d20f561062356ee9f6b9ddcc059b13f4a028c93843ed5d550c77b13b3949ad80e85177baf76cd94d3e15310b1f455b8acf28043e7347d4b772bf26d231b4260a089a08cfcb4e4c657e3da486eb7eac2d72bafbec987ad338189b2a375e8250703538f19815d2af49c9687047f214beeabe9e834281a649722fb72e12938e7ff7b39b9e2c23eef4dab4ffe24a9f476c3589f0e8bfd237f9e31496d6607ab052eaacc06fcc764a6fd5e0eaaa11a138e2c99934dfc1abb8400383a6b9164a5a1f17c11b639e4ae28a707a8ce1d1a96f9a1b870a25d328f8bc7be344768a8de31529afdf365daeff28ca7b7bc7f1b9c467cefc71948b28abf392b9d274a0c0df184d76d4a44494621b576b488d1f2d5596b5d228ccaa040a763f7b7bc76fcc856dbb188fe20ba6407c851a4fd5d815e03d1f5d4e99ca2fcfb76bdc3c2d9e6cc7afb90ab83db5799acfde9b549ffc68a3af0a5ce4c62eb049b7a35f5dec65ac8ac0bf2b0874dd8ef84810afc4d647717b7efbac357264301464662a5002e87e8997a041d5282de85c342d22dc67e84a64fec9e37bbb60821ad8466fd1ff5a0add3ce40463710664124a2ee1f2f9aace92ff13081bfaed84008f7677ca19634bfb9b7856f6be9774b94c6fa8b096d4119c37aeabc15d5b043b6aed771349bdaaa5f82d8cc116a59005824fd473b71d89f37331ecdc1fe445d238c6fdb5b0902c11da60d1c44d7baa5361c20bf0a607067b315cfdb9cc1af73a6279d093691922d1eea2919af1404fee546c1af93cbba3c449cddabde21d48612b5f223e67307ad795a361b94854c65b8bd115ff9e9ab3a4b492cd4a7ba2060790e4b816a183974d7728fe5d987efb3e2a0eb2e01664bc2ef8e14188bdbd188ac8075f78d6c220cc74a43e778fa635aa59d4216dfa42e23b085ad587d08d2866f88820644097dd4f46a6959591b4fe95c80cef483bfad0cbdab091b842bc6d8a0bb7f6421acdec0ad9aa7bed397d05beb8195459da82c8802edbcbda06f65cd94304ffa3950070566d07ae365ec5360d9119774c4747ffacf5a2ef1de05e33fcad8ca9b551fdfbdd8eb5a770e91e3c7aa500fc18cd93713a42da2851c27b4023e01e3b920d3a49338dcbbf188e7da026347c09d765085217c25ebe2742c7057f7a67577dfa525cd08c7c280d4d949a7f583e6835dfba5a9edd318a0eb8bb3c21b0cee59d88d17d589542efdcb2c60e6a798d4bed0f9d70addfdb5e523a210ca1d6b2d7f240f2486d1a907a0815e6063779cf54a7b3a07c7364857a7126de7753de07d29540b7251da58703c043be1e2168b93a62700d66d6b24fdd4d9a875ffb78e1ab2a45720dbb4778a3f2a6c35471cfad6f428b1bc279a4400fbb0179ba0ff327c6c385acdaf15550a8254847e54ac11bf6ebd6205a8df391679807557300a50972d468cb39178a86e2031b4271fbb60090e650a4a3b696edefbf3e32d4c604952cc241a6b1e86d71bd18fc54070a533c3fa34d1b9cc58" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "int32_field_bound", + "id": "bundle-int32-field-bound", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a2f766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b93ebddd3ee7cbd9fbe49eccd8f732af73a12b5c4e9be58431877c9cb6f5e6fb214bc0a0476f1333796740efd4a53f3d83c21c4be9fc51c71b9fbe11d126caf8ea5f7a32433bca84d4031ba83cbcc7ddbc7e38db5dcea225f5faa567b48ede7293779c8b404d25223772a1887afd243a11e57d22ce2e065695548f6d08b56d2b8667a97f74fe0372a311b81a406ab66f8a2bcd8277a5789e018d8f004b9065bebd11269e3906b21cf81b9b971c98849146bc40e7b202b35c02c3d2fc35689745ecf1c859e411c00686a1723bc83e1b7b97a12203311eecbd2e50a0b1351e427f1285acae5fbb6b4563bdb2b716c89362e028de606f9d4a3b9a1e8cc756311fe4b0a2b3c060850d6986a7889c6b8a9f6da7661525528b4fa2295f0c00fcb6bedacb668a9cc5bc887ef92c9b00079159b2b2fad9fe7dcb314cf69960133465491c1fb09aecccbc18881439a7a5461f259f14ec7567871e682292b47db0fc2a798760a35666f32c1b98ab3e53baa2132a2bc0c783cb72102a8d8ffb1f45a8a4b4dfc03595133d9521879f00096e060f8d8db287b437cf24cf3200d5c204dea39d8c4c6cc82ca3a1ac146732359f8c926f12ee49e745be208a53a28f3d8b39c2e7ba7b23f6018756829401762d4fcaec90287a5567c5f0107f1340f566911a3bd7721c35704e05acb27d9caeaaea4439d0118e8ea97980c354c71575d8ebebf780c9895e019355964cdb0c05ba99580e937f5c58a03934229d03c9e945e6a893240a5fabc4f09587034ef945405c75941b66261fab324f668200f9b4f8f17db8946bce63657e999b9a6ac253b823e4c3019149426a9c2880874bd1a234567e2cfeb522e143e89b8bc978a1d010daa78e9e81c083df71e07841edbc1cd96db92bcbe2077266a34e736f76842f9a64581cfa9ea15d762a6ab90955310f863ca4c1a17bea9a8b1faf62294c7aa08a5916d3a4222074df0408846dbcaabdfa3e83f0e3644fa9e61c4cb70245f91558f4129e4be77414c41c237ddbcaaf7e4e1c26bf5cac46eb7fa77b52bfce1f153ed3e2d35fa7af613b24afce7981cde7741e3d83c122067a636ee6b3b6b441a9a2db5d0632b6d0ab56dd3f12901c1ceea4237a87c240abad64b20cc6251270a6c86f674a7d84381e9efddffce6595a3597dbfb2b3deaffdb3a13c7ffc353ee568ee0542b1ed6b947cfe66b5b5d3f4a2616e9260d746f1d5353f00e08d172312d1d54cf85223f17820376ad5bc869c03e4cdae9d230102ac908cfcda6ae2ae16ad3475c545ca680caef091ce2f163dce277e737da9fc14648daa5ae555d09e44e9ea466063fc7629411e84670ba004c7aabe63a9c7560789916497a8225e4a24947baf89730d4b6ad43a201a1d740c80065f74d7a41c9957de7ae025dd5cb0517a0ff4125b49ecabad79fc0dda0b7edaca1d8629c5536f146bb54821b794299d949c51ecdc72f79a9608d07232885cde31cb4cce3a294529a3421cdc3cc70aef07293e16131eebbb53685978bb86cd14b6691074e7934528b797b1dca984d2dac3503acc53ae1b1fa378dbc1c01c757c13c5e1c2fae7c40495e01c6ebbbc3f2c608c73c9f870ffe2f5b76f6b80a2bb3c07fb6152c35c8fc85c1b6eb7149bc34ef20cbaf1fcff82cab29350dee9a8d50f03a93e2abda9c20c53c2481e8ff6f824126a4d425dab956958b38b6fa01ae60e4523d50fa435000860326c7e773b50b05b43e8f68ed2f887794d2693628f9589a58b5c56b95ebea330ba0d3fb5049c677da1391f5410e19f4c1fb7bffa38210044c057aa25acdc14b9cfbeb916e831342bbc2bdcf6b60e6e979a65f19182b5614c852e9afaa215c6219f9dc653461cc2ecc6783a78468ee28baedc98d457d2f8160c77bd2bb6d52499f34c86c02ba18ea0b1414d1a2525d8dd8f958db835dea5a5d3ed14138febb9138f7cd02c0fffb516b00529a4af61099ed9f970eb1a73f1bcde6b17e53bb4dc8b0efed07e836aa44532bfee7e79deeb4ca3cf1bdca0ce12a2bb9b9c52aec8a7eaad98c10067b9448cf00794950badea217853cb0b44a7262b1d0abc490426f5ccf567dfc2dc5c935b29608b7b096d821472a6ab3c1d1206efa835c62155d8e94021dc3f6482532f84aa27e323de681ba83974d0da1247dbe10e0b3f32d3378c9359a710fd01d8dae89c48a4dff704e8f97d51894701a961065fbb0cda7be4c698b702cc61f5fbaef81719800e64c3a6f543f1053f85e9edcb3184f572a2215108810ad80226681f9d7ba8cf7a3674469068509aff45c3a7e9140de910727e43c57722a47af4aa1846b7d4029480280295293a1098ab9fbbdb62d12660ab593e3546af20265d53b09600fb0185361c67bbb39cf4619dbd8eb702b389a03826017a6fca3776a360ca1d0d78c3f02b5ca9e4f4e66688e6e758a56bf91c28d10b8c3fc7fae22ca23723993632809a4096bdd570443a7be83cc336f122514e82bcac7296cc94a70fbcc47a38ed7cb65d16f7fd1e5821a617a8a74c2d647320c6cd025c05eccc67ed0981a5f60e60acea8220d19fdea05d6a160925c1be1acf6060e8d6dea6f1d86d37f51d6a8ef41bc16e24e0b1f49241f1ab3a42203d44cb66311a79e930d189125a93e63f54120044005b5d5ff705b8ff65e1526828d2f24dd613ad0b99161897fe3e05d21f2808137ae1cd2ccd7e433efdc9dbbc73d558571622fc6be2ea73d2cc50fdbee9b075e97d6cdafd8221229387b090c97463fd5a01c7e617ce0ac030a2ae59a49ca4e67447750b95fc254012cd25196f528042d4a84667ed29cf9eaf97a268efe015f25dcf7d1bbf3c99b606613d15f20b49f826d94e1a7285498ee6c78784ce7a11b683ff8d41c7e11cf59d4076494dc1a075156c5cd5648e98a5bb25d14b408c41f11ebaf7c0da08ef43a8da7a6d6f37056a1e85dab5f640156114c5e181f7302a077f1d99f1a575d20dcb4b19c13a3debb058f633e12a3ff8741cb5e8a68996e61adf58df6be761a4a7d1440792551d2382c14706d060d13762f52bf7ef1bc66bb438e1d69306a2022bbcc889ce48edd0e8f5cc25cbb47ced2652f979df41b81cf58fed4055afde3fae7b2e39669b029c85d4bda7e47190a471e7044e53961f640ec9e4feed023802c4824826577a0ea80ec7a8f1c9bfb549b41f05cafccaae95b4ec5ac1ff1e93a7b2096cbf3a41985d71281f33cc086802a418f992eaf7affbacde397b91272553aa0cf1af06e28a93ee1a39cfa83a0a807ac52fc4b378c21c7eafb84fb568853c4c4c72f104a5d67b6ab03e266b12f87d0c370afc4d689a47ebf69c806d33692f1c61dc839236b648aa4254229d8938eb48be69cb901b0ae7e5f3970a5d3af2c4abce93420092a41b8dbbe6de91a3eab049c3337f937be68fbda10691b6f7366661164160fbd2f50ad5da3f518b47f9d099717557b2170f237f9c9883577c14ad0f9f020a924990ac7347335ba91729d1759c5b185bfcc5fb306b1369e3831f1f1d1494c939ed561d395b0fd175f8b69f177f06b9361a84b781408ef99dc44338fbdd393674854f896ab03e5356090cd8ed88bc180b75a2fa9df3b9bc21720be0fc39edcbca33dd157efd34badd8cb9aac7e9c48de64f5edcb86d6471755d56f3098b3a15344cef05f77b9d0ebecd83a6b4388d458602e0f0a70f2565cdeb52f0d3ff96e968d3b4626be006fda00bfe08c1d75dd9644332ef9cce316506fa85129071c8179f68796b68d9563b50957316ccc040b4866faeeebf5560fbcc7d3f200974f9036aa45aea7075d33c0b7e799d7141884b9b845461a2b42929d50d69e2e9b66c79ca10bb2405542aa55ede7def36443e800f4b0de019820c190c2b0df6711fd2c488439bce748884f0c742d8ea0cd756580e1710c701ff8e21daa7c1372875ba6faac9882ff372190dc12566b48027cc7416ce73c72030af60899e3943d2e5a37d34bbfe8babfd4b5e5000c2de74d7fdb0e0e9bba2f39f8ea1b48f9d6b57bd7bed04fa0452ffee57d676930ac33daf34a7d0c570789ea6ea9e8f35e15a89b181df34efc183b992f0dd1f0cd129e11df937b23d60ac9baa5625a227123c44a97cdcd6599b7679c0d04489f36d20725f8271c5af7683edbe2007af527797dcd7fb7b449378b55278ea391bc472debd8fb787949f1974cb382ae123a4d03ae5599e2eed1836dda3763feb3539be4577604a98f838988d859e3ed8b3224fefcd6988be85b1b31e8b9e9eaa62a438c22e655e4208fd29b3cd0bb54cb36db5210ebc6557a2d65ce564f34fa919da4e09476f7a82657ab718f73907b6ac57d56d520c43bec7d70aec20d01398a7ecae158a07fe18bec3610f51" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "transition_chain", + "id": "bundle-invalid-transition", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a30766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bf4b2e11862bef2951545fa91151d4fd609f814d834ef0547ccd517e5799336c63d7fda19621e4ad868dc31269cfd1cab022460b455c4c4dba311fa03e08a0093406dfd727b9be1e1d0cbc72167574fc94ccf0a95545fa77b210a4d8e311ca854e551191f7b29880dae1598197abdd9457f534f69471b4883a043f2d14311523592d616f16b8d88ec53cac4288e9ba13176e98d923cf82c60c1a5caf547729f1ffc1b0fed649721d1a7fd026e21ae6b7a23c914927344dd45cb9ed27e51ab58301e7a299eaa2852cd00a16a1d4cac255f548451c078e290ad5694cc252302ba13edab28bf1d66d4cee9b0d251e242e8af70b9a2aa210f6c0c458c6e0c418401e2cdfd65c3fe2d2134a9527c1b17f472d644eeace376043613a9d0b225e393995cd183e54f73565f87027ea43bd8642d951578bd8312cb7a627224b0887f88d9c9e1d138ea4f8282c116ae162b016d03cf7eca90ae6fecea82443777ff46ade2fdf1be6f347e13c66d266cccb2ba4795c9e8173b7f82f41d16aab11a2fc77f4b36943ba56db93bd8f9dbf532501f3fc322ef8ba1bd3105d5c3f44aaa0b25de1efe3a38d52ec42c3940ff84e784fe36bc97ec1e20e2a91537555ccb60c2bf41adc8a65c25543f15390984ba61f7f8ae7e1f9304d6ddeadec5557e9ee46fe47ce7bb461dee66f05e544d8af1c0e289fe74dc3e19d6813eaf9eab7d7771da0ac519bb632305fb6910f74b882cbca0dde48dc518c8bd59327cc301e225ae853ed4e2130f16dd8849a70befda9a55154bd4cfafc45c3a621432692f35ee8a492d176655e0ebeb50d4bd192f5929b29bd47a76e5d316dd4e44619a1540a05e2145a8caaca881742ec55f3112078cbf430765a3dc8e7d7a372cb0991913bd4b39573adef2f1978f5888f8fbff65b60cf3f905531cb165c0361e83e834783ed82ede49024c2046098aa5e878d88ca3db2a2a32a32c61ab53a9fbe47698e25867192f1be13eac0e57836091ba61d18b587ca07afe816c023211a72e2b10f6ac3ff4e64837f43f0c8fb3c9453112e8b40e31fd5e637fc7a9641d8193d4084de50fe75851414e06d40f4be93601e02484fb96e71cb46636b8b89414c4928c6dd19d3f99a827eedcb1a0d36154653061d359248f797da124e5f4a3c920317829b1be3764334818088f0ff34ccfcd7be87ecd42777e677b4b39c4c633f46752fbeae872d1a9673c76e8a1c2d00f62c7b36b14767c5be370178f37ddd06da7bab043119d25928bf32164375f14b17188f63e5baf7a0a9fa8c6a88459d44cbc6c77ec1a70a779e266d70b3ce81b591455428e66b1da4f136b51bd22a274494bed9187abb116f176added6bd46feb9b3012c82b5f7b75bc0c5617e84b7375a7f6edde8e58c3bb40f60182d1b91c1b895c09bef3cc733eb7a9b295543e18a29fa07b175cd6c25b1202eba6dc70155ea102c45a36c9f1c011a741c7d390f3790c86d2cc168e3be2a36664b57fa2ad134bdafbe64b9812c74abc6f861923b646adc62b85ea766f6e83f233a0f6eea226fce5536210b276c8f878a4e8be181ad453a83ad293b64d231af4326509ec3f9a60a6394c709f3b2bc6112a51134df42867f2bd0a2221a2f23131f9fed3ecf9a35082f7a55bee5053df1824864e5cddfa3af3cd5f9930213a89b9777aacbb219660173bc4d89005967fbf1cf749f00fb3a6657f71640f6381e5163be64b68d2d59d5861edeb13f845c5a92388f356c50881e06874a31bf978270034b675c21ae7f8cdfae7de446edc019810c60d05929d472ca3ce928085bab4388307f9bcfb2f3d6240683202080c1f72befd11b168f33a68784a2f0e0ea2a554c7eda0ed91ca67deda5235074cd3c5f184008c30df7074d962817c101a721f71d9ec0f5e74cb45c731dca1e5f08d0b0c54d70bd6809eac8f5ea6429ed7446966a86e7e5c942e863792bdcc7adff081e7797bb7e540a0c5a35d421a26283ab854d263019206f9fec35a1627801bcf235696de1c3542b4b5e3f6dff07b4eac453756a83e00508eec1fa467859cb2b5cb8b07d38d0060c084800392d06b61819d8bf186038f7131ac94d600acbfb1ed3ece2fc299195bdffd59e132da0c1fce8173f46c76caa99de9879175e18aedfcdb626be5ae45af382ff1fe81062aabe165fbfeeb02a771849f482aeb07aa7e5a5002342c6d100a4102c77f7882c1d964c8482c2f41ca764809eafd91c3911e28071a36c77fd7e705147833f8cff0e0ef0363514195bcfa08f0a494678c67172a02ad764ac418ee6f1bbe27f5696a0e5771f509b09cc4a0906d30d370aa7abdb9919cf75f94003af4523fac1f26aa3711dbe50a955312c0bc6bc8fff0720e24d341dc5e84730a69cd027bbd18a9ace87d6c96569b725ef7a3fcce3fec53884e6c9dc01e13e40a9d44b3e8753b9581b1876713a56be232b48236a7a96e387e5653debaa57b5d550c3e2550bd127fb5077fd6685bc1d1c0289a721132b8e8b77afb39a45f5d305b09f0b40472666e55453cb0847b4379fa788fc0b2f35c40c14bcad63604eedfd69612baef8e3abcfc2a0e917e8309813763570b4263ced155966a1bb5931912a7461c406ae463d665b9701904051a6adfa9855e9d5068ec6d7972efa0b45beda0e45965d84f24fed947a144e2d83df23850fdedf386a8046c72bb804b1bb749c7753dccc9e90c792cc2d40bb92628fe0b900dc2f2360e7400fb68e51a06e9fbb2028a6cc562d51861e2d0f726cc2c4fb16bf931cb6acff824631378e8e1e347675e24b5f1234cdd217623c8fb26a37b0a15cf2aa88172fe8713913745ee4ab39120c3b2878566ef873f4d1c7d0da5b526c4764f3b43b0bb6dc9c4cb798ec3b1714de325522175171bad3a311ada5bfb149390db942b0b5e55c967081209cc44baee74201e1dff51d076f42b413b674e95cc04ddabbe13900eed834a7e781aba60a4ffc7d17c7973490618a29e8cb1b9bd0f316ddbcbac6e7f22325ff83b454e1c7cd74c9dcc9145e46a1088b46730ed61e37a4d05c106c5e2f1abf67046bcef44b15b0506149b6b244b06885c52a005abcf6e1f37c6ab8c3f1f1b4f202f4f6190e2b947ecc3c0f5daf0d3201ffcf777a4595b1e38d2d6abad5d4c8a30c0142d8d1b89e645c1398b8887d365c810e5799500dc61155cee3270577566f72126c2e77c1338a877163a23ee5673873490e29cd30f359ee2fe75627dcebb358689d9f24ca7a5201be4e42ac1a94d59a269af12eb4181ebc29fbf9307aeca070ee1c6844c9e3a84066c3dc5e26917a2cd573e2602e36307feaf976010178972addd787bf9448ad23f0fa7b14e2f4f8a3da53f0f9f4f6839d971224b788f3e66da4b1a5c078e5b0accd913dfaad7de79d0d1f86948fb159579f742577bbc852f71920ddeec800725ef6c6201f4b6c9286a5611264a11c7d16d38bb15d4a32d9e67c01cbe8bc3f7d65785199ee4ba27efdafe63751b3b56ac03de1b3eba485064a3716e344b55e95d71ed56d0ef93fa9c49d6b991e46c80f01ec200cc98bf6ac78ef3d9027a4b80181956e6e36d185f29787b304a72668b5a1e30ac49da1966863ad8491c17e99c20c59d9be37ba674a654802bdb80c1d41f69517d9c3a7268067542b07461225f51bc973d7bebf852832fba2faaa0ac5c1cccf77ec9effa187413c9e1389ac0f7beac2d59694652f29214ca53eedb9243108fcd2e5c2db309054abfc59ae17bfeddd330b55f7ce1cc8e4b58931d5be29625b5fa10f69f9f9a175fcff118836c569e38af48b98ad789c4e85dadb161e8513e5279f75802a2b425884d9de6c21516401d1b8f304e81e341aa267202d033daf35cd06150e38bc2e2857874d6cb2f34f9fafe62ff777f11584cf3f0b390b7f314cd83e639596d08ae502fe935d89eebea6ef3dc363fc30d214c49b0ab0fe1814b147ae6ca7f46c00067c805f95a638640cf4dcc0c2ada911a666d47d21919521a01e37ed6fd9ef31539c1e930f7ec76efa7539adb1db99d6a966553f8553b60ebfec7f99eecd660b8ee6e4bcb03f5d7bd5883947ce440f269114e4cf5227587263c5f6dd7ba77a0e77c0ee28343058fa95515c8fddfa8154cbab2a67187cab390de4fb23584d856d6bb06bd921c06a045e422137e781f81b39f1b60a69d5d21b3ebbbc82d0ecf4b76900bbd6f5fd1e9c8b529bc323d6f6806c5cf3ce7f27596ebd63c5c184a50f5bee56acf978e532473103be2ff44ed535ed8a28a334e84c2b12f0305d" + }, + { + "category": "catalog_contract", + "entrypoint": "bundle", + "expected_failure": "boot_session_bound", + "id": "bundle-long-boot-session", + "input_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a31766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b9a34bf617c5b17787b697f1fae9b138052ef692aae68697529ca475bf98f5057be124acb4b0e8ff262d8e30ccbbd1390bcc6d02cf2b3732088fdbe8c9d93b2eae93150071d339bce7bff8b597daa5b32c64db8b6506f6912a7964521ad70fd0e4af8214a40164ad70e1ae8d2af0ae036548e74d62ba5af0fa97263c307d14e0d24650b06abb99b75e64456cbffd5de88e044a55c1f9fdb6cdee38bc0ad5e3ace73bfbd8bb81ea858c68e5b97f665b1e286efad1de4868ee6bbeb9ba04affed93cdb8c3f67dd1a5929e691b6ec72a369487d396e6557213857a067f3fb21d9a79baa49c86b93747ebe43af390bac5143d52f90291a7662e8b9e1edbda2152b9aaa14ba932647787676ec939757ad79feea4f3183bd3f16fffb8cf559af18b5eb370af72861021c58904a985f1ad257530aa6c0441b463e67db87690370914503d6215027ad78140b09cbb49308ef6cf1cce4589e2d1adc72cfc1b405418be738bb1c0089cbecbba4f20fb69d93d05032c7674360eea59df88310d2aff69cc136d6d160874c1c380909002b4b093c5654b0502b9592e89d49397505ab2847ba5085c09aff02fccd5fa915f7c953f5c777b793168c0db6d1864373a670561a1821c09d920b94d85f9bd2f9e91670848f522c1aeb9f747350eb12397148ecaea93eda009c8b86dedbadd99bf0a3cba87b836cb14b0bcdeaa83dbecbbe20cf1b3c44619d7a484d56b61b43b6eb0273293deab180168b1436ef901761dfecfe82b65376ec5eb7efc9f089652eb27e128cc726a2a5cf6b2be28fa59e34c9d73e4db0cddf2d8104eb91873d037a0e470867da49ef8e31a50c4f63c12891b3391f0d485f83fb7869b7e24801d274cc3a6776756d67df612b731eafd00b008f35322aa623bec94b043c66164f014567333bf6e72e80a86ae4afb0ad860fcf44dfb90ae4e91bb7a96ddacf24d34f0d3df25087c309ea37d9646d364c2415ce218935afc49ea81121ca47d4e99645129507ac82e5bc56dbef22a81eb6583200d8eacea029273fcf08436d68636641a3259d6645598fdc406b22422398534d098baf5620661cc2af5d89ec8068ef42b49040b7f10b04e18c8bcc730183cd4467f6b0e6e8005c4723a1c49fe22803eb056eb1483b7046d0f1c0f4b89a10adffe5f9bc76aa48d41dcdb9b98c861d17fc169056c978e1495fa39439bc9a63b6931d9a756c025d0eb3923a05c215528c110a1b171b8bf2dd422f3a60960e6b6c06f53ff34342e0e55c0b5df331f0dd734dfe3ae7a89b355a79c543ec970d2892738ea8b00dba120da689b52f69bb0dbc25f9e3b06f760b618f686bfc780762534984a0a4b32c174027dcedf32482f2634ea1b0015aad39435a1d0ac66c9bd6c586eaa70c66d3e7c59fb7778d73a9769873f4cff74d2111678d9f878c1fe858e764d6a88dbd924b47af4ec4ae66764b719e7fdb8636928692030b748873e15c443ae63cc47023b541ba20462bf97c8efefd1a49532fbedb04e85a4fc7ff094ffe6868f72147f9f3de80bdc527fe68010cc13fc414ff1d145cdb8668c184abb2ac4cfaa76a33e4bc8045f557139579435d40351b9e795b73083ecab686d1e6e1fb62e803eb96ce5129de5a022ebe9c86f03d6653b71370033de404fa2c9c6aaa20e520cabd8943f1b73ba6aa06fba48fe97c3bfd612d86ca9587105b56b2efa9f59b1e1346febafdcf01c27c496f8db8a8dbc093ad41ab53605f5c3fb072baa565c1a7ab47bb3dc5d7f4a811e2a7749d5f99ff21102a53bc1f2a812fd4a0d7f35e78c6dcf7bae137908a81983bd73cbda9799e14aab1e5c1eaf682dda0a1c3d2d31d0c0f258b24e68bd3e422078e66baf769741dcb166c52959bbaf7c26e626e9210b5801464e206fb3ea49d9bd14dcd12263f17a35e49d1324c5c849436ca0686b816f9eb08e9145307921226da8b5d8f1bdc3948c43d33b9f9dfdbd2a748e15abb25c203e308c34ef220ff39f38859534b173074bcd08200705896ed14f1e04399b8df7b37350dd1a6dfdc556776a334d8a0f6583a4e17e794bb90bdba247691adb2ab2b2841ae21f729b867d65bc70be004594dcfe895129a03a89d8b402f86c2af28c1b2c01225c97633591abfa1a13c9e5a9790ed4ce9c894fca11917cbe84bc88558c8fe6c4026a678270cdc8366f14642f2fb12c6d567cc1a76f3b14898e5c08d0073a83775ea2ecaa8bd409afd3cb249a45edc1f2d91ed621b3ed42d93c57f9dbd5772106efb64dd224f732288d269090018ee690de4d1902666410697e8e1ded391b1defd416fd9f5cba81f2b975cbdadf4cc55255b7a4f4c39b9ff551bc2bfb0ccfd585d76dfb2c57aeeb81e9ada1a8066f315f25d694669ea20723833b90d07421bc9c8d60569ef3ba6f14f418689235e9c179ea0cf80889b674eafdcb1e4058208989533a1e54b0d1e37d9b67c23e4264e6f13233e449888c0c869034dd646e31e382c3218ddbc4d2c243acc1a09dd3524332007146d88b371cdfb9b2144864caafde3568fcb5cbe1e6933378a5a1be2cd6dec4b62bb40d8479bd0005d80611e1e49d293d3c29b9226d85c677bd12d76de7dc11f53ecb319fe57f55262641cccfbe3401852c7f00a2e457be871aa549be6e4b029ec5d3e501647cefe0596360f71e66c6c161cc22bfa28aa83a85d5af14c6e1b9cf41abd3fd57400ca29dc64275bf26a66bfb691898928e48f66d739be784f4336ba6e76eeabf7cde614f1619872c50a5fc58030f3bc4f3d15b85c5b7933208b48fd4c2d1e64635e92d2f9f394e6a0237ab43fe68a78f7be6b50d3adfb1bba37b9724a4c8d327430691fc72de074ae27cc2fa63ecc2e2cd260f4da854346f7f03b035ca00945580ce74ecb044dac4b437376c3638dfb1ed3771d60d4404cede0d42fdebc57101f8716ee8ab611fc53c8dd4d736c99dbd1f652184d7912ab9e3d521333dc64958a445bf2f2b4499688cb2da85cdf3bd13faf398906136e18ee9c3ad5bfdd0f357df08253a40b915a4be6c08479faab15e048faffc608837a45757e3c035f98ebb2376371c56d541e74a387288c110fa24276b1a8cb7159cdc241e217cce4541e1832e83b8868abbf6cbbafb2639a3c1fd47ca24dcc3934ef1fa4880f37f672d495cf3b318b447daccac87a1f6bebff72e3e294291eeb3bbd84fee3fae1d44a18f2250936b09fd3e070cc02dd0cd39d582419a6b89885eb266f35a103692bf25107e796b55420830f47e9c64f7dfde7c8ee6ca88832f10aba62066938d0309b1a502fce5afe6e1a01a32cf21869e35f4f1c186051d9a756ba74712e4707f06c97f9844971681b1b489e61ed536b4c0c2a22cb7cb8ce40f1f7287061e79137876cc284336740007e83dfc4d35e5d9c2d5be8da88e94909b012b4adc8aec57fd3ea9d83e274913bc258f0e69c46cd24abadf03e2265b876e730ee4cdadef97b968711746383f86dd86e015a17a117578e4c3de6866b703f9b4c6038ceb6d0d02128ff48e8b045457eb60a5b9bf1e6a0e5c7917cafa217ba0f2e7caa962667b1529f511b14fe303ee63f6c957f10614164b70f65cb319e9ad28a99618bf9c1922c7507ec2db00da6046f342b515df877cc0459990e9d700d51558fd0c911d8b8a67aa7f0ee3ac63981802adfb5156f3d91d312e5a8514770fbbb4d077ccba19141b854e71a4642925d0dfdc150104a6997a83317e5310953473f78b7e461ea4c50786608f876e108670ab5739175eb7ad1601d7a7295977f8b17c88eee39fd0fbd3ec67ad7d5bd0cc70e8700dbf416e6e74f5f541b27d83b6488b3e38fdb5e5902baf9e309cd150903a4259c842640c95fcb66914a2cea132c494bd44c7d5b1c49dd054fb41611f274a0181b32753e82ae4e16c13fa284e72c9c14147c9a0973e69faeb331b962724a3ce9bcb05c19631af6cedcc9d3144a3bba50c3ca603b40a35ca2163f599f774b4622cf807a13484effe5ad050569376043e85673f52a4f7d32577f12e66d182813840dee356cf6b2cf8c9ec6f71060ff305484cf88f1d5b77910b3b1bb66c16dba272f9e486d0cbc3630aa3307aa67ae67fe5b4a3c53b7b4ded8b161c261471edf99791276ccf879b5788e27d4729b199d202f4c463fc91ddc307d6dfc810dd61312c901897abe124aa1baff3adf94cb7f849b14fe6e3bcd6fad830be0f01227405e9df2670fbae38dbbcb788f740068fba9cd5092d1d6b3fd812559d24c11ec160860c8cfc8756b2d8eb83e7df1548cd3757aa5e9bccf9ea7f95af99e251e472cf9777e78826a07a248f9673e6b518aa3277937e2571b58ee129a4c61de49f6ad0e7a28562f67f66647e54e8afe86bb7d795223890e407d29a0084c2358f4955fe36b7c0d27a3e2be9429e32ff5de479d152806a5f0c7176313ae97964b5cb1828a86ea649" + }, + { + "category": "hpke_context", + "entrypoint": "bundle_unwrap_context", + "expected_failure": "hpke_authentication", + "id": "bundle-wrong-context", + "input_hex": "7b2262756e646c655f666f726d6174223a2272657365617263682d62756e646c652d7631222c2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303938222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d" + }, + { + "category": "trailing_bytes", + "entrypoint": "receipt", + "expected_failure": "noncanonical_json", + "id": "receipt-leading-whitespace", + "input_hex": "207b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2233313831222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a2231222c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d" + }, + { + "category": "integral_bounds", + "entrypoint": "receipt", + "expected_failure": "noncanonical_decimal", + "id": "receipt-leading-zero", + "input_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2233313831222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a2231222c2266697273745f73657175656e63655f6e756d626572223a223031222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d" + }, + { + "category": "integral_bounds", + "entrypoint": "receipt", + "expected_failure": "wrong_type", + "id": "receipt-numeric-count", + "input_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2233313831222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a312c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d" + }, + { + "category": "integral_bounds", + "entrypoint": "receipt", + "expected_failure": "int64_overflow", + "id": "receipt-byte-count-overflow", + "input_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2239323233333732303336383534373735383038222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a2231222c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d" + }, + { + "category": "trailing_bytes", + "entrypoint": "receipt", + "expected_failure": "trailing_byte", + "id": "receipt-trailing-byte", + "input_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2233313831222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a2231222c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d00" + } + ], + "schema_version": 1, + "valid": { + "canonical_json": { + "canonical_jcs_utf8_hex": "7b225c72223a2243617272696167652052657475726e222c2231223a224f6e65222c22c280223a22436f6e74726f6c222c22c3b6223a224c6174696e20536d616c6c204c6574746572204f205769746820446961657265736973222c22e282ac223a224575726f205369676e222c22f09f9880223a22456d6f6a693a204772696e6e696e672046616365222c22efacb3223a22486562726577204c65747465722044616c6574205769746820446167657368227d" + }, + "bundle": { + "bundle_id": "00000000-0000-4000-8000-000000000099", + "container_hex": "414443455850303100000000000040008000000000000099fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7000b101112131415161718191a1b766563746f722d68706b655869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52bef94177bb023d9d7c5100833a3118c84c3622cffdfcf9ef269ab3301b3f743188f72cdcdec411e43f9e3b6ff4726081181a1862d419cec048c04a6740f6bc3dc40971f31f3d75eb2876381215eae53ba1bc3df6ff2732974b93e28dd4b3aebfefc9ef64b302a500b3ba6d9d2b1e80bd41e9e01083165b0ce61ae0568bfdd0636e8bb172c54541023f120fbd80918f5db8b1c22ba347f037bffc938708f0a186b1ff66d42027050f609b06bc367c43ab77b92b476c06dbe812f32f902dc3aa0adbda55006565875cc089e30dba2a649e73a6c91fb015d1127ca004c6678f0760ab4b5a6a37666f1b5dd32a91892153769d4be8a8ee8125ea1f840ebb2dd4ca01a9be433b1c5cf441f704f02e98e2ef48daa2353cb7c7c990b451d06a99eefe79231d13744f9ca044c7212de9e4d78af50805b48c2b6afac6ec497bc9d2f3204f4f2e9852d6ec0c4ecc82728eb02858ea3ec55e60b18c723b9407de281724bfa1760b7a93bf4cb067a4f2eaf16d7fe5b39be6d560b98bf8493bfcdc226eb3ce698af1d8fa7d32166b0627652086c7e01ae8f599341888cd4825ea9fde682d4f74d8fe9aa6b9d2a845eeacb30cde228bf8f018cd6d427c3b28f5748b046efd689542d9a4f6c84130458dbabc9df9fcf2b359ce5426062b1e7efbc37335e2b498163e0ba20fcacbcbd33bfbb25439bbc715f5df6b4643d47fdf479817f5d392cc3f39717ef3ef1fe2110ed5d7988782770ef08bad7625ae4208cfd3f5cd6cc04ba19ef48a7a11795b983ce0264a20ec850e11ff63a6cc589e09f7edf46c18fd72c545661ef9eac54028d9849bec0d88942636b37632ec22e70ea5aaa9566fc489a88a4b4c15138550018178e8900db4b7cff378db2fa43db0b8e9c405586736c26ad9566d36b12431474c383d191342a226182735a225ef7aea8e6f2b0d22d2ba92c6fe645341cdfb29b1d0961d073c1ac6233bdfb53ef8352c5d7285456046f2bac77ebac039c4f5dc6225815401291ea80e5250c361b156e78c570339f5048f62c5b69ec4d797d2114c02cf0123cb62ec92f480271587f0c64d98ef0e0ab0aa2d9c713bf7b74ab2e04d8a662921ff809c9b15b4f2a190462d7b4dda99fda98511ec260d1f22a2a69ef2406d06a0fc207c793ce9a479cc19335c746d48aa29022face0af22f158e2c4369d5a9747b43d12954bb2b82e8e042178b3c24c0b494a634c4b71d106693870a84fbaac03e40591582295bb94e256d7e075da00e78a4d2ff2b3dd9c6eb125c4493612f2c74070dd0487329cfb81244ab9bcb25646e40c3229f8e73a55429c8678d1b872ed674aac356c124d734f66e1cace3e5ecf8235a7734ba37c82c173ea43a2f92bc004e471a343be6fe7f404fb5a821365f32781bfb85347d25fd441d15e577d20901f53704195e8affc03368c0b3bd60aabe9ef14e854db1c9e6cd738d19ef1d20d506f7395fd096de963fee0ab7153be49322416426318aade21f20c02d7656d7bdc9aa54c7bc65e01ae7e0a92046c37385cf24bef7b860f2bfc18f28af6e0fb55c85f1c145a2aa1367daf479c326ff49b4e5f3e658057fe42af469aa29485c8a00cfc12da2d901058ffd2ed50915e76caa917b0e67e17fc5f29157694c30d798898af7f95a0ba07d91947c27199818283e8ce3a0adde1a4432c871e5322d863afd96b2159eb014ef98c6db40d6504355584d8fc47e249039cfffe1d1265ec037820a7a50613d4276caee2974e2e7b3d1651aea7855b0dbdd5555c29ca95f0bf919e6a5b0505148c5695193153aec27bdb02ce8da800a94ab0050fdebbc4c4ff50a169b9f262fca3cef3158bc844a9850ceed6ccee5f671bce70855a8efcee9a9543636c96cbbe6bd62a8b3a50259e8bf76118439f1eedc7d4e6c4aec08fcdd00184282b3d14624c9e0bf8f9c8b9b419bbd4bf3e02025f65b0828c94c4e2d77ca2e4f2781adac547b8fb4d9c75803ab55d72241945473b17dc819b4065f76469008d39825ebe36b9723014f72e528b9c1a083ddaba882a82decc88ff7c98c56e11ab722c79498ea1ce8ee53f367c23645eb78d0df19e6eb1803c8789455c72b2f2b087dc6b96c163f3cc17aec950b38ad776897c4af84afeadae1f0f07c5fc57d6fe7f0498e97c5843aab30da8d9bd8591d52252efd3ad13447cd210fed5985165714d871771a338e4cdf3afd9f8aab2abb7abb54b3096fed9ba818898fefd88db1973808a2c9aacda2515382375738cd3198624e45695fc3f9eb7b773c6373b8cd6cb749ea40bfa42911bdc366cf807a226a6d6020c3a4e838c6b0e1a30cff1c4215533430533d900c7aec49a172647246b02ca14fffdd9758a588423ea8a0c543b77edffb2147aa1801c4303abd780b21a3767341c78cd6ae526dd33a2bcce1c0cce433f08940df280da20a226da5e1f9dfc785b921e6e4c122f87d173999bf72c7a2b1afb925147dc823e9f0db2f3f234ee67d753d8ff2fbc54844fa1db423a45bda9b800680969902026de586b9684ecf6f117a38de98f2ccb4fc30028ab0464d9b781c95b10733f7c93ad61b0f848e8041ee4af5dc0724080836c65d015ba81777aa0ba484cb05e54ccb78bc32a8663a5379672e8ec9bd85d766b482ac661942a2f129f756f806577a09982fb15a39c75230ea1813f2b0f1e7c886100575521a230adbb3d466c020c8c943d7af51df009833d4cae36b2dee8fee49ce5bb892063a98aa9db5aadfc2014abcd69860c851a51afce466e09d0a111a80a331b62bcbd90ce44cda1535e7d8ce1dd3f6aec2b759edc4a258ddb87cc17f50214063f2b8976450aa2c8f07ef7b13152cc0760359a441ab31136554b8fc901ad9e08b312bfb180de3f9cf90e93b5d15a6be4f5482a528d194ce9be31274d185dfd9a8acc9866146393780d55af30f9ede8aa16093477991f1eeb9dab270c64b586496f017c5b80fb5c796faa2e5964d33ec8b61785a2c8b4b725d2033cbeca9a76a1183fd7456628e96cc4787a2c53b04519e5257d4be45d7142b52a733c19a6fee2eb36f67129d315521499a03943bb8d25d869c2f7beecc71102e31b9be1f5c5c999a172647d6003833b5a6f89f3d1ecfc66cf0dd78959b60eaf79adecbc0d7ff6da7cdd4edd85b75d2306445a3ab6dee5d08c1d2ce4a2be5944a962bc03aecb29edc15de96c16cedb810af7d4ca841f940c7a404f7ff5b8fd208ee9009952d87d783464e8611afd05a8767cb90ec3dcd1c697bdf5eb54776aafb7e29d1125642524a7312bec6ab442eea2ab5d9e4ad2cf98108a24a4fe9847611cc5b758470bd93d1af444e7d23f1d883b8b94944191993edf45eb862d37ff1819d87b1c2b8a3bd47dc589bc36d26a2ec2190b97970a29af2fa6d89edb90fb4785142ab8b4f250b0e6a4086f7d4397665c1e4840a9d809bec3b306aa8e5530d69da39289443876b79ffabdce5d4a774acab008efcedaaa671dba891ab013dff640c9799e572e1768e49d91e7abb6110e4e50109e2ba94aa42b3891cba732061a1c341c43f8dcc6242a77f18c7beeac74e4d021454cf1ade2dfdfa61384bb8c0b51913f29657cd0139f9fbdc359424a65a5737c9e2d36d7f44d8b47e4474fbfd5919718f7e95e4fa3b5eaca5cd1f6dcb68cf559e3d3d964db726ce5ae56bc624b301c94d4083411458e8a3acfb37606397dcc70e6a4a1a42b19caaedb0fc77fd41b6bd91541a28e9b2d44130ae38749bb0c6d37a2f00186726af1259b993f7c8965a87b6fe16bb8049fbbb592fa48c90def4b61d95fc123d948ece30e489626b4e004e183dbac2f58554c9e3c3428596ff284cc8ab5f7b3dddb3736dc4212744624a0c7589c0526b3b03ce88a57bb02bcb70f8daf230a56938337f1143dac4367e61ad80e1847bb594b39ee829acd0c2e739a21cda51797a8b96e907fe29bd0dc97697538ebc4b6688e3f61e7172980afe0048d9db619b71f11e087f9a364ffa75183f19c3e6d4d9e2cb9f4379d4f5d1495a6daeefcd2c84931bf9bbce7f0f207496cbfbe28770c821f3c35ad6400808fd7c66fc4eba0a3cc03c7a25e47b8de9579553b45eabc1e09db7ef86362689f7da2352a6ec14cd9bb37387c253705a29444260a1ababa3c6f84885899a194588efeb1115731ab0d41e69ad618a74318ce06b2bd85255982e35f9dabb81d8a89c09920531d340f4feccd9caf7efd7c4062adc092f5974e6df5dfb3bbbbbc9caa8db06428f5d72482bc626968c71e8636268f596517721ca63", + "content_key_hex": "a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf", + "content_nonce_hex": "101112131415161718191a1b", + "context_jcs_utf8_hex": "7b2262756e646c655f666f726d6174223a2272657365617263682d62756e646c652d7631222c2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d", + "document_jcs_utf8_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c2262756e646c655f6b696e64223a226175746f6d617469635f75706c6f6164222c22636f6e66696775726174696f6e223a7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d2c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c22636f6e66696775726174696f6e5f7369676e6174757265223a7b227369676e6174757265223a22495a5a5142345a4b74766e34724d7a627479344c6930354b424756516871645476723659526f5a45396466346e563331763754726d33362d687376656b6d5233696e564a536379467772306e713567464c4e6a784251222c227369676e65725f6b65795f6964223a22766563746f722d7369676e6572227d2c226578706572696d656e74223a7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c2264757261626c655f7468726f7567685f73657175656e6365223a2231222c226576656e745f636f756e74223a2231222c226576656e7473223a5b7b22636f6c6c6563746f725f6964223a226170705f6c6966656379636c652e7631222c226669656c6473223a7b2261637469766974795f636c617373223a22766563746f722e4163746976697479227d2c226f627365727665645f74696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a2232303030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a2231303030227d2c227061796c6f61645f736368656d615f76657273696f6e223a312c227061796c6f61645f74797065223a2241435449564954595f43524541544544222c2273657175656e63655f6e756d626572223a2231227d5d2c226578706572696d656e745f6964223a22766563746f722d7374756479222c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c226e6578745f73657175656e63655f6e756d626572223a2232222c227061727469636970616e745f696e7374616e63655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303137222c2272657461696e65645f66726f6d5f73657175656e6365223a2231222c227374617465223a2252554e4e494e47222c227472616e736974696f6e73223a5b7b2266726f6d223a22494d504f52544544222c22726561736f6e223a22434f4e46494755524154494f4e5f5349474e41545552455f5645524946494544222c2274696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a22313030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a22313030227d2c22746f223a22434f4e4649475f5645524946494544227d2c7b2266726f6d223a22434f4e4649475f5645524946494544222c22726561736f6e223a22434f4e53454e545f5245564945575f4f50454e4544222c2274696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a22323030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a22323030227d2c22746f223a22434f4e53454e545f50454e44494e47227d2c7b2266726f6d223a22434f4e53454e545f50454e44494e47222c22726561736f6e223a22434f4e53454e545f4143434550544544222c2274696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a22333030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a22333030227d2c22746f223a224143434553535f5345545550227d2c7b2266726f6d223a224143434553535f5345545550222c22726561736f6e223a224143434553535f505245464c494748545f504153534544222c2274696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a22343030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a22343030227d2c22746f223a225245414459227d2c7b2266726f6d223a225245414459222c22726561736f6e223a225041525449434950414e545f53544152544544222c2274696d65223a7b22626f6f745f73657373696f6e5f6964223a22626f6f742d766563746f72222c226d6f6e6f746f6e69635f74696d655f6e616e6f73223a22353030222c2277616c6c5f74696d655f7574635f6d696c6c6973223a22353030227d2c22746f223a2252554e4e494e47227d5d2c2275706c6f616465645f7468726f7567685f73657175656e6365223a2230227d2c226578706f727465645f61745f7574635f6d696c6c6973223a223130303030222c22666f726d6174223a2272657365617263682d62756e646c652d7631222c2270726f6475636572223a7b22636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964227d7d", + "hpke_ephemeral_private_key_base64url": "ISIjJCUmJygpKissLS4vMDEyMzQ1Njc4OTo7PD0-P0A", + "hpke_wrapped_content_key_hex": "5869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67bf9ca6ad1d6441a511ad18b4ecc5d2b040d17f440f82f6d1364bbca654daceadfa87148fb29ae53022d45942d6342b52b", + "researcher_private_key_base64url": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVpbXF1eX2A", + "researcher_public_key_base64url": "ZLEBsdC-WocEvQePmJUAH8A-jp-VIvGI3RKNmEbUhGY", + "sha256": "70498d01436008bb453e78daf834f8560228d8246a752522f91f01ac8d31c68e" + }, + "signed_configuration": { + "canonical_jcs_sha256": "fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7", + "canonical_jcs_utf8_hex": "7b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d", + "envelope_hex": "4144434346473031000d000003c9766563746f722d7369676e65727b2261737369676e65645f7061727469636970616e745f6964223a6e756c6c2c22636f6c6c6563746f7273223a5b7b22636f6e666967223a7b226d6178696d756d5f7265706f72745f6c6174656e63795f7573223a313030303030302c2273616d706c696e675f706572696f645f7573223a3130303030307d2c226964223a22616363656c65726f6d657465722e7631222c227265717569726564223a66616c73657d2c7b22636f6e666967223a7b7d2c226964223a226170705f6c6966656379636c652e7631222c227265717569726564223a747275657d5d2c22636f6e66696775726174696f6e5f6964223a22766563746f722d636f6e666967222c22636f6e73656e74223a7b22646f63756d656e745f76657273696f6e223a227631222c2273756d6d617279223a2250726f746f636f6c20766563746f7220636f6e73656e742e227d2c226475726174696f6e5f686f757273223a32342c226578706572696d656e745f6964223a22766563746f722d7374756479222c22657870697265735f6174223a22323033302d30312d30315430303a30303a30305a222c226578706f7274223a7b2268706b655f7075626c69635f6b6579223a225a4c45427364432d576f6345765165506d4a55414838412d6a702d564976474933524b4e6d456255684759222c22726573656172636865725f6b65795f6964223a22766563746f722d68706b65227d2c22696e74657276656e74696f6e73223a5b5d2c226973737565645f6174223a22323032362d30312d30315430303a30303a30305a222c226d696e696d756d5f636c69656e745f76657273696f6e223a2237222c22706c6174666f726d223a22616e64726f6964222c22707572706f7365223a224578657263697365207468652064657374727563746976652050726f746f636f6c20763120636f6e74726163742e222c2272657365617263686572223a7b22636f6e74616374223a22766563746f72406578616d706c652e696e76616c6964222c226e616d65223a2250726f746f636f6c20566563746f72227d2c22736368656d615f76657273696f6e223a312c227369676e6572223a7b226b65795f6964223a22766563746f722d7369676e6572222c227075626c69635f6b6579223a22656256574c6f5f6d56506c41654c4553364b6d4c703541666854726d6c623758344f4f52433630456c6d51227d2c2273746f72616765223a7b226d6178696d756d5f6c6f63616c5f6279746573223a31363737373231367d2c2273757276657973223a5b5d2c227469746c65223a2250726f746f636f6c20766563746f72222c2275706c6f6164223a7b7d7d21965007864ab6f9f8acccdbb72e0b8b4e4a04655086a753bebe98468644f5d7f89d5df5bfb4eb9b7ebe86cbde9264778a754949cc85c2bd27ab98052cd8f105", + "signature_base64url": "IZZQB4ZKtvn4rMzbty4Li05KBGVQhqdTvr6YRoZE9df4nV31v7Trm36-hsvekmR3inVJScyFwr0nq5gFLNjxBQ", + "signer_key_id": "vector-signer", + "signer_private_key_base64url": "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyA", + "signer_public_key_base64url": "ebVWLo_mVPlAeLES6KmLp5AfhTrmlb7X4OORC60ElmQ" + }, + "upload_receipt": { + "canonical_jcs_utf8_hex": "7b2262756e646c655f6964223a2230303030303030302d303030302d343030302d383030302d303030303030303030303939222c22627974655f636f756e74223a2233313831222c22636f6e66696775726174696f6e5f736861323536223a2266623264666561363338636136323130653764313562663132653962663363393130303964353463386135383164633361343737616363643732326262396337222c226576656e745f636f756e74223a2231222c2266697273745f73657175656e63655f6e756d626572223a2231222c226c6173745f73657175656e63655f6e756d626572223a2231222c22736861323536223a2237303439386430313433363030386262343533653738646166383334663835363032323864383234366137353235323266393166303161633864333163363865227d", + "value": { + "bundle_id": "00000000-0000-4000-8000-000000000099", + "byte_count": "3181", + "configuration_sha256": "fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7", + "event_count": "1", + "first_sequence_number": "1", + "last_sequence_number": "1", + "sha256": "70498d01436008bb453e78daf834f8560228d8246a752522f91f01ac8d31c68e" + } + } + } +} diff --git a/protocol/v1/join-link-vectors.json b/protocol/v1/join-link-vectors.json new file mode 100644 index 0000000..635537a --- /dev/null +++ b/protocol/v1/join-link-vectors.json @@ -0,0 +1,48 @@ +{ + "corpus_format": "adc-join-link-conformance-v1", + "schema_version": 1, + "valid": { + "artifact_url": "https://artifacts.example.invalid/join/dGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4", + "artifact_sha256": "0000000000000000000000000000000000000000000000000000000000000000", + "signer_fingerprint": "0123456789ABCDEFFEDCBA9876543210", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fartifacts.example.invalid%2Fjoin%2FdGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + "hostile": [ + { + "id": "url-uppercase-default-port-dot-segment", + "encoded": "adc://join/v1?artifact=https%3A%2F%2FEXAMPLE.invalid%3A443%2Fa%2F..%2Fconfig.adccfg&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-query", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fartifacts.example.invalid%2Fconfig.adccfg%3Fdownload%3D1&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-credentials", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fuser%40artifacts.example.invalid%2Fconfig.adccfg&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-fragment", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fartifacts.example.invalid%2Fconfig.adccfg%23mutable&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-empty-segment", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fartifacts.example.invalid%2Fa%2F%2Fconfig.adccfg&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-percent-escape", + "encoded": "adc://join/v1?artifact=https%3A%2F%2Fartifacts.example.invalid%2Fa%2F%2563onfig.adccfg&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "url-numeric-host", + "encoded": "adc://join/v1?artifact=https%3A%2F%2F127.0.0.1%2Fconfig.adccfg&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "join-lowercase-escape", + "encoded": "adc://join/v1?artifact=https%3A%2f%2Fartifacts.example.invalid%2Fjoin%2FdGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4&sha256=0000000000000000000000000000000000000000000000000000000000000000&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + }, + { + "id": "join-query-order", + "encoded": "adc://join/v1?sha256=0000000000000000000000000000000000000000000000000000000000000000&artifact=https%3A%2F%2Fartifacts.example.invalid%2Fjoin%2FdGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4&signer_fingerprint=0123456789ABCDEFFEDCBA9876543210" + } + ] +} diff --git a/researcher-tools/build.gradle.kts b/researcher-tools/build.gradle.kts index 730b33c..3e0aa10 100644 --- a/researcher-tools/build.gradle.kts +++ b/researcher-tools/build.gradle.kts @@ -23,6 +23,10 @@ tasks.named("run") { workingDir(rootProject.projectDir) } +tasks.withType().configureEach { + systemProperty("adc.repository.root", rootProject.projectDir.absolutePath) +} + dependencies { implementation(project(":core:crypto")) implementation(project(":core:export")) diff --git a/researcher-tools/examples/INSECURE-demo-hpke-private.json b/researcher-tools/examples/INSECURE-demo-hpke-private.json deleted file mode 100644 index 36bdb14..0000000 --- a/researcher-tools/examples/INSECURE-demo-hpke-private.json +++ /dev/null @@ -1 +0,0 @@ -{"primaryKeyId":218992727,"key":[{"keyData":{"typeUrl":"type.googleapis.com/google.crypto.tink.HpkePrivateKey","value":"EioSBggBEAEYAhogGnJDfDh8XH1eARTHmTLNlog9curtphTpEn7L36sY0QAaIEe45n70BzP/OGLnW69bUiUYgff5rmYB9l+zA+tPTeSq","keyMaterialType":"ASYMMETRIC_PRIVATE"},"status":"ENABLED","keyId":218992727,"outputPrefixType":"TINK"}]} diff --git a/researcher-tools/examples/INSECURE-demo-hpke-private.key b/researcher-tools/examples/INSECURE-demo-hpke-private.key new file mode 100644 index 0000000..b02faa1 --- /dev/null +++ b/researcher-tools/examples/INSECURE-demo-hpke-private.key @@ -0,0 +1 @@ +R7jmfvQHM_84Yudbr1tSJRiB9_muZgH2X7MD609N5Ko diff --git a/researcher-tools/examples/INSECURE-demo-signing-private.key b/researcher-tools/examples/INSECURE-demo-signing-private.key index f24e5ac..f2ebdb8 100644 --- a/researcher-tools/examples/INSECURE-demo-signing-private.key +++ b/researcher-tools/examples/INSECURE-demo-signing-private.key @@ -1 +1 @@ -MC4CAQAwBQYDK2VwBCIEIAmi71ZFO/KyvZwT1PajCAqBeBtssu3Hit6BvREAIgMh +CaLvVkU78rK9nBPU9qMICoF4G2yy7ceK3oG9EQAiAyE diff --git a/researcher-tools/examples/README.md b/researcher-tools/examples/README.md index c59f570..2f59cc3 100644 --- a/researcher-tools/examples/README.md +++ b/researcher-tools/examples/README.md @@ -4,12 +4,17 @@ | File | What it is | | --- | --- | -| `INSECURE-demo-signing-private.key` | Ed25519 study signing private key — public fixture | -| `INSECURE-demo-hpke-private.json` | Tink HPKE private keyset for export decryption — public fixture | +| `INSECURE-demo-signing-private.key` | Raw Ed25519 study signing private key, unpadded base64url — public fixture | +| `INSECURE-demo-hpke-private.key` | Raw X25519 HPKE private key, unpadded base64url — public fixture | | `demo-study.json` | An example study configuration, useful as a schema reference | These exist so a debug build can exercise signing and export decryption end to end, and so the example configuration in the [researcher guide](../../docs/researcher-guide.md) is runnable. That is their only purpose. The signing public key is not stored separately: it travels inside the configuration, as `demo-study.json`'s `signer.public_key`. +`demo-study.json` is intentionally formatted for people to read. Run `canonicalize` before `sign`; only the resulting RFC 8785 bytes are valid Protocol v1 signing input. +The signed result mirrored at `app/src/debug/res/raw/demo_study_envelope.txt` is verified by +`DemoStudyAssetTest`; changing this configuration or its signing key requires regenerating that +asset through the same `canonicalize` and `sign` commands. + The example is an anonymous v1 configuration with one localized short-answer survey and one one-time survey intervention. It deliberately has no researcher-assigned participant code. **Never use them for a real study.** A study signed with these keys is not authentic, and an export encrypted to this HPKE key is readable by anyone who clones this repository. diff --git a/researcher-tools/examples/demo-study.json b/researcher-tools/examples/demo-study.json index 5110a1b..b4ea1cd 100644 --- a/researcher-tools/examples/demo-study.json +++ b/researcher-tools/examples/demo-study.json @@ -5,7 +5,8 @@ "assigned_participant_id": null, "issued_at": "2026-01-01T00:00:00Z", "expires_at": "2035-01-01T00:00:00Z", - "minimum_app_version": 1, + "platform": "android", + "minimum_client_version": "1", "title": "Modular sensing demonstration", "researcher": { "name": "Android Data Collector maintainers", @@ -63,7 +64,7 @@ "interval_millis": 10000, "minimum_interval_millis": 5000, "maximum_batch_delay_millis": 30000, - "minimum_displacement_meters": 5, + "minimum_displacement_millimeters": 5000, "priority": "BALANCED" } }, @@ -129,25 +130,11 @@ }, "signer": { "key_id": "demo-signer-2026", - "public_key": "MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0=" + "public_key": "sRSaTpZmTSBL7eN6nS_HBsNmLM8n1hdRmIt1vtLZsC0" }, "export": { "researcher_key_id": "demo-hpke-2026", - "tink_hpke_public_keyset": { - "primaryKeyId": 218992727, - "key": [ - { - "keyData": { - "typeUrl": "type.googleapis.com/google.crypto.tink.HpkePublicKey", - "value": "EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA", - "keyMaterialType": "ASYMMETRIC_PUBLIC" - }, - "status": "ENABLED", - "keyId": 218992727, - "outputPrefixType": "TINK" - } - ] - } + "hpke_public_key": "GnJDfDh8XH1eARTHmTLNlog9curtphTpEn7L36sY0QA" }, "upload": {} } diff --git a/researcher-tools/src/main/kotlin/cool/linc/androiddatacollector/researcher/Main.kt b/researcher-tools/src/main/kotlin/cool/linc/androiddatacollector/researcher/Main.kt index b6bf288..98ec134 100644 --- a/researcher-tools/src/main/kotlin/cool/linc/androiddatacollector/researcher/Main.kt +++ b/researcher-tools/src/main/kotlin/cool/linc/androiddatacollector/researcher/Main.kt @@ -1,21 +1,26 @@ package cool.linc.androiddatacollector.researcher import cool.linc.androiddatacollector.core.crypto.HpkeCrypto +import cool.linc.androiddatacollector.core.definition.ProtocolBase64Url +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec import cool.linc.androiddatacollector.core.export.ResearchExport +import cool.linc.androiddatacollector.core.export.ResearchBundleVerifier import cool.linc.androiddatacollector.core.protocol.ConfigurationVerifier import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec import cool.linc.androiddatacollector.core.protocol.SignedConfigurationEnvelope -import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import java.nio.channels.FileChannel import java.nio.file.Files import java.nio.file.Path import java.nio.file.StandardCopyOption +import java.nio.file.StandardOpenOption +import java.nio.file.attribute.PosixFilePermission +import java.nio.file.attribute.PosixFilePermissions import java.security.KeyFactory import java.security.KeyPairGenerator import java.security.Signature import java.security.spec.PKCS8EncodedKeySpec import java.security.spec.X509EncodedKeySpec import java.time.Instant -import java.util.Base64 fun main(arguments: Array) { require(arguments.isNotEmpty()) { usage() } @@ -34,14 +39,14 @@ fun main(arguments: Array) { private fun signingKeygen(args: Arguments) { val pair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair() - writeNew(args.path("--private"), Base64.getEncoder().encode(pair.private.encoded)) - writeNew(args.path("--public"), Base64.getEncoder().encode(pair.public.encoded)) + writeNew(args.path("--private"), ProtocolBase64Url.encode(pair.private.encoded.requirePrefix(ED25519_PKCS8_PREFIX)).toByteArray()) + writeNew(args.path("--public"), ProtocolBase64Url.encode(pair.public.encoded.requirePrefix(ED25519_X509_PREFIX)).toByteArray()) } private fun hpkeKeygen(args: Arguments) { - val pair = HpkeCrypto.generateKeyset() - writeNew(args.path("--private"), pair.privateKeysetJson.toByteArray(Charsets.UTF_8)) - writeNew(args.path("--public"), pair.publicKeysetJson.toByteArray(Charsets.UTF_8)) + val pair = HpkeCrypto.generateKeyPair() + writeNew(args.path("--private"), ProtocolBase64Url.encode(pair.privateKey).toByteArray()) + writeNew(args.path("--public"), ProtocolBase64Url.encode(pair.publicKey).toByteArray()) } private fun canonicalize(args: Arguments) { @@ -124,7 +129,9 @@ private fun signEnvelope( privateKeyBase64: String, ): ByteArray { val privateKey = KeyFactory.getInstance("Ed25519").generatePrivate( - PKCS8EncodedKeySpec(Base64.getDecoder().decode(privateKeyBase64.trim())), + PKCS8EncodedKeySpec( + ED25519_PKCS8_PREFIX + ProtocolBase64Url.decodeExact(privateKeyBase64.trim(), 32, "Ed25519 private key"), + ), ) val signature = Signature.getInstance("Ed25519").run { initSign(privateKey) @@ -132,7 +139,9 @@ private fun signEnvelope( sign() } val declaredKey = KeyFactory.getInstance("Ed25519").generatePublic( - X509EncodedKeySpec(Base64.getDecoder().decode(declaredPublicKey)), + X509EncodedKeySpec( + ED25519_X509_PREFIX + ProtocolBase64Url.decodeExact(declaredPublicKey, 32, "Ed25519 public key"), + ), ) require(Signature.getInstance("Ed25519").run { initVerify(declaredKey) @@ -158,7 +167,7 @@ private fun checkConfig(args: Arguments) { } ?: emptyMap() val verified = ConfigurationVerifier( trustedSigningKeys = pinned, - appVersionCode = args.optionalValue("--app-version")?.toInt() ?: Int.MAX_VALUE, + clientVersion = args.optionalValue("--app-version")?.toLong() ?: Long.MAX_VALUE, now = { args.optionalValue("--now")?.let(Instant::parse) ?: Instant.now() }, ).verify(Files.readAllBytes(args.path("--envelope"))) val configuration = verified.configuration @@ -171,17 +180,46 @@ private fun decrypt(args: Arguments) { val configuration = StudyConfigurationCodec.decode(Files.readAllBytes(args.path("--config"))) val output = args.path("--output") require(!Files.exists(output)) { "Refusing to overwrite ${output.toAbsolutePath()}" } - // Staged, because the AES-GCM tag is only checked once the last byte has been read: a tampered - // bundle writes plausible-looking plaintext right up until it fails. Nothing appears at the - // destination unless verification succeeded. - val staging = Files.createTempFile(output.toAbsolutePath().parent, ".adc-decrypt", ".tmp") + val outputParent = requireNotNull(output.toAbsolutePath().parent) { "Output needs a parent directory" } + require(Files.isDirectory(outputParent)) { "Output parent does not exist: $outputParent" } + // AEAD authenticates only at EOF, and the authenticated plaintext still has a closed-world + // schema to prove. Keep both phases in a private staging file; publish only their joint result. + val staging = Files.createTempFile( + outputParent, + ".adc-decrypt", + ".tmp", + PosixFilePermissions.asFileAttribute( + setOf(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_WRITE), + ), + ) try { - Files.newInputStream(args.path("--bundle")).use { input -> - Files.newOutputStream(staging).use { plaintext -> - ResearchExport.decrypt(input, plaintext, Files.readString(args.path("--private")), configuration) + val header = Files.newInputStream(args.path("--bundle")).use { input -> + Files.newOutputStream( + staging, + StandardOpenOption.WRITE, + StandardOpenOption.TRUNCATE_EXISTING, + ).use { plaintext -> + ResearchExport.decrypt( + input, + plaintext, + ProtocolBase64Url.decodeExact( + Files.readString(args.path("--private")).trim(), + HpkeCrypto.RAW_KEY_BYTES, + "X25519 private key", + ), + configuration, + ) } } - Files.move(staging, output) + val verified = Files.newInputStream(staging).use { plaintext -> + ResearchBundleVerifier.verify(plaintext, header, configuration) + } + FileChannel.open(staging, StandardOpenOption.WRITE).use { it.force(true) } + Files.move(staging, output, StandardCopyOption.ATOMIC_MOVE) + println( + "verified ${verified.header.bundleId} ${verified.experiment.firstSequenceNumber}-" + + "${verified.experiment.lastSequenceNumber}", + ) } catch (failure: Throwable) { Files.deleteIfExists(staging) throw failure @@ -225,3 +263,17 @@ private fun usage(): String = """ check-config --envelope FILE [--public FILE --key-id ID] [--app-version N] [--now ISO_INSTANT] decrypt --bundle FILE --private FILE --config FILE --output FILE """.trimIndent() + +private fun ByteArray.requirePrefix(prefix: ByteArray): ByteArray { + require(size == prefix.size + 32 && copyOfRange(0, prefix.size).contentEquals(prefix)) { + "Unexpected JCA raw-key encoding" + } + return copyOfRange(prefix.size, size) +} + +private val ED25519_X509_PREFIX = byteArrayOf( + 0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00, +) +private val ED25519_PKCS8_PREFIX = byteArrayOf( + 0x30, 0x2e, 0x02, 0x01, 0x00, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x04, 0x22, 0x04, 0x20, +) diff --git a/researcher-tools/src/test/kotlin/cool/linc/androiddatacollector/researcher/DecryptCommandTest.kt b/researcher-tools/src/test/kotlin/cool/linc/androiddatacollector/researcher/DecryptCommandTest.kt new file mode 100644 index 0000000..472ee5f --- /dev/null +++ b/researcher-tools/src/test/kotlin/cool/linc/androiddatacollector/researcher/DecryptCommandTest.kt @@ -0,0 +1,95 @@ +package cool.linc.androiddatacollector.researcher + +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.attribute.PosixFilePermission +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertThrows +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder + +class DecryptCommandTest { + @get:Rule + val temporary = TemporaryFolder() + + private val corpus: JsonObject by lazy { + val repository = Path.of(requireNotNull(System.getProperty("adc.repository.root"))) + JsonParser.parseString( + Files.readString(repository.resolve("protocol/v1/conformance-vectors.json")), + ).asJsonObject + } + + @Test + fun validBundleIsPrivateAndPublishedOnlyAfterCompleteVerification() { + val files = inputs(corpus.getAsJsonObject("valid").getAsJsonObject("bundle").get("container_hex").asString) + + main(files.arguments()) + + assertArrayEquals( + corpus.getAsJsonObject("valid").getAsJsonObject("bundle") + .get("document_jcs_utf8_hex").asString.hex(), + Files.readAllBytes(files.output), + ) + assertEquals( + setOf(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_WRITE), + Files.getPosixFilePermissions(files.output), + ) + } + + @Test + fun authenticatedButInvalidDocumentPublishesNothingAndDeletesStaging() { + val vector = corpus.getAsJsonArray("hostile") + .map { it.asJsonObject } + .single { it.get("id").asString == "bundle-unknown-payload" } + val files = inputs(vector.get("input_hex").asString) + + assertThrows(Exception::class.java) { main(files.arguments()) } + + assertFalse(Files.exists(files.output)) + Files.list(files.output.parent).use { entries -> + assertFalse(entries.anyMatch { it.fileName.toString().startsWith(".adc-decrypt") }) + } + } + + private fun inputs(bundleHex: String): CommandFiles { + val directory = temporary.newFolder().toPath() + val valid = corpus.getAsJsonObject("valid") + val signed = valid.getAsJsonObject("signed_configuration") + val bundle = valid.getAsJsonObject("bundle") + return CommandFiles( + bundle = directory.resolve("input.adcexp").also { Files.write(it, bundleHex.hex()) }, + privateKey = directory.resolve("private.key").also { + Files.writeString(it, bundle.get("researcher_private_key_base64url").asString) + }, + configuration = directory.resolve("configuration.json").also { + Files.write(it, signed.get("canonical_jcs_utf8_hex").asString.hex()) + }, + output = directory.resolve("output.json"), + ) + } + + private fun String.hex(): ByteArray { + require(length % 2 == 0 && matches(Regex("[0-9a-f]*"))) + return ByteArray(length / 2) { index -> substring(index * 2, index * 2 + 2).toInt(16).toByte() } + } + + private data class CommandFiles( + val bundle: Path, + val privateKey: Path, + val configuration: Path, + val output: Path, + ) { + fun arguments() = arrayOf( + "decrypt", + "--bundle", bundle.toString(), + "--private", privateKey.toString(), + "--config", configuration.toString(), + "--output", output.toString(), + ) + } +} diff --git a/settings.gradle.kts b/settings.gradle.kts index c22c888..de7b86b 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -20,10 +20,16 @@ include( ":app", ":collector:accelerometer", ":collector:app-lifecycle", + ":collector:ambient-light", + ":collector:battery-state", + ":collector:gyroscope", ":collector:keyboard-ime", ":collector:location", ":collector:network-state", ":collector:network-usage", + ":collector:proximity", + ":collector:sensor-common", + ":collector:temporal-context", ":collector:usage-events", ":core:access", ":core:collector-api", diff --git a/tools/__init__.py b/tools/__init__.py new file mode 100644 index 0000000..bbe8f66 --- /dev/null +++ b/tools/__init__.py @@ -0,0 +1 @@ +"""Repository-local validation tools.""" diff --git a/tools/catalog.py b/tools/catalog.py new file mode 100644 index 0000000..53cb16d --- /dev/null +++ b/tools/catalog.py @@ -0,0 +1,504 @@ +#!/usr/bin/env python3 +"""Validate and canonicalize the closed-world ADC collector catalog.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import sys +from pathlib import Path +from typing import Any, Iterable + + +ROOT = Path(__file__).resolve().parents[1] +DEFAULT_CATALOG = ROOT / "protocol/v1/collector-catalog.json" +DEFAULT_KOTLIN_CONTRACT = ( + ROOT + / "core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt" +) +ID = re.compile(r"[a-z][a-z0-9_.-]{2,63}\Z") +FIELD = re.compile(r"[a-z][a-z0-9_]{1,63}\Z") +EVENT_TYPE = re.compile(r"[A-Z][A-Z0-9_]{1,63}\Z") +MODULE = re.compile(r":[a-z][a-z0-9-]*(?::[a-z][a-z0-9-]*)*\Z") + + +class CatalogError(ValueError): + """The catalog is not the one unambiguous language-neutral schema.""" + + +def _reject_duplicate(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise CatalogError(f"duplicate JSON member: {key}") + result[key] = value + return result + + +def _reject_float(value: str) -> None: + raise CatalogError(f"JSON non-integral number is forbidden: {value}") + + +def load(path: Path) -> dict[str, Any]: + try: + value = json.loads( + path.read_text(encoding="utf-8"), + object_pairs_hook=_reject_duplicate, + parse_float=_reject_float, + parse_constant=_reject_float, + ) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise CatalogError(str(error)) from error + if not isinstance(value, dict): + raise CatalogError("catalog root must be an object") + return value + + +def canonical_bytes(value: Any) -> bytes: + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + + +def _kotlin_string(value: str) -> str: + return json.dumps(value, ensure_ascii=False) + + +def _kotlin_double(value: int) -> str: + return f"{value}.0" + + +def render_kotlin_contract(catalog: dict[str, Any]) -> str: + """Generate the sole Kotlin runtime projection of implemented event contracts.""" + collectors = [ + collector + for collector in catalog["collectors"] + if collector["implementation"]["status"] == "implemented" + and "android" in collector["platforms"] + ] + lines = [ + "// Generated by tools/catalog.py from protocol/v1/collector-catalog.json. Do not edit.", + "package cool.linc.androiddatacollector.core.collector", + "", + "object ProtocolEventContracts {", + " val contracts: Map = mapOf(", + ] + for collector in collectors: + lines.extend( + [ + f' {_kotlin_string(collector["id"])} to CollectorEventContract(', + f' payloadSchemaVersion = {collector["payload_schema_version"]},', + f' maximumEncodedEventBytes = {collector["maximum_encoded_event_bytes"]:,},'.replace(",", "_", 1), + " payloads = payloads(", + ] + ) + for payload in collector["payloads"]: + types = ", ".join(_kotlin_string(value) for value in payload["types"]) + lines.extend( + [ + f" listOf({types}) to EventPayloadContract(", + " fields = mapOf(", + ] + ) + for name, field in payload["fields"].items(): + arguments = [ + f'type = EventFieldType.{field["type"].upper()}', + f'required = {str(field["required"]).lower()}', + ] + if "enum" in field: + values = ", ".join(_kotlin_string(value) for value in field["enum"]) + arguments.append(f"enumValues = setOf({values})") + if "minimum" in field: + arguments.append(f'minimum = {_kotlin_double(field["minimum"])}') + if "maximum" in field: + arguments.append(f'maximum = {_kotlin_double(field["maximum"])}') + if "maximum_length" in field: + arguments.append(f'maximumLength = {field["maximum_length"]}') + lines.extend( + [ + f' {_kotlin_string(name)} to EventFieldContract(', + *(f" {argument}," for argument in arguments), + " ),", + ] + ) + lines.extend( + [ + " ),", + " ),", + ] + ) + lines.extend( + [ + " ),", + " ),", + ] + ) + lines.extend( + [ + " )", + "", + " operator fun get(collectorId: String): CollectorEventContract? = contracts[collectorId]", + "", + " private fun payloads(", + " vararg groups: Pair, EventPayloadContract>,", + " ): Map = buildMap {", + " groups.forEach { (types, contract) ->", + " types.forEach { type -> check(put(type, contract) == null) { \"Duplicate payload type\" } }", + " }", + " }", + "}", + "", + ] + ) + return "\n".join(lines) + + +def check_kotlin_contract(catalog: dict[str, Any], path: Path = DEFAULT_KOTLIN_CONTRACT) -> None: + expected = render_kotlin_contract(catalog) + try: + actual = path.read_text(encoding="utf-8") + except OSError as error: + raise CatalogError(f"generated Kotlin contract is absent: {path}") from error + if actual != expected: + raise CatalogError( + "generated Kotlin contract is stale; run " + "python3 tools/catalog.py generate-kotlin" + ) + + +def _object(value: Any, path: str, exact: Iterable[str]) -> dict[str, Any]: + if not isinstance(value, dict): + raise CatalogError(f"{path} must be an object") + expected = set(exact) + actual = set(value) + if actual != expected: + raise CatalogError( + f"{path} members differ: missing={sorted(expected-actual)}, " + f"unknown={sorted(actual-expected)}" + ) + return value + + +def _list(value: Any, path: str) -> list[Any]: + if not isinstance(value, list): + raise CatalogError(f"{path} must be an array") + return value + + +def _sorted_unique(values: list[Any], path: str) -> None: + if values != sorted(values) or len(values) != len(set(values)): + raise CatalogError(f"{path} must be sorted and unique") + + +def _string(value: Any, path: str, pattern: re.Pattern[str] | None = None) -> str: + if not isinstance(value, str) or not value: + raise CatalogError(f"{path} must be a non-empty string") + if pattern and not pattern.fullmatch(value): + raise CatalogError(f"{path} has invalid syntax: {value!r}") + return value + + +def _integer(value: Any, path: str, minimum: int | None = None) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise CatalogError(f"{path} must be an integer") + if minimum is not None and value < minimum: + raise CatalogError(f"{path} must be at least {minimum}") + return value + + +def _enum(values: Any, path: str) -> list[str]: + result = _list(values, path) + if not result or any(not isinstance(item, str) or not item for item in result): + raise CatalogError(f"{path} must contain non-empty strings") + _sorted_unique(result, path) + return result + + +def _validate_config_field(field: Any, path: str) -> None: + if not isinstance(field, dict): + raise CatalogError(f"{path} must be an object") + kind = field.get("type") + common = {"type", "meaning", "unit"} + required = { + "integer": common | {"minimum", "maximum"}, + "boolean": common, + "enum": common | {"enum"}, + "enum_array": common | {"items_enum", "minimum_items", "maximum_items"}, + }.get(kind) + if required is None: + raise CatalogError(f"{path} has unknown type or wrong members") + allowed = required | ({"maximum_field"} if kind == "integer" else set()) + if not required <= set(field) or not set(field) <= allowed: + raise CatalogError(f"{path} has unknown type or wrong members") + _string(field["meaning"], f"{path}.meaning") + _string(field["unit"], f"{path}.unit") + if kind == "integer": + low = _integer(field["minimum"], f"{path}.minimum") + high = _integer(field["maximum"], f"{path}.maximum") + if low > high: + raise CatalogError(f"{path} minimum exceeds maximum") + if "maximum_field" in field: + _string(field["maximum_field"], f"{path}.maximum_field", FIELD) + elif kind == "enum": + _enum(field["enum"], f"{path}.enum") + elif kind == "enum_array": + _enum(field["items_enum"], f"{path}.items_enum") + low = _integer(field["minimum_items"], f"{path}.minimum_items", 0) + high = _integer(field["maximum_items"], f"{path}.maximum_items", 0) + if low > high: + raise CatalogError(f"{path} minimum_items exceeds maximum_items") + + +def _validate_payload_field(field: Any, path: str) -> None: + if not isinstance(field, dict): + raise CatalogError(f"{path} must be an object") + kind = field.get("type") + common = {"type", "meaning", "unit", "clock_basis", "required"} + optional = { + "boolean": set(), + "decimal_string": {"minimum", "maximum"}, + "enum": {"enum"}, + "float32": {"minimum", "maximum"}, + "float64": {"minimum", "maximum"}, + "int32": {"minimum", "maximum"}, + "json_string": {"maximum_length"}, + "string": {"maximum_length"}, + }.get(kind) + required = common | ({"enum"} if kind == "enum" else set()) + if optional is None or not required <= set(field) or not set(field) <= common | optional: + raise CatalogError(f"{path} has unknown type or wrong members") + _string(field["meaning"], f"{path}.meaning") + _string(field["unit"], f"{path}.unit") + if field["clock_basis"] is not None: + _string(field["clock_basis"], f"{path}.clock_basis") + if not isinstance(field["required"], bool): + raise CatalogError(f"{path}.required must be boolean") + if kind == "enum": + _enum(field["enum"], f"{path}.enum") + for key in ("minimum", "maximum"): + if key in field: + _integer(field[key], f"{path}.{key}") + if "maximum_length" in field: + _integer(field["maximum_length"], f"{path}.maximum_length", 1) + if "minimum" in field and "maximum" in field and field["minimum"] > field["maximum"]: + raise CatalogError(f"{path} minimum exceeds maximum") + if kind == "int32": + for key in ("minimum", "maximum"): + if key in field and not -(2**31) <= field[key] <= 2**31 - 1: + raise CatalogError(f"{path}.{key} exceeds signed int32") + + +def _validate_module(module: str, status: str, root: Path | None, path: str) -> None: + _string(module, path, MODULE) + if root is None or status != "implemented": + return + parts = module.removeprefix(":").split(":") + directory = root.joinpath(*parts) + if not directory.is_dir(): + raise CatalogError(f"{path} claims missing implemented module {module}") + + +def validate(catalog: dict[str, Any], project_root: Path | None = None) -> None: + root = _object( + catalog, + "$", + {"catalog_format", "catalog_version", "collectors", "protocol_schema_version"}, + ) + if root["catalog_format"] != "adc-collector-catalog-v1": + raise CatalogError("$.catalog_format must be adc-collector-catalog-v1") + if root["catalog_version"] != 1 or root["protocol_schema_version"] != 1: + raise CatalogError("catalog and protocol schema versions must be integer 1") + collectors = _list(root["collectors"], "$.collectors") + ids: list[str] = [] + for index, value in enumerate(collectors): + path = f"$.collectors[{index}]" + collector = _object( + value, + path, + { + "access", + "configuration", + "id", + "implementation", + "maximum_encoded_event_bytes", + "payload_schema_version", + "payloads", + "platforms", + "privacy_class", + "rate_bound", + "selectable", + }, + ) + identifier = _string(collector["id"], f"{path}.id", ID) + ids.append(identifier) + if collector["privacy_class"] not in {"SENSITIVE", "RESTRICTED"}: + raise CatalogError(f"{path}.privacy_class is unknown") + if not isinstance(collector["selectable"], bool): + raise CatalogError(f"{path}.selectable must be boolean") + if collector["payload_schema_version"] != 1: + raise CatalogError(f"{path}.payload_schema_version must be integer 1") + size = _integer( + collector["maximum_encoded_event_bytes"], + f"{path}.maximum_encoded_event_bytes", + 128, + ) + if size > 65_536: + raise CatalogError(f"{path}.maximum_encoded_event_bytes exceeds 65536") + platforms = _enum(collector["platforms"], f"{path}.platforms") + if any(platform not in {"android", "ios"} for platform in platforms): + raise CatalogError(f"{path}.platforms contains an unknown platform") + + implementation = _object( + collector["implementation"], + f"{path}.implementation", + {"android_module", "status"}, + ) + if implementation["status"] not in {"implemented", "planned"}: + raise CatalogError(f"{path}.implementation.status is unknown") + _validate_module( + implementation["android_module"], + implementation["status"], + project_root, + f"{path}.implementation.android_module", + ) + + access = _list(collector["access"], f"{path}.access") + access_keys: list[tuple[str, str]] = [] + for access_index, item in enumerate(access): + access_path = f"{path}.access[{access_index}]" + entry = _object(item, access_path, {"kind", "mode"}) + kind = _string(entry["kind"], f"{access_path}.kind", EVENT_TYPE) + mode = entry["mode"] + if mode not in { + "hardware", + "install_permission", + "participant_setting", + "runtime_permission", + "special_access", + }: + raise CatalogError(f"{access_path}.mode is unknown") + access_keys.append((kind, mode)) + _sorted_unique(access_keys, f"{path}.access") + + configuration = collector["configuration"] + if configuration is None: + if collector["selectable"]: + raise CatalogError(f"{path}.configuration may be null only for runtime collectors") + else: + config = _object(configuration, f"{path}.configuration", {"fields", "required"}) + fields = config["fields"] + if not isinstance(fields, dict): + raise CatalogError(f"{path}.configuration.fields must be an object") + if list(fields) != sorted(fields): + raise CatalogError(f"{path}.configuration.fields must be sorted") + required_value = _list(config["required"], f"{path}.configuration.required") + required = ( + _enum(required_value, f"{path}.configuration.required") + if required_value + else [] + ) + if any(name not in fields for name in required): + raise CatalogError(f"{path}.configuration.required names an unknown field") + for name, field in fields.items(): + _string(name, f"{path}.configuration field name", FIELD) + _validate_config_field(field, f"{path}.configuration.fields.{name}") + for name, field in fields.items(): + maximum_field = field.get("maximum_field") + if maximum_field is None: + continue + referenced = fields.get(maximum_field) + if maximum_field == name or referenced is None or referenced.get("type") != "integer": + raise CatalogError( + f"{path}.configuration.fields.{name}.maximum_field is not another integer field" + ) + + rate = _object( + collector["rate_bound"], + f"{path}.rate_bound", + {"enforced_by", "kind", "maximum_events_per_hour"}, + ) + _string(rate["enforced_by"], f"{path}.rate_bound.enforced_by") + if rate["kind"] not in { + "android_platform_cap", + "collector_rate_limit", + "not_enforced", + "poll_configuration", + }: + raise CatalogError(f"{path}.rate_bound.kind is unknown") + maximum = rate["maximum_events_per_hour"] + if rate["kind"] == "not_enforced": + if maximum is not None: + raise CatalogError(f"{path}.rate_bound maximum must be null when not enforced") + else: + _integer(maximum, f"{path}.rate_bound.maximum_events_per_hour", 1) + + payloads = _list(collector["payloads"], f"{path}.payloads") + if not payloads: + raise CatalogError(f"{path}.payloads must not be empty") + collector_types: set[str] = set() + for payload_index, item in enumerate(payloads): + payload_path = f"{path}.payloads[{payload_index}]" + payload = _object(item, payload_path, {"fields", "types"}) + types = _enum(payload["types"], f"{payload_path}.types") + for event_type in types: + _string(event_type, f"{payload_path}.types", EVENT_TYPE) + if event_type in collector_types: + raise CatalogError(f"payload type is repeated in {identifier}: {event_type}") + collector_types.add(event_type) + fields = payload["fields"] + if not isinstance(fields, dict) or list(fields) != sorted(fields): + raise CatalogError(f"{payload_path}.fields must be a sorted object") + for name, field in fields.items(): + _string(name, f"{payload_path} field name", FIELD) + _validate_payload_field(field, f"{payload_path}.fields.{name}") + _sorted_unique(ids, "$.collectors[].id") + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("command", choices=("check", "canonical", "digest", "generate-kotlin")) + parser.add_argument("path", nargs="?", type=Path, default=DEFAULT_CATALOG) + parser.add_argument( + "--project-root", + type=Path, + default=ROOT, + help="verify implemented Android module paths; use an empty value only through the API", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + try: + catalog = load(args.path) + validate(catalog, args.project_root.resolve()) + except CatalogError as error: + print(f"catalog error: {error}", file=sys.stderr) + return 1 + encoded = canonical_bytes(catalog) + if args.command == "generate-kotlin": + DEFAULT_KOTLIN_CONTRACT.parent.mkdir(parents=True, exist_ok=True) + DEFAULT_KOTLIN_CONTRACT.write_text(render_kotlin_contract(catalog), encoding="utf-8") + print(f"wrote {DEFAULT_KOTLIN_CONTRACT}") + elif args.command == "canonical": + sys.stdout.buffer.write(encoded + b"\n") + elif args.command == "digest": + print(hashlib.sha256(encoded).hexdigest()) + else: + try: + check_kotlin_contract(catalog) + except CatalogError as error: + print(f"catalog error: {error}", file=sys.stderr) + return 1 + print(f"valid collector catalog: {len(catalog['collectors'])} collectors") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/catalog_parity.py b/tools/catalog_parity.py new file mode 100644 index 0000000..caf49f8 --- /dev/null +++ b/tools/catalog_parity.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +"""Prove the catalog's implemented Android/Web projection matches checked-in code.""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path +from typing import Any + +if __package__: + from tools import catalog as catalog_tool +else: + import catalog as catalog_tool + + +ROOT = Path(__file__).resolve().parents[1] +KOTLIN_MODEL = ROOT / "core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt" +KOTLIN_CODEC = ROOT / "core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt" +WEB_TYPES = ROOT / "web/src/lib/adc/types.ts" +WEB_PARSE = ROOT / "web/src/routes/researcher/parse.ts" +RUNTIME = ROOT / "core/experiment-runtime/src/main/kotlin/cool/linc/androiddatacollector/core/runtime/ExperimentRuntime.kt" +KOTLIN_EVENT_CONTRACT = ( + ROOT + / "core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt" +) + + +class ParityError(ValueError): + """A platform copy has drifted from the language-neutral catalog.""" + + +def _balanced(text: str, start: int, opening: str, closing: str) -> str: + if start < 0 or text[start] != opening: + raise ParityError(f"cannot locate balanced {opening}{closing} block") + depth = 0 + quote: str | None = None + escaped = False + for index in range(start, len(text)): + character = text[index] + if quote: + if escaped: + escaped = False + elif character == "\\": + escaped = True + elif character == quote: + quote = None + continue + if character in {'"', "'"}: + quote = character + elif character == opening: + depth += 1 + elif character == closing: + depth -= 1 + if depth == 0: + return text[start : index + 1] + raise ParityError(f"unterminated {opening}{closing} block") + + +def _number(text: str) -> int: + return int(text.replace("_", "")) + + +def _snake(name: str) -> str: + return re.sub(r"(? list[dict[str, Any]]: + return [ + collector + for collector in catalog["collectors"] + if collector["implementation"]["status"] == "implemented" + and collector["selectable"] + and "android" in collector["platforms"] + ] + + +def _configuration_classes(model: str) -> dict[str, dict[str, Any]]: + starts = list(re.finditer(r"^data class (\w+Configuration)\s*\(", model, re.MULTILINE)) + result: dict[str, dict[str, Any]] = {} + for index, match in enumerate(starts): + end = starts[index + 1].start() if index + 1 < len(starts) else len(model) + section = model[match.start() : end] + identifier = re.search(r'const val ID = "([a-z][a-z0-9_.-]+\.v1)"', section) + if not identifier: + continue + parameters = _balanced(model, match.end() - 1, "(", ")") + fields = { + _snake(name): kind.strip() + for name, kind in re.findall( + r"(?:override\s+)?val\s+(\w+)\s*:\s*([^,\n=]+)", parameters + ) + if name != "required" + } + result[identifier.group(1)] = { + "class": match.group(1), + "fields": fields, + "section": section, + } + return result + + +def _codec_fields(codec: str, class_name: str) -> set[str]: + marker = re.search(rf"{re.escape(class_name)}\.ID\s*->\s*\{{", codec) + if not marker: + raise ParityError(f"Kotlin codec has no {class_name} closed-world branch") + block = _balanced(codec, marker.end() - 1, "{", "}") + marker = block.find("config.requireExactKeys(") + if marker < 0: + raise ParityError(f"Kotlin codec {class_name} branch has no exact config key set") + arguments = _balanced(block, marker + len("config.requireExactKeys"), "(", ")") + if "emptySet()" not in arguments and "setOf(" not in arguments: + raise ParityError(f"Kotlin codec {class_name} key set is not statically auditable") + return set(re.findall(r'"([a-z][a-z0-9_]*)"', arguments)) + + +def _enum_values(model: str, name: str) -> set[str]: + match = re.search(rf"enum class {re.escape(name)}\s*\{{([^}}]+)\}}", model) + if not match: + raise ParityError(f"Kotlin enum {name} is absent") + return set(re.findall(r"\b[A-Z][A-Z0-9_]*\b", match.group(1))) + + +def _check_kotlin_configuration( + collectors: list[dict[str, Any]], model: str, codec: str +) -> None: + classes = _configuration_classes(model) + expected_ids = {collector["id"] for collector in collectors} + if set(classes) != expected_ids: + raise ParityError( + f"Kotlin CollectorConfiguration IDs differ: catalog={sorted(expected_ids)}, " + f"Kotlin={sorted(classes)}" + ) + codec_classes = set(re.findall(r"(\w+Configuration)\.ID\s*->", codec)) + expected_classes = {value["class"] for value in classes.values()} + if codec_classes != expected_classes: + raise ParityError("Kotlin codec collector branches differ from CollectorConfiguration classes") + by_id = {collector["id"]: collector for collector in collectors} + for identifier, implementation in classes.items(): + configuration = by_id[identifier]["configuration"] + fields = configuration["fields"] + if set(implementation["fields"]) != set(fields): + raise ParityError(f"{identifier} Kotlin constructor fields differ from catalog") + if set(configuration["required"]) != set(fields): + raise ParityError(f"{identifier} catalog must mark every Kotlin constructor field required") + if _codec_fields(codec, implementation["class"]) != set(fields): + raise ParityError(f"{identifier} Kotlin codec key set differs from catalog") + section = implementation["section"] + for name, field in fields.items(): + kotlin_type = implementation["fields"][name] + property_name = re.sub(r"_([a-z])", lambda match: match.group(1).upper(), name) + if field["type"] == "integer": + if kotlin_type not in {"Int", "Long"}: + raise ParityError(f"{identifier}.{name} is not an integer in Kotlin") + bound = re.search( + rf"require\(\s*{re.escape(property_name)}\s+in\s+([0-9_]+)\.\.([A-Za-z0-9_]+)", + section, + ) + if not bound or _number(bound.group(1)) != field["minimum"]: + raise ParityError(f"{identifier}.{name} Kotlin minimum differs from catalog") + maximum = bound.group(2) + if maximum[0].isdigit(): + resolved_maximum = _number(maximum) + if "maximum_field" in field: + raise ParityError(f"{identifier}.{name} lacks its catalog cross-field bound") + else: + referenced = _snake(maximum) + if referenced not in fields: + raise ParityError(f"{identifier}.{name} has an unmodeled Kotlin cross-field bound") + if field.get("maximum_field") != referenced: + raise ParityError(f"{identifier}.{name} Kotlin cross-field bound differs from catalog") + resolved_maximum = fields[referenced]["maximum"] + if resolved_maximum != field["maximum"]: + raise ParityError(f"{identifier}.{name} Kotlin maximum differs from catalog") + elif field["type"] == "boolean": + if kotlin_type != "Boolean": + raise ParityError(f"{identifier}.{name} is not Boolean in Kotlin") + elif field["type"] == "enum": + if _enum_values(model, kotlin_type) != set(field["enum"]): + raise ParityError(f"{identifier}.{name} Kotlin enum differs from catalog") + elif field["type"] == "enum_array": + element = re.fullmatch(r"Set<(\w+)>", kotlin_type) + if not element or {item.lower() for item in _enum_values(model, element.group(1))} != set(field["items_enum"]): + raise ParityError(f"{identifier}.{name} Kotlin enum set differs from catalog") + + +def _descriptor_block(source: str) -> str: + marker = source.find("CollectorDescriptor(") + if marker < 0 or source.find("CollectorDescriptor(", marker + 1) >= 0: + raise ParityError("collector module must declare exactly one CollectorDescriptor") + return _balanced(source, marker + len("CollectorDescriptor"), "(", ")") + + +def _check_kotlin_descriptors( + collectors: list[dict[str, Any]], root: Path, model: str +) -> None: + classes = _configuration_classes(model) + for collector in collectors: + module = root.joinpath(*collector["implementation"]["android_module"].lstrip(":").split(":")) + source = "\n".join( + path.read_text(encoding="utf-8") for path in sorted((module / "src/main").rglob("*.kt")) + ) + identifiers = set(re.findall(r'"([a-z][a-z0-9_.-]+\.v1)"', source)) + class_name = classes[collector["id"]]["class"] + if identifiers - {collector["id"]} or ( + collector["id"] not in identifiers and f"{class_name}.ID" not in source + ): + raise ParityError(f"{module} wire IDs differ from catalog: {sorted(identifiers)}") + block = _descriptor_block(source) + event_contract = re.search( + r"eventContract\s*=\s*requireNotNull\(ProtocolEventContracts\[([^\]]+)]\)", + block, + ) + privacy = re.search(r"privacyClass\s*=\s*PrivacyClass\.(\w+)", block) + if not event_contract or not privacy: + raise ParityError(f"{module} descriptor is not statically auditable") + contract_id = event_contract.group(1).strip() + if contract_id == f"{class_name}.ID": + pass + elif re.fullmatch(r"[A-Z][A-Z0-9_]*", contract_id): + definition = re.search( + rf"const val {re.escape(contract_id)}\s*=\s*\"([^\"]+)\"", + source, + ) + if not definition or definition.group(1) != collector["id"]: + raise ParityError(f"{collector['id']} descriptor contract ID differs") + elif contract_id != f'"{collector["id"]}"': + raise ParityError(f"{collector['id']} descriptor contract ID differs") + if "payloadSchemaVersion" in block or "maximumEncodedEventBytes" in block: + raise ParityError(f"{collector['id']} descriptor duplicates generated contract metadata") + if privacy.group(1) != collector["privacy_class"]: + raise ParityError(f"{collector['id']} privacy class differs") + expected_access = { + item["kind"] for item in collector["access"] if item["mode"] != "install_permission" + } + actual_access = set(re.findall(r"AccessKind\.([A-Z][A-Z0-9_]*)", source)) + if actual_access != expected_access: + raise ParityError(f"{collector['id']} runtime access requirements differ") + manifest = (module / "src/main/AndroidManifest.xml") + manifest_text = manifest.read_text(encoding="utf-8") if manifest.is_file() else "" + for access in collector["access"]: + if access["mode"] == "install_permission" and f"android.permission.{access['kind']}" not in manifest_text: + raise ParityError(f"{collector['id']} install permission {access['kind']} is absent") + + +def _quoted(block: str, suffix: str = "") -> set[str]: + return set(re.findall(rf"'([a-z][a-z0-9_.-]+{re.escape(suffix)})'", block)) + + +def _web_case_fields(parse_source: str, identifier: str) -> set[str]: + switch_start = parse_source.index("switch (source.id)") + switch = _balanced(parse_source, parse_source.index("{", switch_start), "{", "}") + marker = re.search(rf"case '{re.escape(identifier)}':(?:\s*\{{)?", switch) + if not marker: + raise ParityError(f"Web structural parser has no {identifier} branch") + tail = switch[marker.end() :] + next_case = re.search(r"\n\s*case '[^']+':", tail) + block = tail[: next_case.start()] if next_case else tail + exact = re.search(r"requireExactKeys\(config,\s*\[(.*?)\]\)", block, re.DOTALL) + if not exact: + raise ParityError(f"Web structural parser {identifier} branch lacks exact keys") + return set(re.findall(r"'([a-z][a-z0-9_]*)'", exact.group(1))) + + +def _check_web(collectors: list[dict[str, Any]], types_source: str, parse_source: str) -> None: + expected_ids = {collector["id"] for collector in collectors} + union = types_source[types_source.index("export type CollectorId") : types_source.index("export const COLLECTOR_ORDER")] + order_match = re.search(r"export const COLLECTOR_ORDER[^=]*=\s*\[(.*?)\]", types_source, re.DOTALL) + if not order_match: + raise ParityError("Web COLLECTOR_ORDER is absent") + if _quoted(union, ".v1") != expected_ids or _quoted(order_match.group(1), ".v1") != expected_ids: + raise ParityError("Web CollectorId/COLLECTOR_ORDER differs from implemented catalog IDs") + bounds_block = re.search(r"export const BOUNDS = \{(.*?)\}\s+as const", types_source, re.DOTALL) + if not bounds_block: + raise ParityError("Web BOUNDS metadata is absent") + bounds = { + name: (_number(low), _number(high)) + for name, low, high in re.findall( + r"^\s*(\w+):\s*\[([0-9_]+),\s*([0-9_]+)\]", bounds_block.group(1), re.MULTILINE + ) + } + for collector in collectors: + fields = collector["configuration"]["fields"] + if _web_case_fields(parse_source, collector["id"]) != set(fields): + raise ParityError(f"{collector['id']} Web structural key set differs from catalog") + for name, field in fields.items(): + if field["type"] != "integer": + continue + camel = re.sub(r"_([a-z])", lambda match: match.group(1).upper(), name) + expected = (field["minimum"], field["maximum"]) + collector_prefix = re.sub( + r"_([a-z])", + lambda match: match.group(1).upper(), + collector["id"].split(".", 1)[0], + ) + specific = collector_prefix + camel[0].upper() + camel[1:] + if bounds.get(camel) != expected and bounds.get(specific) != expected: + raise ParityError(f"{collector['id']}.{name} Web bounds differ from catalog") + for name, field in fields.items(): + if field["type"] == "enum": + values = re.search( + r"export const LOCATION_PRIORITIES[^=]*=\s*\[(.*?)\]", + types_source, + re.DOTALL, + ) + if not values or set(field["enum"]) != set(re.findall(r"'([A-Z][A-Z_]*)'", values.group(1))): + raise ParityError(f"{collector['id']}.{name} Web enum differs from catalog") + if field["type"] == "enum_array": + values = re.search( + r"export const NETWORK_TRANSPORTS[^=]*=\s*\[(.*?)\]", + types_source, + re.DOTALL, + ) + if not values or set(field["items_enum"]) != set(re.findall(r"'([a-z]+)'", values.group(1))): + raise ParityError(f"{collector['id']}.{name} Web enum array differs from catalog") + + +def _check_interventions(catalog: dict[str, Any], root: Path) -> None: + source = RUNTIME.read_text(encoding="utf-8") + if "descriptor.eventContract.accepts(event, Long.MAX_VALUE)" not in source: + raise ParityError("collector runtime does not enforce the generated typed event contract") + intervention_contract = re.search( + r"ProtocolEventContracts\[([A-Z][A-Z0-9_]*)]\)\.accepts\(\s*" + r"draft,\s*metadataAfterState\.nextSequenceNumber,?\s*\)", + source, + ) + if not intervention_contract: + raise ParityError("intervention runtime does not consume its generated event contract") + identifier = intervention_contract.group(1) + definition = re.search( + rf'const val {re.escape(identifier)}\s*=\s*"([^"]+)"', + source, + ) + if not definition or definition.group(1) != "interventions.v1": + raise ParityError("intervention runtime uses the wrong generated event contract") + contract = (root / "core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt").read_text(encoding="utf-8") + if "maximumEncodedEventBytes in 128..65_536" not in contract: + raise ParityError("CollectorEventContract maximumEncodedEventBytes bounds drifted") + + +def check(value: dict[str, Any], root: Path = ROOT) -> None: + catalog_tool.validate(value, root) + expected_contract = catalog_tool.render_kotlin_contract(value) + actual_contract = KOTLIN_EVENT_CONTRACT.read_text(encoding="utf-8") + if actual_contract != expected_contract: + raise ParityError("generated Kotlin event contract differs from the catalog") + collectors = _implemented(value) + model = KOTLIN_MODEL.read_text(encoding="utf-8") + codec = KOTLIN_CODEC.read_text(encoding="utf-8") + _check_kotlin_configuration(collectors, model, codec) + _check_kotlin_descriptors(collectors, root, model) + _check_web( + collectors, + WEB_TYPES.read_text(encoding="utf-8"), + WEB_PARSE.read_text(encoding="utf-8"), + ) + _check_interventions(value, root) + + +def main() -> int: + try: + value = catalog_tool.load(catalog_tool.DEFAULT_CATALOG) + check(value) + except (OSError, UnicodeError, ParityError, catalog_tool.CatalogError) as error: + print(f"catalog parity error: {error}", file=sys.stderr) + return 1 + print("catalog parity passed: Kotlin descriptors/config codec and Web metadata") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/collector_assurance.py b/tools/collector_assurance.py new file mode 100644 index 0000000..7f9bb51 --- /dev/null +++ b/tools/collector_assurance.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +"""Fail closed when an Android collector crosses its constrained capability boundary.""" + +from __future__ import annotations + +import argparse +import json +import re +import struct +import sys +import zipfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable + + +ROOT = Path(__file__).resolve().parents[1] +DEFAULT_POLICY = ROOT / "assurance/collector-policy.json" +IMPORT = re.compile(r"^\s*import\s+([A-Za-z_][\w.]*)", re.MULTILINE) +PROJECT_DEPENDENCY = re.compile(r"\b(?:api|implementation|compileOnly|runtimeOnly)\s*\(\s*project\(\s*\"([^\"]+)\"") +CATALOG_DEPENDENCY = re.compile(r"\b(?:api|implementation|compileOnly|runtimeOnly)\s*\(\s*(libs\.[A-Za-z0-9_.]+)\s*\)") +TEST_CATALOG_DEPENDENCY = re.compile( + r"\btestImplementation\s*\(\s*(libs\.[A-Za-z0-9_.]+)\s*\)" +) +STRING_DEPENDENCY = re.compile( + r"\b(?:api|implementation|compileOnly|runtimeOnly)\s*\(\s*\"([^\"]+)\"\s*\)" +) +DEPENDENCY_DECLARATION = re.compile( + r"\b(?:[A-Za-z][A-Za-z0-9]*(?:Implementation|Api|CompileOnly|RuntimeOnly)|" + r"api|implementation|compileOnly|runtimeOnly)\s*\(" +) +DESCRIPTOR_CLASS = re.compile(r"L([A-Za-z0-9_$/]+);") +ALLOWED_TEST_CATALOG_DEPENDENCIES = frozenset({"libs.junit4"}) + + +class AssuranceError(ValueError): + """An assurance input is malformed or violates policy.""" + + +@dataclass(frozen=True, order=True) +class MemberReference: + owner: str + name: str + + +@dataclass(frozen=True) +class ClassReferences: + classes: frozenset[str] + methods: frozenset[MemberReference] + + +def load_policy(path: Path) -> dict[str, Any]: + try: + policy = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise AssuranceError(f"cannot load policy: {error}") from error + expected = { + "allowed_catalog_dependencies", + "allowed_project_dependencies", + "forbidden_class_prefixes", + "forbidden_classes", + "forbidden_import_prefixes", + "forbidden_method_names", + "forbidden_methods", + } + if not isinstance(policy, dict) or set(policy) != expected: + raise AssuranceError("collector policy has unknown or missing members") + return policy + + +def parse_class(data: bytes) -> ClassReferences: + """Read only the JVM constant pool; executable bytecode is intentionally unnecessary.""" + if len(data) < 10 or data[:4] != b"\xca\xfe\xba\xbe": + raise AssuranceError("not a JVM class file") + count = struct.unpack_from(">H", data, 8)[0] + pool: list[tuple[int, Any] | None] = [None] * count + offset = 10 + index = 1 + while index < count: + if offset >= len(data): + raise AssuranceError("truncated constant pool") + tag = data[offset] + offset += 1 + if tag == 1: + if offset + 2 > len(data): + raise AssuranceError("truncated UTF-8 constant length") + length = struct.unpack_from(">H", data, offset)[0] + offset += 2 + if offset + length > len(data): + raise AssuranceError("truncated UTF-8 constant") + # JVM class files use modified UTF-8. Names relevant to this policy are ASCII; + # replacement decoding safely ignores modified encodings in unrelated literals. + value = data[offset : offset + length].decode("utf-8", errors="replace") + offset += length + elif tag in {3, 4}: + value = data[offset : offset + 4] + offset += 4 + elif tag in {5, 6}: + value = data[offset : offset + 8] + offset += 8 + pool[index] = (tag, value) + index += 2 + continue + elif tag in {7, 8, 16, 19, 20}: + if offset + 2 > len(data): + raise AssuranceError("truncated constant-pool index") + value = struct.unpack_from(">H", data, offset)[0] + offset += 2 + elif tag in {9, 10, 11, 12, 17, 18}: + if offset + 4 > len(data): + raise AssuranceError("truncated constant-pool pair") + value = struct.unpack_from(">HH", data, offset) + offset += 4 + elif tag == 15: + if offset + 3 > len(data): + raise AssuranceError("truncated method handle") + value = (data[offset], struct.unpack_from(">H", data, offset + 1)[0]) + offset += 3 + else: + raise AssuranceError(f"unknown constant-pool tag {tag}") + if offset > len(data): + raise AssuranceError("truncated constant pool") + pool[index] = (tag, value) + index += 1 + + def entry(at: int, tag: int) -> Any: + if at <= 0 or at >= len(pool) or pool[at] is None or pool[at][0] != tag: + raise AssuranceError("invalid constant-pool reference") + return pool[at][1] + + def utf8(at: int) -> str: + return entry(at, 1) + + def class_name(at: int) -> str: + return utf8(entry(at, 7)) + + classes: set[str] = set() + methods: set[MemberReference] = set() + for constant in pool[1:]: + if constant is None: + continue + tag, value = constant + if tag == 7: + classes.add(utf8(value)) + elif tag == 1: + classes.update(DESCRIPTOR_CLASS.findall(value)) + elif tag in {10, 11}: + class_index, name_and_type_index = value + name_index, _descriptor_index = entry(name_and_type_index, 12) + methods.add(MemberReference(class_name(class_index), utf8(name_index))) + return ClassReferences(frozenset(classes), frozenset(methods)) + + +def violations(references: ClassReferences, policy: dict[str, Any]) -> list[str]: + class_prefixes = tuple(policy["forbidden_class_prefixes"]) + exact_classes = set(policy["forbidden_classes"]) + forbidden_methods = { + MemberReference(item["owner"], item["name"]) for item in policy["forbidden_methods"] + } + forbidden_method_names = set(policy["forbidden_method_names"]) + result: list[str] = [] + for name in sorted(references.classes): + normalized = name.lstrip("[").removeprefix("L").removesuffix(";") + if normalized in exact_classes or normalized.startswith(class_prefixes): + result.append(f"forbidden class reference {normalized}") + for member in sorted(references.methods & forbidden_methods): + result.append(f"forbidden method reference {member.owner}.{member.name}") + for member in sorted( + reference + for reference in references.methods - forbidden_methods + if reference.name in forbidden_method_names + ): + result.append(f"forbidden method reference {member.owner}.{member.name}") + return result + + +def _production_class_files(module: Path) -> Iterable[Path]: + build = module / "build" + if not build.is_dir(): + return [] + return ( + path + for path in build.rglob("*.class") + if not any("test" in part.lower() for part in path.relative_to(build).parts) + and ( + "kotlin-classes" in path.parts + or "built_in_kotlinc" in path.parts + or "javac" in path.parts + ) + ) + + +def _source_violations(module: Path, policy: dict[str, Any]) -> list[str]: + forbidden = tuple(policy["forbidden_import_prefixes"]) + result: list[str] = [] + for source in sorted((module / "src/main").rglob("*.kt")): + text = source.read_text(encoding="utf-8") + for imported in IMPORT.findall(text): + if imported.startswith(forbidden): + result.append(f"{source}: forbidden import {imported}") + build_file = module / "build.gradle.kts" + if build_file.is_file(): + text = build_file.read_text(encoding="utf-8") + allowed_projects = set(policy["allowed_project_dependencies"]) + allowed_catalogs = set(policy["allowed_catalog_dependencies"]) + recognized_starts: set[int] = set() + for match in PROJECT_DEPENDENCY.finditer(text): + recognized_starts.add(match.start()) + dependency = match.group(1) + if dependency not in allowed_projects: + result.append(f"{build_file}: forbidden project dependency {dependency}") + for match in CATALOG_DEPENDENCY.finditer(text): + recognized_starts.add(match.start()) + dependency = match.group(1) + if dependency not in allowed_catalogs: + result.append(f"{build_file}: forbidden catalog dependency {dependency}") + for match in TEST_CATALOG_DEPENDENCY.finditer(text): + recognized_starts.add(match.start()) + dependency = match.group(1) + if dependency not in ALLOWED_TEST_CATALOG_DEPENDENCIES: + result.append(f"{build_file}: forbidden test dependency {dependency}") + for match in STRING_DEPENDENCY.finditer(text): + recognized_starts.add(match.start()) + dependency = match.group(1) + result.append(f"{build_file}: forbidden external dependency {dependency}") + for declaration in DEPENDENCY_DECLARATION.finditer(text): + if declaration.start() not in recognized_starts: + result.append( + f"{build_file}: unrecognized dependency declaration " + f"{declaration.group(0).removesuffix('(').strip()}" + ) + return result + + +def scan(root: Path, policy: dict[str, Any], require_classes: bool = True) -> tuple[int, list[str]]: + collector_root = root / "collector" + modules = sorted(path.parent for path in collector_root.glob("*/build.gradle.kts")) + if not modules: + raise AssuranceError(f"no collector modules found below {collector_root}") + errors: list[str] = [] + class_count = 0 + for module in modules: + errors.extend(_source_violations(module, policy)) + for class_file in sorted(_production_class_files(module)): + class_count += 1 + try: + found = violations(parse_class(class_file.read_bytes()), policy) + except (OSError, AssuranceError) as error: + errors.append(f"{class_file}: {error}") + continue + errors.extend(f"{class_file}: {item}" for item in found) + if require_classes and class_count == 0: + errors.append("no production collector class files found; compile collectors before assurance") + return class_count, errors + + +def scan_explicit(paths: Iterable[Path], policy: dict[str, Any]) -> tuple[int, list[str]]: + count = 0 + errors: list[str] = [] + for path in paths: + if path.suffix == ".jar": + with zipfile.ZipFile(path) as archive: + for name in sorted(item for item in archive.namelist() if item.endswith(".class")): + count += 1 + errors.extend( + f"{path}!{name}: {item}" + for item in violations(parse_class(archive.read(name)), policy) + ) + else: + count += 1 + errors.extend(f"{path}: {item}" for item in violations(parse_class(path.read_bytes()), policy)) + return count, errors + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("paths", nargs="*", type=Path, help="optional class or jar files") + parser.add_argument("--root", type=Path, default=ROOT) + parser.add_argument("--policy", type=Path, default=DEFAULT_POLICY) + parser.add_argument("--allow-no-classes", action="store_true") + args = parser.parse_args(argv) + try: + policy = load_policy(args.policy) + if args.paths: + count, errors = scan_explicit(args.paths, policy) + else: + count, errors = scan(args.root, policy, not args.allow_no_classes) + except (AssuranceError, OSError, zipfile.BadZipFile) as error: + print(f"collector assurance error: {error}", file=sys.stderr) + return 1 + if errors: + print("collector assurance violations:", file=sys.stderr) + for error in errors: + print(f"- {error}", file=sys.stderr) + return 1 + print(f"collector assurance passed: {count} production class files") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/conformance/kotlin/ProtocolConformanceTest.kt b/tools/conformance/kotlin/ProtocolConformanceTest.kt new file mode 100644 index 0000000..3ac929e --- /dev/null +++ b/tools/conformance/kotlin/ProtocolConformanceTest.kt @@ -0,0 +1,143 @@ +package adc.conformance + +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import cool.linc.androiddatacollector.core.crypto.HpkeCrypto +import cool.linc.androiddatacollector.core.definition.ProtocolCanonicalJson +import cool.linc.androiddatacollector.core.definition.StudyConfigurationCodec +import cool.linc.androiddatacollector.core.export.ResearchBundleVerifier +import cool.linc.androiddatacollector.core.export.ResearchExport +import cool.linc.androiddatacollector.core.export.UploadReceiptCodec +import cool.linc.androiddatacollector.core.protocol.ConfigurationVerifier +import cool.linc.androiddatacollector.core.protocol.JoinLink +import cool.linc.androiddatacollector.core.protocol.SignedConfigurationCodec +import java.io.ByteArrayOutputStream +import java.io.File +import java.nio.ByteBuffer +import java.time.Instant +import java.util.Base64 +import org.junit.Assert.assertArrayEquals +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Test + +class ProtocolConformanceTest { + private val corpus: JsonObject by lazy { + val root = requireNotNull(System.getProperty("adc.repository.root")) + JsonParser.parseString(File(root, "protocol/v1/conformance-vectors.json").readText()).asJsonObject + } + private val joinCorpus: JsonObject by lazy { + val root = requireNotNull(System.getProperty("adc.repository.root")) + JsonParser.parseString(File(root, "protocol/v1/join-link-vectors.json").readText()).asJsonObject + } + + @Test + fun validCorpusIsConsumedByKotlinProtocolImplementations() { + val valid = corpus.objectAt("valid") + ProtocolCanonicalJson.requireCanonical( + valid.objectAt("canonical_json").hexAt("canonical_jcs_utf8_hex"), + MAXIMUM_FIXTURE_BYTES, + ) + val signed = valid.objectAt("signed_configuration") + val envelope = SignedConfigurationCodec.decode(signed.hexAt("envelope_hex")) + assertArrayEquals(signed.hexAt("canonical_jcs_utf8_hex"), envelope.configurationBytes) + assertEquals(signed.stringAt("signer_key_id"), envelope.signerKeyId) + val verified = ConfigurationVerifier( + trustedSigningKeys = emptyMap(), + clientVersion = 7, + now = { Instant.parse("2027-01-01T00:00:00Z") }, + ).verify(signed.hexAt("envelope_hex")) + + val bundle = valid.objectAt("bundle") + val plaintext = ByteArrayOutputStream() + val header = ResearchExport.decrypt( + bundle.hexAt("container_hex").inputStream(), + plaintext, + bundle.base64UrlAt("researcher_private_key_base64url"), + verified.configuration, + ) + ResearchBundleVerifier.verify(plaintext.toByteArray(), header, verified.configuration) + assertArrayEquals(bundle.hexAt("document_jcs_utf8_hex"), plaintext.toByteArray()) + + val receiptVector = valid.objectAt("upload_receipt") + val receipt = UploadReceiptCodec.decode(receiptVector.hexAt("canonical_jcs_utf8_hex")) + val expected = receiptVector.objectAt("value") + assertEquals(expected.stringAt("bundle_id"), receipt.bundleId.toString()) + assertEquals(expected.stringAt("byte_count"), receipt.byteCount.toString()) + assertEquals(expected.stringAt("sha256"), receipt.sha256) + } + + @Test + fun everyHostileVectorFailsItsKotlinEntrypoint() { + val valid = corpus.objectAt("valid") + val signed = valid.objectAt("signed_configuration") + val configuration = StudyConfigurationCodec.decode(signed.hexAt("canonical_jcs_utf8_hex")) + val bundle = valid.objectAt("bundle") + val privateKey = bundle.base64UrlAt("researcher_private_key_base64url") + val container = bundle.hexAt("container_hex") + corpus.getAsJsonArray("hostile").forEach { element -> + val vector = element.asJsonObject + val input = vector.hexAt("input_hex") + when (vector.stringAt("entrypoint")) { + "canonical_json" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + ProtocolCanonicalJson.requireCanonical(input, MAXIMUM_FIXTURE_BYTES) + } + "configuration_jcs" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + StudyConfigurationCodec.decode(input) + } + "signed_configuration" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + ConfigurationVerifier( + trustedSigningKeys = emptyMap(), + clientVersion = 7, + now = { Instant.parse("2027-01-01T00:00:00Z") }, + ).verify(input) + } + "bundle" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + val plaintext = ByteArrayOutputStream() + val header = ResearchExport.decrypt(input.inputStream(), plaintext, privateKey, configuration) + ResearchBundleVerifier.verify(plaintext.toByteArray(), header, configuration) + } + "bundle_unwrap_context" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + val keyIdLength = ByteBuffer.wrap(container, 56, 2).short.toInt() and 0xffff + val wrapped = container.copyOfRange(70 + keyIdLength, 150 + keyIdLength) + HpkeCrypto.decrypt(privateKey, wrapped, input) + } + "receipt" -> assertThrows(vector.stringAt("id"), Exception::class.java) { + UploadReceiptCodec.decode(input) + } + else -> error("Unknown conformance entrypoint") + } + } + } + + @Test + fun sharedJoinLinkCorpusIsAcceptedAndRejectedByTheKotlinWireParser() { + val valid = joinCorpus.objectAt("valid") + val parsed = JoinLink.parse(valid.stringAt("encoded")) + assertEquals(valid.stringAt("artifact_url"), parsed.artifactUrl.toASCIIString()) + assertEquals(valid.stringAt("artifact_sha256"), parsed.artifactSha256) + assertEquals(valid.stringAt("signer_fingerprint"), parsed.signerFingerprint) + assertEquals(valid.stringAt("encoded"), parsed.encode()) + + joinCorpus.getAsJsonArray("hostile").forEach { element -> + val vector = element.asJsonObject + assertThrows(vector.stringAt("id"), IllegalArgumentException::class.java) { + JoinLink.parse(vector.stringAt("encoded")) + } + } + } + + private fun JsonObject.objectAt(name: String): JsonObject = getAsJsonObject(name) + + private fun JsonObject.stringAt(name: String): String = get(name).asString + + private fun JsonObject.hexAt(name: String): ByteArray { + val text = stringAt(name) + require(text.length % 2 == 0 && text.matches(Regex("[0-9a-f]*"))) { "Invalid vector hex" } + return ByteArray(text.length / 2) { index -> text.substring(index * 2, index * 2 + 2).toInt(16).toByte() } + } + + private fun JsonObject.base64UrlAt(name: String): ByteArray = Base64.getUrlDecoder().decode(stringAt(name)) + + private companion object { const val MAXIMUM_FIXTURE_BYTES = 1_048_576 } +} diff --git a/tools/conformance/typescript/catalog-parity.spec.ts b/tools/conformance/typescript/catalog-parity.spec.ts new file mode 100644 index 0000000..45218ff --- /dev/null +++ b/tools/conformance/typescript/catalog-parity.spec.ts @@ -0,0 +1,167 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { canonicalBytes, canonicalConfigurationBytes, configurationValue } from '../../../web/src/lib/adc/canonical'; +import { defaultCollector, validate } from '../../../web/src/lib/adc/schema'; +import { COLLECTOR_ORDER, type CollectorConfig, type CollectorId } from '../../../web/src/lib/adc/types'; +import { parseConfiguration } from '../../../web/src/routes/researcher/parse'; +import { validConfiguration } from '../../../web/tests/fixture'; + +type ConfigurationField = + | { type: 'integer'; minimum: number; maximum: number; maximum_field?: string } + | { type: 'boolean' } + | { type: 'enum'; enum: string[] } + | { type: 'enum_array'; items_enum: string[]; minimum_items: number; maximum_items: number }; + +type CatalogCollector = { + id: string; + platforms: string[]; + selectable: boolean; + implementation: { status: string }; + configuration: { fields: Record; required: string[] } | null; +}; + +const catalog = JSON.parse( + readFileSync(new URL('../../../protocol/v1/collector-catalog.json', import.meta.url), 'utf8') +) as { collectors: CatalogCollector[] }; + +const implemented = catalog.collectors.filter((collector) => + collector.implementation.status === 'implemented' && + collector.selectable && + collector.platforms.includes('android') +); + +const configRecord = (collector: CollectorConfig): Record => + collector.config as Record; + +function withCollector(collector: CollectorConfig) { + return validConfiguration({ collectors: [collector] }); +} + +function atIntegerBoundary( + collector: CollectorConfig, + field: string, + value: number +): CollectorConfig { + const copy = structuredClone(collector); + configRecord(copy)[field] = value; + if (copy.id === 'location.v1') { + if (field === 'interval_millis' && value < copy.config.minimum_interval_millis) { + copy.config.minimum_interval_millis = value; + } + if (field === 'minimum_interval_millis' && value > copy.config.interval_millis) { + copy.config.interval_millis = value; + } + } + return copy; +} + +function collectorIssues(collector: CollectorConfig) { + return validate(withCollector(collector)).filter((issue) => issue.path.startsWith('collectors.0')); +} + +function rawConfiguration(collector: CollectorConfig): Record { + return configurationValue(withCollector(collector)) as Record; +} + +describe('collector catalog Web projection', () => { + it('selects exactly the catalog\'s implemented Android collectors', () => { + expect(new Set(COLLECTOR_ORDER)).toEqual(new Set(implemented.map((collector) => collector.id))); + expect(COLLECTOR_ORDER).toHaveLength(new Set(COLLECTOR_ORDER).size); + }); + + for (const definition of implemented) { + const id = definition.id as CollectorId; + const schema = definition.configuration!; + + it(`${id} default and closed-world shape match the catalog`, () => { + const collector = defaultCollector(id); + expect(collector.id).toBe(id); + expect(new Set(Object.keys(configRecord(collector)))).toEqual(new Set(Object.keys(schema.fields))); + expect(new Set(schema.required)).toEqual(new Set(Object.keys(schema.fields))); + expect(collectorIssues(collector)).toEqual([]); + expect(parseConfiguration(canonicalConfigurationBytes(withCollector(collector))).collectors) + .toEqual([collector]); + + const unknown = rawConfiguration(collector); + const unknownConfig = ((unknown.collectors as Array>)[0] + .config as Record); + unknownConfig.unknown_catalog_field = true; + expect(() => parseConfiguration(canonicalBytes(unknown))).toThrow('parse_keys'); + + const firstField = Object.keys(schema.fields)[0]; + if (firstField) { + const missing = rawConfiguration(collector); + const missingConfig = ((missing.collectors as Array>)[0] + .config as Record); + delete missingConfig[firstField]; + expect(() => parseConfiguration(canonicalBytes(missing))).toThrow('parse_keys'); + } + }); + + for (const [fieldName, field] of Object.entries(schema.fields)) { + if (field.type === 'integer') { + it(`${id}.${fieldName} enforces the catalog's inclusive integer bounds`, () => { + for (const boundary of [field.minimum, field.maximum]) { + const collector = atIntegerBoundary(defaultCollector(id), fieldName, boundary); + expect(collectorIssues(collector)).toEqual([]); + expect(() => parseConfiguration(canonicalConfigurationBytes(withCollector(collector)))) + .not.toThrow(); + } + for (const outside of [field.minimum - 1, field.maximum + 1]) { + const issues = collectorIssues(atIntegerBoundary(defaultCollector(id), fieldName, outside)); + expect(issues).toContainEqual(expect.objectContaining({ + path: `collectors.0.config.${fieldName}`, + code: 'number_range', + bounds: { min: field.minimum, max: field.maximum } + })); + } + }); + if (field.maximum_field) { + it(`${id}.${fieldName} enforces the catalog's cross-field ceiling`, () => { + const referenced = schema.fields[field.maximum_field]; + expect(referenced?.type).toBe('integer'); + if (!referenced || referenced.type !== 'integer') return; + const collector = defaultCollector(id); + configRecord(collector)[field.maximum_field!] = referenced.minimum; + configRecord(collector)[fieldName] = Math.max(field.minimum, referenced.minimum + 1); + const issues = collectorIssues(collector); + expect(issues).toContainEqual(expect.objectContaining({ + path: `collectors.0.config.${fieldName}` + })); + expect(() => parseConfiguration(canonicalConfigurationBytes(withCollector(collector)))) + .toThrow('parse_invalid'); + }); + } + } else if (field.type === 'boolean') { + it(`${id}.${fieldName} accepts only a JSON boolean`, () => { + const raw = rawConfiguration(defaultCollector(id)); + const config = ((raw.collectors as Array>)[0] + .config as Record); + config[fieldName] = 'false'; + expect(() => parseConfiguration(canonicalBytes(raw))).toThrow('parse_boolean'); + }); + } else if (field.type === 'enum') { + it(`${id}.${fieldName} accepts exactly the catalog enum`, () => { + expect(field.enum).toContain(configRecord(defaultCollector(id))[fieldName]); + const raw = rawConfiguration(defaultCollector(id)); + const config = ((raw.collectors as Array>)[0] + .config as Record); + config[fieldName] = 'NOT_IN_CATALOG'; + expect(() => parseConfiguration(canonicalBytes(raw))).toThrow('parse_collector'); + }); + } else { + it(`${id}.${fieldName} accepts only catalog enum members`, () => { + const actual = configRecord(defaultCollector(id))[fieldName] as string[]; + expect(actual.length).toBeGreaterThanOrEqual(field.minimum_items); + expect(actual.length).toBeLessThanOrEqual(field.maximum_items); + expect(actual.every((value) => field.items_enum.includes(value))).toBe(true); + const raw = rawConfiguration(defaultCollector(id)); + const config = ((raw.collectors as Array>)[0] + .config as Record); + config[fieldName] = ['not-in-catalog']; + expect(() => parseConfiguration(canonicalBytes(raw))).toThrow('parse_collector'); + }); + } + } + } +}); diff --git a/tools/conformance/typescript/protocol-vectors.spec.ts b/tools/conformance/typescript/protocol-vectors.spec.ts new file mode 100644 index 0000000..69f1ae1 --- /dev/null +++ b/tools/conformance/typescript/protocol-vectors.spec.ts @@ -0,0 +1,109 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { openBundle } from '../../../web/src/lib/adc/bundle'; +import { parseCanonicalJson } from '../../../web/src/lib/adc/canonical'; +import { verify } from '../../../web/src/lib/adc/crypto'; +import { decodeEnvelope } from '../../../web/src/lib/adc/envelope'; +import { encodeJoinLink, parseJoinLink } from '../../../web/src/lib/adc/join'; +import { parseConfiguration } from '../../../web/src/routes/researcher/parse'; + +type Vector = { + category: string; + entrypoint: 'canonical_json' | 'configuration_jcs' | 'signed_configuration' | 'bundle' | 'bundle_unwrap_context' | 'receipt'; + expected_failure: string; + id: string; + input_hex: string; +}; + +const corpus = JSON.parse( + readFileSync(new URL('../../../protocol/v1/conformance-vectors.json', import.meta.url), 'utf8') +) as { + hostile: Vector[]; + valid: { + bundle: Record; + canonical_json: { canonical_jcs_utf8_hex: string }; + signed_configuration: Record; + upload_receipt: { canonical_jcs_utf8_hex: string; value: Record }; + }; +}; +const joinCorpus = JSON.parse( + readFileSync(new URL('../../../protocol/v1/join-link-vectors.json', import.meta.url), 'utf8') +) as { + hostile: Array<{ encoded: string; id: string }>; + valid: { + artifact_sha256: string; + artifact_url: string; + encoded: string; + signer_fingerprint: string; + }; +}; +const bytes = (hex: string) => Uint8Array.from(Buffer.from(hex, 'hex')); + +describe('shared Protocol v1 conformance corpus', () => { + const signed = corpus.valid.signed_configuration; + const configurationBytes = bytes(signed.canonical_jcs_utf8_hex); + const configuration = parseConfiguration(configurationBytes); + const bundle = corpus.valid.bundle; + const container = bytes(bundle.container_hex); + + it('is consumed by the actual configuration and bundle readers', async () => { + expect(parseCanonicalJson(bytes(corpus.valid.canonical_json.canonical_jcs_utf8_hex))) + .toBeTypeOf('object'); + const envelope = decodeEnvelope(bytes(signed.envelope_hex)); + expect(envelope.signerKeyId).toBe(signed.signer_key_id); + expect(envelope.configurationBytes).toEqual(configurationBytes); + expect(verify(configurationBytes, envelope.signature, signed.signer_public_key_base64url)).toBe(true); + + const opened = await openBundle(container, configuration, bundle.researcher_private_key_base64url); + expect(opened.ok).toBe(true); + if (!opened.ok) return; + expect(new TextEncoder().encode(opened.bundle.text)).toEqual(bytes(bundle.document_jcs_utf8_hex)); + }); + + for (const vector of corpus.hostile) { + if (vector.entrypoint === 'receipt') continue; // The TypeScript receiver owns this entrypoint. + it(`rejects ${vector.id}`, async () => { + const input = bytes(vector.input_hex); + if (vector.entrypoint === 'canonical_json') { + expect(() => parseCanonicalJson(input)).toThrow(); + } else if (vector.entrypoint === 'configuration_jcs') { + expect(() => parseConfiguration(input)).toThrow(); + } else if (vector.entrypoint === 'signed_configuration') { + expect(() => { + const envelope = decodeEnvelope(input); + parseConfiguration(envelope.configurationBytes); + if (!verify(envelope.configurationBytes, envelope.signature, signed.signer_public_key_base64url)) { + throw new Error('configuration_signature_invalid'); + } + }).toThrow(); + } else if (vector.entrypoint === 'bundle') { + await expect(openBundle(input, configuration, bundle.researcher_private_key_base64url)) + .resolves.toMatchObject({ ok: false }); + } else { + const wrongContext = parseCanonicalJson(input) as { bundle_id: string }; + const mutated = container.slice(); + mutated.set(bytes(wrongContext.bundle_id.replaceAll('-', '')), 8); + await expect(openBundle(mutated, configuration, bundle.researcher_private_key_base64url)) + .resolves.toMatchObject({ ok: false, failure: 'unwrap_failed' }); + } + }); + } +}); + +describe('shared Protocol v1 join-link corpus', () => { + it('accepts the canonical join link byte-for-byte', () => { + const parsed = parseJoinLink(joinCorpus.valid.encoded); + expect(parsed).toEqual({ + artifactUrl: joinCorpus.valid.artifact_url, + artifactSha256: joinCorpus.valid.artifact_sha256, + signerFingerprint: joinCorpus.valid.signer_fingerprint + }); + expect(encodeJoinLink(parsed)).toBe(joinCorpus.valid.encoded); + }); + + for (const vector of joinCorpus.hostile) { + it(`rejects ${vector.id}`, () => { + expect(() => parseJoinLink(vector.encoded)).toThrow(); + }); + } +}); diff --git a/tools/conformance/vitest.config.ts b/tools/conformance/vitest.config.ts new file mode 100644 index 0000000..407e10d --- /dev/null +++ b/tools/conformance/vitest.config.ts @@ -0,0 +1,16 @@ +import { fileURLToPath } from 'node:url'; + +const repository = fileURLToPath(new URL('../..', import.meta.url)); + +export default { + root: `${repository}/web`, + resolve: { + alias: { $lib: `${repository}/web/src/lib` } + }, + server: { + fs: { allow: [repository] } + }, + test: { + include: [`${repository}/tools/conformance/typescript/**/*.spec.ts`] + } +}; diff --git a/tools/generate_protocol_vectors.mjs b/tools/generate_protocol_vectors.mjs new file mode 100644 index 0000000..d53749e --- /dev/null +++ b/tools/generate_protocol_vectors.mjs @@ -0,0 +1,469 @@ +#!/usr/bin/env node +/** Generate the deterministic, language-neutral Protocol v1 conformance corpus. */ + +import { createHash } from 'node:crypto'; +import { readFile, writeFile } from 'node:fs/promises'; +import { ed25519, x25519 } from '../web/node_modules/@noble/curves/ed25519.js'; +import { expand, extract } from '../web/node_modules/@noble/hashes/hkdf.js'; +import { sha256 } from '../web/node_modules/@noble/hashes/sha2.js'; + +const UTF8 = new TextEncoder(); +const EMPTY = new Uint8Array(); +const output = new URL('../protocol/v1/conformance-vectors.json', import.meta.url); +const concat = (...values) => { + const result = new Uint8Array(values.reduce((sum, value) => sum + value.length, 0)); + let offset = 0; + for (const value of values) { + result.set(value, offset); + offset += value.length; + } + return result; +}; +const hex = (value) => Buffer.from(value).toString('hex'); +const base64url = (value) => Buffer.from(value).toString('base64url'); +const digest = (value) => createHash('sha256').update(value).digest(); +const u16 = (value) => Uint8Array.of(value >>> 8, value & 255); +const u32 = (value) => Uint8Array.of(value >>> 24, value >>> 16, value >>> 8, value).map((x) => x & 255); +const canonical = (value) => { + if (value === null || typeof value !== 'object') return JSON.stringify(value); + if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`; + return `{${Object.keys(value).sort().map((key) => `${JSON.stringify(key)}:${canonical(value[key])}`).join(',')}}`; +}; +const bytes = (value) => UTF8.encode(canonical(value)); +const clone = (value) => structuredClone(value); +const xorLast = (value) => { + const result = value.slice(); + result[result.length - 1] ^= 1; + return result; +}; + +const i2osp2 = (value) => u16(value); +const suiteKem = concat(UTF8.encode('KEM'), i2osp2(0x20)); +const suiteHpke = concat(UTF8.encode('HPKE'), i2osp2(0x20), i2osp2(1), i2osp2(2)); +const version = UTF8.encode('HPKE-v1'); +const labeledExtract = (suite, salt, label, ikm) => + extract(sha256, concat(version, suite, UTF8.encode(label), ikm), salt); +const labeledExpand = (suite, prk, label, info, length) => + expand(sha256, prk, concat(i2osp2(length), version, suite, UTF8.encode(label), info), length); + +async function aesGcm(keyBytes, nonce, plaintext, aad) { + const key = await crypto.subtle.importKey('raw', keyBytes, 'AES-GCM', false, ['encrypt']); + return new Uint8Array(await crypto.subtle.encrypt( + { name: 'AES-GCM', iv: nonce, additionalData: aad, tagLength: 128 }, + key, + plaintext + )); +} + +async function hpkeSeal(recipientPublic, ephemeralPrivate, plaintext, info) { + const enc = x25519.getPublicKey(ephemeralPrivate); + const dh = x25519.getSharedSecret(ephemeralPrivate, recipientPublic); + const eaePrk = labeledExtract(suiteKem, EMPTY, 'eae_prk', dh); + const shared = labeledExpand(suiteKem, eaePrk, 'shared_secret', concat(enc, recipientPublic), 32); + const schedule = concat( + Uint8Array.of(0), + labeledExtract(suiteHpke, EMPTY, 'psk_id_hash', EMPTY), + labeledExtract(suiteHpke, EMPTY, 'info_hash', info) + ); + const secret = labeledExtract(suiteHpke, shared, 'secret', EMPTY); + const key = labeledExpand(suiteHpke, secret, 'key', schedule, 32); + const nonce = labeledExpand(suiteHpke, secret, 'base_nonce', schedule, 12); + return concat(enc, await aesGcm(key, nonce, plaintext, EMPTY)); +} + +const signerPrivate = Uint8Array.from({ length: 32 }, (_, index) => index + 1); +const signerPublic = ed25519.getPublicKey(signerPrivate); +const researcherPrivate = Uint8Array.from({ length: 32 }, (_, index) => 0x41 + index); +const researcherPublic = x25519.getPublicKey(researcherPrivate); +const configuration = { + assigned_participant_id: null, + collectors: [ + { + config: { maximum_report_latency_us: 1000000, sampling_period_us: 100000 }, + id: 'accelerometer.v1', + required: false + }, + { config: {}, id: 'app_lifecycle.v1', required: true } + ], + configuration_id: 'vector-config', + consent: { document_version: 'v1', summary: 'Protocol vector consent.' }, + duration_hours: 24, + expires_at: '2030-01-01T00:00:00Z', + experiment_id: 'vector-study', + export: { hpke_public_key: base64url(researcherPublic), researcher_key_id: 'vector-hpke' }, + interventions: [], + issued_at: '2026-01-01T00:00:00Z', + minimum_client_version: '7', + platform: 'android', + purpose: 'Exercise the destructive Protocol v1 contract.', + researcher: { contact: 'vector@example.invalid', name: 'Protocol Vector' }, + schema_version: 1, + signer: { key_id: 'vector-signer', public_key: base64url(signerPublic) }, + storage: { maximum_local_bytes: 16777216 }, + surveys: [], + title: 'Protocol vector', + upload: {} +}; +const configurationBytes = bytes(configuration); +const configurationSha = digest(configurationBytes); +const signature = ed25519.sign(configurationBytes, signerPrivate); +const signerKeyId = UTF8.encode(configuration.signer.key_id); +const envelope = concat( + UTF8.encode('ADCCFG01'), + u16(signerKeyId.length), + u32(configurationBytes.length), + signerKeyId, + configurationBytes, + signature +); + +const bundleId = '00000000-0000-4000-8000-000000000099'; +const bundleIdBytes = Uint8Array.from(Buffer.from(bundleId.replaceAll('-', ''), 'hex')); +const contextValue = { + bundle_format: 'research-bundle-v1', + bundle_id: bundleId, + configuration_sha256: hex(configurationSha), + researcher_key_id: 'vector-hpke' +}; +const context = bytes(contextValue); +const documentValue = { + bundle_id: bundleId, + bundle_kind: 'automatic_upload', + configuration, + configuration_sha256: hex(configurationSha), + configuration_signature: { signature: base64url(signature), signer_key_id: 'vector-signer' }, + experiment: { + assigned_participant_id: null, + configuration_id: 'vector-config', + durable_through_sequence: '1', + event_count: '1', + events: [{ + collector_id: 'app_lifecycle.v1', + fields: { activity_class: 'vector.Activity' }, + observed_time: { + boot_session_id: 'boot-vector', + monotonic_time_nanos: '2000', + wall_time_utc_millis: '1000' + }, + payload_schema_version: 1, + payload_type: 'ACTIVITY_CREATED', + sequence_number: '1' + }], + experiment_id: 'vector-study', + first_sequence_number: '1', + last_sequence_number: '1', + next_sequence_number: '2', + participant_instance_id: '00000000-0000-4000-8000-000000000017', + retained_from_sequence: '1', + state: 'RUNNING', + transitions: [ + { + from: 'IMPORTED', + reason: 'CONFIGURATION_SIGNATURE_VERIFIED', + time: { boot_session_id: 'boot-vector', monotonic_time_nanos: '100', wall_time_utc_millis: '100' }, + to: 'CONFIG_VERIFIED' + }, + { + from: 'CONFIG_VERIFIED', + reason: 'CONSENT_REVIEW_OPENED', + time: { boot_session_id: 'boot-vector', monotonic_time_nanos: '200', wall_time_utc_millis: '200' }, + to: 'CONSENT_PENDING' + }, + { + from: 'CONSENT_PENDING', + reason: 'CONSENT_ACCEPTED', + time: { boot_session_id: 'boot-vector', monotonic_time_nanos: '300', wall_time_utc_millis: '300' }, + to: 'ACCESS_SETUP' + }, + { + from: 'ACCESS_SETUP', + reason: 'ACCESS_PREFLIGHT_PASSED', + time: { boot_session_id: 'boot-vector', monotonic_time_nanos: '400', wall_time_utc_millis: '400' }, + to: 'READY' + }, + { + from: 'READY', + reason: 'PARTICIPANT_STARTED', + time: { boot_session_id: 'boot-vector', monotonic_time_nanos: '500', wall_time_utc_millis: '500' }, + to: 'RUNNING' + } + ], + uploaded_through_sequence: '0' + }, + exported_at_utc_millis: '10000', + format: 'research-bundle-v1', + producer: { client_version: '7', platform: 'android' } +}; +const documentBytes = bytes(documentValue); +const contentKey = Uint8Array.from({ length: 32 }, (_, index) => 0xa0 + index); +const contentNonce = Uint8Array.from({ length: 12 }, (_, index) => 0x10 + index); +const ephemeralPrivate = Uint8Array.from({ length: 32 }, (_, index) => 0x21 + index); +const wrappedKey = await hpkeSeal(researcherPublic, ephemeralPrivate, contentKey, context); +const documentCiphertext = await aesGcm(contentKey, contentNonce, documentBytes, context); +const bundlePrefixForNonce = (nonce) => concat( + UTF8.encode('ADCEXP01'), + bundleIdBytes, + configurationSha, + u16(UTF8.encode('vector-hpke').length), + nonce, + UTF8.encode('vector-hpke'), + wrappedKey +); +const bundlePrefix = bundlePrefixForNonce(contentNonce); +const bundle = concat(bundlePrefix, documentCiphertext); +let semanticNonceCounter = 0; +const authenticatedBundle = async (document) => { + // These are public deterministic fixtures, but each authenticated hostile still models the + // production invariant that one AES-GCM key never repeats a nonce. + const nonce = contentNonce.slice(); + nonce[nonce.length - 1] += ++semanticNonceCounter; + return concat( + bundlePrefixForNonce(nonce), + await aesGcm(contentKey, nonce, bytes(document), context) + ); +}; +const bundleSha = digest(bundle); +const receiptValue = { + bundle_id: bundleId, + byte_count: String(bundle.length), + configuration_sha256: hex(configurationSha), + event_count: '1', + first_sequence_number: '1', + last_sequence_number: '1', + sha256: hex(bundleSha) +}; +const receiptBytes = bytes(receiptValue); +const oldConfig = concat(envelope.slice(0, 14), u16(64), envelope.slice(14)); +const wrongDigestBundle = bundle.slice(); +wrongDigestBundle[24] ^= 1; + +const configurationVariant = (change) => { + const value = clone(configuration); + change(value); + return bytes(value); +}; +const signedVariant = (payload, signatureBytes = signature) => concat( + UTF8.encode('ADCCFG01'), + u16(signerKeyId.length), + u32(payload.length), + signerKeyId, + payload, + signatureBytes +); +const semanticBundle = async (change) => { + const value = clone(documentValue); + change(value); + return authenticatedBundle(value); +}; +const withInnerBundleMismatch = await semanticBundle((value) => { + value.bundle_id = '00000000-0000-4000-8000-000000000098'; +}); +const withEmbeddedConfigurationMember = await semanticBundle((value) => { + value.configuration.unexpected = true; +}); +const withInnerDigestMismatch = await semanticBundle((value) => { + value.configuration_sha256 = '00'.repeat(32); +}); +const withInvalidSignature = await semanticBundle((value) => { + value.configuration_signature.signature = base64url(xorLast(signature)); +}); +const withOldProducer = await semanticBundle((value) => { + value.producer.client_version = '6'; +}); +const withUnknownRootMember = await semanticBundle((value) => { + value.unexpected = true; +}); +const withRangeCountMismatch = await semanticBundle((value) => { + value.experiment.event_count = '2'; + value.experiment.last_sequence_number = '2'; +}); +const withSequenceGap = await semanticBundle((value) => { + const second = clone(value.experiment.events[0]); + second.sequence_number = '3'; + value.experiment.events.push(second); + value.experiment.durable_through_sequence = '3'; + value.experiment.event_count = '2'; + value.experiment.last_sequence_number = '3'; + value.experiment.next_sequence_number = '4'; +}); +const withUnknownCollector = await semanticBundle((value) => { + value.experiment.events[0].collector_id = 'unknown.v1'; +}); +const withUnknownPayload = await semanticBundle((value) => { + value.experiment.events[0].payload_type = 'UNKNOWN_PAYLOAD'; +}); +const withUnknownPayloadSchema = await semanticBundle((value) => { + value.experiment.events[0].payload_schema_version = 2; +}); +const withUnknownEventField = await semanticBundle((value) => { + value.experiment.events[0].fields.unexpected = 'value'; +}); +const withOversizedEventField = await semanticBundle((value) => { + value.experiment.events[0].fields.activity_class = 'x'.repeat(513); +}); +const sensorFloatBundle = (floatValue) => semanticBundle((value) => { + value.experiment.events[0] = { + collector_id: 'accelerometer.v1', + fields: { + accuracy: '3', + source_elapsed_realtime_nanos: '2000', + x_meters_per_second_squared: floatValue, + y_meters_per_second_squared: '0', + z_meters_per_second_squared: '9.81' + }, + observed_time: { + boot_session_id: 'boot-vector', + monotonic_time_nanos: '2000', + wall_time_utc_millis: '1000' + }, + payload_schema_version: 1, + payload_type: 'ACCELEROMETER_SAMPLE', + sequence_number: '1' + }; +}); +const withNonfiniteSensor = await sensorFloatBundle('NaN'); +const withEmptySensorFloat = await sensorFloatBundle(''); +const withWhitespaceSensorFloat = await sensorFloatBundle(' '); +const withHexSensorFloat = await sensorFloatBundle('0x10'); +const withBinarySensorFloat = await sensorFloatBundle('0b10'); +const batteryBundle = (percentage) => semanticBundle((value) => { + value.experiment.events[0] = { + collector_id: 'battery_state.v1', + fields: { + charging_source: 'NONE', + charging_state: 'DISCHARGING', + percentage, + power_save_enabled: 'false' + }, + observed_time: { + boot_session_id: 'boot-vector', + monotonic_time_nanos: '2000', + wall_time_utc_millis: '1000' + }, + payload_schema_version: 1, + payload_type: 'BATTERY_STATE', + sequence_number: '1' + }; +}); +const withNoncanonicalInt32 = await batteryBundle('-0'); +const withOutOfRangeInt32 = await batteryBundle('101'); +const withInvalidTransitions = await semanticBundle((value) => { + value.experiment.transitions[4].reason = 'PARTICIPANT_PAUSED'; +}); +const withLongBootSession = await semanticBundle((value) => { + value.experiment.events[0].observed_time.boot_session_id = 'b'.repeat(129); +}); +const withTrailingBundleByte = concat(bundle, Uint8Array.of(0)); +const signaturePayload = configurationVariant((value) => { value.title = 'Wrong signature input'; }); +const validCanonicalJson = bytes({ + '\r': 'Carriage Return', + '1': 'One', + '\u0080': 'Control', + 'ö': 'Latin Small Letter O With Diaeresis', + '€': 'Euro Sign', + '😀': 'Emoji: Grinning Face', + 'דּ': 'Hebrew Letter Dalet With Dagesh' +}); + +const corpus = { + corpus_format: 'adc-protocol-conformance-v1', + hostile: [ + { category: 'unicode_jcs', entrypoint: 'canonical_json', expected_failure: 'utf16_key_order', id: 'jcs-wrong-utf16-key-order', input_hex: hex(UTF8.encode('{"":0,"𐀀":1}')) }, + { category: 'unicode_jcs', entrypoint: 'canonical_json', expected_failure: 'malformed_utf8', id: 'jcs-malformed-utf8', input_hex: '7b2278223a22c328227d' }, + { category: 'unicode_jcs', entrypoint: 'canonical_json', expected_failure: 'unpaired_surrogate', id: 'jcs-unpaired-surrogate', input_hex: hex(UTF8.encode('{"x":"\\ud800"}')) }, + { category: 'unicode_jcs', entrypoint: 'canonical_json', expected_failure: 'noncanonical_escape', id: 'jcs-noncanonical-unicode-escape', input_hex: hex(UTF8.encode('{"x":"\\u0061"}')) }, + { category: 'integral_bounds', entrypoint: 'canonical_json', expected_failure: 'negative_zero', id: 'jcs-negative-zero', input_hex: hex(UTF8.encode('{"n":-0}')) }, + { category: 'trailing_bytes', entrypoint: 'canonical_json', expected_failure: 'trailing_whitespace', id: 'jcs-trailing-whitespace', input_hex: hex(concat(validCanonicalJson, UTF8.encode('\n'))) }, + { category: 'unknown_field', entrypoint: 'configuration_jcs', expected_failure: 'duplicate_member', id: 'config-duplicate-member', input_hex: hex(UTF8.encode(canonical(configuration).replace('{', '{"assigned_participant_id":null,'))) }, + { category: 'old_v1', entrypoint: 'configuration_jcs', expected_failure: 'legacy_field', id: 'config-old-v1-field', input_hex: hex(UTF8.encode(canonical(configuration).replace('"minimum_client_version":"7"', '"minimum_app_version":7'))) }, + { category: 'integral_bounds', entrypoint: 'configuration_jcs', expected_failure: 'nonintegral_number', id: 'config-nonintegral-duration', input_hex: hex(configurationVariant((value) => { value.duration_hours = 1.5; })) }, + { category: 'integral_bounds', entrypoint: 'configuration_jcs', expected_failure: 'int64_overflow', id: 'config-client-version-overflow', input_hex: hex(configurationVariant((value) => { value.minimum_client_version = '9223372036854775808'; })) }, + { category: 'integral_bounds', entrypoint: 'configuration_jcs', expected_failure: 'physical_bound', id: 'config-zero-duration', input_hex: hex(configurationVariant((value) => { value.duration_hours = 0; })) }, + { category: 'raw_key_encoding', entrypoint: 'configuration_jcs', expected_failure: 'padded_base64url', id: 'config-padded-signing-key', input_hex: hex(configurationVariant((value) => { value.signer.public_key += '='; })) }, + { category: 'raw_key_encoding', entrypoint: 'configuration_jcs', expected_failure: 'wrong_key_length', id: 'config-short-hpke-key', input_hex: hex(configurationVariant((value) => { value.export.hpke_public_key = base64url(researcherPublic.slice(0, -1)); })) }, + { category: 'raw_key_encoding', entrypoint: 'configuration_jcs', expected_failure: 'legacy_tink_keyset', id: 'config-tink-hpke-keyset', input_hex: hex(configurationVariant((value) => { value.export.hpke_public_key = '{"primaryKeyId":1,"key":[]}'; })) }, + { category: 'trailing_bytes', entrypoint: 'configuration_jcs', expected_failure: 'noncanonical_json', id: 'config-leading-whitespace', input_hex: hex(concat(UTF8.encode(' '), configurationBytes)) }, + { category: 'old_v1', entrypoint: 'signed_configuration', expected_failure: 'old_v1_framing', id: 'adccfg-old-signature-length', input_hex: hex(oldConfig) }, + { category: 'malformed_length', entrypoint: 'signed_configuration', expected_failure: 'zero_key_length', id: 'adccfg-zero-key-length', input_hex: hex(concat(envelope.slice(0, 8), u16(0), envelope.slice(10))) }, + { category: 'malformed_length', entrypoint: 'signed_configuration', expected_failure: 'truncated', id: 'adccfg-truncated', input_hex: hex(envelope.slice(0, -1)) }, + { category: 'trailing_bytes', entrypoint: 'signed_configuration', expected_failure: 'trailing_byte', id: 'adccfg-trailing-byte', input_hex: hex(concat(envelope, Uint8Array.of(0))) }, + { category: 'signature_input', entrypoint: 'signed_configuration', expected_failure: 'signature_payload_mismatch', id: 'adccfg-wrong-signature-input', input_hex: hex(signedVariant(signaturePayload)) }, + { category: 'signature_input', entrypoint: 'signed_configuration', expected_failure: 'tampered_signature', id: 'adccfg-tampered-signature', input_hex: hex(signedVariant(configurationBytes, xorLast(signature))) }, + { category: 'old_v1', entrypoint: 'bundle', expected_failure: 'old_v1_framing', id: 'bundle-old-zero-header', input_hex: hex(concat(UTF8.encode('ADCEXP01'), new Uint8Array(128))) }, + { category: 'outer_inner_identity', entrypoint: 'bundle', expected_failure: 'configuration_digest_mismatch', id: 'bundle-wrong-configuration-digest', input_hex: hex(wrongDigestBundle) }, + { category: 'body_tampering', entrypoint: 'bundle', expected_failure: 'aead_authentication', id: 'bundle-tampered-tag', input_hex: hex(xorLast(bundle)) }, + { category: 'malformed_length', entrypoint: 'bundle', expected_failure: 'truncated', id: 'bundle-truncated', input_hex: hex(bundle.slice(0, -1)) }, + { category: 'malformed_length', entrypoint: 'bundle', expected_failure: 'zero_key_length', id: 'bundle-zero-key-length', input_hex: hex(bundle.map((byte, index) => index === 56 || index === 57 ? 0 : byte)) }, + { category: 'trailing_bytes', entrypoint: 'bundle', expected_failure: 'aead_authentication', id: 'bundle-trailing-byte', input_hex: hex(withTrailingBundleByte) }, + { category: 'outer_inner_identity', entrypoint: 'bundle', expected_failure: 'inner_bundle_id', id: 'bundle-inner-id-mismatch', input_hex: hex(withInnerBundleMismatch) }, + { category: 'outer_inner_identity', entrypoint: 'bundle', expected_failure: 'embedded_configuration', id: 'bundle-embedded-configuration-member', input_hex: hex(withEmbeddedConfigurationMember) }, + { category: 'outer_inner_identity', entrypoint: 'bundle', expected_failure: 'inner_configuration_digest', id: 'bundle-inner-configuration-digest', input_hex: hex(withInnerDigestMismatch) }, + { category: 'signature_input', entrypoint: 'bundle', expected_failure: 'embedded_signature', id: 'bundle-invalid-embedded-signature', input_hex: hex(withInvalidSignature) }, + { category: 'integral_bounds', entrypoint: 'bundle', expected_failure: 'minimum_client_version', id: 'bundle-old-producer', input_hex: hex(withOldProducer) }, + { category: 'unknown_field', entrypoint: 'bundle', expected_failure: 'unknown_root_member', id: 'bundle-unknown-root-member', input_hex: hex(withUnknownRootMember) }, + { category: 'range_count', entrypoint: 'bundle', expected_failure: 'event_count', id: 'bundle-range-count-mismatch', input_hex: hex(withRangeCountMismatch) }, + { category: 'range_count', entrypoint: 'bundle', expected_failure: 'noncontiguous_sequence', id: 'bundle-sequence-gap', input_hex: hex(withSequenceGap) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'unknown_collector', id: 'bundle-unknown-collector', input_hex: hex(withUnknownCollector) }, + { category: 'unknown_payload', entrypoint: 'bundle', expected_failure: 'unknown_payload', id: 'bundle-unknown-payload', input_hex: hex(withUnknownPayload) }, + { category: 'unknown_payload', entrypoint: 'bundle', expected_failure: 'unknown_payload_schema', id: 'bundle-unknown-payload-schema', input_hex: hex(withUnknownPayloadSchema) }, + { category: 'unknown_field', entrypoint: 'bundle', expected_failure: 'unknown_event_field', id: 'bundle-unknown-event-field', input_hex: hex(withUnknownEventField) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'field_bound', id: 'bundle-oversized-event-field', input_hex: hex(withOversizedEventField) }, + { category: 'nonfinite_sensor', entrypoint: 'bundle', expected_failure: 'nonfinite_float', id: 'bundle-nonfinite-sensor', input_hex: hex(withNonfiniteSensor) }, + { category: 'float_grammar', entrypoint: 'bundle', expected_failure: 'empty_float', id: 'bundle-empty-sensor-float', input_hex: hex(withEmptySensorFloat) }, + { category: 'float_grammar', entrypoint: 'bundle', expected_failure: 'whitespace_float', id: 'bundle-whitespace-sensor-float', input_hex: hex(withWhitespaceSensorFloat) }, + { category: 'float_grammar', entrypoint: 'bundle', expected_failure: 'hex_float', id: 'bundle-hex-sensor-float', input_hex: hex(withHexSensorFloat) }, + { category: 'float_grammar', entrypoint: 'bundle', expected_failure: 'binary_float', id: 'bundle-binary-sensor-float', input_hex: hex(withBinarySensorFloat) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'noncanonical_int32', id: 'bundle-noncanonical-int32', input_hex: hex(withNoncanonicalInt32) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'int32_field_bound', id: 'bundle-int32-field-bound', input_hex: hex(withOutOfRangeInt32) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'transition_chain', id: 'bundle-invalid-transition', input_hex: hex(withInvalidTransitions) }, + { category: 'catalog_contract', entrypoint: 'bundle', expected_failure: 'boot_session_bound', id: 'bundle-long-boot-session', input_hex: hex(withLongBootSession) }, + { category: 'hpke_context', entrypoint: 'bundle_unwrap_context', expected_failure: 'hpke_authentication', id: 'bundle-wrong-context', input_hex: hex(bytes({ ...contextValue, bundle_id: '00000000-0000-4000-8000-000000000098' })) }, + { category: 'trailing_bytes', entrypoint: 'receipt', expected_failure: 'noncanonical_json', id: 'receipt-leading-whitespace', input_hex: hex(concat(UTF8.encode(' '), receiptBytes)) }, + { category: 'integral_bounds', entrypoint: 'receipt', expected_failure: 'noncanonical_decimal', id: 'receipt-leading-zero', input_hex: hex(UTF8.encode(canonical(receiptValue).replace('"first_sequence_number":"1"', '"first_sequence_number":"01"'))) }, + { category: 'integral_bounds', entrypoint: 'receipt', expected_failure: 'wrong_type', id: 'receipt-numeric-count', input_hex: hex(UTF8.encode(canonical(receiptValue).replace('"event_count":"1"', '"event_count":1'))) }, + { category: 'integral_bounds', entrypoint: 'receipt', expected_failure: 'int64_overflow', id: 'receipt-byte-count-overflow', input_hex: hex(UTF8.encode(canonical(receiptValue).replace(`"byte_count":"${bundle.length}"`, '"byte_count":"9223372036854775808"'))) }, + { category: 'trailing_bytes', entrypoint: 'receipt', expected_failure: 'trailing_byte', id: 'receipt-trailing-byte', input_hex: hex(concat(receiptBytes, Uint8Array.of(0))) } + ], + schema_version: 1, + valid: { + canonical_json: { + canonical_jcs_utf8_hex: hex(validCanonicalJson) + }, + bundle: { + bundle_id: bundleId, + container_hex: hex(bundle), + content_key_hex: hex(contentKey), + content_nonce_hex: hex(contentNonce), + context_jcs_utf8_hex: hex(context), + document_jcs_utf8_hex: hex(documentBytes), + hpke_ephemeral_private_key_base64url: base64url(ephemeralPrivate), + hpke_wrapped_content_key_hex: hex(wrappedKey), + researcher_private_key_base64url: base64url(researcherPrivate), + researcher_public_key_base64url: base64url(researcherPublic), + sha256: hex(bundleSha) + }, + signed_configuration: { + canonical_jcs_sha256: hex(configurationSha), + canonical_jcs_utf8_hex: hex(configurationBytes), + envelope_hex: hex(envelope), + signature_base64url: base64url(signature), + signer_key_id: 'vector-signer', + signer_private_key_base64url: base64url(signerPrivate), + signer_public_key_base64url: base64url(signerPublic) + }, + upload_receipt: { + canonical_jcs_utf8_hex: hex(receiptBytes), + value: receiptValue + } + } +}; + +const encoded = `${JSON.stringify(corpus, null, 2)}\n`; +if (process.argv.includes('--check')) { + const checkedIn = await readFile(output, 'utf8'); + if (checkedIn !== encoded) throw new Error('Protocol conformance corpus is stale; regenerate it'); + console.log('Protocol v1 conformance corpus is reproducible'); +} else { + await writeFile(output, encoded); + console.log(`wrote ${output.pathname}`); +} diff --git a/tools/protocol-conformance.init.gradle b/tools/protocol-conformance.init.gradle new file mode 100644 index 0000000..58860bb --- /dev/null +++ b/tools/protocol-conformance.init.gradle @@ -0,0 +1,14 @@ +// Adds the repository-owned shared-vector consumer without making a protocol test part of one +// platform module's source of truth. +gradle.beforeProject { project -> + if (project.path == ":core:export") { + project.pluginManager.withPlugin("org.jetbrains.kotlin.jvm") { + project.extensions.getByName("kotlin") + .sourceSets.getByName("test") + .kotlin.srcDir(project.rootProject.file("tools/conformance/kotlin")) + project.tasks.withType(org.gradle.api.tasks.testing.Test).configureEach { + systemProperty("adc.repository.root", project.rootProject.projectDir.absolutePath) + } + } + } +} diff --git a/tools/tests/__init__.py b/tools/tests/__init__.py new file mode 100644 index 0000000..ca6aa30 --- /dev/null +++ b/tools/tests/__init__.py @@ -0,0 +1 @@ +"""Tests for repository-local validation tools.""" diff --git a/tools/tests/test_catalog.py b/tools/tests/test_catalog.py new file mode 100644 index 0000000..54cfc97 --- /dev/null +++ b/tools/tests/test_catalog.py @@ -0,0 +1,80 @@ +from __future__ import annotations + +import copy +import json +import tempfile +import unittest +from pathlib import Path + +from tools import catalog + + +class CatalogTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.valid = catalog.load(catalog.DEFAULT_CATALOG) + + def test_repository_catalog_is_valid(self) -> None: + catalog.validate(self.valid, catalog.ROOT) + + def test_checked_in_kotlin_contract_is_generated_from_catalog(self) -> None: + catalog.check_kotlin_contract(self.valid) + + def test_stale_kotlin_contract_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "ProtocolEventContracts.kt" + path.write_text("// stale\n", encoding="utf-8") + with self.assertRaisesRegex(catalog.CatalogError, "stale"): + catalog.check_kotlin_contract(self.valid, path) + + def test_duplicate_json_member_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "catalog.json" + path.write_text('{"catalog_format":"a","catalog_format":"b"}', encoding="utf-8") + with self.assertRaisesRegex(catalog.CatalogError, "duplicate"): + catalog.load(path) + + def test_unknown_member_is_rejected(self) -> None: + hostile = copy.deepcopy(self.valid) + hostile["fallback"] = True + with self.assertRaisesRegex(catalog.CatalogError, "unknown"): + catalog.validate(hostile) + + def test_unsorted_payload_types_are_rejected(self) -> None: + hostile = copy.deepcopy(self.valid) + hostile["collectors"][2]["payloads"][0]["types"].reverse() + with self.assertRaisesRegex(catalog.CatalogError, "sorted and unique"): + catalog.validate(hostile) + + def test_configuration_required_must_be_an_array(self) -> None: + hostile = copy.deepcopy(self.valid) + collector = next(item for item in hostile["collectors"] if item["id"] == "accelerometer.v1") + collector["configuration"]["required"] = "" + with self.assertRaisesRegex(catalog.CatalogError, "must be an array"): + catalog.validate(hostile) + + def test_cross_field_bound_must_reference_an_integer_field(self) -> None: + hostile = copy.deepcopy(self.valid) + collector = next(item for item in hostile["collectors"] if item["id"] == "location.v1") + collector["configuration"]["fields"]["minimum_interval_millis"]["maximum_field"] = "priority" + with self.assertRaisesRegex(catalog.CatalogError, "not another integer field"): + catalog.validate(hostile) + + def test_payload_enum_values_are_required(self) -> None: + hostile = copy.deepcopy(self.valid) + collector = next(item for item in hostile["collectors"] if item["id"] == "keyboard_touch.v1") + del collector["payloads"][0]["fields"]["action"]["enum"] + with self.assertRaisesRegex(catalog.CatalogError, "wrong members"): + catalog.validate(hostile) + + def test_float_configuration_value_is_rejected_while_loading(self) -> None: + encoded = json.dumps(self.valid).replace('"maximum": 60000000', '"maximum": 1.5', 1) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "catalog.json" + path.write_text(encoded, encoding="utf-8") + with self.assertRaisesRegex(catalog.CatalogError, "non-integral"): + catalog.load(path) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/tests/test_catalog_parity.py b/tools/tests/test_catalog_parity.py new file mode 100644 index 0000000..cefd316 --- /dev/null +++ b/tools/tests/test_catalog_parity.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +import copy +import unittest + +from tools import catalog, catalog_parity + + +class CatalogParityTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.valid = catalog.load(catalog.DEFAULT_CATALOG) + + def test_repository_platform_projections_match(self) -> None: + catalog_parity.check(self.valid) + + def test_generated_event_contract_drift_is_rejected(self) -> None: + hostile = copy.deepcopy(self.valid) + collector = next(item for item in hostile["collectors"] if item["id"] == "accelerometer.v1") + collector["maximum_encoded_event_bytes"] += 1 + with self.assertRaisesRegex(catalog_parity.ParityError, "generated Kotlin event contract"): + catalog_parity.check(hostile) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/tests/test_collector_assurance.py b/tools/tests/test_collector_assurance.py new file mode 100644 index 0000000..b274f3b --- /dev/null +++ b/tools/tests/test_collector_assurance.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path + +from tools import collector_assurance + + +@unittest.skipUnless(shutil.which("javac"), "javac is required for class-file assurance tests") +class CollectorAssuranceTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.policy = collector_assurance.load_policy(collector_assurance.DEFAULT_POLICY) + + def compile_references(self, body: str) -> collector_assurance.ClassReferences: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = root / "Fixture.java" + source.write_text(f"public final class Fixture {{ {body} }}", encoding="utf-8") + subprocess.run( + ["javac", "-d", str(root), str(source)], + check=True, + capture_output=True, + text=True, + ) + return collector_assurance.parse_class((root / "Fixture.class").read_bytes()) + + def test_harmless_class_passes(self) -> None: + references = self.compile_references("public int value() { return 7; }") + self.assertEqual([], collector_assurance.violations(references, self.policy)) + + def test_file_api_is_rejected_from_bytecode(self) -> None: + references = self.compile_references( + 'public boolean value() { return new java.io.File("x").exists(); }' + ) + self.assertTrue( + any("java/io/File" in item for item in collector_assurance.violations(references, self.policy)) + ) + + def test_file_type_in_a_method_descriptor_is_rejected(self) -> None: + references = self.compile_references( + "public void value(java.io.BufferedWriter writer) {}" + ) + self.assertTrue( + any( + "java/io/BufferedWriter" in item + for item in collector_assurance.violations(references, self.policy) + ) + ) + + def test_network_api_is_rejected_from_bytecode(self) -> None: + references = self.compile_references( + 'public String value() throws Exception { return new java.net.URL("https://example.invalid").getHost(); }' + ) + self.assertTrue( + any("java/net/URL" in item for item in collector_assurance.violations(references, self.policy)) + ) + + def test_dynamic_loading_is_rejected_from_bytecode(self) -> None: + references = self.compile_references( + 'public Class value() throws Exception { return Class.forName("Fixture"); }' + ) + self.assertTrue( + any("Class.forName" in item for item in collector_assurance.violations(references, self.policy)) + ) + + def test_context_file_database_and_event_log_bypasses_are_rejected(self) -> None: + references = collector_assurance.ClassReferences( + classes=frozenset( + { + "android/app/DownloadManager", + "android/util/EventLog", + } + ), + methods=frozenset( + { + collector_assurance.MemberReference( + "android/content/Context", "deleteFile" + ), + collector_assurance.MemberReference( + "android/content/ContextWrapper", "fileList" + ), + collector_assurance.MemberReference( + "android/app/Application", "deleteDatabase" + ), + collector_assurance.MemberReference( + "android/util/EventLog", "writeEvent" + ), + } + ), + ) + + violations = collector_assurance.violations(references, self.policy) + + self.assertTrue(any("DownloadManager" in item for item in violations)) + self.assertTrue(any("EventLog" in item for item in violations)) + self.assertTrue(any("Context.deleteFile" in item for item in violations)) + self.assertTrue(any("ContextWrapper.fileList" in item for item in violations)) + self.assertTrue(any("Application.deleteDatabase" in item for item in violations)) + + def test_unrecognized_gradle_dependency_form_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as directory: + module = Path(directory) + (module / "src/main").mkdir(parents=True) + (module / "build.gradle.kts").write_text( + 'dependencies { implementation(files("collector.jar")) }\n', + encoding="utf-8", + ) + errors = collector_assurance._source_violations(module, self.policy) + self.assertTrue(any("unrecognized dependency declaration" in item for item in errors)) + + def test_junit_is_the_only_allowed_collector_test_dependency(self) -> None: + with tempfile.TemporaryDirectory() as directory: + module = Path(directory) + (module / "src/main").mkdir(parents=True) + build_file = module / "build.gradle.kts" + build_file.write_text( + "dependencies { testImplementation(libs.junit4) }\n", + encoding="utf-8", + ) + junit_errors = collector_assurance._source_violations(module, self.policy) + build_file.write_text( + "dependencies { testImplementation(libs.mockk) }\n", + encoding="utf-8", + ) + arbitrary_errors = collector_assurance._source_violations(module, self.policy) + + self.assertEqual([], junit_errors) + self.assertTrue(any("forbidden test dependency libs.mockk" in item for item in arbitrary_errors)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/validate_protocol_vectors.py b/tools/validate_protocol_vectors.py new file mode 100644 index 0000000..e248793 --- /dev/null +++ b/tools/validate_protocol_vectors.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +"""Validate the framing, hashes, and hostile coverage of shared Protocol v1 vectors.""" + +from __future__ import annotations + +import hashlib +import json +import re +import struct +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +VECTORS = ROOT / "protocol/v1/conformance-vectors.json" +JOIN_VECTORS = ROOT / "protocol/v1/join-link-vectors.json" + + +def fail(message: str) -> None: + raise ValueError(message) + + +def reject_duplicate_members(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + fail(f"duplicate corpus member: {key}") + result[key] = value + return result + + +def raw(value: str, label: str) -> bytes: + if not isinstance(value, str) or len(value) % 2 or not re.fullmatch(r"[0-9a-f]*", value): + raise ValueError(f"{label} is not lowercase even-length hex") + try: + return bytes.fromhex(value) + except ValueError as error: + raise ValueError(f"{label} is not lowercase even-length hex") from error + + +def canonical_json(value: object) -> bytes: + """Integral-only RFC 8785 bytes, including UTF-16 object-member ordering.""" + if value is None or isinstance(value, (bool, int, str)): + return json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + if isinstance(value, list): + return b"[" + b",".join(canonical_json(item) for item in value) + b"]" + if isinstance(value, dict): + keys = sorted(value, key=lambda item: item.encode("utf-16-be", "surrogatepass")) + return b"{" + b",".join( + canonical_json(key) + b":" + canonical_json(value[key]) for key in keys + ) + b"}" + fail(f"unsupported canonical JSON value: {type(value).__name__}") + + +def validate(path: Path = VECTORS) -> None: + value = json.loads( + path.read_text(encoding="utf-8"), + object_pairs_hook=reject_duplicate_members, + parse_float=lambda value: fail(f"non-integral corpus number: {value}"), + parse_constant=lambda value: fail(f"invalid corpus number: {value}"), + ) + if set(value) != {"corpus_format", "hostile", "schema_version", "valid"}: + fail("corpus root is not closed-world") + if value["corpus_format"] != "adc-protocol-conformance-v1" or value["schema_version"] != 1: + fail("corpus identity is wrong") + valid = value["valid"] + if set(valid) != {"bundle", "canonical_json", "signed_configuration", "upload_receipt"}: + fail("valid corpus is incomplete") + if set(valid["canonical_json"]) != {"canonical_jcs_utf8_hex"}: + fail("valid canonical JSON fixture is not closed-world") + jcs = raw(valid["canonical_json"]["canonical_jcs_utf8_hex"], "canonical JSON") + if canonical_json(json.loads(jcs)) != jcs: + fail("canonical JSON Unicode fixture is not RFC 8785 ordered") + config = valid["signed_configuration"] + if set(config) != { + "canonical_jcs_sha256", + "canonical_jcs_utf8_hex", + "envelope_hex", + "signature_base64url", + "signer_key_id", + "signer_private_key_base64url", + "signer_public_key_base64url", + }: + fail("valid signed configuration fixture is not closed-world") + canonical = raw(config["canonical_jcs_utf8_hex"], "configuration") + if canonical_json(json.loads(canonical)) != canonical: + fail("configuration fixture is not canonical JSON") + if hashlib.sha256(canonical).hexdigest() != config["canonical_jcs_sha256"]: + fail("configuration digest mismatch") + envelope = raw(config["envelope_hex"], "configuration envelope") + if envelope[:8] != b"ADCCFG01": + fail("configuration magic mismatch") + key_length, config_length = struct.unpack(">HI", envelope[8:14]) + if envelope[14 : 14 + key_length].decode() != config["signer_key_id"]: + fail("configuration key ID mismatch") + if envelope[14 + key_length : 14 + key_length + config_length] != canonical: + fail("configuration frame does not contain canonical bytes") + if len(envelope) != 14 + key_length + config_length + 64: + fail("configuration frame has the wrong fixed signature tail") + bundle = valid["bundle"] + if set(bundle) != { + "bundle_id", + "container_hex", + "content_key_hex", + "content_nonce_hex", + "context_jcs_utf8_hex", + "document_jcs_utf8_hex", + "hpke_ephemeral_private_key_base64url", + "hpke_wrapped_content_key_hex", + "researcher_private_key_base64url", + "researcher_public_key_base64url", + "sha256", + }: + fail("valid bundle fixture is not closed-world") + container = raw(bundle["container_hex"], "bundle") + if container[:8] != b"ADCEXP01" or hashlib.sha256(container).hexdigest() != bundle["sha256"]: + fail("bundle framing or digest mismatch") + key_length = struct.unpack(">H", container[56:58])[0] + key_id = container[70 : 70 + key_length].decode() + context_bytes = raw(bundle["context_jcs_utf8_hex"], "context") + context = json.loads(context_bytes) + if canonical_json(context) != context_bytes: + fail("bundle context is not canonical JSON") + if key_id != context["researcher_key_id"] or container[24:56].hex() != context["configuration_sha256"]: + fail("bundle context does not match framing") + if len(raw(bundle["hpke_wrapped_content_key_hex"], "wrapped key")) != 80: + fail("HPKE wrapped key must be 80 bytes") + receipt = valid["upload_receipt"] + if set(receipt) != {"canonical_jcs_utf8_hex", "value"} or set(receipt["value"]) != { + "bundle_id", + "byte_count", + "configuration_sha256", + "event_count", + "first_sequence_number", + "last_sequence_number", + "sha256", + }: + fail("valid receipt fixture is not closed-world") + receipt_bytes = raw(receipt["canonical_jcs_utf8_hex"], "receipt") + if receipt_bytes != canonical_json(receipt["value"]): + fail("receipt is not canonical") + if receipt["value"]["sha256"] != bundle["sha256"] or receipt["value"]["byte_count"] != str(len(container)): + fail("receipt does not describe the valid bundle") + hostile = value["hostile"] + ids = [item["id"] for item in hostile] + if len(ids) != len(set(ids)): + fail("hostile vector IDs are not unique") + required = { + "canonical_json", + "configuration_jcs", + "signed_configuration", + "bundle", + "bundle_unwrap_context", + "receipt", + } + if {item["entrypoint"] for item in hostile} != required: + fail("hostile corpus does not cover every protocol entrypoint") + required_categories = { + "body_tampering", + "catalog_contract", + "hpke_context", + "integral_bounds", + "malformed_length", + "nonfinite_sensor", + "old_v1", + "outer_inner_identity", + "range_count", + "raw_key_encoding", + "signature_input", + "trailing_bytes", + "unicode_jcs", + "unknown_field", + "unknown_payload", + } + actual_categories = {item["category"] for item in hostile} + if not required_categories <= actual_categories: + fail(f"hostile corpus misses normative categories: {sorted(required_categories-actual_categories)}") + for item in hostile: + if set(item) != {"category", "entrypoint", "expected_failure", "id", "input_hex"}: + fail(f"hostile vector {item.get('id')} is not closed-world") + if not all(isinstance(item[key], str) and item[key] for key in ("category", "entrypoint", "expected_failure", "id")): + fail(f"hostile vector {item.get('id')} has an empty label") + raw(item["input_hex"], item["id"]) + + +def validate_join(path: Path = JOIN_VECTORS) -> None: + value = json.loads( + path.read_text(encoding="utf-8"), + object_pairs_hook=reject_duplicate_members, + parse_float=lambda item: fail(f"non-integral join corpus number: {item}"), + parse_constant=lambda item: fail(f"invalid join corpus number: {item}"), + ) + if set(value) != {"corpus_format", "hostile", "schema_version", "valid"}: + fail("join corpus root is not closed-world") + if value["corpus_format"] != "adc-join-link-conformance-v1" or value["schema_version"] != 1: + fail("join corpus identity is wrong") + valid = value["valid"] + if set(valid) != {"artifact_sha256", "artifact_url", "encoded", "signer_fingerprint"}: + fail("valid join fixture is not closed-world") + if not isinstance(valid["artifact_sha256"], str) or not re.fullmatch( + r"[0-9a-f]{64}", valid["artifact_sha256"] + ): + fail("valid join digest is malformed") + if not isinstance(valid["signer_fingerprint"], str) or not re.fullmatch( + r"[0-9A-F]{32}", valid["signer_fingerprint"] + ): + fail("valid join fingerprint is malformed") + if not isinstance(valid["artifact_url"], str) or not valid["artifact_url"]: + fail("valid join artifact URL is malformed") + if ( + not isinstance(valid["encoded"], str) + or not valid["encoded"].isascii() + or len(valid["encoded"]) > 4096 + ): + fail("valid join encoding is malformed") + hostile = value["hostile"] + if not isinstance(hostile, list) or not hostile: + fail("join hostile corpus is empty") + ids = [item.get("id") for item in hostile] + if len(ids) != len(set(ids)): + fail("join hostile vector IDs are not unique") + for item in hostile: + if set(item) != {"encoded", "id"}: + fail(f"hostile join vector {item.get('id')} is not closed-world") + if not all(isinstance(item[key], str) and item[key] for key in ("encoded", "id")): + fail("hostile join vector has an empty value") + if not item["encoded"].isascii() or len(item["encoded"]) > 4096: + fail(f"hostile join vector {item['id']} is malformed") + + +if __name__ == "__main__": + try: + validate() + validate_join() + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as error: + print(f"protocol vector error: {error}", file=sys.stderr) + raise SystemExit(1) + print("valid Protocol v1 conformance corpora") diff --git a/web/CONTRACT.md b/web/CONTRACT.md index 38c9f7a..e3aa796 100644 --- a/web/CONTRACT.md +++ b/web/CONTRACT.md @@ -1,148 +1,132 @@ -# Module contract - -The site is static and runs entirely in the browser: no server, no network calls, no analytics. Keys -are generated in the tab and never leave it. Everything below is the shape each module must expose so -the parts can be written independently. - -Read `src/lib/adc/types.ts` first — it is the schema contract, transcribed from -`core/study-definition`. - -## Non-negotiable encoding facts - -These were measured against the shipped `researcher-tools` CLI on a real build. They are not -guesses, and code that contradicts them produces a file the Android app rejects. - -1. **String escaping** is Gson `JsonWriter`'s default table, *not* HTML-safe. Escape `"` → `\"`, - `\` → `\\`, `\b` `\f` `\n` `\r` `\t` to their short forms, every other code point below `0x20` - as `\u00xx` with lowercase hex, and U+2028 / U+2029 as `
` / `
`. Leave everything - else alone: `/`, `<`, `>`, `&`, `=`, `'`, DEL (0x7F), and all non-ASCII — CJK and emoji are - emitted raw as UTF-8. -2. **`minimum_displacement_meters` is a Kotlin `Float`** and is written by Java's - `Float.toString()`. Round the value to float32, then emit the shortest decimal that round-trips - to that float32, always with at least one digit after the point. Observed: `0` → `0.0`, - `5` → `5.0`, `100` → `100.0`, `0.25` → `0.25`, `0.1` → `0.1`, `0.3` → `0.3`, - `9999.999` → `9999.999`, `1234.5678` → `1234.5677`. JavaScript's `Number.prototype.toString` - gives the shortest form for a *double* and is wrong here. -3. **Every other number** is an integer literal matching `-?(0|[1-9][0-9]*)`. No exponents, no - leading zeros, no trailing `.0`. -4. **`tink_hpke_public_keyset`** is re-emitted from Gson's `JsonObject.toString()`: compact, no - whitespace, keys in the order they appeared. Emit it in the same order the keyset was built in. -5. **Root key order is fixed** by `StudyConfigurationCodec.encode` and is not alphabetical. The v1 - shape includes `assigned_participant_id`, `surveys`, and `interventions`; the former prompt shape - is invalid and has no compatibility path. Take the complete order from that function. -6. **`upload: null`** encodes as `"upload":{}`. - -## `src/lib/adc/canonical.ts` - -```ts -export function formatFloat(value: number): string; -export function escapeJsonString(value: string): string; -export function canonicalize(configuration: StudyConfiguration): string; -export function canonicalBytes(configuration: StudyConfiguration): Uint8Array; +# Web Protocol v1 contract + +The site is static and runs entirely in the browser: no server, network calls, analytics, or key +persistence. The normative cross-language specification lives in `../protocol/v1/`; this document +maps that protocol onto the Web source so a new contributor can find a behavior and its tests. + +## Where to start + +| Concern | Source | Focused tests | +| --- | --- | --- | +| Configuration types and bounds | `src/lib/adc/types.ts` | `tests/hostile.spec.ts` | +| RFC 8785 JCS and configuration projection | `src/lib/adc/canonical.ts` | `tests/canonical.spec.ts` | +| Raw Ed25519/X25519 keys | `src/lib/adc/crypto.ts` | `tests/crypto.spec.ts` | +| Signed configuration framing | `src/lib/adc/envelope.ts` | `tests/crypto.spec.ts` | +| Configuration closed-world reader | `src/routes/researcher/parse.ts` | `tests/hostile.spec.ts` | +| Encrypted bundle reader | `src/lib/adc/bundle.ts` | `tests/bundle.spec.ts` | +| Authoring state and stale-signature rule | `src/routes/researcher/draft.svelte.ts` | `tests/researcher-draft.spec.ts` | +| Downloaded artifacts | `src/routes/researcher/artifacts.ts` | `tests/researcher.spec.ts` | +| Immutable join URI and local QR | `src/lib/adc/join.ts`, `src/routes/researcher/JoinLinkPanel.svelte` | `tests/join.spec.ts`, shared `join-link-vectors.json` | +| Local deterministic protocol boundary | all of the above | `tests/compat.spec.ts` | + +There is no compatibility reader. Former Protocol v1 Tink keysets, protobuf prefixes, PKCS#8, +X.509, padded Base64, floating-point displacement, and variable signature framing are invalid. + +## Canonical configuration + +`canonicalize(value)` is a generic RFC 8785 JCS primitive. It recursively sorts object member names +by UTF-16 code units, uses ECMAScript JSON primitive serialization, and rejects non-finite numbers, +unsupported values, cycles, sparse arrays, and lone surrogates. `parseCanonicalJson(bytes)` accepts +only fatal UTF-8 whose JCS re-encoding is byte-identical; this also rejects whitespace, duplicate +members, alternate number spellings, and trailing content. + +The configuration's typed in-memory model uses `upload: null` when upload is disabled. Protocol v1 +has one wire shape, `"upload":{}`. `configurationValue`, `canonicalizeConfiguration`, and +`canonicalConfigurationBytes` own this explicit boundary projection. They also normalize instants +and the set-like network transport list exactly as the Android codec does. Schema rules do not leak +into the generic JCS primitive. + +Configuration-specific Protocol v1 changes are: + +- `platform` is exactly `"android"`; +- `minimum_client_version` is a positive canonical decimal string; +- `signer.public_key` is a raw 32-byte Ed25519 key; +- `export.hpke_public_key` is a raw 32-byte X25519 key; +- both keys use canonical unpadded base64url; +- `location.v1.minimum_displacement_millimeters` is an integer JSON number. + +`parseConfiguration` first requires canonical, closed-world bytes, builds the typed value without +defaults, re-encodes it to prove Android normalization equality, applies all schema validation, and, +for `.adccfg`, verifies signer identity and Ed25519 signature. It never drops an unknown member or +repairs an old shape. + +## Keys and signed configuration + +Both private artifacts are one unpadded base64url string containing exactly 32 raw bytes. Public +halves are always derived locally. Nothing reads an accompanying public key or wrapper metadata. + +`ADCCFG01` is exactly: + +```text +magic[8] | signer_key_id_length u16 BE | configuration_length u32 BE | +signer_key_id UTF-8 | configuration JCS | Ed25519 signature[64] ``` -`canonicalize` must produce the exact bytes `researcher-tools canonicalize` would. This is the one -module with a byte-level test against the real CLI. +The signature covers the configuration JCS bytes. There is no signature-length member and the +container must end after byte 64 of the signature. -## `src/lib/adc/schema.ts` +## Encrypted bundle reader -```ts -export interface Issue { path: string; code: string } -export function validate(configuration: StudyConfiguration): Issue[]; -export function emptyConfiguration(): StudyConfiguration; -export function defaultCollector(id: CollectorId): CollectorConfig; -``` - -`validate` returns every problem rather than throwing on the first, because the UI marks fields. -`code` is a stable identifier the i18n layer maps to a message; never a sentence. - -## `src/lib/adc/crypto.ts` - -```ts -export interface SigningKeyPair { privatePkcs8Base64: string; publicX509Base64: string } -export function generateSigningKeyPair(): SigningKeyPair; -export function sign(configurationBytes: Uint8Array, privatePkcs8Base64: string): Uint8Array; -export function verify(configurationBytes: Uint8Array, signature: Uint8Array, publicX509Base64: string): boolean; -export function fingerprint(publicX509Base64: string): string; -``` - -Ed25519 via `@noble/ed25519`. The private key is PKCS#8 DER, the public key X.509 -SubjectPublicKeyInfo, both Base64 — byte-identical to what `KeyPairGenerator.getInstance("Ed25519")` -emits, because the CLI reads these files. The fingerprint is SHA-256 over the *decoded* public key, -first 16 bytes, as eight uppercase groups of four hex characters separated by single spaces. - -## `src/lib/adc/tink.ts` +The browser reader is a bounded convenience reader; large-study analysis belongs in the offline +Python pipeline. `ADCEXP01` is exactly: -```ts -export interface HpkeKeyset { publicKeyset: TinkKeyset; privateKeyset: TinkKeyset } -export function generateHpkeKeyset(): HpkeKeyset; +```text +magic[8] | bundle UUID[16] | configuration SHA-256[32] | +researcher_key_id_length u16 BE | content nonce[12] | researcher_key_id UTF-8 | +RFC 9180 wrapped content key[80] | AES-256-GCM encrypted JCS document and tag ``` -Tink `DHKEM_X25519_HKDF_SHA256 / HKDF_SHA256 / AES_256_GCM`. The `value` field is a hand-encoded -protobuf, Base64 (standard alphabet, padded). `HpkePublicKey` is `{2: HpkeParams{1:1, 2:1, 3:2}, -3: publicKey}`; `HpkePrivateKey` is `{2: HpkePublicKey, 3: privateKey}`; field 1 (version 0) is -omitted in both. `keyId` is a random uint32 that must be non-zero and equal to `primaryKeyId`, and -`outputPrefixType` is `TINK`. Compare against `researcher-tools/examples/INSECURE-demo-hpke-*.json` -— those are real, working keysets and the format must match them exactly. +The wrapped key is raw RFC 9180 output (`enc[32] + ciphertext[48]`) with no Tink prefix. The HPKE +info and document AAD are the JCS bytes of: -## `src/lib/adc/envelope.ts` - -```ts -export function encodeEnvelope(signerKeyId: string, configurationBytes: Uint8Array, signature: Uint8Array): Uint8Array; +```json +{ + "bundle_format": "research-bundle-v1", + "bundle_id": "lowercase UUID", + "configuration_sha256": "64 lowercase hex", + "researcher_key_id": "key ID" +} ``` -`ADCCFG01` (8 ASCII bytes), then big-endian `uint16` key-ID length, `int32` configuration length, -`uint16` signature length, then the key ID as UTF-8, the configuration bytes, and the signature. - -## i18n +The authenticated document uses exact root objects for producer, signature provenance, and +experiment state. Every sequence, counter, wall time, and monotonic time is a canonical decimal +string, so values above JavaScript's safe-integer limit are never rounded. A successful read has +verified both AEAD layers, JCS bytes, the embedded configuration digest/signature, every repeated +identity, event count, event range, and event ordering. A failure publishes no plaintext object. -The catalogues live in `src/lib/i18n` (`types.ts` for the `Messages` shape, `en.ts`, `zh-TW.ts`, -and `messages.ts` as the seam). The reactive half is `src/lib/ui/i18n.svelte.ts`, because runes -compile only in a `.svelte.ts` module and every component that reads a message is already importing -from `$lib/ui`. - -```ts -export type Locale = 'en' | 'zh-TW'; -export const messages: Record; -``` +## Authoring and artifacts -The chosen locale persists in `localStorage` under `adc.locale`, which the inline script in -`app.html` also reads so the first paint is not in the wrong language. With nothing stored the -browser decides. `` is written as `zh-Hant-TW` rather than `zh-TW`, because CSS language -matching is prefix-based and the CJK type block hangs off `:lang(zh-Hant)`. Both locales must have -the same keys, and `tests/i18n.spec.ts` asserts it recursively. +`draft.svelte.ts` is the single state owner. The editable configuration carries inert IDs; the +document derives experiment, configuration, signer, and export key IDs. A signature is associated +with one canonical string, and any edit immediately retires the signature and envelope. -## `src/lib/ui` +The four downloads are raw Ed25519 private key, raw X25519 private key, canonical configuration +JSON, and signed `.adccfg`. The two configuration artifacts do not exist until signing succeeds. +Private bytes stay in the tab and are never written to browser storage. -Nord palette, both themes. Shared components: the language control, the step rail, the icon set -(drawn as inline SVG, no icon dependency), field controls, and the download tiles. Text is the last -resort here: the interface is meant to be legible without reading, with words added only where a -picture genuinely cannot carry the meaning. +## Immutable join artifact -## Tests +After an envelope exists, `JoinLinkPanel.svelte` accepts one artifact URL, delegates the exact +Protocol v1 URL / URI bytes to `join.ts`, and renders the resulting QR locally with the bundled +`qrcode` library. There is no QR service, fetch, upload, polling, or browser persistence. The URI +binds the envelope's complete SHA-256 and signing fingerprint; editing the study retires the +envelope and therefore the join artifact. -`pnpm test` runs the unit suites, including `tests/compat.spec.ts`, which shells out to -`researcher-tools` — rebuilding its distribution before the suite — and asserts byte for byte that this -encoder and the Kotlin one agree, then signs a study here and has `check-config` accept it. +`join.ts` implements the same deliberately narrow HTTPS profile and uppercase percent encoding as +Kotlin `JoinLink`. It never delegates canonicalization to WHATWG `URL`: that parser is used only as +an equality check after the lexical profile succeeds. Personalized authoring additionally rejects +the assigned participant ID anywhere in the URL and requires a final opaque base64url path token +of at least 22 characters. The normative syntax, bounds, trust flow, and common Kotlin/TypeScript +corpus are in `../protocol/v1/README.md` and `../protocol/v1/join-link-vectors.json`. -`pnpm e2e` is separate because it needs a build, a static server, and a browser. It drives the -researcher page the way a person would and ends by handing the resulting `.adccfg` to the same CLI. -It exists because the unit suites prove the library and prove nothing about the page: its first run -found a collector card marked `aria-disabled` while switched off, which made the only control that -could switch it on unavailable to assistive technology, and nothing else in the project would have -caught that. +## Verification -Two more browser runs share that server and gate on the researcher page alone. `pnpm e2e:one-line` -measures every leaf run of interface text and fails if one needs a second line, because the length a -string may be is a constraint and this is the thing that measures it. `pnpm e2e:units` switches on -all seven collectors and fails if a control shows a number in the unit the *file* stores rather than -the unit a person states — `100000` beside `Hz`, `1073741824` where `1 GiB` was meant. Both say in -their own headers what they cannot catch; read that before reading a green run as a proof. - -``` +```sh +pnpm test +pnpm check pnpm build -pnpm exec vite preview --port 4173 --strictPort # or any static server over build/ -pnpm e2e -pnpm e2e:one-line -pnpm e2e:units ``` + +`tests/compat.spec.ts` consumes the normative shared valid/hostile corpus under `../protocol/v1/` +and also keeps focused RFC 8032 and deterministic local round-trip cases. Browser E2E scripts +remain separate because they require a built static site and Playwright. diff --git a/web/e2e/researcher-flow.mjs b/web/e2e/researcher-flow.mjs index b14ef69..23e3a30 100644 --- a/web/e2e/researcher-flow.mjs +++ b/web/e2e/researcher-flow.mjs @@ -19,21 +19,21 @@ * comparison worth making — and because agreement alone would still hold for a page that * invented one string and printed it everywhere, both key names are also recomputed here from * the key material in the signed file, by an implementation that is not the site's. - * 2. The bytes are Gson's bytes. The study text below is deliberately hostile — an em dash, CJK, + * 2. The bytes are RFC 8785 JCS bytes. The study text below is deliberately hostile — an em dash, CJK, * an emoji, a quote, a backslash, a newline, and characters Gson leaves alone — and the * canonical JSON the page hands over is fed back through `researcher-tools canonicalize`, - * which must return it unchanged. An encoder that "helpfully" escapes `<` or `&`, or that - * emits `—` for the em dash, fails here and only here. + * which must return it unchanged. * * pnpm build && pnpm exec http-server build -p 4173 # or any static server * node e2e/researcher-flow.mjs */ import { chromium } from 'playwright'; import { execFileSync } from 'node:child_process'; -import { createHash, createPrivateKey, createPublicKey } from 'node:crypto'; +import { createHash } from 'node:crypto'; import { mkdtempSync, existsSync, readFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import { ed25519, x25519 } from '@noble/curves/ed25519.js'; const ORIGIN = process.env.ORIGIN ?? 'http://localhost:4173'; const CLI = join( @@ -92,12 +92,12 @@ await page.goto(`${ORIGIN}/researcher/`, { waitUntil: 'networkidle' }); // Keys, which is the second step now: the page opens on the study, so the rail has to be used to // get here. Nothing is typed and no button is pressed to make the keys — the step generates both // pairs on arrival, and both names derive from the key material. The two tiles are taken by the -// names they carry, which are those derived names with `-private.key` / `-private.json` after them. +// names they carry, which are those derived names with `-private.key` after them. await page.locator('[data-testid="rail-keys"]').click(); await page.waitForSelector('[data-testid="step-keys"]'); await page.waitForTimeout(600); const signingTile = page.getByRole('button', { name: /signer-[0-9a-z]{13}-private\.key$/ }); -const hpkeTile = page.getByRole('button', { name: /export-[0-9a-z]{13}-private\.json$/ }); +const hpkeTile = page.getByRole('button', { name: /export-[0-9a-z]{13}-private\.key$/ }); if ((await signingTile.count()) !== 1) fail('the keys step did not make a signing key on arrival'); if ((await hpkeTile.count()) !== 1) fail('the keys step did not make an export key on arrival'); // The step offers nothing to type at all. A key-ID field here is the thing that was removed, and a @@ -224,7 +224,7 @@ for (const id of ['read-configuration-session', 'read-key-session']) { await page.locator(`[data-testid="${id}"]`).click(); } const staged = await page.locator('[data-testid="step-read"] .note').allTextContents(); -for (const name of ['study-canonical.json', 'export-hpke-private.json']) { +for (const name of ['study-canonical.json', 'export-hpke-private.key']) { if (!staged.some((line) => line.includes(name))) { fail(`the read step did not name ${name} after taking it from this tab`); } @@ -241,7 +241,7 @@ if (problems.length) fail('the page logged errors:\n ' + problems.join('\n ')) // `researcher-tools sign --key-id` wants. for (const name of [ `${signerKeyId}-private.key`, - `${exportKeyId}-private.json`, + `${exportKeyId}-private.key`, 'study-canonical.json', 'study.adccfg' ]) { @@ -323,9 +323,9 @@ claim( 'Required was pressed on Motion alone and App activity came out required too' ); -// Gson's escape table and nothing else: these appear as themselves, and these are escaped. +// JCS preserves non-ASCII text and uses JSON escapes only where the string grammar requires them. for (const raw of ['—', '瀏覽器建立 🔬', '正體中文 & = fine.']) { - claim(text.includes(raw), `the encoder escaped ${JSON.stringify(raw)}, which Gson emits raw`); + claim(text.includes(raw), `the encoder changed ${JSON.stringify(raw)}`); } for (const escaped of ['E2E Lab \\"Verification\\" \\\\ Group', 'app activity.\\nStays on the phone']) { claim(text.includes(escaped), `the encoder did not write ${JSON.stringify(escaped)}`); @@ -342,8 +342,7 @@ for (const escaped of ['E2E Lab \\"Verification\\" \\\\ Group', 'app activity.\\ // // `lib/adc/ids.ts` is re-implemented below from its own specification rather than imported — a // derivation checked against itself proves nothing. Sixty-four bits of SHA-256 over a -// domain-separated *raw* public key (not the DER, not the Tink JSON), as thirteen base-36 -// characters behind a word stem. +// domain-separated raw public key, as thirteen base-36 characters behind a word stem. // --------------------------------------------------------------------------------------------- const keyTag = (domain, raw) => @@ -358,50 +357,10 @@ const keyTag = (domain, raw) => .toString(36) .padStart(13, '0'); -/** The one length-delimited field `number` at the top level of a protobuf message, or `null`. */ -function field(message, number) { - let at = 0; - const varint = () => { - let value = 0; - let shift = 0; - for (;;) { - const byte = message[at]; - at += 1; - value += (byte & 0x7f) * 2 ** shift; - if ((byte & 0x80) === 0) return value; - shift += 7; - } - }; - while (at < message.length) { - const key = varint(); - const wire = key & 7; - if (wire === 2) { - const length = varint(); - if (key >>> 3 === number) return message.subarray(at, at + length); - at += length; - } else if (wire === 0) varint(); - else if (wire === 5) at += 4; - else if (wire === 1) at += 8; - else throw new Error(`unreadable protobuf wire type ${wire}`); - } - return null; -} - -/** X.509 SubjectPublicKeyInfo for Ed25519: fixed length, so the last 32 bytes are the key. */ -const X509_PREFIX = Buffer.from('302a300506032b6570032100', 'hex'); - -const spki = Buffer.from(document.signer.public_key, 'base64'); -claim( - spki.length === 44 && spki.subarray(0, 12).equals(X509_PREFIX), - 'signer.public_key is not an X.509 Ed25519 key' -); -const signerRaw = spki.subarray(12); - -const exportRaw = field( - Buffer.from(document.export.tink_hpke_public_keyset.key[0].keyData.value, 'base64'), - 3 -); -claim(exportRaw?.length === 32, 'the public keyset in the file carries no 32-byte X25519 point'); +const signerRaw = Buffer.from(document.signer.public_key, 'base64url'); +const exportRaw = Buffer.from(document.export.hpke_public_key, 'base64url'); +claim(signerRaw.length === 32, 'the configuration carries no raw 32-byte Ed25519 public key'); +claim(exportRaw.length === 32, 'the configuration carries no raw 32-byte X25519 public key'); claim( document.signer.key_id === `signer-${keyTag('adc:signer-key-id:v1:', signerRaw)}`, @@ -421,46 +380,37 @@ claim( // And the two files the researcher keeps hold those same keys, so a file named after a key is // named after the key that is actually inside it. const signingPrivate = readFileSync(join(out, `${signerKeyId}-private.key`), 'utf8'); -const derivedSpki = createPublicKey( - createPrivateKey({ - key: Buffer.from(signingPrivate.trim(), 'base64'), - format: 'der', - type: 'pkcs8' - }) -).export({ format: 'der', type: 'spki' }); +const derivedSigningPublic = ed25519.getPublicKey(Buffer.from(signingPrivate.trim(), 'base64url')); claim( - Buffer.from(derivedSpki).equals(spki), + Buffer.from(derivedSigningPublic).equals(signerRaw), `${signerKeyId}-private.key is not the key the configuration is signed under` ); -const privateKeyset = JSON.parse(readFileSync(join(out, `${exportKeyId}-private.json`), 'utf8')); -const privateValue = Buffer.from(privateKeyset.key[0].keyData.value, 'base64'); -const publicInPrivate = field(privateValue, 2); -claim(publicInPrivate !== null, `${exportKeyId}-private.json carries no public half`); +const exportPrivate = readFileSync(join(out, `${exportKeyId}-private.key`), 'utf8'); +const derivedExportPublic = x25519.getPublicKey(Buffer.from(exportPrivate.trim(), 'base64url')); claim( - Buffer.from(field(publicInPrivate, 3) ?? []).equals(exportRaw), - `${exportKeyId}-private.json cannot decrypt what this study encrypts to` + Buffer.from(derivedExportPublic).equals(exportRaw), + `${exportKeyId}-private.key cannot decrypt what this study encrypts to` ); // The envelope carries exactly the canonical bytes the page also handed over as a file. A // researcher who archives one and distributes the other is archiving the right thing. -// `ADCCFG01`, uint16 key-id length, int32 configuration length, uint16 signature length. +// `ADCCFG01`, uint16 key-id length, uint32 configuration length, key ID, JCS, signature64. const envelope = readFileSync(join(out, 'study.adccfg')); claim(envelope.subarray(0, 8).toString('latin1') === 'ADCCFG01', 'the envelope has no magic'); const keyIdLength = envelope.readUInt16BE(8); -const configurationLength = envelope.readInt32BE(10); -const signatureLength = envelope.readUInt16BE(14); +const configurationLength = envelope.readUInt32BE(10); claim( - envelope.length === 16 + keyIdLength + configurationLength + signatureLength, + envelope.length === 14 + keyIdLength + configurationLength + 64, 'the envelope lengths do not add up to its size' ); claim( - envelope.subarray(16, 16 + keyIdLength).toString('utf8') === signerKeyId, + envelope.subarray(14, 14 + keyIdLength).toString('utf8') === signerKeyId, 'the envelope names a signer other than the one the page showed' ); claim( envelope - .subarray(16 + keyIdLength, 16 + keyIdLength + configurationLength) + .subarray(14 + keyIdLength, 14 + keyIdLength + configurationLength) .equals(canonical), 'the .adccfg does not carry the canonical JSON the page offered beside it' ); diff --git a/web/e2e/units.mjs b/web/e2e/units.mjs index 115e91c..ee564de 100644 --- a/web/e2e/units.mjs +++ b/web/e2e/units.mjs @@ -14,7 +14,7 @@ * * Exit code: non-zero if any control on `/researcher/` shows a storage-unit number. * - * It drives the researcher page the way a person would — reach the Study step, switch on all seven + * It drives the researcher page the way a person would — reach the Study step, switch on all twelve * collectors, add an intervention, switch on delivery — so that every control the site has is mounted, and * then reads what is actually on screen. It does this in both locales, and it does it again after * clicking every preset chip on every control, so the assertions cover the values the page itself @@ -156,7 +156,9 @@ function ceilings(m, locale) { [m.unit.hours, 24], // the humanisers roll into days here [day, 366], // a year is the longest study `BOUNDS.durationHours` allows [oneDay, 366], - [m.unit.metres, BOUNDS.minimumDisplacementMeters[1]], // the schema's own ceiling + [m.unit.metres, BOUNDS.minimumDisplacementMillimeters[1] / 1_000], + [m.unit.millimetres, BOUNDS.changeThresholdMillimeters[1]], + [m.unit.lux, BOUNDS.changeThresholdMillilux[1] / 1_000], // `binaryBytes` moves to the next prefix at 1024, so no prefix ever carries more. ['B', 1_024], ['KiB', 1_024], diff --git a/web/package.json b/web/package.json index f02fbc7..d5e3a84 100644 --- a/web/package.json +++ b/web/package.json @@ -16,6 +16,7 @@ "@sveltejs/adapter-static": "^3.0.10", "@sveltejs/kit": "^2.70.2", "@sveltejs/vite-plugin-svelte": "^6.2.1", + "@types/qrcode": "1.5.6", "playwright": "^1.62.1", "svelte": "^5.56.8", "svelte-check": "^4.1.5", @@ -26,6 +27,12 @@ "dependencies": { "@noble/curves": "^2.2.0", "@noble/ed25519": "^3.1.0", - "@noble/hashes": "^2.0.1" + "@noble/hashes": "^2.0.1", + "qrcode": "1.5.4" + }, + "pnpm": { + "overrides": { + "cookie": "0.7.2" + } } } diff --git a/web/pnpm-lock.yaml b/web/pnpm-lock.yaml index 1ac3d3d..f8651b0 100644 --- a/web/pnpm-lock.yaml +++ b/web/pnpm-lock.yaml @@ -4,6 +4,9 @@ settings: autoInstallPeers: true excludeLinksFromLockfile: false +overrides: + cookie: 0.7.2 + importers: .: @@ -17,16 +20,22 @@ importers: '@noble/hashes': specifier: ^2.0.1 version: 2.2.0 + qrcode: + specifier: 1.5.4 + version: 1.5.4 devDependencies: '@sveltejs/adapter-static': specifier: ^3.0.10 - version: 3.0.10(@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6)) + version: 3.0.10(@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6(@types/node@26.1.2))) '@sveltejs/kit': specifier: ^2.70.2 - version: 2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6) + version: 2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6(@types/node@26.1.2)) '@sveltejs/vite-plugin-svelte': specifier: ^6.2.1 - version: 6.2.4(svelte@5.56.8)(vite@7.3.6) + version: 6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)) + '@types/qrcode': + specifier: 1.5.6 + version: 1.5.6 playwright: specifier: ^1.62.1 version: 1.62.1 @@ -41,10 +50,10 @@ importers: version: 5.9.3 vite: specifier: ^7.1.5 - version: 7.3.6 + version: 7.3.6(@types/node@26.1.2) vitest: specifier: ^3.0.5 - version: 3.2.7 + version: 3.2.7(@types/node@26.1.2) packages: @@ -425,6 +434,12 @@ packages: '@types/estree@1.0.9': resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + '@types/node@26.1.2': + resolution: {integrity: sha512-Vu4a5UFA9rIIFJ7rB/Vaafh9lrCQszopTCx6KjFboXTGQbPNasehVR5TEiithSDGyd1DEiUByggTZsg8jukeIg==} + + '@types/qrcode@1.5.6': + resolution: {integrity: sha512-te7NQcV2BOvdj2b1hCAHzAoMNuj65kNBMz0KBaxM6c3VGBOhU0dURQKOtH8CFNI/dsKkwlv32p26qYQTWoB5bw==} + '@types/trusted-types@2.0.7': resolution: {integrity: sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==} @@ -462,6 +477,14 @@ packages: engines: {node: '>=0.4.0'} hasBin: true + ansi-regex@5.0.1: + resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==} + engines: {node: '>=8'} + + ansi-styles@4.3.0: + resolution: {integrity: sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==} + engines: {node: '>=8'} + aria-query@5.3.1: resolution: {integrity: sha512-Z/ZeOgVl7bcSYZ/u/rh0fOpvEpq//LZmdbkXyc7syVzjPAhfOa9ebsdTSjEBDU4vs5nC98Kfduj1uFo0qyET3g==} engines: {node: '>= 0.4'} @@ -478,6 +501,10 @@ packages: resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==} engines: {node: '>=8'} + camelcase@5.3.1: + resolution: {integrity: sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg==} + engines: {node: '>=6'} + chai@5.3.3: resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==} engines: {node: '>=18'} @@ -490,12 +517,22 @@ packages: resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} engines: {node: '>= 14.16.0'} + cliui@6.0.0: + resolution: {integrity: sha512-t6wbgtoCXvAzst7QgXxJYqPt0usEfbgQdftEPbLL/cvv6HPE5VgvqCuAIDR0NgU52ds6rFwqrgakNLrHEjCbrQ==} + clsx@2.1.1: resolution: {integrity: sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==} engines: {node: '>=6'} - cookie@0.6.0: - resolution: {integrity: sha512-U71cyTamuh1CRNCfpGY6to28lxvNwPG4Guz/EVjgf3Jmzv0vlDp1atT9eS5dDjMYHucpHbWns6Lwf3BKz6svdw==} + color-convert@2.0.1: + resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==} + engines: {node: '>=7.0.0'} + + color-name@1.1.4: + resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} + + cookie@0.7.2: + resolution: {integrity: sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==} engines: {node: '>= 0.6'} debug@4.4.3: @@ -507,6 +544,10 @@ packages: supports-color: optional: true + decamelize@1.2.0: + resolution: {integrity: sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA==} + engines: {node: '>=0.10.0'} + deep-eql@5.0.2: resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==} engines: {node: '>=6'} @@ -518,6 +559,12 @@ packages: devalue@5.9.0: resolution: {integrity: sha512-RWrqdArjvPbsATEhOPUo6Wndc/iWnkWKlhIrdlF3zMMYo/c3CVtoaVAyLtWxz5h8nSlkHzxnzV2uLydPXmtF+A==} + dijkstrajs@1.0.3: + resolution: {integrity: sha512-qiSlmBq9+BCdCA/L46dw8Uy93mloxsPSbwnm5yrKn2vMPiy8KyAskTF6zuV/j5BMsmOGZDPs7KjU+mjb670kfA==} + + emoji-regex@8.0.0: + resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} + es-module-lexer@1.7.0: resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==} @@ -553,6 +600,10 @@ packages: picomatch: optional: true + find-up@4.1.0: + resolution: {integrity: sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw==} + engines: {node: '>=8'} + fsevents@2.3.2: resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==} engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} @@ -563,6 +614,14 @@ packages: engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} os: [darwin] + get-caller-file@2.0.5: + resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==} + engines: {node: 6.* || 8.* || >= 10.*} + + is-fullwidth-code-point@3.0.0: + resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==} + engines: {node: '>=8'} + is-reference@3.0.3: resolution: {integrity: sha512-ixkJoqQvAP88E6wLydLGGqCJsrFUnqoH6HnaczB8XmDH1oaWU+xxdptvikTgaEhtZ53Ky6YXiBuUI2WXLMCwjw==} @@ -576,6 +635,10 @@ packages: locate-character@3.0.0: resolution: {integrity: sha512-SW13ws7BjaeJ6p7Q6CO2nchbYEc3X3J6WrmTTDto7yMPqVSZTUyY5Tjbid+Ab8gLnATtygYtiDIJGQRRn2ZOiA==} + locate-path@5.0.0: + resolution: {integrity: sha512-t7hw9pI+WvuwNJXwk5zVHpyhIqzg2qTlklJOf0mVxGSbe3Fp2VieZcduNYjaLDoy6p9uGpQEGWG87WpMKlNq8g==} + engines: {node: '>=8'} + loupe@3.2.1: resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==} @@ -602,6 +665,22 @@ packages: resolution: {integrity: sha512-4a+OsYv9UktOJKE+l1A4OufDgdRF9PifWj+tJnHURo/P+WOxpG4GzUFL9qCalmWauao6ogiG+QvnCovwPoyAWA==} engines: {node: '>=12.20.0'} + p-limit@2.3.0: + resolution: {integrity: sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w==} + engines: {node: '>=6'} + + p-locate@4.1.0: + resolution: {integrity: sha512-R79ZZ/0wAxKGu3oYMlz8jy/kbhsNrS7SKZ7PxEHBgJ5+F2mtFW2fK2cOtBh1cHYkQsbzFV7I+EoRKe6Yt0oK7A==} + engines: {node: '>=8'} + + p-try@2.2.0: + resolution: {integrity: sha512-R4nPAVTAU0B9D35/Gk3uJf/7XYbQcyohSKdvAxIRSNghFl4e71hVoGnBNQz9cWaXxO2I10KTC+3jMdvvoKw6dQ==} + engines: {node: '>=6'} + + path-exists@4.0.0: + resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==} + engines: {node: '>=8'} + pathe@2.0.3: resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==} @@ -626,14 +705,30 @@ packages: engines: {node: '>=20'} hasBin: true + pngjs@5.0.0: + resolution: {integrity: sha512-40QW5YalBNfQo5yRYmiw7Yz6TKKVr3h6970B2YE+3fQpsWcrbj1PzJgxeJ19DRQjhMbKPIuMY8rFaXc8moolVw==} + engines: {node: '>=10.13.0'} + postcss@8.5.25: resolution: {integrity: sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw==} engines: {node: ^10 || ^12 || >=14} + qrcode@1.5.4: + resolution: {integrity: sha512-1ca71Zgiu6ORjHqFBDpnSMTR2ReToX4l1Au1VFLyVeBTFavzQnv5JxMFr3ukHVKpSrSA2MCk0lNJSykjUfz7Zg==} + engines: {node: '>=10.13.0'} + hasBin: true + readdirp@4.1.2: resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} engines: {node: '>= 14.18.0'} + require-directory@2.1.1: + resolution: {integrity: sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==} + engines: {node: '>=0.10.0'} + + require-main-filename@2.0.0: + resolution: {integrity: sha512-NKN5kMDylKuldxYLSUfrbo5Tuzh4hd+2E8NPPX02mZtn1VuREQToYe/ZdlJy+J3uCpfaiGF05e7B8W0iXbQHmg==} + rollup@4.62.4: resolution: {integrity: sha512-RXOqwaPsBGjMNMa4sQjDjHieHEZDFoj/Rdr46l2MU5DfEs16wHJPC2RPTPHWhNl+M3aI472LLqFkFKut4SblOg==} engines: {node: '>=18.0.0', npm: '>=8.0.0'} @@ -643,6 +738,9 @@ packages: resolution: {integrity: sha512-xal3CZX1Xlo/k4ApwCFrHVACi9fBqJ7V+mwhBsuf/1IOKbBy098Fex+Wa/5QMubw09pSZ/u8EY8PWgevJsXp1A==} engines: {node: '>=6'} + set-blocking@2.0.0: + resolution: {integrity: sha512-KiKBS8AnWGEyLzofFfmvKwpdPzqiy16LvQfK3yv/fVH7Bj13/wl3JSR1J+rfgRE9q7xUJK4qvgS8raSOeLUehw==} + set-cookie-parser@3.1.2: resolution: {integrity: sha512-5/r/lTwbJ3zQ+qwdUFZYeRNqda7P5HD8zQKqlSjdGt1/S0cjLAphHusj4Y58ahDtWn/g32xrIS58/ikOvwl0Lw==} @@ -663,6 +761,14 @@ packages: std-env@3.10.0: resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==} + string-width@4.2.3: + resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} + engines: {node: '>=8'} + + strip-ansi@6.0.1: + resolution: {integrity: sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==} + engines: {node: '>=8'} + strip-literal@3.1.0: resolution: {integrity: sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==} @@ -709,6 +815,9 @@ packages: engines: {node: '>=14.17'} hasBin: true + undici-types@8.3.0: + resolution: {integrity: sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==} + vite-node@3.2.4: resolution: {integrity: sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==} engines: {node: ^18.0.0 || ^20.0.0 || >=22.0.0} @@ -790,11 +899,29 @@ packages: jsdom: optional: true + which-module@2.0.1: + resolution: {integrity: sha512-iBdZ57RDvnOR9AGBhML2vFZf7h8vmBjhoaZqODJBFWHVtKkDmKuHai3cx5PgVMrX5YDNp27AofYbAwctSS+vhQ==} + why-is-node-running@2.3.0: resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==} engines: {node: '>=8'} hasBin: true + wrap-ansi@6.2.0: + resolution: {integrity: sha512-r6lPcBGxZXlIcymEu7InxDMhdW0KDxpLgoFLcguasxCaJ/SOIZwINatK9KY/tf+ZrlywOKU0UDj3ATXUBfxJXA==} + engines: {node: '>=8'} + + y18n@4.0.3: + resolution: {integrity: sha512-JKhqTOwSrqNA1NY5lSztJ1GrBiUodLMmIZuLiDaMRJ+itFd+ABVE8XBjOvIWL+rSqNDC74LCSFmlb/U4UZ4hJQ==} + + yargs-parser@18.1.3: + resolution: {integrity: sha512-o50j0JeToy/4K6OZcaQmW6lyXXKhq7csREXcDwk2omFPJEwUNOVtJKvmDr9EI1fAJZUyZcRF7kxGBWmRXudrCQ==} + engines: {node: '>=6'} + + yargs@15.4.1: + resolution: {integrity: sha512-aePbxDmcYW++PaqBsJ+HYUFwCdv4LVvdnhBy78E57PIor8/OVvhMrADFFEDh8DHDFRv/O9i3lPhsENjO7QX0+A==} + engines: {node: '>=8'} + zimmerframe@1.1.4: resolution: {integrity: sha512-B58NGBEoc8Y9MWWCQGl/gq9xBCe4IiKM0a2x7GZdQKOW5Exr8S1W24J6OgM1njK8xCRGvAJIL/MxXHf6SkmQKQ==} @@ -991,18 +1118,18 @@ snapshots: dependencies: acorn: 8.18.0 - '@sveltejs/adapter-static@3.0.10(@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6))': + '@sveltejs/adapter-static@3.0.10(@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6(@types/node@26.1.2)))': dependencies: - '@sveltejs/kit': 2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6) + '@sveltejs/kit': 2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6(@types/node@26.1.2)) - '@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6)': + '@sveltejs/kit@2.70.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(typescript@5.9.3)(vite@7.3.6(@types/node@26.1.2))': dependencies: '@standard-schema/spec': 1.1.0 '@sveltejs/acorn-typescript': 1.0.11(acorn@8.18.0) - '@sveltejs/vite-plugin-svelte': 6.2.4(svelte@5.56.8)(vite@7.3.6) + '@sveltejs/vite-plugin-svelte': 6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)) '@types/cookie': 0.6.0 acorn: 8.18.0 - cookie: 0.6.0 + cookie: 0.7.2 devalue: 5.9.0 esm-env: 1.2.2 kleur: 4.1.5 @@ -1011,28 +1138,28 @@ snapshots: set-cookie-parser: 3.1.2 sirv: 3.0.2 svelte: 5.56.8 - vite: 7.3.6 + vite: 7.3.6(@types/node@26.1.2) optionalDependencies: typescript: 5.9.3 '@sveltejs/load-config@0.2.1': {} - '@sveltejs/vite-plugin-svelte-inspector@5.0.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(vite@7.3.6)': + '@sveltejs/vite-plugin-svelte-inspector@5.0.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2))': dependencies: - '@sveltejs/vite-plugin-svelte': 6.2.4(svelte@5.56.8)(vite@7.3.6) + '@sveltejs/vite-plugin-svelte': 6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)) obug: 2.1.4 svelte: 5.56.8 - vite: 7.3.6 + vite: 7.3.6(@types/node@26.1.2) - '@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6)': + '@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2))': dependencies: - '@sveltejs/vite-plugin-svelte-inspector': 5.0.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6))(svelte@5.56.8)(vite@7.3.6) + '@sveltejs/vite-plugin-svelte-inspector': 5.0.2(@sveltejs/vite-plugin-svelte@6.2.4(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)))(svelte@5.56.8)(vite@7.3.6(@types/node@26.1.2)) deepmerge: 4.3.1 magic-string: 0.30.21 obug: 2.1.4 svelte: 5.56.8 - vite: 7.3.6 - vitefu: 1.1.3(vite@7.3.6) + vite: 7.3.6(@types/node@26.1.2) + vitefu: 1.1.3(vite@7.3.6(@types/node@26.1.2)) '@types/chai@5.2.3': dependencies: @@ -1045,6 +1172,14 @@ snapshots: '@types/estree@1.0.9': {} + '@types/node@26.1.2': + dependencies: + undici-types: 8.3.0 + + '@types/qrcode@1.5.6': + dependencies: + '@types/node': 26.1.2 + '@types/trusted-types@2.0.7': {} '@vitest/expect@3.2.7': @@ -1055,13 +1190,13 @@ snapshots: chai: 5.3.3 tinyrainbow: 2.0.0 - '@vitest/mocker@3.2.7(vite@7.3.6)': + '@vitest/mocker@3.2.7(vite@7.3.6(@types/node@26.1.2))': dependencies: '@vitest/spy': 3.2.7 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 7.3.6 + vite: 7.3.6(@types/node@26.1.2) '@vitest/pretty-format@3.2.7': dependencies: @@ -1091,6 +1226,12 @@ snapshots: acorn@8.18.0: {} + ansi-regex@5.0.1: {} + + ansi-styles@4.3.0: + dependencies: + color-convert: 2.0.1 + aria-query@5.3.1: {} assertion-error@2.0.1: {} @@ -1099,6 +1240,8 @@ snapshots: cac@6.7.14: {} + camelcase@5.3.1: {} + chai@5.3.3: dependencies: assertion-error: 2.0.1 @@ -1113,20 +1256,38 @@ snapshots: dependencies: readdirp: 4.1.2 + cliui@6.0.0: + dependencies: + string-width: 4.2.3 + strip-ansi: 6.0.1 + wrap-ansi: 6.2.0 + clsx@2.1.1: {} - cookie@0.6.0: {} + color-convert@2.0.1: + dependencies: + color-name: 1.1.4 + + color-name@1.1.4: {} + + cookie@0.7.2: {} debug@4.4.3: dependencies: ms: 2.1.3 + decamelize@1.2.0: {} + deep-eql@5.0.2: {} deepmerge@4.3.1: {} devalue@5.9.0: {} + dijkstrajs@1.0.3: {} + + emoji-regex@8.0.0: {} + es-module-lexer@1.7.0: {} esbuild@0.28.1: @@ -1174,12 +1335,21 @@ snapshots: optionalDependencies: picomatch: 4.0.5 + find-up@4.1.0: + dependencies: + locate-path: 5.0.0 + path-exists: 4.0.0 + fsevents@2.3.2: optional: true fsevents@2.3.3: optional: true + get-caller-file@2.0.5: {} + + is-fullwidth-code-point@3.0.0: {} + is-reference@3.0.3: dependencies: '@types/estree': 1.0.9 @@ -1190,6 +1360,10 @@ snapshots: locate-character@3.0.0: {} + locate-path@5.0.0: + dependencies: + p-locate: 4.1.0 + loupe@3.2.1: {} magic-string@0.30.21: @@ -1206,6 +1380,18 @@ snapshots: obug@2.1.4: {} + p-limit@2.3.0: + dependencies: + p-try: 2.2.0 + + p-locate@4.1.0: + dependencies: + p-limit: 2.3.0 + + p-try@2.2.0: {} + + path-exists@4.0.0: {} + pathe@2.0.3: {} pathval@2.0.1: {} @@ -1222,14 +1408,26 @@ snapshots: optionalDependencies: fsevents: 2.3.2 + pngjs@5.0.0: {} + postcss@8.5.25: dependencies: nanoid: 3.3.16 picocolors: 1.1.1 source-map-js: 1.2.1 + qrcode@1.5.4: + dependencies: + dijkstrajs: 1.0.3 + pngjs: 5.0.0 + yargs: 15.4.1 + readdirp@4.1.2: {} + require-directory@2.1.1: {} + + require-main-filename@2.0.0: {} + rollup@4.62.4: dependencies: '@types/estree': 1.0.9 @@ -1266,6 +1464,8 @@ snapshots: dependencies: mri: 1.2.0 + set-blocking@2.0.0: {} + set-cookie-parser@3.1.2: {} siginfo@2.0.0: {} @@ -1282,6 +1482,16 @@ snapshots: std-env@3.10.0: {} + string-width@4.2.3: + dependencies: + emoji-regex: 8.0.0 + is-fullwidth-code-point: 3.0.0 + strip-ansi: 6.0.1 + + strip-ansi@6.0.1: + dependencies: + ansi-regex: 5.0.1 + strip-literal@3.1.0: dependencies: js-tokens: 9.0.1 @@ -1339,13 +1549,15 @@ snapshots: typescript@5.9.3: {} - vite-node@3.2.4: + undici-types@8.3.0: {} + + vite-node@3.2.4(@types/node@26.1.2): dependencies: cac: 6.7.14 debug: 4.4.3 es-module-lexer: 1.7.0 pathe: 2.0.3 - vite: 7.3.6 + vite: 7.3.6(@types/node@26.1.2) transitivePeerDependencies: - '@types/node' - jiti @@ -1360,7 +1572,7 @@ snapshots: - tsx - yaml - vite@7.3.6: + vite@7.3.6(@types/node@26.1.2): dependencies: esbuild: 0.28.1 fdir: 6.5.0(picomatch@4.0.5) @@ -1369,17 +1581,18 @@ snapshots: rollup: 4.62.4 tinyglobby: 0.2.17 optionalDependencies: + '@types/node': 26.1.2 fsevents: 2.3.3 - vitefu@1.1.3(vite@7.3.6): + vitefu@1.1.3(vite@7.3.6(@types/node@26.1.2)): optionalDependencies: - vite: 7.3.6 + vite: 7.3.6(@types/node@26.1.2) - vitest@3.2.7: + vitest@3.2.7(@types/node@26.1.2): dependencies: '@types/chai': 5.2.3 '@vitest/expect': 3.2.7 - '@vitest/mocker': 3.2.7(vite@7.3.6) + '@vitest/mocker': 3.2.7(vite@7.3.6(@types/node@26.1.2)) '@vitest/pretty-format': 3.2.7 '@vitest/runner': 3.2.7 '@vitest/snapshot': 3.2.7 @@ -1397,9 +1610,11 @@ snapshots: tinyglobby: 0.2.17 tinypool: 1.1.1 tinyrainbow: 2.0.0 - vite: 7.3.6 - vite-node: 3.2.4 + vite: 7.3.6(@types/node@26.1.2) + vite-node: 3.2.4(@types/node@26.1.2) why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 26.1.2 transitivePeerDependencies: - jiti - less @@ -1414,9 +1629,38 @@ snapshots: - tsx - yaml + which-module@2.0.1: {} + why-is-node-running@2.3.0: dependencies: siginfo: 2.0.0 stackback: 0.0.2 + wrap-ansi@6.2.0: + dependencies: + ansi-styles: 4.3.0 + string-width: 4.2.3 + strip-ansi: 6.0.1 + + y18n@4.0.3: {} + + yargs-parser@18.1.3: + dependencies: + camelcase: 5.3.1 + decamelize: 1.2.0 + + yargs@15.4.1: + dependencies: + cliui: 6.0.0 + decamelize: 1.2.0 + find-up: 4.1.0 + get-caller-file: 2.0.5 + require-directory: 2.1.1 + require-main-filename: 2.0.0 + set-blocking: 2.0.0 + string-width: 4.2.3 + which-module: 2.0.1 + y18n: 4.0.3 + yargs-parser: 18.1.3 + zimmerframe@1.1.4: {} diff --git a/web/src/lib/adc/bundle.ts b/web/src/lib/adc/bundle.ts index f43115a..0f0d684 100644 --- a/web/src/lib/adc/bundle.ts +++ b/web/src/lib/adc/bundle.ts @@ -1,77 +1,99 @@ -/** - * Opening a `.adcexp` — the file a phone hands back at the end of a study. +/** Authenticated Protocol v1 `.adcexp` reader. * - * This is the only reader on the site: everything else in `lib/adc` writes files for the app and - * the CLI to read. `researcher-tools decrypt` is the other implementation of this function, and it - * takes the same three inputs for the same reason — the bundle, the study configuration, and the - * export private key. The configuration is not a convenience. Its `experiment_id`, - * `configuration_id` and `export.researcher_key_id` are the AAD the body was sealed under and the - * `info` the content key was wrapped under, and none of the three is anywhere in the file's - * cleartext except the last. A personalised study issues one configuration per participant, so the - * tag only verifies against *that* participant's file. - * - * The CLI stages its output and writes nothing until the tag verifies. The browser gets the same - * guarantee for free and takes it: the body is one AES-GCM stream with one tag at the end — - * `DECRYPT_CHUNK_BYTES` on the Kotlin side is a read buffer, not a frame size — so a single - * `subtle.decrypt` covers the whole document, and WebCrypto verifies the tag before it resolves. - * Nothing here returns a document that has not been authenticated. - * - * Every way this can fail returns a name rather than throwing one. A researcher holding last - * month's key and this month's bundle has made a mistake with a precise fix, and "decryption - * failed" is the one answer that does not tell them which of the three files to change. + * The browser keeps this as a convenience reader for bounded bundles. The offline Python pipeline + * remains the analysis path for large studies. No plaintext value is returned until both AEAD + * layers, canonical JSON, embedded configuration provenance, identities, and actual event range + * have all verified. */ -import { hpkePublicKey, readHpkePrivateKeyset } from './tink'; -import type { StudyConfiguration, TinkKeyset } from './types'; +import { + canonicalBytes, + canonicalConfigurationBytes, + canonicalize, + canonicalizeConfiguration, + isCanonicalDecimal, + parseCanonicalJson +} from './canonical'; +import { decodeBase64Url, encodeBase64Url, verify } from './crypto'; +import { ID_PATTERN, type StudyConfiguration } from './types'; import { x25519 } from '@noble/curves/ed25519.js'; import { expand, extract } from '@noble/hashes/hkdf.js'; import { sha256 } from '@noble/hashes/sha2.js'; - -/* ---- the container ------------------------------------------------------------------------- */ +import collectorCatalog from '../../../../protocol/v1/collector-catalog.json'; const MAGIC = 'ADCEXP01'; - -/** magic 8 | uint16 keyIdLen | int32 wrappedKeyLen | nonce 12. Everything after it is variable. */ -const HEADER_BYTES = 26; - +const FIXED_HEADER_BYTES = 70; +const BUNDLE_ID_BYTES = 16; +const DIGEST_BYTES = 32; const NONCE_BYTES = 12; +const WRAPPED_KEY_BYTES = 80; const TAG_BYTES = 16; const CONTENT_KEY_BYTES = 32; - -/** Tink's `TINK` output prefix: `0x01` then the key id, big-endian. */ -const PREFIX_BYTES = 5; -const ENCAPSULATED_BYTES = 32; - -/** `ResearchExport.decrypt`'s own bounds, so a length that lies is refused rather than allocated. */ const MINIMUM_KEY_ID_BYTES = 3; const MAXIMUM_KEY_ID_BYTES = 64; -const MINIMUM_WRAPPED_BYTES = 32; -const MAXIMUM_WRAPPED_BYTES = 16_384; - -/** - * WebCrypto has no streaming AEAD, so the ciphertext and the plaintext are both resident at once - * and a bundle is bounded by what a tab can hold rather than by what a phone can write — - * `storage.maximum_local_bytes` reaches 8 GiB. Refusing at a stated size is a sentence a researcher - * can act on; a `RangeError` out of `subtle.decrypt`, or a tab that stops responding, is not. - */ -export const MAXIMUM_BUNDLE_BYTES = 268_435_456; - -/* ---- what comes out ------------------------------------------------------------------------ */ +const MAXIMUM_INT64 = 9_223_372_036_854_775_807n; +const PARTICIPANT_INSTANCE_ID = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/; +const CANONICAL_SIGNED_INTEGER = /^(?:0|-?[1-9][0-9]*)$/; +const DECIMAL_FLOAT = /^[+-]?(?:(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?)$/; +const UTF8 = new TextEncoder(); +const FATAL_UTF8 = new TextDecoder('utf-8', { fatal: true }); + +type CatalogField = { + type: 'boolean' | 'decimal_string' | 'enum' | 'float32' | 'float64' | 'int32' | 'json_string' | 'string'; + required: boolean; + enum?: string[]; + minimum?: number; + maximum?: number; + maximum_length?: number; +}; +type CatalogPayload = { fields: Record; types: string[] }; +type CatalogCollector = { + id: string; + maximum_encoded_event_bytes: number; + payload_schema_version: number; + payloads: CatalogPayload[]; +}; +const EVENT_CONTRACTS = new Map( + (collectorCatalog.collectors as CatalogCollector[]).map((collector) => [collector.id, collector]) +); +const TRANSITION_DESTINATIONS: Record = { + ACCESS_PREFLIGHT_PASSED: 'READY', + CONFIGURATION_SIGNATURE_VERIFIED: 'CONFIG_VERIFIED', + CONSENT_ACCEPTED: 'ACCESS_SETUP', + CONSENT_REVIEW_OPENED: 'CONSENT_PENDING', + PARTICIPANT_FINISHED_EARLY: 'COMPLETED', + PARTICIPANT_PAUSED: 'PAUSED', + PARTICIPANT_RESUMED: 'RUNNING', + PARTICIPANT_STARTED: 'RUNNING', + PARTICIPANT_WITHDREW: 'WITHDRAWN', + STORAGE_FAILURE: 'PAUSED', + STUDY_DURATION_ELAPSED: 'COMPLETED' +}; +const STATE_TRANSITIONS: Record = { + ACCESS_SETUP: ['READY', 'WITHDRAWN'], + COMPLETED: ['WITHDRAWN'], + CONFIG_VERIFIED: ['CONSENT_PENDING', 'WITHDRAWN'], + CONSENT_PENDING: ['ACCESS_SETUP', 'WITHDRAWN'], + IMPORTED: ['CONFIG_VERIFIED', 'WITHDRAWN'], + PAUSED: ['RUNNING', 'COMPLETED', 'WITHDRAWN'], + READY: ['RUNNING', 'WITHDRAWN'], + RUNNING: ['PAUSED', 'COMPLETED', 'WITHDRAWN'], + WITHDRAWN: [] +}; + +// Browser-only memory policy, not a Protocol wire limit. Automatic uploads happen to share this +// bound; larger manual exports remain valid and belong in the streaming adc-analysis CLI. +const MAXIMUM_BROWSER_PREVIEW_BYTES = 33_554_432; +export const BUNDLE_FORMAT = 'research-bundle-v1'; -/** - * The phone's clock, its uptime, and which boot the uptime is measured from. `elapsed_realtime_nanos` - * passes `Number.MAX_SAFE_INTEGER` after about 104 days of uptime, so it is carried and shown and - * never used in arithmetic. - */ export interface ResearchTime { - wall_time_utc_millis: number; - elapsed_realtime_nanos: number; + wall_time_utc_millis: string; + monotonic_time_nanos: string; boot_session_id: string; } -/** `fields` is string→string in every event, including the ones whose values look like numbers. */ export interface ResearchEvent { - sequence_number: number; + sequence_number: string; collector_id: string; payload_schema_version: number; observed_time: ResearchTime; @@ -90,53 +112,38 @@ export interface ResearchExperiment { experiment_id: string; configuration_id: string; participant_instance_id: string; - /** The key is absent in an anonymous study, so absent and null are the same answer here. */ assigned_participant_id: string | null; state: string; - next_sequence_number: number; + retained_from_sequence: string; + uploaded_through_sequence: string; + durable_through_sequence: string; + next_sequence_number: string; + first_sequence_number: string; + last_sequence_number: string; + event_count: string; transitions: ResearchTransition[]; events: ResearchEvent[]; - /** - * The window this file carries, which is not always the whole study: a scheduled upload sends a - * slice, and `next_sequence_number - 1` is what the device has recorded in its lifetime. - */ - first_sequence_number: number; - last_sequence_number: number; } -export const BUNDLE_FORMAT = 'research-bundle-v1'; - export interface ResearchDocument { - format: string; - exported_at_utc_millis: number; - /** - * The study's own canonical JSON, verbatim. Nothing here reads it: the tag verified under a - * context derived from the configuration the caller supplied, which is already the proof that the - * two are the same study. - */ - configuration: unknown; + format: typeof BUNDLE_FORMAT; + bundle_id: string; + bundle_kind: 'manual_export' | 'automatic_upload'; + configuration_sha256: string; + producer: { platform: 'android'; client_version: string }; + exported_at_utc_millis: string; + configuration: StudyConfiguration; + configuration_signature: { signer_key_id: string; signature: string }; experiment: ResearchExperiment; } export interface ResearchBundle { - /** `export.researcher_key_id`, as the file names it. */ keyId: string; document: ResearchDocument; - /** The decrypted JSON exactly as the phone wrote it, which is what `--output` would contain. */ text: string; bytes: number; } -/** - * Why nothing opened, in the order the checks run. Each one names a different file to change. - * - * The last two are a real distinction rather than two words for one failure, and which one comes - * back says where the mismatch is. The context binds the wrap *and* the body, and the wrap is - * opened first — so a configuration whose `experiment_id` or `configuration_id` is not the one this - * file was sealed under fails as `unwrap_failed`, which is the personalised-study case of holding - * another participant's configuration. `tag_failed` can only happen once the context and the key - * have both already proved correct, so it means the bytes changed after the phone wrote them. - */ export type BundleFailure = | 'not_a_bundle' | 'too_large' @@ -151,82 +158,78 @@ export type BundleResult = | { ok: false; failure: BundleFailure }; const failed = (failure: BundleFailure): BundleResult => ({ ok: false, failure }); - -/** - * A `Uint8Array` is not a `BufferSource` to TypeScript, because a view could sit over a - * `SharedArrayBuffer` and WebCrypto refuses those. None of these ever does — every array here comes - * from a `File`, a `TextEncoder`, or `@noble` — so the narrowing is stated once instead of at each - * of the four calls that would otherwise carry the same cast. - */ const source = (bytes: Uint8Array): BufferSource => bytes as unknown as BufferSource; -/* ---- the read ------------------------------------------------------------------------------ */ +export function configurationDigest(configuration: StudyConfiguration): Uint8Array { + return sha256(canonicalConfigurationBytes(configuration)); +} -/** - * The context both layers are bound to: the wrap's RFC 9180 `info`, and the body's AES-GCM AAD. - * The same bytes in two different roles, which is the one thing about this format that is easy to - * get backwards — a reader that passes it as the wrap's *associated data* instead fails with an - * `OperationError` indistinguishable from the wrong key. - */ -export function bundleContext(configuration: StudyConfiguration): Uint8Array { - return utf8( - `${BUNDLE_FORMAT}:${configuration.experiment_id}:${configuration.configuration_id}:` + - configuration.export.researcher_key_id - ); +export function bundleContext( + bundleId: string, + configurationSha256: string, + researcherKeyId: string +): Uint8Array { + return canonicalBytes({ + bundle_format: BUNDLE_FORMAT, + bundle_id: bundleId, + configuration_sha256: configurationSha256, + researcher_key_id: researcherKeyId + }); } export async function openBundle( bytes: Uint8Array, configuration: StudyConfiguration, - privateKeyset: TinkKeyset + privateKey: string ): Promise { - if (bytes.length > MAXIMUM_BUNDLE_BYTES) return failed('too_large'); - if (bytes.length < HEADER_BYTES) return failed('not_a_bundle'); + if (bytes.length > MAXIMUM_BROWSER_PREVIEW_BYTES) return failed('too_large'); + if (bytes.length < FIXED_HEADER_BYTES + MINIMUM_KEY_ID_BYTES + WRAPPED_KEY_BYTES + TAG_BYTES) { + return failed('not_a_bundle'); + } for (let index = 0; index < MAGIC.length; index += 1) { if (bytes[index] !== MAGIC.charCodeAt(index)) return failed('not_a_bundle'); } const header = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const keyIdLength = header.getUint16(8); - // Signed, as the Java writer wrote it, and range-checked before it is used as a length: read - // unsigned, a hostile `0xffffffff` becomes four gigabytes of subarray. - const wrappedLength = header.getInt32(10); + const keyIdLength = header.getUint16(56); if (keyIdLength < MINIMUM_KEY_ID_BYTES || keyIdLength > MAXIMUM_KEY_ID_BYTES) { return failed('not_a_bundle'); } - if (wrappedLength < MINIMUM_WRAPPED_BYTES || wrappedLength > MAXIMUM_WRAPPED_BYTES) { + const keyIdEnd = FIXED_HEADER_BYTES + keyIdLength; + const wrappedEnd = keyIdEnd + WRAPPED_KEY_BYTES; + if (bytes.length <= wrappedEnd + TAG_BYTES) return failed('not_a_bundle'); + + let keyId: string; + try { + keyId = FATAL_UTF8.decode(bytes.subarray(FIXED_HEADER_BYTES, keyIdEnd)); + } catch { return failed('not_a_bundle'); } - const bodyAt = HEADER_BYTES + keyIdLength + wrappedLength; - if (bytes.length <= bodyAt + TAG_BYTES) return failed('not_a_bundle'); + const bundleId = uuid(bytes.subarray(8, 8 + BUNDLE_ID_BYTES)); + if (!bundleId || !ID_PATTERN.test(keyId)) return failed('not_a_bundle'); + const digest = bytes.subarray(24, 24 + DIGEST_BYTES); + const digestHex = hex(digest); + const nonce = bytes.subarray(58, 58 + NONCE_BYTES); + const wrapped = bytes.subarray(keyIdEnd, wrappedEnd); + const body = bytes.subarray(wrappedEnd); - const nonce = bytes.subarray(14, 14 + NONCE_BYTES); - const keyId = new TextDecoder().decode(bytes.subarray(HEADER_BYTES, HEADER_BYTES + keyIdLength)); - const wrapped = bytes.subarray(HEADER_BYTES + keyIdLength, bodyAt); - const body = bytes.subarray(bodyAt); - - // Before any crypto, exactly as the CLI does it. Bundle and configuration from two different - // studies is the commonest mistake of the three, and this is the only check that can name it. if (keyId !== configuration.export.researcher_key_id) return failed('wrong_study'); + if (!same(digest, configurationDigest(configuration))) return failed('wrong_study'); + + let recipientPrivate: Uint8Array; + let recipientPublic: Uint8Array; + try { + recipientPrivate = decodeBase64Url(privateKey.trim(), 32); + recipientPublic = x25519.getPublicKey(recipientPrivate); + if (!same(recipientPublic, decodeBase64Url(configuration.export.hpke_public_key, 32))) { + return failed('wrong_key'); + } + } catch { + return failed('wrong_key'); + } - const recipient = readHpkePrivateKeyset(privateKeyset); - if (!recipient) return failed('wrong_key'); - // The 5-byte prefix names the key that sealed this. A mismatch here is last month's key file, and - // saying so costs one comparison; letting it through costs an HPKE failure with no name. - const prefix = new DataView(wrapped.buffer, wrapped.byteOffset, wrapped.byteLength); - if (wrapped[0] !== 1 || prefix.getUint32(1) !== recipient.keyId) return failed('wrong_key'); - // Deliberately stricter than `ResearchExport.decrypt`, which never looks at the configuration's - // keyset on the read path. `researcher_key_id` is derived from the public key, so a configuration - // naming a different key was already refused above; what is left is a configuration whose own two - // halves disagree — a key id from one keyset beside the bytes of another, which only a hand-edited - // or corrupted file has. Nothing this site or the CLI writes can reach here, and a study whose - // declared key cannot be the one that sealed the bundle is worth saying so rather than letting - // HPKE fail with no name. - const study = hpkePublicKey(configuration.export.tink_hpke_public_keyset); - if (!study || !same(study, x25519.getPublicKey(recipient.scalar))) return failed('wrong_key'); - - const context = bundleContext(configuration); - const contentKey = await unwrap(wrapped, recipient.publicKey, recipient.scalar, context); + const context = bundleContext(bundleId, digestHex, keyId); + const contentKey = await unwrap(wrapped, recipientPublic, recipientPrivate, context); if (!contentKey) return failed('unwrap_failed'); let plaintext: Uint8Array; @@ -245,63 +248,56 @@ export async function openBundle( return failed('tag_failed'); } - const text = new TextDecoder().decode(plaintext); - const document = readDocument(text); + let text: string; + let parsed: unknown; + try { + text = FATAL_UTF8.decode(plaintext); + parsed = parseCanonicalJson(plaintext); + } catch { + return failed('unreadable'); + } + const document = readDocument(parsed, configuration, bundleId, digestHex); return document ? { ok: true, bundle: { keyId, document, text, bytes: plaintext.length } } : failed('unreadable'); } -/* ---- RFC 9180, base mode, DHKEM(X25519, HKDF-SHA256) / HKDF-SHA256 / AES-256-GCM ------------- - * - * The one suite Tink's `HpkeCrypto.validateParameters` accepts and the one this site writes, so - * there is nothing to negotiate and no algorithm agility to implement. `mode` is 0 and the sequence - * number is 0, which makes the AEAD nonce `base_nonce` unchanged. - * ------------------------------------------------------------------------------------------- */ - +/* RFC 9180 base mode: DHKEM(X25519, HKDF-SHA256), HKDF-SHA256, AES-256-GCM. */ const KEM_ID = 0x0020; const KDF_ID = 0x0001; const AEAD_ID = 0x0002; - -const encoder = new TextEncoder(); -const utf8 = (text: string) => encoder.encode(text); const EMPTY = new Uint8Array(0); - const i2osp2 = (value: number) => Uint8Array.of((value >> 8) & 0xff, value & 0xff); - -const SUITE_KEM = concat(utf8('KEM'), i2osp2(KEM_ID)); -const SUITE_HPKE = concat(utf8('HPKE'), i2osp2(KEM_ID), i2osp2(KDF_ID), i2osp2(AEAD_ID)); -const VERSION = utf8('HPKE-v1'); +const SUITE_KEM = concat(UTF8.encode('KEM'), i2osp2(KEM_ID)); +const SUITE_HPKE = concat( + UTF8.encode('HPKE'), + i2osp2(KEM_ID), + i2osp2(KDF_ID), + i2osp2(AEAD_ID) +); +const VERSION = UTF8.encode('HPKE-v1'); const labeledExtract = (suite: Uint8Array, salt: Uint8Array, label: string, ikm: Uint8Array) => - extract(sha256, concat(VERSION, suite, utf8(label), ikm), salt); - + extract(sha256, concat(VERSION, suite, UTF8.encode(label), ikm), salt); const labeledExpand = ( suite: Uint8Array, prk: Uint8Array, label: string, info: Uint8Array, length: number -) => expand(sha256, prk, concat(i2osp2(length), VERSION, suite, utf8(label), info), length); +) => expand(sha256, prk, concat(i2osp2(length), VERSION, suite, UTF8.encode(label), info), length); -/** - * The content key out of `HybridEncrypt`'s output. `info` is the raw context — Tink passes - * `contextInfo` straight into the key schedule, and the output prefix is no part of it — while the - * sealed key's own associated data is empty. - */ async function unwrap( wrapped: Uint8Array, recipientPublic: Uint8Array, - scalar: Uint8Array, + recipientPrivate: Uint8Array, info: Uint8Array ): Promise { - const enc = wrapped.subarray(PREFIX_BYTES, PREFIX_BYTES + ENCAPSULATED_BYTES); - const sealed = wrapped.subarray(PREFIX_BYTES + ENCAPSULATED_BYTES); - if (enc.length !== ENCAPSULATED_BYTES || sealed.length <= TAG_BYTES) return null; + if (wrapped.length !== WRAPPED_KEY_BYTES) return null; + const enc = wrapped.subarray(0, 32); + const sealed = wrapped.subarray(32); try { - // Refuses a shared secret that is all zeroes, which is the low-order-point case RFC 9180 - // requires an implementation to reject. - const dh = x25519.getSharedSecret(scalar, enc); + const dh = x25519.getSharedSecret(recipientPrivate, enc); const eaePrk = labeledExtract(SUITE_KEM, EMPTY, 'eae_prk', dh); const shared = labeledExpand( SUITE_KEM, @@ -337,148 +333,346 @@ async function unwrap( } } -function concat(...parts: Uint8Array[]): Uint8Array { - const out = new Uint8Array(parts.reduce((total, part) => total + part.length, 0)); - let at = 0; - for (const part of parts) { - out.set(part, at); - at += part.length; +function readDocument( + parsed: unknown, + expectedConfiguration: StudyConfiguration, + bundleId: string, + configurationSha256: string +): ResearchDocument | null { + const root = exact(parsed, [ + 'bundle_id', + 'bundle_kind', + 'configuration', + 'configuration_sha256', + 'configuration_signature', + 'experiment', + 'exported_at_utc_millis', + 'format', + 'producer' + ]); + if (!root || root.format !== BUNDLE_FORMAT || root.bundle_id !== bundleId) return null; + if (root.bundle_kind !== 'manual_export' && root.bundle_kind !== 'automatic_upload') return null; + if (root.configuration_sha256 !== configurationSha256) return null; + if (!decimal(root.exported_at_utc_millis)) return null; + + const producer = exact(root.producer, ['client_version', 'platform']); + if (!producer || producer.platform !== 'android' || !positiveDecimal(producer.client_version)) { + return null; + } + if (BigInt(producer.client_version as string) < BigInt(expectedConfiguration.minimum_client_version)) { + return null; } - return out; -} - -function same(left: Uint8Array, right: Uint8Array): boolean { - return left.length === right.length && left.every((byte, index) => byte === right[index]); -} -/* ---- the document -------------------------------------------------------------------------- */ + const configuration = record(root.configuration); + if (!configuration || canonicalize(configuration) !== canonicalizeConfiguration(expectedConfiguration)) { + return null; + } + if (hex(sha256(canonicalBytes(configuration))) !== configurationSha256) return null; -/** - * Structural, and only as deep as anything reads. A summary that maps over `events` cannot be shown - * a `null` there and recover, and a document that fails this is one no version of this page wrote — - * so it is refused with a name rather than half-rendered. Unknown fields are carried: this reads a - * format the phone owns, and refusing a field somebody added would break every future bundle. - */ -function readDocument(text: string): ResearchDocument | null { - let parsed: unknown; + const provenance = exact(root.configuration_signature, ['signature', 'signer_key_id']); + if (!provenance || provenance.signer_key_id !== expectedConfiguration.signer.key_id) return null; + let signature: Uint8Array; try { - parsed = JSON.parse(text); + signature = decodeBase64Url(string(provenance.signature), 64); } catch { return null; } - const root = record(parsed); - if (!root || root.format !== BUNDLE_FORMAT) return null; - if (!isNumber(root.exported_at_utc_millis)) return null; - if (!record(root.configuration)) return null; + if (!verify(canonicalBytes(configuration), signature, expectedConfiguration.signer.public_key)) { + return null; + } + + const experiment = readExperiment(root.experiment, expectedConfiguration); + if (!experiment) return null; + if ( + root.bundle_kind === 'automatic_upload' && + (experiment.event_count === '0' || + BigInt(experiment.first_sequence_number) !== BigInt(experiment.uploaded_through_sequence) + 1n) + ) return null; + return { + format: BUNDLE_FORMAT, + bundle_id: bundleId, + bundle_kind: root.bundle_kind, + configuration_sha256: configurationSha256, + producer: { platform: 'android', client_version: producer.client_version as string }, + exported_at_utc_millis: root.exported_at_utc_millis as string, + configuration: expectedConfiguration, + configuration_signature: { + signer_key_id: provenance.signer_key_id as string, + signature: encodeBase64Url(signature) + }, + experiment + }; +} - const source = record(root.experiment); +function readExperiment(raw: unknown, configuration: StudyConfiguration): ResearchExperiment | null { + const source = exact(raw, [ + 'assigned_participant_id', + 'configuration_id', + 'durable_through_sequence', + 'event_count', + 'events', + 'experiment_id', + 'first_sequence_number', + 'last_sequence_number', + 'next_sequence_number', + 'participant_instance_id', + 'retained_from_sequence', + 'state', + 'transitions', + 'uploaded_through_sequence' + ]); if (!source) return null; if ( - !isString(source.experiment_id) || - !isString(source.configuration_id) || - !isString(source.participant_instance_id) || - !isString(source.state) || - !isNumber(source.next_sequence_number) || - !isNumber(source.first_sequence_number) || - !isNumber(source.last_sequence_number) - ) { - return null; - } - const assigned = source.assigned_participant_id; - if (assigned !== undefined && assigned !== null && !isString(assigned)) return null; + source.experiment_id !== configuration.experiment_id || + source.configuration_id !== configuration.configuration_id || + source.assigned_participant_id !== configuration.assigned_participant_id || + typeof source.participant_instance_id !== 'string' || + !PARTICIPANT_INSTANCE_ID.test(source.participant_instance_id) || + !nonempty(source.state) + ) return null; + + const decimalKeys = [ + 'durable_through_sequence', + 'event_count', + 'first_sequence_number', + 'last_sequence_number', + 'next_sequence_number', + 'retained_from_sequence', + 'uploaded_through_sequence' + ] as const; + if (decimalKeys.some((key) => !decimal(source[key]))) return null; if (!Array.isArray(source.events) || !Array.isArray(source.transitions)) return null; + const durable = BigInt(source.durable_through_sequence as string); + const next = BigInt(source.next_sequence_number as string); + const retained = BigInt(source.retained_from_sequence as string); + const uploaded = BigInt(source.uploaded_through_sequence as string); + const first = BigInt(source.first_sequence_number as string); + const last = BigInt(source.last_sequence_number as string); + if ( + next !== durable + 1n || retained < 1n || retained > next || uploaded >= next || + retained > uploaded + 1n || first < retained || last > durable + ) return null; + const events: ResearchEvent[] = []; - for (const raw of source.events) { - const event = readEvent(raw); + for (const rawEvent of source.events) { + const event = readEvent(rawEvent, configuration); if (!event) return null; events.push(event); } + if (BigInt(source.event_count as string) !== BigInt(events.length)) return null; + if (events.length > 0) { + for (let index = 1; index < events.length; index += 1) { + if (BigInt(events[index].sequence_number) !== BigInt(events[index - 1].sequence_number) + 1n) { + return null; + } + } + if ( + events[0].sequence_number !== source.first_sequence_number || + events[events.length - 1].sequence_number !== source.last_sequence_number + ) return null; + } else if ( + BigInt(source.last_sequence_number as string) + 1n !== + BigInt(source.first_sequence_number as string) + ) { + return null; + } + const transitions: ResearchTransition[] = []; - for (const raw of source.transitions) { - const transition = readTransition(raw); - if (!transition) return null; + let transitionState = 'IMPORTED'; + for (const rawTransition of source.transitions) { + const transition = readTransition(rawTransition); + if ( + !transition || transition.from !== transitionState || + TRANSITION_DESTINATIONS[transition.reason] !== transition.to || + !STATE_TRANSITIONS[transition.from]?.includes(transition.to) + ) return null; + transitionState = transition.to; transitions.push(transition); } + if ( + (transitions.length === 0 && source.state !== 'IMPORTED') || + (transitions.length > 0 && source.state !== transitionState) + ) return null; return { - format: root.format, - exported_at_utc_millis: root.exported_at_utc_millis, - configuration: root.configuration, - experiment: { - experiment_id: source.experiment_id, - configuration_id: source.configuration_id, - participant_instance_id: source.participant_instance_id, - assigned_participant_id: isString(assigned) ? assigned : null, - state: source.state, - next_sequence_number: source.next_sequence_number, - transitions, - events, - first_sequence_number: source.first_sequence_number, - last_sequence_number: source.last_sequence_number - } + experiment_id: configuration.experiment_id, + configuration_id: configuration.configuration_id, + participant_instance_id: source.participant_instance_id as string, + assigned_participant_id: configuration.assigned_participant_id, + state: source.state as string, + retained_from_sequence: source.retained_from_sequence as string, + uploaded_through_sequence: source.uploaded_through_sequence as string, + durable_through_sequence: source.durable_through_sequence as string, + next_sequence_number: source.next_sequence_number as string, + first_sequence_number: source.first_sequence_number as string, + last_sequence_number: source.last_sequence_number as string, + event_count: source.event_count as string, + transitions, + events }; } -function readEvent(raw: unknown): ResearchEvent | null { - const source = record(raw); - if (!source) return null; - const time = readTime(source.observed_time); - const fields = record(source.fields); - if (!time || !fields) return null; +function readEvent(raw: unknown, configuration: StudyConfiguration): ResearchEvent | null { + const source = exact(raw, [ + 'collector_id', + 'fields', + 'observed_time', + 'payload_schema_version', + 'payload_type', + 'sequence_number' + ]); + const fields = source && record(source.fields); + const time = source && readTime(source.observed_time); if ( - !isNumber(source.sequence_number) || - !isString(source.collector_id) || - !isNumber(source.payload_schema_version) || - !isString(source.payload_type) - ) { - return null; - } - // Always strings on the wire, including a survey submission, which arrives as JSON *text*. - for (const value of Object.values(fields)) if (!isString(value)) return null; - return { - sequence_number: source.sequence_number, - collector_id: source.collector_id, - payload_schema_version: source.payload_schema_version, + !source || !fields || !time || !decimal(source.sequence_number) || + !nonempty(source.collector_id) || !nonempty(source.payload_type) || + !Number.isSafeInteger(source.payload_schema_version) || + (source.payload_schema_version as number) < 1 + ) return null; + if (Object.values(fields).some((value) => typeof value !== 'string')) return null; + const event = { + sequence_number: source.sequence_number as string, + collector_id: source.collector_id as string, + payload_schema_version: source.payload_schema_version as number, observed_time: time, - payload_type: source.payload_type, + payload_type: source.payload_type as string, fields: fields as Record }; + return acceptsEvent(event, source, configuration) ? event : null; } function readTransition(raw: unknown): ResearchTransition | null { - const source = record(raw); - if (!source) return null; - const time = readTime(source.time); - if (!time || !isString(source.from) || !isString(source.to) || !isString(source.reason)) { + const source = exact(raw, ['from', 'reason', 'time', 'to']); + const time = source && readTime(source.time); + if (!source || !time || !nonempty(source.from) || !nonempty(source.to) || !nonempty(source.reason)) { return null; } - return { from: source.from, to: source.to, reason: source.reason, time }; + return { from: source.from, to: source.to, reason: source.reason, time } as ResearchTransition; } function readTime(raw: unknown): ResearchTime | null { - const source = record(raw); - if (!source) return null; + const source = exact(raw, ['boot_session_id', 'monotonic_time_nanos', 'wall_time_utc_millis']); if ( - !isNumber(source.wall_time_utc_millis) || - !isNumber(source.elapsed_realtime_nanos) || - !isString(source.boot_session_id) - ) { - return null; - } + !source || !nonempty(source.boot_session_id) || + UTF8.encode(source.boot_session_id as string).length > 128 || + !decimal(source.monotonic_time_nanos) || + !decimal(source.wall_time_utc_millis) + ) return null; return { - wall_time_utc_millis: source.wall_time_utc_millis, - elapsed_realtime_nanos: source.elapsed_realtime_nanos, - boot_session_id: source.boot_session_id + boot_session_id: source.boot_session_id as string, + monotonic_time_nanos: source.monotonic_time_nanos as string, + wall_time_utc_millis: source.wall_time_utc_millis as string }; } +function acceptsEvent( + event: ResearchEvent, + raw: Record, + configuration: StudyConfiguration +): boolean { + const configured = configuration.collectors.some((collector) => collector.id === event.collector_id) || + (event.collector_id === 'interventions.v1' && configuration.interventions.length > 0); + const contract = EVENT_CONTRACTS.get(event.collector_id); + if (!configured || !contract || event.payload_schema_version !== contract.payload_schema_version) { + return false; + } + const payload = contract.payloads.find((candidate) => candidate.types.includes(event.payload_type)); + if (!payload) return false; + const names = Object.keys(event.fields); + if ( + names.some((name) => !Object.hasOwn(payload.fields, name)) || + Object.entries(payload.fields).some(([name, field]) => + field.required ? !Object.hasOwn(event.fields, name) : false + ) || + Object.entries(event.fields).some(([name, value]) => !acceptsField(value, payload.fields[name])) + ) return false; + return canonicalBytes(raw).length <= contract.maximum_encoded_event_bytes; +} + +function acceptsField(value: string, field: CatalogField): boolean { + if (field.maximum_length !== undefined && value.length > field.maximum_length) return false; + let numeric: number; + switch (field.type) { + case 'boolean': + return value === 'true' || value === 'false'; + case 'decimal_string': + return decimal(value); + case 'enum': + return field.enum?.includes(value) === true; + case 'float32': + if (!DECIMAL_FLOAT.test(value)) return false; + numeric = Number(value); + return Number.isFinite(numeric) && Number.isFinite(Math.fround(numeric)) && inRange(numeric, field); + case 'float64': + if (!DECIMAL_FLOAT.test(value)) return false; + numeric = Number(value); + return Number.isFinite(numeric) && inRange(numeric, field); + case 'int32': + if (!CANONICAL_SIGNED_INTEGER.test(value)) return false; + numeric = Number(value); + return Number.isInteger(numeric) && numeric >= -2_147_483_648 && numeric <= 2_147_483_647 && + inRange(numeric, field); + case 'json_string': + try { + JSON.parse(value); + return true; + } catch { + return false; + } + case 'string': + return true; + } +} + +const inRange = (value: number, field: CatalogField) => + (field.minimum === undefined || value >= field.minimum) && + (field.maximum === undefined || value <= field.maximum); + +function exact(value: unknown, keys: readonly string[]): Record | null { + const candidate = record(value); + if (!candidate) return null; + const actual = Object.keys(candidate); + return actual.length === keys.length && keys.every((key) => Object.hasOwn(candidate, key)) + ? candidate + : null; +} + function record(value: unknown): Record | null { - return value && typeof value === 'object' && !Array.isArray(value) + return value !== null && typeof value === 'object' && !Array.isArray(value) ? (value as Record) : null; } -const isString = (value: unknown): value is string => typeof value === 'string'; -const isNumber = (value: unknown): value is number => - typeof value === 'number' && Number.isFinite(value); +const string = (value: unknown): string => typeof value === 'string' ? value : ''; +const nonempty = (value: unknown): value is string => typeof value === 'string' && value.length > 0; +const decimal = (value: unknown): value is string => isCanonicalDecimal(value, MAXIMUM_INT64); +const positiveDecimal = (value: unknown): value is string => decimal(value) && value !== '0'; + +function uuid(bytes: Uint8Array): string | null { + if ( + bytes.length !== BUNDLE_ID_BYTES || + (bytes[6] & 0xf0) !== 0x40 || + (bytes[8] & 0xc0) !== 0x80 + ) return null; + const value = hex(bytes); + return `${value.slice(0, 8)}-${value.slice(8, 12)}-${value.slice(12, 16)}-${value.slice(16, 20)}-${value.slice(20)}`; +} + +function hex(bytes: Uint8Array): string { + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, '0')).join(''); +} + +function concat(...parts: Uint8Array[]): Uint8Array { + const out = new Uint8Array(parts.reduce((total, part) => total + part.length, 0)); + let at = 0; + for (const part of parts) { + out.set(part, at); + at += part.length; + } + return out; +} + +function same(left: Uint8Array, right: Uint8Array): boolean { + return left.length === right.length && left.every((byte, index) => byte === right[index]); +} diff --git a/web/src/lib/adc/canonical.ts b/web/src/lib/adc/canonical.ts index fdda3dd..b4c5e38 100644 --- a/web/src/lib/adc/canonical.ts +++ b/web/src/lib/adc/canonical.ts @@ -1,149 +1,134 @@ /** - * The canonical encoder: a transcription of `StudyConfigurationCodec.encode`, byte for byte. + * RFC 8785 JSON Canonicalization Scheme (JCS) for Protocol v1 values. * - * `decode` on the device re-encodes whatever it parsed and refuses the file unless the bytes come - * back identical, so this is not "a JSON writer that happens to agree". The signature is computed - * over exactly these bytes, which means a key in the wrong place, a float in the wrong form, or an - * escape Gson would not have written produces a file that is correctly signed and rejected by - * every device. - * - * Nothing here validates or throws: the UI encodes as the researcher types, and a draft is - * half-finished most of the time. `schema.ts` is what refuses a document. + * JCS uses ECMAScript's JSON primitive serialization and recursively sorts object member names by + * UTF-16 code units. Protocol values are also I-JSON: lone surrogates and values JSON cannot encode + * are rejected instead of repaired. Schema validation is responsible for the narrower ADC rule + * that configuration numbers are bounded integers. */ -import type { - CollectorConfig, - ChoiceOption, - InterventionConfig, - LocalizedText, - NetworkTransport, - StudyConfiguration, - SurveyDefinition, - SurveyQuestion, - TinkKeyset, - UploadConfig -} from './types'; +import type { StudyConfiguration } from './types'; -/** - * Gson's default replacement table — the non-HTML-safe one. Everything from `0x20` up is written - * as itself apart from `"` and `\`, which is why `/`, `<`, `>`, `&`, `=`, `'`, DEL, and all - * non-ASCII appear raw in the output. - */ -const CONTROL_ESCAPES: readonly string[] = (() => { - const table: string[] = []; - for (let code = 0; code < 0x20; code++) table[code] = `\\u${code.toString(16).padStart(4, '0')}`; - table[0x08] = '\\b'; - table[0x09] = '\\t'; - table[0x0a] = '\\n'; - table[0x0c] = '\\f'; - table[0x0d] = '\\r'; - return table; -})(); +const ENCODER = new TextEncoder(); +const FATAL_DECODER = new TextDecoder('utf-8', { fatal: true }); -/** The escaped body of a JSON string, without the surrounding quotes. */ -export function escapeJsonString(value: string): string { - let escaped = ''; - let plain = 0; - for (let index = 0; index < value.length; index++) { - const code = value.charCodeAt(index); - let replacement: string; - if (code < 0x20) replacement = CONTROL_ESCAPES[code]; - else if (code === 0x22) replacement = '\\"'; - else if (code === 0x5c) replacement = '\\\\'; - else if (code === 0x2028) replacement = '\\u2028'; - else if (code === 0x2029) replacement = '\\u2029'; - else continue; - escaped += value.slice(plain, index) + replacement; - plain = index + 1; - } - return plain === 0 ? value : escaped + value.slice(plain); +export function canonicalize(value: unknown): string { + return encode(value, new Set()); } -function quoted(value: string): string { - return `"${escapeJsonString(value)}"`; +export function canonicalBytes(value: unknown): Uint8Array { + return ENCODER.encode(canonicalize(value)); } -/** - * Java's `Float.toString`, which is what Gson's `JsonWriter.value(Number)` calls for the one - * `Float` in the schema: the shortest decimal that round-trips to the same float32, and among - * equally short ones the closest, with at least one digit after the point. - * - * `Number.prototype.toString` gives the shortest form for a *double* — for 0.1f that is - * `0.10000000149011612`, which is a different string and therefore a different signature. - */ -export function formatFloat(value: number): string { - const float = Math.fround(value); - if (Number.isNaN(float)) return 'NaN'; - if (float === Infinity) return 'Infinity'; - if (float === -Infinity) return '-Infinity'; - - const sign = float < 0 || Object.is(float, -0) ? '-' : ''; - const magnitude = Math.abs(float); - if (magnitude === 0) return `${sign}0.0`; - - // Nine significant digits always distinguish two float32 values, so the search terminates. - let rendered = magnitude.toExponential(8); - for (let precision = 1; precision < 9; precision++) { - const candidate = magnitude.toExponential(precision - 1); - if (Math.fround(Number(candidate)) === magnitude) { - rendered = candidate; - break; +/** Parse only canonical UTF-8 JSON. This rejects whitespace, duplicate members, and alternate + * number/string spellings because re-encoding must reproduce every input byte. */ +export function parseCanonicalJson(bytes: Uint8Array): unknown { + let text: string; + let value: unknown; + try { + text = FATAL_DECODER.decode(bytes); + value = JSON.parse(text); + } catch { + throw new Error('canonical_json_invalid'); + } + if (canonicalize(value) !== text) throw new Error('canonical_json_required'); + return value; +} + +/** The Protocol v1 configuration value before JCS. The in-memory model uses `null` for disabled + * upload, while the one wire shape uses an exact empty object. Kotlin also normalizes instants and + * the set-like transport list before its canonical-byte equality check. */ +export function configurationValue(configuration: StudyConfiguration): unknown { + const normalizeInstant = (value: string) => { + const parsed = parseInstant(value); + return parsed ? formatInstant(parsed) : value; + }; + return { + ...configuration, + issued_at: normalizeInstant(configuration.issued_at), + expires_at: normalizeInstant(configuration.expires_at), + collectors: configuration.collectors.map((collector) => + collector.id === 'network_usage.v1' + ? { + ...collector, + config: { + ...collector.config, + transports: [...new Set(collector.config.transports)].sort() + } + } + : collector + ), + upload: configuration.upload ?? {} + }; +} + +export function canonicalizeConfiguration(configuration: StudyConfiguration): string { + return canonicalize(configurationValue(configuration)); +} + +export function canonicalConfigurationBytes(configuration: StudyConfiguration): Uint8Array { + return ENCODER.encode(canonicalizeConfiguration(configuration)); +} + +function encode(value: unknown, ancestors: Set): string { + if (value === null) return 'null'; + switch (typeof value) { + case 'string': + assertUnicodeScalarString(value); + return JSON.stringify(value); + case 'number': { + if (!Number.isFinite(value)) throw new Error('jcs_number'); + return JSON.stringify(value); } + case 'boolean': + return value ? 'true' : 'false'; + case 'object': + break; + default: + throw new Error('jcs_type'); } - const [mantissa, power] = rendered.split('e'); - let digits = mantissa.replace('.', ''); - let exponent = Number(power); - // One significant digit is the exception to "shortest wins": a two-digit decimal is used when it - // is strictly closer, which is why Float.MIN_VALUE is 1.4E-45 and not 1.0E-45. A two-digit - // rendering ending in zero is the one-digit decimal written again, so it never wins. - if (digits.length === 1) { - const [refined, refinedPower] = magnitude.toExponential(1).split('e'); - const closer = refined.replace('.', ''); - if (!closer.endsWith('0')) { - digits = closer; - exponent = Number(refinedPower); + const object = value as object; + if (ancestors.has(object)) throw new Error('jcs_cycle'); + ancestors.add(object); + try { + if (Array.isArray(object)) { + const entries: string[] = []; + for (let index = 0; index < object.length; index += 1) { + if (!Object.hasOwn(object, index)) throw new Error('jcs_sparse_array'); + entries.push(encode(object[index], ancestors)); + } + return `[${entries.join(',')}]`; } - } - // Two decimals of the same length can be equally close to the value. Java takes the one with the - // even significand; `toExponential` always takes the larger, which renders 4618.53125f as - // 4618.5313 where the JDK writes 4618.5312. - if (Number(digits[digits.length - 1]) % 2 === 1 && isHalfway(magnitude, digits, exponent)) { - digits = String(BigInt(digits) - 1n); + const prototype = Object.getPrototypeOf(object); + if (prototype !== Object.prototype && prototype !== null) throw new Error('jcs_object'); + const record = object as Record; + const members = Object.keys(record).sort().map((key) => { + assertUnicodeScalarString(key); + return `${JSON.stringify(key)}:${encode(record[key], ancestors)}`; + }); + return `{${members.join(',')}}`; + } finally { + ancestors.delete(object); } +} - // Java switches to `1.0E-4` / `1.0E7` outside 10^-3 ..< 10^7. Both ends are outside anything this - // schema accepts for a displacement, but a file carrying one still has to encode the way the - // device would re-encode it. - if (exponent < -3 || exponent >= 7) { - return `${sign}${digits[0]}.${digits.slice(1) || '0'}E${exponent}`; +function assertUnicodeScalarString(value: string): void { + for (let index = 0; index < value.length; index += 1) { + const unit = value.charCodeAt(index); + if (unit < 0xd800 || unit > 0xdfff) continue; + if (unit > 0xdbff || index + 1 >= value.length) throw new Error('jcs_unicode'); + const next = value.charCodeAt(++index); + if (next < 0xdc00 || next > 0xdfff) throw new Error('jcs_unicode'); } - if (exponent < 0) return `${sign}0.${'0'.repeat(-exponent - 1)}${digits}`; - const padded = digits.padEnd(exponent + 1, '0'); - return `${sign}${padded.slice(0, exponent + 1)}.${padded.slice(exponent + 1) || '0'}`; } -const FLOAT_BITS = new DataView(new ArrayBuffer(4)); +export const CANONICAL_DECIMAL = /^(0|[1-9][0-9]*)$/; -/** - * Whether the value sits exactly between `digits` and the decimal one unit below it. Both sides of - * the comparison are integers: the float is `significand × 2^exponent` exactly, and the midpoint of - * two adjacent decimals of the same length is `(2 × digits - 1) × 10^power / 2`. - */ -function isHalfway(magnitude: number, digits: string, exponent: number): boolean { - FLOAT_BITS.setFloat32(0, magnitude); - const bits = FLOAT_BITS.getUint32(0); - const biased = (bits >>> 23) & 0xff; - let left = BigInt(biased === 0 ? bits & 0x7fffff : (bits & 0x7fffff) | 0x800000); - let right = 2n * BigInt(digits) - 1n; - const power2 = (biased === 0 ? -126 : biased - 127) - 23 + 1; - if (power2 >= 0) left <<= BigInt(power2); - else right <<= BigInt(-power2); - const power10 = exponent - digits.length + 1; - if (power10 >= 0) right *= 10n ** BigInt(power10); - else left *= 10n ** BigInt(-power10); - return left === right; +export function isCanonicalDecimal(value: unknown, maximum?: bigint): value is string { + if (typeof value !== 'string' || !CANONICAL_DECIMAL.test(value)) return false; + return maximum === undefined || BigInt(value) <= maximum; } /** A `java.time.Instant`, decomposed. */ @@ -152,20 +137,12 @@ export interface Instant { nano: number; } -// `DateTimeFormatter.ISO_INSTANT` appends an offset written `+HH:MM[:ss]`, so the colon is not -// optional and the minutes are not: `+08`, `+0800`, and `+080000` are all spellings `Instant.parse` -// refuses. Accepting them here would take a hand-written file the CLI would not. const INSTANT = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.(\d{1,9}))?(Z|[+-]\d{2}:\d{2}(?::\d{2})?)$/; +const MINIMUM_INSTANT_SECOND = -62_167_219_200; +const MAXIMUM_INSTANT_SECOND = 253_402_300_799; -const MINIMUM_INSTANT_SECOND = -62_167_219_200; // 0000-01-01T00:00:00Z -const MAXIMUM_INSTANT_SECOND = 253_402_300_799; // 9999-12-31T23:59:59Z - -/** - * `Instant.parse`, narrowed to the years `Instant.toString` renders as four plain digits. Anything - * outside that would need Java's `+10000-…` form, and a study configuration has no business - * carrying one. - */ +/** Parse the ISO-8601 spellings accepted for configuration validity windows. */ export function parseInstant(text: string): Instant | null { const match = INSTANT.exec(text); if (!match) return null; @@ -180,7 +157,7 @@ export function parseInstant(text: string): Instant | null { return { second: epochSecond, nano: match[7] ? Number(match[7].padEnd(9, '0')) : 0 }; } -/** `Instant.toString`: always seconds, fractions in whole groups of three, and never an offset. */ +/** Canonical UTC spelling used by the authoring defaults. */ export function formatInstant(instant: Instant): string { const day = Math.floor(instant.second / 86_400); const time = instant.second - day * 86_400; @@ -206,21 +183,15 @@ function monthLength(year: number, month: number): number { return month === 4 || month === 6 || month === 9 || month === 11 ? 30 : 31; } -const MAXIMUM_OFFSET_SECONDS = 18 * 3_600; - function offsetSeconds(zone: string): number | null { if (zone === 'Z') return 0; const [hour, minute, second = 0] = zone.slice(1).split(':').map(Number); if (minute > 59 || second > 59) return null; - // `ZoneOffset` bounds the whole offset at ±18:00 rather than the hour field, so -18:00:01 and - // -18:16 are both out even though neither has an hour above 18. const magnitude = hour * 3_600 + minute * 60 + second; - if (magnitude > MAXIMUM_OFFSET_SECONDS) return null; + if (magnitude > 18 * 3_600) return null; return zone[0] === '-' ? -magnitude : magnitude; } -// Howard Hinnant's civil-calendar conversions, with March as the start of the year so the leap day -// lands at the end. Exact for every year this module accepts. function epochDay(year: number, month: number, day: number): number { const shifted = year - (month <= 2 ? 1 : 0); const era = Math.floor(shifted / 400); @@ -236,11 +207,8 @@ function civilFromEpochDay(days: number): [number, number, number] { const era = Math.floor(shifted / 146_097); const dayOfEra = shifted - era * 146_097; const yearOfEra = Math.trunc( - (dayOfEra - - Math.trunc(dayOfEra / 1_460) + - Math.trunc(dayOfEra / 36_524) - - Math.trunc(dayOfEra / 146_096)) / - 365 + (dayOfEra - Math.trunc(dayOfEra / 1_460) + Math.trunc(dayOfEra / 36_524) - + Math.trunc(dayOfEra / 146_096)) / 365 ); const dayOfYear = dayOfEra - (365 * yearOfEra + Math.trunc(yearOfEra / 4) - Math.trunc(yearOfEra / 100)); @@ -249,180 +217,3 @@ function civilFromEpochDay(days: number): [number, number, number] { const month = monthIndex < 10 ? monthIndex + 3 : monthIndex - 9; return [yearOfEra + era * 400 + (month <= 2 ? 1 : 0), month, day]; } - -/** - * Kotlin holds a parsed `Instant` and writes `toString()`, so a valid instant spelled some other - * way — `Date.toISOString()`'s trailing `.000`, an offset that is not `Z` — is re-spelled here - * rather than carried through. A value that is not an instant at all is passed on untouched, for - * the preview; `validate` is what refuses it. - */ -function instantText(text: string): string { - const instant = parseInstant(text); - return quoted(instant ? formatInstant(instant) : text); -} - -/** - * Every number in the schema apart from the one displacement is an `Int` or a `Long`, written by - * `JsonWriter.value(long)` as a plain decimal literal. A draft can hold something that is not a - * whole number yet, so this renders what it was given rather than pretending. - */ -function integer(value: number): string { - if (!Number.isFinite(value)) return '0'; - return Number.isInteger(value) ? BigInt(value).toString() : String(value); -} - -/** Same defence, for a draft that reached a boolean field carrying something else. */ -function boolean(value: boolean): string { - return value === true ? 'true' : 'false'; -} - -/** - * The keyset is re-emitted from Gson's `JsonObject.toString()`: compact, and in the order the keys - * were parsed in. Object property order carries that here, and every number in a Tink keyset is a - * key ID. - */ -export function keysetJson(keyset: TinkKeyset): string { - return compact(keyset); -} - -function compact(value: unknown): string { - if (value === null || value === undefined) return 'null'; - if (typeof value === 'string') return quoted(value); - if (typeof value === 'boolean') return String(value); - if (typeof value === 'number') return integer(value); - if (Array.isArray(value)) return `[${value.map(compact).join(',')}]`; - return `{${Object.entries(value as object) - .map(([key, entry]) => `${quoted(key)}:${compact(entry)}`) - .join(',')}}`; -} - -/** `sortedBy { it.name }` over a `Set`: MOBILE before WIFI, and no duplicates. */ -const TRANSPORTS_BY_ENUM_NAME: readonly NetworkTransport[] = ['mobile', 'wifi']; - -function encodeCollector(collector: CollectorConfig): string { - return `{"id":${quoted(collector.id)},"required":${boolean(collector.required)},"config":{${collectorConfig(collector)}}}`; -} - -function collectorConfig(collector: CollectorConfig): string { - switch (collector.id) { - case 'app_lifecycle.v1': - return ''; - case 'accelerometer.v1': - return ( - `"sampling_period_us":${integer(collector.config.sampling_period_us)}` + - `,"maximum_report_latency_us":${integer(collector.config.maximum_report_latency_us)}` - ); - case 'network_state.v1': - return `"include_bandwidth_estimates":${boolean(collector.config.include_bandwidth_estimates)}`; - case 'network_usage.v1': { - const transports = TRANSPORTS_BY_ENUM_NAME.filter((transport) => - collector.config.transports.includes(transport) - ); - return ( - `"transports":[${transports.map(quoted).join(',')}]` + - `,"poll_interval_minutes":${integer(collector.config.poll_interval_minutes)}` - ); - } - case 'usage_events.v1': - return `"poll_interval_minutes":${integer(collector.config.poll_interval_minutes)}`; - case 'location.v1': - return ( - `"interval_millis":${integer(collector.config.interval_millis)}` + - `,"minimum_interval_millis":${integer(collector.config.minimum_interval_millis)}` + - `,"maximum_batch_delay_millis":${integer(collector.config.maximum_batch_delay_millis)}` + - `,"minimum_displacement_meters":${formatFloat(collector.config.minimum_displacement_meters)}` + - `,"priority":${quoted(collector.config.priority)}` - ); - case 'keyboard_touch.v1': - return `"trajectory_sampling_hz":${integer(collector.config.trajectory_sampling_hz)}`; - // An id outside the union cannot reach here through `parse.ts`, which refuses one exactly as - // `decodeCollector` does. The arm exists so that if one ever did, the encoder writes an empty - // object rather than falling off the end and interpolating the token `undefined` — a document - // that is not JSON at all, signed, and refused by every device with nothing to point at. - default: - return ''; - } -} - -function localized(text: LocalizedText): string { - const translations = Object.entries(text.translations).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); - return `{"default":${quoted(text.default)},"translations":{${translations.map( - ([language, value]) => `${quoted(language)}:${quoted(value)}` - ).join(',')}}}`; -} - -function choices(options: ChoiceOption[]): string { - return `[${options.map((option) => `{"id":${quoted(option.id)},"label":${localized(option.label)}}`).join(',')}]`; -} - -function question(value: SurveyQuestion): string { - const common = `"type":${quoted(value.type)},"id":${quoted(value.id)},"prompt":${localized(value.prompt)},"required":${boolean(value.required)}`; - switch (value.type) { - case 'short_text': return `{${common},"maximum_length":${integer(value.maximum_length)}}`; - case 'scale': return `{${common},"minimum":${integer(value.minimum)},"maximum":${integer(value.maximum)},"minimum_label":${localized(value.minimum_label)},"maximum_label":${localized(value.maximum_label)}}`; - case 'single_choice': return `{${common},"options":${choices(value.options)}}`; - case 'multiple_choice': return `{${common},"options":${choices(value.options)},"minimum_selections":${integer(value.minimum_selections)},"maximum_selections":${integer(value.maximum_selections)}}`; - } -} - -function survey(value: SurveyDefinition): string { - return `{"id":${quoted(value.id)},"title":${localized(value.title)},"description":${localized(value.description)},"questions":[${value.questions.map(question).join(',')}]}`; -} - -function intervention(value: InterventionConfig): string { - const action = `{"type":${quoted(value.action.type)},"notification_title":${quoted(value.action.notification_title)},"notification_message":${quoted(value.action.notification_message)}${value.action.type === 'survey' ? `,"survey_id":${quoted(value.action.survey_id)}` : ''}}`; - const triggers = value.triggers.map((trigger) => { - const schedule = trigger.schedule.type === 'one_time' - ? `{"type":"one_time","offset_minutes":${integer(trigger.schedule.offset_minutes)},"clock":${quoted(trigger.schedule.clock)}}` - : trigger.schedule.type === 'interval' - ? `{"type":"interval","start_offset_minutes":${integer(trigger.schedule.start_offset_minutes)},"interval_minutes":${integer(trigger.schedule.interval_minutes)},"clock":${quoted(trigger.schedule.clock)}}` - : `{"type":"daily_local","local_time":${quoted(trigger.schedule.local_time)}}`; - return `{"id":${quoted(trigger.id)},"schedule":${schedule},"availability_minutes":${integer(trigger.availability_minutes)}}`; - }); - return `{"id":${quoted(value.id)},"action":${action},"triggers":[${triggers.join(',')}]}`; -} - -/** The exact string `researcher-tools canonicalize` writes, root key order included. */ -export function canonicalize(configuration: StudyConfiguration): string { - return ( - '{' + - `"schema_version":${integer(configuration.schema_version)}` + - `,"experiment_id":${quoted(configuration.experiment_id)}` + - `,"configuration_id":${quoted(configuration.configuration_id)}` + - `,"assigned_participant_id":${configuration.assigned_participant_id === null ? 'null' : quoted(configuration.assigned_participant_id)}` + - `,"issued_at":${instantText(configuration.issued_at)}` + - `,"expires_at":${instantText(configuration.expires_at)}` + - `,"minimum_app_version":${integer(configuration.minimum_app_version)}` + - `,"title":${quoted(configuration.title)}` + - `,"researcher":{"name":${quoted(configuration.researcher.name)}` + - `,"contact":${quoted(configuration.researcher.contact)}}` + - `,"purpose":${quoted(configuration.purpose)}` + - `,"duration_hours":${integer(configuration.duration_hours)}` + - `,"consent":{"document_version":${quoted(configuration.consent.document_version)}` + - `,"summary":${quoted(configuration.consent.summary)}}` + - `,"collectors":[${configuration.collectors.map(encodeCollector).join(',')}]` + - `,"surveys":[${configuration.surveys.map(survey).join(',')}]` + - `,"interventions":[${configuration.interventions.map(intervention).join(',')}]` + - `,"storage":{"maximum_local_bytes":${integer(configuration.storage.maximum_local_bytes)}}` + - `,"signer":{"key_id":${quoted(configuration.signer.key_id)}` + - `,"public_key":${quoted(configuration.signer.public_key)}}` + - `,"export":{"researcher_key_id":${quoted(configuration.export.researcher_key_id)}` + - `,"tink_hpke_public_keyset":${keysetJson(configuration.export.tink_hpke_public_keyset)}}` + - `,"upload":{${encodeUpload(configuration.upload)}}` + - '}' - ); -} - -/** An absent upload block is an empty object, not `null`: the decoder reads emptiness as "no". */ -function encodeUpload(upload: UploadConfig | null): string { - if (!upload) return ''; - return ( - `"endpoint":${quoted(upload.endpoint)}` + - `,"interval_minutes":${integer(upload.interval_minutes)}` + - `,"allow_metered":${boolean(upload.allow_metered)}` - ); -} - -export function canonicalBytes(configuration: StudyConfiguration): Uint8Array { - return new TextEncoder().encode(canonicalize(configuration)); -} diff --git a/web/src/lib/adc/crypto.ts b/web/src/lib/adc/crypto.ts index 4d69dd8..242c5be 100644 --- a/web/src/lib/adc/crypto.ts +++ b/web/src/lib/adc/crypto.ts @@ -1,64 +1,68 @@ -/** - * Ed25519 signing keys, in the exact encodings Java reads. +/** Raw Protocol v1 key handling. * - * The CLI writes `KeyPairGenerator.getInstance("Ed25519")`'s output straight to disk and the app - * reads `signer.public_key` through `X509EncodedKeySpec`, so both halves are DER — not raw 32-byte - * keys. A researcher has to be able to move between this page and the CLI in either direction: - * a key made here must sign in `researcher-tools`, and a key made there must verify here. That is - * why the two prefixes below are byte constants rather than something assembled at runtime; for - * Ed25519 the DER is fixed-length and every field is known, so there is nothing to compute. + * Every key artifact and wire field is an unpadded base64url encoding of exactly 32 raw bytes. + * There is deliberately no DER, protobuf, Tink prefix, or alternate decoder in this module. */ +import { x25519 } from '@noble/curves/ed25519.js'; import * as ed from '@noble/ed25519'; import { sha256, sha512 } from '@noble/hashes/sha2.js'; -// @noble/ed25519 ships its synchronous API unwired so the hash can be tree-shaken away. Signing -// happens inside a click handler on a key the page already holds, so the sync path is the one we -// want. ed.hashes.sha512 = sha512; +const KEY_BYTES = 32; +const SIGNATURE_BYTES = 64; +const BASE64URL = /^[A-Za-z0-9_-]+$/; + export interface SigningKeyPair { - privatePkcs8Base64: string; - publicX509Base64: string; + privateKey: string; + publicKey: string; } -/** PKCS#8: SEQUENCE { INTEGER 0, SEQUENCE { OID 1.3.101.112 }, OCTET STRING { OCTET STRING seed } } */ -const PKCS8_PREFIX = Uint8Array.of( - 0x30, 0x2e, 0x02, 0x01, 0x00, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x04, 0x22, 0x04, 0x20 -); - -/** X.509 SubjectPublicKeyInfo: SEQUENCE { SEQUENCE { OID 1.3.101.112 }, BIT STRING { key } } */ -const X509_PREFIX = Uint8Array.of( - 0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00 -); - -const KEY_BYTES = 32; +export interface HpkeKeyPair { + privateKey: string; + publicKey: string; +} export function generateSigningKeyPair(): SigningKeyPair { const { secretKey, publicKey } = ed.keygen(); + return { privateKey: encodeBase64Url(secretKey), publicKey: encodeBase64Url(publicKey) }; +} + +export function signingKeyPairFromPrivate(privateKey: string): SigningKeyPair { + const raw = decodeBase64Url(privateKey.trim(), KEY_BYTES); return { - privatePkcs8Base64: encodePkcs8(secretKey), - publicX509Base64: encodeX509(publicKey) + privateKey: encodeBase64Url(raw), + publicKey: encodeBase64Url(ed.getPublicKey(raw)) }; } -export function sign(configurationBytes: Uint8Array, privatePkcs8Base64: string): Uint8Array { - return ed.sign(configurationBytes, decodePkcs8(privatePkcs8Base64)); +export function generateHpkeKeyPair(): HpkeKeyPair { + const privateKey = crypto.getRandomValues(new Uint8Array(KEY_BYTES)); + return hpkeKeyPairFromPrivate(encodeBase64Url(privateKey)); } -/** - * Never throws: this is the self-check that stops a configuration going out with a `public_key` - * that does not match the key it was signed with, and a malformed key is one of the answers it - * exists to give. RFC 8032 semantics rather than the ZIP-215 default, because the app verifies - * through the JDK, which rejects the non-canonical encodings ZIP-215 accepts. - */ +export function hpkeKeyPairFromPrivate(privateKey: string): HpkeKeyPair { + const raw = decodeBase64Url(privateKey.trim(), KEY_BYTES); + return { + privateKey: encodeBase64Url(raw), + publicKey: encodeBase64Url(x25519.getPublicKey(raw)) + }; +} + +export function sign(configurationBytes: Uint8Array, privateKey: string): Uint8Array { + return ed.sign(configurationBytes, decodeBase64Url(privateKey, KEY_BYTES)); +} + +/** Strict RFC 8032 verification; malformed encodings are a false result, never an exception. */ export function verify( configurationBytes: Uint8Array, signature: Uint8Array, - publicX509Base64: string + publicKey: string ): boolean { try { - return ed.verify(signature, configurationBytes, decodeX509(publicX509Base64), { + if (signature.length !== SIGNATURE_BYTES) return false; + return ed.verify(signature, configurationBytes, decodeBase64Url(publicKey, KEY_BYTES), { zip215: false }); } catch { @@ -66,61 +70,38 @@ export function verify( } } -/** - * SHA-256 over the DER, not over the 32-byte key — `SignerIdentity.fingerprint` hashes whatever - * the Base64 decodes to, and this is the string a participant compares against the recruitment - * sheet. Hashing the wrong span produces a fingerprint nobody can match and nobody can debug. - */ -export function fingerprint(publicX509Base64: string): string { - const digest = sha256(decodeBase64(publicX509Base64)).subarray(0, 16); +/** Participant-facing 128-bit fingerprint over the raw Ed25519 public key. */ +export function fingerprint(publicKey: string): string { + const digest = sha256(decodeBase64Url(publicKey, KEY_BYTES)).subarray(0, 16); return Array.from(digest, (byte) => byte.toString(16).padStart(2, '0').toUpperCase()) .join('') .replace(/(.{4})(?=.)/g, '$1 '); } -export function encodePkcs8(seed: Uint8Array): string { - return encodeBase64(wrap(PKCS8_PREFIX, seed, 'pkcs8_length')); -} - -/** The 32-byte seed, which is what @noble/ed25519 and the RFC call the secret key. */ -export function decodePkcs8(privatePkcs8Base64: string): Uint8Array { - return unwrap(PKCS8_PREFIX, decodeBase64(privatePkcs8Base64), 'pkcs8_invalid'); -} - -export function encodeX509(publicKey: Uint8Array): string { - return encodeBase64(wrap(X509_PREFIX, publicKey, 'x509_length')); -} - -export function decodeX509(publicX509Base64: string): Uint8Array { - return unwrap(X509_PREFIX, decodeBase64(publicX509Base64), 'x509_invalid'); -} - -export function encodeBase64(bytes: Uint8Array): string { +export function encodeBase64Url(bytes: Uint8Array): string { let binary = ''; for (const byte of bytes) binary += String.fromCharCode(byte); - return btoa(binary); + return btoa(binary).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/g, ''); } -export function decodeBase64(text: string): Uint8Array { +/** Decode one canonical, unpadded base64url value. */ +export function decodeBase64Url(text: string, expectedBytes?: number): Uint8Array { + if (text.length === 0 || !BASE64URL.test(text) || text.includes('=')) { + throw new Error('base64url_invalid'); + } + const remainder = text.length % 4; + if (remainder === 1) throw new Error('base64url_invalid'); + const padded = text.replace(/-/g, '+').replace(/_/g, '/') + '='.repeat((4 - remainder) % 4); let binary: string; try { - binary = atob(text.trim()); + binary = atob(padded); } catch { - throw new Error('base64_invalid'); + throw new Error('base64url_invalid'); } - return Uint8Array.from(binary, (character) => character.charCodeAt(0)); -} - -function wrap(prefix: Uint8Array, key: Uint8Array, code: string): Uint8Array { - if (key.length !== KEY_BYTES) throw new Error(code); - const der = new Uint8Array(prefix.length + KEY_BYTES); - der.set(prefix); - der.set(key, prefix.length); - return der; -} - -function unwrap(prefix: Uint8Array, der: Uint8Array, code: string): Uint8Array { - if (der.length !== prefix.length + KEY_BYTES) throw new Error(code); - if (prefix.some((byte, index) => der[index] !== byte)) throw new Error(code); - return der.slice(prefix.length); + const bytes = Uint8Array.from(binary, (character) => character.charCodeAt(0)); + if (expectedBytes !== undefined && bytes.length !== expectedBytes) { + throw new Error('base64url_length'); + } + if (encodeBase64Url(bytes) !== text) throw new Error('base64url_noncanonical'); + return bytes; } diff --git a/web/src/lib/adc/envelope.ts b/web/src/lib/adc/envelope.ts index 51832c4..7efed9c 100644 --- a/web/src/lib/adc/envelope.ts +++ b/web/src/lib/adc/envelope.ts @@ -1,38 +1,80 @@ -/** - * The `.adccfg` container: `ADCCFG01`, three big-endian lengths, then the three payloads. +/** Fixed Protocol v1 `.adccfg` framing. * - * The bounds mirror `SignedConfigurationCodec.encode`'s `require`s rather than trusting the caller. - * A file that leaves this page and is refused by every phone it reaches is a worse failure than a - * download that does not start, because nothing about it says which of the two ends was wrong. + * `ADCCFG01 | signer-key-id length u16 BE | configuration length u32 BE | key id UTF-8 | + * canonical configuration | Ed25519 signature[64]`. */ -import { MAXIMUM_CONFIGURATION_BYTES } from './types'; +import { ID_PATTERN, MAXIMUM_CONFIGURATION_BYTES } from './types'; const MAGIC = 'ADCCFG01'; -const HEADER_BYTES = MAGIC.length + 2 + 4 + 2; +const HEADER_BYTES = 14; +const SIGNATURE_BYTES = 64; +const ENCODER = new TextEncoder(); +const DECODER = new TextDecoder('utf-8', { fatal: true }); + +export interface ConfigurationEnvelope { + signerKeyId: string; + configurationBytes: Uint8Array; + signature: Uint8Array; +} export function encodeEnvelope( signerKeyId: string, configurationBytes: Uint8Array, signature: Uint8Array ): Uint8Array { - const keyId = new TextEncoder().encode(signerKeyId); - if (keyId.length < 3 || keyId.length > 64) throw new Error('envelope_key_id'); + const keyId = ENCODER.encode(signerKeyId); + if (!ID_PATTERN.test(signerKeyId) || keyId.length > 64) throw new Error('envelope_key_id'); if (configurationBytes.length < 2 || configurationBytes.length > MAXIMUM_CONFIGURATION_BYTES) { throw new Error('envelope_configuration'); } - if (signature.length < 32 || signature.length > 128) throw new Error('envelope_signature'); + if (signature.length !== SIGNATURE_BYTES) throw new Error('envelope_signature'); const envelope = new Uint8Array( - HEADER_BYTES + keyId.length + configurationBytes.length + signature.length + HEADER_BYTES + keyId.length + configurationBytes.length + SIGNATURE_BYTES ); - const header = new DataView(envelope.buffer); for (let index = 0; index < MAGIC.length; index += 1) envelope[index] = MAGIC.charCodeAt(index); + const header = new DataView(envelope.buffer); header.setUint16(8, keyId.length); - header.setInt32(10, configurationBytes.length); - header.setUint16(14, signature.length); + header.setUint32(10, configurationBytes.length); envelope.set(keyId, HEADER_BYTES); envelope.set(configurationBytes, HEADER_BYTES + keyId.length); envelope.set(signature, HEADER_BYTES + keyId.length + configurationBytes.length); return envelope; } + +export function decodeEnvelope(bytes: Uint8Array): ConfigurationEnvelope { + if (bytes.length < HEADER_BYTES + SIGNATURE_BYTES) throw new Error('envelope_short'); + if (!isEnvelope(bytes)) throw new Error('envelope_magic'); + const header = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + const keyIdLength = header.getUint16(8); + const configurationLength = header.getUint32(10); + if (keyIdLength < 3 || keyIdLength > 64) throw new Error('envelope_key_id'); + if (configurationLength < 2 || configurationLength > MAXIMUM_CONFIGURATION_BYTES) { + throw new Error('envelope_configuration'); + } + const expected = HEADER_BYTES + keyIdLength + configurationLength + SIGNATURE_BYTES; + if (bytes.length !== expected) throw new Error('envelope_length'); + const keyIdEnd = HEADER_BYTES + keyIdLength; + const configurationEnd = keyIdEnd + configurationLength; + let signerKeyId: string; + try { + signerKeyId = DECODER.decode(bytes.subarray(HEADER_BYTES, keyIdEnd)); + } catch { + throw new Error('envelope_key_id'); + } + if (!ID_PATTERN.test(signerKeyId)) throw new Error('envelope_key_id'); + return { + signerKeyId, + configurationBytes: bytes.slice(keyIdEnd, configurationEnd), + signature: bytes.slice(configurationEnd) + }; +} + +export function isEnvelope(bytes: Uint8Array): boolean { + if (bytes.length < MAGIC.length) return false; + for (let index = 0; index < MAGIC.length; index += 1) { + if (bytes[index] !== MAGIC.charCodeAt(index)) return false; + } + return true; +} diff --git a/web/src/lib/adc/ids.ts b/web/src/lib/adc/ids.ts index a5a3989..2fa1a9a 100644 --- a/web/src/lib/adc/ids.ts +++ b/web/src/lib/adc/ids.ts @@ -10,16 +10,15 @@ * a pure function of the public half of the key they name — computed by the document rather than * typed into it. A researcher has no basis for choosing either, and the property that matters falls * out for free: the second configuration under the same signer gets the same `signer.key_id` by - * construction, whether the key was generated here, imported from `signing-keygen`, or read back - * out of a configuration file. The raw 32-byte key is what is hashed — not the DER, not the Tink - * JSON — so the name is a property of the key itself rather than of an encoding of it. + * construction, whether the key was generated here, imported, or read back out of a configuration + * file. The canonical raw 32-byte public key is the hash input. * * Pure, and deliberately ignorant of `canonical.ts`: `deriveConfigurationId` takes the canonical * string as an argument, so the whole module is testable from plain strings. * - * Six base-36 characters is 30 bits, 1.07×10⁹ values. The de-duplication key downstream is - * `experiment_id + configuration_id + collector_id + sequence_number` (see `docs/researcher-guide`), - * so a digest collision would silently merge two arms of a study; at ten configurations the chance + * Six base-36 characters is 30 bits, 1.07×10⁹ values. Analysis partitions by experiment and + * configuration before event-level `(participant_instance_id, sequence_number)` de-duplication, + * so a configuration digest collision could merge two arms of a study; at ten configurations the chance * of one is ≈4×10⁻⁸, and six characters is the shortest width where that number is negligible. * * A key namespace is not a ten-configuration namespace, which is why `keyTag` is a separate @@ -40,7 +39,7 @@ * published recruitment material; publishing a second string that shares its leading characters * would teach prefix comparison, which is exactly what the fingerprint has to resist — grinding * keypairs until SHA-256 of the SPKI opens with a chosen 32 bits is hours of one ordinary CPU. So: - * a different hash input (a 21-byte ASCII domain label and the raw key, versus the unlabelled DER), + * a domain-separated hash input, * a different alphabet and case (lowercase base-36 versus uppercase hex), and a different shape * (one unbroken 13-character run behind a word stem, versus eight groups of four). The tag is never * grouped into quads; the visual difference is load-bearing. What a researcher actually needs from @@ -55,9 +54,7 @@ */ import { sha256 } from '@noble/hashes/sha2.js'; -import { decodeBase64, decodeX509 } from './crypto'; -import { delimited, readMessage } from './tink'; -import type { TinkKeyset } from './types'; +import { decodeBase64Url } from './crypto'; const ENCODER = new TextEncoder(); @@ -149,16 +146,16 @@ function keyTag(domain: string, publicKey: Uint8Array): string { } /** - * `signer.key_id`. `''` when the argument is not a decodable X.509 Ed25519 public key. + * `signer.key_id`. `''` when the argument is not a canonical raw Ed25519 public key. * * Total, and the two outcomes are the only two: a legal ID exactly 20 characters long, or the empty * string a fresh document already carries — so `validate` reports the same `required` issue on the * same path it always did and nothing downstream changes shape. */ -export function deriveSignerKeyId(publicX509Base64: string): string { +export function deriveSignerKeyId(publicKey: string): string { let raw: Uint8Array; try { - raw = decodeX509(publicX509Base64); + raw = decodeBase64Url(publicKey, KEY_BYTES); } catch { return ''; } @@ -166,21 +163,14 @@ export function deriveSignerKeyId(publicX509Base64: string): string { } /** - * `export.researcher_key_id`. `''` when the keyset carries no readable HPKE public key. - * - * Field 3 of `HpkePublicKey` is the raw X25519 point, which is what gets hashed — not the keyset - * JSON, and not the Tink `keyId`, which is 32 bits of randomness rather than key material and which - * two distinct scalars may legitimately share. + * `export.researcher_key_id`. `''` when the argument is not a canonical raw X25519 public key. */ -export function deriveExportKeyId(keyset: TinkKeyset): string { - const value = keyset?.key?.[0]?.keyData?.value; - if (typeof value !== 'string') return ''; - let raw: Uint8Array | null; +export function deriveExportKeyId(publicKey: string): string { + let raw: Uint8Array; try { - raw = delimited(readMessage(decodeBase64(value)), 3); + raw = decodeBase64Url(publicKey, KEY_BYTES); } catch { return ''; } - if (!raw || raw.length !== KEY_BYTES) return ''; return `export-${keyTag(EXPORT_DOMAIN, raw)}`; } diff --git a/web/src/lib/adc/join.ts b/web/src/lib/adc/join.ts new file mode 100644 index 0000000..2125e8e --- /dev/null +++ b/web/src/lib/adc/join.ts @@ -0,0 +1,121 @@ +import { sha256 } from '@noble/hashes/sha2.js'; + +export interface JoinLink { + artifactUrl: string; + artifactSha256: string; + signerFingerprint: string; +} + +const PREFIX = 'adc://join/v1?'; +const SHA256 = /^[0-9a-f]{64}$/; +const FINGERPRINT = /^[0-9A-F]{32}$/; +const OPAQUE_PATH_TOKEN = /^[A-Za-z0-9_-]{22,}$/; +const QUERY_KEYS = ['artifact', 'sha256', 'signer_fingerprint'] as const; +const MAXIMUM_ARTIFACT_URL_BYTES = 2_048; +const MAXIMUM_JOIN_LINK_BYTES = 4_096; +const UTF8 = new TextEncoder(); +const UNRESERVED = /^[A-Za-z0-9._~-]$/; +const ARTIFACT_URL = /^https:\/\/([^/:?#]+)(?::([0-9]+))?(\/[A-Za-z0-9._~\/-]+)$/; +const HOST_LABEL = /^(?:[a-z0-9]|[a-z0-9][a-z0-9-]{0,61}[a-z0-9])$/; +const CANONICAL_PORT = /^[1-9][0-9]{0,4}$/; + +/** Build the exact immutable URI represented by an envelope digest and signer fingerprint. */ +export function createJoinLink( + artifactUrl: string, + artifact: Uint8Array, + fingerprint: string, + assignedParticipantId: string | null = null +): string { + const canonicalUrl = validateArtifactUrl(artifactUrl, assignedParticipantId); + const normalizedFingerprint = fingerprint.replaceAll(' ', ''); + const value: JoinLink = { + artifactUrl: canonicalUrl, + artifactSha256: hex(sha256(artifact)), + signerFingerprint: normalizedFingerprint + }; + return encodeJoinLink(value); +} + +export function encodeJoinLink(value: JoinLink): string { + const artifact = validateArtifactUrl(value.artifactUrl, null); + if (!SHA256.test(value.artifactSha256)) throw new Error('join_sha256_invalid'); + if (!FINGERPRINT.test(value.signerFingerprint)) throw new Error('join_fingerprint_invalid'); + const encoded = + `${PREFIX}artifact=${percentEncode(artifact)}` + + `&sha256=${value.artifactSha256}` + + `&signer_fingerprint=${value.signerFingerprint}`; + if (encoded.length > MAXIMUM_JOIN_LINK_BYTES) throw new Error('join_link_too_long'); + return encoded; +} + +export function parseJoinLink(encoded: string): JoinLink { + if (encoded.length > MAXIMUM_JOIN_LINK_BYTES || !encoded.startsWith(PREFIX)) { + throw new Error('join_link_invalid'); + } + const parts = encoded.slice(PREFIX.length).split('&'); + if (parts.length !== QUERY_KEYS.length) throw new Error('join_query_invalid'); + const values = parts.map((part, index) => { + const separator = part.indexOf('='); + if (separator <= 0 || part.indexOf('=', separator + 1) >= 0) throw new Error('join_query_invalid'); + if (part.slice(0, separator) !== QUERY_KEYS[index]) throw new Error('join_query_invalid'); + return percentDecode(part.slice(separator + 1)); + }); + const value: JoinLink = { + artifactUrl: values[0], + artifactSha256: values[1], + signerFingerprint: values[2] + }; + if (encodeJoinLink(value) !== encoded) throw new Error('join_link_noncanonical'); + return value; +} + +function validateArtifactUrl(value: string, assignedParticipantId: string | null): string { + if (value.length > MAXIMUM_ARTIFACT_URL_BYTES) { + throw new Error('join_artifact_url_invalid'); + } + const match = ARTIFACT_URL.exec(value); + if (match === null) throw new Error('join_artifact_url_invalid'); + const [, host, port = '', path] = match; + if ( + host.length > 253 || + !/[a-z]/.test(host) || + !host.split('.').every((label) => HOST_LABEL.test(label)) || + (port !== '' && + (!CANONICAL_PORT.test(port) || Number(port) > 65_535 || port === '443')) || + !path.slice(1).split('/').every((segment) => segment !== '' && segment !== '.' && segment !== '..') + ) throw new Error('join_artifact_url_invalid'); + try { + if (new URL(value).href !== value) throw new Error('join_artifact_url_invalid'); + } catch { + throw new Error('join_artifact_url_invalid'); + } + if (assignedParticipantId !== null) { + if (value.includes(assignedParticipantId)) throw new Error('join_url_exposes_participant_id'); + const lastSegment = path.split('/').at(-1) ?? ''; + if (!OPAQUE_PATH_TOKEN.test(lastSegment)) throw new Error('join_url_requires_opaque_path'); + } + return value; +} + +function percentEncode(value: string): string { + let encoded = ''; + for (const byte of UTF8.encode(value)) { + const character = String.fromCharCode(byte); + encoded += UNRESERVED.test(character) + ? character + : `%${byte.toString(16).toUpperCase().padStart(2, '0')}`; + } + return encoded; +} + +function percentDecode(value: string): string { + try { + return decodeURIComponent(value); + } catch { + throw new Error('join_escape_invalid'); + } +} + +function hex(value: Uint8Array): string { + return Array.from(value, (byte) => byte.toString(16).padStart(2, '0')).join(''); +} diff --git a/web/src/lib/adc/schema.ts b/web/src/lib/adc/schema.ts index b80251e..3b03716 100644 --- a/web/src/lib/adc/schema.ts +++ b/web/src/lib/adc/schema.ts @@ -8,17 +8,24 @@ * and the keys there are one list, split across two files that must not drift. */ -import { canonicalBytes, formatInstant, keysetJson, parseInstant, type Instant } from './canonical'; -import { isUsableHpkePublicKeyset } from './tink'; +import { + canonicalConfigurationBytes, + formatInstant, + isCanonicalDecimal, + parseInstant, + type Instant +} from './canonical'; +import { decodeBase64Url } from './crypto'; import { BOUNDS, ASSIGNED_PARTICIPANT_ID_PATTERN, - DEFAULT_MINIMUM_APP_VERSION, + DEFAULT_MINIMUM_CLIENT_VERSION, ID_PATTERN, MAXIMUM_INTERVENTION_OCCURRENCES, MAXIMUM_CONFIGURATION_BYTES, MAXIMUM_LOCAL_BYTES, MINIMUM_LOCAL_BYTES, + PLATFORM, SCHEMA_VERSION, UPLOAD_MAXIMUM_INTERVAL_MINUTES, UPLOAD_MINIMUM_INTERVAL_MINUTES, @@ -46,7 +53,7 @@ export type IssueCode = | 'document_too_large' | 'signer_missing' | 'export_key_missing' - | 'keyset_unusable' + | 'key_invalid' | 'language_tag' | 'unknown_reference' | 'selection_bounds' @@ -72,6 +79,11 @@ const DEFAULT_VALIDITY_DAYS = 90; */ export const DEFAULT_LOCAL_BYTES = 1_024 * 1_024 * 1_024; +/** Conservative UTC-18..UTC+18 local-date reach used by every Protocol v1 implementation. */ +export function maximumReachableLocalDates(studyMinutes: number): number { + return Math.ceil((studyMinutes + 36 * 60) / 1_440) + 1; +} + export function validate(configuration: StudyConfiguration): Issue[] { const issues: Issue[] = []; @@ -79,6 +91,9 @@ export function validate(configuration: StudyConfiguration): Issue[] { if (configuration.schema_version !== SCHEMA_VERSION) { issues.push(range('schema_version', [SCHEMA_VERSION, SCHEMA_VERSION])); } + if (configuration.platform !== PLATFORM) { + issues.push({ path: 'platform', code: 'required' }); + } identifier(issues, 'experiment_id', configuration.experiment_id); identifier(issues, 'configuration_id', configuration.configuration_id); if (configuration.assigned_participant_id !== null && @@ -92,7 +107,12 @@ export function validate(configuration: StudyConfiguration): Issue[] { issues.push({ path: 'expires_at', code: 'window_order' }); } - integer(issues, 'minimum_app_version', configuration.minimum_app_version, BOUNDS.minimumAppVersion); + decimal( + issues, + 'minimum_client_version', + configuration.minimum_client_version, + BOUNDS.minimumClientVersion + ); text(issues, 'title', configuration.title, BOUNDS.title); text(issues, 'researcher.name', configuration.researcher.name, BOUNDS.researcherName); text(issues, 'researcher.contact', configuration.researcher.contact, BOUNDS.researcherContact); @@ -172,18 +192,19 @@ export function validate(configuration: StudyConfiguration): Issue[] { ]); identifier(issues, 'signer.key_id', configuration.signer.key_id); - if (!configuration.signer.public_key) { - issues.push({ path: 'signer.public_key', code: 'signer_missing' }); - } else { - text(issues, 'signer.public_key', configuration.signer.public_key, BOUNDS.signerPublicKey); - } + rawPublicKey(issues, 'signer.public_key', configuration.signer.public_key, 'signer_missing'); identifier(issues, 'export.researcher_key_id', configuration.export.researcher_key_id); - keyset(issues, configuration); + rawPublicKey( + issues, + 'export.hpke_public_key', + configuration.export.hpke_public_key, + 'export_key_missing' + ); upload(issues, configuration); - const bytes = canonicalBytes(configuration).length; + const bytes = canonicalConfigurationBytes(configuration).length; if (bytes > MAXIMUM_CONFIGURATION_BYTES) { issues.push({ path: '', @@ -200,6 +221,7 @@ export function emptyConfiguration(): StudyConfiguration { const now = Math.floor(Date.now() / 1_000); return { schema_version: SCHEMA_VERSION, + platform: PLATFORM, // Inert placeholders, and so are the two key IDs below. The editor derives all four — // `lib/adc/ids.ts` — and the document it signs carries the derived values, never these. // `validate` still checks them, because it also judges documents this editor did not build @@ -210,7 +232,7 @@ export function emptyConfiguration(): StudyConfiguration { // revoked, so the default window is short enough to be a mistake worth noticing. issued_at: formatInstant({ second: now, nano: 0 }), expires_at: formatInstant({ second: now + DEFAULT_VALIDITY_DAYS * 86_400, nano: 0 }), - minimum_app_version: DEFAULT_MINIMUM_APP_VERSION, + minimum_client_version: DEFAULT_MINIMUM_CLIENT_VERSION, title: '', researcher: { name: '', contact: '' }, purpose: '', @@ -222,7 +244,7 @@ export function emptyConfiguration(): StudyConfiguration { interventions: [], storage: { maximum_local_bytes: DEFAULT_LOCAL_BYTES }, signer: { key_id: '', public_key: '' }, - export: { researcher_key_id: '', tink_hpke_public_keyset: { primaryKeyId: 0, key: [] } }, + export: { researcher_key_id: '', hpke_public_key: '' }, upload: null }; } @@ -242,6 +264,27 @@ export function defaultCollector(id: CollectorId): CollectorConfig { required: false, config: { sampling_period_us: 100_000, maximum_report_latency_us: 1_000_000 } }; + case 'battery_state.v1': + case 'temporal_context.v1': + return { id, required: false, config: {} }; + case 'gyroscope.v1': + return { + id, + required: false, + config: { sampling_period_us: 100_000, maximum_report_latency_us: 1_000_000 } + }; + case 'ambient_light.v1': + return { + id, + required: false, + config: { sampling_period_us: 1_000_000, change_threshold_millilux: 1_000 } + }; + case 'proximity.v1': + return { + id, + required: false, + config: { minimum_event_interval_ms: 1_000, change_threshold_millimeters: 0 } + }; case 'network_state.v1': return { id, required: false, config: { include_bandwidth_estimates: false } }; case 'network_usage.v1': @@ -260,7 +303,7 @@ export function defaultCollector(id: CollectorId): CollectorConfig { interval_millis: 60_000, minimum_interval_millis: 30_000, maximum_batch_delay_millis: 300_000, - minimum_displacement_meters: 25, + minimum_displacement_millimeters: 25_000, priority: 'BALANCED' } }; @@ -342,6 +385,55 @@ function validateSchedule( schedule: StudyConfiguration['interventions'][number]['triggers'][number]['schedule'], studyMinutes: number ): void { + if (schedule.type === 'random_window') { + if (schedule.local_windows.length < 1 || schedule.local_windows.length > 8) { + issues.push(range(`${path}.local_windows`, [1, 8])); + } + const parsedWindows = schedule.local_windows.map((window, index) => { + const windowPath = `${path}.local_windows.${index}`; + const start = localMinute(issues, `${windowPath}.start_local_time`, window.start_local_time); + const end = localMinute(issues, `${windowPath}.end_local_time`, window.end_local_time); + if (start !== null && end !== null && start >= end) { + issues.push({ path: windowPath, code: 'window_order' }); + } + return { start, end, path: windowPath }; + }); + parsedWindows.forEach((window, index) => { + const previous = parsedWindows[index - 1]; + if (previous && previous.end !== null && window.start !== null && previous.end > window.start) { + issues.push({ path: window.path, code: 'window_order' }); + } + }); + integer(issues, `${path}.occurrences_per_window`, schedule.occurrences_per_window, [1, 8]); + integer(issues, `${path}.maximum_occurrences_per_day`, schedule.maximum_occurrences_per_day, [1, 64]); + integer(issues, `${path}.maximum_occurrences_total`, schedule.maximum_occurrences_total, [1, 512]); + integer(issues, `${path}.minimum_separation_minutes`, schedule.minimum_separation_minutes, [1, 1_440]); + if ( + Number.isInteger(schedule.maximum_occurrences_per_day) && + schedule.maximum_occurrences_per_day > schedule.local_windows.length * schedule.occurrences_per_window + ) { + issues.push({ path: `${path}.maximum_occurrences_per_day`, code: 'schedule_bounds' }); + } + if (Number.isInteger(schedule.occurrences_per_window) && + Number.isInteger(schedule.minimum_separation_minutes)) { + parsedWindows.forEach((window) => { + if (window.start !== null && window.end !== null && + window.end - window.start < 1 + + (schedule.occurrences_per_window - 1) * schedule.minimum_separation_minutes) { + issues.push({ path: window.path, code: 'schedule_bounds' }); + } + }); + parsedWindows.forEach((window, index) => { + const next = parsedWindows[(index + 1) % parsedWindows.length]; + if (!next || window.end === null || next.start === null) return; + const nextStart = next.start + (index === parsedWindows.length - 1 ? 1_440 : 0); + if (nextStart - (window.end - 1) < schedule.minimum_separation_minutes) { + issues.push({ path: next.path, code: 'schedule_bounds' }); + } + }); + } + return; + } if (schedule.type === 'daily_local') { if (!/^(?:[01][0-9]|2[0-3]):[0-5][0-9]$/.test(schedule.local_time)) { issues.push({ path: `${path}.local_time`, code: 'instant' }); @@ -365,13 +457,26 @@ function occurrenceCount( ): number { if (!Number.isInteger(studyMinutes) || studyMinutes <= 0) return 0; if (schedule.type === 'one_time') return 1; - if (schedule.type === 'daily_local') return Math.ceil(studyMinutes / 1_440) + 1; + if (schedule.type === 'daily_local') return maximumReachableLocalDates(studyMinutes); + if (schedule.type === 'random_window') { + return Number.isInteger(schedule.maximum_occurrences_total) ? schedule.maximum_occurrences_total : 0; + } if (!Number.isInteger(schedule.start_offset_minutes) || !Number.isInteger(schedule.interval_minutes) || schedule.start_offset_minutes < 0 || schedule.start_offset_minutes >= studyMinutes || schedule.interval_minutes <= 0) return 0; return Math.ceil((studyMinutes - schedule.start_offset_minutes) / schedule.interval_minutes); } +const LOCAL_TIME = /^(?:[01][0-9]|2[0-3]):[0-5][0-9]$/; + +function localMinute(issues: Issue[], path: string, value: string): number | null { + if (!LOCAL_TIME.test(value)) { + issues.push({ path, code: 'instant' }); + return null; + } + return Number(value.slice(0, 2)) * 60 + Number(value.slice(3)); +} + function integer(issues: Issue[], path: string, value: number, bounds: Bounds): void { if (typeof value !== 'number' || !Number.isInteger(value)) issues.push({ path, code: 'integer' }); else if (value < bounds[0] || value > bounds[1]) issues.push(range(path, bounds)); @@ -401,6 +506,7 @@ function collectorConfig(issues: Issue[], path: string, collector: CollectorConf case 'app_lifecycle.v1': return; case 'accelerometer.v1': + case 'gyroscope.v1': integer( issues, `${path}.sampling_period_us`, @@ -414,6 +520,37 @@ function collectorConfig(issues: Issue[], path: string, collector: CollectorConf BOUNDS.maximumReportLatencyUs ); return; + case 'battery_state.v1': + case 'temporal_context.v1': + return; + case 'ambient_light.v1': + integer( + issues, + `${path}.sampling_period_us`, + collector.config.sampling_period_us, + BOUNDS.ambientLightSamplingPeriodUs + ); + integer( + issues, + `${path}.change_threshold_millilux`, + collector.config.change_threshold_millilux, + BOUNDS.changeThresholdMillilux + ); + return; + case 'proximity.v1': + integer( + issues, + `${path}.minimum_event_interval_ms`, + collector.config.minimum_event_interval_ms, + BOUNDS.minimumEventIntervalMs + ); + integer( + issues, + `${path}.change_threshold_millimeters`, + collector.config.change_threshold_millimeters, + BOUNDS.changeThresholdMillimeters + ); + return; case 'network_state.v1': return; case 'network_usage.v1': @@ -460,12 +597,12 @@ function collectorConfig(issues: Issue[], path: string, collector: CollectorConf config.maximum_batch_delay_millis, BOUNDS.maximumBatchDelayMillis ); - // The one field that is not an integer, so a bare comparison — NaN fails both of these. - const displacement = config.minimum_displacement_meters; - const [floor, ceiling] = BOUNDS.minimumDisplacementMeters; - if (!(displacement >= floor) || !(displacement <= ceiling)) { - issues.push(range(`${path}.minimum_displacement_meters`, BOUNDS.minimumDisplacementMeters)); - } + integer( + issues, + `${path}.minimum_displacement_millimeters`, + config.minimum_displacement_millimeters, + BOUNDS.minimumDisplacementMillimeters + ); return; } case 'keyboard_touch.v1': @@ -479,22 +616,30 @@ function collectorConfig(issues: Issue[], path: string, collector: CollectorConf } } -function keyset(issues: Issue[], configuration: StudyConfiguration): void { - const path = 'export.tink_hpke_public_keyset'; - const value = configuration.export.tink_hpke_public_keyset; - if (!value || !Array.isArray(value.key) || value.key.length === 0) { - issues.push({ path, code: 'export_key_missing' }); +function decimal(issues: Issue[], path: string, value: string, bounds: Bounds): void { + if (!isCanonicalDecimal(value)) { + issues.push({ path, code: 'integer' }); return; } - const length = keysetJson(value).length; - if (length < 32 || length > 16_384) { - issues.push({ path, code: 'length_range', bounds: { min: 32, max: 16_384 } }); + const parsed = BigInt(value); + if (parsed < BigInt(bounds[0]) || parsed > BigInt(bounds[1])) issues.push(range(path, bounds)); +} + +function rawPublicKey( + issues: Issue[], + path: string, + value: string, + missing: 'signer_missing' | 'export_key_missing' +): void { + if (!value) { + issues.push({ path, code: missing }); return; } - // The length is the only thing `ExportConfiguration` checks, so it is the only thing that would - // have been caught before a participant's phone. Everything Tink itself would refuse is refused - // here instead, while the researcher is still in front of the key that produced it. - if (!isUsableHpkePublicKeyset(value)) issues.push({ path, code: 'keyset_unusable' }); + try { + decodeBase64Url(value, 32); + } catch { + issues.push({ path, code: 'key_invalid' }); + } } /** diff --git a/web/src/lib/adc/tink.ts b/web/src/lib/adc/tink.ts deleted file mode 100644 index edb406e..0000000 --- a/web/src/lib/adc/tink.ts +++ /dev/null @@ -1,301 +0,0 @@ -/** - * The HPKE keyset pair, written the way Tink writes it. - * - * Tink's JSON keyset is a thin wrapper around a serialised `HpkePrivateKey` / `HpkePublicKey` - * protobuf, and the app hands the public half straight to `TinkJsonProtoKeysetFormat`. There is no - * protobuf runtime here: the two messages have four fields between them, all of them known, so - * they are emitted directly. `researcher-tools/examples/INSECURE-demo-hpke-private.json` and the - * keyset inside `demo-study.json` are a real pair from the shipped CLI, and `tests/crypto.spec.ts` - * rebuilds both from the demo private scalar and compares the JSON text. - * - * Key order in the objects below is load-bearing. `tink_hpke_public_keyset` is re-emitted by the - * canonicaliser in the order it was built, so a keyset assembled with its fields in another order - * canonicalises to different bytes than the same keyset from the CLI. - */ - -import { decodeBase64, encodeBase64 } from './crypto'; -import type { TinkKeyset } from './types'; -import { x25519 } from '@noble/curves/ed25519.js'; - -export interface HpkeKeyset { - publicKeyset: TinkKeyset; - privateKeyset: TinkKeyset; -} - -const PRIVATE_TYPE_URL = 'type.googleapis.com/google.crypto.tink.HpkePrivateKey'; -const PUBLIC_TYPE_URL = 'type.googleapis.com/google.crypto.tink.HpkePublicKey'; - -/** `HpkeParams { kem: DHKEM_X25519_HKDF_SHA256, kdf: HKDF_SHA256, aead: AES_256_GCM }`. */ -const PARAMS = Uint8Array.of(0x08, 0x01, 0x10, 0x01, 0x18, 0x02); - -const PRIVATE_KEY_BYTES = 32; -const MAXIMUM_KEY_ID = 0xffff_ffff; - -export function generateHpkeKeyset(): HpkeKeyset { - return hpkeKeysetFromPrivateKey(generatePrivateKey(), randomKeyId()); -} - -/** - * The deterministic half of {@link generateHpkeKeyset}, so a known scalar can be checked against a - * known keyset. The public key is the X25519 base-point multiplication of the scalar; Tink stores - * the scalar unclamped and clamps on use, so the bytes here are the scalar exactly as given. - */ -export function hpkeKeysetFromPrivateKey(privateKey: Uint8Array, keyId: number): HpkeKeyset { - if (privateKey.length !== PRIVATE_KEY_BYTES) throw new Error('hpke_private_key_length'); - if (!Number.isInteger(keyId) || keyId < 1 || keyId > MAXIMUM_KEY_ID) { - throw new Error('hpke_key_id'); - } - const publicKey = concat( - lengthDelimited(2, PARAMS), - lengthDelimited(3, x25519.getPublicKey(privateKey)) - ); - return { - publicKeyset: keyset(PUBLIC_TYPE_URL, publicKey, 'ASYMMETRIC_PUBLIC', keyId), - privateKeyset: keyset( - PRIVATE_TYPE_URL, - concat(lengthDelimited(2, publicKey), lengthDelimited(3, privateKey)), - 'ASYMMETRIC_PRIVATE', - keyId - ) - }; -} - -function keyset( - typeUrl: string, - value: Uint8Array, - keyMaterialType: string, - keyId: number -): TinkKeyset { - return { - primaryKeyId: keyId, - key: [ - { - keyData: { typeUrl, value: encodeBase64(value), keyMaterialType }, - status: 'ENABLED', - keyId, - outputPrefixType: 'TINK' - } - ] - }; -} - -/** - * Tink's `X25519.generatePrivateKey`: 32 random bytes with the three bits RFC 7748 clamping - * overwrites set to the opposite of what clamping would leave, so a scalar used without clamping - * fails loudly instead of interoperating by luck one time in eight. - */ -function generatePrivateKey(): Uint8Array { - const key = crypto.getRandomValues(new Uint8Array(PRIVATE_KEY_BYTES)); - key[0] |= 7; - key[31] &= 63; - key[31] |= 128; - return key; -} - -/** Tink's `Util.randKeyId`: any 32-bit pattern but zero, written unsigned in the JSON keyset. */ -function randomKeyId(): number { - const buffer = new Uint32Array(1); - do { - crypto.getRandomValues(buffer); - } while (buffer[0] === 0); - return buffer[0]; -} - -/** Field 1 of both messages is `version`, which is 0, and proto3 omits scalar defaults. */ -function lengthDelimited(fieldNumber: number, value: Uint8Array): number[] { - return [(fieldNumber << 3) | 2, ...varint(value.length), ...value]; -} - -function varint(value: number): number[] { - const bytes: number[] = []; - let rest = value; - do { - bytes.push(rest > 0x7f ? (rest & 0x7f) | 0x80 : rest); - rest >>>= 7; - } while (rest > 0); - return bytes; -} - -function concat(...parts: number[][]): Uint8Array { - return Uint8Array.from(parts.flat()); -} - -/* ---- reading one back --------------------------------------------------------------------- */ - -/** - * Whether a keyset read out of a study file is one the app can actually seal to. - * - * `ExportConfiguration` bounds this document's *length* and checks nothing else, and the codec - * re-emits it verbatim, so a keyset that names the wrong algorithm, is disabled, is not the primary, - * or has a truncated key signs cleanly here and is refused by Tink on a participant's phone at - * export time. Nothing between the two would have said so, and a study can be weeks old before - * anyone finds out. This is Tink's own validation, run before the signature instead of after it. - * - * Both producers of this keyset — this page and `researcher-tools hpke-keygen` — write exactly one - * enabled `TINK` key. A keyset shaped any other way did not come from either, so it is refused - * rather than guessed at. - */ -export function isUsableHpkePublicKeyset(keyset: TinkKeyset): boolean { - return hpkePublicKey(keyset) !== null; -} - -/** - * The 32-byte X25519 key a study seals to, or `null` for a keyset Tink would refuse. Same checks as - * {@link isUsableHpkePublicKeyset}, which is now this function asked as a yes-or-no question: an - * HPKE open needs `pkRm` for its `kem_context`, and deriving it a second way would be a second - * opinion about which key a bundle was sealed to. - */ -export function hpkePublicKey(keyset: TinkKeyset): Uint8Array | null { - const value = keyMaterial(keyset, PUBLIC_TYPE_URL, 'ASYMMETRIC_PUBLIC'); - if (!value) return null; - // A truncated protobuf throws out of `readMessage`, and every way a keyset can fail to be one - // gives the same answer here: it is not a keyset this can use. - try { - const fields = readMessage(value); - const publicKey = delimited(fields, 3); - return publicKey && publicKey.length === PRIVATE_KEY_BYTES && isSuite(delimited(fields, 2)) - ? publicKey - : null; - } catch { - return null; - } -} - -/** What an HPKE open needs from the file `hpke-keygen` wrote. */ -export interface HpkeRecipient { - keyId: number; - /** - * Stored unclamped, which is how Tink stores it; `@noble`'s x25519 clamps on use exactly as Tink - * does, so these are the bytes as read. - */ - scalar: Uint8Array; - publicKey: Uint8Array; -} - -/** - * The private twin of {@link hpkePublicKey}, and the CLI has one too: `HpkeCrypto` validates the - * private handle and its parameters before it will decrypt anything. The whole browser recipe is - * written to one suite and one prefix shape, so a keyset naming another suite has to be refused - * with a sentence rather than left to fail later as an opaque tag error, which is the one failure - * nobody can act on. - * - * `HpkePrivateKey { 2: HpkePublicKey, 3: bytes private_key }`, field 1 being `version = 0` that - * proto3 omits. - */ -export function readHpkePrivateKeyset(keyset: TinkKeyset): HpkeRecipient | null { - const value = keyMaterial(keyset, PRIVATE_TYPE_URL, 'ASYMMETRIC_PRIVATE'); - if (!value) return null; - try { - const fields = readMessage(value); - const scalarBytes = delimited(fields, 3); - const publicMessage = delimited(fields, 2); - if (!scalarBytes || !publicMessage || scalarBytes.length !== PRIVATE_KEY_BYTES) return null; - const publicFields = readMessage(publicMessage); - const publicKey = delimited(publicFields, 3); - if (!publicKey || publicKey.length !== PRIVATE_KEY_BYTES) return null; - if (!isSuite(delimited(publicFields, 2))) return null; - return { keyId: keyset.primaryKeyId, scalar: scalarBytes, publicKey }; - } catch { - return null; - } -} - -/** - * Tink's own validation of the wrapper, run before anything reads the key inside it. Both producers - * of these files — this page and `researcher-tools hpke-keygen` — write exactly one enabled `TINK` - * key, so a keyset shaped any other way did not come from either and is refused rather than guessed - * at. - */ -function keyMaterial( - keyset: TinkKeyset, - typeUrl: string, - keyMaterialType: string -): Uint8Array | null { - if (!keyset || typeof keyset !== 'object') return null; - if (!Array.isArray(keyset.key) || keyset.key.length !== 1) return null; - const [entry] = keyset.key; - if (!entry || typeof entry !== 'object') return null; - if (entry.status !== 'ENABLED' || entry.outputPrefixType !== 'TINK') return null; - if (!Number.isInteger(entry.keyId) || entry.keyId < 1 || entry.keyId > MAXIMUM_KEY_ID) return null; - if (entry.keyId !== keyset.primaryKeyId) return null; - const data = entry.keyData; - if (!data || typeof data !== 'object') return null; - if (data.typeUrl !== typeUrl || data.keyMaterialType !== keyMaterialType) return null; - if (typeof data.value !== 'string') return null; - try { - return decodeBase64(data.value); - } catch { - return null; - } -} - -/** `HpkeParams`, this module's one suite. */ -function isSuite(params: Uint8Array | null): boolean { - if (!params) return false; - const suite = readMessage(params); - // DHKEM_X25519_HKDF_SHA256 / HKDF_SHA256 / AES_256_GCM. Read field by field rather than compared - // as bytes, so a producer that orders the three differently is still recognised. - return scalar(suite, 1) === 1 && scalar(suite, 2) === 1 && scalar(suite, 3) === 2; -} - -interface ProtoField { - field: number; - wire: number; - bytes: Uint8Array; - value: number; -} - -/** - * A protobuf message, flat. Fields this module does not know are carried rather than refused — - * `version` is a varint proto3 omits at 0, and a reader that cannot step over one breaks on the - * first producer that writes it. - */ -export function readMessage(message: Uint8Array): ProtoField[] { - const fields: ProtoField[] = []; - let cursor = 0; - while (cursor < message.length) { - const [tag, afterTag] = readVarint(message, cursor); - const field = tag >>> 3; - const wire = tag & 7; - if (wire === 2) { - const [length, afterLength] = readVarint(message, afterTag); - const end = afterLength + length; - if (end > message.length) throw new Error('proto_truncated'); - fields.push({ field, wire, bytes: message.subarray(afterLength, end), value: 0 }); - cursor = end; - } else if (wire === 0) { - const [value, after] = readVarint(message, afterTag); - fields.push({ field, wire, bytes: EMPTY, value }); - cursor = after; - } else { - const width = wire === 5 ? 4 : wire === 1 ? 8 : -1; - if (width < 0 || afterTag + width > message.length) throw new Error('proto_wire_type'); - fields.push({ field, wire, bytes: EMPTY, value: 0 }); - cursor = afterTag + width; - } - } - return fields; -} - -export function delimited(fields: readonly ProtoField[], field: number): Uint8Array | null { - return fields.find((candidate) => candidate.field === field && candidate.wire === 2)?.bytes ?? null; -} - -function scalar(fields: readonly ProtoField[], field: number): number | null { - return fields.find((candidate) => candidate.field === field && candidate.wire === 0)?.value ?? null; -} - -const EMPTY = new Uint8Array(0); - -function readVarint(bytes: Uint8Array, start: number): [number, number] { - let value = 0; - let shift = 0; - let cursor = start; - for (;;) { - if (cursor >= bytes.length || shift > 28) throw new Error('proto_varint'); - const byte = bytes[cursor++]; - value |= (byte & 0x7f) << shift; - if ((byte & 0x80) === 0) return [value >>> 0, cursor]; - shift += 7; - } -} diff --git a/web/src/lib/adc/types.ts b/web/src/lib/adc/types.ts index 334298f..7b224ef 100644 --- a/web/src/lib/adc/types.ts +++ b/web/src/lib/adc/types.ts @@ -18,10 +18,11 @@ export const MAXIMUM_INTERVENTION_OCCURRENCES = 512; /** * Pinned. The lowest `versionCode` the schema allows, and the only one this page authors — there is * no control for it, because a floor a researcher has no way to measure is a floor they cannot set - * honestly. `BOUNDS.minimumAppVersion` stays: `validate` still has to judge documents written + * honestly. `BOUNDS.minimumClientVersion` stays: `validate` still has to judge documents written * elsewhere, and `researcher-tools` can raise the floor deliberately. */ -export const DEFAULT_MINIMUM_APP_VERSION = 1; +export const PLATFORM = 'android' as const; +export const DEFAULT_MINIMUM_CLIENT_VERSION = '1'; /** `[a-z0-9][a-z0-9-]{2,63}` — stable schema IDs. */ export const ID_PATTERN = /^[a-z0-9][a-z0-9-]{2,63}$/; @@ -39,6 +40,11 @@ export type LocationPriority = 'BALANCED' | 'HIGH_ACCURACY'; export type CollectorId = | 'app_lifecycle.v1' | 'accelerometer.v1' + | 'battery_state.v1' + | 'temporal_context.v1' + | 'gyroscope.v1' + | 'ambient_light.v1' + | 'proximity.v1' | 'network_state.v1' | 'network_usage.v1' | 'usage_events.v1' @@ -53,6 +59,11 @@ export type CollectorId = export const COLLECTOR_ORDER: readonly CollectorId[] = [ 'app_lifecycle.v1', 'accelerometer.v1', + 'battery_state.v1', + 'temporal_context.v1', + 'gyroscope.v1', + 'ambient_light.v1', + 'proximity.v1', 'network_state.v1', 'network_usage.v1', 'usage_events.v1', @@ -83,6 +94,23 @@ export type CollectorConfig = required: boolean; config: { sampling_period_us: number; maximum_report_latency_us: number }; } + | { id: 'battery_state.v1'; required: boolean; config: Record } + | { id: 'temporal_context.v1'; required: boolean; config: Record } + | { + id: 'gyroscope.v1'; + required: boolean; + config: { sampling_period_us: number; maximum_report_latency_us: number }; + } + | { + id: 'ambient_light.v1'; + required: boolean; + config: { sampling_period_us: number; change_threshold_millilux: number }; + } + | { + id: 'proximity.v1'; + required: boolean; + config: { minimum_event_interval_ms: number; change_threshold_millimeters: number }; + } | { id: 'network_state.v1'; required: boolean; @@ -101,8 +129,8 @@ export type CollectorConfig = interval_millis: number; minimum_interval_millis: number; maximum_batch_delay_millis: number; - /** A Kotlin Float. See `formatFloat` in canonical.ts — this does not round-trip as a JS number. */ - minimum_displacement_meters: number; + /** Integer millimetres on the wire; the editor presents metres. */ + minimum_displacement_millimeters: number; priority: LocationPriority; }; } @@ -141,7 +169,15 @@ export type RelativeClock = 'CALENDAR_TIME' | 'ACTIVE_RUNNING_TIME'; export type InterventionSchedule = | { type: 'one_time'; offset_minutes: number; clock: RelativeClock } | { type: 'interval'; start_offset_minutes: number; interval_minutes: number; clock: RelativeClock } - | { type: 'daily_local'; local_time: string }; + | { type: 'daily_local'; local_time: string } + | { + type: 'random_window'; + local_windows: { start_local_time: string; end_local_time: string }[]; + occurrences_per_window: number; + maximum_occurrences_per_day: number; + maximum_occurrences_total: number; + minimum_separation_minutes: number; + }; export interface InterventionTrigger { id: string; @@ -165,26 +201,17 @@ export interface UploadConfig { allow_metered: boolean; } -/** A Tink keyset document, kept as parsed JSON so it can be re-emitted in its original key order. */ -export interface TinkKeyset { - primaryKeyId: number; - key: Array<{ - keyData: { typeUrl: string; value: string; keyMaterialType: string }; - status: string; - keyId: number; - outputPrefixType: string; - }>; -} - export interface StudyConfiguration { schema_version: number; + platform: typeof PLATFORM; experiment_id: string; configuration_id: string; assigned_participant_id: string | null; /** ISO-8601 instant, exactly as `Instant.toString()` renders it. */ issued_at: string; expires_at: string; - minimum_app_version: number; + /** Positive build number, encoded as a canonical decimal string. */ + minimum_client_version: string; title: string; researcher: { name: string; contact: string }; purpose: string; @@ -195,7 +222,7 @@ export interface StudyConfiguration { interventions: InterventionConfig[]; storage: { maximum_local_bytes: number }; signer: { key_id: string; public_key: string }; - export: { researcher_key_id: string; tink_hpke_public_keyset: TinkKeyset }; + export: { researcher_key_id: string; hpke_public_key: string }; /** `null` means the study does not upload; the encoder writes `"upload":{}` for it. */ upload: UploadConfig | null; } @@ -209,23 +236,26 @@ export const BOUNDS = { durationHours: [1, 8_760], consentDocumentVersion: [1, 64], consentSummary: [1, 8_000], - // `require(minimumAppVersion > 0)` on a Kotlin `Int`, so the ceiling is the Int's, not "any - // positive number": `requireInt` throws above it and the file is refused before the bound is read. - minimumAppVersion: [1, 2_147_483_647], + // The Android build number must fit a positive Kotlin `Int`; the wire form is decimal text. + minimumClientVersion: [1, 2_147_483_647], notificationTitle: [1, 120], notificationMessage: [1, 500], availabilityMinutes: [1, 525_600], surveyText: [1, 2_000], shortTextMaximumLength: [1, 4_000], - signerPublicKey: [32, 1_024], + rawPublicKey: [43, 43], uploadEndpoint: [8, 2_048], samplingPeriodUs: [5_000, 1_000_000], + ambientLightSamplingPeriodUs: [200_000, 10_000_000], maximumReportLatencyUs: [0, 60_000_000], + changeThresholdMillilux: [0, 100_000_000], + minimumEventIntervalMs: [100, 60_000], + changeThresholdMillimeters: [0, 10_000], pollIntervalMinutes: [1, 1_440], intervalMillis: [1_000, 3_600_000], minimumIntervalMillis: [500, 3_600_000], maximumBatchDelayMillis: [0, 86_400_000], - minimumDisplacementMeters: [0, 10_000], + minimumDisplacementMillimeters: [0, 10_000_000], trajectorySamplingHz: [1, 120] } as const satisfies Record; diff --git a/web/src/lib/i18n/en.ts b/web/src/lib/i18n/en.ts index f621a74..6942ce1 100644 --- a/web/src/lib/i18n/en.ts +++ b/web/src/lib/i18n/en.ts @@ -48,6 +48,7 @@ export const en: Messages = { addSurvey: 'Add survey', addQuestion: 'Add question', addTrigger: 'Add schedule', + addWindow: 'Add time window', survey: 'Survey', surveyTitle: 'Survey title', surveyDescription: 'Survey description', @@ -68,9 +69,22 @@ export const en: Messages = { offset: 'Offset in minutes', interval: 'Interval in minutes', localTime: 'Local time', + windowStart: 'Window starts', + windowEnd: 'Window ends', + occurrencesPerWindow: 'Prompts per window', + dailyMaximum: 'Maximum prompts per day', + totalMaximum: 'Maximum prompts in study', + minimumSeparation: 'Minimum separation in minutes', + randomWindowSummary: ({ minimum, maximum }) => + `${number.format(minimum)}–${number.format(maximum)} possible prompts; exact times are chosen and stored on the phone.`, availability: 'Available for minutes', types: { shortText: 'Short text', scale: 'Numeric scale', singleChoice: 'Single choice', multipleChoice: 'Multiple choice' }, - schedules: { oneTime: 'One time', interval: 'Recurring interval', dailyLocal: 'Daily local time' }, + schedules: { + oneTime: 'One time', + interval: 'Recurring interval', + dailyLocal: 'Daily local time', + randomWindow: 'Random local windows' + }, clocks: { calendar: 'Calendar time (pauses included)', active: 'Running time (pauses excluded)' } }, @@ -106,6 +120,8 @@ export const en: Messages = { hours: 'h', hertz: 'Hz', metres: 'm', + millimetres: 'mm', + lux: 'lux', mebibytes: 'MiB', bytes: 'bytes' }, @@ -113,7 +129,7 @@ export const en: Messages = { file: { signingPrivate: 'study-signing-private.key', signingPublic: 'study-signing-public.key', - exportPrivate: 'export-hpke-private.json', + exportPrivate: 'export-hpke-private.key', exportPublic: 'export-hpke-public.json', canonical: 'study-canonical.json', signed: 'study.adccfg' @@ -145,6 +161,8 @@ export const en: Messages = { displacement: 'Minimum displacement', priority: 'Priority', trajectoryRate: 'Trajectory sampling', + changeThreshold: 'Change threshold', + minimumEventInterval: 'Minimum event interval', upload: 'Scheduled upload', endpoint: 'Endpoint', uploadInterval: 'Interval', @@ -152,7 +170,7 @@ export const en: Messages = { signerKeyId: 'Signing key ID', signerPublicKey: 'Signing public key', exportKeyId: 'Export key ID', - exportKeyset: 'Export public keyset', + exportPublicKey: 'Export public key', fingerprint: 'Fingerprint' }, /** @@ -170,6 +188,8 @@ export const en: Messages = { storageQuota: 'When it fills, collection stops. Nothing is dropped.', required: 'The study cannot start without this access.', samplingPeriod: 'A request, not a limit. Devices may go faster.', + ambientLightSamplingPeriod: + 'A hard minimum between emitted events. The latest meaningful change is retained for the next eligible emission.', bandwidthEstimates: 'Platform estimates, not measurements.', pollInterval: 'A minute is a pilot setting, not a study setting.', fastestInterval: 'Never longer than the interval.', @@ -196,6 +216,31 @@ export const en: Messages = { records: 'Raw x/y/z acceleration, gravity included', limit: 'No activity, posture, or gesture labels' }, + 'battery_state.v1': { + name: 'Battery state', + records: 'Percentage, charging state and source, and power-save state', + limit: 'No serial, hardware ID, health, or temperature' + }, + 'temporal_context.v1': { + name: 'Time context', + records: 'Time-zone ID, UTC offset, DST state, and clock-change reason', + limit: 'A time zone is not treated as a location or travel record' + }, + 'gyroscope.v1': { + name: 'Rotation', + records: 'Raw x/y/z angular velocity and sensor accuracy', + limit: 'No orientation, activity, posture, or gesture inference' + }, + 'ambient_light.v1': { + name: 'Ambient light', + records: 'Raw illuminance, sensor time, and accuracy', + limit: 'No image, environmental content, or presence inference' + }, + 'proximity.v1': { + name: 'Proximity', + records: 'Raw distance, maximum range, and near/far interpretation', + limit: 'Many phones report only near/far; values are not comparable across devices' + }, 'network_state.v1': { name: 'Connection type', records: 'Transport, metered, roaming, and validation', @@ -241,7 +286,7 @@ export const en: Messages = { `The whole configuration must stay under ${number.format(max)} bytes`, signer_missing: 'Generate the signing key first', export_key_missing: 'Generate the export key first', - keyset_unusable: 'Not a keyset the app can encrypt to. Generate or import the export key again', + key_invalid: 'Not a canonical 32-byte Protocol v1 public key. Generate or import the key again', language_tag: 'Use a valid BCP 47 language tag', unknown_reference: 'Choose a survey defined in this configuration', selection_bounds: 'Selection limits do not match this question', @@ -360,8 +405,7 @@ export const en: Messages = { /** One line each, beside the control the sentence is about. */ note: { irrevocable: 'A file you have handed out cannot be called back.', - /** Above the seven cards, once, because the pill on each card is the same decision seven - * times over and the consequence of it does not change between them. */ + /** Above the twelve cards once, because the consequence is identical on every card. */ required: 'Mark a source Required and a participant who declines it cannot start the study.', disclosure: 'The app lists the data in its own words. Agree with it.', delivery: 'They cannot take part and decline this. Say so in consent.' @@ -386,7 +430,17 @@ export const en: Messages = { archive: 'Needed to decrypt your own data.', publish: 'The fingerprint goes into your recruitment material.', distribute: 'For participants', - pilot: 'Pilot on the phones your study targets before you recruit anyone.' + pilot: 'Pilot on the phones your study targets before you recruit anyone.', + join: { + title: 'Optional join link and QR', + artifactUrl: 'HTTPS address of the signed .adccfg', + artifactHint: 'Host the exact signed file first, then enter its final address. Redirects are refused.', + personalizedHint: 'Use a long opaque final path segment. Do not put the assigned participant ID in the address.', + copy: 'Copy join link', + invalid: 'Enter a valid final HTTPS address. Personalized files require a long opaque path that does not reveal the assigned participant ID.', + immutable: 'The QR is made locally and binds this file’s complete SHA-256 and signing fingerprint. The app downloads it once; it does not poll for changes.', + qrAlt: 'QR code for the immutable study join link' + } }, read: { lede: 'Open an export a participant sent back. Nothing leaves this tab.', diff --git a/web/src/lib/i18n/types.ts b/web/src/lib/i18n/types.ts index 29248cf..3a416b2 100644 --- a/web/src/lib/i18n/types.ts +++ b/web/src/lib/i18n/types.ts @@ -76,7 +76,7 @@ export interface IssueMessages { document_too_large: (bounds: { max: number }) => string; signer_missing: string; export_key_missing: string; - keyset_unusable: string; + key_invalid: string; language_tag: string; unknown_reference: string; selection_bounds: string; @@ -133,6 +133,7 @@ export interface Messages { addSurvey: string; addQuestion: string; addTrigger: string; + addWindow: string; survey: string; surveyTitle: string; surveyDescription: string; @@ -153,9 +154,16 @@ export interface Messages { offset: string; interval: string; localTime: string; + windowStart: string; + windowEnd: string; + occurrencesPerWindow: string; + dailyMaximum: string; + totalMaximum: string; + minimumSeparation: string; + randomWindowSummary: (bounds: { minimum: number; maximum: number }) => string; availability: string; types: { shortText: string; scale: string; singleChoice: string; multipleChoice: string }; - schedules: { oneTime: string; interval: string; dailyLocal: string }; + schedules: { oneTime: string; interval: string; dailyLocal: string; randomWindow: string }; clocks: { calendar: string; active: string }; }; @@ -199,6 +207,8 @@ export interface Messages { hours: string; hertz: string; metres: string; + millimetres: string; + lux: string; mebibytes: string; bytes: string; }; @@ -239,6 +249,8 @@ export interface Messages { displacement: string; priority: string; trajectoryRate: string; + changeThreshold: string; + minimumEventInterval: string; upload: string; endpoint: string; uploadInterval: string; @@ -246,7 +258,7 @@ export interface Messages { signerKeyId: string; signerPublicKey: string; exportKeyId: string; - exportKeyset: string; + exportPublicKey: string; fingerprint: string; }; /** @@ -269,6 +281,7 @@ export interface Messages { storageQuota: string; required: string; samplingPeriod: string; + ambientLightSamplingPeriod: string; bandwidthEstimates: string; pollInterval: string; fastestInterval: string; @@ -381,10 +394,10 @@ export interface Messages { note: { irrevocable: string; /** - * What marking a source Required costs a participant. Said once above the seven cards - * rather than seven times inside them: the pill on each card says *this one is marked*, + * What marking a source Required costs a participant. Said once above the twelve cards + * rather than twelve times inside them: the pill on each card says *this one is marked*, * and this says what being marked does. Every pill names it with `aria-describedby`, so - * one sentence is still the description of all seven controls that cause it. + * one sentence is still the description of all twelve controls that cause it. */ required: string; disclosure: string; @@ -406,6 +419,16 @@ export interface Messages { publish: string; distribute: string; pilot: string; + join: { + title: string; + artifactUrl: string; + artifactHint: string; + personalizedHint: string; + copy: string; + invalid: string; + immutable: string; + qrAlt: string; + }; }; /** * The last step, and the only one that consumes rather than produces. Everything after `session` diff --git a/web/src/lib/i18n/zh-TW.ts b/web/src/lib/i18n/zh-TW.ts index ed550f1..b7e143f 100644 --- a/web/src/lib/i18n/zh-TW.ts +++ b/web/src/lib/i18n/zh-TW.ts @@ -51,6 +51,7 @@ export const zhTW: Messages = { addSurvey: '新增問卷', addQuestion: '新增題目', addTrigger: '新增排程', + addWindow: '新增時段', survey: '問卷', surveyTitle: '問卷標題', surveyDescription: '問卷說明', @@ -71,9 +72,22 @@ export const zhTW: Messages = { offset: '延後時間（分鐘）', interval: '間隔時間（分鐘）', localTime: '當地時間', + windowStart: '時段開始', + windowEnd: '時段結束', + occurrencesPerWindow: '每個時段提示次數', + dailyMaximum: '每日提示上限', + totalMaximum: '整項研究提示上限', + minimumSeparation: '最短間隔（分鐘）', + randomWindowSummary: ({ minimum, maximum }) => + `可能提示 ${number.format(minimum)}–${number.format(maximum)} 次；確切時間由手機抽選並保存。`, availability: '可填寫時間（分鐘）', types: { shortText: '簡短文字', scale: '數字量尺', singleChoice: '單選', multipleChoice: '複選' }, - schedules: { oneTime: '單次', interval: '固定間隔', dailyLocal: '每天的當地時間' }, + schedules: { + oneTime: '單次', + interval: '固定間隔', + dailyLocal: '每天的當地時間', + randomWindow: '隨機當地時段' + }, clocks: { calendar: '日曆時間（暫停期間仍計時）', active: '實際收集時間（暫停期間不計時）' } }, @@ -108,6 +122,8 @@ export const zhTW: Messages = { hours: '小時', hertz: 'Hz', metres: '公尺', + millimetres: '公釐', + lux: 'lux', mebibytes: 'MiB', bytes: '位元組' }, @@ -115,7 +131,7 @@ export const zhTW: Messages = { file: { signingPrivate: 'study-signing-private.key', signingPublic: 'study-signing-public.key', - exportPrivate: 'export-hpke-private.json', + exportPrivate: 'export-hpke-private.key', exportPublic: 'export-hpke-public.json', canonical: 'study-canonical.json', signed: 'study.adccfg' @@ -147,6 +163,8 @@ export const zhTW: Messages = { displacement: '最小位移', priority: '定位模式', trajectoryRate: '軌跡取樣率', + changeThreshold: '變化門檻', + minimumEventInterval: '最短事件間隔', upload: '自動傳送', endpoint: '接收端點', uploadInterval: '傳送間隔', @@ -154,7 +172,7 @@ export const zhTW: Messages = { signerKeyId: '簽章金鑰 ID', signerPublicKey: '簽章公鑰', exportKeyId: '匯出資料的金鑰 ID', - exportKeyset: '匯出資料的公鑰組', + exportPublicKey: '匯出資料的公鑰', fingerprint: '金鑰指紋' }, /** @@ -170,6 +188,8 @@ export const zhTW: Messages = { storageQuota: '空間用完後會停止收集，但不會刪除已收集的資料。', required: '沒有這項權限就無法開始。', samplingPeriod: '這是取樣週期要求，不是頻率上限；裝置可能更快。', + ambientLightSamplingPeriod: + '這是相鄰輸出事件之間的硬性最短間隔；最新一筆達到變化門檻的讀值會保留到下一次可輸出時。', bandwidthEstimates: '平台的估計值，不是實測值。', pollInterval: '一分鐘是試跑用的設定，不是正式研究的設定。', fastestInterval: '不能比定位間隔長。', @@ -196,6 +216,31 @@ export const zhTW: Messages = { records: '裝置座標系中的原始 x、y、z 軸加速度，包含重力', limit: '不含動作、姿勢或手勢標記' }, + 'battery_state.v1': { + name: '電池狀態', + records: '電量百分比、充電狀態與來源，以及省電模式狀態', + limit: '不含序號、硬體 ID、健康狀態或溫度' + }, + 'temporal_context.v1': { + name: '時間脈絡', + records: '時區 ID、UTC 偏移、日光節約狀態與時間變更原因', + limit: '不把時區當作位置或旅行紀錄' + }, + 'gyroscope.v1': { + name: '旋轉', + records: '原始 x、y、z 軸角速度與感測器準確度', + limit: '不推論方向、活動、姿勢或手勢' + }, + 'ambient_light.v1': { + name: '環境光', + records: '原始照度、感測器時間與準確度', + limit: '不含影像或環境內容，也不推論人在不在場' + }, + 'proximity.v1': { + name: '距離感測', + records: '原始距離、最大範圍與遠近判定', + limit: '許多手機只能回報遠近；不同裝置的數值不可直接比較' + }, 'network_state.v1': { name: '連線類型', records: '連線類型、是否按流量計費，以及漫遊與驗證狀態', @@ -240,7 +285,7 @@ export const zhTW: Messages = { document_too_large: ({ max }) => `整份設定檔必須小於 ${number.format(max)} 個位元組`, signer_missing: '請先產生簽章金鑰', export_key_missing: '請先產生匯出資料的加密金鑰', - keyset_unusable: 'App 無法使用這組金鑰加密。請重新產生匯出資料的加密金鑰，或改為匯入另一組', + key_invalid: '這不是 Protocol v1 的 32-byte 標準公開金鑰。請重新產生或匯入金鑰', language_tag: '請使用有效的 BCP 47 語言標籤', unknown_reference: '請選擇這份設定檔中已定義的問卷', selection_bounds: '選取數量限制與這題不相容', @@ -381,7 +426,17 @@ export const zhTW: Messages = { archive: '解密收到的資料時需要使用。', publish: '請將金鑰指紋放入招募資料。', distribute: '給參與者', - pilot: '正式招募前，請先在研究預計支援的 Android 版本與機型上完成測試。' + pilot: '正式招募前，請先在研究預計支援的 Android 版本與機型上完成測試。', + join: { + title: '選用的加入連結與 QR Code', + artifactUrl: '已簽署 .adccfg 的 HTTPS 位址', + artifactHint: '請先託管完全相同的已簽署檔案，再輸入最終位址；App 不接受重新導向。', + personalizedHint: '最後一段路徑必須是夠長的隨機字串，且位址不得包含指定參與者 ID。', + copy: '複製加入連結', + invalid: '請輸入有效的最終 HTTPS 位址。個人化檔案須使用不洩露指定參與者 ID 的長隨機路徑。', + immutable: 'QR Code 完全在本機產生，並綁定此檔案的完整 SHA-256 與簽章指紋。App 只下載一次，不會輪詢更新。', + qrAlt: '不可變研究加入連結的 QR Code' + } }, read: { lede: '開啟參與者回傳的匯出檔。所有內容都只保留在這個分頁。', diff --git a/web/src/lib/participant/SourceGrid.svelte b/web/src/lib/participant/SourceGrid.svelte index cc6058d..2a1c405 100644 --- a/web/src/lib/participant/SourceGrid.svelte +++ b/web/src/lib/participant/SourceGrid.svelte @@ -1,5 +1,5 @@ + +
+
+

{m.researcher.files.join.title}

+
+
+
+ (artifactUrl = value)} + /> + {#if invalid} + + {:else if joinLink} + {joinLink} +
+ joinLink} + label={m.researcher.files.join.copy} + copiedLabel={m.status.copied} + failedLabel={m.error.clipboard} + variant="text" + testid="copy-join-link" + /> +
+ {/if} + +
+ + {#if qrSource && joinLink} + {m.researcher.files.join.qrAlt} + {/if} +
+
+ + diff --git a/web/src/routes/researcher/RateBar.svelte b/web/src/routes/researcher/RateBar.svelte index d4a852b..0b4f918 100644 --- a/web/src/routes/researcher/RateBar.svelte +++ b/web/src/routes/researcher/RateBar.svelte @@ -3,7 +3,7 @@ * How much this collector writes, as a length in one colour. * * Four lit segments is four decades of events per hour, computed from the parameters on the card - * itself. Across seven cards it is the only cross-collector comparison on the page. + * itself. Across twelve cards it is the only cross-collector comparison on the page. * * It used to warm from green through amber to red, which is the battery-meter idiom exactly: a * ramp encodes danger, and there is no measurement behind danger here. Length encodes magnitude, diff --git a/web/src/routes/researcher/StepFiles.svelte b/web/src/routes/researcher/StepFiles.svelte index af0cb29..ba8de8b 100644 --- a/web/src/routes/researcher/StepFiles.svelte +++ b/web/src/routes/researcher/StepFiles.svelte @@ -18,6 +18,7 @@ import Fingerprint from '$lib/ui/Fingerprint.svelte'; import IconButton from '$lib/ui/IconButton.svelte'; import Note from '$lib/ui/Note.svelte'; + import JoinLinkPanel from './JoinLinkPanel.svelte'; import { ARTIFACTS, artifactBytes, @@ -25,7 +26,6 @@ type ArtifactId, type ArtifactSource } from './artifacts'; - import { keysetJson } from '$lib/adc/canonical'; import type { Draft } from './draft.svelte'; import type { Messages } from '$lib/i18n/types'; @@ -40,8 +40,8 @@ const source = $derived({ signingPrivate: - draft.signing.kind === 'held' ? draft.signing.material.privatePkcs8Base64 : null, - hpkePrivate: draft.hpke.kind === 'held' ? keysetJson(draft.hpke.material.privateKeyset) : null, + draft.signing.kind === 'held' ? draft.signing.material.privateKey : null, + hpkePrivate: draft.hpke.kind === 'held' ? draft.hpke.material.privateKey : null, canonical: draft.canonicalBytes, envelope: draft.envelope }); @@ -206,6 +206,15 @@ {/if} + {#if draft.envelope && draft.fingerprint} + + {/if} + {#if !signed}
diff --git a/web/src/routes/researcher/StepKeys.svelte b/web/src/routes/researcher/StepKeys.svelte index b6d54d5..e4a43d6 100644 --- a/web/src/routes/researcher/StepKeys.svelte +++ b/web/src/routes/researcher/StepKeys.svelte @@ -40,7 +40,6 @@ import DropTarget from '$lib/ui/DropTarget.svelte'; import Fingerprint from '$lib/ui/Fingerprint.svelte'; import Note from '$lib/ui/Note.svelte'; - import { keysetJson } from '$lib/adc/canonical'; import { artifactFilename } from './artifacts'; import type { Messages } from '$lib/i18n/types'; import type { Draft } from './draft.svelte'; @@ -68,10 +67,10 @@ // rather than by pulling the canonical document through `artifactBytes` on a step that has none. const encoder = new TextEncoder(); const signingBytes = $derived( - signing.kind === 'held' ? encoder.encode(signing.material.privatePkcs8Base64).length : 0 + signing.kind === 'held' ? encoder.encode(signing.material.privateKey).length : 0 ); const hpkeBytes = $derived( - hpke.kind === 'held' ? encoder.encode(keysetJson(hpke.material.privateKeyset)).length : 0 + hpke.kind === 'held' ? encoder.encode(hpke.material.privateKey).length : 0 ); const savedCount = $derived( @@ -116,7 +115,7 @@ -
+
take('hpke', file)} testid="key-import-hpke" /> diff --git a/web/src/routes/researcher/StepRead.svelte b/web/src/routes/researcher/StepRead.svelte index a3f9abc..1369b45 100644 --- a/web/src/routes/researcher/StepRead.svelte +++ b/web/src/routes/researcher/StepRead.svelte @@ -31,9 +31,9 @@ import type { IconRef } from '$lib/ui/icons'; import { openBundle } from '$lib/adc/bundle'; import { MAXIMUM_CONFIGURATION_BYTES, isCollectorId, COLLECTOR_ORDER } from '$lib/adc/types'; - import type { CollectorId, StudyConfiguration, TinkKeyset } from '$lib/adc/types'; + import type { CollectorId, StudyConfiguration } from '$lib/adc/types'; import { download } from './artifacts'; - import { hpkeKeysetFromPrivate } from './keys'; + import { hpkeKeyPairFromPrivate } from './keys'; import { parseConfiguration } from './parse'; import type { Draft } from './draft.svelte'; import type { Messages } from '$lib/i18n/types'; @@ -54,8 +54,8 @@ let bundleName = $state(''); let configuration = $state.raw(null); let configurationName = $state(''); - let keyset = $state.raw(null); - let keysetName = $state(''); + let privateKey = $state.raw(null); + let privateKeyName = $state(''); let failure = $state(''); let working = $state(false); @@ -69,7 +69,7 @@ */ const session = $derived(draft.envelope !== null && draft.hpke.kind === 'held'); - const ready = $derived(bundleBytes !== null && configuration !== null && keyset !== null); + const ready = $derived(bundleBytes !== null && configuration !== null && privateKey !== null); const opened = $derived(draft.bundle); const experiment = $derived(opened?.document.experiment ?? null); @@ -105,18 +105,17 @@ } /** - * The same import the Keys step runs, so a file that is not a keyset says so as a keyset problem - * rather than surfacing later as a study that will not open. It rebuilds the keyset from the - * scalar, which also means the key id comes from the file rather than being minted again. + * The same strict raw-key import the Keys step runs, so malformed or padded base64 fails before + * any bundle decryption is attempted. */ - async function takeKeyset(file: File) { + async function takePrivateKey(file: File) { try { - keyset = hpkeKeysetFromPrivate(await file.text()).privateKeyset; - keysetName = file.name; + privateKey = hpkeKeyPairFromPrivate(await file.text()).privateKey; + privateKeyName = file.name; failure = ''; } catch { - keyset = null; - keysetName = ''; + privateKey = null; + privateKeyName = ''; failure = m.error.keyFile; } } @@ -127,10 +126,10 @@ failure = ''; } - function useSessionKeyset() { + function useSessionKey() { if (draft.hpke.kind !== 'held') return; - keyset = draft.hpke.material.privateKeyset; - keysetName = m.file.exportPrivate; + privateKey = draft.hpke.material.privateKey; + privateKeyName = m.file.exportPrivate; failure = ''; } @@ -140,12 +139,12 @@ * the summary is somebody's data — the wrong one on screen is worse than none. */ async function open() { - if (!bundleBytes || !configuration || !keyset || working) return; + if (!bundleBytes || !configuration || !privateKey || working) return; working = true; draft.holdBundle(null); failure = ''; try { - const result = await openBundle(bundleBytes, configuration, keyset); + const result = await openBundle(bundleBytes, configuration, privateKey); if (result.ok) draft.holdBundle(result.bundle); else failure = m.error.bundle[result.failure]; } finally { @@ -167,6 +166,11 @@ const GLYPHS: Record = { 'app_lifecycle.v1': 'app', 'accelerometer.v1': 'motion', + 'battery_state.v1': 'data-volume', + 'temporal_context.v1': 'clock', + 'gyroscope.v1': 'motion', + 'ambient_light.v1': 'app', + 'proximity.v1': 'connection', 'network_state.v1': 'connection', 'network_usage.v1': 'data-volume', 'usage_events.v1': 'screen', @@ -205,14 +209,14 @@ */ const span = $derived.by(() => { if (!experiment || experiment.events.length === 0) return null; - let low = Number.POSITIVE_INFINITY; - let high = Number.NEGATIVE_INFINITY; + let low: bigint | null = null; + let high: bigint | null = null; for (const event of experiment.events) { - const at = event.observed_time.wall_time_utc_millis; - if (at < low) low = at; - if (at > high) high = at; + const at = BigInt(event.observed_time.wall_time_utc_millis); + if (low === null || at < low) low = at; + if (high === null || at > high) high = at; } - return { from: instant(low), to: instant(high) }; + return { from: instant(String(low)), to: instant(String(high)) }; }); /** @@ -220,8 +224,9 @@ * than prose: a locale-formatted date would read differently in the two languages for a value * that is the same instant in both, and this one gets compared against a log. */ - function instant(millis: number): string { - const at = new Date(millis); + function instant(millis: string): string { + const numeric = Number(millis); + const at = new Date(numeric); return Number.isNaN(at.getTime()) ? String(millis) : `${at.toISOString().slice(0, 19)}Z`; } @@ -231,8 +236,10 @@ * `n / total` already means "part of" everywhere else on this page and a denominator equal to the * numerator says nothing. */ - const lifetime = $derived(experiment ? experiment.next_sequence_number - 1 : 0); - const partial = $derived(experiment !== null && lifetime > experiment.last_sequence_number); + const lifetime = $derived(experiment ? BigInt(experiment.next_sequence_number) - 1n : 0n); + const partial = $derived( + experiment !== null && lifetime > BigInt(experiment.last_sequence_number) + );
@@ -284,9 +291,9 @@ {#if session} @@ -294,12 +301,12 @@ variant="ghost" icon="key" label={m.researcher.read.session} - onclick={useSessionKeyset} + onclick={useSessionKey} testid="read-key-session" /> {/if}
- {#if keysetName}{/if} + {#if privateKeyName}{/if}
diff --git a/web/src/routes/researcher/StepStudy.svelte b/web/src/routes/researcher/StepStudy.svelte index 3ccddab..1a06bdb 100644 --- a/web/src/routes/researcher/StepStudy.svelte +++ b/web/src/routes/researcher/StepStudy.svelte @@ -5,7 +5,7 @@ * * What is asked here is what only a person can answer. Nothing on this step names the study: * `experiment_id` and `configuration_id` are derived from the title and from the document's own - * bytes (`lib/adc/ids.ts`, shown on the sign step), and `minimum_app_version` is pinned, so three + * bytes (`lib/adc/ids.ts`, shown on the sign step), and `minimum_client_version` is pinned, so three * controls that were arithmetic dressed as questions are gone. * * Two placements are deliberate. `storage.maximum_local_bytes` sits under the collectors even @@ -208,7 +208,7 @@
- - names.exportKeyId ? `${names.exportKeyId}-private.json` : m.file.exportPrivate, - mime: 'application/json' + names.exportKeyId ? `${names.exportKeyId}-private.key` : m.file.exportPrivate, + mime: 'text/plain' }, { id: 'canonical', @@ -102,9 +102,9 @@ const encoder = new TextEncoder(); /** What each artefact is made of, as bytes. `null` means it does not exist yet. */ export interface ArtifactSource { - /** Base64 PKCS#8, exactly as `signing-keygen` writes it — no trailing newline. */ + /** Raw 32-byte Ed25519 secret, canonical unpadded base64url. */ signingPrivate: string | null; - /** The compact Tink keyset JSON, matching `INSECURE-demo-hpke-private.json` byte for byte. */ + /** Raw 32-byte X25519 secret, canonical unpadded base64url. */ hpkePrivate: string | null; canonical: Uint8Array; envelope: Uint8Array | null; diff --git a/web/src/routes/researcher/draft.svelte.ts b/web/src/routes/researcher/draft.svelte.ts index 74c649f..660519e 100644 --- a/web/src/routes/researcher/draft.svelte.ts +++ b/web/src/routes/researcher/draft.svelte.ts @@ -27,12 +27,17 @@ */ import type { ResearchBundle } from '$lib/adc/bundle'; -import { canonicalBytes, canonicalize } from '$lib/adc/canonical'; +import { + canonicalConfigurationBytes, + canonicalizeConfiguration +} from '$lib/adc/canonical'; import { fingerprint as fingerprintOf, + generateHpkeKeyPair, generateSigningKeyPair, sign as signBytes, verify, + type HpkeKeyPair, type SigningKeyPair } from '$lib/adc/crypto'; import { encodeEnvelope } from '$lib/adc/envelope'; @@ -43,7 +48,6 @@ import { deriveSignerKeyId } from '$lib/adc/ids'; import { defaultCollector, emptyConfiguration, validate, type Issue } from '$lib/adc/schema'; -import { generateHpkeKeyset, type HpkeKeyset } from '$lib/adc/tink'; import { COLLECTOR_ORDER, ID_PATTERN, @@ -57,7 +61,7 @@ import type { StepState } from '$lib/ui/types'; import { SvelteSet } from 'svelte/reactivity'; import type { ArtifactId } from './artifacts'; import { estimate } from './estimate'; -import { hpkeKeysetFromPrivate, signingKeyPairFromPrivate } from './keys'; +import { hpkeKeyPairFromPrivate, signingKeyPairFromPrivate } from './keys'; import { parseConfiguration } from './parse'; import { stepForPath, type StepId } from './steps'; @@ -100,7 +104,7 @@ function studyStarted(configuration: StudyConfiguration): boolean { export function createDraft() { let configuration = $state(emptyConfiguration()); let signing = $state>({ kind: 'empty' }); - let hpke = $state>({ kind: 'empty' }); + let hpke = $state>({ kind: 'empty' }); let signature = $state.raw(null); let envelope = $state.raw(null); @@ -165,7 +169,7 @@ export function createDraft() { // not on every keystroke in the study text. const derivedSignerKeyId = $derived(deriveSignerKeyId(configuration.signer.public_key)); const derivedExportKeyId = $derived( - deriveExportKeyId(configuration.export.tink_hpke_public_keyset) + deriveExportKeyId(configuration.export.hpke_public_key) ); const signerKeyId = $derived(signerKeyIdPin !== '' ? signerKeyIdPin : derivedSignerKeyId); const exportKeyId = $derived(exportKeyIdPin !== '' ? exportKeyIdPin : derivedExportKeyId); @@ -178,7 +182,9 @@ export function createDraft() { signer: { ...configuration.signer, key_id: signerKeyId }, export: { ...configuration.export, researcher_key_id: exportKeyId } }); - const configurationId = $derived(deriveConfigurationId(experimentId, canonicalize(unnamed))); + const configurationId = $derived( + deriveConfigurationId(experimentId, canonicalizeConfiguration(unnamed)) + ); /** * What is validated, canonicalised, signed, and downloaded. Never the editable object: the spread @@ -192,8 +198,8 @@ export function createDraft() { */ const document = $derived({ ...unnamed, configuration_id: configurationId }); - const canonical = $derived(canonicalize(document)); - const bytes = $derived(canonicalBytes(document)); + const canonical = $derived(canonicalizeConfiguration(document)); + const bytes = $derived(canonicalConfigurationBytes(document)); const issues = $derived(validate(document)); const cost = $derived(estimate(document)); const stale = $derived(signedCanonical !== null && signedCanonical !== canonical); @@ -306,15 +312,15 @@ export function createDraft() { function generateSigning() { const pair = generateSigningKeyPair(); signing = { kind: 'held', material: pair }; - configuration.signer.public_key = pair.publicX509Base64; + configuration.signer.public_key = pair.publicKey; sent['signing-private'] = false; kept['signing-private'] = false; } function generateHpke() { - const keyset = generateHpkeKeyset(); - hpke = { kind: 'held', material: keyset }; - configuration.export.tink_hpke_public_keyset = keyset.publicKeyset; + const pair = generateHpkeKeyPair(); + hpke = { kind: 'held', material: pair }; + configuration.export.hpke_public_key = pair.publicKey; sent['hpke-private'] = false; kept['hpke-private'] = false; } @@ -426,7 +432,7 @@ export function createDraft() { (studyStarted(configuration) || signedCanonical !== null) ); }, - /** The one thing on this page that came *from* a participant. Null until a tag verifies. */ + /** Participant-supplied content; null until AEAD and the complete Protocol document verify. */ get bundle() { return bundle; }, @@ -521,15 +527,15 @@ export function createDraft() { importSigning(text: string) { const pair = signingKeyPairFromPrivate(text); signing = { kind: 'held', material: pair }; - configuration.signer.public_key = pair.publicX509Base64; + configuration.signer.public_key = pair.publicKey; sent['signing-private'] = false; kept['signing-private'] = false; }, importHpke(text: string) { - const keyset = hpkeKeysetFromPrivate(text); - hpke = { kind: 'held', material: keyset }; - configuration.export.tink_hpke_public_keyset = keyset.publicKeyset; + const pair = hpkeKeyPairFromPrivate(text); + hpke = { kind: 'held', material: pair }; + configuration.export.hpke_public_key = pair.publicKey; sent['hpke-private'] = false; kept['hpke-private'] = false; }, @@ -549,10 +555,10 @@ export function createDraft() { ); exportKeyIdPin = adoptedName( loaded.export.researcher_key_id, - deriveExportKeyId(loaded.export.tink_hpke_public_keyset) + deriveExportKeyId(loaded.export.hpke_public_key) ); - if (signing.kind === 'held') loaded.signer.public_key = signing.material.publicX509Base64; - if (hpke.kind === 'held') loaded.export.tink_hpke_public_keyset = hpke.material.publicKeyset; + if (signing.kind === 'held') loaded.signer.public_key = signing.material.publicKey; + if (hpke.kind === 'held') loaded.export.hpke_public_key = hpke.material.publicKey; configuration = loaded; // The file's own name, adopted. A file whose name this editor could not have written is not // inherited: the title derives one instead, and the researcher can still override it. @@ -568,8 +574,9 @@ export function createDraft() { /** * The whole result of a decryption, or `null` to drop it. Assigned in one move because that is - * the CLI's guarantee reproduced: `researcher-tools decrypt` stages its output and writes - * nothing until the tag verifies, and nothing partial ever reaches this field either. + * the CLI's guarantee reproduced: `researcher-tools decrypt` stages its output and publishes + * nothing until AEAD and the complete closed-world document verify. Nothing partial reaches + * this field either. */ holdBundle(value: ResearchBundle | null) { bundle = value; @@ -601,13 +608,13 @@ export function createDraft() { // One snapshot for the whole act, so the bytes that are signed, the bytes that go in the // envelope, and the string staleness is measured against cannot be three different documents. const target = document; - if (target.signer.public_key !== material.publicX509Base64) return 'mismatch'; - const text = canonicalize(target); - const payload = canonicalBytes(target); + if (target.signer.public_key !== material.publicKey) return 'mismatch'; + const text = canonicalizeConfiguration(target); + const payload = canonicalConfigurationBytes(target); let produced: Uint8Array; let container: Uint8Array; try { - produced = signBytes(payload, material.privatePkcs8Base64); + produced = signBytes(payload, material.privateKey); if (!verify(payload, produced, target.signer.public_key)) return 'mismatch'; container = encodeEnvelope(target.signer.key_id, payload, produced); } catch { diff --git a/web/src/routes/researcher/estimate.ts b/web/src/routes/researcher/estimate.ts index b575dd0..746669d 100644 --- a/web/src/routes/researcher/estimate.ts +++ b/web/src/routes/researcher/estimate.ts @@ -31,6 +31,11 @@ export interface Estimate { const EVENT_BYTES = { 'app_lifecycle.v1': 180, 'accelerometer.v1': 120, + 'battery_state.v1': 140, + 'temporal_context.v1': 180, + 'gyroscope.v1': 120, + 'ambient_light.v1': 120, + 'proximity.v1': 140, 'network_state.v1': 180, 'network_usage.v1': 180, 'usage_events.v1': 180, @@ -48,6 +53,16 @@ export function collectorRate(collector: CollectorConfig): Rate { return { events: 20, bytes }; case 'accelerometer.v1': return { events: 3.6e9 / Math.max(1, collector.config.sampling_period_us), bytes }; + case 'battery_state.v1': + return { events: 4, bytes }; + case 'temporal_context.v1': + return { events: 1, bytes }; + case 'gyroscope.v1': + return { events: 3.6e9 / Math.max(1, collector.config.sampling_period_us), bytes }; + case 'ambient_light.v1': + return { events: 3.6e9 / Math.max(1, collector.config.sampling_period_us), bytes }; + case 'proximity.v1': + return { events: 3.6e6 / Math.max(1, collector.config.minimum_event_interval_ms), bytes }; case 'network_state.v1': return { events: 30, bytes }; case 'network_usage.v1': @@ -90,7 +105,7 @@ export type Volume = 0 | 1 | 2 | 3 | 4; /** * Which decade of events per hour this collector is in. Four steps rather than a continuous * position, because the constants above are order-of-magnitude and a smooth bar would claim a - * precision they do not have. Seven cards side by side then say which one writes the most. + * precision they do not have. Twelve cards side by side then say which one writes the most. */ export function volumeOf(eventsPerHour: number): Volume { if (!(eventsPerHour > 0)) return 0; diff --git a/web/src/routes/researcher/keys.ts b/web/src/routes/researcher/keys.ts index 037acca..84903cc 100644 --- a/web/src/routes/researcher/keys.ts +++ b/web/src/routes/researcher/keys.ts @@ -1,54 +1,12 @@ -/** - * Deriving a key pair from a private half that already exists. +/** Private-key imports for the researcher workspace. * - * `lib/adc` generates both key pairs and has no reason to read one back; this page does. Without - * it a second configuration under the same signer is impossible, and the cross-language workflow — - * one signed configuration per language, same signer, same `experiment_id`, new - * `configuration_id` — needs exactly that. - * - * Both derivations are pure and local: the public half is computed from the private one, never - * taken from the file beside it. - */ - -import { decodeBase64, decodePkcs8, encodeX509, type SigningKeyPair } from '$lib/adc/crypto'; -import { delimited, hpkeKeysetFromPrivateKey, readMessage, type HpkeKeyset } from '$lib/adc/tink'; -import { ed25519 } from '@noble/curves/ed25519.js'; - -/** The CLI writes the key with a trailing newline and trims on read; do the same. */ -export function signingKeyPairFromPrivate(privatePkcs8Base64: string): SigningKeyPair { - const trimmed = privatePkcs8Base64.trim(); - const seed = decodePkcs8(trimmed); - return { privatePkcs8Base64: trimmed, publicX509Base64: encodeX509(ed25519.getPublicKey(seed)) }; -} - -/** - * The private scalar out of a Tink JSON keyset. `HpkePrivateKey` is - * `{2: HpkePublicKey, 3: privateKey}` with field 1 (version 0) omitted, so the scalar is the one - * length-delimited field 3 at the top level. The key ID comes from the file rather than being - * minted again: a keyset that decrypts existing bundles has to keep announcing the same ID. + * Both artifacts are one canonical unpadded base64url string containing 32 raw bytes. Public keys + * are always derived locally; a second value beside the secret is never trusted. */ -export function hpkeKeysetFromPrivate(privateKeysetJson: string): HpkeKeyset { - const parsed: unknown = JSON.parse(privateKeysetJson); - const key = (parsed as { key?: unknown[] })?.key?.[0] as - | { keyData?: { value?: string }; keyId?: number } - | undefined; - const value = key?.keyData?.value; - if (typeof value !== 'string' || typeof key?.keyId !== 'number') { - throw new Error('hpke_keyset_shape'); - } - return hpkeKeysetFromPrivateKey(scalarOf(value), key.keyId); -} - -/** Every way a file can fail to be a keyset reads the same to the researcher: it was not one. */ -function scalarOf(base64: string): Uint8Array { - let bytes: Uint8Array | null; - try { - bytes = delimited(readMessage(decodeBase64(base64)), 3); - } catch { - throw new Error('hpke_keyset_shape'); - } - if (!bytes) throw new Error('hpke_keyset_shape'); - return bytes; -} -export type { HpkeKeyset, SigningKeyPair }; +export { + hpkeKeyPairFromPrivate, + signingKeyPairFromPrivate, + type HpkeKeyPair, + type SigningKeyPair +} from '$lib/adc/crypto'; diff --git a/web/src/routes/researcher/labels.ts b/web/src/routes/researcher/labels.ts index 4122561..e952407 100644 --- a/web/src/routes/researcher/labels.ts +++ b/web/src/routes/researcher/labels.ts @@ -30,8 +30,8 @@ export function fieldLabel(m: Messages, path: string): string { return label.signerPublicKey; case 'export.researcher_key_id': return label.exportKeyId; - case 'export.tink_hpke_public_keyset': - return label.exportKeyset; + case 'export.hpke_public_key': + return label.exportPublicKey; case 'upload.endpoint': return label.endpoint; case 'upload.interval_minutes': @@ -51,7 +51,7 @@ export function fieldLabel(m: Messages, path: string): string { return label.issuedAt; case 'expires_at': return label.expiresAt; - // No `minimum_app_version`: it is pinned, `parse.ts` clamps it, and the path is unreachable. If + // No `minimum_client_version`: it is pinned and the path is unreachable. If // a future rule resurrects it, the fallback at the end renders the path itself, which is what // that fallback is for. case 'title': @@ -66,6 +66,11 @@ export function fieldLabel(m: Messages, path: string): string { return label.samplingPeriod; case 'maximum_report_latency_us': return label.reportLatency; + case 'change_threshold_millilux': + case 'change_threshold_millimeters': + return label.changeThreshold; + case 'minimum_event_interval_ms': + return label.minimumEventInterval; case 'include_bandwidth_estimates': return label.bandwidthEstimates; case 'transports': @@ -78,7 +83,7 @@ export function fieldLabel(m: Messages, path: string): string { return label.fastestInterval; case 'maximum_batch_delay_millis': return label.batchDelay; - case 'minimum_displacement_meters': + case 'minimum_displacement_millimeters': return label.displacement; case 'priority': return label.priority; diff --git a/web/src/routes/researcher/parse.ts b/web/src/routes/researcher/parse.ts index 45c0ab4..4a367f3 100644 --- a/web/src/routes/researcher/parse.ts +++ b/web/src/routes/researcher/parse.ts @@ -22,7 +22,6 @@ import { isCollectorId, isLocationPriority, isNetworkTransport, - MAXIMUM_CONFIGURATION_BYTES, type CollectorConfig, type InterventionConfig, type InterventionSchedule, @@ -32,67 +31,22 @@ import { type SurveyQuestion, type StudyConfiguration } from '$lib/adc/types'; +import { canonicalConfigurationBytes, parseCanonicalJson } from '$lib/adc/canonical'; +import { decodeEnvelope, isEnvelope } from '$lib/adc/envelope'; +import { verify } from '$lib/adc/crypto'; +import { validate } from '$lib/adc/schema'; +import { PLATFORM } from '$lib/adc/types'; -const MAGIC = 'ADCCFG01'; -const HEADER_BYTES = MAGIC.length + 2 + 4 + 2; - -export interface Envelope { - signerKeyId: string; - configurationBytes: Uint8Array; - signature: Uint8Array; -} - -/** The inverse of `encodeEnvelope`, with the same bounds: a length that lies is a refused file. */ -export function decodeEnvelope(bytes: Uint8Array): Envelope { - if (bytes.length < HEADER_BYTES) throw new Error('envelope_short'); - for (let index = 0; index < MAGIC.length; index += 1) { - if (bytes[index] !== MAGIC.charCodeAt(index)) throw new Error('envelope_magic'); - } - const header = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const keyIdLength = header.getUint16(8); - const configurationLength = header.getInt32(10); - const signatureLength = header.getUint16(14); - if (keyIdLength < 3 || keyIdLength > 64) throw new Error('envelope_key_id'); - if (configurationLength < 2 || configurationLength > MAXIMUM_CONFIGURATION_BYTES) { - throw new Error('envelope_configuration'); - } - if (signatureLength < 32 || signatureLength > 128) throw new Error('envelope_signature'); - if (bytes.length !== HEADER_BYTES + keyIdLength + configurationLength + signatureLength) { - throw new Error('envelope_length'); - } - const keyIdEnd = HEADER_BYTES + keyIdLength; - const configurationEnd = keyIdEnd + configurationLength; - return { - signerKeyId: new TextDecoder().decode(bytes.subarray(HEADER_BYTES, keyIdEnd)), - configurationBytes: bytes.slice(keyIdEnd, configurationEnd), - signature: bytes.slice(configurationEnd) - }; -} - -export function isEnvelope(bytes: Uint8Array): boolean { - if (bytes.length < MAGIC.length) return false; - for (let index = 0; index < MAGIC.length; index += 1) { - if (bytes[index] !== MAGIC.charCodeAt(index)) return false; - } - return true; -} +export { decodeEnvelope, isEnvelope } from '$lib/adc/envelope'; /** * A closed-world structural read, field by field. Unknown, absent, or mistyped fields are refused * exactly as the Android codec refuses them. The editor never invents replacement study content. - * - * `tink_hpke_public_keyset` is the exception and is kept exactly as parsed, property order - * included — the canonicaliser re-emits it in the order it was built, so re-ordering it here would - * change the bytes that get signed. */ export function parseConfiguration(bytes: Uint8Array): StudyConfiguration { - const source = isEnvelope(bytes) ? decodeEnvelope(bytes).configurationBytes : bytes; - let parsed: unknown; - try { - parsed = JSON.parse(new TextDecoder().decode(source)); - } catch { - throw new Error('parse_json'); - } + const envelope = isEnvelope(bytes) ? decodeEnvelope(bytes) : null; + const source = envelope?.configurationBytes ?? bytes; + const parsed = parseCanonicalJson(source); if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) throw new Error('parse_shape'); const raw = parsed as Record; if (!('experiment_id' in raw) && !('collectors' in raw)) throw new Error('parse_shape'); @@ -109,17 +63,19 @@ export function parseConfiguration(bytes: Uint8Array): StudyConfiguration { requireExactKeys(consent, ['document_version', 'summary']); requireExactKeys(storage, ['maximum_local_bytes']); requireExactKeys(signer, ['key_id', 'public_key']); - requireExactKeys(exported, ['researcher_key_id', 'tink_hpke_public_keyset']); + requireExactKeys(exported, ['researcher_key_id', 'hpke_public_key']); if (Object.keys(upload).length > 0) requireExactKeys(upload, ['endpoint', 'interval_minutes', 'allow_metered']); - return { + if (raw.platform !== PLATFORM) throw new Error('parse_platform'); + const configuration: StudyConfiguration = { schema_version: numeric(raw.schema_version), + platform: PLATFORM, experiment_id: string(raw.experiment_id), configuration_id: string(raw.configuration_id), assigned_participant_id: nullableString(raw.assigned_participant_id), issued_at: string(raw.issued_at), expires_at: string(raw.expires_at), - minimum_app_version: appVersion(raw.minimum_app_version), + minimum_client_version: string(raw.minimum_client_version), title: string(raw.title), researcher: { name: string(researcher.name), @@ -143,9 +99,7 @@ export function parseConfiguration(bytes: Uint8Array): StudyConfiguration { }, export: { researcher_key_id: string(exported.researcher_key_id), - tink_hpke_public_keyset: isKeysetShaped(exported.tink_hpke_public_keyset) - ? (exported.tink_hpke_public_keyset as StudyConfiguration['export']['tink_hpke_public_keyset']) - : fail('parse_keyset') + hpke_public_key: string(exported.hpke_public_key) }, // `{}` is how an absent upload block is written, so an empty object is "no", not "malformed". upload: @@ -157,11 +111,22 @@ export function parseConfiguration(bytes: Uint8Array): StudyConfiguration { } : null }; + if (!sameBytes(canonicalConfigurationBytes(configuration), source)) { + throw new Error('parse_canonical'); + } + if (validate(configuration).length > 0) throw new Error('parse_invalid'); + if (envelope) { + if (envelope.signerKeyId !== configuration.signer.key_id) throw new Error('envelope_signer'); + if (!verify(source, envelope.signature, configuration.signer.public_key)) { + throw new Error('envelope_signature'); + } + } + return configuration; } const ROOT_KEYS = [ - 'schema_version', 'experiment_id', 'configuration_id', 'assigned_participant_id', 'issued_at', - 'expires_at', 'minimum_app_version', 'title', 'researcher', 'purpose', 'duration_hours', 'consent', + 'schema_version', 'platform', 'experiment_id', 'configuration_id', 'assigned_participant_id', 'issued_at', + 'expires_at', 'minimum_client_version', 'title', 'researcher', 'purpose', 'duration_hours', 'consent', 'collectors', 'surveys', 'interventions', 'storage', 'signer', 'export', 'upload' ] as const; @@ -184,23 +149,12 @@ const boolean = (value: unknown): boolean => typeof value === 'boolean' ? value const array = (value: unknown): unknown[] => Array.isArray(value) ? value : fail('parse_array'); const numeric = (value: unknown): number => - typeof value === 'number' && Number.isFinite(value) ? value : fail('parse_number'); + typeof value === 'number' && Number.isSafeInteger(value) ? value : fail('parse_number'); function fail(message: string): never { throw new Error(message); } -/** - * Kept as a named seam because this field has no editor control. Its value is never clamped: - * `validate` reports an illegal floor, and a legal value round-trips byte for byte. - */ -const appVersion = (value: unknown): number => numeric(value); - -/** - * Whatever it turns out to be, it is kept property-for-property: the canonicaliser re-emits this - * object in the order it was parsed in, so re-ordering it here would change the bytes that get - * signed. Only "is it an object at all" is decided; `validate` decides whether Tink can use it. - */ -function isKeysetShaped(value: unknown): boolean { - return !!value && typeof value === 'object' && !Array.isArray(value); +function sameBytes(left: Uint8Array, right: Uint8Array): boolean { + return left.length === right.length && left.every((byte, index) => byte === right[index]); } /** @@ -229,6 +183,42 @@ function collector(raw: unknown): CollectorConfig { maximum_report_latency_us: numeric(config.maximum_report_latency_us) } }; + case 'battery_state.v1': + requireExactKeys(config, []); + return { id: source.id, required, config: {} }; + case 'temporal_context.v1': + requireExactKeys(config, []); + return { id: source.id, required, config: {} }; + case 'gyroscope.v1': + requireExactKeys(config, ['sampling_period_us', 'maximum_report_latency_us']); + return { + id: source.id, + required, + config: { + sampling_period_us: numeric(config.sampling_period_us), + maximum_report_latency_us: numeric(config.maximum_report_latency_us) + } + }; + case 'ambient_light.v1': + requireExactKeys(config, ['sampling_period_us', 'change_threshold_millilux']); + return { + id: source.id, + required, + config: { + sampling_period_us: numeric(config.sampling_period_us), + change_threshold_millilux: numeric(config.change_threshold_millilux) + } + }; + case 'proximity.v1': + requireExactKeys(config, ['minimum_event_interval_ms', 'change_threshold_millimeters']); + return { + id: source.id, + required, + config: { + minimum_event_interval_ms: numeric(config.minimum_event_interval_ms), + change_threshold_millimeters: numeric(config.change_threshold_millimeters) + } + }; case 'network_state.v1': requireExactKeys(config, ['include_bandwidth_estimates']); return { @@ -256,7 +246,7 @@ function collector(raw: unknown): CollectorConfig { case 'location.v1': { requireExactKeys(config, [ 'interval_millis', 'minimum_interval_millis', 'maximum_batch_delay_millis', - 'minimum_displacement_meters', 'priority' + 'minimum_displacement_millimeters', 'priority' ]); const priority = config.priority; if (!isLocationPriority(priority)) throw new Error('parse_collector'); @@ -267,7 +257,7 @@ function collector(raw: unknown): CollectorConfig { interval_millis: numeric(config.interval_millis), minimum_interval_millis: numeric(config.minimum_interval_millis), maximum_batch_delay_millis: numeric(config.maximum_batch_delay_millis), - minimum_displacement_meters: numeric(config.minimum_displacement_meters), + minimum_displacement_millimeters: numeric(config.minimum_displacement_millimeters), priority } }; @@ -396,11 +386,32 @@ function intervention(raw: unknown): InterventionConfig { requireExactKeys(trigger, ['id', 'schedule', 'availability_minutes']); const schedule = object(trigger.schedule); const type = schedule.type; - if (type !== 'one_time' && type !== 'interval' && type !== 'daily_local') throw new Error('parse_schedule'); + if (type !== 'one_time' && type !== 'interval' && type !== 'daily_local' && + type !== 'random_window') throw new Error('parse_schedule'); let parsedSchedule: InterventionSchedule; if (type === 'daily_local') { requireExactKeys(schedule, ['type', 'local_time']); parsedSchedule = { type, local_time: string(schedule.local_time) }; + } else if (type === 'random_window') { + requireExactKeys(schedule, [ + 'type', 'local_windows', 'occurrences_per_window', 'maximum_occurrences_per_day', + 'maximum_occurrences_total', 'minimum_separation_minutes' + ]); + parsedSchedule = { + type, + local_windows: array(schedule.local_windows).map((rawWindow) => { + const window = object(rawWindow); + requireExactKeys(window, ['start_local_time', 'end_local_time']); + return { + start_local_time: string(window.start_local_time), + end_local_time: string(window.end_local_time) + }; + }), + occurrences_per_window: numeric(schedule.occurrences_per_window), + maximum_occurrences_per_day: numeric(schedule.maximum_occurrences_per_day), + maximum_occurrences_total: numeric(schedule.maximum_occurrences_total), + minimum_separation_minutes: numeric(schedule.minimum_separation_minutes) + }; } else { const clock = schedule.clock; if (clock !== 'CALENDAR_TIME' && clock !== 'ACTIVE_RUNNING_TIME') throw new Error('parse_schedule'); diff --git a/web/src/routes/researcher/presets.ts b/web/src/routes/researcher/presets.ts index 797f0e1..b0bdf93 100644 --- a/web/src/routes/researcher/presets.ts +++ b/web/src/routes/researcher/presets.ts @@ -29,14 +29,18 @@ export const PRESETS = { // a preset row that needs a second line costs 50px on the card, and the sixth value is the one // the box reaches in two keystrokes. 5 Hz is typed. sampling_period_us: [1, 10, 50, 100, 200], + ambient_sampling_period_us: [0.2, 0.5, 1, 5, 10], // Seconds. Zero is unbatched delivery and is a real choice, not an empty box. maximum_report_latency_us: [0, 1, 5, 30, 60], + change_threshold_millilux: [0, 1, 10, 100, 1_000], + minimum_event_interval_ms: [0.1, 0.5, 1, 10, 60], + change_threshold_millimeters: [0, 1, 10, 100, 1_000], // 6 hours comes off the shortlist and stays a rung on the ladder below, one arrow press from 4. poll_interval_minutes: [1, 5, 15, 60, 1_440], interval_millis: [1_000, 10_000, 60_000, 300_000, 3_600_000], maximum_batch_delay_millis: [0, 30_000, 300_000, 3_600_000, 86_400_000], // 1 km is a kilometre of walking between fixes; it is typed, not clicked. - minimum_displacement_meters: [0, 5, 25, 100], + minimum_displacement_millimeters: [0, 5, 25, 100], trajectory_sampling_hz: [30, 60, 120], // `DEFAULT_LOCAL_BYTES` rather than a fourth literal: the study opens on that value, so the chip // showing it has to be the same number and cannot be left behind if the default moves. diff --git a/web/src/routes/researcher/random-window.ts b/web/src/routes/researcher/random-window.ts new file mode 100644 index 0000000..29ad84d --- /dev/null +++ b/web/src/routes/researcher/random-window.ts @@ -0,0 +1,24 @@ +import type { InterventionSchedule } from '$lib/adc/types'; + +type RandomWindowSchedule = Extract; +type RandomLocalWindow = RandomWindowSchedule['local_windows'][number]; + +export function nextRandomWindow(schedule: RandomWindowSchedule): RandomLocalWindow | null { + const minute = (value: string) => Number(value.slice(0, 2)) * 60 + Number(value.slice(3)); + const clock = (value: number) => + `${String(Math.floor(value / 60)).padStart(2, '0')}:${String(value % 60).padStart(2, '0')}`; + const last = schedule.local_windows.at(-1); + if (!last) return { start_local_time: '08:00', end_local_time: '12:00' }; + + const first = schedule.local_windows[0]; + const width = 1 + (schedule.occurrences_per_window - 1) * schedule.minimum_separation_minutes; + const start = minute(last.end_local_time) + schedule.minimum_separation_minutes - 1; + const end = start + width; + if ( + end >= 1_440 || + minute(first.start_local_time) + 1_440 - (end - 1) < schedule.minimum_separation_minutes + ) { + return null; + } + return { start_local_time: clock(start), end_local_time: clock(end) }; +} diff --git a/web/src/routes/researcher/scales.ts b/web/src/routes/researcher/scales.ts index 64e6b94..363f081 100644 --- a/web/src/routes/researcher/scales.ts +++ b/web/src/routes/researcher/scales.ts @@ -28,11 +28,9 @@ * 1. Round-trip. `toHuman(toStored(h)) === h`. A value does not change by passing through a box. * 2. Encodable. `toStored(h)` is an integer inside `BOUNDS`, so the canonical encoder's * `-?(0|[1-9][0-9]*)` still matches. The one exception is - * `minimum_displacement_meters`, a Kotlin `Float` written by `formatFloat`, - * where `toStored` returns a float32 and must. + * Location displacement is presented in metres and stored as integer mm. */ -import { formatFloat } from '$lib/adc/canonical'; import { BOUNDS, MAXIMUM_LOCAL_BYTES, @@ -66,11 +64,15 @@ export type { Scale }; export type ScaleKey = | 'sampling_period_us' + | 'ambient_sampling_period_us' | 'maximum_report_latency_us' + | 'change_threshold_millilux' + | 'minimum_event_interval_ms' + | 'change_threshold_millimeters' | 'poll_interval_minutes' | 'interval_millis' | 'maximum_batch_delay_millis' - | 'minimum_displacement_meters' + | 'minimum_displacement_millimeters' | 'trajectory_sampling_hz' | 'duration_hours' | 'maximum_local_bytes' @@ -122,6 +124,19 @@ export function scales(m: Messages, u: Units): Record { format: u.hertz }, + ambient_sampling_period_us: { + box: true, + affix: m.unit.seconds, + toHuman: (stored) => stored / 1_000_000, + toStored: (human) => Math.round(human * 1_000_000), + min: BOUNDS.ambientLightSamplingPeriodUs[0] / 1_000_000, + max: BOUNDS.ambientLightSamplingPeriodUs[1] / 1_000_000, + step: 0.1, + presets: PRESETS.ambient_sampling_period_us, + scale: 'log', + format: u.seconds + }, + /** * A box rather than a ladder, where its neighbour the batch delay is a ladder: the two are the * same kind of decision three orders of magnitude apart, and this one tops out at a minute, so @@ -141,6 +156,45 @@ export function scales(m: Messages, u: Units): Record { format: u.seconds }, + change_threshold_millilux: { + box: true, + affix: m.unit.lux, + toHuman: (stored) => stored / 1_000, + toStored: (human) => Math.round(human * 1_000), + min: BOUNDS.changeThresholdMillilux[0] / 1_000, + max: BOUNDS.changeThresholdMillilux[1] / 1_000, + step: 0.001, + presets: PRESETS.change_threshold_millilux, + scale: 'log', + format: u.lux + }, + + minimum_event_interval_ms: { + box: true, + affix: m.unit.seconds, + toHuman: (stored) => stored / 1_000, + toStored: (human) => Math.round(human * 1_000), + min: BOUNDS.minimumEventIntervalMs[0] / 1_000, + max: BOUNDS.minimumEventIntervalMs[1] / 1_000, + step: 0.1, + presets: PRESETS.minimum_event_interval_ms, + scale: 'log', + format: u.seconds + }, + + change_threshold_millimeters: { + box: true, + affix: m.unit.millimetres, + toHuman: (stored) => stored, + toStored: (human) => human, + min: BOUNDS.changeThresholdMillimeters[0], + max: BOUNDS.changeThresholdMillimeters[1], + step: 1, + presets: PRESETS.change_threshold_millimeters, + scale: 'log', + format: u.millimetres + }, + // 1 min → 1 day. 2 hours is an ordinary poll interval and is on no chip. poll_interval_minutes: laddered( LADDERS.poll_interval_minutes, @@ -159,22 +213,16 @@ export function scales(m: Messages, u: Units): Record { u.millis ), - /** - * The one non-integer, and the one field where the unit was never the problem: metres are - * metres. The gap is float32 — the box takes a double and the file gets a `Float`, so - * `1234.5678` is written `1234.5677`. `toHuman` is the shortest decimal that round-trips to the - * float32 that will actually be written, which is why the box never shows - * `50.099998474121094` and never shows a number the file will not contain. - */ - minimum_displacement_meters: { + /** Metres in the control, exact integer millimetres in Protocol v1. */ + minimum_displacement_millimeters: { box: true, affix: m.unit.metres, - toHuman: (stored) => Number(formatFloat(Math.fround(stored))), - toStored: (human) => Math.fround(human), - min: BOUNDS.minimumDisplacementMeters[0], - max: BOUNDS.minimumDisplacementMeters[1], - step: 0.1, - presets: PRESETS.minimum_displacement_meters, + toHuman: (stored) => stored / 1_000, + toStored: (human) => Math.round(human * 1_000), + min: BOUNDS.minimumDisplacementMillimeters[0] / 1_000, + max: BOUNDS.minimumDisplacementMillimeters[1] / 1_000, + step: 0.001, + presets: PRESETS.minimum_displacement_millimeters, scale: 'log', format: u.metres }, @@ -244,11 +292,15 @@ export function scales(m: Messages, u: Units): Record { */ export const SCALE_BOUNDS: Record = { sampling_period_us: BOUNDS.samplingPeriodUs, + ambient_sampling_period_us: BOUNDS.ambientLightSamplingPeriodUs, maximum_report_latency_us: BOUNDS.maximumReportLatencyUs, + change_threshold_millilux: BOUNDS.changeThresholdMillilux, + minimum_event_interval_ms: BOUNDS.minimumEventIntervalMs, + change_threshold_millimeters: BOUNDS.changeThresholdMillimeters, poll_interval_minutes: BOUNDS.pollIntervalMinutes, interval_millis: [BOUNDS.minimumIntervalMillis[0], BOUNDS.intervalMillis[1]], maximum_batch_delay_millis: BOUNDS.maximumBatchDelayMillis, - minimum_displacement_meters: BOUNDS.minimumDisplacementMeters, + minimum_displacement_millimeters: BOUNDS.minimumDisplacementMillimeters, trajectory_sampling_hz: BOUNDS.trajectorySamplingHz, duration_hours: BOUNDS.durationHours, maximum_local_bytes: [MINIMUM_LOCAL_BYTES, MAXIMUM_LOCAL_BYTES], diff --git a/web/src/routes/researcher/steps.ts b/web/src/routes/researcher/steps.ts index 1f562e3..86b928f 100644 --- a/web/src/routes/researcher/steps.ts +++ b/web/src/routes/researcher/steps.ts @@ -39,7 +39,7 @@ export const STEPS: readonly StepDefinition[] = [ id: 'study', icon: 'document', // Three root keys the study step no longer owns: both identifiers are derived and shown on the - // sign step, and `minimum_app_version` is pinned with no control anywhere. `stepForPath`'s + // sign step, while platform and `minimum_client_version` are pinned with no control anywhere. // `?? 'sign'` fallback routes them to the step that now holds them. paths: [ 'schema_version', diff --git a/web/src/routes/researcher/units.ts b/web/src/routes/researcher/units.ts index abed857..0fd9526 100644 --- a/web/src/routes/researcher/units.ts +++ b/web/src/routes/researcher/units.ts @@ -15,7 +15,6 @@ * entries are byte-identical to what `Intl` emitted, so no rendered string moved. */ -import { formatFloat } from '$lib/adc/canonical'; import { binaryBytes } from '$lib/ui/format'; import type { Locale, Messages } from '$lib/i18n/types'; @@ -27,6 +26,8 @@ export interface Units { hours(value: number): string; hertz(value: number): string; metres(value: number): string; + millimetres(value: number): string; + lux(value: number): string; bytes(value: number): string; /** One significant figure above ten, because the constants behind it are that good and no better. */ count(value: number): string; @@ -90,9 +91,9 @@ export function units(m: Messages, locale: Locale): Units { // The app shows a whole number of hertz because Android delivers at least that rate, never // less; a fractional rate would suggest a precision the sensor does not offer. hertz: (value) => `${value} ${m.unit.hertz}`, - // The float32 that will actually be written, not the double that was typed: `1234.5678` - // displays as `1234.5677`, and discovering that at diff time is worse than seeing it here. - metres: (value) => `${formatFloat(value)} ${m.unit.metres}`, + metres: (value) => `${number.format(value)} ${m.unit.metres}`, + millimetres: (value) => `${number.format(value)} ${m.unit.millimetres}`, + lux: (value) => `${number.format(value)} ${m.unit.lux}`, bytes: (value) => binaryBytes(value), count: (value) => number.format(coarse(value)), about: (value) => `≈ ${value}` diff --git a/web/tests/bundle.spec.ts b/web/tests/bundle.spec.ts index b356ae7..2fb9bc6 100644 --- a/web/tests/bundle.spec.ts +++ b/web/tests/bundle.spec.ts @@ -1,263 +1,206 @@ -/** - * The reader, against bundles built for the occasion. - * - * Every case here starts from a fresh HPKE key pair and a bundle sealed to it, because the one - * thing that must never be committed to this repository is a key that opens a real export. See - * `tests/seal.ts` for why the fixture is built rather than shelled out for, and `tests/compat.spec.ts` - * for the half of this claim that involves the JVM. - * - * The negative cases are the point. A researcher meets this step holding three files, and the only - * useful failure is one that names which of the three is wrong — so each case here changes exactly - * one thing and asserts the name that comes back. - */ - import { describe, expect, it } from 'vitest'; -import { MAXIMUM_BUNDLE_BYTES, openBundle, type BundleFailure } from '../src/lib/adc/bundle'; -import { generateHpkeKeyset } from '../src/lib/adc/tink'; -import type { StudyConfiguration } from '../src/lib/adc/types'; -import { bundleJson, seal } from './seal'; - -/** - * A study and the key pair it seals to. Fresh on every call, because the one file that must never - * exist in this repository is a key that opens a real export. - */ -function pair(overrides: Partial = {}) { - const keyset = generateHpkeKeyset(); - const configuration: StudyConfiguration = { - schema_version: 1, - experiment_id: 'bundle-harness', - configuration_id: 'bundle-case-001', - assigned_participant_id: null, - issued_at: '2026-01-01T00:00:00Z', - expires_at: '2035-01-01T00:00:00Z', - minimum_app_version: 1, - title: 'Bundle harness 研究', - researcher: { name: 'Harness', contact: 'harness@example.invalid' }, - purpose: 'Prove the browser opens what a phone wrote.', - duration_hours: 24, - consent: { document_version: 'harness-1', summary: 'A fixture. Nothing is collected.' }, - collectors: [{ id: 'app_lifecycle.v1', required: true, config: {} }], - surveys: [], - interventions: [], - storage: { maximum_local_bytes: 16 * 1024 * 1024 }, - signer: { key_id: 'harness-signer', public_key: '' }, - export: { researcher_key_id: 'harness-hpke', tink_hpke_public_keyset: keyset.publicKeyset }, - upload: null, - ...overrides - }; - return { keyset, configuration }; -} - -async function refuses( - bytes: Uint8Array, - configuration: StudyConfiguration, - privateKeyset: Parameters[2], - failure: BundleFailure -) { - expect(await openBundle(bytes, configuration, privateKeyset)).toEqual({ ok: false, failure }); -} - -describe('openBundle', () => { - it('opens a bundle sealed to the study, byte for byte', async () => { - const { keyset, configuration } = pair(); - const text = bundleJson(configuration, { events: 3, assignedParticipantId: 'A-017' }); - const result = await openBundle( - await seal(configuration, text), - configuration, - keyset.privateKeyset +import { bundleContext, openBundle, type ResearchDocument } from '../src/lib/adc/bundle'; +import { canonicalize } from '../src/lib/adc/canonical'; +import { generateHpkeKeyPair } from '../src/lib/adc/crypto'; +import { HPKE, SIGNING, validConfiguration } from './fixture'; +import { sealBundle } from './seal'; + +const clone = (value: T): T => JSON.parse(JSON.stringify(value)); + +describe('ADCEXP01 Protocol v1 reader', () => { + it('opens the fixed RFC 9180 framing and preserves decimal 64-bit values', async () => { + const configuration = validConfiguration(); + const bytes = await sealBundle(configuration, SIGNING.privateKey); + expect(new TextDecoder().decode(bytes.subarray(0, 8))).toBe('ADCEXP01'); + expect(new DataView(bytes.buffer).getUint16(56)).toBe( + new TextEncoder().encode(configuration.export.researcher_key_id).length ); - if (!result.ok) throw new Error(`refused: ${result.failure}`); - expect(result.bundle.text).toBe(text); - expect(result.bundle.keyId).toBe('harness-hpke'); - expect(result.bundle.bytes).toBe(new TextEncoder().encode(text).length); - const experiment = result.bundle.document.experiment; - expect(experiment.events).toHaveLength(3); - expect(experiment.events[0].fields).toEqual({ x: '0.2', y: '-1.2', z: '9.82' }); - expect(experiment.assigned_participant_id).toBe('A-017'); - expect(experiment.first_sequence_number).toBe(1); - expect(experiment.last_sequence_number).toBe(3); - }); - /** The anonymous study writes no such key, and an absent key is not a malformed bundle. */ - it('reads an absent assigned participant as none', async () => { - const { keyset, configuration } = pair(); - const text = bundleJson(configuration, { events: 0 }); - // The configuration inlined alongside carries its own `assigned_participant_id: null`, so the - // absence being asserted is the experiment block's, which is where the phone omits the key. - expect(Object.keys(JSON.parse(text).experiment)).not.toContain('assigned_participant_id'); - const result = await openBundle( - await seal(configuration, text), - configuration, - keyset.privateKeyset + const result = await openBundle(bytes, configuration, HPKE.privateKey); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.bundle.document.bundle_id).toBe('00112233-4455-4677-8899-aabbccddeeff'); + expect(result.bundle.document.experiment.events[0].observed_time.monotonic_time_nanos).toBe( + '9007199254740993' ); - if (!result.ok) throw new Error(`refused: ${result.failure}`); - expect(result.bundle.document.experiment.assigned_participant_id).toBeNull(); - expect(result.bundle.document.experiment.events).toEqual([]); - expect(result.bundle.document.experiment.last_sequence_number).toBe(0); - }); - - /** - * The Kotlin side reads the body in 64 KiB chunks and calls `doFinal` once, so a bundle spanning - * several of those windows still carries exactly one tag. A reader that expected a frame per chunk - * would open the first window of this and nothing else. - */ - it('opens a body several read windows wide with one tag', async () => { - const { keyset, configuration } = pair(); - const text = bundleJson(configuration, { events: 1_000 }); - expect(text.length).toBeGreaterThan(4 * 65_536); - const bytes = await seal(configuration, text); - const result = await openBundle(bytes, configuration, keyset.privateKeyset); - if (!result.ok) throw new Error(`refused: ${result.failure}`); - expect(result.bundle.text).toBe(text); - // Body minus plaintext is one 16-byte tag, whatever the size. - const header = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - const bodyAt = 26 + header.getUint16(8) + header.getInt32(10); - expect(bytes.length - bodyAt - result.bundle.bytes).toBe(16); + expect(result.bundle.document.experiment.event_count).toBe('2'); + expect(canonicalize(JSON.parse(result.bundle.text))).toBe(result.bundle.text); }); - it('carries the whole-study count beside the window this file holds', async () => { - const { keyset, configuration } = pair(); - // A scheduled upload sends a slice: sequences 501–503 out of 900 the device has recorded. - const text = bundleJson(configuration, { events: 3, firstSequenceNumber: 501, lifetime: 900 }); - const result = await openBundle( - await seal(configuration, text), - configuration, - keyset.privateKeyset + it('uses one JCS context for HPKE info and content AAD', () => { + expect( + new TextDecoder().decode( + bundleContext( + '00112233-4455-4677-8899-aabbccddeeff', + '00'.repeat(32), + 'protocol-export' + ) + ) + ).toBe( + '{"bundle_format":"research-bundle-v1","bundle_id":"00112233-4455-4677-8899-aabbccddeeff","configuration_sha256":"' + + '00'.repeat(32) + + '","researcher_key_id":"protocol-export"}' ); - if (!result.ok) throw new Error(`refused: ${result.failure}`); - expect(result.bundle.document.experiment.first_sequence_number).toBe(501); - expect(result.bundle.document.experiment.last_sequence_number).toBe(503); - expect(result.bundle.document.experiment.next_sequence_number).toBe(901); }); - describe('refuses, by name', () => { - it('a file that is not a bundle', async () => { - const { keyset, configuration } = pair(); - await refuses( - new TextEncoder().encode('{"not":"a bundle"}'), - configuration, - keyset.privateKeyset, - 'not_a_bundle' - ); - }); - - it('a header with nothing after it', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - await refuses(bytes.slice(0, 20), configuration, keyset.privateKeyset, 'not_a_bundle'); - await refuses(bytes.slice(0, 40), configuration, keyset.privateKeyset, 'not_a_bundle'); - }); - - /** A length that lies is refused before it is used to slice anything. */ - it('a key id length outside the writer’s bounds', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); - view.setUint16(8, 2); - await refuses(bytes, configuration, keyset.privateKeyset, 'not_a_bundle'); - view.setUint16(8, 65); - await refuses(bytes, configuration, keyset.privateKeyset, 'not_a_bundle'); - }); - - it('a wrapped key length read as a negative int32', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength).setUint32(10, 0xffff_ffff); - await refuses(bytes, configuration, keyset.privateKeyset, 'not_a_bundle'); - }); - - it('a file larger than this tab opens', async () => { - const { keyset, configuration } = pair(); - // Only the length is inspected before the ceiling, so a sparse array proves the ordering - // without allocating a quarter of a gigabyte of real ciphertext. - const huge = { length: MAXIMUM_BUNDLE_BYTES + 1 } as unknown as Uint8Array; - await refuses(huge, configuration, keyset.privateKeyset, 'too_large'); - }); + it.each(['', ' ', '0x10', '0b10', 'NaN', 'Infinity'])( + 'rejects a non-decimal sensor float spelling: %j', + async (hostile) => { + const base = validConfiguration(); + const configuration = validConfiguration({ + collectors: [ + ...base.collectors, + { + id: 'gyroscope.v1', + required: false, + config: { maximum_report_latency_us: 1_000_000, sampling_period_us: 20_000 } + } + ] + }); + const bytes = await sealBundle(configuration, SIGNING.privateKey, { + document: (value) => { + const changed = clone(value); + changed.experiment.events[0] = { + sequence_number: '1', + collector_id: 'gyroscope.v1', + payload_schema_version: 1, + observed_time: changed.experiment.events[0].observed_time, + payload_type: 'GYROSCOPE_SAMPLE', + fields: { + accuracy: '3', + source_elapsed_realtime_nanos: '1000000000', + x_radians_per_second: hostile, + y_radians_per_second: '0.2', + z_radians_per_second: '0.3' + } + }; + return changed; + } + }); - /** The commonest mistake of the three, and the CLI names it before touching any crypto too. */ - it('a bundle from another study', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 }), { - keyId: 'other-hpke' + expect(await openBundle(bytes, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'unreadable' }); - await refuses(bytes, configuration, keyset.privateKeyset, 'wrong_study'); + } + ); + + it('distinguishes wrong configuration, wrong key, HPKE corruption, and body corruption', async () => { + const configuration = validConfiguration(); + const valid = await sealBundle(configuration, SIGNING.privateKey); + + expect(await openBundle(valid, { ...configuration, configuration_id: 'other-config' }, HPKE.privateKey)) + .toEqual({ ok: false, failure: 'wrong_study' }); + expect(await openBundle(valid, configuration, generateHpkeKeyPair().privateKey)).toEqual({ + ok: false, + failure: 'wrong_key' }); - it('last month’s private key', async () => { - const { configuration } = pair(); - const stale = generateHpkeKeyset(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - await refuses(bytes, configuration, stale.privateKeyset, 'wrong_key'); + const wrapped = valid.slice(); + const wrappedAt = 70 + new DataView(valid.buffer).getUint16(56); + wrapped[wrappedAt + 40] ^= 1; + expect(await openBundle(wrapped, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'unwrap_failed' }); - /** - * The same key id in front of a different scalar. The prefix check passes and the derived - * public key does not match the study's, which is the check that catches a keyset somebody - * renumbered by hand. - */ - it('a private key whose id matches but whose scalar does not', async () => { - const { keyset, configuration } = pair(); - const impostor = generateHpkeKeyset(); - impostor.privateKeyset.primaryKeyId = keyset.publicKeyset.primaryKeyId; - impostor.privateKeyset.key[0].keyId = keyset.publicKeyset.primaryKeyId; - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - await refuses(bytes, configuration, impostor.privateKeyset, 'wrong_key'); + const body = valid.slice(); + body[body.length - 1] ^= 1; + expect(await openBundle(body, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'tag_failed' }); + }); - it('a keyset naming another suite', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - const broken = structuredClone(keyset.privateKeyset); - broken.key[0].keyData.typeUrl = 'type.googleapis.com/google.crypto.tink.EciesPrivateKey'; - await refuses(bytes, configuration, broken, 'wrong_key'); + it.each([ + { + name: 'unknown root member', + mutate: (value: ResearchDocument) => ({ ...value, future: true }) + }, + { + name: 'numeric sequence instead of decimal string', + mutate: (value: ResearchDocument) => { + const changed = clone(value) as unknown as { experiment: { events: Array<{ sequence_number: unknown }> } }; + changed.experiment.events[0].sequence_number = 1; + return changed; + } + }, + { + name: 'conflicting event count', + mutate: (value: ResearchDocument) => { + const changed = clone(value); + changed.experiment.event_count = '3'; + return changed; + } + }, + { + name: 'wrong actual range', + mutate: (value: ResearchDocument) => { + const changed = clone(value); + changed.experiment.first_sequence_number = '0'; + return changed; + } + }, + { + name: 'non-contiguous event range', + mutate: (value: ResearchDocument) => { + const changed = clone(value); + changed.experiment.events[1].sequence_number = '3'; + changed.experiment.last_sequence_number = '3'; + return changed; + } + }, + { + name: 'empty automatic upload', + mutate: (value: ResearchDocument) => { + const changed = clone(value); + changed.bundle_kind = 'automatic_upload'; + changed.experiment.events = []; + changed.experiment.event_count = '0'; + changed.experiment.first_sequence_number = '3'; + changed.experiment.last_sequence_number = '2'; + return changed; + } + }, + { + name: 'old padded signature encoding', + mutate: (value: ResearchDocument) => { + const changed = clone(value); + changed.configuration_signature.signature += '=='; + return changed; + } + } + ])('fails closed on $name', async ({ mutate }) => { + const configuration = validConfiguration(); + const bytes = await sealBundle(configuration, SIGNING.privateKey, { document: mutate }); + expect(await openBundle(bytes, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'unreadable' }); + }); - /** - * A personalised study issues one configuration per participant, so this is the shape of - * holding the wrong one. The context is the wrap's `info` as well as the body's AAD, and the - * wrap opens first — which is why this is `unwrap_failed` rather than a body tag failure. - */ - it('another participant’s configuration', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 1 })); - const other = { ...configuration, configuration_id: 'bundle-case-002' }; - await refuses(bytes, other, keyset.privateKeyset, 'unwrap_failed'); + it('rejects old or truncated v1 framing rather than sniffing a fallback', async () => { + const oldHeader = new Uint8Array(64); + oldHeader.set(new TextEncoder().encode('ADCEXP01')); + expect(await openBundle(oldHeader, validConfiguration(), HPKE.privateKey)).toEqual({ + ok: false, + failure: 'not_a_bundle' }); + }); - it('a bundle altered after the phone wrote it', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 3 })); - bytes[bytes.length - 1] ^= 1; - await refuses(bytes, configuration, keyset.privateKeyset, 'tag_failed'); + it('rejects a non-random bundle UUID and a noncanonical key ID before decryption', async () => { + const configuration = validConfiguration(); + const invalidUuid = await sealBundle(configuration, SIGNING.privateKey, { + bundleId: '00112233-4455-0677-8899-aabbccddeeff' }); - - it('a plaintext that is not this format', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, JSON.stringify({ format: 'something-else' })); - await refuses(bytes, configuration, keyset.privateKeyset, 'unreadable'); + expect(await openBundle(invalidUuid, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'not_a_bundle' }); - - /** The summary maps over `events`; a document that cannot be mapped over is refused whole. */ - it('a document whose events are not events', async () => { - const { keyset, configuration } = pair(); - const document = JSON.parse(bundleJson(configuration, { events: 1 })); - document.experiment.events[0].fields.x = 0.2; - await refuses( - await seal(configuration, JSON.stringify(document)), - configuration, - keyset.privateKeyset, - 'unreadable' - ); + const invalidKeyId = await sealBundle(configuration, SIGNING.privateKey, { keyId: 'Bad' }); + expect(await openBundle(invalidKeyId, configuration, HPKE.privateKey)).toEqual({ + ok: false, + failure: 'not_a_bundle' }); }); - - /** Nothing renders before the tag verifies, so nothing partial may come back either. */ - it('returns no document on any failure', async () => { - const { keyset, configuration } = pair(); - const bytes = await seal(configuration, bundleJson(configuration, { events: 2 })); - bytes[bytes.length - 2] ^= 0x40; - const result = await openBundle(bytes, configuration, keyset.privateKeyset); - expect(result.ok).toBe(false); - expect(result).not.toHaveProperty('bundle'); - }); }); diff --git a/web/tests/canonical.spec.ts b/web/tests/canonical.spec.ts index f96cd02..62b04db 100644 --- a/web/tests/canonical.spec.ts +++ b/web/tests/canonical.spec.ts @@ -1,369 +1,76 @@ -/** - * Every string asserted here came out of the other implementation, not out of this one. - * - * The float renderings are `Float.toString` on a real JDK, checked in bulk over two million values - * before the interesting ones were written down; the whole-document snapshots are what - * `researcher-tools canonicalize` writes for the same input. Nothing here proves the encoder agrees - * with itself — a test that did would pass just as happily on bytes no device accepts. - */ - import { describe, expect, it } from 'vitest'; import { - canonicalBytes, + canonicalConfigurationBytes, canonicalize, - escapeJsonString, - formatFloat, + canonicalizeConfiguration, formatInstant, + parseCanonicalJson, parseInstant } from '../src/lib/adc/canonical'; -import type { CollectorConfig, StudyConfiguration, TinkKeyset } from '../src/lib/adc/types'; - -/** - * The demonstration keyset from `researcher-tools/examples`. It is a real, working keyset, which - * makes it the right fixture for the one field this encoder re-emits rather than composes. - */ -const DEMO_KEYSET: TinkKeyset = { - primaryKeyId: 218992727, - key: [ - { - keyData: { - typeUrl: 'type.googleapis.com/google.crypto.tink.HpkePublicKey', - value: 'EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA', - keyMaterialType: 'ASYMMETRIC_PUBLIC' - }, - status: 'ENABLED', - keyId: 218992727, - outputPrefixType: 'TINK' - } - ] -}; - -const DEMO_SUMMARY = - "This demonstration can collect precise location, motion, network state, aggregate Wi-Fi/mobile usage, app and screen usage events, this app's lifecycle, and touch dynamics made inside the optional research keyboard. It never records keyboard text. Data stays encrypted on this device until you choose Export. You can pause, withdraw, export repeatedly, or delete local data."; - -const DEMO_PUBLIC_KEY = 'MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0='; - -/** `researcher-tools/examples/demo-study.json`, as the site would hold it. */ -const demoStudy: StudyConfiguration = { - schema_version: 1, - experiment_id: 'modular-sensing-demo', - configuration_id: 'demo-config-2026', - assigned_participant_id: null, - issued_at: '2026-01-01T00:00:00Z', - expires_at: '2035-01-01T00:00:00Z', - minimum_app_version: 1, - title: 'Modular sensing demonstration', - researcher: { - name: 'Android Data Collector maintainers', - contact: 'research@example.invalid' - }, - purpose: - 'Verify the complete on-device collection, pause, export, and researcher-decryption loop.', - duration_hours: 24, - consent: { document_version: 'demo-1', summary: DEMO_SUMMARY }, - collectors: [ - { id: 'app_lifecycle.v1', required: true, config: {} }, - { - id: 'accelerometer.v1', - required: true, - config: { sampling_period_us: 100000, maximum_report_latency_us: 1000000 } - }, - { id: 'network_state.v1', required: true, config: { include_bandwidth_estimates: true } }, - { - id: 'network_usage.v1', - required: false, - config: { transports: ['mobile', 'wifi'], poll_interval_minutes: 5 } - }, - { id: 'usage_events.v1', required: false, config: { poll_interval_minutes: 15 } }, - { - id: 'location.v1', - required: false, - config: { - interval_millis: 10000, - minimum_interval_millis: 5000, - maximum_batch_delay_millis: 30000, - minimum_displacement_meters: 5, - priority: 'BALANCED' - } - }, - { id: 'keyboard_touch.v1', required: false, config: { trajectory_sampling_hz: 60 } } - ], - surveys: [ - { - id: 'demo-survey', - title: { default: 'Study check-in', translations: { 'zh-TW': '研究確認' } }, - description: { default: 'Tell us how the study is going.', translations: {} }, - questions: [ - { - type: 'short_text', - id: 'status-note', - prompt: { default: 'How is it going?', translations: {} }, - required: false, - maximum_length: 500 - } - ] - } - ], - interventions: [ - { - id: 'demo-check-in', - action: { - type: 'survey', - notification_title: 'Study check-in', - notification_message: 'Please complete the study check-in.', - survey_id: 'demo-survey' - }, - triggers: [ - { - id: 'after-one-hour', - schedule: { type: 'one_time', offset_minutes: 60, clock: 'ACTIVE_RUNNING_TIME' }, - availability_minutes: 1440 - } - ] - } - ], - storage: { maximum_local_bytes: 16777216 }, - signer: { key_id: 'demo-signer-2026', public_key: DEMO_PUBLIC_KEY }, - export: { researcher_key_id: 'demo-hpke-2026', tink_hpke_public_keyset: DEMO_KEYSET }, - upload: null -}; - -function withCollectors(collectors: CollectorConfig[]): StudyConfiguration { - return { ...demoStudy, collectors }; -} - -describe('formatFloat', () => { - it('reproduces the renderings measured against the CLI', () => { - expect(formatFloat(0)).toBe('0.0'); - expect(formatFloat(5)).toBe('5.0'); - expect(formatFloat(100)).toBe('100.0'); - expect(formatFloat(0.25)).toBe('0.25'); - expect(formatFloat(0.1)).toBe('0.1'); - expect(formatFloat(0.3)).toBe('0.3'); - expect(formatFloat(9999.999)).toBe('9999.999'); - expect(formatFloat(1234.5678)).toBe('1234.5677'); - }); - - it('always writes a fractional digit, and keeps the sign of zero', () => { - expect(formatFloat(10000)).toBe('10000.0'); - expect(formatFloat(1)).toBe('1.0'); - expect(formatFloat(-0)).toBe('-0.0'); - expect(formatFloat(-2.5)).toBe('-2.5'); - expect(formatFloat(-0.1)).toBe('-0.1'); - }); - - it('breaks a tie toward the even digit, as the JDK does', () => { - expect(formatFloat(4618.53125)).toBe('4618.5312'); - expect(formatFloat(6806.40625)).toBe('6806.4062'); - expect(formatFloat(0.03125)).toBe('0.03125'); - }); - - it('prefers two digits over one where a two-digit decimal is closer', () => { - // Float.MIN_VALUE. `1.0E-45` also round-trips, and is not what Java writes. - expect(formatFloat(1.401298464324817e-45)).toBe('1.4E-45'); +import { validConfiguration } from './fixture'; + +describe('RFC 8785 JCS', () => { + it('sorts recursively by UTF-16 member names and uses ECMAScript primitives', () => { + expect( + canonicalize({ z: 1, a: { '\u20ac': 'Euro', '\r': 'CR', '1': true }, n: -0 }) + ).toBe('{"a":{"\\r":"CR","1":true,"€":"Euro"},"n":0,"z":1}'); + expect(canonicalize({ numbers: [333333333.33333329, 1e30, 4.5, 2e-3, 1e-27] })).toBe( + '{"numbers":[333333333.3333333,1e+30,4.5,0.002,1e-27]}' + ); }); - it('switches to exponent form exactly where Java does', () => { - expect(formatFloat(0.001)).toBe('0.001'); - expect(formatFloat(0.0001)).toBe('1.0E-4'); - expect(formatFloat(9999999)).toBe('9999999.0'); - expect(formatFloat(1e7)).toBe('1.0E7'); - expect(formatFloat(3.4028235e38)).toBe('3.4028235E38'); + it('rejects values outside I-JSON instead of repairing them', () => { + expect(() => canonicalize({ value: Number.NaN })).toThrow('jcs_number'); + expect(() => canonicalize({ value: BigInt(1) })).toThrow('jcs_type'); + expect(() => canonicalize({ value: '\ud800' })).toThrow('jcs_unicode'); + const cycle: Record = {}; + cycle.self = cycle; + expect(() => canonicalize(cycle)).toThrow('jcs_cycle'); }); - it('round-trips every rendering back to the same float32', () => { - for (let step = 0; step <= 400; step++) { - const value = (step / 400) * 10_000; - expect(Math.fround(Number(formatFloat(value)))).toBe(Math.fround(value)); - expect(formatFloat(value)).toMatch(/^-?(\d+\.\d+|\d\.\d+E-?\d+)$/); + it('accepts only byte-for-byte canonical UTF-8 JSON', () => { + expect(parseCanonicalJson(new TextEncoder().encode('{"a":1,"b":2}'))).toEqual({ a: 1, b: 2 }); + for (const hostile of [' {"a":1}', '{"b":2,"a":1}', '{"a":1,"a":1}', '{"a":1.0}']) { + expect(() => parseCanonicalJson(new TextEncoder().encode(hostile))).toThrow(); } - }); - - it('is shortest for a float32, not for the double holding it', () => { - // What `Number.prototype.toString` would have written for the same value. - expect(String(Math.fround(0.1))).toBe('0.10000000149011612'); - expect(formatFloat(0.1)).toBe('0.1'); - expect(String(Math.fround(1234.5678))).toBe('1234.5677490234375'); - expect(formatFloat(1234.5678)).toBe('1234.5677'); - }); -}); - -describe('escapeJsonString', () => { - it('uses Gson’s default table, not the HTML-safe one', () => { - expect(escapeJsonString('a"b')).toBe('a\\"b'); - expect(escapeJsonString('a\\b')).toBe('a\\\\b'); - expect(escapeJsonString('\b\f\n\r\t')).toBe('\\b\\f\\n\\r\\t'); - expect(escapeJsonString('\u0000')).toBe('\\u0000'); - expect(escapeJsonString('\u000b')).toBe('\\u000b'); - expect(escapeJsonString('\u001f')).toBe('\\u001f'); - expect(escapeJsonString('\u2028\u2029')).toBe('\\u2028\\u2029'); - }); - - it('leaves alone everything an HTML-safe writer would have escaped', () => { - expect(escapeJsonString(' & b = \'c\'')).toBe(' & b = \'c\''); - expect(escapeJsonString('\u007f')).toBe('\u007f'); - expect(escapeJsonString('€é')).toBe('€é'); - expect(escapeJsonString('研究 \u{1f512}')).toBe('研究 \u{1f512}'); - }); - - it('returns the input untouched when there is nothing to escape', () => { - expect(escapeJsonString('')).toBe(''); - expect(escapeJsonString('plain')).toBe('plain'); - }); -}); - -describe('parseInstant', () => { - it('re-spells any accepted instant the way Instant.toString would', () => { - const respell = (text: string) => formatInstant(parseInstant(text)!); - expect(respell('2026-01-01T00:00:00Z')).toBe('2026-01-01T00:00:00Z'); - expect(respell('2026-01-01T00:00:00.000Z')).toBe('2026-01-01T00:00:00Z'); - expect(respell('2026-01-01T00:00:00.120Z')).toBe('2026-01-01T00:00:00.120Z'); - expect(respell('2026-01-01T00:00:00.000001Z')).toBe('2026-01-01T00:00:00.000001Z'); - expect(respell('2026-01-01T00:00:00.000000001Z')).toBe('2026-01-01T00:00:00.000000001Z'); - expect(respell('2026-01-01T08:30:00+08:00')).toBe('2026-01-01T00:30:00Z'); - expect(respell('2024-02-29T23:59:59Z')).toBe('2024-02-29T23:59:59Z'); - }); - - it('refuses what Instant.parse refuses', () => { - expect(parseInstant('')).toBeNull(); - expect(parseInstant('2026-01-01')).toBeNull(); - expect(parseInstant('2026-01-01T00:00Z')).toBeNull(); - expect(parseInstant('2026-01-01T00:00:00')).toBeNull(); - expect(parseInstant('2026-02-30T00:00:00Z')).toBeNull(); - expect(parseInstant('2023-02-29T00:00:00Z')).toBeNull(); - expect(parseInstant('2026-13-01T00:00:00Z')).toBeNull(); - expect(parseInstant('2026-01-01T24:00:00Z')).toBeNull(); + expect(() => parseCanonicalJson(Uint8Array.of(0x7b, 0x22, 0xff, 0x22, 0x3a, 0x31, 0x7d))).toThrow(); }); }); -describe('canonicalize', () => { - it('matches the demonstration study byte for byte', () => { - expect(canonicalize(demoStudy)).toBe( - '{"schema_version":1' + - ',"experiment_id":"modular-sensing-demo"' + - ',"configuration_id":"demo-config-2026"' + - ',"assigned_participant_id":null' + - ',"issued_at":"2026-01-01T00:00:00Z"' + - ',"expires_at":"2035-01-01T00:00:00Z"' + - ',"minimum_app_version":1' + - ',"title":"Modular sensing demonstration"' + - ',"researcher":{"name":"Android Data Collector maintainers"' + - ',"contact":"research@example.invalid"}' + - ',"purpose":"Verify the complete on-device collection, pause, export, and researcher-decryption loop."' + - ',"duration_hours":24' + - ',"consent":{"document_version":"demo-1","summary":"' + - DEMO_SUMMARY + - '"}' + - ',"collectors":[' + - '{"id":"app_lifecycle.v1","required":true,"config":{}}' + - ',{"id":"accelerometer.v1","required":true,"config":{"sampling_period_us":100000,"maximum_report_latency_us":1000000}}' + - ',{"id":"network_state.v1","required":true,"config":{"include_bandwidth_estimates":true}}' + - ',{"id":"network_usage.v1","required":false,"config":{"transports":["mobile","wifi"],"poll_interval_minutes":5}}' + - ',{"id":"usage_events.v1","required":false,"config":{"poll_interval_minutes":15}}' + - ',{"id":"location.v1","required":false,"config":{"interval_millis":10000,"minimum_interval_millis":5000,"maximum_batch_delay_millis":30000,"minimum_displacement_meters":5.0,"priority":"BALANCED"}}' + - ',{"id":"keyboard_touch.v1","required":false,"config":{"trajectory_sampling_hz":60}}' + - ']' + - ',"surveys":[{"id":"demo-survey","title":{"default":"Study check-in","translations":{"zh-TW":"研究確認"}},"description":{"default":"Tell us how the study is going.","translations":{}},"questions":[{"type":"short_text","id":"status-note","prompt":{"default":"How is it going?","translations":{}},"required":false,"maximum_length":500}]}]' + - ',"interventions":[{"id":"demo-check-in","action":{"type":"survey","notification_title":"Study check-in","notification_message":"Please complete the study check-in.","survey_id":"demo-survey"},"triggers":[{"id":"after-one-hour","schedule":{"type":"one_time","offset_minutes":60,"clock":"ACTIVE_RUNNING_TIME"},"availability_minutes":1440}]}]' + - ',"storage":{"maximum_local_bytes":16777216}' + - ',"signer":{"key_id":"demo-signer-2026","public_key":"' + - DEMO_PUBLIC_KEY + - '"}' + - ',"export":{"researcher_key_id":"demo-hpke-2026","tink_hpke_public_keyset":' + - '{"primaryKeyId":218992727,"key":[{"keyData":{"typeUrl":"type.googleapis.com/google.crypto.tink.HpkePublicKey"' + - ',"value":"EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA"' + - ',"keyMaterialType":"ASYMMETRIC_PUBLIC"},"status":"ENABLED","keyId":218992727,"outputPrefixType":"TINK"}]}}' + - ',"upload":{}}' - ); - }); - - it('writes a populated upload block, and text exactly as it was written', () => { - const configuration: StudyConfiguration = { - ...demoStudy, - title: '心理韌性研究 "2026"', - purpose: 'Line one\nline two\ttabbed', - researcher: { name: 'Lin\\Chen', contact: 'lab@example.invalid' }, - assigned_participant_id: 'Cohort_A-017', - surveys: [], - interventions: [], - collectors: [{ id: 'app_lifecycle.v1', required: true, config: {} }], - upload: { - endpoint: 'https://intake.example.invalid/v1/bundles?study=1', - interval_minutes: 720, - allow_metered: false - } - }; - const canonical = canonicalize(configuration); - expect(canonical).toContain('"title":"心理韌性研究 \\"2026\\""'); - expect(canonical).toContain('"purpose":"Line one\\nline two\\ttabbed"'); - expect(canonical).toContain('"name":"Lin\\\\Chen"'); - expect(canonical).toContain('"assigned_participant_id":"Cohort_A-017"'); - expect(canonical).toContain('"surveys":[]'); - expect(canonical).toContain('"interventions":[]'); - expect(canonical).toContain( - '"upload":{"endpoint":"https://intake.example.invalid/v1/bundles?study=1","interval_minutes":720,"allow_metered":false}' - ); - expect(canonical.endsWith('}')).toBe(true); - expect(JSON.parse(canonical).title).toBe(configuration.title); - }); - - it('sorts transports by their Kotlin enum name and drops duplicates', () => { - const encoded = (transports: ('mobile' | 'wifi')[]) => - canonicalize( - withCollectors([ - { - id: 'network_usage.v1', - required: false, - config: { transports, poll_interval_minutes: 30 } - } - ]) - ); - expect(encoded(['wifi', 'mobile'])).toContain('"transports":["mobile","wifi"]'); - expect(encoded(['mobile'])).toContain('"transports":["mobile"]'); - expect(encoded(['wifi', 'wifi'])).toContain('"transports":["wifi"]'); - expect(encoded([])).toContain('"transports":[]'); - }); - - it('re-spells the validity window, so a browser-shaped instant still signs correctly', () => { - const canonical = canonicalize({ - ...demoStudy, - issued_at: new Date(Date.UTC(2026, 0, 1)).toISOString(), - expires_at: '2026-04-01T08:00:00+08:00' +describe('Protocol v1 configuration value', () => { + it('uses JCS, an empty upload object, raw keys, decimal client version, and integer millimetres', () => { + const configuration = validConfiguration(); + const text = canonicalizeConfiguration(configuration); + expect(text).toBe(new TextDecoder().decode(canonicalConfigurationBytes(configuration))); + expect(text).toContain('"minimum_client_version":"1"'); + expect(text).toContain('"minimum_displacement_millimeters":25000'); + expect(text).toContain('"platform":"android"'); + expect(text).toContain('"upload":{}'); + expect(text).toContain(`"hpke_public_key":"${configuration.export.hpke_public_key}"`); + expect(text).not.toMatch(/tink|minimum_app_version|minimum_displacement_meters/); + }); + + it('normalizes set-like transports and instants exactly as the Android codec', () => { + const configuration = validConfiguration({ + issued_at: '2026-01-01T08:00:00+08:00', + collectors: [ + { + id: 'network_usage.v1', + required: false, + config: { transports: ['wifi', 'mobile', 'wifi'], poll_interval_minutes: 15 } + } + ] }); - expect(canonical).toContain('"issued_at":"2026-01-01T00:00:00Z"'); - expect(canonical).toContain('"expires_at":"2026-04-01T00:00:00Z"'); - }); - - it('carries a location displacement through Float.toString', () => { - const encoded = (minimum_displacement_meters: number) => - canonicalize( - withCollectors([ - { - id: 'location.v1', - required: false, - config: { - interval_millis: 60000, - minimum_interval_millis: 30000, - maximum_batch_delay_millis: 0, - minimum_displacement_meters, - priority: 'HIGH_ACCURACY' - } - } - ]) - ); - expect(encoded(0)).toContain('"minimum_displacement_meters":0.0,"priority":"HIGH_ACCURACY"'); - expect(encoded(0.1)).toContain('"minimum_displacement_meters":0.1'); - expect(encoded(1234.5678)).toContain('"minimum_displacement_meters":1234.5677'); + const text = canonicalizeConfiguration(configuration); + expect(text).toContain('"issued_at":"2026-01-01T00:00:00Z"'); + expect(text).toContain('"transports":["mobile","wifi"]'); }); }); -describe('canonicalBytes', () => { - it('is UTF-8, so a CJK document is longer in bytes than in characters', () => { - const configuration = { ...demoStudy, title: '研究' }; - const bytes = canonicalBytes(configuration); - expect(bytes).toBeInstanceOf(Uint8Array); - expect(bytes).toEqual(new TextEncoder().encode(canonicalize(configuration))); - expect(bytes.length).toBeGreaterThan(canonicalize(configuration).length); - expect(new TextDecoder().decode(bytes)).toBe(canonicalize(configuration)); +describe('instant authoring', () => { + it('parses offsets and emits canonical UTC', () => { + expect(formatInstant(parseInstant('2026-01-01T08:00:00+08:00')!)).toBe('2026-01-01T00:00:00Z'); + expect(parseInstant('2026-02-30T00:00:00Z')).toBeNull(); + expect(parseInstant('2026-01-01T00:00Z')).toBeNull(); }); }); diff --git a/web/tests/compat.spec.ts b/web/tests/compat.spec.ts index f965e80..bb3fe16 100644 --- a/web/tests/compat.spec.ts +++ b/web/tests/compat.spec.ts @@ -1,850 +1,82 @@ -/** - * The byte-compatibility harness: the one test that compares this site against the other - * implementation instead of against itself. - * - * Every case here is encoded twice — once by `src/lib/adc/canonical.ts` in this process, once by - * `researcher-tools canonicalize` in a JVM — and the two byte strings have to be identical. That is - * the whole claim the site rests on, because `StudyConfigurationCodec.decode` re-encodes what it - * parsed and refuses the file unless the bytes come back the same. A configuration this page signs - * is verified over exactly those bytes on the device, so an encoder that writes merely *plausible* - * JSON produces files that are correctly signed and rejected everywhere. - * - * The last block goes past canonicalisation: it generates a signing key here, signs here, builds the - * envelope here, and then asks the CLI to verify the result. `check-config` succeeding is the - * end-to-end statement that a `.adccfg` made in a browser is one the Android app will accept. - * - * @vitest-environment node - */ - -import { execFile, execFileSync } from 'node:child_process'; -import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { dirname, join } from 'node:path'; -import { fileURLToPath } from 'node:url'; -import { afterAll, beforeAll, describe, expect, it } from 'vitest'; -import { canonicalBytes, canonicalize, keysetJson } from '../src/lib/adc/canonical'; -import { encodeEnvelope } from '../src/lib/adc/envelope'; -import { fingerprint, generateSigningKeyPair, sign, verify } from '../src/lib/adc/crypto'; -import { generateHpkeKeyset } from '../src/lib/adc/tink'; +/** Cross-language Protocol v1 conformance checks plus focused primitive vectors. */ + +import { describe, expect, it } from 'vitest'; +import { + canonicalBytes, + canonicalConfigurationBytes, + canonicalize, + canonicalizeConfiguration +} from '../src/lib/adc/canonical'; +import { decodeBase64Url, encodeBase64Url, sign, verify } from '../src/lib/adc/crypto'; +import { decodeEnvelope, encodeEnvelope } from '../src/lib/adc/envelope'; import { openBundle } from '../src/lib/adc/bundle'; -import { decodeEnvelope } from '../src/routes/researcher/parse'; -import { bundleJson, seal } from './seal'; -import type { - CollectorConfig, - InterventionConfig, - NetworkTransport, - StudyConfiguration, - TinkKeyset -} from '../src/lib/adc/types'; - -const REPOSITORY = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); -const CLI = join(REPOSITORY, 'researcher-tools/build/install/researcher-tools/bin/researcher-tools'); +import { parseConfiguration } from '../src/routes/researcher/parse'; +import { HPKE, SIGNING, validConfiguration } from './fixture'; +import { sealBundle } from './seal'; -let workspace = ''; -let sequence = 0; +const fromHex = (value: string) => + Uint8Array.from(value.match(/../g) ?? [], (byte) => parseInt(byte, 16)); -beforeAll(async () => { - await new Promise((resolve, reject) => { - execFile( - join(REPOSITORY, 'gradlew'), - [':researcher-tools:installDist'], - { cwd: REPOSITORY }, - (error, stdout, stderr) => { - if (!error) resolve(); - else reject(new Error(`researcher-tools build failed\n${stderr || stdout || error.message}`)); - } +describe('Protocol v1 deterministic compatibility boundary', () => { + it('matches RFC 8032 Ed25519 test vector 1 using raw keys', () => { + const privateKey = encodeBase64Url( + fromHex('9d61b19deffd5a60ba844af492ec2cc44449c5697b326919703bac031cae7f60') + ); + const publicKey = encodeBase64Url( + fromHex('d75a980182b10ab7d54bfed3c964073a0ee172f3daa62325af021a68f707511a') + ); + const expected = fromHex( + 'e5564300c360ac729086e2cc806e828a84877f1eb8e5d974d873e06522490155' + + '5fb8821590a33bacc61e39701cf9b46bd25bf5f0595bbe24655141438e7a100b' ); + expect(sign(new Uint8Array(), privateKey)).toEqual(expected); + expect(verify(new Uint8Array(), expected, publicKey)).toBe(true); }); - workspace = mkdtempSync(join(tmpdir(), 'adc-compat-')); -}, 600_000); -afterAll(() => workspace && rmSync(workspace, { recursive: true, force: true })); - -/** A fresh directory per invocation, because every CLI output path is opened `CREATE_NEW`. */ -function scratch(): string { - const directory = join(workspace, String(sequence++)); - mkdirSync(directory, { recursive: true }); - return directory; -} - -function runCli(...args: string[]): string { - try { - return execFileSync(CLI, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }); - } catch (failure) { - const detail = failure as { stderr?: string; stdout?: string; message?: string }; - throw new Error( - `researcher-tools ${args.join(' ')}\n${detail.stderr || detail.stdout || detail.message}` + it('round-trips JCS configuration bytes through the fixed signed envelope', () => { + const configuration = validConfiguration(); + const payload = canonicalConfigurationBytes(configuration); + const signature = sign(payload, SIGNING.privateKey); + const envelope = encodeEnvelope(configuration.signer.key_id, payload, signature); + const decoded = decodeEnvelope(envelope); + expect(decoded.configurationBytes).toEqual(payload); + expect(decoded.signerKeyId).toBe(configuration.signer.key_id); + expect(verify(decoded.configurationBytes, decoded.signature, SIGNING.publicKey)).toBe(true); + expect(canonicalizeConfiguration(parseConfiguration(envelope))).toBe( + canonicalizeConfiguration(configuration) ); - } -} - -/** - * The configuration as ordinary JSON, which is what a researcher would have written by hand. The - * only translation is the absent upload block: the schema spells "no upload" as an empty object, - * and `null` is not a shape the decoder accepts. - */ -function wireJson(configuration: StudyConfiguration): string { - return JSON.stringify({ ...configuration, upload: configuration.upload ?? {} }, null, 2); -} - -function cliCanonicalize(configuration: StudyConfiguration): Uint8Array { - const directory = scratch(); - const input = join(directory, 'study.json'); - const output = join(directory, 'canonical.json'); - writeFileSync(input, wireJson(configuration), 'utf8'); - runCli('canonicalize', '--input', input, '--output', output); - return readFileSync(output); -} - -/** Where the two encodings part company, with enough either side to see what happened. */ -function difference(site: Uint8Array, cli: Uint8Array): string { - const text = new TextDecoder(); - const limit = Math.max(site.length, cli.length); - for (let index = 0; index < limit; index++) { - if (index < site.length && index < cli.length && site[index] === cli[index]) continue; - const from = Math.max(0, index - 48); - const at = (bytes: Uint8Array) => - index < bytes.length ? `0x${bytes[index].toString(16).padStart(2, '0')}` : 'end of output'; - return [ - `first difference at byte ${index} (site ${site.length} bytes, cli ${cli.length} bytes)`, - ` site ${at(site)} ${JSON.stringify(text.decode(site.subarray(from, index + 48)))}`, - ` cli ${at(cli)} ${JSON.stringify(text.decode(cli.subarray(from, index + 48)))}` - ].join('\n'); - } - return 'identical'; -} - -const DEMO_KEYSET: TinkKeyset = { - primaryKeyId: 218992727, - key: [ - { - keyData: { - typeUrl: 'type.googleapis.com/google.crypto.tink.HpkePublicKey', - value: 'EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA', - keyMaterialType: 'ASYMMETRIC_PUBLIC' - }, - status: 'ENABLED', - keyId: 218992727, - outputPrefixType: 'TINK' - } - ] -}; - -const DEMO_PUBLIC_KEY = 'MCowBQYDK2VwAyEAsRSaTpZmTSBL7eN6nS/HBsNmLM8n1hdRmIt1vtLZsC0='; - -const BASE: StudyConfiguration = { - schema_version: 1, - experiment_id: 'compat-harness', - configuration_id: 'compat-case-001', - assigned_participant_id: null, - issued_at: '2026-01-01T00:00:00Z', - expires_at: '2035-01-01T00:00:00Z', - minimum_app_version: 1, - title: 'Byte compatibility harness', - researcher: { name: 'Harness', contact: 'harness@example.invalid' }, - purpose: 'Prove the browser encoder and the JVM encoder write the same bytes.', - duration_hours: 24, - consent: { document_version: 'harness-1', summary: 'A fixture. Nothing is collected.' }, - collectors: [{ id: 'app_lifecycle.v1', required: true, config: {} }], - surveys: [], - interventions: [], - storage: { maximum_local_bytes: 16 * 1024 * 1024 }, - signer: { key_id: 'compat-signer', public_key: DEMO_PUBLIC_KEY }, - export: { researcher_key_id: 'compat-hpke', tink_hpke_public_keyset: DEMO_KEYSET }, - upload: null -}; - -function study(overrides: Partial): StudyConfiguration { - return { ...BASE, ...overrides }; -} - -function notification( - id: string, - offsetMinutes: number, - message: string, - availabilityMinutes = 1_440 -): InterventionConfig { - return { - id, - action: { type: 'notification', notification_title: 'Study notice', notification_message: message }, - triggers: [{ - id: `${id.slice(0, 56)}-trigger`, - schedule: { type: 'one_time', offset_minutes: offsetMinutes, clock: 'CALENDAR_TIME' }, - availability_minutes: availabilityMinutes - }] - }; -} - -function everyCollector(required: boolean): CollectorConfig[] { - return [ - { id: 'app_lifecycle.v1', required, config: {} }, - { - id: 'accelerometer.v1', - required, - config: { sampling_period_us: 100_000, maximum_report_latency_us: 1_000_000 } - }, - { id: 'network_state.v1', required, config: { include_bandwidth_estimates: true } }, - { - id: 'network_usage.v1', - required, - config: { transports: ['mobile', 'wifi'], poll_interval_minutes: 5 } - }, - { id: 'usage_events.v1', required, config: { poll_interval_minutes: 15 } }, - { - id: 'location.v1', - required, - config: { - interval_millis: 10_000, - minimum_interval_millis: 5_000, - maximum_batch_delay_millis: 30_000, - minimum_displacement_meters: 5, - priority: 'BALANCED' - } - }, - { id: 'keyboard_touch.v1', required, config: { trajectory_sampling_hz: 60 } } - ]; -} - -type LocationConfig = Extract['config']; - -function location(overrides: Partial): StudyConfiguration { - return study({ - collectors: [ - { - id: 'location.v1', - required: false, - config: { - interval_millis: 10_000, - minimum_interval_millis: 5_000, - maximum_batch_delay_millis: 30_000, - minimum_displacement_meters: 5, - priority: 'BALANCED', - ...overrides - } - } - ] }); -} -function usage(transports: NetworkTransport[]): StudyConfiguration { - return study({ - collectors: [ - { id: 'network_usage.v1', required: true, config: { transports, poll_interval_minutes: 30 } } - ] + it('round-trips deterministic RFC 9180/AES-GCM bundle bytes', async () => { + const configuration = validConfiguration(); + const first = await sealBundle(configuration, SIGNING.privateKey); + const second = await sealBundle(configuration, SIGNING.privateKey); + expect(second).toEqual(first); + expect(await openBundle(first, configuration, HPKE.privateKey)).toMatchObject({ ok: true }); }); -} - -/** Written by code point rather than as escapes, so nothing in this file can be read two ways. */ -const character = (code: number) => String.fromCharCode(code); -const LINE_SEPARATOR = character(0x2028); -const PARAGRAPH_SEPARATOR = character(0x2029); -const DELETE = character(0x7f); - -/** Every code point Gson escapes as `\u00xx`, plus the five with short forms, in order. */ -const CONTROLS = Array.from({ length: 0x20 }, (_, code) => character(code)).join(''); - -/** - * Everything Gson's escape table has an opinion about, in one string: the whole control range, the - * two characters it escapes above `0x20`, and a spread of characters it deliberately leaves raw - * because the writer is not in HTML-safe mode. - */ -const NASTY = - '研究「同意」書 😀🇹🇼 — "quoted" \\backslash\\ /slash/ & = \' ' + - CONTROLS + - ' ' + - LINE_SEPARATOR + - 'line' + - PARAGRAPH_SEPARATOR + - 'para ' + - DELETE + - ' ünïcödé ½ ∑ 🇯🇵👩‍👩‍👧‍👦'; - -/** The demonstration study the CLI ships, held the way the site holds it. */ -const DEMO: StudyConfiguration = { - schema_version: 1, - experiment_id: 'modular-sensing-demo', - configuration_id: 'demo-config-2026', - assigned_participant_id: null, - issued_at: '2026-01-01T00:00:00Z', - expires_at: '2035-01-01T00:00:00Z', - minimum_app_version: 1, - title: 'Modular sensing demonstration', - researcher: { - name: 'Android Data Collector maintainers', - contact: 'research@example.invalid' - }, - purpose: - 'Verify the complete on-device collection, pause, export, and researcher-decryption loop.', - duration_hours: 24, - consent: { - document_version: 'demo-1', - summary: - "This demonstration can collect precise location, motion, network state, aggregate Wi-Fi/mobile usage, app and screen usage events, this app's lifecycle, and touch dynamics made inside the optional research keyboard. It never records keyboard text. Data stays encrypted on this device until you choose Export. You can pause, withdraw, export repeatedly, or delete local data." - }, - collectors: [ - { id: 'app_lifecycle.v1', required: true, config: {} }, - { - id: 'accelerometer.v1', - required: true, - config: { sampling_period_us: 100_000, maximum_report_latency_us: 1_000_000 } - }, - { id: 'network_state.v1', required: true, config: { include_bandwidth_estimates: true } }, - { - id: 'network_usage.v1', - required: false, - config: { transports: ['mobile', 'wifi'], poll_interval_minutes: 5 } - }, - { id: 'usage_events.v1', required: false, config: { poll_interval_minutes: 15 } }, - { - id: 'location.v1', - required: false, - config: { - interval_millis: 10_000, - minimum_interval_millis: 5_000, - maximum_batch_delay_millis: 30_000, - minimum_displacement_meters: 5, - priority: 'BALANCED' - } - }, - { id: 'keyboard_touch.v1', required: false, config: { trajectory_sampling_hz: 60 } } - ], - surveys: [], - interventions: [notification('demo-check-in', 60, 'Please check that the study is still running as expected.')], - storage: { maximum_local_bytes: 16_777_216 }, - signer: { key_id: 'demo-signer-2026', public_key: DEMO_PUBLIC_KEY }, - export: { researcher_key_id: 'demo-hpke-2026', tink_hpke_public_keyset: DEMO_KEYSET }, - upload: null -}; - -/** - * The values that separate `Float.toString` from every shorter implementation: the two the schema - * will actually see, the ones whose double rendering differs from their float one, the tie Java - * breaks towards the even digit, and both scientific-notation switches the bounds can still reach. - */ -const DISPLACEMENTS = [ - 0, 5, 0.25, 1234.5678, 0.1, 0.3, 9999.999, 10_000, 4618.53125, 0.001, 0.0001, 1e-5, 1.4e-45 -]; - -const CASES: Array<{ name: string; configuration: StudyConfiguration }> = [ - { name: 'the demonstration study the CLI ships', configuration: DEMO }, - { - name: 'every collector, all required, with upload and interventions', - configuration: study({ - collectors: everyCollector(true), - interventions: [ - notification('check-in-one', 60, 'Still running?'), - notification('check-in-two', 1_200, '研究の確認 😀') - ], - upload: { - endpoint: 'https://uploads.example.invalid/v1', - interval_minutes: 60, - allow_metered: true - } - }) - }, - { - name: 'every collector, none required, no upload, no interventions', - configuration: study({ collectors: everyCollector(false), interventions: [], upload: null }) - }, - - ...everyCollector(true).map((collector) => ({ - name: `only ${collector.id}, required`, - configuration: study({ collectors: [collector] }) - })), - ...everyCollector(false).map((collector) => ({ - name: `only ${collector.id}, optional`, - configuration: study({ collectors: [collector] }) - })), - - ...DISPLACEMENTS.map((meters) => ({ - name: `minimum_displacement_meters ${meters}`, - configuration: location({ minimum_displacement_meters: meters }) - })), - - { - name: 'text: CJK, emoji, quotes, backslashes, newlines and the whole control range', - configuration: study({ - title: '研究 "「」" \\ & 😀 line\nbreak\ttab', - researcher: { name: '林\t"Lin"\\研究員 😀', contact: 'mail@example.invalid\n<研究>' }, - purpose: NASTY, - consent: { document_version: 'v1\\"研究"', summary: NASTY }, - interventions: [notification('nasty-notice', 5, NASTY.slice(0, 500))], - upload: { - endpoint: 'https://uploads.example.invalid/v1?a=b&c=d#研究', - interval_minutes: 15, - allow_metered: false - } - }) - }, - { - name: 'text: the characters Gson leaves raw because it is not HTML-safe', - configuration: study({ - title: `< > & = ' / ${DELETE}`, - purpose: `${DELETE}` - }) - }, - { - name: 'text: nothing but a line separator and a paragraph separator', - configuration: study({ - title: LINE_SEPARATOR + PARAGRAPH_SEPARATOR, - consent: { document_version: LINE_SEPARATOR, summary: PARAGRAPH_SEPARATOR } - }) - }, - - { - name: 'bounds: every numeric field at its minimum', - configuration: study({ - minimum_app_version: 1, - duration_hours: 1, - title: 'A', - researcher: { name: 'A', contact: 'a@b' }, - purpose: 'P', - consent: { document_version: 'v', summary: 'S' }, - experiment_id: 'a-b', - configuration_id: 'a-b', - signer: { key_id: 'a-b', public_key: 'A'.repeat(32) }, - export: { researcher_key_id: 'a-b', tink_hpke_public_keyset: DEMO_KEYSET }, - storage: { maximum_local_bytes: 8 * 1024 * 1024 }, - collectors: [ - { - id: 'accelerometer.v1', - required: false, - config: { sampling_period_us: 5_000, maximum_report_latency_us: 0 } - }, - { - id: 'network_usage.v1', - required: false, - config: { transports: ['mobile'], poll_interval_minutes: 1 } - }, - { id: 'usage_events.v1', required: false, config: { poll_interval_minutes: 1 } }, - { - id: 'location.v1', - required: false, - config: { - interval_millis: 1_000, - minimum_interval_millis: 500, - maximum_batch_delay_millis: 0, - minimum_displacement_meters: 0, - priority: 'BALANCED' - } - }, - { id: 'keyboard_touch.v1', required: false, config: { trajectory_sampling_hz: 1 } } - ], - interventions: [notification('a-b', 0, 'M', 1)], - upload: { endpoint: 'https://a', interval_minutes: 1, allow_metered: false } - }) - }, - { - name: 'bounds: every numeric field at its maximum', - configuration: study({ - minimum_app_version: 2_147_483_647, - duration_hours: 8_760, - title: 'T'.repeat(120), - researcher: { name: 'N'.repeat(120), contact: 'C'.repeat(240) }, - purpose: 'P'.repeat(2_000), - consent: { document_version: 'V'.repeat(64), summary: 'S'.repeat(8_000) }, - experiment_id: `a${'b'.repeat(63)}`, - configuration_id: `9${'-c9'.repeat(21)}`, - signer: { key_id: `z${'-y9'.repeat(21)}`, public_key: 'K'.repeat(1_024) }, - storage: { maximum_local_bytes: 8 * 1024 * 1024 * 1024 }, - collectors: [ - { - id: 'accelerometer.v1', - required: true, - config: { sampling_period_us: 1_000_000, maximum_report_latency_us: 60_000_000 } - }, - { - id: 'network_usage.v1', - required: true, - config: { transports: ['mobile', 'wifi'], poll_interval_minutes: 1_440 } - }, - { id: 'usage_events.v1', required: true, config: { poll_interval_minutes: 1_440 } }, - { - id: 'location.v1', - required: true, - config: { - interval_millis: 3_600_000, - minimum_interval_millis: 3_600_000, - maximum_batch_delay_millis: 86_400_000, - minimum_displacement_meters: 10_000, - priority: 'HIGH_ACCURACY' - } - }, - { id: 'keyboard_touch.v1', required: true, config: { trajectory_sampling_hz: 120 } } - ], - interventions: [notification(`p${'-q8'.repeat(21)}`, 525_599, 'M'.repeat(500), 525_600)], - upload: { - endpoint: `https://e.invalid/${'a'.repeat(2_030)}`, - interval_minutes: 10_080, - allow_metered: true - } - }) - }, - { name: 'transports: mobile only', configuration: usage(['mobile']) }, - { name: 'transports: wifi only', configuration: usage(['wifi']) }, - { name: 'transports: written in the other order', configuration: usage(['wifi', 'mobile']) }, - { name: 'transports: repeated', configuration: usage(['wifi', 'wifi']) }, - - { name: 'interventions: absent', configuration: study({ interventions: [] }) }, - { - name: 'interventions: one-time, interval, and local daily schedules', - configuration: study({ - interventions: [ - notification('first-notice', 0, 'A'), - { - id: 'interval-notice', - action: { type: 'notification', notification_title: 'Check in', notification_message: 'B' }, - triggers: [{ id: 'every-six-hours', schedule: { type: 'interval', start_offset_minutes: 60, interval_minutes: 360, clock: 'ACTIVE_RUNNING_TIME' }, availability_minutes: 120 }] - }, - { - id: 'daily-notice', - action: { type: 'notification', notification_title: 'Check in', notification_message: '請確認研究仍在執行 😀' }, - triggers: [{ id: 'local-evening', schedule: { type: 'daily_local', local_time: '20:30' }, availability_minutes: 720 }] - } - ] - }) - }, - - { - name: 'upload: present, metered allowed, shortest interval', - configuration: study({ - upload: { - endpoint: 'https://uploads.example.invalid/ingest', - interval_minutes: 1, - allow_metered: true - } - }) - }, - { name: 'upload: absent', configuration: study({ upload: null }) }, - - { - name: 'instants: fractional seconds Instant.toString keeps', - configuration: study({ - issued_at: '2026-01-01T00:00:00.123456789Z', - expires_at: '2035-06-30T23:59:59.999999999Z' - }) - }, - { - name: 'instants: spellings Instant.toString rewrites', - configuration: study({ - issued_at: '2025-12-31T16:00:00.000-08:00', - expires_at: '2035-01-01T00:00:00.1Z' - }) - }, - { - name: 'instants: a leap day and the end of a year', - configuration: study({ issued_at: '2028-02-29T23:59:59Z', expires_at: '2035-12-31T23:59:59Z' }) - }, - - { - name: 'keyset: a key ID above the signed 32-bit range', - configuration: study({ - export: { - researcher_key_id: 'compat-hpke', - tink_hpke_public_keyset: { - primaryKeyId: 4_294_967_295, - key: [ - { - keyData: { - typeUrl: 'type.googleapis.com/google.crypto.tink.HpkePublicKey', - value: 'EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA', - keyMaterialType: 'ASYMMETRIC_PUBLIC' - }, - status: 'ENABLED', - keyId: 4_294_967_295, - outputPrefixType: 'TINK' - } - ] - } - } - }) - }, - { - name: 'keyset: fields in another order, which the codec has to preserve', - configuration: study({ - export: { - researcher_key_id: 'compat-hpke', - tink_hpke_public_keyset: { - key: [ - { - outputPrefixType: 'TINK', - keyId: 218992727, - status: 'ENABLED', - keyData: { - keyMaterialType: 'ASYMMETRIC_PUBLIC', - value: 'EgYIARABGAIaIBpyQ3w4fFx9XgEUx5kyzZaIPXLq7aYU6RJ+y9+rGNEA', - typeUrl: 'type.googleapis.com/google.crypto.tink.HpkePublicKey' - } - } - ], - primaryKeyId: 218992727 - } as unknown as TinkKeyset - } - }) - } -]; - -describe('canonicalize matches researcher-tools byte for byte', () => { - for (const testCase of CASES) { - it( - testCase.name, - () => { - const site = canonicalBytes(testCase.configuration); - const report = difference(site, cliCanonicalize(testCase.configuration)); - expect(report).toBe('identical'); - }, - 60_000 - ); - } -}); - -/** - * Past canonicalisation: a study whose keys were made here, signed here, packaged here, and handed - * to the verifier the app uses. Ed25519 is deterministic, so the JVM signing the same bytes with - * the same key has to produce the same signature — which makes the envelope comparison an equality - * rather than merely "both of them verify". - */ -describe('a browser-made .adccfg is one the app accepts', () => { - it( - 'verifies self-certifying and pinned, and matches what the CLI would have signed', - () => { - const signing = generateSigningKeyPair(); - const hpke = generateHpkeKeyset(); - const keyId = 'web-made-signer'; - - const configuration = study({ - experiment_id: 'browser-end-to-end', - configuration_id: 'browser-config-001', - title: '瀏覽器簽署的研究 😀 "end to end"', - consent: { document_version: 'e2e-1', summary: NASTY }, - collectors: everyCollector(true), - interventions: [notification('e2e-notice', 30, '請確認 😀')], - signer: { key_id: keyId, public_key: signing.publicX509Base64 }, - export: { researcher_key_id: 'browser-hpke', tink_hpke_public_keyset: hpke.publicKeyset }, - upload: { - endpoint: 'https://uploads.example.invalid/ingest', - interval_minutes: 120, - allow_metered: false - } - }); - - const bytes = canonicalBytes(configuration); - const signature = sign(bytes, signing.privatePkcs8Base64); - expect(verify(bytes, signature, signing.publicX509Base64)).toBe(true); - const envelope = encodeEnvelope(keyId, bytes, signature); - - const directory = scratch(); - const envelopePath = join(directory, 'study.adccfg'); - const publicPath = join(directory, 'signer.pub'); - const privatePath = join(directory, 'signer.key'); - const canonicalPath = join(directory, 'canonical.json'); - const cliEnvelopePath = join(directory, 'cli.adccfg'); - writeFileSync(envelopePath, envelope); - writeFileSync(publicPath, signing.publicX509Base64, 'utf8'); - writeFileSync(privatePath, signing.privatePkcs8Base64, 'utf8'); - writeFileSync(canonicalPath, bytes); - - const selfCertifying = runCli( - 'check-config', - '--envelope', - envelopePath, - '--now', - '2026-06-01T00:00:00Z' - ); - expect(selfCertifying).toContain('valid browser-end-to-end browser-config-001'); - expect(selfCertifying).toContain(`signer ${keyId} ${fingerprint(signing.publicX509Base64)}`); - expect(selfCertifying).toContain('pinned no (self-certifying)'); - - const pinned = runCli( - 'check-config', - '--envelope', - envelopePath, - '--public', - publicPath, - '--key-id', - keyId, - '--app-version', - '1', - '--now', - '2026-06-01T00:00:00Z' - ); - expect(pinned).toContain('pinned yes'); - - runCli( - 'sign', - '--config', - canonicalPath, - '--private', - privatePath, - '--key-id', - keyId, - '--output', - cliEnvelopePath - ); - expect(difference(envelope, readFileSync(cliEnvelopePath))).toBe('identical'); - }, - 120_000 - ); - - it( - 'bulk-personalizes unique configurations without putting assigned codes in filenames', - () => { - const signing = generateSigningKeyPair(); - const directory = scratch(); - const config = study({ - configuration_id: 'personalization-template', - signer: { key_id: 'personalize-signer', public_key: signing.publicX509Base64 } - }); - const configPath = join(directory, 'template.json'); - const privatePath = join(directory, 'signer.key'); - const mappingPath = join(directory, 'mapping.tsv'); - const output = join(directory, 'artifacts'); - writeFileSync(configPath, canonicalBytes(config)); - writeFileSync(privatePath, signing.privatePkcs8Base64, 'utf8'); - writeFileSync(mappingPath, 'arm-a-config\tAssigned_A-017\narm-b-config\tAssigned_B-018\n', 'utf8'); - - const canonicalWithId = join(directory, 'personalized.json'); - runCli( - 'canonicalize', '--input', configPath, '--output', canonicalWithId, - '--assigned-participant-id', 'Assigned_C-019' - ); - expect(JSON.parse(readFileSync(canonicalWithId, 'utf8')).assigned_participant_id).toBe('Assigned_C-019'); - - const signedWithId = join(directory, 'personalized.adccfg'); - runCli( - 'sign', '--config', configPath, '--private', privatePath, - '--key-id', 'personalize-signer', '--output', signedWithId, - '--assigned-participant-id', 'Assigned_D-020' - ); - const signedConfiguration = JSON.parse( - new TextDecoder().decode(decodeEnvelope(readFileSync(signedWithId)).configurationBytes) - ); - expect(signedConfiguration.assigned_participant_id).toBe('Assigned_D-020'); - - expect(runCli( - 'personalize', '--config', configPath, '--mapping', mappingPath, - '--private', privatePath, '--key-id', 'personalize-signer', '--output-dir', output - )).toContain('personalized 2 configurations'); - const first = JSON.parse(readFileSync(join(output, 'arm-a-config.json'), 'utf8')); - const second = JSON.parse(readFileSync(join(output, 'arm-b-config.json'), 'utf8')); - expect(first.assigned_participant_id).toBe('Assigned_A-017'); - expect(second.assigned_participant_id).toBe('Assigned_B-018'); - expect(first.configuration_id).not.toBe(second.configuration_id); - expect(existsSync(join(output, 'Assigned_A-017.adccfg'))).toBe(false); - expect(runCli( - 'check-config', '--envelope', join(output, 'arm-a-config.adccfg'), - '--now', '2026-06-01T00:00:00Z' - )).toContain('valid compat-harness arm-a-config'); - - const duplicateMapping = join(directory, 'duplicate.tsv'); - writeFileSync(duplicateMapping, 'same-config\tAssigned_A\nsame-config\tAssigned_B\n', 'utf8'); - expect(() => runCli( - 'personalize', '--config', configPath, '--mapping', duplicateMapping, - '--private', privatePath, '--key-id', 'personalize-signer', - '--output-dir', join(directory, 'duplicate-output') - )).toThrow(/Duplicate configuration ID/); - expect(existsSync(join(directory, 'duplicate-output'))).toBe(false); - }, - 120_000 - ); - - /** - * `JSON.stringify(-0)` is `0`, so a negative zero cannot reach the CLI through an ordinary JSON - * file and cannot be a case in the table above. It can still reach a number input, and both ends - * agree it is `-0.0` — which the round trip below is the proof of. - */ - it( - 'writes a negative zero displacement the way the device re-encodes it', - () => { - const bytes = canonicalBytes(location({ minimum_displacement_meters: -0 })); - expect(new TextDecoder().decode(bytes)).toContain('"minimum_displacement_meters":-0.0'); - const directory = scratch(); - const input = join(directory, 'canonical.json'); - const output = join(directory, 'roundtrip.json'); - writeFileSync(input, bytes); - runCli('canonicalize', '--input', input, '--output', output); - expect(difference(bytes, readFileSync(output))).toBe('identical'); - }, - 60_000 - ); - - it( - 'canonicalises its own output to itself, which is what the device checks', - () => { - const directory = scratch(); - const canonicalPath = join(directory, 'canonical.json'); - const output = join(directory, 'roundtrip.json'); - writeFileSync(canonicalPath, canonicalize(DEMO), 'utf8'); - runCli('canonicalize', '--input', canonicalPath, '--output', output); - expect(difference(canonicalBytes(DEMO), readFileSync(output))).toBe('identical'); - }, - 60_000 - ); - - /** - * The export half, and the one direction available: `researcher-tools` has no `encrypt`, because - * the only thing that writes a bundle is a phone. So the bundle is sealed here and the JVM opens - * it — which tests the same wire format the site's reader is written against, from the other side. - * - * Without this, `tests/bundle.spec.ts` would be a writer and a reader agreeing with each other, - * and two halves wrong the same way agree perfectly. `decrypt` succeeding says the container - * layout, the big-endian lengths, the TINK prefix, the RFC 9180 schedule, the context in both of - * its roles, and the single-tag body are all what Kotlin believes them to be. The key exists only - * inside `scratch()` and only for this test. - */ - it( - 'seals a bundle the CLI decrypts, and opens one the site reads', - async () => { - const keyset = generateHpkeKeyset(); - const configuration = study({ - experiment_id: 'bundle-compat', - configuration_id: 'bundle-compat-001', - export: { researcher_key_id: 'bundle-hpke', tink_hpke_public_keyset: keyset.publicKeyset } - }); - const plaintext = bundleJson(configuration, { - events: 4, - assignedParticipantId: 'Assigned_A-017' - }); - const bundle = await seal(configuration, plaintext); - - const directory = scratch(); - const bundlePath = join(directory, 'export.adcexp'); - const configPath = join(directory, 'canonical.json'); - const privatePath = join(directory, 'hpke-private.json'); - const output = join(directory, 'decrypted.json'); - writeFileSync(bundlePath, bundle); - writeFileSync(configPath, canonicalBytes(configuration)); - writeFileSync(privatePath, keysetJson(keyset.privateKeyset), 'utf8'); - - runCli( - 'decrypt', - '--bundle', bundlePath, - '--config', configPath, - '--private', privatePath, - '--output', output - ); - expect(difference(new TextEncoder().encode(plaintext), readFileSync(output))).toBe('identical'); - - // And back the other way, so the two implementations are stated to agree rather than assumed. - const opened = await openBundle(bundle, configuration, keyset.privateKeyset); - if (!opened.ok) throw new Error(`the site refused its own bundle: ${opened.failure}`); - expect(opened.bundle.text).toBe(plaintext); + it('rejects the former Protocol v1 JSON and variable-length envelope', () => { + const current = JSON.parse(canonicalizeConfiguration(validConfiguration())) as Record; + delete current.platform; + delete current.minimum_client_version; + current.minimum_app_version = 1; + current.export = { + researcher_key_id: 'protocol-export', + tink_hpke_public_keyset: { primaryKeyId: 1, key: [] } + }; + expect(() => parseConfiguration(canonicalBytes(current))).toThrow(); + + const payload = canonicalConfigurationBytes(validConfiguration()); + const old = new Uint8Array(16 + payload.length + 64); + old.set(new TextEncoder().encode('ADCCFG01')); + const view = new DataView(old.buffer); + view.setUint16(8, 3); + view.setUint32(10, payload.length); + view.setUint16(14, 64); + expect(() => decodeEnvelope(old)).toThrow(); + }); - // The CLI's own refusal, for the mistake the site names `wrong_study`. - const other = join(directory, 'other.json'); - writeFileSync(other, canonicalBytes(study({ export: configuration.export, configuration_id: 'bundle-compat-002' }))); - expect(() => - runCli( - 'decrypt', - '--bundle', bundlePath, - '--config', other, - '--private', privatePath, - '--output', join(directory, 'never.json') - ) - ).toThrow(); - expect(existsSync(join(directory, 'never.json'))).toBe(false); - }, - 120_000 - ); + it('uses generic JCS rather than a schema-order encoder', () => { + expect(canonicalize({ z: 0, a: { y: 2, x: 1 } })).toBe('{"a":{"x":1,"y":2},"z":0}'); + }); }); diff --git a/web/tests/crypto.spec.ts b/web/tests/crypto.spec.ts index 40a9462..ad71657 100644 --- a/web/tests/crypto.spec.ts +++ b/web/tests/crypto.spec.ts @@ -1,121 +1,69 @@ -/** - * Interop tests against the fixtures in `researcher-tools/examples`, which were produced by the - * shipped CLI on a real JDK and a real Tink. - * - * The point of every case here is that the site and the CLI are interchangeable. A researcher who - * generates a key on this page must be able to sign with the CLI later, and a study signed here - * must verify on a phone, so nothing below asserts "our encoder agrees with itself" — each case - * pins bytes that came from the other implementation. - */ - import { describe, expect, it } from 'vitest'; -import demoSigningPrivateKey from '../../researcher-tools/examples/INSECURE-demo-signing-private.key?raw'; -import demoHpkePrivateKeyset from '../../researcher-tools/examples/INSECURE-demo-hpke-private.json?raw'; -import demoStudyJson from '../../researcher-tools/examples/demo-study.json?raw'; import { - decodeBase64, - decodePkcs8, - decodeX509, - encodeBase64, - encodePkcs8, - encodeX509, - fingerprint, + decodeBase64Url, + encodeBase64Url, + generateHpkeKeyPair, generateSigningKeyPair, + hpkeKeyPairFromPrivate, sign, + signingKeyPairFromPrivate, verify } from '../src/lib/adc/crypto'; -import { hpkeKeysetFromPrivateKey } from '../src/lib/adc/tink'; -import { encodeEnvelope } from '../src/lib/adc/envelope'; - -const demoPrivateKey = demoSigningPrivateKey.trim(); -const demoPrivateKeyset = demoHpkePrivateKeyset.trim(); -const demoStudy = JSON.parse(demoStudyJson); -const demoPublicKey: string = demoStudy.signer.public_key; -const demoPublicKeyset = demoStudy.export.tink_hpke_public_keyset; - -describe('Ed25519 keys', () => { - it('round-trips a generated pair through sign and verify', () => { - const pair = generateSigningKeyPair(); - const message = new TextEncoder().encode('{"schema_version":1}'); - const signature = sign(message, pair.privatePkcs8Base64); +import { decodeEnvelope, encodeEnvelope } from '../src/lib/adc/envelope'; - expect(signature).toHaveLength(64); - expect(verify(message, signature, pair.publicX509Base64)).toBe(true); - - const tampered = Uint8Array.from(message); - tampered[0] ^= 1; - expect(verify(tampered, signature, pair.publicX509Base64)).toBe(false); - expect(verify(message, signature, generateSigningKeyPair().publicX509Base64)).toBe(false); +describe('raw Protocol v1 keys', () => { + it('generates and reopens raw 32-byte Ed25519 and X25519 artifacts', () => { + const signing = generateSigningKeyPair(); + const hpke = generateHpkeKeyPair(); + for (const value of [signing.privateKey, signing.publicKey, hpke.privateKey, hpke.publicKey]) { + expect(value).toMatch(/^[A-Za-z0-9_-]{43}$/); + expect(decodeBase64Url(value, 32)).toHaveLength(32); + } + expect(signingKeyPairFromPrivate(signing.privateKey)).toEqual(signing); + expect(hpkeKeyPairFromPrivate(hpke.privateKey)).toEqual(hpke); }); - it('re-encodes the demo PKCS#8 and X.509 keys byte for byte', () => { - expect(encodePkcs8(decodePkcs8(demoPrivateKey))).toBe(demoPrivateKey); - expect(encodeX509(decodeX509(demoPublicKey))).toBe(demoPublicKey); - - // Both fixtures are 48 and 44 bytes of DER, not raw keys, and they are one key pair: a - // signature made with the file verifies under the key the demo study publishes. - expect(decodeBase64(demoPrivateKey)).toHaveLength(48); - expect(decodeBase64(demoPublicKey)).toHaveLength(44); - const message = new TextEncoder().encode('demo'); - expect(verify(message, sign(message, demoPrivateKey), demoPublicKey)).toBe(true); + it('rejects padded, standard-base64, whitespace, and wrong-length alternatives', () => { + const canonical = encodeBase64Url(new Uint8Array(32).fill(7)); + for (const value of [`${canonical}=`, ` ${canonical}`, `+${canonical.slice(1)}`, 'AA']) { + expect(() => decodeBase64Url(value, 32)).toThrow(); + } }); - it('fingerprints the demo public key the way the app displays it', () => { - expect(fingerprint(demoPublicKey)).toBe('9D0D AE5A 0D20 B29F D642 942A 0E17 4AAE'); - }); -}); - -describe('Tink HPKE keysets', () => { - it('rebuilds both demo keysets from the demo private key material', () => { - // Taken apart by hand rather than through the encoder, so a wrong idea about the layout cannot - // hide by being wrong in both places. Field 2 is the 42-byte HpkePublicKey, field 3 the scalar. - const fixture = JSON.parse(demoPrivateKeyset); - const value = decodeBase64(fixture.key[0].keyData.value); - expect(Array.from(value.subarray(0, 2))).toEqual([0x12, 0x2a]); - expect(Array.from(value.subarray(44, 46))).toEqual([0x1a, 0x20]); - - const keyset = hpkeKeysetFromPrivateKey(value.subarray(46), fixture.primaryKeyId); - expect(JSON.stringify(keyset.privateKeyset)).toBe(demoPrivateKeyset); - expect(JSON.stringify(keyset.publicKeyset)).toBe(JSON.stringify(demoPublicKeyset)); - }); - - it('refuses key material Tink would reject', () => { - const privateKey = new Uint8Array(32); - expect(() => hpkeKeysetFromPrivateKey(privateKey.subarray(0, 31), 1)).toThrow(); - expect(() => hpkeKeysetFromPrivateKey(privateKey, 0)).toThrow(); - expect(() => hpkeKeysetFromPrivateKey(privateKey, 0x1_0000_0000)).toThrow(); + it('signs and verifies with strict RFC 8032 semantics', () => { + const pair = generateSigningKeyPair(); + const message = new TextEncoder().encode('{"schema_version":1}'); + const signature = sign(message, pair.privateKey); + expect(signature).toHaveLength(64); + expect(verify(message, signature, pair.publicKey)).toBe(true); + signature[0] ^= 1; + expect(verify(message, signature, pair.publicKey)).toBe(false); }); }); -describe('envelope', () => { - it('lays the header out big-endian ahead of the three payloads', () => { - const configuration = new TextEncoder().encode('{}'); - const signature = new Uint8Array(64).fill(7); - const envelope = encodeEnvelope('demo-signer-2026', configuration, signature); - - expect(new TextDecoder().decode(envelope.subarray(0, 8))).toBe('ADCCFG01'); - const header = new DataView(envelope.buffer, 8, 8); - expect(header.getUint16(0)).toBe(16); - expect(header.getInt32(2)).toBe(2); - expect(header.getUint16(6)).toBe(64); - expect(new TextDecoder().decode(envelope.subarray(16, 32))).toBe('demo-signer-2026'); - expect(envelope.subarray(32, 34)).toEqual(configuration); - expect(envelope.subarray(34)).toEqual(signature); - expect(envelope).toHaveLength(16 + 16 + 2 + 64); +describe('fixed ADCCFG01 envelope', () => { + it('has no signature-length field and round-trips exactly 64 signature bytes', () => { + const configuration = new TextEncoder().encode('{"a":1}'); + const signature = new Uint8Array(64).fill(9); + const bytes = encodeEnvelope('protocol-signer', configuration, signature); + expect(new TextDecoder().decode(bytes.subarray(0, 8))).toBe('ADCCFG01'); + expect(new DataView(bytes.buffer).getUint16(8)).toBe(15); + expect(new DataView(bytes.buffer).getUint32(10)).toBe(configuration.length); + expect(bytes).toHaveLength(14 + 15 + configuration.length + 64); + expect(decodeEnvelope(bytes)).toEqual({ + signerKeyId: 'protocol-signer', + configurationBytes: configuration, + signature + }); }); - it('refuses payloads the app would refuse to open', () => { - const signature = new Uint8Array(64); - expect(() => encodeEnvelope('ab', new Uint8Array(2), signature)).toThrow(); - expect(() => encodeEnvelope('a'.repeat(65), new Uint8Array(2), signature)).toThrow(); - expect(() => encodeEnvelope('demo', new Uint8Array(1), signature)).toThrow(); - expect(() => encodeEnvelope('demo', new Uint8Array(2), new Uint8Array(31))).toThrow(); - }); -}); - -describe('base64', () => { - it('round-trips every byte value', () => { - const bytes = Uint8Array.from({ length: 256 }, (_, index) => index); - expect(decodeBase64(encodeBase64(bytes))).toEqual(bytes); + it('rejects variable signatures and trailing bytes', () => { + expect(() => encodeEnvelope('protocol-signer', new Uint8Array(2), new Uint8Array(63))).toThrow( + 'envelope_signature' + ); + const valid = encodeEnvelope('protocol-signer', new Uint8Array(2), new Uint8Array(64)); + const trailing = new Uint8Array(valid.length + 1); + trailing.set(valid); + expect(() => decodeEnvelope(trailing)).toThrow('envelope_length'); }); }); diff --git a/web/tests/fixture.ts b/web/tests/fixture.ts new file mode 100644 index 0000000..b5bae5d --- /dev/null +++ b/web/tests/fixture.ts @@ -0,0 +1,52 @@ +import { + encodeBase64Url, + hpkeKeyPairFromPrivate, + signingKeyPairFromPrivate +} from '../src/lib/adc/crypto'; +import type { StudyConfiguration } from '../src/lib/adc/types'; + +const raw = (byte: number) => encodeBase64Url(new Uint8Array(32).fill(byte)); + +export const SIGNING = signingKeyPairFromPrivate(raw(0x11)); +export const HPKE = hpkeKeyPairFromPrivate(raw(0x22)); + +export function validConfiguration( + overrides: Partial = {} +): StudyConfiguration { + return { + schema_version: 1, + platform: 'android', + experiment_id: 'protocol-study', + configuration_id: 'protocol-study-000001', + assigned_participant_id: null, + issued_at: '2026-01-01T00:00:00Z', + expires_at: '2027-01-01T00:00:00Z', + minimum_client_version: '1', + title: 'Protocol v1 study', + researcher: { name: 'ADC Lab', contact: 'adc@example.test' }, + purpose: 'Protocol conformance', + duration_hours: 24, + consent: { document_version: '2026-01', summary: 'Collect lifecycle test events.' }, + collectors: [ + { id: 'app_lifecycle.v1', required: true, config: {} }, + { + id: 'location.v1', + required: false, + config: { + interval_millis: 60_000, + minimum_interval_millis: 30_000, + maximum_batch_delay_millis: 300_000, + minimum_displacement_millimeters: 25_000, + priority: 'BALANCED' + } + } + ], + surveys: [], + interventions: [], + storage: { maximum_local_bytes: 1_073_741_824 }, + signer: { key_id: 'protocol-signer', public_key: SIGNING.publicKey }, + export: { researcher_key_id: 'protocol-export', hpke_public_key: HPKE.publicKey }, + upload: null, + ...overrides + }; +} diff --git a/web/tests/hostile.spec.ts b/web/tests/hostile.spec.ts index 4a75cc4..7994670 100644 --- a/web/tests/hostile.spec.ts +++ b/web/tests/hostile.spec.ts @@ -1,240 +1,98 @@ import { describe, expect, it } from 'vitest'; -import demoStudyJson from '../../researcher-tools/examples/demo-study.json?raw'; -import { parseInstant } from '$lib/adc/canonical'; -import { validate } from '$lib/adc/schema'; -import { isUsableHpkePublicKeyset } from '$lib/adc/tink'; +import { + canonicalBytes, + canonicalConfigurationBytes, + canonicalizeConfiguration +} from '../src/lib/adc/canonical'; +import { sign } from '../src/lib/adc/crypto'; +import { encodeEnvelope } from '../src/lib/adc/envelope'; +import { validate } from '../src/lib/adc/schema'; +import type { StudyConfiguration } from '../src/lib/adc/types'; import { parseConfiguration } from '../src/routes/researcher/parse'; -import type { StudyConfiguration, TinkKeyset } from '$lib/adc/types'; +import { SIGNING, validConfiguration } from './fixture'; -/** - * A study file the editor did not write. - * - * Opening a previous configuration is a first-class flow — the cross-language workflow is exactly - * that — so every byte in these files is attacker-shaped as far as this page is concerned. The - * failures below all shared one outcome before they were closed: the page signed the document - * without complaint and something downstream, a device or a decryption weeks later, refused it. - */ - -const encoder = new TextEncoder(); -const demo = JSON.parse(demoStudyJson) as Record; - -function load(mutate: (document: Record) => void): StudyConfiguration { - const document = JSON.parse(demoStudyJson) as Record; - mutate(document); - return parseConfiguration(encoder.encode(JSON.stringify(document))); -} - -function refuses(mutate: (document: Record) => void): void { - expect(() => load(mutate)).toThrow(); -} - -const keyset = demo.export as { tink_hpke_public_keyset: TinkKeyset }; -const goodKeyset = keyset.tink_hpke_public_keyset; - -function withKeyset(replacement: unknown): StudyConfiguration { - return load((document) => { - (document.export as Record).tink_hpke_public_keyset = replacement; - }); -} - -function codes(configuration: StudyConfiguration): string[] { - return validate(configuration).map((issue) => issue.code); -} - -function clone(value: T): T { - return JSON.parse(JSON.stringify(value)) as T; -} - -describe('the demo study', () => { - it('is valid, so every failure below is the mutation and not the fixture', () => { - expect(validate(parseConfiguration(encoder.encode(demoStudyJson)))).toEqual([]); - }); -}); - -describe('a collector the codec cannot write', () => { - it('refuses the file rather than encoding the token undefined', () => { - refuses((document) => { - document.collectors = [{ id: 'bogus.v1', required: true, config: {} }]; - }); - }); - - it('refuses a missing collector config instead of inventing defaults', () => { - refuses((document) => { - document.collectors = [{ id: 'accelerometer.v1', required: true }]; - }); - }); - - it('survives a collector that is a bare string', () => { - refuses((document) => { - document.collectors = ['accelerometer.v1']; - }); - }); - - it('keeps an out-of-range value so validate can report it', () => { - const configuration = load((document) => { - document.collectors = [ - { id: 'keyboard_touch.v1', required: false, config: { trajectory_sampling_hz: 9_000 } } - ]; - }); - expect(codes(configuration)).toEqual(['number_range']); - }); -}); - -describe('transports', () => { - it('refuses a name NetworkTransport.valueOf would refuse', () => { - refuses((document) => { - document.collectors = [ - { id: 'network_usage.v1', required: false, config: { transports: ['fibre'] } } - ]; - }); - }); - - /** A string has a `length` and an `includes`, so it passed both the check and the encoder. */ - it('does not let a string pass as a one-element set', () => { - refuses((document) => { - document.collectors = [ - { id: 'network_usage.v1', required: false, config: { transports: 'wifi' } } - ]; - }); - }); - - it('reports an empty set rather than encoding one', () => { - const configuration = load((document) => { - document.collectors = [ - { - id: 'network_usage.v1', - required: false, - config: { transports: [], poll_interval_minutes: 15 } - } - ]; - }); - expect(codes(configuration)).toEqual(['transports_empty']); - }); -}); - -describe('location priority', () => { - it('refuses a value LocationPriority.valueOf would refuse', () => { - refuses((document) => { - document.collectors = [ - { id: 'location.v1', required: false, config: { priority: 'FASTEST' } } - ]; - }); - }); -}); - -describe('legacy prompts', () => { - it('rejects the removed prompt shape instead of silently migrating it', () => { - refuses((document) => { - delete document.interventions; - document.prompts = [{ id: 'old-prompt', delay_minutes: 5, message: 'hi' }]; - }); - }); -}); - -describe('closed-world v1 shape', () => { - it('refuses unknown nested fields instead of dropping them before signing', () => { - refuses((document) => { - (document.researcher as Record).legacy_name = 'discard me'; - }); - }); - - it('refuses a mistyped scalar instead of replacing it with a default', () => { - refuses((document) => { - document.duration_hours = '24'; - }); - }); - - it.each(['', 'contains space', 'é', 'a'.repeat(65)])('reports an invalid assigned ID %j', (id) => { - const configuration = load((document) => { document.assigned_participant_id = id; }); - expect(codes(configuration)).toContain('id_format'); - }); - - it('reports a schedule whose durable lifetime occurrence set exceeds the metadata bound', () => { - const configuration = load((document) => { - document.interventions = [{ - id: 'too-frequent', - action: { type: 'notification', notification_title: 'Check-in', notification_message: 'Check in now.' }, - triggers: [{ - id: 'every-minute', - schedule: { type: 'interval', start_offset_minutes: 0, interval_minutes: 1, clock: 'CALENDAR_TIME' }, - availability_minutes: 5 - }] - }]; - }); - expect(codes(configuration)).toContain('schedule_bounds'); - }); -}); - -/** - * `ExportConfiguration` bounds this document's length and nothing else, so before these checks each - * of the six below signed cleanly here and was refused by real Tink on the phone at export time. - */ -describe('the export keyset', () => { - it('accepts the one the CLI ships', () => { - expect(isUsableHpkePublicKeyset(goodKeyset)).toBe(true); - expect(codes(withKeyset(goodKeyset))).toEqual([]); - }); +describe('closed-world configuration parser', () => { + const wire = () => JSON.parse(canonicalizeConfiguration(validConfiguration())) as Record; it.each([ + ['unknown root member', (value: Record) => (value.future = true)], + ['wrong platform', (value: Record) => (value.platform = 'ios')], + ['numeric client version', (value: Record) => (value.minimum_client_version = 1)], + ['padded client version', (value: Record) => (value.minimum_client_version = '01')], [ - 'a primary that names no key', - (value: TinkKeyset) => { - value.primaryKeyId = 999; - } - ], - [ - 'a disabled key', - (value: TinkKeyset) => { - value.key[0].status = 'DISABLED'; + 'old minimum app version', + (value: Record) => { + delete value.minimum_client_version; + value.minimum_app_version = 1; } ], [ - 'a RAW output prefix', - (value: TinkKeyset) => { - value.key[0].outputPrefixType = 'RAW'; - } - ], - [ - 'a symmetric key', - (value: TinkKeyset) => { - value.key[0].keyData.typeUrl = 'type.googleapis.com/google.crypto.tink.AesGcmKey'; - value.key[0].keyData.keyMaterialType = 'SYMMETRIC'; - } - ], - [ - 'a truncated key', - (value: TinkKeyset) => { - const raw = atob(value.key[0].keyData.value); - value.key[0].keyData.value = btoa(raw.slice(0, raw.length - 1)); - } - ], - [ - 'AES-128-GCM instead of AES-256-GCM', - (value: TinkKeyset) => { - const raw = [...atob(value.key[0].keyData.value)].map((c) => c.charCodeAt(0)); - raw[raw.indexOf(0x18) + 1] = 0x01; - value.key[0].keyData.value = btoa(String.fromCharCode(...raw)); + 'old Tink export keyset', + (value: Record) => { + value.export = { + researcher_key_id: 'protocol-export', + tink_hpke_public_keyset: { primaryKeyId: 1, key: [] } + }; } ] - ])('refuses %s', (_name, mutate) => { - const broken = clone(goodKeyset); - mutate(broken); - expect(isUsableHpkePublicKeyset(broken)).toBe(false); - expect(codes(withKeyset(broken))).toContain('keyset_unusable'); + ])('rejects %s', (_name, mutate) => { + const value = wire(); + mutate(value); + expect(() => parseConfiguration(canonicalBytes(value))).toThrow(); + }); + + it('rejects noncanonical member order, duplicate members, malformed UTF-8, and floats', () => { + expect(() => parseConfiguration(new TextEncoder().encode('{"b":2,"a":1}'))).toThrow(); + expect(() => parseConfiguration(new TextEncoder().encode('{"a":1,"a":1}'))).toThrow(); + expect(() => parseConfiguration(Uint8Array.of(0x7b, 0xff, 0x7d))).toThrow(); + const value = wire(); + value.duration_hours = 1.5; + expect(() => parseConfiguration(canonicalBytes(value))).toThrow(); + }); + + it('verifies signer ID and Ed25519 signature when the input is ADCCFG01', () => { + const configuration = validConfiguration(); + const payload = canonicalConfigurationBytes(configuration); + const signature = sign(payload, SIGNING.privateKey); + expect(parseConfiguration(encodeEnvelope(configuration.signer.key_id, payload, signature))).toEqual( + configuration + ); + expect(() => parseConfiguration(encodeEnvelope('other-signer', payload, signature))).toThrow( + 'envelope_signer' + ); + signature[0] ^= 1; + expect(() => + parseConfiguration(encodeEnvelope(configuration.signer.key_id, payload, signature)) + ).toThrow('envelope_signature'); }); }); -/** - * `DateTimeFormatter.ISO_INSTANT` writes an offset as `+HH:MM[:ss]` and `ZoneOffset` caps the whole - * offset at ±18:00. Accepting a spelling `Instant.parse` refuses means taking a hand-written file - * the CLI would not. - */ -describe('instant offsets', () => { - it.each(['+08', '+0800', '+080000', '-18:00:01', '-18:16'])('refuses %s', (offset) => { - expect(parseInstant(`2026-01-01T00:00:00${offset}`)).toBeNull(); +describe('configuration validation', () => { + it('accepts the complete Protocol v1 fixture', () => { + expect(validate(validConfiguration())).toEqual([]); + }); + + it.each(['', '0', '01', '+1', '-1', '2147483648'])('rejects client build %j', (value) => { + const issues = validate(validConfiguration({ minimum_client_version: value })); + expect(issues.some((issue) => issue.path === 'minimum_client_version')).toBe(true); }); - it.each(['Z', '+08:00', '-08:00', '+18:00', '-18:00', '+05:45'])('accepts %s', (offset) => { - expect(parseInstant(`2026-01-01T00:00:00${offset}`)).not.toBeNull(); + it.each(['AA==', 'A'.repeat(42), 'not+a-key'])('rejects noncanonical raw key %j', (value) => { + const configuration = validConfiguration({ + signer: { key_id: 'protocol-signer', public_key: value } + }); + expect(validate(configuration)).toContainEqual({ path: 'signer.public_key', code: 'key_invalid' }); + }); + + it('enforces integer millimetres and their physical bound', () => { + const configuration = validConfiguration(); + const location = configuration.collectors[1] as Extract< + StudyConfiguration['collectors'][number], + { id: 'location.v1' } + >; + location.config.minimum_displacement_millimeters = -1; + expect(validate(configuration).some((issue) => + issue.path.endsWith('minimum_displacement_millimeters') + )).toBe(true); }); }); diff --git a/web/tests/i18n.spec.ts b/web/tests/i18n.spec.ts index ba8acd3..23da22f 100644 --- a/web/tests/i18n.spec.ts +++ b/web/tests/i18n.spec.ts @@ -12,6 +12,7 @@ import { LOCALES, type Messages } from '$lib/i18n/types'; /** Sample arguments for every message that is a template, keyed by its path. */ const ARGUMENTS: Record = { 'control.stepPosition': { index: 2, total: 4 }, + 'intervention.randomWindowSummary': { minimum: 0, maximum: 14 }, 'issue.length_range': { min: 1, max: 120 }, 'issue.number_range': { min: 5_000, max: 1_000_000 }, 'issue.document_too_large': { max: 1_048_576 }, diff --git a/web/tests/ids.spec.ts b/web/tests/ids.spec.ts index aa114df605f9b335c1966471c5497f5c5ef9c5c8..2746bbc32ec6a672389195834dc5c8f6a7084666 100644 GIT binary patch literal 2708 zcmb_e-EJc_6u$3M97QY5L`;$vBqVlKq)>_ot3?&<-R`pMj8774C-z`FZ9^1^hv0$> z5X2bDM#%u!Wgc zOGs{oMPrj^CnA`i06A!Jix*1H#C)Zh6-rKc)?8c?iWW%#w7_bbeh$tn(5k?_6AhX`ZQrjtY1lG9Bd08pX{1_RJBNyT(oWM!SajXb>dHkt@-?*67s5f8F2rnyM~tn`TFRb%(sDMq)$w9Ce|FT(OVRPN93;t* zPh!GDnIUeYWrvCj-c5e>!UUdcv@0zQNEPi$rKct19C=0?QIIYXwKE`jy3`Qf@*}hX8vO#Zo0AG zVMpvL4*2H{{BHphv$am_}T|G0B#^vi%S@*1mi6Ok#41-Z6-iooa7UWqcNpVIPs)=Z z8!Y3CFJgHm_M)&a2U*Nh8HH`Xc+mpMSdK)P&OF81kHpk{cp?X(i1UXjAKbBAyc5W7 zo%_AI^b>~{*5hFm${r8obFhCYrin;DOrkJnKcAS7bfzordyy}8c_KE~HPiI{=(3mG zr>`Ph@b%-3#umNo%Wxp#aV*1>z77Pq=c%A=;0^Y1#AQ6cgR~svWpn@lO5j-8<|qYh8(W%2JsIf^o>ijw2mVJTEgI`Yas^){n9nA9%{LMBq`xxSvA&+@4#k?R22l}OAKqi=&4rz&QmeLyIEbOJ3vJ=fQtq?}t$e*gM_%gwn4lC+`n&#pU8%)lEO_c|G#X1Z{^FgR$~1qvxwdp$z0i9n>9N(u>L#l*9*@{6^TxD` zrOKUiXqk0_UPU`J*KbvpnYYAt-e)&hL8e}197lfE6A2VC5Ca~#iqnr`CJJ3G&4tgv zTTdUJR^_N~PH{<5$G(|Bm|-TE4;Ad zJ`sXt3Z0mQ@K=SNE>eYNrNkXyuvzC=z{@y`5$uylx%7mOpol6e5Szh-Pm|(>_L;8v z<|ea<746onro3r{w!b%MVonO)Tie)c$+1;PfCxIs=r;OS_wP392JtIYXlyZ;JTjyoDv;kQL$ zAd1ade_4VYZNIdtc4&Uo)4_Mg@ut?>VMu`=;rQlZjAzuoG_s~oe&=+g8!nhP+9^c~M5!ZY)4n!@A2Nia^hz>?41U86> z-Fyd;Apt}NiK`V*OtKM9b&9ViucZuX6 z|56mgizWLR5n0o9k^t|9)oQiUN;<94ggPWWvRX^mipgXy>U{+#0c8ViiuLd++8Y(q5Vtl`xd~3c_2XZdjutFfd~ibP>(o3B613~coMfh%Xr{zuFtxzkkAiw zEC>(}5)Gb7ZUsNIFBLW6Us%-dzm2F~0}1{wqy;cl%e{E9i?(k(QX!}$<#C$)5b1&k z@Z}B#47F}G$67|oRDQ#d8JsHR;(oP@6o}k65Je47p@H}z5dwEYbyK84_=2oaAZKI= zP6-}`@(htDJUqEnpe@gm(ui#3E@BAMRs3h&sj_Zh{1=qIK&O!zOjW7^qtFj%nUXEB z_P&}xnU=vli2lPU@D;LExL2=(Bq~5Y3#QnQGN>|P)3(qWVhh6+6(-RoT{ zoS?i1S)FD8+Jjg|Q*5H8$UKn8@1gib|Ep#TJv? zs0ciwJ+M0fdO*>vW+E0=uAIDapp~Buc{Jl{;SAq;U#Nr%b1R$o7jTdu2%-j38}Vy7 z%0`_aLgA;=og(7_?fX#fmB#A2`Lkd?>=6Mn=^(;^mDR>tquN-(&uOEvt$sdPv~Q|l zh`5Go1cC!iH2HW{zgRQ{3jr4+HMQ2)nV+r4RPkbtLazoB74a3-A|UkII7^1!jX7on zB?17jE7=oOm2>2808JFMxKZ^GGpdjB7(Iq@&?Nh{qmUEf4pml|c{$SMCcGXBL^A|L zs}PJK4i8ZGg0h|!Oc}O-8AM#AV^j$`sTlpE!02ZIU{uZ;I0p+PRsH2E%E#m1n}(Psrk%=us0K+Wv-e8H00IEXkY(G4n#{y4&15^Wst<3IsWi*OA35s%5D=bGa;29+$q6&Dkcq@gG8|$zDOZ)Me*GmZPSWu^VrR_L45``*T zH6oX#PDp(~=tg!;dO=Y&7Rs^Ewvfzap*bLXJinn3r+gPFqNQt(oxJ@20(WY9vxltX zCQedi6}pB5XgCn43{YW3XB8@gw|v0zc^pqo@@jzM1mzXOF=ZY#woBOtjgcs+;vbe! z6l&V@2lHw@sU&AVtfog5h za)|}-7mh?(F!GG~ZNF85fjD9k#y2Wqq4(&LbE!_dC>4%)C36dUrA3fa+>@r+%8Tn1J}C2oK;RROfc^CW@~qS7{%wNbuL zfcHTXqU}p$rR<`8c+%Q?dfck|ViYO0#^5R%Ce$6EhHeW@%?aR=pT>T>B?1l@JX`Q*g;W1AGe|lf+wxh zZMP!={|ymiAC$U3q9)chTj{K>t#>vy?{)5PH7HnO3y6?PJ!v%k!=rtkazsJ%0KpWQ z@6#B4h6;VE4WbC`1U{Ata@bK_f34|`5pZ32S)<0#krEjec7p5`4>BrKvnk3W05dHmtK)2(;o4-Q+p{-3jn z>Z8&S%FaQwdGhS>aT9N3iJFqnnau{RyL9)t{ODPeW!3z!--i*oG+_<7V zYFusH3zDlf{(SS^R)3(kI76(ZF!m$TE=^3^X-){v6Pv3xwl>1iW1l{K_2&WjY{OrH)lyz8Hulge8h zda!XY=eTS@04_-jVda`YoKSHljdZs}ah3C&4_i>}vh&KVFWLpTAy_z5_3~C5IbCv0 z2(b9QsDM){(M(Vop+ZfLEW|asyGn@+_JNlG=)*6k`kjTCp-fzX7cER$^sa_ZWRU93 zg??pn(+eu;C{e|k@@Tj@;PeIXveY(@#Yl|0U=1*eppaeww4wFz`lu_(Yc@i`dx)CN zAfiH)VCd2lC`WI?mZaeG0@VwrI+Blmk3zD;i?lk}`NFS)|0{TmlO(!kNpAh{E+y1%0O9>WvdodYx5Z$PL7oJTD5d5U>}PDGBxzSMD!Q=?AB#`=d|yA7s8e#ix&D1- zHH+TKI%VPNGDE%^(!x`6T9WF_x`~17O?Wbi(Zhj*BH+2r*4N=VsfY-Fu-`f^Z#ZNv zU*pP?qetxpr2}`z-8AA}rI9nWPE5I6DJ3rn!Bx$hnVbT(9*7%R#5qEZONIi_8>k>N zbyAnNAfni^wq4@LykXQ;b7|(!HG`__=DashS1oN)o*4==V0f9?1m^kk=cx0cerA?1 zg{|4`<7}I3g5qQq%1T{EQ?f)Xqvo#e6RcUiU0c;hTspu1PrLqL&~7$A+S&Q;trxzB z_`l2cBWmfO3xsNzxm9KogeW8gORY*}!gNnA5duNjskrf`9qeWxs3JKj}1M}1?CT}!Qk($Os zom3sqm;FA>{ggD?@boQXsXShcvps**T_7@N%5nC*s|viElb4^Ol;gTQ=* z)KQy|wx_7usH}qrFVbzlR`@GrI(>T+C+XXcn9oyK)9;xfO%xeCBi>DL0@i!XY#~3u6)Bqq^ux z-rc>IYtr4)@IYRY70|t&}W1lr!8%@^OUv046)vbqYe{+46ZLP23S!?AUTW>a778~g0oGTA*WK{}Q zl=wK?4qxsk8cEnE?m@5gbH7I(vW?7O5M>q-nStnRo1w^5ynswxDj( zl%WgFvgWu)*MVe6$zncJ3j@(fu1PUQ;lHyMD8vNyK!`h7B|kRl&uMc zSq8|ra0?74MOT_j!OP8i>z|(Cav-he{AL5h+b5HJ(y&tqkO>f-x0Nzx1_7`cz!qU1@Y0xV?K`B>3ilz>P;V z6!g&7xm&ec8=_WuBx*Blc7 diff --git a/web/tests/join.spec.ts b/web/tests/join.spec.ts new file mode 100644 index 0000000..f979536 --- /dev/null +++ b/web/tests/join.spec.ts @@ -0,0 +1,85 @@ +import { describe, expect, it } from 'vitest'; +import { createJoinLink, encodeJoinLink, parseJoinLink } from '../src/lib/adc/join'; + +describe('immutable join links', () => { + it('round-trips the exact artifact digest and signer fingerprint', () => { + const link = createJoinLink( + 'https://artifacts.example.invalid/join/dGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4', + new Uint8Array([1, 2, 3]), + '0123 4567 89AB CDEF FEDC BA98 7654 3210' + ); + + expect(parseJoinLink(link)).toEqual({ + artifactUrl: + 'https://artifacts.example.invalid/join/dGhpcy1pcy1hLTEyOC1iaXQtdG9rZW4', + artifactSha256: '039058c6f2c0cb492c533b0a4d14ef77cc0f78abccced5287d84a1a2011cfb81', + signerFingerprint: '0123456789ABCDEFFEDCBA9876543210' + }); + }); + + it('rejects URLs that WHATWG and URI implementations normalize differently', () => { + for (const artifactUrl of [ + 'https://EXAMPLE.invalid:443/a/../config.adccfg', + 'https://artifacts.example.invalid/config.adccfg?download=1', + 'https://artifacts.example.invalid/a//config.adccfg', + 'https://artifacts.example.invalid/a/%63onfig.adccfg', + 'https://127.0.0.1/config.adccfg' + ]) { + expect(() => + encodeJoinLink({ + artifactUrl, + artifactSha256: '0'.repeat(64), + signerFingerprint: 'A'.repeat(32) + }) + ).toThrow('join_artifact_url_invalid'); + } + }); + + it('requires opaque personalized paths and never carries the roster code', () => { + expect(() => + createJoinLink( + 'https://artifacts.example.invalid/alice-001', + new Uint8Array([1]), + 'A'.repeat(32), + 'alice-001' + ) + ).toThrow('join_url_exposes_participant_id'); + expect(() => + createJoinLink( + 'https://artifacts.example.invalid/config.adccfg', + new Uint8Array([1]), + 'A'.repeat(32), + 'roster-code' + ) + ).toThrow('join_url_requires_opaque_path'); + expect( + createJoinLink( + 'https://artifacts.example.invalid/MDEyMzQ1Njc4OWFiY2RlZg', + new Uint8Array([1]), + 'A'.repeat(32), + 'roster-code' + ) + ).not.toContain('roster-code'); + }); + + it('rejects ambiguous, mutable, or noncanonical encodings', () => { + const valid = encodeJoinLink({ + artifactUrl: 'https://artifacts.example.invalid/config.adccfg', + artifactSha256: '0'.repeat(64), + signerFingerprint: 'A'.repeat(32) + }); + const artifact = 'https%3A%2F%2Fartifacts.example.invalid%2Fconfig.adccfg'; + for (const hostile of [ + valid.replace('adc://', 'https://'), + valid.replace('artifact=', 'unknown=x&artifact='), + valid.replace('&sha256=', `&sha256=${'0'.repeat(64)}&sha256=`), + valid.replace('https%3A', 'http%3A'), + valid.replace(artifact, 'https%3A%2F%2Fuser%40artifacts.example.invalid%2Fconfig.adccfg'), + valid.replace(artifact, `${artifact}%23mutable`), + valid.replace('%2F', '%2f'), + `${valid}&extra=1` + ]) { + expect(() => parseJoinLink(hostile), hostile).toThrow(); + } + }); +}); diff --git a/web/tests/node.d.ts b/web/tests/node.d.ts deleted file mode 100644 index 670a24d..0000000 --- a/web/tests/node.d.ts +++ /dev/null @@ -1,46 +0,0 @@ -/** - * The slice of Node that `compat.spec.ts` uses. - * - * That test shells out to the `researcher-tools` CLI, which is the only way to compare this - * encoder against the one the Android app runs. The project has no `@types/node` — nothing it - * ships touches Node — and adding one for a single test would put a dependency in everyone's - * install for a file that never reaches the browser. These are the five modules and twelve - * functions that test calls, and nothing else. - */ - -declare module 'node:child_process' { - export function execFile( - command: string, - args: readonly string[], - options: { cwd?: string }, - callback: (error: Error | null, stdout: string, stderr: string) => void - ): void; - export function execFileSync( - file: string, - args: readonly string[], - options?: { cwd?: string; encoding?: 'utf8'; stdio?: string | readonly string[] } - ): string; -} - -declare module 'node:fs' { - export function existsSync(path: string): boolean; - export function mkdirSync(path: string, options?: { recursive?: boolean }): void; - export function mkdtempSync(prefix: string): string; - export function readFileSync(path: string): Uint8Array; - export function readFileSync(path: string, encoding: 'utf8'): string; - export function rmSync(path: string, options?: { recursive?: boolean; force?: boolean }): void; - export function writeFileSync(path: string, data: string | Uint8Array, encoding?: 'utf8'): void; -} - -declare module 'node:os' { - export function tmpdir(): string; -} - -declare module 'node:path' { - export function dirname(path: string): string; - export function join(...segments: string[]): string; -} - -declare module 'node:url' { - export function fileURLToPath(url: string | URL): string; -} diff --git a/web/tests/p2.spec.ts b/web/tests/p2.spec.ts new file mode 100644 index 0000000..4402c8b --- /dev/null +++ b/web/tests/p2.spec.ts @@ -0,0 +1,157 @@ +import { describe, expect, it } from 'vitest'; +import { canonicalBytes, canonicalConfigurationBytes, canonicalizeConfiguration } from '../src/lib/adc/canonical'; +import { maximumReachableLocalDates, validate } from '../src/lib/adc/schema'; +import type { CollectorConfig, InterventionConfig, StudyConfiguration } from '../src/lib/adc/types'; +import { parseConfiguration } from '../src/routes/researcher/parse'; +import { nextRandomWindow } from '../src/routes/researcher/random-window'; +import { validConfiguration } from './fixture'; + +function p2Collectors(): CollectorConfig[] { + return [ + { id: 'battery_state.v1', required: false, config: {} }, + { id: 'temporal_context.v1', required: false, config: {} }, + { + id: 'gyroscope.v1', + required: false, + config: { sampling_period_us: 100_000, maximum_report_latency_us: 1_000_000 } + }, + { + id: 'ambient_light.v1', + required: false, + config: { sampling_period_us: 1_000_000, change_threshold_millilux: 1_000 } + }, + { + id: 'proximity.v1', + required: false, + config: { minimum_event_interval_ms: 1_000, change_threshold_millimeters: 0 } + } + ]; +} + +function randomIntervention(): InterventionConfig { + return { + id: 'random-ema', + action: { + type: 'notification', + notification_title: 'Check in', + notification_message: 'Please complete the check-in.' + }, + triggers: [ + { + id: 'random-ema-window', + availability_minutes: 60, + schedule: { + type: 'random_window', + local_windows: [ + { start_local_time: '08:00', end_local_time: '12:00' }, + { start_local_time: '14:00', end_local_time: '18:00' } + ], + occurrences_per_window: 2, + maximum_occurrences_per_day: 4, + maximum_occurrences_total: 14, + minimum_separation_minutes: 60 + } + } + ] + }; +} + +function p2Configuration(overrides: Partial = {}): StudyConfiguration { + const base = validConfiguration(); + return { + ...base, + collectors: [base.collectors[0], ...p2Collectors()], + interventions: [randomIntervention()], + ...overrides + }; +} + +describe('P2 Protocol v1 authoring', () => { + it('round-trips all five collectors and random windows through the closed-world parser', () => { + const configuration = p2Configuration(); + expect(validate(configuration)).toEqual([]); + expect(parseConfiguration(canonicalConfigurationBytes(configuration))).toEqual(configuration); + }); + + it('refuses unknown collector config fields instead of silently dropping them', () => { + const wire = JSON.parse(canonicalizeConfiguration(p2Configuration())) as { + collectors: Array<{ id: string; config: Record }>; + }; + const proximity = wire.collectors.find((collector) => collector.id === 'proximity.v1'); + expect(proximity).toBeDefined(); + proximity!.config.infer_presence = true; + expect(() => parseConfiguration(canonicalBytes(wire))).toThrow('parse_keys'); + }); + + it('enforces the collector-specific physical-unit bounds', () => { + const configuration = p2Configuration(); + const ambient = configuration.collectors.find( + (collector): collector is Extract => + collector.id === 'ambient_light.v1' + )!; + const proximity = configuration.collectors.find( + (collector): collector is Extract => + collector.id === 'proximity.v1' + )!; + ambient.config.sampling_period_us = 199_999; + proximity.config.change_threshold_millimeters = 10_001; + const paths = validate(configuration).map((issue) => issue.path); + expect(paths).toContain('collectors.4.config.sampling_period_us'); + expect(paths).toContain('collectors.5.config.change_threshold_millimeters'); + }); + + it('rejects overlapping, undersized, and over-capacity random windows', () => { + const configuration = p2Configuration(); + const schedule = configuration.interventions[0].triggers[0].schedule; + if (schedule.type !== 'random_window') throw new Error('fixture'); + schedule.local_windows[0].end_local_time = '08:30'; + schedule.local_windows[1].start_local_time = '08:15'; + schedule.maximum_occurrences_per_day = 17; + const issues = validate(configuration); + expect(issues.some((issue) => issue.code === 'window_order')).toBe(true); + expect(issues.some((issue) => issue.code === 'schedule_bounds')).toBe(true); + }); + + it('uses each signed random total for the global bound under arbitrary clock edits', () => { + expect(maximumReachableLocalDates(60)).toBe(3); + const configuration = p2Configuration({ duration_hours: 1 }); + const intervention = randomIntervention(); + const template = intervention.triggers[0]; + configuration.interventions = [{ + ...intervention, + triggers: Array.from({ length: 2 }, (_, index) => ({ + ...template, + id: `random-trigger-${index + 1}`, + schedule: { + type: 'random_window' as const, + local_windows: [{ start_local_time: '08:00', end_local_time: '09:00' }], + occurrences_per_window: 8, + maximum_occurrences_per_day: 8, + maximum_occurrences_total: 512, + minimum_separation_minutes: 1 + } + })) + }]; + + expect(validate({ ...configuration, interventions: [{ + ...configuration.interventions[0], + triggers: configuration.interventions[0].triggers.slice(0, 1) + }] })).toEqual([]); + expect(validate(configuration)).toContainEqual({ path: 'interventions', code: 'schedule_bounds' }); + }); + + it('never suggests a random window whose end would be Protocol-invalid 24:00', () => { + const schedule = randomIntervention().triggers[0].schedule; + if (schedule.type !== 'random_window') throw new Error('fixture'); + schedule.local_windows = [{ start_local_time: '08:00', end_local_time: '23:00' }]; + schedule.occurrences_per_window = 1; + + expect(nextRandomWindow(schedule)).toBeNull(); + + schedule.local_windows[0].end_local_time = '22:59'; + expect(nextRandomWindow(schedule)).toEqual({ + start_local_time: '23:58', + end_local_time: '23:59' + }); + }); +}); diff --git a/web/tests/participant-copy.spec.ts b/web/tests/participant-copy.spec.ts new file mode 100644 index 0000000..3cfe5c4 --- /dev/null +++ b/web/tests/participant-copy.spec.ts @@ -0,0 +1,11 @@ +import { describe, expect, it } from 'vitest'; +import { en, zhTW } from '$lib/participant/copy'; + +describe('participant upload disclosure', () => { + it('keeps the installation code inside ciphertext in both locales', () => { + expect(en.delivery.upload.code).toContain('after decrypting'); + expect(en.delivery.upload.metadata).toContain('cannot see your installation code'); + expect(zhTW.delivery.upload.code).toContain('解密後'); + expect(zhTW.delivery.upload.metadata).toContain('看不到安裝代碼'); + }); +}); diff --git a/web/tests/researcher-draft.spec.ts b/web/tests/researcher-draft.spec.ts index 32be762..f7a5d26 100644 --- a/web/tests/researcher-draft.spec.ts +++ b/web/tests/researcher-draft.spec.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'vitest'; import { flushSync } from 'svelte'; import { createDraft } from '../src/routes/researcher/draft.svelte'; -import { canonicalize } from '$lib/adc/canonical'; +import { canonicalizeConfiguration } from '$lib/adc/canonical'; import { decodeEnvelope } from '../src/routes/researcher/parse'; import { verify } from '$lib/adc/crypto'; @@ -83,7 +83,7 @@ describe('draft', () => { expect(decoded.signerKeyId).toMatch(/^signer-[0-9a-z]{13}$/); expect(verify(decoded.configurationBytes, decoded.signature, draft.configuration.signer.public_key)).toBe(true); expect(new TextDecoder().decode(decoded.configurationBytes)).toBe(draft.canonical); - expect(draft.canonical).toBe(canonicalize(draft.document)); + expect(draft.canonical).toBe(canonicalizeConfiguration(draft.document)); draft.configuration.title = 'T2'; flushSync(); @@ -126,7 +126,7 @@ describe('draft', () => { expect(draft.configuration.export.researcher_key_id).toBe(''); const signerName = draft.signerKeyId; - const privateHalf = draft.signing.kind === 'held' ? draft.signing.material.privatePkcs8Base64 : ''; + const privateHalf = draft.signing.kind === 'held' ? draft.signing.material.privateKey : ''; draft.generateSigning(); flushSync(); expect(draft.signerKeyId).not.toBe(signerName); diff --git a/web/tests/researcher.spec.ts b/web/tests/researcher.spec.ts index a5e0642..1c30a95 100644 --- a/web/tests/researcher.spec.ts +++ b/web/tests/researcher.spec.ts @@ -1,170 +1,58 @@ import { describe, expect, it } from 'vitest'; -import demoSigningPrivateKey from '../../researcher-tools/examples/INSECURE-demo-signing-private.key?raw'; -import demoHpkePrivateKeyset from '../../researcher-tools/examples/INSECURE-demo-hpke-private.json?raw'; -import demoStudyJson from '../../researcher-tools/examples/demo-study.json?raw'; -import { canonicalBytes, canonicalize, keysetJson } from '$lib/adc/canonical'; +import { canonicalConfigurationBytes, canonicalizeConfiguration } from '$lib/adc/canonical'; +import { generateHpkeKeyPair, generateSigningKeyPair, sign } from '$lib/adc/crypto'; import { encodeEnvelope } from '$lib/adc/envelope'; -import { fingerprint, generateSigningKeyPair, sign, verify } from '$lib/adc/crypto'; -import { generateHpkeKeyset } from '$lib/adc/tink'; -import { DEFAULT_LOCAL_BYTES, emptyConfiguration, validate } from '$lib/adc/schema'; -import { hpkeKeysetFromPrivate, signingKeyPairFromPrivate } from '../src/routes/researcher/keys'; -import { decodeEnvelope, parseConfiguration } from '../src/routes/researcher/parse'; -import { units } from '../src/routes/researcher/units'; -import { estimate, volumeOf } from '../src/routes/researcher/estimate'; -import { stepForPath } from '../src/routes/researcher/steps'; -import { en } from '$lib/i18n/en'; -import { zhTW } from '$lib/i18n/zh-TW'; - -const demoStudy = JSON.parse(demoStudyJson); -const demoStudyBytes = new TextEncoder().encode(demoStudyJson); - -describe('key import', () => { - it('derives the same public half the CLI would, from the committed demo key', () => { - const pair = signingKeyPairFromPrivate(demoSigningPrivateKey); - expect(pair.publicX509Base64).toBe(demoStudy.signer.public_key); - expect(pair.privatePkcs8Base64).toBe(demoSigningPrivateKey.trim()); - expect(fingerprint(pair.publicX509Base64)).toBe(fingerprint(demoStudy.signer.public_key)); - }); - - it('rebuilds the demo HPKE keyset from its private half, byte for byte', () => { - const keyset = hpkeKeysetFromPrivate(demoHpkePrivateKeyset); - expect(keysetJson(keyset.privateKeyset)).toBe(demoHpkePrivateKeyset.trim()); - expect(keysetJson(keyset.publicKeyset)).toBe( - keysetJson(demoStudy.export.tink_hpke_public_keyset) +import { DEFAULT_LOCAL_BYTES, emptyConfiguration } from '$lib/adc/schema'; +import { artifactBytes } from '../src/routes/researcher/artifacts'; +import { hpkeKeyPairFromPrivate, signingKeyPairFromPrivate } from '../src/routes/researcher/keys'; +import { parseConfiguration } from '../src/routes/researcher/parse'; +import { validConfiguration } from './fixture'; + +describe('researcher Protocol v1 workflow', () => { + it('generates portable raw private artifacts with no wrapper or newline', () => { + const signing = generateSigningKeyPair(); + const hpke = generateHpkeKeyPair(); + expect(signing.privateKey).toMatch(/^[A-Za-z0-9_-]{43}$/); + expect(hpke.privateKey).toMatch(/^[A-Za-z0-9_-]{43}$/); + expect(signingKeyPairFromPrivate(signing.privateKey)).toEqual(signing); + expect(hpkeKeyPairFromPrivate(hpke.privateKey)).toEqual(hpke); + }); + + it('signs, packages, verifies, and reopens one canonical configuration', () => { + const configuration = validConfiguration(); + const pair = generateSigningKeyPair(); + configuration.signer.public_key = pair.publicKey; + const finalPayload = canonicalConfigurationBytes(configuration); + const envelope = encodeEnvelope( + configuration.signer.key_id, + finalPayload, + sign(finalPayload, pair.privateKey) + ); + expect(canonicalizeConfiguration(parseConfiguration(envelope))).toBe( + canonicalizeConfiguration(configuration) ); }); -}); - -describe('parse', () => { - it('round-trips the demo study through canonicalize', () => { - const configuration = parseConfiguration(demoStudyBytes); - expect(validate(configuration)).toEqual([]); - const again = parseConfiguration(canonicalBytes(configuration)); - expect(canonicalize(again)).toBe(canonicalize(configuration)); - }); - - it('round-trips an envelope', () => { - const configuration = parseConfiguration(demoStudyBytes); - const payload = canonicalBytes(configuration); - const pair = signingKeyPairFromPrivate(demoSigningPrivateKey); - const signature = sign(payload, pair.privatePkcs8Base64); - const envelope = encodeEnvelope(configuration.signer.key_id, payload, signature); - const decoded = decodeEnvelope(envelope); - expect(decoded.signerKeyId).toBe(configuration.signer.key_id); - expect(decoded.configurationBytes).toEqual(payload); - expect(verify(decoded.configurationBytes, decoded.signature, pair.publicX509Base64)).toBe(true); - expect(canonicalize(parseConfiguration(envelope))).toBe(canonicalize(configuration)); - }); - - it('refuses bytes that are not a configuration', () => { - expect(() => parseConfiguration(new TextEncoder().encode('{"a":1}'))).toThrow(); - expect(() => parseConfiguration(new TextEncoder().encode('nope'))).toThrow(); - }); -}); -describe('generated artefacts', () => { - it('signs what it generates and verifies against the declared public key', () => { - const pair = generateSigningKeyPair(); - const keyset = generateHpkeKeyset(); + it('authors pinned platform/build defaults and integer physical units', () => { const configuration = emptyConfiguration(); - configuration.experiment_id = 'demo-study'; - configuration.configuration_id = 'demo-study-v1'; - configuration.title = 'T'; - configuration.researcher = { name: 'R', contact: 'r@example.org' }; - configuration.purpose = 'P'; - configuration.consent = { document_version: 'v1', summary: 'S' }; - configuration.collectors = [{ id: 'app_lifecycle.v1', required: false, config: {} }]; - configuration.signer = { key_id: 'demo-signer', public_key: pair.publicX509Base64 }; - configuration.export = { - researcher_key_id: 'demo-export', - tink_hpke_public_keyset: keyset.publicKeyset + expect(configuration.platform).toBe('android'); + expect(configuration.minimum_client_version).toBe('1'); + expect(configuration.storage.maximum_local_bytes).toBe(DEFAULT_LOCAL_BYTES); + expect(configuration.export).toEqual({ researcher_key_id: '', hpke_public_key: '' }); + }); + + it('downloads private keys as exact raw text and the config only after signing', () => { + const signing = generateSigningKeyPair(); + const hpke = generateHpkeKeyPair(); + const canonical = canonicalConfigurationBytes(validConfiguration()); + const source = { + signingPrivate: signing.privateKey, + hpkePrivate: hpke.privateKey, + canonical, + envelope: null }; - expect(validate(configuration)).toEqual([]); - const payload = canonicalBytes(configuration); - const signature = sign(payload, pair.privatePkcs8Base64); - expect(verify(payload, signature, configuration.signer.public_key)).toBe(true); - const envelope = encodeEnvelope(configuration.signer.key_id, payload, signature); - expect(decodeEnvelope(envelope).configurationBytes).toEqual(payload); - // The two private artefacts carry no trailing newline. - expect(pair.privatePkcs8Base64.endsWith('\n')).toBe(false); - expect(keysetJson(keyset.privateKeyset).endsWith('\n')).toBe(false); - }); -}); - -describe('units and estimate', () => { - for (const [locale, m] of [['en', en], ['zh-TW', zhTW]] as const) { - it(`humanises in ${locale}`, () => { - const u = units(m, locale); - expect(u.minutes(1440)).toMatch(/1/); - expect(u.minutes(60)).toBe(`1 ${m.unit.hours}`); - expect(u.minutes(15)).toBe(`15 ${m.unit.minutes}`); - expect(u.millis(0)).toBe('0'); - expect(u.millis(10000)).toMatch(/10/); - expect(u.hertz(10)).toBe(`10 ${m.unit.hertz}`); - // The float32 that will be written, not the double that was typed. - expect(u.metres(1234.5678)).toBe(`1234.5677 ${m.unit.metres}`); - expect(u.metres(0)).toBe(`0.0 ${m.unit.metres}`); - expect(u.bytes(64 * 1024 * 1024)).toBe('64 MiB'); - // The default a study opens on, in the words the control renders it in. - expect(u.bytes(DEFAULT_LOCAL_BYTES)).toBe('1 GiB'); - expect(u.about('x')).toBe('≈ x'); - }); - } - - it('rates the accelerometer as the expensive one', () => { - const configuration = emptyConfiguration(); - configuration.collectors = [ - { id: 'accelerometer.v1', required: false, config: { sampling_period_us: 5_000, maximum_report_latency_us: 0 } }, - { id: 'app_lifecycle.v1', required: false, config: {} } - ]; - const result = estimate(configuration); - expect(result.eventsPerHour).toBeGreaterThan(700_000); - // 200 Hz fills the default quota inside a day. The bound is a day rather than the quota, so - // this says what it means and does not have to move when the default does. - expect(result.hoursToQuota).toBeLessThan(24); - expect(volumeOf(720_000)).toBe(4); - expect(volumeOf(20)).toBe(1); - }); -}); - -describe('paths', () => { - it('routes every issue a document can raise to a named step', () => { - for (const path of [ - 'experiment_id', 'consent.summary', 'collectors.2.config.interval_millis', - 'interventions.0.id', 'surveys.0.questions.0.id', 'storage.maximum_local_bytes', 'upload.endpoint', - 'signer.key_id', 'export.tink_hpke_public_keyset', '' - ]) { - expect(['keys', 'study', 'sign', 'files']).toContain(stepForPath(path)); - } - expect(stepForPath('signer.public_key')).toBe('keys'); - expect(stepForPath('')).toBe('sign'); - expect(stepForPath('collectors.2.config.interval_millis')).toBe('study'); - // The three the study step no longer owns. Both identifiers are read out on the sign step, - // which is where their issue rows now land; `minimum_app_version` has no control at all. - expect(stepForPath('experiment_id')).toBe('sign'); - expect(stepForPath('configuration_id')).toBe('sign'); - expect(stepForPath('minimum_app_version')).toBe('sign'); - }); -}); - -describe('issue messages', () => { - it('covers every code validate can emit, in both catalogues', () => { - const configuration = emptyConfiguration(); - configuration.collectors = [ - { id: 'network_usage.v1', required: false, config: { transports: [], poll_interval_minutes: 0 } }, - { id: 'location.v1', required: false, config: { interval_millis: 1000, minimum_interval_millis: 3000, maximum_batch_delay_millis: 0, minimum_displacement_meters: -1, priority: 'BALANCED' } } - ]; - configuration.interventions = [{ - id: 'a', - action: { type: 'notification', notification_title: '', notification_message: '' }, - triggers: [{ id: 'b', schedule: { type: 'one_time', offset_minutes: -1, clock: 'CALENDAR_TIME' }, availability_minutes: 0 }] - }]; - configuration.upload = { endpoint: 'http://x', interval_minutes: 0, allow_metered: false }; - configuration.expires_at = configuration.issued_at; - for (const m of [en, zhTW]) { - for (const issue of validate(configuration)) { - expect(m.issue, issue.code).toHaveProperty(issue.code); - } - } + expect(new TextDecoder().decode(artifactBytes('signing-private', source)!)).toBe(signing.privateKey); + expect(new TextDecoder().decode(artifactBytes('hpke-private', source)!)).toBe(hpke.privateKey); + expect(artifactBytes('canonical', source)).toBeNull(); }); }); diff --git a/web/tests/scales.spec.ts b/web/tests/scales.spec.ts index cbdd000..f1debfd 100644 --- a/web/tests/scales.spec.ts +++ b/web/tests/scales.spec.ts @@ -6,16 +6,13 @@ * it is the defect `Math.round` on the sampling period would have introduced on 94 of 200 rates. * * 2. Encodable. Whatever the control can produce has to be something the canonical encoder can - * write. Every stored value is an integer inside the schema's bound, because `integer()` emits - * `-?(0|[1-9][0-9]*)` and anything else is a file the app refuses. The one exception is - * `minimum_displacement_meters`, a Kotlin `Float` written by `formatFloat`, where the stored - * value is a float32 and `formatFloat` must round-trip back to the number the box showed. + * write. Every stored value is an integer inside the schema's bound, including location's + * millimetre value. * * The lattice is `min, min+step, …, max` for a control with a box, and the ladder for one without. */ import { describe, expect, it } from 'vitest'; -import { formatFloat } from '../src/lib/adc/canonical'; import { en } from '../src/lib/i18n/en'; import { zhTW } from '../src/lib/i18n/zh-TW'; import { SCALE_BOUNDS, scales, type Scale, type ScaleKey } from '../src/routes/researcher/scales'; @@ -30,11 +27,15 @@ const CATALOGUES = [ function lattice(scale: Scale): number[] { if (scale.ladder) return [...scale.ladder]; const digits = String(scale.step).split('.')[1]?.length ?? 0; + const count = Math.round((scale.max - scale.min) / scale.step); + const stride = Math.max(1, Math.ceil(count / 10_000)); const points: number[] = []; - for (let h = scale.min; h <= scale.max + scale.step / 2; h += scale.step) { + for (let index = 0; index <= count; index += stride) { + const h = scale.min + index * scale.step; points.push(digits > 0 ? Number(h.toFixed(digits)) : h); } - return points; + points.push(scale.max, ...scale.presets); + return [...new Set(points)]; } for (const [name, catalogue, locale] of CATALOGUES) { @@ -43,13 +44,17 @@ for (const [name, catalogue, locale] of CATALOGUES) { describe(`scales (${name})`, () => { it('offers a box only where one human unit names both ends', () => { - // The rule is not a judgement per control: it follows from the bounds. These four are hertz - // at both ends, seconds at both ends, metres at both ends, hertz at both ends. + // The rule is not a judgement per control: it follows from whether one displayed unit names + // both ends of the legal range. const boxed = keys.filter((key) => S[key].box); expect(boxed.sort()).toEqual( [ + 'ambient_sampling_period_us', + 'change_threshold_millilux', + 'change_threshold_millimeters', 'maximum_report_latency_us', - 'minimum_displacement_meters', + 'minimum_displacement_millimeters', + 'minimum_event_interval_ms', 'sampling_period_us', 'trajectory_sampling_hz' ].sort() @@ -102,12 +107,6 @@ for (const [name, catalogue, locale] of CATALOGUES) { const stored = scale.toStored(human); if (stored < low || stored > high) { broken.push(`${human} stores ${stored}, outside ${low}..${high}`); - } else if (key === 'minimum_displacement_meters') { - // A Float, and the shortest decimal that round-trips to it is what the file carries. - if (Math.fround(stored) !== stored) broken.push(`${human} stores a non-float32`); - else if (Number(formatFloat(stored)) !== human) { - broken.push(`${human} writes as ${formatFloat(stored)}`); - } } else if (!Number.isInteger(stored)) { broken.push(`${human} stores ${stored}, which is not an integer`); } @@ -118,8 +117,16 @@ for (const [name, catalogue, locale] of CATALOGUES) { }); it(`${key} can reach every value on its chip row`, () => { - const reachable = new Set(lattice(scale)); - for (const preset of scale.presets) expect(reachable.has(preset), `${key} ${preset}`).toBe(true); + for (const preset of scale.presets) { + const reachable = scale.ladder + ? scale.ladder.includes(preset) + : preset >= scale.min && preset <= scale.max && + Math.abs( + (preset - scale.min) / scale.step - + Math.round((preset - scale.min) / scale.step) + ) < 1e-9; + expect(reachable, `${key} ${preset}`).toBe(true); + } }); } diff --git a/web/tests/seal.ts b/web/tests/seal.ts index 8e293e0..ec1c33a 100644 --- a/web/tests/seal.ts +++ b/web/tests/seal.ts @@ -1,108 +1,170 @@ -/** - * A `.adcexp`, built here, because nothing else in this repository can build one on demand. - * - * `tests/compat.spec.ts` gets its fixtures the honest way: it shells out to `researcher-tools` and - * compares. That trick does not work for bundles — the CLI has a `decrypt` and no `encrypt`, because - * the only thing that ever writes one of these is a participant's phone. Committing a real bundle is - * not an option either: opening one needs the export *private* key, and `.gitignore` refuses - * `*-private.json` and `*.adcexp` for exactly that reason. - * - * So the direction is inverted. This seals a bundle with the writer's half of the same recipe the - * reader implements, and two different things then check it. `tests/bundle.spec.ts` opens it with - * the site's reader, which proves the pair agree; `tests/compat.spec.ts` hands the same three files - * to `researcher-tools decrypt`, which proves the pair agree *with the JVM* — and that second one is - * the claim that matters, because a writer and a reader that are wrong the same way would pass the - * first on their own. - * - * A key exists here only for the length of a test, and only in memory or a `mkdtemp` directory. - */ +/** Deterministic Protocol v1 bundle sealer used only by reader tests. */ -import { canonicalize } from '../src/lib/adc/canonical'; -import { bundleContext } from '../src/lib/adc/bundle'; -import { hpkePublicKey } from '../src/lib/adc/tink'; -import type { StudyConfiguration, TinkKeyset } from '../src/lib/adc/types'; +import { + BUNDLE_FORMAT, + bundleContext, + configurationDigest, + type ResearchDocument +} from '../src/lib/adc/bundle'; +import { + canonicalBytes, + canonicalizeConfiguration +} from '../src/lib/adc/canonical'; +import { decodeBase64Url, encodeBase64Url, sign } from '../src/lib/adc/crypto'; +import type { StudyConfiguration } from '../src/lib/adc/types'; import { x25519 } from '@noble/curves/ed25519.js'; import { expand, extract } from '@noble/hashes/hkdf.js'; import { sha256 } from '@noble/hashes/sha2.js'; -const encoder = new TextEncoder(); -const utf8 = (text: string) => encoder.encode(text); +const UTF8 = new TextEncoder(); const EMPTY = new Uint8Array(0); -const buffer = (bytes: Uint8Array): BufferSource => bytes as unknown as BufferSource; - -const i2osp2 = (value: number) => Uint8Array.of((value >> 8) & 0xff, value & 0xff); -const SUITE_KEM = concat(utf8('KEM'), i2osp2(0x0020)); -const SUITE_HPKE = concat(utf8('HPKE'), i2osp2(0x0020), i2osp2(0x0001), i2osp2(0x0002)); -const VERSION = utf8('HPKE-v1'); - -const labeledExtract = (suite: Uint8Array, salt: Uint8Array, label: string, ikm: Uint8Array) => - extract(sha256, concat(VERSION, suite, utf8(label), ikm), salt); - -const labeledExpand = ( - suite: Uint8Array, - prk: Uint8Array, - label: string, - info: Uint8Array, - length: number -) => expand(sha256, prk, concat(i2osp2(length), VERSION, suite, utf8(label), info), length); +const NONCE = new Uint8Array(12).fill(0x44); +const EPHEMERAL_PRIVATE = new Uint8Array(32).fill(0x33); +const CONTENT_KEY = new Uint8Array(32).fill(0x55); +const DEFAULT_BUNDLE_ID = '00112233-4455-4677-8899-aabbccddeeff'; export interface SealOptions { - /** The name the header carries, when a test needs it to disagree with the configuration. */ + bundleId?: string; keyId?: string; - /** The keyset to seal to, when a test needs a bundle the study's own key cannot open. */ - keyset?: TinkKeyset; + document?: (value: ResearchDocument) => unknown; } -/** - * `ResearchExport.encrypt`, in reverse of the reader: a random content key sealed to the study's - * HPKE public key with the context as `info`, then the body under that key with the same context as - * AAD, behind the 26-byte header. - */ -export async function seal( +export async function sealBundle( configuration: StudyConfiguration, - plaintext: string, + signingPrivateKey: string, options: SealOptions = {} ): Promise { - const keyset = options.keyset ?? configuration.export.tink_hpke_public_keyset; - const recipient = hpkePublicKey(keyset); - if (!recipient) throw new Error('seal_keyset'); - const context = bundleContext(configuration); - - const contentKey = crypto.getRandomValues(new Uint8Array(32)); - const nonce = crypto.getRandomValues(new Uint8Array(12)); - const wrapped = await wrap(recipient, keyset.primaryKeyId, contentKey, context); + const bundleId = options.bundleId ?? DEFAULT_BUNDLE_ID; + const keyId = options.keyId ?? configuration.export.researcher_key_id; + const digest = configurationDigest(configuration); + const digestHex = hex(digest); + const configurationValue = JSON.parse(canonicalizeConfiguration(configuration)); + const signature = sign(canonicalBytes(configurationValue), signingPrivateKey); + const time = { + boot_session_id: 'boot-0001', + monotonic_time_nanos: '9007199254740993', + wall_time_utc_millis: '1767225600000' + }; + const document: ResearchDocument = { + format: BUNDLE_FORMAT, + bundle_id: bundleId, + bundle_kind: 'manual_export', + configuration_sha256: digestHex, + producer: { platform: 'android', client_version: '1' }, + exported_at_utc_millis: '1767225600000', + configuration: configurationValue, + configuration_signature: { + signer_key_id: configuration.signer.key_id, + signature: encodeBase64Url(signature) + }, + experiment: { + experiment_id: configuration.experiment_id, + configuration_id: configuration.configuration_id, + participant_instance_id: '123e4567-e89b-42d3-a456-426614174000', + assigned_participant_id: configuration.assigned_participant_id, + state: 'RUNNING', + retained_from_sequence: '1', + uploaded_through_sequence: '0', + durable_through_sequence: '2', + next_sequence_number: '3', + first_sequence_number: '1', + last_sequence_number: '2', + event_count: '2', + transitions: [ + { + from: 'IMPORTED', + to: 'CONFIG_VERIFIED', + reason: 'CONFIGURATION_SIGNATURE_VERIFIED', + time + }, + { + from: 'CONFIG_VERIFIED', + to: 'CONSENT_PENDING', + reason: 'CONSENT_REVIEW_OPENED', + time + }, + { + from: 'CONSENT_PENDING', + to: 'ACCESS_SETUP', + reason: 'CONSENT_ACCEPTED', + time + }, + { + from: 'ACCESS_SETUP', + to: 'READY', + reason: 'ACCESS_PREFLIGHT_PASSED', + time + }, + { from: 'READY', to: 'RUNNING', reason: 'PARTICIPANT_STARTED', time } + ], + events: [ + { + sequence_number: '1', + collector_id: 'app_lifecycle.v1', + payload_schema_version: 1, + observed_time: time, + payload_type: 'ACTIVITY_CREATED', + fields: { activity_class: 'tests.ProtocolFixtureActivity' } + }, + { + sequence_number: '2', + collector_id: 'app_lifecycle.v1', + payload_schema_version: 1, + observed_time: time, + payload_type: 'ACTIVITY_RESUMED', + fields: { activity_class: 'tests.ProtocolFixtureActivity' } + } + ] + } + }; - const key = await crypto.subtle.importKey('raw', buffer(contentKey), 'AES-GCM', false, [ - 'encrypt' - ]); - const body = new Uint8Array( - await crypto.subtle.encrypt( - { name: 'AES-GCM', iv: buffer(nonce), tagLength: 128, additionalData: buffer(context) }, - key, - buffer(utf8(plaintext)) - ) + const context = bundleContext(bundleId, digestHex, keyId); + const wrapped = await wrap( + CONTENT_KEY, + decodeBase64Url(configuration.export.hpke_public_key, 32), + context ); - - const keyId = utf8(options.keyId ?? configuration.export.researcher_key_id); - const header = new Uint8Array(26); - header.set(utf8('ADCEXP01')); - const view = new DataView(header.buffer); - view.setUint16(8, keyId.length); - view.setInt32(10, wrapped.length); - header.set(nonce, 14); - return concat(header, keyId, wrapped, body); + const plaintext = canonicalBytes(options.document ? options.document(document) : document); + const body = await aesGcm(CONTENT_KEY, NONCE, context, plaintext); + const keyIdBytes = UTF8.encode(keyId); + const out = new Uint8Array(70 + keyIdBytes.length + wrapped.length + body.length); + out.set(UTF8.encode('ADCEXP01')); + out.set(uuidBytes(bundleId), 8); + out.set(digest, 24); + new DataView(out.buffer).setUint16(56, keyIdBytes.length); + out.set(NONCE, 58); + out.set(keyIdBytes, 70); + out.set(wrapped, 70 + keyIdBytes.length); + out.set(body, 70 + keyIdBytes.length + wrapped.length); + return out; } -/** RFC 9180 base-mode seal, with Tink's 5-byte `TINK` prefix in front of it. */ +const KEM_ID = 0x0020; +const KDF_ID = 0x0001; +const AEAD_ID = 0x0002; +const i2osp2 = (value: number) => Uint8Array.of((value >> 8) & 0xff, value & 0xff); +const VERSION = UTF8.encode('HPKE-v1'); +const SUITE_KEM = concat(UTF8.encode('KEM'), i2osp2(KEM_ID)); +const SUITE_HPKE = concat( + UTF8.encode('HPKE'), i2osp2(KEM_ID), i2osp2(KDF_ID), i2osp2(AEAD_ID) +); +const labeledExtract = (suite: Uint8Array, salt: Uint8Array, label: string, ikm: Uint8Array) => + extract(sha256, concat(VERSION, suite, UTF8.encode(label), ikm), salt); +const labeledExpand = ( + suite: Uint8Array, + prk: Uint8Array, + label: string, + info: Uint8Array, + length: number +) => expand(sha256, prk, concat(i2osp2(length), VERSION, suite, UTF8.encode(label), info), length); + async function wrap( + plaintext: Uint8Array, recipientPublic: Uint8Array, - keyId: number, - contentKey: Uint8Array, info: Uint8Array ): Promise { - const ephemeral = x25519.utils.randomSecretKey(); - const enc = x25519.getPublicKey(ephemeral); - const dh = x25519.getSharedSecret(ephemeral, recipientPublic); + const enc = x25519.getPublicKey(EPHEMERAL_PRIVATE); + const dh = x25519.getSharedSecret(EPHEMERAL_PRIVATE, recipientPublic); const eaePrk = labeledExtract(SUITE_KEM, EMPTY, 'eae_prk', dh); const shared = labeledExpand(SUITE_KEM, eaePrk, 'shared_secret', concat(enc, recipientPublic), 32); const schedule = concat( @@ -111,97 +173,38 @@ async function wrap( labeledExtract(SUITE_HPKE, EMPTY, 'info_hash', info) ); const secret = labeledExtract(SUITE_HPKE, shared, 'secret', EMPTY); - const key = await crypto.subtle.importKey( - 'raw', - buffer(labeledExpand(SUITE_HPKE, secret, 'key', schedule, 32)), - 'AES-GCM', - false, - ['encrypt'] - ); - const baseNonce = labeledExpand(SUITE_HPKE, secret, 'base_nonce', schedule, 12); - const sealed = new Uint8Array( + const key = labeledExpand(SUITE_HPKE, secret, 'key', schedule, 32); + const nonce = labeledExpand(SUITE_HPKE, secret, 'base_nonce', schedule, 12); + return concat(enc, await aesGcm(key, nonce, EMPTY, plaintext)); +} + +async function aesGcm( + rawKey: Uint8Array, + nonce: Uint8Array, + aad: Uint8Array, + plaintext: Uint8Array +): Promise { + const source = (value: Uint8Array): BufferSource => value as unknown as BufferSource; + const key = await crypto.subtle.importKey('raw', source(rawKey), 'AES-GCM', false, ['encrypt']); + return new Uint8Array( await crypto.subtle.encrypt( - { name: 'AES-GCM', iv: buffer(baseNonce), tagLength: 128, additionalData: buffer(EMPTY) }, + { name: 'AES-GCM', iv: source(nonce), additionalData: source(aad), tagLength: 128 }, key, - buffer(contentKey) + source(plaintext) ) ); - const prefix = new Uint8Array(5); - prefix[0] = 1; - new DataView(prefix.buffer).setUint32(1, keyId); - return concat(prefix, enc, sealed); } -/** - * A plaintext in the shape `JsonWriter` emits it, key order included. `configuration` is the study's - * own canonical JSON inlined verbatim, which is what makes the bundle self-describing. - */ -export function bundleJson( - configuration: StudyConfiguration, - experiment: { - participantInstanceId?: string; - assignedParticipantId?: string | null; - state?: string; - events?: number; - firstSequenceNumber?: number; - lifetime?: number; - collectors?: readonly string[]; - } = {} -): string { - const first = experiment.firstSequenceNumber ?? 1; - const count = experiment.events ?? 0; - const collectors = experiment.collectors ?? ['app_lifecycle.v1', 'accelerometer.v1']; - const events = Array.from({ length: count }, (_unused, index) => ({ - sequence_number: first + index, - collector_id: collectors[index % collectors.length], - payload_schema_version: 1, - observed_time: { - wall_time_utc_millis: 1_762_000_000_000 + index * 1_000, - elapsed_realtime_nanos: 2_000_000_000 + index * 1_000_000, - boot_session_id: 'boot-a1b2c3' - }, - payload_type: 'SENSOR_SAMPLE', - // Strings, every one of them, as the wire format has it — a number here would be a bundle no - // phone ever wrote. - fields: { x: '0.2', y: '-1.2', z: '9.82' } - })); - const last = count === 0 ? 0 : first + count - 1; - return JSON.stringify({ - format: 'research-bundle-v1', - exported_at_utc_millis: 1_762_000_100_000, - configuration: JSON.parse(canonicalize(configuration)), - experiment: { - experiment_id: configuration.experiment_id, - configuration_id: configuration.configuration_id, - participant_instance_id: - experiment.participantInstanceId ?? '00000000-0000-4000-8000-000000000017', - // Written only when the study assigns one, so an anonymous bundle has no such key at all. - ...(experiment.assignedParticipantId - ? { assigned_participant_id: experiment.assignedParticipantId } - : {}), - state: experiment.state ?? 'RUNNING', - next_sequence_number: (experiment.lifetime ?? last) + 1, - transitions: [ - { - from: 'IMPORTED', - to: 'CONFIG_VERIFIED', - reason: 'CONFIGURATION_SIGNATURE_VERIFIED', - time: { - wall_time_utc_millis: 1_762_000_000_000, - elapsed_realtime_nanos: 2_000_000_000, - boot_session_id: 'boot-a1b2c3' - } - } - ], - events, - first_sequence_number: first, - last_sequence_number: last - } - }); +function uuidBytes(value: string): Uint8Array { + return Uint8Array.from(value.replace(/-/g, '').match(/../g) ?? [], (byte) => parseInt(byte, 16)); +} + +function hex(bytes: Uint8Array): string { + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, '0')).join(''); } function concat(...parts: Uint8Array[]): Uint8Array { - const out = new Uint8Array(parts.reduce((total, part) => total + part.length, 0)); + const out = new Uint8Array(parts.reduce((sum, part) => sum + part.length, 0)); let at = 0; for (const part of parts) { out.set(part, at); From 51e48f310b812a4930f941983d7934a61465708c Mon Sep 17 00:00:00 2001 From: JacobLinCool Date: Wed, 5 Aug 2026 01:32:25 +0800 Subject: [PATCH 2/4] Add the ciphertext receiver and offline analysis pipeline Ship the bounded R2-only Cloudflare Worker with immutable replay receipts and no decrypt or administration path. Add the adc-analysis inventory, verification, reassembly, typed Parquet, provenance, and quality-summary pipeline with dedicated CI coverage. --- .github/workflows/analysis.yml | 60 + .github/workflows/receiver.yml | 60 + adc-analysis/.gitignore | 9 + adc-analysis/README.md | 129 ++ adc-analysis/pyproject.toml | 35 + adc-analysis/src/adc_analysis/__init__.py | 8 + adc-analysis/src/adc_analysis/bundle.py | 840 ++++++++ adc-analysis/src/adc_analysis/catalog.py | 295 +++ adc-analysis/src/adc_analysis/cli.py | 86 + .../src/adc_analysis/configuration.py | 439 ++++ adc-analysis/src/adc_analysis/crypto.py | 103 + adc-analysis/src/adc_analysis/encoding.py | 61 + adc-analysis/src/adc_analysis/errors.py | 13 + adc-analysis/src/adc_analysis/event_store.py | 284 +++ adc-analysis/src/adc_analysis/filesystem.py | 68 + adc-analysis/src/adc_analysis/inventory.py | 263 +++ adc-analysis/src/adc_analysis/jcs.py | 141 ++ adc-analysis/src/adc_analysis/limits.py | 7 + adc-analysis/src/adc_analysis/models.py | 136 ++ adc-analysis/src/adc_analysis/pipeline.py | 156 ++ adc-analysis/src/adc_analysis/py.typed | 1 + adc-analysis/src/adc_analysis/reassembly.py | 300 +++ adc-analysis/src/adc_analysis/sink.py | 417 ++++ adc-analysis/src/adc_analysis/sources.py | 101 + .../src/adc_analysis/streaming_json.py | 202 ++ adc-analysis/src/adc_analysis/summary.py | 42 + adc-analysis/tests/__init__.py | 1 + adc-analysis/tests/fakes.py | 40 + adc-analysis/tests/test_cli.py | 123 ++ .../tests/test_configuration_catalog.py | 289 +++ adc-analysis/tests/test_conformance.py | 232 +++ adc-analysis/tests/test_inventory.py | 144 ++ adc-analysis/tests/test_pipeline.py | 181 ++ adc-analysis/tests/test_reassembly.py | 202 ++ adc-analysis/tests/test_sink.py | 364 ++++ .../tests/test_streaming_filesystem.py | 69 + adc-analysis/uv.lock | 424 ++++ receiver/.gitignore | 5 + receiver/README.md | 120 ++ receiver/package.json | 26 + receiver/pnpm-lock.yaml | 1787 +++++++++++++++++ receiver/src/contract.ts | 345 ++++ receiver/src/index.ts | 166 ++ receiver/src/verified-body.ts | 160 ++ receiver/tests/receiver.test.ts | 577 ++++++ receiver/tests/receiver.workerd.test.ts | 141 ++ receiver/tsconfig.json | 29 + receiver/vitest.config.ts | 8 + receiver/vitest.workerd.config.ts | 22 + receiver/worker-configuration.d.ts | 12 + receiver/wrangler.example.jsonc | 24 + 51 files changed, 9747 insertions(+) create mode 100644 .github/workflows/analysis.yml create mode 100644 .github/workflows/receiver.yml create mode 100644 adc-analysis/.gitignore create mode 100644 adc-analysis/README.md create mode 100644 adc-analysis/pyproject.toml create mode 100644 adc-analysis/src/adc_analysis/__init__.py create mode 100644 adc-analysis/src/adc_analysis/bundle.py create mode 100644 adc-analysis/src/adc_analysis/catalog.py create mode 100644 adc-analysis/src/adc_analysis/cli.py create mode 100644 adc-analysis/src/adc_analysis/configuration.py create mode 100644 adc-analysis/src/adc_analysis/crypto.py create mode 100644 adc-analysis/src/adc_analysis/encoding.py create mode 100644 adc-analysis/src/adc_analysis/errors.py create mode 100644 adc-analysis/src/adc_analysis/event_store.py create mode 100644 adc-analysis/src/adc_analysis/filesystem.py create mode 100644 adc-analysis/src/adc_analysis/inventory.py create mode 100644 adc-analysis/src/adc_analysis/jcs.py create mode 100644 adc-analysis/src/adc_analysis/limits.py create mode 100644 adc-analysis/src/adc_analysis/models.py create mode 100644 adc-analysis/src/adc_analysis/pipeline.py create mode 100644 adc-analysis/src/adc_analysis/py.typed create mode 100644 adc-analysis/src/adc_analysis/reassembly.py create mode 100644 adc-analysis/src/adc_analysis/sink.py create mode 100644 adc-analysis/src/adc_analysis/sources.py create mode 100644 adc-analysis/src/adc_analysis/streaming_json.py create mode 100644 adc-analysis/src/adc_analysis/summary.py create mode 100644 adc-analysis/tests/__init__.py create mode 100644 adc-analysis/tests/fakes.py create mode 100644 adc-analysis/tests/test_cli.py create mode 100644 adc-analysis/tests/test_configuration_catalog.py create mode 100644 adc-analysis/tests/test_conformance.py create mode 100644 adc-analysis/tests/test_inventory.py create mode 100644 adc-analysis/tests/test_pipeline.py create mode 100644 adc-analysis/tests/test_reassembly.py create mode 100644 adc-analysis/tests/test_sink.py create mode 100644 adc-analysis/tests/test_streaming_filesystem.py create mode 100644 adc-analysis/uv.lock create mode 100644 receiver/.gitignore create mode 100644 receiver/README.md create mode 100644 receiver/package.json create mode 100644 receiver/pnpm-lock.yaml create mode 100644 receiver/src/contract.ts create mode 100644 receiver/src/index.ts create mode 100644 receiver/src/verified-body.ts create mode 100644 receiver/tests/receiver.test.ts create mode 100644 receiver/tests/receiver.workerd.test.ts create mode 100644 receiver/tsconfig.json create mode 100644 receiver/vitest.config.ts create mode 100644 receiver/vitest.workerd.config.ts create mode 100644 receiver/worker-configuration.d.ts create mode 100644 receiver/wrangler.example.jsonc diff --git a/.github/workflows/analysis.yml b/.github/workflows/analysis.yml new file mode 100644 index 0000000..7151556 --- /dev/null +++ b/.github/workflows/analysis.yml @@ -0,0 +1,60 @@ +name: Analysis CI + +on: + push: + branches: + - main + paths: + - "adc-analysis/**" + - "protocol/v1/**" + - "tools/generate_protocol_vectors.mjs" + - "tools/validate_protocol_vectors.py" + - ".github/workflows/analysis.yml" + pull_request: + paths: + - "adc-analysis/**" + - "protocol/v1/**" + - "tools/generate_protocol_vectors.mjs" + - "tools/validate_protocol_vectors.py" + - ".github/workflows/analysis.yml" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: analysis-ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + verify: + name: Ruff and offline pipeline tests + runs-on: ubuntu-latest + timeout-minutes: 20 + defaults: + run: + working-directory: adc-analysis + + steps: + - name: Check out source + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Set up locked Python environment + uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 + with: + version: "0.11.26" + python-version: "3.13" + working-directory: adc-analysis + enable-cache: true + cache-dependency-glob: adc-analysis/uv.lock + + - name: Install locked dependencies + run: uv sync --locked + + - name: Check Python sources + run: uv run --frozen ruff check src tests + + - name: Run complete offline pipeline suite + run: uv run --frozen python -m unittest discover -s tests -v diff --git a/.github/workflows/receiver.yml b/.github/workflows/receiver.yml new file mode 100644 index 0000000..2c60fe2 --- /dev/null +++ b/.github/workflows/receiver.yml @@ -0,0 +1,60 @@ +name: Receiver CI + +on: + push: + branches: + - main + paths: + - "receiver/**" + - "protocol/v1/**" + - "docs/p0-p2-implementation-contract.md" + - ".github/workflows/receiver.yml" + pull_request: + paths: + - "receiver/**" + - "protocol/v1/**" + - "docs/p0-p2-implementation-contract.md" + - ".github/workflows/receiver.yml" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: receiver-ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + verify: + name: Type-check, test, and bundle + runs-on: ubuntu-latest + timeout-minutes: 10 + + steps: + - name: Check out source + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Set up pnpm + uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + with: + version: "10.21.0" + + - name: Set up Node + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: "22" + cache: pnpm + cache-dependency-path: receiver/pnpm-lock.yaml + + - name: Install dependencies + working-directory: receiver + run: pnpm install --frozen-lockfile + + - name: Verify receiver + working-directory: receiver + run: | + pnpm typecheck + pnpm test + pnpm build diff --git a/adc-analysis/.gitignore b/adc-analysis/.gitignore new file mode 100644 index 0000000..6e0cf73 --- /dev/null +++ b/adc-analysis/.gitignore @@ -0,0 +1,9 @@ +.venv/ +.ruff_cache/ +.coverage +htmlcov/ +build/ +dist/ +src/*.egg-info/ +__pycache__/ +*.py[cod] diff --git a/adc-analysis/README.md b/adc-analysis/README.md new file mode 100644 index 0000000..fe1af2e --- /dev/null +++ b/adc-analysis/README.md @@ -0,0 +1,129 @@ +# ADC analysis + +`adc-analysis` is the offline, one-way Protocol v1 pipeline. It inventories encrypted `.adcexp` +objects, authenticates an entire bundle, deterministically reassembles events, and atomically +publishes typed Parquet. It never changes a study, contacts a participant, or adds a receiver-side +decrypt path. + +Start with the repository's normative contract: + +- `../protocol/v1/README.md` defines the wire and cryptographic protocol. +- `../protocol/v1/collector-catalog.json` defines every accepted collector, payload, field, unit, + and type. +- `../protocol/v1/conformance-vectors.json` is the shared valid and hostile corpus. +- `src/adc_analysis/` contains the implementation; each pipeline stage has one correspondingly + named module. +- `tests/` covers the shared corpus, source inventory, conflict/gap taxonomy, configuration, and + Parquet round trips. + +| Concern | Source | Primary tests | +| --- | --- | --- | +| Local/R2 reads, source-specific bounds, immutable cache | `sources.py`, `inventory.py`, `limits.py` | `test_inventory.py` | +| JCS, keys, HPKE, configuration, complete streaming bundle verification | `jcs.py`, `streaming_json.py`, `encoding.py`, `crypto.py`, `configuration.py`, `bundle.py` | `test_conformance.py`, `test_configuration_catalog.py`, `test_streaming_filesystem.py` | +| Duplicate/conflict/gap decisions and bounded spill storage | `reassembly.py`, `event_store.py` | `test_reassembly.py` | +| Arrow schema, Parquet, manifest, quality summary | `sink.py`, `summary.py` | `test_pipeline.py`, `test_sink.py` | +| One-way orchestration and CLI | `pipeline.py`, `cli.py` | `test_pipeline.py`, `test_cli.py` | +| Owner-only staging and create-only publication | `filesystem.py` | `test_streaming_filesystem.py` | + +## Pipeline and trust boundary + +```text +LocalBundleSource / S3BundleSource + -> content-addressed ciphertext cache + inventory.json + -> framing, HPKE, AEAD, JCS, signature, config, range, catalog validation + -> deterministic event reassembly + -> typed Parquet + dataset manifest + quality summary +``` + +R2 metadata and paths are untrusted routing claims. Participant/event identity exists only after +the complete authenticated document verifies. Automatic receiver objects retain the 32 MiB wire +bound. A local manual export may reach its signed local-storage quota (at most 8 GiB), so GCM +decryption, JCS checking, event validation, reassembly, and Parquet row groups are streamed or +spilled instead of loading the document into memory. AEAD authentication completes before JSON is +accepted. Every plaintext staging/spill artifact is inside a tightened mode-0700 directory with +owner-only files and is removed on every handled success or failure. One invalid bundle is +quarantined whole and +emits no rows. A conflicting authenticated event identity stops dataset publication; there is no +last-write-wins behavior or unknown-schema fallback. + +## Install and run + +From this directory: + +```sh +uv sync --locked +uv run adc-analysis inventory \ + --workspace /secure/adc-work \ + --local /path/to/manual-exports /path/to/downloaded-r2-objects +``` + +R2 uses its S3-compatible endpoint and boto3's normal credential chain: + +```sh +uv run adc-analysis inventory \ + --workspace /secure/adc-work \ + --s3-bucket adc-ciphertext \ + --s3-endpoint-url https://ACCOUNT_ID.r2.cloudflarestorage.com \ + --s3-region auto \ + --s3-prefix uploads/ +``` + +Both source types may be inventoried into one manifest in a single command by supplying +`--local ...` and `--s3-bucket ...` together. Inventory is an explicit snapshot: rerunning the +command replaces the manifest with exactly the objects supplied during that invocation while +retaining the immutable content-addressed ciphertext cache. + +Materialization needs a local mode-0600 key file. Keys are unpadded base64url raw X25519 private +keys, keyed by the signed configuration's researcher key ID: + +```json +{"format":"adc-analysis-keys-v1","keys":{"researcher-key-id":"RAW_PRIVATE_KEY_BASE64URL"}} +``` + +```sh +chmod 600 /secure/researcher-keys.json +uv run adc-analysis materialize \ + --workspace /secure/adc-work \ + --keys /secure/researcher-keys.json \ + --catalog ../protocol/v1/collector-catalog.json \ + --output /secure/datasets/study-2026-08 +``` + +The output path must not already exist or be a symbolic link. Publication uses an OS-level atomic, +create-only rename of a complete sibling staging directory, so a concurrently appearing empty +directory is never replaced. Hive-style partitions are +`experiment_id/configuration_id/collector_id/payload_schema_version/payload_type`; files contain +explicit Arrow schemas, exact 64-bit clocks/sequences, and source ciphertext provenance. + +Run all checks with: + +```sh +uv run ruff check src tests +uv run python -m unittest discover -s tests -v +``` + +## Operational notes + +- Keep the workspace, key file, and dataset on encrypted researcher-controlled storage. +- Inventory downloads ciphertext before any key is used. The receiver remains R2-only and has no + list/decrypt/admin API. +- `reports/validation-report.json` records quarantine and conflict outcomes even when publication + stops. `quality-summary.json` distinguishes overlaps, identical duplicates, conflicts, interior + gaps, undelivered suffixes, reclaimed prefixes, and achieved mean sampling rates (rounded to the + nearest millihertz, with the exact interval count and duration retained); it does not infer + participant behavior. Sensor rates use the catalog-declared hardware/source + `source_elapsed_realtime_nanos`, not callback-envelope time, so Android FIFO batching does not + collapse the measured duration. +- Potentially large quality collections use the stable + `{"count":"…","examples":[…],"examples_truncated":true|false}` shape. Counts remain exact; + at most 100 deterministic examples are retained. Nested details such as gap ranges and boot + session IDs use the same shape, so consumers must not treat `examples` as the complete set when + `examples_truncated` is true. +- An identical duplicate has the same authenticated event identity and bytes. A conflict has the + same identity but different bytes and stops publication. An interior gap is absent below the + highest arrived sequence; an undelivered suffix is absent after it but at or below the latest + authenticated durable boundary; a reclaimed prefix is absent below the latest authenticated + retained boundary. These labels describe evidence availability, not why a participant did or did + not produce an observation. +- Database connectors are intentionally out of scope. `DatasetSink` is the narrow extension + contract; Parquet is the only implementation in this release. diff --git a/adc-analysis/pyproject.toml b/adc-analysis/pyproject.toml new file mode 100644 index 0000000..67e313f --- /dev/null +++ b/adc-analysis/pyproject.toml @@ -0,0 +1,35 @@ +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "adc-analysis" +version = "0.1.0" +description = "Offline verification and typed Parquet materialization for ADC Protocol v1" +readme = "README.md" +requires-python = ">=3.11" +license = "MIT" +dependencies = [ + "boto3>=1.40,<2", + "cryptography>=48.0.1,<49", + "ijson>=3.4,<4", + "pyarrow>=21,<24", +] + +[dependency-groups] +dev = ["ruff>=0.16,<0.17"] + +[project.scripts] +adc-analysis = "adc_analysis.cli:main" + +[tool.setuptools] +package-dir = {"" = "src"} + +[tool.setuptools.packages.find] +where = ["src"] + +[tool.setuptools.package-data] +adc_analysis = ["py.typed"] + +[tool.ruff] +target-version = "py311" diff --git a/adc-analysis/src/adc_analysis/__init__.py b/adc-analysis/src/adc_analysis/__init__.py new file mode 100644 index 0000000..f1581cc --- /dev/null +++ b/adc-analysis/src/adc_analysis/__init__.py @@ -0,0 +1,8 @@ +"""Fail-closed offline analysis for ADC Protocol v1.""" + +__all__: list[str] = [] +__version__ = "0.1.0" + +from .pipeline import AnalysisPipeline + +__all__ = ["AnalysisPipeline"] diff --git a/adc-analysis/src/adc_analysis/bundle.py b/adc-analysis/src/adc_analysis/bundle.py new file mode 100644 index 0000000..d6e7f3f --- /dev/null +++ b/adc-analysis/src/adc_analysis/bundle.py @@ -0,0 +1,840 @@ +"""Streaming whole-bundle Protocol v1 authentication and semantic verification.""" + +from __future__ import annotations + +import hashlib +import os +import tempfile +import uuid +from collections.abc import Iterator, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from cryptography.exceptions import InvalidSignature, InvalidTag +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey +from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + +from .catalog import CollectorCatalog +from .configuration import validate_configuration +from .crypto import open_base, public_key +from .encoding import base64url_decode, protocol_id, sha256_hex, uuid4_text, uuid_text +from .errors import ValidationError +from .filesystem import private_directory +from .jcs import canonical_decimal, canonicalize, exact_object, parse +from .limits import ( + AUTOMATIC_UPLOAD_MAX_BYTES, + MANUAL_EXPORT_MAX_BYTES, + SIGNED_CONFIGURATION_MAX_BYTES, +) +from .models import EventProvenance, InventoryObject, VerifiedBundle, VerifiedEvent +from .streaming_json import BoundedObjectBuilder, CanonicalJsonEvents + +MAGIC = b"ADCEXP01" +ROOT_KEYS = { + "bundle_id", + "bundle_kind", + "configuration", + "configuration_sha256", + "configuration_signature", + "experiment", + "exported_at_utc_millis", + "format", + "producer", +} +EXPERIMENT_KEYS = { + "assigned_participant_id", + "configuration_id", + "durable_through_sequence", + "event_count", + "events", + "experiment_id", + "first_sequence_number", + "last_sequence_number", + "next_sequence_number", + "participant_instance_id", + "retained_from_sequence", + "state", + "transitions", + "uploaded_through_sequence", +} +EVENT_KEYS = { + "sequence_number", + "collector_id", + "payload_schema_version", + "observed_time", + "payload_type", + "fields", +} +TIME_KEYS = {"boot_session_id", "monotonic_time_nanos", "wall_time_utc_millis"} +ROOT_SCALARS = ROOT_KEYS - { + "configuration", + "configuration_signature", + "experiment", + "producer", +} +EXPERIMENT_SCALARS = EXPERIMENT_KEYS - {"events", "transitions"} +SCALAR_EVENTS = {"null", "boolean", "integer", "number", "string"} +STATES = { + "IMPORTED", + "CONFIG_VERIFIED", + "CONSENT_PENDING", + "ACCESS_SETUP", + "READY", + "RUNNING", + "PAUSED", + "COMPLETED", + "WITHDRAWN", +} +REASON_DESTINATION = { + "CONFIGURATION_SIGNATURE_VERIFIED": "CONFIG_VERIFIED", + "CONSENT_REVIEW_OPENED": "CONSENT_PENDING", + "CONSENT_ACCEPTED": "ACCESS_SETUP", + "ACCESS_PREFLIGHT_PASSED": "READY", + "PARTICIPANT_STARTED": "RUNNING", + "PARTICIPANT_PAUSED": "PAUSED", + "PARTICIPANT_RESUMED": "RUNNING", + "PARTICIPANT_FINISHED_EARLY": "COMPLETED", + "STUDY_DURATION_ELAPSED": "COMPLETED", + "PARTICIPANT_WITHDREW": "WITHDRAWN", + "STORAGE_FAILURE": "PAUSED", +} +ALLOWED_TRANSITIONS = { + "IMPORTED": {"CONFIG_VERIFIED", "WITHDRAWN"}, + "CONFIG_VERIFIED": {"CONSENT_PENDING", "WITHDRAWN"}, + "CONSENT_PENDING": {"ACCESS_SETUP", "WITHDRAWN"}, + "ACCESS_SETUP": {"READY", "WITHDRAWN"}, + "READY": {"RUNNING", "WITHDRAWN"}, + "RUNNING": {"PAUSED", "COMPLETED", "WITHDRAWN"}, + "PAUSED": {"RUNNING", "COMPLETED", "WITHDRAWN"}, + "COMPLETED": {"WITHDRAWN"}, + "WITHDRAWN": set(), +} + + +@dataclass(frozen=True, slots=True) +class _EventContext: + configuration: Mapping[str, Any] + participant_id: str + configured_collectors: frozenset[str] + provenance: EventProvenance + first_sequence: int + event_count: int + + +class _VerifiedEventSpool: + """Owner-only, integrity-checked event stream for every verified bundle.""" + + def __init__( + self, + path: Path, + verifier: BundleVerifier, + context: _EventContext, + sha256: str, + byte_count: int, + ): + self.path = path + self.verifier = verifier + self.context = context + self.sha256 = sha256 + self.byte_count = byte_count + self.closed = False + + def __iter__(self) -> Iterator[VerifiedEvent]: + if self.closed: + raise ValidationError("verified event spool is closed") + yield from self.verifier._iter_event_file( + self.path, self.context, self.sha256, self.byte_count + ) + + def __len__(self) -> int: + return self.context.event_count + + def close(self) -> None: + if not self.closed: + self.path.unlink(missing_ok=True) + self.closed = True + + def __del__(self) -> None: + try: + self.close() + except OSError: + pass + + +class BundleVerifier: + def __init__( + self, + catalog: CollectorCatalog, + researcher_private_keys: Mapping[str, bytes], + staging_directory: Path, + ): + self.catalog = catalog + self.keys = dict(researcher_private_keys) + if not self.keys: + raise ValidationError("at least one researcher key is required") + for key_id, key in self.keys.items(): + protocol_id(key_id, "researcher key ID") + if not isinstance(key, bytes) or len(key) != 32: + raise ValidationError( + "researcher private keys must be raw 32-byte values" + ) + self.staging = private_directory(staging_directory) + + def verify(self, source: InventoryObject) -> VerifiedBundle: + maximum = ( + AUTOMATIC_UPLOAD_MAX_BYTES + if source.source_kind == "receiver" + else MANUAL_EXPORT_MAX_BYTES + ) + if not 1 <= source.byte_count <= maximum: + raise ValidationError("ciphertext size is outside its source bound") + fd, name = tempfile.mkstemp( + prefix="adc-plaintext-", suffix=".json", dir=self.staging + ) + plaintext_path = Path(name) + try: + os.fchmod(fd, 0o600) + with ( + source.cache_path.open("rb", buffering=0) as encoded, + os.fdopen(fd, "wb", buffering=0) as plaintext, + ): + outer, private_key, plaintext_sha256, plaintext_bytes = ( + self._decrypt_to(encoded, plaintext, source) + ) + plaintext.flush() + os.fsync(plaintext.fileno()) + if plaintext_path.stat().st_mode & 0o077: + raise ValidationError("plaintext staging permissions are not private") + return self._validate_staged( + plaintext_path, + plaintext_sha256, + plaintext_bytes, + outer, + source, + private_key, + ) + finally: + plaintext_path.unlink(missing_ok=True) + + def _decrypt_to( + self, encoded, plaintext, source: InventoryObject + ) -> tuple[dict[str, Any], bytes, str, int]: + if os.fstat(encoded.fileno()).st_size != source.byte_count: + raise ValidationError("ciphertext size does not match inventory") + ciphertext_digest = hashlib.sha256() + count = 0 + + def read_exact(length: int) -> bytes: + nonlocal count + chunks = bytearray() + while len(chunks) < length: + chunk = encoded.read(length - len(chunks)) + if not chunk: + raise ValidationError("bundle container is truncated") + chunks.extend(chunk) + count += len(chunk) + ciphertext_digest.update(chunk) + return bytes(chunks) + + fixed = read_exact(70) + if fixed[:8] != MAGIC: + raise ValidationError("unsupported bundle container format") + bundle_id = uuid4_text( + str(uuid.UUID(bytes=fixed[8:24])), "outer bundle ID" + ) + key_length = int.from_bytes(fixed[56:58], "big") + if not 3 <= key_length <= 64: + raise ValidationError("researcher key ID length is invalid") + if source.byte_count <= 150 + key_length + 16: + raise ValidationError("bundle container size is invalid") + key_and_wrapped = read_exact(key_length + 80) + try: + researcher_key_id = key_and_wrapped[:key_length].decode( + "utf-8", errors="strict" + ) + except UnicodeDecodeError as error: + raise ValidationError("researcher key ID is malformed UTF-8") from error + protocol_id(researcher_key_id, "researcher key ID") + try: + private_key = self.keys[researcher_key_id] + except KeyError as error: + raise ValidationError("no private key for researcher key ID") from error + outer = { + "bundle_id": bundle_id, + "configuration_sha256": fixed[24:56].hex(), + "content_nonce": fixed[58:70], + "researcher_key_id": researcher_key_id, + "wrapped_key": key_and_wrapped[key_length:], + } + context = _bundle_context(outer) + content_key = open_base(private_key, outer["wrapped_key"], context) + if len(content_key) != 32: + raise ValidationError("HPKE content key length is invalid") + + decryptor = Cipher( + algorithms.AES(content_key), modes.GCM(outer["content_nonce"]) + ).decryptor() + decryptor.authenticate_additional_data(context) + tail = b"" + plaintext_digest = hashlib.sha256() + plaintext_count = 0 + while chunk := encoded.read(1024 * 1024): + count += len(chunk) + if count > source.byte_count: + raise ValidationError("ciphertext grew while it was being read") + ciphertext_digest.update(chunk) + buffered = tail + chunk + if len(buffered) <= 16: + tail = buffered + continue + body, tail = buffered[:-16], buffered[-16:] + decoded = decryptor.update(body) + plaintext.write(decoded) + plaintext_digest.update(decoded) + plaintext_count += len(decoded) + if count != source.byte_count: + raise ValidationError("ciphertext size does not match inventory") + if ciphertext_digest.hexdigest() != source.sha256: + raise ValidationError("ciphertext digest does not match inventory") + if len(tail) != 16: + raise ValidationError("bundle content authentication tag is missing") + try: + decoded = decryptor.finalize_with_tag(tail) + except (ValueError, InvalidTag) as error: + raise ValidationError("bundle content authentication failed") from error + plaintext.write(decoded) + plaintext_digest.update(decoded) + plaintext_count += len(decoded) + return outer, private_key, plaintext_digest.hexdigest(), plaintext_count + + def _validate_staged( + self, + path: Path, + plaintext_sha256: str, + plaintext_bytes: int, + outer: Mapping[str, Any], + source: InventoryObject, + private_key: bytes, + ) -> VerifiedBundle: + fd, name = tempfile.mkstemp( + prefix="adc-events-", suffix=".jsonl", dir=self.staging + ) + event_path = Path(name) + keep_event_path = False + try: + os.fchmod(fd, 0o600) + with os.fdopen(fd, "wb") as event_stream: + parsed = self._stream_document( + path, + plaintext_sha256, + plaintext_bytes, + event_stream, + outer, + source, + private_key, + ) + event_stream.flush() + os.fsync(event_stream.fileno()) + bundle = self._finish_document(parsed, event_path, outer, source, private_key) + keep_event_path = isinstance(bundle.events, _VerifiedEventSpool) + return bundle + finally: + if not keep_event_path: + event_path.unlink(missing_ok=True) + + def _stream_document( + self, + path: Path, + plaintext_sha256: str, + plaintext_bytes: int, + event_stream, + outer: Mapping[str, Any], + source: InventoryObject, + private_key: bytes, + ) -> dict[str, Any]: + root_keys: set[str] = set() + experiment_keys: set[str] = set() + root_values: dict[str, Any] = {} + experiment_values: dict[str, Any] = {} + configuration = None + signature = None + producer = None + active_name: str | None = None + active: BoundedObjectBuilder | None = None + root_started = False + experiment_started = False + events_started = False + transitions_started = False + parsed_events = 0 + transition_previous = "IMPORTED" + + for prefix, event, value in CanonicalJsonEvents( + path, plaintext_sha256, plaintext_bytes + ): + if active is not None: + active.feed(event, value) + if active.complete: + built = active.value + if active_name == "configuration": + configuration = validate_configuration(built, self.catalog) + elif active_name == "configuration_signature": + signature = built + elif active_name == "producer": + producer = built + elif active_name == "event": + encoded_event = canonicalize(built) + event_stream.write(encoded_event) + event_stream.write(b"\n") + parsed_events += 1 + elif active_name == "transition": + transition_previous = _validate_transition( + built, transition_previous + ) + active_name = None + active = None + continue + + if prefix == "" and event == "start_map": + root_started = True + continue + if prefix == "" and event == "map_key": + if value not in ROOT_KEYS: + raise ValidationError(f"unknown bundle document member: {value}") + root_keys.add(value) + continue + if prefix == "experiment" and event == "start_map": + experiment_started = True + continue + if prefix == "experiment" and event == "map_key": + if value not in EXPERIMENT_KEYS: + raise ValidationError(f"unknown experiment snapshot member: {value}") + experiment_keys.add(value) + continue + if prefix == "experiment.events" and event == "start_array": + events_started = True + continue + if prefix == "experiment.transitions" and event == "start_array": + transitions_started = True + continue + + subtree = _subtree_target(prefix, event, self.catalog.maximum_event_bytes) + if subtree is not None: + active_name, bound = subtree + active = BoundedObjectBuilder(bound) + active.feed(event, value) + continue + if prefix in ROOT_SCALARS and event in SCALAR_EVENTS: + root_values[prefix] = value + continue + if prefix.startswith("experiment."): + name = prefix.removeprefix("experiment.") + if name in EXPERIMENT_SCALARS and event in SCALAR_EVENTS: + experiment_values[name] = value + continue + if prefix == "experiment.events.item": + raise ValidationError("event array items must be objects") + if prefix == "experiment.transitions.item": + raise ValidationError("transition array items must be objects") + + if active is not None: + raise ValidationError("bundle document contains an incomplete subtree") + if not root_started or not experiment_started: + raise ValidationError("bundle document root/experiment must be objects") + if not events_started or not transitions_started: + raise ValidationError("events and transitions must be arrays") + if root_keys != ROOT_KEYS: + raise ValidationError(f"bundle document keys mismatch: {sorted(root_keys)}") + if experiment_keys != EXPERIMENT_KEYS: + raise ValidationError( + f"experiment snapshot keys mismatch: {sorted(experiment_keys)}" + ) + if configuration is None or signature is None or producer is None: + raise ValidationError("bundle document object member has the wrong type") + return { + "configuration": configuration, + "configuration_signature": signature, + "experiment": experiment_values, + "parsed_events": parsed_events, + "producer": producer, + "root": root_values, + "transition_final_state": transition_previous, + } + + def _finish_document( + self, + parsed: Mapping[str, Any], + event_path: Path, + outer: Mapping[str, Any], + source: InventoryObject, + private_key: bytes, + ) -> VerifiedBundle: + root = parsed["root"] + if root.get("format") != "research-bundle-v1": + raise ValidationError("unsupported bundle document format") + if uuid4_text(root.get("bundle_id"), "bundle ID") != outer["bundle_id"]: + raise ValidationError("outer and inner bundle IDs differ") + bundle_kind = root.get("bundle_kind") + if bundle_kind not in {"manual_export", "automatic_upload"}: + raise ValidationError("unknown bundle kind") + if source.source_kind == "receiver" and bundle_kind != "automatic_upload": + raise ValidationError("receiver inventory object is not an automatic upload") + configuration_digest = sha256_hex( + root.get("configuration_sha256"), "configuration digest" + ) + if configuration_digest != outer["configuration_sha256"]: + raise ValidationError("outer and inner configuration digests differ") + + configuration = parsed["configuration"] + configuration_bytes = canonicalize(configuration) + if len(configuration_bytes) > SIGNED_CONFIGURATION_MAX_BYTES: + raise ValidationError("embedded configuration exceeds Protocol v1 bound") + if hashlib.sha256(configuration_bytes).hexdigest() != configuration_digest: + raise ValidationError("embedded configuration digest mismatch") + signature = exact_object( + parsed["configuration_signature"], + {"signer_key_id", "signature"}, + "configuration signature", + ) + if signature["signer_key_id"] != configuration["signer"]["key_id"]: + raise ValidationError("configuration signer provenance mismatch") + signature_bytes = base64url_decode( + signature["signature"], 64, "configuration signature" + ) + signing_key = base64url_decode( + configuration["signer"]["public_key"], 32, "signer public key" + ) + try: + Ed25519PublicKey.from_public_bytes(signing_key).verify( + signature_bytes, configuration_bytes + ) + except (ValueError, InvalidSignature) as error: + raise ValidationError( + "configuration signature verification failed" + ) from error + export = configuration["export"] + if export["researcher_key_id"] != outer["researcher_key_id"]: + raise ValidationError("researcher key ID differs from configuration") + expected_public = base64url_decode( + export["hpke_public_key"], 32, "researcher public key" + ) + if public_key(private_key) != expected_public: + raise ValidationError("researcher private key does not match configuration") + + if bundle_kind == "automatic_upload": + if source.byte_count > AUTOMATIC_UPLOAD_MAX_BYTES: + raise ValidationError("automatic upload exceeds 32 MiB") + elif source.byte_count > configuration["storage"]["maximum_local_bytes"]: + raise ValidationError("manual export exceeds its signed local storage quota") + + producer = exact_object( + parsed["producer"], {"client_version", "platform"}, "producer" + ) + if producer["platform"] != configuration["platform"]: + raise ValidationError("producer platform differs from configuration") + client_version = canonical_decimal( + producer["client_version"], "producer client version" + ) + minimum_version = canonical_decimal( + configuration["minimum_client_version"], "minimum client version" + ) + if client_version < minimum_version: + raise ValidationError("producer client version is too old") + exported_at = canonical_decimal( + root.get("exported_at_utc_millis"), "exported_at_utc_millis" + ) + return self._finish_experiment( + parsed, + event_path, + configuration, + outer, + source, + bundle_kind, + exported_at, + ) + + def _finish_experiment( + self, + parsed: Mapping[str, Any], + event_path: Path, + configuration: Mapping[str, Any], + outer: Mapping[str, Any], + source: InventoryObject, + bundle_kind: str, + exported_at: int, + ) -> VerifiedBundle: + root = parsed["experiment"] + if root.get("experiment_id") != configuration["experiment_id"]: + raise ValidationError("experiment ID differs from configuration") + if root.get("configuration_id") != configuration["configuration_id"]: + raise ValidationError("configuration ID differs from configuration") + if root.get("assigned_participant_id") != configuration[ + "assigned_participant_id" + ]: + raise ValidationError("assigned participant ID differs from configuration") + participant_id = uuid_text( + root.get("participant_instance_id"), "participant instance ID" + ) + state = root.get("state") + if state not in STATES: + raise ValidationError("experiment state is invalid") + if parsed["transition_final_state"] != state: + raise ValidationError("final transition does not match experiment state") + + first = canonical_decimal(root.get("first_sequence_number"), "first_sequence_number") + last = canonical_decimal(root.get("last_sequence_number"), "last_sequence_number") + count = canonical_decimal(root.get("event_count"), "event_count") + durable = canonical_decimal( + root.get("durable_through_sequence"), "durable_through_sequence" + ) + next_sequence = canonical_decimal( + root.get("next_sequence_number"), "next_sequence_number" + ) + retained = canonical_decimal( + root.get("retained_from_sequence"), "retained_from_sequence" + ) + uploaded = canonical_decimal( + root.get("uploaded_through_sequence"), "uploaded_through_sequence" + ) + if ( + first < 1 + or next_sequence != durable + 1 + or not 1 <= retained <= next_sequence + or not retained <= first <= next_sequence + ): + raise ValidationError("experiment sequence boundaries are inconsistent") + if uploaded > durable or retained > uploaded + 1: + raise ValidationError("retained/uploaded boundaries are inconsistent") + expected_last = first - 1 if count == 0 else first + count - 1 + if last != expected_last or last > durable: + raise ValidationError("bundle range/count is inconsistent") + if parsed["parsed_events"] != count: + raise ValidationError("event array count mismatch") + if bundle_kind == "automatic_upload" and count == 0: + raise ValidationError("automatic upload cannot be empty") + if bundle_kind == "automatic_upload" and first != uploaded + 1: + raise ValidationError("automatic upload does not start after its watermark") + + configured_collectors = {item["id"] for item in configuration["collectors"]} + if configuration["interventions"]: + configured_collectors.add("interventions.v1") + context = _EventContext( + configuration, + participant_id, + frozenset(configured_collectors), + EventProvenance( + source.sha256, + outer["bundle_id"], + outer["configuration_sha256"], + source.source_uri, + ), + first, + count, + ) + event_digest = hashlib.sha256() + event_bytes = 0 + for event in self._iter_event_file(event_path, context): + encoded_size = len(event.canonical_bytes) + 1 + event_bytes += encoded_size + event_digest.update(event.canonical_bytes) + event_digest.update(b"\n") + self._validate_receiver_metadata(source, outer, first, last, count) + os.chmod(event_path, 0o400) + verified_events = _VerifiedEventSpool( + event_path, + self, + context, + event_digest.hexdigest(), + event_bytes, + ) + return VerifiedBundle( + outer["bundle_id"], + bundle_kind, + outer["configuration_sha256"], + configuration["experiment_id"], + configuration["configuration_id"], + participant_id, + exported_at, + first, + last, + count, + retained, + uploaded, + durable, + next_sequence, + verified_events, + source, + ) + + def _iter_event_file( + self, + path: Path, + context: _EventContext, + expected_sha256: str | None = None, + expected_bytes: int | None = None, + ) -> Iterator[VerifiedEvent]: + digest = hashlib.sha256() + byte_count = 0 + index = 0 + with path.open("rb") as stream: + while line := stream.readline(self.catalog.maximum_event_bytes + 2): + byte_count += len(line) + digest.update(line) + if ( + len(line) > self.catalog.maximum_event_bytes + 1 + or not line.endswith(b"\n") + ): + raise ValidationError("event spool record exceeds its bound") + event = parse(line[:-1]) + yield self._validate_event( + event, + context.first_sequence + index, + context.configuration, + context.participant_id, + context.configured_collectors, + context.provenance, + ) + index += 1 + if index != context.event_count: + raise ValidationError("event spool count changed after authentication") + if expected_bytes is not None and byte_count != expected_bytes: + raise ValidationError("event spool size changed after authentication") + if expected_sha256 is not None and digest.hexdigest() != expected_sha256: + raise ValidationError("event spool digest changed after authentication") + + def _validate_event( + self, + value: Any, + expected_sequence: int, + configuration: Mapping[str, Any], + participant_id: str, + configured_collectors: frozenset[str], + provenance: EventProvenance, + ) -> VerifiedEvent: + root = exact_object(value, EVENT_KEYS, "event") + sequence = canonical_decimal(root["sequence_number"], "event sequence_number") + if sequence != expected_sequence or sequence < 1: + raise ValidationError("event sequence is not contiguous") + collector_id = root["collector_id"] + if ( + not isinstance(collector_id, str) + or collector_id not in configured_collectors + ): + raise ValidationError("event collector is not enabled by configuration") + schema_version = root["payload_schema_version"] + if isinstance(schema_version, bool) or not isinstance(schema_version, int): + raise ValidationError("payload_schema_version must be an integer") + payload_type = root["payload_type"] + if not isinstance(payload_type, str): + raise ValidationError("payload_type must be a string") + schema = self.catalog.payload(collector_id, schema_version, payload_type) + typed_fields = self.catalog.typed_fields(schema, root["fields"]) + self.catalog.validate_event_size(root, schema) + boot, monotonic, wall = _research_time(root["observed_time"]) + return VerifiedEvent( + configuration["experiment_id"], + configuration["configuration_id"], + participant_id, + configuration["assigned_participant_id"], + sequence, + collector_id, + schema_version, + payload_type, + boot, + monotonic, + wall, + typed_fields, + canonicalize(root), + provenance, + ) + + def _validate_receiver_metadata( + self, + source: InventoryObject, + outer: Mapping[str, Any], + first: int, + last: int, + count: int, + ) -> None: + metadata = source.metadata + if metadata is None: + return + expected = { + "sha256": source.sha256, + "byte_count": str(source.byte_count), + "configuration_sha256": outer["configuration_sha256"], + "researcher_key_id": outer["researcher_key_id"], + "first_sequence_number": str(first), + "last_sequence_number": str(last), + "event_count": str(count), + } + for key, value in expected.items(): + if metadata.get(key) != value: + raise ValidationError(f"receiver metadata mismatch: {key}") + + +def _subtree_target( + prefix: str, event: str, maximum_event_bytes: int +) -> tuple[str, int] | None: + if event != "start_map": + if prefix in { + "configuration", + "configuration_signature", + "producer", + }: + raise ValidationError(f"{prefix} must be an object") + return None + if prefix == "configuration": + return "configuration", SIGNED_CONFIGURATION_MAX_BYTES + if prefix == "configuration_signature": + return "configuration_signature", 4096 + if prefix == "producer": + return "producer", 4096 + if prefix == "experiment.events.item": + return "event", maximum_event_bytes + if prefix == "experiment.transitions.item": + return "transition", 4096 + return None + + +def _validate_transition(value: Any, previous_to: str) -> str: + item = exact_object(value, {"from", "reason", "time", "to"}, "transition") + if item["from"] not in STATES or item["to"] not in STATES: + raise ValidationError("transition state is invalid") + if ( + item["reason"] not in REASON_DESTINATION + or REASON_DESTINATION[item["reason"]] != item["to"] + ): + raise ValidationError("transition reason/destination mismatch") + if ( + item["from"] != previous_to + or item["to"] not in ALLOWED_TRANSITIONS[item["from"]] + ): + raise ValidationError("transition history is discontinuous") + _research_time(item["time"]) + return item["to"] + + +def _bundle_context(outer: Mapping[str, Any]) -> bytes: + return canonicalize( + { + "bundle_format": "research-bundle-v1", + "bundle_id": outer["bundle_id"], + "configuration_sha256": outer["configuration_sha256"], + "researcher_key_id": outer["researcher_key_id"], + } + ) + + +def _research_time(value: Any) -> tuple[str, int, int]: + root = exact_object(value, TIME_KEYS, "research time") + boot = root["boot_session_id"] + if ( + not isinstance(boot, str) + or not boot.strip() + or not 1 <= len(boot.encode("utf-8")) <= 128 + ): + raise ValidationError("boot_session_id is invalid") + monotonic = canonical_decimal(root["monotonic_time_nanos"], "monotonic_time_nanos") + wall = canonical_decimal(root["wall_time_utc_millis"], "wall_time_utc_millis") + return boot, monotonic, wall diff --git a/adc-analysis/src/adc_analysis/catalog.py b/adc-analysis/src/adc_analysis/catalog.py new file mode 100644 index 0000000..641dd88 --- /dev/null +++ b/adc-analysis/src/adc_analysis/catalog.py @@ -0,0 +1,295 @@ +"""Closed-world collector catalog loading and payload conversion.""" + +from __future__ import annotations + +import math +import re +import struct +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from .errors import ValidationError +from .jcs import ( + canonical_decimal, + canonicalize, + exact_object, + parse, + parse_embedded_json, +) + +_SIGNED = re.compile(r"(?:0|-?[1-9][0-9]*)\Z") +_FLOAT = re.compile(r"[+-]?(?:(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?)\Z") +_COLLECTOR_ID = re.compile(r"[a-z][a-z0-9_.-]{2,63}\Z") +_PAYLOAD_TYPE = re.compile(r"[A-Z][A-Z0-9_]{1,63}\Z") + + +@dataclass(frozen=True, slots=True) +class PayloadSchema: + collector_id: str + schema_version: int + payload_type: str + maximum_encoded_event_bytes: int + fields: Mapping[str, Mapping[str, Any]] + + +class CollectorCatalog: + """The one catalog used for validation and Parquet schema generation.""" + + def __init__(self, path: Path): + root = parse(path.read_bytes(), require_canonical=False) + exact_object( + root, + { + "catalog_format", + "catalog_version", + "protocol_schema_version", + "collectors", + }, + "catalog", + ) + if ( + root["catalog_format"] != "adc-collector-catalog-v1" + or isinstance(root["catalog_version"], bool) + or root["catalog_version"] != 1 + or isinstance(root["protocol_schema_version"], bool) + or root["protocol_schema_version"] != 1 + ): + raise ValidationError("unsupported collector catalog") + if not isinstance(root["collectors"], list): + raise ValidationError("catalog collectors must be an array") + self._collectors: dict[str, Mapping[str, Any]] = {} + self._payloads: dict[tuple[str, int, str], PayloadSchema] = {} + self._sampling_clock_fields: dict[tuple[str, int, str], str] = {} + for collector in root["collectors"]: + self._load_collector(collector) + + @property + def collector_ids(self) -> frozenset[str]: + return frozenset(self._collectors) + + @property + def payload_schemas(self) -> tuple[PayloadSchema, ...]: + return tuple(self._payloads[key] for key in sorted(self._payloads)) + + @property + def maximum_event_bytes(self) -> int: + return max( + schema.maximum_encoded_event_bytes for schema in self._payloads.values() + ) + + @property + def sampling_clock_fields(self) -> Mapping[tuple[str, int, str], str]: + """Catalog-selected source clocks for achieved-rate summaries.""" + + return dict(self._sampling_clock_fields) + + def collector_configuration(self, collector_id: str) -> Mapping[str, Any]: + try: + configuration = self._collectors[collector_id]["configuration"] + except KeyError as error: + raise ValidationError(f"unknown collector: {collector_id}") from error + if configuration is None: + raise ValidationError(f"collector is not configurable: {collector_id}") + return configuration + + def payload( + self, collector_id: str, schema_version: int, payload_type: str + ) -> PayloadSchema: + try: + return self._payloads[(collector_id, schema_version, payload_type)] + except KeyError as error: + raise ValidationError( + f"unknown payload: {collector_id}/{schema_version}/{payload_type}" + ) from error + + def validate_collector_config(self, collector: Any) -> str: + root = exact_object(collector, {"id", "required", "config"}, "collector") + collector_id = _string(root["id"], "collector id") + if not isinstance(root["required"], bool): + raise ValidationError("collector required must be boolean") + schema = self.collector_configuration(collector_id) + config = root["config"] + if not isinstance(config, dict): + raise ValidationError("collector config must be an object") + fields = schema["fields"] + required = set(schema["required"]) + if set(config) != set(fields) or set(fields) != required: + raise ValidationError(f"collector config keys mismatch: {collector_id}") + for name, descriptor in fields.items(): + value = config[name] + kind = descriptor["type"] + if kind == "integer": + if isinstance(value, bool) or not isinstance(value, int): + raise ValidationError(f"{collector_id}.{name} must be an integer") + if not descriptor["minimum"] <= value <= descriptor["maximum"]: + raise ValidationError(f"{collector_id}.{name} is outside bounds") + elif kind == "boolean": + if not isinstance(value, bool): + raise ValidationError(f"{collector_id}.{name} must be boolean") + elif kind == "enum_array": + allowed = descriptor["items_enum"] + if ( + not isinstance(value, list) + or not value + or any( + not isinstance(item, str) or item not in allowed + for item in value + ) + or value != sorted(set(value)) + ): + raise ValidationError( + f"{collector_id}.{name} must be a sorted unique enum array" + ) + elif kind == "enum": + if not isinstance(value, str) or value not in descriptor["enum"]: + raise ValidationError(f"{collector_id}.{name} is an invalid enum") + else: + raise ValidationError(f"unknown catalog configuration type: {kind}") + for name, descriptor in fields.items(): + maximum_field = descriptor.get("maximum_field") + if maximum_field and config[name] > config[maximum_field]: + raise ValidationError(f"{collector_id}.{name} exceeds {maximum_field}") + return collector_id + + def typed_fields(self, schema: PayloadSchema, fields: Any) -> dict[str, Any]: + if not isinstance(fields, dict) or not set(fields) <= set(schema.fields): + raise ValidationError(f"payload field set mismatch: {schema.payload_type}") + required = { + name for name, descriptor in schema.fields.items() if descriptor["required"] + } + if not required <= set(fields): + raise ValidationError( + f"required payload fields missing: {schema.payload_type}" + ) + return { + name: _convert_field(name, fields[name], descriptor) + for name, descriptor in schema.fields.items() + if name in fields + } + + def validate_event_size( + self, event: Mapping[str, Any], schema: PayloadSchema + ) -> None: + if len(canonicalize(event)) > schema.maximum_encoded_event_bytes: + raise ValidationError("event exceeds catalog maximum_encoded_event_bytes") + + def _load_collector(self, collector: Any) -> None: + if not isinstance(collector, dict): + raise ValidationError("catalog collector must be an object") + collector_id = _string(collector.get("id"), "catalog collector id") + if not _COLLECTOR_ID.fullmatch(collector_id): + raise ValidationError("catalog collector ID is invalid") + if collector_id in self._collectors: + raise ValidationError(f"duplicate catalog collector: {collector_id}") + version = collector.get("payload_schema_version") + maximum = collector.get("maximum_encoded_event_bytes") + if isinstance(version, bool) or not isinstance(version, int) or version != 1: + raise ValidationError("unsupported payload schema version") + if ( + isinstance(maximum, bool) + or not isinstance(maximum, int) + or not 128 <= maximum <= 65_536 + ): + raise ValidationError("invalid catalog event bound") + self._collectors[collector_id] = collector + for payload in collector.get("payloads", []): + if not isinstance(payload, dict) or not isinstance( + payload.get("fields"), dict + ): + raise ValidationError("invalid catalog payload") + source_clocks = [ + name + for name, descriptor in payload["fields"].items() + if descriptor.get("clock_basis") == "continuous_monotonic_since_boot" + ] + if len(source_clocks) > 1: + raise ValidationError("payload has multiple continuous source clocks") + source_clock = source_clocks[0] if source_clocks else None + if source_clock is not None: + descriptor = payload["fields"][source_clock] + if ( + descriptor.get("type") != "decimal_string" + or descriptor.get("unit") != "nanosecond" + ): + raise ValidationError( + "continuous source clock must be decimal nanoseconds" + ) + for payload_type in payload.get("types", []): + payload_type = _string(payload_type, "payload type") + if not _PAYLOAD_TYPE.fullmatch(payload_type): + raise ValidationError("catalog payload type is invalid") + key = (collector_id, version, payload_type) + if key in self._payloads: + raise ValidationError(f"duplicate catalog payload: {key}") + self._payloads[key] = PayloadSchema( + collector_id, version, payload_type, maximum, payload["fields"] + ) + if source_clock is not None: + self._sampling_clock_fields[key] = source_clock + + +def _convert_field(name: str, value: Any, descriptor: Mapping[str, Any]) -> Any: + if not isinstance(value, str): + raise ValidationError(f"payload field {name} must use its string wire encoding") + kind = descriptor["type"] + if kind == "decimal_string": + return canonical_decimal(value, name) + if kind == "int32": + if not _SIGNED.fullmatch(value): + raise ValidationError(f"{name} must be a signed decimal integer") + number = int(value) + if not -(2**31) <= number < 2**31: + raise ValidationError(f"{name} is outside int32") + if "minimum" in descriptor and number < descriptor["minimum"]: + raise ValidationError(f"{name} is below minimum") + if "maximum" in descriptor and number > descriptor["maximum"]: + raise ValidationError(f"{name} is above maximum") + return number + if kind == "boolean": + if value not in ("true", "false"): + raise ValidationError(f"{name} must be true or false") + return value == "true" + if kind == "enum": + if value not in descriptor["enum"]: + raise ValidationError(f"{name} is an invalid enum") + return value + if kind in ("float32", "float64"): + if not _FLOAT.fullmatch(value): + raise ValidationError(f"{name} is not a finite decimal float") + number = float(value) + if not math.isfinite(number): + raise ValidationError(f"{name} must be finite") + if "minimum" in descriptor and number < descriptor["minimum"]: + raise ValidationError(f"{name} is below minimum") + if "maximum" in descriptor and number > descriptor["maximum"]: + raise ValidationError(f"{name} is above maximum") + if kind == "float32": + try: + number = struct.unpack(">f", struct.pack(">f", number))[0] + except OverflowError as error: + raise ValidationError(f"{name} is outside float32") from error + if not math.isfinite(number): + raise ValidationError(f"{name} is outside float32") + return number + if kind == "string": + if len(value.encode("utf-16-le")) // 2 > descriptor.get( + "maximum_length", 2**31 + ): + raise ValidationError(f"{name} is too long") + return value + if kind == "json_string": + if len(value.encode("utf-16-le")) // 2 > descriptor.get( + "maximum_length", 2**31 + ): + raise ValidationError(f"{name} JSON is too large") + parse_embedded_json(value) + return value + raise ValidationError(f"unknown catalog payload type: {kind}") + + +def _string(value: Any, name: str) -> str: + if not isinstance(value, str): + raise ValidationError(f"{name} must be a string") + return value diff --git a/adc-analysis/src/adc_analysis/cli.py b/adc-analysis/src/adc_analysis/cli.py new file mode 100644 index 0000000..bb1cb1a --- /dev/null +++ b/adc-analysis/src/adc_analysis/cli.py @@ -0,0 +1,86 @@ +"""Command-line boundary for the offline pipeline.""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +from .catalog import CollectorCatalog +from .errors import AnalysisError +from .inventory import CiphertextInventory +from .pipeline import AnalysisPipeline, load_private_keys +from .sink import ParquetSink +from .sources import BundleSource, LocalBundleSource, S3BundleSource + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="adc-analysis", + description="Inventory, authenticate, and materialize ADC Protocol v1 ciphertext bundles.", + ) + commands = parser.add_subparsers(dest="command", required=True) + inventory = commands.add_parser( + "inventory", help="copy ciphertext into the immutable local cache" + ) + inventory.add_argument("--workspace", type=Path, required=True) + inventory.add_argument("--local", type=Path, nargs="+") + inventory.add_argument("--s3-bucket") + inventory.add_argument("--s3-prefix", default="") + inventory.add_argument("--s3-endpoint-url") + inventory.add_argument("--s3-region") + inventory.add_argument("--s3-profile") + + materialize = commands.add_parser( + "materialize", + help="fully verify inventory and atomically publish typed Parquet", + ) + materialize.add_argument("--workspace", type=Path, required=True) + materialize.add_argument("--keys", type=Path, required=True) + materialize.add_argument("--output", type=Path, required=True) + materialize.add_argument("--catalog", type=Path, required=True) + return parser + + +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + try: + if args.command == "inventory": + sources: list[BundleSource] = [] + if args.local: + sources.append(LocalBundleSource(args.local)) + if args.s3_bucket: + sources.append( + S3BundleSource( + args.s3_bucket, + prefix=args.s3_prefix, + endpoint_url=args.s3_endpoint_url, + region_name=args.s3_region, + profile_name=args.s3_profile, + ) + ) + elif ( + args.s3_prefix + or args.s3_endpoint_url + or args.s3_region + or args.s3_profile + ): + raise AnalysisError("S3 options require --s3-bucket") + if not sources: + raise AnalysisError("at least one of --local or --s3-bucket is required") + objects = CiphertextInventory(args.workspace).ingest(sources) + print(f"inventoried {len(objects)} ciphertext object(s)") + return 0 + catalog = CollectorCatalog(args.catalog) + keys = load_private_keys(args.keys) + pipeline = AnalysisPipeline(args.workspace, catalog, keys, ParquetSink(catalog)) + output = pipeline.materialize(args.output) + print(output) + return 0 + except (AnalysisError, OSError) as error: + parser.exit(2, f"adc-analysis: {error}\n") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/adc-analysis/src/adc_analysis/configuration.py b/adc-analysis/src/adc_analysis/configuration.py new file mode 100644 index 0000000..b69e075 --- /dev/null +++ b/adc-analysis/src/adc_analysis/configuration.py @@ -0,0 +1,439 @@ +"""Semantic validator for the exact Android Protocol v1 configuration object.""" + +from __future__ import annotations + +import re +from calendar import timegm +from datetime import datetime +from typing import Any +from urllib.parse import urlsplit + +from .catalog import CollectorCatalog +from .encoding import base64url_decode, protocol_id +from .errors import ValidationError +from .jcs import canonical_decimal, exact_object + +ROOT_KEYS = { + "schema_version", + "experiment_id", + "configuration_id", + "assigned_participant_id", + "issued_at", + "expires_at", + "platform", + "minimum_client_version", + "title", + "researcher", + "purpose", + "duration_hours", + "consent", + "collectors", + "surveys", + "interventions", + "storage", + "signer", + "export", + "upload", +} +_PARTICIPANT = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}\Z") +_BCP47 = re.compile(r"[A-Za-z]{2,3}(?:-[A-Za-z0-9]{2,8})*\Z") +_LOCAL_TIME = re.compile(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]\Z") +_INSTANT = re.compile( + r"(?P[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2})" + r"(?:\.(?P[0-9]{3}(?:[0-9]{3})?(?:[0-9]{3})?))?Z\Z" +) + + +def validate_configuration(value: Any, catalog: CollectorCatalog) -> dict[str, Any]: + root = exact_object(value, ROOT_KEYS, "configuration") + _integer(root["schema_version"], "schema_version", 1, 1) + protocol_id(root["experiment_id"], "experiment ID") + protocol_id(root["configuration_id"], "configuration ID") + participant = root["assigned_participant_id"] + if participant is not None and ( + not isinstance(participant, str) + or not _PARTICIPANT.fullmatch(participant) + or len(participant.encode("utf-8")) > 64 + ): + raise ValidationError("assigned participant ID is invalid") + issued = _instant(root["issued_at"], "issued_at") + expires = _instant(root["expires_at"], "expires_at") + if issued >= expires: + raise ValidationError("configuration expiry must follow issue time") + if root["platform"] != "android": + raise ValidationError("analysis accepts only Android configurations") + if canonical_decimal(root["minimum_client_version"], "minimum_client_version") < 1: + raise ValidationError("minimum client version must be positive") + _bounded_text(root["title"], "title", 1, 120) + researcher = exact_object(root["researcher"], {"name", "contact"}, "researcher") + _bounded_text(researcher["name"], "researcher name", 1, 120) + _bounded_text(researcher["contact"], "researcher contact", 3, 240) + _bounded_text(root["purpose"], "purpose", 1, 2_000) + duration = _integer(root["duration_hours"], "duration_hours", 1, 8_760) + consent = exact_object(root["consent"], {"document_version", "summary"}, "consent") + _bounded_text(consent["document_version"], "consent document version", 1, 64) + _bounded_text(consent["summary"], "consent summary", 1, 8_000) + + collectors = _array(root["collectors"], "collectors") + if not collectors: + raise ValidationError("at least one collector is required") + collector_ids = [catalog.validate_collector_config(item) for item in collectors] + _unique(collector_ids, "collector ID") + + surveys = _array(root["surveys"], "surveys") + survey_ids = [_validate_survey(item) for item in surveys] + _unique(survey_ids, "survey ID") + interventions = _array(root["interventions"], "interventions") + intervention_ids: list[str] = [] + trigger_ids: list[str] = [] + maximum_occurrences = 0 + for item in interventions: + intervention_id, ids, occurrences, survey_reference = _validate_intervention( + item, duration + ) + intervention_ids.append(intervention_id) + trigger_ids.extend(ids) + maximum_occurrences += occurrences + if survey_reference is not None and survey_reference not in survey_ids: + raise ValidationError("intervention references an unknown survey") + _unique(intervention_ids, "intervention ID") + _unique(trigger_ids, "intervention trigger ID") + if maximum_occurrences > 512: + raise ValidationError("too many intervention occurrences") + + storage = exact_object(root["storage"], {"maximum_local_bytes"}, "storage") + _integer(storage["maximum_local_bytes"], "maximum_local_bytes", 8 << 20, 8 << 30) + signer = exact_object(root["signer"], {"key_id", "public_key"}, "signer") + protocol_id(signer["key_id"], "signer key ID") + base64url_decode(signer["public_key"], 32, "signer public key") + export = exact_object( + root["export"], {"researcher_key_id", "hpke_public_key"}, "export" + ) + protocol_id(export["researcher_key_id"], "researcher key ID") + base64url_decode(export["hpke_public_key"], 32, "researcher HPKE public key") + _validate_upload(root["upload"]) + return root + + +def _validate_upload(value: Any) -> None: + if not isinstance(value, dict): + raise ValidationError("upload must be an object") + if not value: + return + upload = exact_object( + value, {"endpoint", "interval_minutes", "allow_metered"}, "upload" + ) + endpoint = _bounded_text(upload["endpoint"], "upload endpoint", 8, 2_048) + if not endpoint.startswith("https://") or any( + ord(character) <= 0x20 for character in endpoint + ): + raise ValidationError("upload endpoint must be an HTTPS authority") + try: + parsed = urlsplit(endpoint) + host = parsed.hostname + except ValueError as error: + raise ValidationError("upload endpoint must be an HTTPS authority") from error + if parsed.scheme != "https" or not host: + raise ValidationError("upload endpoint must be an HTTPS authority") + _integer(upload["interval_minutes"], "upload interval", 1, 10_080) + if not isinstance(upload["allow_metered"], bool): + raise ValidationError("allow_metered must be boolean") + + +def _validate_survey(value: Any) -> str: + root = exact_object(value, {"id", "title", "description", "questions"}, "survey") + survey_id = protocol_id(root["id"], "survey ID") + _localized(root["title"], "survey title") + _localized(root["description"], "survey description") + questions = _array(root["questions"], "survey questions") + if not 1 <= len(questions) <= 100: + raise ValidationError("survey question count is invalid") + ids = [_validate_question(item) for item in questions] + _unique(ids, "survey question ID") + return survey_id + + +def _validate_question(value: Any) -> str: + if not isinstance(value, dict): + raise ValidationError("survey question must be an object") + kind = value.get("type") + if not isinstance(kind, str): + raise ValidationError("unknown survey question type") + common = {"type", "id", "prompt", "required"} + extra = { + "short_text": {"maximum_length"}, + "scale": {"minimum", "maximum", "minimum_label", "maximum_label"}, + "single_choice": {"options"}, + "multiple_choice": {"options", "minimum_selections", "maximum_selections"}, + }.get(kind) + if extra is None: + raise ValidationError("unknown survey question type") + root = exact_object(value, common | extra, "survey question") + question_id = protocol_id(root["id"], "survey question ID") + _localized(root["prompt"], "question prompt") + if not isinstance(root["required"], bool): + raise ValidationError("survey required must be boolean") + if kind == "short_text": + _integer(root["maximum_length"], "maximum_length", 1, 4_000) + elif kind == "scale": + minimum = _integer(root["minimum"], "scale minimum", -1_000, 1_000) + maximum = _integer(root["maximum"], "scale maximum", -1_000, 1_000) + if minimum >= maximum: + raise ValidationError("scale bounds are invalid") + _localized(root["minimum_label"], "minimum label") + _localized(root["maximum_label"], "maximum label") + else: + options = _array(root["options"], "choice options") + if not 2 <= len(options) <= 50: + raise ValidationError("choice option count is invalid") + option_ids = [] + for option in options: + option = exact_object(option, {"id", "label"}, "choice option") + option_ids.append(protocol_id(option["id"], "choice option ID")) + _localized(option["label"], "choice label") + _unique(option_ids, "choice option ID") + if kind == "multiple_choice": + minimum = _integer( + root["minimum_selections"], "minimum selections", 0, len(options) + ) + maximum = _integer( + root["maximum_selections"], + "maximum selections", + max(1, minimum), + len(options), + ) + if root["required"] and minimum == 0: + raise ValidationError("required multiple choice needs a selection") + return question_id + + +def _validate_intervention( + value: Any, duration_hours: int +) -> tuple[str, list[str], int, str | None]: + root = exact_object(value, {"id", "action", "triggers"}, "intervention") + intervention_id = protocol_id(root["id"], "intervention ID") + action = root["action"] + if not isinstance(action, dict): + raise ValidationError("intervention action must be an object") + kind = action.get("type") + keys = {"type", "notification_title", "notification_message"} + if kind == "survey": + keys.add("survey_id") + elif kind != "notification": + raise ValidationError("unknown intervention action") + action = exact_object(action, keys, "intervention action") + _bounded_text(action["notification_title"], "notification title", 1, 120) + _bounded_text(action["notification_message"], "notification message", 1, 500) + survey_reference = ( + protocol_id(action["survey_id"], "survey ID") if kind == "survey" else None + ) + triggers = _array(root["triggers"], "intervention triggers") + if not triggers: + raise ValidationError("intervention requires a trigger") + trigger_ids: list[str] = [] + occurrences = 0 + study_minutes = duration_hours * 60 + for trigger in triggers: + trigger = exact_object( + trigger, {"id", "schedule", "availability_minutes"}, "trigger" + ) + trigger_ids.append(protocol_id(trigger["id"], "trigger ID")) + _integer(trigger["availability_minutes"], "availability_minutes", 1, 525_600) + schedule = trigger["schedule"] + if not isinstance(schedule, dict): + raise ValidationError("schedule must be an object") + schedule_type = schedule.get("type") + if schedule_type == "one_time": + schedule = exact_object( + schedule, {"type", "offset_minutes", "clock"}, "one-time schedule" + ) + offset = _integer( + schedule["offset_minutes"], "offset_minutes", 0, 2**31 - 1 + ) + _clock(schedule["clock"]) + if offset >= study_minutes: + raise ValidationError("one-time trigger is outside the study") + occurrences += 1 + elif schedule_type == "interval": + schedule = exact_object( + schedule, + {"type", "start_offset_minutes", "interval_minutes", "clock"}, + "interval schedule", + ) + start = _integer( + schedule["start_offset_minutes"], "start_offset_minutes", 0, 2**31 - 1 + ) + interval = _integer( + schedule["interval_minutes"], "interval_minutes", 1, 525_600 + ) + _clock(schedule["clock"]) + if start >= study_minutes: + raise ValidationError("interval trigger is outside the study") + occurrences += (study_minutes - start + interval - 1) // interval + elif schedule_type == "daily_local": + schedule = exact_object( + schedule, {"type", "local_time"}, "daily-local schedule" + ) + if not isinstance(schedule["local_time"], str) or not _LOCAL_TIME.fullmatch( + schedule["local_time"] + ): + raise ValidationError("daily local time is invalid") + occurrences += _maximum_reachable_local_dates(study_minutes) + elif schedule_type == "random_window": + schedule = exact_object( + schedule, + { + "type", + "local_windows", + "occurrences_per_window", + "maximum_occurrences_per_day", + "maximum_occurrences_total", + "minimum_separation_minutes", + }, + "random-window schedule", + ) + windows = _array(schedule["local_windows"], "local windows") + if not 1 <= len(windows) <= 8: + raise ValidationError("random-window count is invalid") + previous_end = None + window_minutes: list[tuple[int, int]] = [] + for window in windows: + window = exact_object( + window, {"start_local_time", "end_local_time"}, "local window" + ) + start = window["start_local_time"] + end = window["end_local_time"] + if ( + not isinstance(start, str) + or not isinstance(end, str) + or not _LOCAL_TIME.fullmatch(start) + or not _LOCAL_TIME.fullmatch(end) + or start >= end + or (previous_end is not None and start < previous_end) + ): + raise ValidationError( + "local windows must be sorted, non-overlapping same-day ranges" + ) + previous_end = end + window_minutes.append((_local_minute(start), _local_minute(end))) + per_window = _integer( + schedule["occurrences_per_window"], "occurrences_per_window", 1, 8 + ) + per_day = _integer( + schedule["maximum_occurrences_per_day"], + "maximum_occurrences_per_day", + 1, + 64, + ) + if per_day > len(windows) * per_window: + raise ValidationError("daily occurrence limit exceeds window capacity") + total = _integer( + schedule["maximum_occurrences_total"], + "maximum_occurrences_total", + 1, + 512, + ) + separation = _integer( + schedule["minimum_separation_minutes"], + "minimum_separation_minutes", + 1, + 1_440, + ) + if any( + end - start < 1 + (per_window - 1) * separation + for start, end in window_minutes + ): + raise ValidationError( + "a random window cannot fit its configured occurrences" + ) + for index, (_, end) in enumerate(window_minutes): + next_start = window_minutes[(index + 1) % len(window_minutes)][0] + if index == len(window_minutes) - 1: + next_start += 1_440 + if next_start - (end - 1) < separation: + raise ValidationError( + "random windows are too close for configured separation" + ) + # Repeated wall-clock edits can expose arbitrarily many local dates inside a short + # monotonic study, so only the signed lifetime cap safely contributes here. + occurrences += total + else: + raise ValidationError("unknown intervention schedule") + _unique(trigger_ids, "trigger ID") + return intervention_id, trigger_ids, occurrences, survey_reference + + +def _localized(value: Any, name: str) -> None: + root = exact_object(value, {"default", "translations"}, name) + _bounded_text(root["default"], f"{name} default", 1, 2_000) + translations = root["translations"] + if not isinstance(translations, dict) or len(translations) > 32: + raise ValidationError(f"{name} translations are invalid") + lowered: set[str] = set() + for language, text in translations.items(): + if not _BCP47.fullmatch(language) or language.lower() in lowered: + raise ValidationError(f"{name} language tag is invalid or duplicated") + lowered.add(language.lower()) + _bounded_text(text, f"{name} translation", 1, 2_000) + + +def _clock(value: Any) -> None: + if value not in {"CALENDAR_TIME", "ACTIVE_RUNNING_TIME"}: + raise ValidationError("relative clock is invalid") + + +def _local_minute(value: str) -> int: + return int(value[:2]) * 60 + int(value[3:]) + + +def _maximum_reachable_local_dates(study_minutes: int) -> int: + """Conservative UTC-18..UTC+18 local-date reach shared by Protocol v1 clients.""" + return (study_minutes + 36 * 60 + 1_439) // 1_440 + 1 + + +def _instant(value: Any, name: str) -> int: + match = _INSTANT.fullmatch(value) if isinstance(value, str) else None + if match is None: + raise ValidationError(f"{name} must be an RFC 3339 UTC instant") + fraction = match.group("fraction") + if fraction is not None and ( + int(fraction) == 0 or (len(fraction) > 3 and fraction.endswith("000")) + ): + raise ValidationError(f"{name} is not a canonical Java Instant") + try: + second = datetime.fromisoformat(match.group("base") + "Z") + except ValueError as error: + raise ValidationError(f"{name} must be an RFC 3339 UTC instant") from error + return timegm(second.timetuple()) * 1_000_000_000 + int( + (fraction or "0").ljust(9, "0") + ) + + +def _bounded_text(value: Any, name: str, minimum: int, maximum: int) -> str: + if not isinstance(value, str): + raise ValidationError(f"{name} must be a string") + length = len(value.encode("utf-16-le")) // 2 + if not minimum <= length <= maximum: + raise ValidationError(f"{name} length is invalid") + return value + + +def _integer(value: Any, name: str, minimum: int, maximum: int) -> int: + if ( + isinstance(value, bool) + or not isinstance(value, int) + or not minimum <= value <= maximum + ): + raise ValidationError(f"{name} is outside bounds") + return value + + +def _array(value: Any, name: str) -> list[Any]: + if not isinstance(value, list): + raise ValidationError(f"{name} must be an array") + return value + + +def _unique(values: list[str], name: str) -> None: + if len(values) != len(set(values)): + raise ValidationError(f"duplicate {name}") diff --git a/adc-analysis/src/adc_analysis/crypto.py b/adc-analysis/src/adc_analysis/crypto.py new file mode 100644 index 0000000..dbb6af6 --- /dev/null +++ b/adc-analysis/src/adc_analysis/crypto.py @@ -0,0 +1,103 @@ +"""Minimal RFC 9180 base-mode receiver for the fixed Protocol v1 suite.""" + +from __future__ import annotations + +import hashlib +import hmac + +from cryptography.exceptions import InvalidTag +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.x25519 import ( + X25519PrivateKey, + X25519PublicKey, +) +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +from .errors import ValidationError + +KEM_ID = 0x0020 +KDF_ID = 0x0001 +AEAD_ID = 0x0002 +_KEM_SUITE = b"KEM" + KEM_ID.to_bytes(2, "big") +_SUITE = ( + b"HPKE" + + KEM_ID.to_bytes(2, "big") + + KDF_ID.to_bytes(2, "big") + + AEAD_ID.to_bytes(2, "big") +) +_VERSION = b"HPKE-v1" + + +def public_key(private_key: bytes) -> bytes: + try: + key = X25519PrivateKey.from_private_bytes(private_key) + except ValueError as error: + raise ValidationError("invalid X25519 private key") from error + return key.public_key().public_bytes( + serialization.Encoding.Raw, serialization.PublicFormat.Raw + ) + + +def open_base(private_key: bytes, wrapped: bytes, info: bytes) -> bytes: + """Open `enc || ciphertext` with empty AAD and sequence number zero.""" + + if len(private_key) != 32 or len(wrapped) != 80: + raise ValidationError("invalid HPKE key or wrapped-key length") + enc, ciphertext = wrapped[:32], wrapped[32:] + try: + recipient = X25519PrivateKey.from_private_bytes(private_key) + ephemeral = X25519PublicKey.from_public_bytes(enc) + dh = recipient.exchange(ephemeral) + recipient_public = public_key(private_key) + shared_secret = _extract_and_expand(dh, enc + recipient_public) + key, nonce = _key_schedule(shared_secret, info) + return AESGCM(key).decrypt(nonce, ciphertext, b"") + except (ValueError, InvalidTag) as error: + raise ValidationError("HPKE authentication failed") from error + + +def _extract_and_expand(dh: bytes, kem_context: bytes) -> bytes: + eae_prk = _labeled_extract(b"", _KEM_SUITE, b"eae_prk", dh) + return _labeled_expand(eae_prk, _KEM_SUITE, b"shared_secret", kem_context, 32) + + +def _key_schedule(shared_secret: bytes, info: bytes) -> tuple[bytes, bytes]: + psk_id_hash = _labeled_extract(b"", _SUITE, b"psk_id_hash", b"") + info_hash = _labeled_extract(b"", _SUITE, b"info_hash", info) + key_schedule_context = b"\x00" + psk_id_hash + info_hash + secret = _labeled_extract(shared_secret, _SUITE, b"secret", b"") + key = _labeled_expand(secret, _SUITE, b"key", key_schedule_context, 32) + nonce = _labeled_expand(secret, _SUITE, b"base_nonce", key_schedule_context, 12) + return key, nonce + + +def _labeled_extract(salt: bytes, suite: bytes, label: bytes, ikm: bytes) -> bytes: + return _hkdf_extract(salt, _VERSION + suite + label + ikm) + + +def _labeled_expand( + prk: bytes, suite: bytes, label: bytes, info: bytes, length: int +) -> bytes: + labeled_info = length.to_bytes(2, "big") + _VERSION + suite + label + info + return _hkdf_expand(prk, labeled_info, length) + + +def _hkdf_extract(salt: bytes, ikm: bytes) -> bytes: + return hmac.new( + salt or bytes(hashlib.sha256().digest_size), ikm, hashlib.sha256 + ).digest() + + +def _hkdf_expand(prk: bytes, info: bytes, length: int) -> bytes: + if length > 255 * hashlib.sha256().digest_size: + raise ValidationError("HPKE expand length is invalid") + output = bytearray() + previous = b"" + counter = 1 + while len(output) < length: + previous = hmac.new( + prk, previous + info + bytes([counter]), hashlib.sha256 + ).digest() + output.extend(previous) + counter += 1 + return bytes(output[:length]) diff --git a/adc-analysis/src/adc_analysis/encoding.py b/adc-analysis/src/adc_analysis/encoding.py new file mode 100644 index 0000000..a216e1e --- /dev/null +++ b/adc-analysis/src/adc_analysis/encoding.py @@ -0,0 +1,61 @@ +"""Strict Protocol v1 text encodings.""" + +from __future__ import annotations + +import base64 +import re +import uuid + +from .errors import ValidationError + +ID = re.compile(r"[a-z0-9][a-z0-9-]{2,63}\Z") +SHA256 = re.compile(r"[0-9a-f]{64}\Z") +_BASE64URL = re.compile(r"[A-Za-z0-9_-]*\Z") + + +def base64url_decode(value: object, length: int, name: str) -> bytes: + if not isinstance(value, str) or "=" in value or not _BASE64URL.fullmatch(value): + raise ValidationError(f"{name} is not unpadded base64url") + try: + decoded = base64.urlsafe_b64decode(value + "=" * (-len(value) % 4)) + except ValueError as error: + raise ValidationError(f"{name} is not unpadded base64url") from error + if len(decoded) != length or base64url_encode(decoded) != value: + raise ValidationError(f"{name} must encode exactly {length} bytes") + return decoded + + +def base64url_encode(value: bytes) -> str: + return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii") + + +def protocol_id(value: object, name: str) -> str: + if not isinstance(value, str) or not ID.fullmatch(value): + raise ValidationError(f"{name} is invalid") + return value + + +def sha256_hex(value: object, name: str) -> str: + if not isinstance(value, str) or not SHA256.fullmatch(value): + raise ValidationError(f"{name} must be lowercase SHA-256") + return value + + +def uuid4_text(value: object, name: str) -> str: + value = uuid_text(value, name) + parsed = uuid.UUID(value) + if parsed.version != 4 or parsed.variant != uuid.RFC_4122: + raise ValidationError(f"{name} must be a lowercase RFC 4122 version-4 UUID") + return value + + +def uuid_text(value: object, name: str) -> str: + if not isinstance(value, str): + raise ValidationError(f"{name} must be a UUID") + try: + parsed = uuid.UUID(value) + except ValueError as error: + raise ValidationError(f"{name} must be a UUID") from error + if str(parsed) != value: + raise ValidationError(f"{name} must be a lowercase UUID") + return value diff --git a/adc-analysis/src/adc_analysis/errors.py b/adc-analysis/src/adc_analysis/errors.py new file mode 100644 index 0000000..e33ac2b --- /dev/null +++ b/adc-analysis/src/adc_analysis/errors.py @@ -0,0 +1,13 @@ +"""Public error types with intentionally non-sensitive messages.""" + + +class AnalysisError(Exception): + """Base class for expected analysis failures.""" + + +class ValidationError(AnalysisError): + """An input failed a Protocol v1 invariant.""" + + +class ConflictError(AnalysisError): + """Authenticated events reuse one identity with different content.""" diff --git a/adc-analysis/src/adc_analysis/event_store.py b/adc-analysis/src/adc_analysis/event_store.py new file mode 100644 index 0000000..ba9e38b --- /dev/null +++ b/adc-analysis/src/adc_analysis/event_store.py @@ -0,0 +1,284 @@ +"""Owner-only SQLite spill storage for authenticated event reassembly.""" + +from __future__ import annotations + +import hashlib +import json +import os +import sqlite3 +import tempfile +from collections.abc import Iterator, Mapping +from pathlib import Path + +from .errors import ValidationError +from .filesystem import private_directory +from .models import ( + BootSession, + EventProvenance, + SamplingGroup, + SurveyLifecycleCount, + VerifiedEvent, +) + +EVENT_COLUMNS = ( + "experiment_id, configuration_id, participant_instance_id, sequence_number, " + "assigned_participant_id, collector_id, payload_schema_version, payload_type, " + "boot_session_id, monotonic_time_nanos, wall_time_utc_millis, fields_json, " + "canonical_bytes, source_ciphertext_sha256, source_bundle_id, " + "source_configuration_sha256, source_object, content_sha256" +) +EVENT_PLACEHOLDERS = ",".join("?" for _ in range(18)) + + +class EventDatabase: + def __init__(self, directory: Path): + directory = private_directory(directory) + fd, name = tempfile.mkstemp(prefix="adc-reassembly-", suffix=".sqlite3", dir=directory) + os.close(fd) + self.path = Path(name) + os.chmod(self.path, 0o600) + self.connection = sqlite3.connect(self.path) + self.connection.execute("PRAGMA trusted_schema = OFF") + self.connection.execute("PRAGMA synchronous = FULL") + self.connection.execute("PRAGMA journal_mode = DELETE") + self.connection.execute( + f"CREATE TABLE candidates ({_column_definitions()})" + ) + + def add(self, event: VerifiedEvent) -> None: + self.connection.execute( + f"INSERT INTO candidates ({EVENT_COLUMNS}) VALUES ({EVENT_PLACEHOLDERS})", + _event_row(event), + ) + + def finish_candidates(self) -> None: + self.connection.commit() + self.connection.execute( + "CREATE INDEX candidate_identity ON candidates " + "(experiment_id, configuration_id, participant_instance_id, " + "sequence_number, content_sha256, source_ciphertext_sha256, " + "source_object, source_bundle_id)" + ) + self.connection.execute(f"CREATE TABLE accepted ({_column_definitions()})") + self.connection.commit() + + def candidate_rows(self): + return self.connection.execute( + f"SELECT {EVENT_COLUMNS} FROM candidates ORDER BY " + "experiment_id, configuration_id, participant_instance_id, " + "sequence_number, content_sha256, canonical_bytes, " + "source_ciphertext_sha256, " + "source_object, source_bundle_id" + ) + + def accept(self, row: tuple) -> None: + self.connection.execute( + f"INSERT INTO accepted ({EVENT_COLUMNS}) VALUES ({EVENT_PLACEHOLDERS})", + row, + ) + + def seal(self) -> DiskEventCollection: + self.connection.commit() + self.connection.execute( + "CREATE INDEX accepted_identity ON accepted " + "(experiment_id, configuration_id, participant_instance_id, sequence_number)" + ) + self.connection.execute( + "CREATE INDEX accepted_partition ON accepted " + "(experiment_id, configuration_id, collector_id, " + "payload_schema_version, payload_type, participant_instance_id, " + "sequence_number)" + ) + count = self.connection.execute("SELECT COUNT(*) FROM accepted").fetchone()[0] + self.connection.commit() + self.connection.close() + return DiskEventCollection(self.path, count) + + def abort(self) -> None: + self.connection.close() + self.path.unlink(missing_ok=True) + + +class DiskEventCollection: + """Repeatable, query-ordered event iterable backed by a private database.""" + + def __init__(self, path: Path, count: int): + self.path = path + self.count = count + self.closed = False + + def __len__(self) -> int: + return self.count + + def __iter__(self) -> Iterator[VerifiedEvent]: + yield from self._query( + "experiment_id, configuration_id, participant_instance_id, sequence_number" + ) + + def iter_partitioned(self) -> Iterator[VerifiedEvent]: + yield from self._query( + "experiment_id, configuration_id, collector_id, " + "payload_schema_version, payload_type, participant_instance_id, " + "sequence_number" + ) + + def iter_boot_sessions(self) -> Iterator[BootSession]: + query = ( + "SELECT experiment_id, configuration_id, participant_instance_id, " + "boot_session_id FROM accepted GROUP BY experiment_id, configuration_id, " + "participant_instance_id, boot_session_id ORDER BY experiment_id, " + "configuration_id, participant_instance_id, boot_session_id" + ) + for row in self._raw_query(query): + yield BootSession(*row) + + def iter_sampling_groups( + self, + source_clock_fields: Mapping[tuple[str, int, str], str], + ) -> Iterator[SamplingGroup]: + query = ( + "SELECT experiment_id, configuration_id, participant_instance_id, " + "collector_id, boot_session_id, payload_schema_version, payload_type, " + "fields_json FROM accepted ORDER BY experiment_id, configuration_id, " + "participant_instance_id, collector_id, boot_session_id, sequence_number" + ) + current_key: tuple[str, str, str, str, str] | None = None + current_field = "" + first = 0 + last = 0 + count = 0 + for row in self._raw_query(query): + field = source_clock_fields.get((row[3], row[5], row[6])) + if field is None: + continue + fields = json.loads(row[7]) + timestamp = fields.get(field) + if isinstance(timestamp, bool) or not isinstance(timestamp, int): + raise ValidationError("sampling source clock is not an integer") + key = row[:5] + if current_key is not None and key != current_key: + yield SamplingGroup(*current_key, current_field, first, last, count) + count = 0 + if count == 0: + current_key = key + current_field = field + first = timestamp + last = timestamp + else: + if field != current_field: + raise ValidationError( + "collector uses inconsistent sampling source clocks" + ) + first = min(first, timestamp) + last = max(last, timestamp) + count += 1 + if current_key is not None: + yield SamplingGroup(*current_key, current_field, first, last, count) + + def iter_survey_lifecycle_counts(self) -> Iterator[SurveyLifecycleCount]: + query = ( + "SELECT experiment_id, configuration_id, participant_instance_id, " + "payload_type, COUNT(*) FROM accepted WHERE collector_id = " + "'interventions.v1' AND payload_type LIKE 'SURVEY_%' GROUP BY " + "experiment_id, configuration_id, participant_instance_id, payload_type " + "ORDER BY experiment_id, configuration_id, participant_instance_id, " + "payload_type" + ) + for row in self._raw_query(query): + yield SurveyLifecycleCount(*row) + + def close(self) -> None: + if not self.closed: + self.path.unlink(missing_ok=True) + self.closed = True + + def __del__(self) -> None: + try: + self.close() + except OSError: + pass + + def _query(self, ordering: str) -> Iterator[VerifiedEvent]: + if self.closed: + raise ValidationError("reassembled event store is closed") + connection = sqlite3.connect(f"file:{self.path}?mode=ro", uri=True) + try: + cursor = connection.execute( + f"SELECT {EVENT_COLUMNS} FROM accepted ORDER BY {ordering}" + ) + for row in cursor: + yield _row_event(row) + finally: + connection.close() + + def _raw_query(self, query: str): + if self.closed: + raise ValidationError("reassembled event store is closed") + connection = sqlite3.connect(f"file:{self.path}?mode=ro", uri=True) + try: + yield from connection.execute(query) + finally: + connection.close() + + +def _column_definitions() -> str: + return ( + "experiment_id TEXT NOT NULL, configuration_id TEXT NOT NULL, " + "participant_instance_id TEXT NOT NULL, sequence_number INTEGER NOT NULL, " + "assigned_participant_id TEXT, collector_id TEXT NOT NULL, " + "payload_schema_version INTEGER NOT NULL, payload_type TEXT NOT NULL, " + "boot_session_id TEXT NOT NULL, monotonic_time_nanos INTEGER NOT NULL, " + "wall_time_utc_millis INTEGER NOT NULL, fields_json TEXT NOT NULL, " + "canonical_bytes BLOB NOT NULL, source_ciphertext_sha256 TEXT NOT NULL, " + "source_bundle_id TEXT NOT NULL, source_configuration_sha256 TEXT NOT NULL, " + "source_object TEXT NOT NULL, content_sha256 TEXT NOT NULL" + ) + + +def _event_row(event: VerifiedEvent) -> tuple: + fields = json.dumps( + event.fields, + ensure_ascii=False, + allow_nan=False, + sort_keys=True, + separators=(",", ":"), + ) + return ( + event.experiment_id, + event.configuration_id, + event.participant_instance_id, + event.sequence_number, + event.assigned_participant_id, + event.collector_id, + event.payload_schema_version, + event.payload_type, + event.boot_session_id, + event.monotonic_time_nanos, + event.wall_time_utc_millis, + fields, + event.canonical_bytes, + event.provenance.source_ciphertext_sha256, + event.provenance.source_bundle_id, + event.provenance.source_configuration_sha256, + event.provenance.source_object, + hashlib.sha256(event.canonical_bytes).hexdigest(), + ) + + +def _row_event(row: tuple) -> VerifiedEvent: + return VerifiedEvent( + row[0], + row[1], + row[2], + row[4], + row[3], + row[5], + row[6], + row[7], + row[8], + row[9], + row[10], + json.loads(row[11]), + bytes(row[12]), + EventProvenance(row[13], row[14], row[15], row[16]), + ) diff --git a/adc-analysis/src/adc_analysis/filesystem.py b/adc-analysis/src/adc_analysis/filesystem.py new file mode 100644 index 0000000..234954e --- /dev/null +++ b/adc-analysis/src/adc_analysis/filesystem.py @@ -0,0 +1,68 @@ +"""Small fail-closed filesystem primitives for plaintext artifacts.""" + +from __future__ import annotations + +import ctypes +import errno +import os +import stat +import sys +from pathlib import Path + +from .errors import ValidationError + + +def private_directory(path: Path) -> Path: + """Create or tighten a non-symlink directory to owner-only access.""" + + path = Path(path).absolute() + path.mkdir(parents=True, exist_ok=True, mode=0o700) + if path.is_symlink() or not path.is_dir(): + raise ValidationError("private staging path must be a real directory") + if os.name == "posix": + os.chmod(path, 0o700) + mode = stat.S_IMODE(path.stat(follow_symlinks=False).st_mode) + if mode != 0o700: + raise ValidationError("private staging directory permissions are unsafe") + return path.resolve() + + +def rename_noreplace(source: Path, destination: Path) -> None: + """Atomically publish a directory without replacing a concurrent destination.""" + + source_bytes = os.fsencode(source) + destination_bytes = os.fsencode(destination) + if sys.platform == "darwin": + libc = ctypes.CDLL(None, use_errno=True) + renamex_np = libc.renamex_np + renamex_np.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint] + renamex_np.restype = ctypes.c_int + result = renamex_np(source_bytes, destination_bytes, 0x00000004) + elif sys.platform.startswith("linux"): + libc = ctypes.CDLL(None, use_errno=True) + renameat2 = getattr(libc, "renameat2", None) + if renameat2 is None: + raise ValidationError("atomic create-only publication is unsupported") + renameat2.argtypes = [ + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ] + renameat2.restype = ctypes.c_int + result = renameat2(-100, source_bytes, -100, destination_bytes, 1) + elif os.name == "nt": + try: + os.rename(source, destination) + return + except FileExistsError as error: + raise ValidationError("dataset destination already exists") from error + else: + raise ValidationError("atomic create-only publication is unsupported") + if result == 0: + return + error_number = ctypes.get_errno() + if error_number in {errno.EEXIST, errno.ENOTEMPTY}: + raise ValidationError("dataset destination already exists") + raise OSError(error_number, os.strerror(error_number), str(destination)) diff --git a/adc-analysis/src/adc_analysis/inventory.py b/adc-analysis/src/adc_analysis/inventory.py new file mode 100644 index 0000000..62f1ccd --- /dev/null +++ b/adc-analysis/src/adc_analysis/inventory.py @@ -0,0 +1,263 @@ +"""Immutable, content-addressed ciphertext inventory.""" + +from __future__ import annotations + +import hashlib +import os +import re +import tempfile +from collections.abc import Iterable +from datetime import datetime +from pathlib import Path + +from .errors import ValidationError +from .jcs import canonical_decimal, canonicalize, parse +from .limits import AUTOMATIC_UPLOAD_MAX_BYTES, MANUAL_EXPORT_MAX_BYTES +from .models import InventoryObject +from .sources import BundleSource + +METADATA_KEYS = { + "sha256", + "byte_count", + "configuration_sha256", + "researcher_key_id", + "first_sequence_number", + "last_sequence_number", + "event_count", + "received_at_utc", +} +_HEX = re.compile(r"[0-9a-f]{64}\Z") +_RECEIVE_TIME = re.compile( + r"[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}Z\Z" +) + + +class CiphertextInventory: + def __init__(self, workspace: Path): + self.workspace = Path(workspace).resolve() + self.cache = self.workspace / "cache" / "objects" + self.manifest = self.workspace / "inventory.json" + + def ingest(self, sources: Iterable[BundleSource]) -> tuple[InventoryObject, ...]: + self.cache.mkdir(parents=True, exist_ok=True) + if self.cache.resolve() != self.cache: + raise ValidationError("ciphertext cache must not traverse symbolic links") + objects: list[InventoryObject] = [] + for source in sources: + for item in source.objects(): + objects.append(self._ingest_object(item)) + objects.sort(key=lambda item: (item.sha256, item.source_uri)) + document = { + "format": "adc-ciphertext-inventory-v1", + "objects": [ + { + "byte_count": str(item.byte_count), + "cache_path": str(item.cache_path.relative_to(self.workspace)), + "metadata": dict(sorted(item.metadata.items())) + if item.metadata is not None + else None, + "sha256": item.sha256, + "source_kind": item.source_kind, + "source_uri": item.source_uri, + } + for item in objects + ], + } + _atomic_write(self.manifest, canonicalize(document), 0o600) + return tuple(objects) + + def load(self) -> tuple[InventoryObject, ...]: + try: + document = parse(self.manifest.read_bytes()) + except (OSError, ValidationError) as error: + raise ValidationError("inventory manifest is unreadable") from error + if not isinstance(document, dict) or set(document) != {"format", "objects"}: + raise ValidationError("inventory manifest keys mismatch") + if document["format"] != "adc-ciphertext-inventory-v1" or not isinstance( + document["objects"], list + ): + raise ValidationError("unsupported inventory manifest") + result = [] + for raw in document["objects"]: + if not isinstance(raw, dict) or set(raw) != { + "byte_count", + "cache_path", + "metadata", + "sha256", + "source_kind", + "source_uri", + }: + raise ValidationError("inventory object keys mismatch") + digest = raw["sha256"] + if not isinstance(digest, str) or not _HEX.fullmatch(digest): + raise ValidationError("inventory digest is invalid") + byte_count = canonical_decimal( + raw["byte_count"], + "inventory byte_count", + maximum=MANUAL_EXPORT_MAX_BYTES, + ) + if raw["source_kind"] not in {"local", "receiver"}: + raise ValidationError("inventory source kind is invalid") + if not isinstance(raw["source_uri"], str) or not raw["source_uri"]: + raise ValidationError("inventory source URI is invalid") + expected_relative = ( + Path("cache") / "objects" / digest[:2] / f"{digest}.adcexp" + ) + if raw["cache_path"] != str(expected_relative): + raise ValidationError("inventory cache path is invalid") + candidate = self.workspace / expected_relative + cache_path = candidate.resolve() + if cache_path != candidate or not cache_path.is_file(): + raise ValidationError("inventory cache object is missing") + metadata = raw["metadata"] + if raw["source_kind"] == "receiver": + _validate_metadata(metadata, digest, byte_count) + elif metadata is not None: + raise ValidationError("local inventory objects cannot have metadata") + if cache_path.stat().st_size != byte_count or _sha256(cache_path) != digest: + raise ValidationError("cached ciphertext does not match inventory") + result.append( + InventoryObject( + raw["source_uri"], + digest, + byte_count, + cache_path, + metadata, + raw["source_kind"], + ) + ) + return tuple(result) + + def _ingest_object(self, item) -> InventoryObject: + if item.source_kind not in {"local", "receiver"}: + raise ValidationError("source object kind is invalid") + maximum = ( + AUTOMATIC_UPLOAD_MAX_BYTES + if item.source_kind == "receiver" + else MANUAL_EXPORT_MAX_BYTES + ) + if item.size < 1 or item.size > maximum: + raise ValidationError( + f"source object size outside protocol bound: {item.source_uri}" + ) + temporary = None + digest = hashlib.sha256() + count = 0 + try: + fd, temporary_name = tempfile.mkstemp(prefix=".adc-object-", dir=self.cache) + temporary = Path(temporary_name) + os.fchmod(fd, 0o600) + with os.fdopen(fd, "wb") as output, item.open() as source: + while chunk := source.read(1024 * 1024): + count += len(chunk) + if count > maximum: + raise ValidationError("source object exceeds its source bound") + digest.update(chunk) + output.write(chunk) + output.flush() + os.fsync(output.fileno()) + if count != item.size: + raise ValidationError("source object changed while reading") + sha256 = digest.hexdigest() + if item.source_kind == "receiver": + _validate_metadata(item.metadata, sha256, count) + elif item.metadata is not None: + raise ValidationError("local source objects cannot have metadata") + destination = self.cache / sha256[:2] / f"{sha256}.adcexp" + destination.parent.mkdir(parents=True, exist_ok=True) + if destination.parent.resolve() != destination.parent: + raise ValidationError( + "ciphertext cache must not traverse symbolic links" + ) + if destination.exists(): + if ( + destination.stat().st_size != count + or _sha256(destination) != sha256 + ): + raise ValidationError("content-addressed cache collision") + else: + try: + os.link(temporary, destination) + except FileExistsError: + if ( + destination.stat().st_size != count + or _sha256(destination) != sha256 + ): + raise ValidationError("content-addressed cache collision") + return InventoryObject( + item.source_uri, + sha256, + count, + destination, + item.metadata, + item.source_kind, + ) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + + +def _validate_metadata(metadata: object, digest: str, byte_count: int) -> None: + if not isinstance(metadata, dict) or set(metadata) != METADATA_KEYS: + raise ValidationError( + "receiver metadata must have the exact Protocol v1 key set" + ) + if any(not isinstance(value, str) for value in metadata.values()): + raise ValidationError("receiver metadata values must be strings") + if metadata["sha256"] != digest: + raise ValidationError("receiver metadata digest mismatch") + if ( + canonical_decimal( + metadata["byte_count"], + "metadata byte_count", + maximum=AUTOMATIC_UPLOAD_MAX_BYTES, + ) + != byte_count + ): + raise ValidationError("receiver metadata byte count mismatch") + if not _HEX.fullmatch(metadata["configuration_sha256"]): + raise ValidationError("receiver metadata configuration digest is invalid") + first = canonical_decimal( + metadata["first_sequence_number"], "metadata first_sequence_number" + ) + last = canonical_decimal( + metadata["last_sequence_number"], "metadata last_sequence_number" + ) + count = canonical_decimal(metadata["event_count"], "metadata event_count") + if first < 1 or last != (first - 1 if count == 0 else first + count - 1): + raise ValidationError("receiver metadata sequence range is inconsistent") + key_id = metadata["researcher_key_id"] + if not isinstance(key_id, str) or not re.fullmatch( + r"[a-z0-9][a-z0-9-]{2,63}", key_id + ): + raise ValidationError("receiver metadata researcher key ID is invalid") + received = metadata["received_at_utc"] + if not _RECEIVE_TIME.fullmatch(received): + raise ValidationError("receiver metadata receive time is invalid") + try: + datetime.fromisoformat(received) + except ValueError as error: + raise ValidationError("receiver metadata receive time is invalid") from error + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + while chunk := stream.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def _atomic_write(path: Path, data: bytes, mode: int) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + fd, name = tempfile.mkstemp(prefix=f".{path.name}-", dir=path.parent) + temporary = Path(name) + try: + os.fchmod(fd, mode) + with os.fdopen(fd, "wb") as stream: + stream.write(data) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) diff --git a/adc-analysis/src/adc_analysis/jcs.py b/adc-analysis/src/adc_analysis/jcs.py new file mode 100644 index 0000000..f139494 --- /dev/null +++ b/adc-analysis/src/adc_analysis/jcs.py @@ -0,0 +1,141 @@ +"""The bounded RFC 8785 subset used by Protocol v1. + +Protocol JSON never contains floating-point JSON numbers. Sensor floats are strings, +so rejecting every JSON float gives a considerably smaller and safer implementation. +""" + +from __future__ import annotations + +import json +from typing import Any + +from .errors import ValidationError + + +def _reject_constant(value: str) -> None: + raise ValidationError(f"non-finite JSON value: {value}") + + +def _reject_float(value: str) -> None: + raise ValidationError(f"non-integral JSON number: {value}") + + +def _pairs(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValidationError(f"duplicate JSON member: {key}") + result[key] = value + return result + + +def parse(data: bytes, *, require_canonical: bool = True) -> Any: + """Parse one strict UTF-8 JSON value and optionally require exact JCS bytes.""" + + try: + text = data.decode("utf-8", errors="strict") + value = json.loads( + text, + object_pairs_hook=_pairs, + parse_float=_reject_float, + parse_constant=_reject_constant, + ) + except ( + UnicodeDecodeError, + json.JSONDecodeError, + TypeError, + ValueError, + OverflowError, + RecursionError, + ) as error: + if isinstance(error, ValidationError): + raise + raise ValidationError("malformed JSON") from error + if require_canonical and canonicalize(value) != data: + raise ValidationError("JSON is not canonical JCS") + return value + + +def parse_embedded_json(text: str) -> Any: + """Parse a strict payload field which is JSON text but is not required to be JCS.""" + + try: + return json.loads( + text, + object_pairs_hook=_pairs, + parse_constant=_reject_constant, + ) + except ( + json.JSONDecodeError, + TypeError, + ValueError, + OverflowError, + RecursionError, + ) as error: + if isinstance(error, ValidationError): + raise + raise ValidationError("malformed embedded JSON") from error + + +def canonicalize(value: Any) -> bytes: + """Encode the Protocol v1 integral-only JCS subset.""" + + return _encode(value).encode("utf-8") + + +def _encode(value: Any) -> str: + if value is None: + return "null" + if value is True: + return "true" + if value is False: + return "false" + if isinstance(value, int): + return str(value) + if isinstance(value, float): + raise ValidationError("floating-point JSON numbers are forbidden") + if isinstance(value, str): + _validate_unicode(value) + return json.dumps(value, ensure_ascii=False, separators=(",", ":")) + if isinstance(value, list): + return "[" + ",".join(_encode(item) for item in value) + "]" + if isinstance(value, dict): + if not all(isinstance(key, str) for key in value): + raise ValidationError("JSON object keys must be strings") + keys = sorted(value, key=lambda key: key.encode("utf-16-be")) + return ( + "{" + + ",".join(_encode(key) + ":" + _encode(value[key]) for key in keys) + + "}" + ) + raise ValidationError(f"unsupported JSON type: {type(value).__name__}") + + +def _validate_unicode(value: str) -> None: + try: + value.encode("utf-8", errors="strict") + value.encode("utf-16-be", errors="strict") + except UnicodeEncodeError as error: + raise ValidationError("JSON contains an unpaired surrogate") from error + + +def exact_object(value: Any, keys: set[str], name: str) -> dict[str, Any]: + if not isinstance(value, dict) or set(value) != keys: + actual = sorted(value) if isinstance(value, dict) else type(value).__name__ + raise ValidationError(f"{name} keys mismatch: {actual}") + return value + + +def canonical_decimal(value: Any, name: str, *, maximum: int = 2**63 - 1) -> int: + if ( + not isinstance(value, str) + or not value + or (value != "0" and (value[0] == "0" or not value.isascii())) + ): + raise ValidationError(f"{name} must be a canonical unsigned decimal string") + if not value.isdigit(): + raise ValidationError(f"{name} must be a canonical unsigned decimal string") + number = int(value) + if number > maximum: + raise ValidationError(f"{name} is outside the supported range") + return number diff --git a/adc-analysis/src/adc_analysis/limits.py b/adc-analysis/src/adc_analysis/limits.py new file mode 100644 index 0000000..c7d7758 --- /dev/null +++ b/adc-analysis/src/adc_analysis/limits.py @@ -0,0 +1,7 @@ +"""Protocol v1 resource limits shared by inventory and bundle verification.""" + +AUTOMATIC_UPLOAD_MAX_BYTES = 32 * 1024 * 1024 +MANUAL_EXPORT_MAX_BYTES = 8 * 1024 * 1024 * 1024 +SIGNED_CONFIGURATION_MAX_BYTES = 1024 * 1024 +JSON_STRING_TOKEN_MAX_BYTES = SIGNED_CONFIGURATION_MAX_BYTES +JSON_MAX_DEPTH = 64 diff --git a/adc-analysis/src/adc_analysis/models.py b/adc-analysis/src/adc_analysis/models.py new file mode 100644 index 0000000..1ea5861 --- /dev/null +++ b/adc-analysis/src/adc_analysis/models.py @@ -0,0 +1,136 @@ +"""Small immutable values shared by pipeline stages.""" + +from __future__ import annotations + +from collections.abc import Iterator, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Protocol + + +class VerifiedEvents(Protocol): + def __iter__(self) -> Iterator[VerifiedEvent]: ... + + def __len__(self) -> int: ... + + def close(self) -> None: ... + + +class PartitionedVerifiedEvents(VerifiedEvents, Protocol): + def iter_partitioned(self) -> Iterator[VerifiedEvent]: ... + + def iter_boot_sessions(self) -> Iterator[BootSession]: ... + + def iter_sampling_groups( + self, + source_clock_fields: Mapping[tuple[str, int, str], str], + ) -> Iterator[SamplingGroup]: ... + + def iter_survey_lifecycle_counts(self) -> Iterator[SurveyLifecycleCount]: ... + + +@dataclass(frozen=True, slots=True) +class SourceObject: + source_uri: str + size: int + metadata: Mapping[str, str] | None + opener: Any + source_kind: str = "local" + + def open(self): + return self.opener() + + +@dataclass(frozen=True, slots=True) +class InventoryObject: + source_uri: str + sha256: str + byte_count: int + cache_path: Path + metadata: Mapping[str, str] | None + source_kind: str = "local" + + +@dataclass(frozen=True, slots=True) +class EventProvenance: + source_ciphertext_sha256: str + source_bundle_id: str + source_configuration_sha256: str + source_object: str + + +@dataclass(frozen=True, slots=True) +class BootSession: + experiment_id: str + configuration_id: str + participant_instance_id: str + boot_session_id: str + + +@dataclass(frozen=True, slots=True) +class SamplingGroup: + experiment_id: str + configuration_id: str + participant_instance_id: str + collector_id: str + boot_session_id: str + source_clock_field: str + first_monotonic_time_nanos: int + last_monotonic_time_nanos: int + event_count: int + + +@dataclass(frozen=True, slots=True) +class SurveyLifecycleCount: + experiment_id: str + configuration_id: str + participant_instance_id: str + payload_type: str + event_count: int + + +@dataclass(frozen=True, slots=True) +class VerifiedEvent: + experiment_id: str + configuration_id: str + participant_instance_id: str + assigned_participant_id: str | None + sequence_number: int + collector_id: str + payload_schema_version: int + payload_type: str + boot_session_id: str + monotonic_time_nanos: int + wall_time_utc_millis: int + fields: Mapping[str, Any] + canonical_bytes: bytes + provenance: EventProvenance + + @property + def identity(self) -> tuple[str, str, str, int]: + return ( + self.experiment_id, + self.configuration_id, + self.participant_instance_id, + self.sequence_number, + ) + + +@dataclass(frozen=True, slots=True) +class VerifiedBundle: + bundle_id: str + bundle_kind: str + configuration_sha256: str + experiment_id: str + configuration_id: str + participant_instance_id: str + exported_at_utc_millis: int + first_sequence_number: int + last_sequence_number: int + event_count: int + retained_from_sequence: int + uploaded_through_sequence: int + durable_through_sequence: int + next_sequence_number: int + events: VerifiedEvents + source: InventoryObject diff --git a/adc-analysis/src/adc_analysis/pipeline.py b/adc-analysis/src/adc_analysis/pipeline.py new file mode 100644 index 0000000..2716da2 --- /dev/null +++ b/adc-analysis/src/adc_analysis/pipeline.py @@ -0,0 +1,156 @@ +"""One-way orchestration from immutable inventory to typed Parquet.""" + +from __future__ import annotations + +import hashlib +import os +import shutil +import tempfile +from collections.abc import Mapping +from pathlib import Path + +from .bundle import BundleVerifier +from .catalog import CollectorCatalog +from .encoding import base64url_decode, protocol_id +from .errors import ValidationError +from .inventory import CiphertextInventory +from .jcs import canonicalize, exact_object, parse +from .reassembly import Reassembler +from .sink import DatasetSink + + +class AnalysisPipeline: + def __init__( + self, + workspace: Path, + catalog: CollectorCatalog, + researcher_private_keys: Mapping[str, bytes], + sink: DatasetSink, + ): + self.workspace = Path(workspace).resolve() + self.inventory = CiphertextInventory(self.workspace) + self.verifier = BundleVerifier( + catalog, researcher_private_keys, self.workspace / "staging" / "plaintext" + ) + self.sink = sink + + def materialize(self, destination: Path) -> Path: + bundles = [] + result = None + failures: list[dict[str, str]] = [] + try: + for source in self.inventory.load(): + try: + bundles.append(self.verifier.verify(source)) + except ValidationError as error: + failures.append(self._quarantine(source, str(error))) + if not bundles: + self._write_report( + { + "format": "adc-validation-report-v1", + "validation_failures": failures, + } + ) + raise ValidationError("no valid bundles remain after verification") + result = Reassembler( + self.workspace / "staging" / "reassembly" + ).reassemble(bundles) + self._write_report( + { + "format": "adc-validation-report-v1", + "quality": result.quality, + "validation_failures": failures, + } + ) + return self.sink.write( + result, destination, validation_failures=tuple(failures) + ) + finally: + for bundle in bundles: + bundle.events.close() + if result is not None: + result.events.close() + + def _quarantine(self, source, reason: str) -> dict[str, str]: + directory = self.workspace / "quarantine" / source.sha256[:2] + directory.mkdir(parents=True, exist_ok=True) + ciphertext = directory / f"{source.sha256}.adcexp" + if ciphertext.exists(): + if ( + ciphertext.stat().st_size != source.byte_count + or _sha256(ciphertext) != source.sha256 + ): + raise ValidationError("quarantine ciphertext collision") + else: + fd, name = tempfile.mkstemp(prefix=".quarantine-", dir=directory) + temporary = Path(name) + try: + os.fchmod(fd, 0o600) + with ( + os.fdopen(fd, "wb") as output, + source.cache_path.open("rb") as input_stream, + ): + shutil.copyfileobj(input_stream, output, 1024 * 1024) + output.flush() + os.fsync(output.fileno()) + if _sha256(temporary) != source.sha256: + raise ValidationError("quarantine copy digest mismatch") + os.replace(temporary, ciphertext) + finally: + temporary.unlink(missing_ok=True) + record = { + "ciphertext": str(ciphertext.relative_to(self.workspace)), + "reason": reason, + "sha256": source.sha256, + "source_object": source.source_uri, + } + _atomic_write(directory / f"{source.sha256}.reason.json", canonicalize(record)) + return record + + def _write_report(self, document: dict) -> None: + _atomic_write( + self.workspace / "reports" / "validation-report.json", + canonicalize(document), + ) + + +def load_private_keys(path: Path) -> dict[str, bytes]: + path = Path(path).resolve() + if os.name == "posix" and path.stat().st_mode & 0o077: + raise ValidationError( + "researcher key file must not be group- or world-accessible" + ) + root = parse(path.read_bytes(), require_canonical=False) + exact_object(root, {"format", "keys"}, "researcher key file") + if root["format"] != "adc-analysis-keys-v1" or not isinstance(root["keys"], dict): + raise ValidationError("unsupported researcher key file") + keys = {} + for key_id, encoded in root["keys"].items(): + protocol_id(key_id, "researcher key ID") + keys[key_id] = base64url_decode(encoded, 32, f"private key {key_id}") + if not keys: + raise ValidationError("researcher key file is empty") + return keys + + +def _atomic_write(path: Path, data: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + fd, name = tempfile.mkstemp(prefix=f".{path.name}-", dir=path.parent) + temporary = Path(name) + try: + os.fchmod(fd, 0o600) + with os.fdopen(fd, "wb") as stream: + stream.write(data) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + while chunk := stream.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() diff --git a/adc-analysis/src/adc_analysis/py.typed b/adc-analysis/src/adc_analysis/py.typed new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/adc-analysis/src/adc_analysis/py.typed @@ -0,0 +1 @@ + diff --git a/adc-analysis/src/adc_analysis/reassembly.py b/adc-analysis/src/adc_analysis/reassembly.py new file mode 100644 index 0000000..bc80214 --- /dev/null +++ b/adc-analysis/src/adc_analysis/reassembly.py @@ -0,0 +1,300 @@ +"""Deterministic event reassembly with explicit data-quality states.""" + +from __future__ import annotations + +import heapq +from collections.abc import Iterable +from dataclasses import dataclass +from itertools import groupby, islice +from pathlib import Path +from typing import Any + +from .event_store import DiskEventCollection, EventDatabase +from .models import PartitionedVerifiedEvents, VerifiedBundle, VerifiedEvent +from .summary import SUMMARY_EXAMPLE_LIMIT, BoundedExamples + + +@dataclass(frozen=True, slots=True) +class ReassemblyResult: + bundles: tuple[VerifiedBundle, ...] + events: PartitionedVerifiedEvents + quality: dict[str, Any] + has_conflicts: bool + + +class Reassembler: + def __init__(self, staging_directory: Path): + self.staging_directory = staging_directory + + def reassemble(self, bundles: Iterable[VerifiedBundle]) -> ReassemblyResult: + ordered = tuple( + sorted( + bundles, + key=lambda value: ( + value.experiment_id, + value.configuration_id, + value.participant_instance_id, + value.first_sequence_number, + value.bundle_id, + value.source.sha256, + value.source.source_uri, + ), + ) + ) + return self._reassemble(ordered) + + def _reassemble(self, ordered: tuple[VerifiedBundle, ...]) -> ReassemblyResult: + database = EventDatabase(self.staging_directory) + collection: DiskEventCollection | None = None + try: + for bundle in ordered: + for event in bundle.events: + database.add(event) + database.finish_candidates() + identical_duplicates = BoundedExamples[dict[str, Any]]() + event_conflicts = BoundedExamples[dict[str, Any]]() + current_identity = None + first_row = None + previous_canonical: bytes | None = None + copies = 0 + content_variants = 0 + content_hash_examples: list[str] = [] + source_bundle_id_examples: list[str] = [] + sampled_source_bundle_ids: set[str] = set() + + def finish_identity() -> None: + if current_identity is None or first_row is None: + return + common = { + "configuration_id": current_identity[1], + "experiment_id": current_identity[0], + "participant_instance_id": current_identity[2], + "sequence_number": str(current_identity[3]), + "source_bundle_id_examples": source_bundle_id_examples, + "source_bundle_id_examples_truncated": ( + copies > len(source_bundle_id_examples) + ), + "source_copy_count": str(copies), + } + if content_variants > 1: + event_conflicts.add( + { + **common, + "content_sha256_examples": content_hash_examples, + "content_sha256_examples_truncated": ( + content_variants > len(content_hash_examples) + ), + "content_variant_count": str(content_variants), + } + ) + return + database.accept(first_row) + if copies > 1: + identical_duplicates.add(common) + + for row in database.candidate_rows(): + identity = row[:4] + if identity != current_identity: + finish_identity() + current_identity = identity + first_row = row + previous_canonical = None + copies = 0 + content_variants = 0 + content_hash_examples = [] + source_bundle_id_examples = [] + sampled_source_bundle_ids = set() + copies += 1 + canonical = bytes(row[12]) + if canonical != previous_canonical: + content_variants += 1 + previous_canonical = canonical + if len(content_hash_examples) < SUMMARY_EXAMPLE_LIMIT: + content_hash_examples.append(row[17]) + source_bundle_id = row[14] + if ( + source_bundle_id not in sampled_source_bundle_ids + and len(source_bundle_id_examples) < SUMMARY_EXAMPLE_LIMIT + ): + sampled_source_bundle_ids.add(source_bundle_id) + source_bundle_id_examples.append(source_bundle_id) + finish_identity() + collection = database.seal() + bundle_identity_conflicts = _bundle_identity_conflicts(ordered) + quality = { + "bundle_identity_conflicts": bundle_identity_conflicts.document(), + "event_conflicts": event_conflicts.document(), + "format": "adc-quality-summary-v1", + "identical_event_duplicates": identical_duplicates.document(), + "participant_coverage": _coverage_stream(ordered, collection), + "range_overlaps": _bundle_overlaps(ordered), + } + return ReassemblyResult( + ordered, + collection, + quality, + bool(event_conflicts.count or bundle_identity_conflicts.count), + ) + except Exception: + if collection is None: + database.abort() + else: + collection.close() + raise + + +def _bundle_identity_conflicts( + bundles: tuple[VerifiedBundle, ...], +) -> BoundedExamples[dict[str, Any]]: + summary = BoundedExamples[dict[str, Any]]() + ordered = sorted(bundles, key=lambda bundle: (bundle.bundle_id, bundle.source.sha256)) + for bundle_id, items in groupby(ordered, key=lambda bundle: bundle.bundle_id): + variant_count = 0 + previous_digest = None + digest_examples: list[str] = [] + for bundle in items: + digest = bundle.source.sha256 + if digest == previous_digest: + continue + previous_digest = digest + variant_count += 1 + if len(digest_examples) < SUMMARY_EXAMPLE_LIMIT: + digest_examples.append(digest) + if variant_count > 1: + summary.add( + { + "bundle_id": bundle_id, + "ciphertext_sha256_examples": digest_examples, + "ciphertext_sha256_examples_truncated": ( + variant_count > len(digest_examples) + ), + "ciphertext_variant_count": str(variant_count), + } + ) + return summary + + +def _bundle_overlaps(bundles: tuple[VerifiedBundle, ...]) -> dict[str, object]: + summary = BoundedExamples[dict[str, Any]]() + nonempty = (bundle for bundle in bundles if bundle.event_count) + for identity, grouped_items in groupby(nonempty, key=_bundle_participant): + active: list[tuple[int, int, VerifiedBundle]] = [] + for unique_index, current in enumerate(grouped_items): + while active and active[0][0] < current.first_sequence_number: + heapq.heappop(active) + overlap_count = len(active) + summary.add_count(overlap_count) + capacity = SUMMARY_EXAMPLE_LIMIT - len(summary.examples) + for _, _, previous in islice(active, max(0, capacity)): + summary.add_example( + { + "bundle_ids": sorted([previous.bundle_id, current.bundle_id]), + "configuration_id": identity[1], + "experiment_id": identity[0], + "first_sequence_number": str(current.first_sequence_number), + "last_sequence_number": str( + min( + previous.last_sequence_number, + current.last_sequence_number, + ) + ), + "participant_instance_id": identity[2], + } + ) + heapq.heappush( + active, (current.last_sequence_number, unique_index, current) + ) + return summary.document() + + +def _coverage_stream( + bundles: tuple[VerifiedBundle, ...], events: Iterable[VerifiedEvent] +) -> dict[str, object]: + event_groups = iter(groupby(events, key=lambda event: event.identity[:3])) + current_group = next(event_groups, None) + summary = BoundedExamples[dict[str, Any]]() + for identity, items in groupby(bundles, key=_bundle_participant): + latest = max( + items, + key=lambda item: ( + item.durable_through_sequence, + item.retained_from_sequence, + item.uploaded_through_sequence, + item.exported_at_utc_millis, + item.bundle_id, + item.source.sha256, + ), + ) + if current_group is not None and current_group[0] < identity: + raise RuntimeError("accepted event identity has no verified bundle") + if current_group is not None and current_group[0] == identity: + sequences = (event.sequence_number for event in current_group[1]) + summary.add(_streamed_coverage_record(identity, latest, sequences)) + current_group = next(event_groups, None) + else: + summary.add(_streamed_coverage_record(identity, latest, ())) + if current_group is not None: + raise RuntimeError("accepted event identity has no verified bundle") + return summary.document() + + +def _streamed_coverage_record( + identity: tuple[str, str, str], + latest: VerifiedBundle, + sequences: Iterable[int], +) -> dict[str, Any]: + reclaimed = BoundedExamples[dict[str, str]]() + gaps = BoundedExamples[dict[str, str]]() + reclaimed_cursor = 1 + delivered_cursor = latest.retained_from_sequence + maximum_delivered = 0 + for sequence in sequences: + maximum_delivered = sequence + if sequence < latest.retained_from_sequence: + if sequence > reclaimed_cursor: + reclaimed.add( + {"first": str(reclaimed_cursor), "last": str(sequence - 1)} + ) + reclaimed_cursor = max(reclaimed_cursor, sequence + 1) + else: + if sequence > delivered_cursor: + gaps.add( + {"first": str(delivered_cursor), "last": str(sequence - 1)} + ) + delivered_cursor = max(delivered_cursor, sequence + 1) + if reclaimed_cursor < latest.retained_from_sequence: + reclaimed.add( + { + "first": str(reclaimed_cursor), + "last": str(latest.retained_from_sequence - 1), + } + ) + undelivered_start = max( + maximum_delivered + 1, latest.retained_from_sequence + ) + not_yet_delivered = BoundedExamples[dict[str, str]]() + if undelivered_start <= latest.durable_through_sequence: + not_yet_delivered.add( + { + "first": str(undelivered_start), + "last": str(latest.durable_through_sequence), + } + ) + return { + "configuration_id": identity[1], + "durable_through_sequence": str(latest.durable_through_sequence), + "experiment_id": identity[0], + "interior_gaps": gaps.document(), + "not_yet_delivered": not_yet_delivered.document(), + "participant_instance_id": identity[2], + "reclaimed_prefix": reclaimed.document(), + "retained_from_sequence": str(latest.retained_from_sequence), + } + + +def _bundle_participant(bundle: VerifiedBundle) -> tuple[str, str, str]: + return ( + bundle.experiment_id, + bundle.configuration_id, + bundle.participant_instance_id, + ) diff --git a/adc-analysis/src/adc_analysis/sink.py b/adc-analysis/src/adc_analysis/sink.py new file mode 100644 index 0000000..6ed5ce9 --- /dev/null +++ b/adc-analysis/src/adc_analysis/sink.py @@ -0,0 +1,417 @@ +"""Typed dataset extension point and the sole supported Parquet implementation.""" + +from __future__ import annotations + +import hashlib +import os +import shutil +import tempfile +from collections.abc import Mapping +from itertools import groupby +from pathlib import Path +from typing import Any, Protocol + +import pyarrow as pa +import pyarrow.parquet as pq + +from . import __version__ +from .catalog import CollectorCatalog, PayloadSchema +from .errors import ConflictError, ValidationError +from .filesystem import rename_noreplace +from .jcs import canonicalize +from .models import PartitionedVerifiedEvents, VerifiedEvent +from .reassembly import ReassemblyResult +from .summary import BoundedExamples + +PARQUET_BATCH_MAX_ROWS = 65_536 +PARQUET_BATCH_MAX_ESTIMATED_BYTES = 16 * 1024 * 1024 + + +class DatasetSink(Protocol): + """Extension point for validated events; this release implements only Parquet.""" + + def write( + self, + result: ReassemblyResult, + destination: Path, + *, + validation_failures: tuple[Mapping[str, str], ...] = (), + ) -> Path: ... + + +class ParquetSink: + def __init__(self, catalog: CollectorCatalog): + self.catalog = catalog + + def write( + self, + result: ReassemblyResult, + destination: Path, + *, + validation_failures: tuple[Mapping[str, str], ...] = (), + ) -> Path: + if result.has_conflicts: + raise ConflictError( + "conflicting authenticated identities; dataset was not materialized" + ) + destination = Path(destination).absolute() + if destination.is_symlink(): + raise ValidationError("dataset destination must not be a symbolic link") + if destination.exists(): + raise ValidationError("dataset destination already exists") + destination.parent.mkdir(parents=True, exist_ok=True) + temporary = Path( + tempfile.mkdtemp(prefix=f".{destination.name}-", dir=destination.parent) + ) + try: + partitions = self._write_partitions(result.events, temporary) + quality = dict(result.quality) + quality["observations"] = _observations( + result.events, + self.catalog.sampling_clock_fields, + ) + _write_file(temporary / "quality-summary.json", canonicalize(quality)) + manifest = { + "dataset_format": "adc-parquet-dataset-v1", + "parser_version": __version__, + "partitions": partitions, + "source_ciphertexts": [ + { + "bundle_id": bundle.bundle_id, + "bundle_kind": bundle.bundle_kind, + "byte_count": str(bundle.source.byte_count), + "configuration_id": bundle.configuration_id, + "configuration_sha256": bundle.configuration_sha256, + "event_count": str(bundle.event_count), + "experiment_id": bundle.experiment_id, + "first_sequence_number": str(bundle.first_sequence_number), + "last_sequence_number": str(bundle.last_sequence_number), + "participant_instance_id": bundle.participant_instance_id, + "receiver_received_at_utc_untrusted": ( + bundle.source.metadata["received_at_utc"] + if bundle.source.metadata is not None + else None + ), + "sha256": bundle.source.sha256, + "source_object": bundle.source.source_uri, + } + for bundle in result.bundles + ], + "validation_failures": [dict(item) for item in validation_failures], + } + _write_file(temporary / "dataset-manifest.json", canonicalize(manifest)) + rename_noreplace(temporary, destination) + return destination + finally: + if temporary.exists(): + shutil.rmtree(temporary) + + def _write_partitions( + self, events: PartitionedVerifiedEvents, root: Path + ) -> list[dict[str, str]]: + ordered = events.iter_partitioned() + manifest: list[dict[str, str]] = [] + for key, partition_events in groupby(ordered, key=_partition_key): + experiment, configuration, collector, version, payload_type = key + schema = self.catalog.payload(collector, version, payload_type) + arrow_schema = _arrow_schema(schema) + directory = ( + root + / f"experiment_id={experiment}" + / f"configuration_id={configuration}" + / f"collector_id={collector}" + / f"payload_schema_version={version}" + / f"payload_type={payload_type}" + ) + directory.mkdir(parents=True, exist_ok=True) + path = directory / "part-00000.parquet" + writer = pq.ParquetWriter( + path, + arrow_schema, + compression="zstd", + use_dictionary=False, + write_statistics=True, + version="2.6", + ) + row_count = 0 + rows: list[dict[str, Any]] = [] + estimated_bytes = 0 + try: + for event in partition_events: + event_bytes = _estimated_row_bytes(event) + if rows and ( + len(rows) >= PARQUET_BATCH_MAX_ROWS + or estimated_bytes + event_bytes + > PARQUET_BATCH_MAX_ESTIMATED_BYTES + ): + row_count += _write_rows(writer, rows, arrow_schema) + rows.clear() + estimated_bytes = 0 + rows.append(_row(event, schema)) + estimated_bytes += event_bytes + if ( + len(rows) >= PARQUET_BATCH_MAX_ROWS + or estimated_bytes >= PARQUET_BATCH_MAX_ESTIMATED_BYTES + ): + row_count += _write_rows(writer, rows, arrow_schema) + rows.clear() + estimated_bytes = 0 + if rows: + row_count += _write_rows(writer, rows, arrow_schema) + finally: + writer.close() + relative = str(path.relative_to(root)) + manifest.append( + { + "file": relative, + "row_count": str(row_count), + "sha256": _sha256(path), + } + ) + return manifest + + +def _arrow_schema(schema: PayloadSchema) -> pa.Schema: + fields = [ + pa.field("participant_instance_id", pa.string(), nullable=False), + pa.field("assigned_participant_id", pa.string(), nullable=True), + pa.field("sequence_number", pa.int64(), nullable=False), + pa.field("observed_wall_time_utc_millis", pa.int64(), nullable=False), + pa.field("observed_monotonic_time_nanos", pa.int64(), nullable=False), + pa.field("observed_boot_session_id", pa.string(), nullable=False), + ] + reserved = {field.name for field in fields} | { + "experiment_id", + "configuration_id", + "collector_id", + "payload_schema_version", + "payload_type", + "source_ciphertext_sha256", + "source_bundle_id", + "source_configuration_sha256", + "source_object", + "parser_version", + } + for name, descriptor in sorted(schema.fields.items()): + if name in reserved: + raise ValidationError( + f"payload field collides with dataset provenance: {name}" + ) + metadata = { + b"adc.meaning": str(descriptor["meaning"]).encode(), + b"adc.type": descriptor["type"].encode(), + b"adc.unit": str(descriptor.get("unit", "none")).encode(), + } + if descriptor.get("clock_basis") is not None: + metadata[b"adc.clock_basis"] = str(descriptor["clock_basis"]).encode() + fields.append( + pa.field( + name, + _arrow_type(descriptor["type"]), + nullable=not descriptor["required"], + metadata=metadata, + ) + ) + fields.extend( + [ + pa.field("source_ciphertext_sha256", pa.string(), nullable=False), + pa.field("source_bundle_id", pa.string(), nullable=False), + pa.field("source_configuration_sha256", pa.string(), nullable=False), + pa.field("source_object", pa.string(), nullable=False), + pa.field("parser_version", pa.string(), nullable=False), + ] + ) + return pa.schema( + fields, + metadata={ + b"adc.collector_id": schema.collector_id.encode(), + b"adc.payload_schema_version": str(schema.schema_version).encode(), + b"adc.payload_type": schema.payload_type.encode(), + }, + ) + + +def _arrow_type(kind: str) -> pa.DataType: + return { + "boolean": pa.bool_(), + "decimal_string": pa.int64(), + "enum": pa.string(), + "float32": pa.float32(), + "float64": pa.float64(), + "int32": pa.int32(), + "json_string": pa.string(), + "string": pa.string(), + }[kind] + + +def _row(event: VerifiedEvent, schema: PayloadSchema) -> dict[str, Any]: + row = { + "participant_instance_id": event.participant_instance_id, + "assigned_participant_id": event.assigned_participant_id, + "sequence_number": event.sequence_number, + "observed_wall_time_utc_millis": event.wall_time_utc_millis, + "observed_monotonic_time_nanos": event.monotonic_time_nanos, + "observed_boot_session_id": event.boot_session_id, + "source_ciphertext_sha256": event.provenance.source_ciphertext_sha256, + "source_bundle_id": event.provenance.source_bundle_id, + "source_configuration_sha256": event.provenance.source_configuration_sha256, + "source_object": event.provenance.source_object, + "parser_version": __version__, + } + for name in schema.fields: + row[name] = event.fields.get(name) + return row + + +def _partition_key(event: VerifiedEvent) -> tuple[str, str, str, int, str]: + return ( + event.experiment_id, + event.configuration_id, + event.collector_id, + event.payload_schema_version, + event.payload_type, + ) + + +def _write_rows( + writer: pq.ParquetWriter, + rows: list[dict[str, Any]], + schema: pa.Schema, +) -> int: + writer.write_table( + pa.Table.from_pylist(rows, schema=schema), + row_group_size=len(rows), + ) + return len(rows) + + +def _estimated_row_bytes(event: VerifiedEvent) -> int: + """Conservatively bound each in-memory Arrow construction batch.""" + + strings = ( + event.experiment_id, + event.configuration_id, + event.participant_instance_id, + event.assigned_participant_id or "", + event.collector_id, + event.payload_type, + event.boot_session_id, + event.provenance.source_ciphertext_sha256, + event.provenance.source_bundle_id, + event.provenance.source_configuration_sha256, + event.provenance.source_object, + __version__, + ) + return len(event.canonical_bytes) + sum(len(value.encode()) for value in strings) + 512 + + +def _observations( + events: PartitionedVerifiedEvents, + source_clock_fields: Mapping[tuple[str, int, str], str], +) -> dict[str, Any]: + boot_sessions = BoundedExamples[dict[str, Any]]() + for participant, participant_sessions in groupby( + events.iter_boot_sessions(), + key=lambda item: ( + item.experiment_id, + item.configuration_id, + item.participant_instance_id, + ), + ): + if boot_sessions.has_capacity: + session_ids = BoundedExamples[str]() + for session in participant_sessions: + session_ids.add(session.boot_session_id) + boot_sessions.add( + { + "boot_session_ids": session_ids.document(), + "configuration_id": participant[1], + "experiment_id": participant[0], + "participant_instance_id": participant[2], + } + ) + else: + for _ in participant_sessions: + pass + boot_sessions.add_count(1) + + achieved = BoundedExamples[dict[str, Any]]() + for group in events.iter_sampling_groups(source_clock_fields): + duration_nanos = max( + 0, + group.last_monotonic_time_nanos + - group.first_monotonic_time_nanos, + ) + interval_count = max(0, group.event_count - 1) + achieved.add( + { + "boot_session_id": group.boot_session_id, + "clock_basis": "continuous_monotonic_since_boot", + "collector_id": group.collector_id, + "configuration_id": group.configuration_id, + "duration_monotonic_nanos": str(duration_nanos), + "event_count": str(group.event_count), + "experiment_id": group.experiment_id, + "mean_sampling_rate_millihertz": _mean_rate_millihertz( + interval_count, + duration_nanos, + ), + "participant_instance_id": group.participant_instance_id, + "sampling_interval_count": str(interval_count), + "source_clock_field": group.source_clock_field, + } + ) + + survey_lifecycle = BoundedExamples[dict[str, str]]() + for group in events.iter_survey_lifecycle_counts(): + survey_lifecycle.add( + { + "configuration_id": group.configuration_id, + "event_count": str(group.event_count), + "experiment_id": group.experiment_id, + "participant_instance_id": group.participant_instance_id, + "payload_type": group.payload_type, + } + ) + + temporal_changes = BoundedExamples[dict[str, str]]() + for event in events: + if event.collector_id == "temporal_context.v1": + temporal_changes.add( + { + "change_reason": str(event.fields.get("change_reason", "")), + "configuration_id": event.configuration_id, + "experiment_id": event.experiment_id, + "participant_instance_id": event.participant_instance_id, + "sequence_number": str(event.sequence_number), + "timezone_id": str(event.fields.get("timezone_id", "")), + } + ) + return { + "achieved_sampling_observations": achieved.document(), + "boot_sessions": boot_sessions.document(), + "survey_lifecycle_counts": survey_lifecycle.document(), + "temporal_context_events": temporal_changes.document(), + } + + +def _mean_rate_millihertz(interval_count: int, duration_nanos: int) -> str | None: + if interval_count == 0 or duration_nanos == 0: + return None + numerator = interval_count * 1_000_000_000_000 + return str((numerator + duration_nanos // 2) // duration_nanos) + + +def _write_file(path: Path, data: bytes) -> None: + with path.open("xb") as stream: + stream.write(data) + stream.flush() + os.fsync(stream.fileno()) + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + while chunk := stream.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() diff --git a/adc-analysis/src/adc_analysis/sources.py b/adc-analysis/src/adc_analysis/sources.py new file mode 100644 index 0000000..a18ae06 --- /dev/null +++ b/adc-analysis/src/adc_analysis/sources.py @@ -0,0 +1,101 @@ +"""Ciphertext-only source adapters.""" + +from __future__ import annotations + +from collections.abc import Iterable +from contextlib import closing +from pathlib import Path +from typing import Protocol +from urllib.parse import urlsplit + +from .errors import ValidationError +from .models import SourceObject + + +class BundleSource(Protocol): + """Enumerates bounded ciphertext objects without interpreting their contents.""" + + def objects(self) -> Iterable[SourceObject]: ... + + +class LocalBundleSource: + def __init__(self, paths: Iterable[Path]): + self.paths = tuple(Path(path).resolve() for path in paths) + if not self.paths: + raise ValidationError("at least one local path is required") + + def objects(self) -> Iterable[SourceObject]: + files: set[Path] = set() + for path in self.paths: + if path.is_file(): + files.add(path) + elif path.is_dir(): + files.update(item for item in path.rglob("*.adcexp") if item.is_file()) + else: + raise ValidationError(f"local source does not exist: {path}") + for path in sorted(files, key=str): + size = path.stat().st_size + yield SourceObject( + path.as_uri(), size, None, lambda path=path: path.open("rb") + ) + + +class S3BundleSource: + """S3-compatible/R2 reader. Credentials are resolved by boto3, never stored here.""" + + def __init__( + self, + bucket: str, + *, + prefix: str = "", + endpoint_url: str | None = None, + region_name: str | None = None, + profile_name: str | None = None, + client=None, + ): + if not bucket: + raise ValidationError("S3 bucket is required") + if endpoint_url is not None: + try: + endpoint = urlsplit(endpoint_url) + hostname = endpoint.hostname + except ValueError as error: + raise ValidationError("S3 endpoint must be HTTPS") from error + if endpoint.scheme != "https" or not hostname: + raise ValidationError("S3 endpoint must be HTTPS") + self.bucket = bucket + self.prefix = prefix + if client is None: + import boto3 + + session = boto3.Session(profile_name=profile_name) + client = session.client( + "s3", endpoint_url=endpoint_url, region_name=region_name + ) + self.client = client + + def objects(self) -> Iterable[SourceObject]: + paginator = self.client.get_paginator("list_objects_v2") + pages = paginator.paginate(Bucket=self.bucket, Prefix=self.prefix) + for page in pages: + for item in sorted( + page.get("Contents", []), key=lambda value: value["Key"] + ): + key = item["Key"] + if key.endswith("/"): + continue + head = self.client.head_object(Bucket=self.bucket, Key=key) + size = int(head["ContentLength"]) + metadata = {str(k): str(v) for k, v in head.get("Metadata", {}).items()} + + def opener(key=key): + response = self.client.get_object(Bucket=self.bucket, Key=key) + return closing(response["Body"]) + + yield SourceObject( + f"s3://{self.bucket}/{key}", + size, + metadata, + opener, + source_kind="receiver", + ) diff --git a/adc-analysis/src/adc_analysis/streaming_json.py b/adc-analysis/src/adc_analysis/streaming_json.py new file mode 100644 index 0000000..7213a28 --- /dev/null +++ b/adc-analysis/src/adc_analysis/streaming_json.py @@ -0,0 +1,202 @@ +"""Bounded streaming validation for Protocol v1 canonical JSON documents.""" + +from __future__ import annotations + +import hashlib +from collections.abc import Iterator +from decimal import Decimal +from pathlib import Path +from typing import Any, BinaryIO + +import ijson +from ijson.common import JSONError, ObjectBuilder + +from .errors import ValidationError +from .jcs import canonicalize +from .limits import JSON_MAX_DEPTH, JSON_STRING_TOKEN_MAX_BYTES + + +class CanonicalJsonEvents: + """Yield parser events and prove that the source bytes are exact integral JCS.""" + + def __init__(self, path: Path, expected_sha256: str, expected_bytes: int): + self.path = path + self.expected_sha256 = expected_sha256 + self.expected_bytes = expected_bytes + + def __iter__(self) -> Iterator[tuple[str, str, Any]]: + encoder = _CanonicalEncoder() + try: + with self.path.open("rb") as source: + reader = _BoundedJsonReader(source) + for prefix, event, value in ijson.parse(reader, use_float=False): + encoder.feed(event, value) + yield prefix, event, value + except ValidationError: + raise + except (JSONError, UnicodeError, ValueError, OverflowError) as error: + raise ValidationError("malformed JSON") from error + canonical_digest, canonical_count = encoder.finish() + if reader.count != self.expected_bytes or reader.digest.hexdigest() != self.expected_sha256: + raise ValidationError("plaintext changed while it was being parsed") + if ( + canonical_count != reader.count + or canonical_digest.hexdigest() != reader.digest.hexdigest() + ): + raise ValidationError("JSON is not canonical JCS") + + +class BoundedObjectBuilder: + """Materialize one already-streamed subtree under an explicit byte bound.""" + + def __init__(self, maximum_bytes: int): + self.builder = ObjectBuilder() + self.encoder = _CanonicalEncoder(maximum_bytes=maximum_bytes) + self.depth = 0 + self.complete = False + + def feed(self, event: str, value: Any) -> None: + if self.complete: + raise ValidationError("JSON subtree has trailing values") + self.encoder.feed(event, value) + self.builder.event(event, value) + if event in {"start_map", "start_array"}: + self.depth += 1 + elif event in {"end_map", "end_array"}: + self.depth -= 1 + if self.depth == 0: + self.complete = True + elif self.depth == 0: + self.complete = True + + @property + def value(self) -> Any: + if not self.complete: + raise ValidationError("JSON subtree is incomplete") + self.encoder.finish() + return self.builder.value + + +class _BoundedJsonReader: + def __init__(self, source: BinaryIO): + self.source = source + self.in_string = False + self.escaped = False + self.string_bytes = 0 + self.digest = hashlib.sha256() + self.count = 0 + + def read(self, size: int = -1) -> bytes: + data = self.source.read(size) + self.digest.update(data) + self.count += len(data) + self._scan(data) + return data + + def _scan(self, data: bytes) -> None: + for byte in data: + if not self.in_string: + if byte == 0x22: + self.in_string = True + self.escaped = False + self.string_bytes = 0 + continue + if self.escaped: + self.escaped = False + elif byte == 0x5C: + self.escaped = True + elif byte == 0x22: + self.in_string = False + continue + self.string_bytes += 1 + if self.string_bytes > JSON_STRING_TOKEN_MAX_BYTES: + raise ValidationError("JSON string exceeds the protocol bound") + + +class _CanonicalEncoder: + def __init__(self, *, maximum_bytes: int | None = None): + self.digest = hashlib.sha256() + self.count = 0 + self.maximum_bytes = maximum_bytes + self.stack: list[dict[str, Any]] = [] + self.root_seen = False + + def feed(self, event: str, value: Any) -> None: + if event == "map_key": + self._map_key(value) + return + if event in {"start_map", "start_array"}: + self._before_value() + self._write(b"{" if event == "start_map" else b"[") + self.stack.append( + { + "kind": "map" if event == "start_map" else "array", + "count": 0, + "awaiting": False, + "last_key": None, + } + ) + if len(self.stack) > JSON_MAX_DEPTH: + raise ValidationError("JSON nesting exceeds the protocol bound") + return + if event in {"end_map", "end_array"}: + expected = "map" if event == "end_map" else "array" + if not self.stack or self.stack[-1]["kind"] != expected: + raise ValidationError("malformed JSON container") + context = self.stack.pop() + if context["awaiting"]: + raise ValidationError("JSON object member has no value") + self._write(b"}" if event == "end_map" else b"]") + return + if event not in {"null", "boolean", "integer", "number", "string"}: + raise ValidationError(f"unsupported JSON token: {event}") + if event == "number" and (isinstance(value, Decimal) or not isinstance(value, int)): + raise ValidationError("floating-point JSON numbers are forbidden") + self._before_value() + self._write(canonicalize(value)) + + def finish(self): + if self.stack or not self.root_seen: + raise ValidationError("JSON document is incomplete") + return self.digest, self.count + + def _before_value(self) -> None: + if not self.stack: + if self.root_seen: + raise ValidationError("JSON document has multiple root values") + self.root_seen = True + return + context = self.stack[-1] + if context["kind"] == "array": + if context["count"]: + self._write(b",") + context["count"] += 1 + return + if not context["awaiting"]: + raise ValidationError("JSON object value has no member name") + context["awaiting"] = False + + def _map_key(self, value: Any) -> None: + if not self.stack or self.stack[-1]["kind"] != "map": + raise ValidationError("JSON member name is outside an object") + if not isinstance(value, str): + raise ValidationError("JSON member name must be a string") + context = self.stack[-1] + if context["awaiting"]: + raise ValidationError("JSON object member has no value") + encoded_key = value.encode("utf-16-be", errors="strict") + if context["last_key"] is not None and encoded_key <= context["last_key"]: + raise ValidationError("JSON object members are duplicate or not JCS-sorted") + if context["count"]: + self._write(b",") + self._write(canonicalize(value)) + self._write(b":") + context["count"] += 1 + context["awaiting"] = True + context["last_key"] = encoded_key + + def _write(self, data: bytes) -> None: + self.digest.update(data) + self.count += len(data) + if self.maximum_bytes is not None and self.count > self.maximum_bytes: + raise ValidationError("JSON subtree exceeds its protocol bound") diff --git a/adc-analysis/src/adc_analysis/summary.py b/adc-analysis/src/adc_analysis/summary.py new file mode 100644 index 0000000..169441f --- /dev/null +++ b/adc-analysis/src/adc_analysis/summary.py @@ -0,0 +1,42 @@ +"""Deterministic bounded summaries for arbitrarily large validated datasets.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Generic, TypeVar + +SUMMARY_EXAMPLE_LIMIT = 100 +T = TypeVar("T") + + +@dataclass(slots=True) +class BoundedExamples(Generic[T]): + """Count every occurrence while retaining only a fixed number of examples.""" + + count: int = 0 + examples: list[T] = field(default_factory=list) + + def add(self, example: T) -> None: + self.count += 1 + if len(self.examples) < SUMMARY_EXAMPLE_LIMIT: + self.examples.append(example) + + def add_count(self, count: int) -> None: + if count < 0: + raise ValueError("summary count increment must be non-negative") + self.count += count + + def add_example(self, example: T) -> None: + if len(self.examples) < SUMMARY_EXAMPLE_LIMIT: + self.examples.append(example) + + @property + def has_capacity(self) -> bool: + return len(self.examples) < SUMMARY_EXAMPLE_LIMIT + + def document(self) -> dict[str, object]: + return { + "count": str(self.count), + "examples": self.examples, + "examples_truncated": self.count > len(self.examples), + } diff --git a/adc-analysis/tests/__init__.py b/adc-analysis/tests/__init__.py new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/adc-analysis/tests/__init__.py @@ -0,0 +1 @@ + diff --git a/adc-analysis/tests/fakes.py b/adc-analysis/tests/fakes.py new file mode 100644 index 0000000..8137d88 --- /dev/null +++ b/adc-analysis/tests/fakes.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import io + + +class FakeS3Client: + def __init__(self, objects: dict[str, tuple[bytes, dict[str, str]]]): + self.objects = objects + + def get_paginator(self, name): + if name != "list_objects_v2": + raise AssertionError(name) + return _Paginator(self.objects) + + def head_object(self, *, Bucket, Key): + _ = Bucket + data, metadata = self.objects[Key] + return {"ContentLength": len(data), "Metadata": metadata} + + def get_object(self, *, Bucket, Key): + _ = Bucket + data, _ = self.objects[Key] + return {"Body": io.BytesIO(data)} + + +class _Paginator: + def __init__(self, objects): + self.objects = objects + + def paginate(self, *, Bucket, Prefix): + _ = Bucket + return [ + { + "Contents": [ + {"Key": key} + for key in sorted(self.objects) + if key.startswith(Prefix) + ] + } + ] diff --git a/adc-analysis/tests/test_cli.py b/adc-analysis/tests/test_cli.py new file mode 100644 index 0000000..62fbb5c --- /dev/null +++ b/adc-analysis/tests/test_cli.py @@ -0,0 +1,123 @@ +from __future__ import annotations + +import json +import os +import tempfile +import unittest +from contextlib import redirect_stdout +from io import StringIO +from pathlib import Path +from unittest.mock import patch + +from fakes import FakeS3Client + +from adc_analysis.cli import main +from adc_analysis.inventory import CiphertextInventory +from adc_analysis.sources import S3BundleSource + +REPOSITORY = Path(__file__).resolve().parents[2] +PROTOCOL = REPOSITORY / "protocol" / "v1" + + +class CliTest(unittest.TestCase): + def test_local_inventory_then_materialize(self) -> None: + corpus = json.loads((PROTOCOL / "conformance-vectors.json").read_text()) + bundle = corpus["valid"]["bundle"] + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "bundle.adcexp" + source.write_bytes(bytes.fromhex(bundle["container_hex"])) + workspace = root / "workspace" + output = StringIO() + with redirect_stdout(output): + result = main( + [ + "inventory", + "--workspace", + str(workspace), + "--local", + str(source), + ] + ) + self.assertEqual(0, result) + self.assertIn("inventoried 1", output.getvalue()) + + keys = root / "keys.json" + keys.write_text( + json.dumps( + { + "format": "adc-analysis-keys-v1", + "keys": { + "vector-hpke": bundle["researcher_private_key_base64url"] + }, + } + ) + ) + os.chmod(keys, 0o600) + dataset = root / "dataset" + with redirect_stdout(output): + result = main( + [ + "materialize", + "--workspace", + str(workspace), + "--keys", + str(keys), + "--output", + str(dataset), + "--catalog", + str(PROTOCOL / "collector-catalog.json"), + ] + ) + self.assertEqual(0, result) + self.assertTrue((dataset / "dataset-manifest.json").is_file()) + + def test_one_inventory_snapshot_can_combine_local_and_receiver_sources( + self, + ) -> None: + corpus = json.loads((PROTOCOL / "conformance-vectors.json").read_text()) + bundle = corpus["valid"]["bundle"] + encoded = bytes.fromhex(bundle["container_hex"]) + receipt = corpus["valid"]["upload_receipt"]["value"] + metadata = { + "sha256": receipt["sha256"], + "byte_count": receipt["byte_count"], + "configuration_sha256": receipt["configuration_sha256"], + "researcher_key_id": "vector-hpke", + "first_sequence_number": receipt["first_sequence_number"], + "last_sequence_number": receipt["last_sequence_number"], + "event_count": receipt["event_count"], + "received_at_utc": "2026-08-04T00:00:00.000Z", + } + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + local = root / "manual.adcexp" + local.write_bytes(encoded) + workspace = root / "workspace" + receiver = S3BundleSource( + "bucket", + client=FakeS3Client({"automatic.adcexp": (encoded, metadata)}), + ) + with ( + patch("adc_analysis.cli.S3BundleSource", return_value=receiver), + redirect_stdout(StringIO()), + ): + result = main( + [ + "inventory", + "--workspace", + str(workspace), + "--local", + str(local), + "--s3-bucket", + "bucket", + ] + ) + self.assertEqual(0, result) + objects = CiphertextInventory(workspace).load() + self.assertEqual(2, len(objects)) + self.assertEqual({"local", "receiver"}, {item.source_kind for item in objects}) + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_configuration_catalog.py b/adc-analysis/tests/test_configuration_catalog.py new file mode 100644 index 0000000..1c03ba0 --- /dev/null +++ b/adc-analysis/tests/test_configuration_catalog.py @@ -0,0 +1,289 @@ +from __future__ import annotations + +import copy +import json +import unittest +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +from adc_analysis.catalog import CollectorCatalog +from adc_analysis.configuration import validate_configuration +from adc_analysis.errors import ValidationError +from adc_analysis.jcs import parse + +REPOSITORY = Path(__file__).resolve().parents[2] +PROTOCOL = REPOSITORY / "protocol" / "v1" + + +class ConfigurationAndCatalogTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.catalog = CollectorCatalog(PROTOCOL / "collector-catalog.json") + corpus = json.loads((PROTOCOL / "conformance-vectors.json").read_text()) + encoded = bytes.fromhex( + corpus["valid"]["signed_configuration"]["canonical_jcs_utf8_hex"] + ) + cls.configuration = parse(encoded) + + def test_random_window_exact_schema_and_bounds(self) -> None: + configuration = copy.deepcopy(self.configuration) + configuration["interventions"] = [ + { + "id": "daily-ema", + "action": { + "type": "notification", + "notification_title": "Check in", + "notification_message": "How are you?", + }, + "triggers": [ + { + "id": "daily-ema-trigger", + "availability_minutes": 30, + "schedule": { + "type": "random_window", + "local_windows": [ + { + "start_local_time": "09:00", + "end_local_time": "11:00", + }, + { + "start_local_time": "15:00", + "end_local_time": "17:00", + }, + ], + "occurrences_per_window": 1, + "maximum_occurrences_per_day": 2, + "maximum_occurrences_total": 20, + "minimum_separation_minutes": 60, + }, + } + ], + } + ] + validate_configuration(configuration, self.catalog) + invalid = copy.deepcopy(configuration) + invalid["interventions"][0]["triggers"][0]["schedule"]["local_windows"][1][ + "start_local_time" + ] = "10:00" + with self.assertRaises(ValidationError): + validate_configuration(invalid, self.catalog) + impossible = copy.deepcopy(configuration) + schedule = impossible["interventions"][0]["triggers"][0]["schedule"] + schedule["local_windows"] = [ + {"start_local_time": "09:00", "end_local_time": "09:30"} + ] + schedule["occurrences_per_window"] = 2 + schedule["maximum_occurrences_per_day"] = 2 + schedule["minimum_separation_minutes"] = 30 + with self.assertRaises(ValidationError): + validate_configuration(impossible, self.catalog) + + def test_random_window_global_bound_uses_each_signed_lifetime_cap(self) -> None: + configuration = copy.deepcopy(self.configuration) + configuration["duration_hours"] = 1 + + def trigger(index: int) -> dict[str, Any]: + return { + "id": f"random-trigger-{index}", + "availability_minutes": 30, + "schedule": { + "type": "random_window", + "local_windows": [ + {"start_local_time": "08:00", "end_local_time": "09:00"} + ], + "occurrences_per_window": 8, + "maximum_occurrences_per_day": 8, + "maximum_occurrences_total": 512, + "minimum_separation_minutes": 1, + }, + } + + configuration["interventions"] = [ + { + "id": "date-line-ema", + "action": { + "type": "notification", + "notification_title": "Check in", + "notification_message": "Please respond", + }, + "triggers": [trigger(1)], + } + ] + validate_configuration(configuration, self.catalog) + configuration["interventions"][0]["triggers"].append(trigger(2)) + with self.assertRaises(ValidationError): + validate_configuration(configuration, self.catalog) + + def test_catalog_converts_wire_strings_without_schema_inference(self) -> None: + battery = self.catalog.payload("battery_state.v1", 1, "BATTERY_STATE") + fields = { + "charging_source": "USB", + "charging_state": "CHARGING", + "percentage": "87", + "power_save_enabled": "false", + } + typed = self.catalog.typed_fields(battery, fields) + self.assertEqual( + { + "charging_source": "USB", + "charging_state": "CHARGING", + "percentage": 87, + "power_save_enabled": False, + }, + typed, + ) + with self.assertRaises(ValidationError): + self.catalog.typed_fields(battery, {"percentage": "87"}) + with self.assertRaises(ValidationError): + self.catalog.typed_fields(battery, dict(typed, unknown="value")) + for invalid_percentage in ("-1", "-0", "+1", "01", "101"): + with ( + self.subTest(percentage=invalid_percentage), + self.assertRaises(ValidationError), + ): + self.catalog.typed_fields( + battery, dict(fields, percentage=invalid_percentage) + ) + + def test_optional_payload_fields_remain_nullable(self) -> None: + location = self.catalog.payload("location.v1", 1, "LOCATION_FIX") + required = { + name: _example(descriptor["type"], descriptor) + for name, descriptor in location.fields.items() + if descriptor["required"] + } + converted = self.catalog.typed_fields(location, required) + self.assertEqual(set(required), set(converted)) + + def test_instant_text_must_match_the_signed_configuration_codec(self) -> None: + invalid = copy.deepcopy(self.configuration) + invalid["issued_at"] = "2026-01-01T00:00:00.000Z" + with self.assertRaises(ValidationError): + validate_configuration(invalid, self.catalog) + + def test_every_catalog_payload_has_a_concrete_typed_conversion(self) -> None: + for schema in self.catalog.payload_schemas: + with self.subTest( + collector=schema.collector_id, payload=schema.payload_type + ): + fields = { + name: _example(descriptor["type"], descriptor) + for name, descriptor in schema.fields.items() + } + self.assertEqual( + set(fields), set(self.catalog.typed_fields(schema, fields)) + ) + + def test_all_survey_questions_and_local_schedule_types(self) -> None: + configuration = copy.deepcopy(self.configuration) + text = {"default": "Prompt", "translations": {"zh-TW": "問題"}} + options = [ + {"id": "option-a", "label": text}, + {"id": "option-b", "label": text}, + ] + configuration["surveys"] = [ + { + "id": "daily-survey", + "title": text, + "description": text, + "questions": [ + { + "type": "short_text", + "id": "short-question", + "prompt": text, + "required": True, + "maximum_length": 200, + }, + { + "type": "scale", + "id": "scale-question", + "prompt": text, + "required": True, + "minimum": 1, + "maximum": 7, + "minimum_label": text, + "maximum_label": text, + }, + { + "type": "single_choice", + "id": "single-question", + "prompt": text, + "required": False, + "options": options, + }, + { + "type": "multiple_choice", + "id": "multiple-question", + "prompt": text, + "required": True, + "options": options, + "minimum_selections": 1, + "maximum_selections": 2, + }, + ], + } + ] + configuration["interventions"] = [ + { + "id": "survey-reminder", + "action": { + "type": "survey", + "notification_title": "Survey", + "notification_message": "Please respond", + "survey_id": "daily-survey", + }, + "triggers": [ + { + "id": "one-time-trigger", + "availability_minutes": 30, + "schedule": { + "type": "one_time", + "offset_minutes": 0, + "clock": "ACTIVE_RUNNING_TIME", + }, + }, + { + "id": "interval-trigger", + "availability_minutes": 30, + "schedule": { + "type": "interval", + "start_offset_minutes": 0, + "interval_minutes": 1_440, + "clock": "CALENDAR_TIME", + }, + }, + { + "id": "daily-trigger", + "availability_minutes": 30, + "schedule": {"type": "daily_local", "local_time": "08:30"}, + }, + ], + } + ] + validate_configuration(configuration, self.catalog) + invalid = copy.deepcopy(configuration) + invalid["surveys"][0]["questions"][3]["minimum_selections"] = 0 + with self.assertRaises(ValidationError): + validate_configuration(invalid, self.catalog) + + +def _example(kind: str, descriptor: Mapping[str, Any]) -> str: + if kind == "boolean": + return "false" + if kind == "decimal_string": + return "1" + if kind == "enum": + return descriptor["enum"][0] + if kind in {"float32", "float64"}: + minimum = descriptor.get("minimum", 0) + return str(float(minimum)) + if kind == "int32": + return "1" + if kind == "json_string": + return "{}" + return "value" + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_conformance.py b/adc-analysis/tests/test_conformance.py new file mode 100644 index 0000000..32784e3 --- /dev/null +++ b/adc-analysis/tests/test_conformance.py @@ -0,0 +1,232 @@ +from __future__ import annotations + +import hashlib +import json +import tempfile +import unittest +from pathlib import Path + +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + +from adc_analysis.bundle import BundleVerifier +from adc_analysis.catalog import CollectorCatalog +from adc_analysis.configuration import validate_configuration +from adc_analysis.crypto import open_base +from adc_analysis.encoding import base64url_decode, sha256_hex, uuid4_text +from adc_analysis.errors import ValidationError +from adc_analysis.jcs import canonical_decimal, canonicalize, exact_object, parse +from adc_analysis.models import InventoryObject + +REPOSITORY = Path(__file__).resolve().parents[2] +PROTOCOL = REPOSITORY / "protocol" / "v1" + + +class ProtocolConformanceTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.corpus = json.loads((PROTOCOL / "conformance-vectors.json").read_text()) + cls.catalog = CollectorCatalog(PROTOCOL / "collector-catalog.json") + + def test_valid_configuration_and_bundle(self) -> None: + signed = self.corpus["valid"]["signed_configuration"] + configuration_bytes = bytes.fromhex(signed["canonical_jcs_utf8_hex"]) + validate_configuration(parse(configuration_bytes), self.catalog) + _validate_signed_configuration( + bytes.fromhex(signed["envelope_hex"]), self.catalog + ) + parse( + bytes.fromhex( + self.corpus["valid"]["canonical_json"]["canonical_jcs_utf8_hex"] + ) + ) + bundle = self.corpus["valid"]["bundle"] + verified = self._verify_container(bytes.fromhex(bundle["container_hex"])) + self.assertEqual(bundle["bundle_id"], verified.bundle_id) + self.assertEqual(verified.event_count, len(verified.events)) + + def test_hpke_golden_material(self) -> None: + bundle = self.corpus["valid"]["bundle"] + key = base64url_decode( + bundle["researcher_private_key_base64url"], 32, "private key" + ) + plaintext = open_base( + key, + bytes.fromhex(bundle["hpke_wrapped_content_key_hex"]), + bytes.fromhex(bundle["context_jcs_utf8_hex"]), + ) + self.assertEqual(bundle["content_key_hex"], plaintext.hex()) + + def test_every_hostile_vector_is_consumed_and_rejected(self) -> None: + consumed: set[str] = set() + for vector in self.corpus["hostile"]: + entrypoint = vector["entrypoint"] + encoded = bytes.fromhex(vector["input_hex"]) + with ( + self.subTest(vector=vector["id"]), + self.assertRaises((ValidationError, ValueError)), + ): + if entrypoint == "canonical_json": + parse(encoded) + elif entrypoint == "configuration_jcs": + validate_configuration(parse(encoded), self.catalog) + elif entrypoint == "signed_configuration": + _validate_signed_configuration(encoded, self.catalog) + elif entrypoint == "bundle": + self._verify_container(encoded) + elif entrypoint == "bundle_unwrap_context": + bundle = self.corpus["valid"]["bundle"] + open_base( + base64url_decode( + bundle["researcher_private_key_base64url"], + 32, + "private key", + ), + bytes.fromhex(bundle["hpke_wrapped_content_key_hex"]), + encoded, + ) + elif entrypoint == "receipt": + _validate_receipt(encoded) + else: + self.fail(f"unhandled conformance entrypoint: {entrypoint}") + consumed.add(vector["id"]) + self.assertEqual({item["id"] for item in self.corpus["hostile"]}, consumed) + + def test_manual_export_streams_events_and_receiver_origin_rejects_it(self) -> None: + bundle = self.corpus["valid"]["bundle"] + encoded = bytes.fromhex(bundle["container_hex"]) + document = json.loads(bytes.fromhex(bundle["document_jcs_utf8_hex"])) + document["bundle_kind"] = "manual_export" + key_length = int.from_bytes(encoded[56:58], "big") + ciphertext_start = 70 + key_length + 80 + manual = encoded[:ciphertext_start] + AESGCM( + bytes.fromhex(bundle["content_key_hex"]) + ).encrypt( + bytes.fromhex(bundle["content_nonce_hex"]), + canonicalize(document), + bytes.fromhex(bundle["context_jcs_utf8_hex"]), + ) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + path = root / "manual.adcexp" + path.write_bytes(manual) + verifier = BundleVerifier( + self.catalog, + { + "vector-hpke": base64url_decode( + bundle["researcher_private_key_base64url"], 32, "private key" + ) + }, + root / "staging", + ) + local = InventoryObject( + path.as_uri(), + hashlib.sha256(manual).hexdigest(), + len(manual), + path, + None, + ) + verified = verifier.verify(local) + try: + self.assertEqual("manual_export", verified.bundle_kind) + self.assertEqual(verified.event_count, len(list(verified.events))) + finally: + verified.events.close() + + receiver = InventoryObject( + path.as_uri(), + hashlib.sha256(manual).hexdigest(), + len(manual), + path, + None, + "receiver", + ) + with self.assertRaises(ValidationError): + verifier.verify(receiver) + + def _verify_container(self, encoded: bytes): + bundle = self.corpus["valid"]["bundle"] + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "vector.adcexp" + path.write_bytes(encoded) + source = InventoryObject( + path.as_uri(), + hashlib.sha256(encoded).hexdigest(), + len(encoded), + path, + None, + ) + verifier = BundleVerifier( + self.catalog, + { + "vector-hpke": base64url_decode( + bundle["researcher_private_key_base64url"], 32, "private key" + ) + }, + Path(temporary) / "staging", + ) + return verifier.verify(source) + + +def _validate_signed_configuration(encoded: bytes, catalog: CollectorCatalog) -> None: + if len(encoded) < 14 + 64 or encoded[:8] != b"ADCCFG01": + raise ValidationError("invalid signed configuration framing") + key_length = int.from_bytes(encoded[8:10], "big") + config_length = int.from_bytes(encoded[10:14], "big") + if not 3 <= key_length <= 64 or not 2 <= config_length <= 1_048_576: + raise ValidationError("invalid signed configuration length") + if len(encoded) != 14 + key_length + config_length + 64: + raise ValidationError("signed configuration has trailing or truncated bytes") + try: + key_id = encoded[14 : 14 + key_length].decode("utf-8", errors="strict") + except UnicodeDecodeError as error: + raise ValidationError("signer key ID is malformed UTF-8") from error + configuration_bytes = encoded[14 + key_length : 14 + key_length + config_length] + configuration = parse(configuration_bytes) + validate_configuration(configuration, catalog) + if key_id != configuration["signer"]["key_id"]: + raise ValidationError("signer key ID mismatch") + public = base64url_decode(configuration["signer"]["public_key"], 32, "signer key") + signature = encoded[-64:] + try: + Ed25519PublicKey.from_public_bytes(public).verify( + signature, configuration_bytes + ) + except (ValueError, InvalidSignature) as error: + raise ValidationError("configuration signature failed") from error + + +def _validate_receipt(encoded: bytes) -> None: + receipt = exact_object( + parse(encoded), + { + "bundle_id", + "byte_count", + "configuration_sha256", + "event_count", + "first_sequence_number", + "last_sequence_number", + "sha256", + }, + "receipt", + ) + uuid4_text(receipt["bundle_id"], "receipt bundle ID") + sha256_hex(receipt["configuration_sha256"], "configuration digest") + sha256_hex(receipt["sha256"], "ciphertext digest") + byte_count = canonical_decimal( + receipt["byte_count"], "byte_count", maximum=33_554_432 + ) + first = canonical_decimal(receipt["first_sequence_number"], "first_sequence_number") + last = canonical_decimal(receipt["last_sequence_number"], "last_sequence_number") + count = canonical_decimal(receipt["event_count"], "event_count") + if ( + byte_count == 0 + or first == 0 + or last != (first - 1 if count == 0 else first + count - 1) + ): + raise ValidationError("receipt arithmetic mismatch") + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_inventory.py b/adc-analysis/tests/test_inventory.py new file mode 100644 index 0000000..58c2abe --- /dev/null +++ b/adc-analysis/tests/test_inventory.py @@ -0,0 +1,144 @@ +from __future__ import annotations + +import hashlib +import io +import tempfile +import unittest +from pathlib import Path + +from fakes import FakeS3Client + +from adc_analysis.errors import ValidationError +from adc_analysis.inventory import CiphertextInventory +from adc_analysis.jcs import canonicalize, parse +from adc_analysis.models import SourceObject +from adc_analysis.sources import LocalBundleSource, S3BundleSource + + +class _Source: + def __init__(self, objects): + self._objects = objects + + def objects(self): + return iter(self._objects) + + +class InventoryTest(unittest.TestCase): + def test_local_inventory_is_content_addressed_and_reloadable(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "source" + source.mkdir() + (source / "b.adcexp").write_bytes(b"ADCEXP01-b") + (source / "a.adcexp").write_bytes(b"ADCEXP01-a") + (source / "ignored.txt").write_text("not a bundle") + inventory = CiphertextInventory(root / "workspace") + first = inventory.ingest([LocalBundleSource([source])]) + second = inventory.load() + self.assertEqual(first, second) + self.assertEqual(2, len(first)) + for item in first: + self.assertEqual(item.sha256, item.cache_path.stem) + self.assertEqual("local", item.source_kind) + self.assertEqual(0, item.cache_path.stat().st_mode & 0o077) + + def test_receiver_metadata_is_closed_world_and_checked(self) -> None: + data = b"ADCEXP01-ciphertext" + digest = hashlib.sha256(data).hexdigest() + metadata = { + "sha256": digest, + "byte_count": str(len(data)), + "configuration_sha256": "1" * 64, + "researcher_key_id": "researcher-key", + "first_sequence_number": "1", + "last_sequence_number": "1", + "event_count": "1", + "received_at_utc": "2026-08-04T00:00:00.000Z", + } + with tempfile.TemporaryDirectory() as temporary: + inventory = CiphertextInventory(Path(temporary) / "workspace") + objects = inventory.ingest( + [ + S3BundleSource( + "bucket", + prefix="prefix/", + client=FakeS3Client( + { + "prefix/b.adcexp": (data, metadata), + "prefix/a.adcexp": (data, metadata), + } + ), + ) + ] + ) + self.assertEqual( + ["s3://bucket/prefix/a.adcexp", "s3://bucket/prefix/b.adcexp"], + [o.source_uri for o in objects], + ) + self.assertTrue(all(o.source_kind == "receiver" for o in objects)) + invalid = dict(metadata, unexpected="value") + with self.assertRaises(ValidationError): + inventory.ingest( + [ + S3BundleSource( + "bucket", + client=FakeS3Client({"invalid": (data, invalid)}), + ) + ] + ) + invalid_time = dict(metadata, received_at_utc="2026-99-04T00:00:00.000Z") + with self.assertRaises(ValidationError): + inventory.ingest( + [ + S3BundleSource( + "bucket", + client=FakeS3Client({"invalid-time": (data, invalid_time)}), + ) + ] + ) + + def test_inventory_manifest_cannot_redirect_a_cache_entry(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "source.adcexp" + source.write_bytes(b"ADCEXP01-content") + inventory = CiphertextInventory(root / "workspace") + inventory.ingest([LocalBundleSource([source])]) + document = parse(inventory.manifest.read_bytes()) + document["objects"][0]["cache_path"] = "another-file" + inventory.manifest.write_bytes(canonicalize(document)) + with self.assertRaises(ValidationError): + inventory.load() + + def test_source_size_and_digest_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + inventory = CiphertextInventory(Path(temporary) / "workspace") + oversized = SourceObject( + "memory:large", 33_554_433, None, lambda: io.BytesIO(b"") + ) + with self.assertRaisesRegex(ValidationError, "changed while reading"): + inventory.ingest([_Source([oversized])]) + receiver_oversized = SourceObject( + "memory:receiver-large", + 33_554_433, + {}, + lambda: io.BytesIO(b""), + "receiver", + ) + with self.assertRaisesRegex(ValidationError, "outside protocol bound"): + inventory.ingest([_Source([receiver_oversized])]) + changing = SourceObject( + "memory:changing", 3, None, lambda: io.BytesIO(b"four") + ) + with self.assertRaises(ValidationError): + inventory.ingest([_Source([changing])]) + with self.assertRaises(ValidationError): + S3BundleSource( + "bucket", + endpoint_url="http://example.test", + client=FakeS3Client({}), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_pipeline.py b/adc-analysis/tests/test_pipeline.py new file mode 100644 index 0000000..5cef1a6 --- /dev/null +++ b/adc-analysis/tests/test_pipeline.py @@ -0,0 +1,181 @@ +from __future__ import annotations + +import json +import os +import tempfile +import unittest +from pathlib import Path + +import pyarrow.parquet as pq +from fakes import FakeS3Client + +from adc_analysis.catalog import CollectorCatalog +from adc_analysis.encoding import base64url_decode +from adc_analysis.errors import ValidationError +from adc_analysis.inventory import CiphertextInventory +from adc_analysis.pipeline import AnalysisPipeline, load_private_keys +from adc_analysis.sink import ParquetSink +from adc_analysis.sources import LocalBundleSource, S3BundleSource + +REPOSITORY = Path(__file__).resolve().parents[2] +PROTOCOL = REPOSITORY / "protocol" / "v1" + + +class PipelineTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.corpus = json.loads((PROTOCOL / "conformance-vectors.json").read_text()) + cls.bundle = cls.corpus["valid"]["bundle"] + cls.catalog = CollectorCatalog(PROTOCOL / "collector-catalog.json") + cls.keys = { + "vector-hpke": base64url_decode( + cls.bundle["researcher_private_key_base64url"], 32, "private key" + ) + } + + def test_valid_bundle_materializes_typed_parquet_with_provenance(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "source" + source.mkdir() + (source / "valid.adcexp").write_bytes( + bytes.fromhex(self.bundle["container_hex"]) + ) + workspace = root / "workspace" + CiphertextInventory(workspace).ingest([LocalBundleSource([source])]) + destination = AnalysisPipeline( + workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "dataset") + parquet = next(destination.rglob("*.parquet")) + table = pq.read_table(parquet) + row = table.to_pylist()[0] + self.assertEqual(1, row["sequence_number"]) + self.assertEqual(self.bundle["bundle_id"], row["source_bundle_id"]) + self.assertEqual("vector-study", row["experiment_id"]) + manifest = json.loads((destination / "dataset-manifest.json").read_text()) + self.assertEqual("adc-parquet-dataset-v1", manifest["dataset_format"]) + self.assertEqual([], manifest["validation_failures"]) + + second_workspace = root / "workspace-second" + CiphertextInventory(second_workspace).ingest([LocalBundleSource([source])]) + second = AnalysisPipeline( + second_workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "dataset-second") + self.assertEqual( + parquet.read_bytes(), next(second.rglob("*.parquet")).read_bytes() + ) + self.assertEqual( + [], list((workspace / "staging" / "reassembly").glob("*.sqlite3")) + ) + + def test_s3_and_local_sources_produce_the_same_authenticated_rows(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + encoded = bytes.fromhex(self.bundle["container_hex"]) + local_file = root / "bundle.adcexp" + local_file.write_bytes(encoded) + receipt = self.corpus["valid"]["upload_receipt"]["value"] + metadata = { + "sha256": receipt["sha256"], + "byte_count": receipt["byte_count"], + "configuration_sha256": receipt["configuration_sha256"], + "researcher_key_id": "vector-hpke", + "first_sequence_number": receipt["first_sequence_number"], + "last_sequence_number": receipt["last_sequence_number"], + "event_count": receipt["event_count"], + "received_at_utc": "2026-08-04T00:00:00.000Z", + } + local_workspace = root / "local-workspace" + r2_workspace = root / "r2-workspace" + CiphertextInventory(local_workspace).ingest( + [LocalBundleSource([local_file])] + ) + CiphertextInventory(r2_workspace).ingest( + [ + S3BundleSource( + "bucket", + client=FakeS3Client( + {self.bundle["bundle_id"]: (encoded, metadata)} + ), + ) + ] + ) + local_output = AnalysisPipeline( + local_workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "local-dataset") + r2_output = AnalysisPipeline( + r2_workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "r2-dataset") + local_row = pq.read_table( + next(local_output.rglob("*.parquet")) + ).to_pylist()[0] + r2_row = pq.read_table(next(r2_output.rglob("*.parquet"))).to_pylist()[0] + local_row.pop("source_object") + r2_row.pop("source_object") + self.assertEqual(local_row, r2_row) + + def test_invalid_bundle_is_quarantined_without_partial_rows(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "source" + source.mkdir() + encoded = bytearray.fromhex(self.bundle["container_hex"]) + encoded[-1] ^= 1 + (source / "invalid.adcexp").write_bytes(encoded) + workspace = root / "workspace" + CiphertextInventory(workspace).ingest([LocalBundleSource([source])]) + with self.assertRaises(ValidationError): + AnalysisPipeline( + workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "dataset") + self.assertFalse((root / "dataset").exists()) + self.assertEqual(1, len(list((workspace / "quarantine").rglob("*.adcexp")))) + self.assertEqual([], list((workspace / "staging" / "plaintext").glob("*"))) + + def test_invalid_bundle_does_not_contribute_rows_when_valid_data_remains( + self, + ) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "source" + source.mkdir() + valid = bytes.fromhex(self.bundle["container_hex"]) + invalid = bytearray(valid) + invalid[-1] ^= 1 + (source / "valid.adcexp").write_bytes(valid) + (source / "invalid.adcexp").write_bytes(invalid) + workspace = root / "workspace" + CiphertextInventory(workspace).ingest([LocalBundleSource([source])]) + output = AnalysisPipeline( + workspace, self.catalog, self.keys, ParquetSink(self.catalog) + ).materialize(root / "dataset") + self.assertEqual(1, pq.read_table(next(output.rglob("*.parquet"))).num_rows) + manifest = json.loads((output / "dataset-manifest.json").read_text()) + self.assertEqual(1, len(manifest["validation_failures"])) + + def test_private_key_file_requires_private_permissions_and_exact_shape( + self, + ) -> None: + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "keys.json" + path.write_text( + json.dumps( + { + "format": "adc-analysis-keys-v1", + "keys": { + "vector-hpke": self.bundle[ + "researcher_private_key_base64url" + ] + }, + } + ) + ) + os.chmod(path, 0o600) + self.assertEqual(self.keys, load_private_keys(path)) + os.chmod(path, 0o644) + with self.assertRaises(ValidationError): + load_private_keys(path) + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_reassembly.py b/adc-analysis/tests/test_reassembly.py new file mode 100644 index 0000000..18b0a7a --- /dev/null +++ b/adc-analysis/tests/test_reassembly.py @@ -0,0 +1,202 @@ +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from adc_analysis.catalog import CollectorCatalog +from adc_analysis.errors import ConflictError +from adc_analysis.models import ( + EventProvenance, + InventoryObject, + VerifiedBundle, + VerifiedEvent, +) +from adc_analysis.reassembly import Reassembler +from adc_analysis.sink import ParquetSink + +REPOSITORY = Path(__file__).resolve().parents[2] + + +class _MemoryEvents: + def __init__(self, events: tuple[VerifiedEvent, ...]): + self.events = events + + def __iter__(self): + return iter(self.events) + + def __len__(self) -> int: + return len(self.events) + + def close(self) -> None: + pass + + +def _event( + sequence: int, content: bytes = b"same", bundle_id: str = "bundle-a" +) -> VerifiedEvent: + provenance = EventProvenance("a" * 64, bundle_id, "b" * 64, f"memory:{bundle_id}") + return VerifiedEvent( + "study-id", + "config-id", + "00000000-0000-0000-0000-000000000001", + None, + sequence, + "app_lifecycle.v1", + 1, + "ACTIVITY_CREATED", + "boot", + sequence * 10, + sequence * 20, + {"activity_class": "Activity"}, + content, + provenance, + ) + + +def _bundle( + bundle_id: str, + events: tuple[VerifiedEvent, ...], + *, + first: int | None = None, + last: int | None = None, + retained: int = 1, + durable: int | None = None, + sha: str = "a" * 64, +) -> VerifiedBundle: + first = first if first is not None else (events[0].sequence_number if events else 1) + last = ( + last + if last is not None + else (events[-1].sequence_number if events else first - 1) + ) + durable = durable if durable is not None else max(last, 0) + source = InventoryObject( + f"memory:{bundle_id}:{sha}", sha, 100, Path("/tmp/unused"), None + ) + return VerifiedBundle( + bundle_id, + "manual_export", + "b" * 64, + "study-id", + "config-id", + "00000000-0000-0000-0000-000000000001", + durable, + first, + last, + len(events), + retained, + max(0, retained - 1), + durable, + durable + 1, + _MemoryEvents(events), + source, + ) + + +class ReassemblyTest(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.reassembly_index = 0 + + def _reassemble(self, bundles): + self.reassembly_index += 1 + result = Reassembler( + Path(self.temporary.name) / f"reassembly-{self.reassembly_index}" + ).reassemble(bundles) + self.addCleanup(result.events.close) + return result + + def test_duplicates_overlaps_gaps_and_reclaimed_prefix_are_distinct(self) -> None: + first = _event(3, bundle_id="bundle-a") + duplicate = _event(3, bundle_id="bundle-b") + fourth = _event(4, b"four", bundle_id="bundle-b") + result = self._reassemble( + [ + _bundle("bundle-a", (first,), retained=3, durable=6), + _bundle("bundle-b", (duplicate, fourth), retained=3, durable=6), + ] + ) + self.assertFalse(result.has_conflicts) + self.assertEqual([3, 4], [event.sequence_number for event in result.events]) + self.assertEqual("1", result.quality["identical_event_duplicates"]["count"]) + self.assertEqual("1", result.quality["range_overlaps"]["count"]) + coverage = result.quality["participant_coverage"]["examples"][0] + self.assertEqual( + [{"first": "1", "last": "2"}], + coverage["reclaimed_prefix"]["examples"], + ) + self.assertEqual([], coverage["interior_gaps"]["examples"]) + self.assertEqual( + [{"first": "5", "last": "6"}], + coverage["not_yet_delivered"]["examples"], + ) + + def test_content_conflict_has_no_winner(self) -> None: + result = self._reassemble( + [ + _bundle("bundle-a", (_event(1, b"one", "bundle-a"),)), + _bundle( + "bundle-b", (_event(1, b"different", "bundle-b"),), sha="c" * 64 + ), + ] + ) + self.assertTrue(result.has_conflicts) + self.assertEqual([], list(result.events)) + self.assertEqual("1", result.quality["event_conflicts"]["count"]) + with tempfile.TemporaryDirectory() as temporary: + destination = Path(temporary) / "dataset" + with self.assertRaises(ConflictError): + ParquetSink( + CollectorCatalog( + REPOSITORY / "protocol" / "v1" / "collector-catalog.json" + ) + ).write(result, destination) + self.assertFalse(destination.exists()) + + def test_interior_gap_is_not_a_not_yet_delivered_suffix(self) -> None: + result = self._reassemble( + [ + _bundle("bundle-a", (_event(1),), durable=5), + _bundle( + "bundle-b", + (_event(3, b"three", "bundle-b"),), + durable=5, + sha="c" * 64, + ), + ] + ) + coverage = result.quality["participant_coverage"]["examples"][0] + self.assertEqual( + [{"first": "2", "last": "2"}], + coverage["interior_gaps"]["examples"], + ) + self.assertEqual( + [{"first": "4", "last": "5"}], + coverage["not_yet_delivered"]["examples"], + ) + + def test_reassembled_event_collection_is_repeatable(self) -> None: + bundles = [ + _bundle("bundle-a", (_event(1, bundle_id="bundle-a"),), durable=3), + _bundle( + "bundle-b", + ( + _event(1, bundle_id="bundle-b"), + _event(2, b"two", "bundle-b"), + ), + durable=3, + sha="c" * 64, + ), + ] + result = self._reassemble(bundles) + first = [event.canonical_bytes for event in result.events] + second = [event.canonical_bytes for event in result.events] + self.assertEqual(first, second) + self.assertEqual([b"same", b"two"], first) + + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_sink.py b/adc-analysis/tests/test_sink.py new file mode 100644 index 0000000..2422181 --- /dev/null +++ b/adc-analysis/tests/test_sink.py @@ -0,0 +1,364 @@ +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +import pyarrow.parquet as pq + +from adc_analysis.catalog import CollectorCatalog +from adc_analysis.event_store import EventDatabase +from adc_analysis.models import ( + BootSession, + EventProvenance, + SamplingGroup, + SurveyLifecycleCount, + VerifiedEvent, +) +from adc_analysis.reassembly import ReassemblyResult +from adc_analysis.sink import ParquetSink, _observations + +REPOSITORY = Path(__file__).resolve().parents[2] +CATALOG = REPOSITORY / "protocol" / "v1" / "collector-catalog.json" +PARTICIPANT = "00000000-0000-0000-0000-000000000001" + + +def _event( + sequence: int, + *, + collector_id: str = "app_lifecycle.v1", + payload_type: str = "ACTIVITY_CREATED", + boot_session_id: str = "boot", + fields: dict[str, object] | None = None, + monotonic_time_nanos: int | None = None, +) -> VerifiedEvent: + if fields is None: + fields = ( + { + "change_reason": "TIMEZONE_CHANGED", + "daylight_saving_time": False, + "timezone_id": "Asia/Taipei", + "utc_offset_seconds": 28_800, + } + if collector_id == "temporal_context.v1" + else {"activity_class": "Activity"} + ) + return VerifiedEvent( + "study-id", + "config-id", + PARTICIPANT, + None, + sequence, + collector_id, + 1, + payload_type, + boot_session_id, + sequence * 10 if monotonic_time_nanos is None else monotonic_time_nanos, + sequence * 20, + fields, + f"event-{sequence}".encode(), + EventProvenance("a" * 64, "bundle", "b" * 64, "memory:bundle"), + ) + + +class _Events: + def __init__( + self, + events: list[VerifiedEvent], + *, + boot_sessions: list[BootSession] | None = None, + sampling_groups: list[SamplingGroup] | None = None, + survey_counts: list[SurveyLifecycleCount] | None = None, + ): + self.events = events + self.boot_sessions = boot_sessions or [] + self.sampling_groups = sampling_groups or [] + self.survey_counts = survey_counts or [] + + def __iter__(self): + return iter(self.events) + + def __len__(self) -> int: + return len(self.events) + + def close(self) -> None: + pass + + def iter_partitioned(self): + return iter( + sorted( + self.events, + key=lambda event: ( + event.experiment_id, + event.configuration_id, + event.collector_id, + event.payload_schema_version, + event.payload_type, + event.participant_instance_id, + event.sequence_number, + ), + ) + ) + + def iter_boot_sessions(self): + return iter(self.boot_sessions) + + def iter_sampling_groups(self, source_clock_fields): + return iter(self.sampling_groups) + + def iter_survey_lifecycle_counts(self): + return iter(self.survey_counts) + + +class SinkTest(unittest.TestCase): + def test_all_p2_collectors_materialize_typed_parquet_fixtures(self) -> None: + fixtures = [ + ( + "battery_state.v1", + "BATTERY_STATE", + { + "charging_source": "USB", + "charging_state": "CHARGING", + "percentage": 73, + "power_save_enabled": True, + }, + ), + ( + "temporal_context.v1", + "TEMPORAL_CONTEXT", + { + "change_reason": "TIMEZONE_CHANGED", + "daylight_saving_time": False, + "timezone_id": "Asia/Taipei", + "utc_offset_seconds": 28_800, + }, + ), + ( + "gyroscope.v1", + "GYROSCOPE_SAMPLE", + { + "accuracy": 3, + "source_elapsed_realtime_nanos": 9_223_372_036, + "x_radians_per_second": 1.25, + "y_radians_per_second": -2.5, + "z_radians_per_second": 0.5, + }, + ), + ( + "ambient_light.v1", + "AMBIENT_LIGHT_SAMPLE", + { + "accuracy": 2, + "illuminance_lux": 321.5, + "source_elapsed_realtime_nanos": 9_223_372_037, + }, + ), + ( + "proximity.v1", + "PROXIMITY_SAMPLE", + { + "distance_centimeters": 1.5, + "maximum_range_centimeters": 5.0, + "near": True, + "source_elapsed_realtime_nanos": 9_223_372_038, + }, + ), + ] + events = [ + _event( + index, + collector_id=collector_id, + payload_type=payload_type, + fields=fields, + ) + for index, (collector_id, payload_type, fields) in enumerate(fixtures, start=1) + ] + result = ReassemblyResult((), _Events(events), {"format": "adc-quality-summary-v1"}, False) + + with tempfile.TemporaryDirectory() as temporary: + destination = Path(temporary) / "dataset" + ParquetSink(CollectorCatalog(CATALOG)).write(result, destination) + for collector_id, payload_type, expected in fixtures: + path = next( + ( + destination + / "experiment_id=study-id" + / "configuration_id=config-id" + / f"collector_id={collector_id}" + / "payload_schema_version=1" + / f"payload_type={payload_type}" + ).glob("*.parquet") + ) + table = pq.ParquetFile(path).read() + self.assertEqual(1, table.num_rows) + row = table.to_pylist()[0] + for name, value in expected.items(): + if isinstance(value, float): + self.assertAlmostEqual(value, row[name], places=5) + else: + self.assertEqual(value, row[name]) + self.assertEqual("a" * 64, row["source_ciphertext_sha256"]) + self.assertEqual("bundle", row["source_bundle_id"]) + self.assertEqual("b" * 64, row["source_configuration_sha256"]) + self.assertEqual("memory:bundle", row["source_object"]) + self.assertEqual("int64", str(table.schema.field("sequence_number").type)) + for name, descriptor in self.catalog_fields(collector_id, payload_type).items(): + self.assertEqual(descriptor, str(table.schema.field(name).type)) + + def test_quality_observations_are_partitioned_and_bounded(self) -> None: + events = [ + _event( + sequence, + collector_id="temporal_context.v1", + payload_type="TEMPORAL_CONTEXT", + boot_session_id=f"boot-{sequence:03d}", + ) + for sequence in range(1, 151) + ] + collection = _Events( + events, + boot_sessions=[ + BootSession("study-id", "config-id", PARTICIPANT, f"boot-{index:03d}") + for index in range(1, 151) + ], + sampling_groups=[ + SamplingGroup( + "study-id", + "config-id", + PARTICIPANT, + "gyroscope.v1", + f"boot-{index:03d}", + "source_elapsed_realtime_nanos", + index * 10, + index * 10, + 1, + ) + for index in range(1, 151) + ], + survey_counts=[ + SurveyLifecycleCount( + "study-id", + "config-id", + f"participant-{index:03d}", + "SURVEY_OPENED", + index, + ) + for index in range(1, 151) + ], + ) + + observations = _observations( + collection, + CollectorCatalog(CATALOG).sampling_clock_fields, + ) + + achieved = observations["achieved_sampling_observations"] + self.assertEqual("150", achieved["count"]) + self.assertEqual(100, len(achieved["examples"])) + self.assertTrue(achieved["examples_truncated"]) + boot_sessions = observations["boot_sessions"] + self.assertEqual("1", boot_sessions["count"]) + session_ids = boot_sessions["examples"][0]["boot_session_ids"] + self.assertEqual("150", session_ids["count"]) + self.assertEqual(100, len(session_ids["examples"])) + survey = observations["survey_lifecycle_counts"] + self.assertEqual("150", survey["count"]) + self.assertEqual("study-id", survey["examples"][0]["experiment_id"]) + self.assertEqual("config-id", survey["examples"][0]["configuration_id"]) + temporal = observations["temporal_context_events"] + self.assertEqual("150", temporal["count"]) + self.assertEqual("study-id", temporal["examples"][0]["experiment_id"]) + self.assertEqual("config-id", temporal["examples"][0]["configuration_id"]) + + def test_parquet_batches_obey_the_estimated_byte_cap(self) -> None: + events = [_event(sequence) for sequence in range(1, 4)] + collection = _Events( + events, + boot_sessions=[BootSession("study-id", "config-id", PARTICIPANT, "boot")], + sampling_groups=[ + SamplingGroup( + "study-id", + "config-id", + PARTICIPANT, + "app_lifecycle.v1", + "boot", + "source_elapsed_realtime_nanos", + 10, + 30, + 3, + ) + ], + ) + result = ReassemblyResult( + (), + collection, + {"format": "adc-quality-summary-v1"}, + False, + ) + with tempfile.TemporaryDirectory() as temporary: + destination = Path(temporary) / "dataset" + with patch( + "adc_analysis.sink.PARQUET_BATCH_MAX_ESTIMATED_BYTES", 1 + ): + ParquetSink(CollectorCatalog(CATALOG)).write(result, destination) + parquet = pq.ParquetFile(next(destination.rglob("*.parquet"))) + self.assertEqual(3, parquet.metadata.num_row_groups) + + def test_batched_gyroscope_rate_uses_hardware_source_timestamps(self) -> None: + catalog = CollectorCatalog(CATALOG) + with tempfile.TemporaryDirectory() as temporary: + database = EventDatabase(Path(temporary)) + for sequence in range(1, 52): + database.add( + _event( + sequence, + collector_id="gyroscope.v1", + payload_type="GYROSCOPE_SAMPLE", + monotonic_time_nanos=9_000_000_000, + fields={ + "accuracy": 3, + "source_elapsed_realtime_nanos": (sequence - 1) + * 20_000_000, + "x_radians_per_second": 0.1, + "y_radians_per_second": 0.2, + "z_radians_per_second": 0.3, + }, + ) + ) + database.finish_candidates() + for row in database.candidate_rows(): + database.accept(row) + events = database.seal() + self.addCleanup(events.close) + + observations = _observations(events, catalog.sampling_clock_fields) + + achieved = observations["achieved_sampling_observations"]["examples"] + self.assertEqual(1, len(achieved)) + self.assertEqual("1000000000", achieved[0]["duration_monotonic_nanos"]) + self.assertEqual("50", achieved[0]["sampling_interval_count"]) + self.assertEqual("50000", achieved[0]["mean_sampling_rate_millihertz"]) + self.assertEqual( + "source_elapsed_realtime_nanos", + achieved[0]["source_clock_field"], + ) + + @staticmethod + def catalog_fields(collector_id: str, payload_type: str) -> dict[str, str]: + schema = CollectorCatalog(CATALOG).payload(collector_id, 1, payload_type) + arrow_types = { + "boolean": "bool", + "decimal_string": "int64", + "enum": "string", + "float32": "float", + "float64": "double", + "int32": "int32", + "json_string": "string", + "string": "string", + } + return {name: arrow_types[str(field["type"])] for name, field in schema.fields.items()} + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/tests/test_streaming_filesystem.py b/adc-analysis/tests/test_streaming_filesystem.py new file mode 100644 index 0000000..4b6b6aa --- /dev/null +++ b/adc-analysis/tests/test_streaming_filesystem.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +import hashlib +import os +import tempfile +import unittest +from pathlib import Path + +from adc_analysis.errors import ValidationError +from adc_analysis.filesystem import private_directory, rename_noreplace +from adc_analysis.streaming_json import CanonicalJsonEvents + + +class StreamingAndFilesystemTest(unittest.TestCase): + def test_streaming_jcs_accepts_exact_bytes_and_rejects_other_encodings(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "document.json" + canonical = b'{"a":[true,"value"],"z":1}' + path.write_bytes(canonical) + events = list( + CanonicalJsonEvents( + path, hashlib.sha256(canonical).hexdigest(), len(canonical) + ) + ) + self.assertEqual(("", "start_map", None), events[0]) + + noncanonical = b'{"z":1, "a":[true,"value"]}' + path.write_bytes(noncanonical) + with self.assertRaises(ValidationError): + list( + CanonicalJsonEvents( + path, + hashlib.sha256(noncanonical).hexdigest(), + len(noncanonical), + ) + ) + + floating = b'{"value":1.0}' + path.write_bytes(floating) + with self.assertRaises(ValidationError): + list( + CanonicalJsonEvents( + path, hashlib.sha256(floating).hexdigest(), len(floating) + ) + ) + + def test_private_directory_is_tightened_and_publish_is_create_only(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + staging = root / "staging" + staging.mkdir(mode=0o755) + self.assertEqual(private_directory(staging), staging.resolve()) + if os.name == "posix": + self.assertEqual(0o700, staging.stat().st_mode & 0o777) + + source = root / "source" + destination = root / "destination" + source.mkdir() + destination.mkdir() + (source / "new").write_text("new") + (destination / "existing").write_text("existing") + with self.assertRaises(ValidationError): + rename_noreplace(source, destination) + self.assertTrue((source / "new").is_file()) + self.assertTrue((destination / "existing").is_file()) + + +if __name__ == "__main__": + unittest.main() diff --git a/adc-analysis/uv.lock b/adc-analysis/uv.lock new file mode 100644 index 0000000..e94f9e0 --- /dev/null +++ b/adc-analysis/uv.lock @@ -0,0 +1,424 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" + +[[package]] +name = "adc-analysis" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "boto3" }, + { name = "cryptography" }, + { name = "ijson" }, + { name = "pyarrow" }, +] + +[package.dev-dependencies] +dev = [ + { name = "ruff" }, +] + +[package.metadata] +requires-dist = [ + { name = "boto3", specifier = ">=1.40,<2" }, + { name = "cryptography", specifier = ">=48.0.1,<49" }, + { name = "ijson", specifier = ">=3.4,<4" }, + { name = "pyarrow", specifier = ">=21,<24" }, +] + +[package.metadata.requires-dev] +dev = [{ name = "ruff", specifier = ">=0.16,<0.17" }] + +[[package]] +name = "boto3" +version = "1.43.62" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "botocore" }, + { name = "jmespath" }, + { name = "s3transfer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b9/c7/f7732c5e1abf7270a6bbbce47338d25ea66a30df658cffd1d17bb5f735fb/boto3-1.43.62.tar.gz", hash = "sha256:0bf920e0739346e81c7310b685a3f783bf1fcc62ce7d5c7016508fa25c0d261f", size = 112668, upload-time = "2026-07-31T19:35:17.257Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7c/f0/5e1a392c817e395b140c18c12a00c0c65c69f8d63da26ad4387aebf2172b/boto3-1.43.62-py3-none-any.whl", hash = "sha256:0bb298e7ffd72b91615df44bf71c417df80a29d844971e5d665b8bd743a4bb35", size = 140025, upload-time = "2026-07-31T19:35:15.347Z" }, +] + +[[package]] +name = "botocore" +version = "1.43.62" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "jmespath" }, + { name = "python-dateutil" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4e/8d/36af6d99269a701f83809b87a01f4728699eb825ebdedee3a3d515b18f61/botocore-1.43.62.tar.gz", hash = "sha256:94efc419c9f0f41dc2415e4b6b62f04ae21b3ce3930fac47214c4d3f361ea8b8", size = 15818261, upload-time = "2026-07-31T19:35:06.235Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/65/d5dae96de68ffc55acf87c3bae76e9dabeeca92aadf1223f20e9a7860aef/botocore-1.43.62-py3-none-any.whl", hash = "sha256:76de153de1ba3e242b2e6df6a13ab8a3fb35d17db562462969e661457b63166e", size = 15502622, upload-time = "2026-07-31T19:35:02.697Z" }, +] + +[[package]] +name = "cffi" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pycparser", marker = "implementation_name != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/57/5f/ff100cae70ebe9d8df1c01a00e510e45d9adb5c1fdda84791b199141de97/cffi-2.1.0.tar.gz", hash = "sha256:efc1cdd798b1aaf39b4610bba7aad28c9bea9b910f25c784ccf9ec1fa719d1f9", size = 531036, upload-time = "2026-07-06T21:34:30.382Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d3/67/85c89a59ba36a671e79638f44d466749f08179266a57e4f2ffdf92174072/cffi-2.1.0-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:02cb7ff33ded4f1532476731f89ede53e2e488a8e6205515a82144246ffa7dcc", size = 183845, upload-time = "2026-07-06T21:32:26.32Z" }, + { url = "https://files.pythonhosted.org/packages/ea/dd/e3b0baa2d3d6a857ac72b7efbf18e32e487c9cdafcc13049ad765495b15e/cffi-2.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f5bce581e6b8c235e566a14768a943b172ada3ed73537bb0c0be1edee312d4e7", size = 184186, upload-time = "2026-07-06T21:32:28.025Z" }, + { url = "https://files.pythonhosted.org/packages/65/68/9f3ef890cf3c6ab97bd531c5677f67613d302165d16f8142b2811782a614/cffi-2.1.0-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:30b65779d598c370374fefabf138d456fd6f3216bfa7bedfab1ba82025b0cd93", size = 211892, upload-time = "2026-07-06T21:32:29.565Z" }, + { url = "https://files.pythonhosted.org/packages/22/d7/1a74539db16d8bfd839ff1515948948efbb162e574650fd3d846896eea95/cffi-2.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88023dfe18799507b73f1dbb0d14326a17465de1bc9c9c7655c22845e9ddc3a2", size = 218793, upload-time = "2026-07-06T21:32:30.951Z" }, + { url = "https://files.pythonhosted.org/packages/ec/d1/9a5b7169499e8e8d8e636de70b97ac7c9447104d2ff1a2cd94790cea5162/cffi-2.1.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:0a96b74cda968eebbad56d973efe5098974f0a9fb323865bf99ea1fd24e3e64c", size = 205737, upload-time = "2026-07-06T21:32:32.216Z" }, + { url = "https://files.pythonhosted.org/packages/ba/b0/e131a9c41f10607926278453d9596163594fe1c4ebc46efe3b5e5b34eb84/cffi-2.1.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a5781494d4d400a3f47f8f1da94b324f6e6b440a53387774002890a2a2f4b50f", size = 204909, upload-time = "2026-07-06T21:32:33.655Z" }, + { url = "https://files.pythonhosted.org/packages/fb/d2/4398416cd699b35167947c6e22aca52c47e69ad5695073c9f1f2c52e04aa/cffi-2.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aa7a1b53a2a4452ada2d1b5dade9960b2522f1e61293a811a077439e39029565", size = 217883, upload-time = "2026-07-06T21:32:35.173Z" }, + { url = "https://files.pythonhosted.org/packages/a2/a5/d4fe77b589e5e82d43ebc809bf2e6474afe8e48e32ea050b9357645b6471/cffi-2.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9d8272c0e483b024e1b9ad029821470ed8ec65631dbd90217469da0e7cd89f1c", size = 221251, upload-time = "2026-07-06T21:32:36.527Z" }, + { url = "https://files.pythonhosted.org/packages/22/f0/a2fc43084c0433caf7f461bccc013e28f848d04ee1c5ed7fce71423cf4d9/cffi-2.1.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:7762faa47e8ff7eb80bd261d9a7d8eea2d8baa69de5e95b70c1f338bbe712f02", size = 214250, upload-time = "2026-07-06T21:32:37.852Z" }, + { url = "https://files.pythonhosted.org/packages/04/8c/b925975448cf20634a9fbd5efceb807219db452653648d2897c0989cab2d/cffi-2.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:89095c1968b4ba8285840e131bf2891b09ae137fe2146905acae0354fbce1b5e", size = 219441, upload-time = "2026-07-06T21:32:39.146Z" }, + { url = "https://files.pythonhosted.org/packages/eb/da/5c4918a2d61d86fa927d716cb3d8e4626ef8dc8f605a599d32f33897f59a/cffi-2.1.0-cp311-cp311-win32.whl", hash = "sha256:64c753a0f87a256020004f37a1c8c02c480e725f910f0b2a0f3f07debd1b2479", size = 174496, upload-time = "2026-07-06T21:32:40.467Z" }, + { url = "https://files.pythonhosted.org/packages/f9/c8/6c2de1d55cf35ef8b92885d5ef280790f0fb9634d87ea1cc315176aecd61/cffi-2.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:4f26194e3d95e06501b942642855aed4f953d55e95d7d01b7c4483db3ecff458", size = 185113, upload-time = "2026-07-06T21:32:41.761Z" }, + { url = "https://files.pythonhosted.org/packages/9e/4e/e8d7cb5783f1841a3c8fb3a7735838d7484d08ec08c9f984b14cac1ac0e9/cffi-2.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:35aaea0c7ee0e58a5cd8c2fd1a48fdf7ece0d2699b7ecdda08194e9ce5dd9b3d", size = 179927, upload-time = "2026-07-06T21:32:42.961Z" }, + { url = "https://files.pythonhosted.org/packages/1e/85/990925db5df586ec90beb97529c853497e7f85ba0234830447faf41c3057/cffi-2.1.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:df2b82571a1b30f58a87bf4e5a9e78d2b1eff6c6ce8fd3aa3757221f93f0863f", size = 184829, upload-time = "2026-07-06T21:32:44.324Z" }, + { url = "https://files.pythonhosted.org/packages/4b/92/e7bb136ad6b5352603732cf907ef862ca103f20f2031c1735a46300c20c9/cffi-2.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:78474632761faa0fb96f30b1c928c84ebcf68713cbb80d15bab09dfe61640fde", size = 184728, upload-time = "2026-07-06T21:32:45.683Z" }, + { url = "https://files.pythonhosted.org/packages/c3/c0/d1ec30ffb370f748f2fb54425972bfef9871e0132e82fb589c46b6676049/cffi-2.1.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:5972433ad71a9e46516584ef60a0fda12d9dc459938d1539c3ddecf9bdc1368d", size = 214815, upload-time = "2026-07-06T21:32:48.557Z" }, + { url = "https://files.pythonhosted.org/packages/1b/dc/5620cf930688be01f2d673804291de757a934c90b946dbdc3d84130c2ea4/cffi-2.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b6422532152adf4e59b110cb2808cee7a033800952f5c036b4af047ee43199e7", size = 222429, upload-time = "2026-07-06T21:32:49.848Z" }, + { url = "https://files.pythonhosted.org/packages/4b/a4/77b53abbf7a1e0beb9637edbef2a94d15f9c822f591e85d439ffd91519a6/cffi-2.1.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:46b1c8db8f6122420f32d02fffb924c2fe9bc772d228c7c711748fff56aabb2b", size = 210315, upload-time = "2026-07-06T21:32:51.221Z" }, + { url = "https://files.pythonhosted.org/packages/58/0c/f528df19cc94b675087324d4760d9e6d5bfae97d6217aa4fac43de4f5fcc/cffi-2.1.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9fafc5aa2e2a39aaf7f8cc0c1f044a9b07fca12e558dca53a3cc5c654ad67a7", size = 208859, upload-time = "2026-07-06T21:32:52.512Z" }, + { url = "https://files.pythonhosted.org/packages/62/f2/c9522a81c32132799a1972c39f5c5f8b4c8b9f00488a23feaa6c06f07741/cffi-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1e9f50d192a3e525b15a75ab5114e442d83d657b7ec29182a991bc9a88fd3a66", size = 221844, upload-time = "2026-07-06T21:32:53.704Z" }, + { url = "https://files.pythonhosted.org/packages/6e/28/bd53988b9833e8f8ad539d26f4c07a6b3f6bcb1e9e02e7ca038250b3428d/cffi-2.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:98fff996e983a36d3aa2eca83af40c5821202e7e6f32d13ae94e3d2286f10cfe", size = 225287, upload-time = "2026-07-06T21:32:54.907Z" }, + { url = "https://files.pythonhosted.org/packages/79/99/0d0fd37f055224085f42bbb2c022d002e17dde4a97972822327b07d84101/cffi-2.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:379de10ce1ba048b1448599d1b37b24caee16309d1ac98d3982fc997f768700b", size = 223681, upload-time = "2026-07-06T21:32:56.329Z" }, + { url = "https://files.pythonhosted.org/packages/b0/80/c138990aa2a70b1a269f6e06348729836d733d6f970867943f61d367f8cc/cffi-2.1.0-cp312-cp312-win32.whl", hash = "sha256:9b8f0f26ca4e7513c534d351eca551947d053fac438f2a04ac96d882909b0d3a", size = 175269, upload-time = "2026-07-06T21:32:57.777Z" }, + { url = "https://files.pythonhosted.org/packages/a8/eb/f636456ff21a83fc13c032b58cc5dde061691546ac79efa284b2989b7982/cffi-2.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:c97f080ea627e2863524c5af3836e2270b5f5dfff1f104392b959f8df0c5d384", size = 185881, upload-time = "2026-07-06T21:32:59.253Z" }, + { url = "https://files.pythonhosted.org/packages/dd/2c/400ea43e721727dca8a65c4521390e9196757caba4a45643acb2b63271b8/cffi-2.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:6d194185eabd279f1c05ebe3504265ddfc5ad2b58d0714f7db9f01da592e9eb6", size = 180088, upload-time = "2026-07-06T21:33:02.278Z" }, + { url = "https://files.pythonhosted.org/packages/96/88/a996879e2eeccb815f6e3a5967b12a308257412acec882039d386bd2aa7b/cffi-2.1.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:10537b1df4967ca26d21e5072d7d54188354483b91dc75058968d3f0cf13fbda", size = 194331, upload-time = "2026-07-06T21:33:03.697Z" }, + { url = "https://files.pythonhosted.org/packages/58/85/7ae00d5c8dd6266f4e944c3db630f3c5c9a98b61d469c714d848b1d8138a/cffi-2.1.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:a95b05f9baf29b91171b3a8bd2020b028835243e7b0ff6bb23e2a3c228518b1b", size = 196966, upload-time = "2026-07-06T21:33:05.353Z" }, + { url = "https://files.pythonhosted.org/packages/8c/e9/45c3a76ad8d43ad9261f4c95436da61128d3ca545d72b9612c0ab5be0b1c/cffi-2.1.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:15faec4adfff450819f3aee0e2e02c812de6edb88203aa58807955db2003472a", size = 184795, upload-time = "2026-07-06T21:33:06.699Z" }, + { url = "https://files.pythonhosted.org/packages/84/4c/82f132cb4418ee6d953d982b19191e87e2a6372c8a4ce36e50b69d6ade4a/cffi-2.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:716ff8ec22f20b4d988b12884086bcef0fc99737043e503f7a3935a6be99b1ea", size = 184746, upload-time = "2026-07-06T21:33:08.071Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1c/4ed5a0e5bdca6cbc275556de3328dd1b76fd0c11cc13c88fe66d1d8715f2/cffi-2.1.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:63960549e4f8dc41e31accb97b975abaecfc44c03e396c093a6436763c2ea7db", size = 214747, upload-time = "2026-07-06T21:33:09.671Z" }, + { url = "https://files.pythonhosted.org/packages/3a/a6/e879bb68cc23a2bc9ba8f4b7d8019f0c2694bad2ab6c4a3701d429439f58/cffi-2.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ff067a8d8d880e7809e4ac88eb009bb848870115317b306666502ccad30b147f", size = 222392, upload-time = "2026-07-06T21:33:10.896Z" }, + { url = "https://files.pythonhosted.org/packages/88/f6/01890cfd63c08f8eb96a8319b0443690197d240a8bd6346048cf7bde9190/cffi-2.1.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:3b926723c13eba9f81d2ef3820d63aeceec3b2d4639906047bf675cb8a7a500d", size = 210285, upload-time = "2026-07-06T21:33:12.251Z" }, + { url = "https://files.pythonhosted.org/packages/a6/cf/2b684132056f438567b61e19d690dd31cd0921ace051e0a458be6074369e/cffi-2.1.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:47ff3a8bfd8cb9da1af7524b965127095055654c177fcfc7578debcb015eecd0", size = 208801, upload-time = "2026-07-06T21:33:13.617Z" }, + { url = "https://files.pythonhosted.org/packages/6f/08/f2e7d62c460faae0926f2d6e423694aa409ced3bc1fe2927a0a6e5f05416/cffi-2.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:799416bae98336e400981ff6e532d67d5c709cfb30afb79865a1315f94b0e224", size = 221808, upload-time = "2026-07-06T21:33:15.466Z" }, + { url = "https://files.pythonhosted.org/packages/38/37/04f54b8e63a02f3d908332c9effbf8c366167c6f733ed8a3d4f79b7e2a1e/cffi-2.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:961be50688f7fba2fa65f63712d3b9b341a22311f5253460ce933f52f0de1c8c", size = 225241, upload-time = "2026-07-06T21:33:16.869Z" }, + { url = "https://files.pythonhosted.org/packages/a9/d6/c72eecca433cd3e681c65ed313ab4835d9d4a379704d0f628a6a05f51c2e/cffi-2.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:bf5c6cf48238b0eb4c086978c492ad1cbc22373fc5b2d7353b3a598ce6db887a", size = 223588, upload-time = "2026-07-06T21:33:18.239Z" }, + { url = "https://files.pythonhosted.org/packages/c6/4b/e706f67279140f92939da3475ad610df18bfd52d50f14953a8e5fede71d5/cffi-2.1.0-cp313-cp313-win32.whl", hash = "sha256:db3eb7d46527159a878ec3460e9d40615bc25ba337d477db681aea6e4f05c5d2", size = 175248, upload-time = "2026-07-06T21:33:19.799Z" }, + { url = "https://files.pythonhosted.org/packages/5a/47/59eb7975cb0e4ef0afa764ea945b29a5bb4537a9f771cb7d6c8a5dd74c95/cffi-2.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:8e74a6135550c4748af665b1b1118b6aab33b1fc6a16f9aff630af107c3b4512", size = 185717, upload-time = "2026-07-06T21:33:21.47Z" }, + { url = "https://files.pythonhosted.org/packages/5a/af/34fee85c48f8d94efc8597bc09470c9dd274c145f1c12e0fbc6ab6d38d74/cffi-2.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:2282cd5e38aa8accd03e99d1256af8411c84cdbee6a89d841b563fdbd1f3e50f", size = 180114, upload-time = "2026-07-06T21:33:22.515Z" }, + { url = "https://files.pythonhosted.org/packages/d8/f0/81478e482afa03f6d18dc8f2afb5edc45b3080853b634b5ed91961be0998/cffi-2.1.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:d2117334c3af3bdcb9a88522b844a2bdb5efdc4f71c6c822df55486ae1c3347a", size = 194142, upload-time = "2026-07-06T21:33:23.657Z" }, + { url = "https://files.pythonhosted.org/packages/7d/95/8de304305cd9204974b0ca051b86d307cafca13aa575a0ef1b44d92c0d8c/cffi-2.1.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:702c436735fbe99d59ada02a1f65cfc0d31c0ee8b7290912f8fbc5cd1e4b16c3", size = 196819, upload-time = "2026-07-06T21:33:25.007Z" }, + { url = "https://files.pythonhosted.org/packages/20/71/7c8372d30e42415602ed9f268f7cfd66f1b855fed881ecd168bcb45dbc0b/cffi-2.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:1ff3456eab0d889592d1936d6125bbfbc7ae4d3354a700f8bd80450a66445d4d", size = 184965, upload-time = "2026-07-06T21:33:26.605Z" }, + { url = "https://files.pythonhosted.org/packages/d6/5c/584e626835f0375c928176c04137c96927165cb8733cdb3150ec04e5ee5e/cffi-2.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c4165821e131d6d4ca444347c2b694e2311bcfa3fe5a861cc72968f28867beac", size = 184952, upload-time = "2026-07-06T21:33:27.823Z" }, + { url = "https://files.pythonhosted.org/packages/2e/d2/065fcae1c73979fac8e054462478d0ff8a29c40cdc2ed7ea5676a061df53/cffi-2.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:276f20fffd7b396e12516ba8edf9509210ac248cbbc5acbc39cd512f9f59ebe6", size = 222353, upload-time = "2026-07-06T21:33:29.178Z" }, + { url = "https://files.pythonhosted.org/packages/ed/a5/e8bbb1ce5b3ac2f53ad6a10bde44318a5a8d99d4f4a000d44a6e39aeb3e4/cffi-2.1.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7d5980a3433d4b71a5e120f9dd551403d7824e31e2e67124fe2769c404c06913", size = 210051, upload-time = "2026-07-06T21:33:30.534Z" }, + { url = "https://files.pythonhosted.org/packages/28/ed/c127d3ac36e899c965e3361357c3befacd6578c03f40125183e41c3b219e/cffi-2.1.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6ca4919c6e4f89aa99c42510b42cf54596892c00b3f9077f6bdd1505e24b9c8d", size = 208630, upload-time = "2026-07-06T21:33:31.753Z" }, + { url = "https://files.pythonhosted.org/packages/cc/d7/97d3136f81db489ec8d1d67748c110d6c994268fd7528014aa9f2b085e4e/cffi-2.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d53d10f7da99ae46f7373b9150393e9c5eab9b224909982b43832668de4779f5", size = 221593, upload-time = "2026-07-06T21:33:33.044Z" }, + { url = "https://files.pythonhosted.org/packages/d3/27/93195977168ee63aed233a1a0993a2178798654d1f4bddcdd321d6fd3b21/cffi-2.1.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c351efb95e832a853a29361675f33a7ce53de1a109cd73fd47af0712213aa4ce", size = 225146, upload-time = "2026-07-06T21:33:34.224Z" }, + { url = "https://files.pythonhosted.org/packages/b3/c1/6dbd291ee2ae5a50a034aa057207081f545923bbf15dad4511e985aafff5/cffi-2.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:dbf7c7a88e2bac086f06d14577332760bdeecc42bdec8ac4077f6260557d9326", size = 223240, upload-time = "2026-07-06T21:33:35.57Z" }, + { url = "https://files.pythonhosted.org/packages/0f/6f/ade5ce9863a57992a6ea3d0d10d7e29b8749fc127204b3d493d667b2815f/cffi-2.1.0-cp314-cp314-win32.whl", hash = "sha256:1854b724d00f6654c742097d5387569021be12d3a0f770eae1df8f8acfcc6acd", size = 177723, upload-time = "2026-07-06T21:33:51.626Z" }, + { url = "https://files.pythonhosted.org/packages/41/de/92b9eeed4ae4a21d6fd9b2a2c8505cbed573299902ea73981cc13f7ff62c/cffi-2.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb", size = 187937, upload-time = "2026-07-06T21:33:53.403Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1a/cc6ae6c2913a03aab8898eee57963cf1035b8df5872ed8b9115fcc7e2be8/cffi-2.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:7d28dff1db6764108bc30788d85d61c876beff416d9a49cb9dd7c5a9f34f5804", size = 183001, upload-time = "2026-07-06T21:33:54.74Z" }, + { url = "https://files.pythonhosted.org/packages/14/f0/134c00ce0779ec86dea2aa1aac69339c2741a8045072676763512363a2ea/cffi-2.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7ea6b3e2c4250ff1de21c630fe72d0f63eb95c2c32ffbf64a358cf4a8836d714", size = 188538, upload-time = "2026-07-06T21:33:36.792Z" }, + { url = "https://files.pythonhosted.org/packages/50/d8/3b86aba791cb610d24e8a3e1b2cd529e71fa15096b04e4d4e360049d4a4c/cffi-2.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6af371f3767faeffc6ac1ef57cdfd25844403e9d3f476c5537caee499de96376", size = 188230, upload-time = "2026-07-06T21:33:38.011Z" }, + { url = "https://files.pythonhosted.org/packages/14/d0/117dcd9209255ad8571fbc8c92ef32593a1d294dcec91ddc4e4db50606f2/cffi-2.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb4e8997a49aa2c08a3e43c9045d224448b8941d88e7ac163c7d383e560cbf98", size = 223899, upload-time = "2026-07-06T21:33:39.514Z" }, + { url = "https://files.pythonhosted.org/packages/b6/3d/f20f8b886b254e3ad10e15cd4186d3aed49f3e6a35ab37aab9f8f25f7c03/cffi-2.1.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:bf01d8c84cbea96b944c73b22182e6c7c432b3475632b8111dbfdc95ddad6e13", size = 211652, upload-time = "2026-07-06T21:33:40.851Z" }, + { url = "https://files.pythonhosted.org/packages/28/3b/fad54de07260b93ddeef4b96d0131d57ea900675df1d410ae1deee52d7a6/cffi-2.1.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:33eb1ad83ebe8f313e0df035c406227d55a79456704a863fad9842136af5ad7d", size = 210755, upload-time = "2026-07-06T21:33:42.183Z" }, + { url = "https://files.pythonhosted.org/packages/cc/82/3d5c705acb7abbba9bbd7d79b8e62e0f25b6120eb7ae6ac49f1b721722fe/cffi-2.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ac0f1a2d0cfa7eea3f2aaf006ab6e70e8feeb16b75d65b7e5939982ca2f11056", size = 223933, upload-time = "2026-07-06T21:33:43.603Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d0/47e338384ab6b1004241002fa616301020cea4fc95f283506565d252f276/cffi-2.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c16914df9fb7f500e440e6875fa23ff5e0b31db01fa9c06af98d59a91f0dc2e4", size = 226749, upload-time = "2026-07-06T21:33:45.046Z" }, + { url = "https://files.pythonhosted.org/packages/70/25/65bd5b58ea4bfdfc15cde02cb5365f89ef8ab8b2adfb8fe5c4bd4233382f/cffi-2.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5ecbd0499275d57506d397eebe1981cee87b47fcd9ef5c22cab7ed7644a39a94", size = 225703, upload-time = "2026-07-06T21:33:46.374Z" }, + { url = "https://files.pythonhosted.org/packages/dc/78/aa01ac599a8a4322533d45a1f9bc93b338276d2d59dabbe7c6d92a775c81/cffi-2.1.0-cp314-cp314t-win32.whl", hash = "sha256:7d034dcffa09e9a46c93fa3a3be402096cb5354ac6e41ab8e5cc9cd8b642ad76", size = 182857, upload-time = "2026-07-06T21:33:47.696Z" }, + { url = "https://files.pythonhosted.org/packages/b9/26/d00496b22de4d4228f32dde94ad996f350c8aad676d63bcca0743c8dea4d/cffi-2.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0582a58f3051372229ca8e7f5f589f9e5632678208d8636fea3676711fdf7fe5", size = 194065, upload-time = "2026-07-06T21:33:48.953Z" }, + { url = "https://files.pythonhosted.org/packages/d5/dd/0c7dbf815a579ff005008a2d815a55d6bb047c349eef536d9dc53d3f0a8d/cffi-2.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:510aeeeac94811b138077451da1fb18b308a5feab47dd2b603af55804155e1c8", size = 186404, upload-time = "2026-07-06T21:33:50.309Z" }, + { url = "https://files.pythonhosted.org/packages/55/c7/8c8c50cb11c6750051daf12164098a9a6f027ac4356967fd4d800a07f242/cffi-2.1.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:2e9dabb9abcb7ad15938c7196ad5c1718a4e6d33cc79b4c0209bdb64c4a54a5c", size = 194121, upload-time = "2026-07-06T21:33:56.109Z" }, + { url = "https://files.pythonhosted.org/packages/99/e2/67680bf19a6b60d2bb7ff83baefa2a4c3d2d7dc0f3277034b802e1fc504c/cffi-2.1.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:37f525a7e7e50c017fdebe58b787be310ad59357ae43a053943a6e1a6c526001", size = 196820, upload-time = "2026-07-06T21:33:57.288Z" }, + { url = "https://files.pythonhosted.org/packages/ed/da/4bbe583a3b3a5c8c60892124fe17f3fa3656523faf0d3484eae90f091853/cffi-2.1.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:95f2954c2c9473d892eca6e0409f3568b37ab62a8eedb122461f73cc273476e3", size = 184936, upload-time = "2026-07-06T21:33:58.765Z" }, + { url = "https://files.pythonhosted.org/packages/e5/4b/1f4c36ab273980d7aa75bb126ea4f8971f24a96108acad3a0a084028c57b/cffi-2.1.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:cdf2448aab5f661c9315308ec8b93f4e8a1a67a3c733f8631067a2b67d5913dc", size = 185045, upload-time = "2026-07-06T21:34:00.085Z" }, + { url = "https://files.pythonhosted.org/packages/ef/c3/ad299dc38f3583f8d916b299f028af418a9ec98bc695fcbebeae7420691c/cffi-2.1.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:90bec57cf82089383bd06a605b3eb8daebf7e5a668520beaf6e327a83a947699", size = 222342, upload-time = "2026-07-06T21:34:01.814Z" }, + { url = "https://files.pythonhosted.org/packages/eb/d8/df4543cc087245044ed02ef3ad8e0a26619d0075ac7a77a12dc81177851b/cffi-2.1.0-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6274dcb2d15cef48daa73ed1be5a40d501d74dccd0cd6db364776d12cb6ba022", size = 210073, upload-time = "2026-07-06T21:34:03.255Z" }, + { url = "https://files.pythonhosted.org/packages/2c/0e/fac738d73728c6cea2a88a2883dca54892496cbba88a1dc1f2909cb8a6f5/cffi-2.1.0-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:2b71d409cccee78310ab5dec549aed052aaea483346e282c7b02362596e01bb0", size = 208551, upload-time = "2026-07-06T21:34:04.433Z" }, + { url = "https://files.pythonhosted.org/packages/e6/3f/0b04a700dd64f465c93020253a793a82c9b4dff9961f48facd0df945d9b8/cffi-2.1.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7d3538f9c0e50670f4deb93dbb696576e60590369cae2faf7de681e597a8a1f1", size = 221649, upload-time = "2026-07-06T21:34:06.157Z" }, + { url = "https://files.pythonhosted.org/packages/5d/7c/b7379a5704c79eda57ce075869ba70a0368d1c850f803b3c0d078d39dcaf/cffi-2.1.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:8f9ec95b8a043d3dfbc74d9abc6f7baf524dd27a8dc160b0a32ff9cdab650c28", size = 225203, upload-time = "2026-07-06T21:34:07.489Z" }, + { url = "https://files.pythonhosted.org/packages/5a/02/d5e6c43ea85c41bda2a184a3418f195fe7cf602967a8d2b94e085b83deef/cffi-2.1.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:af5e2915d41fe6c961694d7bfdc8562942638200f3ce2765dfb8b745cf997629", size = 223263, upload-time = "2026-07-06T21:34:08.712Z" }, + { url = "https://files.pythonhosted.org/packages/2c/d8/772b8259bf75749adffb1c546828978381fb516f60cf701f6c83daf60c85/cffi-2.1.0-cp315-cp315-win32.whl", hash = "sha256:0a42c688d19fca6e095a53c6a6e2295a5b050a8b289f109adab02a9e61a25de6", size = 177696, upload-time = "2026-07-06T21:34:26.355Z" }, + { url = "https://files.pythonhosted.org/packages/2f/dd/afa2191fc6d57fedd26e5844a2fe2fcc0bbfa00961bbaa5a41e4921e7cca/cffi-2.1.0-cp315-cp315-win_amd64.whl", hash = "sha256:bccbbb5ee76a61f9d99b5bf3846a51d7fca4b6a732fe46f89295610edaf41853", size = 187914, upload-time = "2026-07-06T21:34:27.58Z" }, + { url = "https://files.pythonhosted.org/packages/05/ef/6cd4f8c671517162379dc79cfae5aea9106bc38abb89628d5c16adf6a838/cffi-2.1.0-cp315-cp315-win_arm64.whl", hash = "sha256:8d35c139744adb3e727cd51b1a18324bbe44b8bd41bf8322bca4d41289f48eda", size = 183004, upload-time = "2026-07-06T21:34:28.905Z" }, + { url = "https://files.pythonhosted.org/packages/11/b6/12fc55092817a5faa26fb8c40c7f9d662e11a46ee248c137aafc42517d92/cffi-2.1.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:f9912624a0c0b834b7520d7769b3644453aabc0a7e1c839da7359f050750e9bc", size = 188378, upload-time = "2026-07-06T21:34:09.926Z" }, + { url = "https://files.pythonhosted.org/packages/8d/2e/cdac88979f295fde5daa69622c7d2111e56e7ceb94f211357fbe452339e4/cffi-2.1.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:df92f2aba50eb4d96718b68ef76f2e57a57b54f2fa62333496d16c6d585a85ca", size = 188319, upload-time = "2026-07-06T21:34:11.101Z" }, + { url = "https://files.pythonhosted.org/packages/e0/27/1d0b408497e41a74795af122d7b603c418c5fed0171450f899afd04e594f/cffi-2.1.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0520e1f4c35f44e209cbbb421b67eec42e6a157f59444dfb6058874ff3610e5d", size = 223904, upload-time = "2026-07-06T21:34:12.606Z" }, + { url = "https://files.pythonhosted.org/packages/8b/31/e115c985105dd7ffb32444505f18ceb874bb42d992af05d5dced7ecf1980/cffi-2.1.0-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:3681e031db29958a7502f5c0c9d6bbc4c36cb20f7b104086fa642d1799631ff8", size = 211554, upload-time = "2026-07-06T21:34:13.987Z" }, + { url = "https://files.pythonhosted.org/packages/5a/67/9e6e09409336d9e515c58367e7cfcf4f89df06ad25252675595a58eb59d5/cffi-2.1.0-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:762f99479dcb369f60ab9017ad4ab97a36a1dd7c1ee5a3b15db0f4b8659120cd", size = 210795, upload-time = "2026-07-06T21:34:15.972Z" }, + { url = "https://files.pythonhosted.org/packages/19/e5/d3cc82a4a0be7902af279c04181ad038449c096734464a5ae1de3e1401bd/cffi-2.1.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0611e7ebf90573a535ebdc33ae9da222d037853983e13359f580fab781ca017f", size = 223843, upload-time = "2026-07-06T21:34:17.509Z" }, + { url = "https://files.pythonhosted.org/packages/b9/65/b434abc97ce7cecc2c640fde160507c0ecc7e21544b483ba3325d2e2ea17/cffi-2.1.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:86cf8755a791f72c85dc287128cc62d4f24d392e3f1e15837245623f4a33cccc", size = 226773, upload-time = "2026-07-06T21:34:19.05Z" }, + { url = "https://files.pythonhosted.org/packages/b5/9f/d4dc66ca651eb1145a133314cda721abf13cfac3d28c4a0402263ae6ad75/cffi-2.1.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:ba00f661f8ba35d075c937174e27c2c421cec3942fd2e0ea3e66996757c0fdd9", size = 225719, upload-time = "2026-07-06T21:34:20.576Z" }, + { url = "https://files.pythonhosted.org/packages/68/5a/e536c528bc8057496c360c0978559a2dc45653f89dd6151078aa7d8fca1a/cffi-2.1.0-cp315-cp315t-win32.whl", hash = "sha256:cb96698e3c7413d906ce83f8ffd245ec1bd94707541f299d0ce4d6b0193e982b", size = 182760, upload-time = "2026-07-06T21:34:22.059Z" }, + { url = "https://files.pythonhosted.org/packages/d3/0b/0ffe8b82d3875bced5fa1e7986a7a46b748262a40ab7f60b475eb9fb1bb3/cffi-2.1.0-cp315-cp315t-win_amd64.whl", hash = "sha256:f146d154428a2523f9cc7936c02353c2459b8f6cf07d3cd1ee1c0a611109c5d5", size = 193769, upload-time = "2026-07-06T21:34:23.589Z" }, + { url = "https://files.pythonhosted.org/packages/a0/17/1073b53b68c9b5ca6914adf5f8bf55aacc2d3be102418c90700160ea8605/cffi-2.1.0-cp315-cp315t-win_arm64.whl", hash = "sha256:cbb7640ce37159548d2147b5b8c241f962143d4c71231431820783f4dc78f210", size = 186405, upload-time = "2026-07-06T21:34:24.857Z" }, +] + +[[package]] +name = "cryptography" +version = "48.0.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/12/45/870e7f4bef50e5f53b9f51d4428aee5290eedf58ba443f16b1ebb7ab8e66/cryptography-48.0.1.tar.gz", hash = "sha256:266f4ee051abb2f725b74ef8072b521ce1feacf685a3364fa6a6b45548db791a", size = 832989, upload-time = "2026-06-09T22:32:31.8Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1b/bc/ee4137cbbe105652c0ee4252792b78fc8e7afa4b8e61d9d5dc05a7f45731/cryptography-48.0.1-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:3e4a1a3232eef2e6c732827d5722db29a0cc8b27af2a4d865b094cf954be9ca1", size = 8008324, upload-time = "2026-06-09T22:31:00.702Z" }, + { url = "https://files.pythonhosted.org/packages/d5/85/6379d42181bfc713094f081360fc5784d6c816b599d45e7f082502d173ce/cryptography-48.0.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:32143b24adb918f078134e1e230f1eb8cc04886b92c28b5f0041aaf3e5699225", size = 4696243, upload-time = "2026-06-09T22:32:33.446Z" }, + { url = "https://files.pythonhosted.org/packages/9c/87/c85d147b53323c7eb4d850920c8901377323c2a0ff8d79c262d4fee89aa2/cryptography-48.0.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f0d27a5696721ef7a672b8c810f6aded391058e0b9486e63e6d93baf765da691", size = 4713235, upload-time = "2026-06-09T22:31:40.141Z" }, + { url = "https://files.pythonhosted.org/packages/79/58/67cbf8cf1ee7c54b439ca07bbecf8362c07afc11a3724fea70f745784add/cryptography-48.0.1-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:eb86ce1af36fe65041b6db9a8bb064ee621a7e5fded0f80d475ec243477cd242", size = 4702323, upload-time = "2026-06-09T22:31:42.191Z" }, + { url = "https://files.pythonhosted.org/packages/89/c6/24266ac10c47f6cd2a865f4446062b466da1d1f10b27189eac00e61bf0c9/cryptography-48.0.1-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:b024e784ad6c077ee0147b35ea9cbfc1e34e1fd4c1dcca214c2794d73a12df08", size = 5300085, upload-time = "2026-06-09T22:31:58.703Z" }, + { url = "https://files.pythonhosted.org/packages/d2/bb/cc4b78784f97efc8c5874c2a9743708d172be6663024b34a0467885ae0c8/cryptography-48.0.1-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:3752f2dbc8f07a30aad2932c986cea495b03bb554887828225da104f732852b6", size = 4746137, upload-time = "2026-06-09T22:31:31.01Z" }, + { url = "https://files.pythonhosted.org/packages/1f/52/0c44de3f5267f8fbe8e835138017522a333436166e406f0db9b9e6e3033f/cryptography-48.0.1-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:bd81490cd5801d755cf97bb68ac191f14b708470b1c7cf4580f669b9c9264cd8", size = 4333867, upload-time = "2026-06-09T22:32:28.096Z" }, + { url = "https://files.pythonhosted.org/packages/9a/2e/772d7adbfa931537bc401640b7cac9976bff689bda187833e5d63b428e49/cryptography-48.0.1-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:66fd0771e7b9c6dcd44cf1120690d2338d16d72795cf40cae2786a39eba65429", size = 4701805, upload-time = "2026-06-09T22:31:38.284Z" }, + { url = "https://files.pythonhosted.org/packages/f8/a3/b06844f303873493c963caf581c04df31c7035e0c1b0f02c4814d319ec80/cryptography-48.0.1-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:3fd2ca57062b241c856670b073487d2e86c4637937ca5601e48f97bf8e11fc8f", size = 5258461, upload-time = "2026-06-09T22:31:04.187Z" }, + { url = "https://files.pythonhosted.org/packages/9f/13/8b765e2e12b07c74941caadb9d1c8fdc006c4dfbf2b8f2d610519758954d/cryptography-48.0.1-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:0ee6ea481db1ab889cba043ec1eda17bb9c1ea79db6722f779c3667f9f70322f", size = 4745488, upload-time = "2026-06-09T22:32:30.07Z" }, + { url = "https://files.pythonhosted.org/packages/2e/aa/48972bce55049b32a94f4907eda4d75fa385aad8a39506cc2fc72196ecf0/cryptography-48.0.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:f2ceef93cb096aa3c4cc4b5c94ca6131f9196d28c64d6111533402a9b2054d41", size = 4830256, upload-time = "2026-06-09T22:31:43.868Z" }, + { url = "https://files.pythonhosted.org/packages/47/a2/e5079a032fb85cf6005046ca92bbd78b0c82dad2b5751ab8c311659da06f/cryptography-48.0.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9bd3f92d76217892b15df84ca256c2c113d386fdda7a7d8691aeeced976507c6", size = 4979117, upload-time = "2026-06-09T22:31:05.845Z" }, + { url = "https://files.pythonhosted.org/packages/b7/a0/8f50cae9c74e718ed769d63ed5c74bd0ea830c9550a74629cebd1b9c7bc7/cryptography-48.0.1-cp311-abi3-win32.whl", hash = "sha256:b9a32b876490d66c8bcc9963ef220199569748434ab01a9d6aaeabf88e7f5158", size = 3304154, upload-time = "2026-06-09T22:32:16.845Z" }, + { url = "https://files.pythonhosted.org/packages/c5/69/0572c77dbace6fef72f33755bd52ea399c71367250d366237f8691826b9e/cryptography-48.0.1-cp311-abi3-win_amd64.whl", hash = "sha256:39489bfca54c7a1f6b297efcd8bc608ab92d16c4ca631b0cad4da46724588b24", size = 3817138, upload-time = "2026-06-09T22:32:00.388Z" }, + { url = "https://files.pythonhosted.org/packages/42/06/3e768b4c3bc78201583fa35a0e18f640dd782ff41afba88f8545481a8874/cryptography-48.0.1-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:f817adc181390bd54f2f700107a7419040fb7c1bdf2fc26f36551a06a68c3345", size = 7989830, upload-time = "2026-06-09T22:31:07.8Z" }, + { url = "https://files.pythonhosted.org/packages/8a/13/6476736484b94041110c8340a3eb63962fea4975baea8cb4a512adb44d4d/cryptography-48.0.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d5d30989c6917b478b5817902e85fddaea2261efa8648383d965381ccb9e1ac4", size = 4689201, upload-time = "2026-06-09T22:31:09.745Z" }, + { url = "https://files.pythonhosted.org/packages/79/62/65a87f34d2a431546e2509b85d55e8c90df86d668f6731da64d538512ac2/cryptography-48.0.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:df637c05205ea7c1d7fbcbe54bbfea648a52951155f997af13d895d0ecc96991", size = 4702822, upload-time = "2026-06-09T22:32:24.409Z" }, + { url = "https://files.pythonhosted.org/packages/7f/59/810b5204b0a9b10f4b6bc06bd551a8b609803cd931806bc3b71884b225e5/cryptography-48.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:869c3b8a53bfe27147832df48b32adadf558249d50e76cb3769d40e986b13265", size = 4694875, upload-time = "2026-06-09T22:32:08.737Z" }, + { url = "https://files.pythonhosted.org/packages/24/dc/d8ca05ffea724eec6d232ea6f18e74c269eb6bdfdcc9bfba689790d1325f/cryptography-48.0.1-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:e361afba8918070d376df76f408a4f67fec0ee9cff81a99e48fe9a233ef59e17", size = 5290385, upload-time = "2026-06-09T22:31:15.212Z" }, + { url = "https://files.pythonhosted.org/packages/03/8c/3be6cb4da181f5bb6c19cf560c2359d60644a6b5fc5b57854e528f47b296/cryptography-48.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:d069066deead00ac7f090be101be875a06855908f7ec004c27b8fefb4acfb411", size = 4737082, upload-time = "2026-06-09T22:32:22.66Z" }, + { url = "https://files.pythonhosted.org/packages/aa/f6/d5f60a5a1434dbfd949e227fd0065d194c7e6b6ac526b17f5c06152b8231/cryptography-48.0.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:09f73a725d582cef64b91281a322cd798d14a33b2b6f2b7ad9531dc336d84c02", size = 4325328, upload-time = "2026-06-09T22:32:10.777Z" }, + { url = "https://files.pythonhosted.org/packages/17/b7/ba75dd947a14b6ad907b01ae8f6b5b348cdd1b48142f0063dee9e20c1d9d/cryptography-48.0.1-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:15254441469dd6bf027039453288e2072124f8b6603563f5d759e1c9b69273fa", size = 4694530, upload-time = "2026-06-09T22:31:53.105Z" }, + { url = "https://files.pythonhosted.org/packages/62/29/50d6b9e8aff12d8b67afaeb3569335e32dc83a5723e3bbded24fdac9f809/cryptography-48.0.1-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:8ace4507d1e6533c125f4fac754f8bb8b6a74c08e92179dabd7e16571a3efbf3", size = 5245046, upload-time = "2026-06-09T22:31:25.774Z" }, + { url = "https://files.pythonhosted.org/packages/9f/04/618f4115cfc0add0838c82507aa18a346089428da8653ad38b3ff36f5cb3/cryptography-48.0.1-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:b4e391975f038e66432328639620a4aff2d307513b004f1ca06d6225bced815c", size = 4736660, upload-time = "2026-06-09T22:32:12.676Z" }, + { url = "https://files.pythonhosted.org/packages/24/9c/06e062462a0de28a3b3911322eded4c16deb9f441b1b7575d3dc59488ab5/cryptography-48.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:42fcd8e26fe555d9b3577a135f5091fefa0aa4e99129c23fb56787a1bd4ada72", size = 4822229, upload-time = "2026-06-09T22:31:17.062Z" }, + { url = "https://files.pythonhosted.org/packages/f4/be/0561971eaaee4b8a0e7d5113c536921063ab91aaf23278ac374eaf881e11/cryptography-48.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c1400da5e32a43253392277eac7490a60e497d810a63dd5608d71bbd7af507c9", size = 4966364, upload-time = "2026-06-09T22:31:32.842Z" }, + { url = "https://files.pythonhosted.org/packages/a4/27/728c77876f12b000820b69ae490f3c4083775e79e07827e9e60be07ad209/cryptography-48.0.1-cp314-cp314t-win32.whl", hash = "sha256:0df56b056bc17c1b7d6821dfa65216e62bd232d8ab05eb3db44e71d235651471", size = 3278498, upload-time = "2026-06-09T22:31:29.154Z" }, + { url = "https://files.pythonhosted.org/packages/06/e3/79a612c6d7b1e6ee0edd43633d53035bec2cfb78c82b76f7864f39e36f34/cryptography-48.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:9de21387aa95e2a895823d0745b430bed4f33503ba9ab5e0b5311f33e37d66d2", size = 3798790, upload-time = "2026-06-09T22:31:56.697Z" }, + { url = "https://files.pythonhosted.org/packages/ca/6c/00fa2a95997164c8b2072ce327c23d4ab20809ccc323ea5fab91e53a4bba/cryptography-48.0.1-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:4fdc69f8e4316bcf0c8c8ec1f26f285d12e8142d88d96c876a59a03be3f6ae67", size = 7987408, upload-time = "2026-06-09T22:32:20.777Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d9/45f309a7e4e5f3f8f121d6d3be9e94024a7726ec598d6e08ae04edb2f04d/cryptography-48.0.1-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:48fe40804d4caa2288f24e70ca8c64c42dd826da0ad7e4f1b41b2128d679e6c8", size = 4690196, upload-time = "2026-06-09T22:31:54.74Z" }, + { url = "https://files.pythonhosted.org/packages/5f/9f/a1bc8bcc798811b8527eb374bbccf30a3f3e806829d967118222bf1125eb/cryptography-48.0.1-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:86be3b1b0b6bf09482fb50a979c508d2950ed95f5621ec77f4e385962006b83a", size = 4696782, upload-time = "2026-06-09T22:31:45.615Z" }, + { url = "https://files.pythonhosted.org/packages/66/c2/81a4fb4e4373c500bb526bc337ac5719dd31dd15b970b84a238168c6aa08/cryptography-48.0.1-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:4ab0a343c807bbcd90c971cd1ecf072937cd01847a9e002bef88fb47ac6be577", size = 4696618, upload-time = "2026-06-09T22:31:11.564Z" }, + { url = "https://files.pythonhosted.org/packages/e5/0b/aa68b221dde92d09cb29a024ede17550ee21e77a404e59fc093c82bb51e1/cryptography-48.0.1-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:9621de99d2da096006b629979efd8ae7eb2d8b822488d0c89ee4000c306c59b1", size = 5289970, upload-time = "2026-06-09T22:31:20.368Z" }, + { url = "https://files.pythonhosted.org/packages/78/13/fba657f958d2af66ea959a4ba01212632089249d34af1ae48054136344d7/cryptography-48.0.1-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:88c852a0ae366e262e5a1744b685e6a433dc8788dd2a277e418bf4904203609d", size = 4731873, upload-time = "2026-06-09T22:31:22.253Z" }, + { url = "https://files.pythonhosted.org/packages/4c/4c/9a964756d24a26b3e34dfcb16f961b89838786e6700b635b0d1e3adff4b6/cryptography-48.0.1-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:43c5835e2cb98c8733d86f57d6fc879b613f5c3478607281c3e36daffc6dd8a6", size = 4330804, upload-time = "2026-06-09T22:31:36.56Z" }, + { url = "https://files.pythonhosted.org/packages/4b/0f/a10f3a6eb12950a10e3a874070283aa2dd5875b2bfd15fad8a3e17b3f13e/cryptography-48.0.1-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:fe0180af5bf9236518a087e35bf2d9a347d5f5f51e63c579d683ddff424e3d46", size = 4696217, upload-time = "2026-06-09T22:31:13.351Z" }, + { url = "https://files.pythonhosted.org/packages/f3/6f/5cd12f951165ea73ef85266775d97e4c763b2474ccfd816dd69d3a18d6f8/cryptography-48.0.1-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:b7a2d1a937a738a881737cec135a38bb61470589b17515b9f73f571d0ae10401", size = 5245252, upload-time = "2026-06-09T22:32:02.193Z" }, + { url = "https://files.pythonhosted.org/packages/68/ab/8aaa12e4516ec4464033ab79b6f3b592bd5a92102467c4ace8a0d970203f/cryptography-48.0.1-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:b74ca3b8e5ecdd833bf6a002ca41b4793bb27fb8f1c06ffaf2643c9e9140e31b", size = 4731388, upload-time = "2026-06-09T22:32:04.019Z" }, + { url = "https://files.pythonhosted.org/packages/1b/24/50027ea4dca85ec1f40688f3c24fb32ccacd520583c9592c3cc95628e6fb/cryptography-48.0.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2c37f2461406063b417837f5f3daab668652acd82423efcd7f0a9f04be972de1", size = 4824186, upload-time = "2026-06-09T22:32:18.707Z" }, + { url = "https://files.pythonhosted.org/packages/52/41/04cb5eb17085ade6f50cc611fb657df6a0f5885350de8764ece89c050197/cryptography-48.0.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:86fe77abb1bd87afb251d4d02ada7ecf53a32cee9b67d976abb2e45a13297475", size = 4964539, upload-time = "2026-06-09T22:31:18.793Z" }, + { url = "https://files.pythonhosted.org/packages/36/bf/ed70785c496e89d7e73b7cda2d21f2447fd6d4e821714b8d04ff217fed92/cryptography-48.0.1-cp39-abi3-win32.whl", hash = "sha256:6b2c0c3e6ccf3ade7750f836ef3ee36eea250cc467d45c256895573ac08cc6f1", size = 3282307, upload-time = "2026-06-09T22:30:53.162Z" }, + { url = "https://files.pythonhosted.org/packages/b3/ff/371ea7d252656ee1eb6d83eeeef3d1d0c6baf1d6497687d081ea03814670/cryptography-48.0.1-cp39-abi3-win_amd64.whl", hash = "sha256:9a49ca6c81417f6a5edb50375a60cccdd70fa0a91a5211829dbea74eba94d2ac", size = 3793408, upload-time = "2026-06-09T22:32:15.191Z" }, + { url = "https://files.pythonhosted.org/packages/a9/d3/eb4e394e587341fdad09a09101fa76478ead3a78b0ad63e55c22f0d75c02/cryptography-48.0.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:08a597acce1ff37f347400087776599e2348a3a8bc53b44120e463cd274efe4a", size = 3951747, upload-time = "2026-06-09T22:31:23.871Z" }, + { url = "https://files.pythonhosted.org/packages/e0/4a/3f43451b4f858bfceaaaffc649e6e787e8d4fb332a1d443af39ab02cc8f1/cryptography-48.0.1-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:735824ec41b7f74a7c45fb1591349333e4c696cb6c044e5f46356e560143e4cd", size = 4641226, upload-time = "2026-06-09T22:31:02.532Z" }, + { url = "https://files.pythonhosted.org/packages/73/4e/855584c2c23b09e4ce2d3b9c30e983e679cd60b068c513c6bbdb91e11782/cryptography-48.0.1-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:92a46e1d638daa264ba2971c0b0489c9409787943efae4d60ffda3d091ef832c", size = 4668958, upload-time = "2026-06-09T22:32:06.213Z" }, + { url = "https://files.pythonhosted.org/packages/42/3b/d35750e41d803d1e516fd6d6011f065424924da7af1748cef4cc9cb3ede1/cryptography-48.0.1-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:7e234ac052af99f2700826a5c29ea99d9c1b1f80341cde62d11c8154dc8e0bd9", size = 4640793, upload-time = "2026-06-09T22:32:26.331Z" }, + { url = "https://files.pythonhosted.org/packages/ca/aa/cdb7181fe865285e87e96825aaab239400f1de0c3bfba9bd9769b79f1a92/cryptography-48.0.1-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:33842cf0888951cef5bc7ac724ab844a42044c1727b967b7f8997289a0464f92", size = 4668505, upload-time = "2026-06-09T22:31:27.534Z" }, + { url = "https://files.pythonhosted.org/packages/5d/8c/ce3823c06c2804f194f9e64f0d67fa3f4094a39f2bb1a990cd03603af8fc/cryptography-48.0.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6184ca7b174f28d7c703f1290d4b297217c45355f77a98f67e9b7f14549ac54a", size = 3742204, upload-time = "2026-06-09T22:31:34.773Z" }, +] + +[[package]] +name = "ijson" +version = "3.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3a/06/b31f040a8764336a11152e474a7abcb3782fedb0d1cdf78f442b82878c56/ijson-3.5.1.tar.gz", hash = "sha256:af40bd1a85f55db0b8b30715c858761306bd92d5590148636f75c3309e6e76bd", size = 69913, upload-time = "2026-07-06T17:37:42.923Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/d3/16d1595d3ef4743fc55129211bc52f52d59c582d0b7be045d8c04be0ae0c/ijson-3.5.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:2aa9d0cf21d4de89fb633e5ec27e9ad02c3f9a4ffa3940d120b23b8aed3acffc", size = 89069, upload-time = "2026-07-06T17:36:15.727Z" }, + { url = "https://files.pythonhosted.org/packages/32/a5/ddba126e2d46cf3b86ad762aeb5e0a02ce0ebc6e4529fe7d06eecb217844/ijson-3.5.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:05eba5268a38809ba1c3dbfa44ea67336e2c353fc11768acc9c6442fe0ccac50", size = 60697, upload-time = "2026-07-06T17:36:16.66Z" }, + { url = "https://files.pythonhosted.org/packages/dc/74/444d8d00a4506a79fc5544614106fa48d5f6f7049511148d8b6cddb8e9d7/ijson-3.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:40ddd236c80a667dd6a1f6b625d18ddac68b8719ff795761b7542f2e1f78e4a4", size = 60747, upload-time = "2026-07-06T17:36:17.927Z" }, + { url = "https://files.pythonhosted.org/packages/ee/b1/bc07831e646aebcc91a7bad9c5a0bf7c3f3395f0b10599e021667a3777f1/ijson-3.5.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:e6cf9e49902f28af7a2e2f8b35c201195c0f0d5c170a5786e0c0a1b8492a4e37", size = 132095, upload-time = "2026-07-06T17:36:19.022Z" }, + { url = "https://files.pythonhosted.org/packages/1d/1f/b4547461d75db40744616e40c0a06cf2f46a14e60742f6d12510f4612985/ijson-3.5.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6ee1e6d59c800aa819952f6cb5ff08707ecd576b29cc9c3d00e33c2b371a92ce", size = 138790, upload-time = "2026-07-06T17:36:20.22Z" }, + { url = "https://files.pythonhosted.org/packages/a7/30/7ecba8377509eaea2666db5b39a1a99e23f5e3e1e7ee371ec366cbfc4f7c/ijson-3.5.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:affb85eb75fa03a21d1f790bbf26a0e66e5701672062a30dc5c3c6a29c5c0a63", size = 135233, upload-time = "2026-07-06T17:36:21.252Z" }, + { url = "https://files.pythonhosted.org/packages/38/36/0679010904b24398336b3099b09ccb1daa41c534e7cb0931e89d5fcdbee4/ijson-3.5.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3060b141ef758be3742315d44476109460c265b88247e3a4e479949f8b134eac", size = 138832, upload-time = "2026-07-06T17:36:22.323Z" }, + { url = "https://files.pythonhosted.org/packages/b0/90/a40f971e78191e423c7b3a23756f37c3a51c27aadd7769b3fb1816e0044d/ijson-3.5.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:ffba9bce60be21b496afc67a05ab8e3f431f87f0282fd6ce3c62004c951a1428", size = 133313, upload-time = "2026-07-06T17:36:23.405Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d7/b012c347d3ab011c0c4f7988dc6e85b83eaab59df1aec089f5db0e7b29c5/ijson-3.5.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:170cc4c209f57decc9b7ee5fd340f2a1602d54020fa222846482ff1c99e88fdc", size = 135706, upload-time = "2026-07-06T17:36:24.464Z" }, + { url = "https://files.pythonhosted.org/packages/f5/48/3eacb96124e78271f4e648c6ce36f9ce15ce2cef2afb6f8dc6e213e43979/ijson-3.5.1-cp311-cp311-win32.whl", hash = "sha256:6d581a071dae8dbee61f8d962e892787707bad6e641e2f6fb30dd89d3e896939", size = 52221, upload-time = "2026-07-06T17:36:25.517Z" }, + { url = "https://files.pythonhosted.org/packages/1b/1a/19eff8576da0b46fa4a5c8751536ea27ab34c44b2609b2bcded9d7808d42/ijson-3.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:1356bca96d015948b601b013defb2d5631e4330e8f5880e4d7c933d472a90c34", size = 54641, upload-time = "2026-07-06T17:36:26.453Z" }, + { url = "https://files.pythonhosted.org/packages/c7/80/86b28f28ebf190fffd4f46790e065311e2758b55d8e6bbd33d92e9a49448/ijson-3.5.1-cp311-cp311-win_arm64.whl", hash = "sha256:c2b83b24be73f0c7a301807a4c3081939524421c7ae1556eb6eac7cff50ddfa7", size = 53954, upload-time = "2026-07-06T17:36:27.542Z" }, + { url = "https://files.pythonhosted.org/packages/5b/6e/f3ded1ebb85ccc89a30f7b10a0076f30db70ae1d1e0b6423ff93c57b7539/ijson-3.5.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ee60c7741012671867678eae71c51872cac938b76f3d4ca40a778e6c361774d2", size = 88643, upload-time = "2026-07-06T17:36:28.529Z" }, + { url = "https://files.pythonhosted.org/packages/ee/f2/18f14a1d79ef4898e746b4f50dcdbe60abab317cc2bd8390f043b9553c4e/ijson-3.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:11c1d7d36a13054b5872ecd5d745dc4009d9abdbcba2312de69e66c2f92a46d2", size = 60611, upload-time = "2026-07-06T17:36:29.597Z" }, + { url = "https://files.pythonhosted.org/packages/30/c7/6e3e591324fd4c7a7a9e1bc23548bacbd84c0d91766b71f09f13e945e7e9/ijson-3.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b9517efbe6604bce16f3e50d49b0cd1bdc58917f98cf2eab026599c5c0422991", size = 60447, upload-time = "2026-07-06T17:36:30.747Z" }, + { url = "https://files.pythonhosted.org/packages/4d/a5/9af7be670381ddac26dd55107ed0110b50f5161673b053311db67f510dcc/ijson-3.5.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:ea4fd7bec203a600b1cc88a492dfe6b75ce4b1b87488a66adcd5406022213f64", size = 139092, upload-time = "2026-07-06T17:36:31.749Z" }, + { url = "https://files.pythonhosted.org/packages/41/fb/f9c1664d75467453e6bd4e5f9cd2211b730b09e049445ab64cbac68cc6a3/ijson-3.5.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:350caea815e53151994b597abc80cf669454276b5ac6aadcec69ef6d48f7e90b", size = 149921, upload-time = "2026-07-06T17:36:32.912Z" }, + { url = "https://files.pythonhosted.org/packages/43/80/d20b1c49c4aa7cc6644131e2e57192b45346ef4816566ed1cd9fd05bae38/ijson-3.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e4fcebfe1685bb7ba06a8255a5d428ea6b4b895d7acf979cb637d8bbc9db2f47", size = 149848, upload-time = "2026-07-06T17:36:34.032Z" }, + { url = "https://files.pythonhosted.org/packages/fd/fc/5baa710869f5ab939e6233583ced1546889b55c35f35b844c518ac10abc3/ijson-3.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d78f362f51c8691798758a9e6ac3c9d385ee1228cb82987c91562a2fae235cd3", size = 150810, upload-time = "2026-07-06T17:36:35.19Z" }, + { url = "https://files.pythonhosted.org/packages/54/16/a12b3d987a5c1677b04557c6f9b9feb7e04b7d4171e9a344856cb9136e9b/ijson-3.5.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:0b184180d45f85fd4479659582749b109e49f4a29c21ac700ccc9c2280fe015e", size = 142989, upload-time = "2026-07-06T17:36:36.23Z" }, + { url = "https://files.pythonhosted.org/packages/ed/63/1026c535671fc334fc85aeb78f0945c825e7a338575edc753c0f455459ae/ijson-3.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e353891d33a2e6aa5caf72c2a5fbadd7a46f5f9b32dcfd0c84113b2444c255b8", size = 151702, upload-time = "2026-07-06T17:36:37.296Z" }, + { url = "https://files.pythonhosted.org/packages/cb/af/b58aa3a2bf4d31c388ea78b49826605f60932891ce97e404d196766b4ea3/ijson-3.5.1-cp312-cp312-win32.whl", hash = "sha256:936f28671f018f8ac4d3f003ae9fa01d0467ab4ef4cfd0c97f23beda485b61c6", size = 52613, upload-time = "2026-07-06T17:36:38.345Z" }, + { url = "https://files.pythonhosted.org/packages/04/66/ce70a92949c2a753dad91fdd5761dc14f3a44517e80cfc3c26612982ed61/ijson-3.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:322c783f3ee0c6b383bbd4db88370b10172168808cc2a0bf811f1253f7435602", size = 54729, upload-time = "2026-07-06T17:36:39.337Z" }, + { url = "https://files.pythonhosted.org/packages/a5/ff/e17784240c9cf1d58de2f2853ebaf9cc54f6bce117a1f12a6150bbb4a5aa/ijson-3.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:e2ac204b59f09e38e16d277f906240e9fd38780e42076599419265af183dc4b4", size = 53714, upload-time = "2026-07-06T17:36:40.308Z" }, + { url = "https://files.pythonhosted.org/packages/fd/c0/5384ccf4fc497ae3dc79a5a28561b05518b503ade29daf3898168d640406/ijson-3.5.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:3c0556d628443d3e871f414855313b2ae6cd9faa0104de3316bd8db03aab1589", size = 88652, upload-time = "2026-07-06T17:36:41.278Z" }, + { url = "https://files.pythonhosted.org/packages/8e/42/58769b8b6d614adb15c2c938c77bcdbfadfba8b1d21a98b5b09cb8961adc/ijson-3.5.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:12aa7fcf46f0fdc8e9e7cf37541e1dc20ac3f9243a23f4d346ab5395f72b0fe2", size = 60607, upload-time = "2026-07-06T17:36:42.697Z" }, + { url = "https://files.pythonhosted.org/packages/db/4a/8322c2824c24184880587bbca45531127a21a4b3bfc897f13427fea02424/ijson-3.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a96066d8c12a18ce2fa90579f2bbf991377cb71725874932e4a5d855226c162a", size = 60447, upload-time = "2026-07-06T17:36:43.791Z" }, + { url = "https://files.pythonhosted.org/packages/f4/43/7bdca8f733c45ce97f61a64fadd3e51d255c4c9b467345cbf71ccc7bb368/ijson-3.5.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:a19413a092d458a57aaa574fec08e265851d3b5c6e018377f426cd5e70b91280", size = 138889, upload-time = "2026-07-06T17:36:45.081Z" }, + { url = "https://files.pythonhosted.org/packages/e7/dc/e8a2e63700ab1d63aaf3fa38c454f8178eaa5b80a6d7c019d1d61b490a6c/ijson-3.5.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:65974568748678165d7e90e3e7ce2f7c233cfe4de6c37fbb0760941c97e14632", size = 149933, upload-time = "2026-07-06T17:36:46.312Z" }, + { url = "https://files.pythonhosted.org/packages/d9/56/640a4d980f7f2c11e399a7fd5ccb9e3d3c9e1dec3a1d5a10024570697c25/ijson-3.5.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bad5d55c99c89de8cd0a4cded51f86427ba3353c4dccca37ec2e32e06f26b437", size = 149857, upload-time = "2026-07-06T17:36:47.309Z" }, + { url = "https://files.pythonhosted.org/packages/3d/a1/c953e22c83992b69ae538a83b3678d28768f1a48042fc7794733423a5ce7/ijson-3.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1a38d503ce343952e88edfd9a27296a4ec96af7073a9db58b3df6233367f75fc", size = 151141, upload-time = "2026-07-06T17:36:48.405Z" }, + { url = "https://files.pythonhosted.org/packages/9e/ab/8fe5b7269b140e6e5f8837a33ce980fd9b67c70d0f8114289ed1cea4dace/ijson-3.5.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:2f41982c73896acab4a2a14faa14e152e444bd69f37c3139204429fd3fe65a10", size = 143112, upload-time = "2026-07-06T17:36:50.353Z" }, + { url = "https://files.pythonhosted.org/packages/78/f3/23d1284edcde50ba337ddfba5b5d59f8273084d98b28af94715e73dd2b64/ijson-3.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3321fede2b638d400de0036889a3a25c3bb689feb8df45e70a393346aad6194f", size = 152184, upload-time = "2026-07-06T17:36:51.536Z" }, + { url = "https://files.pythonhosted.org/packages/82/4e/df61be89dd295e4da722ec96ba03b1765bcb2becdaaaede9c96a7d2365b6/ijson-3.5.1-cp313-cp313-win32.whl", hash = "sha256:af6ddbd10ac9bce87a835f2de3ec61455ec435c54e7e0ba7b17c31c66de6f164", size = 52607, upload-time = "2026-07-06T17:36:52.596Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d9/03e5dbd3ef7e0cee06fbef0f87b91d7ce1c07fae9b5a1b0ca8b895de62c4/ijson-3.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:1de3de278b0ffb40338374ad2a730e1c56f933e0706b1815ebeb07b82239b1a3", size = 54730, upload-time = "2026-07-06T17:36:53.526Z" }, + { url = "https://files.pythonhosted.org/packages/38/30/4f37076c88a96a1a5e44df38b59fade4f59eaef87ef8b5162d55b2d426d5/ijson-3.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:c8a36a19b92cb7172c6448ab94f446033cfa3129dc4894aebe205f96b3fabf42", size = 53719, upload-time = "2026-07-06T17:36:54.592Z" }, + { url = "https://files.pythonhosted.org/packages/f9/17/54f9180c0da9a9e96e5b3791bc74093f029a2344678b4da218c2699465bf/ijson-3.5.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:21e1a250b254edba2f0dd7272a4c56f0a879aabe328d9e306dd1fc115f560e74", size = 89223, upload-time = "2026-07-06T17:36:55.534Z" }, + { url = "https://files.pythonhosted.org/packages/09/70/0ee0d2627c534174455a745ca25284797e71b0d6e2b2a1b31cc914e7b462/ijson-3.5.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e01f95433725e2df62d682ff88e4a57bb694385ff2362bc364adec961167ae04", size = 60831, upload-time = "2026-07-06T17:36:56.554Z" }, + { url = "https://files.pythonhosted.org/packages/8d/e6/56f64ba7a3e7a25d9a9fbbeb4c30597d6b76c1094cc2041d11a3224b562c/ijson-3.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:539e8d6cca079bcbb68c390e55148f908e0a943a34f7dd321248637c6272adca", size = 60752, upload-time = "2026-07-06T17:36:57.826Z" }, + { url = "https://files.pythonhosted.org/packages/3e/2b/5a55db881f1b043cd6d5716578937a60ac16348be1a3afbf846b21cf4b44/ijson-3.5.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:32f64051be2f990d8ae7b614b5abdf4a7bead510ce3666568d7403c6c46ce4d8", size = 140783, upload-time = "2026-07-06T17:36:58.984Z" }, + { url = "https://files.pythonhosted.org/packages/2e/61/f7783cc18672dc31544141139efd187fb34795d24e573fed6abea6b776c7/ijson-3.5.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cd0dfc5a788d0b0c2f1eab258b9dabdeefc631ca8ef87644a999f633b0b2555a", size = 149976, upload-time = "2026-07-06T17:37:00.235Z" }, + { url = "https://files.pythonhosted.org/packages/5f/d6/4182dd63b6b70eae4f5208c53558a050895a40734dff283463033c153742/ijson-3.5.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42bfda7858d99ee9777ec28cb6d347928249eefeb577f9b0a67503c18f7ebb6a", size = 149317, upload-time = "2026-07-06T17:37:01.476Z" }, + { url = "https://files.pythonhosted.org/packages/01/b1/a675e4a9b428a0ef556e7d718bf0e6885e3e5543042248a1a7030899a3d4/ijson-3.5.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c4b9a28e9719d1aebebe93ad8dc2ba87f4e2d9035043b196c1c07ef8530b44cc", size = 150555, upload-time = "2026-07-06T17:37:02.676Z" }, + { url = "https://files.pythonhosted.org/packages/b5/69/52686f56b44af63a93c3dc3f5bcfa07f87427d9aea4d2cbe3e1c94188c74/ijson-3.5.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9a0b25c750a6bde14a0b31f1dcbfc86368e50767e3eaa73bb138e54128055edd", size = 144485, upload-time = "2026-07-06T17:37:03.779Z" }, + { url = "https://files.pythonhosted.org/packages/f0/46/10554e817dde56300a8414e52c0f5a44a29f3440327cd6d829ece57759b3/ijson-3.5.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:bd756f7b22df745ac14b7bc2ab9ed7c190a222e4c8e1bef26ef1162af8e54d0f", size = 151470, upload-time = "2026-07-06T17:37:04.901Z" }, + { url = "https://files.pythonhosted.org/packages/91/82/f37cbb110b48abdb623d169d0e196f2f6e064e2c20fa789ecde6e69b0440/ijson-3.5.1-cp314-cp314-win32.whl", hash = "sha256:e035cdfb2a1446b13881f0dfc0eecd1541cbb17a27a938ded2160ae6ce25051b", size = 53219, upload-time = "2026-07-06T17:37:06.254Z" }, + { url = "https://files.pythonhosted.org/packages/00/58/792df8f001c246c8ff28f860de81d35ea0d797c0d3276c22a2af83089656/ijson-3.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:eeb2fb2daa5dd30326f93db465d0855b34aa6b1f52a7c0ff94522aec5ad57dfb", size = 55485, upload-time = "2026-07-06T17:37:07.242Z" }, + { url = "https://files.pythonhosted.org/packages/c0/3c/db3ccc22c09ed4738787e8d82fff76101aa81ec8de7eaf6572e065e012d3/ijson-3.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:a96ab35d7ce2129dfde49c4c807596443410e260d7f7a4ca8fe4d0035553b589", size = 54390, upload-time = "2026-07-06T17:37:08.497Z" }, + { url = "https://files.pythonhosted.org/packages/26/59/eefa5d9488250c03f24152576804205ae40e29cac0dc65cbbc5f3d422008/ijson-3.5.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:77b68e91f95fb16ac2e7819903cd545db6cffa308c28833cc34911e6b21e91dd", size = 93177, upload-time = "2026-07-06T17:37:09.71Z" }, + { url = "https://files.pythonhosted.org/packages/88/db/6329eb7bb9f1906c1906fc10e7074b8f08bf39b7d50baa58f1b597d48898/ijson-3.5.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:94a95065b1ac67602af0cec852b07505abc37b77e3774d1c801d935d05e48f82", size = 62891, upload-time = "2026-07-06T17:37:10.735Z" }, + { url = "https://files.pythonhosted.org/packages/fc/d0/b3beddb96eef0b20bb9902c36e4de30f145be06d7e5e1d780e1a1689d0ce/ijson-3.5.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:b70b5da6b0571da8f601a437c4fba2d35bc27739637d85f3acdc8f88916ce68e", size = 62575, upload-time = "2026-07-06T17:37:11.681Z" }, + { url = "https://files.pythonhosted.org/packages/5b/01/95f3a7c27d25bb917954ef0c8e86d0e60f585b9db675cbd05d355f54cce8/ijson-3.5.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:0ade373dd765b057b1dec05d7711bfeb5a36f1e825259466d9f545cfd8ef3ba3", size = 200568, upload-time = "2026-07-06T17:37:12.743Z" }, + { url = "https://files.pythonhosted.org/packages/77/61/c94ee4ea1f22318aab9a49b35d0ce8ac87dd24d508ea4c77dcbde362ba5e/ijson-3.5.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:882bc0bdd25d41eae90a15695cd50707edde0978b8b72a2532e30442dd8fd04c", size = 217956, upload-time = "2026-07-06T17:37:14.041Z" }, + { url = "https://files.pythonhosted.org/packages/1a/82/43e8d225aea5ee00eef7998c8ce41f344f7ba451329dfa9e92f4700813af/ijson-3.5.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:451901c36e12fa87cbb1cafe661bd25c08c6bd7900cc738279614f71cea07048", size = 208403, upload-time = "2026-07-06T17:37:15.201Z" }, + { url = "https://files.pythonhosted.org/packages/cf/6f/375f67fad76677aca9bc0817b2b18fdd231d309fe24e26b19a5556ef6cdd/ijson-3.5.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e3c5f660658f2ebfba5d4dfe4bafe8cd3a0defcda410ec08d2205fe08c398940", size = 211967, upload-time = "2026-07-06T17:37:16.484Z" }, + { url = "https://files.pythonhosted.org/packages/dc/53/4c754c3ba18ec70b7086b91a4abd368358fc47cc9b3871afd50deef4fea1/ijson-3.5.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:29eb8f0c77a296a10843a1714ad4a5d561e604cda3c88585e9012cf2c1729b0a", size = 201020, upload-time = "2026-07-06T17:37:18.017Z" }, + { url = "https://files.pythonhosted.org/packages/26/2d/3e7191b3222a31c378b827565b4fa64676a293441279f84db3d971720bf5/ijson-3.5.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:85997568d6b304cfa59d5c3f2b04f95b92e9a8c7f57d312343a7989cf8dfff85", size = 205584, upload-time = "2026-07-06T17:37:19.343Z" }, + { url = "https://files.pythonhosted.org/packages/24/11/55ae9c915e68f37c8698f8b09355071dc808ced5e9d4abf8238dc363f500/ijson-3.5.1-cp314-cp314t-win32.whl", hash = "sha256:c2e2509dc7f2fa5a2ac9ba7d15dd901f4093bd36b0784f65e04b681b7956651c", size = 54438, upload-time = "2026-07-06T17:37:20.656Z" }, + { url = "https://files.pythonhosted.org/packages/96/df/5bf2656447f14a923d25a0401b1cd628ca05c23041d3a4c116ae8d44dc39/ijson-3.5.1-cp314-cp314t-win_amd64.whl", hash = "sha256:2699e838099d056818c5f8e4ba702b345d0304e58847bdc79c5c1616d5d750a5", size = 56467, upload-time = "2026-07-06T17:37:21.615Z" }, + { url = "https://files.pythonhosted.org/packages/4e/e4/dec06e84fac704039625039c6b116a44f17ad72fda48b8f88a2493364b77/ijson-3.5.1-cp314-cp314t-win_arm64.whl", hash = "sha256:c388f85cbb9eec022b2bdedd23ffacfe7ab100c1200b1f47bee6e6ea2c3309fa", size = 55774, upload-time = "2026-07-06T17:37:22.958Z" }, + { url = "https://files.pythonhosted.org/packages/49/ea/f42470cc773c8686dd0823da8aefc31a138cd9aea1ad476d43c8293068da/ijson-3.5.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:077b1b0bcb6a622d460c6674fe6647c7af5a3b06503e1996d1efcf9f78c94512", size = 57830, upload-time = "2026-07-06T17:37:37.005Z" }, + { url = "https://files.pythonhosted.org/packages/d0/2f/64c61edab2c5ecf42a524146a70fa6171c8cf3960b947fb4c5f175660cb3/ijson-3.5.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:e8dbf71b21e65cb7f0d4d387c07fe73be820168070c3be05a0763a80f424f1c7", size = 57325, upload-time = "2026-07-06T17:37:38.017Z" }, + { url = "https://files.pythonhosted.org/packages/9f/5b/553ea8f14dfc756d6b6c9be2e2231ab44877ce96408eb9da3bb3f11ddd13/ijson-3.5.1-pp311-pypy311_pp73-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:0d7c5025a820f36f3e0e64f4b0232b338c690664c12b497e205cf64dcc64fc12", size = 71344, upload-time = "2026-07-06T17:37:38.997Z" }, + { url = "https://files.pythonhosted.org/packages/2e/3e/0248fd00746731074ca01365a25d8aa3c4d54642c8a14490d94f7550bda9/ijson-3.5.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aa7a2c94e43c02e0482088e6ff997e2bd7b9a76e6f1d0fd70891b4b5ff51318f", size = 71335, upload-time = "2026-07-06T17:37:39.965Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b9/1f1259546cc875adad240c468515f428d3a79b3def3ced17be3cdfe29146/ijson-3.5.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:69b5eef70240e9734c5a2fb5cc3742cae411fc833a66b9a50722b9eedb1e27de", size = 68728, upload-time = "2026-07-06T17:37:40.928Z" }, + { url = "https://files.pythonhosted.org/packages/ea/02/aafbf0c3e1468c7c0f607065363b49c381de7e4bb43ae6674684a3fafe92/ijson-3.5.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:4b75b6bf4b0dbb0df24947db6722cd5723ce8d6e6b13fddbfc98db312ba82237", size = 54922, upload-time = "2026-07-06T17:37:41.879Z" }, +] + +[[package]] +name = "jmespath" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload-time = "2026-01-22T16:35:26.279Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload-time = "2026-01-22T16:35:24.919Z" }, +] + +[[package]] +name = "pyarrow" +version = "23.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/88/22/134986a4cc224d593c1afde5494d18ff629393d74cc2eddb176669f234a4/pyarrow-23.0.1.tar.gz", hash = "sha256:b8c5873e33440b2bc2f4a79d2b47017a89c5a24116c055625e6f2ee50523f019", size = 1167336, upload-time = "2026-02-16T10:14:12.39Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b0/41/8e6b6ef7e225d4ceead8459427a52afdc23379768f54dd3566014d7618c1/pyarrow-23.0.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:6f0147ee9e0386f519c952cc670eb4a8b05caa594eeffe01af0e25f699e4e9bb", size = 34302230, upload-time = "2026-02-16T10:09:03.859Z" }, + { url = "https://files.pythonhosted.org/packages/bf/4a/1472c00392f521fea03ae93408bf445cc7bfa1ab81683faf9bc188e36629/pyarrow-23.0.1-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:0ae6e17c828455b6265d590100c295193f93cc5675eb0af59e49dbd00d2de350", size = 35850050, upload-time = "2026-02-16T10:09:11.877Z" }, + { url = "https://files.pythonhosted.org/packages/0c/b2/bd1f2f05ded56af7f54d702c8364c9c43cd6abb91b0e9933f3d77b4f4132/pyarrow-23.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:fed7020203e9ef273360b9e45be52a2a47d3103caf156a30ace5247ffb51bdbd", size = 44491918, upload-time = "2026-02-16T10:09:18.144Z" }, + { url = "https://files.pythonhosted.org/packages/0b/62/96459ef5b67957eac38a90f541d1c28833d1b367f014a482cb63f3b7cd2d/pyarrow-23.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:26d50dee49d741ac0e82185033488d28d35be4d763ae6f321f97d1140eb7a0e9", size = 47562811, upload-time = "2026-02-16T10:09:25.792Z" }, + { url = "https://files.pythonhosted.org/packages/7d/94/1170e235add1f5f45a954e26cd0e906e7e74e23392dcb560de471f7366ec/pyarrow-23.0.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3c30143b17161310f151f4a2bcfe41b5ff744238c1039338779424e38579d701", size = 48183766, upload-time = "2026-02-16T10:09:34.645Z" }, + { url = "https://files.pythonhosted.org/packages/0e/2d/39a42af4570377b99774cdb47f63ee6c7da7616bd55b3d5001aa18edfe4f/pyarrow-23.0.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:db2190fa79c80a23fdd29fef4b8992893f024ae7c17d2f5f4db7171fa30c2c78", size = 50607669, upload-time = "2026-02-16T10:09:44.153Z" }, + { url = "https://files.pythonhosted.org/packages/00/ca/db94101c187f3df742133ac837e93b1f269ebdac49427f8310ee40b6a58f/pyarrow-23.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:f00f993a8179e0e1c9713bcc0baf6d6c01326a406a9c23495ec1ba9c9ebf2919", size = 27527698, upload-time = "2026-02-16T10:09:50.263Z" }, + { url = "https://files.pythonhosted.org/packages/9a/4b/4166bb5abbfe6f750fc60ad337c43ecf61340fa52ab386da6e8dbf9e63c4/pyarrow-23.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:f4b0dbfa124c0bb161f8b5ebb40f1a680b70279aa0c9901d44a2b5a20806039f", size = 34214575, upload-time = "2026-02-16T10:09:56.225Z" }, + { url = "https://files.pythonhosted.org/packages/e1/da/3f941e3734ac8088ea588b53e860baeddac8323ea40ce22e3d0baa865cc9/pyarrow-23.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:7707d2b6673f7de054e2e83d59f9e805939038eebe1763fe811ee8fa5c0cd1a7", size = 35832540, upload-time = "2026-02-16T10:10:03.428Z" }, + { url = "https://files.pythonhosted.org/packages/88/7c/3d841c366620e906d54430817531b877ba646310296df42ef697308c2705/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:86ff03fb9f1a320266e0de855dee4b17da6794c595d207f89bba40d16b5c78b9", size = 44470940, upload-time = "2026-02-16T10:10:10.704Z" }, + { url = "https://files.pythonhosted.org/packages/2c/a5/da83046273d990f256cb79796a190bbf7ec999269705ddc609403f8c6b06/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:813d99f31275919c383aab17f0f455a04f5a429c261cc411b1e9a8f5e4aaaa05", size = 47586063, upload-time = "2026-02-16T10:10:17.95Z" }, + { url = "https://files.pythonhosted.org/packages/5b/3c/b7d2ebcff47a514f47f9da1e74b7949138c58cfeb108cdd4ee62f43f0cf3/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bf5842f960cddd2ef757d486041d57c96483efc295a8c4a0e20e704cbbf39c67", size = 48173045, upload-time = "2026-02-16T10:10:25.363Z" }, + { url = "https://files.pythonhosted.org/packages/43/b2/b40961262213beaba6acfc88698eb773dfce32ecdf34d19291db94c2bd73/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:564baf97c858ecc03ec01a41062e8f4698abc3e6e2acd79c01c2e97880a19730", size = 50621741, upload-time = "2026-02-16T10:10:33.477Z" }, + { url = "https://files.pythonhosted.org/packages/f6/70/1fdda42d65b28b078e93d75d371b2185a61da89dda4def8ba6ba41ebdeb4/pyarrow-23.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:07deae7783782ac7250989a7b2ecde9b3c343a643f82e8a4df03d93b633006f0", size = 27620678, upload-time = "2026-02-16T10:10:39.31Z" }, + { url = "https://files.pythonhosted.org/packages/47/10/2cbe4c6f0fb83d2de37249567373d64327a5e4d8db72f486db42875b08f6/pyarrow-23.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6b8fda694640b00e8af3c824f99f789e836720aa8c9379fb435d4c4953a756b8", size = 34210066, upload-time = "2026-02-16T10:10:45.487Z" }, + { url = "https://files.pythonhosted.org/packages/cb/4f/679fa7e84dadbaca7a65f7cdba8d6c83febbd93ca12fa4adf40ba3b6362b/pyarrow-23.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:8ff51b1addc469b9444b7c6f3548e19dc931b172ab234e995a60aea9f6e6025f", size = 35825526, upload-time = "2026-02-16T10:10:52.266Z" }, + { url = "https://files.pythonhosted.org/packages/f9/63/d2747d930882c9d661e9398eefc54f15696547b8983aaaf11d4a2e8b5426/pyarrow-23.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:71c5be5cbf1e1cb6169d2a0980850bccb558ddc9b747b6206435313c47c37677", size = 44473279, upload-time = "2026-02-16T10:11:01.557Z" }, + { url = "https://files.pythonhosted.org/packages/b3/93/10a48b5e238de6d562a411af6467e71e7aedbc9b87f8d3a35f1560ae30fb/pyarrow-23.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:9b6f4f17b43bc39d56fec96e53fe89d94bac3eb134137964371b45352d40d0c2", size = 47585798, upload-time = "2026-02-16T10:11:09.401Z" }, + { url = "https://files.pythonhosted.org/packages/5c/20/476943001c54ef078dbf9542280e22741219a184a0632862bca4feccd666/pyarrow-23.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9fc13fc6c403d1337acab46a2c4346ca6c9dec5780c3c697cf8abfd5e19b6b37", size = 48179446, upload-time = "2026-02-16T10:11:17.781Z" }, + { url = "https://files.pythonhosted.org/packages/4b/b6/5dd0c47b335fcd8edba9bfab78ad961bd0fd55ebe53468cc393f45e0be60/pyarrow-23.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5c16ed4f53247fa3ffb12a14d236de4213a4415d127fe9cebed33d51671113e2", size = 50623972, upload-time = "2026-02-16T10:11:26.185Z" }, + { url = "https://files.pythonhosted.org/packages/d5/09/a532297c9591a727d67760e2e756b83905dd89adb365a7f6e9c72578bcc1/pyarrow-23.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:cecfb12ef629cf6be0b1887f9f86463b0dd3dc3195ae6224e74006be4736035a", size = 27540749, upload-time = "2026-02-16T10:12:23.297Z" }, + { url = "https://files.pythonhosted.org/packages/a5/8e/38749c4b1303e6ae76b3c80618f84861ae0c55dd3c2273842ea6f8258233/pyarrow-23.0.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:29f7f7419a0e30264ea261fdc0e5fe63ce5a6095003db2945d7cd78df391a7e1", size = 34471544, upload-time = "2026-02-16T10:11:32.535Z" }, + { url = "https://files.pythonhosted.org/packages/a3/73/f237b2bc8c669212f842bcfd842b04fc8d936bfc9d471630569132dc920d/pyarrow-23.0.1-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:33d648dc25b51fd8055c19e4261e813dfc4d2427f068bcecc8b53d01b81b0500", size = 35949911, upload-time = "2026-02-16T10:11:39.813Z" }, + { url = "https://files.pythonhosted.org/packages/0c/86/b912195eee0903b5611bf596833def7d146ab2d301afeb4b722c57ffc966/pyarrow-23.0.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:cd395abf8f91c673dd3589cadc8cc1ee4e8674fa61b2e923c8dd215d9c7d1f41", size = 44520337, upload-time = "2026-02-16T10:11:47.764Z" }, + { url = "https://files.pythonhosted.org/packages/69/c2/f2a717fb824f62d0be952ea724b4f6f9372a17eed6f704b5c9526f12f2f1/pyarrow-23.0.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:00be9576d970c31defb5c32eb72ef585bf600ef6d0a82d5eccaae96639cf9d07", size = 47548944, upload-time = "2026-02-16T10:11:56.607Z" }, + { url = "https://files.pythonhosted.org/packages/84/a7/90007d476b9f0dc308e3bc57b832d004f848fd6c0da601375d20d92d1519/pyarrow-23.0.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c2139549494445609f35a5cda4eb94e2c9e4d704ce60a095b342f82460c73a83", size = 48236269, upload-time = "2026-02-16T10:12:04.47Z" }, + { url = "https://files.pythonhosted.org/packages/b0/3f/b16fab3e77709856eb6ac328ce35f57a6d4a18462c7ca5186ef31b45e0e0/pyarrow-23.0.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:7044b442f184d84e2351e5084600f0d7343d6117aabcbc1ac78eb1ae11eb4125", size = 50604794, upload-time = "2026-02-16T10:12:11.797Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a1/22df0620a9fac31d68397a75465c344e83c3dfe521f7612aea33e27ab6c0/pyarrow-23.0.1-cp313-cp313t-win_amd64.whl", hash = "sha256:a35581e856a2fafa12f3f54fce4331862b1cfb0bef5758347a858a4aa9d6bae8", size = 27660642, upload-time = "2026-02-16T10:12:17.746Z" }, + { url = "https://files.pythonhosted.org/packages/8d/1b/6da9a89583ce7b23ac611f183ae4843cd3a6cf54f079549b0e8c14031e73/pyarrow-23.0.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:5df1161da23636a70838099d4aaa65142777185cc0cdba4037a18cee7d8db9ca", size = 34238755, upload-time = "2026-02-16T10:12:32.819Z" }, + { url = "https://files.pythonhosted.org/packages/ae/b5/d58a241fbe324dbaeb8df07be6af8752c846192d78d2272e551098f74e88/pyarrow-23.0.1-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:fa8e51cb04b9f8c9c5ace6bab63af9a1f88d35c0d6cbf53e8c17c098552285e1", size = 35847826, upload-time = "2026-02-16T10:12:38.949Z" }, + { url = "https://files.pythonhosted.org/packages/54/a5/8cbc83f04aba433ca7b331b38f39e000efd9f0c7ce47128670e737542996/pyarrow-23.0.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0b95a3994f015be13c63148fef8832e8a23938128c185ee951c98908a696e0eb", size = 44536859, upload-time = "2026-02-16T10:12:45.467Z" }, + { url = "https://files.pythonhosted.org/packages/36/2e/c0f017c405fcdc252dbccafbe05e36b0d0eb1ea9a958f081e01c6972927f/pyarrow-23.0.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4982d71350b1a6e5cfe1af742c53dfb759b11ce14141870d05d9e540d13bc5d1", size = 47614443, upload-time = "2026-02-16T10:12:55.525Z" }, + { url = "https://files.pythonhosted.org/packages/af/6b/2314a78057912f5627afa13ba43809d9d653e6630859618b0fd81a4e0759/pyarrow-23.0.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c250248f1fe266db627921c89b47b7c06fee0489ad95b04d50353537d74d6886", size = 48232991, upload-time = "2026-02-16T10:13:04.729Z" }, + { url = "https://files.pythonhosted.org/packages/40/f2/1bcb1d3be3460832ef3370d621142216e15a2c7c62602a4ea19ec240dd64/pyarrow-23.0.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f4763b83c11c16e5f4c15601ba6dfa849e20723b46aa2617cb4bffe8768479f", size = 50645077, upload-time = "2026-02-16T10:13:14.147Z" }, + { url = "https://files.pythonhosted.org/packages/eb/3f/b1da7b61cd66566a4d4c8383d376c606d1c34a906c3f1cb35c479f59d1aa/pyarrow-23.0.1-cp314-cp314-win_amd64.whl", hash = "sha256:3a4c85ef66c134161987c17b147d6bffdca4566f9a4c1d81a0a01cdf08414ea5", size = 28234271, upload-time = "2026-02-16T10:14:09.397Z" }, + { url = "https://files.pythonhosted.org/packages/b5/78/07f67434e910a0f7323269be7bfbf58699bd0c1d080b18a1ab49ba943fe8/pyarrow-23.0.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:17cd28e906c18af486a499422740298c52d7c6795344ea5002a7720b4eadf16d", size = 34488692, upload-time = "2026-02-16T10:13:21.541Z" }, + { url = "https://files.pythonhosted.org/packages/50/76/34cf7ae93ece1f740a04910d9f7e80ba166b9b4ab9596a953e9e62b90fe1/pyarrow-23.0.1-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:76e823d0e86b4fb5e1cf4a58d293036e678b5a4b03539be933d3b31f9406859f", size = 35964383, upload-time = "2026-02-16T10:13:28.63Z" }, + { url = "https://files.pythonhosted.org/packages/46/90/459b827238936d4244214be7c684e1b366a63f8c78c380807ae25ed92199/pyarrow-23.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a62e1899e3078bf65943078b3ad2a6ddcacf2373bc06379aac61b1e548a75814", size = 44538119, upload-time = "2026-02-16T10:13:35.506Z" }, + { url = "https://files.pythonhosted.org/packages/28/a1/93a71ae5881e99d1f9de1d4554a87be37da11cd6b152239fb5bd924fdc64/pyarrow-23.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:df088e8f640c9fae3b1f495b3c64755c4e719091caf250f3a74d095ddf3c836d", size = 47571199, upload-time = "2026-02-16T10:13:42.504Z" }, + { url = "https://files.pythonhosted.org/packages/88/a3/d2c462d4ef313521eaf2eff04d204ac60775263f1fb08c374b543f79f610/pyarrow-23.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:46718a220d64677c93bc243af1d44b55998255427588e400677d7192671845c7", size = 48259435, upload-time = "2026-02-16T10:13:49.226Z" }, + { url = "https://files.pythonhosted.org/packages/cc/f1/11a544b8c3d38a759eb3fbb022039117fd633e9a7b19e4841cc3da091915/pyarrow-23.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a09f3876e87f48bc2f13583ab551f0379e5dfb83210391e68ace404181a20690", size = 50629149, upload-time = "2026-02-16T10:13:57.238Z" }, + { url = "https://files.pythonhosted.org/packages/50/f2/c0e76a0b451ffdf0cf788932e182758eb7558953f4f27f1aff8e2518b653/pyarrow-23.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:527e8d899f14bd15b740cd5a54ad56b7f98044955373a17179d5956ddb93d9ce", size = 28365807, upload-time = "2026-02-16T10:14:03.892Z" }, +] + +[[package]] +name = "pycparser" +version = "3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "ruff" +version = "0.16.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/70/25/7113f6d5498888c5fb7db34081cba7d5971c4cb1bfb26819966eee68f003/ruff-0.16.1.tar.gz", hash = "sha256:fedad7c801dabd3fb9741d76aca39246e6ddd9ca446a015875207bf19f1e6bc7", size = 4877500, upload-time = "2026-07-30T19:37:01.379Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1b/bd/694da69368e0973de65df2ddc73ab18d43c469d5963d9b150911de6bc513/ruff-0.16.1-py3-none-linux_armv6l.whl", hash = "sha256:58edb313b88f0c5460a26adf5f39a37a3be789494a15e3e411e35fa78b89f9a0", size = 10839126, upload-time = "2026-07-30T19:36:13.697Z" }, + { url = "https://files.pythonhosted.org/packages/3f/f0/b626e5d5bd0dd9576263658ef12885e2288afd1029a48e26ffed65ec1ac1/ruff-0.16.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fde5a99e2f97479af66edd6622c6d5a2a7592c77cf4153d9e4428f5eeb55b60c", size = 11070253, upload-time = "2026-07-30T19:36:17.14Z" }, + { url = "https://files.pythonhosted.org/packages/83/63/f40acfb6b35b88623e71684942b552c3edd96035f5d98f313815f7b277de/ruff-0.16.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e0d4c20532fca4f7fa609369161d968dd28f65d83dabbd61d8e9c7edbf7001f6", size = 10561425, upload-time = "2026-07-30T19:36:20.04Z" }, + { url = "https://files.pythonhosted.org/packages/aa/dd/14ec0e9c2b4d315547dd38765004b4863e354e1b52cb308272215d9f6f6d/ruff-0.16.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:30affbcedf59ad5703d9c91f82266e02b47739f797e1a7b6e158e5526a6dae38", size = 10948879, upload-time = "2026-07-30T19:36:22.476Z" }, + { url = "https://files.pythonhosted.org/packages/33/e9/9d870cbae575030fdef595f04b4b97573c525b5497cce4f4498cf2f85446/ruff-0.16.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:24e9c631573cbca9d20f1283f8f479b2afa4a8503504822bd71a293889f16743", size = 10643691, upload-time = "2026-07-30T19:36:24.914Z" }, + { url = "https://files.pythonhosted.org/packages/c4/09/12743d544e2173f53ecd27217c65f90d2bc0f8424a66a60339e56bbc0457/ruff-0.16.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b41bdd48fb420987a9b5212e4957c26ad4abce401fa9ea9d4d85843727945f4f", size = 11435354, upload-time = "2026-07-30T19:36:28.447Z" }, + { url = "https://files.pythonhosted.org/packages/7f/89/a1652b2daee52083c9554a6333b678a8b01d0400f976827bb87857f9449a/ruff-0.16.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b0d1e1393b7648079e13669de1c1f4fde06d4583e84d8fd5c1551e0a77a2aa75", size = 12259033, upload-time = "2026-07-30T19:36:31.326Z" }, + { url = "https://files.pythonhosted.org/packages/16/96/ecdcb8c54ee7b123b487f807eb014e6e019155a0b81dfb669acd52f28ce3/ruff-0.16.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:07bf434b1c95f4e093be4532068ef4fcf00924eb2ade8796075980902d6fd54a", size = 11667981, upload-time = "2026-07-30T19:36:34.394Z" }, + { url = "https://files.pythonhosted.org/packages/cd/90/c52e12e0d862e9572f2a33aa227409143520abe53111e9a6babbac7b4af8/ruff-0.16.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39897739f112253ee4fdd2e8aa9a4f9ded99fb2be367d5f31dfa4ded6025584c", size = 11468183, upload-time = "2026-07-30T19:36:37.339Z" }, + { url = "https://files.pythonhosted.org/packages/2c/6b/4ffb7ad1d83eb16cf8cbb3c8815d3f11c88460fd162d4b372a2059be1c2a/ruff-0.16.1-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:82ae3c0c0d74daf17b968a10b7b3bb3ef297ab7de0c1f749646b25e690ccb150", size = 11470071, upload-time = "2026-07-30T19:36:39.91Z" }, + { url = "https://files.pythonhosted.org/packages/9c/72/32ae7db4c0b5e32ab611787caa19d1546800676d79f7483b7100a3561bf4/ruff-0.16.1-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:4d5f2ed10f8242d83fc08d521301089364e3375375705356f20c0e31606ef3ef", size = 10919503, upload-time = "2026-07-30T19:36:42.65Z" }, + { url = "https://files.pythonhosted.org/packages/f7/ca/3d901ba6ad6fc38da39c3448fc6c59ac945679293a17c3ceb6d6c1cba13e/ruff-0.16.1-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:a4665b309891f83f3e3c25447935f1213e9abbd4b5640af7a1f2def9f8d413c1", size = 10649861, upload-time = "2026-07-30T19:36:45.18Z" }, + { url = "https://files.pythonhosted.org/packages/92/79/894ef1ced26552d5f8c9cf6d85b0687840e1128c55aeab7b9c2d54a0d880/ruff-0.16.1-py3-none-musllinux_1_2_i686.whl", hash = "sha256:26e9ca5c9bc3971f20d3cf18a957f52ffd6a5f6564ff15c4912a144dcac22494", size = 11148137, upload-time = "2026-07-30T19:36:47.936Z" }, + { url = "https://files.pythonhosted.org/packages/2d/69/3609a09fa1cb46cc28b762363e440a354204e5dff01bd0c8d7437874d6b9/ruff-0.16.1-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:67e1e1e3fa4f0c82f0e36d4cd61e661f6e7a6196cb1aa92fe0828fa7b8f257cd", size = 11559211, upload-time = "2026-07-30T19:36:50.448Z" }, + { url = "https://files.pythonhosted.org/packages/fc/8a/fb22af2fd78a736e241fabf67e30ce1799a64244026377a49e133af90762/ruff-0.16.1-py3-none-win32.whl", hash = "sha256:d31765e131295b8445caf301e3e8a85b34d1b9b211b4109b7ba457888b051806", size = 10838258, upload-time = "2026-07-30T19:36:53.298Z" }, + { url = "https://files.pythonhosted.org/packages/d4/35/e57fd9fb5d423961df087a00b12d42c0a830288dc2f3b45ecca299158b4f/ruff-0.16.1-py3-none-win_amd64.whl", hash = "sha256:09b05e8b90c2cb06ad63464350e7a45e8e44a2dfe52072ebfba6666ca8d3f596", size = 11961111, upload-time = "2026-07-30T19:36:56.107Z" }, + { url = "https://files.pythonhosted.org/packages/cb/46/240ea004bf6dc4feb40e9832f2205a476a47dd5b8a3f8211a5fc5f95e20e/ruff-0.16.1-py3-none-win_arm64.whl", hash = "sha256:dbaadaac38c70239f056d306b7476f246b0bf000fa6b3876402acbf5b227eaf8", size = 11309414, upload-time = "2026-07-30T19:36:58.79Z" }, +] + +[[package]] +name = "s3transfer" +version = "0.19.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "botocore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/43/35e4d8aa320bffe8287fe8f65f578fa2d2db0a64212f0e710dce58267854/s3transfer-0.19.2.tar.gz", hash = "sha256:ba0309fd86be3c27dbf78cdd813c13c5e1df16e5874b99d2535ebbdfb9892993", size = 165592, upload-time = "2026-07-22T19:30:44.432Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bc/e7/5c595c75e9f41a44f30e526eda465ea0b4eec93470e074e4a111b253f13a/s3transfer-0.19.2-py3-none-any.whl", hash = "sha256:d8168eccca828cbb2cd573675333f3bddd254313a9c42494b84c76b539e8ba25", size = 90216, upload-time = "2026-07-22T19:30:43.251Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] diff --git a/receiver/.gitignore b/receiver/.gitignore new file mode 100644 index 0000000..9f94d8e --- /dev/null +++ b/receiver/.gitignore @@ -0,0 +1,5 @@ +node_modules/ +.wrangler/ +coverage/ +dist/ +wrangler.jsonc diff --git a/receiver/README.md b/receiver/README.md new file mode 100644 index 0000000..6a7b137 --- /dev/null +++ b/receiver/README.md @@ -0,0 +1,120 @@ +# ADC ciphertext receiver + +This directory contains the complete server-side surface for automatic uploads. It is one +Cloudflare Worker, one deployment-fixed `POST` path, and one R2 binding. The normative request, +bundle, and receipt contract is [`../protocol/v1/README.md`](../protocol/v1/README.md); the phase +boundaries are in +[`../docs/p0-p2-implementation-contract.md`](../docs/p0-p2-implementation-contract.md). + +The Worker accepts a bounded `ADCEXP01` ciphertext stream and stores it under its bundle UUID. It +checks the Protocol v1 content headers, untrusted routing claims, visible outer bundle identities, +actual byte count, and SHA-256. It never buffers a complete bundle. A create-only R2 write returns +`201`; an exact replay returns the same canonical receipt with `200`; reuse of a bundle UUID with +different bytes or metadata returns `409`. No success response is produced before the R2 operation +has completed. The verifier feeds R2 through a `FixedLengthStream`, preserving backpressure while +meeting R2's requirement that streamed uploads have a known length. + +The application-header vocabulary is closed-world. Apart from the ten Protocol v1 headers, the +Worker ignores only the ordinary OkHttp transport headers and headers in Cloudflare's +[edge HTTP header reference](https://developers.cloudflare.com/fundamentals/reference/http-headers/). +Local Wrangler/Miniflare's `MF-Original-Hostname` transport header is also ignored. +Every other client-controlled header is rejected. In particular, credentials, cookies, alternate +content encodings, and extra routing headers cannot silently acquire meaning. + +This is deliberately not an application server. It has no participant or device authentication, +private key, decryption, listing, download, deletion, administration, dashboard, remote control, +runtime configuration endpoint, D1, Queue, KV, or Durable Object. R2 metadata and request headers +are untrusted routing data, not proof of bundle origin or plaintext contents. + +## Fixed deployment inputs + +Copy `wrangler.example.jsonc` to the ignored `wrangler.jsonc` and replace every placeholder: + +- `routes[0].pattern`: the controlled receiver hostname; +- `r2_buckets[0].bucket_name`: the private ciphertext bucket; +- `UPLOAD_PATH`: one absolute path such as `/v1/upload`; query strings are rejected; +- `ALLOWED_CONFIGURATION_SHA256`: one lowercase SHA-256 of canonical configuration bytes; +- `ALLOWED_RESEARCHER_KEY_ID`: the researcher key ID bound into that configuration. + +The checked-in template intentionally cannot accept uploads unchanged. These values are fixed in +a Worker deployment; the Worker exposes no API that changes them. Deploy a separate Worker (and +normally a separate bucket) for a different configuration. + +The R2 object key is exactly the lowercase bundle UUID. Custom metadata has exactly these names: + +```text +sha256 +byte_count +configuration_sha256 +researcher_key_id +first_sequence_number +last_sequence_number +event_count +received_at_utc +``` + +All eight values are untrusted. `received_at_utc` is assigned by the Worker on the first successful +write and remains unchanged on replay. The seven-field receipt omits receive time and researcher +key ID, as required by Protocol v1. + +## Code map + +| Location | Responsibility | +| --- | --- | +| `src/contract.ts` | Closed-world request/outer-header parsing, R2 metadata identity, receipt bytes | +| `src/verified-body.ts` | Backpressured length, prefix, and streaming SHA-256 verification | +| `src/index.ts` | Single-route HTTP flow and create-only R2 transaction | +| `tests/receiver.test.ts` | Fake-R2 replay, race, conflict, bound, and failure-path coverage | +| `tests/receiver.workerd.test.ts` | Real workerd/R2 upload and replay against the shared Protocol corpus | +| `wrangler.example.jsonc` | The complete production binding and deployment template | + +## Develop and verify + +Requires Node.js 22+ and pnpm 10: + +```sh +cd receiver +pnpm install --frozen-lockfile +pnpm typecheck +pnpm test +pnpm build +``` + +The fast tests call the same request handler as production with an in-memory, conditionally written +fake R2 bucket. A separate integration suite runs the production export, native `DigestStream`, and +real local R2 implementation inside workerd, using the shared Protocol v1 valid bundle and receipt +vectors. Together they cover streaming validation, exact replay, concurrent create-only writes, +conditional-write races, conflicts, and storage failures without adding a test-only HTTP route. + +For a local Worker after creating `wrangler.jsonc`: + +```sh +pnpm dev +``` + +When manually posting a fixture with `curl`, add `--header 'Accept:'` to suppress curl's default +`Accept: */*`; it is not one of the exact Protocol v1 request headers and is intentionally rejected. + +## Deploy and operate + +1. Create a private R2 bucket and bind it as `BUNDLES`. Do not enable `r2.dev` public access. +2. Put a retention rule on the bucket before accepting uploads. Cloudflare documents both the + dashboard flow and `wrangler r2 bucket lifecycle` commands in its + [R2 object lifecycle guide](https://developers.cloudflare.com/r2/buckets/object-lifecycles/). +3. Route the Worker through a controlled hostname. Configure a + [WAF rate-limit rule](https://developers.cloudflare.com/waf/rate-limiting-rules/) for the exact + upload path; the ingress is intentionally public and can receive bounded bogus ciphertext. +4. Run `pnpm deploy`. Verify wrong paths and methods fail, then submit a Protocol v1 fixture and + confirm a `201` followed by an identical-receipt `200` replay. +5. Give analysts a separate least-privilege S3-compatible read credential for the bucket. The + Worker itself exposes no retrieval endpoint. + +Application logs contain no request headers, bundle IDs, participant identifiers, or bodies. Use +Cloudflare aggregate request/R2 metrics and WAF controls for operations; consent and governance +documents must still name the endpoint operator, jurisdiction, retention, and authorized analysts. + +The implementation relies only on documented Cloudflare behavior: R2 `put()` accepts streams, +custom metadata, SHA-256 verification, and conditional writes; a failed condition returns `null`; +successful writes are strongly consistent once the promise resolves. See the +[R2 Workers API reference](https://developers.cloudflare.com/r2/api/workers/workers-api-reference/) +and [R2 consistency model](https://developers.cloudflare.com/r2/reference/consistency/). diff --git a/receiver/package.json b/receiver/package.json new file mode 100644 index 0000000..2307e18 --- /dev/null +++ b/receiver/package.json @@ -0,0 +1,26 @@ +{ + "name": "@adc/ciphertext-receiver", + "private": true, + "type": "module", + "scripts": { + "build": "wrangler deploy --dry-run --config wrangler.example.jsonc --outdir dist", + "deploy": "wrangler deploy --config wrangler.jsonc", + "dev": "wrangler dev --config wrangler.jsonc", + "test": "pnpm test:unit && pnpm test:workerd", + "test:unit": "vitest run --config vitest.config.ts", + "test:workerd": "vitest run --config vitest.workerd.config.ts", + "typecheck": "tsc --noEmit" + }, + "devDependencies": { + "@cloudflare/vitest-pool-workers": "0.20.1", + "@cloudflare/workers-types": "5.20260801.1", + "@types/node": "24.2.0", + "typescript": "7.0.2", + "vitest": "4.1.10", + "wrangler": "4.118.0" + }, + "engines": { + "node": ">=22" + }, + "packageManager": "pnpm@10.21.0" +} diff --git a/receiver/pnpm-lock.yaml b/receiver/pnpm-lock.yaml new file mode 100644 index 0000000..6512f39 --- /dev/null +++ b/receiver/pnpm-lock.yaml @@ -0,0 +1,1787 @@ +lockfileVersion: '9.0' + +settings: + autoInstallPeers: true + excludeLinksFromLockfile: false + +importers: + + .: + devDependencies: + '@cloudflare/vitest-pool-workers': + specifier: 0.20.1 + version: 0.20.1(@cloudflare/workers-types@5.20260801.1)(@vitest/runner@4.1.10)(@vitest/snapshot@4.1.10)(vitest@4.1.10(@types/node@24.2.0)(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1))) + '@cloudflare/workers-types': + specifier: 5.20260801.1 + version: 5.20260801.1 + '@types/node': + specifier: 24.2.0 + version: 24.2.0 + typescript: + specifier: 7.0.2 + version: 7.0.2 + vitest: + specifier: 4.1.10 + version: 4.1.10(@types/node@24.2.0)(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1)) + wrangler: + specifier: 4.118.0 + version: 4.118.0(@cloudflare/workers-types@5.20260801.1) + +packages: + + '@cloudflare/kv-asset-handler@0.5.0': + resolution: {integrity: sha512-jxQYkj8dSIzc0cD6cMMNdOc1UVjqSqu8BZdor5s8cGjW2I8BjODt/kWPVdY+u9zj3ms75Q5qaZgnxUad83+eAg==} + engines: {node: '>=22.0.0'} + + '@cloudflare/unenv-preset@2.16.1': + resolution: {integrity: sha512-ECxObrMfyTl5bhQf/lZCXwo5G6xX9IAUo+nDMKK4SZ8m4Jvvxp52vilxyySSWh2YTZz8+HQ07qGH/2rEom1vDw==} + peerDependencies: + unenv: 2.0.0-rc.24 + workerd: '>1.20260305.0 <2.0.0-0' + peerDependenciesMeta: + workerd: + optional: true + + '@cloudflare/vitest-pool-workers@0.20.1': + resolution: {integrity: sha512-eN5jHaX78lY/btWlyWIiNtTIgmXnI0CvwC6CPukgRjHoXs66jqX0AEKUsUJOeybfXEmZUhhy5FP/Xx+6wNwI7A==} + peerDependencies: + '@vitest/runner': ^4.1.0 + '@vitest/snapshot': ^4.1.0 + vitest: ^4.1.0 + + '@cloudflare/workerd-darwin-64@1.20260730.1': + resolution: {integrity: sha512-+MBHmPaiTe2KajryW0T24rZvWFxb41hD3d8anNzQqHzft6vSEb18+sp0znSwxgij7ApPhSM1+vhkNg4f3YMguA==} + engines: {node: '>=16'} + cpu: [x64] + os: [darwin] + + '@cloudflare/workerd-darwin-arm64@1.20260730.1': + resolution: {integrity: sha512-SBHKntPkKvNPgaCrTe99xC1CAl8ygJDzlYfK0LbuJ1muKadIw35WnhO0wu894fKBtllsVQdNzDLee+cm0ppLSQ==} + engines: {node: '>=16'} + cpu: [arm64] + os: [darwin] + + '@cloudflare/workerd-linux-64@1.20260730.1': + resolution: {integrity: sha512-ouyPOSMbiKPeSwUJUvxtMcxGAXs2J4aPE4T5ABIYX5ClcQx5j5bbHTmnqOQEY8sAuLTPjH7dY+iB6UI5ISlwwA==} + engines: {node: '>=16'} + cpu: [x64] + os: [linux] + + '@cloudflare/workerd-linux-arm64@1.20260730.1': + resolution: {integrity: sha512-YQ+Mi78U3TPdgBPtwq+Sm6rJU+Ihl2y0pjYtuuKkdmUbYzL7oLR6Xqq9wljhasnuCFICssDJaqhMep5WizYoEQ==} + engines: {node: '>=16'} + cpu: [arm64] + os: [linux] + + '@cloudflare/workerd-windows-64@1.20260730.1': + resolution: {integrity: sha512-27fAN+vUECW1oYVc1KOcHYpkL8COM2Uxtxql7TL595kxbjoqS5yckw7NLz7bTf2pALFCZWjqXDjZGJ/xbG4ZKQ==} + engines: {node: '>=16'} + cpu: [x64] + os: [win32] + + '@cloudflare/workers-types@5.20260801.1': + resolution: {integrity: sha512-XCv5xWi47WQOK0LpLa6997Mrpz8Ct+nZmp/M5Xp8Z4BFsarf7nYjkznGOcOoYK5m1GfbMFEEuQ2OIZnbIWoe9A==} + + '@cspotcode/source-map-support@0.8.1': + resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==} + engines: {node: '>=12'} + + '@emnapi/runtime@1.11.3': + resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==} + + '@esbuild/aix-ppc64@0.28.1': + resolution: {integrity: sha512-Svl7tq8k/08+p6CXPpRjQ1fKX+1odH/BQbb48fV6fj3CWHhsoIOoY87w1oHXm0qEpkIK3ZfVgp0hed3XBXzXMQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [aix] + + '@esbuild/android-arm64@0.28.1': + resolution: {integrity: sha512-34EGEbCIAgosYz6goLcopX6Mo7NyGv9tfwEM2/7Ce2VcVRk568iSvniGWcUXIy7wEDR1wzolcxcriFVrWYcwBg==} + engines: {node: '>=18'} + cpu: [arm64] + os: [android] + + '@esbuild/android-arm@0.28.1': + resolution: {integrity: sha512-0k2F129Xdio1TdJfzJ8sy1Q47vUD2NnwdhiAf7drUN1EBTfPf4hsFCtmMgu/6m8JSzsBrlmVjudMBQqOfG8usQ==} + engines: {node: '>=18'} + cpu: [arm] + os: [android] + + '@esbuild/android-x64@0.28.1': + resolution: {integrity: sha512-dbwY7ltSMDWsRatcRpCnES4F+im88OCUgGZjy52shC7GqHRE/cYlxNbB4Z4UpJswpcc4Qxd2oE/ufM0p61IKng==} + engines: {node: '>=18'} + cpu: [x64] + os: [android] + + '@esbuild/darwin-arm64@0.28.1': + resolution: {integrity: sha512-TZbWkQY7kvTAXbXUT7uVACR5cMHsDiSz9z7ZKAX/RTq/WJEk3QyRr0wZpNhBDX+/0CtdqUIJlOiodQcta6tY3Q==} + engines: {node: '>=18'} + cpu: [arm64] + os: [darwin] + + '@esbuild/darwin-x64@0.28.1': + resolution: {integrity: sha512-zfdzgK9ACBNZLI/CyHTOx81SyNbM6YXn7rxSgX97VjyiPl9W1i4Ka4fgKECEoFCKGpvBj5qArWIGgQjOwkgskQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [darwin] + + '@esbuild/freebsd-arm64@0.28.1': + resolution: {integrity: sha512-wG2EA8ENdEI0qhkSZMjfqrdY+ziCYCPMmtZjjIwOmXFjmyzEHn+UUxk5of+SYsjtfs3VpnlC7QLzSI5hY/rOAw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [freebsd] + + '@esbuild/freebsd-x64@0.28.1': + resolution: {integrity: sha512-i7dZ9vQgnvSCzi/rYCXNgtF/U+eKZNJBzu3eTQbRgHnM7tNSizLOkRFAl3qzVc/Op/u5YkHHa4pf/3DOYHthLQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [freebsd] + + '@esbuild/linux-arm64@0.28.1': + resolution: {integrity: sha512-yHs+0uc8+nvEAfAfxrWQKK5peSNzBc4PegcMO0EJ2hT71uA7vB8Ihg2e77R2P7SG5uYjPbHlLLmve4LLLRCf0g==} + engines: {node: '>=18'} + cpu: [arm64] + os: [linux] + + '@esbuild/linux-arm@0.28.1': + resolution: {integrity: sha512-qVXBOHQS+d5Y722GwJzJUtOLlX7km3CraOaGormF1pDtPd2C/l1SHRPgjLunLGe51Sh5YYWKMFDyV4SxgMQYTQ==} + engines: {node: '>=18'} + cpu: [arm] + os: [linux] + + '@esbuild/linux-ia32@0.28.1': + resolution: {integrity: sha512-d1z4ZuP0ajrfz/FhGT4vv278rX8KnPPJx8i5+AtK7TYbx9Le9F1hyzurZpkEyjkGa9dUGhQow4C1NmeGvqxN2w==} + engines: {node: '>=18'} + cpu: [ia32] + os: [linux] + + '@esbuild/linux-loong64@0.28.1': + resolution: {integrity: sha512-M5sRjUVZrkm1OAPR3dlOYzNmN+loZKGVi1VUQGrwuqLcbR6qeAz+famMhjASeH3YVKvZz+zT1jlh/keC3Rj/lg==} + engines: {node: '>=18'} + cpu: [loong64] + os: [linux] + + '@esbuild/linux-mips64el@0.28.1': + resolution: {integrity: sha512-mRObBZeHh2OxcBFPWE/FjylkRgZdYuiTR3vaTozquCGOH14iP9oN4x4Ge81CoIDYQrXmIxpFumJBu5MtZpnQJQ==} + engines: {node: '>=18'} + cpu: [mips64el] + os: [linux] + + '@esbuild/linux-ppc64@0.28.1': + resolution: {integrity: sha512-slScBsMAb3GFDcdrCgLwZtPYRoH2H/youv10QiZyRjmsP48fznoveWytSgCI/R0ZcUgpc0ZhIUEx6LHts8yrfQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [linux] + + '@esbuild/linux-riscv64@0.28.1': + resolution: {integrity: sha512-kw0owk1o0GFETUJyW0jc0G4Yzs0BHZn0JDZ8JRT088vjJYX777BAs1fDGxAC+q831qOs2DTC96mNsG2opdfyyQ==} + engines: {node: '>=18'} + cpu: [riscv64] + os: [linux] + + '@esbuild/linux-s390x@0.28.1': + resolution: {integrity: sha512-/lAIjX8aYFRByhh6L5rYtPEDRqa9de/4V/juOXcta5frjvzXO4/sqEtyytse0g3zZFuWu5cDN0MkLz2qRDD2Ag==} + engines: {node: '>=18'} + cpu: [s390x] + os: [linux] + + '@esbuild/linux-x64@0.28.1': + resolution: {integrity: sha512-u/anNYF2mmVOEDwLtnQ1wOr3EZ9sTNGLWrsYGYwHWzGA3Si84IOkHXlbWTD1NB+9/1lcnweYKO54uhxZydNzfA==} + engines: {node: '>=18'} + cpu: [x64] + os: [linux] + + '@esbuild/netbsd-arm64@0.28.1': + resolution: {integrity: sha512-oks0DYbLwWMmaakTsCb+zL4E+aHRVLom9IJZOAthMQEPiQmydXHkziYEsGYRx0uNV/IjEKGAV941JzH02pflqw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [netbsd] + + '@esbuild/netbsd-x64@0.28.1': + resolution: {integrity: sha512-aeL6lAnN89Hz43Mlh1G8ARasbuoYvSITDEx0tHh5b7jJnHcssqgjy9Yx430GDpmCa6OyrKoS0aNRjKundRizGg==} + engines: {node: '>=18'} + cpu: [x64] + os: [netbsd] + + '@esbuild/openbsd-arm64@0.28.1': + resolution: {integrity: sha512-MEFJe5C3R8pwXdZ5Y21oo6m7ePiS0d9pWucn99O/wvyJZChoIQKrQDxKrGeW8F5+T0okTHesAmDeiHDTIq0V/Q==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openbsd] + + '@esbuild/openbsd-x64@0.28.1': + resolution: {integrity: sha512-i/ZLIOafE0Z8cI/XANJAixoJL/uRAoS2xOA3rb0xN+KK0K177cMAsQYkzHtBrtMXAKuAc7HGgcWiZ/sRC1Nxgw==} + engines: {node: '>=18'} + cpu: [x64] + os: [openbsd] + + '@esbuild/openharmony-arm64@0.28.1': + resolution: {integrity: sha512-ge+Z7EXFNt2BO1oAMsVpiQ8EwndV9i1xXerAeTIK7AtPs3bKFXQM7nlRxDSIUIMeueR1CNXxqztLzdNeReKBJg==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openharmony] + + '@esbuild/sunos-x64@0.28.1': + resolution: {integrity: sha512-BEjgtECkL3vY+SaSQ6nzVfiALUeFxpawyp8Jmf5PtYhf1Ug40N1h/hxlhts+f1FvSvarEigdxS3BlSMI2PJLcQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [sunos] + + '@esbuild/win32-arm64@0.28.1': + resolution: {integrity: sha512-lCv9eK/H6ZJWbE7bh2nw54CZ9M2nupBxJcTsdk/QQnWkdSjKGuxmmH8/GWrlT1eMmZfn4dGcCjRte397WqfQXA==} + engines: {node: '>=18'} + cpu: [arm64] + os: [win32] + + '@esbuild/win32-ia32@0.28.1': + resolution: {integrity: sha512-zvb/mB2bSCoJOpoCBgYKKpX6YM6mJBlBUVUtVj41DlZJVEB6/0CKlRYxP5wWl1C1ILiCoAU5wZZ4q1P3qeS6Eg==} + engines: {node: '>=18'} + cpu: [ia32] + os: [win32] + + '@esbuild/win32-x64@0.28.1': + resolution: {integrity: sha512-bm4Mowrv+GXMlpWX++EcXw/iLyd1o3+bJkC2DkWXYVvgZCqD/bSj9ctZeAMC3cIxgjRVR2Dufaiu4YPxr5gW1A==} + engines: {node: '>=18'} + cpu: [x64] + os: [win32] + + '@img/colour@1.1.0': + resolution: {integrity: sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==} + engines: {node: '>=18'} + + '@img/sharp-darwin-arm64@0.35.2': + resolution: {integrity: sha512-eEieHsMksAW4IiO5NzauESRl2D2qz3J/kwUxUrSfV06A93eEaRfMpHXyUb1mAqrR7i8U9A0GRqE9pjn6u1Jjpg==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [darwin] + + '@img/sharp-darwin-x64@0.35.2': + resolution: {integrity: sha512-BaktuGPCeHJMARpodR8jK4uKiZrPAy9WrfQW0sdI37clracq8Bp01AYS3SZgi5FS/y5twa9t4+LIuuxQjqRrWw==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [darwin] + + '@img/sharp-freebsd-wasm32@0.35.2': + resolution: {integrity: sha512-YoAxdnd8hPUkvLHd3bWY+YA8nw3xM/RyRopYucNsWHVSan8NLVM3X2volsfoRDcXdUJPg6tXahSd7HXPK7lRnw==} + engines: {node: '>=20.9.0'} + os: [freebsd] + + '@img/sharp-libvips-darwin-arm64@1.3.1': + resolution: {integrity: sha512-4V/M3roRMTYjiwZY9IOVQOE8OyeCxFAkYmyZDrZl51uOKjibm3oeEJ4WAmLxutAfzFbC9jqUiPs2gbnGflH+7g==} + cpu: [arm64] + os: [darwin] + + '@img/sharp-libvips-darwin-x64@1.3.1': + resolution: {integrity: sha512-c0/DxItpJv2+dGhgycJBBgotdqruGYDvA79drdh0MD1dFpy7JzJ/PlXwi1H4rFf0eTy8tgbI91aHDnZIceY3jQ==} + cpu: [x64] + os: [darwin] + + '@img/sharp-libvips-linux-arm64@1.3.1': + resolution: {integrity: sha512-JznefmcK9j1JKPz8AkQDh89kjojubyfOasWBPKfzMIhPwsgDy9evpE/naJTXXXmghS1iFwR8u/kTwh/I2/+GCw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linux-arm@1.3.1': + resolution: {integrity: sha512-aGGy9aWzXgHBG7HNyQPWorZthlp7+x6fDRoPAQbGO3ThcttuTyKIx3NuSHb6zb4gBNq6/yNn9f1cy9nFKS/Vmg==} + cpu: [arm] + os: [linux] + + '@img/sharp-libvips-linux-ppc64@1.3.1': + resolution: {integrity: sha512-1EkwGNCZk6iWNCMWqrvdJ+r1j0PT1zIz60CNPhYnJlK/zyeWqlsPZIe+ocBVqPF8k/Ssee/NCk+tE9Ryrko6ng==} + cpu: [ppc64] + os: [linux] + + '@img/sharp-libvips-linux-riscv64@1.3.1': + resolution: {integrity: sha512-Ilays+w2bXdnxzxtQdmXR62u8o8GYa3eL4+Gr+1KiE4xperMZUslRaVPJwwPkzlHEjGfXAfRVAa/7CYCtSqsBw==} + cpu: [riscv64] + os: [linux] + + '@img/sharp-libvips-linux-s390x@1.3.1': + resolution: {integrity: sha512-VfBwVHQTbRoj4XlpA/KLZ7ltgMpz+4WSejFzQ+GnoImjo1PtEJ59QB2qR1xQEeRPYIkNrPIm2L4cICMvz4C2ew==} + cpu: [s390x] + os: [linux] + + '@img/sharp-libvips-linux-x64@1.3.1': + resolution: {integrity: sha512-+c8ukgwU62DS54nCAjw7keOfHUkmr0B5QHEdcOqRnodF/MNXJbVI8Eopoj4B/0H8Asr65I+A4Amrn7a85/md6A==} + cpu: [x64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-arm64@1.3.1': + resolution: {integrity: sha512-qlKb/pwbkAi1WMsJrYHk7CuDrd12s27U2QnRhFYUoJNrRCmkosMTttuRFat/DDB3IlDm5qE1TJgZ4JDnHX8Ldw==} + cpu: [arm64] + os: [linux] + + '@img/sharp-libvips-linuxmusl-x64@1.3.1': + resolution: {integrity: sha512-yO21HwoUVLN8Qa+/SBjQLMYwBWAVJjeGPNe+hc0OUeMeifEtJqu5a1c4HayE1nNpDih9y3/KkoltfkDodmKAlg==} + cpu: [x64] + os: [linux] + + '@img/sharp-linux-arm64@0.35.2': + resolution: {integrity: sha512-af12Pnd0ZGu2HfP8NayB0kk6eC/lrfbQE6HlR4jD+34wdJ1Vw9TF6TMn6ZvffT+WgqVsl0hRbmNvz2u/23VmwA==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + + '@img/sharp-linux-arm@0.35.2': + resolution: {integrity: sha512-SE4kzF2mepn6z+6E7L6lsV8FzuLL6IPQdyX8ZiwROAG/G8td+hP/m7FsFPwidtrF19gvajuC9l6TxAVcsA4S7A==} + engines: {node: '>=20.9.0'} + cpu: [arm] + os: [linux] + + '@img/sharp-linux-ppc64@0.35.2': + resolution: {integrity: sha512-hYSBm7zcNtDCozCxQHYZJiu63b/bXsgRZuOxCIBZsStMM9Vap47iFHdbX4kCvQsblPB/k+clhELpdQJHQLSHvg==} + engines: {node: '>=20.9.0'} + cpu: [ppc64] + os: [linux] + + '@img/sharp-linux-riscv64@0.35.2': + resolution: {integrity: sha512-qQt0Kc13+Hoan/Awq/qMSQw3L+RI1NCRPgD5cUJ/1WSSmIoysLOc72jlRM3E0OHN9Yr313jgeQ2T+zW+F03QFA==} + engines: {node: '>=20.9.0'} + cpu: [riscv64] + os: [linux] + + '@img/sharp-linux-s390x@0.35.2': + resolution: {integrity: sha512-E4fLLfRPzDLlEeDaTzI98OFLcv++WL5ChLLMwPoVd0CIoZQqupBSNbOisPL5am9XsbQ9T84+iiMpUvbFtkunbA==} + engines: {node: '>=20.9.0'} + cpu: [s390x] + os: [linux] + + '@img/sharp-linux-x64@0.35.2': + resolution: {integrity: sha512-gi0zFJJRLswfCZmHtJdikXPOc5u7qamSOS3NHedLqLd4W8Q0NqjdBr6TTRIgsfFjqfTsHFgdfvJ9LwqSgcHiAA==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + + '@img/sharp-linuxmusl-arm64@0.35.2': + resolution: {integrity: sha512-siWbOW1u6HFnFLrp0waKyW7VEf7jYvcDWdrXEFa8AkdAQgEvuu5Fz8/Y70w9EeqAdwDtfU012BhEHHaDqvQNzg==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + + '@img/sharp-linuxmusl-x64@0.35.2': + resolution: {integrity: sha512-YBqMMcjDi4QGYiSn4vNOYBhmlC4z5AXqkOUUqI2e0AFA4urNv4ESgOgwNl3K+4etQhha0twXlzeF20bbULm9Yg==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + + '@img/sharp-wasm32@0.35.2': + resolution: {integrity: sha512-Mrv4JQNYVQ94xH+jzZ9r+gowleN8mv2FTgKT+PI6bx5C0G8TdNYndu161pg2i7uoBwxy2ImPMHrJOM2LZef7Bw==} + engines: {node: '>=20.9.0'} + + '@img/sharp-webcontainers-wasm32@0.35.2': + resolution: {integrity: sha512-QNV27pxs9wpApEiCfvHM1RDoP1w1+2KrUWWDPEhEwg+latvOrfuhWrHWZKwdSFwU6jh3myjw/yOCRsUIuOft3g==} + engines: {node: '>=20.9.0'} + cpu: [wasm32] + + '@img/sharp-win32-arm64@0.35.2': + resolution: {integrity: sha512-BiVRYc/t6/Vl3e1hBx0hugG4oN9Pydf4fgMSpxTQJmwGUg/YoXTWHiFeRymHfCZzifxu4F4rpk/I67D0LQ20wQ==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [win32] + + '@img/sharp-win32-ia32@0.35.2': + resolution: {integrity: sha512-YYEhx9PImCC7T0tI8JDMi4DB9LwLCXCU5OWNYEXAxh5Q1ShKkyC6byxzoBJ3gEFDnH2lQckWuDe70G7mB2XJog==} + engines: {node: ^20.9.0} + cpu: [ia32] + os: [win32] + + '@img/sharp-win32-x64@0.35.2': + resolution: {integrity: sha512-imoOyBcoM/iiUr4J6VPpCNjPnjvP/Gks95898yB8YqoGGYmHYbOyCuNv9FMhFgtaiHFGbHW8bxKqRV6VjtXThQ==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [win32] + + '@jridgewell/resolve-uri@3.1.2': + resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==} + engines: {node: '>=6.0.0'} + + '@jridgewell/sourcemap-codec@1.5.5': + resolution: {integrity: sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==} + + '@jridgewell/trace-mapping@0.3.9': + resolution: {integrity: sha512-3Belt6tdc8bPgAtbcmdtNJlirVoTmEb5e2gC94PnkwEW9jI6CAHUeoG85tjWP5WquqfavoMtMwiG4P926ZKKuQ==} + + '@oxc-project/types@0.142.0': + resolution: {integrity: sha512-7W+2q5AKQVU36fkaryontrHn3YDt1RyUYXatw9i5H8ocYe2sPKSFB6eS8WNPeRKiN1qAWWZUPm7gwFzJGrccqQ==} + + '@poppinss/colors@4.1.6': + resolution: {integrity: sha512-H9xkIdFswbS8n1d6vmRd8+c10t2Qe+rZITbbDHHkQixH5+2x1FDGmi/0K+WgWiqQFKPSlIYB7jlH6Kpfn6Fleg==} + + '@poppinss/dumper@0.6.5': + resolution: {integrity: sha512-NBdYIb90J7LfOI32dOewKI1r7wnkiH6m920puQ3qHUeZkxNkQiFnXVWoE6YtFSv6QOiPPf7ys6i+HWWecDz7sw==} + + '@poppinss/exception@1.2.3': + resolution: {integrity: sha512-dCED+QRChTVatE9ibtoaxc+WkdzOSjYTKi/+uacHWIsfodVfpsueo3+DKpgU5Px8qXjgmXkSvhXvSCz3fnP9lw==} + + '@rolldown/binding-android-arm64@1.2.2': + resolution: {integrity: sha512-l7x215OGvo1s52JWmR8U/DAVzEDWBCIbTm28aeJV/WDTSHgcKXaZTuBT0hJMs5NggilfJTW3clZVvd24yfKJxA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@rolldown/binding-darwin-arm64@1.2.2': + resolution: {integrity: sha512-9u9Xv6c1AJZT0FfwH5vrMG5Jjcwhc1MlyrPu0XfTqkzsmqfks2M6W/o5XwAJgVVN/jHpqqngC1WevHKKTIUtIA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@rolldown/binding-darwin-x64@1.2.2': + resolution: {integrity: sha512-9W1mbGZAfW3oqd85bhBkmpyHCCzL1TeG/zFFP3vg7b0rlly8cxOcre5nXwz+LHazCwac2MNWgPdPCHndABjpWQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@rolldown/binding-freebsd-x64@1.2.2': + resolution: {integrity: sha512-0p1lhiCSCyaerFwtrdZQUx7NqGk6LQnaRKWX7tFQqwQgvX0rjM15cIkm3pax1UpEakK14C4mOxx/jSqCBdBRqQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@rolldown/binding-linux-arm-gnueabihf@1.2.2': + resolution: {integrity: sha512-e+cOJXrJ2L3zx6YzqPg+f6Wbk3V1cKB8bOhbaYdVYN3DdquzNdRAmrbETz1qnt5yp/c7JNlNjmITiA2cVneQ7w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@rolldown/binding-linux-arm64-gnu@1.2.2': + resolution: {integrity: sha512-JsSMsj6sNat/MuhG5fnBD7QgbtpHKVe30x5/bAVirDHdhoQRXJkF6xc0Jqk8O4fiCUQAzMOoH9wZi3m60c8wtg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + + '@rolldown/binding-linux-arm64-musl@1.2.2': + resolution: {integrity: sha512-B5G/zJdHaoJn9vD50eGHWkiWfmq8Uhi3IiLPJTzmZTrAalk1bztUikSXo0qga18ibE0IXboyeMUnhPjhAJ45wQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + + '@rolldown/binding-linux-ppc64-gnu@1.2.2': + resolution: {integrity: sha512-6mC/awzKka8W6EoekjegpfGkjz8jXWDX63pqu/HYVpyKtZfu65Jsh4QAH3Kej3CAv/c1oGX7psTmFEbr0mDxLA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + + '@rolldown/binding-linux-s390x-gnu@1.2.2': + resolution: {integrity: sha512-412MX9fJLdA1IK28EZnc8jYv2HRTleOZgfLQumJ5zy7OeJLZlg/CETwFaXjNmGVxG51cFHpKLqb5LKvBC+HsHA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + + '@rolldown/binding-linux-x64-gnu@1.2.2': + resolution: {integrity: sha512-Q/+HI/ToJafZ1iCqGgVQXUEkIjufHCTF0gBQ2a5o3cg7GJ2h0qyq3nvvSmU+bGda2/7ygXpTY4TM6gO9OhQ0ZA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + + '@rolldown/binding-linux-x64-musl@1.2.2': + resolution: {integrity: sha512-ZKp/w41n6wCvxzxQHtQSbuphfX3Y4cCvbjkKHusrLx4lh+JWLTU7StSltO/DKARISzbj368d+qUaCjI8K2wzXw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + + '@rolldown/binding-openharmony-arm64@1.2.2': + resolution: {integrity: sha512-pxE6xD4KS3eAROkKK5yrhB9/3+vhlhVGMvlQLbdpzrBGDbKrnzx3RLwPaHvasLo6jgaiBLn7e04Df9C4tYhjmA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@rolldown/binding-win32-arm64-msvc@1.2.2': + resolution: {integrity: sha512-4MqEue5re+xIZzAWsB8sj0P1kqZySWqIuN4t6QaIO/YA6SFwySOLruvWQFzfmqk8LBK2P30KCSJwf6mJCZZ5/A==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@rolldown/binding-win32-x64-msvc@1.2.2': + resolution: {integrity: sha512-NweNxxD0Nf9t8v7kodun45Ijp3EIwYY+uydPP6qBEYvfBqhIjN6dZMzlQja3tqX/aLs3F3Uz+AxDpKgRhpOZQg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + + '@rolldown/pluginutils@1.0.1': + resolution: {integrity: sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==} + + '@sindresorhus/is@7.2.0': + resolution: {integrity: sha512-P1Cz1dWaFfR4IR+U13mqqiGsLFf1KbayybWwdd2vfctdV6hDpUkgCY0nKOLLTMSoRd/jJNjtbqzf13K8DCCXQw==} + engines: {node: '>=18'} + + '@speed-highlight/core@1.2.23': + resolution: {integrity: sha512-iRoq6i6JDJP6Mt2A5JaPvzw0pgYHH6k92ij+yXiTrB7T2y9N789aWE3EHWj/5ztlJBokcCBja3iYLVdu5wgnkg==} + + '@standard-schema/spec@1.1.0': + resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} + + '@types/chai@5.2.3': + resolution: {integrity: sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==} + + '@types/deep-eql@4.0.2': + resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==} + + '@types/estree@1.0.9': + resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + + '@types/node@24.2.0': + resolution: {integrity: sha512-3xyG3pMCq3oYCNg7/ZP+E1ooTaGB4cG8JWRsqqOYQdbWNY4zbaV0Ennrd7stjiJEFZCaybcIgpTjJWHRfBSIDw==} + + '@typescript/typescript-aix-ppc64@7.0.2': + resolution: {integrity: sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==} + engines: {node: '>=16.20.0'} + cpu: [ppc64] + os: [aix] + + '@typescript/typescript-darwin-arm64@7.0.2': + resolution: {integrity: sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [darwin] + + '@typescript/typescript-darwin-x64@7.0.2': + resolution: {integrity: sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [darwin] + + '@typescript/typescript-freebsd-arm64@7.0.2': + resolution: {integrity: sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [freebsd] + + '@typescript/typescript-freebsd-x64@7.0.2': + resolution: {integrity: sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [freebsd] + + '@typescript/typescript-linux-arm64@7.0.2': + resolution: {integrity: sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [linux] + + '@typescript/typescript-linux-arm@7.0.2': + resolution: {integrity: sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==} + engines: {node: '>=16.20.0'} + cpu: [arm] + os: [linux] + + '@typescript/typescript-linux-loong64@7.0.2': + resolution: {integrity: sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==} + engines: {node: '>=16.20.0'} + cpu: [loong64] + os: [linux] + + '@typescript/typescript-linux-mips64el@7.0.2': + resolution: {integrity: sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==} + engines: {node: '>=16.20.0'} + cpu: [mips64el] + os: [linux] + + '@typescript/typescript-linux-ppc64@7.0.2': + resolution: {integrity: sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==} + engines: {node: '>=16.20.0'} + cpu: [ppc64] + os: [linux] + + '@typescript/typescript-linux-riscv64@7.0.2': + resolution: {integrity: sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==} + engines: {node: '>=16.20.0'} + cpu: [riscv64] + os: [linux] + + '@typescript/typescript-linux-s390x@7.0.2': + resolution: {integrity: sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==} + engines: {node: '>=16.20.0'} + cpu: [s390x] + os: [linux] + + '@typescript/typescript-linux-x64@7.0.2': + resolution: {integrity: sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [linux] + + '@typescript/typescript-netbsd-arm64@7.0.2': + resolution: {integrity: sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [netbsd] + + '@typescript/typescript-netbsd-x64@7.0.2': + resolution: {integrity: sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [netbsd] + + '@typescript/typescript-openbsd-arm64@7.0.2': + resolution: {integrity: sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [openbsd] + + '@typescript/typescript-openbsd-x64@7.0.2': + resolution: {integrity: sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [openbsd] + + '@typescript/typescript-sunos-x64@7.0.2': + resolution: {integrity: sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [sunos] + + '@typescript/typescript-win32-arm64@7.0.2': + resolution: {integrity: sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [win32] + + '@typescript/typescript-win32-x64@7.0.2': + resolution: {integrity: sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [win32] + + '@vitest/expect@4.1.10': + resolution: {integrity: sha512-YsCn+qAk1GWjQOWFEsEcL2gNQ0zmVmQu3T03qP6UyjhtmdtwtbuI+DASn/7iQB3HGTXkdBwGddzxPlmiql5vlA==} + + '@vitest/mocker@4.1.10': + resolution: {integrity: sha512-v0xaezt+DKEmKfaxg133ldzADrwLGd7Ze1MfQQTYfvs8OqZIwbxyxaYURivwV7sWy5fqn3rH5uOrSp07bp44Ow==} + peerDependencies: + msw: ^2.4.9 + vite: ^6.0.0 || ^7.0.0 || ^8.0.0 + peerDependenciesMeta: + msw: + optional: true + vite: + optional: true + + '@vitest/pretty-format@4.1.10': + resolution: {integrity: sha512-W1HsjSH4MXQ9YfmmhLAoIYf1HRfekQCGngeIgcei6MP5QQGWUe0gkopdZQaVCFO+JDJMrAJGwa5pRpNpvy4P8Q==} + + '@vitest/runner@4.1.10': + resolution: {integrity: sha512-IKI6kpIH+LmpROplyLwBBaCfMgOZOMsygVa6BARD6ahA04VRuJSa6OaVG7kRvSEMD870Vd91rSSw0eegtWyLGg==} + + '@vitest/snapshot@4.1.10': + resolution: {integrity: sha512-xRkfOT1qpTAi/Ti4Y1LtfRc3kEuqxGw59eN2jN9pRWMtS/XDevekhcFSqvQqjUNGksfjMJu3Y+oJ+4Ypn2OaJw==} + + '@vitest/spy@4.1.10': + resolution: {integrity: sha512-PLf/Ugvoq5wO/b4rwYCR1h2PSIdXz7wnkQFMiUpLdtM7l6pqVFcQIBEHyT1+l+cj7mNwAfZHzqXqDyjvOuwbDw==} + + '@vitest/utils@4.1.10': + resolution: {integrity: sha512-fy9am/HWxbaGt/Sawrp90vt6Y6jQwf1RX77cz3uwoJwJVMli/e1IEwRPnMNJ7vKfPTwo0diXifkpPvwH9v7nGA==} + + assertion-error@2.0.1: + resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==} + engines: {node: '>=12'} + + blake3-wasm@2.1.5: + resolution: {integrity: sha512-F1+K8EbfOZE49dtoPtmxUQrpXaBIl3ICvasLh+nJta0xkz+9kF/7uet9fLnwKqhDrmj6g+6K3Tw9yQPUg2ka5g==} + + chai@6.2.2: + resolution: {integrity: sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==} + engines: {node: '>=18'} + + cjs-module-lexer@1.2.3: + resolution: {integrity: sha512-0TNiGstbQmCFwt4akjjBg5pLRTSyj/PkWQ1ZoO2zntmg9yLqSRxwEa4iCfQLGjqhiqBfOJa7W/E8wfGrTDmlZQ==} + + convert-source-map@2.0.0: + resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} + + cookie@1.1.1: + resolution: {integrity: sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==} + engines: {node: '>=18'} + + detect-libc@2.1.2: + resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==} + engines: {node: '>=8'} + + error-stack-parser-es@1.0.5: + resolution: {integrity: sha512-5qucVt2XcuGMcEGgWI7i+yZpmpByQ8J1lHhcL7PwqCwu9FPP3VUXzT4ltHe5i2z9dePwEHcDVOAfSnHsOlCXRA==} + + es-module-lexer@2.3.1: + resolution: {integrity: sha512-shc1dbU90Yl/xq1QrC7QRtfcwURZuVRfPhZbDoldJ1cn1gzDvBaBWlv0eFolj5+0znnPJz5TXLxsN77X/12KTA==} + + esbuild@0.28.1: + resolution: {integrity: sha512-HrJrvZv5ayxBzPfwphOoNzkzOIIlifzk0KJrGK2c8R4+LKpMtpYLQeUdjnwjWv/LZlkH2laZk+4w78pi99D4Vw==} + engines: {node: '>=18'} + hasBin: true + + estree-walker@3.0.3: + resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==} + + expect-type@1.4.0: + resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==} + engines: {node: '>=12.0.0'} + + fdir@6.5.0: + resolution: {integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==} + engines: {node: '>=12.0.0'} + peerDependencies: + picomatch: ^3 || ^4 + peerDependenciesMeta: + picomatch: + optional: true + + fsevents@2.3.3: + resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + + kleur@4.1.5: + resolution: {integrity: sha512-o+NO+8WrRiQEE4/7nwRJhN1HWpVmJm511pBHUxPLtp0BUISzlBplORYSmTclCnJvQq2tKu/sgl3xVpkc7ZWuQQ==} + engines: {node: '>=6'} + + lightningcss-android-arm64@1.33.0: + resolution: {integrity: sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [android] + + lightningcss-darwin-arm64@1.33.0: + resolution: {integrity: sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [darwin] + + lightningcss-darwin-x64@1.33.0: + resolution: {integrity: sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [darwin] + + lightningcss-freebsd-x64@1.33.0: + resolution: {integrity: sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [freebsd] + + lightningcss-linux-arm-gnueabihf@1.33.0: + resolution: {integrity: sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==} + engines: {node: '>= 12.0.0'} + cpu: [arm] + os: [linux] + + lightningcss-linux-arm64-gnu@1.33.0: + resolution: {integrity: sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [linux] + + lightningcss-linux-arm64-musl@1.33.0: + resolution: {integrity: sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [linux] + + lightningcss-linux-x64-gnu@1.33.0: + resolution: {integrity: sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [linux] + + lightningcss-linux-x64-musl@1.33.0: + resolution: {integrity: sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [linux] + + lightningcss-win32-arm64-msvc@1.33.0: + resolution: {integrity: sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==} + engines: {node: '>= 12.0.0'} + cpu: [arm64] + os: [win32] + + lightningcss-win32-x64-msvc@1.33.0: + resolution: {integrity: sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==} + engines: {node: '>= 12.0.0'} + cpu: [x64] + os: [win32] + + lightningcss@1.33.0: + resolution: {integrity: sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==} + engines: {node: '>= 12.0.0'} + + magic-string@0.30.21: + resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==} + + miniflare@5.20260730.0-alpha: + resolution: {integrity: sha512-8/dspSXDshP6nSkCpjKO7BYc2qZoYSXm7iM+QxY7qJyJpAB3onnQSaiu0cvKJlfuMGwULl55hG69FJCcCMXU1Q==} + engines: {node: '>=22.0.0'} + + nanoid@3.3.17: + resolution: {integrity: sha512-xQLf0A3HOMlgHq0n247/LRuAOYmB7dXJ/DvAxGvsSBij45XtBSmQycu+F8ODbHwns/XyFZagyL1+J0Offw1E0g==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + + obug@2.1.4: + resolution: {integrity: sha512-4a+OsYv9UktOJKE+l1A4OufDgdRF9PifWj+tJnHURo/P+WOxpG4GzUFL9qCalmWauao6ogiG+QvnCovwPoyAWA==} + engines: {node: '>=12.20.0'} + + path-to-regexp@6.3.0: + resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==} + + pathe@2.0.3: + resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==} + + picocolors@1.1.1: + resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} + + picomatch@4.0.5: + resolution: {integrity: sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==} + engines: {node: '>=12'} + + postcss@8.5.25: + resolution: {integrity: sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw==} + engines: {node: ^10 || ^12 || >=14} + + rolldown@1.2.2: + resolution: {integrity: sha512-opwpo1tQBAcpSUJDt94B7hhLNGOKjCdE//XXjeLrnx9b83bjnw45tXdg1b09yEw/VLFBJGZpwRULMmOZo7ol+A==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + + semver@7.8.5: + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==} + engines: {node: '>=10'} + hasBin: true + + sharp@0.35.2: + resolution: {integrity: sha512-FVtFjtBCMiJS6yb5CX7Sop45WFMpeGw6oRKuJnXYgf/f1ms/D7LE/ZUSNxnW7rZ/dbslQWYkoqFHGPaDBtaK4w==} + engines: {node: '>=20.9.0'} + + siginfo@2.0.0: + resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==} + + source-map-js@1.2.1: + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} + engines: {node: '>=0.10.0'} + + stackback@0.0.2: + resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} + + std-env@4.2.0: + resolution: {integrity: sha512-oCUKSupKTHX53EyjDtuZQ64pjLJ6yYCtpmEw0goYxtjG9KpbRe8KAsl2tBUGU9DyMcJ0RwJ8GqJAFzMXcXW1Rw==} + + supports-color@10.2.2: + resolution: {integrity: sha512-SS+jx45GF1QjgEXQx4NJZV9ImqmO2NPz5FNsIHrsDjh2YsHnawpan7SNQ1o8NuhrbHZy9AZhIoCUiCeaW/C80g==} + engines: {node: '>=18'} + + tinybench@2.9.0: + resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==} + + tinyexec@1.3.0: + resolution: {integrity: sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==} + engines: {node: '>=18'} + + tinyglobby@0.2.17: + resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} + engines: {node: '>=12.0.0'} + + tinyrainbow@3.1.1: + resolution: {integrity: sha512-yau8yJdTt989Mm0Bd/236QnzEiPf2xLLTqUZRUJOo/3CB078LSwzei343DgtJVmfJKJE3TMINY1u42SQsP6mXw==} + engines: {node: '>=14.0.0'} + + tslib@2.8.1: + resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} + + typescript@7.0.2: + resolution: {integrity: sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==} + engines: {node: '>=16.20.0'} + hasBin: true + + undici-types@7.10.0: + resolution: {integrity: sha512-t5Fy/nfn+14LuOc2KNYg75vZqClpAiqscVvMygNnlsHBFpSXdJaYtXMcdNLpl/Qvc3P2cB3s6lOV51nqsFq4ag==} + + undici@7.28.0: + resolution: {integrity: sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==} + engines: {node: '>=20.18.1'} + + unenv@2.0.0-rc.24: + resolution: {integrity: sha512-i7qRCmY42zmCwnYlh9H2SvLEypEFGye5iRmEMKjcGi7zk9UquigRjFtTLz0TYqr0ZGLZhaMHl/foy1bZR+Cwlw==} + + vite@8.2.0: + resolution: {integrity: sha512-pn+CFpM0lwDeKwmOq1ZaBK/9sjorZcgqxki6MbY/jPEVd9vichIlmlD4HmQ5wdP5EgqQCFRaACBxMC7uEGc6lQ==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + peerDependencies: + '@types/node': ^20.19.0 || >=22.12.0 + '@vitejs/devtools': ^0.4.0 + esbuild: ^0.27.0 || ^0.28.0 + jiti: '>=1.21.0' + less: ^4.0.0 + sass: ^1.70.0 + sass-embedded: ^1.70.0 + stylus: '>=0.54.8' + sugarss: ^5.0.0 + terser: ^5.16.0 + tsx: ^4.8.1 + yaml: ^2.4.2 + peerDependenciesMeta: + '@types/node': + optional: true + '@vitejs/devtools': + optional: true + esbuild: + optional: true + jiti: + optional: true + less: + optional: true + sass: + optional: true + sass-embedded: + optional: true + stylus: + optional: true + sugarss: + optional: true + terser: + optional: true + tsx: + optional: true + yaml: + optional: true + + vitest@4.1.10: + resolution: {integrity: sha512-R9jUTe5S4Qb0HCd4TNqpC7oGcrMssMRGXLW80ubjWsW9VH5GF8y1Y0SFLY9AbqSk6nt0PnOx4H4WNJYZ13GUPw==} + engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} + hasBin: true + peerDependencies: + '@edge-runtime/vm': '*' + '@opentelemetry/api': ^1.9.0 + '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 + '@vitest/browser-playwright': 4.1.10 + '@vitest/browser-preview': 4.1.10 + '@vitest/browser-webdriverio': 4.1.10 + '@vitest/coverage-istanbul': 4.1.10 + '@vitest/coverage-v8': 4.1.10 + '@vitest/ui': 4.1.10 + happy-dom: '*' + jsdom: '*' + vite: ^6.0.0 || ^7.0.0 || ^8.0.0 + peerDependenciesMeta: + '@edge-runtime/vm': + optional: true + '@opentelemetry/api': + optional: true + '@types/node': + optional: true + '@vitest/browser-playwright': + optional: true + '@vitest/browser-preview': + optional: true + '@vitest/browser-webdriverio': + optional: true + '@vitest/coverage-istanbul': + optional: true + '@vitest/coverage-v8': + optional: true + '@vitest/ui': + optional: true + happy-dom: + optional: true + jsdom: + optional: true + + why-is-node-running@2.3.0: + resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==} + engines: {node: '>=8'} + hasBin: true + + workerd@1.20260730.1: + resolution: {integrity: sha512-zmfNIjwYSWFY5chGBOjWtH3xAE7p97FTC6vR4Ep98290ho6AeAR/NVcBD274YCLEUYzqm8yxdtZlxMybU8a3jA==} + engines: {node: '>=16'} + hasBin: true + + wrangler@4.118.0: + resolution: {integrity: sha512-9pkBw/b8zWqGx2S+oLhgHMR1M/4VOE8SynUFABnGWiSFGlcOQ4xiI/B71Xf66RYP2xzngU37IQFPtUruij3lYw==} + engines: {node: '>=22.0.0'} + hasBin: true + peerDependencies: + '@cloudflare/workers-types': ^5.20260730.1 + peerDependenciesMeta: + '@cloudflare/workers-types': + optional: true + + ws@8.21.0: + resolution: {integrity: sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==} + engines: {node: '>=10.0.0'} + peerDependencies: + bufferutil: ^4.0.1 + utf-8-validate: '>=5.0.2' + peerDependenciesMeta: + bufferutil: + optional: true + utf-8-validate: + optional: true + + youch-core@0.3.3: + resolution: {integrity: sha512-ho7XuGjLaJ2hWHoK8yFnsUGy2Y5uDpqSTq1FkHLK4/oqKtyUU1AFbOOxY4IpC9f0fTLjwYbslUz0Po5BpD1wrA==} + + youch@4.1.0-beta.10: + resolution: {integrity: sha512-rLfVLB4FgQneDr0dv1oddCVZmKjcJ6yX6mS4pU82Mq/Dt9a3cLZQ62pDBL4AUO+uVrCvtWz3ZFUL2HFAFJ/BXQ==} + + zod@4.4.3: + resolution: {integrity: sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==} + +snapshots: + + '@cloudflare/kv-asset-handler@0.5.0': {} + + '@cloudflare/unenv-preset@2.16.1(unenv@2.0.0-rc.24)(workerd@1.20260730.1)': + dependencies: + unenv: 2.0.0-rc.24 + optionalDependencies: + workerd: 1.20260730.1 + + '@cloudflare/vitest-pool-workers@0.20.1(@cloudflare/workers-types@5.20260801.1)(@vitest/runner@4.1.10)(@vitest/snapshot@4.1.10)(vitest@4.1.10(@types/node@24.2.0)(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1)))': + dependencies: + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + cjs-module-lexer: 1.2.3 + esbuild: 0.28.1 + miniflare: 5.20260730.0-alpha + vitest: 4.1.10(@types/node@24.2.0)(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1)) + wrangler: 4.118.0(@cloudflare/workers-types@5.20260801.1) + zod: 4.4.3 + transitivePeerDependencies: + - '@cloudflare/workers-types' + - bufferutil + - utf-8-validate + + '@cloudflare/workerd-darwin-64@1.20260730.1': + optional: true + + '@cloudflare/workerd-darwin-arm64@1.20260730.1': + optional: true + + '@cloudflare/workerd-linux-64@1.20260730.1': + optional: true + + '@cloudflare/workerd-linux-arm64@1.20260730.1': + optional: true + + '@cloudflare/workerd-windows-64@1.20260730.1': + optional: true + + '@cloudflare/workers-types@5.20260801.1': {} + + '@cspotcode/source-map-support@0.8.1': + dependencies: + '@jridgewell/trace-mapping': 0.3.9 + + '@emnapi/runtime@1.11.3': + dependencies: + tslib: 2.8.1 + optional: true + + '@esbuild/aix-ppc64@0.28.1': + optional: true + + '@esbuild/android-arm64@0.28.1': + optional: true + + '@esbuild/android-arm@0.28.1': + optional: true + + '@esbuild/android-x64@0.28.1': + optional: true + + '@esbuild/darwin-arm64@0.28.1': + optional: true + + '@esbuild/darwin-x64@0.28.1': + optional: true + + '@esbuild/freebsd-arm64@0.28.1': + optional: true + + '@esbuild/freebsd-x64@0.28.1': + optional: true + + '@esbuild/linux-arm64@0.28.1': + optional: true + + '@esbuild/linux-arm@0.28.1': + optional: true + + '@esbuild/linux-ia32@0.28.1': + optional: true + + '@esbuild/linux-loong64@0.28.1': + optional: true + + '@esbuild/linux-mips64el@0.28.1': + optional: true + + '@esbuild/linux-ppc64@0.28.1': + optional: true + + '@esbuild/linux-riscv64@0.28.1': + optional: true + + '@esbuild/linux-s390x@0.28.1': + optional: true + + '@esbuild/linux-x64@0.28.1': + optional: true + + '@esbuild/netbsd-arm64@0.28.1': + optional: true + + '@esbuild/netbsd-x64@0.28.1': + optional: true + + '@esbuild/openbsd-arm64@0.28.1': + optional: true + + '@esbuild/openbsd-x64@0.28.1': + optional: true + + '@esbuild/openharmony-arm64@0.28.1': + optional: true + + '@esbuild/sunos-x64@0.28.1': + optional: true + + '@esbuild/win32-arm64@0.28.1': + optional: true + + '@esbuild/win32-ia32@0.28.1': + optional: true + + '@esbuild/win32-x64@0.28.1': + optional: true + + '@img/colour@1.1.0': {} + + '@img/sharp-darwin-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-darwin-arm64': 1.3.1 + optional: true + + '@img/sharp-darwin-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-darwin-x64': 1.3.1 + optional: true + + '@img/sharp-freebsd-wasm32@0.35.2': + dependencies: + '@img/sharp-wasm32': 0.35.2 + optional: true + + '@img/sharp-libvips-darwin-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-darwin-x64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-arm@1.3.1': + optional: true + + '@img/sharp-libvips-linux-ppc64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-riscv64@1.3.1': + optional: true + + '@img/sharp-libvips-linux-s390x@1.3.1': + optional: true + + '@img/sharp-libvips-linux-x64@1.3.1': + optional: true + + '@img/sharp-libvips-linuxmusl-arm64@1.3.1': + optional: true + + '@img/sharp-libvips-linuxmusl-x64@1.3.1': + optional: true + + '@img/sharp-linux-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-arm64': 1.3.1 + optional: true + + '@img/sharp-linux-arm@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-arm': 1.3.1 + optional: true + + '@img/sharp-linux-ppc64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-ppc64': 1.3.1 + optional: true + + '@img/sharp-linux-riscv64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-riscv64': 1.3.1 + optional: true + + '@img/sharp-linux-s390x@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-s390x': 1.3.1 + optional: true + + '@img/sharp-linux-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linux-x64': 1.3.1 + optional: true + + '@img/sharp-linuxmusl-arm64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-arm64': 1.3.1 + optional: true + + '@img/sharp-linuxmusl-x64@0.35.2': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-x64': 1.3.1 + optional: true + + '@img/sharp-wasm32@0.35.2': + dependencies: + '@emnapi/runtime': 1.11.3 + optional: true + + '@img/sharp-webcontainers-wasm32@0.35.2': + dependencies: + '@img/sharp-wasm32': 0.35.2 + optional: true + + '@img/sharp-win32-arm64@0.35.2': + optional: true + + '@img/sharp-win32-ia32@0.35.2': + optional: true + + '@img/sharp-win32-x64@0.35.2': + optional: true + + '@jridgewell/resolve-uri@3.1.2': {} + + '@jridgewell/sourcemap-codec@1.5.5': {} + + '@jridgewell/trace-mapping@0.3.9': + dependencies: + '@jridgewell/resolve-uri': 3.1.2 + '@jridgewell/sourcemap-codec': 1.5.5 + + '@oxc-project/types@0.142.0': {} + + '@poppinss/colors@4.1.6': + dependencies: + kleur: 4.1.5 + + '@poppinss/dumper@0.6.5': + dependencies: + '@poppinss/colors': 4.1.6 + '@sindresorhus/is': 7.2.0 + supports-color: 10.2.2 + + '@poppinss/exception@1.2.3': {} + + '@rolldown/binding-android-arm64@1.2.2': + optional: true + + '@rolldown/binding-darwin-arm64@1.2.2': + optional: true + + '@rolldown/binding-darwin-x64@1.2.2': + optional: true + + '@rolldown/binding-freebsd-x64@1.2.2': + optional: true + + '@rolldown/binding-linux-arm-gnueabihf@1.2.2': + optional: true + + '@rolldown/binding-linux-arm64-gnu@1.2.2': + optional: true + + '@rolldown/binding-linux-arm64-musl@1.2.2': + optional: true + + '@rolldown/binding-linux-ppc64-gnu@1.2.2': + optional: true + + '@rolldown/binding-linux-s390x-gnu@1.2.2': + optional: true + + '@rolldown/binding-linux-x64-gnu@1.2.2': + optional: true + + '@rolldown/binding-linux-x64-musl@1.2.2': + optional: true + + '@rolldown/binding-openharmony-arm64@1.2.2': + optional: true + + '@rolldown/binding-win32-arm64-msvc@1.2.2': + optional: true + + '@rolldown/binding-win32-x64-msvc@1.2.2': + optional: true + + '@rolldown/pluginutils@1.0.1': {} + + '@sindresorhus/is@7.2.0': {} + + '@speed-highlight/core@1.2.23': {} + + '@standard-schema/spec@1.1.0': {} + + '@types/chai@5.2.3': + dependencies: + '@types/deep-eql': 4.0.2 + assertion-error: 2.0.1 + + '@types/deep-eql@4.0.2': {} + + '@types/estree@1.0.9': {} + + '@types/node@24.2.0': + dependencies: + undici-types: 7.10.0 + + '@typescript/typescript-aix-ppc64@7.0.2': + optional: true + + '@typescript/typescript-darwin-arm64@7.0.2': + optional: true + + '@typescript/typescript-darwin-x64@7.0.2': + optional: true + + '@typescript/typescript-freebsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-freebsd-x64@7.0.2': + optional: true + + '@typescript/typescript-linux-arm64@7.0.2': + optional: true + + '@typescript/typescript-linux-arm@7.0.2': + optional: true + + '@typescript/typescript-linux-loong64@7.0.2': + optional: true + + '@typescript/typescript-linux-mips64el@7.0.2': + optional: true + + '@typescript/typescript-linux-ppc64@7.0.2': + optional: true + + '@typescript/typescript-linux-riscv64@7.0.2': + optional: true + + '@typescript/typescript-linux-s390x@7.0.2': + optional: true + + '@typescript/typescript-linux-x64@7.0.2': + optional: true + + '@typescript/typescript-netbsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-netbsd-x64@7.0.2': + optional: true + + '@typescript/typescript-openbsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-openbsd-x64@7.0.2': + optional: true + + '@typescript/typescript-sunos-x64@7.0.2': + optional: true + + '@typescript/typescript-win32-arm64@7.0.2': + optional: true + + '@typescript/typescript-win32-x64@7.0.2': + optional: true + + '@vitest/expect@4.1.10': + dependencies: + '@standard-schema/spec': 1.1.0 + '@types/chai': 5.2.3 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 + chai: 6.2.2 + tinyrainbow: 3.1.1 + + '@vitest/mocker@4.1.10(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1))': + dependencies: + '@vitest/spy': 4.1.10 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 8.2.0(@types/node@24.2.0)(esbuild@0.28.1) + + '@vitest/pretty-format@4.1.10': + dependencies: + tinyrainbow: 3.1.1 + + '@vitest/runner@4.1.10': + dependencies: + '@vitest/utils': 4.1.10 + pathe: 2.0.3 + + '@vitest/snapshot@4.1.10': + dependencies: + '@vitest/pretty-format': 4.1.10 + '@vitest/utils': 4.1.10 + magic-string: 0.30.21 + pathe: 2.0.3 + + '@vitest/spy@4.1.10': {} + + '@vitest/utils@4.1.10': + dependencies: + '@vitest/pretty-format': 4.1.10 + convert-source-map: 2.0.0 + tinyrainbow: 3.1.1 + + assertion-error@2.0.1: {} + + blake3-wasm@2.1.5: {} + + chai@6.2.2: {} + + cjs-module-lexer@1.2.3: {} + + convert-source-map@2.0.0: {} + + cookie@1.1.1: {} + + detect-libc@2.1.2: {} + + error-stack-parser-es@1.0.5: {} + + es-module-lexer@2.3.1: {} + + esbuild@0.28.1: + optionalDependencies: + '@esbuild/aix-ppc64': 0.28.1 + '@esbuild/android-arm': 0.28.1 + '@esbuild/android-arm64': 0.28.1 + '@esbuild/android-x64': 0.28.1 + '@esbuild/darwin-arm64': 0.28.1 + '@esbuild/darwin-x64': 0.28.1 + '@esbuild/freebsd-arm64': 0.28.1 + '@esbuild/freebsd-x64': 0.28.1 + '@esbuild/linux-arm': 0.28.1 + '@esbuild/linux-arm64': 0.28.1 + '@esbuild/linux-ia32': 0.28.1 + '@esbuild/linux-loong64': 0.28.1 + '@esbuild/linux-mips64el': 0.28.1 + '@esbuild/linux-ppc64': 0.28.1 + '@esbuild/linux-riscv64': 0.28.1 + '@esbuild/linux-s390x': 0.28.1 + '@esbuild/linux-x64': 0.28.1 + '@esbuild/netbsd-arm64': 0.28.1 + '@esbuild/netbsd-x64': 0.28.1 + '@esbuild/openbsd-arm64': 0.28.1 + '@esbuild/openbsd-x64': 0.28.1 + '@esbuild/openharmony-arm64': 0.28.1 + '@esbuild/sunos-x64': 0.28.1 + '@esbuild/win32-arm64': 0.28.1 + '@esbuild/win32-ia32': 0.28.1 + '@esbuild/win32-x64': 0.28.1 + + estree-walker@3.0.3: + dependencies: + '@types/estree': 1.0.9 + + expect-type@1.4.0: {} + + fdir@6.5.0(picomatch@4.0.5): + optionalDependencies: + picomatch: 4.0.5 + + fsevents@2.3.3: + optional: true + + kleur@4.1.5: {} + + lightningcss-android-arm64@1.33.0: + optional: true + + lightningcss-darwin-arm64@1.33.0: + optional: true + + lightningcss-darwin-x64@1.33.0: + optional: true + + lightningcss-freebsd-x64@1.33.0: + optional: true + + lightningcss-linux-arm-gnueabihf@1.33.0: + optional: true + + lightningcss-linux-arm64-gnu@1.33.0: + optional: true + + lightningcss-linux-arm64-musl@1.33.0: + optional: true + + lightningcss-linux-x64-gnu@1.33.0: + optional: true + + lightningcss-linux-x64-musl@1.33.0: + optional: true + + lightningcss-win32-arm64-msvc@1.33.0: + optional: true + + lightningcss-win32-x64-msvc@1.33.0: + optional: true + + lightningcss@1.33.0: + dependencies: + detect-libc: 2.1.2 + optionalDependencies: + lightningcss-android-arm64: 1.33.0 + lightningcss-darwin-arm64: 1.33.0 + lightningcss-darwin-x64: 1.33.0 + lightningcss-freebsd-x64: 1.33.0 + lightningcss-linux-arm-gnueabihf: 1.33.0 + lightningcss-linux-arm64-gnu: 1.33.0 + lightningcss-linux-arm64-musl: 1.33.0 + lightningcss-linux-x64-gnu: 1.33.0 + lightningcss-linux-x64-musl: 1.33.0 + lightningcss-win32-arm64-msvc: 1.33.0 + lightningcss-win32-x64-msvc: 1.33.0 + + magic-string@0.30.21: + dependencies: + '@jridgewell/sourcemap-codec': 1.5.5 + + miniflare@5.20260730.0-alpha: + dependencies: + '@cspotcode/source-map-support': 0.8.1 + sharp: 0.35.2 + undici: 7.28.0 + workerd: 1.20260730.1 + ws: 8.21.0 + youch: 4.1.0-beta.10 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + + nanoid@3.3.17: {} + + obug@2.1.4: {} + + path-to-regexp@6.3.0: {} + + pathe@2.0.3: {} + + picocolors@1.1.1: {} + + picomatch@4.0.5: {} + + postcss@8.5.25: + dependencies: + nanoid: 3.3.17 + picocolors: 1.1.1 + source-map-js: 1.2.1 + + rolldown@1.2.2: + dependencies: + '@oxc-project/types': 0.142.0 + '@rolldown/pluginutils': 1.0.1 + optionalDependencies: + '@rolldown/binding-android-arm64': 1.2.2 + '@rolldown/binding-darwin-arm64': 1.2.2 + '@rolldown/binding-darwin-x64': 1.2.2 + '@rolldown/binding-freebsd-x64': 1.2.2 + '@rolldown/binding-linux-arm-gnueabihf': 1.2.2 + '@rolldown/binding-linux-arm64-gnu': 1.2.2 + '@rolldown/binding-linux-arm64-musl': 1.2.2 + '@rolldown/binding-linux-ppc64-gnu': 1.2.2 + '@rolldown/binding-linux-s390x-gnu': 1.2.2 + '@rolldown/binding-linux-x64-gnu': 1.2.2 + '@rolldown/binding-linux-x64-musl': 1.2.2 + '@rolldown/binding-openharmony-arm64': 1.2.2 + '@rolldown/binding-win32-arm64-msvc': 1.2.2 + '@rolldown/binding-win32-x64-msvc': 1.2.2 + + semver@7.8.5: {} + + sharp@0.35.2: + dependencies: + '@img/colour': 1.1.0 + detect-libc: 2.1.2 + semver: 7.8.5 + optionalDependencies: + '@img/sharp-darwin-arm64': 0.35.2 + '@img/sharp-darwin-x64': 0.35.2 + '@img/sharp-freebsd-wasm32': 0.35.2 + '@img/sharp-libvips-darwin-arm64': 1.3.1 + '@img/sharp-libvips-darwin-x64': 1.3.1 + '@img/sharp-libvips-linux-arm': 1.3.1 + '@img/sharp-libvips-linux-arm64': 1.3.1 + '@img/sharp-libvips-linux-ppc64': 1.3.1 + '@img/sharp-libvips-linux-riscv64': 1.3.1 + '@img/sharp-libvips-linux-s390x': 1.3.1 + '@img/sharp-libvips-linux-x64': 1.3.1 + '@img/sharp-libvips-linuxmusl-arm64': 1.3.1 + '@img/sharp-libvips-linuxmusl-x64': 1.3.1 + '@img/sharp-linux-arm': 0.35.2 + '@img/sharp-linux-arm64': 0.35.2 + '@img/sharp-linux-ppc64': 0.35.2 + '@img/sharp-linux-riscv64': 0.35.2 + '@img/sharp-linux-s390x': 0.35.2 + '@img/sharp-linux-x64': 0.35.2 + '@img/sharp-linuxmusl-arm64': 0.35.2 + '@img/sharp-linuxmusl-x64': 0.35.2 + '@img/sharp-webcontainers-wasm32': 0.35.2 + '@img/sharp-win32-arm64': 0.35.2 + '@img/sharp-win32-ia32': 0.35.2 + '@img/sharp-win32-x64': 0.35.2 + + siginfo@2.0.0: {} + + source-map-js@1.2.1: {} + + stackback@0.0.2: {} + + std-env@4.2.0: {} + + supports-color@10.2.2: {} + + tinybench@2.9.0: {} + + tinyexec@1.3.0: {} + + tinyglobby@0.2.17: + dependencies: + fdir: 6.5.0(picomatch@4.0.5) + picomatch: 4.0.5 + + tinyrainbow@3.1.1: {} + + tslib@2.8.1: + optional: true + + typescript@7.0.2: + optionalDependencies: + '@typescript/typescript-aix-ppc64': 7.0.2 + '@typescript/typescript-darwin-arm64': 7.0.2 + '@typescript/typescript-darwin-x64': 7.0.2 + '@typescript/typescript-freebsd-arm64': 7.0.2 + '@typescript/typescript-freebsd-x64': 7.0.2 + '@typescript/typescript-linux-arm': 7.0.2 + '@typescript/typescript-linux-arm64': 7.0.2 + '@typescript/typescript-linux-loong64': 7.0.2 + '@typescript/typescript-linux-mips64el': 7.0.2 + '@typescript/typescript-linux-ppc64': 7.0.2 + '@typescript/typescript-linux-riscv64': 7.0.2 + '@typescript/typescript-linux-s390x': 7.0.2 + '@typescript/typescript-linux-x64': 7.0.2 + '@typescript/typescript-netbsd-arm64': 7.0.2 + '@typescript/typescript-netbsd-x64': 7.0.2 + '@typescript/typescript-openbsd-arm64': 7.0.2 + '@typescript/typescript-openbsd-x64': 7.0.2 + '@typescript/typescript-sunos-x64': 7.0.2 + '@typescript/typescript-win32-arm64': 7.0.2 + '@typescript/typescript-win32-x64': 7.0.2 + + undici-types@7.10.0: {} + + undici@7.28.0: {} + + unenv@2.0.0-rc.24: + dependencies: + pathe: 2.0.3 + + vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1): + dependencies: + lightningcss: 1.33.0 + picomatch: 4.0.5 + postcss: 8.5.25 + rolldown: 1.2.2 + tinyglobby: 0.2.17 + optionalDependencies: + '@types/node': 24.2.0 + esbuild: 0.28.1 + fsevents: 2.3.3 + + vitest@4.1.10(@types/node@24.2.0)(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1)): + dependencies: + '@vitest/expect': 4.1.10 + '@vitest/mocker': 4.1.10(vite@8.2.0(@types/node@24.2.0)(esbuild@0.28.1)) + '@vitest/pretty-format': 4.1.10 + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 + es-module-lexer: 2.3.1 + expect-type: 1.4.0 + magic-string: 0.30.21 + obug: 2.1.4 + pathe: 2.0.3 + picomatch: 4.0.5 + std-env: 4.2.0 + tinybench: 2.9.0 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + tinyrainbow: 3.1.1 + vite: 8.2.0(@types/node@24.2.0)(esbuild@0.28.1) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 24.2.0 + transitivePeerDependencies: + - msw + + why-is-node-running@2.3.0: + dependencies: + siginfo: 2.0.0 + stackback: 0.0.2 + + workerd@1.20260730.1: + optionalDependencies: + '@cloudflare/workerd-darwin-64': 1.20260730.1 + '@cloudflare/workerd-darwin-arm64': 1.20260730.1 + '@cloudflare/workerd-linux-64': 1.20260730.1 + '@cloudflare/workerd-linux-arm64': 1.20260730.1 + '@cloudflare/workerd-windows-64': 1.20260730.1 + + wrangler@4.118.0(@cloudflare/workers-types@5.20260801.1): + dependencies: + '@cloudflare/kv-asset-handler': 0.5.0 + '@cloudflare/unenv-preset': 2.16.1(unenv@2.0.0-rc.24)(workerd@1.20260730.1) + blake3-wasm: 2.1.5 + esbuild: 0.28.1 + miniflare: 5.20260730.0-alpha + path-to-regexp: 6.3.0 + unenv: 2.0.0-rc.24 + workerd: 1.20260730.1 + optionalDependencies: + '@cloudflare/workers-types': 5.20260801.1 + fsevents: 2.3.3 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + + ws@8.21.0: {} + + youch-core@0.3.3: + dependencies: + '@poppinss/exception': 1.2.3 + error-stack-parser-es: 1.0.5 + + youch@4.1.0-beta.10: + dependencies: + '@poppinss/colors': 4.1.6 + '@poppinss/dumper': 0.6.5 + '@speed-highlight/core': 1.2.23 + cookie: 1.1.1 + youch-core: 0.3.3 + + zod@4.4.3: {} diff --git a/receiver/src/contract.ts b/receiver/src/contract.ts new file mode 100644 index 0000000..4a0d82e --- /dev/null +++ b/receiver/src/contract.ts @@ -0,0 +1,345 @@ +export const BUNDLE_FORMAT = "research-bundle-v1"; +export const BUNDLE_MEDIA_TYPE = "application/vnd.adc.research-bundle"; +export const MAXIMUM_BODY_BYTES = 32 * 1024 * 1024; +const MINIMUM_BODY_BYTES = 170; + +const MAGIC = new TextEncoder().encode("ADCEXP01"); +const MAXIMUM_SIGNED_64 = 9_223_372_036_854_775_807n; +const UUID_V4 = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; +const LOWER_HEX_256 = /^[0-9a-f]{64}$/; +const KEY_ID = /^[a-z0-9][a-z0-9-]{2,63}$/; +const DEPLOYMENT_PATH = /^\/(?:[A-Za-z0-9._~-]+\/)*[A-Za-z0-9._~-]+$/; +const CANONICAL_DECIMAL = /^(?:0|[1-9][0-9]*)$/; +const BASE64_SHA256 = /^[A-Za-z0-9+/]{43}=$/; + +const ADC_HEADERS = new Set([ + "x-adc-bundle-format", + "x-adc-bundle-id", + "x-adc-configuration-sha256", + "x-adc-researcher-key-id", + "x-adc-sequence-from", + "x-adc-sequence-to", + "x-adc-event-count", +]); +const CONTENT_HEADERS = new Set(["content-type", "content-length", "content-digest"]); +// These are transport headers supplied by OkHttp or documented Cloudflare edge transforms. They +// carry no receiver semantics. Everything else under client control is rejected rather than +// accidentally becoming a second authentication, routing, or representation mechanism. +const INFRASTRUCTURE_HEADERS = new Set([ + "accept-encoding", + "cdn-loop", + "cf-connecting-ip", + "cf-connecting-ipv6", + "cf-connecting-o2o", + "cf-ew-via", + "cf-ipcountry", + "cf-pseudo-ipv4", + "cf-ray", + "cf-visitor", + "cf-worker", + "connection", + "host", + // Wrangler/Miniflare adds this local transport header before invoking the Worker. + "mf-original-hostname", + "true-client-ip", + "user-agent", + "x-forwarded-for", + "x-forwarded-proto", + "x-real-ip", +]); + +export const CUSTOM_METADATA_KEYS = [ + "sha256", + "byte_count", + "configuration_sha256", + "researcher_key_id", + "first_sequence_number", + "last_sequence_number", + "event_count", + "received_at_utc", +] as const; + +export interface ReceiverEnv { + BUNDLES: R2Bucket; + UPLOAD_PATH: string; + ALLOWED_CONFIGURATION_SHA256: string; + ALLOWED_RESEARCHER_KEY_ID: string; +} + +export interface Deployment { + uploadPath: string; + configurationSha256: string; + researcherKeyId: string; +} + +export interface UploadClaims { + bundleId: string; + byteCount: number; + byteCountText: string; + configurationSha256: string; + researcherKeyId: string; + firstSequenceNumber: string; + lastSequenceNumber: string; + eventCount: string; + sha256: string; + sha256Bytes: Uint8Array; +} + +export class RequestViolation extends Error { + constructor(message: string) { + super(message); + this.name = "RequestViolation"; + } +} + +export class PayloadTooLargeViolation extends RequestViolation { + constructor() { + super("Content-Length exceeds the Protocol v1 limit"); + this.name = "PayloadTooLargeViolation"; + } +} + +export class DeploymentViolation extends Error { + constructor(message: string) { + super(message); + this.name = "DeploymentViolation"; + } +} + +export function parseDeployment(env: ReceiverEnv): Deployment { + if (!DEPLOYMENT_PATH.test(env.UPLOAD_PATH)) { + throw new DeploymentViolation("UPLOAD_PATH is invalid"); + } + if (!LOWER_HEX_256.test(env.ALLOWED_CONFIGURATION_SHA256)) { + throw new DeploymentViolation("ALLOWED_CONFIGURATION_SHA256 is invalid"); + } + if (!KEY_ID.test(env.ALLOWED_RESEARCHER_KEY_ID)) { + throw new DeploymentViolation("ALLOWED_RESEARCHER_KEY_ID is invalid"); + } + return { + uploadPath: env.UPLOAD_PATH, + configurationSha256: env.ALLOWED_CONFIGURATION_SHA256, + researcherKeyId: env.ALLOWED_RESEARCHER_KEY_ID, + }; +} + +export function parseUploadRequest(request: Request, deployment: Deployment): UploadClaims { + rejectUnknownProtocolHeaders(request.headers); + if (request.headers.get("content-type") !== BUNDLE_MEDIA_TYPE) { + throw new RequestViolation("Content-Type is invalid"); + } + + const byteCountText = requiredHeader(request.headers, "content-length"); + const byteCount = parseBoundedNumber( + "Content-Length", + byteCountText, + MINIMUM_BODY_BYTES, + MAXIMUM_BODY_BYTES, + ); + const digest = parseContentDigest(requiredHeader(request.headers, "content-digest")); + const bundleFormat = requiredHeader(request.headers, "x-adc-bundle-format"); + const bundleId = requiredHeader(request.headers, "x-adc-bundle-id"); + const configurationSha256 = requiredHeader(request.headers, "x-adc-configuration-sha256"); + const researcherKeyId = requiredHeader(request.headers, "x-adc-researcher-key-id"); + const firstSequenceNumber = requiredHeader(request.headers, "x-adc-sequence-from"); + const lastSequenceNumber = requiredHeader(request.headers, "x-adc-sequence-to"); + const eventCount = requiredHeader(request.headers, "x-adc-event-count"); + + if (bundleFormat !== BUNDLE_FORMAT) throw new RequestViolation("Bundle format is invalid"); + if (!UUID_V4.test(bundleId)) throw new RequestViolation("Bundle ID is invalid"); + if (!LOWER_HEX_256.test(configurationSha256)) { + throw new RequestViolation("Configuration digest is invalid"); + } + if (!KEY_ID.test(researcherKeyId)) throw new RequestViolation("Researcher key ID is invalid"); + + const first = parsePositiveInt64("X-ADC-Sequence-From", firstSequenceNumber); + const last = parsePositiveInt64("X-ADC-Sequence-To", lastSequenceNumber); + const count = parsePositiveInt64("X-ADC-Event-Count", eventCount); + if (last < first || last - first + 1n !== count) { + throw new RequestViolation("Sequence range and event count do not agree"); + } + if (configurationSha256 !== deployment.configurationSha256) { + throw new RequestViolation("Configuration digest is not allowed"); + } + if (researcherKeyId !== deployment.researcherKeyId) { + throw new RequestViolation("Researcher key ID is not allowed"); + } + + return { + bundleId, + byteCount, + byteCountText, + configurationSha256, + researcherKeyId, + firstSequenceNumber, + lastSequenceNumber, + eventCount, + sha256: digest.hex, + sha256Bytes: digest.bytes, + }; +} + +function rejectUnknownProtocolHeaders(headers: Headers): void { + for (const [rawName] of headers) { + const name = rawName.toLowerCase(); + if (ADC_HEADERS.has(name) || CONTENT_HEADERS.has(name)) continue; + if (INFRASTRUCTURE_HEADERS.has(name)) continue; + throw new RequestViolation("Unknown request header"); + } +} + +function requiredHeader(headers: Headers, name: string): string { + const value = headers.get(name); + if (value === null || value.length === 0) throw new RequestViolation(`Missing ${name}`); + return value; +} + +function parseBoundedNumber(name: string, raw: string, minimum: number, maximum: number): number { + if (!CANONICAL_DECIMAL.test(raw)) throw new RequestViolation(`${name} is not canonical`); + if (raw.length > String(maximum).length) throw new PayloadTooLargeViolation(); + const value = BigInt(raw); + if (value > BigInt(maximum)) throw new PayloadTooLargeViolation(); + if (value < BigInt(minimum)) throw new RequestViolation(`${name} is out of range`); + return Number(value); +} + +function parsePositiveInt64(name: string, raw: string): bigint { + if (!CANONICAL_DECIMAL.test(raw)) throw new RequestViolation(`${name} is not canonical`); + if (raw.length > 19) throw new RequestViolation(`${name} is out of range`); + const value = BigInt(raw); + if (value < 1n || value > MAXIMUM_SIGNED_64) { + throw new RequestViolation(`${name} is out of range`); + } + return value; +} + +function parseContentDigest(raw: string): { bytes: Uint8Array; hex: string } { + const prefix = "sha-256=:"; + if (!raw.startsWith(prefix) || !raw.endsWith(":")) { + throw new RequestViolation("Content-Digest is invalid"); + } + const encoded = raw.slice(prefix.length, -1); + if (!BASE64_SHA256.test(encoded)) throw new RequestViolation("Content-Digest is invalid"); + let decoded: Uint8Array; + try { + decoded = Uint8Array.from(atob(encoded), (character) => character.charCodeAt(0)); + } catch { + throw new RequestViolation("Content-Digest is invalid"); + } + if (decoded.length !== 32 || encodeBase64(decoded) !== encoded) { + throw new RequestViolation("Content-Digest is not canonical"); + } + return { bytes: decoded, hex: bytesToHex(decoded) }; +} + +function encodeBase64(bytes: Uint8Array): string { + let binary = ""; + for (const byte of bytes) binary += String.fromCharCode(byte); + return btoa(binary); +} + +export function requiredOuterPrefixLength(prefix: Uint8Array): number { + if (prefix.length < 58) throw new RequestViolation("Bundle outer header is truncated"); + const keyIdLength = (prefix[56]! << 8) | prefix[57]!; + if (keyIdLength < 3 || keyIdLength > 64) { + throw new RequestViolation("Bundle researcher key ID length is invalid"); + } + return 70 + keyIdLength; +} + +export function verifyOuterPrefix(prefix: Uint8Array, claims: UploadClaims): void { + const requiredLength = requiredOuterPrefixLength(prefix); + if (prefix.length < requiredLength) throw new RequestViolation("Bundle outer header is truncated"); + if (!equalBytes(prefix.subarray(0, 8), MAGIC)) throw new RequestViolation("Bundle magic is invalid"); + + const outerBundleId = uuidFromBytes(prefix.subarray(8, 24)); + const outerConfigurationSha256 = bytesToHex(prefix.subarray(24, 56)); + const keyIdBytes = prefix.subarray(70, requiredLength); + let outerResearcherKeyId: string; + try { + outerResearcherKeyId = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(keyIdBytes); + } catch { + throw new RequestViolation("Bundle researcher key ID is not UTF-8"); + } + if (!KEY_ID.test(outerResearcherKeyId)) { + throw new RequestViolation("Bundle researcher key ID is invalid"); + } + if (outerBundleId !== claims.bundleId) throw new RequestViolation("Outer bundle ID mismatch"); + if (outerConfigurationSha256 !== claims.configurationSha256) { + throw new RequestViolation("Outer configuration digest mismatch"); + } + if (outerResearcherKeyId !== claims.researcherKeyId) { + throw new RequestViolation("Outer researcher key ID mismatch"); + } + if (claims.byteCount <= 150 + keyIdBytes.length + 16) { + throw new RequestViolation("Bundle ciphertext is truncated"); + } +} + +function uuidFromBytes(bytes: Uint8Array): string { + const hex = bytesToHex(bytes); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} + +export function bytesToHex(bytes: Uint8Array): string { + let output = ""; + for (const byte of bytes) output += byte.toString(16).padStart(2, "0"); + return output; +} + +function equalBytes(left: Uint8Array, right: Uint8Array): boolean { + if (left.length !== right.length) return false; + let difference = 0; + for (let index = 0; index < left.length; index += 1) { + difference |= left[index]! ^ right[index]!; + } + return difference === 0; +} + +export function objectMetadata(claims: UploadClaims, receivedAtUtc: string): Record { + return { + sha256: claims.sha256, + byte_count: claims.byteCountText, + configuration_sha256: claims.configurationSha256, + researcher_key_id: claims.researcherKeyId, + first_sequence_number: claims.firstSequenceNumber, + last_sequence_number: claims.lastSequenceNumber, + event_count: claims.eventCount, + received_at_utc: receivedAtUtc, + }; +} + +export function isExactObject(object: R2Object, claims: UploadClaims): boolean { + const metadata = object.customMetadata; + const httpMetadata = object.httpMetadata; + if (metadata === undefined || httpMetadata === undefined) return false; + if (Object.keys(metadata).sort().join("\n") !== [...CUSTOM_METADATA_KEYS].sort().join("\n")) { + return false; + } + const storedChecksum = object.checksums.sha256; + return object.size === claims.byteCount + && httpMetadata.contentType === BUNDLE_MEDIA_TYPE + && storedChecksum !== undefined + && bytesToHex(new Uint8Array(storedChecksum)) === claims.sha256 + && metadata.sha256 === claims.sha256 + && metadata.byte_count === claims.byteCountText + && metadata.configuration_sha256 === claims.configurationSha256 + && metadata.researcher_key_id === claims.researcherKeyId + && metadata.first_sequence_number === claims.firstSequenceNumber + && metadata.last_sequence_number === claims.lastSequenceNumber + && metadata.event_count === claims.eventCount + && isCanonicalReceiveTime(metadata.received_at_utc); +} + +function isCanonicalReceiveTime(value: string | undefined): boolean { + if (value === undefined || !/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/.test(value)) return false; + const parsed = new Date(value); + return !Number.isNaN(parsed.valueOf()) && parsed.toISOString() === value; +} + +export function encodeReceipt(claims: UploadClaims): string { + return `{"bundle_id":"${claims.bundleId}","byte_count":"${claims.byteCountText}",` + + `"configuration_sha256":"${claims.configurationSha256}","event_count":"${claims.eventCount}",` + + `"first_sequence_number":"${claims.firstSequenceNumber}",` + + `"last_sequence_number":"${claims.lastSequenceNumber}","sha256":"${claims.sha256}"}`; +} diff --git a/receiver/src/index.ts b/receiver/src/index.ts new file mode 100644 index 0000000..ff54cd7 --- /dev/null +++ b/receiver/src/index.ts @@ -0,0 +1,166 @@ +import { + BUNDLE_MEDIA_TYPE, + DeploymentViolation, + PayloadTooLargeViolation, + RequestViolation, + encodeReceipt, + isExactObject, + objectMetadata, + parseDeployment, + parseUploadRequest, + type ReceiverEnv, + type UploadClaims, +} from "./contract"; +import { + VerifiedBody, + createWorkerDigest, + type DigestFactory, +} from "./verified-body"; + +export interface ReceiverDependencies { + createDigest: DigestFactory; + now: () => Date; +} + +const PRODUCTION_DEPENDENCIES: ReceiverDependencies = { + createDigest: createWorkerDigest, + now: () => new Date(), +}; + +export async function handleRequest( + request: Request, + env: ReceiverEnv, + dependencies: ReceiverDependencies = PRODUCTION_DEPENDENCIES, +): Promise { + let deployment; + try { + deployment = parseDeployment(env); + } catch (error) { + if (error instanceof DeploymentViolation) return errorResponse(500, "receiver_misconfigured"); + return errorResponse(500, "receiver_failure"); + } + + const url = new URL(request.url); + if (url.pathname !== deployment.uploadPath || url.search !== "") { + return errorResponse(404, "not_found"); + } + if (request.method !== "POST") { + return errorResponse(405, "method_not_allowed", { Allow: "POST" }); + } + if (request.body === null) return errorResponse(400, "invalid_request"); + + let claims: UploadClaims; + try { + claims = parseUploadRequest(request, deployment); + } catch (error) { + if (error instanceof PayloadTooLargeViolation) return errorResponse(413, "payload_too_large"); + if (error instanceof RequestViolation) return errorResponse(400, "invalid_request"); + return errorResponse(500, "receiver_failure"); + } + + let existing: R2Object | null; + try { + existing = await env.BUNDLES.head(claims.bundleId); + } catch { + return errorResponse(503, "storage_unavailable"); + } + + const body = new VerifiedBody(request.body, claims, dependencies.createDigest); + if (existing !== null) { + const bodyError = await validateBody(body); + if (bodyError !== undefined) return bodyError; + return isExactObject(existing, claims) + ? receiptResponse(200, claims) + : errorResponse(409, "bundle_conflict"); + } + + // R2 accepts request/response bodies or the readable half of a FixedLengthStream. Keep the + // verifier as the single source reader while giving R2 an explicitly sized streaming body. + const fixedLength = new FixedLengthStream(claims.byteCount); + const forwarding = body.stream.pipeTo(fixedLength.writable); + void forwarding.catch(() => undefined); + + let stored: R2Object | null; + try { + stored = await env.BUNDLES.put(claims.bundleId, fixedLength.readable, { + onlyIf: new Headers({ "If-None-Match": "*" }), + httpMetadata: { contentType: BUNDLE_MEDIA_TYPE }, + customMetadata: objectMetadata(claims, dependencies.now().toISOString()), + sha256: exactArrayBuffer(claims.sha256Bytes), + }); + } catch { + await releaseFixedLengthStream(fixedLength.readable, forwarding); + const bodyError = await validateBody(body); + return bodyError ?? errorResponse(503, "storage_unavailable"); + } + + // A conditional create may return without pulling the body. Cancelling the fixed-length + // transport releases pipeTo(); VerifiedBody.complete() then drains and verifies the same + // request source before replay success or conflict is decided. + await releaseFixedLengthStream(fixedLength.readable, forwarding); + const bodyError = await validateBody(body); + if (bodyError !== undefined) return bodyError; + if (stored !== null) { + return isExactObject(stored, claims) + ? receiptResponse(201, claims) + : errorResponse(503, "storage_unavailable"); + } + + // A competing create won after our initial HEAD. R2 writes and metadata reads are strongly + // consistent, so the winning object is now the single authority for replay vs conflict. + try { + const winner = await env.BUNDLES.head(claims.bundleId); + if (winner === null) return errorResponse(503, "storage_unavailable"); + return isExactObject(winner, claims) + ? receiptResponse(200, claims) + : errorResponse(409, "bundle_conflict"); + } catch { + return errorResponse(503, "storage_unavailable"); + } +} + +async function releaseFixedLengthStream( + readable: ReadableStream, + forwarding: Promise, +): Promise { + await readable.cancel().catch(() => undefined); + await forwarding.catch(() => undefined); +} + +async function validateBody(body: VerifiedBody): Promise { + try { + await body.complete(); + return undefined; + } catch (error) { + return error instanceof RequestViolation + ? errorResponse(400, "invalid_request") + : errorResponse(503, "storage_unavailable"); + } +} + +function exactArrayBuffer(bytes: Uint8Array): ArrayBuffer { + return bytes.slice().buffer; +} + +function receiptResponse(status: 200 | 201, claims: UploadClaims): Response { + return new Response(encodeReceipt(claims), { + status, + headers: { + "Cache-Control": "no-store", + "Content-Type": "application/json", + }, + }); +} + +function errorResponse(status: number, code: string, headers?: HeadersInit): Response { + const responseHeaders = new Headers(headers); + responseHeaders.set("Cache-Control", "no-store"); + responseHeaders.set("Content-Type", "text/plain; charset=utf-8"); + return new Response(`${code}\n`, { status, headers: responseHeaders }); +} + +export default { + fetch(request: Request, env: ReceiverEnv): Promise { + return handleRequest(request, env); + }, +} satisfies ExportedHandler; diff --git a/receiver/src/verified-body.ts b/receiver/src/verified-body.ts new file mode 100644 index 0000000..4f3f0ad --- /dev/null +++ b/receiver/src/verified-body.ts @@ -0,0 +1,160 @@ +import { + RequestViolation, + bytesToHex, + requiredOuterPrefixLength, + verifyOuterPrefix, + type UploadClaims, +} from "./contract"; + +export interface DigestSink { + writable: WritableStream; + digest: Promise; +} + +export type DigestFactory = () => DigestSink; + +export function createWorkerDigest(): DigestSink { + const stream = new crypto.DigestStream("SHA-256"); + return { writable: stream, digest: stream.digest }; +} + +/** + * One-pass, bounded-memory verifier. R2 pulls from [stream]; if a conditional write declines to + * consume all bytes, [complete] drains the same source without replaying or buffering it. + */ +export class VerifiedBody { + readonly stream: ReadableStream; + + private readonly source: ReadableStreamDefaultReader; + private readonly digestWriter: WritableStreamDefaultWriter; + private readonly digest: Promise; + private readonly prefix = new Uint8Array(134); + private prefixLength = 0; + private requiredPrefixLength = 58; + private byteCount = 0; + private finished = false; + private outerVerified = false; + private activeRead: Promise | undefined; + private failure: unknown; + private outputCancelled = false; + + constructor( + body: ReadableStream, + private readonly claims: UploadClaims, + createDigest: DigestFactory, + ) { + this.source = body.getReader(); + const digest = createDigest(); + this.digestWriter = digest.writable.getWriter(); + this.digest = digest.digest; + // A protocol error can abort the digest before complete() needs its value. Observe that + // rejection immediately; complete() still reports the original protocol failure. + void this.digest.catch(() => undefined); + this.stream = new ReadableStream( + { + pull: (controller) => this.pull(controller), + // R2 may decline a conditional write without reading. Keep the request reader so + // complete() can validate the exact replay body rather than trust its digest header. + cancel: () => { + this.outputCancelled = true; + }, + }, + // Do not prefetch a request chunk before R2 asks for it. This keeps backpressure attached + // to the durable sink and leaves complete() free to drain a declined conditional write. + { highWaterMark: 0 }, + ); + } + + async complete(): Promise { + if (this.activeRead !== undefined) { + await this.activeRead.catch(() => undefined); + } + while (!this.finished && this.failure === undefined) { + await this.readOne(); + } + if (this.failure !== undefined) throw this.failure; + const actual = bytesToHex(new Uint8Array(await this.digest)); + if (actual !== this.claims.sha256) throw new RequestViolation("Body SHA-256 mismatch"); + } + + private pull(controller: ReadableStreamDefaultController): Promise { + const operation = this.readOne(controller); + this.activeRead = operation; + return operation.finally(() => { + if (this.activeRead === operation) this.activeRead = undefined; + }); + } + + private async readOne(controller?: ReadableStreamDefaultController): Promise { + if (this.finished) { + if (!this.outputCancelled) controller?.close(); + return; + } + if (this.failure !== undefined) { + if (!this.outputCancelled) controller?.error(this.failure); + throw this.failure; + } + try { + const next = await this.source.read(); + if (next.done) { + this.finish(); + await this.digestWriter.close(); + if (!this.outputCancelled) controller?.close(); + return; + } + await this.accept(next.value); + if (!this.outputCancelled) controller?.enqueue(next.value); + } catch (error) { + await this.fail(error); + if (!this.outputCancelled) controller?.error(error); + throw error; + } + } + + private async accept(chunk: Uint8Array): Promise { + this.byteCount += chunk.byteLength; + if (this.byteCount > this.claims.byteCount) { + throw new RequestViolation("Body exceeds Content-Length"); + } + this.captureOuterPrefix(chunk); + await this.digestWriter.write(chunk); + } + + private captureOuterPrefix(chunk: Uint8Array): void { + let chunkOffset = 0; + while (this.prefixLength < this.requiredPrefixLength && chunkOffset < chunk.length) { + const copied = Math.min( + this.requiredPrefixLength - this.prefixLength, + chunk.length - chunkOffset, + ); + this.prefix.set(chunk.subarray(chunkOffset, chunkOffset + copied), this.prefixLength); + this.prefixLength += copied; + chunkOffset += copied; + + if (this.prefixLength === 58 && this.requiredPrefixLength === 58) { + this.requiredPrefixLength = requiredOuterPrefixLength(this.prefix.subarray(0, 58)); + } + if (this.prefixLength === this.requiredPrefixLength && !this.outerVerified) { + verifyOuterPrefix(this.prefix.subarray(0, this.requiredPrefixLength), this.claims); + this.outerVerified = true; + } + } + } + + private finish(): void { + if (this.byteCount !== this.claims.byteCount) { + throw new RequestViolation("Body length does not match Content-Length"); + } + if (!this.outerVerified) throw new RequestViolation("Bundle outer header is truncated"); + this.finished = true; + } + + private async fail(error: unknown): Promise { + if (this.failure !== undefined) return; + this.failure = error; + await Promise.allSettled([ + this.source.cancel(error), + this.digestWriter.abort(error), + ]); + } +} diff --git a/receiver/tests/receiver.test.ts b/receiver/tests/receiver.test.ts new file mode 100644 index 0000000..cf133ff --- /dev/null +++ b/receiver/tests/receiver.test.ts @@ -0,0 +1,577 @@ +import { describe, expect, it } from "vitest"; +import { + BUNDLE_MEDIA_TYPE, + CUSTOM_METADATA_KEYS, + MAXIMUM_BODY_BYTES, + bytesToHex, + type ReceiverEnv, +} from "../src/contract"; +import { handleRequest, type ReceiverDependencies } from "../src/index"; +import type { DigestSink } from "../src/verified-body"; + +// Node does not brand fixed-length streams. The real workerd suite exercises Cloudflare's native +// implementation; the fast fake-R2 suite only needs an identity transform around the verifier. +class TestFixedLengthStream extends TransformStream { + constructor(_expectedLength: number) { + super(); + } +} +Object.defineProperty(globalThis, "FixedLengthStream", { value: TestFixedLengthStream }); + +const BUNDLE_ID = "550e8400-e29b-41d4-a716-446655440000"; +const OTHER_BUNDLE_ID = "f47ac10b-58cc-4372-a567-0e02b2c3d479"; +const CONFIGURATION_SHA256 = "ab".repeat(32); +const OTHER_CONFIGURATION_SHA256 = "cd".repeat(32); +const RESEARCHER_KEY_ID = "researcher-key"; +const UPLOAD_URL = "https://receiver.example.test/v1/upload"; +const RECEIVED_AT = "2026-08-04T01:02:03.004Z"; + +const DEPENDENCIES: ReceiverDependencies = { + createDigest: createTestDigest, + now: () => new Date(RECEIVED_AT), +}; + +describe("ADC Protocol v1 ciphertext receiver", () => { + it("streams a new bundle into a create-only R2 object and returns the canonical receipt", async () => { + const bucket = new FakeR2Bucket(); + const fixture = await requestFixture(); + + const response = await handleRequest(fixture.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(201); + expect(response.headers.get("content-type")).toBe("application/json"); + expect(await response.text()).toBe(canonicalReceipt(fixture)); + expect(bucket.putCalls).toBe(1); + expect(bucket.lastOnlyIf?.get("if-none-match")).toBe("*"); + expect(bucket.lastPutBody).toEqual(fixture.body); + expect(bucket.lastExpectedSha256).toBe(fixture.sha256); + + const stored = bucket.record(BUNDLE_ID); + expect(stored).toBeDefined(); + expect(stored?.object.key).toBe(BUNDLE_ID); + expect(stored?.object.size).toBe(fixture.body.length); + expect(stored?.object.httpMetadata?.contentType).toBe(BUNDLE_MEDIA_TYPE); + expect(Object.keys(stored?.object.customMetadata ?? {})).toEqual([...CUSTOM_METADATA_KEYS]); + expect(stored?.object.customMetadata).toEqual({ + sha256: fixture.sha256, + byte_count: String(fixture.body.length), + configuration_sha256: CONFIGURATION_SHA256, + researcher_key_id: RESEARCHER_KEY_ID, + first_sequence_number: "1", + last_sequence_number: "1", + event_count: "1", + received_at_utc: RECEIVED_AT, + }); + }); + + it("returns the identical receipt for a fully verified exact replay without overwriting", async () => { + const bucket = new FakeR2Bucket(); + const first = await requestFixture(); + const replay = await requestFixture(); + + const created = await handleRequest(first.request, environment(bucket), DEPENDENCIES); + const originalRecord = bucket.record(BUNDLE_ID); + const repeated = await handleRequest(replay.request, environment(bucket), { + ...DEPENDENCIES, + now: () => new Date("2030-01-01T00:00:00.000Z"), + }); + + expect(created.status).toBe(201); + expect(repeated.status).toBe(200); + expect(await repeated.text()).toBe(await created.text()); + expect(bucket.putCalls).toBe(1); + expect(bucket.record(BUNDLE_ID)).toBe(originalRecord); + expect(bucket.record(BUNDLE_ID)?.object.customMetadata?.received_at_utc).toBe(RECEIVED_AT); + }); + + it("resolves concurrent identical creates as one 201 and one 200", async () => { + const bucket = new FakeR2Bucket(); + const left = await requestFixture(); + const right = await requestFixture(); + + const responses = await Promise.all([ + handleRequest(left.request, environment(bucket), DEPENDENCIES), + handleRequest(right.request, environment(bucket), DEPENDENCIES), + ]); + + expect(responses.map((response) => response.status).sort()).toEqual([200, 201]); + expect(await responses[0]!.text()).toBe(await responses[1]!.text()); + expect(bucket.objects.size).toBe(1); + expect(bucket.putCalls).toBe(2); + }); + + it("validates the body when a raced conditional write declines it without reading", async () => { + const bucket = new FakeR2Bucket(); + const original = await requestFixture(); + expect((await handleRequest(original.request, environment(bucket), DEPENDENCIES)).status).toBe(201); + + bucket.hideExistingFromNextHead = true; + bucket.declineExistingBeforeRead = true; + const response = await handleRequest( + (await requestFixture()).request, + environment(bucket), + DEPENDENCIES, + ); + + expect(response.status).toBe(200); + expect(bucket.declinedBeforeRead).toBe(true); + expect(await response.text()).toBe(canonicalReceipt(original)); + }); + + it("accepts documented OkHttp and Cloudflare transport headers without treating them as claims", async () => { + const bucket = new FakeR2Bucket(); + const fixture = await requestFixture({ + headers: { + "Accept-Encoding": "br, gzip", + "CF-Connecting-IP": "192.0.2.1", + "CF-EW-Via": "15", + "CF-Ray": "230b030023ae2822-SJC", + "CF-Visitor": "{\"scheme\":\"https\"}", + "CDN-Loop": "cloudflare", + Connection: "Keep-Alive", + "MF-Original-Hostname": "localhost", + "User-Agent": "okhttp/5", + "X-Forwarded-For": "192.0.2.1", + "X-Forwarded-Proto": "https", + }, + }); + + expect((await handleRequest(fixture.request, environment(bucket), DEPENDENCIES)).status).toBe(201); + }); + + it("never overwrites a reused bundle ID with different ciphertext", async () => { + const bucket = new FakeR2Bucket(); + const original = await requestFixture(); + const changedBody = original.body.slice(); + changedBody[changedBody.length - 1] = changedBody[changedBody.length - 1]! ^ 0xff; + + expect((await handleRequest(original.request, environment(bucket), DEPENDENCIES)).status).toBe(201); + const storedBefore = bucket.record(BUNDLE_ID)?.body.slice(); + const conflict = await requestFixture({ body: changedBody }); + const response = await handleRequest(conflict.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(409); + expect(await response.text()).toBe("bundle_conflict\n"); + expect(bucket.record(BUNDLE_ID)?.body).toEqual(storedBefore); + }); + + it("verifies replay bytes instead of trusting a repeated digest header", async () => { + const bucket = new FakeR2Bucket(); + const original = await requestFixture(); + expect((await handleRequest(original.request, environment(bucket), DEPENDENCIES)).status).toBe(201); + + const changedBody = original.body.slice(); + changedBody[changedBody.length - 1] = changedBody[changedBody.length - 1]! ^ 0xff; + const forgedReplay = await requestFixture({ body: changedBody, digestBody: original.body }); + const response = await handleRequest(forgedReplay.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(400); + expect(bucket.putCalls).toBe(1); + }); + + it("treats a replay with different claimed range metadata as a conflict", async () => { + const bucket = new FakeR2Bucket(); + const original = await requestFixture(); + expect((await handleRequest(original.request, environment(bucket), DEPENDENCIES)).status).toBe(201); + const replay = await requestFixture({ + headers: { + "X-ADC-Sequence-From": "2", + "X-ADC-Sequence-To": "2", + }, + }); + + expect((await handleRequest(replay.request, environment(bucket), DEPENDENCIES)).status).toBe(409); + }); + + it("does not resolve success until the durable R2 write promise resolves", async () => { + const bucket = new FakeR2Bucket(); + bucket.pauseBeforeCommit(); + const fixture = await requestFixture(); + let settled = false; + + const pending = handleRequest(fixture.request, environment(bucket), DEPENDENCIES) + .finally(() => { settled = true; }); + await bucket.putReachedCommit; + await Promise.resolve(); + expect(settled).toBe(false); + + bucket.releaseCommit(); + expect((await pending).status).toBe(201); + }); + + it("returns 503 on R2 failures and never fabricates a receipt", async () => { + const headFailure = new FakeR2Bucket(); + headFailure.failHead = true; + expect((await handleRequest( + (await requestFixture()).request, + environment(headFailure), + DEPENDENCIES, + )).status).toBe(503); + + const putFailure = new FakeR2Bucket(); + putFailure.failPut = true; + const response = await handleRequest( + (await requestFixture()).request, + environment(putFailure), + DEPENDENCIES, + ); + expect(response.status).toBe(503); + expect(putFailure.objects.size).toBe(0); + }); + + it("exposes only the deployment-fixed POST path", async () => { + const bucket = new FakeR2Bucket(); + const wrongPath = await requestFixture({ url: "https://receiver.example.test/other" }); + const query = await requestFixture({ url: `${UPLOAD_URL}?download=1` }); + const get = new Request(UPLOAD_URL, { method: "GET" }); + + expect((await handleRequest(wrongPath.request, environment(bucket), DEPENDENCIES)).status).toBe(404); + expect((await handleRequest(query.request, environment(bucket), DEPENDENCIES)).status).toBe(404); + const methodResponse = await handleRequest(get, environment(bucket), DEPENDENCIES); + expect(methodResponse.status).toBe(405); + expect(methodResponse.headers.get("allow")).toBe("POST"); + expect(bucket.headCalls).toBe(0); + }); + + it("fails closed when deployment inputs are placeholders or malformed", async () => { + const bucket = new FakeR2Bucket(); + const fixture = await requestFixture(); + const env = environment(bucket); + env.ALLOWED_CONFIGURATION_SHA256 = "REPLACE_WITH_64_LOWERCASE_HEX"; + + const response = await handleRequest(fixture.request, env, DEPENDENCIES); + + expect(response.status).toBe(500); + expect(await response.text()).toBe("receiver_misconfigured\n"); + expect(bucket.headCalls).toBe(0); + }); + + it("rejects missing and duplicate protocol headers", async () => { + const missingBucket = new FakeR2Bucket(); + const missing = await requestFixture(); + missing.request.headers.delete("content-digest"); + expect((await handleRequest(missing.request, environment(missingBucket), DEPENDENCIES)).status).toBe(400); + + const duplicateBucket = new FakeR2Bucket(); + const duplicate = await requestFixture(); + duplicate.request.headers.append("x-adc-event-count", "1"); + expect((await handleRequest( + duplicate.request, + environment(duplicateBucket), + DEPENDENCIES, + )).status).toBe(400); + expect(missingBucket.headCalls + duplicateBucket.headCalls).toBe(0); + }); + + it("returns 413 before storage when the declared body exceeds 32 MiB", async () => { + const bucket = new FakeR2Bucket(); + const fixture = await requestFixture({ + headers: { "Content-Length": String(MAXIMUM_BODY_BYTES + 1) }, + }); + + const response = await handleRequest(fixture.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(413); + expect(await response.text()).toBe("payload_too_large\n"); + expect(bucket.headCalls).toBe(0); + }); + + it.each([ + ["media type parameter", { headers: { "Content-Type": `${BUNDLE_MEDIA_TYPE}; charset=binary` } }], + ["transfer encoding", { headers: { "Transfer-Encoding": "chunked" } }], + ["unknown ADC header", { headers: { "X-ADC-Extra": "value" } }], + ["unknown content header", { headers: { "Content-Language": "en" } }], + ["authorization header", { headers: { Authorization: "Bearer ignored-is-not-allowed" } }], + ["noncanonical byte count", { headers: { "Content-Length": "0256" } }], + ["unknown bundle format", { headers: { "X-ADC-Bundle-Format": "research-bundle-v2" } }], + ["non-v4 bundle UUID", { headers: { "X-ADC-Bundle-Id": "550e8400-e29b-11d4-a716-446655440000" } }], + ["uppercase digest", { headers: { "X-ADC-Configuration-SHA256": CONFIGURATION_SHA256.toUpperCase() } }], + ["invalid researcher key", { headers: { "X-ADC-Researcher-Key-Id": "Researcher_Key" } }], + ["leading-zero sequence", { headers: { "X-ADC-Sequence-From": "01" } }], + ["empty event range", { headers: { "X-ADC-Event-Count": "0" } }], + ["range/count mismatch", { headers: { "X-ADC-Sequence-To": "2" } }], + ["configuration not allowed", { headers: { "X-ADC-Configuration-SHA256": OTHER_CONFIGURATION_SHA256 } }], + ["key not allowed", { headers: { "X-ADC-Researcher-Key-Id": "different-key" } }], + ["malformed content digest", { headers: { "Content-Digest": `SHA-256=:${"A".repeat(43)}=:` } }], + ])("rejects %s before writing", async (_name, options) => { + const bucket = new FakeR2Bucket(); + const fixture = await requestFixture(options); + + const response = await handleRequest(fixture.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(400); + expect(bucket.putCalls).toBe(0); + }); + + it.each([ + ["wrong magic", (body: Uint8Array) => { body[0] = body[0]! ^ 0xff; }], + ["outer bundle ID mismatch", (body: Uint8Array) => writeUuid(body, 8, OTHER_BUNDLE_ID)], + ["outer configuration mismatch", (body: Uint8Array) => writeHex(body, 24, OTHER_CONFIGURATION_SHA256)], + ["outer key mismatch", (body: Uint8Array) => { + body.set(new TextEncoder().encode("researcher-kex"), 70); + }], + ])("rejects %s without leaving an object", async (_name, mutate) => { + const bucket = new FakeR2Bucket(); + const body = makeBundle(); + mutate(body); + const fixture = await requestFixture({ body }); + + const response = await handleRequest(fixture.request, environment(bucket), DEPENDENCIES); + + expect(response.status).toBe(400); + expect(bucket.objects.size).toBe(0); + }); + + it("rejects truncation, excess bytes, and a digest mismatch", async () => { + const cases = [ + await requestFixture({ body: makeBundle().subarray(0, 100) }), + await requestFixture({ headers: { "Content-Length": String(makeBundle().length - 1) } }), + await requestFixture({ headers: { "Content-Length": String(makeBundle().length + 1) } }), + await requestFixture({ digestBody: new Uint8Array(makeBundle().length) }), + ]; + + for (const fixture of cases) { + const bucket = new FakeR2Bucket(); + const response = await handleRequest(fixture.request, environment(bucket), DEPENDENCIES); + expect(response.status).toBe(400); + expect(bucket.objects.size).toBe(0); + } + }); +}); + +interface FixtureOptions { + body?: Uint8Array; + digestBody?: Uint8Array; + headers?: Record; + method?: string; + url?: string; +} + +interface RequestFixture { + request: Request; + body: Uint8Array; + sha256: string; + byteCountText: string; + firstSequenceNumber: string; + lastSequenceNumber: string; + eventCount: string; +} + +async function requestFixture(options: FixtureOptions = {}): Promise { + const body = options.body?.slice() ?? makeBundle(); + const digest = new Uint8Array(await crypto.subtle.digest("SHA-256", options.digestBody ?? body)); + const sha256 = bytesToHex(digest); + const headers = new Headers({ + "Content-Type": BUNDLE_MEDIA_TYPE, + "Content-Length": String(body.length), + "Content-Digest": `sha-256=:${toBase64(digest)}:`, + "X-ADC-Bundle-Format": "research-bundle-v1", + "X-ADC-Bundle-Id": BUNDLE_ID, + "X-ADC-Configuration-SHA256": CONFIGURATION_SHA256, + "X-ADC-Researcher-Key-Id": RESEARCHER_KEY_ID, + "X-ADC-Sequence-From": "1", + "X-ADC-Sequence-To": "1", + "X-ADC-Event-Count": "1", + ...options.headers, + }); + const method = options.method ?? "POST"; + const request = new Request(options.url ?? UPLOAD_URL, { + method, + headers, + ...(method === "GET" || method === "HEAD" ? {} : { body }), + }); + return { + request, + body, + sha256, + byteCountText: headers.get("content-length")!, + firstSequenceNumber: headers.get("x-adc-sequence-from")!, + lastSequenceNumber: headers.get("x-adc-sequence-to")!, + eventCount: headers.get("x-adc-event-count")!, + }; +} + +function makeBundle(): Uint8Array { + const keyId = new TextEncoder().encode(RESEARCHER_KEY_ID); + const body = new Uint8Array(256); + for (let index = 0; index < body.length; index += 1) body[index] = index & 0xff; + body.set(new TextEncoder().encode("ADCEXP01"), 0); + writeUuid(body, 8, BUNDLE_ID); + writeHex(body, 24, CONFIGURATION_SHA256); + body[56] = 0; + body[57] = keyId.length; + body.set(keyId, 70); + return body; +} + +function writeUuid(target: Uint8Array, offset: number, uuid: string): void { + writeHex(target, offset, uuid.replaceAll("-", "")); +} + +function writeHex(target: Uint8Array, offset: number, hex: string): void { + for (let index = 0; index < hex.length; index += 2) { + target[offset + index / 2] = Number.parseInt(hex.slice(index, index + 2), 16); + } +} + +function toBase64(bytes: Uint8Array): string { + let binary = ""; + for (const byte of bytes) binary += String.fromCharCode(byte); + return btoa(binary); +} + +function canonicalReceipt(fixture: RequestFixture): string { + return `{"bundle_id":"${BUNDLE_ID}","byte_count":"${fixture.byteCountText}",` + + `"configuration_sha256":"${CONFIGURATION_SHA256}","event_count":"${fixture.eventCount}",` + + `"first_sequence_number":"${fixture.firstSequenceNumber}",` + + `"last_sequence_number":"${fixture.lastSequenceNumber}","sha256":"${fixture.sha256}"}`; +} + +function environment(bucket: FakeR2Bucket): ReceiverEnv { + return { + BUNDLES: bucket as unknown as R2Bucket, + UPLOAD_PATH: "/v1/upload", + ALLOWED_CONFIGURATION_SHA256: CONFIGURATION_SHA256, + ALLOWED_RESEARCHER_KEY_ID: RESEARCHER_KEY_ID, + }; +} + +function createTestDigest(): DigestSink { + const chunks: Uint8Array[] = []; + let resolveDigest!: (digest: ArrayBuffer) => void; + let rejectDigest!: (error: unknown) => void; + const digest = new Promise((resolve, reject) => { + resolveDigest = resolve; + rejectDigest = reject; + }); + return { + writable: new WritableStream({ + write(chunk) { + chunks.push(chunk.slice()); + }, + async close() { + resolveDigest(await crypto.subtle.digest("SHA-256", concatenate(chunks))); + }, + abort(error) { + rejectDigest(error); + }, + }), + digest, + }; +} + +function concatenate(chunks: Uint8Array[]): Uint8Array { + const result = new Uint8Array(chunks.reduce((size, chunk) => size + chunk.length, 0)); + let offset = 0; + for (const chunk of chunks) { + result.set(chunk, offset); + offset += chunk.length; + } + return result; +} + +interface StoredRecord { + body: Uint8Array; + object: R2Object; +} + +class FakeR2Bucket { + readonly objects = new Map(); + headCalls = 0; + putCalls = 0; + failHead = false; + failPut = false; + hideExistingFromNextHead = false; + declineExistingBeforeRead = false; + declinedBeforeRead = false; + lastOnlyIf: Headers | undefined; + lastPutBody: Uint8Array | undefined; + lastExpectedSha256: string | undefined; + + private commitGate: Promise | undefined; + private resolveCommit: (() => void) | undefined; + private resolvePutReached!: () => void; + putReachedCommit: Promise = new Promise((resolve) => { this.resolvePutReached = resolve; }); + + record(key: string): StoredRecord | undefined { + return this.objects.get(key); + } + + pauseBeforeCommit(): void { + this.commitGate = new Promise((resolve) => { this.resolveCommit = resolve; }); + } + + releaseCommit(): void { + this.resolveCommit?.(); + } + + async head(key: string): Promise { + this.headCalls += 1; + if (this.failHead) throw new Error("R2 head failed"); + if (this.hideExistingFromNextHead) { + this.hideExistingFromNextHead = false; + return null; + } + return this.objects.get(key)?.object ?? null; + } + + async put(key: string, value: ReadableStream, options: R2PutOptions): Promise { + this.putCalls += 1; + this.lastOnlyIf = options.onlyIf instanceof Headers ? options.onlyIf : undefined; + if (this.failPut) throw new Error("R2 put failed"); + if (this.declineExistingBeforeRead && this.objects.has(key)) { + this.declinedBeforeRead = true; + return null; + } + const body = await readAll(value as ReadableStream); + const digest = await crypto.subtle.digest("SHA-256", body); + const digestHex = bytesToHex(new Uint8Array(digest)); + const expected = options.sha256; + if (!(expected instanceof ArrayBuffer) || bytesToHex(new Uint8Array(expected)) !== digestHex) { + throw new Error("R2 checksum mismatch"); + } + this.lastPutBody = body; + this.lastExpectedSha256 = digestHex; + this.resolvePutReached(); + await this.commitGate; + + if (this.lastOnlyIf?.get("if-none-match") === "*" && this.objects.has(key)) return null; + const object = fakeObject(key, body.length, digest, options); + this.objects.set(key, { body, object }); + return object; + } +} + +async function readAll(stream: ReadableStream): Promise { + const reader = stream.getReader(); + const chunks: Uint8Array[] = []; + while (true) { + const next = await reader.read(); + if (next.done) break; + chunks.push(next.value.slice()); + } + return concatenate(chunks); +} + +function fakeObject(key: string, size: number, digest: ArrayBuffer, options: R2PutOptions): R2Object { + const httpMetadata = options.httpMetadata instanceof Headers + ? {} + : { ...(options.httpMetadata ?? {}) }; + return { + key, + version: "fake-version", + size, + etag: "fake-etag", + httpEtag: "\"fake-etag\"", + uploaded: new Date(RECEIVED_AT), + httpMetadata, + customMetadata: { ...(options.customMetadata ?? {}) }, + range: { offset: 0, length: size }, + checksums: { + sha256: digest, + toJSON: () => ({ sha256: bytesToHex(new Uint8Array(digest)) }), + }, + storageClass: "Standard", + writeHttpMetadata() {}, + }; +} diff --git a/receiver/tests/receiver.workerd.test.ts b/receiver/tests/receiver.workerd.test.ts new file mode 100644 index 0000000..1d4e255 --- /dev/null +++ b/receiver/tests/receiver.workerd.test.ts @@ -0,0 +1,141 @@ +import { env, exports } from "cloudflare:workers"; +import { describe, expect, it } from "vitest"; +import corpus from "../../protocol/v1/conformance-vectors.json"; +import { type ReceiverEnv } from "../src/contract"; +import { handleRequest } from "../src/index"; + +const UPLOAD_URL = "https://receiver.example.test/v1/upload"; +const RESEARCHER_KEY_ID = "vector-hpke"; + +describe("ADC receiver in workerd", () => { + it("stores and exactly replays the shared Protocol v1 bundle through native R2", async () => { + const body = decodeHex(corpus.valid.bundle.container_hex); + const receipt = corpus.valid.upload_receipt.value; + const expectedReceipt = new TextDecoder().decode( + decodeHex(corpus.valid.upload_receipt.canonical_jcs_utf8_hex), + ); + + const created = await exports.default.fetch(UPLOAD_URL, requestInit(body, receipt)); + const replayed = await exports.default.fetch(UPLOAD_URL, requestInit(body, receipt)); + const createdReceipt = await created.text(); + const replayedReceipt = await replayed.text(); + + expect({ status: created.status, body: createdReceipt }).toEqual({ + status: 201, + body: expectedReceipt, + }); + expect({ status: replayed.status, body: replayedReceipt }).toEqual({ + status: 200, + body: expectedReceipt, + }); + + const stored = await env.BUNDLES.get(receipt.bundle_id); + expect(stored).not.toBeNull(); + expect(new Uint8Array(await stored!.arrayBuffer())).toEqual(body); + const metadata = stored!.customMetadata; + expect(metadata).toBeDefined(); + expect(metadata!).toMatchObject({ + sha256: receipt.sha256, + byte_count: receipt.byte_count, + configuration_sha256: receipt.configuration_sha256, + researcher_key_id: RESEARCHER_KEY_ID, + first_sequence_number: receipt.first_sequence_number, + last_sequence_number: receipt.last_sequence_number, + event_count: receipt.event_count, + }); + expect(Object.keys(metadata!).sort()).toEqual([ + "byte_count", + "configuration_sha256", + "event_count", + "first_sequence_number", + "last_sequence_number", + "received_at_utc", + "researcher_key_id", + "sha256", + ]); + expect(metadata!.received_at_utc).toMatch( + /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/, + ); + }); + + it("drains and verifies a replay when the real R2 conditional create loses a race", async () => { + const body = decodeHex(corpus.valid.bundle.container_hex); + const receipt = corpus.valid.upload_receipt.value; + const expectedReceipt = new TextDecoder().decode( + decodeHex(corpus.valid.upload_receipt.canonical_jcs_utf8_hex), + ); + const prepared = await exports.default.fetch(UPLOAD_URL, requestInit(body, receipt)); + expect([200, 201]).toContain(prepared.status); + + let hideFirstHead = true; + const racingBucket = new Proxy(env.BUNDLES, { + get(target, property) { + if (property === "head") { + return (key: string): Promise => { + if (hideFirstHead) { + hideFirstHead = false; + return Promise.resolve(null); + } + return target.head(key); + }; + } + const value: unknown = Reflect.get(target, property); + return typeof value === "function" ? value.bind(target) : value; + }, + }); + const receiverEnv: ReceiverEnv = { + BUNDLES: racingBucket, + UPLOAD_PATH: env.UPLOAD_PATH, + ALLOWED_CONFIGURATION_SHA256: env.ALLOWED_CONFIGURATION_SHA256, + ALLOWED_RESEARCHER_KEY_ID: env.ALLOWED_RESEARCHER_KEY_ID, + }; + + const replayed = await handleRequest( + new Request(UPLOAD_URL, requestInit(body, receipt)), + receiverEnv, + ); + + expect(hideFirstHead).toBe(false); + expect({ status: replayed.status, body: await replayed.text() }).toEqual({ + status: 200, + body: expectedReceipt, + }); + }); +}); + +function requestInit( + body: Uint8Array, + receipt: typeof corpus.valid.upload_receipt.value, +): RequestInit { + return { + method: "POST", + headers: { + "Content-Type": "application/vnd.adc.research-bundle", + "Content-Length": receipt.byte_count, + "Content-Digest": `sha-256=:${base64(decodeHex(receipt.sha256))}:`, + "X-ADC-Bundle-Format": "research-bundle-v1", + "X-ADC-Bundle-Id": receipt.bundle_id, + "X-ADC-Configuration-SHA256": receipt.configuration_sha256, + "X-ADC-Researcher-Key-Id": RESEARCHER_KEY_ID, + "X-ADC-Sequence-From": receipt.first_sequence_number, + "X-ADC-Sequence-To": receipt.last_sequence_number, + "X-ADC-Event-Count": receipt.event_count, + }, + body, + }; +} + +function decodeHex(hex: string): Uint8Array { + if (hex.length % 2 !== 0 || !/^[0-9a-f]*$/.test(hex)) throw new Error("Invalid fixture hex"); + const bytes = new Uint8Array(hex.length / 2); + for (let index = 0; index < bytes.length; index += 1) { + bytes[index] = Number.parseInt(hex.slice(index * 2, index * 2 + 2), 16); + } + return bytes; +} + +function base64(bytes: Uint8Array): string { + let binary = ""; + for (const byte of bytes) binary += String.fromCharCode(byte); + return btoa(binary); +} diff --git a/receiver/tsconfig.json b/receiver/tsconfig.json new file mode 100644 index 0000000..9d53664 --- /dev/null +++ b/receiver/tsconfig.json @@ -0,0 +1,29 @@ +{ + "compilerOptions": { + "target": "ES2024", + "module": "ESNext", + "moduleResolution": "Bundler", + "resolveJsonModule": true, + "lib": ["ESNext"], + "types": ["@cloudflare/workers-types", "node"], + "strict": true, + "skipLibCheck": true, + "exactOptionalPropertyTypes": true, + "noEmit": true, + "noFallthroughCasesInSwitch": true, + "noImplicitOverride": true, + "noImplicitReturns": true, + "noUncheckedIndexedAccess": true, + "noUnusedLocals": true, + "noUnusedParameters": true, + "useUnknownInCatchVariables": true, + "verbatimModuleSyntax": true + }, + "include": [ + "src/**/*.ts", + "tests/**/*.ts", + "vitest.config.ts", + "vitest.workerd.config.ts", + "worker-configuration.d.ts" + ] +} diff --git a/receiver/vitest.config.ts b/receiver/vitest.config.ts new file mode 100644 index 0000000..1ba0714 --- /dev/null +++ b/receiver/vitest.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + environment: "node", + include: ["tests/receiver.test.ts"], + }, +}); diff --git a/receiver/vitest.workerd.config.ts b/receiver/vitest.workerd.config.ts new file mode 100644 index 0000000..9693029 --- /dev/null +++ b/receiver/vitest.workerd.config.ts @@ -0,0 +1,22 @@ +import { cloudflareTest } from "@cloudflare/vitest-pool-workers"; +import { defineConfig } from "vitest/config"; + +const CONFIGURATION_SHA256 = "fb2dfea638ca6210e7d15bf12e9bf3c91009d54c8a581dc3a477accd722bb9c7"; + +export default defineConfig({ + plugins: [ + cloudflareTest({ + wrangler: { configPath: "./wrangler.example.jsonc" }, + miniflare: { + bindings: { + UPLOAD_PATH: "/v1/upload", + ALLOWED_CONFIGURATION_SHA256: CONFIGURATION_SHA256, + ALLOWED_RESEARCHER_KEY_ID: "vector-hpke", + }, + }, + }), + ], + test: { + include: ["tests/receiver.workerd.test.ts"], + }, +}); diff --git a/receiver/worker-configuration.d.ts b/receiver/worker-configuration.d.ts new file mode 100644 index 0000000..66299ee --- /dev/null +++ b/receiver/worker-configuration.d.ts @@ -0,0 +1,12 @@ +declare namespace Cloudflare { + interface Env { + BUNDLES: R2Bucket; + UPLOAD_PATH: string; + ALLOWED_CONFIGURATION_SHA256: string; + ALLOWED_RESEARCHER_KEY_ID: string; + } + + interface GlobalProps { + mainModule: typeof import("./src/index"); + } +} diff --git a/receiver/wrangler.example.jsonc b/receiver/wrangler.example.jsonc new file mode 100644 index 0000000..71a48f3 --- /dev/null +++ b/receiver/wrangler.example.jsonc @@ -0,0 +1,24 @@ +{ + "$schema": "node_modules/wrangler/config-schema.json", + "name": "adc-ciphertext-receiver", + "main": "src/index.ts", + "compatibility_date": "2026-07-30", + "workers_dev": false, + "routes": [ + { + "pattern": "receiver.example.invalid", + "custom_domain": true + } + ], + "r2_buckets": [ + { + "binding": "BUNDLES", + "bucket_name": "adc-ciphertext-replace-me" + } + ], + "vars": { + "UPLOAD_PATH": "/REPLACE_WITH_UPLOAD_PATH", + "ALLOWED_CONFIGURATION_SHA256": "REPLACE_WITH_64_LOWERCASE_HEX", + "ALLOWED_RESEARCHER_KEY_ID": "REPLACE_WITH_RESEARCHER_KEY_ID" + } +} From 38895d548916b038b9654f74485a225b7291e449 Mon Sep 17 00:00:00 2001 From: JacobLinCool Date: Wed, 5 Aug 2026 01:33:02 +0800 Subject: [PATCH 3/4] Align CI and documentation with the P0-P2 architecture Run Protocol, catalog, Collector capability, Android, Web, receiver, and analysis checks in their appropriate workflows while retaining debug APK and signed release publication. Document the final trust boundaries, participant and researcher flows, component ownership, data semantics, release process, and the simplified non-evidence assurance policy. --- .github/workflows/ci.yml | 25 +- .github/workflows/pages.yml | 28 +- .github/workflows/release.yml | 13 +- README.md | 77 +++- assurance/README.md | 17 + docs/component-boundaries.md | 43 +- docs/data-collector-implementation-guide.md | 463 +++++++------------- docs/data-dictionary.md | 231 ++++++++-- docs/maintainers/release.md | 4 +- docs/p0-p2-implementation-contract.md | 145 ++++++ docs/participant-guide.md | 48 +- docs/researcher-guide.md | 402 ++++++++++------- docs/system-design.md | 315 +++++++------ docs/threat-model.md | 124 +++++- 14 files changed, 1217 insertions(+), 718 deletions(-) create mode 100644 assurance/README.md create mode 100644 docs/p0-p2-implementation-contract.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 93b03c2..88ec2ac 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -52,6 +52,13 @@ jobs: distribution: temurin java-version: "17" + - name: Validate Protocol v1 and collector contracts + run: | + python3 tools/catalog.py check + python3 tools/catalog_parity.py + python3 tools/validate_protocol_vectors.py + python3 -m unittest discover -s tools/tests -v + # The wrapper JAR is a binary in the tree; this checks it against Gradle's published # checksums so nobody has to take it on faith. - name: Validate the Gradle wrapper @@ -81,10 +88,22 @@ jobs: exit 1 fi "$sdkmanager" "$ANDROID_PLATFORM_PACKAGE" "$ANDROID_BUILD_TOOLS_PACKAGE" - echo "$(dirname "$sdkmanager")" >> "$GITHUB_PATH" + dirname "$sdkmanager" >> "$GITHUB_PATH" + + - name: Test, lint, and build + run: >- + ./gradlew --no-daemon + test testDebugUnitTest lintDebug assembleDebug assembleRelease + + - name: Consume shared Protocol v1 vectors in Kotlin + run: >- + ./gradlew --no-daemon + -I tools/protocol-conformance.init.gradle + :core:export:test + --tests adc.conformance.ProtocolConformanceTest - - name: Verify and build - run: ./gradlew --no-daemon test testDebugUnitTest lintDebug assembleDebug assembleRelease + - name: Enforce collector capability boundary + run: python3 tools/collector_assurance.py - name: Upload debug APK uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index f31e853..39aa5a1 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -1,10 +1,8 @@ name: Pages -# A denylist, not the allowlist this used to be, because `tests/compat.spec.ts` byte-compares this -# site's encoder against `researcher-tools` in a JVM. That makes the Kotlin codec an input to the -# web tests, and naming exactly which Kotlin would mean restating researcher-tools' dependency list -# here and keeping it in step forever. Running on a core change that turns out to be irrelevant -# costs two minutes; missing one publishes a site that signs configurations the app then rejects. +# A denylist keeps the Web authoring surface and shared Protocol corpus in step without duplicating +# their dependency graph here. The TypeScript conformance suite consumes the same checked-in +# vectors as Kotlin and Python; this workflow does not build a JVM artifact. on: push: branches: @@ -52,17 +50,6 @@ jobs: with: persist-credentials: false - # The compatibility suite shells out to researcher-tools and builds it if it is missing, so - # the byte-match this workflow claims to enforce needs a JVM to be enforceable at all. - - name: Set up JDK 17 - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 - with: - distribution: temurin - java-version: "17" - - - name: Set up Gradle - uses: gradle/actions/setup-gradle@0f4528296b4bc09e8ae0fc7be30185a4ab435545 # v6.0.0 - - name: Set up pnpm uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 with: @@ -79,15 +66,18 @@ jobs: working-directory: web run: pnpm install --frozen-lockfile - # The site signs study configurations, so it publishes only what type-checks and only what - # still byte-matches the Kotlin encoder these tests compare against. + # The site signs study configurations, so it publishes only what type-checks and consumes + # the reproducible shared Protocol v1 corpus without relaxing a hostile vector. - name: Type-check working-directory: web run: pnpm run check - name: Test working-directory: web - run: pnpm run test + run: | + pnpm run test + node ../tools/generate_protocol_vectors.mjs --check + pnpm exec vitest run --config ../tools/conformance/vitest.config.ts # A project site is served from //, and `svelte.config.js` bakes BASE_PATH into every # URL the build emits. No trailing slash: SvelteKit rejects one. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4c91b53..0d9f35f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -102,7 +102,7 @@ jobs: exit 1 fi "$sdkmanager" "$ANDROID_PLATFORM_PACKAGE" "$ANDROID_BUILD_TOOLS_PACKAGE" - echo "$(dirname "$sdkmanager")" >> "$GITHUB_PATH" + dirname "$sdkmanager" >> "$GITHUB_PATH" - name: Materialize release-signing files shell: bash @@ -129,6 +129,17 @@ jobs: -PreleaseVersionName=${{ steps.release_metadata.outputs.version_name }} -PreleaseVersionCode=${{ github.run_number }} + - name: Recheck Protocol v1 and collector contracts + run: | + python3 tools/catalog.py check + python3 tools/catalog_parity.py + python3 tools/validate_protocol_vectors.py + python3 tools/collector_assurance.py + ./gradlew --no-daemon \ + -I tools/protocol-conformance.init.gradle \ + :core:export:test \ + --tests adc.conformance.ProtocolConformanceTest + - name: Verify and package signed APK id: signed_apk shell: bash diff --git a/README.md b/README.md index ce52766..94633fb 100644 --- a/README.md +++ b/README.md @@ -10,23 +10,28 @@ Standing up a mobile sensing study normally means writing an Android app, gettin ## How a study works -1. **Generate your keys.** One Ed25519 pair to sign study configurations, one HPKE pair to decrypt exports. `researcher-tools` does both. -2. **Write the study.** A strict v1 JSON file naming collectors, reusable surveys, scheduled interventions, anonymous or assigned-code identity mode, duration, storage quota, consent text, and signing/export public keys. +1. **Generate your keys.** One Ed25519 pair to sign study configurations, one X25519 HPKE pair to decrypt bundles. `researcher-tools` writes raw 32-byte keys as unpadded base64url. +2. **Write the study.** A strict Protocol v1 RFC 8785 JSON file naming collectors, reusable surveys, scheduled interventions, anonymous or assigned-code identity mode, duration, storage quota, consent text, and signing/export public keys. 3. **Sign it.** `researcher-tools sign` produces a `.adccfg` file. Because the signing public key travels inside the signed bytes, any build of the app can verify it. 4. **Distribute.** Participants install the app and import your `.adccfg`. Setup is five steps, one screen each — the study details, what each enabled collector records and does not record, the consent text with the signer's key fingerprint, the Android access your collectors need, and the start button — and collection begins only when they press it. 5. **Collect.** Events are written to encrypted on-device storage. Participants can pause, resume, finish early, or withdraw. -6. **Export and analyse.** The participant exports an encrypted bundle and sends it to you. If the study declares an upload endpoint, the app also delivers the same encrypted bundles to it on a schedule. `researcher-tools decrypt` turns either one into JSON. +6. **Export and analyse.** The participant exports an encrypted bundle and sends it to you. If the study declares an upload endpoint, the app also delivers immutable ciphertext bundles to an R2 receiver on a schedule. `adc-analysis` inventories, verifies, decrypts, reassembles, and writes typed Parquet offline. The full procedure, including key handling and study design guidance, is in the [researcher guide](docs/researcher-guide.md). ## What you can collect -Seven collectors ship in v1. A study enables the ones it names and configures each one's parameters within validated ranges. +Twelve selectable collectors ship in v1. A study enables the ones it names and configures each one's parameters within validated ranges. | Collector | Records | Cannot establish | | --- | --- | --- | | `app_lifecycle.v1` | Lifecycle of this app's own activities | Use of any other app | | `accelerometer.v1` | Raw x/y/z acceleration, sensor time, accuracy | Recognised motion, posture, or activity | +| `battery_state.v1` | Battery percentage, charging state/source, power-save state | Battery health, temperature, or hardware identity | +| `temporal_context.v1` | Time-zone ID, UTC offset, DST state, clock-change reason | Location or travel | +| `gyroscope.v1` | Raw x/y/z angular velocity, sensor time, accuracy | Orientation, posture, or activity | +| `ambient_light.v1` | Raw illuminance, sensor time, accuracy | Environmental content or presence | +| `proximity.v1` | Raw distance, sensor range, near/far interpretation | Comparable physical distance across devices or presence | | `network_state.v1` | Default network transport, validated/metered/roaming/VPN flags, bandwidth estimates | Addresses, destinations, or content | | `network_usage.v1` | Device-total Wi-Fi and mobile rx/tx bytes and packets per interval | Instantaneous throughput, per-app attribution, exact timing | | `usage_events.v1` | Raw app, screen, keyguard, and boot events | A complete or real-time session stream | @@ -41,18 +46,21 @@ Some practical notes: package names, location, fine-grained timing, acceleration The collector set is meant to grow. A collector is a Gradle module implementing three things: a typed configuration that appears in the signed study file, a plugin descriptor declaring what access it needs, and a runtime instance that observes its source and emits events. -Collector modules depend only on `core:collector-api` and `core:study-definition`, so a new data source does not touch storage, the runtime, or the protocol layer. The [implementation guide](docs/data-collector-implementation-guide.md) walks through the contract and every registration step, with a complete worked example. +Collector modules depend only on `core:collector-api`, `core:study-definition`, and, for Android +hardware listeners, the narrow `collector:sensor-common` lifecycle helper. A new data +source does not touch storage, the runtime, or protocol/export code. The +[implementation guide](docs/data-collector-implementation-guide.md) walks through the contract and every registration step. ## Participant data protection Studies collect from people's personal phones, so the platform is built to support a defensible ethics submission and honest commitments to participants. - **Encrypted on the device.** Each study's events and metadata are encrypted with a per-study AES-256-GCM key from the Android Keystore, marked non-exportable, in 4 MiB event segments under the app's no-backup storage, up to the quota the configuration set. An event is appended once and never rewritten; the only thing that removes a segment is confirmed delivery to the study's endpoint, and then only under storage pressure. -- **Signed, tamper-evident studies.** A configuration is Ed25519-signed and strictly validated: canonical encoding, exact schema, known collectors, validity window, minimum app version. Verification failures are fail-closed — an unverifiable configuration collects nothing. A signature proves the configuration is unchanged since it was signed; it does not prove who wrote it unless the build pins that signer, and the consent screen states which of the two applies. -- **Durable interventions and native surveys.** Notification actions can use one-time, recurring, or daily-local triggers. Their occurrences survive retries and reboot without duplication. Native surveys support short text, integer scales, single choice, and multiple choice; only a confirmed, complete submission enters the encrypted event stream. -- **Separated participant identities.** Every import gets a fresh random instance UUID. A configuration may additionally carry an opaque researcher-assigned code; both appear inside encrypted exports, while upload headers expose only the instance UUID used for routing and de-duplication. +- **Signed, tamper-evident studies.** A configuration is Ed25519-signed and strictly validated: RFC 8785 bytes, exact schema, known collectors, Android platform, validity window, and minimum client build. Verification failures are fail-closed — an unverifiable configuration collects nothing. A signature proves the configuration is unchanged since it was signed; it does not prove who wrote it unless the build pins that signer, and the consent screen states which of the two applies. +- **Durable interventions and native surveys.** Notification actions can use one-time, recurring, daily-local, or signed random-local-window triggers. Random instants are selected with a CSPRNG and persisted before scheduling, so retries and reboot do not redraw them; clock/time-zone changes do not rewrite already materialized occurrences. Native surveys support short text, integer scales, single choice, and multiple choice; only a confirmed, complete submission enters the encrypted event stream. +- **Separated participant identities.** Every import gets a fresh random instance UUID. A configuration may additionally carry an opaque researcher-assigned code; both appear in the encrypted document. Upload URLs and headers contain no participant, assigned, experiment, or configuration ID. Their bundle UUID, configuration digest, researcher key ID, exact range/count, size, and digest are untrusted routing claims, not participant authentication. - **Encrypted, participant-directed export.** Getting data to the research team is an export the participant performs and directs, encrypted with a fresh key per export and wrapped to your HPKE public key. The app never holds your private key. -- **Scheduled upload, when the study asks for it.** A configuration may name an HTTPS endpoint, interval, and metered-network policy. The endpoint host, cadence, and network condition are shown before consent. Each run sends outstanding events up to a 16 MiB plaintext budget and records its exact boundary. Finishing or withdrawing cancels future interventions and the deadline, while delivery continues until the undelivered tail arrives. Undelivered events are never reclaimed to make room. +- **Scheduled upload, when the study asks for it.** A configuration may name an HTTPS endpoint, interval, and metered-network policy. The endpoint host, cadence, and network condition are shown before consent. Before HTTP starts, the app durably stages one immutable ciphertext bundle in no-backup storage: about 16 MiB of plaintext and at most 32 MiB on the wire. Retries send those exact bytes with fixed length and digest. Only a matching seven-field receipt on `201 Created` or exact-replay `200 OK` advances the watermark; redirects, `202`, malformed receipts, and other terminal responses do not. Finishing or withdrawing leaves delivery running until the tail arrives. Undelivered events are never reclaimed to make room. - **Participant control over the lifecycle.** Collection starts only on an explicit action and can be paused, finished, or withdrawn. Pausing takes a monotonic boundary, so delayed callbacks cannot leak post-pause data into the dataset. - **Storage failures stop collection.** Quota exhaustion or a write failure fail-closes the study to `PAUSED` rather than silently dropping events, so a dataset is complete over the window it declares or absent. @@ -82,6 +90,27 @@ With an emulator or device attached: ./gradlew :core:storage:connectedDebugAndroidTest :app:connectedDebugAndroidTest ``` +The app suite separates the Android signed-configuration regression +([`AndroidConfigurationImportTest`](app/src/androidTest/kotlin/cool/linc/androiddatacollector/AndroidConfigurationImportTest.kt)), +the full participant UI flow ([`CoreFlowTest`](app/src/androidTest/kotlin/cool/linc/androiddatacollector/CoreFlowTest.kt)), +and the five-collector Android integration +([`P2CollectorEmulatorTest`](app/src/androidTest/kotlin/cool/linc/androiddatacollector/P2CollectorEmulatorTest.kt)). +The last test skips when gyro, light, or proximity hardware is absent. Its optional exact-value mode +expects a sensor-capable emulator that the host has already configured; the test does not fake +Android's sensor APIs: + +```bash +adb -s emulator-5554 emu power ac on +adb -s emulator-5554 emu power status charging +adb -s emulator-5554 emu power capacity 73 +adb -s emulator-5554 emu sensor set gyroscope 1.25:-2.5:0.5 +adb -s emulator-5554 emu sensor set light 123 +adb -s emulator-5554 emu sensor set proximity 1 +./gradlew :app:connectedDebugAndroidTest \ + -Pandroid.testInstrumentationRunnerArguments.class=cool.linc.androiddatacollector.P2CollectorEmulatorTest \ + -Pandroid.testInstrumentationRunnerArguments.p2SyntheticInputs=true +``` + The debug APK lands at `app/build/outputs/apk/debug/app-debug.apk`. A clean checkout has no signing material, so `assembleRelease` produces an unsigned release APK. ### Try it without a real study @@ -94,10 +123,10 @@ For that reason a **release build ships no demonstration study** — the signed ```text signing-keygen generate an Ed25519 signing pair -hpke-keygen generate a Tink HPKE keyset pair +hpke-keygen generate a raw X25519 HPKE key pair canonicalize strictly parse and emit a canonical configuration sign sign a canonical configuration into .adccfg -check-config verify envelope, signature, validity window, app version; optionally pin the signer +check-config verify envelope, signature, platform, validity window, and client build; optionally pin the signer decrypt decrypt an .adcexp into research-bundle-v1 JSON ``` @@ -115,6 +144,8 @@ flowchart LR Session --> Export[":core:export"] Export --> Crypto[":core:crypto"] Android[":app Android adapters"] --> Session + Android --> Receiver["receiver/ Cloudflare Worker"] + Receiver --> R2["private R2 ciphertext"] Access[":core:access"] --> Session Protocol[":core:protocol"] --> Definition[":core:study-definition"] Tools[":researcher-tools"] --> Definition @@ -127,9 +158,9 @@ flowchart LR | `:app` | Compose UI, finite UI state, SAF, and Android foreground/work/recovery/upload adapters | | `:core:model` | Bounded study metadata, state and event models, `StudyStore` port and its retained window | | `:core:study-definition` | Strict canonical JSON, closed-world typed study and collector configuration | -| `:core:protocol` | Signed envelope, signature verification, optional signer pinning, validity and version checks | +| `:core:protocol` | Signed envelope, immutable join URI, signature verification, optional signer pinning, validity and version checks | | `:core:collector-api` | Collector lifecycle, health, registry, access contract, shared callback dispatcher | -| `:core:crypto` | Tink HPKE key generation, wrapping, and unwrapping | +| `:core:crypto` | Protocol v1 raw-key Ed25519 verification and fixed-suite RFC 9180 HPKE over raw X25519 keys; Tink is internal only, never a wire keyset | | `:core:access` | Runtime permission, Usage Access, input method, and hardware preflight | | `:core:experiment-runtime` | Command serialisation, state machine, collector supervision, event admission gate | | `:core:study-application` | The single active-study session, recovery, port coordination, and the upload watermark | @@ -137,9 +168,24 @@ flowchart LR | `:core:export` | Streaming JSON to AES-GCM over a sequence window under an optional size budget, HPKE key wrapping, receipts | | `:collector:*` | One isolated module per data source | | `:researcher-tools` | Ed25519 and HPKE keys, canonicalise, sign, verify, decrypt CLI | +| `receiver/` | One bounded Protocol v1 upload POST, immutable ciphertext writes, and canonical receipts | Platform-independent modules contain no `android.*` imports, which keeps the domain logic testable on the JVM. [Component boundaries](docs/component-boundaries.md) documents the contracts. +New contributors should treat [`protocol/v1`](protocol/v1/README.md) as the normative wire contract, the [collector catalog](protocol/v1/collector-catalog.json) as the schema source, and [`docs/p0-p2-implementation-contract.md`](docs/p0-p2-implementation-contract.md) as the implementation decision record. Trace one path through the [configuration codec](core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt), [signed envelope](core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/SignedConfiguration.kt), [bundle exporter](core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchExport.kt), [bundle verifier](core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt), [single-entry outbox](app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt), [HTTP adapter](app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt), [receiver handler](receiver/src/index.ts), and the offline [`adc-analysis`](adc-analysis/README.md) pipeline. The join path is similarly short: [Web authoring](web/src/lib/adc/join.ts), [shared parser](core/protocol/src/main/kotlin/cool/linc/androiddatacollector/core/protocol/JoinLink.kt), [Android staging](app/src/main/kotlin/cool/linc/androiddatacollector/platform/JoinArtifactDownloader.kt), [intent entry](app/src/main/kotlin/cool/linc/androiddatacollector/MainActivity.kt), then the existing [session import](core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt). The [outbox](app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt), [uploader](app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt), and [receiver](receiver/tests/receiver.test.ts) tests make crash/replay and receipt semantics executable. Receiver deployment and R2 operations start at [`receiver/README.md`](receiver/README.md), and the Collector capability policy lives under [`assurance`](assurance/README.md). + +For `random_window`, trace the signed model and bounds in +[`StudyConfiguration.kt`](core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt), +its codec and [Web editor](web/src/routes/researcher/InterventionEditor.svelte), then the CSPRNG +materialization in +[`InterventionSchedulePlanner.kt`](core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/InterventionSchedulePlanner.kt). +The [session](core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt) +persists the occurrence before scheduling; the Android delivery/expiry workers in +[`AndroidStudyPlatform.kt`](app/src/main/kotlin/cool/linc/androiddatacollector/platform/AndroidStudyPlatform.kt) +and [`BootRecoveryReceiver`](app/src/main/kotlin/cool/linc/androiddatacollector/BootRecoveryReceiver.kt) +reconcile the same ID after retries, reboot, clock, or time-zone changes. The adjacent planner, +runtime, session, and app policy tests make each boundary executable. + ## Documentation | Document | For | @@ -151,6 +197,11 @@ Platform-independent modules contain no `android.*` imports, which keeps the dom | [System design](docs/system-design.md) | The implemented v1 architecture in full | | [Component boundaries](docs/component-boundaries.md) | Module contracts and invariants | | [Threat model](docs/threat-model.md) | Trust assumptions and limitations, for ethics review | +| [Normative Protocol v1](protocol/v1/README.md) | JCS, keys, join URI, binary framing, bundle document, upload, receipt, and conformance corpora | +| [P0–P2 implementation contract](docs/p0-p2-implementation-contract.md) | Locked architectural decisions and scope | +| [Collector capability policy](assurance/README.md) | Static source, bytecode, and dependency boundaries for collectors | +| [Ciphertext receiver](receiver/README.md) | R2-only Worker contract, verification commands, deployment, and operations | +| [Offline analysis](adc-analysis/README.md) | Ciphertext inventory, verification, reassembly, and typed Parquet materialization | | [Release process](docs/maintainers/release.md) | Maintainers | ## Contributing diff --git a/assurance/README.md b/assurance/README.md new file mode 100644 index 0000000..a7f304e --- /dev/null +++ b/assurance/README.md @@ -0,0 +1,17 @@ +# Collector capability policy + +`collector-policy.json` defines the static capability boundary enforced by +`tools/collector_assurance.py`. The check covers collector source imports, direct Gradle +dependencies, and compiled production class constant pools. + +`:collector:sensor-common` is the sole allowed collector-to-collector dependency. It is scanned by +the same policy and exposes only shared Android sensor-listener lifecycle ownership. + +Run the check after compiling collector modules: + +```bash +python3 tools/collector_assurance.py +``` + +This policy complements the runtime payload schema and size checks. It is not telemetry and does +not collect data from participant devices. diff --git a/docs/component-boundaries.md b/docs/component-boundaries.md index 63ede25..77be349 100644 --- a/docs/component-boundaries.md +++ b/docs/component-boundaries.md @@ -5,6 +5,15 @@ entry points. Each contract states what a module owns and, where it matters, wha sees. The dependency direction in `settings.gradle.kts` and the module build files is the enforcement mechanism. +New contributors should read this with the [normative Protocol v1 contract](../protocol/v1/README.md), +the [collector catalog](../protocol/v1/collector-catalog.json), the +[P0–P2 implementation contract](p0-p2-implementation-contract.md), and +[`assurance`](../assurance/README.md). The concrete upload seam is deliberately short: +[StudyUploader](../core/study-application/src/main/kotlin/cool/linc/androiddatacollector/core/application/StudyApplication.kt), +[FileUploadOutbox](../app/src/main/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutbox.kt), +[OkHttpStudyUploader](../app/src/main/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploader.kt), +and their neighbouring tests. + ```text participant UI -> study application -> study domain |-> collector runtime -> collector API <- collector features @@ -18,16 +27,18 @@ researcher tools -> study definition + signed protocol + export format - `:core:model` owns finite study metadata, events, state transitions, and the `StudyStore` port. It never owns an unbounded event list. - `:core:study-definition` owns the strict, closed-world study schema and its canonical codec. -- `:core:protocol` owns only the signed envelope and trust verification. -- `:core:collector-api` is the only core module a collector feature depends on directly. It - owns collector lifecycle, health, event admission contracts, capabilities, and shared - serialized callback delivery. Its `CollectorContext` hands a collector a coroutine scope, an +- `:core:protocol` owns the signed envelope, immutable join-URI parser, and trust verification. +- `:core:collector-api` is the only runtime-facing core API a collector feature depends on; a + collector also depends on `:core:study-definition` for its closed typed configuration. Hardware + sensor modules may additionally use the narrow `:collector:sensor-common` lifecycle helper. + `:core:collector-api` owns collector lifecycle, health, event admission contracts, capabilities, + and serialized callback delivery. Its `CollectorContext` hands a collector a coroutine scope, an `EventSink`, and a clock — no store, no state machine, no scheduler, no exporter. - `:core:experiment-runtime` owns command serialization, state transitions, admission, and collector supervision. It is platform independent. - `:core:study-application` owns study use cases and coordinates injected storage, access, foreground-host, scheduling, export, and upload ports. The `StudyUploader` port takes a - sequence window and returns a receipt; the HTTP client behind it lives in `:app`. + sequence window and returns a verified receipt; durable staging and HTTP live in `:app`. - Android entry points and adapters live in `:app`; they do not duplicate recovery or study policy. - `:core:storage`, `:core:export`, and `:core:crypto` implement the encrypted data boundary @@ -43,22 +54,26 @@ researcher tools -> study definition + signed protocol + export format removes whole leading segments and never reuses an index or a sequence number. Runtime snapshots carry bounded metadata — counters, the transition history, and the last event per collector — never the event log. -- A bundle carries exactly the window it holds, while later collection may continue. A - participant export takes the whole retained window with no size budget. An upload asks for - everything after the endpoint's last confirmation and stops at the first event boundary past - its plaintext budget, so where it ends is decided while it streams and is reported in the - receipt. `first_sequence_number` and `last_sequence_number` are written after the `events` - array for that reason: a bundle never declares a range before it knows it. -- The upload watermark advances only on a confirmed delivery, only as far as the receipt says - the bundle reached, and never backwards. A confirmed delivery is the only thing that makes +- A bundle's authenticated JCS document declares exactly the contiguous window it holds. A + participant export takes the retained window and may scale to the local quota. Automatic upload + selects a bounded exact window, then durably stages one complete ciphertext bundle plus its + manifest before HTTP. At most one entry exists; reboot, process death, retry, and response loss + reuse its bundle ID, digest, range, length, and bytes. +- The upload watermark advances only on `201 Created` or exact-replay `200 OK` with a canonical + seven-field receipt matching the outbox manifest, and never backwards. Redirects, `202`, other + statuses, and malformed or mismatched receipts cannot commit. A confirmed delivery is the only thing that makes local data reclaimable. Reclaiming starts only above 80% of the study's quota and takes whole leading segments at or below the watermark. Undelivered data is never released to make room. - Study metadata is self-sufficient: opening a study validates framing and sequence contiguity from the plaintext frame headers and reads `lastEvents` from the metadata, so load cost is - linear in frames rather than in bytes decrypted. Event payloads are authenticated when read. + linear in frames rather than in bytes decrypted. Event payloads are authenticated when read; + the one exception is exact append-journal recovery, which authenticates only the durable tail. Reading a range decrypts only that range and seeks past the frames below it. - Configuration decoding stays canonical, strict, and compile-time allowlisted. There is no legacy or fallback reader. +- The collector catalog is the shared schema source, but not a runtime plugin mechanism. Runtime + validates payload schema and `maximumEncodedEventBytes`; the Collector capability policy + constrains source, bytecode, and dependencies in CI. - One process-scoped study session owns recovery and runtime lifetime. Android receivers, workers, services, and UI delegate to it. diff --git a/docs/data-collector-implementation-guide.md b/docs/data-collector-implementation-guide.md index 3b4db2c..7a3666b 100644 --- a/docs/data-collector-implementation-guide.md +++ b/docs/data-collector-implementation-guide.md @@ -6,10 +6,13 @@ of the system depends on. Read [System design](system-design.md) first for the module map, and [Component boundaries](component-boundaries.md) for the responsibility split. This document -covers only the collector side of that boundary. - -Everything below is current source. Where a field or a rule exists but nothing enforces it, -this guide says so — see [Known gaps](#13-known-gaps). +covers only the collector side of that boundary. The [normative Protocol v1 contract](../protocol/v1/README.md) +defines the enclosing configuration and bundle. Its machine-readable schema source is the +[Protocol v1 collector catalog](../protocol/v1/collector-catalog.json); the generated Kotlin +projection is +[`ProtocolEventContracts.kt`](../core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/ProtocolEventContracts.kt). +Read the [Collector capability policy](../assurance/README.md) before adding a module; CI enforces +its source, bytecode, and dependency boundaries. ## 1. What a collector is @@ -66,16 +69,16 @@ and must narrow it themselves. Every existing plugin rejects a mismatch with ```kotlin data class CollectorDescriptor( val id: String, - val payloadSchemaVersion: Int, val displayName: String, val privacyClass: PrivacyClass, - val maximumEncodedEventBytes: Int, + val eventContract: CollectorEventContract, ) { + val payloadSchemaVersion get() = eventContract.payloadSchemaVersion + val maximumEncodedEventBytes get() = eventContract.maximumEncodedEventBytes + init { require(ID_PATTERN.matches(id)) { "Invalid collector ID" } - require(payloadSchemaVersion > 0) { "Payload schema version must be positive" } require(displayName.isNotBlank()) { "Collector display name must not be blank" } - require(maximumEncodedEventBytes in 128..65_536) { "Invalid maximum event size" } } private companion object { @@ -89,6 +92,10 @@ enum class PrivacyClass { } ``` +`CollectorEventContract` supplies the closed payload-type/field set, field types and bounds, +payload schema version, and maximum encoded size. Plugins obtain it from +`ProtocolEventContracts[ID]`; editing the generated file directly is forbidden. + ### Context and clocks ```kotlin @@ -105,7 +112,7 @@ interface ResearchClocks { Three handles. A coroutine scope, an event sink, and a clock. There is no store, no state machine, no scheduler, no exporter, and no `Activity`. That narrowness is the boundary — -see [section 3](#3-invariants-you-must-not-break). +see [section 3](#3-invariants-a-collector-must-hold). `ResearchTime` carries three readings taken together ([`core/model/.../ExperimentModels.kt`](../core/model/src/main/kotlin/cool/linc/androiddatacollector/core/model/ExperimentModels.kt)): @@ -203,27 +210,27 @@ change one of these, raise it as a design discussion first. | A collector does not | Why | Boundary that enforces it | | --- | --- | --- | -| Write files, databases, or preferences | Every research byte must go through the encrypted store so that sequence numbers stay contiguous and export, upload, and reclaiming can all reason about one window. A side file is invisible to export, to the storage quota, and to deletion. | The only project dependencies in a `:collector:*` build file are `:core:collector-api` and `:core:study-definition`. `:core:storage` is not on the classpath, and `CollectorContext` carries no `StudyStore`. | +| Write files, databases, or preferences | Every research byte must go through the encrypted store so that sequence numbers stay contiguous and export, upload, and reclaiming can all reason about one window. A side file is invisible to export, to the storage quota, and to deletion. | Feature modules depend on `:core:collector-api`, `:core:study-definition`, and optionally `:collector:sensor-common`; `:core:storage` is not on the classpath, and `CollectorContext` carries no `StudyStore`. | | Change study state | `IMPORTED` → … → `WITHDRAWN` is the participant's control surface. A collector that could move it could un-pause a study the participant paused. | `ExperimentStateMachine.transition` is called only from `ExperimentRuntime` in `:core:experiment-runtime`, which no collector module depends on. | | Start an `Activity` or drive UI | The app must never interrupt the participant on a collector's schedule. | Collector modules do not depend on `:app`. Plugins are constructed with `context.applicationContext`. | | Schedule interventions or notifications | Intervention timing and occurrence identity come from the signed configuration and are reconciled by the session manager. | The `StudyWorkScheduler` port is declared in `:core:study-application`; the WorkManager adapter is in `:app`. Neither is on a collector's classpath. | | Export, encrypt, or package data | Export is a participant-initiated act over a bounded sequence window, encrypted to a researcher HPKE key. | `:core:export` and `:core:crypto` are not on any collector's classpath. | -| Open a socket or upload | Network transport lives in the study application layer, where the `StudyUploader` port sends only the encrypted bundle, only to the endpoint the signed configuration names, and only after the consent screen has disclosed it. A collector reaching the network would bypass all three, and the participant guide and deployed consent texts describe a study's transmission as coming from that one place. | The app declares `android.permission.INTERNET` for the upload worker, so the permission is present in the process and no manifest check will catch a collector using it. `:core:export`, `:core:crypto`, and the uploader are off a collector's classpath, and `CollectorContext` exposes no network client. Review is what enforces the rest. | +| Open a socket or upload | Network transport lives in the study application layer, where the `StudyUploader` sends only a staged encrypted bundle to the signed endpoint. A collector reaching the network would bypass signed scope and consent. | `CollectorContext` exposes no network client, and forbidden network classes, imports, and dependencies fail the Collector capability check. | | Record text, characters, or content typed on the research keyboard | The consent text tells participants the keyboard never sees what they write. Touch dynamics research does not need the characters, so the characters are never carried across the boundary. | `ResearchKeyboardView.onTouchEvent` passes `key.category.name` to `ImeObservationBridge.publish`, never `key.text`. `ImeTouchObservation` has no field that could hold a character. The committed text goes to `InputConnection` and stops there. | | Log payload values, paths, package names, or exception messages | A logcat line is readable by anyone with adb access and is not covered by the encrypted store. | `CollectorHealth.reasonCode` is constrained to `[A-Z][A-Z0-9_]{2,63}`, which cannot hold free text. There is no other diagnostic channel in the API. | -### What is not enforced +### Static policy is not a sandbox -None of the above is a sandbox. A collector module is Kotlin compiled into the same process -with the app's permissions; `java.io.File` and `Context.startActivity` are reachable from -any of them. The Gradle dependency graph is the real enforcement point for most of these -rules, and there is currently **no automated architecture test** asserting it. A collector -that broke these rules would compile and pass CI. Review is the backstop. +Collectors still run in the app process with its permissions, so these checks are not operating +system isolation. The repository nevertheless fails CI when a collector crosses its declared +capability boundary. `tools/collector_assurance.py` inspects source imports, direct Gradle +dependencies, and compiled class constant pools against `assurance/collector-policy.json`. -Verify the graph yourself: +Run the same capability check locally after compiling collectors: ```bash ./gradlew :collector:accelerometer:dependencies --configuration debugCompileClasspath +python3 tools/collector_assurance.py ``` Verify the permission set of a built APK, which is stronger than reading a single manifest: @@ -244,8 +251,8 @@ POST_NOTIFICATIONS RECEIVE_BOOT_COMPLETED WAKE_LOCK plus the signature-level `cool.linc.androiddatacollector.DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION` that AndroidX contributes. `INTERNET` belongs to the study application layer's upload worker; -its presence no longer tells you whether any given study transmits, and it means a collector's -own network call would not show up here either. +its presence no longer tells you whether any given study transmits, which is why bytecode and +dependency policy are required in addition to a permission diff. ## 4. Module layout and dependency direction @@ -280,15 +287,15 @@ fun pluginFor(configuration: CollectorConfiguration): CollectorPlugin = ### Identity and versioning - The ID must match `[a-z][a-z0-9_.-]{2,63}`. Every existing collector uses `.v`. -- The ID is immutable once a study has shipped. Changing what an existing payload type - means, changing units, or removing a field is a new major ID and a new configuration type. +- The ID is immutable once a study has shipped. Changing a field's meaning, type, unit, precision, + or clock basis requires a new collector/payload schema identity in the catalog. - `payloadSchemaVersion` versions the payload independently of the configuration schema and independently of the ID. It is stamped on every `EventDraft` and travels into the export. -- `maximumEncodedEventBytes` must be in `128..65_536`. Declare a bound you can actually - justify from the field list. Note that **nothing reads this field today** — see - [Known gaps](#13-known-gaps). The limits the code does enforce are `EventDraft`'s: at most - 32 fields, each value at most 1,024 characters. -- `privacyClass` is `RESTRICTED` for `keyboard_touch.v1` and `SENSITIVE` for the other six. +- `maximumEncodedEventBytes` is generated from the catalog and must be in `128..65_536`. The + runtime validates the complete payload contract and encodes with a worst-case sequence number + before append; an oversized or schema-invalid event is rejected. `EventDraft` independently + limits an event to 32 fields and each value to 60 Ki UTF-16 code units. +- `privacyClass` is `RESTRICTED` for `keyboard_touch.v1` and `SENSITIVE` for the other current collectors. Nothing reads it today either; it documents the author's own classification. ### Strict configuration decoding @@ -299,11 +306,9 @@ It is strict in a specific, checkable way: - `requireExactKeys` demands the exact key set. Unknown keys, missing keys, and renamed keys are all rejected. There are no optional fields with defaults. -- Integers must match `-?(0|[1-9][0-9]*)` as a literal, so `1.0`, `1e3`, and `"1"` are all - rejected for an integer field. -- `decode` re-encodes what it parsed and requires the bytes to be identical - (`require(encode(decoded).contentEquals(bytes))`). Whitespace, key order, and number - formatting are therefore all fixed. Signatures are taken over these canonical bytes. +- Configuration JSON is RFC 8785 JCS. Schema numeric fields are bounded integral JSON numbers; + sequence/time/client-build fields use decimal strings. `decode` re-encodes and requires exact + byte equality, so noncanonical whitespace, ordering, escaping, and number spelling fail. - An unknown collector ID throws. There is no fallback reader and no legacy path. Range checks live in the configuration type's `init` block, not in the codec, so they apply @@ -415,11 +420,12 @@ From [`core/model/.../ExperimentModels.kt`](../core/model/src/main/kotlin/cool/l | `payloadType` | `[A-Z][A-Z0-9_]{1,63}` | | field key | `[a-z][a-z0-9_]{0,63}` | | field count | at most 32 | -| field value length | at most 1,024 characters | +| field value length | at most 60 Ki UTF-16 code units | | field value type | `String` only — encode numbers with `toString()` | -The runtime sorts fields with `toSortedMap()` when it builds the `RecordedEvent`, so field -order in your map does not affect the stored bytes. +Before append, the runtime requires the draft's payload schema, payload type, exact field set, +field values, and worst-case protocol-encoded size to satisfy the catalog-derived event contract. +It then sorts fields with `toSortedMap()`, so map insertion order does not affect stored bytes. ### Source time versus write time @@ -462,6 +468,11 @@ encoding only. | `network_state.v1` | `SerializedCallbackCollector` | 256 | | `location.v1` | `SerializedCallbackCollector` | 512 | | `accelerometer.v1` | `SerializedCallbackCollector` | 2,048 | +| `battery_state.v1` | `SerializedCallbackCollector` | 64 | +| `temporal_context.v1` | `SerializedCallbackCollector` | 64 | +| `gyroscope.v1` | `AndroidSensorCollector` | 2,048 | +| `ambient_light.v1` | `AndroidSensorCollector` | 256 | +| `proximity.v1` | `AndroidSensorCollector` | 256 | | `keyboard_touch.v1` | `SerializedCallbackCollector` | 2,048 | | `network_usage.v1` | `Collector` directly (polling) | none | | `usage_events.v1` | `Collector` directly (polling) | none | @@ -521,7 +532,7 @@ whichever wrote last. Runtime-level incidents (`COMMAND_REJECTED`, `RUNTIME_FAIL `STORAGE_WRITE_FAILED`, `PAUSE_PERSISTENCE_FAILED`) are a separate field, `RuntimeSnapshot.incidentCode`, and are not collector health. -## 10. The seven built-in collectors +## 10. The twelve built-in collectors Each entry states what the collector records and, as importantly, what its data cannot be used to claim. @@ -578,6 +589,43 @@ are Android's, unmodified. The collector performs no filtering, no gravity remov inference. It does not produce step counts, postures, or activity labels; those are the analyst's claims to make and defend. +### `battery_state.v1` + +Empty exact config, no access, 64-event callback queue. A runtime-registered, non-exported +receiver snapshots whole percentage, charging state/source, and power-save mode. It deliberately +does not request serial, health, temperature, current, voltage, or capacity. Exact duplicates are +suppressed and rapid changes retain the newest state under a one-minute bound through the tested +`core:collector-api/LatestValueRateGate`. + +### `temporal_context.v1` + +Empty exact config, no access, 64-event callback queue. A runtime-registered, non-exported +receiver records study start/reconciliation and Android time/time-zone changes as time-zone ID, +UTC offset, DST state, and a bounded reason code. It uses the same tested latest-value rate gate; +a zone setting is never labelled as location or travel. + +### `gyroscope.v1` + +The configuration and Android listener/batching semantics mirror `accelerometer.v1`. +`GYROSCOPE_SAMPLE` carries source elapsed-realtime nanoseconds, raw x/y/z rad/s, and accuracy. +`AndroidSensorCollector` owns its `adc-gyroscope` handler thread and pause/stop cleanup. No +orientation or activity inference is present. + +### `ambient_light.v1` + +`sampling_period_us` is 200,000–10,000,000 and `change_threshold_millilux` is +0–100,000,000. `AndroidSensorCollector` owns a 256-event queue and the `adc-ambient-light` +thread. Non-finite/negative readings are refused; the collector emits raw lux and accuracy only +after both monotonic period and change gates. + +### `proximity.v1` + +`minimum_event_interval_ms` is 100–60,000 and `change_threshold_millimeters` is 0–10,000. +`AndroidSensorCollector` owns a 256-event queue and the `adc-proximity` thread. A tested +latest-value gate retains the newest meaningful sample inside the interval. Payload is raw distance, +declared maximum range, and `distance < maximumRange`; many devices expose binary behavior, so no +cross-device precision or presence claim is made. + ### `network_state.v1` | | | @@ -707,7 +755,7 @@ Configuration fields, all required and exact: | `interval_millis` | 1,000–3,600,000 | | `minimum_interval_millis` | 500 to `interval_millis` | | `maximum_batch_delay_millis` | 0–86,400,000 | -| `minimum_displacement_meters` | 0–10,000 | +| `minimum_displacement_millimeters` | 0–10,000,000 | | `priority` | `BALANCED` or `HIGH_ACCURACY` | Uses Google Play services `FusedLocationProviderClient`. There is no platform @@ -778,117 +826,47 @@ call site. `pressure` and `size` are device-specific normalized values. They are not calibrated newtons or square millimetres and are not comparable across device models. -## 11. Worked example: adding `ambient_light.v1` - -This collector is **not in the repository**. It is written out in full so every file you must -touch appears exactly once. It follows the `accelerometer.v1` pattern, which is the shortest -correct path for a callback source. - -### Step 1 — module - -`settings.gradle.kts`, keeping the include list sorted: - -```kotlin -include( - ":app", - ":collector:accelerometer", - ":collector:ambient-light", - ":collector:app-lifecycle", - // … -) -``` - -`collector/ambient-light/build.gradle.kts`: +## 11. Current example: tracing `ambient_light.v1` -```kotlin -plugins { - alias(libs.plugins.android.library) -} - -android { - namespace = "cool.linc.androiddatacollector.collector.ambientlight" - compileSdk = 37 - defaultConfig { minSdk = 34 } - compileOptions { - sourceCompatibility = JavaVersion.VERSION_17 - targetCompatibility = JavaVersion.VERSION_17 - } -} +`ambient_light.v1` is implemented. This section is a production-code index, not a copied second +implementation. A new engineer should be able to follow the complete feature without searching +for an undocumented registry or convention. -kotlin { - compilerOptions { - jvmTarget = org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17 - allWarningsAsErrors = true - } -} +### Step 1 — module and dependency boundary -dependencies { - implementation(project(":core:collector-api")) - implementation(project(":core:study-definition")) - implementation(libs.coroutines.android) -} -``` +- [`settings.gradle.kts`](../settings.gradle.kts) includes `:collector:ambient-light`. +- [`collector/ambient-light/build.gradle.kts`](../collector/ambient-light/build.gradle.kts) lists + the complete module dependencies: the runtime-facing collector API, typed study definition, + shared sensor lifecycle owner, coroutines, and test-only JUnit. +- [`collector/sensor-common`](../collector/sensor-common) is the sole permitted + collector-to-collector dependency. It owns listener registration, callback serialization, and + teardown, not ambient-light semantics. +- The module needs no permission or Android component, so it has no manifest surface of its own. -Do not add a dependency that is not on this list without understanding which invariant in -[section 3](#3-invariants-you-must-not-break) it weakens. Add a module -`src/main/AndroidManifest.xml` only if you need a permission or an Android component; remember -that anything you declare there is merged into the app manifest. +Any new dependency must remain inside `assurance/collector-policy.json`. Review a new manifest +directly; the Collector capability policy does not inspect app manifests. ### Step 2 — typed configuration -In `core/study-definition/src/main/kotlin/.../StudyConfiguration.kt`, add a member of the -sealed `CollectorConfiguration` interface. Range checks belong here, not in the codec: - -```kotlin -data class AmbientLightConfiguration( - override val required: Boolean, - val samplingPeriodUs: Int, -) : CollectorConfiguration { - override val id: String = ID - - init { - require(samplingPeriodUs in 200_000..10_000_000) { "Invalid ambient-light sampling period" } - } - - companion object { const val ID = "ambient_light.v1" } -} -``` +[`StudyConfiguration.kt`](../core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfiguration.kt) +owns `AmbientLightConfiguration`: `sampling_period_us` is bounded to 200,000–10,000,000 and +`change_threshold_millilux` to 0–100,000,000. Constructor validation is the one range authority; +the catalog and Web editor must match it exactly. ### Step 3 — strict codec -Two edits in `StudyConfigurationCodec.kt`. Decode: - -```kotlin -AmbientLightConfiguration.ID -> { - config.requireExactKeys(setOf("sampling_period_us")) - AmbientLightConfiguration(required, config.requireInt("sampling_period_us")) -} -``` - -Encode — the `when` in `encodeCollector` is exhaustive over the sealed interface, so the -compiler will not let you forget this half: - -```kotlin -is AmbientLightConfiguration -> writer.name("sampling_period_us").value(collector.samplingPeriodUs) -``` - -Both halves must agree exactly, because `decode` re-encodes and compares bytes. +[`StudyConfigurationCodec.kt`](../core/study-definition/src/main/kotlin/cool/linc/androiddatacollector/core/definition/StudyConfigurationCodec.kt) +requires exactly both integer keys and encodes them through the exhaustive sealed-interface +branch. Decode then re-encodes and byte-compares canonical JSON. The compact +[`P2ConfigurationTest`](../core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/P2ConfigurationTest.kt) +covers both boundaries, values just outside them, missing/unknown keys, wrong JSON types, and +canonical round-trip. ### Step 4 — access kind -In `CollectorContracts.kt`: - -```kotlin -enum class AccessKind { - // … - ACCELEROMETER_HARDWARE, - AMBIENT_LIGHT_HARDWARE, -} -``` - -Every `when` over `AccessKind` is exhaustive and every module builds with -`allWarningsAsErrors = true`, so adding a value breaks the build in exactly four places until -you handle it: +`AMBIENT_LIGHT_HARDWARE` is a closed [`AccessKind`](../core/collector-api/src/main/kotlin/cool/linc/androiddatacollector/core/collector/CollectorContracts.kt). +Every exhaustive `when` and the app build fail until the following participant-facing surfaces +agree: | File | What to add | | --- | --- | @@ -897,170 +875,57 @@ you handle it: | `app/.../MainActivity.kt` → `requestAccess` | `Unit` — hardware cannot be requested | | `app/.../CollectorDashboard.kt` → `AccessKind.displayName` | a participant-readable label | -This is the one part of collector registration the compiler enforces for you. Use it. - -### Step 5 — the collector +Required missing hardware blocks enrollment. Optional missing hardware reports blocked access and +starts only if hardware becomes available; it never substitutes another source. -`collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt`: +### Step 5 — catalog and generated contract -```kotlin -package cool.linc.androiddatacollector.collector.ambientlight - -import android.content.Context -import android.hardware.Sensor -import android.hardware.SensorEvent -import android.hardware.SensorEventListener -import android.hardware.SensorManager -import android.os.Handler -import android.os.HandlerThread -import cool.linc.androiddatacollector.core.collector.AccessKind -import cool.linc.androiddatacollector.core.collector.AccessRequirement -import cool.linc.androiddatacollector.core.collector.Collector -import cool.linc.androiddatacollector.core.collector.CollectorContext -import cool.linc.androiddatacollector.core.collector.CollectorDescriptor -import cool.linc.androiddatacollector.core.collector.CollectorPlugin -import cool.linc.androiddatacollector.core.collector.PrivacyClass -import cool.linc.androiddatacollector.core.collector.SerializedCallbackCollector -import cool.linc.androiddatacollector.core.definition.AmbientLightConfiguration -import cool.linc.androiddatacollector.core.definition.CollectorConfiguration -import cool.linc.androiddatacollector.core.model.EventDraft - -class AmbientLightCollectorPlugin( - context: Context, -) : CollectorPlugin { - private val applicationContext = context.applicationContext - - override val descriptor = CollectorDescriptor( - id = AmbientLightConfiguration.ID, - payloadSchemaVersion = 1, - displayName = "Ambient light", - privacyClass = PrivacyClass.SENSITIVE, - maximumEncodedEventBytes = 1_024, - ) +Add the configuration schema, payload contracts, units, clock bases, access/privacy, platform +availability, rate bound, and maximum encoded size to +[`protocol/v1/collector-catalog.json`](../protocol/v1/collector-catalog.json). With the module from +step 1 now present, mark the Android implementation `implemented`, then run: - override fun accessRequirements(configuration: CollectorConfiguration): Set { - val typed = configuration as? AmbientLightConfiguration - ?: throw IllegalArgumentException("Invalid ambient-light configuration") - return setOf(AccessRequirement(AccessKind.AMBIENT_LIGHT_HARDWARE, typed.required)) - } - - override fun create( - configuration: CollectorConfiguration, - context: CollectorContext, - ): Collector = AmbientLightCollector( - applicationContext, - configuration as? AmbientLightConfiguration - ?: throw IllegalArgumentException("Invalid ambient-light configuration"), - context, - ) -} - -private class AmbientLightCollector( - androidContext: Context, - private val configuration: AmbientLightConfiguration, - collectorContext: CollectorContext, -) : SerializedCallbackCollector(collectorContext, CHANNEL_CAPACITY), - SensorEventListener { - private val sensorManager = androidContext.getSystemService(SensorManager::class.java) - private val sensor by lazy { - sensorManager.getDefaultSensor(Sensor.TYPE_LIGHT) - ?: throw IllegalStateException("Ambient light hardware is unavailable") - } - private var handlerThread: HandlerThread? = null - - override fun onSensorChanged(event: SensorEvent) { - if (event.sensor.type != Sensor.TYPE_LIGHT || event.values.isEmpty()) return - capture { - EventDraft( - collectorId = AmbientLightConfiguration.ID, - payloadSchemaVersion = 1, - observedTime = context.clocks.now(), - payloadType = "AMBIENT_LIGHT_SAMPLE", - fields = mapOf( - "source_elapsed_realtime_nanos" to event.timestamp.toString(), - "illuminance_lux" to event.values[0].toString(), - "accuracy" to event.accuracy.toString(), - ), - ) - } - } - - override fun onAccuracyChanged( - sensor: Sensor?, - accuracy: Int, - ) = Unit - - override suspend fun registerSource() { - val thread = HandlerThread("adc-ambient-light").also { it.start() } - try { - check( - sensorManager.registerListener( - this, - sensor, - configuration.samplingPeriodUs, - Handler(thread.looper), - ), - ) { "Android rejected the ambient light listener" } - handlerThread = thread - } catch (failure: Throwable) { - thread.quitSafely() - throw failure - } - } - - override suspend fun unregisterSource() { - sensorManager.unregisterListener(this, sensor) - handlerThread?.quitSafely() - handlerThread = null - } - - private companion object { - const val CHANNEL_CAPACITY = 256 - } -} +```bash +python3 tools/catalog.py generate-kotlin +python3 tools/catalog.py check ``` -Points worth copying, in order of how often they are got wrong: +Never hand-edit the generated Kotlin projection. CI proves that it and the catalog agree. -- The `HandlerThread` is started before registration and quit on the failure path, so a - rejected registration does not leak a thread. -- `registerSource()` throws instead of setting health; the base class turns that into - `SOURCE_REGISTRATION_FAILED` and the runtime into `COLLECTOR_START_FAILED`. -- Units are in the field name (`illuminance_lux`). An analyst reading the export should not - have to consult this document to know what a number means. -- Android's own timestamp is preserved alongside `observedTime`. -- No filtering, no smoothing, no derived "is the participant indoors" field. +### Step 6 — the collector -### Step 6 — register in the app +Use the production [ambient-light collector](../collector/ambient-light/src/main/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollector.kt) +as the compact reference and the shared [sensor lifecycle owner](../collector/sensor-common/src/main/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/AndroidSensorCollector.kt) +for listener-thread ownership. Keeping the example as links instead of a copied implementation +prevents this guide from becoming a second, stale collector. -`app/build.gradle.kts`: +The boundaries worth preserving are: -```kotlin -implementation(project(":collector:ambient-light")) -``` +- `AndroidSensorCollector` owns registration rollback, handler callback removal, and thread release; + collector-specific teardown only clears its pending data. +- A changed on-change reading inside the minimum interval replaces the pending reading rather than + disappearing. When emitted, it keeps the original `observedTime` and hardware timestamp. +- The lux threshold alone decides equivalence; accuracy describes an emitted sample and does not + independently trigger one. +- Units remain in field names and the catalog. There is no smoothing or derived indoor/presence + inference. +- The focused [collector test](../collector/ambient-light/src/test/kotlin/cool/linc/androiddatacollector/collector/ambientlight/AmbientLightCollectorTest.kt) + proves coalescing and capture-time behavior; the shared [lifecycle test](../collector/sensor-common/src/test/kotlin/cool/linc/androiddatacollector/collector/sensorcommon/SensorSourceLifecycleTest.kt) + proves failure cleanup. -`app/src/main/kotlin/cool/linc/androiddatacollector/CollectorApplication.kt`: - -```kotlin -val registry = CollectorRegistry( - listOf( - AppLifecycleCollectorPlugin(this), - AccelerometerCollectorPlugin(this), - AmbientLightCollectorPlugin(this), - // … - ), -) -``` +### Step 7 — register in the app -This list is the whole allowlist. A study configuration can name only IDs that appear here, -and `CollectorRegistry` throws for anything else. Adding a collector to the codec without -adding it here produces a configuration that verifies and then fails to run — which is the -correct failure direction, but check both. +[`app/build.gradle.kts`](../app/build.gradle.kts) takes the module, and +[`CollectorApplication.kt`](../app/src/main/kotlin/cool/linc/androiddatacollector/CollectorApplication.kt) +constructs `AmbientLightCollectorPlugin` in the compiled allowlist. `CollectorRegistry` rejects an +unknown configured ID; the catalog is not runtime plugin loading. The Web control, codec, app +allowlist, and catalog parity checks must all land together. -### Step 7 — tests, example, and disclosure +### Step 8 — tests, target-device exercise, and disclosure - Configuration tests in `core/study-definition/src/test/...` covering nominal values, both - range boundaries, an unknown key, a wrong JSON type, and canonical round-trip. Follow + range boundaries, values outside both bounds, unknown/missing keys, a wrong JSON type, and + canonical round-trip. Follow [`NetworkUsageConfigurationTest`](../core/study-definition/src/test/kotlin/cool/linc/androiddatacollector/core/definition/NetworkUsageConfigurationTest.kt). - Collector tests using the fake sink pattern in [`SerializedCallbackCollectorTest`](../core/collector-api/src/test/kotlin/cool/linc/androiddatacollector/core/collector/SerializedCallbackCollectorTest.kt): @@ -1072,6 +937,9 @@ correct failure direction, but check both. release variant ships no demonstration study. - Update the participant guide, the researcher guide's capability table, and this document. A collector whose data is not described to participants must not ship. +- Add lifecycle/access-revocation and rate/size-bound tests, document power and storage estimates, + and exercise the collector on the target device classes. Run + `python3 tools/collector_assurance.py` after compiling. ## 12. Definition of done @@ -1087,30 +955,21 @@ correct failure direction, but check both. reaches logcat. - [ ] Every payload field's unit, clock, precision, platform limitation, and sensitivity is documented. -- [ ] The release manifest gained no permission and no component beyond what the collector - genuinely needs. Check the merged manifest, not only your module's. +- [ ] The catalog validates, generated Kotlin is current, schema-invalid and over-size events are + rejected before append, and deterministic decoder fixtures exist. +- [ ] Collector capability checks pass for source, compiled bytecode, and dependencies. - [ ] `./gradlew test testDebugUnitTest lintDebug assembleDebug assembleRelease` passes — the same command CI runs. -- [ ] The collector has been run on a physical device, not only an emulator. Sensor - batching, doze, and IME selection behave differently there. +- [ ] Disclosure plus power/storage estimates are complete, and relevant target-device behavior + has been exercised before deployment. ## 13. Known gaps Stated here rather than discovered later. -- **No architecture test enforces the module boundary.** The rules in - [section 3](#3-invariants-you-must-not-break) are enforced by the Gradle dependency graph - and by review. A collector that wrote a file or started an `Activity` would compile and - pass CI. -- **`maximumEncodedEventBytes` is declared but unread.** No code compares an encoded event - against it. The enforced limits are `EventDraft`'s 32 fields and 1,024 characters per value. +- **Static policy is not process isolation.** Source, bytecode, and dependency checks catch the + prohibited capabilities they name, but collectors still execute in the app process. Policy + review remains a security decision whenever Android APIs or build tooling change. - **`privacyClass` is declared but unread.** It documents the author's classification and drives no behaviour. - **`displayName` is not shown to participants.** The dashboard lists collectors by ID. -- **A collector module can widen the app's permissions.** Manifest merging means a - `` in a collector module lands in the app manifest, and nothing in the - build fails when it does. Reviewing the merged manifest is the only check. Note that - `INTERNET` is now declared by the app itself for the upload worker, so a collector that - used the network would leave no trace in the permission set at all. -- **No collector module has its own tests.** Coverage for collector behaviour currently comes - from `SerializedCallbackCollectorTest` and `ExperimentRuntimeTest` in the core modules. diff --git a/docs/data-dictionary.md b/docs/data-dictionary.md index ac74a42..73e6542 100644 --- a/docs/data-dictionary.md +++ b/docs/data-dictionary.md @@ -2,7 +2,7 @@ Every field that can appear in an exported dataset, per collector. This document is written to be quotable in an ethics submission: it describes what the code in this repository actually emits, including the gaps. -Read the "what you cannot claim" column in the [researcher guide](researcher-guide.md) alongside this. This document says what a field *is*; that one says what it does not prove. +Read the "what you cannot claim" column in the [researcher guide](researcher-guide.md) alongside this. This document says what a field *is*; that one says what it does not prove. The machine-readable source of truth is the [Protocol v1 collector catalog](../protocol/v1/collector-catalog.json); [Protocol v1](../protocol/v1/README.md) defines the enclosing document and validation order. ## Reading an export @@ -10,55 +10,65 @@ A decrypted bundle is a `research-bundle-v1` JSON document. ```json { - "format": "research-bundle-v1", - "exported_at_utc_millis": 1767225600000, - "configuration": { }, + "bundle_id": "0a1b2c3d-4e5f-4071-8293-a4b5c6d7e8f9", + "bundle_kind": "automatic_upload", + "configuration": {}, + "configuration_sha256": "<64 lowercase hex characters>", + "configuration_signature": {"signature": "", "signer_key_id": "lab-signer-2026"}, "experiment": { - "experiment_id": "...", - "configuration_id": "...", - "participant_instance_id": "0a1b2c3d-4e5f-6071-8293-a4b5c6d7e8f9", "assigned_participant_id": "cohortA-0042", + "configuration_id": "config-2026", + "durable_through_sequence": "4210", + "event_count": "10", + "events": [], + "experiment_id": "study-2026", + "first_sequence_number": "4201", + "last_sequence_number": "4210", + "next_sequence_number": "4211", + "participant_instance_id": "1a1b2c3d-4e5f-4071-8293-a4b5c6d7e8f9", + "retained_from_sequence": "1", "state": "RUNNING", - "next_sequence_number": 4211, - "transitions": [ { "from": "READY", "to": "RUNNING", "reason": "...", "time": { } } ], - "events": [ ], - "first_sequence_number": 1, - "last_sequence_number": 4210 - } + "transitions": [], + "uploaded_through_sequence": "4200" + }, + "exported_at_utc_millis": "1767225600000", + "format": "research-bundle-v1", + "producer": {"client_version": "1", "platform": "android"} } ``` -The window fields come after `events`, and that placement is deliberate. A bundle is written as -a stream, and an uploaded one stops at the first event boundary past a plaintext budget, so the -last sequence it contains is not known until the events have been written. Declaring the window -before them would let a bundle claim a range it does not contain. JSON object member order -carries no meaning, so a parser that reads by key is unaffected, and decryption is unaffected -too. Code that consumes a bundle as a token stream must still follow the v1 order shown here. +The example is expanded for readability. The authenticated bytes are RFC 8785 JCS, so member +order and number spelling are canonical. An automatic upload's exact event boundary is selected +before its complete ciphertext and manifest are durably staged; retries never regenerate it. `configuration` is the canonical study configuration the participant consented to, reproduced verbatim. Every dataset therefore carries its own definition of what was supposed to be collected — including its `upload` block, so a dataset states whether the study it came from delivered data to an endpoint. -It also includes the `signer` block, so provenance travels with the data: `configuration.signer.key_id` and `configuration.signer.public_key` name the key the configuration was signed with, and the same key fingerprint the participant saw on the consent screen can be recomputed from the public key at analysis time. That identifies which signing key issued the study a dataset came from — useful when a lab runs several studies or rotates keys — and does not by itself attest to who held that key. +`configuration_signature` preserves the original signer key ID and raw Ed25519 signature, while `configuration_sha256` binds the exact embedded configuration to the outer `ADCEXP01` framing. The signature and digest are verified again during decryption; they identify which key issued the artifact but do not attest who held that key or which device submitted the bundle. `producer` records the producing platform and client build. | Field | Meaning | | --- | --- | -| `participant_instance_id` | A random UUID generated on the device for each import. Importing the same signed configuration again creates a different ID and independent sequence space. It is pseudonymous: no name, account, device identifier, or advertising ID. It is disclosed on the consent screen and exposed in upload routing, so treat it as personal data. | +| `participant_instance_id` | A random UUID generated on the device for each import. Importing the same signed configuration again creates a different ID and independent sequence space. It is pseudonymous: no name, account, device identifier, or advertising ID. It is absent from upload URLs and headers, but remains personal data after decryption. | | `assigned_participant_id` | Optional researcher-assigned opaque code copied from the signed configuration. It exists only for a personalized study. It is stored in encrypted metadata and appears in the encrypted export, but is deliberately absent from clear upload headers. It can link the dataset to a research roster and must be governed as personal data. | -| `next_sequence_number` | The device's counter at the moment the bundle was written: one past the last event durably stored, across the whole study rather than this bundle. | -| `first_sequence_number` | The first event sequence this bundle contains, inclusive. | -| `last_sequence_number` | The last event sequence this bundle contains, inclusive. | +| `next_sequence_number` | The device's counter at the snapshot: one past the last event durably stored. Decimal string. | +| `retained_from_sequence` | Lowest sequence still retained locally after confirmed-prefix reclamation. Decimal string. | +| `durable_through_sequence` | Highest event durably stored at the snapshot. Decimal string. | +| `uploaded_through_sequence` | Highest sequence committed after an exact upload receipt. Decimal string. | +| `event_count` | Number of events in this bundle; must agree with the inclusive range. Decimal string. | +| `first_sequence_number` | First event sequence this bundle contains, inclusive. Decimal string. | +| `last_sequence_number` | Last event sequence this bundle contains, inclusive. Decimal string. | ### Whole exports and uploaded chunks The two ways data leaves the device produce the same document. What differs is the window. -- A **manual export** runs to whatever was durable when the participant pressed export. It starts at `first_sequence_number: 1` unless the device has reclaimed a delivered prefix to free space, in which case it starts at the lowest sequence still on the phone. Successive exports from one participant therefore overlap, each containing everything the previous one did that has not since been reclaimed. -- An **uploaded chunk**, from a study whose configuration names an endpoint, starts after the last sequence that endpoint confirmed. There is no configured chunk size: each delivery asks for everything outstanding and stops at the first event boundary past a 16 MiB plaintext budget, so chunk sizes vary with event size and with how much backlog was waiting. Consecutive chunks abut rather than overlap, and reassembling a participant's data means concatenating them in sequence order. +- A **manual export** runs to whatever was durable when the participant pressed export. It starts at `first_sequence_number: "1"` unless the device has reclaimed a delivered prefix to free space, in which case it starts at the lowest sequence still on the phone. Successive exports from one participant therefore overlap, each containing everything the previous one did that has not since been reclaimed. +- An **uploaded chunk**, from a study whose configuration names an endpoint, starts after the last sequence an exact receipt confirmed. The app selects a boundary near a 16 MiB plaintext target, durably stages one immutable ciphertext bundle, and enforces a 32 MiB wire ceiling. Consecutive chunks normally abut; exact replays have the same bundle UUID and bytes. Read `first_sequence_number` and `last_sequence_number` on every bundle rather than assuming a starting point or deriving a boundary from the study's configuration. In an uploading study the complete dataset for a participant is the chunks plus the final export, joined on sequence number. -A bundle has no size ceiling of its own; it is bounded by the study's `storage.maximum_local_bytes`, which is why `researcher-tools decrypt` streams rather than decrypting in memory. +A manual bundle is bounded by `storage.maximum_local_bytes`, which is why `researcher-tools decrypt` streams rather than decrypting in memory. The 32 MiB ceiling applies to automatic upload bodies. -`state`, `transitions`, and `configuration` describe the study as a whole in both cases, not just the window, so the same transition history repeats in every chunk. Only `events`, `first_sequence_number`, and `last_sequence_number` are window-scoped. +`state`, `transitions`, and `configuration` describe the study as a whole in both cases, not just the window, so the same transition history repeats in every chunk. `events`, `event_count`, `first_sequence_number`, and `last_sequence_number` are window-scoped. `format` is bound into the bundle's cryptographic associated data, so a reader built for a different version fails to decrypt rather than silently misreading one: the authentication tag fails before any field is parsed. @@ -68,12 +78,12 @@ Every event has the same shape regardless of collector. ```json { - "sequence_number": 1, + "sequence_number": "1", "collector_id": "accelerometer.v1", "payload_schema_version": 1, "observed_time": { - "wall_time_utc_millis": 1767225600000, - "elapsed_realtime_nanos": 12345678901234, + "wall_time_utc_millis": "1767225600000", + "monotonic_time_nanos": "12345678901234", "boot_session_id": "0a1b2c3d4e5f60718293a4b5c6d7e8f9" }, "payload_type": "ACCELEROMETER_SAMPLE", @@ -83,7 +93,7 @@ Every event has the same shape regardless of collector. | Envelope field | JSON type | Meaning | | --- | --- | --- | -| `sequence_number` | number | Monotonic, starts at 1, **shared across all collectors in a study**. Not per-collector. | +| `sequence_number` | decimal string | Monotonic, starts at 1, **shared across all collectors in a study**. Not per-collector. | | `collector_id` | string | Which collector produced this event | | `payload_schema_version` | number | Version of the `fields` schema for this collector | | `observed_time` | object | See below | @@ -94,9 +104,9 @@ Every event has the same shape regardless of collector. This is the most important thing to know before writing a parser. -`fields` is a string-to-string map. Numbers and booleans are stringified: acceleration appears as `"9.81"`, not `9.81`, and flags appear as `"true"` / `"false"`, not `true` / `false`. Only the envelope's `sequence_number`, `payload_schema_version`, and the three `observed_time` values are real JSON numbers. +`fields` is a string-to-string map. Numbers and booleans are stringified: acceleration appears as `"9.81"`, not `9.81`, and flags appear as `"true"` / `"false"`, not `true` / `false`. `payload_schema_version` is a bounded JSON number; sequence and time values are canonical decimal strings. -Field keys match `[a-z][a-z0-9_]{0,63}` and an event has at most 32 fields. A field value is capped at 60 KiB of characters; storage independently caps the complete encoded event at 64 KiB. Ordinary collectors emit much smaller scalar values. The larger bound exists so one survey submission can be committed as a single immutable value. +Field keys match `[a-z][a-z0-9_]{0,63}` and an event has at most 32 fields. A field value is capped at 60 Ki UTF-16 code units; storage independently caps the complete protocol-encoded event at 64 KiB. Ordinary collectors emit much smaller scalar values. The larger bound exists so one survey submission can be committed as a single immutable value. ### Time @@ -105,21 +115,25 @@ Three clocks are recorded on every event, because no single one is sufficient. | Field | Source | Unit | Caveat | | --- | --- | --- | --- | | `wall_time_utc_millis` | `System.currentTimeMillis()` | ms since Unix epoch, UTC | Can jump forwards or backwards — NTP corrections, manual changes, timezone travel. Do not assume monotonicity. | -| `elapsed_realtime_nanos` | `SystemClock.elapsedRealtimeNanos()` | ns since boot | Monotonic and includes deep sleep, but only comparable **within the same `boot_session_id`** | +| `monotonic_time_nanos` | `SystemClock.elapsedRealtimeNanos()` on Android | ns on a continuous monotonic clock | Includes deep sleep on Android and is only comparable **within the same `boot_session_id`** | | `boot_session_id` | derived | 32 hex characters | Changes on every reboot. A change means the two elapsed-realtime values either side are incomparable. | -**No timezone or UTC offset is recorded anywhere.** If local time matters to your analysis, you have to obtain it another way; it cannot be recovered from an export. +The common event envelope does not carry a time zone or UTC offset. A study that explicitly enables +`temporal_context.v1` receives the bounded snapshots documented below; otherwise local time cannot +be reconstructed from an export. `observed_time` is stamped inside the collector callback at capture time, with two exceptions: `network_usage.v1` and `usage_events.v1` are polling collectors and stamp one `observed_time` per poll, shared by every event in that batch. For those two, use the in-payload source time instead. Several collectors also carry a source-supplied time in their payload. **Do not subtract across clock bases:** -- `elapsed_realtime_nanos` and the `source_elapsed_realtime_nanos` fields (accelerometer, location) use the elapsed-realtime base, which **includes** deep sleep. +- `monotonic_time_nanos` and `source_elapsed_realtime_nanos` fields (accelerometer, gyroscope, + ambient light, proximity, and location) use Android's elapsed-realtime base, which **includes** + deep sleep. - The keyboard's `event_uptime_millis` and `down_uptime_millis` use Android's uptime base, which **excludes** deep sleep. ### Deduplication -Exports overlap by design — a participant can export repeatedly, and each export contains everything from its retained floor up to its boundary. Uploaded chunks do not overlap each other, and a manual export overlaps every chunk after that floor. Deduplicate on `participant_instance_id` + `sequence_number`. Do not merge solely on an assigned ID: two imports for one assigned participant intentionally have different instance IDs and independent sequence spaces. Sequence numbers are never reissued within one instance. +Exports overlap by design — a participant can export repeatedly, and each export contains everything from its retained floor up to its boundary. Partition by `(experiment_id, configuration_id)` and deduplicate on `(participant_instance_id, sequence_number)`, making the complete event identity all four values. Identical repeats are duplicates; different content at one identity is a conflict, never a last-write-wins update. Do not merge solely on an assigned ID. Receiver ingestion instead deduplicates the immutable bundle UUID and exact bytes/metadata; a participant/range pair is not an ingest key. ### Intervention and survey events (`interventions.v1`) @@ -142,7 +156,7 @@ For compliance metrics, start with the lifecycle event that actually supports th ### Gaps are real and are not errors -- Nothing is recorded while a study is `PAUSED`. The polling collectors do not back-fill the paused interval on resume; that data is deliberately never collected. +- Nothing is recorded while a study is `PAUSED`, including intervention lifecycle or survey-submission events. Prompt work and visible intervention notifications are removed; calendar time and availability continue, and durable occurrences are reconciled on resume. Polling collectors do not back-fill the paused interval; that data is deliberately never collected. - Across a process restart, `network_usage.v1` and `usage_events.v1` do resume their query window from the last stored event, so a restart is not the same as a pause. - A collector that loses access reports `BLOCKED_ACCESS` and stops. It never emits a placeholder or interpolated value. @@ -178,7 +192,7 @@ Raw accelerometer samples, **including gravity**. No filtering, orientation esti | Sampling period | `sampling_period_us` | int | microseconds | 5,000–1,000,000 (200 Hz–1 Hz) | | Maximum report latency | `maximum_report_latency_us` | int | microseconds | 0–60,000,000 (batching window) | -The sampling period is a **hint to Android**, not a guarantee. Actual delivery rate varies by device, by sensor, and with system power state. Measure the achieved rate from the data rather than assuming the configured one. +The sampling period is a **hint to Android**, not a guarantee. Actual delivery rate varies by device, by sensor, and with system power state. Measure the achieved rate from `source_elapsed_realtime_nanos`, not the event-envelope callback time: FIFO batching can deliver many earlier hardware samples in one callback burst. **Access:** accelerometer hardware must be present. This is a capability check, not an Android permission — there is no dialog and nothing for the participant to grant. A device without the sensor cannot run a study that requires this collector. @@ -196,6 +210,133 @@ Not recorded: gyroscope, magnetometer, or any other sensor; derived orientation, --- +## `battery_state.v1` + +Event-driven battery context with exact duplicate suppression and a one-minute emission bound. If +several changes arrive inside the bound, the newest distinct state is retained. + +**Configuration:** `{}`. **Access:** none. + +**Payload type:** `BATTERY_STATE`. + +| Field | Type | Unit | Meaning | +| --- | --- | --- | --- | +| `percentage` | int | whole percent | `(level × 100) / scale`, bounded to 0–100 | +| `charging_state` | enum | — | `CHARGING`, `DISCHARGING`, `FULL`, `NOT_CHARGING`, or `UNKNOWN` | +| `charging_source` | enum | — | `AC`, `USB`, `WIRELESS`, `DOCK`, `MULTIPLE`, `NONE`, or `UNKNOWN` | +| `power_save_enabled` | boolean-as-string | — | Current Android power-save mode | + +Not recorded: battery serial or hardware ID, capacity, health, voltage, current, temperature, or +the cause of a battery change. Percentage is an integer platform reading, not a calibrated energy +measurement. + +--- + +## `temporal_context.v1` + +A snapshot at study start/reconciliation and when Android reports a time, time-zone, or UTC-offset +change. Exact duplicates are suppressed and rapid changes retain the newest event under a +one-minute bound. + +**Configuration:** `{}`. **Access:** none. + +**Payload type:** `TEMPORAL_CONTEXT`. + +| Field | Type | Unit | Meaning | +| --- | --- | --- | --- | +| `change_reason` | enum | — | `STUDY_STARTED`, `RECONCILED`, `TIMEZONE_CHANGED`, `TIME_SET`, or `UTC_OFFSET_CHANGED` | +| `timezone_id` | string | IANA/Android zone ID | Current `ZoneId.systemDefault()` setting | +| `utc_offset_seconds` | int | seconds | Zone-rule offset at observation, −64,800 to 64,800 | +| `daylight_saving_time` | boolean-as-string | — | Whether that zone's rules are in DST at observation | + +A time zone is a device setting, not proof of physical location or travel. `TIME_SET` proves that +Android announced a wall-clock change; it does not identify who or what changed it. + +--- + +## `gyroscope.v1` + +Raw angular velocity in device coordinates. No filtering, orientation estimation, or activity +inference is applied. + +**Configuration:** + +| Parameter | JSON key | Type | Unit | Range | +| --- | --- | --- | --- | --- | +| Sampling period | `sampling_period_us` | int | microseconds | 5,000–1,000,000 (200 Hz–1 Hz) | +| Maximum report latency | `maximum_report_latency_us` | int | microseconds | 0–60,000,000 | + +**Access:** gyroscope hardware. **Payload type:** `GYROSCOPE_SAMPLE`. + +| Field | Type | Unit | Meaning | +| --- | --- | --- | --- | +| `source_elapsed_realtime_nanos` | long | ns since boot | Hardware `SensorEvent.timestamp` | +| `x_radians_per_second` | float | rad/s | Raw angular velocity around device X | +| `y_radians_per_second` | float | rad/s | Raw angular velocity around device Y | +| `z_radians_per_second` | float | rad/s | Raw angular velocity around device Z | +| `accuracy` | int | enum ordinal | Raw `SensorEvent.accuracy` | + +The requested period is a hint; use source timestamps to measure achieved rate. Not recorded: +orientation, posture, gesture, or activity labels. + +--- + +## `ambient_light.v1` + +Raw ambient illuminance after a monotonic rate gate and change threshold. Because Android light +sensors are commonly on-change sources, the newest threshold-sized change inside the minimum +interval is retained and emitted when the interval opens; its original observation time and +`source_elapsed_realtime_nanos` are preserved. A later reading equivalent to the last emitted lux +value cancels that pending change. Accuracy is descriptive metadata on emitted lux samples and is +not an independent emission trigger. + +**Configuration:** + +| Parameter | JSON key | Type | Unit | Range | +| --- | --- | --- | --- | --- | +| Minimum sample period | `sampling_period_us` | int | microseconds | 200,000–10,000,000 | +| Change threshold | `change_threshold_millilux` | int | millilux | 0–100,000,000 | + +**Access:** ambient-light hardware. **Payload type:** `AMBIENT_LIGHT_SAMPLE`. + +| Field | Type | Unit | Meaning | +| --- | --- | --- | --- | +| `source_elapsed_realtime_nanos` | long | ns since boot | Hardware `SensorEvent.timestamp` | +| `illuminance_lux` | float | lux | Non-negative raw sensor reading | +| `accuracy` | int | enum ordinal | Raw `SensorEvent.accuracy` | + +Not recorded: images, colour, environmental content, or presence. Lux accuracy and calibration vary +by device; the collector does not normalize across hardware. + +--- + +## `proximity.v1` + +Raw proximity readings with a monotonic minimum interval. The newest meaningful reading inside the +interval is retained; exact duplicates and same-state changes below the configured threshold are +suppressed. + +**Configuration:** + +| Parameter | JSON key | Type | Unit | Range | +| --- | --- | --- | --- | --- | +| Minimum event interval | `minimum_event_interval_ms` | int | milliseconds | 100–60,000 | +| Change threshold | `change_threshold_millimeters` | int | millimetres | 0–10,000 | + +**Access:** proximity hardware. **Payload type:** `PROXIMITY_SAMPLE`. + +| Field | Type | Unit | Meaning | +| --- | --- | --- | --- | +| `source_elapsed_realtime_nanos` | long | ns since boot | Hardware `SensorEvent.timestamp` | +| `distance_centimeters` | float | centimetres | Non-negative raw Android distance | +| `maximum_range_centimeters` | float | centimetres | This sensor's declared maximum range | +| `near` | boolean-as-string | — | `distance_centimeters < maximum_range_centimeters` | + +Many proximity sensors expose only near and maximum range. Values are not assumed precise or +comparable across devices, and neither `near` nor distance proves a person's presence. + +--- + ## `network_state.v1` Capabilities of the system default network. Connection shape only. @@ -311,7 +452,7 @@ Fused Location fixes via Google Play Services. | Interval | `interval_millis` | long | ms | 1,000–3,600,000 | | Minimum interval | `minimum_interval_millis` | long | ms | 500 up to the configured interval | | Maximum batch delay | `maximum_batch_delay_millis` | long | ms | 0–86,400,000 | -| Minimum displacement | `minimum_displacement_meters` | float | metres | 0–10,000 | +| Minimum displacement | `minimum_displacement_millimeters` | int | millimetres | 0–10,000,000 | | Priority | `priority` | `"BALANCED"` or `"HIGH_ACCURACY"` | — | — | This collector always requires precise location. There is no coarse-only mode, and `priority` selects a power/accuracy trade-off within fine location rather than reducing the permission it needs. Say so in your consent text. @@ -396,6 +537,10 @@ In a study that uploads, a confirmed delivery lets the device reclaim space. Abo Study metadata is held separately from the events. It is capped at 1 MiB and kept outside the event budget by a 2 MiB reserve, so the record of what a study is and how far it has been delivered cannot be crowded out by the events it describes. Its container header is `ADCMET01`. -Opening a study does not decrypt its event log. Framing and sequence contiguity are checked from the plaintext frame headers, and each collector's most recent event is persisted in that metadata rather than recovered by scanning, so start-up cost is linear in frames rather than in bytes decrypted. Event payloads are therefore authenticated when they are read rather than when a study is opened, which means a damaged payload surfaces at export or upload time rather than at launch. +Normal study opening does not decrypt its event log. Framing and sequence contiguity are checked from plaintext frame headers, and each collector's most recent event is persisted in metadata rather than recovered by scanning, so start-up cost is linear in frames rather than in bytes decrypted. The sole exception is a one-boundary-ahead append journal with a durable tail: recovery authenticates exactly that tail before applying its metadata. All other event payloads are authenticated when read, so damage outside the recovery tail surfaces at export or upload rather than at launch. -Per-collector byte ceilings appear in each collector's descriptor, but they are declarative and not enforced at runtime. The 64 KiB global limit is the one that applies. +Each collector descriptor declares `maximumEncodedEventBytes`. The runtime encodes every admitted +event with the worst-case sequence width and rejects it before append when it exceeds that +collector-specific ceiling; the store's 64 KiB global cap remains a second boundary. CI also +checks collector source, compiled constants, and module dependencies as documented in the +[Collector capability policy](../assurance/README.md). diff --git a/docs/maintainers/release.md b/docs/maintainers/release.md index 2bdf399..9c602bc 100644 --- a/docs/maintainers/release.md +++ b/docs/maintainers/release.md @@ -6,7 +6,9 @@ For maintainers of this repository. Participants and researchers do not need thi Both workflows live in [`.github/workflows`](../../.github/workflows). -**`Android CI`** (`ci.yml`) runs on pushes to `main`, on pull requests, and on manual dispatch. It runs unit tests, Android lint, and debug and release builds, and retains the debug APK for 14 days. +**`Android CI`** (`ci.yml`) runs on pushes to `main`, on pull requests, and on manual dispatch. It +runs unit tests, Android lint, Protocol/catalog conformance, Collector capability checks, and debug +and release builds. Successful runs retain the debug APK as an artifact for 14 days. **`Android Release`** (`release.yml`) accepts only `v` tags that are reachable from `main` — for example `v0.1.0`. A tag with a prerelease suffix produces a GitHub prerelease. diff --git a/docs/p0-p2-implementation-contract.md b/docs/p0-p2-implementation-contract.md new file mode 100644 index 0000000..0611c7c --- /dev/null +++ b/docs/p0-p2-implementation-contract.md @@ -0,0 +1,145 @@ +# P0–P2 implementation contract + +This document is the implementation contract for roadmap issues #7–#19. It records the +cross-module decisions that must stay stable while the work is delivered in phases. The issue +bodies remain the product requirements; this document says where each responsibility belongs and +which shortcuts are forbidden. + +## Scope + +This delivery includes: + +- P0: Protocol v1 finalization, a replay-safe Android upload outbox, the collector/event catalog, + and the Collector capability policy; +- P1: the R2-only ciphertext receiver, offline Python verification/reassembly, typed Parquet, and + immutable join links; +- P2: battery state, temporal context, gyroscope, ambient light, proximity, and randomized local + EMA windows. + +It excludes iOS/P3, remote configuration, remote triggers, participant/device attestation, +receiver administration, receiver-side decryption, databases other than R2 ciphertext storage, +and analysis sinks other than Parquet. Existing issue #5 is the sole mutable-hosted-content +exception; it does not permit changing an accepted app configuration. + +## Non-negotiable invariants + +1. Protocol names remain `schema_version: 1`, `ADCCFG01`, `ADCEXP01`, and + `research-bundle-v1`. This is a destructive pre-1.0 replacement. No old-v1 reader, migration, + dual parser, compatibility flag, Tink wire keyset, or fallback is retained. +2. Protocol input is closed-world and fail-closed. Unknown members, malformed UTF-8, duplicate + JSON members, noncanonical JSON, invalid key encodings, unsupported collectors, wrong platform, + wrong cryptographic context, malformed framing, and trailing bytes are rejected. +3. Android and future iOS configurations may share an experiment ID but never a configuration ID + or signature. This build accepts only an Android-targeted configuration. +4. Upload bytes are fully encrypted and durably staged before HTTP begins. One bundle ID always + denotes one immutable byte string. The watermark advances only after an exact matching durable + receiver receipt. +5. Receiver routing metadata is untrusted. Receiver success proves only that bounded ciphertext + bytes were stored in R2; it does not prove participant, device, configuration, or plaintext + authenticity. +6. Offline analysis verifies the entire bundle before publishing plaintext-derived records. An + invalid bundle produces no partial rows. Event identity is + `(experiment_id, configuration_id, participant_instance_id, sequence_number)` only after + authenticated decryption. +7. Collector implementations remain compiled, closed-world modules. The catalog describes their + contracts; it does not load code or turn unknown payloads into a generic runtime plugin. + +## Responsibility map + +| Concern | Authoritative location | Implementations/consumers | +| --- | --- | --- | +| Normative Protocol v1, catalog, vectors | `protocol/v1/` | Kotlin, TypeScript, Python | +| Typed study configuration | `:core:study-definition` | Android, researcher tooling | +| Signed configuration envelope/trust | `:core:protocol` | Android, researcher tooling | +| Raw-key Ed25519 verification and RFC 9180 HPKE primitives | `:core:crypto` | `:core:protocol`, `:core:export`, researcher tooling | +| Encrypted bundle framing/document | `:core:export` | Android export/outbox, Python analysis | +| Upload planning/watermark | `:core:study-application` | Android session/runtime | +| Durable body staging and HTTP | `:app` platform adapters | WorkManager upload worker | +| Ciphertext ingress | `receiver/` | Cloudflare Worker and R2 only | +| Offline validation and datasets | `adc-analysis/` | Local/R2 source, Parquet sink | +| Collector implementations | `:collector:*` | Android composition root | +| Collector capability policy | `assurance/collector-policy.json` | CI | + +No generic service/repository/controller hierarchy is introduced across these concerns. + +## Protocol v1 decisions + +- Signed configuration JSON uses RFC 8785 JCS. The schema permits only bounded integral JSON + numbers; semantic 64-bit counters, times, monotonic values, and byte counts use canonical decimal + strings. +- Ed25519 and X25519 keys are raw 32-byte values encoded as unpadded base64url. Private CLI key + files use the same raw encoding. +- Floating-point configuration is removed. Location displacement becomes integer millimeters. +- The configuration carries an explicit platform target and decimal minimum client build number. +- `ADCCFG01` has fixed Ed25519 signature length and no legacy signature-length field. +- `ADCEXP01` carries a UUID bundle ID, complete configuration SHA-256, fixed-suite RFC 9180 + X25519/HKDF-SHA-256/AES-256-GCM wrapped content key, and an AES-256-GCM encrypted document. +- HPKE context and content AAD bind the bundle format, bundle ID, configuration digest, and + researcher key ID. The authenticated document repeats and verifies those identities. +- The document includes canonical configuration bytes, configuration signature provenance, + producer platform/client version, bundle kind, retained/uploaded/durable boundaries, actual + range/count, and decimal-string research clocks/sequences. + +## Android upload transaction + +The transaction order is: + +1. Recover an existing valid staged bundle or create exactly one in no-backup storage. +2. Send that fixed-length file with its exact digest and metadata; redirects and implicit request + replay are disabled. +3. Accept only a matching `201 Created` or exact-replay `200 OK` receipt. +4. Persist the watermark through the staged last sequence. +5. Remove the outbox manifest, then its now-harmless body orphan. +6. Reclaim eligible encrypted event segments under the existing quota policy. + +A crash before step 4 resends identical bytes. A crash after step 4 clears the already-covered +stage before creating another. I/O, 408, 425, 429, and 5xx are retryable; all other HTTP/protocol +failures are terminal for delivery but never stop collection. + +## Receiver and analysis boundaries + +The Worker exposes only the Protocol v1 upload POST and performs bounded streaming into an +immutable R2 object. It has no private key, decrypt path, list/download/delete/admin route, D1, +Queue, KV, Durable Object, dashboard, or runtime configuration. + +`adc-analysis` uses one directional pipeline: + +```text +BundleSource -> immutable ciphertext inventory -> full validation -> deterministic reassembly + -> typed Parquet sink +``` + +Local and R2 sources first copy ciphertext into a content-addressed cache. Plaintext is staged with +mode 0600, and validated intermediate data or a complete dataset is published by atomic rename. +Conflicting duplicates remain explicit conflicts; there is no last-write-wins or unknown-schema +fallback. + +## P2 design limits + +- Each new collector is a separate small Gradle module using the existing collector API and event + sink. Gyroscope, ambient light, and proximity share only the narrow + `collector:sensor-common` listener-thread lifecycle helper; configuration, payload mapping, + rate/change policy, and disclosure stay in their own modules. There is no runtime plugin or + generic sensor-schema framework. +- Battery and temporal context use runtime-registered, non-exported receivers. Gyroscope, ambient + light, and proximity use hardware preflight without new Android permissions or components. +- Every new numeric sensor value must be finite; catalog maximum event size and field schema are + enforced by tests and offline validation. +- Randomized EMA reuses durable `InterventionOccurrence` records. A CSPRNG-selected instant is + committed before WorkManager is enqueued; committed random occurrences are never rescheduled. + Only future, unmaterialized local-date windows follow a later time-zone change. Daily and total + caps consume eligible slots in local-date planning order, then signed window array order, then + ordinal; the CSPRNG selects the minute within the chosen slot, not which window survives + truncation. + +## Phase review criteria + +Each phase is complete only after: + +1. targeted and full relevant test/build checks pass; +2. a correctness/security review checks failure paths and stated invariants; +3. a simplicity review removes unnecessary abstractions, duplicate implementations, legacy paths, + unused code, and avoidable dependencies; +4. scout-rule cleanup leaves touched code clearer than before; and +5. an independent reviewer unfamiliar with the implementation can locate the specification, + source, tests, and operational documentation from the root documentation. diff --git a/docs/participant-guide.md b/docs/participant-guide.md index 93d8f90..788c7f4 100644 --- a/docs/participant-guide.md +++ b/docs/participant-guide.md @@ -46,7 +46,7 @@ Every word of the interface lives in [`app/src/main/res/values/strings.xml`](../ Use only the official installation source your research team gives you, and check that the app name and study description match what they told you. Do not install an APK from an unknown source, one that asks you to turn off Android security features, or one that asks for account passwords. This app does not ask for your passwords. -A study configuration file usually ends in `.adccfg`. When you import one, the app checks the signature on the file, the validity period, the minimum app version, and the full structure of the file. If any check fails, the app stops. It does not fall back to a permissive mode and it does not collect anything. If that happens, ask your research team for a correct file rather than trying to work around it. +A study configuration file usually ends in `.adccfg`. When you import one, the app checks the signature on the file, its Android platform target, the validity period, the minimum client build, and the full structure of the file. If any check fails, the app stops. It does not fall back to a permissive mode and it does not collect anything. If that happens, ask your research team for a correct file rather than trying to work around it. That signature check tells you the file has not been altered since it was signed. It does not, on its own, tell you who wrote it — the next section explains what the app shows you instead, and what you can do about it. @@ -112,7 +112,7 @@ Nothing in the researcher name, contact details, or study description proves who The consent step also says whether the study is **Anonymous or pseudonymous** (匿名或假名研究) or **Personalized** (個人化研究). An anonymous configuration contains no code assigned by the research team. A personalized configuration shows the exact opaque code embedded in your signed file; compare it with the code the team gave you. The app never asks you to enter a name, email address, or phone number. -Every import also creates a fresh random installation code. Importing the same configuration twice therefore produces two different installation codes. A personalized export contains both codes inside its encrypted body; an upload exposes only the random installation code in its headers, never the researcher-assigned code. Treat either code as linkable study data even though neither is required to be a name. +Every import also creates a fresh random installation code. Importing the same configuration twice therefore produces two different installation codes. A personalized export contains both codes inside its encrypted body. Automatic upload headers contain neither code, nor the study or configuration identifier; they contain only bundle-level routing claims. Treat either code as linkable study data after decryption even though neither is required to be a name. ### Whether the study sends data automatically @@ -147,12 +147,17 @@ The **Data** step, step 2 of the setup, lists every one of them before you are a That sentence comes from the app, not from the research team. It is a template with that study's own settings filled into it, so a study that samples your location every ten seconds and one that samples it every ten minutes do not read the same. No field in a configuration can change the wording. -What a source cannot see is not printed on this screen. For that, use the table in section 3 of this guide and the [data dictionary](data-dictionary.md), which are written against the same source the app runs. +The screen also names selected limits the implementation can guarantee, such as omitted battery identity or inference labels. Those short statements are not an exhaustive threat model; use the table below and the [data dictionary](data-dictionary.md) for the complete field definitions and interpretation limits, which are written against the same source the app runs. | Row | The sentence it shows | | --- | --- | | **App activity** (App 使用狀況) — a rounded square with a dot in the middle | When this app itself is opened and closed (這個 App 本身何時被開啟與關閉) | | **Motion** (動作) — a wave, two crests around a centre line | Movement of the phone, about **N** times per second or more (手機的移動，每秒約 **N** 次或更多) | +| **Battery context** (電池情境) — opposing data arrows | Whole percentage, charging state/source, and power-save mode; not health, temperature, or hardware identity (整數電量、充電狀態與來源、省電模式；不含健康、溫度或硬體識別碼) | +| **Time context** (時間情境) — a clock | Time-zone setting, UTC offset, daylight-saving state, and clock changes; not a location or travel claim (時區、UTC 偏移、日光節約與時鐘變更；不代表位置或旅行) | +| **Phone rotation** (手機旋轉) — the motion wave | Raw three-axis rotation, about **N** times per second or more; no orientation or activity labels (三軸旋轉，每秒約 **N** 次以上；不含方向或活動標記) | +| **Ambient light** (環境光線) — the app-shaped sensor mark | Raw light level within the configured interval and threshold; no environmental content (依設定間隔與門檻記錄原始照度；不含環境內容) | +| **Proximity sensor** (接近感測器) — the connection arcs | Raw near/distance state within the configured interval and threshold; near/far transitions are recorded even below the distance threshold, and many phones report only near/far (依設定間隔與門檻記錄遠近／距離；遠近狀態切換不受距離門檻限制，且許多手機只能回報近或遠) | | **Connection type** (連線類型) — three rising arcs over a dot | Whether you are on Wi-Fi or mobile data, and whether it is metered (你目前是 Wi-Fi 還是行動網路，以及是否計費) | | **Data volume** (流量) — two arrows side by side, one up and one down | Total bytes your phone sent and received, every **T** (手機每 **T** 傳送與接收的總位元組數) | | **App and screen use** (App 與螢幕使用) — a phone outline with a short bar near its foot | Which apps open and close, and when the screen turns on, every **T** (每 **T** 記錄哪些 App 被開啟關閉、螢幕何時亮起) | @@ -173,6 +178,11 @@ The sentences are short because they are the summary. This is what each source a | --- | --- | --- | | App activity | Lifecycle timings of this app's own screens | Not your use of any other app | | Motion | Raw x/y/z acceleration from your phone, with timing and an accuracy status | Can be used to study movement or posture, but the app does not label it for you | +| Battery state | Whole percentage, charging state/source, and power-save mode | No health, temperature, capacity, serial, or hardware ID | +| Time context | Time-zone ID, UTC offset, DST state, and clock-change reason | A time-zone setting is not physical location or travel evidence | +| Rotation | Raw x/y/z angular velocity, timing, and accuracy status | No derived orientation, posture, activity, or gesture labels | +| Ambient light | Raw illuminance, timing, and accuracy status | No image or environmental content; values vary by hardware | +| Proximity | Raw distance/range and the phone's near/far interpretation | Many devices are binary; it does not prove presence or comparable distance | | Connection type | Wi-Fi/mobile/ethernet/VPN, whether the connection is validated/metered/roaming, bandwidth estimates | No Wi-Fi network names, IP addresses, web addresses, packets, or content | | Data volume | Total bytes and packets your device sent and received over Wi-Fi and mobile data during a time window | Coarse and possibly delayed; not per-app and not per-website usage | | App and screen use | Package names, app resumed/paused/stopped, screen, keyguard (lock screen), and boot/shutdown events | Android's own record of these can be delayed or incomplete | @@ -199,9 +209,12 @@ Tap anywhere on a row that is not granted yet and the app sends you straight to Used for scheduled study activities and for the notification that stays visible while collection is running. Activities use Android's background work system, which is not an exact alarm: battery saving, Doze, or system scheduling can delay them. -### Motion sensor (加速度感測器) and basic network state +### Sensor hardware (感測器硬體) and basic network state -These usually do not produce an extra Android permission dialog. The motion sensor item is a hardware check — whether your phone has the sensor at all — so its row does nothing when you tap it. Connection type uses only Android's ordinary network-state permission and has no row of its own. +Accelerometer, gyroscope, ambient-light, and proximity items are hardware checks, not permission +dialogs, so their rows do nothing when tapped. A required source blocks enrollment when its sensor +is absent; an optional source remains off. Connection type uses only Android's ordinary +network-state permission and has no row of its own. ### Usage access (使用情況存取權) @@ -252,6 +265,10 @@ After you restart your phone, only a study that was Collecting (收集中) tries Each scheduled activity has a durable occurrence identity. Restarting the phone, reopening the app, changing time zone, pausing, or recovering WorkManager reschedules that same occurrence instead of creating another one. Android can deliver a notification late, but the app records separately when it was scheduled, posted, opened, submitted, or expired; a notification being posted is not evidence that you saw it. +Some studies use signed random local-time windows. The phone chooses an instant locally and stores +it before scheduling; a retry or reboot does not draw a new time, and the research team cannot send +a remote trigger. The configuration fixes the windows, limits, and minimum spacing. + Tapping a survey notification opens exactly that occurrence. Surveys are native app screens with screen-reader labels, progress, required/optional indicators, and four answer types: short text, numeric scale, one choice, or multiple choices. Closing before submission stores no research answer or draft. Reopening returns to the same unanswered survey. Submission requires confirmation and is atomic: after one successful submission, the answer is read-only and cannot be edited or submitted again, even after restart or competing taps. An expired occurrence cannot be submitted. ### The status line @@ -289,7 +306,13 @@ When a send attempt fails, a code in red replaces the sent figure: | `UPLOAD_HTTP_` | The receiving server answered with an error, for example `UPLOAD_HTTP_503` | | `UPLOAD_FAILED` | Anything else | -**A send failure does not stop collection.** The study keeps collecting, your phone keeps the data, and the app tries again on its own schedule. Most of these codes describe an ordinary interruption — no Wi-Fi, a low battery, or the research team's server being briefly unavailable — and clear by themselves. Each attempt sends only what has not already been confirmed as received, so nothing is sent twice. +**A send failure does not stop collection.** The study keeps collecting and your phone keeps the +staged encrypted package. Connection failures and server-busy responses are retried using exactly +the same bytes, so the receiver can recognize a replay even when its success response was lost. +A redirect, an incomplete-success response, most other client-error responses, or a mismatched +receipt is recorded as a terminal delivery error instead of being retried forever. The app does +not mark those events sent or silently discard them; the research team must correct its study or +receiver setup. You do not need to interpret the code. If one is still on screen after several days, quote it to your research team as it appears; it tells them where the attempt failed and contains none of your collected data. @@ -334,9 +357,9 @@ Finishing or withdrawing does not strand data the research team was already owed ### Pause and resume -When you press Pause (暫停), the app writes a pause boundary, stops the sources, and flushes events that were already queued before that boundary. Once the status line shows Paused (已暫停), no new study events are accepted for the period you are paused. Data already collected stays on your phone, encrypted. The ongoing notification goes away. +When you press Pause (暫停), the app writes a pause boundary, stops the sources, and flushes events that were already queued before that boundary. Once the status line shows Paused (已暫停), no new study events are accepted for the period you are paused. Data already collected stays on your phone, encrypted. The ongoing notification and any visible scheduled-activity notification go away. A survey cannot be opened or submitted while paused. -Pressing Resume (繼續收集) starts a new collection interval. Data volume and app and screen use are not backfilled: the app does not go back and collect what happened while you were paused. +Pressing Resume (繼續收集) starts a new collection interval. Data volume and app and screen use are not backfilled: the app does not go back and collect what happened while you were paused. Calendar time and scheduled-activity availability still pass during a pause, so the app expires missed activities and reconciles any remaining ones when you resume. ### Finish early @@ -381,7 +404,7 @@ Confirming deletes: - The encrypted study configuration and state. - All local event segments. -- The study's key in the Android Keystore — the hardware-backed key store your phone uses to hold encryption keys. Once that key is gone, any leftover encrypted bytes cannot be read back. +- The study's non-exportable key in Android Keystore. Some phones may protect it in hardware, but this app does not require or verify hardware backing. Once the key is gone, any leftover encrypted bytes cannot be read back. - The export receipt information currently held in the app. **This cannot be undone.** There is no recovery, no undo, and no backup inside the app. After deleting, you cannot export that study's data again, because the data no longer exists on your phone. If you want the research team to have your data, export and send it before you delete. @@ -395,19 +418,20 @@ Uninstalling the app or clearing its app data also destroys the local keys and d | Screen or situation | What to do | | --- | --- | | The interface is in a language you cannot read | Tap the globe at the top right of the header and pick a language; it is the same setting as Android Settings → Apps → Android Data Collector → Language | -| The configuration file will not import | Check the file, the app version, and the study's validity period; do not modify the `.adccfg`, and contact the research team | +| The configuration file will not import | Check the file, the client build/platform, and the study's validity period; do not modify the `.adccfg`, and contact the research team | | Check this against the fingerprint your research team published (請與研究團隊公佈的金鑰指紋核對) | Ordinary for most studies; compare the fingerprint on the Configuration signature (設定檔簽章) block with the one your research team published, and if you do not have one, ask before consenting | | The fingerprint does not match the one you were given | Do not consent. Contact your research team through details you already had, not details taken from the study screen | | A required item in the access list is not granted | Tap that row, finish on the Android screen it opens, and return to the app; if you do not want to grant it, do not start | | A source shows `ACCESS_UNAVAILABLE` | The access it needs is not granted. The other sources keep working; grant it only if you want to | | Motion sensor unavailable | The device has no compatible accelerometer; the app does not fabricate substitute data, so a study that requires it cannot start | +| Gyroscope, ambient-light, or proximity sensor unavailable | The phone lacks that hardware; required collection cannot start and optional collection stays off. There is no substitute or inferred fallback. | | A source shows a red dot and any other code | Pause first, check that permission or special access, then try to resume; if it still fails, contact the research team and quote the code | | Data volume does not change in real time | Android's accounting is coarse and delayed, and the study sets how often the app polls; this is a normal limit | | There are gaps in location | Check location access, your phone's location services, and the research foreground service; indoors it can still be inaccurate | | A scheduled activity does not arrive on time | Check the notification permission and battery-saving settings; interventions use inexact background scheduling | | A survey says it expired or is unavailable | It cannot be submitted after its signed response window; contact the research team if the timing was unexpected | | Export fails | Check that the destination is writable and has enough space, and retry in another location; the study status does not change because of a failed export | -| An `UPLOAD_…` code appears where the sent figure usually is | Nothing to do. Collection carries on and the app retries by itself; connect to Wi-Fi and charge the phone if it persists for days, then contact the research team and quote the code | +| An `UPLOAD_…` code appears where the sent figure usually is | Collection carries on. Network, timeout, busy-server, and temporary-server errors retry automatically; other protocol/receipt errors can be terminal. Connect to Wi-Fi and charge the phone, then contact the research team and quote a persistent code | | Storage failure / paused | The app fails closed and stops accepting events; do not clear the app's data, contact the research team first, or export if you need to | When the app has something to tell you, it shows a short code in capital letters in a red band directly under the header, beside a solid dot — `STORAGE_WRITE_FAILED`, `CONFIGURATION_IMPORT_FAILED`, `EXPORT_FAILED` and the like. Passing the code to your research team helps them diagnose the problem, and it contains none of your collected data. Two codes in that band are confirmations rather than problems: `EXPORT_COMPLETE` after an export finishes, and `LOCAL_DATA_DELETED` after a deletion. @@ -419,7 +443,7 @@ A study moves through nine states internally, but the screen names only four of | Internal name | What you see | What it means for you | | --- | --- | --- | | `IMPORTED` | Dot 1 of 5, the Study panel | The configuration file has been read in; nothing is being collected | -| `CONFIG_VERIFIED` | Dot 1 of 5, the Study panel | Signature, validity period, app version, source list, and export key all checked, which confirms the file is unaltered rather than who wrote it; nothing is being collected | +| `CONFIG_VERIFIED` | Dot 1 of 5, the Study panel | Signature, validity period, platform, client build, source list, and export key all checked, which confirms the file is unaltered rather than who wrote it; nothing is being collected | | `CONSENT_PENDING` | Dot 2 then dot 3 of 5, the Data panel then the Consent panel | You are reading what would be collected and deciding whether to take part | | `ACCESS_SETUP` | Dot 4 of 5, the Access panel | You are completing the required Android access | | `READY` | Dot 5 of 5, the Start panel | Waiting for you to press Start study (開始研究); nothing is being collected | diff --git a/docs/researcher-guide.md b/docs/researcher-guide.md index e2babf8..711a289 100644 --- a/docs/researcher-guide.md +++ b/docs/researcher-guide.md @@ -8,8 +8,9 @@ and whether it delivers them to an endpoint on a schedule — sign that file wit key, and hand it to participants. The participant app verifies the signature, presents the study, and runs exactly what the configuration specifies. -v1 ships seven collectors — app lifecycle, accelerometer, network state, network usage, -usage events, location, and research-keyboard touch dynamics — and runs the complete +v1 ships twelve selectable collectors — app lifecycle, accelerometer, battery state, temporal +context, gyroscope, ambient light, proximity, network state, network usage, usage events, +location, and research-keyboard touch dynamics — and runs the complete on-device loop on Android 14–17 (`minSdk 34`, `compileSdk`/`targetSdk 37`). Changing which of them a study uses, how often they sample, or how long the study lasts is a configuration change. Adding a collector that does not exist yet is a code change; see @@ -17,7 +18,7 @@ change. Adding a collector that does not exist yet is a code change; see Deploying a study is a short pipeline, and this guide follows it: -1. Generate the study signing key and the export encryption keyset (section 3). +1. Generate the study signing key and the export encryption key (section 3). 2. Write the study configuration (section 4). 3. Canonicalise, sign, and verify it (section 5). 4. Distribute the participant app and the `.adccfg` file, and publish your signing key @@ -29,6 +30,13 @@ Deploying a study is a short pipeline, and this guide follows it: Sections 1 and 2 come first because they shape the design: what the data can and cannot support, and how the two key pairs must be handled. +For implementation work, start with the [normative Protocol v1 contract](../protocol/v1/README.md), +its [collector catalog](../protocol/v1/collector-catalog.json), the +[P0–P2 decision record](p0-p2-implementation-contract.md), and the +[Collector capability policy](../assurance/README.md). Configuration, envelope, export, outbox, and +HTTP behavior live beside their tests in `core/study-definition`, `core/protocol`, `core/export`, +and `app/src/{main,test}/…/platform`; those links are indexed in the repository README. + The app runs the on-device loop; it does not run your study. Before you recruit anyone, the research team is responsible for: @@ -54,6 +62,11 @@ written up. | --- | --- | --- | | `app_lifecycle.v1` | Lifecycle transitions of this app's own Activities, each with the Activity class name | Anything about time spent in other apps. This instruments the collector app, not the participant's phone use. | | `accelerometer.v1` | Raw x/y/z acceleration in m/s² in device coordinates, the sensor's own timestamp, and the platform accuracy code | A recognised movement, posture, or activity. The app ships no classifier and produces no ground-truth label. | +| `battery_state.v1` | Whole battery percentage, charging state/source, and power-save state | Battery health, temperature, capacity, hardware identity, or the cause of a change. | +| `temporal_context.v1` | Time-zone ID, UTC offset, DST state, and a bounded reason for a time-context snapshot | Physical location or travel. A configured time zone is not location evidence. | +| `gyroscope.v1` | Raw x/y/z angular velocity in rad/s, sensor time, and accuracy | Orientation, posture, activity, or gesture labels. No inference is performed. | +| `ambient_light.v1` | Raw illuminance in lux, sensor time, and accuracy | Image content, a calibrated environment across devices, or whether a person is present. | +| `proximity.v1` | Raw sensor distance/range and the device's near/far interpretation | Comparable physical distance across devices or presence; many sensors are binary. | | `network_state.v1` | Transport flags for the default network (`wifi`, `mobile`, `ethernet`, `vpn`), `validated`/`metered`/`roaming`, and optional link bandwidth estimates | SSID, BSSID, IP address, hostname, URL, packet contents, or who the device communicated with. None of it is read. | | `network_usage.v1` | Device-total `rx_bytes`/`tx_bytes`/`rx_packets`/`tx_packets` per transport, over an explicit `[coverage_start_utc_millis, coverage_end_utc_millis]` window | An instantaneous throughput, a per-app attribution, or the precise time traffic occurred. The coverage window is the finest resolution that exists in the data. | | `usage_events.v1` | Raw platform events — activity resumed/paused/stopped, screen interactive/non-interactive, keyguard shown/hidden, device startup/shutdown — with the reporting package name where the platform supplies one | A complete or real-time session stream. The platform delays events, omits events, and does not guarantee that resume and pause pair up. | @@ -84,7 +97,7 @@ v1 uses two key pairs with different purposes. They are not interchangeable. | Key | What the private key does | Where the public key goes | | --- | --- | --- | | Ed25519 study signing key | Signs the canonical study configuration bytes | The signed study configuration itself, as `signer.public_key` | -| Tink X25519/HPKE keyset | Decrypts every bundle, exported or uploaded | The signed study configuration, as `export.tink_hpke_public_keyset` | +| Raw X25519/HPKE key | Decrypts every bundle, exported or uploaded | The signed study configuration, as `export.hpke_public_key` | Both public halves therefore travel inside the configuration, and neither requires an app build. A configuration certifies itself: the app verifies the signature with the key the @@ -104,10 +117,10 @@ The consequences differ, so track them separately. a new key and fingerprint, re-sign under the new key ID, and notify participants. There is no revocation mechanism, so configurations already signed under the old key remain valid until they expire — which is a reason to keep validity windows short. -- **HPKE private key lost.** Every export encrypted to that keyset is permanently +- **HPKE private key lost.** Every export encrypted to that key is permanently unreadable. There is no escrow and no recovery path. Participant devices cannot re-encrypt. - **HPKE private key leaked.** Anyone holding it can decrypt any export bundle for that - study that they can obtain. Rotating the keyset requires a new `configuration_id`, a new + study that they can obtain. Rotating the key requires a new `configuration_id`, a new signature, and fresh consent. Do not reuse one key pair for both roles, and do not reuse either across unrelated studies. @@ -148,18 +161,19 @@ Generate a production signing key: --public /secure/study-signing-public.key" ``` -Generate the export HPKE keyset: +Generate the export HPKE key: ```bash ./gradlew :researcher-tools:run --args="hpke-keygen \ - --private /secure/export-hpke-private.json \ - --public ./export-hpke-public.json" + --private /secure/export-hpke-private.key \ + --public ./export-hpke-public.key" ``` -The public signing key is a base64 X.509 Ed25519 key. Paste it into the study -configuration's `signer.public_key`, alongside the key ID you will sign with, and paste the -HPKE public keyset JSON object into `export.tink_hpke_public_keyset`. Both private keys stay -in the controlled research environment. Neither public key goes into an app build. +Each `.key` file contains one raw 32-byte key encoded as unpadded base64url. Paste the +Ed25519 public value into `signer.public_key` and the X25519 public value into +`export.hpke_public_key`. Tink JSON/protobuf keysets, X.509, PKCS#8, padded base64, and +standard-base64 keys are not Protocol v1 wire values. Both private keys stay in the controlled +research environment; neither public key goes into an app build. An institution that wants one build to accept only its own studies can additionally pin the signer — see the end of section 6. @@ -184,7 +198,7 @@ no fewer: ```text schema_version, experiment_id, configuration_id, assigned_participant_id, -issued_at, expires_at, minimum_app_version, +issued_at, expires_at, platform, minimum_client_version, title, researcher, purpose, duration_hours, consent, collectors, surveys, interventions, storage, signer, export, upload ``` @@ -204,7 +218,8 @@ Constraints enforced by - `issued_at` must precede `expires_at`. Verification requires the current time to be at or after `issued_at` and strictly before `expires_at`; the expiry instant itself is already expired. -- `minimum_app_version` must be positive. +- `platform` is exactly `"android"`. `minimum_client_version` is a positive canonical decimal + string (`"1"`, never a JSON number or a zero-padded string). - `title` 1–120 characters; `researcher.name` 1–120; `researcher.contact` 3–240; `purpose` 1–2,000. - `duration_hours` is 1–8,760, measured from the participant's first explicit start. @@ -227,12 +242,11 @@ Constraints enforced by durable idempotency metadata remains inside its encrypted 1 MiB bound. WorkManager timing is best effort, not an exact alarm. Any intervention requires notification access. - `storage.maximum_local_bytes` is 8 MiB–8 GiB (8,388,608–8,589,934,592). -- `signer` carries exactly `key_id` and `public_key`. `public_key` is the base64 X.509 - Ed25519 public half of the key you sign with, 32–1,024 characters. `key_id` must equal the +- `signer` carries exactly `key_id` and `public_key`. `public_key` is the raw 32-byte Ed25519 + public half encoded as unpadded base64url. `key_id` must equal the `--key-id` you pass to `sign`, and the app checks it against the envelope's signer key ID on import. -- `export.tink_hpke_public_keyset` must be a JSON object, 32–16,384 characters once - serialised. +- `export.hpke_public_key` is a raw 32-byte X25519 public key encoded as unpadded base64url. - `upload` is either the empty object `{}`, meaning the study does not upload, or an object carrying exactly `endpoint`, `interval_minutes`, and `allow_metered`. A partially filled block is rejected, so no endpoint or cadence is ever inherited from a default. @@ -247,7 +261,8 @@ With `"assigned_participant_id": null`, one signed artifact may be distributed t import independently mints a random `participant_instance_id`, including repeated imports of the same file. With a non-null assigned code, make a distinct artifact and `configuration_id` for each participant. The assigned code remains inside encrypted metadata and exports; only the random -instance UUID appears in upload routing headers. +instance UUID distinguishes events after decryption. Neither value appears in upload routing +headers. For a batch, supply a UTF-8 tab-separated mapping with exactly `configuration_idassigned_participant_id` per line, then run: @@ -266,10 +281,33 @@ never placed in filenames or printed. `canonicalize` and `sign` also accept An action is defined once and reused by all of an intervention's triggers. Calendar-relative schedules include pauses; active-running schedules exclude them. Daily local schedules follow the -phone's current time zone and are recomputed after time or zone changes. Each planned firing has a +phone's current time zone and are recomputed after time or zone changes. For both daily-local and +random-window schedules, a local minute that does not exist during a DST gap is skipped instead of +shifted outside the signed time; if a minute occurs twice during a DST overlap, the first +chronological occurrence is used. Each planned firing has a SHA-256 `occurrence_id` derived from its configuration, intervention, trigger, and schedule key, so reboot, process recovery, WorkManager retry, or duplicate execution cannot create a second firing. Occurrences stop at the study lifetime and expire after their availability window. +While a study is paused, the app removes pending prompt work and visible intervention +notifications, and it rejects prompt claims, opens, expiries, and survey submissions. Calendar +time and signed availability windows still advance; on resume the app reconciles the durable +occurrences, expires anything whose window elapsed, and schedules only what remains eligible. + +A `random_window` trigger fixes one to eight sorted, non-overlapping local-time windows plus +`occurrences_per_window`, daily and total caps, and `minimum_separation_minutes` in the signed +configuration. The phone uses a CSPRNG to choose each instant and persists the occurrence before +WorkManager is scheduled. Process death, retry, and reboot therefore reuse the same choice. +If a daily or total cap is smaller than the signed slots, the planner considers local dates in +planning order, then windows in their signed array order, then occurrence ordinals; the first +eligible slots consume capacity. A past or DST-nonexistent slot consumes nothing. The CSPRNG +chooses the minute inside a selected slot, not which window survives the cap. +Already materialized occurrences are never moved after a clock or time-zone change; only future +local dates are planned under the new context. Because repeated wall-clock edits can expose more +local dates than study duration alone implies, each random trigger contributes its full signed +`maximum_occurrences_total` to the global 512-occurrence safety bound. The Web editor shows that +true worst-case prompt count and the window bounds, never a participant's selected instants. Daily +local schedules still use the conservative UTC-18 through UTC+18 reachable-date bound. There is no +server trigger. A survey action references a reusable survey by ID. Display text uses `{ "default": "...", "translations": { "zh-TW": "..." } }`; stable IDs, never labels, appear in answers. Submissions @@ -281,7 +319,7 @@ The `signer` block looks like this: ```json "signer": { "key_id": "lab-signer-2026", - "public_key": "MCowBQYDK2Vw…the contents of study-signing-public.key" + "public_key": "" } ``` @@ -291,10 +329,15 @@ Per-collector configuration: | --- | --- | | `app_lifecycle.v1` | `{}` | | `accelerometer.v1` | `sampling_period_us` 5,000–1,000,000; `maximum_report_latency_us` 0–60,000,000 | +| `battery_state.v1` | `{}` | +| `temporal_context.v1` | `{}` | +| `gyroscope.v1` | `sampling_period_us` 5,000–1,000,000; `maximum_report_latency_us` 0–60,000,000 | +| `ambient_light.v1` | `sampling_period_us` 200,000–10,000,000; `change_threshold_millilux` 0–100,000,000 | +| `proximity.v1` | `minimum_event_interval_ms` 100–60,000; `change_threshold_millimeters` 0–10,000 | | `network_state.v1` | `include_bandwidth_estimates` boolean | | `network_usage.v1` | `transports` non-empty subset of `wifi`/`mobile`; `poll_interval_minutes` 1–1,440 | | `usage_events.v1` | `poll_interval_minutes` 1–1,440 | -| `location.v1` | `interval_millis` 1,000–3,600,000; `minimum_interval_millis` 500 to `interval_millis`; `maximum_batch_delay_millis` 0–86,400,000; `minimum_displacement_meters` 0–10,000; `priority` `BALANCED` or `HIGH_ACCURACY`. This collector always requires precise location; `priority` trades power against accuracy within it, and there is no coarse-only mode. | +| `location.v1` | `interval_millis` 1,000–3,600,000; `minimum_interval_millis` 500 to `interval_millis`; `maximum_batch_delay_millis` 0–86,400,000; `minimum_displacement_millimeters` 0–10,000,000; `priority` `BALANCED` or `HIGH_ACCURACY`. This collector always requires precise location; `priority` trades power against accuracy within it, and there is no coarse-only mode. | | `keyboard_touch.v1` | `trajectory_sampling_hz` 1–120 | Both polling collectors, and scheduled delivery, accept a one-minute floor. That floor exists @@ -318,14 +361,16 @@ into the app, in the participant's app language, that you can neither write nor Each entry is a name and a description filled in from that study's parameters, so a study sampling location every ten seconds and one sampling it every ten minutes do not read alike. -What the screen does not carry is a negative: it states what each source records, not what it -cannot see. Where a participant needs that — and an ethics submission usually does — it has -to come from your consent document. The parameters that -reach the screen are `accelerometer.v1`'s `sampling_period_us` (as a rate in hertz, stated as -"or more" because Android treats a sampling period as a hint and devices deliver faster than -asked), the `poll_interval_minutes` of `network_usage.v1` and `usage_events.v1`, and -`location.v1`'s `interval_millis` and `minimum_displacement_meters`. The other three -collectors read the same in every study. The exact wording is in +The description states what each source records and selected limits the implementation can +guarantee, such as omitted battery identity, text, inference, or presence claims. It is not an +exhaustive threat model, so study-specific risks and every additional participant commitment still +belong in your consent document. The parameters that +reach the screen include the accelerometer and gyroscope `sampling_period_us` (as a rate in hertz, +stated as "or more" because Android treats the period as a hint), the ambient-light interval and +threshold, the proximity interval and threshold, the `poll_interval_minutes` of +`network_usage.v1` and `usage_events.v1`, and +`location.v1`'s `interval_millis` and `minimum_displacement_millimeters` (displayed in metres). +Collectors without configuration fields read the same in every study. The exact wording is in [`app/src/main/res/values/strings.xml`](../app/src/main/res/values/strings.xml) and its `values-zh-rTW` counterpart; read it before you write your consent summary, because your participants will. @@ -363,8 +408,8 @@ The deployment consequence is real and worth planning for. **A study recruiting languages may still need one signed configuration per language for study and consent prose — same collectors, same parameters, its own consent document version, its own `configuration_id`, and its own signature — with each participant given the one written in theirs. Keep `experiment_id` shared across them so -the arms are recognisable as one study, and remember that bundles are de-duplicated on -`experiment_id` + `configuration_id` + `collector_id` + `sequence_number` (section 10), so +the arms are recognisable as one study, and remember that events are de-duplicated on +`experiment_id` + `configuration_id` + `participant_instance_id` + `sequence_number` (section 10), so the split reaches your analysis. Telling a participant to switch the app's language does not change a single word you wrote. @@ -401,8 +446,7 @@ fall back on, so it is re-established whenever the app's session initialises. Every link is constrained to an unmetered network unless `allow_metered` is true, plus a battery that is not low. Those constraints are why `interval_minutes` is a floor and not a -promise: a phone on mobile data all week delivers nothing until it reaches Wi-Fi. A failed -attempt retries with exponential backoff from one minute. +promise: a phone on mobile data all week delivers nothing until it reaches Wi-Fi. Delivery continues while the study is `PAUSED`, for data collected before the pause, and it continues after the study ends: finishing, completing on the duration deadline, and withdrawing @@ -411,30 +455,24 @@ reaches you. The chain stops renewing once the study is `COMPLETED` or `WITHDRAW everything it collected has been delivered. Deleting local data cancels delivery outright, so plan for a tail you may never receive and keep manual export in your protocol as the fallback. -**How much each run sends.** There is no configured chunk size. Each run asks for everything -outstanding, and how much fits is decided while the bundle streams: it stops at the first event -boundary past a 16 MiB plaintext budget, and the receipt records where it actually stopped. The -next run resumes from there. The budget is a transport constant in `OkHttpStudyUploader`, not a -per-study setting — `interval_minutes` is what paces delivery, and the budget only binds while a -backlog is being worked off, so a study keeping up with its cadence never meets it. - -The consequence for you is that chunk boundaries are not predictable from the configuration. -Read `first_sequence_number` and `last_sequence_number` out of each bundle rather than deriving -them from `interval_minutes` or an event count. +**How much each run sends.** Before opening HTTP, the app selects an exact event boundary, creates +one complete `ADCEXP01` bundle under its no-backup directory, flushes it to durable storage, and +records a manifest containing its bundle UUID, exact range/count, byte count, and SHA-256. One +outbox entry exists at a time. Its target plaintext budget is about 16 MiB and its hard wire limit +is 32 MiB. Process death, reboot, timeout, and response loss reuse the same bytes; the app never +regenerates ciphertext for a retry. Chunk boundaries are therefore read from the bundle or +receipt, not derived from cadence or an expected event count. **What upload does not do.** It does not gate collection: a study whose endpoint is down, misconfigured, or never deployed keeps recording, and a delivery failure is not treated as a collection incident on the participant's screen. -**When delivery fails.** The participant's dashboard shows a fixed code for the last failed -attempt, derived from the transport failure and never from response content: `UPLOAD_TIMEOUT`, -`UPLOAD_HOST_UNRESOLVED`, `UPLOAD_CONNECT_REFUSED`, `UPLOAD_TLS_HANDSHAKE_FAILED`, -`UPLOAD_TLS_FAILED`, `UPLOAD_INTERRUPTED`, `UPLOAD_IO_FAILED`, `UPLOAD_HTTP_` for a -non-2xx response, or `UPLOAD_FAILED` for anything else. That code is what to ask a participant -to read out when your endpoint has seen nothing from them, because it separates a name -resolution or TLS problem on your side from a phone that never had a network. It does not -overwrite the incident code a storage or access problem sets, and collection carries on either -way. +**When delivery fails.** Only I/O failures, `408`, `425`, `429`, and `5xx` are retryable. A +redirect, `202`, every other `4xx`, malformed receipt, or receipt mismatch is a terminal failure +for that staged bundle. It remains explicit and collection continues; the app does not silently +drop the staged bytes or advance the watermark. The dashboard's fixed incident code is derived +from transport state rather than response content, so support can distinguish DNS, connection, +TLS, timeout, I/O, and HTTP failures without logging a participant identifier. **What confirmed delivery does to local storage.** A study that comfortably fits its quota keeps every event on the phone. Once storage passes 80% of `storage.maximum_local_bytes`, a @@ -449,53 +487,48 @@ The research consequence is in section 10: once a participant's device has recla prefix, their manual export covers a window rather than the whole study, so an uploading study's dataset is the reassembled chunks plus that final export. -What the endpoint receives is the same `ADCEXP01` bundle described in section 9 — ciphertext -wrapped to your HPKE public key — as an `application/octet-stream` POST body with a chunked -transfer encoding, because the bundle is generated as it is written and its length is not -known up front. Everything needed to file and de-duplicate a chunk travels in request -headers, since reading it out of the body would require the private key: +What the endpoint receives is the same `ADCEXP01` bundle described in section 9: a fixed-length +`application/vnd.adc.research-bundle` POST with `Content-Digest` and no transfer encoding. These +are the only ADC routing headers: | Header | Value | | --- | --- | | `X-ADC-Bundle-Format` | `research-bundle-v1` | -| `X-ADC-Experiment-Id` | `experiment_id` from the configuration | -| `X-ADC-Configuration-Id` | `configuration_id` from the configuration | -| `X-ADC-Participant-Instance` | The participant instance ID | -| `X-ADC-Sequence-From` | The first sequence in this chunk. Exact, and strictly increasing across a participant's chunks | -| `X-ADC-Sequence-To-At-Most` | The last sequence durable on the device when the request began. An upper bound, not the window | - -The two headers are not symmetric, and the asymmetry is in the name for a reason. Headers are -sent before the body is generated, so the device knows where a chunk starts but not yet where it -ends: the request budget can stop it at any earlier event boundary. The range a chunk actually -contains is the `first_sequence_number` and `last_sequence_number` inside it, which your endpoint -cannot read — that is ciphertext. **File and de-duplicate on `X-ADC-Sequence-From` together with -`X-ADC-Participant-Instance`**; the next chunk resumes exactly where this one stopped, so that -pair is unique per chunk. An endpoint that records `X-ADC-Sequence-To-At-Most` as a held range -will claim sequences it does not have, and nothing later will correct it. - -`assigned_participant_id` is intentionally absent from the URL and every header. It is sensitive -join data and exists only inside the HPKE-encrypted configuration/experiment content. Do not add it -to reverse-proxy logs or invent a routing header for it. - -Your endpoint must answer 2xx only once it has durably stored the body. The device advances its -watermark to wherever the bundle actually stopped, never sends those sequences again, and may -release them locally if the study's storage runs high — so a 2xx you have not earned can cost -data that exists nowhere else. Answer 408, 429, or 5xx to ask for a retry; any other 4xx is -treated as a request that will keep failing and is not worth the participant's battery. - -**The participant instance ID.** A fresh random UUID generated on the device for every import when the study is -imported, stored in that study's metadata, and included in every bundle and every upload -request. Without it, bundles from different participants arrive indistinguishable — a manual -export carries that information out of band, an upload does not. It is pseudonymous: it +| `X-ADC-Bundle-Id` | Lowercase bundle UUID; the receiver's immutable object key | +| `X-ADC-Configuration-SHA256` | SHA-256 of the exact canonical configuration bytes | +| `X-ADC-Researcher-Key-Id` | Export recipient key ID | +| `X-ADC-Sequence-From` | Exact claimed first sequence | +| `X-ADC-Sequence-To` | Exact claimed last sequence | +| `X-ADC-Event-Count` | Exact claimed event count | + +`Content-Length` and `Content-Digest` are also required. The headers are untrusted routing claims: +the receiver can check their syntax, arithmetic, body digest, and the identities exposed by the +outer framing, but cannot authenticate the encrypted participant or sequence claims. No +`participant_instance_id`, `assigned_participant_id`, `experiment_id`, or `configuration_id` +appears in the URL or headers. Do not invent such a header or deduplicate ingestion by a +participant/range pair. Receiver replay identity is the bundle UUID plus exact stored bytes and +metadata. + +After durable storage, a new object returns `201 Created`; an exact replay returns `200 OK` with +the original response bytes. Both carry compact canonical JSON containing exactly `bundle_id`, +`byte_count`, `configuration_sha256`, `event_count`, `first_sequence_number`, +`last_sequence_number`, and `sha256`, with counts and sequence values as decimal strings. Only a +receipt that matches every outbox value advances the watermark. A reused bundle ID with different +content is `409 Conflict`; `202` is never success. + +**The participant instance ID.** A fresh random UUID generated on the device for every import, +stored in that study's encrypted metadata, and included in every decrypted bundle. It is +pseudonymous: it contains no name, account, device identifier, or advertising ID, and it is not shared across -studies. Re-importing the same anonymous or personalized artifact generates a different UUID, so -its upload chunk identity cannot collide. Treat it as personal data anyway, because it links every -chunk one import produced. +studies. Re-importing the same anonymous or personalized artifact generates a different UUID. +Treat it as personal data because it links every event one import produced, but do not use it as +receiver authentication. **You must disclose upload in your consent text.** The app renders the endpoint host, the cadence, the network condition, the fact that only your key can open the payload, and the -existence of the instance ID into the consent step, directly below your summary and taken -from the signed configuration rather than from it. That is a floor, not a substitute — the +fact that a random installation code travels inside the encrypted data so datasets can be +distinguished. This block sits directly below your summary and is derived from signed state rather +than your prose. That is a floor, not a substitute — the same relationship the data step has to your summary: your consent document has to say who operates the endpoint, where it is hosted, what jurisdiction it sits in, how long chunks are retained there, and who can reach them. A participant cannot decline upload while @@ -517,9 +550,9 @@ Canonicalise first: --output ./study-canonical.json" ``` -Canonicalisation re-emits the object with a fixed key order and normalised timestamp and -number formatting. The signing step decodes the file again and refuses it if the bytes are -not already canonical, so you cannot accidentally sign a hand-edited draft. +Canonicalisation emits RFC 8785 JCS bytes. The signing step parses the file again and refuses it +unless re-encoding produces exactly the same bytes, so duplicate members, noncanonical numbers, +unknown fields, and hand-edited near-canonical drafts fail closed. Sign the canonical bytes: @@ -537,8 +570,10 @@ written, because a mismatch would produce a file that signs cleanly and then fai device: the second failure reads `signer.public_key in the configuration does not match --private`. -The result is a signed study configuration: an `ADCCFG01` envelope carrying the signer key -ID, the canonical configuration bytes, and the Ed25519 signature over exactly those bytes. +The result is a signed study configuration: an `ADCCFG01` envelope containing magic, a two-byte +signer-key-ID length, a four-byte configuration length, the key ID, exact JCS configuration bytes, +and a fixed 64-byte Ed25519 signature over only those configuration bytes. No signature-length +field or alternate framing is accepted. On success the command prints the IDs it signed and the fingerprint of the signing key, for example: @@ -547,11 +582,11 @@ signed my-study-2026 my-study-config-01 fingerprint 9D0D AE5A 0D20 B29F D642 942A 0E17 4AAE ``` -That fingerprint is SHA-256 over the encoded public key, truncated to 16 bytes and rendered +That fingerprint is SHA-256 over the raw 32-byte public key, truncated to 16 bytes and rendered as eight groups of four hex characters. It is what the consent screen shows the participant, and what you publish in your recruitment material — section 6. -Verify independently — envelope structure, signature, app version floor, and the validity +Verify independently — envelope structure, signature, client build floor, platform, and validity window — before anything reaches a participant: ```bash @@ -573,8 +608,8 @@ Supply both and the check pins the signer instead, reproducing what a build list would enforce; the last line then reads `pinned yes`. A configuration that names the pinned key ID while carrying a different public key is rejected, so pinning cannot be sidestepped. -`--app-version` is the participant app's `versionCode`; if omitted the check treats the app -version floor as satisfied. `--now` takes an ISO instant and lets you confirm that a +`--app-version` is the participant client's `versionCode`; if omitted the check treats the client +build floor as satisfied. `--now` takes an ISO instant and lets you confirm that a configuration is refused before `issued_at` and after `expires_at` without changing the system clock. @@ -596,10 +631,12 @@ Actions release workflow; the required secrets and setup are described in the re Google Play distribution uses the corresponding AAB and track process. Building the app is not part of issuing a study: the same build verifies any correctly signed `.adccfg`. -Distribution of the configuration is manual. There is no download endpoint: participants -import the `.adccfg` through the system file picker. Getting data back is manual too unless -the study declares an upload endpoint, in which case delivery is automatic and manual export -remains available alongside it. Plan the logistics of both directions into your protocol. +Participants can import the `.adccfg` through the system file picker, or open an immutable +`adc://join/v1` link / QR generated by the Web authoring surface. Join hosting is transport only: +the link fixes the artifact's complete SHA-256 and signer fingerprint, and the app downloads once, +verifies the digest before the ordinary signature flow, and never polls for replacement. Getting +data back is manual unless the study declares an upload endpoint, in which case delivery is +automatic and manual export remains available alongside it. Plan both directions separately. The app declares `android.permission.INTERNET` and sets `usesCleartextTraffic="false"`, so the permission list of a build no longer tells you whether a given study transmits — the @@ -618,6 +655,28 @@ jq .upload ./study-canonical.json `{}` means the study never transmits. Run the second check against the exact configuration you are about to sign, and against what your consent document tells participants. +### Optional immutable join link and QR + +After signing in the Web authoring flow, enter the HTTPS location where the exact `.adccfg` bytes +will be served. The browser creates the join URI and QR locally; it does not call a QR service. The +artifact URL must use the narrow Protocol v1 profile: lowercase DNS-style HTTPS host, no credentials, +explicit default port, query, fragment, percent escape, dot segment, or repeated slash, followed by one or +more ASCII filename / token path segments. This restriction keeps Kotlin and browser URL handling +byte-for-byte identical rather than relying on either platform's silent normalization. + +For a personalized configuration, publish each file at a unique path whose final segment is at +least 22 random base64url characters (128 bits or more). Never put the roster code in the path, +filename, query, CDN analytics, or access-log label. The Web authoring control rejects a URL that +contains the assigned ID or lacks the opaque token. An anonymous configuration can use a stable +immutable filename. + +Treat a generated join link as part of the exact signed artifact release: changing the hosted +bytes makes its SHA-256 fail and requires a newly generated link. The app rejects redirects, +implicit retry, an oversized artifact, a fingerprint mismatch, and every join attempted while a +study or deletion is active. Download staging lives under no-backup storage and is removed at app +startup and after every outcome. Join does not add configuration refresh, remote control, or a +second consent path. + ### Publish your fingerprint The consent step shows the key fingerprint under the heading *Configuration signature* @@ -734,10 +793,10 @@ Researchers must not: The app sends no telemetry, no analytics, and no crash reports. In a study with an empty `upload` block you therefore learn nothing about a participant's progress unless they tell you, and completeness can only be assessed once they choose to share an encrypted export. In -an uploading study your endpoint sees delivery activity per participant instance, which is -the closest thing to monitoring available — and it is arrival of ciphertext, not a health -check. A silent instance may have paused, withdrawn, run out of Wi-Fi, or lost the phone, -and the four look alike from the endpoint. +an uploading study your endpoint sees ciphertext object arrivals, but the clear routing metadata +does not identify a participant and is not a health check. Silence can mean pause, withdrawal, +network constraints, a lost phone, or a terminal upload error; those cases look alike from the +endpoint. ## 9. Receive and decrypt bundles @@ -747,47 +806,51 @@ data governance procedure. Filenames and receipt timestamps are metadata a parti controls; they are not evidence of identity or integrity. An uploaded chunk carries no more proof of origin than an emailed export does — see [`threat-model.md`](threat-model.md). -Decrypt with the matching canonical configuration and the HPKE private keyset: +Decrypt with the matching canonical configuration and raw HPKE private key: ```bash ./gradlew :researcher-tools:run --args="decrypt \ --bundle ./participant-export.adcexp \ - --private /secure/export-hpke-private.json \ + --private /secure/export-hpke-private.key \ --config ./study-canonical.json \ --output /controlled/participant-export.json" ``` The same command decrypts an uploaded chunk; point `--bundle` at the stored request body. -`decrypt` streams, so a bundle larger than your machine's memory still decrypts. Size your -controlled environment for that. A bundle has no ceiling of its own: it is bounded by the study's -`storage.maximum_local_bytes`, so a manual export at the end of a long, high-rate study scales -with the quota you asked for, which can be 8 GiB. +`decrypt` streams, so a bundle larger than your machine's memory still decrypts. A manual bundle +has no separate transport ceiling: it is bounded by `storage.maximum_local_bytes`, so an end-of-study +export can scale to the 8 GiB quota. Automatic upload bodies are instead capped at 32 MiB. -The command refuses to overwrite an existing `--output` path, and writes its plaintext to a -temporary file in the destination directory first, moving it into place only after the whole -bundle has decrypted. That staging is not tidiness. The AES-GCM tag is verified only once the -last byte has been read, so a truncated or tampered bundle produces plausible-looking plaintext -right up to the point where it fails; staging means a failed verification leaves nothing behind -that could be mistaken for a partial dataset. +The command refuses to overwrite an existing `--output` path. It creates a mode-`0600` temporary +file in the destination directory, decrypts into it, then rereads it through the sole closed-world +bundle verifier. Only after AEAD, JCS, repeated identities, configuration signature, range/count, +transition history, and every catalog event contract pass does it flush and atomically move the +file into place. The AES-GCM tag is verified only at EOF, so failed decryption or semantic +verification deletes staging and publishes no partial plaintext. -The bundle is an `ADCEXP01` container: a per-bundle AES-256-GCM content key wrapped to your -HPKE public keyset, over a plaintext JSON document with this shape: +The bundle is an `ADCEXP01` container: a per-bundle AES-256-GCM content key wrapped with RFC 9180 +base-mode X25519/HKDF-SHA256/AES-256-GCM HPKE, over one authenticated JCS document with this +shape: ```text -format "research-bundle-v1" -exported_at_utc_millis -configuration the canonical study configuration +bundle_id, bundle_kind, format outer UUID, manual_export/automatic_upload, research-bundle-v1 +configuration_sha256 SHA-256 of the exact embedded canonical configuration +configuration the exact signed configuration object +configuration_signature signer_key_id and raw Ed25519 signature provenance +producer platform and client_version +exported_at_utc_millis decimal string experiment: experiment_id, configuration_id, participant_instance_id, - assigned_participant_id (personalized studies only), - state, next_sequence_number, + assigned_participant_id (nullable), state, + next_sequence_number, retained_from_sequence, durable_through_sequence, + uploaded_through_sequence, event_count, transitions[]: from, to, reason, - time: { wall_time_utc_millis, elapsed_realtime_nanos, boot_session_id } + time: { wall_time_utc_millis, monotonic_time_nanos, boot_session_id } events[]: sequence_number, collector_id, payload_schema_version, - observed_time: { wall_time_utc_millis, elapsed_realtime_nanos, boot_session_id }, + observed_time: { wall_time_utc_millis, monotonic_time_nanos, boot_session_id }, payload_type, fields first_sequence_number, last_sequence_number ``` @@ -799,26 +862,32 @@ phone if the device has reclaimed a delivered prefix. `participant_instance_id` pseudonymous per-import identifier described in section 4. A personalized export additionally carries `assigned_participant_id`; use it only as the researcher's opaque join key. -The two window fields are written after `events`, not before it, because a budget decides where -an uploaded bundle stops while it is still streaming. Declaring the window up front would let a -bundle claim a range it does not contain, which is worse than not declaring one. JSON object -member order carries no meaning, so this changes nothing for a parser that reads by key, and it -changes nothing about decryption — the format string is unchanged and bundles produced by -earlier builds decrypt exactly as before. It does matter to code that streams a bundle and -expects the window before the events it describes. - -Every value inside `fields` is a JSON string, including numeric ones. Parsing and range -checking are your responsibility. - -`research-bundle-v1` is bound into the HPKE and AES-GCM associated data, so a reader built -for a different version fails to decrypt rather than misreading one. Use -the `researcher-tools` build that matches the app you distributed. - -Successful decryption proves that the HPKE context and the AES-GCM tag verified: the bundle -was encrypted to your key, for this `experiment_id`/`configuration_id`/`researcher_key_id` -triple, and has not been altered since. It proves nothing about the participant's legal -identity, nothing about device attestation, and nothing about whether the platform dropped -data before it was recorded. +All sequence, count, wall-time, monotonic-time, byte-count, and client-version values are +canonical decimal strings. Every value inside `fields` is also a JSON string, but its exact field +set, type interpretation, units, clock basis, and bound come from +[`protocol/v1/collector-catalog.json`](../protocol/v1/collector-catalog.json); do not infer a +schema from observed data. The embedded configuration, its digest, its original signature, +producer, outer identities, range/count contiguity, and every catalog payload are verified before +plaintext is published. + +The JCS context binds `research-bundle-v1`, bundle UUID, full configuration SHA-256, and +researcher key ID into HPKE `info` and document AES-GCM AAD. A wrong key, context, framing byte, +embedded identity, or old Protocol v1 artifact fails closed. This Protocol v1 definition is a +destructive pre-1.0 replacement; there is no former-v1 fallback. + +Successful validation proves encryption to the configured researcher key, document integrity, +and the provenance of the exact embedded signed configuration. It proves nothing about the +participant's legal identity, participant/device authenticity, device attestation, or whether the +platform dropped data before it was recorded. + +For a dataset rather than a one-file inspection, use [`adc-analysis`](../adc-analysis/README.md). +Its `inventory` command copies local exports or R2/S3-compatible objects into a +content-addressed ciphertext workspace before keys are used. `materialize` then verifies each +whole bundle, quarantines failures, reassembles by +`(experiment_id, configuration_id, participant_instance_id, sequence_number)`, refuses +conflicting duplicates, and atomically publishes typed Parquet plus a provenance manifest and +quality summary. It performs no schema inference and has no database sink or receiver-side +decryption path. Do not partially analyse a file that fails to decrypt. Quarantine it, and where appropriate ask the participant to export a fresh encrypted bundle. @@ -834,8 +903,8 @@ An export is a snapshot, not a state change: - A later export contains the earlier events plus newer ones. Overlap between bundles from the same participant is expected, not an error. Uploaded chunks are the exception: each one starts after the sequence the previous delivery confirmed, so consecutive chunks abut - rather than overlap. Where they abut is decided by the plaintext budget while each bundle - streams, so chunk sizes vary and are not derivable from the configuration — take the window + rather than overlap. Where they abut is selected before the immutable outbox bundle is staged, + so chunk sizes vary and are not derivable from the configuration — take the window from `first_sequence_number` and `last_sequence_number`. - In a study that does not upload, every export is a whole history and the last one is the dataset. In an uploading study it need not be: once a device has reclaimed a delivered @@ -843,9 +912,11 @@ An export is a snapshot, not a state change: then the reassembled chunks plus the final export, and `first_sequence_number` on each bundle tells you where it starts. Keep the chunks; do not treat a late manual export as a replacement for them. -- De-duplicate events on `participant_instance_id` + `sequence_number`; the sequence is global to - collectors, intervention lifecycle, and survey responses within one import. `experiment_id` and - `configuration_id` identify the signed artifact rather than a unique device run. Sequence numbers come from a single monotonic counter per study, so +- Partition a dataset by `(experiment_id, configuration_id)` and de-duplicate events on + `(participant_instance_id, sequence_number)`. Equivalently, the complete event identity is + `(experiment_id, configuration_id, participant_instance_id, sequence_number)`. If the same key + carries different content, report a conflict; never choose a last writer. The sequence is global + to collectors, intervention lifecycle, and survey responses within one import. Sequence numbers come from a single monotonic counter per study, so they are stable across exports and uploads alike, and reclaiming never reissues one. In an uploading study, `participant_instance_id` is what separates repeated imports and devices. - A gap in the delivered sequence range is not proof of data loss. A chunk may not have been @@ -854,7 +925,7 @@ An export is a snapshot, not a state change: for them in the chunks you already hold. Ask for a manual export before treating a gap as missing data. - Reconstruct running and paused windows from `transitions` together with - `observed_time.elapsed_realtime_nanos` and `observed_time.boot_session_id`. Do not infer + `observed_time.monotonic_time_nanos` and `observed_time.boot_session_id`. Do not infer them from export times. ## 11. Analysis notes @@ -875,6 +946,23 @@ session, and inspect sampling gaps before filtering, feature extraction, or mode Posture requires estimating the gravity direction. Movement classification requires independent labels and independent validation. The app supplies no ground truth. +Gyroscope axes use the same device coordinate system and boot-relative hardware timestamp, but +measure rad/s rather than acceleration. Combining the two can support a model; it does not turn +either stream into orientation or activity ground truth. + +### Battery and temporal context + +Battery percentage is a whole platform reading, and charging/power-save fields are context rather +than a causal explanation for sampling gaps. Temporal-context events identify settings and clock +changes. Treat time-zone ID as a setting, not location; split monotonic analyses by boot session +and use these events when interpreting wall-clock discontinuities. + +### Ambient light and proximity + +Illuminance and distance are raw, device-specific sensor values. Do not compare their numeric +precision across models without calibration. Many proximity sensors are binary, and neither a +near event nor a light change proves participant presence or behavior. + ### Network state and usage A state event marks a change in Android's default network or its capabilities. A VPN or diff --git a/docs/system-design.md b/docs/system-design.md index 3a0be1a..d070901 100644 --- a/docs/system-design.md +++ b/docs/system-design.md @@ -6,6 +6,12 @@ study configuration can only select collectors that are already compiled into th download or execute arbitrary code. Collection and storage never require the network; delivery to a researcher endpoint is an option a study configuration turns on. +The [normative Protocol v1 contract](../protocol/v1/README.md) and its +[collector catalog](../protocol/v1/collector-catalog.json) define the wire and event schemas. The +[P0–P2 implementation contract](p0-p2-implementation-contract.md) records the locked decisions, and +[`assurance`](../assurance/README.md) defines the static Collector capability policy. This document +explains how the current modules realize those contracts. + ## 1. Goals and boundaries - Android 14-17 (`minSdk 34`, `compileSdk`/`targetSdk 37`). @@ -49,6 +55,8 @@ flowchart LR Session --> Runtime[":core:experiment-runtime"] Runtime --> API[":core:collector-api"] Collectors[":collector:*"] --> API + SensorCollectors["hardware sensor collectors"] --> SensorCommon[":collector:sensor-common"] + SensorCommon --> API Session --> StorePort["StudyStore port"] Storage[":core:storage"] --> StorePort Session --> Export[":core:export"] @@ -66,24 +74,28 @@ flowchart LR | Module | Responsibility | | --- | --- | -| `:app` | Compose interface and its localized string resources, bounded UI state, SAF, and the Android foreground/work/recovery/upload adapters | +| `:app` | Compose interface and localized resources, bounded UI state, SAF, Android foreground/work/recovery adapters, single-entry upload outbox, and HTTP adapter | | `:core:model` | Bounded study metadata, the state and event model, and the `StudyStore` port with its retained-window contract | | `:core:study-definition` | Strict canonical JSON, closed-world typed study and collector configuration | -| `:core:protocol` | Signed envelope, signature verification, optional signer pinning, validity-window and version checks | +| `:core:protocol` | Signed envelope, immutable join URI, signature verification, optional signer pinning, validity-window and version checks | | `:core:collector-api` | Collector lifecycle, health, registry, access contract, and the shared callback dispatcher | -| `:core:crypto` | Tink HPKE key handling, wrapping, and unwrapping | +| `:core:crypto` | Protocol v1 raw-key Ed25519 verification and fixed-suite RFC 9180 HPKE over raw X25519 keys; Tink remains internal only, never a wire keyset | | `:core:access` | Runtime permission, Usage Access, input-method, and hardware preflight | | `:core:experiment-runtime` | Command serialization, state machine, collector supervision, event admission gate, durable occurrence lifecycle, and atomic survey submission | | `:core:study-application` | The single active-study session; recovery and coordination of storage/access/host/work/export/upload ports, schedule reconciliation, and the upload watermark | -| `:core:storage` | Android Keystore, encrypted metadata, appended event segments, reclaiming delivered ones, recovery | -| `:core:export` | Streaming JSON/AES-GCM over a requested sequence window under an optional plaintext budget, HPKE key wrapping, and receipts | +| `:core:storage` | Android Keystore, encrypted metadata, appended event segments, reclaiming delivered ones, and strict one-event journal recovery | +| `:core:export` | Authenticated JCS/AES-GCM bundle construction over an exact sequence window, HPKE wrapping, closed-world streaming bundle verification, provenance, and strict receipt parsing | | `:collector:*` | One independent module per data source | +| `:collector:sensor-common` | Listener-thread ownership shared only by raw Android hardware-sensor collectors; no schema, inference, or storage policy | | `:researcher-tools` | CLI for Ed25519/HPKE key generation, canonicalization, signing, configuration checking, and decryption (`signing-keygen`, `hpke-keygen`, `canonicalize`, `sign`, `check-config`, `decrypt`) | -A collector feature depends on `collector-api` and `study-definition` and nothing else in the module -graph. It emits through `EventSink` only. It cannot see storage or the runtime, change state -directly, write files, export, or request permissions. `CollectorRegistry` rejects an ID that is not -compiled in, and rejects duplicate IDs at construction. +A collector feature depends on `collector-api`, `study-definition`, and, for raw hardware listener +ownership only, `collector:sensor-common`. This is the sole collector-to-collector dependency and +the capability policy scans it with the feature modules. A collector emits through `EventSink` +only. It cannot see storage or the runtime, change state directly, write files, export, or request +permissions. +`CollectorRegistry` rejects an ID that is not compiled in, and rejects duplicate IDs at +construction. ### The participant interface @@ -113,37 +125,41 @@ was signed. ## 3. Trust and configuration protocol -A study configuration is strict canonical JSON. Object fields must match the current schema exactly. -Unknown fields, unknown collectors, duplicate IDs, non-canonical encoding, and out-of-range numbers -are rejected outright. There is no guessing and no compatibility fallback. +A study configuration is strict RFC 8785 JCS. Object fields must match the current schema exactly. +Unknown fields, unknown collectors, duplicate IDs, noncanonical bytes, malformed UTF-8, +non-integral or out-of-range numbers, and trailing bytes are rejected. Sequence/time/build values +are canonical decimal strings. Protocol v1 is a destructive pre-1.0 definition: artifacts from the +former v1 implementation have no fallback reader. The configuration is wrapped in an `ADCCFG01` binary envelope. All multi-byte integers in the binary layouts in this document are big-endian. ```text magic(8) | signerKeyIdLength(u16) | configLength(u32) | -signatureLength(u16) | signerKeyId | canonicalConfig | Ed25519Signature +signerKeyId | canonicalConfig | Ed25519Signature(64) ``` The signing public key travels inside the signed bytes, as a mandatory root `signer` block of -`key_id` and a base64 X.509 Ed25519 `public_key`. A configuration therefore certifies itself, which +`key_id` and an unpadded-base64url raw 32-byte Ed25519 `public_key`. The export block similarly +carries a raw 32-byte X25519 key. X.509, PKCS#8, Tink JSON/protobuf keysets, padded base64, and +standard-base64 are invalid wire values. A configuration therefore certifies itself, which is what lets one published app verify any researcher's study. Verification order: 1. Validate the envelope length and format. -2. Strictly decode the configuration and validate the schema and the collector parameters. Nothing - decoded is acted on until step 5 passes. +2. Strictly decode and exactly re-encode the JCS configuration; validate schema, Android platform, + raw keys, collector parameters, and `minimum_client_version`. Nothing decoded is acted on until + the signature passes. 3. Require `signer.key_id` to equal the envelope's `signerKeyId`. 4. Resolve the Ed25519 public key: the pinned key for that key ID if the build has one, otherwise the key the configuration declares. 5. Verify the signature over the raw canonical configuration bytes. -6. Check `issued_at <= now < expires_at` and `minimum_app_version`. -7. Confirm the Tink HPKE public keyset carried in the configuration can build a `HybridEncrypt`. +6. Check `issued_at <= now < expires_at`, platform, and the client build floor. -Steps 1-6 live in `ConfigurationVerifier`, which returns a `VerifiedConfiguration` — the -configuration plus `signerAnchored`. Step 7 is applied by the app's composition root when it -activates a configuration, so the `check-config` CLI covers steps 1-6 only. +These checks live in `ConfigurationVerifier`, which returns a `VerifiedConfiguration` containing +the exact canonical bytes, signer key ID and signature, configuration SHA-256, typed configuration, +and `signerAnchored`. That preserved provenance is embedded and reverified in every bundle. `ConfigurationVerifier` takes a map of pinned signers, `CollectorApplication.TRUSTED_SIGNING_KEYS`. Empty is the shipped default: any correctly signed configuration is accepted and `signerAnchored` is @@ -154,7 +170,7 @@ key ID while carrying a different key. What this establishes is that the configuration is unchanged since it was signed. It does not establish who wrote it unless the build pins that signer; `SignerIdentity.fingerprint` (SHA-256 over -the encoded public key, first 16 bytes, as eight uppercase groups of four hex characters) is what a +the raw public key, first 16 bytes, as eight uppercase groups of four hex characters) is what a participant compares against what the research team published. See [`threat-model.md`](threat-model.md). @@ -163,6 +179,18 @@ bytes, so the same signature covers it. The signing private key and the HPKE pri separate purposes and must not be shared. A build embeds public keys only — and none at all unless it pins a signer; it contains no study private key. +### Immutable join import + +`adc://join/v1` is an immutable transport pointer, not remote configuration. Its fixed query binds +one canonical HTTPS artifact URL, the full envelope SHA-256, and the signer fingerprint. Kotlin and +TypeScript consume one shared corpus and enforce a narrow ASCII URL profile instead of accepting +different normalization from `URI` and WHATWG `URL`. Android handles only the exact exported +`VIEW` filter, rejects an active study before network I/O, performs one bounded non-redirecting GET, +and removes no-backup staging on startup and every outcome. The session then checks digest, +ordinary `ADCCFG01` signature / configuration rules, and fingerprint in that order. The host can +withhold bytes but cannot replace an accepted artifact, schedule a refresh, change collectors, or +assign a participant ID through the link. + ## 4. Study state machine ```mermaid @@ -231,6 +259,11 @@ others. | --- | --- | --- | | `app_lifecycle.v1` | This app's own Activity lifecycle | None | | `accelerometer.v1` | Raw x/y/z in m/s², sensor time, accuracy | Accelerometer hardware | +| `battery_state.v1` | Whole percentage, charging state/source, power-save state | None | +| `temporal_context.v1` | Time-zone ID, UTC offset, DST state, clock-change reason | None | +| `gyroscope.v1` | Raw x/y/z angular velocity in rad/s, sensor time, accuracy | Gyroscope hardware | +| `ambient_light.v1` | Raw illuminance in lux, sensor time, accuracy | Ambient-light hardware | +| `proximity.v1` | Raw distance/range and near/far interpretation | Proximity hardware | | `network_state.v1` | Default network availability, transport, validated, metered, roaming, VPN, optional bandwidth estimates | `ACCESS_NETWORK_STATE` (a manifest normal permission) | | `network_usage.v1` | Device-total Wi‑Fi and mobile rx/tx bytes and packets, plus the interval the query covers | Usage Access | | `usage_events.v1` | App resumed/paused/stopped, screen, keyguard, and startup/shutdown raw events, including the foreground app's `package_name` when the platform reports one | Usage Access | @@ -252,6 +285,8 @@ consent material. Each study uses one non-exportable Android Keystore AES-256-GCM key. The study ID is hashed with SHA-256 into an opaque file and key locator. All data lives under `noBackupFilesDir/experiments`. The manifest disables backup, and the cloud-backup and device-transfer rules exclude all app data. +The app neither requests StrongBox nor verifies hardware backing, so this is a Keystore isolation +claim, not an absolute hardware-protection claim. - Metadata: an `AtomicFile` in the format `ADCMET01 | random 96-bit IV | ciphertext+tag`. `ADCMET01` carries the fresh-per-import instance ID, optional researcher-assigned ID, upload watermark, @@ -271,8 +306,10 @@ The manifest disables backup, and the cloud-backup and device-transfer rules exc - Every event append is followed by an `fsync`; metadata is committed through `AtomicFile`. - An event plus its resulting metadata is one recoverable commit. Before appending, the store writes an encrypted `ADCTXN01` journal containing the resulting metadata before the event append. - Recovery either completes that exact commit or - recognizes it as already complete, then removes the journal. This is the one write path used for + If the journal is one boundary ahead and its event is durable, recovery authenticates that exact + tail event and commits the journal metadata; if the event is absent, it discards the prepared + journal. A same-boundary leftover is discarded with main metadata authoritative. Any other + boundary, malformed journal, or event mismatch fails closed. This is the one write path used for occurrence lifecycle events and survey submissions; there is no independent draft store. - The active signed configuration is held separately, under its own Keystore key, as `ADCACT01 | random 96-bit IV | ciphertext+tag`. @@ -293,21 +330,20 @@ or a missing key all fail closed. Nothing is skipped over. The sequence number sits unencrypted at the front of every frame, which is what lets both paths skip work without giving up a check. -`loadMetadata` walks the frames and decrypts none of them (`scanEvents(…, decryptPayloads = false)`). -Framing, the segment index, and sequence contiguity come from the plaintext headers, and the -surviving sequence range is what the metadata is reconciled against. `lastEvents` — the last event -each collector recorded — is read from the encrypted metadata, where it is persisted, rather than -rebuilt by scanning. Opening a study is therefore linear in frames, with no per-event crypto and no -per-event JSON parsing, which is what makes an 8 GiB quota usable. The consequence is stated in the -[threat model](threat-model.md): an event payload is authenticated when it is read, not when the -study is opened. The metadata's own AEAD tag, the framing, and the contiguity of the sequence are -all still checked at open. +`loadMetadata` normally walks the frames with `scanEvents(…, decryptPayloads = false)`. Framing, the +segment index, and sequence contiguity come from the plaintext headers, and the surviving sequence +range is what the metadata is reconciled against. Only the unique state where a one-boundary-ahead +journal and its complete event tail are both durable causes a second header walk and decrypts that +single tail event before committing its metadata. `lastEvents` — the last event each collector +recorded — is persisted inside encrypted metadata rather than rebuilt by scanning. Opening is thus +linear in frames with no per-event crypto or JSON parsing; normal opening decrypts no events and +crash recovery decrypts at most one. The consequence is stated in the [threat model](threat-model.md): +event payloads are authenticated when read, plus the one recovery tail when required. Metadata AEAD, +framing, and sequence contiguity are always checked at open. `readEvents` walks the same headers and decrypts only the frames at or above the requested start, -seeking past the rest. That matters beyond speed: an upload streams its bundle as it is generated, -so time spent decrypting events the window will discard is time the connection sits silent, and -that silence grows with the study's length until it outlasts the network. Measured on an emulator, -the same delivery took 90-149 s before this change and 31 s after. +seeking past the rest. That keeps manual export and durable upload staging proportional to the +requested retained window instead of repeatedly decrypting an already delivered prefix. ### Reclaiming delivered data @@ -355,65 +391,60 @@ nothing qualified. ## 8. Export and upload -Both paths produce the same bundle. An `ExportSnapshot` carries `fromSequence`, an optional -`toSequence`, and an optional `maximumPlaintextBytes`; a participant export runs from -`retainedFromSequence` to `nextSequenceNumber - 1` with no budget, so their copy is complete, and an -upload asks for everything after the last sequence an endpoint confirmed under a 16 MiB budget. The -runtime hands out a copy of the bounded `StudyMetadata` only. The exporter reads the requested range -from the store one event at a time and streams JSON → AES-GCM → destination. It never loads the full -event history, the full plaintext JSON, or the full ciphertext into memory. During `RUNNING`, later -events keep appending with larger sequence numbers, which is why no pause is required for either -path. - -No size failure can strand a study. The format previously enforced a 256 MiB plaintext ceiling by -throwing mid-stream, which under an 8 GiB quota was a permanent-failure state: once the undelivered -tail passed the ceiling, every upload regenerated the whole bundle and then threw, with no way -forward. The budget replaces that. A bundle stops cleanly at the first event boundary past the -budget — checked every `BUDGET_CHECK_STRIDE` = 256 events, so overshoot is bounded by one stride — -and always takes at least one event, so a run cannot make zero progress. The receipt reports where -it stopped, and the rest goes out on the next run. - -Export uses the Storage Access Framework's `CreateDocument` so the participant chooses the -destination. Upload writes the same stream into an OkHttp request body; because the bundle is -generated as it is written its length is unknown up front, so `contentLength()` is `-1` and OkHttp -falls back to chunked transfer encoding. The body is one-shot, so OkHttp cannot replay it once -sending has started; `retryOnConnectionFailure` is left at its default, which then only recovers a -pooled connection that turns out to be dead before anything was written — a common case between -widely spaced uploads. WorkManager owns retry beyond that. +Both paths use `ResearchExport` and the same authenticated document schema. A manual export reads +`[retainedFromSequence, nextSequenceNumber - 1]` and streams directly to the participant's Storage +Access Framework destination, so it may scale to the configured 8 GiB local quota. An automatic +upload first selects an exact non-empty window near a 16 MiB plaintext target, while enforcing a +32 MiB automatic-upload container ceiling. + +Automatic upload does not stream a newly generated request. `FileUploadOutbox` creates the complete +ciphertext in no-backup storage, flushes and atomically publishes it, then persists a bounded +recovery manifest with bundle ID, exact first/last sequence, event count, byte count, configuration +digest, ciphertext SHA-256, and an optional terminal code. It contains no participant, experiment, +or configuration ID. At most one entry exists. Recovery +accepts it only when manifest, length, digest, and outer framing agree. Process death, reboot, I/O retry, or lost response reuses +the same file byte-for-byte; a new bundle cannot supersede it until an exact receipt commits it. + +The HTTP body is therefore replayable, has fixed `Content-Length` and `Content-Digest`, and is never +chunked. Automatic redirects and OkHttp connection-level request replay are disabled; the outbox +and worker own retry semantics. Collection and later manual export can continue while the staged +file is pending. `ResearchExport.decrypt` streams as well, for the same reason: a bundle is bounded by the study's quota rather than by a fixed ceiling, so it can be larger than a researcher's machine wants to hold in memory. It drives the cipher directly rather than through `CipherInputStream`, which reports an AEAD failure as a normal end of stream and would turn a tampered bundle into a silently truncated -file. The trade-off is that plaintext reaches the caller before the tag is verified, which is why -`researcher-tools decrypt` writes to a temporary file in the destination directory and moves it into -place only after `decrypt` returns. +file. Plaintext therefore reaches only a mode-`0600` staging file before the tag is verified. +`researcher-tools decrypt` then streams that file through +[`ResearchBundleVerifier`](../core/export/src/main/kotlin/cool/linc/androiddatacollector/core/export/ResearchBundleVerifier.kt) +and publishes it +with an atomic move only after the authenticated document, signature, identities, ranges, +transitions, and catalog payloads all pass. Export format: ```text -ADCEXP01 | keyIdLength(u16) | wrappedKeyLength(u32) | nonce(12) | -researcherKeyId | TinkHPKEWrappedAESKey | AES-GCMCiphertext +ADCEXP01 | bundleId(16) | configurationSha256(32) | keyIdLength(u16) | +contentNonce(12) | researcherKeyId | HPKEWrappedContentKey(80) | AES-GCMCiphertext ``` -- Content: `research-bundle-v1` JSON containing the canonical configuration, the snapshot time, the - current state, the participant instance ID, optional assigned participant ID, all transitions, - the events in the window, and then - the bundle's own `first_sequence_number` and `last_sequence_number`. Those two sit *after* the - `events` array, because a budget decides where a bundle stops while it streams; declaring the - window up front would let a bundle claim a range it does not contain. A reader that needs the - window before the events has to buffer or make a second pass. +- Content: one closed-world JCS `research-bundle-v1` document containing the outer bundle identity, + manual/automatic kind, exact embedded configuration, configuration digest and original signature, + producer platform/build, snapshot time, full study metadata, transitions, and the exact contiguous + event window. Sequence, count, time, and client-build values are decimal strings. - Content key: a freshly generated AES-256 key and 96-bit nonce for every bundle. -- Key wrapping: Tink HPKE, `DHKEM_X25519_HKDF_SHA256 / HKDF_SHA256 / AES_256_GCM`. -- The context and AAD bind the format string, `experimentId`, `configurationId`, and - `researcherKeyId`. Because the format string is in the AAD, a reader built for another version - fails on the tag rather than misreading the bundle. -- Receipt: key ID, first sequence, sequence boundary, event count, byte count, and the SHA-256 of - the entire encrypted bundle. +- Key wrapping: RFC 9180 base mode with X25519/HKDF-SHA256/AES-256-GCM. The fixed 80-byte wire + value is `enc[32] || sealed_key[48]`; library-specific keysets or prefixes never appear. +- Context: exact JCS containing bundle format, bundle UUID, full configuration SHA-256, and + researcher key ID. It is HPKE `info` and document AES-GCM AAD. +- Validation: framing and bounds, HPKE, content AEAD, JCS, outer/inner identities, embedded + configuration digest/signature/platform/build, exact range/count, and catalog payloads all pass + before plaintext-derived output is published. A state can be exported any number of times, and each file uses a new random key. Repeated exports -normally overlap, so the research side should deduplicate on -`participant_instance_id + sequence_number`. In a study that has reclaimed space, an export +normally overlap, so the research side partitions by `(experiment_id, configuration_id)` and +deduplicates on `(participant_instance_id, sequence_number)`, treating different content at one +identity as a conflict. In a study that has reclaimed space, an export starts at the retained floor instead of at 1 and its `first_sequence_number` says so, which makes it a window over the events still on the device rather than the whole history. The wrong private key, the wrong configuration, or any tampering with the header or the ciphertext leaves the bundle @@ -421,11 +452,17 @@ undecryptable. Upload advances a durable watermark rather than repeating history. `StudyMetadata.uploadedThroughSequence` holds the highest sequence an endpoint confirmed; it starts at 0, advances only after a successful -response, and never moves backwards. There is no fixed chunk size. Each run asks for everything -outstanding — `[uploadedThroughSequence + 1, nextSequenceNumber - 1]` — the budget decides how much -of that fits, and the watermark follows `ExportReceipt.sequenceBoundary` rather than the range that -was planned. The configured interval is what paces delivery; the budget only binds while a backlog -is being worked off, and the next run picks up where the last one stopped. +receipt, and never moves backwards. Requests use `application/vnd.adc.research-bundle` plus bundle +UUID, format, configuration SHA-256, researcher key ID, exact from/to/count, length, and digest +headers. There are no clear participant, assigned, experiment, or configuration IDs; routing +metadata is explicitly untrusted. + +A new durable object succeeds only with `201 Created`; an exact replay succeeds only with `200 OK`. +Both return the same compact JCS receipt with exactly `bundle_id`, `byte_count`, +`configuration_sha256`, `event_count`, `first_sequence_number`, `last_sequence_number`, and +`sha256`. Every value must equal the outbox manifest before commit. Receipt loss is safe because an +exact replay returns the original receipt; a bundle-ID/content conflict is terminal rather than an +overwrite. The session lock is taken twice and briefly — once to compute the range, once to commit — with the HTTP transfer in between under a separate mutex, so an unresponsive endpoint cannot block the @@ -440,9 +477,12 @@ identifier rather than a message and validates it against the same `[A-Z][A-Z0-9 a collector's health reason, so nothing that reaches a screen or a log can hold study data. `OkHttpStudyUploader` classifies the transport failure into `UPLOAD_TIMEOUT`, `UPLOAD_HOST_UNRESOLVED`, `UPLOAD_CONNECT_REFUSED`, `UPLOAD_TLS_HANDSHAKE_FAILED`, -`UPLOAD_TLS_FAILED`, `UPLOAD_INTERRUPTED`, `UPLOAD_IO_FAILED`, or `UPLOAD_FAILED`, and a non-2xx -response becomes `UPLOAD_HTTP_`. The dashboard renders that code in place of the delivered -count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code the same way. +`UPLOAD_TLS_FAILED`, `UPLOAD_INTERRUPTED`, `UPLOAD_IO_FAILED`, or `UPLOAD_FAILED`, and an HTTP error +becomes `UPLOAD_HTTP_`. Only I/O, `408`, `425`, `429`, and `5xx` retry. Redirects, `202`, +every other `4xx`, malformed receipts, and mismatched receipts are terminal for the staged bundle, +without stopping collection or advancing the watermark. The dashboard renders that code in place +of the delivered count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code +the same way. ## 9. Background execution, interventions, and recovery @@ -455,15 +495,27 @@ count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code restored only when the persisted state was `RUNNING`. - Each intervention combines a reusable action with one or more triggers. Actions are localized notifications or localized native surveys. Triggers are one-time offsets, repeating intervals, - or daily local times; each declares whether elapsed study time means calendar time or active - collecting time. WorkManager timing is inexact and delivery can be late. + daily local times, or signed random local windows. Relative triggers declare whether elapsed + study time means calendar time or active collecting time. WorkManager timing is inexact and + delivery can be late. - `InterventionSchedulePlanner` derives every occurrence ID from configuration, intervention, - trigger, and logical schedule position. The ID is independent of current timezone and process - history. The durable occurrence record owns its scheduled instant, expiry, and lifecycle + trigger, and logical schedule position. Daily-local logical positions are stable indexes; + random-window positions include the current-zone local date selected for a not-yet-materialized + slot. Once materialized, an occurrence ID and instant are durable and independent of later zone + changes or process history. The durable occurrence record owns its scheduled instant, expiry, and lifecycle (`SCHEDULED`, `POSTING`, `NOTIFICATION_POSTED`, `OPENED`, `SURVEY_SUBMITTED`, `EXPIRED`). Recovery, boot, time changes, timezone changes, pause, and resume reconcile by that identity, so they do not enqueue a second logical occurrence. A configuration is bounded to 512 lifetime occurrences so this exact durable set remains inside the encrypted metadata ceiling. +- Prompt lifecycle mutations are accepted only in `RUNNING`. Pause cancels pending intervention + work and visible prompt notifications without freezing calendar time or signed availability; + resume reconciles the durable set, expires elapsed windows, and schedules only still-eligible + occurrences. Survey answers cannot be opened or submitted during the pause. +- Random-window triggers use a CSPRNG and persist the chosen instant before WorkManager receives + it. Restart and retry reuse that record. Time/time-zone changes leave materialized occurrences + unchanged and affect only future local dates. Caps truncate eligible slots in local-date planning + order, then signed window array order, then ordinal; randomness selects only the minute within a + selected slot. No server can trigger or redraw an occurrence. - A notification content intent carries only the exact occurrence ID. Opening resolves its signed action from durable state. Survey answers validate against stable survey/question/option IDs and commit as one immutable `SURVEY_SUBMITTED` event plus terminal occurrence state. Closing the UI @@ -487,20 +539,20 @@ count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code cancels interventions and the deadline but leaves delivery running, so a study that has ended still sends its undelivered tail. The chain is simply not renewed once `uploadDrained()` reports that a terminal study has nothing outstanding; deleting local data cancels it outright. -- A failed run returns `Result.retry()` rather than `failure()`: the usual cause is a network or - endpoint problem that resolves on its own, and the events remain durable on the device meanwhile. - A retry keeps the current link alive, so no successor is enqueued on that path. +- A retryable run returns `Result.retry()` and keeps the immutable outbox entry. A terminal protocol + or receipt failure is persisted explicitly and does not spin forever; collection remains + independent and the staged ciphertext is not acknowledged or reclaimed. ## 10. Security and privacy invariants - No unsigned study configuration is accepted, and none whose signature, canonicality, schema, - validity window, or app-version floor fails. A build that pins signers additionally refuses every + validity window, platform, or client-build floor fails. A build that pins signers additionally refuses every signer it does not list. - No dynamically downloaded collector, no parsing fallback, no legacy reader. - The current shape remains schema v1. Earlier prompt-shaped v1 configurations are rejected; there is no compatibility decoder or schema-version alias. -- The researcher-assigned ID is present only inside the signed configuration and encrypted bundle. - Upload routing exposes the random per-import instance ID but never the assigned ID. +- Automatic upload URLs and headers exclude participant, assigned, experiment, and configuration + IDs. Routing exposes only bundle-level claims and does not authenticate a participant or device. - No plaintext study file, no plaintext export scratch file, no secret key in a log. - Study data leaves the device only as an HPKE-wrapped bundle, and only to a destination the participant chose or to the endpoint the signed configuration names. No analytics, no crash @@ -511,11 +563,9 @@ count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code - Upload is decided by the signed configuration, so it is covered by the same signature, expiry, and canonicality checks as the collector set. There is no runtime toggle and no way to redirect a study to a different endpoint without a new signature and fresh consent. -- The upload watermark only advances, only on a confirmed response, and only as far as the receipt - says the bundle actually reached. A failed or hostile endpoint can delay delivery; it cannot cause - events to be skipped or re-sent as a gap. It can, by confirming a bundle it did not keep, make - those events reclaimable from the device under storage pressure — a 2xx is a statement that the - body is durably stored. +- The upload watermark advances only for a `201 Created` or exact-replay `200 OK` whose canonical + seven-member receipt matches the durable outbox manifest. `202`, redirects, generic `2xx`, and + malformed or mismatched receipts never commit or make events reclaimable. - Both public keys travel inside the signed configuration; the study signing and export private keys belong in neither the app nor a production repository. - Of the app's own components, only the launcher activity and the boot receiver and IME service that @@ -527,30 +577,30 @@ count, and a collector in `FAILED` or `BLOCKED_ACCESS` shows its own reason code ## 11. Verification scope -Automated tests cover the strict protocol and signature handling, the state machine, the admission -gate, runtime pause/resume and repeated export, HPKE and AES-GCM round trips with wrong-context and -wrong-suite rejection, and Android Keystore encrypted segmented storage. The eight protocol tests -include the self-certifying path with an empty anchor map, a populated map refusing every other -signer, a configuration that names a pinned key ID while carrying a substituted key, and an envelope -whose signer key ID disagrees with the one in the signed bytes. For upload they cover the -`upload` block round-tripping in both shapes and rejecting a partial or cleartext one, ranged bundles -carrying and declaring only their window, the watermark advancing on success and holding on failure, -a failure not masking a collection incident, an upload before collection starting as a no-op, a -finished study still delivering its backlog and then reporting itself drained, and a study without -an `upload` block never contacting an endpoint. - -Schedule tests cover calendar and active-time one-shots, intervals, daily local time across timezone -changes, restart reconstruction, terminal occurrences, and pause accounting. Runtime tests cover all +Protocol behavior is executable in the shared valid/hostile +[conformance corpus](../protocol/v1/conformance-vectors.json). The neighbouring Kotlin +[configuration](../core/protocol/src/test/kotlin/cool/linc/androiddatacollector/core/protocol/ConfigurationProtocolTest.kt) +and [bundle](../core/export/src/test/kotlin/cool/linc/androiddatacollector/core/export/ResearchExportTest.kt) +tests cover JCS, raw keys, fixed framing, signature provenance, RFC 9180 context, exact ranges, and +wrong-key/context/tamper rejection; TypeScript consumes the same corpus. Runtime tests cover the +state machine, admission barrier, repeated export, watermark commit, and encrypted segmented +storage. + +Schedule tests cover calendar and active-time one-shots, intervals, daily local time, durable CSPRNG +random windows across restart/time-zone/date-line changes, explicit DST gap/overlap resolution, +separation/caps, terminal occurrences, and pause accounting. Runtime tests cover all four survey question types, required/optional validation, stable IDs without labels, expiry, and concurrent submission proving exactly one immutable event. Identity tests cover distinct import instance IDs, assigned-ID persistence/export, upload-header exclusion, CLI bulk uniqueness, and cross-language canonical bytes. -The budget and streaming decryption have their own export tests: a budget stopping at an event -boundary with the receipt naming that boundary and the bundle declaring the window it actually -holds, a snapshot with no budget sending everything it was asked for, and a one-byte change to a -finished bundle raising `AEADBadTagException` from the chunked decrypt path rather than yielding a -short file. +Upload reliability has focused tests for the +[single-entry outbox](../app/src/test/kotlin/cool/linc/androiddatacollector/platform/FileUploadOutboxTest.kt) +and [HTTP adapter](../app/src/test/kotlin/cool/linc/androiddatacollector/platform/OkHttpStudyUploaderTest.kt): +recovery, exact byte replay, digest/length/range identity, redirect refusal, retry classification, +`201`/exact-replay `200`, generic-`2xx` rejection, and exact seven-field receipt matching. Export +tests separately verify streaming manual decryption publishes no successful output after AEAD +failure. Reclaiming is covered on both sides of the split. `EvictionPlanner`'s rules have JVM tests: oldest delivered segments first, a study under its target keeping everything, undelivered events blocking a @@ -559,12 +609,9 @@ adds instrumentation tests, on real Android Keystore, for segment rollover, recl from the new floor, appending after a reclaim without reusing a sequence, reclaimed events no longer being readable, and a missing prefix that was not reclaimed refusing to open. -These areas are not covered by tests today, and should be read as unverified rather than as working: -trailing-frame recovery in the encrypted store, tampering with an `ADCEXP01` *header* rather than its -ciphertext, the withdraw path through `ExperimentRuntime` and `StudySessionManager`, and the two -Android-side upload classes — `OkHttpStudyUploader` and `UploadWorker` — whose request shaping, -failure classification, status-code handling, chain renewal, and WorkManager constraints have no test -of their own. +Collector admission has two complementary checks. The runtime enforces each descriptor's +`maximumEncodedEventBytes` before append, while CI executes the source, bytecode, and dependency +capability policy. The instrumentation test defines the full Compose participation flow: importing the demo study under the shipped empty anchor map, the study step, a Continue through the data step, consent, access @@ -578,6 +625,14 @@ the signed envelope nor its loader, so the entry point the test drives does not scrolls to the export control but does not perform an export. It has to actually run on an emulator or a device; assembling the test APK is not a device-test pass. +Two narrower Android regressions sit beside that UI flow. `AndroidConfigurationImportTest` +proves raw-key Ed25519 demo import on Android itself, so a JCA provider-order +regression cannot hide behind JVM-only protocol tests. `P2CollectorEmulatorTest` creates the five P2 +plugins against real Android broadcast and `SensorManager` surfaces, validates every emitted draft +against its Protocol v1 descriptor, and checks pause/resume/stop boundaries. It skips when the test +device lacks gyro, light, or proximity hardware; its explicit `p2SyntheticInputs=true` mode requires +host-side emulator injection and checks the fixed readings documented in the root README. + Before real recruitment, a study still needs study-specific testing on the target physical devices and OEMs: permissions, background restrictions, battery, storage volume, location accuracy, Usage Access, the study keyboard, and long-duration stress. Passing on an emulator is not IRB or ethics diff --git a/docs/threat-model.md b/docs/threat-model.md index 708d2f7..50b17ff 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -2,6 +2,12 @@ A reference description of the protections in the current release, the limitations that come with them, and a few checks anyone can run. It is written to be attached to an ethics submission or read by a security reviewer, and it describes the system as implemented in this repository. Where a protection is weaker than it looks, that is stated here. +The [normative Protocol v1 contract](../protocol/v1/README.md), its +[collector catalog](../protocol/v1/collector-catalog.json), the +[implementation decision record](p0-p2-implementation-contract.md), and +[Collector capability policy](../assurance/README.md) are the implementation sources behind the +claims in this document. + ## What is being protected | Asset | Primary concern | @@ -9,6 +15,8 @@ A reference description of the protections in the current release, the limitatio | Study events on the device | Confidentiality while the phone is out of the participant's hands | | The exported or uploaded bundle | Confidentiality in transit and at the destination | | The study configuration | Integrity — a participant gets exactly the study they consented to | +| Researcher Ed25519 and HPKE private keys | Preventing forged studies and unauthorized bundle decryption | +| Receiver R2/S3 credentials and ciphertext objects | Durable, bounded custody without exposing a decrypt path | | Assigned and random participant codes | Confidentiality and controlled linkability to a research roster or import | | Survey answers and intervention history | Atomicity, immutability, and truthful lifecycle interpretation | | The participant's control | That start, pause, withdrawal, and deletion mean what they say | @@ -36,11 +44,20 @@ One detail matters for review: the app does **not** request StrongBox and does * ### Study configuration integrity +The researcher Web tool is a static client-side application. It generates or imports raw private +keys in the browser, keeps the draft and keys only in memory, downloads private-key files directly, +and has no analytics or application network destination. That reduces the number of places a key +can land; it does not make the browser a trusted hardware boundary. A malicious extension, +compromised same-origin deployment, or injected dependency can read keys while the page is open or +substitute what is signed. High-risk deployments should use the CLI in a controlled environment, +archive the canonical configuration beside the key material, and verify the resulting signer +fingerprint through an independent recruitment channel. + A configuration is Ed25519-signed inside an `ADCCFG01` envelope. The signing public key travels inside the signed bytes, in a mandatory `signer` block, so a configuration certifies itself and one published app can verify any researcher's study without a rebuild. **What a signature proves is that the configuration is unchanged since it was signed.** It does not prove who wrote it. A verified configuration establishes that identity mode, collectors, localized surveys, intervention actions and triggers, duration, consent, export key, and `upload` block are exactly the bytes the signer produced. It establishes nothing about the identity behind that key unless the build pins that signer. -On import the app checks envelope framing and length bounds, decodes the configuration strictly, requires the declared `signer.key_id` to equal the envelope's signer key ID, verifies the signature over the canonical configuration bytes, and checks the validity window and minimum app version. Nothing decoded is acted on until the signature verifies. Canonicality is enforced by re-encoding the decoded configuration and requiring a byte-identical match, so reordered keys, altered whitespace, duplicate keys, and reformatted numbers are rejected. Every object has an exact required key set — unknown *and* missing keys both fail — and an unknown collector ID fails even if the collector is marked optional. The current shape deliberately remains schema v1; prompt-shaped older v1 documents fail instead of entering a compatibility branch. +On import the app bounds the fixed `ADCCFG01` framing, strictly decodes and byte-for-byte rechecks RFC 8785 JCS, requires the declared `signer.key_id` to equal the envelope key ID, verifies the fixed 64-byte signature over the exact configuration bytes, and checks validity, Android platform, and the decimal-string client build floor. Ed25519 and X25519 wire keys must be raw 32-byte values encoded as unpadded base64url; X.509, PKCS#8, padded base64, and library keysets are rejected. Nothing decoded is acted on until the signature verifies. Every object has an exact key set and an unknown collector fails even when optional. Protocol v1 is a destructive pre-1.0 replacement: former-v1 artifacts fail rather than entering a compatibility branch. A build may additionally pin signers, as `CollectorApplication.TRUSTED_SIGNING_KEYS`. That map is empty in the shipped build, which therefore accepts any correctly signed configuration and reports the publisher as unverified to the participant. A non-empty map is strictly exclusive: only listed signers are accepted, and the pinned key overrides the one the configuration declares and must equal it, so a configuration cannot claim a pinned key ID while carrying a different key. `ConfigurationVerifier` returns both the configuration and whether its signer was pinned, and the consent screen renders that distinction. @@ -48,9 +65,27 @@ Failures are fail-closed: a failed import does not activate a study, and a faile ### Scope of collection -Collectors are selected by ID from a registry of modules compiled into the APK: no plugin download, no scripting layer, no dynamic class loading. Every collector parameter has a validated range in the schema, study duration is bounded, and the local quota a study may claim is bounded to 8 MiB-8 GiB. The module graph enforces the boundary structurally — a `collector:*` module depends only on `core:collector-api` and `core:study-definition`, so a collector cannot write files, change study state, start activities, or request permissions, because the code that would is not on its classpath. - -**What the participant is told about that scope is the app's text, not the researcher's.** Setup is five steps with one panel each — study, data, consent, access, start — and the data step, which comes before the consent text, lists every collector the signed configuration enables. Each entry is described from a template compiled into the app and filled in from that study's own signed parameters: the accelerometer's rate, the poll interval of each polling collector, the location interval and minimum displacement. No configuration field changes any of it, so a researcher cannot understate what a collector captures on the screen a participant reads immediately before consenting. This is a small integrity property rather than a large one, and its limits are worth stating exactly: it constrains how each enabled source is described, not the honesty of the consent summary beside it, which remains the researcher's own prose — and it is entirely positive. The screen says what each source records; it makes no claim about what a source cannot see. A participant who wants that has the documentation and their research team, not this screen. +Collectors are selected by ID from modules compiled into the APK: no plugin download, scripting, +or dynamic loading. Every parameter has a validated range. The module graph enforces the boundary: +a feature collector depends on `core:collector-api`, `core:study-definition`, and optionally the +narrow `collector:sensor-common` helper, so storage, protocol/export, study-state, and UI code are +not on its classpath. + +That boundary is also checked rather than merely documented. CI scans collector source, compiled +constants, and dependency graphs for network, files/database/preferences, dynamic loading, +logging, activity/service launch, protocol/export/storage, and cryptographic capabilities. The +runtime validates catalog identity/schema and enforces `maximumEncodedEventBytes` before append. A +new collector also needs catalog metadata, disclosure, bounds, lifecycle/access tests, and power +and storage estimates. + +**What the participant is told about that scope is the app's text, not the researcher's.** Setup is +five steps with one panel each — study, data, consent, access, start — and the data step, which comes +before the consent text, lists every collector the signed configuration enables. Each entry is +described from a template compiled into the app and filled in from signed parameters: motion-sensor +rates, ambient-light/proximity gates, polling intervals, and location interval/displacement. No +configuration field changes the wording, so a researcher cannot understate what a collector +captures on the screen a participant reads immediately before consenting. This integrity property +constrains each enabled source's description, not the honesty of the researcher's consent prose. One template hedges on purpose. The accelerometer entry reads "about N times per second **or more**", because Android treats a sampling period as a hint rather than a contract and a device is free to deliver faster than the study asked for — observed on a current emulator image at over ten times the requested rate. Stating the configured rate alone would understate what is recorded. @@ -58,9 +93,32 @@ Every participant-facing app string lives in resources and ships in English and ### Identity and survey integrity -Every import mints a new random UUID, even when the same configuration is imported twice. A personalized configuration may also contain one opaque assigned code restricted to a small ASCII grammar. The consent screen distinguishes the two modes and shows the assigned code for comparison. Both codes live in encrypted metadata and exports; only the random per-import UUID is allowed onto the clear upload-routing surface. - -An intervention occurrence is keyed by a deterministic SHA-256 identity over its signed logical schedule position. Its durable state distinguishes scheduled, notification posted, opened, submitted, and expired; recovery and timezone reconciliation use that identity instead of generating a new occurrence. Survey submission validates stable question and option IDs, then uses an encrypted transaction journal to commit one event and the corresponding terminal metadata together. There is no draft store and no update path after submission. These controls prevent duplicate commits and partial durable answers; they do not prove that a participant saw a notification or personally supplied an answer. +Every import mints a new random UUID, even when the same configuration is imported twice. A personalized configuration may also contain one opaque assigned code restricted to a small ASCII grammar. The consent screen distinguishes the two modes and shows the assigned code for comparison. Both appear in encrypted metadata and bundles. Clear upload URLs and headers contain neither one, nor the experiment or configuration ID; they are not participant or device authentication. + +An optional `adc://join/v1` link adds an untrusted HTTPS host only as artifact transport. The link +binds the complete artifact SHA-256 and signer fingerprint; Android disables redirects and implicit +retry, verifies the digest before the normal signature flow, and never polls for updates. A hostile +host can deny service, but changing bytes fails before consent and cannot alter an accepted study. +Personalized authoring requires a random opaque path and rejects the assigned ID in the URL. The +URL still reaches the host and may appear in infrastructure logs, so researchers must not encode a +roster identifier into its path or operational labels. + +The link controls one unauthenticated HTTPS `GET`. Redirects and connection retries are disabled, +no HTTP cookies or participant credentials are attached, the response is tightly bounded, and +no response becomes a study without the pinned digest, signature, and fingerprint. The parser does +not resolve the DNS name itself or filter the address to which DNS maps it, however. Opening an +untrusted join link can therefore make the participant's device contact an HTTPS service reachable +from its current network even though the app neither reveals that response nor accepts it as a +configuration. Join links should be treated as recruitment links, not as harmless display text. + +An intervention occurrence is keyed by a deterministic SHA-256 identity over its signed logical +schedule position. A random-window trigger selects its instant locally with a CSPRNG and persists +the occurrence before scheduling; retry/reboot do not redraw it, time-zone reconciliation only +plans future local dates, and there is no server trigger. Durable state distinguishes scheduled, +notification posted, opened, submitted, and expired. Survey submission validates stable question +and option IDs, then uses an encrypted transaction journal to commit one event and terminal +metadata together. These controls prevent duplicate commits and partial durable answers; they do +not prove that a participant saw a notification or personally supplied an answer. The configuration admits at most 512 lifetime occurrences. This is a security and reliability bound, not an authoring suggestion: retaining every terminal identity is what prevents an old logical firing from reappearing after recovery, and the bound keeps that set under the authenticated metadata limit instead of silently weakening idempotency. @@ -70,23 +128,32 @@ Study data leaves the device two ways, and both carry the same encrypted bundle. The first is export: the participant picks a Storage Access Framework destination and the app writes the bundle there. The second is upload, which exists only if the signed configuration carries a populated `upload` block naming an `https://` endpoint, an interval, and whether metered networks are allowed. The app then posts bundles to that endpoint and to no other. A study whose `upload` block is empty transmits nothing. Which of the two applies is fixed by the configuration the participant consented to, and the endpoint host, the cadence, and the network condition are rendered into the consent step from the signed bytes rather than from the researcher's free-text summary — as a block the app asserts itself, directly below that summary and on the same panel as it. The manifest declares `android.permission.INTERNET`, so a build's permission list no longer distinguishes an uploading study from a non-uploading one. The signed configuration is what a reviewer should read, and the checks at the end of this document cover both. -The bundle itself is encrypted with a fresh AES-256 key per bundle, wrapped with Tink HPKE (`DHKEM_X25519_HKDF_SHA256` / `HKDF_SHA256` / `AES_256_GCM`) to the researcher public key carried in the signed configuration. The keyset is validated at configuration import — exactly one enabled primary key, with KEM, KDF, AEAD, and variant each checked — so a study with a malformed or downgraded export key is rejected before any data is collected. The app never holds the researcher's private key, and a bundle streams from the store through AES-GCM to its destination without a plaintext or full-ciphertext temporary file. +Each bundle has a fresh AES-256 content key and nonce. RFC 9180 base-mode X25519/HKDF-SHA256/AES-256-GCM HPKE wraps that key to the raw researcher public key in the signed configuration; the wire value is fixed and contains no Tink keyset or prefix. Exact JCS context binds the bundle format, bundle UUID, full configuration SHA-256, and researcher key ID into HPKE and document AEAD. The authenticated document embeds the exact configuration, its original Ed25519 signature, producer platform/build, and exact range/count. The app never holds the researcher private key. + +A participant-directed export streams ciphertext to the chosen destination. Automatic upload is +different by design: before HTTP, the app durably stages one complete ciphertext bundle and a +bounded recovery manifest in no-backup storage. The bundle contains no plaintext. Only one entry exists, +and every retry after process death, reboot, timeout, or response loss sends exactly those bytes. +The manifest contains only bounded bundle/receipt bookkeeping and an optional terminal code—no +participant, assigned, experiment, or configuration ID—and is never placed in an HTTP request. -Decryption on the researcher's machine streams too, because a bundle is bounded by the study's storage quota rather than by a fixed size. One detail is deliberate and worth a reviewer's attention: `ResearchExport.decrypt` drives the AES-GCM cipher directly rather than reading through `CipherInputStream`, which reports an authentication failure as an ordinary end of stream. Through `CipherInputStream` a tampered bundle would decrypt into a silently truncated file that looks like a short study; driving the cipher directly keeps a bad tag an exception. The trade-off is that plaintext reaches the caller before the tag has been verified, so `researcher-tools decrypt` writes to a temporary file in the destination directory and moves it into place only after verification succeeds. Anything else consuming this API inherits the same obligation: do not publish the output until the call returns normally. +Decryption on the researcher's machine streams too, because a bundle is bounded by the study's storage quota rather than by a fixed size. One detail is deliberate and worth a reviewer's attention: `ResearchExport.decrypt` drives the AES-GCM cipher directly rather than reading through `CipherInputStream`, which reports an authentication failure as an ordinary end of stream. Through `CipherInputStream` a tampered bundle would decrypt into a silently truncated file that looks like a short study; driving the cipher directly keeps a bad tag an exception. Plaintext reaches only a mode-`0600` staging file before the tag is verified. The CLI then streams staging through `ResearchBundleVerifier`, which rechecks canonical bytes, embedded configuration and signature, identities, ranges, transition history, and catalog event contracts. It atomically publishes the file only when both phases pass and deletes staging on any failure. Anything else consuming the lower-level decrypt API inherits the same obligation: authenticate and validate the complete document before publishing plaintext. What an upload endpoint therefore sees: | The endpoint learns | The endpoint does not learn | | --- | --- | -| That this install is participating, and when each delivery arrives | Any event content; the body is ciphertext only the researcher's HPKE private key opens | -| How much data was collected, from the body size and the declared sequence range | Anything derived from the payload without that private key | -| The `experiment_id`, `configuration_id`, and random participant instance ID, sent in request headers | The assigned participant ID, survey content, event content, name, account, device identifier, or advertising ID | +| A ciphertext bundle UUID, receive time, body size/digest, configuration digest, researcher key ID, and claimed range/count | Event content; the body is ciphertext only the researcher private key opens | +| That the same bundle UUID was replayed, and whether its bytes/metadata match | The participant instance ID, assigned ID, experiment ID, configuration ID, name, account, device identifier, or advertising ID | +| A stable configuration digest may link bundles from one issued artifact or cohort | Whether a submission came from an enrolled participant or genuine device | -The participant instance ID is a random UUID minted for every import and kept in that study's metadata. An uploading study needs it because encrypted chunks otherwise arrive indistinguishable. It is pseudonymous and disclosed on the consent screen. A researcher-assigned ID, when present, stays inside HPKE ciphertext and must not be copied into endpoint headers or logs. +All clear headers are untrusted routing claims. Receiver ingestion identity is the immutable bundle +UUID plus exact bytes and metadata, never a participant/range pair. The participant UUID and any +assigned code become linkable only after authorized decryption and remain personal data there. Transport is TLS: the endpoint must be `https://`, validated when the configuration is decoded, and `usesCleartextTraffic="false"` remains set, so a plaintext HTTP endpoint cannot be configured or reached. There is **no certificate pinning**. The connection trusts the device's system trust store, so an attacker holding a certificate that store accepts — an enterprise or otherwise installed CA, for example — can see the delivery metadata above and can substitute their own endpoint. They still cannot read a bundle. -Delivery is durable rather than best-effort, and this is deliberate: `StudyMetadata.uploadedThroughSequence` records the highest sequence an endpoint confirmed, advances only after a successful response, and never moves backwards. It advances to what the bundle's receipt says was actually written, not to what the run set out to send, so a delivery that stopped early at its size budget leaves the remainder marked undelivered. A study that fails to upload keeps collecting, and the participant can still export by hand. +Delivery is durable rather than best-effort. A new durable object returns `201 Created`; an exact replay returns `200 OK` with the original canonical seven-field receipt. The client requires its bundle ID, byte count, configuration digest, event count, exact first/last sequence, and SHA-256 to match the outbox manifest before it advances `uploadedThroughSequence`. Redirects, `202`, other `2xx`, malformed receipts, and mismatches do not commit. Only I/O, `408`, `425`, `429`, and `5xx` retry; other protocol and `4xx` failures are terminal for that staged bundle but do not stop collection. The participant can still export by hand. A failed delivery is reported to the participant as a fixed reason code — `UPLOAD_TIMEOUT`, `UPLOAD_TLS_FAILED`, `UPLOAD_HTTP_`, `UPLOAD_IO_FAILED` and a few others. `StudyUploadException` carries the code instead of a message and validates it against the same `[A-Z][A-Z0-9_]{2,63}` pattern a collector's health reason uses, so a string that reaches the screen or the log cannot carry study data, an endpoint's response body, or a URL. The underlying transport exception is written to the Android log, where it is a network error rather than research data. Collector health failures surface their reason code the same way. @@ -98,11 +165,11 @@ Entering `RUNNING` mints an admission epoch token that collectors must present, A storage write failure or exhausted quota force-closes the admission gate, records an incident code, and fail-closes the study to `PAUSED`. There is no ring buffer and no silent dropping of events, and reclaiming space is a different thing from either: it can only release events an endpoint has already confirmed receiving, so a quota that fills with nothing delivered stops the study rather than making room, and what was released is recorded in `retainedFromSequence`, declared in every bundle's `first_sequence_number`, and stated on the participant's dashboard. Nothing that has not reached the research team is ever discarded to free space. -Occurrence lifecycle events and survey submissions have a stronger two-record boundary: the encrypted `ADCTXN01` journal makes the event append and updated metadata recoverable as one idempotent commit. Recovery completes the exact pending transaction or recognizes it as already complete. No unverified fallback reconstructs a response from UI state. +Occurrence lifecycle events and survey submissions have a stronger two-record boundary: the encrypted `ADCTXN01` journal makes the event append and updated metadata recoverable as one idempotent commit. Recovery applies a one-boundary-ahead journal only after authenticating an exactly matching durable tail event; it discards a prepared journal whose event is absent, and treats a same-boundary journal as a stale leftover with main metadata authoritative. Any other boundary or content mismatch fails closed. No unverified fallback reconstructs a response from UI state. -A corrupt segment, index gap, AEAD failure, or missing key is a hard failure, and only an incomplete trailing frame in the final segment may be recovered. Event segments missing *below* the retained floor are a hard failure too, because a prefix that disappeared without being reclaimed is indistinguishable from one that was tampered away. Metadata claiming more events than are durable is rejected in favour of the durable count, and an export that cannot read its whole window to the boundary fails rather than producing a partial file. Missing required access keeps a study from reaching `READY`, and a foreground service that fails to start rolls the runtime back instead of collecting. A dataset is therefore either complete over the window it declares or absent, rather than quietly partial. +A corrupt segment, index gap, AEAD failure, or missing key is a hard failure, and only an incomplete trailing frame in the final segment may be recovered. Event segments missing *below* the retained floor are a hard failure too, because a prefix that disappeared without being reclaimed is indistinguishable from one that was tampered away. Main metadata must name the durable tail unless the authenticated one-event journal proves the single permitted append-recovery state; no durable count is guessed or rebuilt as fallback. An export that cannot read its whole window to the boundary fails rather than producing a partial file. Missing required access keeps a study from reaching `READY`, and a foreground service that fails to start rolls the runtime back instead of collecting. A dataset is therefore either complete over the window it declares or absent, rather than quietly partial. -**When an event payload is authenticated, and when it is not.** Opening a study decrypts no events. The sequence number is stored unencrypted at the front of each frame, so the framing, the segment index, and the contiguity of the sequence are checked from the plaintext headers, and the metadata — which holds each collector's last event — is verified by its own AES-GCM tag. This is what makes a large quota workable: the cost of opening a study is linear in the number of frames rather than in the bytes decrypted. The trade-off is direct. **An event payload's authentication tag is verified when that event is read, not when the study is opened.** Corruption or tampering inside an event body surfaces on export or upload, as a hard failure at that point, rather than at startup. Nothing is accepted unverified — a tampered event still cannot reach a bundle — but the detection is deferred, so a device holding a damaged log can look healthy until its data is next read. +**When an event payload is authenticated, and when it is not.** Normal study opening decrypts no events. The sequence number is stored unencrypted at the front of each frame, so framing, segment index, and contiguity are checked from plaintext headers, while metadata — including each collector's last event — has its own AES-GCM tag. The only exception is the unique crash state where a one-boundary-ahead journal and a complete event tail are both durable: opening then decrypts and authenticates exactly that tail before applying the journal. This keeps open cost linear in frames with no per-event crypto; recovery adds at most one event decrypt. **An event payload's authentication tag is otherwise verified when that event is read.** Corruption or tampering inside another event body surfaces on export or upload as a hard failure at that point. Nothing is accepted unverified — a tampered event still cannot reach a bundle — but detection for non-recovery-tail events is deferred, so a device holding a damaged log can look healthy until its data is next read. ### Deletion @@ -122,19 +189,28 @@ One narrow part of this is closed by the data step described under *Scope of col **Publisher impersonation.** The researcher name and contact shown on the consent screen come from the signed configuration, which makes them text the signer chose. Anyone can generate an Ed25519 key, write a configuration naming any research team, sign it, and produce a file that verifies on a build with no pinned signers. The signature is genuine; what it certifies is the file, not its author. -Three things narrow this. The consent step shows the key fingerprint — SHA-256 over the encoded public key, first 16 bytes, as eight uppercase groups of four hex characters — under the heading *Configuration signature*, and, when the signer is not pinned, asks the participant to check that fingerprint against the one their research team published, noting underneath that a signature shows a file is unaltered rather than who wrote it. Deliberately none of it is in the error colour. An unpinned signer is the deployment model, not a failure, and rendering the ordinary case as an alarm trains a reader to skip the block — the mitigation here depends on the participant actually performing a comparison, so the text is an instruction rather than a warning. That wording is in the app's string resources rather than in Kotlin, so it is translated with the rest of the interface and no configuration can alter it. A team that publishes its fingerprint through the channel that recruited its participants gives them a check that copied prose does not defeat. And a configuration is not an anonymous download: it reaches a participant through a relationship that already exists, so an impersonator has to get their file in front of someone through that channel. +Three things narrow this. The consent step shows the key fingerprint — SHA-256 over the raw 32-byte public key, first 16 bytes, as eight uppercase groups of four hex characters — under the heading *Configuration signature*, and, when the signer is not pinned, asks the participant to check that fingerprint against the one their research team published, noting underneath that a signature shows a file is unaltered rather than who wrote it. Deliberately none of it is in the error colour. An unpinned signer is the deployment model, not a failure, and rendering the ordinary case as an alarm trains a reader to skip the block — the mitigation here depends on the participant actually performing a comparison, so the text is an instruction rather than a warning. That wording is in the app's string resources rather than in Kotlin, so it is translated with the rest of the interface and no configuration can alter it. A team that publishes its fingerprint through the channel that recruited its participants gives them a check that copied prose does not defeat. And a configuration is not an anonymous download: it reaches a participant through a relationship that already exists, so an impersonator has to get their file in front of someone through that channel. A build that pins its signers removes this exposure for the studies it accepts, and accepts nothing else. See the [researcher guide](researcher-guide.md) for both sides of that choice. **What happens to a bundle after it leaves.** Once the participant picks a destination, or the app posts a bundle to the study's endpoint, the bytes are beyond the app's reach. Confidentiality holds — only the researcher's HPKE private key opens them — with three caveats: -- A bundle is not authenticated as to origin. It is encrypted *to* the researcher, not signed *by* the device. Anyone with the researcher's public keyset, which is inside every copy of the signed configuration, can fabricate a syntactically valid bundle that decrypts cleanly. A decryptable bundle is not proof of who produced it, and an endpoint receiving one has no cryptographic evidence that a real participant device sent it. +- A bundle is not authenticated as to origin. It is encrypted *to* the researcher, not signed *by* the device. Anyone with the raw researcher public key, which is inside every signed configuration, can fabricate a syntactically valid bundle that decrypts cleanly. A decryptable bundle is not proof of who produced it, and an endpoint receiving one has no cryptographic evidence that a real participant device sent it. - The bundle header exposes the researcher key ID in cleartext, and the suggested export filename contains the study ID and an export timestamp. Anyone handling the file can tell that this person participated in that study. - Size discloses roughly how much data was collected. The same is true on the device: event segment file sizes and modification times leak collection volume and timing to anyone with filesystem read access, without any decryption. -**A compromised or hostile upload endpoint.** An endpoint that is taken over, misconfigured, or logging more than the study intended still cannot read a bundle without the researcher's HPKE private key, which does not belong on a collection server. What it does get is the metadata above for every delivery: which install, which study, how many events, and when. Whoever operates the endpoint can therefore build a participation timeline per instance ID even while the payloads stay closed. It can also refuse deliveries indefinitely; the effect is that the device keeps the data and the researcher does not receive it, not that collection stops. The converse matters more: an endpoint that answers 2xx without durably storing the body advances the watermark, and under storage pressure the device may then release those events. A success response is a claim to have stored the bundle, and an endpoint that cannot honour it should answer 408, 429, or 5xx instead. Treat the endpoint as part of the study's data governance, keep the decryption key off it, and state its operator in the consent material. +**A compromised or hostile upload endpoint.** An endpoint that is taken over, misconfigured, or logging more than intended still cannot read a bundle without the researcher private key. It does learn the untrusted bundle metadata above, including a stable configuration digest that can link submissions from the same issued artifact. It can refuse delivery indefinitely; the device retains the data and collection continues. Conversely, an endpoint can fabricate a matching seven-field `201`/`200` receipt without keeping the body. That can advance the watermark and eventually make those events reclaimable under storage pressure. Receipt matching makes response loss and accidental mismatch safe; it cannot prove remote durability against the server itself. Treat the endpoint as study infrastructure, keep the decryption key off it, minimize logs, and state its operator in consent material. + +The receiver ingress has no participant authentication or device attestation. An attacker can submit +bounded bogus ciphertext and consume storage. Deployment-time configuration-digest/key allowlists, +the 32 MiB body limit, WAF/rate limits, create-only object writes, and R2 lifecycle policy bound the +cost; they do not establish origin. -**Inference from the data itself.** That location traces, keyboard touch dynamics, and app usage patterns can identify a person is a property of the data, not a defect in the software. Minimisation and consent are the controls. The keyboard collector cannot see text, but its timing and within-key position data are behaviourally distinctive; the [data dictionary](data-dictionary.md) states this per collector. +**Inference from the data itself.** That location traces, motion sensors, temporal context, +keyboard touch dynamics, and app usage patterns can identify a person is a property of the data, +not a defect in the software. Minimisation and consent are the controls. The keyboard collector +cannot see text, but its timing and within-key position data are behaviourally distinctive; the +[data dictionary](data-dictionary.md) states this per collector. **Assigned IDs and survey responses are direct governance responsibilities.** An opaque assigned code can still be identifying to the team that holds its roster, and free-text survey answers can contain names or other sensitive details. Bulk personalization keeps codes out of filenames and logs, and transport keeps them out of headers, but decryption intentionally reveals them to the private-key holder. Ethics review should minimize free text, document the roster join and retention policy, and state that closing an unfinished survey stores no answer while submission is final. @@ -144,7 +220,7 @@ A build that pins its signers removes this exposure for the studies it accepts, **No signer revocation.** There is no revocation list, rotation protocol, or kill switch at any layer. A leaked study signing key can mint configurations that any build with an empty anchor map accepts, and configurations already signed with it stay valid until they expire; a short validity window is the only control. Where a build does pin signers, that set is fixed and auditable at build time, and retiring one of those keys requires shipping a new APK. -**Key loss.** Losing the researcher's HPKE private key makes every export from that study permanently unreadable. There is no escrow, and the keyset validation mandates exactly one key, so multi-recipient encryption is not available. Losing the device's Keystore key — through device wipe, uninstall, or clearing app data — destroys all un-exported local data. Neither case has a recovery path. +**Key loss.** Losing the researcher HPKE private key makes every export for that configuration permanently unreadable. There is no escrow and Protocol v1 names exactly one raw recipient key, so multi-recipient encryption is unavailable. Losing the device's Keystore key — through device wipe, uninstall, or clearing app data — destroys all un-exported local data. Neither case has a recovery path. **Per-collector supervision is not fail-closed.** A collector that crashes or loses its permission is marked `FAILED` or `BLOCKED_ACCESS`, and the study continues with the others. This keeps one flaky data source from ending someone's participation, but a dataset can be missing one collector's data for a period while the study looks healthy overall. The collector's status and the resulting gap are visible in the data. @@ -162,7 +238,9 @@ What a real deployment owes participants is its own key pairs and a published si The dependency set is small: Tink for cryptography, Gson for parsing, AndroidX Compose, Lifecycle and WorkManager, Google Play Services Location for the fused location provider, and OkHttp 5.3.0 for the upload request. There is no analytics, crash reporter, or telemetry library. OkHttp is used from one class, `OkHttpStudyUploader`, which builds a single POST to the endpoint the signed configuration names; Play Services Location is the only other dependency with a plausible network surface. -The release runtime classpath resolves to 136 Maven components. On 2026-08-03, the exact Gradle-resolved graph was exported as a CycloneDX SBOM and scanned with Trivy 0.70.0 against vulnerability and Java databases downloaded that day; no known vulnerability at any severity was found. A separate source and `pnpm-lock.yaml` scan found no secret or known vulnerability. These are point-in-time results rather than a continuing guarantee, so re-run both scans before any deployment. +Gradle and package lockfiles make the selected dependency graph reviewable. Vulnerability and +secret-scan results are point-in-time observations rather than a continuing guarantee, so re-run +the relevant scans before deployment. ## Verifying this yourself From 62ae231566b4b93eb1df7eae32d09053aa35e131 Mon Sep 17 00:00:00 2001 From: JacobLinCool Date: Wed, 5 Aug 2026 01:35:53 +0800 Subject: [PATCH 4/4] Prepare v1.0.0-rc.3 --- CITATION.cff | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CITATION.cff b/CITATION.cff index 7bec066..fff8d28 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -29,5 +29,5 @@ keywords: - android license: MIT # TODO: update version and date-released at each tagged release. -version: 1.0.0-rc.2 -date-released: "2026-08-03" +version: 1.0.0-rc.3 +date-released: "2026-08-05"