feat: plugin protocol 6.9#2
Draft
rhencke wants to merge 7 commits into
Draft
Conversation
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
bac6676 to
0fca0d7
Compare
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
3 times, most recently
from
90f371a to
728f092
Compare
rhencke
changed the title
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
728f092 to
1ac3801
Compare
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
1ac3801 to
cedb5c7
Compare
There was a problem hiding this comment.
Pull request overview
Implements Terraform/OpenTofu Plugin Protocol 6.9 across the framework: upgrades the proto + regenerated Python gRPC bindings, adds ClientCapabilities + deferred responses, introduces write_only attribute support, adds ephemeral resources, and adds resource identity schemas + upgrade RPC handling.
Changes:
- Upgrade protocol definitions and regenerate Python protobuf/gRPC stubs (protobuf 6.x, grpcio 1.78+).
- Add framework support for ClientCapabilities, deferred responses, write_only enforcement, ephemeral resource lifecycle RPCs.
- Add resource identity schema/types and thread identity through CRUD + implement GetResourceIdentitySchemas/UpgradeResourceIdentity; expand unit/e2e coverage.
Reviewed changes
Copilot reviewed 23 out of 25 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tfplugin.proto | Updates protocol definitions to 6.9 (capabilities, deferred, identity, ephemeral, write_only). |
| tf/schema.py | Adds write_only to Attribute + introduces IdentitySchema/IdentityAttribute types. |
| tf/provider.py | Implements capabilities extraction, deferred encoding, identity threading, ephemeral RPC handlers, write_only enforcement. |
| tf/iface.py | Adds ClientCapabilities, DeferReason, context defer/identity support, ephemeral + identity mixin protocols. |
| tf/tests/test_schema.py | Adds unit tests for write_only attribute serialization defaults. |
| tf/tests/test_provider.py | Adds tests for write_only enforcement in plan/apply paths. |
| tf/tests/test_resource_identity.py | New comprehensive unit tests for identity schema/RPCs and CRUD threading. |
| tf/tests/test_ephemeral.py | New unit tests for ephemeral resources and schema advertisement/caching. |
| tf/tests/test_capabilities.py | New unit tests for ClientCapabilities extraction + deferred response encoding. |
| tf/gen/tfplugin_pb2.py | Regenerated protobuf bindings for protocol 6.9 (protobuf 6.x). |
| tf/gen/tfplugin_pb2.pyi | Updated typing stubs for new proto messages/fields and timestamp import. |
| tf/gen/tfplugin_pb2_grpc.py | Regenerated gRPC stubs; updates minimum grpcio version and adds new RPC methods. |
| tf/gen/grpc_stdio_pb2.py | Regenerated protobuf bindings (protobuf 6.x). |
| tf/gen/grpc_stdio_pb2_grpc.py | Regenerated gRPC stubs; bumps minimum grpcio version. |
| tf/gen/grpc_controller_pb2.py | Regenerated protobuf bindings (protobuf 6.x). |
| tf/gen/grpc_controller_pb2_grpc.py | Regenerated gRPC stubs; bumps minimum grpcio version. |
| pyproject.toml | Bumps dependency floors for protobuf/grpcio and grpcio-tools dev dependency. |
| e2e/mathprovider/poetry.lock | Updates e2e lockfile for new dependency versions. |
| e2e/mathprovider/mathprovider/provider.py | Adds example Secret (write_only + identity) and ScaledSecret (ephemeral) implementations. |
| e2e/mathprovider/mathprovider/test_provider.py | Adds e2e tests for capabilities, write_only, identity, and ephemeral resources. |
| e2e/e2e_framework/e2e_framework/framework.py | Adds helper to parse OpenTofu version for conditional e2e tests. |
| .github/workflows/python-package.yml | Expands e2e matrix to include OpenTofu v1.11.5. |
| Makefile | Switches TFPLUGIN_PROTO variable to 6.9 proto name. |
| CHANGELOG.md | Documents newly added protocol features and framework APIs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 25 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
cedb5c7 to
6d29175
Compare
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
6d29175 to
1e4cb55
Compare
There was a problem hiding this comment.
Copilot reviewed 24 out of 27 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
1e4cb55 to
c18d4fe
Compare
rhencke
changed the title
rhencke
changed the title
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
c18d4fe to
2b5659d
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 29 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
force-pushed
the
feat/plugin-protocol-6.9
branch
from
2b5659d to
00bd97b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
tf/provider.py:409
- The ValidateResourceConfig comment suggests providers can “inspect” request.client_capabilities to opt into write_only enforcement or deferred planning, but Resource.validate() does not receive a context object (so it can’t access capabilities) and write_only/defer behavior is handled elsewhere in the framework. Please reword or remove this comment to avoid implying a capability-driven validation hook that doesn’t exist.
deprecated=False,
),
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fetch tfplugin6.9.proto from the OpenTofu plugin-protocol docs directory
(the canonical source for all 6.x revisions) and regenerate the Python
gRPC bindings with `make generate-tfproto`.
Proto 6.9 is the latest available version as of this writing. It adds
the following on the wire; subsequent commits add the corresponding
Python-level APIs:
- write_only attribute field on Schema.Attribute
- ClientCapabilities message (deferral_allowed, write_only_attributes_allowed)
threaded into ValidateResourceConfig, ConfigureProvider, ReadResource,
PlanResourceChange, ImportResourceState, ReadDataSource, and
OpenEphemeralResource request messages
- Deferred message (with Reason enum) added to ReadResource,
PlanResourceChange, ImportResourceState, ReadDataSource, and
OpenEphemeralResource response messages
- ResourceIdentitySchema / ResourceIdentityData messages and the
GetResourceIdentitySchemas / UpgradeResourceIdentity RPCs
- Ephemeral resource lifecycle RPCs: ValidateEphemeralResourceConfig,
OpenEphemeralResource, RenewEphemeralResource, CloseEphemeralResource
- ephemeral_resource_schemas field on GetProviderSchema.Response
- EphemeralResourceMetadata on GetMetadata.Response
Dependency constraint changes:
- protobuf upgraded from >=5.28.3,<6.0.0 to >=6.0.0,<7.0.0. The 6.x
Python library uses a new C extension API that is binary-incompatible
with 5.x; the generated bindings validate this at import time.
- grpcio upgraded from >=1.67.1,<2.0.0 to >=1.78.0,<2.0.0. The
regenerated gRPC stubs embed a minimum runtime version check
(grpcio>=1.78.0); the constraint was raised to match.
- poetry.lock (root) and e2e/mathprovider/poetry.lock are both updated.
Each sub-project has its own independent lock file; changing tf's
constraints does not automatically re-resolve sub-project locks — both
must be regenerated explicitly with `poetry lock`.
Note: one trailing-space character on the CallFunction RPC line has been
removed; the official proto contains it but it is clearly a typo and
causes linter warnings.
v1.9.3 and v1.10.5 test backwards compatibility (proto 6.9 features skipped). v1.11.5 tests the full feature set: write_only, resource identity, ephemeral resources. Version-gated tests use setUp() + skipTest() checks in each e2e test class.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 28 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
commented
Apr 2, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 27 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
rhencke
commented
Apr 2, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 27 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rhencke
commented
Apr 2, 2026
New ClientCapabilities dataclass in tf.iface exposes the capability flags that Terraform advertises on each request: deferral_allowed client can handle deferred responses write_only_attributes_allowed client understands write_only attributes The framework extracts these via _extract_capabilities() and threads the result into every context object (ctx.client_capabilities) across ReadResource, PlanResourceChange, ApplyResourceChange, ReadDataSource, ImportResourceState, and the configure path. Providers that do not need to inspect capabilities can ignore the field entirely. Also makes PlanContext.changed_fields optional (default empty set) — a natural consequence of _Context gaining a default-valued field, which Python's dataclass rules require subclass non-default fields to also have defaults or be declared keyword-only. Adds tf_version() to the e2e ProviderTest base class (parses the version from TF_BINARY_NAME without a subprocess) and wires version-gated setUp() checks in each proto 6.9 test class so that write_only, identity, and ephemeral tests are skipped on OpenTofu < 1.11. Unit tests verify _extract_capabilities(), the dataclass defaults, that all RPC handlers accept requests with arbitrary capabilities, and that ctx.client_capabilities is populated with the values Terraform sent. The e2e test confirms that plan and apply succeed unchanged, demonstrating backward-compatible handling.
Add DeferReason enum, ctx.defer() method on all context types, and automatic Deferred message encoding in ReadResource, PlanResourceChange, and ReadDataSource responses. Providers signal deferred operations by calling ctx.defer(reason); the framework encodes the result in the proto response automatically.
Add EphemeralResource protocol with open(ctx: OpenContext, config) signature, OpenContext context class (supports ctx.defer() and ctx.client_capabilities), and four RPC handlers: ValidateEphemeralResourceConfig, OpenEphemeralResource, RenewEphemeralResource (no-op), and CloseEphemeralResource. Ephemeral schemas are included in GetProviderSchema. The math provider exposes math_scaled_secret as a demonstration ephemeral resource. test_capabilities.py gains test_open_context_defer now that OpenContext is available.
write_only marks an attribute whose value is provided in configuration but must be omitted from state. Typical use cases are passwords and API keys that a provider needs during create/update but that should never be persisted in the Terraform state file. The field is only meaningful for managed resource schemas and must be combined with optional or required (as specified in the proto docs). Serialises to proto field 11 on Schema.Attribute (proto 6.8+). The Terraform plugin protocol places the write_only enforcement responsibility on the plugin SDK, not on Terraform Core itself. Terraform Core only signals support via write_only_attributes_allowed in ClientCapabilities (proto 6.6) and trusts the SDK to do the work. The reference implementation is terraform-plugin-framework (Go): NullifyWriteOnlyAttributes() in internal/fwserver/write_only_nullification.go called from server_planresourcechange.go (plan path) and server_createresource.go / server_updateresource.go (apply path). https://github.com/hashicorp/terraform-plugin-framework/blob/main/internal/fwserver/write_only_nullification.go This framework follows the same design: - PlanResourceChange: null write_only attrs in planned state when caps.write_only_attributes_allowed is True (older clients that don't set this flag see write_only as a regular attribute). - ApplyResourceChange: re-inject the real value from config into planned_state before calling create/update (the Go SDK passes config separately; we inject so providers use the same planned dict they always receive). Strip write_only from returned state. Unit tests verify serialisation to protobuf for the true/false/combined cases. End-to-end tests confirm that the attribute value does not appear in Terraform state after apply and is not echoed in plan output.
Defines the API surface for resource identity without wiring identity
data through CRUD handlers. Full wiring is deferred until a real
integration test can validate the round-trip.
As of OpenTofu 1.11, GetResourceIdentitySchemas and UpgradeResourceIdentity
satisfy the gRPC interface but panic("unimplemented") — the client never
calls them, so CRUD wiring would be untestable. Terraform (HashiCorp)
introduced full support in v1.12.
What is included:
- IdentityAttribute and IdentitySchema in tf.schema
- ResourceWithIdentity and ResourceWithUpgradeIdentity mixin protocols
in tf.iface; has_identity() TypeGuard
- ctx.set_identity() / ctx.current_identity API on context objects
(stored but not yet attached to RPC responses)
- GetResourceIdentitySchemas RPC — returns schemas for all
ResourceWithIdentity resources; omits plain resources
- UpgradeResourceIdentity RPC — delegates to ResourceWithUpgradeIdentity
if implemented, otherwise passes identity through unchanged
What is deferred (TODO):
- Threading identity in/out of ReadResource, PlanResourceChange,
ApplyResourceChange, and ImportResourceState responses
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 27 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Terraform Plugin Protocol 6.9.
Proto upgrade (6.5 → 6.9)
tfplugin6.9.protoreplacestfplugin6.5.proto; Python gRPC bindings areregenerated. Dependency floor bumps:
protobuf5→6 (binary-incompatibleC-extension API),
grpcio≥1.78.0 (stub minimum version check).ClientCapabilities (proto 6.6)
ClientCapabilitiesdataclass intf.ifaceexposes the two capabilityflags Terraform sends on most request types:
deferral_allowed— client will retry a deferred planwrite_only_attributes_allowed— client understands write_onlyThe framework extracts capabilities via
_extract_capabilities()andthreads the result into every context object as
ctx.client_capabilitiesacross all RPC handlers. Providers that do not need to inspect capabilities
ignore it entirely and behave identically to before.
Deferred responses (proto 6.6)
DeferReasonenum andctx.defer(reason)let a provider signal that anoperation cannot complete yet. The framework encodes the
Deferredprotomessage automatically in
ReadResource,PlanResourceChange, andReadDataSourceresponses. Terraform retries in a subsequent planningcycle when
deferral_allowedis True.Ephemeral resources (proto 6.7)
EphemeralResourceprotocol withopen(ctx: OpenContext, config)andclose()for values that exist only during plan/apply and are neverpersisted to state. The framework handles four new RPCs:
ValidateEphemeralResourceConfig,OpenEphemeralResource,RenewEphemeralResource(no-op default), andCloseEphemeralResource.Ephemeral schemas are included in
GetProviderSchema.The math provider exposes
math_scaled_secretas a demonstration: itsoutput feeds directly into a
write_onlyattribute on a managed resource.write_only attributes (proto 6.8)
schema.Attribute(write_only=True)marks a field that is accepted inconfiguration but must never appear in state — the right primitive for
secrets such as API keys and passwords.
The framework enforces this automatically: in
PlanResourceChangethevalue is nulled when the client advertises
write_only_attributes_allowed;in
ApplyResourceChange(which has noclient_capabilitiesfield in theproto) write_only support is detected by inspecting whether the planned
state already carries nulls for those fields, and the real value is
restored from config so the provider receives it during apply. Providers
never need to handle write_only stripping themselves.
Resource identity stubs (proto 6.9)
Defines the API surface for resource identity without wiring identity data
through CRUD handlers. Full wiring is deferred pending client support.
As of OpenTofu 1.11,
GetResourceIdentitySchemasandUpgradeResourceIdentitysatisfy the gRPC interface butpanic("unimplemented")— the client never calls them, making CRUDwiring untestable end-to-end. Terraform (HashiCorp) introduced full
support in v1.12.
What is included:
IdentityAttribute/IdentitySchemaintf.schema;ResourceWithIdentity/ResourceWithUpgradeIdentitymixin protocols intf.iface;ctx.set_identity()/ctx.current_identityAPI on contextobjects;
GetResourceIdentitySchemasandUpgradeResourceIdentityRPCimplementations.
What is deferred: threading identity in/out of
ReadResource,PlanResourceChange,ApplyResourceChange, andImportResourceState.Backwards compatibility
All proto 6.9 features degrade gracefully on older clients:
Falsewhen absent from the requestThe e2e matrix tests three OpenTofu versions: v1.9.3 and v1.10.5
(backwards compatibility — proto 6.7/6.8 tests skipped), and v1.11.5 (full
feature coverage for everything except identity). Version gating uses
setUp()+skipTest()in each test class; no feature-flagging inproduction code.
Work queue
Completed (33)