Integration of OpAMP with system packages

[mmanciop]

Hello from the Packaging SIG!

We are working on providing an automatic instrumentation experience based on the OpenTelemetry Injector for instrumenting applications written in various programming languages. We want to build the experience around the Declarative Configuration format, where there would be declarative configuration files (one per language at the moment, more on this later) used by the SDKs that the injector injects. In this first phase, we are POC-ing with Java, Node.js and .NET, but we know from experience that Python and Ruby are also doable with limited work, with the biggest gap being the support for the declarative configurations.

OpAMP in the scope of system packages

We would like OpAMP to be a first-class citizen of the system packages, but there are gaps in the story we currently cannot reconcile. We see two possibilities:

1. We assume that eventually all SDKs will have OpAMP built in, and we need to devise a way for the system packages to make it simple to configure OpAMP across all languages.
2. We introduce an OpAMP supervisor as a system package, and that is responsible to manage configurations.

We are assuming that the better direction is (1), but it feels like the state of the art of the implementations of OpAMP in the SDKs in not advanced enough for this to be in the foreseeable future. In the system packages SIG we don't have the bandwidth to implement (2).

What is your take on the matter of OpAMP support in the system packages? What is missing from this picture?

What about the collector?

The collector DEB and RPM files are not part of the system packages atm, but we do foresee "bringing them into the fold" and making them feel like a cohesive part of the rest of the story. Since the collector already has OpAMP support, that aspect seems to be covered.

Why one configuration file per language?

Because currently we need language-specific overrides, e.g. in terms of which instrumentations to turn on, but that part of the declarative configuration spec is not marked as stable and not yet implemented in enough SDKs. The moment it is stable and well supported, we will likely consider to move to one single configuration file across all languages by default (maybe still allowing language-specific configuration files as opt-in) to reduce the amount of configuration to be done.

[douglascamata]

I don't think option 2 makes a lot of sense when it comes to OpAMP support for language SDKs in the long term. I do agree with you that it can be much easier to build though, specially taking advantage of existing code in the Collector's OpAMP Supervisor, opamp-go library, even the OpAMP bridge in the Operator (which seems conceptually very similar to this).

One question I have is: how would configuration reload work here? Would this new supervisor know how to reload apps running in all the supported SDKs (i.e. a simple USR2 signal or equivalent in Windows)? Who monitors the health of the apps in case a configuration change causes crashes to try to restore them?

Option 1 is definitely the ideal in the long term and where language SDKs should move towards. It's the one that will work better out of the box.

The collector DEB and RPM files are not part of the system packages atm, but we do foresee "bringing them into the fold" and making them feel like a cohesive part of the rest of the story. Since the collector already has OpAMP support, that aspect seems to be covered.

What would this mean exactly? We already publish deb and rpm packages for the Collector and its Supervisor. What would change there? I'm trying to get an understand of what you have in mind.

Because currently we need language-specific overrides, e.g. in terms of which instrumentations to turn on, but that part of the declarative configuration spec is not marked as stable and not yet implemented in enough SDKs. The moment it is stable and well supported, we will likely consider to move to one single configuration file across all languages by default (maybe still allowing language-specific configuration files as opt-in) to reduce the amount of configuration to be done.

I guess you would also handle multiple different configurations for an individual SDK if the host runs multiple apps, right? For example, I might host multiple Python apps in the same machine and configure the SDK differently in each one of them.

[mmanciop]

One question I have is: how would configuration reload work here? Would this new supervisor know how to reload apps running in all the supported SDKs (i.e. a simple USR2 signal or equivalent in Windows)? Who monitors the health of the apps in case a configuration change causes crashes to try to restore them?

AFAIU, it depends exclusively whether it would work inside the SDK.

The collector DEB and RPM files are not part of the system packages atm, but we do foresee "bringing them into the fold" and making them feel like a cohesive part of the rest of the story. Since the collector already has OpAMP support, that aspect seems to be covered.
What would this mean exactly? We already publish deb and rpm packages for the Collector and its Supervisor. What would change there? I'm trying to get an understand of what you have in mind.

Today there are no relationships in the package metadata between, for example, the opentelemetry metapackage (see the PoC) and the collector's. The default configurations do not assume there is a local collector. We could make it so that, if the local collector is installed via a system package, the SDKs talk to it ootb.

Because currently we need language-specific overrides, e.g. in terms of which instrumentations to turn on, but that part of the declarative configuration spec is not marked as stable and not yet implemented in enough SDKs. The moment it is stable and well supported, we will likely consider to move to one single configuration file across all languages by default (maybe still allowing language-specific configuration files as opt-in) to reduce the amount of configuration to be done.
I guess you would also handle multiple different configurations for an individual SDK if the host runs multiple apps, right? For example, I might host multiple Python apps in the same machine and configure the SDK differently in each one of them.

That is already possible in some languages setting the OTEL_CONFIG_FILE env var to specify the location of the config, but it is a manual setup users must perform, e.g., in their SystemD unit.

[tigrannajaryan]

One question I have is: how would configuration reload work here?

This is the important question. For me the usefulness of OpAMP for SDKs is to allow live changes to SDK configuration without restarting the instrumented application. An example I have is to push changes like trace sampling ratio to Otel trace SDK, and have the changes applied immediately, without any application restarts.

I think a useful application of OpAMP for Otel SDKs is possible with one of the 2 approaches:

• Direct OpAMP implementation in the SDK, with ability to receive new SDK configs and apply them on the fly.
• Add support for hot-reload of Otel SDK config from local files, applying config changes on the fly. Have OpAMP-controlled Supervisor write to that local config file and let Otel SDK config file watcher pickup the changes and hot-reload.

I don't have a strong opinion on which of this approaches is more preferable. Someone needs to do a more thorough analysis to make a good design choice.

An implementation where there is an OpAMP-controlled Supervisor that writes config files to a local file which is only picked by Otel SDK on the startup is not very useful in my opinion.

So, in short, either implement OpAMP directly in SDK or implement config hotreload from local file in SDK. If neither of this can be done I would recommend excluding OpAMP-related functionality from packaging SIGs work in the first phase until the SDK are better prepared.