503: concurrent consumer connections#518
Conversation
…'t add any value. Tweaked wording of introduction to improve clarity and avoid "ensure". Removed repetition of motivation in R0201; its already in the intro and makes more sense there. Clarified operation => SDC service operation (language from annex B of 20701). Added change log entry.
| @@ -0,0 +1,137 @@ | |||
| [#vol2_ch_a_mdpws_concurrent_connections] | |||
| ==== MDPWS Concurrent Consumer Connections | |||
There was a problem hiding this comment.
Should be more clear that this is about TCP connections so that it will not be confused with e.g. HTTP connections or 'SDC connections'.
Also this could be general about limits of SDC providers or SDC participants. So TCP connections could be part of a broader concept 'capability limits'.
There was a problem hiding this comment.
Updated the title to MDPWS Concurrent Consumer TCP Connections to make it clearer this applies to the data transmission protocol; the narrative expands on this.
@d-gregorczyk mentioned PAT-22 that this was an issue specifically with MDPWS. Other limits aren't MDPWS specific so probably go in other places such as the pm:ProductionSpecification for alert-pkp, an extension on the pm:ClockDescriptor for epoch support, certificate EKUs for access control things.
Did you have some particular capabilities in mind?
| ** <<term_transport_address>> and/or, | ||
| ** `[source endpoint]` message property (see <<ref_oasis_ws_addressing_2006>>) and/or, | ||
| ** client credentials used for TLS authentication. | ||
| * Securing the connection may be the most expensive part of an SDC service operation, so a <<vol1_spec_sdpi_p_actor_somds_provider>> may |
There was a problem hiding this comment.
It might be unreliable to identify single consumers using this methods.
There was a problem hiding this comment.
Yes. And annoying too since we run multiple instances of a test consumer on one computer for load testing.
Is there a better way?
| ** client credentials used for TLS authentication. | ||
| * Securing the connection may be the most expensive part of an SDC service operation, so a <<vol1_spec_sdpi_p_actor_somds_provider>> may | ||
| drop concurrent connections from a <<vol1_spec_sdpi_p_actor_somds_consumer>> that exceed its limits, or respond with | ||
| HTTP status code `429 (Too Many Requests)` (after securing the connection) to signal the <<vol1_spec_sdpi_p_actor_somds_consumer>> |
There was a problem hiding this comment.
Returning a HTTP status code requires a successful TLS handshake, if this itself is costly for a certain provider this might not be a suiting way then.
Also this gives 429 a special semantics (so a provider can not use it 'normally' in it's HTTP server component). So it should be considered using a custom code or a soap fault.
There was a problem hiding this comment.
Yes, returning the status code may be too costly but at least it gives the consumer something it can work with. I don't think we can specify a blanket approach here. A busy provider might drop connects without any response (this happens anyway on the accept socket when the backlog is full), a less busy one might complete the handshake and provide an http response. I think a robust consumer would handle these faults in a way that might be application dependent (e.g., retry, user intervention, etc).
Hmm. The semantics here seem consistent with the too many requests status code to me. This chapter is really just describing some scenarios where the consumer might see a too many request response and suggestions on how to deal with it.
There seems advantages using standard codes if possible: the normal response to a "too many requests" seems to fit what's outlined.
Do you have some scenarios where the consumer would respond differently to a fault that generates too many requests?
| @@ -0,0 +1,137 @@ | |||
| [#vol2_ch_a_mdpws_concurrent_connections] | |||
There was a problem hiding this comment.
General questions:
Should a provider actively distribute connections between consumers? This at least would require some justification why this is preferable over just 'first come first serve'.
Should a provider signal limitations beforehand? So a consumer could use a adapted behaviour from the beginning. The approach here is that the consumer gets an error and only then can retry in another way. Also this requires a rejection that can return a value, just no accepting a connection does not fit into this pattern,
Limits could be communicated with e.g. Discovery scopes.
Limits can change over time (e.g. a provider uses a limited resource temporarily for non-SDC purposes), has this to be considered?
There was a problem hiding this comment.
It might be good to distribute connections. We discussed a couple of options for this during PAT-22:
- QoS packets : individual packets in the http request can be tagged with a priority to facilitate routing. It doesn't seem to be a good option to me, but it is the one mentioned in §10.3.2 of 20701 so I suppose it has that going for it.
- HTTP Priority headers from RFC9218. This is a request based approach that makes a lot of sense to me. A consumer could put a higher priority on remote operations requests than renewing subscriptions, for example, which could allow the provider to make meaningful choices.
Providers could also make choices based on the SOAP operation. For example, a connection that was just renewing subscriptions could be closed aggressively because the TLS handshake time (probably) isn't that critical for a renewal. Or we know the typical start-up behaviour is WS-transfer/get, Subscribe, GetMdib so a simple provider might detect that sequence and close when its done. These all leverage behaviours well defined in the HTTP standards though.
We discussed signalling limitations, during PAT-22. In particular putting something in the ws-transfer/get response that could help the consumer plan its approach. E.g.:
I think the conclusion was that all consumers have to handle HTTP faults, like 429 anyway. So maybe http status codes are sufficient. Or at least a place to start with some suggested approaches as in this PR.
I think you have a good use case in splitting connections for remote operations; it seems doable with just status codes but easier with a priori info. Perhaps that should be a separate issue?
Limits can certainly be dynamic and depend on the providers load. I think this is part of the reason to try relying on HTTP status codes anyway. That is, even if a provider declares it can accept 3 concurrent connections from a single consumer it might not be able to do that with 5 consumers all at the same time. So we ultimately always have to handle the http status codes at some point.
There was a problem hiding this comment.
Thought a bit more about all of this... 😀
Maybe the topic should be addressed first more fundamentally (before addressing capability and priority management, and mitigations). Participants have technical limitations themselves and may require certain capabilities from other participants. To have a connected system of medical devices the capabilities have to match.
This is probably most interesting for the customers that are building/integrating SDC systems.
From this perspective it would be interesting to:
- classify/describe capability limits
- How to document them in IfU? (Participant should state their own overall limits and e.g. consumers should state what share of capabilities they need from each provider)
- Make limits/capabilities public in network (Metadata, new discovery scopes, ...)?
- Should devices with certain MDS type have minimal limits? Which would allow connection strategies for different device types, so that e.g. it's from consumers point of view safe to use one subscription for everything.
- Should consumers of devices with certain MDS type not suppose capabilities above certain limits?
I would guess that consumers generally will be hesitant to change there connection behaviour based on errors/status got from some functionality invocation (because they do what they do because they deem this necessary e.g. for risk reason). On the other hand I think this could be possible based on device type, remote controlling a SpO2 sensor has other risks and technical requirements than a ventilator. The device type is also already known from Discovery.
| the risks of opening multiple, concurrent connections to a <<vol1_spec_sdpi_p_actor_somds_provider>>'s | ||
| hosted services, including the potential impact on both the <<vol1_spec_sdpi_p_actor_somds_provider>>'s | ||
| and the <<vol1_spec_sdpi_p_actor_somds_consumer>>'s ability to deliver their respective | ||
| <<acronym_sfc,system function contribution>>. |
There was a problem hiding this comment.
Generally the behavior of the consumer should not have impact on the providers functionality, only the consumer may experience that it cannot do what it want to do on the provider.
I think this is too general.
There was a problem hiding this comment.
Any suggestions how we could fix this? I see it
- partly addressing TR1134 from 10700: "CLINICAL FUNCTIONs of an SDC BASE PARTICIPANT except for network-related functionality SHALL be designed in a way that they are not impaired by the IT NETWORK communication, including but not limited to..."
- to draw attention to the issue from the consumer's side. The note offers one solution.
| ==== | ||
| A consumer may choose to limit the number and/or frequency of concurrent connections it | ||
| opens to a <<vol1_spec_sdpi_p_actor_somds_provider>>'s hosted services when it receives a | ||
| response with HTTP status code `429` (Too Many Requests). |
There was a problem hiding this comment.
again, that looks like '429' gets very special semantic here.
There was a problem hiding this comment.
I'm unclear on that. It seems the normal response to a 429 code would be to make fewer requests or wait a bit and try again (particularly when the retry-after header is present).
There was a problem hiding this comment.
The http standard does have a little bit to say on concurrency: https://www.rfc-editor.org/rfc/rfc9112.html#name-concurrency. Its pretty vague though.
There was a problem hiding this comment.
And seems to me to fit the official description for the 429 status code:
https://datatracker.ietf.org/doc/html/rfc6585#autoid-4
| * implement backoff and retry logic. | ||
|
|
||
| A <<vol1_spec_sdpi_p_actor_somds_consumer>> receiving a `429` response should not interpret this as a operation failure, but rather as an indication that the provider is under load and needs the consumer to adjust its behaviour. | ||
| The consumer may use information in the `Retry-After` header, if present, to determine when to retry the request, but could also consider implementing an exponential backoff strategy to avoid overwhelming the provider with retries. |
There was a problem hiding this comment.
This measures may not resolve the situation.... So how helpful is this approach within a system of connected medical devices? Maybe this requires Hospital IT to reconfigure something.
There was a problem hiding this comment.
Yep. Failure is always an option.
I think the main utility is to bring to light that providers might not support concurrent connections from a single consumer and offer some ideas on how consumers could deal with this. The alternative could be mandating at least X, or no more than 1, connections from any consumer. But it seems there would be more challenges with that approach (even ignoring resource constraints). And I'm not sure it makes sense to pull all participants down the lowest common denominator since there are good reasons for consumers to use multiple connections in many cases.
Co-authored-by: Alexander Pentzel <89145096+alex-pe1@users.noreply.github.com>
|
Thanks for all your comments @alex-pe1 . I think I responded to them all. I'd be happy to do a teams video discussion if that would be helpful. 7pm here is 9pm in Germany right now so any time before your lunchtime could work for me. I see this PR as fleshing out expectations for a fault condition, which might be a starting point for something more advanced. But even with more complex solutions we can still have faults. |
3 participants
📑 Description
Not all providers can support many concurrent consumer connections while some consumers and/or providers may benefit from multiple concurrent connections to mitigate risks. This PR outlines an approach, leveraging the HTTP 429 status code, to support interoperability for both scenarios.
☑ Mandatory Tasks
The following aspects have been respected by the pull request assignee and at least one reviewer: