add sdpi p history service option

[d-gregorczyk]

☑ Mandatory Tasks

The following aspects have been respected by the pull request assignee and at least one reviewer:

• Changelog update (necessity checked and entry added or not added respectively)
  • Pull Request Assignee
  • Reviewer

[PaulMartinsen]

Suggest changing R0512 to something like:

The provider shall transmit hm:HistoricMdib before transmitting each contiguous MDIB Sequence of Changes in the range requested by a consumer.
Note: this allows a provider to skip over gaps in its historical record while providing a MDIB Configuration for the consumer to apply changes to.

Possible need to tweak R0514 to something like:

For an MDIB Sequence of Changes that is requested from a SOMDS Provider, the SOMDS Provider shall provide exactly one MDIB Configuration in hm:ChangeSequence/hm:HistoricMdib for each contiguous MDIB sequence of changes.

Reasoning:

I was thinking more about the case where a provider starts providing history information but some of the history becomes unavailable before it is delivered. This could occur when:

1. the consumer requests all (or most) of the history available, and
2. the provider has limited memory and must discard old history records to make room for new changes to the mdib.

Some available options :

1. End subscription:

• The provider ends the history subscription, possibly with a (new) fault message.
• The consumer starts a new history subscription and gets the next lot of data.
• Repeat above steps each time items are lost from this history.

2. Send with gaps:

• The provider skips over the missing data and continues with what it has.
• This is (probably?) useless to the consumer who doesn't have a mdib starting point to apply changes to.

3. Send with gaps and starting mdib

• The provider sends a full mdib snapshot (if this was requested by the original subscription)
• Provider continues with change reports.

I quite like option 3 because:

• it avoids multiple subscriptions (with potential race conditions) and doesn't require any special handling of fault messages on the consumer end.
• the provider does its best to provide the information requested and while still letting the consumer create coherent mdib state.
• its more likely to avoid lost history than if the consumer has to re-subscribe. The consumer can unsubscribe if it doesn't want any more data following a gap.

If the consumer didn't request a baseline mdib, we could sensibly fallback to option 1: end the subscription.

[PaulMartinsen]

I would quite like to be able to omit waveform streams from the history because:

1. they use up a lot of memory,
2. an hour old waveform stream may often not be relevant anymore, and
3. a deeper history of states other than waveform streams may be more relevant than a shallow history with waveform streams.

But the general case, where providers may omit any information, seems quite complicated and could be problematic. That is, important context may be lost.

Could the provider be allowed to omit waveform streams either:

1. without disclosing this to the consumer (except perhaps in the documentation), or
2. by sending RealTimeSamleArrayMetricState with @Off.

Or maybe there is no value in waveform streams in the historic record and they could just be omitted completely? That is, remove msg:WaveformStream from the HistoricReportType in HistoryModel.xsd

The main problem seems to be missing state versions (R1005), although I think we need to get rid of that one anyway.

[PaulMartinsen]

I wonder if the <hm:LanguageFilter/> here could be more generic because histories are potentially large and some one to manage that may be helpful.

So, for example, <hm:Filter Dialect="urn:oid:language-oid"/>. With 0 or more allowed. And that providers must also support http://docs.oasis-open.org/ws-dd/ns/dpws/2009/01/Action and should support sdc.filter://Handle. (must and should for consistency with DWPS and 20701 R0039 ).

It would be nice if absence of the action or handle filter implied including everything. Is it a problem that's inconsistent with the language filter behaviour?

[d-gregorczyk]

What do you mean by "more generic"? Currently, a consumer may provide a list of languages to filter against, or, omits the language filter if it requests all languages.

[d-gregorczyk]

There is no support for action and handle based filters here - the subscribe is rather the request input to a stream of information that is send back by the provider. As HTTP 1.1 does not support multiple responses to a single request, we introduced this subscription workaround.

[PaulMartinsen]

I was thinking we ought to require support of action filters and optional support for handle filters (must and should for consistency with other profiles/standards). This would allow a consumer to select only the information they care about. For example, an alarm panel might have no interest in waveform streams (even if they were available). So more generic i was just thinking why is language filtering supported but not other filters?

[d-gregorczyk]

Initially, the history service was not supposed to work that way. You always request a full disclosure of reports as a provider needs to store them anyway as otherwise it cannot respond to queries that may select full disclosure of data.

If the provider is not capable of storing everything, it just responds with incomplete data no matter the input filtering. This btw. allows for the provider to efficiently store historic reports (e.g. pre-compressed).

So more generic i was just thinking why is language filtering supported but not other filters

There is also a filter to skip initial MDIBs. It is basically condensed to the essentials:

• Backfilling: no localized texts and and initial MDIBs required
• Full disclosure: initial MDIB and localized texts (perhaps filtered by a certain set of languages) included

We may add further filters, but those will increase complexity, and complexity increases variance which in turn reduces the degree of achievable interoperability. If there is pressing need by the stakeholders to support "action filters (akap report type filters)" or "handle filters", we may add them as additional filters (which can also be done by introducing further HistoryQuery flavors).

[PaulMartinsen]

In designing libraries I think it is worth anticipating use cases that haven't been codified. Perhaps, for example, there is a consumer that wishes to retrieve a day's history from a provider that's been emitting updates every 10 seconds. With maximum message size of 2MB, that would be transfer of about 17GB. At which point we'd have to start wondering if soap was the cleanest approach (hehe), but if we're stuck with soap for a while then filtering on actions and/or handles might be useful.

But I don't think there's a right answer to this one. Perfection is the barrier to progress so if we have to pick I choose progress. There seems an opportunity to incorporate some potentially useful filtering at this point, but I don't feel strongly about it.

[d-gregorczyk]

I partly agree with idea of anticipating use cases. I think, designing libraries in a way that they are closed for modification but open for extension is the art of the deal - and is hard to achieve at times... trying to anticipate use cases will fail most of the times as something will be missed, no matter how hard and comprehensive the thoughts about the use cases had been.

What matters to me first (and this is a painful lesson learned during the development of BICEPS) is real world demands as the driver for development, which usually only gets clear when we are about to cross the bridge.

The strength of SDPi is that we can react upon change requests relatively easy, so let's try to come up with a minimum solution and bake in new features as they are required by at least one provider and one consumer in the system.

[PaulMartinsen]

Do we need to deal with time epochs here? I can see three options:

1. drop support for requesting history by time, or use relative time like subscriptions.
2. leave the provider free to decide which time-reference frame it should use when comparing mdib timestamps and history query time-stamps.
3. add attributes to select epoch.

1 seems best to me, but I'm not sure what the use cases are for this one. 2 seems okay as well because time is supposed to be UTC time, which is independent of non-slewing time-adjustments. If 2, perhaps a note along the lines : "The provider may not reliably select the correct change reports for time ranges that span an abrupt time adjustment".

[PaulMartinsen]

Is there any value for consumers in allowing hm:Range and/or either range limit to be omitted to ask for:

• everything available (hm:Range omitted),
• everything up to a specific mdib version/time @StartVersion or @StartTime omitted,
• everything from a specific mdib version/time to the time the subscription was created (@EndVersion or @EndTime omitted)

[d-gregorczyk]

Please let us not over-complicate the handling here. Selecting epochs would instantly require each SDPi implementation to support epochs, which is not going to happen. We either remove selection of time frames or introduce some sort of error handling to signify that a time-based request cannot be fulfilled due to past time synchronization issues.

[PaulMartinsen]

Time is quite useful for testing; saves figuring out all the mdib versions. Otherwise I don't see the use cases. It could be okay to just have a warning that if you use time selection then it might not be perfect.

Does omitting parts of the range over complicate? I wonder if it could simplify things from the consumer side by not having to figure out what range to provide.

[d-gregorczyk]

Time is quite useful for testing; saves figuring out all the mdib versions.

I agree, but it is also a tough nut if non-slewing adjustments are taken into account - could also be quite problematic for end-users of the system to understand whats happening.

Does omitting parts of the range over complicate?

The range use-case was mainly intended to be used for backfilling, where a consumer knows exactly which information it missed. I agree that requesting by time is the more prominent use-case if you want to analyze historic data. In the end, answering to time-based requests is always a best guess story. In the end, the user has to inspect the assigned patient and plausibility of the sequence of data that has been returned by the history service.

[PaulMartinsen]

The sequence diagrams weren't showing up for me. It looks like the action isn't copying them into the supplement from .ci\asciidoc-converter\images when it makes the zip file I downloaded from https://github.com/IHE/DEV.SDPi/actions/runs/15761385158

[PaulMartinsen]

Is it worth pointing out that the consumer can also control the data rate it accepts the information using the HTTP protocol (i.e., delaying the response code)?

[d-gregorczyk]

Not sure if it's worth. The main issue we formerly had were the providers being overwhelmed by archive requests being too big to transmit in one chunk. If you drop me some lines, I can add it to the text.

[PaulMartinsen]

Sorry for delay; some days it feels like there's more going on than a Saudi Arabian sword juggler.

How about, after …avoid exceeding data size notification limitations:

Similarly, the SOMDS Consumer may slow the data rate that it receives information from the SOMDS Provider by delaying acknowledgement of notifications (e.g., the HTTP response message to the provider's notification for the MDPWS protocol). Excessive delays, either by the SOMDS Provider or SOMDS Consumer, may lead to timeouts.

[d-gregorczyk]

Sorry for delay; some days it feels like there's more going on than a Saudi Arabian sword juggler.

💯 relatable

How about, after …avoid exceeding data size notification limitation

Sounds good, I will add that.

[PaulMartinsen]

I suggest that (for consistency with other profiles/standards) providers must support action based filtering and should support handle based filtering for the history service. So the protocol should support this somehow (see suggestion for one approach in another comment).

Because the history service is a foundational feature that may be used by large and small participants in the future, a little flexibility seems worthwhile even if the business cases aren't 100% clear at the beginning. I know low-resource providers would prefer that consumers only request what they need but providers have no control over that and best they can do is support the mechanisms.

Implementation doesn't seem hard to me if there is a mechanism in the protocol but I don't feel very strongly about this. Mostly because it requires consumers to use it for providers to benefit, which may not happen in practice.

[d-gregorczyk]

I suggest that (for consistency with other profiles/standards) providers must support action based filtering and should support handle based filtering for the history service.

Action based filtering does not have any meaningful effect on this port type as there is only one notification in there: To enable subscribing to one or more Notifications exposed by a HOSTED SERVICE, this profile defines a Filter Dialect designated "http://docs.oasis-open.org/ws-dd/ns/dpws/2009/01/Action". You would either be able to select everything or nothing - to select nothing, just do not send s subscribe :-)

Handle based filtering may be applicable to the history service, however, it is an optional feature in SDC Glue. I do not see embedded devices performing handle-based filtering on huge amounts of historic data. The limited filtering comes at a cost: less flexibility. But it provides embedded devices with the ability to efficiently store their historic data without performing expensive filtering operations when requested. If we receive proof that such an operation is efficiently processable and needed by the market today, I am totally happy to add it as a mandatory feature to the history service. The things for which there is no demand by the market, I would like to keep it out the spec for as long as possible. It's all big enough already...

a little flexibility seems worthwhile even if the business cases aren't 100% clear at the beginning.

Flexibility is always given - the current filter can be replaced by another one that supports nifty filtering options if we realize, the limited filtering does not suffice. Let's be evolutionary over revolutionary. ;-)

[PaulMartinsen]

For action based filtering, I'm thinking of the StateEventService/EpisoidicMetricReport, StateEventService/EpisoidicAlertReport, ContextService/EpisoidicContextReport, etc. That is a consumer might decide they only want alert history from the day, then retrieve metric history for particular epochs around an alert.

We currently do handle based filtering on embedded devices. Mostly because 20701:R0039 said we should. Its only useful if consumers follow 20701:R0037 though and so far I haven't found any to validate interoperability of our prototype implementation unfortunately.

We don't have additional overhead to apply the filter (that is, everything is filtered anyway) and it would be fine in our prototype history service if there was a way to do communicate selection. Mdibs in embedded devices may be smaller making handling filtering easier, though that's also the use-case that could benefit from the most from reduced traffic. Maybe handle filtering is only useful for embedded devices? Assuming consumer's take advantage of it (and there's no way to tell if it is available; makes me think should requirements are a little pointless for feature support).

Anyway, your right; new features can always be added and things can be changed. That can take a while, though, and this is just one opportunity to do that but I don't feel strongly about it either way.

[PaulMartinsen]

I haven't looked at localization really, but reading this I wondered R0509 requires providers to send localized text even if they only have static localized text. Can't the consumer just get the localized text from the localized text service in this case? I think this is worth thinking through because:

1. many (most?) providers may only have static localized text
2. there might be a lot of localized text so it seems quite inefficient to send it to every history subscription if it never changes. Unless we trying to encourage everyone to support gzip :).

[d-gregorczyk]

BICEPS cannot express if localized texts are static or dynamic, hence you need the option to request localized texts from the past, though you are right and almost all devices I know never change their texts (at least not between major updates).

[PaulMartinsen]

I think most (all?) localized text changes require description modification events. So if that service isn't present a consumer could know the text is static :-).

[d-gregorczyk]

I think most (all?) localized text changes require description modification events. So if that service isn't present a consumer could know the text is static :-).

Sorry I have to disappoint you, but there are body sites in states that can have localized texts; and apart from that you can always add localized texts to extensions. ;-)