Is your feature request related to a problem? Please describe.
When ASTM F3411 (and later F3548) were written, there was a desire to include a field in notification messages sent due to subscriptions to make the subscriber aware of the sequence of notifications. That is, when USS1 changed the airspace, the DSS would acknowledge the change and also instruct USS1 to notify any subscribers (e.g., USS2 due to a pre-existing subscription) of the change they just made, including a notification_index value that USS1 would deliver to USS2 along with the notification. The intent of this field was to somewhat reduce the time between a notification being lost or received out of order (and therefore USS2 not having current airspace information) and the USS detecting the information loss and proactively recovering. However, this is merely a small optimization because USS2 will always discover the desynchronization next time they attempt to change the airspace and should generally be able to fairly easily recover at that point.
The problem with this mechanism is a sequential notification index necessarily needs to be synchronized with airspace changes, so a subscription effectively serializes all airspace activity within it. Moreover, all subscriptions in the area need to be locked, so to change a small section of airspace, all of the airspace contained in the union of all subscriptions that intersect that small section needs to be locked until the airspace change is complete (so all of their notification indices can be incremented exactly one for that airspace change and the appropriate index can be returned from the DSS response to be included in the notification). To restate differently: sequential notification_index forces maximum concurrency to 1 over an area that is the union of all subscriptions intersecting any particular airspace change.
ASTM recognized this challenge and decided in the May 1, 2024 plenary to remove notification_index entirely in the next revision of F3548. Up until recently, InterUSS had not prioritized a definitive interim solution to this problem as actual user traffic had proved low enough in practice for this serialization to be acceptable (if each request is completed quickly enough, high throughput can still be achieved with very low concurrency). However, recently we have received a number of reports of high latency observed in practice (e.g., #1241, #1311, #1509, #1512) likely tracing to this root cause, making this a higher priority to solve in the short term. We have a number of potential improvements in progress (e.g., #1517, #1518, #1520, #1522, #1523), but it sounds like the sum effect of all of them may not be large enough to achieve overall product performance goals. And, some improvements (e.g., #1363) involve a tradeoff that improves one area at the expense of bringing a different area below the threshold of acceptability. I believe these difficulties are mostly just the unavoidable consequence of choosing to serialize airspace changes in the way necessary to provide notification indices that have the sequential characteristics previously described.
Describe the solution you'd like
I believe the most definitive resolution to the challenge above is likely to change the InterUSS implementation's notification index behavior to avoid extreme serialization, as the technically-best solution validated by ASTM's documented forward plan. I think the simplest form of this solution would be to stop tracking and synchronizing notification index in the underlying DAR and instead provide values indicating that this full optimization information is not available. For instance, we may just return 0 for every notification_index, though I think we can do better in terms of maximum compatibility by returning a daily-monotonic clock value (e.g., milliseconds since midnight).
The relevant ASTM F3548-21 requirement is DSS0005; basically, implement the API. The relevant portion of the API is:
SubscriptionNotificationIndex:
format: int32
description: >-
Tracks the notifications sent for a subscription so the subscriber can
detect missed notifications more easily.
A clock value does help with this somewhat (for instance, a notification with a widely-skewed clock-valued notification index could be flagged as a potential problem with notifications), though of course not as fully as a per-subscription sequential synchronized index. Therefore, I believe we would still meet the letter of the requirement while much more effectively meeting the forward-looking spirit of the functionality. The situation is similar in ASTM F3411-22a.
Even though I expect implementation of this feature will involve substantively changing the SQL queries to the underlying databases, I think it is probably valuable to implement it as a flag-gated feature so that any users preferring the old behavior are not immediately forced into the new behavior before any unforeseen consequences are identified.
Describe alternatives you've considered
A fixed value of 0 for notification_index may possibly have compatibility challenges if a USS only registers a new notification on the basis of a unique specified index, so I think a different value for each notification is preferred. We could simply not lock the subscription while it is being updated so that two racing operations result in the value only being incremented once, but I think that would have similar challenges to the fixed-zero-value with the additional downside that the inference that can be drawn from a particular set of observations is not always the same. A random non-negative int32 value may be more appropriate if we may have two notifications generated for the same subscription in the same millisecond.
Is your feature request related to a problem? Please describe.
When ASTM F3411 (and later F3548) were written, there was a desire to include a field in notification messages sent due to subscriptions to make the subscriber aware of the sequence of notifications. That is, when USS1 changed the airspace, the DSS would acknowledge the change and also instruct USS1 to notify any subscribers (e.g., USS2 due to a pre-existing subscription) of the change they just made, including a notification_index value that USS1 would deliver to USS2 along with the notification. The intent of this field was to somewhat reduce the time between a notification being lost or received out of order (and therefore USS2 not having current airspace information) and the USS detecting the information loss and proactively recovering. However, this is merely a small optimization because USS2 will always discover the desynchronization next time they attempt to change the airspace and should generally be able to fairly easily recover at that point.
The problem with this mechanism is a sequential notification index necessarily needs to be synchronized with airspace changes, so a subscription effectively serializes all airspace activity within it. Moreover, all subscriptions in the area need to be locked, so to change a small section of airspace, all of the airspace contained in the union of all subscriptions that intersect that small section needs to be locked until the airspace change is complete (so all of their notification indices can be incremented exactly one for that airspace change and the appropriate index can be returned from the DSS response to be included in the notification). To restate differently: sequential notification_index forces maximum concurrency to 1 over an area that is the union of all subscriptions intersecting any particular airspace change.
ASTM recognized this challenge and decided in the May 1, 2024 plenary to remove notification_index entirely in the next revision of F3548. Up until recently, InterUSS had not prioritized a definitive interim solution to this problem as actual user traffic had proved low enough in practice for this serialization to be acceptable (if each request is completed quickly enough, high throughput can still be achieved with very low concurrency). However, recently we have received a number of reports of high latency observed in practice (e.g., #1241, #1311, #1509, #1512) likely tracing to this root cause, making this a higher priority to solve in the short term. We have a number of potential improvements in progress (e.g., #1517, #1518, #1520, #1522, #1523), but it sounds like the sum effect of all of them may not be large enough to achieve overall product performance goals. And, some improvements (e.g., #1363) involve a tradeoff that improves one area at the expense of bringing a different area below the threshold of acceptability. I believe these difficulties are mostly just the unavoidable consequence of choosing to serialize airspace changes in the way necessary to provide notification indices that have the sequential characteristics previously described.
Describe the solution you'd like
I believe the most definitive resolution to the challenge above is likely to change the InterUSS implementation's notification index behavior to avoid extreme serialization, as the technically-best solution validated by ASTM's documented forward plan. I think the simplest form of this solution would be to stop tracking and synchronizing notification index in the underlying DAR and instead provide values indicating that this full optimization information is not available. For instance, we may just return 0 for every notification_index, though I think we can do better in terms of maximum compatibility by returning a daily-monotonic clock value (e.g., milliseconds since midnight).
The relevant ASTM F3548-21 requirement is DSS0005; basically, implement the API. The relevant portion of the API is:
A clock value does help with this somewhat (for instance, a notification with a widely-skewed clock-valued notification index could be flagged as a potential problem with notifications), though of course not as fully as a per-subscription sequential synchronized index. Therefore, I believe we would still meet the letter of the requirement while much more effectively meeting the forward-looking spirit of the functionality. The situation is similar in ASTM F3411-22a.
Even though I expect implementation of this feature will involve substantively changing the SQL queries to the underlying databases, I think it is probably valuable to implement it as a flag-gated feature so that any users preferring the old behavior are not immediately forced into the new behavior before any unforeseen consequences are identified.
Describe alternatives you've considered
A fixed value of 0 for notification_index may possibly have compatibility challenges if a USS only registers a new notification on the basis of a unique specified index, so I think a different value for each notification is preferred. We could simply not lock the subscription while it is being updated so that two racing operations result in the value only being incremented once, but I think that would have similar challenges to the fixed-zero-value with the additional downside that the inference that can be drawn from a particular set of observations is not always the same. A random non-negative int32 value may be more appropriate if we may have two notifications generated for the same subscription in the same millisecond.