Rework of the EPP extension handling for response-only and multiple extensions

[valkum]

The current extension mechanism couples the response type strictly to the request type. This coupling is elegant, but it does not hold for the full reality of EPP. Several important cases cannot be expressed without workarounds, and some of these workarounds are currently rejected by real EPP servers. Therefore, a rework of the extension handling is desired. I do not have the definitive solution. This acts more as a tracking issue to discuss possible solutions.

Current Design

Relevant traits are defined in src/request.rs:

pub trait Command: ToXml + Debug {
    type Response: FromXmlOwned + Debug;
    const COMMAND: &'static str;
}

pub trait Extension: ToXml + Debug {
    type Response: FromXmlOwned + Debug;
}

pub trait Transaction<Ext: Extension>: Command + Sized {}

The response is consequently typed as Response<Cmd::Response, Ext::Response>. The extension response type is thus derived from the request extension type. One request extension corresponds to exactly one response
extension, and there is no other path to obtain a typed extension response.

Problem Statement

The design above is insufficient for the following reasons.

The Response Type Is Bound to the Request Type

The associated type Extension::Response can only be reached if a request extension of that type is actually sent. For extensions that contribute only a response element, no suitable request element exists. Hence a request element must be
invented purely to name the response type.

Empty or Invalid Request Elements Break Strict Servers

The consequence of this is visible in the RGP extension. In src/extensions/rgp/request.rs the following is declared:

impl<'a> Transaction<Update<RgpRestoreRequest<'a>>> for DomainInfo<'a> {}

In order to receive the RGP status on a plain DomainInfo, the caller is forced to attach Update<RgpRestoreRequest>. This serializes an actual <rgp:update><rgp:restore op="request"/></rgp:update> element into an information query, which is
semantically wrong. Furthermore, several EPP servers reject a request that carries an empty or unexpected element.

Multiple Simultaneous Extensions Cannot Be Expressed

transact accepts exactly one extension type. Combinations that occur in practice (for example, Rgp together with a Fee extension, or SecDNS together with an allocation token) require a hand-written wrapper type per combination. This does
not scale and produces a combinatorial explosion of glue types. A type-safe solution like a tree of

struct C<A, B> {
  a: A, 
  b: B
}

does work but is an example of the combinatorial explosion of glue types. You might end up with a Response<DomainInfo, C<SecDnsInfo, C<RgpInfo, FeeInfo> (I made the combination up).

This type-safety is far better encoded in actual registry clients. Any client that targets different registries (and thus different implemented EPP servers), needs some kind of runtime dynamic checks.

Connection and Login Negotiated Extensions Are Not Modelled

Some extensions are negotiated once during the login handshake via the URIs and are thereafter emitted by the server on any response, independently of the per-command request extension. Change Poll and the unhandled-namespace
mechanism of RFC 9038 are typical examples. The present type system has no vocabulary for "this extension may appear on every response of this connection". It can only express "this extension belongs to this one command invocation".

Prior Art and Related Work

hyper

The hyper crate solves a structurally similar problem with a typed extension map: https://docs.rs/hyper/latest/hyper/ext/index.html. Protocol-level data that is not bound to a single request shape is stored in a type-indexed map and retrieved
by type.

Draft PR

I tried to solve parts of this earlier in #67.

Current Internal Workaround

We currently use an internal workaround that allows us to encode expected extension responses by hand. This gives you more control if you know beforehand which extensions are expected in a response for a particular request for a single EPP server but still is affected by the combinatorial explosion of glue types.

pub async fn transact_without_ext<'c, 'e, Cmd, ExtResp>(
        &mut self,
        data: impl Into<RequestData<'c, 'e, Cmd, NoExtension>>,
        id: &str,
    ) -> Result<Response<Cmd::Response, ExtResp>, Error>
    where
        Cmd: Transaction<NoExtension> + Command + 'c,
        ExtResp: FromXmlOwned,

Protocol Extension Registries

We could make extensions a vec over an enum. An enum as an extension registry is also a widely used pattern.
See rustls' ClientExtensions or x509-parser's ParsedExtension

The issue here is that extensibility is baked into EPP. Theoretically there can be EPP servers with closed-source extensions, and there would be no way of maintaining these out of tree.

Options

The options differ along one central axis: which type the element of a response is deserialized into during transact.
instant-xml parses the whole <response> in a single streaming pass; therefore, that field must use a type implementing FromXml.

Extension Trait + Typed Response Map

Introduce an extension trait in the spirit of hyper::ext. The Response then carries a typed map instead of a single E. Offering an API like

let rgp: Option<&RgpInfo> = response.extension::<RgpInfo>()?;

This still requires some kind of registry to name all possible target types. The registry could be expressed via type parameter, providing a default for all in-tree extensions.

#[derive(Debug, FromXml)]
#[xml(forward)]
pub enum DefaultExtensions {
    Rgp(rgp::RgpResponse),
    ChangePoll(change_poll::ChangeData),
    // ...
}
impl ExtensionRegistry for DefaultExtensions {  }

// Default type parameter keeps EppClient<RustlsConnector> working unchanged.
pub struct EppClient<C: Connector, Reg: ExtensionRegistry = DefaultExtensions> { /* ... */ }

This also moves existence checks to call sites. A response with some expected extension missing will deserialize but return None for response.extension::<RgpInfo>()?;

Connection Level Type Parameter:

A way to keep the type safety as is would need some encoding of enabled extensions into the connection.
I tried this in #67.

Currently, transact looks like this:

pub async fn transact<'c, 'e, Cmd, Ext>(
        &mut self,
        data: impl Into<RequestData<'c, 'e, Cmd, Ext>>,
        id: &str,
    ) -> Result<Response<Cmd::Response, Ext::Response>, Error>
    where
        Cmd: Transaction<Ext> + Command + 'c,
        Ext: Extension + 'e

This would need to construct and return some combinator, so that both directions are covered: extensions mapped from a request (for example Namestore) as well as extensions that arrive purely from the session (for example RgpInfo for a DomaInfo request).

This approach does not solve all issues. It might increase complexity by a lot. It does not address multiple simultaneous command extensions, and only one connection-level type is expressible unless a combined enum is hand-rolled, at which point the combinatorial explosion reappears.

Lazy Deserialization

Utilizing the recently added AnyElement we could store raw <extension> children and deserialize them lazily similar to:

let rgp: Option<&RgpInfo> = response.extension::<RgpInfo>()?;

So this is basically a lazy version of the extension trait + typed response map. It would drop the requirement of a registry, though.

Onboard and Iterate on transact_without_ext

This allows easier construction of higher-level APIs. I imagine this could replace transact

pub async fn transact_without_ext<'c, 'e, Cmd, ExtResp>(
        &mut self,
        data: impl Into<RequestData<'c, 'e, Cmd, NoExtension>>,
        id: &str,
    ) -> Result<Response<Cmd::Response, ExtResp>, Error>
    where
        Cmd: Transaction<NoExtension> + Command + 'c,
        ExtResp: FromXmlOwned,

There could be a SimpleClient with limited extension support that keeps the current transact signature.

[djc]

Do you want/need clients to have the same type across different extension capabilities, or would you like to encode server extension capabilities in the type system at the connection level?

[valkum]

I am not settled on that. You will need a router of some kind on top that takes care of selecting the correct registry for a TLD. So any owning machinery (let's call it EppRouter) can use dynamic dispatch if it needs to store a heterogeneous set of clients. Due to different rules enforced by some registries, you might need additional logic, even for the same set of extension capabilities.

It would be nice though, if parts that are used to construct an EppRouter, would use the type system to make extension misuse a compilation error.
E.g. you could do

/// Request side support
pub trait Supports<Ext> {}
/// Response side
pub trait Profile {
  type ResponseExt: FromXmlOwned
}

pub struct Verisign;
impl<'a> Supports<NameStore<'a>> for Verisign {}
impl Profile for Verisign {
  type ResponseExt = Vec<VerisignResponseExt>;
}


impl Client<.., P> {
pub async fn transact<'c, 'e, Cmd, Ext>(...) -> Result<Response<Cmd::Response, P::ResponseExt>, Error>
where
  Cmd: Transaction<Ext> + Command + 'c,
  Ext: Extension + 'e,
  P: Supports<Ext> {...}
}

transact now returns a response with a Vec that can hold all extensions that the profile says it can.

But it still lacks support to correctly identify the matching subset of extension responses for the given request.
For example, we know that the only valid extension response types to an info command (maybe even tied to domain info) request are the info responses of the different extension specs (a sec_dns::InfData and rgp::InfData vs. rgp::updData). I also had one shot a couple of weeks ago at encoding this relation with a trait that had multiple types for the different commands, but had to stop work on that due to other priorities.

Let's ignore the afnic extension which currently does not map well at all to instant-epp

<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
  <extension>
    <frnic:ext xmlns:frnic="http://www.afnic.fr/xml/epp/frnic-2.0"
    xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
      <frnic:command>
        <frnic:recover op="request">
          <frnic:domain keepDS="1">
            <frnic:name>mondomaine.fr</frnic:name>
            <frnic:authInfo>
            <domain:pw>codederecover</domain:pw>
            </frnic:authInfo>
            <frnic:registrant>CTC64742</frnic:registrant>
            <frnic:contact
            type="admin">CTC64742</frnic:contact>
            <frnic:contact
            type="tech">CTC64742</frnic:contact>
          </frnic:domain>
        </frnic:recover>
        <frnic:clTRID>TEST-AUTO-1664371433735-
        lakutakunu</frnic:clTRID>
      </frnic:command>
    </frnic:ext>
  </extension>
</epp>

[djc]

It seems to me that we could define a trait like Profile with implementations that map a combination of a request type and perhaps a tuple of extension types to a response type + another tuple of extension types?

What about the Afnic stuff doesn't map well?

[valkum]

Regarding Afnic: They have this single "command" that is not a command per EPP. The boday just contains the extension. I am not sure we should cover this right now. I believe this single part of the afnic extension is in conflict with the EPP base spec. Users can use the transact_xml version if they really need this single frnic:command.

Let me create an end to end example using something like Profile.