Data Server Development Guide

Document Goal

This wiki page is dedicated to provide an overview of the steps required to create a new data-server implementation and connect it to a proxy server like the CQCM map.

Role of the data-server

The proxy server is a consumer of data which is provided by the data-server. The data-server is responsible for deciding which data should be shared with the proxy server by the scopes which have been assigned to a proxy by a given user on the data-server. All scopes of the spec are defined by this document. They are:

• ReadProducts
• ReadOrders
• ReadEnterprise
• WriteOrders
• WriteProducts
• WriteEnterprise

Once the data-server engineer has decided that they would like to allow a proxy server to consume data from their server, they will need to obtain a Keycloak Client ID and Keycloak Client Secret which they will use for authenticating their platform in the network. Similarly, the proxy server itself is a Keycloak Client, both directions of the relationship will be authenticated by OIDC. They will need to contact the proxy server to share their WebID and the URL of their site. Provide the proxy server engineer wishes to consume data from the data-server then they just need to add the WebID/URL of the data-server to a whitelist.

The proxy server consumes data from the data-server via two mechanisms: a webhook, which the data-server calls to inform the proxy server of changes in assigned scopes, and a cache refresh which is requested by the proxy server in a configured interval (e.g. once per day) to make sure that the information is up-to-date. The latter mechanism is there because data-servers aren't required to guarantee delivery of webhooks (e.g. in the case of network failure) and to retrieve any data which may already be available at the beginning of the relationship.

Implementing Data-Server access for the cache refresh

Once per day, and upon receiving a refresh webhook event for a given enterprise (see documentation on the webhooks below), the proxy server will attempt to import data from the data-server in JSON-LD format. The cache refresh happens in two GET requests:

1. the proxy server will make a request to .well-known/dfc/ on the data-server. If it responds 404, the proxy server will assume the default configuration:

{
    "https://github.com/datafoodconsortium/taxonomies/releases/latest/download/scopes.rdf#ReadEnterprise": "/enterprises/",
    "https://github.com/datafoodconsortium/taxonomies/releases/latest/download/scopes.rdf#ReadProducts": "/supplied_products/",
    "https://github.com/datafoodconsortium/taxonomies/releases/latest/download/scopes.rdf#ReadOrders": "/orders/"
}

The document defines the endpoints where data associated to a scope can be found (the configuration on ReadEnterprise defines where the resource server can retrieve instances of dfc-b:Enterprise.) Any scope which is left from the configuration is presumed missing. Note that in the current version of the CQCM proxy server, we are not consuming orders.

2. The proxy server will make a GET request for each scope which is supported, on the configured endpoint. It will import the data which it finds.

For each Read* scope that you wish to support (i.e. from Products, Orders and Enterprises), you will need to create a GET endpoint to retrieve the associated information, serialized into JSON-LD with the classes and properties from the DFC standard. These endpoints should be authenticated with Keycloak, and only resources granted to the platform/proxy server should be returned to it, once authenticated. For more information consult the data permissioning specification associated to this network.

The information they share SHOULD be filtered by the scopes assigned to the proxy, and these scopes SHOULD be assigned per user (per enterprise).

Data returned by each endpoint MUST be serialized in valid RDF (e.g. JSON-LD) and should use the DFC ontologies for serialization, e.g.

{
    "@id": "https://mywebsite.com/enterprises/1/",
    "@type": "dfc-b:Enterprise",
    "dfc-b:name": "My cool enterprise"
}

@id here is a URL uniquely identifying the enterprise, serialized as a dfc-b:Enterprise.

It's a good idea to paginate your data if there's a lot of it. If you include in the response the key next, the cache refresh will follow this link to continue importing data until next is not returned or is returned with the value null.

The data-server SHOULD accept a query parameter, enterpriseUrlid, which will be given as the url-encoded enterprise urlid (@id). Accepting this query parameter, the data-server SHOULD filter the data returned to the proxy server to include only the information relevant to that enterprise/user. On ReadProducts for example, the key for this is dfc-b:suppliedBy, as a supplied product is supplied by an enterprise.

Authenticating requests with Keycloak

To authenticate your cache refresh endpoints with Keycloak:

• fetch the Keycloak JWKS (JSON Web Key Sets) by making a GET request to {KEYCLOAK_URL}/realms/{KEYCLOAK_REALM}/protocol/openid-connect/certs. The keys are public.
• use a library (typically) to decode the encoded token which the proxy server gave to you in the Authorization header of the request (the value comes in the format Bearer {encoded_token}.)
• validate the claims of the client. Client ID must be present.
• the Client ID is the WebID of the proxy server making the request. You can use this information to retrieve the scopes which have been assigned to the proxy server by a user.

Implementing webhooks for keeping proxy servers up-to-date

A Postman collection has been created to accompany these docs which provides example events. It has not yet been published publicly.

The webhook is called when one of three things happens:

1. a resource which the proxy has access to has been created or updated. In this case, the data-server should POST the serialization of the object to the webhook directly, using the eventType "update". A PUT operation will be performed with the data given.
2. a resource which the proxy has access to has been deleted or permission to the object has been revoked. In this case, the data-server should POST the @id and @type of each resource revoked in a list under the key objects. It should use the eventType "revoke".
3. the proxy has had a scope permission given or revoked on the data-server (for a given enterprise). In this case, the eventType should be "refresh". The enterpriseUrlid should be the @id of the enterprise which granted/revoked their data.

In all cases, the endpoint on the proxy server is /djangoldp-dfc/webhook/, and the request type is POST. Webhook requests MUST be authenticated, using a valid Keycloak client from the KEYCLOAK_URL and KEYCLOAK_REALM. To do so, set the Authorization header of your webhook request to Bearer {encoded_token}. The encoded token can be retrieved from Keycloak by making a POST request to KEYCLOAK_URL with the following body:

client_id=$KEYCLOAK_CLIENT_ID,
client_secret=$KEYCLOAK_CLIENT_SECRET,
grant_type="client_credentials",
scope="WriteEnterprise"

The Content-Type should be x-www-form-urlencoded.