Add end-to-end demo documentation and presentation#182
Merged
Conversation
added 30 commits
May 27, 2026 11:30
Add docs/demo with a Marp presentation that doubles as a self-guided tutorial, covering the full NDP journey for end users and admins: installation from scratch, identity and permissions (AAI + Affinities), using the Endpoint from the web, automating with the Python library, federation, and the secure network (NetBird) as a bonus. Includes a README with rendering instructions and a screenshot checklist. Content will be refined incrementally. Refs #179
Reuse the NDP logo (header) and partner-logo band (footer) from 'ndp ep - presentation.pptx' as layered slide backgrounds via CSS, so every slide carries the official branding. Document the assets and the --allow-local-files flag needed when exporting. Refs #179
Hand-crafted minimal SVG icons (NDP navy) for AAI, Affinities, NDP-EP, Federation, Python library and NetBird; shown next to each component in the 'pieces' slide. Removes the component-logos screenshot placeholder. Refs #179
Redraw the Component interactions slide as a clean navy SVG faithful to the C4 System Landscape view in ep-diagrams: single entry point routing to each service, with NDP-EP as the hub talking to AAI, CKAN, Affinities and Federation. Refs #179
Refined SVG: soft drop shadows, per-component icon badges, grouped 'Platform services' container, curved connectors with white-halo labels, and an emphasized NDP-EP hub. Refs #179
Switch from an NDP-EP-centric hub to the original left-to-right/top-down flow: Data User -> AAI (+Affinities for role) -> NDP-EP (with CKAN/S3 backends) -> Federation, plus a NetBird footnote. Keeps the polished styling (shadows, icon badges, halo labels) without making the EP the center everything radiates from. Refs #179
…roles
Roles live entirely in AAI (Keycloak): the token carries ndp_writer/
ndp_admin/group:{uuid}:tier. Affinities is a non-blocking relationship
registry the EP writes datasets/services/endpoints into. Correct the
interactions diagram (Affinities is a write target of the EP, not between
AAI and the EP), the pieces table, startup order, the Act 2 'grant role'
slide (now AAI), captions and the summary.
Refs #179
Top row: Data User -> AAI -> NDP-EP; bottom row spread full width: CKAN/S3, Affinities, Federation (the EP writes to them). Wider aspect fills the slide and fixes the footnote overflowing off the bottom. Refs #179
The paragraph overflowed the slide below the wide diagram. The diagram's own labels already tell the story, so keep the slide clean and move the narration into the speaker notes. Refs #179
Make the deck self-explanatory: keep the diagram slide clean and add a following slide that narrates the flow as numbered steps (sign in -> use the EP -> register relationships -> federate -> secure transport). Refs #179
- Remove the 'Ana' persona; use generic 'the user'. - Installation now leads with the common case (install only the NDP-EP and connect to the central NDP, which already runs AAI/Affinities/Federation); the full local stack is clearly marked as development/testing only. - Rename the 'Demo walkthrough' slide to 'Overview'. Refs #179
The common case runs the all-in-one Docker Hub image (API + UI) via 'docker run --env-file .env' — no git clone needed. The image does not include the data backends (catalog DB / object storage); AAI, Affinities and Federation are the platform's shared services. Compose with a profile is offered as the alternative that also starts a local database. Refs #179
Use the git clone + docker compose path (not the Docker Hub image) and add a prominent 'for system administrators' warning to the install slide. Refs #179
- Add docs/configuration.md: a complete reference of every .env variable (grouped: general, AAI/access, catalog/CKAN/MongoDB, S3, Kafka, Affinities, Federation, Pelican, Rexec, access requests, OpenTelemetry). - Demo: move 'what you run vs. what's shared' to its own slide and link the configuration reference; fix the template filename (example.env). Refs #179
…all slide - Convert all [📸 ...] markers to hidden HTML comments so they no longer render as literal text on slides (they remain as a guide; the README keeps the capture checklist). - Add a visible pointer to docs/configuration.md on the install slide. - Restore blank lines before slide separators after the conversion. Refs #179
…lues) Rewrite docs/configuration.md so each variable explains what it is, what it controls, where to obtain its value, and whether it is required — grouped by area and covering every variable the app reads (verified against the settings classes, incl. env-prefixed AFFINITIES_/REXEC_/OTEL_ and the os.getenv-based PELICAN_*). Add a 'minimum to get started' section. Also add the missing ENABLE_ACCESS_REQUESTS / ACCESS_REQUESTS_COLLECTION to example.env. Refs #179
…ndary table Replace the two bland bullets with a two-column comparison — what you operate (EP + catalog DB + optional object storage) vs. what the platform provides (AAI, Affinities, Federation) — which is a real takeaway for admins. Refs #179
The AAI repo uses .env_template (not .env.example) and needs a TLS cert step; add the git clone and --build per its README. Refs #179
AAI uses the -old repo and .env_template; Affinities and Federation use .env.example. Add the git clone step to each (verified against their repos and READMEs). Refs #179
Reference the common-case install and only show the difference (point .env at local services; bring up backends with a Compose profile). Refs #179
Clarify that 'docker compose up -d' starts only the Endpoint, and that optional local backends (mongodb, s3, kafka, jupyter, full) are added via Compose profiles — so users can run only the EP or the full set. Refs #179
New slide tabulates each profile and the exact services it brings up (none = EP only; mongodb, s3, kafka, jupyter, pelican, full), so users can run only the Endpoint or any subset of local backends. Install slide points to it. Refs #179
…token On the NDP-EP install slide, clarify that choosing CKAN as the local catalog requires a running/accessible CKAN and an admin API token (CKAN_API_KEY), whereas MongoDB needs none. Refs #179
added 28 commits
May 28, 2026 09:32
Use a consistent '<action> — <system>' title style across appendix slides and split the AAI API group/role assignment across two slides so it fits. Refs #179
…ack) Common case: the first admin is provisioned via the NDP platform onboarding (register the Endpoint). Full stack: assign ndp_admin directly in your own Keycloak. Note the historical onboarding (federation config_id + setup script, sci-ndp/NDP-EP) as a presenter note pending confirmation of the current portal. Refs #179
…d Keycloak) Common case: users are existing nationaldataplatform.com accounts (no creation needed). Full stack: create them in your own Keycloak (see appendix). Drop the duplicate create-user screenshot in favour of the appendix. Refs #179
Replace the 'Grant the role' slide with the real UI flow: a no-role user submits a Request access form; an admin approves it on the Access Requests page with a tier (Viewer/Writer/Admin). Add UI screenshot placeholders (22-request-access, 23-access-requests-approve). Refs #179
Split the request/approve slides so the UI screenshots sit on their own centered, larger (h:500) image slides instead of being shrunk next to text. Refs #179
Reorder Step 3 so Search (available to everyone) comes first, expanded into a landing-page slide and an options slide (category, Local/Global catalog, organization + Yours filters, publish/delete on owned items). Move the create/publish writer flow after it. Refs #179
Identity/login is already covered in Step 2; Step 3 starts directly with Search. Refs #179
…ides Replace the per-item create slides (organization/dataset/resource) with three '+ New' slides: an overview listing the six creation flows (Organization, Dataset, Service, URL, S3, Kafka) with what each is, followed by two slides listing the required and optional fields for each, grouped by catalog containers (Organization/Dataset/Service) and data resources (URL/S3/Kafka). Refs #179
…nings Replace the two grouped field slides with six detail slides — Organization, Dataset, Service, URL resource, S3 resource, Kafka topic — each defining exactly what every required and optional field means (sourced from the backend request models). Refs #179
The header logo, footer partner-logos band and the canonical layout used by the demo presentation are extracted from this deck (see docs/demo/assets/). Keep it in the repo so the brand assets can be re-extracted/updated. Refs #179
Per-field 'what it's for' descriptions plus a realistic example value (all values pass the backend validators: slug format, http(s):// URLs, owner_org = services for services, kafka_port 1-65535). Split heavier slides into required/optional so they fit; use a coherent atmospheric/research narrative. Refs #179
S3 resource: identification (3 fields) + S3 details (3 fields). Kafka topic: broker (3 fields) + options (3 fields). Refs #179
Already covered by 'The three roles' in Step 2; remove the duplicate slide from Step 3 and its 35-viewer-vs-writer screenshot placeholder. Refs #179
…t actions) Three slides: overview with prerequisites (S3_ENABLED + S3_* env vars and the writer role), bucket actions (create, list, search, info, delete, refresh), and object actions (upload, list, search by prefix, download, metadata, presigned URLs, delete) — verified against the UI source. Refs #179
Two new slides under 'Health and metrics' enumerate every field the Endpoint posts to METRICS_ENDPOINT — identity & version, catalog activity, host load, and infrastructure flags — gated by IS_PUBLIC=True. Sourced from api/tasks/metrics_task.py. Refs #179
When the Endpoint is installed via nationaldataplatform.com registration, the onboarding process provisions the CKAN sysadmin token, Pre-CKAN API key, etc. on the host so the EP can use them without manual setup. Refs #179
Mirror the Access Requests pattern: keep the text slides clean and put the NetBird dashboard + access-proof screenshots on their own centered, larger image slides. Refs #179
…d.io) Replace the six-slide NetBird section with a single slide that defines NetBird, links to https://netbird.io, and notes why it matters for NDP (one private overlay connecting components across machines, no per-service public ports). Drop the now-unused NetBird screenshot placeholders from the checklist. Refs #179
On every push touching docs/demo/** (and on workflow_dispatch), build the deck with Marp CLI (HTML + PDF) and publish the site to GitHub Pages so the slides are navigable from a public URL. Refs #179
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Marp-based, self-guided presentation/tutorial under
docs/demo/covering the full NDP journey for end users and administrators:Also includes:
docs/configuration.md— full reference of every.envvariable (purpose + where to get the value).example.env— adds the missingENABLE_ACCESS_REQUESTS/ACCESS_REQUESTS_COLLECTION..github/workflows/demo-deck.yml— builds the deck with Marp and publishes it to GitHub Pages on every push todocs/demo/**.The original brand PowerPoint is committed under
docs/ndp ep - presentation.pptx; the header/footer brand images live underdocs/demo/assets/.Test plan
marprenders the deck without errors (HTML + PDF), verified by the new GitHub Actions workflow.Closes #179