Skip to content

migrid-ux based peers ui and all supporting logic#453

Open
albu-diku wants to merge 41 commits into
nextfrom
feature/admin-ui
Open

migrid-ux based peers ui and all supporting logic#453
albu-diku wants to merge 41 commits into
nextfrom
feature/admin-ui

Conversation

@albu-diku

@albu-diku albu-diku commented Feb 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

@albu-diku albu-diku force-pushed the feature/admin-ui branch 3 times, most recently from 910ecac to 3c361ed Compare February 24, 2026 19:29
@albu-diku albu-diku force-pushed the feature/admin-ui branch 4 times, most recently from 1b0ef8a to d94b039 Compare March 30, 2026 14:53
@albu-diku albu-diku changed the title WIP: admin UI migrid-ux based peers ui and all supporting logic Mar 31, 2026
@albu-diku albu-diku marked this pull request as ready for review April 13, 2026 13:33
Martin-Rehr
Martin-Rehr reviewed Apr 17, 2026
View reviewed changes
Comment thread mig/shared/configuration.py
Martin-Rehr
Martin-Rehr reviewed Apr 17, 2026
View reviewed changes
Comment thread mig/lib/templates/__init__.py
@Martin-Rehr

Martin-Rehr commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

I think this PR should be split into three PRs:

  1. Core configuration changes (introduction of RuntimeConfiguration)
  2. Core templating functionality (introduction of ninja templates)
  3. Peers specific functionality

@albu-diku

albu-diku commented Apr 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I think this PR should be split into three PRs:

1. Core configuration changes (introduction of RuntimeConfiguration)

2. Core templating functionality (introduction of ninja templates)

3. Peers specific functionality

Will get right too that! :)

Let me share a little context: the separation you noted is definitely there. Specifically, RuntimeConfiguration and ninja templates were separate PRs originally that were folded into this as necessary to support and undertake the rest of the work.

So with that said, it should not actually be too difficult.

  • RuntimeConfiguration: it is here and was to a large extent reviewed by @rasmunk already.
  • Same goes with templates, will dust off the old PR (reopening if needed)

The only effort that may be needed, particularly in the latter case, is bring the separated versions in-sync with what's in this branch. There was some iteration and evolution on those bits in the course of rest of the work. I am much much happier with what's here, so it's basically copy the files as they are from this branch into separate branches and push those up.

Thanks for the first look.

@albu-diku

albu-diku commented Apr 19, 2026

Copy link
Copy Markdown
Contributor Author

I think this PR should be split into three PRs:

1. Core configuration changes (introduction of RuntimeConfiguration)

2. Core templating functionality (introduction of ninja templates)

3. Peers specific functionality

Have now done this - PR description is updated with links to those this work dependent upon.

This was referenced Apr 24, 2026
Merged
Closed
albu-diku added 5 commits June 15, 2026 16:28
@albu-diku
Implement support infrastructure for peers ui.
71c5f6f 
Original commit messages:

Cover peersactionwith a basic test.

Heavily extend (any|every)thing peer related within accountreq.

Add direct listing functions for pending and accepted peers.

Test the creation of the pending_peers file via picklesupp. Do so
using a raw comparison of the pending peers file

Introduce a pending peers fixture and assert that manage_pending_peers
produces what is in it then consume it
when testing the pending peers listing function.

make common the identical parts of ONWRITE and ONREAD hints selection

Cnvert template support code to package loading.

* expand template packages up-front in TemplateStore.populated()
* template packages is UNPOPULATED by default
* back template listing onto back_packages and rejig some

The following from original commit was addressed by squashes:

template loading and rendering works, but listing templates needs to be
restored as does priming ...am most of the way convinced that per-package
cache dirs want to be done as part of that preparation process and not a
call to os.makedirs() when we do an average "get a template store" call

do all template cache directory ahead of time during priming

explicitly create configuration for testing of tempate defaults

do this to avoid any changes spilling over - this seemed to be happening
with the previous phrasing in the tests though it is not entirely clean
to me how it can have happened given how the test support code should be

repair support for html fragments in the output path and test the results

Expose template rendering via a handler.

fixup: ensure tmplinterface works when called via the cgi-bin

serve html from tmplinterface

make tmplinterface accessible via cgi

Wire the ability to specify a template cache dir through shared/install.

Allow a value of AUTO for the option which, when read, will construct
the necessary path when the configuration is loaded.

avoid config loading reaching out for log file in the templates cli

template cli: repair prime and restore a "cache" command. add tests

rework the tests added for the CLI

repair the first peersaction test

Place the priming code into a method attached to TemplateStore.

rework tmplinterface

Get tmplinterface returning accepted peers and cover with real data.

Get requested peers being returned via tmplinterface.

switch tmpl interface tests to snapshots

add datainterface

get to a working, JSON **responding**, datainterface

datainterface - implement /peer/new and alwats return JSON with a status

Implement the deletion of accepted peers.

restore test suite to full function (also meaning peersaction itself)

Handle peer deletion as a bulk POST to a .../delete endpoint.

Unify the lack of a need to prefix routes between (data|tmpl)interface.

Define peers UI backend routes as though overrides for the migux package.

update datainerface documentation

Pull out much of the finer grained user test support code into a usersupp.

Place the error key at the top level and nest a data key in JSON responses.

Decide a location for peers app assets and reference its bootstrap.

implement the peers summary endpoint

Set only the necessary content-type header across (data|tmpl)interface

Enforce a header check when expecting JSON responses via content_format.

Remove unused import.

Remove repetetive boilerplate from all (data|tmpl)interface tests.

Switch provisioning of pending peers over to taking a DN argument.

WIP peer acceptance

fix logger local in accountreq

fix condtional attr access in shared/configuration

fix condtional attr access in datainterface

fixup - shared/configuration

Implement deletion of requested peers.

Provide well contained user peers and user pending peers assertions.

We already tarted to arrange this for pending peers, but take a moment to
do this for accepted peers as well. As "User" as an infix in the assertions
to avoid any potential confusion and contain the argless hint application
to the loaded pickle files within the two assertion functions.

Remove the pending user entry when we reject the peer.

rename test support pending user retrieval

Cover peer creation and removal of the pending user file on accept.

Get migux package installed automatically across all local invocations.

replace calling out to wget with download via urllib3

replace calling out to unip with the use of zipfile

urllib3 in local reqs

Axe unused tests and improve comments in datainterface.

restore wget fetch but tie its use to an argument

this is done to allow operation in an environment without urllib3

Move to a feature specific requirement file for migux.

support writing the indexurls to a file

fixup

Always use fresh packages and allow for postinstall (add one for migux).

Be more straightforward when setting templates values from a config.

Make config file arugment required in templates CLI.

Arrange covering empty templates base packages from config and fix it.

enable migux by default

Add a comment for a new commit id.

Nudge to latest version.

Nudge to latest version.

Switch local package installation to a single local file url.

Nudge to latest version.

Nudge to latest version.

Attempt to persuade dumb-pypi to execute within GitHub actions.

Restore fixturesupp chunk lost to rebase.

Remove duplication due to rebase.

Axe wrapper properties for config vars within accountreq tests.

Axe remnants of earlier iterations from tmplinterface.

Fix some fallout from splitting peersaction.

try to account for image differences in CI

maybe?

Are we hitting differences between the CI targets?

Ensure the base now date will always allow month replacement.

(cherry picked from commit c628b6c)

Avoid a variable that can be two types to appease linter.

Avoid a variable that can be two types to appease linter.

Align some request handling basics across data and tmpl interfaces.

alignment 2/x

repair invoke in local dev environments

Remove arg fiddling in tmplinterface and use explicit tmpl route variable.

Fix tmplinterface test case name.

Rework template handlers for direct return of render info.

Use "staging" as the name for the dir serving that purpose within envhelp.

Axe the need for urllib3.

Repair "Rework template handlers for direct return of render info."

A fat fingered convert key broke processing fields.

Restore filtering of peer listings.

Repair make unittest after 'Use "staging" as the name for the dir...'.

Further refinement to plugin interface and a bunch of polish.

Relocate RuntimeConfiguration to align with further iterations.

strip out the _send_email function pointer

Align to RuntimeConfiguration as a proxy object.

Rework email send as a reference in context and provide its fake.

Use the runtime configuration and its persistent context as the means
to expose the email sending logic to codepaths making use of it. This
allow centralised control over the active email sending implementation
without having to thread extra arguments through internal codepaths.

As a result, we are able to centrally intercept email sending. Do this,
and as a consequence sprinkle a bunch of additional assertions into calls
that happen to send email but this side-effect was completely uncovered.

Add a means for tests to check the the number of emails sent and whether
an email was sent to a particular recipient. For cases where the email
sent is not relevant to what is being tested, there is an escape hatch.

Repair email sending in peersaction thus activating it for datainterface.

Use a flag to make peer processing work as before by default.

Overhaul peer validation thereby restoring checking lost during rebase.

Return 400 status codes when a failure occurs during a new peer creation.

appease checks on CI by reworking some code for pylint's benefit alone

Add documentation to datainterface and module header to tmplinterface.

Unify the common portions of request coalescing into a reqinfo lib.

pull out the one-off peer_user_dict given there is a better alternative

use a better on disk fixture format for the single peer fixture

in the process, find out that expire needed to be saved as an mktime
value but ended up being saved in iso format traversing datainterface

Add support for filtering by expire.

contain the AND and OR supporting search within tmplinterface

support requested peers filtering by kind

Nudge to latest version.

patch booleanify so its resilient against in-practice data

Nudge to latest version.

cover peer import support in peersaction

restore the peers import route
@albu-diku
Make use of migux@0.7.0.
dfc8649
@albu-diku
apply change requested during review
e053037
@albu-diku
Nudge version.
0434614
@albu-diku
adjust test so it match the branch after rebase
23a3247
rasmunk
rasmunk reviewed Jun 16, 2026
View reviewed changes
Comment thread envhelp/scripts/expand_sources.py Outdated
rasmunk
rasmunk reviewed Jun 16, 2026
View reviewed changes
Comment thread envhelp/scripts/expand_sources.py Outdated
@albu-diku

albu-diku commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Commenting at a higher level: an in-person discussion with @rasmunk covered the nature of the convert_* functions in datainterface and the fact that they are solely responsible for normalising the data that will end up in handlers which is needed to account for the way the cgi/wsgi stubs that run ahead of the interface functionality do things.

This use made sense, but we discussed ways to make the code clearer. Of particular note was making a clear split between the definitions of handlers and the, optional, definition of convert functions to call. This has been addressed.

rasmunk
rasmunk reviewed Jun 22, 2026
View reviewed changes
Comment thread tests/support/usersupp.py Outdated
albu-diku and others added 18 commits June 22, 2026 11:49
@albu-diku
Bump to a -ux version including accepted peer edit.
6876cf1
@rasmunk
Remove the unused _RequestInfo.create handler option and expand on th…
d354ba6 
…e function description and the expected values for the defined arguments. Also reformat imports
@rasmunk
Add missing assert to UserAssertMixin._provision_pending_peer
78bbcb7
@rasmunk
Remove legacy copied jsoninterface header
36e493c
@rasmunk
Amend the peer_dict_from_fields to ensure that it returns a canonical…
75062a7 
… user, such that a new peers request creates a new peer which format matches how process_peer_action loads the peers
@rasmunk
Add the missing extra_accepted fields to the canonical_user creationg…
5aadaab 
… when creating peer_dict_from_fields
@rasmunk
Restructure the datainterface such that handler and route lookup erro…
bb68a4b 
…rs are returned as early as possible. Also update the handle_POST_peers_new such an requested or accepted peer does not allow does not allow duplicates of the identifying email. Furthermore ensure that when a requested peer is deleted that even if the user_pending peer has been removed, we should still remove the clients personal user_settings/client_id/pending_peers file. Finally the datainterface is reformatted and linted with the make fmt/lint targets
@rasmunk
Add datainterface TODOs about seperating the input validation and the…
839da85 
… underlyng library logic
@albu-diku
Bump -ux version.
76ec487
@rasmunk
Correct the returned handle_POST_peers_new key to align with the expe…
dd1ad54 
…cted fields of mig-ux
@albu-diku
Bump to -ux version that allows an empty error string.
5fbd927
@rasmunk
Add the convert_POST_peers_accepted_fetch normalizer and add a fallba…
766f614 
…ck error message when the returned status from the datainterface handler is not 200
@albu-diku
Bump -ux version for import peers tooltip and CSV example.
34caa44
@rasmunk
Add the remove_peers_from_client and add_peers_to_client functions th…
9491831 
…at support updating pending and accepted peers with ensured locking. Refactor the manage_pending_peers to use these
@albu-diku
Bump -ux to a candidate of its finalised repository.
855a4c6
@rasmunk
Rework peers related accountreq functions to have a common entrypoint…
881f549 
… into peers pending and accepted manipulation for certain workflows
@rasmunk
Add helper functions to create proper handler responses. Also unwrap …
e8295f2 
…the process_peer_action for a number of handlers and introduce their own specific input validation. Furthermore introduce a number of convert functions to align the expected request input the requirements of the existing validation expectations. Also remove the user_pending state mgnt after discussion with other contributors about it is expected to be handled externally from the peers API but will instead be handled via the migadmin functionality
@rasmunk
Merge branch 'feature/admin-ui' of github.com:ucphhpc/migrid-sync int…
fa9315e 
…o feature/admin-ui
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Martin-Rehr Martin-Rehr Martin-Rehr left review comments
@rasmunk rasmunk rasmunk left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@albu-diku @Martin-Rehr @rasmunk