Challenges and Improvements for Large-Scale Labgrid Deployments and Client Usability

[ozan956]

Hi everyone,

I would like to share my current experience and some challenges I encountered while designing and scaling a labgrid setup. My intention is to start a discussion and explore how we can improve usability, especially for larger deployments.

Target Setup

The setup I am working on is designed for approximately 40 clients and 5 exporters, distributed across multiple locations.

These users are not only embedded Linux engineers. They include board designers, electrical engineers, and application-level developers. Therefore, it is important that they can use the system without needing to understand labgrid internals in detail.

Current Client Experience

Let’s walk through the typical flow from a client perspective.

1. Resource Discovery Confusion

After installation, the user tries to list available resources and sees something like:

adi-test/group-1/NetworkSerialPort
adi-test/group-1/NetworkUSBDebugger
adi-test/group-1/NetworkPowerPort
adi-test/group-2/NetworkSerialPort
adi-test/group-2/NetworkUSBDebugger
adi-test/group-2/NetworkPowerPort

At this point, the concept of “groups” becomes confusing.

• Users may assume they can directly use these group names.
• However, they later learn that resources must be mapped into "places" before use.

This creates an unnecessary learning step and introduces ambiguity.

2. Redundancy Between Groups and Places

There are two common ways to structure groups:

a) By hardware type:

adi-test/consoles/NetworkSerialPort
adi-test/debuggers/NetworkUSBDebugger
adi-test/power-ports/NetworkPowerPort

This structure provides no information about which resources belong to which device.

b) By device:

adi-test/SC598_EZKIT-01/NetworkSerialPort
adi-test/SC598_EZKIT-01/NetworkPowerPort
adi-test/SC598_EZKIT-01/NetworkUSBDebugger

This is much clearer and easier to understand.

However, even when groups are already well-structured, users still need to redefine the same mapping again as “places” on the client side.

This creates redundancy, additional configuration effort and confusion for new users.

3. Environment on Client Side

Another major issue is the need for an environment on the client side.

Each client is expected to:

• Create and maintain their own environment
• Define tool configurations (e.g., OpenOCD)
• Specify coordinator settings
• Configure access details

In practice:

• These environments are mostly identical for all users
• Boards and setups are rarely changing
• Yet every user maintains their own copy

This leads to duplication, inconsistency and maintenance overhead. For a setup with many users, this is not scalable.

4. SSH and Access Management Complexity

Each client also needs:

• SSH access to exporters
• Proper SSH configuration
• A correctly defined NetworkService with a username

This introduces two challenges:

• SSH setup must be repeated for each user and exporter
• Environment files become user-specific and this reduces reusability

5. Configuration Management Problem

Effectively, each client needs a local configuration database, including the labgrid environment configuration and SSH configuration. Managing this becomes especially difficult for users who are not familiar with these concepts.

Summary of Current Workflow

Today, a typical client must:

1. Install labgrid
2. Understand concepts like "groups" and "places"
3. Identify and acquire a place
4. Create or obtain an environment configuration
5. Configure SSH access
6. Customize settings (e.g., username for NetworkService)
7. Finally connect and use the board

This is a complex process, especially for non-expert users.

Proposed Improvements

To simplify the experience and improve scalability, I suggest the following:

1. Simplify Resource Abstraction

• Make the distinction between groups and places clearer, or unify them
• Allow exporters to define ready-to-use “places"
• Clients should not need to remap resources manually

2. Centralize Configuration

• Move environment configuration to the exporter side
• Maintain a shared configuration for all users
• Avoid duplication across clients

3. Improve SSH Handling

• Simplify or centralize SSH configuration
• Reduce the need for per-user customization
• Align CLI and pytest workflows

4. Reduce Client Responsibilities

The ideal client workflow should be:

1. Install labgrid
2. Acquire a place
3. Connect to debugger or console
4. Use the device

No deep knowledge of labgrid internals should be required.

Conclusion

The goal is to make labgrid accessible and easy to use for all clients, not only those familiar with its internal architecture.

By reducing configuration overhead and centralizing responsibilities, we can improve usability, reduce maintenance effort and scale more effectively across teams and locations

I would especially like to hear from teams working on similar large-scale setups.

• How do you manage configurations across many users?
• How do you handle SSH access and permissions at scale?
• Do you centralize configurations, or keep them client-side?
• What are your main pain points and best practices?

It would be very valuable to discuss these use cases and experiences.

Looking forward to your feedback.

[sjg20]

Thanks for writing this up. My brief comments:

1. I tend to use roles, which are kind-of a superset of places. I haven't tried to use groups.
2. Yes the separate configuration doesn't seem well motivated. In my case I have it on an NFS share, but it isn't ideal. I agree it should be unified, e.g. by allowing the 'environment' to go in the export file, with perhaps some overrides in the client. In my case I use a single environment file for the whole lab, which has required patching Labgrid.
3. That would be nice! What are you suggesting? So sort of relay on the exporter?
4. Agreed

Re your questions (I have various PRs pending):

Q1: How do you manage configurations across many users?
I use an NFS share which all machines can see

Q2: How do you handle SSH access and permissions at scale?
Just copying credentials around...not great

Q3: Do you centralize configurations, or keep them client-side?
The config is centralised - with the export and environment yaml (and a list of places) in the same NFS directory.

Q4: What are your main pain points and best practices?
Getting traction on PRs. For now I am just doing my own thing.

My lab is here:
https://lab.u-boot.org/

[ozan956]

Hi Simon,

Thank you very much for your detailed feedback!

1. On roles vs. groups/places
   What I was trying to highlight is that we currently have groups on the exporter side and places on the client side, and in practice these are usually initialized to match each other. Because of that, having a single naming or classification mechanism (ideally defined on the exporter side) could simplify things significantly. It should be possible to configure everything on the exporter's side. I would be interested to hear more about how you use roles in your setup.

2. On configuration and access management
   Yes, I assume you had to patch certain aspects (e.g., around network service username requirements). Are you effectively configuring all machines in the lab for a single user, or how are you structuring access? Did you propagate your own user across all systems, or do you treat each machine as an independent "client"? As you also mentioned, this becomes quite challenging to manage, especially across multiple offices and labs.

3. On client–exporter interaction
   The specific problem I would like to solve is the direct communication path between the client and the exporter. In my view, the client should not need to reach the exporter directly, nor know exporter hostnames, SSH users, ports, or local permission details.

As you mentioned, one possible approach would be a relay/agent model where the client communicates only with the coordinator, and the coordinator forwards the requested operation to the relevant exporter-side agent. The exporter would then perform the local actions on the hardware.

Labgrid already follows a partially similar approach, as both exporters and clients connect to the coordinator. In practice, however, the coordinator is mainly responsible for resource discovery and reservation, while the actual execution still often involves direct client-to-exporter communication.

This model could be extended further by routing all actions through the coordinator, effectively turning exporters into backend agents. This would make labgrid easier to scale and easier to explain to non-expert users, as they would interact with "boards" or "targets" rather than exporter-specific infrastructure details.

---

Overall, I agree with your observation, many setups right now seem to rely on custom patches to achieve a stable workflow. Without them, it can be quite difficult to run a robust shared environment. That’s why I think it’s very valuable to discuss these use cases and potential improvements together.

[sjg20]

Hi Simon,

Thank you very much for your detailed feedback!

Thanks for writing this up, Ozan!

1. On roles vs. groups/places
   What I was trying to highlight is that we currently have groups on the exporter side and places on the client side, and in practice these are usually initialized to match each other. Because of that, having a single naming or classification mechanism (ideally defined on the exporter side) could simplify things significantly. It should be possible to configure everything on the exporter's side. I would be interested to hear more about how you use roles in your setup.

You're right that for single-exporter labs there's redundancy, and I assume most labs are like that. The split makes more sense when a place aggregates resources from multiple exporters (e.g. serial from one host, power switch from another), or when resources are moved between places without restarting the exporter. But for a typical lab where one exporter owns all the resources for a board, defining the same thing twice does feel awkward.

One option might be to have optional, auto-creation of places from exporter groups, with explicit place definitions still supported for the multi-exporter case. That would simplify the common path without losing flexibility.

In my lab I have ~40 boards (although a few are just QEMU), and roles let me use a command like ub-int bbb that doesn't need to know the underlying place name. Roles let me have multiple roles share a place (e.g. different U-Boot builds for the same board, such as ff3399 and vbe).

PR: #1411

lab config:

e3b7760

2. On configuration and access management
   Yes, I assume you had to patch certain aspects (e.g., around network service username requirements). Are you effectively configuring all machines in the lab for a single user, or how are you structuring access? Did you propagate your own user across all systems, or do you treat each machine as an independent "client"? As you also mentioned, this becomes quite challenging to manage, especially across multiple offices and labs.

Currently a single shared user ('gitlab-runner') on the 'runner' machine which can ssh to the exporter machine. It isn't great, but OK for my simple setup so far.

3. On client–exporter interaction
   The specific problem I would like to solve is the direct communication path between the client and the exporter. In my view, the client should not need to reach the exporter directly, nor know exporter hostnames, SSH users, ports, or local permission details.

As you mentioned, one possible approach would be a relay/agent model where the client communicates only with the coordinator, and the coordinator forwards the requested operation to the relevant exporter-side agent. The exporter would then perform the local actions on the hardware.

Labgrid already follows a partially similar approach, as both exporters and clients connect to the coordinator. In practice, however, the coordinator is mainly responsible for resource discovery and reservation, while the actual execution still often involves direct client-to-exporter communication.

This model could be extended further by routing all actions through the coordinator, effectively turning exporters into backend agents. This would make labgrid easier to scale and easier to explain to non-expert users, as they would interact with "boards" or "targets" rather than exporter-specific infrastructure details.

Overall, I agree with your observation, many setups right now seem to rely on custom patches to achieve a stable workflow. Without them, it can be quite difficult to run a robust shared environment. That’s why I think it’s very valuable to discuss these use cases and potential improvements together.

Yes I strongly agree :-) The current model where the client SSHes directly to the exporter is not ideal and one of the main reasons I haven't made my lab available for interactive access. A coordinator-mediated execution model would be a significant improvement from this POV.

The hard part is probably the streaming use cases (console, video) where direct connection is currently used for latency. Those would need a forwarding mechanism through the coordinator. But for the request/response style operations (power, IO, file transfer) it should be easy enough to route through the coordinator.

This would also make it possible to expose labgrid through a higher-level service without users needing direct SSH credentials at all.

One other thing to mention is that I found it painful to have to fully specify each USB device. When something is broken, it it hard to figure out which USB hub/port to replug, etc. So I created this:

#1854

It may be that my USB-heavy lab is unusual, not sure.

• Simon

[dlech]

This model could be extended further by routing all actions through the coordinator,

Isn't this what the labgrid-exporter --isolated option does?

[ozan956]

This model could be extended further by routing all actions through the coordinator,

Isn't this what the labgrid-exporter --isolated option does?

I think you are right that labgrid-exporter --isolated is an important part of the solution, and it may already cover many practical cases. My understanding is that --isolated mainly improves the network access model. That is very useful for isolation.

The part I am still trying to separate is whether this also solves the execution model. As far as I understand it, the coordinator still mainly handles discovery, state, release, reservations, etc. Actual resource access or command execution still seems to happen through an SSH or proxy path from the client side.

For example, operations such as bootstrap still appear to run through the client-side driver. With --isolated, the SSH path may go through the proxy or exporter instead of a directly reachable resource host, but it is still not quite the same as the coordinator dispatching the action to an exporter.

This also leaves some user and permission questions. A shared exporter user such as labgrid can make setup simpler, but it can make auditing and per-user isolation weaker. For example, multiple users writing as the same Unix user could also create overwrite, or cleanup problems.

There is also a similar issue with the --isolated when user identity is passed through end to end. In that setup, the client connects to the proxy as their own user and then reaches the exporter as the same user. This is good for traceability, but it still means every exporter needs the relevant users and keys. As the number of users and exporters grows, the key distribution and permission management problem remains. So both approaches have trade-offs.

So I currently see two related but separate layers: network isolation, which --isolated addresses, and coordinator-mediated action execution, which would be a different model. In that second model, clients would send action requests to the coordinator, and the coordinator would dispatch them to the relevant exporter. That could reduce the need for client SSH access to exporters and make the user-facing workflow simpler.

Of course, this would be a larger change, and some use cases may still need a more direct path for latency and throughput. But request-response operations such as power control, file transfer, bootstrap, or some hardware actions might be interesting candidates.

I may also be missing part of the intended model here, so I would be interested in your view. Do you think --isolated is already enough for this kind of multi-user setup, or do you see value in adding some form of coordinator-mediated execution for (selected or every) actions?

[dlech]

I may also be missing part of the intended model here, so I would be interested in your view. Do you think --isolated is already enough for this kind of multi-user setup, or do you see value in adding some form of coordinator-mediated execution for (selected or every) actions?

I don't know. I'm a few steps behind you and Simon. I've just been using local setups so far, so I haven't had to deal with that issue yet.

At BayLibre, we've been looking at using headscale/tailscale to make a virtual network for all exporters/coordinators/clients. This adds another layer of admin and potential problems though.

---

In general, I've run most of the same issues/pain points listed in the OP.

Points 1 and 2 I don't see as such a big deal since just one knowledgeable person has to set those up in the first place, then anyone can use them without having to set them up again.

For points 3 and 5, the direction I have been going is putting as much as possible in the .yaml config file. For example, you can put the place name and the coordinator hostname in there, so one only has to export LG_ENV=my-config.yaml and then they can use labgrid-client without extra args or other environment variables.

This also includes a Python file for strategy definitions. So for distributing, I've been looking at making a Python project and distributing it with git. This also includes a pyproject.toml file for setting up a venv with labgrid. So steps for letting someone else use it would be:

1. git clone ...
2. cd my-project
3. uv sync (or whatever your Python tool of choice is)
4. . .venv/bin/activate
5. export LG_ENV=my-config.yaml

Point 4 I haven't got to yet, but I am not looking forward to having to set up a user for everyone I want to share my exporters with. My thought here is to just create a labgrid user instead and have everyone use that. And it still requires getting a public key from every user and installing it on the exporter.

---

Another pain point for me has been setting up the exporters. I would ideally like to have one exporter per system under test (reasoning being to minimize disruptions to other users when tweaking a config), which means lots of exporters. I like the idea of using docker containers for this. One thing I haven't solved there is how best to deal with hot-plugable devices (maybe something like this) and you still have to deal with udev rules on the host system, so it isn't quite a simple as spinning up a container.

[sjg20]

Quick update: I've made the lab available for interactive remote access since my earlier comment, and a few things have shaken out that are relevant to this discussion.

Isn't this what the labgrid-exporter --isolated option does?

Agreeing with @ozan956 - --isolated helps with network isolation (proxying through the exporter) but doesn't change the execution model. The client still drives drivers locally and SSHes to the exporter to run individual commands. Per-user identity propagation, audit trails, and exposing labgrid as a service all still need true coordinator-mediated execution.

---

What I've set up since the previous comment

I now have a working remote-access setup that addresses some (not all) of the pain points without needing protocol changes:

• Network: self-hosted headscale (open-source Tailscale control server) on a public host. Lab + clients all join the tailnet, which gives a stable IP that works through NAT. No port forwarding, no per-client config.
• Auth: a single shared labgrid UNIX account on the lab host, with each user authenticating via their own SSH key. Each authorized_keys entry forces a small wrapper script (via command="...") that refuses interactive shells and logs every command to an audit file, attributed to the user. Revocation = remove one line.
• Serial audit: small labgrid changes I want to discuss separately upstream:
  • New optional user field in ExporterSetAcquiredRequest so the exporter knows who acquired a place
  • LG_SERIAL_TRACE_DIR env var that, when set, makes the exporter pass trace-both to the per-acquire ser2net. Result: per-board, per-user serial trace files capturing every byte both ways

None of this changes the fundamental architecture (it's still SSH/TCP under the hood) but it gives a usable remote workflow today and per-user accountability. It also demonstrates the value of "clients only need one well-known endpoint" - which is essentially what coordinator-mediated execution would give us natively, without needing a VPN.

---

Lab access for anyone who wants to try

Minimal client setup is now small - no U-Boot tree required for read-only access:

1. Install tailscale and email me your desired username + SSH public key
2. I send back a tailscale preauth key
3. sudo tailscale up --login-server=https://headscale.u-boot.org --auth-key=<key>
4. Install labgrid from U-Boot integration [WIP] #1411 (the U-Boot integration branch - the lab's drivers come from this PR, and it ships an example_env.cfg already configured for the kea coordinator with all ~50 boards as RemotePlace entries):

   git clone https://github.com/labgrid-project/labgrid.git
   cd labgrid && gh pr checkout 1411
   pip install --user .
   

5. Three environment variables:

   export PATH=$HOME/.local/bin:$HOME/labgrid/contrib/u-boot:$PATH
   export LG_COORDINATOR=100.64.0.1:20408
   export LG_ENV=$HOME/labgrid/contrib/u-boot/example_env.cfg
   

6. labgrid-client console -p bbb (or any of ~50 boards), or ub-int -T bbb for the full strategy without rebuilding U-Boot.

Total client footprint is a few MB; no U-Boot tree, no buildman, no per-machine setup. Access is currently manual rather than self-service; could be opened up with OIDC if there's interest. Documented in contrib/u-boot/index.rst of #1411.

--

Now I'm wondering what it would take to keep the environment in the exporter...

[ozan956]

Hi David,

Points 1 and 2 I don't see as such a big deal since just one knowledgeable person has to set those up in the first place, then anyone can use them without having to set them up again.

I believe the challenge begins when you need to manage or modify the setup over time. For example, whenever a new board is introduced, all clients need to update their environments accordingly if they want to use it.

Centralizing this responsibility by having more on exporter side would be valuable. It allows a smaller group of “expert” users to handle configurations, while others can focus on using the system. This separation can scale well within a team.

---

For points 3 and 5, the direction I have been going is putting as much as possible in the .yaml config file...

This approach certainly improves usability, but it still has some limitations. For instance, if your environment file defines multiple targets and you attempt to acquire a single one using:

labgrid-client -c env.yaml -p PLACE1 acquire

Labgrid initially uses -p PLACE1 to select the matching role from the environment. However, during the acquire phase, it switches to operating on all RemotePlace entries defined in the environment, rather than only PLACE1. As a result, it becomes impossible to acquire a single place in isolation without bypassing the environment file and interacting directly with the coordinator.

This behavior suggests that the environment file is fundamentally designed to be client-oriented and tailored to specific use cases, rather than serving as a shared, scalable abstraction. I further think that environment files are designed to exist as separate instances for each use case for each user (multiple envs in every local).

---

Point 4 I haven't got to yet...

I had similar thoughts initially. Using a shared user account (e.g., a labgrid user) for all clients. However, this introduces another issue. When using commands like:

labgrid-client -c env.yaml -p PLACE1 scp images/u-boot-spl :u-boot-spl

the file is copied to a default location such as ~/u-boot-spl on the remote system. If multiple users share the same account and perform similar operations, they may unintentionally overwrite each other’s files.

To mitigate this, users would need to follow strict conventions, such as using separate directories (e.g., /home/labgrid/user1/). However, this shifts the burden onto users and introduces potential for human error, as there is no built-in isolation mechanism.

---

Another pain point for me has been setting up the exporters...

That's an interesting idea. Thanks for bringing this up. I will give it more thought.

[ozan956]

Hi Simon,

Now I'm wondering what it would take to keep the environment in the exporter...

So we would be moving from:

• exporter: knows local hardware and exports resources
• coordinator: knows places and matching
• client: builds targets, drivers, and strategies

to:

• exporter: knows hardware, target configuration, drivers, and strategies
• coordinator: still manages place ownership and matching
• client: becomes much thinner and mostly consumes what the exporter/coordinator provides

We would need a significant design change.

On the client side, the environment defines the "targets." These targets are bound to "places," which are created by clients and managed by the coordinator.

1 - We would need to move "targets" to the exporter. The client would then need to fetch the target, so the _get_target() logic would require a major redesign.

2 - The tricky part would be the strategies. There will likely be multiple design approaches to discuss.

3 - This change would also remove the ability to use multiple devices from multiple exporters within a single place. However, I am not sure whether there is a real use case where someone needs multiple resources from different exporters in one place, so this may not be a significant issue.

There are likely many other points to consider. In the end, it is doable, but it will be challenging.

[sjg20]

Hi Simon,

Now I'm wondering what it would take to keep the environment in the exporter...

So we would be moving from:

• exporter: knows local hardware and exports resources
• coordinator: knows places and matching
• client: builds targets, drivers, and strategies

to:

• exporter: knows hardware, target configuration, drivers, and strategies
• coordinator: still manages place ownership and matching
• client: becomes much thinner and mostly consumes what the exporter/coordinator provides

We would need a significant design change.

On the client side, the environment defines the "targets." These targets are bound to "places," which are created by clients and managed by the coordinator.

1 - We would need to move "targets" to the exporter. The client would then need to fetch the target, so the _get_target() logic would require a major redesign.

2 - The tricky part would be the strategies. There will likely be multiple design approaches to discuss.

3 - This change would also remove the ability to use multiple devices from multiple exporters within a single place. However, I am not sure whether there is a real use case where someone needs multiple resources from different exporters in one place, so this may not be a significant issue.

There are likely many other points to consider. In the end, it is doable, but it will be challenging.

Yes this is not easy. I have been prototyping a way for exporters to send their environment file to the coordinator, which then merges them into a single file, which it then provides to clients. Once I've tested it a bit more I will see if I can create a clean PR.