Per-contact force on scene objects

[jbae49]

I'm using ProtoMotions on IsaacLab to train an object-interaction policy and would like to feed per-contact wrench (force + torque, ideally with contact point and normal) on the scene rigid object back into the policy / reward, rather than just a per-link net force.

What I've checked so far:

• The IsaacLab spawn path in protomotions/simulator/isaaclab/utils/scene.py already creates a ContactSensor for each scene object with filter_prim_paths_expr set to the robot bodies, and protomotions/simulator/isaaclab/simulator.py reads force_matrix_w. That gives one net force per (object body, robot link) pair.
• Setting track_contact_points=True / track_friction_forces=True on the same ContactSensorCfg exposes contact_pos_w and friction_forces_w, but those buffers are shaped (num_envs, num_bodies, num_filter_shapes, 3) (one representative point and friction vector per (body, filter), not one entry per actual contact). For my use case (multiple contacts on the same box face from one hand) that still aggregates the information I need.
• The only IsaacLab API I've found that exposes truly per-contact data (point, normal, separation, force) is contact_sensor.contact_physx_view.get_contact_data(dt) on the underlying physx.RigidContactView. This might be more of an IsaacLab question than a ProtoMotions one, but I'd really appreciate any guidance you can share.

Two questions:

1. Is get_contact_data() considered reliable / stable across IsaacLab + IsaacSim versions for production training? The IsaacLab docstring on contact_physx_view says "Use this view with caution. It requires handling of tensors in a specific way.", which makes me a little nervous about depending on it for every-step reward shaping at 4096 envs.

2. Is there a recommended ProtoMotions-side pattern for this — e.g. wrapping get_contact_data() in the simulator and surfacing a per-contact wrench tensor on simulator_state? If not, do you consider per-contact data out of scope for ProtoMotions, in which case I should look at a sim with first-class contact arrays (e.g. mjlab)?

For context, the alternative could be moving to mjlab where MuJoCo's contact array seems like first-class, but I'd much rather stay on the ProtoMotions stack if get_contact_data() is solid in practice.

Any pointers, advice, or confirmation would be a huge help. Thanks!

[tesslerc]

Thanks for the detailed writeup. Before suggesting an implementation, I want to understand what level of contact detail you actually need.

Would an aggregate force per (object, robot body) pair be enough for your reward/observation? In IsaacLab, ContactSensor.data.force_matrix_w gives that kind of filtered pair aggregate, and contact_pos_w looks like it provides the average contact position for each sensor/filter pair rather than individual contact points: https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.sensors.html#isaaclab.sensors.ContactSensorData.contact_pos_w

If that aggregate pair view is enough, we could expose a cleaner ProtoMotions API around it.

Or do you need the true per-contact/patch entries, where one (object, robot body) pair may produce multiple contact points from contact_physx_view.get_contact_data(dt)?

Put differently: is the output you want a dense matrix over tracked robot bodies × scene objects, or a ragged list of individual contacts?

[jbae49]

Thanks for the response and clarification!

For my use case, I would need the true per-contact / patch entries rather than the aggregate pair view.

I want to reason about how contact forces, normals, and contact locations are distributed across the object surface. Multiple contacts from the same hand on the same object face can create different moments around the object COM, so reducing them to one averaged contact would lose information that is important for detecting potential rotation or slip.

So the output I am looking for is closer to a ragged list of individual contacts, ideally including contact point, normal, force, and the associated robot body / scene object!

[tesslerc]

I see. We want to think a bit more about whether we want to build this into the framework as a natively-supported-feature or not (there's a delicate tradeoff between coverage and ease of use).

That said, here's how we'd implement it on our side:

1. Edit the object contact sensors. Currently they only record contacts with other objects and the ground. Extending the filter_prim_paths_expr to also contain the robot bodies. Set max_contact_data_count_per_prim set high enough for dense hand/object contact.
2. Read raw contacts after physics update. After IsaacLab has stepped and _scene.update(dt=...) has run, call:

     forces, points, normals, distances, counts, starts = (
         sensor.contact_physx_view.get_contact_data(
             dt=simulator._sim.get_physics_dt()
         )
     )

forces/points/normals/distances are packed contact buffers. counts and starts tell you which slice belongs to each (object sensor body, filtered robot body) pair.

Then you can route this information from the simulator to the env (and then env context) and into the obs/reward/termination functions of your MDP.