Run predicators envs in the browser via Pyodide + pybullet WASM#36
Open
yiyunliu wants to merge 1 commit into
Open
Run predicators envs in the browser via Pyodide + pybullet WASM#36yiyunliu wants to merge 1 commit into
yiyunliu wants to merge 1 commit into
Conversation
Split utils.py into a Pyodide-safe lite half and a slim CPython
wrapper, repoint env-load-path modules at the lite variant, and add
a proof-of-concept that boots predicators inside Pyodide and resets
each of the 18 envs in the dropdown.
Architecture:
- predicators/utils_lite.py is the bulk of the old utils.py minus
imports of torch, scipy.stats.beta, imageio, and the
pretrained-model SDKs. predicators/utils.py keeps those 11 helpers
(DelayDistribution subclasses, save_video/save_images, the four
beta_bernoulli_* functions, create_llm_by_name/create_vlm_by_name)
and re-exports the lite module via `from utils_lite import *`, so
CPython callers see no API change.
- structs.py imports utils_lite directly (not utils) to break the
cycle that triggers when utils_lite is loaded first under Pyodide.
torch and pretrained_model_interface are lazy imports inside the
two make_*_process methods that need them (via a shared
`_torch_and_delay` helper).
- 119 env-path modules (predicators/{envs, ground_truth_models,
pybullet_helpers, perception, execution_monitoring}) flipped from
`from predicators import utils` to `from predicators import
utils_lite as utils` so they don't pull torch into Pyodide.
- envs/__init__.py and ground_truth_models/__init__.py call
import_submodules(tolerate_import_errors=True) — catches plain
ImportError too, not just ModuleNotFoundError — so optional-dep
failures (torch / gym_sokoban / gymnasium-robotics) skip the
submodule with a warning instead of aborting auto-discovery.
- base_env.py defers `from predicators.pretrained_model_interface
import OpenAILLM` to inside the one method that uses it.
- setup.py: splits heavy ML/LLM deps into an `[ml]` extra; the base
install is the env-runtime slim set that's installable under
Pyodide via micropip. `[develop]` includes `[ml]` plus formatters.
Strict version pins (numpy/matplotlib/pillow/pyyaml) are qualified
with `sys_platform != 'emscripten'` so they don't conflict with
Pyodide's pre-loaded packages.
- predicators/third_party/{__init__.py, fast_downward_translator/__init__.py}
— empty package markers so `find_packages` actually ships the
third_party tree in the built wheel (the env path actually imports
`predicators.third_party.fast_downward_translator.translate`).
- Replace `assert not params` with `assert len(params) == 0` in 22
ground-truth option policies; numpy ndarray params trigger
"truth value of empty array is ambiguous" otherwise.
Browser POC (web/):
- web/app/{index.html, main.js} drive a Three.js renderer (via
urdf-loader) over Pyodide+pybullet-wasm. JSON.stringify-quoted
option names; mesh primitives with no mesh_url are skipped (they
were rendering as 1m box stand-ins for in-memory vertex meshes
like domino's top triangle).
- web/app/setup.py is the Pyodide-side bridge: per-env CFG
overrides for envs that need legacy options, multi-link visual
transforms, body/color diffs, begin_option/step_option protocol.
begin_option wraps the NSRT grounding in try/except so a sampler
raising KeyError or AssertionError returns {error: ...} instead
of crashing across the Pyodide → JS boundary; step_option also
catches the broad Exception after OptionExecutionFailure so a
policy raising mid-rollout doesn't leave _active_option armed.
Also guards `get_gt_options` with try/except in 3 call sites so
envs without registered options (the domino fan/ramp/stairs
variants) load cleanly with an empty option list.
- web/app/bundle.sh builds the predicators wheel, the gym 0.26 shim
wheel, and packs predicators/envs/assets/ into a tarball
(skipping the 32 MB tar rebuild when no asset is newer than the
existing tarball). The pybullet WASM wheel is fetched from
BasisResearch/pybullet-pyodide.
- All 18 envs in the dropdown reach `action_dim=...` when smoked
headlessly in Chromium via web/app/browser_smoke.mjs:
ants, balance, barrier, blocks, boil, circuit, coffee, cover,
domino, domino_fan, domino_fan_ramp, domino_fan_ramp_stairs,
fan, float, grow, laser, magic_bin, switch.
CI (.github/workflows/web.yml):
- One `import-check` job. Boots Pyodide in Node, installs the three
wheels, asserts every env in web/app/index.html's dropdown
registers as a non-abstract BaseEnv subclass. ~30 s wall. No
Chromium, no env construction, no asset extraction.
- import_check.mjs wraps the Python import in try/except writing the
traceback to stderr (which Pyodide's stderr handler captures), and
Node-level unhandledRejection/uncaughtException handlers print
the stack before exit — so a WASM abort surfaces a real message
instead of just `pyodide.asm.js:8`.
CI (.github/workflows/predicators.yml):
- unit-tests / static-type-checking / lint steps install `[ml]` so
they get the heavy deps slim utils.py imports at module top.
yapf / isort / docformatter stay slim.
mypy.ini: add `web` to the top-level exclude so mypy doesn't bail
out on "Duplicate module named setup" (./setup.py vs
./web/app/setup.py).
.predicators_pylintrc: add `web` to ignore-paths — the browser
bridge monkey-patches pybullet, accesses env internals, and uses
in-function imports; those checks don't help the POC.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Author
|
Btw I notice that the CI python version is really really old. It's using a code formatter that's not available for Python 3.13 anymore. Might open a separate issue to bump up the CI version. |
yichao-liang
reviewed
Jun 4, 2026
| state: State, objects: Sequence[Object], | ||
| params: Array) -> Tuple[Pose, Pose, str]: | ||
| assert not params | ||
| assert len(params) == 0 |
Collaborator
There was a problem hiding this comment.
why are we changing this and other similar assertions?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR pulls out the mostly ML independent components from utils.py into a separate file utils_lite.py. The goal is to allow the predicators environments to run in a web browser.
The python side change should be quite minimal other than minor import changes and the utils_lite.py split. I'm not sure if the web frontend should be part of the repo or maintained separately. I prefer the former so we can ensure future changes don't break the web code (I have included CI workflow to check for web imports).