Skip to content

client-side Python execution via Pyodide (feature-flagged)#111

Open
shubhamkhatri20 wants to merge 22 commits into
urban-toolkit:mainfrom
shubhamkhatri20:shubham/pyodide-migration
Open

client-side Python execution via Pyodide (feature-flagged)#111
shubhamkhatri20 wants to merge 22 commits into
urban-toolkit:mainfrom
shubhamkhatri20:shubham/pyodide-migration

Conversation

@shubhamkhatri20
Copy link
Copy Markdown

@shubhamkhatri20 shubhamkhatri20 commented Apr 29, 2026

Describe your changes

  • Migrates the Python execution layer from the backend sandbox to Pyodide (CPython compiled to WebAssembly), so standard data analytics workflows run entirely in the browser with no server required
  • Gated behind a feature flag (PYODIDE_ENABLED=true in .env) so server mode is completely unaffected
  • Adds IndexedDB persistence for uploaded files and provenance history across page refreshes

What changed

Area Change
PyodideExecutor.ts New singleton — loads Pyodide, exposes /data/ virtual FS, runs user code, stores results in-memory under pyodide:// URIs
PythonInterpreter.ts Routes execution to Pyodide or backend based on flag and box type
api.ts fetchData() intercepts pyodide:// paths and serves from in-memory store instead of making a network request
IndexedDBFiles.ts New — persists uploaded files across page refreshes
IndexedDBProvenance.ts New — persists provenance (execution history) per box across page refreshes
index.tsx Preloads Pyodide at boot to hide load latency
ProvenanceProvider.tsx Provenance writes go to IndexedDB instead of backend
DatasetsWindow.tsx Lists files from Pyodide virtual FS; adds Clear All Files button

What is moved vs what remains only on backend

Feature Status
Standard Python (pandas, numpy, Vega-Lite) ✅ Client-side via Pyodide
File uploads + /data/ access ✅ Client-side via IndexedDB
Provenance tracking ✅ Client-side via IndexedDB
VIS_UTK (3D urban rendering) ⚠️ Backend only — requires geopandas/rasterio/GDAL
LLM / AI assistant ⚠️ Backend only — calls /openAI endpoint

How to test

  # In utk_curio/frontend/urban-workflows/.env
  PYODIDE_ENABLED=true

  cd utk_curio/frontend/urban-workflows
  npm install && npm start

Open the app, upload a CSV, run a pandas code box, and verify
output without any backend running.

Issue resolved by this PR (if any)

  • Issue Number:
  • Link:

Type of change (Check all that apply)

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update
  • Other:

Parts of Curio impacted by this PR:

  • Frontend
  • Backend
  • Sandbox

Testing

  • Executed ./scripts/test.sh and it passed all tests.

Screenshots (if relevant)

Checklist (Check all that apply)

  • I have manually loaded each .json test from the tests/ folder into Curio, ran all the nodes one by one, and checked that they run without errors and give the expected results
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@shubhamkhatri20 shubhamkhatri20 force-pushed the shubham/pyodide-migration branch 2 times, most recently from f55534d to 6da86b8 Compare April 29, 2026 22:54
@karlafalcao
Copy link
Copy Markdown
Collaborator

@shubhamkhatri20 please solve the conflicts so we can review it.

@karlafalcao
Copy link
Copy Markdown
Collaborator

@shubhamkhatri20 can you solve the conflics so we can review it.

@shubhamkhatri20 shubhamkhatri20 force-pushed the shubham/pyodide-migration branch from 6da86b8 to 0d3bc9a Compare May 8, 2026 23:36
@shubhamkhatri20
Copy link
Copy Markdown
Author

shubhamkhatri20 commented May 9, 2026

@karlafalcao Fixed the conflicts.

  • I have set the flag pyodide_enabled = false which makes it run as default behaviour. Setting it as true fails some E2E test that require user to be logged in. All other tests passed except E2E.
  • So it runs in default mode without any errors or breaking existing flow.
  • Some dev effort is required to integrate that since "this feature is for only local user machine with a standalone frontend running".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants