A research-focused prosthetic vision simulator that transforms images and video into phosphene percepts for retinal and cortical prosthesis experiments.
This project is designed as an interactive workbench for exploring how different computer vision pipelines, stimulation strategies, and phosphene renderers affect the final percept. It combines modern visual preprocessing methods such as detection, segmentation, saliency, motion, and edge extraction with biologically inspired phosphene simulation.
Visual prostheses do not restore normal vision. Instead, they produce sparse percepts made of phosphenes, or small spots of light. Designing effective encoding pipelines is therefore a major challenge in prosthetic vision research.
This simulator helps researchers and developers:
- explore how input images and videos are transformed into phosphene percepts
- compare different encoding backends and rendering strategies
- inspect intermediate diagnostic maps to understand why a percept looks the way it does
- prototype task-conditioned stimulation pipelines for assistive vision applications
- Interactive Streamlit app for image and video-based phosphene simulation
- YOLO object detection with bounding boxes, labels, and confidence scores
- DeepLabV3 semantic segmentation for scene understanding
- Saliency estimation using spectral residual and fine-grained attention maps
- Edge-based encoding using multi-scale Canny, Sobel, Scharr, and Laplacian features
- Motion-aware fusion using optical flow
- Biologically inspired phosphene rendering
- retinal-style phosphene generation
- cortical phosphene simulation with Dynaphos / pulse2percept
- End-to-end differentiable training pipeline for encoder-simulator-decoder experiments
Run the interactive app locally:
streamlit run app.py