Biology subjects

de Ruyter van Steveninck, J.

Publications and source records attributed to de Ruyter van Steveninck, J..

5 recordsLinked to original sources

bioRxiv2025

End-to-end Learning of Safe Stimulation Parameters for Cortical Neuroprosthetic Vision

Direct electrical stimulation of the brain via cortical visual neuroprostheses is a promising approach to restore basic sight for the visually impaired by inducing a percept of localized light called phosphenes. Apart from the challenge of condensing complex sensory information into meaningful stimulation patterns at low temporal and spatial resolution, providing safe stimulation levels to the brain is crucial. We propose an end-to-end framework to learn optimal stimulation parameters (amplitude, pulse width and frequency) within safe biological constraints. The learned stimulation parameters are passed to a biologically plausible phosphene simulator which takes into account the size, brightness, and temporal dynamics of perceived phosphenes. Our experiments on naturalistic navigation videos demonstrate that constraining stimulation parameters to safe levels not only maintains task performance in image reconstruction from phosphenes but consistently results in more meaningful phosphene vision, while providing insights into the optimal range of stimulation parameters. Our study presents a stimulus-generating encoder that learns stimulation parameters (1) satisfying safety constraints, and (2) maximizing the combined objective of image reconstruction and phosphene interpretability with a highly realistic phosphene simulator accounting for temporal dynamics of stimulation. End-to-end learning of stimulation parameters this way enables enforcement of critical biological safety constraints as well as technical limits of the hardware at hand.

bioengineering↗

bioRxiv2023

Gaze-contingent processing improves mobility performance and visual orientation in simulated head-steered prosthetic vision

The enabling technology of visual prosthetics for the blind is making rapid progress. However, there are still uncertainties regarding the functional outcomes, which can depend on many design choices in the development. In visual prostheses with a head-mounted camera, a particularly challenging question is how to deal with the gaze-locked visual percept associated with spatial updating conflicts in the brain. A recently proposed compensation strategy is gaze-contingent image processing with eyetracking, which enables natural visual scanning and reestablished spatial updating based on eye movements. The current study evaluates the benefits of gaze-contingent processing versus gaze-locked and gaze-ignored simulations in the context of mobility and orientation, using a simulated prosthetic vision paradigm with sighted subjects. Compared to gaze-locked vision, gaze-contingent processing was found to improve the speed in all experimental tasks, as well as the subjective quality of vision. Similar or further improvements were found in a control condition that ignores gaze-depended effects, a simulation that is unattainable in the clinical reality. Our results suggest that gaze-locked vision and spatial updating conflicts can be debilitating for complex visually-guided activities of daily living such as mobility and orientation. Therefore, for prospective users of head-steered prostheses with an unimpaired oculomotor system, the inclusion of a compensatory eye-tracking system is strongly endorsed.

bioengineering↗

bioRxiv2022

Biologically plausible phosphene simulation forthe differentiable optimization of visual corticalprostheses

Blindness affects millions of people around the world, and is expected to become increasingly prevalent in the years to come. For some blind individuals, a promising solution to restore a form of vision are cortical visual prostheses, which convert camera input to electrical stimulation of the cortex to bypass part of the impaired visual system. Due to the constrained number of electrodes that can be implanted, the artificially induced visual percept (a pattern of localized light flashes, or phosphenes) is of limited resolution, and a great portion of the fields research attention is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is the non-invasive functional evaluation in sighted subjects or with computational models by making use of simulated prosthetic vision (SPV) pipelines. Although the SPV literature has provided us with some fundamental insights, an important drawback that researchers and clinicians may encounter is the lack of realism in the simulation of cortical prosthetic vision, which limits the validity for real-life applications. Moreover, none of the existing simulators address the specific practical requirements for the electrical stimulation parameters. In this study, we developed a PyTorch-based, fast and fully differentiable phosphene simulator. Our simulator transforms specific electrode stimulation patterns into biologically plausible representations of the artificial visual percepts that the prosthesis wearer is expected to see. The simulator integrates a wide range of both classical and recent clinical results with neurophysiological evidence in humans and non-human primates. The implemented pipeline includes a model of the retinotopic organisation and cortical magnification of the visual cortex. Moreover, the quantitative effect of stimulation strength, duration, and frequency on phosphene size and brightness as well as the temporal characteristics of phosphenes are incorporated in the simulator. Our results demonstrate the suitability of the simulator for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioural experiments. The modular approach of our work makes it ideal for further integrating new insights in artificial vision as well as for hypothesis testing. In summary, we present an open-source, fully differentiable, biologically plausible phosphene simulator as a tool for computational, clinical and behavioural neuroscientists working on visual neuroprosthetics.

neuroscience↗

bioRxiv2022

Optimization of Neuroprosthetic Vision via End-to-end Deep Reinforcement Learning

Visual neuroprostheses are a promising approach to restore basic sight in visually impaired people. A major challenge is to condense the sensory information contained in a complex environment into meaningful stimulation patterns at low spatial and temporal resolution. Previous approaches considered task-agnostic feature extractors such as edge detectors or semantic segmentation, which are likely suboptimal for specific tasks in complex dynamic environments. As an alternative approach, we propose to optimize stimulation patterns by end-to-end training of a feature extractor using deep reinforcement learning agents in virtual environments. We present a task-oriented evaluation framework to compare different stimulus generation mechanisms, such as static edge-based and adaptive end-to-end approaches like the one introduced here. Our experiments in Atari games show that stimulation patterns obtained via task-dependent end-to-end optimized reinforcement learning result in equivalent or improved performance compared to fixed feature extractors on high difficulty levels. These findings signify the relevance of adaptive reinforcement learning for neuroprosthetic vision in complex environments.

bioengineering↗

bioRxiv2020

End-to-end optimization of prosthetic vision

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a non-trivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments we show that such an approach is able to automatically find a task-specific stimulation protocol. The presented approach is highly modular and could be extended to dynamically optimize prosthetic vision for everyday tasks and requirements of the end-user.

bioengineering↗