Hemmi, J. M. | bioRxiv Science

bioRxiv2024

Combining Unity with machine vision to create low latency, flexible, and simple virtual realities

O_LIIn recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviors. By using the actions of a constrained animal to control the visual scenery, the animal is provided the perception of moving through a simulated environment. As the animal is constrained in space, this allows detailed behavioral quantification. Additionally, as the world is generally computer-generated this allows for mechanistic quantification of visual triggers of behavior. C_LIO_LIWe created a novel virtual arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi-modal virtual reality arena, MultiMoVR (Kaushik et al., 2020) but tracked hoverfly wing movements using DeepLabCut-live (DLC-live, Kane et al., 2020). For trackball experiments, we recorded the motion of a ball that a tethered crab was walking on using FicTrac (Moore et al., 2014). In both cases, real-time tracking was interfaced with Unity to control the movement of the tethered animals avatars in the virtual world. We developed a user-friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. C_LIO_LIWe show that both the DLC-live-Unity and the FicTrac-Unity configurations close the feedback loop effectively with small delays, less than 50 ms. Our FicTrac-Unity integration highlighted the importance of closed-loop feedback by reducing behavioral artifacts exhibited by the crabs in open-loop scenarios. We show that Eristalis tenax hoverflies, using the DLC-live-Unity integration, navigate towards flowers. The effectiveness of our CAVE interface is shown by implementing experimental sequencing control based on avatar proximity to virtual structures. C_LIO_LIOur results show that combining Unity with machine vision tools such as DLC-live and FicTrac provides an easy and flexible virtual reality (VR) environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design and implement new experiments without programming in code. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community. C_LI

animal behavior and cognition↗

bioRxiv2022

Enhanced short-wavelength sensitivity in the blue-tongued skink,Tiliqua rugosa

The complex visually mediated behaviors of diurnal lizards are enabled by a retina typically containing five types of opsins with the potential for tetrachromatic color vision. Despite lizards using a wide range of color signals, the limited variation in photoreceptor spectral sensitivities across lizards suggests only weak selection for species-specific, spectral tuning of photoreceptors. Some species, however, have enhanced short wavelength sensitivity, which likely helps with the detection of signals rich in ultraviolet and short wavelengths. In this study, we examined the visual system of Tiliqua rugosa, which has a UV/blue tongue, to determine the spectral sensitivity of the eye and to gain insight into this species visual ecology. Electroretinograms coupled with spectral stimulation showed peak sensitivity at 560 nm with high similarity to other lizards at wavelengths greater than 530 nm. However, at shorter wavelengths, sensitivity is enhanced leading to a spectral sensitivity curve that is 28 nm broader (full width at half height) than other lizards studied so far. The width of the curve is partially explained by a population of photoreceptors that respond more strongly to low temporal frequencies with possible peaks in sensitivity between 460 and 470 nm suggesting that they are SWS2 photoreceptors. The lack of a peak in sensitivity at 360 nm at low temporal frequencies and under a monochromatic light that suppresses the response of LWS photoreceptors, suggests that the SWS1 photoreceptors are red-shifted. In addition, the yellow and green oil droplets that are common in other diurnal lizards appear to be missing and instead, only transparent and pale-yellow oil droplets are present. LWS photoreceptors are likely paired with pale-yellow oil droplets to produce LWS photoreceptors with wider spectral sensitivity curves than in other lizards. Opsin sequencing reveals SWS1, SWS2, RH1, RH2 and LWS opsin genes that are very similar to the visual opsins detected in the green anole, Anolis carolinensis, suggesting there is little change in the spectral sensitivity of photoreceptors compared to other diurnal lizards. Since we only obtained a partial sequence of the SWS1 opsin, we were unable to determine whether amino acid substitution at tuning sites could have played a role in red-shifting the SWS1 photoreceptor spectral sensitivity. Photoreceptor densities are typically higher in central and ventral retinal regions than in the dorsal retina suggesting that higher spatial sampling is necessary at eye level and above the animal than on the ground. However, the SWS1 photoreceptors do not follow this pattern potentially due to their low abundance making them less relevant to high acuity visual tasks. Our findings demonstrate that there are possibly multiple mechanisms acting synergistically in the visual system of T. rugosa to enhance short wavelength sensitivity between 360 and 530 nm. While it is tempting to suggest that this is an adaptation to facilitate the detection of the blue tongues of conspecifics, additional experiments are necessary to determine its ecological relevance. Summary statementColor vision and the distribution of photoreceptor subtypes in T. rugosa

neuroscience↗