Barami, T. | bioRxiv Science

bioRxiv2025

Using computer vision to quantify facial expressions of children with autism during naturalistic social interactions

BackgroundDifficulties with non-verbal communication, including atypical use of facial expressions, are a core feature of autism. Quantifying atypical use of facial expressions during naturalistic social interactions in a reliable, objective, and direct manner is difficult, but potentially possible with facial analysis computer vision algorithms that identify facial expressions in video recordings. MethodsWe analyzed >5 million video frames from 100 verbal children, 2-7-years-old (72 with autism and 28 controls), who were recorded during a [~]45-minute ADOS-2 assessment using modules 2 or 3, where they interacted with a clinician. Three different facial analysis algorithms (iMotions, FaceReader, and Py-Feat) were used to identify the presence of six facial expressions (anger, fear, sadness, surprise, disgust, and happiness) in each video frame. We then compared results across algorithms and across autism and control groups using ANCOVA analyses while controlling for the age and sex of participating children. ResultsThere were significant differences in the performance of the three facial analysis algorithms including differences in the proportion of frames identified as containing a face and frames classified as containing each of the examined facial expressions. Nevertheless, analyses across all three algorithms demonstrated that there were no significant differences in the quantity of facial expressions produced by children with autism and controls. LimitationsOur study was limited to verbal children with autism who completed ADOS-2 assessments using modules 2 and 3 and were able to sit in a stable manner while facing a wall mounted camera. In addition, this study focused on comparing the quantity of facial expressions across groups rather than their quality, timing, or social context. ConclusionsCommonly used automated facial analysis algorithms exhibit large variability in their output when identifying facial expressions of young children during naturalistic social interactions. Nonetheless, all three algorithms did not identify differences in the quantity of facial expressions across groups, suggesting that atypical production of facial expressions in verbal children with autism is likely related to their quality, timing, and social context rather than their quantity during natural social interaction.

bioengineering↗

bioRxiv2024

Automated identification and quantification of stereotypical movements from video recordings of children with ASD

ImportanceStereotypical motor movements (SMMs) are a form of restricted and repetitive behavior (RRB), which is a core symptom of Autism Spectrum Disorder (ASD). Current quantification of SMM severity is extremely limited, with studies relying on coarse and subjective caregiver reports or laborious manual annotation of short video recordings. ObjectiveTo demonstrate the utility of a new open-source AI algorithm that can analyze extensive video recordings of children and automatically identify segments with heterogeneous SMMs, thereby enabling their direct and objective quantification. Design, setting, and participantsThis retrospective cohort study analyzed video recordings from 319 behavioral assessments of 241 children with ASD, 1.4 to 8 years old, who participated in research at the Azrieli National Centre for Autism and Neurodevelopment Research in Israel. Behavioral assessments included cognitive, language, and autism diagnostic observation schedule, 2nd edition (ADOS-2) assessments. ExposuresEach assessment was recorded with 2-4 cameras, yielding 580 hours of video footage. We manually annotated 7,352 video segments containing heterogeneous SMMs performed by different children (21.14 hours of video). Main outcomes and measuresWe used a pose-estimation algorithm (OpenPose) to extract skeletal representations of all individuals in each video frame and trained an object-detection algorithm (YOLOv5) to identify and track the child in each movie. We then used the skeletal representation of the child to train an SMM recognition algorithm using a PoseConv3D model. We used data from 220 children for training and data from the remaining 21 children for testing. ResultsThe algorithm accurately detected 92.53% of manually annotated SMMs in our test data with 66.82% precision. Overall number and duration of algorithm identified SMMs per child were highly correlated with manually annotated number and duration of SMMs (r=0.8 and r=0.88, p<0.001 respectively). Conclusion and relevanceThese findings demonstrate the ability of the algorithm to capture a highly diverse range of SMMs and quantify them with high accuracy, enabling objective and direct estimation of SMM severity in individual children with ASD. We openly share the "ASDPose" dataset and "ASDMotion" algorithm for further use by the research community. Key PointsO_ST_ABSQuestionC_ST_ABSIs it possible to train a deep learning algorithm to accurately identify and quantify stereotypical motor movements (SMMs) in video recordings of children with autism? FindingsThe ASDMotion algorithm was trained and tested with the largest video dataset of ASD children curated to date, comprised of 319 behavioral assessment recordings from 241 ASD children. The algorithm successfully detected 92.53% of manually identified SMMs with 66.82% precision, achieving highly accurate quantification of SMMs per child that were strongly correlated (r[≥]0.8) with quantification by manual annotation. MeaningThis study demonstrates the utility of ASDMotion for objective and direct quantification of SMM severity in children with ASD, offering a new freely available, open-source algorithm and dataset that enable transformative basic and clinical ASD research.

neuroscience↗