Francisco Costela, PhD

Data analysis - Training in Low vision

Created a new method to identify the preferred retinal locus (PRL; the peripheral area of the retina used when the fovea is disabled/damaged) used to watch videos by participants with normal vision, blur defocus lenses, and central visual loss (macular degeneration; see image below), as compared to the democratic Center of Interest (gazes from people with normal vision)

Also created a perceptual training using a gaze-contingent display with a simulated scotoma to train people with normal vision to develop a PRL and examined their evolution through increasing scotoma sizes.

Github repository

Publication

Machine Learning - Gaze prediction

A saccade is a fast eye movement that allows the change of visual fixation from one object of interest to another. These movements are characterized by very high angular velocity peaks that can reach up to 1,000º/s, making them as one of the fastest neuromotor activities in the human body. Modeling such a complex movement remains a challenge. Saccadic eye movements can be defined by initial and landing points, duration, amplitude, and velocity profile. The landing point is important as it defines the new fixation region and, therefore, the region of interest of the viewer. Its prediction may reduce problems caused by display-update latency in gaze-contingent systems that make real-time changes in the display based on eye tracking.

Neural Network approach ---- The main contribution of this work is to propose the use of state-of-the-art machine learning techniques (i.e., Recurrent Neural Networks) for saccade landing point prediction in real-world scenarios. The proposed method was evaluated using two large datasets including 219,335 human saccades (collected with an EyeLink 1000 system, median 3°, 95% range 1° to 32°) from 75 subjects acquired during viewing video from Hollywood movies. The results obtained using our proposed methods outperform existing approaches with improvements of up to 40% error reduction. The results show that dynamic temporal relationships exploited by Recurrent Neural Networks can improve the performance of traditional Feed Forward Neural Networks

Taylor series approach ---- Gaze-contingent displays have been widely used in vision research and virtual reality applications. Due to data transmission, image processing, and display preparation, the time delay between the eye tracker and the monitor update may lead to a misalignment between the eye position and the image manipulation during eye movements. I propose a method to reduce the misalignment using a Taylor series to predict the saccadic eye movement. Besides the human saccade dataset, I used data from my electrophysiology experiments (21,844 monkey saccades, collected with a scleral search coil, median 2°, 95% range 1° to 9°). When assuming a 10-ms time delay, the prediction of saccade trajectories using the proposed method could reduce the misalignment greater than two state-of-the-art methods. The average error was about 0.93° for human saccades and 0.26° for monkey saccades. Our results suggest that this proposed saccade prediction method will create more accurate gaze-contingent displays.

Github repository

Publications

"Saccade Landing Point Prediction: A Novel Approach based on Recurrent Neural Networks"
"Dynamic gaze-position prediction of saccadic eye movements using a Taylor series"
"Saccade Landing Point Prediction Based on Fine-Grained Learning Method"

Computer Vision - Visual aids

In these computer vision projects, I computed the Center of Interest by applying a Gaussian Kernel function for each frame based on the gazes of viewers watching videos. I compared the comprehension on video based on different approaches - magnification approach based on democratic COI, static center of image, or random COI.

I also created different visual rehabilitation aids for people with central vision loss (macular degeneration) by applying bubble magnification and zoom magnification, and for people with hemianopia using a dynamic cue around the COI (see video below)

Video example

Github repository

Publications

"An implementation of Bubble Magnification did not improve the video comprehension of individuals with central vision loss""

Machine Learning - Risk detection in driving

I set out to evaluate whether eye movement features extracted from eye tracking data are able to predict risk situations during a risk detection task while watching driving videos. I created predictive models in Python using logistic regressions and feed-forward neural networks.

I collected data from thirty-two normally sighted subjects (15 female) who saw 20 clips of recorded driving scenes while their gaze was tracked. They reported when they considered the car should brake anticipating any hazard.

All subjects reported at least one major collision hazard in each video (average 3.5 reports). I found that hazard situations were predicted by larger saccades, more and longer fixations, fewer blinks, and a smaller gaze dispersion in both X and Y dimensions. I compared performance between models incorporating a different combination of descriptors with the test equality of receiver operating characteristic areas. Accuracy using feedforward neural networks outperformed logistic regressions. The model including saccadic magnitude, fixation duration, dispersion in x, and pupil returned the highest ROC area under the curve (0.73).

Github repository

Publication

Computer Vision - Blink detection

A Matlab program with interface to detect blinks from videos and save the frames automatically. It uses automatic eye detection but can also be disabled to manually indicate the eye region by clicking in the four corners of a rectangle area. The blink detector algorithm is based on a white sclera thresholding method that can be also customised in the interface.

Github repository

Open-access gaze database

I created an open-source database of tracked eye movements, collected using an infra-red, video-camera Eyelink 1000 system, from 95 participants as they viewed ‘Hollywood’ video clips.
There are 206 clips of 30-s and eleven clips of 30-min for a total viewing time of about 60 hours. The database also provides the raw 30-s video clip files, a short preview of the 30-min clips, and subjective ratings of the content of the videos for each in categories: (1) genre; (2) importance of human faces; (3) importance of human figures; (4) importance of man-made objects; (5) importance of nature; (6) auditory information; (7) lighting; and (8) environment type. Precise timing of the scene cuts within the clips and the democratic gaze scanpath position (center of interest) per frame are provided.
At this time, this eye-movement dataset has the widest age range (22–85 years) and is the third largest (in recorded video viewing time) of those that have been made available to the research community. The dataset is freely available in the Open Science Framework repository (link below) and can be used without restriction for educational and research purposes, providing that the paper (link below) is cited in any published work.

Video example

Database repository

Github repository

Publication

Data Analysis - Psychophysics

I analyzed eye movement data and wrote code for multiple psychophysical experiments in Matlab using Psychtoolbox during my career in vision science to measure the role of fixational eye movements in oculomotor control, perception, and cognition. Most of the projects also involved collecting eye movements from eye tracker data (EyeLink 1000).

Github repository

Publications

"Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes"
"Microsaccades restore the visibility of minute foveal targets"
"Fixational eye movement correction of blink-induced gaze position errors"
"Changes in visibility as a function of spatial frequency and microsaccade occurrence" errors "Characteristics of Spontaneous Square-Wave Jerks in the Healthy Macaque Monkey during Visual Fixation"

Data Analysis - Rasch Analysis

There have been few systematic reports of vision-related activity limitations of people with retinitis pigmentosa (RP). I report a merging of data from the National Eye Institute Visual Function Questionnaire (NEI-VFQ) obtained in five previous studies. I asked whether the Vision Function Scale which was developed for cataract patients would apply in this new population (condition).

I used data from 594 individuals who completed a total of 1,753 questionnaires, with 209 participants providing responses over at least four years. Rasch analysis showed that the 15-item VFS was poorly targeted. I created a new instrument by adding four driving-related items to the VFS had better targeting. As an indirect validation, VFS-plus person scores were compared to visual field area measured using a Goldmann perimeter, to the summed score for the combined 30-2 and 30/60-1 Humphrey Field Analyzer programs (HFA), to 30-Hz full-field cone electroretinogram (ERG) amplitude, and to ETDRS visual acuity. Changes in VFS-plus person scores with age and between four common heredity groups were also examined.

My new model (see image below) had person and item separation of 2.66 and 24.43 respectively. The VFS-plus person scores were related to each vision measure (p<0.001). Over a five-year period, there was a reduction in person scores of 0.5 logits (p<0.001). Person scores fell by an average of 0.34 logits per decade (p<0.0001). Participants with an X-linked hereditary pattern had, on average, lower person scores (p<0.001).

In summary, the VFS-plus instrument quantified a highly-significant annual reduction in perceived vision-related ability over a five-year period, was consistent with clinical measures of vision, and detected lower perceived vision-related ability in participants with X-linked disease.

Publications

"Validation of a vision-related activity scale for patients with retinitis pigmentosa""

Machine Learning - NLP in Alzheimer's

I created a new objective measure, the narrative description (ND), of the ability to perceive, understand, and describe a visual scene in a movie clip and used it to assess cognitive status. The process of interpreting and acting upon the visual environment requires both intact cognitive and visual systems. The ND metric uses NLP by comparing the mean number of shared words with a normative dataset after removing stopwords.

I used data from 56 participants with cognitive status ranging from normal cognition to mild dementia (median age 82, range 66 to 99 years). They watched 20 30-s video clips and describe the visual content without time constraints. These verbal responses were transcribed and processed to generate ND shared word scores using a “wisdom of the crowd,” natural-language processing approach. I compared ND scores across diagnostic groups, and used linear mixed models to examine decrements in task performance.

There was a stepwise decline of ND scores with increasing levels of cognitive impairment. Additional analyses showed that ND performance was highly related to performance on the Montreal Cognitive Assessment (MoCA) and domain-specific neuropsychological tests for semantic fluency and set shifting. Other models demonstrated differences in ND performance related video content between cognitively normal and impaired participants.

In summary, the ND test was able to detect decrements in task performance between levels of cognitive impairment and was related to other global neuropsychological measures.

Github repository

Publications

"Narrative video scene description task discriminates between levels of cognitive impairment in Alzheimer’s disease"

Machine Learning - NLP in low vision

In these projects, I created a new objective score of information acquisition (IA) derived by comparing each new response to a control database of descriptions of the same clip using natural language processing. I applied this metric to two low vision groups: central vision loss (CVL;macular degeneration) and hemianopia.

In the CVL study group, subjects with CVL (n= 23) or normal vision (NV, n= 60) described the content of 30-second video clips from movies and documentaries. We derived an objective information acquisition (IA) score for each response using natural-language processing. To test whether the impact of CVL was simply due to reduced resolution, another group of NV subjects (n= 15) described video clips with defocus blur that reduced visual acuity to 20/50 to 20/800. Mixed models included random effects correcting for differences between subjects and between the clips, with age, gender, cognitive status, and education as covariates. IA scores were worse for the CVL group (P< 0.001). IA reduced with worsening visual acuity (P< 0.001), and the reduction with worsening visual acuity was greater for the CVL group than the NV-defocus group (P= 0.01), which was seen as a greater discrepancy at worse levels of visual acuity. Conclusions: The IA method was able to detect difficulties in following the story experienced by people with CVL and HH.

In the hemianopia (HH) group study, I compared 60 participants with normal vision (NV) to 24 participants with HH to test the hypothesis that participants with HH would score lower than NV participants, consistent with reports from people with HH that describe difficulties in video watching. The HH group had a significantly lower IA score, with an average of 2.8, compared with 4.3 shared words of the NV group (mixed-effects regression, P < 0.001). Presence of the content guide significantly increased the IA score by 0.5 shared words (P = 0.03).

Github repository

Publications

"People With Central Vision Loss Have Difficulty Watching Videos""

"People with Hemianopia Report Difficulty with TV, Computer, Cinema Use, and Photography"

"Measuring the Difficulty Watching Video With Hemianopia and an Initial Test of a Rehabilitation Approach"

Machine Learning - (Unsupervised) Archetypal analysis

I set up to provide objective, numeric descriptors of visual field (VF) patterns from Humphrey Field Analyzer (HFA) data from subjects with retinitis pigmentosa (RP), by using an unsupervised machine learning approach (archetypal analysis).

I obtained VFs that were 120-degrees wide from two clinical trials on RP conducted at Massachusetts Eye and Ear Infirmary. Reliable VFs for up to eight visits and up to 7 years were available for 214 individuals (426 eyes) in Study 1 and 235 individuals (469 eyes) in Study 2. I used archetypal analysis to decompose each VF into components that reflected subtypes and affected regions. Each measured VF could be reconstructed from the pattern coefficients. I also examined relationships between archetypal-pattern coefficients and some demographic variables and hereditary-pattern information.

I identified eight archetypal patterns in Study 1, and 12 patterns for Study 2 (see figure below) Progressive changes in archetype coefficients, consistent with expectations, were found: archetypes representing a small central VF only (“tunnel vision”) were more related to increasing age, while archetypes with fuller VFs and peripheral islands were less related to increasing age. There were significant differences between hereditary groups in archetype coefficients for many archetypes, indicating that heredity patterns were associated with phenotypes.

In summary, archetypal analysis, an unsupervised machine learning method, produced patterns (archetypes) that were descriptive of the variety of VF patterns found in two large samples of people with RP. This is a novel quantitative method of describing VFs in RP.

This approach could be used in quantitative analyses, such as genotype-phenotype relationships, or the contribution of peripheral islands to vision-related daily activities, or predicting changes in VF over time or monitoring change, which can be used in evaluations of treatments or inform rehabilitation.
 

On-shelf availability

I trained, deployed, and tested an on-shelf availability detection model on IoT/Edge AI image Sony IMX500 sensors using Azure Custom Vision, Python, Azure cloud processing services, SONY AITRIOS portal, and API tools. IMX500 is the world's first image sensor equipped with AI processing.
Along with my team at Sony Electronics, we installed IMX500 smart cameras in the Microsoft Dynamics 365 Retail Test Lab and showed an on-shelf availability proof of concept in this grocery store like environment.
Our POC ran my model loaded on these cameras which recognized missing items on the shelves. The cameras were set to scan store shelves on a regular basis to look for out of stock products.

 

Service line management

I trained, deployed, and tested a person and magnet detection model on IoT/Edge AI image Sony IMX500 (the world's first image sensor equipped with AI processing) sensors using Azure Custom Vision, Python, Azure cloud processing services, SONY AITRIOS portal, and API tools.

I helped creating a Service Line Management Demo Kit Setup Guide where the user deployed and ran my model on AITRIOS to create value in monitoring workers in stations and managing customer queues.

 

ANPR (plate recognition)

I deployed and tested an ANPR (plate recognition) model on the Sony IMX500 (the world's first image sensor equipped with AI processing) sensors using Azure Custom Vision, Python, Azure cloud processing services, SONY AITRIOS portal, and API tools.

I also helped creating post-processing algorithms to filter and improve the detection of the plates on the POC.

 

Smart Cooler

I created, deployed and tested a beverage detection model using the Sony IMX500 (the world's first image sensor equipped with AI processing) sensors installed in two cameras on the fridge, which triggered detection automatically after beverage removal. I used YOLO and TensorFlow, Azure Custom Vision, Python, Azure cloud processing services, SONY AITRIOS portal, and API tools.

I also helped creating post-processing algorithms to filter and improve the detection of the beverages on the POC.

 

Big Data architecture

I built the Database Architecture in a large-scale project for Visual Speech Recognition using PostgreSQL, PGAdmin4, and RClone to manage/store TBs of data.

 

Lip Reading ETL pipeline

I built the Audio and Video Processing ETL pipeline on a large-scale project for visual speech recognition. I managed two rigs and three computer stations (3-8 Nvidia GPUs each) processing data and performing forced alignment, speech recognition, and face detection using Python, OpenCV, PM2, and HTK.

NLP in fashion (IG)

I applied NLP techniques to a set of Instagram posts with the goal of finding fashion trends. Among others, I used Python, Pandas, Numpy, Scipy, Matplotlib, seaborn and Scikit-learn to conduct an exploratory analysis, data cleaning, sentiment analysis, and topic modeling.

Github repository

Computer vision - Eye tracking

I have wide experience with eye tracking (working both with primates and human eye data). One remarkable project in this field was conducted at Schepens Eye Research Institute where I simulated low vision impairments using gaze-contingent display systems. In the repository, I share the scripts I wrote in Matlab to simulate central vision loss (macular degeneration), hemianopia, and tunnel vision. I have used these methodologies in training programs (check section of PRL training) and also developed gaze-prediction methods to optimize the delay between the eye-tracker and the display system (see section Machine Learning - Gaze prediction). Further, I investigated causes that impact the modeling of saccades based on velocity error and curfature profiles (see paragraph and publication below).

Commonly, saccades are thought to be ballistic eye movements, not modified during flight, with a straight path and a well-described velocity profile. However, they do not always follow a straight path and studies of saccade curvature have been reported previously. In a prior study, I developed a real-time, saccade-trajectory prediction algorithm to improve the updating of gaze-contingent displays and found that saccades with a curved path or that deviated from the expected velocity profile were not well fit by our saccade-prediction algorithm (velocity-profile deviation), and thus had larger updating errors than saccades that had a straight path and had a velocity profile that was fit well by the model. Further, I noticed that the curved saccades and saccades with high velocity-profile deviations were more common than we had expected when participants performed a natural-viewing task. Since those saccades caused larger display updating errors, we sought a better understanding of them. Here we examine factors that could affect curvature and velocity profile of saccades using a pool of 218,744 saccades from 71 participants watching “Hollywood” video clips.

While viewing the video clips, saccades were most likely horizontal or vertical over oblique. Measured curvature and velocity-profile deviation had continuous, skewed frequency distributions. I used mixed-effects regression models that included cubic terms and found a complex relationship between curvature, velocity-profile deviation and saccade duration (or magnitude). Curvature and velocity-profile deviation were related to some video-dependent features such as lighting, face presence, or nature and human figure content. Time during the session was a predictor for velocity profile deviations. Further, I found a relationship for saccades that were in flight at the time of a scene cut to have higher velocity-profile deviations and lower curvature in univariable models. Saccades characteristics vary with a variety of factors, which suggests complex interactions between oculomotor control and scene content that could be explored further.  

Github repository

Publications

"When Watching Video, Many Saccades Are Curved and Deviate From a Velocity Profile Model""

Close Window