Natalie G Wall, Oliver Smith, Linda E Campbell, Carmel Loughland, Mark Wallis, Frans Henskens, Ulrich Schall

Natalie G Wall, Oliver Smith, Linda E Campbell, Carmel Loughland, Mark Wallis, Frans Henskens,Ulrich Schall, Centre for Brain and Mental Health Research, The University of Newcastle,Callaghan 2308, NSW, Australia

Natalie G Wall, Oliver Smith, Linda E Campbell, Carmel Loughland, Mark Wallis, Frans Henskens,Ulrich Schall, The Brain and Mental Health Research Program, Hunter Medical Research Institute, New Lambton Heights 2305, NSW, Australia

Linda E Campbell, School of Psychology, The University of Newcastle, Callaghan 2308, NSW,Australia

Abstract Autism is a neurodevelopmental condition with associated difficulties that present differently across individuals. One such difficulty is recognizing basic and complex facial expressions. Research has previously found that there are many evidence-based support programs available for building non-verbal communication skills. These programs are frequently administered with a therapist or in a group setting, making them inflexible in nature. Programs hosted on e-technology are becoming increasingly popular, with many parents supportive of them. Applications (apps) that are hosted on technology such as iPads or mobile phones allow users to engage in building skills in real-time social settings and own what they are learning. These technologies are frequently used by autistic children, with apps typically focusing on identifying facial features. Yet at this current time, there are mixed reviews of how to design such programs and what their theoretical backing is, with many studies using a mix of observation and psychological assessments as outcome measures. Eye-tracking and electroencephalography are established methodologies that measure neural processing and gaze behaviors while viewing faces. To better support the field moving forward, objective measures such as these are a way to measure outcomes of apps that are designed for helping children on the spectrum build skills in understanding facial expressions.

Key Words: Autism; Facial expression recognition; Technology; Eye-tracking; Electroencephalography

INTRODUCTION

Defining characteristics of autism

The core features of autism spectrum disorder are difficulties with social communication and restricted and repetitive behaviours[1]. However, it is commonly associated with atypical cognitive profiles, executive dysfunction, atypical perceptual and information processing, all of which vary across individuals[2]. Although the diagnostic term for autism is “Autism Spectrum Disorder”, many autistic people prefer identity first language rather than person first language (e.g., autistic person rather than person with autism)[3-6]. Hence, the review will use identity first language hereafter.

In Australia, approximately one in 150 people are diagnosed with autism[7]. Changes in reporting practices over time and increased awareness contribute to the increased rates of diagnosis in many cases[8-10]. Strengths and difficulties vary between individuals with autism being clinically categorised by the type of supports needed[1]. In 2018, school-aged (aged 5-20) autistic people reported the greatest difficulty they faced was fitting in socially[7]. Non-verbal communication skills account for more than 60% of overall communication ability, and has been identified as an especially important issue due to the key role these skills play in the way people socialise[11]. Studies have found that difficulties with non-verbal social skills in autistic people vary widely and include understanding facial expressions, using conversational gestures and joint attention[12-15]. Understanding these subtleties in communication plays a critical role in building trusting friendships and relationships with others, especially during school years.

UNDERSTANDING FACIAL EXPRESSIONS

Understanding and processing facial expressions is considered some of the most important non-verbal communication skills as they are often the first feature we notice when meeting other people. Faces convey important social information about our mood and intentions to those around us. An inability to understand expressions has been linked to greater difficulties in social situations[16,17]. Autistic people are observed to have significant difficulties in recognising facial expressions, and these difficulties have been found to increase over time[17-19]. Although difficulties are experienced with all six basic emotions (happy, sad, fear, disgust, angry and surprise), autistic people are observed to have specific difficulties with more complex emotions such as anger, fear and surprise.

ALEXITHYMIA IN THE AUTISTIC POPULATION

Alexithymia is a trait that occurs in approximately 5% of the neurotypical population and in an estimated 50% of the autistic population[20]. It is defined by difficulties identifying and describing one’s own emotions and a lack of awareness related to physical sensations from emotions[17]. This trait is found to play an important role in interpreting facial expressions. It has been theorized that the facial expression recognition difficulties autistic people experience may actually be due to alexithymia, and not social communication difficulties[17]. However, regardless of the potential cause, the ability to interpret and respond appropriately to the facial expressions of others is important to support social connectedness for autistic people.

CURRENT SOCIAL SUPPORT PROGRAMS

Numerous support programs have been developed for children on the autism spectrum that target skill building in social-cognitive, sensory-integration and relationship-based domains[20]. In fact, many evidence-based practices for working with people on the spectrum are focused on helping social communication, play and joint attention[21]. In clinical practice, there are typically two types of programs, the first being a comprehensive treatment model such as Early Intensive Behaviour Intervention which is often organised around a conceptual framework and overarching model[22]. Such programs often require the child to participate in settings outside of their norm (e.g., a clinic) and requires a minimum (20-h) time commitment by family as well as the participant. In addition, the maintenance and generalisation effects of such programs are observed to be significantly lower compared to when participation occurs in a child’s typical setting[13]. The second type of support programs are those aimed at gaining a new skill or overcoming a specific behaviour; these are typically shorter programs (e.g., 10–20 fortnightly sessions) until the goal is achieved. These can be manual or technology-based and include video modelling. Many video modelling programs were developed to focus on specific socialcommunication behaviours such as social initiation, compliment giving and conversations[23], and are seen as highly effective in helping build these skills[24].

Technology-based support programs have been organised into eight categories: computers and internet, videos, mobile technologies (mobile phones and tablet computers), shared active surfaces, virtual and augmented reality, wearables, robotics and natural use interfaces[25]. Programs deliveredviathese technologies is growing across skill areas, as they allow for on-the-go use and can complement current therapies children and adolescents may be engaged in[26-28]. In fact, van der Meeret al[28] suggested how portable technology such as iPads and iPhones are viable technology aids for people in areas such as communication and transitioning skills. A recent example of this technology introduced the concept of animal filters on a mobile device to understand how the idea of mind-blindness or emotion recognition can vary in contexts[29]. Other software applications or “apps” allow the user to take a photo or video while attaching specific emotions to them, which aides in recognising the emotion in the future[29].

Although there is generally no theoretical backing for the use and design of such technologies, acceptability of e-technology is high across both children and parents, with many children using iPhones and iPads almost daily[30]. Parents reportedly like the idea of tablet-based therapy, especially when it comes with in-built instructions. This is not surprising when we consider that many parents work long hours, are time poor, and struggle with the costs of taking children to regular therapy sessions[31]. Although an appetite clearly exists among parents for support programs delivered on mobile devices, a meta-analysis by Honget al[32], found that there were only 14 studies with a combined 36 participants that examined tablet-mediated support programs focusing on social and communication related skills delivery. Further research has also identified that many allied health professionals do not regularly use tablet computers within their sessions[33]. This means a significant gap exists between what is currently available and what health care consumers would like to be able to access. There is a clear need for further e-technology development and research, with larger sample sizes, and targeted delivery of therapies using a variety of devices, such as tablets and virtual reality, to provide learning and simulation in naturalistic settings to lessen this translational gap.

There have been some developments in designing programs for better interpretation of facial expressions. One early development by Baron-Cohenet al[34] used video modelling. The Transporters DVD targeted expression recognition by adding human faces to animated trains and to engage children in social interactions. Over time, video modelling has proven to be a viable method for helping people on the spectrum learn social skills[34]. Developments in other e-technology are still progressing. Clarket al[35] point out that 15 e-technology therapy-based studies were conducted between 2000 and 2016. A review identified that many tablet applications (apps) are designed as support tools with a strong focus on social skills[36]. Apps can provide flexibility of delivery of therapies using a game-based environment to be played in real-time social settings, decreasing anxiety and supporting skill development[36,37]. However, the generalisation effects for these programs were mixed, due mostly to a lack of follow up and observation outside the research setting[38,39]. A way forward would be evaluating the therapeutic benefits of e-technology support programs by objectively measuring changes in behaviour and underlying brain function that is associated with facial information processing.

TECHNIQUES FOR DESIGNING SOCIAL SUPPORT PROGRAMS

Eye-tracking

An important part of social communication is the ability to interpret social signals such as those displayed through facial expressions. Displays of emotion are generally processed by looking at the eye and mouth regions of the face, then linking the information cognitively to a social context or verbal cue. Eye-tracking technology provides a real-time objective measure of face perception and feature processing. Fixation frequency and saccadic velocity (speed of synchronised eye movements) can be mapped to provide a scan path recording. Scan path recordings in the general population show an ‘upside down’ triangular pattern of performance focused on the eye regions and mouth (Figure 1). Comparatively, scan path patterns for autistic people tend to show more inconsistent viewing of the facial features, with fixations mostly falling on non-salient facial regions such as the chin, ear and hairline[40]. Findings from eye-tracking technology provided important insight to why autistic people may experience difficulties interpreting facial emotions and social situations.

Findings from a number of studies report eye avoidance in autistic people and conclude a general consensus that autistic people scan faces differently when compared to non-autistic people[40,41]. Avoiding the eye area, as observed in autistic people, suggests that facial information is encoded and processed differently to their non-autistic peers[42-44]. In a meta-analysis on eye-tracking and social attention, autistic people spent less time giving attention to social stimuli compared to nonautistic people overall[45]. The research reported that this occurred more frequently when there were more people shown in the content, such as a social scene rather than a singular person. Further evidence suggests that when the facial information is coded in such a way, the greater likelihood of difficulties with social skills[46].

Specific emotions also appear to impact face processing performance. Eye avoidance (with predominant focus on the mouth region) is more frequent in autistic adolescents when specifically viewing negative emotions[47]. In a study examining angry faces, Blacket al[44] showed that autistic children were more likely to avoid viewing angry faces and scored higher on the social communication difficulties subset measure of the Gilliam Autism Rating Scale-3 (GARS-3)[48]. Greater attention to the mouth region when viewing face images is also associated with higher rates of social anxiety[46,49-51]. Innovations in e-technology-based support programs provide a mechanism to deliver targeted supports which assist with facial expression recognition and processing difficulties, particularly those to improve eye-tracking of the most salient facial feature areas such as the eyes and mouth regions. Eye-tracking technology also allows a more objective measure of behavioural changes in face processing.

Event related potentials

Another method to explore social communication is to use electroencephalography (EEG). EEG is an electrophysiological monitoring method to measure the electrical activation in the brain in real time. This measurement is highly sensitive and can measure brain responses to the millisecond (ms) and is therefore complimentary for eye tracking researchers. It is common for these brain responses to be labelled by their polarity; either positive (P) or negative (N) due to their electrical deflection, combined with the time after stimulus onset. Additionally, these brain responses are known as event related potentials (ERPs) and are defined “as the direct result of a specific sensory, cognitive or motor event”[52]. For example, a common ERP used in emotional recognition or social communication research is named the N170. The N170 ERP is observed as a significant amplitude deflection in a negative direction 170 ms after stimulus onset. Figure 1 shows the N170 in response to a face in a non-autistic child.

Figure 1 Event-related potentials in response to face (red) and (blue) object images of a non-autistic child. The grey area marks the facespecific event-related potential referred to as N170. Insert shows a normal scan path covering the eye and mouth region of a non-autistic child when viewing a face image.

The N170 is a robust and reliable measure of facial recognition and has been confirmed by spatial functional magnetic resonance imaging, which demonstrates the response is generated from an area of the brain specific for facial recognition known as the fusiform gyrus. This brain region also encodes other complex visual stimuli but is most responsive when processing facial features[41]. Research has found that the N170 is not different in response to angry, fearful or neutral faces between non-autistic or autistic participants[16,41,43]. This means that the identification of a face as such is very fast, but the identification of a specific emotion expression takes longer as it requires a more detailed evaluation of the facial features and is captured by subsequent ERPs (i.e., N200/P300)[53].

Autistic people usually have a slower processing time than non-autistic people when looking at facial stimuli; with slower processing times being associated with less face processing expertise[41,42,54]. Further research confirmed that there is a link between scanning the eye-region and a faster N170 response[41]. Moreover, a smaller N170 is also associated with less developed socials skills and more atypical social behaviours as is more common in autistic people[55].

CONCLUSION

Recording eye-tracking and ERPs are well suited and established methodologies that allow for a more objective assessment of support program benefits at the behavioural (eye-tracking) and neural (ERPs) level. They are less intrusive and demanding than other methods, such as functional brain imaging. As the general idea, e-technology should support these basic processes which are fundamentally involved in facial stimuli processing. TheFaceTiletask[56] is a good example of how this can be achieved, for instance, as a game-like app for autistic children. The game would ask children to recognise emotions in photographs of same-aged children. The photographs are initially covered by tiles, with the goal to remove as few tiles as possible to make a decision on the expressed emotion. The children would learn to remove tiles over the eye and mouth area to maximise their chances in doing well, with the number of tiles over the photo increasing and becoming smaller on each level up. The aforementioned evaluation methods could then be used to test the autistic children’s eye-tracking performance before and after playing theFaceTilegame (i.e., until reaching the desired game performance level). If theFaceTilegame actually improves the face viewing strategy (i.e., spending more time in the eye and mouth region) the face-specific neural processes should also improve as indexed by a larger N170 ERP compared to the pre-support program level. These measures could be further used to test possibly improved social comprehension more generally, for instance, employing tests such as the NEPSY-II[57], Cambridge Face Memory Test–Children[58], or the Social Communication Questionnaire–Current[59] before and after playing theFaceTilegame. As such, the behavioural and cognitive measures of the application would make it a well-rounded, evidence-based program to help support autistic children to build skills in understanding facial expressions.

ACKNOWLEDGEMENTS

The authors acknowledge the autistic people and their families that have participated in the cited studies.