Ateneo Laboratory for the Learning Sciences

Description of the Project

Eye-tracking is an emerging field in which special video cameras are used to follow and record the movements of a test subject. Eye gaze fixations indicate the test subject’s focus, hence the behavior of the test subject’s eyes can be used as an empirical measure of user attention. Eye gaze data has been used for a variety of fields from autism research (Navab, et al., 2012) to sports (Barfoot, et al., 2012) to animal behavior (Kano & Tomonaga, 2009) to fashion design (Ho, 2014) to reading comprehension (Graesser, et al., 2005).

Because accurate eye tracking requires relatively new and expensive technology, research and publications that make use of eye tracking data is somewhat rare.  This implies that the field is open for contributions.

We propose to develop national capacity to conduct eye tracking related research.  This entails the hosting of eye tracking workshops and the execution of a common research project. Because the proponents are from computer science, the common research project will focus on novice programmer tracing and debugging skills.

Statement of the problem

During the 4th International Outsourcing Summit held in 2012, Silicon Valley-trained Filipino information technology experts noted that “only 10 percent of information technology (IT) or computer science graduates are hireable.”  Although the Philippines produced 70,000 IT graduates in 2011, big companies consider most of these graduates to be undereducated and even uneducated by their standards.  To blame, these experts say, are substandard training and education programs.  Many schools allow their graduates obtained degrees without knowing how to program (Cuevas-Miel, 2012).

Programming is a fundamental skill that all graduates of IT-related courses must possess, yet it is a difficult skill to learn.  Since 2006, Dr. Rodrigo of the Ateneo de Manila has fostered an active research group that has studied novice programmer education.  Funded by several Department of Science and Technology grants, the Ateneo has studied novice programmer errors (Rodrigo et al., 2013), novice programmer affective states such as confusion (Lee, et al, 2011) and frustration (Rodrigo & Baker, 2009), and has built systems to analyzing novice programmer compilation logs (e.g. Dy & Rodrigo, 2010).

Prior works’ data sources have been limited to logs of novice programmer compilations and human observations. Both of these tended to be coarse grained.  The logs were essentially snapshots of novice programmer activity that have no representation for interim thought processes.  The human observations of each student were recorded every three minutes or so. This research project collects finer-grained data that enables us to track what it is that the students are attending to.  This, in turn, helps us determine whether they are in fact tracing the logic of the code or if they are confused.

Purpose of the research

This research project studies the tracing and debugging strategies employed by novice programmers, both as individuals and as pairs. Subsequent analyses will attempt to draw differences between high- and low-performing students and then arrive at recommended debugging strategies that teachers can then convey to students. The research questions that the group hopes to answer include but are not limited to the following:

• What problem solving strategies do novice programmers employ when visually parsing through a code fragment?

• How do the strategies employed by high-performing and low-performing students differ?

How does collaboration between pairs of novices affect the speed and accuracy with which bugs are identified and resolved?

Theoretical Framework

Eye-tracking is a data collection technique in which an individual’s eye movements are measured (Poole & Ball, 2006).  These measurements enable the researcher to determine where a person is looking (point-of-regard) and the sequence in which the eyes shift from one object to another.  Eye-tracking research is founded on the “eye-mind hypothesis” that asserts that eye traces are an indicator of where an individual’s attention is focused.   Visual attention refers to focalization. Individuals withdraw from some environmental stimuli so that they can effectively deal with others (Duchowski, 2007). Hence, eye-tracking data can provide researchers with insight regarding individuals’ visual information processing.

There are typically two types of eye-trackers: table-mounted and head-mounted.  Table-mounted eye trackers, as the name implies, are positioned on a table and are focused on the participant’s eye.  Head-mounted eye-trackers are similar eye glasses (Duchowski, 2007).

Poole and Ball (2006) and (Duchowski, 2007) describe several features used to measure eye-gaze.  Fixations refer to positions of the eyes when the eyes are relatively stationary.  They are an indicator of the amount of attention that is being applied to the point-of-regard.  Saccades refer to movement of the gaze from one point to another. No encoding takes place during a saccade, but regressive saccades, i.e. backward movement across a region that has already been visited, can indicate that an individual is having difficulty processing the material. Scanpaths are sequences of saccades and fixations.  They can indicate compliance with or departure from what is regarded to be an optimal path through the material.  Blink rate and pupil size can both indicate cognitive effort.  A low blink rate indicates a high workload while large pupils imply greater processing.

Eye-tracking research has been used to study human attention in a variety of circumstances.  These include but are not limited to reading, driving, marketing and advertising, and computer science.

Expected Output

This research project should result in prescriptions for teaching code comprehension, tracing, and debugging, some of the hardest computer-programming skills to learn.  It should also lead to research capacity building and publications in highly reputable journals and conferences.

Justification

The project is significant in at least three ways:

1. The addition of the eye-tracking component develops local expertise in the collection, analysis and interpretation of eye-tracking data. Because the technology is not yet widely used for this type of research, it has a high potential to lead to new contributions and collaborations in other areas of research.

2. It will contribute to the body of literature about novice programmer cognition, learning strategies, and debugging strategies.

3. It will contribute to novice programmer education by helping establish expert patterns of parsing code that can then inform instructors about how to teach novices how to read and trace through code fragments.  This can lead to a higher level of IT preparation and training.

Proposal Summary

The JOLLY collaboration will develop innovative web based technology to improve English comprehension and digital skills amongst disadvantaged Filipino children transitioning from primary to secondary school. Disadvantaged children often lack the comprehension skills needed at secondary school due to poor access to resources, learning opportunities or help at home. Up to 40% of 6th grade Filipino students have average to poor mastery of English: one of two official languages. Literacy is significantly related to life chances, long-term aspirations and self-efficacy. Digital skills are also vital in modern technological society, allowing learning across cultures. JOLLY will train students to use digital technology to understand, create and share joking riddles: an approach that has previously
produced improvements in reading comprehension and digital proficiency. JOLLY will motivate children to learn through fun exchanges of humour: a powerful component of Filipino and British cultures.

Objectives

• To increase reading comprehension and digital skills amongst learners aged 11-12 years old by using technology to engage them in understanding, creating and sharing joking riddles;
• To help young people to make the transition from primary to secondary school and increase their chances of using education to break free from the cycle of deprivation and disadvantage;
• To develop a user centred design process with teachers and learners through which they can contribute to the design of the activities and software we develop to support their understanding, creation and sharing of riddles. This should also increase student and teacher effectiveness and self confidence;
• To build a collaborative partnership between academic and non-academic partners in the UK and the Philippines that will endure beyond the lifetime of the Newton grant funding;
• To build capacity across the UK and Philippines through developing the skills and expertise of young researchers, teachers and students.

Details of the work that is to be undertaken in the proposed Institutional Links Collaboration

UNESCO (2011) reports that Philippine government spending on
education only grew by 0.2% from 1999 to 2008, despite the fact that the Philippine economy grew by 5% in the same period. As a result, more and more Filipinos are illiterate, the country has an increasing number of out-of-school youths, and those who do graduate are not prepared for the workforce (Philippines Department of Education, 2008). One of the economic consequences of this inadequate preparation is that a large segment of the population is unable to participate in high-skill, high-value industries such as BPO (Alava, 2010); A clear link between a learner’s understanding of the ambiguity in riddles and their English comprehension has been demonstrated (Yuill, 1988; Yuill and Oakhill, 1988). An trained group of children improved significantly more than those given decoding practice; The role of peer discussion in fostering comprehension improvement has also been demonstrated (Yuill, 2009). 24 children who worked in pairs to discuss and resolve ambiguities in joking riddles showed an increased reading comprehension significantly more than a group of 24 no-treatment controls; Computers have been shown to be effective at structuring and supporting collaborative discussion (Luckin et al., 2012). Specifically technology has been used effectively to scaffold collaborative discussion between pairs of students to promote the joint construction of meaning from ambiguous language (Yuill et. al. 2009); Personal narrative sharing and voicing opinions through online international children’s communities has been shown to lead to increased sense of empowerment, widening of perspectives and development of common forms of communication (e.g. Cassell and Tversky, 2005); Technology is increasingly shown to improve literacy skills of children with varying skills (Cassell, 2004); A wide literature discusses the way in which User Centred Design can benefit stakeholders (e.g. Norman & Draper 1986; Carroll, 2002).

Relevant Stakeholders and Potential Users

Teachers and students: Dr. Rodrigo and ACED have developed digital literacy materials for public school teachers and helped deliver the training in Metro Manila and the provinces. Past collaborations between Dr. Rodrigo and ACED led to the installation of learning software in public schools with the systematic collection of usage data and ACED and its network of schools is eager to learn about new ways in which technology can support learning and to develop teacher competencies both in content and pedagogy. The UK team also has existing relationships with primary and secondary schools and JOLLY will work with them to identify students who would be able to build online relationships with Filipino students. The UCL Institute of Education is the leading teacher-training organisation; this will enable us to effectively disseminate our findings to classroom practitioners. ACED, in partnership with faculty members of the Ateneo, also regularly offers public school teacher training and findings from JOLLY will be disseminated to schoolteachers and administrators as part of these regular training sessions. National & Regional Policy makers: We will work with the Philippines Department of Education and its public schools who are already in partnership with ACED. Existing relationships between ACED and the school system ease access to school leadership, teachers, and students. Existing relationships with policy makers in the UK, in particular the Dept. for International Development will be built upon to communicate what JOLLY can offer disadvantaged learners. Educational Technology companies: Luckin and Porayska-Pomsta work closely with many Educational technology companies, particularly small and medium sized companies and startUps. We will tap into this community to ensure that the work of JOLLY is widely disseminated and further software can be developed. We will assist our partners in the Philippines to develop similar working relationships local companies.