Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - weDRAW (Exploiting the best sensory modality for learning arithmetic and geometrical concepts based on multisensory interactive Information and Communication Technologies and serious games)

Teaser

In the last years, the use of multisensory technology for teaching has been increasingly used in the classroom. Whilst there has been a significant effort in developing technologies, such as tangible, sensor, and mobile technologies in the context of whole-body interaction and...

Summary

In the last years, the use of multisensory technology for teaching has been increasingly used in the classroom. Whilst there has been a significant effort in developing technologies, such as tangible, sensor, and mobile technologies in the context of whole-body interaction and embodied interaction for learning, to date, none of them sufficiently exploited multisensory integration for teaching and, none of them has taken into consideration these latest scientific results. Current technologies do not still sufficiently ground on psychophysics and pedagogical evidence.
In the last few years, researchers have provided a renewed understanding of the role of communication between sensory modalities during development. weDRAW starts from the idea that child has a preferential sensory channel to learn specific properties and that the visual signal is not always the more powerful and effective channel. Scientific studies show that specific sensory systems (such as vision or audition) have specific roles for specific perceptual properties. While vision is crucial for spatial perception, the audition is fundamental for temporal perception. Both space and time are strictly related to geometrical and arithmetic concepts. Today, at school, the visual channel is often the one most frequently exploited sensory modality for teaching while the other channels have a marginal role.
weDRAW creates and evaluates a novel technology for deeper learning of arithmetic and geometry grounded in these novel scientific discoveries and based on the use of multiple sensory modalities. In particular, weDRAW propose to open a new teaching/learning channel, personalized for each student, based on multisensory interactive technology (i.e., audio, tactile motor and visual), including a hardware and software platform to support this approach and three serious games to evaluate it.

Work performed

weDRAW consortium achieved many pedagogical, scientific and technological objectives.
At the scientific level, the weDRAW consortium worked with psychophysical methodologies to understand motoric capabilities of typical children at different stages of their development (6-7yo and 8-10yo) through carrying out psychophysical experiments. It also worked to identify the primary and most suitable sensory modality to teach typically developing children different arithmetical and geometrical concepts.
At the pedagogical level, the weDRAW consortium worked on the pedagogical design and evaluation of the new technological solutions proposed in weDRAW. Working closely with primary school teachers, pedagogics identified key mathematical and geometrical learning concepts for this age group where multimodal and multisensory engagement holds particular promise for understanding different mathematical and geometrical concepts. In particular, it defined the multisensory embodied and enactive pedagogical framework for teaching and learning arithmetical and geometrical concepts with primary school children and how and the extent to which the developed pedagogical framework can be applied to both typically developing and visually impaired children. In particular, the weDRAW consortium identified the learning concepts to be incorporated into the new technological solutions and serious games, the design requirements, and evaluated the results obtained with psychophysical testing.
At the technological level, the weDRAW consortium worked on the development of a library of software modules: i) for analysis of nonverbal motoric expressive and social behaviour of both individuals and small groups of children and for real-time control of sound, music, haptic, and visual feedback and ii) for multimodal (tactile, motor) analysis of nonverbal behaviour to capture the child’s learning-related affective states. It also worked to develop: i) an integrated hardware and software platform, supporting multiple different inputs and output devices, and scalable to different learning environments (e.g., school and home), integrating the above-mentioned libraries and supporting design and development of serious games and ii) three serious games, exploiting both the envisaged platform and libraries, and applying the art-inspired multisensory embodied and enactive pedagogical framework developed in the project. It also investigated different kinds of sensory feedback, namely: auditory feedback by means of sonification and active music listening, haptic feedback and visual feedback. Finally, weDRAW Consortium defined the use cases for the serious games and the requirements for the prototypes, and developed three serious games. The first one to learn arithmetic, the second one to learn geometry and the third one to learn together. We started to validate these technological solutions in children with and without visual impairment with pedagogical and psychophysical methodologies. Tools for early diagnosis of dyslexia have been also preliminary developed and tested in groups of children at elementary school.

Final results

weDRAW is expected to significantly contribute to progress beyond the state of the art along with different directions, having a direct impact on society, especially with respect to the enhancement of the teaching and learning processes. The major directions are:
- Novel concepts and approaches: using the best modality for learning a specific concept. weDRAW research aimed at defining which are the best modalities (vision, touch or audition) for understanding arithmetic and geometrical concepts.
- A novel ICT-supported paradigm to learn arithmetic and geometry: linking body movement and time in music and linking body movement and space in drawing. The serious games and the activities weDRAW are developing for children, link body movement with visual and auditory feedback, taking art (namely music and drawing) as a fundamental basis to design feedback.
- A novel unique language: teaching geometry to children with visual disability. A relevant aspect is that our approach and technology need to enable the application of the same learning paradigm to typically developed and impaired children.
- Novel ICT tools for early diagnosis: screening for dyslexia in children. Preliminary data collected suggest that dyslexic children have a lower precision on time perception. Such evidence would enable providing a fast screening approach to highlight situations in which dyslexia may be present. To this aim, an app was developed for assessing precision on time perception from different sensory modalities.
- A novel embodied and enactive pedagogical approach. weDRAW technologies are expected to provide new forms of interaction and understanding, but will also contribute to innovative pedagogical approaches, enabling teachers to modify the system and design activities to meet the students needs.
- Novel algorithms for analysis of nonverbal affective motoric and social interaction. The weDRAW project is working on algorithms for analysis of nonverbal affective motoric and social interaction in the learning process. Initial techniques were integrated into the first release of the weDRAW libraries and platform.
- Novel ICT products and services for technology-enhanced learning. weDRAW will release a platform, three serious games, and other prototypes that represent novel ICT products and services for technology-enhanced learning. A first version of the platform and of the serious games was already delivered and presented to the public.

Website & more info

More info: https://www.wedraw.eu/.