Teaching Children Musical Perception with MUSIC-AR

Unfortunately in Brazil there is a non compulsory musical education in schools that leads to loss of sound/musical perception of Brazilian children. This fact, associated with the lack of software for the teaching of musical perception, inspired the creation of Music-AR, a set of software that uses Augmented Reality technology for the teaching of sound properties, such as timbre, pitch and sound intensity. There were four small applications for that: the first one allows the child to manipulate virtual objects linked to sounds, this way, the child can loosen and stretch virtual objects relating them to the (bass and treble) sound pitch; the second focus on the concept of sound intensity, associating it to virtual animals been far or near to the children; the third is related to duration of the sound (short or long), and the last is about timbre – the personality of the sound. Tests were applied and the results are presented in this work.


Introduction
According to Azuma et al [1], Augmented Reality (AR) is a technology that enables the user to see the real world, with virtual objects superimposed upon or composited with the real world. AR system has the following three characteristics: combines real and virtual, it is interactive in real time and it is registered in 3D.
AR has emerged another technology called Virtual Reality (VR). While VR can be defined as an advanced interface to computer applications, where the user can navigate and interact in real time in a 3D synthesized environment, using multisensory devices, AR simplifies its use because a nonconventional device is not required, just a webcam and markers are needed [2]. Although it emerged in the 60s, only 30 years later it started being used in various fields of knowledge and educational applications.
AR was indicated by the Horizon Report [3] as one of the technologies that will revolutionize education in the coming years. This is due to their fun and interactive features, among other reasons. The emergence of AR applications in education was made possible by: cheapening and improvement of the hardware and the need for more user-friendly interfaces for interaction with non-expert users; and the indispensability to work with other ways of teaching, using active practices. In this sense, using computers in the classroom allows, in many cases, simulation of situations not previously possible or imagined.
On the other hand, Schafer [4] states that music education should be directed to all and not only to those who demonstrate an aptitude. The author deals with the musical teaching as a creative process, but still little practiced in traditional teaching.
In this context, Australian musical education is distinctive from that in other countries in a number of ways. Its main strengths include the idea that music should be available to all children. Music is identified as a separate and important Key Learning Area within the curriculum [5].
However, in Brazil, musical education is not compulsory in schools which lead to loss of sound/musical perception in Brazilian children; for example, sound intensity is confused with the volume knob of the radios.
There are only few studies in the literature that address the use of AR in the field of Arts; specifically in musical education, it becomes even more scarce [6], [7], [8].
This work aims to present Music-AR, a set of software that uses AR for teaching the sound properties, such as timbre, pitch, duration and sound intensity. Thus, the software allows the child to manipulate virtual objects linked to sounds, such as the notion of loosening and stretching virtual objects relating them to the (bass and treble) sound time and place objects closer or further apart to give the notion of sound intensity. Tests were performed with EAI Endorsed Transactions on e-Learning 01-03 2015 | Volume 2 | Issue 5 | e3 Valéria Farinazzo Martins et al. 2 14 children and a music teacher. The results are presented in this paper.
The paper is structured as follows: Section II "Theoretical Foundation" presents aspects related to AR, an authoring tool for RA, and also on Musical Perception. Section III "Related Work" presents three works connected to music education with AR. Section IV "Music-AR" presents a description of the application, as well as aspects of its development. While in section V "Results" presents the tests conducted with the children. Finally, section VI "Conclusions" outlines the main conclusions obtained with the development of this project as well as brings future work.

Augmented Reality
Augmented Reality (AR) is a research area that combines the capture of real images with projection of 3D virtual images [9]. Combination of graphics provides greater interactivity between users and computers [10].
AR technology is considered low cost when compared to Virtual Reality technology [11]. The use of a non-conventional interaction device, such as data gloves, Head Mounted Display (HMD) and viewing digital caves, is not required [12]. These devices usually have an expensive cost for acquisition and development, because they are considered the new generation technology. In AR systems, interaction occurs with markers of paper along with a camera or webcam [13]. AR does not require high technology webcam, making its applications more accessible economically. Figure 1 describes four components needed to transform a real environment in an AR experience [14], [15]: 1) marker of paper; 2) webcam; 3) computer with AR software; d) screen for viewing images combined.

Figure 1. Operation of Augmented Reality
The AR process works as follow: initially user will point the marker to the webcam. Webcam will capture the image of marker and transmit to AR software. Software will recognize the marker and create a 3D image. So, software will mix 3D image with real image captured by the webcam. Finally, the software will display the two images (real and virtual) combined in preview screen. While user will move around the scene, within the field of view of the webcam, 3D image will move along, associated with marker. This merging of real and virtual images, together with real-time interactivity, causes effect of AR.
Among authoring tools to generate AR software, we opted for Flash Augmented Reality Authoring System (Flaras) [16]. This tool is easy to understand and use, even by people without technical knowledge in computer programming. Applications created can be run directly through the web browser with Adobe Flash Player plug-in. Applications can be used both online so if there is Internet access, as well as offline, otherwise.
Main screen of Flaras tool is shown in Figure 2. As can be seen, generation and creation of AR environment are performed through visual commands in menus located on the side of the screen. There is no need of knowledge in modelling 3D and computer programming. Developer user can select and download ready-made 3D objects, coming from repositories such as Google 3D Warehouse, and use Flaras tool to create applications.
Experiences with AR have been widely disseminated in several areas of knowledge, such as Education [10], Health [17], Entertainment [18], Publicity [19], and others [14]. As this study aimed to develop an AR musical application, during the literature survey, it was possible to identify some works involving music and AR, especially with educational and therapeutic purposes. Some of these works are presented in section 3. Teaching Children Musical Perception with MUSIC-AR 3

Musical Perception
According to Schafer [4], music is an organization of sounds (rhythm, intensity, melody and timbre), with the intention of being heard. In this regard, the vision of musical art transcends the orthodox definitions of classical, popular and folk music and personal preference. Therefore, music can also be described as an imitation of nature or daily sounds.
According to the author [4], the awareness of environmental sounds occurs through much training. The author further states that before the "noise pollution", the man has become "deaf". The musical education provides training people to be able to preserve familiar sounds and create new ones. Thus, it should listen to, analyze and make sounds. Still, for the author, musical education should be constantly linked to the experiences and discovery, just so you can get efficient results. These experiences and discoveries should happen very early in childhood.
Thus, in order to understand music, people must be able to distinguish the meaning of sound properties [4], [20]:  Pitch: refers to the bass, midrange and treble, i.e., how high or low a sound is.  Timbre: sound quality that distinguishes instruments. It is the personality of the sound.  Intensity: It is the strength which the sound occurs.  Amplitude: It is the variation of sound intensity.  Melody: the sound movement by difference frequency.  Duration: It is the variation of sound time; its extremes are: long and short.
Music should elicit different emotional responses in listeners, which are generated because they assume that the different extremes related to these aforementioned sound characteristics (e.g., high and low, strong and smooth, short and long, fast and slow) must have some power over emotions. This can be used to create a composition with a specific character that may affect the listeners' emotional state. [20], [21].

Related Work
Traditional music education often places a great emphasis on individual practice. Studies have shown that individual musical practice is often not very productive due to low motivation and interest of students [6]. System fosters should promote interest and motivation, as this will increase practice time and hence, most likely, learning outcomes. Thus, AR has been identified as a suitable technology for the above goals, due to its ability to create a perceptual and cognitive overlap between instrument (keys), instructions (notes), and music (sound).
Chow et l [6] used AR to create a perceptual and cognitive overlap between instrument and instructions to stimulate development in notation literacy and to create motivation through presenting as a game the task that was perceived as a chore. The user wears the head mounted display and sits in front of the keyboard. The keyboard connects to the computer using a MIDI interface. The video captured by the cameras in front of the device is projected onto the display to create the AR effect. The representation of musical notes in the system is visually indicated which key each written note corresponds to. The text and music are synchronised using visual cues. Each note is represented as a line above the corresponding key, where the length of the line represents the duration of the note. The notes approach the keys in the AR view in a steady time. When the note reaches the keyboard, the corresponding key should be pressed. Similarly, when the end of the note reaches the keyboard, the key should be released.
Huang et al [7] presents another work about AR based piano teaching system, which tracks the real keyboard of piano naturally. Following the virtual hands augmented on the keyboard, beginners of piano can practice playing piano. Instead of the marker in traditional AR system, the geometrical parameters of the piano keyboard are detected to create a marker. We first extract all the contours of the image, from which possible contours of keyboard are found. This way, the keyboard area is identified by the structure of white and black keys predefined. The authors evaluated the system with speed and accuracy.
Klebber et al [8] conducted an experiment using AR to learn a guitar musical instrument. The guitar system uses LED lights embedded in the fretboard to give direct information to the guitarist as to where to place his/her fingers. This was compared to a standard scale diagram. Results indicate that this system led to initial significant gains in performance over a control condition using diagrams.
Many other studies present positive results from the use of AR for learning musical instruments as: piano [22], guitar [23] and percussion [24]. Besides contributing to musical learning, AR musical systems have also aiding in therapeutic activities. [25] presents an experiment conducted with GenVirtual, an Augmented Reality musical system for music therapy. In GenVirtual, colored cards with graphic symbols replace the music keyboard. Unlike the keyboard, composed of unchangeable and fixed keys, in this system the therapist can print cards of different sizes and place them in various ways, according to the desired motor exercise. Each card represents a different musical note in the timbre of a given musical instrument. A colored virtual cube is attached to each card. GenVirtual was tested with children with Cerebral Palsy and Duchenne Muscular Dystrophy. Results showed that GenVirtual could serve therapeutic interventions including learning of cognitive, motor, psychological, social and to stimulate musicality. The GenVirtual was also tested in occupational therapy sessions [26] Francisco et al [27] created an AR application that uses music and visual therapy. It is an alternative tool for managing the mental stress level of an individual. The developed system enables the users to relax and unwind anywhere at any time. The goal of the software is to reduce the user stress level through audio and visual presentation alongside with the integration of augmentation of the original setting and application of appropriate concepts from music and visual therapy, with a web camera support.

Music-AR
Music-AR was designed based on interviews with a teaching music expert, Arts Coordinator of a private teaching school located in the city of São Paulo, Brazil. Information gathered through this interview have shown shortages of software for teaching children musical perception. According to the interviewed, a few software only teaches the trivial, such as musical notes. It is necessary, however, teaching sound properties that precede musical teaching, such as pitch, rhythm and timbre. Thus, the goal was not to teach music or a musical instrument, the core focus was to develop an application that can teach musical perception in children.
Flaras tool was used for developing the application. Although you can import 3D objects ready, for reasons of ease and adequacy of the objects they were created. As the target audience were children, Music-AR should have the following requirements: be attractive, playful, easy to use, easy to learn, fun, intuitive and visually pleasing. The development took about a month and involved two developers, authors of this work.
To achieve these goals, short and intuitive games (exercises) were generated. All games allowed the child could hear / see explanations about it and then use the concepts learned to solve the exercises. Figure 3 shows the sequence that the user performs one of the games -regarding the topic "Pitch". To explain the concept of pitch, the application brings the idea to stretch and loosen a rope -respectively high and low; this is what really happens in the stringed musical instrument.  Figure 4 shows the main screen for the application of sound intensity. This concept is relate to the strength with which the sound occurs In this exercise, the child after hearing an explanation about "Sound Intensity," three bees buzzing are presented, giving the feeling of distance -the closest or farthest beeand the sound with greater or lesser intensity.    Figure 6 presents the main screen for the application of "Timbre". This concept is related to the personality of the sound ;it is the ability to separate two sounds of different instruments, such as the sound of a piano and the sound of a violin. After listening to the explanations about timbre, the child must decide which sound is listening to. A screen with feedback is shown to the child.

Results
In order to validate the applications (games) developed two types of evaluation were performed: tests with potential end users and evaluation with a music teacher.
For tests with potential end users, a group of 14 children of both genders (nine girls and five boys) was selected. The age of the children is presented in Graphic 1. Five children had had contact with AR. Eight children had had contact with musical education.

Graphic 1. Quantity of Children by Age
The following methodology was applied for the tests with potential end users. First, we collected parental permission for the child to participate in testing. Some questions about their profile were asked. So, for each child who did not know AR, its principles were explained, as well as a small demonstration was performed. Subsequently, each child was asked to carry out the execution of applications about sound properties. Finally, some questions were asked about usability.
The physical environment used for the tests was not a controlled environment. The tests took place in the users home in order not to cause nuisance to the children`s usual environment. The users were then asked to run applications. Figure 7 shows users using the application.  Five children had some difficulty performing activities.  This difficulty is not related to the lack of musical education, neither to the lack of interest.  This difficulty is related to problems of usability of the technology itself, such as those caused by inadequate light or lack of detection of the marker in a few moments.  100% of the children could understand the musical perception using the applications.
About the usability questions, the children answered:  100% of the children believed the layout of four applications as appropriate and attractive.  11 children considered the mouse interaction as good and 13 considered the marker interaction as good.
EAI Endorsed Transactions on e-Learning 01-03 2015 | Volume 2 | Issue 5 | e3 Teaching Children Musical Perception with MUSIC-AR 7  100% of the children liked the sounds of the applications.  100% of the children liked using the applications and would like to use this type of technology frequently. On the other hand, by an interview with a music teacher, the following results were obtained:  The teacher did not know the technology of RA, but considered it important to be used for music education.  The teacher considered the applications are very interesting and fun and that could really help in teaching music.  The age range for children to use such games should be between four and six years.  The teacher believes that these games are easy to manipulate and that lets kids would not have a problem with it.  She considered that more games should be developed and then organized into a kit and distributed to children.

Conclusion
This work presents the use of AR for teaching musical perception in Brazilian children, especially on pitch, sound intensity, duration and timbre.
By the observation results, it is possible conclude that children could understand, through using AR, the concepts of pitch, sound intensity, duration and timbre -sound properties that must be learned before teaching music. All children could solve the musical perception exercises.
By the observation in questionnaire results, it could be noticed children felt motivated to use the technology. Another important observation is that children seem to understand how using AR as soon as they begin to interact with technology.
By the interview with a music teacher, it is considered that RA could be used for music education; applications developed can motivate children to learn music. AR seems to be adequate for use with children, due to its playful nature technology. AR can be used as a tool in the learning-teaching process of musical perception.
Flaras is a very easy tool to be used by nonspecialist people. On the other hand, it is a little limited, exactly by the same point. Some types of exercises on musical perception could not be performed due to lack of resources in the tool. For example, it´s impossible include a selection command (if-then and if-then-else). This command is used by programmers to determine what will happen if a certain event occurs. In the case of the "if" statement, if a certain event is true, then the command will be executed.
As future work, more games should be developed in order to create a repository of games for teaching musical perception. But to do so, other AR tools will be tested in order to provide increased functionality to other applications. We wanted to test, for example, Unity 3D game engine that is used on Qualcomm Vuforia platform [28]. Unity is a fully integrated development engine for creating games and other interactive 3D content; Vuforia platform makes it possible to write a single native application that runs on almost all smartphones and tablets.
We intend to create an illustrative interactive book to music education that enables the development of the following activities: a) know the musical instruments; b) analyze and compare the different types of sounds; c) develop auditory acuity; d) exercise the care, concentration and the ability of analysis and selection of sounds.
Another important factor is to use AR games regularly in a classroom in order to verify improvements in musical learning.