2 years of leveling the playfield in musical instrument training
Note Blasters(Democratizing music for all) 🎶 — https://play.noteblast.org
Abstract: Aminjin
Technologies are rapidly evolving, and they are impacting human lives profoundly. And this also applies to the music industry, especially musical instrument training. There is the traditional option of receiving private expensive lessons including how to accurately tune your own instrument from a music teacher. However, it is not always viable for a teacher to be present, let alone the high cost. Furthermore, musical instrument training could overall be enhanced through a more technical approach without being constrained by the cost and human resources. With this goal in mind, we Noteblasters decided to investigate whether technology can address and resolve this common problem facing any musically inclined instrument players. Our Noteblast app is designed to detect pitches and give real-time feedback to the musicians on whether the correct note is being played or not. Through our multiple versions of prototypes, we were able to test our Noteblast app’s functionalities such as the tuner, providing real-time accurate feedback on the user’s playing, creating an easy and convenient tuning experience. The data shows aforementioned musical tuning problems can be resolved significantly more with a technology integrated device, which is our app.
Fig 1.1- Average cost for musical instrument tutoring in America per month in 2021
Introduction: Adelle
Music is often regarded as the universal language of humanity, possessing the ability to transcend cultural and linguistic barriers. Our goal is to make a product, helpful to everyone regardless of the diversity of their backgrounds, languages, traditions and region. In a world where everyone is interconnected by technology and a shared passion for music, the need for a universal platform that can unite musicians and students from diverse places in the world is ever more palpable. And essentially, a product like our Noteblaster that can effortlessly open doors and be a gateway to a global community of music learners(gamers as well). Notwithstanding, Noteblast is the solution.
Cognitive Development: Learning and engaging with music has been shown to enhance cognitive abilities such as memory, problem-solving, and spatial-temporal skills. A musical game that educates students provides a fun and engaging way for them to develop these vital cognitive skills.
According to an article from University of Southern California, a research by Frontiers in Psychology has shown that researchers have found that music had an impact on “positive youth development”. The researchers, Cho and Ilari, also showed that it contributed to students with measures for school connectedness and optimistic future expectations. They have also found that students that started their music learning before 8 were more hopeful about the future, strengthening their mental health.
However, the access to musical education has not been proven to be accessible to all segments of society. Although musical education is widely exposed to 92% of the public schools in the United States, the students’ participation and access is also based on the financial situations of the families.
(From SBO Magazine)
Noteblast can solve this problem. With its free access to anyone with the internet, it does not require students to have to pay for the education. As a team, Noteblast is a step forward to democratize music for all.
Prototype and Solution: Jeffrey
To combat our discovered problem, we decided to build a music app, capable of providing aspiring musicians a chance to tune their instrument, and practice sight reading through an interactive game.
Our app consists of a main page where users can navigate to the Browse, Tuner, and Settings pages. Helpful links and credits are located at the bottom of the screen (Fig 1.1).
Fig 1.1 — Title page with two buttons leading to different subpages
Through the Browse page, users can access a list of levels requested from a server, and play them by selecting the level then selecting play (Fig 1.2), or by clicking the play triangle when hovering over the level. Each level has its appropriate song, notes, tempo, and other miscellaneous metadata. The user can “favorite” a song, which they can later view and check.
Fig 1.2 — Browse page with different songs listed including one song selected
The gameplay goes as follows: a countdown begins, and a metronome before the song begins playing. Then, the song’s notes begin, scrolling from right to left, where a bar indicates the current position of the song over time. Similar to conventional sheet music, though scrolling in nature similar to a 2D video game’s side view camera, the notes must be played accurately, with the correct name, accidental, octave, and within 25 cents of the absolute frequency to be marked as correct. If a note is missed, meaning throughout the duration of the note, no seemingly accurate note was played, it is marked as wrong and turns red (Fig 1.3). This accuracy score is summed at the termination of a level as the number of correctly played notes out of the total number of notes. An according star score is given, out of 3, and further information is displayed. The user is prompted to either retry the level or return to the Browse page, where they can look for more songs or return to the main page (Fig 1.4).
Fig 1.3 — Notes moving from the right to left, including one wrong note
Fig 1.4 — Conclusion screen including accuracy and star count
The Tuner page allows the user to tune their instrument by analyzing their microphone, the same way the game uses microphone input to determine the user’s instrument note. It consists of a circle, which grows in size according to how accurate the current pitch of your instrument is. It also shifts in color, from red signifying extremely out of tune to green signifying an almost perfect match to frequency. Furthermore, the app also displays the believed note that is currently being played: note, accidental, and octave. To the right, the app displays the cents offset value, in a range from -50 to 50, with a negative value denoting flat and a positive value denoting sharp. Additionally, on the very right, the program also displays the hertz value of the microphone input. On the very bottom is a progress bar displaying the total amount of time playing with a note, separated into red, yellow, and green sections. Each section displays the amount of time in which the current note being held was played out of tune, partially in tune, and in tune, respectively. This provides the user with a sort of challenge, to hold their in-tune instrument for as long as possible, as to make sure most of the bar remains green (Fig 1.5).
Fig 1.5 — The tuner in action, showing the note, cents offset, and frequency
The Settings page includes numerous accessibility toggles for people of special needs, or for further customization. This includes the ability to toggle animations, which incorporates any transitions, looping displays, or background elements. There is also the ability to toggle the visual application background, which currently consists of randomly generated and animated Voronoi cells. Furthermore, users can customize the look of the app itself, from modern to legacy, which includes more rounded corners (Fig 1.6). Currently, colors of the application are not up for customization, but that might be considered in the near future.
Fig 1.6 — The Settings page, where users can customize their own preferences
Limitations: Jeffrey
This application is not without its limitations however. Due to the algorithmic nature of the pitch identification script, there is no way for us to determine any other qualities of sound, including specific attributes for specific instruments. Furthermore, since the application currently still does not have an account system nor a way to create levels within the game, the community aspect of our product design is slightly stunted. Furthermore, due to the features required within our app, some aspects of the website might not be supported on older versions, as we have not done extensive testing with varying Javascript, HTML, and CSS versions of varying browsers.
Future: Hyun
So far, our pitch detection algorithm used in Noteblast is called ACF2+. By taking in a sample of audio data, we can approximate which frequency the input signal is at. So far, it has worked decently for us, yielding accuracy and looptimes up to par with what we expect. However, to innovate and further improve our product, we aim to implement an AI model into our pitch detection. We plan to train our machine learning model by running through audio clips from various music professionals in our network. The goal for this new method of pitch detection is not only accuracy, but the ability to identify what needs improvement in the user’s sound.
We currently offer a simple but robust tuning section in our game purely for tuning. The only downside of such a tuner based on cents off a pitch cannot yield qualitative feedback, only quantitative. While musicians with more experience may be able to effectively interpret quantitative feedback, newer musicians getting into their instrument may lack such skills. We aspire to patch this deficiency by offering not only quantitative but also qualitative feedback. Instead of simply saying a user is off by a certain note by a certain amount, our model will return an instrument-specific output that describes the most likely problem in the user’s playing/instrument.
Currently, our UI interfaces for gameplay and tuning are very user-friendly, as shown in Figure 1.1 and Figure 1.2 respectively. However, our goal is to not only give a green percentage of accuracy but also tips/feedback on how the user’s sound was. Specifically, we desire the ability to output the user’s sound’s timbre, not just the pitch.
(Fig 1.1)
(Fig 1.2)
While we believe our current format is sufficient to provide the user with analysis on how their accuracy was while playing, we aspire to provide an even more in-depth analysis on the user’s playing. Figuring out a way to move from an algorithmic accuracy measurement to a qualitative instrument-specific output will be key to further increasing our app’s impact.
Although pitch detection and more beginner-friendly tuning is key to our mission, we also aim to further our user’s enjoyment of our app through our own music synthesis. By creating and training a machine learning model, our big goal for the future is to have a mode where we can endlessly generate our own and unique music for the user to play. This will not only solve the issue of boredom from repetition in our game, but also to provide a fresh and new experience every time a musician opens Noteblast.
Through these features planned for the future, we will more effectively drive our mission of democratizing music education for all.
🎶 Additional tech blogs by each member of Noteblasters:
Aminjin: https://play.noteblast.org/nb/blogarticle/aminjinblog/blog.md
Adell: https://play.noteblast.org/nb/blogarticle/adelleblog/README.md
Jeffrey: https://play.noteblast.org/nb/blogarticle/jeffreyblog/blog.md
Hyun: https://play.noteblast.org/nb/blogarticle/hyunblog/blog.md
🎶 Github Noteblasters: https://github.com/Noteblast/noteblast
Reference:
Raffio, N. (2023) Researchers find music education benefits youth wellbeing USC Today https://today.usc.edu/music-education-research/
Cho, E. & Ilari, B. (2023) Musical participation and positive youth development in middle school Frontiers in Psychology https://www.frontiersin.org/articles/10.3389/fpsyg.2022.1056542/full
Morrison, B (2022) Good News/Bad News While Most Students Have Access to Music Education, Nearly Four Million Students Do Not SBO Magazine https://sbomagazine.com/good-news-bad-news-while-most-students-have-access-to-music-education-nearly-four-million-students-do-not/