The GLOBE: Evaluating Sensors for Interactive Computer Music Performance
For my Masters in Music Thesis, specialization in Sonic Arts, I propose a creative process for designing a DMI (Digital Musical Instrument) through an evaluation of sensors and in the context of interactive computer music performance. In this process the author uses the affordances and limitations of different types of sensors to generate ideas for possible DMI designs. The chosen design to construct is the GLOBE (Gesture-sensing, Luminous, Oscillating Ball Environment). It uses an Adafruit Feather HUZZAH with ESP8266, 16 FSRs that are connected with a CD74HC4067 16-channel analog multiplexer, an Adafruit TCS34725, a WS2812B LED strip, a Sparkfun LIS3DH 3-axis accelerometer, and a Sharp IR distance sensor (4-30cm). The sensors and physical geometry of the GLOBE supports many gestures and instrumental techniques for the performer to control sound in real time. Two Etudes performed with the GLOBE premiered on YouTube Live on May 22nd, 2020. My thesis performance was live streamed due to COVID-19. The online audience was asked to complete a voluntary survey to provide their feedback on the instrument and performance. The evaluation of the GLOBE as an effective musical instrument includes their feedback, as well as my perspective as the designer, composer/programmer, and performer.
Etude 1: The first etude exemplifies how the GLOBE spinning in its original hamster ball stand. The etude is minimalistic in that it mostly employs one-to-one mapping of the y-axis of the accelerometer, and the FSRs; it does not use the colour sensor or the IR sensor. Compositionally, this piece is partly improvised. There are two main sound engines within Etude 1: droning synths and Karplus Strong synthesis. I adapted the drone from Samuel Pearce-Davies’ drone synth Max patch. My iteration is a combination of seven rectangle and sine waves that are slightly detuned from one another. Each of the rectangle waves and sine waves are added together with a sawtooth wave. By randomly changing the multipliers of the waveforms, they move in between the sawtooth waveform and the rectangle or sine waveforms which result in a change of timbre. These changes in the multiplier ramp smoothly from one value to the next with my [CK_Line] abstraction. I mapped the FSRs on the left side of the GLOBE to the pitch of these combined synths, by converting the FSRs’ data to individual MIDI numbers. The last two FSRs toggle the master gain of the synths to ramp on/off. The Y axis of the accelerometer is also mapped to the cut-off frequency of the filters for both the Karplus Strong engine and the drone.
Etude 2: The second etude uses all of the sensors in the GLOBE and is more complicated in its playing techniques and sensor mappings. I base my decisions for mapping sounds to colour on objects and environments that are both associated with the colour and the sound. There is a bed track for each colour and a collection of shorter samples that play overtop. Each group of sounds pass through a unique process and effects so that they each sound very distinct from one another. The sounds mapped to red are crackling fire sounds, firetrucks, heartbeats, and cardinal bird chirps. This red group passes through granular synthesis. I mapped the Y axis of the accelerometer to change the maximum scaled value of the grain’s speed. I also mapped the IR sensor to change the maximum scaled value of the pitch and the maximum scaled value of the length of the grains. The green group contains an ambient forest soundscape with glass, sand, and shell samples. The Y axis of the accelerometer is mapped to the speed of the bed track and the IR sensor pitch shifts the bed track. Using [mc] objects, I make six instances of the pitch shifted bed track and pass them through a resonant bandpass filter. The FSRs are mapped to the six center frequencies of the filters and the Z axis of the accelerometer is mapped to the Q factor of the filters. Various flowing water sounds are mapped to blue. The Y axis of the accelerometer pitch shifts the bed track. The sound then goes through a flanger. The X axis of the accelerometer is mapped to the delay time of the flanger and the IR is mapped to the intensity of the flanger. The sounds in the yellow group are more abstract than the sounds in the other colour groups there are fewer yellow objects or animals that make distinct, recognizable sounds. There are buzzing bees, a rubber duck, canary chirps, dog barks and a baby’s laughter. I set the bees bed track to go through a series of delays with varying rates as well as time stretching and a comb filter. The X axis of the accelerometer changes the time stretching and the Y axis of the changes the rate of the delay. When the delay times change, it creates a glissando effect. The Z axis is mapped to a lowpass filter that smooths out the transitions between delay times. The IR sensor is mapped to the feedback gain of the comb filter. I use a polyphonic sample player formerly employed in my piece, Sound of My Hands, which consists of a [groove] object inside of a [poly] object.With the sample player I layer short samples overtop of the bed tracks. When the performer presses any FSR, the [urn] object randomly selects both a sample and the transposition of that sample.
For DMI designs for specific performers, or for performers with physical disabilities, or limited mobility, the concept for a DMI should always start with the performer's capabilities and gestures. For example, with TRAVIS I and II, the starting limitations are a violinist's typical playing gestures and the violin's physical geometry. But, for an instrument aimed for the capabilities of the typical human body, and not a specific type of instrumentalist, can a concept for a DMI effectively arise from the technology? This is my main research question. I made an application in Max MSP that would randomly output a list of sensors. I then considered the sensors affordances and limitations and I challenged myself to think of new instruments that had to incorporate those sensors. My favourite design from this process, the Ball, is what developed into the GLOBE. I have a github repository of the Max patch here: https://github.com/CLKo1/random-sensor-list
If you don't have Max MSP installed, I turned it into a standalone application. You can download it and try it on your own desktop here. This version is only compatible with Mac OS, not Windows.
I find that the one sensor category is the easiest to come up with ideas for and I often have multiple different DMI design ideas per sensor. I also took note which of these ideas could be made with an alternative sensor. For my best one sensor ideas, I find that I intuitively want to add another sensor or two despite the one sensor restriction. As I work through the two-sensor category, I only have one or two ideas for DMIs for each combination of sensors. When I consider the three-sensor category, there are times when I cannot think of any ideas with some combinations of sensors. Sometimes two of the sensors given are similar enough that it would be redundant to use both of them in one DMI, or it does not make sense to combine them given their limitations. Sometimes I cannot think of any ideas with the three or four provided sensors other than a DIY MIDI touchpad layout. Given these challenges, if I were to repeat this process, I would only consider lists or two random sensors, then com up with as many design ideas as possible, and finally think if there are any other sensor types that would enhance the design. Let me know how it goes if you try it out!
After the Youtube livestream I asked the audience to fill out a voluntary survey for feedback on the GLOBE design and the overall performance. The survey questions were:
Could you see/hear the correlation between how the instrument was played and how it changed the sound? If so, was one piece easier in which to identify this correlation than the other?
How many ways of playing the instrument (instrumental technique) could you identify?
Do you have suggestions for other creative playing techniques that were not used during the performance?
Could you perceive mistakes in the performance, and if so, what were they?
What about the performance did you find MOST effective?
What about the performance did you find LEAST effective?
Any other thoughts about the performance you would like to add?
I will post a summary of the results from this survey here after my thesis is completed.
Andresen, M., Bach, M., Kristensen, K., Nordahl, R., Serafin, S., Fontana, F., & Brewster, S. (2010). The LapSlapper - feel the beat. In Haptic and Audio Interaction Design: 5th International Workshop, HAID 2010, Copenhagen, Denmark, September 16-17, 2010. Proceedings (Vol. 6306, Lecture Notes in Computer Science, pp. 160-168). Berlin, Heidelberg: Springer Berlin Heidelberg.
Dahlstedt, P., & Dahlstedt, A. S. (2019). OtoKin: mapping for sound space exploration through dance improvisation. In Proceedings of the 2019 International Conference on New Interfaces for Musical Expression (NIME19), 156-161.
Freed A., Wessel, D., Zbyszynski, M., & Uitti, F. M. (2006). Augmenting the Cello. In Proceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06). 409-413.
Granieri, N., & Dooley, J. (2019). Reach, a keyboard-based gesture recognition system for live piano sound modulation. In Proceedings of the 2019 International Conference on New Interfaces for Musical Expression (NIME19). 375-376. http://www.open-access.bcu.ac.uk/id/eprint/7623 Gurevich, M., & Cavan Fyans, A. (2011). Digital Musical Interactions: performer–system relationships and their perception by spectators. Organised Sound, 16(2), 166–175. https://doi.org/10.1017/S1355771811000112
Jensenius, A., & Voldsund, A. (2012). The Music Ball Project: concept, design, development, performance. In Proceedings of the 2012 International Conference on New Interfaces for Musical Expression (NIME12). 300-303. http://urn.nb.no/URN:NBN:no-31755
Jordà, S., Geiger, G., Alonso, M., & Kaltenbrunner, M. (2007). The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In Proceedings of the 1st International Conference on Tangible and Embedded Interaction - TEI ’07, 139. https://doi.org/10.1145/1226969.1226998
Kimura, M., Rasamimanana, N., Bevilacqua, F., Zamborlin, B., Schnell, N., & Fléty, E. (2012). Extracting Human Expression for Interactive Composition with the Augmented Violin. In Proceedings of the 2012 International Conference on New Interfaces for Musical Expression (NIME12).
Ko, C., & Oehlberg, L. (2020). Touch Responsive Augmented Violin Interface System II: Integrating sensors into a 3D printed fingerboard. In Proceedings of the 2020 International Conference on New Interfaces for Musical Expression (NIME20).
Lucas, A., Ortiz, M., & Schroeder, F. (2019) Bespoke Design for Inclusive Music: The challenges of evaluation. In Proceedings of the 2019 Conference on New Interfaces for Musical Expression (NIME19). 105-109.
Marshall, M., & Wanderley, M. (2006). Evaluation of Sensors as Input Devices for Computer Music Interfaces. In Computer Music Modeling and Retrieval: Third International Symposium, CMMR 2005, Pisa, Italy, September 26-28, 2005. Revised Papers (Vol. 3902, Lecture Notes in Computer Science, pp. 130-139). Berlin, Heidelberg: Springer Berlin Heidelberg.
Matossian, V., & Gehlhaar, R. (2015). Human Instruments: Accessible musical instruments for people with varied physical ability. In B.K. Wiederhold et al. (Eds.) Annual Review of Cybertherapy and Telemedicine. (pp. 202-207). IOS Press.
Medeiros, C., & Wanderley, M. (2014). A Comprehensive Review of Sensors and Instrumentation Methods in Devices for Musical Expression. Sensors, 14(8), 13556–13591. https://doi.org/10.3390/s140813556
Mikalauskas, C., Viczko, A., & Oehlberg, L. (2019). Beyond the Bare Stage: exploring props as potential improviser-controlled technology. Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction - TEI ’19, 35–43. https://doi.org/10.1145/3294109.3295631
Miranda, E., & Wanderley, M. (2006). New digital musical instruments: Control and interaction beyond the keyboard. (21). Middleton, Wis.: A-R Editions.
Morgan, C. (2018). A Motion-based Controller for Real-time Computer Music with applications for Dance Choreography and Music Composition: the design, construction and programming of a wireless accelerometer-based interface system. In Collin College Sabbatical Leave Archives.https://www.collin.edu/hr/benefits/sabbatical_archive.html
Murray-Browne, T., Aversano, D., Hobbes, W., Lopez, D., & Chapman, D. (2014). The Cave Of Sounds: an interactive installation exploring how we create music together. In Proceedings of the 2014 International Conference on New Interfaces for Musical Expression (NIME14), 307-310.
Neustaedter, C. & Sengers, P. (2012). Autobiographical Design in HCI Research: designing and learning through use-it-yourself. In Proceedings of the Designing Interactive Systems Conference (DIS’12). 514–523. https://doi.org/10.1145/2317956.2318034
Nyce, D. (2003). Linear Position Sensors: Theory and Application. Hoboken: John Wiley & Sons, Incorporated.
Overholt, D. (2011). Violin-Related HCI: A taxonomy elicited by the musical interface technology design space. In Arts and Technology: Second International Conference, ArtsIT 2011, Esbjerg, Denmark, December 10-11, 2011, Revised Selected Papers. 101 (pp. 80-89). Berlin, Heidelberg: Springer Berlin Heidelberg.
Paine, G. (2004). Gesture and Musical Interaction: Interactive engagement through dynamic morphology. In Proceedings of the 2004 Conference on New Interfaces for Musical Expression (NIME04), 80-86.
Rowe, R. (1999). The aesthetics of interactive music systems. Contemporary Music Review, 18(3), 83-87, doi:10.1080/07494469900640361
Rowe, R. (1993). Interactive music systems: Machine listening and composing. Cambridge, Mass.: MIT Press.
Tahiroglu, K., Gurevich, M., & Knapp, R. B. (2018). Contextualising idiomatic gestures in musical interactions with NIMEs. In Proceedings of the 2018 Conference on New Interfaces for Musical Expression (NIME18). 126-131. doi: 10.5281/zenodo.1302701
Tanaka, A. (2000). Musical performance practice on sensor-based instruments. In M. Wanderley & M. Battier (eds), trends in gestural control of music, 389-405. IRCAM - Centre Pompidou.
Tarabella, L., & Bertini, G. (2004). About the Role of Mapping in Gesture-Controlled Live Computer Music. In U. K. Wiil (Ed.), Computer Music Modeling and Retrieval. Springer Berlin Heidelberg. ( 2771)217–224. https://doi.org/10.1007/978-3-540-39900-1_19
Thorn, S. D. (2019). Transference: a hybrid computational system for improvised violin Performance. Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, 541–546. https://doi.org/10.1145/3294109.3301254
Turchet, L. (2018). Smart Mandolin: autobiographical design, implementation, use cases, and lessons learned. In Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion - AM’18, 1–7. https://doi.org/10.1145/3243274.3243280
Visi, F., & Dahl, L. (2018). Real-Time Motion Capture Analysis and Music Interaction with the Modosc Descriptor Library. In Proceedings of the 2018 International Conference on New Interfaces for Musical Expression (NIME18), 144-147.
Wang, J., d'Alessandro, N., Fels, S., & Pritchard, R. (2011). SQUEEZY: extending a multi-touch screen with force sensing objects for controlling articulatory synthesis. In Proceedings of the 2011 International Conference on New Interfaces for Musical Expression (NIME11). 531-532.
Weinberg, G., Orth, M., & Russo, P. (2000). The Embroidered Musical Ball: a squeezable instrument for expressive performance. CHI '00 Extended Abstracts on Human Factors in Computing Systems, 283-284. https://doi.org/10.1145/633292.633457
Winkler, T. (1998). Composing interactive music: Techniques and ideas using Max. Cambridge, Mass.: MIT Press.
Xiao X., Haddad, D.D., Sanchez, T., van Troyer, A., Kleinberger, R., Webb, P., Paradiso, J., Machover, T., Ishii, H. (2016). Kinéphone: exploring the musical potential of an actuated pin-based shape display. In Proceedings of the 2016 Conference on New Interfaces for Musical Expression (NIME16). 259-264.
Zappi, V., & McPherson, A. (2014). Dimensionality and appropriation in digital musical instrument design. In Proceedings of the 2014 Conference on New Interfaces for Musical Expression (NIME14). 455-460. 10.5281/zenodo.1178993