Introduction

Previously, we conducted a formative study using cultural probes and interviews to get a better understanding of our users and their vision for the future of music. The insights from the study helped us brainstorm and come up with 3 promising concepts grounded in constraints such as acceptability, feasibility, and impact. Based on the feedback from our peers and further discussions among team members, we decided to focus on the concept of a looping mat that would support classical musicians practicing alone by connecting them musically with their instructors and peers.

As we continued with our research and design process, we realized that we would need consider how the wide variation in the practice of classical music might constrain our design. Classical musicians employ a variety of instruments. They practice or perform by themselves or in groups of various sizes. Every instrument is positioned in a different way and requires different types of movements from the musician. This led us to our next research question: What kind of form and types of interaction should the looping mat have to accommodate these variations?

We identified that we would need very specific insights into how musicians would use a foot controller for recording and playing musical segments. With this goal in mind, we conducted a user enactment study by placing an open-ended design prototype of a looping mat in the context of a musician’s practice room.

Study Design

After analyzing the results from the cultural probe and interviews, we picked the concept of a looping mat and designed a user enactment matrix with three dimensions – activity, reactivity, and ownership. We developed eight user enactment scenarios based on the matrix, and piloted these scenarios class.

After conducting the two pilot User Enactment studies in class, we gained feedback that indicated we would need to narrow our concept’s scope slightly to make its purpose clearer to our users. We decided to adjust the dimensions of our matrix to reflect our concept’s use in solo practice only.
​
We conducted online research on foot controllers, looping and generative music, by watching videos on YouTube and looking at descriptions of foot controlled products available today. We were especially influenced by Loopy HD, a looping based composer for the iPad. To get further insights into how musicians currently play a foot controlled instrument, we observed a carillon performance by Prof. Tiffany Ng at the Burton Tower at the University of Michigan.
​

Based on our research and further discussions, we revised our user enactment matrix and scenarios to address potential interactions with foot-operated buttons, pedals, and/or foot-based gestures. We kept the dimension regarding personal/shared use, as we learned from our pilot studies that ownership type could impact how users engage with the device. We also narrowed down our focus to recording long tracks for practice, instead of looping or synthesizing clips. Our final revised scenarios tested different foot interfaces and different levels of engagement with external devices and other people.

Interface

1. Foot gesture-based interface for recording and playing back recordings during solo practice
   You are alone in the practice room and you want to record and playback your practice. You stand on the mat (like a large iPad that you stand on), and use your foot to start recording. You then start playing your instrument. Once you are done playing, you stop the recording by using your foot again. This recording gets saved on the cloud. It appears on your mat as a square block. You record a few more pieces and they get collected in a scrollable list of blocks. Then, you tap the square blocks of different recordings to play specific recordings.
   
   
2. Configurable buttons with haptic feedback for your feet, to record and play these recordings during solo practice
   You are at home practicing your part for an upcoming audition. You have the mat in front of you and your instrument in hand. When you tap on the record button, you feel a long vibration and you know it has started recording. You play a track and tap on the record button again to stop recording, this time you feel a very brief vibration.

Use of recordings

1. Organizing, sharing your recorded tracks, and discovering shared recordings through an app
   At this point, you have amassed a significant number of recordings of your practice sessions. You use the companion app to organize these recordings, rate them and take notes. Next, you use your favorite platform to share your recording to your ensemble. You can use recordings created and shared on other platforms and load it into the mat through this app.
   
   
2. Using different recordings simultaneously during practice
   You are practicing a melodic phrase that you feel comfortable playing while alone, but have trouble playing in rehearsal due to the complexity of the rhythm of other lead instruments. You want to practice this section alongside these other parts. How many parts would you want to hear? Find these loops, play them back while you are practicing, and make any adjustments you would normally make while practicing (replaying parts of the piece, moving to other sections of the piece). Stop the loops when you are done practicing.
   ​
   
3. A mat shared by a band and the teacher
   You are playing a part in a quartet. The band have a practice room with mat installed. One day you walk into the practice room and tap on the mat to see what your band mates have recorded. You play along with their recordings and record a version of your part.
   
   

Our study was designed to take place in a practice room at the music school, with a chair and a music stand. We provided a ukulele as a prop instrument to evoke the feeling of practicing an instrument. To test the efficiency of different foot interfaces, we drew out a number of functions/buttons that are commonly found in musical contexts- play, pause, stop, record, tune, metronome- and asked participants to stick the buttons on the interface in locations that would be convenient for them to use while playing their primary instrument.
​
We recruited three music school students on the spot and also tested with two students from other schools who had at least 10 years of experience with playing an instrument. Each team member took different roles including conducting user enactments, note taking, photo taking, and participant recruiting. Team members asked questions to the participants at points of interest during the studies, and debrief questions at the end of each study.

Study Results

Among our five participants, one plays saxophone, three of them take piano as their primary instrument, and one plays cello. When conducting the study, we found that all of them chose to sit on a chair while completing the following tasks. This is very different from what we anticipated when designing the prototypes, as we imagined more of a scenario where the musician would stand and practice. We probably should have invited participants whose instrument may require a standing position like violin or upright bass.

​Another thing we learned was that piano players have unique requirements. The piano itself has pedals, which the player operates by foot from time to time. Also, considering the big size of a piano, it will block player’s vision of their feet. The above two limitations strongly affect our mat design. As one participant suggested, we probably need to design two different mat models, one specifically for piano players and one for others.​

In UE2, participants arranged buttons in the way that would be most conducive to use while playing their primary instrument, then acted out the scenario. Some participants wanted the buttons right in front of their feet. Still, other participants wanted the buttons to be a bit further out, so that they would not accidentally press them while tapping their feet to keep tempo.
​

Other participants wanted the buttons to be right next to their feet. Some participants also indicated the need for buttons to more easily turn on the metronome, tuner, or page turner with their feet (as seen to the right), though not all of the participants made use of these buttons- perhaps due to lack of interest, but also perhaps due to confusion as to what the button icons represented in this study.
​

​We built a virtual affinity wall based on our observations and interviews during the five user enactments. Overall, we gained insights in the following 6 aspects.
​

1. Design and positioning
   1. For piano, the design should consider existing pedals, and a visual feedback closer to the piano's keyboard
   2. The mat should include fewer controls, which should be as close to the feet as possible
2. How to start/stop recording
   1. When asked to use a preferred gesture to start / stop recording, most preferred to tap a specific area with their foot. This positioning would differ for different instruments.
   2. To stop recording / playback, tap the same area again
3. Feedback for start/stop recording
   1. Signal vs Noise: Foot tapping sound should not get recorded
   2. Vibration feedback might not get noticed if the foot is off the mat, and might even get recorded
4. Problems with feedback and recording
   1. Audible countdown followed by long vibration when it is about to start recording
   2. One or two shorter pulses of vibration when it stops recording
5. Recording, sharing, navigating and selecting tracks
   1. Everyone had different opinions on an ensemble's group size that might find this product useful
   2. Some would prefer to use it for personal recording
   3. Some would find the shared recordings to be very useful
   4. They want the ability to control and play sections of multiple recordings
   5. Video recordings would be more useful for learning
   6. Ability to control about 3 tracks at the same time
   7. Ability to select tracks, change tempo, and volume with feet. In some cases, tempo should respond to the instrument.
   8. Prefer navigating the list of recordings through previous / next buttons instead of swipe gestures. Swipe would work better on phone.
6. Other features wanted
   1. Audio lessons through the mat
   2. Would like a metronome feature
   3. They liked the idea of a companion app
   4. Integration of page turning for sheet music (this seems to be one thing they really need, and most participants mentioned this at the end of the interview, unprompted)

Ideation & Selection

We individually sketched new design ideas based on the insights from user enactments and came together to discuss. Our new ideas all point to the direction of a simpler, smaller foot interface and pairing the device with a display to enable more features with simple foot controls.

However, we could not reach a consensus on what kind of screen–speaker–mic combination should the device have. So, we decided to consider further variations of the device.

To select among these options, we revisited our selection criteria from Milestone 2 and made some modifications based on our new findings. Then, we noted down the pros and cons based on these criteria.

• Portability
  • Is it easy to carry with you?
  • Is it compact enough to be stored in a student’s backpack?
• Customizability
  • Does it work well with different instruments?
  • Can it be used along with existing technology that the user owns?
• Acceptability
  • Is it intrusive to use?
  • Will musicians be comfortable using the product?
• Non-Disruptiveness
  • Will the musicians have to remember to ‘activate’ the product?
  • Will they have to be attentive to the product?
• Learnability
  • Will the device be easy to learn?
  • Will it be immediately usable or will there be a learning period?
• Plausibility
  • Is it technologically plausible?
  • Can the product work at different scales of population?

Ultimately, we decided Design #4 best fit our evaluation criteria and would fit into not only the practice spaces utilized by classical musicians, but could also be suitable for musicians practicing contemporary styles (jazz, rock, etc) as well.

System Proposal

​We propose foot-controlled mat and speaker combination that facilitates solo practicing activities of classical musicians. It allows musicians to record and listen to the music they play, load companion music recorded by others to play while they practice, and share the music they recorded with friends.

Based on our research and the feedback from user enactment studies, we are going to implement the following features:
​

• Start / Stop Recording
  • ​Users can start/stop recording by easily tapping one button with their foot
  • The button and speaker will light up to give visual feedback indicating the recording has started
• Load, play and pause up to 3 tracks independently
  • ​​Users can play and pause up to 3 tracks with 3 buttons on the pad
• Setting and adjusting recording tempo
  • When a user rests their foot on the pad and taps ambiently, the pad automatically senses the tapping tempo and sets the tempo of the current recording
• Metronome
  • Users can tap the metronome button to turn the metronome on and off
  • Metronome will be set at the current recording tempo
  • Users can step heavily on the pad's up and down arrows to manually adjust tempo
  • Users can alternatively tap the pad in their desired tempo to automatically set the metronome to their desired tempo​

• Load and share music recordings from and to other devices
  • ​Recordings are sent from the device to the user's account, which can be accessed by the user or their connections via the web or the mobile app
• Mic, speaker and screen
  • ​Built-in high quality mic to ensure the quality of recordings
  • A combination of speaker and screen that can be folded and detached from the foot control pad

The device's portable design enables music students to carry it with them. During a practice session, the user will switch on the device and place it near their left or right foot. Optionally, they can detach the screen, and place it on a table or music stand.

When the music student starts playing, they can rest their foot on the pad, tap their foot in rhythm, and the device will set the recording or metronome tempo automatically. The student can start and stop recording by tapping the left-most button.

When the student is listening to companion tracks while practicing, she/he can navigate to different parts of the piece by sliding a knob at the top of the speaker, which will move the needle along the sound wave on the speaker's visual interface.

After the practice session, the student can share their recordings by syncing the speaker to their smart phones and sending it to other connections in their network. They can also load music from other devices for their next practice session.​

Members of a band can share a device in their practice room and store recordings in the buttons on the pad, so that each person can access other parts easily when they practice alone.

Although it is a popular request, we will not be adding sheet music page turning in the first version of the product. One reason is currently different people have different solutions: some use iPads, some use other specific equipment. We are not sure if we want to reproduce another automatic music page turner or if the users would replace the device they are using with our new design. Another reason is not many people actually buy such a product even though there are many on the market. Is it because of the price, portability or utility? More user research needs to be done if we want to go deeper in this area.

This design is highly plausible and can be implemented with current technologies. In further iterations, however, we also need to consider variations in material and construction. For example, what material would the user like to see for the pad? Currently, plan to use a soft material to dampen tapping sounds that could be picked up from the microphone, but there will be a big difference between silicone, carpet, or other materials. While the silicone pad can be cleaned easily, carpet has a more comfortable and warm feeling. In terms of speaker quality and mic optimization, we think technology will develop dramatically if we set a 10 year timeframe. In that time, the speaker may have a higher sound quality without occupying such a large space, and the mic may be able to detect the sound in high quality wherever it is placed. 

Demo Proposal

Our demo presentation will be based on our system proposal. There are three main steps in this simulated experience:

• how to trigger the device,
• how to respond to the user’s foot operation,
• and how to display the music information on the screen.

To demonstrate the first step, we have decided to use large arcade buttons with LED to act as triggers. The buttons will be embedded on the pad and get triggered once they have been tapped. We will also put force sensors under the foot resting area to detect the tempo of the tapping. This part is pretty plausible. Due to skill limitation, we may not be able to build an embedded response system. Instead, we will connect the buttons and sensors to a laptop computer which will work as the recording and playback device. For the screen display, we plan to use a smartphone or a tablet encased in a 3D printed shell. The input data will be sent to the laptop for analysis, and the final visualization will be sent back to the smartphone/tablet to display. In a 10 year timeframe, the mic and speaker may be able to have a high quality while not occupying too much space. But for this prototype, we will use the phone to act as the mic and speaker. Lastly, we decide to 3D print the entire folding structure. The shape of the case will be carefully designed to achieve an optimal acoustic effect when a phone is put inside.

Conclusion

As we conclude Milestone 3, we have narrowed down our concept based on more extensive research with our target user group; we have a clear vision of how, when, and where it will be used, and by whom; and we have identified clear interactions with the system and have prioritized features of the user interface, both graphically and kinetically. We have a clear plan for building our prototype and demonstrating the system. Some questions do remain, however. We will need to do some testing to find the optimal position for the speaker and mic (e.g. height, angle, distance from instrument). We also want to test between physical buttons and touch screens to identify which forms classical musicians prefer, and which form may have fewer distractions and be easier to use during the practice. Our User Enactments gave us valuable information on macro-interactions, but as we move forward, we will need to potentially conduct some additional rapid prototyping to identify the most effective micro-interactions with individual buttons on the footpad as well as the detachable speaker/microphone/graphical interface.