We are tackling the problem of how to collect sounds for a large, open-source database that can be used to train AI. To do this, we need to collect a large, diverse database of sounds in a sustainable and ethical way.
Working with Team Echoes of the ETC, we developed a novel solution to this problem: deploying a data collection game on the streaming service Twitch. Twitch streamers, who share their gameplay to audiences all over the world, ask their viewers to donate sounds. In their free time, viewers use a mobile app to collect sounds and share them with the streamer – and with us. Finally, streamers reward their community with attention on stream, including playing games that incorporate the sounds their community collected.
Here is a video trailer for the game experience:
Our test deployments have shown that this solution is highly viable. For example, one streamer played our game for ninety minutes, and requested to be notified when the game was released so he could play again.
Many thanks to Bosch and Philips Health for their support for this project.
Publications
Kim, B., Cheng, Y., Li, Z., Li, R., Tan, C., Wang, S., Shi, Y., & Hammer, J. (2019). Games with a purpose to collect home audio data. Extended Abstracts of the 2019 CHI Conference on Computer-Human Interaction in Play.
Collaborators
Faculty
Nikolas Martelaro (HCII)
Students
[Add students here]