We are tackling the problem of how to collect sounds for a large, open-source database that can be used to train AI. To do this, we need to collect a large, diverse database of sounds in a sustainable and ethical way.
Working with Team Echoes of the ETC, we developed a novel solution to this problem: deploying a data collection game on the streaming service Twitch. Twitch streamers, who share their gameplay to audiences all over the world, ask their viewers to donate sounds. In their free time, viewers use a mobile app to collect sounds and share them with the streamer – and with us. Finally, streamers reward their community with attention on stream, including playing games that incorporate the sounds their community collected.
Here is a video trailer for the game experience:
Our test deployments have shown that this solution is highly viable. For example, one streamer played our game for ninety minutes, and requested to be notified when the game was released so he could play again.
Publications
Kim, B., Cheng, Y., Li, Z., Li, R., Tan, C., Wang, S., Shi, Y., & Hammer, J. (2019). Games with a purpose to collect home audio data. Extended Abstracts of the 2019 CHI Conference on Computer-Human Interaction in Play. [pdf]
Collaborators
Faculty
Project Staff
Students
Nikolas Martelaro (HCII)
Bryan Kim
Zoe Bai
Kevin Bergen
Jingya Chen
Carol Cheng
Yifan Deng
Kevin Han
Ann Maria Jose
Charley Li
Jeffrey Li
Liz Li
Ruoxi Li
Ashley Liang
Janine Louie
Yifeng Shi
Jesse Song
Dustin Stephan
Ava Tan
Rocky Wang
Sally Wang
Andrew Yu
Funding
Many thanks to Bosch and Philips Health for their support for this project.