Evaluation of video labeling game ‘Waisda?’

On the 19th of May 2009 the Dutch Institute for Sound and Vision launched ‘Waisda?’ in collaboration with KRO, an innovative annotation game for video. In the form of a challenging game Waisda? asks the player to add keywords (tags) to video material. Its goal is to enhance the searchability of audiovisual archives. Waisda? is based upon the principle that one can speak of a meaningful description when several people individually give the same tag to a video. Waisda? is the world’s first operational video labeling game for audiovisual collections. Waisda? is part of a series of pilots that are being implemented within the project Images for the Future to explore the possibilities of public participation and has now been evaluated after a 6 month trial period.

The pilot phase of Waisda? turned out to be extremely successful. Since its launch the website drew 9.198 unique visitors and 2.296 players added as much as 340.551 tags to 604 items. The website was visited 12.297 times. A total of 137.421 (40,3%) of the 340.551 tags were added by two or more players within ten seconds. The evaluation demonstrates that ‘crowdsourcing’ of video annotations through a serious social game can indeed enable the enhancement of the findability of audiovisual material. Furthermore the report mentions a number of success factors and recommendations for a possible sequel.

The evaluation consists of a quantative and a qualitative analysis of the tags, usability research of the current interface and user research on the motivation of the players. The evaluation report is written by Lotte Belice Baltussen, in collaboration with Maarten Brinkerink en Johan Oomen, employed at Sound and Vision. Researchers of the Web & Media group at the VU University Amsterdam have also contributed significantly due to their involvement in the European research project PrestoPRIME. Within the project the VU – in collaboration with Sound and Vision – will do a follow-up study on automated processes for tag analysis, for improvement and expansion of the game- and interface design and for the integration of the tags in professional catalogue descriptions.

Read more