Published Versions 1 Vol 2 (3) : 417-442 2020
Refining Linked Data with Games with a Purpose
: 2019 - 04 - 15
: 20192019 - 0810 - 1010
128 3 0
Abstract & Keywords
Abstract: With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of data sets, to add missing knowledge and to identify and remove errors. Several approaches – mostly relying on machine learning and natural language processing techniques – have been proposed to address this refinement goal; they usually need a partial gold standard, i.e. some “ground truth” to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions. In this paper, we present an open source software framework to build Games with a Purpose for linked data refinement, i.e. web applications to crowdsource partial ground truth, by motivating user participation through fun incentive. We detail the impact of this new resource by explaining the specific data linking “purposes” supported by the framework (creation, ranking and validation of links) and by defining the respective crowdsourcing tasks to achieve those goals. We also introduce our approach for incremental truth inference over the contributions provided by players of Games with a Purpose (also abbreviated as GWAP): we motivate the need for such a method with the specificity of GWAP vs. traditional crowdsourcing; we explain and formalize the proposed process and we explain its positive consequences and we illustrate the results of an experimental comparison with state-of-the-art approaches. To show this resource’s versatility, we describe a set of diverse applications that we built on top of it; to demonstrate its reusability and extensibility potential, we provide references to detailed documentation, including an entire tutorial which in a few hours guides new adopters to customize and adapt the framework to a new use case.
Keywords: Human computation; Games with a purpose; Linked data; Knowledge graph; Data refinement; Data linking; Truth inference
This work was partially supported by the STARS4ALL project (H2020-688135) co-funded by the European Commission.
M. Acosta, A. Zaveri, E. Simperl, D. Kontokostas, S. Auer & J. Lehmann. Crowdsourcing linked data quality assessment. In: International Semantic Web Conference, 2013, pp. 260–276. doi: 10.1007/978-3-642-41338-4_17.
D.C. Brabham. Crowdsourcing. Cambridge, MA: MIT Press, 2013.
M.A. Brovelli, I. Celino, A. Fiano, M.E. Molinari & V. Venkatachalam. A crowdsourcing-based game for land cover validation. Applied Geomatics 10(1)(2018), 1–11. doi: 10.1007/s12518-017-0201-3.
I. Celino, D. Cerizza, S. Contessa, M. Corubolo, D. Dell'Aglio, E.D. Valle, & S. Fumeo. Urbanopoly: A social and location-based game with a purpose to crowdsource your urban data. In: Privacy, Security, Risk and Trust (PASSAT), 2012 International Conference on and 2012 International Conference on Social Computing (SocialCom), 2012, pp. 910–913. doi: 10.1109/SocialCom-PASSAT.2012.138.
I. Celino, S. Contessa, M. Corubolo, D. Dell'Aglio, E. Della Valle, S. Fumeo & T. Krüger. Linking smart cities datasets with human computation: The case of UrbanMatch. In: P. Cudré-Mauroux et al. (eds) The Semantic Web – ISWC 2012. Berlin: Springer, pp. 34–49. doi: 10.1007/978-3-642-35173-0_3.
I. Celino, I., E. Della Valle & R Gualandris. On the effectiveness of a mobile puzzle game UI to crowdsource linked data management tasks. In: 1st International Workshop on User Interfaces for Crowdsourcing and Human Computation, 2014.
I. Celino, A. Fiano & R Fino. Analysis of a cultural heritage game with a purpose with an educational incentive. In: International Conference on Web Engineering, 2016, pp. 422–430. doi: 10.1007/978-3-319-38791-8_28.
J. Chamberlain, M. Poesio & U. Kruschwitz. Phrase detectives: A web-based collaborative annotation game. In: Proceedings of the International Conference on Semantic Systems (I-Semantics' 08), 2008, pp. 42–49.
A.P. Dawid & A.M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics(1979), pp. 20–28. doi: 10.2307/2346806.
X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy … & W. Zhang. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014, pp. 601–610. doi: 10.1145/2623330.2623623.
I.P. Fellegi & A.B. Sunter. A theory for record linkage. Journal of the American Statistical Association 64(328)(1969), 1183–1210.
A. Ferrara, A. Nikolov & F. Scharffe. Data linking for the semantic web. In: Semantic Web: Ontology and Knowledge Base Enabled Tools, Services, and Applications, 2013, 169–200. doi: 10.4018/978-1-4666-3610-1.ch008.
C. Fürber & M. Hepp. Using SPARQL and SPIN for data quality management on the semantic Web. In: W. Abramowicz & R. Tolksdorf (eds.) Business Information Systems (BIS 2010). Berlin: Springer, pp. 35–46. doi: 10.1007/978-3-642-12814-1_4.
C. Guéret, P. Groth, C. Stadler, & J. Lehmann. Assessing linked data mappings using network measures. In: Extended Semantic Web Conference, 2012, pp. 87–102.
J. Hees, T. Roth-Berghofer, R. Biedert, B. Adrian, & A. Dengel. BetterRelations: Using a game to rate linked data triples. In: Annual Conference on Artificial Intelligence, 2011, pp. 134–138.
J. Howe. The rise of crowdsourcing. Wired magazine 14(6)(2006), 1–4.
D.R. Karger, S. Oh & D. Shah. Iterative learning for reliable crowdsourcing systems. In: Advances in neural information processing systems, 2011, pp. 1953–1961.
E. Law & L.v. Ahn. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5(3)(2011), 1–121.
B. Mozafari, P. Sarkar, M. Franklin, M. Jordan & S. Madden. Scaling up crowd-sourcing to very large datasets: A case for active learning. In: Proceedings of the VLDB Endowment 8(2)(2014), 125–136.
B. Mozafari, P. Sarkar, M.J. Franklin, M.I. Jordan & S. Madden. Active learning for crowd-sourced databases. arXiv preprint. arXiv:1209.3686, 2012.
H. Paulheim. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic Web 8(3)(2017), 489–508.
H. Paulheim & C. Bizer. Improving the quality of linked data using statistical distributions. International Journal on Semantic Web and Information Systems (IJSWIS) 10(2)(2014), 63–86.
A.J. Quinn & B.B. Bederson. Human computation: A survey and taxonomy of a growing eld. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 1403–1412.
W.M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66(336)(1071), 846–850.
G. Re Calegari, G. Nasi & I. Celino. Human computation vs. machine learning: An experimental comparison for image classification. Human Computation Journal 5(1)(2018), 13–30.
B. Settles. Active learning. Synthesis Lectures on Articial Intelligence and Machine Learning 6(1)(2012), 1–114.
R. Shah. Spam Hurts Crowdsourcing But Can't Kill It, (Forbes Contributor Opinions). Available at:
V.S. Sheng, F. Provost & P.G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2008, pp. 614–622.
E. Simperl, B. Norton & D. Vrandecic. Crowdsourcing tasks in linked data management. In: Proceedings of the Second International Conference on Consuming Linked Data-Volume 782, 2012, pp. 61–72.
K. Siorpaes & M. Hepp. Games with a Purpose for the Semantic Web. IEEE Intelligent Systems 23(3)(2008)
J. Sleeman & T. Finin. Type prediction for ecient coreference resolution in heterogeneous semantic graphs. In: Semantic Computing (ICSC), 2013 IEEE Seventh International Conference, 2013, pp. 78–85.
Sleeman, J., Finin, T., Joshi, A.: Topic Modeling for RDF Graphs. In: LD4IE at ISWC. pp. 48{62 (2015)
S. Thaler, E.P.B. Simperl & K. Siorpaes. SpotTheLink: A game for ontology alignment. Wissensmanagement 182(2011), 246–253.
U. Ul Hassan, S. O'Riain & E. Curry. Effects of expertise assessment on the quality of task routing in human computation. In: Proceedings of the 2nd International Workshop on Social Media for Crowdsourcing and Human Computation, 2013.
L. Von Ahn & L. Dabbish. Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2004, pp. 319–326.
L. Von Ahn & L. Dabbish. Designing games with a purpose. Communications of the ACM 51(8)(2008), 58–67.
J. Vuurens, A.P. de Vries & C. Eickho. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In: Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11), 2011, pp. 21–26.
J. Waitelonis, N. Ludwig, M. Knuth & H. Sack. WhoKnows? Evaluating linked data heuristics with a quiz that cleans up DBpedia. Interactive Technology and Smart Education 8(4), 2011, pp. 236–248.
C. Wieser, F. Bry, A. Bérard & R. Lagrange. ARTigo: building an artwork search engine with games and higher-order latent semantic analysis. In: First AAAI Conference on Human Computation and Crowdsourcing, 2013.
A. Zaveri, A. Rula, A. Maurino, R. Pietrobon, J. Lehmann & S. Auer. Quality assessment for linked data: A survey. Semantic Web 7(1)(2016), 63–93.
Y. Zheng, G. Li, Y. Li, C. Shan & R. Cheng. Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment 10(5)(2017), 541–552.
Article and author information
Cite As
I. Celino, G. Re Calegari & A. Fiano. Refining linked data with games with a purpose. Data Intelligence 2(2020), 417-442. doi: 10.1162/dint_a_00056
Irene Celino
The ideas and concepts presented in the paper are the results of at least three years of cooperation between the authors. I. Celino focused on data linking, crowdsourcing tasks and incremental truth inference. All authors contributed to the manuscript writing and they edited and reviewed the final version of the article.
Irene Celino is the Head of the Knowledge Technologies group at Cefriel, where she leads an R&D team and she is Portfolio and Project Manager. With expertise in Semantic Web and Human Computation technologies, her research activities cover the application of such innovative technologies to the design and development of Web applications, search engines, recommendations systems and mobile games, especially in Smart City and transportation-related scenarios. She has over 15 years of experience in over 30 R&D cooperative projects, both at National/Regional level and at European level within FP6, FP7, H2020 and EIT Digital. She is author of over 70 scientific publications in peer-reviewed journals, books and conferences.
Gloria Re Calegari
G. Re Calegari and A. Fiano focused on the framework, its applications, evaluation and tutorial. All authors contributed to the manuscript writing and they edited and reviewed the final version of the article.
Gloria Re Calegari is a researcher at Cefriel. She has a computer science background and her fields of expertise are Data Science and Human Computation technologies. Her research activities cover the design and development of gamified application and Games with a Purpose, next to the development of machine learning solutions that bring together humans and artificial intelligence. During her over 5 years of experience in R&D cooperative projects, both at National and Regional level, she published more than 20 scientific publications in peer-reviewed journals and conferences.
Andrea Fiano
G. Re Calegari and A. Fiano focused on the framework, its applications, evaluation and tutorial. All authors contributed to the manuscript writing and they edited and reviewed the final version of the article.
Andrea Fiano is a senior developer at Cefriel. Starting with the development of Web application in .Net, he continued with the development of backend solutions and REST APIs in Java and practiced with Single Page Applications and Progressive Web App in Angular and Node.js. He provides his expertise in the development of customer tailored solutions as well as in supporting the research branch. In particular, he has helped in the field of Human Computation with the development of some Games with a Purpose in the Smart City and Crowdsourcing scenarios.
Publication records
Published: July 16, 2020 (Versions1
Data Intelligence