Published Versions 1 Vol 1 (4) : 350-361 2019
Toward Training and Assessing Reproducible Data Analysis in Data Science Education
: 2019 - 05 - 27
: 2019 - 08 - 18
: 2019 - 08 - 22
1430 12 0
Abstract & Keywords
Abstract: Reproducibility is a cornerstone of scientific research. Data science is not an exception. In recent years scientists were concerned about a large number of irreproducible studies. Such reproducibility crisis in science could severely undermine public trust in science and science-based public policy. Recent efforts to promote reproducible research mainly focused on matured scientists and much less on student training. In this study, we conducted action research on students in data science to evaluate to what extent students are ready for communicating reproducible data analysis. The results show that although two-thirds of the students claimed they were able to reproduce results in peer reports, only one-third of reports provided all necessary information for replication. The actual replication results also include conflicting claims; some lacked comparisons of original and replication results, indicating that some students did not share a consistent understanding of what reproducibility meant and how to report replication results. The findings suggest that more training is needed to help data science students communicating reproducible data analysis.
Keywords: Data Science Education, Reproducibility, Reproducible Data Analysis, Communication, Action Research
1. National Institutes of Health. Providing new incentives to foster open science and increase reproducibility. (2014). Available at:
2. G. King. Publication, Publication. PS: Political Science and Politics 39(1)(2006), 119–125. doi: 10.1017/S1049096506060252.
3. Nature. 1500 scientists lift the lid on reproducibility. (2016). Available at:
4. J. Kaiser. Rigorous replication effort succeeds for just two of five cancer papers. Available at:
5. W.G. Dewald, G.J. Thursby, & R.G. Anderson. Replication in empirical economics: the Journal of Money, Credit and Banking project. The American Economic Review 76(4)(1986), 587–603. doi: 10.2753/PET1061-1991290677.
6. R.D. Peng, F. Dominici, & S.L. Zeger. Reproducible epidemiologic research. American Journal of Epidemiology 163(9)(2006), 783–89. doi:10.1093/aje/kwj093.
7. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349(6251)(2015), aac4716. doi:10.1126/science.aac4716.
8. R.D. Peng. Reproducible research in computational science. Science 334(6060)(2011), 1226-1227. doi: 10.1126/science.1213847.
9. G. King. Replication, replication. PS: Political Science and Politics 28(3)(1955), 444-452. doi:10.2307/420301.
10. D.L. Donoho. An invitation to reproducible computational research. Biostatistics 11(3)(2010), 385–88. doi: 10.1093/biostatistics/kxq028.
11. V. Stodden, & S. Miguez. Best practices for computational science: Software infrastructure and environments for reproducible and extensible research. Journal of Open Research Software 2(1)(2004), p.e21. doi: 10.5334/jors.ay.
12. B. Howe. Communicating Data Science Results. MOOC course on Coursera. Available at:
13. R. Peng. Disseminating reproducible research is fundamentally a language and communication problem. Blog. Simply Statistics. Available at:
14. R. Gentleman, & D.T. Lang. Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics 16(1)(2012), 1-23. doi: 10.2307/27594227.
15. D.E. Knuth. Literate programming. The Computer Journal 27(2)(1984), 97-111. doi: 10.1093/comjnl/27.2.97.
16. K. Dnuggets. Python eats away at R: Top Software for Analytics. Data Science, Machine Learning in 2018: Trends and Analysis. Available at:
18. C. Gandrud. Reproducible Research with R and R Studio. Boca Raton, FL: CRC Press, 2013. isbn: 9781498715379.
19. Stackoverflow. How to create a minimal, complete, and verifiable example - help center - stack overflow. Available at:
20. E.T. Stringer. Action research in education. Upper Saddle River, NJ: Pearson Prentice Hall, 2008. isbn: 9780132255189.
21. J. Cohen. A coefficient of agreement for nominal scales. Educ Psychol Meas 20 (1960) 37–46. doi: 10.1177/001316446002000104.
Article and author information
Cite As
B. Yu & X. Hu. Toward training and assessing reproducible data analysis in data science education. Data Intelligence 1(2019), 381-392. doi: 10.1162/dint_a_00053
Bei Yu
B. Yu ( proposed the research problems, performed the research, collected and analyzed the data, and wrote and revised the manuscript.
Dr. Bei Yu is an Associate Professor at the School of Information Studies at Syracuse University. She is also the Faculty Lead of the Certificate of Advanced Study in Data Science. Her research area is in applied natural language processing, especially sentiment and opinion analysis. Before joining Syracuse she was a postdoctoral researcher at the Kellogg School of Management at Northwestern University. Dr. Yu earned her PhD from the Graduate School of Library and Information Science at University of Illinois at Urbana-Champaign. She holds both BS and MS degrees in computer science.
Xiao Hu
X. Hu ( proposed the research problems, analyzed the data and wrote and revised the manuscript.
Dr. Xiao Hu is an Associate Professor in the Faculty of Education at the University of Hong Kong. She is also the Program Director of Bachelor of Arts and Science in Social Data Science. Her main research interests are applications of data mining and machine learning in the domains of Information, Education and Culture, including learning analytics, music information retrieval, affective computing and data science. Dr. Hu holds a PhD degree in Library and Information Science, Master’s degrees in Computer Science and Electrical Engineering, and a Bachelor’s degree in Electronics and Information Systems.
Publication records
Published: Oct. 24, 2019 (Versions1
Data Intelligence