Too Many Bots: A Lesson for Online Quantitative Data Collection

Authors

  • Keri A Schwab California Polytechnic State University-San Luis Obispo
  • Ben Sherman California Polytechnic State University-San Luis Obispo
  • Marni Goldenberg California Polytechnic State University-San Luis Obispo

DOI:

https://doi.org/10.18666/JPRA-2023-12011

Keywords:

bots, quantitative analysis, methodology, online surveys, incentives

Abstract

“Bots”, computer software capable of taking surveys for an operator, pose a serious threat to the integrity of research that relies on publicly available online surveys. This paper addresses the issue of bot responses to online surveys and suggests several strategies for reducing and addressing these fraudulent responses. To combat this threat, researchers should employ specific methods for building, distributing, and processing surveys that deter and eliminate bot responses from the data set. Methods for anti-bot survey design include building in bot detection software to the survey, creating trap questions, and writing questions that require specific free-form answers. Survey distribution methods that avoid or hide monetary incentives, use a password-protected link, or employ some other form of population targeting will also receive less bot responses. Finally, data should be screened for bots after collection using a set of reliable criteria to identify and remove bot responses.

References

References

Cheng, Y. (Daniel), Farmer, J. R., Dickinson, S. L., Robeson, S. M., Fischer, B. C., & Reynolds, H. L. (2021). Climate change impacts and urban green space adaptation efforts: Evidence from U.S. municipal parks and recreation departments. Urban Climate, 39, 100962. https://doi.org/10.1016/j.uclim.2021.100962

Das, M., Ester, P., & Kaczmirek, L. (Eds.). (2010). Advances in Applied Methods and Research Strategies. Routledge. https://doi.org/10.4324/9780203844922

Griffin, M., Martino, R. J., LoSchiavo, C., Comer-Carruthers, C., Krause, K. D., Stults, C. B., & Halkitis, P. N. (2022). Ensuring survey research data integrity in the era of internet bots. Quality & Quantity, 56(4), 2841–2852. https://doi.org/10.1007/s11135-021-01252-1

Henderson, K., Grappendorf, H., Bruton, C., & Tomas, S. (2013). The status of women in the parks and recreation profession in the United States. World Leisure Journal, 55(1), 58–71. https://doi.org/10.1080/04419057.2012.759142

Lebeuf, C., Zagalsky, A., Foucault, M., & Storey, M.-A. (2019). Defining and Classifying Software Bots: A Faceted Taxonomy. 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), 1–6. https://doi.org/10.1109/BotSE.2019.00008

Martínez-Jauregui, M., Delibes-Mateos, M., Arroyo, B., & Soliño, M. (2020). Addressing social attitudes toward lethal control of wildlife in national parks. Conservation Biology, 34(4), 868–878. https://doi.org/10.1111/cobi.13468

McMaster, H. S., LeardMann, C. A., Speigle, S., Dillman, D. A., & Millennium Cohort Family Study Team. (2017). An experimental comparison of web-push vs. Paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Medical Research Methodology, 17(1), 73. https://doi.org/10.1186/s12874-017-0337-1

Nwana, H. S. (1996). Software agents: An overview. The Knowledge Engineering Review, 11(3), 205–244. https://doi.org/10.1017/S026988890000789X

Pozzar, R., Hammer, M. J., Underhill-Blazey, M., Wright, A. A., Tulsky, J. A., Hong, F., Gundersen, D. A., & Berry, D. L. (2020). Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire. Journal of Medical Internet Research, 22(10), e23021. https://doi.org/10.2196/23021

Prince, K. R., Litovsky, A. R., & Friedman-Wheeler, D. G. (2012). Internet-mediated research: Beware of bots. The Behavior Therapist, 35, 85–88.

Rooney, T. (2010). Introduction to IP Address Management. John Wiley & Sons.

Schonlau, M., & Couper, M. P. (2017). Options for Conducting Web Surveys. Statistical Science, 32(2), 279–292.

Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks. The Quantitative Methods for Psychology, 16(5), 472–481. https://doi.org/10.20982/tqmp.16.5.p472

Teitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., & Klitzman, R. L. (2015). Detecting, preventing, and responding to “fraudsters” in internet research: Ethics and tradeoffs. The Journal of Law, Medicine & Ethics: A Journal of the American Society of Law, Medicine & Ethics, 43(1), 116–133. https://doi.org/10.1111/jlme.12200

Van Selm, M., & Jankowski, N. W. (2006). Conducting Online Surveys. Quality and Quantity, 40(3), 435–456. https://doi.org/10.1007/s11135-005-8081-8

Wang, Z., Qin, M., Chen, M., & Jia, C. (2018). Hiding Fast Flux Botnet in Plain Email Sight (X. Lin, A. Ghorbani, K. Ren, S. Zhu, & A. Zhang, Eds.; Vol. 239, pp. 182–197). Springer International Publishing. https://doi.org/10.1007/978-3-319-78816-6_14

Yarrish, C., Groshon, L., Mitchell, J. D., Appelbaum, A., Klock, S., Winternitz, T., & Friedman-Wheeler, D. G. (2019). Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research. The Behavior Therapist, 42, 235–242.

Published

2023-09-15

Issue

Section

Research Notes