Too Many Bots: A Lesson for Online Quantitative Data Collection


  • Keri A Schwab California Polytechnic State University-San Luis Obispo
  • Ben Sherman California Polytechnic State University-San Luis Obispo
  • Marni Goldenberg California Polytechnic State University-San Luis Obispo



bots, quantitative analysis, methodology, online surveys, incentives


“Bots”, computer software capable of taking surveys for an operator, pose a serious threat to the integrity of research that relies on publicly available online surveys. This paper addresses the issue of bot responses to online surveys and suggests several strategies for reducing and addressing these fraudulent responses. To combat this threat, researchers should employ specific methods for building, distributing, and processing surveys that deter and eliminate bot responses from the data set. Methods for anti-bot survey design include building in bot detection software to the survey, creating trap questions, and writing questions that require specific free-form answers. Survey distribution methods that avoid or hide monetary incentives, use a password-protected link, or employ some other form of population targeting will also receive less bot responses. Finally, data should be screened for bots after collection using a set of reliable criteria to identify and remove bot responses.



Cheng, Y. (Daniel), Farmer, J. R., Dickinson, S. L., Robeson, S. M., Fischer, B. C., & Reynolds, H. L. (2021). Climate change impacts and urban green space adaptation efforts: Evidence from U.S. municipal parks and recreation departments. Urban Climate, 39, 100962.

Das, M., Ester, P., & Kaczmirek, L. (Eds.). (2010). Advances in Applied Methods and Research Strategies. Routledge.

Griffin, M., Martino, R. J., LoSchiavo, C., Comer-Carruthers, C., Krause, K. D., Stults, C. B., & Halkitis, P. N. (2022). Ensuring survey research data integrity in the era of internet bots. Quality & Quantity, 56(4), 2841–2852.

Henderson, K., Grappendorf, H., Bruton, C., & Tomas, S. (2013). The status of women in the parks and recreation profession in the United States. World Leisure Journal, 55(1), 58–71.

Lebeuf, C., Zagalsky, A., Foucault, M., & Storey, M.-A. (2019). Defining and Classifying Software Bots: A Faceted Taxonomy. 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), 1–6.

Martínez-Jauregui, M., Delibes-Mateos, M., Arroyo, B., & Soliño, M. (2020). Addressing social attitudes toward lethal control of wildlife in national parks. Conservation Biology, 34(4), 868–878.

McMaster, H. S., LeardMann, C. A., Speigle, S., Dillman, D. A., & Millennium Cohort Family Study Team. (2017). An experimental comparison of web-push vs. Paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Medical Research Methodology, 17(1), 73.

Nwana, H. S. (1996). Software agents: An overview. The Knowledge Engineering Review, 11(3), 205–244.

Pozzar, R., Hammer, M. J., Underhill-Blazey, M., Wright, A. A., Tulsky, J. A., Hong, F., Gundersen, D. A., & Berry, D. L. (2020). Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire. Journal of Medical Internet Research, 22(10), e23021.

Prince, K. R., Litovsky, A. R., & Friedman-Wheeler, D. G. (2012). Internet-mediated research: Beware of bots. The Behavior Therapist, 35, 85–88.

Rooney, T. (2010). Introduction to IP Address Management. John Wiley & Sons.

Schonlau, M., & Couper, M. P. (2017). Options for Conducting Web Surveys. Statistical Science, 32(2), 279–292.

Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020). Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks. The Quantitative Methods for Psychology, 16(5), 472–481.

Teitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., & Klitzman, R. L. (2015). Detecting, preventing, and responding to “fraudsters” in internet research: Ethics and tradeoffs. The Journal of Law, Medicine & Ethics: A Journal of the American Society of Law, Medicine & Ethics, 43(1), 116–133.

Van Selm, M., & Jankowski, N. W. (2006). Conducting Online Surveys. Quality and Quantity, 40(3), 435–456.

Wang, Z., Qin, M., Chen, M., & Jia, C. (2018). Hiding Fast Flux Botnet in Plain Email Sight (X. Lin, A. Ghorbani, K. Ren, S. Zhu, & A. Zhang, Eds.; Vol. 239, pp. 182–197). Springer International Publishing.

Yarrish, C., Groshon, L., Mitchell, J. D., Appelbaum, A., Klock, S., Winternitz, T., & Friedman-Wheeler, D. G. (2019). Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research. The Behavior Therapist, 42, 235–242.





Research Notes