Practicing What We Preach (and Sometimes Study): Methodological Issues in Experimental Laboratory Research
Abstract
At the core of any causal claim is an experimental study, and laboratory research often provides the most valid arena for conducting experiments. Laboratory experiments allow researchers to provide internal validity to any phenomena they examine. However, researchers must consider and address certain methodological concerns when conducting lab research. Before participants ever agree to participate, forces are beginning to affect how they will respond to experimental manipulations. These and additional concerns continue during and after the completion of the experiment, yet many researchers do not fully understand their implications. As serious as these issues have always been, the introduction of social media and Internet data collection only highlight the importance of understanding and addressing methodological issues in laboratory experiments. In the following article, we discuss potential methodological flaws in laboratory research, examine research relevant to these problems, review evidence that suggests problems actually do contaminate our data, and suggest areas for future research. We conclude by providing recommendations for eliminating or alleviating these problems in future research, considering the role of replication studies as a post-hoc means of detecting these issues, and apply our conclusions to experimental research performed outside or a laboratory. In all, we believe that addressing these concerns in experimental research and practice will strengthen the validity of our conclusions, the replicability of our findings, and science as a whole.
Get full access to this article
View all access and purchase options for this article.
References
Adair J. G. (1973). The human subject: The social psychology of the psychological experiment. Boston, MA: Little, Brown.
Adair J. G., & Epstein J. S. (1968). Verbal cues in the mediation of experimenter bias. Psychological Reports, 22, 1045–1053. 10.2466/pr0.1968.22.3c.1045
Anderson C. A., & Bushman B. J. (1997). External validity of “trivial” experiments: The case of laboratory aggression. Review of General Psychology, 1, 19–41. 10.1037/1089-2680.1.1.19
Azrin N. H., Holz W., Ulrich R., & Goldiamond I. (1961). The control of the content of conversation through reinforcement. Journal of the Experimental Analysis of Behavior, 4, 25–30. 10.1901/jeab.1961.4-25
Blackhart G. C., Brown K. E., Clark T., Pierce D. L., & Shell K. (2012). Assessing the adequacy of postexperimental inquiries in deception research and the factors that promote participant honesty. Behavior Research Methods, 44, 24–40. 10.3758/s13428-011-0132-6
Carey B. (2011, November 2). Fraud case seen as a red flag for psychology research. The New York Times. Retrieved from http://www.nytimes.com/2011/11/03/health/research/noted-dutch-psychologist-stapel-accused-of-research-fraud.html
Chandler J., Mueller P., & Paolacci G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. 10.3758/s13428-013-0365-7
Crespi L. P. (1946). The cheater problem in polling. Public Opinion Quarterly, 9, 431–445. 10.1086/265760
DeSantis A., & Kayson W. A. (1997). Defendants’ characteristics of attractiveness, race, and sex and sentencing decisions. Psychological Reports, 81, 679–683. 10.2466/pr0.1997.81.2.679
DeYoung C. G., Peterson J. B., & Higgins D. M. (2002). Higher-order factors of the Big Five predict conformity: Are there neuroses of health? Personality and Individual Differences, 33, 533–552. 10.1016/S0191-8869(01)00171-4
Diener E., Matthews R., & Smith R. E. (1972). Leakage of experimental information to potential future subjects by debriefed subjects. Journal of Experimental Research in Personality, 6, 264–267.
Dion K., Berscheid E., & Walster E. (1972). What is beautiful is good. Journal of Personality and Social Psychology, 24, 285–290. 10.1037/h0033731
Durgin F. H., Baird J. A., Greenburg M., Russell R., Shaughnessy K., & Waymouth S. (2009). Who is being deceived? The experimental demands of wearing a backpack. Psychonomic Bulletin & Review, 16, 964–969. 10.3758/PBR.16.5.964
Edlund J. E., Hartnett J. H., Heider J. H., Perez E. J., & Lusk J. (2014). Experimenter characteristics and word choice: Best practices when administering an informed consent. Ethics & Behavior, 24, 397–407. 10.1080/10508422.2013.854171
Edlund J. E., Nichols A. L., Okdie B. M., Guadagno R. E., Eno C. A., Heider J. D., … Wilcox K. T. (2014). The prevalence and prevention of crosstalk: A multi-institutional study. The Journal of Social Psychology, 154, 181–185. 10.1080/00224545.2013.872596
Edlund J. E., Sagarin B. J., & Johnson B. S. (2007). Reciprocity and the belief in a just world. Personality and Individual Differences, 43, 589–596. 10.1016/j.paid.2007.01.007
Edlund J. E., Sagarin B. J., Skowronski J. J., Johnson S. J., & Kutter J. (2009). Whatever happens in the laboratory stays in the laboratory: The prevalence and prevention of participant crosstalk. Personality and Social Psychology Bulletin, 35, 635–642. 10.1177/0146167208331255
Farrow J. M., Farrow B. J., Lohss W. E., & Taub S. I. (1975). Intersubject communication as a contaminating factor in verbal conditioning. Perceptual and Motor Skills, 40, 975–982. 10.2466/pms.1975.40.3.975
Ferguson C. J., & Heene M. (2012). A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science, 7, 555–561. 10.1177/1745691612459059
Giner-Sorolla R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571. 10.1177/1745691612457576
Glinski R. J., Glinski B. C., & Slatin G. T. (1970). Nonnaivety contamination in conformity experiments: Sources, effects, and implications for control. Journal of Personality and Social Psychology, 16, 478–485. 10.1037/h0030073
Golding S. L., & Lichtenstein E. (1970). Confession of awareness and prior knowledge of deception as a function of interview set and approval motivation. Journal of Personality and Social Psychology, 14, 213–223. 10.1037/h0028853
Gross L. R. (1959). Effects of verbal and nonverbal reinforcement in the Rorschach. Journal of Consulting Psychology, 23, 66–68. 10.1037/h0043654
Guest L. (1947). A study of interviewer competence. International Journal of Opinion & Attitude Research, 1, 17–30.
Hakmiller K. L. (1971). Inter-subject communication. Psychological Reports, 28, 956–958. 10.2466/pr0.1971.28.3.956
Hansel C. E. M. (1966). ESP: A scientific evaluation. New York, NY: Charles Scribner's Sons.
Haslam S. A., & Reicher S. D. (2012). Contesting the “nature” of conformity: What Milgram and Zimbardo's studies really show. PLoS Biology, 10, e1001426. 10.1371/journal.pbio.1001426
Haslam S. A., Reicher S. D., & Birney M. (2014). Nothing by mere authority: Evidence that in an experimental analogue of the Milgram paradigm participants are motivated not by orders but by appeals to science. Journal of Social Issues, 70, 473–488. 10.1111/josi.12072
Haslam S. A., Reicher S. D., Millard K., & McDonald R. (2014). “Happy to have been of service”: The Yale archive as a window into the engaged followership of participants in Milgram's ‘obedience’ experiments. British Journal of Social Psychology. Advance online publication. 10.1111/bjso.12074
Highhouse S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554–566. 10.1177/1094428107300396
Horka S. T., & Farrow B. J. (1970). A methodological note on intersubject communication as a contaminating factor in psychological experiments. Journal of Experimental Child Psychology, 10, 363–366. 10.1016/0022-0965(70)90058-5
Horton J. J., Rand D. G., & Zeckhauser R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425. 10.1007/s10683-011-9273-9
Hyman H. H. (1954). Interviewing in social research. Chicago, IL: University of Chicago Press.
Jung J. (1982). The experimenter's challenge: Methods and issues in psychological research. New York, NY: Macmillan.
Kintz B. L., Delprato D. J., Mettee D. R., Persons C. E., & Schappe R. H. (1965). The experimenter effect. Psychological Bulletin, 63, 223–232. 10.1037/h0021718
Klein O., Doyen S., Leys C., Magalhaes de Saldanha da Gama P. A., Miller S., Questienne L., & Cleeremans A. (2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7, 572–584. 10.1177/1745691612463704
Klein R. A., Ratliff K. A., Vianello M., Adams R. B. Jr., Bahník Š., Bernstein M. J., … Nosek B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142–152. 10.1027/1864-9335/a000178
Langlois J. H., Kalakanis L., Rubenstein A. J., Larson A., Hallam M., & Smoot M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychological Bulletin, 126, 390–423.
Levin S. M. (1961). The effects of awareness on verbal conditioning. Journal of Experimental Psychology, 61, 67–75. 10.1037/h0047321
Levy L. H. (1967). Awareness, learning, and the beneficent subject as expert witness. Journal of Personality and Social Psychology, 6, 365–370. 10.1037/h0024716
Lichtenstein E. (1970). “Please don't talk to anyone about this experiment”: Disclosure of deception by debriefed subjects. Psychological Reports, 26, 485–486. 10.2466/pr0.1970.26.2.485
Logue A. W. (1998). Laboratory research on self-control: Applications to administration. Review of General Psychology, 2, 221–238. 10.1037/1089-2680.2.2.221
Makel M. C., Plucker J. A., & Hegarty B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542. 10.1177/1745691612460688
Marwit S. J., & Marcia J. E. (1967). Tester bias and response to projective instruments. Journal of Consulting Psychology, 31, 253–258. 10.1037/h0024662
Mills J. (1976). A procedure for explaining experiments involving deception. Personality and Social Psychology Bulletin, 2, 3–13. 10.1177/014616727600200102
Mills T. M. (1962). A sleeper variable in small groups research: The experimenter. Pacific Sociological Review, 5, 21–28. 10.2307/1388274
Minor M. W. (1970). Experimenter-expectancy effect as a function of evaluation apprehension. Journal of Personality and Social Psychology, 15, 326–332. 10.1037/h0029612
Morton T. A., Haslam S. A., Postmes T., & Ryan M. K. (2006). We value what values us: The appeal of identity-affirming science. Political Psychology, 27, 823–838. 10.1111/j.1467-9221.2006.00539.x
Newberry B. H. (1973). Truth telling in subjects with information about experiments: Who is being deceived? Journal of Personality and Social Psychology, 25, 369–374. 10.1037/h0034229
Nichols A. L., & Maner J. K. (2008). The good-subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151–166. 10.3200/GENP.135.2.151-166
Orne M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783. 10.1037/h0043424
Orne M. T. (1969). Demand characteristics and the concept of quasi-controls. In Rosenthal R. & Rosnow R. L. (), Artifact in behavioral research. New York, NY: Academic Press.
Page M. (1973). On detecting demand awareness by post-experimental questionnaire. The Journal of Social Psychology, 91, 305–323. 10.1080/00224545.1973.9923053
Reicher S. D., Haslam S. A., & Smith J. R. (2012). Working towards the experimenter: Reconceptualizing obedience within the Milgram paradigm as identification-based followership. Perspectives on Psychological Science, 7, 315–324. 10.1177/1745691612448482
Roediger H. L. (2012). Psychology's woes and a partial cure: The value of replication. Observer, 25, 27–29.
Rosenthal R. (1963). On the social psychology of the psychological experiment: The experimenter's hypothesis as unintended determinant of experimental results. American Scientist, 51, 268–283.
Rosenthal R. (1964). Experimental outcome-orientation and the results of the psychological experiment. Psychological Bulletin, 61, 405–412. 10.1037/h0045850
Rosenthal R. (1966). Experimenter effects in behavioral research. New York, NY: Appleton-Century-Crofts.
Rosenthal R. (1968). Experimenter expectancy and the reassuring nature of the null hypothesis decision procedure. Psychological Bulletin, 70, 30–47. 10.1037/h0026727
Rosenthal R., & Fode K. L. (1963). Psychology of the scientist: V. Three experiments in experimenter bias. Psychological Reports, 12, 491–511. 10.2466/pr0.1963.12.2.491
Rosenthal R., Persinger G. W., Vikan-Kline L., & Fode K. L. (1963). The effect of experimenter outcome bias and subject set on awareness in verbal conditioning experiments. Journal of Verbal Learning and Verbal Behavior, 2, 275–283. 10.1016/S0022-5371(63)80094-8
Rosenthal R., & Rosnow R. L. (1975). The volunteer subject. New York, NY: Wiley.
Sagarin B. J., Rhoads K., & Cialdini R. B. (1998). Deceiver's distrust: Denigration as a consequence of undiscovered deception. Personality and Social Psychology Bulletin, 24, 1167–1176. 10.1177/01461672982411004
Schultz D. P. (1969). The human subject in psychological research. Psychological Bulletin, 72, 214–228. 10.1037/h0027880
Seemann E. A., Carroll S. J., Woodard A., & Mueller M. L. (2008). The type of threat matters: Differences in similar magnitude threats elicit differing magnitudes of psychological reactance. North American Journal of Psychology, 10, 583–603.
Shames M. L., & Adair J. G. (1980). Effect of experimenter's expectancy in relation to type and structure of the experimental task. Psychological Reports, 46, 487–497. 10.2466/pr0.1980.46.2.487
Sigall H., Aronson H., & Van Hoose T. (1970). The cooperative subject: Myth or reality? Journal of Experimental Social Psychology, 6, 1–10. 10.1016/0022-1031(70)90072-7
Silverman I. (1968). The effects of experimenter outcome expectancy on latency of word association. Journal of Clinical Psychology, 24, 60–63. 10.1002/1097-4679(196801)24:1<60::AID-JCLP2270240115>3.0.CO;2-4
Silverman I. (1977). The human subject in the psychological laboratory. New York, NY: Pergamon Press.
Smith R. B. III, Helm B., & Tedeschi J. T. (1972). Verbal conditioning of “aware” subjects. The Journal of Social Psychology, 87, 151–152. 10.1080/00224545.1972.9918662
Spielberger C. D., & Levin S. M. (1962). What is learned in verbal conditioning? Journal of Verbal Learning & Verbal Behavior, 1, 125–132. 10.1016/S0022-5371(62)80009-7
Stricker L. J., Messick S., & Jackson D. N. (1967). Suspicion of deception: Implications for conformity research. Journal of Personality and Social Psychology, 5, 379–389. 10.1037/h0021209
Taub S. I., & Farrow B. J. (1973). Reinforcement effects on intersubject communication: The scuttlebutt effect. Perceptual and Motor Skills, 37, 15–22. 10.2466/pms.1973.37.1.15
Taylor K., & Shepperd J. (1996). Probing suspicion among participants in deception research. American Psychologist, 51, 886–887. 10.1037/0003-066X.51.8.886
Walsh W. B. (1976). Disclosure of deception by debriefed subjects: Another look. Psychological Reports, 38, 783–786. 10.2466/pr0.1976.38.3.783
Walsh W. B., & Stillman S. M. (1974). Disclosure of deception by debriefed subjects. Journal of Counseling Psychology, 21, 315–319. 10.1037/h0036683
Weber S. J., & Cook T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77, 273–295. 10.1037/h0032351
Wellman J. A., & Geers A. L. (2009). Rebel without a (conscious) cause: Priming a nonconscious goal for psychological reactance. Basic and Applied Social Psychology, 31, 9–16. 10.1080/01973530802659711
White H. A., & Schumsky D. A. (1972). Prior information and “awareness” in verbal conditioning. Journal of Personality and Social Psychology, 24, 162–165. 10.1037/h0033308
Wickes T. A. Jr. (1956). Examiner influence in a testing situation. Journal of Consulting Psychology, 20, 23–26. 10.1037/h0048983
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: June 1, 2015
Issue published: June 2015
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Review of General Psychology.
VIEW ALL JOURNAL METRICSArticle usage*
Total views and downloads: 1122
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0
Crossref: 0
-
Merging mathematics, biology, and local culture: exploring a tradition...
-
Joint action with human and robotic co-actors: Self-other integration ...
-
Suspicion About Suspicion Probes: Ways Forward
-
Presenting a sham treatment as personalised increases the placebo effe...
-
Sixty Years After Orne’s American Psychologist Article: A Conceptual F...
-
Distinctive roles of mPFC subregions in forming impressions and guidin...
-
Improving mindfulness through self-connection
-
Fostering Stress Resilience Among Business Students: The Role of Stres...
-
Copresence With Virtual Humans in Mixed Reality: The Impact of Context...
-
Construct Validation of Experimental Manipulations in Social Psycholog...
-
Applying insights from magic to improve deception in research: The Swi...
-
Why don’t we care more about carelessness? Understanding the causes an...
-
The Role of Self‐Connection in the Relationship between Mindfulness an...
-
Accelerating the Development of Second‐Order False Belief Reasoning: A...
-
Underlying Causes behind Research Study Participants’ Careless and Bia...
-
Multi-Method Health Co-Inquiry: a Case Illustration for Persons with C...
-
Criminal history and the workplace: a pathway forward
-
An empirical assessment of impact based tornado warnings on shelter in...
-
Participant Crosstalk: Issues When Using the Mechanical Turk
-
Intuitive and Deliberative Empathizers and Systemizers
-
Frightened by an Old Scarecrow: The Remarkable Resilience of Demand Ch...
-
Formatting and Presentation as Confounds in Online Studies
-
Communicating Increased Risk: An Empirical Investigation of the Nation...
-
A Closer Look at Social Psychologists’ Silver Bullet...
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
