Science Corner: History as a threat to internal validity

ASAT's Science Corner Many events will occur in the busy lives of the participants in a research study. These events may never be known to the researchers and may have limited effect on the outcomes. For example, the child may get a new lunchbox or start using a different brand of toothpaste. If these changes have no reasonable connection to the outcomes of interest in the study (i.e., dependent variables), they are not a threat to the internal validity of the study. However, it is possible that events occur during the time frame of an experiment that could affect an experiment’s outcome. These types of events are a cause for concern because they limit the extent to which the researcher can be certain that it was only the intervention (i.e., independent variable) that led to the change in the behavior (Johnston et al., 2019). Researchers refer to these as history threats- as they relate to uncontrolled events in the participants’ lived “histories.”

How History May Impact Outcomes in Single Subject Designs

History threats impact studies that utilize single subject designs such as reversal or multiple baseline designs. The hallmark of a single subject design is the ongoing measurement of the dependent variable (Johnston et al., 2019; Ledford et al., 2018). The use of repeated measures permits researchers to observe variability in the dependent variable that may occur as simply a part of everyday life. A participant may have a poor night of sleep or come down with a virus. These events may or may not impact the dependent variable under investigation. If a change in the level or trend of the dependent variable is observed during baseline, this could suggest a history threat (Ledford, 2018). If the researchers were to introduce the independent variable at that time, it would be difficult to discern what effects are due to the historical event and which to the independent variable. If applying strong experimental tactics, the researchers will simply wait and collect more repeated measures during baseline until stable patterns in these data are observed (Ledford et al., 2023). Once stability is achieved, the researchers can introduce the independent variable and observe the effects upon the dependent variable without interference. Use of repeated measures and close visual inspection of data to detect changes are critical tactics that result in a strong single subject design study (Ledford, 2018).

Given that some historical events may not be controlled or even known by the researchers, it is recommended that the effects of the independent variable on the dependent variable be replicated at different time points a minimum of two additional times (Ledford et al., 2023). If the change in the dependent variable is observed when and only when the independent variable is introduced on three occasions, it stands to reason that the independent variable is truly the cause of the observed change (Ledford et al., 2023). This degree of internal validity builds confidence that the effects obtained in this study can be relied upon and contribute to the larger body of evidence evaluating this intervention’s effects (Hume et al., 2023).

How History May Impact Outcomes in Group Designs

History threats can also impact groups of participants. For example, researchers may evaluate a new math curriculum by comparing results from students receiving this new approach (Treatment A) to those receiving the old curriculum (Treatment B). At the end of the study, it looked like the new curriculum led to greatly improved outcomes. But, as it turns out, all participants in Treatment B attended the same school. During the experiment, their school experienced record levels of snow and multiple school closures interrupted access to their regular educational experiences. This historical event only influenced participants in Treatment B, which may reasonably account for relatively less impressive effects observed for the participants in that group. Treatment A can only be believed to be better than Treatment B if the conditions were relatively similar throughout the course of the experiment. These historical events, which had a negative impact only on participants in Treatment B, threatened this experimental logic and weakened the believability of the outcomes obtained in the research study. Had the researcher randomly assigned participants to groups so that they were not clustered at schools, this threat would have been mitigated as the historical threat would have influenced both groups semi-equally. Use of random assignment is a critical component for a strong group design study (Ledford, 2018).

Conclusion

It would be impossible for a researcher to have control over every single event that occurs in the lives of their participants. Therefore, there will always be events that have the potential to influence outcomes in a study. If they or their child are participating in a research study, caregivers should make an effort to disclose these events to the researcher in a timely manner. As noted by Frampton (2024), researchers must use rigorous research tactics to avoid or at least quickly detect threats to internal validity. Any events that threaten the internal validity of an experiment should be reported in the discussion section of a research article as guided through the peer review process (Tereshko & Marya, 2024). Ultimately, when research outcomes can be replicated time and time again, with new participants, the more the findings can be relied upon when making decisions about care for individuals with autism spectrum disorder (Lang, 2024). As consumers of research, it is critical to remain skeptical of studies making strong claims based on inadequate procedures. Just as we would avoid sitting on a three-legged chair, built with shoddy materials, and inferior tools- we should not put weight on precariously demonstrated scientific findings.

References

Frampton, S. E. (2024). Science Corner: An overview of internal validity: Was it really the treatment that made the difference? Science in Autism Treatment, 21(08).

Hume, K., Steinbrenner, J. R., Odom, S. L., Morin, K. L., Nowell, S. W., Tomaszewski, B., … & Savage, M. N. (2021). Evidence-based practices for children, youth, and young adults with autism: Third generation review. Journal of Autism and Developmental Disorders, 1-20. https://doi.org/10.1007/s10803-020-04844-2

Johnston, J. M., Pennypacker, H. S., & Green, G. (2019). Strategies and tactics of behavioral rand practice. Routledge.

Lang, R. (2024). The crucial role of replication in scientific validation and identification of evidence-based practices. Science in Autism Treatment, 21(05).

Ledford, J. R. (2018). No randomization? No problem: Experimental control and random assignment in single case research. American Journal of Evaluation, 39(1), 71-90. https://doi.org/10.1177/109821401772311

Ledford, J. R., & Zimmerman, K. N. (2023). Rethinking rigor in multiple baseline and multiple probe designs. Remedial and Special Education, 44(2), 154-167. https://doi.org/10.1177/0741932522110253

Tereshko, L., & Marya, V. (2024). Science Corner: Understanding the review process of peer-reviewed articles. Science in Autism Treatment, 21(07).

Citation for this article:

Frampton, S. E. (2024). Science Corner: History as a threat to internal validity. Science in Autism Treatment, 21(10).

Other Science Corner Articles:

Other ASAT Articles:

#Researchers #SavvyConsumer #Educators #Parents

Science Corner: History as a Threat to Internal Validity