Science Corner: An Overview of External Validity

Sarah Frampton, PhD, BCBA-D, LBA (NE)
University of Nebraska Omaha

ASAT's Science Corner As we have reviewed in our ongoing series on internal validity, researchers make efforts to ensure their studies are carefully designed and rigorously implemented (Frampton, 2024a). Researchers work to avoid threats to their study that may be introduced through faulty measurement tactics (i.e., instrumentation; Jackson & Frampton, 2025), low adherence to the treatment (i.e., infidelity; Rangel & Frampton; 2025), and inclusion of overlapping treatments (i.e., multiple treatment interference; Rey & O’Neill, 2024). Data are monitored closely to ensure that any effects on outcomes cannot be attributed to events outside of the experiment (i.e., participant history; Frampton, 2024b) or global development over time (i.e., participant maturation; Frampton & Rocheleau, 2024). As participants are recruited for the study, researchers randomize participants to conditions and may keep them naïve to their group assignment to prevent threats from selection bias, attrition, and diffusion (Stenroos & Kupzyk, 2025). As a result of these efforts, readers can have greater confidence that it was the intervention or treatment (i.e., independent variable) that led to a change in the target outcome (i.e., dependent variable) and not something else.

However, for potential consumers of treatments, there are still many factors that must be considered in relation to external validity. The external validity of a treatment has to do with the likelihood the outcomes observed within the study can be achieved outside of the experimental conditions in the study (Fahmie et al., 2023). To achieve external validity, the effects of a treatment must be repeated across studies (Walker & Carr, 2021), a tactic referred to as replication. Direct replication studies attempt to recreate almost all the same features of the original study but with new participants (Sidman, 1960). Systematic replications vary aspects of the procedures in the original study to clarify under what conditions the outcomes can still be achieved (Sidman, 1960). For example, conducting the study with different types of participants (e.g., older individuals with autism) in a different type of setting (e.g., home, school) or with implementers with different backgrounds (e.g., parents, teachers). Through replication, the scientific community can learn more about which participants will most benefit from a treatment and uncover biases or errors that may have influenced outcomes (Lang, 2024). The more replications demonstrating a consistent outcome, the stronger the external validity for the line of research (Walker & Carr, 2021) and the greater our confidence can be in the potential for it to work in the “real world.”

To further understand the importance of external validity, let’s consider an example. At a conference, a researcher is presenting a new and improved reading intervention that has produced marvelous outcomes with the participants with autism spectrum disorder (ASD). An administrator at this conference might be very interested in purchasing this new reading intervention for use within their school district. Though the researcher took careful steps to ensure strong internal validity within the study, which was verified through the peer review process (Tereshko & Marya, 2024), the administrator should not be so quick to accept this at face value. The administrator must pause to consider the external validity of the line of research investigating this reading intervention.

First, the administrator should ask the researcher if the outcomes shared at the conference have been replicated in other published studies. Though positive outcomes benefiting any group of students with ASD are worth celebrating, if the study has not been replicated, it will lack external validity and cannot be considered an evidence-based practice. Multiple replications, with varying participants, across entirely different research groups, is a common standard for determining whether a treatment can be considered evidence-based (Cook et al., 2015; What Works Clearinghouse, 2020). If this reading treatment has only been conducted with one group of students, it would fall short of this standard. In this case, the administrator would be wise to wait until replications have been conducted before committing the district’s resources.

If the study and its positive outcomes have been repeatedly replicated, the administrator should ask the researcher for more details about the participants included in the study. Though the participants had ASD, and the administrator is seeking resources for students with ASD, this is only one shared characteristic. The autism diagnostic label encompasses a very diverse pool of individuals, with very diverse characteristics. For this reading intervention, it would be important to know the global level of functioning of the participants included in the studies and what specific, prerequisite skills the participants demonstrated related to reading. Collectively, this information could help the administrator determine which students in the local district might benefit from this approach. Additionally, there may be cultural, socioeconomic, and language factors that may have influenced the efficacy of the published study. The administrator should be skeptical of the extent to which a reading intervention conducted with a small group of native English-speaking children in a highly resourced suburban school will produce the same results with students in an under-resourced rural, linguistically diverse school. These factors should be carefully weighed before committing district resources.

However, for caregivers and administrators consuming the research literature, it may be difficult to know what information to look for and how to interpret it when it is provided. In this upcoming series on external validity, the Association for Science in Autism Treatment (ASAT) will aim to bridge this gap. This series will include primers on how caregivers might interpret information included in research studies on participant characteristics, what factors are commonly missing or underreported about participants, and why these are important to consider. We will also address issues related to the settings in which research has been conducted and how these may influence generality (Fahmie et al., 2023). We hope this information empowers caregivers, providers, and other decision makers to sort through the many treatment approaches to find one most likely to work for their loved one.

References

Cook, B. G., Buysse, V., Klingner, J., Landrum, T. J., McWilliam, R. A., Tankersley, M., & Test, D. W. (2015). CEC’s standards for classifying the evidence base of practices in special education. Remedial and Special Education, 36(4), 220-234. https://doi.org/10.1177/07419325145572

Fahmie, T. A., Rodriguez, N. M., Luczynski, K. C., Rahaman, J. A., Charles, B. M., & Zangrillo, A. N. (2023). Toward an explicit technology of ecological validity. Journal of Applied Behavior Analysis, 56(2), 302-322. https://doi.org/10.1002/jaba.972

Frampton, S. (2024a). An overview of internal validity: Was it really the treatment that made a difference? Science in Autism Treatment, 21(08).

Frampton, S. E. (2024b). Science Corner: History as a threat to internal validity. Science in Autism Treatment, 21(10).

Frampton, S., & Rocheleau, A. (2024). Science Corner: Maturation as a threat to internal validity. Science in Autism Treatment, 21(9).

Jackson, K., & Frampton, S. F. (2025). Science Corner: Instrumentation as a threat to internal validity. Science in Autism Treatment, 22(1).

Lang, R. (2024). The crucial role of replication in scientific validation and identification of evidence-based practices. Science in Autism Treatment, 21(05).

Rangel, C., & Frampton, S. F. (2025). Science Corner: Infidelity as a threat to internal validity. Science in Autism Treatment, 22(3).

Rey, C. N., & O’Neill, P. (2024). Science Corner: Multiple treatment interference as a threat to internal validity. Science in Autism Treatment, 21(12).

Sidman, M. (1960). Tactics of scientific research. Basic Books.

Stenroos, S. & Kupzyk, S. (2025). Science Corner: Threats to internal validity in group design studies. Science in Autism Treatment, 22(4).

Tereshko, L., & Marya, V. (2024). Science Corner: Understanding the review process of peer-reviewed articles. Science in Autism Treatment, 21(07).

Walker, S. G., & Carr, J. E. (2021). Generality of findings from single-case designs: It’s not all about the “N”. Behavior Analysis in Practice, 14(4), 991-995. https://doi.org/10.1007/s40617-020-00547-3

What Works Clearinghouse. (2020). What Works Clearinghouse standards handbook (Version 4.1). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/handbooks

Reference for this Article:

Frampton, S. (2025). Science Corner: An overview of external validity. Science in Autism Treatment, 22(7).

Other Science Corner Articles:

Other ASAT Articles:

#Researchers #SavvyConsumer #Educators #Parents