Sarah E. Frampton, PhD, BCBA
University of Nebraska Omaha
“Internal validity?” If you are less familiar with this scientific term, the title of this article may be a little off-putting. Prior to discussing what this all means, it may be helpful to provide our readers with an example of an all-too common scenario.
A caregiver of a child with autism spectrum disorder (ASD) sees an alert on their favorite social media app. A relative has tagged them on a post related to a new type of treatment designed to support persons with ASD. The caregiver sighs. This is not the first time, nor is it likely the last time. As the caregiver peruses the post and the link to the related research article, they note the exciting outcomes. Individuals in Treatment A made positive gains, gains more impressive than individuals in Treatment B. As their child is already in Treatment Intervention B, the caregiver begins to wonder if it is time for a change.
This experience, though hypothetical, is grounded in experiences reported by caregivers all over the world (e.g., Daniels et al., 2017). Caregivers report being inundated with information about potential interventions for their child diagnosed with ASD (Crais et al., 2020). Though the ease of access to information has many positive benefits, it comes with a price. Caregivers are tasked with navigating through increasing numbers of options and discerning the strength of related empirical evidence. The risks associated with selecting the wrong intervention weigh heavily on caregivers (Hughes et al., 2023). Ultimately, the overwhelming array of options, and resulting angst when navigating them, may serve as barriers to accessing effective services for their child (Sapiets et al., 2023).
The Association for Science in Autism Treatment (ASAT) aims to equip caregivers faced with these critical choices. Breaking down these research studies into their components is a useful practice in identifying strong scientific practices. Studies that adhere to strong scientific practices (i.e., methodologies) collectively inform the degree of evidence behind a particular treatment (Hume et al., 2020). Evidence of effectiveness is crucial for caregivers trying to make treatment decisions in support of their loved ones (Celiberti, 2022).
Let’s adopt a scientific lens and walk through some components this caregiver may want to consider when reviewing this research study. Each treatment may be considered a type of independent variable. The independent variable can be thought of as the cause, the change, or the new element that is introduced (Johnston et al., 2019). It may be a large change, like starting a new medication or new type of service model. It may be a small change, like trying a new form of prompting or visual support.
Researchers will then investigate how this independent variable impacts some aspects of the participant’s daily life. It may be the frequency of a particular behavior. It may be the consumption of novel foods. It may be the acquisition of daily living skills. These are all types of dependent variables– so named as changes in these variables are caused by or depend upon the independent variable (Johnston et al., 2019). In a well conducted research study, the change in the dependent variable will be directly attributable to the independent variable with limited possibility of interference from uncontrolled variables (Johnston et al., 2019). In other words, we will have confidence that the treatment made a difference in the lives of the participants and not something else.
When research studies include strong methodologies and convincingly show that the change in the dependent variable was attributable to the independent variable- they are considered to have strong internal validity. In a way, internal validity is the extent to which researchers followed the prescribed “recipe” in a study. To expand on this analogy, when restaurants have immaculately clean kitchens, sanitized tools, fresh ingredients, and well-trained chefs, their dishes will be top of the line. If a restaurant cuts corners on any of these components, their dishes may suffer (and their customers).
Restaurants undergo rigorous inspections and evaluations for adherence to best practices and are graded on their efforts. The same is true in the research world through the peer review process (Mruzek, 2012; Tereshko & Marya, 2024). In the discussion section of a research article, there is typically a section in which the author discloses known threats to internal validity. But it is possible some threats may be missed or not disclosed. Thus, it remains important for consumers of research (and food) to be aware of standards as they make choices between available options. Ultimately, it may be the case that Treatment A is superior to Treatment B. Or it might be that the internal validity of the study was weak, and no conclusions can be drawn from these data. When caregivers are considering a change in treatment, they should be prepared to inspect the research study to make this determination before they make a change to their child’s treatment.
Unfortunately, research is imperfect because it is conducted by humans (Johnston et al., 2019). In this series of Science Corner installments, we will detail different threats to internal validity that may be found within research studies. These threats may be likened to “health code violations” in that they range in severity from minor to major. Threats such as history, maturation, multi-treatment interference, testing, instrumentation, and infidelity will be discussed (see Table 1 for brief definitions informed by Ledford, 2018). We will describe these threats, provide examples, and unpack how the severity of the threat impacts consumer confidence in the treatment. We hope this serves to empower caregivers to recognize strong scientific practices as they navigate the flood of information available to them.
Table 1: Summary of Threats to Internal Validity
| Threat | Outcomes of the study may be due to: |
| History | Non-experimental, outside events. |
| Maturation | Development of the participants over time. |
| Multi-treatment interference | A second treatment; or, the effects cannot be clearly assigned to one treatment vs. another. |
| Testing | Repeated exposure to the evaluation conditions. |
| Instrumentation | Errors, bias, or deviations in measurement. |
| Infidelity | Incorrect implementation of the independent variable. |
References
Celiberti, D. (2022). Becoming a savvy consumer/educator. Science in Autism Treatment, 19(4).
Crais, E., McComish, C. S., Kertcher, E. F., Hooper, S., Pretzel, R., Mendez, L., & Villalobos, M. (2020). Autism spectrum disorder identification, diagnosis, and navigation of services: Learning from the voices of caregivers. Focus on Autism and Other Developmental Disabilities, 35(4), 246-256. https://doi.org/10.1177/1088357620922165
Daniels, A. M., Como, A., Hergüner, S., Kostadinova, K., Stosic, J., & Shih, A. (2017). Autism in Southeast Europe: A survey of caregivers of children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 47, 2314-2325. https://doi.org/10.1007/s10803-017-3145-x
Hughes, J., Roberts, R., Tarver, J., Warters-Louth, C., Zhang, B., Southward, E., … & Pearson, E. (2023). ‘It wasn’t the strategies on their own’: Exploring caregivers’ experiences of accessing services in the development of interventions for autistic people with intellectual disability. Autism, 0(0). https://doi.org/10.1177/13623613231196084
Hume, K., Steinbrenner, J. R., Odom, S. L., Morin, K. L., Nowell, S. W., Tomaszewski, B., … & Savage, M. N. (2021). Evidence-based practices for children, youth, and young adults with autism: Third generation review. Journal of Autism and Developmental Disorders, 51, 1-20. https://doi.org/10.1007/s10803-020-04844-2
Johnston, J. M., Pennypacker, H. S., & Green, G. (2019). Strategies and Tactics of Behavioral Research and Practice. Routledge.
Ledford, J. R. (2018). No randomization? No problem: Experimental control and random assignment in single case research. American Journal of Evaluation, 39(1), 71-90. https://doi.org/10.1177/1098214017723110
Mruzek, D. W. (2012). Focus on science: “Verification” and the peer review process. Science in Autism Treatment, 9(3), 18-19.
Sapiets, S. J., Hastings, R. P., Stanford, C., & Totsika, V. (2023). Families’ access to early intervention and supports for children with developmental disabilities. Journal of Early Intervention, 45(2), 103-121. https://doi.org/10.1177/10538151221083984
Tereshko, L., & Marya, V. (2024). Science Corner: Understanding the review process of peer-reviewed articles. Science in Autism Treatment, 21(07).
Citation for this article:
Frampton, S. (2024). An overview of internal validity: Was it really the treatment that made a difference? Science in Autism Treatment, 21(08).
Other Science Corner Articles:
- Role of replication in scientific validation
- Some cautions on the exclusive use of standardized assessments in recovery-oriented treatment
- Determining the effectiveness of treatments available to persons with autism – Part One
- ASD Intervention: How do we measure effectiveness?
- What is evidence-based practice and why should we care?
- Treatment Integrity: Why it is important regardless of discipline
- “Verification” and the peer review process
Other ASAT Articles:
- Explaining decisions to use science-based treatments
- Becoming a savvy consumer/educator
- Science, pseudoscience and antiscience
- The persistence of fad interventions in the face of negative scientific evidence: Facilitated communication for autism as a case example
#EarlyCareerAndStudents #Faculty #Researchers #SavvyConsumer
