Science Corner: Threats to Internal Validity in Group Design Studies

Shayanne Stenroos and Sara Kupzyk, PhD, LP, BCBA-D
University of Nebraska Omaha

Have you ever read about a study and wondered if an intervention described in the study made a difference in the outcomes? In this article, we will use an example to describe important factors to consider when assessing internal validity when reviewing studies that use group designs.

A teacher is reading an abstract of a research study. The abstract says that Treatment A improved communication and social skills compared to Treatment B. This is exciting news because the teacher has been looking for a treatment to improve these skills with their students!

They read on and see that the study had two groups of participants. Group A received the new treatment approach (the independent variable) and Group B continued with their treatment as usual. This second group served as a control, allowing the researchers to compare the outcomes to learn what would happen without the treatment. In this case, the researchers measured the effect of the treatment on communication and social skills (the dependent variables). As they read more of the summary, they began to wonder if the new treatment was what led to the improvements or if the outcomes were happening due to chance or some other factor.

The extent to which we can attribute a change in the dependent variable(s) to a modification made to the independent variable is called internal validity (Frampton, 2024; Cahit, 2015). In this example, we would want to be confident that any change in communication or social skills was due to the treatment itself and not due to outside factors. If we could be confident that the results were most likely due to the treatment, then the study would have high internal validity.

Since high internal validity is important for substantiating outcomes, it is important to learn about possible threats to internal validity to look for when reading research. Some of these threats include selection bias, attrition, and diffusion (Table 1) and will be discussed below.

Selection Bias

The process in which researchers recruit, select, and assign participants to groups for a study can impact the integrity of the results. Ensuring that a sample is representative of the population in question is crucial, and that the groups do not differ significantly on key variables. Consider our previous example: The teacher notices that the participants in Group A scored above a specific communication score while those in Group B scored below that threshold. This means that the participants who will receive each treatment differ on an important characteristic- their initial communication skills. Thus, we would not know if it was the starting skill level or the treatment that led to differences in outcomes.

This is an example of selection bias, specifically sampling bias, since the sample is not representative of the population in question. In addition to sampling bias, the term selection bias includes self-selection bias, survivorship bias, nonresponse bias, and under-coverage bias (for more information, see Nikolopoulou, 2022). Clearly, there are many ways a sample could be selected that could jeopardize a study’s results, so implementing procedures to avoid selection bias is essential to a study’s design.

The most effective way to combat this bias is through randomization. Randomization can refer to both the initial selection of participants as well as the assignment of participants to treatment groups. It simply means selecting or assigning a participant to treatment groups at random to ensure that the sample is representative and that each treatment group has participants that are similar in important characteristics. Randomization can also aid when inclusion and exclusion criteria cause people from certain demographics to refrain from participating. When randomization cannot be achieved, controlling for preexisting differences and trends in the population is essential (Larzelere et al., 2004). The types of studies most at risk of selection bias are often observational studies, intervention research, and clinical trials since the selection of subjects cannot always be randomized (Nikolopoulou, 2022). Overall, randomization is one of the many tools we can use to minimize selection bias in research.

Attrition

Participant dropout is also a threat to confidence in research outcomes. This dropout rate is called attrition or subject mortality (Bhandari, 2021). While attrition can be expected to some extent in experimental research, especially when studies take place over a long period of time, it can still pose a threat to internal validity if subjects who drop out do so because of a particular factor of their demographics or treatment group (Cahit, 2015).

Recall our treatment study example. If the sample was representative of the population and the treatment groups were randomized, we would have eliminated the selection bias that was present before, but now, halfway through the study, 40% of the participants in Group A dropped out. All the participants who dropped out were not receiving early intensive behavioral intervention (EIBI) services before starting, so all the remaining participants were receiving EIBI. The study now has attrition bias since a large number of the participants who dropped out endorsed a key demographic factor. The results are now what we call skewed (Bhandari, 2021) since the complete data were not available to draw conclusions. Data for Group A now only includes a smaller group of participants that were also receiving EIBI, whereas Group B remained more representative of the population. The differences between the groups observed may be related to who was able to complete the study.

Again, the best way to avoid this attrition bias is to introduce randomization (Cahit, 2015). Randomization allows for participant dropout to be less impactful to the outcome of a study since those dropping out would be dispersed among multiple different treatment groups at random. It is important to note that random attrition is normal and expected, but attrition bias, where dropout is linked to another factor, hinders internal validity (Bhandari, 2021).

Diffusion

Following a different treatment than what a subject was assigned is called diffusion, and it can undermine the validity of research results. Let’s go back to our treatment study example once more. If participants were randomly assigned to Group A and Group B, researchers might trust that they are following that treatment as assigned. However, there are a handful of participants who were not in Group A whose families heard of its positive impact and started using some of the strategies in that treatment. Researchers assume those participants are still in the control group, even though they reaped some of the benefits of the strategy from the treatment group. This study now has diffusion or treatment contamination.

The most effective way to mitigate diffusion is through blinding, otherwise known as masking. In research, blinding means keeping participants and/or researchers unaware of the treatment group individuals are assigned (Daniels et al., 2024). This research tool is especially helpful in research in real-world settings. In our example, blinding may be achieved by requiring both groups to attend clinic sessions even though the targeted assistance tools are only provided to Group A. This way, participants do not know what group they are in because they are all under the assumption that they are receiving the targeted assistance. Blinding can be an effective tool in eliminating diffusion since it “consider(s) the impact of knowledge about study goals and procedures on data collection methods” (Daniels et al., 2024, p.655). In addition, data can be collected to measure the integrity of implementation for each treatment. This would provide information about the extent to which the interventions were implemented as intended. Other strategies for minimizing diffusion include attempting to limit interactions between the two groups by using different times and locations and asking participants to refrain from discussing their experiences with others over the course of the study.

Table 1

Summary of Critical Threats to Internal Validity in Group Design

Possible Threat to Internal Validity	Description	Ways to Decrease the Likelihood of the Threat
Selection Bias	Sample is not representative of the population being examined	Randomize (assign by chance) participants to the groups. Ensure that the groups don’t have big differences at baseline on important characteristics.
Attrition	Participants drop out of the study before its completion	Look to see if participants who dropped out had similar characteristics that could impact outcomes or if dropout differed between the two groups.
Diffusion/ Treatment Contamination	Participants adopt a different treatment than what they were assigned	Ensure the participants and/or researchers do not know which treatment group they are in. Check for treatment integrity- the extent to which the treatment was delivered as intended.

Conclusion

Safeguards such as randomization and blinding can help ensure strong internal validity. These measures can mitigate threats such as selection bias, attrition, and diffusion in research. Selection bias comes in many forms that can sometimes go unnoticed, and attrition can be detrimental to a study’s outcomes, but proper randomization can help decrease the likelihood of these threats. Additionally, diffusion can cause inaccurate results in a study, but blinding can help avoid deviation from assigned treatment groups. With these precautions, we can be more confident that the treatment (independent variable), is most likely responsible for the difference in the dependent variables that are observed.

Nonetheless, it is important to keep in mind that correlational results do not indicate causation. While we can conclude that independent and dependent variables are linked and most likely have a relationship, we cannot definitively conclude that one variable caused changes in the other. When reviewing research, individuals can look to see how the researchers took steps to decrease the threats to internal validity. In our example, the teacher identified threats that were not addressed. In addition, the teacher realized that the study participants were older than the students she worked with. In other words, the participants did not have similar characteristics to their students. Overall, the teacher decided it would be best to wait for additional research to evaluate the outcomes of the treatment before adopting it for use in their classroom.

References

Bhandari, P. (2021, November 1). Attrition bias: Examples, explanation, prevention. Scribbr. https://www.scribbr.com/research-bias/attrition-bias/

Daniels, B., Fallon, L. M., Palacios, A. M., Veiga, M. B., & Cook, A. L. (2024). Exploring threats to internal validity of direct assessment in single-case design research. School Psychology, 39(6), 646-657. https://doi.org/10.1037/spq0000664.

Frampton, S. (2024). An overview of internal validity: Was it really the treatment that made a difference? Science in Autism Treatment, 21(8).

Cahit, K. (2015). Internal validity: A must in research designs. Educational Research and Reviews, 10(2), 111-118. https://academicjournals.org/jounal/ERR/article-abstract/01AA00749743

Larzelere, R. E., Kuhn, B. R., & Johnson, B. (2004). The intervention selection bias: An underrecognized confound in intervention research. Psychological Bulletin, 130(2), 289-303. https://doi.org/10.1037/0033-2909.130.2.289.

Nikolopoulou, K. (2022, September 30). What is selection bias? Definition & examples. Scribbr. https://www.scribbr.com/research-bias/selection-bias/

Reference for this article:

Stenroos, S., & Kupzyk, S. (2025). Science Corner: Threats to internal validity in group design studies. Science in Autism Treatment, 22(5).

Other Science Corner Articles:

• Role of replication in scientific validation
• An overview of internal validity
• Science Corner: Maturation as a threat to internal validity
• Science Corner: History as a threat to internal validity
• Science Corner: Multiple treatment interference as a threat to internal validity
• Science Corner: Instrumentation as a threat to internal validity
• Some cautions on the exclusive use of standardized assessments in recovery-oriented treatment
• Retraction of published research
• Regression to the mean: Expand your science knowledge
• ASD Intervention: How do we measure effectiveness?
• Treatment Integrity: Why it is important regardless of discipline
• Evaluating research
• “Verification” and the peer review process
• Interventions for individuals on the autism spectrum and how best to evaluate their effectiveness

Other ASAT Articles:

• Explaining decisions to use science-based treatments
• Becoming a savvy consumer/educator
• Research Synopsis: Third time’s the charm or three strikes you’re out? An updated review of the efficacy of dolphin‐assisted therapy for autism and developmental disabilities.
• Resources for the implementation of evidence-based practice

 

#Researchers #SavvyConsumer #Educators #Parents