Russell Lang, PhD, BCBA-D
Texas State University
Identifying treatment approaches that are reliably effective in addressing the education and support needs of people with autism is challenging. A 2006 survey of 552 parents identified 111 different treatment approaches that were provided to children with autism (Green et al., 2006). The list of treatments ranged from those with strong scientific backing, supported by numerous well-designed studies, to those lacking any scientific evidence at all, as well as options that have been evaluated in rigorous research but were found to be ineffective or harmful. In a follow-up study, parents identified where they learned about various treatment options and what factors they considered when selecting between options (Green et al., 2007). Two of the most common sources of information included a range of internet sites -many with misleading recommendations- followed by anecdotal reports and testimonials from other parents. Since 2006, the number of treatment options has only increased and, much of the information available online and from personal testimonials is perhaps even more accessible and still unreliable at best. The large and growing number of treatment options that claim to benefit people with autism and the varying levels of evidence underlying those options highlight the critical need to evaluate the strength of scientific evidence when attempting to identify a safe and effective treatment.
Selecting treatments supported by scientific research first requires an understanding that scientific validation progresses across a continuum – from initial, singular findings to replicated, rigorous evidence. Evaluating the underlying scientific support for various treatment options requires consideration of many factors. For example, previous Science Corner articles published by ASAT have noted the importance of treatment integrity (Brand, 2014) and verification (Mruzek, 2012). Another critical factor to add to this list is scientific replication. But what exactly does replication mean in the context of research focused on treatment efficacy, why is it important, and where can reliable information about treatment replication be found?
What is scientific replication?
In terms of treatment research, there are two broad types of replication: within-study replication and across-study replication. Within-study replication is typically emphasized when evaluating the effects of treatment with participants who are all in the same study receiving the same treatment, as in a multiple-baseline across participants’ study. Within-study replication occurs when the treatment effects identified with one participant are reproduced (replicated) with the next participant in the same study. Within-study replication is an important consideration when appraising the rigor (certainty of evidence) of any one single study. However, it is important to remember, a single study can never provide sufficient evidence to be certain that a treatment will be effective again, particularly when participant characteristics and conditions in the next study are dissimilar. For example, a study evaluating a treatment implemented by a practitioner with highly specialized training in a clinical setting cannot provide sufficient evidence to conclude with certainty that the same treatment will be effective or even appropriate if implemented by a teaching assistant in an inclusive school classroom. Likewise, a treatment study that shows language improvements in young children who already used spoken language before starting treatment cannot confidently claim that the same treatment would work as effectively for older children who have never used spoken language. Because a single treatment study, no matter how rigorous, cannot provide all the evidence necessary for the classification of a treatment as an evidence-based practice, this article’s focus is primarily on the need for across-study replication.
Across-study replication involves evaluating the extent to which a treatment consistently produces similar results across multiple studies. In some cases, a replication study is designed to be as similar to the initial study as possible and will involve participants who share similar characteristics, such as the same age range, support needs, and baseline skills that receive identical treatment procedures within settings highly comparable to the initial study. In those cases, consistent results across studies suggest that the treatment itself, rather than random chance or uncontrolled circumstances of the original study, is responsible for the outcomes. This direct replication provides some assurance that the initial study’s outcomes were not skewed by unreported concurrent treatments or by inadvertent data collection or analysis errors by the original researchers. Although this strengthens the evidence for the treatment’s effectiveness under specific conditions, it does not address the treatment’s potential applicability to a broader range of individuals or settings (generalizability).
A ‘replication with extension’ study, however, aims to both replicate findings of previous research and systematically extend treatment applications to participants with different characteristics, in multiple settings such as clinics, schools, and homes, and with different implementers (e.g., parents, clinicians, and teachers). In other words, replication and extension studies involving systematic changes to the initial study’s design, context, and procedures can provide evidence related to a treatment’s generalizability. Specifically, when positive outcomes are reproduced in different settings or with participants of different ages and demographics, the confidence in the treatment’s efficacy and understanding of its applicability increases. For instance, if a behavioral intervention designed to improve play skills shows positive effects not just in a controlled laboratory setting (initial study) but also on a real-world school playground (replication and extension study), that supports the conclusion that the intervention is genuinely effective and provides new information about the treatment’s potential applications. Across-study replication also helps to identify which elements of a treatment are more likely to be universally beneficial versus those that might be effective only under specific conditions.
Why Replication Matters for Autism Interventions?
Considering replication is vital when assessing the reliability of evidence supporting a treatment’s efficacy for individuals with autism, given the wide variability in experiences and responses among this population. Consider a hypothetical treatment designed for children with autism that is intended to improve social skills. Initially, the treatment is developed and tested in a specialized university-based autism therapy center with children 5 to 7 years old from a middle-class community receiving the intervention from a highly trained researcher. The initial study reports that the children demonstrated improved cooperative play skills, better turn-taking, and enriched social interactions following intervention. Across-study replication with extension comes into play when the same treatment procedures are evaluated again in subsequent studies involving different environments (e.g., schools and family homes), children with different characteristics, and is implemented not by specialized therapists or researchers but by teachers, parents, and peers who tend to be more accessible in natural environments. These replication studies help pinpoint effective treatment elements, identify parameters associated with desirable outcomes, protect against ineffective or harmful treatments, and control for biases and errors that may lead to inaccurate conclusions.
Replication helps identify the effective components of a treatment package. Through efforts to replicate and extend the initial study, researchers can distill which components of the treatment package are broadly applicable and which are more context-dependent. For example, replication studies might find that certain elements of the treatment, such as structured play sessions or specific communication prompts, consistently yield positive social skills improvements across various settings and age groups. Of course, they might also discover that other components of the treatment package, like the use of digital devices or one-on-one sessions, are particularly effective only in the original therapy center or only with younger children. In this way, replication enables a fine-grained analysis of the underlying mechanisms (active ingredients) of multicomponent treatments and helps identify why and precisely how a treatment works (or doesn’t).
Replication is a guardrail against pseudoscience. Pseudoscience refers to practices that claim to be scientific and evidence-based despite the absence of rigorous responsible scientific research. The danger of pseudoscience lies in its ability to mislead by offering seemingly simple solutions to complex problems, which can lead to false hope, wasted resources, and, in some cases, direct harm to individuals (e.g., Hemsley et al., 2018). Pseudoscientific treatments often rely on anecdotal evidence and testimonials disseminated to consumers through websites and word of mouth. Proponents of such approaches disregard the need for empirical validation and fail to adhere to the scientific method. When attempts to replicate claims of beneficial outcomes made by proponents of pseudoscientific treatments repeatedly fail, the true nature of a treatment becomes more apparent. In this way, replication not only guides us toward choosing interventions that genuinely benefit those with autism but also shields individuals from the disillusionment and potential hazards associated with unproven methods. In essence, replication ensures that treatments can stand up to rigorous scientific inquiry, offering protection against the allure of quick fixes and unsupported claims.
Replication helps control for biases and errors. Finally, an independent replication is a replication study conducted by researchers who were not involved in the original study or the development of the treatment being studied. Independent replication minimizes the potential influences of researcher biases and unintentional errors that can occur during treatment research (Tincani & Travers, 2019). Independence in replication is particularly valuable because it subjects the initial findings to scrutiny from fresh perspectives and increases confidence that the results can be reproduced without potential biases of the treatment’s original developers or others who may have increased stakes in outcomes. Independent replication is essential in building a robust, unbiased evidence base for autism treatments and is usually held in higher regard because conclusions are not just a reflection of a single team’s preferences, methodologies, potential oversights, or profit motives (Tincani & Travers, 2019).
Where to find information about replication when you’re considering treatment options?
Ask Questions. When reviewing online resources, consulting with professionals, or assessing materials from treatment providers, there’s a chance to pose -or at least ponder- specific questions. Questions like “Has this study been replicated?” “Has this treatment been shown to be effective with children like mine?” or “Has this ever been researched in a school classroom?” can help guide you to information about replication.
Read the Research. In some cases, you can find a study by searching online. But, most of the time, original peer-reviewed research is difficult or expensive to access. However, in many fields, practitioners are required to access research regularly to stay abreast with best practices. So, don’t hesitate to ask your pediatrician, speech therapist, psychologist, behavior analyst, and others for a copy of a study or access to research that could inform decisions. When a research article is technical or filled with jargon, it can help to focus on sections describing participants’ characteristics to see if they are similar to the person who would receive the treatment. The introduction section often indicates how the current study is an extension of previously published research. Further, you can check the reference section to find citations for related research. Please see McKenna (2021) for strategies to consider when conducting a literature search.
Evidence-based Registries and Professional Organizations. In many fields, replication is not merely a methodological preference, but a moral obligation made clear in professional ethics codes (e.g., American Medical Association; American Psychological Association, 2017). Many professional organizations have developed frameworks defining evidence-based practices, placing notable emphasis on the importance of replication. These organizations evaluate treatment options through systematic appraisal of peer-reviewed research to endorse those that qualify as evidence-based practices, particularly those meeting rigorous replication standards. The links below connect to those free resources.
- National Professional Development Center on Autism Spectrum Disorder: https://autismpdc.fpg.unc.edu/
- What Works Clearinghouse: https://ies.ed.gov/ncee/wwc/
- The National Standards Project: https://nationalautismcenter.org/national-standards/phase-2-2015/
References:
American Medical Association. (2016). AMA principles of medical ethics. Retrieved April 4, 2024, from https://www.ama-assn.org/delivering-care/ama-principles-medical-ethics
American Psychological Association. (2017). Ethical principles of psychologists and code of conduct (2002, amended effective June 1, 2010, and January 1, 2017). https://www.apa.org/ethics/code/
Brand, D. (2014). Treatment Integrity: Why it is important regardless of discipline. Science in Autism Treatment, 14(2), 6-7, 9-11. https://asatonline.org/research-treatment/resources/topical-articles/treatment-integrity/
Hemsley, B., Bryant, L., Schlosser, R., Shane, H., Lang, R., Paul, D., Banajee, M., & Ireland, M. (2018). Systematic review of facilitated communication 2014–2018 finds no new evidence that messages delivered using facilitated communication are authored by the person with disability. Autism and Developmental Language Impairments, 3, https://doi.org/10.1177/2396941518821570
Green, V. A. (2007). Parental experience with treatments for autism. Journal of Developmental and Physical Disabilities, 19, 91-107. https://doi.org/10.1007/s10882-007-9035-y
Green, V. A., Pituch, K. A., Itchon, J., Choi, A., O’Reilly, M., & Sigafoos, J. (2006). Internet survey of treatments used by parents of children with autism. Research in Developmental Disabilities, 27(1), 70-84. https://doi.org/10.1016/j.ridd.2004.12.002.
McKenna, K. (2021). Science Corner: Strategies to consider when conducting a comprehensive literature search. Science in Autism Treatment, 18(3).
Mruzek, D. W. (2012). Focus on science: “Verification” and the peer review process. Science in Autism Treatment, 9(3), 18-19. https://asatonline.org/for-parents/becoming-a-savvy-consumer/verification-and-the-peer-review-process/
Tincani, M. & Travers, J. (2019). Replication research, publication bias, and applied behavior analysis. Perspectives on Behavior Science, 42, 59-75. https://doi.org/10.1007/s40614-019-00191-5
Citation for this article:
Lang, R. (2024). The crucial role of replication in scientific validation and identification of evidence-based practices. Science in Autism Treatment, 21(05).
Related ASAT Articles:
- Determining the effectiveness of treatments available to persons with autism – Part One
- ASD Intervention: How do we measure effectiveness?
- What is evidence-based practice and why should we care?
- The persistence of fad interventions in the face of negative scientific evidence: Facilitated communication for autism as a case example
- “Verification” and the peer review process
- A replication and extension of the PEERS
- Explaining decisions to use science-based treatments
