Determining the Effectiveness of Treatments Available to Persons with Autism – Part One

Written by Tristram Smith, PhD and Daniel W. Mruzek, PhD

The goal of helping persons with autism and their families access the “state of the science” in autism treatment is at the heart of the Association for Science and Autism Treatment’s (ASAT) mission. To this end, each of our Treatment Summaries includes a brief assessment regarding its scientific validation (see Learn More About Specific Treatments).

A reasonable question is, “how does ASAT determine if a particular autism treatment has scientific validation?” We cannot simply “tally” the total number of published studies supporting and refuting a particular treatment, because there exists a tremendous range in scientific rigor across published studies. For example, some studies demonstrate tight experimental control, and, thus, afford greater confidence in the results. Other studies have significant methodological weaknesses, rendering their findings tentative, suspect, or altogether worthy of dismissal. Also complicating matters, treatment studies that find “no benefit” are less likely to be published (i.e., the “back of the file cabinet effect”); therefore, we are left with “skewed” samples of published treatment research to review. Even well-designed studies often yield results that other investigators cannot replicate. For that reason, our efforts at assessing the scientific validity of autism treatments must hinge on consideration of not only the number of studies and their outcomes but also the quality of that research.

ASAT’s evaluation of research entails a two-step process: (1) the identification and analysis of each study on a particular intervention, and (2) integrating this information into an overall appraisal and recommendation. Both steps are based on a set of nine criteria developed by Chambless and Hollon (1998), and adopted by the Clinical Psychology Division of the American Psychological Association for the evaluation of research on psychological intervention (see Table 1). Though this set of criteria is not the only one that could be used, it does appear well suited to the purpose of evaluating the “state of the science” for various autism treatments.

In Table 1, the first five criteria correspond to step 1 of our evaluation process (i.e., review of each study of an intervention), and the final four criteria correspond to step 2 (developing an overall appraisal and recommendation). Below, we illustrate application of these criteria for step 1, using the Learning Experiences and Alternative Program for Preschoolers and Their Parents (LEAP) model of early intensive intervention as an example. In the next issue of “Focus on Science” (Spring 2013), we will look more closely at step 2.

The LEAP model was recently the subject of a randomized, controlled study by Strain and Bovey (2011; see the Fall 2012 issue of Science in Autism Treatment, p. 17 for an ASAT Research Review of this study). Using the first five criteria in Table 1, we find the following:

Criterion 1 – Overall Research Design: This was a randomized control trial with 294 children in 56 classrooms, with 28 classrooms randomly assigned to full LEAP participation with follow-along training, and 28 classrooms assigned to a comparison condition (i.e., access to LEAP materials but no follow-along training). The large number of participants and random assignment minimizes risk that a difference between the two groups is the result of chance. A reasonably detailed description of methodology during the two-year study indicates systematic control of potentially confounding variables (e.g., introduction of another “extra” therapy in one of the groups, assurance that the intervention described was actually delivered) and provides opportunity for replication by other researchers.

Criterion 2 – Sample Description: The authors clearly describe their criteria for classroom participation as well as the process of recruiting participation of teaching staff and children. As the authors acknowledge, however, they did not administer measures that require direct observation to confirm that the children qualified for an autism spectrum disorder (ASD) diagnosis. Instead, they relied on diagnosis by outside clinicians and screening questionnaires given through the study. Thus, there is a risk that the authors included some children who did not have ASD.

Criterion 3 – Outcome Assessment: Intervention lasted for two years, and children were assessed at three points: before the start of intervention, after about one year of intervention, and at the conclusion of the second year of intervention. Outcome assessments were standardized and valid measures of autism symptoms, cognition, and adaptive behavior. All measures were administered by appropriately trained investigators. It does not appear that blind raters were employed, introducing potential threat to internal validity (e.g., expectancy bias on the part of the raters).

Criterion 4 – Treatment Implementation: Treatment components and techniques are carefully specified in a detailed manual, allowing other researchers the opportunity to conduct replication studies. Investigators used well-established measures of the fidelity of implementation of the LEAP intervention by teachers in the experimental group, confirming that the treatment was delivered as intended. Teachers were trained and supervised by experts in the LEAP model who were committed to ensuring proficient implementation.

Criterion 5 – Data Analysis: Statistical procedures were appropriate for testing hypotheses. Drop-out rates within and between experimental and control groups were accounted for by researchers in their analyses.

Overall, although not perfect, this study has many methodological strengths that increase confidence in the validity of the findings. Thus, we believe the positive outcome for LEAP participants relative to controls makes a very important contribution to the literature. Nevertheless, in our ASAT description of LEAP (found at: Leap Model), we caution that “additional research…is needed in the LEAP model as a comprehensive treatment program. Important next steps for research are replications by independent investigators and comparisons against established early intensive behavioral treatment models.” We will discuss reasons for this caution in the next installment of “Focus on Science.” More specifically, we will look at criteria six through nine, and integrate this discussion regarding evaluating the scientific validation of autism interventions into specific ways families and practitioners can use our reviews as a resource in making treatment decisions.

Table 1. Evaluating Autism Treatment Research: Key Items for Consideration*

Criteria	Common Considerations
1. Overall Research Design	Benefits of intervention not due to chance Design controls for potentially confounding variables Design described with enough detail to replicate Example: randomized clinical trials
2. Sample Description	Population sampled is specified Standard procedures used to confirm diagnosis
3. Outcome Assessment	Tools measure key clinical concerns Measures have demonstrated reliability and validity Interviewers blind to group status of participants Clinical significance is assessed, not just statistical significance
4. Treatment Implementation	Intervention is manualized for others to further test Study “therapists” are trained and monitored by experts committed to ensuring competent implementation Study is overseen by knowledgeable experts committed to ensuring competent implementation
5. Data Analysis	Procedures for data analysis are planned prior to data collection; “phishing” for significant results through multiple tests does not occur Consideration is given to different group drop-out rates in the analysis of the results
6. Resolution of Conflicting Results	Studies with positive results are weighed alongside studies with results suggesting “no benefit” Meta-analyses (studies that analyze a number of independent studies at one time) are used when warranted
7. Limitation of Efficacy	When reporting positive results, researchers identify for whom the treatment is beneficial Possible “moderator variables” (factors that may systematically influence effectiveness) are considered and acknowledged
8. Generalizability	Consideration is given to the relevancy of the results of a treatment in actual clinical applications Variables that may affect the “external validity” of results (e.g., therapist training, level of supervision) are acknowledged
9. Treatment Feasibility	The degree to which a treatment is acceptable to individuals and their families (i.e., preferred over other options) is acknowledged The ease at which a treatment can be used by practitioners with integrity is considered

*Adapted from Chambless and Hollon (1998)

Click here to continue reading – Part 2

References

Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18.

Strain, P. S., & Bovey, E. H. II. (2011). Randomized controlled trial of the LEAP model of early intervention for young children with autism spectrum disorders. Topics in Early Childhood Special Education, 20(10), 1-22.

Citation for this article:

Smith, T. & Mruzek, D. W. (2013). Determining the effectiveness of treatments available to person with autism part one. Science in Autism Treatment, 10(1), 4-5.