Determining the Effectiveness of Treatments Available to Persons with Autism – Part Two

By Tristram Smith, PhD and Daniel W. Mruzek, PhD

How does one determine if a particular autism treatment is effective?

In Part One of this series, we reported that ASAT uses a two-step process to report on effectiveness in our research synopses, as well as inform our treatment summaries: (1) the identification and analysis of each study on a particular intervention, and (2) integrating this information into an overall appraisal and recommendation (see p. 4 of the Winter 2013 issue of Science in Autism Treatment in Newsletter Archive). Our process is based on a set of nine criteria developed by Chambless and Hollon (1998) and adopted by the Clinical Psychology Division of the American Psychological Association for the evaluation of research on psychological intervention (see Table 1 on p. 14). In Part One, of this series, we reviewed the first five criteria, using our analysis of the LEAP model of intervention as an example (2011; see the Fall 2012 issue of Science in Autism Treatment, p. 17 for an ASAT Research Review of this study). Here, we look at criteria 6 – 9 and integrate this discussion with helping families and practitioners who are making treatment decisions.

Criterion 6 – Resolution of Conflicting Results: Science rarely progresses in a “straight line”. Rather, data from “new” studies may or may not replicate the findings of previous studies. The results of one study may suggest that a particular treatment is helpful for some individuals with ASD, while the results of another study may suggest that the same treatment is of limited benefit, of benefit for only some individuals, or even altogether unhelpful. As a result, our evaluation of the scientific validation of a particular autism treatment must include consideration of these disparate findings. In our example of the LEAP model of intervention, we find that there is only one controlled outcome study (Strain & Bovey, 2011), so, although the study yielded encouraging results, we indicated that replications are needed. Positive outcomes found in one study, even a well-controlled study such as the one on LEAP, do not prove that a treatment is effective.

Criterion 7 – Limitation of Efficacy: Assuming that the findings from studies provide some evidence that a particular treatment is effective, our evaluation of these findings should consider for whom these findings are relevant (e.g., individuals with ASD in a particular age group, at a certain level of cognitive functioning, or with a specific presenting concern), as well as variables that may influence the effectiveness of treatment (e.g., hours of intervention per week, how well staff are trained). For example, in the study investigating the LEAP model of early intervention (2011), Strain and Bovey studied the impact of the LEAP model with preschool children and did not attempt to extend their findings to other age groups. Also, they carefully considered questions related to the number of hours per week that their intervention was delivered and the hours of training required by participating staff.

Criterion 8 – Generalizability: This criterion refers to the degree to which we can expect that the carefully controlled conditions described in a particular study can be maintained in “real life” settings (e.g., amount of expert supervision of practitioners, material resources, physical environment). Even a very powerful treatment effect in a study means little to families and practitioners if the necessary conditions for the treatment cannot be realized in typical intervention settings (e.g., schools, homes). In their discussion of the LEAP model, Strain and Bovey focused squarely upon the degree to which the LEAP model might be implemented in community settings and outside of the relatively tight controls of research. Indeed, this study was implemented in community-based inclusive preschool classrooms in order to assess ease of implementation and satisfaction of participating intervention staff. As mentioned above, additional studies will be necessary to support or refute their positive findings.

Criterion 9 – Treatment Feasibility: In order for a particular treatment to be effective for persons with autism, the treatment must not be overly burdensome for families and practitioners (e.g., too time-consuming, costly, or requiring an unrealistically sophisticated skill set). In their 2011 study, Strain and Bovey included a social validity scale completed by participating lead teachers that examined their impressions of the LEAP model of early intervention along several dimensions related to feasibility, including social acceptability, practicality and flexibility of the intervention. How can ASAT’s synopses of articles and treatment summaries help parents and practitioners with their decision-making? We suggest that they be viewed as “quick reference guides” for checking particular treatment options presented by a practitioner or marketer, as well as “conversation starters” with trusted practitioners (e.g., physician, psychologist, behavior analyst collaborating with family). By the way, if you notice that a key reference is missing, or you suspect that we mischaracterized a treatment or study finding, feel free to contact us at info@asatonline.org. We contend that families making treatment decisions should have the benefit of clear, concise and accurate information. To this end, ASAT’s Treatment Summaries are an important resource.

Criteria	Common Considerations
1. Overall Research Design	Benefits of intervention not due to chance Design controls for potentially confounding variables Design described with enough detail to replicate Example: randomized clinical trials
2. Sample Description	Population sampled is specified Standard procedures used to confirm diagnosis
3. Outcome Assessment	Tools measure key clinical concerns Measures have demonstrated reliability and validity Interviewers blind to group status of participants Clinical significance is assessed, not just statistical significance
4. Treatment Implementation	Intervention is manualized for others to further test Study “therapists” are trained and monitored by experts committed to ensuring competent implementation Study is overseen by knowledgeable experts committed to ensuring competent implementation
5. Data Analysis	Procedures for data analysis are planned prior to data collection; “phishing” for significant results through multiple tests does not occur Consideration is given to different group drop-out rates in the analysis of the results
6. Resolution of Conflicting Results	Studies with positive results are weighed alongside studies with results suggesting “no benefit” Meta-analyses (studies that analyze a number of independent studies at one time) are used when warranted
7. Limitation of Efficacy	When reporting positive results, researchers identify for whom the treatment is beneficial Possible “moderator variables” (factors that may systematically influence effectiveness) are considered and acknowledged
8. Generalizability	Consideration is given to the relevancy of the results of a treatment in actual clinical applications Variables that may affect the “external validity” of results (e.g., therapist training, level of supervision) are acknowledged
9. Treatment Feasibility	The degree to which a treatment is acceptable to individuals and their families (i.e., preferred over other options) is acknowledged The ease at which a treatment can be used by practitioners with integrity is considered

*Adapted from Chambless and Hollon (1998)

Click here to read Part 1

References

Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18.

Strain, P. S., & Bovey, E. H. II. (2011). Randomized controlled trial of the LEAP model of early intervention for young children with autism spectrum disorders. Topics in Early Childhood Special Education, 20(10), 1-22.

Citation for this article:

Smith, T. & Mruzek, D. W. (2013). Determining the effectiveness of treatments available to person with autism part two. Science in Autism Treatment, 10(2), 15-17.