Executive Summary
The routine collection of drug treatment outcomes to manage quality of care, improve patient satisfaction, and allocate treatment resources is currently hampered by two key difficulties: (1) problems locating clients once they leave treatment; and (2) the prohibitive cost of obtaining meaningful and reliable post-treatment data. This study investigated precise methods for an economical staff-based Routine Outcome Monitoring (ROM) system using an 18-item core measure telephone survey. As implemented at Narconon of Oklahoma, a behavioral and social skills-based residential drug rehabilitation program, the system was determined to be psychometrically adequate for aggregate reporting while providing clinically useful information.
Study Overview
The study analyzed the procedure developed and implemented by Narconon of Oklahoma ("Narconon") for monitoring the long-term outcomes achieved by the graduates from its drug and alcohol rehabilitation program. This paper describes the precise methods for an economical staff-based Routine Outcome Monitoring (ROM) system using an 18-item core measure telephone survey as used in the Narconon program and the results of the outcome monitoring process.
Goals of Routine Outcome Monitoring
The goals of routine outcome monitoring are straightforward:
- Ensure reproducible treatment effectiveness, consistency, and cost-effectiveness.
- Improve the overall quality of treatment.
- Ensure accountability of health service providers to funding sources by monitoring their outcomes and maintaining treatment quality.
Challenges in Tracking Post-Treatment Outcomes
The systematic tracking of clients after they have completed a full course of intervention, when they are operating under minimal supervision as a member of their family, workforce, and community, is one of the most convincing methods for demonstrating real-world effectiveness of behavioral health programs. Despite stated aims to gather meaningful post-discharge data, most performance-monitoring efforts are still in development stages. Limitations to data collection include:
- Difficulty tracking clients once they leave the treatment setting.
- Using treatment staff to collect follow-up data when their main function is providing chemical dependency services.
- Relying on personal interviews and other time-consuming protocols that are costly, require substantial staff training, and may result in data drift or loss due to follow-up complexity.
Narconon's Streamlined ROM Process
Narconon has developed a streamlined, telephone-based routine outcome monitoring process within the context of a continuing care facility. Utilizing a short outcomes survey, it is sufficiently simple to fit within the routines of staff and a smaller facility and has the advantage that it can be implemented by individuals who have not been trained in research methods. The procedure enables the program to obtain rapid feedback and thereby locate and work with clients who have encountered difficulties after leaving the program. Additionally, it provides useful data for monitoring the overall effectiveness of the rehabilitation program, allowing program managers to make necessary adjustments to increase effectiveness.
Study Purpose and Methodology
The purpose of this study was to evaluate the efficacy of a post-treatment Routine Outcome Monitoring (ROM) system as a tool for measuring and improving results from drug rehabilitation services. Narconon International and Psychometric Technologies Incorporated developed a scientifically grounded methodology to obtain useful monitoring data, implemented in conjunction with enrollment, case management, and client follow-up systems already in place at Narconon. Initially, a 10-item questionnaire was developed, but this was expanded during the study based on recommendations from the aftercare and case management staff. The final survey included:
- Six items directed at self-reported drug and alcohol use in the past 30 days.
- Two items directed at general drug use since leaving treatment.
- Five items directed at quality of life issues in the past 30 days.
- Several other questions useful in assessing the success of the individual in re-integrating into the community.
Study Participants and Data Collection
For purposes of developing and evaluating this instrument and the ROM methodology, this project was limited to individuals who had completed the full Narconon program (called "graduates"). Narconon staff compiled a list of individuals who had completed the program from 2004 to 2007. Aftercare specialists conducted the ROM surveys by telephone. Initially, surveys were also conducted with close relatives of the graduates to assess the reliability of the graduate data. Once a high statistical correlation was found between the data from the graduates and their closest relatives, relatives were surveyed to obtain data when graduates could not be reached directly after three attempts. Data obtained was de-identified in accordance with Federal and local confidentiality rules and provided to Psychometric Technologies Incorporated for analysis.
Results and Improvements
Data was obtained from 323 of the 419 subjects who returned to their community, leaving 22.9% with missing data. The first sampling point had an inadequate follow-up rate. By reviewing successful actions of other researchers in this field, improvements were made as follows:
- Staff were trained to use a simple enrollment form to collect multiple phone and email contact information from the client, as well as multiple collateral phone and address contact data.
- All contact information was verified and updated at discharge.
- A written checklist was implemented to organize each step of the follow-up process.
As a result of these improvements, post-treatment contact rates consistently improved to upwards of 80 percent.
Data Analysis
The data from graduates and relatives (collateral sources) was analyzed statistically and found to be highly consistent, establishing the validity of the self-report measures used in the ROM approach. Out of the total data set, 72.1% of the data used was self-reported by the graduates. The data was analyzed to determine the degree to which drug problems recurred among Narconon program graduates. The following results were obtained when graduates (or collaterals) were asked about drug use during the 30 days preceding the survey interview:
Recurrence of Drug-Related Problems
| Drug Use and Problems | Graduate (N = 238) | Relative (N = 94) | ||
|---|---|---|---|---|
| None | 1+ days | None | 1+ days | |
| 1. 30 days: any alcohol | 180 (76%) | 58 (24%) | 62 (66%) | 32 (34%) |
| 2. 30 days: alcohol to the point of intoxication | 216 (91%) | 22 (9%) | 85 (90%) | 9 (10%) |
| 3. 30 days: cocaine | 228 (96%) | 10 (4%) | 79 (85%) | 14 (15%) |
| 4. 30 days: marijuana/hashish | 224 (94%) | 14 (6%) | 89 (96%) | 4 (4%) |
| 5. 30 days: heroin | 233 (98%) | 5 (2%) | 83 (89%) | 10 (11%) |
| 6. 30 days: other illegal drugs | 230 (97%) | 8 (3%) | 80 (86%) | 13 (14%) |
| 7. Since graduating: alcohol to intoxication | 165 (69%) | 73 (31%) | 52 (56%) | 41 (44%) |
| 8. Since graduating: used other illegal drugs | 173 (73%) | 65 (27%) | 53 (57%) | 40 (43%) |
| 9. 30 days: been arrested for drug-related offenses | 228 (96%) | 19 (4%) | 85 (91%) | 8 (9%) |
| 10. 30 days: spent the night in jail | 231 (97%) | 7 (3%) | 82 (88%) | 11 (12%) |
| 11. 30 days: been stressed because of your drug use | 119 (90%) | 14 (11%) | 43 (80%) | 11 (20%) |
| 12. 30 days: reduced or given up important activities | 128 (96%) | 5 (4%) | 42 (78%) | 12 (22%) |
| 13. 30 days: experienced emotional problems | 120 (90%) | 13 (10%) | 39 (74%) | 14 (26%) |
Feasibility and Benefits of ROM
One of the additional purposes for instituting the ROM procedure was to determine its feasibility in monitoring program graduates to assist them should they encounter further difficulties with drug or alcohol use. It was found that only 12% of those completing the program reported having a need for further rehabilitation services.
This report shows the feasibility of a Routine Outcome Monitoring (ROM) system for use in drug abuse treatment facilities. Telephone-based recovery management can address several factors including:
- Stabilizing and sustaining recovery congruent with an individualized care philosophy.
- Addressing any stigma associated with returning to a facility after completing treatment.
- Reaching clients who live at a distance from the treatment facility.
- Minimizing potential staff and financial burden.
Data obtained using the ROM system seems adequate for compliance with grant reporting or accrediting agency requirements. Excepting the first follow-up point, this project consistently produced a representative sampling exceeding 80 percent and required very little staff training. Collateral verification in the early part of the project indicated minimal bias from "grateful testimonials"—where the client might not want to hurt the counselor’s feelings by reporting lack of success—possibly due to using non-treatment staff for follow-up interviews. Ready to revolutionize your data collection and ensure seamless compliance? Visit our website today at https://www.narcononafrica.org.za/ and start optimizing your processes with our ROM system!