Journals | Policy | Permission

Journal of Clinical Medicine Research

Short Communication

Volume 11, Number 11, November 2019, pages 764-768

Pragmatic Quality Assessment of Brief Health Behavior Change Interventions: Evidence for Criterion-Related Validity of the Behavior Change Counseling Index

Training healthcare providers in patient-centered counseling approaches such as motivational interviewing (MI) is increasingly popular, both to improve patient-provider communication in general and better support patients in health behavior change [1-3]. A cornerstone of training is to observe trainees using skills and then provide feedback and coaching to improve on skills where needed; however, traditional approaches that rely on expert trainers to review and rate work samples using time-intensive quality measures are not practical in the busy healthcare setting [4-6]. In the area of behavior change counseling, Lane and colleagues [7] developed a brief measure of counseling quality, the behavior change counseling index (BECCI), to assess healthcare provider skill as a pragmatic alternative to resource-intensive “gold standard” measures of MI that had been developed and used in the research context. We are not aware of any studies that examine the criterion-related validity of the BECCI, or the degree to which BECCI ratings are correlated with an alternative gold standard measure [8]; however, the BECCI has gained popularity and been used to assess outcomes in effectiveness trials as well as studies of training outcomes [9-13]. In the majority of these studies the BECCI is rated by a third-party objective rater; however, in routine training it is more practical for the quality of work samples or role-plays to be assessed by a subjective rater, such as a role-play partner or trainer [14]. Trainer-based ratings have also become more common in pragmatic clinical trials of behavior change counseling [11, 15]. In the present study, we harnessed a subset of data from a pragmatic multisite clinical trial to compare patient-centered alcohol brief intervention quality assessment ratings made by a simulated patient after a role-play using the BECCI with those made objectively by a third-party expert rater using the motivational interviewing treatment integrity (MITI) scale. We hypothesized that these two measures of quality would be correlated.

This study is a secondary analysis of data generated through a 25-site clinical trial of screening and intervention for posttraumatic stress disorder and related comorbidities (e.g., risky alcohol use) among traumatically injured inpatients: the Trauma Survivors Outcomes and Support (TSOS) Study [16]. As part of the trial, trauma center providers were trained to deliver alcohol brief interventions using principles of MI. Data for this study come from a simulated patient role-play [17] conducted to assess pre-training skills in patient-centered alcohol brief intervention delivery.

Participants were 64 providers at 25 level I trauma centers in the USA identified for recruitment through the site principal investigator (Table 1). Provider recruitment occurred between January 2016 and February 2018. Providers underwent informed consented procedures with University of Washington research staff. Consenting providers agreed to be TSOS care managers, which included delivering alcohol brief interventions, among other treatment components, and complete a 20-min telephone-based simulated patient role-play assessment of alcohol brief intervention counseling prior to a 1-day TSOS workshop training. Providers received $50.00 for the simulated patient role-play. Previous experience with alcohol brief interventions was not required to participate. This study was approved by the Western Institutional Review Board and conducted in compliance with the ethical standards of the responsible institution on human subjects as well as with the Helsinki Declaration. The trial was registered at Clinical trials.gov (NCT02655354).

Click to view

Table 1. Provider Characteristics (N = 64)

A research staff member was trained as an actor to role-play a hospitalized trauma patient who was drinking alcohol at the time of the injury event. The simulated patient was trained by the trial behavioral interventions trainer (DD), who herself was trained in alcohol screening and brief interventions using MI by an internationally known expert in these areas and member of the Motivational Interviewing Network of Trainers (CD). Simulated patient training included didactics, demonstration, role-play (from both the perspective of a patient and the simulated patient), and feedback on performance as a simulated patient. The simulated patient was taught to be appropriately responsive to MI-adherent and non-adherent behavior of providers and therefore familiar with the distinction between high and low quality patient-centered alcohol brief intervention counseling. Following this training, the simulated patient was taught to rate the BECCI with careful study of the BECCI manual.

Role-plays were conducted by telephone and audio recorded (for MITI coding). The providers were asked to role-play for 20 min, although actual lengths varied (mean (M) = 12.6, standard deviation (SD) = 6.2). The scenario was initially developed and used in a previous implementation trial of alcohol screening and brief intervention services within trauma centers (Table 2) [18].

Click to view

Table 2. Simulated Patient Role-Play Scenario Provider Instructions and Additional Details

The simulated patient actor completed the BECCI immediately following the role-play, without reviewing the audio. We calculated the overall BECCI score comprised of 11 items rated on a Likert-type scale with 0 = not at all, 1 = minimally, 2 = to some extent, 3 = a good deal, and 4 = a great extent. Examples of items are “Practitioner invites the patient to talk about behavior change” and “Practitioner uses empathic listening statements when patient talks about the topic.” The 11 items were summed and divided by 11 to get an overall BECCI score ranging from 0 to 4; higher scores reflect greater provider use of behavior change counseling skills (α = 0.94).

Each simulated patient role-play was coded using the MITI system version 3.1.1 [19] independently by an expert MITI coder and MI trainer (CD) who had no contact or involvement with the study participants. Sessions were rated for 4 of the 5 global scores on a scale from 1 (low) to 5 (high) evaluating the entire session, including evocation, collaboration, empathy, autonomy/support and frequencies of individual provider behaviors (i.e. open-ended questions, simple and complex reflections, MI-adherent and non-adherent behaviors). For the study purposes, we examined empathy as well as the five suggested MITI summary scores, which included MI spirit (an average of three global Likert scales: evocation, collaboration, and autonomy, scores range from 1 to 5) and ratio scores derived from counts of counseling behaviors: 1) The percentage of total reflections that are complex versus simple (i.e. percent complex reflections); 2) The percentage of total questions that are open questions versus closed (i.e. percent open questions); 3) The ratio of total reflections to total questions (i.e. reflection-to-question ratio); 4) The percent of MI adherent responses out of the sum of MI adherent plus non-adherent responses (i.e. percent MI adherent).

We examined descriptive statistics for the overall BECCI score and MITI empathy and summary scores, depicting the median and interquartile range for skewed MITI variables. We examined correlation coefficients between the overall BECCI score and MITI empathy and the five summary scores, using Spearman’s rho for skewed MITI variables.

Providers were primarily female (n = 56, 87.5%) and all had, at minimum, a bachelor’s degree (majority master’s, n = 39, 61.0%). Social work (n = 21, 32.8%) and nursing (n = 18, 28.1%) were the most common professional roles. Of those reporting race/ethnicity (12 had missing data), the majority reported White/Caucasian (n = 41, 64.1%). Providers varied in pre-training skills per the BECCI and MITI (Table 3). All observed correlations between the overall BECCI and six MITI scores were statistically significant and all but one, the correlation between the overall BECCI and the MITI percent complex reflections were large (≥ 0.50; Table 4).

Click to view

Table 3. Descriptive Statistics for Overall BECCI and MITI Empathy and Summary Scores

Click to view

Table 4. Correlations Between Overall BECCI and MITI Scores Based on Simulated Patient Performance

Pragmatic quality assessment is critical to implementing routine healthcare provider training with feedback and coaching in patient-centered behavior change intervention skills. We found evidence of criterion-related validity for the BECCI, specifically, that simulated patient actor ratings of the quality of healthcare provider alcohol brief interventions using the BECCI correlated strongly with objective ratings made by an expert using the MITI. Our findings suggest that having simulated patient actors rate healthcare providers on the BECCI immediately following a role-play is a pragmatic approach to the quality assessment, which can then be used to provide feedback and inform skills coaching efforts. Correlations were particularly strong for the MITI scores corresponding to the spirit of patient-centered counseling. The observation of stronger correlations between the BECCI and MITI scores of MI spirit and empathy than for MITI scores based on frequencies of specific provider behaviors may be due to differences in item-level measurement, specifically, that all of the BECCI items and MI spirit and empathy are based on global impressions.

This investigation capitalized on the availability of a unique dataset generated from a larger study. Although we had a sample of providers representing diverse professional roles, findings may not generalize to types of healthcare providers not captured in our data that engage patients in conversations about behavior change. Further, the simulated patient role-plays focused on the topic of behavior change related to alcohol use and it is unknown whether different correlations would be observed if ratings were based on other behavior change topics. Findings may not generalize to BECCI ratings generated by other types of raters, such as a peer or fellow trainee. Continued research on pragmatic quality assessment and types of raters is warranted, however, given that the time and financial resources needed to use simulated patients may remain a barrier for some routine training environments [20]. Research indicates peer role-plays in patient-centered behavior change interventions can be as effective as those conducted with simulated patients [21], and it may improve the utility of peer role-play to have peers rate each other on the BECCI immediately after role-plays and share the scores as feedback. Research also shows that trainees can improve in their ability to score complex measures like the MITI as a consequence of training in patient-centered counseling [22], which bodes well for the use of the BECCI by trainees as well.

Acknowledgments

The investigators acknowledge the clinical trial principal investigator, Douglas Zatzick, MD, and the 25 trauma centers and staff that participated in the research project.

Financial Disclosure

This study was supported in part by the National Institutes of Health (NIH) Common Fund and National Institute of Mental Health (NIMH), through cooperative agreements (U54 AT007748, 1UH2MH106338-01/4UH3MH106338-02) from the Office of Strategic Coordination within the Office of the NIH Director, and in part by NIMH UH3MH106338-03S1. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.

Conflict of Interest

The authors have no conflict of interest to disclose.

Informed Consent

Providers underwent informed consented procedures with University of Washington research staff.

Author Contributions

DD designed the study, oversaw data collection, reviewed the literature, analyzed and interpreted the data, and drafted the manuscript. LP engaged in data collection, data interpretation, and contributed to drafts of the manuscript. AE engaged in data collection and provided critical reviews of the manuscript. KD contributed to the literature review, data interpretation, and provided critical reviews of the manuscript. DF contributed to data collection and provided critical reviews of the manuscript. CD contributed to the design of the study, literature review, data interpretation, and provided critical reviews of the manuscript.

1. Barwick MA, Bennett LM, Johnson SN, McGowan J, Moore JE. Training health and mental health professionals in motivational interviewing: A systematic review. Child Youth Serv Rev. 2012;34:1786-1795.
   doi
2. Elwyn G, Dehlendorf C, Epstein RM, Marrin K, White J, Frosch DL. Shared decision making and motivational interviewing: achieving patient-centered care across the spectrum of health care problems. Ann Fam Med. 2014;12(3):270-275.
   doi pubmed
3. Soderlund LL, Madson MB, Rubak S, Nilsen P. A systematic review of motivational interviewing training for general health care practitioners. Patient Educ Couns. 2011;84(1):16-26.
   doi pubmed
4. Schwalbe CS, Oh HY, Zweben A. Sustaining motivational interviewing: a meta-analysis of training studies. Addiction. 2014;109(8):1287-1294.
   doi pubmed
5. Beidas RS, Cross W, Dorsey S. Show Me, Don't tell me: behavioral rehearsal as a training and analogue fidelity tool. Cogn Behav Pract. 2014;21(1):1-11.
   doi pubmed
6. Dunn C, Darnell D. Commentary on Schwalbe et al. (2014): Two wishes for the future of motivational interviewing-workshops with fewer learning targets and sustainable coaching. Addiction. 2014;109(8):1295-1296.
   doi pubmed
7. Lane C, Huws-Thomas M, Hood K, Rollnick S, Edwards K, Robling M. Measuring adaptations of motivational interviewing: the development and validation of the behavior change counseling index (BECCI). Patient Educ Couns. 2005;56(2):166-173.
   doi pubmed
8. Salkind NJ. Encyclopedia of research design. Thousand Oaks, CA: SAGE Publications, Inc.; 2010.
9. Noordman J, van Lee I, Nielen M, Vlek H, van Weijden T, van Dulmen S. Do trained practice nurses apply motivational interviewing techniques in primary care consultations? J Clin Med Res. 2012;4(6):393-401.
   doi pubmed
10. Pfister-Minogue KA, Salveson C. Training and experience of public health nurses in using behavior change counseling. Public Health Nurs. 2010;27(6):544-551.
    doi pubmed
11. Shrier LA, Burke PJ, Kells M, Scherer EA, Sarda V, Jonestrask C, et al. Pilot randomized trial of MOMENT, a motivational counseling-plus-ecological momentary intervention to reduce marijuana use in youth. mHealth. 2018;4(29):1-12.
    doi pubmed
12. Spollen JJ, Thrush CR, Mui DV, Woods MB, Tariq SG, Hicks E. A randomized controlled trial of behavior change counseling education for medical students. Med Teach. 2010;32(4):e170-177.
    doi pubmed
13. Thomas T, Passfield L, Coulton S, Crone D. Effectiveness of a tailored training programme in behaviour change counselling for community pharmacists: A pilot study. Patient Educ Couns. 2016;99(1):132-138.
    doi pubmed
14. Cross WF, West JC, Pisani AR, Crean HF, Nielsen JL, Kay AH, Caine ED. A randomized controlled trial of suicide prevention training for primary care providers: a study protocol. BMC Med Educ. 2019;19(1):58.
    doi pubmed
15. Magill N, Graves H, de Zoysa N, Winkley K, Amiel S, Shuttlewood E, Landau S, et al. Assessing treatment fidelity and contamination in a cluster randomised controlled trial of motivational interviewing and cognitive behavioural therapy skills in type 2 diabetes. BMC Fam Pract. 2018;19(1):60.
    doi pubmed
16. Zatzick DF, Russo J, Darnell D, Chambers DA, Palinkas L, Van Eaton E, Wang J, et al. An effectiveness-implementation hybrid trial study protocol targeting posttraumatic stress disorder and comorbidity. Implement Sci. 2016;11:58.
    doi pubmed
17. Kaplonyi J, Bowles KA, Nestel D, Kiegaldie D, Maloney S, Haines T, Williams C. Understanding the impact of simulated patients on health care learners' communication skills: a systematic review. Med Educ. 2017;51(12):1209-1219.
    doi pubmed
18. Darnell D, Dunn C, Atkins D, Ingraham L, Zatzick D. A randomized evaluation of motivational interviewing training for mandated implementation of alcohol screening and brief intervention in trauma centers. J Subst Abuse Treat. 2016;60:36-44.
    doi pubmed
19. Moyers T, Martin T, Manuel J, Miller W, Ernst D. Revised global scales: Motivational interviewing treatment integrity. (MITI 3.1.1). 2010.
20. Lane C, Rollnick S. The use of simulated patients and role-play in communication skills training: a review of the literature to August 2005. Patient Educ Couns. 2007;67(1-2):13-20.
    doi pubmed
21. Lane C, Hood K, Rollnick S. Teaching motivational interviewing: using role play is as effective as using simulated patients. Med Educ. 2008;42(6):637-644.
    doi pubmed
22. Hartzler B, Baer JS, Dunn C, Rosengren DB, Wells E. What is seen through the looking glass: The impact of training on practitioner self-rating of motivational interviewing skills. Behav Cogn Psychother. 2007;35(4):431-445.
    doi

This article is distributed under the terms of the Creative Commons Attribution Non-Commercial 4.0 International License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Journal of Clinical Medicine Research is published by Elmer Press Inc.