Collaboration and the Mental Health Treatment Gap
The University of Washington’s departments of social work, psychiatry and behavioral health, computer science, and the center for game science collaborated on this r34 pilot. Our team on this project included user experience researchers and designers, psychologists, computer scientists, and developers.
Each year millions of Americans with mental illness struggle to find care. Nearly half of the 60 million adults and children living with mental health conditions in the United States go without any treatment. Globally this mental health treatment gap is even greater. More than 70 percent of people who need mental health services lack access to care. While many barriers to treatment exist, a fundamental one, particularly in rural areas, is the shortage of well-trained mental health professionals. Increasing the number of clinicians trained to deliver high-quality care is essential to closing this gap.
The Need for Efficient Clinical Training
The ability to train large numbers of clinicians efficiently and economically is critical. Our goal was to address this issue by creating a tool, an intelligent tutoring system. This system would supplement in-person learning and improve competency in essential skills while being cost-effective and scalable. Intelligent tutoring systems, or ITS, are computerized programs that serve as classroom adjuncts or replacements. They support learning by giving personalized instructions without requiring the presence of a teacher. Ours guides users through simulated cases, decision dilemmas, and other cognitive tasks by providing immediate and customized feedback. We wanted to design a tool that university social workers, psychology students, or in-training medical professionals could use. The tool will allow users to practice core therapeutic concepts such as empathy and redirection at their own pace wherever they are.
The Challenge of Simulation-Based Learning
Typically teachers and students practice these skills through simulations. However this requires the on-site presence of trained professionals. It also incurs the cost of their time to role-play, evaluate, and provide feedback on what they observe.
For our discover phase we talked to both students and experts in training. We focused on those teaching evidence-based psychosocial interventions such as cognitive behavioral therapy to learn about their needs and priorities. 15 senior social work BA students at Heritage University in eastern Washington participated in two focus groups. These helped us understand their study habits and generate their preferences and expectations for apps and technology. We asked students to write down on post-its the main attributes they looked for in an app. We also asked about the qualities that exist in apps that they find most useful or pleasurable to use.
The Design Process: Focus Groups and Research
Using a target diagram, we then asked them to plot their data on the bullseye. They ranked items in order of importance: primary, secondary, and tertiary priorities. The center represented what they felt was the most important. In our expert interviews our goal was to identify common challenges and errors made while teaching evidence-based psychosocial interventions. We wanted to know what areas they found most challenging for their students to master. Also explored what information they thought would help them gauge their students mastery of the subject matter. We spoke to nine experts who talked about core concepts such as the difference between empathy and sympathy, active listening skills, and reflection. They emphasized how therapists need to learn how to adapt to obstacles. These included a patient who is a big talker or belligerent or simply won’t do their homework the treatment requires.
Expert Insights on Core Therapeutic Challenges
The areas they found the most challenging for their students to master leaned more towards the interpersonal skills. These were not about a particular technique but rather required practice for it to become second nature. These discussions coupled with Bloom’s revised taxonomy and the educational competencies the Council on Social Work Education established inform the content of the ITS. They also shaped the seven topics we cover.
Since we created this as a tool for both users, each module begins with a brief lesson. The lesson covers what qualities an empathic response demonstrates, for this example. It also outlines what elements would constitute a poor demonstration of the skill.
Feedback Loops and Design Challenges
The first questions are basic multiple choice reiterating the fundamental rules that the student just learned. When the student gets several correct answers in a row, the ITS progresses to harder questions. The students receive feedback after two tries to answer correctly. Hovering over answers that they didn’t answer right gives a more detailed explanation. This shows why someone should or shouldn’t have chosen it, using different language to explain the core concept behind it. Developing this review feature turned out to be one of our biggest design challenges. In some cases a user gets a question wrong because they selected the wrong answer. In other cases they get it wrong because they failed to choose the right answer. What is the clearest way to convey what is wrong and why?
The Debate Between Game and Quiz Formats
Initially one of the leads in this project had strong feelings about the tool we were developing. This professor in computer science runs the University of Washington’s center for game science. He felt the tool should seem more like a game than a quiz. Because of this we initially focused on correct answers and offering explanations. We shied away from language about getting questions wrong and other things that might seem too harsh such as using an X for incorrect answers. Another problem we found with this design was that the students were not taking the time to read the explanations about their incorrect answers.
Refining the Student User Interface
We experimented with adjusting the placement and color of this feedback so it’d be more prominent. We also convened a group of undergraduate psychology students at the University of Washington. These students had participated in some user testing with our initial prototype. They gave us feedback on the UI and the overall look and feel of the app. They also helped us co-design these more challenging features. The focus group helped us to understand what conventions the students understood and preferred. We had initially chosen softer colors to discourage the notion that someone was quizzing users. What came to pass was that students wanted clear feedback about right and wrong. Despite showing them many color combinations they gravitated towards the mental models of green and red.
Landing on the Final UI Design
This was the design we landed on to address both of those problems. The red indicates an incorrect answer and the green review button shows an answer that the student missed. Hovering over any review button triggers a pop-up with further explanation. After finding our best solutions for the challenges we encountered, we pilot tested a beta version. We worked with six bachelor level social work students enrolled in a class on telephone based cognitive behavioral therapy. This is a low-intensity intervention for mild depression. An expert reviewed the student’s competencies in delivering this therapy through clinical skills role plays for fidelity. Four of the six students submitted role plays for reviews and all four passed. With the success of the pilot we then developed additional patient profiles to expand the cases the students encountered. We also created three additional question types.
Pilot Testing and Advanced Question Modules
The format of these question types differs from the earlier easier multiple choice ones. They all contain a transcript of a dialogue between a patient and a therapist. They ask the user to evaluate the therapist’s responses. In this example the user needs to highlight portions of the therapist’s response where they either successfully or unsuccessfully demonstrate empathy. They then choose the reason why from a drop down menu. As part of my earlier conversation with experts we discussed what sort of information would be useful for them. This would help them gauge their students mastery of a subject. We created this dashboard for professors to check in on the progress of their students as they make their way through the modules.
Using Transcripts to Evaluate Competency
Our hope was that professors would use this information to inform what subjects they address during their in-person class time. The bottom panel shows the amount of time each student has spent on the current module and how far they’ve progressed. In this example they’ve all completed everything. The performance of the class generates the middle panel. The number inside the donut represents the number of students who have struggled with a particular skill. Hovering over the issue gives the professor the details of what students are struggling to grasp. The professor also has access to example errors and practice problems for their own information or to share with the class. This shows how each concept was tested in concept.
Comparing ITS Outcomes to Traditional Learning
We ran a pilot in the fall of 2020 with two 23 undergraduate students already enrolled in psychology and social work classes. We worked with five professors across three universities. Randomly assigned students to a group that used the ITS on their own time completing one module a week. The control group learned the same topics the more traditional way by doing assigned reading at the same pace. At the end of the trial all of the students participated in role-playing a therapist during a one-on-one session. A psychologist trained to evaluate competencies assessed their performance. We found that both groups performed comparatively. What was noticeable was the enthusiasm that students who used the ITS expressed about the experience.
Analyzing the Value of the Review Feature
The review feature continued to be a problem. Students said they used it when they didn’t. When they did use it, they didn’t think it was very valuable and only read the first bolded sentence. But the data did show that it was valuable. When students used the feature after answering a question wrong, they tended to get the next question right. We interviewed all five instructors. Overall the professors felt the ITS app had a lot of potential in identifying where students need more support. They thought it would be particularly good for classes with role playing or a practicum. These settings offer direct opportunities to immediately practice these skills. In terms of our plan of evaluating how the dashboard acted as a useful tool to impact the classroom activity, one snag we encountered was the timing. The pilot occurred during the COVID pandemic when the universities, at least the ones we partnered with, were suspending in-person classes.
Challenges During the Pandemic
Because of a shift to asynchronous course attendance, additional class time was required for announcements, check-ins, and processing current events. This would not normally be needed in a face-to-face environment. The teachers didn’t really feel like they had an opportunity to use it thoroughly. The professors gave us great feedback. They asked us to highlight the students strengths in the dashboard. Currently we only have a section on most common issues. They felt the dashboard shouldn’t only highlight concepts students struggle with. One explained the system is teaching a strengths-based approach but not showing strengths. They like the idea of being able to go over the problems during class. However they felt that we should move access to example errors and practice problems to a different page.
Enhancing Dashboard Utility and Future Goals
Since the dashboard contains students performance data they didn’t want to share that screen for the whole class to see. The professors referenced the active time section of the dashboard the most. They requested a baseline to be included. This would indicate how much time a student really should spend on a module to really understand how well they’re doing. The use of the review feature seems to tie to performance and therefore learning. One of our next focuses is how to get students to both click on this button and then read the explanation. This will involve iterating on both the content and the UI and doing a bunch of a b tests. While there is still room for improvement, overall we have been very pleased with how the ITS has turned out. We’re also pleased with the response of those who’ve used it. ITS has the potential to streamline and scale training and provide an engaging way for students to practice their skills.