How to design AI-powered services — lessons from the emergency room
This article describes how human-centered design approaches can help bridge the gap between what AI can do and what is valuable. Using human-centered design for AI-powered triage technology in an Emergency Room context, we designed for nurses’ skill enhancement, faster and better triage, and a better patient experience.
Defining success criteria for AI models is far from easy. It requires advanced knowledge of both computer science and data science and the ability to translate events and actions in the world into networks of weights, transitions, goal states, and setting precision and accuracy criteria for the desired thresholds. Still, this is far from enough to consider how an AI-powered service provides value to users, customers, organizations, and other stakeholders when the service is implemented. Data-driven and AI-powered functionality create new opportunities and problems — and new skills and approaches are required from a design perspective.
What is valuable?
Take a look at the figure above. It might seem straightforward to think that providing valuable services is just a matter of pushing state-of-the-art machine learning at a problem. But think carefully about the mysterious gray arrow pointing from enabling technology to value-creation. How exactly does that happen? Often, teams fall into a positivist technology trap and define a naive and over-simplified hypothesis about how value-creation happens. One famous and comic example is the automatic soccer camera that followed the referee’s head instead of the ball in the match between Inverness Caledonian Thistle and Ayr United at Caledonian Stadium in Inverness in November 2020. Of course, the problem was that the image detection software only focused on the most obvious task at hand (follow a round light shape on a green background) without taking neither the real world nor what a camera operator’s job is about into account. Even an absolute beginner camera operator who knows nothing about soccer would realize that the referee’s head is not to be followed, even if it shows similar characteristics to a soccer ball. A successful workflow description does not only consist of “what to do” — it also consists of “what not to do.” The value lies in unforeseen real-world applications of both prediction and judgment.
The other problem with the mysterious gray arrow in the figure above is that it presupposes that the application of technology (like Machine Learning, for example) has some innate, almost magical, ability to “create value.” The arrow is highly misleading. Let’s unpack this and update the figure!
Value is created in use
To detect how value is harnessed or created, we need to understand how technology is used. This is the cornerstone of human-centered design and relates to how perception, cognition, and behavior influence value creation. Value occurs then the design takes skills, workflow, and user experience into account.
The various flavors of design — whether it be Business Design, Service Design, UX Design, or Interaction Design — all hone in on the experiences, workflows, skills, and uses in a given real-world situation. This approach is necessary to understand for any value-creation effort and service.
We can skip the gray arrow altogether, because as it turns out, it not only misses the crucial step of usage, it is also going in the wrong direction.
A better flow is to start with what’s valuable — in the unfiltered and authentic use context. Understanding behavior and user experience will arrive at better and sharper conclusions regarding technology requirements. (For example, the camera should not confuse a human head with a soccer ball.) With a firmer grip on authentic use and experiences, you can put better requirements on the technology. The next step becomes interesting for AI-powered services: make sure you design for continuous data collection to improve the AI model.
An updated process starts with insights-driven research to uncover what is valuable in a specific situation. Then we move on to understand behavior and experience and how enabling technology can support them.
For Machine Learing technology in particular, we need to design for continuous data collection through user interaction in order to re-train and enhance the prediction model. This can then lead to increase value in use. This five step process (visualized in the figure above) is in line with how the Virtuous Cycle of AI (as coined by Andrew Ng) works. With data, a Machine Learning service becomes better; a better service attracts more users; more users generate more data that can make the service even better, and so on. But the critical aspect of this is to collect the correct data in a way that does not detract value from the user experience at use time. It also presupposes that the data is used to monitor positive outcomes and impact over time.
In a well–designed service, the data that loops back connect both actions and impact to prediction and judgment, allowing both the prediction model and the human to improve (“super-charging the user” and augmenting her skills). Let’s see how this can be accomplished using design-oriented processes for emergency room triage.
The AI-powered emergency room
A core task of caregivers in the Emergency Room (ER) care is to perform triage. Triage is the process of sorting patients according to the urgency of illness or injury to ascertain which order to treat them in. Triaging involves taking a set of vitality parameters (body temperature, pulse, blood pressure, oxygen saturation, and respiratory rate). As these measurements are standardized, our client developed and constructed an AI-powered “triage camera” with sensors that clinically could measure the vitality parameters non-invasively and in one single measurement by positioning the patient in front of the camera for a few seconds. The original thought behind this technology was to provide the value of faster triage for overworked and stressed ER nurses by placing cameras in the ER waiting rooms in self-service booths. The AI service would then pre-sort patients based on their ratings and unload the vitality measurement operations from the nurses’ workflow. This line of thinking resembles the “mysterious” arrow going from technology to value-creation. Although it is far from clear exactly how that will happen, and whether this is the only or highest value we could get out of the context.
As we shall see, our design approach challenged this hypothesis in several ways.
Our approach: the design sprint
We decided to use the design sprint method and dedicate a focused work week for the first iteration of this design challenge. A team of designers and professional nurses teamed up with the triage camera developers and set up a design sprint project room with large whiteboards and large screens for remote interviews. We cleared our calendars and acquired snacks, coffee, drawing pads, post-its, and prototyping materials for an entire week.
Our first task was to visit our city’s ER clinic. Together with the ER staff, we mapped out a complete workflow starting when a patient enters the ER and reports symptoms, assessment of their vitality parameters, and receiving a full triage. We also got the opportunity to interview a few ER nurses about their work.
Back in the studio in the afternoon, we talked about our findings with professional healthcare advisors and our client’s development team. It was clear that the staff’s presence in the waiting room was critical and that a lot of interesting things were going on in the dynamics between caregivers and patients throughout the process. Professional nurses and physicians rely on what’s known as “the clinical gaze” — a term related to tacit knowledge. Even though the concept of “the clinical gaze” has changed dramatically as technology and sensors have been introduced in the healthcare context, ER staff reports on its value in practice. It is described as an essential, subtle knowledge honed through years of experience but hard to articulate and define. When our first sprint day ended, we realized that an AI-powered camera service for an ER clinic is a significant challenge to implement, not from a technical standpoint but from a nurse–patient interaction and care quality perspective. Reducing triage time is still important, but other qualities need to be considered if the service is to provide real value.
Tuesday came, and we were trying to understand the problem further and map the challenges on a journey timeline from a patient’s point of view, along with the ER nurse perspective captured the day before. To test something useful on Friday, we needed to make the prototype actively “look for” issues regarding the quality of care and the tacit caregiver skills.
We decided to build a fake ER clinic with a waiting room, camera prototypes, information displays, and an examination room. We recruited nurses from the hospital’s ER clinic and extras that would role-play patients for Friday’s test. The camera prototypes were literal black (shoe!) boxes with an image of a camera lens pasted on the side. The digital user interface for the nurses was a few linked screens that displayed triage vitality parameters on a mounted iPad, along with a mockup of the regular patient journal system already in use at the clinic.
We decided to set up a “rich” test environment where different scenarios could unfold to explore variations of patient–nurse interactions. We placed two cameras in the waiting room as self-service booths with instructions for arriving patients on how to use them. We also put a camera in the examination room so that it could be used (again) by the attending nurse to perform complementing measurements.
Friday was testing and debriefing day. The learnings and discussions in the afternoon were rich and had numerous implications for the development of the AI-powered camera service. For example:
- Self-service saves time but misses critical aspects of clinical judgment. This judgment requires human–human contact between nurse and patient.
- Patients (even though they were “fake” patients in this sprint) reported that the self-service camera gave them a sense of confidence. However, it could also add new worries to an already stressful situation if the parameters were “not understandable” or “too extreme.”
- Patients worried that they would not meet a professional fast enough if the camera recorded “too normal” values. “What if the camera says I’m not ill? Will I not receive treatment then?” was one patient comment.
- The nurse fetching the next patient from the waiting room is constantly updating his or her awareness of the situation. A quick scan of the waiting room can tell the nurse important details about patients’ status.
- The manual operation of taking, e.g., blood pressure, can have a calming effect on the patient, instead of just putting the patient in front of a self-service camera booth.
- Nurses develop their skillset and “clinical gaze” by reading vitality signs stepwise as parameters are being measured in real-time by the camera. This allows nurses to expand their skillset and “clinical gaze.” This doesn’t seem to happen when all parameters are delivered to a screen at the same time. There is a crucial design opportunity here for the service. Optimizing only for increased speed may hamper the quality of care and the nurses’ skill development potential.
- On the other hand: getting all parameters from a waiting room AI assessment and then complementing one or two of the five measurements “by hand” boosts the nurses’ “overall awareness” of the patient status and augments their ability to triage the patient correctly.
These findings point to an updated hypothesis: the AI-powered camera service has the potential to speed up triage and enhance the quality of care through human–human contact, as well as to boost skill augmentation for nurses. The service should be designed to maximize these benefits and allow both nurses and AI to learn from experience. This means that we need to think about how this feedback loop will happen in practice, and connect it to both AI prediction and human judgment.
Making sure that the impact of the actions taken is caught and relayed back as feedback to both the human user and the AI model is crucial. By carefully thinking about this feedback loop, both AI–powered prediction and human–powered judgment can be enhanced and provide better impact and value.
Summary — what we learned
The original service hypothesis (or “concept idea” for the AI-powered triage camera) looked like this:
In other words: the triage camera would take the vitality parameters and thereby speed up the triage rate, allowing the ER to push more patients through the system faster.
After one design sprint week, our updated view looked like this (new qualities in green):
By carefully listening in on the nurses’ experiences, synthesized with the engineers’ and designers’ perspectives, it became apparent that several more valuable effects could be achieved. Not only would the modified camera workflow save time, but it would also help nurses provide better and more accurate triage in a combination of manual and AI-powered measurements. Furthermore, nurses were not disconnected from the ER waiting room and patients, which allows them to develop their professional judgment and “clinical gaze” and not rob them of the much-needed calming patient contact. By collecting feedback data from patient encounters downstream, the AI model could be improved. Nurses and the AI-powered camera is considered to be a service-providing team, instead of viewing the AI as an isolated technology that would automate away the nurses from parts of the ER workflow. The teaming approach is critical to achieving long-term positive effects for both patients and nurses.
Often, implicit success criteria (i.e., essential but not defined) are as critical as explicit success criteria. Other times, the AI model is considered disconnected from human operators and contexts. As this case study has shown, using design and research to model task and workflows, testing prototypes in authentic settings, and taking human judgment and augmentation to be a central design goal, we can drastically increase our likelihood to discover implicit success criteria (e.g. the professional “clinical gaze”) as well as design a teaming approach to AI and human co-work.
One more thing…
There is another important strategy aspect beyond individual service design looming in the background of this discussion: the importance of designing learning organizations. There is no doubt that there is immediate value for a company or organization to create strategies for learning and developing co-workers. That means investing in skill development and creativity rather than focusing on short-term productivity by automating existing workflows. The latter approach counteracts the human desire and ability to learn and develop, whereas the former builds strategic value and assets to a company.
Humans thrive in contexts where they develop and grow, which is directly translatable to increased value for the company.
AI-powered services can be used for both ends: automating away people and making them deteriorate and become passive, or enhancing their skills and abilities to provide sustainable value for themselves and their organization. Using a human-centered design approach, you have a higher likelihood of creating functional learning organizations.
