Browsing by Author "Burns, Catherine"

Now showing 1 - 20 of 21

Behavior Change Techniques Present in Wearable Activity Trackers: A Critical Analysis
(JMIR mHealth and uHealth, 2016-04-27) Mercer, Kathryn; Burns, Catherine; Giangregorio, Lora M.; Li, Melissa; Grindrod, Kelly
Background: Wearable activity trackers are promising as interventions that offer guidance and support for increasing physical activity and health-focused tracking. Most adults do not meet their recommended daily activity guidelines, and wearable fitness trackers are increasingly cited as having great potential to improve the physical activity levels of adults. Objective: The objective of this study was to use the Coventry, Aberdeen, and London-Refined (CALO-RE) taxonomy to examine if the design of wearable activity trackers incorporates behavior change techniques (BCTs). A secondary objective was to critically analyze whether the BCTs present relate to known drivers of behavior change, such as self-efficacy, with the intention of extending applicability to older adults in addition to the overall population. Methods: Wearing each device for a period of 1 week, two independent raters used CALO-RE taxonomy to code the BCTs of the seven wearable activity trackers available in Canada as of March 2014. These included Fitbit Flex, Misfit Shine, Withings Pulse, Jawbone UP24, Spark Activity Tracker by SparkPeople, Nike+ FuelBand SE, and Polar Loop. We calculated interrater reliability using Cohen's kappa. Results: The average number of BCTs identified was 16.3/40. Withings Pulse had the highest number of BCTs and Misfit Shine had the lowest. Most techniques centered around self-monitoring and self-regulation, all of which have been associated with improved physical activity in older adults. Techniques related to planning and providing instructions were scarce. Conclusions: Overall, wearable activity trackers contain several BCTs that have been shown to increase physical activity in older adults. Although more research and development must be done to fully understand the potential of wearables as health interventions, the current wearable trackers offer significant potential with regard to BCTs relevant to uptake by all populations, including older adults.
Beyond the Dataset: Understanding Sociotechnical Aspects of the Knowledge Discovery Process Among Modern Data Professionals
(University of Waterloo, 2017-05-01) Ho, Anson; Burns, Catherine; Chilana, Parmit
Data professionals are among the most sought-out professionals in today’s industry. Although the skillsets and training can vary among these professionals, there is some consensus that a combination of technical and analytical skills is necessary. In fact, a growing number of dedicated undergraduate, graduate, and certificate programs are now offering such core skills to train modern data professionals. Despite the rapid growth of the data profession, we have few insights into what it is like to be a data professional on-the-job beyond having specific technical and analytical skills. We used the Knowledge Discovery Process (KDP) as a framework to understand the sociotechnical and collaborative challenges that data professionals face. We carried out 20 semi-structured interviews with data professionals across seven different domains. Our results indicate that KDP in practice is highly social, collaborative, and dependent on domain knowledge. To address the sociotechnical gap, the need for a translator within the KDP has emerged. The main contribution of this thesis is in providing empirical insights into the work of data professionals, highlighting the sociotechnical challenges that they face on the job. Also, we propose a new analytic approach to combine thematic analysis and cognitive work analysis (CWA) on the same dataset. Implications of this research will improve the productivity of data professionals and will have implications for designing future tools and training materials for the next generation of data professionals.
Building Bridges between Pharmacists and Physicians: An Exploratory Investigation via Field, Usability Studies & Control Task Analysis To Enhance Pharmacy Management System Requirements
(University of Waterloo, 2017-04-25) Kerestecioglu, Damla; Burns, Catherine
Miscommunication between healthcare professionals can impact patient care and is costly. The information systems used in pharmacies and hospitals can contribute to this miscommunication. The present research is an initial attempt to investigate the reasons leading to lack of communication between pharmacists and physicians. Twenty-five pharmacists were interviewed to understand how pharmacists use pharmacy management systems (PMS), as well as to explore why communication issues may occur between prescribers and pharmacists. The field study data was transcribed and interview results were organized into three categories. Next, a human factors analysis was completed to understand pharmacists’ needs and challenges with the current PMSs and finally Control Task Analysis was performed to discover weaknesses in the information systems. After discovering the usability issues with PMSs and reasons for miscommunication, mock ups were designed for a decision aid tool called Communication Summary. The intent of the Communication Summary is to enhance pharmacists’ workflow and their communication with prescribers. The mock ups were evaluated in a simulated pharmacy management task by senior pharmacy students. The results suggested that more information on pharmacist-prescriber communication may help in pharmacy management.
A Cognitive Work Analysis Approach to Explainable Artificial Intelligence in Non-Expert Financial Decision-Making
(University of Waterloo, 2022-06-07) Dikmen, Murat; Burns, Catherine
Artificial Intelligence (AI) is being increasingly used to assist complex decision-making such as financial investing. As most AI systems rely on black-box machine learning models, understanding how to support human decision-makers and gaining users' trust becomes important. Explainable Artificial Intelligence (XAI) has been proposed to address these issues by making the decision-making process of AI systems understandable to users. However, existing XAI approaches fail to take into account users' domain experience, and fail to support users with limited domain expertise. This work aims to fill this gap. We presented an approach to integrate domain expertise into XAI, and showed that this approach can have a number of benefits to users of XAI systems such as improved task performance and better assessment of XAI. The main contributions of this work include identifying the benefits of adding domain knowledge to XAI, demonstrating the usefulness of Cognitive Work Analysis (CWA) in XAI, and developing recommendations for future design of AI systems. First, through a Work Domain Analysis (WDA) approach, we identified opportunities to improve the existing XAI approaches by augmenting the explanations with domain knowledge and conducted an online study with 100 participants on users' perceptions of AI in a credit approval context. Results showed some benefits in improving user perceptions and highlighted the importance of contextual factors. Next, we introduced a testbed for exploring user behavior and task performance in a financial decision-making task. We designed decision-support aids based on domain knowledge and explored their effectiveness in an experimental study with 60 participants. In the study, participants engaged with an AI assistant and made investing decisions. Depending on the condition, participants had access to domain knowledge presented on a separate display, domain knowledge embedded in the AI assistant, or no access to domain knowledge. The results showed that participants who had access to domain knowledge relied less on AI when it was incorrect, and obtained better task performance. The effect of domain knowledge on perceptions of AI was limited. Next, we analyzed the user interviews that were part of the previous study. We identified users' mental models of AI and multiple ways they integrated the AI into their decision-making process. The analysis also revealed the complexity of designing for non-expert users, and we developed recommendations for future research and design. Finally, we conducted a Control Task Analysis and Strategies Analysis to synthesize the qualitative and quantitative findings and developed decision ladders and information flow maps. The analyses provided insights into the influence of AI on the decision-making process, challenges associated with non-expert users, and opportunities to improve AI user interface design.
Design and Assessment of External Displays on Autonomous Vehicles for Pedestrian Safety
(University of Waterloo, 2019-09-05) Qin, Zehao; Burns, Catherine
The most vulnerable group in road agents is pedestrians. In the Netherlands, from 2005 to 2009, the average lethality rate for vulnerable road users was 14 per 100 serious road injuries. Prior to autonomous vehicles (AV), conventional vehicles had a human driver that could communicate with pedestrians through signals such as eye gaze, head movements, and hand and arm gestures. With the introduction of AVs, pedestrians can no longer rely on such communication signals. In the future, when all of the control and responsibilities of the human driver gradually transfer to the autonomous driving program, the vehicle’s intent communication to pedestrians must evolve as well. The aim of this proposed research was to investigate the efficacy of different external human-machine-interface in communicating autonomous vehicle intent to pedestrians in crossing situations where negotiation between the AV and the pedestrian is required (i.e. jaywalking). With SAE level 4 high automation enabled vehicles, what impact do external human-machine interfaces have on pedestrians’ crossing behaviour? what impact do external human-machine interfaces have on pedestrians’ general perception of AVs? Three novel design concepts were created to fill the gap of the lack of visual experimentation with displaying the key mental model factors - external speedometer display of the vehicle, speed change indicator (i.e. decelerating/ accelerating), and gap estimation count down timer. The experiment was a within-subject design with 29 levels. The stimulus was structured into a 14 (design cases) x 2 (coloured vs. non-coloured) factorial design. A combination of iconography, text, anthropomorphic features and colour were compared and measured in perceived safety, urgency, usefulness, understandability, emotion comfort, as well as the influence on crossing decisions. A 100-person online study was conducted to understand the impact of external visual displays with high automation (SAE level 4) vehicles on pedestrians’ crossing behaviours. The novel concepts open a new discussion for the perception of warning designs where the new visual concepts (i.e. explicitly displaying and varying the symbolism of speed) had strong performance across all measures.
Dynamic Alert Design Based on Driver’s Cognitive State for Take-over Request in Automated Vehicles
(University of Waterloo, 2024-07-03) Umpaipant, Wachirawit; Samuel, Siby; Burns, Catherine
This thesis investigates the effectiveness of dynamic alert systems tailored to drivers' cognitive states in automated driving environments, focusing on enhancing takeover readiness during critical transitions. Utilizing a large-scale immersive driving simulation, the study evaluated drivers' response times and physiological measures when reacting to various alert intensities and the presence of a secondary typing task. The experiment revealed that dynamic alerts significantly improved response times and takeover performance, especially in high-distraction scenarios. Drivers responded more effectively when alerts were adjusted to their cognitive load, with strong alerts resulting in the fastest reaction times under distracted conditions. On average, dynamic alerts reduced response times by approximately 1.75 seconds compared to static alerts. Additionally, higher lateral accelerations were observed under strong alerts, indicating more decisive maneuvering. Self-rated attention-capturing scores were notably higher with dynamic alerts, particularly under strong alert conditions and in the presence of secondary tasks. The ANOVA results showed significant improvements in attention capturing and overall alert effectiveness when dynamic alerts were employed, demonstrating the robust design’s ability to capture attention and enhance driver responsiveness. The study confirmed that adaptive alert designs, which adjust based on the driver's cognitive state, can markedly enhance overall driving experience and safety. Participants reported higher levels of confidence with dynamic alerts, especially in scenarios involving secondary tasks. Despite the strong alerts, annoyance levels remained low, indicating that dynamic alerts are effective without causing undue stress. These results underscore the potential of using adaptive systems to improve safety and efficiency in automated driving, advocating for a more nuanced approach to system alerts that considers the variable cognitive states of drivers. Future research should validate these findings with on-road studies, explore a broader range of alert modalities, and refine physiological monitoring techniques to further enhance adaptive alert systems.
Ecological Interface Design in Neuro-Critical Care
(University of Waterloo, 2023-05-02) Schaef, Laura Kathleen; Burns, Catherine
Neuro-critical care is a data-intensive environment that requires physicians to integrate information across multiple screens, sources, and software. Despite the advances in neuromonitoring techniques, interfaces that allow for viewing and analyzing of historic data are not common. However, historical data is critical to identify patterns important for patient care. Instead, physicians view the trends of a patient’s neurophysiological variables by continuously watching the bedside monitor or they rely on checking the paper (or digital) charts for a patient where variables have been recorded periodically (usually once an hour). In neuro-critical care, physicians need to understand the historic and current state as well as predict the future state of intracranial pressure (ICP). ICP is the most monitored brain-specific physiologic variable in the Intensive Care Unit (ICU) and is considered a biomarker for secondary brain injury. As a result, ICP would benefit greatly from showing key patterns important to patient state and care. The ICU is a stressful, dynamic, and time-sensitive environment where the performance of physicians and their ability to correctly diagnose and manage patient treatment has a significant impact on patient outcomes. Physicians rely on the bedside physiologic monitor to detect changes in physiologic variables. The monitor must provide the information required to understand the patient’s condition so physicians can determine the optimal treatment plan. With the high cognitive demands and complex sociotechnical environment of the ICU, an opportunity exists for improved neuro-critical care monitoring to support physicians’ decision-making. Ecological Interface Design (EID) is an approach to interface design that has proven effective for complex, sociotechnical, real-time, and dynamic systems. Research suggests that an EID approach combined with user-centered design has a positive impact on performance, especially in unfamiliar scenarios. The objective of this research is to explore an EID design approach combined with user-centered design to enhance the bedside physiologic monitor through the addition of visualizations that help support physicians' understanding of complex relationships and concepts in neuro-critical care. The hope is that providing more-advanced visualizations on the bedside physiologic monitor will lead to improved situation awareness, decreased mental workload, and expertise development acceleration of novice clinicians in the neuro-ICU. The work presented in this thesis builds on the Cognitive Work Analysis (CWA) and observations in the ICU already completed by Uereten et al (2020). The design of the visualizations for use on the bedside physiologic monitor was highly iterative and involved the inputs from the CWA and observations as well as ongoing feedback and focus areas provided by Dr. Victoria McCredie, our clinical collaborator and critical care physician at Toronto Western Hospital. The visualizations were evaluated and validated in semi-structured interviews with trainees (fellows) and experts (staff physicians) in neuro-critical care. The semi-structured interviews with trainees were used as a preliminary usability assessment of the visualizations and the interviews with staff physicians were used to iterate and refine the designs. The results from both sets of interviews were used to create a final design prototype that is currently being tested in a usability study with trainee physicians (January-March 2023).
Exploring Automated Trading: Modeling a Financial System with a Variable Degree of Automation, Display Design, and Evaluation
(University of Waterloo, 2017-11-24) Li, Yeti; Burns, Catherine
Automation has become more pervasive and started to replace human operation in all information-processing stages (e.g., machine-learning based systems). Typically, automated systems have a variable degree of automation (DOA) that challenges modeling and design. Cognitive Work Analysis (CWA) is a modeling approach to support analysts in coping with the complexity of socio-technical domains and has shown success in providing implications for developing ecological displays that are effective in unanticipated situations. CWA allows space for modeling automated systems but has not been well developed to describe variable DOAs. This dissertation explores this problem by focusing on the case of automated trading which is underexplored in both human factors research and finance research. This dissertation starts with developing a modeling approach then focuses on design and evaluation. Modeling. A DOA layering approach on the decision ladder was developed to serve as the theoretical foundation of this research. Two cases of automated trading – basket trading and trend following trading, each using a different DOA – were presented to facilitate the development of this approach. With this approach, the two most commonly used modeling tools in the CWA, the abstraction hierarchy, and the decision ladder each adopted an additional layer for representing human-administrated functions and automated functions. Also, the four information-processing stages in which the automation could take place were marked on the decision ladder to allow for a more detailed level mapping, which is unique in the CWA research. The DOA layering approach was demonstrated to extend the use case of CWA to include automated systems with a variable DOA and have important implications for ecological display design as well as automation design. Design and evaluation. The experimental approach presented later in this dissertation further explored automation and display design implications of the DOA layering approach using AUTRASS (AUtomated TRAding System Simulation), a simulation developed as part of this research program. Two experimental studies on trend following trading are reported in which the design concepts were evaluated. In Experiment 1, inspired by the CWA models, automation was designed as two configurations to represent distinct DOA situations. The moderate DOA configuration simulated a trading situation in which the participants performed a flexible trading task. The high DOA configuration represented a higher DOA situation where a trading algorithm that was unfamiliar to the participants traded in a similar market condition, and the participants monitored the automation and performed a fault detection task. Two types of displays were designed. Conventional displays were typical in information content and form to current trading displays and should support the basic use of the automation. Ecological displays were implemented from the CWA models to support monitoring for unanticipated situations for each DOA condition. Four scenario types were developed by combing the two DOA configurations with the two display types. Experiment 1 involving 24 participants was conducted to thoroughly examine the effectiveness of ecological displays with different DOAs. Based on the literature, the ecological displays were hypothesized to improve task performance and situation awareness and to trigger riskier actions without imposing higher workload. Results of Experiment 1showed that the ecological displays did not provide better support on either trading performance (moderate DOA) or fault detection performance (high DOA). Empirically, a trade-off of situation awareness and workload between the two DOA configurations seemed to exist without the influence of the ecological displays. Interestingly, the results of this study suggested a different pattern of risk preference compared to that in the literature. Specifically, the ecological displays imposed riskier financial trading decisions. The results of Experiment 1 provided implications for identifying system and contextual factors that could influence risk preference and demonstrated sufficient space for improving automation design. Experiment 2 is a follow-up to the first experimental study with separately recruited 24 participants. The high DOA configuration previously used in Experiment 1 was improved with its traits inherited and its flaws in automation design eliminated (i.e., the improved-high DOA configuration was expected to be better supported by the ecological displays). An adaptive configuration was introduced to simulate adaptive automation in the automated trading setting. The conventional displays and the ecological displays continued to be used without any modifications, and similar hypotheses were examined with the two new DOA configurations. Results of Experiment 2 showed that with the improved automation design in the high DOA configuration, the ecological display significantly improved fault detection performance. A consistent pattern of risk preference was found in this study as in Experiment 1. A comparison of the two experimental studies showed new opportunities to derive automation and display design from the DOA layering approach that can support fault detection performance in automated trading, and future research is warranted to explore the influence of ecological displays on risk preference. The development of AUTRASS also makes a unique contribution. The design of the different DOA configurations demonstrated the applicability of the DOA layering approach to guide automation design. Overall, the following conclusions were reached by conducting this dissertation research: CWA has been effective in characterizing the complexity in automated trading that is associated with the variable DOA, and it can further support the design of automation and the ecological displays. Ecological displays may foster risky operation with moderate DOA as well as performance improvement with high DOA. The applicability of the proposed approaches spanning modeling, design, and evaluation should go beyond the limit of automated trading to the brave new world of artificially intelligent automation.
Exploring the Requisites and Design Requirements for Adding ‘Reason for Use’ Information to Prescription Labels
(University of Waterloo, 2018-10-01) Hussein, Thana; Burns, Catherine
Individuals with multiple medical conditions and polypharmacy are at higher risk of inappropriate prescribing and consequent adverse drug events. Moreover, individuals may not adequately understand the therapeutic intention of their medications as this information is conveyed to patients inconsistently from their healthcare providers. One way to address this problem is to inform the patient of the reason for using a prescribed medication. The present research is an initial attempt to design a prescription label that incorporates Reason for Use (RFU) by first understanding how patients are currently receiving RFU information, their feelings of being provided with that information in a future health care system, and secondly, exploring their design preferences for this newly designed prescription label. Twenty patients (10 female; 10 male) throughout the Kitchener-Waterloo region were interviewed using a semi-structured questionnaire and 15 (9 female; 6 male) of these individuals participated in a design workshop that aimed to understand where the RFU information should be placed on a prescription label, the amount of detail, language, and overall layout of the label. Participant responses were analyzed thoroughly to discover important and/or frequent themes. Results from the study revealed that all 20 patients are in favor of having RFU information shared with their pharmacist and overall, would feel more informed if they were provided with RFU information from their healthcare providers. Results from the design workshop revealed that patients preferred RFU information printed in one to three words (80%) with its placement underneath the dosage instructions (33.3%) and/or next to the drug name (46.7%) because it felt the most ‘logical’ (73.3%). The participants showed a high preference for adding RFU to the prescription label for sake of being better-informed and more capable of participating fully in health decisions. Results from this study suggest a need for including RFU into the Ontario medication prescribing practices and adopting recently released United States Pharmacopeia (USP) patient-centered prescription label standards, which also include adding RFU to the labels.
A High-Fidelity VR Simulation Study: Do External Warnings Really Improve Pedestrian Safe Crossing Behavior?
(University of Waterloo, 2021-01-27) He, Fan; Burns, Catherine; Li, Yeti
To better communicate with pedestrians, adding external displays to autonomous vehicles (AVs) has been proposed as a potential communication method to encourage safe crossing behavior by pedestrians. Whereas, most researchers have conducted intercept interviews, lab studies, or simulation studies to explore the efficacy of these displays, these approaches only studied crossing intention but did not explore crossing behavior. We developed a high-fidelity virtual reality scenario where participants could demonstrate actual crossing behavior within an adequately replicated real-world street. We simulated a local street with scalability of the real world in a VR environment, conducted an experiment in an empty space large enough for participants to move across the road in the VR environment. A mixed-method approach assessed attitudinal and behavioral interactions with potential warning patterns. The results showed that the warning patterns contributed significantly to pedestrians’ perceptual vigilance, as in past studies, but safer crossing behavior was not observed. This suggests that crossing intention measures may not be an adequate substitute for behavioral measures of crossing.
Interface Redesign of an Electronic Medical Record Review System Using User-Centered Design
(University of Waterloo, 2019-08-29) Parmar, Harjot Singh; Burns, Catherine
The healthcare industry in recent years has seen a rise in the adoption of Electronic Medical Records (EMRs). These EMRs have replaced the traditional paper-based records at hospitals, clinics, and nursing facilities. This transition has brought with it, numerous advantages of digitization such as improved patient care, timely reminders for checkups, and better health data tracking over time. But the EMR adoption has also come with its own set of challenges. The EMR systems are maintained by the medical coders/nurses at the hospitals. The coders are expected to gather information from different sources such as patient history logs, test results from different labs, etc. followed by entry into the EMR system. Due to the unstructured nature of the task, data entry in EMRs is susceptible to errors which lead to the poor data quality of patient records. Diagnostic decisions taken by the medical practitioners based on erroneous data can adversely affect the patient and at times, even prove to be fatal. To help address this issue of poor data quality of System X, an EMR, employs a unique data review process which allows reviewers (domain experts) to check patient records and communicate back the data entry errors to the coders for required changes to ensure high data quality. In this research, the user-centered design methodology was applied to improve the review process, with the aim of facilitating easier and quicker workflow. The usability issues faced by the reviewers were identified through heuristic evaluations, video walkthroughs, and user interviews methods. To address the issues identified, a new interface design was developed by employing low fidelity and high-fidelity prototyping techniques. Involvement of the reviewers throughout the research ensured that the design proposed was continually assessed and improved qualitatively until they were satisfied. Lastly, the Keystroke Level Model (KLM) was used to quantitatively assess the performance improvement gained from the new design. The final interface design was able to reduce the task-execution time of the patient record review process by 28.51%. This resulted in saving a significant amount of the reviewer’s time, thereby reducing their workload while improving data quality.
Investigating Driver Experience and Augmented Reality Head-Up Displays in Autonomous Vehicles
(University of Waterloo, 2017-05-30) Dikmen, Murat; Burns, Catherine
Autonomous driving is on the horizon. Partially automated vehicles recently started to emerge in the market, and companies are dedicated to bringing more automated driving capabilities to the vehicles in the near future. Over the past twenty years, human factors research has increased our understanding of driver behavior and human-vehicle interaction, as well as human-automation interaction considerably. However, as the technological developments accelerate, there is an urgent need to conduct research to understand the challenges of driving a semi-automated vehicle, the role of cognitive and social factors and driver characteristics, and how interactive technology can be used to increase driving safety in this context. This thesis was an attempt to address some of these challenges. In this work, we present two studies on human factors of automated driving. In the first study, we present the results of a survey conducted with Tesla drivers who have been using partially automated driving features of Tesla cars. Our results revealed that current users of this technology are early adopters. Automation failures were common, but drivers were comfortable in dealing with these situations. Additionally, Tesla drivers have high levels of trust in the automated driving capability of their vehicles, and their trust increases as they experience these features more. The results also revealed that drivers don’t use owner manuals, and seek out information about their cars by using online sources. The majority of Tesla drivers check multiple information sources when their car software receives an update. Overall these findings show that driver needs are changing as the vehicles become smarter and connected. In the second study, we focused on a future technology, augmented reality head-up displays, and explored how this technology can fit into the smart, connected and autonomous vehicle context. Specifically, we conducted an experiment looking into how these displays can be used to monitor the status of automation in automated driving. Participants watched driving videos enhanced with augmented reality cues. Results showed that drivers adjust their trust in the automated vehicle better when information about the vehicle’s sensing capabilities are presented using augmented reality cues, and they have positive attitudes towards these systems. However, there were no major safety-related benefits associated with using these displays. Overall, this work provides several contributions to the knowledge about human-automation interaction in automated driving.
Investigating Technology Implementation in a Canadian Community Hospital
(University of Waterloo, 2025-01-27) Allana, Sana; Burns, Catherine
The integration of technology into healthcare has witnessed significant advancements. However, the widespread adoption of such technologies may not be uniformly positive. While highest levels of adoption are typically found in densely populated urban areas, community healthcare facilities face challenges due to insufficient resources, like infrastructure, funding, and specialized staff, exacerbated by their remote locations. This is cause for concern as community hospitals account for 90% of all hospitals in Canada. This reveals a major opportunity to improve technology adoption and implementation at community hospitals, to aid their existing challenges, increase equity in healthcare, and improve generalizability of healthcare technologies. This research aims to uncover the perceptions, expectations, cultural nuances, and barriers to technology adoption at a community-level hospital in Ontario, Canada. The study began with a contextual inquiry approach, incorporating semi-structured interviews and surveys. Data was collected from nine clinical and managerial staff members whose workflows were impacted by three pilot technology projects. The interviews aimed to explore staff expectations and experiences with how these pilot projects impacted their workflows, patient care, and the overall technology implementation process. The survey included demographic questions and items based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model, designed to predict factors influencing technology acceptance. The pilot technologies included a discharge planning tool, a portable X-ray scanner, and a digital pathology tool. A thematic analysis of the qualitative data was conducted, followed by affinity mapping to identify overarching themes. The Functional Resonance Analysis Method (FRAM) was also used to understand and model the impact of integrating the pilot technologies into preexisting, variable workflows. Finally, survey results were analyzed using frequency distributions to identify trends and triangulate findings. Overall, most staff reported a high level of technology use in both their work and daily lives. They also acknowledged that technology breakdowns at the workplace were inevitable, often resulting in time-consuming, manual workarounds. As well, for all pilot projects, staff felt overburdened by the additional workload required to manage the pilots alongside their regular duties. However, despite these challenges, all staff expressed an appreciation for innovation and a strong willingness to try new tools to improve their work. The discharge planning and X-ray scanner tools did not integrate well into existing workflows or provide additional value. Both tools performed inconsistently and failed to meet expectations for streamlining processes, leading to reluctance and distrust among staff. Additionally, change management planning was insufficient for both tools, with staff experiencing abrupt workflow changes, limited training, and a lack of clarity on project timelines or statuses. As a result, neither tool was requested for purchase following pilot testing. Conversely, staff decided to purchase the digital pathology tool, despite the disruptions to existing workflows, as the perceived benefits to both staff and patient care outweighed these challenges. Staff were excited about the tool’s potential and engaged in close collaboration with the manufacturer and project team. Furthermore, change management was carefully planned, with a phased implementation approach. The pilot was also driven by strong advocacy from a pathologist, which ensured alignment with clinical needs. Based on these findings, several recommendations were uncovered to improve the technology implementation process. First, the challenges with change management highlight the need for better resource allocation. This includes providing sufficient time for introducing new tools, clearly explaining the reasons for their selection, offering personalized training that covers tool usage, troubleshooting, and its impact on existing processes, and ensuring staff have the necessary bandwidth to manage change without disrupting daily operations. Second, communication channels should be improved. Startup companies should collaborate closely with the hospital during the development and testing phases to better understand staff needs and workflows, while also providing tailored support throughout the implementation process. Additionally, communication with hospital leadership must be strengthened to secure strong support, allocate resources effectively, and incorporate feedback on the challenges staff encounter, fostering a more collaborative environment that is better equipped to drive innovation. Finally, it is crucial to define and share specific success metrics for pilot projects. These metrics will help staff assess the technology's impact, make informed decisions about its use, evaluate the implementation process, identify lessons learned, and pinpoint areas for improvement, all of which can refine future technology adoption strategies. Overall, technology implementation and adoption are influenced by a variety of factors, which are further compounded by the high workload, staffing shortages, and unpredictable environments commonly found in community hospitals. By addressing these recommendations, health organizations can enhance the adoption and effectiveness of new technologies, ultimately improving staff workflows and patient care.
Models of Cognitive Work Analysis in Neurocritical Care and Perspectives on Expertise– An Ecological Interface Design Approach
(University of Waterloo, 2023-12-12) Uereten, Ece; Burns, Catherine
Neurocritical care is a complex and data-rich environment in which timely decisions are of utmost importance. Small changes in the neurophysiologic states may imply neurological deterioration that can be fatal if not treated timely and appropriately. A multidisciplinary team of clinicians takes care of the patients with varying expertise levels. Expert physicians in this domain often try to act proactively and prevent further injury (such as secondary brain injury often visible in traumatic brain injury patients) and manage concurrently, assessing patient data represented through various sources. Novice intensive care physicians act rather reactively to specific events and have a sequential approach to management. The scattered data sources across the unit and the lack of centralized views on the patient data make it even more challenging for physicians, in general, to understand the happenings and support the patient. Collecting and identifying all data through various locations and systems can cause a high cognitive workload and is not optimally laid out through meaningful representations on different device interfaces. Further, such interfaces (e.g., the bedside physiologic monitor) lack the incorporation of neurocritical-relevant concept visualizations that would help novices better analyze and understand the context and significant relationships among variables. This is especially important because the complexity of neurocritical care is high, immediate, and proactive treatment is desired. It usually takes much time for novices to learn the specifics of assessing neurological deterioration in a critical context and being aware of the situation to consider the trajectory and required medical support for the patient. Inappropriate actions and long decision-making time can result in unwanted health deteriorations or even fatalities. Bedside physiologic monitors play a significant role in assessing the vital signs of patients; however, in neurocritical care, there are additional variables, such as Intracranial Pressure (ICP), that are significant biomarkers for neurological conditions and need to be assessed in context with other variables. In this work, we have focused on this specific variable as it is a key variable showing the health state of the brain and is interconnected with various other physiologic variables, which requires a detailed examination to assess neurocritical care patients’ conditions. For novice neurocritical care physicians, it is a new variable and challenging context to learn how to deal with this variable as it was often displayed numerically and as a waveform on monitors. Assessing this variable in context needs to be trained to develop an understanding of (dys)functional autoregulation in the brain. Still, interfaces don’t have much flexibility in representing this variable other than in a numeric or waveform pattern. Monitors thus can benefit from new representations of ICP within its context. This work proposes an ecological interface design (EID) approach to represent ICP together with other variables on vital signs monitor to be used in neurocritical care. This work not only shows a new design for such monitor interfaces but also integrates the additional perspective on supporting novice physicians to develop more in the direction of experts to support them in the best way possible. Differences in the context of neurocritical care among physicians are explored in this work, as they are the main decision-makers in the unit. Once the understanding of differences between novices and experts can be disseminated, the steps that are required to develop expertise can be further examined and supported through various ways. Overall, the research objective of this work is to explore how expertise development can be supported through the ecological interface design approach of bedside physiologic monitors in the context of neurocritical care. Methods and Modeling: As a starting point, observations in a neurocritical care unit are summarized and modeled through the Cognitive Work Analysis (CWA) framework using the Work Domain Analysis (WDA) and Control Task Analysis (ConTa). This framework is selected to identify various elements of the neurocritical care context as it provides in-depth insights into complex sociotechnical areas and their constraints. Further, it lays a basis for the design of an ecological interface that enables the end-user to become an adaptive problem solver in any situation. The CWA models thus provide the basis for identifying key relationships and aspects that would be important to display on such interfaces. In the following step, interviews were conducted with critical care experts and novices to further dive into their perspectives on expertise development, challenges, and potential ways of addressing these and supporting novices. These insights were relevant for adding and discussing aspects represented on the CWA models and exploring potential expertise-relevant measures that could be used for assessing the development of expertise through interface design. Identifying both CWA and interview findings, static visualizations were developed and discussed with both novice and expert critical care physicians. Their feedback was incorporated into the new designs of the visualizations and integrated into a prototype (i.e., the ecological interface). This interface was compared to a standard interface used in the neurocritical care unit within a usability study. Two neurocritical care scenarios were presented to the participants in the control and experimental groups while being asked to think out loud about assessing and treating such patient cases based on the interfaces and data shown. Results: The CWA models showed different dimensions of the work domain and how the physical variables are interconnected to provide and support patient recovery. The decision ladders mapped on a common neurocritical care scenario showed differences in expert and novice physicians’ approaches to assessing, evaluating, and acting on certain signs related to the health states of a patient. While novices cognitively go through each step and might miss out on case-relevant information or may come up with misinterpretations even, expert physicians take shortcuts as they have built certain mental models already based on their vast knowledge and experience with similar cases. Through the interviews and discussions with physicians, we further noticed that novice critical care physicians usually have a numerical and threshold approach. At the same time, experts expect them to develop an approach that incorporates understanding the significance of the waveform, trend, and individualize patient care depending on various patient characteristics. To further develop their mental model, they must step back from the “one size fits all” mindset and explore the optimal thresholds for each patient individually. To progress, they further need to develop autonomy and communication skills with various stakeholders, deepen their knowledge and familiarity with tools, perform adequately, and reflect on their actions. The interviews also revealed various aspects of expertise often tracked by the expert physicians training the novices. It was noted that experts often have a great amount of experience, knowledge, skills, training, work where there is no evidence, have excellent performance and specific personal traits. The inputs from both the CWA models and the discussions with the physicians provided initial design ideas that could be further developed to support aspects of the novices' expertise development process in the interface design context. Concepts that are relevant for neurocritical care monitoring and associated challenges have thus been discussed. Both expert and novice critical care physicians provided feedback on the initial visualizations, which helped to iterate the visualizations further and were then incorporated into a prototype interface (i.e., the ecological interface). A standard interface was also developed as a prototype to compare how participants in both groups could be compared. The intention was to show one potential ecological interface for neurocritical care that would specifically support the novice physicians to develop more characteristics of experts identified in the proposed studies. Novices using the ecological interface (experimental group) showed fewer errors when presented with two different neurocritical care scenarios, compared to the control group using the standard interface. At the same time, the experimental group’s reasoning was more focused and included more investigation of the cause for ICP elevations and patterns compared to the group using a standard interface. Reflections on the ecological interface further showed that participants started thinking critically about their strategies and helped them state the reasons and limitations of their actions demonstrated during the scenario presentations. The confidence in handing over the patient to the next team has been rated on a numeric scale showing that the group using the standard interface rated slightly higher on average than the experimental group. Also, the average subjective performance ratings were lower for the ecological interface than the standard interface. It is noteworthy though, that the difference between both groups on confidence and performance ratings wasn’t too far away from another. However, the usability of the ecological interface was perceived as useful, easy to use, and captured key neurocritical care concepts relevant to further develop expertise in the field. The ecological interface was perceived as an improvement to current interfaces. Recommendations on future interface design are shared and can help future research to further improve interface design. Conclusion: This research work consisted of multiple phases uniting ethnographic observations, discussions with clinicians (especially novice and expert physicians in neurocritical care), the development stage of neurocritical care-relevant concepts that are lacking in current bedside physiologic monitor interface designs, an iterative design process of potential visualizations to be displayed on such interfaces, and the usability evaluation of the visualizations in context (with novice neurocritical care physicians). Although the findings in the usability evaluation haven’t shown exact confirmation of all expertise development hypotheses we were expecting (e.g., the experimental group should have more confidence), we could still see relevant tendencies of novices using the ecological interface showing better understanding and reasoning patterns of the data shown to them in context. Different designs of visualizations with different variables can be further evaluated in the future, incorporating more and various perspectives of interface users in neurocritical care. Different scenarios can be tested while also other expertise-relevant measures can be considered. More recommendations for future research are outlined in the next chapters. Contributions: This work contributes to the understanding of ongoing challenges and the importance of data representation on daily-used interfaces and requires in-depth dissemination to support novices in developing expertise in neurocritical care. While this research represented CWA models together with the exploration of expertise in the context of neurocritical care, there is not only a scientific contribution shown in this dissertation but also how technology and specific interfaces can be improved to support different end-users on varying expertise levels. The overall intention is to contribute to a better healthcare system in which physicians can provide best practices and dedication toward patients’ health states and share their expertise with newer generations. In the long run, this also has an educational and economic impact, improving hospital resource allocations.
“My pharmacist”: Creating and maintaining relationship between physicians and pharmacists in primary care settings
(Elsevier, 2020-01) Mercer, Kathryn; Neiterman, Elena; Guirguis, Lisa; Burns, Catherine; Grindrod, Kelly
Background Pharmacists and physicians are being increasingly encouraged to adopt a collaborative approach to patient care, and delivery of health services. Strong collaboration between pharmacists and physicians is known to improve patient safety, however pharmacists have expressed difficulty in developing interprofessional working relationships. There is not a significant body of knowledge around how relationships influence how and when pharmacists and physicians communicate about patient care. Objectives This paper examines how pharmacists and primary care physicians communicate with each other, specifically when they have or do not have an established relationship. Methods Thematic analysis of data from semi-structured interviews with nine primary care physicians and 25 pharmacists, we examined how pharmacists and physicians talk about their roles and responsibilities in primary care and how they build relationships with each other. Results We found that both groups of professionals communicated with each other in relation to the perceived scope of their practice and roles. Three emerging themes emerged in the data focusing on (1) the different ways physicians communicate with pharmacists; (2) insights into barriers discussed by pharmacists; and (3) how relationships shape collaboration and interactions. Pharmacists were also responsible for initiating the relationship as they relied on it more than the physicians. The presence or absence of a personal connection dramatically impacts how comfortable healthcare professionals are with collaboration around care. Conclusion The findings support and extend the existing literature on pharmacist-physician collaboration, as it relates to trust, relationship, and role. The importance of strong communication is noted, as is the necessity of improving ways to build relationships to ensure strong interprofessional collaboration.
Remote Medical Diagnosis in Virtual Reality: A Mixed-methods Approach to Understanding the Perceptions of Patients and Physicians
(University of Waterloo, 2024-12-11) Momoh, Mustapha Unubi; Burns, Catherine; Mikael Mäkelä, Ville
Global healthcare faces challenges, including physician shortages and resource limitations. Telehealth has offered solutions through services such as text and video chats. Yet, these methods have their issues: they provide only limited opportunities for diagnoses, and they do not foster solid patient-physician relationships. Virtual reality (VR) offers a promising future alternative, which could facilitate real-time patient-physician interactions that resemble real-life visits through realistic 3D avatars. However, understanding patients’ and physicians’ needs, attitudes, and concerns is crucial for tailoring such VR solutions to healthcare’s unique demands. Therefore, an online patient survey (n = 402) and physician interviews (n = 6) were conducted to understand these needs. Through thematic analysis, common telehealth concerns, including privacy and limited scope of diagnoses in VR, were identified. Unique elevated concerns, mostly around technology reliability, required expertise, accessibility, and integration with existing workflows, also emerged. Furthermore, the study examined the influence of technology affinity on patients’ acceptance of VR telehealth through a Regression Discontinuity Design (RDD) approach. Overall, this study explores the critical concerns in telehealth and proposes evidence-based considerations for developing VR-based telehealth solutions.
Supporting Caregivers in Complex Home Care: Towards Designing a Voice User Interface
(University of Waterloo, 2021-05-10) Tennant, Ryan; Burns, Catherine; Mercer, Kate
Despite significant advancements in the development of digital health tools and the rising provision of health care services in the home, information management and communication has yet to be standardized through digitization across caregiver teams in complex home care. With the increased risks of adverse events in dynamic and unpredictable home environments, there is a critical need to improve care inconsistencies and prevent communication breakdowns. An opportunity exists for digital health tools to support the standardization of information sharing processes in the home. However, designing digital tools to support complex home care is challenging when considering the uniqueness of patient conditions, the home environment, and caregiving team diversity. Adopting digital health tools in unregulated environments also induces a challenge for standardizing digitization in this complex domain. With advancements in natural language processing and speech recognition, the development of digital health interfaces that provide a natural interaction with information by voice has shown promise to support information management and communication and facilitate engagement with home care technology. The objective of this research is to build a foundation for the future development of a voice user interface or Voice Assistant (VA) to support caregivers in complex home care. The objectives are two-fold: (1) to understand the diverse caregiving experiences related to health information management and communication in complex home care and (2) evaluate the diverse perspectives of caregivers on the design of a VA to support these identified processes. Using a mixed-methods approach of semi-structured interviews and questionnaires with 22 caregivers across North America, this research contributes to understanding both the information and communication processes as well as the design considerations for integrating VA technology in complex home care by potential primary users. This thesis consists of three papers that describe the partial results of one study. One paper focuses on the semi-structured interviews with family caregivers of Children With Special Health Care Needs (CSHCN) to understand the processes involved with managing care in their home. The second paper focuses on the semi-structured interviews with family caregivers and hired caregivers of older adults in the same context. The third paper focuses on the semi-structured interviews and questionnaires with all participants about their expectations for the design of VAs in complex home care. This thesis captures the rich experiences of caregivers who are managing the coordination of care in complex home environments and the considerations for designing VA technology in this domain. The principal findings highlight similarities in caregiving processes and the nuanced complexities among caregiver populations that can inform the design and usability considerations of future digital health tools. There is also the potential for VA technology to provide utility for health information management and communication. However, considerations for functionality and the context of use may impact this innovation's diffusion. Future research should collectively examine home care from caregiving teams' perspectives and objectively measure human-information interaction with this technology in context-specific scenarios.
The Human Factors in the Adoption of Ambient Artificial Intelligence Scribe Technology: Towards Informed and User-centered Implementation of AI in Healthcare
(University of Waterloo, 2024-10-08) Basha, Iman; Burns, Catherine
The landscape of healthcare documentation has undergone substantial transformations over the past few decades, evolving in parallel with technological advancements and shifts in healthcare delivery models. Central to these changes is the electronic medical record (EMR), a digital iteration of patients' paper charts that has become standard in healthcare settings. While EMRs are instrumental in streamlining data management and accessibility, they have introduced new challenges, particularly in terms of administrative burden on healthcare providers. This thesis explores the integration of ambient artificial Intelligence (AI) scribe technology, a solution leveraging advancements in automatic speech recognition (ASR) and natural language processing (NLP), into physicians' workflows. AI scribes semi-automate the documentation process by capturing and synthesizing physician-patient interactions in real time, potentially alleviating the administrative workload on clinicians and improving the quality of care. The potential benefits of this technology are vast, and its adoption raises significant questions regarding privacy, consent, and trust, especially given its capability to record sensitive interactions in detail. The study aims to (1) explore the integration of ambient scribe technology into physicians' workflows and assess its impact on physician-patient interactions, (2) identify and analyze the concerns related to privacy, consent, and trust among patients and physicians regarding the use of the technology, and (3) develop and evaluate a flexible informed consent protocol for patients and physicians. A mixed-method approach was employed, integrating quantitative data from surveys and qualitative insights from semi-structured interviews, providing a comprehensive understanding of the multifaceted impact of the technology. The findings reveal that while AI scribes offer efficiency gains, particularly for complex and lengthy encounters, they are less beneficial for simple cases. Further, the efficiency of documentation with AI scribes compared to without is found to be dependent on individuals, with some physicians reporting negligible improvements due to extensive post-editing and the need for customization, while others noted notable gains. Regarding the impact on interaction, patients and physicians reported enhanced interactions due to reduced distractions but noted instances of self-censorship by patients due to discomfort with the recording process. Patients also expressed worry about self-censorship by physicians due to medicolegal concerns and unintended consequences due to technology over-reliance. Concerning the second objective, patients and physicians expressed significant privacy concerns due to a lack of understanding and transparency in data handling policies. Patients also expressed concerns regarding the autonomy of private data, unauthorized access, and data breaches. The findings underscore the need for transparent data handling policies and robust security measures. Trust in physicians and pre-established patient-physician relationships also played a notable role in patient consent, with patients more likely to consent to AI scribe use with familiar physicians. To address these concerns, the thesis proposed a Multi-Tier Granular Informed Consent (MTGIC) framework, integrating tiered and granular consent models to enhance transparency and participant control over personal data. The empirical evaluation of the MTGIC was well-received by both patients and physicians, though it necessitates ongoing refinement to improve usability and ensure it aligns with user needs. In conclusion, while ambient scribe technology presents a promising tool for enhancing healthcare delivery, its successful implementation is contingent upon careful consideration of its integration into clinical workflows, the management of privacy concerns, and the development of effective consent processes. This study contributes to the ongoing discussion on the best practices for integrating emerging technologies into healthcare systems, aiming to enhance operational efficiency and patient care quality.
Understanding Clinicians’ Requirements, Perception and Acceptance of Clinical Decision Support Systems: User Study for Implementation of Sepsis Best Practice Advisory in General Paediatric Care
(University of Waterloo, 2021-05-11) Olatoye, Olamide; Burns, Catherine
Clinicians are faced with ever-increasing patient data as well as medical evidence which are all required for them to make the best possible decisions. Clinical Decision Support Systems (CDSS) are widely used to support clinicians’ information processing and decision making. However, clinicians as end users are hardly involved in the design and development of these decision support tools. In addition, some of these CDSS designs and processes are not properly implemented to fit into the clinicians’ workflow. The study specifically investigated clinicians’ decision-making regarding Sepsis, design and workflow requirements as well as their perception and acceptance of the Sepsis best practice advisory (BPA). Sepsis is a life-threatening disease, and it is important to identify early manifestations rapidly and reliably for timely interventions as every hour of delay increases mortality by 5-10% (37). The aim was to identify the factors that can aid the implementation of the CDSS such that there is no reduced or incorrect usage and interference with clinicians’ decision making. Successful implementation of the CDSS can further improve patient’s safety especially with regards to Sepsis care. The study was in two phases, a user interview and a moderated usability testing. Both phases were qualitative studies obtaining data from a total of 13 participants from a target population of clinicians working in the general paediatrics unit of the hospital. Decision ladders from control task analysis (ConTa) and cognitive work analysis (CWA) were used to model clinicians’ decision making and the support provided by the Sepsis BPA. The unified theory of acceptance and use of technology (UTUAT) was used to measure clinicians’ satisfaction and acceptance of the tool. The first phase of the study discovered the general experience, knowledge, challenges caring for patients with Sepsis as well as experiences with CDSS and clinicians’ projections or expectations of the Sepsis BPA. Key findings were translated into user requirements which were checked against the minimum viable product (MVP) of the Sepsis BPA and recommendations provided. The second phase discovered particular design feedback and usability issues on the MVP with more recommendations provided. The UTAUT survey results showed highly positive feedback on satisfaction, acceptance and intentions of clinicians to use the Sepsis BPA.
Using a collaborative research approach to develop an interdisciplinary research agenda for the study of mobile health interventions for older adults.
(JMIR Mhealth Uhealth., 2015-02-10) Grindrod, Kelly; Baskerville, Neill; Giangregorio, Lora M.; Tomasson Goodwin, Jill; Sadat Rezai, Leila; Mercer, Kathryn; Burns, Catherine; Chang, Feng
Background: Seniors with chronic diseases are often called on to self-manage their conditions. Mobile health (mHealth) tools may be a useful strategy to help seniors access health information at the point of decision-making, receive real-time feedback and coaching, and monitor health conditions. However, developing successful mHealth interventions for seniors presents many challenges. One of the key challenges is to ensure the scope of possible research questions includes the diverse views of seniors, experts and the stakeholder groups who support seniors as they manage chronic disease. Objective: Our primary objective was to present a case-study of a collaborative research approach to the development of an interdisciplinary research agenda. Our secondary objectives were to report on the results of a nominal group technique (NGT) approach used generate research questions and to assess the success of including non-academic researchers to enrich the scope, priority, and total number of possible research questions. Methods: We invited researchers and stakeholders to participate in a full day meeting that included rapid-style presentations by researchers, health care professionals, technology experts, patients and community groups followed by group discussions. An NGT was used to establish group consensus on the following question: In your opinion, what research needs to be done to better understand the effectiveness, usability and design of mobile health apps and devices for older adults? Results: Overall, the collaborative approach was a very successful strategy to bring together a diverse group of participants with the same end goal. The 32 participants generated 119 items in total. The top three research questions that emerged from the NGT were related to adoption, the need for high quality tools and the digital divide. Strong sub-themes included privacy and security, engagement and design. The NGT also helped us include the perspectives information from non-academic researchers that would not have been captured if the process had been limited to the research team. Conclusions: Developing ways for patients and other stakeholders to have a voice when it comes to developing patient awareness as related to mHealth may guide future research into engagement, ownership, usability and design. It is our intention that our paper be used and adapted by other researchers to engage small or vulnerable populations often excluded from mHealth research and design.