© 2018 Epic Systems Corporation.
1
EHR Usability Test Report
for EpicCare Inpatient
Base August 2018
Report based on NISTIR 7742 Customized Common Industry Format Template for Electronic Health Record Usability Testing
Dates of Usability Study: February, March & August 2018
Date of Report: September 7, 2018
Report Prepared by: Epic
© 2018 Epic Systems Corporation.
2
Table of Contents
1 Executive Summary .........................................................................................................................................4
2 Introduction .......................................................................................................................................................4
3 Method.................................................................................................................................................................7
3.1 Participants................................................................................................................ 7
3.2 Study Design ........................................................................................................... 18
3.3 Tasks ....................................................................................................................... 19
3.4 Procedures............................................................................................................... 28
3.5 Test Location ........................................................................................................... 28
3.6 Test Environment .................................................................................................... 28
3.7 Test Forms and Tools ............................................................................................... 29
3.8 Participant Instructions ............................................................................................ 29
3.9 Usability Metrics...................................................................................................... 29
4 Results ............................................................................................................................................................... 31
4.1 §170.315(a)(9) Clinical Decision Support ................................................................... 31
4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation ........................ 33
4.3 §170.315(a)(1) Computerized Provider Order Entry - Medications............................. 35
4.4 §170.315(a)(2) Computerized Provider Order Entry - Laboratory .............................. 37
4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging ................. 39
4.6 §170.315(a)(5) Demographics.................................................................................... 41
4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks ..................................... 43
4.8 §170.315(b)(3) Electronic Prescribing......................................................................... 45
4.9 §170.315(a)(14) Implantable Device List .................................................................... 47
4.10 §170 315(a)(8) Medication Allergy List .................................................................... 49
4.11 §170 315(a)(7) Medication List ................................................................................ 51
4.12 §170.315(a)(6) Problem List ..................................................................................... 53
5 Results Conclusion....................................................................................................................................... 55
Appendices ....................................................................................................................................................... 561
Appendix 1 ................................................................................................................... 56
Appendix 2 ................................................................................................................... 61
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 3
Appendix 3 ................................................................................................................... 63
Appendix 4 ................................................................................................................... 64
Appendix 5 ................................................................................................................... 69
Appendix 6 ................................................................................................................... 70
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 4
1 Executive Summary Epic staff conducted a usability study of the August 2018 version of EpicCare Inpatient Base1 in February,
March, and August of 2018 at multiple healthcare organizations. The purpose of this study was to evaluate the
usability of the user interface and provide quantitative analysis of the usability of EpicCare Inpatient. During
the usability test, 195 healthcare providers used EpicCare Inpatient in simulated, representative tasks. Each
task was analyzed for risk using the methods detailed in section 3.3.
This study collected performance data on various tasks typically conducted by physicians and nurses. The tasks correspond to certification criteria identified in 45 CFR Part 170 Subpart C of the Health Information
Technology: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic
Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications . For information about the
overall User-Centered Design (UCD) process at Epic, see the User Centered Design document as submitted
for 9.1.4.
Participants in the usability study had experience with a prior version of EpicCare Inpatient. The moderators
introduced the test and instructed participants to complete a series of tasks, given one at a time, using EpicCare
Inpatient, following the testing procedures outlined in section 3.4. After each task, the participants were asked
to complete an Ease of Task Completion rating for that task. The participants’ interactions with the screen,
facial expressions, and verbalizations were recorded electronically during the test and later analyzed to collect
time measurements and evaluate performance. The moderators did not assist the participants in completing
the tasks. All participant data was de-identified.
In accordance with the examples in the NIST 7742 Customized Common Industry Format Template for Electronic
Health Record Usability Testing, various recommended metrics were used to evaluate the usability of the
software. The following quantitative metrics were collected for each eligible participant:
Task completion
Time to complete each task
Number and type of unnecessary steps
Number and type of extra steps
Participant’s Ease of Task Completion ratings
In addition to the performance data, the following qualitative observations were made:
Post-test debrief comments
Major findings
Areas for improvement
2 Introduction
2.1 Procedure Overview
The usability study was conducted on the August 2018 version of EpicCare Inpatient Base, which facilitates
inpatient care workflows and presents healthcare providers in an inpatient setting with integrated medical
information documented in a single patient chart. This usability study included realistic scenarios typically
1 ONC Health IT Certification (for Meaningful Use) information including pricing and limitations is available here: http://www.epic.com/Docs/MUCertification.pdf.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 5
encountered by clinicians who use EpicCare Inpatient. Scenarios were constructed in collaboration with
clinicians to ensure clinical accuracy.
The purpose of the usability study was to validate and provide quantitative evidence of the usability of
EpicCare Inpatient Base. Accordingly, the testing data measured efficiency, effectiveness, and user satisfaction
through a collection of metrics including time spent on each task, extra and unnecessary steps taken per task,
and Ease of Task Completion ratings from participants.
For the purpose of reporting findings from this usability study, commonly used terms are defined as follows:
Participant: A clinician who has experience with a prior version of EpicCare Inpatient, is eligible for
participation in the usability study as determined by the Recruiting Screener (see Appendix 1), and
has completed the usability test
Scenario: A patient synopsis, given to participants to provide clinical context for tasks
Task: A verbal and written clinical workflow that is provided to all participants in the usability study
and has a predefined desired outcome
Subtask: The portion of a task relating to a specific criterion for which data is analyzed
Test: The compilation of tasks specific to studied criteria given in a single sitting to a participant
Path: A series of actions that can be taken in EpicCare Inpatient Base to reach an outcome
The study was performed using the Epic 2018 and August 2018 versions of EpicCare Inpatient Base. For more
information on the testing method, see Section 3.1.
2.2 Certification Usability Testing in Context
The usability testing Epic publically reports for ONC 2015 Edition certification represents a small segment of
the work Epic puts toward user centered design and quality assurance. Epic’s user centered design processes
and quality management system focus on designing and developing high quality software based on a deep
understanding of the user, task, and environment.
Epic approaches usability from multiple angles to create workflows that are simple, intuitive, and attractive
while providing sophisticated functionality and promoting productivity and efficiency.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 6
Fig 1: User Centered Design Process
Research It’s critical that Epic software developers understand users’ requirements, needs, values, and goals. All Epic
software developers conduct extensive ethnographic research including user interviews, workflow
observation, and focus groups.
Collaborative Design and Development
Development projects are selected and prioritized based on feedback from Epic users. Following that input,
the design team, comprised of software developers, clinicians, usability professionals, users, and operational
stakeholders, detail how proposed development addresses the user’s goals.
From wireframe mockups to prototype-based formative usability testing, users are involved in all aspects of
the design process. Epic conducts group design sessions in person and via webcasts and calls with members of
the Epic community. In addition, internal user experience experts evaluate designs based on industry-standard
usability heuristics and Epic’s usability style guide.
User Testing
Epic performs both formative and summative testing. Formative testing early in the cycle informs broad
design decisions. Summative testing on pre-release and released development quantifies usability and
identifies major areas for future focus and improvement. Epic uses industry-standard usability testing
techniques, including card sorts, preference testing, and eye tracking, to collect feedback and establish design
goals.
Customer staff who come to Epic for training and conferences have the opportunity to participate in usability
labs and simulations labs to take part in formative testing and give feedback on in -process development.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 7
3 Method
3.1 Participants
A total of 95 inpatient physicians and 101 inpatient nurses participated in the study. Nurses were asked to
complete 10 tasks. Physician testing was divided into two sessions due to updates in the user interface in the
discharge ordering workflow for the August 2018 release. During the first session, conducted in March of 2018,
physicians were asked to complete 15 tasks in the Epic 2018 version. In the second session, conducted in
August of 2018, physicians were asked to complete three tasks that were affected by the updates in the August
2018 release. Twelve of the 15 physician tasks from the first session were not changed by the updated user
interface in the August 2018 release and were not retested. None of the nursing workflows tested in February
and March of 2018 were affected by the August 2018 release. The demographics, task, and results information
reflect all sessions conducted in February-March and August of 2018.
Participants were recruited by leaders at their organizations and Epic staff. Participants were not compensated
by Epic for their time. EpicCare Inpatient Base is designed to accommodate physician and nurse specialists,
therapists, and other specialized care providers, in addition to general medicine providers, in an inpatient
setting. The participants were actual users of a previous version of EpicCare Inpatient. Participants were not
directly connected to the development of EpicCare Inpatient, nor were they employed by Epic.
Participants completed a recruiting screener that was used to gather demographic data and to verify eligibility
for participation in the study (see Appendix 1 for the Recruiting Screener used for this study). Recruited
participants represented a mix of backgrounds and demographic characteristics. Participants were assigned
an alphanumeric participant identifier at the time of the usability test to de-identify results.
Of the 123 physicians who were given the Recruiting Screener, 95 qualified to participate in the usability study.
Of the 150 nurses who were given the Recruiting Screener, 101 qualified to participate in the usability study.
The most common reason for ineligibility was that the participant did not provide patient care (see Appendix
1 for a full list of eligibility criteria). Participants were scheduled for individual 30-minute testing sessions.
Demographic data is listed in Tables 1-2 and summarized in Appendix 3.
Table 1: Participant Demographics - Inpatient Physicians
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
1 I08 F Physician Doctorate (MD, DO,
PhD, DNP)
7 OB/GYN 30-39 Yes 10 25
2 I15 F Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatrics 40-49 Yes 12 20
3 I28 F Physician Doctorate (MD, DO,
PhD, DNP)
2 Hospitalist 40-49 Yes 11 20
4 J02 F Physician Doctorate (MD, DO,
PhD, DNP)
1.5 Internal Medicine 20-29 Yes 1.5 16
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 8
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
5 J03 M Physician Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 30-39 No 17 20
6 J05 F Physician Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 30-39 No 2 10
7 J06 M Physician Doctorate (MD, DO,
PhD, DNP)
2 OB/GYN 50-59 Yes 25 20
8 J08 M Physician Doctorate (MD, DO,
PhD, DNP)
1.5 OB/GYN 30-39 Yes 2 15
9 J12 F Physician Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 20-29 No 2 18
10 J13 M Physician Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 20-29 Yes 3 20
11 J18 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Internal Medicine 20-29 Yes 2.5 10
12 J19 M Physician Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 30-39 No 2.5 10
13 J23 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatrics 40-49 Yes 21 25
14 J35 F Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatric Hospital
Medicine
40-49 Yes 12 15
15 J36 F Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatrics 30-39 Yes 3 25
16 J39 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Cardiac Surgery 70-79 Yes 37 50
17 K01 M Physician Doctorate (MD, DO,
PhD, DNP)
9 Internal Medicine 40-49 No 13 20
18 K02 M Physician Doctorate (MD, DO,
PhD, DNP)
4 Hospitalist 30-39 Yes 10 15
19 K04 F Physician Doctorate (MD, DO,
PhD, DNP)
3 Family Practice 30-39 Yes 3 20
20 K06 M Physician Doctorate (MD, DO,
PhD, DNP)
9 Internal Medicine 40-49 No 20 15
21 K07 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Internal Medicine 30-39 No 15 20
22 K08 M Physician Doctorate (MD, DO,
PhD, DNP)
1.5 Internal Medicine 30-39 No 7 20
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 9
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
23 K09 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Internal Medicine 50-59 Yes 20 10
24 K10 M Physician Doctorate (MD, DO,
PhD, DNP)
10 Internal Medicine 30-39 Yes 19 15
25 L01 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 40-49 Yes 18 20
26 L03 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
5 Hospitalist 40-49 Yes 20 20
27 L04 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 40-49 Yes 17 10
28 L05 F Physician Doctorate (MD, DO,
PhD, DNP)
5 Internal Medicine 50-59 No 15 15
29 L06 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 40-49 No 9 20
30 L07 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Internal Medicine 30-39 Yes 10 20
31 L08 F Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
5 Neonatology 40-49 Yes 16 20
32 L10 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Pediatric Endocrinology 30-39 Yes 1 25
33 L12 M Physician Doctorate (MD, DO,
PhD, DNP)
0.5 Family Medicine,
Geriatrics
40-49 Yes 24 25
34 L13 M Physician Doctorate (MD, DO,
PhD, DNP)
7 Pediatric 40-49 No 7 18
35 L14 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Family Medicine 30-39 No 6 20
36 L15 M Physician;
CMO
Doctorate (MD, DO,
PhD, DNP)
5 Pulmonology; Internal
Medicine
60-69 Yes 38 20
37 L16 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Family Practice,
Hospitalist
30-39 No 5 20
38 L17 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
7 Neurology; Pediatric
Neurology
40-49 Yes 15 10
39 M01 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 30-39 Yes 10 30
40 M02 M Physician Doctorate (MD, DO,
PhD, DNP)
12 Internal Medicine 40-49 Yes 20 40
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 10
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
41 M03 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Neurology 30-39 No 5 20
42 M04 F Physician Doctorate (MD, DO,
PhD, DNP)
12 Hospitalist 30-39 Yes 10 30
43 N01 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Cardiac Transplant 30-39 Yes 6 10
44 N02 M Physician Doctorate (MD, DO,
PhD, DNP)
7 Critical Care 30-39 Yes 10 20
45 N03 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 40-49 No 9 25
46 N04 M Physician Doctorate (MD, DO,
PhD, DNP)
5 Palliative Medicine 30-39 Yes 5 20
47 N05 M Physician;
CQO
Doctorate (MD, DO,
PhD, DNP)
7 Internal Medicine 30-39 Yes 5 20
48 N06 M Physician Doctorate (MD, DO,
PhD, DNP)
4 Infectious Diseases,
Internal Medicine
40-49 Yes 14 30
49 N07 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Critical Care 30-39 Yes 2 15
50 N08 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
7 Critical Care 30-39 Yes 10 20
51 N09 M Physician Doctorate (MD, DO,
PhD, DNP)
5.5 Internal Medicine 40-49 Yes 2.5 30
52 N10 M Physician;
IT
Doctorate (MD, DO,
PhD, DNP)
7 Hematology 30-39 Yes 7 30
53 N11 F Physician Doctorate (MD, DO,
PhD, DNP)
6 Pediatrics, Internal
Medicine
30-39 Yes 6 20
54 N13 M Physician Doctorate (MD, DO,
PhD, DNP)
9 OB/GYN 50-59 Yes 28 28
55 N15 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Anesthesiologist 30-39 Yes 6 34
56 Z03 M Physician Doctorate (MD, DO,
PhD, DNP)
5 OB/GYN 30-39 Yes 5 20
57 Z04 M Physician Doctorate (MD, DO,
PhD, DNP)
2 Pediatrics 30-39 Yes 6 20
58 Z05 F Physician Doctorate (MD, DO,
PhD, DNP)
4 Pediatrics 30-39 Yes 4 20
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 11
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
59 Z06 F Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
4 Pediatric Critical Care 40-49 Yes 19 40
60 AA01 M Physician Doctorate (MD, DO,
PhD, DNP)
0.25 Internal Medicine 30-39 Yes 3 20
61 AA04 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Psychiatry 40-49 No 20 30
62 BB01 M Physician Doctorate (MD, DO,
PhD, DNP)
0.25 Internal Medicine 40-49 No 15 30
63 BB02 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 30-39 Yes 2 20
64 BB03 F Physician Doctorate (MD, DO,
PhD, DNP)
7 Internal Medicine 30-39 No 4 20
65 BB04 F Physician Doctorate (MD, DO,
PhD, DNP)
1 Pulmonology/Critical
Care
40-49 Yes 20 15
66 CC04 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Internal Medicine 30-39 Yes 3 20
67 CC06 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Psychiatry 30-39 Yes 6 18
68 DD01 F Physician Doctorate (MD, DO,
PhD, DNP)
4 GI 30-39 Yes 5 20
69 DD02 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
6 Family Medicine 40-49 No 9 25
70 DD03 M Physician Doctorate (MD, DO,
PhD, DNP)
1.5 Infectious Disease,
Internal Medicine
40-49 No 10 31
71 DD04 F Physician;
CMIO; Trainer
Doctorate (MD, DO,
PhD, DNP)
1 Family Medicine 30-39 Yes 16 20
72 DD05 M Physician Doctorate (MD, DO,
PhD, DNP)
0.5 Cardiology 40-49 Yes 11 30
73 DD07 M Physician Doctorate (MD, DO,
PhD, DNP)
6 Hospitalist 30-39 Yes 3 20
74 U01 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
6 Anesthesiology, Internal
Medicine
50-59 Yes 20 20
75 U02 M Physician Doctorate (MD, DO,
PhD, DNP)
8 Pediatric Orthopedic
Surgery
50-59 Yes 20 25
76 U03 M Physician Doctorate (MD, DO,
PhD, DNP)
7 Pediatric Pulmonology 30-39 No 5 20
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 12
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
77 U04 F Physician;
CMO
Doctorate (MD, DO,
PhD, DNP)
0.67 Pulmonology/Critical
Care
50-59 No 18 30
78 U06 M Physician Doctorate (MD, DO,
PhD, DNP)
10 Internal Medicine 60-69 Yes 35 40
79 U07 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
8 Physical Medicine and
Rehabilitation
50-59 Yes 27 20
80 U08 M Physician Doctorate (MD, DO,
PhD, DNP)
13 HBS 50-59 Yes 20 20
81 U09 M Physician Doctorate (MD, DO,
PhD, DNP)
4 Hospitalist 30-39 Yes 6 25
82 U10 M Physician;
Trainer
Doctorate (MD, DO,
PhD, DNP)
0.33 Anesthesia 50-59 Yes 16 30
83 U11 M Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatrics 30-39 Yes 3 25
84 U12 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
7 Internal Medicine 50-59 Yes 26 40
85 U13 M Physician Doctorate (MD, DO,
PhD, DNP)
10 Family Medicine,
Hospitalist
30-39 Yes 8 20
86 U14 F Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
8 Internal Medicine 30-39 No 8 25
87 U15 M Physician;
Director
Doctorate (MD, DO,
PhD, DNP)
5 Hospitalist, Internal
Medicine
40-49 Yes 12 20
88 U16 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
2 Internal Medicine 30-39 Yes 8 20
89 U17 M Physician Doctorate (MD, DO,
PhD, DNP)
2 Hospitalist, Internal
Medicine
40-49 Yes 16 20
90 U18 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
10 Hospitalist 30-39 Yes 8 25
91 U19 M Physician;
CMIO
Doctorate (MD, DO,
PhD, DNP)
13 Cardiology 70-79 Yes 40 40
92 U20 M Physician Doctorate (MD, DO,
PhD, DNP)
4 Internal Medicine 40-49 Yes 15 38
93 U22 F Physician Doctorate (MD, DO,
PhD, DNP)
3 Pediatric
Hematology/Oncology
40-49 Yes 14 25
94 U23 M Physician Doctorate (MD, DO,
PhD, DNP)
7 Pediatric
Hematology/Oncology
40-49 Yes 15 25
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 13
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported by
user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
95 U24 M Physician;
Informatics
Doctorate (MD, DO,
PhD, DNP)
6 Infectious Disease 50-59 Yes 20 15
Table 2: Participant Demographics - Inpatient Nurses
ID Gender Role Education Leve l Epic User
Experience (Years)
Specialty (as reported
by user)
Age First
Language English?
Experience
in Fie ld (Years)
Computer
Use (Years)
1
A01 F Nurse Bache lor's
degree
2 Mother-Baby 20-29 Yes 2 20
2 A09 F Nurse Master's degree
(MSN, MS)
13 Med/Surg 50-59 No 30 38
3 A11 F Nurse Bache lor's
degree
9 Med/Surg 50-59 No 16 30
4 A12 F Nurse Master's degree
(MSN, MS)
8 Med/Surg 40-49 No 18 18
5 A15 F Nurse Bache lor's
degree
5 Med/Surg 30-39 Yes 5 10
6 A18 F Nurse Bache lor's
degree
10 Surgical ICU 30-39 Yes 13 20
7 A19 F Nurse Bache lor's
degree
10 Surgical ICU 40-49 Yes 13 25
8 A21 F Nurse Bache lor's
degree
10 Oncology 40-49 Yes 12 22
9 A22 F Nurse Associate degree 3 Med/Surg 40-49 Yes 7 20
10 A23 F Nurse Associate degree 2 Med/Surg 50-59 No 20 5
11 A206 F Nurse Associate degree 5 Med/Surg 30-39 Yes 6 15
12 A25 F Nurse Associate degree 4 Med/Surg 40-49 Yes 4 8
13 A26 F Nurse Bache lor's
degree
9 Transplant; Med/Surg 40-49 No 21 25
14 A27 F Nurse Bache lor's
degree
3 Not reported 20-29 No 5 10
15 A29 F Nurse Bache lor's
degree
2 Neurology 30-39 Yes 6 25
16 A216 F Nurse Bache lor's
degree
3.5 ICU 20-29 Yes 3.5 10
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 14
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
17 A222 F Nurse Bache lor's
degree
10 OB 40-49 Yes 17 10
18 A223 F Nurse Bache lor's
degree
8 Family Centered Care 60-69 Yes 10 30
19 A228 F Nurse Associate degree 13 Neonatal ICU 50-59 Yes 30 30
20 A229 F Nurse Master's degree
(MSN, MS)
5 Med/Surg 30-39 No 5 20
21 A32 F Nurse Associate degree 1 Oncology 30-39 Yes 9 15
22 A38 F Nurse Bache lor's
degree
1 Labor & Delivery 50-59 Yes 8.5 20
23 A49 M Nurse Bache lor's
degree
0.75 Med/Surg 20-29 Yes 0.75 13
24 A54 F Nurse Associate degree 2 Pre -op; Post-op 30-39 Yes 10 15
25 A55 F Nurse Bache lor's
degree
3 Surgical 20-29 Yes 3 10
26 A62 F Nurse Bache lor's
degree
2 Orthopedics 50-59 Yes 6 36
27 A65 F Nurse Bache lor's
degree
2.5 NSICU 30-39 Yes 9 20
28 A66 M Nurse Master's degree
(MSN, MS)
3 Cardiovascular ICU 40-49 Yes 20 30
29 A68 F Nurse Bache lor's
degree
10 CICU 30-39 Yes 10 30
30 A230 F Nurse Bache lor's
degree
9 Mother-Baby 50-59 Yes 9 >20
31 A231 F Nurse Bache lor's
degree
9 Neonatal 30-39 Yes 10 >20
32 B01 F Nurse Bache lor's
degree
2 Surgical Intermediate
Care
20-29 Yes 5 15
32 B02 M Nurse Master's degree
(MSN, MS)
2.5 BMT 40-49 Yes 5 26
34 B06 F Nurse Master's degree
(MSN, MS)
2 Intermediate
Care /Medical
30-39 Yes 4 20
35 B07 M Nurse Bache lor's
degree
2 Trauma 30-39 Yes 2 15
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 15
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
36 B12 F Nurse Bache lor's
degree
8 Cardiovascular ICU 40-49 Yes 9 25
37 B18 F Nurse Master's degree
(MSN, MS)
2 Med/Surg 50-59 Yes 26 5
38 B19 F Nurse Bache lor's
degree
2 Mother-Baby 20-29 Yes 4 15
39 B22 F Nurse Bache lor's
degree
2 Med/Surg 30-39 Yes 6 25
40 B24 F Nurse Bache lor's
degree
1 Te lemetry; Med/Surg 20-29 Yes 5 25
41 B25 F Nurse Bache lor's
degree
1.5 ICU 20-29 Yes 2.5 10
42 B30 M Nurse Bache lor's
degree
2 Not reported 40-49 No 16 >2
43 B43 F Nurse Bache lor's
degree
2 Med/Surg 40-49 Yes 20 20
44 B58 F Nurse Bache lor's
degree
1 Med/Surg 30-39 Yes 4 20
45 B107 F Nurse Master's degree
(MSN, MS)
1.5 Hospice 60-69 Yes 42 40
46 C01 M Nurse ;
Manager
Bache lor's
degree
5 Critical Care 30-39 Yes 12 25
47 C02 M Nurse ;
Manager
Bache lor's
degree
2 Critical Care 50-59 Yes 37 20
48 C03 M Nurse Master's degree
(MSN, MS)
5 ICU, CCU 30-39 Yes 8 26
49 C06 M Nurse Bache lor's
degree
5 CCU 30-39 Yes 13 25
50 C08 F Nurse Bache lor's
degree
5 Postpartum 40-49 Yes 10 >20
51 C09 F Nurse Associate degree 3 Critical Care 50-59 Yes 25 14
52 C13 M Nurse Bache lor's
degree
2 Behavioral Health 20-29 Yes 3 15
53 C14 F Nurse Master's degree
(MSN, MS)
2 Med/Surg 40-49 Yes 8 20
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 16
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
54 C18 F Nurse Bache lor's
degree
1.5 NICU 60-69 No 25 5
55 C20 F Nurse Associate degree 2 Mother-Baby 40-49 Yes 16 25
56 C21 F Nurse Associate degree 2.5 NICU 60-69 Yes 20 38
57 C402 F Nurse Bache lor's
degree
4 Intensive Care 60-69 Yes 25 30
58 C430 F Nurse Associate degree 7 Not reported 50-59 Yes 32 20
59 C433 F Nurse Bache lor's
degree
6 Med/Surg 20-29 Yes 6 20
60 C434 F Nurse Associate degree 6 Surgical 30-39 Yes 6 20
61 C435 F Nurse Associate degree 10 Labor & Delivery 30-39 Yes 2.5 >20
62 C436 M Nurse Bache lor's
degree
5 ICU 40-49 Yes 7 25
63 C437 M Nurse Associate degree 10 Med/Surg 30-39 Yes 15 30
64 C438 F Nurse Associate degree 6 Med/Surg 50-59 No 28 26
65 C442 M Nurse Bache lor's
degree
5 Critical Care 30-39 Yes 5 8
66 C443 F Nurse Bache lor's
degree
5 Psychiatry 50-59 No 10 10
67 C444 F Nurse Bache lor's
degree
0.5 NICU 30-39 Yes 2 20
68 C445 F Nurse Bache lor's
degree
1.5 Psychiatry 30-39 Yes 3 15
69 C448 F Nurse Associate degree 4 Psychiatry 40-49 Yes 4 5
70 C450 F Nurse Bache lor's
degree
2.5 Postpartum 30-39 Yes 2.5 20
71 D01 F Nurse Associate degree 5 Neonatal 50-59 Yes 30 15
72 D02 F Nurse Bache lor's
degree
5 Med/Surg 20-29 Yes 6 18
73 D03 F Nurse Associate degree 5 Pediatrics, PICU 50-59 Yes 32 20
74 E14 M Nurse ;
Analyst
Master's degree
(MSN, MS)
4 Med/Surg 40-49 No 10 26
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 17
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
75 E15 F Nurse Bache lor's
degree
10 PICU 40-49 Yes 26 35
76 E409 F Nurse ;
Analyst
Bache lor's
degree
11 CICU 40-49 Yes 18 20
77 E410 F Nurse ;
Analyst
Master's degree
(MSN, MS)
10 CICU 40-49 Yes 16 20
78 H02 F Nurse Bache lor's
degree
0.5 Med/Surg 20-29 Yes 1 10
79 H04 F Nurse Associate degree 5 Oncology 40-49 Yes 15 20
80 H06 F Nurse Associate degree 5 Trauma Med/Surg 20-29 Yes 5 15
81 H07 F Nurse Bache lor's
degree
2 Med/Surg 20-29 No 1 20
82 H08 F Nurse Master's degree
(MSN, MS)
2 Te lemetry; Cardiac 20-29 Yes 1 10
83 H09 F Nurse Associate degree 5 Med/Surg 30-39 Yes 5 10
84 H10 F Nurse Bache lor's
degree
1 Med/Surg 20-29 Yes 1 10
85 H12 F Nurse Bache lor's
degree
3 Med/Surg 30-39 Yes 4 20
86 I06 F Nurse Bache lor's
degree
6 Pediatrics 30-39 Yes 4 20
87 I12 F Nurse ;
Information
Technology
Bache lor's
degree
2 Surgical ICU 40-49 Yes 10 20
88 I16 F Nurse Bache lor's
degree
3 ICU 50-59 Yes 22 33
89 I18 F Nurse Associate degree 4 Med/Surg 40-49 Yes 10 27
90 I21 F Nurse Bache lor's
degree
1 Rehab 40-49 Yes 20 15
91 I24 F Nurse Master's degree
(MSN, MS)
2 ICU 30-39 Yes 20 20
92 I26 M Nurse Bache lor's
degree
2 ICU 30-39 Yes 3 12
93 I27 M Nurse Associate degree 3 CRRN 50-59 Yes 36 12
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 18
ID Gender Role Education Leve l Epic User
Experience
(Years)
Specialty (as reported
by user)
Age First
Language
English?
Experience
in Fie ld
(Years)
Computer
Use
(Years)
94 I29 F Nurse Bache lor's
degree
2 Trauma, Oncology,
Ortho
20-29 Yes 2 16
95 I31 F Nurse Bache lor's
degree
1 Ortho Trauma 20-29 Yes 6 20
96 I32 F Nurse Bache lor's
degree
8 Rehab 30-39 Yes 8 23
97 I34 F Nurse Master's degree
(MSN, MS)
3 Med/Surg 60-69 Yes 40 15
98 I44 F Nurse Bache lor's
degree
2 Orthopedics 40-49 Yes 20 20
99 J09 F Nurse Bache lor's
degree
2 NICU 50-59 No 30 15
100 J14 F Nurse Bache lor's
degree
3 Te lemetry 30-39 Yes 6 25
101 J16 F Nurse Bache lor's
degree
2 Med/Surg 30-39 Yes 12 20
3.2 Study Design
The objective of this study was to demonstrate areas where the application suite performed well – that is,
effectively, efficiently, and satisfactorily – and identify areas where improvements can be made.
Participants interacted with either the Epic 2018 or August 2018 version of EpicCare Inpatient Base. Each
participant used the system in a designated location, usually a conference or training room at the site where
the participant is employed. All participants were provided with the same instructions by the test moderator.
The system was evaluated for effectiveness, efficiency, and satisfaction as defined by metrics collected and
analyzed for each participant:
Task completion
Time to complete each task
Number and type of unnecessary steps
Number and type of extra steps
Participant’s Ease of Task Completion ratings
For additional information on usability metrics, see section 3.9.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 19
3.3 Tasks
3.3.1 Task List
Tasks were constructed to be realistic and representative of typical activities a user would complete using
EpicCare Inpatient Base. The tasks were prioritized and selected through a combination of the risk analysis
framework outlined in section 3.3.2 and the priorities outlined in NISTIR 7804-012.
NISTIR 7804-01 is an industry standard that provides scenarios and guidelines for usability testing of
Electronic Health Records. Tasks for the usability study were constructed to emphasize priorities articulated
in NISTIR 7804-1 (e.g. identification of information, consistency of information, and integrity of information),
and incorporated the NIST test scenarios when applicable.
Tasks are split into subtasks that are measurable components related to criteria supplied by the ONC. See
Appendix 4 for full task wording.
Physician Tasks/Subtasks, Session 1
Scenario 1: Gertrude is a 55-year-old female who has been admitted to your unit from the ED after
experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition, abrasions,
and a possible concussion.
Task 1: Reconcile a problem from a primary care provider at an outside organization.
§170.315(b)(2) Clinical information reconciliation and incorporation
Task 2: Add a medication allergy to a patient’s allergy list and assess any interactions.
§170.315(a)(4) Drug-drug, drug-allergy interaction checks
§170.315(a)(8) Medication allergy list
Task 3: Modify an order for a diagnostic imaging procedure.
§170.315(a)(3) Computerized provider order entry – Diagnostic imaging
Task 4: Place a medication order and place the consult order suggested by the system’s clinical decision
support.
§170.315(a)(1) Computerized provider order entry – Medications
§170.315(a)(8) Clinical decision support
Scenario 2: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently
admitted to your unit.
Task 5: Review the patient’s problem list.
§170.315(a)(6) Problem list
Task 6: Modify the details of a patient-controlled analgesic (PCA) medication order.
§170.315(a)(1) Computerized provider order entry – Medications
2 NISTIR 7804-1 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability Guidelines for
Standardization
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 20
Task 7: Modify the details of an inpatient laboratory order.
§170.315(a)(2) Computerized provider order entry – Laboratory
Task 8: (Replaced by Task 16)
Task 9: (Replaced by Task 17)
Scenario 3: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and
bradycardia. He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a
CVA last night.
Task 10: Add a problem to the problem list and place the consult order suggested by the system’s clinical
decision support.
§170.315(a)(6) Problem list
§170.315(a)(8) Clinical decision support
Task 11: Modify a medication allergy on a patient’s allergy list.
§170.315(a)(8) Medication allergy list
Task 12: Place a medication order and assess any interactions.
§170.315(a)(4) Drug-drug, drug-allergy interaction checks
§170.315(a)(1) Computerized provider order entry – Medications
Task 13: Order an ultrasound to diagnose symptoms.
§170.315(a)(3) Computerized provider order entry – Diagnostic imaging
Task 14: Resolve a problem on the patient’s problem list.
§170.315(a)(6) Problem list
Task 15: (Replaced by Task 18)
Physician Tasks/Subtasks, Session 2
Scenario 1: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently
admitted to your unit. Sheryl has recovered enough to be discharged.
Task 16: Order an outpatient lab to monitor a discharge medication.
§170.315(a)(2) Computerized provider order entry – Laboratory
Task 17: Electronically prescribe a discharge medication.
§170.315(a)(1) Computerized provider order entry – Medications
§170.315(b)(3) Electronic prescribing
Scenario 2: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and
bradycardia. He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a
CVA a few nights ago. Several days have passed, and Arthur is ready to be discharged.
Task 18: Electronically prescribe a discharge medication.
§170.315(a)(1) Computerized provider order entry – Medications
§170.315(b)(3) Electronic prescribing
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 21
Nurse Tasks/Subtasks
Scenario 1: Walter is a 79-year-old male with a complex medical history that includes CHF, osteoporosis,
dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a wound infection.
Task 1: Document the patient’s preferred language.
§170.315(a)(5) Demographics
Task 2: Reconcile allergy information from an outside urgent care facility.
§170.315(b)(9) Clinical information reconciliation and incorporation
Task 3: Review a complex medication list based on information provided by the patient .
§170.315(a)(6) Medication list
Task 4: Add an implantable device to the patient’s chart using information provided by the patient .
§170.315(a)(14) Implantable device list
Task 5: Determine if the patient has any active devices on the implantable device list that may be contributing
to symptoms.
§170.315(a)(14) Implantable device list
Task 6: Indicate that the patient has declined the intervention suggested by the system’s clinical decision
support.
§170.315(a)(8) Clinical decision support
Scenario 2: Robin is a 50-year-old who has been admitted for diabetic ketoacidosis.
Task 7: Update gender identity as reported by the patient.
§170.315(a)(5) Demographics
Task 8: Reconcile allergy information from an outside facility.
§170.315(b)(9) Clinical information reconciliation and incorporation
Task 9: Add a patient-reported medication to the medication list.
§170.314(a)(7) Medication list
Task 10: Update the information in a patient’s implantable device list.
§170.315(a)(14) Implantable device list
Task selection was based on criticality of function and the risk analysis described in sections 3.3.2 and 3.3.3
below.
3.3.2 Risk Analysis Framework
Risk assessment for each task involves assessing the Likelihood of Risk Occurrence and the Significance of
Risk Materialization (referred to as Likelihood and Significance, respectively). Overall risk for each task is then
categorized as High, Moderate, Low, or Negligible.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 22
Likelihood is determined by a combination of two factors: Frequency of Workflow and Possibility of
Alternative Outcome. Frequency of Workflow reflects the general prevalence of a specific or closely analogous
workflow. Possibility of Alternative Outcome is an assessment of the likelihood that a variation might occur
during a specific or closely analogous workflow. The way in which Frequency of Workflow and Possibility of
Alternative Outcome contribute to Likelihood is outlined in Appendix 6.
Significance is the measurement of the impact of possible outcomes that result from a variation from an
expected task workflow. Impact of possible outcomes of each task was determined in consult with clinicians
and other subject matter experts.
Figure 2 lists representative factors used to evaluate Likelihood and Significance. Human and system factors
that affect the evaluation of each component were discerned from conceptual and historical analysis and
include, but are not limited to, those that appear in Figure 2.3
Fig. 2: Factors which Influence Risk
Likelihood Significance
↑ Factors increasing Frequency of Workflow
Task involves a common sequence of events
↑ Factors increasing Possibility of Alternative Outcome
Clinician must make a decision that requires an increased cognitive load (i.e. the clinician must find information in multiple locations and must synthesize or remember high volumes of information to complete the workflow)
Clinician is limited in ability to recover from an issue in documentation or action
Workflow completes an event or otherwise closes an instance of care to any further documentation
High intrinsic complexity of information or information management
Clinician is likely to experience an interruption during the workflow
↑ Factors increasing Significance
Clinician is likely to take inappropriate action after the issue occurs
Issue affects mostly high-acuity patients, high-risk medication, or urgent/critical workflows
Affected data directly informs significant clinical decisions
3 See Beasley, J. W., Wetterneck, R. B., Temte, J., Lapin, J., Smith, P., Rivera-Rodriguez, J., & Karsh, B. (2011). Information Chaos in Primary Care: Implications for Physician Performance and Patient Safety. J Am Board Fam Med. , 24(6), 745-751. doi:10.3122/jabfm.2011.06.100255; Carayon, P. Sociotechnical systems approach to healthcare quality and patient safety. Work, 4(1). doi:10.3233/WOR-2012-0091-38Carayon; Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A., & Rivera-Rodriquez, A. (2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 53(11). doi:10.1080/00140139.2013.838643; Meeks, D. W., Takian, A., Sittig, D. F., Singh, H., & Barber, N. (2014). Exploring the sociotechnical intersection of patient safety and electronic health record implementation. Journal of the American Medical Informatics Association, 21, 2834. doi:10.1136/amiajnl-2013-001762. Epub 2013 Sep 19; O'Hara, R.,
& Et al. (2014). A qualitative study of decision-making and safety in ambulance service transitions. Health Services and Delivery Research, 2(56). doi:10.3310/hsdr02560; Vincent, C., Taylor-Adams, S., & Stanhope, N. (1995). Framework for Analysing Risk and Safety in Clinical Medicine. BMJ: British Medical Journal, 316(7138), 1154-1157; Wogalter, M. S., & Laughery, K. R. (1996). WARNING! Sign and Label Effectiveness. Current Directions in Psychological Science. doi:10.1111_1467-8721.ep10772712
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 23
↓ Factors decreasing Frequency of Workflow
Task involves an unusual sequence of events
↓ Factors decreasing Possibility of Alternative Outcome
Clinician does not make decisions in the course of the workflow or has robust decision-making support within the workflow
Clinician finds information in one location, or the workflow requires low degree of information synthesis or memory
Clinician has ample opportunity to recover from an issue in documentation or action
Task or workflow occurs in the midst of an event; documentation can easily be edited or added
Low intrinsic complexity of information or information management
Clinician is unlikely to experience an interruption during the workflow
↓ Factors decreasing Significance
Clinician is likely to take appropriate action regardless of the issue
Issue affects mostly low-acuity patients, low-risk medications, or non-urgent/non-critical workflows
Affected data does not inform significant clinical decisions
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 24
Figure 2 illustrates how Likelihood and Significance contribute to overall risk. Significance is weighted more
heavily than Likelihood when completing the overall risk assessment for each task. The relationship between
Significance and Likelihood was created in consult with literature on risk evaluation.4
Fig. 2: Overall Risk Assessment
Significance
Negligible Low Moderate High
Like
lih
oo
d
High Negligible Risk Moderate Risk Moderate Risk High Risk
Moderate Negligible Risk Moderate Risk Moderate Risk High Risk
Low Negligible Risk Low Risk Moderate Risk Moderate Risk
Negligible Negligible Risk Low Risk Moderate Risk Moderate Risk
3.3.3 Risk-Based Task Selection
Tasks were selected to emphasize moderate to high-risk workflows based on the framework presented
above. Risk analysis of physician and nurse tasks is summarized in Tables 3 and 4, respectively.
Table 3: Risk Analysis of Physician Tasks
Physician Task
Criteria Likelihood Significance Risk
1 Reconcile a problem from a primary care provider at an outside organization.
Clinical information reconciliation and incorporation High Moderate Moderate
2 Add a medication allergy to a patient’s allergy list and assess any interactions.
Drug-drug, drug-allergy interaction checks; Medication allergy list
Moderate High High
3 Modify an order for an inpatient diagnostic imaging procedure.
Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate
4 Place a medication order and place the consult order suggested by the system’s clinical decision support.
Computerized provider order entry - Medications; Clinical decision support
Moderate Low Moderate
4 NASA. (1994). Systems Engineering (EIA/IS-632). Electronic Industries Association (EIA); Ben-Asher, J. Z. (2004). Systems engineering aspects in theatre missile defense? Design principles and a case study. Systems Engineering. doi:10.1002/sys.10058; Ben-Asher, J. Z. (2008). Development Program Risk Assessment Based on Utility Theory. Risk Management, 10(4), 285-299. doi:10.1057/rm.2008.9; Ben-
Asher, J. Z., Zack, J., & Prinz, M. (2000). Development Program Risk Management. AIAA Progress in Aeronautics and Astronautics, 192, 341-351.; Blanchard, B. S., & Fabrycky, W. J. (1981). Systems engineering and analysis. Englewood Cliffs, NJ: Prentice-Hall.; Robertson, T. C. (Ed.). (2000). Systems Engineering Handbook (2000 ed.). INCOSE; Tummala, V. M., & Mak, C. L. (2001). A risk management model for improving operation and maintenance activities in electricity transmission networks. Journal of The Operational Research Society, 52, 125-134. doi:10.1057/palgrave.jors.2601044.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 25
Physician Task
Criteria Likelihood Significance Risk
5 Review the patient’s problem list.
Problem list Moderate Moderate Moderate
6 Modify the details of a patient-controlled analgesic medication order.
Computerized provider order entry - Medications Moderate High High
7 Modify the details of an inpatient laboratory order.
Computerized provider order entry - Laboratory Moderate Low Moderate
8 Replaced by Task 16
9 Replaced by Task 17
10 Add a problem to the problem list and place the consult order suggested by the system’s clinical decision support.
Problem list; Clinical decision support Moderate Negligible Negligible
11 Modify a medication allergy on a patient’s allergy list.
Medication allergy list Moderate Moderate Moderate
12 Place a medication order and assess any interactions.
Computerized provider order entry – Medications; Drug-drug, drug-allergy interaction checks
Moderate Moderate Moderate
13 Order an ultrasound to diagnose symptoms.
Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate
14 Resolve a problem on the patient’s problem list.
Problem list Moderate Low Moderate
15 Replaced by Task 18
16 Order an outpatient lab to monitor a discharge medication.
Computerized provider order entry – Laboratory Moderate High High
17 Electronically prescribe a discharge medication.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 26
Physician Task
Criteria Likelihood Significance Risk
Computerized provider order entry - Medications; Electronic prescribing
Moderate High High
18 Electronically prescribe a discharge medication.
Computerized provider order entry - Medications; Electronic prescribing
Moderate High High
Table 4: Risk Analysis of Nurse Tasks
Nurse Task
Criteria Likelihood Significance Risk
1 Document the patient’s preferred language.
Demographics Moderate Low Moderate
2 Reconcile allergy information from an outside urgent care facility.
Clinical information reconciliation Moderate Moderate Moderate
3 Review a complex medication list based on information provided by the patient.
Medication list Moderate Moderate Moderate
4 Add an implantable device to the patient’s chart using information provided by the patient.
Implantable device l ist Moderate Moderate Moderate
5 Determine if the patient has any active devices on the implantable device list that may be contributing to symptoms.
Implantable device l ist Low Low Low
6 Indicate that the patient has declined the intervention suggested by the system’s clinical decision support.
Clinical decision support Moderate Low Moderate
7 Update gender identity as reported by the patient.
Demographics Moderate Moderate Moderate
8 Reconcile allergy information from an outside facility.
Clinical information reconciliation Moderate Moderate Moderate
9 Add a patient reported medication to the medication list.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 27
Nurse Task
Criteria Likelihood Significance Risk
Medication list Moderate Moderate Moderate
10 Update the information in a patient’s implantable device list.
Implantable device l ist Moderate Low Moderate
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 28
3.4 Procedures
Moderators were Epic employees with experience in usability testing who underwent training specific to the
type of usability testing being conducted.
Participants arrived at the testing location where they were greeted by a moderator and oriented to the testing
computer and equipment. Participants were then assigned an alphanumeric participant identifier in order to
de-identify results. Each participant was asked for verbal consent to record the testing session.
To prepare the participant for testing, the moderator outlined the format of th e test and gave general
instructions. The moderator then began recording the session using screen capture, a microphone, and a
portable camera. Before starting the tasks, the moderator showed the participant a brief instructional recording
representative of the training typically given to users before the implementation of new functionality. Pieces
of functionality were included in the instructional recordings when research indicated that not all users were
familiar with the testing setup used, due to organizational variations in configuration. (Epic provides
organizations with a high degree of latitude in setting up the software to meet specific organizational needs.)
Physicians viewed an instructional recording on updated discharge order reconciliation and clinical
information reconciliation. Nurses viewed an instructional recording on clinical information reconciliation and
the implantable device list. At this point, the participant was given an opportunity to ask any questions or
express any concerns. The moderator continued to administer general instruction and tasks during the session.
Participants were instructed to perform tasks:
At their normal pace
Without assistance; moderators were allowed to give immaterial guidance and clarification on tasks,
but not instructions on use of the software
Withholding comments until the test was completed
Before each task, the moderator gave participants a written copy of the task and oral instruction. Task timing
began and ended when the participant reached particular predetermined points in the task.
After the participants finished the tasks, they participated in individual debrief sessions. During these sessions,
the moderators solicited feedback from participants on any areas where the participants had ex tra steps,
unnecessary steps, or incomplete tasks or subtasks. The purpose of this session was to gain additional
information about the possible causes of the unnecessary steps, extra steps, or incomplete tasks or subtasks.
3.5 Test Location
Testing was conducted at healthcare organizations where participants were employed or at the Epic campus.
Participants’ organizations provided testing rooms, typically small conference or training rooms. To ensure
that the surroundings were comfortable for participants, noise levels were kept to a minimum and the ambient
temperature kept within a normal range. See Appendix 2 for a table summarizing the testing dates and
locations.
3.6 Test Environment
EpicCare Inpatient Base is typically used in an inpatient facility. The testing was conducted at a variety of
participating organizations in rooms made available for this purpose. Testing workstations were either a
Lenovo T440P, Intel Core i5-4300M processor (2.60GHz) with 16 GB RAM or a Lenovo T450S, Intel Core i7-
5600U processor (2.60 GHz) with 12 GB RAM. Workstations of both types used either Windows 10 Enterprise
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 29
or Windows 7 Enterprise and Microsoft Internet Explorer 11 and a 23-inch flat panel display in landscape
orientation, 1600 by 900 pixel resolution, and set to thousands of colors.
The application was locally installed and used an InterSystems Caché 2017.1 database server on a loopback
connection.
The participants used a mouse and keyboard when interacting with EpicCare Inpatient Base. The overall
system performance was comparable to what users would experience in a field implementation.
3.7 Test Forms and Tools
During the usability test, the following documents were used:
Recruiting Screener
Moderator Guide
Participant Packet
The participant’s interaction with EpicCare Inpatient Base was captured and recorded digitally with screen
capture software running on the test workstation. Each participant’s facial expressions were recorded, along
with onscreen actions and verbal comments. Recordings were saved and used for further analysis.
3.8 Participant Instructions
The moderator read general introductory statements and instructions aloud to the participant before
administering the test. See Appendix 5 for the Sample Participant Orientation script.
The participant was then asked to complete a number of tasks that were read aloud by the moderator and
provided on paper to the participant for reference.
3.9 Usability Metrics
According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records ,
EHRs should support a process that provides a high level of usability for all users. The goal is for users to
interact with the system effectively, efficiently, and with high satisfaction. As such, metrics for these measures
were captured during the usability testing.
The goals of the test were to assess:
1. Efficiency of EpicCare Inpatient Base by measuring the average task time and extra steps
2. Effectiveness of EpicCare Inpatient Base by measuring task completion rates and unnecessary steps
3. Satisfaction with EpicCare Inpatient Base by measuring Ease of Task Completion ratings
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 30
Data Scoring
Table 5 details how metrics measuring efficiency, effectiveness, and satisfaction were scored.
Table 5: Usability Scoring Metrics
Measures Rationale and Scoring
Efficiency:
Average Task Time
Each task is timed from a predetermined starting point until the participant reaches the predetermined end point for the task or verbalizes completion.
Only task times for tasks that were successfully completed are included in the average task time analysis and standard deviation (reported in parentheses in the table below).
Efficiency:
Average Extra Steps per Task
The number of steps in a participant’s path through the application is recorded and compared to the number of steps in the closest acceptable path. An extra step is recorded if the participant performs a step that is not included in the defined path but is also not counterproductive to completing the task.
The total number of steps taken by a participant is counted and the difference between the steps in their path and the closest defined path is calculated. The average of the differences for the participants is calculated. Only extra steps for tasks that were successfully completed are included in the average extra steps per task analysis and standard deviation.
Effectiveness:
Binary Task Completion Rate
A task is considered a success if the participant achieves the defined task outcome without assistance.
The total number of successes was calculated for each task and then divided by the total number of times that the task was attempted. The results are presented as a percentage.
The task failure percentage can be calculated by subtracting the binary task completion rate from 100.
Effectiveness:
Partial Task Completion Rate
A task is considered 100 percent completed if the participant achieves the defined task outcome without assistance. For participants unable to successfully complete a task, the number of steps completed are counted and divided by the number of steps in the closest defined path to calculate the percentage of the task the participant completed. These results are, in turn, added together and divided by the number of participants who attempted the task to obtain the average partial task completion rate.
Effectiveness:
Average Unnecessary Steps per Task
Unnecessary steps are recorded each time a participant performs an action in the system that is not his intended action. Examples of unnecessary steps include typing mistakes and errant clicks that do not contribute to the completion of the task. The total number of unnecessary steps is calculated and divided by the number of participants to obtain the average number of unnecessary steps committed per participant.
Only unnecessary steps for tasks that were successfully completed are included in the average unnecessary steps per task analysis and standard deviation.
Satisfaction:
Ease of Task Completion Rating
The participant’s subjective impression of the ease of use of the application is recorded for each task. After each task was completed, the moderator asked the participant to rate the task on a 5-point Likert scale: 1 (Very Difficult), 2 (Somewhat Difficult), 3 (Neither Difficult nor Easy), 4 (Somewhat Easy), 5 (Very Easy).
These values are averaged across participants for each task with the calculated standard deviation reported in parentheses in the table in the Data Analysis and Reporting section for each criterion.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 31
4 Results The usability testing results for EpicCare Inpatient Base are detailed below (see Tables 6-17). Results are
organized and analyzed by criteria. The results were calculated according to the methods specified in section 3.9
Usability Metrics.
4.1 §170.315(a)(9) Clinical Decision Support
4.1.1 Data Analysis and Reporting
Table 6: Clinical Decision Support Physician and Nurse Task Results
N=Number of participants
4.1.2 Discussion of the Findings
Clinical decision support testing with inpatient physicians covered two tasks:
Placing a consult order based on a medication order (Task 4)
Placing a consult order based on problem list documentation (Task 10)
Testing with inpatient nurses covered one task:
Indicating that the patient declines intervention (Task 6)
Efficiency
Physicians completed Task 4 in an average of 8.38 seconds with a negligible number of extra steps. (Negligible
is hereafter defined as within one confidence interval of zero.) Physicians completed Task 10 in an average of
5.61 seconds. Nurses completed Task 6 in an average of 13.95 seconds.
Clinical Decision Support
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Placing a consult order based on a
medication order
57 8.38 (5.65) 0.05 (0.23) 98 (2) 99 (1) 0.02 (0.13) 4.75 (0.54)
Placing a consult order based on
problem list documentation
57 5.61 (3.30) 0.07 (0.26) 100 (0) 100 (0) 0 (0) 4.77 (0.46)
Nurse Subtasks
Indicating that the patient declines
intervention
96 13.95
(11.13)
0.05 (0.23) 97 (2) 98 (2) 0.01 (0.10) 4.46 (0.79)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 32
Effectiveness
The binary task completion rates for Task 4 and 10 are 98 percent and 100 percent respectively, and physicians
performed a negligible number of unnecessary steps for Task 4 and no unnecessary steps for Task 10. The binary
task completion rate for Task 6 is 97 percent, and nurses performed a negligible number of unnecessary steps.
Satisfaction
The majority of physicians rated both tasks as Very Easy with scores of 4.75 and 4.77 for Task 4 and Task 10,
respectively. The average satisfaction rating for nurse Task 6 is between Somewhat Easy and Very Easy with a
score of 4.46.
Major Findings
Physicians effectively performed tasks with few or no unnecessary steps and high satisfaction rates. Nurses also
completed the clinical decision support task with negligible unnecessary steps, though physicians' satisfaction
ratings were slightly higher than nurses' ratings overall. Physicians tend to interact with decision support
functionality more frequently, so this familiarity could explain the variation.
Extra steps were observed involving acknowledgement reasons, which are used to provide a reason why a
clinician is dismissing decision support suggestions. Two participants viewed the acknowledgement reasons list
without selecting a reason, and one participant entered an acknowledgement reason but also accepted the
suggested decision support action.
Areas for Improvement
Overall, participants' effectiveness and high satisfaction ratings indicate decision support is easy to use and
understand. Additional study of the wording and display of acknowledgement reasons could potentially lead
to improvements in efficiency by clarifying the intended purpose of acknowledgement reasons.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 33
4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation
4.2.1 Data Analysis and Reporting
Table 7: Clinical Information Reconciliation and Incorporation Physician and Nurse Task Results
4.2.2 Discussion of the Findings
Clinical information reconciliation and incorporation testing with inpatient physicians covered one task:
Reconciling a problem from a primary care provider at an outside organization (Task 1)
Testing with inpatient nurses covered two tasks:
Reconciling allergy information from an outside urgent care facility (Task 2)
Reconciling allergy information from a primary care provider at an outside facility (Task 8)
Efficiency
Physicians completed Task 1 in an average of 18.06 seconds, with 52 out of 55 participants completing the task
with no extra steps. Nurses completed Task 2 and Task 8 in an average of 13.77 seconds and 9.02 seconds,
respectively.
Effectiveness
Physicians performed Task 1 with a binary task completion rate of 98 percent. Nurses performed Task 2 and
Task 8 with a binary task completion rate of 95 percent and 97 percent, respectively, with a negligible number
of unnecessary steps.
Clinical Information Reconciliation and
Incorporation
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Reconciling a problem from an outside organization
55 18.06 (19.24)
0.19 (0.43) 98 (2) 99 (2) 0.06 (0.24) 4.45 (0.90)
Nurse Subtasks
Reconciling an allergy from an outside urgent care facility
100 13.77 (15.34)
0.15 (0.38) 95 (2) 97 (2) 0.01 (0.15) 4.55 (0.90)
Reconciling an allergy from an outside facility
100 9.02 (7.60)
0.04 (0.20) 97 (2) 99 (1) 0.01 (0.10) 4.70 (0.78)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 34
Satisfaction
The average satisfaction rating for physician Task 1 is between Somewhat Easy and Very Easy with a score of
4.45. The majority of nurses rated both tasks as Very Easy with scores of 4.55 and 4.70 for Task 2 and Task 8,
respectively.
Major Findings
Physicians and nurses completed the tasks with few or negligible unnecessary steps and rated tasks as
Somewhat Easy to Very Easy on average. Extra steps were observed when participants navigated to other
areas of the chart to view outside information before pursuing the task workflow. This could be explained by
variations in workflow configuration for how users access outside information at participants' organizations.
Areas for Improvement
Epic 2018 includes visual updates to the clinical information reconciliation activity that participants were using
for the first time during the study. Participants completed all tasks with average ease of task completion ratings
of Somewhat Easy to Very Easy, though nurses rated the tasks somewhat easier than physicians. Results
suggest that improvements could be made to efficiency by addressing variations in workflow configuration.
Further study could explore ways to bolster the consistency of user experience among various organizations.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 35
4.3 §170.315(a)(1) Computerized Provider Order Entry – Medications
4.3.1 Data Analysis and Reporting
Table 8: Computerized Provider Order Entry – Medications Physician Task Results
Computerized Provider Order Entry - Medications
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean
(seconds) (SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)
5 = very easy
Physician Subtasks
Placing a medication order for nicotine replacement therapy
58 22.14 (10.85)
0 (0) 100 (0) 100 (0) 0.09 (0.29) 4.72 (0.59)
Modifying the details of a PCA medication order
58 21.93 (17.79)
0.09 (0.30) 98 (2) 99 (1) 0.05 (0.23) 4.67 (0.63)
Placing a medication order and assessing any interactions
55 12.51 (20.38)
0.18 (0.43) 100 (0) 100 (0) 0.09 (0.30) 4.62 (0.76)
Ordering a PPI for discharge 33 11.12 (12.48)
0.00 (0) 100 (0) 100 (0) 0.06 (0.25) 4.88 (0.33)
Ordering an electrolyte for discharge 35 15.94 (20.41)
0.03 (0.17) 100 (0) 100 (0) 0.00 (0) 4.86 (0.36)
4.3.2 Discussion of the Findings
Computerized provider order entry of medications testing with inpatient physicians covered five tasks:
Placing a medication order (Task 4: Subtask A)
Modifying the details of a PCA medication order (Task 6)
Placing a medication order and assessing any interactions (Task 12: Subtask A)
Ordering a discharge medication (Task 17: Subtask A)
Ordering a discharge medication (Task 18: Subtask A)
Efficiency
Medication ordering tasks have an average task time of 16.73 seconds. Tasks 4 and 6 have above average task
times, which is expected due to the additional complexity inherent in those tasks. Task 4 includes decision
support to select additional orders, and Task 6 includes a PCA order, which has multiple dose components.
Effectiveness
Tasks 4, 12, 17, and 18 all have perfect binary task completion rates. Task 6 has a binary task completion rate of
98 percent and negligible unnecessary steps.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 36
Satisfaction
Each task was rated Very Easy by a majority of participants, indicating a high degree of satisfaction with
medication ordering. The average task ease rating ranges from 4.62 to 4.88.
Major Findings
Four of the five tasks were completed with 100 percent task completion. The relatively short task time of Task
12 demonstrates the benefits of ordering medications from an order set, especially when drug-allergy
interactions are triggered, because they provide inline warnings and quick selection of substitute medications.
Areas for Improvement
Overall, physicians were very successful ordering medications. The effectiveness metrics for Task 6 highlight
PCA medications as a possible opportunity for improvement. PCA orders are a complex area of patient care that
are vital for pain management and require providers to specify multiple interrelated parameters, such as patient
bolus dose and basal rate. The system assists providers with discrete dosing parameter fields and dose warnings.
Additional study could explore ways to clarify the interactions between the various parameters in a dynamic,
intuitive display.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 37
4.4 §170.315(a)(2) Computerized Provider Order Entry – Laboratory
4.4.1 Data Analysis and Reporting
Table 9: Computerized Provider Order Entry – Laboratory Physician Task Results
Computerized Provider Order Entry - Laboratory
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean
(seconds) (SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)
5 = very easy
Physician Subtasks
Modifying the details of an inpatient lab order
58 9.84 (4.98) 0.03 (0.19) 100 (0) 100 (0) 0.02 (0.13) 4.86 (0.40)
Ordering an outpatient lab to monitor a discharge medication
33 12.48 (16.15)
0.00 (0) 100 (0) 100 (0) 0.06 (0.25) 4.76 (0.61)
4.4.2 Discussion of the Findings
Computerized provider order entry of laboratory tests with inpatient physicians covered two tasks:
Modifying the details of an inpatient lab order (Task 7)
Ordering an outpatient lab to monitor a discharge medication (Task 16)
Efficiency
Task 7 has an average task time of 9.84 seconds and negligible extra steps. All participants completed Task 16
without any extra steps. The average task time for Task 16 is 12.48 seconds.
Effectiveness
Both Tasks 7 and 16 have negligible unnecessary steps and perfect task completion .
Satisfaction
Each task was rated Very Easy by a majority of participants, indicating a high degree of satisfaction.
Major Findings
Task 16 was participants' first time interacting with an updated user interface for discharge ordering as part of
the August 2018 release. All participants were able to complete the task successfully, indicating that the new
user interface is highly intuitive.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 38
Areas for Improvement
Results indicate that users are easily able to enter laboratory orders, which may be due to the consistency with
other ordering workflows. As healthcare researchers continue to develop new best practices, we will monitor
this area for further opportunities for improvement with a particular focus on the following areas:
Incorporating decision support more seamlessly at the point of ordering
Preventing duplicate laboratory orders by displaying results from outside organizations
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 39
4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging
4.5.1 Data Analysis and Reporting
Table 10: Computerized Provider Order Entry – Diagnostic Imaging Physician Task Results
Computerized Provider Order Entry – Diagnostic Imaging
Efficiency Effectiveness Satisfaction
N Average
Task Time
Average Extra
Steps per Task
Binary Task
Completion Rate
Partial Task
Completion Rate
Average
Unnecessary Steps per Task
Task Ease
Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Modifying a diagnostic imaging
procedure
57 20.56
(9.37)
0.02 (0.13) 100 (0) 100 (0) 0.05 (0.23) 4.79 (0.45)
Ordering a diagnostic imaging
procedure
58 14.74
(7.99)
0.02 (0.13) 100 (0) 100 (0) 0.09 (0.29) 4.83 (0.53)
4.5.2 Discussion of the Findings
Computerized provider order entry testing for diagnostic imaging tests with inpatient physicians covered two
tasks:
Modifying a diagnostic imaging procedure to include a comment (Task 3)
Ordering a diagnostic imaging procedure (Task 13)
Effectiveness
Tasks 3 and 13 both have negligible extra steps. Task 3 has a slightly longer task time, which is expected
because the task required the participant to type a free-text comment instead of completing discrete fields.
Effectiveness
Both tasks have a perfect task completion rate. Task 3 has a negligible number of extra steps. Most of the extra
steps performed during Task 13 were typing errors while searching for the procedure that were quickly
corrected.
Satisfaction
Both tasks were rated Very Easy by a majority of participants.
Major Findings
All of the imaging order tasks were completed with high levels of effectiveness, efficiency, and satisfaction,
suggesting that physicians are comfortable using this functionality.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 40
Areas for Improvement
Physicians successfully completed all imaging orders tasks. Diagnostic image ordering continues to be an
important topic with implications for patient care and the financial health of organizations. Although the test
results did not indicate specific areas for improvement, as new industry practices arise, future testing should
focus on the following areas:
Decision support for Appropriate Use Criteria
Display of radiation exposure information
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 41
4.6 §170.315(a)(5) Demographics
4.6.1 Data Analysis and Reporting
Table 11: Demographics Nurse Task Results
Demographics
Efficiency Effectiveness Satisfaction
N Average
Task Time
Average Extra
Steps per Task
Binary Task
Completion Rate
Partial Task
Completion Rate
Average
Unnecessary Steps per Task
Task Ease
Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD)
Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Documenting preferred language 98 17.95 (21.21)
0.24 (0.49) 89 (4) 89 (3) 0.01 (0.11) 4.30 (1.09)
Updating sexual orientation and gender identity
99 9.44 (19.58)
0.10 (0.32) 99 (1) 99 (1) 0.07 (0.27) 4.56 (0.81)
4.6.2 Discussion of the Findings
Demographics testing with inpatient nurses covered two tasks:
Documenting the patient’s preferred language (Task 1)
Updating sexual orientation and gender identity as reported by the patient (Task 7)
Efficiency
Participants who completed Tasks 1 and 7 did so with an average time between 9.44 and 17.95 seconds.
Effectiveness
Participants completed Task 1 with negligible number of unnecessary steps and a binary completion rate of 89
percent. Participants completed Task 7 with a 99 percent binary completion rate and 94 out of 99 participants
completed the task without unnecessary steps.
Satisfaction
The majority of participants rated Task 1 Very Easy. Task 7, which a majority of participants also rated Very
Easy, was rated slightly easier overall.
Major Findings
Task 7 received generally high usability metrics, with a satisfaction rating of Very Easy and a 99 percent binary
task completion rate. This indicates a strong user experience, even with the relatively recent addition of gender
identity documentation at most organizations.
The form used to document information for Task 1 has been updated for users to collect more data about social
determinants of health to create a more comprehensive health record. Some participants noted the additional
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 42
fields and visual differences. Comments from participants also indicate that many organizations document
preferred language in a different area of the patient’s chart, which could explain the extra steps taken by some
participants, the slightly lower success rate, and the amount of time taken to complete Task 1. A majority of
participants rated the task as Very Easy, which indicates general satisfaction with the task.
Areas for Improvement
As we continue to update the demographics documentation tool to include more detailed information about
social determinants of health, just-in-time training could help orient users to the updated functionality. Further
study to determine the optimal arrangement of language documentation within other demographics sections
could be beneficial.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 43
4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks
4.7.1 Data Analysis and Reporting
Table 12: Drug-Drug and Drug-Allergy Interaction Checks Physician Task Results
Drug-Drug, Drug-Allergy Interaction Checks
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Adding a medication allergy and assessing any interactions
57 13.16 (11.37)
0.16 (0.40) 100 (0) 100 (0) 0 (0) 4.40 (0.73)
Ordering an inpatient medication and assessing any interactions
55 12.57 (20.44)
0.07 (0.27) 98 (2) 99 (2) 0.02 (0.14) 4.62 (0.76)
4.7.2 Discussion of the Findings
Drug-drug and drug-allergy interaction checks testing with inpatient physicians covered two tasks:
Adding a medication allergy and assessing any interactions (Task 2: Subtask B)
Ordering an inpatient medication and assessing any interactions (Task 12: Subtask B)
Efficiency
On average, participants completed each of the three tasks within 12 or 13 seconds. Four out of 57 participants
took one extra step for Task 2, and two participants took extra steps while completing Task 12. The majority of
participants who took an extra step spent time viewing and selecting interaction override reasons.
Effectiveness
Participants had a 100 percent binary task completion rate for Task 2 and a 98 percent binary task completion
rate for Task 12. Participants took no unnecessary steps while completing Task 2, and some participants took a
negligible number of unnecessary steps while completing Task 12.
Satisfaction
The majority of participants rated Task 2 as Somewhat Easy or Very Easy. Part icipants rated Task 12 between
Somewhat Easy and Very Easy, with the majority of participants rating each task as Very Easy.
Major Findings
Physicians completed Tasks 2 and 12 with high effectiveness, and spent a consistent amount of time on each
task. The satisfaction score for Task 2 was slightly lower than the score for Task 12. However, all participants
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 44
completed Task 2 successfully. Participants took no or a negligible number of unnecessary steps for each task.
Of the participants who took extra steps, most were looking for override reasons specific to their organization.
Areas for Improvement
The efficiency scores indicate that approximately 5 percent of physicians who completed the tasks spent extra
time reviewing and documenting override reasons. Simplifying the selection and documentation screen for
overriding drug-drug and drug-allergy interactions could reduce the number of extra steps taken by physicians
and improve physician satisfaction.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 45
4.8 §170.315(b)(3) Electronic Prescribing
4.8.1 Data Analysis and Reporting
Table 13: Electronic Prescribing Physician Task Results
4.8.2 Discussion of the Findings
Electronic prescribing (e-prescribing) testing with inpatient physicians covered two tasks:
E-prescribing a PPI for discharge (Task 17: Subtask B)
E-prescribing an electrolyte for discharge (Task 18: Subtask B)
Efficiency
Participants took an average of 23.80 seconds to complete Task 17 and an average of 30.58 seconds to complete
Task 18. No participants took extra steps while completing Task 17, and a negligible number of participants took
extra steps during Task 18.
Effectiveness
Thirty out of 32 participants completed Task 17 successfully, with a negligible amount of unnecessary steps.
Thirty-three out of 35 participants successfully completed Task 18, and the majority of unnecessary steps taken
during the task were typing errors that were quickly corrected.
Satisfaction
The majority of participants rated both tasks as Very Easy, indicating a high level of satisfaction with electronic
prescribing.
Major Findings
The partial task completion rate for both Tasks 17 and 18 is 98 percent compared to the 94 percent binary task
completion rate. The two participants who did not complete the tasks were able to queue the expected order and
Electronic Prescribing
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
E-prescribing a PPI for discharge 32 23.80 (13.02)
0.00 (0) 94 (0.05) 98 (0.03) 0.07 (0.26) 4.88 (0.34)
E-prescribing an electrolyte for discharge
35 30.58 (22.51)
0.03 (0.17) 94 (0.04) 98 (0.02) 0.00 (0) 4.86 (0.36)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 46
complete the tasks up to the point of signing the order. In an actual clinical setting, clinical decision support
would prompt the user to sign orders before the discharge ordering process is complete.
Areas for Improvement
Further study is needed that more accurately simulates the clinical setting to ensure that the full e-prescribing
workflow can easily be complete in a real-world context.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 47
4.9 §170.315(a)(14) Implantable Device List
4.9.1 Data Analysis and Reporting
Table 14: Implantable Device List Nurse Task Results
Implantable Device List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Adding a historically implanted device 99 51.67 (20.98)
0.07 (0.27) 99 (1) 99 (1) 0.19 (0.44) 4.65 (0.58)
Verifying the information for a historically implanted device
99 16.86 (19.40)
0.04 (0.21) 95 (2) 97 (2) 0.01 (0.10) 4.61 (0.92)
Updating the information for a historically implanted device
99 7.21 (4.02)
0.02 (0.14) 100 (0) 100 (0) 0.01 (0.10) 4.88 (0.46)
4.9.2 Discussion of the Findings
Implantable device list testing with inpatient nurses covered three tasks:
Adding a historically implanted device based on information provided by the patient (Task 4)
Verifying the information for a historical entry on the implantable device list (Task 5)
Updating the information for a historical entry on the implantable device list based on information
provided by the patient (Task 10)
Efficiency
Tasks 5 and 10 have average task times of 16.86 seconds and 7.21 seconds, respectively. As anticipated, Task 4,
which involved adding a new implant to the chart, has a longer task time than the other two tasks.
Effectiveness
The binary task completion rate for Tasks 4, 5, and 10 is 99 percent, 95 percent, and 100 percent respectively.
Satisfaction
The majority of participants rated each task as Very Easy. This indicates a high degree of satisfaction with the
implantable device list functionality.
Major Findings
Epic 2018 introduced visual updates to the implantable device list functionality. Ninety-eight out of 99 users
effectively completed Task 4 by manually inputting multiple pieces of data to add an implant. All participants
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 48
successfully updated information for an existing implant. In addition, many participants indicated that they
were not familiar with this workflow prior to the testing session. This suggests that the functionality is highly
learnable.
Areas for Improvement
The task completion rates for these tasks indicates that documenting within the implantable device list is highly
usable. The slightly lower task completion rate for Task 5 suggests that reviewing existing data might be an area
for improvement. Further study could explore ways to optimize review activities and improve the findability of
less frequently used fields.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 49
4.10 §170.315(a)(8) Medication Allergy List
4.10.1 Data Analysis and Reporting
Table 15: Medication Allergy List Physician Task Results
4.10.2 Discussion of the Findings
Medication allergy list testing with inpatient physicians covered two tasks:
Adding a medication allergy and assessing any interactions (Task 2: Subtask A)
Entering new patient-reported information for an existing allergy (Task 11)
Efficiency
Physicians took an average amount of time of 24.69 seconds to add a new allergy to the patient's chart and
complete Task 2. They took an average amount of time of 9.02 seconds to update an existing allergy and complete
Task 11. Five out of 57 participants completed Task 2 with only one extra step, and participants completed Task
11 with a negligible amount of extra steps.
Effectiveness
Task 2 had a binary task completion rate of 96 percent and a partial task completion rate of 99 percent. Task 11
had a binary task completion rate of 100 percent, and 54 out of 58 participants completed the task with no
unnecessary steps. The majority of unnecessary steps taken during Task 2 were minor typing errors that were
quickly corrected.
Satisfaction
The majority of participants rated Task 2 as Somewhat Easy or Very Easy and Task 11 as Very Easy. The lower
satisfaction rating for Task 2 as compared to Task 11 could be attributed to the additional complexity of either
adding a new allergy or completing the drug-allergy interaction check.
Medication Allergy List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very easy
Physician Subtasks
Adding a medication allergy and assessing any interactions
57 24.69 (10.83)
0.09 (0.30) 96 (3) 99 (1) 0.11 (0.33) 4.35 (0.79)
Entering new information for an existing allergy
58 9.02 (6.53) 0.03 (0.19) 100 (0) 100 (0) 0.07 (0.26) 4.86 (0.40)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 50
Major Findings
All participants successfully completed Task 11. In Task 2, all participants were able to add the new allergy to
the chart; however, two out of 57 participants did not document that the patient’s reaction was a rash. Task 11
had a high satisfaction rating of 4.86, and Task 2, which also included a drug-allergy interaction check, was rated
4.35 on the ease of task completion scale.
Areas for Improvement
The ease of task completion ratings for Task 2 indicate that drug-allergy interaction checking is an area in which
we could improve physician satisfaction. Improvements to drug-allergy interaction checks are discussed in
section 4.7 §170.315(a)(4)Drug-Drug, Drug-Allergy Interaction Checks.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 51
4.11 §170.315(a)(7) Medication List
4.11.1 Data Analysis and Reporting
Table 16: Medication List Nurse Task Results
4.11.2 Discussion of the Findings
Medication list testing with inpatient nurses covered two tasks:
Reviewing a complex medication list based on information provided by the patient (Task 3)
Adding a patient-reported medication (Task 9)
Efficiency
Participants completed Tasks 3 and 9 with high levels of efficiency. A negligible number of extra steps were
taken during Task 9, and three out of 99 participants took only one extra step during Task 3.
Effectiveness
When reviewing a complex medication list, the majority of participants completed the task with no unnecessary
steps, and 6 percent of participants took only one unnecessary step. When adding a patient-reported medication,
two out of 100 participants took one unnecessary step.
Satisfaction
Tasks 3 and 9 were given task ease ratings of 4.52 and 4.63 respectively, indicating that the majority of
participants found both of the tasks very easy.
Major Findings
For Tasks 3 and 9, the majority of participants reviewed and added a medication to the medication list efficiently
and effectively. All participants added the correct medication to the patient's chart in Task 9. The few participants
who did not complete Tasks 3 or 9 did not document the last taken date for prior-to-admission medications.
Medication List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Nurse Subtasks
Reviewing a complex medication list 99 36.60 (25.75)
0.03 (0.18) 94 (3) 94 (2) 0.06 (0.25) 4.52 (0.75)
Adding a patient-reported medication 100 36.96
(16.99)
0.01 (0.10) 95 (2) 99 (1) 0.02 (0.15) 4.63 (0.60)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 52
Areas for Improvement
Tasks 3 and 9 received high efficiency and satisfaction scores, with low percentages of participants taking an
extra or unnecessary step. The majority of participants rated Tasks 3 and 9 as Very Easy, demonstrating high
satisfaction. The main area for improvement is to better prompt users to document the last taken date for prior-
to-admission medications. The effectiveness of this workflow could be improved by enhancing visual indicators
to highlight this outstanding documentation.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 53
4.12 §170.315(a)(6) Problem List
4.12.1 Data Analysis and Reporting
Table 17: Problem List Physician Task Results
Problem List
Efficiency Effectiveness Satisfaction
N Average Task Time
Average Extra Steps per Task
Binary Task Completion
Rate
Partial Task Completion
Rate
Average Unnecessary
Steps per Task
Task Ease Rating
# Mean (seconds)
(SD)
Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very
easy
Physician Subtasks
Reviewing a patient’s problem list 58 3.05 (2.89) 0.02 (0.13) 100 (0) 100 (0) 0 (0) 4.84 (0.59)
Adding a problem to a patient’s problem list
54 15.72 (7.95)
0 (0) 100 (0) 100 (0) 0.02 (0.14) 4.78 (0.46)
Resolving a problem on a patient’s problem list
58 2.38 (1.93) 0.03 (0.19) 100 (0) 100 (0) 0 (0) 4.91 (0.28)
4.12.2 Discussion of the Findings
Problem list testing with inpatient physicians covered three tasks:
Reviewing a patient’s problem list (Task 5)
Adding a problem to a patient’s problem list (Task 10: Subtask A)
Resolving a problem on a patient’s problem list (Task 14)
Efficiency
Physicians completed Tasks 5, 10, and 14 with average task times of 3.05 seconds, 15.72 seconds, and 2.38
seconds, respectively. All tasks were completed with either zero or negligible extra steps.
Effectiveness
The binary task completion rate for all tasks is 100 percent. Participants completed the tasks with zero or
negligible extra steps.
Satisfaction
The majority of participants rated the tasks as Very Easy with average ease ratings of 4.84, 4.74, and 4.91 for
Tasks 5, 10, and 14, respectively.
Major Findings
Task 5 and Task 14, which involved reviewing and resolving problems from the problem list, have high ease of
completion ratings, indicating that participants could efficiently and effectively complete these tasks. The
average task time for these tasks is less than the average task time for adding a problem to the problem list in
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 54
Task 10. This is likely due to the additional complexity and the necessary clinical documentation involved in
adding a new problem compared to resolving or reviewing an existing problem. The overall completion rate of
100 percent indicates that participants were highly effective at completing these tasks.
Areas for Improvement
Overall, participants reported high satisfaction with problem list tasks, and they performed the tasks effectively.
Though the results indicate positive experiences with problem list documentation, potential areas for future
exploration include:
Integrated use of the problem list across care settings
Development and progression of problems over time
Cooperative documentation by all members of a patient care team
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 55
5 Results Conclusion Physicians and nurses performed tasks that were selected to emphasize areas of Moderate to High risk (see
section 3.3.3 for details). Common areas for improvement include:
Clarifying expected fields to complete
For various criteria, efficiency was impacted by the participant's uncertainty about which fields
needed to be updated to complete the workflow. Although indicators for required and
recommended fields are already used in many places throughout the system, future research
could explore additional ways to clarify whether a response is expected in a given field in a
specific situation. Improvements in this area could support the usability heuristic of "Recognition,
not Recall" and reduce users’ cognitive load.
Bolstering onboarding techniques
EpicCare Inpatient is continually being improved and enhanced to assist users in providing the
highest level of patient care. Incorporating training into the user interface is an important way to
disseminate tips and information so that users can quickly adjust to and make use of new
features. This study highlighted potential areas to further improve onboarding techniques to
make the transition to new and updated activities smoother for users, as observed in sections 4.2,
4.3, 4.6, and 4.9.
The overall average number of extra steps is 0.06 per task, the average binary task completion rate is 98 percent,
and the average ease of task completion rating is 4.69, indicating that tasks were generally completed with a
high degree of efficiency, effectiveness, and satisfaction.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 56
Appendices
Appendix 1
Recruiting Screener
Note: Italicized text indicates information used for internal determination of eligibility and was not included
on the copy given to the participant.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 57
Recruiting Screener Demographic Information
1. Name:
2. Credentials:
3. Highest Level of Education:
a. High school graduate/GED
b. Some college
c. College graduate
d. Postgraduate
e. Other
4. Organization:
5. Primary Work Location:
6. Contact method (please provide one of the following):
a. Work phone:
b. Cell phone:
c. Email:
7. What is your gender?
a. Male
b. Female
c. Other (please specify):__________________
8. Which of these best describes your current age?
a. <20
b. 20-29
c. 30-39
d. 40-49
e. 50-59
f. 60-69
g. 70-79
h. ≥80
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 58
Additional Information
9. Is English your first language?
a. Yes
b. No
10. Are you a fluent English speaker? [if No, disqualify]
a. Yes
b. No
11. Due to logistical restraints and the parameters of this study, we cannot provide assistive technologies
during the testing session. Do you require any assistive technologies to use a computer? [if Yes,
disqualify]
a. Yes
b. No
12. Do you, or does anyone in your household, have a commercial interest in an electronic health record
software or consulting company? [if Yes, disqualify]
a. Yes
b. No
13. How many years of experience do you have using computers for personal and professional activities
(such as email, shopping, record keeping, etc.)?
a. <5 years
b. 5-10 years
c. 10-20 years
d. >20 years
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 59
14. What is your current role? [if not Nurse or Physician, disqualify]
a. Analyst
b. Application Coordinator
c. Certified Nursing Assistant (CNA)
d. CEO, CMIO, CIO, etc.
e. Consultant
f. Director
g. Information Technology
h. Licensed Practical Nurse (LPN)
i. Marketing/Communications
j. Medical Assistant (MA)
k. Nurse
l. Nurse Practitioner (NP)
m. Office Manager
n. Pharmacist
o. Physician
p. Physician Assistant (PA)
q. Project Manager
r. SVP, AVP, VP, etc.
s. Trainer
t. Other (please specify)
15. Do you currently provide direct patient care? [if No, disqualify]
a. Yes
b. No
16. In which setting do you primarily work? [if Ambulatory or Emergency Department, disqualify]
a. Inpatient
b. Emergency Department
c. Ambulatory
17. What is your specialty? [if role is Physician and specialty is Radiology, Ophthalmology or Pathology,
disqualify]
18. How many years have you been working in your field?
a. <5 years
b. 5-10 years
c. 10-20 years
d. >20 years
19. Have you participated in Epic usability testing previously?
a. Yes
b. No
If yes, please describe.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 60
20. How do you capture patient data in your organization? [if Primarily on paper, disqualify]
a. Primarily on paper
b. Primarily electronically
21. Is Epic the EHR you use most often in your organization? [if No, disqualify]
a. Yes
b. No
22. How long have you been using Epic? [if <3 months, disqualify]
23. How frequently do you use Epic? (daily, weekly, monthly)
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 61
Appendix 2
Testing Dates and Locations
Test Dates Locations
1 February 20, 2018 Dallas, TX
2 February 21, 2018 Dallas, TX
3 February 21, 2018 Baltimore, MD
4 February 21, 2018 Walnut Creek, CA
5 February 21, 2018 Concord, CA
6 February 21, 2018 Mountain View, CA
7 February 22, 2018 Dallas, TX
8 February 22, 2018 Fort Worth, TX
9 February 22, 2018 Baltimore, MD
10 February 22, 2018 Concord, CA
11 February 22, 2018 San Pablo, CA
12 February 22, 2018 Martinez, CA
13 February 22, 2018 Rockford, IL
14 February 23, 2018 Dallas, TX
15 February 28, 2018 Skokie, IL
16 March 1, 2018 Chicago, IL
17 March 1, 2018 New Orleans, LA
18 March 1, 2018 Lancaster, PA
19 March 6, 2018 Denver, CO
20 March 7, 2018 Centennial, CO
21 March 7, 2018 New York, NY
22 March 8, 2018 New York, NY
23 March 9, 2018 New York, NY
24 March 21, 2018 Verona, WI
25 March 27, 2018 Wauwatosa, WI
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 62
26 March 27, 2018 Springfield, MO
27 March 27, 2018 Kalamazoo, MI
28 March 28, 2018 Toledo, OH
29 March 28, 2018 Columbus, OH
30 August 6, 2018 Verona, WI
31 August 7, 2018 Verona, WI
32 August 14, 2018 Verona, WI
33 August 15, 2018 Verona, WI
34 August 27, 2018 Verona, WI
35 August 28, 2018 Verona, WI
36 August 29, 2018 Verona, WI
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 63
Appendix 3
Participant Demographics
Following is a high-level summary of participants in this study.
Gender Men 86 Women 110 Total (participants) 196
Occupation/Role RN/BSN 101 Physician 95 Total (participants) 196
Years of Experience Years of experience with Epic (average)
4.66
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 64
Appendix 4
IP Physician Session 1
IP Physician Scenario 1
Your first patient is Gertrude. Gertrude is a 55-year-old female who has been admitted to your unit from the
ED after experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition,
abrasions, and a possible concussion.
Task 1:
Gertrude mentions that she saw her PCP at an outside organization, River Hills Health. She remembers that
he diagnosed her with something new. Reconcile Gertrude’s problem list by adding and accepting the
information from River Hills Health.
Task 2:
Gertrude mentions that sulfa antibiotics give her a rash. Add sulfa antibiotics to her allergy list with a
reaction of rash. You know that any drug-allergy interaction due to her furosemide is unlikely. Use your
clinical judgment to respond to any warnings you receive.
Task 3:
Gertrude complains of pain in her right wrist and you see that it is bruised and swollen. You suspect that
she may have fractured it when she fell. The ED physician already ordered a right wrist x-ray. Update the
order with a comment to the radiologist to look for a possible scaphoid fracture. Sign the order when
complete.
Task 4:
Gertrude is struggling to quit smoking, and she says that she is experiencing symptoms of nicotine
withdrawal since being admitted. Place an order for a nicotine 21 mg patch, and based on the information
provided by the system, place and sign any recommended follow-up orders.
IP Physician Scenario 2
Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to
your unit.
Task 5:
In addition to osteoarthritis of the left knee, Sheryl mentions that her current conditions include high
cholesterol, type 2 diabetes mellitus, and atrial fibrillation . Document that you have reviewed her problem
list.
Task 6:
As you speak with Sheryl, you learn that she is still experiencing significant pain, despite constant use of her
PCA. Increase the dose of her existing HYDROmorphone (Dilaudid) PCA order slightly by decreasing the
lockout interval to 5 minutes and increasing the basal rate to 0.2 mg/hour. After modifying, sign the order.
Task 7:
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 65
There were concerns with how much blood Sheryl lost in surgery, so a CBC was ordered for every 12 hours.
Her hemoglobin levels are looking good, so the frequency of the CBC order can be decreased. Modify the
existing CBC order to have a frequency of daily. After modifying, sign the order.
Task 8:
Replaced by Task 16
Task 9:
Replaced by Task 17
IP Physician Scenario 3
Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.
He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a CVA last
night.
Task 10:
While rounding, you notice that although Arthur is stabilized, he is still exhibiting weakness of his right side.
Add right-sided weakness to his problem list. Based on the information provided by the system, place and
sign any recommended follow-up orders.
Task 11:
While reviewing Arthur’s chart, you notice that there is no reaction for his aspirin allergy. He tells you that
aspirin gave him hives. Update Arthur’s aspirin allergy with a reaction of hives.
Task 12:
Your organization has an initiative to increase antiplatelet utilization for stroke and CVA patients. You’ve
already opened the order set made to support this initiative. Using the order set and your clinical judgment,
select and sign an order for an appropriate antiplatelet therapy.
Task 13:
You decide to order a carotid ultrasound to further investigate the underlying cause of Arthur’s CVA. Place
and sign an order for a bilateral carotid ultrasound.
Task 14:
Several days have passed and Arthur is ready to be discharged. Shortly after admission Arthur was
diagnosed with hypokalemia (low potassium). He responded well to the prescribed potassium drip and
now his levels are much better. Resolve hypokalemia in the problem list.
Task 15:
Replaced by Task 18
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 66
IP Physician Session 2
IP Physician Scenario 1
Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to
your unit. Sheryl has recovered enough to be discharged.
Task 16:
You’ve already completed most of Sheryl’s discharge order reconciliation. Since she will continue her current
warfarin dose, you would like to continue monitoring Sheryl’s INR post -discharge. Enter an order for
Protime-INR. You’ll also be prescribing a medication, so wait to sign this order.
Task 17:
Sheryl mentions that she has been experiencing heartburn, and you want to prescribe her pantoprazole. Your
system automatically sends prescriptions electronically to the patient’s preferred pharmacy, which has
already been specified as Epic Apothecary Pharmacy. Place an order for 40 mg pantoprazole (Protonix). Sign
all orders when complete.
IP Physician Scenario 2
Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.
He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a CVA a few
nights ago. Several days have passed, and Arthur is ready to be discharged.
Task 18:
You’ve already completed most of Arthur’s discharge order reconciliation. His potassium levels are still a
little low, and you would like him to continue taking potassium chloride. Your system automatically sends
prescriptions electronically to the patient’s preferred pharmacy, which has already been specified as Epic
Apothecary Pharmacy. Place and sign an order for potassium chloride (K-dur).
IP Nurse Scenario 1
Your first patient is Walter. Walter is a 79-year-old male with a complex medical history that includes CHF,
osteoporosis, dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a
wound infection.
Task 1:
Walter’s preferred language is Spanish and his PCP speaks Spanish, so he has never needed an interpreter
until now. You have already documented that Walter needs interpreter services. Add Spanish as Walter’s
primary language.
Task 2:
Walter tells you that he went to an outside organization’s urgent care clinic and they determined he has a
new allergy, but he can't remember what the allergy is. You discuss the allergy with Walter and his wife and
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 67
confirm what is displayed as accurate. Reconcile Walter’s allergy list by adding and accepting the
information from River Hills.
Task 3:
Walter’s wife gives you a list of Walter’s at-home medications and states that he took them all yesterday.
Review his medications, adding or changing as needed. Indicate that Walter last took doses for all his
medications yesterday. Document that you have reviewed his medication list.
Task 4:
You’re taking Walter’s vitals when he mentions he has a pacemaker. You see that the pacemaker is not on
his Implants List. Walter does not have his pacemaker card with him but his wife gives you some details.
Add Walter’s pacemaker to the Implants List.
Task 5:
While discussing implants, Walter also mentions he had his hip replaced 6 years ago. You remember that
Depuy Orthopaedics recalled several of their hip replacement systems due to the implants shedding metal
shards. You want to determine if Walter could be affected. Review Walter’s hip implant and state verbally if
the manufacturer is Depuy Orthopaedics.
Task 6:
When giving Walter a snack, you notice he has difficulty chewing. After documenting this, you see that a
nutritional consult is recommended by the system. You discuss this with Walter, but he refuses to see a
nutritionist. Document that Walter has difficulty chewing and move to the next section. Review the
advisory given by the system and indicate that Walter refused the nutritional consult.
IP Nurse Scenario 2
Your next patient is Robin. Robin is a 50-year-old who prefers male gender pronouns. He has been admitted
for diabetic ketoacidosis.
Task 7:
Your organization has an initiative to more accurately document gender identity. There is already
documentation present for Robin and you discuss this with him. Robin informs you that while his sex
assigned at birth was female, he identifies as a man. Update this information in the chart.
Task 8:
Robin recently saw his PCP at an outside organization, River Hills Medical System, for an allergic reaction.
His PCP documented a new allergy which you discuss with Robin and confirm is accurate. Reconcile Robin’s
allergy list by adding and accepting the information from River Hills.
Task 9:
Robin mentions that he is taking Lasix (furosemide) 20 mg at home and last took it yesterday. You also
confirm he is still taking his other medications. Add Lasix to the medication list and indicate that he also
took his other medications yesterday.
Task 10:
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 68
You notice Robin touching his right knee as he mentions some tenderness. He confirms that he had a right
knee replacement last year. You review the implant and notice there is no laterality listed for it. Add the
appropriate laterality to Robin’s chart.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 69
Appendix 5
Sample Participant Orientation
1) We are not testing you or your ability to use the system. We are measuring the usability of the system
itself.
2) You will be taken to the appropriate starting point for each task.
3) You will have a written copy of the task to read.
4) Work at your normal speed and only do what you are specifically asked to do in the system.
5) Because we are testing specific pieces of functionality, you may not complete the entirety of your
normal clinical workflow with a patient.
6) There may be multiple ways to complete a task. You can complete the task in whichever way is
apparent to you or easiest for you.
7) Verbalize that you are done upon completion of each task.
8) Fill out the Ease of Task Completion rating after each task.
9) You will complete a survey about your experience after all tasks are complete.
10) At the end of the test, we may discuss your thought process during specific tasks.
11) Save your comments until all tasks are completed. The facilitator will not offer help or answer any
questions during the test.
12) All of the information you provide will be kept confidential and your name will not be associated with
the results of this session.
________________________________________________________________________________________________
© 2018 Epic Systems Corporation 70
Appendix 6
Frequency of Workflow and Possibility of Alternative Outcome
Possibility of Alternative Outcome
Negligible Low Moderate High
Fre
qu
en
cy o
f W
ork
flo
w
High Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood
Moderate Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood
Low Negligible Likelihood Low Likelihood Moderate Likelihood Moderate Likelihood
Negligible Negligible Likelihood Negligible Likelihood Low Likelihood Low Likelihood