Design and Assessment of a Mobile Health Care Solution for the Military Pediatrician: The DHA Pediatrics App
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
MILITARY MEDICINE, 187, 1/2:232, 2022
Design and Assessment of a Mobile Health Care Solution for the
Military Pediatrician: The DHA Pediatrics App
CPT Patrick T. Reeves, MD, MC, USA*,†; Maj Matthew Royall, MD, USAF, MC*,†;
COL Steven Spencer, MD, MC, USA†,‡; Dr COL Philip L. Rogers, MD, USA (Ret.)*,†;
Dr Steven J. Durning, MD, PhD§,∥; Dr Anita Samuel, PhD§,∥; CAPT Kevin O’Meara, MD, MC, USN*,†;
CAPT Gregory Gorman, MD, MC, USN†,§,¶
ABSTRACT
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
Introduction:
Mobile health technology design and use by patients and clinicians have rapidly evolved in the past 20 years. Never-
theless, the technology has remained in silos of practices, patients, and individual institutions. Uptake across integrated
health systems has lagged.
Materials and Methods:
In 2015, the authors designed a mobile health application (App) aimed at augmenting the capabilities of clinicians who
care for children within the Military Health System (MHS). This App incorporated a curated, system-based collection
of Clinical Practice Guidelines, access to emergency resuscitation cards, call buttons for local market subspecialty and
inpatient teams, links to residency academic calendars, and other web-based resources. Over the next 5 years, three Plan-
Do-Study-Act cycles facilitated multiple enhancements for the App which eventually transitioned from the Android/iOS
stores to a web browser. The “People At the Centre of Mobile Application Development” tool which has validity evi-
dence captured user experience. The team assessed the App’s global effectiveness using Google Analytics. A speed test
measured time saved and accuracy of task completion for clinicians using the App compared to non-users. Finally, MHS
medical librarians critiqued the App using a questionnaire with validity evidence. The Walter Reed National Military
Medical Center Institutional Review Board reviewed the study and deemed it exempt.
Results:
Clinician respondents (n = 68 complete responses across six MTFs, 51% graduate medical trainees representing a 7.4%
response rate of active duty pediatrician forces) perceived the App to have appropriate qualities of efficiency, effec-
tiveness, learnability, memorability, errors, satisfaction, and cognitive properties following App use in clinical practice.
Google Analytics demonstrated more than 1,000 unique users on the App from May 1, 2020 to January 20, 2021. There
were 746 instances (26% of all sessions) when a user navigated between more than one military treatment facility. App
users were faster and more accurate at task completion during a digital scavenger hunt. Medical librarians measured
the App to have acceptable usefulness, accuracy, authority, objectivity, timeliness, functionality, design, security, and
value.
Conclusions:
The App appears to be an effective tool to extend a clinician’s capabilities and inter-professional communication between
world-wide users and six MHS markets. This App was designed—and used—for a large health care network across a
wide geographic footprint. Next steps are establishing an enduring chain of App champions for continued updates and
sharing the App’s code with other military medical disciplines and interested civilian centers.
INTRODUCTION
Over the past two decades, modern medicine has experienced
* Department of Pediatrics, Walter Reed National Military Medical Cen-
an evolution of smart device and mobile health (mHealth)
ter, Bethesda, MD 20814, USA technology innovation with surging use by patients.1 This
‡ Departments of Pediatrics, Brooke Army Medical Center, Fort Sam
unparalleled growth in mHealth consumption by patient users
Houston, TX 78234, USA
† Department of Pediatrics, Uniformed Services University of the Health has outpaced the uptake of these technologies by clinicians
Sciences, Bethesda, MD 20814, USA outside of the Electronic Health Record (EHR).2 With increas-
§ Department of Medicine, Uniformed Services University of the Health ing connectivity, patients and payors have a newfound expec-
Sciences, Bethesda, MD 20814, USA
∥ Center for Health Professions Education, Uniformed Services Univer-
tation that clinicians should provide evidenced-based care at a
sity of the Health Sciences, Bethesda, MD 20814, USA
more rapid pace that meets the expectations of the patient.3,4
¶ Defense Health Agency, Falls Church, VA 22042, USA These expectations, which can increase the time spent on
clinical tasks outside the examination room by upwards of
doi:https://doi.org/10.1093/milmed/usab204
3 hours per day and contribute to burnout, have been incom-
Published by Oxford University Press on behalf of the Association of
Military Surgeons of the United States 2021. This work is written by (a) US pletely addressed by the health care community at large.5
Government employee(s) and is in the public domain in the US. In short, clinicians are in need of tools which can assist in
232 MILITARY MEDICINE, Vol. 187, January/February 2022The DHA Pediatrics App
rapid care delivery, decrease the mental effort (e.g., cogni- 5 years.11 Military pediatricians also operate within an aca-
tive load) required to accomplish tasks both in and outside of demic health system, anchored by the Uniformed Services
the examination room, promote interdisciplinary communi- University, with multiple graduate and undergraduate med-
cation, and increase evidenced-based care efficiency without ical education programs.12 Like their civilian counterparts,
compromising safety. the military pediatrician is heavily tasked with clinical duties,
Recently, mHealth initiatives have shifted focus from leadership roles, education responsibilities and other expecta-
evidence-based guidelines on websites and in journals to the tions.13–15 Despite a shared EHR across the MHS and having
creation of medical applications (Apps) that meet these ongo- a robust, although and often underutilized, TeleHealth plat-
ing clinical needs of clinicians.6 Apps are portable, accessible, form, we identified common clinical problems which lead to
and update automatically with clinically relevant informa- disruption in pediatric patient care such as shared difficulty
tion available to the provider for use at the point of care. accessing specialists and services within and between markets
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
Many Apps developed and implemented to meet niche clinical (e.g., the National Capital Region versus the San Antonio Mil-
needs in pediatric medicine include reference Apps for heart itary Health System), differing sources for commonly used
rate normative values and immunization schedules, support CPGs and non-standardized operating procedures for routine
tools for scoliosis decision-making, and emergency resusci- pediatric care (e.g., infant fever pathway).16 A needs assess-
tation guidance. These tools have addressed their respective ment further informed our team that these burdens contribute
clinical problems to varying levels of success.7 Other Apps to clinician burnout throughout the MHS.17,18
have sought to support the lifelong learning of the clini-
cian by delivering timely, updated clinical practice guidelines Solution Development Strategy
(CPGs), housing digital education webinars for continuing We created a mobile App which augments the clinical care
medical education, or connecting clinicians for secure com- of military-connected children by providing the medical team
munication and structured, ad hoc peer learning.8–10 Given with point-of-care access to shared evidence-based practices
the task saturation faced by many clinicians from patient care and phone access to on-call specialists throughout the MHS.
and administrative responsibilities, the safe implementation of We followed strategies from Karsh for effective technologic
mHealth App clinical support solutions represents a potential design and implementation solutions.19 Our App development
milestone in health care innovation. At this time, a gap exists strategy involved a biweekly virtual meeting of the clini-
in the literature for assessing the effectiveness of Apps used cal team with the Solution Delivery Division (SDD) at the
by clinicians. Defense Health Agency (DHA).
Despite the mHealth resources available to clinicians, no
App designed for both communication and clinical decision General App Setup
support exists in a single platform. The varying nature of Through a targeted needs assessment, we identified the key
provider networks, mostly determined and assembled by mul- elements to form the major characteristics of the App: no cost,
tiple payors, especially limits inclusion of provider communi- reliable, accessible, user-friendly, accurate, and secure. These
cation functionality in Apps to single institutions or limited elements took several forms and functions: a provider one-call
geographic regions. Such a digital tool could represent the and paging system, a telephone/email communication direc-
next phase of mHealth evolution for clinicians seeking to tory, links to online residency academic calendars, a repos-
decrease burnout while simultaneously pursuing of a superior, itory of digitized required hospital badges, and a curated list
safer way to communicate, continue medical education, and of frequently visited pediatric website resources (e.g., ACTion
conduct evidenced-based medicine (EBM) practices. (ACT) Sheets). We also added EBM resources including pub-
We aim to describe our experience developing, dissemi- lished cCPGs organized by body system, hospital-specific
nating, and evaluating a combined reference, communication, standard operating procedures (SOPs), and emergency resus-
and clinical decision support App for a large geographically citation code cards (e.g., Pediatric Advanced Life Support
disparate military health care system. (PALS), Neonatal Resuscitation Program (NRP), Advanced
Cardiac Life Support (ACLS)).
MATERIALS AND METHODS APPROACH The SDD coded the App using the React framework. When
combined with Cordova, this allowed us to format the App ini-
Military Medicine Background
tially for Android/iOS users on the store which was eventually
Today’s military pediatrician belongs to a global health care transitioned to a web browser.
network of 450 geographically separated military treatment Following the establishment of the App’s functional frame-
facilities (MTFs) organized into markets spanning multiple work, the innovation team (P.T.R., M.R., A.S., P.L.R., and
time zones in locations ranging from large metropolitan areas G.G.) expanded to incorporate additional expertise. First, a
to remote, austere, or hostile environments. Of the 9.6 mil- medical graphic artist designed both a group of body system
lion beneficiaries cared for in the Military Health System icons (e.g., stomach and intestines for gastroenterology) to
(MHS), 1.5 million are children, 40% of whom are under age assist user navigation to CPGs, and an App logo icon for user
MILITARY MEDICINE, Vol. 187, January/February 2022 233The DHA Pediatrics App
recognition on smart devices. Second, the DHA Connected were provided a short introduction to DHAPediatricsApp via
Health user experience team reviewed and streamlined the the standardized familiarization video.20 Half of the partici-
App’s aesthetics to ensure that form followed function. We pants completed the hunt using the App and the remaining
completed the design process by delivering an App “How To participants used any means available to them to respond to
Use” and familiarization video to users.20 questions without the App.
Based on the App developmental milestones: a tar- Finally, medical librarians from across the MHS measured
geted needs assessment, key App element identification, the quality of the App using a nine-question survey rubric
functional framework design, key clinical resource acqui- from Hanrahan et al., with grades on a scale from 1 (indi-
sition, and logo/icon design, our App development strat- cating major deficiencies) to 4 (indicating no deficiencies).22
egy included a three-phase Plan-Do-Study-Act (PDSA) The categories analyzed include usefulness, accuracy, author-
process model. First, we designed the App design at our ity, objectivity, timeliness, functionality, design, security, and
value. The team defined an acceptable median score of ≥3 for
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
home institution. Next, we expanded to five sister facil-
ities and culminated in an MHS-wide user assessment of each category (pooled total score >29/36).
the DHA Pediatrics App (DHAPediatricsApp) following Survey response data were presented as medians for indi-
implementation. vidual category scores and mean ± SD for the pooled score.
Categorical comparisons used Rao-Scott Chi-square tests.
Study Methods The Walter Reed National Military Medical Center Insti-
We assessed the App’s impact on clinical team members by tutional Review Board reviewed the study and deemed it
measuring user efficiency, effectiveness, learnability, mem- exempt.
orability, errors, satisfaction, and cognitive load using the
“People At the Centre of Mobile Application Development” Results
(PACMAD, n = 24 questions, Likert scale [1—strongly dis- The “Military Pediatrics” App (Version1) first launched for
agree, 2—disagree, 3—somewhat disagree, 4—neutral, 5— the National Capital Region market on the Google and Apple
somewhat agree, 6—agree, and 7—strongly agree]) tool stores (GS/AS) in 2015 and 2018, respectively (Fig. 1).
which has validity evidence.21 The digital survey tool is repro-
duced as Supplemental Figure S1. Our team targeted an a PDSA Cycle 1 (February 1, 2019-May 1, 2019)
priori score of >5 (i.e., agree or strongly agree) for each Version2 of the App expanded the tool in its GS/AS offerings
category to be considered more than acceptable. (now tablet-enabled) and geographic scope to the San Anto-
Beyond the development of the App, our innovation nio Military Health System market after 18 members of the
project’s central goal was to broaden uptake and use of the clinical team (50% board-certified/board eligible clinicians
App by a larger group of clinicians throughout the MHS. By (BC/BE) [n = 9], 44% graduate medical trainees [n = 8], and
transitioning the App to a web browser, the SDD was able to 6% nursing team [n = 1]) indicated a significant need for
incorporate the Google Analytics tool into the App’s sustain- this tool through the use of a 5-item Likert scale survey
ability and surveillance plan. This free service allows capture (1 = Strongly Disagree; 5 = Strongly Agree) (Table I).
of in-depth details about website visitors and internal site
searches. These insights allow the team to understand the fre- PDSA Cycle 2 (July 1, 2019-April 1, 2020)
quency of use of App functions by the universe of users and Clinicians (n = 86) at four additional facilities reported their
inform follow-on strategies to maintain the longevity of the App needs using a similar 5-item Likert scale survey tool
App. (1 = Strongly Disagree; 5 = Strongly Agree) (Table I). 50%
One aim of our innovation team for the App was to have (n = 43) of respondents were graduate medical trainees, 40%
a positive impact on the individual clinician on a day-to-day (n = 34) of respondents were BC/BE clinicians, and 10%
basis. Practically, we surmised this could be accomplished (n = 9) of respondents served on the nursing team. Informa-
by decreasing the time required to complete routine clini- tion detailing the respondent health care professional type by
cal tasks (e.g., finding the telephone number for a specialist MTF can be found in Table II. Chief residents served as points
and calling) which would reduce cognitive load for the task of contact to implement feedback, enhance capabilities, and
at hand and possibly lessen burnout. We measured this out- eliminate redundancies within the App. The overhaul, with
come by designing a digital scavenger hunt (hunt) (n = 5 multiple locations providing updates, made housing of the
questions) which would assess a clinician’s speed and accu- App on GS/AS ineffective.
racy in response to relevant clinical, real-world scenarios
which routinely affect the care of children within the MHS. PDSA Cycle 3 (May 1, 2019-September 1, 2020)
Specifically, these questions relied upon the App’s major We used the React Javascript library to transition away from
functional components: the provider one-call system, the per- the GS/AS and build a user interface on a Progressive Web
sonnel directory, CPGs, SOPs, and emergency resuscitation Application (PWA) platform. The PWA receives user inputs
code cards. All respondents were new users to the App. All at a web address, pushes rich data through the App, and
234 MILITARY MEDICINE, Vol. 187, January/February 2022The DHA Pediatrics App
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
FIGURE 1. DHA pediatrics app_versions 1-3.
MILITARY MEDICINE, Vol. 187, January/February 2022 235236
TABLE I. Results of General and Targeted Needs Assessment for Application Development
Section A: PDSA Cycle 1 (February 1, 2019-May 1, 2019) Needs Assessment
MTF (number of Questions
respondents) mean (SD)
a Q1: a Q2: a Q3: a Q4: b Q5:
[YES,] Do you have a difficult [YES,] Do you have a difficult [YES,] Do you have a diffi- [YES,] Do you frequently use [YES,] If there was an App
time locating MTF SOPs? time accessing CPGs? cult time communicating a smart device at work? that could provide you these
with consultants/other team capabilities, would you use
members? it?
Mean (SD) Mean (SD) Mean (SD) Mean (SD) n (%)
SAMHS 18 3.2 (1.6) 4.2 (1.5) 3.6 (1.5) 4.1 (1.5) 17 (94)
Section B PDSA Cycle 2 (July 1, 2019-May 1, 2020) Needs Assessment
Q1: Q2: b Q3: a Q4: a Q5: a Q6: a Q7: a Q8: b Q9:
MTF, n (%) Respondents [YES,] My [YES,] I can [YES,] I find it dif- [YES,] I often [YES,] I can [YES,] I have ALL [YES,] There is
Total n = 86 to assess- PALS card is find ALL of ficult to access have a difficult confidently required hospital a current need
USU n = 1 ment that are on my person? the hospital- relevant CPGs dur- time com- tell you what badges. for this type of
trainees. specific SOPs ing my clinical municating lectures are a clinical tool.
even if the duties. with consul- scheduled on
internet sys- tants/other the academic
tem goes team calendar.
down? members?
The DHA Pediatrics App
n (%) n (%) Mean (SD) Mean (SD) Mean (SD) Mean (SD) Mean (SD) n (%)
NCR, 38 (44.2) 13 (34.2) 8 (21.1) 1.4 (1.3) 3.4 (1.7) 3.2 (1.4) 2.0 (1.9) 2.1 (1.5) 37 (97)
SAMHS, 20 11 (55) 210 1.1 (1.5) 3.3 (1.9) 3.8 (1.4) 2.0 (1.9) 3.1 (1.9) 18 (90)
(23.3)
NMCP, 5 (5.8) 5 (100) 2 (40) 2.2 (1.2) 3.6 (0.5) 3.0 (0.6) 1.4 (1.5) 3.0 (1.3) 5 (100)
NMCSD, 8 (9.3) 7 (88) 4 (50) 1.9 (1.8) 3.9 (0.6) 3.5 (1.2) 2.8 (1.9) 2.6 (1.8) 8 (100)
TAMC, 9 (10.5) 3 (33) 222 1.9 (1.0) 2.8 (0.9) 2.6 (1.0) 2.0 (1.7) 2.7 (1.5) 9 (100)
MAMC, 5 (5.8) 4 (80) 120 2.2 (1.3) 2.6 (1.5) 2.4 (1.2) 0.2 (0.4) 1.8 (1.2) 5 (100)
OVERALL 43 (50)
a IndicatesLikert scale 1-5 style question. Scores correspond as responses to each question as: 1 = strongly disagree; 5 = strongly agree.
b Indicates:“Yes or No” style question. So for instance the n (%) reflected for “My PALS card is on my person” demonstrates the total number (and percentage) of respondents who reported that “yes” they
had their PALS card on their person at time of taking the survey.
Abbreviations: MAMC, Madigan Army Medical Center; MTF, Military treatment facility; NCR, National Capital Region; NMCP, Naval Medical Center Portsmouth; NMCSD, Naval Medical Center San
Diego; SAMHS, San Antonio Military Health System; TAMC, Tripler Army Medical Center; USU, Uniformed Services University.
MILITARY MEDICINE, Vol. 187, January/February 2022
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022The DHA Pediatrics App
TABLE II. User Perceptions of the Application
Total respondents BC/BE clinicians Trainees (PGY1-PGY3) Nursing team members
Military treatment facility (time zone),
n (%)
Walter Reed National Military Medi- 27 (40.0) 11 (40.7) 9 (33.3) 7 (25.9)
cal Center/National Capital Region
(EST), n (%)
San Antonio Military Health Sys- 30 (44.1) 2 (6.7) 23 (76.7) 5 (16.7)
tem/Brooke Army Medical Center
(CT), n (%)
Tripler Army Medical Center (HT), 5 (7.4) 1 (20.0) 4 (80.0) 0 (0.0)
n (%)
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
Portsmouth Naval Medical Center 2 (2.9) 1 (50.0) 1 (50.0) 0 (0.0)
(EST), n (%)
San Diego Naval Medical Center (PT), 2 (2.9) 1 (50.0) 1 (50.0) 0 (0.0)
n (%)
Madigan Army Medical Center (PT), 2 (2.9) 2 (100.0) 0 (0.0) 0 (0.0)
n (%)
Frequent App User (Likert Score ≥ 5) 44 (65%)
for Question 3
Total, n (%) 68 (100.0) 18 (26.5) 38 (55.9) 12 (17.6)
Mean (SD)
Efficiency
Question 4 5.16 (1.7) 5.21 (1.66) 5.27 (1.64) 4.86 (1.96)
Question 5 4.81 (1.58) 4.94 (1.52) 5.04 (1.54) 4.71 (1.82)
Question 6 5.07 (1.67) 4.96 (1.67) 5.19 (1.60) 5.07 (1.90)
Pooled results for efficiency 5.01 (1.65) 5.03 (1.62) 5.17 (1.59) 4.88 (1.89)
Effectiveness
Question 7 5.22 (1.57) 5.11 (1.79) 5.38 (1.33) 5.14 (1.61)
Question 8 4.96 (1.56) 4.86 (1.63) 4.96 (1.40) 5.14 (1.79)
Question 9 5.21 (1.64) 5.29 (1.74) 5.15 (1.38) 5.14 (1.99)
Question 10 5 (1.59) 5.07 (1.70) 4.96 (1.48) 4.93 (1.69)
Pooled results for effectiveness 5.1 (1.59) 5.08 (1.71) 5.12 (1.40) 5.09 (1.77)
Learnability
Question 11 5.88 (1.53) 6.07 (1.59) 5.69 (1.35) 5.86 (1.79)
Question 12 5.43 (1.54) 5.46 (1.53) 5.38 (1.30) 5.43 (2.03)
Question 13 5.63 (1.53) 5.82 (1.59) 5.50 (1.42) 5.50 (1.70)
Question 14 5.43 (1.47) 5.43 (1.48) 5.50 (1.39) 5.29 (1.68)
Pooled results for learnability 5.59 (1.52) 5.70 (1.54) 5.52 (1.37) 5.52 (1.80)
Memorability
Question 15 5.81 (1.42) 5.93 (1.39) 5.85 (1.32) 5.50 (1.70)
Question 16 5.28 (1.48) 5.39 (1.47) 5.15 (1.43) 5.29 (1.68)
Pooled results for memorability 5.54 (1.47) 5.66 (1.43) 5.50 (1.38) 5.39 (1.69)
Errors
Question 17 5.22 (1.54) 5.39 (1.52) 5.00 (1.44) 5.29 (1.82)
Question 18 5.12 (1.44) 5.25 (1.29) 5.12 (1.48) 4.86 (1.70)
Pooled results for errors 5.17 (1.49) 5.32 (1.41) 5.06 (1.46) 5.06 (1.76)
Satisfaction
Question 19 5.54 (1.61) 5.64 (1.77) 5.54 (1.33) 5.36 (1.82)
Question 20 5 (1.61) 5.00 (1.76) 4.96 (1.54) 5.07 (1.54)
Question 21 4.75 (1.67) 4.61 (1.81) 4.81 (1.52) 4.93 (1.73)
Question 22 5.03 (1.66) 4.93 (1.74) 5.08 (1.65) 5.14 (1.61)
Pooled results for satisfaction 5.08 (1.65) 5.04 (1.77) 5.10 (1.51) 5.13 (1.68)
Cognitive load
Question 23 5.5 (1.41) 5.82 (1.31) 5.27 (1.37) 5.29 (1.64)
Question 24 5.22 (1.49) 5.21 (1.50) 5.15 (1.43) 5.36 (1.69)
Question 25 4.87 (1.41) 4.89 (1.34) 4.69 (1.38) 5.14 (1.66)
Question 26 5.04 (1.53) 4.86 (1.65) 5.19 (1.30) 5.14 (1.75)
Question 27 4.97 (1.38) 4.82 (1.36) 5.00 (1.26) 5.21 (1.67)
Pooled results for cognitive load 5.12 (1.46) 5.12 (1.43) 5.06 (1.35) 5.23 (1.68)
Abbreviations: CT, Central Time; EST, Eastern Standard Time; HT, Hawaiian Time; PT, Pacific Time.
Bolded text indicates the pooled results for a given category.
MILITARY MEDICINE, Vol. 187, January/February 2022 237The DHA Pediatrics App
TABLE III. Results of the Scavenger Hunt
Pediatric Pediatric Board-certified Board-certified Pediatric Board-certified
Clinical status resident trainee resident trainee staff staff resident trainee staff
Clinical classification PGY-3 PGY-2 Subspecialty General pediatric PGY-1 Subspecialty
staff staff staff
Using the DHAPedi- No No No Yes Yes Yes
atricsApp
Time duration to 7:12 19:32 12:45 6:38 4:22 11:36
complete all tasks
(minutes)
Questions correct, n 4 (80%) 3 (60%) 2 (40%) 5 (100%) 5 (100%) 5 (100%)
(%)
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
outputs various component-based products desired by the breakdown of home screen (6,506 page views, 47.4%), CPGs
user. Version3, “DHAPediatricsApp”, was published in May (1,657 page views, 12.1%), directory or page-one call (1,835
2020. Version3 included an Adult-medicine patch placed on page views, 13.4%), and tools (i.e., resuscitation cards, 775
the San Antonio portal to aid clinicians actively engaged in page views, 5.6%). Notably, there were 746 instances (26%
COVID-19 care.23,24 of sessions) when a user navigated between more than one
MTF, which is common for some clinicians who respond to
User Perceptions consults from more than one facility in the MHS.
The team distributed a user experience survey through mul-
tiple lines of communication in order to reach the largest Scavenger Hunt
proportion of clinical team members as possible throughout Six pediatric clinicians (three board-certified staff and three
the MHS. At the time of data collection, there were 725 pediatric residency trainees) participated in the digital scav-
active duty pediatricians within the DHA (39.7% U.S. Air enger hunt usability test. Notably, Nielson et al. have demon-
Force (n = 288), 43.7% U.S. Army (n = 317), and 16.6% U.S. strated that a minimum of five users are required to conduct
Navy (n = 120)). Sixty eight clinical team members (7.4%) a valid usability study.25 The median time to completion of
responded to the PACMAD survey. Information detailing the hunt for clinicians using the App was 6:38 minutes com-
the respondent health care professional type by MTF can pared to 12:45 minutes for non-App users (Table III). App
be found in Table II. Using the 7-item Likert scaled PAC- users (n = 3, 67% board-certified staff) were 100% correct for
MAD tool, respondents (n = 68) attributed more than appro- the five questions in the hunt compared to an average 67%
priate efficiency (5.01 ± 1.65), effectiveness (5.10 ± 1.59), correct for non-App users.
learnability (5.59 ± 1.52), memorability (5.54 ± 1.47), errors
(5.17 ± 1.49), satisfaction (5.08 ± 1.65), and cognitive prop- Medical Librarian Assessment
erties (5.12 ± 1.46) following App use in clinical practice All surveyed medical librarians (n = 9, 100%) from the MTFs
(Table II). Frequent App users (n = 44) defined as users represented on DHAPediatricsApp graded the App using the
reporting either “6-Agree” or “7-Strongly Agree” to frequent Hanrahan survey. The overall mean score was 32.3 ± 0.74
App use recorded statistically higher levels of improvement (scale 0-36 with 36 indicating no deficiencies). The median
in efficiency, effectiveness, learnability, memorability, errors, score for seven of nine categories (usefulness, objectivity,
satisfaction, and cognitive load compared to other users timeliness, functionality, design, and security) was the maxi-
(P < .05). Additional breakdown of user perceptions by health mum score; the median score for accuracy and authority was
care professional type can be found in Table II. Notably 95% 3 (Supplemental Table S1).
(n = 21) category ratings met the pooled a priori score of >5.
Nursing team members reported less than adequate efficiency DISCUSSION
4.88 ± 1.89. Our results suggest that the DHAPediatricsApp is a mobile,
an effective tool to extend a clinician’s capabilities and inter-
Google Analytics professional communication between world-wide users and
From May 1, 2020 to January 20, 2021, there were 1,292 six MHS markets. Subjectively, users perceived the App to
unique (new) users (77.1% return users, n = 996) who con- significantly decrease the time required to complete important
ducted 2,875 sessions (log-ins) to the DHAPediatricsApp. tasks (for example, looking up phone numbers), PACMAD
Devices used for access to the App were evenly split among Question number 4. Objectively, the Google Analytics report
desktop, smart phone, and tablets. The average session lasted showed that 26% of users viewed more than one MTF and
2:54 minutes and users viewed approximately five pages per 13% of users accessed the directory or page-one call inter-
session. In total, there were 13,740 page navigations with a faces. When combined, these findings demonstrate that the
238 MILITARY MEDICINE, Vol. 187, January/February 2022The DHA Pediatrics App
DHAPediatricsApp eliminates communication barriers and and rapidly updated. Making even straightforward changes to
improves efficiency. This was further evidenced when our Apps on the GS/AS platforms can take weeks to months and
adult medicine at the San Antonio Military Health System requires intimate knowledge of two different coding styles and
adjusted clinical services based on needs to fight the COVID- store regulations. PDSA cycle 2 addressed this drawback and
19 pandemic in May of 2019. Users recognized an imme- required the SDD to transition the App from housing of the
diate need for this communication support tool which we App on the GS/AS to a PWA. The importance of the transition
provided within a 24-hour period and remains today. This to a PWA platform is that the SDD now maintains total com-
further demonstrates the potential for universal App of this mand and control of the tool meaning that the limiting reagent
tool. of this equation is timely feedback from the users. As refer-
Although the DHAPediatricsApp was not specifically enced above, when a need such as the COVID-19 pandemic
compared to other existing applications available to users, presented itself, a new group of users (adult-medicine clini-
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
respondents to the PACMAD demonstrated that the DHA- cians) quickly identified themselves and changes were rapidly
PediatricsApp was superior to the armamentarium available made to meet needs. This approach can be standardized as it
before our App implementation. The App was designed to is implementable across all devices as opposed to strategies
serve as a superior means of accomplishing common clini- using the GS/AS. Our PACMAD data reflected the final PWA
cal tasks compared to the decentralized methods currently in platform used for the DHAPediatricsApp.
place at our MTFs. PACMAD question numbers 6-10 sub- There are several limitations to our study. First, the 7.4%
jectively demonstrated greater effectiveness and efficiency response rate is low and may not be generalizable to the
experienced by App users compared to any other methods broader population of military pediatricians. Response rates
available to them to complete clinical tasks (Table II and could be improved through coordinated efforts that offer sur-
Supplementary Figure S1). The greater effectiveness and effi- vey capture during DHA wide meetings of Pediatric clinical
ciency provided by the App was objectively supported by the team members such as the DHA Complex Pediatrics Care
Scavenger Hunt findings which showed that App users com- Community Forum Meeting or the American Academy of
pleted tasks in half the time and with fewer errors. Finally, Pediatrics National Conference and Exhibition meeting of the
the universal superiority of this App compared to other tools Section on Uniformed Services. Second, while we included
was further demonstrated by querying the medical librarians. nursing teammates and medical administrators in both the
As experts in the field of medical references, the librarians are needs assessment and roll out processes, these clinical team-
oftentimes the first to know about new technologies and Apps mate groups are under-represented in our study. To com-
than can assist at the point of care. The overall App percentile bat this underrepresentation, a new Targeted Nursing Team
score of 89.7 reflected a superior tool with minimal deficien- Members Needs Assessment should occur following annual
cies. In sum, we have supported the concept that our App updates to the App in August of 2021. This assessment will
can decrease cognitive load, is superior to traditional methods seek to address the deficiency identified during PACMAD
employed for completing clinical tasks, and has potential for assessment of nurses. Next, because there is no “cross-talk”
replication to other clinical areas including internal medicine, between Google Analytics and our PACMAD survey tool,
surgery, or the battlefield. there was no way to correlate device type with user percep-
Clinical team members are heterogeneous in their accep- tions. Finally, the Google Analytics report may overestimate
tance of an App into clinical practice. Our data showed that the number of individuals accessing the App since implemen-
App frequent users perceive the greatest improvement in effi- tation. While universal access from any device without need
ciency, learnability, memorability, satisfaction, and cognition for a login is a strength of the App, it makes it difficult to
based on responses to the PACMAD survey. Further, the determine whether “unique” users represents the same per-
PACMAD data indicated that BC/BE and trainee clinicians son logging in from a new device or not. This illuminates
reported overall higher acceptance of the App compared to our team’s greatest challenge: how to balance ease of access,
nursing team members. The low response rate from nurses maintain security of contact information while continuing to
(n = 12, 17.6%) would make any statistical comparison of preserve clinical team members’ time and reducing cognitive
these groups difficult to interpret. The development of the load. Future capture of user perceptions may incorporate sur-
DHAPediatricsApp shows potential to eliminate communica- veys within the PWA or repeat the use of the PACMAD tool
tion barriers between primary care clinicians and consultants, with additional questions to determine the number of devices
especially those separated by geographic distance, to improve employed by a single user operating the App. To date, there
awareness and to increase the usability of published CPGs have been no reports of breach of the App by unwanted parties
and SOPs. Further App improvements could involve a new indicating that our current security framework is appropri-
Targeted Nursing Team Members Needs Assessment which ate. However, future updates could include a username and
would address the deficiency identified during PACMAD password setting along with a Common Access Card login
assessment of nurses. feature to better prevent unwanted access. These security fea-
The most important lesson learned during this endeavor tures would facilitate capture of “unique” user perceptions
was the need for a Progressive Web App (PWA) that was easily over time.
MILITARY MEDICINE, Vol. 187, January/February 2022 239The DHA Pediatrics App
The strength of the PWA architecture is that is can be CONFLICTS OF INTEREST STATEMENT
accessed from clinician desktops, mobile devices, or be inte- Salary support was provided for Drs Reeves, Royall, Spencer, Rogers, Durn-
grated into the EHR (the DHA has recently purchased a ing, Samuel, O’Meara, and Gorman by the U.S. DoD. The authors have no
Cerner license and we are working on this update). financial relationships relevant to this article to disclose.
With the near ubiquitous uptake of smartphones, tablets,
and computers into the daily work of modern medicine, CONTRIBUTORS’ STATEMENT
mobile Apps that integrate references, communication, and P.R. (patrick.t.reeves.mil@mail.mil) conceptualized the study, designed the
clinical decision support serve as a ready means to improve application, created the pictograms on the App, conducted the assess-
ments via survey, interpreted the data analysis, drafted the manuscript,
efficiency and provide better care. The development and evo-
and approved the final manuscript. M.R. (matthew.j.royall.mil@mail.mil)
lution of the DHAPediatricsApp may be generalizable to
contributed to study design, assisted in development of the applica-
other health networks, especially those encompassing large tion, interpreted the data analysis, and revised the manuscript. S.S.
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
geographical areas with remote or rural providers. Iterative (steven.e.spencer.mil@mail.mil) contributed to study design, assisted in
design, responsive to the needs of the user and health system, development of the application, interpreted the data analysis, and revised the
and input by a variety of experts in graphic design and infor- manuscript. P.R. (philip.l.rogers.civ@mail.mil) served as an expert in clinical
mation science are necessary for a successful medical App. quality, and approved the final manuscript. S.D. (steven.durning@usuhs.edu)
Future studies of this and other mHealth Apps should explore served as an expert in health professions education and an expert in health
their potential impact of use of Apps on health outcomes, care innovation, informed the assessment of the application, reviewed
cost, and provider health. Mobile health Apps designed for manuscript drafts, and approved the final manuscript for submission. A.S.
use in specific health systems rather than just individual users (anita.samuel.ctr@usuhs.edu) served as an expert in health professions edu-
cation, assisted in development of the overall mobility strategy, contributed
have the promise to allow evaluations of these higher level
to design of the application, review manuscript drafts, and approved the final
outcomes.
manuscript for submission. K. O’M. (kevin.m.omeara.mil@mail.mil) served
as an expert in clinical quality, and approved the final manuscript. G.G. (Gre-
ACKNOWLEDGMENTS gory.gorman@usuhs.edu) contributed to study design, oversaw the statistical
analysis, interpreted the data, revised the manuscript, and served as subject
To Sofia Echelmeyer BFA, Department of Pediatrics, Uniformed Services
matter expert and project manager. All authors approved the final manuscript
University of the Health Sciences, Bethesda, MD: who served as the team’s
medical graphic artist. She played an invaluable role in design for the App as submitted and agree to be accountable for all aspects of the work.
that enhanced user experience and App effectiveness.
To Heather Soloria MD, Benjamin Smith MD, Nora Mulloy DO, Joshua REFERENCES
Boster MD, Madison Marvel MD, Matthew Timlin DO, Charles Kang MD,
1. Ramirez V, Johnson E, Gonzalez C, Ramirez V, Rubino B, Rossetti G:
Jeanette Traver MD, Meaghan Wido MD, Sterling Kosmach DO, Sarah
Assessing the use of mobile health technology by patients: an obser-
Thompson MD, Dana Gilbert MD: who served as the points-of-contact for
vational study in primary care clinics. JMIR mHealth uHealth 2016;
gathering of data used to inform the application. Thank you to these clinicians
4(2): e41.
who played a vital role in Application development coming from the military
2. Khurana M: Keeping pace: the need for digital health education in
treatment facilities: Naval Medical Center Portsmouth, Tripler Army Medi-
medical schools. Acad Med 2020; 95(11): 1629–30.
cal Center, Brooke Army Medical Center, Madigan Army Medical Center,
3. Coulter A, Dunn N: After Bristol: putting patients at the centreCom-
Naval Medical Center San Diego, and Walter Reed National Military Medi-
mentary: patient centred care: timely, but is it practical? BMJ 2002;
cal Center, respectively. Thank you for your hard work and helping to make
324(7338): 648–51.
this dream a reality.
4. Qudah B, Luetsch K: The influence of mobile health applications
To Michele Mason-Coles MLS, Sarah C. Clarke MSLS, AHIP, Emily
on patient-healthcare provider relationships: a systematic, narrative
Shohfi MLIS, AHIP, Katherine Wolf MLS, Diane Kunichika MLIS, Jeffery
review. Patient Educ Couns 2019; 102(6): 1080–9.
Ring PhD, MILS, Tracy Shields MSIS, AHIP, Christopher Chambers, and
5. Gottschalk A, Flocke SA: Time spent in face-to-face patient care
Alison Rollins MLS, MIS : who serve as the medical librarians for the military
and work outside the examination room. Ann Fam Med 2005; 3(6):
treatment facilities represented on the application: Walter Reed National Mil-
488–93.
itary Medical Center, Naval Medical center San Diego, Tripler Army Medical
6. Baig MM, GholamHosseini H, Connolly MJ: Mobile healthcare appli-
Center, Madigan Army Medical Center, Naval Medical Center Portsmouth,
cations: system design review, critical issues and challenges. Australas
Brooke Army Medical Center and Uniformed Services University of the
Phys Eng Sci Med 2015; 38(1): 23–38.
Health Sciences respectively. Thank you for taking the time to review our
7. Sondhi V, Devgan A: Translating technology into patient care: smart-
application and provide feedback using the questionnaire rubric from Han-
phone applications in pediatric health care. Med J Armed Forces India
rahan et al. Your feedback was crucial to future updates to the App and will
2013; 69(2): 156–61.
prove invaluable to making this clinical tool a reliable clinical extender for
8. Gotschall T: Review of pediatric care online. J Electron Resour Med
years to come.
Libr 2019; 16(2): 67–74.
9. DocMatter: The medical specialist community DocMatter. 2021.
Available at https://www.docmatter.com/company; accessed March
SUPPLEMENTARY MATERIAL 01, 2021.
Supplementary material is available at Military Medicine online. 10. NASPGHAN: NASPGHAN toolbox. 2019. Available at https://
naspghan.org/professional-resources/medical-professional-resources/
naspghan-toolbox/; accessed March 01, 2021.
FUNDING 11. TRICARE: Beneficiary population statistics. Health.mil. 2021. Avail-
There was no funding support beyond the salary disclaimers, as above, for able at https://www.health.mil/I-Am-A/Media/Media-Center/Patient-
the completion of this project. Population-Statistics; accessed March 01, 2021.
240 MILITARY MEDICINE, Vol. 187, January/February 2022The DHA Pediatrics App
12. Judd CA, Dong T, Meyer HS, Hickey PW, Torre DM, 19. Karsh B: Beyond usability: designing effective technology implemen-
Durning SJ: The impact of military pediatrics: assessing clin- tation systems to promote patient safety. BMJ Qual Saf 2004; 13(5):
ical, leadership, academic, and operational experience among 388–94.
pediatric-trained graduates from the Uniformed Services Univer- 20. Reeves PT: How to use the DHA pediatrics app. YouTube. 2020. Avail-
sity of the Health Sciences (USU). Mil Med 2020; 185(9–10): able at https://youtu.be/ZLVqehNJdPc; accessed March 01, 2021.
e1584–9. 21. Saleh A, Isamil RB, Fabil NB: Extension of PACMAD model for
13. Clifton EY: Introduction: the role of the pediatrician in military usability evaluation metrics using Goal Question Metrics (GQM)
medicine.Am Acad Pediatrics 2012; (Suppl): S1–S2. approach. J Theor Appl Inf Technol 2015; 79(1): 90.
14. Musinski R: It’s not just a job, it’s an adventure for 22. Connor Hanrahan TA, Sabrina C: Evaluating mobile medical appli-
military pediatricians. AAP News.2017. Available at cations. eReports. 2014, 18. Available at https://www.ashp.org/-
https://www.aappublications.org/news/2017/06/20/MilitaryPeds062017; /media/store%20files/mobile-medical-apps.pdf; accessed March 01,
accessed March 01, 2021. 2021.
15. Pierce J, Hemming VG: A case for the military pediatrician. Mil Med 23. Percival C: Section on Uniformed Services American Academy of
Downloaded from https://academic.oup.com/milmed/article/187/1-2/232/6295803 by guest on 07 February 2022
1986; 151(10): 559. Pediatrics Uniformed Services Section Newsletter Winter 2020–
16. Pamplin JC, Davis KL, Mbuthia J, et al: Military telehealth: a model 2021. Available at https://collaborate.aap.org/SOUS/_layouts/15/
for delivering expertise to the point of need in austere and operational WopiFrame.aspx?sourcedoc={abfaa765-0c6b-471d-931b-81f611d6
environments. Health Aff 2019; 38(8): 1386–92. 70f8}&action=default; accessed March 01, 2021.
17. Summers SM, Nagy CJ, April MD, Kuiper BW, Rodriguez RG, 24. Patrick Reeves SDDotDHA. The DHA pediatrics app. Health.mil.
Jones WS: The prevalence of faculty physician burnout in military 2020. Available at https://mobile.health.mil/dhapeds/index.html#/;
graduate medical education training programs: a cross-sectional study accessed March 01, 2021.
of academic physicians in the United States Department of Defense. 25. Nielsen J, Landauer TK, eds: A mathematical model of the finding of
Mil Med 2019; 184(9–10): e522–30. usability problems. In: Proceedings of the INTERACT’93 and CHI’93
18. Keyser EA, Weir LF, Valdez MM, Aden JK, Matos RI: Extending peer conference on human factors in computing systems. Amsterdam:
support across the military health system to decrease clinician burnout. 1993.
Mil Med 2021; 186(Suppl 1): 153–9.
MILITARY MEDICINE, Vol. 187, January/February 2022 241You can also read
