A comparison of electronic records to paper records in mental health centers
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
International Journal for Quality in Health Care 2008; Volume 20, Number 2: pp. 136 –143 10.1093/intqhc/mzm064
Advance Access Publication: 12 December 2007
A comparison of electronic records to
paper records in mental health centers
JACK TSAI AND GARY BOND
Department of Psychology, Indiana University-Purdue University Indianapolis (IUPUI)
Abstract
Objective. Medication documentation is a critical aspect of quality patient care. The current study examined whether
electronic medical records provide medication documentation that is more complete and faster to retrieve than traditional
paper records.
Method. This study involves a comparison of archived paper medical records to recent electronic medical records through
chart review. A convenient sample of three large community mental health centers in Indiana was used. Medical charts for
180 patients with schizophrenia were rated on a checklist composed of 16 items that was adapted from a national project.
Documentation that existed before implementation of the electronic medical record system was compared with that after
implementation at each of the three centers. The main outcome measures were completeness and retrieval time of medication
documentation.
Results. Electronic medical records provided medication documentation that was more complete and faster to retrieve than
paper records across all centers and within each center. On average, electronic medical records were 40% more complete and
20% faster to retrieve.
Downloaded from by guest on September 7, 2015
Conclusion. Electronic records have potential to improve medication management for patients in mental health centers over
traditional records. However, medication documentation for patients diagnosed with schizophrenia was found to be deficient
in many areas, regardless of documentation format.
Keywords: community mental health centers, documentation, electronic medical records, medication management,
schizophrenia
Medical records contain treatment history and relevant through patient administration with independent departmen-
experiences pertaining to the care of the individual. As medical tal systems. (ii) Level 2 is Level 1 plus integration via master
records are continually updated, they provide written proof of patient index. (iii) At Level 3, true clinical support is available
the medical life of a patient over time which can aid future with many practical uses, such as electronic clinical orders,
courses of treatments and provide decision support. results reporting, prescribing and multi-professional inte-
Traditionally, clinical documentation has been handwritten on grated care pathways. (iv) Level 4 has Level 3 plus electronic
forms and filed into paper medical records. However, the short- access to knowledge bases, embedded guidelines, electronic
comings of paper records are well known [1]. Handwritten alerts and expert system support. (v) Level 5 has Level 4
medical records can be illegible, incomplete and poorly orga- plus specific clinical models and document imaging. (vi) The
nized, making it difficult to ensure quality of care [2]. most advanced level is Level 6. It has telemedicine and other
The advent of computer technology has introduced enor- multi-media applications such as picture archiving and com-
mous possibilities for electronic documentation and usage of munication systems.
electronic medical records. Electronic medical records are The potential benefits of electronic records in healthcare,
defined as medical records located on a shared computer such as increased communication between users, reduced
network that are both read and written electronically on a paperwork, fewer medical errors and cost savings have been
relational database through a graphic user interface. Dudman widely discussed [4–9]. Electronic records allow for ‘just in
[3] describes six levels of sophistication in electronic medical time’ access and have led to faster data searches and increased
record systems, which were used to characterize electronic physician efficiency [10]. Surprisingly, the direct evidence of
medical records in this study. (i) Level 1 is the most basic the advantages of electronic medical records over paper
level supporting administrative functions of an organization records is meager. Although there is an extensive literature on
Address reprint requests to: Jack Tsai, Department of Psychology, Indiana University-Purdue University Indianapolis (IUPUI),
Tel: þ317 274-6760; Fax: þ317 988-2719; E-mail: jatsai@iupui.edu
International Journal for Quality in Health Care vol. 20 no. 2
# The Author 2007. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved 136Comparison of electronic records to paper records
data accuracy in paper records [11], this does not appear to be The chart inclusion criteria were patient diagnosed with a
the case for electronic records [12]. Despite their potential schizophrenia-spectrum disorder, prescribed an antipsychotic
advantages and strong federal recommendations [13], the and received medication services from the community
mental health field has lagged behind other healthcare special- mental health center for at least a year. At each center, 30
ties in utilizing electronic medical records [14]. paper and 30 electronic records were randomly selected by
Unfortunately, there is reason to be concerned with the each site’s staff using a random numbers table. Patients
quality of documentation in psychiatric records. Surveys in selected for the paper record sample were excluded from the
community settings have found (i) management of antipsy- electronic record sample. A total sample of 90 electronic and
chotic medications is often at variance with evidence-based 90 paper records were sampled across three mental health
recommendations, (ii) documentation of target symptoms centers.
and side effects is frequently inadequate and (iii) documen-
tation of treatments and their outcomes is often missing
Assessors
from medical records [15, 16]. Thus, there is a strong need
to develop methods to assess and improve medication man- Two clinical psychology graduate students served as asses-
agement and documentation. One large project that has sors. Before data collection, they received a brief orientation
made such efforts is Medication Management Approaches in at each site on its medical record system to identify the
Psychiatry (MedMAP) that will now be described. common data locations for the medication-related items.
As part of the National Evidence-Based Practices Project, Assessors also received brief orientations to both the elec-
MedMAP was identified as an evidence-based practice for tronic medical and paper medical record systems at each site.
severe mental illness [17] and a ‘toolkit’ was developed to
facilitate its implementation [18]. The toolkit was limited to
Measures
medications for schizophrenia, with the intent ultimately
to expand to other disorders. The content of this toolkit was The Medication Management Approaches in Psychiatry
guided by a national panel of experts [19], findings from the (MedMAP) Checklist (see Appendix A) was a measure that
Texas Medication Algorithm Project [20] and the was developed specifically for this study and was adapted
Downloaded from by guest on September 7, 2015
Schizophrenia Patient Outcomes Research Team recommen- from the prescriber level fidelity scale from the MedMAP
dations [16, 21]. The MedMAP toolkit provides a set of cri- toolkit [19]. The checklist contains 16 items with dichoto-
teria for medication documentation considered necessary for mous ratings of ‘present or absent’ on items such as ‘year of
adequate patient care and its contents were used in this study. last hospitalization’, ‘level of current medication adherence’
The present study aimed to determine whether electronic and ‘past psychotropic medications’. Assessors rated each
medical records provide higher quality documentation than medical chart for documentation of these items within 1 year
paper records, thereby improving the medication manage- of the most recent note. Medical documentation beyond the
ment of individuals with schizophrenia. No previous study 1 year period was considered outdated. A protocol with
found has examined this and it was hypothesized that elec- descriptions of item rating decisions, such as when to count
tronic records would provide documentation that was more an item as present or absent, was used to ensure
‘complete’ and ‘faster to retrieve’ than paper records. standardization.
Completeness. Completeness was defined as the total
number of items that were found to be present on the
MedMAP Checklist. For each chart, items were summed for
Methods a total score, which ranged from 0 (all items absent) to 16
(all items present).
This study was a retrospective chart review of medication
Retrieval time. Retrieval time was measured by the time
information compiled before and after the adoption of elec-
needed for each assessor to find and rate all items on the
tronic medical records, comparing archived paper records to
MedMAP Checklist. For each chart, assessors self-timed
recent electronic records for the completeness and retrieval
themselves using a stopwatch that was started with the first
time of documentation at three mental health centers. All
item on the MedMAP Checklist and stopped after the last
procedures were approved by the university’s institutional
item was completed. The average times for the two assessors
review board.
for each chart were calculated and used as the measure of
retrieval time in data analyses. All the primary analyses in the
study were repeated using individual assessor times, yielding
Sampling
similar results, suggesting that the mean of the two assessors
A convenience sample of three community mental health was a satisfactory measure.
centers in Indiana was used. Each center served over 4000
patients annually, including over 200 patients diagnosed with
Data collection
a schizophrenia-spectrum disorder who received medication
services. Two sites were located in a large city, whereas the A brief structured interview was conducted with program
third was located in a rural area. All three centers had been directors to obtain information about their medical record
using electronic records for over 2 years. systems and documentation-related changes over the years.
137J. Tsai and G. Bond
Then, the random sample of paper records was indepen- aggregated across sites. At the item level, aggregated across
dently rated and timed by each assessor using the MedMAP sites, there was high agreement between assessors on ratings
Checklist. Assessors rated charts in different order, but the (Cohen’s Kappa ¼ 0.61, P , 0.001).
same group of charts on the same day. At the end of the Learning curve effects were examined to observe whether
day, assessors discussed their individual ratings and reached there were differential learning curves in retrieval time
consensus. After paper records were rated, the same pro- between documentation format and between sites that may
cedure was repeated with the random sample of 30 electronic have influenced or confounded the results. There appeared
records. Paper and electronic records were assessed on sepa- to be learning curves for both documentation formats with
rate days. significant correlations found between retrieval time and the
order that charts were assessed (P , 0.05). But no consistent
pattern emerged across sites. Various supplementary analyses
Data analysis were conducted on the learning curves and they were found
The data were checked for outliers; assumptions of normality to have no material influence on the main results.
and homogeneity of variance were tested. The inter-rater There were no significant patient demographic differences
reliability between assessors was calculated with intra-class between the PMR sample and the electronic medical record
correlation coefficients based on a two-way mixed model sample (see Table 1). As expected, the duration of treatment
using the average measure reliability [22]. To gauge whether documentation was significantly longer for paper records
assessors became faster with more experience in rating charts than electronic records.
at a site, ‘learning curves’ were visually inspected and corre-
lations were conducted between retrieval time and the order Differences on completeness and retrieval time
charts were assessed. Descriptive statistics for patient demo-
graphics were calculated and differences between documen- Two-way analysis of variance analyses found highly signifi-
tation format samples were tested. The main outcomes, cant main effects of documentation on completeness and
completeness and retrieval time were summarized as the retrieval time, showing that documentation in electronic
mean total number of items per chart and mean retrieval records were significantly more complete and faster to
Downloaded from by guest on September 7, 2015
time per chart, respectively. To test each of the main out- retrieve than paper records across the three sites, as shown
comes, two factor analyses of variance were used with the in Table 2. On average, electronic records had about two
family wise error rate set at a two-sided alpha level of 0.05. more items on the MedMAP Checklist documented than
Correlations were conducted between the two main out- paper records or were 40% more complete; each electronic
comes for each documentation format separately and medical record also took 89 s less to rate or was 20% faster
together to observe any relationships. At the item level, fre- to retrieve than paper records. The correlation between com-
quency analyses of completeness were conducted and inde- pleteness and retrieval time for paper records was r ¼ 20.15
pendent t-tests were used to test for differences. (n ¼ 90) and not significant; but for electronic records, it was
r ¼ 20.31 (n ¼ 89) and was significant (P , 0.01).
Table 3 shows item-level analyses of the MedMAP
Checklist and which items tended to be less complete than
Results others. To assess whether paper records and electronic
records showed a similar pattern of completeness, a corre-
Preliminary analysis
lation was calculated using the 16 pairs of item percentages
The mental health centers in this study all used different as data points. This yielded a correlation of 0.86, which
electronic medical record systems. Using Dudman’s [3] suggests similar kinds of information were being omitted in
differentiation of electronic medical records, the first author both documentation formats. All 16 items showed improve-
rated all three electronic medical record systems at a Level 2, ment in completeness moving from paper records to elec-
meaning they had an integrated patient administration system tronic records, except for Item 15—‘Documentation of
that was indexed and had independent departments. One Weight’.
chart had a particularly long retrieval time and was an outlier
so it was excluded from all related analyses. Distributions of
completeness and retrieval time values were roughly normal Discussion
for both documentation formats with skewness and kurtosis
all within the range of 21 to 1. A log transformation was Electronic medical records have begun to be implemented in
performed on retrieval times to meet the assumption of the mental health field. This study looked at three mental
homogeneity of variance. There was adequate inter-rater health centers that have recently replaced their paper medical
reliability on completeness for paper records (intra-class cor- records with electronic records. Documentation in electronic
relation coefficient ¼ 0.65, P , 0.001) and electronic records records was found to be significantly more complete and
(ICC ¼ 0.71, P , 0.001) when aggregated across the three faster to retrieve than paper records. This is a crucial finding
study sites. On retrieval time, there was also adequate inter- because medication documentation chronicles the treatment
rater reliability for paper records (ICC ¼ 0.78, P , 0.001) life of patients and serves as support in making treatment
and electronic records (ICC ¼ 0.66, P , 0.001), when decisions. Although the study focussed on schizophrenia, it
138Comparison of electronic records to paper records
Table 1 Summary of demographic variables
Paper medical record sample Electronic medical record Test of
(n ¼ 90) sample (n ¼ 90) significance
.............................................................................................................................................................................
Mean age (SD) 46.4 (12.1) 46.0 (12.6) P ¼ 0.82
Gender
Male 54 (60.0%) 55 (61.1%) P ¼ 0.88
Female 36 (40.0%) 35 (38.9%)
Ethnicity
White 50 (55.6%) 49 (54.4%) P ¼ 0.87
Black 33 (36.7%) 37 (41.1%)
Other 3 (3.3%) 4 (4.4%)
Diagnosis P ¼ 0.61
Schizophrenia 72 (80.0%) 70 (77.8%)
Schizoaffective/Schizophreniform 17 (18.9%) 20 (22.2%)
Mean duration of documentation (months) 119.1 (83.8) 42.7 (28.9) P , 0.001*
Ethnicity categories of black and other were collapsed into one category for the test of significance.
*p , 0.001.
is plausible to hypothesize that these findings would genera- documentation related to past psychotropic medications and
lize to other mental illnesses as well, which also require documentation of glucose. But all 16 items, except one
careful and extensive medication documentation. Yet, medi- about weight documentation, showed improvement in com-
cation practices have often been found to be deviant from pleteness with using electronic records in place of paper
Downloaded from by guest on September 7, 2015
evidenced-based recommendations [21]. records.
Medical errors are a serious problem in healthcare and are Despite the advantage of electronic medical records over
often a result of documentation errors made in paper paper records, the fact remains that there is still great need
records [1]. This study did not examine documentation for improvement in medication documentation. Despite the
errors per se; it used a proxy measure to assess the adequacy evidence for items on the MedMAP Checklist, less than half
of documentation. So, it looked at whether items that should were found to be documented in the charts in this study.
be documented were documented without determining accu- Thus we can extend the findings of previous research [15,
racy or actual medication practices. But this study is a step 16] that medication practices are still often at variance with
towards documenting a critical strategy for ameliorating evidence-based recommendations and more attention is
errors because omission of documenting items precludes needed in translating research into practice.
correction of errors. The findings of this study suggest that The results of this study corroborate several findings from
implementing electronic medical records to replace paper a pilot study of the MedMAP fidelity scale [19]. In both
records may be a fruitful avenue to advancing the quality of studies, documentation about past medication treatments,
documentation for patients. The findings also suggest that side effects and outcomes were poorly documented. The
the more complete documentation is the faster it is to MedMAP Checklist may have potential to be used as a shor-
retrieve; results found this relationship to be particularly true tened version of the original fidelity scale. As the fidelity
for electronic records. This is another argument for the use scale was a measure that often took several days to complete
of electronic records, in that they may increase the retrieval with the prescriber portion averaging 36 min per chart, the
time of documentation by being more complete. And yet the MedMAP Checklist may be a viable alternative that takes
mental health centers in this study had only begun to use considerably less time to use (7.60 min per chart for paper
electronic records in the last few years compared to decades records, 6.08 min per chart for electronic records). However,
of use with paper records. it is also notable that the checklist is not as comprehensive as
Certain medication documentation elements tended to be the fidelity scale as it does not measure as many items and
more complete than others, with similar items showing up does not measure them on a gradient.
for both electronic and paper records, namely, documen- We speculate that the findings of this study would genera-
tation of diagnosis, medication adherence and patient edu- lize to other community mental health centers. This study
cation. But electronic records had dramatically more found that the documentation in electronic records were
complete documentation on two items related to current more complete and faster to retrieve than paper records
medications and rationale for their prescriptions. This in ‘each’ of the three sites and that can be seen as three
finding has practical significance because it is essential for replications with the same results. It is possible that even
prescribers to know what medications patients are currently greater advantages can be found for electronic records if
taking and why. Electronic records shared some similar items sites were given even more time to adapt and develop their
that were missing with paper records. These were systems.
139140
J. Tsai and G. Bond
Table 2 Summary of outcomes for all three sites
Completeness Test of significance
........................................................................................................................................................................ ................................................................
Site A Site B Site C Total average of 3 sites Documentation Site Interaction
............................................................................................................................................................................................................................................
Paper medical records mean number 4.03 (1.63) 4.47 (1.14) 6.23 (1.74) 4.91 (1.78) P , 0.001** P , 0.001** P ¼ 0.65
of items (SD)
Range (min – max) 2–8 3– 7 3– 9 2– 9
Electronic medical records mean number 6.03 (1.65) 6.37 (2.01) 8.66 (1.49) 7.00 (2.07)
of items (SD)
Range (min – max) 3 – 10 2– 11 6– 12 2– 12
Retrieval time
Paper medical records mean time in 552.27 (134.89) 429.77 (92.75) 379.58 (121.87) 453.87 (137.44) P , 0.001** P , 0.001** P ¼ 0.01*
seconds (SD)
Range (min – max) 337 – 874 264 – 727 166 – 679 166 –874
Electronic medical records mean time in 500.47 (126.18) 336.67 (64.70) 254.15 (58.62) 364.99 (135.27)
seconds (SD)
Range (min – max) 286 – 771 212 – 469 133 – 366 133 –771
Test of significance of Documentation is a comparison of electronic to paper medical records. Test of significance of Site is a comparison of Sites A, B and C.
*p , 0.05. **p , 0.001.
Downloaded from by guest on September 7, 2015Comparison of electronic records to paper records
Table 3 Item by item percentages of completeness
Paper medical records Electronic medical records Difference
(n ¼ 90) (n ¼ 90) (electronic paper)
.............................................................................................................................................................................
Item 1: Diagnosis (%) 97.8 98.9 1.1 P ¼ 0.57
Item 2: First hospitalization (%) 23.3 42.2 18.9 P ¼ 0.01*
Item 3: Prior hospitalizations (%) 25.6 40.0 14.4 P ¼ 0.03*
Item 4: Last hospitalization (%) 43.4 54.4 11.0 P ¼ 0.16
Item 5: Summary of course of Illness (%) 43.3 55.6 12.3 P ¼ 0.12
Item 6: Past psychotropic meds (%) 0.0 2.2 2.2 P ¼ 0.15
Item 7: Current meds (%) 6.7 66.7 60.0 P , 0.001**
Item 8: Rationale for each med (%) 2.2 33.3 31.1 P , 0.001**
Item 9: Med adherence (%) 86.7 92.2 5.5 P ¼ 0.24
Item 10: Patient education (%) 51.5 70.0 18.9 P ¼ 0.01*
Item 11: Outcomes (%) 18.9 33.3 14.4 P ¼ 0.04*
Item 12: Tardive Dyskinesia (%) 17.8 22.2 4.4 P ¼ 0.44
Item 13: EPS symptoms (%) 30.0 36.7 6.7 P ¼ 0.40
Item 14: Glucose (%) 1.1 4.4 3.3 P ¼ 0.17
Item 15: Weight (%) 16.7 11.1 25.6 P ¼ 0.30
Item 16: Patient involvement (%) 26.7 38.9 12.2 P ¼ 0.07
*p , 0.05. **p , 0.001.
Downloaded from by guest on September 7, 2015
However, the value of electronic medical records in the sites. Some organizational changes were reported by
mental health services remains largely unexplored. There are some sites as well that may have influenced the results. Site
many areas for future research. As this study was mainly an A reported that mental health and addiction services were
omnibus test of electronic records versus paper records, the combined for the first time when the electronic medical
mediators responsible for the higher level of completeness record system was created; Site B reported minor organi-
and faster retrieval time of documentation in electronic zational adjustments involving splitting their office manage-
records were not systematically studied. Several untested ment from two geographic areas to three.
hypotheses were formed during this study and may be fertile Staff characteristics could have affected the completeness
areas for future study. In interviews with program directors, and retrieval time of documentation over time. Staff charac-
they expressed the opinion that electronic records held staff teristics may have had a significant influence on medication
more accountable for their documentation. The use of elec- documentation as some prescribers are more adept and dili-
tronic records may create an organizational culture that gent with documentation than others. The clinicians at each
changes attitudes about documentation. Another possible agency may have changed over time or documentation prac-
explanation may be in the forms themselves. Some form tices may have changed. A final limitation of this study is the
fields on the electronic records could be programmed to lack of a gold standard to determine the accuracy of docu-
require staff to complete before progressing through other mentation in medical records, i.e. documentation may be
fields on the form. This was not examined in the current complete but not accurate. Other studies have encountered
study because fields varied even within forms, let alone the same problem and the ideal to capture the true nature of
between agencies. Other possible explanations may be that the patient is difficult, if not impossible to achieve [23].
staff find it easier or more agreeable to type than handwrite However, this study points to areas of medication manage-
data, electronic records can be linked to billing systems, and ment that may improve with using electronic records instead
electronic records exist in virtual space whereas paper of paper records, so that the mental health field can benefit
records are organized by staff and have to be physically from the advantages of technology and translate them into a
sifted through. higher quality of care for patients.
Study limitations
Funding
As the program director interviews revealed, there have been
changes in Health Insurance Portability and Accountability This study was partially funded by an Educational
Act regulations in the past few years that may have altered Enhancement Grant from the Indiana University-Purdue
how and what was documented during the same time elec- University Indianapolis (IUPUI) Graduate Student
tronic medical records were being implemented at each of Organization.
141Downloaded from by guest on September 7, 2015
J. Tsai and G. Bond
142Comparison of electronic records to paper records
References 13. Hogan MF. New freedom commission report: the president’s
new freedom commission: recommendations to transform
1. Institute of Medicine. The Computer-Based Patient Record: An mental health care in America. Psychiatr Serv 2003;54:
Essential Technology for Health Care. Washington, DC: National 1467–74.
Academy Press, 1997. 14. Drake RE, Teague GB, Gersing K. State mental health autho-
2. Young AS, Sullivan G, Burnam MA, Brook RH. Measuring the rities and informatics. Commun Ment Health J 2005;41:365 –70.
quality of outpatient treatment for schizophrenia. Arch Gen 15. Cradock J, Young AS, Sullivan G. The accuracy of medical
Psychiatry 1998;55:611 –7. record documentation in schizophrenia. J Behav Health Serv Res
3. Dudman J. Electronic patient records. Facts at their fingertips. 2001;28:456 –65.
Health Serv J 2000;110:9 –13. 16. Lehman AF, Steinwachs DM, PORT Co-Investigators. At issue:
4. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of translating research into practice: the schizophrenia patient out-
computer-based clinical decision support systems on physician comes research team (PORT) treatment recommendations.
performance and patient outcomes: a systematic review. JAMA Schizophr Bull 1998;24:1 –10.
1998;280:1339 –46. 17. Drake RE, Goldman HE, Leff HS, Lehman AF, Dixon L,
5. Makoul G, Curry RH, Tang PC. The use of electronic medical Mueser KT et al. Implementing evidence-based practices in
records: communication patterns in outpatient encounters. J routine mental health service settings. Psychiatr Serv
Am Med Inform Assoc 2001;8:610 –5. 2001;52:179 –82.
6. Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. 18. Substance Abuse Mental Health Services Administration.
Impact of an electronic medical record system on community- Evidence-based practices: Shaping mental health services
based primary care practices. J Am Board Fam Pract toward recovery. Substance Abuse and Mental Health Services
2000;13:338– 48. Administration’s National Mental Health Information Center
http://mentalhealth.samhsa.gov/cmhs/communitysupport/
7. Ammenwerth E, Eichstadter R, Haux R, Pohl U, Rebel S, toolkits/community/workbook/appendixd.asp (4 November
Ziegler S. A randomized evaluation of a computer-based 2007, date last accessed)
nursing documentation system. Methods Inf Med 2001;40:
Downloaded from by guest on September 7, 2015
61 –8. 19. Bond GR, Miller AL, Taylor A, Tsai J, Howard PB, El-Mallakh
P et al. A scale to assess fidelity to evidence-based principles of
8. Johnson M, Benbow SM, Baldwin RC. An electronic patient medication management for schizophrenia. Schizophr Bull
record system and geriatric psychiatry: Considerations and 2007;33:469.
implications. Aging Ment Health 1999;3:257–63.
20. Gilbert DA, Altshuler KZ, Rago WV, Shon SP, Crismon ML,
9. Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Toprac MG et al. Texas medication algorithm project: defi-
Brown GD. The clinical value of computerized information ser- nitions, rationale, and methods to develop medication algori-
vices: a review of 98 randomized clinical trials. Arch Fam Med thms. J Clin Psychiatry 1998;59(7):345 –51.
1996;5:271 –8.
21. Lehman AF, Kreyenbuhl J, Buchanan RW, Dickerson FB,
10. Kaplan B, Lundsgaarde HP. Toward an evaluation of an inte- Dixon LB, Goldberg R et al. The schizophrenia patient out-
grated clinical imaging system: identifying clinical benefits. comes research team (PORT): updated treatment recommen-
Methods Inf Med 1996;35:221 –9. dations 2003. Schizophr Bull 2004;30(2):193 –217.
11. Romm FJ, Putnam SM. The validity of the medical record. Med 22. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing
Care 1981;19:310 –5. rater reliability. Psychol Bull 1979;86(2):420 –8.
12. Hippisley-Cox J, Pringle M, Cater R, Wynn A, Hammersley V, 23. Hogan WR, Wagner MM. Accuracy of data in computer-based
Coupland C et al. The electronic patient record in primary patient records. J Am Med Inform Assoc 1997;4(5):342 –55.
care—regression or progression? A cross sectional study. BMJ
2003;326:1439 –43. Accepted for publication 4 November 2007
143You can also read
