We launched the Constellation project at Boston’s flagship hospitals – Brigham & Women’s Hospital and Massachusetts General Hospital. The exercise was the first structured effort of its kind in the medical setting.
Here are the details from the paper accepted by American Medical Informatics Association’s SCAMC proceedings in Washington, DC.
AMIA – SCAMC (19th Symposium)
The Constellation Project: Experience and Evaluation of Personal Digital Assistants in the Clinical Environment
Steven E. Labkoff, MD1, Sandeep Shah, MS(CS)2, Joseph Bormel, MD, MPH3, YoonJoon Lee, BS1, Robert A. Greenes, MD, PhD1
- Decision Systems Group, Brigham and Women’s Hospital,
Harvard Medical School, Boston, MA; - K2 Consultants, Nashua, NH;
- Laboratory of Computer Science, Massachusetts General Hospital,
Harvard Medical School, Boston, MA
ABSTRACT
Personal Digital Assistants (PDAs) provide an opportunity to explore and implement ubiquitous computing environments. The Constellation Project is exploring the ramifications of providing internal medicine residents at two university-based residency programs using Apple Newton computers. The 100-day study investigated several areas including attitudes towards information access, deployment and use of PDAs in the clinical environment, content selection and presentation issues, use, and satisfaction evaluation. Issues of content selection and content completeness proved to be the most prominent factors the use of PDAs by second-year residents.
INTRODUCTION
The Constellation Project [1] is an ongoing research effort at the Harvard Medical School designed to analyze the acceptance and utilization of Personal Digital Assistants in medical practice. Studies have shown that physicians’ information seeking behavior is affected by the information resources made available to them and by their expectation that those resources are sufficient to answer their questions[2, 3, 4]. Early work by Fishman at Yale[5] illustrated that portable computers could be used successfully in the hospital setting. With the advent of personal digital assistants (PDAs) in the past two years, it is possible to provide access to information in a convenient, readily available format.
Analysis of access to CD ROM-based medical information has been performed by Osheroff et al [4], who found that only 62% of clinicians’ questions could be resolved with this type of reference tool. House officers on an Internal Medicine Service typically make clinical decisions in many ways. It is often the case that they do not consult reference material because these materials are not easily accessible at the sites where they are most needed. For example, residents commonly rotate in satellite clinics where information resources are not as abundant as at the home institution.
During a typical clinical day, a resident must deal with questions that could benefit from access to a textbook or journal article. Use of the library may be impractical if it is not nearby. Other options include use of a clinical workstation (if it supports electronic textbook or MEDLINE access), asking a colleague, or simply relying on memory. A PDA is likely to be most helpful in situations where libraries or clinical workstations providing textbook and literature resources are not readily available.
In August 1993 Apple Computer introduced the first PDA computer, Newtonª. The PDA is a small device that typically fits in the palm of a hand or in a pocket. A pen is used for input to the device. It has a battery life measured in days, not hours. Some PDAs have wireless modem connections so that network services may be accessed anywhere in building, institution, or city.
Ubiquitous computing environments are being tested in ongoing projects such as the Virtual Hospital Project, University of Iowa [6]. This project utilizes the Internet to disseminate information to physicians. The combination of instant access to information resources and the size of the Newton made a compelling argument for its use in clinical practice.
The Constellation Project analyzed the various factors involved in deploying this technology into clinical practice at two university-affiliated medical services to provide access to electronic textbooks and related resources in clinical medicine. Our hypothesis was that if house staff had immediate access to a relevant collection of information, they would use the PDA instead of the methods they typically use to locate medical information. Another hypothesis was that these devices would be used more in environments where medical information was sparse, such as in satellite clinics.
The project ran at both Brigham and Women’s Hospital (BWH) and the Massachusetts General Hospital (MGH) in Boston, Massachusetts. The BWH has a computing environment with more that 3900 networked 80486 terminals throughout the hospital. This environment, know as BICS, provides functions that include physician order entry, clinical laboratory results, clinical alerts, electronic mail (local and Internet), access to CD-ROM-based textbooks and reference information (Scientific American Medicine CD, PDR-CD), MEDLINE (PaperChase), and a collection of quick medical references and calculations. The MGH system provides laboratory results reporting. During the study the MGH added e-mail and MEDLINE (BRS Colleague) services to its MIS.
METHODS
Application Development
Because residents work in different environments, each may have a different set of needs to answer clinical questions. Early efforts by D’Alassandro et al were made to create simple examples of digital books for both Internet [6] and PDAs [7]. As part of the Constellation Project, we created a new framework for searching and displaying information on the Newton PDA. The software engine developed provided indexed searching for relevant articles or drug information. Linear find routines (in which every word in a document is searched to find a match) were also available. Indexes were searchable from alphabetic lists of key words. Besides indexes, the program also provided searching for tables from separate lists as well as a dynamically displayed table of contents. Inter-book navigation was also developed allowing switching between books at any time while reading. To facilitate data analysis, an audit trail generator program was placed on each machine to monitor the number of uses of each application.
Content Selection
With the permission of the Medical Informatics Division of the American College of Physicians (ACP), we adapted the ACP Journal Club (JC) and the Medical Knowledge Self Assessment Program IX (MKSAP) to PDA format.
These resources provided two distinctly different approaches to medical information. MKSAP is a review text of internal medicine. The JC is a collection of key article abstracts with commentary about specific topics and questions in the field of internal medicine.
Only subsets of both MKSAP and JC were used due to memory constraints. Chapters from MKSAP included cardiovascular medicine, endocrinology, pulmonary and critical care medicine, infectious disease, gastroenterology, and nephrology and hypertension. We selected abstracts and commentary for 1993 from the JC.
We also obtained permission from Education Research Laboratories, Inc. (Ft. Worth, Texas) to create a PDA version of the Monthly Prescribing Guide (MPG) which contained a subset of approximately 1400 ambulatory care drugs along with their dosing, side effects, and interactions.
Because the MPG subset did not have any intensive care drug information, a separate small reference containing the twenty most often used ICU/CCU drugs was obtained from the University of Pittsburgh Division of Pharmacology.
The BWH residency has a 90-page House Staff Manual containing therapeutics information that the residents prepare every year for distribution to the house staff. This, too, was converted to PDA format.
A residency phone book was prepared which contained the phone numbers of all hospitals in the residency as well as all resident beeper numbers for both MGH and BWH.
Lastly, we created a Medical Calculator to handle specific medical equations such as creatinine clearance, fractional excretion of sodium, arteriolar-alveolar gradient equation, and several others.
Experimental Approach
In May 1994, Institutional Review Board approval was obtained for the study.
During June of 1994, before they began their second year residency, the internship class of the internal medical house staff at BWH and MGH were surveyed as to their level of computer expertise and satisfaction with current access to medical information.
On July 13, 1994, two weeks after the start of their second year, 14 residents from each institution were randomized from the pool of residents who answered the initial questionnaire to receive and use a Newton PDA with the above mentioned reference material. Although residents were allowed to decline participation in the study, none did.
A one-hour training session was given to all participants in the study. The residents were also given the instruction book for the PDA and a video tape (provided by Apple) on computer use, care and up-keep. Battery-charging stations were established in both institutions for the purpose of exchanging and charging batteries. In addition, a help desk and a beeper service were available for the duration of the study.
Residents were instructed to use the PDA in any way they saw fit. All applications on the device, including the built-in functions of the Newton, were available for use. Participants were informed about the audit trail program running in the background. Twice during the study, audit trail information was downloaded from the PDAs, and any specific questions the residents had about use of the Newton were answered at that time.
At the conclusion of the study, the residents were re-surveyed about the same issues as in the preliminary questionnaire as well as several specific questions about the Newton, the applications, and their experience using the PDA in general. All computers were returned for data analysis with the exception of two devices that had been stolen during the study period (these two machines were replaced with in 3 weeks – audit trail information was lost in both cases).
EVALUATION
The main questions of interest were: (1) how and when were the resources used, (2) did the clinicians like the presentation of information in this new format , (3) what impact did having these materials available for immediate use have on work habits, (4) were the reverence materials adequate for the needs of the residents. Preliminary results have been previously reported [8].
Resident Surveys
A pre- and post-deployment survey tool was administered to the study group. The pre-deployment survey was administered to the entire internship at both institutions. Due to scheduling conflicts, 31/41 (77%) interns at BWH and 23/31 (56%) interns at MGH responded to the preliminary survey.
The preliminary survey (which elicited Lickert-scale responses) dealt with the following issues: current computer use, where and how often information is needed, which reference texts were currently in use, projected opinions about use of PDA if provided, attitudes about the content provided in the study, and attitudes and opinions concerning point of care and point-of-need information.
The post-deployment survey dealt with many of these same issues, plus questions that dealt specifically with evaluation of project software, satisfaction with the PDA, and opinions about the use of PDAs in general. Space was also provided for open ended answers to address further needs and concerns for the development team.
Audit Trail
The audit software ran in the background while the residents were using the PDA. It did not interfere with the normal use of the device. The start and stop times for access to any application used was recorded as well as power-on and power-off activity. All data was downloaded from each PDA during and at the conclusion of the study and placed into a relational database and a spreadsheet (Microsoft FoxPro 2.5 and Microsoft Excel 4.0) for analysis.
RESULTS
Demographic findings
The residents at MGH rated themselves on average as “computer novices” as compared with the residents at BWH, who rated themselves as “moderately-experienced users”. In addition, residents at BWH were more likely to respond that they used a computer “several times a day” due to their heavy use of BICS for daily clinical activity. There were roughly twice the number of physicians with graduate degrees in addition to the MD at the BWH, but the number of participants with graduate degrees in computer science was comparable (3 at BWH, 1 at MGH).
One striking finding is the use of the hospital information system electronic mail by the residents. At BWH e-mail is part of BICS. During the study e-mail was made available to the MGH residents as well. However, despite this addition to their system their response for usage did not vary much at the conclusion of the study.
Usage Findings
The PDAs were used a total of 3486 times by all 28 residents during the 100-day trial, with approximately 1.25 uses per day per resident. There were clearly several residents at each institution who used the PDAs on a consistent basis, (MGH,5; BWH,3). Likewise, there were residents who did not use the devices after approximately day 10, (MGH,2; BWH,3).
The most frequently used application was the MKSAP (686 uses), followed by MPG (drug reference) (671 uses). Two of the built-in applications, Address Book and Calendar/To-Do List were the next most frequently used programs (358 and 320 uses, respectively). Other resources in the collection were used approximately the same amount (range, 157-185 times).
A precipitous drop in usage was noted between days 1 and 10 (>1000 cumulative uses on day 1 to 50-60 uses per day by day 10. After day 10, a steady-state usage rate was observed, which was for the most part constant for the duration of the trial (0-25 cumulative uses per day).
There was more overall use at MGH vs. BWH when a book vs. usage analysis was prepared for each institution. However, both the Ranked Sum Test and Student’s t test of the usage data suggested that despite a larger number of uses at MGH, the usage differences were not statistically significant between institutions.
Post-Deployment Questionnaire Findings
Of responding residents, 96% requested additional resources on the PDA. The most requested references were therapeutics medical manual and a more complete drug book. One resident at MGH liked having portable information so much that he purchased a laptop computer with a CD-ROM built in and started to use that device approximately two thirds of the way through the study.
The majority of all residents felt that having a portable information resource made them more efficient when looking for medical information (22/28). In addition, they reported that having the PDA did not adversely affect their interaction with patients. They were uncertain if the presence of the PDA made a difference in the way they took care of patients and were also uncertain as to whether the device was useful in diagnostically challenging cases (21/28). All agreed that the interface to the textbooks was adequate for their information retrieval needs .
Seventy-five percent (21/28) of the residents felt that usage was dependent on clinical rotation. They also felt that if they had been given the PDA to keep they would have used it more often (15/28).
When asked to comment on what aspect of the project was “best”, 10/19 (53%) respondents reported that a combination of easy access via the interface and PDA portability.
When asked to comment on the project’s “biggest problems”, 10/22 (45%) said the PDA was too big or too heavy, 10/22 (45%) cited incompleteness of references, 6/22 (27%) mentioned that handwriting recognition was faulty. The majority of residents commented that they were at times frustrated by the omissions in content on the PDA.
When asked to comment on which materials deployed could be omitted in future versions, 8/15 (53%) suggested JC, 6/15 (40%) suggested MKSAP, 2/15 (13%) suggested the House Staff Manual. Textual comments also indicated that their dissatisfaction with these titles was due in large part to the lack of therapeutic information in these resources. Conversely, none of the residents cited the drug books, medical calculator, phone book, or native applications as items that should be omitted.
User Satisfaction
The consensus was that these devices were either useful or very useful to the residents. On average the residents felt the PDA assisted them in 12% of all cases at MGH, whereas they assisted in only 4% of cases at BWH (range 0-40% at MGH, range 0-10% at BWH, n=14 at each institution). Additional content was requested by 96% of respondents (n=28). The most commonly requested content upgrade was information that was problem- or therapeutic-based, the second most requested content was a larger drug database.
Among the aspects of the device and information provided that were considered positive were the rapid accessibility in a pocket format (noted positively by 47% of respondents, with no comment by others) and the presence of the drug reference database (noted by 26% of commenting respondents).
User training was not an issue with regards to usage. After the training session the residents felt capable of using the PDA and the software provided. This was consistent with the lack of calls to the help desk, the lack of pages to the beeper service, the lack of questions during data collection rounds, and the responses in the post-deployment questionnaire.
The primary perceived drawbacks of the PDA were the large size of the device relative to the amount of content that it held and its lack of connectivity to the MIS. The house officers generally felt that with the addition of a wireless connection to the MIS for order entry and data retrieval and larger memory cards permitting more or larger references the device would be much more valuable.
The majority (26/28) of residents noted that the program that ran the Constellation applications was adequate for the task. They felt that it was easy to learn and master searching on the PDA. On the other hand they felt that issues of handwriting recognition, while not needed for navigating digital text, would have made the device much more useful for keeping track of patient information on the PDA. Twenty seven percent of residents felt that had handwriting recognition been better they would have used the device for tasks other than referencing information.
DISCUSSION
Variability in use
The precipitous drop in usage after the first 10 days suggests a novelty effect of the devices in both institutions. MGH residents and BWH residents used the PDAs to different degrees during the study, however, statistical analysis indicated that these differences in usage were not statistically significant.
In addition to the novelty effect, the resources used were found to be somewhat incomplete. Due to memory constraints, various pieces of content were omitted from the PDA (including chapters of MKSAP and two additional years of JC). This led to a problem of frustration in several key areas of searching. As indicated above, a significant number of residents remarked that incompleteness of the provided texts was a major problem with the study.
The most often cited omissions were the chapters on rheumatology, hematology, and oncology in MKSAP. Other chapters were omitted but were not mentioned as concerns by the house staff. Due to only having a subset of information, residents reported frustration in not finding information that was expected to be part of a standard textbook of medicine. Several users reported that they became less likely to use the PDAs later in the study because of content incompleteness. Despite the omission of chapters, MKSAP was the most often used reference on the PDA in both institutions. It should be noted that at the time of the study only 4- megabyte memory cards were available (providing approximately 8-megabytes of text with compression)
, thus limiting the amount of deployable text. At the time of this paper 20-megabyte cards are available and cost efficient for use in PDAs.
Comments
This study is one of the first to investigate the use of PDAs in medical practice. Various problems arose during this trial that were not anticipated, such as the need to omit chapters of MKSAP. As memory cards become larger and less expensive this problem can be easily addressed in the future. In addition, PDAs as a class are new devices. The study took place only 10 months after the first PDAs were available for general consumption. It would be interesting to repeat the study with larger memory cards and more titles that dealt with therapeutics.
On the other hand, the house staff at both institutions were pleased with the applications as well as the concept. It was very clear that different types of content are needed in the future. The most requested book was a therapeutics manual followed by a more complete drug reference. Since none of the applications provided dealt with therapeutics, the available content was less useful. While the drug book provided was complete for the subset of drugs that it covered, it did not cover the entire range of drugs available for use. Instead of using only the ambulatory care subset, it might have been a better decision to select drugs that were on formulary at both institutions and include that subset of the drug book. This would have ensured that specific classes of drugs, such as critical care or chemotherapeutic agents would have been in the PDA. With regard to the user interface to the custom software developed, the majority of residents felt that it was both usable and useful.
While this subject group is small, the study implies that PDA devices may fill some as yet unmet need by house officers. In the future, as PDAs become more powerful, wireless connections to the MIS become more practical, and new applications are written, these devices may well play a significant role in meeting the day-to-day information needs of physicians.
ACKNOWLEDGEMENTS
Funding in the form of content, equipment, and programming time have been provided by the following:
- Grants LM 07092 and LM 04572 from the National Library of Medicine
- K2 Consultants, Nashua, NH
- Apple Computer, Inc., Cupertino CA
- American College of Physicians, Philadelphia, PA
- Education Research Laboratories, Inc., Ft. Worth, TX.
In addition, many thanks to Ron Rouse for his invaluable input.
REFERENCES
[1] Labkoff SE, Shah S, Lee YJ, Greenes RA. The Constellation Project: Access to Medical Reference Information Using Personal Digital Assistants. Proceedings of the 18th Annual Symposium for Computer Applications in Medical Care. Hanley & Belfus, Inc. Philadelphia, PA. 1994;1024.
[2] Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985;103:596-9.
[3] Osheroff JA. Forsythe DE, et al. Physicians’ information needs: analysis of questions posed during clinical teaching. Ann Intern Med. 114(7):576-81, 1991 Apr. 1.
[4] Osheroff JA, Bankowitz RA. Physicians’ use of computer software in answering clinical questions. Bull Med Lib Assn. 81(1):11-19, 1993.
[5] Fishman SM. Palmtop Computers on the Medical Wards. JAMA . 267:1, 169.
[6] D’Alessandro MP, Galvin JR, Erkonen WE, Lacey DL, Curry DS, Holtum EA, Santer DM. The Virtual Hospital: Creating and Organizing a Ubiquitous Health Sciences Organization on the Internet. Proceedings of the 18th Annual Symposium for Computer Applications in Medical Care. Hanley & Belfus, Inc. Philadelphia, PA. 1994; 1061.
[7] D’Alessandro MP, Galvin JR, Santer DM, Erkonen WE. Hand-Held Digital Books in Radiology: Convenient Access to Information. AJR. 164:485-488.
[8] Labkoff SE, Shah S, Lee YJ, Greenes RA. Patterns of Information Resource Access in Patient Care: A Study of the Use of Portable Computers to Support the Clinical Encounter. Proceedings of the 1995 Spring Congress of the American Medical Informatics Association Spring Congress. Bethesda, MD: American Medical Informatics Association: 60.
Questions? Contact Sandeep Shah or Steven Labkoff, MD