by Elizabeth Ferrarini

Staying on stop of best business practices in IT -- especially for privacy, security, and new technologies -- has become a hallmark for the CIO at one of the largest teaching hospital organizations in the United States. Dr. John Halamka has managed to combine his training as a medical doctor with an innate ability to understand all aspects of computer networking.

Dr. Halamka oversees the IT needs for CareGroup Health Systems' three major Boston-area hospitals -- Beth Israel Deaconess Hospital, Mount Auburn Hospital, and New England Baptist Hospital -- and three community hospitals. Together, the six CareGroup facilities have about 12,000 employees, including 3,000 doctors who see about one million patients per year. Halamka is also an associate dean of Harvard Medical School where he spearheads all of the technology programs.

Halamka got a jumpstart on EDI long before HIPAA came along, and his security and privacy practices at CareGroup appear as a case study in a book by the National Academy of Sciences. He took a minute to answer some questions about what he has been doing in EDI, security and privacy, how he keeps up with technology, what he learned from an outage that plagued two hospitals for almost two days, and what types of technology he uses every day.

EL: Can you summarize the high points of your entire network  infrastructure?

JH: About 225 employees maintain the IT infrastructure consisting of 8,000 desktops, 32 terabytes of storage, and 25,000 network ports throughout the 45 miles of wide area network (WAN). A 155MB per second SONET backbone connects the WAN. Most of the networking gear -- firewalls, virtual private network (VPN), routers, and switches -- comes from Cisco. Either Hewlett Packard UNIX servers or Compaq Windows 2000 servers front end several EMC Symmetrix storage area networks. A StorageTek tape library handles all enterprise backups.

EL: Once you were finished planning for Y2K, you had to start worrying about HIPAA. How did you lay the preliminary foundation for HIPAA requirements such as electronic data interchange (EDI)?

JH: Back in 1998, even before Y2K, the CIOs of our provider organizations formed a consortium to enable the entire New England payer provider community to create EDI transactions among ourselves for free. The New England Health EDI Network went live in 1999 before HIPAA EDI transactions for benefits and eligibility.

Since that time, we've used a common infrastructure -- basically Napster for healthcare -- or point-to-point interaction using a VPN between payer and provider. The VPN sends encrypted transactions through a common gateway we've built for referral authorization and our claims, and Web status inquiries. In October 2002, we completed all of the EDI HIPAA transactions for New England.

EL: Privacy is a challenging area for all types of organizations. How would you rate your privacy best practices for the past few years?

JH: I'd rate them as excellent! We're one of the test cases  featured in the leading book about healthcare privacy. For The Record --  Protecting Electronic Healthcare Information, published by the National Academy of Sciences, covers best practices in authentications and access control, auditing, physical security, and disaster recovery.

EL: What kinds of initiatives do you have in place for  privacy?

JH: Since the early 1980s, we've been auditing every transaction that goes through any one of our clinical systems. We've got a Web site called PatientSite where any one of our patients who has received the appropriate authentication credentials can review his or her security audit online. We can also give a patient a printout of the security audit.

We've got a strict no-tolerance policy for confidentiality violations. About three or four employees get terminated every year because of these violations.

EL: What have you been doing to increase privacy?

JH: Each employee needs to be completely trained in all aspects of privacy. For example, every patient needs to be notified about our privacy policy and to sign off on it. A patient needs the opportunity to opt out of certain things, such as automatic enrollment in fundraising activities. We require a great deal of manpower to train our 12,000 employees. So we've selected individuals from key departments, such as IT, human resources, and medical records, to work together to conduct training sessions.

EL: You can't have privacy unless you have security. Unfortunately, HIPAA still doesn't have a hard and fast security rule right now. How did you decide what best practices to use?

JH: You need to sort of make one up. In other words, ask yourself, what are those security elements that are absolutely required to meet the privacy regulations, effective April 2003.

We've had some very good security best practices for many years. For example, every Internet transaction always has 128-bit secure sockets. All strong authentication passwords must have a minimum of six characters, consisting of alphanumeric characters; these passwords expire in 90 days.

Based on the information in For The Record, we created a grid to rank the security provisions for each one of 400 different IT systems. Because there is no security rule, we're not sure if 128-bit secure sockets are good enough. What about Triple DES? We looked at all of those things that didn't meet the spirit of best practices. We've begun to remediate, for example, systems that didn't have passwords or didn't have audit trails.

EL: What are your feelings about security technologies such as PKI  and biometrics?

JH: We tried PKI about four years ago. It didn't work for us. Maintaining 12,000 certificates for that many employees can became an administrative nightmare. We use PKI, in one sense, to do secure email between our trading partners. A company we use offers a secure, SMTP gateway for certification exchange between organizations. Each transaction remains encrypted as it travels over the public Internet from payer to provider or between two large provider organizations. These aren't personal certifications, but organizational ones.

Biometrics doesn't work very well in healthcare. You can't have false negatives. Imagine you're attending to a critical patient. You can't get the patient's chart because the patient has a sweaty thumb print.

EL: Is there any special device you use to handle  authentication?

JH: We use a device from BlueSocket on both our wireless and our wired networks. The device hits the LDAP directory. We think WEP or the wired equivalent privacy protocol isn't sufficient. It uses a single key for all clients. Once someone cracks the key, your security is compromised. With the BlueSocket device, you need to specify your user name password in order to access an application.

EL: Shifting gears from security and privacy, what types of new technologies are you considering that will enhance the quality of care physicians provide to patients?

JH: We're carrying out RFID to track critical medical equipment in the emergency department using devices from Pango Networks. Over the next year, we'll be using bar-coded wrist bands, bar-coded medications, and bar-coded employee badges to track medication administration.

We have two million square feet of wireless to ensure our clinicians have all of the information they need to deliver quality care.

EL: Several years ago, The Boston Globe and all of the computer trade press publications carried the story about a network outage at two of the CareGroup hospitals. Can you briefly tell what happened and what you learned from the experience?

JH: On Wednesday, November 13, 2002, the network experienced a major slowdown for three days. The CISCO technical support team found the Layer 2 structure of the network to be unstable and out of specification with 802.1d standards. The management VLAN in some locations had 10 Layer 2 hops from root. The Spanning Tree Protocol (STP)  imposes a maximum network diameter default of seven. Thus, two distinct bridges in the network should not be more than seven hops away from one to the other.

A major contributor to this STP issue was the network and  Picture Archive Communication System (PACS) network, for sharing high-bandwidth visual files and other clinical data; this was 10 hops away from the closest core network switch, three too many for the spanning tree to handle. To eliminate its influence on the CareGroup network, we isolated it with a Layer 3 boundary. All redundancy in the network was removed to ensure no STP loops were possible.

I learned that infrastructure must be lifecycle-managed per a multi-year strategic plan and not simply replaced at end of life. You need to retire legacy network. You also need to demand review and testing of network changes before you carry them out. Good downtime procedures must accompany each application we carry out. Another lesson is that a disaster recovery plan addresses all the details of a disaster. You need to plan employee logistics, communicate realistically, prepare baseline backups, and focus disaster plans on the network, not just the integrity of the data.

EL: One of your colleagues said that you're really a bionic CIO. What  types of devices do you carry with you at all times.

JH: I'm connected at all times and on call at all times. I have a Blackberry 7290 (Bluetooth enabled GSM/GPRS phone), which I use to answer 500 daily emails. It's also fully integrated via Bluetooth into my 2005 Toyota Prius so I'm completely connected when I drive. I also carry a nationwide pager for redundancy. My medical information is implanted in my right triceps, should I ever need medical care.

--

Elizabeth Ferrarini is a free-lance writer from Boston,  Massachusetts. Reach her at iswive@aol.com.