It's a nightmare that facility operations teams dread. 

They are three and a half hours into a four-hour mandatory test of a critical power system when the monitoring and control system freezes. Real load falls below the required 30 percent. The team takes control manually and works through a time-consuming reboot. Its only recourse: reschedule the entire test. Now that's a nightmare. 

Seared into the team's memory, the experience is one example of a power monitoring and control system past its prime.

"Its operational deficiencies included logging itself off, 'losing' a group of automatic transfer switches from its database and operating very slowly due to its serial communications," said Chris Liedman, operations manager of engineering and maintenance at Park Nicollet Methodist Hospital in Minnesota.

The 426-bed hospital, with more than 3,200 employees and about 950 staff physicians, is renowned for high quality patient care, medical expertise and disease management. It's part of HealthPartners, the nation's largest, consumer-governed, nonprofit integrated health care and financing organization. HealthPartners serves more than 1.4 million medical and dental members and about one million patients, primarily in Minnesota and western Wisconsin.

Although recognized for its patient care, its testing of critical power was an issue. Monthly tests, for example, required transferring one automatic transfer switch at a time from utility to emergency sources. 

The waiting game is no game

"We would wait and wait. And wait," said Bill Tester, facilities manager. Getting all the hospital's 34 automatic transfer switches on line took 15 to 20 minutes. The operations team scheduled an hour on a Wednesday evening each month from 9 p.m. to 10 p.m. for testing.

"The team would hope to complete it in an hour, but often couldn't because of rebooting," Liedman explained. The process took so long they had to reconfigure time delays, or manually transfer each of the 34 transfer switches one at a time...twice.

"It was a very big hassle to conduct monthly tests," Liedman said, "And four-hour tests became  nearly impossible with the system."

That's when the operations team decided to bring its critical power management process into the digital revolution with the help of Emerson Network Power's ASCO® Power business. The revolution is combining advanced electronics and wireless technologies in ways that are changing facilities management.

It's helping manage the increasing complexity of buildings, enhancing their efficiency, ensuring business continuity and emergency preparedness, solving the problem of making decisions based on too little data, and doing more as budgets shrink.

Satisfying the "must haves"

To say the team had a critical eye when they evaluated a new critical power management system is a bit of an understatement. Their list of 'must haves' for a new system included a proven track record of managing critical power for a multi-building campus, thus providing true, interconnected facility management. 

Not surprisingly, other 'must haves' were the ability to better manage multiple automatic transfer switches, the speed to quickly manage the volumes of data generated during tests and producing compliance reports automatically. 

Technically, system performance would need to include distributed processing, prioritized and dynamic data updating, dynamic throttling, diagnostic logging and 1000 Mbps Ethernet. In fact, the system needed to be able to monitor a range of critical power equipment and components, such as automatic transfer switches, paralleling control switchgear, gen-sets, multiple brands of circuit breakers, bus bars and other devices from different manufacturers. 

Compounding the monitoring challenge were  the equipment's varying ages and capabilities for sharing operational information and the different platforms used by equipment vendors. 

System performance also would need power quality analytics to evaluate and diagnose data so the team could make real-time decisions on day-to-day system operation. 

Issues 'A thing of the past'

That was about a year ago. Today, Liedman and Tester report that problems with testing and managing critical power day to day are a thing of the past. The new system the hospital installed, the ASCO 5750 PowerQuest Critical Power Management System, meets the list of 'must haves.' 

What they like best about it is, as Tester said, "It works!" 

Liedman said, "It responds instantaneously. The big win for us is automated reporting and trending, which was all new to us. Before, everything was handwritten. Now, we generate reports that are tailored to meet the requirements of healthcare facilities so they're perfect for our monthly run report. That hits the mark for us."

Trending shows them power demand history so they know where they have adequate capacity and, more importantly, where they don't. They overlay that with future construction plans, which helps them project critical power system scaling to keep it in lock step with hospital growth and kW demand.

And while they still manually enter some data, their confidence in digitally managing critical power continues building and at some point they see themselves relying exclusively on auto-generated data. They also can produce reports on bypass status, critical power settings, diagnostics, alarming and historical logs.

Managing the power infrastructure

They manage the hospital campus' entire power distribution system, beginning at the true dual 13.8 kV back-to-back utility source feeds. 

The on-site power feed design is a main-tie-main configuration with six pairs of downstream, end-to-end substations. Transformers step down power to 480V. The central power plant comprises three 480V, 1000 kW gen-sets. They provide N+1 power redundancy as the total load on the gen-sets is about 1500 kW. 

"Reliability is a big concern of ours, of course," said Tester, "so the redundancy helps ensure power will be there when it's needed." Ensuring reliability also is why they are about to replace the system's 25-year-old generator paralleling control switchgear that synchronizes and manages the gen-sets. 

Thirty-four automatic transfer switches help power loads and range from 150 to 1200 amperes, with most rated between 400 - 800 amps. They are a variety of bypass-isolation, closed transition transfer, open transition transfer and delayed transition transfer modes of operation to satisfy the demands of varying types of loads--from data center compute equipment to motors. 

Liedman said, "With the dual feeds, power generation and transfer capacity and new critical power management capabilities, we have a power infrastructure that's hard to beat." 

They conduct tests with real loads, which include life safety and critical branches, clinical equipment such as MRIs, some business occupancy spaces and a data center with a 125 kVA uninterruptible power system used as a ride through. 

"We're transparent with staff when we conduct a test," Liedman said. "We notify them as a courtesy and they almost never realize there's a difference in the power source." 

Dynamic visualization and the boiler operator

Monitoring critical power 24/7 is a given, of course, so overnight the boiler operator relies on the power management's dynamic visualization to present information in an easily understandable format. 

"The one thing to remember is that modern critical power management makes our regular duty easier," Tester explained. "It's more reliable and secure." In fact, it's protected with 128-bit AES encryption, the same encryption used by NASA. The rule of thumb about data security is that if it has value, it should be encrypted.   

Transitioning to fuller, faster and more accurate and capable critical power management is part of the team's effort to support hospital operation.

"It's all about competing pressures on the provision of healthcare," Liedman said. "It's a very competitive healthcare environment here." Although patients don't see the critical power infrastructure, it has to ensure hospital operations don't miss a beat should the utility source go down. No more nightmares, just sweet dreams.