I received a phone call, on a Friday evening, in 2016 while sitting down for dinner. The on-duty Control Room Operator (CRO) reported all of the steam turbine process values had stopped changing and he had no control over the steam turbine and its associated equipment. The plant is a 250 MW, combined-cycle power plant with a 501FC+ gas turbine, a Westinghouse steam turbine, and an Emerson WDPF Distributed Control System (DCS).
Upon arrival, I found both the primary steam turbine DPU (aka controllers) had failed. The backup DPU had also failed. The I/O was froze, sitting with values from the time the failures occurred. We were fortunate the plant load happened to be steady when the failure occurred. We performed a normal plant shutdown and declared unavailable.
I checked all 4 power supplies and found all were showing < 10.5 VDC. After adjusting them back to their nominal output (13 – 13.5 VDC), I cleared both DPUs’ memories, performed an object code reload from the most recent backup, restarted the primary DPU, and copied to the backup DPU. After a few operational checks, we declared available and were back in business.
The Maintenance Manager came in later and we found almost all of the other DPU power supplies were low and needed adjustment. Further investigation revealed there was no PM’s specified to perform these routine checks. We were fortunate this incident only cost us a few hours. Had the plant been changing load when the failure occurred; things may have been much worse.
More recently, I found CPU errors on most of the T3000 AFC CPU’s at another combined-cycle plant. Further investigation revealed that most of the power supply modules had at least one dead battery.
So, make sure your plant has the necessary Preventative Maintenance (PM) plans for the DCS. This includes software and archive backups.