Smelly stuff all down the street
The metaphor we often use to describe the consequences of a control failure involves smelly liquid being pumped down the street. Well, something much like that happened to a customer in Ghana recently, though the material in question wasn't all that odiferous. Here is the story, in his own words, from Bennedy Otu Ansah, an automation contractor with iPtech in Accra."I want to share with you an experience I encountered about four weeks ago on my Lube Oil Project. The pressure transducer broke down and I had not factored in this possibility in my program. So as it should be, my SPLat program saw this as zero pressure and called in the main pump. This continued to run, and built up the pressure until it cracked the Jockey Pump and spilled water all around. This was a weekend and it continued until the Monday morning, when the Plant Engineer came and pressed the stop button to turn the system off. The security attendants had not yet been trained to stop the system. The plant Engineer confided in me that he had a phone call, but did not pick it up.
"Anyway to them it was the maximum pressure of 10 bar which was too much for the Jockey, and that was why it cracked. So they called me in to reduce the pressure to 8 bar. It was at this point that I realized the transducer was not working. It has since been replaced and I have inserted a zero pressure code to take care of future re-occurrence. I wouldn't mind if you share this experience with your clients, our family."
So what's the moral of this story? Clearly sh#t happens. More importantly, what the control system does when sh#t does happen is of vital concern. When you are designing a control system, you should always be mindful of what will happen when things go wrong. What if that valve sticks? What if the float switch jams? What if, as happened in Ghana, the pressure transducer fails? Or if, say, a tsunami knocks out the reactor cooling system? The more serious the consequences, the harder you need to think about it (and not let bean counters make engineering decisions!).
First you need to be able to detect that something is wrong. That's what Bennedy in Ghana is now doing by checking that the pressure reading is within reasonable bounds. Then you have to decide what to do if your program detects a problem. Usually this will be to raise an alarm or switch the system to a safe condition.
With sensors like a pressure transducer an out of range reading can often be detected. Industrial transducers use 4-20mA, not 0-20mA as their active range for that very reason. 0mA means broken wire or broken transducer. The good old, almost-always-works, standby is to use a timeout. In any sequential logic where one step must be completed before something else can happen, timeout can detect a problem.