A History of Risk Aversion
Legacy control systems are coming into an era of distinction due to the advancement of technology. Traditionally, cutting edge technology is not willingly accepted into the industrial sector due to the potential risks and hazards associated with unproven tools. This hesitation can be broken down into two major contributing factors: baby boomers’ resisted the change in technology, and early Microsoft platforms proved to be very unstable.
These two factors caused a very slow introduction of technology into the industrial control system sector. Now that the baby boomers are retiring, a new breed of engineers are coming into decision-making functions, and the acceptance of technology is much more prevalent. As the new generation of engineers enter into tenure, they are faced with challenges that are not fully understood, and are being forced to navigate the murky waters of risk. Taking advantage of technology enhancements – and introducing a level of unwarranted risk – is a fine line to walk when evaluating all of the options presented while migrating a legacy control system to a new platform.
The information below is intended to provide solid rule-of-thumb concepts that can potentially avoid you from having that uncomfortable conversation with your superior.
Implementing projects and risk management are two things that are linked together through circumstance. Providing a successful solution to meet management-backed objectives essentially boils down to “make good decisions that avoid as much risk as possible.” Obviously it is impossible to avoid risk altogether, but minimizing the exposure of the core factors can lead to the difference between a success and failure. The process of data collection, evaluation and analysis, researching similar experiences, and ultimately making the safe decision is the basis of sound engineering practices.
Alternatively, going out on a limb and interjecting additional risk to achieve a successful project should be well understood by management prior to cutting that path. The concepts listed below provide are fool proof and field tested to minimize risk and provide the best opportunity for a successful project.
Avoid Being Serial Number 001
The first in line is not always a safe place to be when navigating uncharted territories. This is a valuable lesson any serviceman has learned in combat! As engineers, we are taught to be conservative in our decision making process, and liberal in our design process. These methods ensure the highest statistical potential for success and minimize the risk. Therefore, when we approach projects as professionals we do not want to be the first in line to encounter unforeseen circumstances.
Technology’s cutting edge can also be known as the bleeding edge. When evaluating the technology for use in a process control system, you want to ensure that the platform is stable, proven, and tested prior to deployment into your process. “Careful to step and quick to stop” is always a good motto when navigating the process control jungle. The basic principle here is to avoid being serial number 001.
Select Proven Technology
The next case in avoiding undue risk is to not implement proven technology into an unproven application. A perfect example of this is the placement of basic business Ethernet cabling on the plant floor. Ethernet cabling is a standard business practice and has been a proven method for IT integration. However, when the exact same IT practice is implemented on the plant floor, bad things happen due to the ElectroMagnetic Interference (EMI) from associated equipment.
This led to the super slow introduction of Ethernet among industrial applications and the emergence of Cat 6 cabling, which includes a braided shield to prevent EMI from disturbing the digital data passing through the copper conductors. I would not want to explain to my superior that it works in the office just fine but we can’t get that device to communicate with our system. The underlying point here is to not stretch technology to work in an unfamiliar application, and only use proven technology to deploy a process control system.
Don’t Skip Steps
Control system design is a process in itself and it is never good to skip steps. Each step has value and is equally important when developing a final solution. Statistics have revealed that the most common reason for project failures can be traced back to the specifications, which could include the Functional Design Specifications (FDS) and/or the Detail Design Specifications (DSS). These documents do not have to be 100+ pages, but they do provide the basis for project objectives and they clearly outline the criteria for success. As budgets are squeezed and belts tightened, the most important steps toward success may sometimes be skipped.
Some projects might survive without developing front end documentation, but most will absolutely fail because of it. The alternative is having enough money and time (schedule) to develop work-around solutions on the fly to keep the project on track. This is risky, and not a safe place live when the loss of production is based on your decision making skills. The costs for the resulting work-around solutions can be avoided by not skipping steps of the design process. The old saying of “pay it now or pay it later” applies but the issue is the pay it later may have additional repercussions that make the total cost go up. To ensure a successful project and minimize risk, do not skip steps and allow potential risk to live in your house.
Test, Test, Test
Testing is a crucial portion of the project and it is the last opportunity to identify an issue before entering the field. Implementing a minimal or general test is just as risky as not performing any testing. A thorough test shall include the evaluation and/or verification of every item defined with the Functional Design Specification (FDS) and the Detail Design Specification (DDS) document. These documents shall be developed, reviewed, approved, and used to configure and test a system.
Hence, the importance of the principle above (not skipping steps). Not only should the testing be documented, but it should incorporate personnel that are familiar with the system. Sometimes, the FDS and DDS documents are approved by someone that is not aware of a processing detail associated with the operation of a particular piece of equipment. This is a surprisingly common scenario that will require someone with real-work operational experience – specific to the process at hand – to prevent.
It is always smart to get someone from the production side involved with testing to assure a successful project.
The deployment of a newly designed control system is the final piece of the puzzle, and it can make or break a successful project. I observed this on a project that was designed properly and tested thoroughly, but during commissioning, a few I/O points were physically landed on the wrong terminal. The automatic insertion of an NIR probe became damaged, which was extremely expensive to replace. The root cause was not in the specification or software configuration, but in the commissioning phase – the field representative was watching the wrong valve.
This could have been avoided by creating a commissioning procedure, which includes field walk-down of the associated equipment to verify proper tagging identification, wiring practices, and field operation in a specific order to avoid the hazards. This documentation involves both engineering and operations personnel to completely commission a system thoroughly.
- Technology’s bleeding edge is not a safe place.
- Only use situation-specific, proven technological applications.
- The Design Process has value; be careful to not oversimplify.
- Utilize experienced production personnel for testing.
- Plan your commissioning and then commission your plan.
Migrating a control system invokes unavoidable risk, but it can be minimized by following these principles. These points are derived from years of project experiences and should prove useful as more and more migrations are required. If you are interested in legacy migration, researching legacy migration, or have a comment or question about legacy migration, I’d love to hear from you.
Contact the Author
If you have any comments or questions about the material, please do not hesitate to contact me through the comment section. I would love to hear your answers to the questions as well.