In our blog Do You Really Need a Specification Document, we began exploring the value of using Functional Specification (FS) and Detail Specification (DS) for your control projects. These documents serve as a critical pathway for upgrading complex systems to newer versions or migrating to different platforms. Many plant distributed control systems (DCS) or programmable logic controllers (PLCs) have run well for years and, in a perfect world, would never need to be updated. A common perspective for the plant personnel is: “If I can predict and maintain the issues in my system, why should I buy a new system and learn a new process?”
This perspective reflects an optional mentality toward wanting a system upgrade. In our imperfect world, the want-to upgrade will eventually become a need-to upgrade. However, with the right collaboration during project planning, the plant personnel will not only experience the value that specification documents bring to the upgrade process, but they will also have a better understanding of how a successful upgrade replaces obsolete systems with new functionality that benefits all stakeholders in the process.
For clarification, specification documents can be used on migrating systems, upgrading systems, or creating new systems. As we focus on reducing the risk of obsolete systems through the specification process, we will refer to two possible paths–migrating to a different system or performing a version upgrade within the same platform.
Since that strategic choice needs to be thoroughly evaluated during the specification stage, both terms are used as examples in this blog. Version upgrades usually occur when staying with the same platform. Updating hardware and programs within an Allen Bradley ControlLogix system or a Siemens PCS7 system would be a version upgrade. Migrations occur when changing from Vendor A’s system to a Vendor B’s system. An alternative scenario sometimes happens when a vendor no longer supports a platform and a full migration is required, even if staying with the same vendor.
Minimize Risk and Cost of Unexpected Downtime
Change is inevitable due to the obsolescence of hardware and software, as well as the availability programmers with experience on those process control platforms. Once hardware parts are discontinued their price rises exponentially and depending on the system can cost tens of thousands of dollars. In some cases, they may only be available online through unofficial distribution channels. The unofficial online option for control hardware is also temporary and greatly increases the plant’s exposure to risk since these are not supported by the manufacturer or distributor.
If obsolete hardware costs aren’t prohibitive enough, programming changes may be more cumbersome based on the system architecture and industry standards available at the time the system was installed. Since older PLCs had limited capacity, they often required some functionality to be located in the human machine interface (HMI) processor or alternate locations. Programming issues occurring in a combined control system often requires troubleshooting both the PLC and HMI programs and carries the assumption that the plant owns the configuration software for both programs.
Additionally, smaller PLC systems tend to have an HMI database and a PLC database which requires extensive engineering time to maintain, whereas the newer systems typically combine a centralized database for both PLC and HMI. This reduces the overall engineering effort to produce a solution. It also provides an easier system to maintain as well. To connect the financial dots, the increased functionality of newer systems can reduce the number of overall control systems. As a result, less human and financial resources are needed for programming and troubleshooting the system.
Automated reporting functions have also greatly improved over time and can either be a standard feature or an optional feature for the new systems on the market. If the collaboration process identifies the key data points used in the plant’s mandatory reports, then a process historian can be added to the system architecture. The historian stores that key information in a long-term database. When the report-viewing software is customized to meet the requirements, it can publish the historian data in the proper format with the click of a button compared to the hours required for the manual reporting process. This not only reduces labor cost but also improves the quality of the reporting by reducing the chance of human error. There are many additional benefits of a historian that will be addressed in a future blog topic.
System failure is the third strike of an obsolete system. When an obsolete system crashes, extracting the programming may no longer be possible. In older system architectures, perfect storms can occur where the main program and backup programs, if there are any, are lost. Since older control systems typically run in conjunction with an obsolete computer’s operating systems (OS), a migration not only includes changing PLC hardware but also engineering servers and workstations. Finding an operational computer for a discontinued platform can be just as challenging and risky as replacing the other system components.
The Easy Way or the Highway
A broken-down car can be a real-world equivalent to unexpected downtime in plants. One question to ask is whether it’s preferable to replace a car when the current one is still operational or to wait until the active car breaks down on the highway en route to a critical meeting. The prior situation is handled in a controlled manner, allowing more time to prepare for the best replacement, and the latter is acquired under duress caused by the expense of towing, the opportunity cost of missing the meeting, the emotional response to stress, and the greater risk of not receiving a car that meets a majority of your requirements. So if there are warning signs on a system indicating eventual failure, why wait until it shuts down permanently the midst of a critical production run? Loss of that production will cost much more in the short-term and could jeopardize long-term business with affected customers.
In Case of Emergency, Plan!
In unexpected downtime situations, scheduling also plays a major factor in increased project cost and prolonged outage. Integrators often assign their resources to work on multiple projects. Since workload varies at different stages of a project, integration engineers are typically moved during holding patterns in a project. Conversely, there are certain times when resources are not available for reallocation because of the critical stage of their projects. For this reason, engineering resources on major projects are scheduled weeks or months in advance. If a plant suddenly needs a significant amount of onsite time, project costs will most likely increase due to adjusted travel plans, altered schedules, increased project management needs, and required overtime/weekend hours, along with other potential costs for plant personnel and production loss.
Instead of collaboratively planning the pending migration over several weeks or months during standard workday hours, the project executed on emergency conditions condenses engineering efforts into 24/7 urgency. As a result, hourly rates for overtime and weekends will make this option extremely more costly. Even with round-the-clock efforts, it still could take several weeks to get a new system running from a complete failure. In those situations, specifications for the system may be compromised in the name of expediency.
Once the specifications are complete, the design must be approved before the new hardware parts can be ordered, the new panels assembled, and the new programming tested, installed, and commissioned. Each of those steps can be very time intensive and are best completed in a controlled timeline compared to an expedited schedule. If it is necessary that the programming or specification engineers need to have familiarity with the obsolete system, then most likely a smaller pool of resources are available and the need to plan further in advance becomes all the more critical.
Turnkey or Not Turnkey
The traditional bidding method for industry includes an internal expert or team drafting up bid specifications and having multiple parties bid the project. This approach leaves the goals and strategy of the project open to interpretation. The plant’s project team will think they clearly communicated their requirements based on their process experience, but the bidders will have different interpretations based on their engineering philosophies and experience. How does the specification process solve this dilemma?
If a plant was awarding the entire turnkey project, the functional and detail specification process would be handled at the start of the project. By splitting the functional and/or detail specifications out from the entire project, the bidders can quote the development of the specifications only. Once the specifications are developed, they can be used to bid the remainder of the project and help ensure bidders are quoting the same materials and overall strategy. This will identify the platform and overall input and output count along with other key pieces of information needed for accuracy in quoting.
This process is often done in other aspects of projects relating to construction and mechanical aspects, but not always applied to controls. Often, control requirements are tackled last by the project team and that can have a major impact on the project outcome. By performing the functional specification earlier in the process, there will still be some variation among competing bids but the plant should feel confident about choosing between similar solutions.
Also, by having these sections complete, prior to bidding the whole controls project some unexpected costs or issues may be discovered that could have derailed the project otherwise. If the result of the specification process is a higher cost than anticipated, it could either require phasing it in strategic ways, reducing the scope in certain areas or rebidding in a few months once the capital is available. Even if the remainder of the project was temporarily delayed and the plant system failed during that time, the plant could possibly start the project at either the design process or the ordering of hardware and be much further along in their project.
Maximizing the value of specification documents begins with seeing the value of the specification process. Here’s are the reasons for using specification documents that we’ve identified so far:
- Obsolete systems and hardware are typically more costly than current platforms, harder to find from approved distribution channels, and expose the plant to risk if bought elsewhere.
- The functionality of new control systems vastly surpasses obsolete systems in capacity and reporting functions. These are just two examples of improvements that can offer plant personnel time-saving efficiencies.
- Programmers with experience on obsolete systems may not have on-demand availability, which could prolong downtime.
- Migrating a system once it’s failed is more complex, if not impossible, requiring overtime and significantly more manpower to get a new system running.
- Depending on the system, it could take weeks or months of collaborative effort to implement a system that meets the plant’s goals.
- Even an expedited schedule is dependent on design approvals as well as the lead time for parts, assembly, testing, installation, and commissioning. By starting the specification process earlier, in a controlled setting, the project can start at a more advanced point at a later time. It could begin the design or ordering of parts depending on the level of detail in the approved specifications.
- By bidding the specification process separately from the whole project, a more accurate scope can be provided to bidders, which can reduce the variance in estimated cost and help ensure similar solutions are being offered.
- If the project is delayed following the specification process, the plant has useable documents that can aid project timeline when work resumes. If both the functional and detail specifications were completed, then the drawings and control narrative should already be approved. When the remainder of the project is awarded, the winning bidder should be able to order the materials and begin programming.
These are some of the many benefits in developing specification documents. In our next blog, we’ll discuss the impact of code quality on the migration process. By evaluating the criteria from the plant personnel and system integrator’s perspectives, a more realistic picture of the project specifications, costs, and timeline can be developed. Lessons learned about the code can become guideposts for the new system’s programming requirements and help ensure a successful project outcome.
