Large or widely dispersed control systems often have multiple CLx controllers which must communicate with each other. Common goals for inter-controller communication include speed, reliability, detecting failures without nuisance warnings, minimizing network impact, and the ability to modify communication on-line.
Ethernet / IP
While it is possible for controllers to communicate with a wide variety of protocols (ControlNet, DeviceNet, DH, DH+, serial such as RS-485 or RS-232, MODBUS+, Profibus, etc…), CLxs generally communicate with each other over the same Ethernet networks over which they communicate with HMI systems. This should be a network separate from the DLR (device level ring) each processor uses to communicate with its I/O.
A wide variety of protocols is available over Ethernet; CLxs natively communicate via Rockwell’s Ethernet/IP. Third party cards or complicated socket programming are required to use other protocols such as MODBUS/TCP.
User Defined Types
The key to efficient CLx-CLx communication is to put data to be transferred in User Defined Types (UDTs). UDTs allow data of different types to be combined into a single controller tag. If the UDT is 500 bytes or less, it may be transferred together as a single Ethernet packet, so that all the data it contains update at the same time.
UDTs may not be modified on-line – changing them requires a download, so plan ahead! See part 1 of this series for details on proper UDT design.
UDT Requirements for Communication
Communication between CLxs occurs via tags of the same type. Using UDTs as recommended here means that the UDTs:
- Must have the same name on both sides.
- Must have the same member types in the same order on both sides. This includes arrays – if you have an array REAL[20] on one side, and a REAL followed by an array REAL[19] on the other side, communication will NOT work.
- However, member names do not have to be the same on both sides. Ideally the UDTs will be identical, but when spare members are used, it is possible to rename those members & download only one side. The link will still work in the interim before the other side is available to be downloaded.
In addition, UDTs used for produce / consume communication (see below) must be ≤500 bytes.
By contrast UDTs transferred by the MSG blocks recommended here can be any size.
UDT Pairs vs Publishing
For a pair of CLxs communicating with each other, there will be the same two UDTs in both CLxs. In Proc1 there would be tags:
- “to_Proc2” of type Proc1_to_Proc2
- “Proc2” of type Proc2_to_Proc1
In Proc2 there would be tags:
- “to_Proc1” of type Proc2_to_Proc1
- “Proc1” of type Proc1_to_Proc2
Obviously, substitute useful names such as “BLR1”, “BOP”, “FH”, “Oven1”, etc… for the above names “Proc1” and “Proc2”.
But if there are three or more CLxs in the system, you have a choice: either have a pair of UDTs for each communication link, or have each CLx “publish” any data of interest to any other CLxs in a single UDT. UDT pairs for each link might make sense if there is 1↔2 comm and 2↔3 comm, but not 1↔3 comm – and especially if the information 2 is communicating to 1 and 3 deal with different parts of the process.
But if a CLx is sending mostly the same information to more than one other CLx, it makes sense to “publish” a single UDT that every other CLx reads, even if some of the information is not relevant to some of the controllers. That simplifies programming & processor speed. If the published UDTs are ≤500 bytes, there is no network penalty.
HeartBeat
Regardless of the means of communication, the best way to validate comm quality and track data age is with a constantly changing HeartBeat (HB).
Do not use a BOOL.
If the rate of communication is near an even multiple of rate at which it is toggling, it will appear not to be changing for much longer periods than successful comms are actually taking (similar to how spokes on a wheel appear to stop or move slowly on film). Although any non-BOOL type will work, a SINT is an ideal choice – it only consumes one byte, but can repeatedly count from 1 to 127 to eliminate the BOOL “aliasing” problem described above.
The receiving end should run a data age timer, and remember its previous HB value. When the HB ≠ Previous HB, update the previous and RESet the timer. You could set the timer PREset to the length of time at which you will set an alarm, but a standard timer will stop counting at that point – I prefer the PREset at its maximum (2.1 billion milliseconds is about 3½ weeks) and compare the ACCumulated time separately for alarming and/or tripping. That way you know the data age, even if it is far higher than the alarm limit.
A HB is superior to all other methods of detecting communication problems. The ER bit of a MSG block only comes on after an unsuccessful read attempt, so it does not indicate data age. There is a variable delay depending on time-out settings, and it might never come on if something prevents the MSG block from triggering. Also, a timer would still be needed because you would not normally want to alarm based on a single ER event – you’d usually want to let it retry.
A HB also tells you not only that you are communicating with the other processor, but that it is running. MSG blocks will indicate success as long as the correct tag and UDT is loaded in the other processor even if it is in program mode or faulted. If the logic needs to perform certain trip actions on loss of comm, those actions would certainly need to be performed if the other processor is not running, even if comm is OK.
I recommend starting the comm UDT with a HB, followed by at least one more SINT, so that any length of the UDT not including the HB can be COPied internally.
ComHB Add-On-Instruction
I created an AOI to handle heartbeats. It embeds a timer and handles communication warnings, as well as producing a one-shot whenever the HB changes. This AOI is not necessary, but can speed development and minimize code in a system with many CLx comm links.
Spare UDT Members
If one of your goals is to be able to modify the system while it is running (without downloading), spare UDT members are essential. UDTs cannot be changed on-line – changes must be downloaded. So if you discover a new piece of information that needs to be transferred from one CLx to another, you have the following options:
- Take both CLxs off line, change the UDT, and download them both, shutting down the process.
- Create a new UDT on line with the new data and another HB, set up another MSG block & new HB monitoring logic & alarming, etc…
- Create a copy of the original UDT with a different name and with additional members, edit every reference to those tags to use the new tags with the new UDTs. Then whenever each CLx is being downloaded later on, delete the old UDTs and tags.
- My recommendation: Use spare members you wisely inserted into the original UDT.
Any tag 500 bytes or less is transferred in a single network packet, so a 4-byte tag consumes the same network resources as a 500 byte tag. So once the essential members have been added to a UDT, add spare members of each type – especially BOOLs, SINTs, INTs, DINTs, and REALs. Sometimes a string or two is a good idea.
Add spares so that the UDT is an exact multiple of 500 bytes. These spares can then be used on the fly. Then later on when you are downloading for some other reason, rename any used spares to a better name. This can be done at separate times on each side – member names do not need to match.
Do NOT use arrays for spares – that will mean you’d have to rename used spares on both sides and download simultaneously.
Means of Communicating
There are three basic ways you can communicate between processors:
- Produce / Consume tags (good for high speed requirements).
- Write MSG blocks.
- Read MSG blocks (my recommendation for most applications).
Produce / Consume Configuration
Produce/Consume configuration must be done entirely off-line and then downloaded. This includes adding the other CLx to the I/O tree, setting the read rate, etc…
A tag intended to be read by other CLxs is called a “produced tag”. Configure them as shown:
A tag whose values are read from a remote produced tag is a “consumed tag”, configured as shown:
I recommend using produce/consume tags only for high speed applications where every update is critical to occur on schedule. The only times I’ve ever used this is when another CLx is essentially acting as a remote I/O rack rather than a separate controller. This is usually a band-aid situation when combining previously separate equipment or inheriting a poor design – I have never done this in a system over which I’ve had design control from the beginning.
MSG Configuration
I generally recommend reading from, rather than writing to, other CLxs. The exception would be if you do not have permission to modify the other CLx’s configuration because it is provided by another company – then, it might make sense to handle reading and writing from your CLx. But for applications where you are configuring both CLxs, have each CLx read data from the other. Then the ER and DN bits on the MSG blocks are more meaningful. The HB should be used to guarantee good comm, but ER and DN bits are helpful for diagnostics.
The tags to be transferred via MSG need nothing special done to them (in contrast to produce/consume tags). Simply create a MSG instruction and associated MESSAGE tag, and configure its tabs as shown:
In this case, “toProc1” is the tag of type “Proc2_to_Proc1” in the remote Proc2 CLx. Proc2 is a local controller tag (also of type “Proc2_to_Proc1”).
In the communication tab, “Ethernet” is the name of the card through which communication will occur. This can also be just the slot number (in this case, 2). The “2” after that indicates Ethernet/IP protocol – not RIO, DH+ or other possible protocols. The remaining fields are the IP address of the other CLx, a constant 1, and the slot in which the CLx processor containing the data sits.
Triggering MSG Blocks
Unlike produce/consume tag configuration and UDT types themselves, MSG blocks can be added, deleted, and configured on-line, including where they read from, what tags they read from and set locally, and the frequency at which they read. They can even change frequency depending on conditions, or enable/disable communication as needed.
A MSG block is triggered by an off-to-on transition of its EN (enable/rung input). There are many ways to trigger MSG blocks. I do not like triggering on DN or ER bits – that leads to uncontrolled and unpredictable update frequencies, and they can get hung up. Instead, trigger them from a timer. The example below reads data about twice per second (every 500 ms, with a small amount of “slip”).
If you want polling to occur precisely twice per second without any slip, more complicated logic is necessary:
If you want to trigger a MSG block on every scan, this is possible by unlatching its EN bit. This might be useful when the MSG block is in a periodic task with a fairly slow scan rate.
This technique also works for other triggered blocks such as AVErage, or any other block that uses a CONTROL tag.
What If Communication Is Lost?
Configuring control systems is 20% about making it work correctly, and 80% about making it properly handle problems. The above HB scheme is designed to produce a data age, from which comm-loss warning and trip events can be raised. There should always be a warning! I usually set those limits at a little bit over double the read frequency so that a single missed read does not raise an alarm.
Trip values are only necessary if equipment in one CLx has an interlock depending on a condition in the other CLx – so the trip time limit depends on the nature of the interlock. There can be different trip times on the same heartbeat for different interlocks.
I usually leave the values in the read tag as they are when comm fails, and have separate comm-OK interlocks for any equipment depending on those values. If I am transferring data quality (channel fault) bits, I often latch those when comm fails so the blocks looking at that quality know it is bad.
In some circumstances it may make sense to overwrite the read tag with “fail-safe” values – as they would be if everything in that CLx were down. This could be a collection of appropriate MOVe blocks. Or you can zero all the members of the tag other than the HB (you need to leave the HB unchanged or it will look like comms were just restored) as shown here:
Summary
CLx controllers are designed to communicate effectively and easily. Use MSG blocks to exchange data contained in well-designed UDTs with HBs to achieve maximum efficiency, ability to adjust the configuration on-line, and detect failures.
