SCADAPack Memory Issues

We work with SCADAPack PLCs a lot in our line of work. They are pretty flexible devices. However, what many people do not know is that they have a relatively old OS and do not have great memory management. Here are my tips for properly setting up a SCADAPack. Some explanation as to why towards the end.

TL;DR – Setup/Load a SCADAPack

Remember Realflo first! Always get Realflo installed and configured first.

You want to ensure any memory blocks that may change in size in the future (DLOG or SCADA Suite) end up at the end of consumed memory so they can ‘flex’ their size in the largest free block of memory possible. By configuring the SCADAPack with Realflo first way the memory is setup to accommodate this.

  1. Download Realflo C Program
  2. Configure Realflo, start Realflo calculations
  3. Download Telepace Logic program, do not start
  4. Download any other C programs, start if required
  5. Configure other C programs
  6. Start/run Telepace Logic

Explanation

I have been working with SCADAPacks since around 2009. These controllers are very common in Canadian oil and gas and worldwide in water/waste water. They are quite flexible and easy to use, particularly the Telepace based variants.

Memory in the SCADAPack controller is not very well understood. SCADAPacks have three types; volatile memory (RAM), non-volatile memory (NVRAM), and flash disk storage (ROM). Different applications consume this memory differently. But there are some fixed pieces that are always in memory. NVRAM is most commonly used for storing information (logs, calculated values, etc.) in the background as this memory survives power cycles.

The SCADAPack firmware reserves space in NVRAM for the I/O database, roughly 42kB of memory. This I/O database is the Modbus register map. These registers are used to store function block configuration, calculation values and provide a bit of memory to work with practically in your Telepace and C programs. See the Telepace Manual for more information on the I/O database, its uses and limitations.

Pro Tip: SCADAPacks always have the entire Modbus register space reserved. You can poll any Modbus register and the controller with return the contents. This makes SCADAPacks really useful when you need to interface several Modbus devices together, or pull data out for other uses.

In addition to the I/O database there is space reserved for Telepace logic. I cannot remember the exact size, but should be 32 or 64kB of memory. Every network, column and function block element consumes memory words in this reserved space. You can see the usage of this memory space in the bottom right of Telepace. If the display shows 50% used that means Telepace logic is consuming 50% of the memory in the logic reserved space; not 50% of the memory in the SCADAPack in general.

Pro Tip: If you want to optimize your SCADAPack Telepace program look for ways to reduce the number of networks and number of columns consumed on any given network.

One of the things I have found is that SCADAPacks are pretty poor at managing their memory. They are basic controllers and they do not have advanced memory management techniques found on newer modern computing platforms. SCADAPacks will not defragment and re-arrange free memory space to obtain larger free blocks. The programmer needs to download in the correct order to the controller properly to ensure the largest free memory block is at the end of memory.

As I understand it, when a SCADAPack reserves space for a C program, like Realflo, or for DLOGs the operating system reserves that entire block in the first available free block of memory that is large enough. The memory for a particular purpose cannot be split, and will not necessarily be put in the free memory in a way that is useful later. Initially this does not seem so bad. However, in RTUs using Realflo, which is a memory hog, you may end up with not enough free memory in one big block. Lets dig into why this is important.

Scenario 1 – Telepace First

What I have noticed is that if the Telepace and DLOG config goes in first this configuration will consume a good portion of memory, say 10-15%, depending on the number of log records configured. Realflo goes in next but it can consume as much as 75% of the memory on a SCADAPack due to history and log requirements for compliance. Now we may be 90% consumed memory.

Pro Tip: There are now app suites like Simark SCADA Suite that allow for configuration of DLOG blocks outside Telepace logic. Make sure you plan your logging strategy and choose to log data in Telepace or the external app only. Configuring logging in both locations is a recipe for disaster!

Not a big deal, the site runs for a couple years. Now the client adds some more equipment and wants you to add logging on some new data values. If you go from 15% memory required to 20% memory the firmware will attempt to put the DLOGs after Realflo memory.

If there is no block of free space large enough for the logs then the DLOG block status will be invalid. You may also see some weirdness if you are using C programs as these also reserve pages of memory.

Scenario 2 – Realflo First

So my primary recommendation is to download and configure Realflo first. Here is what it does to the SCADAPack memory.

Later on when we go to modify or make the DLOGs larger the DLOG will place itself at the end and we have less of a possibility of memory contention.

I Maybe did it Wrong. What is the Fix?

Cold Boot the RTU and download again. Because the memory in the SCADAPack is so simple you can get in this spot where there is enough free memory, but no single free block big enough for your app or configuration. This is why a lot of weirdness with SCADAPacks are fixed by a cold boot reset to factory settings.

DNP3 – Similar Considerations

DNP3 is a powerful protocol supported natively by SCADAPack RTUs. Not many people use it in oil and gas but if you use DNP3, try to avoid using DLOG since it is redundant. However, the memory considerations are fairly similar. You will want to configure Realflo first and Telepace/DNP3 afterwards.

Holy Terminology Batman!

There are a lot of technical bits and pieces used in this article. Here is a quick glossary.

DLOG – A native logging component unique to SCADAPack. Allows the user to save logs process values for later retrieval. Can be used for high resolution trends (trigger DLOG on time interval), or detailed troubleshooting (trigger a DLOG on an event).
PLC – A programming logic controller. These devices usually support a large number of I/O for connecting sensing devices and can compute logic statements quickly. They have some communication ability but usually not as much as RTUs. Also any calculations are typically handled with logic programming, not with parallel applications on the controller.
RAM – Random Access Memory. Computing hardware uses RAM for storing data. Non-volatile versions are common in industry environments because most industrial controllers need to maintain their state between power cycles. RAM is used for values that frequently changed.
ROM – Read Only Memory. Not as common on computers anymore, but this type of memory is used for storing smaller amounts of read-only data that do not change very often.
Realflo – Realflo is Schneider Electric’s gas flow computer software. It runs on SCADAPack hardware today and is configured using specific software available from Schneider. It can handle all the major gas and liquid flow calculations common to oil and gas measurement.
RTU – Remote Terminal Unit. These are now very similar to PLCs but usually perform less logic and more math. They are also typically given better communication functions (more ports, more protocols).
Telepace – This is the logic programming environment for a SCADAPack. It is comprised of a logic controller on the SCADAPack and a Windows based programming environment.