Time. It flies. It stands still. It is valuable. But time is awful. I dislike time. In particular time zones and the complexity surrounding how to handle time on SCADA systems.
SCADA systems run on time. Daily records are recorded at contract hour. Alarms and Events are logged with time stamps. Should be easy right? Nope. Time is one of the most complicated subjects programmers have to deal with.
What people do not know is that most of the RTU controllers on the market today do not know what time zone they live in. These controllers store all of their historic records in local time without time zone information. Computers and databases since the 1980’s have stored a time stamp in UTC time or local time with a time zone offset. In an RTU with 4MB or less of memory, storage space is limited so most vendors decided not to store the offset.
Up until recently this RTU functionality shortfall did not present much of a problem. SCADA systems were local, operators were local. The server lived at the local plant, the RTUs were in the local field. If the server decided to switch to daylight savings time (DST) the RTUs followed suit and the local operator did not know any different. If the local instrument tech or programmer set a clock in the unit during maintenance it was bound to be pretty close to the SCADA host time.
Now in 2017 it is common for SCADA systems to span multiple time zones. Corporate SCADA systems are the norm because there is an inherent economy of scale when you can place 500 nodes on a server in a data center instead of 50 nodes on a remote server. RTUs from different provinces and states are now connected and reporting back to the same SCADA host. Now we need to handle data from Central time zones and Mountain Standard Time. Some geographic regions observe daylight savings time and others do not.
Further complicating matters is SCADA systems and their users used to be given more time to do daily tasks surrounding daily production. Back in the paper and pen days the production data was sent into the office once a month. When systems became electronic once a day was the standard, but reporting typically happened later, usually early afternoon. Nowadays daily data usually needs to be sent to business intelligence and production accounting systems as soon as a half hour after contract rollover time. There is no margin for error anymore to handle hour long differences of systems on standard time vs. daylight savings time.
I also notice a psychological effect too. Older, smaller, standalone systems could avoid time change fine. The only people with access to the system were local operators and it is fairly easy for a small group of people to adjust to a single localized system being an hour out of sync. You could simply say “the system has to be this way” and that was that. With easy remote access, domain controlled computers, the audience for a SCADA host is larger and explaining why DST is not observed happens not just once or twice but many times a year. It is also difficult to explain to really smart people why SCADA has difficulty adjusting time twice a year.
SCADA host software and polling drivers deal with time in different ways. Most allow you to tell the software what time zone the RTU lives in and whether or not the RTU should observe DST. For example, ClearSCADA handles time by converting all time stamps to UTC time internally. It converts time stamps back to the local time of the user for on screen display. Some host software and polling drivers are ignorant of time zones and treat all time as local, although this is becoming less common.
So what should people do to keep the time chaos to a minimum?
- Write to your local government representative and ask that they abolish daylight savings time. The sooner we can do away with changing our clocks twice a year, the better.
- Decide as an entire business to follow daylight savings time, or not with respect to your SCADA systems. Better yet, standardize on a single time zone (UTC preferably) and convert time to the proper timezone for display locally.
- Make sure your SCADA servers have an outside time source for time sync. Windows domain controller is ok; good quality NTP time server is better; local atomic clock best (but totally overkill!).
- Configure your SCADA system with field equipment time zone information.
- Configure your SCADA host to be the ‘time master’ for the field equipment.
- Ensure field techs set the clocks of RTUs and field equipment upon installation.
- Make sure field techs know that after installation the SCADA host will control the RTU time. Avoid syncing field time to field tech laptops, just in case your SCADA host is an hour different.
Other things we can do as an industry:
- Encourage vendors to implement time sync technologies that already exist. NTP is a common network protocol that is lightweight, well documented and well used worldwide.
- Build some data transmission standards that respect time zones and time information. Protocols, like DNP3, that include time information with their data messages are being used more often.
- Do a better job of educating people on the importance of time and why it can be difficult for field equipment to handle time information.
- Hope to hell we aren’t alive and working when SCADA systems collect data from multiple planets.