Excellent PLC
By Thomas Reed – Lead Automation Integrator
Firmware upgrades are supposed to be routine.
On paper, updating the Honeywell 10012/1/2 CPU module looked straightforward: same hardware, minor revision bump, no functional changes promised.
In reality, it became a lesson in patience — or the lack of it.
Why the Upgrade Was Attempted
The system was stable, but:
-
A newer firmware revision was recommended
-
Minor communication optimizations were advertised
-
Maintenance window was available
No alarms.
No failures.
Just the temptation to “improve things.”
The Upgrade Environment
-
Online system, partially loaded
-
Shared power source with other cabinets
-
No dedicated UPS on the controller rack
-
Flash rewrite performed in-place
Everything worked — until it didn’t.
The Moment Things Went Wrong
Halfway through the firmware write:
-
Power flickered briefly
-
CPU remained powered, but voltage dipped
-
Upgrade tool lost connection
The process did not fail cleanly.
It stopped.
What Was Left Behind in Flash
The 10012/1/2 flash memory ended up with:
-
Old bootloader
-
Partially written firmware image
-
Inconsistent configuration pointers
From the outside, the CPU powered up normally.
Inside, there was nothing coherent to run.
Why Recovery Was Not Possible On-Site
-
Bootloader validation failed
-
Firmware image checksum mismatched
-
CPU never reached a state where re-download was accepted
In effect, the controller became unreachable.
No amount of retries helped.
What Finally Restored the System
-
CPU module replacement
-
Firmware written under controlled lab conditions
-
Application restored from offline archive
Only then did the system start again.
What Should Have Been Done Differently
-
Perform firmware upgrades only with protected power
-
Avoid online upgrades on aging hardware
-
Treat flash write operations as high-risk events
Long-Term Changes After This Incident
-
Firmware upgrades moved to scheduled outages
-
Mandatory UPS required for CPU racks
-
Flash lifecycle tracked alongside firmware revisions
What This Failure Taught Us
-
Flash corruption doesn’t always announce itself
-
Firmware upgrades are write-intensive operations
-
Stable systems don’t need unnecessary changes
-
Recovery paths disappear once flash integrity is lost
Final Reflection
The Honeywell 10012/1/2 CPU flash memory didn’t fail randomly.
It failed during a moment we chose convenience over caution.
In control systems, “almost finished” is often the most dangerous state.
— Thomas Reed