eClaims Fixed and Being Updated
This week marks the half decade anniversary of the submission of the first eClain. As most of you know, after five years of continuous use, one of the eClaims servers crashed last month. We want to let you know what happened and how the administrators are addressing the issues precipitated by that event.
The crash was caused by an automatic update of the Windows Operating System performed remotely by Microsoft on one of the servers that hosts eClaims. It appears that a corrupted Windows update file crippled the operating system and made all files on that system unreadable.
How do we ensure the security and reliability of the data?
Safety and reliability are complex objectives in an ever-shifting technology environment. We all have to pay attention to these issues for our own computers. For a program of state-wide usage such as eClaims, these objectives are even more critical. They involve security from viruses, worms and hackers as well as hardware integrity that handles the inevitable failures of such equipment as hard drives and power supplies.
The eClaims program has had, from the beginning, a comprehensive system to protect it from outside threats - including a system of firewalls, bank-level password protection, and field-specific constraints on data entry. There are also redundancy systems. For example, the eClaims computer servers have of multiple hard drives and multiple power supplies so that the inevitable hardware failures - of which there have been several in the past five years (not noticed by users) - automatically trigger the move to one of the redundant units, requiring no down time whatsoever.
The entire eClaims system is housed in two unrelated limited-access locations in different parts of California, with real-time communications between those locations. Thus, even after a catastrophic event such as an earthquake, the program and data would become available from the secondary location with the flip of a few switches (unless both locations are under water due to the catastrophe). In addition, backups are made and copies sent to both locations so that the data could be restored from them as well, if necessary.
We are now in the process of further improving the system design so that it can continue to run even in the event of corrupted operating system updates from outside sources. We have purchased, set up and tested new computers and have updated operating systems and software. With outside expertise, we have also analyzed the entire system and have identified obscure potential points of hardware and software failure and are in the process of plugging those holes, including unlikely switch, firewall and web access failures.
The changes will be implemented within the next two months. They will require very little down time, and any such time will be well spent to test all backup systems to ensure they work as planned. In the future, we will notify the panel on the Home Page of eClaims prior to performing tests of the backup systems, which will now be done regularly, benefitting everyone.
While our investment in time, energy, hardware, software, and experts has been substantial, the effort is necessary and worthwhile. Technology is not flawless. With these new efforts, however, we are focused on keeping the integrity and security of eClaims at a level that makes you comfortable and confident in accessing the benefits of the program.
Jay Kohorn, who is in charge of eClaims, is available most any time to help panel members and is open to comments and suggestions.
April 19, 2007
Back to News