Sophos blames human error for anti-virus bug

Sophos has blamed its testing procedures and a chain of mistakes by staff for an anti-virus update that last month caused users' machines to identify its software updates as malware.

The update, issued late last month, identified legitimate software updates as malware, leading to an outbreak of panic among customers fearful of virus attacks.

Sophos moved immediately to identify affected users and fix the issue but said it had received 10 times the normal levels of calls to its major call centres in nine countries, and 25 times the average number of views to the company website within the first 48 hours of the problem being identified.

In a post-incident report released this week, the company said it was a string of human errors that led to the bungled rule changes being released to users.

"A Sophos analyst incorrectly coded an update to our detection rules within an IDE file update that caused false positive triggers in the Sophos endpoint product for Windows," the company said.

Although a 12-step testing procedure for each update should have caught the issue before public release, "a combination of human error in code review, human error resulting in incorrect interpretation of test results, and a mismatch in test environments meant that the faulty IDE was allowed to pass through to release".

Sophos said it conducted significant testing of its software updates on multiple platforms, against tens of millions of files and terabytes of data, including a check to ensure the update did not declare false positives.

Although the "vast majority" of Sophos customers had the automatic Live Protection feature enabled or default security settings — minimising the impact of the update — many remained affected by issues with other software on the same computer while attempts to update the Sophos anti-virus, in some cases, failed.

One reader on the Internet Storm Centre at the time said the glitch affected other updater services including Adobe Flash, Oracle Java, Fujitsu AutoUpdater and Dell AutoUpdate Utilities.

"All of the auto-updaters mentioned above were deleted off hundreds of PCs. Now none of these applications will auto-update moving forward," they wrote.

"Even though Sophos may have fixed the problem and fixed their own software, there is a monumental amount of work we have to do to clean up after this mess.

"I've worked in IT for 16 years and have never had a virus/Trojan/spyware/malware cause problems and disrupt our systems the way this did.

"Who can I trust anymore when even my security anti-virus vendor can wreak more havoc on our systems than a virus infection outbreak can?"

Sophos has undertaken several procedures to boost testing and incident response procedures, including fixing the side effect in the software's threat engine that allowed the update to be marked as a false positive in the first place.

It also established an overflow for its call centres, and aimed to make its articles on the issue clearer to understand.

"Sophos will learn from this incident and will emerge an even stronger company," the company said.