Cisco blacklists thousands of legit Aussie mail servers

Thousands of Australian businesses have been unable to send emails this week due to a configuration error in a blacklisting service operated by Cisco Systems.

The issue, which has baffled IT administrators all week, incorrectly gave a large set of IP addresses – many of them Australian customers of Cisco’s IronPort service – a poor reputation score.

Many organisations use Cisco’s IronPort web reputation service (known as SenderBase) to determine whether to accept emails from a given IP address, as a means to cut down on spam.

Business users whose emails have not reached their destination have turned to network managers, ISPs and web hosts in frustration to seek answers.

A technical advisory issued by Canberra-based hosting company AussieHQ suggests legitimate mail has been blocked since as far back as Saturday. The web host released a service advisory Saturday noting “clients may be experiencing bouncebacks when sending through our IronPort mail system.”

By Tuesday it became clear that the problem was affecting mail services across the nation, including customers of ISP Internode.

Network managers running the Catholic Education Network in South Australia noted on Twitter that “much of Internode’s address space seems to be getting a bad mail reputation,” preventing Internode customers from sending mail to recipients on the CESA network. “Senderbase.org seems to have applied a 'guilty by association' policy for email traffic from network blocks with poor reputation.”

It took conversations between this journalist and two resellers in Victoria and South Australia to get to the root of the problem.

It appears that Cisco made some aggressive configuration changes to the IronPort system at some stage late last week, causing numerous false positives.

This publication has learned Cisco Systems has acknowledged the fault internally and told customers to wait 24 hours until a fix updates on IronPort servers. It remains unclear how much longer afterwards customers will have to wait for their reputation score on Senderbase will return to normal.

Customers appear to be especially vulnerable if they have multiple reverse pointer records for a single IP address. This is a common scenario, for example, when there are multiple hosts on a shared web server.

It is also commonplace for an ISP, rather than the customer, to control part of the naming process and the server administrator the other – leading to multiple names for a single IP address.

“We have never bothered getting them updated previously,” one server administrator told iTnews. “It looks like now we will have to.”

Precisely how many email users are impacted is difficult to determine. The spread of affected users in discussion with iTnews to date suggests that it is nationwide, but one reseller said the issue “will have a significant impact on the internet worldwide.”

Representatives from Cisco Systems, Internode and AussieHQ have been asked to respond to the story, but were unable to before going to press.