Inside Telstra's customer information breach

It wasn't just Melbourne's blistering December heat that made Telstra's security operations boffin sweat.

Scott McIntyre, a recent import from the Netherlands's Computer Emergency Response Team (CERT) and now "rock kicker" at the telco, had just found out thousands of Telstra customer records exposed publicly on the internet.

Worse yet, the notification came from a Google News alert.

The story broke after a user on telecommunications forum Whirlpool posted a thread pointing to an internal web page titled 'Telstra Bundle request search', where customer usernames, passwords and telco plan information could be seen by all.

The Sydney Morning Herald was the first to publish the news, shortly after notifying Telstra's media team.

Google had indexed the page, allowing anyone to look up sensitive details of thousands.

The offending website would be taken down 58 minutes later, at 5pm on December 9 but a cached version of the page on Google would be still available.

But Telstra's quick response on the matter showed a willingness to take the threat seriously once realised.

“I was very proud to get hold of the right people,” said McIntyre, who described his role at the company as "incident responder".

Later, some 60,000 potentially at-risk customers would be notified and the entire BigPond email system would be taken offline, affecting up to a million users.

At zero hour, the security veteran was busy gathering facts.

Fact and fiction

“There were people who made very important decisions and they needed very good information,” McIntyre said.

Those charged with crafting statements to customers and the press needed to know the scale of the problem, when it could be closed, and what caused the breach.

Telstra's own Computer Emergency Response Team (TCERT) is well-staffed and experienced, with few security incidents it hadn't had experience handling.

From the outset “telephone bridges were up, and people were working together”, McIntyre said.

McIntyre sought information from news reports, colleagues and, most importantly, logs.

“When you are looking into a breach you need precision, you need definitive and clear answers, and you can't do that with ambiguous information,” he said.

“More often than not, you won't have enough information.

“Document everything.”

There was a laundry list of problems which breached companies could encounter when sourcing information. If cloud computing providers retained log information – and they may often not – they must contain sufficient information.

Simple log time signatures were a spanner in Telstra's incident response machine. The US provider had signed logs in local time and in the US date format.

He said the log format was critically important, along with the scope of information retained within them.

Australia's location at the edge of the world introduced further problems. Not only did it mean overseas experts were asleep but news of the breach would only truly reach fever pitch when the world awoke.

“Timezones are an incredible problem,” McIntyre added.

"Just when you thought you could relax, the world wakes up.”

The Telstra tech appeared to have handled his first breach during his tenure at the company well.

But it was helped by Telstra's rich data retention strategies and helpful industry professionals on hand. Privacy and security were taken seriously as well, he said, beyond the cursory commitment found in the footnote of every breach disclosure note.

Mop up

Technical incident response meetings were held daily at Telstra in the six weeks following the breach as the telco looked to mitigate any lasting damage.

Telstra had engaged Google to torpedo a cached page of the offending web page. Even though an ongoing investigation of the incident found no sensitive information on the cached version, traffic from foreign web users continued to climb.

All information sharing between relevant breach responders was “done with incredible security”, according to McIntyre, including strong encryption and GPG key exchanges verified by telephone calls.

“You'll start developing more questions during an incident and those need to be handled," he said.

Now all changes in functions of the breached system – from wording, to emails and small alterations to PHP or Javascript code - undergo review.

He said many of the passwords were temporary, and much of the data deemed sensitive was in a free-form notation taken by phone operators. Much of the process of identifying data was manual.

The Australian Privacy Commissioner continues to investigate the breach and is yet to make a definitive recommendation on the issue.

But in the end, McIntyre was pragmatic; “you can't put the whipped cream back in the can".