CitiPower and Powercor turn to Melbourne AWS partner Vibrato for smart grid big data project

Australia's energy sector is being asked big questions. What if the answer is, in fact, big data?

That was the thinking for one of the biggest energy companies in the country, the group behind Victoria's largest electricity distribution network. CitiPower and Powercor don't just distribute an enormous amount of electricity but also produce a torrential volume of data.

With help from Amazon Web Services partner Vibrato, they would undertake a truly transformative project combining smart grid technology with cloud data analytics.

CitiPower and Powercor manage more than 70,000km of line with a total serviceable area of over 145,000sq km, servicing over 1.1 million customers.

The arrival of Advanced Metering Infrastructure – smart meters to you and me – represents an initial foray into smart grid applications at scale for a wide variety of power distribution companies. Since deploying Advanced Metering Infrastructure, CitiPower and PowerCor have transitioned from one meter reading every three months to one every 15 minutes. It's an exponential increase in the amount of incoming data.

But while many utilities have invested large amounts of capital in the move to smart grid technology, they often remain unclear on the best path to extract the potential value from these systems.

They can get the data, sure, but if it's under-utilised, what's the benefit?

CitiPower and Powercor turned to Vibrato, an AWS partner based in Melbourne to make sense of it all.

According to Vibrato's entry in the CRN Impact Awards, "advanced analytics is rapidly becoming the next evolutionary step for driving value and insights into the currently deployed networks".

"The ability to ingest, transform and analyse this data is one of the key benefits of deploying a big data solution. Scaling these systems to cope with the amount of data while keeping the cost of processing down represents a sizable challenge for the teams in charge of wrangling this data."

Luke Skinner, head of network technologies at CitiPower and Powercor, said the data and analytics platform built with Vibrato enabled the company to gain far greater insights into the business.

Decision making has been improved in tandem with cost reduction. Customers get a better experience as Powercor can proactively identification and resolve of potential network reliability issues, added Skinner.

Open source tools

Vibrato designed and built a platform to provide previously unknown insights in the way networks are managed, how consumer relationships are handled and to ensure power quality. The project harnessed not only the massive compute power of Amazon's cloud but Docker and open source technology from music streaming service Spotify.

The AWS partner developed "a multi-user notebook environment" based on Apache Zeppelin. This enabled power engineers to write code in any language and execute against a huge parallel compute allowing a vast amount of data to be queried in near real time.

"Enabling an investigatory analysis meant power engineers could test hypotheses easily and rapidly, thus unlocking previously unknown insights. This approach replaced traditional BI capabilities cutting development times and allowing new models to be designed and implemented with relative ease," according to Vibrato.

That AWS computing grunt reduced processing time for enormous data sets from hours or days down to minutes.

"Batch processing pipelines were built using open-source technology from Spotify, and at the highest level each batch job was packaged as a Docker container, which allowed anyone to build new tasks very easily. Utilising the Elastic MapReduce from AWS allowed processing at a huge scale while taking advantage of spot pricing, which ensured that infrastructure costs were kept to a bare minimum. This stack resulted in a fully automated pipeline with data coming in and value coming out constantly."

CitiPower and Powercor are now able to process years' worth of AMI data in mere hours.

If insights were the main benefit, then the cost savings were a welcome bonus.

Vibrato called it a "cost-effective new way of managing data at scale".

"Ultimately, this reduces operational costs, leading to lower consumer costs, and it increases the quality of the service provided. The customer was able to save millions of dollars on traditional data centre costs and leverage compute on demand and at scale. In turn, Powercor was able to leverage various analytical models to determine better and safer solutions for their consumers."

Stefan Jansen, Amazon Web Services head of channels and alliances, praised the partner. "Vibrato has worked with a number of our customers to create a robust and innovative platform that in turn ensures the customers' users get new and innovative product.

"For example, their electricity industry distribution customer Powercor was able to apply data analytics models on AWS to provide Powercor with extremely valuable benefits and insights.

"In particular, this assisted Powercor in detecting faults faster and more precisely for their end users. The Powercor journey will continue to develop as Vibrato looks at new ways to help the company leverage the platform as a whole, not as a one point only solution," added Jansen.

Vibrato is a finalist in the Digital Transformation category at the 2017 CRN Impact Awards. The full list of finalists will be announced at our CRN Pipeline conference in Melbourne on 22-23 March. The winners will be announced at the Impact Awards dinner during the CRN Pipeline conference in Sydney on 6-7 April.

Doing any cool projects this year? Fill out an online expression of interest and we will inform you when we open entries for the 2018 CRN Impact Awards. The form is quick and easy.