Crunching half a petabyte on AWS: Victoria's Dept of Environment taps Bulletproof

Half a petabyte is not only a monstrous volume of data but, in the case of a swathe of aerial photography and satellite imagery purchased by Victoria's Department of Environment, Land, Water and Planning (DELWP), it was expensive too – costing a cool $45 million.

But DELWP faced a problem: the data was sitting in disparate systems and data silos across the organisation in un-networked hard disk drives and out-of-maintenance servers. There was no central catalogue or register, no security and access logs and no means to enable self-service access to the data.

The agency would go on to embark on a cloud migration to Amazon Web Services that reduced time to retrieve archived data from three weeks to less than 12 hours while freeing up full-time staff from mind-numbing data copying to work on more valuable activities.

To make better use of its half-a-million gigabytes of high-value imagery, the agency turned to Sydney-headquartered cloud solution provider Bulletproof.

The project, which was completed in the fourth quarter of 2016, was "complex and time-sensitive", Bulletproof said. "DELWP needed an enterprise system to manage, publish and enable discovery and analysis of a growing catalogue of raster and elevation datasets and a quick solution given their current raster data services weren’t meeting needs."

Other suppliers had already pitched DELWP on an application solution but the agency wanted "a long-term, scalable data solution". Bulletproof swung DELWP's favour by suggesting a data lake built on Amazon Web Services.

The data lake would ultimately make the data easier to discover, providing operational efficiencies for DELWP’s customers, staff and "countless other Victorian Government agencies", according to Bulletproof.

Multiple government departments leverage DELWP data, including Climate Change, Coasts and Marine, Energy, Environment, Forest Fire Management Victoria Heritage Victoria, Land Victoria and Local Government Parks and Forests Water and Catchments Wildlife.

Alena Moison from DELWP's spatial data operations team said: "The crack team of Bulletproof data scientists, architects and engineers easily appreciated our complex and sometimes ambiguous requirements to come up with an elegant, flexible, and extensible solution that will support DELWP’s innovation in the long-term.

"Partnering with Bulletproof to apply pioneering data science strategies, we have built the foundations of an information infrastructure that will improve access and management of significant spatial data assets, and enable DELWP to gain critical insights into the work we do to create a liveable, inclusive and sustainable Victoria," said Moison.

As well as significantly reducing the time to retrieve archived data, the agency achieved the immediate release of two full-time employees from repetitive data copying tasks to "higher-value activities".

Other benefits included increased data security compliance; more flexibility to scale services and costs; improve staff satisfaction with DELWP’s raster data storage offering; reduced risk from reliance on key personnel and no DR for data and critical apps; and better collaboration with other agencies thanks to the removal of internal silos.

Consolidating on the cloud

Bulletproof consolidated data from numerous unsecured data sources, including personal USB hard drives and brought it together on an automated, scalable and highly available architecture.

This solution combined a bevy of AWS services, including S3 storage, Simple Queue Service, Simple Workflow, DynamoDB NoSQL database service and API Gateway.

By using different S3 storage classes, Bulletproof can optimise DELWP's storage costs "through a combination of S3 lifecycle events and custom business logic".

APIs are consumed via the API gateway, which allows Bulletproof and DELWP to control who can access what APIs, apply authentication and have discoverability.

Collections of assets – known collectively as a bundle – are securely stored in S3 buckets that can be accessed through dynamically generated signed URLs.

The schemaless design of DynamoDB was "the perfect choice" for the metadata catalogue because it could facilitate the addition of new bundle metadata requirements over time.

According to Bulletproof, "The entire architecture is purposefully event-driven. Using a combination of event sources including S3, Lambda, API Gateway and SQS events, the system is able to scale. Components are decoupled ensuring flexibility for future requirements."

Bulletproof is a finalist in the Digital Transformation category for the 2017 CRN Impact Awards. The full list of finalists will be announced at CRN Pipeline Melbourne on 22-23 March. The winners will be announced at the Impact Awards dinner during CRN Pipeline Sydney on 6-7 April.

Doing any cool projects this year? Fill out an online expression of interest and we will inform you when we open entries for the 2018 CRN Impact Awards. The form is quick and easy.