Databricks brings new query and dashboard features for big data analytics

Open Source data analytics company Databricks, has launched two new offerings it said will change the way data can be stored and used in traditional data lakes and data warehouses.

The vendor said the new capabilities make it faster and easier for data teams to use its Unified Data Analytics platform for data science, machine learning and a broad range of data analytics use cases. 

The first of the two, Delta Engine, is a high-performance query engine for cloud data lakes, while the second, Redash, is an open source dashboarding and visualization service for data scientists and analysts to do data exploration.

Delta Engine is designed for use with Databricks’ Delta Lake, an open-source structured transaction layer for data lakes. The introduction of Delta Engine allows for much faster query of data which is said to drastically reduce the time it takes to query data.

Databricks' local boss Bede Hackney told CRN that there was a retooling opportunity for partners presented by these and other new features on the open-source platform.

“I think this is almost an existential moment for data warehouses where customers are going to be looking at what kind of data warehouse they have been using for the last [up to] 40 years, and realising it's never quite delivered on the promise. I've had a data lake for the last decade. And it's never quite delivered on the promise,” he said.

Hackney said the features follow a pattern which has emerged in the industry, a move away from data warehouses and data lakes to what Databricks calls a “Lakehouse''.

“We can bring together the best parts of a data lake. And the best capabilities of a data warehouse in a single environment,” he explained.

The new features make it possible to combine the best parts of a data lake and a data warehouse. To build curated data lakes that include structured and semi-structured data and run all analytics in the cloud. 

Databricks has built a base of 200 customers in Australia by working with a group of 20 channel partners across the Asia Pacific and Japan region

“The opportunity for partners is to lead,” said Hackney. “Help their customers move into that new new pattern or new paradigm. We are certainly doing a lot to educate our partners on the technology, and they're out there educating themselves as well, but there's a real responsibility to lead customers through a period of change.” 

Hackney said that while the technology was interesting in and of itself, the business case for taking two poorly performing assets and transforming them into one powerful asset was even more compelling.

“The combination of Delta Engine and Redash really opens up Delta Lake to become your Lakehouse, which will make Databricks a better platform for business analysts to collaborate on as well.”