Survey results recently published by SiliconANGLE showed that enterprise customers remained conflicted on how to rationalize a need to balance data trust with a strong motivation to move fast and innovate. As key data management firms, such as Databricks Inc. and Snowflake Inc.. compete for customers in this area, the end user could end up reaping the benefits through open table formats.
In the modern data landscape, companies have shown an increased interest in adopting open table formats to allow any compute to operate on any data. This is forcing organizations to confront new issues around governance.
“When you talk about open table formats, then you are essentially having to start abstracting governance from the purview of any one vendor,” said George Gilbert (pictured, left) analyst for theCUBE Research. “You have to start thinking about governing all the data in your data and application estate.”
TheCUBE’s Rob Strechay and Sanjeev Mohan discuss open table formats at Supercloud 7.
Gilbert spoke during an analyst panel segment as part of the Supercloud 7: Get Ready for the Next Data Platform event. He was joined on the panel by Sanjeev Mohan (second from right), principal at SanjMo and analyst for theCUBE Collective, along with theCUBE Research’s Dave Vellante (right) and Rob Strechay (second from left), and they discussed the Enterprise Technology Research survey data and key factors influencing enterprise interest in open table formats.
Customers express preferences for open table formats
The survey data from ETR showed results from 105 joint Databricks and Snowflake customers who were asked about preferences for open table formats. While a majority ranked security and governance and lock-in avoidance first and second, ETR uncovered a percentage of customers that could provide a “swing vote” in vendor selection, according to Vellante.
“There’s 14% that said data security is less important than creating a stack that allows for rapid innovation,” Vellante noted. “Why is that the swing vote? It’s because ETR correlated that and found it. Those were the ones that were most likely to [leave] Snowflake and move to Databricks as an example, because they’re saying ‘damn the torpedoes’ in governance.”
Another key element in the open table debate involves multiple formats that have emerged over the years. While there are many choices, Mohan points out that compatibility is a growing issue.
“Right now, if you look at which table format has the most momentum, it is Iceberg,” Mohan said. “But is it the right thing? I think that is a question which people are not asking. Maybe it should be Hudi or Delta. The problem is that these formats are not all compatible to each other, so that is why you have ‘read only’ views. If you write in Delta, you are not going to write in Iceberg — that will kill your entire analytics.”
New competitors for data platform vendors
Another metric generated from ETR’s survey data provided insight into the attractiveness of the Databricks and Snowflake platforms for building generative artificial intelligence applications. While 64% leaned toward Databricks, and 49% leaned toward Snowflake, a healthy 34% expressed an affinity for hyperscaler offerings in this area.
“The data platform vendors think they’re adding new workloads and expanding their total addressable market and reaching new personas, but what’s actually happening is they’re bumping into new competitors,” Gilbert explained. “That number of 34% for the hyperscalers is significant. They are going after that open data for building applications and the data platform vendors aren’t always the best equipped.”
However, Strechay noted that dependence of the Databricks and Snowflake platforms on the cloud could make this number less significant than it might appear. Customers will still be heavily focused on whichever platform allows them to bring data together and achieve meaningful business results.
“I don’t know that it’s one or the other because of the fact that you have Snowflake and Databricks,” Strechay said. “Where do they live? They’re in these hyperscale clouds, they’re not on-prem. It’s going to be a mix, and I think this is why people are looking at it and going, ‘OK, how do I bring this data all together?’”
ETR asked enterprise customers whether they were using or intending to use any open data formats such as Hudi, Iceberg or Delta. The results showed that 70% were evaluating formats, while only 30% had no interest. Results such as these highlight the competitive opportunity facing Databricks and Snowflake as they battle for market share.
“When [Databricks CEO] Ali Ghodsi says, ‘Don’t give your data to any vendor, including Databricks,’ his answer to that is … use open table formats, may the best engine win,” Vellante said.
Stay tuned for the complete video discussion, part of SiliconANGLE’s and theCUBE Research’s coverage of the Supercloud 7: Get Ready for the Next Data Platform event.