Data engineering in the cloud and AI age – News

Over the past 13 years, cloud technology has transformed data engineering and management. Amazon Redshift, Snowflake Inc. and BigQuery reshaped the industry, but a crucial gap remained — how to effectively transform data in the cloud. dbt Labs Inc. has addressed this challenge by creating a cross-cloud structured query language-based transformation layer, integrating best practices from software engineering, such as continuous integration and continuous delivery and Git, into data workflows.

“There was an obvious gaping hole in the ecosystem of data transformation,” said Tristan Handy (pictured), founder and chief executive officer of dbt Labs. “There was no cloud-based way to do it. I think if you want to talk about innovation, innovation is not the nuts and bolts thing that most companies do every single day, and that includes us. Most of the things we need to do on a day-in-day-out basis take the core innovation and extend it to what customers need and how to make it useful in their context.”

Handy spoke with theCUBE’s John Furrier for the Tech Innovation CUBEd Awards 2025 interview series, during an exclusive broadcast on theCUBE, News Media’s livestreaming studio. They discussed the future evolution of data platforms, touching on refinements in data engineering and the increased traction of open standards.

The parallel evolution of data platforms and data engineering

Data engineering has evolved to become as critical as platform engineering and site reliability engineering. Generative artificial intelligence is accelerating this transformation because AI systems rely on structured and well-integrated data. For this reason, organizations that fail to structure and govern their data effectively will struggle to harness AI’s potential, according to Handy.

“What we’ve seen is a lot more data get moved, and a lot more data products get built on top of that, and that feeds to the current moment where we are right now with AI,” he said. “We still have further to go there, but I think we’re in a better place than we have been to feed structured data into AI systems.”

Beyond technology, company culture plays a role in fostering innovation. dbt Labs embraces the philosophy of “We Contribute to the Knowledge Loop,” emphasizing open-source contributions and community-driven development, according to Handy. This altruistic approach, combined with contributions in the data engineering space, earned dbt Labs a “HyperCUBEd Innovation Award – Private Company.”

“The next era in data is not going to be defined by purely the move to cloud,” Handy said. “I think that it is going to come from really two trends. One is standards, and the other one is AI. You put all your data in a certain format and a certain engine, and then they charge you for access to that. And for the first time, that changes. [Chief data officers] want to understand the strategy with respect to Iceberg and OpenTable standards.”

Unlike proprietary solutions that die with the companies that create them, open-source software and knowledge-sharing ensure that valuable insights persist and benefit future generations. This cultural commitment to openness and collaboration is what sets dbt Labs apart, according to Handy.

Here’s the complete video interview, part of News’s and theCUBE Research’s coverage for the Tech Innovation CUBEd Awards 2025 interview series: 

Photo: News