Iceberg project leader is bullish on prospects for unified tables and catalogs – News

Some people hailed Amazon Web Service Inc.’s December announcement that S3 tables will fully support Apache Iceberg as a crowning triumph for the open-source table format. But Russell Spitzer, a principal software engineer at Snowflake Inc. and an Iceberg Project Management Committee member, noted that the announcement was less than a full-throated endorsement.

“I personally thought it would be more tightly coupled with Glue,” which is AWS’s data catalog, “and they chose to integrate via a custom catalog [application program interface] as opposed to like the REST API,” he said. “It exists a bit separately right now.”

Though AWS’ move is a net positive for Iceberg in its march toward becoming a de facto table standard, competitive factors and divided loyalties continue to thwart broad consensus. The issue has become particularly urgent as generative artificial intelligence has come on the scene with its demands for a consistent way to store vast amounts of unstructured and semistructured data.

Apache Iceberg, which is a management layer that sits atop data files in cloud storage, currently has all the momentum, but it and the data catalogs that support it have been the subject of a contentious debate between Snowflake and rival Databricks Inc., which are considered prime platforms for AI development. Both have announced support for Iceberg, but both are also invested in alternatives.

Merger hopes

As someone closely involved in Iceberg’s evolution, Spitzer sees the competing projects eventually merging to create a single foundation for analytics and artificial intelligence model training.

“There’s no reason why we shouldn’t bring valuable Delta Lake features into Iceberg,” he stated, referring to the table format developed by Databricks and released to open source. For example, Iceberg’s streaming capabilities could be improved by adopting proven technology from Delta Lake.

While executives battle publicly over formats and catalogs, engineering teams from rival companies cooperate behind the scenes. “I talk to people from other companies every single day of my life,” he said. “Most engineers just want to make their lives easier.”

Converging multiple table formats and catalogs is a complex and time-consuming process. Though Databricks has expressed confidence that its recent acquisition of Tabular Technologies Inc., whose founders developed Iceberg, will accelerate integration with Delta Lake, Spitzer said there are significant challenges to reconciling proprietary Databricks-specific features with Iceberg’s open-source governance model.

“The hardest thing for them is they have a whole ecosystem around Delta that takes advantage of features that maybe are not well-documented or a good idea for a general-purpose format,” he said.

‘The way forward’

Snowflake is also conflicted. The company says it has fully embraced Iceberg as its open format of choice, but it will continue to offer legacy FDN format and Horizon catalog. “There’s a segment of users who want complete control and security over their data, and they’ll stick with Snowflake’s proprietary format,” Spitzer said. “But for those who want open interoperability, Iceberg is the way forward.”

Data catalogs, which are centralized metadata repositories that help organizations discover, manage and govern their data assets, are another sticking point. The industry has yet to agree on a single catalog standard and may never do so. However, Spitzer believes Iceberg can help the process along.

The 2022 addition of a representational state transfer, or REST, API to Iceberg makes commit logic a server-side service, ensuring that different engines can interact with Iceberg tables in a standardized manner.

Previously, Iceberg relied on a client-side logic approach, where every database engine had to implement its own commit logic. That led to fragmented implementations across Spark, Trino, Python and other platforms that access Iceberg data.

The REST API abstracts away much of that complexity, allowing data catalogs to interact with the underlying data store in a consistent manner.

“We want to get to a point where the clients start detaching from the table format entirely because all they need to know is how to talk to the REST spec,” he said.

Snowflake hopes the Polaris catalog it released to open source last spring will evolve to enable universal catalog federation, enabling multiple distributed catalogs to work together.

“Our goal is to create a single pane of glass where users can see and manage all their datasets, regardless of the underlying catalog,” he said. But that will require cross-vendor collaboration and agreement on identity and access control standards, which will likely be a sticking point.

“Some vendors already have well-established models for access control, and any industry-wide standard would have to accommodate these differences,” he said.

Iceberg ahead

We may be years away from a unified table and catalog standard, but Iceberg continues to evolve. Several significant developments are on the horizon in the forthcoming Iceberg Version 3, Spitzer said. It introduces variant types, allowing more efficient storage of semistructured data like JSON. “You’ll get the same performance as you would with well-structured data, even though your data is not structured,” he said.

Other improvements include row lineage tracking, which allows for detailed record-level history of data changes. A new default column value feature improves schema evolution and simplifies data management by allowing users to specify a default value for a column when inserting new rows without explicitly providing a value. Multi-argument transforms enhance partitioning and sorting strategies to improve query performance and storage efficiency.

Those will be discussion points at the second annual Iceberg Summit in San Francisco April 8 and 9. “We have confirmed headliners from a lot of different companies, including Databricks,” Spitzer said. “If that isn’t saying that things have changed dramatically, I don’t know what is.”

Image: News/DALL-E