Databricks Inc. today announced the general availability of Zerobus Ingest, a new serverless service within its Lakeflow Connect suite that simplifies and accelerates real-time data ingestion into the Databricks Lakehouse.
Zerobus Ingest lets applications stream event-level data directly into governed Delta tables without relying on intermediate message brokers such as Apache Kafka. Databricks said the streamlined process cuts infrastructure overhead, reduces operational costs, and lowers latency to as little as fve seconds.
Zerobus Ingest supports thousands of concurrent clients and can achieve over 10 gigabits per second of aggregate throughput into a single table, with up to 100 megabytes of data per second per connection. The managed service is serverless, meaning it can automatically scale to match workload without manual configuration, according to Databricks.
Instead of the multi-sink architecture typically used by general-purpose message buses that route events to multiple consumers, Zerobus uses a single-sink approach optimized for data ingestion. The company said the approach minimizes engineering overhead, removes the need to tune brokers or monitor consumer groups, and eliminates much of the maintenance typically required for a Kafka-based pipeline.
Developers can integrate with Zerobus Ingest using Google Remote Procedure Calls and representational state transfer application programming interfaces, as well as language-specific software development kits for Python, Java, Rust, Go and TypeScript. That allows applications to write records directly into the lakehouse with minimal code and governance provided by the company’s Unity Catalog.
Databricks said Zerobus Ingest is best for use cases such as telemetry and internet of things data, clickstream analytics, cybersecurity event streams and other scenarios where near-real-time access to large volumes of data improves operational insights. Eliminating external streaming layers allows organizations to reduce costs associated with multiple storage and computing resources while collapsing multi-component ingestion stacks into a single managed service.
The service is available now on major cloud platforms as part of the Lakeflow Connect portfolio.
Image: News/Microsoft Designer
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About News Media
Founded by tech visionaries John Furrier and Dave Vellante, News Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
