Engineering a Time Series Database Using Open Source: Rebuilding InfluxDB 3 in Apache Arrow and Rust

Four years ago, InfluxData began working on a new core for InfluxDB to address the growing demands of modern time series workloads. This decision wasn’t made lightly; it was driven by the need to support specific customer demands, including infinite cardinality, cheaper object storage for historical data, support for the SQL language, and broader ecosystem support for our file format and standards. 

The result of these efforts is InfluxDB 3, a real-time, columnar database optimized for time series built in Rust with the FDAP stack (Apache Flight, DataFusion, Arrow, and Parquet). The new core represents a significant improvement for our core database technology, removing cardinality limits so that users can bring in massive amounts of time series data with unlimited scale, SQL query capabilities, tiered data storage, and fast analytical queries. 

 

This post highlights these core technology choices and the new features that InfluxDB 3 provides.

InfluxDB was driven by a vision to be more than just a time series database from its inception. We wanted a platform that could store raw events at scale, handle all forms of observational data, and generate time series on demand for diverse analytical tasks, both in real-time and historical contexts. Our goal was to create a database that could manage time series, observational, and historical data, transforming it from a mere metrics store into a comprehensive solution for advanced analytics.

The Evolution of InfluxDB

When we released InfluxDB 1.0 in 2016, it was written in Go and had a custom storage engine we built from scratch and specifically optimized for metrics workloads. We called it the Time-Structured Merge Tree (TSM), a variant of a log-structured merge-tree (LSM). The design of TSM paired a time series store with an in-memory inverted index to map metadata to the underlying time series.

This delivered excellent performance for low-to-medium-cardinality workloads, but database performance suffered as the amount of metadata in the inverted index increased. Similar structures were widely adopted by other time series databases, such as Facebook Gorilla, Prometheus, and others.

Cardinality limits have always posed challenges for time series databases, because performance declines with datasets containing large numbers of individual time series. We introduced the Time Series Index (TSI) to address this, extending the index to span disk and memory. That feature was GA in the 1.4 release, which came out in November 2017. With that enhancement, we could support workloads with cardinalities in the tens of millions to potentially one hundred million. Still, it wouldn’t provide the performance we needed for analytic queries. 

Our next release in 2020, InfluxDB 2.x, expanded beyond the core with a new scripting language, UI, and a multi-tenant, usage-based SaaS platform. However, rather than a new scripting language, our users needed unlimited cardinality, more affordable object storage, and enhanced query performance. Achieving these required a fundamental shift in the underlying architecture.

In November 2020, we announced InfluxDB IOx (eventually InfluxDB 3), our plan to reengineer the database to meet the demands of high-resolution workloads. We recognized that delivering fully on our vision for InfluxDB required a complete re-architecture of the database, having earlier identified several potential game-changing OLAP technologies. After validating the benefits, we rebuilt the new core in Rust and Apache Arrow, Apache DataFusion, Apache Parquet, and Arrow Flight. We coined the term FDAP stack to describe the stack. Over time, this choice of foundational tools evolved into a sophisticated stack for building analytic systems. These building blocks are the future of open data systems, real-time analytics, lakehouse, and data warehouse architectures.

The release of InfluxDB 3 delivers a robust, flexible, and high-performance database that meets our original vision of handling all forms of time series, observational, and historical data with unmatched performance and scale. We also incorporated the features customers were requesting: support for high-cardinality data, simultaneous analytical queries against multiple time series, more cost-effective object storage for historical data, and additional enhancements. We aimed to integrate with a broader set of third-party tools, with a primary focus on supporting SQL as a first-class query language.

Rebuilding for the Future: Why Rust?

At the beginning of 2020, I was very interested in Rust and felt that it would be the language of choice for implementing high-performance, server-side software. As we evaluated the future of InfluxDB, it became clear that achieving our long-term vision required a fundamental re-architecture. With that in mind, we chose to build on the best tools available in 2020, rather than the best language that had been available in 2013 when we created InfluxDB 1.0, which was Go. Go served us well due to its simplicity and performance; it’s excellent for building the concurrent, distributed systems that InfluxDB requires. We’ll continue to use it for many projects at InfluxData. 

However, when it came to v3, it was critical to build the core using Rust for maximum efficiency. Rust delivers superior performance, memory safety, and what the Rust community refers to as fearless concurrency. The guarantees that Rust provides for fearless concurrency are game-changing, eliminating entire classes of bugs that can be difficult to track down and fix in other languages, such as Go. Similar to Go, Rust does not suffer from the well-known security and stability issues associated with C/C++-based systems. Rust uses a novel “borrow checker” system to verify safety at compile time, which is faster than Go’s memory manager. Rust’s package management with crates.io and its robust ecosystem allowed us to leverage existing libraries and tools, speeding up our development process. 

Ultimately, the DataFusion query engine’s writing in Rust tipped the balance in our decision to use it as the query engine for InfluxDB 3 rather than adopting an engine written in C++. DataFusion’s alignment with Rust’s performance benefits and advanced SQL execution capabilities made it an ideal choice. This integration allowed InfluxDB 3 to leverage Rust’s performance benefits and DataFusion’s SQL capabilities, enhancing the database’s querying and processing power.

The FDAP Stack: Core Components of InfluxDB 3

One of our goals when developing InfluxDB 3 was to use as many open source libraries and tools as possible from other places. We wanted to build around larger community projects, contributing to them rather than reinventing the wheel on every database component. 

When we started rebuilding the InfluxDB base around FDAP technologies, we made a bet that they would gain an active community that would significantly contribute to their maturity, performance, and functionality. When we selected these projects as the database core in 2020, it wasn’t evident that they would be adopted and contributed to as broadly as they have been over the past four-plus years.

Apache Arrow Flight

For Core and Enterprise, we adopted Flight SQL as the high-performance query option. Flight SQL is gaining popularity in the data science community for its ability to transfer millions of rows from analytical databases like InfluxDB quickly. We’ve also enhanced our HTTP interface to support querying with results in JSONL, CSV, and Parquet response formats. Parquet stands out for enabling the highly efficient transfer of large  datasets, making it ideal for scaling data workflows.

DataFusion

DataFusion is a Rust-based SQL parser, planner, optimizer, and execution engine that uses Apache Arrow as its memory model. It offers high-performance query processing, including streaming vectorized execution, automatic parallelization, and query optimizations. 

Our team of engineers made significant contributions to DataFusion during the development of InfluxDB 3, collaborating closely with a large global community. When we embarked on the InfluxDB rebuild in mid-2020, the project was still in its early stages of development. However, it has since picked up momentum, thanks to years of development from hundreds of individuals and countless engineering hours. It has recently been designated as a Top-Level Project of the Apache Software Foundation. Performance has long been a core focus of the DataFusion community, and speed has attracted significant contributions from an active and growing community. 

As DataFusion continues to mature with additional feature enhancements and performance improvements, these functionalities are directly integrated into InfluxDB 3 and other systems built on DataFusion. This alignment enables us to collaborate with a community of passionate engineers to efficiently and quickly develop advanced database technology in DataFusion. 

Apache Arrow 

Apache Arrow is a columnar, in-memory data format used by InfluxDB 3 for efficient data representation and processing. Arrow standardizes data representation in memory at its core, using best-practice and cache-efficient columnar layouts. It simplifies the representation of complex data types, improving performance and ease of use.

When we began using the Rust implementation of Arrow, arrow-rs, it was relatively new. Over time, InfluxData and many other contributors have invested significant time and expertise in making it one of the best Arrow implementations. This includes many highly optimized computation kernels and complete type support. It is unlikely that we would have had such features in InfluxDB 3 had we tried to reimplement this ourselves.

Apache Parquet

Parquet is the primary format for bulk data exchange with other systems and InfluxDB 3. While Arrow is used as the in-memory and over-wire format for Flight, Parquet enables efficient bulk data transfer and seamless integration with external tools and systems like lakehouses, Apache Iceberg, and many data warehouses.

This marks a move away from InfluxDB as a standalone system to one designed for interoperability. Tools like Snowflake, Databricks, and Spark can read Parquet files directly, providing for easier integration without requiring a translation layer.

By adopting Parquet as the standard exchange format supporting emerging table standards like Apache Iceberg and Delta Lake (with flexibility to adopt others like XTable), InfluxDB 3 is becoming a first-class citizen in the modern data ecosystem, built to integrate with the broader lakehouse and analytics stack.

Each component is well-engineered for its specific function, and by starting with a solid foundation, our team can focus most of its efforts on ensuring they work seamlessly together. They supported the time series use cases with the high performance we required. Achieving this required a deep understanding of each component’s capabilities and limitations, as well as engineering solutions, such as specialized time series optimizations, that could bridge these gaps to form a cohesive and efficient system. 

Figure 1: InfluxDB 3 query architecture

Benefits of the Rebuild

Choosing the FDAP stack has significantly improved the database’s performance, making it much faster and more efficient, with a flexible architecture. It delivers significant gains in ingest efficiency, scalability, data compression, storage costs, and query performance on higher cardinality data. By removing cardinality limits, users can simultaneously bring in unlimited amounts of time series data and analytical queries against many time series, even as data complexity and cardinality increase. 

For developers, building on the FDAP stack means constructing a data engine that processes queries efficiently and manages data storage and retrieval to maximize performance across different workloads. Operationally, it involves setting up a system that handles concurrent data reads and writes, ensures data integrity and consistency, and provides robust fault tolerance mechanisms. Architecturally, it is a design that can scale horizontally and vertically, seamlessly adapting to varying loads and data sizes.

Many modern data tools and technologies can integrate with parts of the FDAP stack to enhance their capabilities. For instance, analytical tools requiring fast query execution can benefit from DataFusion and Arrow’s rapid data processing features. Systems requiring efficient data transport can leverage Arrow Flight, and those requiring optimized storage can utilize Parquet.

Apache Iceberg and the FIDAP Stack

On the horizon, we will add Iceberg support. Iceberg brings table-level abstraction and powerful metadata management to Parquet-backed datasets, enabling direct access from data lakehouses and warehouses like Databricks, Snowflake, and AWS Athena. By writing data in Parquet and exposing it through Iceberg, InfluxDB 3 makes time series data fully queryable, with no custom connectors or translation layers required. 

InfluxDB 3 Open Source 

With that foundation in place, we’re bringing the architecture, performance, and interoperability of InfluxDB 3 to the open source community. The latest InfluxDB 3 Core is a Rust-based, MIT/Apache 2 licensed time series engine built on the same FDAP stack and stateless architecture as our commercial offering. It is purpose-built to provide developers with a powerful, real-time engine for modern data workloads, backed by the same innovations that power our enterprise-grade deployments.

InfluxDB 3 Core is a high-performance, real-time data engine. This focused approach allows Core to deliver exceptional performance for real-time monitoring, data collection, and streaming analytics use cases. By optimizing specifically for this pattern, we’ve achieved query response times under 10ms for last-value queries and under 50ms for hour-long ranges. Core also persists data into Parquet files for long-term storage and access by third-party systems.

InfluxDB 3 Enterprise builds on Core’s foundation for users requiring historical analysis by adding compaction capabilities that enable efficient querying across any time range. This separation allows for Core to maintain its performance characteristics while Enterprise handles the complex requirements of managing historical data.

A key feature of Core and Enterprise is their “diskless” architecture, using object storage as the sole persistence layer. While they can use local disks, their stateless operation with object storage enables dynamic environments where data is seamlessly accessible to third-party systems that can read directly from object storage. 

Both products also include the new Processing Engine that allows users to define Python scripts to collect, process, transform, and monitor data on the fly directly within the database. The plugin API includes the ability to query the database, write data back into the database, and connect to any third-party service enabled through Python’s ecosystem of libraries and tools. We’re excited about the possibilities the plugin system will allow, particularly when paired with the fast, recent data query engine and last-value cache.

InfluxDB 3.2 and InfluxDB 3.3 added the general availability of InfluxDB 3 Explorer, a UI for querying, exploring, and visualizing data, and managed plugins for the Processing Engine to handle common time series tasks. The latest release, InfluxDB 3.4, is available for both Core and Enterprise and includes automation features for setup and workflow.

Figure 2: Write path with the WAL Flush Plugin: Plugins defined in the Processing Engine can intercept write-ahead log events, enabling real-time processing before data is flushed to object storage

Scalable by Design: A Stateless, Diskless Architecture for Modern Workloads

Core and Enterprise introduce a stateless, component-based architecture that separates compute from storage and runs entirely on object storage – what we refer to as a “diskless” architecture. While local disk is still supported, both Core and Enterprise are designed to persist all state to services like S3, GCS, or Azure Blob. Writes go into memory first, where they’re validated and buffered, then flushed to object storage as Parquet for durability. A queryable buffer holds the most recent data, enabling low-latency queries without waiting on files. This design delivers operational simplicity, horizontal scalability, and durability out of the box, without the need for complex clustering logic or data replication layers.

Figure 3: InfluxDB 3 write path.

At the heart of this architecture are modular services that handle distinct responsibilities:

• Ingesters handle real-time write workloads. They buffer incoming data, deduplicate it in memory, and persist it as small Parquet files to object storage.
• Queriers execute low-latency analytical queries, scanning persisted Parquet data or querying hot data held in memory by ingesters.
• Compactors optimize data layout by shuffling data to be sorted by time series in larger blocks of time, improving read performance and enabling long-term historical analysis.
• The centralized catalog implemented on top of object storage (shared across all components) tracks schema metadata, allowing the system to remain fully stateless while maintaining consistency and coordination.

Data flows through these components efficiently. Ingesters validate the schema and partition data by time before writing. Table schemas are inferred automatically at write time (users don’t need to define tables or columns in advance). A custom multi-column sort-merge deduplication strategy (based on DataFusion operators) removes duplicate rows while maintaining write throughput.

To maintain query performance as data ages, compactors continuously merge small files into larger, non-overlapping Parquet files in the background. This merging of files reduces storage overhead, enables fast pruning during queries, and avoids unnecessary recomputation. Compaction levels help balance frequency and computational cost, ensuring the system remains performant at scale.

Queriers serve both recent and historical data. They pull persisted Parquet files from object storage and query recent in-memory buffers from ingesters as needed. Query plans are built using DataFusion, with support for both SQL and InfluxQL, and benefit from DataFusion’s vectorized execution, predicate pushdown, and native support for Parquet.

A key feature of this architecture is the Last Value Cache (LVC) and Distinct Value Cache (DVC), which are in-memory optimizations designed to accelerate familiar time series query patterns. The LVC retains the latest values for configurable tag hierarchies, enabling sub-10ms response times for “last known value” queries. The DVC provides ultra-fast lookups for tag values, improving UI responsiveness and exploratory workflows.

By design, this architecture eliminates local state from critical paths, simplifies upgrades, enables dynamic scale-out, and allows components to recover independently. This separation of concerns also enables workload isolation in Enterprise deployments, where ingest, query, processing, and compaction components can be scaled and scheduled separately.

Together, these capabilities allow InfluxDB 3 Core to operate as a lightweight, real-time analytics engine optimized for recent data. Due to these performance properties, we’ve set a configuration option that limits a query plan to 432 Parquet files, corresponding to a seventy-two-hour time range, given the ten-minute time blocks. The enterprise version lifts this limitation, hence providing long-term query performance, high availability, and advanced multi-node deployment features. This is possible in the enterprise version, due to its compactor, which rewrites the ten-minute files into larger blocks of time. 

Closing Thoughts

The Apache Arrow ecosystem, including Parquet and DataFusion, will serve as the foundation for future OLAP and large-scale data processing systems. In addition to InfluxDB 3 Core, we’re investing our open source efforts in these standards – and sometimes leading those upstream projects – so that the community continues to grow and the Apache Arrow set of projects becomes more straightforward to use, with added features and functionality. We obtain more reliable software because it has been proven across various environments and use cases. That’s the power of open source – a level of maturity and resilience that closed, proprietary systems simply can’t match.

This marks the culmination of a four-and-a-half-year effort to re-architect the system from the ground up to meet the needs of modern time series workloads. With InfluxDB 3, we’ve built a foundation that handles all forms of time series, observational, and historical data with the scale, performance, and flexibility our users need. This isn’t just about solving today’s challenges, it’s about ensuring InfluxDB is built for the kinds of problems developers are solving today and the ones they’ll face next.