On March 17, 2025, Google Cloud introduced a hierarchical namespace (HNS) feature in Cloud Storage, aiming to optimize AI and machine learning (ML) workloads by improving data organization, performance, and reliability.
AI/ML processes, particularly during model training, involve frequent checkpointing to save the model’s state. Traditional flat namespace storage systems handle folder renames by rewriting or deleting each object individually, which can be time-consuming and error-prone. With HNS, Cloud Storage now supports atomic folder-level operations, enabling faster and more reliable checkpointing. According to their blog post, Google’s benchmarking indicates that hierarchical namespace buckets can accelerate checkpoint writes by up to 20 times compared to flat buckets. This improvement is achieved through a new RenameFolder API that performs metadata-only operations, completing tasks in a fraction of the time required by flat namespace buckets.
Google also claims that real-world applications have demonstrated the benefits of HNS. For instance, AssemblyAI reported a 10x increase in throughput from Google Cloud Storage when using HNS with Cloud Storage FUSE, leading to a 15x improvement in training speed.
Additionally, HNS enhances performance by providing an optimized storage layout that supports higher queries per second (QPS) for read and write operations. This is particularly beneficial for AI/ML workloads running on large clusters, where synchronized I/O operations can create bottlenecks. Hierarchical namespace buckets offer up to 8 times higher initial object read and write QPS compared to flat namespace buckets, facilitating quicker ramp-up and better utilization of compute resources.
Jason Stevens, a Senior Director of Engineering at Google, echoed these sentiments, noting that:
Google Cloud Storage (GCS) Hierarchical Namespace (HNS) accelerates storage workloads that rely on filesystem semantics—like folder renames—boosting efficiency for AI workloads. With up to 20x faster checkpointing and 8x higher QPS, HNS helps maximize GPU & TPU utilization for AI/ML pipelines.
To enable the hierarchical namespace feature in Google Cloud Storage, you must configure it at the time of bucket creation, as it cannot be enabled retroactively on existing buckets. Using the gcloud CLI, this is done by running the gcloud storage buckets create command with the -enable-hierarchical-namespace flag, along with specifying the desired bucket name and location. Alternatively, in the Google Cloud Console, you can navigate to the Cloud Storage section, select “Create bucket,” and within the Advanced settings, check the option to enable the hierarchical namespace before completing the rest of the setup. Once enabled, the bucket is optimized for AI and machine learning use cases through support for filesystem-like folders, atomic renames, and improved throughput for read and write operations.
