By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
World of SoftwareWorld of SoftwareWorld of Software
  • News
  • Software
  • Mobile
  • Computing
  • Gaming
  • Videos
  • More
    • Gadget
    • Web Stories
    • Trending
    • Press Release
Search
  • Privacy
  • Terms
  • Advertise
  • Contact
Copyright © All Rights Reserved. World of Software.
Reading: Pinterest Automates Hadoop Cluster Scaling and Migration with Internal Orchestration System
Share
Sign In
Notification Show More
Font ResizerAa
World of SoftwareWorld of Software
Font ResizerAa
  • Software
  • Mobile
  • Computing
  • Gadget
  • Gaming
  • Videos
Search
  • News
  • Software
  • Mobile
  • Computing
  • Gaming
  • Videos
  • More
    • Gadget
    • Web Stories
    • Trending
    • Press Release
Have an existing account? Sign In
Follow US
  • Privacy
  • Terms
  • Advertise
  • Contact
Copyright © All Rights Reserved. World of Software.
World of Software > News > Pinterest Automates Hadoop Cluster Scaling and Migration with Internal Orchestration System
News

Pinterest Automates Hadoop Cluster Scaling and Migration with Internal Orchestration System

News Room
Last updated: 2025/07/31 at 2:07 PM
News Room Published 31 July 2025
Share
SHARE

Recently, Pinterest disclosed its internal orchestration framework, called Hadoop Control Center (HCC), to automate the scaling and migration of its large-scale Hadoop clusters. This move addresses the operational complexity and limitations Pinterest previously faced when managing thousands of nodes across dozens of YARN clusters on AWS.

Historically, Pinterest maintained fixed-size Auto Scaling Groups (ASGs) for its Hadoop infrastructure. This configuration, while safe, meant autoscaling was effectively disabled. Manual adjustments to cluster size, especially scaling in, required extensive human intervention via Terraform. These operations involved complex sequences to drain and decommission nodes safely, while avoiding disruption to batch workloads using MapReduce, Spark, and Flink. The process was both time-consuming and error-prone, often leading to infrastructure duplication and resource waste.

The introduction of HCC transforms this manual workflow into a fully automated system that manages Hadoop cluster resizing and node migration in real-time. Operators can now request scaling actions through a unified command-line interface. Behind the scenes, HCC coordinates interactions with AWS services and Hadoop components to ensure safe decommissioning of nodes, data integrity, and service continuity.

At the heart of HCC is a manager-worker architecture distributed across Pinterest’s Virtual Private Clouds (VPCs). Each VPC runs a manager node that caches cluster state and delegates actions to worker nodes. These workers interact with a custom component called the Hadoop Manager Class (HMC), which orchestrates the step-by-step decommissioning and scaling of nodes. HMC monitors key cluster metrics via JMX, updates configuration files, issues AWS API calls, and schedules internal threads to handle draining, termination, and clean-up operations.

 

HCC logical schema

One of HCC’s key features is its ability to safely migrate nodes during in-place upgrades. Rather than deploying parallel green clusters with new configurations, Pinterest’s engineers can launch a new ASG with updated instance types or AMIs and allow HCC to integrate these new nodes into the existing cluster. HCC then begins draining data and workloads from the old ASG, monitors for completion of shuffle and HDFS replication, and removes decommissioned instances in a controlled fashion. This method minimizes cost, avoids duplicative infrastructure, and removes the need to re-provision capacity or IP space for each migration.

Additionally, the system manages ASG scale-in protection at the instance level, ensuring AWS does not randomly terminate Hadoop nodes that are not ready for removal. After a successful decommission, HCC removes the node from the cluster and updates relevant Terraform variables to maintain configuration consistency and avoid drift during future infrastructure changes.

While HCC has dramatically improved operational efficiency, Pinterest is already looking to extend its capabilities. The team plans to add auto-remediation features for handling unhealthy nodes detected by AWS, enable lifecycle rotation based on OS age or AMI version, and incorporate AWS event triggers for smarter node management. These enhancements are aimed at making Pinterest’s Hadoop infrastructure more autonomous and resilient.

The move to HCC has enabled Pinterest to scale its data processing platforms on demand, reduce the risk of human error, and safely perform in-place migrations with minimal downtime or application impact. As data infrastructure becomes more dynamic and elastic in the cloud, Pinterest’s approach provides a compelling blueprint for managing stateful systems like Hadoop with modern automation principles.

Another tech giant, Uber, disclosed its phased strategy for shifting its massive Hadoop-based batch analytics stack—managing up to exabytes of data and tens of thousands of servers—to Google Cloud Platform. Initially, Uber migrated core components onto GCP-based Infrastructure-as-a-Service to replicate its on‑premises environment with minimal disruption. Subsequently, the team planned gradual adoption of managed services such as Dataproc, BigQuery, and Google Cloud Storage, alongside cloud-native abstractions and reverse‑compatible proxies for Spark, Hive, and Presto. Uber’s layered approach minimizes client impact while enabling a modern, scalable, and elastic platform in the cloud.

These two cases highlight a shared paradigm: large-scale Hadoop systems can be safely migrated or modernized with minimal downtime through well-designed orchestration, replication, and compatibility tooling.

Sign Up For Daily Newsletter

Be keep up! Get the latest breaking news delivered straight to your inbox.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Share This Article
Facebook Twitter Email Print
Share
What do you think?
Love0
Sad0
Happy0
Sleepy0
Angry0
Dead0
Wink0
Previous Article Amazon Discounts USB-C AirPods Max to $449.99
Next Article The Grave Long-Term Effects of the Gaza Malnutrition Crisis
Leave a comment

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Stay Connected

248.1k Like
69.1k Follow
134k Pin
54.3k Follow

Latest News

Starbucks pioneered mobile ordering. Now the coffee giant is ditching a pickup-only store concept.
Computing
UK tech funding roundup: This week’s deals from Ultromics to Fluid Focus – UKTN
News
Best Noise-Cancelling Earbuds 2025: Top picks to block out noise tested
Gadget
More Intel Driver Maintainer Changes In Linux 6.17
Computing

You Might also Like

News

UK tech funding roundup: This week’s deals from Ultromics to Fluid Focus – UKTN

1 Min Read
News

Nintendo Sold Over 6 Million Switch 2 Consoles, Working to Avoid Shortages

5 Min Read
News

AWS sees revenue and profit rise in Q2, bats away competitive concerns | Computer Weekly

4 Min Read
News

Samsung’s Galaxy Buds2 Pro are over $100 off, but this deal won’t last long

2 Min Read
//

World of Software is your one-stop website for the latest tech news and updates, follow us now to get the news that matters to you.

Quick Link

  • Privacy Policy
  • Terms of use
  • Advertise
  • Contact

Topics

  • Computing
  • Software
  • Press Release
  • Trending

Sign Up for Our Newsletter

Subscribe to our newsletter to get our newest articles instantly!

World of SoftwareWorld of Software
Follow US
Copyright © All Rights Reserved. World of Software.
Welcome Back!

Sign in to your account

Lost your password?