High-Performance Remote Shuffle Service on Amazon EMR with Apache Celeborn

In today's Big Data environment, organizations processing bulk loads with Apache Spark face a recurring dilemma: reduce costs or ensure job reliability. This balance becomes especially complex when using Amazon EC2 Spot Instances, as their sudden interruption causes the loss of local shuffle data and forces entire stages to be recomputed, eroding the expected savings. In addition, local shuffle storage leads to oversizing of nodes and keeps resources idle for long periods. In the face of these challenges, Apache Celeborn emerges as a Remote Shuffle Service (RSS) solution that decouples shuffle data from the executor lifecycle, allowing Spark to run on Spot instances without risk of loss and scaling resources independently.

Celeborn's architecture is based on a leader-worker-customer model, where lead nodes manage metadata, workers store and replicate shuffle blocks, and customers integrate with compute engines. By replacing Spark's native ShuffleManager, runners send shuffle data directly to Celeborn workers, who consolidate and replicate it (for example, using the spark.celeborn.client.push.replicate.enabled=true option). This eliminates dependency on the local disk and allows compute nodes to scale freely without triggering recomputations. In a dedicated Amazon EKS cluster, Celeborn can serve multiple EMR environments (both on EKS and on EC2) through an internal load balancer, offering high availability via Raft and persistence on EBS volumes for the leader nodes.

The practical implementation described by AWS shows how to deploy Celeborn in a separate EKS cluster, with workers using ephemeral NVMe storage for performance and cross-worker replication for fault tolerance. Monitoring is done using ADOT Collector, which sends metrics to Amazon Managed Service for Prometheus or a self-managed Prometheus-Grafana stack. Critical settings include disabling Spark's External Shuffle Service, setting Celeborn's shuffle manager, and adjusting options such as spark.sql.adaptive.localShuffleReader.enabled=false. This approach not only improves resiliency, but also optimizes cost by enabling intensive use of Spot Instances without compromising stability.

For companies looking to maximize the performance of their analytics loads in the cloud, integrating a remote shuffle service like Celeborn represents a quantum leap. However, its adoption requires a deep understanding of the cloud infrastructure and the particularities of Spark. At Q2BSTUDIO we offer AWS and Azure cloud services that include design, deployment and optimization of Big Data environments, as well as custom applications to adapt solutions such as Celeborn to the specific needs of each organization. Our team of experts in artificial intelligence, cybersecurity and business intelligence services can help you implement these architectures securely and efficiently, integrating tools such as Power BI for the visualization of metrics or AI agents for process automation.

Ultimately, Apache Celeborn solves the trade-off between cost and reliability in Spark, enabling enterprises to deploy shuffle-intensive workloads with complete confidence. Whether your organization is looking to modernize its data platform or needs advice on implementing a remote shuffle service, Q2BSTUDIO is ready to provide you with tailored software solutions that boost your analytics performance. The combination of EMR, Celeborn, and a well-designed cloud infrastructure is the path to scalable, efficient, and future-proof data processing.