Configuring YARN Capacity Scheduler Queues in AWS EMR

/, Big Data Performance, EMR, Hadoop, Scheduler, Spark/Configuring YARN Capacity Scheduler Queues in AWS EMR

Configuring YARN Capacity Scheduler Queues in AWS EMR


AWS EMR clusters by default are configured with a single capacity scheduler queue and can run a single job at any given time. This blog talks about how you can create and configure multiple capacity scheduler queues in YARN Capacity Scheduler during the creation of a new EMR cluster or when updating existing EMR clusters. For more details on configuration, refer to Apache documentation on Hadoop Capacity Scheduler.

In this example, we will be configuring the following queues:

NumberQueue NameMinimum CapacityMaximum CapacityMaximum Number of Running Jobs

This blog captures examples of how to submit Spark jobs to a specific queue. Mitylytics SaaS functionality models the app’s performance based on history and ensures these jobs are submitted to the best queue to achieve optimum cluster performance. Please visit Mitylytics and request the YARN Capacity scheduler Demo and Trial.

Creation of new EMR cluster with Capacity Scheduler configuration:

  1. In AWS console, Select EMR.
  2. Select “Create Cluster” option and in that select “Go to advanced options”.
  3. For this example, you should select Hadoop and Spark options. Copy and paste the following configuration under “Edit Software settings -> Enter Configuration.    

[  {
“Classification”: “capacity-scheduler”,
“Properties”: {
    “yarn.scheduler.capacity.maximum-am-resource-percent”: “0.6”,
    “yarn.scheduler.capacity.resource-calculator”: “org.apache.hadoop.yarn.util.resource.DominantResourceCalculator”,
     “yarn.scheduler.capacity.root.queues”: “default,gold,silver, bronze”,
     “yarn.scheduler.capacity.root.default.capacity”: “10”,
     “yarn.scheduler.capacity.root.default.user-limit-factor”: “2”,
     “yarn.scheduler.capacity.root.default.maximum-capacity”: “40”,
     “”: “40”,
     “yarn.scheduler.capacity.root.silver.capacity”: “30”,
     “yarn.scheduler.capacity.root.bronze.capacity”: “20”,
     “”: “2”,
     “yarn.scheduler.capacity.root.silver.user-limit-factor”: “2”,
     “yarn.scheduler.capacity.root.bronze.user-limit-factor”: “2”,
     “”: “70”,
     “yarn.scheduler.capacity.root.silver.maximum-capacity”: “60”,
     “yarn.scheduler.capacity.root.bronze.maximum-capacity”: “50”,
     “”: “RUNNING”,
     “yarn.scheduler.capacity.root.silver.state”: “RUNNING”,
     “yarn.scheduler.capacity.root.bronze.state”: “RUNNING”,
     “”: “*”,
     “yarn.scheduler.capacity.root.silver.acl_submit_applications”: “*”,
     “yarn.scheduler.capacity.root.bronze.acl_submit_applications”: “*”
   “Classification”: “yarn-site”,
   “Properties”: {
     “yarn.acl.enable”: “true”,
     “yarn.resourcemanager.scheduler.class”: “org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler”

  1. The rest of the configurations are with your default configuration (Hardware, General Cluster settings and Security). For more details on EMR configuration, refer to the AWS EMR documentation.
  2. Selection of final “Create Cluster” option will launch a new cluster with configured YARN capacity scheduler queues.
  3. Check updated values by selecting Cluster Configurations tab (shown in above picture).
  4. You can also check configuration details in Nodes YARN resource manager. In this example, YARN resource manager details are at http://EMR-Cluster-Master:8088/cluster/scheduler
  5. By default, submitting Spark job without specifying queue details, jobs will be executed on default queue. Here is the example:

Here is the Yarn resource manager UI view:


For submitting Jobs to a specific queue, refer to the section “Submitting a Spark Job to Different Queues”.

Modifying Active EMR Cluster with Capacity Scheduler Configuration

For this example, I have created EMR cluster with default options. Here are screenshots of YARN resource manager.

Scheduler view in YARN resource manager UI:


Applications View in Yarn resource manager UI:

Log in to the Cluster Master node and cd to “/etc/hadoop/conf.empty/” directory. Copy original capacity-scheduler.xml to the capacity-scheduler.xml.default (useful in reverting the changes), similarly copy yarn-site.xml file.

[hadoop@ip-172-31-37-60 ~]$ cd /etc/hadoop/conf.empty/
[hadoop@ip-172-31-37-60 conf.empty]$ sudo cp capacity-scheduler.xml capacity-scheduler.xml.default
[hadoop@ip-172-31-37-60 conf.empty]$ sudo cp yarn-site.xml yarn-site.xml.default

Update yarn-site.xml with following new lines.


capacity-scheduler.xml with the following modifications (Linked to default queue)

<     <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>

>     <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
<     <value>default</value>

>     <value>gold,silver,bronze,default</value>
<     <value>100</value>

>     <value>10</value>
<     <value>1</value>

>     <value>2</value>
<     <value>100</value>

>     <value>40</value>

Update following new lines to the capacity-scheduler.xml


Stop and Start YARN resource manager.

[hadoop@ip-172-31-37-60 conf.empty]$ sudo vi yarn-site.xml
[hadoop@ip-172-31-37-60 conf.empty]$ sudo vi capacity-scheduler.xml
[hadoop@ip-172-31-37-60 conf.empty]$ sudo vi capacity-scheduler.xml
[hadoop@ip-172-31-37-60 conf.empty]$ sudo /sbin/stop hadoop-yarn-resourcemanager
hadoop-yarn-resourcemanager stop/waiting
[hadoop@ip-172-31-37-60 conf.empty]$ sudo /sbin/start hadoop-yarn-resourcemanager
hadoop-yarn-resourcemanager start/running, process 21452
[hadoop@ip-172-31-37-60 conf.empty]$ 

Verify YARN resource manager UI reflects above modifications.


Submitting a Spark Job to Different Queues

Log in to Cluster and submit Spark job to the gold queue with –queue option in spark-submit command. Here is an example

[hadoop@ip-172-31-37-60 spark]$ bin/spark-submit –class org.apache.spark.examples.SparkPi –queue gold –master yarn ./examples/jars/spark-examples_2.11-2.2.0.jar 1000
17/10/17 23:56:20 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Pi is roughly 3.141489791414898
[hadoop@ip-172-31-37-60 spark]$ 

Scheduler view in YARN resource manager UI:

Created two users (user1 and user2) in the Master node. Here is the example of submitting the Spark job from user1 to the silver queue:

[hadoop@ip-172-31-37-60 ~]$ sudo su -c “/usr/lib/spark/bin/spark-submit –class org.apache.spark.examples.SparkPi –queue silver –master yarn /usr/lib/spark/examples/jars/spark-examples_2.11-2.2.0.jar 500” -s /bin/sh user1
17/10/18 00:33:58 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Pi is roughly 3.141932142838643
[hadoop@ip-172-31-37-60 ~]$ 

YARN resource manager UI scheduler view showing user and queue details:

In the next example, I have submitted a Spark-Perf batch job to the Silver queue by user1, Word count job to the gold queue by user hadoop, and SparkPI job to the bronze queue by user2.

With the proposed configuration of the YARN Capacity Scheduler, you can achieve features such as Capacity Guarantees, Elasticity, Multi-tenancy. Once modified queues are in operation, administrators can dynamically change the queue capacity in the file capacity-scheduler.xml and execute the following command to reflect the new changes.

yarn rmadmin -refreshQueues


Please leave a comment on this blog.

By | 2017-12-01T16:01:50+00:00 November 2nd, 2017|Categories: AWS, Big Data Performance, EMR, Hadoop, Scheduler, Spark|7 Comments


  1. Sam December 12, 2017 at 1:03 pm - Reply

    Followed all steps and created queues. While submitting the spark Job with below command, dont see the jobs running in YARN console. However, the job runs and completes without showing which queue it ran. Any idea what could be the issue?

    spark-submit –master yarn –deploy-mode client –executor-memory 2G –queue gold –conf spark.cores.max=4

    • mityweb December 12, 2017 at 4:26 pm - Reply

      Hello Sahoo,

      Thanks for trying input on our Blog.

      FYI, I used similar options to submit SPark-PI job
      spark-submit –class org.apache.spark.examples.SparkPi –deploy-mode client –master yarn –queue gold –executor-memory 2G –conf spark.cores.max=4 ./examples/jars/spark-examples_2.11-2.2.0.jar 1000

      Above JOB was a success and shown in YARN Resource Manager UI (I hope you are referring to this as YARN console).

      I hope you can see all the configured queues in your YARN Resource Manager UI. Can you check respective job details are visible at your apps http://:8088/cluster/apps

  2. Sandeep May 29, 2018 at 8:04 am - Reply

    Hi ,

    Can we connect this to Linux Groups and add users to that group and make sure users from that group can run JOB which queue they are tagged to

    for example:

    Can i create a gold gorup on Linux and tag YARN to that group and all users in this group when submit jobs should run in GOLD

    Sandeep R Diddi

    • mityweb May 30, 2018 at 2:53 am - Reply

      Hello Sandeep,

      I this blog we recommended to directly assign Queue in Spark Submit command. You can ask the respective group to use specific group. We will be presenting how to auto assign groups to individual users through our mitysubmit in the upcoming AI+SPark summit You can refer to that for more details

  3. neeli August 17, 2018 at 10:28 am - Reply

    I used the same setting as above to launch the cluster. Even though i submit the jobs to different queues in EMR they still run in sequential.
    I’m using the same user to submit the spark jobs. Is this the reason why I’m not able to run concurrent jobs in EMR ?

  4. neeli August 17, 2018 at 10:42 am - Reply

    Jobs dont run in concurrent in AWS EMR cluster.

    • mityweb August 18, 2018 at 5:23 am - Reply

      By default, EMR sets user-limit-factor in the capacity-scheduler.xml to 1. Becuase of this JOBS cannot be run in concurrent. If you change this, even with a single queue, you can run concurrent jobs. Check this value in your EMR configuration.

Leave A Comment