Dataproc Flexible VMs

Console

To create a cluster with secondary worker flexible VMs:

1. Open the Dataproc Create a cluster on Compute Engine page in the Google Cloud console.

2. The Set up cluster panel is selected with fields filled in with default values. You can change the suggested name and the cluster region, and make other changes. Make sure that Any is selected as the cluster Zone to allow Dataproc autozone placement to choose the zone that has the best availability of the VM types specified in your flexible VM lists.

3. Select the Configure nodes panel. In the Secondary worker nodes section, specify the number and preemptibility of secondary workers.

   • Click Add a secondary worker for each rank of secondary workers, specifying one or more machine types to include in each rank.

4. After confirming and specifying cluster details in the cluster create panels, click Create.

gcloud

Use the gcloud dataproc clusters create command to add multiple secondary-worker-machine-types flags to specify ranked flexible VM lists for Dataproc secondary workers. The default flexible VM secondary worker type is Spot, which is a preemptible type.

In the following gcloud CLI example, Dataproc attempts to provision secondary workers with n2-standard-8 VMs first (rank 0). If n2-standard-8 machines are not available, Dataproc attempts to provision secondary workers with either e2-standard-8 or t2d-standard-8 VMs (rank 1).

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --zone="" \
    --master-machine-type=n1-standard-8 \
    --worker-machine-type=n1-standard-8 \
    --num-workers=4 \
    --num-secondary-workers=4 \
    --secondary-worker-type=non-preemptible \
    --secondary-worker-machine-types="type=n2-standard-8,rank=0" \
    --secondary-worker-machine-types="type=e2-standard-8,type=t2d-standard-8,rank=1"

Notes:

• --zone="": The Flexible VM feature requires Dataproc autozone placement to allow Dataproc to choose the zone that has your VM types available for use. Passing an empty value ("") to the --zone flag overrides any zone selection specified in your default gcloud config list.

• Dataproc generates component role properties based on machine cores and memory. You can override these system-generated properties with the --properties flag, using the following syntax:

  --properties="ROLE:MACHINE_TYPE:COMPONENT_PREFIX:COMPONENT_PROPERTY=VALUE"
  

  Only the secondary_worker role is the only supported role.

  In the following example, the --properties flag changes the number of cores of e2-standard-8 machines assigned to secondary worker nodes from 8 to 6:

  --properties="secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6"
  

API

Use the instanceFlexibilityPolicy.instanceSelectionList as part of a Dataproc API clusters.create request to specify a ranked list of machineTypes for secondary workers.

Example:

The following JSON snippet from a Dataproc clusters.create request body specifies secondary workers machine types for rank 0 and rank 1.

"config": {
  "secondaryWorkerConfig": {
    "instanceFlexibilityPolicy": {
      "instanceSelectionList": [
        {
          "machineTypes": [
            "n1-standard-4",
            "n2-standard-4"
          ],
          "rank": 0
        },
        {
          "machineTypes": [
            "e2-standard-4",
            "n2d-standard-4"
          ],
          "rank": 1
        }
      ]
    }
  }
}

Use cluster properties to customize component roles: Dataproc generates component role properties based on VM cores and memory. You can override these system-generated properties by adding SoftwareConfig.properties to your clusters.create request, using the following key=value syntax:

ROLE:MACHINE_TYPE:COMPONENT_PREFIX:COMPONENT_PROPERTY=VALUE

Only the secondary_worker role is the only supported role.

In the following example, the properties field changes the number of cores assigned to the secondary worker node of an e2-standard-8 VM from 8 to 6:

"secondary_worker:e2-standard-8:yarn:yarn.nodemanager.resource.cpu-vcores=6"