Isolate your GKE workloads using sole-tenant nodes

This page shows you how to isolate your workloads running on Google Kubernetes Engine (GKE) Standard clusters by using Compute Engine sole-tenant nodes, which are dedicated physical servers that only run a specific project's VMs. You can use sole-tenant nodes to keep your VMs physically separated from VMs in other projects or to group your VMs together on the same host hardware.

This feature is not supported on GKE Autopilot. To learn more about Autopilot security boundaries, see Security boundaries in Autopilot.

To use sole-tenant nodes in GKE, you create a sole-tenant node group in Compute Engine and use that node group to create a GKE node pool. You can create node pools using sole-tenant nodes from the GKE cluster's project or shared sole-tenant nodes using resources across projects within your organization. For example, you can designate an owner project in your organization where you create sole-tenant node groups, and consumer projects where you have clusters with node pools using those node groups from the owner project.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Request additional quota

Sole-tenant nodes are large (for example, 96 vCPUs and 624 GB memory), and the default project quotas are too low to handle them.

You must request a quota increase for Compute Engine API CPUs on the quotas page to create sole-tenant nodes. For more information about requesting an increase, see Quotas.

The number of CPUs you need depends on the following:

• Your current CPU quota usage.
• The size of the node group.
• The node template.

Examine the number of cores of your sole-tenant node type, and request a CPU limit of at least that amount, preferably with an additional 10 CPUs to ensure that your project has adequate capacity.

Create a sole-tenant node template

A sole-tenant node template is a regional resource that defines and applies properties to every node when creating a node group. For more information, see node types.

If you are using shared sole-tenant nodes, ensure that you complete these instructions in the owner project for the node group, not the consumer project of the cluster.

gcloud compute sole-tenancy node-templates create TEMPLATE_NAME \
    --node-type=NODE_TYPE \
    --node-affinity-labels=AFFINITY_LABELS \
    --region=COMPUTE_REGION

Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1/nodeTemplates/template-name].

Create a sole-tenant node group

A node group is a set of sole-tenant nodes in a specific zone from the same sole-tenant node template. For regional clusters and node pools available in multiple zones, you must create a node group with the same name in each zone. Ensure that you have sufficient quota before completing this step.

If you are using shared sole-tenant nodes, ensure that you complete these instructions in the owner project for the node group, not the consumer project of the cluster.

gcloud compute sole-tenancy node-groups create GROUP_NAME \
    --zone COMPUTE_ZONE \
    --node-template TEMPLATE_NAME --target-size TARGET_SIZE \
    --share-settings=SHARE_SETTING \
    --share-with=PROJECTS

Create a GKE sole-tenant node pool

Now that you have created a sole-tenant node group in Compute Engine, you can create a sole-tenant node pool.

If you already have a GKE cluster, you can add a sole-tenant node pool to it. If not, create a cluster with gcloud container clusters create.

If you are using a regional cluster or if your node pool is available in multiple zones, you must create a node group with the same name in each of those zones. If you cannot reuse the name in each zone, create separate node pools for each zone.

Create a node pool with a sole-tenant node group from the cluster's project

To create a node pool using a node group from the cluster's project, pass the name of the sole-tenancy node group. If you want to use a shared sole-tenant node group, or you want to use custom node affinity labels that you configured when you created the sole-tenant node template, see the instructions to create a node pool with shared sole-tenant node using a node affinity file.

Create a new node pool with a specified node group:

For zonal node pools:

  gcloud container node-pools create NODE_POOL_NAME \
      --node-group GROUP_NAME --cluster CLUSTER_NAME \
      --zone COMPUTE_ZONE --machine-type=MACHINE_TYPE \
      --node-locations=COMPUTE_ZONE

For regional node pools:

  gcloud container node-pools create NODE_POOL_NAME \
      --node-group GROUP_NAME --cluster CLUSTER_NAME \
      --region COMPUTE_REGION --machine-type=MACHINE_TYPE \
      --node-locations=COMPUTE_ZONE1,COMPUTE_ZONE2

Replace the following:

• NODE_POOL_NAME: a name for the new node pool.
• GROUP_NAME: the name of the existing sole-tenancy node group you want to use.
• CLUSTER_NAME: the name of the cluster in which you're creating the node pool.
• COMPUTE_REGION: the same region as your cluster.
• MACHINE_TYPE: the node pool machine type.
• COMPUTE_ZONE1, COMPUTE_ZONE2,[...]: the zones for the sole-tenancy node groups.

For a full list of optional flags you can specify, refer to the gcloud container node-pools create documentation.

Create a node pool with shared sole-tenant node using a node affinity file

To create a node pool using a shared sole-tenant node group, you must use node affinity labels. You can also use node affinity to choose node groups from the same project.

To reference labels, follow the instructions to create a node pool, replacing the --node-group flag with --sole-tenant-node-affinity-file flag. Pass a JSON file with those labels. For example, see the following command:

  gcloud container node-pools create sole-tenant-node-pool \
      --sole-tenant-node-affinity-file /path/to/affinity/file --cluster my-cluster \
      --region us-central1 --machine-type n1-standard-4  \
      --node-locations=us-central1-a,us-central1-b,us-central1-c

The following sections describe use cases for node affinity labels with sole-tenant nodes.

Specify a node group name from a project

To use sole-tenant node groups from a specific project and node group name, specify these values in the node affinity file. You can use this affinity if you are creating a multi-zone node pool where the node group names from each zone must match. To use any node group in the specified owner project, omit the entire compute-googleapis.com/node-group-name block from the JSON file.

See the following example:

[
  {
    "key" : "compute.googleapis.com/project",
    "operator" : "IN",
    "values" : ["OWNER_PROJECT"]
  },
  {
    "key" : "compute.googleapis.com/node-group-name",
    "operator" : "IN",
    "values" : ["GROUP_NAME"]
  },
]

Specify custom labels

You can also use node affinity and anti-affinity to create a node pool using node groups with custom labels. Each node group is automatically assigned default affinity labels. You can reference the default labels as with the previous examples, or use custom labels designated when creating the node template. You can use custom labels to isolate or group workloads.

See the following example:

[
  {
    "key" : "compute.googleapis.com/project",
    "operator" : "IN",
    "values" : ["OWNER_PROJECT"]
  },
  {
    "key" : "workload",
    "operator" : "IN",
    "values" : ["frontend"]
  },
]

Limitations

When creating a node pool with sole-tenant nodes, you are responsible for managing the underlying capacity for sole-tenant node groups. Consider how the following features interact with sole-tenant node groups:

• You can enable cluster autoscaling, however it is limited by the capacity of the underlying node group. You can configure the node group autoscaler in Compute Engine to automatically manage the size of your sole-tenant node groups.
• You cannot enable node auto-provisioning for your cluster if you use sole-tenant nodes.
• You can use any node upgrade strategy, however you must ensure you have sufficient capacity to support the node pool using its chosen upgrade strategy.

What's next

• Read about how node pools work.
• Learn more about sole-tenant nodes.