Use service accounts

Some data sources support data transfer authentication by using a service account through the Google Cloud console, API, or the bq command line. A service account is a Google Account associated with your Google Cloud project. A service account can run jobs, such as scheduled queries or batch processing pipelines by authenticating with the service account credentials rather than a user's credentials.

You can update an existing data transfer with the credentials of a service account. For more information, see Update data transfer credentials.

The following situations require updating credentials:

• Your transfer failed to authorize the user's access to the data source:

  Error code 401 : Request is missing required authentication credential. UNAUTHENTICATED

• You receive an INVALID_USER error when you attempt to run the transfer:

  Error code 5 : Authentication failure: User Id not found. Error code: INVALID_USERID

To learn more about authenticating with service accounts, see Introduction to authentication.

Data sources with service account support

BigQuery Data Transfer Service can use service account credentials for transfers with the following:

• Cloud Storage
• Amazon Redshift
• Amazon S3
• Campaign Manager
• Dataset Copy
• Display & Video 360
• Google Ad Manager
• Google Ads
• Google Merchant Center
• Google Play
• Scheduled Queries
• Search Ads 360
• Teradata
• YouTube Content Owner

Before you begin

• Verify that you have completed all actions required in Enabling BigQuery Data Transfer Service.
• Grant Identity and Access Management (IAM) roles that give users the necessary permissions to perform each task in this document.

Required permissions

To update a data transfer to use a service account, you must have the following permissions:

• The bigquery.transfers.update permission to modify the transfer.

  The predefined roles/bigquery.admin IAM role includes this permission.

• Access to the service account. For more information about granting users the service account role, see Service Account User role.

Ensure that the service account you choose to run the transfer has the following permissions:

• The bigquery.datasets.get and bigquery.datasets.update permissions on the target dataset. If the table uses column-level access control, the service account must also have the bigquery.tables.setCategory permission.

  The bigquery.admin predefined IAM role includes all of these permissions. For more information about IAM roles in BigQuery Data Transfer Service, see Introduction to IAM.

• Access to the configured transfer data source. For more information about the required permissions for different data sources, see Data sources with service account support.

• For Google Ads transfers, the service account must be granted domain-wide authority. For more information, see Google Ads API Service Account guide.

Update data transfer credentials

bq update \
--transfer_config \
--update_credentials \
--service_account_name=abcdef-test-sa@abcdef-test.iam.gserviceaccount.com projects/862514376110/locations/us/transferConfigs/5dd12f26-0000-262f-bc38-089e0820fe38 \

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.FieldMaskUtil;
import java.io.IOException;

// Sample to update credentials in transfer config.
public class UpdateCredentials {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    String serviceAccount = "MY_SERVICE_ACCOUNT";
    TransferConfig transferConfig = TransferConfig.newBuilder().setName(configId).build();
    FieldMask updateMask = FieldMaskUtil.fromString("service_account_name");
    updateCredentials(transferConfig, serviceAccount, updateMask);
  }

  public static void updateCredentials(
      TransferConfig transferConfig, String serviceAccount, FieldMask updateMask)
      throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      UpdateTransferConfigRequest request =
          UpdateTransferConfigRequest.newBuilder()
              .setTransferConfig(transferConfig)
              .setUpdateMask(updateMask)
              .setServiceAccountName(serviceAccount)
              .build();
      dataTransferServiceClient.updateTransferConfig(request);
      System.out.println("Credentials updated successfully");
    } catch (ApiException ex) {
      System.out.print("Credentials was not updated." + ex.toString());
    }
  }
}

from google.cloud import bigquery_datatransfer
from google.protobuf import field_mask_pb2

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

service_account_name = "[email protected]"
transfer_config_name = "projects/1234/locations/us/transferConfigs/abcd"

transfer_config = bigquery_datatransfer.TransferConfig(name=transfer_config_name)

transfer_config = transfer_client.update_transfer_config(
    {
        "transfer_config": transfer_config,
        "update_mask": field_mask_pb2.FieldMask(paths=["service_account_name"]),
        "service_account_name": service_account_name,
    }
)

print("Updated config: '{}'".format(transfer_config.name))