BigQuery release notes

The following BigQuery ML features are now available:

• Creating remote models based on the Vertex AI gemini-1.5-flash and gemini-1.5-pro models.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

These features are now generally available (GA).

The following BigQuery ML features are now available:

• You can perform supervised tuning on a remote model based on a Vertex AI Gemini 1.5 flash or Gemini 1.5 pro model.
• You can evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained PaLM and Gemini models and tuned Gemini models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic or the Use tuning and evaluation to improve model performance tutorial.

These BigQuery ML features are generally available (GA).

BigQuery now offers the following Gemini-enhanced SQL translation features:

• In interactive translation mode, you can use Gemini-enhanced SQL translations to customize translated GoogleSQL queries. This feature is generally available (GA).

• You can generate AI suggestions for batch translations using the Gemini model. The suggestions are based on a Gemini-based configuration YAML file. This feature is in Preview.

• After running an interactive SQL translation, you can request a Gemini-generated text explanation that includes a summary of the translated SQL query. This feature is in Preview.

Dataplex automatic discovery lets you scan data in Cloud Storage buckets to extract and catalog metadata. Automatic discovery creates BigLake or external tables and object tables you can use for analytics and AI, and catalogs that data in Dataplex Catalog. This feature is available in public preview.

BigQuery provides context-aware transformation recommendations from Gemini for cleansing data for analysis. Data preparation is available in Preview.

You can now view, trigger, and pause Airflow DAGs in BigQuery. This feature is in Preview.

You can now manage notebook schedules on the Orchestration page. Notebook scheduling is in Preview.

Custom organization policies let you allow or deny specific operations on BigQuery Data Transfer Service transfer configurations to meet your organization's compliance and security requirements. This feature is generally available (GA).

You can now use fine-grained DML to optimize the execution of UPDATE, DELETE, and MERGE statements on tables. This feature is in Preview.

Use the BigQuery migration assessment for Oracle to assess the complexity of migrating data from your Oracle data warehouse to BigQuery. This feature is in preview.

BigQuery tables for Apache Iceberg bring the convenience of BigQuery storage optimization to Apache Iceberg tables that reside in your own cloud buckets. BigQuery tables for Apache Iceberg let you use BigQuery without moving data out of buckets that you control. This feature is now in preview.

You can now export and load Parquet files that include GeoParquet metadata. This feature is generally available (GA).

You can now use pipe syntax anywhere you write GoogleSQL. Pipe syntax supports a linear query structure designed to make your queries easier to read, write, and maintain. This feature is in Preview.

You can now create an external dataset in BigQuery that links to an existing database in Spanner. This feature is in preview.

ODBC driver update, release 3.0.7 1016

• [New] Connector authentication on Google Cloud VMs: The connector now supports authentication through Application Default Credentials using the Google internal metadata server, eliminating the need for a keyfile. This feature works only on Google Cloud Compute Engine VMs.
• [Resolved] The output for PrimaryKeys previously denoted the Key Sequence as a 0-indexed value. This has been corrected to a 1-indexed value, indicating the sequential order of the primary key's column within the primary key itself.

You can now enable, disable, and analyze history-based optimizations for queries. This feature is generally available (GA).

You can now use flexible column names with BigQuery tables and views for extracting, loading, streaming, and querying data. This feature is generally available (GA).

You can now use the operational health dashboard to get a single-pane view of key metrics such as slot usage, shuffle usage, errors, and total storage in real time. This feature is generally available (GA).

You can now create a materialized view replica directly from the Google Cloud console. This feature is generally available (GA).

Cloud console updates: You can now use keyboard shortcuts to control tab navigation in the details pane. This feature is generally available (GA).

You can now use Cloud KMS Autokey to automate the creation and use of customer-managed encryption keys (CMEKs), including the Cloud HSM service. This feature is generally available (GA).

BigQuery ML now offers the following expanded embedding support features:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text, image, and video into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try these capabilities with the following tutorials:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate video embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

These features are generally available (GA).

BigQuery ML now offers the following AI features:

• You can process documents from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Document AI API, including specifying a document processor to use.
  2. Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.

  Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.

• You can transcribe audio files from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  2. Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.

  Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

These BigQuery ML feature are generally available (GA).

You can now create workflows to execute code assets in sequence at a scheduled time. This feature is in Preview.

You can perform model monitoring in BigQuery ML. The following model monitoring functions are now generally available (GA):

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow tfdv.validate_statistics API.

You can now batch migrate classic saved queries to saved queries. This feature is in Preview for projects that have fewer than 2500 classic saved queries.

You can now use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. You can use a contribution analysis model with the ML.GET_INSIGHTS function to generate insights about changes to key metrics in your multi-dimensional data.

Try this feature with the Get data insights from a contribution analysis model tutorial.

This feature is in preview.

You can store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is Generally Available.

You can now use the partial ordering mode in BigQuery DataFrames to generate more efficient queries. This feature is in Preview.

You can now use Terraform to manage IAM tags on datasets and tables. This feature is generally available (GA).

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery, including Creative, Partner, and Advertiser tables. This feature is generally available (GA).

You can now use vector search and vector index features in BigQuery.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

You can try the vector search and vector index capabilities by using the Search embeddings with vector search tutorial.

The BigQuery vector search and vector index features are generally available (GA).

The BigQuery Data Transfer Service now supports incremental transfers when you migrate your data from your Teradata data warehouses to BigQuery. This feature is generally available (GA).

Delta Lake BigLake tables are now generally available (GA). Delta Lake is an open source, tabular data storage format that supports petabyte scale data tables.

You can now use the GROUP BY clause and the SELECT DISTINCT clause with the ARRAY and STRUCT data types. This feature is in Preview.

You can now query data in AlloyDB using a federated query. This feature is now generally available (GA).

The following Gemini in BigQuery features are now generally available (GA):

• Data insights
• Data canvas
• SQL and Python code assistance features:
  • Use the SQL generation tool
  • Prompt to generate SQL queries
  • Explain a SQL query
  • Generate Python code
• Partitioning and clustering recommendations

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

Phrase support for the SEARCH function is now generally available (GA).

You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

This feature is in preview.

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable (reverse ETL). This feature is generally available (GA).

Python code completion is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

You can now perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models. This feature lets you detect anomalies in historical time series data or in new data with multiple feature columns. You can try this feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial. This feature is generally available (GA).

You can now view your BigQuery insights and recommendations using the Recommendations page in the Google Cloud console. You can also view your BigQuery insights and recommendations using the following INFORMATION_SCHEMA views:

• INSIGHTS
• RECOMMENDATIONS
• RECOMMENDATIONS_BY_ORGANIZATION

These features are now in preview.

You can now get lower latency for small queries with the new short query optimized mode. BigQuery automatically determines which queries may be accelerated while other queries continue to run like before. This feature is now in preview.

You can now use time series and range functions to support time series analysis. This feature is now generally available (GA).

The JSON_KEYS function, which extracts unique JSON keys from a JSON expression, is in Preview.

Some JSON functions that take a JSONPath let you specify a mode that allows flexibility in how the JSONPath matches the JSON data structure. This feature is in Preview.

You can now create a materialized view over Apache Iceberg table that is partition aligned with the base table. The materialized view only supports time-based partition transformation, for example, YEAR, MONTH, DAY, and HOUR. This feature is in preview.

When you translate SQL queries from your source database, you can use configuration YAML files to optimize and improve the performance of your translated SQL. This feature is in preview.

Workload management now provides the following benefits:

• The autoscaler now scales up immediately.
• The autoscaler now scales more precisely.
• The autoscaler scales to the nearest multiple of 50 slots, instead of 100.
• You can now purchase capacity commitments, set baseline slots, and set autoscale max slots in incremental steps of 50 slots.
• If one minute or more has passed since the most recent increase in capacity, you can now reduce capacity without resetting the one minute minimum. This allows for multiple consecutive decreases without a one minute delay between them.

These features are now generally available (GA).

You can now use the output_dimensionality argument of the ML.GENERATE_EMBEDDING function when you use the function with a remote model based on a Vertex AI multimodalembedding model. The output_dimensionality argument lets you specify the number of dimensions to use when generating embeddings. This feature is in Preview.

The RANGE data type is now a supported JSON encoding. This feature is Generally Available (GA).

You can now use the administrative jobs explorer to help you quickly monitor jobs activity across your organization. This feature is generally available (GA).

Vector indexes support the TreeAH index type, which uses Google's ScaNN algorithm. The TreeAH index is optimized for batch queries that process hundreds or more query vectors. This feature is in Preview.

You can now use table explorer to examine table data and create data exploration queries. This feature is in preview.

IAM deny policies now support additional permissions, including bigquery.tables.getData which can deny permission to read tables. Consider special cases when you create deny policies for bigquery.tables.getData and other BigQuery permissions. This feature is in preview.

You can now configure SAP Datasphere connections with network attachments to help secure connections. SAP Datasphere connections are in preview.

Manifest files are now supported for Amazon S3 and Azure Blob Storage. This feature is generally available (GA).

The CHANGES change history function is now in preview. This table-valued function provides a history of table changes over a window of time and captures the following operations:

• CREATE TABLE DDL statement
• INSERT DML statement
• Data appended or changed as part of a MERGE DML statement
• UPDATE DML statement
• DELETE DML statement
• Loading data into BigQuery
• Streaming ingestion
• TRUNCATE TABLE DML statement
• Jobs configured with a writeDisposition of WRITE_TRUNCATE
• Individual table partition deletions

You can use data manipulation language (DML) to modify rows that have been recently written to a BigQuery table by the Storage Write API. This is now generally available (GA).

The BigQuery continuous queries feature is now in preview.

Continuous queries let you build long-lived, continuously processing SQL statements that can analyze, process, and perform machine learning (ML) inference on incoming data in BigQuery in real time. You can configure continuous queries to replicate query results to a Pub/Sub topic, Bigtable instance, or another BigQuery table, a process also known as Reverse ETL.

You can use continuous queries to perform the following tasks, using the accessible language of SQL:

• Transform incoming data and act immediately on insights.
• Use Vertex AI to apply real time ML insights.
• Build automated event-driven data pipelines.
• Replicate real-time events to downstream operational systems like Bigtable.

To try BigQuery continuous queries, see Create continuous queries.

You can now use BigQuery Omni Virtual Private Cloud (VPC) allowlists to restrict access to AWS S3 buckets and Azure Blob Storage from specific BigQuery Omni VPCs. This feature is in preview.

The following BigQuery migration assessment features are now generally available (GA):

• When you run a migration assessment, the migration assessment now automatically creates a BigQuery dataset to store the assessment results. You can also choose to store assessment results in an existing empty dataset or manually create a dataset with a custom name.
• While a migration assessment is running, you can view the assessment report with partial data. You can also view its progress and estimated completion time in the status icon tooltip.
• You can view more information and errors about a migration assessment in the assessment details page.

You can now configure the default storage billing model for new datasets. This feature is generally available (GA).

When you run a migration assessment for Amazon Redshift, Teradata, or Snowflake, the service also creates a dataset containing only highly aggregated assessment results. This aggregated dataset doesn't contain any query logs; therefore, no personally identifiable information (PII) or business-sensitive information is visible. You can share this dataset with users that are not in your project. This feature is in preview.

You can now use EXPORT DATA statements to reverse ETL BigQuery data to Spanner. This feature is in preview.

Cloud console updates: You can now drag a tab in the details pane to open a new column and compare tabs. You can also drag the tab to a new position in the current or an adjacent column. This feature is in preview.

The following Analytics Hub features are now generally available:

• Making exchanges and listings publicly discoverable.
• Highlighting listings in the Featured section of the Analytics Hub catalog.
• Generating unauthenticated URLs for public listings.

Data publishers can now share Pub/Sub topics and manage subscriptions in Analytics Hub. This feature is in preview.

You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. This feature is generally available (GA). You can also attach tags to BigQuery datasets during dataset creation to conditionally grant or deny access with IAM policies.

You can now use the BigQuery JupyterLab plugin to explore your data, use BigQuery DataFrames in a Jupyter notebook, and deploy a BigQuery DataFrames notebook to Cloud Composer. This feature is in preview.

The BigQuery migration assessment for Amazon Redshift is now generally available (GA). You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.

Additional collation support for the NULLIF conditional expression has been added. The NULLIF conditional expression is now affected by collation and can be used in collation-supported comparisons with the STRUCT data type. This feature is generally available (GA).

You can now perform supervised tuning on a BigQuery ML remote model based on a gemini-1.0-pro-002 model. This feature is in preview. To try this feature, see Tune a model using your data.

You can also perform supervised tuning by using the BigQuery DataFrames Python API. Use the fit() and score() methods in the bigframes.ml.llm.GeminiTextGenerator model class to perform supervised tuning.

You can now schedule notebooks. This feature is available in preview.

Analytics Hub data egress controls are now generally available (GA). Publishers can now enforce egress restrictions on Analytics Hub listings to prevent subscribers from copying or exporting the shared data.

The slot recommender for editions analyzes historical usage data to recommend optimal capacity purchasing for edition and on-demand workloads. This feature is generally available (GA).

You can now use IAM conditions to control access to BigQuery resources. This feature is generally available (GA).

You can now define a _CHANGE_SEQUENCE_NUMBER for BigQuery change data capture (CDC) to manage streaming UPSERT ordering for BigQuery. This feature is in preview.

The following Generative AI features are now in preview:

• Creating remote models based on the Vertex AI gemini-1.5-flash foundation model.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

BigQuery ML now offers the following Generative AI features:

• Grounding and safety attributes when you use Vertex AI Gemini models with the ML.GENERATE_TEXT function:

  • Use the ground_with_google_search argument to perform grounding. Grounding lets the Gemini model use additional information from the internet when generating a response, in order to make model responses more specific and factual.
  • Use the safety_settings argument to configure safety attributes.The Gemini model filters the responses it returns based on the attributes you specify.

• Video embedding (Preview). You can use the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding model to create multimodal embeddings that include video embeddings.

  To try the new video embedding functionality, see Generate video embeddings by using the ML.GENERATE_EMBEDDING function.

The interactive SQL translator, the translation API, and the batch SQL translator features let you translate the following SQL dialects into GoogleSQL:

• IBM DB2 SQL
• Greenplum SQL
• SQLite

These features are in preview.

You can now query data in AlloyDB using a federated query. This feature is in preview.

The following Generative AI features are now in preview:

• Creating remote models based on the Vertex AI gemini-1.5-pro foundation model.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

You can now use a search index to optimize lookups on the INT64 and TIMESTAMP data types. The feature is in preview.

You can use DLP functions to support encryption and decryption between BigQuery and Sensitive Data Protection, using AES-SIV. This feature is now generally available (GA).

You can now create Gemini-enhanced translation rules to use with the interactive SQL translator. Translation rules let you customize and adjust the results of the interactive translator according to your SQL migration needs. This feature is in preview.

Phrase support for the SEARCH function is in preview.

JavaScript user-defined aggregate functions (UDAFs) are in preview. You can create a JavaScript UDAF with the CREATE AGGREGATE FUNCTION statement.

You can now store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is in preview.

BigQuery Managed Disaster Recovery provides managed failover and redundant compute capacity for business critical workloads. It is intended for use in the case of a total region outage and is supported with the BigQuery Enterprise Plus edition only. This feature is now available in preview.

You can now create AWS Glue federated datasets using the the Google Cloud console. This feature is generally available (GA).

Analytics Hub Subscription Management is generally available (GA). Data Publishers can now manage their subscriptions, view information about their subscribers, and revoke access to their data at any time.

Analytics Hub Provider Usage Metrics is now generally available (GA). The usage metrics include the following:

• Jobs that run against your shared data.
• The consumption details of your shared data by subscribers' projects and organizations.
• The number of rows and bytes processed by the job.

AWS Glue federated datasets are now generally available (GA).

An AWS Glue federated dataset is a connection at the dataset level between BigQuery and an existing database in AWS Glue.

You can now specify translation configurations in the BigQuery interactive SQL translator and use it to debug batch SQL translator jobs. This feature is generally available (GA).

The following BigQuery ML data preprocessing features are now generally available (GA):

• The ML.TRANSFORM function, which you can use to preprocess feature data. This function processes input data by applying the data transformations captured in the TRANSFORM clause of an existing model.
• Transform-only models, which you can use to apply preprocessing functions to input data and return the preprocessed data. Transform-only models decouple data preprocessing from model training, making it easier for you to capture and reuse a set of data preprocessing rules.

You can now reference Iceberg tables in materialized views instead of migrating that data to BigQuery-managed storage. This feature is in preview.

You can now let users that are in Microsoft Entra groups access BigQuery data in Power BI by using Workforce Identity Federation. This feature is generally available.

SQL code generation is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

The BigQuery Data Transfer Service for Google Merchant Center supports the Product Targeting report.

User-defined aggregate functions (UDAFs) that support SQL expressions are in preview. You can create a UDAF with the CREATE AGGREGATE FUNCTION statement.

The quantified LIKE operator is generally available (GA). With this operator, you can check a search value for matches against a list of patterns or an array of patterns, using one of these conditions:

• LIKE ANY: Checks if at least one pattern matches.
• LIKE SOME: Synonym for LIKE ANY.
• LIKE ALL: Checks if every pattern matches.

More permissions are now supported by deny policies. This feature is in preview.

BigQuery now supports subqueries in row level access policies. This feature is now in public preview.

BigQuery ML now offers the following expanded embedding support features in preview:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text and images into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try the new multimodal embedding functionality:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

You can now create a data canvas in BigQuery Studio. A data canvas lets you discover, transform, query, and visualize data using natural language. It provides a graphic interface for your analysis that lets you work with data sources, queries, and visualizations in a directed acyclic graph (DAG), giving you a graphical view of your analysis workflow that maps to your mental model. You can iterate on query results and work with multiple branches of inquiry in a single place. This feature is in preview and access can be requested here.

The following Gemini in BigQuery features are now available in Public Preview:

• Data insights: an automated and intuitive way to explore and understand your data.
• Data canvas: a graphic interface that lets you discover, transform, query, and visualize data using natural language.
• SQL and Python code assistance: Gemini-assisted code generation, completion, and explanation.
• Materialized views, partitioning, and clustering recommendations: recommendations to reduce cost and improve performance.
• Autotune and troubleshoot serverless Spark: optimize and explain Spark workloads.

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

BigQuery Studio is generally available (GA).

BigQuery Studio lets you save, share, and manage versions of code assets such as notebooks and saved queries.

BigQuery DataFrames is generally available (GA).

BigQuery DataFrames is a set of open source Python libraries that implements the pandas and scikit-learn APIs with server-side processing. To get started, you can try BigQuery DataFrames.

The BigQuery materialized view recommender analyzes your past query jobs to identify opportunities to apply materialized views to your queries for potential cost savings. You can view all available materialized view recommendations through the BigQuery UI or Recommender API. This feature is in preview.

Differential privacy is now generally available (GA).

You can now use BigLake to access Delta Lake tables. For more information, see Create Delta Lake BigLake tables. This feature is available in preview.

The allow_non_incremental_definition option and max_staleness option for materialized views are now generally available (GA). The allow_non_incremental_definition option supports an expanded range of SQL queries to create materialized views, and the max_staleness option provides consistently high performance with controlled costs when processing large, frequently changing datasets.

You can now perform model monitoring in BigQuery ML. The following model monitoring functions are now in preview:

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow validate_statistics API.

BigQuery data clean rooms with analysis rules and enhanced usage metrics are now generally available (GA). Data clean rooms provide a security-enhanced and privacy-preserving environment for multiple parties to share and augment data without moving or revealing the underlying data.

Join restrictions, list overlap, differential privacy with privacy budgeting, and aggregation thresholding are now enforceable in BigQuery data clean rooms using analysis rules.

Collation now supports the following generally available (GA) features:

• The underscore in the LIKE operator.
• Comparison support for the STRUCT data type with the following operators and conditional expressions: =, !=, IN and CASE.

You can now configure materialized views with tables enabled for change data capture (CDC) streaming update and delete operations.

The following BigQuery ML features are now in preview:

• Performing supervised tuning on a remote model based on a Vertex AI text-bison large language model (LLM).
• Evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained text-bison, text unicorn, or gemini-pro models and tuned text-bison models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic.

You can now enable, disable, and analyze history-based optimizations for queries. This feature is in preview.

The BigQuery Data Transfer Service for Search Ads 360 now supports the new Search Ads 360 Reporting API. You can use the Search Ads 360 connector to specify custom Floodlight variables and custom columns when transferring Search Ads 360 data to BigQuery. This feature is now generally available (GA).

You can now query data in SAP Datasphere using a federated query. This feature is in preview.

Query optimization using search indexes is now applied to comparisons of string literals and indexed data, including the equal (=), IN, and LIKE operators and the STARTS_WITH function. This feature is generally available (GA).

The Help me code tool lets you use natural language to generate a SQL query that can then be run in BigQuery. This feature is now in preview.

The following Generative AI features are now in preview:

• Creating a remote model based on a Vertex AI gemini-pro-vision large vision model (VLM).
• Using the ML.GENERATE_TEXT function with this remote model to perform Vision Generative AI tasks, such as image or video captioning and visual Q&A, for visual content stored in BigQuery object tables.

Try these features with the Generate text that describes visual content how-to topic.

You can now add Salesforce Data Cloud data to BigQuery. This feature is generally available (GA).

Incremental materialized views now support LEFT OUTER JOIN and UNION ALL. This feature is in preview.

The maximum notebook size has been increased from 10 MB to 20 MB. Notebooks are available in preview.

You can now view lists of all saved queries and all notebooks in your project. These features are available in preview.

You can now create and run Spark stored procedures that are written in Python, Java, and Scala. You can also use the PySpark editor in BigQuery to create stored Python procedures for Apache Spark. This feature is now generally available (GA).

The minimum duration between scheduled queries has been reduced from 15 minutes to 5 minutes. This feature is generally available.

You can now undelete a dataset that is within your time travel window to recover it to the state that it was in when it was deleted. This feature is in preview.

These BigQuery features are now generally available (GA):

Text analysis configuration options for the following:

• CREATE SEARCH INDEX DDL
• Existing LOG_ANALYZER and new PATTERN_ANALYZER analyzers, which are used in various functions, including SEARCH
• The TEXT_ANALYZE function

The following advanced processing functions:

• ML.BAG_OF_WORDS
• ML.TF_IDF
• BAG_OF_WORDS
• TF_IDF
• COSINE_DISTANCE
• EUCLIDEAN_DISTANCE
• EDIT_DISTANCE

You can now perform hierarchical forecasts in BigQuery ML time series models, which let you aggregate and roll up values for all time series in the model. This feature is generally available (GA).

The INFORMATION_SCHEMA.WRITE_API_TIMELINE* views, containing per minute aggregated BigQuery Storage Write API ingestion statistics, are now generally available (GA).

Duet AI in BigQuery can now assist with Python code generation and code completion. This feature is in preview.

You can now selectively refresh the metadata cache for BigLake tables by using the BQ.REFRESH_EXTERNAL_METADATA_CACHE system procedure. This feature is generally available (GA).

The following BigQuery cross-cloud features are now generally available (GA):

• You can take advantage of the benefits of materialized views over Amazon S3 metadata cache-enabled BigLake tables.
• You can create materialized view replicas of materialized views over Amazon S3 metadata cache-enabled Biglake tables. Materialized view replicas let you use the materialized view data in queries while avoiding data egress costs and improving query performance.
• You can get information about materialized view replicas by using SQL, the bq command-line tool, or the BigQuery API.
• You can use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions.

The SQL translation API combines the interactive and batch translator into a single workflow, improving the efficiency and stability of your translation jobs created using the API. This feature is available in preview.

Materialized views can now reference logical views. This feature is in preview.

The ability to perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models is now in preview. This feature enables you to detect anomalies in historical time series data or in new data with multiple feature columns. Try this new feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial.

The following statements are now generally available (GA) with billing enabled:

• CREATE TABLE AS SELECT
• CREATE TABLE IF NOT EXISTS AS SELECT
• CREATE OR REPLACE TABLE AS SELECT
• INSERT INTO SELECT

These statements let you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables.

You can now use time series and range functions to support time series analysis. This feature is in preview.

You can now use data manipulation language (DML) statements to efficiently delete entire partitions. If a DELETE statement targets all rows in a partition, then the entire partition is deleted without scanning bytes or consuming slots. This feature is now generally available (GA).

The BigQuery Data Transfer Service can now transfer data from the following data sources:

• Facebook Ads
• Oracle
• Salesforce
• Salesforce Marketing Cloud
• ServiceNow

Transfers from these data sources are supported in preview.

The following SQL features are now generally available (GA):

• GROUP BY GROUPING SETS clause: Produces aggregated data for one or more grouping sets.
• GROUP BY CUBE clause: Produces aggregated data for all grouping set permutations.
• GROUPING function: Checks if a groupable value in the GROUP BY clause is aggregated.

The GROUP BY ALL clause, which groups rows by inferring grouping keys from the SELECT items, is now in preview.

The following BigQuery text embedding features are now generally available (GA):

• Creating a BigQuery ML remote model that references a Vertex AI textembedding-gecko* text embedding model.
• Using the ML.GENERATE_EMBEDDING function with the remote model to embed text stored in BigQuery.
• Generating text embeddings with the NNLM, SWIVEL, and BERT TensorFlow models.

The following Generative AI features are now generally available (GA):

• Creating a remote model based on the gemini-pro Vertex AI large language model (LLM).
• Using the ML.GENERATE_TEXT function with a remote model based upon gemini-pro to perform generative natural language tasks on text stored in BigQuery tables.
• Use the BigQuery DataFrames GeminiTextGenerator class in the bigframes.ml.llm module to create estimator-like Gemini text generator models.

After you run a query in the query editor, in the Chart tab, you can now see a visualization of your query results. This feature is generally available (GA).

Custom data masking is now generally available (GA). You can define custom masking routines for custom masking capabilities such as salt based hash. The feature is available on the Enterprise Plus edition.

BigQuery now offers entity resolution. This feature lets users match records across datasets even when a common identifier is missing. It utilizes an identity provider for this process; BigQuery supports LiveRamp and provides a framework for other identity providers to offer similar services. This feature is generally available (GA).

You can now view query plans to see details of SQL pushdowns in federated queries. This feature is now generally available.

The following information schema views display the history of configuration changes to the options of your organization and projects:

• ORGANIZATION_OPTIONS_CHANGES view displays the configuration changes to an organization, including all organization and project-level changes.
• PROJECT_OPTIONS_CHANGES view displays the configuration changes to a project.

This feature is now in preview.

BigQuery now supports vector search and vector indexes. These features are in preview.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

Try the new vector search and vector index capabilities with the Search embeddings with vector search tutorial.

You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. This feature is in preview.

Cloud console updates: You can now sort query results by column. Click arrow_drop_down Open sort menu next to the column name and select a sort order. This feature is generally available (GA).

BigQuery now natively supports the Delta Lake format for Amazon S3 and Azure tables. This feature is now in preview.

BigQuery ML has added a new residual column to the output of the ML.EXPLAIN_FORECAST function for ARIMA_PLUS and ARIMA_PLUS_XREG models. The residual column contains the difference between the actual time series and the fitted time series for the historical data. This lets you compare the modeled historical data that is returned in the other output columns of ML.EXPLAIN_FORECAST with the actual historical data.

BigQuery now supports the ST_LINEINTERPOLATEPOINT geography function, which gets a point at a specific fraction in a linestring. This feature is generally available (GA).

You can now use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions. This feature is in preview.

Analytics Hub listings can now include data encrypted with customer-managed encryption keys (CMEK). This feature is generally available (GA).

Operational Health administrative resource charts are now in preview. You can use charts to view slot and shuffle usage, job concurrency, errors, and other metrics.

The BigQuery Data Transfer Service now supports federated workforce identities when creating a data transfer from most data sources. This feature is generally available (GA).

The Apache Hive connector is now generally available (GA) for data analytics pipeline migration.

The following BigQuery cross-cloud features are now in preview:

• You can now take advantage of the benefits of materialized views over Amazon S3 metadata cache-enabled BigLake tables.
• You can create materialized view replicas of materialized views over Amazon S3 metadata cache-enabled BigLake tables. Materialized view replicas let you use the materialized view data in queries while avoiding data egress costs and improving query performance.

The following BigQuery ML data preprocessing features are now in preview:

• The ML.TRANSFORM function, which you can use to preprocess feature data. This function processes input data by applying the data transformations captured in the TRANSFORM clause of an existing model.
• Transform-only models which you can use to apply preprocessing functions to input data and return the preprocessed data. Transform-only models decouple data preprocessing from model training, making it easier for you to capture and reuse a set of data preprocessing rules.

You can use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now generally available (GA).

The slot estimator now supports project level cost-optimal commitment and autoscale recommendations for on-demand workloads. This feature is now in preview.

The following BigQuery ML features for Vertex AI large language models (LLMs) are now generally available (GA):

• The SQL syntax for remote models has been updated to provide access to all text generation and text embedding LLMs (for example, text-bison-32k and textembedding-gecko-multilingual) and also to provide support for different LLM versions.

• Region support for text-bison* LLM models has been expanded to include the following locations in addition to us and us-central1:

  • asia-northeast3
  • asia-southeast1
  • eu
  • europe-west1
  • europe-west2
  • europe-west3
  • europe-west4
  • europe-west9
  • us-west4

You can now see query performance insights about partition skew. This feature is in preview.

The following BigQuery ML point-in-time lookup functions are now generally available (GA). These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

• Use the ML.FEATURES_AT_TIME function to use the same point-in-time cutoff for all entities when retrieving features.
• Use the ML.ENTITY_FEATURES_AT_TIME function to retrieve features from multiple points in time for multiple entities.

The following AI features in BigQuery are now in preview:

• The ability to process documents from BigQuery object tables by doing the following:

  • Creating a remote model based on the Document AI API, including specifying a document processor to use.
  • Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.
    
    Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.
    
    

• The ability to transcribe audio files from BigQuery object tables by doing the following:

  • Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  • Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.
    
    Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

The batch SQL translator has added enhancements when viewing SQL translation reports. You can now see a log summary of all issues during a translation job, as well as a code tab that displays a side-by-side comparison of your input and output files from a translation. This feature is in preview.

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery. This feature is in preview.

The following BigQuery ML features for time series forecasting are now generally available (GA):

• Ensure forecasted values fall within specified limits. The FORECAST_LIMIT_LOWER_BOUND and FORECAST_LIMIT_UPPER_BOUND options of the CREATE MODEL statement let you set the lower and upper bounds of the forecasted values returned by the model.

  Try this feature with the Limit forecasted values for a time series model tutorial.

• Custom holiday modeling:

  • CREATE MODEL syntax lets you specify custom holiday modeling for time series models.
  • The ML.HOLIDAY_INFO function returns the list of holidays being modeled by an ARIMA_PLUS or ARIMA_PLUS_XREG time series forecasting model.
  • The updated ML.EXPLAIN_FORECAST function includes an explanation of the holiday effect for each holiday included in the model.

  Try this feature with the Use custom holidays in a time-series forecasting model tutorial.

BigQuery now supports text analysis configuration options for the following:

• CREATE SEARCH INDEX DDL
• Existing LOG_ANALYZER and new PATTERN_ANALYZER analyzers, which are used in various functions, including SEARCH
• New TEXT_ANALYZE function

BigQuery now also provides support for the following advanced processing functions:

• ML.BAG_OF_WORDS
• ML.TF_IDF
• BAG_OF_WORDS
• TF_IDF
• COSINE_DISTANCE
• EUCLIDEAN_DISTANCE
• EDIT_DISTANCE

These features are now in preview.

The following INFORMATION_SCHEMA views that show metadata for table storage usage are now in preview:

• Use the TABLE_STORAGE_USAGE_TIMELINE view to get total billable bytes per table per day at the project level.
• Use the TABLE_STORAGE_USAGE_TIMELINE_BY_ORGANIZATION view to get total billable bytes per table per day at the organization level.

You can now use cached results from the same query issued by other users in the same project when you use Enterprise or Enterprise Plus edition. This feature is generally available (GA).

BigQuery support for change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API is now generally available (GA).

You can now use data manipulation language (DML) to modify rows that have been recently written by the Storage Write API. This feature is in preview.

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now generally available (GA).

The administrative resource charts now supports the following features in preview:

• View your resource utilization chart at the project level.
• Filter your resource utilization data based on different billing models.

Custom data masking now supports an expanded list of functions, including SHA hash functions with salt. This feature is in preview.

Stored procedures for Apache Spark are now available without enrollment. This feature is in preview.

The BigQuery migration assessment is now available for Apache Hive in preview. You can use this feature to assess the complexity of migrating data from your Apache Hive data warehouse to BigQuery.

You can now use DLP functions to support encryption and decryption between BigQuery and DLP, using AES-SIV. This feature is in preview.

The following geography functions are now generally available (GA):

• ST_LINESUBSTRING: Gets a segment of a single linestring at a specific starting and ending fraction.
• ST_HAUSDORFFDISTANCE: Gets the discrete Hausdorff distance between two geometries.

Queries now support additional ways to work with grouping sets, which include:

• GROUP BY GROUPING SETS clause (new): Produce aggregated data for one or more grouping sets.
• GROUP BY CUBE clause (new): Produce aggregated data for all grouping set permutations.
• GROUP BY ROLLUP clause (update): You can now include groupable items sets in this clause.
• GROUPING function (new): Check if a groupable value in the GROUP BY clause is aggregated.

This feature is in preview.

Adding descriptions to the columns of a view is now generally available (GA). Use the CREATE VIEW or ALTER COLUMN DDL statements to add descriptions.

The BigQuery migration assessment is now available for Snowflake in preview. You can use this feature to assess the complexity of migrating data from your Snowflake data warehouse to BigQuery.

You can now copy tables across regions. This feature is now in preview.

The following Google Cloud Blockchain Analytics datasets are now available in Preview and available through the Public Datasets Program and Analytics Hub:

• Google Cloud's Tron Mainnet data

• Google Cloud's Optimism Mainnet data

• Google Cloud's Avalanche Contract Chain data

• Google Cloud's Fantom Opera data

• Google Cloud's Ethereum Mainnet data

• Google Cloud's Arbitrum One Chain data

• Google Cloud's Cronos Mainnet Chain data

BigQuery native integration in Looker Studio enables monitoring features for Looker Studio queries, improves query performance, and supports many BigQuery features. This feature is in preview.

As a BigQuery administrator, to monitor your organization's slots utilization and BigQuery jobs' performance over time, use can now use administrative query inspector. This feature is now generally available.

The following BigQuery ML point-in-time lookup functions are now in preview. These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

• Use the ML.FEATURES_AT_TIME function to use the same point-in-time cutoff for all entities when retrieving features.
• Use the ML.ENTITY_FEATURES_AT_TIME function to retrieve features from multiple points in time for multiple entities.

You can now use IAM conditions to control access to BigQuery resources. This feature is in preview.

Materialized views over BigLake metadata cache-enabled tables can reference structured data stored in Cloud Storage. These materialized views function like materialized views over BigQuery-managed storage tables, including the benefits of automatic refresh and smart tuning. This feature is now generally available (GA).

Authorized stored procedures are now generally available (GA). This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

The BigQuery Data Transfer Service now supports transfers from Search Ads 360 using the new Search Ads 360 reporting API. This feature is in preview. Customers with existing Search Ads 360 transfers should migrate their workflows to be compatible with the new Search Ads 360. The BigQuery Data Transfer Service will stop its support for the old Search Ads 360 reporting API on May 31st, 2024.

You can now create a federated dataset in BigQuery that federates to an existing database in AWS Glue. This feature is in preview.

The array subscript operator now returns a value in an array directly by index. Previously, only offset and ordinal were available. This feature is generally available (GA).

The struct subscript operator has been added. With this operator, you can access a STRUCT field by index, offset, or ordinal. This feature is generally available (GA).

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable. This feature is in preview.

Data clean rooms is now in preview. Data clean rooms provide a secure environment in which multiple parties can share, join, and analyze their data assets without moving or revealing the underlying data. To learn more, see the following topics:

• Use data clean rooms
• Aggregation threshold for queries and views
• Aggregation threshold clause

Duet AI in BigQuery, an AI-powered collaborator in Google Cloud, can help you complete, generate, and explain SQL queries. This feature is in preview.

BigQuery Studio is now in preview. BigQuery Studio offers features to make it easier for you to discover, explore, analyze, and run inference on data in BigQuery, including:

• Python notebooks, powered by Colab Enterprise. Notebooks provide one-click Python development runtimes, and built-in support for BigQuery DataFrames.
• Asset management and version history for notebooks and saved queries, powered by Dataform.

BigQuery DataFrames is now in preview. BigQuery DataFrames is a Python API that you can use to analyze data and perform machine learning tasks in BigQuery. BigQuery DataFrames consists of the following parts:

• bigframes.pandas implements a DataFrame API (with partial Pandas compatibility) on top of BigQuery.
• bigframes.ml implements a Python API for BigQuery ML (with partial scikit-learn compatibility).

Get started with BigQuery DataFrames by using the BigQuery DataFrames quickstart.

The following Generative AI features are now generally available (GA) in BigQuery ML:

• Creating a remote model based on the Vertex AI large language model (LLM) text-bison.
• Using the ML.GENERATE_TEXT function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

The following text embedding features are now available in preview:

• Creating a BigQuery ML remote model that references the Vertex AI PaLM APIs for embeddings (textembedding-gecko).
• Using the ML.GENERATE_TEXT_EMBEDDING function with the remote embedding model to embed text stored in BigQuery.
• Using the ARRAY<NUMERIC> type as an input feature type to other models.
• Generating text embeddings with the NNLM, SWIVEL, and BERT TensorFlow models.

For more information, see the tutorial for performing basic semantic search with text embeddings.

The following BigQuery ML inference features are now generally available (GA):

• Importing ONNX, XGBoost, and TensorFlow Lite models so that you can run them within the BigQuery ML inference engine.
• Hosting models remotely on Vertex AI Prediction so you can do inference with BigQuery ML, removing the need to build data pipelines manually.
• Using BigQuery ML functions to perform inference on Vertex AI pretrained models so that you can accomplish natural language processing, translation, and computer vision tasks in BigQuery. These functions work with the Cloud Vision, Cloud Natural Language, and Cloud Translation APIs.

BigQuery now allows you to create your own masking routines for your data. You can use the REGEX_REPLACE scalar function to create custom masking rules to obfuscate your sensitive data. This feature is currently in preview.

You can now scan tables to create data profiles and monitor data quality. These features help you better understand your data and ensure it is accurate and reliable. These features are generally available (GA).

Analytics Hub now lets you manage subscriptions. The subscription resource stores relevant information about the subscriber and represents the connection between the shared resource and linked dataset. This feature is now in preview.

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now in preview.

You can now see query performance insights about high cardinality joins. This feature is generally available (GA).

You can now use user-defined functions to export BigQuery data as Protocol Buffer (Protobuf) columns. This feature is generally-available.

The following features are now generally available (GA) in queries and materialized views:

• HAVING MAX and HAVING MIN clauses for the ANY_VALUE function.

• MAX_BY function, which is a synonym for ANY_VALUE(x HAVING MAX y).

• MIN_BY function, which is a synonym for ANY_VALUE(x HAVING MIN y).

The quantitive LIKE operator is now in preview. With this operator, you can check a search value for matches against several patterns, using one of these conditions:

• LIKE ANY: Checks if at least one pattern matches.
• LIKE SOME: Synonym for LIKE ANY.
• LIKE ALL: Checks if every pattern matches.

The following JSON functions are now generally available (GA).

• JSON_ARRAY: Creates a JSON array.
• JSON_ARRAY_APPEND: Appends JSON data to the end of a JSON array.
• JSON_ARRAY_INSERT: Inserts JSON data into a JSON array.
• JSON_OBJECT: Creates a JSON object.
• JSON_REMOVE: Produces JSON with the specified JSON data removed.
• JSON_SET: Inserts or replaces JSON data.
• JSON_STRIP_NULLS: Removes JSON nulls.
• LAX_BOOL: Attempts to convert a JSON value to a SQL BOOL value.
• LAX_FLOAT64: Attempts to convert a JSON value to a SQL FLOAT64 value.
• LAX_INT64: Attempts to convert a JSON value to a SQL INT64 value.
• LAX_STRING: Attempts to convert a JSON value to a SQL STRING value.

BigQuery now supports the ability to deny access to principals via deny policies for the following IAM permissions :

• Managing reservations and capacity commitments: bigquery.googleapis.com/capacityCommitments.*, bigquery.googleapis.com/bireservations.*, bigquery.googleapis.com/reservationAssignments.*, bigquery.googleapis.com/reservations.*
• Resource Deletion: bigquery.googleapis.com/[datasets, tables, models, routines, jobs, connections].delete
• Dataset tag bindings: bigquery.googleapis.com/datasets.[createTagBinding, listTagBinding]
• Row Access Policies: bigquery.rowAccessPolicies.[create, delete, update, setIamPolicy]

Analytics Hub now supports the use of routines in linked datasets. This feature is now in preview.

BigQuery now supports using manifest files for external tables. This feature is now generally available (GA).

With Analytics Hub, you can now track usage metrics of your shared datasets. This feature is generally available (GA). The usage metrics include the following:

• Jobs that run against your shared dataset.
• The consumption details of your shared dataset by subscribers' projects and organizations.
• The number of rows and bytes processed by the job.

Cloud console updates: The following features are now available in preview:

• On the Welcome page, in the Recently accessed section, you can view your 10 most recently accessed resources.
• After you run a query in the query editor, in the Chart tab, you can see the visualization of your query results.

You can now GRANT or REVOKE access to materialized views with a SQL statement. This feature is generally available (GA).

BigQuery Storage Write API multiplexing is now generally available (GA). You can use multiplexing in the default stream to write to multiple destination tables with shared connections.

Query queues are now generally available (GA). With query queues, BigQuery automatically determines your query concurrency based on available slots rather than a fixed limit. Once the maximum concurrency is reached, additional queries are queued until processing resources are available. Query queues are enabled by default and have been rolled out over the last several weeks; no user action is required and you shouldn't see any degradation in your query performance. You can optionally set the maximum concurrency target for a reservation. You can also control the interactive and batch query queue timeout by using default configurations.

Multivariate time series forecasting with the ARIMA_PLUS_XREG model in BigQuery ML is now generally available (GA). This feature lets you perform time series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG information in the end-to-end user journey topic, and try the multivariate time-series forecasting from Seattle air quality data tutorial.

BigQuery ML has introduced new Explainable AI capabilities for better model explainability:

• You can now use the ML.EXPLAIN_FORECAST function with ARIMA_PLUS_XREG models.
• You can use the updated ML.EXPLAIN_FORECAST function to get explanations of the holiday effect for holidays in time series forecasting models (both ARIMA_PLUS and ARIMA_PLUS_XREG).
• You can now use the ML.GLOBAL_EXPLAIN function with AutoML Tables models for global model explainability.
• For Boosted Tree and Random Forest models, you can now use the approx_global_feature_contrib training option to use fast approximation for global feature contribution computation in model training, and the approx_feature_contrib option in the ML.EXPLAIN_PREDICT function to use the fast approximation for local feature contribution computation in model inference.

Now you can also use Vertex Explainable AI on BigQuery ML models that you've registered to the Vertex AI Model Registry. To learn more, see Explainable AI for BigQuery ML models.

BigQuery can now use search indexes to optimize some queries that contain the equal operator (=), IN operator, LIKE operator, or STARTS_WITH function to compare string literals with indexed data. This feature is in preview.

Primary and foreign key table constraints are now generally available. In addition to the features available in preview, you can now also manage constraints through the BigQuery API and view constraints in the BigQuery console.

The following BigQuery ML feature preprocessing functionality is now generally available (GA).

You can export models that use the TRANSFORM clause for feature preprocessing to the TensorFlow SavedModel format. There are 13 data types supported for TRANSFORM clause input, and 127 SQL functions supported for use within the TRANSFORM clause.

You can also now deploy a model trained with the TRANSFORM clause to Vertex AI and locally.

Use the following functions to perform feature preprocessing:

• ML.IMPUTER
• ML.LABEL_ENCODER
• ML.MAX_ABS_SCALER
• ML.MULTI_HOT_ENCODER
• ML.NORMALIZER
• ML.ONE_HOT_ENCODER
• ML.ROBUST_SCALER

Custom holiday modeling for time series forecasting is now in preview. This release offers the following features to improve the transparency, flexibility, and explainability of time series forecasting in BigQuery ML:

• New CREATE MODEL syntax to specify custom holiday modeling for time series models.
• The new ML.HOLIDAY_INFO function, which returns the list of holidays being modeled by an ARIMA_PLUS or ARIMA_PLUS_XREG time series forecasting model.
• An updated ML.EXPLAIN_FORECAST function, which includes an explanation of the holiday effect for each holiday included in the model.

A new public table, bigquery-public-data.ml_datasets.holidays_and_events_for_forecasting, has also been added to provide easy look-up of the built-in holidays used in time series forecasting models.

Try these features with the Use custom holidays in a time-series forecasting model tutorial.

Spanner Data Boost lets you execute analytics queries and data exports with near-zero impact to existing workloads on your provisioned Spanner instance. This feature is now generally available (GA) in the following regions:

• asia-northeast1 (Tokyo)
• us-central1 (Iowa)
• southamerica-east1 (São Paulo)
• europe-west1 (Belgium)
• europe-west2 (London)
• europe-west3 (Frankfurt)

You can use the LOAD DATA SQL statement to load data from Avro, CSV, newline delimited JSON, JSON, ORC, or Parquet files into a table. This feature is generally available (GA).

The slot estimator now provides cost-optimal commitment and autoscale recommendations based on editions pricing and historical performance metrics. This feature is in preview.

The fail-safe period is now generally available (GA). The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. Billed costs won't include the bytes used for fail-safe storage until July 17th, 2023.

The ability to use physical bytes for storage billing is now generally available (GA). When you set your dataset's storage billing model to use physical bytes, the total active storage costs you are billed for includes the bytes used for time travel and fail-safe storage. For more information, see Dataset storage billing models.

The ability to configure the time travel window is now generally available (GA). You can specify the duration of the time travel window from a minimum of two days to a maximum of seven days.

You can now restrict data egress on Analytics Hub listings. This feature is now in preview.

Metadata caching is now available for BigLake tables that reference Amazon S3 data. This feature is in preview. Using cached metadata might improve query performance for BigLake tables.

Support for the following compliance programs is now generally available (GA):

• EU Regions and Support with Sovereignty Controls
• Sovereign Controls by Partners
• International Traffic in Arms Regulation (ITAR)

You can now create stored procedures for Apache Spark using Java or Scala. You can also use the Google Cloud console PySpark editor to add options for stored Python procedures for Apache Spark. This feature is in Preview.

TRUNCATE TABLE is now supported for multi-statement transactions. This feature is now generally available (GA).

Metadata caching is now generally available (GA). Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.

This release includes support for the following new features:

• Protecting metadata cache data with customer-managed encryption keys.
• Statistics on metadata cache usage.
• Table statistics for better query plan performance.

Metadata cache usage is billed going forward. For more information, see Costs.

BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is now generally available (GA).

The following Generative AI features are now in preview with allowlist:

• Creating a remote model based on the Vertex AI large language model (LLM) text-bison.
• Using the ML.GENERATE_TEXT function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

BigQuery now provides information about the fail-safe period. The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. This feature is in preview.

The INFORMATION_SCHEMA views that show table storage metadata are now generally available (GA):

• Use the TABLE_STORAGE view to get a snapshot of current storage usage for tables at the project level.
• Use the TABLE_STORAGE_BY_ORGANIZATION view to get a snapshot of current storage usage for tables at the organization level.

BigLake Metastore is now generally available (GA). You can use BigLake Metastore to access and manage Iceberg table metadata from multiple sources.

The query execution graph is now generally available (GA). You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.

The BigQuery partitioning and clustering recommender is now in preview. The recommender analyzes your BigQuery tables to identify partitioning or clustering opportunities for potential cost savings. You can view partition or cluster recommendations through the BigQuery UI or recommender API. You can also apply recommendations directly to your BigQuery tables.

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Not all data types are supported for filter pushdowns. This feature is generally available (GA).

You can now sort your query results by using the sort menu next to a column name. This feature is in preview.

Object tables are now generally available (GA).

Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. They enable you to analyze and perform inference on images, audio files, documents and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend the data security and governance best practices currently applied to structured data to unstructured data as well.

The GA release includes the following new and updated functions:

• ML.DECODE_IMAGE: Decodes image data so that it can be interpreted by the ML.PREDICT function.
• ML.CONVERT_COLOR_SPACE: Converts images with an RGB color space to a different color space.
• ML.CONVERT_IMAGE_TYPE: Converts the data type of the pixel values in an image.
• ML.RESIZE_IMAGE: Resizes images.
• ML.DISTANCE: Computes the distance between two vectors.
• ML.LP_NORM: Computes the Lᵖ norm for a vector, where ᵖ is the degree.

You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now generally available (GA).

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Pushdowns have limitations, for example not all data types are supported for filter pushdowns. This feature is generally available (GA).

Differential privacy is now in preview and includes four differential privacy aggregate functions that can be used to anonymize data: AVG, COUNT, SUM, and PERCENTILE_CONT. To learn more, see the following topics:

• Use differential privacy
• Differential privacy clause
• Differentially private aggregate functions
• Extending differential privacy

INFORMATION_SCHEMA.MATERIALIZED_VIEW view and enhanced job statistics now let you monitor materialized view usage and refresh jobs. This feature is in preview.

The INSERT INTO SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage and append it into BigQuery tables. This feature is in preview.

You can now use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now in preview.

The table clones feature of BigQuery is now generally available (GA).

You can now add descriptions to the columns of a view. To do this, use the CREATE VIEW or ALTER COLUMN DDL statements. This feature is in preview.

If you use query queues, then you can set the interactive and batch queue timeouts in your default configuration. This feature is in preview.

JSON data type mapping is now available for Cloud Spanner federated queries. This feature is generally available (GA).

BigLake and non-BigLake external tables now support Cloud Storage custom dual-regions. This feature is generally available (GA).

Dynamic data masking has been updated to allow masking on RECORD columns that have been set to REPEATED mode. Previously, querying such columns when data masking had been applied would return internal errors. This feature is generally available (GA).

The BigQuery Data Transfer Service for Google Ads supports the new Google Ads API. The Google Ads connector supports PMax and Discovery campaigns, a limit of 8000 leaf accounts per transfer, the --table_filter flag, and backwards compatibility. This feature is now generally available (GA).

Updates to preferred tables for existing BI engine reservations now take up to ten seconds to propagate, down from five minutes. This feature is generally available (GA).

BigQuery supports setting the rounding mode to ROUND_HALF_EVEN or ROUND_HALF_AWAY_FROM_ZERO for parameterized NUMERIC or BIGNUMERIC columns at the column level. You can specify a default rounding mode at the table or dataset level that is automatically attached to any columns added within those entities. The ROUND() function also accepts the rounding mode as an optional argument. This feature is generally available GA.

The limit for maximum result size (20 GiB logical bytes) when querying Azure or Amazon Simple Storage service (S3) data is now generally available (GA). Querying Azure and Amazon S3 data are now subject to the following quotas and limitations:

• The maximum row size is 10 MiB. For more information, see Quotas for query jobs.

• If your query uses the ORDER BY clause and has a result size larger than 256 MB, then your query fails. Previously, this limit was 2 MB. For more information, see Limitations.

The add data demo guide walks you through the process of adding data to BigQuery through popular sources and is now in preview.

Non-incremental materialized views support most SQL queries, including OUTER JOIN, UNION, and HAVING clauses, as well as analytic functions. This feature is in preview.

BigQuery Partner Center, which can be used to discover and try validated partner applications, is now generally available (GA). In addition, the Google Cloud Ready - BigQuery initiative has added 14 new partners.

Compute (analysis) is now generally available (GA) in three new BigQuery editions: Standard, Enterprise, and Enterprise Plus. These editions support the slots autoscaling model to meet your organizations' needs and budgets.

Autoscaling slots are now generally available (GA). Autoscaling slot reservations and commitments created during the feature's preview have been set to BigQuery Enterprise edition.

You can now use the tf_version training option to specify the Tensorflow (TF) version during model training. By default, tf_version is set as '1.15'. If you want to use TF2 with Keras API, you can add tf_version = '2.8.0' when creating the model.

You can now use the xgboost_version training option to specify the XGBoost version during model training. By default, xgboost_version is set as '0.9'. You can choose XGBoost version 1.1 by specifying xgboost_version = '1.1'.

You can now use the instance_weight_col training option to identify the column containing weights for each data point in the training dataset. Currently the instance_weight_col option is only available for boosted tree and random forest models with non-array feature types.

You can now import model artifacts saved in ONNX, XGBoost, and TensorFlow Lite formats into BigQuery for inference, allowing you to leverage models built in popular frameworks directly within the BigQuery ML inference engine.

You can also host models remotely on Vertex AI Prediction and do inference with BigQuery ML, removing the need to build data pipelines manually.

You can do inference with Google Cloud's state of the art pretrained models using Cloud AI service table-valued functions (TVFs) to get insights from your data. The TVFs work with Cloud Vision API, Cloud Natural Language API and Cloud Translation API.

These features are in preview. To enroll to use this feature, complete the enrollment form.

BigQuery now supports change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API. This feature is in preview.

BigQuery now supports Unicode column naming using international character sets, alphanumeric and special characters. Existing columns can use these new capabilities using the RENAME command. This feature is now in preview.

The following AutoML Tables model features are now generally available:

• Availability in additional regions.
• CMEK support in available regions except multi-regions US and EU.
• OPTIMIZATION_OBJECTIVE now accepts two additional options:
  • MAXIMIZE_PRECISION_AT_RECALL
  • MAXIMIZE_RECALL_AT_PRECISION

The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now generally available (GA).

You can now specify translation configurations in the BigQuery Interactive SQL Translator and use it to debug Batch SQL translator jobs. This feature is now in preview.

The CREATE TABLE AS SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables This feature is in preview.

Case-insensitive collation support is now generally available (GA). In addition to features available in the preview, the GA release includes:

• MIN, MAX, COUNT with DISTINCT, and PERCENTILE_DISC windows functions
• ORDER BY and PARTITION BY in the WINDOWS clause
• LIKE operator with limitations
• Views
• Materialized views with limitations
• Table functions with limitations
• BigQuery BI engine

The WITH RECURSIVE clause is now generally available (GA). This clause lets you include one or more recursive common table expressions (CTEs) in a query.

You can set default values on columns in your BigQuery tables. This feature is now generally available (GA).

Authorized stored procedures are now in preview. This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

Primary and foreign key table constraints are now available in preview. You can define table constraints using the CREATE TABLE statement, the ALTER TABLE ADD PRIMARY KEY statement, or the ALTER TABLE ADD CONSTRAINT statement.

The ALTER TABLE RENAME COLUMN statement and the ALTER TABLE DROP COLUMN statement are now generally available (GA).

You can now make a dataset and the tables in that dataset case-insensitive when you create a dataset or alter a dataset. This feature is generally available (GA).

You can now run bq commands using service account impersonation. This feature is generally available (GA).

In the Explorer pane, the resource corresponding to the focused tab is now selected. This feature is generally available (GA).

In the Explorer pane, you can now see all the resources in the searched resource's level by clicking Show more. This feature is generally available (GA).

You can now create materialized views over BigLake metadata cache-enabled tables to reference structured data stored in Cloud Storage. This feature is in preview.

You can now apply four new types of dynamic data masking to table columns in BigQuery. These new data masking types include date year, email, first four characters, and last four characters masks. This feature is generally available (GA).

Cloud console updates: In the Explorer pane, you can now refresh the contents of a resource (project or dataset). To refresh the contents of a resource, click more_vert View actions, and then click Refresh contents.

The HAVING MAX and HAVING MIN clause for the ANY_VALUE function is now in preview.

You can now view information related to query processing to monitor and optimize queries with the query_info column in INFORMATION_SCHEMA.JOBS, JOBS_BY_FOLDER and JOBS_BY_ORGANIZATION views. This feature is generally available (GA).

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now in preview.

Azure workload identity federation is now generally available (GA) for BigQuery Omni connections. You can now create a connection for federated identity using Google Cloud console.

You can search for BigQuery partners in the BigQuery Partner Center. This feature is in Preview.

More than 20 BigQuery ML components for Vertex AI Managed Pipelines are now generally available. These components benefit AI/ML users for the following:

• Building pipelines using the KFP SDK and TFX SDK
• Linking and tracking metadata automatically
• Seamless integration with Vertex AI for online prediction

Major Google Cloud pipeline components available in Vertex AI are.

• BigqueryQueryJobOp
• BigqueryCreateModelJobOp
• BigqueryExportModelJobOp
• BigqueryPredictModelJobOp
• BigqueryEvaluateModelJobOp
• BigqueryDropModelJobOp
• BigqueryEvaluateModelJobOp
• BigqueryExplainForecastModelJobOp
• BigqueryExplainPredictModelJobOp
• BigqueryForecastModelJobOp

You can now attach Resource Manager tags to datasets, which let you conditionally apply Identity and Access Management (IAM) policies to your resources. This feature is generally available (GA).

You can now use the TRANSFORM clause to train models which you can then export in the Tensorflow SavedModel format. This feature is now available in preview.

The following functions have been added for BigQuery ML:

• ML.ROBUST_SCALER
• ML.NORMALIZER
• ML.ONE_HOT_ENCODER
• ML.IMPUTER
• ML.MAX_ABS_SCALER
• ML.LABEL_ENCODER

These features are now available in preview.

BigQuery ML support for multivariate time-series forecasting with the ARIMA_PLUS_XREG model is now available in preview. This feature lets you perform time-series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG sections in the end-to-end user journey and the multivariate time-series forecasting from Seattle air quality data tutorial.

Sparse input support in BigQuery ML model training is now generally available (GA). This feature improves model training for data whose values are mostly zero or empty. For additional examples, see the sparse features support in BigQuery blog.

The ALTER CAPACITY SET OPTIONS statement and ALTER RESERVATION SET OPTIONS statement are now generally available (GA). Additionally, the CREATE CAPACITY, CREATE RESERVATION, and CREATE ASSIGNMENT statements now support the OPTIONS clause.

The following generally available (GA) features have been added for sessions:

• In a session, temporary functions are now maintained until the session ends.

• In a session, statements that include the TEMP keyword can also include the OR REPLACE and IF NOT EXISTS keywords.

BigQuery ML support for image analytics with vision models is available in preview. Customers can import vision models to perform inference modeling with images to detect objects, perform optical character recognition (OCR), and more. To request access to these features, complete the BigQuery ML interest sign up form.

This new capability uses BigQuery object tables to access image data stored in Cloud Storage and predict results from machine learning models. You can now generate insights from structured and unstructured data with the following steps:

1. Create an object table to access images stored in Cloud Storage.
2. Import vision models in TensorFlow vision models such as ImageNet or ResNet 50, or import your own models to detect objects from images, to annotate photos, and to perform OCR.
3. Unify image data with structured data such as user activities or sales orders to train machine learning models. You can then use prediction results to extract insights from your data.

BigQuery ML integration with Vertex AI Model Registry is now generally available. With this integration, you can now use the following capabilities:

• Register and monitor BigQuery ML models with Vertex AI Model Registry
• Deploy BigQuery ML models directly from Vertex AI Model Registry to Vertex Deployment endpoints
• Use Vertex AI to compare and track evaluation metrics.
• Explainable AI for BigQuery ML models, including built-in XAI, inside Vertex AI
• The seamless integration between BigQuery ML and Vertex AI lets you use Vertex AI for MLOps.

Key features include:

• Model versioning for models registered with Vertex AI Model Registry
• Revision alias for different model versions, and User specified model ID
• List the models by type (custom model, BigQuery ML, AutoML)
• BigQuery ML models can be registered with Vertex AI Model Registry to help you explore, manage, and govern your BigQuery ML models
• Ability to deploy BigQuery ML models to Vertex AI end points
• BigQuery ML models deployed on Vertex AI endpoints can use MLOps features such as model monitoring

The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now in preview.

You can now access and query Cloud SQL data over a private connection. This feature is generally available (GA).

The demo query guide helps you query a public dataset from Google Trends and is now in preview.

BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is in preview.

BigQuery now supports the following features when you load data:

• ASCII control characters for CSV files.
• Reference file with the expected table schema for creating external tables with Avro, ORC, and Parquet files.

These features are generally available (GA).

Object tables are now in preview. Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. These tables enable you to analyze and perform inference on images, audio files, documents, and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend structured data features such as data security and governance best practices to unstructured data.

Metadata caching is now in preview. Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.

The slot estimator helps you manage slot capacity based on historical performance metrics. This feature is now generally available (GA).

You can now transfer data from Amazon S3 and Azure Blob Storage to BigQuery using the LOAD DATA statement. This feature is generally available (GA) and includes support for the following features:

• Transfer files that are hive partitioned.
• Load semi-structured JSON source data into BigQuery without providing a schema by using JSON columns in the destination table.
• Encrypt destination tables using customer managed encryption keys.
• Transfer data to US multi-region and US-EAST-4 regions.

In the Cloud console, the Add data feature lets you access popular ways to search for and ingest data sources that work with BigQuery. For an example, see viewing listings in Analytics Hub. This feature is now generally available (GA).

SQL functions for managing wrapped keysets are generally available (GA). You can now perform the following actions natively in BigQuery with fewer risks and steps:

• Create a wrapped keyset
• Rotate a wrapped keyset
• Rewrap a wrapped keyset
• Encrypt and decrypt a column with a wrapped keyset

Included with this release are the following new key management functions:

• KEYS.NEW_WRAPPED_KEYSET
• KEYS.ROTATE_WRAPPED_KEYSET
• KEYS.REWRAP_KEYSET

The query execution graph is now in preview. You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.

The BigQuery migration assessment is now available for Amazon Redshift in preview. You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.

The max_staleness materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.

Column-level data masking is now generally available (GA). You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column.

Search indexes and the SEARCH() function are now generally available (GA). These enable you to use Google Standard SQL to efficiently pinpoint specific data elements in unstructured text and semi-structured data.

The following geography functions are now generally available (GA):

• ST_ISCLOSED: Returns TRUE for a non-empty geography, where each element in the geography has an empty boundary.
• ST_ISRING: Checks if a geography is a linestring and if the linestring is both closed and simple.

You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now in preview.

Remote functions, which let you invoke functions from Cloud Functions or Cloud Run in your Google Standard SQL queries, are now generally available (GA).

Analytics Hub is now generally available. As an Analytics Hub publisher, you can now view all subscriptions to your listing and remove a subscription from your listing.

You can now use stored procedures for Apache Spark. This feature is in preview.

Multi-statement transactions are now generally available (GA).

The ability to use physical bytes for storage billing is now in Preview. For more information, see Dataset storage billing models.

You can now explore query results in Colab using Python libraries. This feature is now in preview.

BigQuery supports JSON as a native column type. This feature is now generally available (GA).

In addition to standard rounding, BigQuery now supports the rounding mode ROUND_HALF_EVEN for parameterized NUMERIC or BIGNUMERIC columns. The ROUND() function also accepts the rounding mode as an optional argument. This feature is now in preview.

With Datastream for BigQuery, you can now replicate data and schema updates from operational databases directly into BigQuery. This feature is now in preview.

In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available (GA).

BigQuery Omni has introduced support for on-demand pricing model (GA) for a limited duration. For more information, see BigQuery Omni Pricing.

You can now view shuffle usage ratios in the admin resource charts. This feature is now in preview.

BigQuery Omni now supports the following quota and limit:

• The quota for total query result sizes for a project is now 1 TB per day. For more information, see Query jobs.
• The limit for maximum result size for a query has been increased from 2 MB to 10 GB (preview).

For more information, see Limitations.

The BigQuery Data Transfer Service for Google Ads now supports the new Google Ads API. This feature is now in preview.

The is_case_insensitive schema option, which allows you to make a dataset and its table names case-insensitive, is now in preview.

JDBC driver update, release 1.3.0 1001

• You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation).
• You can now configure the connector to use Private Service Connect URLs.
• The connector now supports JDBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
• The connector is now verified to use a default project for datasets. To do this, set the dataset_project_id property in QueryProperties of the connection string to the desired project.
• MATERIALIZED_VIEW has been added to the list of table types when using the getTableTypes function.
• The connector now supports the JSON data type.

ODBC driver update, release 2.5.0 1001

• You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation), with limited support, using Azure AD and Okta identity providers.
• You can now configure the connector to use Private Service Connect URLs.
• The connector now supports ODBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
• The connector is now verified to use a default project for datasets. To do this, set the dataset_project_id property in QueryProperties of the connection string to the desired project.
• MATERIALIZED_VIEW has been added to the list of table types. To retrieve these table types, configure SQLTables to TABLE_TYPES_ONLY.
• The connector now supports the JSON data type.

In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API's concurrent connections and throughput quotas. This feature is now generally available (GA).

The following features are now generally available for ARIMA_PLUS models:

• The HOLIDAY_REGION option can now take more than one region string as input. If you include more than one region string, the union of the holidays in all of the provided regions will be taken into the modeling.
• You can use the new TREND_SMOOTHING_WINDOW_SIZE option to smooth the trend component of the time series by applying a center moving average.

Cloud console updates: Improvements that are related to query execution include the following:

• For long-running queries, the Execution details tab is automatically displayed with the timing details of each stage of the query.

• In the query editor, you can now see the query validation message when your query is completed or canceled.

Customer-managed encryption keys are now integrated with CMEK organization policies. This feature is generally available (GA).

The slot recommender creates recommendations for customers using on-demand billing and is now generally available (GA).

The Random Forest model is now generally available (GA). For more information, see the random forest sections in the end-to-end user journey page.