Google Cloud Bigquery Overview Diagram

So many features

Google has added from the general availability so many features to Bigquery and around Bigquery, that this is now hard to have a full overview, this is the goal of this article to try to list them and to build an overview diagram.

Help wanted : This is a draft, your comments are welcome if some other important adherent feature is missing, or errors.

We will not go into the details of each feature here, to have more details, you can always consult the Google Documentation: https://cloud.google.com/bigquery/docs/introduction

APIs

First of all, we are not talking here of only one single API, the table below lists no less than 9 APIs directly Bigquery related and 5 APIs very close apis (KMS, DLP, …)

• BigQuery API / bigquery.googleapis.com
  A data platform for customers to create, manage, share and query data.Note: includes Bigquery ML.
• BigQuery Storage API / bigquerystorage.googleapis.com
  Google BigQuery API is a data platform for group of users to create, manage, share and query data.
• Bigquery Data Transfer Service API / bigquerydatatransfer.googleapis.com
  Transfers data from partner SaaS applications to Google BigQuery on a scheduled, managed basis.
• BigQuery Connection API / bigqueryconnection.googleapis.com
  Allows users to manage BigQuery connections to external data sources
• Bigquery BigLake API / biglake.googleapis.com
  BigLake is a storage engine that unifies data warehouses and lakes by enabling BigQuery and open source frameworks like Spark to access data with fine-grained access control.
• Google Cloud Data Catalog API / datacatalog.googleapis.com
  A data discovery and metadata management service.
• BigQuery Data Policy API / bigquerydatapolicy.googleapis.com
  Allows users to manage BigQuery data policies.
• Dataplex API / dataplex.googleapis.com
  To manage the lifecycle of data lakes.
• Cloud DLP API / dlp.googleapis.com
  To detect Sensitive Data within the BigQuery data
• Cloud KMS API / cloudkms.googleapis.com
  All data assets residing in BigQuery managed storage support CMEK
• BigQuery Reservation API / bigqueryreservation.googleapis.com
  A service to modify your BigQuery flat-rate reservations.
• Cloud Dataform / dataform.googleapis.com
  Service to develop, version control, and operationalize SQL pipelines in BigQuery.
• BigQuery Analytics Hub API / analyticshub.googleapis.com
  Exchange data and analytics assets securely and efficiently.
• Cloud AI Companion API for Duet AI / cloudaicompanion.googleapis.com
  Cloud AI Companion API for Duet AI
• BigQuery Migration API / bigquerymigration.googleapis.com
  The migration service, exposing apis for migration jobs operations, and agent management.

## Categories

I will now try to list the involved features categories:

• Core Services
  The well known Datasets, tables, views, routines & jobs
• Transfers with the Bigquery Data Transfer API
  for example from external Cloud Providers (AWS S3, Azure Blob Storage)
  from Google Saas such as Campaign Manager, Google Ads, Google Play, …)
  from Datawarehouses (Teradata, AWS Redshift)
  from 3rd parties (Salesforce CRM, Adobe Analytics, …)
  and also copy data from BQ datasets, with scheduled queries), Google Cloud Storage
• Import data from Google Cloud Storage, Google Drive
• Use Bigquery data within Google Spreadsheet with Connected Sheets
• Use live connection on your queries with the Bigquery Connection API
  through the Biglake API to AWS S3, Azure Blob Storage, Apache Iceberg, Cloud Storage structured and unstructured data
  through Federated queries to Google Cloud Spanner & Cloud SQL
  through remote functions on Google Cloud Functions & Cloud Run
• Bigquery Migration Service
  Tool helping to plan your migration project, migrate your legacy queries
• Data Plex, Data Catalog & Bigquery Data Policy
  To manage your data, setup column-level permissions & masking
• Bigquery Analytics Hub
  to expose and even monetize your data
• Optimize your Bigquery costs with the Bigquery Reservation API
• Sensitive Data Protection
  Use Google Cloud DLP API to detect the BQ tables containing sensitive data and then potentially setup specific permissions or measures
• Cloud KMS
  Encrypt Bigquery data / CMEK
• Cloud Dataform
  Build SQL / SQLX workflows & workspaces with git capabilities
• ODBC / JDBC connectivity for Bigquery with the Simba 3rd party drivers
• Vertex AI ML & Gen AI & BQ ML features
  Bigquery ML
  Notebooks
  Use Vertex AI Feature store, fundation models within Bigquery
• Vizualize and use data
  within Tableau, Looker, LookerStudio
  visualize geospatial data with Bigquery Geo Viz

Diagrams

Help wanted : This is a draft, your comments are welcome if some other important adherent feature are missing, or errors.

Core assets

Features