Bitzu

Associate Cloud Engineer ยท Choosing and deploying data services

BigQuery, Pub/Sub, Dataflow and Kafka

Datasets and tables, batch vs streaming pipelines, Pub/Sub topics and subscriptions, Dataflow for stream/batch processing, Managed Service for Apache Kafka, when to pick which.

Key points

  • BigQuery: serverless warehouse; datasets (location set at creation) and tables.
  • Pub/Sub: publishers send to a topic; each subscription is attached to one topic, delivers by pull or push.
  • Unacked messages stay for the retention period (default 7 days).
  • Dataflow: serverless Apache Beam, batch and streaming, templates such as Pub/Sub to BigQuery.
  • Managed Service for Apache Kafka: managed clusters for Kafka APIs and tooling.

Source: https://docs.cloud.google.com/dataflow/docs/overview

Practice questions

Single choice

Which Google Cloud service runs Apache Beam pipelines for both batch and streaming data?

  1. Pub/Sub
  2. Dataflow
  3. BigQuery
  4. Cloud SQL
Show answer

Answer: B. Dataflow

Dataflow is the managed runner for Beam pipelines. Pub/Sub only transports messages and BigQuery stores and queries data.

Select all that apply

Which statements about Dataflow are correct?

  1. It is built on Apache Beam.
  2. It only supports streaming pipelines.
  3. It manages worker VMs for the job and can autoscale them.
  4. You must provision and patch a cluster before running a job.
  5. It supports both batch and streaming processing.
Show answer

Answer: A. It is built on Apache Beam., C. It manages worker VMs for the job and can autoscale them., E. It supports both batch and streaming processing.

Dataflow is serverless, based on Apache Beam and handles both batch and streaming. It is not limited to streaming and you do not provision a cluster yourself.

True or false

True or false: Dataflow can process only streaming data, so batch jobs need a different service.

Show answer

Answer: False

False. Dataflow uses one programming model for both batch and streaming pipelines.

Practice 8 questions on this topic in Bitzu

Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.

Download on the App StoreGet it on Google Play

More Associate Cloud Engineer topics in this module