Associate Cloud Engineer ยท Choosing and deploying data services
BigQuery, Pub/Sub, Dataflow and Kafka
Datasets and tables, batch vs streaming pipelines, Pub/Sub topics and subscriptions, Dataflow for stream/batch processing, Managed Service for Apache Kafka, when to pick which.
Key points
- BigQuery: serverless warehouse; datasets (location set at creation) and tables.
- Pub/Sub: publishers send to a topic; each subscription is attached to one topic, delivers by pull or push.
- Unacked messages stay for the retention period (default 7 days).
- Dataflow: serverless Apache Beam, batch and streaming, templates such as Pub/Sub to BigQuery.
- Managed Service for Apache Kafka: managed clusters for Kafka APIs and tooling.
Source: https://docs.cloud.google.com/dataflow/docs/overview
Practice questions
Which Google Cloud service runs Apache Beam pipelines for both batch and streaming data?
- Pub/Sub
- Dataflow
- BigQuery
- Cloud SQL
Show answer
Answer: B. Dataflow
Dataflow is the managed runner for Beam pipelines. Pub/Sub only transports messages and BigQuery stores and queries data.
Which statements about Dataflow are correct?
- It is built on Apache Beam.
- It only supports streaming pipelines.
- It manages worker VMs for the job and can autoscale them.
- You must provision and patch a cluster before running a job.
- It supports both batch and streaming processing.
Show answer
Answer: A. It is built on Apache Beam., C. It manages worker VMs for the job and can autoscale them., E. It supports both batch and streaming processing.
Dataflow is serverless, based on Apache Beam and handles both batch and streaming. It is not limited to streaming and you do not provision a cluster yourself.
True or false: Dataflow can process only streaming data, so batch jobs need a different service.
Show answer
Answer: False
False. Dataflow uses one programming model for both batch and streaming pipelines.
Practice 8 questions on this topic in Bitzu
Short lessons, instant explanations and spaced review of what you get wrong. Free on iOS and Android.
