What is Pub/Sub?  |  Cloud Pub/Sub Documentation  |  Google Cloud (2022)

Pub/Sub allows services to communicate asynchronously, withlatencies on the order of 100 milliseconds.

Pub/Sub is used for streaming analytics and data integrationpipelines to ingest and distribute data. It's equally effective as amessaging-oriented middleware for service integration or as a queue to parallelize tasks.

Pub/Sub enables you to create systems of event producers andconsumers, called publishers and subscribers. Publishers communicate withsubscribers asynchronously by broadcasting events, rather than bysynchronous remote procedure calls (RPCs).

Publishers send events to the Pub/Sub service, without regard tohow or when these events are to be processed. Pub/Sub thendelivers events to all the services that react to them. In systems communicatingthrough RPCs, publishers must wait for subscribers to receive the data. However,the asynchronous integration in Pub/Sub increases the flexibilityand robustness of the overall system.

To get started with Pub/Sub, check out theQuickstart using Google Cloud console.For a more comprehensive introduction, seeBuilding a Pub/Sub messaging system.

Common use cases

  • Ingestion user interaction and server events. To use userinteraction events from end-user apps or server events from your system, you mightforward them to Pub/Sub. You can then use a stream processingtool, such as Dataflow, which delivers the events to databases.Examples of such databases are BigQuery, Cloud Bigtable, andCloud Storage. Pub/Sub lets you gather events from manyclients simultaneously.

  • Real-time event distribution. Events, raw or processed, may be madeavailable to multiple applications across your team and organization for real-time processing. Pub/Sub supports an "enterprise event bus" andevent-driven application design patterns. Pub/Sub lets youintegrate with many Google systems that export events to Pub/Sub.

  • Replicating data among databases. Pub/Sub is commonly usedto distribute change events from databases. These events can be used toconstruct a view of the database state and state history inBigQuery and other data storage systems.

  • Parallel processing and workflows. You can efficiently distribute manytasks among multiple workers by using Pub/Sub messagesto connect to Cloud Functions. Examples of such tasks are compressing textfiles, sending email notifications, evaluating AI models, and reformattingimages.

  • Enterprise event bus. You can create an enterprise-wide real-time datasharing bus, distributing business events, database updates, and analyticsevents across your organization.

  • Data streaming from applications, services, or IoT devices. For example, aSaaS application can publish a real-time feed of events. Or, a residential sensorcan stream data to Pub/Sub for use in other Google Cloud productsthrough a Dataflow pipeline.

  • Refreshing distributed caches. For example, an application can publishinvalidation events to update the IDs of objects that have changed.

  • Load balancing for reliability. For example, instances of a service may bedeployed on Compute Engine in multiple zones but subscribe to a common topic.When the service fails in any zone, the others can pick up the loadautomatically.

Types of Pub/Sub services

Pub/Sub consists of two services:

  • Pub/Sub service. This messaging service is the defaultchoice for most users and applications. It offers the highest reliability andlargest set of integrations, along with automatic capacity management.Pub/Sub guarantees synchronous replication of all data to atleast two zones and best-effort replication to a third additional zone.

  • Pub/Sub Lite service. A separate but similar messagingservice built for lower cost. It offers lower reliability compared toPub/Sub. It offers either zonal or regional topic storage.Zonal Lite topics are stored in only onezone. Regional Lite topics replicate data to a secondzone asynchronously. Also, Pub/Sub Literequires you to pre-provision and manage storage and throughput capacity.Consider Pub/Sub Lite only for applications where achievinga low cost justifies some additional operational work and lowerreliability.

For more details about the differences between Pub/Sub andPub/Sub Lite, seeChoosing Pub/Sub or Pub/Sub Lite.

Comparing Pub/Sub to other messaging technologies

Pub/Sub combines the horizontal scalability ofApache Kafka andPulsar withfeatures found in traditional messaging middleware such as Apache ActiveMQ andRabbitMQ. Examples of such features are dead-letter queues and filtering.

Another feature that Pub/Sub adopts from messaging middleware isper-message parallelism, rather than partition-based messaging.Pub/Sub "leases" individual messages to subscriber clients, thentracks whether a given message is successfully processed.

By contrast, other horizontally scalable messaging systemsuse partitions for horizontal scaling. This forces subscribersto process messages in each partition in order and limits the number of concurrentclients to the number of partitions. Per-message processingmaximizes the parallelism of subscriber applications, and helps ensurepublisher/subscriber independence.

Compare Service-to-service and service-to-client communication

Pub/Sub is intended for service-to-service communication ratherthan communication with end-user or IoT clients. Other patterns arebetter supported by other products:

  • Client-server. To send messages between a mobile/web app and a service,use products that include Firebase Realtime Database andFirebase Cloud Messaging.
  • IoT-client-service. To send messages between an IoT app and a service,use Cloud IoT Core
  • Asynchronous service calls. Use Cloud Tasks

You can use a combination of these services to build client -> services -> databasepatterns. For example, see the tutorialStreaming Pub/Sub messages over WebSockets.

Integrations

Pub/Sub has many integrations with other Google Cloud products to create a fullyfeatured messaging system:

  • Stream processing and data integration. Supported by Dataflow, including Dataflow templates and SQL, which allow processing anddata integration into BigQuery and data lakes on Cloud Storage. Dataflowtemplates for moving data from Pub/Sub toCloud Storage, BigQuery, and other products are available inthe Pub/Sub and Dataflow UIs in theGoogle Cloud console. Integration with Apache Spark, particularly when managed with Dataproc is also available. Visual composition of integration andprocessing pipelines running on Spark + Dataproc can be accomplished withData Fusion.
  • Monitoring, Alerting and Logging. Supported by Monitoring andLogging products.
  • Authentication and IAM. Pub/Sub relies on a standard OAuthauthentication used by other Google Cloud products and supports granular IAM,enabling access control for individual resources.
  • APIs. Pub/Sub uses standard gRPC and REST service APItechnologies along with client libraries for several languages.
  • Triggers, notifications, and webhooks. Pub/Sub offers push-baseddelivery of messages as HTTP POST requests to webhooks. You can implement workflow automation using Cloud Functionsor other serverless products.
  • Orchestration. Pub/Sub can be integrated into multistep serverlessWorkflows declaratively. Big data and analytic orchestration often done withCloud Composer, which supports Pub/Sub triggers.You can also integrate Pub/Sub withApplication Integration(Preview) which is anIntegration-Platform-as-a-Service (iPaaS) solution. ApplicationIntegration provides aPub/Sub triggerto trigger or start integrations.
  • Integration Connectors.(Preview)These connectors let you connect to various data sources.With connectors, both Google Cloud services and third-party business applications are exposedto your integrations through a transparent, standard interface. For Pub/Sub, you can create a Pub/Sub connection for use in your integrations.

Core concepts

  • Topic. A named resource to which messages are sent by publishers.
  • Subscription. A named resource representing the stream of messages froma single, specific topic, to be delivered to the subscribing application.For more details about subscriptions and message delivery semantics, seetheSubscriber Guide.
  • Message. The combination of data and (optional) attributes that apublisher sends to a topic and is eventually delivered to subscribers.
  • Message attribute. A key-value pair that a publisher can define for amessage. For example, key iana.org/language_tag and value en could beadded to messages to mark them as readable by an English-speaking subscriber.
  • Publisher. An application that creates and sends messages to a single or multiple topics.
  • Subscriber. An application with a subscription to a single or multiple topics to receive messages from it.
  • Acknowledgment (or "ack"). A signal sent by a subscriber to Pub/Sub after it hasreceived a message successfully. Acknowledged messages are removed from the subscription message queue.
  • Push and pull. The two message delivery methods. A subscriber receivesmessages either by Pub/Sub pushing them to the subscriberchosen endpoint, or by the subscriber pulling them from the service.

Publisher-subscriber relationships can be one-to-many (fan-out), many-to-one (fan-in), andmany-to-many, as shown in the following diagram:

What is Pub/Sub? | Cloud Pub/Sub Documentation | Google Cloud (1)

The following diagram illustrates how a message passes from a publisher to asubscriber. For push delivery, the acknowledgment is implicit in the response tothe push request, while for pull delivery it requires a separate RPC.

What is Pub/Sub? | Cloud Pub/Sub Documentation | Google Cloud (2)

Next steps

  • Get started with the Pub/Sub quickstart or the Pub/Sub Lite quickstart.
  • Read the architectural overview of Pub/Sub.
  • Learn how to build a Pub/Sub messaging system.
  • Understand Pub/Sub pricing.
  • Understand quotas and limits for Pub/Sub and Pub/Sub Lite.
  • Read the Pub/Sub release notes.
  • Explore data engineering with Google Cloud services on Qwiklabs.

Top Articles

Latest Posts

Article information

Author: Velia Krajcik

Last Updated: 12/18/2022

Views: 5594

Rating: 4.3 / 5 (54 voted)

Reviews: 93% of readers found this page helpful

Author information

Name: Velia Krajcik

Birthday: 1996-07-27

Address: 520 Balistreri Mount, South Armand, OR 60528

Phone: +466880739437

Job: Future Retail Associate

Hobby: Polo, Scouting, Worldbuilding, Cosplaying, Photography, Rowing, Nordic skating

Introduction: My name is Velia Krajcik, I am a handsome, clean, lucky, gleaming, magnificent, proud, glorious person who loves writing and wants to share my knowledge and understanding with you.