Name: kafka-pubsub-emulator
Owner: Google Cloud Platform
Description: Implementation of Google Cloud Pub/Sub backed by Apache Kafka.
Created: 2018-04-19 12:23:34.0
Updated: 2018-05-21 16:54:30.0
Pushed: 2018-05-21 13:08:08.0
Size: 245
Language: Java
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This project implements a gRPC server that satisfies the Cloud Pub/Sub API as an emulation layer on top of an existing Kafka cluster configuration. The emulator is exposed as a standalone Java application with a mandatory configuration file passed as an argument at runtime. It is fully compatible with the latest versions of the Google Cloud Client libraries and is explicitly tested against the Java version.
Checkout the source and then build the self-contained JAR with mvn package. If you wish to run the
integration test suite, which starts an in-memory Kafka cluster, run mvn verify.
The self-contained JAR is then available at ./target/kafka-pubsub-emulator-<version>.jar.
When invoking the standalone JAR, you must specify the location of a configuration file using
the --configuration.location command line argument.
This is an example using the default configuration from the resources folder. This configuration assumes that you have Kafka running locally on ports 9094, 9095, and 9096 and topics named throughput-testing-3p, throughput-testing-6p, throughput-testing-100p, and testing-10p respectively.
-jar target/kafka-pubsub-emulator-1.0.0.0.jar \
configuration.location=src/main/resources/application.yaml
Most likely, you will need to specify your own configuration file as described in the Configuration Options section below.
To execute docker container, you must provide a volume with configuration file.
This is an example using the default configuration from the resources folder. This configuration assumes that you have Kafka running locally on ports 9094, 9095, and 9096 and topics named throughput-testing-3p, throughput-testing-6p, throughput-testing-100p, and testing-10p respectively.
rt PATH_TO_CONFIG="/insert/path/to/configuration/file.yaml"
er build -t kafka-pubsub-emulator:1.0.0.0 .
er run --mount type=bind,src=${PATH_TO_CONFIG},dst=/etc/config -d kafka-pub-sub-emulator:1.0.0.0 \
configuration.location=/etc/config/application.yaml
The configuration for kubernetes was based on Minikube, to configure see more here.
This is an example using the default configuration from the resources folder. This configuration assumes that you have Kafka running locally on ports 9094, 9095, and 9096 and topics named throughput-testing-3p, throughput-testing-6p, throughput-testing-100p, and testing-10p respectively.
Build application container:
er build -t kafka-pubsub-emulator:1.0.0.0 .
Expose application configuration on Config Map.
ctl create configmap application-config --from-file=src/main/resources/application.yaml
Create deployment see more, with 1 application pods.
ctl create -f kubernetes/kafka-pubsub-emulator-deployment.yaml
Create service load balancer for emulator application see more.
ctl create -f kubernetes/kafka-pubsub-emulator-loadbalancer.yaml
The Pub/Sub Emulator server needs to be started with a YAML-based configuration file that indicates that specifies the addresses of the Kafka brokers, as well as the Topics and Subscriptions that the emulator will support. See src/main/resources/application.yaml for the proper format.
To connect client applications, you can use the official Google Cloud Platform client libraries for your preferred language. The examples below assume you are using the Java libraries. These settings can be adapted for other languages.
By default, the client library attempts to use your Google Cloud Project credentials as explained in the Authentication docs, and connects to pubsub.googleapis.com. Since the emulator does not run in GCP, you'll need to specify a custom CredentialsProvider and create a new Channel bound to the emulator's host and port.
Below is an example that will create a Publisher client connected to the emulator server running
on the local machine at port 8080 using a plaintext connection.
gedChannel channel = ManagedChannelBuilder.forAddress("localhost", 8080)
.usePlaintext(true)
.build();
isher publisher = Publisher.newBuilder(ProjectTopicName.of("my-project", "my-topic")
.setChannelProvider(
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel)))
.setCredentialsProvider(new NoCredentialsProvider())
.build();
If your emulator server is using SSL/TLS, you will need to create a secure Channel using a slightly different instantiation pattern.
certificate = new File("path-to-certificate.crt");
gedChannel channel;
{
annel =
NettyChannelBuilder.forAddress("localhost", 8080)
.maxInboundMessageSize(100000)
.sslContext(GrpcSslContexts.forClient().trustManager(certificate).build())
.build();
tch (SSLException e) {
stem.err.println("Unable to create SSL channel " + e.getMessage());
isher publisher = Publisher.newBuilder(ProjectTopicName.of("my-project", "my-topic")
.setChannelProvider(
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel)))
.setCredentialsProvider(new NoCredentialsProvider())
.build();
One difference between how the client library behaves with the emulator vs. Cloud Pub/Sub is that by default, clients connected to Cloud Pub/Sub open multiple channels (1 per CPU core). Currently, it's not possible to specify that type of multi-channel configuration with the emulator without writing custom Channel code.
For further reference, consult the examples in the integration tests.