Message Queue CKafka Getting Started - Product Documentation - Message Queue CKafka
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Message Queue CKafka
Message Queue CKafka
Getting Started
Product Documentation
©2013-2019 Tencent Cloud. All rights reserved. Page 1 of 38
Message Queue CKafka Copyright Notice ©2013-2019 Tencent Cloud. All rights reserved. Copyright in this document is exclusively owned by Tencent Cloud. You must not reproduce, modify, copy or distribute in any way, in whole or in part, the contents of this document without Tencent Cloud's the prior written consent. Trademark Notice All trademarks associated with Tencent Cloud and its services are owned by Tencent Cloud Computing (Beijing) Company Limited and its affiliated companies. Trademarks of third parties referred to in this document are owned by their respective proprietors. Service Statement This document is intended to provide users with general information about Tencent Cloud's products and services only and does not form part of Tencent Cloud's terms and conditions. Tencent Cloud's products or services are subject to change. Specific products and services and the standards applicable to them are exclusively provided for in Tencent Cloud's applicable terms and conditions. ©2013-2019 Tencent Cloud. All rights reserved. Page 2 of 38
Message Queue CKafka
Contents
Getting Started
Process Overview
VPC Access
Step 1. Create an Instance
Step 2. Create a Topic
Step 3. Send/Receive Messages
Using SDK to Receive/Send Message (Recommended)
Running Kafka Client (Optional)
Access via Public Domain Name
Step 1. Create an Instance
Step 2. Add a Public Route
Step 3. Create a Topic
Step 4. Configure an ACL policy
Step 5. Send/Receive Messages
Using SDK to Receive/Send Message (Recommended)
Running Kafka Client (Optional)
©2013-2019 Tencent Cloud. All rights reserved. Page 3 of 38Message Queue CKafka Getting Started Process Overview Last updated：2021-09-26 17:17:02 The process of accessing CKafka varies according to the network type: Access via a VPC: you can use the VPC of an existing CVM to access the CKafka service, eliminating the need to add an extra route. Access via a public network route: you need to enable a separate public route and configure an ACL policy for the topic. Operation Procedures ©2013-2019 Tencent Cloud. All rights reserved. Page 4 of 38
Message Queue CKafka ©2013-2019 Tencent Cloud. All rights reserved. Page 5 of 38
Message Queue CKafka
VPC Access
Step 1. Create an Instance
Last updated：2021-09-22 18:05:10
Overview
This document describes how to create an instance and deploy a VPC via the CKafka console.
Prerequisites
You have signed up for a Tencent Cloud account.
You have created a VPC.
Directions
1. Log in to the CKafka console.
2. Choose Instance List in the left sidebar, click Create to go to the instance purchase page, and
enter the purchase information as needed.
Billing Mode: monthly subscription
Specs Type: select the Standard or Pro edition based on your business needs.
Kafka Version: select a Kafka version based on your business needs. For more information,
please see Suggestions for CKafka Edition Selection.
Region: select a region close to the resource for client deployment.
AZ:
Standard Edition: does not support multi-AZ deployment.
Pro Edition: if the current region supports multi-AZ deployment, you can select up to 2 AZs for
deployment. For more information on multi-AZ deployment, please see Multi-AZ Deployment.
Product Specification: select a model based on the peak bandwidth and disk capacity.
Message Retention: select a value between 24 and 2160 hours.
When the disk capacity is insufficient (i.e., the disk utilization reaches 90%), previous messages
will be deleted in advance to ensure the service availability.
Instance name: when purchasing multiple instances, you can batch create instances by its
numeric suffix (which is numbered in an ascending order) or its designated pattern string. For
©2013-2019 Tencent Cloud. All rights reserved. Page 6 of 38Message Queue CKafka
detailed directions, please see Naming with Consecutive Numeric Suffixes or Designated Pattern
String.
3. Select the VPC where the CVM resides.
If you want to use other VPCs, follow the steps in Adding Routing Policy to modify the routing
rules.
4. Click Buy Now. The instance created is displayed in the instance list in about 3-5 minutes.
©2013-2019 Tencent Cloud. All rights reserved. Page 7 of 38Message Queue CKafka Step 2. Create a Topic Last updated：2021-06-28 14:16:08 Overview This document describes how to create a topic under an existing instance in the CKafka console. Directions 1. Log in to the CKafka console. 2. On the Instance List page, click the ID of the instance created in step 1 to go to the instance details page. 3. On the instance details page, click the Topic Management tab and click Create. ©2013-2019 Tencent Cloud. All rights reserved. Page 8 of 38
Message Queue CKafka
4. In the Create Topic dialog box, set parameters as needed.
Name: topic name, which cannot be changed once entered and can contain only letters, digits,
underscores (_), hyphens (-), and dots (.).
Partition Count: number of partitions (physical). A topic can contain one or multiple partitions.
CKafka allocates resources by partition.
Replica Count: number of partition replicas, which ensure the availability of partitions. For data
reliability concerns, CKafka does not support single-replica topics currently. Two replicas are
created for each partition by default.
Replicas are also counted into the number of partitions. For example, if you create 1 topic with 6
partitions, and 2 replicas for each partition, then you have a total of 12 partitions (1 x 6 x 2).
Allowlist: if the allowlist is enabled, the topic can be accessed only from IP addresses in the
allowlist, which ensures data security. You can enable allowlist in either the Create Topic or
Edit Topic window.
5. Click Submit.
©2013-2019 Tencent Cloud. All rights reserved. Page 9 of 38Message Queue CKafka Step 3. Send/Receive Messages Using SDK to Receive/Send Message (Recommended) Last updated：2021-07-14 15:54:38 Overview This document describes how to access CKafka to receive/send messages with the SDK for Java in a VPC. If you need to use SDKs for other languages, please see SDK Documentation. Prerequisites Install JDK 1.8 or later Install Maven 2.5 or later Download demo Directions Step 1. Add the Java dependent library Add the following Java dependent library information to the pom.xml file: org.apache.kafka kafka-clients 0.10.2.2 Step 2. Prepare configurations 1. Create the CKafka configuration file kafka.properties . ## Configure the accessed network by copying the information in the **Network** column in the **A ccess Mode** section on the **Instance Details** page in the console ©2013-2019 Tencent Cloud. All rights reserved. Page 10 of 38
Message Queue CKafka
bootstrap.servers=ckafka-xxxxxxxxxxxxxxxxx
## Configure the topic by copying the information on the **Topic Management** page in the console
topic=XXX
## Configure the consumer group as needed
group.id=XXX
Parameter Description
Accessed network, which can be copied from the Network column in the
Access Mode section on the Instance Details page in the console.
bootstrap.servers
Topic name, which can be copied from the Topic Management page in the
console.
topic
You can customize it. After the demo runs successfully, you can see the
group.id
consumer on the Consumer Group page.
2. Create the configuration file loading program CKafkaConfigurer.java .
public class CKafkaConfigurer {
private static Properties properties;
public synchronized static Properties getCKafkaProperties() {
if (null != properties) {
return properties;
}
// Get the content of the configuration file `kafka.properties`
Properties kafkaProperties = new Properties();
try {
kafkaProperties.load(CKafkaProducerDemo.class.getClassLoader().getResourceAsStream("kafka.propert
ies"));
} catch (Exception e) {
System.out.println("getCKafkaProperties error");
}
©2013-2019 Tencent Cloud. All rights reserved. Page 11 of 38Message Queue CKafka
properties = kafkaProperties;
return kafkaProperties;
}
}
Step 3. Send a message
1. Write the message production program CKafkaProducerDemo.java .
public class CKafkaProducerDemo {
public static void main(String args[]) {
// Load `kafka.properties`
Properties kafkaProperties = CKafkaConfigurer.getCKafkaProperties();
Properties properties = new Properties();
// Set the access point of the corresponding topic, which can be obtained in the console
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getProperty("bootstrap.se
rvers"));
// Set the method for serializing Kafka messages. `StringSerializer` is used in this demo.
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
// Set the maximum time to wait for a request
properties.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 30 * 1000);
// Set the number of retries for the client
properties.put(ProducerConfig.RETRIES_CONFIG, 5);
// Set the internal retry interval for the client
properties.put(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, 3000);
// Construct a producer object
KafkaProducer producer = new KafkaProducer(properties);
// Construct a CKafka message
String topic = kafkaProperties.getProperty("topic"); // Topic of the message. Enter the topic you
created in the console
String value = "this is ckafka msg value"; // Message content
try {
// Batch getting future objects can speed up the process. Note that the batch size should not be
too large
List futureList = new ArrayList(128);
for (int i = 0; i < 10; i++) {
// Send the message and get a future object
ProducerRecord kafkaMsg = new ProducerRecord(topic,
value + ": " + i);
Future metadataFuture = producer.send(kafkaMsg);
futureList.add(metadataFuture);
}
producer.flush();
©2013-2019 Tencent Cloud. All rights reserved. Page 12 of 38Message Queue CKafka
for (Future future : futureList) {
// Sync the obtained future object
RecordMetadata recordMetadata = future.get();
System.out.println("produce send ok: " + recordMetadata.toString());
}
} catch (Exception e) {
// If the sending still fails after client internal retries, the system needs to report and handl
e the error.
System.out.println("error occurred");
}
}
}
2. Compile and run CKafkaProducerDemo.java to send the message.
3. View the execution result.
Produce ok:ckafka-topic-demo-0@198
Produce ok:ckafka-topic-demo-0@199
4. On the Topic Management page in the CKafka console, select the corresponding topic and click
More > Message Query and view the just sent message.
Step 4. Consume the message
1. Create the subscribed message consumer program CKafkaConsumerDemo.java .
©2013-2019 Tencent Cloud. All rights reserved. Page 13 of 38Message Queue CKafka
public class CKafkaConsumerDemo {
public static void main(String args[]) {
// Load `kafka.properties`
Properties kafkaProperties = CKafkaConfigurer.getCKafkaProperties();
Properties props = new Properties();
// Set the access point of the corresponding topic, which can be obtained in the console
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getProperty("bootstrap.server
s"));
// Set the maximum interval between two polls
// If the consumer does not return a heartbeat message within the interval, the broker determines
that the consumer is not alive. the broker removes the consumer from the consumer group and trigg
ers rebalancing. The default value is 30s.
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
// Set the maximum number of messages that can be polled at a time
// Do not set this parameter to an excessively large value. If polled messages are not all consum
ed before the next poll starts, load balancing is triggered and lagging occurs.
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 30);
// Set the method for deserializing messages.
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
// The instances in the same consumer group consume messages in load balancing mode
props.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaProperties.getProperty("group.id"));
// Construct a consumer object. This generates a consumer instance
KafkaConsumer consumer = new KafkaConsumer(props);
// Set one or more topics to which the consumer group subscribes
// We recommend you configure consumer instances with the same `GROUP_ID_CONFIG` value to subscri
be to the same topics
List subscribedTopics = new ArrayList();
// If you want to subscribe to multiple topics, add the topics here
// You must create the topics in the console in advance
String topicStr = kafkaProperties.getProperty("topic");
String[] topics = topicStr.split(",");
for (String topic : topics) {
subscribedTopics.add(topic.trim());
}
consumer.subscribe(subscribedTopics);
// Consume messages in loop
while (true) {
try {
ConsumerRecords records = consumer.poll(1000);
// All messages must be consumed before the next poll, and the total duration cannot exceed the t
imeout interval specified by `SESSION_TIMEOUT_MS_CONFIG`
// We recommend you create an independent thread to consume messages and then return the result i
n async mode
for (ConsumerRecord record : records) {
©2013-2019 Tencent Cloud. All rights reserved. Page 14 of 38Message Queue CKafka
System.out.println(
String.format("Consume partition:%d offset:%d", record.partition(), record.offset()));
}
} catch (Exception e) {
System.out.println("consumer error!");
}
}
}
}
2. Compile and run CKafkaConsumerDemo.java to consume messages.
3. View the execution result.
Consume partition:0 offset:298
Consume partition:0 offset:299
4. On the Consumer Group page in the CKafka console, select the corresponding consumer group,
enter the topic name in Topic Name, and click Query Details to view the consumption details.
©2013-2019 Tencent Cloud. All rights reserved. Page 15 of 38Message Queue CKafka
Running Kafka Client (Optional)
Last updated：2021-07-14 15:54:38
Overview
This document explains how to start using Kafka APIs after you purchase the CKafka service. After
setting up a CKafka environment on a CVM instance, you need to download and decompress the
Kafka installation file and perform simple testing on Kafka APIs.
Directions
Step 1. Install a JDK
1. Check Java installation
Open a terminal window and run this command:
java -version
If the Java version number is output, Java installation is successful. If Java has not been installed,
please download and install the JDK.
2. Set up the Java environment
Set the JAVA_HOME environment variable and point it to the Java installation directory on your
machine.
For example, if you use Java JDK 1.8.0_20, the outputs on different operating systems are as follows:
OS Output
Set the environment variable JAVA_HOME to
Windows
C:\Program Files\Java\jdkjdk1.8.0_20
Linux export JAVA_HOME=/usr/local/java-current
Mac OSX export JAVA_HOME=/Library/Java/Home
Add the Java compiler path to the system path:
OS Output
©2013-2019 Tencent Cloud. All rights reserved. Page 16 of 38Message Queue CKafka
OS Output
Add ;C:\Program Files\Java\jdk1.8.0_20\bin to the end of the system variable
Windows
PATH .
Linux export PATH=$PATH:$JAVA_HOME/bin/
Mac OSX not required
Use the java -version command to check your Java installation.
Step 2. Download the Kafka installation file
Download and decompress the Kafka installation file.
Currently, CKafka is fully compatible with Kafka 0.9, 0.10, 1.1, and 2.4. We recommend that you
download one of these versions. This document uses kafka_2.12-1.1.1 as an example.
Step 3. Test Kafka APIs
Go to the ./bin directory and produce and consume a message with CLI commands.
1. Open a terminal window to start the consumer.
bash kafka-console-consumer.sh --bootstrap-server XXXX:port --topic XXXX --consumer.config ../
config/consumer.properties
Note：
Replace XXXX:port with the domain name and port for VPC access, which can be
obtained in the Access Mode section on the Instance Details page in the console.
topic: replace XXXX with the topic name, which can be obtained on the Topic
Management page in the console.
2. Open another terminal window to start the producer.
©2013-2019 Tencent Cloud. All rights reserved. Page 17 of 38Message Queue CKafka
bash kafka-console-producer.sh --broker-list XXXX:port --topic XXXX --producer.config ../confi
g/producer.properties
Note：
Replace XXXX:port with the domain name and port for VPC access, which can be
obtained in the Access Mode section on the Instance Details page in the console.
topic: replace XXXX with the topic name, which can be obtained on the Topic
Management page in the console.
Enter the content of the message, press Enter, and you can see that the consumer received the
message almost at the same time.
Producing a message:
Consuming a message:
©2013-2019 Tencent Cloud. All rights reserved. Page 18 of 38Message Queue CKafka 3. In the message querying page of the CKafka console, query the message sent. The details of the message are as follows: ©2013-2019 Tencent Cloud. All rights reserved. Page 19 of 38
Message Queue CKafka
Access via Public Domain Name
Step 1. Create an Instance
Last updated：2021-08-26 10:14:34
Overview
This document describes how to create an instance and deploy a VPC via the CKafka console.
Prerequisites
You have signed up for a Tencent Cloud account.
You have created a VPC.
Directions
1. Log in to the CKafka console.
2. Choose Instance List in the left sidebar, click Create to go to the instance purchase page, and
enter the purchase information as needed.
Billing Mode: monthly subscription
Specs Type: select the Standard or Pro edition based on your business needs.
Kafka Version: select a Kafka version based on your business needs. For more information,
please see Suggestions for CKafka Edition Selection.
Region: select a region close to the resource for client deployment.
AZ:
Standard Edition: does not support multi-AZ deployment.
Pro Edition: if the current region supports multi-AZ deployment, you can select up to 2 AZs for
deployment. For more information on multi-AZ deployment, please see Multi-AZ Deployment.
Product Specification: select a model based on the peak bandwidth and disk capacity.
Message Retention: select a value between 24 and 2160 hours.
When the disk capacity is insufficient (i.e., the disk utilization reaches 90%), old messages will
be deleted in advance to ensure the service availability.
VPC: select the created VPC.
Instance name: when purchasing multiple instances, you can batch create instances by its
numeric suffix (which is numbered in an ascending order) or its designated pattern string. For
©2013-2019 Tencent Cloud. All rights reserved. Page 20 of 38Message Queue CKafka
specific operations, please see Naming with Consecutive Numeric Suffixes or Designated
Pattern String.
3. Click Buy Now. The instance created is displayed in the instance list in about 3-5 minutes.
©2013-2019 Tencent Cloud. All rights reserved. Page 21 of 38Message Queue CKafka Step 2. Add a Public Route Last updated：2021-07-14 15:54:38 Overview To enable public network access, you need to add a public route for the instance. This document describes how to add a public route for a created instance in the CKafka console. Prerequisites You have created an instance. Directions 1. On the Instance List page, click the ID/name of the instance created in step 1. 2. On the Instance Details page, click Add a routing policy in the Access Mode section to add a public route. Then, you can get the domain name and port for public network access. ©2013-2019 Tencent Cloud. All rights reserved. Page 22 of 38
Message Queue CKafka Step 3. Create a Topic Last updated：2021-07-14 15:54:39 Overview This document describes how to create a topic under an existing instance in the CKafka console. Directions 1. Log in to the CKafka console. 2. On the Instance List page, click the ID/Name of the instance created in step 1 to enter the Instance Details page. 3. On the Instance Details page, click Topic Management at the top and click Create. ©2013-2019 Tencent Cloud. All rights reserved. Page 23 of 38
Message Queue CKafka
4. In the Edit Topic window, set the number of partitions and replicas and other parameters.
Name: the topic name, which cannot be changed once entered and can only contain letters,
digits, underscores, hyphens, and dots.
Partition Count: it is a concept in physical partition, where one topic can contain one or more
partitions. CKafka uses partition as an allocation unit.
Replica Count: the number of partition replicas is used to ensure the high availability of the
partition. To ensure data reliability, creating a single-replica topic is not supported. Two replicas
are enabled by default.
Replicas are also counted into the number of partitions. For example, if you create 1 topic with 6
partitions, and 2 replicas for each partition, then you have a total of 12 partitions (1 x 6 x 2).
Allowlist: if the allowlist is enabled, the topic can be accessed only from IP addresses in the
allowlist, which ensures data security. You can enable allowlist in either the Create Topic or
Edit Topic window.
5. Click Submit to complete topic creation.
©2013-2019 Tencent Cloud. All rights reserved. Page 24 of 38Message Queue CKafka Step 4. Configure an ACL policy Last updated：2021-07-14 15:54:39 Overview To enable public network access, you need to configure an ACL policy for a topic. This document describes how to configure an ACL policy for a created topic in the CKafka console. Prerequisites You have created a topic. Directions 1. On the Instance Details page, select User Management and click Create to add a user and set the username and password. 2. On the ACL Policy Management page, select Edit ACL Policy in the Operation column of the topic created in step 3 to add read/write permission for the user. ©2013-2019 Tencent Cloud. All rights reserved. Page 25 of 38
Message Queue CKafka ©2013-2019 Tencent Cloud. All rights reserved. Page 26 of 38
Message Queue CKafka
Step 5. Send/Receive Messages
Using SDK to Receive/Send Message
(Recommended)
Last updated：2021-07-14 15:54:39
Overview
This document describes how to access CKafka to receive/send messages with the SDK for Java on
the public network.
If you need to use SDKs for other languages, please see SDK Documentation.
Prerequisites
Install JDK 1.8 or later
Install Maven 2.5 or later
Download demo
Directions
Step 1. Add the Java dependent library
Add the following Java dependent library information to the pom.xml file:
org.apache.kafka
kafka-clients
0.10.2.2
Step 2. Prepare configurations
1. Create the JAAS configuration file ckafka_client_jaas.conf .
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
©2013-2019 Tencent Cloud. All rights reserved. Page 27 of 38Message Queue CKafka
username="yourinstance#yourusername"
password="yourpassword";
};
Note：
username is the combination of the instance ID + # + configured username , and password
is the configured password.
2. Create the CKafka configuration file kafka.properties .
## Configure the accessed network by copying the information in the **Network** column in the **A
ccess Mode** section on the **Instance Details** page in the console
bootstrap.servers=xx.xx.xx.xx:xxxx
## Configure the topic by copying the information on the **Topic Management** page in the console
topic=XXX
## Configure the consumer group as needed
group.id=XXX
## Path of the JAAS configuration file `ckafka_client_jaas.conf`.
java.security.auth.login.config.plain=/xxxx/ckafka_client_jaas.conf
Parameter Description
Accessed network, which can be copied from the
Network column in the Access Mode section on the
Instance Details page in the console.
bootstrap.servers
Topic name, which can be copied from the Topic
Management page in the console.
topic
You can customize it. After the demo runs successfully,
group.id you can see the consumer on the Consumer Group
page.
©2013-2019 Tencent Cloud. All rights reserved. Page 28 of 38Message Queue CKafka
Parameter Description
Enter the path of the JAAS configuration file
java.security.auth.login.config.plain
ckafka_client_jaas.conf .
3. Create the configuration file loading program CKafkaConfigurer.java .
public class CKafkaConfigurer {
private static Properties properties;
public static void configureSaslPlain() {
// If you have used the `-D` parameter or another method to set the path, do not set it again her
e
if (null == System.getProperty("java.security.auth.login.config")) {
// Replace `XXX` with your own path
System.setProperty("java.security.auth.login.config",
getCKafkaProperties().getProperty("java.security.auth.login.config.plain"));
}
}
public synchronized static Properties getCKafkaProperties() {
if (null != properties) {
return properties;
}
// Get the content of the configuration file `kafka.properties`
Properties kafkaProperties = new Properties();
try {
kafkaProperties.load(CKafkaProducerDemo.class.getClassLoader().getResourceAsStream("kafka.propert
ies"));
} catch (Exception e) {
System.out.println("getCKafkaProperties error");
}
properties = kafkaProperties;
return kafkaProperties;
}
}
Step 3. Send a message
1. Create the message sending program KafkaSaslProducerDemo.java .
public class KafkaSaslProducerDemo {
public static void main(String args[]) {
// Set the path of the JAAS configuration file
CKafkaConfigurer.configureSaslPlain();
// Load `kafka.properties`
©2013-2019 Tencent Cloud. All rights reserved. Page 29 of 38Message Queue CKafka
Properties kafkaProperties = CKafkaConfigurer.getCKafkaProperties();
Properties props = new Properties();
// Set the access point of the corresponding topic, which can be obtained in the console
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getProperty("bootstrap.serv
ers"));
// Set the access protocol
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
// Set the PLAIN mechanism
props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
// Set the method for serializing Kafka messages
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.S
tringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serializatio
n.StringSerializer");
// Set the maximum time to wait for a request
props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 30 * 1000);
// Set the number of retries for the client
props.put(ProducerConfig.RETRIES_CONFIG, 5);
// Set the internal retry interval for the client
props.put(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, 3000);
// Construct a producer object. Note: a producer object is thread-safe, and generally one Prod
ucer object is sufficient for a process.
KafkaProducer producer = new KafkaProducer(props);
// Construct a CKafka message
String topic = kafkaProperties.getProperty("topic"); // Topic of the message. Enter the topic
you created in the console
String value = "this is ckafka msg value"; // Message content
try {
// Batch getting future objects can speed up the process. Note that the batch size should not
be too large.
List futures = new ArrayList(128);
for (int i =0; i < 100; i++) {
// Send the message and get a future object
ProducerRecord kafkaMessage = new ProducerRecord(topic, value + ": " + i);
Future metadataFuture = producer.send(kafkaMessage);
futures.add(metadataFuture);
}
producer.flush();
for (Future future: futures) {
// Sync the obtained future object
RecordMetadata recordMetadata = future.get();
System.out.println("Produce ok:" + recordMetadata.toString());
}
} catch (Exception e) {
// If the sending still fails after client internal retries, the system needs to report and ha
ndle the error.
System.out.println("error occurred");
}
©2013-2019 Tencent Cloud. All rights reserved. Page 30 of 38Message Queue CKafka
}
}
2. Compile and run KafkaSaslProducerDemo.java to send the message.
3. View the execution result (output).
Produce ok:ckafka-topic-demo-0@198
Produce ok:ckafka-topic-demo-0@199
4. On the Topic Management page in the CKafka console, select the corresponding topic and click
More > Message Query and view the just sent message.
Step 4. Consume the message
1. Create the subscribed message consumer program KafkaSaslConsumerDemo.java .
public class KafkaSaslConsumerDemo {
public static void main(String args[]) {
// Set the path of the JAAS configuration file
CKafkaConfigurer.configureSaslPlain();
// Load `kafka.properties`
©2013-2019 Tencent Cloud. All rights reserved. Page 31 of 38Message Queue CKafka
Properties kafkaProperties = CKafkaConfigurer.getCKafkaProperties();
Properties props = new Properties();
// Set the access point of the corresponding topic, which can be obtained in the console
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getProperty("bootstrap.server
s"));
// Set the access protocol
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
// Set the PLAIN mechanism
props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
// Set the maximum interval between two polls
// If the consumer does not return a heartbeat message within the interval, the broker determines
that the consumer is not alive. the broker removes the consumer from the consumer group and trigg
ers rebalancing. The default value is 30s.
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
// Set the maximum number of messages that can be polled at a time
// Do not set this parameter to an excessively large value. If polled messages are not all consum
ed before the next poll starts, load balancing is triggered and lagging occurs.
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 30);
// Set the method for deserializing messages.
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.St
ringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.
StringDeserializer");
// Set the consumer group of the current consumer instance. Enter the topic you created in the co
nsole
// The instances in the same consumer group consume messages in load balancing mode
props.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaProperties.getProperty("group.id"));
// Construct a consumer object. This generates a consumer instance
KafkaConsumer consumer = new KafkaConsumer(props);
// Set one or more topics to which the consumer group subscribes
// We recommend you configure consumer instances with the same `GROUP_ID_CONFIG` value to subscri
be to the same topics
List subscribedTopics = new ArrayList();
// If you want to subscribe to multiple topics, add the topics here
// You must create the topics in the console in advance
String topicStr = kafkaProperties.getProperty("topic");
String[] topics = topicStr.split(",");
for (String topic: topics) {
subscribedTopics.add(topic.trim());
}
consumer.subscribe(subscribedTopics);
// Consume messages in loop
while (true){
try {
ConsumerRecords records = consumer.poll(1000);
// All messages must be consumed before the next poll, and the total duration cannot exceed the t
imeout interval specified by `SESSION_TIMEOUT_MS_CONFIG`
for (ConsumerRecord record : records) {
©2013-2019 Tencent Cloud. All rights reserved. Page 32 of 38Message Queue CKafka
System.out.println(String.format("Consume partition:%d offset:%d", record.partition(), record.off
set()));
}
} catch (Exception e) {
System.out.println("consumer error!");
}
}
}
}
2. Compile and run KafkaSaslConsumerDemo.java to consume messages.
3. View the execution result.
Consume partition:0 offset:298
Consume partition:0 offset:299
4. On the Consumer Group page in the CKafka console, select the corresponding consumer group,
enter the topic name in Topic Name, and click Query Details to view the consumption details.
©2013-2019 Tencent Cloud. All rights reserved. Page 33 of 38Message Queue CKafka
Running Kafka Client (Optional)
Last updated：2021-07-14 15:54:39
Overview
This document explains how to start using Kafka APIs after you purchase the CKafka service. After
setting up a CKafka environment on a CVM instance, you need to download and decompress the
Kafka installation file and perform simple testing on Kafka APIs.
Directions
Step 1. Install a JDK
1. Check Java installation
Open a terminal window and run this command:
java -version
If the Java version number is output, Java installation is successful. If Java has not been installed,
please download and install the JDK.
2. Set up the Java environment
Set the JAVA_HOME environment variable and point it to the Java installation directory on your
machine.
For example, if you use Java JDK 1.8.0_20, the outputs on different operating systems are as follows:
OS Output
Set the environment variable JAVA_HOME to
Windows
C:\Program Files\Java\jdkjdk1.8.0_20
Linux export JAVA_HOME=/usr/local/java-current
Mac OSX export JAVA_HOME=/Library/Java/Home
Add the Java compiler path to the system path:
OS Output
©2013-2019 Tencent Cloud. All rights reserved. Page 34 of 38Message Queue CKafka
OS Output
Add ;C:\Program Files\Java\jdk1.8.0_20\bin to the end of the system variable
Windows
PATH .
Linux export PATH=$PATH:$JAVA_HOME/bin/
Mac OSX not required
Use the java -version command to check your Java installation.
Step 2. Download the Kafka installation file
Download and decompress the Kafka installation file.
Currently, CKafka is fully compatible with Kafka 0.9, 0.10, 1.1, and 2.4. We recommend that you
download one of these versions. This document uses kafka_2.12-1.1.1 as an example.
Step 3. Test Kafka APIs
1. Configure an ACL policy locally.
i. In the ./config directory of the installation file, add the following content at the end of
producer.properties and consumer.properties .
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
ii. Create a file named ckafka_client_jaas.conf , and add the following content.
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="yourinstance#yourusername"
password="yourpassword";
};
Note：
username is the combination of the instance ID + # + configured username , and
password is the configured password.
©2013-2019 Tencent Cloud. All rights reserved. Page 35 of 38Message Queue CKafka
iii. In the ./bin directory of the installation file, add the statement of the full path of the JAAS file
at the beginning of kafka-console-producer.sh and kafka-console-consumer.sh .
export KAFKA_OPTS="-Djava.security.auth.login.config=****/config/ckafka_client_jaas.conf"
2. Go to the ./bin directory, and produce and consume a message via CLI commands.
i. Open a terminal window to start the consumer.
bash kafka-console-consumer.sh --bootstrap-server XXXX:port --topic XXXX --consumer.config
../config/consumer.properties
Note：
broker-list: replace XXXX:port with the domain name and port for public network
access, which can be obtained in the Access Mode section on the Instance Details
page in the console.
topic: replace XXXX with the topic name, which can be obtained on the Topic
Management page in the console.
ii. Open another terminal window to start the producer.
bash kafka-console-producer.sh --broker-list XXXX:port --topic XXXX --producer.config ../con
fig/producer.properties
Note：
broker-list: replace XXXX:port with the domain name and port for public network
access, which can be obtained in the Access Mode section on the Instance Details
©2013-2019 Tencent Cloud. All rights reserved. Page 36 of 38Message Queue CKafka
page in the console.
topic: replace XXXX with the topic name, which can be obtained on the Topic
Management page in the console.
Enter the content of the message, press Enter, and you can see that the consumer received the
message almost at the same time.
Producing a message:
Consuming a message:
©2013-2019 Tencent Cloud. All rights reserved. Page 37 of 38Message Queue CKafka 3. In the message querying page of the CKafka console, query the message sent. The details of the message are as follows: ©2013-2019 Tencent Cloud. All rights reserved. Page 38 of 38
You can also read
