user guide - huawei cloud · distributed message service for kafka user guide issue 01 date...

Distributed Message Service for Kafka

User Guide

Issue 01

Date 2019-05-17

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.Address: Huawei Industrial Base

Bantian, LonggangShenzhen 518129People's Republic of China

Website: https://www.huawei.com

Email: [email protected]

Issue 01 (2019-05-17) Copyright © Huawei Technologies Co., Ltd. i

https://www.huawei.com

mailto:[email protected]

Contents

1 Permissions Management..................................................................................................... 11.1 Creating a User and Granting DMS Permissions..........................................................................................................11.2 DMS Custom Policies............................................................................................................................................................. 2

2 Preparing the Environment...................................................................................................4

3 Buying an Instance.................................................................................................................. 6

4 Accessing a Kafka Premium Instance............................................................................... 114.1 Accessing a Kafka Premium Instance Without SASL................................................................................................ 114.2 Accessing a Kafka Premium Instance with SASL....................................................................................................... 144.3 Connecting to Kafka Manager and Viewing Kafka Information.......................................................................... 17

5 Operating Kafka Premium Instances................................................................................235.1 Viewing an Instance............................................................................................................................................................. 235.2 Restarting an Instance........................................................................................................................................................ 255.3 Deleting an Instance............................................................................................................................................................ 265.4 Modifying the Information About an Instance...........................................................................................................275.5 Configuring Public Access.................................................................................................................................................. 295.6 Modifying Instance Specifications...................................................................................................................................305.7 Changing an Instance to Yearly/Monthly Billing....................................................................................................... 315.8 Creating a Topic in a Kafka Premium Instance.......................................................................................................... 325.9 Deleting a Topic in a Kafka Premium Instance.......................................................................................................... 335.10 Resetting Kafka Password............................................................................................................................................... 345.11 Resetting Kafka Manager Password............................................................................................................................ 355.12 Querying Messages............................................................................................................................................................365.13 Enabling Dumping............................................................................................................................................................. 375.14 Managing Tags.................................................................................................................................................................... 395.15 Viewing Background Tasks.............................................................................................................................................. 41

6 Monitoring.............................................................................................................................. 426.1 Kafka Metrics..........................................................................................................................................................................426.2 Configuring Alarms.............................................................................................................................................................. 506.3 Viewing Metrics..................................................................................................................................................................... 53

7 Auditing................................................................................................................................... 557.1 Operations That Can Be Recorded by CTS................................................................................................................... 55

Distributed Message Service for KafkaUser Guide Contents

Issue 01 (2019-05-17) Copyright © Huawei Technologies Co., Ltd. ii

7.2 Viewing Traces on the CTS Console................................................................................................................................58

Distributed Message Service for KafkaUser Guide Contents

Issue 01 (2019-05-17) Copyright © Huawei Technologies Co., Ltd. iii

1 Permissions Management

1.1 Creating a User and Granting DMS Permissions

1.2 DMS Custom Policies

1.1 Creating a User and Granting DMS PermissionsThis chapter describes how to use IAM to implement fine-grained permissionscontrol for your DMS resources. With IAM, you can:

● Create IAM users for employees based on your enterprise's organizationalstructure. Each IAM user will have their own security credentials for accessingECS resources.

● Grant only the permissions required for users to perform a specific task.● Entrust a HUAWEI CLOUD account or cloud service to perform efficient O&M

on your DMS resources.

If your HUAWEI CLOUD account does not require individual IAM users, skip thischapter.

This section describes the procedure for granting permissions (see Figure 1-1).

PrerequisitesLearn about the permissions (see System-defined policies supported by DMS)supported by DMS and choose policies according to your requirements. For thesystem policies of other services, see Permission Policy.

Distributed Message Service for KafkaUser Guide 1 Permissions Management

Issue 01 (2019-05-17) Copyright © Huawei Technologies Co., Ltd. 1

https://support.huaweicloud.com/en-us/usermanual-iam/iam_01_0001.html

https://support.huaweicloud.com/en-us/productdesc-kafka/ProductDescPrivilegeManagement.html

https://support.huaweicloud.com/en-us/usermanual-permissions/en-us_topic_0063498930.html

Process Flow

Figure 1-1 Process for granting DMS permissions

1. Create a user group and assign permissions to it.Create a user group on the IAM console, and assign the DMS Viewer policyto the group.

2. Create an IAM user.Create a user on the IAM console and add the user to the group created in 1.

3. Log in and verify permissions.Log in to the DMS console as the created user, and verify that it only has readpermissions for DMS.

1.2 DMS Custom PoliciesCustom policies can be created to supplement the system-defined policies of DMS.For the actions that can be added to custom policies, see Permissions Policiesand Supported Actions.

You can create custom policies in either of the following ways:

● Visual editor: Select cloud services, actions, resources, and request conditions.This does not require knowledge of policy syntax.

● JSON: Edit JSON policies from scratch or based on an existing policy.

For details, see Creating a Custom Policy. The following section containsexamples of common DMS custom policies.






https://support.huaweicloud.com/en-us/api-kafka/api-grant-policy.html

https://support.huaweicloud.com/en-us/api-kafka/api-grant-policy.html


● Kafka permissions policies are based on DMS. Therefore, when assigning permissions,select DMS permissions policies.

● Due to data caching, a policy involving OBS actions will take effect five minutes after itis attached to a user, user group, or project.

Example Custom Policies● Example 1: Allowing users to delete and restart instances

{ "Version": "1.1", "Statement": [ { "Effect": "Allow", "Action": [ " dms:instance:delete dms:instance:modifyStatus " ] } ]}

● Example 2: Denying instance deletionA policy with only "Deny" permissions must be used in conjunction with otherpolicies to take effect. If the permissions assigned to a user contain both"Allow" and "Deny", the "Deny" permissions take precedence over the "Allow"permissions.The following method can be used if you need to assign permissions of theDMS FullAccess policy to a user but you want to prevent the user fromdeleting instances. Create a custom policy for denying instance deletion, andattach both policies to the group to which the user belongs. Then, the usercan perform all operations on DMS except deleting instances. The following isan example deny policy:{ "Version": "1.1", "Statement": [ { "Effect": "Deny", "Action": [ "dms:instance:delete" ] } ]}



2 Preparing the Environment

Before creating Kafka premium instances, you need to create a VPC and configuresecurity groups and subnets for it. A VPC creates an isolated virtual networkenvironment for users to configure and manage Kafka instances, improvingresource security and simplifying network deployment.

Once you have created a VPC, you can use it for all Kafka premium instances yousubsequently create.

Procedure

Step 1 Log in to the management console.

Step 2 Click in the upper left corner of the management console and select a region.

Select the same region as your application service.

Step 3 Click Service List. Choose Network > Virtual Private Cloud to launch the VPCconsole.

Step 4 On the Dashboard page, click Create VPC.

Step 5 Create a VPC as guided by the VPC console, retaining the default values of theVPC parameters unless otherwise required. For details on how to create a VPC, seethe Virtual Private Cloud User Guide.

After a VPC is created, a subnet is also created in the VPC. If the VPC needs moresubnets, perform step Step 6. Otherwise, go to Step 7.

Step 6 Click the name of the VPC for which you want to create a subnet. On the Subnetstab page, click Create Subnet. Create a subnet as prompted, and retaining thedefault values unless otherwise required.

For details on how to create a subnet, see the Virtual Private Cloud User Guide.

Step 7 In the navigation pane, choose Security Group. On the Security Group page, clickCreate Security Group. Create a security group as prompted, retaining the defaultvalues unless otherwise required.

Distributed Message Service for KafkaUser Guide 2 Preparing the Environment


https://support.huaweicloud.com/en-us/usermanual-vpc/en-us_topic_0013935842.html


For details on how to create a security group, see the Virtual Private Cloud UserGuide.

----End

Distributed Message Service for KafkaUser Guide 2 Preparing the Environment




3 Buying an Instance

Scenario

DMS for Kafka provides Kafka premium instances, which are physically isolatedand exclusively occupied by each tenant. You can customize the computingcapabilities and storage space of an instance based on service requirements.

Prerequisites

A VPC configured with security groups and subnets is available.

Procedure




Step 3 Click Service List. Choose Application > Distributed Message Service for Kafkato open the Kafka console.

Step 4 Click Buy Instance in the upper right corner of the page.

By default, you can create a maximum of 100 Kafka premium instances for eachproject. To create more instances, contact customer service to increase your quota.

Step 5 Select a billing mode.

Step 6 Select a region closest to your application to reduce latency and accelerate access.

Step 7 Select one or no less than three AZs.

Step 8 Specify Instance Name and Description.

Step 9 Specify Enterprise Project.

Select an enterprise project to which the instance belongs from the drop-downlist.

Distributed Message Service for KafkaUser Guide 3 Buying an Instance


Step 10 Configure the following instance parameters:

1. Version: Kafka version. Currently, only 1.1.0 is supported.2. Bandwidth:

Attainable bandwidth during intra-VPC access. Options: 100 MB/s, 300 MB/s,600 MB/s, and 1200 MB/s.To ensure service stability, a higher bandwidth is recommended if manyconnections or topics are involved.

3. Maximum Partitions: Retain the default value.This parameter indicates the maximum number of partitions that can becreated for a Kafka premium instance. If the total number of partitions of alltopics exceeds this threshold, topic creation will fail.– 100 MB/s bandwidth: Maximum Partitions is 300.– 300 MB/s bandwidth: Maximum Partitions is 900.– 600 MB/s bandwidth: Maximum Partitions is 1800.– 1200 MB/s bandwidth: Maximum Partitions is 1800.

4. Storage Space: Total disk space for storing the instance data.If the available disk space is less than 5%, you can retrieve but cannot createmessages. Messages do not occupy any disk space after the retention periodends.Disks are formatted when an instance is created. As a result, the actualavailable disk space is 93% to 95% of the total disk space.– 100 MB/s bandwidth: The value range of Storage Space is 600–90000

GB.– 300 MB/s bandwidth: The value range of Storage Space is 1200–90000



GB.

– High I/O + 100 MB/s bandwidth: If the average message size is 1 KB, thetransactions per second (TPS) can reach 100,000 in high throughput scenarios and60,000 in synchronous replication scenarios.

– High I/O + 300 MB/s bandwidth: If the average message size is 1 KB, the TPS canreach 300,000 in high throughput scenarios and 150,000 in synchronous replicationscenarios.

– Ultra-high I/O + 100 MB/s bandwidth: If the average message size is 1 KB, the TPScan reach 100,000 in high throughput scenarios and 80,000 in synchronousreplication scenarios.



– Ultra-high I/O + 1200 MB/s bandwidth: If the average message size is 1 KB, theTPS can reach 1,200,000 in high throughput scenarios and 400,000 in synchronousreplication scenarios.



5. Select a VPC.– A VPC provides an isolated virtual network for your Kafka premium

instances. You can configure and manage the network as required.– Click View VPC. On the displayed VPC console, choose the selected VPC

to view details about it, including security group rules.6. Select a subnet.

Click View Subnet. On the displayed VPC console, choose the selected subnetto view details about it, including the private IP addresses.

7. Select a security group.A security group is a set of rules that control access to ECSs. It provides accesspolicies for mutually trusted ECSs with the same security protectionrequirements in the same VPC.Click Manage Security Group. On the displayed console, view or createsecurity groups.

Figure 3-1 Buying a Kafka premium instance

Step 11 Configure Capacity Threshold Policy.

A capacity threshold policy is the policy used when the disk usage reaches thethreshold. The default capacity threshold is 95%.

● Stop production: New messages cannot be created, but existing messagescan still be retrieved.

● Automatically delete: Messages can be created and retrieved, but 10% ofthe earliest messages will be deleted to ensure sufficient disk space.

Step 12 Configure the username and password for logging in to Kafka Manager.

Step 13 Specify the required duration.

This parameter is displayed only if the billing mode is yearly/monthly.

Step 14 Click More Settings to configure more parameters.

1. Configure public access.



Public access is disabled by default. You can enable or disable it as required.You can create a maximum of three Kafka instances with public accessenabled. If you want to create more instances with public access, contactcustomer service to increase your quota.If public access is enabled, configure the bandwidth.

2. Configure Kafka SASL_SSL.You can configure whether to enable SSL authentication for clients to accessthe instance. This configuration cannot be changed once the instance hasbeen created. If you need to change the configuration, you must createanother instance.If Kafka SASL_SSL is enabled, data will be encrypted before transmission toenhance security.If you enable Kafka SASL_SSL, you must also set the username and passwordfor accessing the instance.Passwords for Kafka premium instances must meet the followingrequirements:– Cannot be empty.– Must be 8 to 32 characters long.– Must contain at least three of the following character types:

▪ Lowercase letters

▪ Uppercase letters

▪ Digits

▪ Special characters (`~!@#$%^&*()-_=+\|[{}]:'",<.>/?)

▪ Cannot start with a hyphen (-).

3. Configure Dumping.After you enable dumping, data will be dumped to OBS buckets for backup.

– If a message contains a key-value attribute, the key cannot be dumped while thevalue can be dumped.

– You cannot disable dumping once it has been enabled.

4. Configure Automatic Topic Creation.You can enable or disable automatic topic creation.If it is enabled, a topic will be automatically created with 3 partitions and 3replicas when a message is created in or retrieved from a topic that does notexist.

5. Specify Time Window.This parameter indicates the time period when O&M personnel performmaintenance operations on the instance.Services can still be used, but occasionally there may be temporary serviceinterruptions. Scheduled maintenance occurs infrequently (typically onceevery few months).

6. Specify Tags.



Tags are used to identify cloud resources. When you have many cloudresources of the same type, you can use tags to classify cloud resources bydimension (for example, use, owner, or environment).– If you have created predefined tags, select a predefined pair of tag key

and value. Click View predefined tags. On the Tag Management Service(TMS) console, view or create predefined tags.

– You can also create new tags by specifying Tag key and Tag value.Up to 20 tags can be added to each Kafka instance. For details about therequirements on tags, see 5.14 Managing Tags.

Step 15 Click Buy Now.

Step 16 Confirm the instance information.

Step 17 After the new Kafka premium instance has been created, return to the KafkaPremium page to view and manage your Kafka premium instances.

1. It takes 3 to 15 minutes to create a Kafka premium instance.2. After a Kafka premium instance has been successfully created, by default its

status should be Running.3. If the new Kafka premium instance fails to be created, delete the unsuccessful

instance creation task by following the procedure in 5.3 Deleting an Instanceand then create the Kafka premium instance again. If the Kafka premiuminstance creation fails again, contact customer service.

----End



4 Accessing a Kafka Premium Instance

4.1 Accessing a Kafka Premium Instance Without SASL

4.2 Accessing a Kafka Premium Instance with SASL

4.3 Connecting to Kafka Manager and Viewing Kafka Information

4.1 Accessing a Kafka Premium Instance Without SASLDMS for Kafka of HUAWEI CLOUD provides Kafka premium instances, which arephysically isolated and exclusively occupied by each tenant. After creating a Kafkapremium instance, you can use an open-source Kafka client to create and retrievemessages in the instance.

This section describes how to use an open-source Kafka client to access a Kafkapremium instance if SASL access is not enabled for the instance.

For details on how to use Kafka clients in different languages, visit https://cwiki.apache.org/confluence/display/KAFKA/Clients.

● The following describes the procedure of accessing a Kafka instance using CLI. To accessan instance in your service code, see the Kafka Premium Developer Guide.

● The Kafka server allows a maximum of 200 clients to be connected using a single IPaddress. If the number of clients exceeds 200, the connection fails.

Prerequisites1. Security group rules have been correctly configured.

A Kafka premium instance with SASL disabled can be accessed within a VPCor over public networks.– For intra-VPC access, ensure that the ECS and the Kafka premium

instance are in the same VPC and that security group rules have beencorrectly configured for the instance. For details on how to configure asecurity group, see How to Select and Configure a Security Group?

– For public access, there are no specific requirements on the VPC.However, you still need to ensure that security group rules have beencorrectly configured for the Kafka premium instance.

Distributed Message Service for KafkaUser Guide 4 Accessing a Kafka Premium Instance


https://cwiki.apache.org/confluence/display/KAFKA/Clients

https://cwiki.apache.org/confluence/display/KAFKA/Clients

https://support.huaweicloud.com/en-us/devg-kafka/Kafka-summary.html

https://support.huaweicloud.com/en-us/kafka_faq/kafka-faq-180604024.html

To access the instance without SASL, allow inbound access through port9094.

Figure 4-1 Security group rules configured for the Kafka instance (withSASL disabled)

2. The instance connection address has been obtained.

– For intra-VPC access, use port 9092. Obtain the instance connectionaddress on the instance details page.

Figure 4-2 Obtaining Kafka instance connection addresses for intra-VPCaccess without SASL

– For public access, use port 9094. Obtain the instance access address onthe instance details page.

Figure 4-3 Obtaining Kafka instance connection addresses for publicaccess without SASL

3. At least one topic has been created for the Kafka premium instance.

4. The ECS environment has been correctly configured, and the Kafka open-source client has been downloaded. If the configuration or the installation hasnot been completed, perform the following procedure.

a. Log in to the ECS.

The following takes Linux ECS as an example. For more information onhow to install JDK and configure the environment variables for aWindows ECS, please search the Internet.



b. Install JDK or JRE, and add the following lines to .bash_profile in thehome directory to configure the environment variables JAVA_HOME andPATH:export JAVA_HOME=/opt/java/jdk1.8.0_151 export PATH=$JAVA_HOME/bin:$PATH

Run the source .bash_profile command for the modification to takeeffect.

Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), becauseECS's default JDK may not be suitable for the sample project. Obtain Oracle JDK1.8.111 or later from Oracle's official website.

c. Download an open-source Kafka client at https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz for an instance of Kafka 1.1.0.wget https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz

d. Run the following command to decompress the package:tar -zxf [kafka_tar]In the preceding command, [kafka_tar] indicates the name of the clientpackage.For example:tar -zxf kafka_2.11-1.1.0.tgz

Accessing the Instance in CLI Mode

Step 1 Log in to the Linux ECS.

Step 2 Run the following commands to create messages:

./kafka-console-producer.sh --broker-list [connection-address] --topic [topic-name]

Parameter description:

● [connection-address] is the address obtained in 2.● [topic-name] is the name of the topic created for the Kafka instance.

The following example uses connection addresses10.3.196.45:9094,10.78.42.127:9094,10.4.49.103:9094. After running thepreceding command, you can send a message to the Kafka instance by writing itand pressing Enter. Each line of content is sent as a message.

[root@ecs-heru bin]# ./kafka-console-producer.sh --broker-list 10.3.196.45:9094,10.78.42.127:9094,10.4.49.103:9094 --topic topic-heru>Hello>DMS>Kafka!>^C[root@ecs-heru bin]#

To stop creating messages, press Ctrl+C to exit.

Step 3 Run the following commands to retrieve messages:

./kafka-console-consumer.sh --bootstrap-server [connection-address] --topic[topic-name] --from-beginning[root@ecs-heru bin]# ./kafka-console-consumer.sh --bootstrap-server 10.3.196.45:9094,10.78.42.127:9094,10.4.49.103:9094 --topic topic-heru --from-beginning



https://www.oracle.com/technetwork/java/javase/downloads/index.html

https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz

https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz

Kafka!DMSHello^CProcessed a total of 3 messages[root@ecs-heru bin]#

To stop retrieving messages, press Ctrl+C to exit.

----End

4.2 Accessing a Kafka Premium Instance with SASLIf you enable SASL_SSL when creating an instance, data will be encrypted beforetransmission for enhanced security.

This section describes how to use an open-source Kafka client to access a Kafkapremium instance if SASL has been enabled for the instance.

● The Kafka server allows a maximum of 200 clients to be connected using a single IPaddress. If the number of clients exceeds 200, the connection fails.

● When accessing a Kafka instance with SASL, map hosts to IP addresses to facilitateinstance broker domain name resolution. Otherwise, latency may occur.Set the IP address to the connection address of the instance. Set hosts to the names ofinstance hosts. Specify a unique name for each host.Examples:10.154.48.120 server0110.154.48.121 server0210.154.48.122 server03

Prerequisites1. Security group rules have been correctly configured.

A Kafka premium instance with SASL enabled can be accessed within a VPCor over public networks.– For intra-VPC access, ensure that the ECS and the Kafka premium

instance are in the same VPC and that security group rules have beencorrectly configured for the instance. For details on how to configure asecurity group, see How to Select and Configure a Security Group?

– For public access, there are no specific requirements on the VPC.However, you still need to ensure that security group rules have beencorrectly configured for the Kafka premium instance.To access the instance with SASL, allow inbound access through port9095.



https://support.huaweicloud.com/en-us/kafka_faq/kafka-faq-180604024.html

Figure 4-4 Security group rules configured for the Kafka instance (withSASL enabled)

2. The instance connection address has been obtained.– For intra-VPC access, use port 9093. Obtain the instance connection

address on the instance details page.

Figure 4-5 Obtaining Kafka instance connection addresses for intra-VPCaccess with SASL enabled

– For public access, use port 9095. Obtain the instance access address onthe instance details page.

Figure 4-6 Obtaining Kafka instance connection addresses for publicaccess with SASL enabled

3. At least one topic has been created for the Kafka premium instance.4. The client.truststore.jks certificate has been installed. The certificate can be

obtained by performing the following procedure.Download the certificate from https://dms-demo.obs.myhuaweicloud.com/cert.zip. Then, decompress the package to obtain the client certificate fileclient.truststore.jks.

5. The ECS has been correctly configured and an open-source Kafka client hasbeen installed. If the configuration or the installation has not been completed,perform the following procedure.

a. Log in to the ECS.The following uses a Linux ECS as an example. For more information onhow to install JDK and configure the environment variables for aWindows ECS, please search the Internet.

b. Install JDK or JRE, and add the following lines to .bash_profile in thehome directory to configure the environment variables JAVA_HOME andPATH:export JAVA_HOME=/opt/java/jdk1.8.0_151 export PATH=$JAVA_HOME/bin:$PATH



https://dms-demo.obs.myhuaweicloud.com/cert.zip

https://dms-demo.obs.myhuaweicloud.com/cert.zip

Run the source .bash_profile command for the modification to takeeffect.

Use Oracle JDK instead of ECS's default JDK (for example, OpenJDK), becauseECS's default JDK may not be suitable for the sample project. Obtain Oracle JDK1.8.111 or later from Oracle's official website.

c. Run the following command to decompress the package:tar -zxf [kafka_tar]In the preceding command, [kafka_tar] indicates the name of the clientpackage.For example:tar -zxf kafka_2.11-1.1.0.tgz

Accessing the Instance in CLI Mode

Step 1 Log in to the Linux ECS.

Step 2 Add the following lines to the consumer.properties and producer.properties files:sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \username="**********" \password="**********"; sasl.mechanism=PLAIN

security.protocol=SASL_SSLssl.truststore.location=/opt/kafka_2.11-1.1.0/config/client.truststore.jksssl.truststore.password=dms@kafkassl.endpoint.identification.algorithm=

In the preceding lines, username and password indicate the username andpassword specified when SASL_SSL is enabled, and ssl.trustore.location indicatesthe path for storing the certificate file. ssl.truststore.password is the servercertificate password which must be set to dms@kafka and cannot bechanged.

Step 3 Access the [base_dir]/kafka_2.11-1.1.0/bin directory.

[base_dir] indicates the installation directory of the Kafka client.

Step 4 Run the following commands to create messages:

./kafka-console-producer.sh --broker-list [connection-address] --topic [topic-name] --producer.config ../config/producer.properties

Parameter description:

● [connection-address] is the address obtained in 2.● [topic-name] is the name of the topic created for the Kafka instance.

The following example uses connection addresses 10.xxx.xxx.202:9095,10.xxx.xxx.197:9095,10.xxx.xxx.68:9095.

After running the preceding command, you can send a message to the Kafkainstance by writing it and pressing Enter. Each line of content is sent as amessage.

[root@ecs-heru bin]#./kafka-console-producer.sh --broker-list 10.xxx.xxx.202:9095,10.xxx.xxx.197:9095,10.xxx.xxx.68:9095 --topic topic-heru --producer.config ../config/producer.properties



https://www.oracle.com/technetwork/java/javase/downloads/index.html

>hello>DMS>Kafka!>^C[root@ecs-heru bin]#

To stop creating messages, press Ctrl+C to exit.

Step 5 Run the following commands to retrieve messages:

./kafka-console-consumer.sh --bootstrap-server [connection-address] --topic[topic-name] --from-beginning --consumer.config ../config/consumer.properties

[root@ecs-heru bin]# ./kafka-console-consumer.sh --bootstrap-server 10.xxx.xxx.202:9095,10.xxx.xxx.197:9095,10.xxx.xxx.68:9095 --topic topic-heru --from-beginning --consumer.config ../config/consumer.propertieshelloKafka!abcdDMSheuuhello^CProcessed a total of 6 messages[root@ecs-heru bin]#

To stop retrieving messages, press Ctrl+C to exit.

----End

4.3 Connecting to Kafka Manager and Viewing KafkaInformation

Kafka Manager is an open-source tool for managing Kafka, which can be usedonly through a web browser. In Kafka Manager, you can view the monitoringstatistics and broker information about your Kafka clusters.

If you did not enable public access for your Kafka premium instance, you mustaccess Kafka Manager using an ECS that runs Windows and has the same VPCand security group configurations as the instance.

If you have enabled public access for your Kafka premium instance, you can visitKafka Manager through a web browser.

In the following example, public access is not enabled for the Kafka premiuminstance.

Logging In to Kafka Manager

Step 1 Create a Windows ECS with the same VPC and security group configurations asthe Kafka premium instance available. For details, see Purchasing and Logging Into a Windows ECS.

Step 2 Obtain the Kafka Manager address.

The address can be obtained on the instance details page as shown in Figure 4-7.



https://support.huaweicloud.com/en-us/qs-ecs/en-us_topic_0021831611.html

https://support.huaweicloud.com/en-us/qs-ecs/en-us_topic_0021831611.html

Figure 4-7 Obtaining the Kafka Manager address

Step 3 Enter the Kafka Manager address in the web browser in the Window ECS.

Step 4 Enter the username and password for logging in to Kafka Manager, which you setwhen creating the instance.

----End

Viewing Information in Kafka ManagerIn Kafka Manager, you can view the monitoring statistics and broker informationabout your Kafka clusters.

● Information about clustersClick Clusters to view the information about clusters Figure 4-8 shows anexample of the storage configuration.– The top navigation bar provides the following functions, as shown in the

red box 1 in the figure.

▪ Cluster: Viewing the list of clusters and cluster information.

▪ Brokers: Viewing information about brokers of a cluster.

▪ Topic: Viewing information about topics in a cluster.

▪ Preferred Replica Election: Electing the leader (preferred replica) ofa topic. This operation is not recommended.

▪ Reassign Partitions: Reassigning partitions. This operation is notrecommended.

▪ Consumers: Viewing the status of consumer groups in a cluster.

– Red box 2 shows an example of the cluster information summary,including the number of topics and brokers in the cluster.



Figure 4-8 Information about clusters

● Combined information about all brokers of a clusterThis page shows statistics of brokers of a cluster. Figure 4-9 shows anexample of the storage configuration.– Red box 1 shows the list of brokers, including number of incoming and

outgoing bytes of different brokers.– Red box 2 shows the monitoring metrics of the cluster.

Figure 4-9 Viewing the combined information about all brokers of acluster

● Information about a specific brokerClick the ID of a broker to view its statistics. Figure 4-10 shows an example ofthe storage configuration.– Red box 1 shows the statistics of the broker, including the numbers of

topics, partitions, and leaders, and percentages of messages, incomingtraffic and outgoing traffic.

– Red box 2 shows the monitoring metrics of the broker.



Figure 4-10 Viewing information about a broker

● Topics of an instanceIn the navigation bar, choose Topic > List. The displayed page shows the listof topics and information about the topics, as shown in Figure 4-11.

Topics starting with "__" are internal topics. Do not perform any operation onthese topics to avoid service faults.

Figure 4-11 Topics of an instance

● Details of a topic



Click the name of a topic to view its details on the displayed page, as shownin Figure 4-12.– Red box 1: Basic information about the topic, including Replication,

Number of Partitions, and Sum of Partition Offsets.– Red box 2: Information about partitions of different brokers.– Red box 3: Consumer groups of the topic. Click the name of a consumer

group name to view its details.– Red box 4: Configurations of the topic. For details, see http://

kafka.apache.org/documentation/#topicconfigs.– Red box 5: Monitoring metrics of the topic.– Red box 6: Information about partitions in the topic, including Latest

Offset, Leader of a partition, Replicas, and In Sync Replicas.

Figure 4-12 Details of a topic

● List of consumersClick the Consumers tab to view the list of consumers in a cluster.

Figure 4-13 Viewing the list of consumers



http://kafka.apache.org/documentation/#topicconfigs

http://kafka.apache.org/documentation/#topicconfigs

● Details of a specific consumerClick the name of a consumer to view its details, including the list of topics inthe consumer and the number of messages that can be retrieved in each topic(Total Lag).

Figure 4-14 Viewing consumer details

● Details of topics in a consumerClick the name of a topic to view retrieval details of different partitions in thetopic, including Partition, the number of messages in a partition (LogSize),progress of the retrieval (Consumer Offset), number of remaining messagesin the partition that can be retrieved (Lag), and the latest consumer thatretrieved from the partition (Consumer Instance Owner).

Figure 4-15 Viewing details of a topic



5 Operating Kafka Premium Instances

5.1 Viewing an Instance

5.2 Restarting an Instance

5.3 Deleting an Instance

5.4 Modifying the Information About an Instance

5.5 Configuring Public Access

5.6 Modifying Instance Specifications

5.7 Changing an Instance to Yearly/Monthly Billing

5.8 Creating a Topic in a Kafka Premium Instance

5.9 Deleting a Topic in a Kafka Premium Instance

5.10 Resetting Kafka Password

5.11 Resetting Kafka Manager Password

5.12 Querying Messages

5.13 Enabling Dumping

5.14 Managing Tags

5.15 Viewing Background Tasks

5.1 Viewing an Instance

Scenario

View detailed information about a Kafka premium instance on the Kafka console,for example, the IP address and port number for accessing the instance.

Prerequisites

A Kafka premium instance has been created.

Distributed Message Service for KafkaUser Guide 5 Operating Kafka Premium Instances


Procedure





Step 4 Search for a Kafka premium instance by specifying the status and name. Table 5-1describes the various possible statuses of a Kafka premium instance.

Table 5-1 Kafka premium instance status description

Status Description

Creating The instance is being created.

Running The instance is running properly.Only Kafka premium instances whose status is Running canprovide the DMS for Kafka service.

Faulty The instance is not running properly.

Starting The status between Frozen and Running.

Restarting The instance is being restarted.

Changing The instance is in the process of changing specifications, enablingdumping, or enabling public access.

Changefailed

The instance specifications failed to be changed, or dumping orpublic access failed to be enabled.

Frozen The instance is frozen.

Freezing The status between Running and Frozen.

Creationfailed

The instance failed to be created.

Step 5 Click the name of the chosen Kafka premium instance and view detailedinformation about the instance on the displayed page.

Table 5-2 describes the parameters for connecting to a Kafka instance. For detailsabout other parameters, see Basic Information tab page of the Kafka instance onthe console.



Table 5-2 Parameters for connecting to a Kafka instance

Section Parameter Description

PublicAccess

PublicAccess

Indicates whether to enable public access for theinstance. Click to modify the public networkfeature.

PublicNetworkBandwidth

Indicates the public network bandwidth of theinstance. This parameter is displayed only whenpublic access is enabled.

KafkaManagerAddress

Indicates the connection address of the KafkaManager.

KafkaAccessAddress

Indicates the public network connection address ofthe instance. This parameter is displayed only whenpublic access is enabled.

Connection Address

IPV4 Indicates the address for accessing the instance in aVPC.

----End

5.2 Restarting an Instance

Scenario

Restart one or more Kafka premium instances at a time on the Kafka console.

When a Kafka premium instance is being restarted, message retrieval and creationrequests of the client will be rejected.

Prerequisites

The status of the Kafka premium instance you want to restart is either Running orFaulty.

Procedure







Step 4 Select one or more Kafka premium instances in the instance list.

Step 5 Click Restart on the top of the instance list.

Step 6 Click Yes.

It takes 3 to 15 minutes to restart a Kafka premium instance. After it issuccessfully restarted, the Kafka premium instance status should be Running.

Restarting a Kafka premium instance only restarts the instance process and does not restartthe VM where the instance is located.

To restart a Kafka premium instance, you can also choose Operation > Restart in the samerow as the chosen Kafka premium instance on the Kafka Premium page.

----End

5.3 Deleting an Instance

ScenarioDelete one or more Kafka premium instances at a time. With a few clicks on theKafka console, you can delete multiple Kafka premium instances that failed to becreated.

Deleting a Kafka premium instance will delete the data in the instance withoutany backup. Exercise caution when performing this operation.

Prerequisites● The Kafka premium instance you want to delete exists.● The status of the Kafka premium instance you want to delete is in the

Running, Faulty, or Frozen state.● Kafka premium instances in yearly/monthly billing mode cannot be deleted.

To disable such a Kafka premium instance, choose Operation > More >Unsubscribe in the same row as the instance.

Deleting a Kafka Premium Instance







Step 4 Select one or more Kafka premium instances in the instance list.

Kafka premium instances in the Creating, Starting, Changing, Change failed, orRestarting state cannot be deleted.

Step 5 Click Delete on the top of the instance list.

Step 6 Click Yes.

It takes 1 to 60 seconds to delete a Kafka premium instance.

To delete a Kafka premium instance, you can also choose Operation > Delete in the samerow as the chosen Kafka premium instance on the Kafka Premium page.

----End

Deleting a Kafka Premium Instance That Failed to be Created





If there are Kafka premium instances that failed to be created, Instance CreationFailures and quantity information will be displayed.

Step 4 Click the icon or quantity next to Instance Creation Failures.

The Instance Creation Failures page is displayed.

Step 5 Delete Kafka premium instances that failed to be created in either of the followingways:● To delete all Kafka premium instances that failed to be created at once, click

Clear Failed Instance.● To delete a single Kafka premium instance that failed to be created, click

Delete in the same row as the chosen Kafka premium instance.

----End

5.4 Modifying the Information About an InstanceAfter creating a Kafka premium instance, you can modify some parameters of theinstance based on service requirements, including the instance name, description,maintenance time window, security group, and capacity threshold policy.



Prerequisites

A Kafka premium instance has been created.

Procedure





Step 4 Click the name of a Kafka premium instance for which you want to modify itsinformation.

Step 5 Click behind a parameter to modify it.

You can modify the following parameters:

● Instance Name● Time Window● Enterprise Project● Description● Security Group● Public Access● Capacity Threshold Policy● Automatic Topic Creation

For details about how to change the public access configuration, see 5.5 ConfiguringPublic Access.

Step 6 Click to save the modification.

If Capacity Threshold Policy has been modified, you will be redirected to theBackground Task Management page, which displays the modification progressand result.

If Instance Name, Description, Time Window, or Security Group has beenmodified, the modification result will be displayed on the upper right corner of thepage.

Click to undo modification.

----End



5.5 Configuring Public AccessTo access a Kafka instance over a public network, you can enable public accessand configure public network bandwidth for the instance. If you no longer needpublic access to the instance, you can disable it as required.

Procedure





Step 4 In the navigation pane, choose Kafka Platinum.

Step 5 Click the name of an instance.

Step 6 In the Public Access section, click .

The Change Public Network Bandwidth page is displayed.

Step 7 Perform the following operations as required:● Enable public access

Click to enable public access and set the bandwidth.

– You can enable public access for a maximum of three instances. If you want toenable public access for more instances, contact customer service to increase yourquota.

– If you have enabled and disabled public access before, the public access addresswill be different when you enable public access again.

– Bandwidth value range:

▪ When the instance specification is 100 MB/s, the public network bandwidthmust be a multiple of 3 in the range from 3 to 900.

▪ When the instance specification is 300 MB/s, the public network bandwidthmust be a multiple of 3 in the range from 3 to 900.

▪ When the instance specification is 600 MB/s, the public network bandwidthmust be a multiple of 4 in the range from 4 to 1,200.

▪ When the instance specification is 1,200 MB/s, the public network bandwidthmust be a multiple of 8 in the range from 8 to 2,400.

● Disable public access

Click to disable public access.



● Modify public network bandwidthNext to Bandwidth (Mbit/s), slide the bar or enter a number in the text boxto set the bandwidth.

– The public network bandwidth can only be changed to the higher value.– During the public network bandwidth expansion, a few services may fail. You are

advised to perform this operation during off-peak hours.

Step 8 Click Submit to save the changes.

A message is displayed, indicating that the task is successfully submitted. You canview the operation progress on the Background Task Management page. If thetask status is Successful, the modification has succeeded.

----End

5.6 Modifying Instance Specifications

ScenarioAfter creating a Kafka premium instance, you can increase the storage space andbandwidth of the instance. The storage space can be expanded for a maximum of20 times.

NO TICE

● To ensure that the instance runs properly, do not perform other operations onthe instance during the increase.

● The price may change after the specifications of a pay-per-use instance aremodified.

● After you successfully change the specification, the instance is billed based onnew specifications.

● The storage space and bandwidth cannot be expanded at the same time.● During bandwidth increase, the storage space is proportionally expanded based

on the current disk space.

PrerequisitesA Kafka premium instance has been created and is in the Running state.

Procedure







Step 4 In the same row as the Kafka premium instance for which you want to modify thespecifications, choose More > Modify Specifications.

Step 5 Specify the required storage space or bandwidth.● Specify a new storage space and click Submit.

View the new storage space in the Used/Available Storage Space (GB)column on the Kafka Premium page.

Available storage space = Actual storage space – Storage space for storing logs andZooKeeper data – Disk formatting lossFor example, if the storage space is expanded to 700 GB, the storage space for storinglogs and ZooKeeper data is 100 GB, and the disk formatting loss is 7 GB, then theavailable storage space after capacity expansion will be 593 GB.

● Specify a new bandwidth and click Submit.View the new Bandwidth of the instance on the instance details page.

– Bandwidth is increased by scaling out nodes. The original nodes and services arenot affected.

– Only new topics are created in new nodes. After the bandwidth is increased, theoriginal topics are still in the original node. If you want to use new nodes, you canmigrate the topics or create new topics by using Kafka Manager.

----End

5.7 Changing an Instance to Yearly/Monthly BillingInstances billed in the pay-per-use mode can be changed to the yearly/monthlybilling mode.

Procedure





Step 4 In the same row as the chosen instance, click More > Change to Yearly/MonthlyBilling.

To change multiple instances to yearly/monthly billing in batches, select the choseninstances in the same state and click Change to Yearly/Monthly Billing above the instancelist.



Step 5 Click Yes.

Step 6 Specify the required duration and click Submit.

----End

5.8 Creating a Topic in a Kafka Premium InstanceAfter creating a Kafka premium instance, you can create a topic in the instance.

Prerequisites● A Kafka premium instance has been created.● The Kafka premium instance is in the Running state.

Procedure





Step 4 Click the name of a Kafka premium instance for which you want to create a topic.

The instance details page is displayed.

Step 5 Click the Topic Management tab, and click Create Topic.

The Create Topic page is displayed.

Step 6 Specify the topic parameters listed in the following table.

Table 5-3 Topic parameters

Parameter Description

Topic Name When you create a topic, a default topic name is generated,which you can change as required. A topic name contains 4to 64 characters. Only letters, digits, underscores (_), andhyphens (-) are allowed.The topic name cannot be changed after a topic is created.

Partitions A larger number of partitions for a topic indicates moremessages retrieved concurrently.If this parameter is set to 1, messages will be retrieved in theFIFO order.Value range: 1 to 20Default value: 3



Parameter Description

Replicas A higher number of replicas delivers higher reliability. Data isautomatically backed up on each replica. After one Kafkabroker becomes faulty, data is still available on other brokers.If this parameter is set to 1, only one set of data is available.Value range: 1 to 3Default value: 3

MessageRetention Period(h)

The number of hours for which messages will be preserved ina topic. Messages older than that period will be deleted anddeleted messages are not retrievable to consumer groups.Value range: 1 to 168Default value: 72

SynchronousReplication

A message is returned to the client only after the messagecreation request has been received and the message hasbeen acknowledged by all replicas.After enabling synchronous replication, configure acks = -1on the client. Otherwise, this function will not take effect.If there is only one replica, synchronous replication cannot beenabled.

SynchronousFlushing

An indicator of whether a message is immediately flushed todisk once it is created.Enabled: A message is immediately flushed to disk once it iscreated, featuring higher reliability.Disabled: A message is stored in the memory instead ofbeing immediately flushed to disk once it is created.

Step 7 Click OK.

----End

5.9 Deleting a Topic in a Kafka Premium Instance

Prerequisites● A Kafka premium instance has been created, and a topic has been created in

this instance.

● The Kafka premium instance is in the Running state.

Procedure







Step 4 Click the name of a Kafka premium instance for which you want to delete a topic.

The instance details page is displayed.

Step 5 Click the Topic Management tab.

Step 6 Select one or more topics in the topic list.

Step 7 Click Delete Topic on the top of the topic list.

To delete a topic, you can also choose Operation > Delete in the same row as the chosentopic on the Topic Management page.

Step 8 Click Yes.

----End

5.10 Resetting Kafka Password

ScenarioIf you forget the password of a Kafka premium instance, you can reset thepassword.

● You can reset the password of a Kafka premium instance only if Kafka SASL_SSL hasbeen enabled for the instance.

● You can only reset the password of a Kafka premium instance in the Running state.

PrerequisitesA Kafka premium instance has been created.

Procedure





Step 4 Choose More > Reset Kafka Password in the same row as the Kafka premiuminstance for which you want to reset the password.



Step 5 In the displayed Reset Kafka Password dialogue box, enter a new password andconfirm the password.

Passwords for Kafka premium instances must meet the following requirements:

● Cannot be empty.● Must be 8 to 32 characters long.● Must contain at least three of the following character types:

– Lowercase letters– Uppercase letters– Digits– Special characters (`~!@#$%^&*()-_=+\|[{}]:'",<.>/?)– Cannot start with a hyphen (-).

Step 6 Click OK.● If the password is successfully reset, a success message will be displayed.● If the password failed to be reset, a failure message will be displayed. Reset

the password again. If you still fail to reset the password after multipleattempts, contact customer service.

The system will display a success message only after the password is successfully reset onall nodes.

----End

5.11 Resetting Kafka Manager Password

ScenarioIf you forget the password of Kafka Manager of a Kafka premium instance, youcan reset the password.

You can reset the password of Kafka Manager only for a Kafka Premium instance in theRunning state.

PrerequisitesA Kafka premium instance has been created.

Procedure







Step 4 Choose More > Reset Manager Password in the same row for which you want toreset the Kafka Manager password.

Step 5 Enter a new password and the confirm password.

Step 6 Click OK.

● If the password is successfully reset, a success message will be displayed.

● If the password failed to be reset, a failure message will be displayed. Resetthe password again. If you still fail to reset the password after multipleattempts, contact customer service.

The system will display a success message only after the password is successfully reset onall nodes.

----End

5.12 Querying Messages

Scenario

You can view the size, creation time, and body of messages in topics.

Procedure






Step 5 Click the Message Query tab. Then specify the topic name and the searchmethod.

You can search by the following methods:

● Creation time: Search by the time that messages are created.

● Offset: Search by the message position.

The following shows messages that have been queried.



Figure 5-1 Querying topic messages

Click View Message Body to view details about a message.

----End

5.13 Enabling Dumping

Scenario

After you enable dumping, data will be dumped to OBS buckets for backup.

● If a message contains a key-value attribute, the key cannot be dumped while the valuecan be dumped.

● You cannot disable dumping once it has been enabled.

Prerequisites

A Kafka premium instance has been created and is in the Running state.

Procedure





Step 4 Choose More > Enable Dumping in the Operation column of a Kafka premiuminstance for which you want to enable dumping.

● For instances with dumping enabled, the Enable Dumping button is not available.

● You can also open the instance details page and click the Dumping Task Managementtab to enable the dump function.



Step 5 On the displayed Enable Dumping for Kafka Premium Instance page, ensurethat Dumping is enabled and click Submit.

Step 6 On the instance details page, click the Dumping Task Management tab.

Step 7 Click Create Dumping Task.

Step 8 Specify the parameters as shown in Table 5-4.

Table 5-4 Parameters for creating a dumping task

Parameter Description Remarks

Source DataType

Type of the sourcedata. Currently, onlyBLOB is supported.Retain the defaultvalue.

-

DumpingService

Service to which datais dumped. Currently,you can only dumpdata to OBS buckets.Retain the defaultvalue.

-

Task Name Name of a dumpingtask.

Used to identify dumping tasks.

Topics Name of the topicswhose data you wantto dump.

If Regular expression is selected, aregular expression is used tosubscribe to topics whose messagesyou want to dump.

Dump FileFormat

Format of the dumpfile. Currently, onlyTEXT is supported.Retain the defaultvalue.

-

Offset ● Maximum offset,which meansdumping thelatest data.

● Minimum offset,which meansdumping theearliest data.

-

Dumping Period(s)

Interval forperiodically dumpingdata. The time unit issecond and thedefault value is 300seconds.

No package files will be generated ifthere is no data within a timesegment.



Parameter Description Remarks

AK Access key ID. For details about how to obtain theAK, see Access Keys.

SK Secret access keyused together withthe access key ID.

For details about how to obtain theSK, see Access Keys.

DumpingAddress

Name of the OBSbucket used to storethe topic data.

Click Select and select an OBSbucket.

DumpingDirectory

Directory for storingtopic data in theOBS.

-

Time DirectoryFormat

Format of the lower-level directory forstoring topic data inthe OBS bucket. Forexample, the storagedirectory may be"bucket_name/dump_file_directory/year/month/day".

If N/A is selected, time is not used inthe storage directory.

RecordSeparator

Select a separator toseparate OBSdumping records.

-

Step 9 Click OK.

----End

5.14 Managing TagsTags facilitate Kafka instance identification and management.

You can add tags to a Kafka instance when creating the instance or add tags onthe details page of the created instance. Up to 10 tags can be added to aninstance. Tags can be modified and deleted.

A tag consists of a tag key and a tag value. Table 5-5 lists the tag key and valuerequirements.



https://support.huaweicloud.com/en-us/usermanual-ca/en-us_topic_0046606340.html

https://support.huaweicloud.com/en-us/usermanual-ca/en-us_topic_0046606340.html

Table 5-5 Tag key and value requirements

Name Requirements

Tag key ● Cannot be left blank.● Must be unique for the same

instance.● Can contain a maximum of 36

characters.● Cannot contain the following

characters: =*<>\,|/● Cannot start or end with a space.

Tag value ● Can contain a maximum of 43characters.

● Cannot contain the followingcharacters: =*<>\,|/

● Cannot start or end with a space.

Procedure






Step 5 Click the Tags tab.

View the tags of the instance.

Step 6 Perform the following operations as required:● Add a tag

a. Click Add Tag.If you have created predefined tags, select a predefined pair of tag keyand value. To view or create predefined tags, click View predefined tags.Then you will be directed to the TMS console.You can also create new tags by specifying Tag key and Tag value.

b. Click OK.● Modify a tag

In the same row as the tag to be modified, choose Operation > Edit. Enterthe new tag value and click OK.



● Delete a tagIn the same row as the tag to be deleted, choose Operation > Delete. Thenclick Yes.

----End

5.15 Viewing Background TasksAfter you initiate certain instance operations such as configuring public access andmodifying the capacity threshold capacity, a background task will start for eachoperation. On the console, you can view the background task status and clear taskinformation by deleting task records.

Procedure





Step 4 Click the name of the Kafka premium instance. The Basic Information page isdisplayed.

Step 5 Click the Background Task Management tab.

A list of background tasks is displayed.

Step 6 Click , specify Start Date and End Date, and click OK to view tasks started inthe corresponding time segment.

● Click to refresh the task status.● To clear the record of a background task, choose Operation > Delete.

You can only delete the records of tasks in the Successful or Failed state.

----End



6 Monitoring

6.1 Kafka Metrics

6.2 Configuring Alarms

6.3 Viewing Metrics

6.1 Kafka Metrics

IntroductionThis section describes DMS metrics reported to Cloud Eye as well as theirnamespaces and dimensions. You can use the Cloud Eye console or APIs to querythe DMS metrics and alarms.

NamespaceSYS.DMS

Distributed Message Service for KafkaUser Guide 6 Monitoring


https://support.huaweicloud.com/en-us/api-ces/en-us_topic_0171212514.html

Kafka Premium Instance Metrics

Table 6-1 Metrics of Kafka premium instances

MetricID

Metric Description ValueRange

MonitoredObjectandDimension

MonitoringPeriod(RawData)

current_partitions

Partitions

Number of used partitions inthe instanceUnit: Count

0–18,000 Monitoredobject:DMSDimension:kafka_instance_id

1minute

current_topics

Topics Number of created topics inthe instanceUnit: Count

0–600 Monitoredobject:DMSDimension:kafka_instance_id

1minute

group_messages

AccumulatedMessages

Total number of accumulatedmessages in all consumergroups of the instanceUnit: Count

0–1,000,000,000

Monitoredobject:DMSDimension:kafka_instance_id

1minute



MetricID




broker_bytes_in_rate

MessageCreation

Number of bytes created persecondUnit: byte/s

0–500,000,000

Monitoredobject:DMSDimension:kafka_broker

1minute

broker_bytes_out_rate

MessageRetrieval

Number of bytes retrievedper secondUnit: byte/s

0–500,000,000


1minute

broker_data_size

Message Size

Total size of messages in thenodeUnit: byte

0–5,000,000,000,000


1minute

broker_fetch_mean

AverageMessageRetrievalProcessingDuration

Average time that the brokerspends processing messageretrieval requestsUnit: ms

0–10,000 Monitoredobject:DMSDimension:kafka_broker

1minute



MetricID




broker_produce_mean

AverageMessageCreationProcessingDuration

Average time that the brokerspends processing messagecreation requestsUnit: ms


1minute

broker_messages_in_rate

MessageCreation Rate

Number of messages createdper secondUnit: count/s


1minute

broker_public_bytes_in_rate

PublicInboundTraffic

Inbound traffic over publicnetworks per secondUnit: byte/s

0–500,000,000


1minute

broker_public_bytes_out_rate

PublicOutboundTraffic

Outbound traffic over publicnetworks per secondUnit: byte/s

0–500,000,000


1minute



MetricID




topic_bytes_in_rate

MessageCreation

Number of bytes created persecond.Unit: byte/sNOTE

This metric is available onlywhen Scope is set to Basicmonitoring on the Queues tabpage.

0–500,000,000

Monitoredobject:DMSDimension:kafka_topics

1minute

topic_bytes_out_rate

MessageRetrieval

Number of bytes retrievedper secondUnit: byte/sNOTE


0–500,000,000


1minute

topic_data_size

Message Size

Total size of messages in thequeueUnit: byteNOTE


0–5,000,000,000,000


1minute

topic_messages

TotalMessages

Total number of messages inthe queueUnit: CountNOTE


≥ 0 Monitoredobject:DMSDimension:kafka_topics

1minute



MetricID




topic_messages_in_rate

MessageCreation Rate

Number of messages createdper secondUnit: count/sNOTE


0–500,000 Monitoredobject:DMSDimension:kafka_topics

1minute

produced_messages

PartitionMessages

Total number of messages inthe partitionUnit: CountNOTE

This metric is available onlywhen Scope is set to Partitionmonitoring on the Queues tabpage.

≥ 0 Monitoredobject:DMSDimension:kafka_partitions

1minute

partition_messages

CreatedMessages

Number of messages thathave been createdUnit: CountNOTE

This metric is available onlywhen Scope is set to Partitionmonitoring on the Queues tabpage.

≥ 0 Monitoredobject:DMSDimension:kafka_partitions

1minute

messages_consumed

RetrievedMessages

Number of messages thathave been retrieved in theconsumer groupUnit: Count

≥ 0 Monitoredobject:DMSDimension:kafka_groups-partitions

1minute



MetricID




messages_remained

AvailableMessages

Number of remainingmessages that can beretrieved in the consumergroupUnit: Count

≥ 0 Monitoredobject:DMSDimension:kafka_groups-partitions

1minute

topic_messages_remained

TopicAvailableMessages

Number of remainingmessages that can beretrieved from the specifiedtopic in the consumer groupUnit: Count

0 to 263–1 Monitoredobject:DMSDimension:kafka_groups_topics

1minute

topic_messages_consumed

TopicRetrievedMessages

Number of messages thathave been retrieved from thespecified topic in theconsumer groupUnit: Count

0 to 263–1 Monitoredobject:DMSDimension:kafka_groups_topics

1minute

consumer_messages_remained

ConsumerAvailableMessages

Number of remainingmessages that can beretrieved in the consumergroupUnit: Count

0 to 263–1 Monitoredobject:DMSDimension:kafka_groups

1minute



MetricID




consumer_messages_consumed

ConsumerRetrievedMessages

Number of messages thathave been retrieved in theconsumer groupUnit: Count

0 to 263–1 Monitoredobject:DMSDimension:kafka_groups

1minute

broker_cpu_core_load

Average LoadperCPUCore

Average load of each CPUcore of the Kafka VM

0–20 Monitoredobject:DMSDimension:kafka_broker

1minute

broker_disk_usage

DiskCapacityUsage

Disk usage of the Kafka VMUnit: %


1minute

broker_memory_usage

MemoryUsage

Memory usage of the KafkaVMUnit: %


1minute



MetricID




broker_heap_usage

JVMHeapMemoryUsageofKafka

Heap memory usage of theKafka JVMUnit: %


1minute

DimensionKey Value

kafka_instance_id Kafka premium instances

kafka_broker Kafka premium instance brokers

kafka_topics Topics in Kafka premium instances

kafka_partitions Partitions in Kafka premium instances

kafka_groups-partitions Partition consumer groups in Kafkapremium instances

kafka_groups_topics Topic consumer groups in Kafkapremium instances

kafka_groups Consumer groups in Kafka premiuminstances

6.2 Configuring AlarmsThis section describes the alarm policies of some metrics and how to configure thepolicies. In actual services, you are advised to configure alarm rules for metricsbased on the following alarm policies:



Approach Upper Limit indicates whether the performance of the current resource is closeto the upper limit. If the performance is close to the upper limit, the performancesupported by the current resource is the alarm threshold set in the alarm policy. If theperformance continues to increase, services may be abnormal.

Table 6-2 Configuring alarm metrics for a Kafka instance

Metric NormalRange

AlarmPolicy

Approach UpperLimit

Description

MemoryUsage

0-100 Alarmthreshold:originalvalue > 85Number ofconsecutive periods:3Alarmseverity:critical

Yes Memory usage of the KafkaVMUnit: %

DiskCapacityUsage


Yes Disk usage of the Kafka VMUnit: %

AverageLoad perCPUCore

0-2 Alarmthreshold:originalvalue > 1.5Number ofconsecutive periods:3Alarmseverity:major

Yes Average load of each CPU coreof the Kafka VM



Metric NormalRange

AlarmPolicy

Approach UpperLimit

Description

JVMHeapMemoryUsage ofKafka


Yes Heap memory usage of theKafka JVMUnit: %

Procedure





Step 4 In the same row as the instance for which you want to view the metrics, chooseMore > View Metric.

You are redirected to the Cloud Eye console page displaying metrics of theselected instance.

Step 5 Hover over a metric and click to create an alarm rule for the metric.

The Create Alarm Rule page is displayed.

Step 6 Specify the alarm details.

For more information about creating alarm rules, see Creating an Alarm Rule.

1. Specify the alarm policy and alarm severity.As shown in the following figure, if the original memory usage exceeds thepreset value for three consecutive periods, an alarm is generated. If the alarmis not handled in a timely manner, an alarm notification is sent every day.



https://support.huaweicloud.com/en-us/usermanual-ces/en-us_topic_0084572213.html

Figure 6-1 Setting the alarm policy and alarm severity

2. Set the alarm notification configurations. If you enable Alarm Notification,set the validity period, notification object, and trigger condition.

3. Click Next to set the alarm name and description.4. Click Finish.

----End

6.3 Viewing Metrics

Scenario

Cloud Eye monitors Kafka instance metrics in real time. You can view thesemetrics on the Cloud Eye console.

Prerequisites

At least one Kafka premium instance has been created. The instance has at leastone available message.

Procedure




Step 3 Click Service List, and choose Application > Distributed Message Service forKafka to open the Kafka console.

Step 4 Perform the following steps to view Kafka premium instance metrics:

1. In the navigation pane, choose Kafka Premium.2. In the same row as the instance for which you want to view the metrics,

choose More > View Metric.



On the Cloud Eye console, view the metrics of the instance, nodes, queues,and consumer groups. Metric data is reported to Cloud Eye every minute.

----End



7 Auditing

7.1 Operations That Can Be Recorded by CTS

7.2 Viewing Traces on the CTS Console

7.1 Operations That Can Be Recorded by CTSWith Cloud Trace Service (CTS), you can record operations associated with DMSfor later query, audit, and backtrack operations.

Table 7-1 DMS operations that can be recorded by CTS

Operation Resource Type Trace Name

Successfully deleting abackground task

Kafka deleteDMSBackendJobSuccess

Failing to delete abackground task

Kafka deleteDMSBackendJobFailure

Successfully creatingan order for creatingan instance

Kafka createDMSInstanceOrderSuccess

Failing to creating anorder for creating aninstance

Kafka createDMSInstanceOrderFailure

Successfullysubmitting a requestto modify an instanceorder

Kafka modifyDMSInstanceOrderSuc-cess

Failing to submit arequest to modify aninstance order

Kafka modifyDMSInstanceOrderFailure

Distributed Message Service for KafkaUser Guide 7 Auditing



Successfullysubmitting a requestto scale up aninstance

Kafka extendDMSInstanceSuccess

Failing to submit arequest to scale up aninstance

Kafka extendDMSInstanceFailure

Successfullysubmitting a requestto reset instancepassword

Kafka resetDMSInstancePasswordSuc-cess

Failing to submit arequest to resetinstance password

Kafka resetDMSInstancePasswordFai-lure

Successfully creating atopic for a Kafkapremium instance

Kafka Kafka_platinum_create_topicSuccess

Failing to create atopic for a Kafkapremium instance

Kafka Kafka_platinum_create_topicFailure

Successfully deleting atopic from a Kafkapremium instance

Kafka Kafka_platinum_delete_topicsSuccess

Failing to delete atopic for a Kafkapremium instance

Kafka Kafka_platinum_delete_topicsFailure

Successfully deletingan instance failed tobe created

Kafka deleteDMSCreateFailureInstan-cesSuccess

Failing to delete aninstance failed to becreated

Kafka deleteDMSCreateFailureInstan-cesFailure

Successfullysubmitting a requestto restart an instance

Kafka restartDMSInstanceSuccess

Failing to submit arequest to restart aninstance

Kafka restartDMSInstanceFailure

Successfullysubmitting a requestto delete multipleinstances at a time

Kafka batchDeleteDMSInstanceSuccess




Failing to submit arequest to deletemultiple instances ata time

Kafka batchDeleteDMSInstanceFailure

Successfullysubmitting a requestto restart multipleinstances at a time

Kafka batchRestartDMSInstanceSuc-cess

Failing to submit arequest to restartmultiple instances ata time

Kafka batchRestartDMSInstanceFailure

Successfullysubmitting a requestto modify instanceinformation

Kafka modifyDMSInstanceInfoSuccess

Failing to submit arequest to modifyinstance information

Kafka modifyDMSInstanceInfoFailure

Deleting multipleinstance tasks at atime

Kafka batchDeleteDMSInstanceTask

Successfully enablingdumping

Kafka createConnectorTaskSuccess

Failing to enabledumping

Kafka createConnectorTaskFailure

Successfully creating adumping task

Kafka createConnectorSinkTaskSuccess

Failing to create adumping task

Kafka createConnectorSinkTaskFailure

Successfullyunfreezing aninstance

Kafka unfreezeDMSInstanceTaskSuc-cess

Failing to unfreeze aninstance

Kafka unfreezeDMSInstanceTaskFai-lure

Successfully freezingan instance

Kafka freezeDMSInstanceTaskSuccess

Failing to freeze aninstance

Kafka freezeDMSInstanceTaskFailure

Successfully deletingan instance

Kafka deleteDMSInstanceTaskSuccess




Failing to delete aninstance

Kafka deleteDMSInstanceTaskFailure

Successfully creatingan instance

Kafka createDMSInstanceTaskSuccess

Failing to create aninstance

Kafka createDMSInstanceTaskFailure

Successfully scaling upan instance

Kafka extendDMSInstanceTaskSuccess

Failing to scale up aninstance

Kafka extendDMSInstanceTaskFailure

Successfully restartingan instance

Kafka restartDMSInstanceTaskSuccess

Failing to restart aninstance

Kafka restartDMSInstanceTaskFailure

Successfully restartingmultiple instances ata time

Kafka batchRestartDMSInstanceTask-Success

Failing to restartmultiple instances ata time

Kafka batchRestartDMSInstanceTask-Failure

Successfully modifyinginstance information

Kafka modifyDMSInstanceInfoTaskSuc-cess

Failing to modifyinstance information

Kafka modifyDMSInstanceInfoTaskFai-lure

7.2 Viewing Traces on the CTS Console

Scenario

View traces of the last seven days on the CTS console.

This section describes how to view operation records for the last 7 days on the CTSconsole.

Procedure






Step 3 Click Service List, and choose Management & Deployment > Cloud TraceService.

Step 4 In the navigation pane, choose Trace List.

Step 5 On the Trace List page, query traces based on a combination of the followingquerying dimensions:● Trace Source: Select DMS.● Resource Type: Select group, queue, or kafka.● Search By: Select an option from the drop-down list.

When you select Trace name, you also need to select a specific trace name.When you select Resource ID, you also need to select a specific resource ID.When you select Resource name, you also need to select a specific resourcename.

● Operator: Select a specific operator (a user other than tenant).● Trace Status: Available options include All trace status, normal, warning,

and incident. You can only select one of them.● Start time and end time: You can specify the time period to query traces.

Step 6 Click on the left of a trace to expand its details as shown in Figure 7-1.

Figure 7-1 Expanding trace details

Step 7 Click View Trace in the Operation column. In the displayed dialog box, the tracestructure details are displayed

----End



user guide - huawei cloud · distributed message service for kafka user guide issue 01 date...

Documents