Quotas in Apache Kafka®

Introduction

In Kafka, quotas are used to limit the resources (such as network bandwidth or request rate) that clients or users can consume. This helps prevent resource exhaustion and enforce fairness among clients. Clients can exhaust resources accidentally (software bugs or misconfiguration) or on purpose (Denial of Service attacks). Quotas are enforced per broker, so a client hitting multiple brokers may have higher-than-expected aggregate throughput. In other words, there is no global coordinator keeping track of total throughput.

Types

There are several different types of quotas you can enforce in Kafka. If you have not configured any quotas, clients will not be throttled in any manner. All the currently available quotas are detailed below.

• producer_byte_rate - Limits the bytes per second a producer can send.
• consumer_byte_rate - Limits the bytes per second a consumer can fetch.
• request_percentage - Limits the share of CPU usage a client can use within each quota window. The total available percentage per quota window is (num.io.threads + num.network.threads) * 100%. In other words, the request percentage quota can exceed 100. The size of the quota window is one second by default, and can be changed using the quota.window.size.seconds config setting. This quota was introduced in KIP-124.
• controller_mutation_rate - Limits the number of admin operations (create topic, create partition, delete topic) that a client can perform per second. These are heavy operations that can potentially affect the stability of the whole cluster if a misbehaving client floods the system with these requests. This quota was introduced in KIP-599.
• connection_creation_rate - Limits the rate of connections from a specific IP-address. Only available for the IP Address entity type. This quota was introduced in KIP-612.

Configuration - Command Line

The quotas can be configured on the command link using the kafka-configs.sh shell script. For example, to add a quota of 1048576 bytes/sec for producer_byte_rate you can use the --add-config flag as follows

kafka-configs.sh --bootstrap-server broker:9092 \
  --alter --add-config 'producer_byte_rate=1048576' \
  --entity-type users --entity-name producerx

The available values for the quota are the ones detailed in the previous section - producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate, connection_creation_rate

The entity-type argument specifies the entity that the quota should apply to. The following values are available for the entity type argument.

• users - Authenticated Kafka user name
• clients - Logical client ID from the client configuration (CLIENT_ID_CONFIG)
• ips - IP address of the client. This is only applicable to the connection_creation_rate quota, which cannot be used with other entity types.

Configuration - Web UI

You can also manage quotas in the gradient fox Web UI. You can find quotas under the Quotas-tab of the cluster you want to manage. A new quota can be added using the Add-button, which will bring up the dialog shown below.

The dialog has sections for the entity that the quota applies to as well as limits-section where quotas for the entity in question can be specified. The entity types and the limits are the same ones you can use on the command line. As explained earlier, the IP Address entity type can only be used with the Connection Creation Rate limit (and the limit can only be used with the IP Address entity type).

Under the same Quotas-tab you can also delete and modify existing quotas using the actions shown for each displayed quota.

Exceeding Quotas

If a consumer or producer exceeds their allowed quota, the brokers will start throttling the client in question. When a producer exceeds the producer_byte_rate quota, brokers will accept the new messages(s), but will add artificial delays before responding. This will result in increased latency seen by the producer. When a consumer exceeds the configured consumer_byte_rate quota, brokers will delay the fetch responses. The consumer will see increased fetch times for their requests.

When a client exceeds the request_percentage limit, the brokers will again delay the response to bring the client request rate within the configured limits.

Exceeding the controller_mutation_rate quota will be handled differently from the other quota violations. When the limits are exceeded, the brokers will return a QuotaViolatedException exception to the caller. The admin client can then retry the call (unless configured not to) until the default.api.timeout.ms setting has been exceeded. In that case, the admin client will throw an ThrottlingQuotaExceededException.

When a client exceeds the connection_creation_rate limit, the broker will delay processing the connection by an amount of time that will result in the client (ip) to be within the configured limits or by one second, whichever will happen soonest. If the client is still in quota violation after the delay, the connection will be dropped. Since the quota is based on the IP address of the client, the broker must accept the connection first to see what the client IP address is. In other words, the quota enforcement can only happen after the connection has been established.

Cloud Providers

Many of the cloud providers do you expose quotas thru the Kafka APIs as they have implemented custom quota systems for their environments. If you try to access the quotas thru the command line tools or gradient fox Web UI, you may encounter some of the following errors.

ClusterAuthorizationException
UnsupportedVersionException