This KIP proposes to control request handler (I/O) thread and network thread utilization using request processing time quotas that limit the amount of time within each quota window that can be used by users/clients. Network thread utilization needs to be taken into account to address this.

In clusters that enable both TLS and non-TLS endpoints, the load on the network thread is significantly higher for TLS.

While quotas of processing time on the request handler thread limit the request rate in many cases above, for a complete request rate quota solution, network thread utilization also needs to be taken into account. disk read for fetches happen on the network threads).

Some requests may use more of their quota on the network threads rather than the request handler threads (eg.

Quotas based on request count per second may not be easy to configure in this case. Number of requests may double, so request counts increase, even though the load on the broker doesn't double since the number of partitions per fetch request has halved.

Consumer group starts with 10 instances and then increases to 20 instances.

Request processing time quotas are required since neither produce byte rate quotas nor request count quotas will be sufficient to limit the broker resources allocated to users/clients in this case.

Client sends produce requests with compressed messages where decompression takes a long time, blocking the request handler thread.

Either request count quotas or request processing time quotas that limit all unauthorized requests and all non-broker (client) requests is required. DoS attack from clients that overload brokers with continuous authorized or unauthorized requests.Fetch byte rate quotas are not sufficient here, either request count quotas or request processing time quotas are required. a consumer with =0 that polls continuously. Clients send requests too quickly - eg.It will be useful to additionally support throttling by request rate to ensure that broker resources are not monopolized by some users/clients. However, if a client sends requests too quickly (e.g., a consumer with =0), it can still overwhelm the broker even though individual request/response size may be small.

Clients that produce or fetch messages at a byte rate that exceeds their quota are throttled by delaying the response by an amount that brings the byte rate within the configured quota. Kafka currently supports quotas by data volume. Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Request throttled new world