【原创】大叔经验分享（58）kudu写入压力大时报错

kudu写入压力大时报错

19/05/18 16:53:12 INFO AsyncKuduClient: Invalidating location fd52e4f930bc45458a8f29ed118785e3(server002:7050) for tablet 4259921cdcca4776b37771659a8cafb3: Service unavailable: Soft memory limit exceeded (at 106.05% of capacity). See https://kudu.apache.org/releases/1.7.0-cdh5.16.1/docs/troubleshooting.html

这是个INFO日志，只是在内存超过hard的80%时拒绝写入，可能请求导致部分写入超时；

参数调优（增大以下参数）：

• 1 --memory_limit_hard_bytes
• 2 --maintenance_manager_num_threads （Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.）
• 3 --block_cache_capacity_mb （memory_limit_hard_bytes的30%到50%）

原文：

Kudu has a hard and soft memory limit. The hard memory limit is the maximum amount a Kudu process is allowed to use, and is controlled by the --memory_limit_hard_bytes flag. The soft memory limit is a percentage of the hard memory limit, controlled by the flag memory_limit_soft_percentage and with a default value of 80%, that determines the amount of memory a process may use before it will start rejecting some write operations.

If the logs or RPC traces contain messages like

Service unavailable: Soft memory limit exceeded (at 96.35% of capacity)

then Kudu is rejecting writes due to memory backpressure. This may result in write timeouts. There are several ways to relieve the memory pressure on Kudu:

• If the host has more memory available for Kudu, increase --memory_limit_hard_bytes.
• Increase the rate at which Kudu can flush writes from memory to disk by increasing the number of disks or increasing the number of maintenance manager threads --maintenance_manager_num_threads. Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.
• Reduce the volume of writes flowing to Kudu on the application side.

Finally, on versions of Kudu prior to 1.8, check the value of --block_cache_capacity_mb. This setting determines the maximum size of Kudu’s block cache. While a higher value can help with read and write performance, do not raise --block_cache_capacity_mb above the memory pressure threshold, which is --memory_pressure_percentage (default 60%) of --memory_limit_hard_bytes, as this will cause Kudu to flush aggressively even if write throughput is low. Keeping --block_cache_capacity_mb below 50% of the memory pressure threshold is recommended. With the defaults, this means --block_cache_capacity_mb should not exceed 30% of --memory_limit_hard_bytes. On Kudu 1.8 and higher, servers will refuse to start if the block cache capacity exceeds the memory pressure threshold.

调优：

https://kudu.apache.org/docs/scaling_guide.html#memory

https://kudu.apache.org/docs/scaling_guide.html#_verifying_if_a_memory_limit_is_sufficient

参考：
https://kudu.apache.org/docs/troubleshooting.html
https://kudu.apache.org/docs/troubleshooting.html#memory_limits