ES_DEVOPS-1

When dealing with a large number of parallel operations in elasticsearch, such as search requests or bulk indexing operations, you may encounter thread pool related errors. Here we explain thread pools and discuss a typical search request thread pool error that might be faced, as well as how to handle such issues.

Thread pools

Thread pools are a collection of threads which are made available to perform specific tasks in elasticsearch. The main goal of thread pools is to make the memory management easier by managing the life-cycles of the threads when it comes to executing large number of requests.

A single node in elasticsearch holds several thread pools for different operations such as search, indexing, bulk operations, and more. Most thread pools are associated with a queue whose purpose is to deal with pending requests. For ease of understanding, let's consider the search thread pool and discuss more about the concept and structure.

Search Thread Pools

Search operations in elasticsearch have a dedicated search pool and an associated queue for each node. The thread pools will have "N" workers, which accept the requests. The number of workers is equal to the number of cores in that node. These N workers can accept a total of N search requests at a time. The number of workers is determined by the number of cores the node is equipped with.

Thread pool queues are deployed when request numbers are exceeded. The default size of the search queue is 1000, and the requests which exceed the thread pool worker numbers are queued in here.

We're Looking For Contributors For Our Open Source Container Orchestration Software Supergiant. Care To Check It Out?

The below picture shows the search thread pool and its queue existing in a single node.  The first set of requests are handled by the "search thread pool" with its "N" workers. The next set of search requests go to the search queue whose default size is 1000.

Requests Exceeding Limits

What happens when requests exceed the number of workers and search queue size? As in the figure shown below, if the number of parallel requests are more than the thread pool and the queue can handle, Elasticsearch throws an exception error.

Solutions

There are a few solutions that can be applied, according to the use case to solve the request, which include:

1. Increasing the size of the thread pool. We can increase the size of the thread pool by setting the "threadpool.search.size" parameter. But, it will affect the query speed performance.
2. Increasing the size of the search queue. The ideal and most common practice is increasing the search queue size. It should be noted that the search queue resides in the memory. Increasing it extensively can lead to out-of-memory exception errors.
3. Increasing the nodes and replicas. This issue is caused by the overload of parallel requests, depending on use case. This method is rarely employed.
4. Changing input data to serial data. This less popular method is used to serialize the data. This method depends on the use case and is employed only if the query performance is not considered.

Conclusion

In this post we discussed search thread pools and the exception errors that might incur in the case of multiple requests. We also explained solutions for the exception errors thrown in cases of multiple requests. Questions/Comments? Drop us a line below.