When dealing with a large number of parallel operations in elasticsearch, such as search requests or bulk indexing operations, you may encounter thread pool related errors. Here we explain thread pools and discuss a typical search request thread pool error that might be faced, as well as how to handle such issues.

Thread pools

Thread pools are a collection of threads which are made available to perform specific tasks in elasticsearch. The main goal of thread pools is to make the memory management easier by managing the life-cycles of the threads when it comes to executing large number of requests.

A single node in elasticsearch holds several thread pools for different operations such as search, indexing, bulk operations, and more. Most thread pools are associated with a queue whose purpose is to deal with pending requests. For ease of understanding, let's consider the search thread pool and discuss more about the concept and structure.

Search Thread Pools

Search operations in elasticsearch have a dedicated search pool and an associated queue for each node. The thread pools will have "N" workers, which accept the requests. The number of workers is equal to the number of cores in that node. These N workers can accept a total of N search requests at a time. The number of workers is determined by the number of cores the node is equipped with.

Thread pool queues are deployed when request numbers are exceeded. The default size of the search queue is 1000, and the requests which exceed the thread pool worker numbers are queued in here.

We're Looking For Contributors For Our Open Source Container Orchestration Software Supergiant. Care To Check It Out?

The below picture shows the search thread pool and its queue existing in a single node.  The first set of requests are handled by the "search thread pool" with its "N" workers. The next set of search requests go to the search queue whose default size is 1000.

Requests Exceeding Limits

What happens when requests exceed the number of workers and search queue size? As in the figure shown below, if the number of parallel requests are more than the thread pool and the queue can handle, Elasticsearch throws an exception error.

Solutions

There are a few solutions that can be applied, according to the use case to solve the request, which include:

  1. Increasing the size of the thread pool. We can increase the size of the thread pool by setting the "threadpool.search.size" parameter. But, it will affect the query speed performance.
  2. Increasing the size of the search queue. The ideal and most common practice is increasing the search queue size. It should be noted that the search queue resides in the memory. Increasing it extensively can lead to out-of-memory exception errors.
  3. Increasing the nodes and replicas. This issue is caused by the overload of parallel requests, depending on use case. This method is rarely employed.
  4. Changing input data to serial data. This less popular method is used to serialize the data. This method depends on the use case and is employed only if the query performance is not considered.

Conclusion

In this post we discussed search thread pools and the exception errors that might incur in the case of multiple requests. We also explained solutions for the exception errors thrown in cases of multiple requests. Questions/Comments? Drop us a line below.

随机推荐

  1. 一、Open CV3.0.0 与 VS2012配置

    原创:博乐Bar,转载请注明出处. 第一步,准备软件及开发环境 1.OpenCV 3.0.0 下载地址:http://www.opencv.org.cn/index.php/Download ,下载最 ...

  2. asp.net mvc webconfig配置文件操作

    读取web.config数据,可以不用编译.如发布后,非常有用web.config文件<configuration> <appSettings> <add key=&qu ...

  3. MySQL(二)之服务管理与配置文件修改和连接MySQL

    上一篇给大家介绍了怎么在linux和windows中安装mysql,本来是可以放在首页的,但是博客园说“安装配置类文件”不让放在首页.接下来给大家介绍一下在linux和windows下MySQL的一下 ...

  4. 使用定时器判断确保某个标签有值才执行方法, 控制js代码执行先后顺序

    使用定时器判断确保某个标签有值才执行方法: var wait = setInterval(function(){ var diqu = $("#diqu").val(); //确保 ...

  5. Centos6.8/7.0搭建Git服务http访问方式

    安装Git版本:git 2.10.0 Git访问方式:基于http的基本验证(非SSL) 1. 安装Apache软件: [root@localhost ~]# yum install httpd 设置 ...

  6. Unity5.4新版AssetBundle资源打包

    (1)新版本 唯一打包API Buildpipeline.BuildAssetBundle (2)在资源的Inpector界面最下方可设置该资源的assetbundleName, 每个assetbun ...

  7. error: Allowed memory size

    错误提示 error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 35 bytes) in D:\www\Th ...

  8. mybatis由浅入深day02_9.3.5使用生成的代码_9.4逆向工程注意事项

    9.3.5 使用生成的代码 需要将生成工程中所生成的代码拷贝到自己的工程中. 拷这4个到我们原来的spring_mybatis1216工程下 ItemsMapper.java package cn.i ...

  9. mybatis由浅入深day02_2一对一查询_2.2方法一:resultType

    2 一对一查询 2.1 需求(查询所有订单信息,关联查询创建订单的用户信息) 查询所有订单信息,关联查询创建订单的用户信息 注意:因为一个订单信息只会是一个人下的订单,所以从查询订单信息出发关联查询用 ...

  10. Win10 安装msi 提示2502、2503的错误代码 -- 命令提示符(管理员) -- msiexec /package

    前言: 归根到底是权限不够导致的.win7应该不会有这个问题.     解决方法: 方法1:临时安装方法 1.鼠标移到桌面左下角->右键(或者直接: WIN+X键),命令提示符(管理员):2.输 ...