生产环境elasticsearch5.0报错IllegalArgumentException: number of documents in the index cannot exceed 2147483519的处理

最近几天的push:user:req数据写不到集群，报错如下：

[--13T09::,][DEBUG][o.e.a.b.TransportShardBulkAction] [yunva_etl_es8] [push:user:req][] failed to execute bulk item (index) index {[push:user:req][/push/user/req][AVyfE_JChh2DyKB0G1Kj], source[{"appId":"","yunvaId":"362b103623aa4f75911b374615c6a682","providerId":"","proviceId":"","mac":"B0:C4:E7:61:E6:E7","imsi":"","imei":"","factory":"samsung","model":"GT-S7568","osType":"android","osVersion":,"networkType":"wifi","ip":"14.152.80.239","sessionId":"","time":"2017-06-13 09:11:39.061.+0800"}]}
java.lang.IllegalArgumentException: number of documents in the index cannot exceed
at org.apache.lucene.index.DocumentsWriterPerThread.reserveOneDoc(DocumentsWriterPerThread.java:) ~[lucene-core-6.2..jar:6.2. 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - -- ::]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:) ~[lucene-core-6.2..jar:6.2. 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - -- ::]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:) ~[lucene-core-6.2..jar:6.2. 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - -- ::]
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:) ~[lucene-core-6.2..jar:6.2. 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - -- ::]
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:) ~[lucene-core-6.2..jar:6.2. 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - -- ::]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:) ~[elasticsearch-5.0..jar:5.0.]

搜索了下：
是lucene的文档数限制，每个分片最大支持2的31次方个文档数量
https://discuss.elastic.co/t/lucene-max-documents-limit/34761
Lucene max documents limit，Yep, there is a 2^31 limit per shard in Lucene.

查询当前es分片文档：

# curl 10.26.241.237:/_cat/shards?v
index shard prirep state docs store ip node

push:user:req  p STARTED  .7gb 10.30.136.143 yunva_etl_es8
push:user:req  r STARTED  .7gb 10.25.135.215 yunva_etl_es2
push:user:req  p STARTED  .7gb 10.45.150.115 yunva_etl_es9
push:user:req  r STARTED  .7gb 10.30.136.143 yunva_etl_es8
push:user:req  r STARTED  .9gb 10.30.136.143 yunva_etl_es8
push:user:req  p STARTED  .9gb 10.174.12.230 yunva_etl_es10
push:user:req  p STARTED  884gb 10.174.12.230 yunva_etl_es10
push:user:req  r STARTED  884gb 10.27.78.228 yunva_etl_es5
push:user:req  r STARTED  883gb 10.45.150.115 yunva_etl_es9
push:user:req  p STARTED  883gb 10.174.12.230 yunva_etl_es10

查询出来的2147483519和2的31次方2147483648接近，所以新增加的数据无法插入

临时的解决办法：先把kafka的磁盘加到数据多保留一段时间20天

原来的集群数据抽取到hbase中作为冷数据处理，再把kafka里面的数据导入数据到新的集群中