HBase HFile

HFile index

HFile index, which is proportional to the total number of Data Blocks. The total amount of memory needed to hold the index can be estimated as (56+AvgKeySize)*NumBlocks.

HFile的索引记录数量是与HFile所包含的Data Block数目成正比的。整个索引所需要的内存空间大约是(56+AvgKeySize)*NumBlocks。

Suggestions on performance optimization
Minimum block size. We recommend a setting of minimum block size between 8KB to 1MB for general usage. Larger block size is preferred if files are primarily for sequential access. However, it would lead to inefficient random access (because there are more data to decompress). Smaller blocks are good for random access, but require more memory to hold the block index, and may be slower to create (because we must flush the compressor stream at the conclusion of each data block, which leads to an FS I/O flush). Further, due to the internal caching in Compression codec, the smallest possible block size would be around 20KB-30KB.

在一般的使用场景中推荐将HBase Data Block的大小设置为8KB到1MB之间。

如果顺序访问的场景居多，可以将Data Block的大小调大，但是这会导致HBase随机访问效率低下，因为需要对更多的数据进行解压。

小一点的Data Block非常有利于HBase的随机访问，但是需要消耗更多的内存来维护索引记录；在创建这些Data Block的时候也会比较慢，因为我们需要在每一个Data Block被写满之后对其进行Flush，这会导致低层HDFS的I/O Flush。而且，由于一些压缩库在使用时内部会包含一些Cache，所以Data Block的大小最好控制在20KB到30KB之间。

HFile Format

File is made of data blocks followed by meta data blocks (if any), a fileinfo block, data block index, meta data block index, and a fixed size trailer which records the offsets at which file changes content type.

 <data blocks><meta blocks><fileinfo><data index><meta index><trailer>
 

Each block has a bit of magic at its start. Block are comprised of key/values. In data blocks, they are both byte arrays. Metadata blocks are a String key and a byte array value. An empty file looks like this:

 <fileinfo><trailer>
 

That is, there are not data nor meta blocks present.