Hbase is a distributed data storage systems.

A Bigtable is spare , distributed , persistent multidimensional sorted map. The map is indexed by a row key, column key, and a timestamp. each value in the map is an uninterrupted arry of bytes.

HBase uses a data model very similar to that of Bigtable. Users store data rows in labelled tables. A data row has a sortable key and an arbitrary number of columns. The table is stored sparsely, so that rows in the same table can have crazily-varying columns, if the user likes.

map

At its core, HBase/BigTable is a map. From the wikipedia article, a map is "an abstract data type composed of a collection of keys and a collection of values, where each key is associated with one value."

Using JavaScript Object Notation, here's an example of a simple map where all the values are just strings:

{
"zzzzz" : "woot",
"xyz" : "hello",
"aaaab" : "world",
"1" : "x",
"aaaaa" : "y"
}

persistent

Persistence merely means that the data you put in this special map "persists" after the program that created or accessed it is finished.

distributed

HBase and BigTable are built upon distributed filesystems so that the underlying file storage can be spread out among an array of independent machines.

HBase sits atop either HDFS or Amazon's S3, while a BigTable makes use of the GFS.

Data is replicated across a number of participating nodes in an analogous manner to how data is striped across discs in a RAID system.

sorted

Unlike most map implementations, in HBase/BigTable the key/value pairs are kept in strict alphabetical order. That is to say that the row for the key "aaaaa" should be right next to the row with key "aaaab" and very far from the row with key "zzzzz".

Continuing our JSON example, the sorted version looks like this:

{
"1" : "x",
"aaaaa" : "y",
"aaaab" : "world",
"xyz" : "hello",
"zzzzz" : "woot"
}

Because these systems tend to be so huge and distributed, this sorting feature is actually very important. The spacial propinquity of rows with like keys ensures that when you must scan the table, the items of greatest interest to you are near each other.

This is important when choosing a row key convention. For example, consider a table whose keys are domain names. It makes the most sense to list them in reverse notation (so "com.jimbojw.www" rather than "www.jimbojw.com") so that rows about a subdomain will be near the parent domain row.

Continuing the domain example, the row for the domain "mail.jimbojw.com" would be right next to the row for "www.jimbojw.com" rather than say "mail.xyz.com" which would happen if the keys were regular domain notation.

It's important to note that the term "sorted" when applied to HBase/BigTable does not mean that "values" are sorted. There is no automatic indexing of anything other than the keys, just as it would be in a plain-old map implementation.

multidimensional

It is easier to think about this like a multidimensional map - a map of maps if you will. Adding one dimension to our running JSON example gives us this:

{
"1" : {
"A" : "x",
"B" : "z"
},
"aaaaa" : {
"A" : "y",
"B" : "w"
},
"aaaab" : {
"A" : "world",
"B" : "ocean"
},
"xyz" : {
"A" : "hello",
"B" : "there"
},
"zzzzz" : {
"A" : "woot",
"B" : "1337"
}
}

In the above example, you'll notice now that each key points to a map with exactly two keys: "A" and "B". From here forward, we'll refer to the top-level key/map pair as a "row". Also, in BigTable/HBase nomenclature, the "A" and "B" mappings would be called "Column Families".

A table's column families are specified when the table is created, and are difficult or impossible to modify later. It can also be expensive to add new column families, so it's a good idea to specify all the ones you'll need up front.

Fortunately, a column family may have any number of columns, denoted by a column "qualifier" or "label". Here's a subset of our JSON example again, this time with the column qualifier dimension built in:

{
// ...
"aaaaa" : {
"A" : {
"foo" : "y",
"bar" : "d"
},
"B" : {
"" : "w"
}
},
"aaaab" : {
"A" : {
"foo" : "world",
"bar" : "domination"
},
"B" : {
"" : "ocean"
}
},
// ...
}

Notice that in the two rows shown, the "A" column family has two columns: "foo" and "bar", and the "B" column family has just one column whose qualifier is the empty string ("").

When asking HBase/BigTable for data, you must provide the full column name in the form "<family>:<qualifier>". So for example, both rows in the above example have three columns: "A:foo", "A:bar" and "B:".

Note that although the column families are static, the columns themselves are not. Consider this expanded row:

{
// ...
"zzzzz" : {
"A" : {
"catch_phrase" : "woot",
}
}
}

In this case, the "zzzzz" row has exactly one column, "A:catch_phrase". Because each row may have any number of different columns, there's no built-in way to query for a list of all columns in all rows. To get that information, you'd have to do a full table scan. You can however query for a list of all column families since these are immutable (more-or-less).

The final dimension represented in HBase/BigTable is time. All data is versioned either using an integer timestamp (seconds since the epoch), or another integer of your choice. The client may specify the timestamp when inserting data.

Consider this updated example utilizing arbitrary integral timestamps:

{
// ...
"aaaaa" : {
"A" : {
"foo" : {
15 : "y",
4 : "m"
},
"bar" : {
15 : "d",
}
},
"B" : {
"" : {
6 : "w"
3 : "o"
1 : "w"
}
}
},
// ...
}

Each column family may have its own rules regarding how many versions of a given cell to keep (a cell is identified by its rowkey/column pair) In most cases, applications will simply ask for a given cell's data, without specifying a timestamp. In that common case, HBase/BigTable will return the most recent version since it stores these in reverse chronological order.

If an application asks for a given row at a given timestamp, HBase will return cell data where the timestamp is less than or equal to the one provided.

Using our imaginary HBase table, querying for the row/column of "aaaaa"/"A:foo" will return "y" while querying for the row/column/timestamp of "aaaaa"/"A:foo"/10 will return "m". Querying for a row/column/timestamp of "aaaaa"/"A:foo"/2 will return a null result.

sparse

The last keyword is sparse. As already mentioned, a given row can have any number of columns in each column family, or none at all. The other type of sparseness is row-based gaps, which merely means that there may be gaps between keys.

Understanding HBase and BigTable的更多相关文章

  1. HBase vs. BigTable Comparison - HBase对比BigTable

    HBase vs. BigTable Comparison HBase is an open-source implementation of the Google BigTable architec ...

  2. 理解Hbase和BigTable(转)

    add by zhj: 这篇文章写的通俗易懂,介绍了HBase最重要的几点特性. 英文原文:https://dzone.com/articles/understanding-hbase-and-big ...

  3. HBase 数据模型(Data Model)

    HBase Data Model--HBase 数据模型(翻译) 在HBase中,数据是存储在有行有列的表格中.这是与关系型数据库重复的术语,并不是有用的类比.相反,HBase可以被认为是一个多维度的 ...

  4. (zz) 谷歌技术"三宝"之BigTable

    006年的OSDI有两篇google的论文,分别是BigTable和Chubby.Chubby是一个分布式锁服务,基于Paxos算法:BigTable是一个用于管理结构化数据的分布式存储系统,构建在G ...

  5. [转载] 谷歌技术"三宝"之BigTable

    转载自http://blog.csdn.net/opennaive/article/details/7532589 2006年的OSDI有两篇google的论文,分别是BigTable和Chubby. ...

  6. HBase 数据模型

    在HBase中,数据是存储在有行有列的表格中.这是与关系型数据库重复的术语,并不是有用的类比.相反,HBase可以被认为是一个多维度的映射. HBase数据模型术语 Table(表格) 一个HBase ...

  7. Hbase学习03

    第3章 Hbase数据存储模型与工作组件 Data格式设计的的总体原则是按照需求要求,依据Hbase性能的相关标准规范和文件,并遵循“统一规范.统一数据模型.统一规划集群.分步实施”的原则,注重实际应 ...

  8. 谷歌技术"三宝"之BigTable

    转自:https://blog.csdn.net/OpenNaive/article/details/7532589 2006年的OSDI有两篇google的论文,分别是BigTable和Chubby ...

  9. HBase 架构与工作原理1 - HBase 的数据模型

    本文系转载,如有侵权,请联系我:likui0913@gmail.com 一.应用场景 HBase 与 Google 的 BigTable 极为相似,可以说 HBase 就是根据 BigTable 设计 ...

随机推荐

  1. “HtmlAgilityPack”已拥有为“System.Net.Http”定义的依赖项的解决方案

    #事故现场 在vs2013上用nuget安装 HtmlAgilityPack -Version 1.8.9时,报错如下: Install-Package : “HtmlAgilityPack”已拥有为 ...

  2. 第一节: 结合EF的本地缓存属性来介绍【EF增删改操作】的几种形式

    一. 背景 说起EF的增删改操作,相信很多人都会说,有两种方式:① 通过方法操作  和  ② 通过状态控制. 相信你在使用EF进行删除或修改操作的时候,可能会遇到以下错误:“ The object c ...

  3. [浏览器事件循环] javaScript事件循环 EventLoop

    前言 Event Loop即事件循环,是指浏览器或Node的一种解决javaScript单线程运行时不会阻塞的一种机制,也就是我们经常使用异步的原理. 先熟悉基本概念 [堆Heap] 堆是一种数据结构 ...

  4. [译]Ocelot - Big Picture

    原文 目录 Big Picture Getting Started Configuration Routing Request Aggregation Service Discovery Authen ...

  5. jQuery1.9及以上版本检测IE版本号

    jQuery 从 1.9 版开始,移除了 $.browser 和 $.browser.version , 取而代之的是 $.support . 在更新的 2.0 版本中,将不再支持 IE 6/7/8. ...

  6. C++创建对象的三种方法

    我自己以前经常弄混 A a(1); 栈内存中分配 A b = A(1); 栈内存中分配,和第一种无本质区别 A c = new A(1); 堆内存中分配 前两种在函数体执行完毕之后会被释放,第三种需要 ...

  7. PHP循环语句深度理解分析——while, for, foreach, do while

    循环结构   一.while循环  while(表达式)  {   循环体;//反复执行,直到表达式为假  } 代码: $index = 1; while ($index<5) {        ...

  8. Django之AJAX

    一.预备知识JSON python中的json: json.dumps( )      json.loads( ) JavaScript中的json:JSON.stringify( )       J ...

  9. pytorch对可变长度序列的处理

    主要是用函数torch.nn.utils.rnn.PackedSequence()和torch.nn.utils.rnn.pack_padded_sequence()以及torch.nn.utils. ...

  10. 快速搭建一个Spring Boot + MyBatis的开发框架

    前言:Spring Boot的自动化配置确实非常强大,为了方便大家把项目迁移到Spring Boot,特意总结了一下如何快速搭建一个Spring Boot + MyBatis的简易文档,下面是简单的步 ...