Kyoto Cabinet 使用及原理
Kyoto Cabinet 基本规格书
如果你知道 Tokyo Cabinet ,那么就应该知道 Kyoto Cabinet,因为他们都是同一个作者(平林幹雄)开发出来的 Key-Value 数据库。
- 改进的空间效率:更小的数据库文件
- 改进的时间效率:更快的处理速度
- 改进的并行性:多线程环境下的高性能
- 改进的可用性:简单的API
- 改进的健壮性:即使在灾难情况下数据库文件也不会损坏
- 支持64位架构:巨大的内存空间和数据库文件可用
- 改进的空间效率:更小的数据库文件
- 改进的并行性:多线程环境下的高性能
- 改进的可移植性:对底层的抽象来支持 非POSIX系统
- 改进的可用性:简单的API,面向对象的设计
- 改进的健壮性:即使在灾难情况下数据库文件也不会损坏
- tune_buckets:设置hash数据库的 bucket 数量
- tune_options:设置可选特性(optional features)
- tune_buckets:设置hash数据库的 bucket 数量
- tune_compressor:设置数据压缩方法
- cap_count:设置记录数的容量
- cap_size:设置内存使用的容量
db.tune_buckets(10LL * * );
db.cap_count(10LL * * );
db.cap_size(8LL << );
db.open(...);
- tune_page:设置每个页大小
- tune_page_cache:设置页缓存(page cache)容量大小
- tune_comparator:设置记录比较器
db.tune_options(GrassDB::TCCOMPESS);
db.tune_buckets(500LL * );
db.tune_page();
db.tune_page_cache(1LL << );
db.open(...);
- tune_alignment:设置记录的对齐幂数
- tune_fbp:设置空闲块池的容量幂数
- tune_options:设置可选特性
- tune_buckets:设置哈希表的bucket数量
- tune_map:设置内部内存映射区域的大小
- tune_defrag:设置自动碎片整理的单位步数
- tune_compressor:设置数据压缩器
db.tune_alignment();
db.tune_options(HashDB::TSMALL | HashDB::TLINEAR);
db.tune_buckets(10LL * );
db.tune_defrag();
db.open(...);
db.tune_options(HashDB::TLINEAR);
db.tune_buckets(20LL * * * );
db.tune_map(300LL << );
db.open(...);
- tune_page:设置每个页大小
- tune_page_cache:设置页缓存(page cache)容量大小
- tune_comparator:设置记录比较器
db.tune_options(TreeDB::TLINEAR | TreeDB::TCCOMPESS); db.tune_buckets(1LL * 1000); db.tune_defrag(8); db.tune_page(32768); db.open(...);
db.tune_options(TreeDB::TLINEAR); db.tune_buckets(1LL * 1000 * 1000 * 1000); db.tune_map(300LL << 30); db.tune_page_cache(8LL << 30); db.open(...);
- tune_options:设置可选特性
- 时间效率:CacheDB > StashDB > ProtoHashDB > ProtoTreeDB > GrassDB
- 空间效率:GrassDB > StashDB > CacheDB > ProtoHashDB > ProtoTreeDB
- 时间效率:HashDB > TreeDB > DirDB > ForestDB
- 空间效率:TreeDB > HashDB > ForestDB > DirDB
db.begin_transaction();
db.set("japan", "tokyo");
db.set("korea", "seoul");
db.end_transaction();
db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE | HashDB::OAUTOTRAN);
db.set("japan", "tokyo");
db.set("china", "beijing");
db.copy("backup.kch");
class BackupImpl : public FileProcessor {
bool process(const std::string& path, int64_t size, int64_t count) {
char cmd[];
sprintf(cmd, "snapshot.sh %s", path.c_str());
return system(cmd) == ;
}
} proc;
db.synchronize(&proc);
db.dump_snapshot("backup.kcss");
db.load_snapshot("backup.kcss");
ArcfourCompressor comp;
comp.set_key("foobarbaz", );
TreeDB db;
db.tune_options(kc::TreeDB::TCOMPRESS);
db.tune_compressor(&comp);
db.open(...);
comp.begin_cycle((uint64_t)getpid() << + (uint64_t)time());
PolyDB db;
db.open("casket.kct#zcomp=arc#zkey=foobarbaz", ...);
GNU General Public License
Kyoto Cabinet is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.
Kyoto Cabinet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see `http://www.gnu.org/licenses/
'.
FOSS License Exception
The FOSS License Exception is also provided in order to accommodate products under other free and open source licenses. See the body text for details.
Commercial License
If you use Kyoto Cabinet within a proprietary software, a commercial license is required.
The commercial license allows you to utilize Kyoto Cabinet by including it in your applications for purpose of developing and producing your applications and to utilize Kyoto Cabinet in order to transfer, sale, rent, lease, distribute or sublicense your applications to any third parties. See the license guide for details.
Author
Kyoto Cabinet was written and is maintained by FAL Labs. You can contact the author by e-mail to `info@fallabs.com
'.
Examples
C++Example:
The following code is an example to use a polymorphic database.
#include <kcpolydb.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open("casket.kch", PolyDB::OWRITER | PolyDB::OCREATE)) {
cerr << "open error: " << db.error().name() << endl;
} // store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")) {
cerr << "set error: " << db.error().name() << endl;
} // retrieve a record
string value;
if (db.get("foo", &value)) {
cout << value << endl;
} else {
cerr << "get error: " << db.error().name() << endl;
} // traverse records
DB::Cursor* cur = db.cursor();
cur->jump();
string ckey, cvalue;
while (cur->get(&ckey, &cvalue, true)) {
cout << ckey << ":" << cvalue << endl;
}
delete cur; // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The following code is a more complex example, which uses the Visitor pattern.
#include <kcpolydb.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open("casket.kch", PolyDB::OREADER)) {
cerr << "open error: " << db.error().name() << endl;
} // define the visitor
class VisitorImpl : public DB::Visitor {
// call back function for an existing record
const char* visit_full(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t *sp) {
cout << string(kbuf, ksiz) << ":" << string(vbuf, vsiz) << endl;
return NOP;
}
// call back function for an empty record space
const char* visit_empty(const char* kbuf, size_t ksiz, size_t *sp) {
cerr << string(kbuf, ksiz) << " is missing" << endl;
return NOP;
}
} visitor; // retrieve a record with visitor
if (!db.accept("foo", , &visitor, false) ||
!db.accept("dummy", , &visitor, false)) {
cerr << "accept error: " << db.error().name() << endl;
} // traverse records with visitor
if (!db.iterate(&visitor, false)) {
cerr << "iterate error: " << db.error().name() << endl;
} // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The following code is an example of word counting with the MapReduce framework. #include <kcpolydb.h>
#include <kcdbext.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open()) {
cerr << "open error: " << db.error().name() << endl;
} // store records
db.set("", "this is a pen");
db.set("", "what a beautiful pen this is");
db.set("", "she is beautiful"); // define the mapper and the reducer
class MapReduceImpl : public MapReduce {
// call back function of the mapper
bool map(const char* kbuf, size_t ksiz, const char* vbuf, size_t vsiz) {
vector<string> words;
strsplit(string(vbuf, vsiz), ' ', &words);
for (vector<string>::iterator it = words.begin();
it != words.end(); it++) {
emit(it->data(), it->size(), "", );
}
return true;
}
// call back function of the reducer
bool reduce(const char* kbuf, size_t ksiz, ValueIterator* iter) {
size_t count = ;
const char* vbuf;
size_t vsiz;
while ((vbuf = iter->next(&vsiz)) != NULL) {
count++;
}
cout << string(kbuf, ksiz) << ": " << count << endl;
return true;
}
} mr; // execute the MapReduce process
if (!mr.execute(&db)) {
cerr << "MapReduce error: " << db.error().name() << endl;
} // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The C language binding is also provided as a wrapper of the polymorphic database API. The following code is an example.
#include <kclangc.h> /* call back function for an existing record */
const char* visitfull(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t *sp, void* opq) {
fwrite(kbuf, , ksiz, stdout);
printf(":");
fwrite(vbuf, , vsiz, stdout);
printf("\n");
return KCVISNOP;
} /* call back function for an empty record space */
const char* visitempty(const char* kbuf, size_t ksiz, size_t *sp, void* opq) {
fwrite(kbuf, , ksiz, stdout);
printf(" is missing\n");
return KCVISNOP;
} /* main routine */
int main(int argc, char** argv) {
KCDB* db;
KCCUR* cur;
char *kbuf, *vbuf;
size_t ksiz, vsiz;
const char *cvbuf; /* create the database object */
db = kcdbnew(); /* open the database */
if (!kcdbopen(db, "casket.kch", KCOWRITER | KCOCREATE)) {
fprintf(stderr, "open error: %s\n", kcecodename(kcdbecode(db)));
} /* store records */
if (!kcdbset(db, "foo", , "hop", ) ||
!kcdbset(db, "bar", , "step", ) ||
!kcdbset(db, "baz", , "jump", )) {
fprintf(stderr, "set error: %s\n", kcecodename(kcdbecode(db)));
} /* retrieve a record */
vbuf = kcdbget(db, "foo", , &vsiz);
if (vbuf) {
printf("%s\n", vbuf);
kcfree(vbuf);
} else {
fprintf(stderr, "get error: %s\n", kcecodename(kcdbecode(db)));
} /* traverse records */
cur = kcdbcursor(db);
kccurjump(cur);
while ((kbuf = kccurget(cur, &ksiz, &cvbuf, &vsiz, )) != NULL) {
printf("%s:%s\n", kbuf, cvbuf);
kcfree(kbuf);
}
kccurdel(cur); /* retrieve a record with visitor */
if (!kcdbaccept(db, "foo", , visitfull, visitempty, NULL, ) ||
!kcdbaccept(db, "dummy", , visitfull, visitempty, NULL, )) {
fprintf(stderr, "accept error: %s\n", kcecodename(kcdbecode(db)));
} /* traverse records with visitor */
if (!kcdbiterate(db, visitfull, NULL, )) {
fprintf(stderr, "iterate error: %s\n", kcecodename(kcdbecode(db)));
} /* close the database */
if (!kcdbclose(db)) {
fprintf(stderr, "close error: %s\n", kcecodename(kcdbecode(db)));
} /* delete the database object */
kcdbdel(db); return ;
}
Java example:
The following code is a typical example to use a database.
import kyotocabinet.*; public class KCDBEX1 {
public static void main(String[] args) { // create the object
DB db = new DB(); // open the database
if (!db.open("casket.kch", DB.OWRITER | DB.OCREATE)){
System.err.println("open error: " + db.error());
} // store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")){
System.err.println("set error: " + db.error());
} // retrieve records
String value = db.get("foo");
if (value != null){
System.out.println(value);
} else {
System.err.println("set error: " + db.error());
} // traverse records
Cursor cur = db.cursor();
cur.jump();
String[] rec;
while ((rec = cur.get_str(true)) != null) {
System.out.println(rec[0] + ":" + rec[1]);
}
cur.disable(); // close the database
if(!db.close()){
System.err.println("close error: " + db.error());
} }
}
The following code is a more complex example, which uses the Visitor pattern.
import kyotocabinet.*; public class KCDBEX2 {
public static void main(String[] args) { // create the object
DB db = new DB(); // open the database
if (!db.open("casket.kch", DB.OREADER)) {
System.err.println("open error: " + db.error());
} // define the visitor
class VisitorImpl implements Visitor {
public byte[] visit_full(byte[] key, byte[] value) {
System.out.println(new String(key) + ":" + new String(value));
return NOP;
}
public byte[] visit_empty(byte[] key) {
System.err.println(new String(key) + " is missing");
return NOP;
}
}
Visitor visitor = new VisitorImpl(); // retrieve a record with visitor
if (!db.accept("foo".getBytes(), visitor, false) ||
!db.accept("dummy".getBytes(), visitor, false)) {
System.err.println("accept error: " + db.error());
} // traverse records with visitor
if (!db.iterate(visitor, false)) {
System.err.println("iterate error: " + db.error());
} // close the database
if(!db.close()){
System.err.println("close error: " + db.error());
} }
}
Kyoto Cabinet 使用及原理的更多相关文章
- Kyoto Cabinet(DBM) + Kyoto Tycoon(网络层)
项目原地址kyotocabinet: http://fallabs.com/kyotocabinet/ kyototycoon: http://fallabs.com/kyototyc ...
- 实现键值对存储(三):Kyoto Cabinet 和LevelDB的架构比較分析
译自 Emmanuel Goossaert (CodeCapsule.com) 在本文中,我将会逐组件地把Kyoto Cabinet 和 LevelDB的架构过一遍.目标和本系列第二部分讲的差点儿相 ...
- Kyoto Cabinet--nosql型单机数据库
摘要: Kyoto Cabinet是轻量级nosql型本地内存数据库 简介 Kyoto Cabinet是一个数据库管理的 lib,是 Tokyo Cabinet 的改进版本.数据库是一个简单的包含记录 ...
- 大型web系统数据缓存设计
1. 前言 在高访问量的web系统中,缓存几乎是离不开的:但是一个适当.高效的缓存方案设计却并不容易:所以接下来将讨论一下应用系统缓存的设计方面应该注意哪些东西,包括缓存的选型.常见缓存系统的特点和数 ...
- NoSQL数据库笔谈(转)
NoSQL数据库笔谈 databases , appdir , node , paper颜开 , v0.2 , 2010.2 序 思想篇 CAP 最终一致性 变体 BASE 其他 I/O的五分钟法则 ...
- 大数据架构师NoSQL建模技术
从数据建模的角度对NoSQL家族系统做了比较简单的比较,并简要介绍几种常见建模技术. 1.前言 为了适应大数据应用场景的要求,Hadoop以及NoSQL等与传统企业平台完全不同的新兴架构迅速地崛起.而 ...
- H2的MVStore
翻译自http://www.h2database.com/html/mvstore.html 转载请著名出处,及译者信息. 第一次翻译,诸多不妥请谅解,谢谢. 概述 MVStore是一个持久化的.日志 ...
- 大型web系统数据缓存设计-l转载
原文地址:http://www.wmyouxi.com/a/60368.html#ixzz3tGYG9JwC 1. 前言 在高访问量的web系统中,缓存几乎是离不开的:但是一个适当.高效的缓存方案设计 ...
- NoSQL数据库笔谈
NoSQL数据库笔谈 databases , appdir , node , paper颜开 , v0.2 , 2010.2 序 思想篇 CAP 最终一致性 变体 BASE 其他 I/O的五分钟法则 ...
随机推荐
- android 滑动冲突
韩梦飞沙 韩亚飞 313134555@qq.com yue31313 han_meng_fei_sha 通过move事件的 拦截. 在滑动组件中,重写 在拦截触摸事件的时候 这个方法, 然后 ...
- P4812 D’s problem(d)
P4812 D’s problem(d)From: admin 时间: 1000ms / 空间: 65536KiB / Java类名: Main 背景 NOIP春季系列课程 描述 小D是一名魔法师,它 ...
- Android任务和返回栈完全解析(转)
转载请注明出处:http://blog.csdn.net/guolin_blog/article/details/41087993 本篇文章主要内容来自于Android Doc,我翻译之后又做了些加工 ...
- tesseract-ocr识别中文扫描图片实例讲解
当我浏览http://code.google.com/p/tesseract-ocr并下载了几个文件下来之后顿时感到一头雾水,不知该如何下手.网上看到有人在linux操作系统下的实现, 如: 利用开源 ...
- OpAmp Voltage Follower/Regulator
LDO Regulator High accuracy voltage regulator Vout = 2.5V * (1 + ( 5.6 / 6.8 ) ) = 4.55V Recently th ...
- 《Go语言实战》摘录:6.5 并发 - 通道
6.5 通道
- 分享下使用 svn,测试服务器代码自动更新、线上服务器代码手动更新的配置经验
分享下使用 svn,测试服务器代码自动更新.线上服务器代码手动更新的配置经验 利用SVN的POST-COMMIT钩子自动部署代码 Linux SVN 命令详解 Linux SVN 命令详解2 使用sv ...
- cocos 主循环
CCApplication的run为主循环,负责在空闲的时候,调用CCDirector的mainloop,setAnimationInterval设置多少秒一帧.m_nAnimationInterva ...
- Cocos2d-x3.1TestCpp之NewRenderTest Demo分析
1.代码构成 VisibleRect.h VisibleRect.cpp AppDelegate.h AppDelegate.cpp HelloWorldScene.h HelloWorldScene ...
- Make the DbContext Ambient with UnitOfWorkScope(now named DbContextScope by mehdime)
The Entity Framework DbContext (or LINQ-to-SQL DataContext) is a Unit Of Work implementation. That m ...