Kyoto Cabinet 使用及原理
Kyoto Cabinet 基本规格书
如果你知道 Tokyo Cabinet ,那么就应该知道 Kyoto Cabinet,因为他们都是同一个作者(平林幹雄)开发出来的 Key-Value 数据库。
- 改进的空间效率:更小的数据库文件
- 改进的时间效率:更快的处理速度
- 改进的并行性:多线程环境下的高性能
- 改进的可用性:简单的API
- 改进的健壮性:即使在灾难情况下数据库文件也不会损坏
- 支持64位架构:巨大的内存空间和数据库文件可用
- 改进的空间效率:更小的数据库文件
- 改进的并行性:多线程环境下的高性能
- 改进的可移植性:对底层的抽象来支持 非POSIX系统
- 改进的可用性:简单的API,面向对象的设计
- 改进的健壮性:即使在灾难情况下数据库文件也不会损坏
- tune_buckets:设置hash数据库的 bucket 数量
- tune_options:设置可选特性(optional features)
- tune_buckets:设置hash数据库的 bucket 数量
- tune_compressor:设置数据压缩方法
- cap_count:设置记录数的容量
- cap_size:设置内存使用的容量
db.tune_buckets(10LL * * );
db.cap_count(10LL * * );
db.cap_size(8LL << );
db.open(...);
- tune_page:设置每个页大小
- tune_page_cache:设置页缓存(page cache)容量大小
- tune_comparator:设置记录比较器
db.tune_options(GrassDB::TCCOMPESS);
db.tune_buckets(500LL * );
db.tune_page();
db.tune_page_cache(1LL << );
db.open(...);
- tune_alignment:设置记录的对齐幂数
- tune_fbp:设置空闲块池的容量幂数
- tune_options:设置可选特性
- tune_buckets:设置哈希表的bucket数量
- tune_map:设置内部内存映射区域的大小
- tune_defrag:设置自动碎片整理的单位步数
- tune_compressor:设置数据压缩器
db.tune_alignment();
db.tune_options(HashDB::TSMALL | HashDB::TLINEAR);
db.tune_buckets(10LL * );
db.tune_defrag();
db.open(...);
db.tune_options(HashDB::TLINEAR);
db.tune_buckets(20LL * * * );
db.tune_map(300LL << );
db.open(...);
- tune_page:设置每个页大小
- tune_page_cache:设置页缓存(page cache)容量大小
- tune_comparator:设置记录比较器
db.tune_options(TreeDB::TLINEAR | TreeDB::TCCOMPESS); db.tune_buckets(1LL * 1000); db.tune_defrag(8); db.tune_page(32768); db.open(...);
db.tune_options(TreeDB::TLINEAR); db.tune_buckets(1LL * 1000 * 1000 * 1000); db.tune_map(300LL << 30); db.tune_page_cache(8LL << 30); db.open(...);
- tune_options:设置可选特性
- 时间效率:CacheDB > StashDB > ProtoHashDB > ProtoTreeDB > GrassDB
- 空间效率:GrassDB > StashDB > CacheDB > ProtoHashDB > ProtoTreeDB
- 时间效率:HashDB > TreeDB > DirDB > ForestDB
- 空间效率:TreeDB > HashDB > ForestDB > DirDB
db.begin_transaction();
db.set("japan", "tokyo");
db.set("korea", "seoul");
db.end_transaction();
db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE | HashDB::OAUTOTRAN);
db.set("japan", "tokyo");
db.set("china", "beijing");
db.copy("backup.kch");
class BackupImpl : public FileProcessor {
bool process(const std::string& path, int64_t size, int64_t count) {
char cmd[];
sprintf(cmd, "snapshot.sh %s", path.c_str());
return system(cmd) == ;
}
} proc;
db.synchronize(&proc);
db.dump_snapshot("backup.kcss");
db.load_snapshot("backup.kcss");
ArcfourCompressor comp;
comp.set_key("foobarbaz", );
TreeDB db;
db.tune_options(kc::TreeDB::TCOMPRESS);
db.tune_compressor(&comp);
db.open(...);
comp.begin_cycle((uint64_t)getpid() << + (uint64_t)time());
PolyDB db;
db.open("casket.kct#zcomp=arc#zkey=foobarbaz", ...);
GNU General Public License
Kyoto Cabinet is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.
Kyoto Cabinet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see `http://www.gnu.org/licenses/
'.
FOSS License Exception
The FOSS License Exception is also provided in order to accommodate products under other free and open source licenses. See the body text for details.
Commercial License
If you use Kyoto Cabinet within a proprietary software, a commercial license is required.
The commercial license allows you to utilize Kyoto Cabinet by including it in your applications for purpose of developing and producing your applications and to utilize Kyoto Cabinet in order to transfer, sale, rent, lease, distribute or sublicense your applications to any third parties. See the license guide for details.
Author
Kyoto Cabinet was written and is maintained by FAL Labs. You can contact the author by e-mail to `info@fallabs.com
'.
Examples
C++Example:
The following code is an example to use a polymorphic database.
#include <kcpolydb.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open("casket.kch", PolyDB::OWRITER | PolyDB::OCREATE)) {
cerr << "open error: " << db.error().name() << endl;
} // store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")) {
cerr << "set error: " << db.error().name() << endl;
} // retrieve a record
string value;
if (db.get("foo", &value)) {
cout << value << endl;
} else {
cerr << "get error: " << db.error().name() << endl;
} // traverse records
DB::Cursor* cur = db.cursor();
cur->jump();
string ckey, cvalue;
while (cur->get(&ckey, &cvalue, true)) {
cout << ckey << ":" << cvalue << endl;
}
delete cur; // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The following code is a more complex example, which uses the Visitor pattern.
#include <kcpolydb.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open("casket.kch", PolyDB::OREADER)) {
cerr << "open error: " << db.error().name() << endl;
} // define the visitor
class VisitorImpl : public DB::Visitor {
// call back function for an existing record
const char* visit_full(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t *sp) {
cout << string(kbuf, ksiz) << ":" << string(vbuf, vsiz) << endl;
return NOP;
}
// call back function for an empty record space
const char* visit_empty(const char* kbuf, size_t ksiz, size_t *sp) {
cerr << string(kbuf, ksiz) << " is missing" << endl;
return NOP;
}
} visitor; // retrieve a record with visitor
if (!db.accept("foo", , &visitor, false) ||
!db.accept("dummy", , &visitor, false)) {
cerr << "accept error: " << db.error().name() << endl;
} // traverse records with visitor
if (!db.iterate(&visitor, false)) {
cerr << "iterate error: " << db.error().name() << endl;
} // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The following code is an example of word counting with the MapReduce framework. #include <kcpolydb.h>
#include <kcdbext.h> using namespace std;
using namespace kyotocabinet; // main routine
int main(int argc, char** argv) { // create the database object
PolyDB db; // open the database
if (!db.open()) {
cerr << "open error: " << db.error().name() << endl;
} // store records
db.set("", "this is a pen");
db.set("", "what a beautiful pen this is");
db.set("", "she is beautiful"); // define the mapper and the reducer
class MapReduceImpl : public MapReduce {
// call back function of the mapper
bool map(const char* kbuf, size_t ksiz, const char* vbuf, size_t vsiz) {
vector<string> words;
strsplit(string(vbuf, vsiz), ' ', &words);
for (vector<string>::iterator it = words.begin();
it != words.end(); it++) {
emit(it->data(), it->size(), "", );
}
return true;
}
// call back function of the reducer
bool reduce(const char* kbuf, size_t ksiz, ValueIterator* iter) {
size_t count = ;
const char* vbuf;
size_t vsiz;
while ((vbuf = iter->next(&vsiz)) != NULL) {
count++;
}
cout << string(kbuf, ksiz) << ": " << count << endl;
return true;
}
} mr; // execute the MapReduce process
if (!mr.execute(&db)) {
cerr << "MapReduce error: " << db.error().name() << endl;
} // close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
} return ;
}
The C language binding is also provided as a wrapper of the polymorphic database API. The following code is an example.
#include <kclangc.h> /* call back function for an existing record */
const char* visitfull(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t *sp, void* opq) {
fwrite(kbuf, , ksiz, stdout);
printf(":");
fwrite(vbuf, , vsiz, stdout);
printf("\n");
return KCVISNOP;
} /* call back function for an empty record space */
const char* visitempty(const char* kbuf, size_t ksiz, size_t *sp, void* opq) {
fwrite(kbuf, , ksiz, stdout);
printf(" is missing\n");
return KCVISNOP;
} /* main routine */
int main(int argc, char** argv) {
KCDB* db;
KCCUR* cur;
char *kbuf, *vbuf;
size_t ksiz, vsiz;
const char *cvbuf; /* create the database object */
db = kcdbnew(); /* open the database */
if (!kcdbopen(db, "casket.kch", KCOWRITER | KCOCREATE)) {
fprintf(stderr, "open error: %s\n", kcecodename(kcdbecode(db)));
} /* store records */
if (!kcdbset(db, "foo", , "hop", ) ||
!kcdbset(db, "bar", , "step", ) ||
!kcdbset(db, "baz", , "jump", )) {
fprintf(stderr, "set error: %s\n", kcecodename(kcdbecode(db)));
} /* retrieve a record */
vbuf = kcdbget(db, "foo", , &vsiz);
if (vbuf) {
printf("%s\n", vbuf);
kcfree(vbuf);
} else {
fprintf(stderr, "get error: %s\n", kcecodename(kcdbecode(db)));
} /* traverse records */
cur = kcdbcursor(db);
kccurjump(cur);
while ((kbuf = kccurget(cur, &ksiz, &cvbuf, &vsiz, )) != NULL) {
printf("%s:%s\n", kbuf, cvbuf);
kcfree(kbuf);
}
kccurdel(cur); /* retrieve a record with visitor */
if (!kcdbaccept(db, "foo", , visitfull, visitempty, NULL, ) ||
!kcdbaccept(db, "dummy", , visitfull, visitempty, NULL, )) {
fprintf(stderr, "accept error: %s\n", kcecodename(kcdbecode(db)));
} /* traverse records with visitor */
if (!kcdbiterate(db, visitfull, NULL, )) {
fprintf(stderr, "iterate error: %s\n", kcecodename(kcdbecode(db)));
} /* close the database */
if (!kcdbclose(db)) {
fprintf(stderr, "close error: %s\n", kcecodename(kcdbecode(db)));
} /* delete the database object */
kcdbdel(db); return ;
}
Java example:
The following code is a typical example to use a database.
import kyotocabinet.*; public class KCDBEX1 {
public static void main(String[] args) { // create the object
DB db = new DB(); // open the database
if (!db.open("casket.kch", DB.OWRITER | DB.OCREATE)){
System.err.println("open error: " + db.error());
} // store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")){
System.err.println("set error: " + db.error());
} // retrieve records
String value = db.get("foo");
if (value != null){
System.out.println(value);
} else {
System.err.println("set error: " + db.error());
} // traverse records
Cursor cur = db.cursor();
cur.jump();
String[] rec;
while ((rec = cur.get_str(true)) != null) {
System.out.println(rec[0] + ":" + rec[1]);
}
cur.disable(); // close the database
if(!db.close()){
System.err.println("close error: " + db.error());
} }
}
The following code is a more complex example, which uses the Visitor pattern.
import kyotocabinet.*; public class KCDBEX2 {
public static void main(String[] args) { // create the object
DB db = new DB(); // open the database
if (!db.open("casket.kch", DB.OREADER)) {
System.err.println("open error: " + db.error());
} // define the visitor
class VisitorImpl implements Visitor {
public byte[] visit_full(byte[] key, byte[] value) {
System.out.println(new String(key) + ":" + new String(value));
return NOP;
}
public byte[] visit_empty(byte[] key) {
System.err.println(new String(key) + " is missing");
return NOP;
}
}
Visitor visitor = new VisitorImpl(); // retrieve a record with visitor
if (!db.accept("foo".getBytes(), visitor, false) ||
!db.accept("dummy".getBytes(), visitor, false)) {
System.err.println("accept error: " + db.error());
} // traverse records with visitor
if (!db.iterate(visitor, false)) {
System.err.println("iterate error: " + db.error());
} // close the database
if(!db.close()){
System.err.println("close error: " + db.error());
} }
}
Kyoto Cabinet 使用及原理的更多相关文章
- Kyoto Cabinet(DBM) + Kyoto Tycoon(网络层)
项目原地址kyotocabinet: http://fallabs.com/kyotocabinet/ kyototycoon: http://fallabs.com/kyototyc ...
- 实现键值对存储(三):Kyoto Cabinet 和LevelDB的架构比較分析
译自 Emmanuel Goossaert (CodeCapsule.com) 在本文中,我将会逐组件地把Kyoto Cabinet 和 LevelDB的架构过一遍.目标和本系列第二部分讲的差点儿相 ...
- Kyoto Cabinet--nosql型单机数据库
摘要: Kyoto Cabinet是轻量级nosql型本地内存数据库 简介 Kyoto Cabinet是一个数据库管理的 lib,是 Tokyo Cabinet 的改进版本.数据库是一个简单的包含记录 ...
- 大型web系统数据缓存设计
1. 前言 在高访问量的web系统中,缓存几乎是离不开的:但是一个适当.高效的缓存方案设计却并不容易:所以接下来将讨论一下应用系统缓存的设计方面应该注意哪些东西,包括缓存的选型.常见缓存系统的特点和数 ...
- NoSQL数据库笔谈(转)
NoSQL数据库笔谈 databases , appdir , node , paper颜开 , v0.2 , 2010.2 序 思想篇 CAP 最终一致性 变体 BASE 其他 I/O的五分钟法则 ...
- 大数据架构师NoSQL建模技术
从数据建模的角度对NoSQL家族系统做了比较简单的比较,并简要介绍几种常见建模技术. 1.前言 为了适应大数据应用场景的要求,Hadoop以及NoSQL等与传统企业平台完全不同的新兴架构迅速地崛起.而 ...
- H2的MVStore
翻译自http://www.h2database.com/html/mvstore.html 转载请著名出处,及译者信息. 第一次翻译,诸多不妥请谅解,谢谢. 概述 MVStore是一个持久化的.日志 ...
- 大型web系统数据缓存设计-l转载
原文地址:http://www.wmyouxi.com/a/60368.html#ixzz3tGYG9JwC 1. 前言 在高访问量的web系统中,缓存几乎是离不开的:但是一个适当.高效的缓存方案设计 ...
- NoSQL数据库笔谈
NoSQL数据库笔谈 databases , appdir , node , paper颜开 , v0.2 , 2010.2 序 思想篇 CAP 最终一致性 变体 BASE 其他 I/O的五分钟法则 ...
随机推荐
- MVC+easyui,写个树
前言:网上关于编写组织机构树的教程并不少,我第一次写树的时候也是在网上借鉴别人的技术,走了一些弯路写下了树.是因为这些教程都不是很全面,对于编程新手来说跳跃性太强.所以趁着闲暇时期,我用心的写个树,供 ...
- Oracle - 数据库巡检脚本
分享一个oracle数据库巡检脚本,欢迎大家使用,希望大家在用的过程中发现脚本中的错误并提出改进意见. -- 数据库巡检脚本 -- 版本号2.1 -- 该脚本仅对数据库的做一个初步的巡检,具体的优化方 ...
- PBR Step by Step(四)Lambertian反射模型
光照可分为局部光照和全局光照. 局部光照:直接照射到物体表面的光照 全局光照:物体表面受周围环境影响的光照 左图中点x接收到周围环境的光线照射,来自周围表面的反射光照称为全局光照:右图中点x接收来自太 ...
- 一个简单的日志函数C++
有时候程序总是会发生意想不到的情况,为了方便排查错误的情况,还是写日志比较方便.这里自己写了一个简单的函数,能实现基本的功能. BOOL WriteLog(char * DataBuffer) { C ...
- markdown编辑器使用指南
欢迎使用Markdown编辑器写博客 本Markdown编辑器使用StackEdit修改而来,用它写博客,将会带来全新的体验哦: Markdown和扩展Markdown简洁的语法 代码块高亮 图片链接 ...
- BZOJ4280 : [ONTAK2015]Stumilowy sad
线段树每个区间维护上下界以及要整体增加的标记即可,时间复杂度$O(m\log n)$. #include<cstdio> #define inf 1500000000 int n,m,op ...
- bzoj1503 郁闷的出纳员
Description OIER公司是一家大型专业化软件公司,有着数以万计的员工.作为一名出纳员,我的任务之一便是统计每位员工的 工资.这本来是一份不错的工作,但是令人郁闷的是,我们的老板反复无常,经 ...
- 喵哈哈村的魔法考试 Round #3 (Div.2) 题解
A 题解:保证一个三角形的话,得两边之和大于第三边才行,所以都拿来判一判就好了. #include <iostream> using namespace std; int main(){ ...
- ios数据保存
- 初始化collectionViewCell
#import <UIKit/UIKit.h> @interface TonyCollectionViewCell : UICollectionViewCell @property UII ...