Berkeley DB

最近用BDB写点东西，写了挺多个测试工程。列下表，也理清楚最近的思路

1.测试BDB程序，包括打开增加记录，查询记录，获取所有记录。将数据转存mysql

程序的不足，增加记录仅仅只有key和value，查询记录仅仅是简单的根据key或者同时制定key和value来查，只获取所有记录，没有获取部分记录，转存mysql消耗比较大

2.mysql和BDB插入和选择数据比较。

在java虚拟机内存相同的条件下，mysql插入数据量比较大时会报内存溢出，BDB 插入数据的量可以比mysql大。而且速度也快。查询所有记录mysql非常快，BDB相对来说比较慢，hbase只测试了下插入记录，插入一千条记录都慢得要死。单个数据选择BDB相对于mysql来说有绝对的优势。

3.修改1中的数据转存mysql。通过两台电脑，两个BDB+mysql进行。

大致流程是一个一台电脑A上有BDB，另外一台电脑B上有BDB和mysql。A上采集数据存储在BDB中。然后自己写一个文件发送的程序把BDB在硬盘上的.jdb文件发送到另外一台电脑。另外一台电脑通过BDB读取.jdb文件写进mysql里面。运行效果比较好，但是时间比直接插入一千万条记录进BDB要长。现在没看到应用领域。

4.序列化存储。实现用BDB存储学生信息

5.BDB游标的使用。对比getSearchKey,getSearchBoth,getSearchKeyRange,getSearchBothRange函数。最后写了个key的模糊查询。

以上基本上是一个星期学到的有关Berkeley DB的知识。

getSearchKey,getSearchBoth,getSearchKeyRange,getSearchBothRange对比程序

 import java.io.File;
 import java.io.UnsupportedEncodingException;
 import java.util.ArrayList;

 import com.sleepycat.je.Cursor;
 import com.sleepycat.je.Database;
 import com.sleepycat.je.DatabaseConfig;
 import com.sleepycat.je.DatabaseEntry;
 import com.sleepycat.je.Environment;
 import com.sleepycat.je.EnvironmentConfig;
 import com.sleepycat.je.LockMode;
 import com.sleepycat.je.OperationStatus;

 public class Query {
     private Environment dbEnvironment = null;
     private Database db = null;

     private void DataConstruct(int t) throws UnsupportedEncodingException {
         int Total = t;
         long BDBTime = 0;
         long start = System.currentTimeMillis();
         /*
          *
          */
         OpenBDB();
         while(Total!=0)
         {

             int k = Total;
             int v = Total+1;
             DatabaseEntry iKey = new DatabaseEntry(Integer.valueOf(k).toString().getBytes("UTF-8"));
             DatabaseEntry iValue = new DatabaseEntry(Integer.valueOf(v).toString().getBytes("UTF-8"));

             //System.out.println("Writing : Key= " + k + "  Value = " + v + " count: " + Total);

             db.put(null, iKey, iValue);
             Total--;
             }
         long end = System.currentTimeMillis();
         BDBTime = end - start;
         System.out.println(BDBTime);

     }

     //各种查询比较

     public void QueryCompare(int t) throws UnsupportedEncodingException{

         DataConstruct(t);
         //OpenBDB();

         Cursor cursor = null;
         cursor = db.openCursor(null, null);

         DatabaseEntry thekey = new DatabaseEntry();
         String key = "12345";
         thekey.setData(key.getBytes("utf-8"));

         DatabaseEntry thedata = new DatabaseEntry();

         System.out.println("   >======== getSearchKey Start ========<");
         cursor.getSearchKey(thekey, thedata, LockMode.DEFAULT);
         byte[] temp = thedata.getData();
         String res = new String(temp,"utf-8");
         System.out.println("          Key " + key + " Value " + res);
         System.out.println("   >======== getSearchKey End ========<");
         System.out.println();

         System.out.println("   >======== getSearchKeyRange Start ========<");
         cursor.getSearchKeyRange(thekey, thedata, LockMode.DEFAULT);
         temp = thedata.getData();
         res = new String(temp,"utf-8");
         System.out.println("          Key " + key + " Value " + res);
         System.out.println("   >======== getSearchKeyRange End ========<");
         System.out.println();

         //如果要执行这个查询最后不用随机数，用随机数相查到结果不易
         System.out.println("   >======== getSearchBoth Start ========<");
         thedata.setData("12346".getBytes("utf-8"));
         OperationStatus status = cursor.getSearchBoth(thekey, thedata, LockMode.DEFAULT);
         if(status == OperationStatus.SUCCESS){
             temp = thedata.getData();
             res = new String(temp,"utf-8");
             System.out.println("          Key " + key + " Value " + res);
         }
         else
             System.out.println("   Not Such Record");

         System.out.println("   >======== getSearchKeyBoth End ========<");
         System.out.println();

         System.out.println("   >======== getSearchBothRange Start ========<");
         thedata.setData("123451".getBytes("utf-8"));
         status = cursor.getSearchBoth(thekey, thedata, LockMode.DEFAULT);
         if(status == OperationStatus.SUCCESS){
             temp = thedata.getData();
             res = new String(temp,"utf-8");
             System.out.println("          Key " + key + " Value " + res);
         }
         else
             System.out.println("   Not Such Record");
         System.out.println("    RecordCount: " + cursor.count());

         System.out.println("   >======== getSearchKeyBothRange End ========<");
         System.out.println();

         /*
          * getSearchKeyRange和getSearchKey都是将游标定位到匹配key值相同的地方，如果再用Cursor.getNext无论如何都是能得到游标下一个。区别是当用到B树的时候
          * 根据getSearchKeyRange和getSearchKey和Cursor.getNext的输出结果的规律可以看出，BDB内部存储采用树形结构，类似字符数
          * getSearchBoth同时匹配Key和Value才有结果，才能返回OperationStatus.SUCCESS才算操作成功
          *
          */
         /*
          * 实现模糊查询，用getSearchKey配合getNext即可
          *
          */
         //单条件模糊查询测试代码,匹配12345,结束应该是12346

         System.out.println("   >======== Simple  Pattern key = 12345 getSearchKey Start ========<");
         thedata = new DatabaseEntry();//单条件查询，把之前的thedata设置为空
         cursor.getSearchKeyRange(thekey, thedata, LockMode.DEFAULT);
         temp = thedata.getData();  //
         res = new String(temp,"utf-8");
         System.out.println("          Key " + key + " Value " + res);
         thekey = new DatabaseEntry();
         while(cursor.getNext(thekey, thedata, LockMode.DEFAULT) != OperationStatus.NOTFOUND){        //结束条件是OperationStatus.NOTFOUND
             temp = thedata.getData();
             byte[] tempkey = thekey.getData();
             res = new String(temp,"utf-8");
             String res1 = new String(tempkey,"utf-8");
             if(res1.equals("12346"))
                 break;
             System.out.println("          Key " + res1 + " Value " + res);
         }
         System.out.println("   >======== Simple  getSearchKey End ========<");
         System.out.println();

         //多条件模糊查询，如果没有相应的机制只能自己编写来查了,
         //失败

         System.out.println("   >======== multiplication  Pattern Key=12345 Value = 123451 getSearchBothRange Start ========<");
         thedata.setData("123451".getBytes("utf-8"));
         thekey.setData(key.getBytes("utf-8"));
         status = cursor.getSearchBothRange(thekey, thedata, LockMode.DEFAULT);
         //首先会去匹配12345 123456
         if(status==OperationStatus.SUCCESS){
             temp = thedata.getData();  //
             res = new String(temp,"utf-8");
             System.out.println("          Key " + key + " Value " + res);
             while(cursor.getNext(thekey, thedata, LockMode.DEFAULT) != OperationStatus.NOTFOUND){        //结束条件是OperationStatus.NOTFOUND
                 temp = thedata.getData();
                 byte[] tempkey = thekey.getData();
                 res = new String(temp,"utf-8");
                 String res1 = new String(tempkey,"utf-8");
                 if(res1.equals("12346"))
                     break;
                 System.out.println("          Key " + res1 + " Value " + res);
             }
         }
         else
             System.out.println("   Not Such Record");

         System.out.println("   >======== multiplication  getSearchBothRange End ========<");
         System.out.println();
         cursor.close();
         CloseBDB();

     }

     //创建数据库
     public void OpenBDB(){
         EnvironmentConfig envConfig = new EnvironmentConfig();
         envConfig.setAllowCreate(true);
         dbEnvironment = new Environment( new File("D:/dbEnv"), envConfig );
          DatabaseConfig dbConfig = new DatabaseConfig();

          dbConfig.setAllowCreate(true);
          dbConfig.setSortedDuplicates(true);
          db = dbEnvironment.openDatabase(null,"BDB", dbConfig);
          System.out.println("打开数据库成功");
     }
     //关闭数据库
     public void CloseBDB(){
         if(db != null){
             db.close();
         }
         if(dbEnvironment != null){
             dbEnvironment.close();
         }
     }

     //获取数据库中所有记录，
     public ArrayList<String> getEveryItem() throws UnsupportedEncodingException {
         Cursor cursor = null;
         OpenBDB();
         cursor= db.openCursor(null,null);
         DatabaseEntry theKey=null;
         DatabaseEntry theData=null;
         theKey = new DatabaseEntry();
         theData = new DatabaseEntry();
         long total = 0;
         ArrayList<String> list = new ArrayList<String>(); 

         while (cursor.getNext(theKey, theData, LockMode.DEFAULT) == OperationStatus.SUCCESS) {
             //System.out.println("Reading: Key: " + new String( theKey.getData()) + "Value: " + new String(theData.getData()));
             list.add(new String( theKey.getData()) );
             list.add(new String( theData.getData()) );
             total++;
         }
         cursor.close();
         System.out.println("total:" + total);
         return list;

     }

     //getSearchKey

 }

序列化创建学生信息

import java.io.File;

import com.sleepycat.bind.EntryBinding;
import com.sleepycat.bind.serial.SerialBinding;
import com.sleepycat.bind.serial.StoredClassCatalog;
import com.sleepycat.je.Cursor;
import com.sleepycat.je.Database;
import com.sleepycat.je.DatabaseConfig;
import com.sleepycat.je.DatabaseEntry;
import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.je.LockMode;
import com.sleepycat.je.OperationStatus;

public class SeriableDemo {

    public static void main(String args[]){
      Environment dbEnv = null;
      Database db = null;
      Cursor cursor=null;
      try {

       EnvironmentConfig envconfig = new EnvironmentConfig();
       envconfig.setAllowCreate(true);
       dbEnv = new Environment(new File("D:/dbEnv/"),envconfig);

       DatabaseConfig dbconfig = new DatabaseConfig();
       dbconfig.setAllowCreate(true);
       dbconfig.setReadOnly(false);
       dbconfig.setSortedDuplicates(false);
       //data
       db = dbEnv.openDatabase(null, "Student_Data", dbconfig);
       //class
       Database myclassdb=dbEnv.openDatabase(null, "Student_class", dbconfig);
       //cataglog存储类信息
       StoredClassCatalog classCatalog =new StoredClassCatalog(myclassdb);

     //创建绑定对象
       EntryBinding dataBinging=new SerialBinding(classCatalog,Student.class);

       DatabaseEntry thekey=new DatabaseEntry("".getBytes("utf-8"));
       DatabaseEntry thedata=new DatabaseEntry();
       Student st=new Student();

       st.setStudentId();
       st.setStudentName("邹颖玖");
       st.setStudentAge();
       st.setStudentDept("计算机科学与工程");

       //绑定数据
       dataBinging.objectToEntry(st, thedata);
       //插入
       db.put(null, thekey, thedata);

       DatabaseEntry data = new DatabaseEntry();
       //DatabaseEntry key = new DatabaseEntry();
       DatabaseEntry key = new DatabaseEntry("".getBytes("utf-8"));
       Student st1 = new Student();
       st1.setStudentId();
       dataBinging.objectToEntry(st1, data);
       //获取
       OperationStatus status=db.get(null, key, data, LockMode.DEFAULT);
       if(status==OperationStatus.SUCCESS){
        Student s=(Student) dataBinging.entryToObject(thedata);
        System.out.println(s.getStudentId()+"   "+s.getStudentName()+"    "+
        s.getStudentAge() + "   " + s.getStudentDept());
       }

      }catch(Exception e){
          e.printStackTrace();
      }
    }
}

上面代码用到学生类。

import java.io.Serializable;

public class Student implements Serializable {

    private long StudentId;
    private String StudentName;
    private int StudentAge;
    private String StudentDept;

    public long getStudentId() {
        return StudentId;
    }

    public void setStudentId(long studentId) {
        StudentId = studentId;
    }

    public String getStudentName() {
        return StudentName;
    }

    public void setStudentName(String studentName) {
        StudentName = studentName;
    }

    public int getStudentAge() {
        return StudentAge;
    }

    public void setStudentAge(int studentAge) {
        StudentAge = studentAge;
    }

    public String getStudentDept() {
        return StudentDept;
    }

    public void setStudentDept(String studentDept) {
        StudentDept = studentDept;
    }

}

以上是老师将来项目需要，估计以后我也有这种需要。没有时间，还是想自己实践弄一套音乐推荐系统出来。