zookeeper 删除snapshot和transaction log的源码解读

转载请注明源地址http://www.cnblogs.com/dongxiao-yang/p/4910059.html

zookeeper具有自动清除快照日志和事务日志的工能，可以在配置文件设置autopurge.purgeInterval来实现，问题是这个属性的时间单位是小时，

有些情况下，一小时的日志过大（比如把事务日志放到内存），需要手动删除，所以需要研究下zk删除日志文件的源码。

清理日志主类:org.apache.zookeeper.server.PurgeTxnLog,包含如下几个静态工具方法

static void printUsage(){

System.out.println("PurgeTxnLog dataLogDir [snapDir] -n count");

System.out.println("\tdataLogDir -- path to the txn log directory");

System.out.println("\tsnapDir -- path to the snapshot directory");

System.out.println("\tcount -- the number of old snaps/logs you want to keep");

System.exit(1);

}

常见的帮助方法，告诉使用者参数的传入顺序，其中snapdir参数为可选，假如两种日志配置在同一个路径下，只传一个路径参数就好。

main方法，没什么好说的，只是解析参数。

public static void purge(File dataDir, File snapDir, int num) throws IOException {

if (num < 3) {

throw new IllegalArgumentException("count should be greater than 3");

}

FileTxnSnapLog txnLog = new FileTxnSnapLog(dataDir, snapDir);

List<File> snaps = txnLog.findNRecentSnapshots(num);

retainNRecentSnapshots(txnLog, snaps);

}

删除文件的主方法，主要分两个部分

1：txnLog.findNRecentSnapshots(num);

找到需要保留的文件

主要逻辑代码为

public List<File> findNRecentSnapshots(int n) throws IOException {

List<File> files = Util.sortDataDir(snapDir.listFiles(), "snapshot", false);

int i = 0;

List<File> list = new ArrayList<File>();

for (File f: files) {

if (i==n)

break;

i++;

list.add(f);

}

return list;

}

private static class DataDirFileComparator

implements Comparator<File>, Serializable

{

private static final long serialVersionUID = -2648639884525140318L;

private String prefix;

private boolean ascending;

public DataDirFileComparator(String prefix, boolean ascending) {

this.prefix = prefix;

this.ascending = ascending;

}

public int compare(File o1, File o2) {

long z1 = Util.getZxidFromName(o1.getName(), prefix);

long z2 = Util.getZxidFromName(o2.getName(), prefix);

int result = z1 < z2 ? -1 : (z1 > z2 ? 1 : 0);

return ascending ? result : -result;

}

}

/**

* Sort the list of files. Recency as determined by the version component

* of the file name.

*

* @param files array of files

* @param prefix files not matching this prefix are assumed to have a

* version = -1)

* @param ascending true sorted in ascending order, false results in

* descending order

* @return sorted input files

*/

public static List<File> sortDataDir(File[] files, String prefix, boolean ascending)

{

if(files==null)

return new ArrayList<File>(0);

List<File> filelist = Arrays.asList(files);

Collections.sort(filelist, new DataDirFileComparator(prefix, ascending));

return filelist;

}

2 删除文件

// VisibleForTesting

static void retainNRecentSnapshots(FileTxnSnapLog txnLog, List<File> snaps) {

// found any valid recent snapshots?

if (snaps.size() == 0)

return;

File snapShot = snaps.get(snaps.size() -1);

int ii=snaps.size() -1;

System.out.println(ii);

final long leastZxidToBeRetain = Util.getZxidFromName(

snapShot.getName(), PREFIX_SNAPSHOT);

class MyFileFilter implements FileFilter{

private final String prefix;

MyFileFilter(String prefix){

this.prefix=prefix;

}

public boolean accept(File f){

if(!f.getName().startsWith(prefix + "."))

return false;

long fZxid = Util.getZxidFromName(f.getName(), prefix);

if (fZxid >= leastZxidToBeRetain) {

return false;

}

return true;

}

}

// add all non-excluded log files

List<File> files = new ArrayList<File>(Arrays.asList(txnLog

.getDataDir().listFiles(new MyFileFilter(PREFIX_LOG))));

// add all non-excluded snapshot files to the deletion list

files.addAll(Arrays.asList(txnLog.getSnapDir().listFiles(

new MyFileFilter(PREFIX_SNAPSHOT))));

// remove the old files

for(File f: files)

{

System.out.println("Removing file: "+

DateFormat.getDateTimeInstance().format(f.lastModified())+

"\t"+f.getPath());

if(!f.delete()){

System.err.println("Failed to remove "+f.getPath());

}

}

}

Util.getZxidFromName工具方法代码

public static long getZxidFromName(String name, String prefix) {

long zxid = -1;

String nameParts[] = name.split("\\.");

if (nameParts.length == 2 && nameParts[0].equals(prefix)) {

try {

zxid = Long.parseLong(nameParts[1], 16);

} catch (NumberFormatException e) {

}

}

return zxid;

}