sphinx（coreseek）——1、增量索引

首先介绍一下     CoreSeek/Sphinx的发布包

indexer: 用于创建全文索引;
    search: 一个简单的命令行(CLI) 的测试程序，用于测试全文索引;
    searchd: 一个守护进程，其他软件（例如WEB程序）可以通过这个守护进程进行全文检索;
    sphinxapi: 一系列searchd 的客户端API 库，用于流行的Web脚本开发语言(PHP, Python, Perl, Ruby, Java).
    spelldump: 一个简单的命令行工具，用于从 ispell 或者 MySpell (OpenOffice内置绑定) 格式的字典中提取词条。当使用 wordforms时可用这些词条对索引进行定制.
    indextool: 工具程序，用来转储关于索引的多项调试信息。此工具是从版本Coreseek 3.1(Sphinx 0.9.9-rc2)开始加入的。
    mmseg: 工具程序和库，Coreseek用于提供中文分词和词典处理。

在Sphinx+LibMMSeg搭建中文全文搜索引擎_安装配置中安装试验了Sphinx 的使用，但是还有几方面的问题有待处理。

1. 动态增量索引
2. 区段查询
3. 实时索引
4. 匹配到的字段在前端界面表现

本篇主要是对动态增量更新的一些研究
在利用 Sphinx 做搜索引擎的时候，一般他的索引建立构成有如下几个部分：

1. 固定不变的主索引
2. 增量索引重建
3. 索引数据合并

• 在实际操作中，需要为增量索引的建立创建辅助表，这样才可以记住最后建立索引的记录ID,做实际的增量部分的索引建立。

CREATE TABLE `sph_counter` (
  `counter_id` int(11) NOT NULL,
  `max_doc_id` int(11) NOT NULL,
  PRIMARY KEY (`counter_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

• 在主索引的数据源中作如下方式的取数据设置

source src
{
        # data source type. mandatory, no default value
        # known types are mysql, pgsql, mssql, xmlpipe, xmlpipe2, odbc
        type                    = mysql
        sql_host                = localhost
        sql_user                = root
        sql_pass                = xxxxx
        sql_db                  = test
        sql_port                =   # optional, default is
        sql_query_pre          = SET NAMES utf8
        sql_query_pre          = SET SESSION query_cache_type=OFF
        sql_query_pre          = REPLACE INTO sph_counter SELECT , MAX(id) FROM cn
        sql_query               = SELECT id,title,content from cn where id<=(SELECT max_doc_id FROM sph_counter WHERE counter_id=)
        sql_query_info          = SELECT * FROM cn WHERE id=$id
}

• 在增量索引的数据源中作如下方式的取数据设置，需要注意的是sql_query_pre要和主索引数量相同，不然查询结果不是想要的内容

#表示增量数据源
source moresrc : src
{
        sql_query_pre          = SET NAMES utf8
        sql_query_pre          = SET SESSION query_cache_type=OFF
        sql_query              = SELECT id,title,content from cn where id>(SELECT max_doc_id FROM sph_counter WHERE counter_id=)
        #sql_ranged_throttle    =
}

• 主索引index定义配置

index src
{
        source                  = src
        path                    = /usr/local/coreseek/var/data/test1
        docinfo                 = extern
        mlock                   =
        morphology              = none
        # 启用中文分词功能source 数据源中需要 设置读取的数据编码字符集为UTF-，否则无
法正确处理；如果是xml，则正确输出为UTF-8编码格式即可；如果是MySQL，则设置读取数据输出
字符集为UTF-8即可
        charset_type            = zh_cn.utf-
        # 中文分词词库位置
        charset_dictpath        =/usr/local/mmseg/etc/
}

• 增量索引index定义配置

index moresrc : src
{
        source = moresrc
        path                    = /usr/local/coreseek/var/data/moresrc
        morphology              = stem_en
}
indexer
{
        # memory limit, in bytes, kiloytes (16384K) or megabytes (256M)
        # optional, default is 32M, max is 2047M, recommended is 256M to 1024M
        mem_limit               = 32M
}

• 创建更新全部索引

root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft.increment.conf  --all --rotate
Coreseek Fulltext 4.1 [ Sphinx 2.0.-dev (r2922)]
Copyright (c) -,
Beijing Choice Software Technologies Inc (http://www.coreseek.com)

 using config file '/usr/local/coreseek/etc/csft.increment.conf'...
indexing index 'src'...
WARNING: Attribute count is : switching to none docinfo
collected  docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total  docs,  bytes
total 0.011 sec,  bytes/sec, 531.82 docs/sec
indexing index 'moresrc'...
WARNING: Attribute count is : switching to none docinfo
collected  docs, 0.0 MB
total  docs,  bytes
total 0.001 sec,  bytes/sec, 0.00 docs/sec
total  reads, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
total  writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=).

• 更新增量索引

数据库中添加数据  然后运行，更新增量索引    记得  执行 indeser   search   还有启动searchd时候要    要指定-c    配置文件。

root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer moresrc -c /usr/local/coreseek/etc/csft.increment.conf --rotate
Coreseek Fulltext 4.1 [ Sphinx 2.0.-dev (r2922)]
Copyright (c) -,
Beijing Choice Software Technologies Inc (http://www.coreseek.com)

 using config file '/usr/local/coreseek/etc/csft.increment.conf'...
indexing index 'moresrc'...
WARNING: Attribute count is : switching to none docinfo
collected  docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total  docs,  bytes
total 0.008 sec,  bytes/sec, 115.24 docs/sec
total  reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total  writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=).

• 合并增量索引到主索引

root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# /usr/local/coreseek/bin/indexer --merge src moresrc  -c /usr/local/coreseek/etc/csft.increment.conf --rotate
Coreseek Fulltext 4.1 [ Sphinx 2.0.-dev (r2922)]
Copyright (c) -,
Beijing Choice Software Technologies Inc (http://www.coreseek.com)

 using config file '/usr/local/coreseek/etc/csft.increment.conf'...
read 0.0 of 0.0 MB, 100.0% done
merged 0.0 Kwords
merged in 0.000 sec
total  reads, 0.000 sec, 6.4 kb/call avg, 0.0 msec/call avg
total  writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=).

• 测试数据如下所示

root@timeless-HP-Pavilion-g4-Notebook-PC:/usr/local/coreseek/bin# ./search -c /usr/local/coreseek/etc/csft.increment.conf  测试
Coreseek Fulltext 4.1 [ Sphinx 2.0.-dev (r2922)]
Copyright (c) -,
Beijing Choice Software Technologies Inc (http://www.coreseek.com)

 using config file '/usr/local/coreseek/etc/csft.increment.conf'...
index 'src': query '测试 ': returned  matches of  total in 0.000 sec

displaying matches:
. document=, weight=
    id=
    title=?????
    content=?? ?????? ????  ??
    addtime=
. document=, weight=
    id=
    title=??
    content=????????
    addtime=
. document=, weight=
    id=
    title=???
    content=????????    ?????? ????
    addtime=

words:
. '测试':  documents,  hits

index 'moresrc': query '测试 ': returned  matches of  total in 0.000 sec

displaying matches:
. document=, weight=
    id=
    title=??
    content=????????
    addtime=

words:
. '测试':  documents,  hits

通过测试数据看到    index  moresrc  增量索引 查询到  “测试” 关键字 5 次    合并之后的住索引变为 14个匹配。 之前是 9个 匹配。