【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件
nutch-site.xml
在nutch2.2.1中,有两份配置文件:nutch-default.xml与nutch-site.xml。
其中前者是nutch自带的默认属性,一般情况下不要修改。
如果需要修改默认属性,可以在nutch-site.xml中增加一个同名的属性,并修改其值。nutch-site.xml中的属性值会覆盖nutch-default.xml中的值。
1、db.ignore.external.links
若为true,则只抓取本域名内的网页,忽略外部链接。
可以在 regex-urlfilter.txt中增加过滤器达到同样效果,但如果过滤器过多,如几千个,则会大大影响nutch的性能。
<property>
<name>db.ignore.external.links</name>
<value>true</value>
<description>If true, outlinks leading from a page to external hosts
will be ignored. This is an effective way to limit the crawl to include
only initially injected hosts, without creating complex URLFilters.
</description>
</property>
2、fetcher.parse
能否在抓取的同时进行解释:可以,但不 建议这样做。
<property>
<name>fetcher.parse</name>
<value>false</value>
<description>If true, fetcher will parse content. NOTE: previous releases would
default to true. Since 2.0 this is set to false as a safer default.</description>
</property>
官方解释
N.B. In a parsing fetcher, outlinks are processed in the reduce phase (at least when outlinks are followed). If a fetcher's reducer stalls you may run out of memory or disk space,
usually after a very long reduce job. Behaviour typical to this is usually
observed in this situation.
In summary, if it is possible, users are advised not to use a parsing fetcher as it is heavy on IO and often leads to the above outcome.
3、db.max.outlinks.per.page
默认情况下,Nutch只抓取某个网页的100个外部链接,导致部分链接无法抓取。若要改变此情况,可以修改此配置项。
<property>
<name>db.max.outlinks.per.page</name>
<value>100</value>
<description>The maximum number of outlinks that we'll process for a page. If this value is nonnegative (>=0), at most db.max.outlinks.per.page outlinks will be processed for a page; otherwise, all outlinks will be processed.
</description>
</property>
官方说明如下:http://wiki.apache.org/nutch/FAQ/
Nutch doesn't crawl relative URLs? Some pages are not indexed but my regex file and everything else is okay - what is going on?
The crawl tool has a default limitation of 100 outlinks of one page that are being fetched. To overcome this limitation change thedb.max.outlinks.per.page property to a higher
value or simply -1 (unlimited).
file: conf/nutch-default.xml
<property>
<name>db.max.outlinks.per.page</name>
<value>-1</value>
<description>The maximum number of outlinks that we'll process for a page.
If this value is nonnegative (>=0), at most db.max.outlinks.per.page outlinks
will be processed for a page; otherwise, all outlinks will be processed.
</description>
</property>
see also: http://www.mail-archive.com/nutch-user@lucene.apache.org/msg08665.html
4、file.content.limit http.content.limit ftp.content.limit
默认情况下,nutch只抓取网页的前65536个字节,之后的内容将被丢弃。
但对于某些大型网站,首页的内容远远不止65536个字节,甚至前面65536个字节里面均是一些布局信息,并没有任何的超链接。
因此修改默认值如下:
<property>
<name>file.content.limit</name>
<value>-1</value>
<description>The length limit for downloaded content using the file
protocol, in bytes. If this value is nonnegative (>=0), content longer
than it will be truncated; otherwise, no truncation at all. Do not
confuse this setting with the http.content.limit setting.
</description>
</property> <property>
<name>http.content.limit</name>
<value>-1</value>
<description>The length limit for downloaded content using the http
protocol, in bytes. If this value is nonnegative (>=0), content longer
than it will be truncated; otherwise, no truncation at all. Do not
confuse this setting with the file.content.limit setting.
</description>
</property> <property>
<name>ftp.content.limit</name>
<value>-1</value>
<description>The length limit for downloaded content, in bytes.
If this value is nonnegative (>=0), content longer than it will be truncated;
otherwise, no truncation at all.
Caution: classical ftp RFCs never defines partial transfer and, in fact,
some ftp servers out there do not handle client side forced close-down very
well. Our implementation tries its best to handle such situations smoothly.
</description>
</property>
【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件的更多相关文章
- 【Nutch2.2.1基础教程之2.2】集成Nutch/Hbase/Solr构建搜索引擎之二:内容分析
请先参见"集成Nutch/Hbase/Solr构建搜索引擎之一:安装及运行",搭建测试环境 http://blog.csdn.net/jediael_lu/article/deta ...
- 【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件 分类: H3_NUTCH 2014-08-18 16:33 1376人阅读 评论(0) 收藏
nutch-site.xml 在nutch2.2.1中,有两份配置文件:nutch-default.xml与nutch-site.xml. 其中前者是nutch自带的默认属性,一般情况下不要修改. 如 ...
- 【Nutch2.2.1基础教程之6】Nutch2.2.1抓取流程
一.抓取流程概述 1.nutch抓取流程 当使用crawl命令进行抓取任务时,其基本流程步骤如下: (1)InjectorJob 开始第一个迭代 (2)GeneratorJob (3)FetcherJ ...
- 【Nutch2.2.1基础教程之1】nutch相关异常
1.在任务一开始运行,注入Url时即出现以下错误. InjectorJob: Injecting urlDir: urls InjectorJob: Using class org.apache.go ...
- 【Nutch2.2.1基础教程之2.1】集成Nutch/Hbase/Solr构建搜索引擎之一:安装及运行【单机环境】
1.下载相关软件,并解压 版本号如下: (1)apache-nutch-2.2.1 (2) hbase-0.90.4 (3)solr-4.9.0 并解压至/usr/search 2.Nutch的配置 ...
- 【Nutch2.2.1基础教程之6】Nutch2.2.1抓取流程 分类: H3_NUTCH 2014-08-15 21:39 2530人阅读 评论(1) 收藏
一.抓取流程概述 1.nutch抓取流程 当使用crawl命令进行抓取任务时,其基本流程步骤如下: (1)InjectorJob 开始第一个迭代 (2)GeneratorJob (3)FetcherJ ...
- 【Nutch2.2.1基础教程之1】nutch相关异常 分类: H3_NUTCH 2014-08-08 21:46 1549人阅读 评论(2) 收藏
1.在任务一开始运行,注入Url时即出现以下错误. InjectorJob: Injecting urlDir: urls InjectorJob: Using class org.apache.go ...
- OpenVAS漏洞扫描基础教程之OpenVAS概述及安装及配置OpenVAS服务
OpenVAS漏洞扫描基础教程之OpenVAS概述及安装及配置OpenVAS服务 1. OpenVAS基础知识 OpenVAS(Open Vulnerability Assessment Sys ...
- Python基础教程之List对象 转
Python基础教程之List对象 时间:2014-01-19 来源:服务器之家 投稿:root 1.PyListObject对象typedef struct { PyObjec ...
随机推荐
- 几个个实用的PHP代码片段【自己备份】
检查服务器是否是 HTTPS 这个PHP代码片段能够读取关于你服务器 SSL 启用(HTTPS)信息. if ($_SERVER['HTTPS'] != "on") { echo ...
- iOS 第三方 需要 引用的库
================================================== AFNetWorking 是基于 nsurlconnection 所以不需要引入库 === ...
- ROC曲线和PR曲线
转自:http://www.zhizhihu.com/html/y2012/4076.html分类.检索中的评价指标很多,Precision.Recall.Accuracy.F1.ROC.PR Cur ...
- gtest以及测试小结
所有的测试,都是让未知的东西和已知的东西进行比较,如果测试结果和预期的一样,那么就认为被测对象是OK的否则视为有问题. python的单元测试是写一堆继承了unittest.TestCase类,每个类 ...
- CentOS7配置Nodejs环境安装记录
今天购买了阿里云服务器,系统选的是CentOS7,下面记录下在它上面安装Nodejs环境的过程,本次操作是直接连接的阿里云服务器的管理终端. 1.由于是纯净的环境,先通过以下命令安装nodejs编译及 ...
- android-support-v7-appcompat的配置使用
直接将android-support-v7-appcompat.jar包拷贝到项目的libs/下面是不能使用的,具体做法官方文档给出了详细说明: (开发环境是ADT) Using Eclipse Cr ...
- PHP页面静态化(转)
在很多地方都看到有PHP整站静态化的东东,怪唬人的..其实,你会静态化一个页面,那么别说整站了,想静态化多少都可以.所以关键是,首先要知道怎么静态化一个页面,了解静态化的原理是关键.. 这里就说下我个 ...
- paip.hadoop的应用研究总结
paip.hadoop的应用研究总结 作者Attilax , EMAIL:1466519819@qq.com 来源:attilax的专栏 地址:http://blog.csdn.net/attil ...
- tab切换jquery代码
http://immmmm.com/jquery-tab-switch-code-improved.html html <div id="sidebar-tab"> ...
- iphone UIScrollView缩放
allImageScrollView = [[UIScrollView alloc] initWithFrame:CGRectMake(0, 0, 768, 1024)]; allImageScrol ...