从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel
https://github.com/goinaction/code
[root@hadoop3 sample]# go run main.go
2018/07/30 17:45:39 Register default matcher
2018/07/30 17:45:39 Register rss matcher
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1001]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=46]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=7]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=10]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=39]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=43]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/business/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1008]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1006]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1007]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_world.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1057]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_us.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1021]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_allpolitics.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1012]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_crime.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1003]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=2]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_tech.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_health.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/opinion?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/politics?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/national?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/world?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/topstories]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/usnews]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491043/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=3]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491571/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=5]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/28180066/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=13]
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_allpolitics.rss: read tcp 192.168.3.103:54478->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54484->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_health.rss: read tcp 192.168.3.103:54482->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_crime.rss: read tcp 192.168.3.103:54476->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_us.rss: read tcp 192.168.3.103:54488->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_world.rss: read tcp 192.168.3.103:54486->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Description:
If Ronald Reagan was the “Teflon president,” Donald Trump must be the Incredible Hulk president. The more the left and the media attack him, the stronger he gets.<img src="http://feeds.feedburner.com/~r/foxnews/opinion/~4/bVrx9JLgLWI" height="1" width="1" alt=""/> 2018/07/30 17:45:39 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:39 Description:
Freeman Hrabowski, president of the University of Maryland, Baltimore County, talks about the importance of mentoring in academia, especially in science, technology, engineering and math fields. 2018/07/30 17:45:39 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Retail and fast-food workers protesting for higher pay are creating a new kind of U.S. labor movement. <em>Chicago Tribune</em> columnist Clarence Page argues that the president could "set a good example" by requiring fast-food vendors who have contracts with the federal government to pay minimum wage. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
The moment when the president articulated his attitude with pithy perfection came when he told the Veterans of Foreign Wars that "what you're seeing and what you're reading is not what's happening." 2018/07/30 17:45:40 Description:
The U.S. president publicly humiliated our most important allies and then said everything was great. We need positive, unifying leadership, writes a former U.S. Army Europe commander. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:40 Description:
The president goes back and forth about what he accepts and denies about Russian election interference. Even so, his recent warning about an attack that helps Democrats is a cogent one. 2018/07/30 17:45:40 Description:
The president demanded cooperation from the Democrats over immigration, threatening to close the government in order to achieve his goals. 2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_tech.rss: read tcp 192.168.3.103:54480->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54474->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/business/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
[root@hadoop3 sample]# tree
.
├── data
│ └── data.json
├── main.go
├── matchers
│ └── rss.go
└── search
├── default.go
├── feed.go
├── match.go
└── search.go 3 directories, 7 files
[root@hadoop3 sample]# tree /home/ctGO/goInAction/
/home/ctGO/goInAction/
└── code
├── chapter1
│ └── channels
│ └── hellochannels.go
├── chapter10
│ ├── listing01
│ │ └── listing01.go
│ ├── listing02
│ │ └── listing02.go
│ ├── listing03
│ │ └── listing03.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing05
│ │ └── listing05.go
│ └── listing06
│ ├── listing06
│ ├── listing06.go
│ └── pubsub
│ └── pubsub.go
├── chapter2
│ └── sample
│ ├── data
│ │ └── data.json
│ ├── main.go
│ ├── matchers
│ │ └── rss.go
│ └── search
│ ├── default.go
│ ├── feed.go
│ ├── match.go
│ └── search.go
├── chapter3
│ ├── dbdriver
│ │ ├── main.go
│ │ └── postgres
│ │ └── postgres.go
│ ├── wordcount
│ │ ├── gowords.txt
│ │ └── wordcount.go
│ └── words
│ └── count.go
├── chapter5
│ ├── listing11
│ │ └── listing11.go
│ ├── listing34
│ │ └── listing34.go
│ ├── listing35
│ │ └── listing35.go
│ ├── listing36
│ │ └── listing36.go
│ ├── listing46
│ │ └── listing46.go
│ ├── listing48
│ │ └── listing48.go
│ ├── listing50
│ │ └── listing50.go
│ ├── listing56
│ │ └── listing56.go
│ ├── listing60
│ │ └── listing60.go
│ ├── listing64
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing64.go
│ ├── listing68
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing68.go
│ ├── listing71
│ │ ├── entities
│ │ │ └── entities.go
│ │ └── listing71.go
│ └── listing74
│ ├── entities
│ │ └── entities.go
│ └── listing74.go
├── chapter6
│ ├── listing01
│ │ └── listing01.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing07
│ │ └── listing07.go
│ ├── listing09
│ │ └── listing09.go
│ ├── listing13
│ │ └── listing13.go
│ ├── listing15
│ │ └── listing15.go
│ ├── listing16
│ │ └── listing16.go
│ ├── listing20
│ │ └── listing20.go
│ ├── listing22
│ │ └── listing22.go
│ └── listing24
│ └── listing24.go
├── chapter7
│ └── patterns
│ ├── pool
│ │ ├── main
│ │ │ └── main.go
│ │ └── pool.go
│ ├── runner
│ │ ├── main
│ │ │ └── main.go
│ │ └── runner.go
│ ├── search
│ │ ├── main
│ │ │ └── main.go
│ │ ├── searchers.go
│ │ └── search.go
│ ├── semaphore
│ │ └── semaphore.go
│ └── work
│ ├── main
│ │ └── main.go
│ └── work.go
├── chapter8
│ ├── listing03
│ │ └── listing03.go
│ ├── listing11
│ │ └── listing11.go
│ ├── listing24
│ │ └── listing24.go
│ ├── listing27
│ │ └── listing27.go
│ ├── listing29
│ │ └── listing29.go
│ ├── listing31
│ │ └── listing31.go
│ ├── listing37
│ │ └── listing37.go
│ └── listing46
│ └── listing46.go
├── chapter9
│ ├── listing01
│ │ └── listing01_test.go
│ ├── listing08
│ │ └── listing08_test.go
│ ├── listing12
│ │ └── listing12_test.go
│ ├── listing17
│ │ ├── handlers
│ │ │ ├── handlers_example_test.go
│ │ │ ├── handlers.go
│ │ │ └── handlers_test.go
│ │ └── listing17.go
│ └── listing28
│ └── listing28_test.go
└── README.md 77 directories, 75 files
[root@hadoop3 sample]#
https://github.com/goinaction/code/blob/master/chapter2/sample/search/search.go
// Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}()
// Run performs the search logic.
func Run(searchTerm string) {
// Retrieve the list of feeds to search through.
feeds, err := RetrieveFeeds()
if err != nil {
log.Fatal(err)
} // Create an unbuffered channel to receive match results to display.
results := make(chan *Result) // Setup a wait group so we can process all the feeds.
var waitGroup sync.WaitGroup // Set the number of goroutines we need to wait for while
// they process the individual feeds.
waitGroup.Add(len(feeds)) // Launch a goroutine for each feed to find the results.
for _, feed := range feeds {
// Retrieve a matcher for the search.
matcher, exists := matchers[feed.Type]
if !exists {
matcher = matchers["default"]
} // Launch the goroutine to perform the search.
go func(matcher Matcher, feed *Feed) {
Match(matcher, feed, searchTerm, results)
waitGroup.Done()
}(matcher, feed)
} // Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}() // Start displaying results as they are available and
// return after the final result is displayed.
Display(results)
}
go通道消息共享
从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel的更多相关文章
- FLUME安装&环境（二）:拉取MySQL数据库数据到Kafka
Flume安装成功,环境变量配置成功后,开始进行agent配置文件设置. 1.agent配置文件(mysql+flume+Kafka) #利用Flume将MySQL表数据准实时抽取到Kafka a1. ...
- 使用Jenkins的Git Parameter插件来从远程仓库拉取指定目录的内容
更换插件安装源 系统管理---插件管理---高级---升级站点,把默认的插件下载地址换成下面这个: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updat ...
- 解决 MySQL 比如我要拉取一个消息表中用户id为1的前10条最新数据
我们都知道,各种主流的社交应用或者阅读应用,基本都有列表类视图,并且都有滑到底部加载更多这一功能, 对应后端就是分页拉取数据.好处不言而喻,一般来说,这些数据项都是按时间倒序排列的,用户只关心最新的动 ...
- canal从mysql拉取数据，并以protobuf的格式往kafka中写数据
大致思路: canal去mysql拉取数据,放在canal所在的节点上,并且自身对外提供一个tcp服务,我们只要写一个连接该服务的客户端,去拉取数据并且指定往kafka写数据的格式就能达到以proto ...
- 版本控制git之三-多人协作 变基 推送 拉取 删除远程分支
版本控制git之三-多人协作 wangfeng7399已关注0人评论350人阅读2019-02-20 21:33:08 如果你想获得一份已经存在了的 Git 仓库的拷贝,比如说,你想为某个开源 ...
- 源码分析Kafka 消息拉取流程
目录 1.KafkaConsumer poll 详解 2.Fetcher 类详解 本节重点讨论 Kafka 的消息拉起流程. @(本节目录) 1.KafkaConsumer poll 详解 消息拉起主 ...
- Mysql日期类型大小比较---拉取给定时间段的记录
我们知道,mysql里边,日期类型有很多表现形式,date, datetime,timestamp等类型.考虑这样一种场景: 按时间段拉取给定时间段的内容,这时,我们就得使用日期类型的比较了. 表结构 ...
- idea配置git，查看git代码&拉取git项目至本地
1.点击file,右键选择setting 选择本地git安装路径 Ps:从git上导入一个全新的maven项目 点击clone按钮后,会弹出如下截图弹窗,点击 NO 项目已经拉取到本地,然后点击ope ...
- Spark Streaming中向flume拉取数据
在这里看到的解决方法 https://issues.apache.org/jira/browse/SPARK-1729 请是个人理解,有问题请大家留言. 其实本身flume是不支持像KAFKA一样的发 ...
随机推荐
- redis主从复制以及SSDB主主复制环境部署记录（四）
redis配置主从和ssdb主主复制 参考:散尽浮华Linux下Redis主从复制以及SSDB主主复制环境部署记录 只做自己查阅. 环境要求:三台虚拟机一台做主,两台做从. 通过SSDB在redis做 ...
- BZOJ 2176 Strange string ——最小表示法
本来想用来练习后缀自动机的,但是100w有点虚(事实证明确实T掉了). 只好上最小表示法. #include <cstdio> #include <cstring> #incl ...
- 2013 Asia acm Hangzhou Regional Contest 杭州现场赛
B Stealing Harry Potter's Precious 题目大意:给定一个n*m的地图,某些点可以走,某些点可以走某些点不可以走,给定一个起点,又给出了k个点k<=4,要求从起点 ...
- P3799 妖梦拼木棒 (组合数学)
题目背景 上道题中,妖梦斩了一地的木棒,现在她想要将木棒拼起来. 题目描述 有n根木棒,现在从中选4根,想要组成一个正三角形,问有几种选法? 输入输出格式 输入格式: 第一行一个整数n 第二行n个整数 ...
- CSS选择器与XPath语言
一 在爬取页面信息的过程中,需要到想要的信息进行定位,主要有两种方法.CSS选择器和XPath语言.查找某一个标签,两种方法都可以做到. 二 CSS选择器 http://www.w3school.co ...
- BZOJ4725: [POI2017]Reprezentacje ró?nicowe
$n \leq 1e5$,$x \leq 1e9$. 1e9呵呵,暴力处理$a_n$的前几项直到1e9.然后处理出差的数列,每次在这里面找,找得到就回答,找不到,那有贡献的只有$a_i-a_{i-1} ...
- 湘潭oj1203/邀请赛A题 数论+java大数
求 n%1+n%2+n%3+n%4+.........n%n=,n<=10^12次. 开始时盲目地找规律,结果一无所获.后来经学长点拨,天资愚钝,搞了半天才明白. 先上图: 对于该题,在求区间( ...
- npm 安装vue-cli
TIP:win10下安装,使用管理员身份进行,否则会有权限限制. 1,安装完成node,node有自带的npm,可以直接在cmd中,找到nodeJs安装的路径下,进行命令行全局安装vue-cli.(n ...
- CentOS 5.4 final下Systemtap的安装
CentOS 5.4 final下Systemtap的安装 时间:2015-02-11来源:linux网站 作者:zklth 一.Systemtap运行环境需求 (1)linux kernel ...
- 【sourcetree】sourcetree连接远程仓库需要登陆但是一直登陆不上的问题 解决方法
授权类型选用 基础 .只需要登陆你在bitbucket的用户名和密码 如下 .即可成功连接远程仓库