从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel
https://github.com/goinaction/code
[root@hadoop3 sample]# go run main.go
2018/07/30 17:45:39 Register default matcher
2018/07/30 17:45:39 Register rss matcher
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1001]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=46]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=7]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=10]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=39]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=43]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/business/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1008]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1006]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[bbci] For URI[http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1007]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_world.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1057]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_us.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1021]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_allpolitics.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1012]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_crime.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=1003]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=2]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_tech.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_health.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[cnn] For URI[http://rss.cnn.com/rss/cnn_topstories.rss]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/opinion?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/politics?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/national?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[foxnews] For URI[http://feeds.foxnews.com/foxnews/world?format=xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/topstories]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://feeds.nbcnews.com/feeds/usnews]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491043/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=3]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/21491571/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=5]
2018/07/30 17:45:39 Search Feed Type[rss] Site[nbcnews] For URI[http://rss.msnbc.msn.com/id/28180066/device/rss/rss.xml]
2018/07/30 17:45:39 Search Feed Type[rss] Site[npr] For URI[http://www.npr.org/rss/rss.php?id=13]
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_allpolitics.rss: read tcp 192.168.3.103:54478->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54484->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_health.rss: read tcp 192.168.3.103:54482->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_crime.rss: read tcp 192.168.3.103:54476->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_us.rss: read tcp 192.168.3.103:54488->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Get http://rss.cnn.com/rss/cnn_world.rss: read tcp 192.168.3.103:54486->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:39 Description:
If Ronald Reagan was the “Teflon president,” Donald Trump must be the Incredible Hulk president. The more the left and the media attack him, the stronger he gets.<img src="http://feeds.feedburner.com/~r/foxnews/opinion/~4/bVrx9JLgLWI" height="1" width="1" alt=""/> 2018/07/30 17:45:39 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:39 Description:
Freeman Hrabowski, president of the University of Maryland, Baltimore County, talks about the importance of mentoring in academia, especially in science, technology, engineering and math fields. 2018/07/30 17:45:39 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Retail and fast-food workers protesting for higher pay are creating a new kind of U.S. labor movement. <em>Chicago Tribune</em> columnist Clarence Page argues that the president could "set a good example" by requiring fast-food vendors who have contracts with the federal government to pay minimum wage. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
The moment when the president articulated his attitude with pithy perfection came when he told the Veterans of Foreign Wars that "what you're seeing and what you're reading is not what's happening." 2018/07/30 17:45:40 Description:
The U.S. president publicly humiliated our most important allies and then said everything was great. We need positive, unifying leadership, writes a former U.S. Army Europe commander. 2018/07/30 17:45:40 Description:
A dispute between the president and<em> The New York Times</em> publisher revives questions about the impact of Trump's anti-press rhetoric. He decried a supposed media conspiracy against his administration. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
Last year a song about freedom became a rallying cry as Zimbabwe unseated longtime president Robert Mugabe. Now the anthem has become tainted by the lack of tangible change for people there. 2018/07/30 17:45:40 Description:
We look at the latest strong economic numbers and the effect tariffs may have on the president's base. 2018/07/30 17:45:40 Description:
The president goes back and forth about what he accepts and denies about Russian election interference. Even so, his recent warning about an attack that helps Democrats is a cogent one. 2018/07/30 17:45:40 Description:
The president demanded cooperation from the Democrats over immigration, threatening to close the government in order to achieve his goals. 2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_tech.rss: read tcp 192.168.3.103:54480->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:40 Get http://rss.cnn.com/rss/cnn_topstories.rss: read tcp 192.168.3.103:54474->216.58.200.51:80: read: connection reset by peer
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:45:41 expected element type <rss> but have <html>
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/business/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
2018/07/30 17:46:09 Get http://feeds.bbci.co.uk/news/world/us_and_canada/rss.xml: dial tcp 66.220.155.12:80: i/o timeout
[root@hadoop3 sample]# tree
.
├── data
│ └── data.json
├── main.go
├── matchers
│ └── rss.go
└── search
├── default.go
├── feed.go
├── match.go
└── search.go 3 directories, 7 files
[root@hadoop3 sample]# tree /home/ctGO/goInAction/
/home/ctGO/goInAction/
└── code
├── chapter1
│ └── channels
│ └── hellochannels.go
├── chapter10
│ ├── listing01
│ │ └── listing01.go
│ ├── listing02
│ │ └── listing02.go
│ ├── listing03
│ │ └── listing03.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing05
│ │ └── listing05.go
│ └── listing06
│ ├── listing06
│ ├── listing06.go
│ └── pubsub
│ └── pubsub.go
├── chapter2
│ └── sample
│ ├── data
│ │ └── data.json
│ ├── main.go
│ ├── matchers
│ │ └── rss.go
│ └── search
│ ├── default.go
│ ├── feed.go
│ ├── match.go
│ └── search.go
├── chapter3
│ ├── dbdriver
│ │ ├── main.go
│ │ └── postgres
│ │ └── postgres.go
│ ├── wordcount
│ │ ├── gowords.txt
│ │ └── wordcount.go
│ └── words
│ └── count.go
├── chapter5
│ ├── listing11
│ │ └── listing11.go
│ ├── listing34
│ │ └── listing34.go
│ ├── listing35
│ │ └── listing35.go
│ ├── listing36
│ │ └── listing36.go
│ ├── listing46
│ │ └── listing46.go
│ ├── listing48
│ │ └── listing48.go
│ ├── listing50
│ │ └── listing50.go
│ ├── listing56
│ │ └── listing56.go
│ ├── listing60
│ │ └── listing60.go
│ ├── listing64
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing64.go
│ ├── listing68
│ │ ├── counters
│ │ │ └── counters.go
│ │ └── listing68.go
│ ├── listing71
│ │ ├── entities
│ │ │ └── entities.go
│ │ └── listing71.go
│ └── listing74
│ ├── entities
│ │ └── entities.go
│ └── listing74.go
├── chapter6
│ ├── listing01
│ │ └── listing01.go
│ ├── listing04
│ │ └── listing04.go
│ ├── listing07
│ │ └── listing07.go
│ ├── listing09
│ │ └── listing09.go
│ ├── listing13
│ │ └── listing13.go
│ ├── listing15
│ │ └── listing15.go
│ ├── listing16
│ │ └── listing16.go
│ ├── listing20
│ │ └── listing20.go
│ ├── listing22
│ │ └── listing22.go
│ └── listing24
│ └── listing24.go
├── chapter7
│ └── patterns
│ ├── pool
│ │ ├── main
│ │ │ └── main.go
│ │ └── pool.go
│ ├── runner
│ │ ├── main
│ │ │ └── main.go
│ │ └── runner.go
│ ├── search
│ │ ├── main
│ │ │ └── main.go
│ │ ├── searchers.go
│ │ └── search.go
│ ├── semaphore
│ │ └── semaphore.go
│ └── work
│ ├── main
│ │ └── main.go
│ └── work.go
├── chapter8
│ ├── listing03
│ │ └── listing03.go
│ ├── listing11
│ │ └── listing11.go
│ ├── listing24
│ │ └── listing24.go
│ ├── listing27
│ │ └── listing27.go
│ ├── listing29
│ │ └── listing29.go
│ ├── listing31
│ │ └── listing31.go
│ ├── listing37
│ │ └── listing37.go
│ └── listing46
│ └── listing46.go
├── chapter9
│ ├── listing01
│ │ └── listing01_test.go
│ ├── listing08
│ │ └── listing08_test.go
│ ├── listing12
│ │ └── listing12_test.go
│ ├── listing17
│ │ ├── handlers
│ │ │ ├── handlers_example_test.go
│ │ │ ├── handlers.go
│ │ │ └── handlers_test.go
│ │ └── listing17.go
│ └── listing28
│ └── listing28_test.go
└── README.md 77 directories, 75 files
[root@hadoop3 sample]#
https://github.com/goinaction/code/blob/master/chapter2/sample/search/search.go
// Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}()
// Run performs the search logic.
func Run(searchTerm string) {
// Retrieve the list of feeds to search through.
feeds, err := RetrieveFeeds()
if err != nil {
log.Fatal(err)
} // Create an unbuffered channel to receive match results to display.
results := make(chan *Result) // Setup a wait group so we can process all the feeds.
var waitGroup sync.WaitGroup // Set the number of goroutines we need to wait for while
// they process the individual feeds.
waitGroup.Add(len(feeds)) // Launch a goroutine for each feed to find the results.
for _, feed := range feeds {
// Retrieve a matcher for the search.
matcher, exists := matchers[feed.Type]
if !exists {
matcher = matchers["default"]
} // Launch the goroutine to perform the search.
go func(matcher Matcher, feed *Feed) {
Match(matcher, feed, searchTerm, results)
waitGroup.Done()
}(matcher, feed)
} // Launch a goroutine to monitor when all the work is done.
go func() {
// Wait for everything to be processed.
waitGroup.Wait() // Close the channel to signal to the Display
// function that we can exit the program.
close(results)
}() // Start displaying results as they are available and
// return after the final result is displayed.
Display(results)
}
go通道消息共享
从数据源拉取数据，将数据内容与一组搜索项做比对 go func() chanel的更多相关文章
- FLUME安装&环境（二）:拉取MySQL数据库数据到Kafka
Flume安装成功,环境变量配置成功后,开始进行agent配置文件设置. 1.agent配置文件(mysql+flume+Kafka) #利用Flume将MySQL表数据准实时抽取到Kafka a1. ...
- 使用Jenkins的Git Parameter插件来从远程仓库拉取指定目录的内容
更换插件安装源 系统管理---插件管理---高级---升级站点,把默认的插件下载地址换成下面这个: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updat ...
- 解决 MySQL 比如我要拉取一个消息表中用户id为1的前10条最新数据
我们都知道,各种主流的社交应用或者阅读应用,基本都有列表类视图,并且都有滑到底部加载更多这一功能, 对应后端就是分页拉取数据.好处不言而喻,一般来说,这些数据项都是按时间倒序排列的,用户只关心最新的动 ...
- canal从mysql拉取数据，并以protobuf的格式往kafka中写数据
大致思路: canal去mysql拉取数据,放在canal所在的节点上,并且自身对外提供一个tcp服务,我们只要写一个连接该服务的客户端,去拉取数据并且指定往kafka写数据的格式就能达到以proto ...
- 版本控制git之三-多人协作 变基 推送 拉取 删除远程分支
版本控制git之三-多人协作 wangfeng7399已关注0人评论350人阅读2019-02-20 21:33:08 如果你想获得一份已经存在了的 Git 仓库的拷贝,比如说,你想为某个开源 ...
- 源码分析Kafka 消息拉取流程
目录 1.KafkaConsumer poll 详解 2.Fetcher 类详解 本节重点讨论 Kafka 的消息拉起流程. @(本节目录) 1.KafkaConsumer poll 详解 消息拉起主 ...
- Mysql日期类型大小比较---拉取给定时间段的记录
我们知道,mysql里边,日期类型有很多表现形式,date, datetime,timestamp等类型.考虑这样一种场景: 按时间段拉取给定时间段的内容,这时,我们就得使用日期类型的比较了. 表结构 ...
- idea配置git，查看git代码&拉取git项目至本地
1.点击file,右键选择setting 选择本地git安装路径 Ps:从git上导入一个全新的maven项目 点击clone按钮后,会弹出如下截图弹窗,点击 NO 项目已经拉取到本地,然后点击ope ...
- Spark Streaming中向flume拉取数据
在这里看到的解决方法 https://issues.apache.org/jira/browse/SPARK-1729 请是个人理解,有问题请大家留言. 其实本身flume是不支持像KAFKA一样的发 ...
随机推荐
- 九度oj 题目1179：阶乘
题目描述: 输入n, 求y1=1!+3!+...m!(m是小于等于n的最大奇数)y2=2!+4!+...p!(p是小于等于n的最大偶数). 输入: 每组输入包括1个整数:n 输出: 可能有多组测试数据 ...
- LINQ学习笔记 Join 与 Group join
LINQ中的Join对应T-SQL中的内连接,并无左连接的方法,当然也没有右连接. 要达成Left join必须依靠GroupJoin来完成. GroupJoin顾名思义就是先集团在做加入,加入的不同 ...
- [luoguP3317] [SDOI2014]重建（矩阵树定理）
传送门 为了搞这个题又是学行列式,又是学基尔霍夫矩阵. 矩阵树定理 本题题解 无耻地直接发链接,反正我也是抄的题解.. #include <cstdio> #include <cma ...
- P1410 子序列 (动态规划)
题目描述 给定一个长度为N(N为偶数)的序列,问能否将其划分为两个长度为N/2的严格递增子序列. 输入输出格式 输入格式: 若干行,每行表示一组数据.对于每组数据,首先输入一个整数N,表示序列的长度. ...
- SpringBoot消失的Web.xml
Filter 过滤器作为web.xml中重要的一部分,有着相当高的出场率,SpringBoot会默认注册几个Filter ApplicationContextHeaderFilter Characte ...
- 济南学习 Day 5 T2 am
車(Rook) [题目描述] 众所周知,車是中国象棋最厉害的棋子之一,他能吃到同一行或者同一列的其他棋子.車显然不能和車在一打起来,于是rly有借来了许多许多車在棋盘上摆了起来...... 他想知道, ...
- 487. Max Consecutive Ones II
Given a binary array, find the maximum number of consecutive 1s in this array if you can flip at mos ...
- SGU103+POJ 1158 最短路/dp
题意:一个无向图,求起点到终点最少时间,限制:每个路口有灯,要灯颜色一样才能过去,灯之有俩种颜色,周期 变化,给定每个灯初态,时间. 思路:开始就想到直接DP,方程dp[k]=dp[i]+distan ...
- C# 编程中的堆栈（Stack）和队列（Queue）
一.什么是堆?(Heap) 堆是无序的,是一片不连续的内存域,由用户自己来控制和释放,如果用户自己不释放的话,当内存达到一定的特定值时,通过垃圾回收器(GC)来回收. 是程序运行期 ...
- android 获得电池状态
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools= ...