Another attempt about LSI
Last week I was here Natural Language Processing in NZ.
Someone asked a question, is there any existed library or solution which can extract certain information out of a natural language dataset of a specific topic, for example, to find objective facts and subjective sentimental information out of a bunch of customer complaints.
I was doing something similar in this realm, and I think LSI or Word2Vec is the best solution of this problem now. But my poor english speaking could not match either my confidence or LSI's power, my explanation of LSI was so terrible that someone gave me a few hints very politely and cryptically. And I could tell some of you were far from convinced.
I felt sorry for LSI, it was not its fault that I didn't present it in a proper way. Thanks to Alyona and everyone in the meetup, I appreciate the opportunity to meet nice people and interesting ideas there. So, here is my 2 cent.
We have been trying to build different models of natural language. The problem of querying natural language dataset is the problem of how to build a model which could interpret a query into results. The question raised at last meetup was how to build such a model which can extract information from natural language dataset based on specific syntactic structure and some specific semantics.
We have plenty natural language parsers now, so I guess syntactic structure is not the problem here.
As for semantics, there are some rule-based approach, such as, WordNet , FrameNet, however, I found it was difficult to map a word to similar words in semantics because a word usaually has multiple meanings, and there is no way to find the right semantics of a word in a specific context with these approach. And you could end up with different results with different meaning of a word. Furthermore, there is not any threshold which can determine how far you can go in a WordNet graph, at least not in a mathematically decidable way.
LSI is a tool of finding similarity among documents, it stems from SVD. After mapping all documents into a space, we could find similarities between documents in different dimension. And of course we can get a distance between any two words in a specific dimension.
Word2Vec is based on the same idea but the algorithm is different.
With these tools, by training a model with a dataset of some topics, we can determine the distance of any two words in a specific dataset(or topic) with this model. Then we can map a query into a set of sentences with this similarity or distance in respect to the specific topic.
LSI or similar algorithm is actually another piece of the puzzle.
'''Note'''
We still have to deal with the problem of how to model multi meanings of a word
Another attempt about LSI的更多相关文章
- yarn关于app max attempt深度解析,针对长服务appmaster平滑重启
在YARN上开发长服务,需要注意fault-tolerance,本篇文章对appmaster的平滑重启的一个参数做了解析,如何设置可以有助于达到appmaster平滑重启. 在yarn-site.xm ...
- ORA-14450: attempt to access a transactional temp table already in use
在ORACLE数据中修改会话级临时表时,有可能会遇到ORA-14550错误,那么为什么会话级全局临时表会报ORA-14450错误呢,如下所示,我们先从一个小小案例入手: 案例1: SQL> CR ...
- Attempt to fetch logical page (...) in database 2 failed. It belongs to allocation unit xxxx not to xxx
今天一个同事说在一个生产库执行某个存储过程,遇到了错误: Fatal error 605 occurred at jul 29 2014 我试着执行该存储过程,结果出现下面错误,每次执行该存储过程,得 ...
- 错误信息:attempt to create saveOrUpdate event with null entity
错误信息:attempt to create saveOrUpdate event with null entity; 这个错误网上答案比较多,我也不多说了. 我遇到的问题是在前台传过来的参数是nul ...
- lua协程一则报错解决“attempt to yield across metamethod/C-call boundary”
问题 attempt to yield across metamethod/C-call boundary 需求跟如下帖子中描述一致: http://bbs.chinaunix.net/forum.p ...
- Warning: Attempt to present on whose view is not in the window hierarchy!
当我想从一个VC跳转到另一个VC的时候,一般会用 - (void)presentViewController:(UIViewController *)viewControllerToPresent a ...
- tomcat提示警告: An attempt was made to authenticate the locked user"tomcat"
启动tomcat7之后,运行正常,但是运行一段时间就会提示以下警告: 十二月 04, 2013 5:10:15 下午 org.apache.catalina.realm.LockOutRealm au ...
- Attempt to present <vc> on <vc> which is already presenting <vc>/(null)
在给 tableViewCell 添加长按手势弹出一个 popViewController 的时候,遇到的这个变态问题: Warning: Attempt to present <UINavig ...
- An attempt was made to load a program with an incorrect format
用.net调用一个C++ 32位的DLL, 编译的时候选择x86, 在部署到一个64位的机器上的时候报错:"An attempt was made to load a program w ...
随机推荐
- 关于WPF中承载 ArcGIS控件。
原文 http://www.cnblogs.com/zoe-j/archive/2011/05/18/2050208.html 之前就做过WPF的应用,之前承载的MapGIS的二次开发控件,今天写一下 ...
- bzoj1641 [Usaco2007 Nov]Cow Hurdles 奶牛跨栏
Description Farmer John 想让她的奶牛准备郡级跳跃比赛,贝茜和她的伙伴们正在练习跨栏.她们很累,所以她们想消耗最少的能量来跨栏. 显然,对于一头奶牛跳过几个矮栏是很容易的,但是高 ...
- VBA:Google翻译(含tk算法)
完整的tk算法: //源自http://translate.google.cn/ TKK=eval('((function(){var a\x3d618632403;var b\x3d14854840 ...
- UGUI Silder
来我们看看这个像温度计的控件, 比如音量面板声音大小的控制.它是一个组合型控件由多个Image 和一个Slider组合而成 它的核心是Slider组件实现的. 简单介绍下Slider组件的属性: Fi ...
- 手动调整导航控制器中的viewcontroller
NSMutableArray *array = [[NSMutableArray alloc]initWithArray: [self.navigationController viewControl ...
- [置顶] Hibernate从入门到精通(五)一对一单向关联映射
上次的博文中Hibernate从入门到精通(四)基本映射我们已经讲解了一下基本映射和相关概念,接下来我们会讲稍微复杂点的映射——关系映射. 关系映射分类 关系映射即在基本映射的基础上处理多个相关对象和 ...
- 安装MySQLdb-python时无法找到-lprobes_mysql处理一则
安装MySQLdb时,我已经做好了如下两点:1)配置好mysql_config 有两种方式可以做: ① 做个软连接,把mysql_config链接到/usr/local/bin下 ② ...
- ff与ie 的关于js兼容性
FF的FIREBUG,不仅能测试JS还能检查CSS错误,是一般常用的.但它主要检查FF方面的错误,对IE就无能为力了.要测试IE,就用ieTester,它可以测试IE几乎所有版本(1.0恐怕也用不到测 ...
- 000-C#基础
C#中数据类型的继承关系如下 System.Object |-------------System.ValueType | |-------System.Boolean | |-------Syste ...
- sql 函数 总结
聚合函数 Avg(numeric_expr)返回integer.decimal.money.float---返回组中各值的平均值 Count(*|column_name|distinct column ...