Another attempt about LSI
Last week I was here Natural Language Processing in NZ.
Someone asked a question, is there any existed library or solution which can extract certain information out of a natural language dataset of a specific topic, for example, to find objective facts and subjective sentimental information out of a bunch of customer complaints.
I was doing something similar in this realm, and I think LSI or Word2Vec is the best solution of this problem now. But my poor english speaking could not match either my confidence or LSI's power, my explanation of LSI was so terrible that someone gave me a few hints very politely and cryptically. And I could tell some of you were far from convinced.
I felt sorry for LSI, it was not its fault that I didn't present it in a proper way. Thanks to Alyona and everyone in the meetup, I appreciate the opportunity to meet nice people and interesting ideas there. So, here is my 2 cent.
We have been trying to build different models of natural language. The problem of querying natural language dataset is the problem of how to build a model which could interpret a query into results. The question raised at last meetup was how to build such a model which can extract information from natural language dataset based on specific syntactic structure and some specific semantics.
We have plenty natural language parsers now, so I guess syntactic structure is not the problem here.
As for semantics, there are some rule-based approach, such as, WordNet , FrameNet, however, I found it was difficult to map a word to similar words in semantics because a word usaually has multiple meanings, and there is no way to find the right semantics of a word in a specific context with these approach. And you could end up with different results with different meaning of a word. Furthermore, there is not any threshold which can determine how far you can go in a WordNet graph, at least not in a mathematically decidable way.
LSI is a tool of finding similarity among documents, it stems from SVD. After mapping all documents into a space, we could find similarities between documents in different dimension. And of course we can get a distance between any two words in a specific dimension.
Word2Vec is based on the same idea but the algorithm is different.
With these tools, by training a model with a dataset of some topics, we can determine the distance of any two words in a specific dataset(or topic) with this model. Then we can map a query into a set of sentences with this similarity or distance in respect to the specific topic.
LSI or similar algorithm is actually another piece of the puzzle.
'''Note'''
We still have to deal with the problem of how to model multi meanings of a word
Another attempt about LSI的更多相关文章
- yarn关于app max attempt深度解析,针对长服务appmaster平滑重启
在YARN上开发长服务,需要注意fault-tolerance,本篇文章对appmaster的平滑重启的一个参数做了解析,如何设置可以有助于达到appmaster平滑重启. 在yarn-site.xm ...
- ORA-14450: attempt to access a transactional temp table already in use
在ORACLE数据中修改会话级临时表时,有可能会遇到ORA-14550错误,那么为什么会话级全局临时表会报ORA-14450错误呢,如下所示,我们先从一个小小案例入手: 案例1: SQL> CR ...
- Attempt to fetch logical page (...) in database 2 failed. It belongs to allocation unit xxxx not to xxx
今天一个同事说在一个生产库执行某个存储过程,遇到了错误: Fatal error 605 occurred at jul 29 2014 我试着执行该存储过程,结果出现下面错误,每次执行该存储过程,得 ...
- 错误信息:attempt to create saveOrUpdate event with null entity
错误信息:attempt to create saveOrUpdate event with null entity; 这个错误网上答案比较多,我也不多说了. 我遇到的问题是在前台传过来的参数是nul ...
- lua协程一则报错解决“attempt to yield across metamethod/C-call boundary”
问题 attempt to yield across metamethod/C-call boundary 需求跟如下帖子中描述一致: http://bbs.chinaunix.net/forum.p ...
- Warning: Attempt to present on whose view is not in the window hierarchy!
当我想从一个VC跳转到另一个VC的时候,一般会用 - (void)presentViewController:(UIViewController *)viewControllerToPresent a ...
- tomcat提示警告: An attempt was made to authenticate the locked user"tomcat"
启动tomcat7之后,运行正常,但是运行一段时间就会提示以下警告: 十二月 04, 2013 5:10:15 下午 org.apache.catalina.realm.LockOutRealm au ...
- Attempt to present <vc> on <vc> which is already presenting <vc>/(null)
在给 tableViewCell 添加长按手势弹出一个 popViewController 的时候,遇到的这个变态问题: Warning: Attempt to present <UINavig ...
- An attempt was made to load a program with an incorrect format
用.net调用一个C++ 32位的DLL, 编译的时候选择x86, 在部署到一个64位的机器上的时候报错:"An attempt was made to load a program w ...
随机推荐
- mac安装GNU命令行工具
mac安装GNU命令行工具 2.添加的repo tap home/dupes brew install coreutils binutils diffutils ed -- ...
- shell脚本语法基础汇总
shell脚本语法基础汇总 将命令的输出读入一个变量中,可以将它放入双引号中,即可保留空格和换行符(\n) out=$(cat text.txt) 输出1 2 3 out="$(cat te ...
- lowerCaseTableNames
数据库表,数据库名大小写铭感问题 mysql lower-case-table-names参数 线上有业务用到开源的产品,其中SQL语句是大小写混合的,而建表语句都是小写的,mysql默认设置导致这些 ...
- 微软GitHub组织
微软aspnet团队的GitHub 微软dotnet团队的GitHub 微软的GitHub 微软云团队的GitHub 微软在GitHub的开源底部有其它组织
- 用Robocod游戏来学习JAVA
Robocode(用游戏来学习Java技术还是用Java来玩游戏?)用你的JAVA编程技术来玩游戏吧!不会JAVA?那就用游戏来学习JAVA吧!什么是Robocode? 其实我对机器人一直很感兴趣.我 ...
- C#.Net前台线程与后台线程的区别
本文来自:http://www.cnblogs.com/zfanlong1314/archive/2012/02/26/2390455.html .Net的公用语言运行时(Common Languag ...
- 从3dmax中导入模型到UDK Editor(供个人备忘)
笔记从3dmax中导入模型到UDK Editor 1) 在3dmax中导出 2) 选择FBX格式,保存 3) 在UDK中打开content browser,自己选个pac ...
- mock server相关解决方案
前后端分离之后 前后端分离后, 大家从此进入了所谓的并行开发时代. 一旦完成前后端的(边界)分工, 大家就可以各司其职了. 前端在与后端交互时, 要想有效地提高工作效率, 后端的接口文档就是重中之重了 ...
- 这个更新需要花去 50.6 M 磁盘上总计 /boot 的空间。请在 7737k 磁盘上留出 /boot 空间。清空您的回收站和临时文件,用“sudo apt-get clean
系统升级会下载多余的内核,删除即可. 1.命令:dpkg --get-selections|grep linux //带image的为系统内核 2.命令: uname -a ...
- .net 时间操作[摘抄]
.Net时间运算 - DateTime类,TimeSpan类http://www.cnblogs.com/kissdodog/archive/2013/03/02/2939927.html 时间戳转换 ...