N-Gram
- 中文名
- 汉语语言模型
- 外文名
- N-Gram
- 定 义
- 计算出具有最大概率的句子
- 基 于
- 该模型基于这样一种假设
In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sequence of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech corpus. When the items are words, n-grams may also be called shingles.[1]
An n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" (or, less commonly, a "digram"); size 3 is a "trigram". Larger sizes are sometimes referred to by the value of n, e.g., "four-gram", "five-gram", and so on.
Contents
Applications
An n-gram model is a type of probabilistic language model for predicting the next item in such a sequence in the form of a (n − 1)–order Markov model.[2] n-gram models are now widely used in probability, communication theory, computational linguistics (for instance, statistical natural language processing), computational biology (for instance, biological sequence analysis), and data compression. Two benefits of n-gram models (and algorithms that use them) are simplicity and scalability – with larger n, a model can store more context with a well-understood space–time tradeoff, enabling small experiments to scale up efficiently.
Examples
Field | Unit | Sample sequence | 1-gram sequence | 2-gram sequence | 3-gram sequence |
---|---|---|---|---|---|
Vernacular name | unigram | bigram | trigram | ||
Order of resulting Markov model | 0 | 1 | 2 | ||
Protein sequencing | amino acid | … Cys-Gly-Leu-Ser-Trp … | …, Cys, Gly, Leu, Ser, Trp, … | …, Cys-Gly, Gly-Leu, Leu-Ser, Ser-Trp, … | …, Cys-Gly-Leu, Gly-Leu-Ser, Leu-Ser-Trp, … |
DNA sequencing | base pair | …AGCTTCGA… | …, A, G, C, T, T, C, G, A, … | …, AG, GC, CT, TT, TC, CG, GA, … | …, AGC, GCT, CTT, TTC, TCG, CGA, … |
Computational linguistics | character | …to_be_or_not_to_be… | …, t, o, _, b, e, _, o, r, _, n, o, t, _, t, o, _, b, e, … | …, to, o_, _b, be, e_, _o, or, r_, _n, no, ot, t_, _t, to, o_, _b, be, … | …, to_, o_b, _be, be_, e_o, _or, or_, r_n, _no, not, ot_, t_t, _to, to_, o_b, _be, … |
Computational linguistics | word | … to be or not to be … | …, to, be, or, not, to, be, … | …, to be, be or, or not, not to, to be, … | …, to be or, be or not, or not to, not to be, … |
Figure 1 shows several example sequences and the corresponding 1-gram, 2-gram and 3-gram sequences.
Here are further examples; these are word-level 3-grams and 4-grams (and counts of the number of times they appeared) from the Google n-gram corpus.[3]
- ceramics collectables collectibles (55)
- ceramics collectables fine (130)
- ceramics collected by (52)
- ceramics collectible pottery (50)
- ceramics collectibles cooking (45)
4-grams
- serve as the incoming (92)
- serve as the incubator (99)
- serve as the independent (794)
- serve as the index (223)
- serve as the indication (72)
- serve as the indicator (120)
n-gram models
An n-gram model models sequences, notably natural languages, using the statistical properties of n-grams.
This idea can be traced to an experiment by Claude Shannon's work in information theory. Shannon posed the question: given a sequence of letters (for example, the sequence "for ex"), what is the likelihood of the next letter? From training data, one can derive a probability distribution for the next letter given a history of size n {\displaystyle n} : a = 0.4, b = 0.00001, c = 0, ....; where the probabilities of all possible "next-letters" sum to 1.0...
More concisely, an n-gram model predicts x i {\displaystyle x_{i}} based on x i − ( n − 1 ) , … , x i − 1 {\displaystyle x_{i-(n-1)},\dots ,x_{i-1}} . In probability terms, this is P ( x i ∣ x i − ( n − 1 ) , … , x i − 1 ) {\displaystyle P(x_{i}\mid x_{i-(n-1)},\dots ,x_{i-1})} . When used for language modeling, independence assumptions are made so that each word depends only on the last n − 1 words. This Markov model is used as an approximation of the true underlying language. This assumption is important because it massively simplifies the problem of learning the language model from data. In addition, because of the open nature of language, it is common to group words unknown to the language model together.
Note that in a simple n-gram language model, the probability of a word, conditioned on some number of previous words (one word in a bigram model, two words in a trigram model, etc.) can be described as following a categorical distribution (often imprecisely called a "multinomial distribution").
In practice, the probability distributions are smoothed by assigning non-zero probabilities to unseen words or n-grams; see smoothing techniques.
Applications and considerations
n-gram models are widely used in statistical natural language processing. In speech recognition, phonemes and sequences of phonemes are modeled using a n-gram distribution. For parsing, words are modeled such that each n-gram is composed of n words. For language identification, sequences of characters/graphemes (e.g., letters of the alphabet) are modeled for different languages.[4] For sequences of characters, the 3-grams (sometimes referred to as "trigrams") that can be generated from "good morning" are "goo", "ood", "od ", "d m", " mo", "mor" and so forth (sometimes the beginning and end of a text are modeled explicitly, adding "__g", "_go", "ng_", and "g__"). For sequences of words, the trigrams that can be generated from "the dog smelled like a skunk" are "# the dog", "the dog smelled", "dog smelled like", "smelled like a", "like a skunk" and "a skunk #".
Practitioners[who?] more interested in multiple word terms might preprocess strings to remove spaces.[who?] Many simply collapse whitespace to a single space while preserving paragraph marks, because the whitespace is frequently either an element of writing style or introduces layout or presentation not required by the prediction and deduction methodology. Punctuation is also commonly reduced or removed by preprocessing and is frequently used to trigger functionality.
n-grams can also be used for sequences of words or almost any type of data. For example, they have been used for extracting features for clustering large sets of satellite earth images and for determining what part of the Earth a particular image came from.[5] They have also been very successful as the first pass in genetic sequence search and in the identification of the species from which short sequences of DNA originated.[6]
n-gram models are often criticized because they lack any explicit representation of long range dependency. This is because the only explicit dependency range is (n − 1) tokens for an n-gram model, and since natural languages incorporate many cases of unbounded dependencies (such as wh-movement), this means that an n-gram model cannot in principle distinguish unbounded dependencies from noise (since long range correlations drop exponentially with distance for any Markov model). For this reason, n-gram models have not made much impact on linguistic theory, where part of the explicit goal is to model such dependencies.
Another criticism that has been made is that Markov models of language, including n-gram models, do not explicitly capture the performance/competence distinction. This is because n-gram models are not designed to model linguistic knowledge as such, and make no claims to being (even potentially) complete models of linguistic knowledge; instead, they are used in practical applications.
In practice, n-gram models have been shown to be extremely effective in modeling language data, which is a core component in modern statistical language applications.
Most modern applications that rely on n-gram based models, such as machine translation applications, do not rely exclusively on such models; instead, they typically also incorporate Bayesian inference. Modern statistical models are typically made up of two parts, a prior distribution describing the inherent likelihood of a possible result and a likelihood function used to assess the compatibility of a possible result with observed data. When a language model is used, it is used as part of the prior distribution (e.g. to gauge the inherent "goodness" of a possible translation), and even then it is often not the only component in this distribution.
Handcrafted features of various sorts are also used, for example variables that represent the position of a word in a sentence or the general topic of discourse. In addition, features based on the structure of the potential result, such as syntactic considerations, are often used. Such features are also used as part of the likelihood function, which makes use of the observed data. Conventional linguistic theory can be incorporated in these features (although in practice, it is rare that features specific to generative or other particular theories of grammar are incorporated, as computational linguists tend to be "agnostic" towards individual theories of grammar[citation needed]).
Out-of-vocabulary words
An issue when using n-gram language models are out-of-vocabulary (OOV) words. They are encountered in computational linguistics and natural language processing when the input includes words which were not present in a system's dictionary or database during its preparation. By default, when a language model is estimated, the entire observed vocabulary is used. In some cases, it may be necessary to estimate the language model with a specific fixed vocabulary. In such a scenario, the n-grams in the corpus that contain an out-of-vocabulary word are ignored. The n-gram probabilities are smoothed over all the words in the vocabulary even if they were not observed.[7]
Nonetheless, it is essential in some cases to explicitly model the probability of out-of-vocabulary words by introducing a special token (e.g. <unk>) into the vocabulary. Out-of-vocabulary words in the corpus are effectively replaced with this special <unk> token before n-grams counts are cumulated. With this option, it is possible to estimate the transition probabilities of n-grams involving out-of-vocabulary words.[8]
n-grams for approximate matching
n-grams can also be used for efficient approximate matching. By converting a sequence of items to a set of n-grams, it can be embedded in a vector space, thus allowing the sequence to be compared to other sequences in an efficient manner. For example, if we convert strings with only letters in the English alphabet into single character 3-grams, we get a 26 3 {\displaystyle 26^{3}} -dimensional space (the first dimension measures the number of occurrences of "aaa", the second "aab", and so forth for all possible combinations of three letters). Using this representation, we lose information about the string. For example, both the strings "abc" and "bca" give rise to exactly the same 2-gram "bc" (although {"ab", "bc"} is clearly not the same as {"bc", "ca"}). However, we know empirically that if two strings of real text have a similar vector representation (as measured by cosine distance) then they are likely to be similar. Other metrics have also been applied to vectors of n-grams with varying, sometimes better, results. For example, z-scores have been used to compare documents by examining how many standard deviations each n-gram differs from its mean occurrence in a large collection, or text corpus, of documents (which form the "background" vector). In the event of small counts, the g-score (also known as g-test) may give better results for comparing alternative models.
Another method for approximate matching is signature files. The study reported in [9] shows that a bit-sliced signature file can be compressed to a smaller size than an inverted file which is the standard way of implementing a vector space approach. With a signature width less than half the number of unique n-grams, the signature file method is about as fast as the inverted file method, and significantly smaller.
It is also possible to take a more principled approach to the statistics of n-grams, modeling similarity as the likelihood that two strings came from the same source directly in terms of a problem in Bayesian inference.
n-gram-based searching can also be used for plagiarism detection.
Other applications
n-grams find use in several areas of computer science, computational linguistics, and applied mathematics.
They have been used to:
- design kernels that allow machine learning algorithms such as support vector machines to learn from string data
- find likely candidates for the correct spelling of a misspelled word
- improve compression in compression algorithms where a small area of data requires n-grams of greater length
- assess the probability of a given word sequence appearing in text of a language of interest in pattern recognition systems, speech recognition, OCR (optical character recognition), Intelligent Character Recognition (ICR), machine translation and similar applications
- improve retrieval in information retrieval systems when it is hoped to find similar "documents" (a term for which the conventional meaning is sometimes stretched, depending on the data set) given a single query document and a database of reference documents
- improve retrieval performance in genetic sequence analysis as in the BLAST family of programs
- identify the language a text is in or the species a small sequence of DNA was taken from
- predict letters or words at random in order to create text, as in the dissociated press algorithm.
- cryptanalysis
Bias-versus-variance trade-off
To choose a value for n in an n-gram model, it is necessary to find the right trade off between the stability of the estimate against its appropriateness. This means that trigram (i.e. triplets of words) is a common choice with large training corpora (millions of words), whereas a bigram is often used with smaller ones.
Smoothing techniques
There are problems of balance weight between infrequent grams (for example, if a proper name appeared in the training data) and frequent grams. Also, items not seen in the training data will be given a probability of 0.0 without smoothing. For unseen but plausible data from a sample, one can introduce pseudocounts. Pseudocounts are generally motivated on Bayesian grounds.
In practice it is necessary to smooth the probability distributions by also assigning non-zero probabilities to unseen words or n-grams. The reason is that models derived directly from the n-gram frequency counts have severe problems when confronted with any n-grams that have not explicitly been seen before – the zero-frequency problem. Various smoothing methods are used, from simple "add-one" (Laplace) smoothing (assign a count of 1 to unseen n-grams; see Rule of succession) to more sophisticated models, such as Good–Turing discounting or back-off models. Some of these methods are equivalent to assigning a prior distribution to the probabilities of the n-grams and using Bayesian inference to compute the resulting posterior n-gram probabilities. However, the more sophisticated smoothing models were typically not derived in this fashion, but instead through independent considerations.
- Linear interpolation (e.g., taking the weighted mean of the unigram, bigram, and trigram)
- Good–Turing discounting
- Witten–Bell discounting
- Lidstone's smoothing
- Katz's back-off model (trigram)
- Kneser–Ney smoothing
Skip-gram
In the field of computational linguistics, in particular language modeling, skip-grams[10] are a generalization of n-grams in which the components (typically words) need not be consecutive in the text under consideration, but may leave gaps that are skipped over.[11] They provide one way of overcoming the data sparsity problem found with conventional n-gram analysis.
Formally, an n-gram is a consecutive subsequence of length n of some sequence of tokens w1 … wn. A k-skip-n-gram is a length-n subsequence where the components occur at distance at most k from each other.
For example, in the input text:
- the rain in Spain falls mainly on the plain
the set of 1-skip-2-grams includes all the bigrams (2-grams), and in addition the subsequences
- the in, rain Spain, in falls, Spain mainly, falls on, mainly the, and on plain.
Syntactic n-grams
Syntactic n-grams are n-grams defined by paths in syntactic dependency or constituent trees rather than the linear structure of the text.[12][13][14] For example, the sentence "economic news has little effect on financial markets" can be transformed to syntactic n-grams following the tree structure of its dependency relations: news-economic, effect-little, effect-on-markets-financial.[12]
Syntactic n-grams are intended to reflect syntactic structure more faithfully than linear n-grams, and have many of the same applications, especially as features in a Vector Space Model. Syntactic n-grams for certain tasks gives better results than the use of standard n-grams, for example, for authorship attribution.[15]
See also
N-Gram的更多相关文章
- Gram格拉姆矩阵在风格迁移中的应用
Gram定义 n维欧式空间中任意k个向量之间两两的内积所组成的矩阵,称为这k个向量的格拉姆矩阵(Gram matrix) 根据定义可以看到,每个Gram矩阵背后都有一组向量,Gram矩阵就是由这一组向 ...
- Gram矩阵 迁移学习 one-shot 之类
格拉姆矩阵是由内积空间中的向量两两内积而得.格拉姆矩阵在向量为随机的情况下也是协方差矩阵.每个数字都来自于一个特定滤波器在特定位置的卷积,因此每个数字代表一个特征的强度,而Gram计算的实际上是两两特 ...
- LG Gram 2018 z980 白
因为今年8代处理器i5的双核变成了四核,感觉是个换电脑的好时机,本来打算买macbook,但是6月的发布会并没有发布,于是开始寻找一些比较有特点的笔记本电脑. 了解了这样一款笔记本 LG GRAM 1 ...
- Gram 矩阵性质及应用
v1,v2,-,vn 是内积空间的一组向量,Gram 矩阵定义为: Gij=⟨vi,vj⟩,显然其是对称矩阵. 其实对于一个XN⋅d(N 个样本,d 个属性)的样本矩阵而言,X⋅X′ 即为 Gram ...
- 【线性代数】4-4:正交基和Gram算法(Orthogonal Bases and Gram-Schmidt)
title: [线性代数]4-4:正交基和Gram算法(Orthogonal Bases and Gram-Schmidt) categories: Mathematic Linear Algebra ...
- 如何给LG gram写一个Linux下的驱动?
其实就是实现一下几个Fn键的功能,没有标题吹得那么牛. 不知道为啥,LG gram这本子意外的小众. 就因为这个,装Linux遇到的硬件问题就没法在网上直接搜到解决办法了. Fn + F9 实现阅读模 ...
- LG gram 双系统全指南
LG gram 双系统全指南 为了和同学联机玩帝国时代2,以及为了下学期的编程课,五年没用过 Windows 的我决定装 Ubuntu20.04 LTS / WIndows 10 双系统了. 我的 L ...
- Gram 矩阵与向量到子空间的距离
设 $W$ 是 $n$ 维 Euclidean 空间 $V$ 的子空间, $\beta\in V$, 定义 $\beta$ 到 $W$ 的距离 $$\bex \rd (\beta,W)=|\bet ...
- 让IIS7.0.0.0支持 .iso .7z .torrent .apk等文件下载的设置方法
IIS默认支持哪些MIME类型呢,我们可以这样查看:打开IIS管理器(计算机--管理--服务和应用程序--Internet信息服务(IIS)管理器:或者Win+R,输入inetmgr,Enter),在 ...
- CFD冲蚀模拟的一些理论
[TOC] 在CFD中计算颗粒对固体壁面的冲蚀往往采用冲蚀模型(Erosion Model). 1 冲蚀速率(Erosion Rate) 冲蚀速率定义为壁面材料在单位时间单位面积上损失的质量(单位:\ ...
随机推荐
- MyEclipse下Maven的安装配置
Maven常用命令: •mvn archetype:generate :创建 Maven 项目 •mvn compile :编译源代码 •mvn test-compile :编译测试代码 •mvn t ...
- [转]为什么我要用 Node.js? 案例逐一介绍
原文地址:http://blog.jobbole.com/53736/ 介绍 JavaScript 高涨的人气带来了很多变化,以至于如今使用其进行网络开发的形式也变得截然不同了.就如同在浏览器中一样, ...
- IOS APP开发中View的几种实现方式
xib文件有以下几个重要的属性: xib文件名 File’s Owner xib文件中的视图的Class xib文件中的视图的Outlet指向 File’s Owner 可以关联到某类,然后通过IBO ...
- Xmanager注册吗
xmanager4.0注册吗 --
- Maven-搭建普通maven项目
点击Eclipse菜单栏File->New->Ohter->Maven得到如下图所示对话框: 选中Maven Project并点击Next,到下一个对话框(默认)继续点击Next得到 ...
- 【深入Java虚拟机】之二:Java垃圾回收机制
[深入Java虚拟机]之:Java垃圾收集机制 对象引用 Java中的垃圾回收一般是在Java堆中进行,因为堆中几乎存放了Java中所有的对象实例.谈到Java堆中的垃圾回收,自然要谈到引用.在JDK ...
- PyCharm2016.23专业版注册码
43B4A73YYJ-eyJsaWNlbnNlSWQiOiI0M0I0QTczWVlKIiwibGljZW5zZWVOYW1lIjoibGFuIHl1IiwiYXNzaWduZWVOYW1lIjoiI ...
- 虚拟机NAT模式无法上网问题的解决办法
在使用CentOS虚拟机时,出现了无法上网的情况,使用主机ping虚机地址可以ping通,而虚机ping不通主机,同时虚机也无法ping通其他的网址或ip,显示内容为Network is unreac ...
- 【转】Private Libraries、Referenced Libraries、Dependency Libraries的区别
一.v4.v7.v13的作用和用法 1.Android Support V4, V7, V13是什么? 本质上就是三个java library. 2.为什么要有support库? 是为了解决软件的 ...
- OPENGGL深度测试
深度测试是为了解决那些在绘图过程中本应该被隐藏的面结果却出现了,例如: 绘图代码中先绘制了一个一个近处的立方体,后绘制了一个远处的立方体,结果在绘制过程中,远处的立方体总是在近处的立方体后绘制,所以在 ...