bert中的分词
直接把自己的工作文档导入的,由于是在外企工作,所以都是英文写的
chinese and english tokens result
input: "我爱中国",tokens:["我","爱","中","国"]
input: "I love china habih", tokens:["I","love","china","ha","##bi","##h"] (here "##bi","##h" are all in vocabulary)
Implementation
chinese and english text would call two tokens,one is basic_tokenizer and other one is wordpiece_tokenizer as you can see from the the code below.
basic_tokenizer
if the input is chinese, the _tokenize_chinese_chars would add whitespace between chinese charater, and then call whitespace_tokenizer which separate text with whitespace,so
if the input is the query "我爱中国",would return ["我","爱","中","国"],if the input is the english query "I love china", would return ["I","love","china"]
wordpiece_tokenizer
if the input is chinese ,if would iterate tokens from basic_tokenizer result, if the character is in vocabulary, just keep the same character ,otherwise append unk_token.
if the input is english, we would iterate over one word ,for example: the word is "shabi", while it is not in vocabulary,so the end index would rollback until it found "sh" in vocabulary,
in the following process, once it found a substr in vocabulary ,it would append "##" then append it to output tokens, so we can get ["sh","##ab","##i"] finally.
#65 How are out of vocabulary words handled for Chinese?
The top 8000 characters are character-tokenized, other characters are mapped to [UNK]. I should've commented this section better but it's here
Basically that's saying if it tries to apply WordPiece tokenization (the character tokenization happens previously), and it gets to a single character that it can't find, it maps it to unk_token.
#62 Why Chinese vocab contains ##word?
This is the character used to denote WordPieces, it's just an artifact of the WordPiece vocabulary generator that we use, but most of those words were never actually used during training (for Chinese). So you can just ignore those tokens. Note that for the English characters that appear in Chinese text they are actually used.
bert中的分词的更多相关文章
- 一文读懂BERT中的WordPiece
1. 前言 2018年最火的论文要属google的BERT,不过今天我们不介绍BERT的模型,而是要介绍BERT中的一个小模块WordPiece. 2. WordPiece原理 现在基本性能好一些的N ...
- 广告行业中那些趣事系列8:详解BERT中分类器源码
最新最全的文章请关注我的微信公众号:数据拾光者. 摘要:BERT是近几年NLP领域中具有里程碑意义的存在.因为效果好和应用范围广所以被广泛应用于科学研究和工程项目中.广告系列中前几篇文章有从理论的方面 ...
- Python中结巴分词使用手记
手记实用系列文章: 1 结巴分词和自然语言处理HanLP处理手记 2 Python中文语料批量预处理手记 3 自然语言处理手记 4 Python中调用自然语言处理工具HanLP手记 5 Python中 ...
- Elasticsearch中的分词器比较及使用方法
Elasticsearch 默认分词器和中分分词器之间的比较及使用方法 https://segmentfault.com/a/1190000012553894 介绍:ElasticSearch 是一个 ...
- ES中的分词器
基本概念: 全文搜索引擎会用某种算法对要建索引的文档进行分析, 从文档中提取出若干Token(词元), 这些算法称为Tokenizer(分词器), 这些Token会被进一步处理, 比如转成小写等, 这 ...
- 开源自然语言处理工具包hanlp中CRF分词实现详解
CRF简介 CRF是序列标注场景中常用的模型,比HMM能利用更多的特征,比MEMM更能抵抗标记偏置的问题. [gerative-discriminative.png] CRF训练 这类耗时的任务,还 ...
- sklearn中的分词函数countVectorizer()的改动--保留长度为1的字符串
1简述问题 使用countVectorizer()将文本向量化时发现,文本中长度唯一的字符串会被自动过滤掉,这对于我在做的情感分析来讲,一些表较重要的表达情感倾向的词汇被过滤掉,比如文本'没用的东西, ...
- nlp任务中的传统分词器和Bert系列伴生的新分词器tokenizers介绍
layout: blog title: Bert系列伴生的新分词器 date: 2020-04-29 09:31:52 tags: 5 categories: nlp mathjax: true ty ...
- 中文分词中的战斗机-jieba库
英文分词的第三方库NLTK不错,中文分词工具也有很多(盘古分词.Yaha分词.Jieba分词等).但是从加载自定义字典.多线程.自动匹配新词等方面来看. 大jieba确实是中文分词中的战斗机. 请随意 ...
随机推荐
- shopnc 手机网站配置
完整版SHOPNC下载地址:http://www.fenxiangweb.com/thread-7643-1-1.html SHOPNC手机wap版安装教程 1.进入根目录下mobile/readme ...
- office 激活教程
首先在我的百度云里下载:http://pan.baidu.com/share/link?shareid=2200272243&uk=1985086665激活工具安装包 如果遇到问题,可以加我Q ...
- 安装MACOS操作步骤详解
安装MACOS操作步骤详解 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 关于安装MAC的操作系统其实大家都知道可以让客服帮忙提供软件上的支持,而且苹果客服都很有礼貌呢,而且非常的 ...
- <T extends Comparable<? super T>>什么意思
<T extends Comparable<? super T>>首先这是运用了java的泛型①extends后面跟的类型如<任意字符 extends 类/接口>表 ...
- cdqz2017-test8-Tree(点分树)
n个点的带点权带边权的树,设点权为a[i],边权为b[i] 一棵树有n*(n-1)/2个点对, 定义这棵树的价值为任意两点对的(a[x]^a[y])*dis(x,y) 有m次修改一个点的点权的操作 输 ...
- POJ No.3255 Roadblocks 求次短路径
#define _CRT_SECURE_NO_WARNINGS /* 7 10 0 1 5 0 2 2 1 2 4 1 3 2 2 3 6 2 4 10 3 5 1 4 5 3 4 6 5 5 6 9 ...
- ThinkPHP 3.2 vendor()方法的深入研究及Phpqrcode的正确扩展
ThinkPHP vendor 方法导入第三方类库 第三方类库 第三方类库指除了 ThinkPHP 框架.应用项目类库之外的其他类库,一般由第三方系统或产品提供,如 Smarty.Zend 等系统的类 ...
- js_倒计时去执行函数或则跳转页面
js_倒计时去执行函数或则跳转页面: var wait = 5; $(document).ready(function () { returnPage(); }); function returnPa ...
- vue 裁剪图片,插件Cropper的使用
全局安装 npm install cropperjs 如果想本项目安装,方便移植: import Cropper from 'cropperjs' --save 这样的话,本地 p ...
- C# CEF 封装UserControl
using System; using System.Collections.Generic; using System.ComponentModel; using System.Drawing; u ...