Initial commit
This commit is contained in:
Spencer Pincott committed
commit
97737ca1ae
16618 files changed
+934131
No files matched your search
+21
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2017 lin onetwo
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+194
@@ -0,0 +1,194 @@
|
||||
<p>
|
||||
<a href='https://badge.fury.io/js/segmentit' style='margin: 0 0.5rem;'>
|
||||
<img src='https://badge.fury.io/js/segmentit.svg' alt='npm version' height='18'/>
|
||||
</a>
|
||||
|
||||
<a href='https://coveralls.io/github/linonetwo/segmentit?branch=master' style='margin: 0 0.5rem;'>
|
||||
<img src='https://coveralls.io/repos/github/linonetwo/segmentit/badge.svg?branch=master' alt='Coverage Status' height='18'/>
|
||||
</a>
|
||||
|
||||
<a href='https://travis-ci.org/linonetwo/segmentit#' style='margin: 0 0.5rem;'>
|
||||
<img src='https://api.travis-ci.org/linonetwo/segmentit.svg?branch=master' alt='CI Status' height='18'/>
|
||||
</a>
|
||||
|
||||
<a href='https://img.shields.io/bundlephobia/minzip/segmentit.svg' style='margin: 0 0.5rem;'>
|
||||
<img src='https://img.shields.io/bundlephobia/minzip/segmentit.svg' alt='Min Zip Size' height='18'/>
|
||||
</a>
|
||||
<p>
|
||||
|
||||
# 中文分词模块
|
||||
|
||||
本模块基于 [node-segment](https://github.com/leizongmin/node-segment) 魔改，增加了 electron、浏览器支持，并准备针对 electron 多线程运行环境进行优化。
|
||||
|
||||
之所以要花时间魔改，是因为 `segment` 和 `nodejieba` 虽然在 node 环境下很好用，但根本无法在浏览器和 electron 环境下运行。我把代码重构为 ES2015，并用 babel 插件内联了字典文件，全部载入的话大小是 3.8M，但如果有些字典你并不需要，字典和模块是支持 tree shaking 的（请使用 ESM 模块）。
|
||||
|
||||
## Usage
|
||||
|
||||
```javascript
|
||||
import { Segment, useDefault } from 'segmentit';
|
||||
|
||||
const segmentit = useDefault(new Segment());
|
||||
const result = segmentit.doSegment('工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作。');
|
||||
console.log(result);
|
||||
```
|
||||
|
||||
对于 runkit 环境：
|
||||
|
||||
```javascript
|
||||
const { Segment, useDefault } = require('segmentit');
|
||||
const segmentit = useDefault(new Segment());
|
||||
const result = segmentit.doSegment('工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作。');
|
||||
console.log(result);
|
||||
```
|
||||
|
||||
[在 Runkit 上免费试用](https://npm.runkit.com/segmentit)
|
||||
|
||||
## 获取词类标注
|
||||
|
||||
结巴分词风格的词类标注：
|
||||
|
||||
```javascript
|
||||
// import Segment, { useDefault, cnPOSTag, enPOSTag } from 'segmentit';
|
||||
const = require('segmentit').default;
|
||||
const { Segment, useDefault, cnPOSTag, enPOSTag } = require('segmentit');
|
||||
|
||||
const segmentit = useDefault(new Segment());
|
||||
|
||||
console.log(segmentit.doSegment('一人得道，鸡犬升天').map(i => `${i.w} <${cnPOSTag(i.p)}> <${enPOSTag(i.p)}>`))
|
||||
// ↑ ["一人得道 <习语,数词 数语素> <l,m>", "， <标点符号> <w>", "鸡犬升天 <成语> <i>"]
|
||||
```
|
||||
|
||||
## 只使用部分词典或使用自定义词典
|
||||
|
||||
useDefault 的具体实现是这样的：
|
||||
|
||||
```javascript
|
||||
// useDefault
|
||||
import { Segment, modules, dicts, synonyms, stopwords } from 'segmentit';
|
||||
|
||||
const segmentit = new Segment();
|
||||
segmentit.use(modules);
|
||||
segmentit.loadDict(dicts);
|
||||
segmentit.loadSynonymDict(synonyms);
|
||||
segmentit.loadStopwordDict(stopwords);
|
||||
```
|
||||
|
||||
因此你实际上可以 import 所需的那部分字典和模块，然后一个个如下载入。没有 import 的那些字典和模块应该会被 webpack 的 tree shaking 去掉。你也可以这样载入自己定义的字典文件，只需要主要 loadDict 的函数签名是 `(dicts: string | string[]): Segment`。
|
||||
|
||||
```javascript
|
||||
// load custom module and dicts
|
||||
import {
|
||||
Segment,
|
||||
ChsNameTokenizer,
|
||||
DictOptimizer,
|
||||
EmailOptimizer,
|
||||
PunctuationTokenizer,
|
||||
URLTokenizer,
|
||||
ChsNameOptimizer,
|
||||
DatetimeOptimizer,
|
||||
DictTokenizer,
|
||||
ForeignTokenizer,
|
||||
SingleTokenizer,
|
||||
WildcardTokenizer,
|
||||
pangu,
|
||||
panguExtend1,
|
||||
panguExtend2,
|
||||
names,
|
||||
wildcard,
|
||||
synonym,
|
||||
stopword,
|
||||
} from 'segmentit';
|
||||
|
||||
const segmentit = new Segment();
|
||||
|
||||
// load them one by one, or by array
|
||||
segmentit.use(ChsNameTokenizer);
|
||||
segmentit.loadDict(pangu);
|
||||
segmentit.loadDict([panguExtend1, panguExtend2]);
|
||||
segmentit.loadSynonymDict(synonym);
|
||||
segmentit.loadStopwordDict(stopword);
|
||||
```
|
||||
|
||||
盘古的词典比较复古了，像「软萌萝莉」这种词都是没有的，请有能力的朋友 PR 一下自己的词库。
|
||||
|
||||
## 创造自己的分词中间件（Tokenizer）和结果优化器（Optimizer）
|
||||
|
||||
### Tokenizer
|
||||
|
||||
Tokenizer 是分词时要经过的一个个中间件，类似于 Redux 的 MiddleWare，它的 split 函数接受分词分到一半的 token 数组，返回一个同样格式的 token 数组（这也就是不要对太长的文本分词的原因，不然这个数组会巨爆大）。
|
||||
|
||||
例子如下：
|
||||
|
||||
```javascript
|
||||
// @flow
|
||||
import { Tokenizer } from 'segmentit';
|
||||
import type { SegmentToken, TokenStartPosition } from 'segmentit';
|
||||
export default class ChsNameTokenizer extends Tokenizer {
|
||||
split(words: Array<SegmentToken>): Array<SegmentToken> {
|
||||
// 可以获取到 this.segment 里的各种信息
|
||||
const POSTAG = this.segment.POSTAG;
|
||||
const TABLE = this.segment.getDict('TABLE');
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Optimizer
|
||||
|
||||
Optimizer 是在分词结束后，发现有些难以利用字典处理的情况，却可以用启发式规则处理时，可以放这些启发式规则的地方，它的 doOptimize 函数同样接收一个 token 数组，返回一个同样格式的 token 数组。
|
||||
|
||||
除了 token 数组以外，你还可以自定义余下的参数，比如在下面的例子里，我们会递归调用自己一次，通过第二个参数判断递归深度：
|
||||
|
||||
```javascript
|
||||
// @flow
|
||||
import { Optimizer } from './BaseModule';
|
||||
import type { SegmentToken } from './type';
|
||||
export default class DictOptimizer extends Optimizer {
|
||||
doOptimize(words: Array<SegmentToken>, isNotFirst: boolean): Array<SegmentToken> {
|
||||
// 可以获取到 this.segment 里的各种信息
|
||||
const POSTAG = this.segment.POSTAG;
|
||||
const TABLE = this.segment.getDict('TABLE');
|
||||
// ...
|
||||
// 针对组合数字后无法识别新组合的数字问题，需要重新扫描一次
|
||||
return isNotFirst === true ? words : this.doOptimize(words, true);
|
||||
}
|
||||
```
|
||||
|
||||
例如目前各种分词工具都没法把「一条红色内裤」中的红色标对词性，但在 segmentit 里我加了个简单的 AdjectiveOptimizer 来处理它：
|
||||
|
||||
```javascript
|
||||
// @flow
|
||||
// https://github.com/linonetwo/segmentit/blob/master/src/module/AdjectiveOptimizer.js
|
||||
import { Optimizer } from './BaseModule';
|
||||
import type { SegmentToken } from './type';
|
||||
|
||||
import { colors } from './COLORS';
|
||||
|
||||
// 把一些错认为名词的词标注为形容词，或者对名词作定语的情况
|
||||
export default class AdjectiveOptimizer extends Optimizer {
|
||||
doOptimize(words: Array<SegmentToken>): Array<SegmentToken> {
|
||||
const { POSTAG } = this.segment;
|
||||
let index = 0;
|
||||
while (index < words.length) {
|
||||
const word = words[index];
|
||||
const nextword = words[index + 1];
|
||||
if (nextword) {
|
||||
// 对于<颜色>+<的>，直接判断颜色是形容词（字典里颜色都是名词）
|
||||
if (nextword.p === POSTAG.D_U && colors.includes(word.w)) {
|
||||
word.p = POSTAG.D_A;
|
||||
}
|
||||
// 如果是连续的两个名词，前一个是颜色，那这个颜色也是形容词
|
||||
if (word.p === POSTAG.D_N && nextword.p === POSTAG.D_N && colors.includes(word.w)) {
|
||||
word.p = POSTAG.D_A;
|
||||
}
|
||||
}
|
||||
// 移到下一个单词
|
||||
index += 1;
|
||||
}
|
||||
return words;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT LICENSED
|
||||
+1
@@ -0,0 +1 @@
|
||||
"use strict";module.exports="production"===process.env.NODE_ENV?require("./segmentit.min.js"):require("./segmentit.js");
|
||||
+2357
File diff suppressed because it is too large.
Load diff
+169
@@ -0,0 +1,169 @@
|
||||
"use strict"
|
||||
function _classCallCheck(n,x){if(!(n instanceof x))throw new TypeError("Cannot call a class as a function")}function _defineProperties(n,x){for(var e=0;e<x.length;e++){var t=x[e]
|
||||
t.enumerable=t.enumerable||!1,t.configurable=!0,"value"in t&&(t.writable=!0),Object.defineProperty(n,t.key,t)}}function _createClass(n,x,e){return x&&_defineProperties(n.prototype,x),e&&_defineProperties(n,e),n}function _defineProperty(n,x,e){return x in n?Object.defineProperty(n,x,{value:e,enumerable:!0,configurable:!0,writable:!0}):n[x]=e,n}function _inherits(n,x){if("function"!=typeof x&&null!==x)throw new TypeError("Super expression must either be null or a function")
|
||||
n.prototype=Object.create(x&&x.prototype,{constructor:{value:n,writable:!0,configurable:!0}}),x&&_setPrototypeOf(n,x)}function _getPrototypeOf(n){return(_getPrototypeOf=Object.setPrototypeOf?Object.getPrototypeOf:function(n){return n.__proto__||Object.getPrototypeOf(n)})(n)}function _setPrototypeOf(n,x){return(_setPrototypeOf=Object.setPrototypeOf||function(n,x){return n.__proto__=x,n})(n,x)}function _assertThisInitialized(n){if(void 0===n)throw new ReferenceError("this hasn't been initialised - super() hasn't been called")
|
||||
return n}function _possibleConstructorReturn(n,x){return!x||"object"!=typeof x&&"function"!=typeof x?_assertThisInitialized(n):x}function _toConsumableArray(n){return _arrayWithoutHoles(n)||_iterableToArray(n)||_nonIterableSpread()}function _arrayWithoutHoles(n){if(Array.isArray(n)){for(var x=0,e=Array(n.length);x<n.length;x++)e[x]=n[x]
|
||||
return e}}function _iterableToArray(n){if(Symbol.iterator in Object(n)||"[object Arguments]"===Object.prototype.toString.call(n))return Array.from(n)}function _nonIterableSpread(){throw new TypeError("Invalid attempt to spread non-iterable instance")}Object.defineProperty(exports,"__esModule",{value:!0})
|
||||
var POSTAG={D_A:1073741824,D_B:536870912,D_C:268435456,D_D:134217728,D_E:67108864,D_F:33554432,D_I:16777216,D_L:8388608,A_M:4194304,D_MQ:2097152,D_N:1048576,D_O:524288,D_P:262144,A_Q:131072,D_R:65536,D_S:32768,D_T:16384,D_U:8192,D_V:4096,D_W:2048,D_X:1024,D_Y:512,D_Z:256,A_NR:128,A_NS:64,A_NT:32,A_NX:16,A_NZ:8,D_ZH:4,D_K:2,UNK:0,URL:1},CN_POS_NAMES={D_A:"形容词 形语素",D_B:"区别词 区别语素",D_C:"连词 连语素",D_D:"副词 副语素",D_E:"叹词 叹语素",D_F:"方位词 方位语素",D_I:"成语",D_L:"习语",A_M:"数词 数语素",D_MQ:"数量词",D_N:"名词 名语素",D_O:"拟声词",D_P:"介词",A_Q:"量词 量语素",D_R:"代词 代语素",D_S:"处所词",D_T:"时间词",D_U:"助词 助语素",D_V:"动词 动语素",D_W:"标点符号",D_X:"非语素字",D_Y:"语气词 语气语素",D_Z:"状态词",A_NR:"人名",A_NS:"地名",A_NT:"机构团体",A_NX:"外文字符",A_NZ:"其他专名",D_ZH:"前接成分",D_K:"后接成分",UNK:"未知",URL:"网址 邮箱地址"},EN_POS_NAMES={D_A:"a",D_B:"b",D_C:"c",D_D:"d",D_E:"e",D_F:"f",D_I:"i",D_L:"l",A_M:"m",D_MQ:"mq",D_N:"n",D_O:"o",D_P:"p",A_Q:"q",D_R:"r",D_S:"s",D_T:"t",D_U:"u",D_V:"v",D_W:"w",D_X:"x",D_Y:"y",D_Z:"z",A_NR:"nr",A_NS:"ns",A_NT:"nt",A_NX:"nx",A_NZ:"nz",D_ZH:"h",D_K:"k",UNK:"un",URL:"uri"}
|
||||
function getPOSTagTranslator(t,c){return function(n){if(isNaN(n))return c[n]||c.UNK
|
||||
var x=[]
|
||||
for(var e in t)0<(n&t[e])&&x.push(c[e])
|
||||
return x.length<1?c.UNK:""+x}}var cnPOSTag=getPOSTagTranslator(POSTAG,CN_POS_NAMES),enPOSTag=getPOSTagTranslator(POSTAG,EN_POS_NAMES),Tokenizer=function(){function x(n){_classCallCheck(this,x),this.segment=n}return _createClass(x,[{key:"split",value:function(n,x){if(x.length<1)throw Error("No tokenizer module!")
|
||||
var e=[{w:n}]
|
||||
return x.forEach(function(n){e=n.split(e)}),e}}]),x}(),Optimizer=function(){function x(n){_classCallCheck(this,x),this.segment=n}return _createClass(x,[{key:"doOptimize",value:function(n,x){var e=_toConsumableArray(n)
|
||||
return x.forEach(function(n){e=n.doOptimize(e)}),e}}]),x}(),Segment=function(){function n(){var i=this
|
||||
_classCallCheck(this,n),_defineProperty(this,"use",function(n){if(Array.isArray(n))n.forEach(i.use)
|
||||
else if("function"==typeof n.init)n.init(i),i.modules[n.type].push(n)
|
||||
else{var x=new n(i)
|
||||
i.modules[x.type].push(x)}return i}),_defineProperty(this,"loadDict",function(n){var type=1<arguments.length&&void 0!==arguments[1]?arguments[1]:"TABLE",x=2<arguments.length&&void 0!==arguments[2]&&arguments[2]
|
||||
if(Array.isArray(n))n.forEach(function(n){return i.loadDict(n)})
|
||||
else{i.DICT[type]||(i.DICT[type]={}),i.DICT["".concat(type,"2")]||(i.DICT["".concat(type,"2")]={})
|
||||
var t=i.DICT[type],c=i.DICT["".concat(type,"2")]
|
||||
n.split(/\r?\n/).map(function(n){return x?n.toLowerCase():n}).forEach(function(n){var x=n.split("|")
|
||||
if(2<x.length){var e=x[0].trim()
|
||||
0<e.length&&(t[e]={f:+x[2],p:+x[1]},c[e.length]||(c[e.length]={}),c[e.length][e]=t[e])}})}return i}),_defineProperty(this,"getDict",function(type){return i.DICT[type]}),_defineProperty(this,"loadSynonymDict",function(n){if(Array.isArray(n))n.forEach(function(n){return i.loadSynonymDict(n)})
|
||||
else{i.DICT.SYNONYM||(i.DICT.SYNONYM={})
|
||||
var t=i.DICT.SYNONYM
|
||||
n.split(/\r?\n/).map(function(n){return n.split(",")}).forEach(function(n){if(1<n.length){var x=n[0].trim(),e=n[1].trim()
|
||||
t[x]=e,t[e]===x&&delete t[e]}})}return i}),_defineProperty(this,"loadStopwordDict",function(n){if(Array.isArray(n))n.forEach(function(n){return i.loadStopwordDict(n)})
|
||||
else{var type="STOPWORD"
|
||||
i.DICT[type]||(i.DICT[type]={})
|
||||
var x=i.DICT[type]
|
||||
n.split(/\r?\n/).map(function(n){return n.trim()}).forEach(function(n){n&&(x[n]=!0)})}return i}),_defineProperty(this,"doSegment",function(n,x){var t=i
|
||||
x=x||{}
|
||||
var e=[]
|
||||
function c(n){var x=0,e=t.getDict("SYNONYM")
|
||||
return n=n.map(function(n){return n.w in e?(x++,{w:e[n.w],p:n.p}):n}),{count:x,list:n}}if(n.replace(/\r/g,"\n").split(/\n+/).forEach(function(n){if(!((n=n.trim()).length<1)){var x=t.tokenizer.split(n,t.modules.tokenizer)
|
||||
0<(x=t.optimizer.doOptimize(x,t.modules.optimizer)).length&&(e=e.concat(x))}}),x.stripPunctuation&&(e=e.filter(function(n){return n.p!==POSTAG.D_W})),x.convertSynonym)do{var r=c(e)
|
||||
e=r.list}while(0<r.count)
|
||||
if(x.stripStopword){var f=t.getDict("STOPWORD")
|
||||
e=e.filter(function(n){return!(n.w in f)})}return x.simple&&(e=e.map(function(n){return n.w})),e}),this.POSTAG=POSTAG,this.DICT={},this.modules={tokenizer:[],optimizer:[]},this.tokenizer=new Tokenizer(this),this.optimizer=new Optimizer(this)}return _createClass(n,[{key:"toString",value:function(n){return n.map(function(n){return n.w}).join("")}},{key:"split",value:function(n,x){for(var e=[],t=0,c=0,r="string"==typeof x?"w":"p";c<n.length;)n[c][r]===x?(t<c&&e.push(n.slice(t,c)),e.push(n.slice(c,c+1)),t=++c):c++
|
||||
return t<n.length-1&&e.push(n.slice(t,n.length)),e}},{key:"indexOf",value:function(n,x,e){e=isNaN(e)?0:e
|
||||
for(var t="string"==typeof x?"w":"p";e<n.length;){if(n[e][t]===x)return e
|
||||
e++}return-1}}]),n}(),Module=function n(x){_classCallCheck(this,n),_defineProperty(this,"type",void 0),this.segment=x},Tokenizer$1=function(){function r(){var n,x
|
||||
_classCallCheck(this,r)
|
||||
for(var e=arguments.length,t=Array(e),c=0;c<e;c++)t[c]=arguments[c]
|
||||
return _defineProperty(_assertThisInitialized(x=_possibleConstructorReturn(this,(n=_getPrototypeOf(r)).call.apply(n,[this].concat(t)))),"type","tokenizer"),x}return _inherits(r,Module),r}(),Optimizer$1=function(){function r(){var n,x
|
||||
_classCallCheck(this,r)
|
||||
for(var e=arguments.length,t=Array(e),c=0;c<e;c++)t[c]=arguments[c]
|
||||
return _defineProperty(_assertThisInitialized(x=_possibleConstructorReturn(this,(n=_getPrototypeOf(r)).call.apply(n,[this].concat(t)))),"type","optimizer"),x}return _inherits(r,Module),r}(),COLOR_WITH_RGB=[["薰衣草紫红","#fff0f5","255,245,245"],["淡藕合","#f0dfee","240,238,238"],["甘石粉","#ffe8f3","255,243,243"],["浅粉红","#ffd9e6","255,230,230"],["浅血牙","#ffd7ea","255,234,234"],["粉凤仙","#fccce6","252,230,230"],["浅珍珠红","#ffb3e6","255,230,230"],["淡粉","#ffb3cc","255,204,204"],["火鹤红","#e68ab8","230,184,184"],["豆棕","#502e3e","80,62,62"],["茄皮紫","#3a212f","58,47,47"],["蒲萄灰","#4e2c3f","78,63,63"],["荷花","#eb7faf","235,175,175"],["灰紫红","#db7093","219,147,147"],["浅珊瑚红","#ff80bf","255,191,191"],["暗粉","#ff77bb","255,187,187"],["陈玫红","#b85798","184,152,152"],["肉红","#d26091","210,145,145"],["尖晶石红","#ff73b3","255,179,179"],["银红","#c85b92","200,146,146"],["暖粉红","#ff69b4","255,180,180"],["浅玫瑰红","#ff66cc","255,204,204"],["紫扇贝色","#923a60","146,96,96"],["长春花色","#ff47d1","255,209,209"],["山茶红","#e63995","230,149,149"],["中青紫红","#c71585","199,133,133"],["深粉红","#ff1493","255,147,147"],["洋玫瑰红","#ff0da6","255,166,166"],["玫瑰红","#ff007f","255,127,127"],["品红","#f400a1","244,161,161"],["胭脂红","#e6005c","230,92,92"],["艶粉","#e6005c","230,92,92"],["红宝石色","#cc0080","204,128,128"],["枢机红","#990036","153,54,54"],["毅红","#8a002e","138,46,46"],["牵牛紫","#800040","128,64,64"],["紫粉","#80002b","128,43,43"],["鶏冠花红","#660033","102,51,51"],["枣红","#640021","100,33,33"],["紫绛色","#51001b","81,27,27"],["玫瑰灰","#4d0026","77,38,38"],["勃艮第酒红","#470024","71,36,36"],["黒紫","#3e0014","62,20,20"],["雪紫","#33001a","51,26,26"],["深红","#33001a","51,26,26"],["雪色","#fffafa","255,250,250"],["浅肉粉","#f5eeeb","245,235,235"],["粉棕","#ebdad3","235,211,211"],["雾玫瑰色","#ffe4e1","255,225,225"],["米红","#fddfdf","253,223,223"],["沙棕","#e6c3c3","230,195,195"],["肉粉","#ffd7d7","255,215,215"],["奶棕","#deb6b6","222,182,182"],["玫瑰褐","#bc8f8f","188,143,143"],["红灰莲","#ab8282","171,130,130"],["粉红","#ffc0cb","255,203,203"],["亮粉红","#ffb6c1","255,193,193"],["壳黄红","#ffb3bf","255,191,191"],["浅粉","#ffb3b3","255,179,179"],["十様锦","#ffb3b3","255,179,179"],["深烟","#311f1e","49,30,30"],["凤仙粉","#ff9393","255,147,147"],["亮珊瑚色","#f08080","240,128,128"],["暗鲑红","#e9967a","233,122,122"],["绛紫","#492525","73,37,37"],["浅鲑红","#ff8099","255,153,153"],["紫铜色","#6b3636","107,54,54"],["珈琲色","#623131","98,49,49"],["烟栗色","#221311","34,17,17"],["亮鲑红","#ffa07a","255,122,122"],["红莲灰","#c76079","199,121,121"],["鲑红","#fa8072","250,114,114"],["印度红","#cd5c5c","205,92,92"],["银朱","#c85554","200,84,84"],["赦红","#d1634b","209,75,75"],["二硃","#c24545","194,69,69"],["芯木色","#c14444","193,68,68"],["銹红","#4d1919","77,25,25"],["珊瑚红","#ff7f50","255,80,80"],["辰沙","#ce3b3b","206,59,59"],["红扇贝色","#ce3b3b","206,59,59"],["蕃茄红","#ff6347","255,71,71"],["柿子橙","#ff4d40","255,64,64"],["妃红","#b8542e","184,46,46"],["甎红","#ab2b2b","171,43,43"],["褐色","#a52a2a","165,42,42"],["硃砂","#972626","151,38,38"],["土红","#bc2e2e","188,46,46"],["樱桃红","#de3163","222,99,99"],["砖红色","#b22222","178,34,34"],["茜红","#e32636","227,54,54"],["银星海棠","#f52443","245,67,67"],["橙色","#fe621f","254,31,31"],["桔红","#ff441a","255,26,26"],["柿子色","#ff531a","255,26,26"],["绯红","#dc143c","220,60,60"],["月季红","#e10b2b","225,43,43"],["鸟罗松红","#c80926","200,38,38"],["蕉红色","#d70419","215,25,25"],["艶红","#bf0417","191,23,23"],["烟色","#2e0a01","46,1,1"],["牡丹红","#b80233","184,51,51"],["榴花红","#b02502","176,2,2"],["枣紫","#540101","84,1,1"],["红色","#ff0000","255,0,0"],["腥红","#ff2400","255,0,0"],["橙红","#ff4500","255,0,0"],["朱红","#ff4d00","255,0,0"],["鲜红","#e60000","230,0,0"],["曙红","#e60039","230,57,57"],["章丹","#dd3700","221,0,0"],["茉莉花红","#cc0033","204,51,51"],["猩红","#cc0000","204,0,0"],["象牙红","#b30000","179,0,0"],["血红","#990000","153,0,0"],["深釉红","#971b00","151,0,0"],["暗红","#8b0000","139,0,0"],["栗色","#800000","128,0,0"],["釉红","#7b1800","123,0,0"],["臙脂","#700000","112,0,0"],["花卉白","#fffaf0","255,240,240"],["海贝色","#fff5ee","255,238,238"],["浅棕灰","#eee7dd","238,221,221"],["亚麻色","#faf0e6","250,230,230"],["胡粉","#fffae8","255,232,232"],["旧蕾丝色","#fdf5e6","253,230,230"],["银白色","#fffae6","255,230,230"],["铁灰色","#625b57","98,87,87"],["蝋白","#fef8de","254,222,222"],["玉米丝色","#fff8dc","255,220,220"],["古董白","#faebd7","250,215,215"],["红梍","#3a3932","58,50,50"],["蕃木瓜色","#ffefd5","255,213,213"],["芽灰","#e3dbbf","227,191,191"],["米Line truncated
|
||||
c&&(c.p===x.D_U&&colors.includes(t.w)&&(t.p=x.D_A),t.p===x.D_N&&this.isNominal(c.p)&&colors.includes(t.w)&&(t.p=x.D_A)),e+=1}return n}},{key:"isNominal",value:function(n){if(Array.isArray(n))return this.isNominal(n[0])
|
||||
var x=this.segment.POSTAG
|
||||
return n===x.D_N||n===x.A_NT||n===x.A_NX||n===x.A_NZ||n===x.A_NR||n===x.A_NS||n===x.URL}}]),n}()
|
||||
function addOrderInfo(n,x){var e={}
|
||||
return n.forEach(function(n){e[n]=x}),e}var FAMILY_NAME_1=addOrderInfo(["王","张","黄","周","徐","胡","高","林","马","于","程","傅","曾","叶","余","夏","钟","田","任","方","石","熊","白","毛","江","史","候","龙","万","段","雷","钱","汤","易","常","武","赖","文","查","赵","肖","孙","李","吴","郑","冯","陈","褚","卫","蒋","沈","韩","杨","朱","秦","尤","许","何","吕","施","桓","孔","曹","严","华","金","魏","陶","姜","戚","谢","邹","喻","柏","窦","苏","潘","葛","奚","范","彭","鲁","韦","昌","俞","袁","酆","鲍","唐","费","廉","岑","薛","贺","倪","滕","殷","罗","毕","郝","邬","卞","康","卜","顾","孟","穆","萧","尹","姚","邵","湛","汪","祁","禹","狄","贝","臧","伏","戴","宋","茅","庞","纪","舒","屈","祝","董","梁","杜","阮","闵","贾","娄","颜","郭","邱","骆","蔡","樊","凌","霍","虞","柯","昝","卢","柯","缪","宗","丁","贲","邓","郁","杭","洪","崔","龚","嵇","邢","滑","裴","陆","荣","荀","惠","甄","芮","羿","储","靳","汲","邴","糜","隗","侯","宓","蓬","郗","仲","栾","钭","历","戎","刘","詹","幸","韶","郜","黎","蓟","溥","蒲","邰","鄂","咸","卓","蔺","屠","乔","郁","胥","苍","莘","翟","谭","贡","劳","冉","郦","雍","璩","桑","桂","濮","扈","冀","浦","庄","晏","瞿","阎","慕","茹","习","宦","艾","容","慎","戈","廖","庾","衡","耿","弘","匡","阙","殳","沃","蔚","夔","隆","巩","聂","晁","敖","融","訾","辛","阚","毋","乜","鞠","丰","蒯","荆","竺","盍","单","欧"],1),FAMILY_NAME_2=addOrderInfo(["司马","上官","欧阳","夏侯","诸葛","闻人","东方","赫连","皇甫","尉迟","公羊","澹台","公冶","宗政","濮阳","淳于","单于","太叔","申屠","公孙","仲孙","轩辕","令狐","徐离","宇文","长孙","慕容","司徒","司空","万俟"],2),DOUBLE_NAME_1=addOrderInfo(["阿","建","小","晓","文","志","国","玉","丽","永","海","春","金","明","新","德","秀","红","亚","伟","雪","俊","桂","爱","美","世","正","庆","学","家","立","淑","振","云","华","光","惠","兴","天","长","艳","慧","利","宏","佳","瑞","凤","荣","秋","继","嘉","卫","燕","思","维","少","福","忠","宝","子","成","月","洪","东","一","泽","林","大","素","旭","宇","智","锦","冬","玲","雅","伯","翠","传","启","剑","安","树","良","中","梦","广","昌","元","万","清","静","友","宗","兆","丹","克","彩","绍","喜","远","朝","敏","培","胜","祖","先","菊","士","向","有","连","军","健","巧","耀","莉","英","方","和","仁","孝","梅","汉","兰","松","水","江","益","开","景","运","贵","祥","青","芳","碧","婷","龙","鹏","自","顺","双","书","生","义","跃","银","佩","雨","保","贤","仲","鸿","浩","加","定","炳","飞","锡","柏","发","超","道","怀","进","其","富","平","全","阳","吉","茂","彦","诗","洁","润","承","治","焕","如","君","增","善","希","根","应","勇","宜","守","会","凯","育","湘","凌","本","敬","博","延","乐","三","二","四","五","六","七","八","九","十"],1),DOUBLE_NAME_2=addOrderInfo(["华","平","明","英","军","林","萍","芳","玲","红","生","霞","梅","文","荣","珍","兰","娟","峰","琴","云","辉","东","龙","敏","伟","强","丽","春","杰","燕","民","君","波","国","芬","清","祥","斌","婷","飞","良","忠","新","凤","锋","成","勇","刚","玉","元","宇","海","兵","安","庆","涛","鹏","亮","青","阳","艳","松","江","莲","娜","兴","光","德","武","香","俊","秀","慧","雄","才","宏","群","琼","胜","超","彬","莉","中","山","富","花","宁","利","贵","福","发","义","蓉","喜","娥","昌","仁","志","全","宝","权","美","琳","建","金","贤","星","丹","根","和","珠","康","菊","琪","坤","泉","秋","静","佳","顺","源","珊","达","欣","如","莹","章","浩","勤","芹","容","友","芝","豪","洁","鑫","惠","洪","旺","虎","远","妮","森","妹","南","雯","奇","健","卿","虹","娇","媛","怡","铭","川","进","博","智","来","琦","学","聪","洋","乐","年","翔","然","栋","凯","颖","鸣","丰","瑞","奎","立","堂","威","雪","鸿","晶","桂","凡","娣","先","洲","毅","雅","月","旭","田","晖","方","恒","亚","泽","风","银","高","贞","九","薇"],2),SINGLE_NAME=addOrderInfo(["家","民","敏","伟","勇","军","斌","静","丽","涛","芳","杰","萍","强","俊","明","燕","磊","玲","华","平","鹏","健","波","红","丹","辉","超","艳","莉","刚","娟","峰","婷","亮","洁","颖","琳","英","慧","飞","霞","浩","凯","宇","毅","林","佳","云","莹","娜","晶","洋","文","鑫","欣","琴","宁","琼","兵","青","琦","翔","彬","锋","阳","璐","旭","蕾","剑","虹","蓉","建","倩","梅","宏","威","博","君","力","龙","晨","薇","雪","琪","Line truncated
|
||||
else{var r=s.matchName(x.w)
|
||||
if(r.length<1)t.push(x)
|
||||
else{for(var f,i=0,a=0;f=r[a];a++)i<f.c&&t.push({w:x.w.substr(i,f.c-i)}),t.push({w:f.w,p:e.A_NR}),i=f.c+f.w.length
|
||||
var o=r[r.length-1]
|
||||
o.c+o.w.length<x.w.length&&t.push({w:x.w.substr(o.c+o.w.length)})}}return t}}],[{key:"matchName",value:function(n,x){var e=0
|
||||
isNaN(x)||(e=x)
|
||||
for(var t=[];e<n.length;){var c=!1,r=n.substr(e,2)
|
||||
if(r in FAMILY_NAME_2){var f=n[0|e+2],i=n[0|e+3]
|
||||
f in DOUBLE_NAME_1&&i in DOUBLE_NAME_2?c=r+f+i:f in SINGLE_NAME&&(c=r+f+(f==i?i:""))}var a=n[0|e]
|
||||
if(!1===c&&a in FAMILY_NAME_1){var o=n[0|e+1],s=n[0|e+2]
|
||||
o in DOUBLE_NAME_1&&s in DOUBLE_NAME_2?c=a+o+s:o in SINGLE_NAME&&(c=a+o+(o==s?s:""))}!1===c?e++:(t.push({w:c,c:e}),e+=c.length)}return t}}]),s}(),DictOptimizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Optimizer$1),_createClass(n,[{key:"doOptimize",value:function(n,x){void 0===x&&(x=!1)
|
||||
for(var e=this.segment.getDict("TABLE"),t=this.segment.POSTAG,c=0,r=n.length-1;c<r;){var f=n[c],i=n[c+1],a=f.w+i.w
|
||||
if(f.p===i.p&&a in e)n.splice(c,2,{w:a,p:e[a].p}),r--
|
||||
else if(0<(f.p&t.D_A)&&i.p&t.D_U)n.splice(c,2,{w:a,p:t.D_A}),r--
|
||||
else{if(0<(f.p&t.A_M)){if(0<(i.p&t.A_M)||"%"===i.w){n.splice(c,2,{w:f.w+i.w,p:t.A_M}),r--
|
||||
continue}if(0<(i.p&t.A_Q)){n.splice(c,2,{w:f.w+i.w,p:t.D_MQ}),r--
|
||||
continue}if((l=n[c+2])&&0<(l.p&t.A_M)&&("."===i.w||"点"===i.w||"分之"===i.w)){n.splice(c,3,{w:f.w+i.w+l.w,p:t.A_M}),r-=2
|
||||
continue}}if(0<(f.p&t.D_MQ)&&"点"==f.w.substr(-1)&&i.p&t.A_M){for(var o=2,s="",u=c+o;u<r;u++){var l
|
||||
if(!(0<((l=n[u]).p&t.A_M)))break
|
||||
s+=l.w,o++}n.splice(c,o,{w:f.w+i.w+s,p:t.D_MQ}),r-=o-1}else c++}}return!0===x?n:this.doOptimize(n,!0)}}]),n}(),_EMAILCHAR="!\"#$%&'*+-/0123456789=?@ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz{|}~.".split(""),EMAILCHAR={}
|
||||
for(var i in _EMAILCHAR)EMAILCHAR[_EMAILCHAR[i]]=1
|
||||
var EmailOptimizer=function(){function o(){return _classCallCheck(this,o),_possibleConstructorReturn(this,_getPrototypeOf(o).apply(this,arguments))}return _inherits(o,Optimizer$1),_createClass(o,[{key:"doOptimize",value:function(n){for(var x=this.segment.POSTAG,e=0,t=n.length-1,c=!1,r=!1;e<t;){var f,i=!!((f=n[e]).p===x.A_NX||f.p===x.A_M&&f.w.charCodeAt(0)<128)
|
||||
if(!1===c&&i)c=e,e++
|
||||
else if(!1!==r||"@"!==f.w)if(!1===r||"@"==n[e-1].w||!1!==i||f.w in EMAILCHAR)!1!==c&&(i||f.w in EMAILCHAR)||(r=c=!1),e++
|
||||
else{var a=n.slice(c,e)
|
||||
n.splice(c,a.length,{w:o.toEmailAddress(a),p:x.URL}),e=c+1,t-=a.length-1,r=c=!1}else r=!0,e++}if(c&&r&&n[t]&&(i=!!((f=n[t]).p===x.A_NX||f.p===x.A_M&&f.w in EMAILCHAR))){a=n.slice(c,n.length)
|
||||
n.splice(c,a.length,{w:o.toEmailAddress(a),p:x.URL})}return n}}],[{key:"toEmailAddress",value:function(n){for(var x,e=n[0].w,t=1;x=n[t];t++)e+=x.w
|
||||
return e}}]),o}(),_STOPWORD=" ,.;+-|/\\'\":?<>[]{}=!@#$%^&*()~`。，、＇：∶；?‘’“”〝〞ˆˇ﹕︰﹔﹖﹑·¨….¸;！´？！～—ˉ｜‖＂〃｀@﹫¡¿﹏﹋﹌︴々﹟#﹩$﹠&﹪%*﹡﹢﹦﹤‐￣¯―﹨ˆ˜﹍﹎+=<＿_-ˇ~﹉﹊（）〈〉‹›﹛﹜『』〖〗［］《》〔〕{}「」【】︵︷︿︹︽_﹁﹃︻︶︸﹀︺︾ˉ﹂﹄︼＋－×÷﹢﹣±／＝≈≡≠∧∨∑∏∪∩∈⊙⌒⊥∥∠∽≌＜＞≤≥≮≯∧∨√﹙﹚[]﹛﹜∫∮∝∞⊙∏┌┬┐┏┳┓╒╤╕─│├┼┤┣╋┫╞╪╡━┃└┴┘┗┻┛╘╧╛┄┆┅┇╭─╮┏━┓╔╦╗┈┊│╳│┃┃╠╬╣┉┋╰─╯┗━┛╚╩╝╲╱┞┟┠┡┢┦┧┨┩┪╉╊┭┮┯┰┱┲┵┶┷┸╇╈┹┺┽┾┿╀╁╂╃╄╅╆○◇□△▽☆●◆■▲▼★♠♥♦♣☼☺◘♀√☻◙♂×▁▂▃▄▅▆▇█⊙◎۞卍卐╱╲▁▏↖↗↑←↔◤◥╲╱▔▕↙↘↓→↕◣◢∷▒░℡™"
|
||||
_STOPWORD=_STOPWORD.split("")
|
||||
var STOPWORD={},STOPWORD2={}
|
||||
for(var i$1 in _STOPWORD)if(""!==_STOPWORD[i$1]){var len=_STOPWORD[i$1].length
|
||||
STOPWORD2[STOPWORD[_STOPWORD[i$1]]=len]||(STOPWORD2[len]={}),STOPWORD2[len][_STOPWORD[i$1]]=len}var PunctuationTokenizer=function(){function s(){return _classCallCheck(this,s),_possibleConstructorReturn(this,_getPrototypeOf(s).apply(this,arguments))}return _inherits(s,Tokenizer$1),_createClass(s,[{key:"split",value:function(n){for(var x,e=this.segment.POSTAG,t=[],c=0;x=n[c];c++)if(0<x.p)t.push(x)
|
||||
else{var r=s.matchStopword(x.w)
|
||||
if(r.length<1)t.push(x)
|
||||
else{for(var f,i=0,a=0;f=r[a];a++)i<f.c&&t.push({w:x.w.substr(i,f.c-i)})," "!=f.w&&t.push({w:f.w,p:e.D_W}),i=f.c+f.w.length
|
||||
var o=r[r.length-1]
|
||||
o.c+o.w.length<x.w.length&&t.push({w:x.w.substr(o.c+o.w.length)})}}return t}}],[{key:"matchStopword",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],t=!1;x<n.length;){for(var c in STOPWORD2){var r=n.substr(x,c)
|
||||
if(r in STOPWORD2[c]){e.push({w:r,c:x}),t=!0
|
||||
break}}x+=!1===t?1:r.length,t=!1}return e}}]),s}(),PROTOTAL=["http://","https://","ftp://","news://","telnet://"],MIN_PROTOTAL_LEN=100
|
||||
for(var i$2 in PROTOTAL)PROTOTAL[i$2].length<MIN_PROTOTAL_LEN&&(MIN_PROTOTAL_LEN=PROTOTAL[i$2].length)
|
||||
var _URLCHAR=["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","0","1","2","3","4","5","6","7","8","9","!","#","$","%","&","‘","(",")","*","+",",","-",".","/",":",";","=","?","@","[","\\","]","^","_","`","|","~"],URLCHAR={}
|
||||
for(var i$2 in _URLCHAR)URLCHAR[_URLCHAR[i$2]]=1
|
||||
var URLTokenizer=function(){function s(){return _classCallCheck(this,s),_possibleConstructorReturn(this,_getPrototypeOf(s).apply(this,arguments))}return _inherits(s,Tokenizer$1),_createClass(s,[{key:"split",value:function(n){for(var x,e=this.segment.POSTAG,t=[],c=0;x=n[c];c++)if(0<x.p)t.push(x)
|
||||
else{var r=s.matchURL(x.w)
|
||||
if(r.length<1)t.push(x)
|
||||
else{for(var f,i=0,a=0;f=r[a];a++)i<f.c&&t.push({w:x.w.substr(i,f.c-i)}),t.push({w:f.w,p:e.URL}),i=f.c+f.w.length
|
||||
var o=r[r.length-1]
|
||||
o.c+o.w.length<x.w.length&&t.push({w:x.w.substr(o.c+o.w.length)})}}return t}}],[{key:"matchURL",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],t=!1;x<n.length;){if(!1===t&&x<n.length-MIN_PROTOTAL_LEN){for(var c,r=0;c=PROTOTAL[r];r++)if(n.substr(x,c.length)===c){t=x,x+=c.length-1
|
||||
break}}else!1===t||n[0|x]in URLCHAR||(e.push({w:n.substr(t,x-t),c:t}),t=!1)
|
||||
x++}return!1!==t&&e.push({w:n.substr(t,x-t),c:t}),e}}]),s}(),ChsNameOptimizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Optimizer$1),_createClass(n,[{key:"doOptimize",value:function(n){for(var x=this.segment.POSTAG,e=0;e<n.length;){var t=n[e],c=n[e+1]
|
||||
if(c){if(c&&("小"===t.w||"老"===t.w)&&(c.w in FAMILY_NAME_1||c.w in FAMILY_NAME_2)){n.splice(e,2,{w:t.w+c.w,p:x.A_NR}),e++
|
||||
continue}if((t.w in FAMILY_NAME_1||t.w in FAMILY_NAME_2)&&0<(c.p&x.A_NR)&&c.w.length<=2){n.splice(e,2,{w:t.w+c.w,p:x.A_NR}),e++
|
||||
continue}if((!t.p||!c.p)&&(t.w in SINGLE_NAME&&t.w===c.w||t.w in DOUBLE_NAME_1&&c.w in DOUBLE_NAME_2)){n.splice(e,2,{w:t.w+c.w,p:x.A_NR})
|
||||
var r=n[e-1]
|
||||
r&&(r.w in FAMILY_NAME_1||r.w in FAMILY_NAME_2)?n.splice(e-1,2,{w:r.w+t.w+c.w,p:x.A_NR}):e++
|
||||
continue}!(t.w in FAMILY_NAME_1||t.w in FAMILY_NAME_2)||t.p&&c.p||n.splice(e,2,{w:t.w+c.w,p:x.A_NR})}e++}for(e=0;e<n.length;){var f=n[e],i=n[e+1]
|
||||
i&&(f.w in FAMILY_NAME_1||f.w in FAMILY_NAME_2)&&i.w in SINGLE_NAME?(n.splice(e,2,{w:f.w+i.w,p:x.A_NR}),e++):e++}return n}}]),n}(),DATETIME_WORDS=["世纪","年","年份","年度","月","月份","月度","日","号","时","点","点钟","分","分钟","秒","毫秒"],DATETIME={}
|
||||
for(var i$3 in DATETIME_WORDS)DATETIME[DATETIME_WORDS[i$3]]=DATETIME_WORDS[i$3].length
|
||||
var DatetimeOptimizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Optimizer$1),_createClass(n,[{key:"doOptimize",value:function(n){this.segment.getDict("TABLE")
|
||||
for(var x=this.segment.POSTAG,e=0,t=n.length-1;e<t;){var c=n[e+1]
|
||||
if(0<((i=n[e]).p&x.A_M)&&c.w in DATETIME){for(var r=i.w+c.w,f=2;;){var i
|
||||
c=n[e+f+1]
|
||||
if(!((i=n[e+f])&&c&&0<(i.p&x.A_M)&&c.w in DATETIME))break
|
||||
f+=2,r+=i.w+c.w}n.splice(e,f,{w:r,p:x.D_T}),t-=f-1}else e++}return n}}]),n}(),_DATETIME=["世纪","年","年份","年度","月","月份","月度","日","号","时","点","点钟","分","分钟","秒","毫秒"],DATETIME$1={}
|
||||
for(var i$4 in _DATETIME)DATETIME$1[_DATETIME[i$4]]=_DATETIME[i$4].length
|
||||
var DictTokenizer=function(){function b(){return _classCallCheck(this,b),_possibleConstructorReturn(this,_getPrototypeOf(b).apply(this,arguments))}return _inherits(b,Tokenizer$1),_createClass(b,[{key:"split",value:function(n){for(var x,e=this.segment.getDict("TABLE"),t=[],c=0;x=n[c];c++)if(0<x.p)t.push(x)
|
||||
else{var r=this.matchWord(x.w,0,n[c-1])
|
||||
if(r.length<1)t.push(x)
|
||||
else{for(var f,i=0,a=0;f=r[a];a++)i<f.c&&t.push({w:x.w.substr(i,f.c-i)}),t.push({w:f.w,p:e[f.w].p}),i=f.c+f.w.length
|
||||
var o=r[r.length-1]
|
||||
o.c+o.w.length<x.w.length&&t.push({w:x.w.substr(o.c+o.w.length)})}}return t}},{key:"matchWord",value:function(n,x,e){isNaN(x)&&(x=0)
|
||||
for(var t=[],c=this.segment.getDict("TABLE2");x<n.length;){for(var r in c){var f=n.substr(x,r)
|
||||
f in c[r]&&t.push({w:f,c:x,f:c[r][f].f})}x++}return this.filterWord(t,e,n)}},{key:"filterWord",value:function(n,x,e){for(var t,c=this.segment.POSTAG,r=this.segment.getDict("TABLE"),f=b.getPosInfo(n,e),i=b.getChunks(f,0,e),a=[],o=0;t=i[o];o++){var s=e.length/t.length,u=!(a[o]={x:t.length,a:0,b:0,c:0,d:0})
|
||||
if(x)var l={w:x.w,p:x.p,f:x.f}
|
||||
else l=!1
|
||||
for(var p,_=0;p=t[_];_++){if(p.w in r){if(p.p=r[p.w].p,a[o].a+=p.f,l){0<(l.p&c.A_M)&&(0<(r[p.w].p&c.A_Q)||p.w in DATETIME$1)&&a[o].d++,0<(p.p&c.D_V)&&(u=!0,0<(l.p&c.D_A)&&a[o].d++),(0<(l.p&c.A_NS)||l.p&c.A_NT||0<(l.p&c.D_A))&&(0<(p.p&c.D_N)||0<(p.p&c.A_NR)||0<(p.p&c.A_NS)||0<(p.p&c.A_NZ)||0<(p.p&c.A_NT))&&a[o].d++,0<(l.p&c.D_F)&&(p.p&0<c.A_M||p.p&0<c.D_MQ)&&a[o].d++,(l.w in FAMILY_NAME_1||l.w in FAMILY_NAME_2)&&(0<(p.p&c.D_N)||0<(p.p&c.A_NZ))&&a[o].d++
|
||||
var d=t[_+1]
|
||||
d&&(d.w in r&&(d.p=r[d.w].p),0<(p.p&c.D_C)&&l.p===d.p&&a[o].d++,"的"!==p.w&&"之"!==p.w||!(0<(d.p&c.D_N)||0<(d.p&c.A_NR)||0<(d.p&c.A_NS)||0<(d.p&c.A_NZ)||0<(d.p&c.A_NT))||(a[o].d+=1.5))}}else a[o].c++
|
||||
a[o].b+=Math.pow(s-p.w.length,2),l=t[_]}!1===u&&(a[o].d-=.5),a[o].a=a[o].a/t.length,a[o].b=a[o].b/t.length}var h,A=i[b.getTops(a)]
|
||||
for(o=0;h=A[o];o++)h.w in r||A.splice(o--,1)
|
||||
return A}}],[{key:"getPosInfo",value:function(n,x){for(var e,t={},c=0;e=n[c];c++)t[e.c]||(t[e.c]=[]),t[e.c].push(e)
|
||||
for(c=0;c<x.length;c++)t[c]||(t[c]=[{w:x[0|c],c:c,f:0}])
|
||||
return t}},{key:"getChunks",value:function(n,x){for(var e=n[x]||[],t=[],c=0;c<e.length;c++){var r=e[c],f=r.c+r.w.length
|
||||
if(n[f])for(var i=b.getChunks(n,f),a=0;a<i.length;a++)t.push([r].concat(i[a]))
|
||||
else t.push([r])}return t}},{key:"getTops",value:function(n){for(var x={x:n[0].x,a:n[0].a,b:n[0].b,c:n[0].c,d:n[0].d},e=1;t=n[e];e++)x.a<t.a&&(x.a=t.a),t.b<x.b&&(x.b=t.b),x.c<t.c&&(x.c=t.c),t.d<x.d&&(x.d=t.d),x.x<t.x&&(x.x=t.x)
|
||||
var t,c=[]
|
||||
for(e=0;t=n[e];e++)c[e]=0,c[e]+=1.5*(x.x-t.x),x.a<=t.a&&(c[e]+=1),t.b<=x.b&&(c[e]+=1),c[e]+=x.c-t.c,c[e]+=1*(t.d<0?x.d+t.d:t.d-x.d)
|
||||
var r=0,f=c[0]
|
||||
for(var e in c){var i=c[e]
|
||||
if(f<i)r=e,f=i
|
||||
else if(i===f){var a=0,o=0
|
||||
n[e].c<n[r].c?a++:o++,n[r].a<n[e].a?a++:o++,n[e].x<n[r].x?a++:o++,o<a&&(r=e,f=i)}}return r}}]),b}(),ForeignTokenizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Tokenizer$1),_createClass(n,[{key:"split",value:function(n){for(var x,e=[],t=0;x=n[t];t++)x.p?e.push(x):e=e.concat(this.splitForeign(x.w))
|
||||
return e}},{key:"splitForeign",value:function(n){var x=this.segment.POSTAG,e=[],t=0,c=0
|
||||
65296<=(f=n.charCodeAt(0))&&f<=65370&&(f-=65248),c=48<=f&&f<=57?x.A_M:65<=f&&f<=90||97<=f&&f<=122?x.A_NX:x.UNK
|
||||
for(var r=1;r<n.length;r++){var f
|
||||
if(65296<=(f=n.charCodeAt(r))&&f<=65370&&(f-=65248),48<=f&&f<=57){if(c!==x.A_M){var i={w:n.substr(t,r-t)}
|
||||
c!==x.UNK&&(i.p=c),e.push(i),t=r}c=x.A_M}else if(65<=f&&f<=90||97<=f&&f<=122){if(c!==x.A_NX){i={w:n.substr(t,r-t)}
|
||||
c!==x.UNK&&(i.p=c),e.push(i),t=r}c=x.A_NX}else c!==x.UNK&&(e.push({w:n.substr(t,r-t),p:[c]}),t=r),c=x.UNK}i={w:n.substr(t,r-t)}
|
||||
return c!==x.UNK&&(i.p=c),e.push(i),e}}]),n}(),SingleTokenizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Tokenizer$1),_createClass(n,[{key:"split",value:function(n){for(var x,e=[],t=0;x=n[t];t++)x.p?e.push(x):e=e.concat(this.splitSingle(x.w))
|
||||
return e}},{key:"splitSingle",value:function(n,x){var e=this.segment.POSTAG
|
||||
isNaN(x)&&(x=0)
|
||||
for(var t=[];x<n.length;)t.push({w:n[0|x],p:e.UNK}),x++
|
||||
return t}}]),n}(),WildcardTokenizer=function(){function n(){return _classCallCheck(this,n),_possibleConstructorReturn(this,_getPrototypeOf(n).apply(this,arguments))}return _inherits(n,Tokenizer$1),_createClass(n,[{key:"split",value:function(n){for(var x,e=this.segment.getDict("WILDCARD"),t=[],c=0;x=n[c];c++)if(0<x.p)t.push(x)
|
||||
else{var r=this.matchWord(x.w)
|
||||
if(r.length<1)t.push(x)
|
||||
else{for(var f,i=0,a=0;f=r[a];a++)i<f.c&&t.push({w:x.w.substr(i,f.c-i)}),t.push({w:f.w,p:e[f.w.toLowerCase()].p}),i=f.c+f.w.length
|
||||
var o=r[r.length-1]
|
||||
o.c+o.w.length<x.w.length&&t.push({w:x.w.substr(o.c+o.w.length)})}}return t}},{key:"matchWord",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],t=this.segment.getDict("WILDCARD2"),c=n.toLowerCase();x<n.length;){var r=!1
|
||||
for(var f in t)c.substr(x,f)in t[f]&&(r={w:n.substr(x,f),c:x})
|
||||
!1!==r?(e.push(r),x+=r.w.length):x++}return e}}]),n}(),modules=[URLTokenizer,WildcardTokenizer,PunctuationTokenizer,ForeignTokenizer,DictTokenizer,ChsNameTokenizer,EmailOptimizer,ChsNameOptimizer,DictOptimizer,DatetimeOptimizer,AdjectiveOptimizer],pangu="\ufeff爱|0x1000|323\n安|0x1000|1651\n凹|0x40000000|10293\n熬|0x1000|36595\n八|0x400000|355\n扒|0x1000|16883\n拔|0x1000|3784\n把|0x40000|22\n摆|0x1000|951\n拜|0x1000|3985\n搬|0x1000|2305\n办|0x1000|261\n扮|0x1000|10508\n拌|0x1000|11672\n绊|0x1000|12776\n帮|0x1000|889\n绑|0x1000|7942\n傍|0x1000|12168\n包|0x1000|884\n饱|0x40000000|4431\n保|0x1000|981\n报|0x100000|438\n背|0x1000|3930\n倍|0x8000000|973\n本|0x10000|115\n绷|0x100000|14002\n逼|0x1000|2473\n比|0x40000|98\n必|0x8000000|1335\n毙|0x1000|8263\n避|0x1000|4562\n编|0x1000|2131\n贬|0x1000|9759\n拚|0x0400|0\n变|0x1000|264\n遍|0x20000|1624\n辨|0x1000|7007\n别|0x8000000|40734\n瘪|0x1000|15221\n并|0x10000000|45\n病|0x100000|383\n拨|0x1000|3172\n剥|0x1000|6852\n播|0x1000|5785\n驳|0x100000|9902\n搏|0x1000|11464\n补|0x1000|1726\n捕|0x1000|5370\n不|0x8000000|5\n步|0x20000|397\n擦|0x1000|2620\n才|0x8000000|82\n采|0x1000|2522\n操|0x1000|4061\n层|0x20000|640\n插|0x1000|2659\n查|0x1000|1652\n岔|0x100000|12396\n差|0x40000000|1299\n产|0x1000|1945\n长|0x40000000|1368\n常|0x8000000|648\n场|0x20000|6159\n抄|0x1000|5307\n超|0x1000|1760\n朝|0x100000|993\n吵|0x1000|5029\n炒|0x1000|5094\n扯|0x1000|3761\n闯|0x1000|2295\n称|0x1000|448\n乘|0x1000|1581\n吃|0x1000|184\n持|0x1000|1894\n斥|0x1000|0\n冲|0x1000|1093\n丑|0x40000000|5722\n出|0x1000|49\n除|0x40000|663\n穿|0x1000|774\n吹|0x1000|1510\n纯|0x40000000|3093\n蠢|0x1000|0\n此|0x10000|119\n次|0x20000|57\n从|0x40000|27\n促|0x1000|3235\n存|0x1000|2089\n寸|0x20000|5741\n搓|0x1000|7703\n打|0x1000|17890\n大|0x40000000|19\n代|0x1000|503\n带|0x1000|178\n待|0x1000|4627\n贷|0x1000|6275\n单|0x8000000|816\n但|0x10000000|37\n淡|0x40000000|5053\n当|0x40000|591\n挡|0x1000|5085\n倒|0x1000|1212\n到|0x1000|18\n得|0x2000|4434\n的|0x2000|1\n登|0x1000|1810\n等|0x2000|24\n低|0x40000000|254\n敌|0x1000|2139\n地|0x2000|1382\n第|0x100000|0\n点|0x20000|116\n电|0x100000|462\n钓|0x100000|11268\n钉|0x1000|9869\n顶|0x1000|1298\n订|0x1000|4503\n定|0x1000|694\n动|0x1000|772\n冻|0x1000|5210\n都|0x8000000|1244\n斗|0x1000|4900\n毒|0x100000|2969\n独|0x8000000|2378\n堵|0x1000|4272\n短|0x40000000|788\n段|0x20000|301\n堆|0x20000|1577\n对|0x40000|14\n兑|0x100000|10085\n顿|0x20000|1803\n夺|0x1000|2841\n朵|0x20000|3952\n恶|0x40000000|2459\n而|0x10000000|28\n二|0x40400000|88\n发|0x1000|3954\n罚|0x1000|3677\n翻|0x1000|1476\n凡|0x8000000|1749\n反|0x1000|315\n犯|0x1000|1800\n方|0x20000|372\n防|0x100000|1637\n仿|0x1000|9692\n放|0x1000|425\n非|0x8000000|185\n肥|0x100000|3240\n废|0x1000|3560\n费|0x1000|1015\n分|0x20000|168\n份|0x20000|490\n封|0x20000|1050\n否|0x1000|6368\n扶|0x1000|3217\n拂|0x1000|9728\n付|0x1000|3646\n负|0x1000|1282\n附|0x1000|1327\n赴|0x100000|1530\n副|0x20000000|122\n傅|0x0400|7535\n富|0x40000000|1725\n赋|0x100000|7857\n缚|0x1000|10335\n该|0x10000|107\n改|0x1000|665\n盖|0x1000|1401\n干|0x1000|8006\n赶|0x1000|1951\n敢|0x1000|805\n刚|0x8000000|542\n高|0x40000000|76\n搞|0x1000|339\n告|0x1000|2472\n割|0x1000|4622\n个|0x20000|8\n给|0x40000|54\n跟|0x40000|434\n更|0x8000000|56\n攻|0x1000|2631\n供|0x1000|1332\n共|0x8000000|358\n够|0x1000|1816\n估|0x1000|15021\n古|0x40000000|1283\n故|0x10000000|1574\n关|0x1000|903\n管|0x1000|368\n光|0x100000|558\n广|0x40000000|1429\n归|0x1000|1388\n滚|0x1000|3632\n过|0x2000|2722\n含|0x1000|1541\n喊|0x1000|1617\n好|0x40000000|722\n耗|0x1000|6572\n喝|0x1000|2636\n合|0x1000|4244\n何|0x10000|1081\n和|0x10000000|373\n黑|0x40000000|628\n狠|0x8000000|5760\n恨|0x1000|2846\n烘|0x1000|17075\n红|0x40000000|537\n后|0x2000000|43\n乎|0x0200|0\n划|0x1000|1961\n坏|0x40000000|1379\n还|0x8000000|550\n换|0x1000|833\n黄|0x40000000|865\n灰|0x100000|2751\n挥|0x1000|3379\n会|0x1000|35\n或|0x10000000|81\n及|0x10000000|780\n即|0x1000|172\n急|0x40000000|1074\n几|0x400000|2502\n己|0x100000|3585\n挤|0x1000|2218\n计|0x1000|1755\n继|0x8000000|0\n加|0x1000|543\n件|0x20000|166\n将|0x8000000|1460\n讲|0x1000|215\n交|0x1000|738\n叫|0x1000|309\n较|0x1000|272\n她|0x10000|55\n仅|0x1000|538\n紧|0x40000000|1415\n尽|0x1000|4931\n近|0x40000000|192\n竟|0x8000000|786\n九|0x400000|533\n久|0x40000000|1260\n旧|0x40000000|681\n就|0x8000000|13\n举|0x1000|1466\n句|0x20000|367\n具|0x1000|1610\n锯|0x100000|12388\n捐|0x100000|5574\n卷|0x20000|3230\n看|0x1000|4382\n考|0x1000|2255\n烤|0x1000|6817\n靠|0x1000|354\n可|0x1000|61\n肯|0x1000|2490\n空|0x40000000|2292\n苦|0x40000000|1231\n夸|0x1000|7755\n块|0x20000|362\n快|0x40000000|210\n困|0x1000|4885\n拉|0x1000|843\n啦|0x0200|1414\n来|0x1000|23\n兰|0x100000|5326\n蓝|0x40000000|3073\n老|0x40000000|132\n擂|0x1000|14715\n冷|0x400000Line truncated
|
||||
function useDefault(n){return n.use(modules),n.loadDict(dicts),n.loadSynonymDict(synonyms),n.loadStopwordDict(stopwords),n}exports.CHS_NAMES=CHS_NAMES,exports.ChsNameOptimizer=ChsNameOptimizer,exports.ChsNameTokenizer=ChsNameTokenizer,exports.DatetimeOptimizer=DatetimeOptimizer,exports.DictOptimizer=DictOptimizer,exports.DictTokenizer=DictTokenizer,exports.EmailOptimizer=EmailOptimizer,exports.ForeignTokenizer=ForeignTokenizer,exports.Module=Module,exports.Optimizer=Optimizer$1,exports.POSTAG=POSTAG,exports.PunctuationTokenizer=PunctuationTokenizer,exports.Segment=Segment,exports.SingleTokenizer=SingleTokenizer,exports.Tokenizer=Tokenizer$1,exports.URLTokenizer=URLTokenizer,exports.WildcardTokenizer=WildcardTokenizer,exports.cnPOSTag=cnPOSTag,exports.dicts=dicts,exports.enPOSTag=enPOSTag,exports.getPOSTagTranslator=getPOSTagTranslator,exports.modules=modules,exports.names=names,exports.pangu=pangu,exports.panguExtend1=panguExtend1,exports.panguExtend2=panguExtend2,exports.stopword=stopword,exports.stopwords=stopwords,exports.synonym=synonym,exports.synonyms=synonyms,exports.useDefault=useDefault,exports.wildcard=wildcard
|
||||
//# sourceMappingURL=segmentit.min.js.map
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"segmentit.min.js","sources":["../../src/POSTAG.js","../../src/Tokenizer.js","../../src/Optimizer.js","../../src/Segment.js","../../src/module/BaseModule.js","../../src/module/COLORS.js","../../src/module/AdjectiveOptimizer.js","../../src/module/CHS_NAMES.js","../../src/module/ChsNameTokenizer.js","../../src/module/DictOptimizer.js","../../src/module/EmailOptimizer.js","../../src/module/PunctuationTokenizer.js","../../src/module/URLTokenizer.js","../../src/module/ChsNameOptimizer.js","../../src/module/DatetimeOptimizer.js","../../src/module/DictTokenizer.js","../../src/module/ForeignTokenizer.js","../../src/module/SingleTokenizer.js","../../src/module/WildcardTokenizer.js","../../src/module/index.js","../../src/knowledge/index.js","../../src/index.js"],"sourcesContent":["// @flow\n\n/**\n * 单词类型\n */\n\nconst POSTAG = {\n D_A: 0x40000000, // 形容词 形语素\n D_B: 0x20000000, // 区别词 区别语素\n D_C: 0x10000000, // 连词 连语素\n D_D: 0x08000000, // 副词 副语素\n D_E: 0x04000000, // 叹词 叹语素\n D_F: 0x02000000, // 方位词 方位语素\n D_I: 0x01000000, // 成语\n D_L: 0x00800000, // 习语\n A_M: 0x00400000, // 数词 数语素\n D_MQ: 0x00200000, // 数量词\n D_N: 0x00100000, // 名词 名语素\n D_O: 0x00080000, // 拟声词\n D_P: 0x00040000, // 介词\n A_Q: 0x00020000, // 量词 量语素\n D_R: 0x00010000, // 代词 代语素\n D_S: 0x00008000, // 处所词\n D_T: 0x00004000, // 时间词\n D_U: 0x00002000, // 助词 助语素\n D_V: 0x00001000, // 动词 动语素\n D_W: 0x00000800, // 标点符号\n D_X: 0x00000400, // 非语素字\n D_Y: 0x00000200, // 语气词 语气语素\n D_Z: 0x00000100, // 状态词\n A_NR: 0x00000080, // 人名\n A_NS: 0x00000040, // 地名\n A_NT: 0x00000020, // 机构团体\n A_NX: 0x00000010, // 外文字符\n A_NZ: 0x00000008, // 其他专名\n D_ZH: 0x00000004, // 前接成分\n D_K: 0x00000002, // 后接成分\n UNK: 0x00000000, // 未知词性\n URL: 0x00000001, // 网址、邮箱地址\n};\n\nconst CN_POS_NAMES = {\n D_A: '形容词 形语素',\n D_B: '区别词 区别语素',\n D_C: '连词 连语素',\n D_D: '副词 副语素',\n D_E: '叹词 叹语素',\n D_F: '方位词 方位语素',\n D_I: '成语',\n D_L: '习语',\n A_M: '数词 数语素',\n D_MQ: '数量词',\n D_N: '名词 名语素',\n D_O: '拟声词',\n D_P: '介词',\n A_Q: '量词 量语素',\n D_R: '代词 代语素',\n D_S: '处所词',\n D_T: '时间词',\n D_U: '助词 助语素',\n D_V: '动词 动语素',\n D_W: '标点符号',\n D_X: '非语素字',\n D_Y: '语气词 语气语素',\n D_Z: '状态词',\n A_NR: '人名',\n A_NS: '地名',\n A_NT: '机构团体',\n A_NX: '外文字符',\n A_NZ: '其他专名',\n D_ZH: '前接成分',\n D_K: '后接成分',\n UNK: '未知',\n URL: '网址 邮箱地址',\n};\n\nconst EN_POS_NAMES = {\n D_A: 'a',\n D_B: 'b',\n D_C: 'c',\n D_D: 'd',\n D_E: 'e',\n D_F: 'f',\n D_I: 'i',\n D_L: 'l',\n A_M: 'm',\n D_MQ: 'mq',\n D_N: 'n',\n D_O: 'o',\n D_P: 'p',\n A_Q: 'q',\n D_R: 'r',\n D_S: 's',\n D_T: 't',\n D_U: 'u',\n D_V: 'v',\n D_W: 'w',\n D_X: 'x',\n D_Y: 'y',\n D_Z: 'z',\n A_NR: 'nr',\n A_NS: 'ns',\n A_NT: 'nt',\n A_NX: 'nx',\n A_NZ: 'nz',\n D_ZH: 'h',\n D_K: 'k',\n UNK: 'un',\n URL: 'uri',\n};\n\nexport function getPOSTagTranslator(POSTagDict: Object, I18NDict: Object): string {\n return (posTagNumber: number): string => {\n if (isNaN(posTagNumber)) {\n return I18NDict[posTagNumber] || I18NDict.UNK;\n }\n const result = [];\n for (const key in POSTagDict) {\n if ((posTagNumber & POSTagDict[key]) > 0) {\n result.push(I18NDict[key]);\n }\n }\n if (result.length < 1) {\n return I18NDict.UNK;\n }\n return result.toString();\n };\n}\n\nexport const cnPOSTag = getPOSTagTranslator(POSTAG, CN_POS_NAMES);\nexport const enPOSTag = getPOSTagTranslator(POSTAG, EN_POS_NAMES);\n\nexport default POSTAG;\n","// @flow\nimport type { SegmentToken } from './module/type';\n\nexport default class Tokenizer {\n /**\n * 分词模块管理器\n *\n * @param {Segment} 分词接口\n */\n constructor(segment) {\n this.segment = segment;\n }\n\n /**\n * 对一段文本进行分词\n *\n * @param {string} text 文本\n * @param {array} modules 分词模块数组\n * @return {array}\n */\n split(text, modules): SegmentToken[] {\n if (modules.length < 1) {\n throw Error('No tokenizer module!');\n }\n // 按顺序分别调用各个module来进行分词 ： 各个module仅对没有识别类型的单词进行分词\n let result = [{ w: text }];\n modules.forEach(module => {\n result = module.split(result);\n });\n return result;\n }\n}\n","// @flow\nimport type { SegmentToken } from './module/type';\n\nexport default class Optimizer {\n /**\n * 优化模块管理器\n *\n * @param {Segment} 分词接口\n */\n constrLine truncated
|
||||
+2323
File diff suppressed because it is too large.
Load diff
+1
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"segmentit.js","sources":["../../src/POSTAG.js","../../src/Tokenizer.js","../../src/Optimizer.js","../../src/Segment.js","../../src/module/BaseModule.js","../../src/module/COLORS.js","../../src/module/AdjectiveOptimizer.js","../../src/module/CHS_NAMES.js","../../src/module/ChsNameTokenizer.js","../../src/module/DictOptimizer.js","../../src/module/EmailOptimizer.js","../../src/module/PunctuationTokenizer.js","../../src/module/URLTokenizer.js","../../src/module/ChsNameOptimizer.js","../../src/module/DatetimeOptimizer.js","../../src/module/DictTokenizer.js","../../src/module/ForeignTokenizer.js","../../src/module/SingleTokenizer.js","../../src/module/WildcardTokenizer.js","../../src/module/index.js","../../src/knowledge/index.js","../../src/index.js"],"sourcesContent":["// @flow\n\n/**\n * 单词类型\n */\n\nconst POSTAG = {\n D_A: 0x40000000, // 形容词 形语素\n D_B: 0x20000000, // 区别词 区别语素\n D_C: 0x10000000, // 连词 连语素\n D_D: 0x08000000, // 副词 副语素\n D_E: 0x04000000, // 叹词 叹语素\n D_F: 0x02000000, // 方位词 方位语素\n D_I: 0x01000000, // 成语\n D_L: 0x00800000, // 习语\n A_M: 0x00400000, // 数词 数语素\n D_MQ: 0x00200000, // 数量词\n D_N: 0x00100000, // 名词 名语素\n D_O: 0x00080000, // 拟声词\n D_P: 0x00040000, // 介词\n A_Q: 0x00020000, // 量词 量语素\n D_R: 0x00010000, // 代词 代语素\n D_S: 0x00008000, // 处所词\n D_T: 0x00004000, // 时间词\n D_U: 0x00002000, // 助词 助语素\n D_V: 0x00001000, // 动词 动语素\n D_W: 0x00000800, // 标点符号\n D_X: 0x00000400, // 非语素字\n D_Y: 0x00000200, // 语气词 语气语素\n D_Z: 0x00000100, // 状态词\n A_NR: 0x00000080, // 人名\n A_NS: 0x00000040, // 地名\n A_NT: 0x00000020, // 机构团体\n A_NX: 0x00000010, // 外文字符\n A_NZ: 0x00000008, // 其他专名\n D_ZH: 0x00000004, // 前接成分\n D_K: 0x00000002, // 后接成分\n UNK: 0x00000000, // 未知词性\n URL: 0x00000001, // 网址、邮箱地址\n};\n\nconst CN_POS_NAMES = {\n D_A: '形容词 形语素',\n D_B: '区别词 区别语素',\n D_C: '连词 连语素',\n D_D: '副词 副语素',\n D_E: '叹词 叹语素',\n D_F: '方位词 方位语素',\n D_I: '成语',\n D_L: '习语',\n A_M: '数词 数语素',\n D_MQ: '数量词',\n D_N: '名词 名语素',\n D_O: '拟声词',\n D_P: '介词',\n A_Q: '量词 量语素',\n D_R: '代词 代语素',\n D_S: '处所词',\n D_T: '时间词',\n D_U: '助词 助语素',\n D_V: '动词 动语素',\n D_W: '标点符号',\n D_X: '非语素字',\n D_Y: '语气词 语气语素',\n D_Z: '状态词',\n A_NR: '人名',\n A_NS: '地名',\n A_NT: '机构团体',\n A_NX: '外文字符',\n A_NZ: '其他专名',\n D_ZH: '前接成分',\n D_K: '后接成分',\n UNK: '未知',\n URL: '网址 邮箱地址',\n};\n\nconst EN_POS_NAMES = {\n D_A: 'a',\n D_B: 'b',\n D_C: 'c',\n D_D: 'd',\n D_E: 'e',\n D_F: 'f',\n D_I: 'i',\n D_L: 'l',\n A_M: 'm',\n D_MQ: 'mq',\n D_N: 'n',\n D_O: 'o',\n D_P: 'p',\n A_Q: 'q',\n D_R: 'r',\n D_S: 's',\n D_T: 't',\n D_U: 'u',\n D_V: 'v',\n D_W: 'w',\n D_X: 'x',\n D_Y: 'y',\n D_Z: 'z',\n A_NR: 'nr',\n A_NS: 'ns',\n A_NT: 'nt',\n A_NX: 'nx',\n A_NZ: 'nz',\n D_ZH: 'h',\n D_K: 'k',\n UNK: 'un',\n URL: 'uri',\n};\n\nexport function getPOSTagTranslator(POSTagDict: Object, I18NDict: Object): string {\n return (posTagNumber: number): string => {\n if (isNaN(posTagNumber)) {\n return I18NDict[posTagNumber] || I18NDict.UNK;\n }\n const result = [];\n for (const key in POSTagDict) {\n if ((posTagNumber & POSTagDict[key]) > 0) {\n result.push(I18NDict[key]);\n }\n }\n if (result.length < 1) {\n return I18NDict.UNK;\n }\n return result.toString();\n };\n}\n\nexport const cnPOSTag = getPOSTagTranslator(POSTAG, CN_POS_NAMES);\nexport const enPOSTag = getPOSTagTranslator(POSTAG, EN_POS_NAMES);\n\nexport default POSTAG;\n","// @flow\nimport type { SegmentToken } from './module/type';\n\nexport default class Tokenizer {\n /**\n * 分词模块管理器\n *\n * @param {Segment} 分词接口\n */\n constructor(segment) {\n this.segment = segment;\n }\n\n /**\n * 对一段文本进行分词\n *\n * @param {string} text 文本\n * @param {array} modules 分词模块数组\n * @return {array}\n */\n split(text, modules): SegmentToken[] {\n if (modules.length < 1) {\n throw Error('No tokenizer module!');\n }\n // 按顺序分别调用各个module来进行分词 ： 各个module仅对没有识别类型的单词进行分词\n let result = [{ w: text }];\n modules.forEach(module => {\n result = module.split(result);\n });\n return result;\n }\n}\n","// @flow\nimport type { SegmentToken } from './module/type';\n\nexport default class Optimizer {\n /**\n * 优化模块管理器\n *\n * @param {Segment} 分词接口\n */\n constructoLine truncated
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
!function(n,x){"object"==typeof exports&&"undefined"!=typeof module?x(exports):"function"==typeof define&&define.amd?define(["exports"],x):x((n=n||self).Segmentit={})}(this,function(n){"use strict"
|
||||
function r(n,x){if(!(n instanceof x))throw new TypeError("Cannot call a class as a function")}function c(n,x){for(var e=0;e<x.length;e++){var c=x[e]
|
||||
c.enumerable=c.enumerable||!1,c.configurable=!0,"value"in c&&(c.writable=!0),Object.defineProperty(n,c.key,c)}}function e(n,x,e){return x&&c(n.prototype,x),e&&c(n,e),n}function a(n,x,e){return x in n?Object.defineProperty(n,x,{value:e,enumerable:!0,configurable:!0,writable:!0}):n[x]=e,n}function x(n,x){if("function"!=typeof x&&null!==x)throw new TypeError("Super expression must either be null or a function")
|
||||
n.prototype=Object.create(x&&x.prototype,{constructor:{value:n,writable:!0,configurable:!0}}),x&&f(n,x)}function i(n){return(i=Object.setPrototypeOf?Object.getPrototypeOf:function(n){return n.__proto__||Object.getPrototypeOf(n)})(n)}function f(n,x){return(f=Object.setPrototypeOf||function(n,x){return n.__proto__=x,n})(n,x)}function o(n){if(void 0===n)throw new ReferenceError("this hasn't been initialised - super() hasn't been called")
|
||||
return n}function u(n,x){return!x||"object"!=typeof x&&"function"!=typeof x?o(n):x}function t(n){return function(n){if(Array.isArray(n)){for(var x=0,e=Array(n.length);x<n.length;x++)e[x]=n[x]
|
||||
return e}}(n)||function(n){if(Symbol.iterator in Object(n)||"[object Arguments]"===Object.prototype.toString.call(n))return Array.from(n)}(n)||function(){throw new TypeError("Invalid attempt to spread non-iterable instance")}()}var s={D_A:1073741824,D_B:536870912,D_C:268435456,D_D:134217728,D_E:67108864,D_F:33554432,D_I:16777216,D_L:8388608,A_M:4194304,D_MQ:2097152,D_N:1048576,D_O:524288,D_P:262144,A_Q:131072,D_R:65536,D_S:32768,D_T:16384,D_U:8192,D_V:4096,D_W:2048,D_X:1024,D_Y:512,D_Z:256,A_NR:128,A_NS:64,A_NT:32,A_NX:16,A_NZ:8,D_ZH:4,D_K:2,UNK:0,URL:1}
|
||||
function p(c,f){return function(n){if(isNaN(n))return f[n]||f.UNK
|
||||
var x=[]
|
||||
for(var e in c)0<(n&c[e])&&x.push(f[e])
|
||||
return x.length<1?f.UNK:""+x}}function d(n){r(this,d),a(this,"type",void 0),this.segment=n}var l=p(s,{D_A:"形容词 形语素",D_B:"区别词 区别语素",D_C:"连词 连语素",D_D:"副词 副语素",D_E:"叹词 叹语素",D_F:"方位词 方位语素",D_I:"成语",D_L:"习语",A_M:"数词 数语素",D_MQ:"数量词",D_N:"名词 名语素",D_O:"拟声词",D_P:"介词",A_Q:"量词 量语素",D_R:"代词 代语素",D_S:"处所词",D_T:"时间词",D_U:"助词 助语素",D_V:"动词 动语素",D_W:"标点符号",D_X:"非语素字",D_Y:"语气词 语气语素",D_Z:"状态词",A_NR:"人名",A_NS:"地名",A_NT:"机构团体",A_NX:"外文字符",A_NZ:"其他专名",D_ZH:"前接成分",D_K:"后接成分",UNK:"未知",URL:"网址 邮箱地址"}),h=p(s,{D_A:"a",D_B:"b",D_C:"c",D_D:"d",D_E:"e",D_F:"f",D_I:"i",D_L:"l",A_M:"m",D_MQ:"mq",D_N:"n",D_O:"o",D_P:"p",A_Q:"q",D_R:"r",D_S:"s",D_T:"t",D_U:"u",D_V:"v",D_W:"w",D_X:"x",D_Y:"y",D_Z:"z",A_NR:"nr",A_NS:"ns",A_NT:"nt",A_NX:"nx",A_NZ:"nz",D_ZH:"h",D_K:"k",UNK:"un",URL:"uri"}),b=function(){function x(n){r(this,x),this.segment=n}return e(x,[{key:"split",value:function(n,x){if(x.length<1)throw Error("No tokenizer module!")
|
||||
var e=[{w:n}]
|
||||
return x.forEach(function(n){e=n.split(e)}),e}}]),x}(),w=function(){function x(n){r(this,x),this.segment=n}return e(x,[{key:"doOptimize",value:function(n,x){var e=t(n)
|
||||
return x.forEach(function(n){e=n.doOptimize(e)}),e}}]),x}(),D=function(){function n(){var i=this
|
||||
r(this,n),a(this,"use",function(n){if(Array.isArray(n))n.forEach(i.use)
|
||||
else if("function"==typeof n.init)n.init(i),i.modules[n.type].push(n)
|
||||
else{var x=new n(i)
|
||||
i.modules[x.type].push(x)}return i}),a(this,"loadDict",function(n){var type=1<arguments.length&&void 0!==arguments[1]?arguments[1]:"TABLE",x=2<arguments.length&&void 0!==arguments[2]&&arguments[2]
|
||||
if(Array.isArray(n))n.forEach(function(n){return i.loadDict(n)})
|
||||
else{i.DICT[type]||(i.DICT[type]={}),i.DICT["".concat(type,"2")]||(i.DICT["".concat(type,"2")]={})
|
||||
var c=i.DICT[type],f=i.DICT["".concat(type,"2")]
|
||||
n.split(/\r?\n/).map(function(n){return x?n.toLowerCase():n}).forEach(function(n){var x=n.split("|")
|
||||
if(2<x.length){var e=x[0].trim()
|
||||
0<e.length&&(c[e]={f:+x[2],p:+x[1]},f[e.length]||(f[e.length]={}),f[e.length][e]=c[e])}})}return i}),a(this,"getDict",function(type){return i.DICT[type]}),a(this,"loadSynonymDict",function(n){if(Array.isArray(n))n.forEach(function(n){return i.loadSynonymDict(n)})
|
||||
else{i.DICT.SYNONYM||(i.DICT.SYNONYM={})
|
||||
var c=i.DICT.SYNONYM
|
||||
n.split(/\r?\n/).map(function(n){return n.split(",")}).forEach(function(n){if(1<n.length){var x=n[0].trim(),e=n[1].trim()
|
||||
c[x]=e,c[e]===x&&delete c[e]}})}return i}),a(this,"loadStopwordDict",function(n){if(Array.isArray(n))n.forEach(function(n){return i.loadStopwordDict(n)})
|
||||
else{var type="STOPWORD"
|
||||
i.DICT[type]||(i.DICT[type]={})
|
||||
var x=i.DICT[type]
|
||||
n.split(/\r?\n/).map(function(n){return n.trim()}).forEach(function(n){n&&(x[n]=!0)})}return i}),a(this,"doSegment",function(n,x){var c=i
|
||||
x=x||{}
|
||||
var e=[]
|
||||
function f(n){var x=0,e=c.getDict("SYNONYM")
|
||||
return n=n.map(function(n){return n.w in e?(x++,{w:e[n.w],p:n.p}):n}),{count:x,list:n}}if(n.replace(/\r/g,"\n").split(/\n+/).forEach(function(n){if(!((n=n.trim()).length<1)){var x=c.tokenizer.split(n,c.modules.tokenizer)
|
||||
0<(x=c.optimizer.doOptimize(x,c.modules.optimizer)).length&&(e=e.concat(x))}}),x.stripPunctuation&&(e=e.filter(function(n){return n.p!==s.D_W})),x.convertSynonym)do{var t=f(e)
|
||||
e=t.list}while(0<t.count)
|
||||
if(x.stripStopword){var r=c.getDict("STOPWORD")
|
||||
e=e.filter(function(n){return!(n.w in r)})}return x.simple&&(e=e.map(function(n){return n.w})),e}),this.POSTAG=s,this.DICT={},this.modules={tokenizer:[],optimizer:[]},this.tokenizer=new b(this),this.optimizer=new w(this)}return e(n,[{key:"toString",value:function(n){return n.map(function(n){return n.w}).join("")}},{key:"split",value:function(n,x){for(var e=[],c=0,f=0,t="string"==typeof x?"w":"p";f<n.length;)n[f][t]===x?(c<f&&e.push(n.slice(c,f)),e.push(n.slice(f,f+1)),c=++f):f++
|
||||
return c<n.length-1&&e.push(n.slice(c,n.length)),e}},{key:"indexOf",value:function(n,x,e){e=isNaN(e)?0:e
|
||||
for(var c="string"==typeof x?"w":"p";e<n.length;){if(n[e][c]===x)return e
|
||||
e++}return-1}}]),n}(),_=function(){function t(){var n,x
|
||||
r(this,t)
|
||||
for(var e=arguments.length,c=Array(e),f=0;f<e;f++)c[f]=arguments[f]
|
||||
return a(o(x=u(this,(n=i(t)).call.apply(n,[this].concat(c)))),"type","tokenizer"),x}return x(t,d),t}(),A=function(){function t(){var n,x
|
||||
r(this,t)
|
||||
for(var e=arguments.length,c=Array(e),f=0;f<e;f++)c[f]=arguments[f]
|
||||
return a(o(x=u(this,(n=i(t)).call.apply(n,[this].concat(c)))),"type","optimizer"),x}return x(t,d),t}(),v=["赤","朱","丹","绯","彤","绛","茜","纁","赭","栗","褐","驼","赭","橘","曙","翠","碧","金","米","缃","靛","紫","藕","桃","青","玄","皂","乌","墨","黛","黝","素","杏","缟","鹤","皓","苍","华","银"].concat(t([["薰衣草紫红","#fff0f5","255,245,245"],["淡藕合","#f0dfee","240,238,238"],["甘石粉","#ffe8f3","255,243,243"],["浅粉红","#ffd9e6","255,230,230"],["浅血牙","#ffd7ea","255,234,234"],["粉凤仙","#fccce6","252,230,230"],["浅珍珠红","#ffb3e6","255,230,230"],["淡粉","#ffb3cc","255,204,204"],["火鹤红","#e68ab8","230,184,184"],["豆棕","#502e3e","80,62,62"],["茄皮紫","#3a212f","58,47,47"],["蒲萄灰","#4e2c3f","78,63,63"],["荷花","#eb7faf","235,175,175"],["灰紫红","#db7093","219,147,147"],["浅珊瑚红","#ff80bf","255,191,191"],["暗粉","#ff77bb","255,187,187"],["陈玫红","#b85798","184,152,152"],["肉红","#d26091","210,145,145"],["尖晶石红","#ff73b3","255,179,179"],["银红","#c85b92","200,146,146"],["暖粉红","#ff69b4","255,180,180"],["浅玫瑰红","#ff66cc","255,204,204"],["紫扇贝色","#923a60","146,96,96"],["长春花色","#ff47d1","255,209,209"],["山茶红","#e63995","230,149,149"],["中青紫红","#c71585","199,133,133"],["深粉红","#ff1493","255,147,147"],["洋玫瑰红","#ff0da6","255,166,166"],["玫瑰红","#ff007f","255,127,127"],["品红","#f400a1","244,161,161"],["胭脂红","#e6005c","230,92,92"],["艶粉","#e6005c","230,92,92"],["红宝石色","#cc0080","204,128,128"],["枢机红","#990036","153,54,54"],["毅红","#8a002e","138,46,46"],["牵牛紫","#800040","128,64,64"],["紫粉","#80002b","128,43,43"],["鶏冠花红","#660033","102,51,51"],["枣红","#640021","100,33,33"],["紫绛色","#51001b","81,27,27"],["玫瑰灰","#4d0026","77,38,38"],["勃艮第酒红","#470024","71,36,36"],["黒紫","#3e0014","62,20,20"],["雪紫","#33001a","51,26,26"],["深红","#33001a","51,26,26"],["雪色","#fffafa","255,250,250"],["浅肉粉","#f5eeeb","245,235,235"],["粉棕","#ebdad3","235,211,211"],["雾玫瑰色","#ffe4e1","255,225,225"],["米红","#fddfdf","253,223,223"],["沙棕","#e6c3c3","230,195,195"],["肉粉","#ffd7d7","255,215,215"],["奶棕","#deb6b6","222,182,182"],["玫瑰褐","#bc8f8f","188,143,143"],["红灰莲","#ab8282","171,130,130"],["粉红","#ffc0cb","255,203,203"],["亮粉红","#ffb6c1","255,193,193"],["壳黄红","#ffb3bf","255,191,191"],["浅粉","#ffb3b3","255,179,179"],["十様锦","#ffb3b3","255,179,179"],["深烟","#311f1e","49,30,30"],["凤仙粉","#ff9393","255,147,147"],["亮珊瑚色","#f08080","240,128,128"],["暗鲑红","#e9967a","233,122,122"],["绛紫","#492525","73,37,37"],["浅鲑红","#ff8099","255,153,153"],["紫铜色","#6b3636","107,54,54"],["珈琲色","#623131","98,49,49"],["烟栗色","#221311","34,17,17"],["亮鲑红","#ffa07a","255,122,122"],["红莲灰","#c76079","199,121,121"],["鲑红","#fa8072","250,114,114"],["印度红","#cd5c5c","205,92,92"],["银朱","#c85554","200,84,84"],["赦红","#d1634b","209,75,75"],["二硃","#c24545","194,69,69"],["芯木色","#c14444","193,68,68"],["銹红","#4d1919","77,25,25"],["珊瑚红","#ff7f50","255,80,80"],["辰沙","#ce3b3b","206,59,59"],["红扇贝色","#ce3b3b","206,59,59"],["蕃茄红","#ff6347","255,71,71"],["柿子橙","#ff4d40","255,64,64"],["妃红","#b8542e","184,46,46"],["甎红","#ab2b2b","171,43,43"],["褐色","#a52a2a","165,42,42"],["硃砂","#972626","151,38,38"],["土红","#bc2e2e","188,46,46"],["樱桃红","#de3163","222,99,99"],["砖红色","#b22222","178,34,34"],["茜红","#e32636","227,54,54"],["银星海棠","#f52443","245,67,67"],["橙色","#fe621f","254,31,31"],["桔红","#ff441a","255,26,26"],["柿子色","#ff531a","255,26,26"],["绯红","#dc143c","220,60,60"],["月季红","#e10b2b","225,43,43"],["鸟罗松红","#c80926","200,38,38"],["蕉红色","#d70419","215,25,25"],["艶红","#bf0417","191,23,23"],["烟色","#2e0a01","46,1,1"],["牡丹红","#b80233","184,51,51"],["榴花红","#b02502","176,2,2"],["枣紫","#540101","84,1,1"],["红色","#ff0000","255,0,0"],["腥红","#ff2400","255,0,0"],["橙红","#ff4500","255,0,0"],["朱红","#ff4d00","255,0,0"],["鲜红","#e60000","230,0,0"],["曙红","#e60039","230,57,57"],["章丹","#dd3700","221,0,0"],["茉莉花红","#cc0033","204,51,51"],["猩红","#cc0000","204,0,0"],["象牙红","#b30000","179,0,0"],["血红","#990000","153,0,0"],["深釉红","#971b00","151,0,0"],["暗红","#8b0000","139,0,0"],["栗色","#800000","128,0,0"],["釉红","#7b1800","123,0,0"],["臙脂","#700000","112,0,0"],["花卉白","#fffaf0","255,240,240"],["海贝色","#fff5ee","255,238,238"],["浅棕灰","#eee7dd","238,221,221"],["亚麻色","#faf0e6","250,230,230"],["胡粉","#fffae8","255,232,232"],["旧蕾丝色","#fdf5e6","253,230,230"],["银白色","#fffae6","255,230,230"],["铁灰色","#625b57","98,87,87"],["蝋白","#fef8de","254,222,222"],["玉米丝色","#fff8dc","255,220,220"],["古董Line truncated
|
||||
f&&(f.p===x.D_U&&v.includes(c.w)&&(c.p=x.D_A),c.p===x.D_N&&this.isNominal(f.p)&&v.includes(c.w)&&(c.p=x.D_A)),e+=1}return n}},{key:"isNominal",value:function(n){if(Array.isArray(n))return this.isNominal(n[0])
|
||||
var x=this.segment.POSTAG
|
||||
return n===x.D_N||n===x.A_NT||n===x.A_NX||n===x.A_NZ||n===x.A_NR||n===x.A_NS||n===x.URL}}]),n}()
|
||||
function N(n,x){var e={}
|
||||
return n.forEach(function(n){e[n]=x}),e}var F=N(["王","张","黄","周","徐","胡","高","林","马","于","程","傅","曾","叶","余","夏","钟","田","任","方","石","熊","白","毛","江","史","候","龙","万","段","雷","钱","汤","易","常","武","赖","文","查","赵","肖","孙","李","吴","郑","冯","陈","褚","卫","蒋","沈","韩","杨","朱","秦","尤","许","何","吕","施","桓","孔","曹","严","华","金","魏","陶","姜","戚","谢","邹","喻","柏","窦","苏","潘","葛","奚","范","彭","鲁","韦","昌","俞","袁","酆","鲍","唐","费","廉","岑","薛","贺","倪","滕","殷","罗","毕","郝","邬","卞","康","卜","顾","孟","穆","萧","尹","姚","邵","湛","汪","祁","禹","狄","贝","臧","伏","戴","宋","茅","庞","纪","舒","屈","祝","董","梁","杜","阮","闵","贾","娄","颜","郭","邱","骆","蔡","樊","凌","霍","虞","柯","昝","卢","柯","缪","宗","丁","贲","邓","郁","杭","洪","崔","龚","嵇","邢","滑","裴","陆","荣","荀","惠","甄","芮","羿","储","靳","汲","邴","糜","隗","侯","宓","蓬","郗","仲","栾","钭","历","戎","刘","詹","幸","韶","郜","黎","蓟","溥","蒲","邰","鄂","咸","卓","蔺","屠","乔","郁","胥","苍","莘","翟","谭","贡","劳","冉","郦","雍","璩","桑","桂","濮","扈","冀","浦","庄","晏","瞿","阎","慕","茹","习","宦","艾","容","慎","戈","廖","庾","衡","耿","弘","匡","阙","殳","沃","蔚","夔","隆","巩","聂","晁","敖","融","訾","辛","阚","毋","乜","鞠","丰","蒯","荆","竺","盍","单","欧"],1),y=N(["司马","上官","欧阳","夏侯","诸葛","闻人","东方","赫连","皇甫","尉迟","公羊","澹台","公冶","宗政","濮阳","淳于","单于","太叔","申屠","公孙","仲孙","轩辕","令狐","徐离","宇文","长孙","慕容","司徒","司空","万俟"],2),m=N(["阿","建","小","晓","文","志","国","玉","丽","永","海","春","金","明","新","德","秀","红","亚","伟","雪","俊","桂","爱","美","世","正","庆","学","家","立","淑","振","云","华","光","惠","兴","天","长","艳","慧","利","宏","佳","瑞","凤","荣","秋","继","嘉","卫","燕","思","维","少","福","忠","宝","子","成","月","洪","东","一","泽","林","大","素","旭","宇","智","锦","冬","玲","雅","伯","翠","传","启","剑","安","树","良","中","梦","广","昌","元","万","清","静","友","宗","兆","丹","克","彩","绍","喜","远","朝","敏","培","胜","祖","先","菊","士","向","有","连","军","健","巧","耀","莉","英","方","和","仁","孝","梅","汉","兰","松","水","江","益","开","景","运","贵","祥","青","芳","碧","婷","龙","鹏","自","顺","双","书","生","义","跃","银","佩","雨","保","贤","仲","鸿","浩","加","定","炳","飞","锡","柏","发","超","道","怀","进","其","富","平","全","阳","吉","茂","彦","诗","洁","润","承","治","焕","如","君","增","善","希","根","应","勇","宜","守","会","凯","育","湘","凌","本","敬","博","延","乐","三","二","四","五","六","七","八","九","十"],1),C=N(["华","平","明","英","军","林","萍","芳","玲","红","生","霞","梅","文","荣","珍","兰","娟","峰","琴","云","辉","东","龙","敏","伟","强","丽","春","杰","燕","民","君","波","国","芬","清","祥","斌","婷","飞","良","忠","新","凤","锋","成","勇","刚","玉","元","宇","海","兵","安","庆","涛","鹏","亮","青","阳","艳","松","江","莲","娜","兴","光","德","武","香","俊","秀","慧","雄","才","宏","群","琼","胜","超","彬","莉","中","山","富","花","宁","利","贵","福","发","义","蓉","喜","娥","昌","仁","志","全","宝","权","美","琳","建","金","贤","星","丹","根","和","珠","康","菊","琪","坤","泉","秋","静","佳","顺","源","珊","达","欣","如","莹","章","浩","勤","芹","容","友","芝","豪","洁","鑫","惠","洪","旺","虎","远","妮","森","妹","南","雯","奇","健","卿","虹","娇","媛","怡","铭","川","进","博","智","来","琦","学","聪","洋","乐","年","翔","然","栋","凯","颖","鸣","丰","瑞","奎","立","堂","威","雪","鸿","晶","桂","凡","娣","先","洲","毅","雅","月","旭","田","晖","方","恒","亚","泽","风","银","高","贞","九","薇"],2),E=N(["家","民","敏","伟","勇","军","斌","静","丽","涛","芳","杰","萍","强","俊","明","燕","磊","玲","华","平","鹏","健","波","红","丹","辉","超","艳","莉","刚","娟","峰","婷","亮","洁","颖","琳","英","慧","飞","霞","浩","凯","宇","毅","林","佳","云","莹","娜","晶","洋","文","鑫","欣","琴","宁","琼","兵","青","琦","翔","彬","锋","阳","璐","旭","蕾","剑","虹","蓉","建","倩","梅","宏","威","博","君","力","龙","晨","薇","雪","琪","欢","荣","江","炜","成","庆","冰","东","帆","雷","楠","锐","进","海","凡","巍","维","迪","媛",Line truncated
|
||||
else{var t=s.matchName(x.w)
|
||||
if(t.length<1)c.push(x)
|
||||
else{for(var r,i=0,a=0;r=t[a];a++)i<r.c&&c.push({w:x.w.substr(i,r.c-i)}),c.push({w:r.w,p:e.A_NR}),i=r.c+r.w.length
|
||||
var o=t[t.length-1]
|
||||
o.c+o.w.length<x.w.length&&c.push({w:x.w.substr(o.c+o.w.length)})}}return c}}],[{key:"matchName",value:function(n,x){var e=0
|
||||
isNaN(x)||(e=x)
|
||||
for(var c=[];e<n.length;){var f=!1,t=n.substr(e,2)
|
||||
if(t in y){var r=n[0|e+2],i=n[0|e+3]
|
||||
r in m&&i in C?f=t+r+i:r in E&&(f=t+r+(r==i?i:""))}var a=n[0|e]
|
||||
if(!1===f&&a in F){var o=n[0|e+1],s=n[0|e+2]
|
||||
o in m&&s in C?f=a+o+s:o in E&&(f=a+o+(o==s?s:""))}!1===f?e++:(c.push({w:f,c:e}),e+=f.length)}return c}}]),s}(),O=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,A),e(n,[{key:"doOptimize",value:function(n,x){void 0===x&&(x=!1)
|
||||
for(var e=this.segment.getDict("TABLE"),c=this.segment.POSTAG,f=0,t=n.length-1;f<t;){var r=n[f],i=n[f+1],a=r.w+i.w
|
||||
if(r.p===i.p&&a in e)n.splice(f,2,{w:a,p:e[a].p}),t--
|
||||
else if(0<(r.p&c.D_A)&&i.p&c.D_U)n.splice(f,2,{w:a,p:c.D_A}),t--
|
||||
else{if(0<(r.p&c.A_M)){if(0<(i.p&c.A_M)||"%"===i.w){n.splice(f,2,{w:r.w+i.w,p:c.A_M}),t--
|
||||
continue}if(0<(i.p&c.A_Q)){n.splice(f,2,{w:r.w+i.w,p:c.D_MQ}),t--
|
||||
continue}if((p=n[f+2])&&0<(p.p&c.A_M)&&("."===i.w||"点"===i.w||"分之"===i.w)){n.splice(f,3,{w:r.w+i.w+p.w,p:c.A_M}),t-=2
|
||||
continue}}if(0<(r.p&c.D_MQ)&&"点"==r.w.substr(-1)&&i.p&c.A_M){for(var o=2,s="",u=f+o;u<t;u++){var p
|
||||
if(!(0<((p=n[u]).p&c.A_M)))break
|
||||
s+=p.w,o++}n.splice(f,o,{w:r.w+i.w+s,p:c.D_MQ}),t-=o-1}else f++}}return!0===x?n:this.doOptimize(n,!0)}}]),n}(),S="!\"#$%&'*+-/0123456789=?@ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz{|}~.".split(""),M={}
|
||||
for(var B in S)M[S[B]]=1
|
||||
var P=function(){function o(){return r(this,o),u(this,i(o).apply(this,arguments))}return x(o,A),e(o,[{key:"doOptimize",value:function(n){for(var x=this.segment.POSTAG,e=0,c=n.length-1,f=!1,t=!1;e<c;){var r,i=!!((r=n[e]).p===x.A_NX||r.p===x.A_M&&r.w.charCodeAt(0)<128)
|
||||
if(!1===f&&i)f=e,e++
|
||||
else if(!1!==t||"@"!==r.w)if(!1===t||"@"==n[e-1].w||!1!==i||r.w in M)!1!==f&&(i||r.w in M)||(t=f=!1),e++
|
||||
else{var a=n.slice(f,e)
|
||||
n.splice(f,a.length,{w:o.toEmailAddress(a),p:x.URL}),e=f+1,c-=a.length-1,t=f=!1}else t=!0,e++}if(f&&t&&n[c]&&(i=!!((r=n[c]).p===x.A_NX||r.p===x.A_M&&r.w in M))){a=n.slice(f,n.length)
|
||||
n.splice(f,a.length,{w:o.toEmailAddress(a),p:x.URL})}return n}}],[{key:"toEmailAddress",value:function(n){for(var x,e=n[0].w,c=1;x=n[c];c++)e+=x.w
|
||||
return e}}]),o}(),z=" ,.;+-|/\\'\":?<>[]{}=!@#$%^&*()~`。，、＇：∶；?‘’“”〝〞ˆˇ﹕︰﹔﹖﹑·¨….¸;！´？！～—ˉ｜‖＂〃｀@﹫¡¿﹏﹋﹌︴々﹟#﹩$﹠&﹪%*﹡﹢﹦﹤‐￣¯―﹨ˆ˜﹍﹎+=<＿_-ˇ~﹉﹊（）〈〉‹›﹛﹜『』〖〗［］《》〔〕{}「」【】︵︷︿︹︽_﹁﹃︻︶︸﹀︺︾ˉ﹂﹄︼＋－×÷﹢﹣±／＝≈≡≠∧∨∑∏∪∩∈⊙⌒⊥∥∠∽≌＜＞≤≥≮≯∧∨√﹙﹚[]﹛﹜∫∮∝∞⊙∏┌┬┐┏┳┓╒╤╕─│├┼┤┣╋┫╞╪╡━┃└┴┘┗┻┛╘╧╛┄┆┅┇╭─╮┏━┓╔╦╗┈┊│╳│┃┃╠╬╣┉┋╰─╯┗━┛╚╩╝╲╱┞┟┠┡┢┦┧┨┩┪╉╊┭┮┯┰┱┲┵┶┷┸╇╈┹┺┽┾┿╀╁╂╃╄╅╆○◇□△▽☆●◆■▲▼★♠♥♦♣☼☺◘♀√☻◙♂×▁▂▃▄▅▆▇█⊙◎۞卍卐╱╲▁▏↖↗↑←↔◤◥╲╱▔▕↙↘↓→↕◣◢∷▒░℡™"
|
||||
z=z.split("")
|
||||
var R={},U={}
|
||||
for(var I in z)if(""!==z[I]){var L=z[I].length
|
||||
U[R[z[I]]=L]||(U[L]={}),U[L][z[I]]=L}var j=function(){function s(){return r(this,s),u(this,i(s).apply(this,arguments))}return x(s,_),e(s,[{key:"split",value:function(n){for(var x,e=this.segment.POSTAG,c=[],f=0;x=n[f];f++)if(0<x.p)c.push(x)
|
||||
else{var t=s.matchStopword(x.w)
|
||||
if(t.length<1)c.push(x)
|
||||
else{for(var r,i=0,a=0;r=t[a];a++)i<r.c&&c.push({w:x.w.substr(i,r.c-i)})," "!=r.w&&c.push({w:r.w,p:e.D_W}),i=r.c+r.w.length
|
||||
var o=t[t.length-1]
|
||||
o.c+o.w.length<x.w.length&&c.push({w:x.w.substr(o.c+o.w.length)})}}return c}}],[{key:"matchStopword",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],c=!1;x<n.length;){for(var f in U){var t=n.substr(x,f)
|
||||
if(t in U[f]){e.push({w:t,c:x}),c=!0
|
||||
break}}x+=!1===c?1:t.length,c=!1}return e}}]),s}(),G=["http://","https://","ftp://","news://","telnet://"],K=100
|
||||
for(var W in G)G[W].length<K&&(K=G[W].length)
|
||||
var X=["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","0","1","2","3","4","5","6","7","8","9","!","#","$","%","&","‘","(",")","*","+",",","-",".","/",":",";","=","?","@","[","\\","]","^","_","`","|","~"],Q={}
|
||||
for(var W in X)Q[X[W]]=1
|
||||
var Y=function(){function s(){return r(this,s),u(this,i(s).apply(this,arguments))}return x(s,_),e(s,[{key:"split",value:function(n){for(var x,e=this.segment.POSTAG,c=[],f=0;x=n[f];f++)if(0<x.p)c.push(x)
|
||||
else{var t=s.matchURL(x.w)
|
||||
if(t.length<1)c.push(x)
|
||||
else{for(var r,i=0,a=0;r=t[a];a++)i<r.c&&c.push({w:x.w.substr(i,r.c-i)}),c.push({w:r.w,p:e.URL}),i=r.c+r.w.length
|
||||
var o=t[t.length-1]
|
||||
o.c+o.w.length<x.w.length&&c.push({w:x.w.substr(o.c+o.w.length)})}}return c}}],[{key:"matchURL",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],c=!1;x<n.length;){if(!1===c&&x<n.length-K){for(var f,t=0;f=G[t];t++)if(n.substr(x,f.length)===f){c=x,x+=f.length-1
|
||||
break}}else!1===c||n[0|x]in Q||(e.push({w:n.substr(c,x-c),c:c}),c=!1)
|
||||
x++}return!1!==c&&e.push({w:n.substr(c,x-c),c:c}),e}}]),s}(),Z=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,A),e(n,[{key:"doOptimize",value:function(n){for(var x=this.segment.POSTAG,e=0;e<n.length;){var c=n[e],f=n[e+1]
|
||||
if(f){if(f&&("小"===c.w||"老"===c.w)&&(f.w in F||f.w in y)){n.splice(e,2,{w:c.w+f.w,p:x.A_NR}),e++
|
||||
continue}if((c.w in F||c.w in y)&&0<(f.p&x.A_NR)&&f.w.length<=2){n.splice(e,2,{w:c.w+f.w,p:x.A_NR}),e++
|
||||
continue}if((!c.p||!f.p)&&(c.w in E&&c.w===f.w||c.w in m&&f.w in C)){n.splice(e,2,{w:c.w+f.w,p:x.A_NR})
|
||||
var t=n[e-1]
|
||||
t&&(t.w in F||t.w in y)?n.splice(e-1,2,{w:t.w+c.w+f.w,p:x.A_NR}):e++
|
||||
continue}!(c.w in F||c.w in y)||c.p&&f.p||n.splice(e,2,{w:c.w+f.w,p:x.A_NR})}e++}for(e=0;e<n.length;){var r=n[e],i=n[e+1]
|
||||
i&&(r.w in F||r.w in y)&&i.w in E?(n.splice(e,2,{w:r.w+i.w,p:x.A_NR}),e++):e++}return n}}]),n}(),H=["世纪","年","年份","年度","月","月份","月度","日","号","时","点","点钟","分","分钟","秒","毫秒"],V={}
|
||||
for(var q in H)V[H[q]]=H[q].length
|
||||
var $=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,A),e(n,[{key:"doOptimize",value:function(n){this.segment.getDict("TABLE")
|
||||
for(var x=this.segment.POSTAG,e=0,c=n.length-1;e<c;){var f=n[e+1]
|
||||
if(0<((i=n[e]).p&x.A_M)&&f.w in V){for(var t=i.w+f.w,r=2;;){var i
|
||||
f=n[e+r+1]
|
||||
if(!((i=n[e+r])&&f&&0<(i.p&x.A_M)&&f.w in V))break
|
||||
r+=2,t+=i.w+f.w}n.splice(e,r,{w:t,p:x.D_T}),c-=r-1}else e++}return n}}]),n}(),J=["世纪","年","年份","年度","月","月份","月度","日","号","时","点","点钟","分","分钟","秒","毫秒"],nn={}
|
||||
for(var xn in J)nn[J[xn]]=J[xn].length
|
||||
var en=function(){function D(){return r(this,D),u(this,i(D).apply(this,arguments))}return x(D,_),e(D,[{key:"split",value:function(n){for(var x,e=this.segment.getDict("TABLE"),c=[],f=0;x=n[f];f++)if(0<x.p)c.push(x)
|
||||
else{var t=this.matchWord(x.w,0,n[f-1])
|
||||
if(t.length<1)c.push(x)
|
||||
else{for(var r,i=0,a=0;r=t[a];a++)i<r.c&&c.push({w:x.w.substr(i,r.c-i)}),c.push({w:r.w,p:e[r.w].p}),i=r.c+r.w.length
|
||||
var o=t[t.length-1]
|
||||
o.c+o.w.length<x.w.length&&c.push({w:x.w.substr(o.c+o.w.length)})}}return c}},{key:"matchWord",value:function(n,x,e){isNaN(x)&&(x=0)
|
||||
for(var c=[],f=this.segment.getDict("TABLE2");x<n.length;){for(var t in f){var r=n.substr(x,t)
|
||||
r in f[t]&&c.push({w:r,c:x,f:f[t][r].f})}x++}return this.filterWord(c,e,n)}},{key:"filterWord",value:function(n,x,e){for(var c,f=this.segment.POSTAG,t=this.segment.getDict("TABLE"),r=D.getPosInfo(n,e),i=D.getChunks(r,0,e),a=[],o=0;c=i[o];o++){var s=e.length/c.length,u=!(a[o]={x:c.length,a:0,b:0,c:0,d:0})
|
||||
if(x)var p={w:x.w,p:x.p,f:x.f}
|
||||
else p=!1
|
||||
for(var d,l=0;d=c[l];l++){if(d.w in t){if(d.p=t[d.w].p,a[o].a+=d.f,p){0<(p.p&f.A_M)&&(0<(t[d.w].p&f.A_Q)||d.w in nn)&&a[o].d++,0<(d.p&f.D_V)&&(u=!0,0<(p.p&f.D_A)&&a[o].d++),(0<(p.p&f.A_NS)||p.p&f.A_NT||0<(p.p&f.D_A))&&(0<(d.p&f.D_N)||0<(d.p&f.A_NR)||0<(d.p&f.A_NS)||0<(d.p&f.A_NZ)||0<(d.p&f.A_NT))&&a[o].d++,0<(p.p&f.D_F)&&(d.p&0<f.A_M||d.p&0<f.D_MQ)&&a[o].d++,(p.w in F||p.w in y)&&(0<(d.p&f.D_N)||0<(d.p&f.A_NZ))&&a[o].d++
|
||||
var h=c[l+1]
|
||||
h&&(h.w in t&&(h.p=t[h.w].p),0<(d.p&f.D_C)&&p.p===h.p&&a[o].d++,"的"!==d.w&&"之"!==d.w||!(0<(h.p&f.D_N)||0<(h.p&f.A_NR)||0<(h.p&f.A_NS)||0<(h.p&f.A_NZ)||0<(h.p&f.A_NT))||(a[o].d+=1.5))}}else a[o].c++
|
||||
a[o].b+=Math.pow(s-d.w.length,2),p=c[l]}!1===u&&(a[o].d-=.5),a[o].a=a[o].a/c.length,a[o].b=a[o].b/c.length}var b,w=i[D.getTops(a)]
|
||||
for(o=0;b=w[o];o++)b.w in t||w.splice(o--,1)
|
||||
return w}}],[{key:"getPosInfo",value:function(n,x){for(var e,c={},f=0;e=n[f];f++)c[e.c]||(c[e.c]=[]),c[e.c].push(e)
|
||||
for(f=0;f<x.length;f++)c[f]||(c[f]=[{w:x[0|f],c:f,f:0}])
|
||||
return c}},{key:"getChunks",value:function(n,x){for(var e=n[x]||[],c=[],f=0;f<e.length;f++){var t=e[f],r=t.c+t.w.length
|
||||
if(n[r])for(var i=D.getChunks(n,r),a=0;a<i.length;a++)c.push([t].concat(i[a]))
|
||||
else c.push([t])}return c}},{key:"getTops",value:function(n){for(var x={x:n[0].x,a:n[0].a,b:n[0].b,c:n[0].c,d:n[0].d},e=1;c=n[e];e++)x.a<c.a&&(x.a=c.a),c.b<x.b&&(x.b=c.b),x.c<c.c&&(x.c=c.c),c.d<x.d&&(x.d=c.d),x.x<c.x&&(x.x=c.x)
|
||||
var c,f=[]
|
||||
for(e=0;c=n[e];e++)f[e]=0,f[e]+=1.5*(x.x-c.x),x.a<=c.a&&(f[e]+=1),c.b<=x.b&&(f[e]+=1),f[e]+=x.c-c.c,f[e]+=1*(c.d<0?x.d+c.d:c.d-x.d)
|
||||
var t=0,r=f[0]
|
||||
for(var e in f){var i=f[e]
|
||||
if(r<i)t=e,r=i
|
||||
else if(i===r){var a=0,o=0
|
||||
n[e].c<n[t].c?a++:o++,n[t].a<n[e].a?a++:o++,n[e].x<n[t].x?a++:o++,o<a&&(t=e,r=i)}}return t}}]),D}(),cn=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,_),e(n,[{key:"split",value:function(n){for(var x,e=[],c=0;x=n[c];c++)x.p?e.push(x):e=e.concat(this.splitForeign(x.w))
|
||||
return e}},{key:"splitForeign",value:function(n){var x=this.segment.POSTAG,e=[],c=0,f=0
|
||||
65296<=(r=n.charCodeAt(0))&&r<=65370&&(r-=65248),f=48<=r&&r<=57?x.A_M:65<=r&&r<=90||97<=r&&r<=122?x.A_NX:x.UNK
|
||||
for(var t=1;t<n.length;t++){var r
|
||||
if(65296<=(r=n.charCodeAt(t))&&r<=65370&&(r-=65248),48<=r&&r<=57){if(f!==x.A_M){var i={w:n.substr(c,t-c)}
|
||||
f!==x.UNK&&(i.p=f),e.push(i),c=t}f=x.A_M}else if(65<=r&&r<=90||97<=r&&r<=122){if(f!==x.A_NX){i={w:n.substr(c,t-c)}
|
||||
f!==x.UNK&&(i.p=f),e.push(i),c=t}f=x.A_NX}else f!==x.UNK&&(e.push({w:n.substr(c,t-c),p:[f]}),c=t),f=x.UNK}i={w:n.substr(c,t-c)}
|
||||
return f!==x.UNK&&(i.p=f),e.push(i),e}}]),n}(),fn=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,_),e(n,[{key:"split",value:function(n){for(var x,e=[],c=0;x=n[c];c++)x.p?e.push(x):e=e.concat(this.splitSingle(x.w))
|
||||
return e}},{key:"splitSingle",value:function(n,x){var e=this.segment.POSTAG
|
||||
isNaN(x)&&(x=0)
|
||||
for(var c=[];x<n.length;)c.push({w:n[0|x],p:e.UNK}),x++
|
||||
return c}}]),n}(),tn=function(){function n(){return r(this,n),u(this,i(n).apply(this,arguments))}return x(n,_),e(n,[{key:"split",value:function(n){for(var x,e=this.segment.getDict("WILDCARD"),c=[],f=0;x=n[f];f++)if(0<x.p)c.push(x)
|
||||
else{var t=this.matchWord(x.w)
|
||||
if(t.length<1)c.push(x)
|
||||
else{for(var r,i=0,a=0;r=t[a];a++)i<r.c&&c.push({w:x.w.substr(i,r.c-i)}),c.push({w:r.w,p:e[r.w.toLowerCase()].p}),i=r.c+r.w.length
|
||||
var o=t[t.length-1]
|
||||
o.c+o.w.length<x.w.length&&c.push({w:x.w.substr(o.c+o.w.length)})}}return c}},{key:"matchWord",value:function(n,x){isNaN(x)&&(x=0)
|
||||
for(var e=[],c=this.segment.getDict("WILDCARD2"),f=n.toLowerCase();x<n.length;){var t=!1
|
||||
for(var r in c)f.substr(x,r)in c[r]&&(t={w:n.substr(x,r),c:x})
|
||||
!1!==t?(e.push(t),x+=t.w.length):x++}return e}}]),n}(),rn=[Y,tn,j,cn,en,k,P,Z,O,$,g],an="\ufeff爱|0x1000|323\n安|0x1000|1651\n凹|0x40000000|10293\n熬|0x1000|36595\n八|0x400000|355\n扒|0x1000|16883\n拔|0x1000|3784\n把|0x40000|22\n摆|0x1000|951\n拜|0x1000|3985\n搬|0x1000|2305\n办|0x1000|261\n扮|0x1000|10508\n拌|0x1000|11672\n绊|0x1000|12776\n帮|0x1000|889\n绑|0x1000|7942\n傍|0x1000|12168\n包|0x1000|884\n饱|0x40000000|4431\n保|0x1000|981\n报|0x100000|438\n背|0x1000|3930\n倍|0x8000000|973\n本|0x10000|115\n绷|0x100000|14002\n逼|0x1000|2473\n比|0x40000|98\n必|0x8000000|1335\n毙|0x1000|8263\n避|0x1000|4562\n编|0x1000|2131\n贬|0x1000|9759\n拚|0x0400|0\n变|0x1000|264\n遍|0x20000|1624\n辨|0x1000|7007\n别|0x8000000|40734\n瘪|0x1000|15221\n并|0x10000000|45\n病|0x100000|383\n拨|0x1000|3172\n剥|0x1000|6852\n播|0x1000|5785\n驳|0x100000|9902\n搏|0x1000|11464\n补|0x1000|1726\n捕|0x1000|5370\n不|0x8000000|5\n步|0x20000|397\n擦|0x1000|2620\n才|0x8000000|82\n采|0x1000|2522\n操|0x1000|4061\n层|0x20000|640\n插|0x1000|2659\n查|0x1000|1652\n岔|0x100000|12396\n差|0x40000000|1299\n产|0x1000|1945\n长|0x40000000|1368\n常|0x8000000|648\n场|0x20000|6159\n抄|0x1000|5307\n超|0x1000|1760\n朝|0x100000|993\n吵|0x1000|5029\n炒|0x1000|5094\n扯|0x1000|3761\n闯|0x1000|2295\n称|0x1000|448\n乘|0x1000|1581\n吃|0x1000|184\n持|0x1000|1894\n斥|0x1000|0\n冲|0x1000|1093\n丑|0x40000000|5722\n出|0x1000|49\n除|0x40000|663\n穿|0x1000|774\n吹|0x1000|1510\n纯|0x40000000|3093\n蠢|0x1000|0\n此|0x10000|119\n次|0x20000|57\n从|0x40000|27\n促|0x1000|3235\n存|0x1000|2089\n寸|0x20000|5741\n搓|0x1000|7703\n打|0x1000|17890\n大|0x40000000|19\n代|0x1000|503\n带|0x1000|178\n待|0x1000|4627\n贷|0x1000|6275\n单|0x8000000|816\n但|0x10000000|37\n淡|0x40000000|5053\n当|0x40000|591\n挡|0x1000|5085\n倒|0x1000|1212\n到|0x1000|18\n得|0x2000|4434\n的|0x2000|1\n登|0x1000|1810\n等|0x2000|24\n低|0x40000000|254\n敌|0x1000|2139\n地|0x2000|1382\n第|0x100000|0\n点|0x20000|116\n电|0x100000|462\n钓|0x100000|11268\n钉|0x1000|9869\n顶|0x1000|1298\n订|0x1000|4503\n定|0x1000|694\n动|0x1000|772\n冻|0x1000|5210\n都|0x8000000|1244\n斗|0x1000|4900\n毒|0x100000|2969\n独|0x8000000|2378\n堵|0x1000|4272\n短|0x40000000|788\n段|0x20000|301\n堆|0x20000|1577\n对|0x40000|14\n兑|0x100000|10085\n顿|0x20000|1803\n夺|0x1000|2841\n朵|0x20000|3952\n恶|0x40000000|2459\n而|0x10000000|28\n二|0x40400000|88\n发|0x1000|3954\n罚|0x1000|3677\n翻|0x1000|1476\n凡|0x8000000|1749\n反|0x1000|315\n犯|0x1000|1800\n方|0x20000|372\n防|0x100000|1637\n仿|0x1000|9692\n放|0x1000|425\n非|0x8000000|185\n肥|0x100000|3240\n废|0x1000|3560\n费|0x1000|1015\n分|0x20000|168\n份|0x20000|490\n封|0x20000|1050\n否|0x1000|6368\n扶|0x1000|3217\n拂|0x1000|9728\n付|0x1000|3646\n负|0x1000|1282\n附|0x1000|1327\n赴|0x100000|1530\n副|0x20000000|122\n傅|0x0400|7535\n富|0x40000000|1725\n赋|0x100000|7857\n缚|0x1000|10335\n该|0x10000|107\n改|0x1000|665\n盖|0x1000|1401\n干|0x1000|8006\n赶|0x1000|1951\n敢|0x1000|805\n刚|0x8000000|542\n高|0x40000000|76\n搞|0x1000|339\n告|0x1000|2472\n割|0x1000|4622\n个|0x20000|8\n给|0x40000|54\n跟|0x40000|434\n更|0x8000000|56\n攻|0x1000|2631\n供|0x1000|1332\n共|0x8000000|358\n够|0x1000|1816\n估|0x1000|15021\n古|0x40000000|1283\n故|0x10000000|1574\n关|0x1000|903\n管|0x1000|368\n光|0x100000|558\n广|0x40000000|1429\n归|0x1000|1388\n滚|0x1000|3632\n过|0x2000|2722\n含|0x1000|1541\n喊|0x1000|1617\n好|0x40000000|722\n耗|0x1000|6572\n喝|0x1000|2636\n合|0x1000|4244\n何|0x10000|1081\n和|0x10000000|373\n黑|0x40000000|628\n狠|0x8000000|5760\n恨|0x1000|2846\n烘|0x1000|17075\n红|0x40000000|537\n后|0x2000000|43\n乎|0x0200|0\n划|0x1000|1961\n坏|0x40000000|1379\n还|0x8000000|550\n换|0x1000|833\n黄|0x40000000|865\n灰|0x100000|2751\n挥|0x1000|3379\n会|0x1000|35\n或|0x10000000|81\n及|0x10000000|780\n即|0x1000|172\n急|0x40000000|1074\n几|0x400000|2502\n己|0x100000|3585\n挤|0x1000|2218\n计|0x1000|1755\n继|0x8000000|0\n加|0x1000|543\n件|0x20000|166\n将|0x8000000|1460\n讲|0x1000|215\n交|0x1000|738\n叫|0x1000|309\n较|0x1000|272\n她|0x10000|55\n仅|0x1000|538\n紧|0x40000000|1415\n尽|0x1000|4931\n近|0x40000000|192\n竟|0x8000000|786\n九|0x400000|533\n久|0x40000000|1260\n旧|0x40000000|681\n就|0x8000000|13\n举|0x1000|1466\n句|0x20000|367\n具|0x1000|1610\n锯|0x100000|12388\n捐|0x100000|5574\n卷|0x20000|3230\n看|0x1000|4382\n考|0x1000|2255\n烤|0x1000|6817\n靠|0x1000|354\n可|0x1000|61\n肯|0x1000|2490\n空|0x40000000|2292\n苦|0x40000000|1231\n夸|0x1000|7755\n块|0x20000|362\n快|0x40000000|210\n困|0x1000|4885\n拉|0x1000|843\n啦|0x0200|1414\n来|0x1000|23\n兰|0x100000|5326\n蓝|0x40000000|3073\n老|0x40000000|132\n擂|0x1000|14715\n冷|0x40000000|1717\n里|0x2000000|53\n连|0x1000|242\n练|0x100000|2935\n炼|0x1000|8304\n两|0x400000|26\n辆|0x20000|793\n量|0x100000|734\n了|0x2000|590\n列|0x1000|1906\nLine truncated
|
||||
n.CHS_NAMES=T,n.ChsNameOptimizer=Z,n.ChsNameTokenizer=k,n.DatetimeOptimizer=$,n.DictOptimizer=O,n.DictTokenizer=en,n.EmailOptimizer=P,n.ForeignTokenizer=cn,n.Module=d,n.Optimizer=A,n.POSTAG=s,n.PunctuationTokenizer=j,n.Segment=D,n.SingleTokenizer=fn,n.Tokenizer=_,n.URLTokenizer=Y,n.WildcardTokenizer=tn,n.cnPOSTag=l,n.dicts=hn,n.enPOSTag=h,n.getPOSTagTranslator=p,n.modules=rn,n.names=un,n.pangu=an,n.panguExtend1=on,n.panguExtend2=sn,n.stopword=ln,n.stopwords=wn,n.synonym=dn,n.synonyms=bn,n.useDefault=function(n){return n.use(rn),n.loadDict(hn),n.loadSynonymDict(bn),n.loadStopwordDict(wn),n},n.wildcard=pn,Object.defineProperty(n,"__esModule",{value:!0})})
|
||||
//# sourceMappingURL=segmentit.js.map
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"segmentit.js","sources":["../../src/POSTAG.js","../../src/module/BaseModule.js","../../src/Tokenizer.js","../../src/Optimizer.js","../../src/Segment.js","../../src/module/COLORS.js","../../src/module/AdjectiveOptimizer.js","../../src/module/CHS_NAMES.js","../../src/module/ChsNameTokenizer.js","../../src/module/DictOptimizer.js","../../src/module/EmailOptimizer.js","../../src/module/PunctuationTokenizer.js","../../src/module/URLTokenizer.js","../../src/module/ChsNameOptimizer.js","../../src/module/DatetimeOptimizer.js","../../src/module/DictTokenizer.js","../../src/module/ForeignTokenizer.js","../../src/module/SingleTokenizer.js","../../src/module/WildcardTokenizer.js","../../src/module/index.js","../../src/knowledge/index.js","../../src/index.js"],"sourcesContent":["// @flow\n\n/**\n * 单词类型\n */\n\nconst POSTAG = {\n D_A: 0x40000000, // 形容词 形语素\n D_B: 0x20000000, // 区别词 区别语素\n D_C: 0x10000000, // 连词 连语素\n D_D: 0x08000000, // 副词 副语素\n D_E: 0x04000000, // 叹词 叹语素\n D_F: 0x02000000, // 方位词 方位语素\n D_I: 0x01000000, // 成语\n D_L: 0x00800000, // 习语\n A_M: 0x00400000, // 数词 数语素\n D_MQ: 0x00200000, // 数量词\n D_N: 0x00100000, // 名词 名语素\n D_O: 0x00080000, // 拟声词\n D_P: 0x00040000, // 介词\n A_Q: 0x00020000, // 量词 量语素\n D_R: 0x00010000, // 代词 代语素\n D_S: 0x00008000, // 处所词\n D_T: 0x00004000, // 时间词\n D_U: 0x00002000, // 助词 助语素\n D_V: 0x00001000, // 动词 动语素\n D_W: 0x00000800, // 标点符号\n D_X: 0x00000400, // 非语素字\n D_Y: 0x00000200, // 语气词 语气语素\n D_Z: 0x00000100, // 状态词\n A_NR: 0x00000080, // 人名\n A_NS: 0x00000040, // 地名\n A_NT: 0x00000020, // 机构团体\n A_NX: 0x00000010, // 外文字符\n A_NZ: 0x00000008, // 其他专名\n D_ZH: 0x00000004, // 前接成分\n D_K: 0x00000002, // 后接成分\n UNK: 0x00000000, // 未知词性\n URL: 0x00000001, // 网址、邮箱地址\n};\n\nconst CN_POS_NAMES = {\n D_A: '形容词 形语素',\n D_B: '区别词 区别语素',\n D_C: '连词 连语素',\n D_D: '副词 副语素',\n D_E: '叹词 叹语素',\n D_F: '方位词 方位语素',\n D_I: '成语',\n D_L: '习语',\n A_M: '数词 数语素',\n D_MQ: '数量词',\n D_N: '名词 名语素',\n D_O: '拟声词',\n D_P: '介词',\n A_Q: '量词 量语素',\n D_R: '代词 代语素',\n D_S: '处所词',\n D_T: '时间词',\n D_U: '助词 助语素',\n D_V: '动词 动语素',\n D_W: '标点符号',\n D_X: '非语素字',\n D_Y: '语气词 语气语素',\n D_Z: '状态词',\n A_NR: '人名',\n A_NS: '地名',\n A_NT: '机构团体',\n A_NX: '外文字符',\n A_NZ: '其他专名',\n D_ZH: '前接成分',\n D_K: '后接成分',\n UNK: '未知',\n URL: '网址 邮箱地址',\n};\n\nconst EN_POS_NAMES = {\n D_A: 'a',\n D_B: 'b',\n D_C: 'c',\n D_D: 'd',\n D_E: 'e',\n D_F: 'f',\n D_I: 'i',\n D_L: 'l',\n A_M: 'm',\n D_MQ: 'mq',\n D_N: 'n',\n D_O: 'o',\n D_P: 'p',\n A_Q: 'q',\n D_R: 'r',\n D_S: 's',\n D_T: 't',\n D_U: 'u',\n D_V: 'v',\n D_W: 'w',\n D_X: 'x',\n D_Y: 'y',\n D_Z: 'z',\n A_NR: 'nr',\n A_NS: 'ns',\n A_NT: 'nt',\n A_NX: 'nx',\n A_NZ: 'nz',\n D_ZH: 'h',\n D_K: 'k',\n UNK: 'un',\n URL: 'uri',\n};\n\nexport function getPOSTagTranslator(POSTagDict: Object, I18NDict: Object): string {\n return (posTagNumber: number): string => {\n if (isNaN(posTagNumber)) {\n return I18NDict[posTagNumber] || I18NDict.UNK;\n }\n const result = [];\n for (const key in POSTagDict) {\n if ((posTagNumber & POSTagDict[key]) > 0) {\n result.push(I18NDict[key]);\n }\n }\n if (result.length < 1) {\n return I18NDict.UNK;\n }\n return result.toString();\n };\n}\n\nexport const cnPOSTag = getPOSTagTranslator(POSTAG, CN_POS_NAMES);\nexport const enPOSTag = getPOSTagTranslator(POSTAG, EN_POS_NAMES);\n\nexport default POSTAG;\n","// @flow\nimport Segment from '../Segment';\n\nexport class Module {\n type: 'optimizer' | 'tokenizer';\n\n constructor(segment: Segment) {\n this.segment = segment;\n }\n}\n\nexport class Tokenizer extends Module {\n type = 'tokenizer';\n}\n\nexport class Optimizer extends Module {\n type = 'optimizer';\n}\n","// @flow\nimport type { SegmentToken } from './module/type';\n\nexport default class Tokenizer {\n /**\n * 分词模块管理器\n *\n * @param {Segment} 分词接口\n */\n constructor(segment) {\n this.segment = segment;\n }\n\n /**\n * 对一段文本进行分词\n *\n * @param {string} text 文本\n * @param {array} modules 分词模块数组\n * @return {array}\n */\n split(text, modules): SegmentToken[] {\n if (modules.length < 1) {\n throw Error('No tokenizer module!');\n }\n // 按顺序分别调用各个module来进行分词 ： 各个module仅对没有识别类型的单词进行分词\n let result = Line truncated
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
{
|
||||
"name": "segmentit",
|
||||
"main": "dist/cjs",
|
||||
"module": "dist/esm/segmentit.js",
|
||||
"browser": "dist/umd/segmentit.js",
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"version": "2.0.3",
|
||||
"description": "Chinese word segmentation 中文分词模块 with browser && electron support",
|
||||
"keywords": [
|
||||
"segment",
|
||||
"chinese",
|
||||
"POS",
|
||||
"mlp",
|
||||
"中文",
|
||||
"electron",
|
||||
"浏览器",
|
||||
"browser",
|
||||
"nodejs",
|
||||
"分词"
|
||||
],
|
||||
"homepage": "https://github.com/linonetwo/segmentit#readme",
|
||||
"bugs": {
|
||||
"url": "https://github.com/linonetwo/segmentit/issues"
|
||||
},
|
||||
"scripts": {
|
||||
"clean": "rimraf dist & rimraf build & rimraf coverage",
|
||||
"flow-typed": "rimraf flow-typed/npm && flow-typed install --overwrite || true",
|
||||
"lint": "eslint .",
|
||||
"lintfix": "eslint . --fix",
|
||||
"postlint": "npm run typecheck",
|
||||
"typecheck": "flow check",
|
||||
"test": "npm run test-only",
|
||||
"test-only": "jest --coverage",
|
||||
"test:watch": "jest --watch",
|
||||
"prepare": "npm run build",
|
||||
"prepublishOnly": "npm run test",
|
||||
"build": "NODE_ENV=production && npm run clean && rollup -c",
|
||||
"build:watch": "babel --watch src -d dist"
|
||||
},
|
||||
"authors": [
|
||||
"Lei Zongmin <[email protected]>",
|
||||
"Linonetwo <[email protected]> (https://onetwo.ren/)"
|
||||
],
|
||||
"contributors": [
|
||||
{
|
||||
"name": "Lei Zongmin",
|
||||
"email": "[email protected]"
|
||||
},
|
||||
{
|
||||
"name": "Linonetwo",
|
||||
"email": "[email protected]"
|
||||
}
|
||||
],
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git://github.com/linonetwo/segmentit.git"
|
||||
},
|
||||
"browserslist": "> 0.25%, not dead",
|
||||
"jest": {
|
||||
"testEnvironment": "node",
|
||||
"moduleFileExtensions": [
|
||||
"js"
|
||||
],
|
||||
"watchman": false,
|
||||
"coverageDirectory": "coverage",
|
||||
"collectCoverageFrom": [
|
||||
"src/**/*.js"
|
||||
]
|
||||
},
|
||||
"dependencies": {
|
||||
"preval.macro": "^4.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/cli": "^7.7.7",
|
||||
"@babel/core": "^7.7.7",
|
||||
"@babel/plugin-proposal-class-properties": "^7.7.4",
|
||||
"@babel/plugin-transform-runtime": "^7.7.6",
|
||||
"@babel/preset-env": "^7.7.7",
|
||||
"@babel/preset-flow": "^7.7.4",
|
||||
"babel-eslint": "^10.0.3",
|
||||
"babel-jest": "^24.9.0",
|
||||
"babel-plugin-macros": "^2.8.0",
|
||||
"eslint": "^6.8.0",
|
||||
"eslint-config-airbnb": "^18.0.1",
|
||||
"eslint-plugin-compat": "^3.3.0",
|
||||
"eslint-plugin-flowtype": "~4.5.2",
|
||||
"eslint-plugin-import": "^2.19.1",
|
||||
"eslint-plugin-jsx-a11y": "^6.2.3",
|
||||
"eslint-plugin-promise": "^4.2.1",
|
||||
"eslint-plugin-react": "^7.17.0",
|
||||
"flow-bin": "^0.114.0",
|
||||
"flow-typed": "^2.6.2",
|
||||
"jest": "^24.9.0",
|
||||
"mocha": "^6.2.2",
|
||||
"rimraf": "^3.0.0",
|
||||
"rollup": "^1.27.13",
|
||||
"rollup-plugin-babel": "^4.3.3",
|
||||
"rollup-plugin-commonjs": "^10.1.0",
|
||||
"rollup-plugin-replace": "^2.2.0",
|
||||
"rollup-plugin-uglify": "^6.0.4",
|
||||
"should": "^13.2.3",
|
||||
"uglify-es": "^3.3.9"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user