Jekyll2023-09-09T08:57:56+00:00https://www.chungkwong.cc/feed.xml陈颂光全栈工程师,能够独立开发从解释器到网站和桌面/移动端应用的各类软件。陈颂光从CROHME 2023看手写数学公式识别在过去四年的发展2023-07-03T00:00:00+00:002023-07-03T00:00:00+00:00https://www.chungkwong.cc/crohme<p><a href="https://crohme2023.ltu-ai.dev/">联机手写数学公式识别竞赛</a>(Competition on Recognition of Handwritten Mathematical Expressions,CROHME)一直是公式识别领域认受性最高的评测平台。<a href="http://icdar2023.org/">作为文档分析与识别国际会议</a>(International Conference on Document Analysis and Recognition,ICDAR)的一部分,2023年这个竞赛继2011-2014、2016和2019年后再度举办。作为本届的其中一个参赛者,我有幸在全部三个任务上夺冠,现在是时候总结一下本次参赛的经验,并展望手写公式识别领域的前景。</p>
<h2 id="竞赛结果">竞赛结果</h2>
<p>竞赛结果已经在<a href="https://crohme2023.ltu-ai.dev/results/">官网</a>公布。各参赛队伍的成绩如下:</p>
<p><img src="image/results_crohme_2023.png" alt="CROHME 2023测试集上各参赛队伍的表现" /></p>
<p>为了与上一届竞赛各参赛队伍和近年发现论文中的结果对比,这里也列出各参赛队伍在上一届测试集上的成绩:</p>
<p><img src="image/results_crohme_2019.png" alt="CROHME 2019测试集上各参赛队伍的表现" /></p>
<h2 id="联机手写识别与脱机手写识别相比仍然有显著优势">联机手写识别与脱机手写识别相比仍然有显著优势</h2>
<p>在CROHME 2023中联机任务的最高准确率达到82.34%,而脱机任务的最高准确率却只有70.81%,差距超过11.5%。可见,联机手写识别与脱机手写识别相比仍然有非常显著的优势。不过,双模态任务的最高准确率达到84.12%,说明现有分别针对两个模态的方法仍然有互补性,但要利用好也不是直截了当的,亚军队伍就似乎在双模态任务上完全放弃脱机输入,使得它在联机任务和双模态任务上准确率完全相同。更重要的是,联机手写公式识别的准确率在过去四年间仍然在攀升,在CROHME 2019测试集上,最高准确率从80.73%大幅提升到88.24%,本届竞赛的前两名的表现都明显优于上届前三名共同的水平。相反,脱机手写公式识别的最高准确率只从77.15%轻微提升到77.83%。这说明,脱机手写识别可能已经到达一个瓶颈,不容易再取得突破。可惜的是,上届前三名科大讯飞、三星和MyScript都没有参加本届竞赛,未能知晓他们的近况。</p>
<p>与脱机手写识别(光学字符识别,OCR)相比,联机手写识别有时间维度的额外信息可用,而且排除了大部分的背景噪声。在CROHME 2019中,图片都是根据轨迹简单地渲染出来的,背景是干净的纯白而前景是干净的纯黑,在这个条件下,联机与脱机识别的准确率差距还不太明显。在该竞赛中,在两项任务均夺冠的科大讯飞的系统在联机和脱机任务的识别率分别为80.73%和77.15%,差距不到4个百分点。事实上,只要设计一个基于规则的笔画提取算法,并用提取出的笔画去训练一个联机手写识别系统,就能轻易得到一个准确率超过75%的脱机手写识别系统。即使直接调用现成的联机手写识别系统,也可以得到65.22%的准确率。在2020年的<a href="http://offrashme.xmutpr.com/">脱机手写数学公式识别与定位竞赛</a>(Competition on Offline Recognition and Spotting of Handwritten Mathematical Expressions,OffRaSHME) 竞赛中,图片背景仍然相当干净,但前景的笔画已经有粗细和浓淡等变化,这时科大讯飞的脱机手写数学公式识别系统已经达到79.85%的准确率,而基于笔画提取的方法只有61.35%的准确率(后面可以改进到65%左右)。到了CROHME 2023,背景也不再是纯色的,同时带有一些噪声,这时基于笔画提取方法的预估准确率已经掉到45%左右。按照这个趋势,当面对通过拍照获得的图片时,脱机手写识别的准确率还会进一步下降,而且还会面临公式检测与定位的额外困难。这表明,联机手写识别的准确率优势主要源于较低水平的背景噪声,时间信息反而是次要的。</p>
<h2 id="本届新增样本更具挑战性">本届新增样本更具挑战性</h2>
<p>近年,在脱机手写识别方面,出现了HME100k和MLHME38k等更真实和具挑战性的数据集。本届CROHME的数据集也同样变得更有挑战性了,反映在各参赛系统在2023年的测试集的准确率明显低于它们在2019测试集上的表现。</p>
<p>每一届CROHME竞赛都会扩充前一届的训练集的并提供全新的测试集。与CROHME 2019相比,联机手写公式训练集增加了:</p>
<ul>
<li>来自CROHME 2014测试集(同时是CROHME 2019的检验集)的986条公式。</li>
<li>由东京农工大学合成约15万条公式。这些公式大幅增加了公式的多样性,但不少样本看起来很奇怪,例如许多减号非常宽(估计是由于按符号高度做规范化造成的)。</li>
<li>由南特大学和东京农工大学1045条新收集的联机手写公式。</li>
</ul>
<p>检验集则由CROHME 2016测试集的1147条公式和555条新收集公式组成。测试集由2354条新收集公式组成。</p>
<p>从上一届首次设立脱机识别任务时,全部图片都是通过直接渲染轨迹得到的,本届则增加了一些更真实的扫描数据。脱机手写公式训练集:</p>
<ul>
<li>OffRaSHME训练集中的10,000条公式。</li>
<li>1604条扫描的公式。与之前的样本相比,这些图片是彩色的且背景带有明显的噪点/线。</li>
<li>1045条新收集的与联机手写公式配对的扫描公式。这些图片是彩色的且背景不再全是白色的。</li>
</ul>
<p>检验集则由CROHME 2016测试集的1147条公式和555条新收集公式组成。测试集由2354条新收集公式组成。</p>
<p>不过,这些新增样本似乎是先有LaTeX标注再转换为LG和SymLG等格式的,一度造成与过往的数据集的众多不一致性,而这些不一致性足以大幅影响评价指标(超过10个百分点)。</p>
<ul>
<li>LaTeX到LG/SymLG的工具会把多位数看作单个符号并对部分根式产生多余的Inside关系。在提醒主办方后,他们迅速地修正了该问题,但Inside关系的问题回归了至少两次,有一次甚至出现在开放提交结果后。</li>
<li>未能区分subscript和underscript。在提醒主办方后,他们决定手动重新标注数据,最终导致正式竞赛日期推迟了约半个月。</li>
<li>一些INKML中小于号没有转义,导致不能使用的XML解析器解析。虽然在提醒主办方后,他们修正了该问题,但这类问题揭露了他们企图通过拼接方式来生成XML,而正确的方式应该是使用成熟的XML库而应该避免把XML当作字符串来处理。</li>
<li>新增样本不再提供笔画与符号间可靠的对应关系。过往INKML文件的MathML标注容许笔画级引用对应的符号,但新收集的样本不再提供MathML标注。部分合成的样本在INKML中提供了traceGroup标签,但无效笔画ID引用、未被引用的笔画ID和其它标注错误层出不穷,而且在同一符号在公式中出现多次时也无法区分它们。LG文件也存在大量无效引用和非树结构。原则上,通过规则合成的公式理应可以自动生成准确的对应关系,但可惜主办方最终决定移除合成样本的所有traceGroup标注,并任由LG文件继续出错。</li>
<li>1604条扫描的公式中大约10%与INKML文件对不上,但主办方决定抛弃有关样本而未有修正对应。
虽然在我向主办方指出了这些明显的问题,他们均积极处理,最终基本上确保了训练集、检验集和测试集标注的一致性,但这反映了当前这个领域的一个通病,就是对数据缺乏敏感性,无论是竞赛组织者还是其它参赛者都没有仔细检查数据。</li>
</ul>
<p>在联机识别任务上,亚军<a href="http://www.cvte.com/news/index_detail_353.html">视源中央研究院</a>在CROHME 2019测试集上的准确率只比我们低不到4个百分点,但在CROHME 2023测试集上的准确率却比我们低近10个百分点,有点怀疑他们在某种程度上把CROHME 2019当作了检验集,导致过拟合了该数据集。在某种意义下,在CROHME 2023测试集上,笔顺的相对重要性更高。在脱机识别任务上,许多数学公式都只占新收集的图片中的一小部分,在没有公式检测的情况下,多余的背景区域可能会影响某些算法。</p>
<h2 id="相关研究文献没有的充分重视数据增广">相关研究文献没有的充分重视数据增广</h2>
<p>过去四年间,端到端可训练的编码器-解码器神经网络已经成为识别公式的绝对主流方法。文献中提出了各式各样的方法去提高准确率:</p>
<ul>
<li>修改网络结构。
<ul>
<li>使用Transformer的解码器。堆叠带多头注意力机制的层可以更好地发现长距离依赖关系,不过计算量一般会较大,而且更容易过度拟合。</li>
<li>使用树解码器。由于数学公式可以自然地被看作树,这种方法企图让神经网络更直接地输出树状结构,例如每一步预测一个节点的类型、它的父节点和两个节点间的关系。</li>
<li>使用图编码器。这种方法的基本思想是把每条笔画看作图中的一个顶点,而把有可能属于同一或有关系符号的笔画对应的顶点用边连接,于是就能用图神经网络去提取顶点和边的特征。不过,这种建模在存在符号间连笔时并不合理,而且难以改造成能识别真实场景的图片。</li>
</ul>
</li>
<li>使用辅助的损失函数。
<ul>
<li>引导采样点/像素与输出符号间的对应。这方法可以使注意力机制更精确,但要求训练数据标注各个符号的位置。</li>
<li>引导各符号的存在性甚至出现次数。这方法也可以让编码器提取出能更直接地用于分类的信息,从而加速收敛。</li>
</ul>
</li>
<li>增加语言模型。
<ul>
<li>生成式语言模型。它们的常见的用法是在解码过程中结合识别模型和语言模型分别给出的置信度,或者用语言模型对多个候选公式分别打分(例如语言困惑度)再重新排序。不过,自回归的解码器本身就隐含地包含了语言模型,外加额外语言模型的效果似乎并不明显。
<ul>
<li>使用基于n-gram的语言模型。</li>
<li>使用基于RNN的语言模型。</li>
<li>使用基于Transformer的语言模型。</li>
</ul>
</li>
<li>形式语言模型。在解码过程中可以确保输出严格符合人手设计的上下文无关语法,例如解析LL(1)语法的语法分析就可以与序列到序列模型的集束搜索解码过程无缝结合。这类语言模型可以防止出现不合法的识别结果(例如序列到序列模型不时会产生花括号不匹配的LaTeX代码),方便在运行期定制识别范围,但对提高总体识别率的作用也不大。</li>
</ul>
</li>
<li>数据增广。
<ul>
<li>局部和整体的几何形变。这种方法可以提高模型对输入的鲁棒性,但未能生成整体结构不同的公式。</li>
<li>分解。这种方法可以改善模型对递归结构的理解,但子公式的数量和形式还是有比较大的局限性。</li>
<li>随机替换手写样本中符号为其它符号。这种做法可以加强模型区分不同符号的能力,但未能生成整体结构不同的公式,同时会弱化语言模型。</li>
<li>利用印刷体样本。其想法是利用对比学习或对抗生成网络的思想来引导神经网络对配对的印刷体和手写体样本提取出相近的特征,从而让网络得益于容易自动生成大量印刷体图片。这方法可以用于打造同时适用于印刷体和脱机手写体的公式识别系统,但对联机手写识别来说不够直接。</li>
</ul>
</li>
</ul>
<p>在我看来大部分公式识别领域的论文都侧重于小样本泛化能力,但通过数据增广来扩大训练集似乎是更可取的方向。同时文献中的方法缺乏正交性,十多种方法都声称可以把公式识别的准确率从45%提高到55%,但我自己的发现是结合多种技巧往往不能产生明显的叠加效果。</p>
<p>我的经验是,基于语料合成相应的轨迹/图片是提高公式识别准确率的最有效方法,这种方法不但可以让模型的训练期间接触到更多内容不同的公式,而且能加强隐含的语言模型(通过使用随机生成的语料也可以反过来用来削弱语言模型),并且适用于各种网络结构,又不会增加推理时的计算量或部署难度。粗暴的合成工具和公开的数据集足以把单模型准确率从50%推进到80%。在生产环境中,这方法有助于扩展支持的数学符号和结构清单。相对而言,网络结构反而似乎不是关键。</p>
<h2 id="从手写识别技术到成功的产品仍然有一段距离">从手写识别技术到成功的产品仍然有一段距离</h2>
<p>竞赛固然能够在相对公平的条件下对比不同队伍的技术实力,但竞赛与产品有着不尽相同的要求,让公式识别对大众实用才能体现其价值。竞赛只关注准确性,但部署(特别是端侧部署)时对速度、内存占用和模型体积等方面是有要求的。在竞赛中,我们可以通过集成多个较大且非量化的模型来用资源占用换准确率;在产品中,我们需要在两者之间选取一个平衡点。此外,CROHME竞赛只涉及不到一百个符号,而且没有矩阵和帽子等常用的结构,数学公式识别系统产品往往还需要支持更多的符号和结构,其中不少是容易与其它符号混淆的。</p>
<p>由于公式识别不太可能做到完全准确,一个协助用户在短时间内发现并修正识别错误机制是重要的。除了可以用常规的公式编辑器编辑识别结果外,还可以利用识别系统提供的信息,比如说,许多数学公式识别系统都能给出公式级和字符级的候选。一个前一候选正确率为80%左右的识别系统的前四候选正确率很可能接近于90%左右,这说明只要向用户提供四个候选,用户就能一键修正大约一半识别错误的公式。如果容许用户通过长按/右键/手势之类的方式把公式中的字符改成其它候选字符的话,能修正的错误比例还会更高。此外,通过用不同颜色显示置信度偏低的公式或符号,也可能可协助用户更有效地发现识别错误。</p>
<p>不少应用场景下不仅要知道用户写的公式是什么,还需要知道各个符号的位置以支持一些编辑和搜索功能。传统的数学公式识别系统一般会生成大量可能可把公式切割成符号集合的切分方式,然后结合符号分类器、关系分类器和语言模型去选取最佳的识别结果,因而一般能比较可靠地给出输入笔画与输出符号间的对应关系。虽然近年端到端可训练的神经网络极大地降低了做公式识别的门槛,只要标注足够的样本就能开发出一个看起来还行的系统,不需要具备扎实的计算机理论知识和编程功底,但这类方法不能显式生成对应关系。虽然也有人尝试让端到端系统生成对应关系,例如我们的系统在预测一个符号时也会预测各采样点属于它的可能性,并且取得了一定的效果,在超过90%识别对的公式中符号与笔画的对应关系也全对,但输出的对应关系仍然不像传统方法那样可靠。</p>
<p>数学公式难以输入的问题长期以来为人诟病,期望手写有一天能成为解决这个问题的工具之一。</p>陈颂光联机手写数学公式识别竞赛(Competition on Recognition of Handwritten Mathematical Expressions,CROHME)一直是公式识别领域认受性最高的评测平台。作为文档分析与识别国际会议(International Conference on Document Analysis and Recognition,ICDAR)的一部分,2023年这个竞赛继2011-2014、2016和2019年后再度举办。作为本届的其中一个参赛者,我有幸在全部三个任务上夺冠,现在是时候总结一下本次参赛的经验,并展望手写公式识别领域的前景。联机手写文档分析的现状与未来2022-10-06T00:00:00+00:002022-10-06T00:00:00+00:00https://www.chungkwong.cc/online-handwriting<p>随着触摸屏和笔式输入设备的普及,人们可以用电子设备代替传统的纸张作为书写的介质,从而更便于共享。不过,仅仅记录书写的轨迹并未充分体现电子化的优势,为了更好地支持编辑、搜索和各种自动化处理,还需要把手写的内容转换为更结构化的形式。与通过扫描或拍照等方式获取的手写文档图片相比,轨迹不但提供了时间、坐标和其它信息(如压感、倾斜角),而且夹杂了更少的的背景噪声,这就使得联机手写文档分析有比脱机手写文档分析做得更好的可能性。然而,做到完美仍然不太可能,所以如何利用不准确的结果就成为实用化的关键。</p>
<h2 id="文档分析的任务">文档分析的任务</h2>
<p>传统的联机手写文档分析一般先把笔画分成若干个对象并判断它们的类型,然后再分别识别它们的详细内容,前者被称为版面分析而后者被称为手写识别。这里所说的典型对象包括文本行、公式和绘画等等。</p>
<p>以下是一些现成的SDK:</p>
<ul>
<li>MyScript的<a href="https://developer.myscript.com/">Interactive ink</a>支持约70种自然语言的文本,另外支持数学公式识别和图表识别。</li>
<li>Google的<a href="https://developers.google.cn/ml-kit/vision/digital-ink-recognition">ML Kit</a>支持超过300种语言的文本,另外支持形状识别。</li>
<li>Microsoft的<a href="https://learn.microsoft.com/en-us/windows/win32/tablet/ink-analysis-overview">Win32 API</a>同时支持版面分析和文本识别。</li>
</ul>
<h2 id="现有的主流技术">现有的主流技术</h2>
<p>固然,我们可以把轨迹渲染为图片再使用脱机文档分析的技术,甚至可以通过把时间等信息嵌入到图像中以避免信息丢失,但这里主要简介联机手写能用的额外技术。</p>
<h3 id="版面分析">版面分析</h3>
<ul>
<li>循环神经网络。假设对象间不存在倒笔的情况(即用户写完一个对象再写下一对象),通过判断每条笔画是否与前一笔画属于同一对象,即可以完成对笔迹的分割。同时,还可以通过判断每条笔画所属对象的类型来得到各对象的类型。利用简单的循环神经网络来提取笔画级特征就能实现这两个目的。在需要支持对象间倒笔时,就需要再加上一个步骤来把被过切分的对象合并回来。这个方法简单高效,但不适合倒笔较多的乱写场景。更多细节参见:
<ul>
<li><a href="https://doi.org/10.1007/978-3-030-86331-9_13">HCRNN: A Novel Architecture for Fast Online Handwritten Stroke Classification</a></li>
<li><a href="https://doi.org/10.1109/ICFHR2020.2020.00058">Spatio-temporal Clustering for Grouping in Online Handwriting Document Layout Analysis with GRU-RNN</a></li>
</ul>
</li>
<li>图神经网络。通过把笔划当作顶点而把在空间或时间上接近的笔画对应的顶点间用边连起,就可以把笔迹看作一个图。于是,把笔迹分割为对象的问题就可以化为对边进行分类的问题(区分两个端点对应的笔画是否属于同一对象),而对对象分类的问题则化为对顶点进行分类的问题(区分相应笔画所属对象的类型)。在这个基础上,还可以得到文档的层次结构,例如可以把分行和分段一起做。这样,图注意力神经网络就可以用上了。这个方法相当一般,文本检测、行分割、表格分析、流程图分析以及数学公式识别等任务都可以用本法处理。更多细节参见:
<ul>
<li><a href="https://doi.org/10.1016/j.patcog.2021.107859">Joint stroke classification and text line grouping in online handwritten documents with edge pooling attention networks</a></li>
<li><a href="https://doi.org/10.1145/3444685.3446295">Table detection and cell segmentation in online handwritten documents with graph attention networks</a></li>
</ul>
</li>
</ul>
<h3 id="识别">识别</h3>
<p>以典型的文字识别为例,我们列出一些标准的方法:</p>
<ul>
<li>切分解码方法。这种方法先生成把行分成字符的多种切分方式,然后对各字符假设进行评分,最后选取最优的切分方式和字符串,参见<a href="https://doi.org/10.1109/TPAMI.2016.2572693">Multi-Language Online Handwriting Recognition</a>。</li>
<li>基于CTC解码的方法。这种方法一般先用双向循环神经网络提取点级别的特征,然后用CTC算法解码,参见<a href="https://doi.org/10.1007/s10032-020-00350-4">Fast multi-language LSTM-based online handwriting recognition</a>。这种方法的识别速度较快,但对输入作出了较强的假设。比如说,它假设输入和输出间有单调的对齐,即写完一个字符再写下一个字符,因此在存在影响识别的字间倒笔时,可能要检测它们并重排,或者另加特征。同时,它不擅长生成格式化标记(如LaTeX代码),故用于识别数学公式时,就只能做字符切分与识别,结构分析要留给其它算法,参见<a href="https://doi.org/10.1007/978-3-030-20518-8_37">Acceleration of Online Recognition of 2D Sequences Using Deep Bidirectional LSTM and Dynamic Programming</a>。</li>
<li>基于自回归解码的方法。编码器一般仍然是双向循环神经网络,解码器则一般带有注意力机制以优化对非单调的对齐的支持,并加上某种某种收敛机制以抑制多识别和漏识别的情况。这种方法通用性较强而且可以学习语言模型,但由于要多次调用解码器以逐个输出符号或树结点,识别速度相对较慢。
<ul>
<li>序列解码器。参见<a href="https://doi.org/10.1109/TMM.2018.2844689">Track, Attend, and Parse (TAP): An End-to-End Framework for Online Handwritten Mathematical Expression Recognition</a>。</li>
<li>树解码器。参见<a href="https://doi.org/10.1109/tmm.2020.3011316">SRD: A Tree Structure Based Decoder for Online Handwritten Mathematical Expression Recognition</a>。</li>
</ul>
</li>
</ul>
<p>以公式识别为例,我们列出调参以外的一些可能可提高准确率的策略:</p>
<ul>
<li>数据增强。为了提高识别系统的泛化能力,可以通过合成数据来提高训练样本的多样性,从而抑制过度拟合。
<ul>
<li>取子公式和进行随机仿射变换。参见<a href="https://doi.org/10.1016/j.patrec.2021.12.002">Syntactic data generation for handwritten mathematical expression recognition</a>。</li>
<li>通过笔画提取利用图片形式的样本。参见<a href="https://doi.org/10.1109/ACCESS.2020.2984627">Stroke Extraction for Offline Handwritten Mathematical Expression Recognition</a>。</li>
<li>设计某种按LaTeX或MathML代码合成相应手写样本的工具,然后基于能批量爬取的语料(也可以混入一些随机生成的)去生成大量样本。这些样本天然地带符号与笔画间的对应关系,可以用于引导注意力系数接近真实的对齐。</li>
</ul>
</li>
<li>多任务学习。通过设计其它与识别任务共用部分参数的任务并与识别任务一起训练,可以引导模型学习到更通用的知识。
<ul>
<li>预测从后到前的序列。参见<a href="https://doi.org/10.1007/978-3-030-86331-9_37">Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer</a>。</li>
<li>预测各符号的出现次数。参见<a href="https://doi.org/10.48550/arXiv.2207.11463">When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition</a>。</li>
</ul>
</li>
<li>限制识别范围。序列解码器有时会输出不合法的输出(如产生花括号不配对的LaTeX代码)或者输出应用场景不允许的结果(如在数字字段输出“lO”而非“10”)。通过约束识别范围可以确保结果合法并区分相似的符号。
<ul>
<li>利用上下文无关语法来限制识别识别。参见<a href="https://doi.org/10.1109/CVPR52688.2022.00451">Syntax-Aware Network for Handwritten Mathematical Expression Recognition</a>。</li>
<li>在语法为LL(1)的情况下(原理上也适用于LL(k)语法),对序列解码过程作轻微的修改即可把预测分析法整合到集束搜索中。</li>
</ul>
</li>
<li>与脱机识别模型或语言模型相结合。联机识别模型更着重于时间信息,脱机识别模型更着重于空间信息,语言模型则着重于上下文信息,它们之间有互补性,故组合模型往往比单个模型更可靠。
<ul>
<li>利用语言困惑度或其它模型对候选公式进行重新排序。</li>
<li>利用n-gram或其它生成模型调整各候选符号的置信度。</li>
</ul>
</li>
</ul>
<h2 id="参考资源">参考资源</h2>
<p>要进入联机手写文档分析领域的话,在公开的数据集上复现前沿论文的工作是一个不错的出发点。</p>
<h3 id="数据集">数据集</h3>
<p>数据集是开发联机手写技术的基本材料,以下是笔者知道的部分联机手写数据集:</p>
<table>
<thead>
<tr>
<th>数据集</th>
<th>来源</th>
<th>内容</th>
<th>主要用途</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="http://www.iapr-tc11.org/mediawiki/index.php/IAM_Online_Document_Database_(IAMonDo-database)">IAMonDo</a></td>
<td>伯恩大学</td>
<td>英文文档</td>
<td>版面分析</td>
</tr>
<tr>
<td><a href="http://web.tuat.ac.jp/~nakagawa/database/en/kondate_about.html">Kondate</a></td>
<td>东京农工大学</td>
<td>日文、泰文和英文文档</td>
<td>版面分析</td>
</tr>
<tr>
<td><a href="http://www.nlpr.ia.ac.cn/databases/CASIA-onDo/index.html">CASIA-onDo</a></td>
<td>中科院</td>
<td>中文和英文文档</td>
<td>版面分析</td>
</tr>
<tr>
<td><a href="https://zenodo.org/record/1195803">Unipen</a></td>
<td>Unipen基金会</td>
<td>英文字符、词和行</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="https://fki.tic.heia-fr.ch/databases/iam-on-line-handwriting-database">IAMonDB</a></td>
<td>伯恩大学</td>
<td>英文文本行</td>
<td>文字识别、手写合成</td>
</tr>
<tr>
<td><a href="https://tc11.cvc.uab.es/datasets/HANDS-VNOnDB2018_1">VNOnDB</a></td>
<td>东京农工大学</td>
<td>越南文词、行和段落</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://www.nlpr.ia.ac.cn/databases/handwriting/Online_database.html">CASIA OLHWDB</a></td>
<td>中科院</td>
<td>汉字字符和中文文本行</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://hcii-lab.net/data/scutcouch/EN/introduction.html">SCUT-COUCH2009</a></td>
<td>华南理工大学</td>
<td>汉字字符、词、拼音和中文文本行</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://hcii-lab.net/data/onHCCTestdataset/onHCCTest.html">SCUT-onHCCTestDB</a></td>
<td>华南理工大学</td>
<td>汉字字符</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://www.iapr-tc11.org/mediawiki/index.php/Harbin_Institute_of_Technology_Opening_Recognition_Corpus_for_Chinese_Characters_(HIT-OR3C)">HIT-OR3C</a></td>
<td>哈工大</td>
<td>汉字字符</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://web.tuat.ac.jp/~nakagawa/database/en/about_nakayosi.html">Nakayosi</a></td>
<td>东京农工大学</td>
<td>日文字符</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://web.tuat.ac.jp/~nakagawa/database/en/about_kuchibue.html">Kuchibue</a></td>
<td>东京农工大学</td>
<td>日文字符</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="https://ieee-dataport.org/open-access/adab-database">ADAB</a></td>
<td>斯法克斯大学</td>
<td>阿拉伯文地名</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="https://www.altec-center.org/filescms/files/">AltecOnDB</a></td>
<td>ALTEC</td>
<td>阿拉伯文</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="http://onlinekhatt.ideas2serve.net/">Online-KHATT</a></td>
<td>混合</td>
<td>阿拉伯文字符和句子</td>
<td>文字识别</td>
</tr>
<tr>
<td><a href="https://github.com/brendenlake/omniglot">Omniglot</a></td>
<td>混合</td>
<td>多语言字符</td>
<td>语言检测</td>
</tr>
<tr>
<td><a href="https://tc11.cvc.uab.es/datasets/ICDAR2019-CROHME-TDF_1">CROHME</a></td>
<td>混合</td>
<td>数学公式</td>
<td>公式识别</td>
</tr>
<tr>
<td><a href="https://github.com/kirel/detexify-data">Detexify</a></td>
<td>Kirel</td>
<td>数学符号</td>
<td>公式识别</td>
</tr>
<tr>
<td><a href="https://zenodo.org/record/50022">HWRT</a></td>
<td>Martin Thoma</td>
<td>数学符号</td>
<td>公式识别</td>
</tr>
<tr>
<td><a href="https://grfia.dlsi.ua.es/homus/">HOMUS</a></td>
<td>阿利坎特大学</td>
<td>音符</td>
<td>音符识别</td>
</tr>
<tr>
<td><a href="https://github.com/googlecreativelab/quickdraw-dataset">Quick draw</a></td>
<td>Google</td>
<td>手绘</td>
<td>形状识别</td>
</tr>
<tr>
<td><a href="http://www.nlpr.ia.ac.cn/databases/CASIA-OHFC/index.html">CASIA OHFC</a></td>
<td>中科院</td>
<td>流程图</td>
<td>图表识别</td>
</tr>
<tr>
<td><a href="https://tc11.cvc.uab.es/datasets/OHFCD_1">OHFCD</a></td>
<td>南特大学</td>
<td>流程图</td>
<td>图表识别</td>
</tr>
<tr>
<td><a href="https://cmp.felk.cvut.cz/~breslmar/flowcharts/">FC</a></td>
<td>捷克技术大学</td>
<td>流程图</td>
<td>图表识别</td>
</tr>
<tr>
<td><a href="https://cmp.felk.cvut.cz/~breslmar/finite_automata/">FA</a></td>
<td>捷克技术大学</td>
<td>有限自动机</td>
<td>图表识别</td>
</tr>
<tr>
<td><a href="http://www.iapr-tc11.org/mediawiki/index.php/ICDAR_2009_Signature_Verification_Competition_(SigComp2009)">SigComp2009</a></td>
<td>DFKI</td>
<td>签名</td>
<td>签名鉴定</td>
</tr>
<tr>
<td><a href="http://www.iapr-tc11.org/mediawiki/index.php/ICDAR_2011_Signature_Verification_Competition_(SigComp2011)">SigComp2011</a></td>
<td>DFKI</td>
<td>中文和荷兰文签名</td>
<td>签名鉴定</td>
</tr>
<tr>
<td><a href="https://tc11.cvc.uab.es/datasets/SigWiComp2013_1">SigWiComp2013</a></td>
<td>DFKI</td>
<td>日文签名</td>
<td>签名鉴定</td>
</tr>
<tr>
<td><a href="https://github.com/BiDAlab/DeepSignDB">DeepSignDB</a></td>
<td>混合</td>
<td>签名</td>
<td>签名鉴定</td>
</tr>
</tbody>
</table>
<h3 id="论文">论文</h3>
<p>如果希望了解本领域的进展,可以跟进本领域比较有影响力的期刊和会议上发表论文。</p>
<table>
<thead>
<tr>
<th>会议</th>
<th>年份</th>
</tr>
</thead>
<tbody>
<tr>
<td>International Conference on Document Analysis and Recognition (ICDAR)</td>
<td>单数年份</td>
</tr>
<tr>
<td>International Conference on Frontiers of Handwriting Recognition (ICFHR)</td>
<td>双数年份</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>期刊</th>
<th>类型</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://www.springer.com/journal/10032">International Journal on Document Analysis and Recognition (IJDAR)</a></td>
<td>季刊</td>
</tr>
<tr>
<td><a href="https://www.sciencedirect.com/journal/pattern-recognition">Pattern Recognition</a></td>
<td>月刊</td>
</tr>
<tr>
<td><a href="https://www.sciencedirect.com/journal/pattern-recognition-letters">Pattern Recognition Letters</a></td>
<td>月刊</td>
</tr>
</tbody>
</table>
<p>另外,可以重点关注在本领域有影响力的团队发表的论文,以及它们的参考文献与引用它们的论文。</p>
<table>
<thead>
<tr>
<th>团队带头人</th>
<th>单位</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://people.ucas.ac.cn/~liuchenglin">Cheng-Lin Liu (刘成林)</a></td>
<td>中科院</td>
</tr>
<tr>
<td><a href="http://www.hcii-lab.net/lianwen/">Lianwen Jin (金连文)</a></td>
<td>华南理工大学</td>
</tr>
<tr>
<td><a href="http://staff.ustc.edu.cn/~jundu/">Jun Du</a></td>
<td>中国科学技术大学</td>
</tr>
<tr>
<td><a href="http://pagesperso.ls2n.fr/~mouchere-h/index.php?lg=en">Harold Mouchère</a></td>
<td>南特大学</td>
</tr>
<tr>
<td><a href="http://web.tuat.ac.jp/~nakagawa/en/nakagawa.html">Masaki Nakagawa</a></td>
<td>东京农工大学</td>
</tr>
<tr>
<td><a href="https://www.cs.rit.edu/~rlaz/">Richard Zanibbi</a></td>
<td>罗切斯特理工学院</td>
</tr>
<tr>
<td>Olga Radyvonenko</td>
<td>三星研发部(乌克兰)</td>
</tr>
</tbody>
</table>
<p>最后,也可以去预印本网站如<a href="https://arxiv.org/">ArXiv</a>或论文搜索引擎如<a href="https://www.semanticscholar.org/">Semantic scholar</a>搜索有关关键词。</p>
<h2 id="未来的可能方向">未来的可能方向</h2>
<p>纵使未来无法预知,但凭经验来说,笔者认为以下方向有较大的发展空间:</p>
<ol>
<li>交互设计。在文档分析系统不容易取得突破的前提下,良好的交互设计正是提高用户体验的关键。呈现和修正文档分析结果的方式有很多可以探索的地方,与键盘和语音等其它输入方式的结合有时也可以弥补手写的弱点。</li>
<li>结果的自动化验证。假如可以较准确地判断结果是否正确并向用户提示可能的错误位置,将可以大幅减低用户校对所需的时间。即使现在的文档分析系统往往能给出置信度,但它们通常并不能准确地反映结果正确的概率(例如不意味着100个置信度99%的结果中大约有99个正确),还需要进行校准或利用其它算法估算。</li>
<li>错误感知的应用。在一些场景下,应用可能应该尝试利用不完全准确识别结果。例如,检索系统可以利用识别候选或识别系统的混淆矩阵做查询扩展,从而提高召回率。又如,在电话号码、证件号和邮箱等输入框,可以把结果限制/修正到相应范围。</li>
<li><a href="https://doi.org/10.1109/SMC52423.2021.9658867">无标注笔迹的利用</a>。云端存储、分析与识别服务可以收集到大量笔迹,而人手对它们进行标注成本高昂且难以保障用户的隐私,因此如何自动化地利用这些笔迹来优化文档分析系统就是一个重要的课题。半监督学习和联邦学习中的技巧也许可以派上用场。</li>
<li>少样本定制。在只有少量样本或先验知识的情况下支持一个字符或者一种版面有利于提高文档分析系统的可定制性。这可能可以通过设计或学习字符或版面的抽象特征来实现。</li>
<li>用户习惯自适应。每个用户的书写风格和惯用语都不同,通过逐渐学习用户的特点,可以让软件越用越好用。</li>
<li>其它手写对象的识别。设计通用的框架去识别乐谱、化学结构式和各种工程图等等可扩展手写的应用范围。</li>
</ol>陈颂光随着触摸屏和笔式输入设备的普及,人们可以用电子设备代替传统的纸张作为书写的介质,从而更便于共享。不过,仅仅记录书写的轨迹并未充分体现电子化的优势,为了更好地支持编辑、搜索和各种自动化处理,还需要把手写的内容转换为更结构化的形式。与通过扫描或拍照等方式获取的手写文档图片相比,轨迹不但提供了时间、坐标和其它信息(如压感、倾斜角),而且夹杂了更少的的背景噪声,这就使得联机手写文档分析有比脱机手写文档分析做得更好的可能性。然而,做到完美仍然不太可能,所以如何利用不准确的结果就成为实用化的关键。基于笔划提取的脱机手写数学公式识别2021-04-25T00:00:00+00:002021-04-25T00:00:00+00:00https://www.chungkwong.cc/stroke-extraction<p>手写体识别传统上被分为联机识别和脱机识别,前者识别动态的笔迹而后者识别静态的图片。与联机识别相比,由于缺乏动态信息和存在背景噪声,脱机识别的准确度通常较低,同时往往占用更多资源。于是,把脱机识别问题转化为联机识别问题是一个有吸引力的想法。本文以由于具有紧凑的二维结构而极具挑战性的数学公式识别问题为例,探讨了基于笔划提取的脱机手写识别的可行性和实用性,这方法夺得了国际性权威竞赛CROHME 2019中有关任务的季军。</p>
<h2 id="为什么要识别数学公式">为什么要识别数学公式</h2>
<p>数学公式是在各种手写和印刷文档中常见的信息载体,如何充分地利用数学公式中的价值是一个有意义的问题。由于紧凑的二维结构有助于简明地表达一些概念,数学公式在经管、科学、工程和其它领域中都十分重要。为了提高数学公式的可发现性,把数学公式电子化是很有必要的,因为这有助于长时间保存和传播。进一步把数学公式结构化将容许有效的数学公式检索 ,从而使问答系统、抄袭检测和其它自动化任务成为可能,它们在教育、科研和其它方面都有很多潜在的应用。</p>
<p>与普通文本相比,数学公式更不方便于输入。传统的输入设备如键盘是为单方向的字符串而设计的,虽然符号间的平面位置关系可以用 TeX 或 MathML 之类的代码表示,但用计算机语言表达数学公式对新手而言非常不友好,因为他们不仅要另外学习一门语言和记忆一系列命令,还要与令人困惑的错误打交道。另一方面,用图形化的公式编辑器输入数学公式对老手而言效率太低,因为重复地从工具箱中选取结构元素和符号费时失事。</p>
<p>既然人们已经习惯了在纸上或黑板上书写数学公式,用相同的方式向计算机输入数学公式就是最理想的。识别系统的目标就是把手写的数学公式转化为方便机器处理的语法树。手写识别又分为联机识别和脱机识别,两者间的区别在于联机识别系统接受动态笔迹为输入,而脱机识别则接受位图为输入。联机识别容许人们通过在触摸设备上书写或拖动鼠标来输入数学公式,适合用于记笔记或与计算机代数系统交互。脱机识别则容许人们录入图片中的数学公式,适合用于电子化手稿的扫描版或者黑板的照片。</p>
<p>值得一提的是,数学公式是一类有代表性的二维记号,可以被用来识别手写数学公式的技术也可能可以被用于识别手写的表格、化学结构式、乐谱、电路图、流程图等等,又或是对手写文档进行版面分析。这些任务和数学公式识别同样受符号切分、符号识别和结构分析问题困扰,也和手写数学公式识别一样有实际的用途。</p>
<h2 id="为什么会想到笔划提取">为什么会想到笔划提取</h2>
<p>通过简单地把笔迹渲染为图片,我们可以轻易地把联机识别问题转化为脱机识别问题。也就是说,给定一个脱机识别系统,我们总可以构造一个和它一样准确的联机识别系统。反过来,如果能够从位图中恢复笔迹信息,联机识别系统也能被用于脱机识别,这样做有以下潜在好处:</p>
<ul>
<li>提高准确率。联机识别的准确率一般比脱机识别高,而且笔划提取容许同时用联机和脱机数据进行训练。</li>
<li>加快识别和降低内存占用。基于卷积神经网络的脱机识别是计算密集的,往往需要GPU等硬件来加速。</li>
<li>降低安装包大小。对于需要同时支持联机与脱机识别的应用,基于笔划提取时只需部署一套模型就能完成两种任务。</li>
<li>减低维护成本。由于联机与脱机识别有一定的相似性,平行地分别开发两套识别系统难免有重复劳动的成分。</li>
</ul>
<h2 id="如何基于笔划提取做脱机识别">如何基于笔划提取做脱机识别</h2>
<p>首先,我们需要一个笔划提取器来把位图转换为笔划序列,这样就可以对提取出的笔划调用联机识别系统了。如果笔划提取完全准确,脱机识别就能做到和联机识别一样准确。虽然这不太容易实现,但在笔划提取大致准确时,已经足以构造准确性有竞争力的脱机识别系统。以下这个笔划提取器纵使不完美,但它的每一步都是有用的。</p>
<p><img src="/image/stroke-extraction.png" alt="笔划提取" /></p>
<p>其次,在可行的情况下用提取出的笔划代替原始笔划去重新训练所基于的联机手写数学公式识别系统,从而避免笔划提取器的单点故障问题。借助这做法有望得到准确性与联机识别系统相近的脱机识别系统。这还使得我们可以利用脱机手写数学公式和印刷体数学公式来训练联机手写数学公式识别系统,有利于增大训练集。</p>
<h2 id="如何利用不完美的识别系统">如何利用不完美的识别系统</h2>
<p>目前即使最好的联机识别系统的准确率也只有80%,也就是说写几条公式就很可能会遇到误识的情况,因此应用必须考虑这种情况。</p>
<ul>
<li>对于要求高准确性的场合,例如作为计算机代数系统的输入时,需要设计方便用户高效地修正数学公式的界面。以下是一些可能可提高公式编辑器的可用性的思路:
<ul>
<li>提供多个备选公式供用户选取,因为识别系统的前k准确率往往明显比前1准确率高;</li>
<li>提供修改单个符号的操作,因为很多情况下识别结果结构正确而只是少数符号错了;</li>
<li>呈现输入笔划与输出符号间的对应关系,让用户更容易地发现错误;</li>
<li>整合键盘、鼠标和手势等输入方式,让用户用最熟悉的方式修正结果。</li>
</ul>
</li>
<li>对于其它场合,例如数学公式检索,少量的识别错误往往不太影响最终结果,而且存在一些技术手段进一步限制这种影响:
<ul>
<li>利用多个候选识别结果去扩展查询,因为识别系统的前k准确率往往明显比前1准确率高;</li>
<li>利用错误感知的相似性度量,因为识别系统一般有一些常见的混淆模式。</li>
</ul>
</li>
</ul>
<h2 id="全文">全文</h2>
<p>如欲了解更多细节,请参阅以下文献:</p>
<ul>
<li><a href="https://doi.org/10.1109/ACCESS.2020.2984627">Stroke Extraction for Offline Handwritten Mathematical Expression Recognition</a>是一篇正式发表的英文学术论文</li>
<li><a href="https://assets.chungkwong.cc/document/stroke-extraction.pdf">《基于笔划提取的脱机手写数学公式识别研究》</a>是一份更详细但非正式的中文技术报告</li>
</ul>陈颂光手写体识别传统上被分为联机识别和脱机识别,前者识别动态的笔迹而后者识别静态的图片。与联机识别相比,由于缺乏动态信息和存在背景噪声,脱机识别的准确度通常较低,同时往往占用更多资源。于是,把脱机识别问题转化为联机识别问题是一个有吸引力的想法。本文以由于具有紧凑的二维结构而极具挑战性的数学公式识别问题为例,探讨了基于笔划提取的脱机手写识别的可行性和实用性,这方法夺得了国际性权威竞赛CROHME 2019中有关任务的季军。局部自适应二值化方法的内存高效快速实现2020-12-05T00:00:00+00:002020-12-05T00:00:00+00:00https://www.chungkwong.cc/binarization-sauvola<p>二值化被广泛用作识别前从背景分离出文本等对象的预处理步骤。通过逐个像素计算阀值,局部自适应二值化方法能较好地分割光照不均或带噪声的受损文档图片。由于阀值往往依赖于矩形窗口中灰度值的一些基于矩的统计量如均值和方差,过往人们常用积分图像去加速计算,代价则是需要额外占用大量内存空间。注意到按行主序,矩形窗口中矩和直方图都可以增量地计算,进而容易得到局部平均、标准差以至分位数,积分图像实际上是不必要的。特别地,这个想法导致Bernsen方法和Niblack型方法如Sauvola方法的新串行实现,它们的时间复杂度与输入图像的大小成正比且与窗口大小无关,而辅助空间关于输入图像大小次线性。</p>
<h2 id="常見的二值化方法">常見的二值化方法</h2>
<p>二值化是把灰度图像转化为二值图像的图像处理操作,通常用于从背景中区分出对象。特别地,二值化被广泛用作文档分析系统的预处理步骤,使光学文本识别引擎能够假设文本成分已经被提取出来。二值化的质量同时影响版面分析和字符识别的准确性,在处理受损文档图像和车牌等场景文本时尤其重要。</p>
<p>许多流行的二值化方法通过比较像素的灰度值和计算出的阀值工作。按照阀值是否因像素而异,可以把这类方法细分为局部自适应方法和整体方法。与整体方法相比,局部自适应方法对常见的受损情况如噪声、光照不均和穿透更健壮。例如单一阀值对于分离文本和咖啡迹而言可能并不足够。以下,我们把一个$H\times W$灰度输入图像记为$(I_{ij})_{0\leq i <H, 0\leq j <W}$,其中按惯例灰度值$I_{ij}\in\left\{0, \ldots, 255\right\}$。相应地,我们把二值图像记为$(B_{ij})_{0\leq i <H, 0\leq j <W}$,其中按惯例$B_{ij}\in\left\{0, 1\right\}$。</p>
<p>整体方法对每个输入图像计算一个阀值$T$,然后令
$B_{ij}=\begin{cases}0&, I_{ij}\leq T\\1&, I_{ij}> T\end{cases}$。
其中最有代表性的Otsu方法选择阀值使类间灰度值方差最大以顾及图像的亮度和对比度。</p>
<p>局部自适应方法即使对同一图像的不同像素也使用可能不同的阀值。像素$(i, j)$的阀值$T_{ij}$用它为心的邻域中像素灰度值的某些统计量来计算,然后令
$B_{ij}=\begin{cases}0&, I_{ij}\leq T_{ij}\\1&, I_{ij}> T_{ij}\end{cases}$。
通常把邻域取为以$(i, j)$为心的$h\times w$矩形窗口,常用的统计量有平均值$\mu_{ij}$和标准差$\sigma_{ij}$。特别地,Niblack取
$T_{ij}=\mu_{ij}+K\sigma_{ij}$, 而Sauvola和Pietikäinen取$T_{ij}=\mu_{ij}\left(1+K\left(\frac{\sigma_{ij}}{R}-1\right)\right)$, 其中$K$为一个可调整的正参数而$R$通常取为$128$,注意$\sigma_{ij}\leq \left(255-0\right)/2<128\text{。}$按照定义
$\mu_{ij}=\frac{1}{hw}\displaystyle\sum_{\substack{i-\frac{h}{2}<k\leq i+\frac{h}{2}\\j-\frac{w}{2}<\ell\leq j+\frac{w}{2}}}I_{k\ell}$而
$\sigma_{ij}=\sqrt{\frac{1}{hw}\displaystyle\sum_{\substack{i-\frac{h}{2}<k\leq i+\frac{h}{2}\\j-\frac{w}{2}<\ell\leq j+\frac{w}{2}}}I_{k\ell}^2-\mu_{ij}^2}$。
Bernsen则取$T_{ij}=\begin{cases}\frac{1}{2}\left(\min_{ij}+\max_{ij}\right)&, \max_{ij}-\min_{ij}\geq C\\ -1&, \max_{ij}-\min_{ij}<C\end{cases}$,其中$C$为参数,
$\min\nolimits_{ij}=\min\left\{I_{k\ell}\vert i-\frac{h}{2}<k\leq i+\frac{h}{2}, j-\frac{w}{2}<\ell\leq j+\frac{w}{2}\right\}$,而
$\max\nolimits_{ij}=\max\left\{I_{k\ell}\vert i-\frac{h}{2}<k\leq i+\frac{h}{2}, j-\frac{w}{2}<\ell\leq j+\frac{w}{2}\right\}$。</p>
<p>下图比较了一些二值化方法应用于某张样本图片时的效果。Otsu方法把相对暗的一片背景当作了前景,这观察验证了Otsu方法Otsu对背景亮度变化不健壮,因为它是一个整体二值化方法。Bernsen方法基本上去除了暗的背景,但一行文本也失去了,这观察验证了Bernsen方法在低对比度区域效果欠佳,因为它们都被视为背景。Sauvola方法在视觉上几乎提取了所有文本,虽然很多字符断裂了且留下了不少噪声。因此,本文系统使用Sauvola方法。</p>
<p><img src="image/sauvola/binarization.png" alt="二值化方法的例子" /></p>
<p>存在许多其它二值化方法,而且直到近期仍然有新方法被不断地提出来。除启发式方法外,还有一些基于机器学习的方法。有监督学习可以用于直接把像素分类,例如Tensmeyer和Martinez设计了一个全卷积神经网络来整合来自不同尺度的信息。非监督学习可以用于对像素聚类,例如Bhowmik等人用$k$-均值算法去聚类受博弈论启发的特征。不过,机器学习算法通常都是计算密集的,所以当受限的计算资源是关注点时不会是首选,特别是在其它方法的结果可以接受时。</p>
<h2 id="局部自适应方法现有实现及其局限性">局部自适应方法现有实现及其局限性</h2>
<p>虽然局部自适应方法一般来说比整体方法更鲁棒,但也更计算密集,现有实现要么慢要么挥霍内存。具体地说,Otsu方法的实现只需$\Theta \left(HW\right)$时间和常量辅助空间:先遍历输入图像中所有像素一遍就能得到灰度值直方图,然后就能得到阀值,最后二值图像在下一轮迭代中得到。相反,直接用定义公式计算阀值的话,Sauvola方法的时间复杂度为$\Theta (HWhw)$,速度随窗口变大而递减。然而,随着高分辨率图片变得普遍,大窗口正变得更有用。</p>
<p>一个以空间换取时间的著名方法是维护积分图像。Bradley和Roth用积分图像去加速窗口中灰度均值的计算。Shafait等人进一步用这技巧去计算标准差。注意到加速Sauvola方法的关键是高效地计算以下形式的量:
$\displaystyle\sum_{\substack{a<k\leq b\\c<\ell\leq d}}f(I_{k\ell})$,
其中$f$为一个函数。令$J_{ij}=\displaystyle\displaystyle\sum_{\substack{k\leq i\\\ell\leq j}}f(I_{k\ell})$, 则
$\displaystyle\sum_{\substack{a<k\leq b\\c<\ell\leq d}}f(I_{k\ell})=J_{bd}-J_{ad}-J_{bc}+J_{ac}$,
如下图所示。积分图像$(J_{ij})_{0\leq i <H, 0\leq j <W}$可以用以下递推关系计算:
$J_{i, j}=\begin{cases}0 &, i<0 \textrm{ 或 }j<0\\\left(J_{i, j-1}-J_{i-1, j-1}\right)+f\left(I_{i, j}\right)+J_{i-1, j} & , \textrm{其它}\end{cases}$。
因此,Sauvola方法的时间复杂度可以降至$\Theta (HW)$,并且与窗口大小无关。然而,为了保存两个积分图像,过渡性内存消耗从$\Theta (1)$剧增到$\Theta (HW)$。它们要占用比输入的灰度图像更大的空间,这种内存消耗对一些应用场景而言过大。</p>
<p><img src="image/sauvola/integral.svg" alt="局部自适应二值化方法的积分图像法实现的原理" /></p>
<p>人们还提出了硬件实现。Chen等人提出了一个Sauvola方法基于GPU的并行实现,但积分图像的构造未有并行化。Najafi和Salehi则提出了一个基于随机电路的完整硬件实现,但它局限于一个小且固定的窗口大小。</p>
<p>许多应用需要局部自适应二值化方法又快又省内存的实现,而且不能依赖于专门的硬件。对于实时应用场景如识别黑板上的内容,学生手中的移动设备要用有限的资源迅速地完成工作。对于批处理应用场景如大型历史文献电子化计划,高效的二值化算法将有助于节省大量时间或硬件。</p>
<h2 id="sauvola方法的新实现">Sauvola方法的新实现</h2>
<p>本节提出Sauvola方法的一个内存高效且快速的实现,它不要求任何特殊的硬件。在加速Sauvola方法的同时尽可能节约内存的想法来源于两个观察。</p>
<p>基本的观察是两个相近的窗口内含有许多公共的元素,因此灰度值的$f$在一个窗口内的总和可以被重用来计算另一个窗口内的总和。形式地,对整数$a, b, c\text{ 和 }d$,
$\displaystyle\sum_{\substack{a<k\leq b\\c<\ell\leq d}}f(I_{k\ell})=\displaystyle\sum_{\substack{a<k\leq b\\ c-1<\ell\leq d-1}}f(I_{k\ell})+\displaystyle\sum_{a<k\leq b}f(I_{kd})-\displaystyle\sum_{a<k\leq b}f(I_{kc})$,
如下图所示。</p>
<p><img src="image/sauvola/window.svg" alt="局部自适应二值化方法的本文实现的原理(窗口级)" /></p>
<p>类似地,对整数$a, b\text{ 和 }e$,
$\displaystyle\sum_{a<k\leq b}f(I_{ke})=\displaystyle\sum_{a-1<k\leq b-1}f(I_{ke})+f(I_{be})-f(I_{ae})$
,如下图所示。</p>
<p><img src="image/sauvola/stripe.svg" alt="局部自适应二值化方法的本文实现的原理(列级)" /></p>
<p>因此,若按行主序计算各像素的阀值,实现只需维护量
$\displaystyle\displaystyle\sum_{i-\lfloor\frac{h+1}{2}\rfloor<k\leq i+\lfloor\frac{h}{2}\rfloor}f(I_{k\ell})$,
其中$\ell=0, \ldots, W-1$,而不需要积分图像。由公式可见本法中每个像素只需四次加/减法,而由公式可见积分图像法中每个像素需要五次加/减法。由于一旦有了一个像素的阀值就能立即二值化该像素,实现不用记忆阀值。</p>
<p>另一个观察是我们不用显式地计算阀值$T_{ij}$以达到二值化的目的。注意到在$K$非负的假设下,
$I_{ij}\leq \mu_{ij}\left(1+K\left(\frac{\sigma_{ij}}{R}-1\right)\right)$等价于
$I_{ij}+\mu_{ij}(K-1)\leq 0$或$\left(I_{ij}+\mu_{ij}\left(K-1\right)\right)^2\leq \frac{K^2\mu_{ij}^2\sigma_{ij}^2}{R^2}$。
因此,开根号运算可以代之以远比它快的乘法运算,从而达到强度削减的效果。</p>
<p>结合两个观察,Sauvola方法可以按以下算法实现。为简单起见,我们假设输入图像和输出图像在内存隔离,否则$I_{i-o, j}$可能被意外地改写。可以修改算法以容许原地二值化,但需要额外空间以保存$oW$个灰度值。</p>
<ul>
<li>输入:
<ul>
<li>灰度位图$(I_{ij})_{0\leq i <H, 0\leq j <W}$</li>
<li>窗口的宽度$w$和高度$h$</li>
<li>正参数$K$ 和 $R$</li>
</ul>
</li>
<li>输出:
<ul>
<li>二值位图$(B_{ij})_{0\leq i <H, 0\leq j <W}$</li>
</ul>
</li>
<li>步骤
<ol>
<li>$o\leftarrow \lfloor \frac{h+1}{2} \rfloor$;</li>
<li>$u\leftarrow \lfloor \frac{h}{2} \rfloor$;</li>
<li>$l\leftarrow \lfloor \frac{w+1}{2} \rfloor$;</li>
<li>$r\leftarrow \lfloor \frac{w}{2} \rfloor$;</li>
<li>对于 $j\leftarrow 0$ 直到 $W-1$ 进行
<ol>
<li>$P_j\leftarrow 0$;</li>
<li>$Q_j\leftarrow 0$;</li>
<li>对于 $i\leftarrow 0$ 直到 $\min\left\{u-1, H-1\right\}$ 进行
<ol>
<li>$P_j\leftarrow P_j+I_{ij}$;</li>
<li>$Q_j\leftarrow Q_j+I_{ij}^2$;</li>
</ol>
</li>
</ol>
</li>
<li>对于 $i\leftarrow 0$ 直到 $H-1$ 进行
<ol>
<li>对于 $j\leftarrow 0$ 直到 $W-1$ 进行
<ol>
<li>若 $i-o\geq 0$ 则
<ol>
<li>$P_j\leftarrow P_j-I_{i-o, j}$;</li>
<li>$Q_j\leftarrow Q_j-I_{i-o, j}^2$;</li>
</ol>
</li>
<li>若 $i+u<H$ 则
<ol>
<li>$P_j\leftarrow P_j+I_{i+u, j}$;</li>
<li>$Q_j\leftarrow Q_j+I_{i+u, j}^2$;</li>
</ol>
</li>
</ol>
</li>
<li>$p\leftarrow 0$</li>
<li>$q\leftarrow 0$</li>
<li>对于 $j\leftarrow 0$ 直到 $\min\left\{r-1, W-1\right\}$ 进行
<ol>
<li>$p\leftarrow p+P_j$;</li>
<li>$q\leftarrow q+Q_j$;</li>
</ol>
</li>
<li>对于 $j\leftarrow 0$ 直到 $W-1$ 进行
<ol>
<li>若 $j-l\geq 0$ 则
<ol>
<li>$p\leftarrow p-P_{j-l}$;</li>
<li>$q\leftarrow q-Q_{j-l}$;</li>
</ol>
</li>
<li>若 $j+r<W$ 则
<ol>
<li>$p\leftarrow p+P_{j+r}$;</li>
<li>$q\leftarrow q+Q_{j+r}$;</li>
</ol>
</li>
<li>$n\leftarrow \left(\min\left\{j+r, W-1\right\}-\max\left\{j-l, -1\right\}\right)\times \left(\min\left\{i+u, H-1\right\}-\max\left\{i-o, -1\right\}\right)$;</li>
<li>$m\leftarrow\frac{p}{n}$;</li>
<li>$v\leftarrow\frac{q}{n}-m^2$;</li>
<li>若$I_{ij}+m(K-1)\leq 0$ 或 $\left(I_{ij}+m\left(K-1\right)\right)^2\leq \frac{K^2m^2v}{R^2}$(实现其它Niblack型方法时修改这条件),则$B_{ij}\leftarrow 0$,否则$B_{ij}\leftarrow 1$。
}
}</li>
</ol>
</li>
</ol>
</li>
</ol>
</li>
</ul>
<p>算法需要$\Theta (W)$辅助空间,不包括输入和输出图像占用的空间。通过调换两个坐标轴,可以实现需要$\Theta (H)$辅助空间的变种。因此,通过在运行时因应高宽比选择一个变种可以使辅助空间需求降至$\Theta (\min\{H, W\})$,它明显低于积分图像法所需的$\Theta (HW)$。事实上,$\min\left\{H, W\right\}\leq\sqrt{HW}\ll HW$。</p>
<p>另外,更小的中间量意味着可以用更短的整数类型保存。在通常应用中,灰度值从0到255,窗口大小不超过257,由直接计算
$255\times 257=65535=2^{16}-1$而$255\times 255\times 257<2^{32}-1$,可见数组$P$的每个元素只需占用一个无符号16位整数而数组$Q$的每个元素只需占用一个32位整数。作为对比,一页A4文档按600DPI扫描后有多达$8.27\times 11.7 \times 600^2=34833240>2^{32}/255$个像素,所以积分图像的每个元素需要占用一个64位整数以防溢出。综上所述,积分图像法需要约$16HW$个字节的辅助空间,而本算法只需要约$6\min\left\{H, W\right\}$个字节的辅助空间。</p>
<p>算法的时间复杂度为$\Theta (HW)$且与窗口大小无关,和积分图像法一样在量级上最优,Sauvola方法不可能有次线性时间的串行实现。因为更低的内存占用意味着更高的缓存命中率,加上加减法次数的减少,速度还可能有所提升。当有多个处理器时,算法可以通过用若干个矩形覆盖输入图像来并行化。</p>
<h2 id="其它方法的实现">其它方法的实现</h2>
<p>虽然前文用Sauvola方法为例进行阐述,但同样的技巧也可以用于加速很多其它二值化方法。</p>
<p>只用修改算法中的一个<code class="language-plaintext highlighter-rouge">if</code>语句中的条件后即可实现大部分基于Niblack方法的局部二值化方法,因为它们的阀值公式也只涉及矩形窗口中的均值和方差(可能还有一些整体量),下表列出了其中一些,其中$L$、$M$和$S$分别为整体最小值、均值和标准差,$\alpha$、$\gamma$、$K$、$K_1$、$K_2$、$p$、$q$和$R$均为参数,更多的例子参见有关的综述。显然,这种修改不影响对时间和空间复杂度的分析,所以依然只需$\Theta \left(HW\right)$时间和$O \left(\min\left\{H, W\right\}\right)$辅助空间即可二值化一个$H\times W$图像。</p>
<table>
<thead>
<tr>
<th>Niblack型方法的例子</th>
<th>阀值</th>
</tr>
</thead>
<tbody>
<tr>
<td>Niblack</td>
<td>$\mu_{ij}+K\sigma_{ij}$</td>
</tr>
<tr>
<td>Sauvola和Pietikäinen</td>
<td>$\mu_{ij}\left(1+K\left(\frac{\sigma_{ij}}{R}-1\right)\right)$</td>
</tr>
<tr>
<td>Wolf等人</td>
<td>$\mu_{ij}-K\left(\mu_{ij}-L\right)\left(1-\frac{\sigma_{ij}}{R}\right)$</td>
</tr>
<tr>
<td>Feng和Tan</td>
<td>$\alpha\mu_{ij}+K_1\left(\frac{\sigma_{ij}}{R}\right)^{1+\gamma}\left(\mu_{ij}-L\right)+K_2\left(\frac{\sigma_{ij}}{R}\right)^{\gamma}L$</td>
</tr>
<tr>
<td>Rais 等人</td>
<td>$\mu_{ij}+0.3\frac{\mu_{ij}\sigma_{ij}-MS}{\max\{\mu_{ij}\sigma_{ij}, MS\}}\sigma_{ij}$</td>
</tr>
<tr>
<td>Khurshid 等人</td>
<td>$\mu_{ij}+K\sqrt{\sigma_{ij}^2+\mu_{ij}^2\frac{hw-1}{hw}}$</td>
</tr>
<tr>
<td>Phansalkar 等人</td>
<td>$\mu_{ij}\left(1+pe^{-q\mu_{ij}}+K\left(\frac{\sigma_{ij}}{R}-1\right)\right)$</td>
</tr>
</tbody>
</table>
<p>注意到局部直方图也可以递推地计算,而最大值、最小值、中位数以至任何分位数都可以从直方图轻易得到,类似方法也能够加速用到相应局部统计量的局部自适应二值化方法。事实上,中值滤波器的一种快速实现就用了这原理。特别地,Bernsen方法只需$\Theta \left(HW\right)$时间和$\Theta \left(\min\left\{W, H\right\}\right)$辅助空间。对于Bernsen方法,由于只用到用局部最大值和局部最小值,还有一种巧妙地用最长单调子序列取代直方图的实现,它的时间复杂度量级同样最优但常数因子较低。注意Bernsen方法的直接实现需要$\Theta \left(HWhw\right)$时间和$\Theta \left(1\right)$辅助空间。Otsu方法的局部版本也基于局部直方图,所以可以省去极度耗费内存的积分直方图(典型地每像素占用1 KiB内存)。</p>
<p>基于本法所支持的二值化方法的复合方法也会受惠。例如可以加入预处理和后处理步骤以降噪,修改阀值以消除离群值,或者应用多次局部自适应二值化来找不同大小的文本。另外,一些预处理过程和后处理过程也可用同样方法加速。例如灰度值的范围可用于估计灰度图像的局部对比度,一阶矩可以用于在二值图像中寻找孤立的前景或背景像素。</p>
<p>一般地,本文所用技巧的应用决不限于二值化,它能被应用到图像处理中的一些其它任务以消除积分图像的若干典型用法,并且仍然有在大致保持速度的前提下节省内存的效果。比如说,双边滤镜和导向滤镜之类的滤镜的实现既不需要积分直方图也不需要积分图像,它们的应用包括细节增强、去雾、光照调整、样式化、联合上采样和匹配。由于它们可能需要应用在移动设备如相机上,故计算开销是一个考虑因素。通过一些小修改可得到更大的灵活性,例如滑动窗口的中心不一定为当前像素,多个滑动窗口可以同时考虑等等。然而,它并不能在所有情况下取代积分图像,因为积分图像容许随机访问任意矩形中的总和,而这技巧只能按一定顺序获取总和。</p>
<h2 id="速度评测">速度评测</h2>
<p>为了检验本文实现的性能,我们实现了Sauvola方法的三种串行算法:直接算法、基于积分图像的算法和本文提出的算法。我们也同时评测了一些其它方法。所有实验用到的程序已经公开,代码见<a href="https://github.com/chungkwong/binarizer">https://github.com/chungkwong/binarizer</a>。</p>
<p>我们测量了二值化来自2009年与2018年间历次文档图像二值化竞赛(the Document Image Binarization COntests,DIBCO)和手写文档图像二值化竞赛(Handwritten Document Image Binarization COmpetitions,H-DIBCO)的116张图片所用的时间。实验在一台带Intel® Core™ i5-7500 CPU @ 3.40GHz和8 GB 内存的机器上进行。</p>
<p>不论对Sauvola方法还是Bernsen方法,下图都验证了本文实现的运行时间与窗口大小无关,与直接实现不同。</p>
<p><img src="image/sauvola/time.svg" alt="窗口大小与二值化运行时间的关系" /></p>
<p>下表显示了窗口大小为21时具体的计算时间。对Sauvola方法,本文实现大约比基于积分图像的实现快了一倍,但仍然比Otsu方法慢。值得注意的是,用于避免积分图像的增量方法与强度削减技巧都有助于加速,虽然前者主要为节省内存而引入。</p>
<table>
<thead>
<tr>
<th>二值化方法</th>
<th>平均运行时间(秒/图片)</th>
</tr>
</thead>
<tbody>
<tr>
<td>固定阀值</td>
<td>0.00096</td>
</tr>
<tr>
<td>Otsu方法</td>
<td>0.00151</td>
</tr>
<tr>
<td>Sauvola方法(直接实现)</td>
<td>0.47614</td>
</tr>
<tr>
<td>Sauvola方法(积分图像实现)</td>
<td>0.02739</td>
</tr>
<tr>
<td>Sauvola方法(积分图像实现加上强度削减)</td>
<td>0.01619</td>
</tr>
<tr>
<td>Sauvola方法(本文实现去除强度削减)</td>
<td>0.02458</td>
</tr>
<tr>
<td><strong>Sauvola方法(本文实现)</strong></td>
<td><strong>0.01349</strong></td>
</tr>
<tr>
<td>Bernsen方法(直接实现)</td>
<td>0.48895</td>
</tr>
<tr>
<td><strong>Bernsen方法(本文实现)</strong></td>
<td><strong>0.07483</strong></td>
</tr>
</tbody>
</table>
<h2 id="结论">结论</h2>
<p>我们为一大类常见的局部自适应二值化方法提出了内存高效的快速实现。特别地,假定输入灰度图像大小为$H\times W$而窗口大小为$h\times w$,Sauvola方法和其它Niblack型方法的各种实现的复杂度如表下表所示,本文方法在理论和实验上都明显优于流行的积分图像法,同时时空权衡比直接算法更平衡。</p>
<table>
<thead>
<tr>
<th>算法</th>
<th>时间复杂度</th>
<th>辅助空间复杂度</th>
</tr>
</thead>
<tbody>
<tr>
<td>直接算法(Sauvola等,2000)</td>
<td>$\Theta (HWhw)$</td>
<td>$\mathbf{\Theta (1)}$</td>
</tr>
<tr>
<td>积分图像法(Shafait等,2008)</td>
<td>$\mathbf{\Theta (HW)}$</td>
<td>$\Theta (HW)$</td>
</tr>
<tr>
<td><strong>本文方法(Chan,2019)</strong></td>
<td>$\mathbf{\Theta (HW)}$</td>
<td>$\Theta (\min\{H,W\})$</td>
</tr>
</tbody>
</table>
<p>类似技巧也适用于图像处理中的一些其它任务以消除积分图像的若干典型用法,从而在大致保持速度的前提下节省内存。由于空间和时间对资源受限设备上的实时应用都是重要的,本文算法对其它问题有借镜意义。</p>
<p><strong>如果您希望在学术论文中引用本算法,请引用<a href="https://arxiv.org/abs/1905.13038">https://arxiv.org/abs/1905.13038</a>,谢谢。</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Chungkwong Chan. Memory-efficient and fast implementation of local adaptive binarization methods. CoRR abs/1905.13038 (2019)
</code></pre></div></div>
<div class="language-bibtex highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">@article</span><span class="p">{</span><span class="nl">DBLP:journals/corr/abs-1905-13038</span><span class="p">,</span>
<span class="na">author</span> <span class="p">=</span> <span class="s">{Chungkwong Chan}</span><span class="p">,</span>
<span class="na">title</span> <span class="p">=</span> <span class="s">{Memory-efficient and fast implementation of local adaptive binarization
methods}</span><span class="p">,</span>
<span class="na">journal</span> <span class="p">=</span> <span class="s">{CoRR}</span><span class="p">,</span>
<span class="na">volume</span> <span class="p">=</span> <span class="s">{abs/1905.13038}</span><span class="p">,</span>
<span class="na">year</span> <span class="p">=</span> <span class="s">{2019}</span><span class="p">,</span>
<span class="na">url</span> <span class="p">=</span> <span class="s">{http://arxiv.org/abs/1905.13038}</span><span class="p">,</span>
<span class="na">archivePrefix</span> <span class="p">=</span> <span class="s">{arXiv}</span><span class="p">,</span>
<span class="na">eprint</span> <span class="p">=</span> <span class="s">{1905.13038}</span><span class="p">,</span>
<span class="na">timestamp</span> <span class="p">=</span> <span class="s">{Thu, 30 May 2019 13:17:18 UTC}</span><span class="p">,</span>
<span class="na">biburl</span> <span class="p">=</span> <span class="s">{https://dblp.org/rec/journals/corr/abs-1905-13038.bib}</span><span class="p">,</span>
<span class="na">bibsource</span> <span class="p">=</span> <span class="s">{dblp computer science bibliography, https://dblp.org}</span>
<span class="p">}</span>
</code></pre></div></div>陈颂光二值化被广泛用作识别前从背景分离出文本等对象的预处理步骤。通过逐个像素计算阀值,局部自适应二值化方法能较好地分割光照不均或带噪声的受损文档图片。由于阀值往往依赖于矩形窗口中灰度值的一些基于矩的统计量如均值和方差,过往人们常用积分图像去加速计算,代价则是需要额外占用大量内存空间。注意到按行主序,矩形窗口中矩和直方图都可以增量地计算,进而容易得到局部平均、标准差以至分位数,积分图像实际上是不必要的。特别地,这个想法导致Bernsen方法和Niblack型方法如Sauvola方法的新串行实现,它们的时间复杂度与输入图像的大小成正比且与窗口大小无关,而辅助空间关于输入图像大小次线性。如何在保障隐私的同时跟踪传染病传播2020-07-29T00:00:00+00:002020-07-29T00:00:00+00:00https://www.chungkwong.cc/contact-tracking<p>新冠肺炎(SARS-CoV-2)于2020年在全球范围的爆发引起了广泛关注,为了快速而有效地减慢病毒的传播,群体免疫以外的另一种可能有效的策略就是找出与感染者有密切接触的人士并让他们自我隔离。由于手机等移动电子通信设备的普及,部分国家强迫人民安装的某种称为“健康码”的应用程序,用以评估用户带病的风险。然而,这类程序往往被指为政府收集过多信息,这种大规模监控严重侵犯了公民的隐私权,并可能被用于其它不可告人的目的。其实,精准的传染病链跟踪不需要以牺牲人民的隐私为代价,<a href="https://www.pepp-pt.org/">泛欧保隐私接触追踪(Pan-European Privacy-Preserving Proximity Tracing,PEPP-PT)</a>和<a href="https://github.com/DP-3T/documents">去中心化保隐私接触跟踪(Decentralized Privacy-Preserving Proximity Tracing,DP-3T)</a>就让用户匿名且完全不收集任何定位信息。</p>
<h2 id="设计目标">设计目标</h2>
<p>实用的传染病传播链跟踪系统应该符合以下要求:</p>
<ol>
<li>能够快速而精准地找出潜在的传播链。为了精确地找出潜在的感染者并提醒他们自我隔离,需要找出所有与确诊者曾经在物理上接近的人。一方面,由于即使在疫情最严重的地区,感染者仍然是少数,鲁莽地封城或封区等极端措施打击面太大,不必要地重创经济,甚至可能使饿死的人数远多于因感染传染病而致命的人数,精确的跟踪可以防止这种过度牵连。另一方面,人手查问确诊人士的行踪容易出现遗漏的情况,而且需要大量受过训练的工作人员。</li>
<li>尊重隐私。只能收集对传染病跟踪而言绝对必须的数据,以防止被挪用于其它用途。</li>
<li>可以在全球范围内工作。不但要保证数以十亿计的手机参与时系统仍然能运作,而且能够跨越地理边界以便恢复正常的跨境往来。</li>
<li>能够迅速投入使用。只能使用当前可用的基础设施和硬件,不能依赖于未来可能的技术突破。</li>
</ol>
<h2 id="基本原理">基本原理</h2>
<p>不论是PEPP-PT还是DP-3T,基本原理是相同的:每部手机不断用蓝牙与周边的手机交换随机数。以下用隐私保护更佳的DP-3T为例简单说明这类传染病跟踪系统的要素:</p>
<ul>
<li>每一个人随身携带一部装有跟踪应用程序的手机。这样,两个人有密切接触的话,她们的手机之间距离也会很短。</li>
<li>在每一个时刻,每部手机有一个匿名的标识,手机每隔一段时间(如10至15分钟)修改其标识。这个标识用以区分不同手机。即使标识的身份一时被泄露(例如手机身处一个附近很少人的地方),由于标识会在短时间内自动失效,仍然不能被用于持续监控。以下是生成标识的一个方法:
<ul>
<li>标识在手机端(伪)随机地生成。只要标识的长度足够(例如128位),就可以使出现碰撞的机会足够低,同时保证其它人(包括中央服务器)都不知道一个标识具体属于哪台手机。更准确地说,刚安装时随机生成一个随机种子,然后每天通过对上一天的随机种子应用某个散列函数得到当天的随机种子,再用该随机种子用某伪随机数生成器生成足够标识供当天各时段乱序使用。</li>
</ul>
</li>
<li>每部手机每隔一段时间(如200至270毫秒)用低功耗蓝牙广播自己当时的标识。由于蓝牙的传输距离大于飞沫的有效传播距离(约2米),足以让密切接触者的手机间通信。同时,部署蓝牙设备以覆盖全境对最极权的人权侵犯者而言都不可行,这可以防止系统被用于大范围监控,但在特定场所部署设备以获知近期访问过场所的确诊人数是可能的。相反,假如系统基于定位信息,则是把用户的行踪暴露在巨大的泄露风险中。</li>
<li>每部手机每隔一段时间(不超过5分钟)扫描蓝牙并把周边其它手机的标识保存下来,同时保存信号强度和当前时间。这样,可以估算接触时间长度和距离,以便评估风险(接触时间越长或距离越短,传染机会越大)。</li>
<li>一旦有人被确诊,她在卫生部门的协助下向中央服务器上传有关数据。以下是其中一个方案:
<ul>
<li>用户上传其手机若干天(如14天)前用过的随机种子,再重新初始化随机种子。由于随机种子的匿名性,可以保护确认者身份的保密性。</li>
</ul>
</li>
<li>手机定期或需要时从中央服务器(或内容分享网络,CDN)取得确诊者相关资料,并向用户提示其风险。以下是其中一个方案:
<ul>
<li>手机从中央服务器下载确诊者近期用过的随机种子,然后生成她们用过的所有标识,再检查近期有否收到过其中的标识,有的话结合接触时间长度和距离计算风险。由于风险计算在手机进行,且非确诊从不向中央服务器上传任何数据,其它人难以窃取接触纪录或评估结果。</li>
</ul>
</li>
<li>每个国家或地区有一个中央服务器以保存可能到过该地方的确诊者的相关数据。处理跨境往来的一个可能方案如下:
<ul>
<li>确诊用户上传时同时呈报她近期(如过去14天)去过的地方列表,以便把数据转交有关地区的中央服务器。</li>
</ul>
</li>
<li>手机自动删除太旧(如14天前)的数据。这样可以避免占用太多空间。</li>
</ul>
<h2 id="不足之处">不足之处</h2>
<p>虽然传染病跟踪系统在设计时已经经过重重论证,但仍然有一些已知缺陷。其中有一些只影响个别系统,而其它则影响所有同类系统。</p>
<ul>
<li>信息安全
<ul>
<li>可用性。
<ul>
<li>中央服务器可能受到分布式拒绝服务攻击,使用户无法同步数据,从而不能及时知道自己的风险。</li>
</ul>
</li>
<li>完整性。接触跟踪所需的必要数据可能被破坏,从而影响风险评估的准确性。
<ul>
<li>蓝牙可能受到阻塞,使手机无法收到其它手机广播的信息,导致接触纪录出现遗漏。</li>
<li>用户可能不安装有关应用程序,确诊者也可能不上传数据,导致确诊者的资料不齐全。</li>
<li>用户可以在一日内广播他人用过的标识,从而制造虚假的接触纪录,导致别人以为自己是高风险人士,以制造恐慌。</li>
<li>确诊者可能用他人的手机来上传,污染中央服务器的数据。</li>
<li>入侵或仿冒中央服务器来发布虚假资料。</li>
</ul>
</li>
<li>匿名性。在特殊情况下,确诊者身份的匿名性可能被破坏,以下是一些场景:
<ul>
<li>其它用户可以修改应用程序并记录更多接触资料(如地点和周边的监控录像)。于是一旦有接触者确诊,确诊者的身份就有可能被锁定。同类攻击适用所有接触者跟踪系统。</li>
<li>其它用户可能可以知道两个标识属于同一确诊者。采用上述上传随机种子方案的话,攻击者可以知道确诊用户近期用过的所有标识,从而得到“在什么什么时间多次碰到同一确诊者”之类的信息。</li>
<li>互联网服务商、WiFi提供者、中央服务器可能记录确诊者的IP从而可能可确定其身份。手机制造多余流量和中央服务器批量更新可以防范前两者。</li>
</ul>
</li>
</ul>
</li>
<li>其它方面
<ul>
<li>续航时间。持续用蓝牙通信会增加手机的耗电量,使电池续航时间下降。</li>
</ul>
</li>
</ul>
<h2 id="总结">总结</h2>
<p>健康与隐私是可以并存的。精确的传染病跟踪并不需要收集任何个人信息和位置信息,而只需要现有的简单技术。世界各地的行政机关都在用防疫为名来圆加强中央集权的夙愿之实,推出各种不必要地限制人身自由的措施,大规模地监控人们的活动,并助长对种族、地域、年龄、职业的歧视。转而通过精准的传播链跟踪来善用有限的检测资源和控制隔离人数,可能是株连九族和完全放任之间的一个平衡点。</p>陈颂光新冠肺炎(SARS-CoV-2)于2020年在全球范围的爆发引起了广泛关注,为了快速而有效地减慢病毒的传播,群体免疫以外的另一种可能有效的策略就是找出与感染者有密切接触的人士并让他们自我隔离。由于手机等移动电子通信设备的普及,部分国家强迫人民安装的某种称为“健康码”的应用程序,用以评估用户带病的风险。然而,这类程序往往被指为政府收集过多信息,这种大规模监控严重侵犯了公民的隐私权,并可能被用于其它不可告人的目的。其实,精准的传染病链跟踪不需要以牺牲人民的隐私为代价,泛欧保隐私接触追踪(Pan-European Privacy-Preserving Proximity Tracing,PEPP-PT)和去中心化保隐私接触跟踪(Decentralized Privacy-Preserving Proximity Tracing,DP-3T)就让用户匿名且完全不收集任何定位信息。用OpenLayers设计地理信息网站2020-03-14T00:00:00+00:002020-03-14T00:00:00+00:00https://www.chungkwong.cc/openlayers<p>在网页上提供地理信息是一个常见的需求,例如连锁店的网站需要展示各门店的位置和前往路线。<a href="https://openlayers.org/en/latest/examples/">OpenLayers</a>就是其中一个经常用于显示地图的Javascript库,可以让用户交互式地与各种地理信息打交道。</p>
<h2 id="何时使用openlayers">何时使用OpenLayers</h2>
<p>为了显示一张地图,最简单的方法是用<code class="language-plaintext highlighter-rouge"><iframe></code>标签嵌入各大地图网站。例如<a href="https://www.openstreetmap.org/">OpenStreetMap</a>、<a href="https://www.bing.com/maps/embed-a-map">必应地图</a>、<a href="https://lbs.amap.com/console/show/card">高德地图</a>和<a href="http://api.map.baidu.com/lbsapi/creatmap/">百度地图</a>都可以生成可用来显示当前地图的HTML代码。比如说OpenStreetMap生成的代码形如:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><iframe</span> <span class="na">width=</span><span class="s">"425"</span> <span class="na">height=</span><span class="s">"350"</span> <span class="na">frameborder=</span><span class="s">"0"</span> <span class="na">scrolling=</span><span class="s">"no"</span> <span class="na">marginheight=</span><span class="s">"0"</span> <span class="na">marginwidth=</span><span class="s">"0"</span> <span class="na">src=</span><span class="s">"https://www.openstreetmap.org/export/embed.html?bbox=114.14517045021059%2C22.48035154717487%2C114.15284156799318%2C22.484599442229353&amp;layer=mapnik&amp;marker=22.48247551099446%2C114.14900600910187"</span> <span class="na">style=</span><span class="s">"border: 1px solid black"</span><span class="nt">></iframe><br/><small><a</span> <span class="na">href=</span><span class="s">"https://www.openstreetmap.org/?mlat=22.48248&amp;mlon=114.14901#map=18/22.48248/114.14901"</span><span class="nt">></span>查看更大的地图<span class="nt"></a></small></span>
</code></pre></div></div>
<p>实际显示效果像下面这样:</p>
<iframe width="425" height="350" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" src="https://www.openstreetmap.org/export/embed.html?bbox=114.14517045021059%2C22.48035154717487%2C114.15284156799318%2C22.484599442229353&layer=mapnik&marker=22.48247551099446%2C114.14900600910187" style="border: 1px solid black"></iframe>
<p><br /><small><a href="https://www.openstreetmap.org/?mlat=22.48248&mlon=114.14901#map=18/22.48248/114.14901">查看更大的地图</a></small></p>
<p>上述方法对于标示单一个位置的用途而言已经足够,但有时我们需要更大的灵活性。例如连锁店的网站可能需要同时展示多个门店的位置,并容许客户在地图上选取一个门店以查看营业时间和前往路线等信息。这时一个解决方案就是使用类似于<a href="https://openlayers.org/en/latest/examples/">OpenLayers</a>或者<a href="https://leafletjs.com/">Leaflet</a>的库。</p>
<p>为了了解OpenLayers能做什么,不妨先看一下我们制作的一个地图网站(<a href="https://www.viewfact.org/map.html">https://www.viewfact.org/map.html</a>)。它包含了以下功能:</p>
<ul>
<li>基本地图和影像地图切换器</li>
<li>地名语言切换器</li>
<li>定位</li>
<li>比例尺</li>
<li>全屏开关</li>
<li>缩略图</li>
<li>下载</li>
<li>持久链接</li>
<li>搜索</li>
<li>用户定义标记
<ul>
<li>标点和坐标显示</li>
<li>画线和长度显示</li>
<li>画多边形和面积显示</li>
<li>画圆和面积显示</li>
<li>颜色设置</li>
<li>撤销和重做</li>
</ul>
</li>
</ul>
<h2 id="基本用法">基本用法</h2>
<p>以下HTML文件用OpenLayers展示一个简单的地图:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!doctype html></span>
<span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/css/ol.css"</span> <span class="na">type=</span><span class="s">"text/css"</span><span class="nt">></span>
<span class="nt"><style></span>
<span class="nc">.map</span> <span class="p">{</span>
<span class="nl">height</span><span class="p">:</span> <span class="m">400px</span><span class="p">;</span>
<span class="nl">width</span><span class="p">:</span> <span class="m">100%</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt"></style></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://cdn.polyfill.io/v2/polyfill.min.js?features=requestAnimationFrame,Element.prototype.classList"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/build/ol.js"</span><span class="nt">></script></span>
<span class="nt"><title></span>OpenLayers基本例子<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"map"</span> <span class="na">class=</span><span class="s">"map"</span><span class="nt">></div></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></span>
<span class="kd">var</span> <span class="nx">map</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nb">Map</span><span class="p">({</span>
<span class="na">target</span><span class="p">:</span> <span class="dl">'</span><span class="s1">map</span><span class="dl">'</span><span class="p">,</span>
<span class="na">layers</span><span class="p">:</span> <span class="p">[</span>
<span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">layer</span><span class="p">.</span><span class="nx">Tile</span><span class="p">({</span>
<span class="na">source</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">source</span><span class="p">.</span><span class="nx">OSM</span><span class="p">()</span>
<span class="p">})</span>
<span class="p">],</span>
<span class="na">view</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">View</span><span class="p">({</span>
<span class="na">center</span><span class="p">:</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">proj</span><span class="p">.</span><span class="nx">fromLonLat</span><span class="p">([</span><span class="mf">114.1490</span><span class="p">,</span><span class="mf">22.4825</span><span class="p">]),</span>
<span class="na">zoom</span><span class="p">:</span> <span class="mi">17</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="nt"></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>效果如下:</p>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/css/ol.css" type="text/css" />
<style>
.map {
height: 400px;
width: 100%;
}
</style>
<script src="https://cdn.polyfill.io/v2/polyfill.min.js?features=requestAnimationFrame,Element.prototype.classList"></script>
<script src="https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/build/ol.js"></script>
<div id="map" class="map"></div>
<script type="text/javascript">
var map = new ol.Map({
target: 'map',
layers: [
new ol.layer.Tile({
source: new ol.source.OSM()
})
],
view: new ol.View({
center: ol.proj.fromLonLat([114.1490,22.4825]),
zoom: 17
})
});
</script>
<p>以下我们仔细分析这个例子。</p>
<p>首先,我们在HTML文件的头部包含了OpenLayers的样式表和Javascript库(版本为6.2.1),同时也包含了<code class="language-plaintext highlighter-rouge">polyfill</code>以支持老旧的IE和Android 4.x。这里我们使用了jsDelivr提供的CDN服务来获取资源,但由于这种CDN不总是可用,而且相对臃肿(包含了所有模块),在正式部署时应考虑把有关URL切换到自己构建的版本。</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/css/ol.css"</span> <span class="na">type=</span><span class="s">"text/css"</span><span class="nt">></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://cdn.polyfill.io/v2/polyfill.min.js?features=requestAnimationFrame,Element.prototype.classList"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"https://cdn.jsdelivr.net/gh/openlayers/openlayers.github.io@master/en/v6.2.1/build/ol.js"</span><span class="nt">></script></span>
</code></pre></div></div>
<p>接着,我们在文档体内增加一个用于放置地图的容器元素,它的高度和宽度可以如常用CSS控制。</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">id=</span><span class="s">"map"</span> <span class="na">class=</span><span class="s">"map"</span><span class="nt">></div></span>
</code></pre></div></div>
<p>最后,我们创建类型为<code class="language-plaintext highlighter-rouge">ol.Map</code>的地图,<code class="language-plaintext highlighter-rouge">target</code>参数指定把地图放到哪里(在本例中是<code class="language-plaintext highlighter-rouge">id</code>为<code class="language-plaintext highlighter-rouge">'map'</code>的元素),<code class="language-plaintext highlighter-rouge">layers</code>参数指定地图的内容(在本例中只有一个图层,来自OpenStreetMap),<code class="language-plaintext highlighter-rouge">view</code>参数指定地图初始的视图(例如本例中地图中心对应于东经114.1490北纬22.4825的位置,而缩放级为17)。</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></span>
<span class="kd">var</span> <span class="nx">map</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nb">Map</span><span class="p">({</span>
<span class="na">target</span><span class="p">:</span> <span class="dl">'</span><span class="s1">map</span><span class="dl">'</span><span class="p">,</span>
<span class="na">layers</span><span class="p">:</span> <span class="p">[</span>
<span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">layer</span><span class="p">.</span><span class="nx">Tile</span><span class="p">({</span>
<span class="na">source</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">source</span><span class="p">.</span><span class="nx">OSM</span><span class="p">()</span>
<span class="p">})</span>
<span class="p">],</span>
<span class="na">view</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">View</span><span class="p">({</span>
<span class="na">center</span><span class="p">:</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">proj</span><span class="p">.</span><span class="nx">fromLonLat</span><span class="p">([</span><span class="mf">114.1490</span><span class="p">,</span><span class="mf">22.4825</span><span class="p">]),</span>
<span class="na">zoom</span><span class="p">:</span> <span class="mi">17</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="nt"></script></span>
</code></pre></div></div>
<h2 id="定制地图">定制地图</h2>
<p>既然直接使用OpenLayers之类的库的主要优势在于可定制性,我们以下就看看如何灵活地定制地图的各个方面。</p>
<h3 id="调整图层">调整图层</h3>
<p>地图由若干个图层组成,例如地形图和地名标签可以分成两个图层以简化需支持多种底图和多种语言的地图渲染服务器。我们除了可以在创建地图时通过<code class="language-plaintext highlighter-rouge">layers</code>参数设定组件集合,也可以通过<code class="language-plaintext highlighter-rouge">getLayers()</code>取得图层集合并加以修改。</p>
<p>图层大致可以分为位图和向量图两大类,也可以分为完整图和拼合图两大类,于是共有四小类。</p>
<p>完整位图的每个视图由一张完整的图片给出,创建方法如<code class="language-plaintext highlighter-rouge">new ol.layer.Image({source: 来源})</code>,其中来源提供图片。OpenLayers支持多种来源,例如ArcGISRest、Canvas、MapGuide、静态图片、WMS和光栅。</p>
<p>拼合位图的每个视图由多张预先渲染的图片拼合而成,创建方法如<code class="language-plaintext highlighter-rouge">new ol.layer.Tile({source: 来源})</code>,其中来源提供各图片。OpenLayers支持多种来源,例如Bing地图、CartoDB、IIIF、OSM、Stamen、TileArcGISRest、TileJSON、TileWMS、WMTS、XYZ、Zoomify和块坐标。前面我们已经用过<code class="language-plaintext highlighter-rouge">new ol.source.OSM()</code>来获取OpenStreetMap的数据。一般地,我们可以指定图片来源的URL模板,不必过于依赖个别的地图数据供应商。比如以下代码指定了缩放级<code class="language-plaintext highlighter-rouge">{z}</code>对应块坐标<code class="language-plaintext highlighter-rouge">({x},{y})</code>的地图块可从<code class="language-plaintext highlighter-rouge">https://mapapi.geodata.gov.hk/gs/api/v1.0.0/xyz/label/hk/tc/wgs84/{z}/{x}/{y}.png</code>取得,顺便也指出了版权信息。</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>new ol.source.XYZ({
url:'https://mapapi.geodata.gov.hk/gs/api/v1.0.0/xyz/label/hk/tc/wgs84/{z}/{x}/{y}.png',
attributions:'<a href="https://api.portal.hkmapservice.gov.hk/disclaimer" target="_blank" class="copyrightDiv">&copy; 香港特區政府</a>',
minZoom: 10,
maxZoom: 18,
crossOrigin: "Anonymous"
})
</code></pre></div></div>
<p>完整位图通常用于显示客户端作出的标记,创建方法如<code class="language-plaintext highlighter-rouge">new ol.layer.Vector({source: 来源})</code>。如果来源用<code class="language-plaintext highlighter-rouge">new ol.source.Vector()</code>创建,我们可以用<code class="language-plaintext highlighter-rouge">addFeature(特征)</code>和<code class="language-plaintext highlighter-rouge">removeFeature(特征)</code>方法向来源增删特征。在用<code class="language-plaintext highlighter-rouge">new ol.Feature()</code>新建特征后,可用<code class="language-plaintext highlighter-rouge">setGeometry(几何对象)</code>设置几何对象,也可用<code class="language-plaintext highlighter-rouge">setStyle(样式)</code>设置样式。</p>
<table>
<thead>
<tr>
<th>代码</th>
<th>几何对象</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.Circle(中心坐标,半径)</code></td>
<td>圆</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.LineString(坐标数组)</code></td>
<td>折线</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.MultiLineString(折线数组)</code></td>
<td>若干条折线</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.MultiPoint(坐标数组)</code></td>
<td>若干个点</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.MultiPolygon(多边形数组)</code></td>
<td>若干个多边形</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.Point(坐标)</code></td>
<td>点</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.Polygon(坐标数组)</code></td>
<td>多边形</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.grom.GeometryCollection(形状数组)</code></td>
<td>若干个形状</td>
</tr>
</tbody>
</table>
<p>拼合位图通常用于显示服务器提供的原始地理数据,创建方法如<code class="language-plaintext highlighter-rouge">new ol.layer.VectorTile({source: new ol.source.VectorTile({format:格式,url:网址模板})})</code>,其中支持的格式包括EsriJSON、GeoJSON、GML、GPX、IGC、KML、MVT、OSMXML、Polyline、TopoJSON、WFS、WKT和WMSGetFeatureInfo。</p>
<p>另外,<code class="language-plaintext highlighter-rouge">ol.layer.Graticule</code>可用于显示<a href="https://openlayers.org/en/latest/examples/graticule.html">网格</a>,<code class="language-plaintext highlighter-rouge">ol.layer.Heatmap</code>可用于显示<a href="https://openlayers.org/en/latest/examples/heatmap-earthquakes.html">热力图</a>。</p>
<h3 id="调整视图">调整视图</h3>
<p>我们除了可以在创建地图时通过<code class="language-plaintext highlighter-rouge">view</code>参数设定组件集合,也可以通过<code class="language-plaintext highlighter-rouge">getView()</code>取得视图并加以修改,以下是一些常用的参数:</p>
<table>
<thead>
<tr>
<th>参数</th>
<th>获得方法</th>
<th>设置方法</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">center</code></td>
<td><code class="language-plaintext highlighter-rouge">getCenter()</code></td>
<td><code class="language-plaintext highlighter-rouge">setCenter(中心)</code></td>
<td>地图中心对应的坐标</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">rotation</code></td>
<td><code class="language-plaintext highlighter-rouge">getRotation()</code></td>
<td><code class="language-plaintext highlighter-rouge">setRotation(角度)</code></td>
<td>方位角,0表示地图上方对应北</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">zoom</code></td>
<td><code class="language-plaintext highlighter-rouge">getZoom()</code></td>
<td><code class="language-plaintext highlighter-rouge">setZoom(缩放级)</code></td>
<td>缩放级,通常在0到20之间</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">minZoom</code></td>
<td><code class="language-plaintext highlighter-rouge">getMinZoom()</code></td>
<td><code class="language-plaintext highlighter-rouge">setMinZoom(缩放级)</code></td>
<td>最小缩放级</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">maxZoom</code></td>
<td><code class="language-plaintext highlighter-rouge">getMaxZoom()</code></td>
<td><code class="language-plaintext highlighter-rouge">setMaxZoom(缩放级)</code></td>
<td>最大缩放级</td>
</tr>
</tbody>
</table>
<p>由于用户可以拖动或缩放地图,我们可以用<code class="language-plaintext highlighter-rouge">map.on('moveend', 回调函数)</code>监察这些变化以便更新持久链接或者视图内的标记。</p>
<p>值得注意的是地图中位置的坐标一般按墨托卡投影(EPSG:3857)而非经纬度(EPSG:4326)计算,所以需要时要用<code class="language-plaintext highlighter-rouge">ol.proj.fromLonLat(坐标)</code>和<code class="language-plaintext highlighter-rouge">ol.proj.toLonLat(坐标)</code>进行转换。</p>
<h3 id="调整组件">调整组件</h3>
<p>在默认情况下,地图上有分别用于放大和缩小的“+”按钮与“-”按钮、一个用于显示地图版权信息的“i”按钮和一个用于重置旋转的按钮(仅在上方不向北时显示)。我们可以在创建地图时通过<code class="language-plaintext highlighter-rouge">controls</code>参数设定控件集合,比如以下把默认组件的工具提示改成中文:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">map</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nb">Map</span><span class="p">({</span>
<span class="na">target</span><span class="p">:</span> <span class="dl">'</span><span class="s1">map</span><span class="dl">'</span><span class="p">,</span>
<span class="na">controls</span><span class="p">:</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">defaults</span><span class="p">({</span>
<span class="na">attributionOptions</span><span class="p">:{</span><span class="na">tipLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">资料来源</span><span class="dl">'</span><span class="p">},</span>
<span class="na">rotateOptions</span><span class="p">:{</span><span class="na">tipLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">重置方位</span><span class="dl">'</span><span class="p">},</span>
<span class="na">zoomOptions</span><span class="p">:{</span><span class="na">zoomInTipLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">放大</span><span class="dl">'</span><span class="p">,</span><span class="na">zoomOutTipLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">缩小</span><span class="dl">'</span><span class="p">}</span>
<span class="p">}),</span>
<span class="na">layers</span><span class="p">:</span> <span class="p">[</span>
<span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">layer</span><span class="p">.</span><span class="nx">Tile</span><span class="p">({</span>
<span class="na">source</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">source</span><span class="p">.</span><span class="nx">OSM</span><span class="p">()</span>
<span class="p">})</span>
<span class="p">],</span>
<span class="na">view</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">View</span><span class="p">({</span>
<span class="na">center</span><span class="p">:</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">proj</span><span class="p">.</span><span class="nx">fromLonLat</span><span class="p">([</span><span class="mf">114.1490</span><span class="p">,</span><span class="mf">22.4825</span><span class="p">]),</span>
<span class="na">zoom</span><span class="p">:</span> <span class="mi">17</span>
<span class="p">})</span>
<span class="p">});</span>
</code></pre></div></div>
<p>也可以稍后通过<code class="language-plaintext highlighter-rouge">getControls()</code>取得控件集合并加以修改。比如说,以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/scale-line.html">比例尺</a>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">ScaleLine</span><span class="p">({</span><span class="na">unit</span><span class="p">:</span> <span class="dl">"</span><span class="s2">metric</span><span class="dl">"</span><span class="p">}));</span>
</code></pre></div></div>
<p>以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/full-screen.html">全屏开关</a>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">FullScreen</span><span class="p">());</span>
</code></pre></div></div>
<p>以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/zoomslider.html">缩放级滑动条</a>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">ZoomSlider</span><span class="p">());</span>
</code></pre></div></div>
<p>以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/navigation-controls.html">转到预定视图的按钮</a>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">ZoomToExtent</span><span class="p">({</span>
<span class="na">extent</span><span class="p">:</span> <span class="p">[</span><span class="nx">minx</span><span class="p">,</span> <span class="nx">miny</span><span class="p">,</span> <span class="nx">maxx</span><span class="p">,</span> <span class="nx">maxy</span><span class="p">]</span>
<span class="p">}));</span>
</code></pre></div></div>
<p>以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/overviewmap.html">缩略图开关</a>(打开后会在较大尺度的地图上用矩形标出当前视图对应的位置):</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">OverviewMap</span><span class="p">({</span>
<span class="na">layers</span><span class="p">:</span> <span class="p">[</span>
<span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">layer</span><span class="p">.</span><span class="nx">Tile</span><span class="p">({</span><span class="na">source</span><span class="p">:</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">source</span><span class="p">.</span><span class="nx">OSM</span><span class="p">()})</span>
<span class="p">],</span>
<span class="na">tipLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">缩略图</span><span class="dl">'</span><span class="p">,</span>
<span class="na">collapseLabel</span><span class="p">:</span><span class="dl">'</span><span class="s1">◹</span><span class="dl">'</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span><span class="dl">'</span><span class="s1">◳</span><span class="dl">'</span>
<span class="p">}));</span>
</code></pre></div></div>
<p>以下代码可以向地图<code class="language-plaintext highlighter-rouge">map</code>加上<a href="https://openlayers.org/en/latest/examples/mouse-position.html">显示鼠标位置对应坐标的控件</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>map.getControls().push(new ol.control.MousePosition({
coordinateFormat: ol.coordinate.createStringXY(4),
projection: 'EPSG:4326',
undefinedHTML: '&nbsp;'
}))
</code></pre></div></div>
<p>一般地,我们可以自己创建元素并用作控件:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">getControls</span><span class="p">().</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">control</span><span class="p">.</span><span class="nx">Control</span><span class="p">({</span><span class="na">element</span><span class="p">:</span><span class="nx">元素</span><span class="p">}));</span>
</code></pre></div></div>
<p>除了控件外,我们还可以在地图上放置一些<a href="https://openlayers.org/en/latest/examples/overlay.html">覆盖物</a>以呈现地理信息,例如以下代码可以把指定元素放置在对应于指定坐标的位置对上7个像素:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">map</span><span class="p">.</span><span class="nx">addOverlay</span><span class="p">(</span><span class="k">new</span> <span class="nx">ol</span><span class="p">.</span><span class="nx">Overlay</span><span class="p">({</span>
<span class="na">element</span><span class="p">:</span> <span class="nx">元素</span><span class="p">,</span>
<span class="na">position</span><span class="p">:</span> <span class="nx">坐标</span><span class="p">,</span>
<span class="na">offset</span><span class="p">:</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">7</span><span class="p">],</span>
<span class="na">stopEvent</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span>
<span class="na">positioning</span><span class="p">:</span> <span class="dl">'</span><span class="s1">bottom-center</span><span class="dl">'</span>
<span class="p">}));</span>
</code></pre></div></div>
<p>当然我们也可以用<code class="language-plaintext highlighter-rouge">removeOverlay</code>移除它们。</p>
<h3 id="调整交互">调整交互</h3>
<p>在默认的情况下,地图支持以下交互:</p>
<ul>
<li>同时按下<code class="language-plaintext highlighter-rouge">Shift</code>和<code class="language-plaintext highlighter-rouge">Alt</code>键时点击再拖动会旋转地图</li>
<li>双击时放大地图</li>
<li>拖动时移动视图</li>
<li>在触摸屏上扭手指时旋转地图</li>
<li>在触摸屏上夹手指时缩放地图</li>
<li>按方向键时移动视图</li>
<li>按加减键时缩放地图</li>
<li>滚动鼠标轮时缩放地图</li>
<li>按下<code class="language-plaintext highlighter-rouge">Shift</code>键再拖放时把选定区域放大</li>
</ul>
<p>此外,我们可以用<code class="language-plaintext highlighter-rouge">map.addInteraction(交互)</code>和<code class="language-plaintext highlighter-rouge">map.removeInteraction(交互)</code>增删交互操作。以下是一些常见交互:</p>
<table>
<thead>
<tr>
<th>操作</th>
<th>用途</th>
<th>相关事件</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.DragBox({})</code></td>
<td>通过拖动选取矩形</td>
<td><code class="language-plaintext highlighter-rouge">boxdrag</code>、<code class="language-plaintext highlighter-rouge">boxend</code>、<code class="language-plaintext highlighter-rouge">boxstart</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.DragAndDrop({source: 向量来源})</code></td>
<td>通过拖放接收特征</td>
<td><code class="language-plaintext highlighter-rouge">addfeatures</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.DragRotateAndZoom()</code></td>
<td>按下<code class="language-plaintext highlighter-rouge">Shift</code>键后拖动可关于缩放和旋转地图而保持中心不变</td>
<td> </td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Draw({source: 向量来源,type: 形状,style:样式})</code></td>
<td>绘画形状(<code class="language-plaintext highlighter-rouge">'Point'</code>、<code class="language-plaintext highlighter-rouge">'LineString'</code>、<code class="language-plaintext highlighter-rouge">'Polygon'</code>或<code class="language-plaintext highlighter-rouge">'Circle'</code>),可以用<code class="language-plaintext highlighter-rouge">getGeometry()</code>取得最近画的形状</td>
<td><code class="language-plaintext highlighter-rouge">drawend</code>、<code class="language-plaintext highlighter-rouge">drawstart</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Extent({})</code></td>
<td>通过拖动选取和修改矩形</td>
<td><code class="language-plaintext highlighter-rouge">extentchanged</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Modify({source: 向量来源})</code></td>
<td>通过拖动修改特征中的顶点和边</td>
<td><code class="language-plaintext highlighter-rouge">modifyend</code>、<code class="language-plaintext highlighter-rouge">modifystart</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Select({style:样式,multi:false,features:[]})</code></td>
<td>通过拖动选中特征</td>
<td><code class="language-plaintext highlighter-rouge">select</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Snap({source: 向量来源})</code></td>
<td>吸附到特征中的顶点和边</td>
<td> </td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">new ol.interaction.Translate({})</code></td>
<td>通过拖动移动特征</td>
<td><code class="language-plaintext highlighter-rouge">translateend</code>、<code class="language-plaintext highlighter-rouge">translatestart</code>、<code class="language-plaintext highlighter-rouge">translating</code></td>
</tr>
</tbody>
</table>
<h2 id="下一步">下一步</h2>
<p>我们仅仅介绍了OpenLayers的基本用法,但它其实还能实现很多其它功能,官方的<a href="https://openlayers.org/en/latest/examples/">例子</a>展示了它们的用法,更详细的信息参见<a href="https://openlayers.org/en/latest/apidoc/">API 文档</a>。</p>陈颂光在网页上提供地理信息是一个常见的需求,例如连锁店的网站需要展示各门店的位置和前往路线。OpenLayers就是其中一个经常用于显示地图的Javascript库,可以让用户交互式地与各种地理信息打交道。Deeplearning4j深度学习框架初探2019-09-07T00:00:00+00:002019-09-07T00:00:00+00:00https://www.chungkwong.cc/deeplearning4j<p>人工神经网络近年在机器学习算法中可谓一枝独秀,对自然语言处理(如机器翻译)和计算机视觉(如图像识别)等领域的进步起了重大作用。受惠于JVM久经考验的丰富生态,<a href="https://deeplearning4j.org">Deeplearning4j</a>(DL4J)深度学习库不但容易在各种平台上使用,而且性能优异(特别是方便借助Hadoop和Spark处理大数据),适合用于开发各类模式识别软件产品。</p>
<h2 id="深度学习概论">深度学习概论</h2>
<h3 id="机器学习与人工智能">机器学习与人工智能</h3>
<p>映射是数学最核心的概念,众多问题都可以视为映射:</p>
<ul>
<li>人脸识别可以视为一个从人脸图片集到人集的映射,把人脸图片对应到人脸所属的人</li>
<li>机器翻译可以视为一个从字符串集到字符串集的映射,把来自一个语言的字符串对应到另一语言中语义相同的某个字符串</li>
</ul>
<p>为了解决问题,对于给定的映射$f:X\to Y$,我们希望对于每个定义域中的值$x$,都可以用计算机算出$f(x)$。对于一些对计算机来说是精确叙述的问题如排序,这是可能严格做到的。但对于人脸识别和机器翻译之类的问题,由于涉及到计算机以外的复杂概念,通常不能期望能完全准确地算出。于是,我们退而求其次,找一个能够计算的映射$g$去近似$f$,并且在某种意义下$f$与$g$接近。在过去,人们通过观察去设计这个近似映射$g$,但对于复杂问题人手设计的启发式规则很快会变得难以维护,而且针对个别问题的方法往往过于特殊,导致重复劳动还不利于总结提高。为了用统一的框架解决不同的问题,人们又想办法自动地构造近似映射$g$。当然我们必须给出关于$f$的一些信息才可能完成这构造,通常给出映射在部分点处的值$(x_1,f(x_1)),\dots,(x_M,f(x_M))$,由此构造$g$的方法叫机器学习。显然即使给定了一个映射在一些点的值,它在其它点处的值仍然可以是任意的,机器学习只能建基于映射能够被相对“简单”的映射逼近的信念。机器学习通常的做法是先作出一类映射,然后从其中找出一个与已知数据最吻合的映射。可见,与基于大数定律的统计方法一样,只有在数据足够多时我们才能指望通过学习得出的映射$g$确实能近似$f$。幸运的是,随着互联网的兴起,数据源源不断地从人和各种传感器产生,收集数据变得容易,机器学习因而在许多问题变得可行。</p>
<p>机器学习现阶段还只能在一些特殊类型的问题上击败其它方法,不过它有更远大的愿景。在某种意义下,人也就是一个映射而已,考虑这个简单的模型:在时刻$t$,人接受来自感官的刺激$I_t$,结合经验$E_t$作出反应$O_t$,同时把经验丰富为$E_{t+1}$。也就是说,人的行为就由映射$(I_t,E_t)\mapsto (E_{t+1},O_t)$完全刻画。假如我们能够完美地拟合这个映射,我们就能用机器去逼真地模仿人的全部行为。既然人们自认为人有智能,与人越接近就越智能,那么可以认为这机器具备所谓的智能(而且是比图灵的会话不可分性更强意义下的智能)。因此,机器学习的一个终极目标就是实现人工智能。</p>
<h3 id="人工神经网络">人工神经网络</h3>
<p>由于数学方法一般在欧氏空间中最好处理,因此往往设计一种编码$e: X\to \mathbb{R}^k$把$f$定义域中的对象对应到某个固定维数的欧氏空间,再设计一种解码单射$d: \mathbb{R}^l\to Y$把某个固定维数的欧氏空间中向量对应到$f$值域中的对象,从而可以把问题归结为寻找近似映射$h:\mathbb{R}^k\to\mathbb{R}^l$再令$g=d\circ h\circ e$即可。例如做图像识别时,图像可以对应到各像素亮度组成的向量;而做文章分类时,文章可以对应到各单词频率(或TF-IDF)组成的向量。对于机器翻译之类的问题,输入和输出的字符串都可以是任意长度的,但通过引入一个特殊的结束符和中间量,也可以和上述的人一样归结为输入和输出有固定维数的映射。</p>
<p>给定一列数据点$\{(X_i=e(x_i),Y_i=d^{-1}(f(x_i)))\}^M_{i=1}$,机器学习首先假定$h$可以被某族映射$\{h(\cdot;\Theta)\}$逼近,然后算出参数$\Theta =(\theta_1,\dots,\theta_N)$的估计值使$f$与$g$在已知点处的值接近。</p>
<p>首先,我们考虑映射族的构造,前馈神经网络的思想就是用一些称为神经元的基本的函数族的多重复合来构造。神经元形如$h(z_1,\dots,z_k;a_1,\dots,a_k,b)=\phi (a_1z_1+\dots+a_kz_k+b)$,其中$\phi$称为激活函数,只有单个神经元且$\phi$为恒同时神经网络退化为线性回归。因为线性函数与线性函数的复合仍然线性,$\phi$大多取非线性函数。而在有待训练时确定的参数中,各$a_i$称为权重,$b$则称为偏移。</p>
<p>人工神经网络可以用图直观地表示,其中每个顶点是神经元,值沿着有向边在神经元间流动,直至到达输出神经元成为整个逼近映射的值的一个分量。设计人工神经网络时往往把它分为若干层,首层表示输入的各分量,最后一层表示输出的分量,边通常仅从一层指向下一层,一种粗暴的全连接设计就是让一层中各节点与下一层的所有节点都连接起来。所谓深度学习说是就是层数较多的意思,通常认为后面的层次保存了较高层次(更整体)的信息。反馈神经网络的图中也可能存在回路,与时序电路实现记忆的原理类似,这种设计被认为可模拟人“越想越像”的记忆,有利于上下文感知。神经网络的设计很大程度上是一门艺术,在实验前难以评判。</p>
<p>接下来,我们需要一个评估拟合好坏的指标,给定某损失函数$L$,值$L(Y_i,h(X_i))$越小越好。于是现在的问题是求出$\Theta$的值使$\frac{1}{M}\sum^M_{i=1}L(Y_i,h(X_i;\Theta))$最小,这是一个优化问题。标准的数值解法是梯度下降法,从由启发式规则得出的参数值$\Theta_0$出发,然后逐步向负梯度方向修正$\Theta_{k}=\Theta_{k-1}-\ell\nabla_{\Theta}(\frac{1}{M}\sum^M_{i=1}L(Y_i,h(X_i;\Theta)))\vert_{\Theta=\Theta_{k-1}}$,其中$\ell$称为学习率(太大容易不收敛,太小收敛则慢)。现实中因为$M$太大,直接按上式计算会太慢且占内存,所以通常不会把所有样本一起用来算梯度,而是把样本分成多个小批次,每次迭代轮流用。另外,迭代公式也有一些变种,例如引入动量项。应当指出,由于$h$不见得可微,而且往往不是凸函数,难以保证数值方法收敛于最小点。</p>
<p>最后指出,虽然上面主要谈监督学习。但某些非监督学习问题可转化为监督学习问题。例如有损压缩问题(类似的有语义散列问题)相当于寻找压缩函数$f:X\to Y$和解压函数$g:Y\to X$使$g\circ f$在某种意义下接近恒同映射,于是我们可以设计一个神经网络,各层中神经元个数先是递减再递增,输入数与输出数相同,数据集中数据同时用作输入和输出去训练网络,最后前半个神经网络就可作为压缩器而后半个神经网络就可作为解压器。类似技术还可以用于生成文本或图像之类。</p>
<h2 id="deeplearning4j的基本用法">Deeplearning4j的基本用法</h2>
<p>和使用其它库一样,首先需要满足依赖关系,例如对Maven项目,在<code class="language-plaintext highlighter-rouge">pom.xml</code>中加入以下样子的内容:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependencies></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.nd4j<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>${nd4j.backend}<span class="nt"></artifactId></span>
<span class="nt"><version></span>${nd4j.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.deeplearning4j<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>deeplearning4j-core<span class="nt"></artifactId></span>
<span class="nt"><version></span>${dl4j.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.slf4j<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>slf4j-jdk14<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.7.25<span class="nt"></version></span>
<span class="nt"></dependency></span>
<span class="nt"></dependencies></span>
<span class="nt"><properties></span>
<span class="c"><!-- 如果你有支持CUDA的GPU可以后端应改为:nd4j-cuda-9.0-platform、nd4j-cuda-9.2-platform 或 nd4j-cuda-10.0-platform --></span>
<span class="nt"><nd4j.backend></span>nd4j-native-platform<span class="nt"></nd4j.backend></span>
<span class="nt"><nd4j.version></span>1.0.0-beta4<span class="nt"></nd4j.version></span>
<span class="nt"><dl4j.version></span>1.0.0-beta4<span class="nt"></dl4j.version></span>
<span class="nt"></properties></span>
</code></pre></div></div>
<p>接着我们用一个例子说明deeplearning4j的用法,这个例子使用卷积神经网络识别MNIST数据集中的手写数字图片。</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/** *****************************************************************************
* Copyright (c) 2015-2019 Skymind, Inc.
*
* This program and the accompanying materials are made available under the
* terms of the Apache License, Version 2.0 which is available at
* https://www.apache.org/licenses/LICENSE-2.0.
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations
* under the License.
*
* SPDX-License-Identifier: Apache-2.0
***************************************************************************** */
package org.deeplearning4j.examples.mnist;
import java.io.*;
import java.util.*;
import org.datavec.api.io.labels.*;
import org.datavec.api.split.*;
import org.datavec.image.loader.*;
import org.datavec.image.recordreader.*;
import org.deeplearning4j.datasets.datavec.*;
import org.deeplearning4j.nn.conf.*;
import org.deeplearning4j.nn.conf.inputs.*;
import org.deeplearning4j.nn.conf.layers.*;
import org.deeplearning4j.nn.multilayer.*;
import org.deeplearning4j.nn.weights.*;
import org.deeplearning4j.optimize.listeners.*;
import org.deeplearning4j.util.*;
import org.nd4j.evaluation.classification.*;
import org.nd4j.linalg.activations.*;
import org.nd4j.linalg.dataset.api.iterator.*;
import org.nd4j.linalg.dataset.api.preprocessor.*;
import org.nd4j.linalg.learning.config.*;
import org.nd4j.linalg.lossfunctions.*;
import org.nd4j.linalg.schedule.*;
import org.slf4j.*;
/**
* Implementation of LeNet-5 for handwritten digits image classification on
* MNIST dataset (99% accuracy)
* <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf">[LeCun et al.,
* 1998. Gradient based learning applied to document recognition]</a>
* Some minor changes are made to the architecture like using ReLU and identity
* activation instead of sigmoid/tanh, max pooling instead of avg pooling and
* softmax output layer.
* <p>
* This example will download 15 Mb of data on the first run.
*
* @author hanlon
* @author agibsonccc
* @author fvaleri
* @author dariuszzbyrad
*/
public class MnistClassifier{
private static final Logger LOGGER=LoggerFactory.getLogger(MnistClassifier.class);
//从http://github.com/myleott/mnist_png/raw/master/mnist_png.tar.gz下载再解压到以下目录
private static final String BASE_PATH="数据集路径";
public static void main(String[] args) throws Exception{
int height=28; // height of the picture in px
int width=28; // width of the picture in px
int channels=1; // single channel for grayscale images
int outputNum=10; // 10 digits classification
int batchSize=54; // number of samples that will be propagated through the network in each iteration
int nEpochs=1; // number of training epochs
int seed=1234; // number used to initialize a pseudorandom number generator.
Random randNumGen=new Random(seed);
LOGGER.info("加载数据...");
File trainData=new File(BASE_PATH+"/mnist_png/training");
FileSplit trainSplit=new FileSplit(trainData,NativeImageLoader.ALLOWED_FORMATS,randNumGen);
ParentPathLabelGenerator labelMaker=new ParentPathLabelGenerator(); // use parent directory name as the image label
ImageRecordReader trainRR=new ImageRecordReader(height,width,channels,labelMaker);
trainRR.initialize(trainSplit);
DataSetIterator trainIter=new RecordReaderDataSetIterator(trainRR,batchSize,1,outputNum);
// pixel values from 0-255 to 0-1 (min-max scaling)
DataNormalization imageScaler=new ImagePreProcessingScaler();
imageScaler.fit(trainIter);
trainIter.setPreProcessor(imageScaler);
// vectorization of test data
File testData=new File(BASE_PATH+"/mnist_png/testing");
FileSplit testSplit=new FileSplit(testData,NativeImageLoader.ALLOWED_FORMATS,randNumGen);
ImageRecordReader testRR=new ImageRecordReader(height,width,channels,labelMaker);
testRR.initialize(testSplit);
DataSetIterator testIter=new RecordReaderDataSetIterator(testRR,batchSize,1,outputNum);
testIter.setPreProcessor(imageScaler); // same normalization for better results
LOGGER.info("网络配置和训练...");
Map<Integer,Double> learningRateSchedule=new HashMap<>();
learningRateSchedule.put(0,0.06);
learningRateSchedule.put(200,0.05);
learningRateSchedule.put(600,0.028);
learningRateSchedule.put(800,0.0060);
learningRateSchedule.put(1000,0.001);
MultiLayerConfiguration conf=new NeuralNetConfiguration.Builder()
.seed(seed)
.l2(0.0005) // ridge regression value
.updater(new Nesterovs(new MapSchedule(ScheduleType.ITERATION,learningRateSchedule)))
.weightInit(WeightInit.XAVIER)
.list()
.layer(new ConvolutionLayer.Builder(5,5)
.nIn(channels)
.stride(1,1)
.nOut(20)
.activation(Activation.IDENTITY)
.build())
.layer(new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2,2)
.stride(2,2)
.build())
.layer(new ConvolutionLayer.Builder(5,5)
.stride(1,1) // nIn need not specified in later layers
.nOut(50)
.activation(Activation.IDENTITY)
.build())
.layer(new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2,2)
.stride(2,2)
.build())
.layer(new DenseLayer.Builder().activation(Activation.RELU)
.nOut(500)
.build())
.layer(new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.nOut(outputNum)
.activation(Activation.SOFTMAX)
.build())
.setInputType(InputType.convolutionalFlat(height,width,channels)) // InputType.convolutional for normal image
.build();
MultiLayerNetwork net=new MultiLayerNetwork(conf);
net.init();
net.setListeners(new ScoreIterationListener(20));
LOGGER.info("Total num of params: {}",net.numParams());
// evaluation while training (the score should go down)
for(int i=0;i<nEpochs;i++){
net.fit(trainIter);
LOGGER.info("Completed epoch {}",i);
Evaluation eval=net.evaluate(testIter);
LOGGER.info(eval.stats());
trainIter.reset();
testIter.reset();
}
File ministModelPath=new File(BASE_PATH+"/minist-model.zip");
ModelSerializer.writeModel(net,ministModelPath,true);
LOGGER.info("The MINIST model has been saved in {}",ministModelPath.getPath());
}
}
</code></pre></div></div>
<p>下载并解压数据集后把<code class="language-plaintext highlighter-rouge">BASE_PATH</code>指向数据集所有目录,再运行上述类即可训练出一个卷积神经网络模型,模型会被保存到目录<code class="language-plaintext highlighter-rouge">BASE_PATH</code>下的<code class="language-plaintext highlighter-rouge">minist-model.zip</code>,另外会得到类似以下的测试结果(准确度约99%):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>========================Evaluation Metrics========================
# of classes: 10
Accuracy: 0.9900
Precision: 0.9900
Recall: 0.9899
F1 Score: 0.9899
Precision, recall & F1: macro-averaged (equally weighted avg. of 10 classes)
=========================Confusion Matrix=========================
0 1 2 3 4 5 6 7 8 9
---------------------------------------------------
972 0 1 0 0 0 1 2 3 1 | 0 = 0
0 1130 0 1 0 1 1 2 0 0 | 1 = 1
0 2 1022 1 1 0 0 4 2 0 | 2 = 2
0 0 1 1001 0 4 0 1 3 0 | 3 = 3
0 0 1 0 974 0 1 0 2 4 | 4 = 4
2 0 0 8 0 879 1 1 1 0 | 5 = 5
3 2 0 0 2 3 948 0 0 0 | 6 = 6
1 1 7 0 0 0 0 1014 1 4 | 7 = 7
0 0 2 1 0 0 0 2 964 5 | 8 = 8
0 0 1 2 5 0 0 4 1 996 | 9 = 9
Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times
==================================================================
</code></pre></div></div>
<p>值得一提,由于DL4J用到了本地库,默认情况下Maven会把所有所有平台的本地库都纳入到类路径,这可能导致类加载失败。这时可以在<code class="language-plaintext highlighter-rouge">mvn</code>命令后加上类似<code class="language-plaintext highlighter-rouge">-Djavacpp.platform=linux-x86_64</code>(改成你的平台)的选项来限制只加载适用平台的本地库。</p>
<h2 id="预处理">预处理</h2>
<p>如前所述,原始数据如文本、声音、图像等等要转化为欧氏空间中张量形式的才能进入神经网络,因此需要提取、转换和加载的过程(ETL)。</p>
<h3 id="提取">提取</h3>
<p><code class="language-plaintext highlighter-rouge">InputSplit</code>接口管理数据位置,它的实现包括:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">CollectionInputSplit</code>记录Uri数组或容器</li>
<li><code class="language-plaintext highlighter-rouge">FileSplit</code>记录根目录,并可设置递归、随机化和容许的文件格式</li>
<li><code class="language-plaintext highlighter-rouge">InputStreamInputSplit</code>记录<code class="language-plaintext highlighter-rouge">InputStream</code></li>
<li><code class="language-plaintext highlighter-rouge">ListStringSplit</code>记录<code class="language-plaintext highlighter-rouge">java.util.List<java.util.List<java.lang.String>></code></li>
<li><code class="language-plaintext highlighter-rouge">NumberedFileInputSplit</code>记录含序号文件名模式(模式中用<code class="language-plaintext highlighter-rouge">%d</code>表示序号)和序号范围</li>
<li><code class="language-plaintext highlighter-rouge">OutputStreamInputSplit</code>记录<code class="language-plaintext highlighter-rouge">OutputStream</code></li>
<li><code class="language-plaintext highlighter-rouge">StringSplit</code>记录<code class="language-plaintext highlighter-rouge">String</code></li>
<li><code class="language-plaintext highlighter-rouge">TransformSplit</code>记录一个<code class="language-plaintext highlighter-rouge">BaseInputSplit</code>和到URI的转换</li>
</ul>
<p><code class="language-plaintext highlighter-rouge">RecordReader</code>接口用于把原始数据转换为一系列记录,其中使用前用<code class="language-plaintext highlighter-rouge">void initialize(Configuration conf, InputSplit split)</code>或<code class="language-plaintext highlighter-rouge">void initialize(InputSplit split)</code>初始化它。它的常用实现包括:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">CollectionRecordReader</code>读入指定集合的元素集合作为记录</li>
<li><code class="language-plaintext highlighter-rouge">ComposableRecordReader</code>把多个指定<code class="language-plaintext highlighter-rouge">RecordReader</code>读入的对应记录连接成记录</li>
<li><code class="language-plaintext highlighter-rouge">ConcatenatingRecordReader</code>分别读入多个指定<code class="language-plaintext highlighter-rouge">RecordReader</code>读入的记录</li>
<li><code class="language-plaintext highlighter-rouge">CSVRecordReader</code>用于读入CSV记录,可指定跳过行数、分隔符和引号</li>
<li><code class="language-plaintext highlighter-rouge">CSVRegexRecordReader</code>用于读入CSV记录并对每个域按正则表达式的捕获组进一步分解,可指定跳过行数、分隔符、引号和各正则表达式</li>
<li><code class="language-plaintext highlighter-rouge">ExcelRecordReader</code>用于读入Excel记录,可指定跳过行数</li>
<li><code class="language-plaintext highlighter-rouge">FileRecordReader</code>用于读入文件</li>
<li><code class="language-plaintext highlighter-rouge">ImageRecordReader</code>用于读入图像,可指定高度、宽度、通道数、图像变换和标签生成器</li>
<li><code class="language-plaintext highlighter-rouge">InMemoryRecordReader</code>读入指定列表的元素列表作为记录</li>
<li><code class="language-plaintext highlighter-rouge">JacksonLineRecordReader</code>用于读入文件行,可指定域选择和对象映射</li>
<li><code class="language-plaintext highlighter-rouge">JacksonRecordReader</code>用于读入JSON、XML或YAML文件为记录,可指定域选择、对象映射、打乱与否、随机种子、标签生成器和标签位置</li>
<li><code class="language-plaintext highlighter-rouge">JDBCRecordReader</code>用于从关系式数据库读入记录,可指定SQL查询、数据元、元数据到记录查询和元数据索引</li>
<li><code class="language-plaintext highlighter-rouge">LibSvmRecordReader</code>用于读入libsvm记录</li>
<li><code class="language-plaintext highlighter-rouge">LineRecordReader</code>用于读入文件行</li>
<li><code class="language-plaintext highlighter-rouge">ListStringRecordReader</code>用于读入字符串列表</li>
<li><code class="language-plaintext highlighter-rouge">LocalTransformProcessRecordReader</code>用于转换各个记录可指定原<code class="language-plaintext highlighter-rouge">RecordReader</code>和<code class="language-plaintext highlighter-rouge">TransformProcess</code></li>
<li><code class="language-plaintext highlighter-rouge">MapFileRecordReader</code>用于读入Hadoop MapFile对应于同一键的全体值,可指定键索引和随机化</li>
<li><code class="language-plaintext highlighter-rouge">MatlabRecordReader</code>用于读入Matlab记录</li>
<li><code class="language-plaintext highlighter-rouge">NativeAudioRecordReader</code>用于读入音频,可指定是否加标签和标签列表</li>
<li><code class="language-plaintext highlighter-rouge">ObjectDetectionRecordReader</code>用于读入图像及对象在其中位置,可指定高度、宽度、通道数、网格高度、网格宽度、图像变换和标签生成器</li>
<li><code class="language-plaintext highlighter-rouge">RegexLineRecordReader</code>用于把文本的各行按捕获组分解为记录,可指定正则表达式和跳过行数</li>
<li><code class="language-plaintext highlighter-rouge">SVMLightRecordReader</code>用于读入SVMLight格式的记录</li>
<li><code class="language-plaintext highlighter-rouge">TfidfRecordReader</code>用于把记录转换为TF-IDF向量</li>
<li><code class="language-plaintext highlighter-rouge">WavFileRecordReader</code>用于读入WAV音频,可指定是否加标签和标签列表</li>
</ul>
<p>特别地<code class="language-plaintext highlighter-rouge">SequenceRecordReader</code>接口用于把原始数据转换为一系列记录列表:</p>
<table>
<thead>
<tr>
<th>类</th>
<th>记录列表</th>
<th>记录</th>
<th>参数</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">CodecRecordReader</code></td>
<td>视频</td>
<td>帧</td>
<td> </td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CollectionSequenceRecordReader</code></td>
<td>集合的集合</td>
<td>集合</td>
<td>集合的集合的集合</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CSVLineSequenceRecordReader</code></td>
<td>CSV文件的记录行</td>
<td>域</td>
<td>跳过行数、分隔符和引号</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CSVMultiSequenceRecordReader</code></td>
<td>CSV文件由指定列表分隔符分隔的记录列表</td>
<td>域</td>
<td>跳过行数、分隔符、引号、模式(<code class="language-plaintext highlighter-rouge">CONCAT</code>、<code class="language-plaintext highlighter-rouge">EQUAL_LENGTH</code>、<code class="language-plaintext highlighter-rouge">PAD</code>)、填充串</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CSVNLinesSequenceRecordReader</code></td>
<td>CSV文件每若干条记录</td>
<td>域</td>
<td>列表行数、跳过行数和分隔符</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CSVSequenceRecordReader</code></td>
<td>CSV文件</td>
<td>域</td>
<td>跳过行数和分隔符</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CSVVariableSlidingWindowRecordReader</code></td>
<td>CSV文件的滑动窗口中记录</td>
<td>域</td>
<td>列表行数上限、跳过行数、分隔符和滑动距离</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">InMemorySequenceRecordReader</code></td>
<td>列表的列表</td>
<td>列表</td>
<td>列表的列表的列表</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">JacksonLineSequenceRecordReader</code></td>
<td>文件的行</td>
<td>域</td>
<td>可指定域选择和对象映射</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LocalTransformProcessSequenceRecordReader</code></td>
<td>记录列表</td>
<td>转换后记录</td>
<td>原<code class="language-plaintext highlighter-rouge">SequenceRecordReader</code>和<code class="language-plaintext highlighter-rouge">TransformProcess</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">MapFileSequenceRecordReader</code></td>
<td>Hadoop MapFile对应于同一键的全体值</td>
<td>值</td>
<td>键索引和随机化</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">NativeCodecRecordReader</code></td>
<td>视频</td>
<td>帧</td>
<td> </td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RegexSequenceRecordReader</code></td>
<td>文本文件的行</td>
<td>正则表达式的捕获组</td>
<td>正则表达式、跳过行数、字符编码和错误处理器</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VideoRecordReader</code></td>
<td>图片目录</td>
<td>图片</td>
<td>高度、宽度、是否加标签和标签列表</td>
</tr>
</tbody>
</table>
<h3 id="转换">转换</h3>
<p>记录流往往需要经过转换才适合进入神经网络,这时就需要设置<code class="language-plaintext highlighter-rouge">TransformProcess</code>。</p>
<h4 id="设置输入模式">设置输入模式</h4>
<p>为此,我们需要描述转换前记录的模式,方法是先<code class="language-plaintext highlighter-rouge">new Schema.Builder()</code>,然后用以下方法增加列,最后调用<code class="language-plaintext highlighter-rouge">Schema build()</code>方法。</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumn</span><span class="o">(</span><span class="nc">ColumnMetaData</span> <span class="n">metaData</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnCategorical</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">stateNames</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnCategorical</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">stateNames</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">maxAllowedValue</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowNaN</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowInfinite</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">maxAllowedValue</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowNaN</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowInfinite</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnLong</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnLong</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnNDArray</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="kt">long</span><span class="o">[]</span> <span class="n">shape</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Double</span> <span class="n">maxAllowedValue</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowNaN</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowInfinite</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsFloat</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Float</span> <span class="n">maxAllowedValue</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowNaN</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">allowInfinite</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">names</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsLong</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">names</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsLong</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsLong</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">minAllowedValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">maxAllowedValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnsString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">pattern</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minIdxInclusive</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxIdxInclusive</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">regex</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">minAllowedLength</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">maxAllowedLength</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">regex</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">minAllowableLength</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span> <span class="n">maxAllowableLength</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnTime</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">joda</span><span class="o">.</span><span class="na">time</span><span class="o">.</span><span class="na">DateTimeZone</span> <span class="n">timeZone</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnTime</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">joda</span><span class="o">.</span><span class="na">time</span><span class="o">.</span><span class="na">DateTimeZone</span> <span class="n">timeZone</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">minValidValue</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">maxValidValue</span><span class="o">)</span>
<span class="nc">Schema</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addColumnTime</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">TimeZone</span> <span class="n">timeZone</span><span class="o">)</span>
</code></pre></div></div>
<p>另外也可以借助<code class="language-plaintext highlighter-rouge">Join.Builder</code>把多个模式连接起来。</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Join</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">setJoinColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">joinColumnNames</span><span class="o">)</span>
<span class="nc">Join</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">setJoinColumnsLeft</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">joinColumnNames</span><span class="o">)</span>
<span class="nc">Join</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">setJoinColumnsRight</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">joinColumnNames</span><span class="o">)</span>
<span class="nc">Join</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">setSchemas</span><span class="o">(</span><span class="nc">Schema</span> <span class="n">left</span><span class="o">,</span> <span class="nc">Schema</span> <span class="n">right</span><span class="o">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">Schema</code>(对应有<code class="language-plaintext highlighter-rouge">SequenceSchema</code>)类的以下静态方法可尝试推断模式:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Schema</span> <span class="nf">infer</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="nc">Writable</span><span class="o">></span> <span class="n">record</span><span class="o">)</span>
<span class="nc">Schema</span> <span class="nf">inferMultiple</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="nc">Writable</span><span class="o">>></span> <span class="n">record</span><span class="o">)</span>
</code></pre></div></div>
<h4 id="设置转换">设置转换</h4>
<p>为设置<code class="language-plaintext highlighter-rouge">TransformProcess</code>,方法是先<code class="language-plaintext highlighter-rouge">new TransformProcessBuilder()</code>,然后用以下方法设置步骤,最后调用<code class="language-plaintext highlighter-rouge">TransformProcess build()</code>方法。</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addConstantColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">ColumnType</span> <span class="n">newColumnType</span><span class="o">,</span> <span class="nc">Writable</span> <span class="n">fixedValue</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addConstantDoubleColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="kt">double</span> <span class="n">value</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addConstantIntegerColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">value</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">addConstantLongColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="kt">long</span> <span class="n">value</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">appendStringColumnTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">toAppend</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">calculateSortedRank</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">sortOnColumn</span><span class="o">,</span> <span class="nc">WritableComparator</span> <span class="n">comparator</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">calculateSortedRank</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">sortOnColumn</span><span class="o">,</span> <span class="nc">WritableComparator</span> <span class="n">comparator</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">ascending</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">categoricalToInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">categoricalToOneHot</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">conditionalCopyValueTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnToReplace</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">sourceColumn</span><span class="o">,</span> <span class="nc">Condition</span> <span class="n">condition</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">conditionalReplaceValueTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="nc">Writable</span> <span class="n">newValue</span><span class="o">,</span> <span class="nc">Condition</span> <span class="n">condition</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">conditionalReplaceValueTransformWithDefault</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="nc">Writable</span> <span class="n">yesVal</span><span class="o">,</span> <span class="nc">Writable</span> <span class="n">noVal</span><span class="o">,</span> <span class="nc">Condition</span> <span class="n">condition</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertFromSequence</span><span class="o">()</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToDouble</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">inputColumn</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToInteger</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">inputColumn</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToSequence</span><span class="o">()</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToSequence</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">keyColumns</span><span class="o">,</span> <span class="nc">SequenceComparator</span> <span class="n">comparator</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToSequence</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">keyColumn</span><span class="o">,</span> <span class="nc">SequenceComparator</span> <span class="n">comparator</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">convertToString</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">inputColumn</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">doubleColumnsMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">doubleMathFunction</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathFunction</span> <span class="n">mathFunction</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">doubleMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="kt">double</span> <span class="n">scalar</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">duplicateColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newName</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">duplicateColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">columnNames</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">newNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">filter</span><span class="o">(</span><span class="nc">Condition</span> <span class="n">condition</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">filter</span><span class="o">(</span><span class="nc">Filter</span> <span class="n">filter</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">floatColumnsMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">floatMathFunction</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathFunction</span> <span class="n">mathFunction</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">floatMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="kt">float</span> <span class="n">scalar</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">integerColumnsMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">integerMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="kt">int</span> <span class="n">scalar</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">integerToCategorical</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">categoryStateNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">integerToCategorical</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">Map</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Integer</span><span class="o">,</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">categoryIndexNameMap</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">integerToOneHot</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">minValue</span><span class="o">,</span> <span class="kt">int</span> <span class="n">maxValue</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">longColumnsMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">longMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="kt">long</span> <span class="n">scalar</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">ndArrayColumnsMathOpTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">ndArrayDistanceTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newColumnName</span><span class="o">,</span> <span class="nc">Distance</span> <span class="n">distance</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">firstCol</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">secondCol</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">ndArrayMathFunctionTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathFunction</span> <span class="n">mathFunction</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">ndArrayScalarOpTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">op</span><span class="o">,</span> <span class="kt">double</span> <span class="n">value</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">normalize</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="nc">Normalize</span> <span class="n">type</span><span class="o">,</span> <span class="nc">DataAnalysis</span> <span class="n">da</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">offsetSequence</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">columnsToOffset</span><span class="o">,</span> <span class="kt">int</span> <span class="n">offsetAmount</span><span class="o">,</span> <span class="nc">SequenceOffsetTransform</span><span class="o">.</span><span class="na">OperationType</span> <span class="n">operationType</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">reduce</span><span class="o">(</span><span class="nc">IAssociativeReducer</span> <span class="n">reducer</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">reduceSequence</span><span class="o">(</span><span class="nc">IAssociativeReducer</span> <span class="n">reducer</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">reduceSequenceByWindow</span><span class="o">(</span><span class="nc">IAssociativeReducer</span> <span class="n">reducer</span><span class="o">,</span> <span class="nc">WindowFunction</span> <span class="n">windowFunction</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">removeAllColumnsExceptFor</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">Collection</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">removeAllColumnsExceptFor</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">removeColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">Collection</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">removeColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">columnNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">renameColumn</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">oldName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">newName</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">renameColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">oldNames</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">newNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">reorderColumns</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">...</span> <span class="n">newOrder</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">replaceStringTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">Map</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">,</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">mapping</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">sequenceMovingWindowReduce</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="kt">int</span> <span class="n">lookback</span><span class="o">,</span> <span class="nc">ReduceOp</span> <span class="n">op</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">splitSequence</span><span class="o">(</span><span class="nc">SequenceSplit</span> <span class="n">split</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">stringMapTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">Map</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">,</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">mapping</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">stringRemoveWhitespaceTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">stringToCategorical</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">List</span><span class="o"><</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span><span class="o">></span> <span class="n">stateNames</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">stringToTimeTransform</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">column</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">format</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">joda</span><span class="o">.</span><span class="na">time</span><span class="o">.</span><span class="na">DateTimeZone</span> <span class="n">dateTimeZone</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">timeMathOp</span><span class="o">(</span><span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">String</span> <span class="n">columnName</span><span class="o">,</span> <span class="nc">MathOp</span> <span class="n">mathOp</span><span class="o">,</span> <span class="kt">long</span> <span class="n">timeQuantity</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">util</span><span class="o">.</span><span class="na">concurrent</span><span class="o">.</span><span class="na">TimeUnit</span> <span class="n">timeUnit</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">transform</span><span class="o">(</span><span class="nc">Transform</span> <span class="n">transform</span><span class="o">)</span>
<span class="nc">TransformProcess</span><span class="o">.</span><span class="na">Builder</span> <span class="nf">trimSequence</span><span class="o">(</span><span class="kt">int</span> <span class="n">numStepsToTrim</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">trimFromStart</span><span class="o">)</span>
</code></pre></div></div>
<h3 id="加载">加载</h3>
<p><code class="language-plaintext highlighter-rouge">DataSetIterator</code>接口用于迭代小批次数据(一个小批次应该足够大以保证有代表性,同时不宜太大以减低内存需求,通常每小批次32至128个样本比较合理),每次返回一个<code class="language-plaintext highlighter-rouge">DataSet</code>,最多有一个输入和一个输出数组。<code class="language-plaintext highlighter-rouge">MultiDataSetIterator</code>类似,但返回<code class="language-plaintext highlighter-rouge">MultiDataSet</code>。前者的实现包括:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">RecordReaderDataSetIterator</code>从<code class="language-plaintext highlighter-rouge">RecordReader</code>读取数据,可指定<code class="language-plaintext highlighter-rouge">RecordReader</code>、<code class="language-plaintext highlighter-rouge">WritableConverter</code>、批次大小、首个标签列、最后标签列、可能的标签数、最大批次数、回归/分类</li>
<li><code class="language-plaintext highlighter-rouge">CachingDataSetIterator</code>支持缓存</li>
<li><code class="language-plaintext highlighter-rouge">ExistingMiniBatchDataSetIterator</code>读入现有的小批次数据</li>
<li><code class="language-plaintext highlighter-rouge">KFoldIterator</code>支持k趟交叉验证</li>
<li><code class="language-plaintext highlighter-rouge">MiniBatchFileDataSetIterator</code>支持把数据分成小批次</li>
<li><code class="language-plaintext highlighter-rouge">MultipleEpochsIterator</code>支持多趟处理</li>
<li><code class="language-plaintext highlighter-rouge">SamplingDataSetIterator</code>支持随机抽样</li>
<li><code class="language-plaintext highlighter-rouge">ViewIterator</code>支持视图</li>
</ul>
<p>它们可以通过<code class="language-plaintext highlighter-rouge">setPreProcessor(DataSetPreProcessor preProcessor)</code>方法设置预处理器:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ImagePreProcessingScaler</code>通常用于把0到255转化为0到1</li>
<li><code class="language-plaintext highlighter-rouge">NormalizerMinMaxScaler</code>通常用于把最小值和最大值分别放到0和1</li>
<li><code class="language-plaintext highlighter-rouge">NormalizerStandardize</code>通常用于把各特征分别正规化为均值0方差1</li>
<li><code class="language-plaintext highlighter-rouge">VGG16ImagePreProcessor</code>通常用于减去平均RGB值</li>
</ul>
<p>注意部分需要统计值的预处理器用之前需要调用调用其<code class="language-plaintext highlighter-rouge">fit</code>方法。另外它们也有适用于<code class="language-plaintext highlighter-rouge">MultiDataSet</code>的版本如<code class="language-plaintext highlighter-rouge">ImageMultiPreProcessingScaler</code>、<code class="language-plaintext highlighter-rouge">MultiNormalizerHybrid</code>、<code class="language-plaintext highlighter-rouge">MultiNormalizerMinMaxScaler</code>、<code class="language-plaintext highlighter-rouge">MultiNormalizerStandardize</code>。</p>
<h2 id="网络配置">网络配置</h2>
<p>人工神经网络配置用类<code class="language-plaintext highlighter-rouge">MultiLayerConfiguration</code>的对象表示,要配置它可以使用的链式API,先创建Builder:<code class="language-plaintext highlighter-rouge">new NeuralNetConfiguration.Builder()</code>,然后通过调用它的各个方法进行配置,最后调用<code class="language-plaintext highlighter-rouge">build()</code>。</p>
<h3 id="训练配置">训练配置</h3>
<h4 id="激活函数">激活函数</h4>
<p>网络中的激活函数(神经元映射)可以用<code class="language-plaintext highlighter-rouge">activation</code>方法配置,部分常见值有(也可传递实现<code class="language-plaintext highlighter-rouge">IActivation</code>的类的对象):</p>
<table>
<thead>
<tr>
<th><code class="language-plaintext highlighter-rouge">Activation</code>枚举常量</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">CUBE</code></td>
<td>$f(x) = x^3$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ELU</code></td>
<td>$f(x) = \begin{cases}x & ,x>0\\ \alpha (\exp(x) - 1.0) & , 其它\end{cases}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">HARDSIGMOID</code></td>
<td>$f(x) = \min\{1, \max\{0, 0.2x + 0.5\}\}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">HARDTANH</code></td>
<td>$f(x) = \begin{cases}1 & ,x>1\\x & , 其它\end{cases}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">IDENTITY</code></td>
<td>$f(x) = x$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LEAKYRELU</code></td>
<td>$f(x) = \max\{0, x\} + \alpha \min\{0, x\}$ 其中默认 $\alpha=0.01$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RATIONALTANH</code></td>
<td>$f(x)= \mathrm{sgn}(x) ( 1 - 1/(1+\vert x\vert +x^2+1.41645x^4))\sim \tanh(x) $</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RECTIFIEDTANH</code></td>
<td>$f(x) = \max\{0, \tanh(x)\}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RELU</code></td>
<td>$f(x) = \begin{cases}x & ,x>0\\0 & , 其它\end{cases}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RELU6</code></td>
<td>$f(x) = \min\{\max\{x, \theta\}, 6\}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RRELU</code></td>
<td>$f(x) = \max\{0,x\} + alpha \min\{0, x\}$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SELU</code></td>
<td>正规化指数线性单位</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SIGMOID</code></td>
<td>$f(x) = 1 / (1 + \exp (-x))$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SOFTMAX</code></td>
<td>$f_i(x)=\exp(x_i - \theta) / \sum_j \exp(x_j - \theta)$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SOFTPLUS</code></td>
<td>$f(x) = \log (1+e^x)$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SOFTSIGN</code></td>
<td>$f(x) = x / (1+\vert x\vert)$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SWISH</code></td>
<td>$f(x) = x /(1+e^{-x})$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">TANH</code></td>
<td>$\tanh$</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">THRESHOLDEDRELU</code></td>
<td>$f(x) = \begin{cases}x & ,x>\theta\\0 & , 其它\end{cases}$</td>
</tr>
</tbody>
</table>
<h4 id="参数初始值">参数初始值</h4>
<p>网络中的初始参数设置方式可以用<code class="language-plaintext highlighter-rouge">weightInit</code>方法配置(类似有<code class="language-plaintext highlighter-rouge">biasInit</code>),部分常见值有:</p>
<table>
<thead>
<tr>
<th><code class="language-plaintext highlighter-rouge">WeightInit</code>枚举常量</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">DISTRIBUTION</code></td>
<td>由<code class="language-plaintext highlighter-rouge">dist</code>方法指定分布给出</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ZERO</code></td>
<td>0</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ONES</code></td>
<td>1</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SIGMOID_UNIFORM</code></td>
<td>均匀分布 U(-r,r) 其中 r=4*sqrt(6/(fanIn + fanOut))</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">NORMAL</code></td>
<td>正态分布,均值 0 ,方差 1/sqrt(fanIn)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LECUN_UNIFORM</code></td>
<td>均匀分布 U[-a,a] 其中 a=3/sqrt(fanIn).</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">UNIFORM</code></td>
<td>均匀分布 U[-a,a] 其中 a=1/sqrt(fanIn)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">XAVIER</code></td>
<td>正态分布,均值 0, 方差 2.0/(fanIn + fanOut)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">XAVIER_UNIFORM</code></td>
<td>均匀分布 U(-s,s) 其中 s = sqrt(6/(fanIn + fanOut))</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">XAVIER_FAN_IN</code></td>
<td>正态分布,均值0, 方差 1/fanIn</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RELU</code></td>
<td>正态分布,方差 2.0/nIn</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RELU_UNIFORM</code></td>
<td>均匀分布 U(-s,s) 其中 s = sqrt(6/fanIn)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">IDENTITY</code></td>
<td>单位方阵(只适用于方阵参数)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_NORMAL_FAN_IN</code></td>
<td>正态分布,均值 0, 方差 1.0/(fanIn)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_NORMAL_FAN_OUT</code></td>
<td>正态分布,均值 0, 方差 1.0/(fanOut)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_NORMAL_FAN_AVG</code></td>
<td>正态分布,均值 0, 方差 1.0/((fanIn + fanOut)/2)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_UNIFORM_FAN_IN</code></td>
<td>均匀分布 U[-a,a] 其中 a=3.0/(fanIn)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_UNIFORM_FAN_OUT</code></td>
<td>均匀分布 U[-a,a] 其中 a=3.0/(fanOut)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">VAR_SCALING_UNIFORM_FAN_AVG</code></td>
<td>均匀分布 U[-a,a] 其中 a=3.0/((fanIn + fanOut)/2)</td>
</tr>
</tbody>
</table>
<h4 id="优化器">优化器</h4>
<p>网络中的优化器可以用<code class="language-plaintext highlighter-rouge">updater</code>方法配置(类似有<code class="language-plaintext highlighter-rouge">biasUpdater</code>),部分常见类有<code class="language-plaintext highlighter-rouge">AdaDelta</code>、<code class="language-plaintext highlighter-rouge">AdaGrad</code>、<code class="language-plaintext highlighter-rouge">AdaMax</code>、<code class="language-plaintext highlighter-rouge">Adam</code>、<code class="language-plaintext highlighter-rouge">AMSGrad</code>、<code class="language-plaintext highlighter-rouge">Nadam</code>、<code class="language-plaintext highlighter-rouge">Nesterovs</code>、<code class="language-plaintext highlighter-rouge">NoOp</code>、<code class="language-plaintext highlighter-rouge">RmsProp</code>、<code class="language-plaintext highlighter-rouge">Sgd</code>。</p>
<p>支持学习率的优化器也支持学习率调度,以便在不同迭代使用不同的学习率(通常在后面迭代使用更小的学习率),以下是一些实现<code class="language-plaintext highlighter-rouge">ISchedule</code>的类:</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ExponentialSchedule</code></td>
<td>value(i) = initialValue * gamma^i</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">InverseSchedule</code></td>
<td>value(i) = initialValue * (1 + gamma * i)^(-power)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">MapSchedule</code></td>
<td>基于用户提供的映射必须为iteration/epoch为 0时提供值</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">PolySchedule</code></td>
<td>value(i) = initialValue * (1 + i/maxIter)^(-power)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SigmoidSchedule</code></td>
<td>value(i) = initialValue * 1.0 / (1 + exp(-gamma * (iter - stepSize)))</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">StepSchedule</code></td>
<td>value(i) = initialValue * gamma^( floor(iter/step) )</td>
</tr>
</tbody>
</table>
<h4 id="正则化">正则化</h4>
<p>可以用<code class="language-plaintext highlighter-rouge">l1(0.1)</code>、<code class="language-plaintext highlighter-rouge">l2(0.2)</code>对参数作L1、L2正则化。可以用<code class="language-plaintext highlighter-rouge">l1Bias(0.1)</code>、<code class="language-plaintext highlighter-rouge">l2Bias(0.2)</code>对偏移作L1、L2正则化。另外每轮迭代后可以作梯度正规化和其它约束。</p>
<h4 id="预防过度拟合">预防过度拟合</h4>
<p>如要在训练阶段中修改激活的值,可用<code class="language-plaintext highlighter-rouge">dropOut</code>方法设置保持概率或修改方法:</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">AlphaDropout</code></td>
<td>企图同时保持均值和方差</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Dropout</code></td>
<td>每个激活x以概率1-p置为0,以概率p设为x/p</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">GaussianDropout</code></td>
<td>加入乘性1均值的高斯噪声</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">GaussianNoise</code></td>
<td>加入加性0均值的高斯噪声</td>
</tr>
</tbody>
</table>
<p>如要在训练阶段把修改参数的值,可用<code class="language-plaintext highlighter-rouge">weightNoise</code>方法设置:</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">DropConnect</code></td>
<td>每个参数x以概率1-p置为0,以概率p设为x/p</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">WeightNoise</code></td>
<td>把加性或乘性的特定分布噪声加入到权重</td>
</tr>
</tbody>
</table>
<h3 id="层">层</h3>
<p>要创建<code class="language-plaintext highlighter-rouge">MultiLayerNetwork</code>,调用<code class="language-plaintext highlighter-rouge">list()</code>方法后再使用<code class="language-plaintext highlighter-rouge">layer</code>方法可以加入层。另外可以设置以下选项:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">pretrain(boolean)</code>方法可设置非监督训练</li>
<li><code class="language-plaintext highlighter-rouge">backprop(boolean)</code>方法可设置向后传播</li>
<li><code class="language-plaintext highlighter-rouge">setInputType(InputType)</code>方法可设置输入类型</li>
</ul>
<h4 id="前馈层">前馈层</h4>
<p>前馈层是基本的层的构造。</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">DenseLayer</code></td>
<td>全连通层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">EmbeddingLayer</code></td>
<td>输入正整数输出向量,只能用于首层</td>
</tr>
</tbody>
</table>
<h4 id="输出层">输出层</h4>
<p>输出层用作最后一层,可设置损失函数。</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">OutputLayer</code></td>
<td>标准的MLP/CNN分类/回归输出层,内置全连通层,二维输入和输出(每个样本一个行向量)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LossLayer</code></td>
<td>无参输出层,只有损失和激活函数,二维输入和输出(每个样本一个行向量),要求 nIn = nOut</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RnnOutputLayer</code></td>
<td>用于反馈神经网络,3维(时间序列)输入和输出,内置时分全连通层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">RnnLossLayer</code></td>
<td><code class="language-plaintext highlighter-rouge">RnnOutputLayer</code>的无参版本,3维输入和输出</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CnnLossLayer</code></td>
<td>CNN中对每个位置作出预测,无参数,输入输出形如[minibatch, depth, height, width]</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Yolo2OutputLayer</code></td>
<td>用于图像中对象检测</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CenterLossOutputLayer</code></td>
<td><code class="language-plaintext highlighter-rouge">OutputLayer</code>企图最小化类中激活间距离的变种</td>
</tr>
</tbody>
</table>
<h4 id="卷积层">卷积层</h4>
<p>卷积层用于构建卷积神经网格,通常在图像处理中用于提取特征。</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ConvolutionLayer</code>/<code class="language-plaintext highlighter-rouge">Convolution2D</code></td>
<td>标准二维卷积层,输入输出形如 [minibatch,depth,height,width]</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Convolution1DLayer</code>/<code class="language-plaintext highlighter-rouge">Convolution1D</code></td>
<td>标准一维卷积层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Deconvolution2DLayer</code></td>
<td>转置卷积,输出通常比输入大</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SeparableConvolution2DLayer</code></td>
<td>分深度的卷积层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SubsamplingLayer</code></td>
<td>通过最大值、平均或p范数缩小</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Subsampling1DLayer</code></td>
<td>一维的采样</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Upsampling2D</code></td>
<td>通过重复行/列的值放大</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Upsampling1D</code></td>
<td>一维的放大</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Cropping2D</code></td>
<td>裁剪层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ZeroPaddingLayer</code></td>
<td>在边沿填充0</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ZeroPadding1DLayer</code></td>
<td>一维版本的填充</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SpaceToDepth</code></td>
<td>把两个空间维数据按块转换为通道维</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SpaceToBatch</code></td>
<td>把两个空间维数据按块转换为批次维</td>
</tr>
</tbody>
</table>
<h4 id="反馈层">反馈层</h4>
<p>反馈层用于构建反馈神经网格,通常用于处理时间序列如文本。</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">LSTM</code></td>
<td>没有窥孔连接的LSTM RNN,支持CuDNN</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">GravesLSTM</code></td>
<td>有窥孔连接LSTM RNN,不支持CuDNN (故对于GPU, 宜用LSTM)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">Bidirectional</code></td>
<td>把单向的RNN包装成双向RNN(前向和反向有独立的参数)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SimpleRnn</code></td>
<td>标准/‘vanilla’ RNN层,由于大多长时依赖而不实用</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LastTimeStep</code></td>
<td>提取所包装(非双向)RNN层的最后时间步把[minibatch, size, timeSeriesLength]转换为 [minibatch, size]</td>
</tr>
</tbody>
</table>
<h4 id="非监督层">非监督层</h4>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">VariationalAutoencoder</code></td>
<td>编码解码器的可变实现,支持多种重构分布</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">AutoEncoder</code></td>
<td>标准去噪自动编码器层</td>
</tr>
</tbody>
</table>
<h4 id="其它层">其它层</h4>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">GlobalPoolingLayer</code></td>
<td>求和、平均、最大值或p范数,对RNN/时间序列输入[minibatch, size, timeSeriesLength]输出[minibatch, size],对CNN输入[minibatch, depth, h, w]输出[minibatch, depth]</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ActivationLayer</code></td>
<td>对输入应用激活函数</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">DropoutLayer</code></td>
<td>把丢弃实现为层</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">BatchNormalization</code></td>
<td>批次正规化 2d (前馈), 3d (时间序列,参数与时间无关) 或 4d (CNN,参数与空间位置无关)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">LocalResponseNormalization</code></td>
<td>CNN的局部响应正规化层,不常用</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">FrozenLayer</code></td>
<td>用于转移学习的冻结层(进一步训练时参数不变)</td>
</tr>
</tbody>
</table>
<h3 id="顶点">顶点</h3>
<p>要创建更灵活的<code class="language-plaintext highlighter-rouge">ComputationGraph</code>,可以调用<code class="language-plaintext highlighter-rouge">graphBuilder()</code>后使用<code class="language-plaintext highlighter-rouge">addVertex</code>方法。</p>
<table>
<thead>
<tr>
<th>类</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ElementWiseVertex</code></td>
<td>对输入元素进行按分量运算如加、减、乘、平均、最值</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">L2NormalizeVertex</code></td>
<td>用L2范数正规化输入</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">L2Vertex</code></td>
<td>计算两个数组间的L2距离</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">MergeVertex</code></td>
<td>沿维数1合并输入产生更大的输出数组</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">PreprocessorVertex</code></td>
<td>只有<code class="language-plaintext highlighter-rouge">InputPreProcessor</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ReshapeVertex</code></td>
<td>进行任意数组重整,但通常应首先考虑预处理器</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ScaleVertex</code></td>
<td>把输入乘以一个常数</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ShiftVertex</code></td>
<td>把输入加上一个常数</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">StackVertex</code></td>
<td>沿维数0合并输入产生更大的输出数组</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SubsetVertex</code></td>
<td>沿维数1(nOut/通道)取输入的子集</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">UnstackVertex</code></td>
<td>沿维数0(小批次)取输入的子集</td>
</tr>
</tbody>
</table>
<h2 id="训练与使用">训练与使用</h2>
<p>有个神经网络配置<code class="language-plaintext highlighter-rouge">conf</code>后,可以通过<code class="language-plaintext highlighter-rouge">new MultiLayerNetwork(conf)</code>创建网络,然后应该调用<code class="language-plaintext highlighter-rouge">init()</code>初始化它。</p>
<p>为了在训练过程中能了解神经网络的状态,可以通过调用<code class="language-plaintext highlighter-rouge">setListeners(TrainingListener...)</code>注册监听器,它们会在训练期每个迭代完成后(或其它元事件)被调用。以下是一些有用的监听器:</p>
<table>
<thead>
<tr>
<th>类</th>
<th>用途</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ScoreIterationListener</code></td>
<td>每若干个迭代记录损失函数得分到日志</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">PerformanceListener</code></td>
<td>每若干个迭代记录性能信息到日志</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">EvaluativeListener</code></td>
<td>每若干个迭代用测试集评估性能</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CheckpointListener</code></td>
<td>周期性地保存检查点</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">StatsListener</code></td>
<td>用于web训练界面</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">CollectScoresIterationListener</code></td>
<td>每若干个迭代记录损失函数得分到一个列表</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">TimeIterationListener</code></td>
<td>估计训练所需的剩余时间</td>
</tr>
</tbody>
</table>
<p>例如为了可视化地观察,可以使用以下代码在<code class="language-plaintext highlighter-rouge">localhost:9000</code>设立基于web的用户界面:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">UIServer</span> <span class="n">uiServer</span> <span class="o">=</span> <span class="nc">UIServer</span><span class="o">.</span><span class="na">getInstance</span><span class="o">();</span>
<span class="nc">StatsStorage</span> <span class="n">statsStorage</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">InMemoryStatsStorage</span><span class="o">();</span>
<span class="n">网络</span><span class="o">.</span><span class="na">setListeners</span><span class="o">(</span><span class="k">new</span> <span class="nc">StatsListener</span><span class="o">(</span><span class="n">statsStorage</span><span class="o">),</span><span class="k">new</span> <span class="nc">ScoreIterationListener</span><span class="o">(</span><span class="mi">10</span><span class="o">));</span>
<span class="n">uiServer</span><span class="o">.</span><span class="na">attach</span><span class="o">(</span><span class="n">statsStorage</span><span class="o">);</span>
</code></pre></div></div>
<p>要训练网络只用调用<code class="language-plaintext highlighter-rouge">fit</code>方法传入数据集迭代器,然后可以用<code class="language-plaintext highlighter-rouge">evaluate</code>等方法评估分类或回归效果。</p>
<p>训练完的网络可以保存起来供以后使用,这时可以使用<code class="language-plaintext highlighter-rouge">ModelSerializer</code>类的静态方法<code class="language-plaintext highlighter-rouge">writeModel</code>和<code class="language-plaintext highlighter-rouge">restoreMultiLayerNetwork</code>可以把网络序列化和反序列化。这使得增量训练成为可能。</p>
<p>最后指出,有时我们希望微调一个现有的神经网络,修改部分层和部分参数而保持其它部分不变,这称为转移学习。<code class="language-plaintext highlighter-rouge">TransferLearning.Builder</code>类可以做这种事情。</p>
<h2 id="下一步">下一步</h2>
<p>我们仅仅介绍了deeplearning4j的基本用法,但它其实还有很多其它功能,例如:</p>
<ul>
<li>当需要训练大型神经网络时,可以借助Spark用多台机器实现分布式训练(推荐的梯度分享实现需要依赖项<code class="language-plaintext highlighter-rouge">dl4j-spark-parameterserver_2.11</code>,老的参数平均实现则需要依赖项<code class="language-plaintext highlighter-rouge">dl4j-spark_2.11</code>)。</li>
<li>神经网络配置中有许多元参数如学习率,Arbiter可以自动寻找适合您的数据的元参数(需要依赖项<code class="language-plaintext highlighter-rouge">arbiter-deeplearning4j</code>、<code class="language-plaintext highlighter-rouge">arbiter-ui_2.11</code>)。</li>
<li>如果希望处理自然语言,不但有分句、分词、词频计算等基本工具,还有生成和比对语义散列等工具。</li>
<li>如果您有现成的Keras的HDF5格式模型,运气不太差的话可以通过<code class="language-plaintext highlighter-rouge">KerasModelImport.importKerasSequentialModelAndWeights</code>之类的方法导入它(需要依赖项<code class="language-plaintext highlighter-rouge">deeplearning4j-modelimport</code>)。</li>
<li>zoo提供了一些现成的模型(部分更提供训练过的参数)如AlexNet、Darknet19、FaceNetNN4Small2、InceptionResNetV1、LeNet、ResNet50、SimpleCNN、TextGenerationLSTM、TinyYOLO、VGG16、VGG19,可以直接使用或作为修改的基础(需要依赖项<code class="language-plaintext highlighter-rouge">deeplearning4j-zoo</code>)。</li>
<li>如果您需要与底层的张量<code class="language-plaintext highlighter-rouge">INDArray</code>打交道,Nd4j库提供了相关的运算。</li>
</ul>
<p>更详细的信息参见<a href="https://deeplearning4j.org/api/latest/">API 文档</a>,另外可以参考官方的<a href="https://github.com/deeplearning4j/dl4j-examples">例子</a>。</p>陈颂光人工神经网络近年在机器学习算法中可谓一枝独秀,对自然语言处理(如机器翻译)和计算机视觉(如图像识别)等领域的进步起了重大作用。受惠于JVM久经考验的丰富生态,Deeplearning4j(DL4J)深度学习库不但容易在各种平台上使用,而且性能优异(特别是方便借助Hadoop和Spark处理大数据),适合用于开发各类模式识别软件产品。用ANTLR解析领域特定语言2019-05-02T00:00:00+00:002019-05-02T00:00:00+00:00https://www.chungkwong.cc/antlr<p>在开发各种软件的过程中难免要与各式各样的小语言打交道,例如要读取不同格式的数据(特别是配置)文件。虽然我们可以从头开始自己写程序去解析它们,但这样往往过于耗时且难以维护。ANTLR是Lex(或Flex)与YACC(或Bison)在Java世界的一个代用品,可以根据词法和语法自动生成解析器。</p>
<h2 id="入门">入门</h2>
<h3 id="编写语法">编写语法</h3>
<p>处理语言的常规方法是先把输入文本分成单词序列,然后生成以词为叶子的解析树。分词的规则称为词法,而构建解析树的规则称为语法,不同语言有不同的词法和语法。在下例中,我们参考<a href="http://json.org/">json.org</a>写下JSON的词法和语法到文件<code class="language-plaintext highlighter-rouge">JSON.g4</code>中:</p>
<pre><code class="language-antlr">grammar JSON;
json
: value
;
value
:
| object
| array
| STRING
| NUMBER
| 'true'
| 'false'
| 'null'
;
object
: '{' member (',' member)* '}'
| '{' '}'
;
member
: STRING ':' value
;
array
: '[' value (',' value)* ']'
| '[' ']'
;
STRING
: '"' (ESC | ~ ["\\])* '"'
;
fragment ESC
: '\\' (["\\/bnrt] | 'u' HEX HEX HEX HEX)
;
fragment HEX
: [0-9a-fA-F]
;
NUMBER
: '-'? INT ('.' [0-9]+)? EXP?
;
fragment INT
: '0' | [1-9] [0-9]*
;
fragment EXP
: [Ee] [+\-]? [0-9]+
;
WS
: [ \t\n\r] + -> skip
;
</code></pre>
<p>应该说为ANTLR写的这种上下文无关语法很易读,我们下面还会更详细地介绍语法文件怎样写。以下指出几点:</p>
<ul>
<li>开首的<code class="language-plaintext highlighter-rouge">grammar JSON;</code>中<code class="language-plaintext highlighter-rouge">JSON</code>必须与文件名<code class="language-plaintext highlighter-rouge">JSON.g4</code>中的<code class="language-plaintext highlighter-rouge">JSON</code>相同,要改就要一起改。</li>
<li>然后是一些形如<code class="language-plaintext highlighter-rouge">规则名 : 分支1 | ... | 分支N ;</code>的规则,一段文本匹配规则相当于它匹配其中一个分支,分支中可用单引号包围要按字面匹配的字符串、用子规则名表示按子规则匹配,另外还可用一些类似正则表达式的记号如<code class="language-plaintext highlighter-rouge">?</code>、<code class="language-plaintext highlighter-rouge">*</code>、<code class="language-plaintext highlighter-rouge">+</code>、<code class="language-plaintext highlighter-rouge">|</code>和<code class="language-plaintext highlighter-rouge">()</code>。
<ul>
<li>语法规则名以小写字母开始,不同语法规则生成解析树不同类型的结点,各分支说明这种结点的子结点序列可以是什么样子</li>
<li>词法规则名以大写字母开始,不同词法规则生成不同语类的词,各分支说明这类词可以是什么样子</li>
<li>以<code class="language-plaintext highlighter-rouge">fragment </code>开首的规则可以被词法规则引用以便重用代码,但本身不会生成对解析器可见的词</li>
</ul>
</li>
</ul>
<h3 id="生成解析器">生成解析器</h3>
<p>写好语法后我们让ANTLR自动为我们生成解析器代码。在首次使用前需要先下载ANTLR如<code class="language-plaintext highlighter-rouge">wget https://www.antlr.org/download/antlr-4.7.2-complete.jar</code>。接着就可以运行ANTLR<code class="language-plaintext highlighter-rouge">java -jar antlr-4.7.2-complete.jar -package com.github.chungkwong.json -o src JSON.g4</code>,其中<code class="language-plaintext highlighter-rouge">-o</code>选项用于指定输出位置、<code class="language-plaintext highlighter-rouge">-package</code>选项用于指定生成类所属的包。(版本号可以自行改成最新的)</p>
<table>
<thead>
<tr>
<th>选项</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">-o 目录</code></td>
<td>指定输出目录</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-lib 目录</code></td>
<td>指定存放被导入文件的目录</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-atn</code></td>
<td>生成ATN图</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-encoding 编码</code></td>
<td>语法文件的编码方式</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-message-format 格式</code></td>
<td>指定信息的格式:<code class="language-plaintext highlighter-rouge">antlr</code>、<code class="language-plaintext highlighter-rouge">gnu</code>或<code class="language-plaintext highlighter-rouge">vs2005</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-long-messages</code></td>
<td>显示异常的详情</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-listener</code></td>
<td>生成解析树事件侦听器(默认)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-no-listener</code></td>
<td>不生成解析树事件侦听器</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-visitor</code></td>
<td>生成解析树访问器</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-no-visitor</code></td>
<td>不生成解析树访问器(默认)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-package 包名</code></td>
<td>指定生成代码所在的包</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-depend</code></td>
<td>生成文件依赖关系</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-D键=值</code></td>
<td>设置或覆盖语法文件的选项</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-Werror</code></td>
<td>把警告当作错误</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-XdbgST</code></td>
<td>打开StringTemplate可视化器查看生成代码</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-XdbgSTWait</code></td>
<td>等待可视化器被关闭</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-Xforce-atn</code></td>
<td>对所有预测用ATN模拟器</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-Xlog</code></td>
<td>把日志写到<code class="language-plaintext highlighter-rouge">antlr-时间.log</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">-Xexact-output-dir</code></td>
<td>把所有输出放到<code class="language-plaintext highlighter-rouge">-o</code>指定的目录中(忽略包)</td>
</tr>
</tbody>
</table>
<p>为了方便使用也可以把JAR文件加到类路径并设置别名,如:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /usr/local/lib
wget https://www.antlr.org/download/antlr-4.7.2-complete.jar
<span class="nb">export </span><span class="nv">CLASSPATH</span><span class="o">=</span><span class="s2">".:/usr/local/lib/antlr-4.7.2-complete.jar:</span><span class="nv">$CLASSPATH</span><span class="s2">"</span>
<span class="nb">alias </span><span class="nv">antlr4</span><span class="o">=</span><span class="s1">'java -Xmx500M -cp "/usr/local/lib/antlr-4.7.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool'</span>
<span class="nb">alias </span><span class="nv">grun</span><span class="o">=</span><span class="s1">'java -Xmx500M -cp "/usr/local/lib/antlr-4.7.2-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'</span>
</code></pre></div></div>
<p>以下我们了解一下生成文件的结构。<code class="language-plaintext highlighter-rouge">JSONParser</code>是解析器本身,对每条语法规则有一个同名方法用于以它作为开始规则展开解析,还有一个<code class="language-plaintext highlighter-rouge">Context</code>类记录上下文(包括取得子规则上下文的方法),以下略去部分代码:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.github.chungkwong.json</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.atn.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.dfa.DFA</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.misc.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.tree.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Iterator</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.ArrayList</span><span class="o">;</span>
<span class="nd">@SuppressWarnings</span><span class="o">({</span><span class="s">"all"</span><span class="o">,</span> <span class="s">"warnings"</span><span class="o">,</span> <span class="s">"unchecked"</span><span class="o">,</span> <span class="s">"unused"</span><span class="o">,</span> <span class="s">"cast"</span><span class="o">})</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">JSONParser</span> <span class="kd">extends</span> <span class="nc">Parser</span> <span class="o">{</span>
<span class="kd">static</span> <span class="o">{</span> <span class="nc">RuntimeMetaData</span><span class="o">.</span><span class="na">checkVersion</span><span class="o">(</span><span class="s">"4.7.2"</span><span class="o">,</span> <span class="nc">RuntimeMetaData</span><span class="o">.</span><span class="na">VERSION</span><span class="o">);</span> <span class="o">}</span>
<span class="kd">protected</span> <span class="kd">static</span> <span class="kd">final</span> <span class="no">DFA</span><span class="o">[]</span> <span class="n">_decisionToDFA</span><span class="o">;</span>
<span class="kd">protected</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">PredictionContextCache</span> <span class="n">_sharedContextCache</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">PredictionContextCache</span><span class="o">();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span>
<span class="no">T__0</span><span class="o">=</span><span class="mi">1</span><span class="o">,</span> <span class="no">T__1</span><span class="o">=</span><span class="mi">2</span><span class="o">,</span> <span class="no">T__2</span><span class="o">=</span><span class="mi">3</span><span class="o">,</span> <span class="no">T__3</span><span class="o">=</span><span class="mi">4</span><span class="o">,</span> <span class="no">T__4</span><span class="o">=</span><span class="mi">5</span><span class="o">,</span> <span class="no">T__5</span><span class="o">=</span><span class="mi">6</span><span class="o">,</span> <span class="no">T__6</span><span class="o">=</span><span class="mi">7</span><span class="o">,</span> <span class="no">T__7</span><span class="o">=</span><span class="mi">8</span><span class="o">,</span> <span class="no">T__8</span><span class="o">=</span><span class="mi">9</span><span class="o">,</span>
<span class="no">STRING</span><span class="o">=</span><span class="mi">10</span><span class="o">,</span> <span class="no">NUMBER</span><span class="o">=</span><span class="mi">11</span><span class="o">,</span> <span class="no">WS</span><span class="o">=</span><span class="mi">12</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span>
<span class="n">RULE_json</span> <span class="o">=</span> <span class="mi">0</span><span class="o">,</span> <span class="n">RULE_value</span> <span class="o">=</span> <span class="mi">1</span><span class="o">,</span> <span class="n">RULE_object</span> <span class="o">=</span> <span class="mi">2</span><span class="o">,</span> <span class="n">RULE_member</span> <span class="o">=</span> <span class="mi">3</span><span class="o">,</span> <span class="n">RULE_array</span> <span class="o">=</span> <span class="mi">4</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">ruleNames</span> <span class="o">=</span> <span class="o">{</span>
<span class="s">"json"</span><span class="o">,</span> <span class="s">"value"</span><span class="o">,</span> <span class="s">"object"</span><span class="o">,</span> <span class="s">"member"</span><span class="o">,</span> <span class="s">"array"</span>
<span class="o">};</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">_LITERAL_NAMES</span> <span class="o">=</span> <span class="o">{</span>
<span class="kc">null</span><span class="o">,</span> <span class="s">"'true'"</span><span class="o">,</span> <span class="s">"'false'"</span><span class="o">,</span> <span class="s">"'null'"</span><span class="o">,</span> <span class="s">"'{'"</span><span class="o">,</span> <span class="s">"','"</span><span class="o">,</span> <span class="s">"'}'"</span><span class="o">,</span> <span class="s">"':'"</span><span class="o">,</span> <span class="s">"'['"</span><span class="o">,</span>
<span class="s">"']'"</span>
<span class="o">};</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">_SYMBOLIC_NAMES</span> <span class="o">=</span> <span class="o">{</span>
<span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="s">"STRING"</span><span class="o">,</span>
<span class="s">"NUMBER"</span><span class="o">,</span> <span class="s">"WS"</span>
<span class="o">};</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Vocabulary</span> <span class="no">VOCABULARY</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">VocabularyImpl</span><span class="o">(</span><span class="n">_LITERAL_NAMES</span><span class="o">,</span> <span class="n">_SYMBOLIC_NAMES</span><span class="o">);</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">Vocabulary</span> <span class="nf">getVocabulary</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="no">VOCABULARY</span><span class="o">;</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">getGrammarFileName</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="s">"JSON.g4"</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span><span class="o">[]</span> <span class="nf">getRuleNames</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">ruleNames</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">getSerializedATN</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">_serializedATN</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="no">ATN</span> <span class="nf">getATN</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">_ATN</span><span class="o">;</span> <span class="o">}</span>
<span class="kd">public</span> <span class="nf">JSONParser</span><span class="o">(</span><span class="nc">TokenStream</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">input</span><span class="o">);</span>
<span class="n">_interp</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ParserATNSimulator</span><span class="o">(</span><span class="k">this</span><span class="o">,</span><span class="n">_ATN</span><span class="o">,</span><span class="n">_decisionToDFA</span><span class="o">,</span><span class="n">_sharedContextCache</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">JsonContext</span> <span class="kd">extends</span> <span class="nc">ParserRuleContext</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">ValueContext</span> <span class="nf">value</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">ValueContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="mi">0</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nf">JsonContext</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">parent</span><span class="o">,</span> <span class="kt">int</span> <span class="n">invokingState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">parent</span><span class="o">,</span> <span class="n">invokingState</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getRuleIndex</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">RULE_json</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">enterJson</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">exitJson</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">JsonContext</span> <span class="nf">json</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">RecognitionException</span> <span class="o">{</span>
<span class="nc">JsonContext</span> <span class="n">_localctx</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">JsonContext</span><span class="o">(</span><span class="n">_ctx</span><span class="o">,</span> <span class="n">getState</span><span class="o">());</span>
<span class="n">enterRule</span><span class="o">(</span><span class="n">_localctx</span><span class="o">,</span> <span class="mi">0</span><span class="o">,</span> <span class="n">RULE_json</span><span class="o">);</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">enterOuterAlt</span><span class="o">(</span><span class="n">_localctx</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
<span class="o">{</span>
<span class="n">setState</span><span class="o">(</span><span class="mi">10</span><span class="o">);</span>
<span class="n">value</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">catch</span> <span class="o">(</span><span class="nc">RecognitionException</span> <span class="n">re</span><span class="o">)</span> <span class="o">{</span>
<span class="n">_localctx</span><span class="o">.</span><span class="na">exception</span> <span class="o">=</span> <span class="n">re</span><span class="o">;</span>
<span class="n">_errHandler</span><span class="o">.</span><span class="na">reportError</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">re</span><span class="o">);</span>
<span class="n">_errHandler</span><span class="o">.</span><span class="na">recover</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">re</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">finally</span> <span class="o">{</span>
<span class="n">exitRule</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">_localctx</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ValueContext</span> <span class="kd">extends</span> <span class="nc">ParserRuleContext</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">ObjectContext</span> <span class="nf">object</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">ObjectContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="mi">0</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">ArrayContext</span> <span class="nf">array</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">ArrayContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="mi">0</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">TerminalNode</span> <span class="nf">STRING</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">getToken</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span> <span class="o">}</span>
<span class="kd">public</span> <span class="nc">TerminalNode</span> <span class="nf">NUMBER</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">getToken</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">NUMBER</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span> <span class="o">}</span>
<span class="kd">public</span> <span class="nf">ValueContext</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">parent</span><span class="o">,</span> <span class="kt">int</span> <span class="n">invokingState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">parent</span><span class="o">,</span> <span class="n">invokingState</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getRuleIndex</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">RULE_value</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">enterValue</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">exitValue</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">ValueContext</span> <span class="nf">value</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">RecognitionException</span> <span class="o">{</span><span class="cm">/* omitted */</span><span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ObjectContext</span> <span class="kd">extends</span> <span class="nc">ParserRuleContext</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">List</span><span class="o"><</span><span class="nc">MemberContext</span><span class="o">></span> <span class="nf">member</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContexts</span><span class="o">(</span><span class="nc">MemberContext</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">MemberContext</span> <span class="nf">member</span><span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">MemberContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nf">ObjectContext</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">parent</span><span class="o">,</span> <span class="kt">int</span> <span class="n">invokingState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">parent</span><span class="o">,</span> <span class="n">invokingState</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getRuleIndex</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">RULE_object</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">enterObject</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">exitObject</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">ObjectContext</span> <span class="nf">object</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">RecognitionException</span> <span class="o">{</span><span class="cm">/* omitted */</span><span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">MemberContext</span> <span class="kd">extends</span> <span class="nc">ParserRuleContext</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">TerminalNode</span> <span class="nf">STRING</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">getToken</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="mi">0</span><span class="o">);</span> <span class="o">}</span>
<span class="kd">public</span> <span class="nc">ValueContext</span> <span class="nf">value</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">ValueContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="mi">0</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nf">MemberContext</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">parent</span><span class="o">,</span> <span class="kt">int</span> <span class="n">invokingState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">parent</span><span class="o">,</span> <span class="n">invokingState</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getRuleIndex</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">RULE_member</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">enterMember</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">exitMember</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">MemberContext</span> <span class="nf">member</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">RecognitionException</span> <span class="o">{</span><span class="cm">/* omitted */</span><span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ArrayContext</span> <span class="kd">extends</span> <span class="nc">ParserRuleContext</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">List</span><span class="o"><</span><span class="nc">ValueContext</span><span class="o">></span> <span class="nf">value</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContexts</span><span class="o">(</span><span class="nc">ValueContext</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">ValueContext</span> <span class="nf">value</span><span class="o">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">getRuleContext</span><span class="o">(</span><span class="nc">ValueContext</span><span class="o">.</span><span class="na">class</span><span class="o">,</span><span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nf">ArrayContext</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">parent</span><span class="o">,</span> <span class="kt">int</span> <span class="n">invokingState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">parent</span><span class="o">,</span> <span class="n">invokingState</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getRuleIndex</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">RULE_array</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">enterArray</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitRule</span><span class="o">(</span><span class="nc">ParseTreeListener</span> <span class="n">listener</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span> <span class="n">listener</span> <span class="k">instanceof</span> <span class="nc">JSONListener</span> <span class="o">)</span> <span class="o">((</span><span class="nc">JSONListener</span><span class="o">)</span><span class="n">listener</span><span class="o">).</span><span class="na">exitArray</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="nc">ArrayContext</span> <span class="nf">array</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">RecognitionException</span> <span class="o">{</span><span class="cm">/* omitted */</span><span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">_serializedATN</span> <span class="o">=</span>
<span class="s">"\3\u608b\ua72a\u8133\ub9ed\u417c\u3be7\u7786\u5964\3\16;\4\2\t\2\4\3\t"</span><span class="o">+</span>
<span class="s">"\3\4\4\t\4\4\5\t\5\4\6\t\6\3\2\3\2\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\5\3"</span><span class="o">+</span>
<span class="s">"\27\n\3\3\4\3\4\3\4\3\4\7\4\35\n\4\f\4\16\4 \13\4\3\4\3\4\3\4\3\4\5\4"</span><span class="o">+</span>
<span class="s">"&\n\4\3\5\3\5\3\5\3\5\3\6\3\6\3\6\3\6\7\6\60\n\6\f\6\16\6\63\13\6\3\6"</span><span class="o">+</span>
<span class="s">"\3\6\3\6\3\6\5\69\n\6\3\6\2\2\7\2\4\6\b\n\2\2\2@\2\f\3\2\2\2\4\26\3\2"</span><span class="o">+</span>
<span class="s">"\2\2\6%\3\2\2\2\b\'\3\2\2\2\n8\3\2\2\2\f\r\5\4\3\2\r\3\3\2\2\2\16\27\3"</span><span class="o">+</span>
<span class="s">"\2\2\2\17\27\5\6\4\2\20\27\5\n\6\2\21\27\7\f\2\2\22\27\7\r\2\2\23\27\7"</span><span class="o">+</span>
<span class="s">"\3\2\2\24\27\7\4\2\2\25\27\7\5\2\2\26\16\3\2\2\2\26\17\3\2\2\2\26\20\3"</span><span class="o">+</span>
<span class="s">"\2\2\2\26\21\3\2\2\2\26\22\3\2\2\2\26\23\3\2\2\2\26\24\3\2\2\2\26\25\3"</span><span class="o">+</span>
<span class="s">"\2\2\2\27\5\3\2\2\2\30\31\7\6\2\2\31\36\5\b\5\2\32\33\7\7\2\2\33\35\5"</span><span class="o">+</span>
<span class="s">"\b\5\2\34\32\3\2\2\2\35 \3\2\2\2\36\34\3\2\2\2\36\37\3\2\2\2\37!\3\2\2"</span><span class="o">+</span>
<span class="s">"\2 \36\3\2\2\2!\"\7\b\2\2\"&\3\2\2\2#$\7\6\2\2$&\7\b\2\2%\30\3\2\2\2%"</span><span class="o">+</span>
<span class="s">"#\3\2\2\2&\7\3\2\2\2\'(\7\f\2\2()\7\t\2\2)*\5\4\3\2*\t\3\2\2\2+,\7\n\2"</span><span class="o">+</span>
<span class="s">"\2,\61\5\4\3\2-.\7\7\2\2.\60\5\4\3\2/-\3\2\2\2\60\63\3\2\2\2\61/\3\2\2"</span><span class="o">+</span>
<span class="s">"\2\61\62\3\2\2\2\62\64\3\2\2\2\63\61\3\2\2\2\64\65\7\13\2\2\659\3\2\2"</span><span class="o">+</span>
<span class="s">"\2\66\67\7\n\2\2\679\7\13\2\28+\3\2\2\28\66\3\2\2\29\13\3\2\2\2\7\26\36"</span><span class="o">+</span>
<span class="s">"%\618"</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="no">ATN</span> <span class="n">_ATN</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">ATNDeserializer</span><span class="o">().</span><span class="na">deserialize</span><span class="o">(</span><span class="n">_serializedATN</span><span class="o">.</span><span class="na">toCharArray</span><span class="o">());</span>
<span class="kd">static</span> <span class="o">{</span>
<span class="n">_decisionToDFA</span> <span class="o">=</span> <span class="k">new</span> <span class="no">DFA</span><span class="o">[</span><span class="n">_ATN</span><span class="o">.</span><span class="na">getNumberOfDecisions</span><span class="o">()];</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">_ATN</span><span class="o">.</span><span class="na">getNumberOfDecisions</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">_decisionToDFA</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="k">new</span> <span class="no">DFA</span><span class="o">(</span><span class="n">_ATN</span><span class="o">.</span><span class="na">getDecisionState</span><span class="o">(</span><span class="n">i</span><span class="o">),</span> <span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">JSONListener</code>接口中对于每条语法规则声明了一个<code class="language-plaintext highlighter-rouge">enter</code>方法和一个<code class="language-plaintext highlighter-rouge">exit</code>方法,以便监听开始尝试和结束尝试一条规则:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.github.chungkwong.json</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.tree.ParseTreeListener</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">JSONListener</span> <span class="kd">extends</span> <span class="nc">ParseTreeListener</span> <span class="o">{</span>
<span class="kt">void</span> <span class="nf">enterJson</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">JsonContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">exitJson</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">JsonContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">enterValue</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ValueContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">exitValue</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ValueContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">enterObject</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ObjectContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">exitObject</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ObjectContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">enterMember</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">MemberContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">exitMember</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">MemberContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">enterArray</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ArrayContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="kt">void</span> <span class="nf">exitArray</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ArrayContext</span> <span class="n">ctx</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">JSONBaseListener</code>类是上述接口的一个空实现,可以作为一个出发点去修改:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.github.chungkwong.json</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.ParserRuleContext</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.tree.ErrorNode</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.tree.TerminalNode</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">JSONBaseListener</span> <span class="kd">implements</span> <span class="nc">JSONListener</span> <span class="o">{</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterJson</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">JsonContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitJson</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">JsonContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterValue</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ValueContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitValue</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ValueContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterObject</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ObjectContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitObject</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ObjectContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterMember</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">MemberContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitMember</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">MemberContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterArray</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ArrayContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitArray</span><span class="o">(</span><span class="nc">JSONParser</span><span class="o">.</span><span class="na">ArrayContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">enterEveryRule</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">exitEveryRule</span><span class="o">(</span><span class="nc">ParserRuleContext</span> <span class="n">ctx</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">visitTerminal</span><span class="o">(</span><span class="nc">TerminalNode</span> <span class="n">node</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">visitErrorNode</span><span class="o">(</span><span class="nc">ErrorNode</span> <span class="n">node</span><span class="o">)</span> <span class="o">{</span> <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>顺便也给出分词器<code class="language-plaintext highlighter-rouge">JSONLexer</code>类的内容:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.github.chungkwong.json</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.Lexer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.CharStream</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.Token</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.TokenStream</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.atn.*</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.dfa.DFA</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.antlr.v4.runtime.misc.*</span><span class="o">;</span>
<span class="nd">@SuppressWarnings</span><span class="o">({</span><span class="s">"all"</span><span class="o">,</span> <span class="s">"warnings"</span><span class="o">,</span> <span class="s">"unchecked"</span><span class="o">,</span> <span class="s">"unused"</span><span class="o">,</span> <span class="s">"cast"</span><span class="o">})</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">JSONLexer</span> <span class="kd">extends</span> <span class="nc">Lexer</span> <span class="o">{</span>
<span class="kd">static</span> <span class="o">{</span> <span class="nc">RuntimeMetaData</span><span class="o">.</span><span class="na">checkVersion</span><span class="o">(</span><span class="s">"4.7.2"</span><span class="o">,</span> <span class="nc">RuntimeMetaData</span><span class="o">.</span><span class="na">VERSION</span><span class="o">);</span> <span class="o">}</span>
<span class="kd">protected</span> <span class="kd">static</span> <span class="kd">final</span> <span class="no">DFA</span><span class="o">[]</span> <span class="n">_decisionToDFA</span><span class="o">;</span>
<span class="kd">protected</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">PredictionContextCache</span> <span class="n">_sharedContextCache</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">PredictionContextCache</span><span class="o">();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span>
<span class="no">T__0</span><span class="o">=</span><span class="mi">1</span><span class="o">,</span> <span class="no">T__1</span><span class="o">=</span><span class="mi">2</span><span class="o">,</span> <span class="no">T__2</span><span class="o">=</span><span class="mi">3</span><span class="o">,</span> <span class="no">T__3</span><span class="o">=</span><span class="mi">4</span><span class="o">,</span> <span class="no">T__4</span><span class="o">=</span><span class="mi">5</span><span class="o">,</span> <span class="no">T__5</span><span class="o">=</span><span class="mi">6</span><span class="o">,</span> <span class="no">T__6</span><span class="o">=</span><span class="mi">7</span><span class="o">,</span> <span class="no">T__7</span><span class="o">=</span><span class="mi">8</span><span class="o">,</span> <span class="no">T__8</span><span class="o">=</span><span class="mi">9</span><span class="o">,</span>
<span class="no">STRING</span><span class="o">=</span><span class="mi">10</span><span class="o">,</span> <span class="no">NUMBER</span><span class="o">=</span><span class="mi">11</span><span class="o">,</span> <span class="no">WS</span><span class="o">=</span><span class="mi">12</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">channelNames</span> <span class="o">=</span> <span class="o">{</span>
<span class="s">"DEFAULT_TOKEN_CHANNEL"</span><span class="o">,</span> <span class="s">"HIDDEN"</span>
<span class="o">};</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">modeNames</span> <span class="o">=</span> <span class="o">{</span>
<span class="s">"DEFAULT_MODE"</span>
<span class="o">};</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">ruleNames</span> <span class="o">=</span> <span class="o">{</span>
<span class="s">"T__0"</span><span class="o">,</span> <span class="s">"T__1"</span><span class="o">,</span> <span class="s">"T__2"</span><span class="o">,</span> <span class="s">"T__3"</span><span class="o">,</span> <span class="s">"T__4"</span><span class="o">,</span> <span class="s">"T__5"</span><span class="o">,</span> <span class="s">"T__6"</span><span class="o">,</span> <span class="s">"T__7"</span><span class="o">,</span> <span class="s">"T__8"</span><span class="o">,</span>
<span class="s">"STRING"</span><span class="o">,</span> <span class="s">"ESC"</span><span class="o">,</span> <span class="s">"HEX"</span><span class="o">,</span> <span class="s">"NUMBER"</span><span class="o">,</span> <span class="s">"INT"</span><span class="o">,</span> <span class="s">"EXP"</span><span class="o">,</span> <span class="s">"WS"</span>
<span class="o">};</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">_LITERAL_NAMES</span> <span class="o">=</span> <span class="o">{</span>
<span class="kc">null</span><span class="o">,</span> <span class="s">"'true'"</span><span class="o">,</span> <span class="s">"'false'"</span><span class="o">,</span> <span class="s">"'null'"</span><span class="o">,</span> <span class="s">"'{'"</span><span class="o">,</span> <span class="s">"','"</span><span class="o">,</span> <span class="s">"'}'"</span><span class="o">,</span> <span class="s">"':'"</span><span class="o">,</span> <span class="s">"'['"</span><span class="o">,</span>
<span class="s">"']'"</span>
<span class="o">};</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span><span class="o">[]</span> <span class="n">_SYMBOLIC_NAMES</span> <span class="o">=</span> <span class="o">{</span>
<span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="s">"STRING"</span><span class="o">,</span>
<span class="s">"NUMBER"</span><span class="o">,</span> <span class="s">"WS"</span>
<span class="o">};</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Vocabulary</span> <span class="no">VOCABULARY</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">VocabularyImpl</span><span class="o">(</span><span class="n">_LITERAL_NAMES</span><span class="o">,</span> <span class="n">_SYMBOLIC_NAMES</span><span class="o">);</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">Vocabulary</span> <span class="nf">getVocabulary</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="no">VOCABULARY</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nf">JSONLexer</span><span class="o">(</span><span class="nc">CharStream</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">input</span><span class="o">);</span>
<span class="n">_interp</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">LexerATNSimulator</span><span class="o">(</span><span class="k">this</span><span class="o">,</span><span class="n">_ATN</span><span class="o">,</span><span class="n">_decisionToDFA</span><span class="o">,</span><span class="n">_sharedContextCache</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">getGrammarFileName</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="s">"JSON.g4"</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span><span class="o">[]</span> <span class="nf">getRuleNames</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">ruleNames</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">getSerializedATN</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">_serializedATN</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span><span class="o">[]</span> <span class="nf">getChannelNames</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">channelNames</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span><span class="o">[]</span> <span class="nf">getModeNames</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">modeNames</span><span class="o">;</span> <span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="no">ATN</span> <span class="nf">getATN</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">_ATN</span><span class="o">;</span> <span class="o">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="n">_serializedATN</span> <span class="o">=</span>
<span class="s">"\3\u608b\ua72a\u8133\ub9ed\u417c\u3be7\u7786\u5964\2\16~\b\1\4\2\t\2\4"</span><span class="o">+</span>
<span class="s">"\3\t\3\4\4\t\4\4\5\t\5\4\6\t\6\4\7\t\7\4\b\t\b\4\t\t\t\4\n\t\n\4\13\t"</span><span class="o">+</span>
<span class="s">"\13\4\f\t\f\4\r\t\r\4\16\t\16\4\17\t\17\4\20\t\20\4\21\t\21\3\2\3\2\3"</span><span class="o">+</span>
<span class="s">"\2\3\2\3\2\3\3\3\3\3\3\3\3\3\3\3\3\3\4\3\4\3\4\3\4\3\4\3\5\3\5\3\6\3\6"</span><span class="o">+</span>
<span class="s">"\3\7\3\7\3\b\3\b\3\t\3\t\3\n\3\n\3\13\3\13\3\13\7\13C\n\13\f\13\16\13"</span><span class="o">+</span>
<span class="s">"F\13\13\3\13\3\13\3\f\3\f\3\f\3\f\3\f\3\f\3\f\3\f\5\fR\n\f\3\r\3\r\3\16"</span><span class="o">+</span>
<span class="s">"\5\16W\n\16\3\16\3\16\3\16\6\16\\\n\16\r\16\16\16]\5\16`\n\16\3\16\5\16"</span><span class="o">+</span>
<span class="s">"c\n\16\3\17\3\17\3\17\7\17h\n\17\f\17\16\17k\13\17\5\17m\n\17\3\20\3\20"</span><span class="o">+</span>
<span class="s">"\5\20q\n\20\3\20\6\20t\n\20\r\20\16\20u\3\21\6\21y\n\21\r\21\16\21z\3"</span><span class="o">+</span>
<span class="s">"\21\3\21\2\2\22\3\3\5\4\7\5\t\6\13\7\r\b\17\t\21\n\23\13\25\f\27\2\31"</span><span class="o">+</span>
<span class="s">"\2\33\r\35\2\37\2!\16\3\2\n\4\2$$^^\t\2$$\61\61^^ddppttvv\5\2\62;CHch"</span><span class="o">+</span>
<span class="s">"\3\2\62;\3\2\63;\4\2GGgg\4\2--//\5\2\13\f\17\17\"\"\2\u0085\2\3\3\2\2"</span><span class="o">+</span>
<span class="s">"\2\2\5\3\2\2\2\2\7\3\2\2\2\2\t\3\2\2\2\2\13\3\2\2\2\2\r\3\2\2\2\2\17\3"</span><span class="o">+</span>
<span class="s">"\2\2\2\2\21\3\2\2\2\2\23\3\2\2\2\2\25\3\2\2\2\2\33\3\2\2\2\2!\3\2\2\2"</span><span class="o">+</span>
<span class="s">"\3#\3\2\2\2\5(\3\2\2\2\7.\3\2\2\2\t\63\3\2\2\2\13\65\3\2\2\2\r\67\3\2"</span><span class="o">+</span>
<span class="s">"\2\2\179\3\2\2\2\21;\3\2\2\2\23=\3\2\2\2\25?\3\2\2\2\27I\3\2\2\2\31S\3"</span><span class="o">+</span>
<span class="s">"\2\2\2\33V\3\2\2\2\35l\3\2\2\2\37n\3\2\2\2!x\3\2\2\2#$\7v\2\2$%\7t\2\2"</span><span class="o">+</span>
<span class="s">"%&\7w\2\2&\'\7g\2\2\'\4\3\2\2\2()\7h\2\2)*\7c\2\2*+\7n\2\2+,\7u\2\2,-"</span><span class="o">+</span>
<span class="s">"\7g\2\2-\6\3\2\2\2./\7p\2\2/\60\7w\2\2\60\61\7n\2\2\61\62\7n\2\2\62\b"</span><span class="o">+</span>
<span class="s">"\3\2\2\2\63\64\7}\2\2\64\n\3\2\2\2\65\66\7.\2\2\66\f\3\2\2\2\678\7\177"</span><span class="o">+</span>
<span class="s">"\2\28\16\3\2\2\29:\7<\2\2:\20\3\2\2\2;<\7]\2\2<\22\3\2\2\2=>\7_\2\2>\24"</span><span class="o">+</span>
<span class="s">"\3\2\2\2?D\7$\2\2@C\5\27\f\2AC\n\2\2\2B@\3\2\2\2BA\3\2\2\2CF\3\2\2\2D"</span><span class="o">+</span>
<span class="s">"B\3\2\2\2DE\3\2\2\2EG\3\2\2\2FD\3\2\2\2GH\7$\2\2H\26\3\2\2\2IQ\7^\2\2"</span><span class="o">+</span>
<span class="s">"JR\t\3\2\2KL\7w\2\2LM\5\31\r\2MN\5\31\r\2NO\5\31\r\2OP\5\31\r\2PR\3\2"</span><span class="o">+</span>
<span class="s">"\2\2QJ\3\2\2\2QK\3\2\2\2R\30\3\2\2\2ST\t\4\2\2T\32\3\2\2\2UW\7/\2\2VU"</span><span class="o">+</span>
<span class="s">"\3\2\2\2VW\3\2\2\2WX\3\2\2\2X_\5\35\17\2Y[\7\60\2\2Z\\\t\5\2\2[Z\3\2\2"</span><span class="o">+</span>
<span class="s">"\2\\]\3\2\2\2][\3\2\2\2]^\3\2\2\2^`\3\2\2\2_Y\3\2\2\2_`\3\2\2\2`b\3\2"</span><span class="o">+</span>
<span class="s">"\2\2ac\5\37\20\2ba\3\2\2\2bc\3\2\2\2c\34\3\2\2\2dm\7\62\2\2ei\t\6\2\2"</span><span class="o">+</span>
<span class="s">"fh\t\5\2\2gf\3\2\2\2hk\3\2\2\2ig\3\2\2\2ij\3\2\2\2jm\3\2\2\2ki\3\2\2\2"</span><span class="o">+</span>
<span class="s">"ld\3\2\2\2le\3\2\2\2m\36\3\2\2\2np\t\7\2\2oq\t\b\2\2po\3\2\2\2pq\3\2\2"</span><span class="o">+</span>
<span class="s">"\2qs\3\2\2\2rt\t\5\2\2sr\3\2\2\2tu\3\2\2\2us\3\2\2\2uv\3\2\2\2v \3\2\2"</span><span class="o">+</span>
<span class="s">"\2wy\t\t\2\2xw\3\2\2\2yz\3\2\2\2zx\3\2\2\2z{\3\2\2\2{|\3\2\2\2|}\b\21"</span><span class="o">+</span>
<span class="s">"\2\2}\"\3\2\2\2\17\2BDQV]_bilpuz\3\b\2\2"</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="no">ATN</span> <span class="n">_ATN</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">ATNDeserializer</span><span class="o">().</span><span class="na">deserialize</span><span class="o">(</span><span class="n">_serializedATN</span><span class="o">.</span><span class="na">toCharArray</span><span class="o">());</span>
<span class="kd">static</span> <span class="o">{</span>
<span class="n">_decisionToDFA</span> <span class="o">=</span> <span class="k">new</span> <span class="no">DFA</span><span class="o">[</span><span class="n">_ATN</span><span class="o">.</span><span class="na">getNumberOfDecisions</span><span class="o">()];</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">_ATN</span><span class="o">.</span><span class="na">getNumberOfDecisions</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">_decisionToDFA</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="o">=</span> <span class="k">new</span> <span class="no">DFA</span><span class="o">(</span><span class="n">_ATN</span><span class="o">.</span><span class="na">getDecisionState</span><span class="o">(</span><span class="n">i</span><span class="o">),</span> <span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">JSON.tokens</code>和<code class="language-plaintext highlighter-rouge">JSONLexer.tokens</code>文件为每个词类指定了一个代号:</p>
<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="py">T__0</span><span class="p">=</span><span class="s">1</span>
<span class="py">T__1</span><span class="p">=</span><span class="s">2</span>
<span class="py">T__2</span><span class="p">=</span><span class="s">3</span>
<span class="py">T__3</span><span class="p">=</span><span class="s">4</span>
<span class="py">T__4</span><span class="p">=</span><span class="s">5</span>
<span class="py">T__5</span><span class="p">=</span><span class="s">6</span>
<span class="py">T__6</span><span class="p">=</span><span class="s">7</span>
<span class="py">T__7</span><span class="p">=</span><span class="s">8</span>
<span class="py">T__8</span><span class="p">=</span><span class="s">9</span>
<span class="py">STRING</span><span class="p">=</span><span class="s">10</span>
<span class="py">NUMBER</span><span class="p">=</span><span class="s">11</span>
<span class="py">WS</span><span class="p">=</span><span class="s">12</span>
<span class="err">'true'=1</span>
<span class="err">'false'=2</span>
<span class="err">'null'=3</span>
<span class="err">'{'=4</span>
<span class="err">','=5</span>
<span class="err">'}'=6</span>
<span class="err">':'=7</span>
<span class="err">'['=8</span>
<span class="err">']'=9</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">JSON.interp</code>和<code class="language-plaintext highlighter-rouge">JSONLexer.interp</code>文件给出了内部表示,一般不用管它。</p>
<h3 id="使用解析器">使用解析器</h3>
<p>现在我们可以使用分词器和解析器了:</p>
<ol>
<li>创建分词器。如<code class="language-plaintext highlighter-rouge">JSONLexer lexer = new JSONLexer(输入);</code>,其中输入可以是用以下方法获取:
<ul>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromChannel(ReadableByteChannel channel)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromChannel(ReadableByteChannel channel, Charset charset)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromChannel(ReadableByteChannel channel, Charset charset, int bufferSize, CodingErrorAction decodingErrorAction, String sourceName, long inputSize)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromChannel(ReadableByteChannel channel, int bufferSize, CodingErrorAction decodingErrorAction, String sourceName)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromPath(Path path)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromPath(Path path, Charset charset)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromReader(Reader r)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromReader(Reader r, String sourceName)</code>
<code class="language-plaintext highlighter-rouge">CharStreams.fromStream(InputStream is)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromStream(InputStream is, Charset charset)</code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromStream(InputStream is, Charset charset, long inputSize) </code></li>
<li><code class="language-plaintext highlighter-rouge">CharStreams.fromString(String s)</code></li>
</ul>
</li>
<li>取得词流。如<code class="language-plaintext highlighter-rouge">CommonTokenStream tokens = new CommonTokenStream(lexer);</code>。</li>
<li>创建解析器。如<code class="language-plaintext highlighter-rouge">JSONParser parser = new JSONParser(tokens)</code>。</li>
<li>取得解析树。如<code class="language-plaintext highlighter-rouge">JSONParser.ObjectContext tree = parser.object();</code>以<code class="language-plaintext highlighter-rouge">object</code>为开始规则解析。</li>
<li>使用解析树
<ul>
<li>遍历解析树。如<code class="language-plaintext highlighter-rouge">ParseTreeWalker.DEFAULT.walk(new JSONBaseListener(), tree);</code>。</li>
<li>按XPath寻找子树。如<code class="language-plaintext highlighter-rouge">Collection<ParseTree> subtrees=XPath.findAll(tree,xpath,parser);</code>,其中字符串<code class="language-plaintext highlighter-rouge">xpath</code>是由<code class="language-plaintext highlighter-rouge">/</code>(表示孩子)或<code class="language-plaintext highlighter-rouge">//</code>(表示后代)分隔的一些<code class="language-plaintext highlighter-rouge">规则名</code>(匹配指定规则生成的结点)、<code class="language-plaintext highlighter-rouge">'字符串'</code>(匹配字面上的词)、<code class="language-plaintext highlighter-rouge">*</code>(通配)或它们之一前面加<code class="language-plaintext highlighter-rouge">!</code>(表示否定)。</li>
</ul>
</li>
</ol>
<p>如果要处理大文件,生成完整的解析树可能占用太多内存。类似于XML的SAX解析器,这时可以用<code class="language-plaintext highlighter-rouge">parser.setBuildParseTree(false);</code>禁止生成完整语法树,并通过<code class="language-plaintext highlighter-rouge"> parser.addParseListener(new JSONBaseListener());</code>设置侦听器。</p>
<h3 id="动态加载语法">动态加载语法</h3>
<p>如果更改语法时不容许重新编译代码,ANTLR也可以动态加载语法,不过性能可能会差一点点。例如以下代码可以按动态给出的词法和语法解析输入流以得出语法树:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">LexerGrammar</span> <span class="n">lg</span><span class="o">=</span><span class="k">new</span> <span class="nc">LexerGrammar</span><span class="o">(</span><span class="n">词法</span><span class="o">);</span>
<span class="nc">Grammar</span> <span class="n">g</span><span class="o">=</span><span class="k">new</span> <span class="nc">Grammar</span><span class="o">(</span><span class="n">语法</span><span class="o">);</span>
<span class="nc">LexerInterpreter</span> <span class="n">lexEngine</span> <span class="o">=</span><span class="n">lg</span><span class="o">.</span><span class="na">createLexerInterpreter</span><span class="o">(</span><span class="nc">CharStreams</span><span class="o">.</span><span class="na">fromStream</span><span class="o">(</span><span class="n">输入流</span><span class="o">));</span>
<span class="nc">CommonTokenStream</span> <span class="n">tokens</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">CommonTokenStream</span><span class="o">(</span><span class="n">lexEngine</span><span class="o">);</span>
<span class="nc">ParserInterpreter</span> <span class="n">parser</span> <span class="o">=</span> <span class="n">g</span><span class="o">.</span><span class="na">createParserInterpreter</span><span class="o">(</span><span class="n">tokens</span><span class="o">);</span>
<span class="nc">ParseTree</span> <span class="n">t</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="na">parse</span><span class="o">(</span><span class="n">g</span><span class="o">.</span><span class="na">rules</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">开始规则名</span><span class="o">).</span><span class="na">index</span><span class="o">);</span>
</code></pre></div></div>
<p>注意,动态加载语法中所有动作代码(包括谓词)会被忽略。</p>
<h3 id="maven插件">Maven插件</h3>
<p>利用Maven插件,解析器可以在构建项目时自动生成。为此,在<code class="language-plaintext highlighter-rouge">pom.xml</code>中<code class="language-plaintext highlighter-rouge">build</code>元素的<code class="language-plaintext highlighter-rouge">plugins</code>元素中加入:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.antlr<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>antlr4-maven-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>4.7.2<span class="nt"></version></span>
<span class="nt"><executions></span>
<span class="nt"><execution></span>
<span class="nt"><id></span>antlr<span class="nt"></id></span>
<span class="nt"><goals></span>
<span class="nt"><goal></span>antlr4<span class="nt"></goal></span>
<span class="nt"></goals></span>
<span class="nt"></execution></span>
<span class="nt"></executions></span>
<span class="nt"></plugin></span>
</code></pre></div></div>
<p>然后把语法文件放到<code class="language-plaintext highlighter-rouge">src/main/antlr4/</code>目录下并按包名组织。如果有其它需要被包含的语法文件,则放到<code class="language-plaintext highlighter-rouge">src/main/antlr4/imports</code>目录下。</p>
<h2 id="语法文件">语法文件</h2>
<p>描述ANTLR语法和/或词法文件名形如<code class="language-plaintext highlighter-rouge">标题.g4</code>,内容由以下部分组成:</p>
<ol>
<li>头,形如以下之一:
<ul>
<li><code class="language-plaintext highlighter-rouge">grammar 标题;</code>表示这文件同时描述词法和语法</li>
<li><code class="language-plaintext highlighter-rouge">lexer grammar 标题;</code>表示这文件只描述词法</li>
<li><code class="language-plaintext highlighter-rouge">parser grammar 标题;</code>表示这文件只描述语法</li>
</ul>
</li>
<li>选项(可选),形如<code class="language-plaintext highlighter-rouge">options { 键1=值1; ... 键N=值N; }</code>,其中可指定的键有:
<ul>
<li><code class="language-plaintext highlighter-rouge">superClass</code>:分词器或解析器的父类</li>
<li><code class="language-plaintext highlighter-rouge">language</code>:生成用指定语言编写的代码</li>
<li><code class="language-plaintext highlighter-rouge">tokenVocab</code>:使用指定文件(加后缀<code class="language-plaintext highlighter-rouge">.token</code>的属性文件)给出的词类代码</li>
<li><code class="language-plaintext highlighter-rouge">TokenLabelType</code>:表示词类的类型,默认为<code class="language-plaintext highlighter-rouge">Token</code></li>
<li><code class="language-plaintext highlighter-rouge">contextSuperClass</code>:表示语法树的类型(应派生自<code class="language-plaintext highlighter-rouge">RuleContext</code>),默认为<code class="language-plaintext highlighter-rouge">ParserRuleContext</code></li>
</ul>
</li>
<li>导入(可选),形如<code class="language-plaintext highlighter-rouge">import 导入文件的标题,...;</code>。导入的效果是依次把被导入文件中规则加到最后(从而规则同名时以当前文件中的为准),词类、通道和命名动作分别合并。纯词法只能导入纯词法,纯语法只能导入纯语法,混合语法可以导入纯语法或没有模式的纯词法。导入可以递归。</li>
<li>词类声明(可选),形如<code class="language-plaintext highlighter-rouge">tokens { 词类名, ... }</code>,列出额外词类(没有词法规则的)以便动作代码使用。</li>
<li>通道(可选,只适用于纯词法),形如<code class="language-plaintext highlighter-rouge">channels {通道名,...}</code>,列出自定义通道。</li>
<li>命名动作(可选),形如<code class="language-plaintext highlighter-rouge">@动作名 {代码}</code>,用于把代码注入到解析器中。其中动作名可以是:
<ul>
<li><code class="language-plaintext highlighter-rouge">header</code>表示把代码注入到类声明前</li>
<li><code class="language-plaintext highlighter-rouge">members</code>表示把代码注入到类内作为字段或方法</li>
</ul>
</li>
<li>一条或以上规则,简单的规则形如<code class="language-plaintext highlighter-rouge">规则名 : 分支1 | ... | 分支N ;</code>,其中词法规则名由大写字母开始而语法规则名由小写字母开始。更复杂的语法规则形如:</li>
</ol>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>规则名[参数声明,...] returns [返回值声明,...] locals [局部总量声明,...] : 分支1 | ... | 分支N ;
</code></pre></div></div>
<p>各种名称可以由字母、数字、下划线组成(支持Unicode),但不能是关键词<code class="language-plaintext highlighter-rouge">import</code>、<code class="language-plaintext highlighter-rouge">fragment</code>、<code class="language-plaintext highlighter-rouge">lexer</code>、<code class="language-plaintext highlighter-rouge">parser</code>、<code class="language-plaintext highlighter-rouge">grammar</code>、<code class="language-plaintext highlighter-rouge">returns</code>、<code class="language-plaintext highlighter-rouge">locals</code>、<code class="language-plaintext highlighter-rouge">throws</code>、<code class="language-plaintext highlighter-rouge">catch</code>、<code class="language-plaintext highlighter-rouge">finally</code>、<code class="language-plaintext highlighter-rouge">mode</code>、<code class="language-plaintext highlighter-rouge">options</code>或<code class="language-plaintext highlighter-rouge">tokens</code>。另外文件中可以使用Java风格的注释<code class="language-plaintext highlighter-rouge">//行末注释</code>、<code class="language-plaintext highlighter-rouge">/* 注释 */</code>和<code class="language-plaintext highlighter-rouge">/** Javadocs */</code>。</p>
<h3 id="词法规则">词法规则</h3>
<table>
<thead>
<tr>
<th>分支</th>
<th>匹配</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">词类名</code></td>
<td>词类中的词</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">'字符序列'</code></td>
<td>字面上的字符序列,除了转义序列<code class="language-plaintext highlighter-rouge">\n</code>(换行)、<code class="language-plaintext highlighter-rouge">\r</code>(回车)、<code class="language-plaintext highlighter-rouge">\t</code>(制表符)、<code class="language-plaintext highlighter-rouge">\b</code>(退格)、<code class="language-plaintext highlighter-rouge">\f</code>(换页)、<code class="language-plaintext highlighter-rouge">\uXXXX</code>(Unicode四位十六进制代码点)或<code class="language-plaintext highlighter-rouge">\u{XXXXXX}’</code>(Unicode十六进制代码点)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">[字符集]</code></td>
<td>字符集中的一个字符,其中字符集由单字符(包括上述转义序列、<code class="language-plaintext highlighter-rouge">\\</code>、<code class="language-plaintext highlighter-rouge">\]</code>、<code class="language-plaintext highlighter-rouge">\-</code>)、形如<code class="language-plaintext highlighter-rouge">单字符-单字符</code>的字符区间、形如<code class="language-plaintext highlighter-rouge">\p{属性名}</code>或<code class="language-plaintext highlighter-rouge">\p{枚举属性=值}</code>的Unicode子集、以及它们形如<code class="language-plaintext highlighter-rouge">\P{属性名}</code>或<code class="language-plaintext highlighter-rouge">\P{枚举属性=值}</code>的补集组成</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">'字符'..'字符'</code></td>
<td>字符区间中的字符(包括这两个字符)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">.</code></td>
<td>任何一个字符</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">词法规则</code></td>
<td>匹配指定词法规则(包括<code class="language-plaintext highlighter-rouge">fragment</code>规则)的字符串,可以递归但不能左递归(需要手动改成右递归)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">{动作代码}</code></td>
<td>空,用于在读取到这位置时执行指定代码,当代码中花括号不配对时额外的花括号要用<code class="language-plaintext highlighter-rouge">\{</code>或<code class="language-plaintext highlighter-rouge">\}</code>转义</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">{谓词代码}?</code></td>
<td>空,布尔表达式的值为假时放弃继续尝试当前规则</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">~子规则</code></td>
<td>一个不匹配指定子规则的字符</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则 子规则</code></td>
<td>由分别匹配子规则的字符串接起来的</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则*</code></td>
<td>由零个或多个匹配子规则的字符串串接起来,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则+</code></td>
<td>由一个或多个匹配子规则的字符串串接起来,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则?</code></td>
<td>由零个或一个匹配子规则的字符串串接起来,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则*?</code></td>
<td>由零个或多个匹配子规则的字符串串接起来,匹配尽可能短</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则+?</code></td>
<td>由一个或多个匹配子规则的字符串串接起来,匹配尽可能短</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">子规则??</code></td>
<td>由零个或一个匹配子规则的字符串串接起来,匹配尽可能短</td>
</tr>
</tbody>
</table>
<p>在动作代码或谓词代码中可以通过<code class="language-plaintext highlighter-rouge">$规则名</code>引用匹配子规则的词(当有多个同名子规则时可在规则前加上<code class="language-plaintext highlighter-rouge">名称=</code>来指定别名),进而可通过<code class="language-plaintext highlighter-rouge">.</code>引用其字段或方法,例如以下只读属性:</p>
<table>
<thead>
<tr>
<th>属性</th>
<th>方法</th>
<th>类型</th>
<th>值</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">text</code></td>
<td><code class="language-plaintext highlighter-rouge">getText</code></td>
<td><code class="language-plaintext highlighter-rouge">String</code></td>
<td>匹配的文本</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">type</code></td>
<td><code class="language-plaintext highlighter-rouge">getType</code></td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>词类代号</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">line</code></td>
<td><code class="language-plaintext highlighter-rouge">getLine</code></td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>词开始的行号(从1开始)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">pos</code></td>
<td><code class="language-plaintext highlighter-rouge">getCharPositionInLine</code></td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>词在行的偏移(从0开始)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">index</code></td>
<td><code class="language-plaintext highlighter-rouge">getTokenIndex</code></td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>当前词的序号(从0开始</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">channel</code></td>
<td><code class="language-plaintext highlighter-rouge">getChannel</code></td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>通道代码,默认为0 (<code class="language-plaintext highlighter-rouge">Token.DEFAULT_CHANNEL</code>),隐藏通道为<code class="language-plaintext highlighter-rouge">Token.HIDDEN_CHANNEL</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td> </td>
<td><code class="language-plaintext highlighter-rouge">int</code></td>
<td>匹配文本表示的整数</td>
</tr>
</tbody>
</table>
<p>在一个分支的最后可以加上<code class="language-plaintext highlighter-rouge">->命令,...</code>,其中可用的命令有:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">skip</code>用于放弃当前词</li>
<li><code class="language-plaintext highlighter-rouge">mode(模式)</code>修改栈顶模式(栈可用于实现仅用正则表达式无法描述的模式,如某些语言容许的嵌套注释)</li>
<li><code class="language-plaintext highlighter-rouge">pushMode(模式)</code>推入栈顶模式</li>
<li><code class="language-plaintext highlighter-rouge">popMode</code>弹出栈顶模式</li>
<li><code class="language-plaintext highlighter-rouge">more</code>要求继续匹配以延长当前词</li>
<li><code class="language-plaintext highlighter-rouge">type(词类)</code>用于修改当前词所属的词类</li>
<li><code class="language-plaintext highlighter-rouge">channel(通道)</code>用于把当前词送到指定通道</li>
</ul>
<p>如果一条词法规则纯粹为了共用代码或提高可读性,而不是要实际生成词,可在规则前加上<code class="language-plaintext highlighter-rouge">fragment </code>。</p>
<h3 id="语法规则">语法规则</h3>
<p>语法分支可以由以下元素组成:</p>
<table>
<thead>
<tr>
<th>构造</th>
<th>描述</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">词类名</code></td>
<td>匹配词类中的词,特殊词<code class="language-plaintext highlighter-rouge">EOF</code>用于标记输入结束</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">'字符串</code>’</td>
<td>匹配恰由指定字符串组成的词</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">语法规则</code></td>
<td>匹配指定规则的一列词</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">语法规则 [参数,...]</code></td>
<td>匹配指定规则的一列词,参数的值将传递给该规则</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">{动作代码}</code></td>
<td>马上执行指定代码,其中可以通过<code class="language-plaintext highlighter-rouge">$x</code>或<code class="language-plaintext highlighter-rouge">$x.y</code>引用属性,并可通过<code class="language-plaintext highlighter-rouge">$x::y</code>引用非局部属性(动态作用域)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">{谓词代码}?</code></td>
<td>若代码的值为<code class="language-plaintext highlighter-rouge">false</code>则放弃尝试当前分支</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">.</code></td>
<td>匹配任何词(除了<code class="language-plaintext highlighter-rouge">EOF</code>)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">(分支|...)</code></td>
<td>匹配匹配其中一个子分支的词列</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素?</code></td>
<td>匹配匹配零个或一个元素的词列,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素*</code></td>
<td>匹配匹配零个或以上元素的词列,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素+</code></td>
<td>匹配匹配一个或以上元素的词列,匹配尽可能长</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素??</code></td>
<td>匹配匹配零个或一个元素的词列,匹配尽可能短</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素*?</code></td>
<td>匹配匹配零个或以上元素的词列,匹配尽可能短</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">元素+?</code></td>
<td>匹配匹配一个或以上元素的词列,匹配尽可能短</td>
</tr>
</tbody>
</table>
<p>在动作代码或谓词代码中可以通过<code class="language-plaintext highlighter-rouge">$规则名</code>引用匹配子规则的文本(当有多个同名子规则时可在规则前加上<code class="language-plaintext highlighter-rouge">名称=</code>来指定别名),进而可通过<code class="language-plaintext highlighter-rouge">.</code>引用其字段或方法,例如以下只读属性:</p>
<table>
<thead>
<tr>
<th>属性</th>
<th>类型</th>
<th>值</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">text</code></td>
<td><code class="language-plaintext highlighter-rouge">String</code></td>
<td>已匹配的文本(包括隐藏通道中的)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">start</code></td>
<td><code class="language-plaintext highlighter-rouge">Token</code></td>
<td>主通道中首个匹配的词</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">stop</code></td>
<td><code class="language-plaintext highlighter-rouge">Token</code></td>
<td>主通道中最后一个匹配的词(只适用于末尾的动作和<code class="language-plaintext highlighter-rouge">finally</code>动作)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ctx</code></td>
<td><code class="language-plaintext highlighter-rouge">ParserRuleContext</code></td>
<td>上下文对象</td>
</tr>
</tbody>
</table>
<p>如果希望通过侦听器监视什么时候开始和结束结束尝试分支,可以在分支后加上<code class="language-plaintext highlighter-rouge">#名称</code>(一个规则要么所有分支都有名称,要么都没有),多个分支可以有相同名称。</p>
<h2 id="总结">总结</h2>
<p>如果对用ANTLR描述语法有疑问,可以参考<a href="https://github.com/antlr/grammars-v4">各种常见语言的ANTLR语法</a>。关于ANTLR的细节,可参阅<a href="https://github.com/antlr/antlr4/blob/master/doc/index.md">官方文档</a>和<a href="https://www.antlr.org/api/">API</a>。顺带一提,ANTLR不仅可以生成用Java编写的解析器,也支持C#、Python、JavaScript、Go、C++、Swift等等。</p>陈颂光在开发各种软件的过程中难免要与各式各样的小语言打交道,例如要读取不同格式的数据(特别是配置)文件。虽然我们可以从头开始自己写程序去解析它们,但这样往往过于耗时且难以维护。ANTLR是Lex(或Flex)与YACC(或Bison)在Java世界的一个代用品,可以根据词法和语法自动生成解析器。编写浏览器插件:适用于Chrome、Firefox、Edge和Opera2019-02-26T00:00:00+00:002019-02-26T00:00:00+00:00https://www.chungkwong.cc/webextension<p>浏览器插件(或称扩展)可以修改浏览器的行为,比如可以屏蔽广告、防止跟踪、下载视频、翻译网页、增加搜索引擎等等。在过往不同浏览器有互不兼容的插件机制,但现在包括Chrome、Firefox、Edge和Opera在内的主流浏览器都在某程度上支持了WebExtensions API,因而我们能轻松地编写同时适用于多个浏览器的插件。</p>
<h2 id="流程">流程</h2>
<p>我们演示如何制作一个简单的插件,它在开启时可以屏蔽网页中的视频。</p>
<p>首先,我们在一个新的目录下编写一个清单文件<code class="language-plaintext highlighter-rouge">manifest.json</code>描述扩展的元数据:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"manifest_version"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="s2">"VideoBlock"</span><span class="p">,</span><span class="w">
</span><span class="nl">"description"</span><span class="p">:</span><span class="s2">"Block all videos"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="s2">"0.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"permissions"</span><span class="p">:[</span><span class="s2">"webRequest"</span><span class="p">,</span><span class="s2">"webRequestBlocking"</span><span class="p">,</span><span class="s2">"<all_urls>"</span><span class="p">],</span><span class="w">
</span><span class="nl">"background"</span><span class="p">:{</span><span class="nl">"scripts"</span><span class="p">:[</span><span class="s2">"background.js"</span><span class="p">]},</span><span class="w">
</span><span class="nl">"browser_action"</span><span class="p">:{</span><span class="nl">"default_icon"</span><span class="p">:{</span><span class="nl">"32"</span><span class="p">:</span><span class="s2">"icons/off.png"</span><span class="p">}}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>在这例子中:</p>
<ul>
<li>清单语法版本为<code class="language-plaintext highlighter-rouge">2</code>(不要改)</li>
<li>扩展名为<code class="language-plaintext highlighter-rouge">VideoBlock</code></li>
<li>简介为<code class="language-plaintext highlighter-rouge">Block all videos</code></li>
<li>版本号为<code class="language-plaintext highlighter-rouge">0.1</code></li>
<li>需要权限为可拦截到所有URI的请求</li>
<li>加载扩展后马上运行脚本<code class="language-plaintext highlighter-rouge">background.js</code></li>
<li>在浏览器工具栏中加入一个按钮,默认图标为<code class="language-plaintext highlighter-rouge">icons/off.png</code></li>
</ul>
<p>其次我们编写<code class="language-plaintext highlighter-rouge">background.js</code>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">blocking</span><span class="o">=</span><span class="kc">false</span><span class="p">;</span>
<span class="nx">browser</span><span class="p">.</span><span class="nx">webRequest</span><span class="p">.</span><span class="nx">onBeforeRequest</span><span class="p">.</span><span class="nx">addListener</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">requestDetails</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="nx">blocking</span><span class="p">){</span>
<span class="k">return</span> <span class="p">{</span><span class="na">redirectUrl</span><span class="p">:</span><span class="nx">requestDetails</span><span class="p">.</span><span class="nx">url</span><span class="p">};</span>
<span class="p">}</span><span class="k">else</span><span class="p">{</span>
<span class="k">return</span> <span class="p">{</span><span class="na">redirectUrl</span><span class="p">:</span><span class="kc">null</span><span class="p">};</span>
<span class="p">}</span>
<span class="p">},{</span><span class="na">urls</span><span class="p">:[</span><span class="dl">"</span><span class="s2"><all_urls></span><span class="dl">"</span><span class="p">],</span><span class="na">types</span><span class="p">:[</span><span class="dl">"</span><span class="s2">media</span><span class="dl">"</span><span class="p">]},[</span><span class="dl">"</span><span class="s2">blocking</span><span class="dl">"</span><span class="p">]);</span>
<span class="nx">browser</span><span class="p">.</span><span class="nx">browserAction</span><span class="p">.</span><span class="nx">onClicked</span><span class="p">.</span><span class="nx">addListener</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="k">if</span><span class="p">(</span><span class="nx">blocking</span><span class="p">){</span>
<span class="nx">browser</span><span class="p">.</span><span class="nx">browserAction</span><span class="p">.</span><span class="nx">setIcon</span><span class="p">({</span><span class="na">path</span><span class="p">:</span> <span class="dl">"</span><span class="s2">icons/off.png</span><span class="dl">"</span><span class="p">});</span>
<span class="p">}</span><span class="k">else</span><span class="p">{</span>
<span class="nx">browser</span><span class="p">.</span><span class="nx">browserAction</span><span class="p">.</span><span class="nx">setIcon</span><span class="p">({</span><span class="na">path</span><span class="p">:</span> <span class="dl">"</span><span class="s2">icons/on.png</span><span class="dl">"</span><span class="p">});</span>
<span class="p">}</span>
<span class="nx">blocking</span><span class="o">=!</span><span class="nx">blocking</span><span class="p">;</span>
<span class="p">});</span>
</code></pre></div></div>
<p>这脚本中用变量<code class="language-plaintext highlighter-rouge">blocking</code>记录屏蔽功能当前是否已启用,然后监视两个事件:</p>
<ul>
<li>当有类型为<code class="language-plaintext highlighter-rouge">media</code>的请求且屏蔽功能启用时屏蔽之</li>
<li>当按钮被点击时启用或关闭屏蔽功能,同时通过修改按钮图标反映新状态</li>
</ul>
<p>接着我们在下画两个32像素高和宽的图标<code class="language-plaintext highlighter-rouge">icons/off.png</code>和<code class="language-plaintext highlighter-rouge">icons/on.png</code>分别用于关闭和开启屏蔽功能时的图标。</p>
<p>现在我们让浏览器加载这个扩展:</p>
<ul>
<li>在Firefox中
<ol>
<li>访问URL<code class="language-plaintext highlighter-rouge">about:debugging</code></li>
<li>选中<code class="language-plaintext highlighter-rouge">启用附加组件调试</code></li>
<li>点击<code class="language-plaintext highlighter-rouge">载入临时附加组件</code></li>
<li>选择<code class="language-plaintext highlighter-rouge">manifest.json</code>文件</li>
</ol>
</li>
<li>在Chrome中
<ol>
<li>访问URL<code class="language-plaintext highlighter-rouge">chrome://extensions</code></li>
<li>选中<code class="language-plaintext highlighter-rouge">Developer Mode</code></li>
<li>点击<code class="language-plaintext highlighter-rouge">LOAD UNPACKED</code></li>
<li>选择<code class="language-plaintext highlighter-rouge">manifest.json</code>所在文件夹</li>
</ol>
</li>
<li>在Edge中
<ol>
<li>访问URL<code class="language-plaintext highlighter-rouge">about:flags</code></li>
<li>选中<code class="language-plaintext highlighter-rouge">Enable extension developer features</code></li>
<li>点击<code class="language-plaintext highlighter-rouge">More (...)</code></li>
<li>点击<code class="language-plaintext highlighter-rouge">Extensions</code></li>
<li>点击<code class="language-plaintext highlighter-rouge">Load extension</code></li>
<li>选择<code class="language-plaintext highlighter-rouge">manifest.json</code>所在文件夹</li>
</ol>
</li>
</ul>
<p>加载后点击扩展在工具栏中的按钮,再打开一个带视频的网页,应该就会发现视频播放不了。</p>
<p>最后我们还可以把扩展发布到网上,让别人通过浏览器的扩展中心发现并安装我们开发的扩展:</p>
<ul>
<li>对于Firefox,发布到<a href="https://addons.mozilla.org/zh-CN/developers/">Mozilla附加组件开发者中心</a></li>
<li>对于Chrome,<a href="https://developer.chrome.com/webstore/publish">参考这里</a></li>
<li>对于Edge,<a href="https://docs.microsoft.com/en-us/microsoft-edge/extensions/guides/packaging">参考这里</a></li>
</ul>
<h2 id="清单">清单</h2>
<p>每个扩展都有一个描述元数据的清单,它是直接位于扩展目录、名为<code class="language-plaintext highlighter-rouge">manifest.json</code>的JSON文件,常见字段如下:</p>
<table>
<thead>
<tr>
<th>键</th>
<th>必要性</th>
<th>用途</th>
<th>格式例子</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">name</code></td>
<td>必须</td>
<td>给用户看的扩展名称,不超过45个字符</td>
<td><code class="language-plaintext highlighter-rouge">"Video blocker"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">version</code></td>
<td>必须</td>
<td>版本号,由一到四个用句点分隔的整数组成</td>
<td><code class="language-plaintext highlighter-rouge">"1"</code>、<code class="language-plaintext highlighter-rouge">"1.0"</code>、<code class="language-plaintext highlighter-rouge">"2.10.2"</code>、<code class="language-plaintext highlighter-rouge">"3.1.2.4567"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">default_locale</code></td>
<td>当且仅当扩展目录有子目录<code class="language-plaintext highlighter-rouge">_locales</code></td>
<td>存放默认字符串的<code class="language-plaintext highlighter-rouge">_locales</code>子目录名</td>
<td><code class="language-plaintext highlighter-rouge">en</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">description</code></td>
<td>可选</td>
<td>给用户看的扩展简介</td>
<td><code class="language-plaintext highlighter-rouge">"Block all videos"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">icons</code></td>
<td>可选</td>
<td>扩展不同大小(以<code class="language-plaintext highlighter-rouge">px</code>为单位)的图标文件位置</td>
<td><code class="language-plaintext highlighter-rouge">{"24":"small.jpg","48":"medium.jpg"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">developer</code></td>
<td>可选</td>
<td>开发者的名称和/或网址</td>
<td><code class="language-plaintext highlighter-rouge">{"name": "Bighard", "url": "https://www.bighard.com"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">browser_action</code></td>
<td>可选</td>
<td>加到浏览器工具栏的按钮,可指定图标、提示和点击时打开的对话框页面,不能与<code class="language-plaintext highlighter-rouge">page_action</code>共存</td>
<td><code class="language-plaintext highlighter-rouge">{"default_icon":{"16": "button/geo-16.png","32":"button/geo-32.png"},"default_title": "Whereami?","default_popup": "popup/geo.html"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">page_action</code></td>
<td>可选</td>
<td>加到浏览器工具栏的按钮,可指定图标、提示和点击时打开的对话框页面,不能与<code class="language-plaintext highlighter-rouge">browser_action</code>共存</td>
<td>同上</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">browser_specific_settings</code></td>
<td>可选</td>
<td>浏览器特定的值</td>
<td><code class="language-plaintext highlighter-rouge">{"gecko":{"strict_min_version": "57.0"}}</code>(表示要求Firefox的版本至少为57才能安装)</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">background</code></td>
<td>可选</td>
<td>浏览器加载扩展后即开始运行的脚本,还可指定运行于的页面和是否保持这页面于已加载状态以记忆状态</td>
<td><code class="language-plaintext highlighter-rouge">{"page": "background.html","scripts": ["background.js"],"persistent":true}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">content_scripts</code></td>
<td>可选</td>
<td>自动把<code class="language-plaintext highlighter-rouge">js</code>指定的脚本和/或<code class="language-plaintext highlighter-rouge">css</code>指定的样式表加入每个页面(<code class="language-plaintext highlighter-rouge">all_frames</code>为<code class="language-plaintext highlighter-rouge">false</code>时则修改模式<code class="language-plaintext highlighter-rouge">matches</code>指定的页面或不修改模式<code class="language-plaintext highlighter-rouge">exclude_matches</code>指定的页面)由<code class="language-plaintext highlighter-rouge">run_at</code>指定的位置<code class="language-plaintext highlighter-rouge">"document_start"</code>、<code class="language-plaintext highlighter-rouge">"document_end"</code>或<code class="language-plaintext highlighter-rouge">"document_idle"</code></td>
<td><code class="language-plaintext highlighter-rouge">{"all_frames": <boolean>,"css": [...],"exclude_matches": [...],"js": [...],"matches": [...],"run_at": "document_end"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">content_security_policy</code></td>
<td>可选</td>
<td>防止意外执行恶意代码的<a href="https://www.w3.org/TR/CSP3/">内容安全策略</a></td>
<td><code class="language-plaintext highlighter-rouge">"script-src 'self'; object-src 'self';"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">options_page</code></td>
<td>可选</td>
<td>选项页</td>
<td><code class="language-plaintext highlighter-rouge">"options/options.html"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">manifest_version</code></td>
<td>可选</td>
<td>清单版本,一般为2</td>
<td><code class="language-plaintext highlighter-rouge">2</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">web_accessible_resources</code></td>
<td>可选</td>
<td>容许页面通过<code class="language-plaintext highlighter-rouge">browser.extension.getURL()</code>访问的资源</td>
<td><code class="language-plaintext highlighter-rouge">["images/useless.png"]</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">permissions</code></td>
<td>可选</td>
<td>需要的额外权限,包括危险API和作危险访问(如绕过同源策略访问有关页面、用<code class="language-plaintext highlighter-rouge">tabs.executeScript</code>向有关页面插入脚本、用<code class="language-plaintext highlighter-rouge">webRequest</code>API从有关页面接收事件)的URL</td>
<td><code class="language-plaintext highlighter-rouge">["activeTab","contextMenus","tabs","webNavigation","webRequestBlocking","*://*.w3.org/*","<all_urls>"]</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">required_keys</code></td>
<td>可选</td>
<td>要求浏览器支持指定字段才安装</td>
<td><code class="language-plaintext highlighter-rouge">["theme"]</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">externally_connectable</code></td>
<td>可选</td>
<td>接受来自其它扩展或URL模式的消息</td>
<td><code class="language-plaintext highlighter-rouge">["ids": [],"matches": []]</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">author</code></td>
<td>可选</td>
<td>作者</td>
<td><code class="language-plaintext highlighter-rouge">Zhang</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">chrome_settings_overrides</code></td>
<td>可选</td>
<td>主页和搜索引擎</td>
<td><code class="language-plaintext highlighter-rouge">{"homepage":"https://www.viewfact.org/","search_provider":{"name":"睇料","search_url":"https://www.viewfact.org/search?q={searchTerms}","keyword":"viewfact","favicon_url":"https://www.viewfact.org/favicon.ico"}}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">chrome_url_overrides</code></td>
<td>可选</td>
<td>特殊页面如新标签页</td>
<td><code class="language-plaintext highlighter-rouge">{"newtab": "my-new-tab.html"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">commands</code></td>
<td>可选</td>
<td>命令(包括特殊命令<code class="language-plaintext highlighter-rouge">_execute_browser_action</code>、<code class="language-plaintext highlighter-rouge">execute_page_action</code>、<code class="language-plaintext highlighter-rouge">_execute_sidebar_action</code>)的描述和键盘快捷键(对不同平台<code class="language-plaintext highlighter-rouge">"default"</code>、<code class="language-plaintext highlighter-rouge">"mac"</code>、<code class="language-plaintext highlighter-rouge">"linux"</code>、<code class="language-plaintext highlighter-rouge">"windows"</code>、<code class="language-plaintext highlighter-rouge">"chromeos"</code>、<code class="language-plaintext highlighter-rouge">"android"</code>、<code class="language-plaintext highlighter-rouge">"ios"</code>可指定不同快捷键)</td>
<td><code class="language-plaintext highlighter-rouge">{"toggle-feature":{"suggested_key":{"default": "Ctrl+Shift+Y","linux": "Ctrl+Shift+U"},"description": "Send a 'toggle-feature' event"}}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">devtools_page</code></td>
<td>可选</td>
<td>开发者工具</td>
<td><code class="language-plaintext highlighter-rouge">"devtools/my-page.html"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">homepage_url</code></td>
<td>可选</td>
<td>扩展的主页</td>
<td><code class="language-plaintext highlighter-rouge">"https://example.org/my-addon"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">incognito</code></td>
<td>可选</td>
<td>区分隐私窗口</td>
<td><code class="language-plaintext highlighter-rouge">"spanning"</code>(隐私和非隐私窗口的事件对扩展可见)、 <code class="language-plaintext highlighter-rouge">"split"</code>(运行两套扩展)、<code class="language-plaintext highlighter-rouge">"not_allowed"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">offline_enabled</code></td>
<td>可选</td>
<td>是否预期在离线时使用</td>
<td><code class="language-plaintext highlighter-rouge">true</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">omnibox</code></td>
<td>可选</td>
<td>用户在地址栏输入指定关键词后,其余部分会发给应用</td>
<td><code class="language-plaintext highlighter-rouge">{"keyword": "mdn"}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">optional_permissions</code></td>
<td>可选</td>
<td>运行时可能请求用户批准的权限</td>
<td><code class="language-plaintext highlighter-rouge">[]</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">options_ui</code></td>
<td>可选</td>
<td>选项页</td>
<td><code class="language-plaintext highlighter-rouge">{"page": "options/options.html","open_in_tab":false,"browser_style":true}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">name</code></td>
<td>可选</td>
<td>给用户看的扩展简称,不超过12个字符</td>
<td><code class="language-plaintext highlighter-rouge">"NoVideo"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">theme</code></td>
<td>可选</td>
<td>外观主题</td>
<td><code class="language-plaintext highlighter-rouge">{"images":{"headerURL": "images/sun.jpg"},"colors": {"bookmark_text":[0,0,0],"frame_inactive":[0,0,0],"ntp_background":[0,0,0],"ntp_text":[0,0,0],"tab_background_text":[0,0,0],"tab_text":[0,0,0],"toolbar": [0,0,0]},"properties":{}}</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">version_name</code></td>
<td>可选</td>
<td>给人看的版本号</td>
<td><code class="language-plaintext highlighter-rouge">"0.1 beta"</code></td>
</tr>
</tbody>
</table>
<h2 id="国际化与本地化">国际化与本地化</h2>
<p>为了让扩展支持多种语言,首先找出扩展中所有需要翻译的信息加以国际化,方法是为它们分别起个互不相同的名字(符合正则表达式<code class="language-plaintext highlighter-rouge">[A-Za-z0-9@_]+?</code>),然后按场合把信息替换为:</p>
<ul>
<li>若在清单或样式表中,则替换为<code class="language-plaintext highlighter-rouge">__MSG_名字__</code></li>
<li>若在脚本中,则替换为<code class="language-plaintext highlighter-rouge">browser.i18n.getMessage('名字',字符串或其数组)</code></li>
</ul>
<table>
<thead>
<tr>
<th>预定义的信息名</th>
<th>说明</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@extension_id</code></td>
<td>扩展ID,不能用于清单</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@ui_locale</code></td>
<td>当前地区</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@bidi_dir</code></td>
<td>当前地区的文字方向:<code class="language-plaintext highlighter-rouge">"ltr"</code>或<code class="language-plaintext highlighter-rouge">"rtl"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@bidi_reversed_dir</code></td>
<td>当前地区的文字反方向:<code class="language-plaintext highlighter-rouge">"ltr"</code>或<code class="language-plaintext highlighter-rouge">"rtl"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@bidi_start_edge</code></td>
<td>当前地区的文字方向:<code class="language-plaintext highlighter-rouge">"left"</code>或<code class="language-plaintext highlighter-rouge">"right"</code></td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">@@bidi_end_edge</code></td>
<td>当前地区的文字反方向:<code class="language-plaintext highlighter-rouge">"right"</code>或<code class="language-plaintext highlighter-rouge">"left"</code></td>
</tr>
</tbody>
</table>
<p>接下来是本地化,方法是在扩展目录的子目录<code class="language-plaintext highlighter-rouge">_locales</code>下为每种语言创建分别创建子目录(目录名为<code class="language-plaintext highlighter-rouge">语言代码</code>或<code class="language-plaintext highlighter-rouge">语言代码_地区代码</code>),再分别在这些子目录创建JSON文件。JSON文件的字段名为各消息的名字,值则是消息模板:</p>
<ul>
<li>对于没有参数的消息模板,形如<code class="language-plaintext highlighter-rouge">"extensionName":{"message": "Image Blocker","description": "Name of the extension"}</code></li>
<li>对于有参数的消息模板
<ul>
<li>可以用<code class="language-plaintext highlighter-rouge">$序号</code>指代位置参数,形如<code class="language-plaintext highlighter-rouge">"of":{"message": "$2的$1","description": "of something"}</code></li>
<li>也可以给参数名字,形如<code class="language-plaintext highlighter-rouge">"{"message": "You clicked $URL$.","description":"Tells the user which link they clicked.","placeholders":{"url":{"content":"$1","example":"https://developer.mozilla.org"}}}</code></li>
</ul>
</li>
</ul>
<h2 id="api">API</h2>
<p>浏览器开放给扩展用Javascript调用的API大多在全局对象(即<code class="language-plaintext highlighter-rouge">window</code>的字段)<code class="language-plaintext highlighter-rouge">browser</code>中,用WebIDL可以如下描述各主要类型:</p>
<pre><code class="language-webidl">[CheckAnyPermissions="browserExt"]
interface BrowserExtGlobal {
readonly attribute Browser browser;
};
Window implements BrowserExtGlobal;
[NoInterfaceObject]
interface Browser {
};
dictionary BrowserExtIcon {
DOMString path;
DOMString size;
};
typedef sequence<BrowserExtIcon> BrowserExtIconArray;
dictionary BrowserExtDeveloper {
DOMString name;
DOMString? url;
};
dictionary BrowserExtBrowserOrPageAction {
BrowserExtIcon? defaultIcon;
DOMString? defaultPopup;
DOMString? defaultTitle;
};
dictionary BrowserExtKeyValue {
DOMString key;
DOMString? value;
};
dictionary BrowserExtBrowserSpecificSettings {
DOMString browserName;
sequence<BrowserExtKeyValue> keyValue;
};
dictionary BrowserExtBackroundOrEvent {
DOMString? page;
boolean persistent;
sequence<DOMString>? scripts;
};
dictionary BrowserExtContentScripts {
boolean allFrames;
sequence<DOMString>? css;
sequence<DOMString>? excludeMatches;
sequence<DOMString>? js;
sequence<DOMString> matches;
DOMString? runAt;
};
dictionary BrowserExtManifest {
BrowserExtBackroundOrEvent? background;
BrowserExtBrowserOrPageAction? browserAction;
BrowserExtBrowserSpecificSettings? browserSpecificSettings;
BrowserExtContentScripts? contentScripts;
DOMString? contentSecurityPolicy;
DOMString? defaultLocale;
DOMString? description;
BrowserExtDeveloper? developer;
BrowserExtIconArray? icons;
DOMString? manifestVersion;
DOMString name;
DOMString? optionsPage;
BrowserExtBrowserOrPageAction? pageAction;
sequence<DOMString>? permissions;
sequence<DOMString>? requiredKeys;
DOMString? version;
sequence<DOMString> webAccessibleResources;
};
[NoInterfaceObject, Exposed=(Window,ContentScript), CheckAnyPermissions="browserExt"]
interface BrowserExtEvent {
void addListener(Function callback);
void removeListener(Function callback);
boolean hasListener(Function callback);
boolean hasListeners();
};
</code></pre>
<p>在WebIDL中,我们用<code class="language-plaintext highlighter-rouge">Exposed</code>属性描述可用上下文(<code class="language-plaintext highlighter-rouge">Window</code>表示<code class="language-plaintext highlighter-rouge">background</code>、<code class="language-plaintext highlighter-rouge">browser_action</code>、<code class="language-plaintext highlighter-rouge">page_action</code>、<code class="language-plaintext highlighter-rouge">options_page</code>等字段引入的脚本,<code class="language-plaintext highlighter-rouge">ContentScript</code>表示<code class="language-plaintext highlighter-rouge">content_scripts</code>字段引入的脚本)。</p>
<h3 id="browseraction">BrowserAction</h3>
<p><code class="language-plaintext highlighter-rouge">browser.browserAction</code>给出了浏览器按钮的API:</p>
<pre><code class="language-webidl">dictionary BrowserExtBrowserActionDefaults {
BrowserExtIconArray? defaultIcon;
DOMString? defaultPopup;
DOMString? defaultTitle;
};
partial dictionary BrowserExtManifest {
BrowserExtBrowserActionDefaults? browserActionDefaults;
};
typedef sequence<byte> BrowserExtColorArray;
dictionary BrowserExtTabIdDetails {
long tabId;
};
dictionary BrowserExtBadgeColorArrayTabId {
BrowserExtColorArray color;
long tabId;
};
dictionary BrowserExtBadgePathTabId {
DOMString path;
long tabId;
};
dictionary BrowserExtBadgeTextTabId {
long tabId;
DOMString text;
};
dictionary BrowserExtBadgeTabIdPopup {
DOMString popupHTMLFileName;
long tabId;
};
callback BrowserExtBrowserActionOnClickedCallback = void (BrowserExtTabsTab tab);
[NoInterfaceObject]
interface BrowserExtBrowserAction {
void disable(long tabId);
void enable(long tabId);
Promise<BrowserExtColorArray> getBadgeBackgroundColor(BrowserExtTabIdDetails details);
Promise<DOMString?> getBadgeText(BrowserExtTabIdDetails details);
Promise<DOMString?> getPopup(BrowserExtTabIdDetails details);
BrowserExtEvent onClicked(BrowserExtBrowserActionOnClickedCallback callback);
void setBadgeBackgroundColor(BrowserExtBadgeColorArrayTabId details);
void setBadgeText(BrowserExtBadgeTextTabId details);
Promise<void> setIcon(BrowserExtBadgePathTabId details);
void setPopup(BrowserExtBadgeTabIdPopup details);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtBrowserAction"]
interface BrowserExtBrowserActionAPI {
readonly attribute BrowserExtBrowserAction browserAction;
};
Browser implements BrowserExtBrowserActionAPI;
</code></pre>
<h3 id="contextmenus">ContextMenus</h3>
<p><code class="language-plaintext highlighter-rouge">browser.contextMenus</code>给出了上下文菜单的API:</p>
<pre><code class="language-webidl">enum BrowserExtContextType { "all", "audio", "browser_action", "editable", "frame", "image", "link", "page", "page_action", "selection", "video" };
enum BrowserExtItemType { "checkbox","normal","radio","separator" };
dictionary BrowserExtContextMenuCreateDetails {
boolean checked;
sequence<ContextType>? contexts;
sequence<DOMString>? documentUrlPatterns;
boolean enabled;
DOMString? id;
Function onclick;
(long or DOMString?) parentId;
sequence<DOMString>? targetUrlPatterns;
DOMString? title;
DOMString? type;
};
dictionary BrowserExtContextMenuUpdateDetails {
boolean checked;
sequence<BrowserExtContextType>? contexts;
sequence<DOMString>? documentUrlPatterns;
boolean enabled;
Function onclick;
long parentId;
sequence<DOMString>? targetUrlPatterns;
DOMString? title;
ItemType? type;
};
dictionary BrowserExtContextMenuOnClickedDetails {
boolean checked;
boolean editable;
long frameId;
DOMString? frameUrl;
DOMString? linkUrl;
DOMString? mediaType;
(long or DOMString) menuItemId;
DOMString? pageUrl;
(long or DOMString?) parentMenuItemId;
DOMString? selectionText;
DOMString? srcUrl;
boolean wasChecked;
};
callback BrowserExtContextMenuOnClickedCallback = void (BrowserExtContextMenuOnClickedDetails details, BrowserExtTabsTab tab);
[NoInterfaceObject]
interface BrowserExtContextMenus {
Promise<long> create(BrowserExtContextMenuCreateDetails details);
BrowserExtEvent onClicked(BrowserExtContextMenuOnClickedCallback callback);
Promise<void> remove((long or DOMString) itemId);
Promise<void> removeAll();
Promise<void> update((long or DOMString) itemId, BrowserExtContextMenuUpdateDetails details);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtContextMenus"]
interface BrowserExtContextMenusAPI {
readonly attribute BrowserExtContextMenus contextMenus;
};
Browser implements BrowserExtContextMenusAPI;
</code></pre>
<h3 id="i18n">i18n</h3>
<p><code class="language-plaintext highlighter-rouge">browser.i18n</code>给出了国际化的API:</p>
<pre><code class="language-webidl">[NoInterfaceObject]
interface BrowserExtI18n {
DOMString getMessage(DOMString messageName, sequence<DOMString?>? substitutions);
};
[NoInterfaceObject, Exposed=(Window,ContentScript), CheckAnyPermissions="browserExt"]
interface BrowserExtI18nAPI {
readonly attribute BrowserExtI18n i18n;
};
Browser implements BrowserExtI18nAPI;
partial dictionary BrowserExtManifest {
DOMString? defaultLocaleValue;
};
</code></pre>
<h3 id="pageaction">pageAction</h3>
<p><code class="language-plaintext highlighter-rouge">browser.pageAction</code>给出了页面按钮的API:</p>
<pre><code class="language-webidl">dictionary BrowserExtPageActionDefaults {
BrowserExtIconArray? defaultIcon;
DOMString? defaultPopup;
DOMString? defaultTitle;
};
partial dictionary BrowserExtManifest {
BrowserExtPageActionDefaults? pageActionDefaults;
};
dictionary BrowserExtBadgePath {
DOMString path;
};
callback BrowserExtPageActionOnClickedCallback = void (BrowserExtTabsTab tab);
interface BrowserExtPageAction {
Promise<DOMString?> getPopup(BrowserExtTabIdDetails details);
Promise<DOMString?> getTitle(BrowserExtTabIdDetails details);
void hide(long tabId);
BrowserExtEvent onClicked(BrowserExtPageActionOnClickedCallback callback);
Promise<void> setIcon(BrowserExtBadgePath path);
void setPopup(BrowserExtBadgePathTabId details);
void setTitle(BrowserExtBadgeTextTabId details);
void show(long tabId);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtPageAction"]
interface BrowserExtPageActionAPI {
readonly attribute BrowserExtPageAction pageAction;
};
Browser implements BrowserExtPageActionAPI;
</code></pre>
<h3 id="runtime">runtime</h3>
<p><code class="language-plaintext highlighter-rouge">browser.runtime</code>给出了运行时的API(用于扩展与页面相互通信):</p>
<pre><code class="language-webidl">enum BrowserExtRuntimeOnInstalledReason {"browser_update", "extension_install", "extension_update"};
dictionary BrowserExtRuntimeOnInstalledDetails {
BrowserExtRuntimeOnInstalledReason reason;
DOMString? previousVersion;
DOMString? id;
};
dictionary BrowserExtRuntimePort {
Function disconnect;
DOMString name;
object onDisconnect;
object onMessage;
Function postMessage;
BrowserExtRuntimeMessageSender? sender;
};
dictionary BrowserExtRuntimeMessageSender {
DOMString? id;
long frameId;
BrowserExtTabsTab? tab;
DOMString? url;
};
callback BrowserExtRuntimeOnConnectCallback = void (BrowserExtRuntimePort port);
callback BrowserExtRuntimeOnInstalledCallback = void (BrowserExtRuntimeOnInstalledDetails details);
callback BrowserExtRuntimeOnMessageCallback = void (optional any message, BrowserExtRuntimeMessageSender sender, BrowserExtRuntimeSendResponseCallback callback);
[NoInterfaceObject]
interface BrowserExtBrowserRuntime {
object getManifest();
DOMString getURL(DOMString path);
attribute DOMString id;
void onConnect(BrowserExtRuntimeOnConnectCallback callback);
void onInstalled(BrowserExtRuntimeOnInstalledCallback callback);
void onMessage(BrowserExtRuntimeOnMessageCallback callback);
Promise<any> sendMessage(optional DOMString extensionId, any message);
attribute BrowserExtRuntimePort Port;
attribute BrowserExtRuntimeMessageSender MessageSender;
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExt"]
interface BrowserExtRuntimeAPI {
readonly attribute BrowserExtRuntime runtime;
};
Browser implements BrowserExtRuntimeAPI;
</code></pre>
<h3 id="tabs">tabs</h3>
<p><code class="language-plaintext highlighter-rouge">browser.tabs</code>给出了标签页的API:</p>
<pre><code class="language-webidl">enum BrowserExtTabMutedReasonDetails { "user", "capture", "extension" };
enum BrowserExtRunAt { "document_end", "document_idle", "document_start" };
enum BrowserExtTabStatus { "complete", "loading" };
enum BrowserExtWindowTypes { "app", "normal", "panel", "popup" };
enum BrowserExtTabsCaptureVisibleTabFormat { "jpeg", "png" };
dictionary BrowserExtTabsTab {
boolean active;
boolean audible;
DOMString? favIconUrl;
boolean highlighted;
long height;
long id;
boolean incognito;
long index;
TabMutedDetails? mutedDetails;
long openerTabId;
boolean pinned;
DOMString? sessionId;
DOMString? status;
DOMString? title;
DOMString? url;
long width;
long windowId;
};
dictionary BrowserExtTabMutedDetails {
DOMString? extensionId;
boolean muted;
TabMutedReasonDetails? reason;
};
dictionary BrowserExtTabConnectDetails {
long frameId;
DOMString? name;
};
dictionary BrowserExtTabCreateDetails {
boolean active;
long index;
long openerTabId;
boolean pinned;
DOMString? url;
long windowId;
};
dictionary BrowserExtTabScriptAndCSSDetails {
boolean allFrames;
DOMString? code;
DOMString? file;
long frameId;
boolean matchAboutBlank;
BrowserExtRunAt? runAt;
};
dictionary BrowserExtTabsCaptureVisibleTabDetails {
BrowserExtTabsCaptureVisibleTabFormat imageCaptureFormat;
long jpegQuality;
};
dictionary BrowserExtTabQueryDetails {
boolean active;
boolean audible;
boolean currentWindow;
boolean highlighted;
long index;
boolean lastFocusedWindow;
boolean muted;
boolean pinned;
TabStatus? status;
DOMString? title;
(DOMString? or sequence<DOMString>?) url;
long windowId;
WindowType? windowType;
};
dictionary BrowserExtTabReloadDetails {
boolean bypassCache;
};
dictionary BrowserExtTabSendMessageDetails {
long frameId;
};
dictionary BrowserExtTabUpdateDetails {
boolean active;
boolean highlighted;
boolean muted;
long openerTabId;
boolean pinned;
DOMString? url;
};
dictionary BrowserExtTabIdWindowId {
long tabId;
long windowId;
};
dictionary BrowserExtTabsWindowIdIsWindowClosing {
boolean isWindowClosing;
long windowId;
};
dictionary BrowserExtTabsOnUpdatedChangeDetails {
boolean audible;
DOMString? favIconUrl;
TabMutedDetails? mutedDetails;
boolean pinned;
DOMString? status;
DOMString? title;
DOMString? url;
};
callback BrowserExtTabsOnActivatedCallback = void (BrowserExtTabIdWindowId activeDetails);
callback BrowserExtTabsOnCreatedCallback = void (BrowserExtTabTab tab);
callback BrowserExtTabsOnRemovedCallback = void (long tabId, BrowserExtTabsWindowIdIsWindowClosing removeDetails);
callback BrowserExtTabsOnUpdatedCallback = void (long tabId, BrowserExtTabsOnUpdatedChangeDetails details, BrowserExtTabsTab tab);
[NoInterfaceObject]
interface BrowserExtBrowserTabs {
Promise<DOMString> captureVisibleTab(long windowId, BrowserExtTabsCaptureVisibleTabDetails details);
RuntimePort connect(long tabId, BrowserExtTabConnectDetails details);
Promise<BrowserExtTabsTab> create(BrowserExtTabCreateDetails details);
void executeScript(long tabId, BrowserExtTabScriptAndCSSDetails details);
Promise<BrowserExtTabsTab> get(long tabId);
Promise<BrowserExtTabsTab> getCurrent();
void insertCSS(long tabId, BrowserExtTabScriptAndCSSDetails details);
void onActivated(BrowserExtTabsOnActivatedCallback callback);
void onCreated(BrowserExtTabsOnCreatedCallback callback);
void onRemoved(BrowserExtTabsOnRemovedCallback callback);
void onUpdated(BrowserExtTabsOnUpdatedCallback callback);
Promise<BrowserExtTabsTab> query(BrowserExtTabQueryDetails queryDetails);
Promise<void> reload(BrowserExtTabReloadDetails details);
Promise<void> remove((long or sequence<long>) tabIds);
Promise<any> sendMessage(long tabId, any message, optional BrowserExtTabSendMessageOptions details);
Promise<BrowserExtTabsTab> update(optional long tabId, BrowserExtTabUpdateDetails details);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtTabs"]
interface BrowserExtTabsAPI {
readonly attribute BrowserExtTabs tabs;
};
Browser implements BrowserExtTabsAPI;
</code></pre>
<h3 id="webnavigation">webNavigation</h3>
<p><code class="language-plaintext highlighter-rouge">browser.webNavigation</code>给出了导航的API:</p>
<pre><code class="language-webidl">enum BrowserExtTransitionType { "link", "typed" };
dictionary BrowserExtWebNavigationOnBeforeNavigateDetails {
long frameId;
long parentFrameId;
long tabId;
double timeStamp;
DOMString url;
};
dictionary BrowserExtWebNavigationOnCommittedDetails {
long frameId;
long processId;
long tabId;
double timeStamp;
TransitionType transitionType;
DOMString url;
};
dictionary BrowserExtWebNavigationOnCompletedDetails {
long frameId;
long processId;
long tabId;
double timeStamp;
DOMString url;
};
dictionary BrowserExtWebNavigationOnDOMContentLoadedDetails {
long frameId;
long processId;
long tabId;
double timeStamp;
DOMString url;
};
dictionary BrowserExtWebNavigationOnErrorOccurredDetails {
DOMString error;
long frameId;
long tabId;
double timeStamp;
DOMString url;
};
dictionary BrowserExtWebNavigationOnReferenceFragmentUpdatedDetails {
long frameId;
long processId;
long tabId;
double timeStamp;
BrowserExtTransitionType transitionType;
DOMString url;
};
callback BrowserExtWebNavigationOnBeforeNavigateCallback = void (BrowserExtWebNavigationOnBeforeNavigateDetails details);
callback BrowserExtWebNavigationOnCommittedCallback = void (BrowserExtWebNavigationOnCommittedDetails details);
callback BrowserExtWebNavigationOnCompletedCallback = void (BrowserExtWebNavigationOnCompletedDetails details);
callback BrowserExtWebNavigationOnDOMContentLoadedCallback = void (BrowserExtWebNavigationOnDOMContentLoadedDetails details);
callback BrowserExtWebNavigationOnErrorOccurredCallback = void (BrowserExtWebNavigationOnErrorOccurredDetails details);
callback BrowserExtWebNavigationOnReferenceFragmentUpdated = void (BrowserExtWebNavigationOnReferenceFragmentUpdatedDetails details);
[NoInterfaceObject]
interface BrowserExtBrowserWebNavigation {
void onBeforeNavigate(BrowserExtWebNavigationOnBeforeNavigateCallback callback);
void onCommitted(BrowserExtWebNavigationOnCommittedCallback callback);
void onCompleted(BrowserExtWebNavigationOnCompletedCallback callback);
void onDOMContentLoaded(BrowserExtWebNavigationOnDOMContentLoadedCallback callback);
void onErrorOccurred(BrowserExtWebNavigationOnErrorOccurredCallback callback);
void onReferenceFragmentUpdated(BrowserExtWebNavigationOnReferenceFragmentUpdated callback);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtWebNavigation"]
interface BrowserExtWebNavigationAPI {
readonly attribute BrowserExtWebNavigation webNavigation;
};
Browser implements BrowserExtWebNavigationAPI;
</code></pre>
<h3 id="webrequest">webRequest</h3>
<p><code class="language-plaintext highlighter-rouge">browser.webRequest</code>给出了web请求的API:</p>
<pre><code class="language-webidl">enum BrowserExtResourceType { "font", "image", "main_frame", "object", "other", "ping", "script", "stylesheet", "sub_frame", "xmlhttprequest" };
dictionary BrowserExtWebRequestUploadDetails {
any bytes;
DOMString? file;
};
dictionary BrowserExtWebRequestRequestBody {
DOMString? error;
object? formData;
sequence<BrowserExtWebRequestUploadData>? rawData;
};
dictionary BrowserExtWebRequestHttpHeader {
DOMString keyName;
any value;
};
dictionary BrowserExtWebRequestHttpHeaders {
sequence<BrowserExtWebRequestHttpHeader> data;
};
dictionary BrowserExtWebRequestOnBeforeRequestDetails {
long frameId;
DOMString method;
long parentFrameId;
BrowserExtWebRequestRequestBody? requestBody;
DOMString requestId;
long tabId;
double timeStamp;
BrowserExtResourceType type;
DOMString url;
};
dictionary BrowserExtWebRequestOnBeforeSendHeadersDetails {
long frameId;
DOMString method;
long parentFrameId;
BrowserExtWebRequestHttpHeaders? requestHeaders;
DOMString requestId;
long tabId;
double timeStamp;
BrowserExtResourceType type;
DOMString url;
};
dictionary BrowserExtWebRequestOnCompletedDetails {
long frameId;
boolean fromCache;
DOMString method;
long parentFrameId;
DOMString requestId;
BrowserExtWebRequestHttpHeaders? responseHeaders;
DOMString? serverIP;
long statusCode;
DOMString statusLine;
long tabId;
double timeStamp;
BrowserExtResourceType type;
DOMString url;
};
dictionary BrowserExtWebRequestOnHeadersReceivedDetails {
long frameId;
DOMString method;
long parentFrameId;
DOMString requestId;
BrowserExtWebRequestHttpHeaders? responseHeaders;
long statusCode;
DOMString statusLine;
long tabId;
double timeStamp;
BrowserExtResourceType type;
DOMString url;
};
dictionary BrowserExtWebRequestOnSendHeadersDetails {
long frameId;
DOMString method;
long parentFrameId;
BrowserExtWebRequestHttpHeaders? requestHeaders;
DOMString requestId;
long tabId;
double timeStamp;
BrowserExtResourceType type;
DOMString url;
};
callback BrowserExtWebRequestOnBeforeRequestCallback = void (BrowserExtWebRequestOnBeforeRequestDetails details);
callback BrowserExtWebRequestOnBeforeSendHeadersCallback = void (BrowserExtWebRequestOnBeforeSendHeadersDetails details);
callback BrowserExtWebRequestOnCompletedCallback = void (BrowserExtWebRequestOnCompletedDetails details);
callback BrowserExtWebRequestOnHeadersReceivedCallback = void (BrowserExtWebRequestOnHeadersReceivedDetails details);
callback BrowserExtWebRequestOnSendHeadersCallback = void (BrowserExtWebRequestOnSendHeadersDetails details);
[NoInterfaceObject]
interface BrowserExtBrowserWebRequest {
void onBeforeRequest(BrowserExtWebRequestOnBeforeRequestCallback callback);
void onBeforeSendHeaders(BrowserExtWebRequestOnBeforeSendHeadersCallback callback);
void onCompleted(BrowserExtWebRequestOnCompletedCallback callback);
void onHeadersReceived(BrowserExtWebRequestOnHeadersReceivedCallback callback);
void onSendHeaders(BrowserExtWebRequestOnSendHeadersCallback callback);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExtWebRequest"]
interface BrowserExtWebRequestAPI {
readonly attribute BrowserExtWebRequest webRequest;
};
Browser implements BrowserWebRequestAPI;
</code></pre>
<h3 id="windows">windows</h3>
<p><code class="language-plaintext highlighter-rouge">browser.windows</code>给出了浏览器窗口的API:</p>
<pre><code class="language-webidl">enum BrowserExtWindowsWindowType { "detached_panel", "normal", "panel", "popup" };
enum BrowserExtWindowsWindowState { "docked", "fullscreen", "maximized", "minimized", "normal" };
dictionary BrowserExtCreateWindowDetails {
boolean focused;
long height;
boolean incognito;
long left;
BrowserExtWindowsWindowState? state;
long tabId;
long top;
(DOMString? or sequence<DOMString>?) url;
long width;
BrowserExtWindowsWindowType? windowType;
};
dictionary BrowserExtGetWindowDetails {
boolean populate;
sequence<BrowserExtWindowsWindowType>? windowTypes;
};
dictionary BrowserExtWindowUpdateDetails {
boolean drawAttention;
boolean focused;
long height;
long left;
long top;
long width;
BrowserExtWindowState? windowState;
};
dictionary BrowserExtWindowsWindow {
boolean alwaysOnTop;
boolean focused;
long height;
long id;
boolean incognito;
long left;
sequence<BrowserExtTabsTab>? tabs;
long top;
DOMString? sessionId;
BrowserExtWindowsWindowState? state;
long width;
BrowserExtWindowsWindowType? windowType;
};
callback BrowserExtWindowsOnFocusChangedCallback = void (long windowId);
[NoInterfaceObject]
interface BrowserExtWindows {
Promise<BrowserExtWindowsWindow> create(CreateWindowDetails details);
Promise<BrowserExtWindowsWindow> get(long windowId, optional BrowserExtGetWindowDetails details);
Promise<sequence<BrowserExtWindowsWindow>> getAll(optional BrowserExtGetWindowDetail details);
Promise<BrowserExtWindowsWindow> getCurrent(optional BrowserExtGetWindowDetail details);
Promise<BrowserExtWindowsWindow> getLastFocused(optional BrowserExtGetWindowDetail details);
void onFocusChanged(BrowserExtWindowsOnFocusChangedCallback callback);
Promise<BrowserExtWindowsWindow> update(long windowId, optional BrowserExtWindowUpdateDetails details);
};
[NoInterfaceObject, Exposed=Window, CheckAnyPermissions="browserExt"]
interface BrowserExtWindowsAPI {
readonly attribute BrowserExtWindows windows;
};
Browser implements BrowserExtWindowsAPI;
</code></pre>
<h2 id="更多信息">更多信息</h2>
<p>各浏览器还支持大量其它API在这里未能尽录,请参考:</p>
<ul>
<li><a href="https://developer.mozilla.org/zh-CN/docs/Mozilla/Add-ons/WebExtensions/Browser_support_for_JavaScript_APIs">MDN</a>汇总了跨浏览器支持情况。</li>
<li><a href="https://docs.microsoft.com/en-us/microsoft-edge/extensions">Microsoft docs</a>描述了Edge扩展。</li>
<li><a href="https://developer.chrome.com/extensions">Chrome文档</a>。</li>
<li><a href="https://browserext.github.io/browserext/">W3C浏览器扩展社区小组</a>提供了一个标准草案。</li>
</ul>陈颂光浏览器插件(或称扩展)可以修改浏览器的行为,比如可以屏蔽广告、防止跟踪、下载视频、翻译网页、增加搜索引擎等等。在过往不同浏览器有互不兼容的插件机制,但现在包括Chrome、Firefox、Edge和Opera在内的主流浏览器都在某程度上支持了WebExtensions API,因而我们能轻松地编写同时适用于多个浏览器的插件。善用Wikidata语义网2019-01-01T00:00:00+00:002019-01-01T00:00:00+00:00https://www.chungkwong.cc/wikidata<p>人们用他们熟悉的自然语言如中文在网上编辑了海量的信息,但自然语言固有的极端复杂性使这些文本难以为机器所理解,于是人们因为找不到已有的信息而重复劳动,这个恶性循环的结果是互联网上文章泛滥但质量普遍低下,特别是过时信息得不到清理。相反,语义网提倡用更规范的方式表达知识,并把它们连接起来。维基基金会的<a href="https://www.wikidata.org">Wikidata</a>就是一个人人可编辑和利用的知识库,有大量志愿者持续扩展和修正它,这使它的数据无论在数量上还是质量上都处于领先地位,维基百科中条目右上方卡片的信息就来自Wikidata。值得一提的是Wikidata的数据位于公众领域,可以(接近)无条件地合法使用。</p>
<h2 id="探索语义网">探索语义网</h2>
<p>世界上有许多实体,其中包括具体的对象如“铅笔”和抽象的概念如“实数”。在自然语言中实体用名词表示,在语义网中我们用IRI表示实体。在自然语言中实体与名词的关系是多对多的(比如一个人可以有多个绰号,不同人也可以同名同姓),而在语义网中实体与IRI的关系是一对一的,这使得语义网中可以避开通常句子中的歧义问题。同时,使用IRI表示实体让我们可方便地获取关于实体的有用信息:把它当作URL去访问即可。</p>
<p>断言指出实体的单个特点,大致对应于自然语言中最简单的陈述句。自然语言中的陈述句往往相当复杂,同时表达了多种含义,但语义网中的每个断言应该只表达一项单一的知识。语义网中最常见的断言由一个主体、一个属性谓词和一个客体组成,多数情况下它们都是实体,但也可能是数值、字符串等等。</p>
<p>语义网的基本模型就这么简单,让我们来体验一下Wikidata如何体现语义网的原则。在浏览器打开一个实体IRI
,比如说<a href="http://www.wikidata.org/entity/Q753535">http://www.wikidata.org/entity/Q753535</a>,其中以Q开始的是Wikidata中实体的编号:</p>
<p><img src="/image/entity.png" alt="Wikidata的一个实体页面" /></p>
<p>我们可以看到以下信息:</p>
<ul>
<li>这个实体在不同语言中的惟一最常见名称,如在中文是“金瓶梅”。</li>
<li>这个实体在不同语言中的各个别名,如在英文有“The Plum in the Golden Vase”、“.The Golden Lotus”、“Plum in the Golden Vase”、“Golden Lotus”和“Jinpingmei”。</li>
<li>这个实体在不同语言中的补充描述(用于区分自然语言中同名的实体),如在中文是“中国古典小说”。</li>
<li>其它以这实体为主体的断言(不同实体有不同的属性),比如说由此我们可知:
<ul>
<li>这个实体是一本书</li>
<li>这个实体的某一张图</li>
<li>这个实体始于1610年</li>
<li>这个实体源于中国</li>
<li>这个实体在维基百科的页面</li>
</ul>
</li>
</ul>
<p>我们看到,谓词和大部分客体都是实体,在上述页面中我们可以通过链接继续浏览它们,例如我们可进一步了解作者“兰陵笑笑生”,相关实体之间的断言使Wikidata成为真正的网(用图论的话,实体对应顶点而断言对应边),我们可沿这个网逐渐展开知识。部分断言更带有出处,方便进行考证。</p>
<p>当然在浏览语义网前要知道实体的IRI,幸运的是Wikidata网站右上角的搜索框可以按实体名称或别名搜索实体。</p>
<p>HTML格式虽然方便人们用浏览器查看,但用程序处理时类似JSON的格式更方便且节省流量,这时可以改用<a href="https://www.wikidata.org/wiki/Special:EntityData/Q753535.json">https://www.wikidata.org/wiki/Special:EntityData/Q753535.json</a>,类似地按要求的格式可改用<code class="language-plaintext highlighter-rouge">rdf</code>、<code class="language-plaintext highlighter-rouge">ttl</code>或<code class="language-plaintext highlighter-rouge">nl</code>后缀,以下演示返回结果的结构(但内容经过大幅删减):</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"entities"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Q753535"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"pageid"</span><span class="p">:</span><span class="w"> </span><span class="mi">708936</span><span class="p">,</span><span class="w">
</span><span class="nl">"ns"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535"</span><span class="p">,</span><span class="w">
</span><span class="nl">"lastrevid"</span><span class="p">:</span><span class="w"> </span><span class="mi">805212899</span><span class="p">,</span><span class="w">
</span><span class="nl">"modified"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2018-12-05T03:30:00Z"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535"</span><span class="p">,</span><span class="w">
</span><span class="nl">"labels"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"zh-hans"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh-hans"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zh-hant"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh-hant"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zh-hk"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh-hk"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jin Ping Mei"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zh"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"descriptions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Chinese naturalistic novel"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zh"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"中国古典小说"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"aliases"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"zh"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅词话"</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The Plum in the Golden Vase"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">".The Golden Lotus"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Plum in the Golden Vase"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Golden Lotus"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jinpingmei"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"claims"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"P50"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P50"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">832785</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q832785"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$fe287f80-43c8-22f7-a0a0-32315931fa20"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P373"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P373"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jinpingmei"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"q753535$7B87F320-CD1A-4A8F-A36D-0F562AFD6550"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="p">,</span><span class="w">
</span><span class="nl">"references"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fa278ebfc458360e5aed63d5058cca83c46134f1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"snaks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"P143"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P143"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">328</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q328"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"snaks-order"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"P143"</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P31"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P31"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">571</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q571"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"q753535$D7105EE3-F496-4639-B951-43C0F52D53BA"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="p">,</span><span class="w">
</span><span class="nl">"references"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fa278ebfc458360e5aed63d5058cca83c46134f1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"snaks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"P143"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P143"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">328</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q328"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"snaks-order"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"P143"</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P957"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P957"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2-07-031490-1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"external-id"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$1A821766-DDAF-4185-BB09-FCCEEBA8C0EC"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P136"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P136"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">8261</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q8261"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$63866EAC-F73C-4B89-9211-7771FE802D7B"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P136"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">11452132</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q11452132"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$E7AFFA14-BD4A-4F6A-BDE6-1BE66555758E"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="p">,</span><span class="w">
</span><span class="nl">"references"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"fa278ebfc458360e5aed63d5058cca83c46134f1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"snaks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"P143"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P143"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">328</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q328"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"snaks-order"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"P143"</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P577"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P577"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"+1610-00-00T00:00:00Z"</span><span class="p">,</span><span class="w">
</span><span class="nl">"timezone"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"before"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"after"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nl">"precision"</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w">
</span><span class="nl">"calendarmodel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://www.wikidata.org/entity/Q1985727"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"time"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"time"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$b7a4cc71-4f11-0068-df03-a566332ebf45"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nl">"P18"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"mainsnak"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P18"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jin Ping Mei.jpg"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"commonsMedia"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"statement"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q753535$B4550D8C-62D4-4868-8020-23973E43313A"</span><span class="p">,</span><span class="w">
</span><span class="nl">"rank"</span><span class="p">:</span><span class="w"> </span><span class="s2">"normal"</span><span class="p">,</span><span class="w">
</span><span class="nl">"references"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dbc378ca38aa06d2ac2a1b790ffe3a5ee9d0bd17"</span><span class="p">,</span><span class="w">
</span><span class="nl">"snaks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"P143"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nl">"snaktype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"value"</span><span class="p">,</span><span class="w">
</span><span class="nl">"property"</span><span class="p">:</span><span class="w"> </span><span class="s2">"P143"</span><span class="p">,</span><span class="w">
</span><span class="nl">"datavalue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"entity-type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"numeric-id"</span><span class="p">:</span><span class="w"> </span><span class="mi">53464</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q53464"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-entityid"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"datatype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"wikibase-item"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"snaks-order"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"P143"</span><span class="p">]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"sitelinks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"enwiki"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"site"</span><span class="p">:</span><span class="w"> </span><span class="s2">"enwiki"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jin Ping Mei"</span><span class="p">,</span><span class="w">
</span><span class="nl">"badges"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://en.wikipedia.org/wiki/Jin_Ping_Mei"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"enwikiquote"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"site"</span><span class="p">:</span><span class="w"> </span><span class="s2">"enwikiquote"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jin Ping Mei"</span><span class="p">,</span><span class="w">
</span><span class="nl">"badges"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://en.wikiquote.org/wiki/Jin_Ping_Mei"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zhwiki"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"site"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zhwiki"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="p">,</span><span class="w">
</span><span class="nl">"badges"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://zh.wikipedia.org/wiki/%E9%87%91%E7%93%B6%E6%A2%85"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"zhwikisource"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"site"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zhwikisource"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"金瓶梅"</span><span class="p">,</span><span class="w">
</span><span class="nl">"badges"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://zh.wikisource.org/wiki/%E9%87%91%E7%93%B6%E6%A2%85"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>如果需要更仔细的控制,如只要求部分语言和属性,又或一次性取得多个实体的信息,可以参考<a href="https://www.wikidata.org/w/api.php">Wikidata API</a>的文档,比如说<code class="language-plaintext highlighter-rouge">https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q1&props=labels|aliases|descriptions&languages=en|zh&format=json</code>可取得实体<code class="language-plaintext highlighter-rouge">Q1</code>在中文和英文的名称、别名和描述:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"entities"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Q1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"item"</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"labels"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"zh"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">5b87</span><span class="se">\u</span><span class="s2">5b99"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Universe"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"descriptions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"zh"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zh"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\u</span><span class="s2">4e00</span><span class="se">\u</span><span class="s2">5207</span><span class="se">\u</span><span class="s2">7a7a</span><span class="se">\u</span><span class="s2">95f4</span><span class="se">\u</span><span class="s2">3001</span><span class="se">\u</span><span class="s2">65f6</span><span class="se">\u</span><span class="s2">95f4</span><span class="se">\u</span><span class="s2">3001</span><span class="se">\u</span><span class="s2">7269</span><span class="se">\u</span><span class="s2">8d28</span><span class="se">\u</span><span class="s2">548c</span><span class="se">\u</span><span class="s2">80fd</span><span class="se">\u</span><span class="s2">91cf</span><span class="se">\u</span><span class="s2">6784</span><span class="se">\u</span><span class="s2">6210</span><span class="se">\u</span><span class="s2">7684</span><span class="se">\u</span><span class="s2">603b</span><span class="se">\u</span><span class="s2">4f53"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"totality of space and all contents"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"aliases"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"en"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Our Universe"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The Universe"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Universe (Ours)"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The Cosmos"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cosmos"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"success"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>同样地,搜索也可用API,如<code class="language-plaintext highlighter-rouge">https://www.wikidata.org/w/api.php?action=wbsearchentities&search=apache&language=en&limit=2&format=json</code>可搜索英文词”apache“并返回最多两个结果:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"searchinfo"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"search"</span><span class="p">:</span><span class="w"> </span><span class="s2">"apache"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"search"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"repository"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q102090"</span><span class="p">,</span><span class="w">
</span><span class="nl">"conceptIRI"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://www.wikidata.org/entity/Q102090"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q102090"</span><span class="p">,</span><span class="w">
</span><span class="nl">"pageid"</span><span class="p">:</span><span class="w"> </span><span class="mi">104694</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"//www.wikidata.org/wiki/Q102090"</span><span class="p">,</span><span class="w">
</span><span class="nl">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Apache"</span><span class="p">,</span><span class="w">
</span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"several culturally related groups of Native Americans in the United States"</span><span class="p">,</span><span class="w">
</span><span class="nl">"match"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"label"</span><span class="p">,</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"text"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Apache"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"repository"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q11354"</span><span class="p">,</span><span class="w">
</span><span class="nl">"conceptIRI"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://www.wikidata.org/entity/Q11354"</span><span class="p">,</span><span class="w">
</span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Q11354"</span><span class="p">,</span><span class="w">
</span><span class="nl">"pageid"</span><span class="p">:</span><span class="w"> </span><span class="mi">12854</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"//www.wikidata.org/wiki/Q11354"</span><span class="p">,</span><span class="w">
</span><span class="nl">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Apache HTTP Server"</span><span class="p">,</span><span class="w">
</span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"open-source web server software"</span><span class="p">,</span><span class="w">
</span><span class="nl">"match"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"label"</span><span class="p">,</span><span class="w">
</span><span class="nl">"language"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en"</span><span class="p">,</span><span class="w">
</span><span class="nl">"text"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Apache HTTP Server"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"search-continue"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"success"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h2 id="发掘语义网">发掘语义网</h2>
<p>上面介绍了获取关于一个实体的信息的方法,但现实中我们通常想要的是关于满足一定条件的所有实体的信息。虽然把Wikidata中所有数据从<a href="https://dumps.wikimedia.org/wikidatawiki/entities/">https://dumps.wikimedia.org/wikidatawiki/entities/</a>下载下来是可能的,但下载至少几十GB的压缩包下来本地处理对于许多用途而言太笨重了,而且如果不定期与上游同步(如下载最近30天的变更)数据就会开始变得过时。为此,Wikidata提供了一个查询服务<a href="https://query.wikidata.org/">https://query.wikidata.org/</a>,大家可以在浏览器使用其IDE编辑查询,也可以写程序向URL<code class="language-plaintext highlighter-rouge">https://query.wikidata.org/sparql?query=经编码的查询</code>发送<code class="language-plaintext highlighter-rouge">GET</code>或<code class="language-plaintext highlighter-rouge">POST</code>请求,返回结果的格式可用<code class="language-plaintext highlighter-rouge">HTTP</code>头<code class="language-plaintext highlighter-rouge">Accept: </code>的值<code class="language-plaintext highlighter-rouge">application/sparql-results+xml</code>、<code class="language-plaintext highlighter-rouge">application/sparql-results+json</code>、<code class="language-plaintext highlighter-rouge">text/tab-separated-values</code>、<code class="language-plaintext highlighter-rouge">text/csv</code>或<code class="language-plaintext highlighter-rouge">application/x-binary-rdf-results-table</code>决定。</p>
<p>这个查询服务接受的查询语言是<a href="https://www.w3.org/TR/sparql11-query/">SPARQL</a>,与受<a href="/prolog.html">Prolog</a>启发的SQL高度类似。为了尽早熟悉这语言,不妨先尝试<a href="https://query.wikidata.org/">https://query.wikidata.org/</a>页面提供的示例,然后可以尝试调整过滤器看看结果有什么变化。接着可以从空白查询出发通过新增过滤器建立查询,例如先输入”黄霑“选取Q709317作为客体,再选择属性作词者(P676)就会得到查询(含义是“哪些歌的作词人是黄霑?”):</p>
<pre><code class="language-SPARQL">SELECT ?__ ?__Label WHERE {
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
?__ wdt:P676 wd:Q709317.
}
LIMIT 100
</code></pre>
<p>提交查询就会得到结果如:</p>
<table>
<thead>
<tr>
<th><code class="language-plaintext highlighter-rouge">__</code></th>
<th><code class="language-plaintext highlighter-rouge">__Label</code></th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">wd:Q1743796</code></td>
<td>我的中國心</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">wd:Q10867969</code></td>
<td>上海灘</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">wd:Q15909571</code></td>
<td>獅子山下</td>
</tr>
</tbody>
</table>
<h3 id="sparql">SPARQL</h3>
<p>以下我们说明怎么写SPARQL查询。</p>
<h4 id="简单查询">简单查询</h4>
<p>如前所述,一个断言本质上是由主体、属性谓词和客体组成的三元组,它们三者通常都是用IRI表示的实体,因而一个典型的断言形如<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/entity/Q444381 http://www.wikidata.org/prop/direct/P50 http://www.wikidata.org/entity/Q23114</code>(这里<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/entity/Q444381</code>表示《孔乙己》、<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/prop/direct/P50</code>表示属性“作者是”、<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/entity/Q23114</code>表示鲁迅,因而这断言的意思是“《孔乙己》的作者是鲁迅”)。由于每次都使用完整的IRI太冗长,我们通常把常用的前缀如<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/entity/</code>和<code class="language-plaintext highlighter-rouge">http://www.wikidata.org/prop/direct/</code>分别简记为<code class="language-plaintext highlighter-rouge">wd:</code>和<code class="language-plaintext highlighter-rouge">wdt:</code>,于是上述断言可以简化为<code class="language-plaintext highlighter-rouge">wd:Q444381 wdt:P50 wd:Q23114</code>。当然,我们也可以自定义其它名称空间,如在查询前加上<code class="language-plaintext highlighter-rouge">PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#></code>后查询中就可用<code class="language-plaintext highlighter-rouge">rdfs:</code>代替IRI前缀<code class="language-plaintext highlighter-rouge">http://www.w3.org/2000/01/rdf-schema#</code>。</p>
<p>现在我们想在已知断言的一部分的前提下找出断言的其余部分,于是我们把断言中未知的部分用<code class="language-plaintext highlighter-rouge">?变量名</code>代替(变量名是我们选取的,不同变量名代表不同的未知量,如果不关心变量值也可用<code class="language-plaintext highlighter-rouge">_:变量名</code>),然后在前面加上<code class="language-plaintext highlighter-rouge">SELECT * WHERE{</code>而在后面加上<code class="language-plaintext highlighter-rouge">}</code>就得到查询,比如说:</p>
<table>
<thead>
<tr>
<th>查询</th>
<th>效果</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">SELECT * WHERE {?work wdt:P50 wd:Q23114}</code></td>
<td>查找鲁迅有哪些作品</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SELECT * WHERE {wd:Q444381 ?relationship wd:Q23114}</code></td>
<td>查找《孔乙己》与鲁迅有什么关系</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SELECT * WHERE {wd:Q444381 wdt:P50 ?author}</code></td>
<td>查找《孔乙己》的作者是谁</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">SELECT * WHERE {?work wdt:P50 ?author}</code></td>
<td>查找所有作品和它们分别的作者是谁</td>
</tr>
</tbody>
</table>
<p>尝试这些查询会分别得到一个表格,表格的各个列分别表示各未知变量的值而各行表示不同断言。不过,由于表格中的值都是实体的IRI,我们并不知道它们分别代表什么,我们更想要的是实体的名称。由于实体到它的名称通过属性谓词<code class="language-plaintext highlighter-rouge">rdfs:label</code>给出(顺便一提,别名由<code class="language-plaintext highlighter-rouge">skos:altLabel</code>给出、描述由<code class="language-plaintext highlighter-rouge">schema:description</code>给出),我们把查询修改为<code class="language-plaintext highlighter-rouge">SELECT * WHERE {?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName}</code>(这里两个元组用句点<code class="language-plaintext highlighter-rouge">.</code>分隔)。读者可能已经看出这相当于Prolog中的合一或SQL中的表连接,现在我们得到了鲁迅各作品的IRI和对应的名称。</p>
<p>不过,我们注意到同一作品在不同语言中名称都在不同行出来了,而我们只对中文名称感兴趣。由于语义网中的字符串是带语言标签的(如英文字符串表示为类似<code class="language-plaintext highlighter-rouge">"Hello world"@en</code>、繁体中文字符串表示为类似<code class="language-plaintext highlighter-rouge">"軟體"@zh-hant</code>),于是我们把查询改为<code class="language-plaintext highlighter-rouge">SELECT * WHERE {?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName. FILTER(LANG(?workName)='zh')}</code>。</p>
<p>假如我们还想要作品的英文名称,我们自然会写出<code class="language-plaintext highlighter-rouge">SELECT * WHERE {?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName. FILTER(LANG(?workName)='zh'). ?work rdfs:label ?workNameEn. FILTER(LANG(?workNameEn)='en')}</code>,但你会看到结果变少了,原因是许多作品没有英文名。如果我们只要求在有英文名时返回英文名,则可把查询改为:</p>
<pre><code class="language-SPARQL">SELECT * WHERE {
?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName.
FILTER(LANG(?workName)='zh').
OPTIONAL{?work rdfs:label ?workNameEn.FILTER(LANG(?workNameEn)='en')}
}
</code></pre>
<p>假如我们对实体IRI完全不感兴趣,可以把查询改为<code class="language-plaintext highlighter-rouge">SELECT ?workName WHERE {?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName. FILTER(LANG(?workName)='zh')}</code>,这样结果中只会有<code class="language-plaintext highlighter-rouge">?workName</code>一列,就像SQL中的投影。顺便一提,连接再投影后可能有重复结果,例如用<code class="language-plaintext highlighter-rouge">SELECT ?coAuthorName WHERE {?work wdt:P50 wd:Q173746.?work wdt:P50 ?coAuthor.FILTER(?coAuthor != wd:Q173746)?coAuthor rdfs:label ?coAuthorName.FILTER(LANG(?coAuthorName)='en')}</code>查询埃尔德什的合著者英文姓名的名单时,结果中有重复是由于有人与他合作多次的结果。要消除重复只用在<code class="language-plaintext highlighter-rouge">SELECT</code>后加上<code class="language-plaintext highlighter-rouge">DISTINCT</code>即可(如果我们只容许但不要求消除重复,则改用<code class="language-plaintext highlighter-rouge">REDUCED</code>)。</p>
<p>反过来,我们可能想要在结果可增加列,新列的值由原有列的值计算得到。例如我们可能想把英文标题变成大写:</p>
<pre><code class="language-SPARQL">SELECT ?workName ?workNameEnCap WHERE {
?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName.
FILTER(LANG(?workName)='zh').
OPTIONAL{?work rdfs:label ?workNameEn.FILTER(LANG(?workNameEn)='en')}
BIND(UCASE(?workNameEn) AS ?workNameEnCap)
}
</code></pre>
<p>一般来说结果集是无序的,但我们可以要求按变量值或更一般表达式的值排序,例如以下按英文名倒序排(排序则把<code class="language-plaintext highlighter-rouge">DESC</code>改为<code class="language-plaintext highlighter-rouge">ASC</code>):</p>
<pre><code class="language-SPARQL">SELECT ?workName ?workNameEnCap WHERE {
?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName.
FILTER(LANG(?workName)='zh').
OPTIONAL{?work rdfs:label ?workNameEn.FILTER(LANG(?workNameEn)='en')}
BIND(UCASE(?workNameEn) AS ?workNameEnCap)
}ORDER BY DESC(?workNameEnCap)
</code></pre>
<p>有时候结果多了看不过来,这就需要对结果集分页。如只要求返回第10个结果开始的5个结果:</p>
<pre><code class="language-SPARQL">SELECT ?workName ?workNameEnCap WHERE {
?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName.
FILTER(LANG(?workName)='zh').
OPTIONAL{?work rdfs:label ?workNameEn.FILTER(LANG(?workNameEn)='en')}
BIND(UCASE(?workNameEn) AS ?workNameEnCap)
}ORDER BY DESC(?workNameEnCap)
OFFSET 10 LIMIT 5
</code></pre>
<p>SPARQL还提供了一些语法糖让查询写起来更紧凑一点,以下举例说明:</p>
<table>
<thead>
<tr>
<th>片段</th>
<th>简化</th>
<th>附注</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">?work wdt:P50 wd:Q23114. ?work rdfs:label ?workName</code></td>
<td><code class="language-plaintext highlighter-rouge">?work wdt:P50 wd:Q23114; rdfs:label ?workName</code></td>
<td>相同主体的三元组</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">?work rdfs:label ?workName. ?work rdfs:label ?workNameEn</code></td>
<td><code class="language-plaintext highlighter-rouge">?work rdfs:label ?workName,?workNameEn</code></td>
<td>相同主体和谓词的元组</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">wd:Q444381 wdt:P50 _:author._:author wdt:P1412 ?language</code></td>
<td><code class="language-plaintext highlighter-rouge">wd:Q444381 wdt:P50/wdt:P1412 ?language</code></td>
<td>间接关系</td>
</tr>
</tbody>
</table>
<h4 id="聚合查询">聚合查询</h4>
<p>与SQL类似,SPARQL也有聚合子句。比如我们想查询鲁迅不同年份的作品数,可以使用以下查询:</p>
<pre><code class="language-SPARQL">SELECT ?year (COUNT(?work) AS ?count) WHERE {
?work wdt:P50 wd:Q23114.
?work wdt:P577 ?date.
}GROUP BY (YEAR(?date) AS ?year)
ORDER BY ?year
</code></pre>
<p>我们还可以过滤分组,比如只要作品数多于一的年份:</p>
<pre><code class="language-SPARQL">SELECT ?year (COUNT(?work) AS ?count) WHERE {
?work wdt:P50 wd:Q23114.
?work wdt:P577 ?date.
}GROUP BY (YEAR(?date) AS ?year)
HAVING (COUNT(?work)>1)
ORDER BY ?year
</code></pre>
<p>除了<code class="language-plaintext highlighter-rouge">COUNT</code>外,还有<code class="language-plaintext highlighter-rouge">MIN</code>、<code class="language-plaintext highlighter-rouge">MAX</code>、<code class="language-plaintext highlighter-rouge">AVG</code>、<code class="language-plaintext highlighter-rouge">SAMPLE</code>和<code class="language-plaintext highlighter-rouge">GROUP_CONCAT</code>可作聚合函数,需要去重复再聚合的话可以在<code class="language-plaintext highlighter-rouge">(</code>后加上<code class="language-plaintext highlighter-rouge">DISTINCT</code>。</p>
<h4 id="完整的查询语法">完整的查询语法</h4>
<p>有了前面的例子,我们已经了解SPARQL中最常用的<code class="language-plaintext highlighter-rouge">SELECT</code>语句的结构,另外还有用于判断结果集是否非空但不返回结果集的<code class="language-plaintext highlighter-rouge">ASK</code>语句、用于用结果集构造图的<code class="language-plaintext highlighter-rouge">CONSTRUCT</code>语句和返回实体全部属性的<code class="language-plaintext highlighter-rouge">DESCRIBE</code>语句。以下给出形式化的定义(关键词除用于表示类型谓词的<code class="language-plaintext highlighter-rouge">a</code>外都不区分大小写):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[1] QueryUnit ::= Query
[2] Query ::= Prologue
( SelectQuery | ConstructQuery | DescribeQuery | AskQuery )
ValuesClause
[4] Prologue ::= ( BaseDecl | PrefixDecl )*
[5] BaseDecl ::= 'BASE' IRIREF
[6] PrefixDecl ::= 'PREFIX' PNAME_NS IRIREF
[7] SelectQuery ::= SelectClause DatasetClause* WhereClause SolutionModifier
[8] SubSelect ::= SelectClause WhereClause SolutionModifier ValuesClause
[9] SelectClause ::= 'SELECT' ( 'DISTINCT' | 'REDUCED' )? ( ( Var | ( '(' Expression 'AS' Var ')' ) )+ | '*' )
[10] ConstructQuery ::= 'CONSTRUCT' ( ConstructTemplate DatasetClause* WhereClause SolutionModifier | DatasetClause* 'WHERE' '{' TriplesTemplate? '}' SolutionModifier )
[11] DescribeQuery ::= 'DESCRIBE' ( VarOrIri+ | '*' ) DatasetClause* WhereClause? SolutionModifier
[12] AskQuery ::= 'ASK' DatasetClause* WhereClause SolutionModifier
[13] DatasetClause ::= 'FROM' ( DefaultGraphClause | NamedGraphClause )
[14] DefaultGraphClause ::= SourceSelector
[15] NamedGraphClause ::= 'NAMED' SourceSelector
[16] SourceSelector ::= iri
[17] WhereClause ::= 'WHERE'? GroupGraphPattern
[18] SolutionModifier ::= GroupClause? HavingClause? OrderClause? LimitOffsetClauses?
[19] GroupClause ::= 'GROUP' 'BY' GroupCondition+
[20] GroupCondition ::= BuiltInCall | FunctionCall | '(' Expression ( 'AS' Var )? ')' | Var
[21] HavingClause ::= 'HAVING' HavingCondition+
[22] HavingCondition ::= Constraint
[23] OrderClause ::= 'ORDER' 'BY' OrderCondition+
[24] OrderCondition ::= ( ( 'ASC' | 'DESC' ) BrackettedExpression )
| ( Constraint | Var )
[25] LimitOffsetClauses ::= LimitClause OffsetClause? | OffsetClause LimitClause?
[26] LimitClause ::= 'LIMIT' INTEGER
[27] OffsetClause ::= 'OFFSET' INTEGER
[28] ValuesClause ::= ( 'VALUES' DataBlock )?
[52] TriplesTemplate ::= TriplesSameSubject ( '.' TriplesTemplate? )?
</code></pre></div></div>
<p>其中用于过滤断言的WHERE子句的组成如下:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[53] GroupGraphPattern ::= '{' ( SubSelect | GroupGraphPatternSub ) '}'
[54] GroupGraphPatternSub ::= TriplesBlock? ( GraphPatternNotTriples '.'? TriplesBlock? )*
[55] TriplesBlock ::= TriplesSameSubjectPath ( '.' TriplesBlock? )?
[56] GraphPatternNotTriples ::= GroupOrUnionGraphPattern | OptionalGraphPattern | MinusGraphPattern | GraphGraphPattern | ServiceGraphPattern | Filter | Bind | InlineData
[57] OptionalGraphPattern ::= 'OPTIONAL' GroupGraphPattern
[58] GraphGraphPattern ::= 'GRAPH' VarOrIri GroupGraphPattern
[59] ServiceGraphPattern ::= 'SERVICE' 'SILENT'? VarOrIri GroupGraphPattern
[60] Bind ::= 'BIND' '(' Expression 'AS' Var ')'
[61] InlineData ::= 'VALUES' DataBlock
[62] DataBlock ::= InlineDataOneVar | InlineDataFull
[63] InlineDataOneVar ::= Var '{' DataBlockValue* '}'
[64] InlineDataFull ::= ( NIL | '(' Var* ')' ) '{' ( '(' DataBlockValue* ')' | NIL )* '}'
[65] DataBlockValue ::= iri | RDFLiteral | NumericLiteral | BooleanLiteral | 'UNDEF'
[66] MinusGraphPattern ::= 'MINUS' GroupGraphPattern
[67] GroupOrUnionGraphPattern ::= GroupGraphPattern ( 'UNION' GroupGraphPattern )*
[68] Filter ::= 'FILTER' Constraint
[69] Constraint ::= BrackettedExpression | BuiltInCall | FunctionCall
[70] FunctionCall ::= iri ArgList
[71] ArgList ::= NIL | '(' 'DISTINCT'? Expression ( ',' Expression )* ')'
[72] ExpressionList ::= NIL | '(' Expression ( ',' Expression )* ')'
[73] ConstructTemplate ::= '{' ConstructTriples? '}'
[74] ConstructTriples ::= TriplesSameSubject ( '.' ConstructTriples? )?
[75] TriplesSameSubject ::= VarOrTerm PropertyListNotEmpty | TriplesNode PropertyList
[76] PropertyList ::= PropertyListNotEmpty?
[77] PropertyListNotEmpty ::= Verb ObjectList ( ';' ( Verb ObjectList )? )*
[78] Verb ::= VarOrIri | 'a'
[79] ObjectList ::= Object ( ',' Object )*
[80] Object ::= GraphNode
[81] TriplesSameSubjectPath ::= VarOrTerm PropertyListPathNotEmpty | TriplesNodePath PropertyListPath
[82] PropertyListPath ::= PropertyListPathNotEmpty?
[83] PropertyListPathNotEmpty ::= ( VerbPath | VerbSimple ) ObjectListPath ( ';' ( ( VerbPath | VerbSimple ) ObjectList )? )*
[84] VerbPath ::= Path
[85] VerbSimple ::= Var
[86] ObjectListPath ::= ObjectPath ( ',' ObjectPath )*
[87] ObjectPath ::= GraphNodePath
[88] Path ::= PathAlternative
[89] PathAlternative ::= PathSequence ( '|' PathSequence )*
[90] PathSequence ::= PathEltOrInverse ( '/' PathEltOrInverse )*
[91] PathElt ::= PathPrimary PathMod?
[92] PathEltOrInverse ::= PathElt | '^' PathElt
[93] PathMod ::= '?' | '*' | '+'
[94] PathPrimary ::= iri | 'a' | '!' PathNegatedPropertySet | '(' Path ')'
[95] PathNegatedPropertySet ::= PathOneInPropertySet | '(' ( PathOneInPropertySet ( '|' PathOneInPropertySet )* )? ')'
[96] PathOneInPropertySet ::= iri | 'a' | '^' ( iri | 'a' )
[97] Integer ::= INTEGER
[98] TriplesNode ::= Collection | BlankNodePropertyList
[99] BlankNodePropertyList ::= '[' PropertyListNotEmpty ']'
[100] TriplesNodePath ::= CollectionPath | BlankNodePropertyListPath
[101] BlankNodePropertyListPath ::= '[' PropertyListPathNotEmpty ']'
[102] Collection ::= '(' GraphNode+ ')'
[103] CollectionPath ::= '(' GraphNodePath+ ')'
[104] GraphNode ::= VarOrTerm | TriplesNode
[105] GraphNodePath ::= VarOrTerm | TriplesNodePath
[106] VarOrTerm ::= Var | GraphTerm
[107] VarOrIri ::= Var | iri
[108] Var ::= VAR1 | VAR2
[109] GraphTerm ::= iri | RDFLiteral | NumericLiteral | BooleanLiteral | BlankNode | NIL
</code></pre></div></div>
<p>接下来是表达式语法,虽然看起来可能有点复杂,但由于语法和语义上都与其它常见语言差不多,没有必要作太多讲解。</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[110] Expression ::= ConditionalOrExpression
[111] ConditionalOrExpression ::= ConditionalAndExpression ( '||' ConditionalAndExpression )*
[112] ConditionalAndExpression ::= ValueLogical ( '&&' ValueLogical )*
[113] ValueLogical ::= RelationalExpression
[114] RelationalExpression ::= NumericExpression ( '=' NumericExpression | '!=' NumericExpression | '<' NumericExpression | '>' NumericExpression | '<=' NumericExpression | '>=' NumericExpression | 'IN' ExpressionList | 'NOT' 'IN' ExpressionList )?
[115] NumericExpression ::= AdditiveExpression
[116] AdditiveExpression ::= MultiplicativeExpression ( '+' MultiplicativeExpression | '-' MultiplicativeExpression | ( NumericLiteralPositive | NumericLiteralNegative ) ( ( '*' UnaryExpression ) | ( '/' UnaryExpression ) )* )*
[117] MultiplicativeExpression ::= UnaryExpression ( '*' UnaryExpression | '/' UnaryExpression )*
[118] UnaryExpression ::= '!' PrimaryExpression
| '+' PrimaryExpression
| '-' PrimaryExpression
| PrimaryExpression
[119] PrimaryExpression ::= BrackettedExpression | BuiltInCall | iriOrFunction | RDFLiteral | NumericLiteral | BooleanLiteral | Var
[120] BrackettedExpression ::= '(' Expression ')'
[121] BuiltInCall ::= Aggregate
| 'STR' '(' Expression ')'
| 'LANG' '(' Expression ')'
| 'LANGMATCHES' '(' Expression ',' Expression ')'
| 'DATATYPE' '(' Expression ')'
| 'BOUND' '(' Var ')'
| 'IRI' '(' Expression ')'
| 'IRI' '(' Expression ')'
| 'BNODE' ( '(' Expression ')' | NIL )
| 'RAND' NIL
| 'ABS' '(' Expression ')'
| 'CEIL' '(' Expression ')'
| 'FLOOR' '(' Expression ')'
| 'ROUND' '(' Expression ')'
| 'CONCAT' ExpressionList
| SubstringExpression
| 'STRLEN' '(' Expression ')'
| StrReplaceExpression
| 'UCASE' '(' Expression ')'
| 'LCASE' '(' Expression ')'
| 'ENCODE_FOR_IRI' '(' Expression ')'
| 'CONTAINS' '(' Expression ',' Expression ')'
| 'STRSTARTS' '(' Expression ',' Expression ')'
| 'STRENDS' '(' Expression ',' Expression ')'
| 'STRBEFORE' '(' Expression ',' Expression ')'
| 'STRAFTER' '(' Expression ',' Expression ')'
| 'YEAR' '(' Expression ')'
| 'MONTH' '(' Expression ')'
| 'DAY' '(' Expression ')'
| 'HOURS' '(' Expression ')'
| 'MINUTES' '(' Expression ')'
| 'SECONDS' '(' Expression ')'
| 'TIMEZONE' '(' Expression ')'
| 'TZ' '(' Expression ')'
| 'NOW' NIL
| 'UUID' NIL
| 'STRUUID' NIL
| 'MD5' '(' Expression ')'
| 'SHA1' '(' Expression ')'
| 'SHA256' '(' Expression ')'
| 'SHA384' '(' Expression ')'
| 'SHA512' '(' Expression ')'
| 'COALESCE' ExpressionList
| 'IF' '(' Expression ',' Expression ',' Expression ')'
| 'STRLANG' '(' Expression ',' Expression ')'
| 'STRDT' '(' Expression ',' Expression ')'
| 'sameTerm' '(' Expression ',' Expression ')'
| 'isIRI' '(' Expression ')'
| 'isIRI' '(' Expression ')'
| 'isBLANK' '(' Expression ')'
| 'isLITERAL' '(' Expression ')'
| 'isNUMERIC' '(' Expression ')'
| RegexExpression
| ExistsFunc
| NotExistsFunc
[122] RegexExpression ::= 'REGEX' '(' Expression ',' Expression ( ',' Expression )? ')'
[123] SubstringExpression ::= 'SUBSTR' '(' Expression ',' Expression ( ',' Expression )? ')'
[124] StrReplaceExpression ::= 'REPLACE' '(' Expression ',' Expression ',' Expression ( ',' Expression )? ')'
[125] ExistsFunc ::= 'EXISTS' GroupGraphPattern
[126] NotExistsFunc ::= 'NOT' 'EXISTS' GroupGraphPattern
[127] Aggregate ::= 'COUNT' '(' 'DISTINCT'? ( '*' | Expression ) ')'
| 'SUM' '(' 'DISTINCT'? Expression ')'
| 'MIN' '(' 'DISTINCT'? Expression ')'
| 'MAX' '(' 'DISTINCT'? Expression ')'
| 'AVG' '(' 'DISTINCT'? Expression ')'
| 'SAMPLE' '(' 'DISTINCT'? Expression ')'
| 'GROUP_CONCAT' '(' 'DISTINCT'? Expression ( ';' 'SEPARATOR' '=' String )? ')'
[128] iriOrFunction ::= iri ArgList?
[129] RDFLiteral ::= String ( LANGTAG | ( '^^' iri ) )?
[130] NumericLiteral ::= NumericLiteralUnsigned | NumericLiteralPositive | NumericLiteralNegative
[131] NumericLiteralUnsigned ::= INTEGER | DECIMAL | DOUBLE
[132] NumericLiteralPositive ::= INTEGER_POSITIVE | DECIMAL_POSITIVE | DOUBLE_POSITIVE
[133] NumericLiteralNegative ::= INTEGER_NEGATIVE | DECIMAL_NEGATIVE | DOUBLE_NEGATIVE
[134] BooleanLiteral ::= 'true' | 'false'
[135] String ::= STRING_LITERAL1 | STRING_LITERAL2 | STRING_LITERAL_LONG1 | STRING_LITERAL_LONG2
[136] iri ::= IRIREF | PrefixedName
[137] PrefixedName ::= PNAME_LN | PNAME_NS
[138] BlankNode ::= BLANK_NODE_LABEL | ANON
</code></pre></div></div>
<p>最后来到最基本的词法。SPARQL的单词可分为:</p>
<ul>
<li>IRI。
<ul>
<li>由<code class="language-plaintext highlighter-rouge"><</code>与<code class="language-plaintext highlighter-rouge">></code>包围的IRI。对于相对IRI,会基于<code class="language-plaintext highlighter-rouge">BaseDecl</code>中给出的IRI解析。</li>
<li>形如<code class="language-plaintext highlighter-rouge">名字空间:本地名</code>,表示由名字空间表示的前缀后紧接本地名,两个部分都可以省略。</li>
</ul>
</li>
<li>字面值。用成对的<code class="language-plaintext highlighter-rouge">'</code>、<code class="language-plaintext highlighter-rouge">"</code>、<code class="language-plaintext highlighter-rouge">'''</code>或<code class="language-plaintext highlighter-rouge">"""</code>包围的文本,其中可用常见的反斜杠转义序列。字符串后可接<code class="language-plaintext highlighter-rouge">@</code>和语言标签来指定语言,或者接<code class="language-plaintext highlighter-rouge">^^</code>和用IRI表示的数据类型。对于一些常见类型有缩写,比如:
<ul>
<li><code class="language-plaintext highlighter-rouge">1</code>相当于<code class="language-plaintext highlighter-rouge">"1"^^xsd:integer</code></li>
<li><code class="language-plaintext highlighter-rouge">1.3</code>相当于<code class="language-plaintext highlighter-rouge">"1.3"^^xsd:decimal</code></li>
<li><code class="language-plaintext highlighter-rouge">1.0e6</code>相当于<code class="language-plaintext highlighter-rouge">"1.0e6"^^xsd:double</code></li>
<li><code class="language-plaintext highlighter-rouge">true</code>相当于<code class="language-plaintext highlighter-rouge">"true"^^xsd:boolean</code></li>
<li><code class="language-plaintext highlighter-rouge">false</code>相当于<code class="language-plaintext highlighter-rouge">"false"^^xsd:boolean</code></li>
</ul>
</li>
<li>空白结点。形如<code class="language-plaintext highlighter-rouge">_:标签</code>或<code class="language-plaintext highlighter-rouge">[]</code>,不同标签空白结点类似于不同的变量,可用于匹配但不需要引用值的场合。</li>
<li>变量。由<code class="language-plaintext highlighter-rouge">$</code>或<code class="language-plaintext highlighter-rouge">?</code>开首接变量名组成,其中变量名用一般非特殊字符都可以,至少字母、数字、下划线总是容许的,包括多数Unicode字符。</li>
<li>行末注释由<code class="language-plaintext highlighter-rouge">#</code>开始,会被忽略。</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[139] IRIREF ::= '<' ([^<>"{}|^`\]-[#x00-#x20])* '>'
[140] PNAME_NS ::= PN_PREFIX? ':'
[141] PNAME_LN ::= PNAME_NS PN_LOCAL
[142] BLANK_NODE_LABEL ::= '_:' ( PN_CHARS_U | [0-9] ) ((PN_CHARS|'.')* PN_CHARS)?
[143] VAR1 ::= '?' VARNAME
[144] VAR2 ::= '$' VARNAME
[145] LANGTAG ::= '@' [a-zA-Z]+ ('-' [a-zA-Z0-9]+)*
[146] INTEGER ::= [0-9]+
[147] DECIMAL ::= [0-9]* '.' [0-9]+
[148] DOUBLE ::= [0-9]+ '.' [0-9]* EXPONENT | '.' ([0-9])+ EXPONENT | ([0-9])+ EXPONENT
[149] INTEGER_POSITIVE ::= '+' INTEGER
[150] DECIMAL_POSITIVE ::= '+' DECIMAL
[151] DOUBLE_POSITIVE ::= '+' DOUBLE
[152] INTEGER_NEGATIVE ::= '-' INTEGER
[153] DECIMAL_NEGATIVE ::= '-' DECIMAL
[154] DOUBLE_NEGATIVE ::= '-' DOUBLE
[155] EXPONENT ::= [eE] [+-]? [0-9]+
[156] STRING_LITERAL1 ::= "'" ( ([^#x27#x5C#xA#xD]) | ECHAR )* "'"
[157] STRING_LITERAL2 ::= '"' ( ([^#x22#x5C#xA#xD]) | ECHAR )* '"'
[158] STRING_LITERAL_LONG1 ::= "'''" ( ( "'" | "''" )? ( [^'\] | ECHAR ) )* "'''"
[159] STRING_LITERAL_LONG2 ::= '"""' ( ( '"' | '""' )? ( [^"\] | ECHAR ) )* '"""'
[160] ECHAR ::= '\' [tbnrf\"']
[161] NIL ::= '(' WS* ')'
[162] WS ::= #x20 | #x9 | #xD | #xA
[163] ANON ::= '[' WS* ']'
[164] PN_CHARS_BASE ::= [A-Z] | [a-z] | [#x00C0-#x00D6] | [#x00D8-#x00F6] | [#x00F8-#x02FF] | [#x0370-#x037D] | [#x037F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
[165] PN_CHARS_U ::= PN_CHARS_BASE | '_'
[166] VARNAME ::= ( PN_CHARS_U | [0-9] ) ( PN_CHARS_U | [0-9] | #x00B7 | [#x0300-#x036F] | [#x203F-#x2040] )*
[167] PN_CHARS ::= PN_CHARS_U | '-' | [0-9] | #x00B7 | [#x0300-#x036F] | [#x203F-#x2040]
[168] PN_PREFIX ::= PN_CHARS_BASE ((PN_CHARS|'.')* PN_CHARS)?
[169] PN_LOCAL ::= (PN_CHARS_U | ':' | [0-9] | PLX ) ((PN_CHARS | '.' | ':' | PLX)* (PN_CHARS | ':' | PLX) )?
[170] PLX ::= PERCENT | PN_LOCAL_ESC
[171] PERCENT ::= '%' HEX HEX
[172] HEX ::= [0-9] | [A-F] | [a-f]
[173] PN_LOCAL_ESC ::= '\' ( '_' | '~' | '.' | '-' | '!' | '$' | '&' | "'" | '(' | ')' | '*' | '+' | ',' | ';' | '=' | '/' | '?' | '#' | '@' | '%' )
</code></pre></div></div>
<p>其实,SPARQL的完整语法还支持修改数据:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[3] UpdateUnit ::= Update
[29] Update ::= Prologue ( Update1 ( ';' Update )? )?
[30] Update1 ::= Load | Clear | Drop | Add | Move | Copy | Create | InsertData | DeleteData | DeleteWhere | Modify
[31] Load ::= 'LOAD' 'SILENT'? iri ( 'INTO' GraphRef )?
[32] Clear ::= 'CLEAR' 'SILENT'? GraphRefAll
[33] Drop ::= 'DROP' 'SILENT'? GraphRefAll
[34] Create ::= 'CREATE' 'SILENT'? GraphRef
[35] Add ::= 'ADD' 'SILENT'? GraphOrDefault 'TO' GraphOrDefault
[36] Move ::= 'MOVE' 'SILENT'? GraphOrDefault 'TO' GraphOrDefault
[37] Copy ::= 'COPY' 'SILENT'? GraphOrDefault 'TO' GraphOrDefault
[38] InsertData ::= 'INSERT DATA' QuadData
[39] DeleteData ::= 'DELETE DATA' QuadData
[40] DeleteWhere ::= 'DELETE WHERE' QuadPattern
[41] Modify ::= ( 'WITH' iri )? ( DeleteClause InsertClause? | InsertClause ) UsingClause* 'WHERE' GroupGraphPattern
[42] DeleteClause ::= 'DELETE' QuadPattern
[43] InsertClause ::= 'INSERT' QuadPattern
[44] UsingClause ::= 'USING' ( iri | 'NAMED' iri )
[45] GraphOrDefault ::= 'DEFAULT' | 'GRAPH'? iri
[46] GraphRef ::= 'GRAPH' iri
[47] GraphRefAll ::= GraphRef | 'DEFAULT' | 'NAMED' | 'ALL'
[48] QuadPattern ::= '{' Quads '}'
[49] QuadData ::= '{' Quads '}'
[50] Quads ::= TriplesTemplate? ( QuadsNotTriples '.'? TriplesTemplate? )*
[51] QuadsNotTriples ::= 'GRAPH' VarOrIri '{' TriplesTemplate? '}'
</code></pre></div></div>
<h2 id="结语">结语</h2>
<p>Wikidata作为一个协作项目,我们在从中获益的同时也理应积极向它捐赠数据,成为一个负责任的公民。例如开发基于它的应用时不妨加入容许用户纠错的机制,并把经审查的纠错信息反馈给上游。</p>陈颂光人们用他们熟悉的自然语言如中文在网上编辑了海量的信息,但自然语言固有的极端复杂性使这些文本难以为机器所理解,于是人们因为找不到已有的信息而重复劳动,这个恶性循环的结果是互联网上文章泛滥但质量普遍低下,特别是过时信息得不到清理。相反,语义网提倡用更规范的方式表达知识,并把它们连接起来。维基基金会的Wikidata就是一个人人可编辑和利用的知识库,有大量志愿者持续扩展和修正它,这使它的数据无论在数量上还是质量上都处于领先地位,维基百科中条目右上方卡片的信息就来自Wikidata。值得一提的是Wikidata的数据位于公众领域,可以(接近)无条件地合法使用。