forked from aymara/lima
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlima-user-manual.xml
294 lines (284 loc) · 19 KB
/
lima-user-manual.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" "/usr/share/sgml/docbook/xml-dtd-4.1.2/docbookx.dtd" []>
<article>
<title>LIMA User Manual</title>
<articleinfo>
<author>
<firstname>Gaël</firstname>
<surname>de Chalendar</surname>
</author>
<releaseinfo>Version 0.1</releaseinfo>
<copyright>
<year>2011</year>
<holder>CEA LIST</holder>
</copyright>
</articleinfo>
<abstract>
<title>Abstract</title>
<para>This documents explains how to install, run, configure and integrate into applications LIMA, the CEA LIST Multilingual Analyzer.</para>
</abstract>
<sect1>
<title>Installation</title>
<sect2>
<title>Debian Lenny deb packages</title>
<para>Install the following packages using dpkg: </para>
<itemizedlist>
<listitem>
<para>svmtool++-1.1.6-Linux.deb</para>
</listitem>
<listitem>
<para>mmcommon-1.0.0-Linux.deb</para>
</listitem>
<listitem>
<para>mmlinguisticprocessing-1.0.0-Linux.deb</para>
</listitem>
<listitem>
<para>mmlinguisticdata-prereq-1.0.0-Linux.deb</para>
</listitem>
<listitem>
<para>mmlinguisticdata-1.0.0-Linux.deb</para>
</listitem>
</itemizedlist>
</sect2>
<sect2>
<title>Mandriva 10.2 RPM packages</title>
<para>Install the following packages using urpmi: </para>
<itemizedlist>
<listitem><para>svmtool++-1.1.6-Linux.rpm</para></listitem>
<listitem><para>mmcommon-1.0.0-Linux.rpm</para></listitem>
<listitem><para>mmlinguisticprocessing-1.0.0-Linux.rpm</para></listitem>
<listitem><para>mmlinguisticdata-prereq-1.0.0-Linux.rpm</para></listitem>
<listitem><para>mmlinguisticdata-1.0.0-Linux.rpm</para></listitem>
</itemizedlist>
<para>If not already installed, the following dependencies will be automatically downloaded and installed: </para>
<itemizedlist>
<listitem><para>libstdc++</para></listitem>
<listitem><para>stdc++-devel</para></listitem>
<listitem><para>boost_date_time</para></listitem>
<listitem><para>...</para></listitem>
</itemizedlist>
<para>If the simple urpmi command fail because of possible errors in dependencies from package libicu or another one, you could have to use options <emphasis>--force</emphasis> and <emphasis>--nodeps</emphasis> like:
<programlisting>
sudo rpm -i --nodeps --force mmcommon-1.0.0-Linux.rpm
</programlisting>
</para>
<para>To know where all the files are installed (include, libraries, configuration...), you can run the following command:
<programlisting>
rpm -ql mmcommon-1.0.0
</programlisting>
This will be usefull to edit configuration files and resources in order to customize your installation.
</para>
</sect2>
<sect2>
<title>Sources compilation</title>
<para>TO BE WRITTEN</para>
</sect2>
</sect1>
<sect1>
<title>Running LIMA for the first time</title>
<para>In this section, we suppose that LIMA has been installed using binary packages (RPM or DEB depending on your distribution).</para>
<para>Choose UTF-8 encoded text files in one language supported by your LIMA version (English in this example) and run the following commands:</para>
<para><command>cd /path/to/your/text/files/folder</command></para>
<para><command>analyzeText -l fre file.txt</command><footnote>
<para>If you have several files to analyze and a multprocessor system, you can replace "analyzeText" by "threadedAnalyzeText -t n" where n is the number of threads you want to run.</para>
</footnote></para>
<para>This will produce a file named <command>file.txt.bin</command> containing a "bag of words" (BoW) binary representation of the file. You can view the content of this file using the <command>readBowFile</command> command:</para>
<para><command>readBowFile file.txt.bin</command></para>
<para>This kind of file contains lemmas of simple terms (nouns, adjectives and verbs), representation of complex terms and named entities found in the text. Suppose that the content of the text file was "On 4th April 2011, Bill Williams looked at Paris from his home window.". Then, the output of the <command>readBowFile</command> command will be:</para>
<para><programlisting>(4th_April_2011-8192-4)->[*(4th-8192-4)(April-16384-8)(2011-8192-14)]:DateTime.DATE:date=2011-04-04;value=4th April 2011
(Bill_Williams-16384-20)->[*(Bill-16384-20)(Williams-16384-25)]:Person.PERSON:firstname=Bill;lastname=Williams;value=Bill Williams
(look-49152-34)
(Paris-16384-44)->[*(Paris-16384-44)]:Location.LOCATION:value=Paris
(home-8192-59)
(window_home-8192-59)->[*(window-8192-64)/-3/-2/(home-8192-59)]</programlisting>As you can see, dates, person names and locations are recognized as such. Also, each line describes a recognized term, either simple (the verb "to look") complex. Complex terms can be nominal compounds ("home window") or named entities ("Paris", "Bill Williams", "4th April 2011"). Each term is described by its normalized form, the numerical value of its category (more on this later) and its position in the text. This is followed by the details of its structure.</para>
<para>This text representation of BoW is not designed to be used later by other programs. It's just to get an idea of its content. A dedicated C++ API allows to manipulate bags of words.</para>
</sect1>
<sect1>
<title>Configuring LIMA</title>
<para>If you want to configure LIMA for your own needs, you will have to make a copy of the configuration directory and define the environment variable <computeroutput>LIMA_CONF</computeroutput> to point to this copy:</para>
<programlisting>mkdir ~/MyLima
cp -R /usr/share/config/lima ~/MyLima/conf
setenv LIMA_CONF=/~/MyLima/conf</programlisting>
<para>Later versions of this manual will describe in details the various configuration possibilities, but suppose for now that you don't need the complex terms extracted by LIMA. Then, you only have to disable syntactic analysis and complex term computing. To do that for English, edit the file <computeroutput>$LIMA_CONF/lima-lp-eng.xml</computeroutput> and comment out the following lines in the <computeroutput>main</computeroutput> group of the <computeroutput>Processors</computeroutput> module:</para>
<programlisting><item value="syntacticAnalyzerChains"/>
<item value="syntacticAnalyzerDeps"/>
<item value="syntacticAnalyzerDepsHetero"/>
<item value="dotDepGraphWriter"/>
<item value="syntacticAnalysisXmlLogger"/>
<item value="compoundBuilderFromSyntacticData"/></programlisting>
</sect1>
<sect1>
<title>Creating a Modex: extracting new kind of entities</title>
<sect2>
<title>Introduction</title>
<para>A Modex ("Module d'Extraction") is a set of compiled regular expression-like rules with their accompanying configuration file. It is the base tool in Lima for various things, including idiomatic expression recognizing and named entities extraction but also syntactic analysis. You can create you own Modexes to extract entities specific to your application. For example, Twitter ids and Twitter hash tags are not natively supported by Lima. So, if your application is targeted at analyzing Tweets, then you will have to write your own Modex to extract them.</para>
</sect2>
<sect2>
<title>Preparing resources folder</title>
<para>In the current version, all Lima resources, including Modex binary rules files, must be under the same parent directory. So, to use you own Modex, you will have either to copy your files under the installation resource directory, as root if you use RPM or DEB packages, or to copy all the resources in a directory of your own and define the <computeroutput>LIMA_RESOURCES</computeroutput> environment variable to point to this one. In this user manual , we will use this second solution:</para>
<programlisting>cp -R /usr/share/apps/lima/resources ~/MyLima/resources
setenv LIMA_RESOURCES=~/MyLima/resources</programlisting>
</sect2>
<sect2>
<title>TwitterModex</title>
<sect3>
<title>The configuration file</title>
<para>There is only one configuration file for all languages supported by the Modex (e.g.: <filename>Twitter-modex.xml</filename>). It must be installed in the <computeroutput>LIMA_CONF</computeroutput> folder. It contains:</para>
<itemizedlist>
<listitem>
<para>a groups and entity types definition module;</para>
</listitem>
<listitem>
<para>a processing units (processUnit) definition module;</para>
</listitem>
<listitem>
<para>for each language, a resources to use definition module;</para>
</listitem>
</itemizedlist>
<sect4>
<title>Groups and Types</title>
<para>This first module, named entities contains a group for each entities group and, in this group, the list (named entityList) of the entity types. For example:
</para>
<programlisting><module name="entities">
<group name="Twitter">
<list name="entityList">
<item value="TWITTERID"/>
<item value="TWITTERHASH"/>
</list>
</group>
</module></programlisting>
</sect4>
<sect4>
<title>Process units</title>
<para>This module, named Processors, defines the processing units groups available for this Modex. These processing units can be pipelines (class ProcessUnitPipeline), which allows to define a global process unit for the Modex chaining up several rules application.
<programlisting><module name="Processors">
<group name="TwitterModex" class="ProcessUnitPipeline" >
<list name="processUnitSequence">
<item value="TwitterRecognition"/>
</list>
</group></programlisting>As in the analysis configuration file, each process unit is defined in its own group with its parameters. For an ApplyRecognizer process unit, these parameters are:
</para>
<itemizedlist>
<listitem>
<para>automaton: the name of a resource defined later in the resources specific to each language (Cf. next section);</para>
</listitem>
<listitem>
<para>applyOnGraph: declare the name of the analysis graph on which to apply the rules (AnalysisGraph if before part-of-speech tagging or PosGraph
after);</para>
</listitem>
<listitem>
<para>useSentenceBounds: (yes or no) defines if the automaton will be applied bbetween each sentence limits or on the whole graph.
Use "no" if this Modex will be used before the "sentenceBoundariesFinder" process unit.</para>
</listitem>
</itemizedlist>
<para><programlisting><group name="TwitterRecognition" class="ApplyRecognizer">
<param key="automaton" value="TwitterRules"/>
<param key="applyOnGraph" value="AnalysisGraph"/>
<param key="useSentenceBounds" value="no"/>
</group></programlisting></para>
</sect4>
<sect4>
<title>Resources</title>
<para>The resources definition modules for each language are called resources-xyz, with xyz the language trigram:
<computeroutput><module name=“resources-xyz”></computeroutput>. They contain a group for each automaton defined above. This group, of the class AutomatonRecognizer defines the extractor parameters and particularly the path to the compiled rules file (relative to the global Lima resources directory):
<programlisting><group name="TwitterRules" class="AutomatonRecognizer">
<param key="rules" value="Twitter/Twitter-eng.bin"/>
</group></programlisting>
Next, the module contains groups to define the microcategories<footnote>
<para>this will be defined in a later version of this document. Currently, you can use the values L_NOM_PROPRE for English and L_NC_GEN for French.</para>
</footnote> that will be affected to the token that will replace each recognized entity. The name of each of these groups must be the one of the corresponding group in the entities module concatenated to the string "Micros". It contains a list of microcategories for each entity. This list is named by the fully qualified name of the entity (<Group name>.<Entity name>):
<programlisting><group name="TwitterMicros" class="SpecificEntitiesMicros">
<list name="Twitter.TWITTERID">
<item value="L_NOM_PROPRE"/>
</list>
</group></programlisting></para>
</sect4>
<sect4>
<title>A complete configuration file</title>
<para><programlisting><?xml version='1.0' encoding='UTF-8'?>
<modulesConfig>
<module name="entities">
<group name="Twitter">
<list name="entityList">
<item value="TWITTERID"/>
<item value="TWITTERHASH"/>
</list>
</group>
</module>
<module name="Processors">
<group name="TwitterModex" class="ProcessUnitPipeline" >
<list name="processUnitSequence">
<item value="TwitterRecognition"/>
</list>
</group>
<group name="TwitterRecognition" class="ApplyRecognizer">
<param key="automaton" value="TwitterRules"/>
<param key="applyOnGraph" value="AnalysisGraph"/>
<param key="useSentenceBounds" value="no"/>
</group>
</module>
<module name="resources-eng">
<group name="TwitterRules" class="AutomatonRecognizer">
<param key="rules" value="Twitter/Twitter-eng.bin"/>
</group>
<group name="TwitterMicros" class="SpecificEntitiesMicros">
<list name="Twitter.TWITTERID">
<item value="L_NOM_PROPRE"/>
</list>
<list name="Twitter.TWITTERHASH">
<item value="L_NOM_PROPRE"/>
</list>
</group>
</module>
</modulesConfig></programlisting></para>
</sect4>
</sect3>
<sect3>
<title>The rules files</title>
<para>The full syntax of rules files is described in the document lima-modex-rules-format.pdf. Here, we just describe the following example:</para>
<programlisting>set encoding=utf8
using modex Twitter-modex.xml
using groups Twitter
set defaultAction=>CreateSpecificEntity()
#----------------------------------------------------------------------
# recognition of Twitter ids
#----------------------------------------------------------------------
@arobase=(\@)
@arobase::*:TWITTERID:
\#::*:TWITTERHASH:</programlisting>
<para>The first four lines are metadata stating that the file is encoded in UTF-8, that it is a rules file for the Twitter Modex, that the entities created by rules will belong to the Twitter group and finally that by default, the action associated to the rules will be to create a specific entity.</para>
<para>Next comes a line describing a class of tokens, here just the tokens composed of the arobase character. After this line, there is two rules, one triggered by the encountering of an arobase and matching any token after it. If matching, a TWITTERID entity is created. The second one has the same format but triggered by a hash character token and creating a TWITTERHASH entity.</para>
<para>Please refer to the full syntax description for details, but let's say here that rules are defined by a triggering token, followed by a regular expression describing the left context of the triggering token and a second one describing its right context, followed by the type of the expression and possibly constraint functions.</para>
<para>When the rules file is ready, you have to compile it with the following command (don't forget to install the configuration file beforehand):</para>
<para><command>compile-rules --language=eng --modex=Twitter-modex.xml -oTwitter-eng.bin Twitter-eng.rules </command></para>
<para>Then copy the binary file to the <computeroutput>LIMA_RESOURCES/Twitter</computeroutput> folder (as stated in the configuration file).</para>
</sect3>
<sect3>
<title>Using your new Modex</title>
<para>In the analysis configuration file (<filename>lima-lp-xyz.xml</filename>), a Modex is included by including explicitly its processings and its resources :
<programlisting><module name="Processors">
<group name="include">
<list name="includeList">
<item value="Twitter-modex.xml/Processors"/>
</list>
</group>
...
</module>
<module name="Resources">
<group name="include">
<list name="includeList">
<item value="Twitter-modex.xml/resources-xyz"/>
</list>
</group>
...
</module></programlisting></para>
<para>The Modex process unit(s) can then be called in the various pipelines.</para>
</sect3>
</sect2>
</sect1>
<sect1>
<title>Integrating LIMA</title>
<para>There is several ways to integrate LIMA in your application. In this early version of this user manual, we will suppose that you just need to get the following information about the tokens present in the analyzed text: lemma, morphosyntactic category, position and entity type in case of specific entities. We also suppose that you want to directly invoke LIMA from your C++ code and support multithreading the calls to the analyzer.</para>
<para>The code accompanying this user manual implements this. The dowork function initializes the analyzer, prepares the list of files to be analyzed, this list being protected by a mutex, and creates the specified number of threads, binding them to the analyze_thread function. This latter one repeatedly peeks a file to analyze, prepare the handler that will allow to access the analysis result (here a BoWText), call the analysis client and then dumps the output using the BoW API.</para>
</sect1>
</article>