cfg: other languages compatibility

[nlpirate]

I am interested in some verticalization of lambeq (and consequently discocat) to languages other than English, particularly Italian. As far as I read from the documentation from a linguistic point of view, at the base of the framework, there are cfg grammars. I know this theoretical formalism very well. How is it possible to visualize and extract the structures and formalisms of these grammars from the library so that they can be extended/modified?

[chirico85]

Good point.
In my case German would be interesting.

[Thommy257]

Hi @nlpirate and @chirico85,

Our parser works based on the Combinatory categorial grammar (CCG) formalism, which is a bit different to context-free grammar (CFG). While CFGs are generative, i.e produce valid sentences, CCG models are used to infer grammar trees from well-formed sentences. Hence, CCGs are parsable, which we leverage using our BobcatParser.

Bobcat works in two stages: First, we apply a BERT model to determine the most likely CCG types per word. The outcome of that step is a weighted list of the k most likely types. After that, we apply a deterministic chart parser that aims to find the most probable CCG reduction tree from the possible word types. You can read more about our parser here.

As you can see, we use a statistical model for the first step, which needs to be trained on data. The data we use to train Bobcat is the CCGbank, which is a translation from the Penn treebank. Hence, to support multiple languages, we need to have such CCG banks for each language, which might require a lot of work.

However, if you don't require a fully-comprehensive CCG parser, you can always create your own (deterministic) parser based on our abstract CCGParser class:

lambeq/lambeq/text2diagram/ccg_parser.py

Line 34 in 70a1fe8

class CCGParser(Reader):

I hope this helps!

[Thommy257]

Also, DisCoPy supports CFG grammars (https://docs.discopy.org/en/0.5/discopy/grammar.cfg.html?highlight=cfg#module-discopy.grammar.cfg), therefore CFGs are also supported by lambeq. Furthermore, CCGs can express CFGs, i.e. also be used to generate sentences.

[dimkart]

I think there was an effort for creating an Italian CCGBank in the past (Turin univ.?) , not sure however what happened with that project.

[nlpirate]

yes, indeed it exists and is available on the site (tut-ccg), but the annotation is different than the one used in lambeq

[dimkart]

Not sure if there is anything more to say here, since as @Thommy257 explained above, without an annotated corpus like CCGBank you can't train a statistical parser. However we are very much interested in adding to lambeq support for languages other than English (and we welcome any community work towards this goal), so this issue will be converted into a Discussion to stay alive.