🪲 Keyword translation fails for Catalan: "at random" -> "att random"

[rix0rrr]

Describe the bug

If we translate the keywords in a program from Catalan to English, but the English keywords at random are used, the translation output is att random.

I think this is because the Catalan translation of this keyword (a) is a prefix of the English translation of this keyword (at).

The following tests reproduces it (put it in test_translation_level_06.py):

    def test_specific_catalan_example(self):
        code = textwrap.dedent("""\
            ruleta1 = ruleta at random
            """)

        result = hedy_translation.translate_keywords(code, from_lang="ca", to_lang="en", level=self.level)

        self.assertEqual(result, 'ruleta1 = ruleta at random')

When I add a print statement to the the innards of translate_keywords:

                replaced_line = replace_token_in_line(line, rule, original, target)
                print(line, '|', replaced_line, '# replacing', rule, original, target)    # <-- debugging

I see the following:

ruleta1 = ruleta at random | ruleta1 = ruleta at random # replacing Rule(keyword='random', line=1, start=20, end=25, value='random') random random
ruleta1 = ruleta at random | ruleta1 = ruleta att random # replacing Rule(keyword='at', line=1, start=16, end=19, value=' at ') a at

• For the first line: we find the keyword random and replace it with random.
• For the second line, we see that we think we matched the keyword a and are replacing it with at, whereas we actually should have matched the keyword a.

This is not a pure language confusion bug, as the Catalan keyword for random is aleatori, so we definitely can distinguish whether we matched an English vs a Catalan keyword. This makes me the think the problem is that a is a prefix of at. Maybe it's a regex thing somewhere?

[rix0rrr]

Oh no, the problem is probably here:

def get_original_keyword(keyword_dict, keyword, line):
    for word in keyword_dict[keyword]:
        if word in line:
            return word

    # If we can't find the keyword, it means that it isn't part of the valid keywords for this language
    # so return original instead
    return keyword

It's not that a is a prefix of at, but it's a proper substring.

[rix0rrr]

Keyword replacing on a string basis after we've already parsed is error prone.

Solution: replace based on parse tree, but do substitutions in reverse order so that later string indexes don't shift after a substitution.