[CHORE, TESTING] Test code can run form the front-end in all languages

[TiBiBa]

Verify that the code can be ran in all languages:

This PR fixes #3760

[Felienne]

Hi @TiBiBa!

I think we said in the meeting that we'd close this and do the testing of the run button in a "manual" way, right (by repeatedly 'clicking' a language and then the run button)

[martinj2001]

@TiBiBa I think I fixed your problems with this tests, let me know if there is more to fix!

[jpelay]

Hello everyone! I'm turning this into a draft while we keep working on it!

[ghost]

👇 Click on the image for a new way to code review

Legend

[jpelay]

Hello, everyone! I made a few changes to the test so it can run in every language!

[Felienne]

Maybe @TiBiBa can take a look? 🙏

[Felienne]

Hi @jpelay

I tried to test the tests as follows, but it seems to still work?

1. Break a print keyword, f.e. in French:

_PRINT: ("affiche" | "prt") _SPACE?

2. Be sure to not rerun on-deploy.sh or it will restore the grammar!

3. Observe the app is now broken:

4. But the tests still passes:

[jpelay]

Hi @Felienne! Thank you for pointing it out!

It does seem like the page is refreshing, but the language is not changing? This one was supposed to show content from Bulgarian (notice that switch-language-bg was selected, but instead is still showing content from Arabic. It does appear that after changing the language once, that one keeps locked!

[Felienne]

Hi @jpelay!!

I might have a hunch as to why this does not test all languages!!! (@rix0rrr gave me the idea)

The current language is not present in the dropdown, so if you start with the first one (Arabic) will be selected:

Then we pick the second one, Bulgarian:

Now... the next one is Bengali:

This might mean we are skipping languages (I am not sure) So we first need to populate a stable list of languages and iterate over these.

[jpelay]

It might be, but I'm actually iterating trhough the list of languages, and clicking each one individually. Each language have their own id, so you can select then individually!

[Felienne]

It might be, but I'm actually iterating trhough the list of languages, and clicking each one individually. Each language have their own id, so you can select then individually!

Ah ok.... then it is probably something else!

[jpelay]

I'm still kinda blocked here, but found out something. This is how the logs look after printing g.lang in each setup_language. Here the first language, ar, have already been selected and processed and we're onto the next language, bg, the section with ### is in change_language so the language is being changed, and the section with *** is in each setup_language:

####################################################################################################
bg
####################################################################################################
[2023-03-14 11:45:05,153] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/change_language', 'method': 'POST', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:05.152911Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'http_code': 200, 'end_time': '2023-03-14T15:45:05.153467Z', 'user_ms': 0, 'sys_ms': 0, 'max_rss': 66756, 'inc_max_rss': 0, 'duration_ms': 0}
[2023-03-14 11:45:05,153] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:05] "POST /change_language HTTP/1.1" 200 -
****************************************************************************************************
ar
****************************************************************************************************
[2023-03-14 11:45:05,194] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/hedy', 'method': 'GET', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:05.175177Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'load_yaml_pickled_ms': 0, 'load_yaml_pickled_cnt': 5, 'render_template_ms': 6, 'render_template_cnt': 1, 'http_code': 200, 'end_time': '2023-03-14T15:45:05.194598Z', 'user_ms': 19, 'sys_ms': 0, 'max_rss': 66864, 'inc_max_rss': 108, 'duration_ms': 19}
[2023-03-14 11:45:05,195] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:05] "GET /hedy HTTP/1.1" 200 -
****************************************************************************************************
bg
****************************************************************************************************
[2023-03-14 11:45:05,819] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/vendor/skulpt.min.js.map', 'method': 'GET', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:05.817109Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'render_template_ms': 0, 'render_template_cnt': 1, 'http_code': 404, 'end_time': '2023-03-14T15:45:05.819514Z', 'user_ms': 2, 'sys_ms': 0, 'max_rss': 66896, 'inc_max_rss': 0, 'duration_ms': 2}
[2023-03-14 11:45:05,819] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:05] "GET /vendor/skulpt.min.js.map HTTP/1.1" 404 -
****************************************************************************************************
bg
****************************************************************************************************
[2023-03-14 11:45:05,872] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/js/appbundle.js.map', 'method': 'GET', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:05.871905Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'http_code': 200, 'end_time': '2023-03-14T15:45:05.872597Z', 'user_ms': 0, 'sys_ms': 0, 'max_rss': 66896, 'inc_max_rss': 0, 'duration_ms': 0}
[2023-03-14 11:45:05,872] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:05] "GET /js/appbundle.js.map HTTP/1.1" 200 -
****************************************************************************************************
bg
****************************************************************************************************
[2023-03-14 11:45:07,231] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/parse', 'method': 'POST', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:06.103452Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'level': 1, 'lang': 'ar', 'session_id': '72809ae8a04d4d18a8489bbcccfb6365', 'username': None, 'db_update:program-stats': 1, 'db_update_ms': 2, 'db_update_cnt': 1, 'transpile_ms': 508, 'transpile_cnt': 1, 'server_error': None, 'http_code': 200, 'end_time': '2023-03-14T15:45:07.231256Z', 'user_ms': 1126, 'sys_ms': 0, 'max_rss': 67408, 'inc_max_rss': 512, 'duration_ms': 1127}
[2023-03-14 11:45:07,231] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:07] "POST /parse HTTP/1.1" 200 -
****************************************************************************************************
bg
****************************************************************************************************
[2023-03-14 11:45:07,306] app.py teardown_request_finish_logging 320 <DEBUG>: {'path': '/vendor/skulpt-stdlib-extensions.js', 'method': 'GET', 'remote_ip': '127.0.0.1', 'user_agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.0.2 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36', 'start_time': '2023-03-14T15:45:07.305516Z', 'pid': 6373, 'loadavg': 0.73, 'fault': 0, 'http_code': 304, 'end_time': '2023-03-14T15:45:07.305984Z', 'user_ms': 0, 'sys_ms': 0, 'max_rss': 67408, 'inc_max_rss': 0, 'duration_ms': 0}
[2023-03-14 11:45:07,306] _internal.py _log 113 <INFO>: 127.0.0.1 - - [14/Mar/2023 11:45:07] "GET /vendor/skulpt-stdlib-extensions.js HTTP/1.1" 304 -
****************************************************************************************************

Notice how the only path that retains the language ar is in the path /hedy, and that is try for every language!

[Felienne]

Yes so that does look suspiciously like it is trying the first language (ar) and then the first one again (bg) and then again the first one (ar)!

[jpelay]

Somehow is working now, after changing... nothing

[rix0rrr]

I changed the premise a little, maybe this is a cheat technique that might help!

There are three things I did:

• First, we need to get a list of languages "from somewhere". For now, I hardcoded a list into the source. Potentially you could generate a JSON file with list of languages at build time or something, and have cy.readFile() read that list.
• Then, I generate a test for every language in that list. By doing a for loop, and calling it('...', () => { .. }) inside the for loop, we get a new test for every language in the list.
• Finally, to make it easy to switch to a particular language without using the UI (cheating, I know), I added the ability to force the current language to a particular language by passing the query argument ?language=. So if you add ?language=fr to a URL, it will immediately cause a switch to French.

Does this help as a start? Up to you to finish it by finding a way to get the list of languages in Cypress (potentially copy/paste might even be good enough for a bit...?) and making the test check what you want it to.

---

If you want to do multiple tests per language, it might even be nicer to put the for loop around the describe. Do something like:

const LANGUAGES_TO_TEST = [ /* ... */ ];

for (const language of LANGUAGES_TO_TEST) {

  describe(`in the language ${language}`, () => {
    it('is possible to run a program', () => { ... });

    it('is possible to do something else', () => { ... });
  });
}

---

For some reason, this for loop that creates closures closing over the loop variable worked "as you'd expect" the first time for me (in fact, I didn't even think twice about it), but it might be a little dangerous nevertheless. Traditionally, there can be problems if you create a closure that refers to a loop variable. This Stack Overflow post might be good to have a read through so that you can recognize the problem quickly if you ever run into it.

An easy fix if we are uneasy might be:

for (const x of LANGUAGES_TO_TEST) {
  const language = x; // Make a fresh variable for every loop iteration

  // ...
}

Although it just might be that declaring the loop variable const is 100% guaranteed to be good enough.

[reedspool]

@rix0rrr below is not a response about your for loop strategy. I think you just nailed it your first time and no more changes are necessary but good to keep everyone aware of that sticky JS confusion!

I followed my (and Felienne's) suspicions about the for loop during the collaborator meeting. I just did a quick search for loop cypress testing and happened to find this SO answer with a comment that confirms those suspicions.

Tl;dr: this problem is exactly what cy.each solves. See that answer and that those docs and if you still have questions about how to proceed I'd be happy to look further!

This is of course if we are not satisfied with @rix0rrr 's solid approach! That seems like a very reasonable tactic!

[rix0rrr]

@reedspool that SO post seems to be about using loops inside the test, vs what this code is doing which is creating multiple tests at module load time.

But it might be you are referring back to a discussion I wasn’t part of?

[jpelay]

Hi! @rix0rrr and @reedspool!

@reedspool I quickly tried the SO approach about wrapping the array and running each but it still didn't work.
@rix0rrr I tried yours and it ineed is a good start! I will work on this further.

Thanks to both of you for the help!

[reedspool]

Glad you've got a good path forward @jpelay!

@rix0rrr Yes sorry about the confusion, I was continuing on thoughts we shared at the last Collaborators meeting.

[jpelay]

Hi all!! @Felienne I added the languages to test just hard coding there in the test. I will improve this once we tackle the test for the pygame features. But it now works! I edited keywords-es.lark and it fails.

Thanks @rix0rrr and @reedspool again for the help offered!

[Felienne]

OMG what a tour de force this was!! So happy it is working now. I made an issue for the hard-coded list of languages, just so we don't forget: #4168

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request is now going on the merge train (choo choo! Do not click update from main anymore, and be sure to allow changes to be pushed to your fork).