Centralize conformance tests in this repository

[damienmg]

Hi,

The Go, the Rust and the Java implementation have their own integration tests that test different feature. When writing the integration tests for the rust language I basically took the Java and the Go one and made some variation: the error messages were not the same, the feature set not exactly the same (when importing the Go one) and the test framework is not exactly the same (assert_eq, fails, ....)

It would be great to standardize those test files and have them in this repo so change to the language would be translated here easily.

[laurentlb]

We have this: https://github.com/bazelbuild/bazel/tree/master/src/test/skylark
We can make the script ignore the error messages.

[damienmg]

I used a different test runner for rust. I was not speaking about the test runner itself but the tests themselves. The format of tests in Bazel is the one I used in fine, just the error were different.

To have a nice conformance test suite, the format should probably be standardized and have go use the same.

Those tests should also live in this repository in my opinion so everyone could contribute to the same test suite.

[Quarz0]

Hi,

I’m going to work on ensuring starlark’s implementation are in sync as part of GSoC’19 under @laurentlb mentoring.
Regarding the common test suite, I'd like to discuss a few things.
First, If we are going to add a test runner then it could be similar to the one for Java for example.
It’s a python script that executes the starlark binary as a process and passes the location of the test files as arguments.
However, If we did the same for a common test runner for the 3 implementations, then we’d need to add the path of the 3 executables of the go, rust and java implementations. Which I’m not sure if it’s a nice idea.

An alternative I’ve thought of is to use docker or something similar to automatically fetch and build the latest version of the 3 implementations, installing all dependencies in the docker image and then running the test suite inside the docker. This wouldn't require having a local copy of any of the implementations' codes/binaries to run the test cases.

[laurentlb]

I think we can use external repositories to test the 3 implementations.

[Quarz0]

Concerning the java implementation, wouldn't this require the visibility of the "//src/main/java/com/google/devtools/starlark:Starlark" rule to be public as mentioned here, so that I can use it from another project?

[laurentlb]

Yes, we can do that.