Skip to content

Commit

Permalink
added REST test suites runner
Browse files Browse the repository at this point in the history
The REST layer can now be tested through tests that are shared between all the elasticsearch official clients.
The tests are based on REST specification that can be found on the elasticsearch-rest-api-spec project and consist of YAML files that describe the operations to be executed and the obtained results that need to be tested.

REST tests can be executed through the ElasticsearchRestTests class, which relies on the rest-spec git submodule that contains the rest spec and tests pulled from the elasticsearch-rest-spec-api project. The rest-spec submodule gets automatically initialized and updated through maven (generate-test-resources phase).

The REST runner and the needed classes are distributed within the test artifact.

The following are the options supported by the REST tests runner:

- tests.rest[true|false|host:port]: determines whether the REST tests need to be run and if so whether to rely on an external cluster (providing host and port) or fire a test cluster (default)
- tests.rest.suite: comma separated paths of the test suites to be run (by default loaded from /rest-spec/test classpath). it is possible to run only a subset of the tests providing a sub-folder or even a single yaml file (the default /rest-spec/test prefix is optional when files are loaded from classpath) e.g. -Dtests.rest.suite=index,get,create/10_with_id
- tests.rest.spec: REST spec path (default /rest-spec/api from classpath)
- tests.iters: runs multiple iterations
- tests.seed: seed to base the random behaviours on
- tests.appendseed[true|false]: enables adding the seed to each test section's description (default false)
- tests.cluster_seed: seed used to create the test cluster (if enabled)

Closes elastic#4469
  • Loading branch information
javanna committed Dec 17, 2013
1 parent 3fed65e commit d97a00d
Show file tree
Hide file tree
Showing 71 changed files with 6,869 additions and 55 deletions.
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "rest-spec"]
path = rest-spec
url = [email protected]:elasticsearch/elasticsearch-rest-api-spec.git
42 changes: 42 additions & 0 deletions TESTING.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,45 @@ even if tests are passing.
------------------------------
mvn test -Dtests.output=always
------------------------------

== Testing the REST layer

The available integration tests make use of the java API to communicate with
the elasticsearch nodes, using the internal binary transport (port 9300 by
default).
The REST layer is tested through specific tests that are shared between all
the elasticsearch official clients and can be found on the
https://github.com/elasticsearch/elasticsearch-rest-api-spec[elasticsearch-rest-api-spec project].
They consist of
https://github.com/elasticsearch/elasticsearch-rest-api-spec/tree/master/test[YAML files]
that describe the operations to be executed and the obtained results that
need to be tested.

`ElasticsearchRestTests` is the executable test class that runs all the
yaml suites available through a git submodule within the `rest-spec` folder.
The submodule gets automatically initialized through maven right before
running tests (generate-test-resources phase).
The REST tests cannot be run without the files pulled from the submodule,
thus if the `rest-spec` folder is empty on your working copy, it means
that it needs to be initialized with the following command:

------------------------------
git submodule update --init
------------------------------

The following are the options supported by the REST tests runner:

* `tests.rest[true|false|host:port]`: determines whether the REST tests need
to be run and if so whether to rely on an external cluster (providing host
and port) or fire a test cluster (default)
* `tests.rest.suite`: comma separated paths of the test suites to be run
(by default loaded from /rest-spec/test). It is possible to run only a subset
of the tests providing a sub-folder or even a single yaml file (the default
/rest-spec/test prefix is optional when files are loaded from classpath)
e.g. -Dtests.rest.suite=index,get,create/10_with_id
* `tests.rest.spec`: REST spec path (default /rest-spec/api)
* `tests.iters`: runs multiple iterations
* `tests.seed`: seed to base the random behaviours on
* `tests.appendseed[true|false]`: enables adding the seed to each test
section's description (default false)
* `tests.cluster_seed`: seed used to create the test cluster (if enabled)
75 changes: 69 additions & 6 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
<tests.shuffle>true</tests.shuffle>
<tests.output>onerror</tests.output>
<tests.client.ratio></tests.client.ratio>
<rest.pull.skip>false</rest.pull.skip>
<rest.init.skip>false</rest.init.skip>
<es.logger.level>INFO</es.logger.level>
</properties>

Expand Down Expand Up @@ -65,6 +67,12 @@
<version>${lucene.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.lucene</groupId>
Expand Down Expand Up @@ -313,6 +321,14 @@
<include>**/*.*</include>
</includes>
</testResource>
<testResource>
<directory>${basedir}/rest-spec</directory>
<targetPath>rest-spec</targetPath>
<includes>
<include>api/*.json</include>
<include>test/**/*.yaml</include>
</includes>
</testResource>
</testResources>

<plugins>
Expand Down Expand Up @@ -396,6 +412,7 @@
<systemProperties>
<!-- RandomizedTesting library system properties -->
<tests.jvm.argline>${tests.jvm.argline}</tests.jvm.argline>
<tests.appendseed>${tests.appendseed}</tests.appendseed>
<tests.iters>${tests.iters}</tests.iters>
<tests.maxfailures>${tests.maxfailures}</tests.maxfailures>
<tests.failfast>${tests.failfast}</tests.failfast>
Expand All @@ -412,6 +429,9 @@
<tests.integration>${tests.integration}</tests.integration>
<tests.cluster_seed>${tests.cluster_seed}</tests.cluster_seed>
<tests.client.ratio>${tests.client.ratio}</tests.client.ratio>
<tests.rest>${tests.rest}</tests.rest>
<tests.rest.suite>${tests.rest.suite}</tests.rest.suite>
<tests.rest.spec>${tests.rest.spec}</tests.rest.spec>
<es.node.local>${env.ES_TEST_LOCAL}</es.node.local>
<es.node.mode>${es.node.mode}</es.node.mode>
<es.logger.level>${es.logger.level}</es.logger.level>
Expand Down Expand Up @@ -1002,14 +1022,50 @@
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>java</executable>
<arguments>
<argument>-version</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>Init Rest Spec</id>
<phase>generate-test-resources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<skip>${rest.init.skip}</skip>
<executable>git</executable>
<arguments>
<argument>submodule</argument>
<argument>update</argument>
<argument>--init</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>Pull Rest Spec</id>
<phase>generate-test-resources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<skip>${rest.pull.skip}</skip>
<executable>git</executable>
<arguments>
<argument>submodule</argument>
<argument>foreach</argument>
<argument>git</argument>
<argument>pull</argument>
<argument>origin</argument>
<argument>master</argument>
</arguments>
</configuration>
</execution>
</executions>
<configuration>
<executable>java</executable>
<arguments>
<argument>-version</argument>
</arguments>
</configuration>

</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand All @@ -1026,7 +1082,14 @@
<include>org/elasticsearch/test/**/*</include>
<include>org/apache/lucene/util/AbstractRandomizedTest.class</include>
<include>org/apache/lucene/util/AbstractRandomizedTest$*.class</include> <!-- inner classes -->
<include>com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.class</include>
</includes>
<excludes>
<!-- we only distribute the REST tests runner, not the executable test -->
<exclude>org/elasticsearch/test/rest/ElasticsearchRestTests.class</exclude>
<!-- unit tests for yaml suite parser & rest spec parser need to be excluded -->
<exclude>org/elasticsearch/test/rest/test/**/*</exclude>
</excludes>
</configuration>
</execution>
</executions>
Expand Down
1 change: 1 addition & 0 deletions rest-spec
Submodule rest-spec added at 2f5f78
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
/*
* Licensed to ElasticSearch and Shay Banon under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. ElasticSearch licenses this
* file to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations under
* the License.
*/

package com.carrotsearch.randomizedtesting;

/**
* Exposes methods that allow to use a {@link RandomizedContext} without using a {@link RandomizedRunner}
* This was specifically needed by the REST tests since they run with a custom junit runner ({@link org.elasticsearch.test.rest.junit.RestTestSuiteRunner})
*/
public final class StandaloneRandomizedContext {

private StandaloneRandomizedContext() {

}

/**
* Creates a new {@link RandomizedContext} associated to the current thread
*/
public static void createRandomizedContext(Class<?> testClass, Randomness runnerRandomness) {
//the randomized runner is passed in as null, which is fine as long as we don't try to access it afterwards
RandomizedContext randomizedContext = RandomizedContext.create(Thread.currentThread().getThreadGroup(), testClass, null);
randomizedContext.push(runnerRandomness.clone(Thread.currentThread()));
}

/**
* Destroys the {@link RandomizedContext} associated to the current thread
*/
public static void disposeRandomizedContext() {
RandomizedContext.current().dispose();
}

public static void pushRandomness(Randomness randomness) {
RandomizedContext.current().push(randomness);
}

public static void popAndDestroy() {
RandomizedContext.current().popAndDestroy();
}

/**
* Returns the string formatted seed associated to the current thread's randomized context
*/
public static String getSeedAsString() {
return SeedUtils.formatSeed(RandomizedContext.current().getRandomness().getSeed());
}

/**
* Util method to extract the seed out of a {@link Randomness} instance
*/
public static long getSeed(Randomness randomness) {
return randomness.getSeed();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
*/
package org.elasticsearch.test;

import com.carrotsearch.randomizedtesting.SeedUtils;
import com.google.common.base.Joiner;
import org.apache.lucene.util.AbstractRandomizedTest;
import org.elasticsearch.ExceptionsHelper;
Expand Down Expand Up @@ -68,6 +67,8 @@
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;

import static org.elasticsearch.test.TestCluster.SHARED_CLUSTER_SEED;
import static org.elasticsearch.test.TestCluster.clusterName;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures;
import static org.hamcrest.Matchers.emptyIterable;
Expand Down Expand Up @@ -123,7 +124,7 @@
* This class supports the following system properties (passed with -Dkey=value to the application)
* <ul>
* <li>-D{@value #TESTS_CLIENT_RATIO} - a double value in the interval [0..1] which defines the ration between node and transport clients used</li>
* <li>-D{@value #TESTS_CLUSTER_SEED} - a random seed used to initialize the clusters random context.
* <li>-D{@value TestCluster#TESTS_CLUSTER_SEED} - a random seed used to initialize the clusters random context.
* <li>-D{@value #INDEX_SEED_SETTING} - a random seed used to initialize the index random context.
* </ul>
* </p>
Expand All @@ -132,24 +133,13 @@
@AbstractRandomizedTest.IntegrationTests
public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase {


/**
* The random seed for the shared test cluster used in the current JVM.
*/
public static final long SHARED_CLUSTER_SEED = clusterSeed();

private static final TestCluster GLOBAL_CLUSTER = new TestCluster(SHARED_CLUSTER_SEED, TestCluster.clusterName("shared", ElasticsearchTestCase.CHILD_VM_ID, SHARED_CLUSTER_SEED));
private static final TestCluster GLOBAL_CLUSTER = new TestCluster(SHARED_CLUSTER_SEED, clusterName("shared", ElasticsearchTestCase.CHILD_VM_ID, SHARED_CLUSTER_SEED));

/**
* Key used to set the transport client ratio via the commandline -D{@value #TESTS_CLIENT_RATIO}
*/
public static final String TESTS_CLIENT_RATIO = "tests.client.ratio";

/**
* Key used to set the shared cluster random seed via the commandline -D{@value #TESTS_CLUSTER_SEED}
*/
public static final String TESTS_CLUSTER_SEED = "tests.cluster_seed";

/**
* Key used to retrieve the index random seed from the index settings on a running node.
* The value of this seed can be used to initialize a random context for a specific index.
Expand Down Expand Up @@ -830,7 +820,7 @@ public Settings settings(int nodeOrdinal) {
};
}

return new TestCluster(currentClusterSeed, numNodes, TestCluster.clusterName(scope.name(), ElasticsearchTestCase.CHILD_VM_ID, currentClusterSeed), nodeSettingsSource);
return new TestCluster(currentClusterSeed, numNodes, clusterName(scope.name(), ElasticsearchTestCase.CHILD_VM_ID, currentClusterSeed), nodeSettingsSource);
}

/**
Expand Down Expand Up @@ -859,14 +849,6 @@ public Settings settings(int nodeOrdinal) {
double transportClientRatio() default -1;
}

private static long clusterSeed() {
String property = System.getProperty(TESTS_CLUSTER_SEED);
if (property == null || property.isEmpty()) {
return System.nanoTime();
}
return SeedUtils.parseSeed(property);
}

/**
* Returns the client ratio configured via
*/
Expand Down
Loading

0 comments on commit d97a00d

Please sign in to comment.