Skip to content

Latest commit

 

History

History
256 lines (162 loc) · 7 KB

DEVELOP.rst

File metadata and controls

256 lines (162 loc) · 7 KB
Raw

DEVELOPMENT

Prerequisites

Crate Data is developed in Java 7, so a working Java 7 virtual machine installation is required. We recommend using Oracle's Java on OSX and OpenJDK on Linux Systems.

Submodules

Init/Update

In order to build the project the first time it is necessary to initialize and update all submodules. Which is done via the following command from within the crate repository:

$ git submodule update --init

Build

This project uses Gradle as build tool. It can be invoked by executing ./gradlew. The first time this command is executed it is bootstrapped automatically, therefore there is no need to install gradle on the system.

In order to build the documentation and run integration tests a Python executable called python2.7 needs to be in the $PATH. The documentation is generated by Sphinx.

Before creating a new distribution, a new version and tag should be created:

  • Add a new version to the io.crate.Version class.
  • Point the CURRENT version in that class to the newly added version.
  • Add a note for the new version at the CHANGES.txt file.
  • Commit e.g. using message 'prepare release x.x.x'.
  • Push to origin
  • Create a tag using the create_tag.sh script (run ./devtools/create_tag.sh).

Now everything is ready for building a new distribution, either manually or let jenkins do the job as usual :-)

Building a tarball and a zip is done by gradle with the command:

$ ./gradlew dist

Building a release tarball is done via the release task. This task actually only runs the distTar task but additionally checks that the output of git describe --tag matches the current version of Crate:

$ ./gradlew release

The resulting tarball and zip will reside in the folder ./app/build/distributions.

It is also possible to install the distribution locally by running:

$ ./gradlew installDist

The root of the installation can then be found under ./app/build/install/crate. Start the server with:

$ ./app/build/install/crate/bin/crate

IntelliJ

To use this project in intellij do not use the built-in JetGradle Plugin. Instead use gradle directly by running:

./gradlew idea

This will create the project files.

Run/Debug Configurations

It is also possible to run Crate Data nodes directly from within IntelliJ. But before that can be done a bit of preparation is necessary.

First create the folders for the configuration and data:

for i in {1..2}; do mkdir -p sandbox/crate_$i/{config,data,plugins}; done

In order for the admin interface to work an additional folder and symbolic links are necessary:

for i in {1..2}; do mkdir -p sandbox/crate_$i/plugins/crate-admin/; done

ln -s $PWD/admin-ui/crate-admin/dist/ sandbox/crate_1/plugins/crate-admin/_site
ln -s $PWD/admin-ui/crate-admin/dist/ sandbox/crate_2/plugins/crate-admin/_site

Then create the configuration files for both nodes:

touch sandbox/crate_1/config/crate.yml
touch sandbox/crate_2/config/crate.yml

And add the following settings:

node.name: local1

http.port: 19201
transport.tcp.port: 19301
network.host: localhost

multicast.enabled: false
discovery.zen.ping.unicast.hosts:
  - 127.0.0.1:19301
  - 127.0.0.1:19302

Note

In the second files the port number and node name has to be changed. 19201 to 19202 and 19301 to 19302.

In addition to the crate.yml file it is also recommended to create a logging configuration file for both nodes. To do so create the files sandbox/crate_1/config/logging.yml and sandbox/crate_2/config/logging.yml.

A minimal example for the logging configuration looks like this:

rootLogger: INFO, console
logger:
  # log action execution errors for easier debugging
  action: DEBUG
  crate.elasticsearch.blob: TRACE

appender:
  console:
    type: console
    layout:
      type: consolePattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

After that the Run/Debug Configurations can be added within IntelliJ. Go to the Run/Debug Configurations window and add a new Application configuration (one for each node) with the following settings:

Main class io.crate.bootstrap.CrateF
VM Options -Des.path.home=/full/path/to/sandbox/crate_1/
Use classpath of module: app

Debugging

To run the app locally for debugging use:

./gradlew runDebug

It will then listen for a remote debugger on port 5005.

Testing

Unit tests can be run like this:

$ ./gradlew test

It is also possible to run tests in parallel in order to speed up execution time by setting the project property testForks like this:

$ ./gradlew test -PtestForks=5

Test Coverage

Create test coverage reports with jacoco in every module's build/reports/jacoco directory:

$ ./gradlew jacocoTestReport

Findbugs

Running FindBugs against our code base:

$ ./gradlew findBugsMain

the findbugs check will also be executed when running:

$ ./gradlew check

Benchmark

A Benchmark for our SQL Interface can be run by calling:

$ ./gradlew bench

It will output some results to stdout (read between the lines) and finally you will receive information where more detailed benchmark-results got stored.

Writing Documentation

The documentation is maintained under the docs directory and written in ReStructuredText and processed with Sphinx.

Note that the documentation is no longer distributed with this project.

Normally the documentation is built by Read the Docs. However if you work on the documentation you can run sphinx directly, which can be done by just running bin/sphinx in the docs directory. The output can then be found in the out/html and out/text directories.

Before you can run bin/sphinx you need to setup a development environment by running bootstrap.py inside the docs directory:

python bootstrap.py

And afterwards run buildout:

./bin/buildout -N

The documentation is tested when running the itest task. Also the itest task is run as a dependency of the dist task.