verinice.veo is a cloud-native application for managing information security and data protection.
- Install Java 11.
- If you are using an IDE (Eclipse, IntelliJ, VSCode,...) you need to run it with the lombok compile time preprocessor. Simply download lombok.jar (i.e. from Maven Central) and run the helper: "java -jar lombok.jar". This will add the required parameter to your eclipse.ini or other configuration. If you want to do it by hand, add the following parameter at the end of your eclipse.ini:
-javaagent:/home/user/eclipse/lombok.jar
git clone https://github.com/SerNet/verinice-veo.git
cd verinice-veo
export JAVA_HOME=/path/to/jdk-11
./gradlew build [-x test]
If you want to build a Docker image, you can then run
docker build --build-arg VEO_VERSION='1.2.3-SNAPSHOT'
You can configure the application by changing properties either as system environment variables, Java system property or by setting the properties in an application.properties file. Which means the following are equivalent:
-
System Environment:
export spring_datasource_url=jdbc:postgresql://127.0.0.1:5432/veo java -jar ./veo-rest/build/libs/veo-rest-VERSION.jar
-
Java system property value:
java -Dspring.datasource.url=jdbc:postgresql://127.0.0.1:5432/veo -jar ./veo-rest/build/libs/veo-rest-VERSION.jar
-
Adding the line to application.properties:
spring.datasource.url=jdbc:postgresql://127.0.0.1:5432/veo
The system environment variable approach has the advantage, that it works when running the built jar.
Note The system environment variable name is the same as the name of the property except dots are replaced by underscore, i.e.
The following sections describe commonly changed properties.
The database connection can be modified by setting the following properties as needed:
export spring_datasource_url=jdbc:postgresql://127.0.0.1:5432/veo
export spring_datasource_username=veo
export spring_datasource_password=veo
Spring profiles can be used to tailor the appication for different runtime and deployment environments. Profiles can be activated via the command line on launch:
./gradlew -P springProfiles=stats veo-rest:bootRun
or
java -Dspring.profiles.active=stats,local -jar veo-rest/build/libs/veo-rest-VERSION.jar
Profiles can also be set using environment variables:
SPRING_PROFILES_ACTIVE: stats,local
For more information on how to use profiles see the reference documentation
The following profiles are supported by the application:
This profile is active when unit or integration tests are running. It can be used to deactivate certain beans and configurations that should not be running during tests because they are replaced by mocks/stubs or because they are not required.
This profile enables a data source proxy that logs all generated SQL. It can be used for debugging. It is also used for tests that analyze the generated SQL queries for possible performance issues.
This profile should not be enabled for clustered (autoscaled) instances. It activates background tasks that only need to be run once in each environment. It will enable the following background tasks:
- an event-dispatcher that forwards the generated application events to an external AMQP
message broker. The dispatcher needs to be configured using the corresponding settings found in
application.yaml
.
(Or any other self-defined profile name). Loads additional configuration files that can be used to set up
a dev environment, increase log levels and such. This profile will look for and load an additional property
file called application-local.[yaml|properties]
This profile will be active while the integration test for the REST API are running. These tests use Spring's TestRestTemplate
which needs to be configured with URLs and credentials for the production services that are to be used
during the integration test, such as Keycloak, PostgreSQL or RabbitMQ.
If you want to run the REST API integration tests, simply run:
./gradlew veo-rest:restTest
You can use application-resttest.yaml
or your own application-local.yaml
to set the required
parameters.
If you need to pass proxy parameters (i.e. if the backend server has to use a proxy to reach the Keycloak service to verify access tokens) you can pass them on the command line as project properties:
./gradlew veo-rest:restTest \
-Phttp.proxyHost=cache.myhost.com -Phttp.proxyPort=3128 \
-Phttps.proxyHost=cache.myhost.com -Phttps.proxyPort=3128
The REST API tests will spin up a local server. All tests execute purely as HTTP requests against this server. No address space is shared between REST client and server. It is not possible to access repositories or any other server-side objects from the tests directly. All communication has to occur using HTTP calls like a regular client would use the service.
This profile is active in the dev environment only.
You can change the logging level per package by setting the property logging.level.<qualified package name>
, e.g.
export logging_level_org_veo_rest_security=DEBUG
export logging_level_org_veo=TRACE
Note that you have to replace the dots in the package name with underscores.
This even works for specific test runs:
env 'logging_level_org_veo=DEBUG' ./gradlew veo-rest:test --tests 'AuthenticationMockMvcITSpec'
Valid logging levels in ascending order are
ALL
TRACE
DEBUG
INFO
WARN
ERROR
FATAL
OFF
You can log to files by setting the property logging.file
, e.g.
export logging_file=/var/log/veo.log
Of course if you want to see the logs and pipe them to a file you can always run
./gradlew sR | tee /var/log/veo.log
This also works for specific test runs:
env 'logging_file=/tmp/test.log' 'logging_level_org_veo=DEBUG' ./gradlew veo-rest:test --tests 'AuthenticationMockMvcITSpec'
To configure more complex setups you can reference a logback.xml configuration file, by setting
the property logging.config
export logging_config=/etc/veo/logback.xml
When you do so, you may like to enable configuration auto scanning by setting
the scanPeriod
attribute of the configuration
tag.
e.g. you could change the logging level in the following logback.xml from error
to debug
while the application is running:
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="30 seconds" >
<include
resource="org/springframework/boot/logging/logback/base.xml" />
<logger name="org.veo" level="error" />
</configuration>
- Install Java 11.
- Install PostgreSQL
- Create an empty database v2020
Set your database properties in file veo-rest/src/main/resources/application.properties and rebuild the application.
./gradlew veo-rest:bootRun
or
./gradlew veo-rest:jar
java -jar veo-rest/build/libs/veo-rest-VERSION.jar
./gradlew jmGui
./gradlew jmRun
Swagger-UI can be accessed online at api.verinice.com/veo or after the REST service has been started locally at:
http://localhost:8070/swagger-ui.html
CAUTION: this URL will redirect you to http://localhost:8070/swagger-ui/index.html?configUrl=/v3/api-docs/swagger-config
. Do NOT use this URL directly or the redirects for OpenID Connect authentication will not work.
You can use postman to access and test the REST API. To set up the necessary OAuth 2.0 authentication, got to the tab 'Authorization -> Configure New Token'.
Enter the following details:
Use the following values:
- Token Name: Can be freely set
- Callback URL:
https://api.verinice.com
- Authorize using browser: disabled
- Auth URL:
https://auth.verinice.com/auth/realms/verinice-veo/protocol/openid-connect/auth
- Access Token URL:
https://auth.verinice.com/auth/realms/verinice-veo/protocol/openid-connect/token
- Client ID:
veo-prod
- Client Secret: empty
- Scope:
veo-user
- State: empty
- Client Authentication:
Send client credentials in body
Click on 'Get New Access Token'. Enter your credentials. Then select the acquired token in the dropdown box 'Available Tokens'.
You can now send your HTTP request. The access token will time out (usually after 1-5 minutes) nad has to be requested again for another request.
This module contains the implementation of the REST services of the REST API.
veo-rest uses OIDC to authorize users. To get an access token and access the API you can use any library supporting oAuth2. To test on the command line you can also use the script
misc/scripts/authenticate
to log in and get an access token and
misc/scripts/authorize
to use the access token and send a HTTP request to the API. See misc/scripts/README or
call ./misc/scripts/authenticate -h
for more details.
(See the section on using postman for another alternative.)
VEO records changes to entities and publishes them to an external message broker (see Profile 'background-tasks').
Veo uses a local event store to save model changes and corresponding events in the same transaction boundary. This ensures transactional consistency between events and model changes without the use of the XA protocol (two-phase commits). Please note: in rare cases an event may be published multiple times. This can happen if the message broker crashes after sending the event but before confirming the transmission to the publisher. Therefore:
- consumers of veo-events MUST perform message deduplication for all received events. Each event carries a unique ID that may be used for this purpose.
- Due to how those events are handled internally, gaps may occur in the IDs, so consumers MUST NOT rely on event IDs being consecutive.
Code styles are enforced using the gradle plugins pmd and spotless.
To verify your code run
./gradlew pmdMain
./gradlew spotlessCheck
or just
./gradlew check
You can find formatting settings for IDEs in misc/eclipse.
Spotless can fix code formatting violations by running
./gradlew spotlessApply
For some reason the ktlint
plugin cannot apply fixes. To automatically format Kotlin files
you can install ktlint
and run
ktlint -F
In the project root.
Sample git hooks are provided in misc/git, e.g. pre-commit, which checks the code style on each commit. To install, copy the file to .git/hooks/pre-commit, etc. Make sure the scripts are executable.
Veo uses flyway for DB migrations. It runs groovy migration scripts from org.veo.persistence migrations when starting the service / spring test environment before JPA is initialized.
- Modify DB model code (persistence-layer entity classes).
./gradlew bootRun
. The service might complain that the DB doesn't match the model but will silently generate the update DDL inveo-rest/schema.local.sql
.- Copy SQL from
veo-rest/schema.local.sql
. - Create a new migration script (e.g.
veo-persistence/src/main/groovy/org/veo/persistence/migrations/V3__add_fancy_new_columns.groovy
) and let it execute the SQL you copied (see existing migration scripts). - Append a semicolon to every SQL command
- Add some DML to your migration if necessary.