Developer Guide

• 1: Setting up releases to Sonatype & Maven Central
• 2: Building the documentation
• 3: Components
• 4: Release Process
• 5: Running in Docker
• 6: Running Stroom in an IDE

1 - Setting up releases to Sonatype & Maven Central

Create a Sonatype account

You need to create an account on Sonatype and you will need to raise a jira ticket on Sonatype’s jira to get approved on the uk.gov.gchq group. This will require an existing user approved for the group to approve you on the ticket.

Setting up a GPG key

You can use the following commands for setting up a GPG2 key for signing.

# Generate the GPG2 key
gpg2 --gen-key

# To list all keys
gpg2 --list-keys

# To get the key ID
gpg2  --list-secret-keys | grep "\[SC\]" | tr -s ' ' | cut -d' ' -f2 | cut -d'/' -f2

# To send the public keys to a key server
gpg2 --keyserver hkp://pool.sks-keyservers.net --send-keys <key id>
gpg2 --keyserver hkp://keyserver.ubuntu.com --send-keys <key id>
gpg2 --keyserver hkp://pgp.mit.edu --send-keys <key id>

# To display the secret key in base64 form, for use in GH actions
key="$(gpg2 --armor --export-secret-keys <key id> | base64 -w0)"; echo -e "-------\n$key\n-------"; key=""

Setting up the gradle build

The signing and release to Sonatype is done by various gradle plugins.

id "io.github.gradle-nexus.publish-plugin" version "1.0.0"
id "signing"
id "maven-publish"

See the root and event-logging-api gradle build files (in the event-logging repo) for an example of how to set up gradle.

The credentials can be passed to the gradle build using special gradle env vars Project Properties (external). The credentials required are:

• ORG_GRADLE_PROJECT_SIGNINGKEY - The key as produced by the gpg2 --armor command.
• ORG_GRADLE_PROJECT_SIGNINGPASSWORD - The password for the GPG key.
• ORG_GRADLE_PROJECT_SONATYPEUSERNAME - The account username on Sonatype.
• ORG_GRADLE_PROJECT_SONATYPEPASSWORD - The account password on Sonatype.

Setting up Github Actions

You will need to provide Github with the four secrets listed above by setting them as repository secrets at https://github.com/gchq//settings/secrets/actions. For each one create a secret with the ORG_GRADLE_... bit as the name.

So that the action can create the Github release you will also need to set up an SSH key pair and provide it with the public and private key. To generate the key pair do:

ssh-keygen -t rsa -b 4096 -f <repo>_deploy_key

The key pair will be created in ~/.ssh/.

Create a repo deploy key with the public key, named ‘Actions Deploy Key’ and with write access at https://github.com///settings/keys/new. Create a repo secret with the private key, named ‘SSH_DEPLOY_KEY’ at https://github.com///settings/secrets/actions/new.

2 - Building the documentation

Stroom’s documentation is built using GitBook (external link).

Prerequisites

NPM

You need NPM to install the GitBook command line toolchain. To get NPM install node (external link).

GitBook command line tools

npm install -g gitbook-cli

Build the book

GitBook uses plugins, e.g. anchorjs allows us to create links to headings within a file. You need to install these plugins first. The below commands should be run in the project root.

gitbook install

You can build the documentation like this:

gitbook build

Or you can run the GitBook server which will watch your files as you work and server them on localhost:4000.

gitbook serve

Troubleshooting

I get an error when trying to run gitbook serve

If you see Errpr: watch /path/to/stroom ENOSPC then run the following: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Links don’t work when I load _book/index.html

It won’t, because CORS is not and cannot be enabled when viewing local files. You need to run gitbook serve or if you really don’t want to do that try cd _book && python -m SimpleHTTPServer.

3 - Components

Some examples of components in Stroom include

• stroom-activity - Component for recording a users actions against a current activity
• stroom-dictionary - Component for storing lists of words.
• stroom-statistics - Component for recording statistical data, e.g. amount of data received in X minutes.

In the project structure a component appears as a first level subdirectory of the root project folder. Components have further subdirectories (modules) that make up the various parts of the component, e.g.

• stroom - Root project
  • stroom-activity - The component
    • stroom-activity-api - API module for stroom-activity
    • stroom-activity-impl - Implementation of the API and other module implementation code
    • stroom-activity-impl-db - Database persistence implementation used by impl
    • stroom-activity-impl-db-jooq - JOOQ generated classes used by stroom-activity-impl-db
    • stroom-activity-mock - Mock implementation for the stroom-activity API

Dependencies between a modules components

The diagram below shows the dependencies between the different modules that make up a component as well as the internal dependencies within the impl module. The actual implementations used at runtime are determined by Guice bindings in whichever Guice modules are loaded by the application. Tests can bind mock implementations of a components API just by using the Guice module within the mock module.

---

Dependencies between components

Typically a component will need to call out to other components to apply security constraints and to log user activity. These typical relationships are shown in the diagram below.

---

Component API, e.g. modules ending in -api

API layer

All communication between components in stroom must be made via a component’s API. The API provides the minimum surface area for communication between components and decouples dependencies between components to just the API code. For component testing purposes mock implementations of these APIs can be used to limit testing to just a single component.

Component API and service implementation, e.g. modules ending in -impl

Client interaction - REST services and GWT Action Handlers

The uppermost layer of the server side code services requests from the client. The client may make restful calls as is the case for the new UI or will use Actions that are handles with ActionHandlers as is the case for the legacy GWT UI.

Since this layer deals with all client interaction it should be responsible for creating audit logs for all user activity, e.g. accessing documents, searching etc. No audit logging should need to be performed at a lower level within the application as deeper levels have less knowledge of user intent since they may just be playing a part in the wider request.

The client interaction layer adds no logic and asks the underlying service layer to service the encapsulated request away from the REST endpoint wrapping code or GWT action handler code. This allows multiple types of endpoint to use the same underlying service layer. If a request requires the use of multiple services to form a response, this must be handled within the service layer by the primary service which will be responsible for any such orchestration.

Service layer

The service layer applies permission constraints to any requests being made so that only calls from identified and permitted users are allowed to proceed. The service layer performs all orchestration and business logic, and is responsible for all mutations of objects that will be persisted by the underlying persistence layer such as stamping objects to be updated with the current user and update time.

The service layer provides implementations for any API that the component may have.

The service layer provides the DAO (Data Access Object) API for the persistence layer to implement but maintains no knowledge of underlying persistence implementation, e.g. database queries.

Persistence implementation, e.g. modules ending in -impl-db

Persistence layer - DAOs

The persistence layer is an implementation of one or more DAOs specified in the service layer. The persistence layer provides no logic, it just stores and retrieves objects in a database or other persistence technology. If serialisation/de-serialisation is required in order to persist the object then that should also be performed by this layer so that no code above this layer has to care about this implementation detail.

The persistence layer does not apply security or permissions checking so should not need to reference the security API.

4 - Release Process

Performing a named release

This is a general guide for performing a release of a named version of one of the stroom family of products.

1. Ensure the CHANGELOG has been updated for the new x.y.z version, i.e. ensure there is a section for vx.y.z and the links to git diffs at the bottom are up to date. e.g.

   ## [v3.1.1_schema-v3.1.2] - 2017-11-14
   
   ### Added
   
   * Add sources and javadoc jars
   
   ### Changed
   
   * Uplift schema to v3.1.2
   
   * Change build to Gradle
   

   and

   [v3.1.2_schema-v3.1.2]: https://github.com/gchq/event-logging/compare/v3.1.1_schema-v3.1.2...v3.1.2_schema-v3.1.2
   

2. If the changes include merges in from other branches, e.g. merging a big fix from v1.0 to v2.0 then add a Merged section to make it clear that the changes in the merge have been applied, e.g.

   ### Merged
   
   * Merged in [v1.0.1]
   
   

3. Test the named release build locally by adding the argument -Pversion=vx.y.z to the gradle build command.

4. Ensure all code is committed to the master branch, or a release branch, such as v5.0.

5. To perform the release create an annotated git tag as follows:

   git tag -a vx.y.z

6. Complete the tag’s commit message using the following format

   stroom-something-vx.y.z
   
   
   @@@
   

   Where @@@ is the content of the x.y.z section from the CHANGELOG. GitHub takes the top line as the title for the release. Git will ignore any lines starting with a # so remove them from the markdown headings. Keep the * bullets. E.g.

   event-logging-v3.1.1_schema-v3.1.2
   
   Added 
   
   * Add sources and javadoc jars
   
   Changed
   
   * Uplift schema to v3.1.2
   
   * Change build to Gradle
   

7. Push the tag. Travis will run a build and on detecting the tag, will release any build artefacts to GitHUb Releases. If the project has any published Maven artefacts these will be releases to Bintray.

SNAPSHOT releases

SNAPSHOT releases should not and cannot be released to Bintray. If a development version of a library needs to be shared between projects then you can either use the Gradle task publishToMavenLocal to publish a SNAPSHOT version to your local Maven repository and change your dependency version to SNAPSHOT, or perform a named release along the lines of vx.y.z-alpha.n.

Release Versioning conventions

Semantic versioning is used, and this should be adhered to, see SemVer (external link). The following are examples of valid version names

• SNAPSHOT - Used only for local development, never to be published publicly.

• v3.3.0 - Initial release of v3.3, with an associated v3.3 branch.

• v3.3.1 - A patch release to v3.3 on the v3.3 branch.

• v3.4.0-alpha.1 - An alpha release of v3.4, either on master or a v3.4 branch

• v3.4.0-beta.1 - An beta release of v3.4, either on master or a v3.4 branch

To Perform a Local Build

Full build:

./gradlew clean build

Build without unit tests

./gradlew clean build -x test

Build without integration tests

./gradlew clean build -x integrationTest

Build without any tests or GWT compilation (GWT compilation applies to stroom only)

./gradlew clean build -x test -x integrationTest -x gwtCompile

5 - Running in Docker

NOTE: The published docker images are intended for small scale testing or evaluation purposes and are currently un-tested in a production environment.

Stroom v6.x release

Prerequisites

In order to run Stroom v6.x using Docker you will need the following installed on the machine you intend to run Stroom on:

• A Linux-like shell environment.
• docker CE (v17.12.0+) - e.g docs.docker.com/install/linux/docker-ce/centos/ (external link) for Centos
• docker-compose (v1.21.0+) - docs.docker.com/compose/install/ (external link)
• bash (v4+)
• jq - stedolan.github.io/jq/ (external link) e.g. sudo yum install jq
• curl

Install steps

This will install the core stack (Stroom and the peripheral services required to run Stroom).

Visit stroom-resources/releases (external link) and find the latest stroom_core release and copy the link to the associated stroom_core*.tar.gz archive file.

Using stroom_core-v6.0.19 as an example:

# Set the release version to download
export STROOM_STACK_VER="stroom_core-v6.0.19"

# Make the stack directory
mkdir ${STROOM_STACK_VER}

# Download and extract the Stroom stack into the directory stroom_core-vX.Y.Z
curl -sL https://github.com/gchq/stroom-resources/releases/download/${STROOM_STACK_VER}/${STROOM_STACK_VER}.tar.gz | tar xz -C ${STROOM_STACK_VER}

# Navigate into the new stack directory, where xxxx is the directory that has just been created
cd ${STROOM_STACK_VER}

# Start the stack
./start.sh

Alternatively if you understand the risks of redirecting web sourced content direct to bash, you can get the latest release using:

# Download and extract the Stroom stack
bash <(curl -s https://gchq.github.io/stroom-resources/get_stroom.sh)

# Navigate into the new stack directory
cd stroom_core_test/stroom_core_test*

# Start the stack
./start.sh

On first run stroom will build the database schemas so this can take a minute or two. The start.sh script will provide details of the various URLs that are available.

Open a browser (preferably Chrome) at https://localhost/stroom and login with:

• username: admin (NOT an email address)
• password: admin

The stroom stack comes supplied with self-signed certificates so you may need to accept a prompt warning you about visiting an untrusted site.

Stroom v5.x release

Prerequisites

In order to run Stroom v5.x using Docker you will need the following installed on the machine you intend to run Stroom on:

• A Linux-like shell environment with bash and GNU sed/grep
• docker CE (v17.12.0+)
• docker-compose (v1.21.0+)
• git

Install steps

# Clone the stroom-resources git repository
git clone https://github.com/gchq/stroom-resources.git

# Navigate to the bin directory in the repository
cd stroom-resources/bin

# Start the stroom v5.x stack using docker/docker-compose
./bounceIt.sh -f env/stroom5.env

Open a browser (preferably Chrome) at http://localhost:8080/stroom and login with:

• username: admin
• password: admin

Docker Hub links

The Stroom image (external link)

The GCHQ organisation (external link)

6 - Running Stroom in an IDE

We tend to use IntelliJ as our Java IDE of choice. This is a guide for running Stroom in IntelliJ for the purposes of developing/debugging Stroom.

Prerequisites

In order to build/run/debug Stroom you will need the following:

• Java 8 JDK
• Git
• Gradle
• IntelliJ
• Docker CE
• Docker Compose

These instructions assume that all servcies will either run in the IDE or in Docker containers.

Environment variables

Stroom git repositories

To develop stroom you will need to clone/fork multiple git repositories. To quickly clone all of the Stroom repositories you can use the helper script described in stroom-resource (external link).

Database setup

Stroom requires a MySQL database to run. You can either point stroom at a MySQL server or preferably at a MySQL Docker containers.

MySQL in a Docker container

See the section below on stroom-resources.

Host based MySQL server

With an instance of MySQL server 5.5 running on your local machine do the following to create the stroom database:

# log into your MySQL server using your root credentials
mysql --user=root --password=myrootpassword

Then run the following commands in the MySQL shell:

drop database stroom;
create database stroom;
grant all privileges on stroom.* to stroomuser@localhost identified by 'stroompassword1';
quit;

Local configuration file

When running stroom in an IDE it is advisable to have a local configuration file to allow you to change settings locally without affecting the repository. The local configuration file is expected to live at ~/.stroom/stroom.conf. To create a default version of this file run the script stroom.conf.sh from within the root of the stroom git repository.

Add any properties from stroom.properties that you want different values for, e.g.

stroom-resources

As a minimum to develop stroom you will need clones of the stroom and stroom-resources git repositories. stroom-resources provides the docker-compose configuration for running the many docker containers needed.

Having cloned stroom-resources navigate to the directory stroom-resources/bin and run the script

./bounceIt.sh

On first run this will create a default version of the git-ignored file stroom-resources/bin/local.env which is intended for use by developers to configure the docker stacks to run.

This file is used to set a number of environment variables that docker compose will use to configure the various containers. The key environment variable in there is SERVICE_LIST. This is a space delimited list of the services for docker-compose to run. The services are all defined in stroom-resources/bin/compose/everything.yml and its dependencies. By default SERVICE_LIST runs a core stroom stack entirely in docker.

To run stroom in an IDE stroom needs to be removed from SERVICE_LIST, i.e. by commenting out the line SERVICE_LIST="${SERVICE_LIST} stroom" in local.env. Having done this run stroom’s core dependencies as follows:

./bounceIt.sh

Verify the Gradle build

Before trying to run Stroom in an IDE it is worth performing a Gradle build without the integration tests (as these take ~20mins to run) to verify the code compiles and all dependencies are present.

./gradlew clean build -x integrationTest

Sample Data

Some of the tests are dependant on some sample data and content being present in the database. This sample data/content can also be useful for manually testing the application in development. The sample data/content is generated by a class called SetupSampleData.java. This class assumes that the database being used for Stroom is completely empty.

First you need to create a run configuration for SetupSampleData.java

1. Click Run -> Edit Configurations…
2. Click the green + icon to add a new configuration
3. Select Application as the configuration type
4. In the Main Class field enter SetupSampleData
5. In the Use classpath of module field select stroom-integrationtest
6. If you have set the STROOM_TMP environment variable in your .bashrc / .zshrc then ignore this step. In the Environment variables field click the … icon and add STROOM_TMP = ~/tmp/stroom/ (or whatever directory you choose)
7. Click the OK button

Now run SetupSampleData

1. Click Run -> Run…
2. Select SetupSampleData

You should now have a database populated with tables and data, providing you with some predefined feeds, data, translations, pipelines, dashboards, etc.

Running Stroom from the IDE

The user interface for Stroom is built using GWT (see GWT Project for more information or GWT specific documentation). As a result Stroom needs to be started up with GWT Dev Mode. Dev Mode handles the compilation of the Java user interface source into JavaScript and the source map that links client JavaScript back to Java source for client side debugging.

Stroom is run from the main method in Startup.java. Before running this you need to setup the run configuration:

1. Click Run -> Edit Configurations…
2. Click the green + icon to add a new configuration
3. Select Application as the configuration type
4. In the Main class field enter Startup
5. In the Programme arguments field enter

-startupUrl stroom.jsp -logLevel INFO -war . -logdir . -gen . -extra . -workDir . stroom.app.AppSuperDevMode

1. In the Use classpath of module field select stroom-startup
2. If you have set the STROOM_TMP environment variable in your .bashrc / .zshrc then ignore this step. In the Environment variables field click the … icon and add STROOM_TMP = ~/tmp/stroom/ (or whatever directory you choose)
3. Click the OK button

Now run Startup

1. Click Run -> Run…
2. Select Startup

You should eventually see the GWT Dev Mode window appear.

---

Initially, some of the buttons shown above will not be visible as it is in the process of starting up. As soon as the Launch Default Browser button appears you are ready to open Stroom in a browser. You have two options:

• Click the Launch Default Browser button
• Open your preferred browser and enter the URL http://127.0.0.1:8888

NOTE: Stroom has been written with Google’s Chrome browser in mind so has only been tested on Chrome. Behaviour in other browsers may vary. We would like to improve cross-browser support so please let us know about any browser incompatibilities that you find.

In the browser you will initially see the following:

Starting Stroom

Initialising context…

Once the context has been initialised you will see the Stroom blue background and a spinner while GWT compiles the front-end code. Once the code has been compiled you will be presented with the Stroom login page. Stroom in development mode uses simple username/password authentication. Enter the following credentials:

• Username: admin
• Password: admin

Right click behaviour

Stroom overrides the defualt right click behaviour in the browser with its own context menu. For UI development it is often required to have access to the browser’s context menu for example to inspect elements. To enable the browser’s context menu you need to uncomment the following property in your stroom.conf file, e.g.

#Uncomment this to enable browser's right click menu for development
stroom.ui.oncontextmenu=