Stroom – TODO

Docs: Single Node Docker Installation

Tue, 18 Oct 2022 00:00:00 +0000

Running Stroom in Docker is the quickest and easiest way to get Stroom up and running. Using Docker means you don’t need to install the right versions of dependencies like Java or MySQL or get them configured correctly for Stroom.

This section details how to install single instances of of Stroom and Stroom-Proxy using Docker.

Note

If you want to deploy a Stroom cluster using containers then you should use Kubernetes, see Kubernetes Cluster.

Stroom Docker stacks

Stroom has a number of predefined stacks that combine multiple docker containers into a fully functioning Stroom environment. The Docker stacks are aimed primarily at single node instances or for evaluation/test. The stack makes use of various shell scripts combined with Docker Compose to integrate the various Docker containers and make them easy to run.

At the moment the usable stacks are:

stroom_core - A single node stroom stack geared towards production use.
stroom_core_test - A single node stroom for test/evaluation, pre-loaded with content. Also includes a remote proxy for demonstration purposes. If you just want to try out Stroom, this is the one to use.
stroom_proxy - A remote proxy stack for aggregating and forwarding logs to stroom(-proxy). Intended for use as a remote proxy that will forward received/aggregated data into a downstream stroom/stroom-proxy.
stroom_services - An Nginx instance for running stroom without Docker.

Each stack contains the following docker compose services.

stroom_core

stroom
stroom-proxy-local
stroom-log-sender
nginx
mysql

stroom_core_test

stroom
stroom-proxy-local
stroom-proxy-remote
stroom-log-sender
nginx
mysql

stroom_proxy

stroom-proxy-remote
stroom-log-sender
nginx

stroom_services

stroom-log-sender
nginx

The services are as follows:

stroom - A Stroom instance.
stroom-proxy-local - A Stroom-Proxy instance that is typically local to Stroom and acts as its front door for data reception.
stroom-proxy-remote - A Stroom-Proxy instance that is remote from Stroom (e.g. owned by another team) and is intended to pass data to a downstream Stroom-Proxy.
nginx - An instance of nginx that is configured to reverse proxy to Stroom and Stroom-Proxy as appropriate. It can also be configured to act as a load balancer to multiple Stroom instances if Stroom is being installed without using Docker.
mysql - An instance of MySQL that is configured to create the database and users required by Stroom.
stroom-log-sender - A simple container that is configured to gather all the log files produced by Stroom, Stroom-Proxy and nginx, to then forward them to Stroom so Stroom can process its own logs.

Prerequisites

In order to run Stroom using Docker you will need the following installed on the machine you intend to run Stroom on:

An internet connection. If you don’t have one see Air Gapped Environments.
A Linux-like shell environment.
Docker CE (v17.12.0+) - e.g. docs.docker.com/install/linux/docker-ce/centos/ for Centos
docker-compose (v1.21.0+) - docs.docker.com/compose/install/
bash (v4+)
jq - stedolan.github.io/jq/ e.g. sudo yum install jq
curl
A non-root user to perform the install as, e.g. stroomuser

Note

jq is not a hard requirement but improves the functionality of the health checks and is a useful thing to have, e.g. for using Stroom’s REST API.

Install steps

This will install the core stack (Stroom and the peripheral services required to run Stroom).

Visit stroom-resources/releases to find the latest stack release. The Stroom stack comes in a number of different variants:

stroom_core_test - If you are just evaluating Stroom or just want to see it running then download the stroom_core_test*.tar.gz stack which includes some pre-loaded content.
stroom_core - If it is for an actual deployment of Stroom then download stroom_core*.tar.gz, which has no content and requires some configuration.

Using stroom_core_test-v7.10.11.tar.gz as an example:

# Define the version to download
VERSION="v7.10.11"; STACK="stroom_core_test"

# Download and extract the Stroom stack
curl -sL "https://github.com/gchq/stroom-resources/releases/download/stroom-stacks-${VERSION}/${STACK}-${VERSION}.tar.gz" | tar xz

# Navigate into the new stack directory, where xxxx is the directory that has just been created
cd "${STACK}-${VERSION}"

# Start the stack
./start.sh

Alternatively if you understand the risks of redirecting web sourced content direct to bash, you can get the latest stroom_core_test release using:

# Download and extract the latest Stroom stack
bash <(curl -s https://gchq.github.io/stroom-resources/v7.1/get_stroom.sh)
(out)
# Navigate into the new stack directory
cd stroom_core_test/stroom_core_test*
(out)
# Start the stack
./start.sh

On first run stroom will build the database schemas so this can take a minute or two. The start.sh script will provide details of the various URLs that are available.

Open a browser (preferably Chrome) at https://localhost and login with:

username: admin
password: admin

Note

If you have installed the stroom_core stack no user accounts are created by default. You will need to manually create an administrator account.

The stroom stack comes supplied with self-signed certificates so you may need to accept a prompt warning you about visiting an untrusted site.

Configuration

To configure your new instance see Configuration.

Docker Hub links

Docs: MySQL Setup

Fri, 20 Aug 2021 00:00:00 +0000

TODO

This needs updating to MySQL 8. Stroom v7 requires MySQL 8.

Prerequisites

MySQL 8.0.x server installed (e.g. yum install mysql-server)
Processing User Setup

A single MySQL database is required for each Stroom instance. You do not need to setup a MySQL instance per node in your cluster.

Check Database installed and running

/sbin/chkconfig --list mysqld
(out)mysqld          0:off   1:off   2:on    3:on    4:on    5:on    6:off
mysql --user=root -p

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">(out)Enter password:
(out)Welcome to the MySQL monitor.  Commands end with ; or \g.
(out)...
quit

The following commands can be used to auto start mysql if required:

/sbin/chkconfig –level 345 mysqld on
/sbin/service httpd start

Overview

MySQL configuration can be simple to complex depending on your requirements.

For a very simple configuration you simply need an out-of-the-box mysql install and create a database user account.

Things get more complicated when considering:

Security
Replication
Tuning memory usage
Running Stroom Stats in a different database to Stroom
Performance Monitoring

Simple Install

Ensure the database is running, then create the database and grant access to it:

mysql --user=root

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">(out)Welcome to the MySQL monitor.  Commands end with ; or \g.
(out)...
create database stroom;
(out)Query OK, 1 row affected (0.02 sec)

grant all privileges on stroom.* to 'stroomuser'@'host' identified by 'password';
(out)Query OK, 0 rows affected (0.00 sec)

create database stroom_stats;
(out)Query OK, 1 row affected (0.02 sec)

grant all privileges on stroom_stats.* to 'stroomuser'@'host' identified by 'password';
(out)Query OK, 0 rows affected (0.00 sec)

flush privileges;
(out)Query OK, 0 rows affected (0.00 sec)

Advanced Security

It is recommended to run /usr/bin/mysql_secure_installation to remove test database and accounts.

./stroom-setup/mysql_grant.sh is a utility script that creates accounts for you to use within a cluster (or single node setup). Run to see the options:

./mysql_grant.sh
(out)usage : --name=<instance name (defaults to my for /etc/my.cnf)>
(out)        --user=<the stroom user for the db>
(out)        --password=<the stroom password for the db>
(out)        --cluster=<the file with a line per node in the cluster>
(out)--user=<db user> Must be set

N.B. name is used when multiple mysql instances are setup (see below).

You need to create a file cluster.txt with a line for each member of your cluster (or single line in the case of a one node Stroom install). Then run the utility script to lock down the server access.

hostname >> cluster.txt
./stroom-setup/mysql_grant.sh --name=mysql56_dev --user=stroomuser --password= --cluster=cluster.txt
(out)Enter root mysql password :
(out)--------------
(out)flush privileges
(out)--------------
(out)
(out)--------------
(out)delete from mysql.user where user = 'stroomuser'
(out)--------------
(out)...
(out)...
(out)...
(out)--------------
(out)flush privileges
(out)--------------

Advanced Install

The below example uses the utility scripts to create 3 custom mysql server instances on 2 servers:

server1 - stroom (source),
server2 - stroom (replica), stroom_stats

As root on server1:

yum install "mysql56-mysql-server"

Create the master database:

./stroom-setup/mysqld_instance.sh --name=mysqld56_stroom --port=3106 --server=mysqld56 --os=rhel6

(out)--master not set ... assuming master database
(out)Wrote base files in tmp (You need to move them as root).  cp /tmp/mysqld56_stroom /etc/init.d/mysqld56_stroom; cp /tmp/mysqld56_stroom.cnf /etc/mysqld56_stroom.cnf
(out)Run mysql client with mysql --defaults-file=/etc/mysqld56_stroom.cnf

cp /tmp/mysqld56_stroom /etc/init.d/mysqld56_stroom; cp /tmp/mysqld56_stroom.cnf /etc/mysqld56_stroom.cnf
/etc/init.d/mysqld56_stroom start

(out)Initializing MySQL database:  Installing MySQL system tables...
(out)OK
(out)Filling help tables...
(out)...
(out)...
(out)Starting mysql56-mysqld:                                   [  OK  ]

Check Start up Settings Correct

chkconfig mysqld off
chkconfig mysql56-mysqld off
chkconfig --add mysqld56_stroom
chkconfig mysqld56_stroom on

chkconfig --list | grep mysql
(out)mysql56-mysqld  0:off   1:off   2:off   3:off   4:off   5:off   6:off
(out)mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
(out)mysqld56_stroom    0:off   1:off   2:on    3:on    4:on    5:on    6:off
(out)mysqld56_stats  0:off   1:off   2:on    3:on    4:on    5:on    6:off

Create a text file with all members of the cluster:

vi cluster.txt

(out)node1.my.org
(out)node2.my.org
(out)node3.my.org
(out)node4.my.org

Create the grants:

./stroom-setup/mysql_grant.sh --name=mysqld56_stroom --user=stroomuser --password=password --cluster=cluster.txt

As root on server2:

yum install "mysql56-mysql-server"


./stroom-setup/mysqld_instance.sh --name=mysqld56_stroom --port=3106 --server=mysqld56 --os=rhel6 --master=node1.my.org --user=stroomuser --password=password

(out)--master set ... assuming slave database
(out)Wrote base files in tmp (You need to move them as root).  cp /tmp/mysqld56_stroom /etc/init.d/mysqld56_stroom; cp /tmp/mysqld56_stroom.cnf /etc/mysqld56_stroom.cnf
(out)Run mysql client with mysql --defaults-file=/etc/mysqld56_stroom.cnf

cp /tmp/mysqld56_stroom /etc/init.d/mysqld56_stroom; cp /tmp/mysqld56_stroom.cnf /etc/mysqld56_stroom.cnf
/etc/init.d/mysqld56_stroom start

(out)Initializing MySQL database:  Installing MySQL system tables...
(out)OK
(out)Filling help tables...
(out)...
(out)...
(out)Starting mysql56-mysqld:                                   [  OK  ]

Check Start up Settings Correct

chkconfig mysqld off
chkconfig mysql56-mysqld off
chkconfig --add mysqld56_stroom
chkconfig mysqld56_stroom on

chkconfig --list | grep mysql
(out)mysql56-mysqld  0:off   1:off   2:off   3:off   4:off   5:off   6:off
(out)mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off
(out)mysqld56_stroom    0:off   1:off   2:on    3:on    4:on    5:on    6:off

Create the grants:

./stroom-setup/mysql_grant.sh --name=mysqld56_stroom --user=stroomuser --password=password --cluster=cluster.txt

Make the slave database start to follow:

cat /etc/mysqld56_stroom.cnf | grep "change master"
(out)# change master to MASTER_HOST='node1.my.org', MASTER_PORT=3106, MASTER_USER='stroomuser', MASTER_PASSWORD='password';

mysql --defaults-file=/etc/mysqld56_stroom.cnf

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">change master to MASTER_HOST='node1.my.org', MASTER_PORT=3106, MASTER_USER='stroomuser', MASTER_PASSWORD='password';
start slave;

As processing user on server1:

mysql --defaults-file=/etc/mysqld56_stroom.cnf --user=stroomuser --password=password

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">create database stroom;
(out)Query OK, 1 row affected (0.00 sec)

use stroom;
(out)Database changed

create table test (a int);
(out)Query OK, 0 rows affected (0.05 sec)

As processing user on server2 check server replicating OK:

mysql --defaults-file=/etc/mysqld56_stroom.cnf --user=stroomuser --password=password

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">show create table test;
(out)+-------+----------------------------------------------------------------------------------------+
(out)| Table | Create Table                                                                           |
(out)+-------+----------------------------------------------------------------------------------------+
(out)| test  | CREATE TABLE `test` (`a` int(11) DEFAULT NULL  ) ENGINE=InnoDB DEFAULT CHARSET=latin1  |
(out)+-------+----------------------------------------------------------------------------------------+
(out)1 row in set (0.00 sec)

As root on server2:

/home/stroomuser/stroom-setup/mysqld_instance.sh --name=mysqld56_stats --port=3206 --server=mysqld56 --os=rhel6 --user=statsuser --password=password
cp /tmp/mysqld56_stats /etc/init.d/mysqld56_stats; cp /tmp/mysqld56_stats.cnf /etc/mysqld56_stats.cnf
/etc/init.d/mysqld56_stats start
chkconfig mysqld56_stats on

Create the grants:

./stroom-setup/mysql_grant.sh --name=mysqld56_stats --database=stats  --user=stroomstats --password=password --cluster=cluster.txt

As processing user create the database:

mysql --defaults-file=/etc/mysqld56_stats.cnf --user=stroomstats --password=password

" 
    data-filter-output="(out)"
    data-continuation-prompt="->"
    data-filter-continuation="(con)">(out)Welcome to the MySQL monitor.  Commands end with ; or \g.
(out)....
create database stats;
(out)Query OK, 1 row affected (0.00 sec)

Docs: Stroom Proxy Installation

Fri, 20 Aug 2021 00:00:00 +0000

Stroom-Proxy can be installed in 4 main ways:

App - There is an app version that runs Stroom-Proxy as a Java JAR JAR Java Archive is a file format for distributing Java class files, associated metadata and resource files. It is a compressed archive based on the {{< glossary “ZIP” >}} format, so can be inspected with any tool capable of reading a ZIP file. Stroom and Stroom-Proxy are distributed as JAR files.Click to see more details... file locally on the server and has settings contained in a configuration file that controls access to the stroom server and database.
Docker Stack - Stroom-Proxy, Nginx and Stroom-Log-Sender run in Docker containers, orchestrated using Docker Compose and some shell scripts. The stroom-proxy image is essentially a minimal Alpine Linux container with the appropriate Java version installed and the Stroom-Proxy JAR contained within it.
Docker Images - Manually run containers based on the Stroom-Proxy docker image.
Kubernetes - Deploy Stroom-Proxy into a Kubernetes cluster.

The document will cover the installation and configuration of the Stroom-Proxy software for both the ‘app’ and Docker stack deployments.

Note

Older versions of Stroom-Proxy ‘sent’ data to Stroom by being co-located with Stroom and using a shared repository location. Stroom-Proxy would write to the repository and Stroom would retrieve data from there.

This method is no longer used. Current versions of Stroom-Proxy forward data to Stroom over HTTP so do not need to be co-located with Stroom.

Typical Deployments

Stroom-Proxy is typically deployed in front of Stroom to act as a proxy for data receipt into Stroom. This abstracts Stroom from the clients sending the data and ensures that received data is aggregated into sensibly sized streams.

For a production Stroom cluster, it is likely that you will want multiple Stroom-Proxy instances behind a load balancer for resiliency and load management.

Note

Stroom-Proxy instances are independent and cannot form a coherent cluster. If you have a ‘cluster’ of Stroom-Proxy instances for resilience, each Stroom-Proxy instance can only work with the data it has received and loss of an instance means the loss of any data it was in the process of aggregating.

Also if you have a large ‘cluster’ of Stroom-Proxy instances, this can have an impact on aggregation as data for the same aggregation key (Feed and Stream Type) will be spread over multiple instances and thus aggregates. For low volume Feeds, this can mean smaller aggregates than is preferred.

To mitigate against this, you can use sticky sessions when load balancing or ideally use a load balancer that allows selection of the upstream instance using HTTP headers, i.e. Feed.

Assumptions

The following assumptions are used in this document.

The user has reasonable RHEL/CentOS/Rocky System administration skills.
Installation is on a fully patched minimal RHEL/CentOS/Rocky instance.
The application user stroomuser has been created in the OS.
The user has set up the Stroom processing user as described here.
The prerequisite software has been installed.

Firewall Configuration

For both methods of deployment, the ports used are as follows: Some may need to be opened to allow access to the ports from outside the host.

80 - Nginx listens on port 80 but redirects onto 443.
443 - Nginx listens on port 443.
8090 - Stroom-Proxy listens on port 8090 for its main public APIs (/datafeed, REST endpoints, etc).
8091 - Stroom-Proxy listens on port 8091 for its administration APIs. Access to this port should probably be carefully controlled.

It is therefore likely that you will only want to expose 443 and maybe 80 to other hosts.

For example on a RHEL/CentOS server using firewalld the commands would be as root user:

firewall-cmd --zone=public --permanent --add-port=80/tcp
firewall-cmd --zone=public --permanent --add-port=443/tcp
firewall-cmd --reload

Stroom Proxy (docker version)

The build of a stroom proxy where the Stroom-Proxy Java application (and associated services) are running in docker containers.

Because everything is running in Docker containers, the only requirement for the host is for the following:

Docker Engine
Docker Compose Plugin
bash v4 or greater - Used by the stack scripts.
GNU coreutils - Used by the stack scripts.
jq - Used by the stack scripts.

Download and install docker

To install Docker Engine and the Docker Composer Plugin see:

All the Stroom-Proxy logs and data will be stored in Docker managed volumes that will, by default, reside in /var/lib/docker. It is typical that this directory will be on small mount point for the OS. It is therefore recommended to relocate this directory to a mount with more space and sufficient resilience, i.e. RAID mirroring.

To do this you need to follow these steps:

Stop the Docker engine.
Move the directory to its new location.
Edit the file /etc/docker/daemon.json and ensure this field is present with the new location as the value.
```
{
  "data-root": "/path/to/new/location"
}
```
Start the Docker engine.

Download and Install Docker Stack

The stroom_proxy Docker stack is available from stroom-resources releases on GitHub. The stack distribution is simply a collection of shell scripts and Docker Compose configuration files. The Docker images will get pulled down from DockerHub when the stack is started.

The installation example below is for stroom version 7.10.20 - but is applicable to other stroom v7 versions. As a suitable stroom user e.g. stroomuser - download and unpack the stroom software.

mkdir -p ~/stroom-proxy
cd ~/stroom-proxy
wget https://github.com/gchq/stroom-resources/releases/download/stroom-stacks-v7.10.20/stroom_proxy-v7.10.20.tar.gz
tar -zxf stroom_proxy-v7.10.20.tar.gz
cd stroom_proxy-v7.10.20

For a stroom proxy, the configuration file stroom_proxy/stroom_proxy-v7.10.20/stroom_proxy.env needs to be edited, with the connection details of the stroom server that data files will be sent to. The default network port for connection to the stroom server is 8080.

The values that need to be set are:

STROOM_PROXY_REMOTE_FEED_STATUS_API_KEY  
STROOM_PROXY_REMOTE_FEED_STATUS_URL  
STROOM_PROXY_REMOTE_FORWARD_URL

The ‘API key’ is generated on the stroom server and is related to a specific user e.g. proxyServiceUser. The 2 URL values also refer to the stroom server and can be a fully qualified domain name (fqdn) or the IP Address.

e.g. if the stroom server was - stroom-serve.somewhere.co.uk - the URL lines would be:

export STROOM_PROXY_REMOTE_FEED_STATUS_URL="http://stroom-serve.somewhere.co.uk:8080/api/feedStatus/v1"
export STROOM_PROXY_REMOTE_FORWARD_URL="http://stroom-serve.somewhere.co.uk:8080/stroom/datafeed"

To Start Stroom Proxy

As the stroom user, run the ‘start.sh’ script found in the stroom install:

cd ~/stroom_proxy/stroom_proxy-v7.10.20/
./start.sh

The first time the script is run it will download the docker images from DockerHub:

stroom-proxy-remote
stroom-log-sender
stroom-nginx

Once the script has completed the Stroom-Proxy server should be running.

The stack directory contains the following scripts for managing the Stroom-Proxy stack.

health.sh - Tests and displays the health of the stack.
info.sh* - Displays info about the stack.
pull_images.sh - Pulls all the docker images used in the stack.
logs.sh - Tails the logs from all services in the stack.
remove.sh - Removes all services and volumes in the stack. Warning: this will delete any data held in Stroom-Proxy.
restart.sh - Restarts all or named services it the stack.
send_data.sh - Script to aid POSTing data into Stroom-Proxy.
set_log_levels.sh - Sets log levels for classes/packages on the running Stroom-Proxy.
set_services.sh - Used for disabling services in the stack.
show_config.sh - Displays the effective docker compose config taking the env file into account.
start.sh - Starts all or named services it the stack.
status.sh - Shows the status of the services in the stack.
stop.sh - Stops all or named services it the stack.

Stroom Proxy (app version)

This is the bare bones installation method that requires installing everything manually. If you are able to use Docker we recommend doing this as there are less things to install and configure, e.g. nginx, send_to_stroom.sh, cron, etc.

Stroom-Proxy is distributed as a ( JAR JAR Java Archive is a file format for distributing Java class files, associated metadata and resource files. It is a compressed archive based on the {{< glossary “ZIP” >}} format, so can be inspected with any tool capable of reading a ZIP file. Stroom and Stroom-Proxy are distributed as JAR files.Click to see more details...) file so this method will run this JAR using the java executable.

The pre-requisites for this deployment are:

RHEL/CentOS/Rocky
Java 25+ JDK (JDK is preferred over JRE as it provides additional tools (e.g. jmap) for capturing heap histogram statistics).
bash v4 or greater - Used by the helper scripts.
GNU coreutils - Used by the helper scripts.

For details about which Java distribution and version to use, and how to install it, see Java.

Note

Disable selinux to avoid issues with access and file permissions.

Download and install Stroom v7 (app version)

Stroom-Proxy releases are available from github.com/gchq/stroom/releases . Each release has a number of artefacts, the Stroom-Proxy application is stroom-proxy-app-v*.zip.

The installation example below is for stroom version v7.10.20, but is applicable to other stroom v7 versions. As a suitable stroom user e.g. stroomuser - download and unpack the stroom software.

wget https://github.com/gchq/stroom/releases/download/v7.10.20/stroom-proxy-app-v7.10.20.zip
unzip stroom-proxy-app-v7.10.20.zip

The configuration file – stroom-proxy/config/config.yml – is the principal file that controls the configuration of Stroom-Proxy. See Stroom Proxy Configuration.

Docs: Securing Stroom

Fri, 04 Feb 2022 00:00:00 +0000

NOTE This document was written for stroom v4/5. Some parts may not be applicable for v6+.

Firewall

The following firewall configuration is recommended:

Outside cluster drop all access except ports HTTP 80, HTTPS 443, and any other system ports your require SSH, etc.
Within cluster allow all access

This will enable nodes within the cluster to communicate on:

8080 - Stroom HTTP.
8081 - Stroom HTTP (admin).
8090 - Stroom Proxy HTTP.
8091 - Stroom Proxy HTTP (admin).
3306 - MySQL

MySQL

TODO

Update this for MySQL 8

It is recommended that you run mysql_secure_installation to set a root password and remove the test database:

mysql_secure_installation

When prompted, answer as follows (providing a root password when asked):

Set root password? → Y
Remove anonymous users? → Y
Disallow root login remotely? → Y
Remove test database and access to it? → Y
Reload privilege tables now? → Y

Docs: Naming Conventions

Mon, 14 Feb 2022 00:00:00 +0000

Stroom has been in use by GCHQ for many years and is used to process logs from a large number of different systems. This sections aims to provide some guidelines on how to name and organise your content, e.g. Feeds, XSLTs, Pipelines, Folders, etc. These are not hard rules and you do not have to follow them, however it may help when it comes to sharing content.

TODO

Complete this section

Community: Developing Content

Wed, 12 Jan 2022 00:00:00 +0000

TODO

Add content

Docs: Java Key Store Setup

Fri, 20 Aug 2021 00:00:00 +0000

TODO

This is out of date for stroom 7.

In order that the java process communicates over https (for example Stroom Proxy forwarding onto Stroom) the JVM requires relevant keystore’s setting up.

As the processing user copy the following files to a directory stroom-jks in the processing user home directory :

CA.crt - Certificate Authority
SERVER.crt - Server certificate with client authentication attributes
SERVER.key - Server private key

As the processing user perform the following:

First turn your keys into der format:

cd ~/stroom-jks

SERVER=<SERVER crt/key PREFIX>
AUTHORITY=CA

openssl x509 -in ${SERVER}.crt -inform PEM -out ${SERVER}.crt.der -outform DER
openssl pkcs8 -topk8 -nocrypt -in ${SERVER}.key -inform PEM -out ${SERVER}.key.der -outform DER

Import Keys into the Key Stores:

Stroom_UTIL_JAR=`find ~/*app -name 'stroom-util*.jar' -print | head -1`

java -cp ${Stroom_UTIL_JAR} stroom.util.cert.ImportKey keystore=${SERVER}.jks keypass=${SERVER} alias=${SERVER} keyfile=${SERVER}.key.der certfile=${SERVER}.crt.der
keytool -import -noprompt -alias ${AUTHORITY} -file ${AUTHORITY}.crt -keystore ${AUTHORITY}.jks -storepass ${AUTHORITY}

Update Processing User Global Java Settings:

PWD=`pwd`
echo "export JAVA_OPTS=\"-Djavax.net.ssl.trustStore=${PWD}/${AUTHORITY}.jks -Djavax.net.ssl.trustStorePassword=${AUTHORITY} -Djavax.net.ssl.keyStore=${PWD}/${SERVER}.jks -Djavax.net.ssl.keyStorePassword=${SERVER}\"" >> ~/env.sh

Any Stroom or Stroom Proxy instance will now additionally pickup the above JAVA_OPTS settings.

Docs: Dictionaries

Tue, 27 Jul 2021 00:00:00 +0000

Creating

Right click on a folder in the explorer tree that you want to create a dictionary in. Choose ‘New/Dictionary’ from the popup menu:

Call the dictionary something like ‘My Dictionary’ and click OK .

Now just add any search terms you want to the newly created dictionary and click .

You can add multiple terms.

Terms on separate lines act as if they are part of an ‘OR’ expression when used in a search.
```
apple
banana
orange
```
Terms on a single line separated by spaces act as if they are part of an ‘AND’ expression when used in a search.
```
apple,banana,orange
```

Using the Dictionary

To perform a search using your dictionary, just choose the newly created dictionary as part of your search expression:

TODO: Fix image

Docs: Stroom 6 Installation

Fri, 20 Aug 2021 00:00:00 +0000

TODO

Update this for Stroom 7.

Running on a single box

Running a release

Download a release , for example Stroom Core v6.0 Beta 3 , unpack it, and run the start.sh script. When you’ve given it some time to start up go to http://localhost/stroom. There’s a README.md file inside the tar.gz with more information.

Admin Account creation

By default, Stroom does not come with an administrator account/user so one or more administrators will need to be setup in order to login and continue provisioning Stroom via the UI.

See Creating an Internal IDP Administrator or Creating an External IDP Administrator depending on the type of Identity Provider (IDP) Identity Provider (IDP) An Identity Provider is a system or service that can authenticate a user and assert their identity. Identity providers can support single sign on (SSO), which allows the user to sign in once to the Identity Provider so they are then authenticated to all systems using that IDP.Click to see more details... that is configured.

Post-install hardening

Before first run

Change database passwords

If you don’t do this before the first run of Stroom then the passwords will already be set and you’ll have to change them on the database manually, and then change the .env.

This change should be made in the .env configuration file. If the values are not there then this service is not included in your Stroom stack and there is nothing to change.

STROOM_DB_PASSWORD
STROOM_DB_ROOT_PASSWORD
STROOM_STATS_DB_ROOT_PASSWORD
STROOM_STATS_DB_PASSWORD
STROOM_AUTH_DB_PASSWORD
STROOM_AUTH_DB_ROOT_PASSWORD
STROOM_ANNOTATIONS_DB_PASSWORD
STROOM_ANNOTATIONS_DB_ROOT_PASSWORD

On first run

Create yourself an account

After first logging in as admin you should create yourself a normal account (using your email address) and add yourself to the Administrators group. You should then log out of admin, log in with your new administrator account and then disable the admin account.

If you decide to use the admin account as your normal account you might find yourself locked out. The admin account has no associated email address, so the Reset Password feature will not work if your account is locked. It might become locked if you enter your password incorrectly too many times.

Delete un-used users and API keys

If you’re not using stats you can delete or disable the following:
- the user statsServiceUser
- the API key for statsServiceUser

Change the API keys

First generate new API keys. You can generate a new API key using Stroom. From the top menu, select:

The following need to be changed:

STROOM_SECURITY_API_TOKEN
- This is the API token for user stroomServiceUser.

Then stop Stroom and update the API key in the .env configuration file with the new value.

Troubleshooting

I’m trying to use certificate logins (PKI) but I keep being prompted for the username and password!

You need to be sure of several things:

When a user arrives at Stroom the first thing Stroom does is redirect the user to the authentication service. This is when the certificate is checked. If this redirect doesn’t use HTTPS then nginx will not get the cert and will not send it onwards to the authentication service. Remember that all of this stuff, apart from back-channel/service-to-service chatter, goes through nginx. The env var that needs to use HTTPS is STROOM_AUTHENTICATION_SERVICE_URL. Note that this is the var Stroom looks for, not the var as set in the stack, so you’ll find it in the stack YAML.
Are your certs configured properly? If nginx isn’t able to decode the incoming cert for some reason then it won’t pass anything on to the service.
Is your browser sending certificates?

Docs: Nodes

Tue, 27 Jul 2021 00:00:00 +0000

All nodes in a Stroom cluster must be configured correctly for them to communicate with each other.

Configuring nodes

Open Monitoring/Nodes from the top menu. The nodes screen looks like this:

TODO

Screenshot

You need to edit each line by selecting it and then clicking the edit icon at the bottom. The URL for each node needs to be set as above but obviously substituting in the host name of the individual node, e.g. http://<HOST_NAME>:8080/stroom/clustercall.rpc

Nodes are expected to communicate with each other on port 8080 over http. Ensure you have configured your firewall to allow nodes to talk to each other over this port. You can configure the URL to use a different port and possibly HTTPS but performance will be better with HTTP as no SSL termination is required.

Once you have set the URLs of each node you should also set the master assignment priority for each node to be different to all of the others. In the image above the priorities have been set in a random fashion to ensure that node3 assumes the role of master node for as long as it is enabled. You also need to check all of the nodes are enabled that you want to take part in processing or any other jobs.

Keep refreshing the table until all nodes show healthy pings as above. If you do not get ping results for each node then they are not configured correctly.

Once a cluster is configured correctly you will get proper distribution of processing tasks and search will be able to access all nodes to take part in a distributed query.

Docs: Pipeline Elements

Thu, 03 Feb 2022 00:00:00 +0000

Reader

Reader elements decode the data in raw byte form using the Feed’s configured character encoding. Some of them will also transform the data at the character level before the data are parsed into a structured form.

BOMRemovalFilterInput

Removes the Byte Order Mark (if present) from the stream.

BadTextXMLFilterReader

Escapes the content of a configured list of named XML elements that are known to potentially contain un-escaped XML reserved characters. For example the element <Expression>$time < now()</Expression> would be transformed to <Expression>$time < now()</Expression> if property leafList is set to Expression.

Element properties:

Name	Description	Default Value	Value Type
tags	A comma separated list of XML element names (case sensitive) between which non-escaped XML characters will be escaped, e.g. ‘>’ => ‘>’.	-	String

FindReplaceFilter

Replaces strings or regexes with new strings.

Element properties:

Name	Description	Default Value	Value Type
bufferSize	The number of characters to buffer when matching the regex.	1000	Integer
dotAll	Let ‘.’ match all characters in a regex.	false	Boolean
escapeFind	Whether or not to escape find pattern or text.	true	Boolean
escapeReplacement	Whether or not to escape replacement text.	true	Boolean
find	The text or regex pattern to find and replace.	-	String
maxReplacements	The maximum number of times to try and replace text. There is no limit by default.	-	String
regex	Whether the pattern should be treated as a literal or a regex.	false	Boolean
replacement	The replacement text.	-	String
showReplacementCount	Show total replacement count	true	Boolean

InvalidCharFilterReader

Removes any characters that are not in the standard XML character set. The version of XML (e.g. 1.0 or 1.1) can be set using the ‘xmlVersion’ property.

Element properties:

Name	Description	Default Value	Value Type
warnOnRemoval	Log a warning if any characters have been removed from the input stream.	true	Boolean
xmlVersion	XML version, e.g. ‘1.0’ or ‘1.1’	1.1	String

InvalidXMLCharFilterReader

Replaces any characters that are not in the standard XML character set with a ‘�’. The version of XML (e.g. 1.0 or 1.1) can be set using the ‘xmlVersion’ property.

Element properties:

Name	Description	Default Value	Value Type
warnOnReplacement	Log a warning if any characters have been replaced in the input stream.	true	Boolean
xmlVersion	XML version, e.g. ‘1.0’ or ‘1.1’	1.1	String

Reader

A basic reader that simply decodes the raw bytes using the Feed’s configured character encoding. It does not transform the data in any other way.

Parser

Parser elements parse raw text data that has an expected structure (e.g. XML, JSON, CSV) into XML events (elements, attributes, text, etc.) that can be further validated or transformed using XSLT. The choice of Parser will be dictated by the structure of the data. If no Reader is used before the Parser, the Parser will also do the job of a simple Reader and decode the raw bytes using the Feed’s configured character encoding.

CombinedParser

The original general-purpose reader/parser that covers all source data types but provides less flexibility than the source format-specific parsers such as dsParser. It effectively combines a BOMRemovalFilterInput, an InvalidCharFilterReader and Parser (based on the type property.

Warning

It is strongly recommended to instead use a combination of Readers and one of the type specific Parsers. This will make the intent of the pipeline much clearer and allow for much greater control.

Element properties:

Name	Description	Default Value	Value Type
fixInvalidChars	Fix invalid XML characters from the input stream.	false	Boolean
namePattern	A name pattern to load a text converter dynamically.	-	String
suppressDocumentNotFoundWarnings	If the text converter cannot be found to match the name pattern suppress warnings.	false	Boolean
textConverter	The text converter configuration that should be used to parse the input data.	-	Document
type	The parser type, e.g. ‘JSON’, ‘XML’, ‘Data Splitter’.	-	String

DSParser

A parser for handling structured plain text data (e.g. CSV or fixed width fields) using the Data Splitter domain specific language. For more details see Data Splitter.

Element properties:

Name	Description	Default Value	Value Type
namePattern	A name pattern to load a data splitter dynamically.	-	String
suppressDocumentNotFoundWarnings	If the data splitter cannot be found to match the name pattern suppress warnings.	false	Boolean
textConverter	The data splitter configuration that should be used to parse the input data.	-	Document

JSONParser

A built-in parser for parsing JSON source data (in JSON fragment format) into an XML representation of the JSON. The Resulting XML will conform to the http://www.w3.org/2013/XSL/json namespace.

Element properties:

Name	Description	Default Value	Value Type
addRootObject	Add a root map element.	true	Boolean
allowBackslashEscapingAnyCharacter	Feature that can be enabled to accept quoting of all character using backslash quoting mechanism: if not enabled, only characters that are explicitly listed by JSON specification can be thus escaped (see JSON spec for small list of these characters)	false	Boolean
allowComments	Feature that determines whether parser will allow use of Java/C++ style comments (both ‘/’+’*’ and ‘//’ varieties) within parsed content or not.	false	Boolean
allowMissingValues	Feature allows the support for “missing” values in a JSON array: missing value meaning sequence of two commas, without value in-between but only optional white space.	false	Boolean
allowNonNumericNumbers	Feature that allows parser to recognize set of “Not-a-Number” (NaN) tokens as legal floating number values (similar to how many other data formats and programming language source code allows it).	false	Boolean
allowNumericLeadingZeros	Feature that determines whether parser will allow JSON integral numbers to start with additional (ignorable) zeroes (like: 000001).	false	Boolean
allowSingleQuotes	Feature that determines whether parser will allow use of single quotes (apostrophe, character ‘'’) for quoting Strings (names and String values). If so, this is in addition to other acceptable markers but not by JSON specification).	false	Boolean
allowTrailingComma	Feature that determines whether we will allow for a single trailing comma following the final value (in an Array) or member (in an Object). These commas will simply be ignored.	false	Boolean
allowUnquotedControlChars	Feature that determines whether parser will allow JSON Strings to contain unquoted control characters (ASCII characters with value less than 32, including tab and line feed characters) or not. If feature is set false, an exception is thrown if such a character is encountered.	false	Boolean
allowUnquotedFieldNames	Feature that determines whether parser will allow use of unquoted field names (which is allowed by Javascript, but not by JSON specification).	false	Boolean
allowYamlComments	Feature that determines whether parser will allow use of YAML comments, ones starting with ‘#’ and continuing until the end of the line. This commenting style is common with scripting languages as well.	false	Boolean

XMLFragmentParser

A parser to convert multiple XML fragments into an XML document. For example the data may contain:

<Event>...</Event>
<Event>...</Event>

i.e. with no root element, so not valid XML. The XMLFragmentParser will wrap the fragments with a root element as defined in the TextConverter document configured with the textConverterRef property.

Element properties:

Name	Description	Default Value	Value Type
namePattern	A name pattern to load a text converter dynamically.	-	String
suppressDocumentNotFoundWarnings	If the text converter cannot be found to match the name pattern suppress warnings.	false	Boolean
textConverter	The XML fragment wrapper that should be used to wrap the input XML.	-	Document

XMLParser

A parser to parse data that is expected to be XML into a series of XML events that can be consumed by a Filter element.

Filter

Filter elements work with XML events that have been generated by a parser. They can consume the events without modifying them, e.g. RecordCountFilter or modify them in some way, e.g. XSLTFilter. Multiple filters can be used one after another with each using the output from the last as its input.

DynamicIndexingFilter

A filter to send source data to an index.

Element properties:

Name	Description	Default Value	Value Type
index	The index to send records to.	-	Document

DynamicSearchResultOutputFilter

Used in a search extraction pipeline for extracting field values that have not been stored in the index and where the fields are dynamic and derived from the data rather than being defined in the Index settings. Consumes XML events in the index-documents:1 namespace to convert them into a form so that they can be used in a Dashboard/Query/Analytic.

ElasticIndexingFilter

A filter consuming XML events in the records:2 namespace to index/store the fields and their values in an Elasticsearch Index.

Element properties:

Name	Description	Default Value	Value Type
batchSize	Maximum number of documents to index in each bulk request.	10000	Integer
cluster	Target Elasticsearch cluster.	-	Document
indexName	Name of the Elasticsearch index. Variables specified such as `{year}` are replaced with the corresponding field values contained in the document root. Field names beginning with an underscore are not written to the document and are only used in the index name pattern.	-	String
ingestPipeline	Name of the Elasticsearch ingest pipeline to execute when indexing.	-	String
purgeOnReprocess	When reprocessing a stream, first delete any documents from the index matching the source stream ID.	true	Boolean
refreshAfterEachBatch	Refresh the index after each batch is processed, making the indexed documents visible to searches.	false	Boolean

HttpPostFilter

This element is deprecated, you should instead use the much more flexible HTTPAppender . This element will simply POST the output of the XML events to the configured URL.

Element properties:

Name	Description	Default Value	Value Type
receivingApiUrl	The URL of the receiving API.	-	String

IdEnrichmentFilter

Adds the attributes ‘StreamId’ and ‘EventId’ to the ’event’ element to enrich the event with its ordinal number in the stream and the ID of the stream that it belongs to. ID enrichment is required to be able to index events as it provides them with an ID that is unique within Stroom. It assumes that an record/event is an XML element at the first level below the root element, i.e. for ’event-logging:3’ XML this means the <Event> element.

IndexingFilter

A filter consuming XML events in the records:2 namespace to index/store the fields and their values in a Lucene Index.

Element properties:

Name	Description	Default Value	Value Type
index	The index to send records to.	-	Document

MergeFilter

Merges XML that has been split into separate XML instances.

Element properties:

Name	Description	Default Value	Value Type
mergeDepth	The depth of XML elements to merge at.	1	Integer

PlanBFilter

Takes XML input (conforming to the reference-data:2 schema) and loads the data into the Plan B State Store. Reference data values can be either simple strings or XML fragments.

RecordCountFilter

Counts events/records in the stream. An event/record is taken to be an XML element that is at the first level below the root element, i.e. for ’event-logging:3’ XML this means the <Event> element.

Element properties:

Name	Description	Default Value	Value Type
countRead	Is this filter counting records read or records written?	true	Boolean
recordDepth	The depth of XML elements to count records.	1	Integer

RecordOutputFilter

Filters out records/events that have raised an Error or Fatal Error during processing. If all records/events have raised at least an Error then no XML events will be output. It assumes that an record/event is an XML element at the first level below the root element, i.e.
for ’event-logging:3’ XML this means the <Event> element.""",

ReferenceDataFilter

Takes XML input (conforming to the reference-data:2 schema) and loads the data into the Reference Data Store. Reference data values can be either simple strings or XML fragments.

Element properties:

Name	Description	Default Value	Value Type
overrideExistingValues	Allow duplicate keys to override existing values?	true	Boolean
warnOnDuplicateKeys	Warn if there are duplicate keys found in the reference data?	false	Boolean

SafeXMLFilter

Restricts the characters to a very simple set consisting of [a-zA-Z0-9] and [ .:-_/]. All other characters are replaced by ~NNN, where NNN is a three digit codepoint for the replaced character.

SchemaFilter

Checks the format of the source data against one of a number of XML schemas. This ensures that if non-compliant data is generated, it will be flagged as in error and will not be passed to any subsequent processing elements.

Element properties:

Name	Description	Default Value	Value Type
namespaceURI	Limits the schemas that can be used to validate data to those with a matching namespace URI.	-	String
schemaGroup	Limits the schemas that can be used to validate data to those with a matching schema group name.	-	String
schemaLanguage	The schema language that the schema is written in.	http://www.w3.org/2001/XMLSchema	String
schemaValidation	Should schema validation be performed?	true	Boolean
systemId	Limits the schemas that can be used to validate data to those with a matching system id.	-	String

SearchResultOutputFilter

Used in a search extraction pipeline for extracting field values that have not been stored in the index and where the field definitions are defined in the Index settings. Consumes XML events in the records:2 namespace to convert them into a form so that they can be used in a Dashboard/Query/Analytic.

SolrIndexingFilter

Delivers source data to the specified index in an external Solr instance/cluster.

Element properties:

Name	Description	Default Value	Value Type
batchSize	How many documents to send to the index in a single post.	1000	Integer
commitWithinMs	Commit indexed documents within the specified number of milliseconds.	-1	Integer
index	The index to send records to.	-	Document
softCommit	Perform a soft commit after every batch so that docs are available for searching immediately (if using NRT replicas).	true	Boolean

SplitFilter

Splits multi-record source data into smaller groups of records prior to delivery to an XSLT. This allows the XSLT to process data more efficiently than loading a potentially huge input stream into memory.

Element properties:

Name	Description	Default Value	Value Type
splitCount	The number of elements at the split depth to count before the XML is split.	10000	Integer
splitDepth	The depth of XML elements to split at.	1	Integer
storeLocations	Should this split filter store processing locations.	true	Boolean

StateFilter

Takes XML input (conforming to the reference-data:2 schema) and loads the data into the State Store. Reference data values can be either simple strings or XML fragments.

StatisticsFilter

An element to allow the source data (conforming to the statistics XML Schema) to be sent to the MySQL based statistics data store.

Element properties:

Name	Description	Default Value	Value Type
statisticsDataSource	The statistics data source to record statistics against.	-	Document

StroomStatsFilter

An element to allow the source data (conforming to the statistics XML Schema) to be sent to an external stroom-stats service.

Element properties:

Name	Description	Default Value	Value Type
flushOnSend	At the end of the stream, wait for acknowledgement from the Kafka broker for all the messages sent. This ensures errors are caught in the pipeline process.	true	Boolean
kafkaConfig	The Kafka config to use.	-	Document
statisticsDataSource	The stroom-stats data source to record statistics against.	-	Document

XPathExtractionOutputFilter

TODO - Add description

Element properties:

Name	Description	Default Value	Value Type
multipleValueDelimiter	The string to delimit multiple simple values.	,	String

XSLTFilter

An element used to transform XML data from one form to another using XSLT. The specified XSLT can be used to transform the input XML into XML conforming to another schema or into other forms such as JSON, plain text, etc.

Element properties:

Name	Description	Default Value	Value Type
suppressXSLTNotFoundWarnings	If XSLT cannot be found to match the name pattern suppress warnings.	false	Boolean
usePool	Advanced: Choose whether or not you want to use cached XSLT templates to improve performance.	true	Boolean
xslt	The XSLT to use.	-	Document
xsltNamePattern	A name pattern to load XSLT dynamically.	-	String

Writer

Writers consume XML events (from Parsers and Filters) and convert them into a stream of bytes using the character encoding configured on the Writer (if applicable). The output data can then be fed to a Destination.

JSONWriter

Writer to convert XML data conforming to the http://www.w3.org/2013/XSL/json XML Schema into JSON format.

Element properties:

Name	Description	Default Value	Value Type
encoding	The output character encoding to use.	UTF-8	String
indentOutput	Should output JSON be indented and include new lines (pretty printed)?	false	Boolean

TextWriter

Writer to convert XML character data events into plain text output.

Element properties:

Name	Description	Default Value	Value Type
encoding	The output character encoding to use.	UTF-8	String
footer	Footer text that can be added to the output at the end.	-	String
header	Header text that can be added to the output at the start.	-	String

XMLWriter

Writer to convert XML events data into XML output in the specified character encoding.

Element properties:

Name	Description	Default Value	Value Type
encoding	The output character encoding to use.	UTF-8	String
indentOutput	Should output XML be indented and include new lines (pretty printed)?	false	Boolean
suppressXSLTNotFoundWarnings	If XSLT cannot be found to match the name pattern suppress warnings.	false	Boolean
xslt	A previously saved XSLT, used to modify the output via xsl:output attributes.	-	Document
xsltNamePattern	A name pattern for dynamic loading of an XSLT, that will modify the output via xsl:output attributes.	-	String

Destination

Destination elements consume a stream of bytes from a Writer and persist them to a destination. This could be a file on a file system or to Stroom’s stream store.

AnnotationWriter

Consume XML documents in the annotation:1 namespace and writes them as Stroom Annotations. Allows for the annotating of events that meet some criteria.

FileAppender

A destination used to write an output stream to a file on the file system. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random to write to.

Element properties:

Name	Description	Default Value	Value Type
compressionMethod	Compression method to apply, if compression is enabled. Supported values: bzip2, deflate, gz, lz4-block, lz4-framed, lzma, pack200, snappy-framed, xz, zip, zstd.	gz	String
filePermissions	Set file system permissions of finished files (example: ‘rwxr–r–’)	-	String
outputPaths	One or more destination paths for output files separated with commas. Replacement variables can be used in path strings such as ${feed}.	-	String
rollSize	When the current output file exceeds this size it will be closed and a new one created.	-	String
splitAggregatedStreams	Choose if you want to split aggregated streams into separate output files.	false	Boolean
splitRecords	Choose if you want to split individual records into separate output files.	false	Boolean
useCompression	Apply compression to output files.	false	Boolean

HDFSFileAppender

A destination used to write an output stream to a file on a Hadoop Distributed File System. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random.

Element properties:

Name	Description	Default Value	Value Type
fileSystemUri	URI for the Hadoop Distributed File System (HDFS) to connect to, e.g. hdfs://mynamenode.mydomain.com:8020	-	String
outputPaths	One or more destination paths for output files separated with commas. Replacement variables can be used in path strings such as ${feed}.	-	String
rollSize	When the current output file exceeds this size it will be closed and a new one created.	-	String
runAsUser	The user to connect to HDFS as	-	String
splitAggregatedStreams	Choose if you want to split aggregated streams into separate output files.	false	Boolean
splitRecords	Choose if you want to split individual records into separate output files.	false	Boolean

HTTPAppender

A destination used to write an output stream to a remote HTTP(S) server.

This element should be preferred over the deprecated HttpPostFilter .

Element properties:

Name	Description	Default Value	Value Type
compressionMethod	Compression method to apply, if compression is enabled. Supported values: bzip2, deflate, gz, lz4-block, lz4-framed, lzma, pack200, snappy-framed, xz, zip, zstd.	gz	String
connectionRequestTimeout	Returns the connection lease request timeout used when requesting a connection from the connection manager. Default: 3 minutes. The timeout is specified as either milliseconds, e.g. ‘60000’ or with a duration suffix, e.g. ‘500ms’, ‘2s’, ‘1m’, etc.	-	String
connectionTimeout	Determines the timeout until a new connection is fully established. This may also include transport security negotiation exchanges such as SSL or TLS protocol negotiation. A timeout value of zero is interpreted as an infinite timeout. Default: 3 minutes. The timeout is specified as either milliseconds, e.g. ‘60000’ or with a duration suffix, e.g. ‘500ms’, ‘2s’, ‘1m’, etc.	-	String
contentType	The content type	application/json	String
forwardChunkSize	Should data be sent in chunks and if so how big should the chunks be. Size is either specified in bytes e.g. ‘1024’ or with a IEC unit suffix, e.g. ‘1K’, ‘1M’, ‘1G’, etc.	-	String
forwardUrl	The URL to send data to.	-	String
hostnameVerificationEnabled	Set this to true to verify that the destination host name matches against the host names in the certificate supplied by the destination server.	true	Boolean
httpHeadersIncludeStreamMetaData	Provide stream metadata as HTTP headers	true	Boolean
httpHeadersStreamMetaDataAllowList	Comma delimited list of stream meta data keys to include as HTTP headers. Only works when httpHeadersIncludeStreamMetaData is set to true. If empty all headers are sent, unless httpHeadersStreamMetaDataDenyList is used. If httpHeadersStreamMetaDataAllowList contains keys, httpHeadersStreamMetaDataDenyList is ignored.	-	String
httpHeadersStreamMetaDataDenyList	Comma delimited list of stream meta data keys to exclude as HTTP headers. Only works when httpHeadersIncludeStreamMetaData is set to true. If empty all headers are sent, unless httpHeadersStreamMetaDataAllowList is used. If httpHeadersStreamMetaDataAllowList contains keys, httpHeadersStreamMetaDataDenyList is ignored.	-	String
httpHeadersUserDefinedHeader1	Additional HTTP Header 1, format is ‘HeaderName: HeaderValue’	-	String
httpHeadersUserDefinedHeader2	Additional HTTP Header 2, format is ‘HeaderName: HeaderValue’	-	String
httpHeadersUserDefinedHeader3	Additional HTTP Header 3, format is ‘HeaderName: HeaderValue’	-	String
keyStorePassword	The key store password	-	String
keyStorePath	The key store file path on the server	-	String
keyStoreType	The key store type. Valid values are [‘JCEKS’, ‘JKS’, ‘DKS’, ‘PKCS11’, ‘PKCS12’].	JKS	String
logMetaKeys	Specifies Which meta data keys will have their values logged in the send log. A Comma delimited string of keys.	guid,receiptid,feed,system,environment,remotehost,remoteaddress	String
readTimeout	How long to wait for data to be available before closing the connection. The timeout is specified as either milliseconds, e.g. ‘60000’ or with a duration suffix, e.g. ‘500ms’, ‘2s’, ‘1m’, etc.	-	String
requestMethod	The HTTP request method. Valid values are GET, POST, HEAD, OPTIONS, PUT, DELETE and TRACE.	POST	String
rollSize	When the current output exceeds this size it will be closed and a new one created. Size is either specified in bytes e.g. ‘1024’ or with a IEC unit suffix, e.g. ‘1K’, ‘1M’, ‘1G’, etc.	-	String
splitAggregatedStreams	Choose if you want to split aggregated streams into separate output.	false	Boolean
splitRecords	Choose if you want to split individual records into separate output.	false	Boolean
sslProtocol	The SSL protocol to use	TLSv1.2	String
timeToLive	The maximum time a pooled connection can stay idle (not leased to any thread) before it is shut down. Default: 1 hour. The timeout is specified as either milliseconds, e.g. ‘60000’ or with a duration suffix, e.g. ‘500ms’, ‘2s’, ‘1m’, etc.	-	String
timeout	Determines the timeout until arrival of a response from the opposite endpoint. A timeout value of zero is interpreted as an infinite timeout. Default: 3 minutes. The timeout is specified as either milliseconds, e.g. ‘60000’ or with a duration suffix, e.g. ‘500ms’, ‘2s’, ‘1m’, etc.	-	String
trustStorePassword	The trust store password	-	String
trustStorePath	The trust store file path on the server	-	String
trustStoreType	The trust store type Valid values are [‘JCEKS’, ‘JKS’, ‘DKS’, ‘PKCS11’, ‘PKCS12’].	JKS	String
useCompression	Should data be compressed when sending	true	Boolean
useContentEncodingHeader	Whether to use the ‘Content-Encoding’ HTTP header when useCompression is ’true’. If ‘false’ (the default), the ‘Compression’ header will be used, which is supported by .Stroom/Stroom-Proxy destinations. ‘Content-Encoding’ would be required for other destinations, but is only applicable for compression types ‘gz’, ‘zstd’ or ‘deflate’.	false	Boolean
useJvmSslConfig	Use JVM SSL config. Set this to true if the Stroom node has been configured with key/trust stores using java system properties like ‘javax.net.ssl.keyStore’.Set this to false if you are explicitly setting key/trust store properties on this HttpAppender.	true	Boolean

RollingFileAppender

A destination used to write an output stream to a file on the file system. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random to write to. This is distinct from the FileAppender in that when the rollSize is reached it will move the current file to the path specified in rolledFileName and resume writing to the original path. This allows other processes to follow the changes to a single file path, e.g. when using tail. On system shutdown all active files will be rolled.

Element properties:

Name	Description	Default Value	Value Type
compressionMethod	Compression method to apply, if compression is enabled. Supported values: bzip2, deflate, gz, lz4-block, lz4-framed, lzma, pack200, snappy-framed, xz, zip, zstd.	gz	String
fileName	Choose the name of the file to write.	-	String
filePermissions	Set file system permissions of finished files (example: ‘rwxr–r–’)	-	String
frequency	Choose how frequently files are rolled.	1h	String
outputPaths	One or more destination paths for output files separated with commas. Replacement variables can be used in path strings such as ${feed}.	-	String
rollSize	When the current output file exceeds this size it will be closed and a new one created, e.g. 10M, 1G.	100M	String
rolledFileName	Choose the name that files will be renamed to when they are rolled.	-	String
schedule	Provide a cron expression to determine when files are rolled.	-	String
useCompression	Apply GZIP compression to output files	false	Boolean

RollingStreamAppender

A destination used to write one or more output streams to a new stream which is then rolled when it reaches a certain size or age. A new stream will be created after the size or age criteria has been met. On system shutdown all active streams will be rolled.

Element properties:

Name	Description	Default Value	Value Type
feed	The feed that output stream should be written to. If not specified the feed the input stream belongs to will be used.	-	Document
frequency	Choose how frequently streams are rolled.	1h	String
rollSize	Choose the maximum size that a stream can be before it is rolled.	100M	String
schedule	Provide a cron expression to determine when streams are rolled.	-	String
segmentOutput	Should the output stream be marked with indexed segments to allow fast access to individual records?	true	Boolean
streamType	The stream type that the output stream should be written as. This must be specified.	-	String
volumeGroup	Optionally override the default volume group of the destination feed.	-	String

S3Appender

A destination used to write an output stream to an S3 bucket.

Element properties:

Name	Description	Default Value	Value Type
bucketNamePattern	Set the bucket name pattern if you want to override the one provided by the S3 config.	-	String
compressionMethod	Compression method to apply, if compression is enabled. Supported values: bzip2, deflate, gz, lz4-block, lz4-framed, lzma, pack200, snappy-framed, xz, zip, zstd.	gz	String
keyNamePattern	Set the key name pattern if you want to override the one provided by the S3 config.	${type}/${year}/${month}/${day}/${idPath}/${feed}/${idPadded}.gz	String
rollSize	When the current output object exceeds this size it will be closed and a new one created.	-	String
s3Config	The S3 bucket config to use.	-	Document
splitAggregatedStreams	Choose if you want to split aggregated streams into separate output objects.	false	Boolean
splitRecords	Choose if you want to split individual records into separate output objects.	false	Boolean
useCompression	Apply compression to output objects.	true	Boolean

StandardKafkaProducer

Consumes XML documents in the kafka-records:2 namespace. For each <kafkaRecord> element converts it into a Kafka message that is passed to the Kafka producer defined by the kafkaConfig property

Element properties:

Name	Description	Default Value	Value Type
flushOnSend	At the end of the stream, wait for acknowledgement from the Kafka broker for all the messages sent. This ensures errors are caught in the pipeline process.	true	Boolean
kafkaConfig	Kafka configuration details relating to where and how to send Kafka messages.	-	Document

StreamAppender

A destination used to write the output stream to a new stream in the stream store. The configuration allows for starting a new stream once a size threshold is reached.

Element properties:

Name	Description	Default Value	Value Type
feed	The feed that output stream should be written to. If not specified the feed the input stream belongs to will be used.	-	Document
rollSize	When the current output stream exceeds this size it will be closed and a new one created.	-	String
segmentOutput	Should the output stream be marked with indexed segments to allow fast access to individual records?	true	Boolean
splitAggregatedStreams	Choose if you want to split aggregated streams into separate output streams.	false	Boolean
splitRecords	Choose if you want to split individual records into separate output streams.	false	Boolean
streamType	The stream type that the output stream should be written as. This must be specified.	-	String
volumeGroup	Optionally override the default volume group of the destination feed.	-	String

StroomStatsAppender

This element is deprecated and should not be used.

Element properties:

Name	Description	Default Value	Value Type
flushOnSend	At the end of the stream, wait for acknowledgement from the Kafka broker for all the messages sent. This ensures errors are caught in the pipeline process.	true	Boolean
kafkaConfig	The Kafka config to use.	-	Document
maxRecordCount	Choose the maximum number of records or events that a message will contain	1	String
statisticsDataSource	The stroom-stats data source to record statistics against.	-	Document

Docs: Lucene Indexes

Tue, 15 Feb 2022 00:00:00 +0000

Stroom uses Apache Lucene for its built-in indexing solution. Index documents are stored in a Volume Volume In Stroom a Volume is a logical storage area that Stroom can write data to. Volumes are associated with a path on a file system that can either be local to the Stroom node or on a shared file system.Click to see more details....

TODO

Complete this page.

Field configuration

Field Types

Id - Treated as a Long.
Boolean - True/False values.
Integer - Whole numbers from -2,147,483,648 to 2,147,483,647.
Long - Whole numbers from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.
Float - Fractional numbers. Sufficient for storing 6 to 7 decimal digits.
Double - Fractional numbers. Sufficient for storing 15 decimal digits.
Date - Date and time values.
Text - Text data.
Number - An alias for Long.

Stored fields

If a field is Stored then it means the complete field value will be stored in the index. This means the value can be retrieved from the index when building search results rather than using the slower Search Extraction process. Storing field values comes at the cost of high storage requirements for the index. If storage space is not an issue then storing all fields that you want to return in search results is the optimum.

Indexed fields

An Indexed field is one that will be processed by Lucene so that the field can be queried. How the field is indexed will depend on the Field type and the Analyser used.

If you have fields that you do not want to be able to filter (i.e. that you won’t use as a query term) then you can include them as non-Indexed fields. Including a non-indexed field means it will be available for the user to select in the Dashboard Dashboard A Dashboard is a configurable entity for querying one or more Data Sources and displaying the results as a table, a visualisation or some other form.Click to see more details... table. A non-indexed field would either need to be Stored in the index or added via Search Extraction to be available in the search results.

Positions

If Positions is selected then Lucene will store the positions of all the field terms in the document.

Analyser types

The Analyser determines how Lucene reads the fields value and extracts tokens from it. The choice of Analyser will depend on the data in the field and how you want to search it.

Keyword - Treats the whole field value as one token. Useful for things like IDs and post codes. Supports the Case Sensitivity setting.
Alpha - Tokenises on any non-letter characters, e.g. one1 two2 three 3 => one two three. Strips non-letter characters. Supports the Case Sensitivity setting.
Numeric -
Alpha numeric - Tokenises on any non-letter/digit characters, e.g. one1 two2 three 3 => one1 two2 three 3. Supports the Case Sensitivity setting.
Whitespace - Tokenises only on white space. Not affected by the Case Sensitivity setting, case sensitive.
Stop words - Tokenises based on non-letter characters and removes Stop Words, e.g. and. Not affected by the Case Sensitivity setting. Case insensitive.
Standard - The most common analyser. Tokenises the value on spaces and punctuation but recognises URLs and email addresses. Removes Stop Words, e.g. and. Not affected by the Case Sensitivity setting. Case insensitive. e.g. Find Stroom at github.com/stroom => Find Stroom at github.com/stroom.

Stop words

Some of the Analysers use a set of stop words for the tokenisers. This is the list of stop words that will not be indexed.

a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will, with

Case sensitivity

Some of the Analyser types support case (in)sensitivity. For example if the Analyser supports it the value TWO two would either be tokenised as TWO two or two two.

Docs: Solr Integration

Tue, 15 Feb 2022 00:00:00 +0000

TODO

Complete this section.

Docs: MySQL Configuration

Mon, 07 Jun 2021 00:00:00 +0000

General configuration

MySQL is configured via the .cnf file which is typically located in one of these locations:

/etc/my.cnf
/etc/mysql/my.cnf
$MYSQL_HOME/my.cnf
<data dir>/my.cnf
~/.my.cnf

Key configuration properties

lower_case_table_names - This property controls how the tables are stored on the filesystem and the case-sensitivity of table names in SQL. A value of 0 means tables are stored on the filesystem in the case used in CREATE TABLE and sql is case sensitive. This is the default in linux and is the preferred value for deployments of stroom of v7+. A value of 1 means tables are stored on the filesystem in lowercase but sql is case insensitive. See also Identifier Case Sensitivity
max_connections - The maximum permitted number of simultaneous client connections. For a clustered deployment of stroom, the default value of 151 will typically be too low. Each stroom node will hold a pool of open database connections for its use, therefore with a large number of stroom nodes and a big connection pool the total number of connections can be very large. This property should be set taking into account the values of the stroom properties of the form *.db.connectionPool.maxPoolSize. See also Connection Interfaces
innodb_buffer_pool_size/innodb_buffer_pool_instances - Controls the amount of memory available to MySQL for caching table/index data. Typically this will be set to 80% of available RAM, assuming MySQL is running on a dedicated host and the total amount of table/index data is greater than 80% of available RAM. Note: innodb_buffer_pool_size must be set to a value that is equal to or a multiple of innodb_buffer_pool_chunk_size * innodb_buffer_pool_instances. See also Configuring InnoDB Buffer Pool Size

TODO

Add additional key configuration items

Deploying without Docker

When MySQL is deployed without a docker stack then MySQL should be installed and configured according to the MySQL documentation. How MySQL is deployed and configured will depend on the requirements of the environment, e.g. clustered, primary/standby, etc.

As part of a docker stack

Where a stroom docker stack includes stroom-all-dbs (MySQL) the MySQL instance is configured via the .cnf file. The .cnf file is located in volumes/stroom-all-dbs/conf/stroom-all-dbs.cnf. This file is read-only to the container and will be read on container start.

Database initialisation

When the container is started for the first time the database will be initialised with the root user account. It will also then run any scripts found in volumes/stroom-all-dbs/init/stroom. The scripts in here will be run in alphabetical order. Scripts of the form .sh, .sql, .sql.gz and .sql.template are supported.

.sql.template files are proprietary to stroom stacks and are just templated .sql files. They can contain tags of the form <<<ENV_VAR_NAME>>> which will be replaced with the value of the named environment variable that has been set in the container.

If you need to add additional database users then either add them to volumes/stroom-all-dbs/init/stroom/001_create_databases.sql.template or create additional scripts/templates in that directory.

The script that controls this templating is volumes/stroom-all-dbs/init/000_stroom_init.sh. This script MUST not have its executable bit set else it will be executed rather than being sourced by the MySQL entry point scripts and will then not work.

Stroom – TODO

Docs: Single Node Docker Installation

Note

Stroom Docker stacks

Prerequisites

Note

Install steps

Note

Configuration

Docker Hub links

Docs: MySQL Setup

TODO

Prerequisites

Check Database installed and running

Overview

Simple Install

Advanced Security

Advanced Install

Docs: Stroom Proxy Installation

Note

Typical Deployments

Note

Assumptions

Firewall Configuration

Stroom Proxy (docker version)

Download and install docker

Download and Install Docker Stack

To Start Stroom Proxy

Stroom Proxy (app version)

Note

Download and install Stroom v7 (app version)

Docs: Securing Stroom

Firewall

MySQL

TODO

Docs: Naming Conventions

See Also

TODO

Community: Developing Content

TODO

Docs: Java Key Store Setup

TODO

Docs: Dictionaries

Creating

Using the Dictionary

Docs: Stroom 6 Installation

TODO

Running on a single box

Running a release

Admin Account creation

Post-install hardening

Before first run

Change database passwords

On first run

Create yourself an account

Delete un-used users and API keys

Change the API keys

Troubleshooting

I’m trying to use certificate logins (PKI) but I keep being prompted for the username and password!

Docs: Nodes

Configuring nodes

TODO

Docs: Pipeline Elements

Reader

BOMRemovalFilterInput

BadTextXMLFilterReader

FindReplaceFilter

InvalidCharFilterReader

InvalidXMLCharFilterReader

Reader

Parser

CombinedParser

Warning

DSParser

JSONParser

XMLFragmentParser

XMLParser

Filter

DynamicIndexingFilter

DynamicSearchResultOutputFilter