This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Stroom Proxy

Stroom Proxy acts as a proxy when sending data to a Stroom instance. Stroom Proxy has various modes such as storing, aggregating and forwarding the received data. Stroom Proxies can be used to forward to other Stroom Proxy instances.

1 - Apache Forwarding

Warning

This document refers to v5.

Stroom Proxy defaults to listening for HTTP on port 9080. It is recommended that Apache is used to listen on the standard HTTP port 80 and forward requests on via the Apache mod_jk module and the AJP protocol (on 9009). Apache can also perform HTTPS on port 443 and pass over requests to Tomcat using the same AJP protocol.

It is additionally recommended that Stroom Proxy is used to front data ingest and so Apache is configured to route traffic to http(s)://server/stroom/datafeed to Stroom Proxy.

Prerequisites

  • tomcat-connectors-1.2.31-src.tar.gz

Setup Apache

  • As root
  • Patch mod_jk
cd ~/tmp
tar -xvzf tomcat-connectors-1.2.31-src.tar.gz 
cd tomcat-connectors-1.2.31-src/native
./configure --with-apxs=/usr/sbin/apxs 
make
sudo cp apache-2.0/mod_jk.so /etc/httpd/modules/
cd
  • Put the web server cert, private key, and CA cert into the web servers conf directory /etc/httpd/conf. E.g.
[user@node1 stroom-doc]$ ls -al /etc/httpd/conf
....
-rw-r--r-- 1 root root  1729 Aug 27  2013 host.crt
-rw-r--r-- 1 root root  1675 Aug 27  2013 host.key
-rw-r--r-- 1 root root  1289 Aug 27  2013 CA.crt
....
  • Make changes to /etc/http/conf.d/ssl.conf as per below
JkMount /stroom/datafeed* loadbalancer_proxy
JkMount /stroom* loadbalancer_proxy

JkOptions +ForwardKeySize +ForwardURICompat +ForwardSSLCertChain -ForwardDirectories

SSLCertificateFile /etc/httpd/conf/[YOUR SERVER].crt
SSLCertificateKeyFile /etc/httpd/conf/[YOUR SERVER].key
SSLCertificateChainFile /etc/httpd/conf/[YOUR CA].crt
SSLCACertificateFile /etc/httpd/conf/[YOUR CA APPENDED LIST].crt
  • Remove /etc/httpd/conf.d/nss.conf to avoid a 8443 port clash
rm /etc/httpd/conf.d/nss.conf
  • Create a /etc/httpd/conf.d/mod_jk.conf configuration
LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"
JkOptions +ForwardKeySize +ForwardURICompat +ForwardSSLCertChain -ForwardDirectories
JkRequestLogFormat "%w %V %T"

JkMount /stroom/datafeed* loadbalancer_proxy
JkMount /stroom* loadbalancer_proxy

JkShmFile logs/jk.shm
<Location /jkstatus/>
    JkMount status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Location>
  • Setup stroom-setup/cluster.txt, generate the workers file and copy into Apache. (as root and replace stroomuser with your processing user)
/home/stroomuser/stroom-setup/workers.properties.sh --cluster=/home/stroomuser/cluster.txt > /etc/httpd/conf/workers.properties
  • Inspect /etc/httpd/conf/workers.properties to make sure it looks as you expect for your cluster text
worker.list=loadbalancer_proxy,local_proxy
worker.stroom_1_proxy.port=9009
worker.stroom_1_proxy.host=localhost
worker.stroom_1_proxy.type=ajp13
worker.stroom_1_proxy.lbfactor=1
worker.stroom_1_proxy.max_packet_size=65536
....
....
worker.loadbalancer_proxy.type=lb
worker.loadbalancer_proxy.balance_workers=stroom_1_proxy,stroom_2_proxy
worker.loadbalancer_proxy.sticky_session=1
worker.local_proxy.type=lb
worker.local_proxy.balance_workers=stroom_1_proxy
worker.local_proxy.sticky_session=1
  • Create a simple redirect page to the stroom web app for the root URL (e.g. DocumentRoot “/var/www/html”, index.html)
&lt;html&gt;&lt;head&gt;&lt;meta http-equiv="Refresh" content="0; URL=stroom"&gt;&lt;/head&gt;&lt;/html&gt;
  • Restart Apache and then test default http / https access.
sudo /etc/init.d/httpd restart

2 - Running with docker

Clone and build stroom-proxy

git clone https://github.com/gchq/stroom-proxy.git
mvn clean install

Unpack the stroom-proxy distribution in preparation for building the docker iamge

cd stroom-proxy-distribution
unzip target/stroom-proxy-distribution-<version>-bin.zip -d target

Building and running the docker image

Here you need to say where you want data to be sent by stroom-proxy. This is done using a --build-arg (external link) parameter of STROOM_PROXY_TYPE. These values can be forward, store, or store_nodb. See the here for more details about these options.

docker stop stroom-proxy
docker rm stroom-proxy
docker rmi stroom-proxy

docker build --build-arg STROOM_PROXY_TYPE=store_nodb --tag=stroom-proxy:latest target/stroom-proxy
docker run -p 8080:8080 --name=stroom-proxy stroom-proxy

3 - Stroom Proxy Installation

Warning

This document refers to v5.

Prerequisites

  • Linux Server’s with at least 4GB RAM
  • Install files stroom-proxy-X-Y-Z-distribution.zip, stroom-deploy-X-Y-Z-distribution.zip
  • Temporarily allow port 9080 if not relying on Apache Forwarding (see below)

Processing User Setup

See Processing User Setup.

Installing Stroom Proxy

As the processing user unpack the stroom-proxy-X-Y-Z-distribution.zip installation files in the processing users home directory.

unzip stroom-proxy-X-Y-Z-distribution.zip

Stroom Proxy can be setup as follows:

  • forward - as an aggregation point to store and forwarding onto another Stroom or Stroom / Proxy
  • store - to front Stroom for data ingest

Stroom Proxy - forward

In forward mode you need to know the server address that data is being sent onto. Run the setup script to capture the basic settings required to run Stroom Proxy in forward mode.

  • @@ NODE @@ - Each Stroom instance in the cluster needs a unique name, if this is a reinstall ensure you use the previous deployment. This name needs match the name used in your worker.properties (e.g. ’node1’ in the case ’node1.my.org’)
  • @@ PORT PREFIX @@ - By default Stroom Proxy will run on port 9080
[stroomuser@node1 ~]$ ./stroom-proxy/bin/setup.sh forward

[stroomuser@dev1 ~]$ ./stroom-proxy/bin/setup.sh forward





...

Parameters
==========

@@NODE@@           : Unique node name for install [node1                                                     ] : node1
@@PORT_PREFIX@@    : HTTP prefix to use           [90                                                        ] : 90
@@REPO_DIR@@       : Stroom Proxy Repository Dir     [/stroomdata/stroom-proxy                                        ] : /home/stroomuser/stroom-proxy-repo
@@FORWARD_SERVER@@ : Server to forward data on to [hostname                                                  ] : audit.my.org
@@JAVA_OPTS@@      : Optional tomcat JVM settings ["-Xms512m -Xmx1g"                                         ] :

...

Stroom Proxy - store

In store mode you need to know the mysql details to validate incoming data with.

[stroomuser@node1 ~]$ ./stroom-proxy-app/bin/setup.sh store

...

@@NODE@@           : Unique node name for install            [node                                                      ] :
@@PORT_PREFIX@@    : HTTP prefix to use                      [90                                                        ] : 72
@@REPO_DIR@@       : Stroom Proxy Repository Dir                [/stroomdata/stroom-proxy                                        ] : /home/stroomuser/stroom-proxy-repo-2
@@JDBC_CLASSNAME@@ : JDBC class name                         [com.mysql.jdbc.Driver                                     ] :
@@JDBC_URL@@       : JDBC URL (jdbc:mysql://[HOST]/[DBNAME]) [jdbc:mysql://localhost/stroom                                ] :
@@DB_USERNAME@@    : Database username                       [                                                          ] : stroomuser
@@DB_PASSWORD@@    : Database password                       [                                                          ] :
@@JAVA_OPTS@@      : Optional tomcat JVM settings            ["-Xms512m -Xmx1g"                                         ] :

Install Check

Start the installed instance:

./stroom-deploy/start.sh

Inspect the logs:

tail -f stroom-proxy-app/instance/logs/stroom.log

Stroom Proxy Properties

The following properties can be configured in the stroom.properties file.

TODO - Could do with a column indicating which proxy mode these properties apply to, e.g. store/forward

Property Name Description
repoDir Optional Repository DIR. If set any incoming request will be written to the file system.
logRequest Optional log line with header attributes output as defined by this property
bufferSize Override default (8192) JDK buffer size to use
forwardUrl Optional The URL’s to forward onto This is pass-through mode if repoDir is not set
forwardThreadCount Number of threads to forward with
forwardTimeoutMs Time out when forwarding
forwardChunkSize Chunk size to use over http(s) if not set requires buffer to be fully loaded into memory
rollCron Interval to roll any writing repositories.
readCron Cron style interval (e.g. every hour ‘0 * *’, every half hour ‘0,30 * *’) to read any ready repositories (if not defined we read all the time).
maxAggregation Aggregate size to break at when building an aggregate.
zipFilenameDelimiter The delimiter used to separate the id ihe zip filename from the templated part
zipFilenameTemplate A template for naming the zip files in the repository where files will be named nnn!zipFilenameTemplate.zip where nnn is the id prefix, ! is the configurable delimiter and zipFilenameTemplate will be something like ‘${feed}!${headerMapKey1}!${headerMapKey2}’. The naem of each variable must exactly match a key in the meta data else it will resolve to ‘’.
requestDelayMs Sleep time used to aid with testing
forwardDelayMs Debug setting to add a delay
dbRequestValidatorContext Database Feed Validator - Data base JDBC context
dbRequestValidatorJndiName Database Feed Validator - Data base JDBC JNDI name
dbRequestValidatorFeedQuery Database Feed Validator - SQL to check feed status
dbRequestValidatorAuthQuery Database Feed Validator - SQL to check authorisation required
remotingUrl Url to use for remoting services
remotingReadTimeoutMs Change from the default JVM settings.
remotingConnectTimeoutMs Change from the default JVM settings.
maxStreamSize Stream size to break at when building an aggregate.
maxFileScan Max number of files to scan over during forwarding. Once this limit is it it will wait until next read interval
cacheTimeToIdleSeconds Time to idle settings to used for validating feed information
cacheTimeToLiveSeconds Time to live settings to used for validating feed information

Apache Forwarding

See Apache Forwarding.

Java Key Store Setup

If you require that Stroom Proxy communicates over 2-way https you will need to set up Java Key Stores.

See Java Key Store Setup.

Securing Stroom

See Securing Stroom.