The page that you are currently viewing is for an old version of Stroom (7.0). The documentation for the latest version of Stroom (7.8) can be found using the version drop-down at the top of the screen or by clicking here.
Testing Stroom Installation
Assumptions
- Stroom Single or Multi Node Cluster Testing
- the Multi Node Stroom Cluster (Proxy and Application) has been deployed
- a Test Feed,
TEST-FEED-V1_0has been added - Proxy aggregation has been turned off on all Stroom Store Proxies
- the Stroom Proxy Repository Format (
REPO_FORMAT) chosen was the default -${pathId}/${id - Stroom Forwarding Proxy Testing
- the Multi Node Stroom Cluster (Proxy and Application) has been deployed
- the Stroom Forwarding Proxy has been deployed
- a Test Feed,
TEST-FEED-V1_0has been added - the Stroom Proxy Repository Format (
REPO_FORMAT) chosen was the default -${pathId}/${id - Stroom Standalone Proxy Testing
- the Stroom Standalone Proxy has been deployed
- the Stroom Proxy Repository Format (
REPO_FORMAT) chosen was the default -${pathId}/${id
Stroom Single or Multi Node Cluster Testing
Data Post Tests
Simple Post tests
These tests are to ensure the Stroom Store proxy and it’s connection to the database is working along with the Apache mod_jk loadbalancer.
We will send a file to the load balanced stroomp.strmdev00.org node (really stroomp00.strmdev00.org) and each time we send the file,
it’s receipt should be managed by alternate proxy nodes. As a number of elements can effect load balancing, it is not always guaranteed
to alternate every time but for the most part it will.
Perform the following
- Log onto the Stroom database node (stroomdb0.strmdev00.org) as any user.
- Log onto both Stroom nodes and become the
stroomuserand monitor each node’s Stroom proxy service using theTpbash macro. That is, on each node, run
You will note events of the form from
stroomp00.strmdev00.org:
and from stroomp01.strmdev00.org:
- On the Stroom database node, execute the command
If you are monitoring the proxy log of stroomp00.strmdev00.org you would see two new logs indicating the successful arrival of the file
- On the Stroom database node, again execute the command
If you are monitoring the proxy log of stroomp01.strmdev00.org you should see a new log. As foreshadowed, we didn’t as the time delay resulted
in the first node getting the file. That is stroomp00.strmdev00.org log file gained the two entries
- Again on the database node, execute the command and this time we see that node
stroomp01.strmdev00.orgreceived the file as per
- Running the curl post command in quick succession shows the loadbalancer working … four executions result in seeing our pair of logs appearing on alternate proxies.
stroomp00:
stroomp01:
stroomp00:
stroomp01:
At this point we will see what the proxies have received.
- On each node run the command
On stroomp00 we see
and on stroomp01 we see
which corresponds to the seven posts of data and the associated events in the proxy logs. To see the contents of one of these files we execute on either node, the command
to see
Checking the /etc/group file on stroomdb0.strmdev00.org confirms the above contents. For the present, ignore
the metadata file present in the zip archive.
If you execute the same command on the other files, all that changes is the value of the ReceivedTime: attribute in the .meta file.
For those curious about the file size differences, this is a function of the compression process within the proxy.
Using stroomp01’s files and extracting them manually and renaming them results in the six files
We have effectively tested the receipt of our data and the load balancing of the Apache mod_jk installation.
Simple Direct Post tests
In this test we will use the direct feed interface of the Stroom application, rather than sending data via the proxy. One would normally use this interface for time sensitive data which shouldn’t aggregate in a proxy waiting for the Stroom application to collect it. In this situation we use the command
To prepare for this test, we monitor the Stroom application log using the T bash alias on each node. So on each node run the command
On each node you should see LifecyleTask events, for example,
To perform the test, on the database node, run the posting command a number of times in rapid succession. This will result in server.DataFeedServiceImpl events in both log files. The Stroom application log is quite busy, you may have to look for these logs.
In the following we needed to execute the posting command three times before seeing the data arrive on both nodes. Looking at the arrival
times, the file turned up on the second node twice before appearing on the first node.
strooomp00:
and on stroomp01:
To confirm this data arrived, we need to view the Data pane of our TEST-FEED-V1_0 tab. To do this, log onto the Stroom UI then
move the cursor to the TEST-FEED-V1_0 entry in the Explorer tab and select the item with a left click
and double click on the entry to see our TEST-FEED-V1_0 tab.
and it is noted that we are viewing the Feed’s attributes as we can see the Setting hyper-link highlighted. As we want to see the Data we have received for this feed, move the cursor to the Data hyper-link and select it to see .
These three entries correspond to the three posts we performed.
We have successfully tested direct posting to a Stroom feed and that the Apache mod_jk loadbalancer also works for this posting method.
Test Proxy Aggregation is Working
To test that the Proxy Aggregation is working, we need to enable on each node.
By enabling the Proxy Aggregation process, both nodes immediately performed the task as indicated by each node’s Stroom application logs as per
stroomp00:
and stroomp01:
And on refreshing the top pane of the TEST-FEED-V1_0 tab we see that two more batches of data have arrived.
.
This demonstrates that Proxy Aggregation is working.
Stroom Forwarding Proxy Testing
Data Post Tests
Simple Post tests
This test is to ensure the Stroom Forwarding proxy and it’s connection to the central Stroom Processing system is working.
We will send a file to our Forwarding proxy (stroomfp0.strmdev00.org) and monitor this nodes’ proxy log files as well as all the
destination nodes proxy log files. The reason for monitoring all the destination system’s proxy log files is that the destination system is
probably load balancing and hence the forwarded file may turn up on any of the destination nodes.
Perform the following
- Log onto any host where you will perform the
curlpost - Monitor all proxy log files
- Log onto the Forwarding Proxy node and become the
stroomuserand monitor the Stroom proxy service using theTpbash macro. - Log onto the destination Stroom nodes and become the
stroomuserand monitor each node’s Stroom proxy service using theTpbash macro. That is, on each node, run
- On the ‘posting’ node, run the command
In the Stroom Forwarding proxy log, ~/stroom-proxy/instance/logs/stroom.log, you will see the arrival of the
file as per the datafeed.DataFeedRequestHandler$1 event running under, in this case, the ajp-apr-9009-exec-1 thread.
And then at the next
periodic interval (60 second intervals) this file will be forwarded to the main stroom proxy
server stroomp.strmdev00.org as shown by the handler.ForwardRequestHandler events running under the pool-10-thread-2 thread.
On one of the central processing nodes, when the file is send by the Forwarding Proxy, you will see the file’s arrival as per the datafeed.DataFeedRequestHandler$1 event in the ajp-apr-9009-exec-3 thread.
Stroom Standalone Proxy Testing
Data Post Tests
Simple Post tests
This test is to ensure the Stroom Store NODB or Standalone proxy is working.
We will send a file to our Standalone proxy (stroomsap0.strmdev00.org) and monitor this nodes’ proxy log files as well the directory the
received files are meant to be stored in.
Perform the following
- Log onto any host where you will perform the
curlpost - Log onto the Standalone Proxy node and become the
stroomuserand monitor the Stroom proxy service using theTpbash macro. That is run
- On the ‘posting’ node, run the command
In the stroom proxy log, ~/stroom-proxy/instance/logs/stroom.log, you will see the arrival of the file via both the handler.LogRequestHandler and datafeed.DataFeedRequestHandler$1 events running under, in this case, the ajp-apr-9009-exec-1 thread.
Further, if you check the proxy’s storage directory, you will see the file 001.zip. The file names number upwards from 001.
shows
On viewing the contents of this file we see both a .dat and .meta file.
The .dat file holds the content of the file we posted - /etc/group.
The .meta file is generated by the proxy and holds information about the posted file