Docs: Volumes

Tue, 27 Jul 2021 00:00:00 +0000

TODO

Describe volumes

Docs: Volume Maintenance

Mon, 12 Jul 2021 00:00:00 +0000

Stroom stores data in volumes. These are the logical link to the Storage hierarchy we setup on the operating system. This HOWTO will demonstrate how one first sets up volumes and also how to add additional volumes if one expanded an existing Stroom cluster.

Assumptions

an account with the Administrator Application Permission is currently logged in.
we will add volumes as per the Multi Node Stroom deployment Storage hierarchy

Configure the Volumes

We need to configure the volumes for Stroom. The follow demonstrates adding the volumes for two nodes, but demonstrates the process for a single node deployment as well the volume maintenance needed when expanding a Multi Node Cluster when adding in a new node.

To configure the volumes, move to the Tools item of the Main Menu and select it to bring up the Tools sub-menu.

Stroom UI Tools sub-menu

then move down and select the Volumes sub-item to be presented with the Volumes configuration window as seen below.

Stroom UI Volumes - configuration window

The attributes we see for each volume are

Node - the processing node the volume resides on (this is just the node name entered when configuring the Stroom application)
Path - the path to the volume
Volume Type - The type of volume
Public - to indicate that all nodes would access this volume
Private - to indicate that only the local node will access this volume
Stream Status
Active - to store data within the volume
Inactive - to NOT store data within the volume
Closed - had stored data within the volume, but now no more data can be stored
Index Status
Active - to store index data within the volume
Inactive - to NOT store index data within the volume
Closed - had stored index data within the volume, but now no more index data can be stored
Usage Date - the date and time the volume was last used
Limit - the maximum amount of data the system will store on the volume
Used - the amount of data in use on the volume
Free - the amount of available storage on the volume
Use% - the usage percentage

If you are setting up Stroom for the first time and you had accepted the default for the CREATE_DEFAULT_VOLUME_ON_START configuration option (true) when configuring the Stroom service application, you will see two default volumes have already been created. Had you set this option to false then the window would be empty.

Add Volumes

Now from our two node Stroom Cluster example, our storage hierarchy was

Node: stroomp00.strmdev00.org
/stroomdata/stroom-data-p00 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p00 - location to store Stroom application index files
/stroomdata/stroom-working-p00 - location to store Stroom application working files (e.g. temporary files, output, etc.) for this node
/stroomdata/stroom-working-p00/proxy - location for Stroom proxy to store inbound data files
Node: stroomp01.strmdev00.org
/stroomdata/stroom-data-p01 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p01 - location to store Stroom application index files
/stroomdata/stroom-working-p01 - location to store Stroom application working files (e.g. temporary files, output, etc.) for this node
/stroomdata/stroom-working-p01/proxy - location for Stroom proxy to store inbound data files

From this we need to create four volumes. On stroomp00.strmdev00.org we create

/stroomdata/stroom-data-p00 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p00 - location to store Stroom application index files

and on stroomp01.strmdev00.org we create

/stroomdata/stroom-data-p01 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p01 - location to store Stroom application index files

So the first step to configure a volume is to move the cursor to the New icon in the top left of the Volumes window and select it. This will bring up the Add Volume configuration window

Stroom UI Add Volume - Volume configuration window

As you can see, the entry box titles reflect the attributes of a volume. So we will add the first nodes data volume

/stroomdata/stroom-data-p00 - location to store Stroom application data files (events, etc.) for this node for node stroomp00.

If you move the Node drop down entry box and select it you will be presented with a choice of available nodes - in this case stroomp00 and stroomp01 as we have a two node cluster with these node names.

Stroom UI Add Volume - select node

By selecting the node stroomp00 we see

Stroom UI Add Volume - selected node

To configure the rest of the attributes for this volume, we:

enter the Path to our first node’s data volume
select a Volume Type of Public as this is a data volume we want all nodes to access
select a Stream Status of Active to indicate we want to store data on it
select an Index Status of Inactive as we do NOT want index data stored on it
set a Limit of 12GB for allowed storage

Stroom UI Add Volume - adding first data volume

and on selection of the OK we see the changes in the Volumes configuration window

Stroom UI Add Volume - added first data volume

We next add the first node’s index volume, as per

Stroom UI Add Volume - adding first index volume

And after adding the second node’s volumes we are finally presented with our configured volumes

Stroom UI Add Volume - all volumes added

Delete Default Volumes

We now need to deal with our default volumes. We want to delete them.

Stroom UI Delete Default - display default

So we move the cursor to the first volume’s line (stroomp00 /home/stroomuser/stroom-app/volumes/defaultindexVolume …) and select the line then move the cursor to the Delete icon in the top left of the Volumes window and select it. On selection you will be given a confirmation request

Stroom UI Delete Default - confirm deletion

at which we press the OK button to see the first default volume has been deleted

Stroom UI Delete Default - first volume deleted

and after we select then delete the second default volume(stroomp00 /home/stroomuser/stroom-app/volumes/defaultStreamVolume …), we are left with

Stroom UI Delete Default - all deleted

At this point, one can close the Volumes configuration window by pressing the Close button.

NOTE: At the time of writing there is an issue regarding volumes

Stroom Github Issue 84 -

Due to Issue 84 , if we delete volumes in a multi node environment, the deletion is not propagated to all other nodes in a cluster. Thus if we attempted to use the volumes we would get a database error. The current workaround is to restart all the Stroom applications which will cause a reload of all volume information. This MUST be done before sending any data to your multi-node Stroom cluster.

Adding new Volumes

When one expands a Multi Node Stroom cluster deployment, after the installation of the Stroom Proxy and Application software and services on the new node, one has to configure the new volumes that are on the new node. The following demonstrates this assuming we are adding

the new node is stroomp02
the storage hierarchy for this node is
/stroomdata/stroom-data-p02 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p02 - location to store Stroom application index files
/stroomdata/stroom-working-p02 - location to store Stroom application working files (e.g. tmp, output, etc.) for this node
/stroomdata/stroom-working-p02/proxy - location for Stroom proxy to store inbound data files

From this we need to create two volumes on stroomp02

/stroomdata/stroom-data-p02 - location to store Stroom application data files (events, etc.) for this node
/stroomdata/stroom-index-p02 - location to store Stroom application index files

To configure the volumes, move to the Tools item of the Main Menu and select it to bring up the Tools sub-menu.

Stroom UI Tools sub-menu

then move down and select the Volumes sub-item to be presented with the Volumes configuration window as seen below. We then move the cursor to the New icon in the top left of the Volumes window and select it. This will bring up the Add Volume configuration window where we select our volume’s node stroomp02.

Stroom UI Volumes - New Node configuration window start data volume

We select this node and then configure the rest of the attributes for this data volume

Stroom UI Volumes - New Node configuration window data volume

then press the title button.

We then add another volume for the index volume for this node with attributes as per

Stroom UI Volumes - New Node configuration window index volume added

And on pressing the OK button we see our two new volumes for this node have been added.

Stroom UI Volumes - New Node configuration window volumes added