Docs: Auto Scaler

Fri, 04 Mar 2022 00:00:00 +0000

Motivation

Setting optimal Stroom stream processor task limits is a crucial factor in running a healthy, performant cluster. If a node is allocated too many tasks, it may become unresponsive or crash. Conversely, if allocated too few tasks, it may have CPU cycles to spare.

The optimal number of tasks is often time-dependent, as load will usually fluctuate during the day and night. In large deployments, it’s not ideal to set static limits, as doing so risks over-committing nodes during intense spikes in activity (such as backlog processing or multiple concurrent searches). Therefore an automated solution, factoring in system load, is called for.

Stroom task autoscaling

When a StroomTaskAutoscaler resource is deployed to a linked StroomCluster, the Operator will periodically compare each Stroom node’s average Pod CPU usage against user-defined thresholds.

Enabling autoscaling

Create an `StroomTaskAutoscaler` resource manifest

Use the example autoscaler.yaml .

Below is an explanation of some of the main parameters. The rest are documented here .

adjustmentIntervalMins Determines how often the Operator will check whether a node has exceeded its CPU parameters. It should be often enough to catch brief load spikes, but not too often as to overload the Operator and Kubernetes cluster through excessive API calls and other overhead.
metricsSlidingWindowMin is the window of time over which CPU usage is averaged. Should not be too small, otherwise momentary load spikes could cause task limits to be reduced unnecessarily. Too large and spikes may not cause throttling to occur.
minCpuPercent and maxCpuPercent should be set to a reasonably tight range, in order to keep the task limit as close to optimal as possible.
minTaskLimit and maxTaskLimit are considered safeguards to avoid nodes ever being allocated an unreasonable number of task. Setting maxTaskLimit to be equal to the number of assigned CPUs would be a reasonable starting point.

Note

A node’s task limits will only be adjusted while its task queue is full. That is, unless a node is fully-committed, it will not be scaled. This is to avoid continually downscaling each node to the minimum during periods of inactivity. Because of this, be realistic with setting maxTaskLimit to ensure the node is actually capable of hitting that maximum. If it can’t, the autoscaler will continue adjusting upwards, potentially causing the node to become unresponsive.

Deploy the resource manifest

kubectl apply -f autoscaler.yaml

Disable autoscaling

Delete the StroomTaskAutoscaler resource

kubectl delete -f autoscaler.yaml

Docs: Task Management

Fri, 09 Jul 2021 00:00:00 +0000

Various Tasks run in the background within Stroom. This HOWTO demonstrates how to manage these tasks

Assumptions

All Sections
- an account with the Administrator Application Permission is currently logged in.
Proxy Aggregation Tasks
- we have a multi node Stroom cluster with two nodes, stroomp00 and stroomp01.
Stream Processor Tasks
- we have a multi node Stroom cluster with two nodes, stroomp00 and stroomp01.
- when demonstrating adding a new node to an existing cluster, the new node is stroomp02.

Proxy Aggregation

Turn Off Proxy Aggregation

We first select the Monitoring item of the Main Menu to bring up the Monitoring sub-menu.

Stroom UI Monitoring sub-menu

Then move down and select the Jobs sub-item to be presented with the Jobs configuration tab as seen below.

Stroom UI Jobs Management - management tab

At this point, we can select the Proxy Aggregation Job whose check-box is selected and the tab will show the individual Stroom Processor nodes in the deployment.

Stroom UI Jobs Management - Proxy Aggregation Job

At this point, uncheck the Enabled check-boxes for both nodes and also the main Proxy Aggregation check-box to see.

Stroom UI Jobs Management - Proxy Aggregation Job Off

At this point, no new proxy aggregation will occur and any inbound files received by the Store Proxies will accumulate in the proxy storage area.

Turn On Proxy Aggregation

We first select the Monitoring item of the Main Menu to bring up the Monitoring sub-menu.

Stroom UI Monitoring sub-menu

Then move down and select the Jobs sub-item then select the Proxy Aggregation Job to be presented with the Jobs configuration tab as seen below.

Stroom UI Jobs Management - Proxy Aggregation Job Off

Now, re-enable each node’s Proxy Aggregation check-box and the main Proxy Aggregation check-box.

After checking the check-boxes, perform a refresh of the display by pressing the Refresh icon .

on the top right of the lower (node display) pane. You should note the Last Executed date/time change to see

Stroom UI Test Feed - Re-enable Proxy Aggregation

Stream Processors

Enable Stream Processors

To enable the Stream Processors task, move to the Monitoring item of the Main Menu and select it to bring up the Monitoring sub-menu.

Stroom UI Monitoring sub-menu

Then move down and select the Jobs sub-item to be presented with the Jobs configuration tab as seen below.

Stroom UI Jobs Management - management tab

At this point, we select the Stream Processor Job whose check-box is not selected and the tab will show the individual Stroom Processor nodes in the Stroom deployment.

Stroom UI Jobs Management - Stream Processor

Clearly, if it was a single node Stroom deployment, you would only see the one node at the bottom of the Jobs configuration tab.

We enable nodes by selecting their check-boxes as well as the main Stream Processors check-box. Do so.

Stroom UI Jobs Management - Stream Processor enabled

That is it. Stroom will automatically take note of these changes and internally start each node’s Stroom Processor task.

Enable Stream Processors On New Node

When one expands a Multi Node Stroom cluster deployment, after the installation of the Stroom Proxy and Application software and services on the new node, we need to enable its Stream Processors task.

To enable the Stream Processors for this new node, move to the Monitoring item of the Main Menu and select it to bring up the Monitoring sub-menu.

Stroom UI Monitoring sub-menu

Then move down and select the Jobs sub-item to be presented with the Jobs configuration tab as seen below.

Stroom UI Jobs Management - management tab

At this point, we select the Stream Processor Job whose check-box is selected

Stroom UI Jobs Management - Stream Processor new node

We enable the new node by selecting its check-box.

Stroom UI Jobs Management - Stream Processor enabled on new node