Version 7.4

• 1: New Features
• 2: Preview Features (experimental)
• 3: Breaking Changes
• 4: Upgrade Notes
• 5: Change Log

1 - New Features

Scheduler

The scheduler in Stroom that is used to schedule all the background jobs and Analytic Rules has been changed. The existing cron /frequency scheduler was quite simplistic and only supported a limited set of features of a cron schedule.

The cron format has changed from a three value cron expression (e.g. * * * to run every minute) to six value one. Existing three value cron expressions will be migrated to the new syntax when deploying Stroom v7.4.

For full details of the new scheduler see Scheduler

---

---

---

General User Interface Changes

Keyboard Shortcuts

Various new keyboard shortcuts for performing actions in Stroom. See Keyboard Shortcuts for details.

• Add the keyboard shortcut Ctrl ^ + Enter ↵ to the code pane of the stepper to perform a step refresh .
• Add the keyboard shortcut Ctrl ^ + Enter ↵ to the Dashboards to execute all queries.
• Add multiple Goto type shortcuts to jump directly to a screen. See Direct Access to Screens for details.

Documentation

The Documentation entity will now default to edit mode if there is no documentation, e.g. on a newly created Documentation entity.

Copy As

A Copy As group has been added to the explorer tree context menu. This replaces, but includes the Copy Link to Clipboard menu item.

• Copy Name to Clipboard - Copies the name of the entity to the clipboard.
• Copy UUID to Clipboard - Copies the UUID of the entity to the clipboard.
• Copy link to Clipboard - Copies a URL to the clipboard that will link directly to the selected entity.

---

API Specification Link

Previous versions of Stroom have included an interactive user interface for navigating Stroom’s API specification. A link to this has been added to the Help menu.

---

---

Dashboard Conditional Formatting

• Make the Enabled and Hide Row checkboxes clickable in the table without having open the rule edit dialog.
• Dim disabled rules in the table.
• Add colour swatches for the background and text colours.

---

Help Icons on Dialogs

Functionality has been added to included help icons on dialogs. Clicking the icon will display a popup containing help text relating to the thing the icon is next to.

Currently help icons have only been added to a few dialogs, but more will follow.

---

Stepping Location

The way you change the step location in the stepper has changed. Previously you could click and then directly edit each of the three numbers. Now the location label is a clickable link that opens an edit dialog.

---

---

2 - Preview Features (experimental)

Analytic Rules

Analytic Rules were introduced in Stroom v7.2 as a preview feature. They remain an experimental preview feature but have undergone more changes/improvements.

Analytic rules are a means of writing a query to find matching data.

Processing Types

Analytic rules have three different processing types:

Streaming

A streaming rule uses a processor filter to find streams that match the filter and runs the query against the stream.

Scheduled Query

A scheduled query will run the rule’s query against a time window of data on a scheduled basis. The time window can be absolute or relative to when the scheduled query fires.

Table Builder

Multiple Notifications

Rules now support having multiple notification types/destinations, for example sending an email as well a stream . Currently Email and Stream are the only notification types supported.

Email Templating

The email notifications have been improved to allow templating of the email subject and body. The template enables static text/HTML to be mixed with values taken from the detection.

---

The templating uses a template syntax called jinja and specifically the JinJava library. The templating syntax includes support for variables, filters, condition blocks, loops, etc. Full details of the syntax see Templating and for details of the templating context available in email subject/body templates see Rule Detections Context.

Example Template

The following is an example of a detection template producing a HTML email body that includes conditions and loops:

<!DOCTYPE html>
<html lang="en">
<meta charset="UTF-8" />
<title>Detector '{{ detectorName | escape }}' Alert</title>
<body>
  <p>Detector <em>{{ detectorName | escape }}</em> {{ detectorVersion | escape }} fired at {{ detectTime | escape }}</p>

  {%- if (values | length) > 0 -%}
  <p>Detail: {{ headline | escape }}</p>
  <ul>
    {% for key, val in values | dictsort -%}
      <li><strong>{{ key | escape }}</strong>: {{ val | escape }}</li>
    {% endfor %}
  </ul>
  {% endif -%}

  {%- if (linkedEvents | length) > 0 -%}
  <p>Linked Events:</p>
  <ul>
    {% for linkedEvent in linkedEvents -%}
      <li>Environment: {{ linkedEvent.stroom | escape }}, Stream ID: {{ linkedEvent.streamId | escape }}, Event ID: {{ linkedEvent.eventId | escape }}</li>
    {% endfor %}
  </ul>
  {% endif %}
</body>

Improved Date Picker

Scheduled queries make use of a new data picker dialog which makes the process of setting a date/time value much easier for the user. This new date picker will be rolled out to other screens in Stroom in some later release.

---

3 - Breaking Changes

Analytic Rules

Any Analytic Rules created in versions 7.2 or 7.3 will need to be deleted and re-created in v7.4. Analytic Rules has been an experimental feature therefore there is no migration in place for rules from previous versions of Stroom.

4 - Upgrade Notes

Java Version

Stroom v7.5 requires Java 21. This is the same java version as Stroom v7.3. Ensure the Stroom and Stroom-Proxy hosts are running the latest patch release of Java v21.

Database Migrations

When Stroom boots for the first time with a new version it will run any required database migrations to bring the database schema up to the correct version.

On boot, Stroom will ensure that the migrations are only run by a single node in the cluster. This will be the node that reaches that point in the boot process first. All other nodes will wait until that is complete before proceeding with the boot process.

It is recommended however to use a single node to execute the migration. To avoid Stroom starting up and beginning processing you can use the migrage command to just migrate the database and not fully boot Stroom. See migrage command for more details.

Migration Filename Change

If you are deploying onto a v7.4-beta.1 instance you will need to modify the database table used to log migration history. If the job_schema_history table contains version 07.03.00.001 then you will need to run the following SQL against the database to prevent Stroom from failing the migration on boot.

delete from job_schema_history where version = '07.03.00.001';

Migration Scripts

For information purposes only, the following are the database migrations that will be run when upgrading to 7.4.0 from the previous minor version.

Note, the legacy module will run first (if present) then the other module will run in no particular order.

Module stroom-analytics

Script V07_04_00_001__execution_schedule.sql

Path: stroom-analytics/stroom-analytics-impl-db/src/main/resources/stroom/analytics/impl/db/migration/V07_04_00_001__execution_schedule.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

--
-- Create the table
--
CREATE TABLE IF NOT EXISTS execution_schedule (
   id int NOT NULL AUTO_INCREMENT,
   name varchar(255) NOT NULL,
   enabled tinyint NOT NULL DEFAULT '0',
   node_name varchar(255) NOT NULL,
   schedule_type varchar(255) NOT NULL,
   expression varchar(255) NOT NULL,
   contiguous tinyint NOT NULL DEFAULT '0',
   start_time_ms bigint DEFAULT NULL,
   end_time_ms bigint DEFAULT NULL,
   doc_type varchar(255) NOT NULL,
   doc_uuid varchar(255) NOT NULL,
   PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

CREATE INDEX execution_schedule_doc_idx ON execution_schedule (doc_type, doc_uuid);
CREATE INDEX execution_schedule_enabled_idx ON execution_schedule (doc_type, doc_uuid, enabled, node_name);

CREATE TABLE IF NOT EXISTS execution_history (
   id bigint(20) NOT NULL AUTO_INCREMENT,
   fk_execution_schedule_id int NOT NULL,
   execution_time_ms bigint NOT NULL,
   effective_execution_time_ms bigint NOT NULL,
   status varchar(255) NOT NULL,
   message longtext,
   PRIMARY KEY (id),
   CONSTRAINT execution_history_execution_schedule_id FOREIGN KEY (fk_execution_schedule_id) REFERENCES execution_schedule (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

CREATE TABLE IF NOT EXISTS execution_tracker (
   fk_execution_schedule_id int NOT NULL,
   actual_execution_time_ms bigint NOT NULL,
   last_effective_execution_time_ms bigint DEFAULT NULL,
   next_effective_execution_time_ms bigint NOT NULL,
   PRIMARY KEY (fk_execution_schedule_id),
   CONSTRAINT execution_tracker_execution_schedule_id FOREIGN KEY (fk_execution_schedule_id) REFERENCES execution_schedule (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

-- --------------------------------------------------

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set shiftwidth=2 tabstop=2 expandtab:

Module stroom-job

Script V07_04_00_005__job_node.sql

Path: stroom-job/stroom-job-impl-db/src/main/resources/stroom/job/impl/db/migration/V07_04_00_005__job_node.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

update job_node set schedule = concat('0 ', schedule, ' * ?') where job_type = 1 and regexp_like(schedule, '^[^ ]+ [^ ]+ [^ ]+$');

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set tabstop=4 shiftwidth=4 expandtab:

5 - Change Log

For a detailed list of all the changes in v7.4 see: v7.4 CHANGELOG