Version 7.5

• 1: New Features
• 2: Preview Features (experimental)
• 3: Breaking Changes
• 4: Upgrade Notes
• 5: Change Log

1 - New Features

This section contains the significant new features or changes in Stroom. For a full list of changes see Change Log.

User Interface

Jobs Screen

---

• Rows are now greyed out if one of the following is true:

  • The parent job is disabled.

  • The job is disabled on a specific node.

  • The node itself is disabled.

• The column Next Scheduled has been added to the detail pane to show the time that the job will next run.

• The details pane now has an Actions menu ( ) where it is possible to do the follow actions to a job on a specific node.

  • Edit the schedule for job on the selected node.

  • Run the job on the selected node now. This will run the job within 10s or so once clicked. Monitor the Last Executed column to see when it has run.

  • Show in Server Tasks (selected node). This will open the Server Tasks screen with the quick filter set to the selected job and node.

  • Show in Server Tasks (all nodes). This will open the Server Tasks screen with the quick filter set to the selected job.

  It is now possible to link directly to the Nodes screen by clicking the Open hover button next to the node name.

Nodes Screen

---

• A jobs detail pane has been added to show the jobs on that node. This shows all jobs and allows jobs to be enabled/disabled on a node. It also shows the state of the parent job and the node. The button can be used to show only the enabled jobs on that node.

  An auto-refresh button has also been added to keep refreshing the jobs detail pane so the Last Executed and Next Scheduled columns are updated.

  It is also possible to link directly to the Jobs screen by clicking the Open hover button next to the job name.

• Nodes rows are now greyed out if the node is disabled.

• Job rows are not greyed out if one of the following is true:

  • The node is disabled.
  • The parent job is disabled.
  • The job is disabled on a specific node.

• The column Build Version has been added to show the version of Stroom that the node is running. This is to highlight any nodes running the wrong version.

• The column Up Date have been added to the Nodes screen to show the time that Stroom was last booted on that node.

• The Ping column screen has been changed so an enabled node with no ping stands out while a disabled node does not.

New Look and Feel

The Login, Change Password and Manage Accounts screens have been changed to bring their look and feel in line with the rest of the application. The screens may look a little different but are functionally the same.

---

---

---

---

---

Dictionaries

The Dictionary screen has been changed to make it easier to manage the import of other Dictionaries.

The Import sub-tab has been changed to include a detail pane that shows the effective word list for each imported Dictionary. This will show all words in the imported dictionary along with any from Dictionaries that it imports from.

A Effective Words sub-tab has been added to show the effective list of words in the Dictionary, i.e. combining all words from the dictionary, its imports and any dictionaries imported by those imports.

Authentication Error Screen

If there is an authentication error during user login, e.g. the account is disabled or locked, the user will now be redirected to a configurable error screen rather than back to the login screen.

The content of the lower part of the dialog is configurable via the property stroom.ui.authErrorMessage. This property accepts HTML content. This the message to contain details of how to contact the appropriate Stroom admin team.

Queries

Editor Code Completion

The code completion in the Query editor has been changed to make the code completion suggestions context aware. For example if you have just typed in dictionary and then hit Ctrl ^ + Space ␣ it will suggest the names of Dictionary documents that are visible to the user.

Dictionaries and Visualisations have also been added to the list of Query Help items (in the left hand pane) and to the available code completions.

Table Download

You can now download the results of a Query using the icon.

Functions in select

Functions, e.g. count() can now be used within the select clause of a StroomQL query.

Dashboards

Sorting

Now when you change an existing table sort on a dashboard it does not require the query to be executed again. The table data will change to reflect the new sort settings. This is particularly useful on complex queries or those operating on large amounts of data.

Dictionary List Input

When using a List Input pane on a Dashboard that is configured with a Dictionary , the drop-down now shows the source of each Dictionary word.

Other UI Changes

• The explorer tree now shows the name of the item in the hover tooltip. This is useful if the name extends beyond the limit of the explorer tree pane.

• Issue #4339 : Allow user selection of analytic duplicate columns.

• Issue #3989 : Improve pause behaviour in dashboards and general presentation of busy state throughout UI.

• The way dialogs can be moved or resized has been improved so that they can be resized on any edge or corner. The area for clicking and dragging to move a dialog has been increased to include all of the title section.

Permissions

• Document deletion will now also delete all associated document permissions granted to user/groups. This previously did not happen on document delete so orphaned document permissions would build up in the database.

  The DB migration V07_04_00_005__Orphaned_Doc_Perms which will delete all document permissions (in table doc_permission) for docs that are not a folder, not the System doc, are not a valid doc (i.e. in the doc table) and are not a pipeline filter. Deleted document permission records will first be copied to a backup table doc_permission_backup_V07_04_00_005.

• Document Copy and Move has been changed to check that the user has Owner permission (or admin) on the document being copied/moved if the permissions mode is None, Destination or Combined. This is because those modes will change the permissions which is something only an Owner/admin can do.

Content Search

The Find in Content screen added in v7.3 has been changed to add Lucene indexing to speed up content searches. The Lucene index will be created when a user first uses the content search. The user may see an error message on first use telling them to wait for the index to be built.

The index is located in <stroom.path.temp>/doc-index, which unless explicitly configured will likely be /tmp/doc-index.

Volumes

• The tables on the Data Volumes and Index Volumes screens have been changed to low-light CLOSED/INACTIVE volumes.

• Tooltips have been added to the Path and Last Updated columns.

• The Use% column has been changed to a percentage bar.

• Red/green colouring has been added to the to the Full column values to make it clearer which volumes are full.

Dependency Documents

Missing Dependencies

Various screens include document pickers to select a dependency document, e.g. selecting an extraction Pipeline in the Dashboard table settings. The document picker will now show a icon to indicate the previously selected document is not longer visible to the user or has been deleted.

Tagged Documents

Various screens require the selection of an extraction, or reference loader Pipeline, i.e.:

• View - Extraction pipeline
• Index - Default extraction pipeline
• Dashboard - Extraction Pipeline
• Pipeline - Reference loader pipeline

To distinguish processing pipelines from extraction or reference loading pipelines, the Pipeline documents can be tagged with pre-configured tags such as extraction and reference-loader. This means the Pipeline picker screen can be pre-filtered on the appropriate tag to make finding the right document easier.

It is recommended to tag all such pipelines using these tags to make document selection easier for other users.

This system defined tagging is configured using the following properties

• stroom.explorer.suggestedTags
• stroom.ui.query.dashboardPipelineSelectorIncludedTags
• stroom.ui.query.indexPipelineSelectorIncludedTags
• stroom.ui.query.viewPipelineSelectorIncludedTags
• stroom.ui.referencePipelineSelectorIncludedTags

API Keys

The API keys screen has changed to allow selection of the hashing algorithm used to store a hash of the API key.

Processing

S3 Appender

A new S3 pipeline element S3Appender has been added to enable the streaming of data to an S3 bucket.

The S3 Appender requires the creation of an S3 Config document to provide the credentials and role details for connecting to the S3 bucket. The content of the S3 Config document is JSON and the JSON Schema describing its structure can be found here .

Stepping

The stepper has been changed to allow termination of the step. This is useful when stepping large streams or when using filtered steps.

The fact that the step is in progress is indicated by a label above the pipeline elements.

Other Changes

• Issue #4444 : Change the hash() expression function to allow the algorithm and salt arguments to be the result of functions rather than just static values, e.g. hash(${field1}, concat('SHA-', ${algoLen}), ${salt}).

2 - Preview Features (experimental)

State Store

• Issue #2126 : Add experimental state store.

3 - Breaking Changes

There are no breaking changes in v7.5.

4 - Upgrade Notes

Java Version

Stroom v7.5 requires Java 21. This is the same java version as Stroom v7.4. Ensure the Stroom and Stroom-Proxy hosts are running the latest patch release of Java v21.

Configuration File Changes

The following changes requiring action have been made to the stroom.yml configuration file.

Removed Properties

stroom.ui.theme.backgroundColour

This property has been removed. You will need to remove it (if present) from your configuration files else Stroom will not boot.

New Properties

stroom.contentIndex

This property has been added. It enables the indexing of stroom content for fast content searching.

state.*

This block of properties has been added to control the new state store functionality.

appConfig:
  state:
    scyllaDbDocCache:
      expireAfterAccess: null
      expireAfterWrite: "PT10M"
      maximumSize: 100
      refreshAfterWrite: null
    sessionCache:
      expireAfterAccess: "PT1H"
      expireAfterWrite: null
      maximumSize: 10
      refreshAfterWrite: null
    stateDocCache:
      expireAfterAccess: null
      expireAfterWrite: "PT10M"
      maximumSize: 100
      refreshAfterWrite: null

Changed Property Values

stroom.ui.helpSubPathJobs

The value of this property has changed from /user-guide/jobs/ to /reference-section/jobs/.

stroom.ui.helpUrl

The value of this property has changed from https://gchq.github.io/stroom-docs/7.4/docs to https://gchq.github.io/stroom-docs/7.5/docs.

Servlets

Stroom presents various servlets. The paths to these servlets have changed but the existing paths remain.

• /stroom/noauth/datafeed => /datafeed
• /stroom/noauth/debug => /debug
• /stroom/noauth/echo => /echo
• /stroom/noauth/status => /status
• /stroom/noauth/swagger-ui => /swagger-ui
• /stroom/sessionList => /sessionList

The /sessionList (and /stroom/sessionList) servlet has been changed to require manage users permission.

Database Migrations

When Stroom boots for the first time with a new version it will run any required database migrations to bring the database schema up to the correct version.

On boot, Stroom will ensure that the migrations are only run by a single node in the cluster. This will be the node that reaches that point in the boot process first. All other nodes will wait until that is complete before proceeding with the boot process.

It is recommended however to use a single node to execute the migration. To avoid Stroom starting up and beginning processing you can use the migrage command to just migrate the database and not fully boot Stroom. See migrage command for more details.

update cross_module_schema_history
set version = '07.05.00.005',
script = 'stroom.app.db.migration.V07_05_00_005__Orphaned_Doc_Perms'
where version = '07.04.00.005'
and script = 'stroom.app.db.migration.V07_04_00_005__Orphaned_Doc_Perms';

Migration Scripts

For information purposes only, the following are the database migrations that will be run when upgrading to 7.5.0 from the previous minor version.

Note, the legacy module will run first (if present) then the other module will run in no particular order.

Module stroom-app

Script V07_05_00_005__Orphaned_Doc_Perms.java

Path: stroom-app/src/main/java/stroom/app/db/migration/V07_05_00_005__Orphaned_Doc_Perms.java

It is not possible to display the content here. The file can be viewed on : GitHub

Module stroom-docstore

Script V07_05_00_005__Add_index_on_doc.sql

Path: stroom-docstore/stroom-docstore-impl-db/src/main/resources/stroom/docstore/impl/db/migration/V07_05_00_005__Add_index_on_doc.sql

-- ------------------------------------------------------------------------
-- Copyright 2022 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- stop note level warnings about objects (not)? existing
SET @old_sql_notes=@@sql_notes, sql_notes=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE docstore_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_create_non_unique_index$$

-- DO NOT change this without reading the header!
CREATE PROCEDURE docstore_create_non_unique_index (
    p_table_name varchar(64),
    p_index_name varchar(64),
    p_index_columns varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        CALL docstore_run_sql(CONCAT(
            'create index ', p_index_name,
            ' on ', database(), '.', p_table_name,
            ' (', p_index_columns, ')'));
    ELSE
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- Improve lookup by name and to remove need to hit the table when we
-- list all docs by type.
CALL docstore_create_non_unique_index(
    "doc",
    "doc_type_name_uuid_idx",
    "type, name, uuid");

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_create_non_unique_index;

DROP PROCEDURE IF EXISTS docstore_run_sql;

-- --------------------------------------------------


-- Reset to the original value
SET SQL_NOTES=@OLD_SQL_NOTES;

Module stroom-node

Script V07_05_00_005__add_build_ver_last_boot.sql

Path: stroom-node/stroom-node-impl-db/src/main/resources/stroom/node/impl/db/migration/V07_05_00_005__add_build_ver_last_boot.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

DROP PROCEDURE IF EXISTS node_run_sql_v1 $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE node_run_sql_v1 (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS node_add_column_v1$$

CREATE PROCEDURE node_add_column_v1 (
    p_table_name varchar(64),
    p_column_name varchar(64),
    p_column_type_info varchar(64) -- e.g. 'varchar(255) default NULL'
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.columns
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND column_name = p_column_name;

    IF object_count = 0 THEN
        CALL node_run_sql_v1(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' add column ', p_column_name, ' ', p_column_type_info));
    ELSE
        SELECT CONCAT(
            'Column ',
            p_column_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

CALL node_add_column_v1(
    'node',
    'build_version',
    'varchar(255) DEFAULT NULL');

CALL node_add_column_v1(
    'node',
    'last_boot_ms',
    'bigint DEFAULT NULL');

DROP PROCEDURE IF EXISTS node_add_column_v1;

DROP PROCEDURE IF EXISTS node_run_sql_v1;

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set tabstop=4 shiftwidth=4 expandtab:

Module stroom-security

Script V07_05_00_005__api_key_relax_uniqeness.sql

Path: stroom-security/stroom-security-impl-db/src/main/resources/stroom/security/impl/db/migration/V07_05_00_005__api_key_relax_uniqeness.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_create_non_unique_index$$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_create_non_unique_index (
    p_table_name varchar(64),
    p_index_name varchar(64),
    p_index_columns varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        CALL security_run_sql(CONCAT(
            'create index ', p_index_name,
            ' on ', database(), '.', p_table_name,
            ' (', p_index_columns, ')'));
    ELSE
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_drop_index $$

-- e.g. security_drop_index('MY_TABLE', 'MY_IDX');
CREATE PROCEDURE security_drop_index (
    p_table_name varchar(64),
    p_index_name varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' does not exist on table ',
            database(),
            '.',
            p_table_name);
    ELSE
        CALL security_run_sql(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' drop index ', p_index_name));
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- We need to make this column case sensitive (_as_cs) else we limit the range of keys we can generate
-- as the keys have mixed case.
-- Note the api_key_prefix col contains lower case data so can stay as _ai_ci
ALTER TABLE api_key MODIFY
    api_key_hash VARCHAR(255)
      CHARACTER SET utf8mb4
      COLLATE utf8mb4_0900_as_cs
      NOT NULL;

-- Drop the old unique index so we can re-create it as non-unique
CALL security_drop_index(
    "api_key",
    "api_key_prefix_idx");

-- We have to look up records by prefix. This will usually return 1 row
-- but may return >1. We test the hash of all returned rows.
CALL security_create_non_unique_index(
    "api_key",
    "api_key_prefix_idx",
    "api_key_prefix");

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_create_non_unique_index;

DROP PROCEDURE IF EXISTS security_drop_index;

DROP PROCEDURE IF EXISTS security_run_sql;

-- --------------------------------------------------

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set shiftwidth=4 tabstop=4 expandtab:

Script V07_05_00_010__api_key_add_algo_column.sql

Path: stroom-security/stroom-security-impl-db/src/main/resources/stroom/security/impl/db/migration/V07_05_00_010__api_key_add_algo_column.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_add_column$$

CREATE PROCEDURE security_add_column (
    p_table_name varchar(64),
    p_column_name varchar(64),
    p_column_type_info varchar(64) -- e.g. 'varchar(255) default NULL'
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.columns
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND column_name = p_column_name;

    IF object_count = 0 THEN
        CALL security_run_sql(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' add column ', p_column_name, ' ', p_column_type_info));
    ELSE
        SELECT CONCAT(
            'Column ',
            p_column_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$


-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- idempotent
CALL security_add_column(
    "api_key",
    "hash_algorithm",
    'tinyint NOT NULL default 0');

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_add_column;

DROP PROCEDURE IF EXISTS security_run_sql;

-- --------------------------------------------------

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set shiftwidth=4 tabstop=4 expandtab:

5 - Change Log

The follow changes are in 7.5 but not in 7.4:

• Issue #4501 : Fix Query editor syntax highlighting.

• Add query help and editor completions for Dictionary Docs for use with in dictionary.

• Issue #4487 : Fix nasty error when running a stats query with no columns.

• Issue #4498 : Make the explorer tree Expand/Collapse All buttons respect the current Quick Filter input text.

• Issue #4518 : Change the Stream Upload dialog to default the stream type to that of the feed.

• Issue #4470 : On import of Feed or Index docs, replace unknown volume groups with the respective configured default volume group (or null if not configured).

• Issue #4460 : Change the way we display functions with lots of arguments in query help and code completion popup.

• Issue #4526 : Change Dictionary to not de-duplicate words as this is breaking JSON when used for holding SSL config in JSON form.

• Issue #4528 : Make the Reindex Content job respond to stroom shutdown.

• Issue #4532 : Fix Run Job Now so that it works when the job or jobNode is disabled.

• Issue #4444 : Change the hash() expression function to allow the algorithm and salt arguments to be the result of functions, e.g. hash(${field1}, concat('SHA-', ${algoLen}), ${salt}).

• Issue #4534 : Fix NPE in include/exclude filter.

• Issue #4527 : Change the non-regex search syntax of Find in Content to not use Lucene field based syntax so that : works correctly. Also change the regex search to use Lucene and improve the styling of the screen.

• Issue #4536 : Fix NPE.

• Issue #4539 : Improve search query logging.

• Improve the process of (re-)indexing content. It is now triggered by a user doing a content search. Users will get an error message if the index is still being initialised. The stroom.contentIndex.enabled property has been removed.

• Issue #4513 : Add primary key to doc_permission_backup_V07_05_00_005 table for MySQL Cluster support.

• Issue #4514 : Fix HTTP 307 with calling /api/authproxy/v1/noauth/fetchClientCredsToken.

• Issue #4475 : Change mask() function to period() and add using to apply a function to window.

• Issue #4341 : Allow download from query table.

• Issue #4507 : Fix index shard permission issue.

• Issue #4510 : Fix right click in editor pane.

• Issue #4511 : Fix StreamId, EventId selection in query tables.

• Issue #4485 : Improve dialog move/resize behaviour.

• Issue #4492 : Make Lucene behave like SQL for OR(NOT()) queries.

• Issue #4494 : Allow functions in StroomQL select, e.g. count().

• Issue #4202 : Fix default destination not being selected when you do Save As.

• Issue #4475 : Add mask() function and deprecate countPrevious().

• Issue #4491 : Fix tab closure when deleting items in the explorer tree.

• Issue #4502 : Fix inability to step an un-processed stream.

• Issue #4503 : Make the enabled state of the delete/restore buttons on the stream browser depend on the user’s permissions. Now they will only be enabled if the user has the require permission (i.e. DELETE/UPDATE) on at least one of the selected items.

• Issue #4486 : Fix the format-date XSLT function for date strings with the day of week in, e.g. stroom:format-date('Wed Aug 14 2024', 'E MMM dd yyyy').

• Issue #4458 : Fix explorer node tags not being copied. Also fix copy/move not selecting the parent folder of the source as the default destination folder.

• Issue #4454 : Show the source dictionary name for each word in the Dashboard List Input selection box. Add sorting and de-duplication of words.

• Issue #4455 : Add Goto Document links to the Imports sub-tab of the Dictionary screen. Also add new Effective Words tab to list all the words in the dictionary that include those from its imports (and their imports).

• Issue #4468 : Improve handling of key sequences and detection of key events from ACE editor.

• Issue #4472 : Change the User Preferences dialog to cope with redundant stroom/editor theme names.

• Issue #4479 : Add ability to assume role for S3.

• Issue #4202 : Fix problems with Dashboard Extraction Pipeline picker incorrectly changing the selected pipeline.

• Change the DocRef picker so that it shows a warning icon if the selected DocRef no longer exists or the user doesn’t have permission to view it.

• Change the Extraction Pipeline picker on the Index Settings screen to pre-filter on tag:extraction. This is configured using the property stroom.ui.query.indexPipelineSelectorIncludedTags.

• Issue #4146 : Fix audit events for deleting/restoring streams.

• Change the alert dialog message styling to have a max-height of 600px so long messages get a scrollbar.

• Issue #4468 : Fix selection box keyboard selection behavior when no quick filter is visible.

• Issue #4471 : Fix NPE with stepping filter.

• Issue #4451 : Add S3 pipeline appender.

• Issue #4401 : Improve content search.

• Issue #4417 : Show stepping progress and allow termination.

• Issue #4436 : Change the way API Keys are verified. Stroom now finds all valid api keys matching the api key prefix and compares the hash of the api key against the hash from each of the matching records. Support has also been added for using different hash algorithms.

• Issue #4448 : Fix query refresh tooltip when not refreshing.

• Issue #4457 : Fix ctrl+enter shortcut for query start.

• Issue #4441 : Improve sorted column matching.

• Issue #4449 : Reload Scheduled Query Analytics between executions.

• Issue #4420 : Make app title dynamic.

• Issue #4453 : Dictionaries will ignore imports if a user has no permission to read them.

• Issue #4404 : Change the Query editor completions to be context aware, e.g. it only lists Datasources after a from .

• Issue #4450 : Fix editor completion in Query editor so that it doesn’t limit completions to 100. Added the property stroom.ui.maxEditorCompletionEntries to control the maximum number of completions items that are shown. In the event that the property is exceeded, Stroom will pre-filter the completions based on the user’s input.

• Add Visualisations to the Query help and editor completions. Visualisation completion inserts a snippet containing all the data fields in the Visualisation, e.g. TextValue(field = Field, gridSeries = Grid Series).

• Issue #4424 : Fix alignment of Current Tasks heading on the Jobs screen.

• Issue #4422 : Don’t show Edit Schedule in actions menu on Jobs screen for Distributed jobs.

• Issue #4418 : Fix missing css for /stroom/sessionList.

• Issue #4435 : Fix for progress spinner getting stuck on.

• Issue #4426 : Add INFO message when an index shard is created.

• Issue #4425 : Fix Usage Date heading alignment on Edit Volume Group screen for both data/index volumes.

• Uplift docker image JDK to eclipse-temurin:21.0.4_7-jdk-alpine.

• Issue #4416 : Allow dashboard table sorting to be changed post query.

• Issue #4421 : Change session state XML structure.

• Issue #4419 : Automatically unpause dashboard result components when a new search begins.

• Rename migration from V07_04_00_005__Orphaned_Doc_Perms to V07_05_00_005__Orphaned_Doc_Perms.

• Issue #4383 : Add an authentication error screen to be shown when a user tries to login and there is an authentication problem or the user’s account has been locked/disabled. Previously the user was re-directed to the sign-in screen even if cert auth was enabled. Added the new property stroom.ui.authErrorMessage to allow setting generic HTML content to show the user when an authentication error occurs.

• Issue #4400 : Fix missing styling on sessionList servlet.

• Fix broken description pane in the stroomQL code completion.

• Change API endpoint /Authentication/v1/noauth/reset from GET to POST and from a path parameter to a POST body.

• Fix various issues relating to unauthenticated servlets. Add new servlet paths e.g. /stroom/XXX becomes /XXX and /stroom/XXX. The latter will be removed in some future release. Notable new servlet paths are /dashboard, /status, /swagger-ui, /echo, /debug, /datafeed, /sessionList.

• Change sessionList servlet to require manage users permission.

• Issue #4360 : Fix quick time settings popup.

• Improve styling of Jobs screen so disabled jobs/nodes are greyed out.

• Add Next Scheduled column to the detail pane of the Job screen.

• Add Build Version and Up Date columns to the Nodes screen. Also change the styling of the Ping column so an enabled node with no ping stands out while a disabled node does not. Also change the row styling for disabled nodes.

• Add a Run now icon to the jobs screen to execute a job on a node immediately.

• Change the FS Volume and Index Volume tables to low-light CLOSED/INACTIVE volumes. Add tooltips to the path and last updated columns. Change the Use% column to a percentage bar. Add red/green colouring to the Full column values.

• Issue #4327 : Add a Jobs pane to the Nodes screen to view jobs by node. Add linking between job nodes on the Nodes screen and the Jobs screen.

• Issue #4339 : Allow user selection of analytic duplicate columns.

• Issue #2126 : Add experimental state store.

• Issue #4334 : Popup explorer text on mouse hover.

• Issue #4278 : Make document deletion also delete the permission records for that document. Also run migration V07_04_00_005__Orphaned_Doc_Perms which will delete all document permissions (in table doc_permission) for docs that are not a folder, not the System doc, are not a valid doc (i.e. in the doc table) and are not a pipeline filter. Deleted document permission records will first be copied to a backup table doc_permission_backup_V07_04_00_005.

• Change document Copy and Move to check that the user has Owner permission (or admin) on the document being copied/moved if the permissions mode is None, Destination or Combined. This is because those modes will change the permissions which is something only an Owner/admin can do.

• Issue #3989 : Improve pause behaviour in dashboards and general presentation of busy state throughout UI.

• Issue #2111 : Add index assistance to find content feature.

For a detailed list of all the changes in v7.5 see: v7.5 CHANGELOG