Upgrade Notes

Required actions and information relating to upgrading to Stroom version 7.5.

Warning

Please read this section carefully in case any of it is relevant to your Stroom instance.

Java Version

Stroom v7.5 requires Java 21. This is the same java version as Stroom v7.4. Ensure the Stroom and Stroom-Proxy hosts are running the latest patch release of Java v21.

Configuration File Changes

The following changes requiring action have been made to the stroom.yml configuration file.

Removed Properties

`stroom.ui.theme.backgroundColour`

This property has been removed. You will need to remove it (if present) from your configuration files else Stroom will not boot.

New Properties

`stroom.contentIndex`

This property has been added. It enables the indexing of stroom content for fast content searching.

`state.*`

This block of properties has been added to control the new state store functionality.

appConfig:
  state:
    scyllaDbDocCache:
      expireAfterAccess: null
      expireAfterWrite: "PT10M"
      maximumSize: 100
      refreshAfterWrite: null
    sessionCache:
      expireAfterAccess: "PT1H"
      expireAfterWrite: null
      maximumSize: 10
      refreshAfterWrite: null
    stateDocCache:
      expireAfterAccess: null
      expireAfterWrite: "PT10M"
      maximumSize: 100
      refreshAfterWrite: null

Changed Property Values

`stroom.ui.helpSubPathJobs`

The value of this property has changed from /user-guide/jobs/ to /reference-section/jobs/.

`stroom.ui.helpUrl`

The value of this property has changed from https://gchq.github.io/stroom-docs/7.4/docs to https://gchq.github.io/stroom-docs/7.5/docs.

Servlets

Stroom presents various servlets. The paths to these servlets have changed but the existing paths remain.

Warning

While the old paths still work, they will be removed in a future version of Stroom so you are advised to change any scripts or similar that point to these servlets.

/stroom/noauth/datafeed => /datafeed
/stroom/noauth/debug => /debug
/stroom/noauth/echo => /echo
/stroom/noauth/status => /status
/stroom/noauth/swagger-ui => /swagger-ui
/stroom/sessionList => /sessionList

Note

These servlet paths are those presented by the stroom application itself. Stroom may be fronted by nginx in which case that may already be doing path mapping to abstract the end client from Stroom’s servlet paths.

The /sessionList (and /stroom/sessionList) servlet has been changed to require manage users permission.

Database Migrations

When Stroom boots for the first time with a new version it will run any required database migrations to bring the database schema up to the correct version.

Warning

It is highly recommended to ensure you have a database backup in place before booting stroom with a new version. This is to mitigate against any problems with the migration. It is also recommended to test the migration against a copy of your database to ensure that there are no problems when you do it for real.

On boot, Stroom will ensure that the migrations are only run by a single node in the cluster. This will be the node that reaches that point in the boot process first. All other nodes will wait until that is complete before proceeding with the boot process.

It is recommended however to use a single node to execute the migration. To avoid Stroom starting up and beginning processing you can use the migrage command to just migrate the database and not fully boot Stroom. See migrage command for more details.

Warning

If you are upgrading a v7.5 Stroom instance that has a version less than or equal to v7.5-beta.9 to a version higher than v7.5-beta.9 then you must run the following SQL on the database to correct the renaming of a migration script. If you don’t, Stroom will not boot as it will detect a mismatch in the DB migration scripts.

update cross_module_schema_history
set version = '07.05.00.005',
script = 'stroom.app.db.migration.V07_05_00_005__Orphaned_Doc_Perms'
where version = '07.04.00.005'
and script = 'stroom.app.db.migration.V07_04_00_005__Orphaned_Doc_Perms';

Migration Scripts

For information purposes only, the following are the database migrations that will be run when upgrading to 7.5.0 from the previous minor version.

Note, the legacy module will run first (if present) then the other module will run in no particular order.

Module `stroom-app`

Script `V07_05_00_005__Orphaned_Doc_Perms.java`

Path: stroom-app/src/main/java/stroom/app/db/migration/V07_05_00_005__Orphaned_Doc_Perms.java

It is not possible to display the content here. The file can be viewed on : GitHub

Module `stroom-docstore`

Script `V07_05_00_005__Add_index_on_doc.sql`

Path: stroom-docstore/stroom-docstore-impl-db/src/main/resources/stroom/docstore/impl/db/migration/V07_05_00_005__Add_index_on_doc.sql

-- ------------------------------------------------------------------------
-- Copyright 2022 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- stop note level warnings about objects (not)? existing
SET @old_sql_notes=@@sql_notes, sql_notes=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE docstore_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_create_non_unique_index$$

-- DO NOT change this without reading the header!
CREATE PROCEDURE docstore_create_non_unique_index (
    p_table_name varchar(64),
    p_index_name varchar(64),
    p_index_columns varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        CALL docstore_run_sql(CONCAT(
            'create index ', p_index_name,
            ' on ', database(), '.', p_table_name,
            ' (', p_index_columns, ')'));
    ELSE
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- Improve lookup by name and to remove need to hit the table when we
-- list all docs by type.
CALL docstore_create_non_unique_index(
    "doc",
    "doc_type_name_uuid_idx",
    "type, name, uuid");

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS docstore_create_non_unique_index;

DROP PROCEDURE IF EXISTS docstore_run_sql;

-- --------------------------------------------------


-- Reset to the original value
SET SQL_NOTES=@OLD_SQL_NOTES;

Module `stroom-node`

Script `V07_05_00_005__add_build_ver_last_boot.sql`

Path: stroom-node/stroom-node-impl-db/src/main/resources/stroom/node/impl/db/migration/V07_05_00_005__add_build_ver_last_boot.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

DROP PROCEDURE IF EXISTS node_run_sql_v1 $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE node_run_sql_v1 (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS node_add_column_v1$$

CREATE PROCEDURE node_add_column_v1 (
    p_table_name varchar(64),
    p_column_name varchar(64),
    p_column_type_info varchar(64) -- e.g. 'varchar(255) default NULL'
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.columns
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND column_name = p_column_name;

    IF object_count = 0 THEN
        CALL node_run_sql_v1(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' add column ', p_column_name, ' ', p_column_type_info));
    ELSE
        SELECT CONCAT(
            'Column ',
            p_column_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

CALL node_add_column_v1(
    'node',
    'build_version',
    'varchar(255) DEFAULT NULL');

CALL node_add_column_v1(
    'node',
    'last_boot_ms',
    'bigint DEFAULT NULL');

DROP PROCEDURE IF EXISTS node_add_column_v1;

DROP PROCEDURE IF EXISTS node_run_sql_v1;

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set tabstop=4 shiftwidth=4 expandtab:

Module `stroom-security`

Script `V07_05_00_005__api_key_relax_uniqeness.sql`

Path: stroom-security/stroom-security-impl-db/src/main/resources/stroom/security/impl/db/migration/V07_05_00_005__api_key_relax_uniqeness.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_create_non_unique_index$$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_create_non_unique_index (
    p_table_name varchar(64),
    p_index_name varchar(64),
    p_index_columns varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        CALL security_run_sql(CONCAT(
            'create index ', p_index_name,
            ' on ', database(), '.', p_table_name,
            ' (', p_index_columns, ')'));
    ELSE
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_drop_index $$

-- e.g. security_drop_index('MY_TABLE', 'MY_IDX');
CREATE PROCEDURE security_drop_index (
    p_table_name varchar(64),
    p_index_name varchar(64)
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.statistics
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND index_name = p_index_name;

    IF object_count = 0 THEN
        SELECT CONCAT(
            'Index ',
            p_index_name,
            ' does not exist on table ',
            database(),
            '.',
            p_table_name);
    ELSE
        CALL security_run_sql(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' drop index ', p_index_name));
    END IF;
END $$

-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- We need to make this column case sensitive (_as_cs) else we limit the range of keys we can generate
-- as the keys have mixed case.
-- Note the api_key_prefix col contains lower case data so can stay as _ai_ci
ALTER TABLE api_key MODIFY
    api_key_hash VARCHAR(255)
      CHARACTER SET utf8mb4
      COLLATE utf8mb4_0900_as_cs
      NOT NULL;

-- Drop the old unique index so we can re-create it as non-unique
CALL security_drop_index(
    "api_key",
    "api_key_prefix_idx");

-- We have to look up records by prefix. This will usually return 1 row
-- but may return >1. We test the hash of all returned rows.
CALL security_create_non_unique_index(
    "api_key",
    "api_key_prefix_idx",
    "api_key_prefix");

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_create_non_unique_index;

DROP PROCEDURE IF EXISTS security_drop_index;

DROP PROCEDURE IF EXISTS security_run_sql;

-- --------------------------------------------------

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set shiftwidth=4 tabstop=4 expandtab:

Script `V07_05_00_010__api_key_add_algo_column.sql`

Path: stroom-security/stroom-security-impl-db/src/main/resources/stroom/security/impl/db/migration/V07_05_00_010__api_key_add_algo_column.sql

-- ------------------------------------------------------------------------
-- Copyright 2020 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------

-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;

-- --------------------------------------------------

DELIMITER $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_run_sql $$

-- DO NOT change this without reading the header!
CREATE PROCEDURE security_run_sql (
    p_sql_stmt varchar(1000)
)
BEGIN

    SET @sqlstmt = p_sql_stmt;

    SELECT CONCAT('Running sql: ', @sqlstmt);

    PREPARE stmt FROM @sqlstmt;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_add_column$$

CREATE PROCEDURE security_add_column (
    p_table_name varchar(64),
    p_column_name varchar(64),
    p_column_type_info varchar(64) -- e.g. 'varchar(255) default NULL'
)
BEGIN
    DECLARE object_count integer;

    SELECT COUNT(1)
    INTO object_count
    FROM information_schema.columns
    WHERE table_schema = database()
    AND table_name = p_table_name
    AND column_name = p_column_name;

    IF object_count = 0 THEN
        CALL security_run_sql(CONCAT(
            'alter table ', database(), '.', p_table_name,
            ' add column ', p_column_name, ' ', p_column_type_info));
    ELSE
        SELECT CONCAT(
            'Column ',
            p_column_name,
            ' already exists on table ',
            database(),
            '.',
            p_table_name);
    END IF;
END $$


-- --------------------------------------------------

DELIMITER ;

-- --------------------------------------------------

-- idempotent
CALL security_add_column(
    "api_key",
    "hash_algorithm",
    'tinyint NOT NULL default 0');

-- --------------------------------------------------

DROP PROCEDURE IF EXISTS security_add_column;

DROP PROCEDURE IF EXISTS security_run_sql;

-- --------------------------------------------------

SET SQL_NOTES=@OLD_SQL_NOTES;

-- vim: set shiftwidth=4 tabstop=4 expandtab:

Last modified November 13, 2024: Merge branch '7.4' into 7.5 (03eacad)

Upgrade Notes

Warning

Java Version

Configuration File Changes

Removed Properties

stroom.ui.theme.backgroundColour

New Properties

stroom.contentIndex

state.*

Changed Property Values

stroom.ui.helpSubPathJobs

stroom.ui.helpUrl

Servlets

Warning

Note

Database Migrations

Warning

Warning

Migration Scripts

Module stroom-app

Script V07_05_00_005__Orphaned_Doc_Perms.java

Module stroom-docstore

Script V07_05_00_005__Add_index_on_doc.sql

Module stroom-node

Script V07_05_00_005__add_build_ver_last_boot.sql

Module stroom-security

Script V07_05_00_005__api_key_relax_uniqeness.sql

Script V07_05_00_010__api_key_add_algo_column.sql

`stroom.ui.theme.backgroundColour`

`stroom.contentIndex`

`state.*`

`stroom.ui.helpSubPathJobs`

`stroom.ui.helpUrl`

Module `stroom-app`

Script `V07_05_00_005__Orphaned_Doc_Perms.java`

Module `stroom-docstore`

Script `V07_05_00_005__Add_index_on_doc.sql`

Module `stroom-node`

Script `V07_05_00_005__add_build_ver_last_boot.sql`

Module `stroom-security`

Script `V07_05_00_005__api_key_relax_uniqeness.sql`

Script `V07_05_00_010__api_key_add_algo_column.sql`