This is the multi-page printable view of this section. Click here to print.
Version 7.13
- 1: New Features
- 2: Preview Features (experimental)
- 3: Breaking Changes
- 4: Upgrade Notes
- 5: Change Log
1 - New Features
Node Groups and Processor Profiles
Nodes can now be placed into named Node Groups, and a Processor Profile says which node group does the processing, during which periods of the day and week, and how many threads it may use in each period. A processor filter can then be given a profile, so that processing can be confined to particular nodes, held back until quiet times of day, or given more of the cluster when it is free.
Both are managed from
and
Task creation now takes profiles into account, so tasks that no node is allowed to process are no longer queued, and each profile is given a share of the tasks that are created.
See Also
Document History and Safe Delete
Stroom now keeps a proper audit trail for every document, recording who changed it and when, along with a snapshot of the document data at each change.
Document dependencies are also now held in the database, which makes it possible to ask what a document depends on and what depends on it. This powers the new Safe Delete feature, which tells you what would break before you delete something, and shows you what a folder contains before you delete it.
See Also
Data Generator
A new DataGen document type sends a defined block of data into a Feed Feed A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.Click to see more details... on a schedule. It is intended for generating test data, for example to exercise an analytic rule.
See Also
Tab Sessions
A set of open document tabs can be saved under a name and reopened later, which is useful when you regularly work on the same group of documents, or want to put an investigation down and pick it up again.
Tab sessions are held per user and managed from the Navigation menu, alongside Find and Recent Items, using
Opening a tab session closes the tabs you currently have open, then reopens the saved documents in the order they were saved in.
Pipeline Editing Improvements
- Pipeline stepping can now step across multiple streams rather than stopping at the end of one.
- When saving a pipeline you can choose which of the changed documents it references are saved with it.
- XSLTs and text converters can be created as documents embedded in the pipeline that uses them, rather than as separate items in the explorer tree.
Dashboard and Query Functions
Two functions have been added for pulling structured values apart in dashboard and query tables:
xpath- extract a value from an XML string.jq- extract a value from a JSON string.
The link function now accepts a title, and a new XSLT function computes the similarity of two float vectors.
Search Improvements
- Elasticsearch nested field types are now supported in search.
- Elasticsearch rerank search now supports multiple
dense_vectorfields, and numeric comparators work withfloatanddoublefields. - The number of dimensions used for vector embedding is now configurable, and optional.
Sessions and Tokens
Administrators can now see the sessions a user holds and end them, and revoke the tokens issued to that user. Any user can end all of their own sessions other than the one they are using.
See Also
Account Self Service
An account locked by repeated failed sign-ins now unlocks itself after a period rather than needing an administrator, and the ‘Forgot password’ flow has been rebuilt so that a user can complete a reset from an emailed link. The three account states, Enabled, Locked and Inactive, are now independent of one another, each with a single owner.
See Also
Authenticating Edge Proxies
Stroom can now sit behind a proxy that has already authenticated the user, such as an AWS Application Load Balancer with Cognito, or NGINX with oauth2-proxy.
In this arrangement the proxy is the OpenID Connect relying party and Stroom trusts the identity it passes on, rather than running a sign-in flow of its own.
See Also
Other Authentication Changes
- The signing keys used by the internal identity provider are now rotated automatically, controlled by
stroom.security.identity.token.jwkRotationInterval. - Provider specific parameters can be added to the OpenID authentication request using
authenticationRequestExtraParams, which is how Google is asked for a refresh token. - An optional
requiredAccessTokenTypesetting refuses a token of the wrong type on the API, so that anid_tokencannot be used in place of an access token. - The Stroom user interface no longer needs redirects to authenticate, so it can be served from another location by a backend for frontend proxy.
See Also
Visualisation Assets
A Visualisation now has an Assets tab holding the files it is built from, e.g. JavaScript, CSS, HTML and images, in a folder structure that can be edited and uploaded to through the user interface.
See Also
Git Repositories
Git repositories can now be given HTTP and TLS configuration, so that Stroom can reach a repository through a proxy or one that presents a private certificate.
See Also
Content Index
The content index is no longer rebuilt on first use after a node restarts.
It can now be held locally on each node for performance, or on shared storage, controlled by stroom.contentIndex.storageType.
JSON Parsing Limits
The JSONParser pipeline element now streams string values to the downstream elements rather than reading each one into memory, and has new properties to protect Stroom from very large or deeply nested documents.
See Also
Smaller Changes
- Multiple instances of the same dashboard can be opened at once.
- A new System Info admin servlet, and a menu of all admin servlets at
<admin port>/<admin path>/menu. - Standard annotation comments are now content rather than configuration.
- Annotation history entries are grouped and expandable, and annotation decoration of tables is faster.
- Additional properties on the
S3Appender. - Query table customisations can be reset.
- Node selection for node groups supports select all and inverting the selection.
- Editing a document only marks it as needing to be saved once something has actually changed.
2 - Preview Features (experimental)
Ask Stroom AI
Ask Stroom AI, introduced as a preview feature in v7.11, has been substantially improved and remains a preview feature.
- The chat can be docked as a panel rather than only opening as a dialog.
- Chat history is kept, and individual messages and attachments can be viewed, opened or deleted.
- Tables can be attached to a conversation, and are named so that the model can refer to them by source.
- Larger tables are analysed in batches and the results merged, rather than the request simply being too big.
- Answers can be copied and downloaded, and messages show when they were sent.
- Use of Ask Stroom AI is now recorded in the event log.
The chat history is held in the database, which is why a new ai configuration branch appears in this release.
See Also
Dense Vector Search
Dense vector fields, introduced as a preview feature in v7.11, remain experimental in both the Lucene and Elasticsearch implementations.
- Search results can be reranked using a model, including across multiple
dense_vectorfields in Elasticsearch. - The number of dimensions used for embedding is configurable, and can be left unset.
- A new XSLT function computes the similarity of two float vectors.
3 - Breaking Changes
Warning
Please read this section carefully in case any of the changes affect you.Most of the breaking changes in this release relate to authentication. If you use an external identity provider, read Audience Validation Is Now on by Default and External Identity Provider Redirect URI before upgrading. If you use Stroom’s internal identity provider, read Duplicate Account Email Addresses before upgrading, as the database migration will stop with an error if any are found.
Stroom
Breaking changes relating to Stroom.
A number of configuration properties have been removed. If any of these properties have been set in the YAML configuration file, Stroom will no longer boot. Some properties have also had their default values changed. See Upgrade Notes for details.
Authentication and Accounts
These changes affect Stroom’s own sign-in and account handling. Further authentication changes that affect Stroom-Proxy as well are in Stroom & Stroom-Proxy below.
Duplicate Account Email Addresses
Account email addresses must now be unique, so that the ‘Forgot password’ flow can identify an account from the email address it is given. An account may still have no email address at all, and any number of accounts may have none.
If two or more accounts currently share an email address then the database migration will stop with an error and Stroom will not start. See Upgrade Notes for how to check for this before upgrading.
External Identity Provider Redirect URI
If you use an external identity provider, the client registered at that provider must now list Stroom’s sign-in callback as an allowed redirect URI. A provider that does not have it registered will reject the login.
See Upgrade Notes.
Password Complexity Regex Removed
The internal identity provider no longer supports a character class complexity regex for passwords.
Password strength is now checked on the server using the same estimator the sign-in screen already showed, governed by minimumPasswordStrength, a score of 0 to 4 that defaults to 3.
If your configuration sets passwordComplexityRegex then remove it, as the property no longer exists and Stroom will not boot with it set.
Account Lockout Now Expires
An account locked by repeated failed sign-ins is now unlocked automatically after stroom.security.identity.failedLoginLockDuration, which defaults to 30 minutes, rather than staying locked until an administrator intervenes.
This removes a denial of service in which a few failed sign-ins could lock any named user out permanently.
Set the duration to "PT0S" if you need locks to be permanent.
Note that the duration governs locks that are already held rather than only new ones, so shortening it releases people who are already locked.
See Also
Administrators Can No Longer Lock Accounts
The three account states now have one owner each. An administrator decides whether an account is Enabled, the sign-in process Locks an account after repeated wrong passwords, and the account maintenance job marks an unused account Inactive. An administrator can still undo the latter two, but can no longer apply them.
To prevent an account being used, disable it.
Two kinds of account are converted to disabled on upgrade, so review the accounts list afterwards. See Upgrade Notes.
Account Fields Renamed
The account fields have been renamed in the REST API and in the accounts screen’s quick filter.
| Was | Now |
|---|---|
loginFailures |
failureCount |
failureLocked |
failureLockedMs, the time the lock was applied |
status:Locked, status:Enabled, status:Inactive, status:Disabled |
locked:true, enabled:true, inactive:true |
The status term has been removed rather than renamed, as the three states are now independent of one another.
Update any saved quick filters or scripts that use status: or loginFailures.
Internal Identity Provider Request Validation
The internal identity provider’s authorization endpoint now requires response_type=code, a nonce and the openid scope, and rejects a request that omits any of them.
Stroom’s own sign-in sends all three, so no change is needed for a normal deployment.
If you have set requestScopes to an empty list then sign-in will now fail, as Stroom omits the scope parameter entirely.
Restore the default, or ensure the list contains openid.
Removed email.allowPasswordResets Property
This property has been removed, so Stroom will not boot if it is set.
Nothing read it, so if you set it to false believing it switched password reset emails off, it did not.
Password resets are governed by allowPasswordResets under the password policy section.
CSRF Checks on Browser Requests
Cross site request forgery checks now apply to state-changing requests whose credential was injected by an authenticating edge proxy, where previously only session cookie identities were checked.
An in-browser client that attaches its own bearer token must now send an X-CSRF: 1 header on state-changing requests when edgeAuthentication.enabled is set.
Automation outside a browser, and traffic between cluster nodes, are unaffected.
Other Changes
Removal of HBase Statistics
The HBase backed statistics store, i.e. StroomStatsStore, has been removed along with all of its configuration.
See Upgrade Notes for the properties concerned.
Removal of the ScyllaDB State Store
The ScyllaDB backed state store has been removed, along with the whole state configuration branch.
Use a Plan B store instead.
Standard Annotation Comments
Standard annotation comments are now content rather than configuration.
Any comments previously configured in the standardComments property will need to be saved as annotation comments.
JSON Parser Limits
The JSONParser pipeline element now applies limits to protect Stroom from very large or deeply nested documents.
| Property | Default | Effect |
|---|---|---|
stringTruncateLength |
10,000 | String values longer than this are truncated. |
maxStringLength |
100,000,000 | A string value longer than this is a fatal error. |
maxDepth |
500 | Nesting deeper than this is a fatal error. |
If you parse JSON containing string values longer than 10,000 characters then they will now be truncated unless you raise stringTruncateLength.
The underlying JSON library has also been uplifted to a new major version, so the output of the JSONParser element should be checked to confirm it is still as expected.
Stroom-Proxy
There are no Stroom-Proxy specific breaking changes in v7.13. The authentication changes below apply to Stroom-Proxy as well as Stroom.
Stroom & Stroom-Proxy
Breaking changes that are common to both Stroom and Stroom Proxy.
Authentication
These apply wherever an external identity provider is used, so to Stroom and to Stroom-Proxy.
Audience Validation is Now on by Default
The audience, i.e. aud, claim of a token from an external identity provider is now validated.
Previously, if allowedAudiences was not configured then no audience validation was performed at all, so a token minted for a different application at the same provider could be replayed against Stroom.
Where allowedAudiences is empty the audience is now validated against the configured clientId instead.
If your provider issues tokens to Stroom whose audience is not the clientId, e.g. an API or resource identifier, then list the expected values under allowedAudiences.
Stroom also now refuses to start with an external identity provider unless one of allowedAudiences or clientId is configured, so that validation cannot be silently skipped for lack of anything to validate against.
See Also
An Audience Claim is Now Required
audienceClaimRequired now defaults to true, so an access token from an external identity provider that carries no aud claim is rejected.
Some Cognito and Okta access token shapes have no aud claim.
If yours is one of them, set audienceClaimRequired to false to restore the previous behaviour.
Removal of the TEST_CREDENTIALS Identity Provider Type
This identity provider type has been removed, along with the publicly known credentials it shipped with.
Stroom and Stroom-Proxy will now fail validation on start if identityProviderType is set to it.
Use INTERNAL_IDP for Stroom, and NO_IDP or EXTERNAL_IDP for Stroom-Proxy.
A replacement is available for test and demonstration environments, but it is off unless deliberately enabled at runtime and must never be used in production.
See Also
4 - Upgrade Notes
Warning
Please read this section carefully in case any of it is relevant to your Stroom/Stroom-Proxy instance.Upgrade Path
You can upgrade to v7.13.x from any v7.x release that is older than the version being upgraded to.
If you want to upgrade to v7.13.x from v5.x or v6.x we recommend you do the following:
- Upgrade v5.x to the latest patch release of v6.0.
- Upgrade v6.x to the latest patch release of v7.0.
- Upgrade v7.x to the latest patch release of v7.13.
Warning
v7.13 cannot migrate content in legacy formats, i.e. content created in v5/v6. You must therefore upgrade to v7.0.x first to migrate this content, before upgrading to v7.13.x.Java Version
Stroom v7.13 requires Java 25.
Warning
This is different to the java version required for Stroom v7.9 (Java 21).Ensure the Stroom and Stroom-Proxy hosts are running the latest patch release of Java v25.
Before Upgrading
Duplicate Account Email Addresses
This applies only if you use Stroom’s internal identity provider.
Account email addresses must now be unique, so that the ‘Forgot password’ flow can identify an account from the email address it is given. The database migration will stop with an error listing the addresses concerned if any are shared, and Stroom will not start.
Check for shared addresses before upgrading.
SELECT email, COUNT(1), GROUP_CONCAT(user_id)
FROM account
WHERE email IS NOT NULL
GROUP BY email
HAVING COUNT(1) > 1;
If this returns any rows then give each of those accounts its own email address, or clear the address of all but one of them. An account with no email address is fine, and any number of accounts may have none, but such an account cannot reset its password by email.
External Identity Provider Redirect URI
This applies only if you use an external identity provider, e.g. Keycloak, AWS Cognito or Google.
The client registered at the provider must list Stroom’s sign-in callback under its valid redirect URIs, substituting your Stroom public URL.
https://<stroom-host>/api/auth/flow/v1/signin-oidc
This is the single redirect_uri that Stroom sends on the authorization request, so a provider that does not have this exact value registered will reject the login with an invalid redirect URI error.
If you also use provider-side logout, add the post logout landing page to the client’s valid post logout redirect URIs.
https://<stroom-host>/
The internal identity provider needs no configuration for this.
See Also
Upgrading from a 7.13 Beta Release
This applies only if you have deployed a 7.13 beta release up to and including v7.13-beta.10.
The database migration scripts in those releases were named V07_14... rather than V07_13..., and have since been renamed to match the branch.
The migration will fail on the next release unless the recorded script names are updated first.
Run the following script against the Stroom database before starting the new version.
v07_13_migration_script_rename.sqlAfter Upgrading
Review Disabled Accounts
This applies only if you use Stroom’s internal identity provider.
An administrator can no longer lock an account, so two kinds of account are converted to disabled by the migration and will appear in the accounts list as such.
- Any account that was locked at the point of upgrade. Locking was how an administrator barred an account, and disabling is now the control for that.
- An account flagged as a processing account, a setting that has been removed.
This fails closed, i.e. access stays barred until an administrator acts, so review the accounts list after upgrading and re-enable anything that was disabled in error.
See Also
Configuration File Changes
Stroom’s config.yml
New ai Branch
The following ai branch has been added for configuring the database connection details for the ai module.
You should not need to configure any of this unless you want a dedicated database for this module.
appConfig:
ai:
db:
connection:
jdbcDriverClassName: null
jdbcDriverPassword: null
jdbcDriverUrl: null
jdbcDriverUsername: null
connectionPool:
cachePrepStmts: false
connectionTimeout: "PT30S"
idleTimeout: "PT10M"
leakDetectionThreshold: "PT0S"
maxLifetime: "PT30M"
maxPoolSize: 30
minimumIdle: 10
prepStmtCacheSize: 25
prepStmtCacheSqlLimit: 256
Changes to askStroomAi Branch
This branch controls the Ask Stroom AI feature.
The following properties have been removed.
appConfig:
askStroomAi:
chatMemory:
timeToLive:
time: 1
timeUnit: "HOURS"
tokenLimit: 30000
tableSummary:
maximumBatchSize: 16384
maximumTableInputRows: 100
The following properties have been added.
appConfig:
askStroomAi:
tableAnalysis:
maxTotalRows: 10000
maxRowsPerBatch: 1000
maxParallelBatches: 4
tableQuerySystemPrompt: " You are a data analysis AI. You will answer user\
\ questions using ONLY the markdown-formatted DATA TABLE records provided.\
\ If the records do not contain relevant details, say \"No relevant information.\"\
\n"
tableQueryUserPrompt: " CONVERSATION CONTEXT:\n {{context}}\n\n USER\
\ QUERY:\n {{query}}\n\n DATA TABLE:\n {{table}}\n\n Provide findings\
\ relevant only to these records, in a concise structured format. Use\
\ the conversation context to understand what has been previously discussed.\n"
multiSummaryMergePrompt: " Merge the following summaries into a single unified,\
\ concise summary. Preserve important details, numerical findings, and\
\ remove duplicates.\n\n {{summaries}}\n"
chatSystemPrompt: "You are a helpful data analysis assistant within the Stroom\
\ data platform. When table data is attached to the conversation, it appears\
\ as markdown tables prefixed with [Attached Table: ...] labels identifying\
\ the source. Use data from all relevant attached tables to answer the user's\
\ questions. If multiple tables are present, cite the source table name in your\
\ answer. If you don't have enough information, say so."
historySummaryPrompt: "Summarise the following conversation history in 2-3 concise\
\ sentences. Preserve key facts, decisions, data findings, and any table names\
\ or sources referenced. Do not include greetings or filler."
maxHistorySafetyCapMessages: 200
attachmentDownloadTimeoutMs: 60000
enableDebugDetail: true
New contentIndex Branch
This section is used for controlling the indexing of Stroom’s content.
appConfig:
contentIndex:
contentIndexDir: "content_index"
minRebuildAge: "PT1M"
storageType: "LOCAL"
New docstore Properties
appConfig:
docstore:
docRefInfoCache:
expireAfterAccess: "PT10M"
expireAfterWrite: null
maximumSize: 1000000
refreshAfterWrite: null
statisticsMode: "INTERNAL"
docRefNameCache:
expireAfterAccess: "PT10M"
expireAfterWrite: null
maximumSize: 1000000
refreshAfterWrite: null
statisticsMode: "INTERNAL"
physicalDeleteAge: "P30D"
Changes to explorer Branch
The following properties have been removed.
appConfig:
explorer:
docRefInfoCache:
expireAfterAccess: "PT10M"
expireAfterWrite: null
maximumSize: 1000
refreshAfterWrite: null
statisticsMode: "INTERNAL"
Changes to node Branch
The following properties have been added.
appConfig:
node:
nodeGroupCache:
expireAfterAccess: null
expireAfterWrite: "PT1H"
maximumSize: 1000
refreshAfterWrite: "PT10S"
statisticsMode: "INTERNAL"
Changes to pipeline Branch
The following property has been added.
appConfig:
pipeline:
parser:
disableExternalEntities: true
Changes to processor Branch
The following properties have been added.
appConfig:
processor:
processorProfileCache:
expireAfterAccess: null
expireAfterWrite: "PT1H"
maximumSize: 1000
refreshAfterWrite: "PT10S"
statisticsMode: "INTERNAL"
Changes to the security Branch.
The following configuration properties have been removed.
appConfig:
security:
authorisation:
userCache:
expireAfterAccess: "PT30M"
identity:
email:
allowPasswordResets: false
passwordResetUrl: "/s/resetPassword/?user=%s&token=%s"
openid:
refreshTokenCache:
expireAfterAccess: "P1D"
expireAfterWrite: null
maximumSize: 10000
refreshAfterWrite: null
statisticsMode: "INTERNAL"
passwordPolicy:
passwordComplexityRegex: ".*"
The following new properties have been added.
appConfig:
security:
authentication:
csrf:
protectBrowserOriginatedRequests: true
edgeAuthentication:
enabled: false
logout:
cookiesToExpire: []
signOutUrl: null
openId:
authenticationRequestExtraParams: {}
requiredAccessTokenType: null
validateAudience: true
identity:
passwordResetRequestCooldown: "PT1M"
reactivateInactiveAccountsOnLogin: false
token:
jwkRotationInterval: "P30D"
The default value of passwordPolicyMessage has changed from
To conform with our Strong Password policy, you are
required to use a sufficiently strong password. Password must be more
than 8 characters.
To
To conform with our Strong Password policy, you are
required to use a sufficiently strong password. Password must be at least
8 characters.
The default value of audienceClaimRequired has changed from false to true.
appConfig:
security:
authentication:
openId:
audienceClaimRequired: true
The default cache configuration for userCache has changed from expireAfterAccess to expireAfterWrite.
appConfig:
security:
authorisation:
userCache:
expireAfterAccess: "PT30M"
Removed state Branch
The state config was used for a ScyllaDB backed sate store which has now been removed from Stroom.
appConfig:
state:
scyllaDbDocCache:
expireAfterAccess: null
expireAfterWrite: "PT10M"
maximumSize: 100
refreshAfterWrite: null
statisticsMode: "INTERNAL"
sessionCache:
expireAfterAccess: "PT1H"
expireAfterWrite: null
maximumSize: 10
refreshAfterWrite: null
statisticsMode: "INTERNAL"
stateDocCache:
expireAfterAccess: null
expireAfterWrite: "PT10M"
maximumSize: 100
refreshAfterWrite: null
statisticsMode: "INTERNAL"
Changes to the statistics Branch
The following properties for the HBase based statistics store have been removed along with this statistics store functionality.
appConfig:
statistics:
hbase:
docRefType: "StroomStatsStore"
eventsPerMessage: 100
kafkaConfigUuid: null
kafkaTopics:
count: "statisticEvents-Count"
value: "statisticEvents-Value"
internal:
benchmarkCluster:
- type: "StroomStatsStore"
uuid: "2503f703-5ce0-4432-b9d4-e3272178f47e"
name: "Benchmark-Cluster Test"
cpu:
- type: "StroomStatsStore"
uuid: "1edfd582-5e60-413a-b91c-151bd544da47"
name: "CPU"
enabledStoreTypes:
- "StatisticStore"
eventsPerSecond:
- type: "StroomStatsStore"
uuid: "cde67df0-0f77-45d3-b2c0-ee8bb7b3c9c6"
name: "EPS"
heapHistogramBytes:
- type: "StroomStatsStore"
uuid: "b0110ab4-ac25-4b73-b4f6-96f2b50b456a"
name: "Heap Histogram Bytes"
heapHistogramInstances:
- type: "StroomStatsStore"
uuid: "bdd933a4-4309-47fd-98f6-1bc2eb555f20"
name: "Heap Histogram Instances"
memory:
- type: "StroomStatsStore"
uuid: "d8a7da4f-ef6d-47e0-b16a-af26367a2798"
name: "Memory"
metaDataStreamSize:
- type: "StroomStatsStore"
uuid: "3b25d63b-5472-44d0-80e8-8eea94f40f14"
name: "Meta Data-Stream Size"
metaDataStreamsReceived:
- type: "StroomStatsStore"
uuid: "5535f493-29ae-4ee6-bba6-735aa3104136"
name: "Meta Data-Streams Received"
pipelineStreamProcessor:
- type: "StroomStatsStore"
uuid: "efd9bad4-0bab-460f-ae98-79e9717deeaf"
name: "PipelineStreamProcessor"
refDataStoreEntryCount:
- type: "StroomStatsStore"
uuid: "TODO"
name: "Reference Data Store Entry Count"
refDataStoreSize:
- type: "StroomStatsStore"
uuid: "TODO"
name: "Reference Data Store Size"
refDataStoreStreamCount:
- type: "StroomStatsStore"
uuid: "TODO"
name: "Reference Data Store Stream Count"
searchResultsStoreCount:
- type: "StroomStatsStore"
uuid: "TODO"
name: "Search Results Store Count"
searchResultsStoreSize:
- type: "StroomStatsStore"
uuid: "TODO"
name: "Search Results Store Size"
streamTaskQueueSize:
- type: "StroomStatsStore"
uuid: "4ce8d6e7-94be-40e1-8294-bf29dd089962"
name: "Stream Task Queue Size"
volumes:
- type: "StroomStatsStore"
uuid: "60f4f5f0-4cc3-42d6-8fe7-21a7cec30f8e"
name: "Volumes"
New visualisationAsset Properties
appConfig:
visualisationAsset:
aceEditorModes:
css: "CSS"
htm: "HTML"
html: "HTML"
js: "JAVASCRIPT"
svg: "XML"
txt: "TEXT"
xml: "XML"
assetCacheDir: "asset_cache"
clearAssetCacheOnStartup: false
default: "application/octet-stream"
defaultAceEditorMode: "TEXT"
mimetypes:
apng: "image/apng"
bmp: "image/bmp"
css: "text/css"
gif: "image/jpeg"
htm: "text/html"
html: "text/html"
jpeg: "image/jpeg"
jpg: "image/jpeg"
js: "text/javascript"
png: "image/png"
svg: "image/svg+xml"
tif: "image/tiff"
tiff: "image/tiff"
txt: "text/plain"
webp: "image/webp"
xml: "application/xml"
visualisationAssetDb:
connection:
jdbcDriverClassName: null
jdbcDriverPassword: null
jdbcDriverUrl: null
jdbcDriverUsername: null
connectionPool:
cachePrepStmts: false
connectionTimeout: "PT30S"
idleTimeout: "PT10M"
leakDetectionThreshold: "PT0S"
maxLifetime: "PT30M"
maxPoolSize: 30
minimumIdle: 10
prepStmtCacheSize: 25
prepStmtCacheSqlLimit: 256
Changes to the ui Branch
The values of the following properties have changed, so that the in-application help links point at the right pages of this documentation site. These are only relevant if you have overridden them, e.g. to point at a locally published copy of the documentation.
| Property | Was | Now |
|---|---|---|
stroom.ui.helpUrl |
https://gchq.github.io/stroom-docs/7.5/docs |
https://gchq.github.io/stroom-docs/7.13/docs |
stroom.ui.helpSubPathExpressions |
/user-guide/dashboards/expressions/ |
/reference-section/expressions/ |
stroom.ui.helpSubPathQuickFilter |
/user-guide/finding-things/ |
/user-guide/content/finding-things/ |
stroom.ui.helpSubPathStroomQueryLanguage |
/user-guide/dashboards/stroom-query-language/ |
/user-guide/search/queries/stroom-query-language/ |
Stroom-Proxy’s config.yml
Changes to the security Branch.
The following new properties have been added.
appConfig:
security:
authentication:
openId:
authenticationRequestExtraParams: {}
requiredAccessTokenType: null
validateAudience: true
The default value of audienceClaimRequired has changed from false to true.
appConfig:
security:
authentication:
openId:
audienceClaimRequired: true
Database Migrations
When Stroom boots for the first time with a new version it will run any required database migrations to bring the database schema up to the correct version.
Warning
It is highly recommended to ensure you have a database backup in place before booting stroom with a new version. This is to mitigate against any problems with the migration. It is also recommended to test the migration against a copy of your database to ensure that there are no problems when you do it for real.On boot, Stroom will ensure that the migrations are only run by a single node in the cluster. This will be the node that reaches that point in the boot process first. All other nodes will wait until that is complete before proceeding with the boot process.
It is recommended however to use a single node to execute the migration.
To avoid Stroom starting up and beginning processing you can use the migrage command to just migrate the database and not fully boot Stroom.
See migrage command for more details.
Migration Scripts
For information purposes only, the following are the database migrations that will be run when upgrading to 7.13.0 from the previous minor version.
Note, the legacy module will run first (if present) then the other module will run in no particular order.
Module stroom-ai
Script V07_13_00_001__ai.sql
Path: stroom-ai/stroom-ai-impl-db/src/main/resources/stroom/ai/impl/db/migration/V07_13_00_001__ai.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
--
-- Create the ai_chat table
--
CREATE TABLE IF NOT EXISTS ai_chat (
id int NOT NULL AUTO_INCREMENT,
version int NOT NULL,
create_time_ms bigint NOT NULL,
create_user varchar(255) NOT NULL,
update_time_ms bigint NOT NULL,
update_user varchar(255) NOT NULL,
user_uuid varchar(255) NOT NULL,
title varchar(255) NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
--
-- Create the ai_chat_attachment table
--
CREATE TABLE IF NOT EXISTS ai_chat_attachment (
id int NOT NULL AUTO_INCREMENT,
fk_ai_chat_id int NOT NULL,
create_time_ms bigint NOT NULL,
update_time_ms bigint NOT NULL,
status int NOT NULL,
attachment_type int NOT NULL,
description varchar(255),
context_json longtext,
row_count int,
truncated tinyint(1) NOT NULL DEFAULT 0,
error_message varchar(1024),
PRIMARY KEY (id),
CONSTRAINT fk_ai_chat_attachment_chat
FOREIGN KEY (fk_ai_chat_id) REFERENCES ai_chat (id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
--
-- Create the ai_chat_message table
--
CREATE TABLE IF NOT EXISTS ai_chat_message (
id int NOT NULL AUTO_INCREMENT,
fk_ai_chat_id int NOT NULL,
create_time_ms bigint NOT NULL,
message_type int NOT NULL,
fk_attachment_id int DEFAULT NULL,
message longtext NOT NULL,
PRIMARY KEY (id),
CONSTRAINT fk_ai_chat_message_chat
FOREIGN KEY (fk_ai_chat_id) REFERENCES ai_chat (id) ON DELETE CASCADE,
CONSTRAINT fk_ai_chat_message_attachment
FOREIGN KEY (fk_attachment_id) REFERENCES ai_chat_attachment (id) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
CREATE INDEX idx_ai_chat_user_uuid ON ai_chat (user_uuid);
CREATE INDEX idx_ai_chat_attachment_chat_id ON ai_chat_attachment (fk_ai_chat_id);
CREATE INDEX idx_ai_chat_message_chat_id ON ai_chat_message (fk_ai_chat_id);
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set tabstop=4 shiftwidth=4 expandtab:
Script V07_13_00_002__ai_remove_cascade.sql
Path: stroom-ai/stroom-ai-impl-db/src/main/resources/stroom/ai/impl/db/migration/V07_13_00_002__ai_remove_cascade.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
-- Remove CASCADE DELETE from ai_chat_attachment -> ai_chat
ALTER TABLE ai_chat_attachment
DROP FOREIGN KEY fk_ai_chat_attachment_chat;
ALTER TABLE ai_chat_attachment
ADD CONSTRAINT fk_ai_chat_attachment_chat
FOREIGN KEY (fk_ai_chat_id) REFERENCES ai_chat (id);
-- Remove CASCADE DELETE from ai_chat_message -> ai_chat
ALTER TABLE ai_chat_message
DROP FOREIGN KEY fk_ai_chat_message_chat;
ALTER TABLE ai_chat_message
ADD CONSTRAINT fk_ai_chat_message_chat
FOREIGN KEY (fk_ai_chat_id) REFERENCES ai_chat (id);
-- Remove SET NULL from ai_chat_message -> ai_chat_attachment
ALTER TABLE ai_chat_message
DROP FOREIGN KEY fk_ai_chat_message_attachment;
ALTER TABLE ai_chat_message
ADD CONSTRAINT fk_ai_chat_message_attachment
FOREIGN KEY (fk_attachment_id) REFERENCES ai_chat_attachment (id);
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set tabstop=4 shiftwidth=4 expandtab:
Module stroom-annotation
Script V07_13_00_001__collation_fix.sql
Path: stroom-annotation/stroom-annotation-impl-db/src/main/resources/stroom/annotation/impl/db/migration/V07_13_00_001__collation_fix.sql
-- ------------------------------------------------------------------------
-- Copyright 2016-2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
ALTER TABLE annotation_link CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
ALTER TABLE annotation_subscription CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
ALTER TABLE annotation_tag CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
ALTER TABLE annotation_tag_link CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Module stroom-app
Script V07_13_00_005__populate_doc_dependency_processor_filters.java
Path: stroom-app/src/main/java/stroom/app/db/migration/V07_13_00_005__populate_doc_dependency_processor_filters.java
It is not possible to display the content here. The file can be viewed on : GitHub
Module stroom-docstore
Script V07_13_00_001__split_doc_table.sql
Path: stroom-docstore/stroom-docstore-impl-db/src/main/resources/stroom/docstore/impl/db/migration/V07_13_00_001__split_doc_table.sql
--
-- Copyright 2016-2025 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-- ============================================================================
-- Split doc table into doc (identity) + doc_data (typed content) +
-- doc_audit (operation trail) + doc_data_snapshot (deduplicated snapshots) +
-- doc_audit_data_snapshot (audit-to-snapshot links)
-- ============================================================================
-- stop note level warnings about objects (not)? existing
SET @old_sql_notes=@@sql_notes, sql_notes=0;
-- ---------------------------------------------------------------------------
-- Step 1: Create doc_data table with sparse typed columns
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS doc_data (
id bigint NOT NULL AUTO_INCREMENT,
fk_doc_id bigint NOT NULL,
ext varchar(255) NOT NULL,
data_type tinyint NOT NULL,
json_data json,
text_data longtext,
bin_data longblob,
PRIMARY KEY (id),
UNIQUE KEY doc_data_fk_doc_id_ext_idx (fk_doc_id, ext),
CONSTRAINT doc_data_fk_doc_id FOREIGN KEY (fk_doc_id) REFERENCES doc (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
-- ---------------------------------------------------------------------------
-- Step 2a: Migrate VALID meta JSON -> doc_data.json_data
-- The meta row references itself (fk_doc_id = doc.id where ext = 'meta')
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, json_data)
SELECT id, ext, 1, CAST(CONVERT(data USING utf8mb4) AS JSON)
FROM doc
WHERE ext = 'meta'
AND data IS NOT NULL
AND CONVERT(data USING utf8mb4) IS NOT NULL
AND JSON_VALID(CONVERT(data USING utf8mb4)) = 1;
-- ---------------------------------------------------------------------------
-- Step 2b: Migrate INVALID meta rows -> doc_data.text_data as fallback
-- These rows had data that could not be parsed as JSON.
-- Stored as text for manual inspection and correction.
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, text_data)
SELECT id, ext, 2, CONVERT(data USING utf8mb4)
FROM doc
WHERE ext = 'meta'
AND data IS NOT NULL
AND (CONVERT(data USING utf8mb4) IS NULL
OR JSON_VALID(CONVERT(data USING utf8mb4)) = 0);
-- ---------------------------------------------------------------------------
-- Step 3a: Migrate VALID content JSON -> doc_data.json_data
-- These are non-meta rows with ext = 'json', joined to the meta row
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, json_data)
SELECT dm.id, d.ext, 1, CAST(CONVERT(d.data USING utf8mb4) AS JSON)
FROM doc d
JOIN doc dm ON dm.type = d.type AND dm.uuid = d.uuid AND dm.ext = 'meta'
WHERE d.ext = 'json'
AND d.data IS NOT NULL
AND CONVERT(d.data USING utf8mb4) IS NOT NULL
AND JSON_VALID(CONVERT(d.data USING utf8mb4)) = 1;
-- ---------------------------------------------------------------------------
-- Step 3b: Migrate INVALID content JSON rows -> doc_data.text_data as fallback
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, text_data)
SELECT dm.id, d.ext, 2, CONVERT(d.data USING utf8mb4)
FROM doc d
JOIN doc dm ON dm.type = d.type AND dm.uuid = d.uuid AND dm.ext = 'meta'
WHERE d.ext = 'json'
AND d.data IS NOT NULL
AND (CONVERT(d.data USING utf8mb4) IS NULL
OR JSON_VALID(CONVERT(d.data USING utf8mb4)) = 0);
-- ---------------------------------------------------------------------------
-- Step 4: Migrate text content -> doc_data.text_data
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, text_data)
SELECT dm.id, d.ext, 2, CONVERT(d.data USING utf8mb4)
FROM doc d
JOIN doc dm ON dm.type = d.type AND dm.uuid = d.uuid AND dm.ext = 'meta'
WHERE d.ext IN ('xsl', 'xsd', 'xml', 'js', 'txt');
-- ---------------------------------------------------------------------------
-- Step 5: Migrate remaining content -> doc_data.bin_data
-- ---------------------------------------------------------------------------
INSERT INTO doc_data (fk_doc_id, ext, data_type, bin_data)
SELECT dm.id, d.ext, 3, d.data
FROM doc d
JOIN doc dm ON dm.type = d.type AND dm.uuid = d.uuid AND dm.ext = 'meta'
WHERE d.ext NOT IN ('meta', 'json', 'xsl', 'xsd', 'xml', 'js', 'txt')
AND d.ext IS NOT NULL;
-- ---------------------------------------------------------------------------
-- Step 6: Delete all non-meta rows from doc
-- ---------------------------------------------------------------------------
DELETE FROM doc WHERE ext != 'meta' OR ext IS NULL;
-- ---------------------------------------------------------------------------
-- Step 7: Restructure doc table - drop data/ext, add deleted, update indexes
-- ---------------------------------------------------------------------------
ALTER TABLE doc DROP KEY doc_type_uuid_ext_idx;
ALTER TABLE doc DROP KEY doc_type_uuid_idx;
ALTER TABLE doc DROP KEY doc_uuid_idx;
ALTER TABLE doc DROP KEY doc_type_name_uuid_idx;
ALTER TABLE doc DROP COLUMN data;
ALTER TABLE doc DROP COLUMN ext;
ALTER TABLE doc ADD COLUMN deleted bigint DEFAULT NULL;
ALTER TABLE doc ADD UNIQUE KEY doc_uuid_idx (uuid);
ALTER TABLE doc ADD KEY doc_type_name_uuid_idx (type, name, uuid);
ALTER TABLE doc ADD KEY doc_deleted_idx (deleted);
-- ---------------------------------------------------------------------------
-- Step 8: Create doc_audit table
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS doc_audit (
id bigint NOT NULL AUTO_INCREMENT,
fk_doc_id bigint NOT NULL,
action tinyint NOT NULL,
action_time bigint NOT NULL,
user_uuid varchar(255) DEFAULT NULL,
user_name varchar(255) DEFAULT NULL,
PRIMARY KEY (id),
KEY doc_audit_fk_doc_id_idx (fk_doc_id),
KEY doc_audit_action_time_idx (action_time),
CONSTRAINT doc_audit_fk_doc_id FOREIGN KEY (fk_doc_id) REFERENCES doc (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
-- ---------------------------------------------------------------------------
-- Step 9: Create doc_data_snapshot table
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS doc_data_snapshot (
id bigint NOT NULL AUTO_INCREMENT,
fk_doc_id bigint NOT NULL,
ext varchar(255) NOT NULL,
data_type tinyint NOT NULL,
data_hash bigint NOT NULL,
json_data json,
text_data longtext,
bin_data longblob,
PRIMARY KEY (id),
KEY doc_data_snapshot_dedup_idx (fk_doc_id, ext, data_hash),
KEY doc_data_snapshot_fk_doc_id_idx (fk_doc_id),
CONSTRAINT doc_data_snapshot_fk_doc_id FOREIGN KEY (fk_doc_id) REFERENCES doc (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
-- ---------------------------------------------------------------------------
-- Step 10: Create doc_audit_data_snapshot link table
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS doc_audit_data_snapshot (
id bigint NOT NULL AUTO_INCREMENT,
fk_doc_audit_id bigint NOT NULL,
fk_doc_data_snapshot_id bigint NOT NULL,
PRIMARY KEY (id),
KEY doc_audit_data_snapshot_fk_doc_audit_id_idx (fk_doc_audit_id),
KEY doc_audit_data_snapshot_fk_doc_data_snapshot_id_idx (fk_doc_data_snapshot_id),
CONSTRAINT doc_audit_data_snapshot_fk_doc_audit_id
FOREIGN KEY (fk_doc_audit_id) REFERENCES doc_audit (id),
CONSTRAINT doc_audit_data_snapshot_fk_doc_data_snapshot_id
FOREIGN KEY (fk_doc_data_snapshot_id) REFERENCES doc_data_snapshot (id)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
-- ---------------------------------------------------------------------------
-- Step 11: Seed initial audit entries from existing meta JSON
-- The user_uuid is NULL because the current meta JSON only stores user
-- display names, not their UUIDs.
-- ---------------------------------------------------------------------------
-- CREATE entry from createTimeMs / createUser
INSERT INTO doc_audit (fk_doc_id, action, action_time, user_uuid, user_name)
SELECT d.id, 1,
COALESCE(JSON_VALUE(dd.json_data, '$.createTimeMs' RETURNING SIGNED), 0),
NULL,
JSON_VALUE(dd.json_data, '$.createUser' RETURNING CHAR(255))
FROM doc d
JOIN doc_data dd ON dd.fk_doc_id = d.id AND dd.ext = 'meta'
WHERE dd.json_data IS NOT NULL;
-- UPDATE entry from updateTimeMs / updateUser (only if updateTimeMs is set)
INSERT INTO doc_audit (fk_doc_id, action, action_time, user_uuid, user_name)
SELECT d.id, 2,
COALESCE(JSON_VALUE(dd.json_data, '$.updateTimeMs' RETURNING SIGNED), 0),
NULL,
JSON_VALUE(dd.json_data, '$.updateUser' RETURNING CHAR(255))
FROM doc d
JOIN doc_data dd ON dd.fk_doc_id = d.id AND dd.ext = 'meta'
WHERE dd.json_data IS NOT NULL
AND JSON_VALUE(dd.json_data, '$.updateTimeMs' RETURNING SIGNED) IS NOT NULL;
-- Reset to the original value
SET SQL_NOTES=@OLD_SQL_NOTES;
Script V07_13_00_002__add_version_column.sql
Path: stroom-docstore/stroom-docstore-impl-db/src/main/resources/stroom/docstore/impl/db/migration/V07_13_00_002__add_version_column.sql
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Copyright 2024-2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Add version column for optimistic concurrency control.
-- The version UUID is currently embedded inside the meta JSON blob ($.version).
-- Extract it to a dedicated column so the DB can enforce version matching atomically.
-- stop note level warnings about objects (not)? existing
SET @old_sql_notes=@@sql_notes, sql_notes=0;
-- Step 1: Add the column with a default that lets us identify unpopulated rows.
ALTER TABLE doc ADD COLUMN version varchar(36) NOT NULL DEFAULT '';
-- Step 2: Populate from existing meta JSON.
-- Populate from existing meta JSON stored in doc_data.json_data.
UPDATE doc d
JOIN doc_data dd ON dd.fk_doc_id = d.id AND dd.ext = 'meta'
SET d.version = JSON_UNQUOTE(JSON_EXTRACT(dd.json_data, '$.version'))
WHERE dd.json_data IS NOT NULL
AND JSON_UNQUOTE(JSON_EXTRACT(dd.json_data, '$.version')) IS NOT NULL;
-- Step 3: Fallback — generate a UUID for rows with missing/invalid JSON
-- (e.g. those with meta stored in text_data rather than json_data).
UPDATE doc
SET version = UUID()
WHERE version = '';
-- Reset to the original value
SET SQL_NOTES=@OLD_SQL_NOTES;
Script V07_13_00_003__doc_dependency.sql
Path: stroom-docstore/stroom-docstore-impl-db/src/main/resources/stroom/docstore/impl/db/migration/V07_13_00_003__doc_dependency.sql
--
-- Copyright 2016-2025 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-- ============================================================================
-- Tracks document-to-document dependency edges.
-- Each row means: the doc identified by (from_type, from_uuid) depends on
-- (to_type, to_uuid).
--
-- Uses bare UUIDs (no FK to doc table) because:
-- - Dependency targets may reference docs that don't exist yet (broken deps)
-- - Pseudo-refs (e.g. Annotations, SearchableIndex) are not in the doc table
-- - Non-doc entities (e.g. ProcessorFilter) are not in the doc table
-- ============================================================================
-- stop note level warnings about objects (not)? existing
SET @old_sql_notes=@@sql_notes, sql_notes=0;
CREATE TABLE IF NOT EXISTS doc_dependency (
id BIGINT NOT NULL AUTO_INCREMENT,
from_type VARCHAR(255) NOT NULL,
from_uuid VARCHAR(255) NOT NULL,
from_name VARCHAR(255) NOT NULL DEFAULT '',
to_type VARCHAR(255) NOT NULL,
to_uuid VARCHAR(255) NOT NULL,
to_name VARCHAR(255) NOT NULL DEFAULT '',
PRIMARY KEY (id),
-- A given (from, to) edge should be unique
UNIQUE KEY doc_dependency_from_to (from_uuid, to_uuid),
-- Query pattern: "what does doc X depend on?"
KEY doc_dependency_from_uuid (from_uuid),
-- Query pattern: "what depends on doc X?" (for safe-delete, dependants view)
KEY doc_dependency_to_uuid (to_uuid)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
-- Reset to the original value
SET sql_notes=@old_sql_notes;
Script V07_13_00_004__populate_doc_dependency.java
Path: stroom-docstore/stroom-docstore-impl-db/src/main/java/stroom/docstore/impl/db/migration/V07_13_00_004__populate_doc_dependency.java
It is not possible to display the content here. The file can be viewed on : GitHub
Module stroom-security
Script V07_13_00_005__json_web_key.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_005__json_web_key.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
-- Remove the old defunct test JSON web key inserted by V07_00_00_040__json_web_key_seed_data.sql.
DELETE FROM json_web_key
WHERE key_id = '1ec7a983-317d-46ce-ae93-ce42bc217e52';
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_010__account_self_service_unlock.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_010__account_self_service_unlock.sql
-- ------------------------------------------------------------------------
-- Copyright 2025 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Supports self service account unlocking for the internal identity provider, see GH-5656.
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
DROP PROCEDURE IF EXISTS V07_13_00_010__account_self_service_unlock;
DELIMITER $$
CREATE PROCEDURE V07_13_00_010__account_self_service_unlock ()
BEGIN
DECLARE object_count integer;
DECLARE duplicate_count integer;
-- The SHA-256 hash of the secret in the most recently issued password reset link for the account.
-- The link is an opaque random string; only its hash is held here, so the link cannot be recovered
-- from the database. Issuing a new link replaces this, so an earlier one stops working, and it is
-- cleared whenever the password is set, so a link cannot be used twice or survive a password change.
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'reset_token_hash';
IF object_count = 0 THEN
ALTER TABLE account ADD COLUMN reset_token_hash varchar(64) DEFAULT NULL;
END IF;
-- When the current password reset link expires, as epoch millis. Held explicitly rather than derived
-- so that changing the configured link lifetime does not retroactively change links already issued.
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'reset_token_expiry_ms';
IF object_count = 0 THEN
ALTER TABLE account ADD COLUMN reset_token_expiry_ms bigint DEFAULT NULL;
END IF;
-- When a password reset email was last requested for this account, used to stop the unauthenticated
-- reset endpoint being used to send mail to someone's inbox over and over. Held against the account
-- rather than in a table of its own so that it is bounded by the number of accounts, and against the
-- database rather than in memory so that the limit holds across a cluster.
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'reset_email_requested_ms';
IF object_count = 0 THEN
ALTER TABLE account ADD COLUMN reset_email_requested_ms bigint DEFAULT NULL;
END IF;
-- A blank email address is no email address at all, and is held as NULL. Held as an empty string it
-- would be a value like any other, so the second account left blank would clash with the first under
-- the unique index below, where any number of accounts may have no address. Earlier versions wrote
-- blanks through from the account screen, so normalise what is already stored before indexing it.
UPDATE account SET email = NULL WHERE TRIM(email) = '';
-- 'Forgot password' finds the account to reset by its email address, so an address must identify at
-- most one account. An account may still have no email address at all, in which case it simply
-- cannot be reset by email; a UNIQUE index permits any number of NULLs.
SELECT COUNT(1)
INTO object_count
FROM information_schema.statistics
WHERE table_schema = database()
AND table_name = 'account'
AND index_name = 'account_email_idx';
IF object_count = 0 THEN
-- Adding the index would fail with a bare duplicate key error, naming only the first clash, and
-- only after the columns above had been added. Check first so that the operator is told what is
-- wrong rather than being shown a key name.
SELECT COUNT(1)
INTO duplicate_count
FROM (
SELECT email
FROM account
WHERE email IS NOT NULL
GROUP BY email
HAVING COUNT(1) > 1
) duplicates;
IF duplicate_count > 0 THEN
SET @message_text = CONCAT(
'Cannot make account.email unique: ', duplicate_count,
' email addresses are each used by more than one account');
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = @message_text;
END IF;
ALTER TABLE account ADD UNIQUE KEY account_email_idx (email);
END IF;
END $$
DELIMITER ;
CALL V07_13_00_010__account_self_service_unlock;
DROP PROCEDURE IF EXISTS V07_13_00_010__account_self_service_unlock;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_015__drop_oauth_client_uri_pattern.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_015__drop_oauth_client_uri_pattern.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Drop the unused oauth_client.uri_pattern column. redirect_uri validation is an exact match against
-- the application's public root and does not use this column.
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
DROP PROCEDURE IF EXISTS V07_13_00_015__drop_oauth_client_uri_pattern;
DELIMITER $$
CREATE PROCEDURE V07_13_00_015__drop_oauth_client_uri_pattern ()
BEGIN
DECLARE object_count integer;
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'oauth_client'
AND column_name = 'uri_pattern';
IF object_count = 1 THEN
ALTER TABLE oauth_client DROP COLUMN uri_pattern;
END IF;
END $$
DELIMITER ;
CALL V07_13_00_015__drop_oauth_client_uri_pattern;
DROP PROCEDURE IF EXISTS V07_13_00_015__drop_oauth_client_uri_pattern;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_020__account_lockout_expiry.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_020__account_lockout_expiry.sql
-- ------------------------------------------------------------------------
-- Copyright 2025 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Supports time-limited (auto-expiring) account lockout for the internal identity provider.
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
DROP PROCEDURE IF EXISTS V07_13_00_020__account_lockout_expiry;
DELIMITER $$
CREATE PROCEDURE V07_13_00_020__account_lockout_expiry ()
BEGIN
DECLARE object_count integer;
-- When a failure-driven lock auto-expires, as epoch millis. A lock set by repeated failed logins
-- carries an expiry so that it clears itself after a configured period, rather than needing an
-- administrator. NULL means the lock (if any) never expires: this is the case for a lock set
-- manually by an administrator, and is the default for a freshly created account.
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'locked_until_ms';
IF object_count = 0 THEN
ALTER TABLE account ADD COLUMN locked_until_ms bigint DEFAULT NULL;
END IF;
END $$
DELIMITER ;
CALL V07_13_00_020__account_lockout_expiry;
DROP PROCEDURE IF EXISTS V07_13_00_020__account_lockout_expiry;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_025__oauth_token.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_025__oauth_token.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
--
-- Create the oauth_token table: an inventory of the tokens the INTERNAL IdP has minted, so that they can
-- be listed and revoked before they expire. Externally minted tokens never get a row here - their
-- revocation is the external IdP's business.
--
-- Deliberately NOT called `token`. A legacy `token` table (V07_00_00_025) is dropped by a migration in a
-- DIFFERENT module, stroom-security-impl-db/V07_02_00_115__drop_token_table.sql, which reaches across the
-- schema boundary because cross-module migration order is non-deterministic. That drop is guarded only by
-- an existence check, so it cannot tell a brand new table from the legacy one - reusing the name would
-- mean silent data loss on whichever install orderings ran identity's migrations first.
--
CREATE TABLE IF NOT EXISTS oauth_token (
id int NOT NULL AUTO_INCREMENT,
version int NOT NULL,
create_time_ms bigint NOT NULL,
create_user varchar(255) NOT NULL,
update_time_ms bigint NOT NULL,
update_user varchar(255) NOT NULL,
-- ACCESS | ID | REFRESH. A plain discriminator, NOT an FK to the legacy token_type table.
token_type varchar(10) NOT NULL,
-- The JWT id, for ACCESS/ID rows only. Refresh tokens are opaque random strings, not JWTs, and have
-- no jti - which is why this table has a surrogate id PK rather than keying on jti.
jti varchar(255) DEFAULT NULL,
-- SHA-256 of the opaque refresh token, for REFRESH rows only. The redeemable credential is looked up
-- by presentation and must never be stored in the clear.
token_hash varchar(255) DEFAULT NULL,
-- A subject string, NOT an FK to account.id: service and external subjects have no account row.
subject_id varchar(255) NOT NULL,
client_id varchar(255) DEFAULT NULL,
-- Rotation lineage for refresh tokens (grant id for access/id), so reuse can revoke a whole family.
family_id varchar(255) DEFAULT NULL,
-- The scope granted at authentication, carried forward onto every successor token in the family.
-- Needed to mint the successor when a refresh token is redeemed.
scope longtext,
-- When the end user actually authenticated. Carried forward so a refreshed id token reports the
-- original login time rather than the time of the refresh.
auth_time_ms bigint DEFAULT NULL,
issued_ms bigint NOT NULL,
expires_ms bigint NOT NULL,
-- Set when a refresh token is redeemed. The row is KEPT until it expires rather than being deleted,
-- because a consumed-but-unexpired row is what makes a replay of that token detectable for the whole of
-- its lifetime.
consumed_ms bigint DEFAULT NULL,
revoked tinyint NOT NULL DEFAULT '0',
revoked_ms bigint DEFAULT NULL,
revoked_by varchar(255) DEFAULT NULL,
PRIMARY KEY (id),
-- Two nullable natural keys, exactly one of which is populated per token_type.
UNIQUE KEY oauth_token_jti_idx (jti),
UNIQUE KEY oauth_token_token_hash_idx (token_hash),
-- Admin grouping and revoke-by-user.
KEY oauth_token_subject_id_idx (subject_id),
-- Family revocation on refresh reuse.
KEY oauth_token_family_id_idx (family_id),
-- Drives both the read-time `expires_ms > now` predicate and the purge job.
KEY oauth_token_expires_ms_idx (expires_ms),
-- The revoked-and-still-live lookup that builds the verify path's denylist. `revoked` leads because it
-- is the selective column - almost no rows are revoked - whereas `expires_ms > now` matches nearly
-- every row and so cannot narrow anything on its own.
KEY oauth_token_revoked_idx (revoked, expires_ms)
) ENGINE=InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_030__drop_processing_account.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_030__drop_processing_account.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Removes account.processing_account, which nothing sets and which grants nothing.
--
-- The flag marked an account as a non-interactive identity: hidden from the account list and search,
-- and refused both interactive sign in and password reset. No code path ever set it. Every migration
-- that populates the table writes false, and the create request carries no field for it, so the only
-- writer was an account update accepting the whole account object from the caller - which no longer
-- happens now that an update carries only the fields being changed.
--
-- It is also not how either machine identity in Stroom works. The internal processing user holds a
-- short lived self issued token and has no account row at all, and a data sender authenticating with
-- a client certificate is identified by the certificate's common name, deliberately without an account.
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
DROP PROCEDURE IF EXISTS V07_13_00_030__drop_processing_account;
DELIMITER $$
CREATE PROCEDURE V07_13_00_030__drop_processing_account ()
BEGIN
DECLARE object_count integer;
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'processing_account';
IF object_count = 1 THEN
-- Fail closed before the column goes. Such an account is currently barred from signing in, so
-- dropping the column without this would silently turn it into an ordinary, usable account.
-- Disabling is the control that expresses "barred" now, and unlike the flag an administrator
-- can see it. This is expected to match no rows on any database that has not been hand edited.
UPDATE account
SET enabled = 0
WHERE processing_account = 1;
ALTER TABLE account DROP COLUMN processing_account;
END IF;
END $$
DELIMITER ;
CALL V07_13_00_030__drop_processing_account;
DROP PROCEDURE IF EXISTS V07_13_00_030__drop_processing_account;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
Script V07_13_00_035__account_lock_state.sql
Path: stroom-security/stroom-security-identity-db/src/main/resources/stroom/security/identity/db/migration/V07_13_00_035__account_lock_state.sql
-- ------------------------------------------------------------------------
-- Copyright 2026 Crown Copyright
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- ------------------------------------------------------------------------
-- Reduces the account lockout to the one thing it is for: blunting repeated wrong passwords.
--
-- locked )
-- locked_until_ms ) -> failure_locked_ms
-- login_failures -> failure_count
--
-- Three changes, in one migration because they are one idea.
--
-- Naming. Locking is not an administrative act. An administrator prevents an account being used by
-- disabling it, and only ever unlocks - and then only when the lock will not clear itself and self service
-- unlock is unavailable. Naming these columns as a group makes it harder to read them as general state.
--
-- Meaning. The old column held when the lock was due to end, worked out when the lock was applied, so it
-- was the configured duration already spent. Changing that duration then had no effect on any lock already
-- in force. Holding the moment the lock was applied and adding the configured duration at the point of
-- asking makes the setting mean what an administrator expects, for existing locks as well as new ones. A
-- duration of zero now means the lock does not lapse - including for locks already held, so setting it
-- during an incident makes every one of them permanent until an administrator releases it.
--
-- Shape. Once the column holds when the lock was applied, the boolean says nothing the timestamp does not:
-- a lock exists exactly when there is a time it was applied. Two columns that must agree are two columns
-- that can disagree, and a writer that clears one without the other leaves a lock that no longer means what
-- it says. One column cannot disagree with itself.
-- Stop NOTE level warnings about objects (not)? existing
SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0;
DROP PROCEDURE IF EXISTS V07_13_00_035__account_lock_state;
DELIMITER $$
CREATE PROCEDURE V07_13_00_035__account_lock_state ()
BEGIN
DECLARE object_count integer;
SELECT COUNT(1)
INTO object_count
FROM information_schema.columns
WHERE table_schema = database()
AND table_name = 'account'
AND column_name = 'locked';
IF object_count = 1 THEN
-- Carry over locks an administrator applied by hand, while `locked` still means "locked for any
-- reason". A lock with no end time is the best available reading of that: a failure lock normally
-- carries one. Disabling is the control that expresses an administrator barring an account now, and
-- unlike a permanent lock it is visible for what it is.
--
-- This cannot tell a hand applied lock from a failure lock in a deployment configured with a zero
-- lock duration, where failure locks are also stored with no end time. Those accounts become
-- disabled. That fails closed - access stays barred until an administrator acts - but it is visible,
-- and such deployments should expect to re-enable deliberately after upgrading.
UPDATE account
SET enabled = 0,
locked = 0
WHERE locked = 1
AND locked_until_ms IS NULL;
ALTER TABLE account
CHANGE COLUMN locked_until_ms failure_locked_ms bigint DEFAULT NULL,
CHANGE COLUMN login_failures failure_count int NOT NULL DEFAULT '0';
-- When each surviving lock was applied cannot be recovered: the old column held only when it was
-- due to end, and the duration that produced it is configuration rather than data. Treat them as
-- locked now, so each serves at most one further full duration. That errs towards keeping an
-- account locked rather than releasing one early, and it is bounded.
UPDATE account
SET failure_locked_ms = UNIX_TIMESTAMP() * 1000
WHERE locked = 1;
-- No lock means no lock time. Any end time left on an unlocked account is stale and must go, or the
-- single column would report a lock that is not there.
UPDATE account
SET failure_locked_ms = NULL
WHERE locked = 0;
ALTER TABLE account
DROP COLUMN locked;
END IF;
END $$
DELIMITER ;
CALL V07_13_00_035__account_lock_state;
DROP PROCEDURE IF EXISTS V07_13_00_035__account_lock_state;
SET SQL_NOTES=@OLD_SQL_NOTES;
-- vim: set shiftwidth=4 tabstop=4 expandtab:
5 - Change Log
New Features and Changes
-
Feature #5582 : Add proper audit trail to doc history and store snapshots of data changes to allow future restore.
-
Feature #2109 : Add doc dependencies to DB to improve capability.
-
Feature #1556 : Add safe delete feature now we can depend on a reliable dependency discovery service.
-
Feature #4111 : Add confirmation details when deleting a folder.
-
Feature #5697 : Apply CSRF Origin/X-CSRF checks to state-changing requests whose credential was injected by an authenticating edge proxy (previously only session-cookie identities were checked), and reject cross-site browser requests carrying a request token unless they send
X-CSRF: 1. In-browser clients that attach their own bearer token must now sendX-CSRF: 1on state-changing requests whenedgeAuthentication.enabledis set; non-browser automation and inter-node traffic are unaffected. -
Feature #5697 : Add
security.authentication.openId.authenticationRequestExtraParamsto append provider-specific parameters to the OIDC authentication request, e.g. Google’saccess_type: offlinewithout which Google issues no refresh token and the session cannot outlive the first access token. -
Feature #5656 : Add feature to view sessions and revoke them and associated user tokens.
-
Feature #5675 : Add HTTP and TLS configuration to Git repositories.
-
Feature #5656 : Add self service account unlocking for the internal identity provider, controlled by the new properties
stroom.security.identity.reactivateInactiveAccountsOnLoginandstroom.security.identity.allowLockedAccountPasswordReset, and rebuild the ‘Forgot password’ reset page so that an emailed reset link can be completed. -
Feature #5652 : Support Elasticsearch nested field types in search.
-
Feature #5654 : Support multiple dense_vector fields in Elasticsearch rerank search.
-
Feature #5599 : Add XPath to query functions so that users can pull XML apart in Dashboard Tables.
-
Feature #5600 : Add JQ to query functions so that users can pull JSON apart in Dashboard Tables.
-
Feature #5559 : Improve node selection for node groups to allow select all and selection inversion.
-
Feature #5561 : Add feature to delete individual attachments and messages from AI chat history.
-
Feature #5561 : Add time tooltips to AI chat messages.
-
Feature #5561 : Fix user preferences resetting stroom AI preferences.
-
Feature #5561 : Open and view attachments in the AI chat window.
-
Feature #5561 : Add names to tables so they can be identified by stroom AI.
-
Feature #5565 : Make vector embedding dimension count configurable.
-
Feature #5616 : Add XSLT function for computing the similarity of two float vectors.
-
Feature #5622 : Change Stroom UI auth flow so redirects are no longer required. Allows Stroom UI to be served from another location with BFF proxy.
-
Feature #5630 : Make embedding dimensions optional.
-
Feature #5567 : Stop the content index rebuilding on first use after a node reboot. Add config props
stroom.contentIndex.contentIndexDir(defaults tocontent_index),stroom.contentIndex.storageType(one ofTEMP|LOCAL|SHARED, defaults toLOCAL) andstroom.contentIndex.minRebuildAge(defaults toPT1M). Thus the content index can now be stored locally on each node for better performance or on shared storage. Stroom now eagerly builds the content index on boot if the storage type isSHARED. -
Feature #5515 : Change JSONParser pipeline element to truncate very long strings values. Currently very long string values can result in Out of Memory errors in Stroom. The following configuration properties have been added to the JSONParser element;
stringTruncateLength(default 10,000) to truncate very long strings,maxStringLength(default 100,000,000) to cause a fatal error if a long string is encountered,maxDepth(default 500) to limit the depth of deeply nested documents. The JSONParser has also been changed so that the characters of string values are streamed to the downstream pipeline elements rather than reading the whole string into memory. NOTE: It is still possible for downstream XSLT XPATH functions to result in the entire string being read into memory. -
Feature #5303 : Improve Stroom AI to add dockable panel, chat history, attachments, copy, download, chat details etc.
-
Feature #5282 : Add pipeline scheduling.
-
Feature #5346 : Choose which external document changes to save when saving a pipeline.
-
Feature #5377 : Embedded pipeline docs.
-
Feature #5387 : Allow pipeline stepping across multiple streams.
-
Feature #3103 : Allow multiple dashboard instances.
-
Feature : Editing items in the UI now indicates save required only when changes are made.
-
Feature #3206 : User tab sessions.
-
Feature #5366 : Rebase on master.
-
Feature #5366 : Cleanup.
-
Feature #5366 : Checkstyle.
-
Feature #5232 : Add standard annotation comments. MIGRATION: Any comments previously configured in standardComments property will need to saved as annotation comments.
-
Feature #5366 : Change icon for DataGen.
-
Feature #5366 : Add feed name to rule detection.
-
Feature : Improve the ProgressMonitor task creation logging output to included counts of errored and skipped filters. Change task creation to re-test filter enabled/deleted state just prior to creating tasks. Add validation to ProcessorConfig to enforce a minimum value of
1on some properties. -
Feature #5366 : Fix DataGen destination feed not firing dirty event.
-
Feature #5366 : Checkstyle.
Bug Fixes
-
Bug #5553 : Fix DocRefInfo cache bug.
-
Bug #4073 : Stop explorer scrolling to the top on deleting an item.
-
Bug #5697 : Fix the UI bootstrap never recognising a user authenticated by an edge proxy (e.g. AWS ALB + Cognito, NGINX + oauth2-proxy):
/api/auth/flow/v1/statusnow accepts a verified request token, and the newsecurity.authentication.edgeAuthenticationconfig block suppresses stroom’s own OIDC flow and supports edge-aware logout when the proxy is the relying party. -
Bug #5669 : Fix
HttpClientConfigConverternot mappingverifyHostname, which prevented TLS hostname verification being disabled on HTTP clients. -
Bug #5671 : Run directory-scanner file ingest as the processing user so that receipt checks requiring a user succeed.
-
Bug #5674 : Fix dirty behaviour on pipeline structure changes.
-
Bug #5680 : Fix account migration script.
-
Bug #5679 : Fix slow processor task assignment on large clusters. Task queueing now takes account of processing profiles so that tasks no node is allowed to process are not queued, and are released if a profile stops allowing them. Task assignment no longer repeatedly fills the queue when there is nothing to add, and only one request fills the queue at a time while the others wait for it.
-
Bug #5679 : Fix processor task creation not recording errors against the filter tracker, and not stopping when a task creation limit has been reached.
-
Bug #5685 : Fix inability to unset Max Processing Tasks on a processor filter.
-
Bug #5678 : Fix processor task retention only using the
stroom.processor.deleteAgevalue that was current when the node started. The Processor Task Retention job now reads the property on each run, so a change to it takes effect without a node restart. -
Bug #5663 : Fix OpenAPI spec for polymorphic types.
-
Bug #5647 : Fix user entered name being ignored when creating a new volume group.
-
Bug #5646 : Fix onChange() behaviour for document edits.
-
Bug #5651 : Fix file uploads bug introduced by CSRF change.
-
Bug #5560 : Fix processing schedule list label.
-
Bug #5548 : Fix PlanB filter XML value bug.
-
Bug #5562 : Add missing tab types to session restore.
-
Bug #5617 : Fix tab visibility on resize.
-
Bug #5621 : Support numeric comparators for Elasticsearch float and double fields.
-
Bug #5573 : Fix Ask Stroom AI error handling behaviour when requests are too large.
-
Bug #5574 : Fix Ask Stroom AI dock behaviour.
-
Bug #5575 : Change ask Stroom AI table page menu item.
-
Bug #5576 : Increase default AI model HTTP timeouts to 10 minutes.
-
Bug #5585 : Fix dashboard tab rename bug.
-
Bug #5577 : Fix bug affecting AI chat model selection.
-
Bug #5601 : Fix bug stopping embedded queries being edited.
-
Bug #5568 : Add analytic rule info to error stream messages.
-
Bug #5636 : Fix expression term quote removal bug.
-
Bug #5640 : Fix CSRF checks.
-
Bug : Change the behaviour of JSON deserialisation to not error when a null value is encountered for a primitive type. This is how it used to behave in 7.12. However it now logs an error if a null primitive is encountered, so the corresponding Java class can be fixed to properly support null values.
-
Bug #5579 : Change test collation to utf8mb4_0900_ai_ci.
-
Bug #5558 : Fix processor profiles allowing processing for disabled node groups.
-
Bug : Fix DocRef hover copy/open links not appearing.
-
Bug #5535 : Fix simple string values not appearing in Pipeline Property table.
-
Bug : Fix Null Pointer type bug on Data Receipt Rules screen.
-
Bug : Fix output of the manage_users –listPermissions command.
-
Bug : Fix missing arg validation on reset_password CLI command. Obfuscate password in logging.
Dependency Changes
-
Dependency #5624 : Upgrade langchain4j and openai-java libs.
-
Dependency : Uplift AWS SDK to 2.46.7 and hbase-shaded-netty to 4.1.13.
-
Dependency : Uplift Dropwizard to 5.0.2.
-
Dependency : Uplift DropWizard to v5.0.1.
-
Dependency : Uplift
net.openhft:zero-allocation-hashingto2026.0. -
Dependency : Add Jackson JSON library
3.1.2in addition to the existing2.21.2version. Stroom/Stroom-Proxy are now using v3 with the exception of a few specific areas that need legacy capability only available in v2. v3 is a significant change from v2 with some breaking changes and some differences in behaviour. Special attention should be paid to the output of JSONParser pipeline element to ensure it is behaving as expected. -
Dependency : Uplift base docker images to
eclipse-temurin:25.0.3_9-jdk-alpine-3.23. -
Dependency : Uplift org.apache.commons:commons-pool2 from 2.12.1 to 2.13.1.
-
Dependency : Uplift org.flywaydb:flyway-core from 11.20.0 to 12.0.0.
-
Dependency : Uplift org.eclipse.jgit:org.eclipse.jgit from 7.3.0.202506031305-r to 7.5.0.202512021534-r.
-
Dependency : Uplift org.apache.solr:solr-solrj from 9.8.0 to 9.10.1.
-
Dependency : Uplift swagger from 2.2.41 to 2.2.42.
-
Dependency : Uplift gwt from 2.12.2 to 2.13.0.
-
Dependency : Uplift co.elastic.clients:elasticsearch-java from 9.2.1 to 9.3.2.
-
Dependency : Uplift org.yaml:snakeyaml from 2.2 to 2.6.
Code Refactor
-
Refactor #5557 : Change config class constructors to correctly handle and default null primitive values on deserialisation from YAML.
-
Refactor : Change json (de)serialisation to not go via a String when dealing only with byte[] data.
-
Refactor : Refactor the feedKey locking in PreAggregator and make AttributeMapUtil#readKeys() more lenient when reading
.metafiles. -
Refactor : Remove HBase statistics (may require change to default config).
-
Refactor : Remove ScyllaDB based state store.
-
Refactor : Replace NullSafe.requireNonNullElse() with Objects.requireNonNullElse().
-
Refactor : Replace NullSafe.requireNonNullElseGet() with Objects.requireNonNullElseGet().
Build Changes
-
Build : Fix CI build failure due to missing Docker container prefix.
-
Build : Uplift gradle-wrapper from 9.3.0 to 9.3.1.
Uncategorised Issues
-
Issue #5366 : Add level and status to rules.
-
Issue #5366 : Create new doc object DataGen.
-
Issue #5366 : Add Execution tab to DataGen.
-
Issue #5366 : Implement job scheduling for DataGen.
-
Issue #5366 : Refactor schedulers.
-
Issue #5366 : Refactor schedulers to interface and cleanup.
-
Issue #5366 : Fix typo.