Reference Section

• 1: Cron Syntax
• 2: Dates & Times
• 3: Documents
• 4: Editor Completion Snippets
• 5: Expression functions
• 5.1: Aggregate Functions
• 5.2: Cast Functions
• 5.3: Date Functions
• 5.4: Link Functions
• 5.5: Logic Funtions
• 5.6: Mathematics Functions
• 5.7: Selection Functions
• 5.8: String Functions
• 5.9: Type Checking Functions
• 5.10: URI Functions
• 5.11: Value Functions
• 6: Background Jobs
• 7: Keyboard Shortcuts
• 8: Pipeline Elements
• 9: Templating

1 - Cron Syntax

cron is a syntax for expressing schedules.

Stroom uses a scheduler called Quartz which supports cron expressions for scheduling. The full details of the cron syntax supported by Quartz can be found here .

Cron expressions are used in:

• Stroom Jobs

Field Specification

Special Characters

• * (all values) - used to select all values within a field. For example, * in the minute field means every minute.

• ? (no specific value) - useful when you need to specify something in one of the two fields in which the character is allowed, but not the other. For example, if I want my trigger to fire on a particular day of the month (say, the 10th), but don’t care what day of the week that happens to be, I would put 10 in the day-of-month field, and ? in the day-of-week field. See the examples below for clarification.

• - - used to specify ranges. For example, 10-12 in the hour field means the hours 10, 11 and 12.

• , - used to specify additional values. For example, MON,WED,FRI in the day-of-week field means the days Monday, Wednesday, and Friday.

• / - used to specify increments. For example, 0/15 in the seconds field means the seconds 0, 15, 30, and 45. And 5/15 in the seconds field means the seconds 5, 20, 35, and 50. You can also specify ‘/’ after the ‘’ character - in this case ‘’ is equivalent to having ‘0’ before the ‘/’. ‘1/3’ in the day-of-month field means fire every 3 days starting on the first day of the month.

• L (last) - has different meaning in each of the two fields in which it is allowed. For example, the value L in the day-of-month field means the last day of the month - day 31 for January, day 28 for February on non-leap years. If used in the day-of-week field by itself, it simply means 7 or SAT. But if used in the day-of-week field after another value, it means the last xxx day of the month - for example 6L means the last friday of the month. You can also specify an offset from the last day of the month, such as L-3 which would mean the third-to-last day of the calendar month. When using the ‘L’ option, it is important not to specify lists, or ranges of values, as you’ll get confusing/unexpected results.

• W (weekday) - used to specify the weekday (Monday-Friday) nearest the given day. As an example, if you were to specify 15W as the value for the day-of-month field, the meaning is: the nearest weekday to the 15th of the month. So if the 15th is a Saturday, the trigger will fire on Friday the 14th. If the 15th is a Sunday, the trigger will fire on Monday the 16th. If the 15th is a Tuesday, then it will fire on Tuesday the 15th. However if you specify 1W as the value for day-of-month, and the 1st is a Saturday, the trigger will fire on Monday the 3rd, as it will not ‘jump’ over the boundary of a month’s days. The ‘W’ character can only be specified when the day-of-month is a single day, not a range or list of days.

  Note

  The ‘L’ and ‘W’ characters can also be combined in the day-of-month field to yield ‘LW’, which translates to “last weekday of the month”.

• # - used to specify the nth XXX day of the month. For example, the value of 6#3 in the day-of-week field means the third Friday of the month (day 6 = Friday and #3 = the 3rd one in the month). Other examples: 2#1 = the first Monday of the month and 4#5 = the fifth Wednesday of the month. Note that if you specify #5 and there is not 5 of the given day-of-week in the month, then no firing will occur that month.

  Note

  The legal characters and the names of months and days of the week are not case sensitive. MON is the same as mon.

Examples

2 - Dates & Times

Standard Format

Stroom’s standard format for displaying dates and times is ISO 8601 and specifically in the format

yyyy-MM-dd'T'HH:mm:ss.SSSXX

(where 'T' is the constant T and XX is the timezone offset or Z for Zulu/UTC).

The time part is always represented with three digits for the millisecond part.

Parsing

Parsing is the action of reading a string like 2010-01-01T23:59:59.123Z and converting it into a date/time value.

There are two types of parsing, standard parsing and parsing with an explicit format.

Standard Parsing

The standard format is used when parsing dates with no specific date format, for example in the EffectiveTime header that is used when sending reference data to Stroom.

There is a degree of leniency when parsing date time strings with no explicit format. The following table shows the acceptable date time strings and how they are represented in the standard form.

Custom Date Formats

Parsing and formatting with an explicit date time format is done in a few places in Stroom.

• The XSLT function format-date(). This function is a bit of a misnomer as it is doing both parsing and formatting.

• The Dashboard/Query expression parseDate().

Stroom uses Java’s DateTimeFormatter syntax for expressing an explicit date format.

All letters A to Z and a to z are reserved as pattern letters. The following pattern letters are defined:

The count of pattern letters determines the format.

Presentation Types

• Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form Exactly 4 pattern letters will use the full form Exactly 5 pattern letters will use the narrow form Pattern letters L, c, and q specify the stand-alone form of the text styles.

• Number: If the count of letters is one, then the value is output using the minimum number of digits and without padding Otherwise, the count of digits is used as the width of the output field, with the value zero-padded as necessary The following pattern letters have constraints on the count of letters Only one letter of c and F can be specified Up to two letters of d, H, h, K, k, m, and s can be specified Up to three letters of D can be specified.

• Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above Otherwise use the Number rules above.

• Fraction: Outputs the nano-of-second field as a fraction-of-second The nano-of-second value has nine digits, thus the count of pattern letters is from 1 to 9 If it is less than 9, then the nano-of-second value is truncated, with only the most significant digits being output.

• Year: The count of letters determines the minimum field width below which padding is used If the count of letters is two, then a reduced two digit form is used For printing, this outputs the rightmost two digits For parsing, this will parse using the base value of 2000, resulting in a year within the range 2000 to 2099 inclusive If the count of letters is less than four (but not two), then the sign is only output for negative years as per SignStyle.NORMAL Otherwise, the sign is output if the pad width is exceeded, as per SignStyle.EXCEEDS_PAD.

• ZoneId: This outputs the time-zone ID, such as Europe/Paris If the count of letters is two, then the time-zone ID is output Any other count of letters throws IllegalArgumentException.

• Zone names: This outputs the display name of the time-zone ID If the pattern letter is z the output is the daylight saving aware zone name If there is insufficient information to determine whether DST applies, the name ignoring daylight saving time will be used If the count of letters is one, two or three, then the short name is output If the count of letters is four, then the full name is output Five or more letters throws IllegalArgumentException.

  If the pattern letter is v the output provides the zone name ignoring daylight saving time If the count of letters is one, then the short name is output If the count of letters is four, then the full name is output Two, three and five or more letters throw IllegalArgumentException.

• Offset X and x: This formats the offset based on the number of pattern letters One letter outputs just the hour, such as +01, unless the minute is non-zero in which case the minute is also output, such as +0130 Two letters outputs the hour and minute, without a colon, such as +0130 Three letters outputs the hour and minute, with a colon, such as +01:30 Four letters outputs the hour and minute and optional second, without a colon, such as +013015 Five letters outputs the hour and minute and optional second, with a colon, such as +01:30:15 Six or more letters throws IllegalArgumentException Pattern letter X (upper case) will output Z when the offset to be output would be zero, whereas pattern letter x (lower case) will output +00, +0000, or +00:00.

• Offset O: With a non-zero offset, this formats the localized offset based on the number of pattern letters One letter outputs the short form of the localized offset, which is localized offset text, such as GMT, with hour without leading zero, optional 2-digit minute and second if non-zero, and colon, for example GMT+8 Four letters outputs the full form, which is localized offset text, such as GMT, with 2-digit hour and minute field, optional second field if non-zero, and colon, for example GMT+08:00 If the offset is zero, only localized text is output Any other count of letters throws IllegalArgumentException.

• Offset Z: This formats the offset based on the number of pattern letters One, two or three letters outputs the hour and minute, without a colon, such as +0130 The output will be +0000 when the offset is zero Four letters outputs the full form of localized offset, equivalent to four letters of Offset-O The output will be the corresponding localized offset text if the offset is zero Five letters outputs the hour, minute, with optional second if non-zero, with colon It outputs Z if the offset is zero Six or more letters throws IllegalArgumentException.

• Optional section: The optional section markers work exactly like calling DateTimeFormatterBuilder.optionalStart() and DateTimeFormatterBuilder.optionalEnd().

• Pad modifier: Modifies the pattern that immediately follows to be padded with spaces The pad width is determined by the number of pattern letters This is the same as calling DateTimeFormatterBuilder.padNext(int).

For example, ppH outputs the hour-of-day padded on the left with spaces to a width of 2.

Any unrecognized letter is an error Any non-letter character, other than [, ], {, }, # and the single quote will be output directly Despite this, it is recommended to use single quotes around all characters that you want to output directly to ensure that future changes do not break your application.

For further details, see the DateTimeFormatter documentation.

For examples of the parsing/formatting patterns in use, see format-date().

Formatting

Stroom can format dates with an explicit format in a few places:

• The XSLT function format-date(). This function is a bit of a misnomer as it is both parsing and formatting.

• The Dashboard/Query expression formatDate().

• The User Preferences dialog.

When formatting a date time, the format syntax is the same as used in parsing, as shown above.

Durations

Durations are represented in Stroom in two different forms, Stroom Durations and ISO 8601 Durations.

Stroom Duration

Stroom’s standard duration syntax takes the form of a numeric value followed by an optional unit suffix, e.g. 10m for ten minutes.

Stroom Duration strings are used in a number of places in Stroom:

• Frequency Schedules.
• Date Expressions.
• Configuration properties.
• Dashboard/Query expression functions parseDuration and formatDuration.

ISO 8601 Duration

ISO 8601 durations are an international standard format for expressing durations.

ISO 8601 duration strings are used in a number of places in Stroom:

• Configuration properties.
• Dashboard/Query expression functions parseISODuration and formatISODuration.

Date Expressions

Date expressions are a way to represent relative dates or to express simple date arithmetic. They can be used in the following places in Strom:

• Dashboard expression term values.
• Dashboard/Query time range settings.
• Dashboard/Query expression language.

Date expressions consist of a mixture of:

• StroomDuration.
• Relative date functions like minute().
• Absolute date/times in the Standard Format

The available relative date functions are:

In the examples above, the current time is taken to be 2024-04-26T17:41:55.239Z which is a Friday.

The following are some examples of date expressions:

In the examples above, the current time is taken to be 2024-04-26T17:41:55.239Z which is a Friday.

3 - Documents

All Documents in Stroom share some common elements:

• UUID - Uniquely identifies the document within Stroom and when exported into another stroom.
• Type - This is the type as used in the DocRef .
• Documentation - Every Document has a Documentation tab for recording any documentation that relates to the Document, see Documenting Content.

Some Documents are very simple with just text content and documentation, e.g. XSLT. Others are much more complex, e.g. Pipeline, with various different tabs to manage the content of the Document.

The following is a list of all Document types in Stroom.

Configuration

Documents that are used as configuration for other documents.

Dictionary

• Icon:
• Type: Dictionary

A Dictionary is essentially a list of ‘words’, where each ‘word’ is separated by a new line. Dictionaries can be used in filter expressions, i.e. IN DICTIONARY. They allow for the reuse of the same set of values across many search expressions. Dictionaries also support inheritance so one dictionary can import the contents of other dictionaries.

Documentation

• Icon:
• Type: Documentation

A Document type for simply storing user created documentation, e.g. adding a Documentation document into a folder to describe the contents of that folder.

Elastic Cluster

• Icon:
• Type: ElasticCluster

Defines the connection details for a single Elasticsearch cluster. This Elastic Cluster Document can then be used by one or more Elastic Index Documents.

Kafka Configuration

• Icon:
• Type: KafkaConfig

Defines the connection details for a single Kafka cluster. This Kafka Configuration Document can then be used by one or more StandardKafkaProducer pipeline elements.

S3 Configuration

• Icon:
• Type: S3Config

Defines the config for S3

Script

• Icon:
• Type: Script

Contains a Javascript script that is used as the source for a visualisation Document. Scripts can have dependencies on other Script Documents, e.g. to allow re-use of common code.

ScyllaDB

• Icon:
• Type: ScyllaDB

Defines the connection details for a ScyllaDB state store instance.

Visualisation

• Icon:
• Type: Visualisation

Defines a data visualisation that can be used in a Dashboard Document. The Visualisation defines the settings that will be available to the user when it is embedded in a Dashboard. A Visualisation is dependent on a Script Document for the Javascript code to make it work.

Data Processing

Documents relating to the processing of data.

Feed

• Icon:
• Type: Feed

The Feed is Stroom’s way of compartmentalising data that has been ingested or created by a Pipeline. Ingested data must specify the Feed that is it destined for.

The Feed Document defines the character encoding for the data in the Feed, the type of data that will be received into it (e.g. Raw Events) and optionally a Volume Group to use for data storage. The Feed Document can also control the ingest of data using its Feed Status property and be used for viewing data that belonging to that feed.

Pipeline

• Icon:
• Type: Pipeline

A Pipeline defines a chain of Pipeline elements that consumes from a source of data (a Stream of raw data or cooked events) then processes it according to the elements used in the chain. Pipelines can be linear or branching and support inheritance of other pipelines to allow re-use of common structural parts.

The Pipeline Document defines the structure of the pipeline and the configuration of each of the elements in that pipeline. It also defines the filter(s) that will be used to control what data is passed through the pipeline and the priority of processing. The Pipeline Document can be used to view the data produced by the pipeline and to monitor its processing state and progress.

Indexing

Documents relating to the process of adding data into an index, such as Lucene or Elasticsearch.

Elastic Index

• Icon:
• Type: ElasticIndex

Defines an index that exists within an Elasticsearch cluster. This Document is used in the configuration of the ElasticIndexingFilter pipeline element.

Lucene Index

• Icon:
• Type: Index

Lucene Index is the standard built-in index within Stroom and is one of may data sources. An index is like a catalog in a library and provides a very fast way to access documents/records/events when searching using fields that have been indexed. The index stores the field values and pointers to the document they came from (the Stream and Event IDs). Data can be indexed using multiple indexes to allow fast access in different ways.

The Lucene Index Document optionally defines the fields that will be indexed (it is possible to define the fields dynamically) and their types. It also allows for configuration of the way the data in the index will be stored, partitioned and retained.

The Lucene Index Document is used by the IndexingFilter and DynamicIndexingFilter pipeline elements.

Solr Index

• Icon:
• Type: SolrIndex

Solr Index represents an index on a Solr cluster. It defines the connection details for connecting to that cluster and the structure of the index. It is used by the SolrIndexingFilter pipeline element.

State Store

• Icon:
• Type: StateStore

Defines a place to store state

Statistic Store

• Icon:
• Type: StatisticStore

Defines a logical statistic store used to hold statistical data of a particular type and aggregation window. Statistics in Stroom is a way to capture counts or values from events and record how they change over time, with the counts/values aggregated (sum/mean) across time windows.

The Statistic Store Document configures the type of the statistic (Count or Value), the tags that are used to qualify a statistic event and the size of the aggregation windows. It also supports the definition of roll-ups that allow for aggregation over all values of a tag. Tags can be things like user, node, feed, etc. and can be used to filter data when querying the statistic store in a Dashboard/Query.

It is used by the StatisticsFilter pipeline element.

Stroom-Stats Store

• Icon:
• Type: StroomStatsStore

The Stroom-Stats Store Document is deprecated and should not be used.

Search

Documents relating to searching for data in Stroom.

Analytic Rule

• Icon:
• Type: AnalyticRule

Defines an analytic rule which can be run to alert on events meeting a criteria. The criteria is defined using a StroomQL query. The analytic can be processed in different ways:

• Streaming
• Table Builder
• Scheduled Query

Dashboard

• Icon:
• Type: Dashboard

The Dashboard Document defines a data querying and visualisation dashboard. The dashboard is highly customisable to allow querying of many different data sources of different types. Queried data can be displayed in tabular form, visualised using interactive charts/graphs or render as HTML.

The Dashboard Doc can either be used for ad-hoc querying/visualising of data, to construct a dashboard for others to view or to just view an already constructed dashboard. Dashboards can be parameterised so that all queries on the dashboard are displaying data for the same user, for example. For ad-hoc querying of data from one data source, you are recommended to use a Query instead.

Query

• Icon:
• Type: Query

A Query Document defines a StroomQL query and is used to execute that query and view its results. A Query can query main types of data source including Views, Lucene Indexes, and Searchables .

View

• Icon:
• Type: View

A view is an abstraction over a data source (such as a Lucene Indexe) and optionally an extraction pipeline. Views provide a much simpler way for users to query data as the user can simply query against the View without any knowledge of the underlying data source or extraction of that data.

Transformation

Documents relating to the transformation of data.

Text Converter

• Icon:
• Type: TextConverter

A Text Converter Document defines the specification for splitting text data into records/fields using Data Splitter or for wrapping fragment XML with a XMLFragmentParser pipeline element. The content of the Document is either XML in the data-splitter:3 namespace or a fragment parser specification (see Pipeline Recipies).

This Document is used by the following pipeline elements:

• DSParser
• XMLFragmentParser
• CombinedParser

XML Schema

• Icon:
• Type: XMLSchema

This Document defines an XML Schema that can be used within Stroom for validation of XML documents. The XML Schema Document content is the XMLSchema text. This Document also defines the following:

• Namespace URI - The XML namespace of the XMLSchema and the XML document that the schema will validate.
• System Id - An ID (that is unique in Stroom) that can be used in the xsi:schemaLocation attribute, e.g. xsi:schemaLocation="event-logging:3 file://event-logging-v3.4.2.xsd".
• Schema Group - A name to group multiple versions of the same schema. The SchemaFilter can be configured to only use schemas matching a configured group.

The XML Schema Document also provides a handy interactive viewer for viewing and navigating the XMLSchema in a graphical representation.

This Document is used by the SchemaFilter pipeline element.

XSL Translation

• Icon:
• Type: XSLT

The content of this Document is an XSLT document for transforming data in a pipeline. This Document is used by the XSLTFilter pipeline element.

4 - Editor Completion Snippets

Overview

Completion snippets are a way to quickly insert snippets of text into the editor. This is very useful for example when editing XSLT documents as you can quickly insert common chunks of XSLT.

This page lists all the snippets available in Stroom. Snippets are specific to the type of content being edited, e.g. When editing an XSLT, you will only be able to use XML/XSLT snippets.

Tab Positions

A snippet is not just a static block of text, it can contain various tab position placeholders. The following is an example of a snippet with multiple tab positions:

<xsl:apply-templates select="${1:*}">
  <xsl:with-param name="${2:param}">${3}</xsl:with-param>
  ${0}
</xsl:apply-templates>

Tab positions are expressed like ${n} or ${n:xxx}, where n is a number indication the order of the tab position and xxx is the default value at that tab position. Sometimes xxx will not be a default value, but will instead be a string to hint at the kind of thing the user is expected to overtype with where no obvious default is available. ${0} is a special tab position in that it defines the last tab position.

To insert a snippet do one of the following:

• Type the whole Tab Trigger then hit Tab ↹ .
• Type some characters from the Name then hit Ctrl ^ + Space ␣ to select it from the list.

The snippet will be inserted (replacing the Tab Trigger if used) and the cursor will be position on the first tab position (${1} if present, otherwise ${0}). If the tab position has a default value then the whole of that default value will be selected allowing the user to quickly over type it. Once the user is happy with the first tab position (whether they inserted text, over typed or did nothing), they can hit tab to move to the next tab position.

Advanced Tab Positions

It is possible for a tab position to be used multiple times in a snippet, like in the following example. All subsequent uses of a tab stop will take the value entered by the user on the first use of it. The subsequent uses will not act as tab stops.

<xsl:stylesheet xmlns="${1}" xpath-default-namespace="${1}">
  ${0}
</xsl:stylesheet>

If you want a reused tab position to also be a tab stop so that the user can chose to override the re-used value, then you can nest the tab stops like in this example:

<xsl:stylesheet xmlns="${1}" xpath-default-namespace="${2:${1}}">
  ${0}
</xsl:stylesheet>

Adding Snippets to Stroom

We have plans for improving how completion snippets are defined, i.e. allowing users to define their own. Therefore, available snippets may be subject to change.

However, until then, if there are any generic snippets that you think would be useful to add to Stroom then please raise an issue on GitHub .

XML/XSLT Snippets

Apply-templates with-param (wapply)

Name: Apply-templates with-param, Tab Trigger: wapply

<xsl:apply-templates select="${1:*}">
  <xsl:with-param name="${2:param}">${3}</xsl:with-param>
  ${0}
</xsl:apply-templates>

Apply-templates sort-by (applysort)

Name: Apply-templates sort-by, Tab Trigger: applysort

<xsl:apply-templates select="${1:*}">
  <xsl:sort select="${2:node}" order="${3:ascending}" data-type="${4:text}">${5}
</xsl:apply-templates>
${0}

Apply-templates plain (apply)

Name: Apply-templates plain, Tab Trigger: apply

<xsl:apply-templates select="${1:*}" />
${0}

Attribute blank (attr)

Name: Attribute blank, Tab Trigger: attr

<xsl:attribute name="${1:name}">${2}</xsl:attribute>
${0}

Attribute value-of (attrval)

Name: Attribute value-of, Tab Trigger: attrval

<xsl:attribute name="${1:name}">
  <xsl:value-of select="${2:*}" />
</xsl:attribute>
${0}

Call-template (call)

Name: Call-template, Tab Trigger: call

<xsl:call-template name="${1:template}" />
${0}

Call-template with-param (wcall)

Name: Call-template with-param, Tab Trigger: wcall

<xsl:call-template name="${1:template}">
  <xsl:with-param name="${2:param}">${3}</xsl:with-param>${4}
</xsl:call-template>
${0}

Choose (choose)

Name: Choose, Tab Trigger: choose

<xsl:choose>
  <xsl:when test="${1:value}">
    ${2}
  </xsl:when>${3}
</xsl:choose>
${0}

Copy-of (copyof)

Name: Copy-of, Tab Trigger: copyof

<xsl:copy-of select="${1:*}" />
${0}

Element blank (elem)

Name: Element blank, Tab Trigger: elem

<xsl:element name="${1:name}">
  ${2}
</xsl:element>
${0}

For-each (foreach)

Name: For-each, Tab Trigger: foreach

<xsl:for-each select="${1:*}">
  ${2}
</xsl:for-each>
${0}

If (if)

Name: If, Tab Trigger: if

<xsl:if test="${1:test}">
  ${2}
</xsl:if>
${0}

Import (imp)

Name: Import, Tab Trigger: imp

<xsl:import href="${1:stylesheet}" />
${0}

Include (inc)

Name: Include, Tab Trigger: inc

<xsl:include href="${1:stylesheet}" />
${0}

Otherwise (otherwise)

Name: Otherwise, Tab Trigger: otherwise

<xsl:otherwise>
  ${1}
</xsl:otherwise>
$0

Param (param)

Name: Param, Tab Trigger: param

<xsl:param name="${1:name}">
  ${2}
</xsl:param>
${0}

Stylesheet (style)

Name: Stylesheet, Tab Trigger: style

<xsl:stylesheet
    version="1.0"
    xmlns="${1}"
    xpath-default-namespace="${2:${1}}"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  ${0}
</xsl:stylesheet>

Template (temp)

Name: Template, Tab Trigger: temp

<xsl:template match="${1:*}">
  ${2}
</xsl:template>
$0

Template named (ntemp)

Name: Template named, Tab Trigger: ntemp

<xsl:template name="${1:name}">
  ${2}
</xsl:template>
$0

Text (text)

Name: Text, Tab Trigger: text

<xsl:text>${1}</xsl:text>
$0

Value-of (valof)

Name: Value-of, Tab Trigger: valof

<xsl:value-of select="${1:*}" />
${0}

Variable blank (var)

Name: Variable blank, Tab Trigger: var

<xsl:variable name="${1:name}">
  ${0}
</xsl:variable>

Variable select (varsel)

Name: Variable select, Tab Trigger: varsel

<xsl:variable select="${1:*}" />
${0}

When (when)

Name: When, Tab Trigger: when

<xsl:when test="${1:test}">
  ${0}
</xsl:when>

With-param (wparam)

Name: With-param, Tab Trigger: wparam

<xsl:with-param name="${1:name}">${2}</xsl:with-param>
${0}

With-param select (wparamsel)

Name: With-param select, Tab Trigger: wparamsel

<xsl:with-param name="${1:name}" select="${2:*}" />
${0}

Fatal message (fatal)

Name: Fatal message, Tab Trigger: fatal

<xsl:message terminate="yes">${1}</xsl:message>
${0}

Error message (error)

Name: Error message, Tab Trigger: error

<xsl:message><error>${1}</error></xsl:message>
${0}

Warning message (warn)

Name: Warning message, Tab Trigger: warn

<xsl:message><warn>${1}</warn></xsl:message>
${0}

Info message (info)

Name: Info message, Tab Trigger: info

<xsl:message><info>${1}</info></xsl:message>
${0}

Identity skeleton (ident)

Name: Identity skeleton, Tab Trigger: ident

<xsl:stylesheet version="1.0" xpath-default-namespace="${1:event-logging:3}" xmlns="${2:${1}}" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <!-- Whenever you match any node or any attribute -->
  <xsl:template match="node( )|@*">

    <!-- Copy the current node -->
    <xsl:copy>

      <!-- Including any attributes it has and any child nodes -->
      <xsl:apply-templates select="@*|node( )"/>
    </xsl:copy>
  </xsl:template>

  ${0}
</xsl:stylesheet>

Records identity skeleton (recident)

Name: Records identity skeleton, Tab Trigger: recident

<xsl:stylesheet version="1.0" xpath-default-namespace="records:2" xmlns="event-logging:3" xmlns:stroom="stroom" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <!-- Match Root Object -->
  <xsl:template match="records">
    <Events xmlns="event-logging:3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="event-logging:3 file://event-logging-v3.4.2.xsd" Version="3.4.2">
      <xsl:apply-templates />
    </Events>
  </xsl:template>
  <xsl:template match="record">
    <Event>
      <EventTime>
        <TimeCreated>${1:time}</TimeCreated>
      </EventTime>
      <EventSource>
        <System>
          <Name>${2:name}</Name>
          <Environment>${3:environment}</Environment>
        </System>
        <Generator>${4:generator}</Generator>
        <Device>${5:device}</Device>
      </EventSource>
      <EventDetail>
        <TypeId>${6:type}</TypeId>
        ${0}
        <xsl:apply-templates />
      </EventDetail>
    </Event>
  </xsl:template>

  <!-- Whenever you match any node or any attribute -->
  <xsl:template match="node( )|@*">

    <!-- Copy the current node -->
    <xsl:copy>

      <!-- Including any attributes it has and any child nodes -->
      <xsl:apply-templates select="@*|node( )" />
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Events identity skeleton (evtident)

Name: Events identity skeleton, Tab Trigger: evtident

<xsl:stylesheet version="1.0" xpath-default-namespace="event-logging:3" xmlns="event-logging:3" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <!-- Match Root Object -->
  <xsl:template match="Events">
    <Events xmlns="event-logging:3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="event-logging:3 file://event-logging-v3.4.2.xsd" Version="3.4.2">
      <xsl:apply-templates />
    </Events>
  </xsl:template>
  ${0}

  <!-- Whenever you match any node or any attribute -->
  <xsl:template match="node( )|@*">

    <!-- Copy the current node -->
    <xsl:copy>

      <!-- Including any attributes it has and any child nodes -->
      <xsl:apply-templates select="@*|node( )" />
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Data Splitter Snippets

CSV Splitter (csv)

Name: CSV Splitter, Tab Trigger: csv

<?xml version="1.0" encoding="UTF-8"?>
<dataSplitter
    xmlns="data-splitter:3"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="data-splitter:3 file://data-splitter-v3.0.xsd"
    version="3.0">

  <!-- Match each line using a new line character as the delimiter -->
  <split delimiter="\n">

    <!-- Take the matched line (using group 1 ignores the delimiters,
    without this each match would include the new line character) -->
    <group value="\$1">

    <!-- Match each value separated by a comma as the delimiter -->
    <split delimiter=",">

      <!-- Output the value from group 1 (as above using group 1
        ignores the delimiters, without this each value would include
      the comma) -->
      <data value="\$1"/>
      ${0}
    </split>
    </group>
  </split>
</dataSplitter>

CSV Splitter with heading (csvh)

Name: CSV Splitter with heading, Tab Trigger: csvh

<?xml version="1.0" encoding="UTF-8"?>
<dataSplitter
    xmlns="data-splitter:3"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="data-splitter:3 file://data-splitter-v3.0.xsd"
    version="3.0">

  <!-- Match heading line (note that maxMatch="1" means that only the
  first line will be matched by this splitter) -->
  <split delimiter="\n" maxMatch="1">

    <!-- Store each heading in a named list -->
    <group>
      <split delimiter=",">
        <var id="heading" />
      </split>
    </group>
  </split>

  <!-- Match each record -->
  <split delimiter="\n">

    <!-- Take the matched line -->
    <group value="\$1">

      <!-- Split the line up -->
      <split delimiter=",">

        <!-- Output the stored heading for each iteration and the value
        from group 1 -->
        <data name="\$heading\$1" value="\$1" />
        ${0}
      </split>
    </group>
  </split>
</dataSplitter>

Data Splitter Template (ds)

Name: Data Splitter Template, Tab Trigger: ds

<?xml version="1.0" encoding="UTF-8"?>
<dataSplitter
    xmlns="data-splitter:3"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="data-splitter:3 file://data-splitter-v3.0.xsd"
    version="3.0">
  ${0}
</dataSplitter>

Data element with name attribute (nval)

Name: Data element with name attribute, Tab Trigger: nval

<data name="${1}" value="${2}"/>
${0}

Data element without name attribute (val)

Name: Data element without name attribute, Tab Trigger: val

<data value="${1}"/>
${0}

Var element (var)

Name: Var element, Tab Trigger: var

<var id="${1}"/>
${0}

Split element (spl)

Name: Split element, Tab Trigger: spl

<split delimiter="${1:\n}">
  <group value="${2:\$1}">
    ${3}
  </group>
</split>
${0}

Group element (gr)

Name: Group element, Tab Trigger: gr

<group value="${1:\$1}">
  ${2}
</group>
${0}

All element (all)

Name: All element, Tab Trigger: all

<all>
  ${1}
</all>
${0}

Regex element (reg)

Name: Regex element, Tab Trigger: reg

<regex ${1:dotall="true" }${2:caseInsensitive="true" }pattern="${3}">
  <group>
    ${0}
  </group>
</regex>

XMLFragmentParser Snippets

Events fragment template (evt)

Name: Events fragment template, Tab Trigger: evt

<?xml version="1.1" encoding="utf-8"?>
<!DOCTYPE Events [
<!ENTITY fragment SYSTEM "fragment">]>
<Events
    xmlns="event-logging:${1:3}"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="event-logging:${1} file://event-logging-v${2:3.4.2}.xsd"
    version="${2}">
&fragment;
</records>
${0}

Records fragment template (rec)

Name: Records fragment template, Tab Trigger: rec

<?xml version="1.1" encoding="utf-8"?>
<!DOCTYPE Records [
<!ENTITY fragment SYSTEM "fragment">]>
<records
    xmlns="records:${1:2}"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="records:${1} file://records-v${2:2.0}.xsd"
    version="${2}">
&fragment;
</records>
${0}

Documentation (Markdown) Snippets

Heading level 1 (h1)

Name: Heading level 1, Tab Trigger: h1

# ${1:heading}

${0}

Heading level 2 (h2)

Name: Heading level 2, Tab Trigger: h2

## ${1:heading}

${0}

Heading level 3 (h3)

Name: Heading level 3, Tab Trigger: h3

### ${1:heading}

${0}

Heading level 4 (h4)

Name: Heading level 4, Tab Trigger: h4

#### ${1:heading}

${0}

Heading level 5 (h5)

Name: Heading level 5, Tab Trigger: h5

##### ${1:heading}

${0}

Heading level 6 (h6)

Name: Heading level 6, Tab Trigger: h6

###### ${1:heading}

${0}

Fenced Block (fence)

Name: Fenced Block, Tab Trigger: fence

```${1:language}
${2}
```
${0}

Fenced block of XML (fxml)

Name: Fenced block of XML, Tab Trigger: fxml

```xml
${1}
```
${0}

Fenced block of plain text (ftext)

Name: Fenced block of plain text, Tab Trigger: ftext

```text
${1}
```
${0}

Inline code (inline)

Name: Inline code, Tab Trigger: inline

`${1:code}`${0}

Bold text (b)

Name: Bold text, Tab Trigger: b

**${1:bold_text}**${0}

Italic text (i)

Name: Italic text, Tab Trigger: i

_${1:italic_text}_${0}

Strike-through text (s)

Name: Strike-through text, Tab Trigger: s

~~${1:strikethrough_text}~~${0}

Bold italic text (bi)

Name: Bold italic text, Tab Trigger: bi

***${1:bold_italic_text}***${0}

Stroom Query Language Snippets

All Expression Functions are available as snippets. They do not currently have tab triggers.

Eval first StreamId (str)

Name: Eval first StreamId, Tab Trigger: str

eval StreamId = first(StreamId)
$0

Eval first EventId (evt)

Name: Eval first EventId, Tab Trigger: evt

eval EventId = first(EventId)
$0

Eval first Stream/EventIds (ids)

Name: Eval first Stream/EventIds, Tab Trigger: ids

eval StreamId = first(StreamId)
eval EventId = first(EventId)
$0

Eval first first value (first)

Name: Eval first first value, Tab Trigger: first

eval ${1:field_name} = first(${1})
$0

Dashboard Table Expression Editor Snippets

All Expression Functions are available as snippets. They do not currently have tab triggers.

5 - Expression functions

Expressions can be used to manipulate data on Stroom Dashboards and Queries.

Each function has a name, and some have additional aliases.

In some cases, functions can be nested. The return value for some functions being used as the arguments for other functions.

The arguments to functions can either be other functions, literal values, or they can refer to fields on the input data using the field reference ${val} syntax.

5.1 - Aggregate Functions

Aggregate functions require that the dashboard columns without aggregate functions have a grouping level applied. The aggregate function will then be evaluated against the values in the group.

Average

Takes an average value of the arguments

average(arg)
mean(arg)

Examples

average(${val})
${val} = [10, 20, 30, 40]
> 25

mean(${val})
${val} = [10, 20, 30, 40]
> 25

Count

Counts the number of records that are passed through it. Doesn’t take any notice of the values of any fields.

count()

Example

Supplying 3 values...

count()
> 3

Count Groups

This is used to count the number of unique values where there are multiple group levels. For Example, a data set grouped as follows

1. Group by Name
2. Group by Type

A groupCount could be used to count the number of distinct values of ’type’ for each value of ’name'

Count Unique

This is used to count the number of unique values passed to the function where grouping is used to aggregate values in other columns. For Example, a data set grouped as follows

1. Group by Name
2. Group by Type

countUnique() could be used to count the number of distinct values of ’type’ for each value of ’name'

Example

countUnique(${val})
${val} = ['bill', 'bob', 'fred', 'bill']
> 3

Distinct

Concatenates all distinct (unique) values together into a single string. Works in the same way as joining() except that it discards duplicate values. Values are concatenated in the order that they are given to the function. If a delimiter is supplied then the delimiter is placed between each concatenated string. If a limit is supplied then it will only concatenate up to limit values.

distinct(values)
distinct(values, delimiter)
distinct(values, delimiter, limit)

Examples

distinct(${val}, ', ')
${val} = ['bill', 'bill', 'bob', 'fred', 'bill']
> 'bill, bob, fred'

distinct(${val}, '|', 2)
${val} = ['bill', 'bill', 'bob', 'fred', 'bill']
> 'bill|bob'

Joining

Concatenates all values together into a single string. Works in the same way as distinct() except that duplicate values are included. Values are concatenated in the order that they are given to the function. If a delimiter is supplied then the delimiter is placed between each concatenated string. If a limit is supplied then it will only concatenate up to limit values.

joining(values)
joining(values, delimiter)
joining(values, delimiter, limit)

Example

joining(${val}, ', ')
${val} = ['bill', 'bob', 'fred', 'bill']
> 'bill, bob, fred, bill'

Max

Determines the maximum value given in the args.

max(arg)

Examples

max(${val})
${val} = [100, 30, 45, 109]
> 109

# They can be nested
max(max(${val}), 40, 67, 89)
${val} = [20, 1002]
> 1002

Min

Determines the minimum value given in the args.

min(arg)

Examples

min(${val})
${val} = [100, 30, 45, 109]
> 30

# They can be nested
min(max(${val}), 40, 67, 89)
${val} = [20, 1002]
> 20

StDev (Standard Deviation)

Calculate the standard deviation for a set of input values.

stDev(arg)

Examples

round(stDev(${val}))
${val} = [600, 470, 170, 430, 300]
> 147

Sum

Sums all the arguments together

sum(arg)

Examples

sum(${val})
${val} = [89, 12, 3, 45]
> 149

Variance

Calculate the variance of a set of input values.

variance(arg)

Examples

variance(${val})
${val} = [600, 470, 170, 430, 300]
> 21704

5.2 - Cast Functions

To Boolean

Attempts to convert the passed value to a boolean data type.

toBoolean(arg1)

Examples:

toBoolean(1)
> true
toBoolean(0)
> false
toBoolean('true')
> true
toBoolean('false')
> false

To Double

Attempts to convert the passed value to a double data type.

toDouble(arg1)

Examples:

toDouble('1.2')
> 1.2

To Integer

Attempts to convert the passed value to a integer data type.

toInteger(arg1)

Examples:

toInteger('1')
> 1

To Long

Attempts to convert the passed value to a long data type.

toLong(arg1)

Examples:

toLong('1')
> 1

To String

Attempts to convert the passed value to a string data type.

toString(arg1)

Examples:

toString(1.2)
> '1.2'

5.3 - Date Functions

Parse Date

Parse a date and return a long number of milliseconds since the epoch. For details of the pattern syntax, see Dates & Times.

parseDate(aString)
parseDate(aString, pattern)
parseDate(aString, pattern, timeZone)

Example

parseDate('2014 02 22', 'yyyy MM dd', '+0400')
> 1393012800000

Format Date

Format a date supplied as milliseconds since the epoch. For details of the format pattern syntax, see Dates & Times.

formatDate(aLong)
formatDate(aLong, pattern)
formatDate(aLong, pattern, timeZone)

Example

formatDate(1393071132888, 'yyyy MM dd', '+1200')
> '2014 02 23'

Ceiling Year/Month/Day/Hour/Minute/Second

ceilingYear(args...)
ceilingMonth(args...)
ceilingDay(args...)
ceilingHour(args...)
ceilingMinute(args...)
ceilingSecond(args...)

Examples

ceilingSecond("2014-02-22T12:12:12.888Z"
> "2014-02-22T12:12:13.000Z"
ceilingMinute("2014-02-22T12:12:12.888Z"
> "2014-02-22T12:13:00.000Z"
ceilingHour("2014-02-22T12:12:12.888Z"
> "2014-02-22T13:00:00.000Z"
ceilingDay("2014-02-22T12:12:12.888Z"
> "2014-02-23T00:00:00.000Z"
ceilingMonth("2014-02-22T12:12:12.888Z"
> "2014-03-01T00:00:00.000Z"
ceilingYear("2014-02-22T12:12:12.888Z"
> "2015-01-01T00:00:00.000Z"

Floor Year/Month/Day/Hour/Minute/Second

floorYear(args...)
floorMonth(args...)
floorDay(args...)
floorHour(args...)
floorMinute(args...)
floorSecond(args...)

Examples

floorSecond("2014-02-22T12:12:12.888Z"
> "2014-02-22T12:12:12.000Z"
floorMinute("2014-02-22T12:12:12.888Z"
> "2014-02-22T12:12:00.000Z"
floorHour("2014-02-22T12:12:12.888Z"
> 2014-02-22T12:00:00.000Z"
floorDay("2014-02-22T12:12:12.888Z"
> "2014-02-22T00:00:00.000Z"
floorMonth("2014-02-22T12:12:12.888Z"
> "2014-02-01T00:00:00.000Z"
floorYear("2014-02-22T12:12:12.888Z"
> "2014-01-01T00:00:00.000Z"

Round Year/Month/Day/Hour/Minute/Second

roundYear(args...)
roundMonth(args...)
roundDay(args...)
roundHour(args...)
roundMinute(args...)
roundSecond(args...)

Examples

roundSecond("2014-02-22T12:12:12.888Z")
> "2014-02-22T12:12:13.000Z"
roundMinute("2014-02-22T12:12:12.888Z")
> "2014-02-22T12:12:00.000Z"
roundHour("2014-02-22T12:12:12.888Z"
> "2014-02-22T12:00:00.000Z"
roundDay("2014-02-22T12:12:12.888Z"
> "2014-02-23T00:00:00.000Z"
roundMonth("2014-02-22T12:12:12.888Z"
> "2014-03-01T00:00:00.000Z"
roundYear("2014-02-22T12:12:12.888Z"
> "2014-01-01T00:00:00.000Z"

5.4 - Link Functions

Links can be inserted into dashboard tables using the link function. All link types described in Internal Links can be added to dashboard tables using the link function. In addition to the link function there are convenience functions such as annotation, dashboard, data and stepping that make it easier to supply the required link parameters.

Annotation

A helper function to make forming links to annotations easier than using Link. The Annotation function allows you to create a link to open the Annotation editor, either to view an existing annotation or to begin creating one with pre-populated values.

annotation(text, annotationId)
annotation(text, annotationId, [streamId, eventId, title, subject, status, assignedTo, comment])

If you provide just the text and an annotationId then it will produce a link that opens an existing annotation with the supplied ID in the Annotation Edit dialog.

Example

annotation('Open annotation', ${annotation:Id})
> [Open annotation](?annotationId=1234){annotation}
annotation('Create annotation', '', ${StreamId}, ${EventId})
> [Create annotation](?annotationId=&streamId=1234&eventId=45){annotation}
annotation('Escalate', '', ${StreamId}, ${EventId}, 'Escalation', 'Triage required')
> [Escalate](?annotationId=&streamId=1234&eventId=45&title=Escalation&subject=Triage%20required){annotation}

If you don’t supply an annotationId then the link will open the Annotation Edit dialog pre-populated with the optional arguments so that an annotation can be created. If the annotationId is not provided then you must provide a streamId and an eventId. If you don’t need to pre-populate a value then you can use '' or null() instead.

Example

annotation('Create suspect event annotation', null(), 123, 456, 'Suspect Event', null(), 'assigned', 'jbloggs')
> [Create suspect event annotation](?streamId=123&eventId=456&title=Suspect%20Event&assignedTo=jbloggs){annotation}

Dashboard

A helper function to make forming links to dashboards easier than using Link.

dashboard(text, uuid)
dashboard(text, uuid, params)

Example

dashboard('Click Here','e177cf16-da6c-4c7d-a19c-09a201f5a2da')
> [Click Here](?uuid=e177cf16-da6c-4c7d-a19c-09a201f5a2da){dashboard}
dashboard('Click Here','e177cf16-da6c-4c7d-a19c-09a201f5a2da', 'userId=user1')
> [Click Here](?uuid=e177cf16-da6c-4c7d-a19c-09a201f5a2da&params=userId%3Duser1){dashboard}

Data

A helper function to make forming links to data easier than using Link.

data(text, id, partNo, [recordNo, lineFrom, colFrom, lineTo, colTo, viewType, displayType])

Example

data('Quick View', ${StreamId}, 1)
> [Quick View]?id=1234&&partNo=1)

Example of non-segmented raw data section, viewed un-formatted in a stroom tab:

data('View Raw', ${StreamId}, ${partNo}, null(), 5, 1, 5, 342, 'source', 'tab')

Example of a single record (event) from a segmented stream, viewed formatted in a popup dialog:

data('View Cooked', ${StreamId}, 1, ${eventId})

Example of a single record (event) from a segmented stream, viewed formatted in a stroom tab:

data('View Cooked', ${StreamId}, 1, ${eventId}, null(), null(), null(), null(), 'preview', 'tab')

Link

Create a string that represents a hyperlink for display in a dashboard table.

link(url)
link(text, url)
link(text, url, type)

Example

link('https://www.somehost.com/somepath')
> [https://www.somehost.com/somepath](https://www.somehost.com/somepath)
link('Click Here','https://www.somehost.com/somepath')
> [Click Here](https://www.somehost.com/somepath)
link('Click Here','https://www.somehost.com/somepath', 'dialog')
> [Click Here](https://www.somehost.com/somepath){dialog}
link('Click Here','https://www.somehost.com/somepath', 'dialog|Dialog Title')
> [Click Here](https://www.somehost.com/somepath){dialog|Dialog Title}

Stepping

Open the Stepping tab for the requested data source.

stepping(text, id)
stepping(text, id, partNo)
stepping(text, id, partNo, recordNo)

Example

stepping('Click here to step',${StreamId})
> [Click here to step](?id=1)

5.5 - Logic Funtions

Equals

Evaluates if arg1 is equal to arg2

arg1 = arg2
equals(arg1, arg2)

Examples

'foo' = 'bar'
> false
'foo' = 'foo'
> true
51 = 50
> false
50 = 50
> true

equals('foo', 'bar')
> false
equals('foo', 'foo')
> true
equals(51, 50)
> false
equals(50, 50)
> true

Note that equals cannot be applied to null and error values, e.g. x=null() or x=err(). The isNull() and isError() functions must be used instead.

Greater Than

Evaluates if arg1 is greater than to arg2

arg1 > arg2
greaterThan(arg1, arg2)

Examples

51 > 50
> true
50 > 50
> false
49 > 50
> false

greaterThan(51, 50)
> true
greaterThan(50, 50)
> false
greaterThan(49, 50)
> false

Greater Than or Equal To

Evaluates if arg1 is greater than or equal to arg2

arg1 >= arg2
greaterThanOrEqualTo(arg1, arg2)

Examples

51 >= 50
> true
50 >= 50
> true
49 >= 50
> false

greaterThanOrEqualTo(51, 50)
> true
greaterThanOrEqualTo(50, 50)
> true
greaterThanOrEqualTo(49, 50)
> false

If

Evaluates the supplied boolean condition and returns one value if true or another if false

if(expression, trueReturnValue, falseReturnValue)

Examples

if(5 < 10, 'foo', 'bar')
> 'foo'
if(5 > 10, 'foo', 'bar')
> 'bar'
if(isNull(null()), 'foo', 'bar')
> 'foo'

Less Than

Evaluates if arg1 is less than to arg2

arg1 < arg2
lessThan(arg1, arg2)

Examples

51 < 50
> false
50 < 50
> false
49 < 50
> true

lessThan(51, 50)
> false
lessThan(50, 50)
> false
lessThan(49, 50)
> true

Less Than or Equal To

Evaluates if arg1 is less than or equal to arg2

arg1 <= arg2
lessThanOrEqualTo(arg1, arg2)

Examples

51 <= 50
> false
50 <= 50
> true
49 <= 50
> true

lessThanOrEqualTo(51, 50)
> false
lessThanOrEqualTo(50, 50)
> true
lessThanOrEqualTo(49, 50)
> true

And

If all supplied arguments evaluate to true then return true, else false.

and(booleanValue, booleanValue)

Or

If one or more of the supplied arguments evaluate to true then return true, else false.

or(booleanValue, booleanValue)

Not

Inverts boolean values making true, false etc.

not(booleanValue)

Examples

not(5 > 10)
> true
not(5 = 5)
> false
not(false())
> true

5.6 - Mathematics Functions

Add

arg1 + arg2

Or reduce the args by successive addition

add(args...)

Examples

34 + 9
> 43
add(45, 6, 72)
> 123

Divide

Divides arg1 by arg2

arg1 / arg2

Or reduce the args by successive division

divide(args...)

Examples

42 / 7
> 6
divide(1000, 10, 5, 2)
> 10
divide(100, 4, 3)
> 8.33

Max

Determines the maximum value given in the args

max(args...)

Examples

max(100, 30, 45, 109)
> 109

# They can be nested
max(max(${val}), 40, 67, 89)
${val} = [20, 1002]
> 1002

Min

Determines the minimum value given in the args

min(args...)

Examples

min(100, 30, 45, 109)
> 30

They can be nested

min(max(${val}), 40, 67, 89)
${val} = [20, 1002]
> 20

Modulo

Determines the modulus of the dividend divided by the divisor.

modulo(dividend, divisor)

Examples

modulo(100, 30)
> 10

Multiply

Multiplies arg1 by arg2

arg1 * arg2

Or reduce the args by successive multiplication

multiply(args...)

Examples

4 * 5
> 20
multiply(4, 5, 2, 6)
> 240

Negate

Multiplies arg1 by -1

negate(arg1)

Examples

negate(80)
> -80
negate(23.33)
> -23.33
negate(-9.5)
> 9.5

Power

Raises arg1 to the power arg2

arg1 ^ arg2

Or reduce the args by successive raising to the power

power(args...)

Examples

4 ^ 3
> 64
power(2, 4, 3)
> 4096

Random

Generates a random number between 0.0 and 1.0

random()

Examples

random()
> 0.78
random()
> 0.89
...you get the idea

Subtract

arg1 - arg2

Or reduce the args by successive subtraction

subtract(args...)

Examples

29 - 8
> 21
subtract(100, 20, 34, 2)
> 44

Sum

Sums all the arguments together

sum(args...)

Examples

sum(89, 12, 3, 45)
> 149

Rounding Functions

These functions require a value, and an optional decimal places. If the decimal places are not given it will give you nearest whole number.

Ceiling

ceiling(value, decimalPlaces<optional>)

Examples

ceiling(8.4234)
> 9
ceiling(4.56, 1)
> 4.6
ceiling(1.22345, 3)
> 1.223

Floor

floor(value, decimalPlaces<optional>)

Examples

floor(8.4234)
> 8
floor(4.56, 1)
> 4.5
floor(1.2237, 3)
> 1.223

Round

round(value, decimalPlaces<optional>)

Examples

round(8.4234)
> 8
round(4.56, 1)
> 4.6
round(1.2237, 3)
> 1.224

Statistical Functions

Average

Takes an average value of the arguments. The alias mean can be used instead.

average(args...)
mean(args...)

Examples

average(10, 20, 30, 40)
> 25
mean(8.9, 24, 1.2, 1008)
> 260.525

StDev (Standard Deviation)

Calculate the standard deviation for a set of input values.

stDev(args...)

Examples

round(stDev(600, 470, 170, 430, 300))
> 147

Variance

Calculate the variance of a set of input values.

variance(args...)

Examples

variance(600, 470, 170, 430, 300)
> 21704

5.7 - Selection Functions

Selection functions are a form of aggregate function operating on grouped data. They select a sub-set of the child values.

Any

Selects the first value found in the group that is not null() or err(). If no explicit ordering is set then the value selected is indeterminate.

any(${val})

Examples

any(${val})
${val} = [10, 20, 30, 40]
> 10

Bottom

Selects the bottom N values and returns them as a delimited string in the order they are read.

bottom(${val}, delimiter, limit)

Example

bottom(${val}, ', ', 2)
${val} = [10, 20, 30, 40]
> '30, 40'

First

Selects the first value found in the group even if it is null() or err(). If no explicit ordering is set then the value selected is indeterminate.

first(${val})

Example

first(${val})
${val} = [10, 20, 30, 40]
> 10

Last

Selects the last value found in the group even if it is null() or err(). If no explicit ordering is set then the value selected is indeterminate.

last(${val})

Example

last(${val})
${val} = [10, 20, 30, 40]
> 40

Nth

Selects the Nth value in a set of grouped values. If there is no explicit ordering on the field selected then the value returned is indeterminate.

nth(${val}, position)

Example

nth(${val}, 2)
${val} = [20, 40, 30, 10]
> 40

Top

Selects the top N values and returns them as a delimited string in the order they are read.

top(${val}, delimiter, limit)

Example

top(${val}, ', ', 2)
${val} = [10, 20, 30, 40]
> '10, 20'

5.8 - String Functions

Concat

Appends all the arguments end to end in a single string

concat(args...)

Example

concat('this ', 'is ', 'how ', 'it ', 'works')
> 'this is how it works'

Contains

Tests if inputString contains subString.

contains(inputString, subString)

Example

contains('this', 'this')
> true
contains('this', 'that')
> false

Current User

Returns the username of the user running the query.

currentUser()

Example

currentUser()
> 'jbloggs'

Decode

The arguments are split into 3 parts

1. The input value to test
2. Pairs of regex matchers with their respective output value
3. A default result, if the input doesn’t match any of the regexes

decode(input, test1, result1, test2, result2, ... testN, resultN, otherwise)

It works much like a Java Switch/Case statement

Example

decode(${val}, 'red', 'rgb(255, 0, 0)', 'green', 'rgb(0, 255, 0)', 'blue', 'rgb(0, 0, 255)', 'rgb(255, 255, 255)')
${val}='blue'
> rgb(0, 0, 255)
${val}='green'
> rgb(0, 255, 0)
${val}='brown'
> rgb(255, 255, 255) // falls back to the 'otherwise' value

in Java, this would be equivalent to

String decode(value) {
    switch(value) {
        case "red":
            return "rgb(255, 0, 0)"
        case "green":
            return "rgb(0, 255, 0)"
        case "blue":
            return "rgb(0, 0, 255)"
        default:
            return "rgb(255, 255, 255)"
    }
}

decode('red')
> 'rgb(255, 0, 0)'

DecodeUrl

Decodes a URL

decodeUrl('userId%3Duser1')
> userId=user1

EncodeUrl

Encodes a URL

encodeUrl('userId=user1')
> userId%3Duser1

Exclude

If the supplied string matches one of the supplied match strings then return null, otherwise return the supplied string

exclude(aString, match...)

Example

exclude('hello', 'hello', 'hi')
> null
exclude('hi', 'hello', 'hi')
> null
exclude('bye', 'hello', 'hi')
> 'bye'

Hash

Cryptographically hashes a string

hash(value)
hash(value, algorithm)
hash(value, algorithm, salt)

Example

hash(${val}, 'SHA-512', 'mysalt')
> A hashed result...

If not specified the hash() function will use the SHA-256 algorithm. Supported algorithms are determined by Java runtime environment.

Include

If the supplied string matches one of the supplied match strings then return it, otherwise return null

include(aString, match...)

Example

include('hello', 'hello', 'hi')
> 'hello'
include('hi', 'hello', 'hi')
> 'hi'
include('bye', 'hello', 'hi')
> null

Index Of

Finds the first position (zero based) of subString in inputString or -1 if it cannot be found. Uses a simple literal match.

indexOf(inputString, subString)

Example

indexOf('aa-bb-cc', '-')
> 2

Last Index Of

Finds the last position (zero based) of subString in inputString or -1 if it cannot be found. Uses a simple literal match.

lastIndexOf(inputString, subString)

Example

lastIndexOf('aa-bb-cc', '-')
> 5

Lower Case

Converts the string to lower case

lowerCase(aString)

Example

lowerCase('Hello DeVeLoPER')
> 'hello developer'

Match

Test an input string using a regular expression to see if it matches

match(input, regex)

Example

match('this', 'this')
> true
match('this', 'that')
> false

Query Param

Returns the value of the requested query parameter.

queryParam(paramKey)

Examples

queryParam('user')
> 'jbloggs'

Query Params

Returns all query parameters as a space delimited string.

queryParams()

Examples

queryParams()
> 'user=jbloggs site=HQ'

Replace

Perform text replacement on an input string using a regular expression to match part (or all) of the input string and a replacement string to insert in place of the matched part

replace(input, regex, replacement)

Example

replace('this', 'is', 'at')
> 'that'

String Length

Takes the length of a string

stringLength(aString)

Example

stringLength('hello')
> 5

Substring

Take a substring based on start/end index of letters

substring(aString, startIndex, endIndex)

Example

substring('this', 1, 2)
> 'h'

Substring After

Get the substring from the first string that occurs after the presence of the second string

substringAfter(firstString, secondString)

Example

substringAfter('aa-bb', '-')
> 'bb'

Substring Before

Get the substring from the first string that occurs before the presence of the second string

substringBefore(firstString, secondString)

Example

substringBefore('aa-bb', '-')
> 'aa'

Upper Case

Converts the string to upper case

upperCase(aString)

Example

upperCase('Hello DeVeLoPER')
> 'HELLO DEVELOPER'

5.9 - Type Checking Functions

Is Boolean

Checks if the passed value is a boolean data type.

isBoolean(arg1)

Examples:

isBoolean(toBoolean('true'))
> true

Is Double

Checks if the passed value is a double data type.

isDouble(arg1)

Examples:

isDouble(toDouble('1.2'))
> true

Is Error

Checks if the passed value is an error caused by an invalid evaluation of an expression on passed values, e.g. some values passed to an expression could result in a divide by 0 error. Note that this method must be used to check for error as error equality using x=err() is not supported.

isError(arg1)

Examples:

isError(toLong('1'))
> false
isError(err())
> true

Is Integer

Checks if the passed value is an integer data type.

isInteger(arg1)

Examples:

isInteger(toInteger('1'))
> true

Is Long

Checks if the passed value is a long data type.

isLong(arg1)

Examples:

isLong(toLong('1'))
> true

Is Null

Checks if the passed value is null. Note that this method must be used to check for null as null equality using x=null() is not supported.

isNull(arg1)

Examples:

isNull(toLong('1'))
> false
isNull(null())
> true

Is Number

Checks if the passed value is a numeric data type.

isNumber(arg1)

Examples:

isNumber(toLong('1'))
> true

Is String

Checks if the passed value is a string data type.

isString(arg1)

Examples:

isString(toString(1.2))
> true

Is Value

Checks if the passed value is a value data type, e.g. not null or error.

isValue(arg1)

Examples:

isValue(toLong('1'))
> true
isValue(null())
> false

Type Of

Returns the data type of the passed value as a string.

typeOf(arg1)

Examples:

typeOf('abc')
> string
typeOf(toInteger(123))
> integer
typeOf(err())
> error
typeOf(null())
> null
typeOf(toBoolean('false'))
> false

5.10 - URI Functions

Fields containing a Uniform Resource Identifier (URI) in string form can queried to extract the URI’s individual components of authority, fragment, host, path, port, query, scheme, schemeSpecificPart and userInfo. See either RFC 2306: Uniform Resource Identifiers (URI): Generic Syntax or Java’s java.net.URI Class for details regarding the components. If any component is not present within the passed URI, then an empty string is returned.

The extraction functions are

• extractAuthorityFromUri() - extract the Authority component
• extractFragmentFromUri() - extract the Fragment component
• extractHostFromUri() - extract the Host component
• extractPathFromUri() - extract the Path component
• extractPortFromUri() - extract the Port component
• extractQueryFromUri() - extract the Query component
• extractSchemeFromUri() - extract the Scheme component
• extractSchemeSpecificPartFromUri() - extract the Scheme specific part component
• extractUserInfoFromUri() - extract the UserInfo component

If the URI is http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details the table below displays the extracted components

extractAuthorityFromUri

Extracts the Authority component from a URI

extractAuthorityFromUri(uri)

Example

extractAuthorityFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'foo:bar@w1.superman.com:8080'

extractFragmentFromUri

Extracts the Fragment component from a URI

extractFragmentFromUri(uri)

Example

extractFragmentFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'more-details'

extractHostFromUri

Extracts the Host component from a URI

extractHostFromUri(uri)

Example

extractHostFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'w1.superman.com'

extractPathFromUri

Extracts the Path component from a URI

extractPathFromUri(uri)

Example

extractPathFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> '/very/long/path.html'

extractPortFromUri

Extracts the Port component from a URI

extractPortFromUri(uri)

Example

extractPortFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> '8080'

extractQueryFromUri

Extracts the Query component from a URI

extractQueryFromUri(uri)

Example

extractQueryFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'p1=v1&p2=v2'

extractSchemeFromUri

Extracts the Scheme component from a URI

extractSchemeFromUri(uri)

Example

extractSchemeFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'http'

extractSchemeSpecificPartFromUri

Extracts the SchemeSpecificPart component from a URI

extractSchemeSpecificPartFromUri(uri)

Example

extractSchemeSpecificPartFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> '//foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2'

extractUserInfoFromUri

Extracts the UserInfo component from a URI

extractUserInfoFromUri(uri)

Example

extractUserInfoFromUri('http://foo:bar@w1.superman.com:8080/very/long/path.html?p1=v1&p2=v2#more-details')
> 'foo:bar'

5.11 - Value Functions

Err

Returns err

err()

False

Returns boolean false

false()

Null

Returns null

null()

True

Returns boolean true

true()

6 - Background Jobs

The jobs in the list are in the order they appear in the Stroom UI.

Data Delete

Before data is physically removed from the database and file system it is marked as logically deleted by adding a flag to the metadata record in the database. Data can be logically deleted by a user from the UI or via a process such as data retention. Data is deleted logically as it is faster to do than a physical delete (important in the UI), and it also allows for data to be restored (undeleted) from the UI. This job performs the actual physical deletion of data that has been marked logically deleted for longer than the duration configured with stroom.data.store.deletePurgeAge. All data files associated with a metadata record are deleted from the file system before the metadata is physically removed from the database.

Data Processor

Processes data by finding data that matches processing filters on each pipeline. When enabled, each worker node asks the master node for data processing tasks. The master node creates tasks based on processing filters added to the Processors screen of each pipeline and supplies them to the requesting workers.

Node Status

How frequently we try write stats about node status including JVM and memory usage.

Processor Task Creator

Create Processor Tasks from Processor Filters.

Query History Clean

How frequently items in the query history are removed from the history if their age is older than stroom.history.daysRetention or if the number of items in the history exceeds stroom.history.itemsRetention.

Account Maintenance

This job checks user accounts on the system and de-activates them under the following conditions:

• An unused account that has been inactive for longer than the age configured by stroom.security.identity.passwordPolicy.neverUsedAccountDeactivationThreshold.
• An account that has been inactive for longer than the age configured by stroom.security.identity.passwordPolicy.unusedAccountDeactivationThreshold.

Analytic Executor: Scheduled Query

Run scheduled index query analytics periodically

Analytic Executor: Table Builder

Analytic Executor: Table Builder

Attribute Value Data Retention

Deletes Meta attribute values (additional and less valuable metadata) older than stroom.data.meta.metaValue.deleteAge.

Elastic Index Retention

Logically delete indexed documents in Elasticsearch indexes based on the specified deletion query.

File System Volume Status

Scans your data volumes to ensure they are available and determines how much free space they have. Records this status in the Volume Status table.

Index Shard Delete

How frequently index shards that have been logically deleted are physically deleted from the file system.

Index Shard Retention

How frequently index shards that are older then their retention period are logically deleted.

Index Volume Status

Scans your index volumes to ensure they are available and determines how much free space they have. Records this status in the Index Volume Status table.

Index Writer Cache Sweep

How frequently entries in the Index Shard Writer cache are evicted based on the time-to-live, time-to-idle and cache size settings.

Index Writer Flush

How frequently in-memory changes to the index shards are flushed to the file system and committed to the index.

Java Heap Histogram Statistics

How frequently heap histogram statistics will be captured. This can be useful for diagnosing issues or seeing where memory is being used. Each run will result in a JVM pause so car should be taken when running this on a production system.

Orphan File Finder

Job to find files that do not exist in the meta store.

Orphan Meta Finder

Job to find items in the meta store that have no associated data.

Pipeline Destination Roll

How frequently rolling pipeline destinations, e.g. a Rolling File Appender are checked to see if they need to be rolled. This frequency should be at least as short as the most frequent rolling frequency.

Policy Based Data Retention

Run the policy based data retention rules over the data and logically delete and data that should no longer be retained.

Processor Task Manager Disown Dead Tasks

Tasks that seem to be stuck processing due to the death of a processing node are disowned and added back to the task queue for processing after stroom.processor.disownDeadTasksAfter.

Processor Task Manager Release Old Queued Tasks

Release queued tasks from old master nodes.

Processor Task Queue Statistics

How frequently statistics about the state of the stream processing task queue are captured.

Processor Task Retention

This job is responsible for cleaning up redundant processors, tasks and filters. If it is not run then these will build up on the system consuming space in the database.

This job relies on the property stroom.processor.deleteAge to govern what is deemed old. The deleteAge is used to derive the delete threshold, i.e. the current time minus deleteAge.

When the job runs it executes the following steps:

1. Logically Delete Processor Tasks - Logically delete all processor tasks belonging to processor filters that have been logically deleted.

2. Logically Delete Processor Filters - Logically delete old processor filters with a state of COMPLETE and no associated tasks. Filters are considered old if the last poll time is less than the delete threshold.

3. Physically Delete Processor Tasks - Physically delete all old processor tasks with a status of COMPLETE or DELETED. Tasks are considered old if they have no status time or the status time (the time the status was last changed) is less than the delete threshold.

4. Physically Delete Processor Filters - Physically delete all old processor filters that have already been logically deleted. Filters are considered old if the last update time is less than the delete threshold. A filter can be logically deleted either by the step above or explicitly by a user in the user interface.

5. Physically Delete Processors - Physically delete all old processors that have already been logically deleted. Processors are considered old if the last update time is less than the delete threshold. A processor can only be logically deleted by the user in the user interface.

Therefore for items not deleted by a user, there will be a delay equal to deleteAge before logical deletion, then another delay equal to deleteAge before final physical deletion.

Property Cache Reload

Stroom’s configuration properties can each be configured globally in the database. This job controls the frequency that each node refreshes the values of its properties cache from the global database values. See also Properties.

Ref Data Off-heap Store Purge

Purges all data older than the purge age defined by property stroom.pipeline.purgeAge. See also Reference Data.

SQL Stats Database Aggregation

This job controls the frequency that the database statistics aggregation process is run. This process takes the entries in SQL_STAT_VAL_SRC and merges them into the main statistics tables SQL_STAT_KEY and SQL_STAT_KEY. As this process is reliant on data flushed by the SQL Stats In Memory Flush job it is advisable to schedule it to run after that, leaving some time for the in-memory flush to finish.

SQL Stats In Memory Flush

SQL Statistics are initially held and aggregated in memory. This job controls the frequency that the in memory statistics are flushed from the in memory buffer to the staging table SQL_STAT_VAL_SRC in the database.

Solr Index Optimise

How frequently Solr index segments are explicitly optimised by merging them into one.

Solr Index Retention

How frequently a process is run to delete items from the Solr indexes that don’t meet the retention rule of that index.

7 - Keyboard Shortcuts

This section describes all the keyboard shortcuts that are available to use in Stroom. Some shortcuts apply everywhere and some are specific to the screen that you are in or the user interface component that has focus.

Keyboard shortcuts can take two forms:

• Combination - Multiple keys pressed at the same time, e.g. Ctrl ^ + Enter ↵ .
• Sequence* - Multiple keys pressed in sequence with only one key pressed at a time, e.g. g , t , press

General Shortcuts

Movement

Movement within lists, selection boxes or tab bars you can use the cursor keys, hjkl or wasd to move between items.

You can also move up or down by page using:

Finding Things

Direct Access to Screens

The following shortcuts are all of the sequential type with the mnemonic Goto X. These shortcuts may not do anything if you do not have the required permissions for the screen in question.

Creating New Documents

The following shortcuts will open the dialog to create a new document in the currently selected explorer tree folder. If a document is currently selected in the explorer tree then the new document will be created in the same folder as the selected document. If nothing is selected or multiple items are selected then these key sequences have no effect.

Screen Specific Shortcuts

Dashboard

The following shortcuts are available when editing a Dashboard .

Pipeline Stepping

The following shortcuts are available when stepping a pipeline.

Query

The following shortcuts are available when editing a Query .

Text Editor

The following common shortcuts are available when editing text editing text in the Ace text editor that is used on many screens in Stroom, e.g. when editing a Pipeline or Query.

See Ace Default keyboard shortcuts for more.

If you know Vim key bindings then the Ace editor supports a reasonable sub-set of them, see Vim Key Bindings.

8 - Pipeline Elements

Reader

Reader elements decode the data in raw byte form using the Feed’s configured character encoding. Some of them will also transform the data at the character level before the data are parsed into a structured form.

BOMRemovalFilterInput

BOMRemovalFilterInput  

Removes the Byte Order Mark (if present) from the stream.

BadTextXMLFilterReader

BadTextXMLFilterReader  

Escapes the content of a configured list of named XML elements that are know to potentially contain un-escaped XML reserved characters. For example the element <Expression>$time < now()</Expression> would be transformed to <Expression>$time &lt; now()</Expression> if property leafList is set to Expression.

Element properties:

FindReplaceFilter

FindReplaceFilter  

Replaces strings or regexes with new strings.

Element properties:

InvalidCharFilterReader

InvalidCharFilterReader  

Removes any characters that are not in the standard XML character set. The version of XML (e.g. 1.0 or 1.1) can be set using the ‘xmlVersion’ property.

Element properties:

InvalidXMLCharFilterReader

InvalidXMLCharFilterReader  

Replaces any characters that are not in the standard XML character set with a ‘�’. The version of XML (e.g. 1.0 or 1.1) can be set using the ‘xmlVersion’ property.

Element properties:

Reader

Reader  

A basic reader that simply decodes the raw bytes using the Feed’s configured character encoding. It does not transform the data in any other way.

Parser

Parser elements parse raw text data that has an expected structure (e.g. XML, JSON, CSV) into XML events (elements, attributes, text, etc) that can be further validated or transformed using XSLT. The choice of Parser will be dictated by the structure of the data. If no Reader is used before the Parser, the Parser will also do the job of a simple Reader and decode the raw bytes using the Feed’s configured character encoding.

CombinedParser

CombinedParser  

The original general-purpose reader/parser that covers all source data types but provides less flexibility than the source format-specific parsers such as dsParser. It effectively combines a BOMRemovalFilterInput, an InvalidCharFilterReader and Parser (based on the type property.

Element properties:

DSParser

DSParser  

A parser for handling structured plain text data (e.g. CSV or fixed width fields) using the Data Splitter domain specific language. For more details see Data Splitter.

Element properties:

JSONParser

JSONParser  

A built-in parser for parsing JSON source data (in JSON fragment format) into an XML representation of the JSON. The Resulting XML will conform to the http://www.w3.org/2013/XSL/json namespace.

Element properties:

XMLFragmentParser

XMLFragmentParser  

A parser to convert multiple XML fragments into an XML document. For example the data may contain:

<Event>...</Event>
<Event>...</Event>

i.e. with no root element, so not valid XML. The XMLFragmentParser will wrap the fragments with a root element as defined in the TextConverter document configured with the textConverterRef property.

Element properties:

XMLParser

XMLParser  

A parser to parse data that is expected to be XML into a series of XML events that can be consumed by a Filter element.

Filter

Filter elements work with XML events that have been generated by a parser. They can consume the events without modifying them, e.g. RecordCountFilter or modify them in some way, e.g. XSLTFilter. Multiple filters can be used one after another with each using the output from the last as its input.

DynamicIndexingFilter

DynamicIndexingFilter  

A filter to send source data to an index.

Element properties:

DynamicSearchResultOutputFilter

DynamicSearchResultOutputFilter  

Used in a search extraction pipeline for extracting field values that have not been stored in the index and where the fields are dynamic and derived from the data rather than being defined in the Index settings. Consumes XML events in the index-documents:1 namespace to convert them into a form so that they can be used in a Dashboard/Query/Analytic.

ElasticIndexingFilter

ElasticIndexingFilter  

A filter consuming XML events in the records:2 namespace to index/store the fields and their values in an Elasticsearch Index.

Element properties:

HttpPostFilter

HttpPostFilter  

This element is deprecated, you should instead use the much more flexible HTTPAppender . This element will simply POST the output of the XML events to the configured URL.

Element properties:

IdEnrichmentFilter

IdEnrichmentFilter  

Adds the attributes ‘StreamId’ and ‘EventId’ to the ’event’ element to enrich the event with its ordinal number in the stream and the ID of the stream that it belongs to. ID enrichment is required to be able to index events as it provides them with an ID that is unique within Stroom. It assumes that an record/event is an XML element at the first level below the root element, i.e. for ’event-logging:3’ XML this means the <Event> element.

IndexingFilter

IndexingFilter  

A filter consuming XML events in the records:2 namespace to index/store the fields and their values in a Lucene Index.

Element properties:

RecordCountFilter

RecordCountFilter  

Counts events/records in the stream. An event/record is taken to be an XML element that is at the first level below the root element, i.e. for ’event-logging:3’ XML this means the <Event> element.

Element properties:

RecordOutputFilter

RecordOutputFilter  

Filters out records/events that have raised an Error or Fatal Error during processing. If all records/events have raised at least an Error then no XML events will be output. It assumes that an record/event is an XML element at the first level below the root element, i.e.
for ’event-logging:3’ XML this means the <Event> element.""",

ReferenceDataFilter

ReferenceDataFilter  

Takes XML input (conforming to the reference-data:2 schema) and loads the data into the Reference Data Store. Reference data values can be either simple strings or XML fragments.

Element properties:

SafeXMLFilter

SafeXMLFilter  

Restricts the characters to a very simple set consisting of [a-zA-Z0-9] and [ .:-_/]. All other characters are replaced by ~NNN, where NNN is a three digit codepoint for the replaced character.

SchemaFilter

SchemaFilter  

Checks the format of the source data against one of a number of XML schemas. This ensures that if non-compliant data is generated, it will be flagged as in error and will not be passed to any subsequent processing elements.

Element properties:

SearchResultOutputFilter

SearchResultOutputFilter  

Used in a search extraction pipeline for extracting field values that have not been stored in the index and where the field definitions are defined in the Index settings. Consumes XML events in the records:2 namespace to convert them into a form so that they can be used in a Dashboard/Query/Analytic.

SolrIndexingFilter

SolrIndexingFilter  

Delivers source data to the specified index in an external Solr instance/cluster.

Element properties:

SplitFilter

SplitFilter  

Splits multi-record source data into smaller groups of records prior to delivery to an XSLT. This allows the XSLT to process data more efficiently than loading a potentially huge input stream into memory.

Element properties:

StateFilter

StateFilter  

Takes XML input (conforming to the reference-data:2 schema) and loads the data into the State Store. Reference data values can be either simple strings or XML fragments.

StatisticsFilter

StatisticsFilter  

An element to allow the source data (conforming to the statistics XML Schema) to be sent to the MySQL based statistics data store.

Element properties:

StroomStatsFilter

StroomStatsFilter  

An element to allow the source data (conforming to the statistics XML Schema) to be sent to an external stroom-stats service.

Element properties:

XPathExtractionOutputFilter

XPathExtractionOutputFilter  

TODO - Add description

Element properties:

XSLTFilter

XSLTFilter  

An element used to transform XML data from one form to another using XSLT. The specified XSLT can be used to transform the input XML into XML conforming to another schema or into other forms such as JSON, plain text, etc.

Element properties:

Writer

Writers consume XML events (from Parsers and Filters) and convert them into a stream of bytes using the character encoding configured on the Writer (if applicable). The output data can then be fed to a Destination.

JSONWriter

JSONWriter  

Writer to convert XML data conforming to the http://www.w3.org/2013/XSL/json XML Schema into JSON format.

Element properties:

TextWriter

TextWriter  

Writer to convert XML character data events into plain text output.

Element properties:

XMLWriter

XMLWriter  

Writer to convert XML events data into XML output in the specified character encoding.

Element properties:

Destination

Destination elements consume a stream of bytes from a Writer and persist then to a destination. This could be a file on a file system or to Stroom’s stream store.

AnnotationWriter

AnnotationWriter  

Consume XML documents in the annotation:1 namespace and writes them as Stroom Annotations. Allows for the annotating of events that meet some criteria.

FileAppender

FileAppender  

A destination used to write an output stream to a file on the file system. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random to write to.

Element properties:

HDFSFileAppender

HDFSFileAppender  

A destination used to write an output stream to a file on a Hadoop Distributed File System. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random.

Element properties:

HTTPAppender

HTTPAppender  

A destination used to write an output stream to a remote HTTP(S) server.

This element should be preferred over the deprecated HttpPostFilter .

Element properties:

RollingFileAppender

RollingFileAppender  

A destination used to write an output stream to a file on the file system. If multiple paths are specified in the ‘outputPaths’ property it will pick one at random to write to. This is distinct from the FileAppender in that when the rollSize is reached it will move the current file to the path specified in rolledFileName and resume writing to the original path. This allows other processes to follow the changes to a single file path, e.g. when using tail. On system shutdown all active files will be rolled.

Element properties:

RollingStreamAppender

RollingStreamAppender  

A destination used to write one or more output streams to a new stream which is then rolled when it reaches a certain size or age. A new stream will be created after the size or age criteria has been met. On system shutdown all active streams will be rolled.

Element properties:

S3Appender

S3Appender  

A destination used to write an output stream to an S3 bucket.

Element properties:

StandardKafkaProducer

StandardKafkaProducer  

Consumes XML documents in the kafka-records:2 namespace. For each <kafkaRecord> element converts it into a Kafka message that is passed to the Kafka producer defined by the kafkaConfig property

Element properties:

StreamAppender

StreamAppender  

A destination used to write the output stream to a new stream in the stream store. The configuration allows for starting a new stream once a size threshold is reached.

Element properties:

StroomStatsAppender

StroomStatsAppender  

This element is deprecated and should not be used.

Element properties:

9 - Templating

Overview

Templating is the process of creating a reusable format or layout for presenting data in a consistent manner. Templating is currently used in Stroom for creating email templates for Analytic Rule detections.

Stroom’s templating uses a sub-set of the template syntax called jinja and specifically the JinJava library. The templating syntax includes support for variables, filters, condition blocks, loops, etc. Full details of the syntax can be found here .

When a template is rendered, Stroom will populate the template context with data that can be used by the template.

Basic Templating Syntax

Jinja templating is very powerful and has a rich language so this is quick guide to the very basic features. See the full syntax here here .

Data Types

The following data types are available in the Jinja language.

Accessing Collection Items

A List/tuple item can be accessed by its index (zero based), e.g. fruits[0] returns Apple.

A value in a dictionary can be accessed using its key, e.g. myDict['fruit'] returns Apple. If the key does not contain special characters (with the exception of _, then you can also used this form myDict.fruit to get the same value.

Conditions

Valid conditions are:

• == Compares two objects for equality
• != Compares two objects for inequality
• > true if the left-hand side is greater than the right-hand side
• >= true if the left-hand side is greater or equal to the right-hand side
• < true if the left-hand side is lower than the right-hand side
• <= true if the left-hand side is lower or equal to the right-hand side

Logical Operators

The logic operators for use with boolean values are:

• and Boolean and
• or Boolean or
• not Boolean negation

Expressions

Syntax: {{ ... }}

Expressions will render the value of a variable that has been defined in the template context or in a previous statement. An example of a simple value expression is

The detection time is {{detectionTime}}.

The detection time is 2024-05-03T09:15:13.454Z.

An expression can also contain variables that are passed through one or more Filters.

Statements

Syntax: {% ... %}

Statements are used to create conditional blocks, loops, define variables, etc.

Setting Variables

Syntax: {% set <variable name> = <value or expression> %}

{% set colour = '#F7761F' %}
Colour: {{colour}}
{% set fruits = ["Apple", "Orange"] %}
Top Fruit: {{ fruits | first }}
{% set fullName = "% %" | format(firstName, surname) %}
Name: {{fullName}}

Colour: #F7761F
Top Fruit: Apple
Name: Joe Bloggs

Conditional Blocks

Syntax:

{% if <value, variable or expression> <condition> <value, variable or expression> %}
  < optional content, expressions or statements>
{% elif <value, variable or expression> <condition> <value, variable or expression>%}
  < optional content, expressions or statements>
{% else %}
  < optional content, expressions or statements>
{% endif %}

Conditional blocks can be used to optional render content depending on the value of a variable. See conditions for the list of valid conditions.

{% if (values | length) > 0 -%}
This detection has {{ values | length }} values.
{%- else -%}
This detection has no values.
{%- endif -%}

This detection has 10 values.

Loops

Syntax:

{% for <item name> in <variable or expression> %}
  <content, expressions or statements to repeat for each item>
{% endif %}

For loops allow you to loop over items in a list/tuple or entries in a dictionary.

{% for key, val in values | dictsort %}
{{ key }}: {{ val }}
{% endfor %}

fruit: Apple
weight: 320

Filters

Syntax: ... | <filter name>

Filters are essentially functions that take an input and return an output. Some functions have additional parameters to control the behaviour of the function.

Filters can be chained together with the output of one filter becoming the input to the next filter in the chain in the same way that Stroom pipeline elements work.

Some useful filters are:

For a full list of filters see here or here .

Comments

Syntax: {# <comment text> #}

Non-rendered Comments can be added to the template.

{#- Show the time -#}
The detection time is {{detectionTime}}.

The detection time is 2024-05-03T09:15:13.454Z.

White Space

When JinJava renders the template, each expression or statement is evaluated and then removed or replaced by it’s output, but any white space around them, e.g. line breaks remain. This can result in unwanted line breaks in the output.

To avoid unwanted white space you can add the - character to the opening and/or closing tag to strip leading/trailing whitespace outside the block, e.g.

• {{ ... }} => {{- ... -}}
• {% ... %} => {%- ... -%}

{{ firstName -}}
{{ surname }}

JoeBloggs

For a detailed guide to how white space works see here .

Template Context

The context is a data structure that contains the dynamic content to use when rendering the template. The variables and values in the context are set by Stroom depending on where the template is being used.

Rule Detections Context

When an email subject/body template is rendered for an Analytic Rule detection, field values from the detection are placed in the context which allows them to be used in the template.

For example {{ detectTime }} will render the date/time the detection happened. The fields available in the context are those taken from the detection:1 XMLSchema with some additional fields.

The template has access to the following fields from the detection:

DetectionLinkedEvent fields: