Stroom Pipeline Functions

Functions for obtaining information about the current pipeline process.

`add-meta`

add-meta(key, value)➡Empty Sequence

Adds a metadata key and value to the output destination. Blank keys are ignored.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The metadata key to write.
`value`	String		-	-		-	The metadata value to write.

Return Value:

Type	Description
Empty Sequence	An empty sequence.

`bitmap-lookup`

bitmap-lookup(map, key, [time], [ignoreWarnings], [trace], [delimiter])➡Sequence

The bitmap-lookup() function looks up a bitmap key from reference or context data a value (which can be an XML node set) for each set bit position and adds it to the resultant XML.

If the look up fails no result will be returned.

Function Arguments:

Param	Type	Default Value	Allowed Values	Min Count	Description
`map`	String	-	-	-	The name of the map to perform the lookup in.
`key`	String	-	-	-	The key to lookup in the named map.
`time`	String	-	-	-	Determines which set of reference data was effective at the requested time. If no reference data exists with an effective time before the requested time then the lookup will fail. Time is in the format `yyyy-MM-dd'T'HH:mm:ss.SSSXX`, e.g. `2010-01-01T00:00:00.000Z`.
`ignoreWarnings`	Boolean	-	-	-	If true, any lookup failures will be ignored, else they will be reported as warnings.
`trace`	Boolean	-	-	-	If true, additional trace information is output as INFO messages.
`delimiter`	String	`<space>`	-	-	The delimiter to use when concatenating multiple lookup values. The default delimiter is a space.

Return Value:

Type	Description
Sequence	The matching value or XML nodes, or an empty sequence if no value is found.

Examples

For the purposes of these examples, the reference data store map contains:

Key (Bit position)	Value
0	Administrator
1	Manage_Users
2	Perform_Export
3	View_Data
4	Manage_Jobs
5	Delete_Data
6	Manage_Volumes

The following are example lookups using the above reference data:

Lookup Key (decimal)	Lookup Key (Hex)	Bitmap	Result
`0`	`0x0`	`0000000`	-
`1`	`0x1`	`0000001`	`Administrator`
`74`	`0x4A`	`1001010`	`Manage_Users View_Data Manage_Volumes`
`2`	`0x2`	`0000010`	`Manage_Users`
`96`	`0x60`	`1100000`	`Delete_Data Manage_Volumes`

`classification`

classification()➡String

Returns the display classification of the feed for the data being processed.

Return Value:

Type	Description
String	The feed’s display classification, if available.

`col-from`

col-from()➡String

Returns the input column where the current record begins. The column can be 0.

Return Value:

Type	Description
String	The starting input column, if available.

`col-to`

col-to()➡String

Returns the input column where the current record ends.

Return Value:

Type	Description
String	The ending input column, if available.

`dictionary`

dictionary(name)➡String

Returns the contents of a Dictionary document for use during translation. The dictionary may be identified by name or UUID.

The main use for this function is to allow users to abstract the management of a set of keywords from the XSLT so that it is easier for some users to make quick alterations to a dictionary that is used by some XSLT, without the need for the user to understand the complexities of XSLT.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`name`	String		-	-		-	The name or UUID of the Dictionary document.

Return Value:

Type	Description
String	The dictionary contents, if the document is found.

`feed-attribute`

Is Alias of: meta

feed-attribute(key)➡String

Looks up a metadata value for the current stream part.

The key can be Feed, StreamType, CreatedTime, EffectiveTime, Pipeline or any other attribute supplied when the stream was sent to Stroom, e.g. meta('System').

feed-attribute() is a deprecated alias.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The metadata key (case-insensitive), such as `Feed`, `StreamType` or a supplied attribute.

Return Value:

Type	Description
String	The value for the key, or an empty sequence if the key is not found.

`feed-name`

feed-name()➡String

Returns the name of the feed for the data currently being processed.

Return Value:

Type	Description
String	The current feed name, if available.

`get`

get(key)➡String

Reads a value stored under a key by put() during the current pipeline process.

Values are stored against a key name so that multiple values can be stored. These functions can be used for many purposes but are most commonly used to count a number of records that meet certain criteria.

The map is in the scope of the current pipeline process so values do not live after the stream has been processed. Also, the map will only contain entries that were put() within the current pipeline process.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The key used when storing the value.

Return Value:

Type	Description
String	The stored value, or an empty sequence if the key is absent.

Example

An example of how to count records is shown below:

<!-- Get the current record count -->
<xsl:variable name="currentCount" select="number(stroom:get('count'))" />

<!-- Increment the record count -->
<xsl:variable name="count">
  <xsl:choose>
    <xsl:when test="$currentCount">
      <xsl:value-of select="$currentCount + 1" />
    </xsl:when>
    <xsl:otherwise>
      <xsl:value-of select="1" />
    </xsl:otherwise>
  </xsl:choose>
</xsl:variable>

<!-- Store the count for future retrieval -->
<xsl:value-of select="stroom:put('count', $count)" />

<!-- Output the new count -->
<data name="Count">
  <xsl:attribute name="Value" select="$count" />
</data>

`line-from`

line-from()➡String

Returns the input line where the current record begins, counting from 1.

Return Value:

Type	Description
String	The starting input line, if available.

`line-to`

line-to()➡String

Returns the input line where the current record ends.

Return Value:

Type	Description
String	The ending input line, if available.

`log`

log(severity, message)➡Empty Sequence

The log() function writes a message to the processing log with the specified severity. Severities of INFO, WARN, ERROR and FATAL can be used. Severities of ERROR and FATAL will result in records being omitted from the output if a RecordOutputFilter is used in the pipeline.

The counts for RecWarn, RecError will be affected by warnings or errors generated in this way therefore this function is useful for adding business rules to XML output.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`severity`	String		-	`INFO`, `WARN`, `ERROR`, `FATAL`		-	The message severity, such as `INFO`, `WARN`, `ERROR` or `FATAL`.
`message`	String		-	-		-	The message to output to the log.

Return Value:

Type	Description
Empty Sequence	An empty sequence.

Example

Heading

This call will warn if a SID is not the correct length.

<xsl:if test="string-length($sid) != 7">
  <xsl:value-of select="stroom:log('WARN', concat($sid, ' is not the correct length'))"/>
</xsl:if>

The same functionality can also be achieved using the standard xsl:message element, see <xsl:message>.

`lookup`

lookup(map, key, [time], [ignoreWarnings], [trace])➡Sequence

Looks up a key in reference or context data and return the matching string or XML. A map path separated by / performs chained lookups. Numeric keys can also match key ranges in the reference data.

If the look up fails no result will be returned.

Function Arguments:

Param	Type	Default Value	Allowed Values	Min Count	Description
`map`	String	-	-	-	The reference data map name, or `/`-separated map path.
`key`	String	-	-	-	The lookup key.
`time`	String	-	-	-	Set the time to use when determining which reference data stream to use based on its effective time. which set of reference data was effective at the requested time. The reference data stream with the latest effective time that is before this value will be used. If no reference data exists with an effective time before the requested time then the lookup will fail. Time is in the format `yyyy-MM-dd'T'HH:mm:ss.SSSXX`, e.g. `2010-01-01T00:00:00.000Z`.
`ignoreWarnings`	Boolean	-	-	-	Suppress warnings for failed lookups when `true`.
`trace`	Boolean	-	-	-	Output additional trace information as `INFO` messages when `true`.

Return Value:

Type	Description
Sequence	The matching value or XML nodes, or an empty sequence if no value is found.

Example

By testing the result, a default value may be output if no result is returned.

The following example uses lookup to convert a staff number into an employee ID, assuming the STAFF_NO_TO_EMP_ID reference data map has been loaded with the staff number to employee ID mappings.

<xsl:variable name="staffNo" select="StaffNumber"/>
<xsl:if test="$staffNo">
   <xsl:variable name="sid" select="stroom:lookup('STAFF_NO_TO_EMP_ID', $staffNo, $formattedDateTime)"/>

   <xsl:choose>
      <xsl:when test="$empId">
         <User>
             <Id><xsl:value-of select="$empId"/></Id>
         </User>
      </xsl:when>
      <xsl:otherwise>
         <data name="StaffNumber">
            <xsl:attribute name="Value"><xsl:value-of select="$staffNo"/></xsl:attribute>
         </data>
      </xsl:otherwise>
   </xsl:choose>
</xsl:if>

Range Lookups

Reference data entries can either be stored with single string key or a key range that defines a numeric range, e.g. 1-100. When a lookup is preformed the passed key is looked up as if it were a normal string key. If that lookup fails Stroom will try to convert the key to an integer (long) value. If it can be converted to an integer than a second lookup will be performed against entries with key ranges to see if there is a key range that includes the requested key.

Range lookups can be used for looking up an IP address where the reference data values are associated with ranges of IP addresses. In this use case, the IP address must first be converted into a numeric value using numeric-ip(), e.g.:

<xsl:value-of select="stroom:lookup('IP_TO_LOCATION', numeric-ip($ipAddress))"/>

Similarly the reference data must be stored with key ranges whose bounds were created using this function.

Nested Maps

The lookup function allows you to perform chained lookups using nested maps. For example you may have a reference data map called USER_ID_TO_LOCATION that maps user IDs to some location information for that user and a map called USER_ID_TO_MANAGER that maps user IDs to the user ID of their manager. If you wanted to decorate a user’s event with the location of their manager you could use a nested map to achieve the lookup chain. To perform the lookup set the map argument to the list of maps in the lookup chain, separated by a /, e.g. USER_ID_TO_MANAGER/USER_ID_TO_LOCATION.

This will perform a lookup against the first map in the list using the requested key. If a value is found the value will be used as the key in a lookup against the next map. The value from each map lookup is used as the key in the next map all the way down the chain. The value from the last lookup is then returned as the result of the lookup() call. If no value is found at any point in the chain then that results in no value being returned from the function.

In order to use nested map lookups each intermediate map must contain simple string values. The last map in the chain can either contain string values or XML fragment values.

`manifest`

manifest()➡Sequence

Returns the manifest attributes of the current stream as an XML document in the stroom-meta namespace. The document has a manifest root element and a string element for each attribute, with its name in the key attribute. The attributes are ordered by name.

If the stream cannot be read, an empty manifest element is returned.

Return Value:

Type	Description
Sequence	An XML document containing the stream’s manifest attributes.

`manifest-for-id`

manifest-for-id(streamId)➡Sequence

Returns the manifest attributes of the specified stream as an XML document in the stroom-meta namespace. The document has a manifest root element and a string element for each attribute, with its name in the key attribute. The attributes are ordered by name.

If the stream cannot be read, an empty manifest element is returned.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`streamId`	String		-	-		-	The ID of the stream whose manifest is required.

Return Value:

Type	Description
Sequence	An XML document containing the stream’s manifest attributes.

`meta`

Aliases:

feed-attribute

meta(key)➡String

Looks up a metadata value for the current stream part.

The key can be Feed, StreamType, CreatedTime, EffectiveTime, Pipeline or any other attribute supplied when the stream was sent to Stroom, e.g. meta('System').

feed-attribute() is a deprecated alias.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The metadata key (case-insensitive), such as `Feed`, `StreamType` or a supplied attribute.

Return Value:

Type	Description
String	The value for the key, or an empty sequence if the key is not found.

Note

This is not the same as meta-attribute().

meta-attribute() - reads the manifest of the whole stream from the data store, so it will open the stream's source.
meta() - reads the meta data of the part currently being processed, and is what most translations want.

`meta-attribute`

meta-attribute(key)➡String

Reads one attribute from the current stream’s manifest. This opens the manifest from the stream’s source, unlike meta() which reads metadata for the current part.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The manifest attribute key to read.

Return Value:

Type	Description
String	The manifest attribute value, or an empty sequence if the key is not found.

Note

This is not the same as meta().

meta() - reads the meta data of the part currently being processed, and is what most translations want.
meta-attribute() - reads the manifest of the whole stream from the data store, so it will open the stream's source.

`meta-keys`

meta-keys()➡Sequence

Returns the metadata keys for the current stream part as an array of strings.

Return Value:

Type	Description
Sequence	An array containing the available metadata keys.

Warning

When calling this function and assigning the result to a variable, you must specify the variable data type of xs:string* (array of strings).

Example

The following fragment is an example of using meta-keys() to emit all meta values for a given stream, into an Event/Meta element:

<Event>
  <xsl:variable name="metaKeys" select="stroom:meta-keys()" as="xs:string*" />
  <Meta>
    <xsl:for-each select="$metaKeys">
      <string key="{.}"><xsl:value-of select="stroom:meta(.)" /></string>
    </xsl:for-each>
  </Meta>
</Event>

`meta-stream`

meta-stream()➡Sequence

Returns metadata for the current stream and part as an XML document in the stroom-meta namespace. The document has a meta-stream root element and a string element for each metadata entry, with its name in the key attribute. The entries are ordered by name.

If the stream part cannot be read, an empty meta-stream element is returned.

Return Value:

Type	Description
Sequence	An XML document containing the stream part’s metadata.

`meta-stream-for-id`

meta-stream-for-id(streamId, partNo)➡Sequence

Returns metadata for the specified streamId and partNo as an XML document in the stroom-meta namespace. The document has a meta-stream root element and a string element for each metadata entry, with its name in the key attribute. The entries are ordered by name.

If the stream part cannot be read, an empty meta-stream element is returned.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`streamId`	Integer		-	-		-	The ID of the stream containing the part.
`partNo`	Integer		-	-		-	The part number within the stream, starting at `1`.

Return Value:

Type	Description
Sequence	An XML document containing the stream part’s metadata.

`parent-for-id`

parent-for-id(streamId)➡String

Returns the parent stream ID of the specified input stream.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`streamId`	String		-	-		-	The ID of the input stream.

Return Value:

Type	Description
String	The parent stream ID, if one exists.

`parent-id`

parent-id()➡String

Returns the parent stream ID of the current input stream.

Return Value:

Type	Description
String	The parent stream ID, if one exists.

`part-no`

part-no()➡String

Returns the current part number within a multipart input stream, counting from 1.

Return Value:

Type	Description
String	The current part number.

`pipeline-name`

pipeline-name()➡String

Returns the name of the pipeline currently processing the stream.

Return Value:

Type	Description
String	The pipeline name, if available.

`put`

put(key, value)➡Empty Sequence

Stores a string under a key for retrieval by get() during the current pipeline process.

Function Arguments:

Param	Type	Mandatory	Default Value	Allowed Values	Multiple	Min Count	Description
`key`	String		-	-		-	The key under which to store the value.
`value`	String		-	-		-	The value to store.

Return Value:

Type	Description
Empty Sequence	An empty sequence.

`record-no`

record-no()➡String

Returns the current record number within the current stream part, counting from 1.

Return Value:

Type	Description
String	The current record number, if available.

`search-id`

search-id()➡String

Returns the batch search ID when a pipeline is processing as part of a batch search.

Return Value:

Type	Description
String	The batch search ID, if available.

`source`

source()➡Sequence

Returns an XML document in the stroom-meta namespace describing the current source location.

Return Value:

Type	Description
Sequence	The XML source location, or an empty sequence if no location is available.

`source-id`

Aliases:

stream-id

source-id()➡String

Returns the ID of the current input stream. stream-id() is an alias retained for compatibility.

Return Value:

Type	Description
String	The current input stream ID, if available.

`stream-id`

Is Alias of: source-id

stream-id()➡String

Returns the ID of the current input stream. stream-id() is an alias retained for compatibility.

Return Value:

Type	Description
String	The current input stream ID, if available.

Last modified October 20, 2026: Merge pull request #144 from gchq/add-7.14-rel-notes (2c6fb22)

Stroom Pipeline Functions

add-meta

bitmap-lookup

Examples

classification

col-from

col-to

dictionary

feed-attribute

feed-name

get

See Also

Example

line-from

line-to

log

Example

lookup

Example

Range Lookups

Nested Maps

manifest

manifest-for-id

meta

Note

meta-attribute

Note

meta-keys

Warning

Example

meta-stream

meta-stream-for-id

parent-for-id

parent-id

part-no

pipeline-name

put

See Also

record-no

search-id

source

source-id

stream-id

`add-meta`

`bitmap-lookup`

`classification`

`col-from`

`col-to`

`dictionary`

`feed-attribute`

`feed-name`

`get`

`line-from`

`line-to`

`log`

`lookup`

`manifest`

`manifest-for-id`

`meta`

`meta-attribute`

`meta-keys`

`meta-stream`

`meta-stream-for-id`

`parent-for-id`

`parent-id`

`part-no`

`pipeline-name`

`put`

`record-no`

`search-id`

`source`

`source-id`

`stream-id`