Stroom uses Pipelines to process its data.
A pipeline is a set of pipeline elements connected together.
Pipelines are very powerful and flexible and allow the user to transform, index, store and forward data in a wide variety of ways.
Example Pipeline
Pipelines can take many forms and be used for a wide variety of purposes, however a typical pipeline to convert CSV data into cooked events might look like this:
Pipelines process data in batches.
This batch of data is referred to as a StreamStream
A Stream is the unit of data that Stroom works with and will typically contain many Events.
Click to see more details....
The input for the pipeline is a single Stream that exists within a Feed and this data is fed into the left-hand side of the pipeline at
Source
.
Pipelines can accept streams from multiple Feeds assuming those feeds contain similar data.
The data in the Stream is always text data (XML, JSON, CSV, fixed-width, etc.) in a known Character encodingCharacter encoding
Character Encoding is the means of encoding character data (i.e. text) into binary form. Therefore to decode character data from a stream of bytes, the character encoding must be known (or guessed).
Click to see more details....
Stroom does not currently support processing binary formats.
XML
The working format for pipeline processing is XMLXML
Extensible Markup Language is a markup language for storing/transmitting structured data. It is the working format for most Pipeline processing in Stroom and is the standard normalised format for event data.
Click to see more details... (with the exception of raw streaming).
Data can be input and output in other forms, e.g. JSON, CSV, fixed-width, etc. but the majority of pipelines do most of their processing in XML.
Input data is converted into XML SAX
events, processed using XSLT to transform it into different shapes of XML then either consumed as XML (e.g. an
IndexingFilter
) or converted into a desired output format for storage/forwarding.
Forks
Pipelines can also be forked at any point in the pipeline.
This allows the same data to be processed in different ways.
Note
Rather than creating complicated pipelines with forks, it is sometimes better to create multiple pipelines as this makes it easier to handle errors in one fork of the processing.
It also makes it easier to re-use common simple pipelines.
For example if you have a pipeline to transform CSV events into normalised XML then index it and forward it to a remote server, it may be better to have a pipeline to cook the events, then a common one to index those XML events and one to forward XML events.
Pipeline Inheritance
It is possible for pipelines to inherit from other pipelines.
This allows for the creation of a standard abstract pipelines with a set structure, though not fully configured, to be inherited by many concrete pipelines.
For example you may have a standard pipeline for indexing XML events, i.e. read XML data and pass it to an
IndexingFilter
, but the IndexingFilter is not configured with the actual Index to send documents to.
A pipeline that inherits this one can then be simply configured with the Index to use.
Pipeline inheritance allows for changes to the inherited structure, e.g. adding additional elements in line.
Multi level inheritance is also supported.
Pipeline Element Types
Stroom has a number of categories of pipeline element.
Reader
Readers are responsible for reading the raw bytes of the input data and converting it to character data using the Feed’s character encoding.
They also provide functionality to modify the data before or after it is decoded to characters, e.g. Byte order markByte order mark
A special Unicode character at the start of a text stream that indicates the byte order (or endianness) of the stream.
Click to see more details... removal, or doing find/replace on the character data.
You can chain multiple Readers.
Parser
A parser is designed to convert the character data into XML for processing.
For example, the
JSONParser
will use a JSON parser to read the character data as JSON and convert it into XML elements and attributes that represent the JSON structure, so that it can be transformed downstream using XSLT.
Parsers have a built in reader so if they are not preceded by a Reader they will decode the raw bytes into character data before parsing.
Filter
A filter is an element that handles XML SAX events (e.g. element, attribute, character data, etc.) and can either return those events unchanged or modify them.
An example of Filter is an
XSLTFilter
element.
Multiple filters can be chained, with each one consuming the events output by the one preceding it, therefore you can have lots of common reusable XSLTFilters that all do small incremental changes to a document.
Writer
A writer is an element that handles XML SAX events (e.g. element, attribute, character data, etc.) and converts them into encoded character data (using a specified encoding) of some form.
The preceding filter may have been an
XSLTFilter
which transformed XML into plain text, in which case only character data events will be output and a
TextWriter
can just write these out as text data.
Other writers will handle the XML SAX events to convert them into another format, e.g. the
JSONWriter
before encoding them as character data.
Destination
A destination element is a consumer of character data, as produced by a writer.
A typical destination is a
StreamAppender
that writes the character data (which may be XML, JSON, CSV, etc.) to a new Stream in Stroom’s stream store.
Other destinations can be used for sending the encoded character data to Kafka, a file on a file system or forwarding to an HTTP URL.
1 - Pipeline Recipes
A set of basic pipeline structure recipes for common use cases.
The following are a basic set of pipeline recipes for doing typical tasks in Stroom.
It is not an exhaustive list, as the possibilities with Pipelines are vast.
They are intended as a rough guide to get you started with building Pipelines.
Comma Separated Values is a file format with typically one record per line and fields delimited by a ,. Fields may be optionally enclosed with double quotes, though there is no fixed standard for CSV data, particularly when it comes to escaping of double quotes and/or commas.
Click to see more details... data is ingested.
The Data Splitter parser parses the records and fields into records format XML using an XML based TextConverter document.
The first XSLTFilter is used to normalise the events in records XML into event-logging XML.
The second XSLTFilter is used to decorate the events with additional data, e.g. <UserDetails> using reference data lookups.
The SchemaFilter ensures that the XML output by the stages of XSLT transformation conforms to the event-logging XMLSchema.
The XML events are then written out as an Event Stream to the Stream store.
The same as ingesting CSV data above, except the input JSON is converted into an XML representation of the JSON by the JSONParser.
The Normalise XSLTFilter will be specific to the format of the JSON being ingested.
The Decorate XSLTFilter will likely be identical to that used for the CSV ingest above, demonstrating reuse of pipeline element content.
As above except that the input data is already XML, though not in event-logging format.
The XMLParser simply reads the XML character data and converts it to XML SAX events for processing.
The Normalise XSLTFilter will be specific to the format of this XML and will transform it into event-logging format.
XML Fragments are where the input data looks like:
<Event>
...
</Event>
<Event>
...
</Event>
In other words, it is technically badly formed XML as it has no root element or declaration.
This format is however easier for client systems to send as they can send multiple <Event> blocks in one stream (e.g. just appending them together in a rolled log file) but don’t need to wrap them with an outer <Events> element.
The XMLFragmentParser understands this format and will add the wrapping element to make well-formed XML.
If the XML fragments are already in event-logging format then no Normalise XSLTFilter is required.
In some cases client systems may send XML containing characters that are not supported by the XML standard.
These can be removed using the
InvalidXMLCharFilterReader
.
The input data may also be known to contain other sets of characters that will cause problems in processing.
The
FindReplaceFilter
can be used to remove/replace either a fixed string or a Regex pattern.
In cases where you want to export the raw (or cooked) data from a feed you can have a very simple pipeline to pipe the source data directly to an appender.
This may be so that the raw data can be ingested into another system for analysis.
In this case the data is being written to disk using a file appender.
Be careful when specifying the directory structure for the FileAppender so that you don’t end up with too many files in one folder, which can cause some OS issues.
Indexing
XML to Stroom Lucene Index
This use case is for indexing XML event data that had already been normalised using one of the ingest pipelines above.
The
XSLTFilter
is used to transform the event into records format, extracting the fields to be indexed from the event.
The
IndexingFilter
reads the records XML and loads each one into Stroom’s internal Lucene index .
Dynamic indexing in Stroom allows you to use the XSLT to define the fields that are being indexed and how each field should be indexed.
This avoids having to define all the fields up front in the Index and allows for the creation of fields based on the actual data received.
The only difference with normal indexing in Stroom is that is uses the
DynamicIndexingFilter
and rather than transforming the event into records:2 XML, it is transformed into index-documents:1 XML as shown in the example below.
This use case is for indexing XML event data that had already been normalised using one of the ingest pipelines above.
The
XSLTFilter
is used to transform the event into records format, extracting the fields to be indexed from the event.
The
ElasticIndexingFilter
reads the records XML and loads each one into an external Elasticsearch index .
Search extraction is the process of combining the data held in the index with data obtained from the original indexed document, i.e. the event.
Search extraction is useful when you do not want to store the whole of an event in the index (to reduce storage used) but still want to be able to access all the event data in a Dashboard/View.
An extraction pipeline is required to combine data in this way.
Search extraction pipelines are referenced in Dashboard and View settings.
Standard Lucene Index Extraction
This is a non-dynamic search extraction pipeline for a Lucene index.
XSLTFilter
- An XSLT transforming event-logging:3 => index-documents:1.
Data Egress
XML to CSV File
A recipe for writing normalised XML events (as produced by an ingest pipeline above) to a file, but in a flat file format like CSV.
The
XSLTFilter
transforms the events XML into CSV data with XSLT including this:
The
TextWriter
converts the XML character events into a stream of characters encoded using the desired output character encoding.
The data is appended to a file on a file system, with one file per Stream.
This is similar to the above recipe for writing out CSV, except that the
XSLTFilter
converts the event XML into XML conforming to the https://www.w3.org/2013/XSL/json/
XMLSchema.
The
JSONWriter
can read this format of XML and convert it into JSON using the desired character encoding.
The
RollingFileAppender
will append the encoded JSON character data to a file on the file system that is rolled based on a size/time threshold.
This recipe is for sending normalised XML events to another system over HTTP.
The
HTTPAppender
is configured with the URL and any TLSTransport Sayer Security (TLS)
Transport Sayer Security (TLS) is the evolution of Secure Sockets Layer (SSL) and refers to the encryption of traffic between client and server.
Click to see more details... certificates/keys/credentials.
A typical pipeline for loading XML reference data (conforming to the reference-data:2 XMLSchema) into the reference data store.
The
ReferenceDataFilter
reads the reference-data:2 format data and loads each entry into the appropriate map in the store.
As an example, the reference-data:2 XML for mapping userIDs to staff numbers looks something like this:
This recipe takes normalised XML data and converts it into statistic events (conforming to the statistics:4 XMLSchema).
Stroom’s Statistic Stores are a way to store aggregated counts or averaged values over time periods.
For example you may want counts of certain types of event, aggregated over fixed time buckets.
Each XML event is transformed using the
XSLTFilter
to either return no output or a statistic event.
An example of statistics:4 data for two statistic events is:
The following capabilities are available to parse input data:
XML - XML input can be parsed with the XML parser.
XML Fragment - Treat input data as an XML fragment, i.e. XML that does not have an XML declaration or root elements.
Data Splitter - Delimiter and regular expression based language for turning non XML data into XML (e.g. CSV)
2.1 - XML Fragments
Handling XML data without root level elements.
Some input XML data may be missing an XML declaration and root level enclosing elements.
This data is not a valid XML document and must be treated as an XML fragment.
To use XML fragments the input type for a translation must be set to ‘XML Fragment’.
A fragment wrapper must be defined in the XML conversion that tells Stroom what declaration and root elements to place around the XML fragment data.
Here is an example:
<?xml version="1.1" encoding="UTF-8"?>
<!DOCTYPE records [
<!ENTITY fragment SYSTEM "fragment">
]>
<records
xmlns="records:2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="records:2 file://records-v2.0.xsd"
version="2.0">
&fragment;
</records>
During conversion Stroom replaces the fragment text entity with the input XML fragment data.
Note that XML fragments must still be well formed so that they can be parsed correctly.
3 - XSLT Conversion
Using Extensible Stylesheet Language Transformations (XSLT) to transform data.
Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents. XSLTs are the primary means of transforming data in Stroom.
Click to see more details... is a language that is typically used for transforming XML documents into either a different XML document or plain text.
XSLT is key part of Stroom’s pipeline processing as it is used to normalise bespoke events into a common XML audit event document conforming to the event-loggingXML SchemaXML Schema
XML Schema is a language used to define the permitted structure of an XML document. An XML Schema can be used to validate an XML document to ensure it conforms to that schema such that onward processing of the XML document can be done with confidence that the document is correct.
Click to see more details....
Once a text file has been converted into intermediary XML (or the feed is already XML), XSLTXSLT
Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents. XSLTs are the primary means of transforming data in Stroom.
Click to see more details... is used to
translate the XML into the event-logging XML format.
The
XSLTFilter
pipeline element defines the XSLT document and is used to do the transformation of the input XML into XML or plain text.
You can have multiple XSLTFilter elements in a pipeline if you want to break the transformation into steps, or wish to have simpler XSLTs that can be reused.
Raw Event Feeds are typically translated into the event-logging:3 schema and Raw Reference into the reference-data:2 schema.
3.1 - XSLT Basics
The basics of using XSLT and the XSLTFilter element.
Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents. XSLTs are the primary means of transforming data in Stroom.
Click to see more details... is a very powerful language and allows the user to perform very complex transformations of XML data.
This documentation does not aim to document how to write XSLT documents, for that, we strongly recommend you refer to online references (e.g. W3Schools
or obtain a book covering XSLT 2.0 and XPathXPath
XPath is an expression language for selecting a node or nodes in an XML document. It is used heavily in XSLT to define the match criteria for templates and to select values.
Click to see more details...).
It does however aim to document aspects of XSLT that are specific to the use of XSLT in Stroom.
Examples
Event Normalisation
Here is an example XSLT document that transforms XML data in the records:2NamespaceNamespace
In Stroom Namespace typically refers to an XML Namespace. Namespaces are used in XML to distinguish different elements, e.g. where an XSLT is transforming XML in the records:2Namespace into XML in the event-logging:3Namespace.
Click to see more details... (which is the output of the
DSParser
element) into event XML in the event-logging:3 namespace.
It is an example of event normalisation from a bespoke format.
Warning
This example aims to show some typical uses of XSLT in a typical Stroom use case.
It does not necessarily represent best practice in terms of creation of a normalised event.
In Stroom Namespace typically refers to an XML Namespace. Namespaces are used in XML to distinguish different elements, e.g. where an XSLT is transforming XML in the records:2Namespace into XML in the event-logging:3Namespace.
Click to see more details... (which is the output of the
DSParser
element) into XML in the reference-data:2 namespace that is suitable for loading using the
ReferenceDataFilter
If you want an XSLT to decorate an Events XML document with some additional data or to change it slightly without changing its namespace then a good starting point is the identity transformation.
<xsl:stylesheet
version="1.0"
xpath-default-namespace="event-logging:3"
xmlns="event-logging:3"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!-- Match Root Object -->
<xsl:template match="Events">
<Events
xmlns="event-logging:3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="event-logging:3 file://event-logging-v3.4.2.xsd"
Version="3.4.2">
<xsl:apply-templates />
</Events>
</xsl:template>
<!-- Whenever you match any node or any attribute -->
<xsl:template match="node( )|@*">
<!-- Copy the current node -->
<xsl:copy>
<!-- Including any attributes it has and any child nodes -->
<xsl:apply-templates select="@*|node( )" />
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
This XSLT will copy every node and attribute as they are, returning the input document completely un-changed.
You can then add additional templates to match on specific elements and modify them, for example decorating a user’s UserDetails elements with value obtained from a reference data lookup on a user ID.
Note
You can insert this identity skeleton into an XSLT editor using this editor snippet.
The log() function writes a message to the processing log with the specified severity.
Severities of INFO, WARN, ERROR and FATAL can be used.
Severities of ERROR and FATAL will result in records being omitted from the output if a
RecordOutputFilter is used in the pipeline.
The counts for RecWarn, RecError will be affected by warnings or errors generated in
this way therefore this function is useful for adding business rules to XML output.
Click to see more details...) XSLT function.
The element text is logged to the Error stream with a default severity of ERROR.
A child element can optionally be used to set the severity level (one of FATAL|ERROR|WARN|INFO).
The namespace of this element does not matter.
You can also set the attribute terminate="yes" to log the message at severity FATAL and halt processing of that stream part.
If the stream is multi-part then processing will continue with the next part.
Note
Setting terminate="yes" will trump any severity defined by a child element.
It will always be logged at FATAL.
The following are some examples of using <xsl:message>.
<!-- Log a message using default severity of ERROR -->
<xsl:message>Invalid length</xsl:message>
<!-- terminate="yes" means log the message as a FATAL ERROR and halt processing of the stream part -->
<xsl:message terminate="yes">Invalid length</xsl:message>
<!-- Log a message with a child element name specifying the severity. -->
<xsl:message>
<warn>Invalid length</warn>
</xsl:message>
<!-- Log a message with a child element name specifying the severity. -->
<xsl:message>
<info>Invalid length</info>
</xsl:message>
<!-- Log a message, specifying the severity and using a dynamic value. -->
<xsl:message>
<info>
<xsl:value-of select="concat('User ID ', $userId, ' is invalid')" />
</info>
</xsl:message>
3.2 - XSLT Functions
Custom XSLT functions available in Stroom.
Stroom provides a set of custom functions for use in your XSLT translations.
See the XSLT Functions reference section for details of the available functions.
3.3 - XSLT Includes
Using an XSLT import to include XSLT from another translation.
You can use an XSLT import to include XSLT from another translation.
E.g.:
<xsl:import href="ApacheAccessCommon" />
This would include the XSLT from the ApacheAccessCommon translation.
4 - File Output
Substitution variables for use in output file names and paths.
When outputting files with Stroom, the output file names and paths can include various substitution variables to form the file and path names.
Context Variables
The following replacement variables are specific to the current processing context.
${feed} - The name of the feed that the stream being processed belongs to
${pipeline} - The name of the pipeline that is producing output
${sourceId} - The id of the input data being processed
${partNo} - The part number of the input data being processed where data is in aggregated batches
${searchId} - The id of the batch search being performed.
This is only available during a batch search
${node} - The name of the node producing the output
Time Variables
The following replacement variables can be used to include aspects of the current time in UTC.
${year} - Year in 4 digit form, e.g. 2000
${month} - Month of the year padded to 2 digits
${day} - Day of the month padded to 2 digits
${hour} - Hour padded to 2 digits using 24 hour clock, e.g. 22
${minute} - Minute padded to 2 digits
${second} - Second padded to 2 digits
${millis} - Milliseconds padded to 3 digits
${ms} - Milliseconds since the epoch
System (Environment) Variables
System variables (environment variables) can also be used, e.g. ${TMP}.
File Name References
rolledFileName in RollingFileAppender can use references to the fileName to incorporate parts of the non rolled file name.
${fileName} - The complete file name
${fileStem} - Part of the file name before the file extension, i.e. everything before the last ‘.’
${fileExtension} - The extension part of the file name, i.e. everything after the last ‘.’
Other Variables
${uuid} - A randomly generated UUID to guarantee unique file names
5 - Reference Data
Performing temporal reference data lookups to decorate event data.
In Stroom reference data is primarily used to decorate events using stroom:lookup() calls in XSLTs.
For example you may have reference data feed that associates the FQDN of a device to the details of its physical location, or reference data that associates usernames with employee details such as name and job title.
You can then perform a stroom:lookup() in the XSLT to decorate an event with the physical location of a device by looking up the FQDN found in the event.
Reference Data Sources
Reference data can come from two different sources:
Reference stream - A separate Feed containing Raw Reference streams that have been translated into Reference streams conforming to reference-data:2 XML.
Context sub-stream - A Context stream that is a sub-stream of the Events stream.
This sub-stream contains contextual data relevant to the events in the Events stream.
Reference Stream
Reference data is often time sensitive, e.g. an employee’s job title will change over time, so each stream of reference data has an Effective Date set against it.
This allows reference data lookups to be performed using the date of the event to ensure the reference data that was actually effective at the time of the event is used.
Warning
There is currently no mechanism in Stroom to place dependencies between Feeds, i.e. an Events Feed being dependent on a Reference Feed.
When performing a lookup Stroom can only use the Reference streams that it has available.
If a Reference stream has been delayed (in receipt or translation), then Stroom will have to use the most recent one it has.
Using Reference stream based reference data involves the following steps/processes:
A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.
Click to see more details... in Stroom.
Creating a pipeline (and associated processors/filters) to transform the Raw Reference into a Reference stream containing reference-data:2 format XML.
Creating a reference loader pipeline with a
ReferenceDataFilter
element to load cooked reference data into the reference data store.
This pipeline does not need any processors or processor filters as it is used on demand when a lookup is performed and the required reference stream is not in the store.
Adding reference pipeline/feeds to an XSLT Filter in your event pipeline.
The Pipeline is set to the reference loader pipeline created above.
The Feed is set to the Feed containing the Reference streams.
The Stream Type is set to Reference.
Adding the lookup call to the XSLT.
Processing the raw events through the event pipeline.
The following is an example of a reference loader pipeline:
The process of creating a reference data pipeline is described in the HOWTO linked at the top of this document.
Note
It is possible to not eagerly convert the Raw Reference streams into Reference streams and instead do this as part of the reference loader pipeline.
While this saves having to store all the cooked Reference streams, it means every node that uses a Raw Reference stream for a lookup will have to do the transformation before it can go into the reference data store.
This can result in a lot of duplicated work on a multi-node Stroom.
Context Sub-Stream
Some Events streams have a Context sub-stream associated with them.
Context streams allow the system sending the events to Stroom to supply an additional stream of data that provides context to the Raw Events stream.
This can be useful when the system sending the events has no control over the event content but needs to supply additional information.
The Context stream can be used in lookups as a reference source to decorate events on receipt.
Context reference data is specific to a single event stream so is transient in nature, therefore the On Heap Store is used to hold it for the duration of the event stream processing only.
As soon as the Events stream has been processed, the transient store will be destroyed.
As the Context sub-stream sits alongside the Events stream, there is no concept of Effective Date so no date/time argument is needed in the lookup() call.
Using Context sub-stream based reference data involves the following steps/processes:
Creating a context loader pipeline to transform the raw context data into reference-data:2 XML and pass that into a
ReferenceDataFilter
element.
This pipeline does not need any processors or processor filters as it is used on demand when the first lookup call is made in an Events stream.
Adding reference pipeline/feeds to an XSLT Filter in your event pipeline.
The Pipeline is set to the context loader pipeline created above.
The Feed is set to the Feed containing the Events streams.
The Stream Type is set to Context.
Adding the lookup call to the XSLT.
Processing the raw events through the event pipeline.
The following is an example of a context loader pipeline that takes in raw context data in a text format that is converted into reference-data:2 XML using a
DSParser
and
XSLTFilter
.
The ReferenceDataFilter requires the reference data to be XML that conforms to the reference-data:2 XMLSchema.
A reference data entry essentially consists of the following:
Effective time - The data/time that the entry was effective from, i.e. the time the raw reference data was received.
Map name - A unique name for the key/value map that the entry will be stored in.
The name only needs to be unique within all map names that may be loaded within an XSLT Filter.
In practice it makes sense to keep map names globally unique.
A reference stream can contain more than one map and the entries for a map can be spread over more than one reference feed.
A map can contain a mixture of key and range based entries.
Key - The text that will be used to lookup the value in the reference data map.
The key is case sensitive.
Mutually exclusive with Range.
Range - The inclusive range of integer keys that the entry applies to.
Mutually exclusive with Key.
Value - The value can either be simple text, e.g. an IP address, or an XML fragment that can be inserted into another XML document.
XML values must be correctly namespaced.
The following is an example of some reference data that has been converted from its raw form into reference-data:2 XML.
In this example the reference data contains three entries that each belong to a different map.
Two of the entries are simple text values and the last has an XML value.
<?xml version="1.1" encoding="UTF-8"?>
<referenceData
xmlns="reference-data:2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stroom="stroom"
xmlns:evt="event-logging:3"
xsi:schemaLocation="reference-data:2 file://reference-data-v2.0.xsd"
version="2.0.1">
<!-- A simple string value -->
<reference>
<map>FQDN_TO_IP</map>
<key>stroomnode00.strmdev00.org</key>
<value>192.168.2.245</value>
</reference>
<!-- A simple string value -->
<reference>
<map>IP_TO_FQDN</map>
<key>192.168.2.245</key>
<value>stroomnode00.strmdev00.org</value>
</reference>
<!-- A key range -->
<reference>
<map>USER_ID_TO_COUNTRY_CODE</map>
<range>
<from>1</from>
<to>1000</to>
</range>
<value>GBR</value>
</reference>
<!-- An XML fragment value -->
<reference>
<map>FQDN_TO_LOC</map>
<key>stroomnode00.strmdev00.org</key>
<value>
<evt:Location>
<evt:Country>GBR</evt:Country>
<evt:Site>Bristol-S00</evt:Site>
<evt:Building>GZero</evt:Building>
<evt:Room>R00</evt:Room>
<evt:TimeZone>+00:00/+01:00</evt:TimeZone>
</evt:Location>
</value>
</reference>
</referenceData>
Reference Data Namespaces
When XML reference data values are created, as in the example XML above, the XML elements in the value must be qualified with a namespace to distinguish them from the reference-data:2 XML that surrounds them.
In the above example the XML fragment will become as follows when injected into an event:
Even if evt is already declared in the XML being injected into it, if it has been declared for the reference fragment then it will be explicitly declared in the destination.
While duplicate namespacing may appear odd it is valid XML.
The namespacing can also be achieved like this:
<?xml version="1.1" encoding="UTF-8"?>
<referenceData
xmlns="reference-data:2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stroom="stroom"
xsi:schemaLocation="reference-data:2 file://reference-data-v2.0.xsd"
version="2.0.1">
<!-- An XML value -->
<reference>
<map>FQDN_TO_LOC</map>
<key>stroomnode00.strmdev00.org</key>
<value>
<Location xmlns="event-logging:3">
<Country>GBR</Country>
<Site>Bristol-S00</Site>
<Building>GZero</Building>
<Room>R00</Room>
<TimeZone>+00:00/+01:00</TimeZone>
</Location>
</value>
</reference>
</referenceData>
This reference data will be injected into event XML exactly as is, i.e.:
Reference data is stored in two different places on a Stroom node.
All reference data is only visible to the node where it is located.
Each node that is performing reference data lookups will need to load and store its own reference data.
While this will result in duplicate data being held by nodes it makes the storage of reference data and its subsequent lookup very performant.
On-Heap Store
The On-Heap store is the reference data store that is held in memory in the Java Heap.
This store is volatile and will be lost on shut down of the node.
The On-Heap store is only used for storage of context data which is destroyed once processing of the Events stream is complete.
Off-Heap Store
The Off-Heap store is the reference data store that is held in memory outside of the Java Heap and is persisted to local disk.
As the store is also persisted to local disk it means the reference data will survive the shutdown of the stroom instance.
Storing the data off-heap means Stroom can run with a much smaller Java Heap size.
The Off-Heap store is based on the Lightning Memory-Mapped Database (LMDB).
LMDB makes use of the Linux page cache to ensure that hot portions of the reference data are held in the page cache (making use of all available free memory).
Infrequently used portions of the reference data will be evicted from the page cache by the Operating System.
Given that LMDB utilises the page cache for holding reference data in memory the more free memory the host has the better as there will be less shifting of pages in/out of the OS page cache.
When storing large amounts of data you may experience the OS reporting very little free memory as a large amount will be in use by the page cache.
This is not an issue as the OS will evict pages when memory is needed for other applications, e.g. the Java Heap.
Local Disk
The Off-Heap store is intended to be located on local disk on the Stroom node.
The location of the store is set using the property stroom.pipeline.referenceData.localDir.
Using LMDB on remote storage is NOT advised, see http://www.lmdb.tech/doc.
Using the fastest storage (i.g. fast SSDs) is advised to reduce load times and lookups of data that is not in memory.
Warning
If you are running stroom on AWS EC2 instances then you will need to attach some local instance storage to the host, e.g. SSD, to use for the reference data store.
In tests EBS storage was found to be VERY slow.
It should be noted that AWS instance storage is not persistent between instance stops, terminations and hardware failure.
However any loss of the reference data store will mean that the next time Stroom boots a new store will be created and reference data will be loaded on demand as normal.
Transactions
LMDB is a transactional database with ACID semantics.
All interaction with LMDB is done within a read or write transaction.
There can only be one write transaction at a time so if there are a number of concurrent reference data loads then they will have to wait in line.
Read transactions, i.e. lookups, are not blocked by each other but may be blocked by a write transaction depending on the value of the system property stroom.pipeline.referenceData.lmdb.readerBlockedByWriter.
LMDB can operate such that readers are not blocked by writers but if there is an open read transaction while a write transaction is writing data to the store then it is unable to make use of free space (from previous deletes, see Store Size & Compaction) so will result in the store increasing in size.
If read transactions are likely while writes are taking place then this can lead to excessive growth of the store.
Setting stroom.pipeline.referenceData.lmdb.readerBlockedByWriter to true will block all reads while a load is happening so any free space can be re-used, at the cost of making all lookups wait for the load to complete.
Use of this setting will depend on how likely it is that loads will clash with lookups and the store size should be monitored.
Read-Ahead Mode
When data is read from the store, if the data is not already in the page cache then it will be read from disk and added to the page cache by the OS.
Read-ahead is the process of speculatively reading ahead to load more pages into the page cache than were requested.
This is on the basis that future requests for data may need the pages speculatively read into memory as it is more efficient to read multiple pages at once.
If the reference data store is very large or is larger than the available memory then it is recommended to turn read-ahead off as the result will be to evict hot reference data from the page cache to make room for speculative pages that may not be needed.
It can be tuned off with the system property stroom.pipeline.referenceData.readAheadEnabled.
Key Size
When reference data is created care must be taken to ensure that the Key used for each entry is less than 507 bytes.
For simple ASCII characters then this means less than 507 characters.
If non-ASCII characters are in the key then these will take up more than one byte per character so the length of the key in characters will be less.
This is a limitation inherent to LMDB.
Commit Intervals
The property stroom.pipeline.referenceData.maxPutsBeforeCommit controls the number of entries that are put into the store between each commit.
As there can be only one transaction writing to the store at a time, committing periodically allows other process to jump in and make writes.
There is a trade off though as reducing the number of entries put between each commit can seriously affect performance.
For the fastest single process performance a value of 0 should be used which means it will not commit mid-load.
This however means all other processes wanting to write to the store will need to wait.
Low values (e.g. in the hundreds) mean very frequent commits so will hamper performance.
Cloning the Off Heap Store
If you are provisioning a new stroom node it is possible to copy the off heap store from another node.
Stroom should not be running on the node being copied from.
Simply copy the contents of stroom.pipeline.referenceData.localDir into the same configured location on the other node.
The new node will use the copied store and have access to its reference data.
Store Size & Compaction
Due to the way LMDB works the store can only grow in size, it will never shrink, even if reference data is deleted.
Deleted data frees up space for new writes to the store so will be reused but will never be freed back to the operating system.
If there is a regular process of purging old data and adding new reference data then this should not be an issue as the new reference data will use the space made available by the purged data.
If store size becomes an issue then it is possible to compact the store.
lmdb-utils is package that is available on some package managers and this has an mdb_copy command that can be used with the -c switch to copy the LMDB environment to a new one, compacting it in the process.
This should be done when Stroom is down to avoid writes happening to the store while the copy is happening.
The following is an example of how to compact the store assuming Stroom has been shut down first.
# Navigate to the 'stroom.pipeline.referenceData.localDir' directory
cd /some/path/to/reference_data
# Verify contents
ls
(out) data.mdb lock.mdb
# Create a directory to write the compacted file to
mkdir compacted
# Run the compaction, writing the new data.mdb file to the new sub-dir
mdb_copy -c ./ ./compacted
# Delete the existing store
rm data.mdb lock.mdb
# Copy the compacted store back in (note a lock file gets created as needed)
mv compacted/data.mdb ./
# Remove the created directory
rmdir compacted
Now you can re-start Stroom and it will use the new compacted store, creating a lock file for it.
The compaction process is fast.
A test compaction of a 4Gb store, compacted down to 1.6Gb took about 7s on non-flash HDD storage.
Alternatively, given that the store is essentially a cache and all data can be re-loaded another option is to delete the contents of stroom.pipeline.referenceData.localDir when Stroom is not running.
On boot Stroom will create a brand new empty store and reference data will be re-loaded as required.
This approach will result in all data having to be re-loaded so will slow lookups down until it has been loaded.
The Loading Process
Reference data is loaded into the store on demand during the processing of a stroom:lookup() method call.
Reference data will only be loaded if it does not already exist in the store, however it is always loaded as a complete stream, rather than entry by entry.
The test for existence in the store is based on the following criteria:
The UUID of the reference loader pipeline.
The version of the reference loader pipeline.
The Stream ID for the stream of reference data that has been deemed effective for the lookup.
The Stream Number (in the case of multi part streams).
If a reference stream has already been loaded matching the above criteria then no additional load is required.
IMPORTANT: It should be noted that as the version of the reference data pipeline forms part of the criteria, if the reference loader pipeline is changed, for whatever reason, then this will invalidate ALL existing reference data associated with that reference loader pipeline.
Typically the reference loader pipeline is very static so this should not be an issue.
Standard practice is to convert raw reference data into reference:2 XML on receipt using a pipeline separate to the reference loader.
The reference loader is then only concerned with reading cooked reference:2 into the Reference Data Filter.
In instances where reference data streams are infrequently used it may be preferable to not convert the raw reference on receipt but instead to do it in the reference loader pipeline.
Duplicate Keys
The Reference Data Filter pipeline element has a property overrideExistingValues which if set to true means if an entry is found in an effective stream with the same key as an entry already loaded then it will overwrite the existing one.
Entries are loaded in the order they are found in the reference:2 XML document.
If set to false then the existing entry will be kept.
If warnOnDuplicateKeys is set to true then a warning will be logged for any duplicate keys, whether an overwrite happens or not.
Value De-Duplication
Only unique values are held in the store to reduce the storage footprint.
This is useful given that typically, reference data updates may be received daily and each one is a full snapshot of the whole reference data.
As a result this can mean many copies of the same value being loaded into the store.
The store will only hold the first instance of duplicate values.
Querying the Reference Data Store
The reference data store can be queried within a Dashboard in Stroom by selecting Reference Data Store in the data source selection pop-up.
Querying the store is currently an experimental feature and is mostly intended for use in fault finding.
Given the localised nature of the reference data store the dashboard can currently only query the store on the node that the user interface is being served from.
In a multi-node environment where some nodes are UI only and most are processing only, the UI nodes will have no reference data in their store.
Purging Old Reference Data
Reference data loading and purging is done at the level of a reference stream.
Whenever a reference lookup is performed the last accessed time of the reference stream in the store is checked.
If it is older than one hour then it will be updated to the current time.
This last access time is used to determine reference streams that are no longer in active use and thus can be purged.
The Stroom job Ref Data Off-heap Store Purge is used to perform the purge operation on the Off-Heap reference data store.
No purge is required for the On-Heap store as that only holds transient data.
When the purge job is run it checks the time since each reference stream was accessed against the purge cut-off age.
The purge age is configured via the property stroom.pipeline.referenceData.purgeAge.
It is advised to schedule this job for quiet times when it is unlikely to conflict with reference loading operations as they will fight for access to the single write transaction.
Lookups
Lookups are performed in XSLT Filters using the XSLT functions.
In order to perform a lookup one or more reference loaders must be specified on the XSLT Filter pipeline element.
Each reference loader feed is specified along with a reference loader pipeline that will ingest the specified feed (optionally converting it into reference:2 XML if it is not already) and pass it into a Reference Data Filter pipeline element.
The following is an example of making a lookup() call in an XSLT that will return a simple string value.
The above examples are equally applicable for a bitmap-lookup() call.
Reference Feeds & Loaders
In the XSLT Filter pipeline element multiple combinations of feed and reference loader pipeline can be specified.
There must be at least one in order to perform lookups.
If there are multiple then when a lookup is called for a given time, the effective stream for each feed/loader combination is determined.
The effective stream for each feed/loader combination will be loaded into the store, unless it is already present.
When the actual lookup is performed Stroom will try to find the key in each of the effective streams that have been loaded and that contain the map in the lookup call.
If the lookup is unsuccessful in the effective stream for the first feed/loader combination then it will try the next, and so on until it has tried all of them.
For this reason if you have multiple feed/loader combinations then order is important.
It is possible for multiple effective streams to contain the same map/key so a feed/loader combination higher up the list will trump one lower down with the same map/key.
Also if you have some lookups that may not return a value and others that should always return a value then the feed/loader for the latter should be higher up the list so it is searched first.
To perform a lookup using context data, set the stream type to Context.
Effective Streams
Reference data lookups have the concept of Effective Streams.
An effective stream is the most recent stream for a given FeedFeed
A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.
Click to see more details... that has an Effective Date that is less than or equal to the date used for the lookup() call.
If an effective time argument is not supplied, then it will default to the create time of the Raw Events stream being processed and if that is not available, the current time.
When performing a lookup, Stroom will search the stream store to find all the effective streams in a time bucket that surrounds the lookup time.
These sets of effective streams are cached so if a new reference stream is created it will not be used until the cached set has expired.
To rectify this you can clear the cache Reference Data - Effective Stream Cache on the Caches screen accessed from:
Monitoring
Caches
Standard Key/Value Lookups
Standard key/value lookups consist of a simple string key and a value that is either a simple string or an XML fragment.
Standard lookups are performed using the various forms of the stroom:lookup()stroom:lookup()
Looks up a key in reference or context data and return the matching string or XML.
A map path separated by / performs chained lookups.
Numeric keys can also match key ranges in the reference data.
If the look up fails no result will be returned.
Click to see more details... XSLT function.
Note
If the key is not found and the key is an integer then it will attempt a range lookup using the same key.
This is to allow for maps that contain a mixture of key/value pairs and range/value pairs.
Range Lookups
Range lookups consist of a key that is an integer and a value that is either a simple string or an XML fragment.
For more detail on range lookups see the XSLT function stroom:lookup()stroom:lookup()
Looks up a key in reference or context data and return the matching string or XML.
A map path separated by / performs chained lookups.
Numeric keys can also match key ranges in the reference data.
If the look up fails no result will be returned.
Click to see more details....
Note
The lookup will initially look for a single key that matches the lookup key.
If an exact match is not found then it will look for a range that contains the key.
This is to allow for maps that contain a mixture of key/value pairs and range/value pairs.
Nested Map Lookups
Nested map lookups involve chaining a number of lookups with the value of each map being used as the key for the next.
For more detail on nested lookups see the XSLT function stroom:lookup()stroom:lookup()
Looks up a key in reference or context data and return the matching string or XML.
A map path separated by / performs chained lookups.
Numeric keys can also match key ranges in the reference data.
If the look up fails no result will be returned.
Click to see more details....
Bitmap Lookups
A bitmap lookup is a special kind of lookup that actually performs a lookup for each enabled bit position of the passed bitmap value.
For more detail on bitmap lookups see the XSLT function stroom:lookup()stroom:lookup()
Looks up a key in reference or context data and return the matching string or XML.
A map path separated by / performs chained lookups.
Numeric keys can also match key ranges in the reference data.
If the look up fails no result will be returned.
Click to see more details....
Values can either be a simple string or an XML fragment.
Context Data Lookups
Some event streams have a Context stream associated with them.
Context streams allow the system sending the events to Stroom to supply an additional stream of data that provides context to the raw event stream.
This can be useful when the system sending the events has no control over the event content but needs to supply additional information.
The context stream can be used in lookups as a reference source to decorate events on receipt.
Context reference data is specific to a single event stream so is transient in nature, therefore the On Heap Store is used to hold it for the duration of the event stream processing only.
Typically the reference loader for a context stream will include a translation step to convert the raw context data into reference:2 XML.