Templating
Overview
Templating is the process of creating a reusable format or layout for presenting data in a consistent manner. Templating is currently used in Stroom for creating email templates for Analytic Rule
detections.Stroom’s templating uses a sub-set of the template syntax called jinja and specifically the JinJava library. The templating syntax includes support for variables, filters, condition blocks, loops, etc. Full details of the syntax can be found here .
When a template is rendered, Stroom will populate the template context with data that can be used by the template.
Basic Templating Syntax
Jinja templating is very powerful and has a rich language so this is quick guide to the very basic features. See the full syntax here here .
Data Types
The following data types are available in the Jinja language.
Data type | What does it represent | Example values |
---|---|---|
None | Represents no or lack of value. | none , None |
Integer | Whole numbers. | 42 , 12 , 3134 |
Boolean | Boolean value, i.e. true/false. | true , True , false , False |
Float | Real numbers (with decimal separator). | 12.34 , 423.52341 , 3.1415 , 15.0 |
String | Any string of characters. | "dog" , 'Cat' |
List | List/array of items of any type. Can be modified after being assigned. | [1, 2, 3] , ["Apple", "Orange"] |
Tuple | Like a list but cannot be modified. | (1, 2, 3) , ("Apple", "Orange") |
Dictionary | Object containing key/value pairs, also known as a map. | { "fruit": "Apple", "weight": 320 } |
Accessing Collection Items
A List/tuple item can be accessed by its index (zero based), e.g. fruits[0]
returns Apple
.
A value in a dictionary can be accessed using its key, e.g. myDict['fruit']
returns Apple
.
If the key does not contain special characters (with the exception of _
, then you can also used this form myDict.fruit
to get the same value.
Conditions
Valid conditions are:
==
Compares two objects for equality!=
Compares two objects for inequality>
true if the left-hand side is greater than the right-hand side>=
true if the left-hand side is greater or equal to the right-hand side<
true if the left-hand side is lower than the right-hand side<=
true if the left-hand side is lower or equal to the right-hand side
Logical Operators
The logic operators for use with boolean values are:
and
Boolean andor
Boolean ornot
Boolean negation
Expressions
Syntax: {{ ... }}
Expressions will render the value of a variable that has been defined in the template context or in a previous statement. An example of a simple value expression is
The detection time is {{detectionTime}}.
The detection time is 2024-05-03T09:15:13.454Z.
An expression can also contain variables that are passed through one or more Filters.
Statements
Syntax: {% ... %}
Statements are used to create conditional blocks, loops, define variables, etc.
Setting Variables
Syntax: {% set <variable name> = <value or expression> %}
{% set colour = '#F7761F' %}
Colour: {{colour}}
{% set fruits = ["Apple", "Orange"] %}
Top Fruit: {{ fruits | first }}
{% set fullName = "% %" | format(firstName, surname) %}
Name: {{fullName}}
Colour: #F7761F
Top Fruit: Apple
Name: Joe Bloggs
Conditional Blocks
Syntax:
{% if <value, variable or expression> <condition> <value, variable or expression> %}
< optional content, expressions or statements>
{% elif <value, variable or expression> <condition> <value, variable or expression>%}
< optional content, expressions or statements>
{% else %}
< optional content, expressions or statements>
{% endif %}
Conditional blocks can be used to optional render content depending on the value of a variable. See conditions for the list of valid conditions.
{% if (values | length) > 0 -%}
This detection has {{ values | length }} values.
{%- else -%}
This detection has no values.
{%- endif -%}
This detection has 10 values.
Loops
Syntax:
{% for <item name> in <variable or expression> %}
<content, expressions or statements to repeat for each item>
{% endif %}
For loops allow you to loop over items in a list/tuple or entries in a dictionary.
{% for key, val in values | dictsort %}
{{ key }}: {{ val }}
{% endfor %}
fruit: Apple
weight: 320
Note
Note, the filterdictsort
is used here to sort the dictionary by its keys.
Note
Note the use of -
to prevent an additional line break appearing in the rendered output, see White Space below.
Filters
Syntax: ... | <filter name>
Filters are essentially functions that take an input and return an output. Some functions have additional parameters to control the behaviour of the function.
Filters can be chained together with the output of one filter becoming the input to the next filter in the chain in the same way that Stroom pipeline elements work.
Some useful filters are:
Filter | Description | Example |
---|---|---|
length |
The number of items in the list/sequence/tuple/string/dictionary | {{ "hello" | length }} => 5 |
escape |
Escapes any HTML special characters | <p>{{ "10 > 3" | escape }}</p> => <p>10 > 3</p> |
default |
Return the first argument if the input is undefined or empty | {{ None | default("foo", true) }} => foo |
For a full list of filters see here or here .
Comments
Syntax: {# <comment text> #}
Non-rendered Comments can be added to the template.
{#- Show the time -#}
The detection time is {{detectionTime}}.
The detection time is 2024-05-03T09:15:13.454Z.
Note
Note the use of -
to prevent an additional line break appearing in the rendered output, see White Space below.
White Space
When JinJava renders the template, each expression or statement is evaluated and then removed or replaced by it’s output, but any white space around them, e.g. line breaks remain. This can result in unwanted line breaks in the output.
To avoid unwanted white space you can add the -
character to the opening and/or closing tag to strip leading/trailing whitespace outside the block, e.g.
{{ ... }}
=>{{- ... -}}
{% ... %}
=>{%- ... -%}
{{ firstName -}}
{{ surname }}
JoeBloggs
For a detailed guide to how white space works see here .
Template Context
The context is a data structure that contains the dynamic content to use when rendering the template. The variables and values in the context are set by Stroom depending on where the template is being used.
Rule Detections Context
When an email subject/body template is rendered for an Analytic Rule
detection, field values from the detection are placed in the context which allows them to be used in the template.For example {{ detectTime }}
will render the date/time the detection happened.
The fields available in the context are those taken from the detection:1
XMLSchema with some additional fields.
The template has access to the following fields from the detection:
Field Name | Type | Description |
---|---|---|
detectTime | String | When the detection occurred. |
detectorName | String | Recommended detector detail - name of detector. This should be unique within the system. Some detectors are very specific and only ever create one kind of detection, and in these cases it is likely that the name of the detector will be very similar to the name of the detection headline that it creates. Example: detectorSourceName= Bad Thing Detector headline=Bad Thing Detected .However, it is possible to create detectors that are sufficiently configurable that they can create many kinds of detection. In these cases, this field should always be assigned to the same literal regardless of the detection/headline. Example: detectorSourceName= Really Configurable Detector headline=Good Thing Detected , and detectorSourceName=Really Configurable Detector headline=Bad Thing Detected .For detectors that run within Stroom pipelines, the name of the XSLT can be specified here. |
detectorUuid | String | This is the UUID of the Analytic Rule document. |
detectorVersion | String | Recommended detector detail - version of detector. This is the version of the detector identified in detectorSourceName field. Different versions might produce different detections. For detectors that run within Stroom pipelines, the version of the XSLT can be specified here. Example: v1.0 . |
detectorEnvironment | String | Recommended detector detail - where the detector was deployed. For analytics that run within Stroom itself, the name of the processing pipeline can be used. Note: the XSLT function stroom: pipeline-name() can be used within Stroom XSLT processing to determine pipeline name.Other analytics might run within an external processing framework, such as Apache Spark.Example: DevOps Spark Cluster |
headline | String | |
detailedDescription | String | Recommended detection detail. A more detailed description of what was detected than provided by headline. This will normally include some information relating to what triggered the detection, such as a specific device, location, or user. In addition to descriptive text that will be the same for all detections of this kind, there are typically several possible variable dimensions that could be used to populate parts of the string that is assigned to this field. Normally, only one such dimension is selected, based on the likely triage process (the kind of analysis that takes place, and principal area of interest of the analysts). It should be possible to group by this field value to collect all detections that relate to the thing that the analysts are most interested in during triage. Example: Charitable Donation By 'Freya Bloggs' Detected or Charitable Donation To 'Happy Cats Cattery' Detected depending on anticipated area of analyst interest(perhaps philanthropic activities of individuals or financial transactions to organisations, respectively).For some detections, this field will have the same value as that for headline as no further information is available. |
fullDescription | String | Recommended detection detail. Complete description of what was detected. This will normally include some detail relating to what triggered the detection. All dimensions with ordinal (literal) values that are useful for triage are output. Numeric and other continuous values such as timestamps are not included in this full description, in order that grouping by this single field will work effectively. Example: Charitable Donation By 'Freya Bloggs' to 'Happy Cats Cattery' Detected .For some detections, this field will have the same value as that for detailedDescription as no further information is available. |
detectionUniqueId | String | This field does not need to be assigned. Any assignment should be to a value that is sufficiently unique to identify a specific detection from a specific detector. Typically, but not necessarily, the value of this field is a UUID . It can be useful to assign this field in order to support analytic development / debugging. It is necessary to assign this field if detectionRevision field is assigned a value. |
detectionRevision | Integer | Can be used, in conjunction with detectionUniqueId to allow detectors that run continuously, in a streaming fashion to revise their detections in the light of new information. For example, it might be useful to revise the same detection with additional linked events and a new standard deviation. Where more than one detection has the same detectionUniqueId value, then the one with the highest detectionRevision will be the current one and all previous revisions (lower numbers in detectionRevision field) are superseded / ignored. |
defunct | Boolean | This field allows a detection to declare that all previous revisions (same detectionUniqueId, lower detectionRevision numbers) are now considered invalid. For example, new data might arrive later than expected and invalidate a detection that has already been sent into Stroom. Default value is false . |
executionSchedule | String | The name of the schedule that fired this detection, if the detection was fired by a Scheduled Query. |
executionTime | String | This is the actual wall clock time that the rule ran. |
effectiveExecutionTime | String | This is the time used in any relative date expressions relative data expressions name in the rule’s query or time range, e.g. day() - 1w . The effective time is relevant when executing historic time ranges in a scheduled query. |
values | Dictionary | This a dictionary with all the field/column names from the Query (with the exception of StreamId and EventId ) as keys and their respective cell values as the value. |
linkedEvents | List of DetectionLinkedEvent | This is a list of the event(s) that are linked to this detection. |
DetectionLinkedEvent fields:
Field Name | Type | Description |
---|---|---|
stroom | String | The Stroom instance within which this event exists, assumed to be this instance of Stroom if not supplied. |
streamId | String | The ID of the Stream that contains the associated event. |
eventId | String | The ID of the Event that is associated with this detection. |
Warning
When choosing the names of the columns in your rule it may be beneficial to use snake_case
or UpperCamelCase
to make it easier to reference those columns in the detection template (see Accessing Collection Items above). E.g. myDict.some_key
vs myDict['some key']
.