Analytic Rules

• 1: Rule Queries
• 2: Rule Settings
• 3: Rule Execution
• 4: Rule Notifications
• 5: Detections
• 6: Duplicate Management
• 7: Streaming Rules
• 8: Table Builder Rules

1 - Rule Queries

The Query tab holds the StroomQl query that defines the rule. It is the same query editor used by the Query document, so you can write a query, run it and inspect the results while you are developing the rule.

Every row the query returns becomes one detection. This is the single most important thing to understand about a rule. A query that returns a million rows will raise a million detections, so it is worth running the query and looking at the row count before enabling the rule.

Choosing a Data Source

A rule query can use any data source that a Query document can use, but the choice interacts with the processing type.

• A Scheduled Query rule can query anything, including Searchables and Lucene indexes.
• A Streaming rule must query a View , because the View’s filter is what decides which streams are fed through the rule.

Time Ranges and Relative Times

A rule that is going to run repeatedly should express its time range relative to when it runs, not as absolute dates.

When a scheduled rule runs, Stroom sets the reference time for the query to the rule’s effective execution time rather than to the current clock time. Every relative date expression in the query and in the time range is then resolved against that effective time.

This is what makes it possible for a rule to work through history. A rule whose query says where EventTime >= day() - 1d will, when it runs for an effective time of last Tuesday, select last Monday’s data. The same query run for an effective time of today selects yesterday’s data.

Columns

The columns your query selects become the content of each detection.

Every column is added to the detection as a name/value pair, using the column’s name as the key, and is available to email templates through the values dictionary. Two column names are treated specially.

Including StreamId and EventId in the query is what allows a detection to link back to the Event that caused it. Without them the detection still carries all its other values, but there is nothing to click through to.

Table Preferences

Some things about a result table cannot be expressed in StroomQl , such as which columns are hidden, how values are formatted and how rows are sorted. These are set by interacting with the results table in the UI and are stored on the rule alongside the query.

For an Analytic Rule these preferences affect only what you see while developing the query. For a Report they also affect the file that is produced, because the report is written from the same table.

Parameters

A rule can define parameters in the same way as a Query document, and refer to them in the query. Parameters are stored on the rule, so a scheduled rule always runs with the values held against it.

2 - Rule Settings

The Settings tab holds the settings that describe the rule itself, rather than its query, its schedule or the notifications it sends.

Level and Status

Neither is mandatory. A rule that sets neither produces detections without them, which is how every rule created before these settings existed behaves.

Both are written to every detection the rule produces, whatever its processing type, and both are available to notification email templates as {{ level }} and {{ status }}.

Level describes how urgent a detection from this rule is, so that whoever receives it can prioritise.

Status describes how much the rule can be trusted, which tells an analyst how much weight to give a detection and how many false positives to expect.

Include Rule Documentation

The content of the rule’s Documentation tab is written to each detection as detailedDescription, which means it leaves Stroom with the detection, for example by email. Untick this where the documentation is for whoever maintains the rule rather than for whoever receives its detections.

Feed for Errors

Reports

A Report has a Settings tab of its own, holding Feed For Errors along with the settings that decide what kind of file it produces.

A report has no Level, Status or Include Rule Documentation, because it delivers a file rather than detections.

3 - Rule Execution

The Execution tab sets the rule’s processing type, and for a Scheduled Query rule it is where you define when the rule runs.

A scheduled rule does not have a single schedule field. Instead it has a list of execution schedules, each of which runs the rule independently. The tab is split, with the schedules in the upper pane and the history of what those schedules have done in the lower one.

Everything on this page applies equally to Reports , which use the same execution machinery.

Execution Schedules

Use the button above the schedule list to add a schedule.

Having more than one schedule on a rule is useful when you want the same query run over different periods. A common pattern is a frequent schedule that keeps up with new data, plus a second schedule with a start and end time that is used once to work back over history.

Run as User

The query runs with the permissions of the Run As User, not those of whoever created the rule. If that user cannot see some of the data, the rule will silently detect nothing in it.

How Time Works

Each run of a schedule has two different times associated with it, and telling them apart is the key to understanding scheduled rules.

For a rule that is keeping up with new data the two are almost the same. They diverge whenever a rule is catching up, because a rule that is behind will run now for an effective time in the past.

The effective time of the first run comes from the schedule’s Start Time, or from the next occurrence of the schedule if no start time is set. After each successful run Stroom records where it has got to, and the next run picks up from there.

Contiguous Execution

Whether a schedule catches up or skips ahead is decided by whether it is contiguous.

• A contiguous schedule takes its next effective time from the last effective time. If the rule has not run for a week, it will work forward through that week one run at a time until it catches up. No window of data is missed.
• A non-contiguous schedule takes its next effective time from the current time. If the rule has not run for a week, that week is skipped.

Contiguous execution is what you want for a rule that must not miss anything. Be aware that it means a rule which has been disabled for a long time, or whose start time is set well in the past, will have a lot of catching up to do when it is enabled.

Replaying an Execution

Selecting a row in the execution history and clicking Replay Execution sets up a re-run of that one execution.

Replaying does not run anything immediately. It adds a new execution schedule, named after the original with (replay) appended, whose start and end times are both the effective execution time of the row you selected. The new schedule is created disabled, so nothing happens until you enable it.

This is deliberate, as it gives you the chance to check the bounds before the rule runs again. Once enabled, the schedule runs the rule once for that effective time and then has nothing further to do.

Replay works on a single history row at a time. To re-run a range of executions, add an ordinary schedule with the start and end times you want instead.

Execution History

The lower pane lists what each schedule has done.

History is kept for the period set by the stroom.analytics.executionHistoryRetention property and is deleted by the Analytic Execution History Retention job.

The Execution Schedule Manager

The rule’s Execution tab only shows that rule’s schedules.

The list identifies each schedule by the document it belongs to as well as by its own name, and can be filtered.

Run Schedules Now and Batch Edit Schedules can be applied either to the schedules you have selected or to everything matching the current filter.

Error Feed

Errors raised while a rule is running are written as Error streams to the rule’s error feed.

The feed is set by Feed For Errors on the Settings tab rather than on this one. Where the rule does not name one, the feed configured in stroom.ui.analyticUiDefaultConfig.defaultErrorFeed is used, and a run for which neither is set fails.

A run can fail before there is anywhere to write to, for example when neither the rule nor the system configuration names an error feed. Failures of that kind are recorded against the execution history instead, which is why it is worth looking at both.

4 - Rule Notifications

The Notifications tab decides what happens to the detections a rule raises. Settings that describe the rule as a whole, such as what its detections contain, are on the Settings tab instead.

The tab is a list of notifications. A rule can have any number of notifications and each one is delivered independently. A rule that both writes detections to a feed for later analysis and emails a few people about them has two notifications, not one.

Everything on this page applies to Reports as well, except that a report delivers a file rather than a detection. The differences are covered in Report Delivery.

Stream Destinations

A Stream notification writes detections into a Feed as XML conforming to the detection:1 XML Schema . This is the right choice when detections are themselves data, to be indexed, searched, or fed into another rule.

Use Source Feed If Possible is useful for a streaming rule that runs across many feeds and whose detections are best kept alongside the data that produced them.

Email Destinations

An Email notification sends each detection to people as an email.

Each field accepts more than one address.

Templates

The subject and body are Jinja templates rather than fixed text, so that each email can describe the detection that triggered it.

The values from the detection are put into the template context before the template is rendered, so {{ detectTime }} renders when the detection happened and values.SourceIp renders the value of the query’s SourceIp column.

If the rendered body looks like HTML it is sent as an HTML email, otherwise it is sent as plain text.

Testing

Two buttons let you check a template without waiting for a real detection.

• Test Template renders the template against an example detection and shows you the result, including any errors in the template.
• Send Test Email sends that example detection to the configured recipients, which also proves the SMTP configuration works.

Limiting Notifications

A rule that suddenly matches far more data than expected can send a very large number of emails. The limit settings exist to bound that.

Once Maximum Notifications have been sent the notification switches itself off and nothing further is delivered through it. It comes back on when Resume Notifications After has elapsed since the last notification was sent, at which point the count starts again from zero.

The count is held per notification and is not persisted, so restarting the node resets it.

5 - Detections

A detection is what an Analytic Rule produces. One is raised for each row the rule’s query returns, and it is the detection, rather than the row, that is delivered to the rule’s notifications.

Detections written to a Feed are XML in the detection:1 namespace, so they can be indexed and searched like any other data in Stroom. Detections sent by email are rendered through a template that has access to the same fields.

How a Detection is Built

Most of a detection is filled in for you from the rule and from the run that raised it.

Because detailedDescription comes from the rule’s documentation, what you write there is delivered with the detections the rule raises. It is worth writing it for whoever receives the detection rather than for whoever maintains the rule.

Whether it is included is controlled by Include Rule Documentation on the Settings tab, which applies to every processing type. Where it is unticked, detailedDescription is left empty.

Values

Every column the query selects becomes an entry in values, keyed by the column name. In a feed these appear as <value name="..."> elements, and in an email template they are reachable through the values dictionary.

The two exceptions are StreamId and EventId. Rather than appearing in values, they are used to build a linked event, which records the Stream and Event the detection relates to. A detection whose query selects neither will have no linked events.

Fields a Rule Does Not Set

The detection:1 schema has several fields that an Analytic Rule never populates, because they only make sense for a detector that revises its own findings over time.

These exist because the same schema is used by detections produced outside Stroom, for example by an XSLT in a translation pipeline or by an external processing framework, which can set them.

Detections as a Data Source

Writing detections to a feed means they can be treated as data in their own right. Once indexed they can be searched from a Dashboard or Query, reported on, and used as the input to further rules.

A rule whose query runs over a feed of detections is a common way of spotting a pattern across detections that no single detection shows on its own.

6 - Duplicate Management

Some rules will detect the same thing over and over. A rule that runs hourly and looks for a misconfigured device will find that device every hour until someone fixes it, and nobody needs telling twenty four times a day.

Duplicate Management suppresses detections that match ones the rule has already raised. The tab is only shown for Scheduled Query rules.

This is different from limiting notifications, which bounds how many notifications a rule may send regardless of what they say. Duplicate management suppresses repeats while still delivering anything new.

Settings

The two switches are independent, and the store is used if either of them is set.

Setting Remember Notifications on its own is a useful way to see what a rule would suppress before committing to suppressing it.

The lower part of the tab lists what the rule has stored, which is how you check why something was or was not suppressed. It also lets you clear the store.

Choosing Columns

Where Choose Columns is not set, the columns used depend on whether the query groups its results.

• A query with grouped columns uses those grouped columns, and ignores the rest.
• A query with no grouping uses every column.

Using every column is often too strict, because a column that changes on every run will make every detection unique. A rule whose query selects a timestamp, or a count, or an event id, will never suppress anything unless the columns are narrowed down to the ones that identify the thing being detected.

Choose Columns is how you narrow them. For the misconfigured device example, choosing just the device name means the rule tells you about that device once rather than hourly, however much the other columns change.

Column names are case sensitive and must match the names in the query.

The Duplicate Check Store

Each rule has its own store, held on the processing node and configured by the stroom.analytics.duplicateCheckStore property.

The store is keyed on the rule, so it survives the rule being edited. It is deleted when the rule is deleted.

7 - Streaming Rules

A Streaming rule is run by a Processor filter rather than by a schedule. Each Stream that matches the filter is passed through the rule as it is processed, so a detection is raised without waiting for the next scheduled run.

Use this type where the delay imposed by a schedule is unacceptable. For most rules a Scheduled Query is the better choice, because it can look across a window of data rather than at one stream at a time.

Data Source

A streaming rule must query a View , and that View must specify an extraction pipeline.

The View is what makes streaming possible. Its filter determines which streams are selected for processing, and its extraction pipeline determines the fields the rule’s query can use.

A rule whose query names anything other than a View, or whose View has no pipeline, fails when it is processed rather than when it is saved. The error is reported against the rule, so a streaming rule that raises nothing at all is worth checking here first.

Processing

The Execution tab of a streaming rule shows the rule’s processors and filters, in the same form as the Processors tab of a pipeline.

When a filter is created for the rule, its expression is defaulted from the filter on the View the query uses. You can then narrow it further, for example to a subset of feeds or a range of stream creation times.

Because streaming rules use the normal stream processing machinery, they behave like any other processing task.

• They are picked up by the standard stream processor jobs, not by the Analytic Executor jobs.
• Their progress can be tracked, and they can be reprocessed, from the Processors screen.
• Their priority competes with other processing on the node.

Differences from a Scheduled Rule

A streaming rule has no execution schedules, and consequently no execution history, no effective execution time and no catch up behaviour. Relative date expressions in the query resolve against the time of processing.

It also has no Duplicate Management tab, so repeated detections must be controlled by limiting notifications instead.

Streaming rules are the only type that can use Use Source Feed If Possible on a stream notification, which writes each detection to the feed the source data came from rather than to the destination feed, see Stream Destinations.

8 - Table Builder Rules

A Table Builder rule builds and maintains its own table of data from the streams it processes, and raises detections from that table. The table is held in shards on the processing node, separately from the rest of Stroom’s data.

The rule’s Shards tab, which is only shown for this processing type, lists those shards and lets you look at what they contain.

Settings