Preview Features (experimental)

Pathways

Trace logs describe the operation of applications and user interactions with applications. Pathways is designed to analyse trace logs over a period of time and remember patterns of activity. The intent is for Pathways to identify unexpected behaviour or changes to services once regular behaviour has been learnt.

Once trace logs have been added to a Plan B store it can be analysed with Pathways. Pathways examines trace logs and identifies unique paths in trace logs between methods or service calls, e.g. A -> B -> C. It also records alternate paths, e.g. A -> B -> D. Each path is remembered by Pathways and logged to an output stream. Whenever a new unique path is found the fact is logged so that it is easy to identify changes.

Pathways also records and monitors changes to span attributes within traces.

Pathways makes no judgement about the changes it logs, it is up to users to add analytic rules to fire against pathway logs.

Data Receipt Rules

Data Receipt Rules is a new feature that serves as an alternative to the existing feed status checking performed by Stroom Proxy and Stroom. It provides a much richer mechanism for controlling which received data streams are Received, Rejected or Dropped. It allows anyone with the Manage Data Receipt Rules Application Permission to create one or more rules to controls the receipt of data.

Data Receipt Rules can be accessed as follows:

Each rule is defined by a boolean expression (as used in Dashboards and Stream filtering) and the Action (Receive, Reject, Drop_ that will be performed if the data matches the rule. Rules are evaluated in ascending order by Rule Number. The action is taken from the first rule to match.

If no rules match then the data will be rejected by default, i.e. the rules are include rather than exclude filters. If you want data to be received if no rules match then you can create a rule at the end of the list with an Action of Receive and no expression terms.

If a stream matches a rule that has an Accept action, it will still be subject to a check to see if the Feed actually exists. This means that the rules do not need to contain an Accept rule to cover all of the Feeds in the system. They only need to cover The client will receive a 101 Feed is not defined error if it does not exist.

The screen operates in a similar way to Data Retention Rules in that rules can be moved up/down to change their importance, or enabled/disabled.

Fields

The fields available to use in the expression terms can be defined in the Fields tab. The terms will be evaluated against the stream’s meta data, i.e. a combination of the HTTP headers sent by the client and any that have been populated by Stroom Proxy or Stroom. This allows for the use of custom headers to aid in the filtering of data into Stroom.

Dictionaries are supported for use with the in dictionary condition. The contents of the dictionary and any of the dictionaries that it inherits will be included in the data fetched by Stroom Proxy.

Stroom Configuration

Data Receipt Rules are controlled by the following configuration:

appConfig:
  receiptPolicy:
    # List of fields whose values will be obfuscated when the rules
    # are fetched by Stroom Proxy
    obfuscatedFields:
    - "AccountId"
    - "AccountName"
    - "Component"
      # ... truncated
    - "UploadUserId"
    - "UploadUsername"
    - "X-Forwarded-For"
    # The hash algorithm used to hash obfuscated values, one of:
    # * SHA3_256
    # * SHA2_256
    # * BCRYPT
    # * ARGON_2
    # * SHA2_512
    obfuscationHashAlgorithm: "SHA2_512"
    # The initial list of fields to bootstrap a Stroom environment.
    # Changing this has no effect one an environment has been started up.
    receiptRulesInitialFields:
      AccountId: "Text"
      Component: "Text"
      Compression: "Text"
      content-length: "Text"
      # ... truncated
      Type: "Text"
      UploadUsername: "Text"
      UploadUserId: "Text"
      user-agent: "Text"
      X-Forwarded-For: "Text"
  receive:
    # The action to take if there is a problem with the data receipt rules, e.g.
    # Stroom Proxy has been unable to contact Stroom to fetch the rules.
    fallbackReceiveAction: "RECEIVE"
    # The data receipt checking mode, one of:
    # * FEED_STATUS - Use the legacy Feed Status Check method
    # * RECEIPT_POLICY - Use the new Data Receipt Rules
    # * RECEIVE_ALL - Receive ALL data with no checks
    # * DROP_ALL - Drop ALL data with no checks
    # * REJECT_ALL - Reject ALL data with no checks
    receiptCheckMode: "RECEIPT_POLICY"

Stroom Proxy Configuration

appConfig:
  receiptPolicy:
    # Only set this if you need to supply a non-standard full url
    # By default Proxy will used the known path for the Data Receipt Rules resource
    # combined with the host/port/scheme from the `downstreamHost` config property.
    receiveDataRulesUrl: null
    # The frequency that the rules will be fetched from the downstream Stroom instance.
    syncFrequency: "PT1M"

  # Identical configuration to Stroom as described above.
  # Stroom and Stroom Proxy can use different `receiptCheckMode` values, but typically
  # they will be the same.
  receiptPolicy:

Stroom Proxy Rule Synchronisation

If Stroom Proxy is configured with receiptCheckMode set to RECEIPT_POLICY and has downstreamHost configured, then it will periodically send a request to Stroom to fetch the latest copy of the Data Receipt Rules. If Stroom Proxy is unable to contact Stroom it will use the latest copy of the rules that it has.

Given that Stroom Proxy will only synchronise periodically, once a change is made to the rule set, there will be a delay before the new rules take effect.

Term Value Obfuscation

As a Stroom administrator you may not want the values used in the Data Receipt Rule expression terms to be visible when they are fetched by a remote Stroom Proxy (that may be maintained by another team). It is therefore possible to obfuscate the values used for the expression terms for certain configured fields. The fields that are obfuscated are controlled by the property stroom.receiptPolicy.obfuscatedFields.

For example, in the default configuration, Feed is an obfuscated field. Thus a term like Feed != FEED_XYZ would have its value obfuscated when fetched by Stroom Proxy. Stroom Proxy is able to similarly obfuscated meta data values for obfuscated fields in the same way to allow it to test the rule expression.

This prevents the Stroom Proxy administrator from being able to see the values used in the rules as they are not in plain text. Each value is salted with its own unique salt then hashed. The hash algorithm can be configured using stroom.receiptPolicy.obfuscationHashAlgorithm.