Stroom – document

Docs: Documents

Wed, 21 Feb 2024 00:00:00 +0000

All Documents in Stroom share some common elements:

UUID UUID A Universally Unique Identifier for uniquely identifying something. UUIDs are used as the identifier in Doc Refs. An example of a UUID is 4ffeb895-53c9-40d6-bf33-3ef025401ad3.Click to see more details... - Uniquely identifies the document within Stroom and when exported into another stroom.
Type - This is the type as used in the Doc Ref Doc Ref A Doc Ref (or Document Reference) is an identifier used to identify most documents/entities in Stroom, e.g. an XSLT will have a Doc Ref.Click to see more details....
Documentation - Every Document has a Documentation tab for recording any documentation that relates to the Document, see Documenting Content.

Some Documents are very simple with just text content and documentation, e.g. XSLT. Others are much more complex, e.g. Pipeline, with various different tabs to manage the content of the Document.

The following is a list of all Document types in Stroom.

Configuration

Documents that are used as configuration for other documents.

Dictionary

Icon:
Type: Dictionary

A Dictionary is essentially a list of ‘words’, where each ‘word’ is separated by a new line. Dictionaries can be used in filter expressions, i.e. IN DICTIONARY. They allow for the reuse of the same set of values across many search expressions. Dictionaries also support inheritance so one dictionary can import the contents of other dictionaries.

Documentation

Icon:
Type: Documentation

A Document type for simply storing user created documentation, e.g. adding a Documentation document into a folder to describe the contents of that folder.

Elastic Cluster

Icon:
Type: ElasticCluster

Defines the connection details for a single Elasticsearch cluster. This Elastic Cluster Document can then be used by one or more Elastic Index Documents.

Git Repo

Icon:
Type: GitRepo

Contains the configuration for a connection to a Git repository.

Kafka Configuration

Icon:
Type: KafkaConfig

Defines the connection details for a single Kafka cluster. This Kafka Configuration Document can then be used by one or more StandardKafkaProducer pipeline elements.

OpenAI Model

Icon:
Type: OpenAIModel

Defines the settings required to connect to an OpenAI-compatible API and interact with a model.

S3 Configuration

Icon:
Type: S3Config

Defines the config for S3

Script

Icon:
Type: Script

Contains a Javascript script that is used as the source for a visualisation Document. Scripts can have dependencies on other Script Documents, e.g. to allow re-use of common code.

Scylla DB

Icon:
Type: ScyllaDB

Defines the connection details for a ScyllaDB state store instance.

Visualisation

Icon:
Type: Visualisation

Defines a data visualisation that can be used in a Dashboard Document. The Visualisation defines the settings that will be available to the user when it is embedded in a Dashboard. A Visualisation is dependent on a Script Document for the Javascript code to make it work.

Data Processing

Documents relating to the processing of data.

Feed

Icon:
Type: Feed

The Feed Feed A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.Click to see more details... is Stroom’s way of compartmentalising data that has been ingested or created by a Pipeline. Ingested data must specify the Feed that it is destined for.

The Feed Document defines the character encoding for the data in the Feed, the type of data that will be received into it (e.g. Raw Events) and optionally a Volume Group to use for data storage. The Feed Document can also control the ingest of data using its Feed Status property and be used for viewing data that belonging to that feed.

Pipeline

Icon:
Type: Pipeline

A Pipeline defines a chain of Pipeline elements that consumes from a source of data (a Stream of raw data or cooked events) then processes it according to the elements used in the chain. Pipelines can be linear or branching and support inheritance of other pipelines to allow re-use of common structural parts.

The Pipeline Document defines the structure of the pipeline and the configuration of each of the elements in that pipeline. It also defines the filter(s) that will be used to control what data is passed through the pipeline and the priority of processing. The Pipeline Document can be used to view the data produced by the pipeline and to monitor its processing state and progress.

Indexing

Documents relating to the process of adding data into an index, such as Lucene or Elasticsearch.

Elastic Index

Icon:
Type: ElasticIndex

Defines an index that exists within an Elasticsearch cluster. This Document is used in the configuration of the ElasticIndexingFilter pipeline element.

Lucene Index

Icon:
Type: Index

Lucene Index is the standard built-in index within Stroom and is one of many data sources. An index is like a catalog in a library and provides a very fast way to access documents/records/events when searching using fields that have been indexed. The index stores the field values and pointers to the document they came from (the Stream and Event IDs). Data can be indexed using multiple indexes to allow fast access in different ways.

The Lucene Index Document optionally defines the fields that will be indexed (it is possible to define the fields dynamically) and their types. It also allows for configuration of the way the data in the index will be stored, partitioned and retained.

The Lucene Index Document is used by the IndexingFilter and DynamicIndexingFilter pipeline elements.

Pathways

Icon:
Type: Pathways

TODO - Add description

Plan B

Icon:
Type: PlanB

Defines a place to store state

Solr Index

Icon:
Type: SolrIndex

Solr Index represents an index on a Solr cluster. It defines the connection details for connecting to that cluster and the structure of the index. It is used by the SolrIndexingFilter pipeline element.

State Store

Icon:
Type: StateStore

Defines a place to store state

Statistic Store

Icon:
Type: StatisticStore

Defines a logical statistic store used to hold statistical data of a particular type and aggregation window. Statistics in Stroom is a way to capture counts or values from events and record how they change over time, with the counts/values aggregated (sum/mean) across time windows.

The Statistic Store Document configures the type of the statistic (Count or Value), the tags that are used to qualify a statistic event and the size of the aggregation windows. It also supports the definition of roll-ups that allow for aggregation over all values of a tag. Tags can be things like user, node, feed, etc. and can be used to filter data when querying the statistic store in a Dashboard/Query.

It is used by the StatisticsFilter pipeline element.

Stroom-Stats Store

Icon:
Type: StroomStatsStore

The Stroom-Stats Store Document is deprecated and should not be used.

Search

Documents relating to searching for data in Stroom.

Analytic Rule

Icon:
Type: AnalyticRule

Defines an analytic rule which can be run to alert on events meeting a criteria. The criteria is defined using a StroomQL query. The analytic can be processed in different ways:

Streaming
Table Builder
Scheduled Query

Annotation

Icon:
Type: Annotation

TODO - Add description

Dashboard

Icon:
Type: Dashboard

The Dashboard Document defines a data querying and visualisation dashboard. The dashboard is highly customisable to allow querying of many different data sources of different types. Queried data can be displayed in tabular form, visualised using interactive charts/graphs or rendered as HTML.

The Dashboard Doc can either be used for ad-hoc querying/visualising of data, to construct a dashboard for others to view or to just view an already constructed dashboard. Dashboards can be parameterised so that all queries on the dashboard are displaying data for the same user, for example. For ad-hoc querying of data from one data source, you are recommended to use a Query instead.

Query

Icon:
Type: Query

A Query Document defines a StroomQl StroomQl Stroom Query Language is Stroom’s own query language. It has similarities with Structured Query Language (SQL) as used in databases. StroomQL is sometimes referred to as sQL to distinguish it from SQL.Click to see more details... query and is used to execute that query and view its results. A Query can query main types of data source including Views, Lucene Indexes, and Searchables Searchable A Searchable is the term given the special searchable data sources that appear at the root of the explorer tree picker when selecting a data source. These data sources are special internal data sources that are not user managed content, unlike an Index. They provide the means to search various aspects of Stroom’s internals, such as the Meta Store or Processor Tasks.Click to see more details....

Report

Icon:
Type: Report

Defines a report that can be run at scheduled intervals and sent to individuals via email. The criteria is defined using a StroomQL query.

View

Icon:
Type: View

A view is an abstraction over a data source (such as a Lucene Index) and optionally an extraction pipeline. Views provide a much simpler way for users to query data as the user can simply query against the View without any knowledge of the underlying data source or extraction of that data.

Transformation

Documents relating to the transformation of data.

Text Converter

Icon:
Type: TextConverter

A Text Converter Document defines the specification for splitting text data into records/fields using Data Splitter or for wrapping fragment XML with a XMLFragmentParser pipeline element. The content of the Document is either XML in the data-splitter:3 namespace or a fragment parser specification (see Pipeline Recipies).

This Document is used by the following pipeline elements:

XML Schema

Icon:
Type: XMLSchema

This Document defines an XML Schema XML Schema XML Schema is a language used to define the permitted structure of an XML document. An XML Schema can be used to validate an XML document to ensure it conforms to that schema such that onward processing of the XML document can be done with confidence that the document is correct.Click to see more details... that can be used within Stroom for validation of XML documents. The XML Schema Document content is the XMLSchema text. This Document also defines the following:

Namespace URI - The XML namespace of the XMLSchema and the XML document that the schema will validate.
System Id - An ID (that is unique in Stroom) that can be used in the xsi:schemaLocation attribute, e.g. xsi:schemaLocation="event-logging:3 file://event-logging-v3.4.2.xsd".
Schema Group - A name to group multiple versions of the same schema. The SchemaFilter can be configured to only use schemas matching a configured group.

The XML Schema Document also provides a handy interactive viewer for viewing and navigating the XMLSchema in a graphical representation.

This Document is used by the SchemaFilter pipeline element.

XSL Translation

Icon:
Type: XSLT

The content of this Document is an XSLT XSLT Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents. XSLTs are the primary means of transforming data in Stroom.Click to see more details... document for transforming data in a pipeline. This Document is used by the XSLTFilter pipeline element.

Docs: Document Permissions

Fri, 01 Nov 2024 00:00:00 +0000

TODO

The Users, Groups and Permissions screens are undergoing significant change in Stroom v7.6. Therefore this section will be updated with more detail in v7.6.

Document Permissions are permissions that are granted to Users or Groups for a specific Document Document Typically refers to an item that can be created in the Explorer Tree, e.g. a Feed, a Pipeline, a Dashboard, etc. May also be known as an Entity.Click to see more details.... They control what documents a user/group can see and what they can do to those documents. They allow very fine grained control over what a user/group can see or do in Stroom.

For example, User jbloggs may be granted Use permission on the Index named Alert Index in order for him to be able to query that index in a dashboard, but not be able to see it in the explorer tree or change it in any way.

By default a new user with no Application Permissions, Document Permissions or Group memberships cannot view/use/modify any documents. They do not even have permission to create any documents. When logging into Stroom, they will simply see an empty explorer tree.

A user can gain varying levels of access to documents in a number of ways:

Being added to a Group that has direct or inherited permissions on one or more existing documents.
Being added to a Group that has direct or inherited permissions to create one or more document types.
Being directly granted permissions one or more existing documents.
Being directly granted the permission to create one or more document types.
Being granted the Administrator Application Permission which gives them access to ALL documents.

In order to modify the permissions on a document, you must either hold Owner permission on the document or have the Administrator Application Permission. The Document Permissions screen for a document/folder can be accessed by right clicking on it in the explorer tree and selecting:

Permission Types

The following is the list of different permissions that can be granted to users/groups on a document.

Permission	Description
Owner	Same as delete plus ability to change the document’s permissions (i.e. grant permissions on this document to other users/groups).
Delete	Same as edit plus permission to delete the document.
Edit	Same as view plus permission to edit, move, rename or add tags to the document.
View	Permission to see the document in the explorer tree, open it as read-only, copy it or export it (subject to also having the `Export Configuration` application permission).
Use	Only allow use of a document, e.g. allow use of an index as part of a search process but do not allow viewing of the document itself.

The following is the list of different permissions that can be granted to users/groups on a folder .

Permission	Description
Owner	Same as delete plus ability to change the folder’s permissions (i.e. grant permissions on this folder to other users/groups).
Delete	Same as edit plus permission to delete the folder.
Edit	Same as view plus permission to edit, move, rename or add tags to the folder.
View	Permission to see the folder in the explorer tree (and its child items that you also have View permission on), open it as read-only, copy it or export it (subject to also having the `Export Configuration` application permission).
Use	Only allow use of a folder, e.g. allow use of an index as part of a search process but do not allow viewing of the folder itself.

Implied Permissions

Note that each permission in the two tables above also includes all the permissions below it in the table, e.g. a user with Edit permission on a document will also have the implied permissions View and Use. There is no need to grant these lower permissions to the user, though doing so will have no impact as Stroom will use the highest value permission when checking permissions.

Inherited Permissions

If a User jbloggs is a member of Group Team A and that group is a member of group Division 123, then jbloggs will inherit all permissions from both Team A and Division 123. A User/Group will inherit all permissions of the groups that they are a member of and also from any ancestor groups of those groups.

User/Group	Permissions	Direct/Inherited
Division 123	View on Dictionary IP Allow List	Direct
Team A	View on Dictionary IP Allow List	Inherited
Team A	Owner on Dashboard Team Dashboard	Direct
jbloggs	View on Dictionary IP Allow List	Inherited
jbloggs	Owner on Dashboard Team Dashboard	Inherited
jbloggs	View on Dashboard Frank’s Dashboard	Direct

`Owner` Permission

A document can have multiple owners. An owner can be a user or a group. When a document is created by a user they are automatically made an owner of it. Any user with the Administrator role has implied ownership of ALL documents.

Having Owner permission on a document means the user can grant permissions on that document to other users, or revoke permissions from other users.

`Use` Permission

This permission allows users to access a document but not actually see it in the explorer tree or open the document in Stroom. They can however make use of the document, e.g. selecting and querying an Index in a Dashboard .

The Use permission is not relevant to all document types.

Permissions on Folders

Folders in the explorer tree work mostly in the same way as documents when it comes to permissions. There are a couple of exceptions to this.

Permission on Folder Contents

The permissions on a folder apply only to the folder itself and has no bearing on what you can/can’t do to its child items. The permissions on each child item in the folder control what you can/can’t do to those items.

For example, if you only have View permission on a folder, but have Delete on a document in that folder, then you are able to delete that document and thus change the contents of the folder.

Similarly, if you have View permission on a folder but have no permission on any of its child items, then you will just see an empty folder.

Ancestor Folder Visibility

A folder will be visible to a user in the explorer tree if the user has View permission on it OR if the user has View permission on any single document/folder that is a descendant of it.

For example, if a user has View permission on a Dictionary Dictionary_XYZ with path

System / Folder_A / Folder_B / Dictionary_XYZ

but no permissions on Folder A or Folder B, they will be able to see both Folders in the explorer tree in addition to the Dictionary. They will however not be able to open those Folders as they do not have the permission.

Therefore, when granting permissions on a document/folder to a user/group, you are also implicitly granting visibility (but not View permission) on all ancestor folders.

Create Permissions

Folders can have one or more Create Permissions granted on them to users/groups.

There is a Create Permission for each document type, e.g. Index, Dictionary, Feed, etc. A Create Permission is the ability to create a new document of that type in that folder.

For example, user jbloggs is an analyst and is granted Create Dashboard and Create Query permissions on the Folder named Joe's Folder. This means Joe can only create Dashboard or Query documents in that folder and nothing else.

Applying Changes to Descendants

When making changes to the permissions on a Folder you have the option of making the changes to just that folder or to all descendants of that folder. Selecting to apply to all descendants will make all permission changes apply to every descendant, i.e. including any sub-folders and their contents or own sub-folders.

Moving and Copying Documents

When you move or copy a document/folder you have the choice of how the destination document/folder’s permissions should be derived. The move/copy dialog offers the following choices:

None - Removes all current permissions. Ignores permissions of the destination folder. You will be the owner of the moved document if not already.
Source - Keep the current permissions and ownership as they are.
Destination - Removes all current permissions. Adds the permissions of the destination folder. You will be the owner of the moved document if not already.
Combined - Keep the current permissions and add the permissions of the destination folder. You will be the owner of the moved document if not already.

Note

You must have Owner permission on the source document/folder (or Administrator Application permission Application permission This is a permission that is not specific to a single document. It applies to all documents or is not related to documents in any way.Click to see more details…) if you wish to use None, Destination or Combined as these all involve a change of permissions.

Stroom – document

Docs: Documents

Configuration

Dictionary

Documentation

Elastic Cluster

Git Repo

Kafka Configuration

OpenAI Model

S3 Configuration

Script

Scylla DB

Visualisation

Data Processing

Feed

Pipeline

See Also

Indexing

Elastic Index

See Also

Lucene Index

See Also

Pathways

Plan B

Solr Index

See Also

State Store

Statistic Store

Stroom-Stats Store

Search

Analytic Rule

Annotation

Dashboard

Query

Report

View

Transformation

Text Converter

XML Schema

XSL Translation

Docs: Document Permissions

TODO

Permission Types

Implied Permissions

Inherited Permissions

Owner Permission

Use Permission

Permissions on Folders

Permission on Folder Contents

Ancestor Folder Visibility

Create Permissions

Applying Changes to Descendants

Moving and Copying Documents

Note

`Owner` Permission

`Use` Permission