This is the multi-page printable view of this section. Click here to print.
P
- 1: Parser
- 2: Pipeline
- 3: Pipeline element
- 4: Processor
- 5: Processor filter
- 6: Property
1 - Parser
See Also
- Pipeline Element Reference
- Pipeline Pipeline A Pipeline is an entity that is constructed to take a single input of stream data and process/transform it with one or more outputs. A Pipeline can have many elements within it to read, process or transform the data flowing through it.Click to see more details...
- Raw Events Raw Events This is a Stream Type used for Streams received by Stroom. Streams received by Stroom will be in a variety of text formats (CSV, delimited, fixed width, XML, JSON, etc.). Until they have been processed by a pipeline they are essentially just unstructured character data with no concept of what is a record/event. A Parser in a pipeline is required to provide the demarcation between records/events.Click to see more details...
-
Records
Records
This is a Stream Type for Streams containing data conforming to the
records:2XML Schema. It also refers more generally to any XML conforming to therecords:2XML Schema which is used in a number of places in Stroom, including as the output format for the DSParser and input for the IndexingFilter.Click to see more details... - Field Field A named data Field within some form of record or entity, and where each Field can have an associated value. In Stroom, Fields can be the Fields in an Index (or other queryable Data Source or the fields of Metadata associated with a Stream, e.g. Stream ID, Feed, creation time, etc.Click to see more details...
2 - Pipeline
See Also
3 - Pipeline element
See Also
- Pipeline Element Reference
- Pipeline Pipeline A Pipeline is an entity that is constructed to take a single input of stream data and process/transform it with one or more outputs. A Pipeline can have many elements within it to read, process or transform the data flowing through it.Click to see more details...
4 - Processor
The Processor can be enabled/disabled to enable/disable the processing of data through the Pipeline. A processor will have one or more Processor Filters Processor filter A Processor Filter is used to used to find Streams to process through the Pipeline associated with the Processor Filter. A Processor Filter consists of an expression tree to select which Streams to process and a tracker to track the what Streams have been processed.Click to see more details... associated with it.
See Also
5 - Processor filter
For example a typical Processor Filter would have an Expression Tree that selected all Streams of type Raw Events in a particular Feed Feed A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.Click to see more details.... A filter could also select a single Stream by its ID, e.g. when Re-processing Re-processing The act of repeating the processing of a set of input data (Stream) that have already been processed at least once. Re-Processing can be done for an individual Stream or multiple Streams using a Processor Filter.Click to see more details... a Stream.
A Pipeline can have multiple Processor Filters. Filters can be enabled/disabled independently of their parent Processor to control processing.
See Also
- Expression tree Expression tree A tree of expression terms that each evaluate to a boolean (True/False) value. Terms can be grouped together within an expression operator (AND, OR, NOT).Click to see more details...
- Feed Feed A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.Click to see more details...
- Pipeline Pipeline A Pipeline is an entity that is constructed to take a single input of stream data and process/transform it with one or more outputs. A Pipeline can have many elements within it to read, process or transform the data flowing through it.Click to see more details...
- Re-processing Re-processing The act of repeating the processing of a set of input data (Stream) that have already been processed at least once. Re-Processing can be done for an individual Stream or multiple Streams using a Processor Filter.Click to see more details...
- Stream Stream A Stream is the unit of data that Stroom works with and will typically contain many Events.Click to see more details...
6 - Property
config.yml configuration file.