Stroom – api

Docs: API Specification

Thu, 02 Feb 2023 00:00:00 +0000

Swagger UI

The APIs are available as a Swagger Open API specification in the following forms:

JSON - stroom.json
YAML - stroom.yaml

A dynamic Swagger user interface is also available for viewing all the API endpoints with details of parameters and data types. This can be found in two places.

Published on GitHub for each minor version Swagger user interface .
Published on a running stroom instance at the path /stroom/noauth/swagger-ui.

API Endpoints in Application Logs

The API methods are also all listed in the application logs when Stroom first boots up, e.g.

INFO  2023-01-17T11:09:30.244Z main i.d.j.DropwizardResourceConfig The following paths were found for the configured resources:

    GET     /api/account/v1/ (stroom.security.identity.account.AccountResourceImpl)
    POST    /api/account/v1/ (stroom.security.identity.account.AccountResourceImpl)
    POST    /api/account/v1/search (stroom.security.identity.account.AccountResourceImpl)
    DELETE  /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
    GET     /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
    PUT     /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
    GET     /api/activity/v1 (stroom.activity.impl.ActivityResourceImpl)
    POST    /api/activity/v1 (stroom.activity.impl.ActivityResourceImpl)
    POST    /api/activity/v1/acknowledge (stroom.activity.impl.ActivityResourceImpl)
    GET     /api/activity/v1/current (stroom.activity.impl.ActivityResourceImpl)
    ...

You will also see entries in the logs for the various servlets exposed by Stroom, e.g.

INFO  ... main s.d.common.Servlets            Adding servlets to application path/port: 
INFO  ... main s.d.common.Servlets            	stroom.core.servlet.DashboardServlet          => /stroom/dashboard 
INFO  ... main s.d.common.Servlets            	stroom.core.servlet.DynamicCSSServlet         => /stroom/dynamic.css 
INFO  ... main s.d.common.Servlets            	stroom.data.store.impl.ImportFileServlet      => /stroom/importfile.rpc 
INFO  ... main s.d.common.Servlets            	stroom.receive.common.ReceiveDataServlet      => /stroom/noauth/datafeed 
INFO  ... main s.d.common.Servlets            	stroom.receive.common.ReceiveDataServlet      => /stroom/noauth/datafeed/* 
INFO  ... main s.d.common.Servlets            	stroom.receive.common.DebugServlet            => /stroom/noauth/debug 
INFO  ... main s.d.common.Servlets            	stroom.data.store.impl.fs.EchoServlet         => /stroom/noauth/echo 
INFO  ... main s.d.common.Servlets            	stroom.receive.common.RemoteFeedServiceRPC    => /stroom/noauth/remoting/remotefeedservice.rpc 
INFO  ... main s.d.common.Servlets            	stroom.core.servlet.StatusServlet             => /stroom/noauth/status 
INFO  ... main s.d.common.Servlets            	stroom.core.servlet.SwaggerUiServlet          => /stroom/noauth/swagger-ui 
INFO  ... main s.d.common.Servlets            	stroom.resource.impl.SessionResourceStoreImpl => /stroom/resourcestore/* 
INFO  ... main s.d.common.Servlets            	stroom.dashboard.impl.script.ScriptServlet    => /stroom/script 
INFO  ... main s.d.common.Servlets            	stroom.security.impl.SessionListServlet       => /stroom/sessionList 
INFO  ... main s.d.common.Servlets            	stroom.core.servlet.StroomServlet             => /stroom/ui

Docs: Application Programming Interfaces (API)

Wed, 16 Feb 2022 00:00:00 +0000

Stroom has many public REST APIs API Application Programming Interface. An interface that one system can present so other systems can use it to communicate. Stroom has a number of APIs, e.g. its many REST APIs and its /datafeed interface for data receipt.Click to see more details... to allow other systems to interact with Stroom. Everything that can be done via the user interface can also be done using the API.

All methods on the API will are authenticated and authorised, so the permissions will be exactly the same as if the API user is using the Stroom user interface directly.

Docs: Calling an API

Tue, 17 Jan 2023 00:00:00 +0000

Authentication

In order to use the API endpoints you will need to authenticate. Authentication is achieved using an API Key API Key API Keys are a form of authentication token that are created within Stroom for use by Stroom-Proxy instances or other clients that want to use Stroom’s API. It is an encrypted string that contains details of the user and the expiration date of the token. Possession of a valid API Key for a user account means that you can do anything that the user can do in the user interface via the API.Click to see more details... or Token Token Typically refers to an authentication token that may be used for user authentication. A Stroom API Key is a form of authentication token.Click to see more details....

You will either need to create an API key for your personal Stroom user account or for a shared processing user account. Whichever user account you use it will need to have the necessary permissions for each API endpoint it is to be used with.

To create an API key (token) for a user:

In the top menu, select:

Click Create.
Enter a suitable expiration date. Short expiry periods are more secure in case the key is compromised.
Select the user account that you are creating the key for.
Click OK
Select the newly created API Key from the list of keys and double click it to open it.
Click Copy Key to copy the key to the clipboard.

To make an authenticated API call you need to provide a header of the form Authorization:Bearer ${TOKEN}, where ${TOKEN} is your API Key as copied from Stroom.

Calling an API method with `curl`

This section describes how to call an API method using the command line tool curl as an example client. Other clients can be used, e.g. using python, but these examples should provide enough help to get started using another client.

HTTP Requests Without a Body

Typically HTTP GET requests will have no body/payload Often PUT and DELETE requests will also have no body/payload.

The following is an example of how to call an HTTP GET method (i.e. a method that does not require a request body) on the API using curl.

TOKEN='API KEY GOES IN HERE' \
curl \
  --silent \
  --insecure \
  --header "Authorization:Bearer ${TOKEN}" \
  https://stroom-fqdn/api/node/v1/info/node1a

Example output:

{"discoverTime":"2022-02-16T17:28:37.710Z","buildInfo":{"buildDate":"2022-01-19T15:27:25.024677714Z","buildVersion":"7.0-beta.175","upDate":"2022-02-16T09:28:11.733Z"},"nodeName":"node1a","endpointUrl":"http://192.168.1.64:8080","itemList":[{"nodeName":"node1a","active":true,"master":true}],"ping":2}

Warning

The --insecure argument is used in this example which means certificate verification will not take place. It is recommended not to use this argument and instead supply curl with client and certificate authority certificates to make a secure connection.

TOKEN='API KEY GOES IN HERE' \
curl \
  --silent \
  --cert /path/to/client-cert \
  --key /path/to/client-key \
  --cacert /path/to/ca-cert \
  --header "Authorization:Bearer ${TOKEN}" \
  https://stroom-fqdn/api/some/path

You can either call the API via Nginx (or similar reverse proxy) at https://stroom-fqdn/api/some/path or if you are making the call from one of the stroom hosts you can go direct using http://localhost:8080/api/some/path. The former is preferred as it is more secure.

Requests With a Body

A lot of the API methods in Stroom require complex bodies/payloads for the request. The following example is an HTTP POST to perform a reference data lookup on the local host.

Create a file req.json containing:

{
  "mapName": "USER_ID_TO_STAFF_NO_MAP",
  "effectiveTime": "2024-12-02T08:37:02.772Z",
  "key": "user2",
  "referenceLoaders": [
    {
      "loaderPipeline" : {
        "name" : "Reference Loader",
        "uuid" : "da1c7351-086f-493b-866a-b42dbe990700",
        "type" : "Pipeline"
      },
      "referenceFeed" : {
        "name": "STAFF-NO-REFERENCE",
        "uuid": "350003fe-2b6c-4c57-95ed-2e6018c5b3d5",
        "type" : "Feed"
      }
    }
  ]
}

Now send the request with curl.

TOKEN='API KEY GOES IN HERE' \
curl \
  --json @req.json \
  --request POST \
  --header "Authorization:Bearer ${TOKEN}" \
  http://localhost:8080/api/refData/v1/lookup
(out)staff2

This API method returns plain text or XML depending on the reference data value.

Note

This assumes you are using curl version 7.82.0 or later that supports the --json argument. If not you will need to replace --json with --data and add these arguments:

--header "Content-Type: application/json"
--header "Accept: application/json"

Handling JSON

jq is a utility for processing JSON and is very useful when using the API methods.

For example to get just the build version from the node info endpoint:

TOKEN='API KEY GOES IN HERE' \
curl \
    --silent \
    --insecure \
    --header "Authorization:Bearer ${TOKEN}" \
    https://localhost/api/node/v1/info/node1a \
  | jq -r '.buildInfo.buildVersion'
(out)7.0-beta.175

Docs: Query APIs

Wed, 16 Feb 2022 00:00:00 +0000

The Query APIs use common request/response models and end points for querying each type of data source held in Stroom. The request/response models are defined in stroom-query .

Currently Stroom exposes a set of query endpoints for the following data source types. Each data source type will have its own endpoint due to differences in the way the data is queried and the restrictions imposed on the query terms. However they all share the same API definition.

stroom-index Queries - The Lucene based search indexes.
Sql Statistics Query - Stroom’s SQL Statistics store.
Searchable - Searchables are various data sources that allow you to search the internals of Stroom, e.g. local reference data store, annotations, processor tasks, etc.

The detailed documentation for the request/responses is contained in the Swagger definition linked to above.

Common endpoints

The standard query endpoints are

Datasource

The Data source Data source The source of data for a Query, e.g. a Lucene based Index, a SQL Statistics Data source, etc.Click to see more details... endpoint is used to query Stroom for the details of a data source with a given Doc Ref Doc Ref A Doc Ref (or Document Reference) is an identifier used to identify most documents/entities in Stroom, e.g. an XSLT will have a Doc Ref.Click to see more details.... The details will include such things as the fields available and any restrictions on querying the data.

Search

The search endpoint is used to initiate a search against a data source or to request more data for an active search. A search request can be made using iterative mode, where it will perform the search and then only return the data it has immediately available. Subsequent requests for the same queryKey will also return the data immediately available, expecting that more results will have been found by the query. Requesting a search in non-iterative mode will result in the response being returned when the query has completed and all known results have been found.

The SearchRequest model is fairly complicated and contains not only the query terms but also a definition of how the data should be returned. A single SearchRequest can include multiple ResultRequest sections to return the queried data in multiple ways, e.g. as flat data and in an alternative aggregated form.

Stroom as a query builder

Stroom is able to export the json form of a SearchRequest model from its dashboards. This makes the dashboard a useful tool for building a query and the table settings to go with it. You can use the dashboard to define the data source, define the query terms tree and build a table definition (or definitions) to describe how the data should be returned. Then, clicking the download icon on the query pane of the dashboard will generate the SearchRequest json which can be immediately used with the /search API or modified to suit.

Destroy

This endpoint is used to kill an active query by supplying the queryKey for query in question.

Keep alive

Stroom will only hold search results from completed queries for a certain length of time. It will also terminate running queries that are too old. In order to prevent queries being aged off you can hit this endpoint to indicate to Stroom that you still have an interest in a particular query by supplying the query key.

Docs: Export Content API

Tue, 17 Jan 2023 00:00:00 +0000

Stroom has API API Application Programming Interface. An interface that one system can present so other systems can use it to communicate. Stroom has a number of APIs, e.g. its many REST APIs and its /datafeed interface for data receipt.Click to see more details... methods for exporting Content Content Content in Stroom typically means the user created documents/entities created in Stroom and as seen in the explorer tree. Content can be created/modified by Stroom users and imported/exported for sharing between different Stroom instances.Click to see more details... in Stroom to a single zip file.

Export All - `/api/export/v1`

This method will export all content in Stroom to a single zip file. This is useful as an alternative backup of the content or where you need to export the content for import into another Stroom instance.

In order to perform a full export, the user (identified by their API Key) performing the export will need to ensure the following:

Have created an API Key
The system property stroom.export.enabled is set to true.
The user has the application permission Export Configuration or Administrator.

Only those items that the user has Read permission on will be exported, so to export all items, the user performing the export will need Read permission on all items or have the Administrator application permission.

Performing an Export

To export all readable content to a file called export.zip do something like the following:

TOKEN="API KEY GOES IN HERE"
curl \
  --silent \
  --request GET \
  --header "Authorization:Bearer ${TOKEN}" \
  --output export.zip \
  https://stroom-fqdn/api/export/v1/

Note

If you encounter problems then replace --silent with --verbose to get more information.

Export Zip Format

The export zip will contain a number of files for each Document Document Typically refers to an item that can be created in the Explorer Tree, e.g. a Feed, a Pipeline, a Dashboard, etc. May also be known as an Entity.Click to see more details... exported. The number and type of these files will depend on the type of document, however every document will have the following two file types:

.node - This file represents the document’s location in the explorer tree along with its name and UUID.
.meta - This is the metadata for the document independent of the explorer tree. It contains the name, type and UUID of the document along with the unique identifier for the version of the document.

Documents may also have files like these (a non-exhaustive list):

.json - JSON data holding the content of the document, as used for Dashboards.
.txt - Plain text data holding the content of the document, as used for Dictionaries.
.xml - XML data holding the content of the document, as used for Pipelines.
.xsd - XML Schema content.
.xsl - XSLT content.

The following is an example of the content of an export zip file:

TEST_FEED_CERT.Feed.fcee4270-a479-4cc0-a79c-0e8f18a4bad8.meta
TEST_FEED_CERT.Feed.fcee4270-a479-4cc0-a79c-0e8f18a4bad8.node
TEST_FEED_PROXY.Feed.f06d4416-8b0e-4774-94a9-729adc5633aa.meta
TEST_FEED_PROXY.Feed.f06d4416-8b0e-4774-94a9-729adc5633aa.node
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.meta
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.xsl
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.node
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.xml
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.meta
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.node

Filenames

When documents are added to the zip, they are added with a directory structure that mirrors the explorer tree.

The filenames are of the form:

<name>.<type>.<UUID>.<extension>

As Stroom allows characters in document and folder names that would not be supported in operating system paths (or cause confusion), some characters in the name/directory parts are replaced by _ to avoid this. e.g. Dashboard 01/02/2020 would become Dashboard_01_02_2020.

If you need to see the contents of the zip as if viewing it within Stroom you can run this bash script in the root of the extracted zip.

#!/usr/bin/env bash

shopt -s globstar
for node_file in **/*.node; do
  name=
  name="$(grep -o -P "(?<=name=).*" "${node_file}" )"
  path=
  path="$(grep -o -P "(?<=path=).*" "${node_file}" )"

  echo "./${path}/${name}   (./${node_file})"
done

This will output something like:

(out) ./Standard Pipelines/Json/Events to JSON   (./Standard_Pipelines/Json/Events_to_JSON.XSLT.1c3d42c2-f512-423f-aa6a-050c5cad7c0f.node)
(out) ./Standard Pipelines/Json/JSON Extraction   (./Standard_Pipelines/Json/JSON_Extraction.Pipeline.13143179-b494-4146-ac4b-9a6010cada89.node)
(out) ./Standard Pipelines/Json/JSON Search Extraction   (./Standard_Pipelines/Json/JSON_Search_Extraction.XSLT.a8c1aa77-fb90-461a-a121-d4d87d2ff072.node)
(out) ./Standard Pipelines/Reference Loader   (./Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.node)

Docs: Reference Data API

Wed, 01 Feb 2023 00:00:00 +0000

The reference data store has an API to allow other systems to access the reference data store.

`/api/refData/v1/lookup`

The /lookup endpoint requires the caller to provide details of the reference feed and loader pipeline so if the effective stream is not in the store it can be loaded prior to performing the lookup. It is useful for forcing a reference load into the store and for performing ad-hoc lookups.

Note

As reference data stores are local to a node, it is best to send the request to a node that does processing as it is more likely to have already loaded the data. If you send it to a UI node that does not do processing, it is likely to trigger a load as the data will not be there.

Below is an example of a lookup request file req.json.

{
  "mapName": "USER_ID_TO_LOCATION",
  "effectiveTime": "2020-12-02T08:37:02.772Z",
  "key": "jbloggs",
  "referenceLoaders": [
    {
      "loaderPipeline" : {
        "name" : "Reference Loader",
        "uuid" : "da1c7351-086f-493b-866a-b42dbe990700",
        "type" : "Pipeline"
      },
      "referenceFeed" : {
        "name": "USER_ID_TOLOCATION-REFERENCE",
        "uuid": "60f9f51d-e5d6-41f5-86b9-ae866b8c9fa3",
        "type" : "Feed"
      }
    }
  ]
}

This is an example of how to perform the lookup on the local host.

curl \
  --json @req.json \
  --request POST \
  --header "Authorization:Bearer ${TOKEN}" \
  http://localhost:8080/api/refData/v1/lookup
(out)staff2

Docs: Explorer API

Tue, 11 Jun 2024 00:00:00 +0000

Creating a New Document

The explorer API is responsible for creation of all document types. The explorer API is used to create the initial skeleton of a document then the API specific to the document type in question is used to update the document skeleton with additional settings/content.

This is an example request file req.json:

{
  "docType": "Feed",
  "docName": "MY_FEED",
  "destinationFolder": {
    "type": "Folder",
    "uuid": "3dfab6a2-dbd5-46ee-b6e9-6df45f90cd85",
    "name": "My Folder",
    "rootNodeUuid": "0"
  },
  "permissionInheritance": "DESTINATION"
}

You need to set the following properties in the JSON:

docType - The type of the document being created, see Documents.
docName - The name of the new document.
destinationFolder.uuid - The UUID UUID A Universally Unique Identifier for uniquely identifying something. UUIDs are used as the identifier in Doc Refs. An example of a UUID is 4ffeb895-53c9-40d6-bf33-3ef025401ad3.Click to see more details... of the destination folder (or 0 if the document is being created in the System root.
rootNodeUuid - This is always 0 for the System root.

To create the skeleton document run the following:

curl \
  -s \
  -X POST \
  -H "Authorization:Bearer ${TOKEN}" \
  -H 'Content-Type: application/json' \
  --data @"req.json" \
  http://localhost:8080/api/explorer/v2/create/ \
| jq -r '.uuid'

This will create the document and return its new UUID to stdout.

Docs: Feed API

Tue, 11 Jun 2024 00:00:00 +0000

Creating a Feed

In order to create a Feed you must first create the skeleton document using the Explorer API.

Updating a Feed

To modify a feed you must first fetch the existing Feed document. This is done as follows:

curl \
  -s \
  -H "Authorization:Bearer ${TOKEN}" \
  "http://localhost:8080/api/feed/v1/${feed_uuid}" \

Where ${feed_uuid} is the UUID UUID A Universally Unique Identifier for uniquely identifying something. UUIDs are used as the identifier in Doc Refs. An example of a UUID is 4ffeb895-53c9-40d6-bf33-3ef025401ad3.Click to see more details... of the feed in question.

This will return the Feed document JSON.

{
  "type": "Feed",
  "uuid": "0dafc9c2-dcd8-4bb6-88ce-5ee228babe78",
  "name": "MY_FEED",
  "version": "32dae12f-a696-4e0e-8acb-47cf0ad3c77f",
  "createTimeMs": 1718103980225,
  "updateTimeMs": 1718103980225,
  "createUser": "admin",
  "updateUser": "admin",
  "reference": false,
  "streamType": "Raw Events",
  "status": "RECEIVE"
}

You can use jq to modify this JSON to add/change any of the document settings.

Example Script

The following is an example bash script for creating and modifying multiple Feeds. It requires curl and jq to run.

#!/usr/bin/env bash

set -e -o pipefail

main() {

  # Your API key
  local TOKEN="sak_d5752a32b2_mv1JYUYUuvRUDpikW75G5w4kQUq7EEjShQ9DiRjN14yEFonKTW42KbeQogui52gTjq9RDRufNEz2MXt1PRCThudzHU5RVpLMbZKThCgyyEX2y2sBrk31rYMJRKNg2yMG"
  # UUID of the dest folder
  local FOLDER_UUID="fc617580-8cf0-4ac3-93dd-93604603aef0"

  local feed_name
  local create_feed_req
  local feed_uuid
  local feed_doc

  for i in {1..2}; do
    # Use date to make a unique name for the test
    feed_name="MY_FEED_$(date +%s)_${i}"

    # Set the feed name and its destination
    create_feed_req=$(cat <<-END
      {
        "docType": "Feed",
        "docName": "${feed_name}",
        "destinationFolder": {
          "type": "Folder",
          "uuid": "${FOLDER_UUID}",
          "rootNodeUuid": "0"
        },
        "permissionInheritance": "DESTINATION"
      }
END
    )

    # Create the skeleton feed and extract its new UUID from the response
    feed_uuid=$( \
      curl \
        -s \
        -X POST \
        -H "Authorization:Bearer ${TOKEN}" \
        -H 'Content-Type: application/json' \
        --data "${create_feed_req}" \
        http://localhost:8080/api/explorer/v2/create/ \
      | jq -r '.uuid'
    )

    echo "Created feed $i with name '${feed_name}' and UUID '${feed_uuid}'"

    # Fetch the created feed
    feed_doc=$( \
      curl \
        -s \
        -H "Authorization:Bearer ${TOKEN}" \
        "http://localhost:8080/api/feed/v1/${feed_uuid}" \
      )

    echo -e "Skeleton Feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"

    # Add/modify properties on the feed doc
    feed_doc=$(jq '
      .classification="HUSH HUSH" 
      | .encoding="UTF8" 
      | .contextEncoding="ASCII" 
      | .streamType="Events"
      | .volumeGroup="Default Volume Group"' <<< "${feed_doc}")

    #echo -e "Updated feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"

    # Update the feed with the new properties
    curl \
      -s \
      -X PUT \
      -H "Authorization:Bearer ${TOKEN}" \
      -H 'Content-Type: application/json' \
      --data "${feed_doc}" \
      "http://localhost:8080/api/feed/v1/${feed_uuid}" \
    > /dev/null

    # Fetch the created feed
    feed_doc=$( \
      curl \
        -s \
        -H "Authorization:Bearer ${TOKEN}" \
        "http://localhost:8080/api/feed/v1/${feed_uuid}" \
      )

    echo -e "Updated Feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"
    echo
  done
}

main "$@"

Stroom – api

Docs: API Specification

Swagger UI

API Endpoints in Application Logs

Docs: Application Programming Interfaces (API)

Docs: Calling an API

Authentication

Calling an API method with curl

HTTP Requests Without a Body

Warning

Requests With a Body

Note

Handling JSON

Docs: Query APIs

Common endpoints

Datasource

Search

Stroom as a query builder

Destroy

Keep alive

Docs: Export Content API

Export All - /api/export/v1

Performing an Export

Note

Export Zip Format

Filenames

Docs: Reference Data API

/api/refData/v1/lookup

Note

Docs: Explorer API

Creating a New Document

Docs: Feed API

Creating a Feed

Updating a Feed

Example Script

Calling an API method with `curl`

Export All - `/api/export/v1`

`/api/refData/v1/lookup`