This is the multi-page printable view of this section. Click here to print.
Stroom has many public REST APIs to allow other systems to interact with Stroom. Everything that can be done via the user interface can also be done using the API.
All methods on the API will are authenticated and authorised, so the permissions will be exactly the same as if the API user is using the Stroom user interface directly.
The APIs are available as a Swagger Open API specification in the following forms:
A dynamic Swagger user interface is also available for viewing all the API endpoints with details of parameters and data types. This can be found in two places.
/stroom/noauth/swagger-ui.The API methods are also all listed in the application logs when Stroom first boots up, e.g.
INFO 2023-01-17T11:09:30.244Z main i.d.j.DropwizardResourceConfig The following paths were found for the configured resources:
GET /api/account/v1/ (stroom.security.identity.account.AccountResourceImpl)
POST /api/account/v1/ (stroom.security.identity.account.AccountResourceImpl)
POST /api/account/v1/search (stroom.security.identity.account.AccountResourceImpl)
DELETE /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
GET /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
PUT /api/account/v1/{id} (stroom.security.identity.account.AccountResourceImpl)
GET /api/activity/v1 (stroom.activity.impl.ActivityResourceImpl)
POST /api/activity/v1 (stroom.activity.impl.ActivityResourceImpl)
POST /api/activity/v1/acknowledge (stroom.activity.impl.ActivityResourceImpl)
GET /api/activity/v1/current (stroom.activity.impl.ActivityResourceImpl)
...
You will also see entries in the logs for the various servlets exposed by Stroom, e.g.
INFO ... main s.d.common.Servlets Adding servlets to application path/port:
INFO ... main s.d.common.Servlets stroom.core.servlet.DashboardServlet => /stroom/dashboard
INFO ... main s.d.common.Servlets stroom.core.servlet.DynamicCSSServlet => /stroom/dynamic.css
INFO ... main s.d.common.Servlets stroom.data.store.impl.ImportFileServlet => /stroom/importfile.rpc
INFO ... main s.d.common.Servlets stroom.receive.common.ReceiveDataServlet => /stroom/noauth/datafeed
INFO ... main s.d.common.Servlets stroom.receive.common.ReceiveDataServlet => /stroom/noauth/datafeed/*
INFO ... main s.d.common.Servlets stroom.receive.common.DebugServlet => /stroom/noauth/debug
INFO ... main s.d.common.Servlets stroom.data.store.impl.fs.EchoServlet => /stroom/noauth/echo
INFO ... main s.d.common.Servlets stroom.receive.common.RemoteFeedServiceRPC => /stroom/noauth/remoting/remotefeedservice.rpc
INFO ... main s.d.common.Servlets stroom.core.servlet.StatusServlet => /stroom/noauth/status
INFO ... main s.d.common.Servlets stroom.core.servlet.SwaggerUiServlet => /stroom/noauth/swagger-ui
INFO ... main s.d.common.Servlets stroom.resource.impl.SessionResourceStoreImpl => /stroom/resourcestore/*
INFO ... main s.d.common.Servlets stroom.dashboard.impl.script.ScriptServlet => /stroom/script
INFO ... main s.d.common.Servlets stroom.security.impl.SessionListServlet => /stroom/sessionList
INFO ... main s.d.common.Servlets stroom.core.servlet.StroomServlet => /stroom/ui
In order to use the API endpoints you will need to authenticate. Authentication is achieved using an API Key or Token .
You will either need to create an API key for your personal Stroom user account or for a shared processing user account. Whichever user account you use it will need to have the necessary permissions for each API endpoint it is to be used with.
To create an API key (token) for a user:
To make an authenticated API call you need to provide a header of the form Authorization:Bearer ${TOKEN}, where ${TOKEN} is your API Key as copied from Stroom.
curlThis section describes how to call an API method using the command line tool curl as an example client.
Other clients can be used, e.g. using python, but these examples should provide enough help to get started using another client.
Typically HTTP GET requests will have no body/payload
Often PUT and DELETE requests will also have no body/payload.
The following is an example of how to call an HTTP GET method (i.e. a method that does not require a request body) on the API using curl.
The --insecure argument is used in this example which means certificate verification will not take place.
It is recommended not to use this argument and instead supply curl with client and certificate authority certificates to make a secure connection.
You can either call the API via Nginx (or similar reverse proxy) at https://stroom-fddn/api/some/path or if you are making the call from one of the stroom hosts you can go direct using http://localhost:8080/api/some/path. The former is preferred as it is more secure.
A lot of the API methods in Stroom require complex bodies/payloads for the request.
The following example is an HTTP POST to perform a reference data lookup on the local host.
Create a file req.json containing:
{
"mapName": "USER_ID_TO_STAFF_NO_MAP",
"effectiveTime": "2024-12-02T08:37:02.772Z",
"key": "user2",
"referenceLoaders": [
{
"loaderPipeline" : {
"name" : "Reference Loader",
"uuid" : "da1c7351-086f-493b-866a-b42dbe990700",
"type" : "Pipeline"
},
"referenceFeed" : {
"name": "STAFF-NO-REFERENCE",
"uuid": "350003fe-2b6c-4c57-95ed-2e6018c5b3d5",
"type" : "Feed"
}
}
]
}
Now send the request with curl.
This API method returns plain text or XML depending on the reference data value.
This assumes you are using curl version 7.82.0 or later that supports the --json argument.
If not you will need to replace --json with --data and add these arguments:
--header "Content-Type: application/json"
--header "Accept: application/json"
jq is a utility for processing JSON and is very useful when using the API methods.
For example to get just the build version from the node info endpoint:
The Query APIs use common request/response models and end points for querying each type of data source held in Stroom. The request/response models are defined in stroom-query .
Currently Stroom exposes a set of query endpoints for the following data source types. Each data source type will have its own endpoint due to differences in the way the data is queried and the restrictions imposed on the query terms. However they all share the same API definition.
The detailed documentation for the request/responses is contained in the Swagger definition linked to above.
The standard query endpoints are
The Data Source endpoint is used to query Stroom for the details of a data source with a given DocRef . The details will include such things as the fields available and any restrictions on querying the data.
The search endpoint is used to initiate a search against a data source or to request more data for an active search. A search request can be made using iterative mode, where it will perform the search and then only return the data it has immediately available. Subsequent requests for the same queryKey will also return the data immediately available, expecting that more results will have been found by the query. Requesting a search in non-iterative mode will result in the response being returned when the query has completed and all known results have been found.
The SearchRequest model is fairly complicated and contains not only the query terms but also a definition of how the data should be returned. A single SearchRequest can include multiple ResultRequest sections to return the queried data in multiple ways, e.g. as flat data and in an alternative aggregated form.
Stroom is able to export the json form of a SearchRequest model from its dashboards. This makes the dashboard a useful tool for building a query and the table settings to go with it. You can use the dashboard to defined the data source, define the query terms tree and build a table definition (or definitions) to describe how the data should be returned. The, clicking the download icon on the query pane of the dashboard will generate the SearchRequest json which can be immediately used with the /search API or modified to suit.
This endpoint is used to kill an active query by supplying the queryKey for query in question.
Stroom will only hold search results from completed queries for a certain lenght of time. It will also terminate running queries that are too old. In order to prevent queries being aged off you can hit this endpoint to indicate to Stroom that you still have an interest in a particular query by supplying the query key.
Stroom has API methods for exporting content in Stroom to a single zip file.
/api/export/v1This method will export all content in Stroom to a single zip file. This is useful as an alternative backup of the content or where you need to export the content for import into another Stroom instance.
In order to perform a full export, the user (identified by their API Key) performing the export will need to ensure the following:
stroom.export.enabled is set to true.Export Configuration or Administrator.Only those items that the user has Read permission on will be exported, so to export all items, the user performing the export will need Read permission on all items or have the Administrator application permission.
To export all readable content to a file called export.zip do something like the following:
--silent with --verbose to get more information.
The export zip will contain a number of files for each document exported. The number and type of these files will depend on the type of document, however every document will have the following two file types:
.node - This file represents the document’s location in the explorer tree along with its name and UUID..meta - This is the metadata for the document independent of the explorer tree.
It contains the name, type and UUID of the document along with the unique identifier for the version of the document.Documents may also have files like these (a non-exhaustive list):
.json - JSON data holding the content of the document, as used for Dashboards..txt - Plain text data holding the content of the document, as used for Dictionaries..xml - XML data holding the content of the document, as used for Pipelines..xsd - XML Schema content..xsl - XSLT content.The following is an example of the content of an export zip file:
TEST_FEED_CERT.Feed.fcee4270-a479-4cc0-a79c-0e8f18a4bad8.meta
TEST_FEED_CERT.Feed.fcee4270-a479-4cc0-a79c-0e8f18a4bad8.node
TEST_FEED_PROXY.Feed.f06d4416-8b0e-4774-94a9-729adc5633aa.meta
TEST_FEED_PROXY.Feed.f06d4416-8b0e-4774-94a9-729adc5633aa.node
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.meta
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.xsl
TEST_REFERENCE_DATA_EVENTS_XXX.XSLT.4f74999e-9d69-47c7-97f7-5e88cc7459f7.node
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.xml
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.meta
Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.node
When documents are added to the zip, they are added with a directory structure that mirrors the explorer tree.
The filenames are of the form:
<name>.<type>.<UUID>.<extension>
As Stroom allows characters in document and folder names that would not be supported in operating system paths (or cause confusion), some characters in the name/directory parts are replaced by _ to avoid this. e.g. Dashboard 01/02/2020 would become Dashboard_01_02_2020.
If you need to see the contents of the zip as if viewing it within Stroom you can run this bash script in the root of the extracted zip.
#!/usr/bin/env bash
shopt -s globstar
for node_file in **/*.node; do
name=
name="$(grep -o -P "(?<=name=).*" "${node_file}" )"
path=
path="$(grep -o -P "(?<=path=).*" "${node_file}" )"
echo "./${path}/${name} (./${node_file})"
done
This will output something like:
./Standard Pipelines/Json/Events to JSON (./Standard_Pipelines/Json/Events_to_JSON.XSLT.1c3d42c2-f512-423f-aa6a-050c5cad7c0f.node)
./Standard Pipelines/Json/JSON Extraction (./Standard_Pipelines/Json/JSON_Extraction.Pipeline.13143179-b494-4146-ac4b-9a6010cada89.node)
./Standard Pipelines/Json/JSON Search Extraction (./Standard_Pipelines/Json/JSON_Search_Extraction.XSLT.a8c1aa77-fb90-461a-a121-d4d87d2ff072.node)
./Standard Pipelines/Reference Loader (./Standard_Pipelines/Reference_Loader.Pipeline.da1c7351-086f-493b-866a-b42dbe990700.node)
The reference data store has an API to allow other systems to access the reference data store.
/api/refData/v1/lookupThe /lookup endpoint requires the caller to provide details of the reference feed and loader pipeline so if the effective stream is not in the store it can be loaded prior to performing the lookup.
It is useful for forcing a reference load into the store and for performing ad-hoc lookups.
Below is an example of a lookup request file req.json.
{
"mapName": "USER_ID_TO_LOCATION",
"effectiveTime": "2020-12-02T08:37:02.772Z",
"key": "jbloggs",
"referenceLoaders": [
{
"loaderPipeline" : {
"name" : "Reference Loader",
"uuid" : "da1c7351-086f-493b-866a-b42dbe990700",
"type" : "Pipeline"
},
"referenceFeed" : {
"name": "USER_ID_TOLOCATION-REFERENCE",
"uuid": "60f9f51d-e5d6-41f5-86b9-ae866b8c9fa3",
"type" : "Feed"
}
}
]
}
This is an example of how to perform the lookup on the local host.
The explorer API is responsible for creation of all document types. The explorer API is used to create the initial skeleton of a document then the API specific to the document type in question is used to update the document skeleton with additional settings/content.
This is an example request file req.json:
{
"docType": "Feed",
"docName": "MY_FEED",
"destinationFolder": {
"type": "Folder",
"uuid": "3dfab6a2-dbd5-46ee-b6e9-6df45f90cd85",
"name": "My Folder",
"rootNodeUuid": "0"
},
"permissionInheritance": "DESTINATION"
}
You need to set the following properties in the JSON:
docType - The type of the document being created, see Documents.docName - The name of the new document.destinationFolder.uuid - The
UUID
of the destination folder (or 0 if the document is being created in the System root.rootNodeUuid - This is always 0 for the System root.To create the skeleton document run the following:
This will create the document and return its new UUID to stdout.
In order to create a Feed you must first create the skeleton document using the Explorer API.
To modify a feed you must first fetch the existing Feed document. This is done as follows:
Where ${feed_uuid} is the
UUID
of the feed in question.
This will return the Feed document JSON.
{
"type": "Feed",
"uuid": "0dafc9c2-dcd8-4bb6-88ce-5ee228babe78",
"name": "MY_FEED",
"version": "32dae12f-a696-4e0e-8acb-47cf0ad3c77f",
"createTimeMs": 1718103980225,
"updateTimeMs": 1718103980225,
"createUser": "admin",
"updateUser": "admin",
"reference": false,
"streamType": "Raw Events",
"status": "RECEIVE"
}
You can use jq to modify this JSON to add/change any of the document settings.
The following is an example bash script for creating and modifying multiple Feeds.
It requires curl and jq to run.
#!/usr/bin/env bash
set -e -o pipefail
main() {
# Your API key
local TOKEN="sak_d5752a32b2_mv1JYUYUuvRUDpikW75G5w4kQUq7EEjShQ9DiRjN14yEFonKTW42KbeQogui52gTjq9RDRufNEz2MXt1PRCThudzHU5RVpLMbZKThCgyyEX2y2sBrk31rYMJRKNg2yMG"
# UUID of the dest folder
local FOLDER_UUID="fc617580-8cf0-4ac3-93dd-93604603aef0"
local feed_name
local create_feed_req
local feed_uuid
local feed_doc
for i in {1..2}; do
# Use date to make a unique name for the test
feed_name="MY_FEED_$(date +%s)_${i}"
# Set the feed name and its destination
create_feed_req=$(cat <<-END
{
"docType": "Feed",
"docName": "${feed_name}",
"destinationFolder": {
"type": "Folder",
"uuid": "${FOLDER_UUID}",
"rootNodeUuid": "0"
},
"permissionInheritance": "DESTINATION"
}
END
)
# Create the skeleton feed and extract its new UUID from the response
feed_uuid=$( \
curl \
-s \
-X POST \
-H "Authorization:Bearer ${TOKEN}" \
-H 'Content-Type: application/json' \
--data "${create_feed_req}" \
http://localhost:8080/api/explorer/v2/create/ \
| jq -r '.uuid'
)
echo "Created feed $i with name '${feed_name}' and UUID '${feed_uuid}'"
# Fetch the created feed
feed_doc=$( \
curl \
-s \
-H "Authorization:Bearer ${TOKEN}" \
"http://localhost:8080/api/feed/v1/${feed_uuid}" \
)
echo -e "Skeleton Feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"
# Add/modify propeties on the feed doc
feed_doc=$(jq '
.classification="HUSH HUSH"
| .encoding="UTF8"
| .contextEncoding="ASCII"
| .streamType="Events"
| .volumeGroup="Default Volume Group"' <<< "${feed_doc}")
#echo -e "Updated feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"
# Update the feed with the new properties
curl \
-s \
-X PUT \
-H "Authorization:Bearer ${TOKEN}" \
-H 'Content-Type: application/json' \
--data "${feed_doc}" \
"http://localhost:8080/api/feed/v1/${feed_uuid}" \
> /dev/null
# Fetch the created feed
feed_doc=$( \
curl \
-s \
-H "Authorization:Bearer ${TOKEN}" \
"http://localhost:8080/api/feed/v1/${feed_uuid}" \
)
echo -e "Updated Feed doc for '${feed_name}'\n$(jq '.' <<< "${feed_doc}")"
echo
done
}
main "$@"