Indexing and Search

• 1: Elasticsearch
• 2: Apache Solr
• 3: Stroom Search API

1 - Elasticsearch

See: Elasticsearch indexing in the User Guide.

2 - Apache Solr

Assumptions

1. You are familiar with Lucene indexing within Stroom
2. You have some data to index

Points to note

1. A Solr core is the home for exactly one Stroom index.
2. Cores must initially be created in Solr.
3. It is good practice to name your Solr core the same as your Stroom Index.

Method

1. Start a docker container for a single solr node.

   docker run -d -p 8983:8983 --name my_solr solr

2. Check your Solr node. Point your browser at http://yourSolrHost:8983

3. Create a core in Solr using the CLI.

   docker exec -it my_solr solr create_core -c test_index

4. Create a SolrIndex in Stroom

   

   ---

5. Update settings for your new Solr Index in Stroom then press “Test Connection”. If successful then press Save. Note the “Solr URL” field is a reference to the newly created Solr core.

   

   ---

6. Add some Index fields. e.g.EventTime, UserId

7. Retention is different in Solr, you must specify an expression that matches data that can be deleted.

   

   ---

8. Your Solr Index can now be used as per a Stroom Lucene Index. However, your Indexing pipeline must use a SolrIndexingFilter instead of an IndexingFilter.

3 - Stroom Search API

1. Create an API Key for yourself, this will allow the API to authenticate as you and run the query with your privileges.

2. Create a Dashboard that extracts the data you are interested in. You should create a Query and Table.

3. Download the JSON for your Query. Press the download icon in the Query Pane to generate a file containing the JSON. Save the JSON to a file named query.json.

4. Use curl to send the query to Stroom.

   API_KEY='<put your API Key here' \
   URI=stroom.host/api/searchable/v2/search \
   curl \
   -s \
   --request POST \
   ${URL} \
   -o response.out \
   -H "Authorization:Bearer ${API_KEY}" \
   -H "Content-Type: application/json" \
   --data-binary @query.json

5. The query response should be in a file named response.out.

6. Optional step: reformat the response to csv using jq.

   cat response.out | jq '.results[0].rows[].values | @csv'