Community

• 1: Roadmap
• 2: Developer Guide
• 2.1: Setting up releases to Sonatype & Maven Central
• 2.2: Building the documentation
• 2.3: Components
• 2.4: Release Process
• 2.5: Running in Docker
• 2.6: Running Stroom in an IDE
• 3: Developing Content
• 4: Documenting Stroom
• 4.1: Building the Documentation
• 4.2: Managing Stroom Versions
• 4.3: Documentation Style Guide
• 4.3.1: Site Structure
• 4.3.2: Front Matter
• 4.3.3: Markdown Style Conventions
• 4.3.4: Additional Page Elements
• 4.3.5: Images
• 4.3.6: Icon Gallery

1 - Roadmap

v7.0

Reference data storage

Reference data uses a memory-mapped disk-based store rather than direct memory to reduce the memory overhead associated with storing reference data.

Search result storage

Search results are stored on disk rather than in memory during creation to reduce the memory overhead incurred by search.

Modularisation

Separation of Stroom components into discreet modules that have clear APIs and separate persistence where required to reduce coupling.

Modernisation of libraries

Changing Stroom libraries to replace Spring with Guice and Hibernate with JOOQ.

Annotations

Search results in dashboards can be annotated to provide status and notes relating to the result item, e.g. an event. These annotations can later be searched to see which events have annotations associated with them.

v7.1

Elastic search integration

Elastic search can be used for indexing data. Data can be sent to an elastic index via a pipeline element and an elastic index can be queried from a Stroom dashboard.

Interactive Visualisations

Selecting or manipulating parts of visualisations can be used to trigger further queries to zoom in or select specific data etc.

Improved Proxy Aggregation

Proxy aggregation can better match user defined aggregate sizes and forward to multiple destinations.

User Preferences

The UI can be customised to meet the needs of an end user including theme (dark mode), date and time format, font, layout.

v7.2

XSLT 3

Add support for XSLT 3.

Accessibility Improvements

Refactoring some elements of the UI to improve accessibility.

v8+

Authorisation enhancements

The Stroom authorisation system is split out into a separate service and provides integration with external authorisation mechanisms.

Proxy processing

Stroom proxy is capable of pipeline processing in the same way as a full Stroom application. Pipeline configuration content can be pushed to proxies so that they can perform local processing prior to sending data to Stroom.

Multiple input sources

Stroom is capable of processing data from a Kafka topic, HDFS, the local file system, HTTP POST in addition to the included stream store.

Multiple output destinations

Stroom has improved support for writing to various destinations such as Kafka, HDFS, etc. Improvements include compression and meta data wrapping for future import.

Improved field extraction

Enhancements to data splitter and associated UI to make the process of extracting field data from raw content much easier.

Kafka analytics

Stroom exposes the use of Apache Kafka Streams for performing certain complex analytics.

Query fusion

Stroom allows multiple data sources to be queried at the same time and the results of the queries to be fused. This might be for fusing data from multiple search indexes, e.g. events and annotations, or to effectively decorate results with additional data at search time.

Reference data deltas

Reference data is enhanced to cope with changes (additions and removals) of state information rather than always relying on complete snapshots.

2 - Developer Guide

2.1 - Setting up releases to Sonatype & Maven Central

Create a Sonatype account

You need to create an account on Sonatype and you will need to raise a jira ticket on Sonatype’s jira to get approved on the uk.gov.gchq group. This will require an existing user approved for the group to approve you on the ticket.

Setting up a GPG key

You can use the following commands for setting up a GPG2 key for signing.

# Generate the GPG2 key
gpg2 --gen-key

# To list all keys
gpg2 --list-keys

# To get the key ID
gpg2  --list-secret-keys | grep "\[SC\]" | tr -s ' ' | cut -d' ' -f2 | cut -d'/' -f2

# To send the public keys to a key server
gpg2 --keyserver hkp://pool.sks-keyservers.net --send-keys <key id>
gpg2 --keyserver hkp://keyserver.ubuntu.com --send-keys <key id>
gpg2 --keyserver hkp://pgp.mit.edu --send-keys <key id>

# To display the secret key in base64 form, for use in GH actions
key="$(gpg2 --armor --export-secret-keys <key id> | base64 -w0)"; echo -e "-------\n$key\n-------"; key=""

Setting up the gradle build

The signing and release to Sonatype is done by various gradle plugins.

id "io.github.gradle-nexus.publish-plugin" version "1.0.0"
id "signing"
id "maven-publish"

See the root and event-logging-api gradle build files (in the event-logging repo) for an example of how to set up gradle.

The credentials can be passed to the gradle build using special gradle env vars Project Properties (external). The credentials required are:

• ORG_GRADLE_PROJECT_SIGNINGKEY - The key as produced by the gpg2 --armor command.
• ORG_GRADLE_PROJECT_SIGNINGPASSWORD - The password for the GPG key.
• ORG_GRADLE_PROJECT_SONATYPEUSERNAME - The account username on Sonatype.
• ORG_GRADLE_PROJECT_SONATYPEPASSWORD - The account password on Sonatype.

Setting up Github Actions

You will need to provide Github with the four secrets listed above by setting them as repository secrets at https://github.com/gchq//settings/secrets/actions. For each one create a secret with the ORG_GRADLE_... bit as the name.

So that the action can create the Github release you will also need to set up an SSH key pair and provide it with the public and private key. To generate the key pair do:

ssh-keygen -t rsa -b 4096 -f <repo>_deploy_key

The key pair will be created in ~/.ssh/.

Create a repo deploy key with the public key, named ‘Actions Deploy Key’ and with write access at https://github.com///settings/keys/new. Create a repo secret with the private key, named ‘SSH_DEPLOY_KEY’ at https://github.com///settings/secrets/actions/new.

2.2 - Building the documentation

Stroom’s documentation is built using GitBook (external link).

Prerequisites

NPM

You need NPM to install the GitBook command line toolchain. To get NPM install node (external link).

GitBook command line tools

npm install -g gitbook-cli

Build the book

GitBook uses plugins, e.g. anchorjs allows us to create links to headings within a file. You need to install these plugins first. The below commands should be run in the project root.

gitbook install

You can build the documentation like this:

gitbook build

Or you can run the GitBook server which will watch your files as you work and server them on localhost:4000.

gitbook serve

Troubleshooting

I get an error when trying to run gitbook serve

If you see Errpr: watch /path/to/stroom ENOSPC then run the following: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Links don’t work when I load _book/index.html

It won’t, because CORS is not and cannot be enabled when viewing local files. You need to run gitbook serve or if you really don’t want to do that try cd _book && python -m SimpleHTTPServer.

2.3 - Components

Some examples of components in Stroom include

• stroom-activity - Component for recording a users actions against a current activity
• stroom-dictionary - Component for storing lists of words.
• stroom-statistics - Component for recording statistical data, e.g. amount of data received in X minutes.

In the project structure a component appears as a first level subdirectory of the root project folder. Components have further subdirectories (modules) that make up the various parts of the component, e.g.

• stroom - Root project
  • stroom-activity - The component
    • stroom-activity-api - API module for stroom-activity
    • stroom-activity-impl - Implementation of the API and other module implementation code
    • stroom-activity-impl-db - Database persistence implementation used by impl
    • stroom-activity-impl-db-jooq - JOOQ generated classes used by stroom-activity-impl-db
    • stroom-activity-mock - Mock implementation for the stroom-activity API

Dependencies between a modules components

The diagram below shows the dependencies between the different modules that make up a component as well as the internal dependencies within the impl module. The actual implementations used at runtime are determined by Guice bindings in whichever Guice modules are loaded by the application. Tests can bind mock implementations of a components API just by using the Guice module within the mock module.

---

Dependencies between components

Typically a component will need to call out to other components to apply security constraints and to log user activity. These typical relationships are shown in the diagram below.

---

Component API, e.g. modules ending in -api

API layer

All communication between components in stroom must be made via a component’s API. The API provides the minimum surface area for communication between components and decouples dependencies between components to just the API code. For component testing purposes mock implementations of these APIs can be used to limit testing to just a single component.

Component API and service implementation, e.g. modules ending in -impl

Client interaction - REST services and GWT Action Handlers

The uppermost layer of the server side code services requests from the client. The client may make restful calls as is the case for the new UI or will use Actions that are handles with ActionHandlers as is the case for the legacy GWT UI.

Since this layer deals with all client interaction it should be responsible for creating audit logs for all user activity, e.g. accessing documents, searching etc. No audit logging should need to be performed at a lower level within the application as deeper levels have less knowledge of user intent since they may just be playing a part in the wider request.

The client interaction layer adds no logic and asks the underlying service layer to service the encapsulated request away from the REST endpoint wrapping code or GWT action handler code. This allows multiple types of endpoint to use the same underlying service layer. If a request requires the use of multiple services to form a response, this must be handled within the service layer by the primary service which will be responsible for any such orchestration.

Service layer

The service layer applies permission constraints to any requests being made so that only calls from identified and permitted users are allowed to proceed. The service layer performs all orchestration and business logic, and is responsible for all mutations of objects that will be persisted by the underlying persistence layer such as stamping objects to be updated with the current user and update time.

The service layer provides implementations for any API that the component may have.

The service layer provides the DAO (Data Access Object) API for the persistence layer to implement but maintains no knowledge of underlying persistence implementation, e.g. database queries.

Persistence implementation, e.g. modules ending in -impl-db

Persistence layer - DAOs

The persistence layer is an implementation of one or more DAOs specified in the service layer. The persistence layer provides no logic, it just stores and retrieves objects in a database or other persistence technology. If serialisation/de-serialisation is required in order to persist the object then that should also be performed by this layer so that no code above this layer has to care about this implementation detail.

The persistence layer does not apply security or permissions checking so should not need to reference the security API.

2.4 - Release Process

Performing a named release

This is a general guide for performing a release of a named version of one of the stroom family of products.

1. Ensure the CHANGELOG has been updated for the new x.y.z version, i.e. ensure there is a section for vx.y.z and the links to git diffs at the bottom are up to date. e.g.

   ## [v3.1.1_schema-v3.1.2] - 2017-11-14
   
   ### Added
   
   * Add sources and javadoc jars
   
   ### Changed
   
   * Uplift schema to v3.1.2
   
   * Change build to Gradle
   

   and

   [v3.1.2_schema-v3.1.2]: https://github.com/gchq/event-logging/compare/v3.1.1_schema-v3.1.2...v3.1.2_schema-v3.1.2
   

2. If the changes include merges in from other branches, e.g. merging a big fix from v1.0 to v2.0 then add a Merged section to make it clear that the changes in the merge have been applied, e.g.

   ### Merged
   
   * Merged in [v1.0.1]
   
   

3. Test the named release build locally by adding the argument -Pversion=vx.y.z to the gradle build command.

4. Ensure all code is committed to the master branch, or a release branch, such as v5.0.

5. To perform the release create an annotated git tag as follows:

   git tag -a vx.y.z

6. Complete the tag’s commit message using the following format

   stroom-something-vx.y.z
   
   
   @@@
   

   Where @@@ is the content of the x.y.z section from the CHANGELOG. GitHub takes the top line as the title for the release. Git will ignore any lines starting with a # so remove them from the markdown headings. Keep the * bullets. E.g.

   event-logging-v3.1.1_schema-v3.1.2
   
   Added 
   
   * Add sources and javadoc jars
   
   Changed
   
   * Uplift schema to v3.1.2
   
   * Change build to Gradle
   

7. Push the tag. Travis will run a build and on detecting the tag, will release any build artefacts to GitHUb Releases. If the project has any published Maven artefacts these will be releases to Bintray.

SNAPSHOT releases

SNAPSHOT releases should not and cannot be released to Bintray. If a development version of a library needs to be shared between projects then you can either use the Gradle task publishToMavenLocal to publish a SNAPSHOT version to your local Maven repository and change your dependency version to SNAPSHOT, or perform a named release along the lines of vx.y.z-alpha.n.

Release Versioning conventions

Semantic versioning is used, and this should be adhered to, see SemVer (external link). The following are examples of valid version names

• SNAPSHOT - Used only for local development, never to be published publicly.

• v3.3.0 - Initial release of v3.3, with an associated v3.3 branch.

• v3.3.1 - A patch release to v3.3 on the v3.3 branch.

• v3.4.0-alpha.1 - An alpha release of v3.4, either on master or a v3.4 branch

• v3.4.0-beta.1 - An beta release of v3.4, either on master or a v3.4 branch

To Perform a Local Build

Full build:

./gradlew clean build

Build without unit tests

./gradlew clean build -x test

Build without integration tests

./gradlew clean build -x integrationTest

Build without any tests or GWT compilation (GWT compilation applies to stroom only)

./gradlew clean build -x test -x integrationTest -x gwtCompile

2.5 - Running in Docker

NOTE: The published docker images are intended for small scale testing or evaluation purposes and are currently un-tested in a production environment.

Stroom v6.x release

Prerequisites

In order to run Stroom v6.x using Docker you will need the following installed on the machine you intend to run Stroom on:

• A Linux-like shell environment.
• docker CE (v17.12.0+) - e.g docs.docker.com/install/linux/docker-ce/centos/ (external link) for Centos
• docker-compose (v1.21.0+) - docs.docker.com/compose/install/ (external link)
• bash (v4+)
• jq - stedolan.github.io/jq/ (external link) e.g. sudo yum install jq
• curl

Install steps

This will install the core stack (Stroom and the peripheral services required to run Stroom).

Visit stroom-resources/releases (external link) and find the latest stroom_core release and copy the link to the associated stroom_core*.tar.gz archive file.

Using stroom_core-v6.0.19 as an example:

# Set the release version to download
export STROOM_STACK_VER="stroom_core-v6.0.19"

# Make the stack directory
mkdir ${STROOM_STACK_VER}

# Download and extract the Stroom stack into the directory stroom_core-vX.Y.Z
curl -sL https://github.com/gchq/stroom-resources/releases/download/${STROOM_STACK_VER}/${STROOM_STACK_VER}.tar.gz | tar xz -C ${STROOM_STACK_VER}

# Navigate into the new stack directory, where xxxx is the directory that has just been created
cd ${STROOM_STACK_VER}

# Start the stack
./start.sh

Alternatively if you understand the risks of redirecting web sourced content direct to bash, you can get the latest release using:

# Download and extract the Stroom stack
bash <(curl -s https://gchq.github.io/stroom-resources/get_stroom.sh)

# Navigate into the new stack directory
cd stroom_core_test/stroom_core_test*

# Start the stack
./start.sh

On first run stroom will build the database schemas so this can take a minute or two. The start.sh script will provide details of the various URLs that are available.

Open a browser (preferably Chrome) at https://localhost/stroom and login with:

• username: admin (NOT an email address)
• password: admin

The stroom stack comes supplied with self-signed certificates so you may need to accept a prompt warning you about visiting an untrusted site.

Stroom v5.x release

Prerequisites

In order to run Stroom v5.x using Docker you will need the following installed on the machine you intend to run Stroom on:

• A Linux-like shell environment with bash and GNU sed/grep
• docker CE (v17.12.0+)
• docker-compose (v1.21.0+)
• git

Install steps

# Clone the stroom-resources git repository
git clone https://github.com/gchq/stroom-resources.git

# Navigate to the bin directory in the repository
cd stroom-resources/bin

# Start the stroom v5.x stack using docker/docker-compose
./bounceIt.sh -f env/stroom5.env

Open a browser (preferably Chrome) at http://localhost:8080/stroom and login with:

• username: admin
• password: admin

Docker Hub links

The Stroom image (external link)

The GCHQ organisation (external link)

2.6 - Running Stroom in an IDE

We tend to use IntelliJ as our Java IDE of choice. This is a guide for running Stroom in IntelliJ for the purposes of developing/debugging Stroom.

Prerequisites

In order to build/run/debug Stroom you will need the following:

• Java 8 JDK
• Git
• Gradle
• IntelliJ
• Docker CE
• Docker Compose

These instructions assume that all servcies will either run in the IDE or in Docker containers.

Environment variables

Stroom git repositories

To develop stroom you will need to clone/fork multiple git repositories. To quickly clone all of the Stroom repositories you can use the helper script described in stroom-resource (external link).

Database setup

Stroom requires a MySQL database to run. You can either point stroom at a MySQL server or preferably at a MySQL Docker containers.

MySQL in a Docker container

See the section below on stroom-resources.

Host based MySQL server

With an instance of MySQL server 5.5 running on your local machine do the following to create the stroom database:

# log into your MySQL server using your root credentials
mysql --user=root --password=myrootpassword

Then run the following commands in the MySQL shell:

drop database stroom;
create database stroom;
grant all privileges on stroom.* to stroomuser@localhost identified by 'stroompassword1';
quit;

Local configuration file

When running stroom in an IDE it is advisable to have a local configuration file to allow you to change settings locally without affecting the repository. The local configuration file is expected to live at ~/.stroom/stroom.conf. To create a default version of this file run the script stroom.conf.sh from within the root of the stroom git repository.

Add any properties from stroom.properties that you want different values for, e.g.

stroom-resources

As a minimum to develop stroom you will need clones of the stroom and stroom-resources git repositories. stroom-resources provides the docker-compose configuration for running the many docker containers needed.

Having cloned stroom-resources navigate to the directory stroom-resources/bin and run the script

./bounceIt.sh

On first run this will create a default version of the git-ignored file stroom-resources/bin/local.env which is intended for use by developers to configure the docker stacks to run.

This file is used to set a number of environment variables that docker compose will use to configure the various containers. The key environment variable in there is SERVICE_LIST. This is a space delimited list of the services for docker-compose to run. The services are all defined in stroom-resources/bin/compose/everything.yml and its dependencies. By default SERVICE_LIST runs a core stroom stack entirely in docker.

To run stroom in an IDE stroom needs to be removed from SERVICE_LIST, i.e. by commenting out the line SERVICE_LIST="${SERVICE_LIST} stroom" in local.env. Having done this run stroom’s core dependencies as follows:

./bounceIt.sh

Verify the Gradle build

Before trying to run Stroom in an IDE it is worth performing a Gradle build without the integration tests (as these take ~20mins to run) to verify the code compiles and all dependencies are present.

./gradlew clean build -x integrationTest

Sample Data

Some of the tests are dependant on some sample data and content being present in the database. This sample data/content can also be useful for manually testing the application in development. The sample data/content is generated by a class called SetupSampleData.java. This class assumes that the database being used for Stroom is completely empty.

First you need to create a run configuration for SetupSampleData.java

1. Click Run -> Edit Configurations…
2. Click the green + icon to add a new configuration
3. Select Application as the configuration type
4. In the Main Class field enter SetupSampleData
5. In the Use classpath of module field select stroom-integrationtest
6. If you have set the STROOM_TMP environment variable in your .bashrc / .zshrc then ignore this step. In the Environment variables field click the … icon and add STROOM_TMP = ~/tmp/stroom/ (or whatever directory you choose)
7. Click the OK button

Now run SetupSampleData

1. Click Run -> Run…
2. Select SetupSampleData

You should now have a database populated with tables and data, providing you with some predefined feeds, data, translations, pipelines, dashboards, etc.

Running Stroom from the IDE

The user interface for Stroom is built using GWT (see GWT Project for more information or GWT specific documentation). As a result Stroom needs to be started up with GWT Dev Mode. Dev Mode handles the compilation of the Java user interface source into JavaScript and the source map that links client JavaScript back to Java source for client side debugging.

Stroom is run from the main method in Startup.java. Before running this you need to setup the run configuration:

1. Click Run -> Edit Configurations…
2. Click the green + icon to add a new configuration
3. Select Application as the configuration type
4. In the Main class field enter Startup
5. In the Programme arguments field enter

-startupUrl stroom.jsp -logLevel INFO -war . -logdir . -gen . -extra . -workDir . stroom.app.AppSuperDevMode

1. In the Use classpath of module field select stroom-startup
2. If you have set the STROOM_TMP environment variable in your .bashrc / .zshrc then ignore this step. In the Environment variables field click the … icon and add STROOM_TMP = ~/tmp/stroom/ (or whatever directory you choose)
3. Click the OK button

Now run Startup

1. Click Run -> Run…
2. Select Startup

You should eventually see the GWT Dev Mode window appear.

---

Initially, some of the buttons shown above will not be visible as it is in the process of starting up. As soon as the Launch Default Browser button appears you are ready to open Stroom in a browser. You have two options:

• Click the Launch Default Browser button
• Open your preferred browser and enter the URL http://127.0.0.1:8888

NOTE: Stroom has been written with Google’s Chrome browser in mind so has only been tested on Chrome. Behaviour in other browsers may vary. We would like to improve cross-browser support so please let us know about any browser incompatibilities that you find.

In the browser you will initially see the following:

Starting Stroom

Initialising context…

Once the context has been initialised you will see the Stroom blue background and a spinner while GWT compiles the front-end code. Once the code has been compiled you will be presented with the Stroom login page. Stroom in development mode uses simple username/password authentication. Enter the following credentials:

• Username: admin
• Password: admin

Right click behaviour

Stroom overrides the defualt right click behaviour in the browser with its own context menu. For UI development it is often required to have access to the browser’s context menu for example to inspect elements. To enable the browser’s context menu you need to uncomment the following property in your stroom.conf file, e.g.

#Uncomment this to enable browser's right click menu for development
stroom.ui.oncontextmenu=

3 - Developing Content

4 - Documenting Stroom

This site is built using Hugo with the Docsy Hugo theme. The content is pre-dominantly authored in Markdown with some Hugo shortcodes.

Useful external links

• Hugo static site generator.
• Docsy theme for Hugo.
• Commonmark Markdown cheat sheet and tutorial.

4.1 - Building the Documentation

Prerequisites

In order to build and contribute to the documentation you will need the following installed:

• bash
• Docker (external link)

Docker is required as all the build steps are performed in docker containers to ensure a consistent and known build environment. It also ensures that the local build environment matches that used in GitHub actions.

It is possible to build the docs without docker but you would need to install all the other dependencies that are provided in the docker images, e.g. java, plantuml, puppeteer, hugo, npm, html2canvas, jspdf, graphviz etc.

It is assumed that you have a reasonable understanding of how to use Git and Github, specifically:

• Github
  • Forks
  • Pull requests
• Git
  • Branching
  • Pulling from upstream remotes

Cloning the stroom-docs git repository

The git repository for this site is stroom-docs (external link). stroom-docs uses the Docsy theme (themes/docsy/) via a Go module so the theme will be pulled in when Hugo is first run.

Unless you are a committer on the gchq/stroom-docs repository, you will need to fork this repository into your own Github account. You can then clone your fork of the repository like so (in this example it is assumed your Github username is jbloggs):

# Clone the repo
git clone https://github.com/jbloggs/stroom-docs.git
(out)Cloning into 'stroom-docs'...
(out)remote: Enumerating objects: 66006, done.
(out)remote: Counting objects: 100% (7916/7916), done.
(out)remote: Compressing objects: 100% (1955/1955), done.
(out)remote: Total 66006 (delta 3984), reused 7417 (delta 3603), pack-reused 58090
(out)Receiving objects: 100% (66006/66006), 286.61 MiB | 7.31 MiB/s, done.
(out)Resolving deltas: 100% (34981/34981), done.

Configuring the fork

You now need to configure the clone of your fork with details of the gchq/stroom-docs upstream, which will allow you to pull changes from it.

git remote add upstream https://github.com/gchq/stroom-docs.git

Checking out the correct branch

There is a version of the documentation for each minor Stroom version (see Stroom Versions.

Before you start editing/viewing the documentation you need to be working from the correct branch of the Git repository. If for example you are adding some content that is applicable to Stroom v7.0 then you need to checkout branch 7.0.

git checkout 7.0

Creating a feature branch

If you are making changes to the documentation then the recommended working practice is to always make changes on a feature branch. A feature branch would typically contain all commits/changes relating to a single feature, e.g. the addition of a new section, or updating the documentation for a specific change in Stroom. Having only these changes on a branch makes it easy to merge them into the release branch (e.g. 7.0) or to just delete the branch if they are no longer needed. A feature branch would typically be merged into the appropriate release branch by creating a pull request in GitHub.

Assuming you have checked out the desired release branch that you would like your changes to ultimately be merged into, do something like the following to create a feature branch (where add-properties-section is the name of your branch):

git checkout -b add-properties-section

This will create the named branch and check it out in one step.

Converting the PlantUML files to SVG

stroom-docs makes used of PlantUML (external link) for a lot of its diagrams. These are stored in the repository as .puml text files. In order that they can be rendered in the site they need to be converted into SVGs first.

To convert all .puml files into sibling .puml.svg files do the following:

./container_build/runInPumlDocker.sh SVG

This command will find all .puml files (in content/ and assets/) and convert each one to SVG. It only needs to be run on first clone of the repo or when .puml files are added/changed. The generated .puml.svg files are ignored by git. This command will be run as part of the GitHub Actions automated build.

Error: Error building site: "/builder/shared/content/en/docs/user-guide/concepts/streams.md:57:1":
failed to render shortcode "image":
failed to process shortcode: "/builder/shared/layouts/shortcodes/image.html:54:21":
execute of template failed: template: shortcodes/image.html:54:21:
executing "shortcodes/image.html" at <$image.Name>: nil pointer evaluating resource.Resource.Name

Running a local server

The documentation can be built and served locally while developing it. To build and serve the site run

./container_build/runInHugoDocker.sh server

This uses Hugo to build the site in memory and then serve it from a local web server. When any source files are changed or added Hugo will detect this and rebuild the site as required, including automatically refreshing the browser page to update the rendered view.

Once the server is running the site is available at localhost:1313/stroom-docs.

Building the site locally

To perform a full build of the static site run:

./container_build/runInHugoDocker.sh build

This will generate all the static content and place it in public/.

Generating the PDF

Every page has a Print entire section link that will display a printable view of that section and its children. In addition to this the GitHub Actions we generate a PDF of the docs section and all its children, i.e. all of the documentation (but not News/Releases or Community) in one PDF. This makes the documentation available for offline use.

To test the PDF generation do:

./container_build/runInPupeteerDocker.sh PDF

Updating the Docsy theme

The Docsy theme is a dependency of this Hugo site. See Update the Hugo Docsy Module for details on how to update the version of the Docsy theme.

In those instructions when it says to run a hugo command you need to do it from within the hugo docker container.

./container_build/runInHugoDocker.sh bash

This will give you a bash prompt inside the container that has the hugo binary available. The container has access to your local stroom-docs repository (from where you ran runInHugoDocker.sh) and the initial directory is the root of the repository. Inside the container the root of the repository is mounted as /builder/shared.

To leave the container’s shell type exit.

Alternatively you can run a single command directly, e.g. to update Docsy to 0.6.0 do:

./container_build/runInHugoDocker.sh 'hugo mod get -u github.com/google/docsy@v0.6.0'

4.2 - Managing Stroom Versions

The Docsy theme supports site versioning so that multiple versions of the site/documentation can exist and link between each other. For this documentation site, each version of the site is tied to a minor release of Stroom, e.g. 7.0, 7.1, 7.2, 8.0 etc. Each Stroom version is represented by a git branch with the same name. There is an additional git branch legacy that is for all Stroom versions prior to 7.0 and will be removed once the versions prior to 7.0 are no longer supported. Documentation changes for an as yet unreleased Stroom version would be performed on the master branch.

When the combined site is built, each version will exist within a directory as siblings of each other, i.e.

/7.0/
/7.1/
/7.2/
/8.0/
/legacy/

Versioned Site Configuration

To configure each version of the site so that it knows what version it is and what the other versions are you need to edit config.toml. This needs to be done on each branch in a way that is appropriate to each branch. If a change needs to be applied to all branches then it is best to make it in the oldest branch for which the documentation is published and then merged the changes up the chain, e.g. legacy => 7.0 => 7.1 => 7.2 => 8.0 => master.

The following config properties needed to be amended on each branch. This example is based on there being version legacy, 7.0 and 7.1, with 7.1 being the latest.

[params]
  # Menu title if your navbar has a versions selector to access old versions of your site.
  version_menu = "Stroom Version (7.1)"

  # If true, displays a banner on each page warning that it is an old version.
  # Set this to true on each git branch of stroom-docs that is not the latest release branch
  archived_version = false

  # Used in the banner on each archived page.
  # Must match the value in brackets in "version_menu" above
  version = "7.1"

  # A link to latest version of the docs. Used in the "version-banner" partial to
  # point people to the main doc site.
  url_latest_version = "/../7.1"

  # The name of the github branch that this version of the documentation lives on.
  # Used for the github links in the top of the right hand sidebar.
  # Should match the last part of url_latest_version.
  github_branch= "7.1"

  # A set of all the versions that are available.
  [[params.versions]]
    version = "7.1"
    url = "/../7.1"
  [[params.versions]]
    version = "7.0"
    url = "/../7.0"
  [[params.versions]]
    version = "Legacy"
    url = "/../legacy"

Automated build process

The site is built by Gihub Actions on a nightly basis. This schedule is controlled by build_and_release.yml (external link) on the master branch.

This automated build will look for any branches matching the pattern (legacy|[0-9]+\.[0-9]+) and for each one will do the following:

• Checkout that branch
• Build the site for that version using Hugo
  • Add the site files to a combined site
  • Generate the documenation PDF
• Build the site with no other versions configured
  • Create a zip of the single version site

Once each site has been processed it will:

• Create a single zip file containing the combined site
• Tag the release with a version number
• Add the following release artefacts:
  • Single version site zips
  • Combined site zip
  • Single version PDFs
• Create a root index.hml file that will redirect to the latest version sub-directory.
• Publish the combined site to GitHub Pages https://gchq.github.io/stroom-docs (external link).

Although the build is run on the master branch it will use the HEAD commit of each of the release branches to build the site(s).

The build and release can be forced by adding the text [publish] to the commit message on master. This is useful in testing, or if updated documentation is needed for any reason.

Where to make changes

The nature of a change to the site/documentation will determine which git branch the change is made on.

Changes specific to a Stroom version

Any changes that are specific to a Stroom version, e.g. documenting a new feature in that version should be made on the oldest branch that contains that feature.

Changes to the News/Releases

Adding news items or release notes for new versions should be done on the latest release branch.

Changing the site look

Ideally changes to the look of the site, e.g. upgrading the Docsy theme sub-module to a new commit, should be done on all branches so when switching between branches the look doesn’t change. This means changes to the site framework should be done on the oldest published version branch and then merged up the chain to the others, e.g. legacy => 7.0 => 7.1 => master.

In some cases a change to the look may require significant refactoring of the content, e.g. changes to a shortcode. In the event of this it may be necessary to only make the change on the latest release branch and for different versions to have a slightly different look. The decision on how best to tackle these situations will have to be on a case by case basis.

Building a mock multi-version site

To make it easier to test how the combined site will look with multiple versions the following script can be run to mock up a multi-version site. It does the following:

1. Copies the content of the local repository.
2. Amends the config file to set appropriate versions.
3. Builds the site for that version.
4. Copies the built site into a sub-directory matching the version in /tmp/stroom-docs_mock_combined_site/.

To run this script do:

create_mock_combined_site.sh

The combined site can be served using something like the Python simple HTTP server, e.g.

cd /tmp/stroom-docs_mock_combined_site
python -m SimpleHTTPServer 8888

Then open a browser at localhost:8888.

As each version of the site is a copy of the same thing the content will be all the same but it allows you to test the version drop down and archived banner.

4.3 - Documentation Style Guide

Overview

Stroom’s documentation is created using the static site generator Hugo (external link). This converts markdown content into a rich HTML site. The markdown content in stroom-docs is not intended to be read as-is in GitHub, it needs to be rendered first.

The full documentation for Hugo can be found here (external link). The site also uses the Docsy (external link) theme for Hugo. The documentation for Docsy can be found here (external link). The Docsy theme provides a lot of the styling but also adds other features and shortcodes. You should consult the Docsy documentation in the first instance.

To maintain a degree of consistency in the documentation you should use this section as a reference for how to layout your content.

Shortcodes

The documentation makes heavy use of Hugo shortcodes for adding page elements such as links, icons, images, etc. Shortcodes make it easy to change how a page element is styled by just changing the shortcode.

Hugo includes many shortcodes (external link), the Docsy theme adds some more (external link) and there are some bespoke stroom-docs ones in layouts/shortcodes/.

To make your life easier when editing the documentation it is recomended to use an editor that supports text snippets. Snippets make it very quick to add shortcodes into the documentation.

For example the following snippet adds a skeleton front matter to a page.

snippet hfront "Hugo markdown metadata front matter" b
---
title: "${1:Title}"
linkTitle: "$1"
#weight:
date: `date "+%Y-%m-%d"`
tags: $2
description: >
  $3
---

$0
endsnippet

{
  "hugo-front-matter": {
    "prefix": "hfront",
    "body": [
      "---",
      "title: \"${1:Title}\"",
      "linkTitle: \"$1\"",
      "#weight:",
      "date: `date \"+%Y-%m-%d\"`",
      "tags: $2",
      "description: >",
      "  $3",
      "---",
      "",
      "$0"
    ],
    "description": "Hugo markdown metadata front matter"
  }
}

4.3.1 - Site Structure

File names

Ideally files and directories should be named using lower-kebab-case, e.g. site-structure.md.

Directory structure

All page content, i.e. markdown, is located underneath content/en. This directory has one sub-directory for each of the top nav bar items.

Stroom-Docs top level sections

Each of the following sections can have a different styling that is appropriate to its content, e.g. documentation vs blog.

Landing Page

The landing page is located at content/en/_index.html. This is the page that users will initially see, unless visiting via a direct link.

About (about)

A single page describing what stroom is.

Documentation (docs)

This is where all the documentation for installing, administering and using stroom is located.

News/Releases (news)

This is a blog type section that contains a page for each new Stroom release and a set of blog posts for Stroom news items. The pages in the two sub-sections (news and releases) are displayed in chronalogical order based on the date key in the page’s front matter.

News (news/news)

This sub section has a flat structure with one .md file per news item. Each file should be named in the form YYYYMMDD-<name>.md. The date key should be set in the front matter to match the date of the file. Hugo will use this date key to order the files in the menu. The date should be set in ISO 8601 date format, i.e.

date: 2021-07-09

Releases (news/releases)

Each new minor version release of Stroom should have a file in this directory. They should be named in the form vXX.YY.md where XX is the zero padded major version and YY is the zero padded minor version, e.g. v07.01. The zero padding is to ensure correct ordering by default withouth having to resort to using weight in the front matter.

The front matter should be set along these lines:

---
title: "Version 7.0"
linkTitle: "7.0"
date: 2021-07-07
description: >
  Key new features and changes present in v7.0 of Stroom and Stroom-Proxy.
---

Community (community)

This section provides information for people wanting to contribute to the development of Stroom and its peripheral repositories. This can include developer documentation for building and developing Stroom. This has the same structure as docs.

Documentation content

The docs and community top level sections have a tree structure for their content. Each of these directories will contain three different types of entities:

• Section directories
• Section index files (_index.md)
• Pages (e.g. building.md)

The following is an example of part of the structure of the community section.

├── documentation/                 Documentation section directory
│   ├── building.md                Page
│   ├── _index.md                  Documentation section index page
│   ├── style-guide/               Style guide section directory
│   │   ├── basic-elements.md      Page
│   │   ├── icon-gallery.md        Page     
│   │   ├── _index.md              Style guide section index page
│   │   ├── site-structure.md      Page
│   │   └── using-images.md        Page
│   └── versions.md                Page
└── roadmap.md                     Page

A section can essentially contain branches (sections) and leaves (pages). A branch (i.e. a section) is defined by a directory that contains an _index.md file. The front matter in this index file defines the meta data for that section, e.g. the title, date, tags, description, etc. A leaf (i.e. a page with no children) is just a markdown file with front matter. The front matter for branches and leaves works in the same way.

How do I…?

Add a child page

If you already have a section that you want to add a new child page to then you will already have a structure like this:

└── my-section/
    ├── _index.md
    ├── a-page.md
    └── another-page.md

To add a new page simple create a .md file for the new page in the section directory, e.g.

└── my-section/
    ├── _index.md
    ├── a-page.md
    ├── another-page.md
    └── new-page.md

Add the front matter to the top of the new page file. e.g.

---
title: "My Long Wordy Title"
linkTitle: "My Short Title"
weight: 20
date: 2022-01-27
tags: 
description: >
  A new page describing stuff.
---

The new page should now appear in the list of child pages on the section page and in the left hand navigation pane.

If you want to control the position of the new page relative to its siblings then adjust the weight of this page and that of its siblings to get the order that you want.

Add a new section

If you want to add a sub-section to an existin section then you will already have a structure like this:

└── my-section/
    ├── _index.md
    ├── a-page.md
    └── another-page.md

To add a new sub-section create a sub-directory within the existing directory (with a name that roughly matches the section name) and create an _index.md file within it, e.g.

└── my-section/
    ├── my-new-sub-section/
    │   └── _index.md
    ├── _index.md
    ├── a-page.md
    └── another-page.md

In the _index.md file add front matter to the top, e.g.

---
title: "My Long Wordy Section Title"
linkTitle: "My Short Section Title"
weight: 20
date: 2022-01-27
tags: 
description: >
  A new section for stuff.
---

As with pages, adjust the weight of the section and its siblings to get the order you require.

To add pages to this new section see Add a new child page above.

If you are not sure if you need to add a whole new sub-section or just a child page you can just add a new sub-section and add content to the _index.md file as if it were a simple page. If you later realise that you need child pages then move the content out into a child page and create other sibling pages to it.

4.3.2 - Front Matter

Front matter in Hugo is a set of meta data at the top of each page that controls which menus include the page as well as providing information about the page, e.g.

---
title: "Site Structure"
linkTitle: "Site Structure"
weight: 10
date: 2021-07-20
tags:
  - style
  - TODO
description: >
  Describes the file and directory structure for the site.
---

Hugo supports front matter in YAML, JSON and TOML, however for consistency all front matter in stroom-docs should be in YAML format.

Title and link title

This is the section/page title and will become the h1 heading (in HTML/Markdown terms) on the section/page. The linkTitle is the text that is displayed in the left hand navigation sidebar. It should be the same as title unless the title is quite long, in which case a shorter version should be used so it fits in the sidebar.

Weight

The weight controls the position of the page/section within the other pages/sections of its parent. i.e. it controls the order of the pages/sections in the left hand navigation bar and the list of child items on a section index. If no weight is provided then Hugo will use date, then linkTitle, then the file path.

To assist with re-ordering pages you can use the script change_weights.sh in the root of the repo. E.g. to change the order of the child items of the user-guide section do the following:

./change_weight.sh content/en/docs/user-guide

Date

The date should be set/updated to the current date when a page/section is created or modified. This data is show at the bottom of the page and tells the reader when the page was last updated.

Tags

In the front matter, tags is a list of tags that are applicable to the document and allow documents to be searched for by tag. Tag names should conform to the following conventions:

• Lower kebab case, i.e. reference-data, even for abbreviations.
• The only exception to the case rule is TODO, which is always upper case.
• Singular, i.e. pipeline rather than pipelines.

Avoid using too many unique tag names as it will make the list of tags in the sidebar to large to be useful. When setting a tag on a document consult the list of existing tags to ensure consistency and to see if a more applicable tag already exists.

Add the TODO tag to a page when the page is incomplete. This makes it easy to find areas of the documentation that are in need of attention.

Cascading tags

If you want to apply a tag to all descendant pages of a section you can add this to the front matter of the section:

cascade:
  tags:
    - install

This will then apply all of tags added to tags to each descendant page.

Description

A short description of what the page/section covers. This will be shown on the index page of its parent.

4.3.3 - Markdown Style Conventions

Line breaks

Sentence per line

Each sentence must start on a new line, even in numbered/bulleted lists. This makes it easier to move sentences around or to remove them and limits the scope of changes when it comes to git diffs and merges. When the markdown is rendered into HTML/PDF, the sentences will be joined into a single paragraph.

See this site for more of the reasons behind sentence per line. Though this link relates to Asciidoc, the same logic applies to markdown.

For example:

This is the first sentence of the paragraph.
This is the second.
This it the third and final one.

This is the start of a new paragraph.

This is the first sentence of the paragraph. This is the second. This it the third and final one.

This is the start of a new paragraph.

No hard line breaks.

Long lines should not be hard wrapped by adding line breaks. You should instead rely on your editor to soft wrap long lines that cannot fit on the visible screen area. The process of hard wrapping long lines will vary from editor to editor and not all editors support re-wrapping lines after the content has changed. It also relies on each person’s editor being configured to the same wrap column. Adding hard wraps also means a slight change at the start of a paragraph will potentially cause all subsequent lines to be re-wrapped and thus appear as a substantial difference in the commit.

## Don't do this

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum.


## Do this instead

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Forced line breaks

In some circumstances, e.g. a list of items that is not bulleted, you may want to prevent the joining of adjacent lines when rendered. You can force a line break by adding two spaces ␣␣ at the end of a line.

Paragraph 1.

One␣␣
Two␣␣
Three

Paragraph 2.

Paragraph 1.

One
Two
Three

Paragraph 2.

Blank lines and spacing

• A heading line should be preceded by two blank lines and followed by one blank line. This makes the headings clearer in the markdown source.
• A fenced code block should be surrounded by one blank line.
• Paragraphs should be separated by one blank line.
• Bulleted and numbered lists should be surrounded by one blank line.
• Additional sentences in bulleted/numbered lists should be indented for clarity in the raw markdown.

e.g:

The text belonging to the previous heading.


## A Heading


## A sub heading

The text of this heading.
A second sentence in this paragraph.

A new paragraph.
A second sentence in this paragraph.

Here are some bullets:

* Bullet Z.
* Bullet A.
  This is an additional sentence for this bullet point.
  And so is this.
  * Sub-bullet AX.
    This is an additional sentence for this bullet point.
    And so is this.
* Bullet F.

Here are some numbered steps:

1. Step 1.
1. Step 2.
1. Step 3.

Another random line.

Headings

The page title uses heading level one (# in markdown) so all markdown headings should be >= 2 (## , ### , #### , etc.). Headings should have two blank lines above them for clarity in the raw markdown. The # characters should always be followed by one space character

The following is an example of the heading levels.

# Heading level 1

DON'T use this level in your documents.
Level one headings will be generated from the `title` in the document's front matter.


## Heading level 2


### Heading level 3


#### Heading level 4


##### Heading level 5


###### Heading level 6

Markdown supports an alternate style for headings, as shown below. Don’t use this style as it is not clear from the symbols what the heading level is.

Heading level 1
===============

Don't use this style.


Heading level 2
---------------

Don't use this style.

Table of contents

The page table of contents (right hand pane) is controlled by this in config.toml.

[markup]
  [markup.tableOfContents]
    endLevel = 4
    ordered = false
    startLevel = 2

The maximum depth of the table of contents can be controlled with endLevel.

Heading example (level 2)

This is an example of a level 2 heading.

Heading example (level 3)

This is an example of a level 3 heading.

Heading example (level 4)

This is an example of a level 4 heading.

Heading example (level 5)

This is an example of a level 5 heading.

Heading example (level 6)

This is an example of a level 6 heading.

Block quotes

Single line

A simple paragraph block quote.

> This is a simple block quote.
> This is the second sentence on the same line.

This is a simple block quote. This is the second sentence on the same line.

Multi line

A pair of spaces at the end of a line can be used to force line breaks, e.g.:

> This is a multi line block quote.␣␣
> This is the second line.␣␣
> This is the third line.

This is a multi line block quote.
This is the second line.
This is the third line.

Lists

Bulleted list

* Fruit.
  Make sure you get your five-a-day.
* Meat
    * Beef
    * Chicken
* Vegetables.

Numbered List

Numbered list items should all be numbered with number 1 so that the markdown render handles the consecutive numbering. This makes the file easier to edit and means the addition of one item in the middle does not cause a change to all the lines after it, e.g:

1. Item one.
   This is some extra content for step 1.
1. Item two.
    1. Sub-item A.
    1. Sub-item B.
1. Item three.

Check List

* [ ] Item 1.
      This is some extra content for item 1.
* [ ] Item 2.
    * [x] Item 2a.
    * [ ] Item 2b.
* [x] Item 3.

Definition list

Name
: Godzilla

Birthplace
: Japan

Color
: Green

Tables

Tables should ideally have its columns aligned in the markdown for clarity in the raw markdown.

## Ideally do this

| Artist          | Album          | Year |
|-----------------|----------------|------|
| Michael Jackson | Thriller       | 1982 |
| Prince          | Purple Rain    | 1984 |
| Beastie Boys    | License to Ill | 1986 |

## But this is acceptable

| Artist | Album | Year |
|-|-|-|
| Michael Jackson | Thriller | 1982 |
| Prince | Purple Rain | 1984 |
| Beastie Boys | License to Ill | 1986 |

Both will produce the same result but the latter can be harder to read in markdown form.

4.3.4 - Additional Page Elements

Links

Links can be added using either standard markdown link syntax or using a Hugo shortcode. The advantage of the shortcode is that hugo will check for broken links when building the site so there are preferred.

Links to external sites, i.e. on the internet, should have (external link) appended to the link title. This makes it clear to readers which links are local and which are external and therefore possibly not available if there is no access to the internet.

Anchors

You can link to headings on a page using its anchor. The anchor for a heading is the heading text with:

• All non-alphanumeric characters removed
• Spaces replaced with a -
• All characters made lower case
• Multiple consecutive - characters, e.g. --- are replaced with a single -

For example the heading Mr O'Neil's 1st Event (something) becomes as an anchor #mr-oneils-1st-event-something.

Shortcode page link examples

Shortcode links are slightly more verbose to type but are preferable to markdown style links as the link target will be checked at site build time so you know all the links are correct.

Hugo has ref and relref link shortcodes. Use only relref as this will be translated into a relative path when the site it built ensuring that the site can be run from any base path.

The following are some example of different links to internal content.

Heading anchor

• A link to a heading anchor on this page.

  [link]({{< relref "#alerts" >}})
  

Absolute path

• A link to a heading anchor on another page, using an absolute path.

  [link]({{< relref "community/documentation/style-guide/using-images#captions" >}})
  

Relative path

• A link to a heading anchor on page above this one, using a relative path.

  [link]({{< relref "../../../docs/proxy/install.md#prerequisites" >}})
  

Unique page name

• A link to a heading anchor on another page, using only the unique page name. This is quicker to type and won’t break if pages are moved but will not work if the page name is not unique.

  [link]({{< relref "running#basic-configuration" >}})
  

Section link

• A link to a branch/section in the document tree, i.e. to the \_index.md.

  [link]({{< relref "docs/HOWTOs" >}})
  

Next page in section link

• A link to the next page in the current section. If the current page is the last page in the section then no link will be displayed. What the next page is is defined by the page weights or the default ordering of pages in the section.

  This is useful when you have a set of pages in a section that have a natural flow, e.g. where the pages in a section are the sequential steps in an installation guide.

  The link looks like this(with the page title in the link text and the hover tip showing both the page title and the description):

  Next page - Images

  {{< next-page >}}
  

Previous page in section link

• A link to the previous page in the current section. If the current page is the last page in the section then no link will be displayed. What the previous page is is defined by the page weights or the default ordering of pages in the section.

  This is useful when you have a set of pages in a section that have a natural flow, e.g. where the pages in a section are the sequential steps in an installation guide.

  The link looks like this(with the page title in the link text and the hover tip showing both the page title and the description):

  Previous page - Markdown Style Conventions

  {{< prev-page >}}
  

Markdown page link examples

Markdown style links should not contain the .md extension as this will be stripped when the site is generated, e.g. for the following content:

/content
   /en
      /docs
         /section-x
            /sub-section-a
               _index.md
               page1.md
               page2.md
            /sub-section-b
               _index.md
               page1.md
               page2.md

This will become:

/docs
   /section-x
      /sub-section-a
         /page1
         /page2
      /sub-section-b
         /page1
         /page2

in the rendered site.

• A link to a heading anchor on this page.

  [link](#alerts)
  

• A link to a heading anchor on another page, using a relative link.

  [link](../using-images#captions)
  

• A link to a heading anchor on another page, using an absolute link.

  [link](/docs/style-guide/using-images#captions)
  

Download file links

To create a link to download a file, like this , that is served by this site you need to do:

{{< file-link "quick-start-guide/mock_stroom_data.csv" >}}Link Title{{< /file-link >}}

Paths are relative to /assets/files/.

Code

Inline code

Inline code `looks like this`.

Code blocks (simple)

Code blocks should be surrounded with fences ```language-type and ``` with the language type always specified to ensure correct syntax highlighting. If the language type is not supplied then styling will be different to fenced blocks with a language.

This is a markdown example of a fenced code block containing XML content.

<root>
  <child attr="xxx">some val</child>
</root>

```xml
<root>
  <child attr="xxx">some val</child>
</root>
```

The following are some example of rendered code blocks:

id,date,time
1,6/2/2018,10:18
2,12/6/2017,5:58
3,6/7/2018,11:58

---
root:
  someKey: "value"

<root>
  <child attr="xxx">
    some val
  </child>
</root>

echo "${VAR}"

code blocks (advanced)

If you need to show line numbers or to highlight sections of a code block then you can use the code-block shortcode. This shortcode takes the following named arguments:

• language - The language for syntax highlighting. Defaults to text.
• lines - Whether to display line numbers or not (true|false). Defaults to true.
• highlight - A comma separate list of lines to highlight. Ranges of line numbers can be specified, e.g. 2-8,13,25-30. Defaults to no line highlighting.
• start - The line number to start at. Defaults to 1.

The following is an example of how to use the shortcode.

{{< code-block language="xml" highlight="2-5,8" >}}
<root>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
</root>
{{< /code-block >}}

Default behaviour

{{< code-block >}}
...
{{< /code-block >}}

<root>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
</root>

With line numbers, a language and a starting line number

{{< code-block language="xml" start="5" >}}
...
{{< /code-block >}}

  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
</root>

With line numbers, a language and highlighted lines

{{< code-block language="xml" highlight="2-5,8" >}}
...
{{< /code-block >}}

<root>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
</root>

With a language and highlighted lines, but without line numbers

{{< code-block language="xml" lines="false" highlight="2-5,8" >}}
...
{{< /code-block >}}

<root>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
  <child attr="xxx">
    some val
  </child>
</root>

Command line blocks

To demonstrate commands being run on the command line you can use the command-line shortcode. This results in a code block with the shell prompt displayed on the left of each line. It also means the prompt part is not selectable when the user selects the text for copy/pasting.

The shortcode takes the following positional arguments:

1. username - The username to appear in the prompt, defaults to user.
2. hostname - The hostname to appear in the prompt, defaults to localhost.
3. language - The language of the content (a valid and installed prism language name language name), defaults to bash.

If you want to display shell output then prefix each output line with (out). It will then be displayed without a prompt.

echo "hello world"
(out)hello world
id
(out)uid=1000(david) gid=1000(david)

{{< command-line "david" "wopr" >}}
echo "hello world"
(out)class
id
(out)uid=1000(david) gid=1000(david)
{{< command-line >}}

Inline files

Some code or text examples may be too large for a fenced block so you can put the content in a separate file and include it in-line like so.

<?xml version="1.1" encoding="UTF-8"?>
<dataSplitter
    xmlns="data-splitter:3"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="data-splitter:3 file://data-splitter-v3.0.1.xsd"
    version="3.0">
  <!-- 
  GEOHOST REFERENCE FEED:
  
  CHANGE HISTORY
  v1.0.0 - 2020-02-09 John Doe
  
  This is a reference feed for device Logical and Geographic data.
  
  The feed provides for each device
  * the device FQDN
  * the device IP Address
  * the device Country location (using ISO 3166-1 alpha-3 codes)
  * the device Site location
  * the device Building location
  * the device Room location
  *the device TimeZone location (both standard then daylight timezone offsets from UTC)
  
  The data is a TAB delimited file with the first line providing headings.
  
  Example data:
  
  FQDN	IPAddress	Country	Site	Building	Room	TimeZones
stroomnode00.strmdev00.org	192.168.2.245	GBR	Bristol-S00	GZero	R00	+00:00/+01:00
stroomnode01.strmdev01.org	192.168.3.117	AUS	Sydney-S04	R6	5-134	+10:00/+11:00
host01.company4.org	192.168.4.220	USA	LosAngeles-S19	ILM	C5-54-2	-08:00/-07:00
  
  -->
  
  <!-- Match the heading line - split on newline and match a maximum of one line  -->
  <split delimiter="\n" maxMatch="1">
   
   <!-- Store each heading and note we split fields on the TAB (&#9;) character -->
     <group>
       <split delimiter="&#9;">
         <var id="heading"/>
       </split>
     </group>
   </split>
   
  !-- Match all other data lines - splitting on newline -->
  <split delimiter="\n">
    <group>
      <!-- Store each field using the column heading number for each column ($heading$1) and note we split fields on the TAB (&#9;) character -->
       <split delimiter="&#9;">
         <data name="$heading$1" value="$1"/>
       </split>
    </group>
  </split>
</dataSplitter>

The card has a maximum height and will show scrollbars as required.

Examples of how to use in-line files are:

{{< textfile "style-guide/in_line_file_example.xml" "xml" >}}My caption{{< /textfile >}}
{{< textfile "style-guide/in_line_file_example.xml" >}}My caption{{< /textfile >}}
{{< textfile "style-guide/in_line_file_example.xml" />}}

Supported languages

This site uses Prismjs for syntax highlighing code blocks. PrismJs supports a large number of different languages however only certain languages have been included with this site. The language specified in the markdown code fence or in the command-line shortcode must be in the list of included languages.

The list of included language names are:

sh
bash
css
csv
groovy
http
java
javascript
jq
json
python
regex
scss
sql
text
toml
xml
yaml

To include extra languages in this site you need to build a new version of the prism.js and prism.css files. This can be done here . When creating new versions of these file you must included the languages and plugins already included else you may break this site. The generated files are then copied over the top of /static/css/prism.css and /static/js/prism.js. Both files include a comment at the top with the link to the PrismJs download page with the currently included items selected. Use this link then add any additional items, bearing in mind the size of the download and its impact on page load times.

An example of the download link is https://prismjs.com/download.html#themes=prism&languages=markup+css+clike+javascript+bash+csv+groovy+java+jq+json+markdown+python+regex+scss+sql+toml+yaml&plugins=line-highlight+line-numbers+command-line+toolbar+copy-to-clipboard+treeview */

Alerts

Warning block Quote

The markdown for this is:

{{% warning %}}
This is a warning that can contain _markdown_.
{{% /warning %}}

Page level warning

The markdown for this is:

{{% page-warning %}}
This is a warning that can contain _markdown_.
{{% /page-warning %}}

Note block Quote

The markdown for this is:

{{% note %}}
This is a note that can contain **markdown**.
{{% /note %}}

Page level info

The markdown for this is:

{{% pageinfo %}}
This is some info that can contain **markdown**.
{{% /pageinfo %}}

TODO block Quote

The markdown for this is:

{{% todo %}}
This is a TODO that can contain `markdown`.
{{% /todo %}}

Cards

Cards can be used to display related content side by side. Each card can contain markdown and/or Hugo short codes. The cards will be displayed horizontally to fill the width of the page.

---
root:
  someKey: "value"

<root>
  <child attr="xxx">some val</child>
</root>

The markdown for this is:

{{< cardpane >}}
  {{< card header="YAML" >}}
```yaml
---
root:
  someKey: "value"
```
  {{< /card >}}
  {{< card header="XML" >}}
```xml
<root>
  <child attr="xxx">some val</child>
</root>
```
  {{< /card >}}
{{< /cardpane >}}

Tabbed panes

Hugo/Docsy have shortcodes for tabbed panes however these mean only one tab will be printed or visible in the generated PDF so there use should be avoided.

4.3.5 - Images

Image assets

All Images should be placed in /assets/images/ or within a descendent directory of it. Images should be added to the markdown using the image shortcode. This ensures a consistent look for all images and allows control of the size of the image.

The short code can be added in the following ways.

{{< image "style-guide/stroom-oo.svg" "200x" >}}

{{< image "style-guide/stroom-oo.png" >}}

{{< image "style-guide/stroom-oo.svg" "200x" >}}Image caption{{< /image >}}

All paths used in the image shortcode are relative to /assets/images.

The following shows an example of the directory structure.

/assets
   /images
      /section-x
         /sub-section-y
            /page1
               /image.svg
/content
   /en
      /docs
         /section-x
            /sub-section-y
               page1.md - #image shortcode used in here

Where images are only used by one page then the directory structure in /assets ought to mirror that of the page the images are used in, as shown above. If images are used by multiple pages then some other sensible organisation of the images should be used, e.g. /assets/images/section-x if they are limited to pages in one section, or /aaets/images/common if they are used in multiple sections.

Captions

The image can be displayed with a caption:

---

Size

The image can be defined with a maximum width (e.g. "300x") or a maximum height ("x200").

.png files

.png files can be rendered in their original size by omitting the size argument.

Or they can be rendered with a set size.

Using global /assets/ resources

For images that are shared by multiple page bundles, e.g. stroom icons, place them in /assets/images/. The image path is relative to /assets/images/, e.g. file /assets/images/style-guide/svg-example.svg becomes:

{{< image "style-guide/svg-example.svg" "200x" >}}

---

PlantUML

PlantUML is the favoured tool for producing diagrams such as UML diagrams, entity relationship diagrams and other more general architecture diagrams. The diagrams are written in plain text which makes them easy to version control. The plantUML syntax can then be converted into image files, e.g. .svg files.

A plantUML diagram looks like this in plain text form:

@startuml
Alice -> Bob: Authentication Request
Bob --> Alice: Authentication Response

Alice -> Bob: Another authentication Request
Alice <-- Bob: Another authentication Response
@enduml

PlantUML diagrams should be saved in a dedicated .puml file in the same folder as the page that will display it. If it is needed by multiple pages then it should be saved in /assets/images/.

The build process will convert all .puml files to a corresponding .puml.svg file. These generated .puml.svg files are ignored by git so need to be regenerated at build time or when you change a .puml file. The images can be generated by doing:

./container_build/runInPumlDocker.sh SVG

You should embed an PlantUML image like this, using the .puml.svg file (that may not yet exist, but will have to exist before the site is built):

Using page resources

Images can be located in a page bundle (external link). This is where the page is defined as a named directory (rather than a .md file) with an associated index.md file for the markdown contnet. All other items in the directory are page resources that can be used by the page, i.e. image files.

Use the image short code to display an image file that is located in the same directory as the page. For the short code to work the page must be a leaf bundle (index.md) or a branch bundle (_index.md), i.e:

/docs
   /MyPage
      index.md - #image shortcode used in here
      example.svg

or

/docs
   /MySection
      /Page1
         index.md
      /Page2
         index.md
      _index.md - #image shortcode used in here
      example.svg

In the above example, the shortcode would look like:

{{< image "example.svg" "200x" >}}

Stroom user interface Components

Sometimes it is useful to display an image of certain user interface elements to explain something. Rather than use screenshots which are very difficult to keep up to date, you can instead use some simple shortcodes to display some UI elements, e.g. buttons, pipeline elements, etc. By using shortcodes, any change to the look of the Stroom UI means only the shortcode content and their styling need to change, without having to recreate tens or hundreds of screenshots.

Stroom icons

Stroom UI icons such as or can be added in line using the shortcode stroom-icon.

Arguments:

• icon_file_path - The filename (and path) of the icon file. The path is relative to /assets/images/stroom-ui/.
• title (optional) - The hover tip title that will be given to the icon.
• state (optional) [enabled|disabled] - Whether the button is enabled or disabled. Defaults to enabled.

The full list of available icons can be found in the Icon Gallery.

E.g:

Simple use: {{< stroom-icon "save.svg" >}}
Custom name: {{< stroom-icon "key.svg" "Lock" >}}
Disabled document icon: {{< stroom-icon "document/Feed.svg" "Feed" "disabled" >}}

Buttons

To display a UI button you can use the shortcode stroom-btn with the button title as its only argument.

Arguments:

• title - The text to display on the button.

{{< stroom-btn "Ok" >}}
and
{{< stroom-btn "Cancel" >}}

Pipeline elements

To display a pipeline element (as seen on the Structure sub-tab on the Pipeline screen), like splitFilter , you can use the shortcode pipe-elm.

Arguments:

• element_name - The name of the pipeline element (case insensitive), e.g. XsltFilter.
• display_name (optional) - The display name for the pipeline element. If a display name is not provided then the lower camel case form of the element name will be used as the display name.

The icon will be selected to match the element name provided.

Rendered

This is an xsltFilter with its default name:

xsltFilter

This is a splitFilter with a custom name:

My Split Filter

Markdown

This is an xsltFilter with its default name:

{{< pipe-elm "xsltFilter" >}}

This is a splitFilter with a custom name:

{{< pipe-elm "splitFilter" "My Split Filter" >}}

Stroom document tabs

To display a top level Stroom document tab, like Big Index × , you can use the shortcode stroom-tab.

Arguments:

• icon_filename - The filename of the icon to use (relative to assets/images/stroom-ui/document/) (case sensitive), e.g. Pipeline.svg.
• title - The text to display in the tab, e.g. Indexing Pipeline.
• state (optional) - Whether the tab is active or not (active or inactive). Defaults to inactive.

For a full list of all available icons see the Icon Gallery

Rendered

Simple:

MY_FEED ×

Custom name:

My Translation ×

Active:

My Translation ×

Unsaved:

* My Translation ×

Markdown

Simple: {{< stroom-tab "Feed.svg" "MY_FEED" >}}
Custom name: {{< stroom-tab "XSLT.svg" "My Translation" >}}
Active: {{< stroom-tab "XSLT.svg" "My Translation" "active" >}}
Unsaved: {{< stroom-tab "XSLT.svg" "My Translation" "active" "unsaved" >}}

Stroom selected menu items

To display a selected menu item you can use the stroom-menu shortcode.

For example to demonstrate creating a folder using the context menu:

New Folder

The markdown for this example is:

{{< stroom-menu "New" "Folder" >}}

The shortcode takes one argument per menu item. The list of available menu items with an icon are in this gallery. Menu item names are case-sensitive. If the menu item name is not in the gallery then it will be displayed without an icon, e.g.:

New Banana

This shortcode can be used in-line like Tools Users this.

4.3.6 - Icon Gallery

This page can be used as a reference for finding icons and their filenames to use them in the documentation.

See here for how to add icons to pages.

General icons

General icons used in Stroom.

Example: {{< stroom-icon "edit.svg" >}}

add-above.svg

add-below.svg

add.svg

alert.svg

clear.svg

close-grey.svg

close.svg

collapse-up.svg

copy.svg

database.svg

delete.svg

deleted.svg

dependencies.svg

disable.svg

down.svg

download.svg

drop-down.svg

edit.svg

ellipses-horizontal.svg

ellipses-vertical.svg

error.svg

expand-down.svg

explorer.svg

fast-backward-green.svg

fast-backward.svg

fast-forward-green.svg

fast-forward.svg

fatal.svg

favourites.svg

feed.svg

field.svg

file-formatted.svg

file-raw.svg

file.svg

filter.svg

folder-tree.svg

folder.svg

format.svg

function.svg

generate.svg

help.svg

hide.svg

history.svg

info-deleted.svg

info-warning.svg

info.svg

jobs.svg

key.svg

lock-green.svg

locked-amber.svg

logo.svg

logout.svg

monitoring.svg

move.svg

nodes.svg

oo.svg

operator.svg

password.svg

pause.svg

play-green.svg

play.svg

process.svg

properties.svg

raw.svg

refresh-green.svg

refresh.svg

remove.svg

ruleset.svg

save.svg

saveas.svg

settings-blue.svg

settings-grey.svg

settings.svg

shard-close.svg

shard-flush.svg

show.svg

spinner.svg

step-backward-green.svg

step-backward.svg

step-forward-green.svg

step-forward.svg

step.svg

stepping.svg

stop-red.svg

stop.svg

table-nested.svg

table.svg

tree-closed.svg

tree-leaf.svg

tree-open.svg

undo.svg

unlock-amber.svg

unlocked-green.svg

up.svg

upload.svg

user-disabled.svg

user.svg

users-disabled.svg

users.svg

volumes.svg

Pipeline element icons

Icons used for the different pipeline elements.

Base path: pipeline
Example: {{< stroom-icon "pipeline/stream.svg" >}}

ElasticSearch.svg

StroomStatsStore.svg

analyticOutput.svg

apache_kafka-icon.svg

file.svg

files.svg

hadoop-elephant-logo.svg

id.svg

index.svg

json.svg

kafka.svg

recordCount.svg

recordOutput.svg

referenceData.svg

search.svg

solr.svg

split.svg

statistics.svg

stream.svg

stroomStats.svg

text.svg

xml.svg

xmlSearch.svg

xsd.svg

xslt.svg

Document type icons

Icons used for the different document entity types, i.e. those seen in the explorer tree.

Base path: document
Example: {{< stroom-icon "document/Index.svg" >}}

AnalyticOutputStore.svg

AnnotationsIndex.svg

Dashboard.svg

Dictionary.svg

ElasticIndex.svg

Feed.svg

Folder.svg

Index.svg

KafkaConfig.svg

Pipeline.svg

ReceiveDataRuleSet.svg

Script.svg

SelectAllOrNone.svg

SolrIndex.svg

StatisticStore.svg

StroomStatsStore.svg

System.svg

TextConverter.svg

Visualisation.svg

XMLSchema.svg

XSLT.svg

searchable.svg

Assorted UI elements

Assorted images used in the user interface.

Base path: assorted
Example: {{< stroom-icon "assorted/popup.png" "Menu selection" >}}

popup.png

Menu items

Menu items with icons that are available for use with the stroom-menu shortcode.

API Keys

Caches

Change password

Close

Close All

Copy

Dashboard

Data Retention

Data Volumes

Database Tables

Delete

Dependencies

Dictionary

Elastic Cluster

Elastic Index

Export

Feed

Folder

Help

Import

Index Volumes

Info

Jobs

Kafka Configuration

Logout

Lucene Index

Move

New

Nodes

Permissions

Pipeline

Preferences

Properties

Rename

Rule Set

Save

Save All

Script

Server Tasks

Solr Index

Statistic Store

Stroom-Stats Store

Text Converter

User Permissions

Users

Visualisation

XML Schema

XSL Translation

Updating this gallery

Icons

This gallery of icons can be updated by running the following script which will copy all the icons from the stroom repository. You should ensure your local stroom repository is up to date and on the correct branch before running this.

./update_stroom_icons.sh <stroom repo root>

e.g.

./update_stroom_icons.sh ../v7stroom/

Menu Items

To update the available menu items edit the shortcode file layouts/shortcodes/stroom-menu.html and modify the icon_map variable..