Extending Gaffer
Extending Gaffer can mean a few things to a developer, this guide aims to cover general use cases for how and why you may want to extend the capabilities or add customisation to Gaffer.
Writing Custom Classes
As Gaffer is Java based to create and load additional libraries you will first need to write custom classes. Gaffer has many places where custom classes can be used, such as custom operations, aggregation functions, element generators etc.
Depending on what type of class you are writing, e.g. an Operation, you may
need to extend or implement one of the interface classes already in Gaffer.
Assuming you are using Maven for package management you can add Gaffer as
a dependency to gain access to its classes like below:
<dependency>
<groupId>uk.gov.gchq.gaffer</groupId>
<artifactId>${gaffer.component}</artifactId>
<version>${gaffer.version}</version>
</dependency>
Tip
See the components breakdown for a list of key Gaffer modules.
Writing an Operation
In Gaffer we use the term Operation
to describe some component of a query affecting a graph, these are
chained together in OperationChains and this is how we execute
queries to our graphs. Whilst there are a wealth of existing
operations and their associated handlers provided out of the box with
Gaffer. In this section we'll talk about how you might go about adding a new
operation/handler with some simple examples.
The Operation Interface
All Operations must adhere to and implement the Operation
interface, this interface describes the building blocks of an operation.
One important distinction here is that an implementation of Operation should not
contain the actual logic for an Operation, think of these as a task definition
or an instruction manual for what an Operation should do. This then goes hand in
hand with our OperationHandler(s) which define the actual functionality,
either generically or split into multiple handlers for different stores.
Selecting an Interface
In general, most Operation implementations don't actually directly implement the
Operation Interface, they actually implement one or more extended interfaces.
For example the Input and Output interfaces extend on top of the core Operation
interface and concordantly the InputOutput interface is a combination of these.
These all live inside the Gaffer Core library under the Operation subdirectory
(see the component breakdown for
more details).
Implementing an Interface
The example we'll use here is the
ToSet
Operation, this takes an Iterable of Elements and converts it into a
Set, a handy way to remove duplicate Elements if you're doing multiple hops
and end up getting duplicates back.
Taking a look at the Javadoc
for this class you should be able to see this class implements both
InputOutput, with an input type of Iterable<? extends T> and an output type
of Set<? extends T> and MultiInput<T>. Within the class it also implements
all the requisite methods eg, shallowClone(), setInput(), getOptions()
etc.
How to Write the Handler
Using the same ToSet Operation from before, now that we have this definition
of an operation, we need to actually tell Gaffer how to do it. We do this by
defining either a single generic Handler and linking it to the Operation or in
many cases, we might want different implementations based on the store being used
so we'd write a number of operation handlers and decide which one is needed at
runtime.
Here is the ToSetHandler
which implements the appropriate Handler interface (in this case the
OutputOperationHandler). It takes in the ToSet Operation that has been built
with all the options etc. defined and then does the actual doOperation() logic to
stream our Iterable's contents into a LinkedHashSet. This and most other
handlers live inside the Store library.
How does Gaffer know which Handler to use?
Handlers are registered in the Store class via the addOpHandlers() method,
specifically for our example addOperationHandler(ToSet.class, new
ToSetHandler<>()) is called to register it. Once this is done we can then
use our Operation and its Handler in a user defined OperationChain to effect
some query on our graph data.
You can alternatively use the an operations declarations config file (typically
called operationsDeclarations.json) which can be used to define the
Operation and its Handler like below:
{
"operations": [
{
"operation": "uk.gov.gchq.gaffer.operation.impl.output.ToSet",
"handler": {
"class": "uk.gov.gchq.gaffer.store.operation.handler.output.ToSetHandler"
}
}
]
}
Writing an Element/Object Generator
There are two types of generators used, one to transform a domain object into a
Gaffer Element and one to transform the other way. These generators are the
ElementGenerator and ObjectGenerator.
Living within the Core/Data section of the repository these generator interfaces
are quite simple and as with Operation are extended by other interfaces to
provide a more nuanced set of interfaces you could choose to use over the core
Element/Object generator interfaces.
As an example we will walk through the one of the existing generators that can
take a Neo4j CSV file and convert the entries to Gaffer Elements.
/**
* An {@link OpenCypherCsvElementGenerator}s that will generate
* Gaffer Elements from Neo4j CSV strings.
*/
@Since("2.0.0")
@Summary("Generates elements from a Neo4j CSV")
public class Neo4jCsvElementGenerator extends OpenCypherCsvElementGenerator {
@Override
protected LinkedHashMap<String, String> getFields() {
return new Neo4jCsvGenerator().getFields();
}
}
This class may appear very simple but the real functionality for this generator
is in the OpenCypherCsvElementGenerator that it extends. This generator is an
abstract class that implements the ElementGenerator interface and has a lot of
functionality for handling Open Cypher CSVs.
You can view the full class in the repository
which has the logic for converting the CSV format but the key bit to take note
of is the apply(final Iterable<? extends String> strings) method. This method
is the main override which will be passed the lines of a CSV file to convert and
return a Iterable<? extends Element> e.g. a list of Elements.
This pattern is repeated across many of the different generators and can be used as inspiration for your custom generator. As with the Operations so long as you're implementing the appropriate interfaces it's hard to go far wrong.
Writing an Aggregation Function
We can provide a custom aggregation function in a number of ways, most of the
aggregation functionality is provided by the Koryphe library
sat adjacent to Gaffer. If we want to provide a new "generic" aggregator we
would add it in this Koryphe library. For example lets take a look at the very
simple Max comparator, this takes a pair of Java 8 Comparables and finds the
highest value one, this function is applied as an aggregation.
@Since("1.0.0")
@Summary("Calculates the max value")
public class Max extends KorypheBinaryOperator<Comparable> {
@Override
protected Comparable _apply(final Comparable a, final Comparable b) {
return a.compareTo(b) >= 0 ? a : b;
}
}
If we want to add something very specific to a store type or some other
restriction we can add this into the appropriate Store location within Gaffer.
An example of this is the HyperLogLogPlusAggregator in the Sketches library,
this merges HLLPs together.
@Since("1.0.0")
@Summary("Aggregates HyperLogLogPlus objects")
@Deprecated
public class HyperLogLogPlusAggregator extends KorypheBinaryOperator<HyperLogLogPlus> {
@Override
protected HyperLogLogPlus _apply(final HyperLogLogPlus a, final HyperLogLogPlus b) {
try {
a.addAll(b);
} catch (final CardinalityMergeException exception) {
throw new RuntimeException("An Exception occurred when trying to aggregate the HyperLogLogPlus objects", exception);
}
return a;
}
}
You can also add aggregators by loading a JAR into Accumulo and
accessing it on the classpath, this is more something an administrator
customising their usage of the platform might do rather than a developer. For
development purposes it's usually good to check if you can make use of an
existing function in Koryphe first, if not then you can then write your own to
be loaded into the classpath. As long as the new function extends a
KorypheBinaryOperator<T> then it can be used as an aggregation function for
Gaffer.
Loading Custom Libraries
Once you have written your custom classes to make them available the simplest way is to compile to a JAR and load on the Java class path at runtime.
If you are using the container images this is as simple as adding your JAR(s) to a directory in the image, this is explained in detail in the deployment guide.
Created: September 13, 2023