Class GafferAdder

  • All Implemented Interfaces:
    Serializable

    public class GafferAdder
    extends Object
    implements Serializable

    Helper class to add Elements to a Gaffer store.

    The Flink sink is given a single element at a time. Adding a single element at a time to Gaffer would be really inefficient so we add these individual elements to a blocking queue that we simultaneously add elements to whilst the Gaffer Store is consuming them.

    The queue is a BlockingQueue with a maximum size to prevent the queue from getting too large and running out of memory. If the maximum size is reached Flink will be blocked from adding elements to the queue. The maximum size of the queue can be configured using the operation option: gaffer.flink.operation.handler.max-queue-size. By default the maximum size is 1,000,000. The blocking queue is only blocked on addition, it does not cause the Gaffer Store to block if the queue is empty. In the case where a Kafka queue has just a single Gaffer element the Store can immediately add this rather than blocking and waiting for a batch to fill up. A side affect of this is that AddElements operation may complete if no elements are added to the queue for some time. In this situation we just restart the AddElements operation next time an element is received.

    See Also:
    Serialized Form
    • Field Detail

    • Constructor Detail

      • GafferAdder

        public GafferAdder​(OP operation,
                           Store store)
    • Method Detail

      • initialise

        public void initialise()
      • add

        public void add​(Element element)