Flowpipeline Example Collection

This collection of example configs is supposed to help users get started using different use cases. A grouped and alphabetically sorted reference, might be the best resource when trying to achieve a specific outcome, this short guide however tries to give new users some idea of what is possible with this tool and present existing users with additional options.

The most natural way to group these examples is to list them by which input segment they use, aka "where they get flows from". Note that these input segments can be freely interchanged for one another, and all examples work with all inputs.

bpf

This segment accesses local network interfaces using raw sockets, as for instance tcpdump does.

Relevant examples are:

• ./flowdump/bpf.yml -- create a tcpdump style view with custom filtering from CLI using local interfaces

goflow

This segment allows listening for raw IPFIX, Netflow, or sFlow by using goflow2's API.

Relevant examples are:

• ./localkafka/write.yml -- emulate plain goflow2 and write flows to a Kafka topic for the following section to use

kafkaconsumer

This segment accesses streams of flows generated by another pipeline using kafkaproducer or goflow2.

Relevant examples are:

• ./flowdump/kafkaflowdump.yml -- create a tcpdump style view with custom filtering from CLI
• ./flowdump/highlight.yml -- create a tcpdump style view but use the filtering conditional to highlight desired flows instead of dropping undesired flows
• ./enricher -- enrich flows with various bits of data and store them back in Kafka
• ./reducer -- strip flows of fields and store them back in Kafka
• ./splitter -- distribute flows to multiple Kafka topics based on a field
• ./anonymizer -- anonymize IP addresses using Crypto PAn