Skip to content

Commit

Permalink
doc: rearrange the entries in a more consumable way
Browse files Browse the repository at this point in the history 
This also changes the order of some sections in index.md.
The idea is to keep this logical order:

- collection
- filtering & tracking
- profiles
- post-processing

Signed-off-by: Paolo Valerio <[email protected]>
  • Loading branch information
@vlrpl @atenart
vlrpl authored and atenart committed Dec 17, 2024
1 parent 64eb7f1 commit b4bff3b
Show file tree
Hide file tree
Showing 2 changed files with 98 additions and 95 deletions.
156 changes: 78 additions & 78 deletions docs/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,66 +64,54 @@ line arguments.

[^1]: Probes for tracking packets are always installed by the core.

## Post-processing
## Filtering

Events stored in a file can be formatted and displayed to the console using the
simple `print` command (the events filename used for the input defaults to
`retis.data`).
Tracing packets can generate a lot of events, some of which are not
interesting. Retis implements a filtering logic to only report
packets matching the filter or being tracked (see [tracking](#tracking)).
Retis has two ways of filtering and both can coexist.
One is based on the [packet content](filtering.md#packet), e.g.:

```none
$ retis print
$ retis collect -f 'arp or tcp port 443'
...
```

But events can also be post-processed. Retis allows to trace packets across the
networking stack and as such the same packet can be seen multiple times (e.g. in
the IP stack, TCP stack, OvS stack & netfilter stack; sometimes multiple times
in each subsystem depending on which probes were loaded). The `sort` command
uses information reported by the `skb-tracking` and the `ovs` collectors to
identify unique packets and group/reorder the events so the same packet can be
efficiently tracked in the stack.
and the other is based on [metadata](filtering.md#metadata), e.g.:

```none
$ retis collect --allow-system-changes -p ip_local_deliver \
--nft-verdicts drop -f 'udp port 8080' -o --print
$ retis collect -m 'sk_buff.dev.nd_net.net.ns.inum == 4026531840'
...
$ retis sort
3316376152002 [swapper/2] 0 [k] ip_local_deliver #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 0
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
+ 3316376220653 [swapper/2] 0 [k] __nft_trace_packet #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 1
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
table firewalld (1) chain filter_IN_FedoraServer (202) handle 215 drop
+ 3316376224687 [swapper/2] 0 [tp] skb:kfree_skb #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 2 drop (NETFILTER_DROP)
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
```

Another post-processing command, `pcap`, can be used to generate `pcap-ng` files
from a set of stored Retis events. For this to work the collection has to be
done using (at least) the `pcap` profile. For now `pcap-ng` files can be
generated by filtering the events using a single probe.
The [filtering](filtering.md) page provides a more detailed
explanation of their respective features, covering aspects such as how
to use different filter types, the specific syntax rules, and examples
of filters.

```none
$ retis -p pcap,generic collect -o
$ retis pcap --probe net:netif_receive_skb | tcpdump -nnr -
$ retis pcap --probe net:net_dev_start_xmit -o retis.pcap
$ wireshark retis.pcap
```
## Tracking

Some post-processing commands (eg. `print`, `sort`) can generate a long output.
In such case a pager is automatically used in case the output is larger than the
current terminal. By default `less` is used but the pager can be explicitly
chosen by setting the `PAGER` environment variable, or unset by setting
`NOPAGER`.
Retis does its best to track packets in the networking stack, and does it in
different ways. Note that tracking packets is not a built-in feature of the
Linux kernel and doing so is complex and cannot be 100% foolproof (but of course
bugs should be reported and fixed).

```none
$ PAGER=more retis sort
$ NOPAGER=1 retis sort
```
1. A Retis core built-in feature generates unique identifiers by tracking the
data part of [socket buffers](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/skbuff.h?h=v6.3#n737).
The socket buffer is also included in the identifier so we can track clones
and friends. This core skb tracking logic is used by the filtering part for
Retis to track packets after they were modified (e.g. NAT). Full details on
the implementation can be found
[in the sources](https://github.com/retis-org/retis/blob/main/retis/src/core/tracking/skb_tracking.rs).

In addition to built-in post-processing commands, it is possible to use the
python bindings to implement custom processing. See the
[python bindings documentation](python.md).
2. A collector, `skb-tracking`, retrieves the core tracking information (unique
identifier and socket buffer address) and reports it in the event. Without
enabling this collector, skb tracking information won't be reported and can't
be used at post-processing time.

3. The `ovs` collector tracks packets in upcalls so we can follow a packet
being sent to the OpenVSwitch user-space daemon, even if it is re-injected
later on.

## Profiles and customization

Expand Down Expand Up @@ -159,51 +147,63 @@ New profiles can be written and used if stored in `/etc/retis/profiles` or
[example profile](https://github.com/retis-org/retis/blob/main/retis/test_data/profiles/example.yaml)
with inlined comments. If a profile is generic enough, consider contributing it!

## Filtering
## Post-processing

Tracing packets can generate a lot of events, some of which are not
interesting. Retis implements a filtering logic to only report
packets matching the filter or being tracked (see [tracking](#tracking)).
Retis has two ways of filtering and both can coexist.
One is based on the [packet content](filtering.md#packet), e.g.:
Events stored in a file can be formatted and displayed to the console using the
simple `print` command (the events filename used for the input defaults to
`retis.data`).

```none
$ retis collect -f 'arp or tcp port 443'
$ retis print
...
```

and the other is based on [metadata](filtering.md#metadata), e.g.:
But events can also be post-processed. Retis allows to trace packets across the
networking stack and as such the same packet can be seen multiple times (e.g. in
the IP stack, TCP stack, OvS stack & netfilter stack; sometimes multiple times
in each subsystem depending on which probes were loaded). The `sort` command
uses information reported by the `skb-tracking` and the `ovs` collectors to
identify unique packets and group/reorder the events so the same packet can be
efficiently tracked in the stack.

```none
$ retis collect -m 'sk_buff.dev.nd_net.net.ns.inum == 4026531840'
$ retis collect --allow-system-changes -p ip_local_deliver \
--nft-verdicts drop -f 'udp port 8080' -o --print
...
```
$ retis sort
The [filtering](filtering.md) page provides a more detailed
explanation of their respective features, covering aspects such as how
to use different filter types, the specific syntax rules, and examples
of filters.
3316376152002 [swapper/2] 0 [k] ip_local_deliver #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 0
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
+ 3316376220653 [swapper/2] 0 [k] __nft_trace_packet #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 1
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
table firewalld (1) chain filter_IN_FedoraServer (202) handle 215 drop
+ 3316376224687 [swapper/2] 0 [tp] skb:kfree_skb #304276b119fffff9847c36ba800 (skb 18446630032886128640) n 2 drop (NETFILTER_DROP)
if 2 (eth0) rxif 2 172.16.42.1.40532 > 172.16.42.2.8080 ttl 64 tos 0x0 id 14042 off 0 [DF] len 32 proto UDP (17) len 4
```

## Tracking
Another post-processing command, `pcap`, can be used to generate `pcap-ng` files
from a set of stored Retis events. For this to work the collection has to be
done using (at least) the `pcap` profile. For now `pcap-ng` files can be
generated by filtering the events using a single probe.

Retis does its best to track packets in the networking stack, and does it in
different ways. Note that tracking packets is not a built-in feature of the
Linux kernel and doing so is complex and cannot be 100% foolproof (but of course
bugs should be reported and fixed).
```none
$ retis -p pcap,generic collect -o
$ retis pcap --probe net:netif_receive_skb | tcpdump -nnr -
$ retis pcap --probe net:net_dev_start_xmit -o retis.pcap
$ wireshark retis.pcap
```

1. A Retis core built-in feature generates unique identifiers by tracking the
data part of [socket buffers](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/skbuff.h?h=v6.3#n737).
The socket buffer is also included in the identifier so we can track clones
and friends. This core skb tracking logic is used by the filtering part for
Retis to track packets after they were modified (e.g. NAT). Full details on
the implementation can be found
[in the sources](https://github.com/retis-org/retis/blob/main/retis/src/core/tracking/skb_tracking.rs).
Some post-processing commands (eg. `print`, `sort`) can generate a long output.
In such case a pager is automatically used in case the output is larger than the
current terminal. By default `less` is used but the pager can be explicitly
chosen by setting the `PAGER` environment variable, or unset by setting
`NOPAGER`.

2. A collector, `skb-tracking`, retrieves the core tracking information (unique
identifier and socket buffer address) and reports it in the event. Without
enabling this collector, skb tracking information won't be reported and can't
be used at post-processing time.
```none
$ PAGER=more retis sort
$ NOPAGER=1 retis sort
```

3. The `ovs` collector tracks packets in upcalls so we can follow a packet
being sent to the OpenVSwitch user-space daemon, even if it is re-injected
later on.
In addition to built-in post-processing commands, it is possible to use the
python bindings to implement custom processing. See the
[python bindings documentation](python.md).
37 changes: 20 additions & 17 deletions mkdocs.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,26 @@ repo_url: https://github.com/retis-org/retis
edit_uri: edit/main/docs/

nav:
- Overview: index.md
- Profiles: profiles.md
- Filtering: filtering.md
- Installation: install.md
- Requirements: requirements.md
- Limitations: limitations.md
- Contributing: CONTRIBUTING.md
- External resources: resources.md
- Event sections: event_sections.md
- Python bindings: python.md
- Collectors:
- skb: modules/skb.md
- skb-drop: modules/skb_drop.md
- skb-tracking: modules/skb_tracking.md
- ovs: modules/ovs.md
- ct: modules/ct.md
- nft: modules/nft.md
- Getting started:
- Overview: index.md
- Requirements: requirements.md
- Installation: install.md
- Collectors:
- Event sections: event_sections.md
- skb: modules/skb.md
- skb-drop: modules/skb_drop.md
- skb-tracking: modules/skb_tracking.md
- ovs: modules/ovs.md
- ct: modules/ct.md
- nft: modules/nft.md
- Learn more:
- Filtering: filtering.md
- Profiles: profiles.md
- Python bindings: python.md
- Limitations: limitations.md
- Resourses:
- External: resources.md
- Contributing: CONTRIBUTING.md

markdown_extensions:
- footnotes
Expand Down

0 comments on commit b4bff3b

Please sign in to comment.