forked from vectordotdev/vector
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.md.erb
175 lines (121 loc) · 10.2 KB
/
README.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
<p align="center">
<strong>
<a href="<%= metadata.links.fetch("urls.vector_components") %>">Components<a/> • <a href="<%= metadata.links.fetch("urls.vector_guides") %>">Guides<a/> • <a href="<%= metadata.links.fetch("urls.vector_docs") %>">Docs<a/> • <a href="<%= metadata.links.fetch("urls.vector_blog") %>">Blog<a/> • <a href="<%= metadata.links.fetch("urls.vector_download") %>">Download v<%= metadata.latest_version %><a/>
</strong>
</p>
---
<p align="center">
<img src="./website/static/img/readme_diagram.svg" alt="Vector">
</p>
<%- if metadata.new_post -%>
---
<p align="center">
<strong>
New post! <a href="<%= HOST %><%= metadata.new_post.permalink %>"><%= metadata.new_post.title %></a>
</strong>
</p>
---
<%- end -%>
## What is Vector?
<%= vector_summary() %>
## [Documentation](https://vector.dev/docs/)
### About
* [**Concepts**][docs.concepts]
* [**Data model**][docs.data_model] - [log event][docs.data-model.log], [metric event][docs.data-model.metric]
* [**Guarantees**][docs.guarantees]
### Setup
* [**Installation**][docs.installation] - [operating systems][docs.operating_systems], [package managers][docs.package_managers], [platforms][docs.platforms], [from archives][docs.from-archives], [from source][docs.from-source]
* [**Configuration**][docs.configuration]
* [**Deployment**][docs.deployment] - [strategies][docs.strategies], [topologies][docs.topologies]
### Reference
* [**Sources**][docs.sources] - <%= common_component_links(:source) %>
* [**Transforms**][docs.transforms] - <%= common_component_links(:transform) %>
* [**Sinks**][docs.sinks] - <%= common_component_links(:sink) %>
### Administration
* [**Process management**][docs.process-management]
* [**Monitoring**][docs.monitoring]
* [**Updating**][docs.updating]
* [**Validating**][docs.validating]
### Resources
* [**Community**][urls.vector_community] - [chat][urls.vector_chat], [@vectordotdev][urls.vector_twitter], [mailing list][urls.mailing_list]
* [**Releases**][urls.vector_releases] - [v<%= metadata.latest_version %> (latest)][urls.v<%= metadata.latest_version %>]
* [**Roadmap**][urls.vector_roadmap] - [vote on new features][urls.vote_feature]
* **Policies** - [Security][urls.vector_security_policy], [Privacy][urls.vector_privacy_policy], [Code of Conduct][urls.vector_code_of_conduct]
## Performance
The following performance tests demonstrate baseline performance between
common protocols with the exception of the Regex Parsing test.
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
| ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| [TCP to Blackhole](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_blackhole_performance) | _**86mib/s**_ | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
| [File to TCP](https://github.com/timberio/vector-test-harness/tree/master/cases/file_to_tcp_performance) | _**76.7mib/s**_ | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
| [Regex Parsing](https://github.com/timberio/vector-test-harness/tree/master/cases/regex_parsing_performance) | 13.2mib/s | n/a | _**20.5mib/s**_ | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
| [TCP to HTTP](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_http_performance) | _**26.7mib/s**_ | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
| [TCP to TCP](https://github.com/timberio/vector-test-harness/tree/master/cases/tcp_to_tcp_performance) | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | _**70.4mib/s**_ | 7.6mib/s |
To learn more about our performance tests, please see the [Vector test harness][urls.vector_test_harness].
## Correctness
The following correctness tests are not exhaustive, but they demonstrate
fundamental differences in quality and attention to detail:
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | Splunk UF | Splunk HF |
| ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| [Disk Buffer Persistence](https://github.com/timberio/vector-test-harness/tree/master/cases/disk_buffer_persistence_correctness) | ✅ | ✅ | ❌ | ❌ | ⚠️ | ✅ | ✅ |
| [File Rotate (create)](https://github.com/timberio/vector-test-harness/tree/master/cases/file_rotate_create_correctness) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [File Rotate (copytruncate)](https://github.com/timberio/vector-test-harness/tree/master/cases/file_rotate_truncate_correctness) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| [File Truncation](https://github.com/timberio/vector-test-harness/tree/master/cases/file_truncate_correctness) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Process (SIGHUP)](https://github.com/timberio/vector-test-harness/tree/master/cases/sighup_correctness) | ✅ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ✅ |
| [JSON (wrapped)](https://github.com/timberio/vector-test-harness/tree/master/cases/wrapped_json_correctness) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
To learn more about our correctness tests, please see the [Vector test harness][urls.vector_test_harness].
## The Little Details
### Data Model
* **All data types** - [Logs][docs.data-model.log], [metrics][docs.data-model.metric], and traces (coming soon).
* **Customizable log schema** - [Change Vector's log schema][docs.global-options#log_schema] to anything you like.
* **Rich type system** - Support for [JSON primitive types][docs.data-model.log#types] and [timestamps][docs.data-model.log#timestamps].
* **Metrics interoperability** - A [sophisticated metrics data model][docs.data-model.metric] ensures _correct_ interoperability between systems.
* **Metrics aggregation** - Aggregated [histgorams][docs.data-model.metric#aggregated_histogram] and [summaries][docs.data-model.metric#aggregated_summary] reduce volume without loss of precision.
### Control Flow
* **Pipelining** - A [directed acyclic graph processing model][docs.configuration] allows for flexible topologies.
* **Control-flow** - Transforms like the [`swimlanes` transform][docs.transforms.swimlanes] allow for complex control-flow logic.
* **Dynamic partitioning** - Create [dynamic partitions on the fly][docs.sinks.aws_s3#partitioning] with Vector's [templating syntax][docs.reference.templating].
### Data Processing
* **Programmable transforms** - [Lua][docs.transforms.lua], [Javascript (coming soon)][urls.pr_721], and [WASM (coming soon)][urls.issue_1802] transforms.
* **Rich parsing** - [Regex][docs.transforms.regex_parser], [Grok][docs.transforms.grok_parser], and [more][urls.vector_parsing_transforms] allow for rich parsing.
* **Smart timestamp coercion** - All [parsing transforms][urls.vector_parsing_transforms] implement a [`types` option][docs.transforms.regex_parser#types] that can automatically parse timestamps.
* **Context enrichment** - [Enrich data with environment context][urls.vector_enriching_transforms].
* **Metrics derivation** - [Derive logs from metrics][docs.transforms.log_to_metric].
* **Multi-line merging** - [Merge multi-line logs][docs.sources.file#multiline] into one event, such as stacktraces.
### Operations
* **Hot reload** - [Reload configuration on the fly][docs.process-management#reloading] without disrupting data flow.
* **Zero delay start** - [Starts and restarts][docs.administration.process-management] without a delay.
* **Multi-platform** - [Linux, MacOS, Windows, x86_64, ARM64, and ARMv7][docs.installation].
* **CI friendly** - [Config validating][docs.administration.validating] and [unit tests][guides.advanced.unit-testing] make Vector CI friendly.
* **Configurable concurrency** - All CPU cores ([service][docs.strategies#service]) or just one ([daemon][docs.strategies#daemon]) via the [`--threads` flag][docs.process-management#starting].
* **Custom DNS** - [Custom DNS][docs.global-options#dns_servers] makes service discovery possible.
* **Optional static binary** - [Optional MUSL static binaries][pages.releases] mean zero required dependencies.
* **TLS support** - All relevant Vector components offer TLS options for secure communication.
### Reliability
* **Memory safety** - Vector is built in Rust and is [memory safe][urls.rust_memory_safety], avoiding a large class of memory related errors.
* **Decoupled buffer design** - Buffers are per-[sink][docs.sinks]; a bad sink won't bring the entire pipeline to a halt.
* **Intelligent retries** - A fibonacci backoff algorithsm with jitter makes Vector a good citizen during outages.
* **Backpressure & load shedding** - Buffers can be configured to provide backpressure or shed load.
* **Rate-limited internal logging** - Vector's internal logging is rate-limited avoiding IO saturation if errors occur.
* **Sink healthchecks** - Healthchecks provide startup safety and prevent deploys with bad configuration.
* **Robust disk buffering** - Vector uses `leveldb` for robust data durability across restarts.
### UX
* **Clear Guarantees** - A [guarantee support matrix][docs.guarantees] helps you make the appropriate tradeoffs with components.
* **Config unit tests** - [Develop Vector config files like code][guides.advanced.unit-testing]. Avoid the frustrating dev style required by other tools.
* **Config linting** - [Quickly lint][docs.administration.validating] Vector config files to spot errors and prevent bad configs in CI.
* **Thoughtful docs** - [Quality documentation][docs.what-is-vector] that respects your time and reduces communication overhead.
## Installation
Run the following in your terminal, then follow the on-screen instructions.
```bash
<%= install_command %>
```
Or use your own [preferred method][docs.installation].
## Latest Highlights
<%- metadata.highlights.reverse[0..4].each do |highlight| -%>
* [<%= highlight.date.strftime("%Y/%m/%d") %> - <%= highlight.title %>](<%= HOST %><%= highlight.permalink %>) - available in [<%= highlight.release %>][<%= highlight.release == "nightly" ? "urls.vector_download_nightly" : "urls.vector_download" %>]
<%- end -%>
[view all...][urls.vector_highlights]
---
<p align="center">
Developed with ❤️ by <strong><a href="https://timber.io">Timber.io</a></strong> - <a href="https://github.com/timberio/vector/security/policy">Security Policy</a> - <a href="https://github.com/timberio/vector/blob/master/PRIVACY.md">Privacy Policy</a>
</p>