Skip to content

Commit

Permalink
Typechecking for inputs, outputs and filters
Browse files Browse the repository at this point in the history
  • Loading branch information
@deric
deric committed Jul 20, 2023
1 parent 0201f99 commit f2aab5a
Show file tree
Hide file tree
Showing 6 changed files with 82 additions and 144 deletions.
121 changes: 16 additions & 105 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,117 +1,28 @@
# fluentbit
# puppet-fluentbit [![Test](https://github.com/deric/puppet-fluentbit/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/deric/puppet-fluentbit/actions/workflows/test.yml)

Welcome to your new module. A short overview of the generated parts can be found
in the [PDK documentation][1].

The README template below provides a starting point with details about what
information to include in your README.

## Table of Contents

1. [Description](#description)
1. [Setup - The basics of getting started with fluentbit](#setup)
* [What fluentbit affects](#what-fluentbit-affects)
* [Setup requirements](#setup-requirements)
* [Beginning with fluentbit](#beginning-with-fluentbit)
1. [Usage - Configuration options and additional functionality](#usage)
1. [Limitations - OS compatibility, etc.](#limitations)
1. [Development - Guide for contributing to the module](#development)
A Puppet module to manage [Fluent Bit](https://fluentbit.io/) installation.

## Description

Briefly tell users why they might want to use your module. Explain what your
module does and what kind of problems users can solve with it.

This should be a fairly short description helps the user decide if your module
is what they want.

## Setup

### What fluentbit affects **OPTIONAL**

If it's obvious what your module touches, you can skip this section. For
example, folks can probably figure out that your mysql_instance module affects
their MySQL instances.

If there's more that they should know about, though, this is the place to
mention:

* Files, packages, services, or operations that the module will alter, impact,
or execute.
* Dependencies that your module automatically installs.
* Warnings or other important notices.

### Setup Requirements **OPTIONAL**

If your module requires anything extra before setting up (pluginsync enabled,
another module, etc.), mention it here.

If your most recent release breaks compatibility or requires particular steps
for upgrading, you might want to include an additional "Upgrading" section here.

### Beginning with fluentbit

The very basic steps needed for a user to get the module up and running. This
can include setup steps, if necessary, or it can be an example of the most basic
use of the module.

## Usage

Include usage examples for common use cases in the **Usage** section. Show your
users how to use your module to solve problems, and be sure to include code
examples. Include three to five examples of the most important or common tasks a
user can accomplish with your module. Show users how to accomplish more complex
tasks that involve different types, classes, and functions working in tandem.

## Reference

This section is deprecated. Instead, add reference information to your code as
Puppet Strings comments, and then use Strings to generate a REFERENCE.md in your
module. For details on how to add code comments and generate documentation with
Strings, see the [Puppet Strings documentation][2] and [style guide][3].
Fluent Bit is a fast, lightweight agent for trasporting logs, metrics, traces, etc.

If you aren't ready to use Strings yet, manually create a REFERENCE.md in the
root of your module directory and list out each of your module's classes,
defined types, facts, functions, Puppet tasks, task plans, and resource types
and providers, along with the parameters for each.

For each element (class, defined type, function, and so on), list:

* The data type, if applicable.
* A description of what the element does.
* Valid values, if the data type doesn't make it obvious.
* Default value, if any.

For example:
In order to install the package and setup `fluent-bit` service, simply include the main class:

```puppet
include fluentbit
```
### `pet::cat`

#### Parameters
## Usage

##### `meow`
[Fluent Bit supports wide range of inputs](https://docs.fluentbit.io/manual/pipeline/inputs)

Enables vocalization in your cat. Valid options: 'string'.
![fluentbit pipeline]("doc/pipeline.png")

Default: 'medium-loud'.
```yaml
fluentbit::inputs:
'tail-syslog':
pipeline: input
plugin: tail
properties:
Path: /var/syslog
```

## Limitations

In the Limitations section, list any incompatibilities, known issues, or other
warnings.

## Development

In the Development section, tell other users the ground rules for contributing
to your project and how they should submit their work.

## Release Notes/Contributors/Etc. **Optional**

If you aren't using changelog, put your release notes here (though you should
consider using changelog). You can also add any additional sections you feel are
necessary or important to include here. Please use the `##` header.

[1]: https://puppet.com/docs/pdk/latest/pdk_generating_modules.html
[2]: https://puppet.com/docs/puppet/latest/puppet_strings.html
[3]: https://puppet.com/docs/puppet/latest/puppet_strings_style.html
20 changes: 10 additions & 10 deletions manifests/init.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
# @see https://docs.fluentbit.io/manual/
#
# @param manage_package_repo Installs the package repositories
# @param input_plugins Hash of the INPUT plugins to be configured
# @param output_plugins Hash of the OUTPUT plugins to be configured
# @param filter_plugins Hash of the filter to be configured
# @param inputs Hash of the INPUT plugins to be configured
# @param outputs Hash of the OUTPUT plugins to be configured
# @param filters Hash of the filter to be configured
#
# @param package_ensure
# Whether to install the Fluentbit package, and what version to install.
Expand Down Expand Up @@ -237,13 +237,13 @@
String $scripts_dir,

Fluentbit::Parser $parsers,
Hash[String, Hash] $input_plugins = {},
Hash[String, Hash] $output_plugins = {},
Hash[String, Hash] $filter_plugins = {},
Fluentbit::PipelinePlugin $inputs = {},
Fluentbit::PipelinePlugin $outputs = {},
Fluentbit::PipelinePlugin $filters = {},
Hash[String, Hash] $upstreams = {},
Hash $variables = {},
Fluentbit::MultilineParser $multiline_parsers = {},
Fluentbit::Stream $streams = {},
Hash[String, Hash] $upstreams = {},
Array[Stdlib::Absolutepath] $plugins = [],
) {
$plugins_path = "${config_dir}/${plugins_dir}"
Expand All @@ -259,8 +259,8 @@
-> Class['fluentbit::config']
~> Class['fluentbit::service']

create_resources(fluentbit::pipeline, $input_plugins, { pipeline_type => 'input' })
create_resources(fluentbit::pipeline, $output_plugins, { pipeline_type => 'output' })
create_resources(fluentbit::pipeline, $filter_plugins, { pipeline_type => 'filter' })
create_resources(fluentbit::pipeline, $inputs, { pipeline => 'input' })
create_resources(fluentbit::pipeline, $outputs, { pipeline => 'output' })
create_resources(fluentbit::pipeline, $filters, { pipeline => 'filter' })
create_resources(fluentbit::upstream, $upstreams)
}
48 changes: 21 additions & 27 deletions manifests/pipeline.pp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,78 +6,72 @@
#
# @example
# fluentbit::pipeline { 'input-dummy':
# type => 'input',
# plugin_name => 'dummy',
# pipeline => 'input',
# plugin => 'dummy',
# }
# @example
# fluentbit {
# input_plugins => {
# 'input-dummy' => { 'plugin_name' => 'dummy' },
# 'input-dummy' => { 'plugin' => 'dummy' },
# },
# }
# @see https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file#config_pipeline
#
# @param pipeline_type Defines the pipeline type to be configured
# @param plugin_name fluent-bit plugin name to be used
# @param pipeline Defines the pipeline type to be configured
# @param plugin fluent-bit plugin name to be used
# @param order Order to be applied to concat::fragment
# @param properties Hash of rest of properties needed to configure the pipeline-plugin
# @param data_dir
# @param config_dir
# @param scripts_path
# @param plugins_path
#
define fluentbit::pipeline (
Fluentbit::PipelineType $pipeline_type,
String[1] $plugin_name,
String[1] $order = '10',
Fluentbit::PipelineType $pipeline,
String[1] $plugin,
Integer $order = 10,
Hash[String, Any] $properties = {},
Stdlib::Absolutepath $data_dir = $fluentbit::data_dir,
Stdlib::Absolutepath $config_dir = $fluentbit::config_dir,
Stdlib::Absolutepath $scripts_path = $fluentbit::scripts_path,
Stdlib::Absolutepath $plugins_path = $fluentbit::plugins_path,
) {
$db_compatible_plugins = ['tail', 'systemd']

if $pipeline_type == 'input' and $plugin_name in $db_compatible_plugins {
if $pipeline == 'input' and $plugin in $db_compatible_plugins {
$db_settings = {
'db' => "${data_dir}/${title}",
'db' => "${fluentbit::data_dir}/${title}",
}
} else {
$db_settings = {}
}

if $pipeline_type == 'output' and $plugin_name == 'forward' {
if $pipeline == 'output' and $plugin == 'forward' {
$upstream_settings = $properties['upstream'] ? {
undef => {},
default => {
upstream => "${config_dir}/upstream-${properties['upstream']}.conf",
upstream => "${fluentbit::config_dir}/upstream-${properties['upstream']}.conf",
},
}
} else {
$upstream_settings = {}
}

if $pipeline_type == 'filter' and $plugin_name == 'lua' and $properties['code'] {
if $pipeline == 'filter' and $plugin == 'lua' and $properties['code'] {
# Catch 'code' property for lua scripts and write it to disk
file { "${scripts_path}/${title}.lua":
file { "${fluentbit::scripts_path}/${title}.lua":
ensure => file,
mode => $fluentbit::config_file_mode,
content => $properties['code'],
notify => Service[$fluentbit::service_name],
}
$script_settings = {
'script' => "${scripts_path}/${title}.lua",
'script' => "${fluentbit::scripts_path}/${title}.lua",
'code' => undef,
}
} else {
$script_settings = {}
}

concat::fragment { "fragment-${title}":
target => "${plugins_path}/${pipeline_type}s.conf",
concat::fragment { "pipeline-${title}":
target => "${fluentbit::plugins_path}/${pipeline}s.conf",
content => epp('fluentbit/pipeline.conf.epp',
{
name => $plugin_name,
pipeline_type => $pipeline_type,
name => $plugin,
pipeline_type => $pipeline,
order => $order,
properties => merge(
$db_settings,
{
Expand Down
21 changes: 21 additions & 0 deletions spec/classes/fluentbit_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,4 +101,25 @@
.with_content(%r{Name\s+json\n\s+Format\s+json\n\s+Time_key\s+time})
}
end

context 'configure inputs' do
let(:params) do
{
inputs: {
'tail-syslog': {
'pipeline': 'input',
'plugin': 'tail',
'properties': {
'Path': '/var/log/syslog',
}
}
},
}
end

it {
is_expected.to contain_file('/etc/fluent-bit/parsers.conf')
.with_content(%r{Name\s+json\n\s+Format\s+json\n\s+Time_key\s+time})
}
end
end
10 changes: 8 additions & 2 deletions spec/defines/pipeline_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,17 @@
let(:title) { 'input' }
let(:params) do
{
pipeline_type: 'input',
plugin_name: 'dummy',
pipeline: 'input',
plugin: 'tail',
properties: {
path: '/var/log/syslog',
}
}
end

it { is_expected.to compile }

it { is_expected.to contain_concat__fragment('pipeline-input').with_content(%r{Name\s+tail\n}) }
it { is_expected.to contain_concat__fragment('pipeline-input').with_content(%r{Path\s+/var/log/syslog\n}) }
end
end
6 changes: 6 additions & 0 deletions types/pipelineplugin.pp
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type Fluentbit::PipelinePlugin = Hash[String, Struct[{
pipeline => Fluentbit::PipelineType,
plugin => String[1],
Optional[order] => Integer,
Optional[properties] => Hash,
}]]

0 comments on commit f2aab5a

Please sign in to comment.