Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter/date: Improvements #66

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

filter/date: Improvements #66

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d384c5a
filter/date: Add add'l fields for greater compatibility with logstash.
jbuchbinder Oct 18, 2018
a6fd909
Revert "grok: Fix not properly terminating once valid match is found."
jbuchbinder Oct 18, 2018
1a3bc63
Revert "grok: Add ability to accept multiple patterns for filter."
jbuchbinder Oct 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
filter/date: Add add'l fields for greater compatibility with logstash.
  • Loading branch information
@jbuchbinder
jbuchbinder committed Oct 18, 2018
commit d384c5a16d8c19f15a4756568c2d117170592f52
30 changes: 20 additions & 10 deletions filter/date/filterdate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,15 @@ import (
// ModuleName is the name used in config file
const ModuleName = "date"

// ErrorTag tag added to event when process module failed
const ErrorTag = "gogstash_filter_date_error"

// FilterConfig holds the configuration json fields and internal objects
type FilterConfig struct {
config.FilterConfig

Format []string `json:"format"` // date parse format
Source string `json:"source"` // source message field name
Joda bool `json:"joda"` // whether using joda time format
Format []string `json:"format"` // date parse format
Source string `json:"source"` // source message field name
Joda bool `json:"joda"` // whether using joda time format
TagOnFailure []string `json:"tag_on_failure"` // tags to append on failure
Target string `json:"target"` // target field

timeParser func(layout, value string) (time.Time, error)
}
Expand All @@ -35,8 +34,10 @@ func DefaultFilterConfig() FilterConfig {
Type: ModuleName,
},
},
Format: []string{time.RFC3339Nano},
Source: "message",
Format: []string{time.RFC3339Nano},
Source: "message",
Target: "@timestamp",
TagOnFailure: []string{"gogstash_filter_date_error", "_dateparsefailure"},
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why TagOnFailure "_dateparsefailure" as a default value?
I mean there was "gogstash_filter_date_error"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was specifically to ensure some sort of compatibility with the output that logstash's date plugin dumps into elasticsearch, so I had added it to the list of tags added on failure to not break compatibility with anything anyone had been doing.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to remove from default value, because not all users migrate from logstash, and it's unnecessary for pure gogstash user.
Write some examples or hints in README for your case?

}
}

Expand Down Expand Up @@ -70,11 +71,20 @@ func (f *FilterConfig) Event(ctx context.Context, event logevent.LogEvent) logev
}

if err != nil {
event.AddTag(ErrorTag)
for _, t := range f.TagOnFailure {
event.AddTag(t)
}
goglog.Logger.Error(err)
return event
}

event.Timestamp = timestamp.UTC()
switch f.Target {
case "@timestamp":
event.Timestamp = timestamp.UTC()
break
default:
event.SetValue(f.Target, timestamp.UTC())
break
}
return event
}