Feature Request: CLI Flag to filter user information out of logging trace

[obloomfield]

Feature request

This feature is to optionally omit user information from logging traces (spans) by toggling some new CLI flag / environment variable.

Motivation

For many teams to maintain security compliance, this feature would be critical.

If you are using TEI in a production service, there is likely a security policy that prohibits you from accessing user data without consent. Without this change, user info like queries and document snippets would be leaked.

Your contribution

This can be likely done by:

• Adding a new disabled_fields Layer to logging.rs to allow for control of which named log fields to filter out (for instance, "inputs").
• Adding a new CLI flag / env variable DISABLE_LOG_FIELDS to list out disabled log fields
• Add README to describe how this could be used to filter customer sensitive info

I would be happy to work on this once the feature is approved. Let me know if you think there may be a better approach.