| title | description | services | documentationcenter | author | manager | editor | ms.assetid | ms.service | ms.devlang | ms.topic | ms.tgt_pltfrm | ms.workload | ms.date | ms.author |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Collect custom logs in OMS Log Analytics | Microsoft Docs |
Log Analytics can collect events from text files on both Windows and Linux computers. This article describes how to define a new custom log and details of the records they create in the OMS repository. |
log-analytics |
bwren |
jwhit |
tysonn |
aca7f6bb-6f53-4fd4-a45c-93f12ead4ae1 |
log-analytics |
na |
article |
na |
infrastructure-services |
11/17/2017 |
bwren |
The Custom Logs data source in Log Analytics allows you to collect events from text files on both Windows and Linux computers. Many applications log information to text files instead of standard logging services such as Windows Event log or Syslog. Once collected, you can parse each record in the log in to individual fields using the Custom Fields feature of Log Analytics.
The log files to be collected must match the following criteria.
-
The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry.
YYYY-MM-DD HH:MM:SS
M/D/YYYY HH:MM:SS AM/PM
Mon DD,YYYY HH:MM:SS -
The log file must not allow circular updates where the file is overwritten with new entries.
-
The log file must use ASCII or UTF-8 encoding. Other formats such as UTF-16 are not supported.
Note
If there are duplicate entries in the log file, Log Analytics will collect them. However, the search results will be inconsistent where the filter results show more events than the result count. It will be important that you validate the log to determine if the application that creates it is causing this behavior and address it if possible before creating the custom log collection definition.
Use the following procedure to define a custom log file. Scroll to the end of this article for a walkthrough of a sample of adding a custom log.
The Custom Log Wizard runs in the OMS portal and allows you to define a new custom log to collect.
- In the OMS portal, go to Settings.
- Click on Data and then Custom logs.
- By default, all configuration changes are automatically pushed to all agents. For Linux agents, a configuration file is sent to the Fluentd data collector. If you wish to modify this file manually on each Linux agent, then uncheck the box Apply below configuration to my Linux machines.
- Click Add+ to open the Custom Log Wizard.
You start by uploading a sample of the custom log. The wizard will parse and display the entries in this file for you to validate. Log Analytics will use the delimiter that you specify to identify each record.
New Line is the default delimiter and is used for log files that have a single entry per line. If the line starts with a date and time in one of the available formats, then you can specify a Timestamp delimiter which supports entries that span more than one line.
If a timestamp delimiter is used, then the TimeGenerated property of each record stored in OMS will be populated with the date/time specified for that entry in the log file. If a new line delimiter is used, then TimeGenerated is populated with date and time that Log Analytics collected the entry.
- Click Browse and browse to a sample file. Note that this may button may be labeled Choose File in some browsers.
- Click Next.
- The Custom Log Wizard will upload the file and list the records that it identifies.
- Change the delimiter that is used to identify a new record and select the delimiter that best identifies the records in your log file.
- Click Next.
You must define one or more paths on the agent where it can locate the custom log. You can either provide a specific path and name for the log file, or you can specify a path with a wildcard for the name. This supports applications that create a new file each day or when one file reaches a certain size. You can also provide multiple paths for a single log file.
For example, an application might create a log file each day with the date included in the name as in log20100316.txt. A pattern for such a log might be log*.txt which would apply to any log file following the application’s naming scheme.
The following table provides examples of valid patterns to specify different log files.
| Description | Path |
|---|---|
| All files in C:\Logs with .txt extension on Windows agent | C:\Logs\*.txt |
| All files in C:\Logs with a name starting with log and a .txt extension on Windows agent | C:\Logs\log*.txt |
| All files in /var/log/audit with .txt extension on Linux agent | /var/log/audit/*.txt |
| All files in /var/log/audit with a name starting with log and a .txt extension on Linux agent | /var/log/audit/log*.txt |
- Select Windows or Linux to specify which path format you are adding.
- Type in the path and click the + button.
- Repeat the process for any additional paths.
The name that you specify will be used for the log type as described above. It will always end with _CL to distinguish it as a custom log.
- Type in a name for the log. The _CL suffix is automatically provided.
- Add an optional Description.
- Click Next to save the custom log definition.
It may take up to an hour for the initial data from a new custom log to appear in Log Analytics. It will start collecting entries from the logs found in the path you specified from the point that you defined the custom log. It will not retain the entries that you uploaded during the custom log creation, but it will collect already existing entries in the log files that it locates.
Once Log Analytics starts collecting from the custom log, its records will be available with a Log Search. Use the name that you gave the custom log as the Type in your query.
Note
If the RawData property is missing from the search, you may need to close and reopen your browser.
The entire log entry will be stored in a single property called RawData. You will most likely want to separate the different pieces of information in each entry into individual properties stored in the record. You do this using the Custom Fields feature of Log Analytics.
Detailed steps for parsing the custom log entry are not provided here. Please refer to the Custom Fields documentation for this information.
You cannot remove a custom log definition once it's been created, but you can disable it by removing all of its collection paths.
- In the OMS portal, go to Settings.
- Click on Data and then Custom logs.
- Click Details next to the custom log definition to disable.
- Remove each of the collection paths for the custom log definition.
Log Analytics will collect new entries from each custom log approximately every 5 minutes. The agent will record its place in each log file that it collects from. If the agent goes offline for a period of time, then Log Analytics will collect entries from where it last left off, even if those entries were created while the agent was offline.
The entire contents of the log entry are written to a single property called RawData. You can parse this into multiple properties that can be analyzed and searched separately by defining Custom Fields after you have created the custom log.
Custom log records have a type with the log name that you provide and the properties in the following table.
| Property | Description |
|---|---|
| TimeGenerated | Date and time that the record was collected by Log Analytics. If the log uses a time-based delimiter then this is the time collected from the entry. |
| SourceSystem | Type of agent the record was collected from. OpsManager – Windows agent, either direct connect or System Center Operations Manager Linux – All Linux agents |
| RawData | Full text of the collected entry. |
| ManagementGroupName | Name of the management group for System Center Operations Manage agents. For other agents, this is AOI-<workspace ID> |
Records from custom logs are stored in the OMS repository just like records from any other data source. They will have a type matching the name that you provide when you define the log, so you can use the Type property in your search to retrieve records collected from a specific log.
The following table provides different examples of log searches that retrieve records from custom logs.
| Query | Description |
|---|---|
| MyApp_CL | All events from a custom log named MyApp_CL. |
| MyApp_CL | where Severity_CF=="error" | All events from a custom log named MyApp_CL with a value of error in a custom field named Severity_CF. |
The following section walks through an example of creating a custom log. The sample log being collected has a single entry on each line starting with a date and time and then comma-delimited fields for code, status, and message. Several sample entries are shown below.
2016-03-10 01:34:36 207,Success,Client 05a26a97-272a-4bc9-8f64-269d154b0e39 connected
2016-03-10 01:33:33 208,Warning,Client ec53d95c-1c88-41ae-8174-92104212de5d disconnected
2016-03-10 01:35:44 209,Success,Transaction 10d65890-b003-48f8-9cfc-9c74b51189c8 succeeded
2016-03-10 01:38:22 302,Error,Application could not connect to database
2016-03-10 01:31:34 303,Error,Application lost connection to database
We provide one of the log files and can see the events that it will be collecting. In this case New Line is a sufficient delimiter. If a single entry in the log could span multiple lines though, then a timestamp delimiter would need to be used.
The log files will be located in C:\MyApp\Logs. A new file will be created each day with a name that includes the date in the pattern appYYYYMMDD.log. A sufficient pattern for this log would be C:\MyApp\Logs\*.log.
We use a name of MyApp_CL and type in a Description.
We use a query of Type=MyApp_CL to return all records from the collected log.
We use Custom Fields to define the EventTime, Code, Status, and Message fields and we can see the difference in the records that are returned by the query.
- Use Custom Fields to parse the entries in the custom log in to individual fields.
- Learn about log searches to analyze the data collected from data sources and solutions.