{agent} uses data streams to store time series data across multiple indices. Each data stream ships with a customizable index lifecycle policy that automates data retention as your indices grow and age.
The storage and sizing guide attempts to define a "typical" storage reference for Elastic APM, and there are additional settings you can tweak to reduce storage, or to tune data ingestion in {es}.
In addition, the APM UI makes it easy to visualize your APM data usage with {kibana-ref}/storage-explorer.html[storage explorer]. Storage explorer allows you to analyze the storage footprint of each of your services to see which are producing large amounts of data—so you can better reduce the data you’re collecting or forecast and prepare for future storage needs.
APM processing and storage costs are largely dominated by transactions, spans, and stack frames.
-
Transactions describe an event captured by an Elastic {apm-agent} instrumenting a service. They are the highest level of work being measuring within a service.
-
Spans belong to transactions. They measure from the start to end of an activity, and contain information about a specific code path that has been executed.
-
Stack frames belong to spans. Stack frames represent a function call on the call stack, and include attributes like function name, file name and path, line number, etc. Stack frames can heavily influence the size of a span.
Due to the high variability of APM data, it’s difficult to classify a transaction as typical. Regardless, this guide will attempt to classify Transactions as Small, Medium, or Large, and make recommendations based on those classifications.
The size of a transaction depends on the language, agent settings, and what services the agent instruments. For instance, an agent auto-instrumenting a service with a popular tech stack (web framework, database, caching library, etc.) is more likely to generate bigger transactions.
In addition, all agents support manual instrumentation. How little or much you use these APIs will also impact what a typical transaction looks like.
If your sampling rate is very small, transactions will be the dominate storage cost.
Here’s a speculative reference:
| Transaction size | Number of Spans | Number of stack frames |
|---|---|---|
Small |
5-10 |
5-10 |
Medium |
15-20 |
15-20 |
Large |
30-40 |
30-40 |
There will always be transaction outliers with hundreds of spans or stack frames, but those are very rare. Small transactions are the most common.
Consider the following typical storage reference. These numbers do not account for {es} compression.
-
1 unsampled transaction is ~1 KB
-
1 span with 10 stack frames is ~4 KB
-
1 span with 50 stack frames is ~20 KB
-
1 transaction with 10 spans, each with 10 stack frames is ~50 KB
-
1 transaction with 25 spans, each with 25 spans is 250-300 KB
-
100 transactions with 10 spans, each with 10 stack frames, sampled at 90% is 600 KB
APM data compresses quite well, so the storage cost in {es} will be considerably less:
-
Indexing 100 unsampled transactions per second for 1 hour results in 360,000 documents. These documents use around 50 MB of disk space.
-
Indexing 10 transactions per second for 1 hour, each transaction with 10 spans, each span with 10 stack frames, results in 396,000 documents. These documents use around 200 MB of disk space.
-
Indexing 25 transactions per second for 1 hour, each transaction with 25 spans, each span with 25 stack frames, results in 2,340,000 documents. These documents use around 1.2 GB of disk space.
|
Note
|
These examples were indexing the same data over and over with minimal variation. Because of that, the compression ratios observed of 80-90% are somewhat optimistic. |
The amount of storage for APM data depends on several factors: the number of services you are instrumenting, how much traffic the services see, agent and server settings, and the length of time you store your data.
Here are some ways you can reduce either the amount of APM data you’re ingesting or the amount of data you’re retaining.
Distributed tracing can generate a substantial amount of data. More data can mean higher costs and more noise. Sampling aims to lower the amount of data ingested and the effort required to analyze that data.
See [sampling] to learn more.
In some cases, APM agents may collect large amounts of very similar or identical spans in a transaction. These repeated, similar spans often don’t provide added benefit, especially if they are of very short duration. Span compression takes these similar spans and compresses them into a single span-- retaining important information but reducing processing and storage overhead.
See [span-compression] to learn more.
Elastic APM agents collect stacktrace information under certain circumstances.
This can be very helpful in identifying issues in your code,
but it also comes with an overhead at collection time and increases the storage usage.
Stack trace collection settings are managed in each agent.
You might want to only keep data for a defined time period. This might mean deleting old documents periodically, deleting data collected for specific services or customers, or deleting specific indices.
Depending on your use case, you can delete data:
-
periodically with {ilm}
-
with the {kib} Index Management UI
If you want to delete data for security or privacy reasons, see [apm-data-security].
Index lifecycle management enables you to automate how you want to manage your indices over time. You can base actions on factors such as shard size and performance requirements. See [ilm-how-to] to learn more.
You can delete all APM documents matching a specific query with the {ref}/docs-delete-by-query.html[Delete By Query API].
For example, to delete all documents with a given service.name, use the following request:
POST /.ds-*-apm*/_delete_by_query
{
"query": {
"term": {
"service.name": {
"value": "old-service-name"
}
}
}
}{kib}'s {ref}/index-mgmt.html[Index Management] allows you to manage your cluster’s indices, data streams, index templates, and much more.
In {kib}, navigate to Stack Management > Index Management > Data Streams. Select the data streams you want to delete, and click Delete data streams.
You might want to update documents that are already indexed. For example, if you your service name was set incorrectly.
To do this, you can use the {ref}/docs-update-by-query.html[Update By Query API]. To rename a service, send the following request:
POST /.ds-*-apm*/_update_by_query?expand_wildcards=all
{
"query": {
"term": {
"service.name": {
"value": "current-service-name"
}
}
},
"script": {
"source": "ctx._source.service.name = 'new-service-name'",
"lang": "painless"
}
}|
Tip
|
Remember to also change the service name in the {apm-agents-ref}/index.html[{apm-agent} configuration]. |