The Aliyun LOG Java Producer is a high concurrency writing library written for Java applications. The Producer Library and Consumer Library is a reading and writing encapsulation for Loghub. It aims to reduce the threshold for data collection and consumption.
- Provide an asynchronous sending interface, thread safety.
- You can configure more than one project.
- You can control the thread number for sending data.
- You can control the number and size of logs being merged into one package.
- You can control the memory usage. When the memory usage reach the threshold configured by user, the producer's send interface will be blocked until the free memory is available.
Disk Free: the generation data will be send to AliCloud Log Service in real time through network.High Throughput: support more than 100 write operations per second.The Computing and IO is separated: recoding logs do not affect computing time.
In the above scenarios, the Aliyun LOG Java Producer will alleviate the burden of your development. You don't need to care about log collection implementation details. And there is no need to worry about that log collection will impact on the normal operation of your business. Reducing the threshold of data acquisition greatly.
The comparison of all kinds of access methods
| Access Mode | Pros/Cons | Use Case |
|---|---|---|
| log file + Logtail | Log writing and log collection decoupling, there is no need to modify your code. | common case |
| syslog + Logtail | Good performance (80MB/S), disk free, the syslog protocol needs to be supported. | syslog |
| Aliyun Log Java SDK | Disk free, the data will be send to AliCloud Log Service directly, you need to care about the switch of network IO and program IO. | The log will not be written to a file |
| Aliyun LOG Java Producer | Disk free, asynchronous and high throughput. | The log will not be written to a file, high QPS. |
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>aliyun-log</artifactId>
<version>0.6.27</version>
</dependency>
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>log-loghub-producer</artifactId>
<version>0.1.16</version>
</dependency>
The configuration format is as follows. Please refer to the parameter description part for parameter value.
public class ProducerConfig {
// Specify the timeout for sending package, in milliseconds, default is 3000
public int packageTimeoutInMS = 3000;
// Specify the maximum log count per package, the upper limit is 4096
public int logsCountPerPackage = 4096;
// Specify the maximum cache size per package, the upper limit is 3MB, in bytes
public int logsBytesPerPackage = 3 * 1024 * 1024;
// The upper limit of the memory that can be used by each producer instance, in bytes, default is 100MB
public int memPoolSizeInByte = 100 * 1024 * 1024;
// If shardHash is specified when you send data, you should care about this parameter, otherwise there is no need to care about it.
// The backend thread will merge the data being sent to the same shard together, and shard is associated with a hash interval.
// The producer will pull the hash interval information for each shard from AliCloud Log Service regularly and update the local value, this parameter stands for the time interval.
public int shardHashUpdateIntervalInMS = 10 * 60 * 1000;
// Specify the retry times, default is 10. If exceeds this value, the exception will pass to the callback.
public int retryTimes = 10;
// protobuf
public String logsFormat = "protobuf";
// Specify the I/O thread pool's maximum pool size, the main function of the I/O thread pool is to send data, default is 8
public int maxIOThreadSizeInPool = 8;
// userAgent
public String userAgent = "loghub-producer-java";
}
The main function of callback is to handle the results of sending data. The results include successful status or exception. If you don't care about the result, there is no need to extends ILogCallback.
| Parameter | Description | Value |
|---|---|---|
| packageTimeoutInMS | Specify the timeout for sending package. | Integer, in milliseconds |
| logsCountPerPackage | Specify the maximum log count per package. | Integer, 1~4096 |
| logsBytesPerPackage | Specify the maximum cache size per package. | Integer, 1~3145728(3M),in bytes |
| memPoolSizeInByte | The upper limit of the memory that can be used by each producer instance. | Integer, in bytes |
| maxIOThreadSizeInPool | Specify the I/O thread pool's maximum pool size, the main function of the I/O thread pool is to send data. | Integer, default is 8 |
| shardHashUpdateIntervalInMS | If shardHash is specified when you send data, you should care about this parameter, otherwise there is no need to care about it. The backend thread will merge the data being sent to the same shard together, and shard is associated with a hash interval. The producer will pull the hash interval information for each shard from AliCloud Log Service regularly and update the local value, this parameter stands for the time interval. | Integer, in milliseconds |
| retryTimes | Specify the retry times, default is 10. If exceeds this value, the exception will pass to the callback. | Integer |
If you want to write data to logstore through Aliyun LOG Java Producer, you should configure the following permissions for the account you use. RAM doc
| Action | Resource |
|---|---|
| log:PostLogStoreLogs | acs:log:${regionName}:${projectOwnerAliUid}:project/${projectName}/logstore/${logstoreName} |
If the interface provided by producer can't meet your log collection requirements, you can develop your own log collection API based on Aliyun Log Java SDK.
- Alicloud Log Service homepage
- Alicloud Log Service doc
- Alicloud Log Servic official forum
- Alicloud Log Servic official technical support: submit tickets
@zzboy made a great contribution to this project.
Thanks for the excellent work by @zzboy