Six Costly Pitfalls in Log Collection: From Local Management Missteps to Looming System Failures | HackerNoon

Background

When monitoring the running status of the system and troubleshooting complex issues, logs have long served as an indispensable observability method. Scientific local log management strategies not only retain more complete historical records locally and minimize performance overhead but also facilitate log collection and subsequent analysis. However, in actual O&M, we often encounter counterexamples. The collection problems caused by such management defects cannot be perfectly solved by mainstream collection tools such as LoongCollector (formerly iLogtail), Filebeat, FluentBit, Vector, and OpenTelemetry Collector. The best practice is to solve the root cause. Here we summarize our experience, hoping to provide some inspiration and collectively enhance log utility for all.

1. Using the copy truncate mode for log rotation may cause log loss or duplicate collection due to non-atomic operations and new file creation.

The principle of using the copy truncate mode of Logrotate to rotate logs is to first copy the original log file and then truncate it. This method raises the following issues:

1. New files generated by the copy operation may be repeatedly collected as new content. Due to inode changes in the file system, the collector may fail to correctly identify this as the rotated old file.
2. Logs generated between copy and truncate operations may be lost. There is a time window between these two operations, during which the content written exists neither in the copied file nor in the original file (as it will be cleared by the truncate operation).
3. The truncate operation may reduce the file size and change the header content. Shrinking the file or changing its header signature will cause the collector to misjudge it as a new file, resulting in duplicate collection.

Therefore, the copy truncate mode may lead to issues like duplicate log collection, content loss, or inconsistency.

It is recommended to use the create mode for log rotation, that is, to create a new file and rename the old file, which ensures file integrity and continuity. If unavoidable, use the exact path name when configuring the collection settings.

Network-attached storage (NAS) typically employs an eventual consistency model, which is a common design in distributed systems. In real-time collection scenarios, this may cause the following issues:

1. Inconsistency between object metadata and actual content. Due to eventual consistency, metadata such as file size may be updated before the actual content.
2. Read operations returning file holes. If the metadata shows that the file has grown, but the actual content has not been synchronized, the read operation may return 0 characters (file holes).
3. Data latency. The results of write operations may not be immediately visible to read operations, causing collection latency.
4. Data loss. NAS does not support inotify, and objects are listed at a low speed, so files may not be detected, resulting in data loss.

These issues may cause the collected data to be inconsistent with the final content.

It is recommended to use EBS and use local disks for on-premises servers to ensure the efficiency and consistency of log reading and writing. If unavoidable, implement the compatibility logic for exception logs on the consumer.

3. Multi-process log writing may cause incomplete data collection due to mutual data overwriting.

It is a common but not recommended practice for multiple processes to write to the same log file concurrently, which may lead to the following problems:

1. Interleaved file content. Writes by multiple processes may intersect with each other, causing disorder in log entries.
2. Incomplete collection. When a write event occurs on the file, the collector starts collecting data. However, if other processes continue to write data during the collection process, these newly written contents may be skipped.
3. File lock contention. Multi-process writes may cause file lock contention, which affects write performance and reliability.

This pattern may result in incomplete data collection that deviates from the final file contents.

It is recommended that multiple processes write their respective files to ensure log integrity and order. If unavoidable, implement the compatibility logic for exception logs on the consumer.

4. Creating file holes to release log file space may cause duplicate log collection or data loss due to changes in the file signature and content.

Releasing log file space by creating holes in the file header is a risky practice for the following reasons:

1. Changed file signature. LoongCollector (formerly iLogtail) additionally uses the header content of the file as the basis for determining file uniqueness to prevent missed collection from inode reuse. Creating holes may change this signature, causing the collector to misjudge it as a new file.
2. Data integrity issues. Creating holes essentially replaces the original content with 0 characters, which may result in the loss of important historical logs.
3. File system fragmentation. Frequent creation of holes may lead to file system fragmentation, degrading read and write performance.

This practice may result in duplicate data collection and loss of historical data.

It is recommended to use the standard log rotation mechanism to manage log file size, such as using the Logrotate tool for regular log rotation, which ensures log integrity and traceability. If unavoidable, we recommend that you use fallocate instead of truncate or dd, and implement compatibility logic for exception logs on the consumer.

5. Frequent overwriting of files may result in incomplete or inconsistent collected data due to constantly changing file content.

Frequently overwriting the entire log file is an insecure log management method. It may cause the following issues:

1. Inconsistent metadata and content. During the overwriting process, metadata such as the file size may be updated before the actual content. As a result, the collector reads incomplete or inconsistent content.
2. Data loss risk. If overwrite operations occur during log collection, the collected data may be disordered or lost.
3. Difficulty retaining historical data. Frequent overwriting causes failure to retain historical logs, which is not conducive to issue tracing and analysis. This practice may result in inconsistency between the collected content and the final file content, or complete loss of file content.

It is recommended to record logs in append mode and use the log rotation mechanism to manage the file size. If unavoidable, implement the compatibility logic for exception logs on the consumer.

6. Saving files edited with Vim may cause duplicate log collection due to creating new files to replace the original ones.

When you use Vim to edit and save a file, its saving mechanism can cause the following issues:

1. The inode change. When Vim creates a new file to replace the original one, the new file has a different inode from the original, which may cause the collector to misjudge it as a new file.
2. Changed file signature. The header content of the new file may differ from that of the original file, which changes the file signature. As a result, the collector cannot correctly identify the file.
3. File content loss. When Vim replaces the file, the writing program may not switch to the newly saved log file, potentially resulting in loss of log content.

This editing mode may lead to duplicate log collection or data loss.

If you only need to view logs, we recommend that you use read-only tools such as less and grep. If unavoidable, implement deduplication and exception handling logic on the consumer.

Summary

Log is the “black box” of system operations, and its management quality directly affects the troubleshooting efficiency and system reliability. By avoiding the anti-patterns mentioned in this article and following the best practices, such as using Logstore rotation, local disk writing, and single-threaded appending, you can significantly lower log collection risks and improve observability. It is hoped that this article can provide practical references for teams to build a robust and efficient log management system.