This blog post covers a recent use case we worked on, where we separated sensitive logs into a separate destination storage.
We recently worked on a use case where the system generated around 10,000 logs per second. Some logs contained sensitive information, and we wanted only specific teams to have access to these logs. So, we needed an efficient log routing solution to route the sensitive logs to particular destinations and storage based on their properties.
Existing system
The system in the context was a Java Spring-boot-based service. The logs generated were stored on the disk. A Syslog server running within the private network was used for sensitive information collection.
Initial implementation
Our first approach was to update the code to write the sensitive logs to the syslog server directly. However, this approach would mean a change in many places in the code. It would also be maintenance overhead managing separate logic implementation for specific logs . We wanted to have a system that could be configured centrally for the application and be able to specify the destination of the log for a specific class of code.
Introducing Log4J2 Appenders
After some research, we came across Log4J2 Syslog Appender. Appenders in Log4J2 allow us to specify different destinations for the logger. Then, we can select which appender to use based on references. This approach allows us to
- Keep the solution simple without introducing custom logic in the code
- Use the existing log4j library, which is battle-tested and has good community support
- Keep all the routing configurations in a single place for easier maintainability.
So, let’s dive into how we implemented it.
1. Create Log4j2 Configuration File
- Create a
log4j2-spring.xmlfile in theresource directory. This file defines the logging behaviour of your application. - Appenders: Two appenders are defined – one for the
Consoleand one forSyslog. The console appender uses a specified pattern layout for formatting. - Loggers: Two loggers are defined – one for the specific class (
com.one2n.logrouter.service.LogService) that contains sensitive information, and a root logger for all other logs.
2. Configure Logger
In the LogService class, we do not have any logic for the custom routing of the logs. It stays clean and simple.
Once we run the code, we can see that the sensitive logs are sent to the Syslog server. The rest of the logs are sent to the STDOUT.
Syslog Server:
logs received by the syslog server.
Application Server:
Console logs received by the application server.
Conclusion
With this approach, we were able to achieve configurable routing of logs based on a simple centralized configuration. Additionally using Log4J allows us to benefit from all the nice features such as configurable batch sizes for better utilization of network bandwidth and much more.
