In part 1 of my guest post series for the FOSE Insights Blog, I talked about the importance of enterprise log management and the questions one should ask before implementing a log management solution. In this post, I’ll be covering the different types of log management solutions to help you determine the best one for your organization.
When implementing a log management solution, there are various types of solutions, architectures and definitions. Here are a few you should be familiar with:
- Sinkhole – Traditional single “syslog” server that “receives” remote logs from one or more sources
- Hierarchy – Multi-Tiered sinkhole’s divided by department, network (vlan), or other logical fashion such as accounting, marketing and engineering to collect log data.
- Aggregator – Usually located at the top of a hierarchy, where major functionality such as alerting, reporting, searching and correlations occur.
- Distributed – Independent log repositories, may be searchable/accessible from a central location.
- Store and Forward – logs are written to a local disk or network disk to be spooled and sent later.
- Streaming – Real-time distribution of log data to a remote logging server as they are being generated.
- Agent Based – Operating systems that do not support remote logging often require assistance from software to send log data. Even those operating systems that are capable of sending log data may use agents to send out specific data in a secure manner to a logging server.
- Agent Less – Systems do not send log data directly to the log server, the logger itself obtains the data via secure file copy (store & grab) or WMI (Windows Management Interface)
- Combo – Most mature log management infrastructures use part or all the above in some way or fashion.
There are a number of useful tools in both the open-source and commercial space that can assist in the creation of a log management solution or the upgrade of an existing solution. In the open-source area, the following solutions are:
- Syslog-ng – Unix based tool. Swiss army knife of log management. Can read any file and “tail” it to the network. Commercial versions available (sinkhole/ forwarding agent)
- Rsyslog – Like syslog-ng , with enhanced filtering, encryption, buffering
- OSSEC – Host Based or Server Based SIM/IDS (Aggregator/Agent)
- –SEC.pl – Simple Event Correlator (Aggregator)
- –PHP-Syslog, MySQL – PHP interface to logs in a database
- –Lasso – Agent-less collection agent for Windows (WMI based)
While in the commercial realm, there are many formidable solutions available, but are just a here are a few notable ones:
Splunk – In the “Pro” column, Splunk provides relevant search information very fast due to its use of data indexing. It has a wide support for various operating systems such as Windows, Mac OS X and Linux-based systems and it is extremely easy to use. One can also have use the software at no cost up to 500MB of log data. In the “Con” column, Splunk seem to have quick development cycles that requires numerous software updates and the advanced features, such as “app” development has a bit of a steep learning curve.
Log Logic – Log Logic is an appliance-based solution that is also fast and has a wide operating system support based. As for the “Cons”, the cost is a bit high due to its appliance only option and it lacks user specific customization.
LogRhythm – LogRhythm has the ability to collect any type of log data regardless of source and the ability to collect log data with or without installing an agent on the log source device. As for the “Cons”, its use of a database backend may cause insertion delay if the events per second are too high for the setup. This may lead to a delay to access information.
Conclusion
The difference between a log management solution and other types of monitoring tools is that the data is already available on your devices and applications; it is just a matter of setting it up, collecting and using it. In addition, log management is considered an industry and security best practice regardless if your organization has to meet regulatory compliance or not.
One should budget accordingly for the tools selected, even free tools have a cost when factoring hardware and storage components. Some vendors of commercially available tools publish their cost, while others may not, but there is a value on obtaining their professional services for initial deployment at an added cost. It allows for better long-term planning and it is helpful for the initial setup and deployment phases.
