Tag Archive for: log management

You are here: / log management

Posts

Enterprise Log Management: An Overview (Part 2)

In part 1 of my guest post series for the FOSE Insights Blog, I talked about the importance of enterprise log management and the questions one should ask before implementing a log management solution. In this post, I’ll be covering the different types of log management solutions to help you determine the best one for your organization.

When implementing a log management solution, there are various types of solutions, architectures and definitions.  Here are a few you should be familiar with:

  • Sinkhole – Traditional single “syslog” server that “receives” remote logs from one or more sources
  • Hierarchy – Multi-Tiered sinkhole’s divided by department, network (vlan), or other logical fashion such as accounting, marketing and engineering to collect log data.
  • Aggregator – Usually located at the top of a hierarchy, where major functionality such as alerting, reporting, searching and correlations occur.
  • Distributed – Independent log repositories, may be searchable/accessible from a central location.
  • Store and Forward – logs are written to a local disk or network disk to be spooled and sent later.
  • Streaming – Real-time distribution of log data to a remote logging server as they are being generated.
  • Agent Based – Operating systems that do not support remote logging often require assistance from software to send log data.  Even those operating systems that are capable of sending log data may use agents to send out specific data in a secure manner to a logging server.
  • Agent Less – Systems do not send log data directly to the log server, the logger itself obtains the data via secure file copy (store & grab) or WMI (Windows Management Interface)
  • Combo – Most mature log management infrastructures use part or all the above in some way or fashion.

There are a number of useful tools in both the open-source and commercial space that can assist in the creation of a log management solution or the upgrade of an existing solution.  In the open-source area, the following solutions are:

  • Syslog-ng – Unix based tool. Swiss army knife of log management. Can read any file and “tail” it to the network. Commercial versions available (sinkhole/ forwarding  agent)
  • Rsyslog – Like syslog-ng , with enhanced filtering, encryption, buffering
  • OSSEC   – Host Based or Server Based SIM/IDS (Aggregator/Agent)
  • –SEC.pl – Simple Event Correlator (Aggregator)
  • –PHP-Syslog, MySQL – PHP interface to logs in a database
  • –Lasso – Agent-less collection agent for Windows (WMI based)

While in the commercial realm, there are many formidable solutions available, but are just a here are a few notable ones:

Splunk – In the “Pro” column, Splunk provides relevant search information very fast due to its use of data indexing.  It has a wide support for various operating systems such as Windows, Mac OS X and Linux-based systems and it is extremely easy to use.  One can also have use the software at no cost up to 500MB of log data.  In the “Con” column, Splunk seem to have quick development cycles that requires numerous software updates and the advanced features, such as “app” development has a bit of a steep learning curve.

Log Logic – Log Logic is an appliance-based solution that is also fast and has a wide operating system support based.  As for the “Cons”, the cost is a bit high due to its appliance only option and it lacks user specific customization.

LogRhythm – LogRhythm has the ability to collect any type of log data regardless of source and the ability to collect log data with or without installing an agent on the log source device.  As for the “Cons”, its use of a database backend may cause insertion delay if the events per second are too high for the setup.  This may lead to a delay to access information.

Conclusion

The difference between a log management solution and other types of monitoring tools is that the data is already available on your devices and applications; it is just a matter of setting it up, collecting and using it.  In addition, log management is considered an industry and security best practice regardless if your organization has to meet regulatory compliance or not.

One should budget accordingly for the tools selected, even free tools have a cost when factoring hardware and storage components. Some vendors of commercially available tools publish their cost, while others may not, but there is a value on obtaining their professional services for initial deployment at an added cost.  It allows for better long-term planning and it is helpful for the initial setup and deployment phases.

Automated Log Management and Analysis using Splunk for Computer Incident Investigations

I define “Log Analysis” as a process of collecting system logs (syslog) and event data from computer systems, network devices and applications to look for anomalous events that are malicious or are in violation of organizational policies.

Many organizations spend thousands of dollars on equipment deployment, but ignore the system and event logs from those exact systems. Log analysis is one of the most overlooked aspects of operational computer and network security today.

Traditionally, security teams would use outdated methods and inefficient analysis techniques such as command lines and scripts to review log files. Furthermore, the security team has limited access to data, and when that data has to be collected from multiple locations and equipment to be analyzed, that often increases the amount of time necessary to produce a conclusion of an incident.

By introducing Splunk a search engine for log data that supports many log sources such as Apache access logs, mysql database logs, and any log in standard syslog format, we were able to be more productive in our log analysis.

Splunk comes in two versions, basic and professional. The basic version is free as long as you keep the data limited to 500MB a day while the professional version cost is dependent on the amount of data collected as well as some other neat features.

Splunk provides both real-time and historical visibility into all network, application, server and user activity to support investigations, alerting and reporting. It provides that bridge security and computer investigators need to do their jobs right.

For more information on Splunk and log management you can visit:
www.splunk.com