Without the right tools to aggregate and parse your log data, finding and understanding the information you’re looking for is nearly impossible

There are endless uses for logs because the logs themselves are endless. Application logs, security logs, BI logs, Lincoln logs (Ok, maybe not)…

Setting aside security, BI and childhood fun for a moment, even looking specifically at uses for application logs alone, there are countless uses. Production monitoring, Performance monitoring, Support, Troubleshooting, QA resolution, and more. It really all comes down to how and what you log. All of the tools here can be used to get a better understanding and more value out of your logs, but they also have their own strengths and weaknesses. In order to get the most out of your tooling, it’s important to select it based on your needs and goals.

Logging Best Practices (Getting the Most Out of Your Tooling)

First and foremost, we need to be aware of what our goals are. Why are we adding log statements to our code in the first place? Do we want to use them for application monitoring? Support and troubleshooting? Security? Depending on what your goals are, your entire approach to logging and the tools you will need may be change.

Once the purpose of the logs is settled, it’s important to structure them so that they are understandable both to yourself and the members of your team and to whichever logging tool you choose to use. JSON and KVP (Key Value Pair) are both good choices.

Logs generate a massive amount of data, and this data may be coming from multiple environments across many servers. To ensure that this data isn’t lost and can be used effectively, they should be consolidated and centralized to a single storage location. Of course, this can be costly which is why, in almost all situations, TRACE, INFO and DEBUG-level logs are turned off in production.

Finally, we need to be aware of the limitations of our logged data. Our first warning sign is that those log-levels that we’re turning off in production (TRACE, INFO, DEBUG) makeup close to two-thirds of all of our logs. That’s a lot of missing information.

Logs have been used for troubleshooting and support for ages, but considering the limited context they provide for application errors, they’re much better suited for other purposes. These log aggregation and analysis tools are most helpful for security and BI purposes or for identifying trends in user events and activities.

And a final note, don’t think that logs are the only data you can get your hands on. These tools all integrate with others which allows you to combine the data parsing power of your logging tools with more meaningful application data.

1. Splunk

Splunk is the biggest tool in the log management space. It’s well-established, full-featured, and enterprise-class. It’s unique in this space as an on-premises tool (although they have come out with a Cloud version as well).

When to use it: Enterprise companies with lots of feature needs and a variety of data that needs analyzing.

Pricing:

• Enterprise: $50-173/GB/month, depending on data volume
• Cloud: Contact for pricing info
• Light: $87/GB/month for up to 20GB/day (up to 5 users only)

Pros:

• Most feature-rich (more than 500 apps)
• Built-in search and visualization tools
• Good for security, BI and infrastructure monitoring

Cons:

• Complex setup and maintenance
• High cost to support real-world applications

2. Elastic

Elastic (formerly ELK – ElasticSearch, Logstash, Kibana) is an open-source project made up of many different tools for application data analysis and visualization. Logstash, specifically, was made for the collection and management of log files. Beyond log aggregation, it includes ElasticSearch for indexing and searching through data and Kibana for charting and visualizing data. Together they form a powerful log management solution.

When to use it: If you want an open-source tool. If you’re interested in implementing the entire Elastic stack or at least see value separately in using ElasticSearch or Kibana and want the interactive benefits that come from combining these tools.

Pricing: Free

Pros:

• Open-source tools come with a lot of control
• Uses 3 mature components to form 1 powerful solution
• Quick and easy setup for an open-source solution

Cons:

• Components need to be treated essentially as individual products
• Logstash filters are written in Ruby, Kibana is pure Javascript, and ElasticSearch has its own REST API as well as JSON templates

3. Sumo Logic

Sumo Logic was founded as a SaaS version of Splunk, going so far as to imitate some of Splunk’s features and visuals early on. Since then, Sumo Logic has developed into a full-fledged enterprise-class log management solution in its own right. Sumo Logic is the most enterprise-focused of the cloud-native log analyzers.

When to use it: If you’re an enterprise-type company but are willing to sacrifice some features for the benefits of SaaS, Sumo Logic is worth exploring. It’s also good if you have a strong focus on security. It’s not just developer-oriented as a tool either, with benefits for security teams and business purposes.

Pricing:

• Enterprise: $150/GB/month for minimum 3GB
• Professional: $90/GB/month for minimum 3GB
• Free: Free up to 500MB

Pros:

• Easy setup compared to on-prem solutions
• Advanced analytics and machine learning for logs, metrics and external data
• Baselines are tracked for anomaly detection purposes

Cons:

• Less extensive feature list than Splunk

4. Loggly

Loggly is a robust log analyzer, focusing on simplicity and ease of use. It’s targeted for developers and DevOps – making it less enterprise-focused.

When to use it: Primary use cases are for troubleshooting and customer support scenarios. It’s a good tool for a DevOps team.

Pricing:

• Enterprise: Starting at $349/month
• Pro: Starting at $199/month, up to 5 users
• Standard: Starting at $79/month, up to 3 users
• Free: Single User

Pros:

• Very developer friendly
• Easily create custom performance and DevOps dashboards

‍

Cons:

• Best suited for parsing app data for smaller Dev or DevOps teams
• Will not perform as full-blown infrastructure/security/analytics solution

5. PaperTrail

PaperTrail is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. It’s a SaaS tool designed to enhance the logs you already collect or generate.

When to use it: If you want a simple and straightforward tool without lots of extra bells and whistles. If you want a stripped down and basic log analyzer that is good for looking at log files in aggregation and doesn’t try to be anything more.

Pricing:

Ranges from $7/month for 1GB to $395/month for 50GB with option to customize a plan to suit volume requirements

Pros:

• Simple way to look at log files from multiple machines in a singular view in the cloud
• Quick setup
• Very affordable

Cons:

• Mostly text-based, UX is similar to looking at an actual log
• No advanced integrations, predictive, or reporting capabilities

6. Graylog

Graylog is an open-source log analyzer backed by MongoDB as well as ElasticSearch (similar to Logstash) for storing and searching log errors. It’s mainly focused on helping developers detect and fix errors in their apps, but they’ve also released an official enterprise-ready platform.

When to use it: Graylog is more targeted towards developers than other open source log management tools. Plus, if you want a log management tool that aims to be both enterprise-ready and is open source, Graylog definitely deserves to be in the mix with Elastic/Logstash.

Pricing: Free

Pros:

• Can handle an extensive range of data formats
• Fine grained control for authentication and user permissions
• Alerting for streams you designate makes for quick analytics
• Uses a REST API for distributing and being sent data

Cons:

• Not particularly management friendly on the dashboard front
• Reporting functionality is a bit lacking
• As a newer tool, particularly on the enterprise front, it’s not as mature and fully vetted as other options

Final Thoughts

That was a lot of information to absorb. Let’s just sum that all up with the basic use cases for each tool, shall we?

Splunk is the best “out-of-the-box” tool for enterprise companies where money is less of a concern. Elastic is the strongest open-source project with complex setup and maintenance being the downside. Sumo Logic is basically the SaaS version of Splunk but it’s less expensive and has a less-extensive feature list. Loggly is a solid solution for smaller Dev and DevOps teams focusing on monitoring and troubleshooting. PaperTrail is a simple and affordable tool for viewing log files from multiple machines in a singular view in the cloud. Graylog is a solid alternative to Logstash within the Elastic Stack framework.

One last final thought… These log management tools help you understand your log files, but only based on the information that your log files contain. In cases where an error occurs in production, the logs most likely won’t contain the data that you need (the source code and variable state, for example).

OverOps integrates with all of these tools by appending links to the logs leading to the code and variable state at the time of every error. So, instead of seeing just a single error line in the log and trying to use it to troubleshoot, you can get the complete picture.