Blog
Service Reliability Management

The Top 10 Exception Types in Production Java Applications – Based on 1B Events | Harness Blog

Learn about the top 10 exceptions in production Java applications and how to avoid them.

TL;DR

Harness Service Reliability Management's analysis reveals that 97% of logged errors in production environments stem from just 10 unique exceptions, with NullPointerException leading at 70%. This insight enables teams to prioritize fixing the most impactful errors, significantly reducing error volume and improving system reliability.

The Pareto logging principle: 97% of logged error statements are caused by 3% of unique errors

We received a lot of feedback and questions following a recent data-crunching post where we showed that 97% of logged errors are caused by 10 unique errors. By popular demand, we’ll go a step deeper into the top exceptions types in over a thousand applications that were included in this research.

Let’s roll.

Without Further Ado: The Top Exceptions by Types


To pull out the data, we crunched anonymized stats from over a thousand applications monitored by Harness Service Reliability Management’s error analysis micro-agent, and checked what were the top 10 exception types for each company. Then we combined all the data and came up with the overall top 10 list.


Every production environment is different, R&D teams use different 3rd party libraries, and also have custom exception types of their own. Looking at the bigger picture, the standard exceptions stand out and some interesting patterns become visible.

1. NullPointerException – 70% of Production Environments

Yes. The infamous NullPointerException is in at #1. Sir Charles Antony Richard Hoare, inventor of the Null Reference was not mistaken when he said:
“I call it my billion-dollar mistake. It was the invention of the null reference in 1965… This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.”


With a top 10 spot at 70% of the production environments that we looked at, NPEs take the first place. At Harness, we actually have a special alert that lets us know whenever a new NullPointerException is introduced on our system.

2. NumberFormatException – 55% of Production Environments

In at #2 is the NumberFormatException which happens when you try to convert a string to a numeric value and the String is not formatted correctly. It extends IllegalArgumentException which also makes an appearance here at #3.


One easy fix to make sure that the input you’re passing to the parse method passes these regular expression:

  1. For integer values: “-?\\d+”
  2. For float values: “-?\\d+.\\d+”

3. IllegalArgumentException – 50% of Production Environments

Next up at #3, IllegalArgumentException, appearing at the top 10 exceptions in 50% of the production environments in this survey.
An IllegalArgumentException actually saves you from trouble, and thrown when you’re passing arguments from an unexpected type to your methods. For example, some method that expects type X and you’re calling it with type Y as an argument. Once again, an error that’s caused by not checking what you’re sending out as input to other methods.

4. RuntimeException – 23% of Production Environments

All exception objects in the top 10 list (Apart from Exception) are unchecked and extend RuntimeException. However, at #4 we’re facing a “pure” RuntimeException, where Java the language actually doesn’t throw any itself. So what’s going on here?
There are 2 main use cases to explicitly throw a RuntimeException from your code:

  1. Throwing a new “generic” unchecked exception
  2. Rethrows:
  3. “Wrapping” a general unchecked exception around another exception that extends RuntimeException
  4. Making a checked exception unchecked

One famous story around checked vs. unchecked and the last use case we described here comes from Amazon’s AWS SDK which ONLY throws unchecked exceptions and refuses to use checked exceptions.

5. IllegalStateException – 22% of Production Environments

In at #5, featured at the top 10 exceptions in 22% of over a 1,000 applications covered in this post is the IllegalStateException.
An IllegalStateException is thrown when you’re trying to use a method in an inappropriate time.


A more realistic Java example would be if you use URLConnection, trying to do something assuming you’re not connected, and get “IllegalStateException: Already Connected”.

6. NoSuchMethodException – 16% of Production Environments

16% of the production environments in this data crunch had NoSuchMethodException in their top 10.
Since most of us don’t write code while drunk, at least during day time, this doesn’t necessarily mean that we’re that delirious to think we’re seeing something that’s not there. That way the compiler would have caught that way earlier in the process.


This exception is thrown when you’re trying to use a method that doesn’t exist, which happens when you’re using reflection and getting the method name from some variable or when you’re building against a version of a class and using a different one at production.

7. ClassCastException – 15% of Production Environments

A ClassCastException occurs when we’re trying to cast a class to another class of which it is not an instance. 15% of production environments have it in their top 10 exceptions, quite troublesome.


The rule is that you can’t cast an object to a different class which it doesn’t inherit from. Nature did it once, when no one was looking, and that’s how we got the… Java mouse-deer. Yep, that’s a real creature.

8. Exception – 15% of Production Environments

In at #8 is the mother of all exceptions, Exception. Java never throws plain Exceptions, so this is another case like RuntimeException where it must be… you, or 3rd party code, that throws it explicitly because:

  1. You need an exception and just too lazy to specify what it actually is.
  2. Or… More specifically, you need a checked exception to be thrown for some reason

9. ParseException – 13% of Production Environments

Parsing errors strike again! Whenever we’re passing a string to parse into something else, and it’s not formatted the way it’s supposed to, we’re hit by a ParseException. Bummer.


It’s more common than you might have thought with 13% of the production environments tested in this posted featuring this exception in their top 10.
The solution is… yet again, check yo’ self.

10. InvocationTargetException – 13% of Production Environments

Another exception that’s thrown at us from the world of Java Reflection is the InvocationTargetException. This one is actually a wrapper, if something goes wrong in an invoked method, that exception is then wrapped with an InvocationTargetException.


To get the original exception, you’d have to use the getTargetException method.


We see that 13% of production environments tested in this post had it in their list of top 10 exceptions. The second exception type here that’s directly related to Java’s reflection features.

Final Thoughts

The world of Java exceptions is indeed quite colorful, and it’s amazing to see how much impact the top 10 exceptions have on our logs. 97% of all logged errors come from 10 unique exceptions.


Try Harness Service Reliability Management and find out what are the top 10 exceptions in your own production environment. It only takes a few minutes to get started and you’ll also get all the data you need in order to fix them. Source, Stack, State.

← Previous:
Next: →

Related Resources

How to Automatically Recover from Java Deadlocks in Legacy Production Systems

AI SRE

How to Automatically Recover from Java Deadlocks in Legacy Production Systems

August 3, 2021

Harness Team

+ more
Time to Read

The quick and dirty deadlock solution to get those threads off your back

What exactly is a code hack? We’ve asked developers: What’s the most useful debugging trick you use, what are some of the things you do that most developers aren’t aware of and how you managed to solve that issue that was bugging you for way too long. Basically, any non-straightforward solution using a piece of gum and a paperclip.

This post is the first story in the series, kicking it off with Uri Shamay(https://cmpxchg16.me/), Principal Lead System Software Engineer at Akamai, who shares a story of deadlock madness.

The Set Up

10 Years of Legacy Undocumented Dode with a Hint of Deadlocks

At a company I previously worked in, we were developing a system that dramatically decreased the lead time on some business logic at a bank. Operations that used to take 30 days to complete now only take a single day. The flow starts from a validation process where the system receives a customer and runs all kinds of checks against external government APIs and internal APIs. To get a hold of this data, the old model’s implementation used to invoke a native thread for each of these API fetches.

A tech lead decided this model for API calls can’t scale, so he moved on to a newer implementation with a Non-Blocking I/O using Java NIO that allowed him to create only a few threads that were based on CPU architecture. The new complex flow also contained JNI and DLLs for file system notifications when changes occur on a local path that belongs to a Message Queue system.

The Problem

The Curious Case of a Jstack Deadlock

The unexpected behavior occurred with the new deployment, but the tech lead that was previously in charge quit, so no one knew where the source code was and what it actually did. The bank’s authorization policy required us to use the existing API library to get the data from the government’s systems. Every few hours it would all stop functioning, and jstack told us there is a deadlock in place. Actually, there were a lot of deadlocks when we used those libraries, but we never fully understood why, as we haven’t seen any patterns in the jstack output. To top it all, some of the code was just Jars, JNI, and DLLs without the source files and no documentation, so I used the DJ – Java Decompiler for the Jars, and WinDbg for the DLL interception.

Deadlocks investigations are hard, and deadlocks in code you’re not familiar with are even harder. But Deadlocks that happen when other developers don’t understand concurrency patterns are the hardest! When I started to investigate the deadlocks in production I was sure it’s just a small issue and I can quickly fix it. That optimism quickly became a joke in the R&D group, with many developers going through the jstack output when no one really understood the pattern and the root cause for the deadlocks.

We looked for open-source tools to get more information that will help us, and tried some tracing tools but none of them helped. We were trying to get the state of the sytem, and reproduce it in a QA environment but the data that we simulated wasn’t diverse enough, so the deadlocks didn’t reappear. Since real production data couldn’t be provided due to a regulation policy we reached a dead end.
[adrotate group=”11″]

“We didn’t want DevOps to kill us”

After months of problems and manual restarts in the middle of night, DevOps jokes 24x7, budget concerns and a new strict deadline for the project, we decided to rewrite the problematic libraries. They contained so many problems that we needed to fallback to the old version with native threads for each call. After a deep analysis of the code, we understood this not the behavior we need. No running away from rewriting the massive codebase. Right now we’re working on rewriting everything from scratch.

The Workaround

So, we decided to rewrite everything, but in the meantime, we didn’t want devops to kill us, so we wrote some JMX code that polls ‘findDeadlockedThreads’ and does an automatic restart when it happens. This is how we automatically restarted the app after running into a deadlock. The DeadLockDetector thread polls the JMX of findDeadlockedThreads and exits brutally by System.exit($CODE) when a deadlock is detected:

The error code is then returned to the main loop and in case it’s the deadlock code, the app will be executed again from scratch. Note that you can use that JMX external to your program with Remote Management in case you don’t have the source or you want to do it on a live system:

** The full code example with a simulated deadlock is available on GitHub.

Get Started

Get Started with Harness AI

Try the full platform free. No module restrictions, no credit card.

Harness Team
Harness
Harness delivers intelligent AI automation, so your team ships code faster, safer, and smarter.
harness-team
Harness Team
https://www.linkedin.com/company/harnessinc/