Why SAST Tools Trigger False Positives

All automated testing solutions produce results when run against a target system. That’s expected. What’s more interesting about this process is that we want these results to be as related as possible; we don’t want false alarms.

False alarms or in more technical terms false positives (FPs) are the anticipated byproducts of any automated testing. But, what makes a static application security tool to produce them? In this post, we’ll try to pinpoint some of the key sources of FPs and what the tools do about this situation.

The obvious reason that nobody likes false alarms is that because they create clutter and we have to spend more time than we have to analyzing those results.

Unknown Protection Mechanisms

A dynamic query construction doesn’t always mean that we are dealing with a SQL Injection vulnerability. Most important of all, one of the dynamic parts should stem from an untrusted (tainted) input source, such as the end user.

Even with this taint analysis in-place, there might be a validation method that neutralizes or worse blacklists any SQL meta characters in the input at runtime.

A static code analysis tool can’t possibly infer the validation tracing through with the data flow in the code. Blacklisting is a bad practice of doing validation, so, in case of a solid neutralization method, the tool will probably produce an FP.

This problem is solvable since the tools are flexible enough to be taught about the validation methods. So that, they will realize the validation when it’s applied on a traced tainted data and suppress their findings.

This is cool, however, there are still many factors that make a weakness a vulnerability; such as hardened configuration directives, the version number of the framework or third party libraries used, etc.

Nevertheless, security static code analysis tools analyze configuration files and apply software composition analysis process to find out unpatched code.

Unpredictable Semantics of String Operations

Strings are the most dangerous data type that are used as a medium resulting in serious software security bugs. All of the injections bugs, such as XSS, SQL Injection, LDAP Injection, XPath Injection, OS Command Injection, Open Redirect, etc. are the result of using unvalidated strings.

However, the operations applied on a string may be the decisive criteria whether an reported issue is really a vulnerability, that is it really exists or it’s just another FP.

Here’s an example code;

Runtime.getRuntime().exec(userProvidedInput

We have a weakness called OS Command Injection in Java here where the user can execute any command he/she chooses on the target operating system. How about the code below?

Runtime.getRuntime().exec("java -version " + userProvidedInput

As per Java’s API the malicious user will not be able to trigger OS Command Injection here. Since Java implementation of exec will tokenize the string, take the first token as the command and rest as its arguments. True for certain prefix command that might still pose a problem, but not this particular one, hence a source of FP.

Most of the static analysis tools stay on the safe side with these problems. They still consider this to be a source of weakness that should be addressed.

Over Assumptions on Hardcoded String Values

How a tool understand whether a string variable name holds a sensitive data, such as passwords, keys, authentication tokens at runtime? Well, one popular way of doing it is looking for certain words in these variable names.
For example, if a variable name contains “password” in it, then with a high probability it will contain a sensitive data at runtime. But this is just a probability and result in bad looking FPs.

The same goes with efforts finding out privacy violation issues by trying to locate sensitive data being sent to client side.

Let’s look at the code below;

myPasswordLabel.Text = "Enter your password"

Here the component name contains “password” in it, however, that doesn’t really mean that the text it is set is really a password. This is just a label. There might be some venues for a static analysis tool to further use to give such a precise looking decision, however, this is still another source for FPs.

Weakly Typed Languages

What makes a static analysis tool to decide whether a variable holds an untrusted value or not. First of all, it has to figure out the ways that untrusted input enters to the program.

Let’s look at the JavaScript code below.

function MassSale(sales, installments, products)
{
   // process the sales
}

If there’s no way for a static analysis tool to understand the parameters to MassSale function comes from an end-user, then it’s impossible to trace them inside the code and find possible weaknesses.

One way of understanding this is querying the name MassSale from a predefined list of function names that accept end-user requests with parameters. This is possible for strongly typed languages where you can query a function’s signature in a pre-determined, or inherited class. This way the tool can understand that MassSale is an inherited public member of a certain controller class, therefore it should be a function accepting user provided parameters.

Conversely, this is not always possible for weakly typed languages, since you can’t always query the signature of a callback function accepting untrusted inputs.

Dynamic languages such as JavaScript, PHP or Python makes it hard for a static analysis tool to deduce the semantic model of the code in question. Obviously, the assumptions here for a tool are a source of FPs.

Approximation & Missing Sensitivites

A fundamental source of FPs in security testing automation is due to approximation pursued when designing such tools. Approximations is the set of assumptions that has to be made in order to make a testing tool scalable and meaningful.

Sensitivities are one way of describing these approximations. Here’s a simple example from a previous post;

a = 3; b = 5;
input = String.Empty;
if (a < b)
{
    input = GetUserInput();
}
else
{
    input = String.Empty;
}
Execute(input);

When the above code is analyzed manually, we can see that the variable input can never be set to an untrusted input, since a is smaller than b. If a static code analyzer report an issue here, it is a false alarm. What if the values for variables a and b had dynamic values? This makes the static code analysis a real challenge and force any design to make assumptions, here called certain sensitivities.

Here, in order not to miss a valid issue a static analysis tool assumes that both branches of the if statement will eventually work so, at some point in runtime, the input executed by the last line will stem from an untrusted user leading to a vulnerability. This is definitely a source of FP.

Dealing with False Alarms

It’s a generally accepted design choice to have an accepted number of false alarms than having real alarms missed instead. Those are missed opportunities.

I would rather have a COVID-19 test which falsely diagnose an infection than to miss a real one. But, of course, in meaningful ratios.

Sure it’s not a beautiful sight to get an awful amount of FPs after hours of manual analysis on reported issues. So, SAST or DAST, automated solutions try their best to lower the number of false alarms they report. But they have valid reasons for the inevitable ones. Moreover, they also provide additional mechanisms to make these “errors” bearable for the end-users by choosing smart sensitivity design choices and applying different levels of trust on the reported issues.