When it comes to the quality of software, there is no getting around static code analyses, especially for safety-critical software. Roughly speaking, there is hardly any difference between all static analysis tools. They examine code without executing it, uncover defects and error-prone code parts in the software and generate warnings and reports. They are usually integrated into the development environment and CI/CD/build systems. However, if you want to successfully integrate a coding tool into daily development and get the most out of your investment, it is worth examining the options carefully.

Decision-makers often simply follow a uniform pattern: they let the tools in question process the same code, then compare the results and select the tool that reports the most rule violations. This is not a true evaluation, because the winner of this battle is not necessarily the tool that is best suited to setting up a sustainable and scalable static analysis process in a company. In fact, many key factors that make the difference between implementing a successful static analysis approach and another failed attempt are easily overlooked.

Needs analysis and current status

Before you start looking for a tool, you should take a clear look at the company and check the following things:

What is needed? In order for the static analysis to be successful, it must first be determined what it should actually achieve:

• Which critical points should it address?
• Do conformity requirements have to be met?
• Is the security aspect of the applications an issue?
• What is already being done in this regard?
• Who evaluates the analysis reports and must act on their basis?

Where does the company stand now? It is important to know which problems the tools are intended to solve and whether they fit the company:

• Is the current development process sufficiently stable, reproducible and streamlined to serve as a solid foundation for static analysis?
• What have been the results of previous evaluations or implementations of static analysis tools?

Criteria for the selection process

Selecting a static analysis tool for practical use and long-term integration into the development process requires effort and planning. Instead of a purely technical review, it is much more important to examine how well the tool fits into the company. It is also worth evaluating the provider who sells and supports the tool.

Criteria for evaluating the tools

The following criteria must be taken into account in the technical assessment of the tools in question:

• Coverage of the required guidelines,
• Quality of the built-in checkers for the required guidelines,
• Coverage of the standards specified by the industry or the company,
• Depth and breadth of analysis,
• practical ways to reduce the "noise", i.e. the rule violations reported by the checkers but which can be ignored,
• acceptable number of false positives and how to deal with them,
• acceptable number of false negatives,
• Effort to adapt the built-in checkers to the company policy,
• Effort to add new, individual checkers for special requirements
• Supported complexity level for new, specific checkers

Considerations regarding the provider

Just as important as choosing the right tool is choosing the right vendor, because the buyer inevitably enters into a relationship with the vendor. Behind the most successful tool deployments is always a vendor that is committed to helping the organization achieve its business objectives, overcome challenges and drive adoption of the tool. During the evaluation process, it is important to test the providers in several qualification and evaluation rounds:

• Are the provider's scaling, growth and foresight capabilities in line with the organization's own requirements and objectives?
• Does the provider have a coherent strategy for deployment across the organization and for evolving as the organization's needs change?
• What are the provider's best practices for using its tool?

You should also be aware of the provider's reputation on the market:

• Which companies are already using the tool?
• What do the case studies say about deployment, use and benefits of the tool?
• What do industry experts say in test reports, reviews and awards?

Quality or quantity: coverage counts

A common question is: How many checkers does the product have? This implies that the quality of a tool depends on how many different errors it detects - and is an inadequate assessment criterion, especially for static analysis tools. Rather, a user of static analysis tools should be concerned with how well the respective tool covers different error types and coding standards and how well-founded its analysis is. The key question is therefore: How well does a tool cover the types of coding problems that are relevant to your own company?