Static Source Code Security Scanning Tools Evaluation Benchmark

1. Background Introduction

1.1 Purpose

Static Source Code Security Scanning Tools or Static Application Security Testing (SAST) Tools
Static Application Security Testing (SAST) tools are designed to analyze application source code, bytecode, or binaries to identify security vulnerabilities. These tools do not require program execution during analysis and detection, enabling efficient code security auditing based on static code review. As a result, they facilitate early-stage code auditing during software development and testing phases, allowing teams to detect and resolve security vulnerabilities proactively.

Currently, numerous commercial and open-source source code scanning tools are available in the market, and organizations are increasingly prioritizing source code security. However, the absence of a comprehensive evaluation benchmark for static source code security scanning tools has left enterprises without clear criteria for selecting and procuring such tools.

The purpose of this benchmark is to establish a comprehensive, objective, and neutral evaluation framework for static source code security scanning tools. It serves as a reference for enterprises, institutions, or teams to assess and select SAST tools that align with their security requirements and operational workflows.

1.2 Scope

This benchmark exclusively evaluates software tools that statically detect security defects or vulnerabilities in source code, binary files, or compiled files without executing the program.
Included:

• Tools analyzing source code, binaries, or compiled artifacts through static methods.

Excluded:

• Tools scanning non-code mediums (e.g., databases, documents).
• Tools employing dynamic analysis (e.g., runtime code execution, behavioral monitoring).

1.3 Technical Background

Early intervention in identifying and remediating security risks and vulnerabilities during the software development lifecycle yields significant returns, driving the growing emphasis on shift-left security (integrating security practices into early development stages). However, current security analysis and detection tools remain limited in fully addressing architecture-level security, quality, or correctness issues in software systems. Vulnerabilities identified through manual security testing or dynamic vulnerability testing still require source code modifications for resolution.

Static analysis techniques, by contrast, enable comprehensive, direct, and simplified identification of security vulnerabilities and defects in source code, presenting findings clearly to both security professionals and developers.

As a result, static source code security scanning tools remain indispensable in security testing. To maximize their effectiveness, these tools must align with software design principles and continuously refine detection rules to enhance accuracy and coverage. This integration ensures vulnerabilities are detected and addressed efficiently, minimizing risks throughout the software lifecycle.

2. Normative References

The following documents were referenced during the development of this benchmark:

• GB/T 25069–2022: Information Security Technology — Terminology
  A Chinese national standard defining terminology for information security technologies.
• GB/T 39412–2020: Information Security Technology — Code Security Audit Specification
  A Chinese national standard outlining principles and methodologies for code security auditing.
• OWASP Top 10:2021
  A globally recognized guide by the Open Web Application Security Project (OWASP), listing the ten most critical web application security risks.
• OWASP ASVS (Application Security Verification Standard)
  A security verification framework by the OWASP Foundation, defining security requirements across three levels (Level 1 to Level 3) to help organizations build secure web applications.

Additionally, this benchmark incorporates insights from:

• Industry Standards and Best Practices
  Relevant industry standards and best practices for static source code security scanning.
• Open-Source Projects and Communities
  Documentation, guidelines, and best practices from open-source static source code security scanning tools and their communities.

3. Terms and Definitions

• Security
  The properties of a system that ensure confidentiality, integrity, availability, auditability, authenticity, and reliability.
• Security Function
  Functionality designed to meet security requirements and correctly implement corresponding security policies.
• Risk
  The effect of uncertainty on objectives.
• Vulnerability
  A weakness, flaw, or non-compliant component in an information system that can be exploited by malicious actors to cause unauthorized access, data breaches, service disruption, or other security issues.
• SAST (Static Application Security Testing)
  A technique for analyzing application source code, bytecode, or binaries to identify security vulnerabilities without executing the program.
• OWASP (Open Web Application Security Project)
  A global community-led initiative focused on improving the security of web applications through open-source tools, resources, and best practices.

4. Evaluation Benchmark Overview

The evaluation benchmark defines the functional categories that static source code security scanning tools should support. These categories determine the applicability, detection capabilities, and compatibility of the tools for enterprises, institutions, or teams when conducting source code security scans.

The benchmark for static source code security scanning tools includes the following components:

• Deployment Environment
• Security Scanning
• Vulnerability Detection
• Source Code Support
• Extension and Integration
• Product Interaction
• Report Output

5. Evaluation Benchmark Specifications

5.1 Deployment Environment

The supported deployment environments of a tool determine its adaptability and compatibility across different usage scenarios, including:

• End-user environments: Testing and usage by individual users on local machines.
• Development and testing environments: Server-side testing and usage in development or QA setups.

5.1.1. Operating System Support

The tool must support mainstream operating systems to meet diverse development and testing needs, including but not limited to:

Operating System Support

5.1.2. Containerization Support

The tool should provide seamless integration with containerized workflows, such as:

• Docker: Enable scanning within Docker containers for CI/CD pipelines.
• Kubernetes: Optional support for orchestrated environments.

5.2 Security Scanning

5.2.1. Scanning Speed

Scanning speed refers to the efficiency of vulnerability detection in source code. Faster scans reduce waiting time for security teams and accelerate vulnerability discovery and feedback. Therefore, faster scanning speeds are prioritized for static source code security scanning tools.

The tool must support scanning for at least 3 mainstream programming languages (e.g., Java, Python, JavaScript, C/C++, C#), and performance consistency across languages must be ensured.

The tool should efficiently handle codebases of varying sizes:

• Small Projects: <500k lines of code (LoC).
• Medium Projects: 500k–5M LoC.
• Large Projects: >5M LoC.

Based on industry standards, acceptable scan durations are:

Scan Duration Benchmarks

5.2.2. Scanning Configuration

To enhance adaptability across diverse testing and scanning scenarios, static source code security scanning tools should include the following configuration capabilities:

Scanning Configuration Capability

5.2.3. False Positive Rate(FPR)

The False Positive Rate refers to the percentage of incorrectly flagged vulnerabilities in scan results relative to the total number of reported vulnerabilities:

FPR Definition

The FP Rate is a key metric for assessing the precision of static application security testing (SAST) tools and a primary concern for users. Excessive false positives increase the time and effort required for security or development teams to validate or triage alerts, thereby extending vulnerability remediation cycles, raising costs, and reducing stakeholder motivation to address security issues. Persistently high FP Rates may erode developer confidence in static analysis tools.

Variables such as code complexity and framework design inherently impact FP Rates, thus, it is impossible to entirely eliminate false positives or maintain an extremely low FP rate.

Based on industry research, an FP Rate of 10%–20% or lower is considered optimally acceptable for static source code security scanning tools in development environments.

5.2.4. False Negative Rate(FNR)

The False Negative Rate refers to the percentage of genuine vulnerabilities not identified by the scanning tool relative to the total number of actual vulnerabilities:

FNR Definition

The FN Rate is a critical metric for evaluating the accuracy and reliability of static application security testing (SAST) tools. A high FN Rate indicates that the tool fails to detect legitimate vulnerabilities, introducing security risks to applications. Conversely, a low FN Rate reflects fewer missed vulnerabilities in scan results. Consequently, the False Negative Rate and False Positive Rate are the two most critical performance indicators for users of security scanning tools.

Based on industry survey reports, an FN Rate of 20%–30% or lower is generally deemed acceptable for enterprise-grade static source code security scanning tools.

5.2.5. Compiled Code Support

Security vulnerability scanning of source code may encounter situations where the source code is unavailable, particularly with compiled files in C/C++ or other compiled code results within a source code project, such as:

• Precompiled linked libraries or components embedded within source projects
• Compiled C/C++ binaries
• Third-party dependencies distributed solely in binary form

Therefore, static source code security scanning tools support scanning of binaries or bytecode after code compilation, enabling the detection of security vulnerabilities or risks in various forms of static files and improving the coverage of static source code file scanning.

5.2.6. Mobile App Support

Static source code security scanning tools should be capable of supporting security scans for mobile application source code, including but not limited to:

• iOS Applications: Support for Swift, Objective-C, and associated frameworks (UIKit, CoreData)
• Android Applications: Analysis of Java, Kotlin, and Android SDK components
• Cross-Platform Frameworks: Compatibility with React Native, Flutter, and Xamarin codebases

5.2.7. Vulnerability Rule Customization

Static source code security scanning tools must support granular customization of vulnerability detection rules to adapt to specific use cases, code patterns, and framework implementations. This capability ensures efficient vulnerability discovery, reduces false positives, and prevents overwhelming users with non-actionable results.

5.2.7.1. Rule Modification Capability

Tools shall allow modification of existing detection rules when:

• Source code employs design patterns or frameworks that mitigate vulnerabilities unrecognized by default rules.
• Custom implementations render generic rule logic inaccurate or irrelevant.

The following are common scenarios where the source code has taken a specific design and approach that requires a vulnerability rule modification:

• Unrecognized Input Sanitization Methods: Custom input validation logic bypassed by default rules
• Unrecognized Output Encoding Mechanisms: Proprietary encoding functions not mapped to secure output practices
• Custom Filtering Functions: Internal sanitization routines not accounted for in rule logic
• Proprietary Database APIs: Unique query builders or ORM implementations misclassified as unsafe
• Unsupported Frameworks: Detection gaps for niche or legacy frameworks
• Rule Logic Errors: Overly broad or context-insensitive pattern matching

5.2.7.2. Rule Creation Capability

Static source code security scanning tools must enable users to define new vulnerability detection rules to address emerging threats or framework-specific risks not covered by default rule sets.

Mandatory components for custom rules should includes:

• Vulnerability Type Name: canonical identifier.
• Vulnerability Description: contextual impact analysis and exploitation scenarios.
• Detection Logic: code pattern matching (AST-based or regex), data flow analysis constraints or framework-specific heuristic checks.

5.2.8. Vulnerability Tagging

Vulnerability tagging refers to the ability of static source code security scanning tools to automatically label, classify, and archive detected vulnerabilities.

5.2.8.1. Tagging Vulnerabilities:

Mark identified vulnerabilities as “detected” and provide detailed information, including:

• Severity level
• Code location (file, line number)
• Vulnerability type (e.g., SQL injection, buffer overflow)
• Contextual code snippets

5.2.8.2. Classifying Vulnerabilities:

Categorize vulnerabilities by:

• Severity level (Critical/High/Medium/Low)
• Type (e.g., injection flaws, misconfigurations)
• Ownership (e.g., team, component)
• Enable filtering and prioritization based on impact and remediation urgency.

5.2.8.3. Archiving Vulnerabilities:

Archive resolved vulnerabilities to:

• Track remediation history (fix dates, patch versions)
• Analyze trends (e.g., recurring vulnerability types, MTTR metrics)
• Generate audit reports for compliance purposes

The benefits of vulnerability tagging including:

• Improves vulnerability management efficiency and accuracy.
• Facilitates data-driven analysis of security risks.
• Supports historical tracking and compliance reporting.

5.3 Vulnerability Detection

5.3.1. Supported Vulnerability Types

Static source code security scanning tools must cover common security vulnerability types encountered during system design and development. While vulnerabilities can be classified in multiple ways, the OWASP Top 10 — a widely recognized industry-standard classification based on real-world impact and exploitability — serves as the primary benchmark. Tools shall detect the following high-risk, low-attack-cost vulnerabilities from the OWASP Web Top 10 2021:

(1) Broken Access Control

(2) Cryptographic Failures

(3) Injection

(4) Insecure Design

(5) Security Misconfiguration

(6) Vulnerable and Outdated Components

(7) Identification and Authentication Failures

(8) Software and Data Integrity Failures

(9) Security Logging and Monitoring Failures

(10) Server-Side Request Forgery (SSRF)

Note: The above 10 categories are derived from the OWASP Web Top 10 2021 project, which periodically updates its threat models.

5.3.2. Vulnerability Information Requirements

For each detected vulnerability, the tool shall provide clear, actionable documentation to facilitate understanding and remediation:

• Vulnerability Name: Canonical identifier (e.g., “SQL Injection”)
• Vulnerability Description: Concise explanation of the flaw and its root cause
• Impact Analysis: Potential consequences (e.g., data leakage, privilege escalation)
• Remediation Guidance: Step-by-step fixes, including secure code examples and configuration adjustments

5.3.3. Development Framework Support
Tools must analyze codebases built with diverse frameworks by:

• Syntax Parsing: Accurately interpreting framework-specific syntax (e.g., Angular directives, React hooks)
• Dependency Mapping: Identifying framework dependencies (name, type, version) and associated risks
• Framework-Specific Rule Sets: Detecting vulnerabilities unique to popular frameworks (e.g., Spring, Django, Laravel)

5.4 Source Code Support

5.4.1. Supported Programming Languages
Static source code security scanning tools must support mainstream programming languages to address the increasing diversity of software development ecosystems. Required language coverage includes:

• Python
• PHP
• Java/JSP
• C/C++
• C#
• JavaScript
• .NET
• Objective-C
• Go
• Swift

5.4.2. Source Code Import Methods
Tools shall accommodate diverse codebase ingestion scenarios by supporting:

• Local Upload: Accept compressed source packages (ZIP, RAR, TAR.GZ)
• Repository Integration: Pull code directly from version control systems (Git, SVN, Mercurial)
• Snippet Analysis: Analyze partial code fragments via CLI or IDE plugins

5.5 Integration Capabilities

To reduce operational overhead and enhance SDL/DevSecOps adoption, tools must integrate with critical development lifecycle systems.

5.5.1. Source Code Management (SCM) Integration
Static source code security scanning tools must support integration with source code management systems (or code hosting platforms). By integrating with version control tools such as Git or SVN, the tool should automatically retrieve source code based on preconfigured version management settings without manual intervention.

Automated code retrieval is critical for reducing human effort and time in the source code scanning lifecycle while improving test coverage. Integrating with source code management systems to obtain code files also aligns with security requirements, mitigating the risk of code leakage associated with manual source code uploads.

5.5.2. Issue Tracking System Integration
Static source code security scanning tools must integrate with defect tracking systems to streamline vulnerability remediation workflows. This includes:

5.5.2.1. Single Vulnerability Synchronization:

Submit individual vulnerabilities to the tracking system with the following details:

• Vulnerability Name
• Vulnerability Type (e.g., SQL Injection, XSS)
• Description (root cause and attack vector)
• Impact (e.g., data breach, privilege escalation)
• Location (file path, line number, code snippet)
• Remediation Guidance (specific code fixes or configuration changes)

5.5.2.2. Bulk Vulnerability Synchronization:

• Export all detected vulnerabilities in a project to the defect management system with identical metadata fields as above.
• Support batch operations to minimize manual effort in large-scale projects.

5.5.3. Continuous Integration (CI) System Integration
To enhance security scanning efficiency during testing workflows, static source code security scanning tools must provide integration capabilities with continuous integration systems such as Jenkins.

5.6 Product Interaction Modes

To adapt to diverse software development workflows and tooling environments, static source code security scanning tools must support the following interaction modes:

Interaction Modes

5.7 Report Generation Requirements

Static source code security scanning tools must have the capability to generate security vulnerability scan reports. The output reports must include, at minimum:

• Vulnerability Statistics (including severity levels and vulnerability types)
• Vulnerability Locations
• Vulnerability Descriptions
• Code Snippets
• Remediation Recommendations

These elements help report users understand the overall status of vulnerabilities in the source code, quickly locate vulnerabilities, and facilitate remediation.

The tool should support exporting reports in multiple file formats, including but not limited to:

• PDF
• HTML
• Word
• Excel
• XML

Release Date: May 30, 2023

Guidance Organizations: OWASP China, ASA Application Security Alliance

Acknowledgments:

• Pei Weiwei (InsBug Lab)
• Xie Lifei (InsBug Lab)