Software is gradually defining everything, and its forms are becoming increasingly diverse. Software is no longer limited to the applications or apps we see on computers or smartphones. It is now an integral part of hardware devices and many unseen areas, such as cars, televisions, airplanes, warehouses, cash registers, and more. Besides sensors and other electronic components, the actions and data of hardware often rely on software, whether in small amounts of code or in hidden or visible forms.
Regardless of the type of software, the development process inevitably encounters bugs that need to be identified and fixed. While major bugs are often detected and resolved before release or deployment by developers or testers, security vulnerabilities don’t always receive the same attention. It’s akin to a driver rushing to get passengers to the airport, only to forget to properly secure the trunk — an issue that’s easy to overlook.
Between 1992 and 2005, at least 52 people in the United States lost their lives due to unintended acceleration issues with Toyota Corolla vehicles. Subsequent code audits of Toyota’s software revealed multiple design and implementation flaws, including buffer overflows, invalid pointer references, and race conditions.
Michael Barr, the security expert responsible for auditing the code, stated in his report:
“System safety can’t be an afterthought. It must be designed from the very beginning into a system.”
Over my years in the security industry, whether conducting penetration testing, security assessments, or code audits, I have encountered various security vulnerabilities and risks. These can be summarized into seven categories:
Misunderstanding of Security Protection Technologies
This issue arises when engineers have only a superficial understanding of security technologies during system design. As a result, even though security measures are implemented, they fail to provide effective protection.
A common example is the use of encryption or hashing for data. MD5 is so well-known that many developers instinctively think of it when hashing user passwords. However, they often apply only a single round of MD5 computation, which is insufficient.
In an internal technical sharing session, a colleague discussed the frontend encryption of a website. The site encrypted account information on the frontend before sending it to the server. Although the JavaScript code was slightly obfuscated, the structure and logic remained clear: what appeared to be RSA encryption was, in fact, simple DES encryption. Moreover, the encryption process was obfuscated by combining plaintext and ciphertext strings.
In this case, the site failed to implement proper slow encryption, used the outdated DES algorithm, and employed a custom encryption structure — all of which deviated from best practices.
Another common issue involves misunderstanding security configurations. For instance, many enterprise email systems configure SPF (Sender Policy Framework) but neglect to configure DMARC (Domain-based Message Authentication, Reporting, and Conformance). This incomplete setup fails to prevent email spoofing effectively.
Before adopting any security technology, it is crucial to thoroughly understand its applicability and best practices. This principle applies to security products as well. Blindly using popular solutions or relying on superficial knowledge can lead to suboptimal outcomes.
Component Integration and Hidden Security Designs
Integrating third-party SDKs during system design and development is common. However, the security of these third-party components is often not evaluated in the context of their application scenarios. Additionally, these components themselves may contain vulnerabilities, and developers may lack the resources or expertise to conduct security testing. Consequently, systems built on these components share their risks.
For example, a hotel business plan involved using facial recognition devices for unattended check-ins. Customers could book a room online and verify their identity by scanning their ID card and face at the front desk. Testing a device from a well-known AI company revealed that by exploiting a third-party input method, the system’s recognition mechanism could be bypassed to access the device’s management backend. This allowed viewing photos of scanned IDs and faces stored in the system directory, posing significant privacy risks.
Similar issues exist in many intelligent devices found in public spaces, such as toy or beverage vending machines and shopping guide systems. Additionally, vulnerabilities in third-party open-source components or frameworks, such as Log4j, Fastjson, or Struts2, are also common. These illustrate why ensuring and tracing software supply chain security can be challenging. By analyzing the past vulnerabilities, support, and development history of supply chain components, their reliability and security can be evaluated, informing future decisions.
Furthermore, system designs often fail to fully consider application scenarios. Unnecessary features may be retained or exposed, such as including interfaces for future versions in current code or merely commenting out testing environment interfaces. Developers often assume that nobody would bother probing these interfaces. However, the principle of “security by obscurity” is a flawed understanding of protection; any hidden functionality is likely to be discovered sooner or later. As the saying goes, “If you don’t want others to know, don’t do it.”
Ignoring Security in System Design
This issue frequently arises in early foundational internet protocols. The pioneers of computing and networking, often physicists or mathematicians, did not anticipate the complex security challenges we face today. Similarly, developers who lack knowledge of network and system threats may overlook security during design and development. Sometimes, security designs are even abandoned due to conflicts with business requirements.
For instance, a well-known domestic CRM company allows customers to write and execute code to flexibly solve their data processing or business workflow needs. Although the code is simplified and encapsulated, it is still possible to craft payloads to trigger XSS vulnerabilities, enabling silent data exfiltration without manual downloading.
Similar problems exist in foundational network protocols (e.g., DNS, NTP, ARP, HTTP, IP, FTP) and basic software or frameworks (e.g., Redis, MongoDB, Flask). Many of these either lack security considerations or fail to follow security-by-default principles.
The value of threat modeling lies in analyzing potential security risks based on requirements and design, then addressing these risks through thoughtful security design to mitigate, transfer, or eliminate them, thereby reducing the cost of security implementation and maintenance.
Security Risks from Poor Exception Handling
One of the principles of security design is fail-safe, which ensures that systems remain secure even during failures or anomalies. However, exception handling is often neglected in system design and implementation. For developers, exception handling may seem unrelated to security, primarily serving to maintain normal operations or assist in diagnosing issues. In reality, poorly designed exception handling can lead to security issues like denial-of-service attacks or other vulnerabilities, as business continuity itself is an aspect of security.
For example, a system required users to input a series of numbers. The system used a linked list to store and sort the numbers, with a time complexity of O(nlogn). If an attacker inputted an excessively long string of numbers, the sorting process could cause the system to hang or crash.
In another instance, during a trip, my wife interacted with a tourism navigation screen at an airport. By continuously swiping the page up and down, she eventually triggered an error message, bypassing the display application and accessing the Windows operating system.
Exception handling design involves more than throwing an exception. It also requires validating input, managing system failures or crashes, maintaining security during downtime, and planning appropriate responses.
Discontinuous or Inconsistent Trust Relationships
In software function design or implementation, trust relationships often exist between functionalities or components due to business needs. For example, when using Function A, it is assumed that Function B, which A relies on, is also secure. Similarly, Component A is assumed to be safe because it is associated with Component B. However, in complex business scenarios, such trust relationships can break down due to complexity or iteration, leading to cascading or indirect security risks.
For example, during a security evaluation of a system, we found a feature designed to protect files by encrypting them, preventing unauthorized access. However, once a protected file was opened, it could be saved as plaintext using the “Save As” function without any restrictions. This inconsistency in trust relationships between file protection and the save function compromised the entire security mechanism.
Effective security protection requires comprehensive design and evaluation, considering system requirements, functionality, and trust relationships. Weak links in trust relationships can render protection measures ineffective.
Over-Reliance on Single-Point Security Measures
The principle of defense-in-depth emphasizes building multiple layers of security, similar to safeguarding a man’s hidden stash beyond neighborhood security and household measures (e.g., safes, phone passwords, and account credentials). Relying on a single security measure is risky, as it can be bypassed or compromised.
For example, in a review of a domestic signal system, we identified potential vulnerabilities that could allow traffic signal manipulation under specific circumstances. While these vulnerabilities were acknowledged by traffic experts, the manufacturer downplayed them, believing the physical security of the signal box at intersections was sufficient. However, relying solely on a simple physical lock for protection is far from adequate.
No matter the operational environment or context, security measures at the system design level must follow defense-in-depth strategies while balancing implementation costs.
Insufficient Assessment of Scenarios or Environments
A significant source of security risks arises from the specific operating environment and usage scenarios. If these are not clearly understood during system design, addressing the resulting vulnerabilities may become prohibitively expensive or even infeasible. Threat modeling fundamentally involves creating abuse cases to simulate scenarios where users deviate from normal use or act maliciously.
For example, a bank required visitors to fill out personal information and health details via a link sent to their phone. However, the system did not consider that recipients might forward the link to others, allowing them to access sensitive personal and health data.
Similarly, an online lottery system for a company’s year-end event had a flaw in prize distribution logic. Employees who drew later had a higher chance of winning, as the pool of prizes gradually increased in proportion to remaining participants. Recognizing this, I deliberately participated late and secured a second-place prize.
Security is inherently tied to business attributes, necessitating an understanding of the scenarios, environments, and user base. Security design aims not only to protect systems but also to prevent user risks. Collaborating with business teams to evaluate the rationale and necessity of security designs is both logical and essential.
Conclusion
To minimize security risks and vulnerabilities in software design and development, one must possess solid technical expertise and a robust background in security offense and defense. Developing secure software is akin to crafting fine art — it requires meticulous thought, constant consideration of potential threats, and thoughtful design solutions. This makes upfront security design critically important.
