A Security Stack - Part 2

Smart contract development is a relatively new discipline of computer science. It has unique constraints such as compute limits, high IO costs, and full program transparency. Traditional software development allowed developers to hide certain parts of their applications from end users and attackers, and make upgrades very quickly. However, this is not the case for smart contract systems. These programs operate autonomously on the blockchain, with millions and sometimes billions of dollars in assets. Oftentimes, they have limited capacity for upgrade even if issues are discovered. Consequently, developers must adopt an adversarial mindset, proactively thinking through all the ways a security incident can occur.

A helpful mental model is to think about development and a stack of tools that can be used to harden a smart contract system. Each tool a developer incorporates adds an additional layer of protection, increasing the likelihood of detecting and fixing critical bugs before an attacker can exploit them. The lower the layer at which a vulnerability is discovered, the less costly it is to remediate. In an ideal world, where a project is infinitely resourced, all of these layers of the stack can be implemented. In the real world, with resource constraints, developers must pick and choose which items have the highest impact for their project's needs.

First consider the most foundational layer, threat modeling. This layer is often missed, and allows a developer to construct a mental map of system components most likely to contain vulnerabilities. This could be public or external functions that involve changing state across multiple contracts, or functions that handle user funds. Having a keen understanding and intuition around where things are likely to go wrong can help in designing and redesigning a system if needed. This layer does not directly prevent bugs, but it can help in the subsequent layers to find issues that are further beneath the surface.

Building on this foundation is the first layer of defense: unit testing. Now the developer has a good understanding of where things are likely to go wrong from threat modeling. The first layer of active defense against bugs is to write simple unit tests. Writing these tests are table stakes for building smart contract systems.

The subsequent layer, integration tests, are crucial for simulating code execution against a chain-forked network. This layer reveals discrepancies between developer expectations and actual blockchain behavior. Notable examples of bugs that can be caught this way include using a solidity version above 0.8.19 which fails due to the absence of the PUSH0 opcode in layer-2 scaling solutions such as Arbitrum, Base, and Optimism. Additionally, there are issues specific to non-EVM equivalent platforms: for instance, zkSync did not support the Solidity language feature .transfer(), which led to raw Ethereum getting stuck in a contract. These nuanced issues underscore the importance of this testing layer in identifying challenges not evident during the unit testing phase.

At the third layer, we introduce fuzz and invariant tests, employing advanced tools and methods to probe for elusive vulnerabilities. Tools like Foundry natively support these tests but don't utilize symbolic execution for identifying issues. On the other hand, stateful fuzzers like Echidna use symbolic execution to challenge both contract and system-wide invariants. Another valuable resource is Hevm, originally developed by dapptools engineers and now maintained by the Ethereum Foundation. This tool rapidly identifies property violations on contract bytecode. Symbolic tests are particularly effective for DeFi primitives with minimal external dependencies. However, their utility diminishes in larger, monolithic applications that integrate directly with other protocols. Combined with prior layers, these advanced methods can help find non-obvious issues.

Proceeding to the fourth layer, mutation testing tools come into play. These tools, like Certora's Gambit, alter the program code and check if the existing unit and integration test suite identifies the mutated code change by failing. Failing tests after code mutation confirms the test suite's quality.

The fifth layer employs static analysis tools such as Slither, MythX, and Pyrometer to identify low-hanging issues in the codebase. While these tools have a high false positive rate, they offer valuable insights with a relatively small time investment. For example, just an hour spent reviewing Slither's output could flag high-severity issues like reentrancy vulnerabilities or divide-before-multiply problems. A recent study from the Imperial College of London indicated that static analyzers could have prevented $149 million in losses related to reentrancy issues alone, had their output been properly reviewed and findings remediated. This layer is fast due to its automated nature, but requires human review to ensure the accuracy of its findings.

Adding more depth, the sixth layer involves internal code reviews conducted by another engineer within the organization. This second set of eyes provides valuable feedback on design, identifies potential issues in the code by walking through things line by line, and questions the assumptions made by the developer. It serves as a comprehensive review of everything done or not done from the prior layers, finding bugs, filling gaps in test coverage and highlighting design and architecture flaws. This is a recommended template for structuring synchronous code reviews.

The seventh layer introduces external reviewers or auditors in a time-boxed engagement, offering an additional, unbiased layer of scrutiny. These experts meticulously examine the codebase for vulnerabilities and design flaws. If a significant number of issues are identified at this stage, it often signals a lapse in earlier phases of the Software Development Life Cycle (SDLC). For example, if 10 critical or high issues arise despite thorough testing, and reviews, this may indicate shortcomings not only in the testing phase but potentially in the architecture and threat modeling stages as well. This external review serves to validate the security measures and development processes undertaken up to this point.

Taking a formal approach, the eighth layer involves formal verification through the use of specialized tools like the Certora Prover or the K Framework. In this phase, develop a formal specification of the system and leverage SMT solvers on the backend to rigorously examine whether there exists a state in which the program could violate the specified formulas. By applying mathematical logic in this way, one validates that the system conforms to its desired specifications, ensuring the code meets rigorously defined criteria.

Next, the ninth layer introduces auditing contests, a strategy where the codebase is made publicly accessible for a time-boxed audit. Here, security experts from various levels of expertise can submit findings, which are then evaluated by judges who surface only the valid issues in a report format for the sponsor's consideration. These competitions not only engage a broader community of security experts but also offer a high leverage on security spending by pooling collective intelligence to compete for a fixed prize. This enhances the security measures of previous layers by adding a diversified layer of scrutiny.

The tenth layer incorporates bug bounties, serving as an ongoing audit contest without a time constraint and involving live software with high stakes. Platforms like Immunefi facilitate responsible disclosure, and some protocols offer incentives as high as 10 % of the at-risk funds for revealing vulnerabilities. This layer provides a continuous incentive for review of live systems, catching vulnerabilities that might have been overlooked in earlier stages.

The eleventh and more exploratory layer offers a unique approach, allowing ethical hackers and security researchers to exploit smart contracts under real-world conditions. In a single operation, researchers can siphon funds and then deposit them into a secured address, and then later claim a share of the Total Value Locked (TVL) or the maximum fee specified by the project. To legitimize this approach, comprehensive legal contracts are necessary. These contracts serve to indemnify the hackers, absolving them of liability across jurisdictions, provided they return the stolen funds to a designated address within an agreed-upon timeframe. This is an area still under active research, and is worth watching if only to see how these white hat hackers are treated in their local jurisdictions when income is reported from this source.

Developing smart contracts is a trade-off space between speed and security. This essay presents a multi-layered approach for creating correct and secure software. It emphasizes that security is not an afterthought to be outsourced. Rather, it is an integral part of the SDLC that developers must manage actively. Given the unique challenges and high stakes in smart contract development, these security layers serve as a guide for informed decision-making. It's important to note that the sequence and repetition of these layers can vary based on the project's specific needs. A single layer can also be implemented multiple times at different stages of the SDLC. Selecting and adapting these layers will help mitigate risks effectively. As the field evolves, we anticipate the introduction of new layers to further enhance the security of smart contract systems.