Auditing Compound Protocol

[JacobPPhillips]

@allthecolors – as always, your commentary is extremely thoughtful. I’m sure @sukernik will explain, but OZ and TOB were contacted around the same time — OZ just happened to move a lot quicker on this. And the process run here seemed to give ample time for other auditors to throw in a competing proposal. OZ’s request for the other auditors & the COMP community to move quickly on this seems reasonable given the timeline they’ve worked with so far (though I think the Dec 6th deadline is too tight, given the holidays).

This idea had been discussed by a number of folks in the community long before Larry finally took the initiative to have the conversations and make this happen. There’s an intricate web of potential conflicts of interests (some obvious, some less obvious) for just about any leader in the crypto community, so it’s not surprising to see this here. I think the best we can do is just be transparent about potential COI’s. In some ways, Larry’s close relationship with OZ may have been valuable in helping mobilize OZ to make the prop it did (and perhaps he should even be compensated for this!). COIs can be dangerous if they lead to proposals that are misaligned with the long-term health of COMP, but in this situation, COMP would be very lucky to have OZ or TOB on our side, both are excellent options.

The [scope of the proposal] and [cost] concerns are also very valid. I think the service providers should be free to propose whatever they want… if their proposals are off the mark from what the COMP community would like, we should negotiate the services/terms. This highlights a key problem — it’s really hard to negotiate the right deals with so many voices on an open forum, especially when considering multiple proposals. (h/t @sukernik who planned to bring up this exact issue on the community call).

Perhaps forming a pod (committee) of people with expertise in structuring and negotiating these types of deals and giving them the autonomy to negotiate off-forum with OZ, TOB, and others to get a deal done would be a better way to accomplish the objective here.

(p.s. Happy Thanksgiving to anyone celebrating!)

[dvf]

I’m the security guy for OUSD. Currently we are the #8 holder of cUSDT and the #10 holder of cUSDC, and growing rapidly - yesterday 29 million stables were deposited into OUSD. I don’t want something happening to that money. My goal here is that Compound not lose a pile of money suddenly.

---

The Compound codebase needs a first tier audit. It is madness to continue the current ad-hoc deployment of new code changes. This is one of the biggest protocols in Defi. It is vital that each code change pass a serious code review.

As far as I can tell, everyone here is in agreement on the need for an audit, and the need for code change reviews.

Beyond that, Compound’s single collateral pool design means that a security failure of any listed coin could empty all the liquidity of all Compound lending pools. Both coins that destroyed CREAM, and the coin that could have destroyed AAVE, would have wrecked Compound had they been listed here. Compound governance’s strong bias for not listing coins has probably saved the protocol.

A listed coin is a much a part of the security of Compound as the Compound codebase itself. Listing a new coin requires a high bar of security and economic analysis. It’s an integral part of the security of Compound.

---

Individuals making code contributions can’t be expected to negotiate, schedule, and pre-fund an audit of their own code that then may or may not even be approved for payment. Even for good teams with existing auditor relationships, it is difficult to quickly get an auditor lined up to review code changes. If we want Compound to have any development velocity, there needs to be a friction free way of getting excellent code reviews on all changes that is not up to the individual contributor. Both of these proposals provide that, and this is good.

This capability is not cheap no matter how it is done, and that is okay.

---

One long term alternative is for Compound to build out their own internal security team to handle the majority of the security work.

The proposals from OZ, Trail of Bits, and Certora all assume that Compound has an internal development/security team. This isn’t the case as far as I can tell from forum lurking. Compound development currently seems to be individuals making ad-hoc proposals, and no one really checking them, outside sometimes getting a random external audit.

Four million dollars, even in this market, does buy a substantial amount of developer/security time. The ideal mix of internal and and external from both a cost and security perspective probably isn’t being 100% external.

But while the long term solution may be different, it is urgent to put Compound on a better security footing. I’d run a version of one of these proposals for a year and use that time to plan and build out the longer term solution.

---

It would be good for external security proposals to reflect the current Compound security staffing, rather than that of a typical project.

---

I am strongly against any form of “performance fee”. That seems like it would reward gaming the system, more than just doing the job and getting paid / earning the reputation. I think performance fees have has strong downsides. It’s a distraction for everyone involved. The auditors don’t want to be the ones that signed off on the code change that got one of the largest defi projects hacked. That’s enough.

Unless someone has an epic, genius idea, let’s just nix anything to do with performance bonuses.

If an auditor is willing to do so, denominating the payment amount in COMP would be an alignment that would be good, and easy to do. If not, I understand.

---

I don’t have any friendships with anyone at either OZ or Trail of Bits. I’ve had audits by both. Both were professional, competent, and reasonably through.

I preferred the OZ experience, but I know other people who have preferred working with Trail of Bits. I’m assuming your experience and strength of audit is based on which individuals are assigned to your audit team.

In either case, either company is first tier and capable of providing a good team.

The two proposals are currently at the same price. Of the two current proposals, the OZ proposal offers more, and I lean that way.

Both proposals would make a marked security improvement over the current situation. In my opinion, given the funds in Compound, both bring more value than they cost.

Both the OZ proposal and the Trail of Bits are obviously overpriced, and also contain low value filler to pad the dollar value of the proposal.

Because these proposals are currently overpriced there’s a lot of weird dynamics going on in this thread from both parties, as they try to get the cash cow, while not getting into a bidding war and driving the price down.

How to properly negotiate in this open forum is beyond the scope of what I’m willing to think about on my Thanksgiving day off.

Compound needs an audit, needs a system for auditing future changes, and needs it now. We should have a deadline, and go for it.

Happy Thanksgiving!

[sukernik]

Plenty of things to respond to here; it may be a good idea to do so before the post-Thanksgiving dinner food coma.

First of all, I’d like to thank @allthecolors for his pragmatic reply. It’s always a bit difficult to communicate all of the information one has on the forums: private conversations happen of course, and relaying every bit of detail that happens in private can be cumbersome. As a result, many forum posts are (rightfully) questioned by the outside observer.

Now to address @allthecolors’ comment directly. A sizable chunk of my net worth is in COMP, so when the Proposal 62 exploit happened, I was quite frustrated that a simple and avoidable process failure wasn’t being rectified. In my mind, the fastest fix was to get a top-tier auditor to review every single governance proposal for Compound. I ran the idea by a few folks in the community, and after they responded favorably to the idea, I decided to reach out to the audit firms. When you need help with something, you naturally turn to the folks you’re closest with first. I had pre-existing relationships with the OZ folks, so I reached out to them first. A few days after speaking with OZ, I reached out to Trail of Bits, ConsenSys Diligence, and Sigma Prime (it took some time for me to get connected with them). In each case, I told each of them the same information: Compound protocol needs an auditor, you should bid for the work, and I personally have a personal and economic relationship with OZ. Conflicts of interest are unavoidable; as long we’re transparent about them, we should be able to achieve fair and favorable outcomes.

I’d also like to share some thoughts on the process by which Compound protocol chooses an auditor. Clearly, this whole shebang is a first for Compound: we’ve never had to deal with multiple vendors bidding for the same work, which means we’ve never had to establish a process for choosing the winner. Is this an annoyance to vendors looking to work with Compound? Absolutely. Is Compound protocol the one spending millions of dollars for the work? Yeah. In my book, the guy who is opening his wallet is the one who dictates the terms of engagement. To that end, it’s my opinion that Compound should be laying out the process by which it buys things rather than the other way around.

Practically speaking, Compound doesn’t make any decisions on its own (wouldn’t that be nice); it’s the tokenholders/community that make the decisions. How then should tokenholders/community members proceed on making the decision of which vendor wins the audit work? I’d like to propose several options:

• Tokenholders delegate purchasing authority to a group of individuals. The idea here is it’s impractical to have a tokenholders/community members evaluate vendors. Most large tokenholders aren’t experienced in evaluating audit vendors; instead, they could delegate authority to a group of experienced folks to make the purchasing decision on their behalf. This group of folks could evaluate the vendors, publish an analysis, and make a recommendation to tokenholders. If the tokenholders agree, they would vote for the “recommended” vendor.

• Tokenholders vote on the vendor. A faster but slightly more chaotic process would be for tokenholders/community members to informally assign a deadline by which interested audit firms must submit proposals. After that date, no new proposals will be accepted. The “accepted” proposals can then be put up for an on-chain vote. Let the best vendor win!

As far as timing goes (if we go with the second option), I think it’s fair to honor OZ’s request of having the vote start on December 6. As part of that, we can set an informal rule saying that all proposals must be submitted on the forums a week before that — November 29. Would that mean that vendors posting their proposals after that are automatically disqualified from the vote? No. But it would mean that they would be tardy to the informal deadline set by the community, which reflects poorly on them. And if they can’t get their act together by then, then do we really want them as our vendor of choice? Probably not.

Anyway, that’s how I’d go about it. Now for some turkey…

[dguido]

@OZSecure

We would request that a vote start NLT 1200 PST, Dec 6, 2021. Secondly, whatever vote date is selected, we would kindly ask that the proposals be locked one week before the vote, so we can revise our proposal one last time, given that our proposal has been open for scrutiny and copying since we proposed it on Nov 4, 2021.

For what it’s worth, I agree with the dates suggested by Open Zeppelin here. We’ll post a full proposal within the next day and think it’s appropriate to have a vote shortly after.

[OZSecure]

@jacobpphillips, @sukernik, @allthecolors, @dvf, @dguido thank you, great recent comments and feedback.

It seems like now is a good opportunity to “clear the air” and address several items as well as specificities related to this Proposal and discussions.

First and foremost, we are new to working and partnering with DAOs and made some assumptions about the speed at which we should approach this process. We were also surprised by the lack of competing bids during the 2-3 week period that we presented our proposal. We assumed, incorrectly, that given the lack of bids we should go ahead and press forward with a vote. We did consult with various folks privately and they also felt it was strange that no other firms jumped into the fray. As such we went forward, even though we understand that people like @Getty were publicly pushing for competition. We understand this could have looked like we were pushing through a vote without proper competition or time for the community to reflect. However if you put yourself in OpenZeppelin's position, it made sense to continue moving forward with the process. Even in retrospect we still are not sure how we would have known when the right time to pull the trigger would have been, but understand how this could have been perceived as high pressure. @allthecolors and others hope that give some context

Secondly in regards to the streaming payment method we chose, it had been used before, and given the parameters available made more sense to use than asking for a lump sum payment for a year or half year etc. We felt that it would have been wrong for us to ask for a bunch of money upfront as opposed to having it stream out and let governance end the stream when appropriate. Lesson learned here as well…@arr00 and others hoping this gives our perspective. Please let us know what you would have suggested here...

Thirdly, in regards to the performance fee and its binary nature. We listened to the initial feedback and given the complexities of changing it, specifically suggested that the community continue discussing the fee and delay it until the end of March next year. We did not include this fee in our formal proposal for the vote Also, at the last community call, we discussed that we were postponing the decision on this fee and also discussed that it would make sense for us to roll in concepts that blockchain @ berkeley were proposing vis a vis bug bounties.

We now understand that even though competition was demanded, even necessary for some, Compound Governance doesn’t have a tried and true mechanism for evaluating competing vendors and that we are treading on new ground here. Therefore we are happy to be patient as the Community deliberates on what the best way forward is here.

In the meantime, we will be revising our proposal to reflect community feedback and our new thinking on the performance fees, as well as various Q&A that has come from this process.

Please keep the feedback coming and we look forward to all your thoughts and comments!

[HRitzdorf]

Security Solutions for DAOs

Overview

We live in a new world where more and more DAOs are in control of a project. Ensuring that safe choices are made is a major challenge and DAOs have been exploring different approaches. According to current proposals, Compound would effectively outsource their security to a vendor who would take care of various security-related needs.

We believe that this approach is not the route to take. The chosen audit company would become a critical dependency for the protocol. The suggested solutions with their intransparent, quarterly, lump-sum payment for a mix of services do not focus on the most effective tools. Hence, we would like to suggest a process that can leverage the strength of the security community.

The main security challenge for a protocol like Compound is the growing complexity that needs to be taken into account with every change:

• Due to upgradability, Compound needs to ensure compatibility with previous versions. Failure to do so can have significant consequences.
• As cTokens are used in various other DeFi protocols, these integrations must not be affected by the proposed changes.
• While certain governance votes are being voted on, others are in development or under review. This creates complicated dependencies.
• As the Compound protocol grows and starts to consist of more contracts it must be ensured that all contracts still interact correctly. A current compound liquidation involves roughly ten contracts and roughly fifty calls between these contracts.

Overall, we conclude that such systems exhibit quadratic complexity growth. To tackle this, we aim for scalability. Furthermore, we believe that the security can best be ensured by a broad base of security providers. This prevents any form of vendor lock-in and conflicts of interest, while allowing to combine the strengths of different providers.

Suggested Process in a Nutshell

Our suggested process is as follows and acts on different stages of the DAO:

In detail our process has the following steps:

• Early feedback during RFP stage with code reviewers adding security consideration paragraphs

  • This often reduces review cycles and hence development costs, review costs and time-to-market as code reviewers can foresee likely issues
  • Example: Security considerations for RFP 8: Balancer V2 liquidity as collateral | $25k

    The RFP aims to allow Balancer V2 liquidity as collateral. Recent attacks against cream.finance have shown that repeated borrowing can create an extremely leveraged position. When reviewing the security, special care must be taken to ensure that the value of the liquidity tokens cannot be artificially inflated. This would cause the protocol to lose funds as the collateralization ratio is insufficient to compensate for the price change. An attacker could extract these protocol losses and hence retroactively finance the artificial inflation of the liquidity token values.

• Perform thorough code review of newly developed code

  • Thorough expert code review of to-be-deployed code is still the most effective tool available today.
  • Code reviewers inform the developers about all shortcomings related to security, efficiency or design.
  • During code review, code reviewers jointly contribute to an audit suite by encoding all successful attacks as well as attacks that are currently not possible but might work in the future.

• Develop an audit suite

  • Composed primarily of extensive tests executable against forked mainnet at any point in time, allowing to continuously evaluate the security of the protocol and new proposal
  • The audit suite is open source under the same license as Compound Smart Contracts (BSD-3). It must be executable publicly without restriction by having suitable licenses in place for potentially proprietary components.

• Leverage tool-based analysis where efficiently possible

  • Program Verification Tools currently cannot show correctness for a protocol as complex as Compound.
  • Use tools selectively with more complicated techniques to validate correct behavior of subsystems, e.g., formal verification, static analysis or fuzzing.

• Perform the Proposal Verification

  • Once the development has completed, the proposal is prepared on-chain.
  • The extended test against forked mainnet of the audit suite allow anyone to verify the compatibility of the proposal. In particular they can check that the proposal does not break any integrations and is compatible with previous versions.
  • As the proposal includes the concrete parameters chosen, these parameters can be verified as part of this step.

In contrasts to other suggested processes, we focus on prevention. While monitoring is certainly nice to have, Compound already has some monitoring and attacks often take place in one (large) transaction. Hence monitoring and alerting have limited effectiveness, especially considering the present timelocks.

Finding Suitable Code Reviewers

The biggest hurdle for code reviewers getting into a new complex system is the required initial investment to understand it. Fluctuation and internal availability of employees also require any review companies to constantly invest into training, which can reduce either quality in times of high demand.

To tackle these problems, we propose the following general solution:

• Compound Governance votes in several code reviewers each year to onboard into auditing through a paid training phase.

• Auditors apply to become wardens of Compound by providing:

  • An example of how a code review for the Comptroller and the cToken would be performed, detailing methods and time required.
  • The cost and amount of hours reserved for Compound per quarter, which are if accepted guaranteed to be paid by Compound.
  • The expected compensation for the training phase.

• On-boarding code reviewers can also greatly benefit from the existing audit suite which contains many relevant attack scenarios.

  • Any time reserved for Compound not spent on reviewing code or RFPs is used to improve this audit suite.

• Auditors who are selected by Governance and complete the training take shared ownership of the audit suite and collectively improve the coverage.

  • Delivering good work in improving the Compound Audit Suite will likely be used to evaluate the quality of the auditor which aligns interests long-term by increasing the likelihood of being voted in again next year.

Benefits of the Audit Suite

• Auditors think of many cases, most of which are not exploitable currently. Putting them in the suite helps to test for this case going forward. It aggregates knowledge from multiple code reviews and companies.
• Expert know-how of the system and the possible attacks against it are captured for future auditors.
• With testing against forked mainnet it is possible to simulate the effects of all the currently open proposals independently as well as combined.
• The Audit Suite enforces the definition of explicit value ranges, making code reviews more effective and simplifying certain governance proposals which just update parameters within those ranges.
• It provides extremely good feedback for developers reducing review cycles and thereby financial and temporal overhead.
• Auditors will gain confidence in having audited the system holistically even when reviewing seemingly small changes.
• Tasks that needs to be performed repeatedly, such as the compatibility check with past versions, can be automated once and efficiently repeated.
• The suite ensures the correctness of on-chain proposals including correctly chosen parameters which are generally not part of the code review.
• Given its design the audit suite provides a decentralized and permissionless mechanism for community members to inspect the state of the protocol and to evaluate ongoing proposals.

Considering all these benefits, we believe that the audit suite can provide the necessary scalability by capturing a lot of the growing complexity. It requires an initial investment to build it up, but we consider it worthwhile.

Summary

Our suggestion incentivizes broad participation from small to large security companies. Thereby, it broadens the base of experts in Compound and leverages proven techniques. Using the audit suite, we can aggregate the knowledge and insights from multiple audits (instead of only relying on the last audit) and thereby tackle the growing complexity. We believe that this provides a viable long-term solution.

While even a single audit company chosen to shepherd the protocol could write such an audit suite, only by having multiple companies who rely on each other to have covered critical parts of the system enforces that a public, high-quality audit suite gets created. It also allows every security provider to play to their strengths when contributing. Lastly, it allows a decentralized verification of proposals by any community member.

[Rk2357]

@sukernik @dguido @OZSecure
Hi everyone, I’m Rishabh from Blockchain at Berkeley.

Thank you to both the OpenZeppelin and TrailOfBits teams for starting a discussion of the need for more Compound security. While the entire community seems to agree on the need for security, there is the question of which company to procure those services from. I also agree with Larry that we should allow other security companies to forward their bid for the contract as well.

Overall, I believe, of the two options that Larry presents, the second is overall more beneficial to the protocol.

1. Direct voting by tokenholders is most in-line with the spirit of decentralized governance. Voting should not be filtered through the lens of some pre-selected committee; it adds additional layers of unnecessary bureaucracy.
2. In OZ’s original proposal, they highlighted the need for quick iteration on this issue. Other commenters have tended to agree. Direct voting lets one company start work on this issue as fast as possible, rather than having to wait for some committee to deliberate, then possibly multiple governance proposal cycles (if the tokenholders reject the committee’s recommendation).

However, the immediate problem with this idea is that Compound proposals have traditionally only supported only Yes, No, or Abstain votes. However, in the Governor Bravo contract, there is an option for voters to attach text to their on-chain votes. Thus, if we assign each candidate in our vote a two-letter code, we can measure the support for each option through the comments.

Supporting multiple options in proposals is also a necessary step for Compound governance, so that tokenholders can make more complex decisions in the future (i.e. elected positions, setting variable parameters, etc.).

[RexShinka]

Hi Guys,

This is Rex from DeFiSafety. As I indicated in an article I wrote about the Compound Proposal 62, I believe this latest proposal is trying to solve the wrong problem. What Compound (and other protocols) need is not more 3rd party review but rather more staff. The Proposal 62 needed a PM and an internal software and process quality staff more than a third party auditor.

Protocols need more on the internal team watching and reviewing tasks before they present the results to an external auditor or a DAO. Based on my analysis of 62, the dev was mostly on his own. He wrote the code, wrote the tests, ran the tests, put the code on the testnet and asked the community to review and test. I never saw any evidence of actual community review. Finally, an external auditor looked it over and the result went to the DAO. I did write to the dev to get his comments on the article, but I did not get a response.

There did not seem to be an internal quality person or a PM. There did not seem to be internal dev process that might have said dev can’t write his own tests.

I believe a large protocol like Compound should have at least one internal QA person, a PM and a written process.

Is there a reason I have missed why protocols do not build internal staff? Could the staff be a target of regulatory attacks? But if that was true, it would be true of the auditors too, especially if they take over tasks that would be internal in the corporate world.

Another weakness I see is a lack of internal process documents. An example of this is that latest bZx EOA failure. The dev should not have used his main computer to execute trades on a major protocol account (remember Hughs’s exploit?) and the account should have had a multi-sig. There should be a public process document where bZx says how things should be done. In the case of COMP 62, the process should not have allowed one dev to do the entire requirements/code/test process on his own.

Actually, that inspires me. DeFiSafety can write a process doc, get community content and allow individual DAO’s to vote the document into their process. Perhaps Compound community could contribute?

I am not against more auditor participation and have nothing but praise for all the auditors in the chain of messages. DeFiSafety is not bidding for any work. We do not do consulting to protocols. Our goal is revenue from DeFi users so when we rate protocols like yours we are not in any conflict of interest.

However, our team watches DeFi protocols every day and thinks about problems from a process quality perspective. Please take this as our comments from that perspective. We are always available for discussions.

[dguido]

Please find our anticipated proposal below. We iterated on several items and terms that were present in our first draft, so there are a few refinements versus what I posted earlier!

I also spoke on Discord at the Compound Community Dev Call earlier today. I carved out my part of the presentation so you can listen to it briefly since it may add color to the proposal.

(For convenience, I also extracted the audio from ChainSecurity’s portion of the call.)

Executive summary

Compound seeks to prevent insecure proposals from being merged via the decentralized governance process. Trail of Bits will provide comprehensive software assurance services to mitigate this risk across three specific activities: consulting services, security engineering, and process creation.

We believe it is essential to create an easy-to-follow process with highly robust tools that makes security transparent, with or without a code review from Trail of Bits. We consider the primary goal as building capacity in the Compound ecosystem to secure itself. To that end, we will provide engineering efforts to develop critical security infrastructure and processes.

Trail of Bits will be paid the equivalent of $1 million USD in COMP every quarter for one year to provide the baseline services. This payment is all-inclusive of all services defined in the proposal.

Goals and Non-Goals

Goals of this effort include:

• Prevent insecure code from being merged into Compound through the governance process, and ensure that any remaining risks of proposals under consideration are well known before a vote. Compound desires services that eliminate and illuminate these risks.
• Provide first-class security tools and analytical capabilities to Compound developers. Compound developers must have every opportunity to analyze their code and understand its security ramifications. Compound desires a variety of tools that enable thorough inspection of code by developers and users.
• Create repeatable processes that build capacity for security and avoid dependence on external audits or third-party services for security. Security efforts should result in higher quality code being developed by the Compound community over time through the adoption of consistent processes.

Non-goals of this effort include:

• Deploy, configure, or maintain on-chain monitoring software. This goal does not address the security of governance proposals. These systems are an active field of research, may apply to fewer attacks than expected, or may have undesirable performance costs. Instead, we will commit to evaluating their opportunities and limitations during the project.
• Further development of the bug bounty program. We support the formation of a bug bounty proposal to specifically address this issue. We will provide consultative advice to this proposal based on our years of experience seeing both ends of the bounty ecosystem.

Description of the Services

1. Consulting services

Trail of Bits will provide at least one full-time security engineer, with additional engineers supporting the project as needed, to perform the following consulting services:

• Maintain a presence on Discord and the Compound forums. We will actively engage in conversation, reach out to developers as needed, and identify any new proposals.
• Provide proposal authors with 1:1 counseling sessions. We will host a video call with the author to understand their goals and provide immediate feedback, including on the architectural design of their proposal and suggestions to reduce complexity.
• Review and report any identified security issues in the code for proposals. We will describe the issue, a scenario to abuse it, and a recommendation to address it. We will work with proposal authors to validate fixes that result from these reports.
• Define security properties for proposals. We will work with the authors to provide reasonable security invariants alongside the proposal and tests for them in Certora, Echidna, or other tools, as appropriate for the specific invariants under test.
• Document “Security Considerations” for every proposal. We will contribute a standardized section to reviewed proposals to inform developers and users of limitations, risks, or other considerations to form an opinion about their vote on it.
• Provide our analysis directly to the community. Before a vote on any governance issue, we will host a public community call, walk attendees through the documented Security Considerations, and run test suites, fully informing their decisions to vote.
• Host bi-weekly Office Hours with developers. We will cover testing and verification tools, demonstrate new security processes and tools for Compound, and solicit feedback for new areas of development and guidance needed by the community.
• Evaluate new security techniques for adoption by Compound. We will perform detailed evaluations of the applicability of these techniques directly on the Compound codebase, sharing our empirical results of efficacy and utility.
• Ad-hoc services sourced from across Trail of Bits, as needed. This includes expertise from our separate teams for cryptography, application security, cloud-native security, threat modeling, machine learning, and other research teams. Our firm employs more than 80 researchers servicing clients in tech, defense, and finance on high-risk security challenges.

2. Security engineering

Trail of Bits will engineer solutions in software to critical security risks faced by Compound. We will ensure that first-class security tools are available, easy to use, and customized for use on Compound, with knowledge of Compound-specific risks built in.

• Ensure that Slither, our static analysis framework, and Echidna, our rapid security property tester, always work on Compound code. These security tools are delicate machines that must ingest all code possible to write in Solidity and EVM. We will ensure that no breaking changes affect Compound for an extended period of time and these tools always “just work.”
• Customize Slither and Echidna to the Compound codebase. We will extend each tool to understand Compound’s architecture, expected security properties, and third-party protocols, vastly enhancing their depth of results. For example, we can build static analyses that understand and aggregate data from Certora properties or that understand specific Compound interfaces.
• Customize Slither to evaluate the security of upgrades. Upgradeability exposes low-level complexity with possibly disastrous results. Slither already evaluates for 17 such flaws, and we will enhance this analysis with Compound-specific conditions.
• Develop scaffolding for new proposals with pre-integrated security analyses from Slither, Echidna, Certora, or others, as appropriate. These templates will provide secure beginnings for parameter changes, new tokens, protocol features, and governance changes.
• Continuously define and evaluate security properties across the Compound codebase with analyses from Slither, Echidna, Certora, or other techniques, as appropriate. New proposals may expose under-specified areas of Compound that require greater formalization. We will work to fill in these gaps with new properties.

3. Repeatable processes

Trail of Bits will build the capacity of the Compound community to secure itself, minimizing dependency on external security audits for security and most efficiently using their time when engaged. We will define and socialize repeatable processes that encapsulate common tasks with security integrated within them. Adoption of these processes will set a floor on proposal quality, continuously secure proposals regardless of security auditor review, and improve the quality of proposals over time.

• Design a repeatable process for starting a new proposal. We will develop onboarding guidance for developers that describes the end-to-end process for securely creating new proposals. Touchpoints will include security training, using pre-created templates, example security properties, guidance on tools, self-assessment, engaging with Trail of Bits for code review.
• Design a repeatable process for proposal self-assessment. We will provide guidance to assess the security of proposals before sharing them publicly. This will facilitate more effective conversations about security with the community and Trail of Bits, knowing that an initial baseline has been met.
• Design a repeatable process for risk assessment by the community. We will share the risk factors that security experts focus on when reviewing proposals and document steps to run testing and verification tools, thus providing a map for those voting on proposals to become better informed and obtain empirical evidence.
• Design a repeatable process for evaluating third-party protocol integration risks. We will design a repeatable process to investigate the security considerations of new token implementations and other linkages to DeFi building blocks from Compound. We will document existing pitfalls and concerns in third-party code already used by Compound, and facilitate the discovery and documentation of others.
• Design a repeatable process for other protocols to securely integrate with Compound. In the reverse of the above, we will describe security considerations for DeFi users of Compound. In our mind, any compromise involving Compound, even if it does not originate from our own code, will compromise its reputation.
• Regularly update a “treasure map” for bug hunting in Compound. We will guide other security researchers to less specified, more risky areas of the code. We will regularly report statistics on these identified hotspots, such as % coverage for specifications.

Risks

Compound may have unknown, latent security vulnerabilities already present in the code. Our proposal focuses on new code added to Compound and, therefore, these issues may continue undiscovered. To mitigate this risk, we have included a) a task focused on specifying new security properties, as needed, and b) a task to build a treasure map to aid other bug hunters.

COMP holders may vote for proposals that contain documented security risk or that are otherwise highly risky, considering the reward to be greater in that circumstance. To mitigate this risk, we will a) add documented Security Considerations to every proposal and b) host a community call to walk through those considerations, demonstrate how to run included test suites, and evaluate the coverage of them.

Compound is highly complex and new proposals may use new features of Solidity or combine features of Solidity in ways that break the security tools upon which it depends. To mitigate this risk, Trail of Bits has specifically prioritized the development and testing of new features for Slither and Echidna against the Compound codebase.

Proposals may require swift approval without waiting for input from Trail of Bits, or input from Trail of Bits may be otherwise unavailable within the timeframe required due to unknown circumstances. To mitigate this risk, Trail of Bits has proposed a robust sequence of processes and tools for the community to enhance trust in themselves and better understand the risks of their actions.

Financial Terms

Trail of Bits will be paid the equivalent of $1 million USD in COMP every quarter for one year to provide the services. This payment is all-inclusive of all services defined in the proposal.

[jared]

I have to say that for the long-term, the model proposed by ChainSecurity seems to me to be the most comprehensive and beneficial to the protocol. I agree that cultivating an RFP process, and in general focusing on the codified process, regardless of auditor, is a direction I believe we should move in to put the protocol in the strongest position going forward. That said, I can see how signing an agreement with a single auditor to guarantee their time is something that could be more useful in the shorter term. I think it would be a great if we could find some sort of balance that moves us towards a robust process, without lock-in, but acknowledges some of the difficulties implementing the full RFP process immediately.

[OZSecure]

@jared as we said on the community call this week, OpenZeppelin would be open and happy to work with other auditors where appropriate or desired by the DAO. We agree that the community will best benefit from a codified robust process that does not involve vendor lock-in, and we have tried to make that clear in our proposal. We are also open to working closely with firms like Certora if and when Formal Verification is required by the DAO. Overall we view partnerships and codified processes as critical to a layered security approach to ensure the security of the DAO and complementary to our continuous audit proposal.

Please find our revised proposal summary below:

OpenZeppelin’s Updated Proposal Summary

The Compound DAO’s long-term security requires a comprehensive and continuous set of audit and security solutions to prevent loss of funds and protect its reputation resulting from risks to the Compound protocol, specifically those introduced by community-proposed upgrades

OpenZeppelin will provide dedicated continuous audit services for all Compound governance proposals and will work with the Compound community to develop comprehensive security requirements and to implement best practice security monitoring.

OpenZeppelin's services will be coordinated by a dedicated Security Advisor who along with the OpenZeppelin team, the Compound DAO and the community will work to:

1. Improve the overall process to ensure the security of community proposed upgrades to the Compound Protocol
2. Provide continuous audits and dedicated resources to respond rapidly to all community proposed upgrades and changes
3. Coordinate the creation of documented security checklists and requirements that can be shared with all proposal authors
4. Implement an open security monitoring and security dashboard solution that will allow the community to validate security
5. Integrate, support, and analyze other possible future important security program components such as formal verification, bug bounties, and white hat monitoring approved by the DAO.

The combined effort of the OpenZeppelin team, the Security Advisor, and the Compound community will thereby reduce potential security risks and further assure the DAOs trusted reputation.

OpenZeppelin has revised its original proposal to focus on community feedback and excludes performance fees. OpenZeppelin’s fee will be the equivalent of $1 million USD in COMP every quarter for one year, to provide these services. This fee covers all services defined in the proposal. Please see our full revised proposal here :

OZ Final Proposal

We believe that no other firm in the market can bring the same breadth and depth of offerings to the DAO. We provide best-in-class continuous auditing and security advisory services; established leadership in secure development and secure operations; and external relationships and partnerships at a cost to value no other firm can match.

Next Steps

Assuming a vote begins on Dec 6th and OpenZeppelin’s Governance Proposal is selected by Dec 13th, the Compound community can expect the following timeline:

1. Proposal Security Process (starting on Dec 13th) - the Security Advisor (SA) will immediately engage with the Compound community to create a well-defined process for auditing protocol changes prior to being proposed. This will include a regular and agreed upon set of KPIs and communication plan to update the community on a regular basis, including but not limited to community calls.
2. Comprehensive Compound Audit (starting in January): A team of dedicated OpenZeppelin security auditors will perform a comprehensive review of the currently deployed smart contracts. Many of whom have worked with Compound smart contracts in prior audits.
3. Continuous Audits (starting in February): With the Proposal Security Process defined and a Comprehensive Audit complete, OpenZeppelin will be fully prepared to provide audits on all protocol change proposals. Note: We can begin auditing proposal changes earlier but given the lack of protocol changes currently pending, we expect February to be the ideal time to start.
4. Security Requirements (starting in January): the SA and assigned members of the OpenZeppelin team will engage with Compound Labs and the Compound community to gather current security requirements and begin building comprehensive security requirements documentation and checklists.
5. Security Monitoring: (starting in January): the SA and assigned members of the OpenZeppelin team will begin bi-weekly community workshops to gather requirements and share progressive design and implementation plans on comprehensive security monitoring and building a security dashboard.

We would be honored to partner with the Compound DAO to not only deliver continuous auditing but to also work together to be leaders and innovators in how to securely and efficiently run an effective DAO security program!

[Emilie_ChainSecurity]

Dear Compound Community,

ChainSecurity thanks you for the feedback on our proposal.

We’ve received positive feedback regarding increased competitiveness, redundancy of audits, collaboration between auditors. However, the following questions were raised:

• How can we ensure continuity with multiple auditors? How can we avoid things falling through the cracks? How can we avoid the risks and costs of switching auditors?
• What is the cost of this proposal?
• What exactly is the audit suite and what role does it play in the governance process?
• What role would ChainSecurity play?

Please find our answers below.

1. The shared audit suite provides a common work basis and alleviates the risks of switching between auditors

Risks of discontinuity can be best alleviated with an extended test suite, called the “audit suite”. The audit suite is a collection of test cases designed by each auditor as they review or study Compound code.

When an auditor reviews the code manually, (s)he imagines numerous potential attacks and their impact on the system. If the system is vulnerable to a few of these attacks, only those end up in the audit report, while the countless other attacks imagined for the audit remain undocumented, only stored in the mind of the auditor.

With the audit suite, the auditor would create tests for each of these attacks, even if they are not successful with this specific code. By storing this thought process in the shared testing suite, we can build up on the thinking of that auditor in future updates of the system, even if (s)he is not present for the next audit.

Recording all test cases also helps to monitor the quality of an auditor’s methodology, and offers more transparency to the community.

Multiple auditors would be working on Compound, each of them contributing to the test suite. Each auditor would have an incentive to create the best tests possible, as it would be a great reputation builder if one of their tests identifies a vulnerability in the future. The common test suite would expand the Compound auditing knowledge as it is shared, instead of keeping it segregated. As time goes by, the audit suite will comprise more and more attack scenarios, and allow an increasingly performant review of the protocol.

We believe this extended test suite could recreate the continuity of knowledge you’d find in one auditing firm, or even exceed it: even in a single auditing firm, employees may get sick, go on holidays, or leave the company, and their unrecorded knowledge will go with them.

(See more details on the audit suite in our initial proposal.)

2. Multiple auditors are trained on Compound codebase and ready to jump in the audit process at any time

To clarify this point, let us describe the next steps of the voting process with our proposal:

→ STEP 1: Compound governance votes for our proposal

• This doesn’t select a specific auditor, it just approves the system.

→ STEP 2: Auditors apply in a standardized way, which allows easy comparison between service providers. As stated in our original proposal:

• “Auditors apply to become wardens of Compound by providing:
  • An example of how a code review for the Comptroller and the cToken would be performed, detailing methods and time required.
  • The cost and amount of hours reserved for Compound per quarter, which are if accepted guaranteed to be paid by Compound.
  • The expected compensation for the training phase.”

→ STEP 3: Compound votes for the auditors

• Compound governance can vote for several auditors and/or proposals

→ STEP 4: Auditors are onboarded into the system

• Selected auditors start the training phase (which is a full audit of the system). This training phase brings the auditor up to speed, secures the system even more and improves the shared testing suite: the auditor will add tests coming from a new perspective
• During the training phase, previously onboarded auditors can spend time helping new auditors (if time allows)

→ STEP 5: Auditors work in partnership

• Auditors secure the protocol and contribute to the test suite as per their proposal
• Every year, Compound governance votes to keep the auditors in the system or not

As an example, please find here the proposal we intend to submit.

---

PS: Thanks @dguido for recording the audio of our intervention expalining the proposal in the Compound dev call: chainsecurity.m4a - Google Drive

[sukernik]

It’s been over a month since I posted the original request for auditors. Since then, we’ve received several comprehensive proposals from top-tier smart contract auditors. Candidly, I thought there would be some interest to audit Compound protocol, but I didn’t expect we’d get as many proposals as we did. It’s spectacular to see so many vendors vying for Compound’s business. If you’re looking for what a B2DAO process looks like, this is it.

As mentioned in an earlier post from this thread, Compound protocol does not have a clean and efficient process for evaluating several vendors. As a result, tokenholders and vendors aren’t sure what happens next. This is problematic for both parties: Compound needs an auditor as soon as possible, and the audit firms have a business to run and need to assign resources to their customers ahead of time. In short, we need a process for picking the vendor of choice.

I spoke to several community members about this, and folks suggested a quick and easy process. In a nutshell, we’d be using the community multisig to run a simple process. The community multisig would first whitelist each vendor to make a proposal. After that, tokenholders would vote on their favorite proposal. The proposal with the most “For” votes wins. To prevent more than one proposal from winning, the community multisig would then cancel the losing proposals after the vote is completed.

I’m including a more detailed version of the process below.

Audit Selection Process

1. Reverie to create a form for vendors to submit their Ethereum address - 12/7
2. Reverie to confirm the address belongs to the vendor through a video call - 12/8
3. Reverie to share addresses with the community multisig - 12/8
4. Community multisig to whitelist the address for each vendor - 12/9
5. After being whitelisted, each vendor will submit their proposal for an on-chain vote - 12/13
6. Tokenholders vote for their favorite proposal after a two-day review period - 12/15-12/17
7. Winning proposal is queued for execution in the timelock; losing proposals are cancelled by the community multisig - 12/17

Since Reverie (the company I started with Derek Hsue) initiated the audit search process as part of this thread, we propose that we also complete the vendor selection process on behalf of Compound protocol. It took considerable time for us to reach out to the audit firms, explain Compound’s needs to them, encourage them to submit proposals, and to walk them through the process. We would appreciate a $75k COMP grant for the work we’ve done here.

[getty]

This structure makes a lot of sense and is a good use of the whitelisting proposers feature.

While I don’t think we need to worry about the firms making other proposals, setting a relatively short expiry on the ability to propose is a good idea. Maybe two weeks? It could be even shorter, like 10-days.

This whole process has been a great learning experience for DAO, and this setup will be good for future B2DAO situations.

[jamico]

Thanks @sukernik. This structure makes sense and provides a clear path to finalizing the process. A couple thoughts:

Timing. The proposal deadline Larry suggested above (12/13) is reasonable. The community has given ample time to receive new bids and needs to be respectful of the auditors who have committed time, effort and resources towards the process to date. Also, it makes sense to require all proposals to be submitted at the same time so they run simultaneously. To that end, it might make sense to specify an exact time on 12/13 for the whitelisted auditors to introduce their proposals. That way we can ensure the votes run as close to simultaneous as possible.

Voting Mechanics. Under this model, we should make clear that tokenholders should only vote for one proposal. In other words, tokenholders should vote “yes” on the proposal they prefer, and abstain from the others (since the multi-sig will cancel all proposals except the one with the most “yes” votes). This will ensure that we avoid any weird overlapping vote dynamics between the various proposals.

Otherwise, this approach looks great. Thanks again to @sukernik and the Reverie team in taking the initiative and creating a framework that works for all sides.

Jeff Amico
a16z

[JacobPPhillips]

@sukernik

Since Reverie (the company I started with Derek Hsue) initiated the audit search process as part of this thread, we propose that we also complete the vendor selection process on behalf of Compound protocol. It took considerable time for us to reach out to the audit firms, explain Compound’s needs to them, encourage them to submit proposals, and to walk them through the process. We would appreciate a $75k COMP grant for the work we’ve done here.

No complaints yet for the $75k fee? good to see. We’re making progress people.

[massnomis]

“Community Multisig”

[dguido]

No complaints yet for the $75k fee? good to see. We’re making progress people.

It’s really clear from our end that Larry has been a tremendous help in getting this done, and the multisig-based voting process is just the latest effort that he’s invested. He’s definitely an asset for all you folks!

[JacobPPhillips]

Hey folks, apologize for the tone of this post. I feel quite strongly that anyone who does work to add value to a protocol (especially one as big as this) should be able to capture meaningful value/ownership from that work. COMP has not done this well in the past, but I see things like this (meaningful fee for arranging an auditing firm to work with the community) as a great step in the right direction. This should be the standard going forward (and we should even consider going back to retroactively reprice past contributions).

[ratankaliani]

Under this model, we should make clear that tokenholders should only vote for one proposal. In other words, tokenholders should vote “yes” on the proposal they prefer, and abstain from the others (since the multi-sig will cancel all proposals except the one with the most “yes” votes). This will ensure that we avoid any weird overlapping vote dynamics between the various proposals.

---

In terms of voting mechanics it seems like this adds additional and unnecessary friction. If voters & the community multisig are required to do this every time we pick a vendor for a specific function on Compound, it sets a bad standard for future votes (though this model should work in the short-term for this vote). We should look into doing one of the following:

Direct Voting for Vendor Selection. Adding additional functionality to the GovernorBravoDelegate contract to enable multiple & customizable options for voting on governance proposals. In this model, a “Audit Vendor Selection” Proposal would be created with ToB & OZ & Abstain listed as potential options by the community multisig or an unbiased third party, with the ToB & OZ correlated to a specific action (i.e. sending COMP tokens to the auditor as the first payment for auditing). There should be an arbitrary limit on the number of options allowed on a governance proposals (i.e. 2-4 to start), and voters would only be allowed to pick one option.

Correlating Proposals. Similar to the role that the community multisig is playing in @jamico’s model, we would create a function that would enable proposals to be batched together, with only the proposal in the batch with the highest votes FOR (where the proposal is passing) being approved. In this model, a third party proposer would create the batched proposals, and launch them together. Should any proposal pass by the end of the voting period, all proposals but the one with the most FOR votes would be cancelled by the contract. Additionally, voters would only be able to vote for one proposal within a batch.

In either of these two options, we solve the problems of 1) relying on the community multisig & 2) requiring voters to be cognizant of not voting on more than one proposal. Still trying to figure out the specific details for implementation, but would love to hear thoughts on implementing such functionality into the governance module.

s/o to @Rk2357 & @annamira for the inspiration here