Auditing Compound Protocol

Proposals
1

Summary

A few weeks ago, Compound suffered an exploit that led to the loss of ~$50 million worth of COMP. The exploit happened due to a process failure in the governance process: a governance proposal that tweaked COMP distribution rates (Proposal 62) had a flaw that allowed certain lenders on Compound to drain the treasury for hundreds of millions of dollars worth of COMP.

Although Proposal 62 was reviewed by several community members, none spotted the exploit in the proposal code in time. In short, the process failure came down to a leaky proposal review process.

I believe an additional complication made something like Proposal 62’s exploit more likely: historically, community members have had to shoulder the burden of arranging an audit for their proposal. This has resulted in extremely long integration times or improvements never getting implemented at all. That’s no bueno.

Process failures like the one we recently had should not be occurring for a protocol of Compound’s size. I’d like to propose a simple but effective solution to fix the process failure: hiring a top-tier smart contract audit firm to review every governance proposal prior to its on-chain execution. In my opinion, having a top-quality auditor review every proposal is a no-brainer that the protocol should be willing to pay for (since the magnitude of potential losses far exceed the cost of the service).

Background

Shortly after the exploit, I reached out to OpenZeppelin, which is one of the top smart contract security firms in the industry. I told them about the Compound exploit and encouraged them to consider submitting a proposal with a comprehensive set of security solutions to protect the protocol. After a few weeks of brainstorming, OpenZeppelin is now ready to post the proposal here on the forums to gather feedback from the wider community.

To be as transparent as possible, I’ll share just a few more details. I reached out to OpenZeppelin first because I had a personal relationship with the team: we worked on a few projects together in the past and I have great respect for the work they do and the team they put together. In addition, Reverie (the company I co-founded with Derek Hsue) is an advisor to OpenZeppelin and Forta (a new protocol launched by the OpenZeppelin team). In a nutshell, we stand to benefit if they do well.

I should also note that I spoke to Trail of Bits to encourage them to propose something for Compound too. Finally, I made an attempt to provide the same background to ConsenSys Diligence, although we haven’t had a chance to speak about Compound’s needs yet.

What’s Next

As a next step, I’d love to invite OpenZeppelin as well as the other audit firms to post their proposals in this thread. The community could then discuss the proposals and offer feedback on the scope of service, price, and other factors that are relevant to the purchasing decision.

11 Likes
2

Security Solutions For Compound Governance

Summary

A proposal to implement Security Solutions to prevent and mitigate loss of funds resulting from security risks introduced by community-proposed upgrades to the Compound protocol.

Background

For the past two years, OpenZeppelin has worked formally and informally with Compound to:

During that time, OpenZeppelin developed deep expertise and set industry standards on secure smart contract development, including best practices on protocol functionally, incentive structures, oracle integrations, and governance processes for protocols like Compound.

As events from the past weeks demonstrate, governance upgrades can introduce new security risk vectors that could result in reputational damage to the protocol and possible loss of user funds. A proven way to mitigate risk is performing ongoing security audits. OpenZeppelin prides itself on auditing each line of code manually; automated code analysis tools are no substitute for seasoned, dedicated professionals with deep knowledge of smart contract security.

However, as helpful as security audits and code libraries are in identifying or preventing vulnerabilities in code, effective security requires a continuous and holistic approach.

Proposal

In the following sections, we outline the case and scope for OpenZeppelin to develop a comprehensive set of best-in-class Security Solutions for Compound throughout all stages of the governance proposal lifecycle:

c1
c11093Ă—576 11.2 KB

Security Advisory

OpenZeppelin will provide a dedicated Protocol Security Officer (PSO) to act as Compound’s security advisor. The PSO will be the main point of contact for the Community and provide advice and guidance to help improve and maintain Compound’s governance process using industry best practices.

The PSO’s advice, guidance, and feedback will be focused on:

  • Governance process improvements
    • Incorporating security audits and input from security experts throughout the proposal lifecycle
    • Setting up procedures for threat detection, monitoring, and alerting
    • Process recommendations on use of timelock; and specific recommendations on how to extend Pause Guardian functionality into incident response procedures
  • Community support by answering smart contract development security related questions during the governance proposal creation process, including:
    • Development best practices based on known and potential attack vectors, and types of vulnerabilities to avoid
    • Recommendations on available market solutions (security tools, bug bounties, services) for improving overall technical risk

Security Training

In order to reduce the probability of introducing new security risks in new proposals, the PSO, together with OpenZeppelin’s Education team, will provide tailored training and resources to the Community on secure smart contract development best practices. OpenZeppelin will host bi-annual workshops led by senior smart contract security researchers including:

  • An additional set of four workshops to outline and educate the community on unique aspects of Compound’s security landscape
  • A customized, six-session training course to take place over the span of three months
  • Unique course materials and certifications for all attendees, which could become a potential requirement for community members to introduce new governance proposals

For examples of educational content developed by OpenZeppelin, please refer to our Secure Smart Contract Development Series on Strategies for Safer Governance systems, The Dangers of Price Oracles, Strategies for Secure Access Controls, and The Dangers of Token Integration.

Security Audits

OpenZeppelin will continuously audit and provide actionable security recommendations on new governance proposals. OpenZeppelin will assign a dedicated team of security auditors to review proposed changes on an ongoing basis, and will verify that the transaction to be executed matches the code submitted. The Compound protocol will be audited as these changes occur within the governance process, and the related deliverables will be communicated upon an agreed-upon schedule.

For new governance proposals that meet an agreed-upon threshold (e.g. minimum # of votes), OpenZeppelin will commit to provide within an agreed-upon time frame:

  • A detailed report with an actionable description of any issue found
  • An outline of potential exploitable vulnerabilities in the code and any unexpected behavior caused by errors in the architecture and/or logic, and additional recommendations to increase security, and a general analysis of the contract dynamics reflecting state-of-the-art security patterns
  • All governance proposals including parameter changes will be reviewed and audited
  • A classification of issues according to severity with proposed, concrete fixes
  • A review of any fixes to issues identified in the audit process

As the first step in implementing this Solution, OpenZeppelin will perform an initial, comprehensive audit of Compound’s deployed smart contracts over a fixed period of time. For examples of OpenZeppelin’s security audit reports, please see our blog.

Threat Detection Monitoring and Alerting

In order to enable faster issue prevention and mitigate the risk associated with newly passed proposals, OpenZeppelin will provide operational and security monitoring to detect issues and threats during and after the deployment of passed proposals. OpenZeppelin will implement or enhance existing monitoring and alerting systems to provide complete coverage. This will include (as appropriate):

  • Setup of comprehensive operational and security monitoring allowing near real-time threat detection of attacks or exploits related to Compound smart contracts and specific governance proposals
  • Configuration of alert notifications and dashboards for Compound’s designated parties and for OpenZeppelins PSO as appropriate
  • Validation that all governance proposals are deployed accurately
  • Supply chain monitoring (libraries, oracles or protocols used by Compound Protocol in association with new governance proposals)

Examples on how the threat detection monitoring and alerting Solution works:

  • Video demonstrating threat detection developed by OpenZeppelin on the Forta Protocol of the issues encountered after the deployment of Compound’s proposal 062
  • Example of an agent script monitoring the Compound protocol on Ethereum mainnet
  • Screenshot of alert notifications configuration

c3
c31624Ă—1068 64.9 KB

  • Screenshot of the Forta Explorer, showing OpenZeppelin-developed alerts from Ethereum mainnet associated with the start of the recent Indexed Finance exploit on October 15th, 2021, with losses of $16m

c2
c21600Ă—910 136 KB

Expectations

  • Duration: the nature of Security Solutions requires certain predictability to ensure the availability of teams and resources. Therefore we propose a 1-year engagement given the high and growing demand for smart contract security services.
  • Communications:
    • PSO Community responses and recommendations (2 business day/48 hour response time)
    • Audit Reports published as required by community-proposed upgrades to the Compound protocol
    • Initial education event series and Bi-Annual education workshops and training sessions thereafter
    • Monthly forum posts and participation of PSO on community calls with explanation relating to audit and threat reporting
    • Discord Developer & Twitter Spaces Community Calls as necessary
  • Terms of Service: applicable to OpenZeppelin’s provision of Security Solutions
  • Out of scope of this proposal: Setup and administration of bug bounty services, incident and emergency response services not explicitly laid out in this proposal, crisis communications and public relations support, compliance risk management, fees associated with external monitoring services (e.g. future Forta Protocol fees), as well as any other product or service not explicitly described here.

Fee Structure

OpenZeppelin charges a service fee for its best-in-class Security Solutions that seeks to be commensurate with the value that our solutions add to protocols. OpenZeppelin also wants to provide a strong signal of our alignment with the protocol. At the start of every quarter for one year OpenZeppelin will create a formal Compound Governance Proposal to update the service fee payment in accordance with the simple formula below:

Screen Shot 2021-11-02 at 10.47.29 AM
Screen Shot 2021-11-02 at 10.47.29 AM1328Ă—124 5.94 KB

  • Retainer Fee [COMP]
    • $1,000,000/quarter, payable in COMP at the Volume Weighted Average Price of COMP of the previous month (source: Messari)
    • Payable at the beginning of each quarter
  • Major Security Event Occurrence (𝞪) [1,0]
    • 𝞪=1 no Major Security Event occurred during the previous quarter
    • 𝞪=0 major security event has occurred during the previous quarter
    • “Major Security Event’’ means any material smart contract vulnerability that is exploited arising from code audited by OpenZeppelin which causes a major loss of funds to users generally (i.e. a loss is not particular to any specific individual user), or OpenZeppelin audited vulnerabilities that are exploited and cause significant loss of protocol function or service to the community. OpenZeppelin will not be responsible for events that occur when OpenZeppelin’s audit or governance process recommendations are ignored and not implemented into the Compound Protocol, which are beyond OpenZeppelin’s control.
  • Performance Fee [COMP]
    • $1,000,000/quarter, payable in COMP at the Volume Weighted Average Price of COMP of the previous month (source: Messari)
    • Payable at the end of each quarter

About OpenZeppelin

OpenZeppelin is the premier smart contract security technology and services company, trusted by the most used DeFi, NFT, and DAO projects. Founded in 2015 with the mission to protect the open economy, OpenZeppelin safeguards tens of billions of dollars in funds for leading crypto organizations including Coinbase, Ethereum Foundation, Compound, Aave, TheGraph, and many others.

11 Likes
3

I really like the idea of a top-tier audit firm entering into an ongoing agreement to audit the protocol. As @sukernik stated, historically, the burden of getting an audit has typically been on the contributor rather than the protocol. Establishing a structured process for making protocol changes and an additional review step by OZ will be valuable.

In my mind, one of the critical roles of governance is to ensure the protocol grows and, in doing so, balances security with innovation.

OZ’s proposal at a high level seems to be robust yet flexible to the protocol’s needs. Everyone is on the same page as to the goal of attaining an auditor. So lets skip to the elephant in the room: 1-year engagement, $4m base + another $4m assuming the protocol doesn’t experience a “Major security event.”

For $8m, Compound gets OZ, a team who is considered the best in the industry, who has audited the protocol extensively, a dedicated OZ advisor (they call them a Protocol Security Officer), tailored training resources, four workshops to educate the community on Compound’s security, a six-session training course (a course that could be used for onboarding full-time community members), all parameter changes and protocol changes thoroughly audited, and a threat detection system.

The services and resources provided here seem quite valuable, but when asked to put a number on it, $8m looks relatively high.

To date, there have been 67 governance proposals. Of which 39 occurred in the last year. If I am generous in counting, 15 of the 39 had protocol changes. That would put each of these at just over $500k each. Even if we were to say the protocol will make 25 code changes over the next year, that would average $320k. Lets also keep in mind many of these changes are pretty small in scope. I only expect 2-5 significant changes in a year.

30: Contributor Grants
32: Dai liquidations pt1
33: Remove automatic COMP claims and COMP speed refresh
34: DAI DSR
37: Sweep function
42: Migration to Gov Bravo
47: Oracle Improvement
49: Upgrade ctokens
50: Comptroller upgrade
59: Dai liquidations pt2
60: Address Whitelist for Governance
62-65: Split COMP rewards

I know it is a relatively new concept that DAOs hire services providers and that OZ is the best, but $8m seems very high. Not to mention a performance fee based on their ability to secure the protocol is laughable. If Compound experiences a “Major Security Event” and OZ (or any auditor) checked the code, they should not expect to get paid further.

@OZSecure Lets get some clarity on the pricing. A couple of weeks back, I did some back-of-the-envelope math and came up with a $3-$5m range (which I thought was already really generous).

13 Likes
4

The performance fee is a strong mismatch for aligning incentives here. Let’s be pessimistic, and consider that the odds of Compound suffering a giant loss in the next year is 5%. That means that if there was zero effort to improve the security of Compound, the odds of OZ getting the performance bonus would still be 95%.

On the other hand, an excellent audit, and perfect checking of all proposals through the year, might reduce the odds of a major loss to 2%. This only increases the odds of a performance bonus from 95% to 98%. That’s only a 3% change in expected payoff amount between no effort vs insane effort. It just doesn’t make sense here. Compound would be paying 4 million dollars to create $120,000 worth of incentives.

I agree that the price is surprisingly high and would love another look at it from them.

We at OUSD have had audits from Trail Of Bits, Solidified, Certora, and Open Zeppelin in the last year. Open Zeppelin would be my first pick for this audit. They did excellent, comprehensive, through work, and were a pleasure to work with.

I also do think that a re-audit of the current state of the code and systems would be a wonderful thing.

5 Likes
6

@Getty and @dvf thanks for your thoughtful and direct response to our proposal and your kind words about our past work and position in the crypto security space.

Before addressing your questions, it’s important to note that our proposal goes well beyond ad hoc auditing or tool-based approaches to protocol security, which is what you assume in your price analysis. We priced our fees in this proposal across two dimensions 1) comprehensive security products and services that extend beyond ad hoc or tool-based auditing services and 2) reputational risk for both parties -- Compound and OpenZeppelin. See below for a more detailed breakdown of the various components of our proposed security solution:

Security of the Compound protocol:

While state-of-the-art tools and automation are evolving, audits of smart contract systems remain costly because — to be truly effective — they must be manual in nature. In order to conduct timely continuous audits, as needed and related to governance code changes, OpenZeppelin will reserve the bandwidth of dedicated, knowledgeable blockchain security professionals and auditors. Our proposed fee structure accounts for the opportunity cost to us when we dedicate our top teams’ availability for the Compound DAO. We are also including a full audit of the Compound protocol in this fee and it’s relevant to note that in many cases our current audit pipeline is booked 12 months in advance.

That said, your breakdown of our fee structure only focuses on ad hoc audits. Our proposal offers a more comprehensive security solution, which includes --

  • Threat Monitoring and Threat Dashboarding and related customization services and integration: not only will we provide Defender and Forta integration (OpenZeppelin-developed software) but also customized reporting, dashboarding and Forta agent integration and development. This will require a significant investment in customized dashboard development, third party integration, and interaction with the community. Threat Monitoring and Threat Dashboarding is a critical cost component of our fees, which includes our software. The value of this solution to Compound will be:
    • Improved visibility and Intelligence of threats
    • Integration of relevant risk and open-source data
    • Reporting and analytics to help prioritize and act on the most relevant information
    • With the goal of faster threat detection and response thereby reducing overall protocol risk
  • Security Education: The Compound community will be provided with dedicated OpenZeppelin content and education events that will allow them to learn about security issues and best practices to apply in their Compound related activities. By utilizing this training and educational content, the Community will become more aware of coding practices that lead to vulnerabilities, being more proactive in avoiding these issues, thereby reducing overall risk in code submissions and governance decisions. (For examples of content and education and see our original proposal please)
  • PSO Advisory Service: We are including senior project management and advisory services to recommend improvements to the incident response and emergency response procedures and response times. Improving the ability to respond in a more timely manner is critically important to a secure and resilient protocol.

Reputational risk:

When we conduct a one-time regular audit we caveat our reports based on specific code, timing, and scope; through this proposal we become the Compound DAOs trusted security solution provider. As such, our reputation and brand is tied directly to the protocol's security.

  • The value of reduced risk: The goal of continuous security solutions is to reduce risk of downtime due to a significant misconfiguration event or security incident, which can not only disrupt protocol continuity and lead to loss of funds but also materially damage the long term reputation of the DAO. Attaching a value to prevention of reputational damage always presents a challenge, however we suggest that the value of the security solutions we can provide will protect the overall reputation as well as the value of the project.
  • Aligning our reputation to Compound’s reputation: The proposal aligns OpenZeppelin’s reputation to yours. If an incident occurs under our watch, it impacts our brand as well as Compound’s.
  • Risk fee aligns our interests: Keeping the above in mind, we proposed a fee structure such that Compound DAO pays the performance fee only after a successful quarter with no incidents. As you suggest, we get nothing if an incident occurs, which aligns our mutual interests.

In sum, our proposal is valued such that security is a continuous and dedicated process, that goes beyond ad-hoc audits, and takes into account our role as a Trusted Security Solution Provider, directly aligned to Compound’s needs and reputation.

2 Likes
7

Hi, we at Blockchain @ Berkeley support this proposal and appreciate how clear the duties of OZ will be once they are hired to audit each proposal. We share @getty’s concerns, and look at the pricing from another perspective:

Existing bug bounty programs in the DeFi space offer comparable rewards (i.e., 100-500k) per bug found in existing core contract code, not for auditing individual proposals before they are passed and in timelock.

The time of OZ professionals is very valuable, but would it make more sense to reserve such high per-code-change payments as a bonus for extreme bugs avoided rather than as a baseline, no protocol incidents rate?

We support adding, in addition to the OZ auditing proposal, a bug bounty program like this: www.comp.xyz/t/bug-bounty-program-for-compound-proposals/2590

Curious to hear more folks’ thoughts on pricing for auditing versus finding a bug, especially at different stages in protocol development (proposed vs. timelock vs. deployed code).

3 Likes
8

Since the launch of governance last summer, the Compound community has shepherded the protocol through a period of remarkable growth. Under the community’s watch the protocol has seen significant expansion across every relevant metric, from loans outstanding to TVL to net revenue. This period also witnessed a number of community-led upgrades to the core protocol itself, including the Governor Bravo migration and the Oracle Improvement, among others. These not only represent meaningful upgrades to the system, they also speak to the quality of the Compound community and its ability to shepherd the protocol into the future.

Like any community-led project though, growing pains are to be expected. To leverage the power of open-source development is to accept certain new risk tradeoffs, and Compound is no different. The protocol has benefited greatly from the open-source model, drawing on a great deal of third-party developer talent and resources to upgrade the protocol and extend its core capabilities. At the same time, the very properties that make this model so compelling (openness, permissionless, etc) can also introduce new risk vectors when not properly managed. We saw this recently with Proposal 62.

But the takeaway from 62 should not be a retreat from the open-source model that has otherwise served Compound so well. It should instead be an opportunity to give the community the resources and support it needs to contribute to the protocol in a safe and responsible manner. Doing so will allow Compound to continue to harness the benefits of community development while mitigating the accompanying risks.

This brings us to the OpenZeppelin proposal, which we think is positioned to do exactly that. OpenZeppelin not only has an industry-leading reputation among blockchain security firms, it also has extensive experience working on the Compound protocol itself, including 10+ audits as well as its work on the OpenZeppelin Governor contracts. Enlisting a firm of this caliber will give community developers the support they need to do their job effectively, while also ensuring the protocol remains safe.

While the proposal does come with a high nominal price tag, it ultimately seems reasonable given the quality of OpenZeppelin’s offering, the demand for similar services across the industry, and the need to safeguard Compound following Proposal 62. And furthermore the short contract term (and quarterly repricing options) allow the community to assess OpenZeppelin’s performance in relatively short-order and determine whether it’s worthwhile to renew.

We appreciate OpenZeppelin’s hard work in crafting a thoughtful and detailed proposal and also incorporating feedback from the community. In terms of next steps, we defer to the community’s view on timing/process, though at this point we are supportive of this proposal moving towards a formal vote. We welcome any additional thoughts or feedback that the community might have.

Jeff Amico
a16z

Disclosure: a16z is an investor in Forta, a blockchain security platform that was recently spun out of OpenZeppelin. a16z is not an investor in OpenZeppelin.

2 Likes
9

It’s great to see this proposal from OpenZeppelin, who has generally been a great partner to Compound Labs over the last few years. The scope of the proposal is quite large however, and includes services which Compound Labs has never used before, and therefore cannot attest to the value of. Audits have never been a silver bullet, and never will be, so personally I think it’s important to think about and balance all of the quality assurance resources we have available. This should probably include things like formal verification and investment into the tools developers actually use to build and test the protocol themselves.

While the price tag seems extremely high, I would like to clarify what exactly would be covered by the continuous security audits? Would this include all new contracts developed by the community and adopted by governance?

Logistically speaking, when can a proposal be submitted for audit? If we are truly just monitoring proposals that have been made on-chain, that seems not nearly enough time for an audit and a response to the audit. In that case, what would the actual process of scheduling an audit look like? What sort of lead times should the community expect?

I think it’s important we understand the answers to these questions in order to understand the offering. Thanks to everyone on this thread!

5 Likes
10

@jared thanks for joining the discussion!

I have tried to break out each of your questions with answers below:

What about Formal Verification (FV)?

As you may be aware we are trying to use formal verification on critical components of the OpenZeppelin Contracts Library. We see FV as helping to lower the overall risk footprint but FV also has its clear limitations. Therefore, we see FV as a component of an overall holistic security process that must include coding best practices, manual auditing, education, threat monitoring and bug bounties (our internal process encompasses all of these). Formal verification is only as good as the initial description of the desired properties of critical elements of the protocol, which is why manual auditing is still a critical step in the overall program. We have relationship with Certora and are fans of the progress their technology is making but would be happy to work with whichever formal verification vendor Compound chooses and implements. Security as you indicated is not about looking for a “silver bullet” but layering in services, processes, and technologies to lower overall risk exposure. We would be happy to include additional work and cooperation with Compound’s desired FV vendor as part of this existing proposal. Our PSO and auditors can aid in ensuring that the desired properties of the protocol are described properly from a security perspective, so the FV vendor can write the rules effectively for implementation.

Would this include all new contracts developed by the community and adopted by governance?

Yes, that's the expectation. Any substantial changes to the Compound protocol would be covered by an OpenZeppelin audit. Audit times would be set aside in advance for upcoming proposals and protocol changes. We would commit to auditing all community-supported proposals on a timely basis. That said, not all proposals are created equal; some will be high priority for the protocol and some will be low priority. We expect to work with the community on helping us separate the signal from the noise

Logistically speaking, when can a proposal be submitted for audit? If we are truly just monitoring proposals that have been made on-chain, that seems not nearly enough time for an audit and a response to the audit. In that case, what would the actual process of scheduling an audit look like? What sort of lead times should the community expect?

As part of the Advisory Solution, the first task of the OpenZeppelin Protocol Security Officer (PSO) is to propose to the community a clear approach and process as to how security audits and security related issues would be incorporated into the proposal lifecycle. From our perspective, security audits should be a prerequisite for submitting a governance proposal that makes changes to the protocol.

Once a potential community proposal has initial support and begins development, the submitter will work with the PSO and dedicated OpenZeppelin auditors to ensure the agreed defined process is followed, and everything is ready for code completion in the most efficient way possible.

The initial proposal is considered code complete with a frozen commit, it would then be submitted to the OpenZeppelin Audit team for audit.

Based on similar audits (e.g. Celo governance upgrades) our initial estimates suggest 2 weeks for auditing minor protocol proposals although exact time required depends on the lines of code and complexity. This includes 1 week for the audit and another week for reviewing fixes although this timing also depends on how quickly the proposal submitter responds with fixes. Once the final report is finalized and shared with the Community, the proposal will be able to move forward with the governance process starting with the two day review. Good communication between the OpenZeppelin team, the submitter, and the community will be critical and will be managed by the PSO. Please note that we have been conservative in these estimates, it's our assumption that we can improve the timeliness based on improved processes, developer education and good community communication.

Before auditing new proposals it's our expectation that we will do a full protocol audit, starting out with existing critical components of the Compound protocol which should include the governance contract, Comptroller, and other components typically involved in governance proposals. This initial audit will ensure that OpenZeppelin can take full ownership for the security for the entire protocol and give our auditors the necessary context to review new protocol changes.This would take approximately 7-9 weeks based on our knowledge of the existing Compound code base, we would begin this audit as soon as possible upon acceptance of this proposal.

I have attached an example to illustrate the process:


Proposal 62 would have consisted of the following steps:

  1. Late July - Forum post responding to RFP16 is made and development begins on the proposal changes. The PSO is made aware of the proposal in progress and works with the submitter to prepare for an audit in accordance with community guidelines.
  2. September 13th - Proposal code is completed as a Pull Request with a frozen commit one week prior to audit start
  3. September 20th - Audit begins the week of Sep 20th and ends that week on Sep 24th. An audit report is delivered to the submitter no later than Sep 27th.
  4. September 27th - Submitter addresses bugs found and submits fixes to the audit team to review. The audit team reviews the fixes and updates the report.
  5. October 4th - The report is finalized and shared with the Community. The proposal is able to move forward with the governance process starting with the two day review.

I hope this clarifies our willingness to support FV as a component of an overall holistic security process, and the way we are thinking about integrating our offering into the Compound Community governance process.

Steve Gant

OpenZeppelin

2 Likes
11

On behalf of the OpenZeppelin team, I want to thank everyone for bringing their thoughts and expertise to a constructive and open exchange about the Compound Security Solutions Proposal over the past two weeks.

Overall, Compound stakeholders and community members agree that a set of comprehensive security solutions are essential to the Compound DAO’s overall security and continued strong reputation. To this end, this week we will submit only the retainer fee portion of the OpenZeppelin Proposal for a vote. This will allow us to move forward on this important project and work with the Compound DAO on immediate governance and security matters.

Separately, in March 2022 we will submit the performance fee for a vote, which will allow time for further discussion while moving forward with this important matter. We believe the foundation of our initial work will demonstrate the merit of our proposal’s overall approach and look forward to the prospect of working with Compound!

7 Likes
12

This sounds workable then, @OZSecure.

I am in favor of doing it.

3 Likes
13

Why is does OpenZeppelin believe that using the built in contributor comp speed is a good idea? It is not the way to go about this proposal at all. The community should reject any proposal that creates indefinite payments to anyone.

5 Likes
14

Hey all,

Trail of Bits has a proposed solution in the works for the issue that Larry Sukernik described. We are aware that there’s an active vote on Open Zeppelin’s proposal. Please vote “no” on this governance proposal if you’d like to consider how Trail of Bits would provide these services before deciding on a vendor.

In advance of a finished proposal from us, I’d like to lay out why I think Trail of Bits could be the right choice for this job, a few key ways that our proposal will differ from Open Zeppelin, and why I think the performance fee may not provide the intended incentives.

Introducing Trail of Bits

Trail of Bits has worked extensively with Compound over nearly 1,000 hours of security review through four focused engagements covering their governance, core protocol, and internal security procedures. Many firms in DeFi, including MakerDAO, Balancer, Uniswap, and Rocketpool trust our expertise to help secure their code, and you can find many more in our Publications repository. We’ve succeeded in finding vulnerabilities in highly verified systems and providing the best solutions regardless of whether we invented them. We’re relentless about raising the baseline in the communities we work, and have developed and made freely available some of the most-used security tools, reference guides, and security research in the industry.

Trail of Bits is differentiated from other firms by our diversity of experience and expertise. We work on securing software wherever the risks are high in the technology, defense, and finance industries and employ a wide-ranging team of experts in programming language theory, cryptography, cloud-native software, and low-level exploitation to do so. Roughly one-third of our firm works on fundamental research in their field through long-term contracts with DARPA on automated program analysis, zero-knowledge proofs, and software verification. Access to these competency areas improves our ability to consult with all of our clients, and particularly on blockchain systems.

Trail of Bits proposal preview

At our core, we’ll be proposing many of the same services as Open Zeppelin. Therefore, I’ll keep these sections brief by focusing on the key differences in our approach.

Security review of governance proposals

Trail of Bits believes this is the most important service provided by the proposal. We plan to review all governance proposals, including parameter changes, new token integrations, and more extensive code changes. For each proposal, we will fully describe any identified security issues, including scenarios for abusing the issue and specific recommendations to address it.

  • Reviewing the security of a proposal after voting has already begun is too late. We will begin reviewing new proposals after it becomes clear they are seriously considered by users on the Compound Discourse.
  • Proposal authors will receive a one-on-one counseling session. We will host a video call with the author to understand their goals and provide immediate feedback. These video conversations will ensure information is effectively shared.
  • We will review and report any identified security issues. These issues will take the standard form of a description of the issue, a scenario for abusing it, and a recommendation for addressing it. We will work with proposal authors to validate any fixes that result from these reports.
  • We will contribute a “Security Considerations” section to every proposal. The absence of specific security issues does not ensure the safety of a proposal. We will contribute a standardized section to reviewed proposals that informs developers and users of limitations, risks, monitoring guidance, or other considerations.
  • We will help define security properties for the proposal. Human review is necessary but insufficient to provide for the security of DeFi systems. We will work with the authors to provide reasonable security invariants alongside the proposal.
  • We will provide our analysis directly to the community. Prior to a vote on any governance issue, we will host a public community call and walk attendees through any specific issues we discovered and the documented Security Considerations.

Community training and continuous improvement

Trail of Bits will develop scaffolding for proposal authors to write safer proposals and provide continuous training and guidance to improve the quality of proposals over time. In particular, our approach takes after the Security TAG process for improving the CNCF security baseline by engaging directly with developers.

  • We will iterate on public guidance for developing secure proposals. This will consist of a template repository with testing and verifications tools pre-configured for different proposal types, prompts to elicit a self-assessment of the proposal’s security by its authors, and a public walkthrough for navigating the security review process with Trail of Bits.
  • We will host bi-weekly workshop sessions or “office hours” with developers. These sessions will cover testing and verification tools, review previous DeFi incidents, and deeply investigate common areas of risk. As appropriate, these will be recorded and made available for community use.
  • We will publish a minimum-security checklist for new proposals. Specific to Compound, we will describe the minimum viable due diligence steps we believe are required to make a reasonable governance proposal. Like our Token Integration Checklist, this checklist will include specific, actionable steps that authors can complete on their own, before circulating a proposal.

Security intelligence

Trail of Bits takes an expansive, intelligence-driven view of security monitoring. On-chain monitoring is important, however, if you are finding issues at the time that transactions are being processed then you are typically too late. Furthermore, on-chain monitoring tools have substantial limitations regarding the types of issues they can detect. Like human-only code review, on-chain monitoring is necessary but not sufficient to ensure safety in DeFi.

We will evaluate and recommend an on-chain monitoring vendor. We will evaluate on-chain monitoring vendors (including Forta), recommend the selection of one, and assist in configuring it appropriately. Furthermore, the code invariants developed through security review will be regularly provided to the owner of this system. The quality of your on-chain monitoring is directly dependent on the quality of your system’s specifications.

We will provide security intelligence and facilitate community engagement. We will facilitate community development of new attacks against Compound. Rather than wait to report a fully-developed attack to Compound, Trail of Bits will be available to co-investigate unproven potential vulnerabilities with reporters and prepare tailored remediations and fixes to Compound alongside their reports. With our earlier involvement, we will shorten the window of exposure for new attacks reported against Compound.

Pricing structure

Fees to Trail of Bits will be split into three discrete categories:

  1. Quarterly retainer. Trail of Bits will be paid the equivalent of $1 million USD in COMP every quarter for one year to provide the baseline services.
  2. Trail of Bits will be eligible for bounty awards in cases where it co-investigates and co-reports issues to Compound, outside of the retainer fees.
  3. Performance fee. The motivations behind this fee are positive, however, the design as a binary all or nothing with factors outside the auditor’s control are not.

To be clear, we don’t have a perfectly designed performance fee incentive to propose at this time and desire additional time to work on this aspect of the proposal specifically. We’re eager to work with the community on a model that benefits everyone.

Finally, we are happy to cut the bounty co-investigation piece from this proposal and incorporate it into the Bug Bounty for Compound Proposals discussion. I included a discussion of it solely due to the mention of security monitoring by Open Zeppelin and because I see security intelligence as an inseparable part of security monitoring.

We’re very excited to hear your thoughts!

9 Likes
15

While we welcome competition, ToB’s last minute proposal -- that by their own admission is nearly identical to ours -- is undifferentiated and lacks a solution for threat monitoring and security intelligence. ToB had several weeks to respond but chose not to, making it clear that Compound’s security is not a priority for them. Delays in moving forward with security solutions, regardless of with whom, compromise the security of high value projects (DAOs or others), which must make a decision to act, not to delay.

In the 6-8 weeks since the Proposal 62 event, we have actively and publicly engaged the community and stakeholders to determine the most effective method of providing security given OpenZeppelin’s current and future capabilities. (Note, the proposal in question was in development within OpenZeppelin for our DAO strategy and we believed that our long standing Compound relationship and the incident merited a custom, early version of it given our long history of working with Compound).

A counterproposal from another security firm, particularly a proposal with significant alternatives, rather than a copy with small tactical differences, would have been valuable had it been brought forward in a timely fashion. Then the community could have determined the best option -- comparing all proposals with the appropriate time and attention it deserves. However, given security has urgency, we strongly advise the vote continue as planned and invite other vendors with valuable, original approaches to offer a counter proposal next year when the solution comes up for a vote again.

Given we are all working together to shape the future of B2DAO relationships and how engagements like this take off, we have no doubt that we and the community will have the opportunity to improve and iterate our solutions in the open and in competition with other solution providers in the space. OpenZeppelin is confident in our ability to deliver best-in-class security solutions, as our clients can attest (https://www.comp.xyz/t/auditing-compound-protocol/2543/4).

We ask that those who are genuinely interested in Compounds Protocol Security vote for our current proposal. (https://compound.finance/governance/proposals/70)

1 Like
16

I certainly understand the desire to swiftly plan a path forward but urgency on this issue is driven by new governance proposals, of which I only see a parameter change right now. For the record, the first we heard there was any activity on this topic was October 26th. It was a lot to wrap our heads around and deserved some time thinking about an approach rather than diving in with recommendations. I was surprised to see a vote so quickly given the magnitude of this effort!

At the end of the day, we’re all offering software assurance services so of course a lot of this is going to look the same. If I had to characterize the differences, the Trail of Bits proposal focuses most of its effort on a) finding and reducing risk in governance proposals, and b) increasing security by reducing the cost of control (e.g., with templates, checklists, assessment guides, specification development, and other systems that continuously reinforce themselves). I think we were much more specific on that front. On the other hand, the OZ proposal contemplates a broader set of activities, like security training and threat detection, that apply only indirectly to the core issue of merging insecure or unsafe proposals. E.g., I’m less sure how to take a Q&A channel staffed with OZ employees to the bank.

Reading between the lines, this seems like a similar take that Jared had earlier in the thread:

The scope of the proposal is quite large … and includes services which Compound Labs has never used before, and therefore cannot attest to the value of. Audits have never been a silver bullet, and never will be, so personally I think it’s important to think about and balance all of the quality assurance resources we have available. This should probably include things like formal verification and investment into the tools developers actually use to build and test the protocol themselves.

In my opinion, on-chain monitoring in particular, while possibly valuable, probably should not get bundled as a must-purchase alongside the auditing service for governance proposals. These types of systems are an active field of research (see also, flashbots), may apply to less attacks than expected, or may have undesirable performance impacts or other costs. I’d want to study them further to understand their value and limitations.

4 Likes
17

If you want Compound to be protected by OpenZeppelin as of Dec 1st, then vote Yes.

If you believe the question of Compound’s security can wait for further deliberation at this time and be addressed at an unknown future date, then vote No.

The facts are that we had a timely and immediate response to the needs of Compound and the Compound community. We did not show up opportunistically, or late, focused solely on missing out on a business opportunity. Our priority at OpenZeppelin is and has been to protect the open economy, including DAOs and communities like Compound.

We are past the stage of deciding whether Compound needs a security partner. There is no formal RFP process for DAO security solutions today, based on feedback and responses, it's clear that we not only conducted a fair and open process, but have set the standard for what a Security Solution should look like.

Delaying a decision to an unknown future date and vendor could come at the cost of Compound’s long-term security preparedness.

Thank you for your consideration,

2 Likes
18

We really appreciate the hard work of both OpenZeppelin and Trail of Bits in bringing forward their respective proposals. To receive competing bids of this quality speaks to the importance of the task at hand. It’s now incumbent on the community to do its part and evaluate these proposals fairly and in a timely manner.

As we wrote earlier in this thread, we supported OZ’s original proposal, and delegated to their CAP to help kick off a live vote. However, we chose to abstain from voting on Proposal 70 once we saw Trail of Bits submit a competing bid. We did so to ensure that ToB had a fair chance to pitch its proposal and be heard by the community. While this will extend the timeline, we think a competitive process is in the best interest of the protocol, and will ultimately serve Compound well in the end.

To that end, we would support extending the process for another 1-2 weeks. This would allow the community to hear more about ToB’s proposal, to evaluate the two proposals side-by-side, and to answer any remaining questions before making a decision. Following this period, we would suggest that the two proposals be introduced as simultaneous governance votes (or something similar), with the token holders ultimately deciding which one they prefer.

We think a simple approach along these lines would give ToB an opportunity to pitch its proposal, while also setting a firm deadline to ensure the process does not drag out indefinitely. This latter point is important both for the security of the Compound protocol and to ensure we remain respectful of both vendors’ time and effort.

We welcome any thoughts or ideas that the community might have, and look forward to identifying a process that works for all parties.

Jeff Amico
a16z

9 Likes
20

Thanks Jeff!

We strongly believe that OpenZeppelin has the best vision, team, and Security Solutions for Compound. We also know that we are trailblazing a new process here of how the DAO will ensure security.

On November 3rd, OpenZeppelin was invited to submit a proposal and has led a well paced, open, and transparent process. But we accept that this process will now be extended.

Given the significant time, effort and leadership OpenZeppelin has shown here, we simply ask that you and other community members come up with a fair process with a clear timeline, and decision-making criteria.

We would request that a vote start NLT 1200 PST, Dec 6, 2021. Secondly, whatever vote date is selected, we would kindly ask that the proposals be locked one week before the vote, so we can revise our proposal one last time, given that our proposal has been open for scrutiny and copying since we proposed it on Nov 4, 2021.

We appreciate that this is a new process for Compound - and DAOs in general - and look forward to continuing to work with you all in breaking new ground!

4 Likes
21

I appreciate the write-ups from @OZSecure (OpenZeppelin) and @dguido (Trail of Bits). These teams’ work with Compound Labs on the protocol’s core contracts, especially pre-governance, was invaluable for the bootstrapping of the protocol. As a community member and user, I deeply appreciate your past work for Compound and your broader efforts as good actors in the smart contract security space.

To me, the tenor and character of this current thread feel out of step with the bona fides of OZ’s and ToB’s prior work with Compound Labs. Let me try to lay out the key discrepancies I’m seeing. I am hoping it could help us think through the origins of the current tension; increase community buy-in for @jamico’s suggestion on a path forward; and hopefully identify some ways to circumvent such tensions in future efforts as stewards of the protocol.

Conflicts of interest: I appreciate @sukernik’s transparency at the top of this thread in explaining how the proposal from OZ came to us. That said, declaration of a conflict of interest doesn’t absolve us of the need to agree on how to work around it. In this case, it seems likely that there was some degree of information or timing asymmetry between the vendor with whom @sukernik has a business relationship (OZ) and the vendor with whom he presumably does not (ToB).

Now, we have one vendor (OZ) that has been, at times in this thread, projecting a sense of entitlement to an up-down vote as a sole-source provider, and another vendor (ToB) stuck playing catch-up through no fault of their own while trying to manage the optics of the awkward situation that we have placed them in.

I have to say, I am a bit baffled by the number and variety of high-pressure sales tactics employed by @OZSecure throughout this thread, from aggressive up-selling relative to @sukernik’s initial outreach, to intentionally promoting an atmosphere of urgency, to rather bizarrely accusing ToB of copying aspects of OZ’s proposal that any reasonable vendor would include. In my eyes, these tactics are incongruous with, and damaging to, OZ’s good reputation.

I see two take-aways here, one for community members taking on new initiatives, and one for vendors proposing fee-based services to the protocol:

  • Community members: Let’s please at least start a thread here on the forums before engaging in direct outreach to potential vendors for any new initiatives that could possibly involve a distribution of COMP, especially if there are known conflicts of interest involved. Otherwise, vendors like ToB (and others who might be well-qualified but whom we might not personally know) have every reason to question the integrity of the process.

  • Service providers: Especially in cases where a conflict of interest may exist, prospective service providers should be careful to keep proposals clearly in-scope of the request. Here, the request was audit review of new governance proposals prior to on-chain execution; what OZ proposed in response is a much more elaborate engagement, some of which is necessary to deliver the core service, some of which is almost certainly not.
    For example, I love the idea of an experienced auditing firm providing educational materials for community members seeking to improve the protocol, and I would likely be a user of this resource if it existed. Furthermore, OZ’s suite of public contract templates offer ample evidence that they would likely do a fantastic job. However, it is related, but quite far removed from, the core aims of this security audit request and, at least in my view, should be considered as a separate proposal.

Cost model: I’m not qualified to objectively assess how reasonable the costs are, but I will state as an outsider to the security audit industry that they seem absurdly, even comically high to me (like, by about one order of magnitude). While I acknowledge that the community is likely to support paying a premium relative to reasonable market rates, any additional supporting evidence OZ can provide to justify the cost structure relative to industry standards would be helpful here.

I also agree with community members who question the cost structure of OZ’s proposal and its performance fee: I do not consider a performance fee model to be appropriate for this work. If the service rendered is to secure the protocol, why should the service provider get extra cookies every quarter the protocol is not exploited? Compound does not need to offer an extra incentive here. The service provider’s motivation to do their work well should be to score the next security agreement with the protocol, not to earn extra compensation for services already rendered.

Finally, I agree with @arr00 that compensation for this work should not be streamed to the vendor. It makes more sense to me that it be paid out periodically (perhaps quarterly for one year, then annually) with a new proposal to authorize the payment for each following period.

The question of payment schedules leads to an important final follow-up question: how can the community secure auditing services beyond the runway of the current COMP distribution schedule, which is only a few years iirc? While this question goes beyond the core focus of the audit proposals, it is worth discussing in this context as we continue what I consider to be the most important discussion we’re having: how to adjust COMP distribution for the health, security, and sustainability of the protocol.

9 Likes
22

@allthecolors – as always, your commentary is extremely thoughtful. I’m sure @sukernik will explain, but OZ and TOB were contacted around the same time — OZ just happened to move a lot quicker on this. And the process run here seemed to give ample time for other auditors to throw in a competing proposal. OZ’s request for the other auditors & the COMP community to move quickly on this seems reasonable given the timeline they’ve worked with so far (though I think the Dec 6th deadline is too tight, given the holidays).

This idea had been discussed by a number of folks in the community long before Larry finally took the initiative to have the conversations and make this happen. There’s an intricate web of potential conflicts of interests (some obvious, some less obvious) for just about any leader in the crypto community, so it’s not surprising to see this here. I think the best we can do is just be transparent about potential COI’s. In some ways, Larry’s close relationship with OZ may have been valuable in helping mobilize OZ to make the prop it did (and perhaps he should even be compensated for this!). COIs can be dangerous if they lead to proposals that are misaligned with the long-term health of COMP, but in this situation, COMP would be very lucky to have OZ or TOB on our side, both are excellent options.

The [scope of the proposal] and [cost] concerns are also very valid. I think the service providers should be free to propose whatever they want… if their proposals are off the mark from what the COMP community would like, we should negotiate the services/terms. This highlights a key problem — it’s really hard to negotiate the right deals with so many voices on an open forum, especially when considering multiple proposals. (h/t @sukernik who planned to bring up this exact issue on the community call).

Perhaps forming a pod (committee) of people with expertise in structuring and negotiating these types of deals and giving them the autonomy to negotiate off-forum with OZ, TOB, and others to get a deal done would be a better way to accomplish the objective here.

(p.s. Happy Thanksgiving to anyone celebrating!)

5 Likes