I believe Compound needs a more robust deployment process if it is to stand the test of time. The recent COMP distribution bug and our inability to fix it for the next 7 days highlight the gaps in the current process. Improving the code review and testing process is a must, but still insufficient. Any stable software system needs the ability to quickly rollback changes, and rollout updates in a phased manner. No amount of code review and testing can compensate for this.
- Rollback capability - it currently takes at least 7 days to make changes to the protocol. I know there are good reasons for this process, but we also need the ability to quickly deploy fixes in emergencies. As such, I propose that we create a separate process for rolling back changes. To prevent abuse, we can have different parameters around the process such as an elevated minimum votes required, max timelapse since deploy, versions that are eligible to be rolled back to (if possible), etc. IMO, as engineers, we are responsible to ensure a minimum level of safety in the systems we build, and I don’t think we will ever get there with the 7-day delay constraint. It’s like we are building a major bridge in San Francisco and not making it earthquake proof. Safety needs to come first.
- Phased Rollout capability - a standard practice of traditional software that I believe we can implement on the blockchain to make Compound more robust. Similar to how AB tests are implemented, we can require all major code changes to branch users (into new code vs old code) based on a predefined schedule (e.g. Day 0 - 1%, Day 2 - 5%, Day 7 - 10%, etc). This gives us time to monitor changes, deploy fixes, and minimize the impact of bugs.
Robert mentioned on Discord recently that Compound is decentralized to “ensure that the protocol can run for 100 years”. That’s the future I want to see, and I believe we need to engineer a more robust system so that we can realize that dream. We are lucky that the recent bug was only around COMP distribution. It could have been much worse, and the story would have been that Compound is decentralized which caused the protocol to only last for 3 years. Let’s learn from this and ensure our future is the former and not the latter.