Proposal To Upgrade To UAV V3

This would mean aggregating and averaging V2 and V3 prices weighted by their respective liquidity, correct? Many newer tokens have a majority of liquidity concentrated on V3 - would hate to see low liquidity on V2 be an attack vector or a barrier to asset listing.

Thanks for working on this @GFXlabs - low liquidity on V2 has been a barrier to asset listing for many great projects.

I’m curious what Compound’s past with using the failover has been. Do chainlink oracles go down for extended periods of time? Have any TWAP attacks been tried before?

The failover has never been activated. So far we haven’t experienced any issues although the price feeds aren’t being monitored as far as I’m aware.

This would mean aggregating and averaging V2 and V3 prices weighted by their respective liquidity, correct? Many newer tokens have a majority of liquidity concentrated on V3 - would hate to see low liquidity on V2 be an attack vector or a barrier to asset listing.

Aggregating prices across multiple DEXs would be best. There’s not always going to be sufficient liquidity on Uniswap V2 or even V3. Sometimes the majority of liquidity can even be on Sushiswap or Curve in rare cases such as stETH-ETH. Fragmented liquidity has been a barrier to asset listing in the past and will likely continue to be a barrier, unfortunately.

I’d like to note that the minimum liquidity required for a Uniswap V2 pool is a lot lower than most people think. If we want the price to be 99% accurate, it must be profitable to perform arbitrage on the pool resulting in about a 1% change in price. This is a function of both liquidity and gas price. Because we use TWAPs (assuming the pool has minimum required liquidity for an accurate price), an attacker would have to manipulate the price every block over the TWAP period. It becomes even more expensive with miners performing arbitrage as well.

The real question becomes how do we guarantee that the fallback pool always has enough liquidity for an accurate price.

1 Like

I also have a comment on Uniswap v3’s observation cardinalities.

All of the UAV price updates call the Uniswap v3 pool oracle’s observe function. This function will revert if the pool oracle doesn’t have an observation for the past 15 minutes (i.e. the TWAP period).

Up to one observation is written per block (if there’s at least one trade in the block). So we must handle the worst case scenario where there’s a trade in the pool every block. Therefore, the underlying pool oracle’s observation cardinality must be at least 15 minutes / blockTime = 900 seconds / 10 seconds = 90. We use 10 seconds for block time as that’s what’s expected after the merge. Since blocks aren’t produced exactly every 10 seconds (or 11 seconds on average currently), we must use a bit higher than a cardinality of 90. I’d suggest requiring an observation cardinality of 120 to be safe.

Hi Compound Community,

The Chainlink Labs team is pleased to announce that we have successfully deployed the UniswapAnchoredView (UAV) V3 using the latest UAV configurations!

UniswapAnchoredView Contract

To recap, the update from Uniswap V2 to Uniswap V3 pools is crucial because V2’s liquidity continues to drop as liquidity providers migrate to V3. Compound’s UAV Oracle must follow this shift to provide a highly reliable and highly-secure price feed.

Below is the summary of the changes:

In addition, we’ve also increased the Uniswap observation cardinality to 150 on pools that were below 150. To elaborate, the “cardinality” in Uniswap is a term for the TWAP lookback history. It represents the state changes per block, so ten trades in a block get condensed and recorded as one update in the lookback. To save on gas costs, the cardinality starts at 2, and anyone can pay the gas cost for a deeper lookback. Therefore, increasing the cardinality to 150 ensures a safe minimum for the lookback history.

Uniswap V3 Liquidity Pools - As of Aug 5

Listed below are the Uniswap V3 pools and liquidity levels for each price feed in the UAV3. Note that it is required to reference the ETH-paired pools specifically:

  • ETH: $185M - 0x88e6a0c2ddd26feeb64f039a2c41296fcb3f5640
  • DAI: $19M - 0xc2e9f25be6257c210d7adf0d4cd6e3e881ba25f8
  • USDC: N/A
  • USDT: N/A
  • TUSD: N/A
  • WBTC: $244M - 0xcbcdf9626bc03e24f779434178a73a0b4bad62ed
  • BAT: $827K - 0xae614a7a56cb79c04df2aeba6f5dab80a39ca78e
  • ZRX: $928K - 0x14424eeecbff345b38187d0b8b749e56faa68539
  • REPv2: $30.5K - 0xb055103b7633b61518cd806d95beeb2d4cd217e7
  • SAI: N/A
  • UNI: $16.2M - 0x1d42064fc4beb5f8aaf85f4617ae8b3b5b8bd801
  • COMP: $4.3M - 0xea4ba4ce14fdd287f380b55419b1c5b6c3f22ab6
  • LINK: $8.1M - 0xa6cc3c2531fdaa6ae1a3ca84c2855806728693e8
  • AAVE: $3.9M - 0x5ab53ee1d50eef2c1dd3d5402789cd27bb52c1bb
  • SUSHI: $56.5K - 0x73a6a761fe483ba19debb8f56ac5bbf14c0cdad1
  • MKR: $11.8M - 0xe8c6c9227491c0a8156a0106a0204d881bb7e531
  • YFI: $178.5K - 0x04916039b1f59d9745bf6e0a21f191d1e0a84287
  • FEI: $233 - 0x2028D7Ef0223C45caDBF05E13F1823c1228012BF

We note that low liquidity in Uniswap v3 pools makes the TWAP price in those pools easier to manipulate. That may cause the Uniswap anchor to be less accurate than expected. However, this will not affect the accuracy of the primary price feed from Chainlink.

Next Steps

If the Compound community supports this upgrade, the following are the recommended next steps to proceed with this proposal:

  1. The Compound community will signal support using the above Uniswap V3 pools for the current UAV3 deployment.
  2. If supported, Chainlink Labs will configure all existing validator proxies to push prices to the new UAV candidate while continually pushing to the existing one. This is so that Compound voters can see the new UAV working as it should while the current UAV maintains full operation.
  3. Lastly, if the vote is passed and executed on-chain, The Comptroller will point to UAV V3 and use it for prices. At this stage, Chainlink will decommission UAV V2 by setting the ValidatorProxies to only point to the new one.
3 Likes

Really excited for Proposal 117 - fantastic collaboration all around to get to this :raised_hands:

2 Likes

integration with uniswap v3 pools sounds great.
and I’d like to check something before my voting activity.

I have checked compound v3 document but can not find out an anchor mechanism in there. It looks like comet is only depend on Chainlink price feed. isn’t it?

Hi @dakeshi, that’s right - Compound III is designed to use Chainlink Price Feeds directly. This is in line with how other major protocols use Chainlink. It also allows the Compound community to onboard new markets and deploy on more chains without a dependency on Uniswap.

An hour ago, Proposal 117 was executed, which updated the price feed that Compound v2 uses. This price feed contained an error that is causing transactions for ETH suppliers and borrowers to revert. Effectively, the cETH market is temporarily frozen.

OpenZeppelin, Chainlink, Compound Labs, and many members of the community are working to diagnose and fix this issue ASAP; GFX has created Proposal 119 which will revert the price feed to it’s original state.

You can still supply Ether collateral, and no users should be at risk of liquidation, or at risk of losing funds. If you have any questions, please join the community in Discord.

3 Likes

Here is an update on the incident and next steps for the Compound community:

On Aug 30th at 6:20 PM UTC, Compound Proposal 117 was executed to upgrade the Oracle Contract to a new version that uses Uniswap V3 instead of V2 for price feeds. It had passed a governance vote after being proposed by GFX Labs on behalf of ChainLink. The changes were reviewed by OpenZeppelin along with Dedaub and ABDK.

At 6:38 PM UTC, the proposal executor noticed that all calls coming from the Comptroller to getUnderlyingPrice for the cETH market were reverting and notified Compound developers of the issue. All other cToken markets appeared to be unaffected. Compound Labs, ChainLink, GFX Labs and OpenZeppelin teams were immediately notified and jumped into a war room.

At around 6:47 PM UTC, the proposal executor reported that because cETH does not have an underlying() method assumed to be present in every cToken contract by the new oracle implementation, the getUnderlyingPrice function returns empty bytes that cannot be decoded and the call reverts. This source of the issue was verified by Compound Labs and OpenZeppelin shortly after. This leaves withdrawals and liquidations in the cETH market effectively frozen while the issue remains in place.

At 7:14 PM UTC, a new proposal was submitted by GFX Labs to revert the upgrade and return to using the V2 Oracle contract. It is set to be passed and executed after a 7 day governance process. In the meantime, Compound Labs issued Twitter and Discord posts notifying users of the price feed issue while the war room participants continued to investigate the issue further to determine the impact on existing users.

The primary issue right now is a temporary denial of service for the cETH market which will be resolved by the new governance proposal. No funds are at risk at this time. The rest of the cToken markets on Compound V2 and all of V3 remain functional.

However, any users that deposited ETH and obtained cETH for opening borrow positions must be aware that they might get instantly liquidated whenever the fix proposal executes IF by that time the price of ETH has dropped significantly. These users can add collateral and repay borrowed assets normally to cover for eventual price drops by monitoring their borrow positions accordingly.

We’ll continue to work with the Compound community to ensure a speedy resolution and keep everyone informed on updates as they come. Once the immediate situation is resolved, a post-mortem will be forthcoming.

7 Likes

What is the eventual longterm plan? To alter UAV3 to accommodate cETH not having an underlying() function and redeploy?

Yes, there appears to be a simple fix to UAV3 that will accommodate cETH. We’ll focus on that after resolving the current disruption.

1 Like

Here is an update on the price feed issue for the cETH market. The fix proposal is still underway and expected to be executed in ~6 days.

As mentioned previously, users should be able to avoid any sudden liquidations caused by price changes once the fix proposal goes through by adding collateral or repaying borrowed assets. However, we’ve identified that some downstream protocols may not be capable of managing their positions at this time. For that reason, there may be a need to pause liquidations immediately before the proposal is executed to give these protocols time to adjust their positions.

OpenZeppelin has worked with Arr00 to prepare a proactive proposal that can be used to unpause liquidation IF the Pause Guardian decides that pausing is necessary. If market conditions do not change drastically and/or all positions are safe, then this will be unnecessary and the proposal will be canceled. We have also tested this proposal thoroughly to ensure it will work as intended since this will be the first time the unpause functionality may be used.

If the Pause Guardian does need to pause liquidations following the fix to protect some positions from unfair liquidations, this will only last for ~23 hours as the unpause proposal will reverse it shortly afterwards. Since this pause will impact all cToken markets, not just cETH, there are risks assumed by the protocol in case any other markets suffer a drop in price but it should be minor over the course of ~23 hours. If you are part of a protocol that is unable to manage positions or may otherwise be at risk of liquidations when price feeds turn on, please let us know in the community Discord.

We’ll continue to keep the community updated as we work with the Pause Guardian and other Compound participants to ensure the resolution is as smooth as possible. As we monitor the situation over the next week, we will also begin to work on a post-mortem that can be shared shortly after the fix is finalized.

3 Likes

Thanks @cylon. For transparency, could you share a preliminary list of the aforementioned protocols?

Currently, only Index Coop as mentioned in the Compound discord. I’ve confirmed that a few others are not at risk and will take the next few days to see if anyone else comes forward with concerns.

2 Likes

@cylon Got it, thank you for clarifying. Have you/the team been in contact with the folks at Index Coop?

We have indeed been in touch with the Compound team, in discord and in DMs.

Thank you for flagging this on both forums!

2 Likes

Minor update: Proposal 118 was cancelled by Gauntlet to reduce changes to the v2 protocol until after the fix passes. Gauntlet intends to re-submit the proposal at a future date

2 Likes

Hey @CL_Michael, is it possible to add RAI (Pool: 0x14de8287adc90f0f95bf567c0707670de52e3813) to the UAV when it is redeployed? :pray:

The fix proposal has been executed and the cETH v2 markets are fully functional again. Pausing was not necessary to prevent unfair liquidations so Proposal 121 has been cancelled.

More updates will be shared in today’s community call followed by a post-mortem later this week.

2 Likes

A post-mortem of the price feed incident has been posted here: cETH Price Feed Incident: Post-Mortem