On the goverance call today, I introduced building a medianizer and received a warm reception. @sorawit and the Band Protocol Team have stepped up and taken on the role of developing a solution. They believe they can build a system that meets the guidelines I posted. On today’s call, the Band Team said they hope to present to the community at the next governance call.
The team at Band has developed their own oracle system that “allows developers to query any data including real-world events, sports, weather, random numbers and more.” You can learn more about their work here.
I am thrilled there is a strong team interested in executing this upgrade, and I think it would be a great example of governance working if this succeeds. In addition, this medianizer has the potential to be a tool for the whole DeFi community to utilize. Unlike other solutions, this would be a totally open source and free way for developers to get price info.
Just to clarify, Chainlink has numerous independent nodes and data sources per oracle feed, such as 21 nodes on the ETH/USD price feed currently and soon 31 in their OCR launch. It’s already decentralized, so one node or even several nodes going down will not cause a failure. Also, not as well known, but Chainlink has two separate node clients running at the same time, so if there were a problem in the core Chainlink software client then it would just failover to the previous version. With its OCR launch appearing to be very soon, it will actually have 3 client versions, for extra redundancy and failover protection. Where would this point of failure you described be if there are numerous independent nodes, data sources, and software client versions?
You are right. From a technical point of view Chainlink alone would be the perfect solution. The problem is, that for unknown reasons, the original team decided not to use Chainlink and the sentiment against Chainlink seems to be unchanged. So, I think to argue technically for Chainlink doesn’t make sense, as this is a political decision.
I would like to raise the community attention to the fact that coinbase oracle api is sent with different delays to different IP addresses.
The delay is of full minutes (not only few seconds), and probably stem from a sub-optimal REST API system on coinbase side.
This give unfair advantage to certain liquidators who get the price feed update before others, and this advantage can amount to millions of $ (e.g., in the $90M liquidation event).
We first realized this issue while working on the B.Protocol integration with Compound, and @blck also confirmed it after we raised the issue at the discord channel.
Moreover @blck pointed out to us that different IP addresses have different delay.
Centralized exchanges are notorious for giving different service classes to different players, with 0 transparency.
And when designing new price feed, this should be taken into account. Relying on off-chain could bring CeFi dirty tricks to DeFi.
Thank you for pointing this out. I would love to see some test results that prove this if you have them available. I am not surprised that this is happening, unfortunately.
Part of the big picture with building the medianizer is adding many more price feeds, and in particular onchain sources, so that not any one source can exert pressure on an asset’s price.
The issue here is not price manipulation, but rather that knowing even epsilon change in price could worth millions in certain scenarios.
A practical approach is to demand a functional websocket service from coinbase (and okex) with no delays.
On the smart contract level, one can think of other mitigations as well, e.g., some build in delay between the price update and the liqudiation.
But if there is actually governance here then that shouldn’t matter. It should be the best solution as voted on by the community. The idea that the team’s wanting or not wanting something for political reasons is still a thing, either means that:
Governance does not matter because a few people /entities control the majority of votes
We have not put it to a vote to actually find out.
Thanks for sharing this @yaronvel - this a good example of the disparity in data quality/transparency we are going to continue running into while constructing a patchwork of various price feeds. I’m concerned people underestimate the impact one bad apple can have on the bunch.
We could build the best medianizer in the world, but it is nonetheless beholden to the quality of data it receives. Chainlink, whose nodes source from top aggregators, will always provide the best data. And just think of how much time/energy we could save to put towards things like Compound Chain, adding new collateral etc.
Good point. A quick look at the leaderboard shows that 5 entities have over 50% of the voting weight, meaning together can effectively pass/reject any proposal. Meanwhile I have no idea what they think about the oracle situation, which is no doubt odd. It would be nice to have a clearer understanding of their intentions, reasoning etc. to help better guide our conversation. If it is political, I’d like to know why.
If you have questions, comments, ideas about the medianizer, you are welcome to post them here. At this point, we have an initial consensus that the community would like this idea developed and are no longer debating the merits of developing a medianizer in this channel. If you disagree with this development, then I suggest you try to enact a change yourself or vote against this when given the opportunity.
I will happily debate transitioning the Open Oracle Feed to a Chainlink service, but this is no longer the place to have that debate.
When discussing a major protocol change, there shouldn’t be a point at which discussing its merits is over, even if there is ‘initial consensus.’
We are already uncovering issues with data quality, and if this approach doesn’t end up solving the problems the community thinks it will then it’s a lot of wasted time, development effort, or worse.
Out of respect I will dial it back and allow this thread to be more development focused, but I will be keeping eye on the progress here and continuing to voice any new concerns.
Chainlink updates on price deviations, such as every .5% change in price will trigger an update. So the volatility of the asset over the 24 hour period will determine how much the price feed is updated.
Is there a list of data sources that will be used for this? I’m not sure how such a medianizer will ensure market coverage if it’s taking the median value from a mixture of oracle price feeds that track many exchanges directly with raw exchange data from single exchanges. Not all data sources are created equal, sources with more market coverage should be weighted higher. Decentralization for pricing data is obviously important, but I don’t think it should come at the expense of market coverage or data quality in general. Market coverage is the primary issue that this solution should seek to solve.
Great question, picking which data sources are safe to add will be interesting once the medianizer is built. For now, the goal is to develop a system that can take input from both offchain and onchain sources, and it will be up to the community to decide what sources are safe to add. Once we know how the medianizer is going to be structured, we can begin to discuss which sources the community wants to add in addition to Coinbase. I will likely start a separate forum thread once it is time to start that conversation. I will say I am partial to have many sources rather than just a handful, here is an interesting article to read related to the topic.
I still don’t understand how this medianizer solves the issue of market coverage if it’s still going to be weighting every data source equally. Raw data from a single exchange is of a very different quality/utility than a decentralized oracle network that pulls from multiple data aggregators who track all exchanges by weighing each by real volume. Not every oracle network is of the same quality either so that’s why I think the medianizer shouldn’t just be strictly a “median picker” because that doesn’t account for different data qualities from different sources.
I can give an example, if we used a medianizer that pulls from five sources: Coinbase, OKEx, Uniswap, Chainlink price feeds, Band price feeds, and then volume/liquidity of an asset consolidates to an exchange that isn’t Coinbase, OKEx, or Uniswap (let’s say it goes to Binance), then those three exchanges being used in the medianizer can become manipulated for very little cost and end up controlling the median value because those three exchanges are being weighted equally to the two price feeds that aggregate from a much broader collection of exchanges including Binance (the median of five sources takes the third value in an ordered list). This may or may not be an issue for a highly liquid assets like ETH, but for something less liquid like ZRX or for a new less liquid collateral as Compound scales, it could cause a abnormal deviation which could allow for an undercollateralized loan to be created on or for false liquidations to occur.
The medianizer can be built, but we still need to discuss its merits and determine if it will end up solving the problem we think it is before a full integration. I worry that it is simply increasing the attack surface of Compound by mixing lower quality raw data with higher quality refined data.
Adding more oracles requires increased gas usage. Compound’s current OPF oracles are extremely gas efficient since they just parrot back a stored answer. Other on-chain oracles are not as efficient and can cost a lot more gas.
Given this, a possible solution would be to have an intermediary contract that periodically does the gas expensive tasks of collecting prices from a variety of sources, then distilling that prices down to a single trusted price that it stores. Compound would just fetch this stored price. This could use the same fetching API as the current OPF oracles, so that the ctoken code does not need to change. This would still maintain the gas efficiency of the current setup.
Agree, that we can react to event. Aggressive liquidation with 8% premium for liquidator is great incentive for manipulator, and users are not protected.
We cant get any report or explanation from centralize exchange when event like “DAI liquidation” happens. I wish I could investigate an “unforeseen event” on the blockchain.
I dont think that is good job, I think we stuck with Coinbase.
Currently, Compound Finance is using one source (Coinbase) for getting prices. The medianizer will be a huge upgrade and allow the protocol to consider adding more price feeds safely. It will be up to governance to only add high-quality sources for price data.
I think you are underestimating how much it costs to move markets of the assets on Compound. As a market maker, I can tell you the market is fairly efficient and has matured significantly over the last year.
Compound Finance should not be adding any assets to the platform that are only traded at a few exchanges and have limited liquidity. We should be only adding high-quality ERC20s.
First post here. I built Maker’s Oracles so I think I can provide a bit of perspective here.
Maker uses a Medianizer which takes the median of a set of Feeds. Each Feed queries a wide sample of exchanges (both on-chain and offchain) to compose it’s own price. This gives us good market-coverage without relying on any single entity. A core piece of Maker’s collateral onboarding process is determining the Data Model (the set of sources) for an asset.
So I think the Medianizer model works great, but I’m a bit concerned by the implementation being discussed here. I want to caution against mixing individual sources like Uniswap and Coinbase along with aggregated sources like Chainlink, Band, or Maker Oracles because you end up overweighting some sources over others which facilitates Oracle manipulation. On top of that, many of the aggregators have another layer of obscurity where they use data provider services rather than exchange data itself. So you end up with this really opaque mess where there are hidden vulnerabilities for exploiting the Oracle, but it’s not clear where they are.
My suggestion is to keep things simple.
Using signed exchange data is very gas-efficient and negates the need for trusted parties. However, not enough exchanges currently sign their price data, making it very difficult to build a robust manipulation-resistant Oracle. Doing the hybrid model (Open Oracle + aggregator sources) doesn’t fix anything and just creates hidden risks. So in my opinion, the Compound community has three options.
Convince more exchanges to sign price data. In the interim period before sufficient exchanges are onboarded the Compound Protocol might be vulnerable.
Replace the Compound Open Oracle with an aggregated Oracle service like Chainlink, Maker, or Band. This is the easiest solution, but reduces Compound’s flexibility by adding an external dependency. This reduced flexibility is why Maker created its own Oracles rather than going with something like Chainlink.
Fork one of the Oracle protocols and onboard your own set of trusted actors (Feeds) via Compound Governance. This is a lot more work, since you have additional infra to maintain and support, and have the governance overhead of onboarding “Feeds”. The benefit here is that Compound’s community would have maximum flexibility to integrate whatever asset they want with their own Data Model and don’t have to rely on an external org for deliverables.
I’ll throw Maker’s medianizer in the ring that’s been in use in Multi-Collateral Dai since launch as an option for both (2) and (3). Here’s a dashboard showing the medianized Oracle prices for Maker.
