How We Monitor Performance

The most important thing for our appraisals is that they are accurate, so that our users and partners can have confidence in our work and use them reliably in their applications, protocols, trading strategies, etc.

At Upshot, we spend a lot of time measuring appraisal performance and updating our models to ensure that our appraisals are as accurate as possible. The primary way we do this is by comparing appraisals to realized sale values:

first, we gather together all sales for a set period of time (usually a few weeks)
for each sale price we gather, we look for the most recent appraisal value that we made prior to the sale
then we summarize the difference between the Upshot appraisal and real-world sale using some performance metrics

We use three metrics for measuring appraisal performance, shown in the table below:

Metric	Description	Rationale
MAPE	Median absolute percentage error (%)	Captures the ‘typical’ error. A MAPE of 10% would mean that half of our appraisals have an error of 10% or less.
90% APE	90th percentile of absolute percentage errors	1 in 10 appraisals would be as bad or worse than this value. We measure this to keep track of performance at the tails (not just what’s average or typical).
95% APE	95th percentile of absolute percentage errors	1 in 20 appraisals would be as bad or worse than this value. This provides an estimate of a worst case error rate.

Why These Three Metrics?

It’s not enough to know that the average performance is good - because in many collections with high liquidity, it’s relatively easy to estimate the value of less-rare items simply by tracking floor prices.

We spend a lot of time tracking the worst mistakes our models make, because this helps us to understand how we perform on rarer or complex assets that are more difficult to appraise. We use this insight to focus where changes are most needed in our models, creating a virtuous cycle of improvement. The changes we make in our models are never hand-engineered for specific assets though. We always make changes that have a positive effect on appraisal quality across and within multiple collections at the same time, thus eliminating the inefficiencies that arise from targeting assets too specifically. This principle has been allowing us to scale our models without sacrifice in appraisal quality or speed of onboarding new collections.

We’re particularly proud of the accuracy of our appraisals across these rarer, complex assets. We feel as though this is where our appraisals uniquely showcase their robustness.

Performance Measurement is the Foundation for How Upshot Appraisals Iterate and Improve

The work of making accurate appraisal models is never done - new collections mint with new types of utility and different trading dynamics, new metas emerge and of course, the market never stands still. Our laser focus on performance monitoring provides the foundation for continuous improvement.

We have adopted a formal model development cycle which has appraisal performance at its core, and is composed of the 3 steps shown in the diagram below: