sellerator/README.md

Sellerator

In online marketplaces, product listings often include some seller statistics such as percentage of positive ratings, out of how many total ratings. Sellers often defend these ratings vigorously, and for good reason: if a seller has had 10,000 reviews, and 98% of those reviews are positive, it implies that they are reliably getting something wrong about 1 in 50 times. Compare them to a seller with the same number of reviews, but a 99.5% positive rate: now something only goes wrong once in every 200 times, which sounds like a much better bet.

Now, it doesn't pay to only focus on the percentage. A seller with a 100% score who has only sold ten items so far is much less certain than a seller with 100% who has received 1,000 reviews. In statistics, such "track records" are modelled using confidence intervals: the more reviews a seller has, the narrower the confidence interval for their positive rate.

We think of each seller as a weighted coin which might come up heads most of the time, but we want to know whether it's tails one in four or one in a hundred times. If the coin has only been flipped a small number of times, it's hard to guess the exact probability of heads.

Furthermore, when assessing a seller, we usually aren't interested in the best case. We're interested in the risk that something will go wrong with our order. Thus it pays to focus on the lower end of the confidence interval. For all we know, the seller with only 10 reviews so far might get something wrong every four times, but they just got lucky so far.

This program calculates the worst case odds of something going wrong.

Examples

10 reviews, 100% positive

$ ./sellerator 10 100
2.902387

So the worst case scenario is they get something wrong about one package in every three. Maybe they're better than that, but for all we know, they just got lucky so far.

You shouldn't conclude they're a bad seller. They're just new, is all. But you might want to wait for them to establish more of a track record before trusting them with your order.

100 reviews, varying positive levels

$ echo 98 99 99.2 99.5 99.9 100 | xargs -n1 ./sellerator 100
12.925678
16.022218
16.866848
18.352825
20.909134
21.690726

For the higher percentages, something going wrong 1 in 20 times may or may not be okay by you.

1000 reviews, varying positive levels

$ echo 98 99 99.2 99.5 99.9 100 | xargs -n1 ./sellerator 1000
31.958048
52.767013
61.112010
81.014310
154.585006
209.605337

Now we're talking. Even at 99%, the seller has established enough consistency that a 1 in 50 risk doesn't sound so bad.

Thousands of reviews and competing on percentages close to 100%

Big sellers often have tens of thousands of reviews and a rating above 99.5% percent. The competition is fierce here and you might just be better off focusing on price. But to give you a feel for how the different numbers can play out:

for i in 10000 20000 50000 100000; do
    printf "%-7s %6.2f %6.2f %6.2f %6.2f %6.2f %6.2f %6.2f %6.2f\n" ${i}: \
        $(./sellerator $i 99.1) \
        $(./sellerator $i 99.3) \
        $(./sellerator $i 99.5) \
        $(./sellerator $i 99.6) \
        $(./sellerator $i 99.7) \
        $(./sellerator $i 99.8) \
        $(./sellerator $i 99.9) \
        $(./sellerator $i 100)
done
10000:   90.06 112.49 150.56 181.83 230.51 317.67 525.06 2088.75
20000:   95.85 120.76 163.81 199.89 257.24 363.52 635.12 4176.70
50000:  101.25 128.53 176.43 217.23 283.30 409.39 752.23 10440.53
100000: 104.06 132.60 183.09 226.45 297.28 434.40 818.67 20880.25

This shows that if your personal risk threshold is a 1-in-200 chance of an order going wrong, you would require a 99.6% positive rating for a seller with only 10 or 20 thousand reviews, but for 50,000 or more you could settle for 99.5% :)

A risk matrix approach

You might decide that if a seller has a weaker reputation, it's okay to risk small amounts of money on them; it's only for more expensive orders that you are more risk averse. Venturing small orders can then help new seller build their reputation, which is good if it establishes more competition in the market.

A simple way to do this would be to decide an "online risk tax" for yourself of, say, $5 per order. If you multiply this by the seller's odds, you get an upper limit on how much you should feel comfortable spending with that seller.

Let's consider our new seller who so far got good reviews, but we don't yet know whether this trend will continue:

$ bc -l <<< "5*$(./sellerator 10 100)"
14.511935

So we might try the seller out by buying something worth less than $15. If it works out, we give them a good review, and their score improves slightly:

$ ./sellerator 11 100
3.110860

$ bc -l <<< "5*$(./sellerator 11 100)"
15.554300

On the other hand, we might be looking to buy a new laptop with a budget of $1500. Our $5 hedge is 1/300 of the budget. Looking at the table above for highly competitive sellers, we should obviously stick to any sellers with 10,000+ reviews who have ratings of 99.8% or better.