13  Estimating Things You Can’t See

Builds the Hill estimator, but frames it around the philosophical problem first: the tail is exactly the region where you have the fewest observations.

import math

def hill_estimator(data, k):
    top_k = sorted(data, reverse=True)[:k]
    x_k = top_k[-1]
    log_ratios = [math.log(x / x_k) for x in top_k[:-1]]
    return len(log_ratios) / sum(log_ratios)

The central exercise is a Hill plot — sweeping k and plotting the resulting alpha estimates — rather than trusting any single value:

def hill_plot_values(data, k_range):
    return [(k, hill_estimator(data, k)) for k in k_range]

For genuine Pareto data, a plateau emerges across a middle range of k. For Gaussian data, no plateau appears — the estimate drifts. The lesson: the estimator will always return a number; only the plateau (or its absence) tells you whether that number means anything.