Fat Tailed Data Science from Scratch

First Principles of Pragmatic Data Science

1 Preface

In college I read Data Science from Scratch by Joel Gus and remember first principles examples and hands-on code snippets. I remember spending time in some of my boring classes implementing the examples in Atom and playing with the data, the values, and some of the assumptions. I really resonate with the premise of the book: do not treat data science as a collection of library calls whose assumptions are hidden behind an opaque API. Build the tools yourself, in small pieces, and you can see what they compute and when they deserve your trust.

At the time, I couldn’t help but think about how unrealistic some of the math and statistics in it were, though. I figured, very few things in the world actually seem to work with the formulas and distributions listed in the book, but at the time, it seemed to look similar to most other stats or probability textbook material, so, what’s there to do. Well, since then, I’ve managed to get exposure to a lot more statistics and probability and, at least in my network, so-called “Fat Tailed” distributions have been all the rage. These distributions seem to break all the classical assumptions where much of the familiar introductory toolkit is most comfortabl e.g. observations don’t cluster around a center, very large deviations aren’t negligible, variance is useless, and averages never settle down, etc. etc.. This kind of data seems to make much more sense in modeling the real world. So what would a from-scratch data-science book look like if it started instead with this kind of statistics and probability?

That question is the origin of Fat Tailed Data Science from Scratch.

Doing math in this way ruins a lot of assumptions even professionals make when, for example, simulating real world conditions or cleaning data for training machine learning models. Aome of its most common tools no longer work:

• Means, standard deviations, and z-scores rely on moments that may not exist or may settle too slowly to estimate reliably.
• Covariance, correlation, and PCA rely on squared variation; a principal component can be driven by a few extreme observations rather than broad structure in the data.
• Least squares and squared-error loss give an extreme residual disproportionate influence over a fitted model.
• Gaussian confidence intervals, standard errors, and the ordinary central-limit heuristic rely on assumptions that heavy tails can violate.
• K-means and other Euclidean-distance methods can let a small number of large values pull centroids and distances away from the observations they are meant to summarize.

So while many of the traditional techniques will still be more useful in job interviews or getting a passing grade in class but learning what I’ll call the “the dark arts” of probability can give someone a much richer understanding of the world. In many practical settings, the most consequential observations are also the least well described by the comfortable center-and-variance narrative e.g. unusually large losses, demand spikes, long latencies, city sizes, highly connected nodes, and concentrated wealth.

When talking about this kind of math, there’s some jargon that makes sense to define first:

• An outlier may have come from a different process: an error, a recording problem, or contamination.
• A rare event has low probability; that alone says nothing about the distribution’s tail.
• Skew describes asymmetry; it does not tell us how much probability remains far from the center.
• A fat tail is a property of the generating process itself. Its extremes are expected outputs of that process, even when they are infrequent.

Calling every extreme value an outlier is therefore not caution. It can be a way of discarding the evidence the data is offering about the system that produced it.

The book follows the same first-principles spirit as its inspiration. We write the small mechanisms before relying on polished implementations e.g. empirical distribution and survival functions, sampling routines, means and robust alternatives, order statistics, tail-index estimators, loss functions, and simple dependence diagnostics. The reusable versions live alongside the text as ordinary, tested Python code. The chapters use them rather than quietly maintaining separate versions of an algorithm.

This is a practical book and thus doesn’t try to explain everything in the world as a fat-tailed distributioned and if anything, this kind of math leaves one asking more questions than answers e.g. a straight line on a log-log plot is suggestive, not proof of a power law, an estimator always returning a number does not make that number reliable, finite samples may not distinguish a Pareto tail from a lognormal or another plausible model, especially in the region that matters most. Math is hard and the world is complicated thus the responsible conclusion is that sometimes the data cannot yet tell us enough. It is not a failure of technique but it is a fact a decision should acknowledge.

If this book succeeds, its most durable contribution will be a habit rather than a formula: when someone hands you an average, a variance, a confidence interval, a correlation, or a squared-error fit, ask what happened in the tail before you believe it.

1.1 Who This Book Is For

This book is for beginner and intermediate programmers who are curious about mathematics, statistics, and probability. You do not need to arrive as an extreme-value theorist. More important is a willingness to inspect what a summary statistic assumes before acting on it. This book can be read as a continuation of Data Science from Scratch: readers who enjoyed its first-principles approach will recognize the spirit of the exercises and can treat this as a Part 2 that puts tails at the center.

Reading Data Science from Scratch is not required. This book stands on its own and builds the probability and statistical ideas it needs as it goes. Curiosity and a willingness to work through small examples matter more here than prior exposure to extreme-value theory or formal statistics.

1.2 How to Read This Book

If this is your first sustained encounter with Python, begin with Chapter 0 and then read the book in order. Each part supplies the vocabulary and mechanism needed for the next. Readers coming from Data Science from Scratch can treat Chapter 0 as a quick refresher and begin with Chapter 1. In either path, run the examples, change a value, and see which conclusions move with it. The experiments are part of the argument, not decoration around it.

1.3 What You Need

A computer with Python, basic algebra, and the patience to work through a small program are enough to begin. Chapter 0 provides a practical Python on-ramp; it is optional for experienced Python readers. It is not a replacement for a full programming course, so readers entirely new to programming may want a slower introductory resource alongside this book. No prior course in probability, statistics, machine learning, or extreme-value theory is required.

1.4 Code, Experiments, and Claims

The repository accompanying this book (https://github.com/jonaylor89/FatTailedDataScienceFromScratch) contains the reusable implementations and the tests that support computational claims. The chapters deliberately start with small, inspectable Python rather than a library call whose assumptions are hidden. Run the code, but do not mistake numerical output for a universal result: simulations illustrate a mechanism, and diagnostics can suggest a tail model without proving one.