Nassim Nicholas Taleb

Black Swan

Antifragile

Skin in the Game

Bed of Procrustes

Strange Phrases

artificia docuit fames - sophistication is born out of hunger

ingenium mala saepe movent - When life gives you a lemon, make a lemonade

necessity really is the mother of invention

peter drucker, jacques lacan

alan Greenspan

---

Roadmap

• Me:

do you know a roadmap to be like nassim taleb?

• Chat.openai.com :

Nassim Taleb is a well-known author, statistician, and risk analyst. To become like him, one might consider the following steps:

Read widely and deeply in a variety of fields, including mathematics, economics, philosophy, and history.

Develop a strong understanding of probability theory, statistics, and decision-making under uncertainty.

Develop critical thinking skills and a habit of questioning assumptions and conventional wisdom.

Cultivate a contrarian perspective and be willing to challenge prevailing ideas and theories.

Write and publish on topics related to your interests and expertise.

Build a personal brand and gain recognition in your field through speaking engagements, media appearances, and other public-facing activities.

Be ready to face criticism and have thick skin to face it.

Please note that becoming like Nassim Taleb, who is a renowned author and thinker with a unique perspective and set of experiences, will likely be a long-term and difficult journey. But, reading his books, articles and following his thoughts can give you a good idea of his perspective and you can adapt those to your own perspective and ideas.

• Me:

is there a curriculum to follow?

• Chat.openai.com :

There is no specific curriculum to follow to become like Nassim Taleb, as his background and expertise span a wide range of fields. However, some potential steps you can take to gain knowledge and skills in areas related to his work include:

Studying mathematics, particularly probability theory and statistics.

Taking courses in economics, finance, and risk management.

Reading classic works in philosophy, particularly those related to epistemology and decision-making.

Reading books and articles by Nassim Taleb and other thinkers in the fields of risk, uncertainty, and complex systems.

Practicing critical thinking and reasoning by analyzing and evaluating various arguments and perspectives.

Building your own intellectual toolkit through writing, debating, and other forms of active engagement.

Learning about the history of ideas, economics, science and mathematics.

Keep in mind that this is not an exhaustive list and you should tailor your studies to your own interests and goals, and also consider taking courses or pursuing degrees in fields that align with your interests. Additionally, it’s important to gain practical experience and develop your own ideas, either through internships, research projects, or independent study.

MINI LECTURES IN PROBABILITY

MINI-LESSON 1: Standard Deviation

Understanding Statistical Concepts: Standard Deviation and Correlation

We could address a fundamental issue with how statistical concepts are understood without delving too deeply into correlation. Specifically, there is a significant misunderstanding among both the general public and practicing statisticians regarding basic statistical measures.

Standard Deviation: Common Misconceptions

Let’s start by discussing standard deviation, a fundamental statistical measure used to quantify the amount of variation or dispersion in a set of data.

Definition and Calculation:

1. Mean Calculation:

   $$\bar{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$$

   where $\bar{x}$ is the mean of the data, $x_i$ are the individual observations, and $n$ is the total number of observations.

2. Variance Calculation:

   $$\text{Variance}=\frac{1}{n}\sum _{i=1}^n(x_i-\bar{x}{)}^2$$

3. Standard Deviation Calculation:

   $$\sigma =\sqrt{\text{Variance}}$$

The standard deviation measures how much each observation deviates from the mean, with larger deviations being given more weight because the deviations are squared in the variance calculation.

Common Misconception:

Many people mistakenly believe that standard deviation is simply the average amount by which individual data points deviate from the mean. For example, if you ask someone how much the temperature changes daily, they might describe it as the average change, not considering the squared deviations used in standard deviation.

Mathematical Misunderstanding:

Consider a dataset where most values are zero except for one value that is very large (e.g., 1,000,000), and the rest are zero:

• Data: $\{0,0,\dots ,0,1,000,000\}$ (with 1,000,000 observations, only one being 1,000,000)

  • Standard Deviation:

    $$\sigma =\sqrt{\frac{1}{n}\left((1,000,000-\bar{x}{)}^2\right)}\approx 1,000$$

  • Mean Absolute Deviation (MAD):

    $$\text{MAD}=\frac{1}{n}\sum _{i=1}^n|x_i-\bar{x}|$$

    For this dataset, MAD is significantly smaller.

The standard deviation can be disproportionately large compared to the mean absolute deviation because squaring the deviations emphasizes larger deviations more heavily.

Correlation: A Brief Overview

Definition and Calculation:

1. Covariance:

   $$\text{Cov}(X,Y)=\frac{1}{n}\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$$

2. Correlation Coefficient:

   $${\rho }_{X,Y}=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$$

   where ${\sigma }_X$ and ${\sigma }_Y$ are the standard deviations of $X$ and $Y$, respectively.

Misunderstandings:

People often think of correlation in terms of linear relationships between variables. However, correlation as a metric might fail in non-linear contexts or when there are deviations from the assumptions of linearity and normality.

In the context of Gaussian distributions, the ratio of standard deviation to mean absolute deviation (MAD) is:

$$\frac{\sigma }{\text{MAD}}=\sqrt{\frac{\varphi }{2}}\approx 1.25$$

where $\varphi$ is the Gaussian distribution parameter. This ratio increases for distributions with fat tails.

Conclusion

Understanding the correct definitions and implications of standard deviation and correlation is crucial for accurate statistical analysis. Misconceptions can lead to significant errors in interpreting data and making decisions based on statistical measures.

MINI-LESSON 2: Fat Tails

Understanding Fat Tails in Probability

Welcome back to our series on statistics and probability, with a special focus on probability concepts. Today, we’ll explore the concept of fat tails, also known as thick tails. Fat tails are a crucial topic because many professionals, including statisticians, often misunderstand or overlook them, leading to significant errors in analysis.

What Are Fat Tails?

Fat tails refer to the phenomenon where extreme events (outliers) have a higher probability of occurring than what is predicted by standard statistical models. This contrasts with the usual assumption in normal distributions where extreme events are rare.

• Misconception: People often think fat tails mean that extreme events are more frequent and have a bigger impact. However, it’s not about frequency but about impact. Extreme events are less frequent but have disproportionately large effects when they do occur.

Definition: In a fat-tailed distribution, a small number of observations contribute disproportionately to the total statistical properties of the dataset.

Example of a Fat-Tailed Distribution

Consider a dataset where most values are zero except for one extreme value. For instance:

• Data: $\{0,0,\dots ,0,1,000,000\}$ (with 1,000,000 observations, where one value is 1,000,000)

Average Calculation:

$$\text{Mean}=\frac{1}{1,000,000}\left(\sum _{i=1}^{999,999}0+1,000,000\right)=\frac{1,000,000}{1,000,000}=1$$

The mean is 1, but most of the data is concentrated in the extreme value.

Impact: In real-world scenarios, this is similar to having billions of people with negligible wealth and a few billionaires whose wealth skews the average.

Understanding Fat Tails in Probability Distributions

Let’s compare the normal distribution (Gaussian distribution) with a fat-tailed distribution:

• Normal Distribution: For a normal distribution, about 68% of observations lie within one standard deviation ($\sigma$) from the mean, and 95% lie within two standard deviations.

• Fat-Tailed Distribution: In a fat-tailed distribution, extreme values (beyond one standard deviation) become less frequent, but those that do occur have a significant impact. This means:

  • Within One Sigma: Fewer observations fall within this range.
  • Beyond One Sigma: The probability of observing extreme values is low, but those extreme values have a large impact.

Misconceptions and Practical Implications

• Misconception: Professionals often overlook the effects of fat tails, assuming they can use standard statistical measures (like standard deviation) effectively. This misunderstanding leads to faulty conclusions.

• Practical Impact: Fat tails imply that many statistical models, including regression models, might not be appropriate. For example, the Gauss-Markov theorem (which underpins linear regression) assumes normally distributed errors. In the presence of fat tails, these assumptions are violated, leading to unreliable results.

Conclusion: Recognizing and understanding fat tails is crucial for accurate statistical analysis. Professionals should be aware that standard metrics and models might not apply in fat-tailed contexts and should seek alternative approaches for analysis.

MINI-LESSON 3: The Law of Large Numbers

Understanding the Law of Large Numbers

Welcome back to our series on statistics and probability. Today, we’ll delve into the Law of Large Numbers (LLN). The goal is to explain this fundamental concept clearly and concisely in under 10 minutes.

What is the Law of Large Numbers?

The Law of Large Numbers is a foundational principle in probability and statistics. It states that as the number of trials or observations increases, the sample mean (average) will converge to the expected value (the theoretical mean) of the distribution from which the samples are drawn.

Importance:

• Scientific Significance: LLN is essential for statistical analysis and modeling, providing a basis for making predictions and understanding variability.
• Philosophical Significance: It addresses the problem of induction, which deals with the reliability of inferences based on observed data.

Formal Definition

Let $X_1,X_2,\dots ,X_n$ be a sequence of independent and identically distributed (i.i.d.) random variables with a finite mean $\mu$. The Law of Large Numbers states that the sample average ${\bar{X}}_n$ converges to the true mean $\mu$ as $n$ approaches infinity:

$${\bar{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i$$ $$\underset{n\to \infty }{\lim }{\bar{X}}_n=\mu$$

Why is LLN Important?

1. Predictability: LLN allows us to predict that the average of a large number of observations will be close to the expected value, providing stability and predictability.
2. Portfolio Theory: In finance, LLN explains why a diversified portfolio tends to be more stable than individual stocks. The combined average performance of a large number of assets is less volatile than the performance of any single asset.

Example: Coin Tosses

Consider a simple example of tossing a fair coin:

• Single Toss: The outcome is either 0 (tails) or 1 (heads), with a probability $p=0.5$.

• Expectation: The expected value for each toss is $\mu =0.5$.

• Average of Tosses: If you toss the coin many times and calculate the average number of heads, this average will get closer to 0.5 as the number of tosses increases.

  Calculations:

  • For a small number of tosses (e.g., 2): $$\text{Possible Outcomes}=\{0,1,2\}$$
  • Standard Deviation: $$\text{SD}=\sqrt{p(1-p)}=\sqrt{0.5\times 0.5}=0.5$$
  • For a large number of tosses: $$\text{Standard Deviation}=\frac{\text{SD}}{\sqrt{n}}=\frac{0.5}{\sqrt{n}}$$

As $n$ increases, the standard deviation of the average decreases, meaning the average becomes more stable.

Fat Tails and Limitations

While the LLN works well for many distributions, it can be problematic in the presence of fat tails (heavy-tailed distributions). Fat-tailed distributions have extreme values that are more common than those predicted by normal distributions.

• Example: In a Pareto distribution (80/20 rule), a small number of observations account for most of the effect. For fat-tailed distributions:

  • Variance: Often undefined or very large, making standard deviation and mean less informative.

  • Sample Size Requirement: To achieve stability in fat-tailed distributions, you need a significantly larger sample size compared to normal distributions.

Comparing Distributions

Gaussian Distribution:

• Standard Deviation Drop: For a normal distribution, the standard deviation of the sample mean decreases as $\sqrt{n}$, making the sample mean more stable with more observations.

Pareto Distribution:

• Mean Deviation: In a Pareto distribution, the mean deviation does not decrease at the same rate. You may need an enormous number of observations (e.g., $10^{13}$) to achieve similar stability to that of a Gaussian distribution with fewer observations.

Summary

• The Law of Large Numbers assures that as we collect more data, the sample mean converges to the true mean, providing greater certainty.
• In practice, this means that aggregating large amounts of data can stabilize predictions and results.
• Fat-tailed distributions present challenges as they require much larger samples to achieve the same level of certainty and stability.

MINI-LESSON 4: The Central Limit Theorem

Understanding the Central Limit Theorem (CLT)

Welcome back! In this lesson, we’ll cover the Central Limit Theorem (CLT)—a cornerstone of statistical theory that simplifies the analysis of complex data. The CLT explains why many distributions approximate a normal (Gaussian) distribution, even if the underlying data does not.

What is the Central Limit Theorem?

The Central Limit Theorem states that when you take the sum (or average) of a large number of independent and identically distributed (i.i.d.) random variables, the distribution of the sum (or average) approaches a normal (Gaussian) distribution, regardless of the shape of the original distribution, provided that the original distribution has a finite variance.

Formally:

• If $X_1,X_2,\dots ,X_n$ are i.i.d. random variables with mean $\mu$ and variance ${\sigma }^2$,
• Then the distribution of the sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ approaches a normal distribution as $n$ becomes large.

Mathematically:

$${\bar{X}}_n\approx \mathcal{N}\left(\mu ,\frac{{\sigma }^2}{n}\right)$$

where $\mathcal{N}$ denotes the normal distribution.

Why is CLT Important?

1. Simplifies Analysis: The CLT allows us to use the properties of the normal distribution to make inferences about data, even if we don’t know the exact distribution of the data.
2. Statistical Methods: It underpins many statistical methods and tests, such as hypothesis testing and confidence intervals.
3. Predictability: It helps in making predictions and understanding the behavior of averages or sums.

Case Studies and Examples

1. Uniform Distribution

Let’s start with the simplest case—a uniform distribution:

• Uniform Distribution: All outcomes are equally likely in a given range. For instance, if we have a discrete uniform distribution between 0 and 10, each integer in this range is equally likely.

  • Example: Suppose we have 10 random variables $X_1,X_2,\dots ,X_{10}$ uniformly distributed between 0 and 10. When we sum these variables:

    • Single Observation: Flat distribution.
    • Sum of Two Observations: Results in a triangular distribution.
    • Sum of Five Observations: Approaches a normal distribution more closely.

  Visual: As you add more observations, the shape of the distribution of the sum starts to look more like a bell curve.

2. Binomial Distribution

Now consider a binomial distribution, which results from a series of Bernoulli trials (e.g., coin tosses):

• Bernoulli Trials: Outcomes are either 0 or 1 (e.g., heads or tails), with a probability $p$ for heads.

  • Example: If you sum 10 Bernoulli trials, the resulting distribution starts to resemble a normal distribution.

  Visual: Adding more trials refines the bell curve shape, demonstrating how the binomial distribution approximates a normal distribution as $n$ increases.

3. Pareto Distribution

Finally, let’s examine a Pareto distribution—a fat-tailed distribution often used to model wealth distribution:

• Pareto Distribution: Characterized by a heavy tail, meaning that extreme values (e.g., very high wealth) are more common.

  • Example: Summing a large number of Pareto-distributed variables:

    • Single Variable: Exhibits a heavy-tailed shape.
    • Sum of Variables: With a large number of variables, the distribution of the sum eventually approximates a normal distribution, but the convergence is slower compared to distributions with lighter tails.

  Visual: Even with a large sample size, it takes longer for the Pareto distribution to approximate a normal distribution compared to lighter-tailed distributions.

Summary

• Central Limit Theorem (CLT): When summing a large number of i.i.d. random variables, their sum (or average) tends to follow a normal distribution, provided the original distribution has finite variance.
• Applications: The CLT simplifies statistical analysis and allows the use of Gaussian distribution properties for inference.
• Fat-Tailed Distributions: The convergence to a normal distribution is slower with fat-tailed distributions, requiring larger sample sizes for accurate approximation.

MINI-LESSON 5: Correlation

Understanding Correlation: A Deep Dive

Welcome back! Today, we’re diving into the concept of correlation. While it’s a common term, its true meaning and implications are often misunderstood, particularly when it comes to non-linear relationships. Let’s break down what correlation is, how it’s calculated, and where its limitations lie.

1. What is Correlation?

Correlation measures the linear relationship between two variables, $X$ and $Y$. It quantifies how changes in one variable are associated with changes in another. Mathematically, the Pearson correlation coefficient $\rho$ between two variables $X$ and $Y$ is defined as:

$${\rho }_{XY}=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\frac{\mathbb{E}[(X-{\mu }_X)(Y-{\mu }_Y)]}{\sqrt{\mathbb{E}[(X-{\mu }_X{)}^2]}\cdot \sqrt{\mathbb{E}[(Y-{\mu }_Y{)}^2]}}$$

Where:

• $\mathbb{E}$ is the expectation operator.
• ${\mu }_X$ and ${\mu }_Y$ are the means of $X$ and $Y$, respectively.
• ${\sigma }_X$ and ${\sigma }_Y$ are the standard deviations of $X$ and $Y$.
• $\text{Cov}(X,Y)$ is the covariance of $X$ and $Y$.

2. Correlation vs. Dependence

A common misconception is that correlation fully captures the dependence between two variables. However, correlation only measures linear dependence. There are cases where two variables are highly dependent, but their correlation is zero. Consider the following example:

• Example: Non-linear Dependence
  • If $Y=X$ for $X\ge 0$ and $Y=-X$ for $X<0$, the correlation ${\rho }_{XY}=0$. However, $X$ and $Y$ are perfectly dependent in a non-linear way.

This shows that while correlation is useful in simple, linear models, it fails to capture more complex dependencies.

3. Where Does Correlation Work?

Correlation is effective in simple, linear models where $Y$ can be expressed as a linear function of $X$:

$$Y=\beta X+ϵ$$

Here, $\beta$ is a constant, and $ϵ$ is noise. If we normalize the variables (subtract the mean and divide by the standard deviation), the correlation $\rho$ corresponds to $\beta$.

However, this measure becomes less informative as the noise increases or the relationship between $X$ and $Y$ becomes non-linear.

4. Correlation is Not Additive

Another critical limitation of correlation is its non-additivity. Suppose you have two variables, $X$ and $Y$, with a correlation of 0.5 (${\rho }_{XY}=0.5$). If you break down the correlation within specific quadrants (e.g., when both $X$ and $Y$ are positive), the local correlations can be very different from the global correlation. This subadditivity can lead to misinterpretations.

• Example: Correlation in Quadrants
  • Imagine $X$ and $Y$ are both positive or both negative. The local correlation might be 0.26, but this does not simply add up to the global correlation of 0.5. This non-additivity is a significant limitation when using correlation in more complex scenarios.

5. Visualizing Correlation

To truly understand correlation, it’s often better to visualize the data. Let’s look at how correlation changes with different values:

• Zero Correlation ($\rho =0$): Points are scattered with no clear linear pattern.
• Moderate Correlation ($\rho =0.5$): A weak linear trend begins to appear.
• High Correlation ($\rho =0.8$): A strong linear pattern is visible, but still, it’s not perfect.
• Perfect Correlation ($\rho =1$): Points lie exactly on a straight line.

Interestingly, a correlation of 0.5 is closer to 0 than it is to 1, indicating that even moderate correlations don’t imply a strong linear relationship.

6. Beyond Correlation: Mutual Information

For non-linear relationships, mutual information is a more powerful metric. Unlike correlation, mutual information can capture all types of dependencies, both linear and non-linear.

• Mutual Information Definition:

  • Mutual information measures the amount of information obtained about one random variable through another. It’s based on the concept of entropy and is calculated as:
  $$I(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)\log \left(\frac{p(x,y)}{p(x)p(y)}\right)$$

  Where $p(x,y)$ is the joint probability distribution of $X$ and $Y$, and $p(x)$, $p(y)$ are the marginal distributions.

Mutual information is particularly useful in fields like genetics, where complex, non-linear relationships are common.

7. Key Takeaways

• Correlation measures linear relationships but is limited in non-linear contexts.
• Mutual Information is a better measure for capturing all dependencies, especially in non-linear scenarios.
• Always visualize data before relying on correlation to understand relationships.
• Be cautious with studies that use correlation, especially in social sciences, as they may overlook non-linearities.

Conclusion

Correlation is a useful tool, but it has significant limitations, especially outside of simple, linear models. When dealing with real-world data, especially in non-linear systems, relying solely on correlation can be misleading. Always consider more robust measures like mutual information and, whenever possible, visualize your data to truly understand the relationships at play.

MINI-LESSON 6: Fooled by Metrics (Correlation)

This lecture dives into two major points regarding how people are often misled by metrics:

1. Metrics as Random Variables

• Metrics Aren’t Deterministic: Metrics like correlation aren’t fixed values but random variables that can vary with each sample. This randomness means that what you observe might not represent the true relationship between variables.
• Example with Correlation: Consider two independent, uncorrelated variables $X$ and $Y$. If you repeatedly sample these and calculate the Pearson correlation, you’ll find it varies widely from sample to sample, even though the true correlation is zero. This variability can lead to false confidence in a relationship that doesn’t actually exist.

2. Gaming Metrics

• Exploiting the Upper Bound: Researchers might exploit the randomness of metrics by cherry-picking the highest correlation out of many tests. This is especially problematic when dealing with a large number of variables (high dimensionality), where the chance of finding a seemingly significant correlation purely by chance increases.
• Example with Spurious Correlations: The lecture provides humorous but serious examples, like correlating the number of films Nicolas Cage appeared in with the number of people who drowned in pools. Such spurious correlations arise when researchers, knowingly or unknowingly, pick the highest correlations from a large set of data, mistaking noise for meaningful relationships.

Real-World Implications

• Dimensionality Problem: As the number of variables $P$ increases, the number of possible correlations grows quadratically (approximately $P^2/2$). This explosion in possible correlations increases the likelihood of finding spurious relationships unless the sample size $N$ is sufficiently large to mitigate this effect.
• Misleading Research: Fields like psychology or political science, which often rely on observational data, can produce misleading results when they fail to account for the randomness and spuriousness inherent in their metrics. The lecture suggests that many studies in these fields might be based on shaky statistical grounds, leading to “garbage” results.

Conclusion

• Critical Approach to Metrics: Always be skeptical of metrics, especially correlations, unless you’ve seen the data or the graph. Remember that correlation does not imply causation, and often, correlation does not even imply true correlation due to the issues of randomness and gaming.
• Consider Dimensionality: Be aware of the dimensionality problem—having too many variables with insufficient data can lead to an abundance of misleading correlations. Proper statistical techniques are needed to manage this, but even then, skepticism and careful analysis are essential.

MINI-LESSON 7: P-Values and P-Value Hacking

This lecture tackles the concept of the p-value, highlighting its limitations and the potential for misuse in research:

1. The Problem with P-Values

• P-Value as a Misleading Metric: The p-value is widely used in statistics to measure the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. However, the lecture argues that the p-value is not a solid concept with a strong probabilistic basis, making it a problematic tool.
• Stochastic Nature of P-Values: P-values are random variables, which means they can vary significantly with different samples, even when the underlying conditions are the same. This variability implies that a single p-value from one experiment may not reliably indicate the true probability of observing the result by chance.

2. The Illusion of Determinism

• Misunderstanding Sample Size: When calculating a statistic like the mean, researchers often assume that the sample size $n$ accounts for all uncertainties, especially when using metrics like the z-score. However, this isn’t true. The scaling by the square root of $n$ doesn’t eliminate all the subtleties, and the resulting p-value can still be highly variable.
• False Confidence: The lecture emphasizes that p-values can create an illusion of certainty. For instance, even if the true p-value is 0.11 (indicating a weak result), running multiple experiments might yield a p-value of 0.01 by chance, leading researchers to falsely believe they have found something significant.

3. Gaming P-Values

• Exploiting the Upper Bound: Just as with correlations, researchers can game p-values by conducting multiple experiments and selecting the smallest p-value (i.e., the one that appears most significant). This practice leads to inflated claims of significance, as the distribution of the maximum p-value is skewed lower than that of the true p-value.
• Misinterpretation: A p-value of 0.01, obtained through multiple trials, isn’t truly 0.01 in terms of statistical significance—it’s just a product of selective reporting. To avoid this issue, the p-value should be much smaller than the standard threshold (e.g., 0.05) to ensure it hasn’t been gamed.

4. Implications for Research

• Psychology and Small Samples: The lecture points out that fields like psychology often rely on small sample sizes, making p-values particularly unreliable. The use of p-values in such contexts can give a false sense of precision and significance, leading to flawed conclusions.
• Need for Better Practices: Researchers should be cautious when using p-values and consider alternative methods or metrics that provide a more reliable measure of significance. The lecture suggests that the problem is especially severe in fields with small sample sizes, where the variability of p-values is more pronounced.

Conclusion

• Skepticism Towards P-Values: The lecture advises being wary of p-values, especially in studies with small sample sizes or when multiple tests are conducted. The stochastic nature of p-values means they can be easily misinterpreted or manipulated, leading to unreliable research findings.
• Broader Issues in Research: The discussion ties back to earlier points about the stochastic nature of metrics in general and the risks of “hacking” these metrics to produce seemingly significant results. Researchers need to be aware of these pitfalls and strive for more robust statistical practices.

MINI-LESSON 8: Power Laws

The lecture provides an overview of power laws, the Pareto distribution, and their significance in understanding various statistical distributions. Here’s a summary of the key points:

1. Introduction to Power Laws and Pareto Principle

• Pareto Principle (80/20 Rule): The idea that 80% of effects come from 20% of causes, originally observed by Vilfredo Pareto in land ownership.
• Fractality: The 80/20 distribution can be recursively applied, leading to an even more skewed distribution (e.g., 1% owning 50% of the land).

2. Classes of Distributions

• Gaussian Distribution:

  • Common in nature due to the Central Limit Theorem.
  • Rapid decline in tail probabilities (e.g., probabilities of exceeding 3, 4, 5 sigmas).
  • The ratio of tail probabilities increases as you move further out in the distribution.

• Sub-Exponential Distribution:

  • Includes distributions like log-normal, exponential, Laplace, and gamma.
  • These distributions have thicker tails than Gaussian but are not power laws.

• Power Law Distributions:

  • Includes Pareto, Student’s t, Cauchy, and stable distributions.
  • Characterized by a constant ratio between different levels of magnitude (e.g., doubling income levels).
  • No finite variance or mean in extreme cases, yet they are still understandable and usable.

3. Gaussian vs. Power Law Characteristics

• Gaussian:

  • Life expectancy shrinks as age increases (e.g., after 80 years, the expected additional lifespan decreases).
  • Tail probabilities decline rapidly.

• Power Law:

  • Invariance in ratios (e.g., doubling any quantity keeps the ratio constant).
  • The expected value conditional on exceeding a threshold remains a multiple of that threshold.

4. Mathematical Representation of Power Laws

• Formula: The probability of exceeding a value $x$ in a power law is proportional to $x^{-\alpha }$.
  • Log-log plots of these distributions yield a straight line with a slope of $-\alpha$.
  • The lower the $\alpha$, the fatter the tail of the distribution.
  • Special cases: $\alpha =1$ (no mean), $\alpha =2$ (no variance).

5. Implications for Statistics and Real-World Applications

• Traditional statistical measures like mean, variance, and kurtosis lose their significance in power law distributions.
• These concepts are critical in understanding phenomena such as wealth distribution, market movements, and natural events like earthquakes.

---

Here’s a summary of the key points:

1. Wars and Power Laws

• The speaker critiques Steven Pinker’s assertion that violence has declined based on a small sample of historical data. He argues that wars follow a power law distribution with a very low alpha (around 0.5), indicating extremely fat tails, which means the probability of very large wars is significant.
• The claim that the world is safer because there hasn’t been a large war since WWII is criticized for not considering the long inter-arrival times of such wars, which could average around 80 years. Statistically, one would need to wait much longer (potentially three times this period) to make a confident statement about a decline in war.
• Historical data on wars is often unreliable due to the inconsistencies in records, where the number of casualties can vary widely depending on the source. The speaker suggests a method to account for this by using a range of possible values (low and high estimates) to create multiple possible histories and test hypotheses.

2. Pandemics

• Similar to wars, pandemics also follow a power law distribution with a low alpha, making them a significant existential risk. The speaker mentions that by using bootstrapping and other statistical methods, the same conclusion about the dangerous nature of pandemics is reached.

3. Understanding Infinite Mean

• The speaker explains what it means for a distribution to have an “infinite mean.” In essence, with power law distributions, especially those with an alpha less than 1, the mean can be very unstable, varying widely depending on the sample, and never converging to a single value.

4. Power Laws and Constraints

• Power laws arise naturally in many systems, but constraints (like physical or biological limits) can prevent them from fully manifesting. For instance, human height doesn’t follow a power law due to biological constraints, whereas something like wealth distribution, with fewer constraints, can follow a power law more closely.

5. Preferential Attachment and Social Contagion

• The speaker discusses mechanisms like the Matthew Effect (rich-get-richer phenomenon) and preferential attachment, where initial advantages snowball, leading to power law distributions in things like wealth, popularity, and other social phenomena.

6. Entropy and Distribution

• There’s a brief mention of how constraints like energy can lead to different types of distributions, with power laws emerging when such constraints are relaxed.

The overall message is that power laws are pervasive in many natural and social phenomena, but one must be careful in interpreting them, especially when dealing with historical data or making predictions about rare, high-impact events like wars and pandemics.

---

Understanding Fat Tails and Power Laws

1. Introduction to Fat Tails and Their Importance

When we say a phenomenon has “fat tails,” it means that rare events play a significant role in determining the overall behavior of the system. In statistical distributions with fat tails, the extreme values (rare events) contribute most of the distribution’s properties. This is different from distributions like the normal distribution, where most of the information lies in the center (or “body”) of the distribution.

2. Power Laws and the Role of the Alpha Parameter

A common way to describe fat-tailed distributions is through power laws. The power law is often represented as:

$P(X>x)\propto x^{-\alpha }$

Here, $\alpha$ is a crucial parameter:

• The smaller the $\alpha$, the fatter the tail.
• The fatter the tail, the more significant the contribution of rare events.

3. Survival Function and Its Importance

The survival function (also known as the tail distribution function) tells us the probability that a value exceeds a certain threshold $K$. For a power law distribution, the survival function $S(x)$ is given by:

$S(x)={\left(\frac{K}{x}\right)}^{\alpha }$

Where:

• $K$ is the threshold value.
• $x$ is the value above the threshold.
• $\alpha$ is the power law exponent.

This function helps us understand how likely it is to observe extreme events.

4. Inverse Survival Function

One powerful property of power laws is that we can invert the survival function to find thresholds for given probabilities. The inverse survival function $K(p)$ gives us the value of $x$ such that the probability of exceeding it is $p$:

$K(p)={\left(\frac{1}{p}\right)}^{1/\alpha }$

5. Practical Application: The 80/20 Rule

A well-known principle, the 80/20 rule (Pareto Principle), states that 20% of the causes lead to 80% of the effects. We can model this with a power law distribution.

For example:

• If $\alpha =1.5$, then approximately 20% of the population holds 80% of the wealth.
• Similarly, a much smaller fraction, say 0.1%, could hold a significantly larger portion of the wealth.

Using the survival function and its inverse, we can compute specific values. For instance, let’s compute for $\alpha =1.5$:

• $P(X>x)$ where $x=1\%$ of the population.
• If $\alpha =1.5$, 1% of the population might control about 55% of the wealth.

6. Financial Markets Example

In financial markets, fat tails are particularly important. Consider an options portfolio:

• 1% of trading days might contribute to 95% of the portfolio’s profit/loss (P&L).
• Even 0.1% of the days could account for 90% of the P&L.

This illustrates that most of the financial outcomes are driven by a small number of extreme events, reinforcing the importance of understanding fat tails.

7. Conclusion

Fat-tailed distributions and power laws highlight the importance of rare events in various phenomena. By using the survival function and its inverse, we can quantify these effects and make predictions about the contribution of different segments of a distribution.

Key Equations Summary

1. Power Law Distribution: $P(X>x)\propto x^{-\alpha }$

2. Survival Function: $S(x)={\left(\frac{K}{x}\right)}^{\alpha }$

3. Inverse Survival Function: $K(p)={\left(\frac{1}{p}\right)}^{1/\alpha }$

These concepts are critical in fields such as finance, economics, and risk management, where understanding the impact of rare events can be the difference between success and failure.

MINI-LESSON 9: Evidence Based Science & Mistakes in Particularizing the General

1. Evidence-Based Science: Definition and Importance

Evidence-based science relies on empirical data and rigorous testing to draw conclusions. The idea is to avoid biases and ensure that the treatments or interventions being used are genuinely effective. It’s about using data from well-designed studies to guide decision-making.

Example: Randomized Control Trials (RCTs)

• RCTs are the gold standard in evidence-based medicine. They involve randomly assigning participants into two groups:

  • Group A: Receives the treatment (e.g., hydroxychloroquine for COVID-19).
  • Group B: Receives a placebo or standard treatment.

  The outcomes are then compared statistically to determine if the treatment is effective.

2. The Law of Large Numbers and Its Misinterpretations

The Law of Large Numbers states that as the number of observations (n) increases, the average of the observed values (mean) converges to the expected value. However, this principle is often misunderstood or misapplied.

A. Understanding the Flow of Information

• Information about the average (Aₙ) does not necessarily apply to each individual observation (X₁, X₂, …, Xₙ).
• For example, the “average human” might have one breast and one testicle, but this doesn’t reflect the reality of any individual person.

B. Implications in Medicine and Risk Management

• Clinical Knowledge (n=1): Focuses on individual cases.
• Statistical Knowledge (n=100): Focuses on group data and averages.
• Risk Management (n=very large): Deals with tail risks, where rare but extreme events dominate.

3. Pseudo-Empiricism and the Danger of Misleading Generalizations

Pseudo-empiricism occurs when observations that are not statistically significant are used to make broad generalizations. This is a critical mistake in both scientific and public discourse.

Example: Misinterpreting COVID-19 Data

• Consider the claim that hydroxychloroquine (HCQ) is effective because one hospital reports a lower death rate. This doesn’t account for differences in population demographics, such as age, between hospitals. Without randomization, these observations are not scientifically valid.

4. The Misuse of Averages and Group Characteristics

A common mistake is to apply group averages to individuals, which leads to incorrect conclusions.

A. Racism and Stereotyping

• Applying the average characteristics of a group (e.g., a nationality’s supposed weakness in math) to an individual from that group is flawed reasoning.

B. Risk Analysis in Pandemics

• The risk of dying from COVID-19 cannot be directly compared to the risk of dying in a car accident because the dynamics of the two risks are different.
  • Car Accident: The risk is individual and stable.
  • COVID-19: The risk is collective and can increase exponentially with the spread.

5. Understanding Extremistan vs. Mediocristan

In risk management, it’s crucial to distinguish between different types of randomness:

• Mediocristan: Events with low variability, where averages are meaningful.
• Extremistan: Events with high variability, where extreme events (tail risks) dominate outcomes.

A. Pandemics as Extremistan Events

• Deaths from pandemics follow a fat-tailed distribution, meaning a small number of events (e.g., a super-spreader event) can have a massive impact.

6. Conclusion: The Dangers of Generalizing and Particularizing

The key takeaway is to avoid the following errors:

1. Generalizing from Small Samples: Drawing broad conclusions from a limited number of observations.
2. Particularizing from General Averages: Applying the characteristics of a group average to individual cases.

Final Thoughts

To navigate complex phenomena like pandemics or economic crises, it’s crucial to understand the limitations of statistical averages and the importance of rigorous empirical testing. Misinterpreting data, especially in fields like medicine and risk management, can lead to catastrophic errors.

MINI-LESSON 10: Simpson’s Paradox & Its Exploitation

Understanding Simpson’s Paradox: A Guide

Hello again, friends! Today, I want to discuss an important concept in statistics known as Simpson’s Paradox. This paradox is often misused, particularly by groups like anti-vaxxers and conspiracy theorists. It’s essential to understand how this paradox works to avoid being misled by faulty arguments.

What is Simpson’s Paradox?

Simpson’s Paradox occurs when a trend observed in several different groups of data disappears or reverses when these groups are combined. This can lead to misleading conclusions if one doesn’t account for the underlying group differences.

Example: Imagine you’re looking at two universities (University A and University B) and their performance across different departments. University A might outperform University B in every individual department, yet when you combine all departments, University B could appear to outperform University A. This reversal happens due to differences in group sizes or distributions, not because of the actual performance.

The Anti-Vax Argument Misusing Simpson’s Paradox

A recent example of Simpson’s Paradox being misused is in arguments against vaccines. Specifically, people like Alex Berenson have claimed that in the age group of 10 to 59, vaccinated individuals are dying at twice the rate of unvaccinated individuals. On the surface, this might seem to suggest that being vaccinated increases the risk of death.

But let’s break it down:

• Age Distribution: The age group 10 to 59 is broad. Older individuals (e.g., 50-59) naturally have a higher mortality rate than younger individuals (e.g., 10-20). If most older people are vaccinated and most younger people are unvaccinated, this can skew the overall mortality statistics.

• Group Segmentation: If you segment the data by smaller age brackets (e.g., 10-20, 20-30, etc.), you might find that within each bracket, vaccinated individuals have a lower mortality rate than unvaccinated ones.

• Simpson’s Paradox in Action: The paradox appears because the overall data combines different age groups with different vaccination and mortality rates. The over-representation of older, vaccinated individuals (who naturally have a higher mortality rate) in the overall analysis can make it seem like vaccination increases mortality, which is misleading.

Visualizing the Concept

To illustrate, let’s break this down with some simple math.

Suppose:

• Age Group 50-59:

  • Vaccinated: 20,000 people with 200 deaths (1% mortality).
  • Unvaccinated: 1,000 people with 30 deaths (3% mortality).

• Age Group 10-20:

  • Vaccinated: 1,000 people with 1 death (0.1% mortality).
  • Unvaccinated: 10,000 people with 2 deaths (0.02% mortality).

If you combine these groups without considering the age factor, you might get a misleading overall mortality rate:

• Total Vaccinated: 21,000 people with 201 deaths.
• Total Unvaccinated: 11,000 people with 32 deaths.

Mortality Rate:

• Vaccinated: $\frac{201}{21,000}\approx 0.96\%$
• Unvaccinated: $\frac{32}{11,000}\approx 0.29\%$

This combined data wrongly suggests that vaccinated individuals have a higher mortality rate, ignoring the fact that the vaccinated group has more older individuals who are at a higher risk of death due to age, not the vaccine.

Conclusion

Simpson’s Paradox teaches us to be careful when interpreting statistics, especially when groups of different sizes or characteristics are involved. It’s crucial to consider the context and segment data appropriately to avoid misleading conclusions.

MINI-LESSON 11: Path dependence in 2 ½ minutes

Understanding Path Dependence

Path dependence is a concept that illustrates how the sequence of events or decisions affects the final outcome. To make this clearer, let’s break it down with your example.

The Scenario

You have a process that starts at 100, and after 10 steps, you end up back at 100. The steps consist of either adding (+1) or subtracting (-1). You can:

1. Add (+1) five times and Subtract (-1) five times in different sequences.

Three Different Paths

1. Path 1: Alternating Steps

   • You alternate between +1 and -1.
   • Example Sequence: +1, -1, +1, -1, +1, -1, +1, -1, +1, -1.
   • Result: The value oscillates between 99 and 101, ending at 100.
   • Stability: Appears stable as it fluctuates minimally around the starting point.

2. Path 2: First All Up, Then All Down

   • First, you add +1 five times, then subtract -1 five times.
   • Example Sequence: +1, +1, +1, +1, +1, -1, -1, -1, -1, -1.
   • Result: The value first rises to 105 and then drops back down to 100.
   • Stability: This path creates a peak and then a drop, showing significant variation over time.

3. Path 3: First All Down, Then All Up

   • First, you subtract -1 five times, then add +1 five times.
   • Example Sequence: -1, -1, -1, -1, -1, +1, +1, +1, +1, +1.
   • Result: The value first drops to 95 and then rises back to 100.
   • Stability: This path creates a dip followed by a rise, also showing significant variation over time.

Key Observations

• Mean and Variance: All three paths have the same mean (100) and the same variance (because the number of steps up and down is equal in each case).
• Temporal Structure: Despite having the same mean and variance, the paths are very different when you look at how they evolve over time.
  • Path 1: Minimal variation, stable.
  • Path 2: Sharp rise followed by a sharp drop.
  • Path 3: Sharp drop followed by a sharp rise.

Path Dependence in Action

• Path 1 suggests stability, as the value fluctuates around 100.
• Path 2 and Path 3 show instability, with significant deviations from the starting point before returning to 100.

The concept of path dependence shows that the sequence in which events occur can matter greatly, even if traditional statistical measures like mean and variance remain unchanged. This is why it’s crucial to consider the temporal structure of processes, not just their average outcomes.

Conclusion: Path dependence highlights the importance of the sequence of events. Two processes with identical overall outcomes can behave very differently depending on the order of steps taken. This is a crucial consideration in fields ranging from finance (like stock prices) to physics (like particle motion).

MINI-LESSON 11b: Drawdowns and Logs

Let’s break down the concept of maximum drawdown and why logarithms are used in finance, making it easier to understand the reasoning behind these approaches.

What is Maximum Drawdown?

Maximum Drawdown (MDD) is a measure of the largest decline from a peak to a trough in the value of an investment over a specific period. It helps assess the riskiness of the investment by showing the worst-case scenario in terms of loss.

Steps to Calculate Maximum Drawdown:

1. Identify Peaks and Troughs:

   • Track the value of an investment over time.
   • Note every peak (local maximum) and the lowest subsequent point (trough).

2. Calculate Drawdowns:

   • For each peak, calculate the drop to the subsequent trough.
   • This drop is called a drawdown.

3. Find Maximum Drawdown:

   • Among all drawdowns, the maximum drawdown is the largest one.

Visual Representation:

If you plot the value of a security over time, the MDD is the largest vertical drop from a peak to the lowest point following that peak.

Why Not Use Percentages?

Using percentages for drawdowns can be misleading. Here’s why:

• Non-Additivity of Percentages:

  • Suppose a stock price rises from $100to$200 (a 100% gain) and then falls back to $100 (a 50% loss). The sum of these percentages (100% - 50%) doesn’t correctly represent the return, as the stock is back to its original value, meaning no net return.

• Symmetry Issue:

  • A 50% loss followed by a 50% gain doesn’t bring you back to the original value. For example, if you drop from $100to$50 (a 50% loss) and then gain 50%, you only go back to $75,not$100.

Why Use Logarithms?

Logarithms address the problems with percentage calculations:

1. Continuous Compounding:

   • When you model returns over time with continuous compounding, the formula becomes: $$S(t)=S(0)\cdot e^{rt}$$ where $S(t)$ is the value at time $t$, $S(0)$ is the initial value, $r$ is the continuous return, and $e$ is the base of the natural logarithm.

2. Additivity:

   • Taking the logarithm of returns ensures that they are additive. If $S(t)=S(0)\cdot e^{rt}$, then: $$\log \left(\frac{S(t)}{S(0)}\right)=rt$$ This form allows for straightforward summation and comparison of returns over different periods.

Application to Maximum Drawdown

When calculating maximum drawdown using logarithms:

• Logarithmic Drawdown: $$\text{Logarithmic Drawdown}=\log \left(\frac{\text{Peak Value}}{\text{Trough Value}}\right)$$
• This ensures that the drawdowns are comparable and correctly represent the risk over different periods.

Summary

1. Maximum Drawdown measures the largest drop from a peak to a trough in an investment’s value, reflecting its riskiness.
2. Percentages are misleading due to their non-additivity, especially when dealing with large returns and losses.
3. Logarithms provide a better measure because they ensure returns are additive, allowing for more accurate risk assessment.

Using these concepts, you can better understand and compare the riskiness of different investments over various periods.

MINI LECTURE 12: How to Look at the Risks of Covid Vaccines

Let’s break down the key points from your discussion about vaccines and the risk assessment around them, using a more structured approach to make it easier to understand.

1. Understanding the Risk of Vaccines

• Purpose of Vaccines: Vaccines are not taken for entertainment or mood enhancement; they are a preventive measure against serious diseases like COVID-19, which is lethal.
• Initial Skepticism: Initially, there might be skepticism about vaccines due to uncertainties, but over time, the effectiveness in preventing infections and saving lives becomes clearer.

2. Time vs. Sample Size in Risk Assessment

• Traditional View of Risk Over Time:
  • People often think that if a disease takes 12 or 20 years to develop (e.g., cancer from exposure to radiation in Hiroshima), the risk is only relevant after that time.
• Large Sample Size Consideration:
  • When you have a large population (like 8 billion people), the distribution of events (e.g., adverse effects from vaccines) spreads out, and you might see some effects earlier. This is because with more people, rare events can occur more frequently.

3. Distribution of Adverse Events

• Distribution Shapes:
  • The occurrence of adverse events can follow different distributions. These distributions can be front-loaded (more events early on), bell-shaped (events peak in the middle), or have long tails (events spread out over time).
• Time-Trade-Off:
  • With a large enough sample size, you don’t need to wait the full 12 or 20 years to see patterns. If there were significant risks associated with vaccines, they would likely appear within the first few years.

4. Interpreting the Data

• Current Observations:
  • So far, the data doesn’t show significant long-term risks associated with vaccines. While conspiracy theories might claim that adverse effects are hidden, this is unlikely because it’s hard to hide such information on a large scale.
• Misattribution of Cause and Effect:
  • People might wrongly attribute an unrelated event (e.g., someone dying after eating an ice cream cone) to the vaccine simply because the vaccine was administered beforehand.

5. Conclusion: Favoring Vaccination

• Risk Distribution:
  • The risk distribution of vaccines seems to have a very thin left tail, meaning severe adverse events are rare.
• COVID-19 vs. Vaccine Risks:
  • Given that COVID-19 is a real and significant threat, the benefits of vaccination in saving lives outweigh the minimal risks.

Summary

• Vaccines are essential in combating lethal diseases like COVID-19.
• Large sample sizes allow us to see the effects of vaccines more clearly and earlier.
• So far, no significant long-term risks have been observed with vaccines, making them a crucial tool in public health.

This structured breakdown should help you understand the reasoning behind supporting vaccination, focusing on the interplay between time, sample size, and the distribution of risks.

MINI LECTURE 13: Claims that Violence Has Dropped Are Not Statistical

Introduction

Hello again, and thank you for joining us. Today, we’ll explore common errors people make when dealing with variables from Extremistan and apply standard statistical techniques to address these issues.

Key Concepts

1. Extremistan vs. Mediocristan

   • Mediocristan: Characterized by distributions with thin tails, where most variables are close to the mean, and the Central Limit Theorem applies. For example, heights of people or IQ scores follow this type of distribution.
   • Extremistan: Characterized by distributions with fat tails, where extreme values can be very large, and the Central Limit Theorem does not apply well. Examples include wealth distribution or catastrophic events.

2. Statistical Analysis of Violence

   • Stephen Pinker’s Theory: Pinker argues that violence has decreased over time. However, establishing this claim scientifically is challenging due to the nature of the data.
   • Statistical Challenges: In Extremistan, the Law of Large Numbers and Central Limit Theorem are less effective. This makes it difficult to draw meaningful conclusions from historical data on violence.

Statistical Methods for Analyzing Violence

1. Event Analysis

   • We categorize events based on their magnitude, such as those causing more than 20 million or 50 million casualties.

   • Mean Inter-Arrival Time: For events causing 20 million casualties, the mean time between such events is about 50 years. For those causing 50 million casualties, it is about 70 years.

   • Exponential Distribution: The time between extreme events follows an exponential distribution, and statistical claims about their frequency require analyzing data over a longer time period (e.g., a century or more).

2. Quantitative Historiography

   • Synthetic Histories: We create synthetic historical data by simulating 100,000 different scenarios to account for variability and estimate parameters robustly.
   • Estimates of Historical Events: Estimates of casualties from historical events, such as the An Lushan Rebellion or Mongolian invasions, may be revised as new data or methods become available.

3. Handling Errors and Reporting

   • Historical Estimates: Historical estimates are prone to errors due to biases or incomplete data. For example, estimates of casualties during the Syrian conflict under Assad have inflated over time.
   • Robust Analysis: To ensure robustness, we compare estimates from different sources and adjust for potential biases.

Technical Considerations

1. Power Law Distribution

   • In Extremistan, events may follow a power law distribution, where the probability of extreme events decreases polynomially. However, in practice, data often do not fit a perfect power law.
   • Log Transformation: To handle data that do not strictly follow a power law, we apply log transformations to approximate power law behavior.

2. Applications

   • The techniques used for analyzing historical violence also apply to financial risks, pandemics, and other domains characterized by fat-tailed distributions.
   • Comparison: In terms of fat-tailed behavior:
     • Pandemics have more extreme events than wars.
     • Financial derivatives exhibit fat tails, though not as pronounced as pandemics.

Conclusion

In summary, analyzing violence and other extreme events requires careful statistical handling, especially in Extremistan where traditional methods may not apply. By using synthetic data, adjusting for historical inaccuracies, and employing advanced statistical techniques, we can make more robust claims about the nature of extreme events.

Thank you for your attention, and have a great day!

MINI LECTURE 13b: How to fix the problem of power laws with compact support

Introduction

In our work on violence and related topics, Professor Cherilo and I developed a transformation method to handle power laws that may not fit the strict definition. This method allows us to work with distributions that resemble power laws but have boundaries or other deviations.

Power Laws and Their Limits

A true power law has the form: $P(x)\propto x^{-\alpha }$ where $\alpha$ is the exponent. However, real-world data may not always fit this ideal. For example, a distribution with lower bound $l$ and upper bound $h$ might have a power-law-like tail but is not strictly a power law.

The Transformation Function

To address this, we use a transformation function $y=f(x)$ to map the original data $x$ into a new variable $y$ with different support: $f(x)=l-(h-l)\frac{\log (\log (h-x))}{\log (h-l)}$ where:

• $l$ is the lower bound.
• $h$ is the upper bound.
• $\log$ denotes the logarithm function, and it can use any base.

Example with Population Data

Consider the population of the world, which is approximately 7 billion. If we apply the transformation:

• For values up to about 3 billion, the function $f(x)$ will approximate the population distribution.
• As $x$ approaches $h$, the function $f(x)$ approaches infinity.

Relation to Log-Normal Distribution

The approach is similar to the log-normal distribution used in finance:

• Stock Price Transformation: Logarithmic returns of stock prices are often modeled using a normal distribution.

For a random variable $x$ following a normal distribution $N(\mu ,{\sigma }^2)$:

• The log return is often used to model $x$, which also follows a normal distribution but with different parameters.

Parameter Interpretation

For the transformed variable $y$:

• The mean ${\mu }_y$ of $y$ is given by: ${\mu }_y=e^{\mu +\frac{1}{2}{\sigma }^2}$
• The variance ${\sigma }_y^2$ is an exponential function of ${\sigma }^2$ with some shift.

Fitting Power Laws

By fitting the power law to the transformed variable $y$, we obtain a model that approximates the power law more accurately for large values:

• The analytics performed on $y$ are similar to those on $x$, but with adjustments for the transformed support.

Summary

The dual distribution method allows us to handle distributions that approximate power laws but have finite bounds. The transformation helps map one distribution to another, making it possible to analyze and compare them effectively. The conditions for this transformation to work were thoroughly examined to ensure accuracy.

MINI LECTURE 14: Fragility, Convexity, and Antifragility

Distinction Between X and F(X)

• X: Represents a random variable or an event (e.g., rain, a stock market fluctuation).
• F(X): Represents the outcome or response to that event (e.g., income for an Uber driver, profit/loss in trading).
• Many people mistakenly focus on predicting X (the event) instead of F(X) (the response to the event), which can lead to errors in judgment.

Non-Linearity and Fragility

• Fragility is characterized by a non-linear, typically concave response to stress or shocks. For example, a fragile object (like a coffee cup) doesn’t break gradually but rather catastrophically after a certain threshold of stress.
• Example: If you drop a cup from increasing heights, it remains intact up to a point but eventually shatters completely. The damage is not proportional to the height but accelerates past a certain threshold.
• This non-linearity means that the average impact of small shocks doesn’t necessarily predict the impact of a larger shock.

The Concept of Anti-Fragility

• Anti-fragile systems benefit from variability and stress up to a certain point. They have a convex response to stress, where small shocks can improve the system’s resilience or condition.
• Example: Human bones become stronger when subjected to moderate stress (like exercise), but too much stress (e.g., a high fall) can cause harm.

Practical Applications

• Medicine: A patient might survive better with varying doses of a treatment rather than a constant dose, due to the non-linear response of the body.
• Finance: A financial institution might seem stable under normal conditions but collapse under extreme conditions because of its exposure to risks that accelerate in the tails of the distribution.
• Supply Chains: A supply chain designed to handle average demand might fail when faced with a sudden spike in demand, highlighting the need for redundancy and flexibility in systems.

Key Takeaways

• Convexity vs. Concavity: Fragile systems have a concave response to stress, where harm accelerates rapidly after a certain point. Anti-fragile systems have a convex response, where they benefit from small amounts of stress.
• Survivorship Bias: Systems that survive over time tend to be non-linear, as linear systems (those without resilience to shocks) would have already failed.
• Redundancy and Resilience: Systems need to be designed with a buffer (redundancy) to handle unexpected shocks, rather than being optimized only for average conditions.

This talk emphasizes understanding and leveraging non-linear responses in various systems to build resilience and avoid catastrophic failures. The concept of anti-fragility is about designing systems that thrive in the face of volatility, rather than merely surviving or breaking under stress.

MINI LECTURE 15: Conditional vs. unconditional correlation: twin studies overestimate heredity

The text you’ve shared covers a broad and complex critique of heredity studies, especially in relation to IQ and environmental factors. The main points seem to be:

1. Overestimation of Heredity: The argument is that studies focusing on heredity often overestimate its role by not properly accounting for the environment. The speaker criticizes the reliance on correlations in these studies, particularly when dealing with non-linear environments where correlations may become uninformative.

2. Critique of IQ as a Measure: IQ is questioned as a reliable predictor of real-world outcomes. The speaker points out that while IQ may have some predictive power, the methods used to measure this power (e.g., correlations) are often flawed, especially when environmental factors are not properly considered.

3. Issues with Correlation: There’s a strong emphasis on the limitations of using correlation as a metric in non-linear and complex environments. The speaker suggests that relying solely on correlation can lead to misleading conclusions, especially when environmental variability is not accounted for.

4. Twin Studies and Falconer’s Formula: The speaker criticizes the methodology behind twin studies, particularly the use of Falconer’s formula, which estimates heritability based on the difference in correlations between identical and fraternal twins. The critique suggests that this method oversimplifies the complex interactions between genetics and environment.

5. Environmental Influence: The discussion highlights the importance of considering different environments and their impact on heritability estimates. The speaker suggests that varying environments can lead to vastly different outcomes, which are often not adequately captured by traditional heritability studies.

6. Critique of the Scientific Community: The speaker expresses frustration with what they see as intellectual dishonesty or ignorance within certain segments of the scientific community, particularly those focused on heredity and genetics. They argue that many researchers in this field fail to grasp the complexities of correlation and environment.

The overarching message seems to be a call for more sophisticated and nuanced approaches to understanding heredity, particularly by integrating better models of environmental influence and moving beyond simple correlation-based methods.

MINI LECTURE 16: On the Absurdity of Standard Genetics Reports

Understanding Genetic Ancestry: The Absurdity of Percentages

When you receive a genetic report from companies like 23andMe, they might tell you something like, “You’re 30% Greek and 15% Sicilian.” This might sound straightforward, but it’s actually misleading. Here’s why.

1. Genetic Overlap and Population Mixing

• Overlap of Populations: Populations like Greeks and Sicilians aren’t isolated or completely separate groups. Throughout history, people have moved, intermarried, and shared genetic material across regions. So, it’s not accurate to think of them as completely distinct categories.
• Not Independent Categories: Because of this overlap, you can’t neatly separate someone’s genes into strict percentages like “30% Greek” or “15% Sicilian.” These populations share a lot of genetic similarities, and the idea of being a specific percentage of one is not meaningful.

2. Principal Component Analysis (PCA)

• Mapping Genetic Data: To understand how populations relate genetically, scientists use a method called Principal Component Analysis (PCA). This method plots genetic samples on a map based on their similarities and differences.
• Orthogonal Basis: PCA works by creating a set of independent axes (like X and Y on a graph) that capture the most significant genetic variations. These axes are orthogonal, meaning there’s no overlap between them.
• Position on the Map: Different populations, like Greeks and Sicilians, are plotted on this map. Often, these populations overlap on the PCA plot, showing their genetic similarities. For example, you might see Greek samples very close to Sicilian ones, with some individuals falling in between, reflecting their mixed heritage.

3. Why Percentages Don’t Make Sense

• No Clear Boundaries: Since these populations overlap and share genetic traits, saying you’re 30% Greek and 15% Sicilian doesn’t capture the true genetic picture. It’s more accurate to say that your genes are positioned somewhere on a spectrum between these populations.
• Individual Variation: Each person’s genetic makeup is unique, and trying to fit it into neat percentages based on populations that aren’t strictly defined is not scientifically accurate.

4. PCA in Practice

• Eigenvalues and Variance: In PCA, the first principal component captures the most significant genetic variation, often more than 50%. The second component might capture around 20%, and the third even less. This shows that most of the genetic variation can be explained by just a couple of these components.
• Positioning Individuals: By looking at where your genes fall on these principal components, you can see how closely related you are to different populations. But again, these positions reflect a continuous gradient, not discrete categories.

5. A Word on Methodology

• Alternative Methods: There are other ways to analyze genetic data, like using mutual information or entropy-based techniques instead of just looking at correlations. These methods might give a slightly different picture but generally confirm that populations overlap and are not entirely distinct.

Conclusion

So, when someone says you’re “30% Greek and 15% Sicilian,” it’s best to take it with a grain of salt. These percentages don’t accurately reflect the complexity of human genetics. Instead, think of your ancestry as a point on a map that blends various genetic influences, rather than a set of fixed percentages.

MINI LECTURE 17: Maximum Ignorance Probability

Introduction

In trading and statistical analysis, making informed decisions often relies on understanding probabilities and their underlying distributions. This guide explores the concept of Maximum Ignorance Probability, its relationship with Maximum Entropy Probability, and practical applications in estimating error rates and coverage probabilities. We’ll delve into mathematical formulations and provide Python code snippets to illustrate these concepts.

1. Maximum Entropy Probability

Maximum Entropy Principle is a method for estimating probability distributions by selecting the distribution with the highest entropy among all those that satisfy certain constraints. This approach embodies the idea of making the least biased inference based on the given information.

Example: Fair Coin Toss

Consider a fair coin toss with two possible outcomes: Heads (H) and Tails (T).

• Entropy (H) for a discrete distribution is defined as:

  $$H(P)=-\sum _i{p}_i\log p_i$$

• For a fair coin:

  $$P(H)=P(T)=0.5$$

H(P) = -[0.5 \log 0.5 + 0.5 \log 0.5] = \log 2

This distribution maximizes entropy, representing maximum uncertainty or ignorance given the constraint that the probabilities sum to 1. ### 2. Extending to Maximum Ignorance Probability **Maximum Ignorance Probability** extends the concept of maximum entropy by considering scenarios with more complex constraints or lack of information. It is particularly useful in real-world situations where additional layers of uncertainty exist. #### Application: Estimating Error Rates in Surgical Procedures Imagine a surgeon performing 60 transplant surgeries. Without prior data, we aim to estimate the error rate—the probability that a surgery results in an error. **Challenges:** - No historical error rate data. - Uncertainty about whether the surgeon's performance is above or below a benchmark. **Approach:** 1. **Model the Number of Errors:** Assume the number of errors follows a **Binomial Distribution**:

X \sim \text{Binomial}(n=60, p)

where $p$ is the error rate. 2. **Determine the Fair Error Rate:** To find a probability $p$ such that there's a 50% chance the true error rate is above or below $p$, we seek the **median** of the binomial distribution. Mathematically, find $p$ where:

P(X \leq k) = 0.5

where $k$ is the observed number of errors. 3. **Connecting to Bayesian Probability:** The median of the binomial distribution relates to Bayesian inference, where prior beliefs are updated with observed data to form posterior distributions. ### 3. Mathematical Formulation #### Binomial Distribution The probability mass function (PMF) of a binomial distribution is:

P(X = k) = \binom{n}{k} p^k (1 - p)^{n - k}

where: - $n$ = number of trials (e.g., surgeries) - $k$ = number of successes (e.g., errors) - $p$ = probability of success on a single trial #### Finding the Median Probability To find the median probability $p$ such that $P(X \leq k) = 0.5$, we typically use numerical methods or inverse functions, as closed-form solutions are rare. **Relation to Beta Distribution:** The Beta distribution is the conjugate prior for the binomial distribution in Bayesian statistics. The median of the binomial distribution can be related to the parameters of the Beta distribution. ### 4. Practical Implications #### Estimating Fair Error Rate - **No Observations (k = 0):** Even with zero observed errors, the fair error rate isn't necessarily zero. Using maximum ignorance probability, we derive a non-zero error rate that reflects uncertainty. - **With Observations:** As the number of observations increases, the estimated error rate converges to the true error rate, though it never quite reaches zero. #### Example Calculation Suppose a surgeon has performed 60 surgeries with zero errors observed. 1. **Define the Objective:** Find $p$ such that:

P(X = 0) = (1 - p)^{60} = 0.5

2. **Solve for $p$:**

(1 - p)^{60} = 0.5

\ln(1 - p) = \frac{\ln(0.5)}{60}

1 - p = e^{\frac{\ln(0.5)}{60}} \approx e^{-0.01155} \approx 0.9885

p \approx 1 - 0.9885 = 0.0115 \text{ or } 1.15%

This **fair error rate** of 1.15% is neither zero (as naive frequency would suggest) nor based on an arbitrary Bayesian prior, but rather derived from the principle of maximum ignorance. ### 5. Coverage Probability **Coverage Probability** refers to the proportion of times a confidence interval contains the true parameter value. It is a measure of the reliability of an interval estimate. #### Importance in Probability Estimation A well-calibrated model should have coverage probabilities that match the nominal confidence levels. For example, a 95% confidence interval should contain the true parameter 95% of the time. ### 6. Connection to Bayesian Probability The exploration of maximum ignorance probability has influenced Bayesian approaches, particularly in defining priors that lead to accurate coverage probabilities. #### Example: Bayesian Coverage A Bayesian might use a Beta prior to model the error rate $p$. The posterior distribution, given observed data, allows for the construction of credible intervals with desired coverage properties. ### 7. Practical Example: Estimating Die Probabilities Consider a die that, on average, yields a score of 4.5 instead of the fair expectation of 3.5. #### Objective Determine the most likely probability distribution for the die's outcomes based on maximum ignorance. #### Approach 1. **Define Constraints:** - The die has outcomes 1 through 6. - The expected value $E[X] = 4.5$. 2. **Maximize Entropy:** Use the maximum entropy principle to find the distribution $P(X = x)$ that maximizes entropy subject to the given constraints. 3. **Resulting Distribution:** The higher outcomes (e.g., 6) will have higher probabilities than lower ones (e.g., 1), reflecting the increased average. ### 8. Python Implementation Here’s a Python example using the `scipy` library to find the median probability for the binomial distribution scenario described earlier. ```python import numpy as np from scipy.stats import binom from scipy.optimize import bisect def find_median_p(n, k, target_cdf=0.5): """ Find the probability p such that P(X <= k) = target_cdf for X ~ Binomial(n, p) """ def func(p): return binom.cdf(k, n, p) - target_cdf # p must be between 0 and 1 p_median = bisect(func, 0, 1) return p_median # Example: n=60, k=0 errors observed n = 60 k = 0 median_p = find_median_p(n, k) print(f"Median probability p for n={n}, k={k} is {median_p:.4f} ({median_p*100:.2f}%)") ``` **Output:** ``` Median probability p for n=60, k=0 is 0.0115 (1.15%) ``` This aligns with our manual calculation, confirming that with 60 trials and zero successes, the median probability is approximately 1.15%. ### 9. Conclusion **Maximum Ignorance Probability** provides a robust framework for estimating probabilities in scenarios with limited information. By extending the maximum entropy principle, it allows for the derivation of meaningful probability estimates even when data is scarce or entirely absent. This approach is particularly valuable in fields like trading and medical procedures, where understanding and mitigating error rates is crucial. By integrating mathematical formulations and practical examples, this guide aims to enhance your intuition and application of maximum ignorance probability in real-world situations. Here's a **refined and well-structured version** of your content with improved formatting, clearer organization, and professional tone while preserving all the technical detail: --- ## **Mini Lecture 17: Maximum Ignorance Probability** ### **Overview** In trading and statistical analysis, informed decision-making often depends on estimating probabilities under uncertainty. This lecture explores the concept of **Maximum Ignorance Probability**, its roots in the **Maximum Entropy Principle**, and how it applies to real-world problems such as estimating error rates and ensuring statistical fairness. --- ### **1. Maximum Entropy Principle** The **Maximum Entropy Principle** states that, among all probability distributions satisfying given constraints, the one with the highest entropy is the most unbiased choice. It reflects the state of **maximum uncertainty**, or **minimum prior knowledge**, consistent with known facts. #### Example: Fair Coin Toss For a fair coin (Heads = H, Tails = T): * The entropy of a discrete distribution:

H(P) = -\sum_{i} p_i \log p_i

* For $P(H) = P(T) = 0.5$:

H(P) = -[0.5 \log 0.5 + 0.5 \log 0.5] = \log 2

This configuration maximizes entropy, representing complete ignorance beyond knowing the outcomes are equally likely. --- ### **2. Maximum Ignorance Probability** This concept **extends maximum entropy** to scenarios where prior knowledge is severely limited or absent. It’s useful in high-stakes domains (e.g., medicine, finance) where decisions must be made with little or no historical data. #### Case Study: Surgical Error Rates * **Scenario**: A surgeon performs 60 transplants. No prior data exists on error frequency. * **Goal**: Estimate a “fair” error rate without assumptions. #### Modeling Approach: 1. **Binomial Distribution**:

X \sim \text{Binomial}(n=60, p)

2. **Objective**: Find the value of $p$ such that:

P(X \leq k) = 0.5

where $k$ is the observed number of errors (e.g., $k = 0$) — i.e., find the **median** of the binomial distribution. 3. **Interpretation**: This estimate reflects the idea that we are equally uncertain about whether the true rate is higher or lower than this value — hence, **maximum ignorance**. --- ### **3. Mathematical Details** #### Binomial PMF:

P(X = k) = \binom{n}{k} p^k (1 - p)^{n - k}

#### Solving for Median $p$: * With $k = 0$, solve:

(1 - p)^{60} = 0.5

p \approx 1 - e^{\ln(0.5)/60} \approx 0.0115 \ (\text{or } 1.15%)

This gives a non-zero estimate even when no errors are observed — a key insight of the **maximum ignorance** approach. --- ### **4. Connection to Bayesian Inference** This method parallels **Bayesian thinking** without requiring explicit priors: * The binomial model corresponds to using a **Beta** prior in Bayesian statistics. * The maximum ignorance approach yields similar results as using an uninformative prior, but derived from symmetry and fairness principles. --- ### **5. Coverage Probability** **Coverage probability** is the proportion of times a confidence or credible interval contains the true parameter value. * A model is **well-calibrated** if the nominal level (e.g., 95%) matches empirical performance. * Maximum ignorance approaches help construct such intervals by avoiding overconfident or overly narrow estimates in data-scarce environments. --- ### **6. Practical Example: Biased Die** **Problem**: A die yields an average value of 4.5 (vs. 3.5 for a fair die). What is the most likely distribution? #### Steps: 1. **Constraints**: * Outcomes: 1 through 6 * Expected value: $E[X] = 4.5$ 2. **Solution**: Maximize entropy subject to $\sum p_i = 1$ and $\sum i \cdot p_i = 4.5$ 3. **Outcome**: Higher face values (e.g., 5, 6) will have higher probabilities, reflecting the biased mean. --- ### **7. Python Implementation** Here’s Python code to compute the median probability using numerical methods: ```python import numpy as np from scipy.stats import binom from scipy.optimize import bisect def find_median_p(n, k, target_cdf=0.5): def func(p): return binom.cdf(k, n, p) - target_cdf return bisect(func, 0, 1) # Example: n=60 surgeries, k=0 errors n = 60 k = 0 median_p = find_median_p(n, k) print(f"Median error rate: {median_p:.4f} ({median_p*100:.2f}%)") ``` **Output**: ``` Median error rate: 0.0115 (1.15%) ``` --- ### **8. Key Takeaways** * **Maximum Ignorance Probability** provides rational estimates in the face of extreme uncertainty. * It reflects symmetry and fairness without relying on arbitrary priors. * Useful in fields like medicine, risk management, and algorithmic trading. **✅ Condensed Learning Notes:** --- ## MINI LECTURE 18: Positive Definite Correlation Matrix for Monte Carlo ### 🔹 Purpose & Challenge * Needed for simulations to represent correlation/covariance. * **High dimensionality (large *n*)** makes construction/calibration difficult. ### 🔹 Sample Size Effect on Correlation * **Small sample** → high variability in observed correlations (even if true correlation = 0). * **Large sample** → observed correlations converge to true values. * As *n* → ∞, correlations of uncorrelated variables → 0. ### 🔹 Constructing Positive Definite Matrices * Use samples from high-variance, fat-tailed distributions (e.g., chi-square, low-DoF Student's *t*). * Check matrix is positive definite. * Analyze with PCA. ### 🔹 PCA & Dimensionality * **1st Principal Component (PC)** may explain large variance (e.g., 55%) → low dimensionality. * Use higher DoF Student’s *t* to distribute variance more evenly. * PCA helps refine sample distribution & dimensionality. ### 🔹 Key Takeaways * Small samples can cause **spurious correlations**. * Distribution choice and sample size directly shape **correlation structure**. * PCA = diagnostic tool to assess/fine-tune matrix structure. --- ## Gates Foundation & GMO Mosquitoes: Warnings from History ### 🔹 The Precautionary Principle * Avoid actions with unknown consequences, especially in ecosystems. ### 🔹 Genetic Modification Risks * **Unpredictable ecological effects**. * GM organisms can **spread, evolve**, disrupt ecosystems. * Changes are **rapid**, unlike slow **natural selection**. ### 🔹 Historical Parallel: Mao’s Four Pests * Eliminated sparrows → ecosystem imbalance → insect overpopulation → famine. * Shows **ecological complexity** & risks of tampering. ### 🔹 Mathematical Insight: Speed vs. Harm * Risk of harm **rises non-linearly** with speed of change. * $\text{Speed} ↑ → \text{Harm} ↑↑↑$ * Natural evolution = slow, allows adjustment. * GMO = **fast change**, nature can’t adapt → high risk. ### 🔹 Ecosystem Complexity * **Reintroduction of wolves** → widespread ecological changes. * **Small species changes** can shift entire ecosystems. ### 🔹 Technological Hubris * Gates’ mosquito control = risk of **Mao-style miscalculation**. * Arrogance in thinking we can “control nature” → dangerous. --- ## Bitcoin Black Paper: Elementary Security Pricing ### 🔹 Currency vs. Security * Bitcoin ≠ stable currency (high volatility). * Lacks intrinsic value/cash flows → speculative. ### 🔹 Security Pricing Formula

P_t = \frac{C_{t+1} + E[P_{t+2}]}{1 + r_d}

* Recursively based on **expected future cash flows**. ### 🔹 Bitcoin’s Special Case * No cash flows/dividends → value = **what next buyer pays**. * Resembles **Ponzi dynamics**. ### 🔹 Absorption Barrier Concept * Bitcoin can hit value = 0 if system/tech fails. * Unlike gold, Bitcoin lacks **intrinsic use value**. ### 🔹 Adjusted Valuation Under Failure Risk * Modified discount factor:

\frac{1 - \pi}{1 + r_d}

$$\ast Pricemustgrowat:$$

1 + r_d + \pi

to remain viable. ### 🔹 Conclusion * Bitcoin = speculative asset, not true currency. * Vulnerable to loss of belief, tech disruption, or structural plateau. * **Sustainability = dependent on continued buyer demand**.