{"id":421,"date":"2021-03-08T14:53:37","date_gmt":"2021-03-08T14:53:37","guid":{"rendered":"https:\/\/firemymoneymanager.com\/?p=421"},"modified":"2021-03-10T15:27:22","modified_gmt":"2021-03-10T15:27:22","slug":"do-equities-really-follow-a-normal-distribution","status":"publish","type":"post","link":"https:\/\/firemymoneymanager.com\/do-equities-really-follow-a-normal-distribution\/","title":{"rendered":"Do Equities Really Follow a Normal Distribution? [2021]"},"content":{"rendered":"\n

The basis for modern portfolio theory, as well as many quantitative strategies for investing or trading is that financial instruments — especially equities — follow a normal distribution. In our articles on finding alpha<\/a>, CAPM, or even pairs trading<\/a>, we assume a generally normal (but not necessarily perfectly normal) distribution.<\/p>\n\n\n\n

But do they still do that in 2021? Did they ever do that? We will attempt to answer that question in this article.<\/p>\n\n\n\n

How to identify a normal distribution<\/h2>\n\n\n\n

A normal distribution is a sample distribution that is roughly bell shaped. One of the most important characteristics of a normal distribution is that it is symmetrical around the center.<\/p>\n\n\n\n

We can create a random normal distribution to see this using Python:<\/p>\n\n\n\n

z = np.random.normal(size=1000)\nsns.distplot(z)<\/pre>\n\n\n\n
The resulting plot shows the 1000 samples form this symmetrical bell curve shape:<\/p>\n\n\n\n
\"the<\/figure><\/div>\n\n\n\n
But how do we determine that this is really a normal distribution?  How do we quantify its normalness? <\/em><\/p>\n\n\n\n
First, let’s look at a QQ plot.  A QQ plot is a good way for visually determining normalness.  We won’t go into the specifics of how it works<\/a>, but a normal distribution should show up on the QQ plot as a straight line.  Here is the plot of the above distribution:<\/p>\n\n\n\n
\"a<\/figure><\/div>\n\n\n\n
But we need to do this for a lot of equities.  How can we do this quantitatively? We can use a Kolmogorov-Smirnov test.<\/p>\n\n\n\n
Here is how the NIST <\/a>explains this test:<\/p>\n\n\n\n
The Kolmogorov-Smirnov test is used to decide if a sample comes from a population with a specific distribution.<\/p>
The Kolmogorov-Smirnov (K-S) test is based on the empirical distribution function (ECDF). Given N<\/em> ordered<\/em> data points Y1<\/sub>, Y2<\/sub>, …, YN<\/sub><\/em><\/strong>, the ECDF is defined as<\/p>
EN<\/sub>=n(i)\/N<\/p>
where n(i)<\/em><\/strong> is the number of points less than Yi<\/sub><\/em><\/strong> and the Yi<\/sub><\/em><\/strong> are ordered from smallest to largest value. This is a step function that increases by 1\/N<\/em> at the value of each ordered data point.<\/p><\/blockquote>\n\n\n\n
Essentially, what this test tells us is the probability that one of the samples is in the red zone in the graph below (the zone that deviates from the normal distribution):<\/p>\n\n\n\n
\"how<\/figure><\/div>\n\n\n\n
If the probability is large, that means that the data is not very normally distributed.  If the probability is small, then the data is normally distributed.  (Note that the way the actual test works is more complicated and beyond the scope of this article.)<\/p>\n\n\n\n
Generally, people interpret the results as a normal distribution if the test returns more than .05.  Let’s try this test on our sample data:<\/p>\n\n\n\n
scipy.stats.kstest(rvs=z, cdf='norm')\nOut[18]: KstestResult(statistic=0.021468169352233724, pvalue=0.737569346346318)<\/code><\/pre>\n\n\n\n
So the KS statistic is .73, which is far above our .05 cutoff.  This is clearly a strong normal distribution.<\/p>\n\n\n\n
Determining normal distribution for a stock<\/h2>\n\n\n\n
In order to determine whether equities follow this normal distribution, we need to load some data.  We use Quandl data to load in equities on the NASDAQ or NYSE.  For this test, we will also look at ETFs.<\/p>\n\n\n\n
We use the following Python code to load the data (this code is based on the code we developed in Getting Started<\/a>):<\/p>\n\n\n\n
alldata = pd.DataFrame()\n\n\ntry: \n    ticker_data = pd.read_csv(\"tickers.csv\")\nexcept FileNotFoundError as e:\n    pass\n\n\nticker_data = ticker_data.loc[((ticker_data['table'] == 'SF1')) & (ticker_data['isdelisted'] == 'N') & (ticker_data['currency'] == 'USD') & ((ticker_data.exchange == 'NYSE') | (ticker_data.exchange == 'NYSEMKT') | (ticker_data.exchange == 'NYSEARCA') | (ticker_data.exchange == 'NASDAQ'))]\n\n\nfor t in ticker_data['ticker']:\n    if t == 'TRUE': continue\n        \n    print(t)\n    \n    try:\n        data = pd.read_csv(t + \".csv\", \n                            header = None,\n                            usecols = [0, 1, 12],\n                            names = ['ticker', 'date', 'adj_close'])\n    \n        if (len(data) < 2000): #ignore bad data (we have to look into wjhy)\n            continue\n        \n        alldata = alldata.append(data)\n        \n    except FileNotFoundError as e:\n        pass\n\ndf = alldata.set_index('date')\ntable = df.pivot(columns='ticker')\ntable.columns = [col[1] for col in table.columns]\n<\/pre>\n\n\n\n
Now we can look at some distribution charts.  Let’s try Ford (F) first.  We clean the data by calculating percentage changes (for comparison) and droping N\/A values.<\/p>\n\n\n\n
ford = table['F'].pct_change().dropna()\nsns.distplot(ford)<\/pre>\n\n\n\n
And here is the chart output.  Looks pretty normal, right?<\/p>\n\n\n\n
\"is<\/figure><\/div>\n\n\n\n
Now, let’s run the KS test on this distribution and determine how normal it actually is:<\/p>\n\n\n\n
scipy.stats.kstest(rvs=ford, cdf='norm')\nOut[16]: KstestResult(statistic=0.4686455878905682, pvalue=0.0)<\/code><\/pre>\n\n\n\n
Uh oh…why is it 0?  The function uses the standard normal distribution.  To tell it to use a different normal distribution, we have to add the mean and standard deviation as arguments:<\/p>\n\n\n\n
scipy.stats.kstest(ford, 'norm', args=(ford.mean(),ford.std()))\nOut[21]: KstestResult(statistic=0.07156629413923976, pvalue=1.3427916967898793e-54)<\/code><\/pre>\n\n\n\n
So the actual value is very small.  This isn’t a normal distribution.  The  QQ plot makes this pretty clear:<\/p>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
If this were a normal distribution, the plot would be linear; however, it’s much more of an S shape.  We can remove the outliers to make it look more like a normal distribution, but it still doesn’t fit that well.<\/p>\n\n\n\n
The tails of the plot show that the stock has more extreme values than the normal distribution would predict.<\/p>\n\n\n\n
Determining normal distribution over all equities<\/h2>\n\n\n\n
In the last section, we found that Ford stock didn’t meet the criteria for a real normal distribution.  <\/p>\n\n\n\n
Now, we want to determine whether equities as a whole are normally distributed.  So we will try to run the same code to get an average p-value for all equities in our dataset:<\/p>\n\n\n\n
ks = pd.DataFrame(columns = ['ticker', 'ks', 'p'])\ni = 0\n\nfor ticker in table.columns:\n    pct_returns = table[ticker].pct_change().dropna()\n    r = scipy.stats.kstest(pct_returns, 'norm', args=(pct_returns.mean(),pct_returns.std()))\n\n    ks.loc[i] = [ticker, r.statistic, r.pvalue]\n    i = i+1\n    <\/pre>\n\n\n\n
Running that code, we get an average p value that is significantly less than .05.  Let’s look at a QQ plot of the S&P 500 ETF to understand how it diverges from a normal distribution:<\/p>\n\n\n\n
\"QQ<\/figure><\/div>\n\n\n\n
Let’s do a sort on all of the KS p-values and see if there are any that show a true normal distribution:<\/p>\n\n\n\n
     ticker        ks         p\n2499   NVCN  0.374057  0.000000\n612     CKX  0.291689  0.000000\n2547    OEG  0.282855  0.000000\n2055    LEU  0.300688  0.000000\n1775   IKNX  0.251512  0.000000\n    ...       ...       ...\n3694   VGLT  0.037264  0.001079\n3551    TYO  0.035553  0.001320\n3639    UST  0.033326  0.005089\n1751   IGOV  0.031124  0.006564\n3606    UNL  0.025611  0.056922\n\n[3937 rows x 3 columns]<\/code><\/pre>\n\n\n\n
So there are some tickers that do (just barely) meet the criteria for a normal distribution.  However, these aren’t stocks.  They’re treasury ETFs and other non-equity ETFs.  The equity tickers are all under the threshold for a normal distribution.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
From our analysis, it is clear that stocks returns don’t follow a totally normal distribution.  The question we need to ask in the future is whether the distribution is sufficiently normal for modern portfolio theory to work.  We will study that in other articles.<\/p>\n","protected":false},"excerpt":{"rendered":"
The basis for modern portfolio theory, as well as many quantitative strategies for investing or trading is that financial instruments — especially equities — follow a normal distribution. In our articles on finding alpha, CAPM, or even pairs trading, we assume a generally normal (but not necessarily perfectly normal) distribution. But do they still do […]<\/p>\n","protected":false},"author":1,"featured_media":425,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6],"tags":[7,20,19],"_links":{"self":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts\/421"}],"collection":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/comments?post=421"}],"version-history":[{"count":0,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts\/421\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/media\/425"}],"wp:attachment":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/media?parent=421"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/categories?post=421"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/tags?post=421"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}