{"id":963,"date":"2021-12-24T15:16:00","date_gmt":"2021-12-24T15:16:00","guid":{"rendered":"https:\/\/firemymoneymanager.com\/?p=963"},"modified":"2022-04-01T01:53:07","modified_gmt":"2022-04-01T01:53:07","slug":"using-autocorrelation-to-predict-stock-returns","status":"publish","type":"post","link":"https:\/\/firemymoneymanager.com\/using-autocorrelation-to-predict-stock-returns\/","title":{"rendered":"Using autocorrelation to predict stock returns [2022]"},"content":{"rendered":"\n<p>In this article we will find stocks and ETFs that have high levels of autocorrelation, and try to determine whether that data can be used to build a trading strategy.<\/p>\n\n\n\n<p>Related articles:<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/firemymoneymanager.com\/getting-started-using-python-to-find-alpha\/\" data-type=\"post\" data-id=\"221\">Getting started: using Python to find alpha [2021]<\/a><\/li><li><a href=\"https:\/\/firemymoneymanager.com\/do-capm-efficient-portfolios-really-outperform-random-ones\/\" data-type=\"post\" data-id=\"829\">Do CAPM efficient portfolios really outperform random ones? [2021]<\/a><\/li><li><a href=\"https:\/\/firemymoneymanager.com\/do-equities-really-follow-a-normal-distribution\/\" data-type=\"post\" data-id=\"421\">Do Equities Really Follow a Normal Distribution? [2021]<\/a><\/li><\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2>Using the partial autocorrelation function<\/h2>\n\n\n\n<p>To determine whether there is autocorrelation &#8212; and if so, how much there is &#8212; we use the PACF, or Partial Autocorrelation Function.  This function determines the correlation of a time series with itself a number of periods prior.<\/p>\n\n\n\n<p>In order to illustrate, we will use the SPY ETF.  It&#8217;s very simple to run the PACF in Python.  In the following code, we load the SPY data and plot the PACF:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">data = pd.read_csv(\"spy.csv\")\ndata.index = pd.to_datetime(data['date'])\ndata = data['adj_close'].pct_change()\ndata = data['2018-01-01':]\nplot_pacf(data)<\/pre>\n\n\n\n<p>The resulting plot looks like this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"640\" height=\"480\" src=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1.png\" alt=\"A plot of the PACF function for the autocorrelation of SPY by day\" class=\"wp-image-1031\" srcset=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1.png 640w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-300x225.png 300w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-620x465.png 620w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-50x38.png 50w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><figcaption>A plot of the PACF function for the autocorrelation of SPY by day<\/figcaption><\/figure>\n\n\n\n<p>This plot shows the degree of autocorrelation between the return of SPY today and the return of SPY <em>n<\/em> days back, where <em>n <\/em>is the x axis.<\/p>\n\n\n\n<p>From this plot, we see that the autocorrelation between the SPY returns on a given day and the returns on the previous day (where the x axis = 1) is around -.25. We can also see that the correlation between the SPY returns on a given day and two days prior is around .1.<\/p>\n\n\n\n<p>The light blue bands tell us whether this correlation is statistically significant.  If a value is outside of the band, it is statistically significant.  Otherwise, it is not and we should ignore it.  Note that for nearly all days after day 9, the values are not statistically significant. <\/p>\n\n\n\n<p>As an aside, we could also use the <em>plot_acf<\/em> function, which would give us this plot:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"640\" height=\"480\" src=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_2.png\" alt=\"A plot of the ACF function for SPY by day\" class=\"wp-image-1034\" srcset=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_2.png 640w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_2-300x225.png 300w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_2-620x465.png 620w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_2-50x38.png 50w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><figcaption>A plot of the ACF function for SPY by day<\/figcaption><\/figure>\n\n\n\n<p>Note that the values in the second plot are higher than those in the previous plot.  The PACF ignores indirect correlations, whereas the ACF includes them.  <\/p>\n\n\n\n<p>For instance, if the return at day <em>t <\/em>is correlated with the return at day <em>t-1<\/em>, and the return at day <em>t-1 <\/em>is correlated with the return at day <em>t-2<\/em>, then the ACF shows a correlation between days <em>t<\/em> and <em>t-2<\/em>.  The PACF would not show this correlation since it is already taken into account in the correlation between days <em>t<\/em> and <em>t-1.<\/em><\/p>\n\n\n\n<h2>Determining autocorrelation for the universe of stocks and ETFs<\/h2>\n\n\n\n<p>We can now run a test for correlation on our entire dataset, and look for the most correlated tickers.  Since we don&#8217;t want to plot every ticker, we will replace <em>plot_pacf <\/em>with <em>pacf.  <\/em>Note that <em>pacf <\/em>returns the correlation as well as a shifted confidence interval.  If that interval contains 0, it means that the result is statistically significant.<\/p>\n\n\n\n<p>Here&#8217;s our code to do this.  We use stock data and ticker info from Quandl:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">try: \n    ticker_data = pd.read_csv(\"tickers.csv\")\nexcept FileNotFoundError as e:\n    pass\n\n\nticker_data = ticker_data.loc[((ticker_data['table'] == 'SF1') | (ticker_data['table'] == 'SFP')) &amp; (ticker_data['isdelisted'] == 'N') &amp; (ticker_data['currency'] == 'USD') &amp; ((ticker_data.exchange == 'NYSE') | (ticker_data.exchange == 'NYSEMKT') | (ticker_data.exchange == 'NYSEARCA') | (ticker_data.exchange == 'NASDAQ'))]\n\nresults = pd.DataFrame(columns=['ticker','l1', 'l2', 'l3', 'l4', 'l5'])\n\nfor t in ticker_data['ticker']:\n    if t == 'TRUE': continue\n    try:\n        data = pd.read_csv(t + \".csv\", \n                            header = None,\n                            usecols = [0, 1, 12],\n                            names = ['ticker', 'date', 'adj_close'])\n    \n        data.index = pd.to_datetime(data['date'])\n        data = data['adj_close'].pct_change()\n        data = data['2014-01-01':]\n        \n        if (len(data) &lt; 15):\n            continue\n        \n        [res,ci] = pacf(data, nlags=5, alpha=.05)\n        for j in range(1, 6):\n            if res[j] > 0:\n                res[j] = ci[j][1]\n            else:\n                res[j] = ci[j][0]\n            \n\n        results.loc[len(results)]= [t] + res[1:].tolist()\n    except FileNotFoundError:\n        pass<\/pre>\n\n\n\n<p>The <em>pacf<\/em> function returns a list of results and a confidence interval for each one.  The function is written in such a way that the confidence interval is shifted by the result.  Therefore, we can simply look at the distance of the boundary of the confidence interval from 0.<\/p>\n\n\n\n<p>After running this code, we have a <em>results<\/em> DataFrame which contains 5 days of autocorrelations by ticker.  Let&#8217;s look at the tickers with the highest day 1 correlation:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>     ticker        l1        l2        l3        l4        l5\n2172   KODK  0.438201 -0.250540  0.055132 -0.070692  0.081623\n6496    TFI  0.349147  0.149567 -0.121855 -0.338199 -0.049053\n6113    PZA  0.345831 -0.119451  0.062336 -0.238694 -0.194027\n5450   IGSB  0.333443 -0.055681  0.115657 -0.131023  0.083334\n6620   VCSH  0.332864  0.079902  0.062387 -0.127547  0.062723\n5372   HYLS  0.317959  0.202700  0.050186 -0.166055 -0.144543\n5779   MINT  0.316034  0.065577  0.222451  0.136379 -0.052238\n5924    NYF  0.313376  0.201103 -0.186961 -0.245647 -0.130453\n909    CNTY  0.294874  0.139255 -0.179101 -0.161458 -0.053789\n3959   UONE  0.284736 -0.055110  0.184917 -0.161995  0.087707\n5157   FTSL  0.278337  0.207237  0.119671 -0.056239 -0.072306\n4382    BAB  0.273436 -0.236646 -0.196171  0.107094  0.127834\n3337    RWT  0.264686 -0.163005 -0.170003  0.051506 -0.189675\n1253    EFC  0.261613 -0.078656 -0.181922  0.171257 -0.297007\n5801   MORT  0.257654  0.056583 -0.135256  0.086857 -0.144170\n2687   NMFC  0.251705  0.145104 -0.101689 -0.267327 -0.114394\n6354   SLQD  0.250446  0.134402  0.089237 -0.232306  0.124600\n6401   SPIB  0.248056  0.092669  0.083518 -0.157038  0.090045\n1922   IBIO  0.244930 -0.175744  0.098071  0.094607 -0.051653\n23      ABR  0.241716  0.070130 -0.104922 -0.090260 -0.091360<\/code><\/pre>\n\n\n\n<p>Let&#8217;s look at some of these tickers individually.  Here is the partial autocorrelation plot for KODK:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"638\" height=\"462\" src=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa1.png\" alt=\"A partial autocorrelation stock return plot for KOD\" class=\"wp-image-1085\" srcset=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa1.png 638w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa1-300x217.png 300w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa1-620x449.png 620w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa1-50x36.png 50w\" sizes=\"(max-width: 638px) 100vw, 638px\" \/><figcaption>A partial autocorrelation stock return plot for KODK<\/figcaption><\/figure>\n\n\n\n<p>We can see that there is an extremely strong one day lagged autocorrelation for this stock.  Let&#8217;s look at another (TFI):<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" width=\"638\" height=\"462\" src=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa2.png\" alt=\"\" class=\"wp-image-1096\" srcset=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa2.png 638w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa2-300x217.png 300w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa2-620x449.png 620w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/pa2-50x36.png 50w\" sizes=\"(max-width: 638px) 100vw, 638px\" \/><\/figure>\n\n\n\n<h2>Is there autocorrelation of stock returns on larger time scales?<\/h2>\n\n\n\n<p>Now we will look at monthly time scales.  We adjust our code to resample by business month:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">try: \n    ticker_data = pd.read_csv(\"tickers.csv\")\nexcept FileNotFoundError as e:\n    pass\n\n\nticker_data = ticker_data.loc[((ticker_data['table'] == 'SF1') | (ticker_data['table'] == 'SFP')) &amp; (ticker_data['isdelisted'] == 'N') &amp; (ticker_data['currency'] == 'USD') &amp; ((ticker_data.exchange == 'NYSE') | (ticker_data.exchange == 'NYSEMKT') | (ticker_data.exchange == 'NYSEARCA') | (ticker_data.exchange == 'NASDAQ'))]\n\nresults = pd.DataFrame(columns=['ticker','l1', 'l2', 'l3', 'l4', 'l5'])\n\nfor t in ticker_data['ticker']:\n    if t == 'TRUE': continue\n    try:\n        data = pd.read_csv(t + \".csv\", \n                            header = None,\n                            usecols = [0, 1, 12],\n                            names = ['ticker', 'date', 'adj_close'])\n    \n        data.index = pd.to_datetime(data['date'])\n        data = data['adj_close'].pct_change()\n        data = data['2014-01-01':]\n        \n        if (len(data) &lt; 15):\n            continue\n        \n        [res,ci] = pacf(data, nlags=5, alpha=.05)\n        for j in range(1, 6):\n            if res[j] > 0:\n                res[j] = ci[j][1]\n            else:\n                res[j] = ci[j][0]\n            \n\n        results.loc[len(results)]= [t] + res[1:].tolist()\n    except FileNotFoundError:\n        pass<\/pre>\n\n\n\n<p>Here we have the top monthly auto-correlated tickers, with the last 5 months&#8217; lag showing. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>     ticker        l1        l2        l3        l4        l5\n4155    BIL  1.062948  0.671578  0.406545  0.296207  0.267860\n2030   KIRK  0.741970  0.273084 -0.483974  0.308310  0.236260\n5896    SHV  0.693188  0.515499  0.327983  0.318500  0.461331\n1694   HEAR  0.647345  0.263520 -0.328804 -0.238193 -0.389770\n689     CCV  0.587156  0.310129  0.583598 -0.552379 -0.401946\n3365   SPNV  0.557627  0.260339 -0.302074  0.403751  0.240059\n5593    PFH  0.555545  0.234674 -0.280491 -0.271333  0.283133\n3779   VBIV  0.555185 -0.345410 -0.337042 -0.324246  0.310560\n3194   SCOR  0.545700 -0.297076 -0.406047 -0.368039 -0.321558\n5496    NMZ  0.543346 -0.437787 -0.305792 -0.396759 -0.256477\n125     AIM  0.538091  0.280120 -0.369887 -0.329034 -0.257873\n2730   OSTK  0.537406  0.276835  0.495272 -0.391599 -0.367689\n5495    NMY  0.533144 -0.401574  0.255636 -0.376926 -0.260971\n1683   HCAP  0.530111 -0.391663  0.304328 -0.269658  0.380937\n2603   NVAX  0.529282  0.264697  0.377803  0.306428  0.308998\n5430    MUJ  0.516649 -0.255428 -0.330438 -0.346465  0.221122\n119    AHPI  0.514086 -0.409486  0.222293  0.227159 -0.263185\n4181    BNY  0.508837 -0.338257  0.313896 -0.415420  0.336402\n2810   PEIX  0.503102  0.337251  0.265116 -0.280291 -0.313332\n2056   KOPN  0.500044  0.364696 -0.384475  0.236588 -0.332640<\/code><\/pre>\n\n\n\n<p>Let&#8217;s plot some of these individually.  <\/p>\n\n\n\n<h2>Can autocorrelation of stock returns be used in a trading strategy?<\/h2>\n\n\n\n<p>In this section we will examine whether we can use autocorrelation of stock returns in order to build a profitable trading strategy.  For simplicity, we will assume that our carrying fees and trading costs are zero (though in practice, these could affect the profitability significantly).<\/p>\n\n\n\n<p>Let&#8217;s start with a monthly strategy using the most autocorrelated ETF we found.  We will run a simple strategy using BIL, where we buy the ETF at closing whenever the previous month had a positive return.  We will short the ETF if the previous month had a negative return.  <\/p>\n\n\n\n<p>We use the <em>bt<\/em> package in Python to run this strategy, and get the following result:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Total Return         4.45%\nDaily Sharpe         2.37\nDaily Sortino        4.15\nCAGR                 0.56%\nMax Drawdown         -0.35%\nCalmar Ratio         1.61<\/code><\/pre>\n\n\n\n<p>This is a pretty good result considering the low volatility.  Here is a chart of our returns:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"341\" src=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-1024x341.png\" alt=\"Returns on an autocorrelation trading strategy on BIL\" class=\"wp-image-1111\" srcset=\"https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-1024x341.png 1024w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-300x100.png 300w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-768x256.png 768w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-620x207.png 620w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-800x267.png 800w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2-50x17.png 50w, https:\/\/firemymoneymanager.com\/wp-content\/uploads\/2021\/12\/Figure_1-2.png 1500w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Returns on an autocorrelation trading strategy on BIL<\/figcaption><\/figure>\n\n\n\n<p>Unfortunately, if we factor in short selling and trading costs, this trade will likely become a loser.  Let&#8217;s try another ticker.  MUJ is a municipal bond fund, which exhibits low volatility.  If we run the backtest on this ticker, we get:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Total Return         50.29%\nDaily Sharpe         0.55\nDaily Sortino        0.80\nCAGR                 5.41%\nMax Drawdown         -28.19%\nCalmar Ratio         0.19<\/code><\/pre>\n\n\n\n<p>A 50% return contrasts with the 5-10% return you would have received with a buy and hold strategy. So this is profitable, but we would have done much better if we just invested in the S&amp;P500.<\/p>\n\n\n\n<h2>Conclusion<\/h2>\n\n\n\n<p>So can autocorrelation predict stock returns? The answer is: somewhat.  We can find certain tickers that are reasonably autocorrelated, and those are somewhat predictable.  However, building a successful trading strategy using just autocorrelation is very difficult &#8212; if not impossible.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this article we will find stocks and ETFs that have high levels of autocorrelation, and try to determine whether that data can be used to build a trading strategy. Related articles: Getting started: using Python to find alpha [2021] Do CAPM efficient portfolios really outperform random ones? [2021] Do Equities Really Follow a Normal [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1050,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[37,21,38,9],"_links":{"self":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts\/963"}],"collection":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/comments?post=963"}],"version-history":[{"count":46,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts\/963\/revisions"}],"predecessor-version":[{"id":1123,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/posts\/963\/revisions\/1123"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/media\/1050"}],"wp:attachment":[{"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/media?parent=963"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/categories?post=963"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/firemymoneymanager.com\/wp-json\/wp\/v2\/tags?post=963"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}