Forecast Combination and Machine-Learning Market Timing: Out-of-Sample and Replication Evidence on the US Equity Premium
Contents11 sections
- Q1 The skeptical benchmark: Goyal Welch 2008 and Goyal, Welch Zafirov 2024
- Q2 Campbell Thompson 2008: economic restrictions and small R², large utility
- Q3 Forecast combination and dimension reduction: RSZ 2010, NRTZ 2014, Kelly–Pruitt, DLRZ 2022
- Q4 Machine-learning timing: GKX 2020, KMZ 2024 and its critique, Avramov et al.
- Q5 Data mining, multiple testing and statistical pitfalls; local predictability
- Q6 Independent replication: which methods survive?
- Q7 When predictability appears: recession concentration and failure periods
- Q8 Transaction costs, leverage limits and the buy-and-hold benchmark
- Q9 Turning points vs average returns; application to Nasdaq / QQQ; retail feasibility
- Further reading
- Frequently asked questions
This note compiles the skeptical benchmark on whether the US equity premium (S&P 500 / CRSP VW) is predictable (the Goyal–Welch series), together with out-of-sample (OOS) figures, source table numbers and the strongest critiques for forecast combination, PCA / PLS and machine-learning timing methods. Anything marked "insufficient evidence / not verified" means that this round could not obtain verbatim figures from the original text, and the report writer must not fill in numbers on their own.
Site note: This is one of five articles in the series "Ranking the Empirical Evidence on VOO / QQQ Market-Timing Methods" (this article: forecast combination and machine learning). It is the full text of the site owner's research note, published as is: content, figures, tables, citations and the "not found / not verified" passages appear as in the original, with nothing deleted or rewritten; only the table-of-contents block at the top of the original draft was removed. The source file is
combination_ml_replication.mdin this series' research-notes folder (research_notes/, folder name in Chinese); this note does not state a research date, and the other notes in the same series state a research date of 2026-10-26. The default prior throughout the series is "timing does not work" (Goyal & Welch 2008; Goyal, Welch & Zafirov 2024). Evidence levels follow the original definitions: "verified against the original" = text extracted directly from the paper PDF during research; "secondary source" = paraphrased from an abstract page or someone else's replication paper; "not verified" = based only on general knowledge of the literature, without obtaining the original. This article is a research record, not investment advice.
Evidence levels: "verified against the original" = text extracted directly from the paper PDF in this round; "secondary source" = paraphrased from an abstract page or someone else's replication paper; "not verified" = based only on general knowledge of the literature, without obtaining the original.
Q1 The skeptical benchmark: Goyal & Welch 2008 and Goyal, Welch & Zafirov 2024
Takeaway
GW 2008 concludes that, as of 2005, most variables were already unstable in sample, generally lost to the historical mean out of sample, and could not help investors time the market. GWZ 2024 extends the data to the end of 2021 and finds that only 11 of 46 variables "should not be dismissed outright." Of 184 investment strategies, only 3 (accrul, gpce, gip, all annual variables) statistically beat "always fully in stocks," and their t-statistics are only 1.3–1.5, i.e. weakly significant.
Cited Findings
GW 2008 (RFS 21(4):1455–1508, verified against the original)
- Abstract conclusion: these models "have performed poorly both in-sample and out-of-sample for 30 years," the models are unstable, and they "would not have helped an investor with access only to available information to profitably time the market" — Welch & Goyal 2008, RFS, doi:10.1093/rfs/hhm014
- For many variables, early significance "comes exclusively from the years of the 1973–1975 oil shock"; OOS failure is concentrated in the later part of the sample, not the earlier part. This indicates the problem is model instability, not lack of test power — GW 2008
- Table 1 (annual, OLS): significant IS variables are b/m, i/k, ntis, eqis and the kitchen sink. OOS R² (forecasts starting 20 years after the sample begins): eqis +2.04%** (ΔRMSE +0.30), the only OOS significantly positive one; b/m −1.72%; i/k −1.77%; ntis −5.07%; kitchen sink −139.03%; model selection −22.50%. If forecasts start in 1965: eqis −1.00%, b/m −12.71%, ntis −6.79%. Variables that are not significant IS all have negative OOS R², e.g. d/p −2.06%, d/y −1.93%, e/p −1.78%, tbl −3.37%, lty −7.72%, svar −27.14% — GW 2008 Table 1
- The text states that eqis became the only OOS-significant variable because it performed very well during the oil shock; for i/k, "the good OOS performance occurs during the oil shock" — GW 2008
- Table 2 (5-year returns): i/k IS R² 33.99%*, OOS R² 12.99%. The original says i/k's performance is "driven by its ability to predict the 2000 crash" — GW 2008 Table 2 (these are overlapping long-horizon returns; see the Boudoukh et al. critique in Q5)
- Table 3 (monthly, Campbell–Thompson procedure, 1927/12–2004/12): after applying the CT truncation (forecasts may not be negative) and sign restrictions, "only csp, eqis, d/p and tbl are positive and significant both IS and OOS, with tms borderline." CEV is computed with γ=3 and a 150% weight cap; benchmark CEVs are: historical-mean timer 82bp, market 79bp, risk-free 40bp (monthly). eqis OOS R² (after truncation) +0.30%, tbl +0.25%, tms +0.21%*. In the same table, variables that are significant IS but still have negative OOS R² after truncation are cay3, ntis, e10/p, b/m, e/p, d/y and dfy — GW 2008 Table 3. The unit of the CEV column in the table (% per month or otherwise) could not be verified column by column in this round.
GWZ 2024 (RFS 37(11):3490–3557, verified against the original, Open Access)
- Scope: 29 variables from 26 post-2008 top-journal papers, plus GW's original 17, for 46 in total. The sample is extended to December 2021, usually about 10 years more data than the original papers — Goyal, Welch & Zafirov 2024, doi:10.1093/rfs/hhae044
- With data extended through 2021, 13 new variables lose IS predictive power. Of the 29 variables, only 9 (31%) see their t-statistics improve with more data; 69% deteriorate. Simulations (IA.I) show this performance "is more consistent with a scenario in which these 29 variables were spurious from the start" — GWZ 2024 p.3491–3492
- The higher the original t threshold, the higher the share that improves after extension: 36% at t≥2.0, 39% at 2.5, 40% at 3.0, 43% at 3.5; at 4.0, gip and gpce improve and vrp does not. This gives "modest" support to the Harvey–Liu–Zhu high-threshold recommendation — GWZ 2024 p.3492
- Under the "homologous" specification (uniformly forecasting non-overlapping S&P 500 log excess returns at each variable's native frequency), another 3 lose IS significance, leaving 10. Of those, 4 also have good OOSCT (OOS R² with forecasts forced to be non-negative): tchi (the 14 technical indicators of Neely et al. 2014), shtint (the short-interest ratio of Rapach, Ringgenberg & Zhou 2016), accrul (Hirshleifer, Hou & Teoh 2009) and gpce (fourth-quarter consumption growth, Møller & Rangvid 2015). Among the old pre-GW variables, 3 more perform well: tby (Campbell 1987), i/k (Cochrane 1991) and eqis (Baker & Wurgler 2000). If the sample is only extended forward and not backward, ogap, tail (Kelly & Jiang 2014) and avgcor (Pollet & Wilson 2010) also perform better — GWZ 2024 p.3492
- Final list (original text of the conclusion section): "Based on data through 2021, of the 46 variables, the 11 we would not dismiss outright are: tchi, shtint, tby, lty, tms; pce, crdstd; i/k; accrul, gpce, eqis." The strongest post-WWII is gpce; "inference on the remaining 35 variables is underwhelming" — GWZ 2024 p.3549
- By frequency (conclusion section):
- Monthly: the 20 new monthly variables "all" lose significance under the homologous IS specification; only tchi and avgcor retain joint IS-OOSCT significance. The best monthly predictors are tby and lty, but neither beats being fully in stocks in any of the four simple strategies, and tby even loses in absolute terms in all four strategies.
- Quarterly: pce and i/k are good IS; i/k and crdstd are good OOSCT. i/k predicted poorly in 1975–1998 and consistently well after 1998, but its IS coefficient fell from −3.74 in the first half to −1.66 in the second half, and all four timing strategies lose to being fully in stocks (two lose in absolute terms).
- Annual: accrul, cfacc, gpce and eqis are good both IS and OOSCT. The "stock-tilted Z-scaled" strategies of accrul and gpce beat being fully in stocks, with t≈1.3, 1.5 and Sharpe about 0.2.
- Source: GWZ 2024 p.3547–3549
- Table 3 figures (homologous IS and OOSCT, monthly R² in %; extracted with pdftotext raw mode, check column alignment against the original PDF):
- Monthly: tchi IS R² 0.51* / OOSCT 0.48**; shtint 0.67* / 1.26**; tby 0.29* / 0.37; lty 0.19 / 0.25; tms 0.11 / 0.02; d/p 0.10 / −0.06; e/p 0.26 / −0.64; infl 0.15 / 0.10*; Martin rsvix 0.12 / −3.27; Kelly–Pruitt fbm 3.16 / −1.50.
- Quarterly: crdstd 4.56 / 4.54**.
- Annual (calendar year): accrul 8.25** / 4.40*; cfacc 11.19** / 16.35*; gpce 12.48* / 5.43; eqis 8.17 / 1.49*; gip 0.02 / −2.05.
- Annual (6-month publication lag): accrul 10.76** / 6.31**; gpce 7.50* / 8.33**; eqis 4.05 / −6.99.
- Source: GWZ 2024 Table 3
- accrul is a "one-trick pony": its performance "comes entirely from one event," namely the dot-com bubble and crash around 2000; after 2002 it barely moves. Since the 1970s gpce has missed only once, failing to call the bear market of the 2008 Great Recession — GWZ 2024 p.3493, p.3549
- Combination forecast (smart consensus, weighted by t-statistic): OOSCT performance is good, but the good performance is concentrated from the mid-1940s to the mid-1970s, and "is largely unremarkable afterwards"; no simple strategy beats being fully in stocks either — GWZ 2024 p.3493
- Investment strategies: four strategies (75/25 stock-tilted or 50/50; $1 fixed or Z-scaled) give 184 in total, and only 3 beat being fully in stocks (accrul, gpce, gip, t=1.31, 1.54, 1.35). Another 16% beat it without significance, and 84% trail being fully in stocks; about one third of the laggards even lose in absolute terms — GWZ 2024 p.3493
- CEV (γ=5, as CT recommend): no variable statistically reliably beats either being fully in stocks or a 20% stock strategy. crdstd and shtint perform well in economic terms, but the authors note themselves that "among 46 variables, such a result is not surprising" — GWZ 2024 p.3493
- The new post-2000 sample includes three recessions (2001, 2008, 2020) and two major bear markets (2000–2002, 2008), which the authors regard as a "fascinating new sample" for testing new variables. VIX-type predictors "have performed extremely well in the last 10 years" — GWZ 2024 p.3491, p.3493
- The working-paper version (SSRN 3929119, 2021) is more pessimistic: "most variables have already lost their empirical support," overall "disappointing" — Welch personal page card-eqpredict; SSRN 3929119
- Data source: GW data are on Amit Goyal's website (formerly http://www.hec.unil.ch/agoyal/); Löffler 2021 mentions that the 2019 version of the files differs from older versions — Löffler 2021
Inferences
- For monthly S&P 500 timing, the GWZ 2024 survivors (tchi, shtint, tby, lty) all have OOS R² of only 0.25%–1.26%, and none of them lets a simple strategy beat being fully in stocks. "Significant OOS R²" and "a strategy that beats buy-and-hold" are two different things, and this is an explicit conclusion in GWZ.
- Annual variables (gpce, accrul) are stronger, but the effective sample at annual frequency is very small, and accrul's performance essentially rests on a single event.
- NRTZ's technical indicators (tchi) are among the few "new" variables still counted as survivors in the GWZ extended sample, consistent with the conclusion of Baetje & Menkhoff (see Q6).
Gaps
- The i/k and pce rows of GWZ Table 3 were missing or misaligned in extraction; figures not verified.
- The per-variable CEV figures in GWZ Table 5 were not extracted one by one.
- The GWZ Internet Appendix (longer-horizon forecasts, IA.VI) was not read.
- The unit of the CEV column in GW 2008 Table 3 could not be confirmed.
Q2 Campbell & Thompson 2008: economic restrictions and "small R², large utility"
Takeaway
CT 2008 argue that after imposing two restrictions, "the forecast premium may not be negative" and "coefficient signs must match theory," the monthly OOS R² of most variables turns slightly positive. Because monthly returns are extremely noisy, even an R² of only about 0.5% has considerable utility value for a mean–variance investor. This OOS R² definition and the CER / Sharpe conversion formulas became standard tools in the subsequent literature.
Cited Findings
- OOS R² definition: R²_OS = 1 − Σ(r_t − r̂_t)² / Σ(r_t − r̄_t)², where r̄ is the prevailing historical mean available at the time. GWZ follow this and call the truncated version "OOSCT R²" — GWZ 2024 around p.3496
- CT formula (as cited by GKX 2020): the Sharpe ratio of an active investor using forecast information is SR* = √[(SR² + R²) / (1 − R²)], where SR is the buy-and-hold Sharpe. GKX use this to compute that the S&P 500 buy-and-hold Sharpe of 0.51 over 1987–2016 can be raised to 0.71 with NN3's R² — Gu, Kelly & Xiu 2020 NBER WP version; original Campbell & Thompson 2008, RFS 21(4):1509–1531, doi:10.1093/rfs/hhm055
- GW 2008 respond to CT: truncation is a "very effective constraint," e.g. d/y is truncated to 0 in 54.2% of months and csp in 44.7%. But after truncation, 7 of the 11 IS-significant models still have negative OOS R² — GW 2008 Table 3 and text
- GWZ 2024 mention that CT argue full-sample IS tests should be favored when the model is stable and known; GWZ disagree with that premise and cite Martin & Nagel 2022 in support of OOS testing — GWZ 2024 p.3492–3493
- The per-variable restricted monthly OOS R² figures in the CT original, and exact values such as "the annualized utility corresponding to a monthly R² of 0.5%": insufficient evidence / not verified (the CT original PDF was not obtained in this round).
Inferences
- The "small R², large utility" calculation assumes the investor can adjust weights precisely according to the forecast, faces no transaction costs, can use leverage, and that the benchmark is "an investor adjusting weights using the historical mean," not buy-and-hold. Löffler 2021 shows that once the benchmark is switched to buy-and-hold, most of the utility advantage disappears (see Q8).
Gaps
- The per-variable figures in CT 2008 Table 2 / 3 were not verified.
Q3 Forecast combination and dimension reduction: RSZ 2010, NRTZ 2014, Kelly–Pruitt, DLRZ 2022
Takeaway
Combination forecasts (RSZ 2010) and "macro + technical indicator PCA" (NRTZ 2014) both show significantly positive OOS R² and CER gains of 2–5% per year in the original papers, but the gains are highly concentrated in NBER recessions. Once the benchmark is switched to buy-and-hold, RSZ's economic gains fall to almost zero (Löffler 2021). In GWZ 2024, the t-statistic-weighted combination forecast is also mediocre after the mid-1970s.
Cited Findings
Rapach, Strauss & Zhou 2010 (RFS 23(2):821–862)
- Design: quarterly S&P 500 excess returns, sample 1947:1–2005:4, OOS starting 1965:1. Univariate regressions on each of 15 GW predictors, then combined (mean, median, trimmed mean, DMSPE) — Löffler 2021 description of RSZ; RSZ 2010, doi:10.1093/rfs/hhp063
- Claim: model uncertainty and instability undermine single models; combination reduces forecast volatility and is linked to the real economy, giving "consistent" OOS gains relative to the historical mean — RSZ 2010 abstract (RePEc)
- The most accurate is the arithmetic mean of the 15 forecasts; the largest utility gain comes from the DMSPE (θ=0.9) combination — Löffler 2021
- Löffler replication values (relative to a historical-mean benchmark, γ=3, weights between about 0.70–1.50): utility gains of 2.39% for the mean combination and 2.64% for DMSPE (annualized); Sharpe differences of 0.090 and 0.102. With a buy-and-hold benchmark instead, the utility gains fall to −0.14% and 0.11%, Sharpe differences to 0.002 and 0.014, and most p-values are > 85% — Löffler 2021, Critical Finance Review
- The often-cited "mean combination quarterly R²_OS = 3.58%": insufficient evidence / not verified (the RSZ original tables were not obtained in this round).
- Subsequent status: NRTZ 2014 apply the same method as a mean combination of 14 macro variables, with a monthly OOS R² of 1.16% (1966:01–2011:12) — NRTZ WP 2010-008 footnote 29. A master's thesis at the University of Liège re-tested with post-2010 data and found the link to the real economy weaker than before 2010 (student thesis, low evidence level) — matheo.uliege.be
Neely, Rapach, Tu & Zhou 2014 (MS 60(7):1772–1791; figures below are from the revised St. Louis Fed WP 2010-008, verified against the original; the published version may differ slightly)
- Data: 14 macro variables (GW) and 14 technical indicators (MA(1,9)…MA(3,12), momentum MOM(9)/MOM(12), OBV volume moving averages), sample 1950:12–2011:12, OOS period 1966:01–2011:12 (552 months), monthly, forecasting S&P 500 log excess returns — NRTZ WP
- Table 4 (OOS R², monthly):
- PC-ECON is negative, but significant at the 1% level by Clark–West (MSFE-adjusted).
- PC-TECH 0.65%, significant.
- PC-ALL 1.79% (MSFE-adj 3.28, significant at 1%); expansions −2.80%, recessions 11.24%.
- Simple-mean combination of technical indicators 0.53% (MSFE-adj 1.38, significant at 10%); macro mean combination 1.16%; sum-of-the-parts 1.32%.
- After controlling for data mining with the Clark–McCracken (2012) reality check, PC-ALL remains significant.
- Source: NRTZ WP Table 4 and p.21–23
- Table 5 (asset allocation): mean–variance investor with γ=5, weights restricted to [0, 1.5], variance estimated with a 5-year rolling window; the historical-mean benchmark has a CER of 3.54% and monthly turnover of 2.09%.
- PC-ALL: CER gain 4.94% (expansions 0.70%, recessions 26.21%); monthly Sharpe 0.16 (about 0.55 annualized); relative turnover 7.51 times; CER gain after a 50bp one-way cost 4.12%.
- PC-ECON: 2.24% (recessions 12.69%, expansions 0.09%), monthly Sharpe 0.10, 1.51% after costs.
- PC-TECH: 2.49% (recessions 16.37%, expansions −0.29%), monthly Sharpe 0.10, > 2% after costs.
- Individual technical indicators have CER gains of up to 3.17%, up to 2.82% after costs, all positive. Among macro variables, NTIS, LTR, DFR and INFL turn negative after costs because of high turnover.
- Source: NRTZ WP Table 5 and p.24–26; published version doi:10.1287/mnsc.2013.1838
- Mechanism: technical indicators better capture the "decline" in the premium near business-cycle peaks, while macro variables better capture the "rise" in the premium near troughs — NRTZ (SMU abstract page)
- Buncic (St. Gallen WP) improves on NRTZ, with OOS R² up to 1.75% and CER gains up to 3.35% higher than NRTZ's ALL factor (relative value) — Buncic, St. Gallen EWP 1522 (only the abstract was seen, full text not read)
Kelly & Pruitt 2013 JF / 2015 JoE (three-pass regression filter / PLS)
- GWZ 2024 include KP's fbm (a factor extracted from cross-sectional B/M): homologous IS R² 3.16%, but OOSCT R² −1.50%, a failure case of "significant IS but negative OOS" — GWZ 2024 Table 3; the working-paper figure on Welch's page is −2.21% — ivo-welch.org
- As paraphrased by GKX 2020: the PLS OOS R² for market returns in the KP original is about 1% per month — GKX NBER WP p.40; originals Kelly & Pruitt 2013, JF 68(5), doi:10.1111/jofi.12060; Kelly & Pruitt 2015, JoE 186(2), doi:10.1016/j.jeconom.2015.02.011
- Table numbers and exact sample periods in the KP originals: insufficient evidence / not verified.
Dong, Li, Rapach & Zhou 2022 JF "Anomalies and the Expected Market Return"
- Forecasts market returns with about 100 cross-sectional anomaly return portfolios (combination / shrinkage methods). Exact OOS R², CER and table numbers: insufficient evidence / not verified (the original was not obtained in this round) — DLRZ 2022, JF 77(1):639–681, doi:10.1111/jofi.13099
- No independent replication seen; GWZ 2024 do not include this variable.
Rapach & Zhou surveys
- Rapach & Zhou 2013 Handbook of Economic Forecasting Vol.2A Ch.6 "Forecasting Stock Returns," and Rapach & Zhou 2022 Oxford Research Encyclopedia: full texts not obtained in this round; content insufficient evidence / not verified — Handbook chapter doi:10.1016/B978-0-444-53683-9.00006-2
Inferences
- Of PC-ALL's 4.94% CER gain, expansions contribute only 0.70%. In effect, nearly all the value comes from a small number of recession months, equivalent to a bet on "avoiding bear markets." NBER recession dates are announced after the fact, but the strategy itself uses only real-time data, so this is not look-ahead bias; the problem is that recession samples are few and statistically fragile.
- The CER gains of RSZ and NRTZ both use "an investor adjusting weights with the historical mean" as the benchmark, and that benchmark's own CER is only 3.54%, clearly below being fully in stocks over the same period, so the gains are inflated.
Gaps
- The exact R²_OS and CER in RSZ 2010 original Table 1 / Table 3 were not verified.
- Whether the figures in the published NRTZ version match the WP version was not verified.
- The DLRZ 2022 figures were not verified.
Q4 Machine-learning timing: GKX 2020, KMZ 2024 and its critique, Avramov et al.
Takeaway
GKX 2020's "bottom-up" neural networks have a monthly OOS R² of about 1.1–1.8% for the S&P 500, and the CT timing strategy raises the Sharpe from 0.51 to 0.77 (1987–2016). KMZ 2024 claim that high-complexity RFF models raise timing Sharpe by about 0.47 even when OOS R² is negative. Nagel 2025 shows that KMZ's short-window strategy is essentially a "volatility-scaled momentum strategy" and that the model has not learned anything from the data. Avramov et al. 2023 show that ML stock-selection profits shrink sharply under economic restrictions (cross-sectional evidence).
Cited Findings
Gu, Kelly & Xiu 2020 (RFS 33(5):2223–2273; figures below from NBER WP 25398, verified against the original)
- Data: CRSP individual stocks 1957/03–2016/12 (about 30,000 stocks), 30-year OOS test (1987–2016). The S&P 500 is aggregated bottom-up from individual stock forecasts (the self-built S&P 500 has a correlation > 0.99 with the actual index) — GKX NBER WP
- Table 5 (S&P 500 monthly OOS R²): OLS-3+H −0.22%; regularized linear methods "cannot beat a constant forecast"; GLM 0.71%; NN1–NN5 about 1.08 / 1.13 / 1.80 / 1.63 / 1.17% (NN3 best) — GKX Table 5 and p.39–40 (the table extraction is misaligned; for the NN series rely on the text's "0.71% GLM" and "1.80% NN3")
- Table 6 (CT timing Sharpe gains, maximum leverage 50%, no short sales): NN3 timing of the S&P 500 has an annualized Sharpe of 0.77 versus 0.51 for buy-and-hold (+0.26); NN1–NN5 gains are about 0.22 / 0.20 / 0.26 / 0.22 / 0.19 — GKX Table 6 and p.40–41; published version doi:10.1093/rfs/hhaa009
- Transaction costs, maximum drawdown and hit rate are not reported: insufficient evidence (not seen in the original).
- Critique: Avramov, Cheng & Metzker 2023 point out in the cross-section that ML signal profits come from hard-to-arbitrage stocks (microcaps, unrated or distressed firms, high limits to arbitrage) and high turnover. After excluding these stocks or including costs, profits fall sharply — Avramov, Cheng & Metzker, MS 69(5), doi:10.1287/mnsc.2022.4449. Specific figures could not be verified from the original in this round; moreover, this is a cross-sectional stock-selection result, not index timing, and cannot be applied directly to S&P 500 timing.
Kelly, Malamud & Zhou 2024 (JF 79(1):459–503, verified against the original)
- Design: forecasts the CRSP VW index using 15 variables (GW predictors plus lagged returns), expanded with Random Fourier Features to as many as P=12,000, with rolling training windows of T=12 / 60 / 120 months; both returns and predictors are volatility-standardized. The analysis sample starts in 1930 (1926–2020, with the first 36 months used for standardization) — KMZ 2024, doi:10.1111/jofi.13298
- Main result: the OOS timing Sharpe (relative to buy-and-hold) improves by "about 0.47 per year, with a t-statistic close to 3.0," while "the vast majority of models have significantly negative OOS R²." The authors argue from this that R² is unsuitable for evaluating the economic value of ML — KMZ 2024 introduction
- Table I: the linear model with z=10³ has OOS R² −3.8%, Sharpe 0.46 (t=4.4), IR versus the market 0.33 (t=3.1). "Nonlinear" (c=1,000, z=10³) has OOS R² 1% per month, Sharpe 0.46, IR versus the market 0.31, and IR versus the best linear model 0.26 (t=2.5). At T=120 the linear model's Sharpe is 0.49, but its alpha versus the market is not significant — KMZ 2024 Table I and p.495–496
- At T=60 / 120 the IR of high-complexity models is about 0.25, with t > 2.0 — KMZ 2024 Fig. 9
Nagel 2025 "Seemingly Virtuous Complexity in Return Prediction" (NBER w34104 / BFI WP 2025-104, verified against the original)
- Core: when P ≫ T, the ridgeless RFF forecast equals a weighted average of the T training-period returns, with weights determined by the "similarity" of the predictor vectors. In short windows similarity mainly reflects closeness in time, so the forecast degenerates into a "weighted average of recent returns," i.e. momentum. And because similarity falls as predictor volatility rises, the result is a volatility-scaled momentum strategy. The good performance comes from "this strategy happening to work in historical data," not from the model extracting a signal — Nagel 2025 abstract; NBER w34104
- Empirics: a simple strategy (linearly declining weights on the past 12 months of returns, divided by predictor volatility) produces positions highly similar to the RFF strategy with comparable OOS performance, and when included as a factor it "absorbs most of the RFF abnormal return." Both reduce exposure before recessions, because the strategy systematically de-risks when predictor volatility rises, not because of foresight — Nagel 2025 p.2–3
- Placebo-data test: adding an MA(2) component with strong negative autocorrelation to actual returns makes returns show reversal. RFF still produces the same positive momentum weights, and the OOS abnormal return turns negative, showing the model does not learn momentum or reversal from the data — Nagel 2025
- Theoretical calibration: under KMZ's own numerical settings, with T=12 and very large P the achievable Sharpe is only 0.005 (0.017 annualized), and about 0.06 annualized at P=1,200, far below KMZ's empirical values — Nagel 2025 p.~40
- Also cites Berk (2023) questioning how KMZ report Sharpe ratios — Nagel 2025 p.~6
- Rebuttal: the AQR / Kelly side has "Understanding the Virtue of Complexity" (2025-07), arguing these challenges have "little bearing" on KMZ's theory and empirics, and proposing a distinction between nominal and effective complexity (an interested party) — AQR working paper
- Extension: Federal Reserve FEDS 2025-089 (Kılıç) applies the same framework to exchange rates; the average OOS R² across currencies is ≤ 0 over the entire regularization range — FEDS 2025-089 (only the search snippet was seen)
Other ML literature: Buncic 2024 / 2025 replications of KMZ, Cartea et al., Chinco, Clark-Joseph & Ye 2019 (JF, LASSO on high-frequency individual stocks), Rossi 2018 (boosted trees forecasting market returns), Bianchi, Büchner & Tamoni 2021 RFS (bond excess returns), Leippold et al. (Chinese market): originals not obtained in this round; all figures insufficient evidence / not verified. Of these, Chinco et al. and Bianchi et al. are not US equity index timing and are only indirectly related to this topic.
Inferences
- GKX's S&P 500 result is an aggregation of bottom-up individual stock forecasts, requiring firm characteristic data for the whole market (94 characteristics × thousands of stocks), which retail investors can almost never reproduce; and it does not account for transaction costs.
- Under Nagel's reading, KMZ's "negative R² but high Sharpe" amounts to a "volatility-scaled momentum" that can be replicated with a few lines of rules. For practice, this actually implies that ML is not needed and simple trend-plus-volatility rules capture most of the effect; but its effectiveness is likewise only historical experience, not the result of model learning.
Gaps
- Whether the table numbers and figures in the published RFS version of GKX match the WP was not verified.
- The specific figures of Avramov et al. and Buncic's replication results for KMZ were not verified.
- In this round, no independent (non-original-author) OOS replication of S&P 500 timing with any ML method was seen.
Q5 Data mining, multiple testing and statistical pitfalls; local predictability
Takeaway
Highly persistent predictors (Stambaugh bias, spurious regression), overlapping long-horizon returns and large-scale variable searches (the HLZ threshold) all overstate IS significance. GWZ 2024 find that variables with higher original t-statistics are more likely to improve in the extended sample, mild support for the high-threshold recommendation. The "pockets of predictability" of Farmer–Schmidt–Timmermann 2023 were shown by a subsequent JF replication to suffer from look-ahead bias (use of a two-sided kernel); after correction, most conclusions do not hold.
Cited Findings
- GWZ cite Harvey, Liu & Zhu 2016 and provide "modest evidence": variables passing higher thresholds (t≥3.0 and above) have a higher improvement share after extension (40–43% versus 31%) — GWZ 2024 p.3491–3492; HLZ 2016, RFS 29(1), doi:10.1093/rfs/hhv059
- GWZ simulations show that IS performance in the extended sample "is more consistent with a scenario in which all are spurious" — GWZ 2024 IA.I summary p.3491
- NRTZ discuss Ferson, Sarkissian & Simin 2003: test size distortion from Stambaugh bias worsens data mining; NRTZ address it with a wild bootstrap and consider the FSS concern "not too severe" for their OOS analysis — NRTZ WP footnote 30; FSS 2003, JF 58(4), doi:10.1111/1540-6261.00579 (DOI not verified)
- GW 2008 Table 2 (5-year returns) shows high R² for i/k, tms and others. Overlapping long-horizon returns artificially inflate R², which is the argument of Boudoukh, Richardson & Whitelaw 2008, "The Myth of Long-Horizon Predictability" — BRW 2008, RFS 21(4):1577–1605, doi:10.1093/rfs/hhl042 (argument based on general knowledge of the literature, figures not verified)
- After GWZ's homologous specification switches to "non-overlapping, native frequency," several variables lose significance; the authors explain in a footnote that this is mainly because overlapping returns are no longer used — GWZ 2024 footnote 1
- Farmer, Schmidt & Timmermann 2023 JF "Pockets of Predictability": using a time-varying-parameter (kernel) method, they find US equity predictability appears only in short "pockets," separated by long unpredictable stretches, inconsistent with affine pricing models and more consistent with "investors' incomplete learning about persistent cash-flow growth" — Alpha Architect summary; CEPR DP 12885
- Replication of FST: Cakici, Fieberg, Neumaier, Poddig & Zaremba, "Pockets of Predictability: A Replication" (JF), point out that the original authors used a two-sided kernel in in-sample estimation, letting future information leak into the forecasting model; switching to a one-sided kernel "qualitatively alters the findings, invalidating most conclusions" — Replication paper page (ICM open access) (only the abstract was seen; specific figures not verified)
- Bossaerts & Hillion 1999, Inoue & Kilian 2005 (comparing IS and OOS test power, arguing OOS tests have lower power), Stambaugh 1999, Cochrane 2008 "The Dog That Did Not Bark" (inferring return predictability from the unpredictability of dividend growth), Pettenuzzo, Timmermann & Valkanov 2014 JFE (constraining forecasts with economic restrictions such as Sharpe bounds), Dangl & Halling 2012 JFE (time-varying-coefficient Bayesian model averaging), Henkel, Martin & Nardari 2011 JFE (predictability concentrated in recessions), Timmermann 2008 IJF "Elusive return predictability" (local, short-lived predictability): originals not obtained in this round; all specific figures insufficient evidence / not verified. Reference DOIs:
- Cochrane 2008, doi:10.1093/rfs/hhm046
- Stambaugh 1999, doi:10.1016/S0304-405X(99)00041-0
- Timmermann 2008, doi:10.1016/j.ijforecast.2008.01.008
- Henkel, Martin & Nardari 2011, doi:10.1016/j.jfineco.2010.09.008
- Dangl & Halling 2012, doi:10.1016/j.jfineco.2012.04.003
- Bossaerts & Hillion 1999, doi:10.1093/rfs/12.2.405
- Inoue & Kilian 2005, doi:10.1081/ETC-200040785
- Pettenuzzo, Timmermann & Valkanov 2014 JFE (DOI not verified; please check against JFE 114(3))
- The DOIs above are listed from general knowledge of the literature; clicking through to confirm them before writing is recommended.
Inferences
- The narrative that "predictability exists only in local stretches" (FST, Timmermann 2008) is itself consistent with the "recession concentration" of NRTZ / HMN. But FST's specific evidence has been shown to have look-ahead bias, and any citation must also note the replication result.
Gaps
- The volume, issue, DOI and corrected figures of the Cakici et al. replication paper were not verified.
- The OOS R² / CER figures of PTV 2014 and Dangl–Halling 2012 were not verified.
Q6 Independent replication: which methods survive?
Takeaway
Technical indicators (NRTZ's tchi) count as survivors in two independent re-examinations: GWZ 2024 list them among the 11 variables that should not be dismissed, and Baetje & Menkhoff consider them relatively stable over 1950–2013. Traditional macro / valuation variables mostly failed after the 1970s. The economic gains of RSZ combination forecasts and the RRZ short-interest indicator are not significant when Löffler 2021 retests them against a buy-and-hold benchmark. KP's PLS factor is negative OOS in GWZ.
Cited Findings
- Survival or failure overview (GWZ 2024):
- Survive: tchi, shtint, tby, lty, tms, pce, crdstd, i/k, accrul, gpce, eqis.
- Significant IS but negative OOS (working-paper version): KP fbm, HJTZ sntm (investor sentiment), AMP (pce in some specifications).
- BW, related to Baker–Wurgler, disqualified for instability (working-paper version wording).
- Source: GWZ 2024; Welch page (WP version)
- Baetje & Menkhoff (WP 2015-01; the published version was not verified in this round, usually cited as 2016): the "relative stability" of technical indicators over 1950–2013 is key to their economic value, and results "improve slightly" over time; macro variables "were effective only before the 1970s and have essentially lost predictive power since, even when the 2008 crisis is included." Entering at any point after the mid-1960s, technical-indicator strategies have an average Sharpe of about 0.6 (before transaction costs). Structural breaks for macro variables are mostly in the 1990s, and for interest-rate variables in the 1970s; profits of technical trading rules weakened after the late 1990s (the paper cites Park & Irwin 2007) — Baetje & Menkhoff WP
- Löffler 2021 (Critical Finance Review): across a large number of hypothetical sample periods, buy-and-hold beats the "historical-mean weight adjustment" benchmark in utility in 84.1% of samples and has a higher Sharpe in 92.2% of samples; at γ=5 it still exceeds three quarters. The replication of RRZ 2016 (short-interest indicator SII, OOS 1990:01–2014:12) matches exactly: relative to the historical-mean benchmark, all indicators are significant at the 5% level; relative to buy-and-hold, none of the indicators is significant, p > 20% — Löffler 2021
- Replication studies such as Zhang, Ma & Liao and Lima & Meng: not obtained in this round; insufficient evidence / not verified.
Inferences
- The most relevant message for retail investors: the only monthly signal category that holds up in several independent tests is price trend / momentum / moving-average technical indicators, and Nagel's dissection of KMZ also lands on "volatility-scaled momentum." But GWZ's four simple strategies still show that tchi cannot statistically beat being fully in stocks.
Gaps
- The journal, volume and issue of the published Baetje & Menkhoff paper were not verified.
- Replication papers such as Zhang–Ma–Liao and Lima–Meng could not be obtained.
Q7 When predictability appears: recession concentration and failure periods
Takeaway
OOS predictability and economic value are highly concentrated in NBER recessions: NRTZ's PC-ALL has a recession R² of 11.24% and an expansion R² of −2.80%; its CER gain is 26.21% in recessions and 0.70% in expansions. GW 2008 find many variables' significance comes from the 1973–75 oil shock; GWZ 2024 find accrul's performance rests only on 2000–2002; GWZ's combination forecast works only in the 1940s–1970s.
Cited Findings
- NRTZ: both IS and OOS predictability are "substantially higher for recessions"; the predictive power of technical indicators is "highly concentrated during recessions" — NRTZ WP Tables 2, 4, 5
- Technical indicators have recession CER gains > 10% (annualized) for 10 / 14 individual indicators — NRTZ WP p.25
- GW 2008: significance often comes "exclusively" from the oil-shock years; models predict poorly "late in the sample" — GW 2008
- GWZ: accrul relies on the dot-com episode alone; gpce missed 2008; i/k failed in 1975–1998 and improved afterwards; the combination forecast is flat after the mid-1970s — GWZ 2024
- Baetje–Menkhoff: macro variables failed after the 1970s; technical-rule profits weakened after the late 1990s — Baetje & Menkhoff WP
- Nagel: both RFF and the volatility-momentum strategy reduce exposure before recessions, because rising volatility leads to de-risking, not foresight — Nagel 2025
- Specific figures of Henkel, Martin & Nardari 2011 JFE: insufficient evidence / not verified — doi:10.1016/j.jfineco.2010.09.008
Inferences
- In long expansions (e.g. 2010–2019), these methods should be expected to trail buy-and-hold, because the expansion R² is negative and the strategy often reduces exposure when forecasts are revised down. This is consistent with Löffler's buy-and-hold advantage.
Gaps
- After 2012 (outside the NRTZ sample), apart from GWZ's test of tchi, no independent figures on PC-ALL's real extended performance were seen.
Q8 Transaction costs, leverage limits and the buy-and-hold benchmark
Takeaway
Most papers' CER is "before costs" and "relative to a historical-mean investor," with weights restricted to [0, 1.5] (allowing 50% leverage, no short sales). NRTZ are among the few that report after-cost figures (50bp one-way, PC-ALL still has 4.12%). But with a buy-and-hold benchmark, RSZ's gains go to zero (Löffler), and 84% of GWZ's simple strategies trail being fully in stocks.
Cited Findings
- Weight restriction [0, 1.5]: NRTZ footnote 31 ("precluding short sales and preventing more than 50% leverage"); GKX's CT strategy likewise has maximum leverage of 50% and no short sales; GW 2008 Table 3 also caps weights at 150% — NRTZ WP; GKX WP; GW 2008
- NRTZ after-cost CER gains: PC-ALL 4.12%, PC-ECON 1.51%, PC-TECH > 2%, individual technical indicators up to 2.82% (50bp one-way cost) — NRTZ WP Table 5
- Löffler: RSZ utility gains relative to buy-and-hold are −0.14% / 0.11%; RRZ not significant (p > 20%) — Löffler 2021
- GWZ: only 3 of 184 strategies significantly beat being fully in stocks (t about 1.3–1.5) — GWZ 2024
- GWZ footnote 2: making only "very tiny" allocation adjustments might beat being fully in stocks, but that is not an economically meaningful strategy — GWZ 2024
- Taxes: none of the papers read address taxes (insufficient evidence).
- Maximum drawdown and hit rate: among the papers read, only KMZ mention improvements in maximum loss and skewness (qualitative statement in the original). NRTZ and GKX do not report maximum drawdown; insufficient evidence / not verified.
Inferences
- In a retail ETF implementation, the 50% leverage cap requires margin or leveraged ETFs, which adds cost. If restricted to [0,1] (switching only between stocks and cash), CER gains should be lower, but no [0,1] version of the figures was seen in the original papers in this round.
Gaps
- Performance of NRTZ and RSZ under a [0,1] weight restriction was not seen reported.
- No data on after-tax performance.
Q9 Turning points vs average returns; application to Nasdaq / QQQ; retail feasibility
Takeaway
Almost all of this literature forecasts "conditional mean excess returns" (regression-style point forecasts), evaluated with OOS R² and CER, rather than forecasting turning points. So-called "avoiding bear markets" is a by-product of mean forecasts being revised down during recessions. This round found no top-journal OOS study applying these methods to the Nasdaq-100 / QQQ. Retail investors can obtain the GW data, but technical-indicator methods are the easiest to implement.
Cited Findings
- Evaluation metrics: GW, GWZ, CT, RSZ, NRTZ, GKX and KMZ all evaluate mean return forecasts with OOS R² (MSFE), CER and Sharpe — GWZ 2024; NRTZ WP; GKX WP
- NRTZ's description of turning points is qualitative: technical indicators detect the decline in the premium near business-cycle peaks, and macro variables detect the rise in the premium near troughs — NRTZ
- Both KMZ's high-complexity strategy and Nagel's volatility-momentum strategy "reduce exposure before recessions" — Nagel 2025; KMZ 2024
- Nasdaq-100 / QQQ: all papers read use the S&P 500 or CRSP VW as the target. GKX's 25 characteristic-sorted portfolios and factor portfolios do not include the Nasdaq-100 — GKX WP. This round found no combination-forecast or ML timing OOS study targeting the Nasdaq-100; insufficient evidence.
- Data availability: the original GW data are public on Goyal's website, and the GWZ data are public as RFS supplementary data — GWZ 2024 (Supplementary data statement); Löffler 2021 data URL mention. Some variables have publication lags, e.g. annual variables such as accrul and gpce; GWZ Table 3 Panel D retests with a 6-month lag, and accrul and gpce still perform well, while eqis turns to OOSCT −6.99% — GWZ 2024 Table 3
Inferences
- Ranking of retail feasibility (inference):
- Technical indicators (MA, momentum) need only price and volume, are the easiest to implement, and are the more stable category in GWZ and Baetje–Menkhoff;
- Interest-rate variables (tby, lty) have public data, but cannot beat being fully in stocks in GWZ;
- Combination or PCA (RSZ, NRTZ) requires updating 14–28 variables every month with recursive estimation; doable but quite tedious, and some GW variables have publication lags and revisions;
- GKX bottom-up ML requires firm characteristic data for the whole market, which is practically infeasible for retail investors;
- KMZ's RFF can be reproduced, but per Nagel's analysis it can be replaced by simple volatility-scaled momentum.
- When applied to QQQ, the Nasdaq-100's volatility and tech concentration are higher than the S&P 500's. The coefficients, CER and cost assumptions in these studies cannot be transplanted directly; any QQQ timing conclusion needs its own OOS test.
Gaps
- Any OOS timing evidence for the Nasdaq-100 (none found in this round).
- Hit rate (directional accuracy) is not reported in any of the papers read.
Further reading
The other four articles in this series:
- Out-of-sample (OOS) evidence on valuation-based timing signals (S&P 500 / US large caps; Nasdaq covered where possible)
- Trend-following and time-series momentum (TSMOM) timing: out-of-sample (OOS) evidence for the S&P 500 and Nasdaq
- Positioning / sentiment and option-implied risk-pricing timing signals: out-of-sample evidence
- Turning-point forecasting, what is special about the Nasdaq-100 / QQQ, and the reality of retail implementation
Related pages on this site:
- VOO returns and technicals (Chinese), QQQ returns and technicals (Chinese), CSPX (Chinese), CNDX (Chinese)
- Custom comparison: VOO vs QQQ one-year returns (Chinese)
- Quantitative Investing in US Stocks: Before Quant Trading, Quantify the Market Regime
- RSI Oversold Crossovers Buy Low, but Rarely at the Bottom: full-history backtest of QQQ / VOO
- Signal Combinations Can Bracket a Bottom, but Cannot Pinpoint the Low
- Leveraged ETF return convergence and the market-support hypothesis: test report
Frequently asked questions
Can machine learning forecast the S&P 500 well enough to time the market?
Gu, Kelly & Xiu (2020) bottom-up neural networks have a monthly OOS R² for the S&P 500 of about 1.1–1.8% (NN3 best at 1.80%), and the CT timing strategy raises the Sharpe from 0.51 to 0.77 (1987–2016). But the original does not report transaction costs, maximum drawdown or hit rate, and it requires firm characteristic data for the whole market, which retail investors can almost never reproduce; this round also found no independent (non-original-author) OOS replication of S&P 500 timing with any ML method.
Does the "virtue of complexity" (Kelly, Malamud & Zhou 2024) really time the market?
KMZ claim that high-complexity RFF models raise timing Sharpe by about 0.47 even when OOS R² is negative. Nagel (2025) points out that in short windows this strategy is essentially a "volatility-scaled momentum strategy" and that the model has not learned anything from the data; on placebo data with a reversal component added, RFF still gives the same momentum weights, and the OOS abnormal return turns negative. The AQR / Kelly side has a separate rebuttal (an interested party).
Can forecast combination beat buy-and-hold?
Löffler (2021) replicates Rapach, Strauss & Zhou (2010): relative to the "historical-mean weight adjustment" benchmark, the annualized utility gains of the mean combination and DMSPE are 2.39% and 2.64%; but against a buy-and-hold benchmark they fall to −0.14% and 0.11%, and most p-values are > 85%.
Which predictors still survive after Goyal-Welch-Zafirov 2024?
GWZ 2024 conclude that 11 of 46 variables should not be dismissed outright: tchi, shtint, tby, lty, tms; pce, crdstd; i/k; accrul, gpce, eqis. But of 184 investment strategies only 3 (accrul, gpce, gip) statistically beat always being fully in stocks, with t-statistics of only 1.31, 1.54, 1.35.
When is market-timing predictability concentrated?
Highly concentrated in NBER recessions. NRTZ's PC-ALL has a recession OOS R² of 11.24% and an expansion figure of −2.80%; its CER gain is 26.21% in recessions and 0.70% in expansions. In long expansions, these methods should be expected to trail buy-and-hold.
Can these methods be applied directly to QQQ?
Not directly. All papers read use the S&P 500 or CRSP VW as the target, and this round found no combination-forecast or ML timing OOS study targeting the Nasdaq-100. The Nasdaq-100's volatility and tech concentration are higher than the S&P 500's, so any QQQ timing conclusion needs its own OOS test.