Lingering Quandaries about System Development (Part 4)

In this series, I’m trying to bring my confusion about System Development principles to the surface through a serial discussion of each and every one.  In http://www.optionfanatic.com/2013/01/23/lingering-quandaries-about-system-development-part-3/, I discussed a case of backtesting multiple tickers.  If a given ETF basket backtests well then I still need to rule out selection bias before feeling confident about trading the basket live.

Selection bias would be present if the specified ETF basket performs well when other baskets perform poorly.  To test this, I could randomly create other ETF baskets and backtest those.  If I identify 200 liquid ETFs, for example, and I want to trade five of them then the number of possible combinations is 2.54 billion.  I could backtest all combinations and calculate the mean and standard deviation (SD) performance statistics.  Ideally, the original basket I backtested should be within 1 SD of the mean performance for all combinations.  If the original backtest is more than 2 SD better than the mean performance for all combinations then perhaps I decide this is an outlier and should not be trusted.

While this seems statistically sound, it does imply that no ETF basket should be significantly better than the rest.

Say what?

Many people believe different tickers to have unique trading personalities!  Institutions, which dominate market action, would be the best explanation why.  If many institutions trading a given ticker follow 50/200-SMA crossovers then I am then likely to see an edge when using a 50/200-SMA crossover trading rule.  To the extent that different tickers are traded by different institutions, different tickers may be more or less influenced by different technical [or other types of] criteria applied by those institutions.  Doesn’t it therefore seem plausible that any given trading system might work well for one ticker (or ETF basket) and not others?  In theory, this system would continue to be effective until the institutions trading that ticker significantly change or until the traders responsible for those institutions’ trades significantly change (e.g. fund managers are replaced).