Does Technical Analysis Work? Here’s Proof! (Part 4)

Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.

I was a bit confused where we left off. Kul continues:

     > It appears as though there may be Alpha reversing filtered technical
     > indicators… We’d need to keeping [sic] rolling this forwards to
     > actually find if this relationship continually holds.

I think he’s basically suggesting we test the worst performers from the training set for outperformance. That is a very interesting idea. I would want to know if the worst training indicators do better on the test set than the best training indicators. This reminds me of the Callan Periodic Table of Investment Returns, which I mentioned in the middle of this post.

     > Obviously adding transaction costs and bid/offer would mean we can’t…
     > capture this but this does give us something to investigate further.

Does he mean we can’t realize any profits from this or just diminished profits? He could have included sample transaction fees to get more clarity on this.*

He then teleports ahead to Bitcoin. Say whaaaaaat? Speaking of transaction fees, though, exactly what vehicle is being used to trade it and what are the usual slippage and commissions to do that? I (and most veteran investors, probably) would be very interested to know since Bitcoin is relatively new.

     > So our train period has a monthly average of 20.4% and our test period
     > has annualised returns of 14.3%…it appears as though there may be
     > some Alpha on all technical indicators for Bitcoin.

That sounds encouraging…

     > Interestingly in our train period we outperform Bitcoin but in the test
     > period Bitcoin outperforms.

If buy-and-hold outperforms, then the indicators have no alpha. Why did he just say otherwise?

     > In order to say with certainty if this relationship holds we’d again
     > need to test again over a longer period of time.

Kul then repeats the backtest for all 12 months of 2018. This extends the backtest by five months since the first six months were the training set and July was the testing set.

     > I think it’s fairly safe to say that the performance of all the
     > indicators decays over time however we do actually outperform
     > buying and holding Bitcoin (although, granted, 2018 was a terrible
     > year for Bitcoin).

I think it’s fairly safe to say we really can’t make any conclusions over such a short period of time where the results are so inconsistent with what we saw before.

Kul concludes:

     > We found… reversing filtered indicators may have Alpha for non-
     > Bitcoin instruments and for Bitcoin… our regular indicators
     > may have… Alpha although it does severely decay over time.

Indicator performance declined over the course of these several months, which is still a short time interval. I wouldn’t generalize to “over time,” which sounds much more substantial.

     > We’d need to test on a much larger data set to see if these
     > relationships do actually hold.

Kul catches himself here and I totally agree. Indeed, the biggest critique I have of this article is the limited backtesting interval. Although he uses a 5-minute time frame, the total study period is one year or less. In case we are looking at a large sample size, Kul could have boosted credibility by reporting number of trades in each group, which he never mentions.

In the final analysis, I can’t help but respond to Kul’s title with “Where’s the beef?”

I will continue next time.

* — I feel strongly about including transaction fees in backtesting as discussed in paragraphs 2-3 here.

Does Technical Analysis Work? Here’s Proof! (Part 3)

Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.

I left off at the point where I think Kul’s article gets really special:

     > Now if by some miracle this does work, just to prove it was all one
     > big fluke, we should be able to roll forwards another 3 months to
     > produce positive P&L again.

The main takeaway from this article is right here. It’s one thing to train a model, which by definition is going to demonstrate good performance, and then follow through with more good performance. I became disillusioned when I was unable to accomplish this repeatedly, which is basically what walk-forward optimizmation does. I then became disillusioned again when I incorporated one additional incubation period as Kul mentions here. I wrote about OOS2 in the third bullet point of this post.

     > Note… the average across all these indicators across all instruments
     > is 0.095% per month so I think it’s reasonable to deduce that the
     > indicators used by themselves without any filtering have pretty much
     > zero Alpha…
     >
     > Now… we want to… run [the winners]… for 1 month forward… the
     > average of these is -1.92% for this 1 month period so if anything we
     > might be able to deduce that filtering positive indicators is actually
     > mean-reverting. Annualised performance here would be 23.0%.

I’m guessing he meant negative 23.0%, here.

In what follows, Kul falls apart a bit. I will do my best to tie things together:

     > If you look back to our train period MACD on Boeing stock… is the
     > best performing and here it’s the worst performing so instead of
     > filtering above 0 P&L we may actually find more Alpha filtering above
     > some +ve threshold (feel free to do this yourself!).

It’s not clear to me how he arrives at this conclusion. Maybe he’s saying not to take the best performers because they could subsequently revert and be the worst performers? Simply raising the threshold above zero would not resolve this, though.

     > The way we’d be able to deduce if this relationship holds is just to
     > roll our train/test period forwards one month and run again. If we do
     > this (i.e. use Feb 18 to Jul 18 as the train period and Aug 18 as the
     > test period) we get 15% annualised returns.

Initially, Kul trained on Jan 1, 2018, to June 30, 2018, and tested on July 1, 2018, to July 31, 2018. How does rolling forward one month start on Feb 18? It should be Feb 1 through July 31, 2018. Maybe he got the year confused with the date?

In addition to being uncertain about what dates he’s addressing, I also don’t know if the 15% annualized returns are positive or negative since he made that mistake just above. I’m a bit confused overall.

I will continue next time.

Does Technical Analysis Work? Here’s Proof! (Part 2)

Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.

Kul explains p-hacking:

     > If we run multiple permutations over and over and we just stop when
     > we reach one that looks favourable, this lands us in a situation
     > statisticians call p-hacking.
     >
     > Much like a series of coin tosses, there is a chance, however small,
     > that we continually land on heads.

Indeed, I have now learned about how to run Bonferroni and Šidák corrections for multiple comparisons in Python.

Kul continues by saying we need a better test for comparison to avoid what could be a mirage of significance caused by multiple comparisons. One possibility is to compare with a buy-and-hold group, but:

     > The problem… is that some instruments are inflationary (like Gold
     > and Stocks…) and some aren’t (like USD — in an inflationary
     > environment the dollar would likely depreciate).
     >
     > This isn’t a fair test because if a technical indicator is… right 51% of
     > the time, we may be able to reasonably deduce there’s Alpha, but
     > if we compare it against stock, well we’d expect stocks to be positive
     > more than 51% of the time given the economy grows over time
     > (historically on a daily basis the S&P 500 is +ve 55% of the time).

Kul is essentially claiming inflation to be a confounding variable (see fourth paragraph here) when looking for alpha. I don’t know that I agree. One internet source states long-term historical inflation to be ~3.2%. Regardless of the exact number, it’s positive and it happens most years. Any TA strategy that does not exceed this is not worth trading, in my opinion, regardless of whether inflation actually boosts baseline buy-and-hold performance.*

For whatever reason, stocks generally melt higher to such a degree that most long equity strategies I studied outperformed over the long-term. I believe (not yet studied) real estate melts higher. Gold seems to melt higher, but my studies did not show consistent outperformance. Contrary to Kul’s inflation hypothesis, I found oil—a commodity priced in USD—to face increased headwinds when traded long (see third paragraph here). This may be due to a particular 4-year time interval of oil prices, though: I need to look at a longer-term chart for verification.

Kul then goes on to say a better approach is the (in Python parlance) train_test_split method, which is to say use IS and OOS periods for comparison:

     > [Acceptable performance would be] over 50% right in both the train
     > period and test period (i.e. do both produce positive P&L) or we
     > require some arbitrary threshold like 0.8x of the outperformance
     > from the train period to conclude whether a particular indicator
     > “works” or not.
     >
     > The easiest way… to test this is… to run a simulation of every
     > indicator (x4) on every instrument (x10) for, say, the first 6 months
     > of 2018 so that’s 40 P&L scenario’s across x3 charts. Then we take
     > the top 10 best performing combinations (or we could even take all
     > of the ones that have produced positive P&L) and run them for
     > another 3 months then look at the performance.

I think this is all legit, but the true brilliance come next.

* — For starters, one way to study this would be to look for differences in annual stock
       returns between inflationary versus deflationary years.

Does Technical Analysis Work? Here’s Proof! (Part 1)

While the title may strike you as clickbait, it’s really based on an August 2019 TDS article written by Janny Kul that is (as of the time of this writing) available online. In this blog mini-series, I am going to do some analysis of the article followed by a bit of extra digging at the end that you really won’t want to miss: stay tuned.

Also in 2019, I wrote a blog mini-series on the same topic where I presented and commented on a sampling of others’ beliefs about technical analysis (TA). This is more of the same except I will be focusing solely on the Janny Kul article. What I particularly like about the article pertains to this fourth paragraph: Kul gives us supporting data, which I find comparable to some of my studies described here.

I will go through Kul’s article quoting and commenting on different parts. I strongly encourage you to find the whole article online for a very interesting read.

     > Given computing power nowadays, in a matter of moments we can simply
     > test every possible indicator (~27), across every standard timeframe
     > (~15), across every possible tradable instrument (>100,000).

This would be 27 * 15 * 100,000 = 40.5M strategies. What kind of computer does Kul have access to that can do this in a “matter of moments?!” Backtesting over 8 – 12 years, my computer was taking 20+ minutes to backtest a double-digit number of strategies. If I round up to 100 and only consider 40M strategies, then it would take 400,000 times as long to run Kul’s backtest on my computer, which is ~15 years. A computer 1,000 times faster than mine would [only] take ~6 days. I have been thinking of upgrading so…

I would question whether it’s reasonable to undertake a backtest of that complexity. Even at that, he’s talking about applying TA “strictly how the indicators were intended to be used.” Not only do many people not believe those settings to be the best,* I strongly believe it important to explore the surrounding parameter space as I describe in paragraphs 4-5 here. This is more what I was doing in my studies where I had up to double digit iterations. This turns the 40.5M strategies into 4.5 billion or many, many more because even 100 iterations is small compared to the finer parameter grids most strategy developers seem to use. I discuss this in paragraphs 2-3 here and the last paragraph here.

I therefore disagree with Kul in his assessment of how simple a complete analysis of TA can be.

Nevertheless, Kul does present some actual data in the article and I will get to studying that next time.

* — Many comments imply this if you read closely in Part 2, Part 3, and Part 4;
       only the reply in Part 6 suggests simple strategies can actually work.