Does Technical Analysis Work? Here’s Proof! (Part 4)
Posted by Mark on April 15, 2021 at 07:10 | Last modified: March 11, 2021 12:01Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.
I was a bit confused where we left off. Kul continues:
> It appears as though there may be Alpha reversing filtered technical
> indicators… We’d need to keeping [sic] rolling this forwards to
> actually find if this relationship continually holds.
I think he’s basically suggesting we test the worst performers from the training set for outperformance. That is a very interesting idea. I would want to know if the worst training indicators do better on the test set than the best training indicators. This reminds me of the Callan Periodic Table of Investment Returns, which I mentioned in the middle of this post.
> Obviously adding transaction costs and bid/offer would mean we can’t…
> capture this but this does give us something to investigate further.
Does he mean we can’t realize any profits from this or just diminished profits? He could have included sample transaction fees to get more clarity on this.*
He then teleports ahead to Bitcoin. Say whaaaaaat? Speaking of transaction fees, though, exactly what vehicle is being used to trade it and what are the usual slippage and commissions to do that? I (and most veteran investors, probably) would be very interested to know since Bitcoin is relatively new.
> So our train period has a monthly average of 20.4% and our test period
> has annualised returns of 14.3%…it appears as though there may be
> some Alpha on all technical indicators for Bitcoin.
That sounds encouraging…
> Interestingly in our train period we outperform Bitcoin but in the test
> period Bitcoin outperforms.
If buy-and-hold outperforms, then the indicators have no alpha. Why did he just say otherwise?
> In order to say with certainty if this relationship holds we’d again
> need to test again over a longer period of time.
Kul then repeats the backtest for all 12 months of 2018. This extends the backtest by five months since the first six months were the training set and July was the testing set.
> I think it’s fairly safe to say that the performance of all the
> indicators decays over time however we do actually outperform
> buying and holding Bitcoin (although, granted, 2018 was a terrible
> year for Bitcoin).
I think it’s fairly safe to say we really can’t make any conclusions over such a short period of time where the results are so inconsistent with what we saw before.
Kul concludes:
> We found… reversing filtered indicators may have Alpha for non-
> Bitcoin instruments and for Bitcoin… our regular indicators
> may have… Alpha although it does severely decay over time.
Indicator performance declined over the course of these several months, which is still a short time interval. I wouldn’t generalize to “over time,” which sounds much more substantial.
> We’d need to test on a much larger data set to see if these
> relationships do actually hold.
Kul catches himself here and I totally agree. Indeed, the biggest critique I have of this article is the limited backtesting interval. Although he uses a 5-minute time frame, the total study period is one year or less. In case we are looking at a large sample size, Kul could have boosted credibility by reporting number of trades in each group, which he never mentions.
In the final analysis, I can’t help but respond to Kul’s title with “Where’s the beef?”
I will continue next time.
* — I feel strongly about including transaction fees in backtesting as discussed in paragraphs 2-3 here.
Does Technical Analysis Work? Here’s Proof! (Part 3)
Posted by Mark on April 12, 2021 at 06:50 | Last modified: March 11, 2021 09:00Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.
I left off at the point where I think Kul’s article gets really special:
> Now if by some miracle this does work, just to prove it was all one
> big fluke, we should be able to roll forwards another 3 months to
> produce positive P&L again.
The main takeaway from this article is right here. It’s one thing to train a model, which by definition is going to demonstrate good performance, and then follow through with more good performance. I became disillusioned when I was unable to accomplish this repeatedly, which is basically what walk-forward optimizmation does. I then became disillusioned again when I incorporated one additional incubation period as Kul mentions here. I wrote about OOS2 in the third bullet point of this post.
> Note… the average across all these indicators across all instruments
> is 0.095% per month so I think it’s reasonable to deduce that the
> indicators used by themselves without any filtering have pretty much
> zero Alpha…
>
> Now… we want to… run [the winners]… for 1 month forward… the
> average of these is -1.92% for this 1 month period so if anything we
> might be able to deduce that filtering positive indicators is actually
> mean-reverting. Annualised performance here would be 23.0%.
I’m guessing he meant negative 23.0%, here.
In what follows, Kul falls apart a bit. I will do my best to tie things together:
> If you look back to our train period MACD on Boeing stock… is the
> best performing and here it’s the worst performing so instead of
> filtering above 0 P&L we may actually find more Alpha filtering above
> some +ve threshold (feel free to do this yourself!).
It’s not clear to me how he arrives at this conclusion. Maybe he’s saying not to take the best performers because they could subsequently revert and be the worst performers? Simply raising the threshold above zero would not resolve this, though.
> The way we’d be able to deduce if this relationship holds is just to
> roll our train/test period forwards one month and run again. If we do
> this (i.e. use Feb 18 to Jul 18 as the train period and Aug 18 as the
> test period) we get 15% annualised returns.
Initially, Kul trained on Jan 1, 2018, to June 30, 2018, and tested on July 1, 2018, to July 31, 2018. How does rolling forward one month start on Feb 18? It should be Feb 1 through July 31, 2018. Maybe he got the year confused with the date?
In addition to being uncertain about what dates he’s addressing, I also don’t know if the 15% annualized returns are positive or negative since he made that mistake just above. I’m a bit confused overall.
I will continue next time.
Categories: Backtesting | Comments (0) | PermalinkDoes Technical Analysis Work? Here’s Proof! (Part 2)
Posted by Mark on April 9, 2021 at 07:04 | Last modified: March 10, 2021 13:33Today I continue with commentary and analysis of Janny Kul’s TDS article with the same title.
Kul explains p-hacking:
> If we run multiple permutations over and over and we just stop when
> we reach one that looks favourable, this lands us in a situation
> statisticians call p-hacking.
>
> Much like a series of coin tosses, there is a chance, however small,
> that we continually land on heads.
Indeed, I have now learned about how to run Bonferroni and Šidák corrections for multiple comparisons in Python.
Kul continues by saying we need a better test for comparison to avoid what could be a mirage of significance caused by multiple comparisons. One possibility is to compare with a buy-and-hold group, but:
> The problem… is that some instruments are inflationary (like Gold
> and Stocks…) and some aren’t (like USD — in an inflationary
> environment the dollar would likely depreciate).
>
> This isn’t a fair test because if a technical indicator is… right 51% of
> the time, we may be able to reasonably deduce there’s Alpha, but
> if we compare it against stock, well we’d expect stocks to be positive
> more than 51% of the time given the economy grows over time
> (historically on a daily basis the S&P 500 is +ve 55% of the time).
Kul is essentially claiming inflation to be a confounding variable (see fourth paragraph here) when looking for alpha. I don’t know that I agree. One internet source states long-term historical inflation to be ~3.2%. Regardless of the exact number, it’s positive and it happens most years. Any TA strategy that does not exceed this is not worth trading, in my opinion, regardless of whether inflation actually boosts baseline buy-and-hold performance.*
For whatever reason, stocks generally melt higher to such a degree that most long equity strategies I studied outperformed over the long-term. I believe (not yet studied) real estate melts higher. Gold seems to melt higher, but my studies did not show consistent outperformance. Contrary to Kul’s inflation hypothesis, I found oil—a commodity priced in USD—to face increased headwinds when traded long (see third paragraph here). This may be due to a particular 4-year time interval of oil prices, though: I need to look at a longer-term chart for verification.
Kul then goes on to say a better approach is the (in Python parlance) train_test_split method, which is to say use IS and OOS periods for comparison:
> [Acceptable performance would be] over 50% right in both the train
> period and test period (i.e. do both produce positive P&L) or we
> require some arbitrary threshold like 0.8x of the outperformance
> from the train period to conclude whether a particular indicator
> “works” or not.
>
> The easiest way… to test this is… to run a simulation of every
> indicator (x4) on every instrument (x10) for, say, the first 6 months
> of 2018 so that’s 40 P&L scenario’s across x3 charts. Then we take
> the top 10 best performing combinations (or we could even take all
> of the ones that have produced positive P&L) and run them for
> another 3 months then look at the performance.
I think this is all legit, but the true brilliance come next.
* — For starters, one way to study this would be to look for differences in annual stock
returns between inflationary versus deflationary years.
Does Technical Analysis Work? Here’s Proof! (Part 1)
Posted by Mark on April 6, 2021 at 07:26 | Last modified: March 9, 2021 15:09While the title may strike you as clickbait, it’s really based on an August 2019 TDS article written by Janny Kul that is (as of the time of this writing) available online. In this blog mini-series, I am going to do some analysis of the article followed by a bit of extra digging at the end that you really won’t want to miss: stay tuned.
Also in 2019, I wrote a blog mini-series on the same topic where I presented and commented on a sampling of others’ beliefs about technical analysis (TA). This is more of the same except I will be focusing solely on the Janny Kul article. What I particularly like about the article pertains to this fourth paragraph: Kul gives us supporting data, which I find comparable to some of my studies described here.
I will go through Kul’s article quoting and commenting on different parts. I strongly encourage you to find the whole article online for a very interesting read.
> Given computing power nowadays, in a matter of moments we can simply
> test every possible indicator (~27), across every standard timeframe
> (~15), across every possible tradable instrument (>100,000).
This would be 27 * 15 * 100,000 = 40.5M strategies. What kind of computer does Kul have access to that can do this in a “matter of moments?!” Backtesting over 8 – 12 years, my computer was taking 20+ minutes to backtest a double-digit number of strategies. If I round up to 100 and only consider 40M strategies, then it would take 400,000 times as long to run Kul’s backtest on my computer, which is ~15 years. A computer 1,000 times faster than mine would [only] take ~6 days. I have been thinking of upgrading so…
I would question whether it’s reasonable to undertake a backtest of that complexity. Even at that, he’s talking about applying TA “strictly how the indicators were intended to be used.” Not only do many people not believe those settings to be the best,* I strongly believe it important to explore the surrounding parameter space as I describe in paragraphs 4-5 here. This is more what I was doing in my studies where I had up to double digit iterations. This turns the 40.5M strategies into 4.5 billion or many, many more because even 100 iterations is small compared to the finer parameter grids most strategy developers seem to use. I discuss this in paragraphs 2-3 here and the last paragraph here.
I therefore disagree with Kul in his assessment of how simple a complete analysis of TA can be.
Nevertheless, Kul does present some actual data in the article and I will get to studying that next time.
* — Many comments imply this if you read closely in Part 2, Part 3, and Part 4;
only the reply in Part 6 suggests simple strategies can actually work.
Can a Retail Trader Succeed at Algorithmic Trading? (Part 7)
Posted by Mark on April 1, 2021 at 06:56 | Last modified: March 9, 2021 10:44Today I will conclude with presentation and commentary in an algorithmic trading thread that took place on a popular online forum about 18 months ago.
> What if you want to make more money; a higher SR? Then you are going
> to have to move towards (a) the world of HFT and / or (b) the world of
> weirder, shorter lived alpha-decaying, non linear patterns and/or
> (c) the world of ‘alternative data’. And away from classical linear
> statistical methods, towards the wacky world of ML. To play in these
> worlds you are going to need to make serious investment in automated
> trading technology, but more importantly you are going to have to be
> able to use ML properly.
>
> The average person using ML in finance does so very badly, and this
> is based on an observation of ‘professionals’ that doesn’t include the
> hoards of amateurs who’ve just downloaded a python package and have
> no idea what they are doing. It’s much easier to overfit with fancy…
Notice the implication here that he has been in position to observe professionals work the craft (of ML). Few can say we have done this. He claims to be an industry professional and I think his writing is very polished. He’s also a book author.
> ML techniques than with classical ones. Given how much overfitting
> goes on just using old-fashioned grid searches and regressions, it’s
> no surprise that overfitting is absolutely endemic within the neural
> network, AI, non-linear classifying crowd.
This is very consistent with what I learned in Datacamp ML courses.
> You need a team to do this properly, first because of the alpha
> decay you are going to spend so much time finding new effects you
> don’t have time to do anything else like actually implement them.
> Second, because it’s less likely that a single person will have
> the full range of skills required to test and implement ML based
> trading strategies. Such people do exist, but they are rare: after
> all it’s rare enough to find people with the full set of skills
> to test and implement classical trading strategies.
Here’s yet another call for a team-based approach (also mentioned in Part 1 and Part 5).
> What does this mean for the individual trader? Simply put, don’t
> use ML unless you know exactly what you are doing. And stay
> away from trading arenas where you need to be able to use ML to
> discover the edges that exist, plus have access to the technology
> that will allow you to exploit those edges. There are plenty of
> areas where you can still compete, but you will have to lower your
> expectations for SR, and thus increase your bankroll or remain
> as a part time trader.
Is RC right about these claims? I really don’t know. His credentials look good, but those can be phony. Many details seem consistent with comments I’ve heard elsewhere and for me, convergence usually boosts credibility.
Categories: System Development | Comments (0) | PermalinkCan a Retail Trader Succeed at Algorithmic Trading? (Part 6)
Posted by Mark on March 29, 2021 at 07:12 | Last modified: March 9, 2021 09:28Today I will continue with presentation and commentary in an algorithmic trading thread that took place on a popular online forum about 18 months ago.
I believe flippant replies like the following are of little use with their absolute, elitist, and hollow claims:
> The basic fact of the matter: if you love problem solving, the market is just
> an other [sic] puzzle, and you can have as much money as you want.
>
> Its [sic] not in the numbers or indicators, its [sic] the edge around which you
> design the trading system.
>
> Running tests won’t give you the edge, spend time finding your edge [sic:
> comma splice, John Rubadeau].
>
> Thats the core of your system, manual or automated [sic, sic, sic].
I often find such replies to be sorely in need of proofreading, too.
In total contrast, I think the following reply sounds very educated and even brilliant in places:
> So I think it’s possible for pretty much anyone to make money using
> simple systematic trading strategies. Most of this money comes from
> being exposed to diversified sources of risk, so no ‘secret sauce’
> or fancy ML techniques are needed. These strategies will mostly be
> quite slow in nature, so not HFT. They will be based on sources of
> risk premia that decay very slowly, if at all. They will not be high
> Sharpe Ratio, but by diversifying over a large number of uncorrelated
> instruments and a number of different strategies I think an expected
> SR of 1.0 is feasible.
Based on my limited research, I don’t know if I agree with this. I’ve looked at roughly 300 simple strategies on three different markets and none came close to passing the KD criteria although I do believe these criteria to be stringent (probably requiring better than SR of 1.0). Could I trade some of these markets during uncorrelated times (I don’t think it makes as much sense to speak of “uncorrelated markets” because correlation changes) and get an overall SR of 1.0? I would regard this as pretty good because my testing shows SR for the S&P 500 to be around 0.35.
> Does this count as success? A SR of 1.0 achieved over ten years or
> more would be top quartile for nearly every hedge fund category.
> But you will struggle to make a living as a trader with a SR of 1.0,
> if trading is your only source of income, unless you are very
> well-capitalized (equates to lower risk and return target).
This is very interesting and theoretical. I would need to backtest and try and correlate SR with annualized return, drawdown, and then figure out reasonable position sizing to determine whether profits could exceed annual living expenses.
I will conclude next time.
Categories: System Development | Comments (0) | PermalinkCan a Retail Trader Succeed at Algorithmic Trading? (Part 5)
Posted by Mark on March 26, 2021 at 07:00 | Last modified: March 9, 2021 08:54I have been presenting and discussing a thread regarding algorithmic trading that took place in a popular online forum about 18 months ago.
I left off with a reply that concludes:
> We live in a time where funding is not extremely difficult. From instant loans to
> credit cards to friends and family to mortgages — if you have an extremely good
> algo then you’ll get the funds. If you cannot get the funds then you’re probably
> not smart enough to participate in the industry anyway. Consider it a test.
This reply included some really good points until now when some credibility is lost. I think fundraising is extremely difficult. Then again: 1. I have yet to generate an “extremely good algo” and; 2. maybe I am not smart enough (see third paragraph here about nothing unique) to participate in the industry.
> As KD said: fitting a strategy to past data is the biggest “crime” I see developers
> making. But every strategy is dependent on past data at the very least for pattern
> recognition. When we say a strategy doesn’t “work” anymore, what we’re really
> saying is that we can’t wait around 50-100 years for it to work again. Because it
> probably will work again given enough time.
Five stars for this reply, baby! Countless times have I heard/read people say that trading systems break never to be profitable again. One reason for this could be because more and more people trade them, which can kill the edge. As time passes, the number of available strategies approaches infinity faster than the number of traders, which suggests a decreasing likelihood that a particular strategy will be traded live. I therefore believe that strategies no longer in vogue can be profitable again years after their trading frequency diminishes.
Incidentally, this reminds me of the Callan Periodic Table of Investment Returns where we frequently see asset classes that significantly underperform in previous year(s) suddenly start to outperform going forward (the cycle may repeat).
> De Prado’s point is that since all strategies have a shelf life, you need a team to
> build a strategy factory for true long term success. He is also not talking about
> retail but about other’s people money. You can’t take the risk free or index rate
> for 2 years with OPM while you figure out a new winning strategy without going
> out of business. A single retail trader could do exactly that.
This is interesting rationale for a team-based approach to trading strategy development. For this and other reasons, I agree with the approach (also discussed in Part 1).
Use of the term “strategy factory” is notable here, too.
I will continue next time.
Categories: System Development | Comments (0) | PermalinkCan a Retail Trader Succeed at Algorithmic Trading? (Part 4)
Posted by Mark on March 23, 2021 at 06:44 | Last modified: March 8, 2021 13:44Today I continue presentation and commentary on an internet thread (subtitled “KD vs. EC”) regarding algorithmic trading that took place about 18 months ago.
> I think they are both “right” in their own way. If you look at what KD does,
> it’s a little different from EC’s strategies.
KD gets his customers to work for him (see second paragraph here)! I wonder how many are aware of this fact.
> The “trading robots” you see on sites like collective2 aren’t serious edges.
> Only an idiot would sell a serious proven edge to random Internet anons for
> pennies on the dollar, and idiots don’t develop profitable trading algos.
> If you have a proven edge, you raise capital to trade it yourself or sell
> it to a quant fund for serious lump-sum cash and a job.
As I discussed near the end of Part 3, it’s not so easy to just “raise capital to trade it yourself or sell it to a quant fund.” Proving the edge probably means creating an incubator fund (or friends and family fund, if one can be lucky enough to gather a reasonable minimum amount of capital this way) for purposes of trading and auditing. That can cost many thousands of dollars, which singularly establishes a high barrier to entry.
> Both of them are nice guys, and I’ve complimented them for their contributions,
> but neither one of them is really “killing it” in this game. I tend to learn a lot
> more from those with serious (1M+) skin in the game, and I don’t mean OPM.
We don’t know whether either is “killing it,” much less how much “skin in the game” (actual capital deployed) either has (see fourth paragraph here).
> And there lies the bait. Yes, in theory a large account can withstand large
> drawdown if low leverage is used, but most people do not have large accounts,
> and those that do, will pull the plug if drawdown gets to 20 percent or more.
> I have seen clients closing accounts after 10 percent loss.
This max drawdown is consistent with the kind of performance I have come to believe is necessary to have something of institutional quality.
> The algorithms the largest banks and funds use depend on infrastructure or size.
> There’s almost no way an at-home type trader can participate in this. Many
> markets have large capital requirements — 100s on millions just to enter.
> For others you need best co-location or contacts in the industry.
I would imagine these algos are of the HFT variety. I don’t know whether this person knows from experience or is just hypothesizing, but s/he makes some really good logical points. The post continues:
> I’ve been following people who sell signals for a long time. There are very rarely
> profitable systems but they always go private fast because selling something that
> can make millions for thousands is not a business. On top of that you have people
Speaking of someone who has been following signal sellers for a long time, I’d like to hear Mark Hulbert’s take on this.
I will continue next time.
Categories: System Development | Comments (0) | PermalinkCan a Retail Trader Succeed at Algorithmic Trading? (Part 3)
Posted by Mark on March 18, 2021 at 07:51 | Last modified: March 8, 2021 10:58Today I continue presentation and commentary on an algo trading thread that took place about 18 months ago.
> Ok, wait a minute: every time traders are talking about big players and market
> manipulation, everyone seems to agree that algos are running the game. So,
> assuming that this is right – someone stating that 85% of market is driven by
> them – that means the big banks, players, etc. actually have financial
> algorithms that work and are profitable.
>
> If this is true, why is there, after decades of banks using them, no market for
> retail traders to buy them, even if it is not the best/newest version of all?!
This is a very interesting point. Question the premise, though: who is to say that many big firms don’t have profitable algos?
Aside from banks, the “etc.” is probably asset managers and hedge funds. With regard to the former, passage of the Dodd-Frank Wall Street Reform and Consumer Protection Act (July 2010) made it illegal for deposit-taking institutions to engage in prop[rietary] trading. That would include banks and many asset managers. With regard to hedge funds, I studied 2008 – 2016 and found them to underperform (see table here). This may leave some smaller firms that attempt prop trading for a period only to realize they can’t make [enough] money at it and quit. I therefore think it likely that the overall universe for prop trading, where algos are being used to generate the huge Wall Street profits of legend, is significantly diminished.
I’m not convinced that “everyone” (who “seems to agree that algos are running the game”) is all that informed—especially in the face of the kind of reasonable doubt just discussed. People repeat what the financial media writes or broadcasts. Does anyone actually check and scrutinize sources or underlying motives? Not nearly enough, I suspect.
One person responded:
> The reason is only bank and large firms’ algos can work for them such as
> market making and HFT. Hardware barrier of entry is in 10s of millions.
I can’t verify the claim of eight figures, but I do believe in a large and formidable barrier to entry.
> What you are looking for is a strategy can can be deployed on retail
> level. While those exist, no one in their right mind would disclose it.
This would imply the strategies given by KD (see second-to-last paragraph here) are bogus.
> It is not that difficult to raise funds and trade it, as long as it
> indeed has positive expectancy.
>
> You will not find it anywhere in public domain.
I don’t believe it is ever easy to raise funds for investment. This article claims the financial services industry has a 12% success rate. Another article claims 90% of financial advisors will fail within the first three years. Claims like these along with the personal roadblocks I have discovered around the wealth management industry make me believe raising money is anything but “easy”—a word that I as a marathon runner will never take for granted (injuries!).
And once again: forget the public domain. You will have to uncover it yourself.
I will continue next time.
Categories: System Development | Comments (0) | Permalink