Sunday, March 9, 2008

Backtesting Methodology

I will utilize backtesting results often on this blog to illustrate points and that warrants a brief description of my technique and a disclaimer. The IBD100 is a dynamic universe of stocks and therefore, nearly impossible to backtest accurately. The purpose is not to try and predict exactly what a system will make but to understand how different concepts tend to affect results. We are not testing variables, we are testing concepts. I will not optimize anything and I will generally run the tests on the same basket of 300 IBD scan generated stocks over a 10 year period (1/1/1998-12/31/2007). The idea is to generate enough results for statistical relevancy. Except when backtesting money management and position sizing, I will use $1000 per trade, will not include slippage and broker’s fees, and will exit all open trades at the end of the test.

The backtesting software used for these tests allow split adjusted and raw historic data to be used together. One important point is one must use raw, unadjusted data to calculate entry and the number of shares that are purchased and split-adjusted data to calculate returns. This is important when backtesting stocks. If split-adjusted data is used for entry, it would reflect you purchasing many more shares than the funds allocated per trade would permit at that time. CSI is the data vendor, backtesting software is TradersStudio by Murray Ruggiero.

The data used is end of day only. Unless noted otherwise, all entries are market orders at open on the day following the entry signal. Protective and trailing stops are standing stop/market orders and profit target stops are standing limit orders that execute during market hours. You will note in the results that the largest loss is typically larger than our protective stops and this is becuase of large price shocks where the market gaps down at the open. These "real-world" situations are not omitted from the results and are part of the risk associated with holding positions overnight. These can wipe out an undercapitalized or over-leveraged account even with a good system with a positive expectancy.

All backtesting introducues survivorship bias and many of the stocks in the current IBD list were barely more than penny stocks 10 years ago, if around at all. The yearly results of these tend to show much better performance in rescent years as they work their way onto the IBD list. Again, I must stress, these tests are not attempting to predict future results, they are only to understand how different methods affect performance and I will only post results for test who's results are consistant across almost ANY basket of stocks.

The point is to find robust methods that work across as wide a universe as possible under a wide range of conditions. Sometimes I will run tests on other baskets of stocks or etfs to illustrate concepts of robustness.

In addition to William O’Neil’s books, another book I highly recommend is Dr. Van Tharp’s Trade Your Way to Financial Freedom. In discussing testing results, I will utilize his concepts of R-Multiples which is basically a way of evaluating risk-reward through the lens of expectancy. An overly simplified version is as follows:

Your R-value is the amount risked per trade. Often this is a fixed number but sometimes when backtesting variable exits like n-day lows, this number varies from trade to trade and an exact number is difficult to determine through backtesting results only and the Average Losing Trade value can be used as a substitute. A system’s R-value is evaluated in relationship to the Average Trade value to determine your expectancy per trade per dollar risk. If this is confusing, it should become clear later and a thorough read of Dr. Tharps’s book will clear up any confusion.


Anonymous said...

"I will generally run the tests on the same basket of 300 IBD scan generated stocks over a 10 year period (1/1/1998-12/31/2007)"

Are you referring to 300 IBD stocks that from last few years?

If so it does not make sense since these are the stocks that were done well in IBD radar for some time historically.

The feel the dynamic historical weekly list of the IBD 200 should be used for such test and all the results should be added.

You can try to extract the list of IBD200 stock from Mr Bonde and apply the test for each week's list for few month range. Then summarize the results. That will make it more sensible realtime test.

Thanks for all the efforts put into this. But put a little more effort to get all the IBD200 historical data and then we can analyze.

bhh said...

Your comments are absolutely correct. My objective is not to test Pradeep's system. All backtesting is hypothetical and introduces survivorship bias. These tests are no different. My goal in these tests is to explore how different stops affect expectancy, not to show what trading that system would have made. This weekend, I will post the same system and stops applied to the 100 stocks that make up the Nasdaq100 and the 500 stocks that make up the S&P500. The goal will be to understand and illustrate how these different stops affect expectancy across a broad range of portfolios. These test will have survivorship bias built-in as well but the structural affects should reveal themselves. The important thing to look at is the expectancy ratio between the different tests, not the absolute values. Thank you for your keen observations.

Damian said...

In terms of specifics in your testing, how many positions can you hold at any one time in your system - that is, is there a max, or does the system just take all trades that have breakout and allocates $1000 to each trade? I believe that Tradersstudio (I own it but don't use it that much) can say the maximum amount of capital required - that would be interesting to look at as well. Great stuff...

bhh said...

It can test money management at the trade plan level but for these early tests, I am mainly interested in the statistical and structural results and therefore just allocating $1000 to each and every trade at the "session" level. At some point, I will post on and discuss different money management strategies.

generic cialis said...

I, of course, a newcomer to this blog, but the author does not agree