Although we have been believers in Design of Experiments for nearly twenty years and builders of automated trading systems for nearly ten, we never put the two together until recently. A designed experiment simply our trading system would have taken years as recently as three years ago. As a result, we did like everybody else does. That is, we set our system parameters using intuition and experience, and tweaked those each time we reviewed the findings from any test run. We made significant progress in that way, but it was plain to see that we could learn more faster.
We started applying Design of Experiments to automated trading systems about a year ago, but the tests were taking a long time. We were working with intraday data, and that is really dense. It took many months to complete a study that used just two years of market data. Even at that cost, we were able to run a couple of studies and became convinced that the approach was valid. Unfortunately, the systems were proprietary and secret so we were having a little trouble talking about our approach in any detail without betraying a confidence.
A couple of months ago, Ernest Chan suggested that we test something from the book Short Term Trading Strategies That Work. We needed to talk about something other than our own systems. There were two major surprises.
The first was the speed of the tests. Our tests for the options trading systems that we built took six hours for each trial for each expiry (month). An eighteen month study takes about three months running ten of these tests at a time in parallel. The strategies in the book used only end-of-day data. As a result, a test on a twenty-year set of data takes one quarter of a second! We had the study completed, and article on it accepted by a publisher in less than a week.
This is mind-boggling because in most experimentation the cost of the trials is the overwhelming majority of the cost of the experiment. The challenge of most of Design of Experiments is to get as much information out with as few trials as possible. One of the problems with our business model when we built Strategy was that there very few people who needed enough experiment designs to keep us in business because they would get a design, take months or even years to run the experiment, then need our help or software for about two hours to interpret the results. When the trials are free and run this fast, the design of the experiment and the analysis become significantly more difficult and time-consuming. That’s fine with us because we have some unique advantages there.
The response to our efforts was the second surprise. We had never heard of this book, but it turns out to be very popular in the trading community. Many people have read, and those that haven’t, had at least heard of it. Many people had a common basis for understanding our work, and we are starting to have real conversations about applying this technology. We are moving on to models that we couldn’t even consider when the cost of trials was an issue.