Usually, when you run an experiment, the primary cost of the experiment is the cost of running the trials. Designing the experiment is usually fairly simple, and never very expensive. The analysis requires some moderately specialized skills, but you can usually afford the analysis if you can afford the trials. Now, for the first time in our experience, we have discovered a field of investigation where the cost of the trials is almost nothing. This has profound side effects that we did not anticipate.
We should have because the most complex experiments were always run by people who had the least expensive trials, but even testing print cartridges makes a noticeable dent in a budget when you want to run a hundred trials. At the very least, the tests will still take hours, perhaps days, to run.
When we started applying Design of Experiments to financial engineering, we were strictly interested in intraday trading systems. Testing these systems had a moderate cost. For example, if we wanted to run sixteen trials on three years of data, it did not cost us much money, but it did take several months. That has a significant cost when you want to know the answers right now.
We wanted to talk about those experiments, but a number of factors prevented that We looked around for another system that we could experiment on. Ernest Chan recommended that we look at systems in Short Term Trading Strategies That Work and High Probability ETF Trading, two books by Larry Connors and Cesar Alvarez. As Ernie correctly perceived, Connors and Alvarez described systems that lent themselves very well to a Design of Experiments. We came up with lots of ideas for experiments right away, and we picked one out to try it out.
My partner Ron is very fast at this kind of thing, and he had a simulator ready to run the test after about a day of work. He had to organize the data because we had never worked with an end-of-day system before. Then he started running the test. We knew they would go quickly, but they only took about a quarter of a second to run a trial on fifteen years of data. It took far less than half a minute to run 56 trials.
This is as close to instant gratification as an experimenter will ever get! It took a few hours to analyze the results, but it was essentially “same day service”. You usually have months separating the planning of the experiment and the analysis.
Immediately, we began to think of more complex experiments looking at more factors and using higher order models. Since the higher order models required more trials, we rarely even thought in that direction. The cubic model was the highest order model supported by the software we were using, and we very rarely had the opportunity to use that. Most people are looking for experiment designs with less trials, not more. In fact, the point of most experiment design software on the market today is to get the most information out of the fewest number of trials.
Notice how the goal changes when the cost of the trials approaches zero. The goal in this environment is to get the best possible predictions without regard to the number of trials it requires. Another fifty trials? Another hundred trials? How about another thousand trials? We don’t care when the trials are free.
Rust Never Sleeps 