Error Doesn’t Matter, Trial Does
February 27th, 2013
If you want to learn, to really learn, experiment.
But I’m not talking about elaborate experiments; I’m talking about crude ones. Not simple ones, crude ones.
We were taught the best experiments maximize learning, but that’s dead wrong. The best experiments are fast, and the best way to be fast is to minimize the investment.
In the name of speed, don’t maximize learning, minimize the investment.
Let’s get right to it. One of the best tricks to minimize investment is to minimize learning – learning per experiment, that is. Define learning narrowly, design the minimum experiment, and run the trial. Learning per trial is low, but learning per month skyrockets because the number of trials per month skyrockets. But it gets better. There’s an interesting learning exponential at work. The first trial informs the second which shapes the third. But instead of three units of learning, it’s cubic. And minimizing learning doesn’t just half the time to run a trial, it reduces it by 100 or more. It’s earning to the hundredth power.
Another way to minimize investment is to minimize resolution. Don’t think nanometers, think thumbs up, thumbs down. Design the trial so the coarsest measuring stick gives an immediate and unambiguous response. There’s no investment in expensive measurement gear and no time invested in interpretation of results. Think sledgehammer to the forehead.
A third way to minimize investment is to evaluate relative differences. The best example is the simple, yet powerful, A-B test . Run two configurations, decide which is better, and run quickly in the direction of goodness. No need to fret about how much better, just sprint toward it. The same goes for trial 1 versus trial 2 comparisons. Here’s the tricky algorithm: If trial 2 is better, do more of that. And the good news applies here too – the learning exponential is still in play. Better to the hundredth power, in record time.
I don’t care what norms you have to bend or what rules you have to break. If you do one thing, run more trials.
But don’t take my word for it. Dr. Seuss had it right:
And I would run them in a boat!
And I would run them with a goat…
And I will run them in the rain.
And in the dark. And on a train.
And in a car. And in a tree.
They are so good so good you see!
Posted in 
Tags: 4 Responses to “Error Doesn’t Matter, Trial Does”
Leave a Reply
Mike Shipulski
I couldn’t have articulated it as you just did in this article, but I must confess on this one. I would not want to be seen or caught cutting out paper-dolls from AutoCad plots and folding pieces of paper into strange origami shapes to determine if an idea/design was even possible. If it passed the office-desk test, I would then scrounge-up aluminum, plastic or other suitable material from the machine-shop scrap bins to try some modeling in my own home DIY shop. Often, I would be able to return the next morning with something tangible in my had able to demonstrate my idea and get acceptance to try it with a little more investment and talent from work. Though my background was in Industrial Engineering, I loved tinkering with mechanical designs and would surprise the ME’s with a new direction to try in solving some of our problems. If my idea didn’t work or was not preferred – there was virtually no loss or cost, and we all moved on!
I agree completely. See also “Rapid Results,” by Robert H. Schaffer and Ronald N. Ashkenas.
This must be a good post.
By my yardstick , the ones that evoke the most thought rank at the top.
I agree, more trials, more tests, more experiments.
“A test under controlled conditions that is made to demonstrate a known truth, examine the validity of a hypothesis, or determine the efficacy of something previously untried.” (http://www.thefreedictionary.com/experiment)
BUT…
The best experiments fail. This is where real learning occurs.
What can we possibly learn from an experiment that follows our hypothesis exactly? We must already posses sufficient knowledge if our hypothesis was a prediction rather than just our best guess. What knowledge do we gain by demonstrating a known truth?
This can lead us to a better definition for a design experiment. I think Mike’s “trials” refers to the third part of the definition given above. Something that has never been tried before. Something we don’t already know. Something to provide knowledge and an opportunity for learning. The magic engineering word “WHY?”.
Don’t spend time proving what you know during the design phase (this can be the plural “you” if we’re a team). Proof comes after, during compliance testing or design verification testing.
Managers don’t like failure. It doesn’t reinforce “we know what we are doing”.
I’ve had my share of discussions where failure isn’t an option.
Quick, simple, informal and inexpensive gets you around most objections.
How about a post on spectacular failures (either side of the coin)?
Thanks;
Doug
OOPS! April 25, 2012, “The forbidden fruit of failure”
AND I responded to that one as well.
So I’m getting old and forgetful. I still think it’s a good post.
Doug