Design of Experiments

Most people are quite familiar with the concept of experimenting, but in scientific contexts experiments are treated with a special level of reverence, and they come with all sorts of philosophical, ethical, and technical issues that the average nonscientist may not be aware of. If the goal of science is to advance human understanding of the world around us, then experimenters, to be considered credible, must carry a gigantic responsibility. To deal with this, the scientific community has developed a set of standards for experiment design, and any credible experiment must meet these standards.

The design of experiments derives from the scientific method, that long-developed set of techniques considered valid for investigating phenomena and advancing knowledge. In this sense, the concept of experiment design goes back centuries, at least to the Renaissance and Enlightenment periods in European history and equivalent periods elsewhere in the world. Scientists and philosophers in these periods recognized that scientific experimentation must conform rigidly to a set of standards, or else there is too much potential for it to veer from the truth.

Why is experiment design so important?

There are good reasons why the scientific method and the standards for the design of experiments are taken so seriously by scientists around the world. Consider, for example, a world-changing, life-and-death phenomenon like global climate change. The fact that the Earth’s climate is changing has been well documented by untold numbers of scientific experiments and studies. Yet even amid all those thousands of experiments, climate change skeptics latch on to just a small handful of poorly executed experiments and use these to disparage the science as a whole. As a result of such efforts, developments in climate policy are now at a virtual standstill, thus endangering the entire human population.

Ultimately, experiment design serves one thing: truth. A well-designed experiment, successfully implemented, reveals at least a small scrap of truth that was not previously known. A poorly designed experiment, no matter how certain its conclusions seem, only muddies the waters of truth and leads to more confusion, more skepticism, and lower credibility for science in general.

That’s why all reputable scientific bodies throughout the world have implemented rigid structures for the types of experiments they perform. It also helps that most legitimate studies are thoroughly vetted by their respective communities before going to publication. In this way, the scientific community is self policing. Even when a poorly designed experiment is carried out, there is little chance that its findings will be published.

The early stages of experiment design

The first stage of experiment design is deciding what question needs to be answered. Some questions are formulated as hypotheses. A hypothesis is a statement that the scientist seeks to prove or disprove. For example, one might begin with the hypothesis, Sugar dissolves in warm water more rapidly than in cold water. Whether the scientist actually believes this statement is beside the point. Hypotheses are sometimes defined as educated guesses, but this ignores the fact that scientists are supposed to be dispassionate observers in their experiments, and indeed a hypothesis may be formulated because the scientist suspects it is false and wants to try to disprove it.

There are many types of hypotheses, and there are complex issues that scientists have to consider when formulating them. Perhaps most important, a hypothesis must be testable. Having an overambitious scope is a quick way to sink an experiment. Meanwhile, the best hypotheses are those that can be definitely disproven. They can only very rarely be proven, however. One scrap of evidence that runs counter to the hypothesis can prove that hypothesis wrong, while volumes of evidence aren’t enough to prove that a positive statement is true under all possible conditions.

But a hypothesis is not required for an experiment to be well designed. Sometimes the scientific inquirer merely wants to see what happens under certain test conditions and may have no preconceived ideas about what is going to happen. For example, if a scientist wishes to study the effects of different types of baby formula with regard to growth, a hypothesis may be useful but is by no means essential.

In any case, it is important to have a clear view of the type of information you are trying to collect and the purpose that this information is to serve. Plus, it helps to do some research at the beginning to find out what results have been achieved through similar experiments in the past. While there is always a risk that knowing such information will bias the experiment, it also helps provide useful context and makes the objectives clearer.

It’s also important to have a sense of all the factors and variables that will be involved. An experiment should obviously have a factor that is controlled among all the objects or subjects of the experiment, but there will always be other factors that cannot be controlled and yet must be accounted for. Sometimes an experiment must be run multiple times before the scientist has a clear understanding of all the factors that will come into play. In fact, repeatability is another crucial factor of experiment design. If it can’t be done more than once, then it is not a well-designed experiment.

Elements of the experiment

Every experiment should have a control. For those who are unfamiliar with scientific terms, a control in this context is a standard of comparison to check against results. So, for example, in the study of different baby formulas, the scientist might choose from a few different possibilities. Perhaps he or she may look at studies of growth rates among children and find the average. Another possibility is to take the average growth rate for children who are exclusively breastfed—though this would add a whole different dimension to this study, which the scientist may not wish to take on.

In medical experiments, a control is also crucial in that it guards against bias resulting from the placebo effect. The placebo effect is the tendency for patients to perceive themselves as benefiting from a treatment even when that treatment actually has no measurable effect. It’s been well documented that in medical studies, patients who believe they are receiving medications but who actually receive sugar pills tend to believe that they are experiencing the benefits of the drug. In many medical experiments, the studiers intentionally include a number of subjects who receive nothing but placebo, and they also measure this data against individuals receiving no treatment at all. In this case, there are two controls.

Because the placebo effect is such a well-documented phenomenon, many studies take a double-blind approach, which simply means that neither the patients nor the experimenters themselves are aware of which subjects receive real treatment and which receive placebos. This way, there is no chance that the effect will be tainted by the interactions between researchers and subjects.

Another important factor in the design of experiments is randomization. This, too, is particularly relevant in medical experiments, but it applies in many other contexts as well. There are many types of randomization, but here are a few of the most common ones.

• Completely randomized: In completely randomized experiment design, the pool of subjects are randomly assigned to control and treatment groups, and if there are other subgroups within the experiment, these are randomly assigned as well. If the pool of subjects is large enough to eliminate anomalous data conditions, then completely randomized design is great for controlling for the effects of many types of variables. Though groups may vary slightly in their results when the experiment is repeated, complete randomization should lead to results that are more or less replicable. If not, there must be something else wrong with the experiment.
• Randomized blocks: If it is useful for experimental purposes to divide the pools of subjects into different groups while still controlling for extraneous variables, one strategy is to use randomized blocks. To use an obvious example, medical research often divides subjects into blocks of males and females, with the individuals within these groups randomly chosen. Of course, there are other ways to divide the subject pool into blocks. Studies often divide the pool based on age, race, and medical history.
• Matched pairs: Matched pairs design uses two randomly selected individuals from within a block and compares their results under two different treatment conditions. It often helps if the blocks are relatively small so that a pair may be, for example, two African-American women in their 20s instead of two women of vastly different ages and races.

Where experiments go wrong

In experimental design, there are many types of things that can wrong. Sometimes taking a look at some of the most common mistakes can be instructive. Here are some mistakes to avoid in experiment design.

• Lack of controls: When an experiment does not include a rigid system for gathering control information, the researchers end up with a pile of data that is more or less meaningless for experimental purposes. For example, if a scientist merely measures the growth rates of babies drinking formula without having any controls, this may give information about how different formulas stack up against each other, but it doesn’t indicate whether formula in general leads to fast, slow, or normal growth.
• Not enough subjects: When testing on a relatively small pool of subjects, an experiment risks letting erroneous variables taint the results. As a rule of thumb, the greater the pool of subjects, the more reliable the data will be. That’s why many studies and experiments are quite laborious and repetitive in their execution. But for reliable results, all that extra work is often with it.
• Poorly chosen subjects: Consider, for example, a poll seeking to gauge public opinion on a proposed tax hike on wealthy people. If the pollsters only interview people on the higher end of the income spectrum, the results will obviously be skewed against the tax hike proposal. This is just an easy to understand example of a type of mistake that can happen in many types of experiments. It, again, underlines the importance of both large subject pools and randomization. A randomly chosen selection of 1000 poll subjects is very likely to provide accurate information on a given question.
• Researcher bias: As mentioned above, there are different approaches one can take to hypotheses. Some hypotheses come from a disinterested perspective, and it’s not important to the experimenters whether they are proven or disproven. Other hypotheses are statements the experimenters strongly believe to be true. In the latter case, there is always a risk of the experimenter’s bias tainting the execution, the gathering of data, and the interpretation of the results. That’s why controls and other regulating elements must be built into any credible experiment.
• Lack of repeatability: It’s always an exciting moment when an experiment points to surprising conclusions and upends the scientific community’s understanding of a given subject. But even the most amazing experiment cannot be considered credible if it cannot be repeated by other scientists. This is why new scientific developments must always be approached with skepticism until more researchers get the chance to repeat the process.
• Drawing correlative rather than causative conclusions: Consider again the example of the baby formula experiment. Imagine that one group of babies who take a certain type of very inexpensive formula tend to grow at a meager rate compared with the other groups. At first glance, this suggests that that cheap type of formula leads to poor growth. But, digging deeper, the researchers may find other factors. For instance, perhaps the cheap formulas are more often used by poor families or single mothers who have comparatively little time to spend with their babies. Perhaps the poor growth can actually be attributed to a shortage of physical closeness between parents and baby. Experimenters must always be hypercritical of their own conclusions in order to avoid such mistakes of causation.