Planning and conducting experiments:
DOE with Jackie Graham
By Jackie Graham, Ph.D.
Managing Director, PQ Systems, Australia
In November, we explored some of the terminology used in experimental design and looked at some of the basic concepts. This time we will overview experimental design in more detail. We will look at what is involved in planning and conducting experiments; analyzing results; and avoiding pitfalls. This article will provide a step-by-step methodology to follow. In later articles we will look at these steps in greater detail.
From the outset, we need to be clear about what we are trying to achieve with our experiments; the experimental methods to be used; and the analysis techniques to employ. Generally, we design experiments so that the variables which have the most impact on outputs are clearly identified and the extent of the impact is measured. From the results, we can determine the optimum levels for the variables so that the best possible product can be produced.
It’s a really good idea to assess process stability as part of the experiment. We do this using a control chart to assure ourselves that no special cause variation is present. Experimental results will be far more reliable if they come from a stable process. However, if this cannot be achieved, the process instability must be accounted for when analyzing the experiment.
Normally, experimental design is performed in stages as outlined below.
1. Define the objectives of the study
Although this seems straightforward, in practice it is not simple. Experimental designs are often complex and expensive to conduct, so they require the agreement of all parties involved. The output should be a clear statement of the problem or issue to be tackled. Everyone should clearly understand the aim.
2. Define the variables or factors of the process
Start by identifying as many variables/factors as possible; these are generally reduced through experimentation or assessment.
3. Define the output measure/s
Next, define how the output from the experiments will be measured. You can have more than one output measure, but they all must be reliable and robust. If the measure is not reliable, any differences observed in the data may be due to errors in measuring the output, rather than genuine differences caused by the changing factors.
We almost always recommend that a measurement study is completed prior to commencing the experiments so that any differences observed can be attributed to changes in the process, not to the way the measurements are taken.
4. Design a preliminary experiment
The aim of the preliminary experiment is to identify the variables that are active in comparison to those that are inert. We make the experiment as simple as possible, with the factors generally set at only two levels. The levels should aim to show the factor’s effect on the output.
Two levels are usually sufficient, but more can be chosen if required. At this stage, it is easy to discount factors and levels; but be careful--experimental designs often produce unexpected results. Also, at this stage, consider the methods that are going to be used to analyze the data. The way the data is collected can have a negative impact or hamper the effectiveness of the analysis tool.
5. Perform the preliminary experiment
Perform the experiment exactly to the plan. Do not be tempted to take short cuts or deviate from the original plan. Errors in experimental procedure can destroy the usefulness of the data. It is easy to underestimate the importance of this step: what seems to be a minor change can reduce the validity of the experiment and the final result. The experiment is repeated at least twice to improve the statistical validity of the data.
6. Assess the data from the preliminary experiment
The aim here is to find the factors that are active and to see if any of the factors interact together. This analysis does not need to be complex; simple analysis tools can be just as useful and are certainly more understandable than complex methods.
7. Design experiments to establish the optimum levels for the factors
Now take what you have learned from the preliminary experiment and design one, or a series of experiments, that will identify the optimum level for each of the factors, to give the best output result.
8. Conduct the experiments
Once again, follow the plan exactly--don’t deviate! As mentioned earlier, errors in experimental procedure can destroy the usefulness of the data. Each experiment is repeated at least twice to improve the statistical validity of the data.
9. Assess the data from the experiments
Our main aim is to find the optimum levels for the factors. The analysis does not need to be complex. Simple analysis tools can be just as useful and are certainly more understandable than complex methods.
10. Repeat the analysis, checking the optimum levels of the key factors
At least part of the analysis needs to be repeated to establish if the experimentation results are correct.
11. Conclusions and recommendations from the experiment
The analysis gives the results; explaining the results to others is the challenging part. Graphic methods are useful and should be utilized.
In summary, when beginning to put an experimental design together, some essential ingredients must be understood:
- The objective of the study.
- The method of experimentation to be employed.
- The analysis methods to be employed.
Experimental design will not be successful unless all these ingredients are known and understood from the outset. For example, if you don’t know how you are going to assess the results of the experiments, you cannot define the data collection method and the level of detail required.
The emphasis in an experimental design should always be to keep the design and analysis as simple as possible. It is easy to ‘over complex’ the situation and the result will be confusion and difficulty in explaining the results to others.
Copyright 2009 PQ Systems.
Please direct questions or problems regarding this web site to the Webmaster.