MSA: Determining significant bias

By Jackie Graham, Ph.D.

This article continues our discussion of measurement system variation. Previously, we have examined the process of setting up a statistical assessment of a measurement system, discussed initial analysis calculating R&R (repeatability and reproducibility), assessed the variation in each of the appraiser’s results, and started to analyse for appraiser bias. This time we will look at the statistical significance of bias: when can we say that a statistically significant bias exists?

The same example data will be used to illustrate additional techniques for analysis. In this study, 5 samples were measured for length, using 4 appraisers. Each sample was measured twice. The results are shown below.

Sample 1

Sample 2

Sample 3

Sample 4

Sample 5

Appraiser 1

9.0

9.0

9.7

10.1

10.0

9.1

9.3

9.3

10.3

10.4

Appraiser 2

9.6

9.1

9.0

10.3

10.5

9.6

10.1

10.2

10.8

10.6

Appraiser 3

9.5

9.7

9.9

10.0

9.8

9.6

10.0

9.9

10.1

9.4

Appraiser 4

9.1

8.2

9.3

9.1

9.5

8.8

9.3

9.7

10.3

9.0

The R&R (repeatability and reproducibility) results are shown below.

Value

% of specification

Equipment variation EV

1.96

65.44

Appraiser variation AV (appraiser)

1.61

53.62

Repeatability and reproducibility R&R

2.54

84.60

The R&R% should be no more than 30%, making the R&R in this example study of 84.6% unacceptable. To begin finding the causes of this unacceptable variation, an average and range chart was constructed, see chart 1.

Text Box: Chart 1
Chart 1

Chart 1 was constructed in exactly the same way as any other average and range chart. However, the interpretation is quite different. Last time, we focused on differences in the data as shown in the average chart. We noted that appraisers 1 and 2 were somewhat similar in comparison to the other appraisers and appraiser 4 appears to be low and to have a bias. The question is whether appraiser 4 has a bias that is statistically significant.

The answer is found by using an analysis method called a main effect chart. This chart indicates whether the differences between the appraisers are statistically significant. The chart compares the overall average for each of the appraisers. The process is started by calculating the overall averages for each appraiser, using the data from the previous table.

Appraiser 1.
Data: 9.0, 9.1, 9.0, 9.3, 9.7, 9.3, 10.1, 10.3, 10.0, and 10.4.
Averaging this data gives an overall average of 9.62.

Appraiser 2.
Data: 9.6, 9.6, 9.1, 10.1, 9.0, 10.2, 10.3, 10.8, 10.5, and 10.6.
Averaging this data gives an overall average of 9.98.

Appraiser 3.
Data: 9.5, 9.6, 9.7, 10.0, 9.9, 9.9, 10.0, 10.1, 9.8, and 9.4.
Averaging this data gives an overall average of 9.79.

Appraiser 4.
Data was: 9.1, 8.8, 8.2, 9.3, 9.3, 9.7, 9.1, 10.3, 9.5, and 9.0.
Averaging this data gives an overall average of 9.23.

Differences in the overall averages ranging from 9.23 to 9.98 can be seen. These averages are plotted onto a main effect chart, as shown in chart 2.


Chart 2

This chart looks very much like an average chart; the difference is that special control limits are calculated. If any of the points lies outside these limits, this indicates that the related appraiser has a significant bias issue relative to the other appraisers.

If the main effect chart shows all the points inside the control limits, any difference in the appraisers is due to normal variation, and there is no significant bias.

In the example, the difference between appraiser 4 and the other appraisers is apparent. The chart demonstrates that the difference is significant. Now, we tend to assume that appraiser 4 is wrong and the other three are correct. But can we be sure? After all, maybe appraiser 4 is correct and the other three have a bias. One thing is for sure, there is a significant bias here, and it must be resolved. Whenever a bias issue is present, using different appraisers will impact the data appearing as a shift in a control chart. The bias must be resolved.

In this situation, the best approach is to repeat the measurement study using standard solutions or samples. Standard solutions have known values, so any bias can be determined, as the correct result is defined. It is essential when completing a study with known samples that the appraisers do not know the value of the standard sample, as this knowledge can affect the outcome of the study.

Summarizing the analysis in this measurement study, we found the following:

  • R&R was poor at 84.6%, far in excess of the maximum requirement of 30%.
  • The control chart showed considerable differences between the appraisers.
  • The inconsistency chart showed that no appraiser was more inconsistent than another.
  • The main effect chart showed a statistically significant bias issue.

This measurement system requires further investigation. The next step should be to complete a study using standard solutions or samples. Complete the same analysis as before. This time, any bias will be quantifiable because standard solutions or samples are used. An investigation and appropriate remedial action can be initiated to overcome the bias issue. Resolving the bias issue will significantly reduce the variation in the measurement system.

Frequently, when standard solutions are used, the R&R improves. Standard solutions are often easier to measure than actual products, making the measurement system look better compared to the reality of day-to-day testing. In order to fully assess measurement variation, it is necessary to use actual products or a mix of standard samples and products.

Originally published in the August 2008 edition of Quality eLine, our free monthly newletter.

Subscribe now!