Calibration Risk Assessment: Where do you start?

We recently received a question from one of our blog readers in response to a blog post on instrument calibration...

Dear Paul,
I have a question about Risk Assessment analyses for product impact from sensing instruments that have gone out of calibration, specifically in GxP-compliant applications. Do you have any recommendations on books, training courses, or other resources for managing this?
Thanks!/I

Dear I,
That’s a big question!  First, let’s imagine the situation: The instrument in question was last calibrated 6 months ago.  Calibration is now due and you find the instrument is out of spec.  Let’s say it's a thermometer and it should be ±0.5°C, but we find that it is off by +1.5°C.  So we have a +1.0°C deviation. We actually have no data to tell us exactly when it went out of calibration.  It’s easy to think it was a slow change and it just slid towards +1.0°C.  But, the truth is that we actually don’t know this, it could have been out by +3.0°C a week ago.
The first thing we need to do is quarantine the instrument, then we investigate the situation.  The instrument should not be adjusted or placed back into service until we know what went wrong.  Likely we will want to open a CAPA so that we can determine, if possible, why the instrument has gone out of specification. This knowledge may help prevent the problem from reoccurring.  Additionally, it can guide the other part of our investigation – determining if there was any impact on product quality.  

Hopefully, we get lucky and we figure out what went wrong, then maybe we can place the failure at a specific point in time, such as during a cleaning or preventive maintenance event.  Likely, we won’t know this. But the nature of the instrument going out of spec, combined with information from the instrument vendor, can help understand the failure enough that we can make some good guesses as to whether the failure was gradual or instantaneous, constant or fluctuating, or if it was constantly one-directional or flipping in both directions.  This information will guide further investigation.

Determining impact on product quality is a lot harder.  You really need to understand your manufacturing process, the properties of your product, and how a change in the variable under investigation (in this case +1.0°C) will affect the product at that point in its manufacturing lifecycle. Some of the time we get lucky (again) and see that there was no impact (such is the case if we know that the deviation was in only one direction and was not variable).  For instance, if the specs for the process are 2-8°C and a review of the temperature history shows that the (failed) device gave us values of 4 to 7°C, then we can assume that although the device was reading high, the actual values would have been 3 to 6°C, and therefore within specification.

However, we may need a much deeper investigation.  In this case, background information can help so that we can draw on similar investigations and other information such as stability data to determine the impact.  We may need to pull retained samples from past lots (during a time when the device could have been out of calibration) and test them to see if there was any quality impact.  We may actually need to do a recall if we discover a meaningful quality issue.

Finally, we need to accept that we might not find the answer.  If, however, we find enough information to determine whether or not there was negative product impact, we can protect the health of our customers.

The ability to perform this sort of analyses is dependent on a few important bits of expertise:
1)Good diagnostics, perhaps with vendor support, to determine the cause, nature, and timing of the failure.
2)Solid process knowledge for your product to guide your investigation to determine if there was a quality impact. This is much easier with a robust quality system with well-documented product history.

Thanks for reading our blog and thanks for sending your question! Let me know if you have any further questions...
Yours,
Paul

BONUS Post:

Our Continuous Monitoring System Product Manager Jon Aldous weighed in on this question as well:
Jon wrote:  "If process parameters are being continuously monitored and alarmed correctly, there are a few things that can indicate an out-of-spec instrument. You may see multiple nuisance alarms or a linear change in value trends that is noticeably different from the trends seen immediately after the last calibration. Typically you need to go back to the last calibration and determine the latest monitored values and then compare trends over time."

PLUS: Our Calibration Development Manager Michael Boeztkes has written about this problem ("my instrument's out of spec... now what?") for PharmaManufacturing.com.