Uncertainty Analysis is an Important Part of Validation

“Testing” is a term that covers a vast range of activities. Not every test is a measurement; some are visual inspections or can be qualified with nominal descriptions such as “pass or fail” or “hot or cold”. However, for tests that either are measurements or include measurements, understanding the uncertainty of those measurements is important to interpreting resulting data. An important component of test method validation is an uncertainty analysis that systematically assesses the factors influencing the measurement.

Instruments and the process of calibration to reference standards are not perfect. Neither is the person performing the test, the environment, or the test method itself. Therefore, every measurement has some uncertainty associated with it. The use of recognized standard procedures reduces many potential sources of measurement uncertainty, however even when the method is standardized, the uncertainty of the method should be understood.

image

A measurement is a function of all the input quantities that affect the measurement. For example, for the measurement of volume there might be three measurements (length, width, height). The uncertainty in the volume measurement can be determined by calculating and combining each measurement uncertainty along with the uncertainty associated with other factors (e.g., temperature or hydration).

Measurement uncertainty is a statistical estimate of experimental error. Thus, when estimating the uncertainty of a measurement, uncertainty components which are of importance in the given situation should be taken into account using appropriate statistics.

When a test method validation includes an uncertainty analysis to systematically assess the factors influencing the measurement, use of the test method for inappropriate applications can be avoided. The test method may also be improved, controlled, or monitored in areas of high uncertainty, ultimately building more confidence into the associated results.

For more information, please visit our website at  medinstitute.com.

 

 

 

The Goals of Test Method Validation

Test method validation is a documented process that is used to confirm that the procedure to be employed for a specific test is suitable for its intended purpose. The validation of a test method by a laboratory should be a planned activity. Validation plans should be updated as development proceeds. The plan should include a description of the method and describe the various validation activities.

When planning a test method validation, consider the scope of the test method and the associated risk (link to previous blog) before the necessary validation activities are selected. The scope of the method is defined in terms of the method’s purpose and the breadth of medical devices that the method is intended for. The range and accuracy of the test method should be adequate for the test method’s intended purpose. The activities selected should be a balance between the rigor needed for a credible validation, breadth of applicability of the test method, and associated effort.

A validation should establish that the test method has: image-2

  • Accuracy within the needed range
  • Repeatability
  • Reproducibility
  • Robustness

A test method that has adequate accuracy will have instruments that are calibrated to reference standards and will have compared well to other methods or compared well in an inter-laboratory study.

A test method that is repeatable will yield successive measurements in close agreement under tightly controlled conditions within the same laboratory.

A test method that is reproducible will yield measurements in close agreement from different laboratories or within the same laboratory over a longer period of time.

A test method that is robust will demonstrate minimal sensitivity to external factors, such as operator skill or changes in ambient conditions.

Validated test methods, used appropriately, provide data that can be relied upon to make important decisions related to the verification and validation of medical devices with confidence.

To have a more thorough discussion or to answer any questions you may have regarding the test method validation process please contact MED Institute via our website at http://www.medinstitute.com

Why do I need test method validation and what level of rigor is necessary?

The medical device industry has long understood the requirements related to process and software validation, however, US FDA Title 21 Code of Federal Regulations Part 820 Quality System Regulations does not have explicit requirements related to test method validation. Despite the lack of requirements, the FDA does have expectations around test method validation and has issued 483s and warning letters for insufficient method validation activities. Test methods that have not been validated yield a risk of the method being inadequate for evaluating medical devices and can lead to failed clinical devices, market recall and patient harm. Thus, to satisfy FDA expectations and avoid the pitfalls of using an inadequate test method, test methods used to produce data in support of regulatory filings or the manufacture of medical devices for human use need to be validated.

Validation

21 CFR 820.3 states: Validation means confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled. Method validation is a documented process that is used to confirm that the procedure to be employed for a specific test is suitable for its intended purpose.  Test method validation gives an overall understanding of uncertainty of the method. A validated method provides confidence that the method is appropriate and that the data generated are reliable and repeatable.

Incorporate Risk

When establishing test method validation activities, first consider the purpose and risk associated with the test. ISO 17025 states: The validation shall be as extensive as is necessary to meet the needs of the given application or field of application. The extent of validation activities required for a given method is driven by its intended purpose and closely related to the risk of patient harm. For example, a test for conformity of appearance may not require the same level of validation activities as a durability test replicating the clinical usage of the device.

aa-1-image-use

The level of validation should be commensurate with the risk associated with the test method being wrong.

Test method developers should determine the risk associated with the test method by evaluating the severity of potential harm and the probability that the harm occurs. With the purpose of the method and the associated risk assessed, the validation activities and level of rigor needed may be planned.

As an example, consider a test method to measure the length of medical device that is 100 cm +/- 1 cm long. The severity and probability associated with the device length being incorrect is low. Thus when planning the test method validation activities associated with a metal ruler, uncertainty determination including accuracy, resolution and repeatability is needed. However determination of the uncertainty related to the thermal expansion of a metal ruler isn’t likely warranted.

A deeper look at validation activities including uncertainty determination will be covered in future blog posts.

For more information, please visit our website at http://www.medinstitute.com

What Should You Do When a Testing Standard Doesn’t Exist? Part 5: Forming an Industry Consortium

In part four of this series, we talked about employing a third party to build testing equipment to develop a new test method. In this part, we’ll discuss forming an industry consortium as another means to develop a test method.

OMTEC Blog post 5 imageThrough the use of Cooperative Research and Development Agreements, an industry consortium can be formed. This can be a useful means of obtaining and sharing data that are expensive to collect.

Collaboration through an industry consortium can result in test method development, new instrumentation, and innovative approaches to shared problems. The knowledge and data developed in the industry consortium can be used to educate others and to support standard development.

For example, in 2006, SRI International, Stanford University, and a consortium of stent manufacturers teamed together to investigate loads on stents in the superficial femoral artery (SFA) and to improve the durability of peripheral vascular implants. An important outcome of this collaborative effort was the identification of bending, torsion, and axial deformations that would need to be simulated in the in vivo loading experienced by the stents. The results suggested possible fracture mechanisms and also provided important parameters for future stent design.

In the final part of this series, we’ll review the options you have when a testing standard doesn’t exist and discuss your recourse if the standards community disagrees with the testing method that you propose.

For more information, please visit our website at http://www.medinstitute.com

sign up

What Should You Do When a Testing Standard Doesn’t Exist? Part 4: Employ a third party to build testing equipment

In part three of this series, we talked about validating your own test method and gaining FDA approval. In this part, we’ll discuss another option to consider when a testing standard doesn’t exist: Employ a third party to build testing equipment to allow the standardization of the test method.

OMTEC Part 4 ImageMedical device technology combines engineering, science, and medicine to provide technical solutions to medical problems. The design of a new medical device presents a number of engineering challenges throughout the design process. A device can only be considered safe after undergoing tests that prove its safety. Demonstrating that a device is safe starts by devising the right tests.

For example, in the case of vascular stents, there was not a suitable method, nor was the appropriate equipment available to measure radial force.

The American Society for Testing and Materials (ASTM) attempted to develop a standardized method to measure radial force for over ten years without success. It wasn’t until Machine Solutions Incorporated developed new equipment that ASTM was able to standardize a test method capable of measuring stent radial force in a highly repeatable way. This equipment has become the industry standard and has been in use for many years.

This work led to the publication of ASTM F3067, “Guide for Radial Loading of Balloon Expandable and Self Expanding Vascular Stents”, in 2014. It is a guide that covers in vitro bench testing methods and equipment that can be used to measure the radial force, or collapse pressure, of vascular stents.

In the next part of this series, we’ll look at a final option to consider when a testing standard doesn’t exist.

For more information, please visit our website at http://www.medinstitute.com

sign up

What Should You Do When a Testing Standard Doesn’t Exist? Part 3: Develop and Validate your own Test Method

In part two of this series, we talked about working together with the standard organizations to develop a new test method. In this part, we’ll discuss another option to consider when a testing standard doesn’t exist: develop and validate your own test method and obtain FDA approval.Image

Test method validation is the documented process of ensuring that a test method is appropriate for its intended use. It is essential that a test method produce reliable results so that product quality and safety can be assured.

The rigor of the test method validation should be dependent upon the risk associated with the use of the test method. Risk is the combination of the probability of harm and the severity of harm. A high level of rigor in the test method validation results in a higher level of validation credibility, thus reducing the risk associated with the use of the test method.

Test method validation is recommended when a new method is developed or an established method is revised. It can also be a useful tool to compare methods used for the same purpose.

Validating your own test method can be a quicker process than working through a standards organization. One disadvantage to this process is that you will be working without external input from industrial peers. Additionally, once the FDA deems your test method unacceptable, continuing to work with the FDA to gain their approval can be an expensive and time consuming process.

In the next part of this series, we’ll look at more options to consider when a testing standard doesn’t exist.

For more information, please visit our website at http://www.medinstitute.com

sign up

4 common medical device testing questions and how to find balance from an engineering perspective.

Metcalf_Justin_001.jpg

Being an engineer in the medical device profession for the past 15 years has been quite an education.  Working with a team of talented engineers has been fun and together we’ve been able to learn from each other and solve almost any problem from an engineering perspective.  Our team has worked on several first to market Class II and Class III medical devices from the ground up, these experiences have been very helpful for us in transitioning from a large medical device company to a consulting firm.  As anyone who has worked in this field knows, sometimes you learn the most when your device fails, but it is what you do after your device fails that defines your career.  To sum up in a few words, what we’re really trying to do as medical device engineers is  find balance between risk and benefit.  Engineers write standards, guidance documents, papers and books to communicate how to find the right risk balance. Even with appropriate references in place, we have several questions that come up during most device development lifecycles.

1. When do you justify and when do you test?

You can have the perfect scientific justification to not test a device, yet the 1000 words that it will take to justify often become more expensive than a simple test. Tests have their own associated risks and costs.  How do you find the balance?

2. How do you choose the right confidence and coverage for a potential failure mode?

Often times test plans are developed with 90% confidence that 90% of the population will not encounter a failure mode. How does one know that this confidence interval is good enough for patient safety? This is a tough question that when put into a FMEA procedure can become systematic.  I’m not sure this should be a systematic question when a potential failure mode is death. How do you find the balance?

3. When is the worst case model a bad enough case?

The terms worst case conditions or worst case model often come up during device development.  What is the worst case scenario and will my device survive when this scenario happens?  And if you design your device for the worst case, what happens to the rest of the population?  Questions like this are hard to answer and can slow product development to a halt. How do you find the balance?

4. How close to the acceptance criteria are too close?

What if your safety factor is right on top of the acceptance criteria?  How does one justify questions like this?  Is there uncertainty analysis incorporated and what was that acceptance criteria based upon anyway?  When pushing the limits of device design to treat a new disease, questions like this arise. How do you find the balance?

Moving a medical device from concept to clinical use is a rewarding accomplishment.  There is quite a grind along the way and many tough questions to answer.  Finding your way through these questions is difficult, but when you get it right, patients benefit.  When you do find the right balance, everyone wins!

Visit our website to learn more about our services.

sign up

What Should You Do When a Testing Standard Doesn’t Exist? Part 2: Working through Standards Organizations

In part one of this series, we discussed how medical device technology can frequently outpace the development of standardized testing methods. One potential solution to this dilemma is to work with the standard organizations to modify an existing standardized method or to publish a new test method.

ReviseStandards committees are typically very receptive to suggestions that might improve an existing standard.

The modification of an existing standardized method can be of great benefit to your company as well as to the industry as a whole. Revisions can occur as often as necessary and can reflect the rapid changes in technology.

ImageWorking together with the American Society for Testing and Materials (ASTM), we helped to publish a new guide for the axial, bending, and torsional durability testing of vascular stents. Each company involved was able to incorporate their own validated methods into this new standard.

Helping to develop new standards can be a time consuming process, but it allows you to collaborate with others in the industry that are struggling with the same issues and can often be an opportunity to build industry synergy and world-wide acceptance of universal design criteria.

In the next part of this series, we’ll look at more options for establishing a new test method.

For more information, please visit our website at http://www.medinstitute.com

What Should You Do When a Testing Standard Doesn’t Exist? Part 1: Introduction

This is the first part in a six-part series that will help explain your options.

When a company has a new design for a medical device, appropriate testing is essential to demonstrate the safety and effectiveness of the device for its intended use. It is convenient when there is an applicable testing standard, but that is not always the case.

OMTEC Part 1_ASME    OMTEC Part1_ASTM    OMTEC Part 1_AAMI.png   OMTEC Part 1_ISO.png

Medical device technology typically advances more quickly than the creation of standard test methods; therefore, many medical device industries have a limited number of standard test methods available for their use.

Without a standard test method in place for a particular type of device, there is an increase in risk. Risk for the patient, risk for the regulatory body and financial risk for the medical device company.

When a suitable testing standard is unavailable, you have the following options:

  • Work with the standard organizations to develop a test method.
  • Develop and validate a new test method and gain FDA agreement.
  • Approach a third party to build testing equipment to allow the standardization of the test method.
  • Form an industry consortium to develop consensus.

In the next part of this series, we’ll look at how you can partner with the standard organizations to develop a test method.

For more information, please visit our website at medinstitute.com

sign up

Medical Device MRI Safety Testing: Where should a hip implant be placed in an ASTM F2182 test to measure the maximum RF-induced heating?

As mentioned in our previous case study , testing the radio-frequency-induced tissue heating by implanted medical devices is important to prevent harm to patients during MR imaging. The ASTM F2182 standard describes the method for testing to evaluate MRI safety with respect to RF-induced heating.  The ASTM standard defines a phantom for testing (i.e., rectangular acrylic container filled with conductive gel) that acts as an approximate simulation of the human body. However, many devices have asymmetrical or complex shapes (e.g., hip implants) that present a challenge for the safety test engineer:

  • Where and in what orientation should the device be placed within the ASTM F2182 phantom for adequate safety assessment?
  • Where on the implant should the temperature probes be placed?

Image

The location of the implant’s placement in the phantom also affects RF-induced heating as shown by the highly non-uniform electrical field magnitude in the ASTM F2182 phantom (see figures at right). In most cases, physical testing to determine the worst case orientation and location within the phantom as well as the location of the temperature probes would require significant effort.

Computational simulation is a Image2cost effective way of determining the worst case orientation and placement of the implant within the phantom. Using the computational simulation predictions, locations may be selected to place temperature probes for the physical tests.
See an animation of a computational simulation of an example hip implant that is positioned in the worst case location and orientation within the ASTM F2182 phantom below. From the results of this cost effective simulation, MRI safety device testing may be performed with the implant in the worst case location and the temperature probes placed to identify the locations of maximum heating.

 

For more information, please visit our website http://www.medinstitute.com/.

sign up