Summative evaluation is one of the methodology specified by the IEC 62366 for the evaluation of the usability of medical devices. Usability is one of the key factors to take in consideration for design of medical devices and it is highly connected with safety, labelling and risk management.

We have already been discussing about usability, and in particular about the main requirements associated to IEC 62366. Moreover, within our QualityMedDev Shop, a checklist to support compliance to IEC 62366 has also been made available.

In this article, we will discuss in details about summative evaluation, that is one of the key factor for the Human Factor Engineering Process and the evaluation of the usability for a medical device.

Few Considerations about Summative Usability Evaluation

Summative evaluation is the last step of the usability process and it basically coincides with the so-called usability validation.

The requirements for Summative Evaluation of usability for medical devices have been established within IEC 62366. The process of usability evaluation can be summarised based on the following steps:

• Preparing use specification
• Identifying UI characteristics related to safety and possible use errors
• Identifying known or foreseeable hazards and hazardous situations
• Identifying and describing hazard-related use scenarios
• Selecting hazard-related use-scenarios for a summative evaluation
• Establishing a UI specification
• Establishing a UI evaluation plan (formative and summative evaluations)
• Implementing and verifying the UI
• Validating the UI

If we focus specifically on the summative evaluations we can say that is a formal evaluation with specific acceptance criteria. The summative evaluation shall be performed on the final product, on the version that will be brought to the market after regulatory approval. The primary goal of the summative evaluation of usability is to get an objective evidence that the medical device is safe to use. This means that the chance of committing dangerous use errors is minimized.

Usually summative evaluation is performed for specific selected hazard-related use scenario. It is possible to perform a single test event or multiple tests , depending on the nature of the product and variation of use cases. If during summative evaluation it turns out that specific risk control measures are not adequate to mitigate specific risks, then the usability process shall restart and the formative evaluation shall be repeated.

Requirements for Summative Evaluation

Based on the IEC 62366, there are specific requirements associated to the summative evaluation for medical device usability assessment.

First fo all, there are specific requirements associated to the preparation of the summative usability plan, that shall contain the following elements:

• the evaluation method being used and a rationale that the method produces objective evidence;
• which part of the user interface is being evaluated;
• where applicable, the criteria for determining whether the information for safety is perceivable, understandable and supports correct use of the medical devices;
• the availability of the accompanying documentation and provision of training during the summative evaluation;
• For a usability test:

– the test environment and conditions of use and a rationale for how they are adequately representative of the actual conditions of use; and

– the method of collecting data during the usability test for the subsequent analysis of observed use errors.

Steps to Properly perform a Usability Evaluation

In this last paragraph, we want to provide the overall steps that need to be considered for the design of a summative usability study. The general steps to do are the following:

• Creating and Finalizing the Test Plan : this involves Choosing the Test Location, Test Personnel, Choosing the Tasks, Pilot Testing
• Recruiting Participants;
• Conducting the Test;
• Analyzing the Data;
• Writing the Report;

The activity related to the recruitment of the participants is of key importance and the following steps shall be undertaken:

• Take a decision on the number of participants;
• Definition of the user profile;
• Perform an initial evaluation of the participants through specific screening questionnaire;
• Inviting participants to actually perform the test.

Subscribe to QualityMedDev Newsletter

QualityMedDev is an online platform focused on Quality & Regulatory topics for medical device business; Follow us on LinkedIn and Twitter to stay up to date with most important news on the Regulatory field.

QualityMedDev is one of the largest online platform supporting medical device business for regulatory compliance topics. We provide regulatory consulting services over a broad range of topics, from EU MDR & IVDR to ISO 13485, including risk management, biocompatibility, usability and software verification and validation and, in general, support in preparation of technical documentation for MDR.

Our sister platform QualityMedDev Academy provides the possibility to follow online and self-paced training courses focused on regulatory compliance topics for medical device. These training courses, developed in collaboration with highly skilled professionals in the medical device sector, allows you to exponentially increase your competencies over a broad range of quality and regulatory topics for medical device business operations.

Do not hesitate to subscribe to our Newsletter!