Note: This is post #3 of a series which focuses on product development best practices for software-enabled medical device teams that want to ship faster.

In our first post, we compared two approaches to product development of software-enabled devices: 1) traditional, document-based, and 2) modern, platform-based.

In our second post, we explored how a modern, platform-based approach can streamline Design History File (DHF) documentation, enhance traceability, and improve flexibility in product development of software-enabled devices.

In this post, Ultralight has continued our collaboration with Aaron Joseph–principal consultant with Sunstone Pilot and a seasoned expert in best practices for software-enabled devices–to explore how modern, paperless tools can transform V&V testing into a faster, more flexible, and error-resistant process.

The Challenge of Modern V&V

V&V testing is difficult! It’s a set of activities that are highly technical, process intensive, and involve cross-functional synchronization across the product development organization. Because regulatory submissions and ultimately product launch depend on completion of V&V, testers are often under intense time pressure. Modern software-enabled medical devices can make V&V even more difficult due to:

• Greater product complexity

  
  

• More software components and distributed architectures

  
  

• More changes late in development

Mistakes in V&V can be very costly in terms of project delays and regulatory approvals. Product teams therefore need all the help they can get for tackling V&V efficiently and flexibly.

Before we dive in, some clarification is in order on the difference between Verification and Validation.

Paper Pains

The traditional approach to V&V testing is document-based and involves writing and approving test protocols (test methods) then executing those protocols with pen-on-paper recording of test results then scanning the executed protocols and then assembling test reports. There are numerous shortcomings of V&V testing with paper documents:

• Error prone: poor handwriting, crossed out test results, results written in wrong place, missing results

  
  

• Difficult to manage repeated testing (due to design changes)

  
  

• Laborious to scan executed protocols and attach to test reports

  
  

• Difficult to translate into other languages

  
  

• Cannot electronically search the test results

  
  

  
  

How to support rapid design iterations of medical devices while still maintaining systematic, rigorous V&V testing?

Two Approaches to V&V

With “paperless V&V”, the software tools manage the testing and generate the test documentation. This allows the testers to focus on development of test methods and keeping up with product design changes instead of endlessly editing documents. This diagram compares the traditional document-based approach (top half) to the paperless V&V approach using The Hub (bottom half).  In The Hub the tester writes a series of “test cases” that define test methods and acceptance criteria. These test cases are grouped together into test protocol documents that the software automatically generates. Execution of the test cases is done in the software and the results are documented in test reports which are also automatically generated.

Paperless V&V with The Hub

Let’s see what this looks like with some examples. First, to verify a design requirement, a tester writes one or more test cases. The screenshot below shows what this looks like for an imaging test of a hypothetical medical device (an ultrasound system). The test case has a unique ID (“VER-1”) which allows it to be referenced throughout the product documentation. During test development this test case, and others, may be revised multiple times as the test method and test tools are refined or as the product design evolves. When finalized, the test case is approved as part of a test protocol document. 

Importantly, The Hub tracks links between requirements and test cases and flags any changes throughout development. If you’re a test engineer, you don’t want to find out when you submit your test protocol for signoff that a requirement you’re verifying has been changed. With The Hub everyone sees the same product information (a single source of truth) and can immediately be notified of any changes. For example, the software can flag a modified requirement that will require a revision to the test case linked to it.

Formal execution of each test case is performed in The Hub. The software serves up each test case and guides the tester step-by-step to run the tests and record all the required information. It records the tester name, date and time of execution, test results, and the pass/fail conclusion.

If any information for a test case is missing, the software will flag it as incomplete. 

Because the software is managing test executions, it can provide an instant summary of the status of V&V testing to anyone using The Hub. This is especially important when coordinating testing resources, scheduling external testing, and estimating the date of completion for V&V.

Documentation Automation

Automatic generation (and regeneration) of test documentation is a key advantage of The Hub. When test execution is complete the software generates detailed individual test reports such as the example below, a summary test report of all testing, and a traceability matrix linking requirements to test results. If retesting is required, which it often is, the software can readily regenerate the reports and traceability matrix to reflect the latest information.

Risk-Based Testing

Importantly, The Hub also keeps track of links between risks, requirements, and tests. This means a tester can see, by following the traceability links, if a requirement is safety-critical and deserves special treatment in testing.  For example, a hardware design requirement for a safety feature, driven by risk analysis, may require a larger sample size and more detailed testing for design verification. 

Because testers are able to see this risk information when developing their test methods, they can avoid expensive delays later on to rewrite a test protocol (for example, due to inadequate sample size or testing tools).

Conclusion

This platform-based approach of The Hub provides multiple advantages for V&V testing. 

• Visibility: during development and preparation for V&V, everyone on the team can see which requirements have test coverage and which don’t; during test execution everyone can see the status of testing

  
  

• Change control: the software tracks every change that’s made and can flag linked items that are affected

  
  

• Paperless: minimize chance of errors in recording test results

  
  

• Automation: quickly generate accurate test documentation at any point in time

In our next blog post we’ll discuss how The Hub supports “dynamic risk management” to allow rigorous control of risk analysis and tracking of risk controls through frequent design changes.

Aaron Joseph, principal consultant with Sunstone Pilot, is a biomedical engineer based in Waltham, Mass. With over 20 years of experience across a broad range of medical devices from surgical robotics to medical imaging to IOT and SaMD products, he helps clients efficiently tackle risk management and design controls for new product development.

Ultralight Labs develops software to streamline product development and documentation for the most innovative medical device teams. Book an Ultralight demo here.