Beyond Metrics - Part 1: An introduction to meaningful metrics

By Ian Mullan, A&O co-founder, 10 March, 2021.

How often do we hear in our weekly clinical trial team meeting the data manager verbally spouting study backlogs in a monotonous tone? A ten minute segment that could have been communicated via an email!

Site 120, 20 open queries, 8 of those 30+ days, 5 visits backlogged. Site 450, 50 open queries, 18 30+ days, 11 visits backlogged, etc, etc”.

While an informative and necessary as part of study conduct, metrics as flat as the tone they are presented in (believe me, no one is more bored than the data managers themselves). They are rarely provided with context! What if site 450 has 3 times as many patients for example? Any metric visualisations are rare or non-existent (too complex/time consuming to create and getting your head around Tableau is another thing). And, to cap it off, it might well have taken the data manager an hour or two to compile, only for other savvy team members to be doing it for themselves anyway. There's too much duplication in the clinical arena already!

They generally tell the audience counts and whole numbers, but never what needs to be done and what needs to prioritised.

Indeed, there may be duplicated efforts as different functional teams may need to have metrics available at different times due to meetings (cross-functional vs DM, Ops etc), with a specific focus on their own subset of tasks. How many hours have now been invested into these efforts and do the parties actually end up with the same metrics?

What’s missing here is a unifying ‘metrics tool’. So what should a ‘metrics tool’ be? Ideally, it should be an on-demand, visually pleasing, and informative tool that is available in the same format from study to study, using a platform that every team member has available.

Figure 01: a universal metrics tool should be able to handle all types of study builds and data sources for maximum familiarity and reach

What do I mean by ‘on-demand’? I mean a tool that can be spoken to, and speak for itself, in live time. For example, a screenshare session in which a monitor might ask for query ageing for a particular set of sites; press a button or use a dropdown, and there it is. No need to drag and drop columns, create queries and wait for a few minutes after you're hit 'RUN'.

Figure 02: a metrics tool needs to be quick-fire with metrics relevant to its audience, as per this Excel example. Learn more:

Not everyone likes tools with bright lights and whistles, so it is important that a metrics tool can be adapted accordingly. Personally, I feel introducing dynamic visuals only serves to help bring focus where needed.

Figure 03: a metrics tool can have visual elements introduced, as per this Excel example. Learn more:

After 20 years in the industry, barring the occasional company with well-developed own-internal data management systems, there remains still the gap of a single unifying tool that can harness, in relative live time, data extracts that provide a full context and visuals study teams can rely upon.

Contextual metrics

Before going any further, I want to introduce a concept: Contextual Metrics. What do I mean by this? In its simplest form, as in the above example, a site might have more query backlogs than another, but might also have many more patients. Taken into context, the site with the highest number of backlogs might not be the most under-performing, nor the site that will jeopardise your interim analysis timelines or the one that needs a monitoring visit ASAP.

Taking to the next level, metrics is not simply about quantity, but quality: the number of queries per 1000 datapoints is a prime example. Another level, further, is to not penalise or lose focus on a site with historical performance issues that may have improved. Indeed, why not look at queries per 1000 datapoints over a recent 2 or 3 month period, for example?

And there I have just introduced an additional concept: 'time'. Why not create a tool where the metric items have metadata in the background associating them with dates? Visit backlogs is an easy example (associated with the visit date), and many interim locks focus on dates, so a tool can have a data-cut date introduced bringing a new degree of relevance to metrics.

Figure 04: a metrics tool should have the ability to cut by date or by visits, as per this Excel example taken from A&O's PK Reconciliation tool. Learn more:

Contextual metrics is nothing new, at least it shouldn't be, but I'm constantly surprised at the lack of its inclusion when working with clients. It's really not difficult but just requires some planning, end user interaction (before the specifications are finalised) and someone experienced to undertake the project.

CRF + External data sources = more meaningful metrics

Aside from the CRF/EDC metrics, what about including external data metrics? Missing samples, PK/PD sampling information, IWRS information, patient diary information, etc? These are, after all, also clinical data that will require reconciliation at some point – so why assume these always fall under another department’s umbrella? To have an effective metrics approach requires a cohesive methodology that will rely on access (obvious but not always easy!) and and understanding of what the data is (misunderstanding variables can create some 'interesting' metrics!)

The examples can go on and on, deeper and deeper, but I think the point here is made. Look out for my next blog entry where I discuss why so many metric projects fail to hit the mark.

Search By Tags
Follow Us
  • Facebook Basic Square
  • Twitter Basic Square
  • Google+ Basic Square

Allow us to help you sort your apples from your oranges!


  • Black YouTube Icon
  • Black LinkedIn Icon
  • Black Facebook Icon

© 2017-2020 Meijboom Consulting Ltd trading as Apples & Oranges Data Solutions