Summary of Diagnostic Studies

This panel is used for summarizing the diagnostic studies

1. The format of input data

  • The variable names must contain TP, FP, TN, and FN, denoting True Positive, False Positive, True Negative, and False Negative
  • The order of TP, FP, TN, FN does not matter
  • No missing values in the data
  • Re-click the "Update data and results" button, after you revised the data

2. You can get the following results:

  • The scatter plot of the studies with the corresponding confidence interval
  • The descriptive statistics of sensitivity (Sens), specificity (Spec), diagnostic odds ratio (DOR), and likelihood ratios (LRs) of each study
  • Test of equality of Sens and Spec
  • Forest plots of Sens, Spec, DOR, LRs

Example: real-word meta-analysis of diagnostic studies

The example insert is the meta-analysis for diagnosing intravascular device-related bloodstream infection.

Reference: Safdar N, Fine JP, Maki DG. Meta-analysis: methods for diagnosing intravascular device–related bloodstream infection. Ann Intern Med. 2005;142(6):451-466. doi:10.7326/0003-4819-142-6-200503150-00011


Data Preview




Note: logit transformation is $\mathrm{logit}(x)=\log(\dfrac{x}{1-x})$
  • Sens: sensitivity$=\dfrac{TP}{TP+FN}$; Spec: specificity$=\dfrac{TN}{TN+FP}$
  • y1: $\mathrm{logit}(\mathrm{Sens})$; y2: $\mathrm{logit}(\mathrm{Spec})$
  • v1: variance of y1; v2: variance of y2

Descriptive Statistics


1. Configurations of the following plot
2. ROC scatter plot of the Sens and Spec of each study


1. Summary of sensitivity (Sens) and specificity (Sepc)
Note:
  • Sens: sensitivity$=\dfrac{TP}{TP+FN}$
  • Sens.CI.lower: lower bound of CI; Sens.CI.upper: upper bound of CI
  • Spec: specificity$=\dfrac{TN}{TN+FP}$
  • Spec.CI.lower: lower bound of CI; Spec.CI.upper: upper bound of CI

2. Test of equality

3. Forest plots


1. Summary of the diagnostic odds ratio (DOR)
Note:
  • DOR: diagnostic odds ratio
  • DOR.CI.lower: lower bound of CI; DOR.CI.upper: upper bound of CI
  • lnDOR.se: standard error of log-transformed DOR

2. Forest plots


1. Summary of the positive or negative likelihood ratios
Note:
  • PosLR: positive likelihood ratio
  • PosLR.CI.lower: lower bound of CI; PosLR.CI.upper: upper bound of CI
  • lnPosLR.se: standard error of log-transformed positive likelihood ratio
  • NegLR: negative likelihood ratio
  • NegLR.CI.lower: lower bound of CI; NegLR.CI.upper: upper bound of CI
  • lnNegLR.se: standard error of log-transformed negative likelihood ratio

2. Forest plots

The Reitsma Model and the Generalized Linear Mixed Model (GLMM) for Meta-Analysis of Diagnostic Test Accuracy

This panel is used for meta-analysis of diagnostic studies without accounting for publication bias.

Two bivariate models are available: linear mixed model (LMM, or the Reitsma model) and generalized linear mixed model (GLMMM, or GLM model)

1. About the Reitsma model

  • This is the random effects model based on the bivariate normal distribution
  • The parameters of interest are: the summarized sensitivity and specificity, and the SAUC
  • When data are sparse (have many 0 values), this model is not recommended

Reference: Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58(10):982-990. doi:10.1016/j.jclinepi.2005.02.022

2. About GLMM

  • This model is a mixture of binomial model and normal model
  • The parameters of interest are: the summarized sensitivity and specificity, the the SAUC
  • When data are sparse (have many 0 values), this model is recommended
  • When data are not sparse, this model has similar results with the Reitsma model

Reference: 1. Chu H, Cole SR. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol. 2006;59(12):1331-1332. doi:10.1016/j.jclinepi.2006.06.011

2. You can get the following results:

  • The estimated summary ROC (SROC) curve
  • The estimates from the Reitsma model
  • The estimates from the GLM model
See more details in the Help and Install App panel.

Meta-analysis


Summary ROC (SROC) Plot


Estimates from the model

                    
Note:

Fixed-effects coefficients: the estimates in the between-study level

  • tsens.: the estimated summarized sensitivity in the logit-scale, $\mu_1$ in the Reitsma model
  • tfpr.: the estimated summarized 1-specificity (false positive rate) in the logit-scale, $-\mu_2$ in the Reitsma model
  • sensitivity: the estimated summarized sensitivity, logit-scaled $\mu_1$ in the Reitsma model
  • false pos. rate:: the estimated summarized 1-specificity (false positive rate), logit-scaled $\mu_2$ in the Reitsma model

See the details of the Reitsma model in Help and Download panel


Estimates from the model

                    
Note:
see the details of the GLM model in Help and Download panel

Random effects: the estimates in the within-study level

Fixed effects: the estimates in the between-study level

  • sens.: the estimated summarized sensitivity in the logit-scale
  • spec: the estimated summarized specificity in the logit-scale

See the details of the GLM model in Help and Download panel


Funnel Plots for Detecting the Existence of Publication Bias

This panel presents funnel plots for detecting potential publication bias on lnDOR, logit-scaled Sens and Spec.

Funnel plot method is not suggested unless all the studies adopt the common cutoff values for diagnosis

You can get the following results:

  • The funnel plot with trim-and-fill for lnDOR, logit-scaled Sens and Sens
  • The test asymmetry of the funnel plot

Trim-and-fill Plot


              



Likelihood-based Sensitivity Analysis for Publication Bias

This panel is used for sensitivity analysis on the estimates of meta-analysis of diagnostic test accuracy.

Publication bias (PB) is the threat to the validity of the estimates. The method in this panel provides a method to quantify the magnitude of the potential PB.

1. The input parameter

  • Marginal selection probability ($p$): the expected proportion of published studies from the population; while $1-p$ indicated the expected proportion of the unpublished
For example, there are 33 studies in the Example. When we assign $p=0.9$, we expect
  • 90% studies (33 studies) were published from the population ($33/0.93 \approx 7$ studies)
  • 10% studies ($37 \times 0.1 \approx 4$ studies) were not published or missing not at random
When we assign $p=1$, we expect there was no unpublished studies.

2. Other parameters

Contrast $c_1, c_2$: it defines the mechanism of selective publication process
  • $c_1=c_2$: the selective publication mechanism is influenced by both Sens and Spec; (the $t$-type statistic of lnDOR influences PB)
  • $c_1=1, c_2=0$: the selective publication mechanism is influenced in by Sens only; (the $t$-type statistic of Sens influences PB)
  • $c_1=0, c_2=1$: the selective publication mechanism is influenced in by Spec only; (the $t$-type statistic of Spec influences PB)
  • $\hat c_1, \hat c_2$: the selective publication mechanism is estimated from data

3. Optional parameters in the optimization

  • Initial value of $\beta$: the initial value of $\beta$ in the optimization
  • Initial value of $c_1^2$: the initial value of $c_1^2$ in the optimization
  • Range of $\beta$: lower and upper constraint of $\beta$ in the optimization
  • Range of $\alpha$: lower and upper constraint of $\alpha$ in the optimization

4. You can get the following results:

  • The bias adjusted SROC curve, given the input marginal selection probability
  • The bias adjusted SAUC, given the input marginal selection probability
  • The bias adjusted $\mu_1, \mu_2, \tau_1, \tau_2, \rho$, given the input marginal selection probability
See more details in Help and Install App panel.

Sensitivity analysis on SROC curves under four scenarios of selective publication mechanisms
Note: there may be issues of non-convergence in the results; in such case, some results are missing.

For the detailed estimates, please check the results table in the Results Table.

A. c1, c2 are estimated

B. c1=c2

C. c1=1, c2=0

D. c1=0, c2=1

E. c1, c2 are specified


Sensitivity analysis on SROC curves under four scenarios of selective publication mechanisms
Note: there may be issues of non-convergence in the results; in such case, some results are missing.

For the detailed estimates, please check the results table in the Results Table.


Sensitivity analysis on SAUC under four scenarios of selective publication mechanisms
Note: click the button to start calculation at the first time.

There may be issues of non-convergence in the results; in such case, some results are missing.

A. c1, c2 are estimated

B. c1=c2

C. c1=1, c2=0

D. c1=0, c2=1

E. c1, c2 are specified


Sensitivity analysis on SAUC under four scenarios of selective publication mechanisms
Note: there may be issues of non-convergence in the results; in such case, some results are missing.


A. c1, c2 are estimated


B. c1=c2


C. c1=1, c2=0


D. c1=0, c2=1


Help and Reference

Note: click the "Help" button on the top

Models in this App

1. The Reitsma model

Suppose that $N~(i=1, \dots, N)$ diagnostic studies are included in meta-analysis. In other words, there are $N$ rows in the input data. The observed data are the number of true positives (TP), true negative (TN), false positives (FP), and false negatives (FN), so sensitivity (Sens) and specificity (Spec) can be estimated from data.

In the within-study level, we denote that,

  • $y_{1i}$ and $y_{2i}$: the logit-Sens and logit-Spec after continuity correction
  • $s_{1i}^2$ and $s_{2i}^2$: the observed variances of $y_{1i}$ and $y_{2i}$ within each study
  • $\mu_{1i}$ and $\mu_{2i}$: the logit-transformed true Sens and Spec of the $i$th study
Given $(\mu_{1i}, \mu_{2i})$, it is assumed that: \begin{align} \binom{y_{1i}}{y_{2i}} \sim N \left (\binom{\mu_{1i}}{\mu_{2i}}, \boldsymbol{\Sigma}_i \right ) \mathrm{~with~} \boldsymbol{\Sigma}_i = \begin{pmatrix} s_{1i}^2 & 0\\ 0 & s_{2i}^2 \end{pmatrix}, \label{eq:b2} \end{align}

In the within-study level, we denote that,

  • $\mu_1$ and $\mu_2$: the common means of the logit-transformed sensitivity and specificity
  • $\tau_1^2$ and $\tau_2^2$: between-study variances
  • $\tau_{12} = \rho\tau_{1}\tau_{2}$ is the covariance between $\mu_{1i}$ and $\mu_{2i}$, $\rho~(-1 \le \rho \le 1)$ is the correlation coefficient
It is assumed that $(\mu_{1i}, \mu_{2i})^T$ is normally distributed: \begin{align} \binom{\mu_{1i}}{\mu_{2i}} \sim N\left ( \binom{\mu_1}{\mu_2}, \boldsymbol{\Omega} \right ) \mathrm{~with~} \boldsymbol{\Omega} = \begin{pmatrix} \tau_1^2 & \tau_{12} \\ \tau_{12} & \tau_2^2 \end{pmatrix}, \label{eq:b1} \end{align}

The combination of \eqref{eq:b2} and \eqref{eq:b1} leads to the Reitsma model:

\begin{align} \boldsymbol{y}_i | \boldsymbol{\Sigma}_i \sim N_2 \left (\boldsymbol{\mu}, \boldsymbol{\Omega} + \boldsymbol{\Sigma}_i \right ), \label{eq:b12} \end{align} where,$\boldsymbol{y}_i = (y_{1i},y_{2i})^T$, $\boldsymbol{\mu} = (\mu_1,\mu_2)^T$

The Reitsma model requires the number of diseased and non-diseased to be large.

2. Generalized linear mixed model (GLMM)

GLM model differs the Reitsma model in the within-study level.

Suppose,

  • $n_{00i}, n_{11i}, n_{01i}, n_{10i}$: TP, TN, FP, FN; $N_{1i}=n_{11i}+n_{10i}, N_{0i}=n_{00i}+n_{01i}$
  • $s_{1i}^2$ and $s_{2i}^2$: the observed variances of $y_{1i}$ and $y_{2i}$ within each study
  • $\mu_{1i}$ and $\mu_{2i}$: the logit-transformed true Sens and Spec of the $i$th study

In the within-study level, it is assumed that:,

\begin{align} n_{00i}\sim \text{Binomial}(N_{0i}, \text{Spec}_i); ~~~~ n_{11i}\sim \text{Binomial}(N_{1i}, \text{Sens}_i) \label{eq:b3} \end{align}

In the between-study level, we denote that:

  • $\mu_{1i}=\text{logit}(\text{Sens}_i)$ and $\mu_{2i}=\text{logit}(\text{Spec}_i)$
  • $\mu_1$ and $\mu_2$: the common means of the logit-Sens and logit-Spec
  • $\tau_1^2$ and $\tau_2^2$: between-study variances
  • $\tau_{12} = \rho\tau_{1}\tau_{2}$ is the covariance between $\mu_{1i}$ and $\mu_{2i}$, $\rho~(-1 \le \rho \le 1)$ is the correlation coefficient

In the between-study level, $\mu_{1i}, \mu_{2i}$ have the same distribution with \ref{eq:b2}

The combination of \eqref{eq:b3} and \eqref{eq:b1} leads to the GLM model.

GLM model does not require continuity correction.

3. SROC and SAUC

The SROC curve: \begin{align} SROC(x; \boldsymbol{\mu}, \boldsymbol{\Omega}) = \mathrm{logit}^{-1} \left[ \mu_1 - \dfrac{\tau_{12}}{\tau_2^2}\{\mathrm{logit}(x)+\mu_2\} \right]. \label{eq:sroc} \end{align} The HSROC curve is given by the SROC curve \eqref{eq:sroc} with $\rho=-1$. The SAUC: \begin{align} SAUC(\boldsymbol{\mu}, \boldsymbol{\Omega}) = \int_{0}^{1}SROC(x; \boldsymbol{\mu}, \boldsymbol{\Omega})dx. \label{sauc} \end{align}

4. Likelihood based sensitivity analysis method

Publication bias is the phenomenon that studies with significant results are more likely to be published or selected for meta-analysis. In meta-analysis of DTA, we consider to model the selective publication of each study by the selection function \begin{align*} P(\text{Publish}|t_i)=\Phi(\alpha+\beta\times t_i\}, \end{align*} which is the probit model on the $t$-type statistic of the linear combination of the logit-transformed sensitivity and specificity: $ \boldsymbol{c}^T \boldsymbol{y}_i = c_1y_{1i}+c_2y_{2i}, $ The $t$-type statistic of the linear combination is $ t_i = {\boldsymbol{c}^T \boldsymbol{y}_i}/{\sqrt{\boldsymbol{c}^T\boldsymbol{\Sigma}_i\boldsymbol{c}}}. $
When $(c_1, c_2) = (1/\sqrt{2}, 1/\sqrt{2})$, it gives the $t$-statistic of the lnDOR. By taking different contrast vectors, the $t$-type statistic can determine a variety of selective publication mechanisms. For example, $(c_1, c_2) = (1, 0)$ and $(c_1, c_2) = (0, 1)$ indicate that the selective publication mechanisms are determined by the significance of sensitivity and specificity, respectively. The contrast vector $\boldsymbol{c}$ can be regarded as unknown parameters to be estimated or specified by user.
In the sensitivity analysis, we introduce the marginal probability of selective publication, $p=P(\mathrm{select})=E\{P(\text{Publish}|t_i)\}$. The estimating model based on the joint distribution of observed data involving the selection probabilities can be constructed. Given various specified values of $p$, the bias-adjusted SROC curves and SAUC can be estimated from the maximum likelihood estimation.

R packages used in this Application:

  • mada, DT, ggplot2, plotly, shiny, shinythemes, shinyWidgets, mvmeta, mada, shinyAce, latex2exp, base,

References:

  • Rutter, Carolyn M. and Gatsonis, Constantine A. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001; 19:2865--2884
  • Reitsma, Johannes B. and Glas, Afina S. and Rutjes, Anne W.S. and Scholten, Rob J.P.M. and Bossuyt, Patrick M. and Zwinderman, Aeilko H. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005; 10:982--990
  • Zhou, Yi and Huang, Ao and Hattori, Satoshi. A likelihood based sensitivity analysis for publication bias on summary ROC in meta-analysis of diagnostic test accuracy. arXiv. 2021; 2106.04253:1--19
  • Chu, Haitao and Guo, Hongfei and Zhou, Yijie. Bivariate random effects meta-analysis of diagnostic studies using generalized linear mixed models. Med Decis Making. 2010; 30:499-508

  • Release History

    Date (yyyy-mm-dd) Version Details
    2023-02-01 0.9.0 Beta First release
    2023-06-30 0.9.1 Beta We updated the followings:

    - Change the color of SROC plot in Meta-analysis to distinguish the results from different models; added the legends

    - Added the Feedback button

    - Added the Reporting issue button

    - Change the names of Help tab into Wiki tab

    - Added Release history in Wiki

    - Added button for taking the survey