Summary of Diagnostic Studies

This panel is used for summarizing the diagnostic studies

1. The format of input data

  • The variable names must contain TP, FP, TN, and FN, denoting True Positive, False Positive, True Negative, and False Negative
  • The order of TP, FP, TN, FN does not matter
  • No missing values in the data
  • Re-click the "Update data and results" button, after you revised the data

2. You can get the following results:

  • The scatter plot of the studies with the corresponding confidence interval
  • The descriptive statistics of sensitivity (Sens), specificity (Spec), diagnostic odds ratio (DOR), and likelihood ratios (LRs) of each study
  • Test of equality of Sens and Spec
  • Forest plots of Sens, Spec, DOR, LRs

Example: real-word meta-analysis of diagnostic studies

The example insert is the meta-analysis for diagnosing intravascular device-related bloodstream infection.

Reference: Safdar N, Fine JP, Maki DG. Meta-analysis: methods for diagnosing intravascular device–related bloodstream infection. Ann Intern Med. 2005;142(6):451-466. doi:10.7326/0003-4819-142-6-200503150-00011


Data Preview




Note: logit transformation is $\mathrm{logit}(x)=\log(\dfrac{x}{1-x})$
  • Sens: sensitivity$=\dfrac{TP}{TP+FN}$; Spec: specificity$=\dfrac{TN}{TN+FP}$
  • y1: $\mathrm{logit}(\mathrm{Sens})$; y2: $\mathrm{logit}(\mathrm{Spec})$
  • v1: variance of y1; v2: variance of y2

Descriptive Statistics


1. Configurations of the following plot
2. ROC scatter plot of the Sens and Spec of each study


1. Summary of sensitivity (Sens) and specificity (Sepc)
Note:
  • Sens: sensitivity$=\dfrac{TP}{TP+FN}$
  • Sens.CI.lower: lower bound of CI; Sens.CI.upper: upper bound of CI
  • Spec: specificity$=\dfrac{TN}{TN+FP}$
  • Spec.CI.lower: lower bound of CI; Spec.CI.upper: upper bound of CI

2. Test of equality

3. Forest plots


1. Summary of the diagnostic odds ratio (DOR)
Note:
  • DOR: diagnostic odds ratio
  • DOR.CI.lower: lower bound of CI; DOR.CI.upper: upper bound of CI
  • lnDOR.se: standard error of log-transformed DOR

2. Forest plots


1. Summary of the positive or negative likelihood ratios (LRs)
Note:
  • PosLR: positive likelihood ratio
  • PosLR.CI.lower: lower bound of CI; PosLR.CI.upper: upper bound of CI
  • lnPosLR.se: standard error of log-transformed positive likelihood ratio
  • NegLR: negative likelihood ratio
  • NegLR.CI.lower: lower bound of CI; NegLR.CI.upper: upper bound of CI
  • lnNegLR.se: standard error of log-transformed negative likelihood ratio

2. Forest plots

Reitsma's Model and GLM model of Meta-Analysis of Diagnostic Test Accuracy

This panel is used for meta-analysis of diagnostic studies without accounting for publication bias.

Two bivariate models are available: linear mixed model (LMM, or Reitsma's model) and generalized linear mixed model (GLMMM, or GLM model)

1. About the Reitsma's model

  • This is the random effects model based on the bivariate normal distribution
  • The parameters of interest are: the summarized sensitivity and specificity, and the SAUC
  • When data are sparse (have many 0 values), this model is not recommended

Reference: Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58(10):982-990. doi:10.1016/j.jclinepi.2005.02.022

2. About GLM model

  • This model is a mixture of binomial model and normal model
  • The parameters of interest are: the summarized sensitivity and specificity, the the SAUC
  • When data are sparse (have many 0 values), this model is recommended
  • When data are not sparse, this model has similar results with Reitsma's model

Reference: 1. Chu H, Cole SR. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol. 2006;59(12):1331-1332. doi:10.1016/j.jclinepi.2006.06.011

2. You can get the following results:

  • The estimated summary ROC (SROC) curve
  • The estimates from the Reitsma's model
  • The estimates from the GLM model
See more details in the Help and Install App panel.

Output: Reitsma's model



Summary ROC (SROC) Plot



Estimates from the model



                  
Note:

Fixed-effects coefficients: the estimates in the between-study level

  • tsens.: the estimated summarized sensitivity in the logit-scale, $\mu_1$ in the Reitsma's model
  • tfpr.: the estimated summarized 1-specificity (false positive rate) in the logit-scale, $-\mu_2$ in the Reitsma's model
  • sensitivity: the estimated summarized sensitivity, logit-scaled $\mu_1$ in the Reitsma's model
  • false pos. rate:: the estimated summarized 1-specificity (false positive rate), logit-scaled $\mu_2$ in the Reitsma's model

See the details of the Reitsma's model in Help and Download panel

Estimates table


Estimates from the model



                  
Note:
see the details of the GLM model in Help and Download panel

Random effects: the estimates in the within-study level

Fixed effects: the estimates in the between-study level

  • sens.: the estimated summarized sensitivity in the logit-scale
  • spec: the estimated summarized specificity in the logit-scale

See the details of the GLM model in Help and Download panel Estimates table


Likelihood-based Sensitivity Analysis for Publication Bias

This panel is used for sensitivity analysis on the estimates of meta-analysis of diagnostic test accuracy.

Publication bias (PB) is the threat to the validity of the estimates. The method in this panel provides a method to quantify the magnitude of the potential PB.

1. The input parameter

  • Marginal selection probability ($p$): the expected proportion of published studies from the population; while $1-p$ indicated the expected proportion of the unpublished
For example, there are 33 studies in the Example. When we assign $p=0.9$, we expect
  • 90% studies (33 studies) were published from the population ($33/0.93$\approx 7$ studies)
  • 10% studies ($37 \times 0.1 \approx 4$ studies) were not published and caused the potential publication bias
When we assign $p=1$, we expect there was no unpublished studies.

2. Other parameters

Contrast $c_1, c_2$: it defines the mechanism of selective publication process in the diagnostic studies
  • $c_1=c_2$: the selective publication mechanism is influenced by both Sens and Spec; precisely, the $t$-type statistic of lnDOR influences PB
  • $c_1=1, c_2=0$: the selective publication mechanism is influenced in by Sens only; precisely, the $t$-type statistic of Sens influences PB
  • $c_1=0, c_2=1$: the selective publication mechanism is influenced in by Spec only; precisely, the $t$-type statistic of Spec influences PB
  • $\hat c_1, \hat c_2$: the selective publication mechanism is estimated from data

3. Optional parameters in the optimization

  • Initial value of $\beta$: the initial value of $\beta$ in the optimization
  • Initial value of $c_1^2$: the initial value of $c_1^2$ in the optimization
  • Range of $\beta$: lower and upper constraint of $\beta$ in the optimization
  • Range of $\alpha$: lower and upper constraint of $\alpha$ in the optimization

4. You can get the following results:

  • The bias adjusted SROC curve, given the input marginal selection probability
  • The bias adjusted SAUC, given the input marginal selection probability
  • The bias adjusted $\mu_1, \mu_2, \tau_1, \tau_2, \rho$, given the input marginal selection probability
See more details in Help and Install App panel.

Sensitivity analysis on SROC curves under four scenarios of selective publication mechanisms

Note: there may be issues of non-convergence in the results; in such case, some results are missing.

For the detailed estimates, please check the results table in the Results Table.

A. c1, c2 are estimated

B. c1=c2

C. c1=1, c2=0

D. c1=0, c2=1

E. c1, c2 are specified


Sensitivity analysis on SROC curves under four scenarios of selective publication mechanisms

Note: there may be issues of non-convergence in the results; in such case, some results are missing.

For the detailed estimates, please check the results table in the Results Table.


Note: click the button to start calculation at the first time.

There may be issues of non-convergence in the results; in such case, some results are missing.

A. c1, c2 are estimated

B. c1=c2

C. c1=1, c2=0

D. c1=0, c2=1

E. c1, c2 are specified


Sensitivity analysis on SAUC under four scenarios of selective publication mechanisms

Note: there may be issues of non-convergence in the results; in such case, some results are missing.



Summary of the estimates


A. c1, c2 are estimated


B. c1=c2


C. c1=1, c2=0


D. c1=0, c2=1


Funnel Plots for Detecting the Significance of Publication Bias

This panel presents funnel plots for detecting potential publication bias on lnDOR, logit-scaled Sens and Spec.

Funnel plot method is not suggested unless all the studies adopt the common cutoff values for diagnosis

You can get the following results:

  • The funnel plot with trim-and-fill for lnDOR, logit-scaled Sens and Sens
  • The test asymmetry of the funnel plot

Trim-and-fill Plot




Test of Asymmetry



                  



Note: these 2 plots are presented to show the whether the publication bias could be detected on sensitivity or specificity only.

Download and Install App in Your Desktop


Download App You can get an *.exe installation file. Currently, it only supports the Windows system.

Help and Reference

Note: click the "Help" button on the top

Models in this App

1. Reitsma's model

Suppose that $N~(i=1, \dots, N)$ diagnostic studies are included in meta-analysis. In other words, there are $N$ rows in the input data. The observed data are the number of true positives (TP), true negative (TN), false positives (FP), and false negatives (FN), so sensitivity (Sens) and specificity (Spec) can be estimated from data.

In the within-study level, we denote that,

  • $y_{1i}$ and $y_{2i}$: the logit-Sens and logit-Spec after continuity correction
  • $s_{1i}^2$ and $s_{2i}^2$: the observed variances of $y_{1i}$ and $y_{2i}$ within each study
  • $\mu_{1i}$ and $\mu_{2i}$: the logit-transformed true Sens and Spec of the $i$th study
Given $(\mu_{1i}, \mu_{2i})$, it is assumed that: \begin{align} \binom{y_{1i}}{y_{2i}} \sim N \left (\binom{\mu_{1i}}{\mu_{2i}}, \boldsymbol{\Sigma}_i \right ) \mathrm{~with~} \boldsymbol{\Sigma}_i = \begin{pmatrix} s_{1i}^2 & 0\\ 0 & s_{2i}^2 \end{pmatrix}, \label{eq:b2} \end{align}

In the within-study level, we denote that,

  • $\mu_1$ and $\mu_2$: the common means of the logit-transformed sensitivity and specificity
  • $\tau_1^2$ and $\tau_2^2$: between-study variances
  • $\tau_{12} = \rho\tau_{1}\tau_{2}$ is the covariance between $\mu_{1i}$ and $\mu_{2i}$, $\rho~(-1 \le \rho \le 1)$ is the correlation coefficient
It is assumed that $(\mu_{1i}, \mu_{2i})^T$ is normally distributed: \begin{align} \binom{\mu_{1i}}{\mu_{2i}} \sim N\left ( \binom{\mu_1}{\mu_2}, \boldsymbol{\Omega} \right ) \mathrm{~with~} \boldsymbol{\Omega} = \begin{pmatrix} \tau_1^2 & \tau_{12} \\ \tau_{12} & \tau_2^2 \end{pmatrix}, \label{eq:b1} \end{align}

The combination of \eqref{eq:b2} and \eqref{eq:b1} leads to the Reitsma's model:

\begin{align} \boldsymbol{y}_i | \boldsymbol{\Sigma}_i \sim N_2 \left (\boldsymbol{\mu}, \boldsymbol{\Omega} + \boldsymbol{\Sigma}_i \right ), \label{eq:b12} \end{align} where,$\boldsymbol{y}_i = (y_{1i},y_{2i})^T$, $\boldsymbol{\mu} = (\mu_1,\mu_2)^T$

Reitsma's model requires the number of diseased and non-diseased to be large.

2. Generalized linear mixed (GLM) model

GLM model differs Reitsma's model in the within-study level.

Suppose,

  • $n_{00i}, n_{11i}, n_{01i}, n_{10i}$: TP, TN, FP, FN; $N_{1i}=n_{11i}+n_{10i}, N_{0i}=n_{00i}+n_{01i}$
  • $s_{1i}^2$ and $s_{2i}^2$: the observed variances of $y_{1i}$ and $y_{2i}$ within each study
  • $\mu_{1i}$ and $\mu_{2i}$: the logit-transformed true Sens and Spec of the $i$th study

In the within-study level, it is assumed that:,

\begin{align} n_{00i}\sim \text{Binomial}(N_{0i}, \text{Spec}_i); ~~~~ n_{11i}\sim \text{Binomial}(N_{1i}, \text{Sens}_i) \label{eq:b3} \end{align}

In the between-study level, we denote that:

  • $\mu_{1i}=\text{logit}(\text{Sens}_i)$ and $\mu_{2i}=\text{logit}(\text{Spec}_i)$
  • $\mu_1$ and $\mu_2$: the common means of the logit-Sens and logit-Spec
  • $\tau_1^2$ and $\tau_2^2$: between-study variances
  • $\tau_{12} = \rho\tau_{1}\tau_{2}$ is the covariance between $\mu_{1i}$ and $\mu_{2i}$, $\rho~(-1 \le \rho \le 1)$ is the correlation coefficient

In the between-study level, $\mu_{1i}, \mu_{2i}$ have the same distribution with \ref{eq:b2}

The combination of \eqref{eq:b3} and \eqref{eq:b1} leads to the GLM model.

GLM model does not require continuity correction.

3. SROC and SAUC

The SROC curve: \begin{align} SROC(x; \boldsymbol{\mu}, \boldsymbol{\Omega}) = \mathrm{logit}^{-1} \left[ \mu_1 - \dfrac{\tau_{12}}{\tau_2^2}\{\mathrm{logit}(x)+\mu_2\} \right]. \label{eq:sroc} \end{align} The HSROC curve is given by the SROC curve \eqref{eq:sroc} with $\rho=-1$. The SAUC: \begin{align} SAUC(\boldsymbol{\mu}, \boldsymbol{\Omega}) = \int_{0}^{1}SROC(x; \boldsymbol{\mu}, \boldsymbol{\Omega})dx. \label{sauc} \end{align}

4. Likelihood based sensitivity analysis method

Publication bias is the phenomenon that studies with significant results are more likely to be published or selected for meta-analysis. In meta-analysis of DTA, we consider to model the selective publication of each study by the selection function \begin{align*} P(\text{Publish}|t_i)=\Phi(\alpha+\beta\times t_i\}, \end{align*} which is the probit model on the $t$-type statistic of the linear combination of the logit-transformed sensitivity and specificity: $ \boldsymbol{c}^T \boldsymbol{y}_i = c_1y_{1i}+c_2y_{2i}, $ where $\boldsymbol{c} = (c_1, c_2)^T$ is a contrast vector. The $t$-type statistic of the linear combination is $ t_i = {\boldsymbol{c}^T \boldsymbol{y}_i}/{\sqrt{\boldsymbol{c}^T\boldsymbol{\Sigma}_i\boldsymbol{c}}}. $
When $(c_1, c_2) = (1/\sqrt{2}, 1/\sqrt{2})$, it gives the $t$-statistic of the lnDOR. By taking different contrast vectors, the $t$-type statistic can determine a variety of selective publication mechanisms. For example, $(c_1, c_2) = (1, 0)$ and $(c_1, c_2) = (0, 1)$ indicate that the selective publication mechanisms are determined by the significance of sensitivity and specificity, respectively. The contrast vector $\boldsymbol{c}$ can be regarded as unknown parameters to be estimated or specified by user.
In the sensitivity analysis, we introduce the marginal probability of selective publication, $p=P(\mathrm{select})=E\{P(\text{Publish}|t_i)\}$. The estimating model based on the joint distribution of observed data involving the selection probabilities can be constructed. Given various specified values of $p$, the bias-adjusted SROC curves and SAUC can be estimated from the maximum likelihood estimation.

R packages used in this Application:

  • mada, DT, ggplot2, plotly, shiny, shinythemes, shinyWidgets, mvmeta, mada, shinyAce, latex2exp, base,