Original File Name: Franken1 trap area jackknife anova.txt

 06/25/2020

                           OR-DBM MRMC 2.51

                     <beta> Build  20181028 </beta>

              MULTIREADER-MULTICASE ROC ANALYSIS OF VARIANCE
                          TRAPEZOIDAL AREA ANALYSIS
 
 
 |====================================================================|
 |*****                         Credits                          *****|
 |====================================================================|
 |                                                                    |
 | ANOVA Computations & Display:                                      |
 | -----------------------------                                      |
 | Kevin M. Schartz, Stephen L. Hillis, Lorenzo L. Pesce, &           |
 | Kevin S. Berbaum                                                   |
 |                                                                    |
 | Expected Utility Computations:                                     |
 | -----------------------------                                      |
 | Craig K. Abbey                                                     |
 |                                                                    |
 |====================================================================|
 
 
 
 |====================================================================|
 |***************************     NOTE     ***************************|
 |====================================================================|
 | The user agreement for this software stipulates that any           |
 | publications based on research data analyzed using this software   |
 | must cite references 1-5 given below.                              |
 |                                                                    |
 | Example of citing the software:                                    |
 |                                                                    |
 |      Reader performance analysis was performed using the software  |
 | package OR-DBM MRMC 2.51, written by Kevin M.Schartz, Stephen L.   |
 | Hillis, Lorenzo L. Pesce, and Kevin S. Berbaum, and freely         |
 | available at http://perception.radiology.uiowa.edu. This program   |
 | is based on the methods initially proposed by Dorfman, Berbaum,    |
 | and Metz [1] and Obuchowski and Rockette [2] and later unified and |
 | improved by Hillis and colleagues [3-5].                           |
 |====================================================================|
 
 Data file: \\vmware-host\Shared Folders\VmWareShared\Franken\Franken1.lrc                                                                                                                                                                                                  
 
 2 treatments, 4 readers, 100 cases (33 normal, 67 abnormal)
 
 Curve fitting methodology is TRAPEZOIDAL/WILCOXON
 Dependent variable is AUC
 
 Study Design:  Factorial
 Covariance Estimation Method:  Jackknifing
 
 ===========================================================================
 *****                            Estimates                            *****
 ===========================================================================
 
TREATMENT x READER AUC ESTIMATES

                  TREATMENT
           -----------------------
 READER         1            2
 ------    ----------   ----------   
     1     0.85345997   0.84961556
     2     0.86499322   0.84350972
     3     0.85730439   0.84011759
     4     0.81524197   0.81433740
 

 TREATMENT AUC MEANS (averaged across readers)
 ---------------------------------------------
       1      0.84774989
       2      0.83689507
 
 

 TREATMENT AUC MEAN DIFFERENCES
 ------------------------------
     1 - 2     0.01085482
 
 
 
 ===========================================================================
 *****            ANOVA Tables (OR analysis of reader AUCs)            *****
 ===========================================================================
 
 TREATMENT X READER ANOVA of AUCs
 (Used for global test of equal treatment AUCs and for treatment differences
  confidence intervals in parts (a) and (b) of the analyses)
 
Source            SS               DF             MS        
------   --------------------    ------   ------------------
     T             0.00023565         1           0.00023565
     R             0.00205218         3           0.00068406
   T*R             0.00015061         3           0.00005020
 
 
 
 READER ANOVAs of AUCs for each treatment
 (Used for single treatment confidence intervals in part (c) of the analyses)
 

                        Mean Squares
 Source     df   Treatment 1   Treatment 2
 ------    ---   -----------   -----------
      R      3    0.00049266    0.00024160
 
 
 ===========================================================================
 *****        Variance component and error-covariance estimates        *****
 ===========================================================================
 
 Obuchowski-Rockette variance component and covariance estimates
 (for sample size estimation for future studies)
 Note: These are ANOVA estimates which can be negative
 
     OR Component             Estimate         Correlation  
 -----------------------  ----------------  ----------------
 Var(R)                         0.00003776
 Var(T*R)                      -0.00071276
 COV1                           0.00079168        0.51887172
 COV2                           0.00048364        0.31697811
 COV3                           0.00051251        0.33590059
 Var(Error)                     0.00152578
 
 
 Corresponding DBM variance component and covariance estimates
 
     DBM Component            Estimate    
 -----------------------  ----------------
 Var(R)                         0.00003776
 Var(C)                         0.05125091
 Var(T*R)                      -0.00071276
 Var(T*C)                      -0.00288715
 Var(R*C)                       0.02791730
 Var(T*R*C) + Var(Error)        0.07629656
 
 
 ===========================================================================
 *****    Analysis 1 (OR Analysis): Random Readers and Random Cases    *****
 ===========================================================================
 (Results apply to the population of readers and cases)


    a) Test for H0: Treatments have the same AUC
 
 Source        DF      Mean Square    F value  Pr > F 
 ----------  ------  ---------------  -------  -------
 Treatment        1       0.00023565     4.69   0.1188
 Error term    3.00       0.00005020
 Error term = MS(T*R) + r*max[Cov2 - Cov3,0]
 
 Conclusion: The treatment AUCs are not significantly different [F(1,3) = 4.69, p = .1188].
 
 Df(error term) = [MS(T*R) + r*max(Cov2 - Cov3,0)]**2/{MS(T*R)**2/[(t-1)(r-1)]}
 Note: "Error term" is the denominator of the F statistic and is a linear
 combination of mean squares, as defined above.  The value of this linear 
 combination is given under the "Mean Square" column
 Note: Df(error term) is called "ddf_H" in Hillis (2007).
 

    b) 95% confidence intervals and hypothesis tests (H0: difference = 0)
       for treatment AUC differences
 
 Treatment
 Comparison  Difference   StdErr      DF      t     Pr >|t|          95% CI       
 ----------  ----------  --------  -------  ------  -------  ---------------------
   1 - 2        0.01085   0.00501     3.00    2.17   0.1188  (-0.00509 ,  0.02680)
 
 StdErr = sqrt{(2/r)*[MS(T*R) + r*max(Cov2 - Cov3,0)]}
 Df same as df(error term) from (a)
 95% CI: Difference +- t(.025;df) * StdErr
 

    c) Single-treatment 95% confidence intervals
       (Each analysis is based only on data for the specified treatment, i.e., 
       on the treatment-specific reader ANOVA of AUCs and Cov2 estimates.)
 
  Treatment      AUC      Std Err       DF     95% Confidence Interval      Cov2   
 ----------  ----------  ----------  -------  -------------------------  ----------
          1  0.84774989  0.02440215    70.12  (0.79908282 , 0.89641696)  0.00047230
          2  0.83689507  0.02356642   253.64  (0.79048429 , 0.88330585)  0.00049498
 
 StdErr = sqrt{1/r * [MS(R) + r*max(Cov2,0)]}
 Df = [MS(R)+ max(r*cov2,0)]**2/[(MS(R)**2/(r-1)]
 Note: Df is called "ddf_H_single" in Hillis (2007)
 95% CI: AUC +- t(.025;df) * StdErr
 
 
 ===========================================================================
 *****    Analysis 2 (OR Analysis): Fixed Readers and Random Cases     *****
 ===========================================================================
 (Results apply to the population of cases but only for the readers used in
 this study. Chi-square or Z tests are used; these are appropriate for 
 moderate or large case sample sizes.)
 
    a) Chi-square test for H0: Treatments have the same AUC
    Note: The chi-square statistic is denoted by X2 or by X2(df), where df is its 
    corresponding degrees of freedom.
 
 
     X2 value       DF    Pr > X2
 ---------------  ------  -------
         0.32101       1   0.5710
 
 Conclusion: The treatment AUCs are not significantly different [X2(1) = .32, p = 0.5710].
 
 X2 = (t-1)*MS(T)/[Var(error) - Cov1 + (r-1)*max(Cov2 - Cov3,0)]


    b) 95% confidence intervals and hypothesis tests (H0: difference = 0)
       for treatment AUC differences
 
 Treatment
 Comparison  Difference   StdErr     z     Pr >|z|          95% CI       
 ----------  ----------  --------  ------  -------  ---------------------
   1 - 2        0.01085   0.01916    0.57   0.5710  (-0.02670 ,  0.04840)
 
 StdErr = sqrt{2/r * [(Var(error) - Cov1 + (r-1)*max(Cov2 - Cov3,0)]}
 95% CI: difference +- z(.025) * StdErr
 

    c) Single treatment AUC 95% confidence intervals
       (Each analysis is based only on data for the specified treatment, i.e., on
        the specific reader ANOVA of AUCs and error-variance and Cov2 estimates.)
 
  Treatment      AUC      Std Error   95% Confidence Interval 
 ----------  ----------  ----------  -------------------------
          1  0.84774989  0.02710939  (0.79461647 , 0.90088331)
          2  0.83689507  0.02744860  (0.78309680 , 0.89069334)
 
  Treatment  Var(Error)     Cov2   
 ----------  ----------  ----------
          1  0.00152278  0.00047230
          2  0.00152877  0.00049498
 
 StdErr = sqrt{1/r * [Var(error) + (r-1)*max(Cov2,0)]}
 95% CI: AUC +- z(.025) * StdErr


    d) Single-reader 95% confidence intervals and tests (H0: difference = 0) for 
    treatment AUC differences.
       (Each analysis is based only on data for the specified reader, i.e, on the 
        reader-specific AUC, error-variance and Cov1 estimates.)
 
         Treatment
 Reader  Comparison  Difference  StdErr      z     Pr >|z|          95% CI       
 ------  ----------  ----------  --------  ------  -------  ---------------------
      1    1 - 2        0.00384   0.04279    0.09   0.9284  (-0.08003 ,  0.08772)
      2    1 - 2        0.02148   0.04007    0.54   0.5919  (-0.05705 ,  0.10002)
      3    1 - 2        0.01719   0.03499    0.49   0.6233  (-0.05140 ,  0.08577)
      4    1 - 2        0.00090   0.03481    0.03   0.9793  (-0.06731 ,  0.06912)
 
 Reader  Var(Error)     Cov1   
 ------  ----------  ----------
      1  0.00147717  0.00056158
      2  0.00151861  0.00071581
      3  0.00137738  0.00076509
      4  0.00172995  0.00112425
 
 StdErr = sqrt[2*(Var(error) - Cov1)]
 95% CI: Difference +- z(.025) * StdErr
 
 
 ===========================================================================
 *****    Analysis 3 (OR Analysis): Random Readers and Fixed Cases     *****
 ===========================================================================
 (Results apply to the population of readers but only for the cases used in
 this study)

     These results result from using the OR model, but treating reader as a random 
 factor and treatment and case as fixed factors.  Because case is treated as a fixed
 factor, it follows that Cov1 = Cov2 = Cov3 = 0; i.e., there is no correlation
 between reader-performance measures (e.g, AUCs) due to reading the same
 cases.  Thus the OR model reduces to a conventional treatment x reader ANOVA
 for the reader-performance outcomes, where reader is a random factor and
 treatment is a fixed factor.  This is the same as a repeated measures ANOVA
 where treatment is the repeated measures factor, i.e., readers provide an
 outcome (e.g., AUC) under each treatment.
     Note that the DBM and OR papers do not discuss this approach, but rather 
 it is included here for completeness.

    a) Test for H0: Treatments have the same AUC
 
 Source        DF    Mean Square      F value  Pr > F 
 ----------  ------  ---------------  -------  -------
 Treatment        1       0.00023565     4.69   0.1188
 T*R              3       0.00005020
 
 Conclusion: The treatment AUCs are not significantly different [F(1,3) = 4.69, p = .1188].
 Note: If there are only 2 treatments, this is equivalent to a paired t-test applied
 to the AUCs


    b) 95% confidence intervals and hypothesis tests (H0: difference = 0)
       for treatment AUC differences
 
 Treatment
 Comparison  Difference   StdErr      DF      t     Pr >|t|          95% CI       
 ----------  ----------  --------  -------  ------  -------  ---------------------
   1 - 2        0.01085   0.00501        3    2.17   0.1188  (-0.00509 ,  0.02680)
 
 StdErr = sqrt[2/r * MS(T*R)]
 DF = df[MS(T*R)] = (t-1)(r-1)
 95% CI: Difference +- t(.025;df) * StdErr
 Note: If there are only 2 treatments, this is equivalent to a paired t-test applied
 to the AUCs
 

    c) Single treatment AUC 95% confidence intervals
       (Each analysis is based only on data for the specified treatment, 
       i.e. on the treatment-specfic reader ANOVA of AUCs
 
  Treatment      AUC        MS(R)     Std Error     DF     95% Confidence Interval 
 ----------  ----------  ----------  ----------  -------  -------------------------
          1  0.84774989  0.00049266  0.01109801        3  (0.81243106 , 0.88306871)
          2  0.83689507  0.00024160  0.00777173        3  (0.81216196 , 0.86162818)
 
 StdErr = sqrt[1/r * MS(R)]
 DF = df[MS(R)] = r-1
 95% CI: AUC +- t(.025;df) * StdErr
 Note: this is the conventional CI, treating the reader AUCs as a random sample.
 
 
 
 #=> Reference resources are missing. Default references provided. <=#
 


                               REFERENCES

 1.   Dorfman, D.D., Berbaum, K.S., & Metz, C.E. (1992). Receiver operating
 characteristic rating analysis: Generalization to the population of 
 readers and patients with the jackknife method. Investigative Radiology,
 27, 723-731.

 2.    Obuchowski, N.A., & Rockette, H.E. (1995). Hypothesis testing of diagnostic
 accuracy for multiple readers and multiple tests: An ANOVA approach with dependent
 observations. Communications in Statistics-Simulation and Computation, 24, 285-308.

 3.   Hillis, S.L., Obuchowski, N.A., Schartz, K.M., & Berbaum, K.S.
 (2005). A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette
 methods for receiver operating characteristic (ROC) data. 
 Statistics in Medicine, 24, 1579-1607  DOI:10.1002/sim.2024.

 4.   Hillis, S.L. (2007). A comparison of denominator degrees of freedom for
 multiple observer ROC analysis.  Statistics in Medicine, 26:596-619  DOI:10.1002/sim.2532.

 6.   Hillis, S.L., Berbaum, K.S., & Metz, C.E. (2008). Recent developments in the
 Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Academic Radiology, 15, 
 647-661. DOI:10.1016/j.acra.2007.12.015
