You are here: Vanderbilt Biostatistics Wiki>Main Web>Clinics>ClinicOmics>ComplexDataJournalClub (revision 8)~~Edit~~~~Attach~~

- Sebastiani et al, Nature Genetics 37:435;2005: Genetic dissection and prognostic modeling of overt stroke in sickle cell anemia.
- Lewin et al (2006), Biometrics 62(2):1-9: Bayesian modeling of differential gene expression; Gottardo et al (2006), Biometrics 62(2):10-18: Bayesian robust inference for differential gene expression in microarray with multiple samples.

- Full Text | Commentary | Supplemental Information
- 18 Apr 2006, noon-1pm, Room D-2221 Med Ctr North, RSVP to biostat@vanderbilt.edu
- Discussed by FrankHarrell and ConstantinAliferis

- Find persons with sickle cell anemia at risk of overt stroke (6-8% incidence)
- Genetic dissection: disentangling interactions among genes, environment, phenotype
- 108 SNPs in 80 (?39) candidate genes
- Unspecified extensive clinical variables also used
- 1398 African Americans with SCA, 92 strokes

- Focused on networks describing dependencies of genotypes on phenotype
- Authors state that strategy is more diagnostic than prognostic
- Did not take variable follow-up into account
- Used Bayesware Discoverer which claims to do "automated discovery of Bayesian networks from data"
- $5,500 for one user license; first 2 authors are owners; not open source
- Found that 31 SNPs in 12 genes interact with fetal hemoglobin to predict stroke risk
- Final network had 73 nodes (fig. 2, 69 SNPs, avg. 1.3 strokes per node)
- "the model is able to describe the determinant effects of genetic variants on stroke"
- Stability benefits from smart choices of candidate genes
- Bayes factor of quantifying the relative likelihood that a SNP is associated with stroke vs. being independent may be problematic
- BN is not dissimilar to tree modeling using recursive partitioning
- Example simulations showing instability
- Simulated data from additive logistic model with sample size of 1490 patients having a target of 92 strokes
- 108 SNPs (binary, prevalence 0.5) having importance (weights) shown below
- 30 clincial variables (normal with mean 0 and standard deviation .5), weights uniform between -1 and 1
- 3 independent repetitions of recursive partitional
- Most important SNPs: s18 and s27; most important clinical variables: c13 and c8

- Multiply sample size by 10 and repeat

library(Hmisc); library(rpart); library(lattice) strokes <- 92 n <- 92 + 1398 snips <- 108 clinical <- 30 ## Generate set of true regression coefficients set.seed(1) snipImportance <- c(runif(snips/3, -2.5, 2.5), runif(2*snips/3, -1, 1)) png('/tmp/snipImportance.png') hist(snipImportance, nclass=20) dev.off() clinImportance <- runif(clinical, -1, 1) which(snipImportance==max(snipImportance)) # 18 which(snipImportance==min(snipImportance)) # 27 which(clinImportance==max(clinImportance)) # 13 which(clinImportance==min(clinImportance)) # 8 ## Generate three sets of data under the fixed model, for original ## sample size and n*10 set.seed(2) for(m in c(1,10)) { for(i in 1:3) { s <- matrix(sample(0:1, m*n*snips, replace=TRUE), ncol=snips) c <- matrix(.5*rnorm(m*n*clinical), ncol=clinical) ## Generate stroke (y=1) and non-stroke (0) under additive logistic model L <- .5*matxv(s, snipImportance) + .5*matxv(c, clinImportance) - 6 y <- rbinom(m*n, 1, plogis(L)) prn(c(mean(y), strokes/n)) f <- rpart(y ~ c + s, control=rpart.control(minsplit=m*40,cp=.001)) prn(f$cptable) options(digits=3) png(paste('/tmp/seb',m,'-',i,'.png',sep=''), pointsize=8) plot(f); text(f) dev.off() } }

- Proportional classified correctly is an improper scoring rule (optimized by a miscalibrated model)
- Arbitrary and covers up "gray zone" (authors used cutoff of 0.5 risk)
- Loss of statistical power
- Authors did independent validation on 114 patients (7 with stroke), got 98.2% correct (112/114, predicted all strokes correctly, predicted 2 strokes in patients without stroke)
- Fig. 3 treats stroke as the predictor and the probability of stroke as the response variable
- Need a calibration plot

- Authors gave no confidence limits on their validated proportion correct [0.938, 0.995] but will the classification rule transport?

binconf(112,114)

- Need a more continuous measure of discrimination ability that does not require arbitrary classification
- Area under ROC curve (C-index), Brier score

library(Hmisc) P <- .8 # true probability of a correct prediction N <- round(10*(2^seq(0,13,by=.25))) p <- low <- hi <- N set.seed(3) y <- rbinom(max(N), 1, P) i <- 0 for(n in N) { i <- i+1 s <- sum(y[1:n]) p[i] <- s/n cat(n,'') lim <- binconf(s, n, method='wilson') low[i] <- lim[,'Lower'] hi[i] <- lim[,'Upper'] } #pdf('/tmp/validation.pdf') png('/tmp/propCorrect.png') plot(log2(N), p, ylim=range(c(low,hi)), axes=FALSE, type='b', xlab='Number of Patients in Validation Sample', ylab='Estimated Accuracy of Diagnostic Patterns', main='Estimated Accuracy and Its Margin of Error\nWhen True Classification Accuracy is 0.8') axis(2) w <- 10*(2^seq(0,13,by=1)) axis(1, log2(w), w) lines(log2(N), low, col=gray(.7)) lines(log2(N), hi, col=gray(.7)) abline(h=P, lty=2, col=gray(.7)) dev.off()

- Meschia and Pankratz (p. 340-1): "Despite their promise, it is important to temper enthusiasm for Bayesian networks and similar analytical techniques. Results from these analytical techniques are dependent on the structure of the model, and their statistical properties have not yet been fully studied. The complex data analaytical techniques that can be applied to large data sets make it possible to obtain persuasive results for any given data set. If nuances in sample set identification, data collection and application of the method are overlooked, then the results that are obtained may be difficult to replicate, or even misleading. ... we recommend that extensive detail be provided about every component of the study design and analysis, including a detailed description of the intermediate steps in model creation."

- Full Text for Lewin et al | Supplemental Information | Full Text for Gottado et al
- 2 May 2006, noon-1pm, Room D-2221 MCN, RSVP to biostat@vanderbilt.edu
- Discussed by Chuan Zhou and Frank Harrell

- Detecting differentially expressed genes
- Simultaneously model expression-dependent array effect
- Simple decision rule taking in account uncertainty in parameter estimates

I | Attachment | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|

ppt | SebastianiAliferis.ppt | manage | 140.0 K | 19 Apr 2006 - 14:19 | FrankHarrell | Aliferis Commentary on Sebastiani et al |

png | propCorrect.png | manage | 5.9 K | 14 Apr 2006 - 16:20 | FrankHarrell | Accuracy of Classification Accuracy as a Function of Sample Size |

png | seb1-1.png | manage | 4.4 K | 17 Apr 2006 - 17:11 | FrankHarrell | |

png | seb1-2.png | manage | 4.6 K | 17 Apr 2006 - 17:11 | FrankHarrell | |

png | seb1-3.png | manage | 4.5 K | 17 Apr 2006 - 17:12 | FrankHarrell | |

png | seb10-1.png | manage | 4.3 K | 17 Apr 2006 - 17:12 | FrankHarrell | |

png | seb10-2.png | manage | 4.3 K | 17 Apr 2006 - 17:12 | FrankHarrell | |

png | seb10-3.png | manage | 4.6 K | 17 Apr 2006 - 17:13 | FrankHarrell | |

png | snipImportance.png | manage | 3.1 K | 17 Apr 2006 - 17:11 | FrankHarrell |

Topic revision: r8 - 26 Apr 2006, ChuanZhou

Copyright © 2013-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.

Ideas, requests, problems regarding Vanderbilt Biostatistics Wiki? Send feedback

Ideas, requests, problems regarding Vanderbilt Biostatistics Wiki? Send feedback