Omics Data Clinic Notes
The notes and questions are from Constantin Aliferis with a few responses by
FrankHarrell.
Handling the demand side of the equation
It seems to me that scheduling/prioritizing the researchers' projects will be necessary. For example, it would be unfortunate if multiple research groups wish to receive consultation on the same day and find that only one can be served. A queuing mechanism seems to be unavoidable and the clinic will need to establish fair and transparent criteria for prioritizing the queue. Such criteria may include: urgency of deadlines, budget size/complexity of the project/grant whether this is a "vanilla" problem, scientific interest on behalf of the clinic's faculty, etc.
If two
DBMI faculty and two biostat faculty can always attend the clinic, we can split into 2 rooms when needed. Regarding prioritization I would like to avoid for now getting into a formula. For one thing, most urgent deadlines come from bad planning and I don't want to let that dictate how we run it.
Executing the recommendation
Who does the analysis work? If I recommend that researcher Smith needs to run method X, how do I respond to Smith's immediate request that I implement and run X for him? What happens if Smith cannot support my staff to execute X? I propose that an explicit statement is made to all researchers seeking advice to the effect that clinic participants are not paid by Vanderbilt to execute the recommendations,
and if such services are requested the researcher must provide effort coverage and any other necessary resources.
Yes. This is a "what we can do in 1.25 - 1.5 hours we will do for you" clinic.
Preparatory work: work load and effort recovery
Often the nature of the project or assaying technology and platform may require non-trivial preparation time on behalf of the consultant. At a minimum the clinic faculty should have early access to a short description of the project, essential literature references, prior work, clear statement of hypotheses, and characteristics of data (if it exists). For cutting-edge projects the preparatory effort needed to consult may be very large. For example, a researcher wants to analyze data from 500K SNP arrays. There is no prior published work on the matter and significant thought has to be given to how to code the data, normalize it, filter variables etc., all in the context of statistical genomics which
may also require catching up with the related specialized literature. I am using this example because both biostatisticians, and myself faced this situation for the
Scuening leukemia transplant study recently and we had to work extensively just to
come up with a credible data analysis plan without many of the specifics that
will eventually be needed for the actual analysis. How will the consultant avoid
sounding "too fuzzy" and at the same time be compensated for her time in case
substantial work is needed just to come up with an analysis plan?
This is an excellent point and right now I have limited insight about it. Perhaps at the start we will emphasize the simpler projects, and for the complex ones we will only sketch an approach or at least an approach to finding the best approach.
Evaluation of service
How do we know if we succeeded? Usage and customer satisfaction are important criteria, but only indirectly related to the real outcomes, that is quality and volume of good science produced for dollar spent compared to when this service is not being offered. We need to form an evaluation criteria set, design, and timeline.
Yes. We should all work on this. I will also keep statistics on the number of attendees from each division in the SOM. I will collect e-mail addresses and later we can put up a web survey asking all past attendees to take it.
Academic side benefits and opportunities:
Can we get a grant to support this effort? It seems sufficiently innovative. We could also augment the service with comparative evaluations of methods (and get grant support for it). We could convert experience to protocols (paper or computerized). Myself and colleagues in DSL have significant experience in the area of comparative evaluations and automating data analysis protocols, which may be useful here. I have been independently planning to apply for support for data analysis tools and comparative evaluations some time now, perhaps the clinic can be a part of these application or independent ones can be pursued.
Scalability and re-use
Obviously, it would be great if in the process we train staff to learn how to handle routine situations and recognize non-routine problems, so that staff eventually can triage requests to the clinic faculty or staff consultants as appropriate. Toward that end I believe we need to establish staff (training) participation and keep a record of the problems presented and advice given. This knowledge base can be used to solve future requests more efficiently, and even to guide student training and faculty/staff recruitment.
Liability
We need to clarify that the advice provided does not make the consultants liable for damages.
Dear Colleagues,
A few quick thoughts about todays -omics clinic.
1. It would be great if the participating researchers prepare a brief statement of the
biological hypotheses or scientific purpose their study aims for. I believe we consumed a good half of the discussion today eliciting the actual hypotheses that were largely hidden within (rather misleading) procedural language. For example, the researcher said she wanted to study variability of certain markers in certain populations rather than that she wanted to see which markers differ between subpopulations and may play a biological role in what subpopulation a patient belongs to (e.g., early vs late HIV progressors).
2. There were too many consultants to have enough time each to participate fully. I concur entirely with Dr Shyr's proposal that in the first few minutes we should familiarize ourselves with the projects of the day and split up the consultants in two groups according to skills and interest.
3. There were some tricky issues (such as the longitudinal nature of data and research design as well as that power-size analysis has to be tied to that design) that we did not address. Will we come back to this next time? If not, does the researcher appreciate this difficulty?
4. I believe Deans explanation to the researcher at the end about follow-up limitations is a very important one. I am sure that over time the value of close collaboration with the consulting faculty/departments will become evident.
5. I suspect that the discussion was generally useful even for the researchers who did not get specific advice in a learn by osmosis fashion. They seemed to be interested and stayed until the end.
6. Mini-tutorials proposed by Dr Masys is a great idea for the benefit of both consulting faculty and researchers. We had a similar interest group that I organized between 2000 and 2001. It did very well, we discussed many topics and learned a lot, but we run out of major topics and since I did not have time to organize it back then it came to a close.
For this round, I volunteer as a speaker for the following:
- basic concepts of biomarker selection and overview of biomarker selection algorithm families
- methodological differences and similarities between machine learning and statistics
- a review of data analysis problems associated with very large dimensionality data
I would very much like to learn more (if colleagues are willing to explain) about the specific assays mentioned in the discussion (e.g., RT PCR, PCR array, flow cytometry, normalizing those, capabilities and pros and cons of these methods etc.).
Constantin
Nice thoughts - be sure to update the twiki page you and I started with anything you want there. I like the idea of having tutorials to us on PCR etc. And having 15-20 minute presentations ready to present, such as the ones you listed, is a terrific idea too.
And remember we will not have so many consultants there for too long.
Frank