Demonstration and Review of PercipEnz Oncore System

Interspersed are some responses (marked DM) from David DeMets, and Sorena Nadaf (marked SN), key personnel involved with Oncore. There are some rejoinders after some of these comments from Frank Harrell (marked FH).

Goals of System in Cancer Center

• Replace multiple systems into a single integrated system
• Roll-out in several months.
• Handle multi-center trials within Tennessee network and nationwide ECOG network. Collect much of the data at point of care.
• DM: Oncore technology is a work in progress and is designed to manage a portfolio of clinical trials throughout their life cycles and provide increasing levels of operational integration for an entire clinical trails enterprise, eventually spanning multiple centers. A key point is that this level of integration entails much more than technology. It requires common standards and practice and a coherent and effective strategy for both organizational and software engineering. This is the reason for the existence of the Oncore consortium (see http://www.oncore.org) which now has seven members. The technology is only part of the story, although it is a critical enabler.
• SN: Also a note worth mentioning is that one of the Consortiums key objectives, is to promote sharing and interactions between data gathered in the basic sciences, and its natural path towards clinical and epidemiological information. This is one of the long term "keys", to our success.

Basis Software

• Java (server side)
• Oracle
• Apache web server

Pros

• HIPAA compliant; uses user roles for access control (PI, CRA, data monitor, statistician, RN, etc.). Can also setup a user to have as many roles simultaneously as needed. In future want to authenticate to VU LDAP database.
• DM: New roles can be defined by each center. Security and access control that works for a distributed clinical trails enterprise and supports the implementation of appropriate HIPPA policies is one of the stregths of Oncore technology. Access control is managed at the application level, not the database level. This is necessary to take advantage of domain semantics. The access mechanisms are more sophisticated than those usually associated with "role based" approaches (e.g. they are dynamic and context dependent).
• Monitoring functions, e.g. accrual, open protocols by IRB manager, PI; status of contract, protocol development and tracking of deviations, IRB review (including a breakdown of IRBs across sites), site contact info. Adaptable - can add new committee review stage designation. Manages DMC audits. Automatic safety alerts to PI or RN when SAEs (serious adverse events) occur. SAEs by drug, number of SAEs per patient. Report of changes over time in lab parameters. Subject calendar. Can add own reports.
• Easy to navigate multiple CRFs (case report forms), completed forms, todo forms.
• Library of forms can pull from, can update forms and save in repository.
• Can enter first few letters of categories and get quick lookups.
• Different consoles for different types of users.
• A grant has been submitted to expand Oncore to non-cancer trials.
• Dave DeMets (Chair, Dept. of Biostatistics, U. Wisconsin) is intimately involved; everything Dave does is first-rate
• DM: The strongest advantage of Oncore technology is that it works. This is the result of, collectively, decades of experience with various approaches and technologies. This includes open source software, locally developed systems, the use of conslultants, etc.. The Oncore model is the result of this experience and the hard won lessons learned. The most telling advantage of this technology is that it is the only system being used by multiple cancer centers for all of their clinical trials management needs. As far as we know this is a historic first and we are the only group that can make this claim. In some centers versions have been in use for about three years.

Cons (and Discussion)

• Possible scaling issues due to Java (it is difficult to understand why Oncore chose to base their system on Oracle for scalability then negated this advantage by using Java)
• DM: Oracle wasn't just chosen for scalability and this comment is unclear about the sense in which Java does not scale. Java has a long history and many technologies support it in various ways. The system architecture for Oncore is based on the J2EE platform that was developed to be highly scalable and has been used successfully in numerous high volume transactional, web-based applications. A web-search on "J2EE platform" will yield many examples. These include, if memory serves, eBay, stock exchanges, Amazon, e-Trade, etc. Because Oncore technology is designed to support a distributed clinical trials enterprise, removing the need for local support at a site by using a "thin client", high performance and scalability are all design goals and we believe that these have been optimally achieved within the constraints of available technology.
• SN: The VICC (Cancer Center) , thus far, has not encountered any scalability issues due to Java, in any environment. In my interactions with the other sites using ONCORE, scalability issues has never been posted.
• FH: Speed problems with Java have already been observed at some of the Oncore sites. This seems to have been fixed now but needs to be watched.
• SN: A reminder that the demo performed here, was in a test environment, and not a production one. Regardless, the speed problems were due to some Data Migration "breaks", we recently experienced in our latest upload. These data breaks have been assessed and fixed. As you know, we are well underway in our migration of over 10 years of data into the ONCORE environment. During the past 7 months of stringent testing, and "scripting", we have been very happy and actually are very pleased with the speed of the system when fully loaded, and accessed via multiple users onsite and off-site. We will maintain our test server environment well after the final migration onto our production systems for testing and future upgrade implementations. There has been no reports of any speed problems from any of the other sites using ONCORE.
• Need to verify that can be generalized to non-cancer clinical trials.
• DM: Neither the Consortium nor PercipEnz make any claim at this point in time that Oncore technology can or even should be used for non-cancer clinical trials. As noted above, there are efforts underway to create SCORE, a derivative of Oncore that can be used for other disease areas. Oncore technology works as well as it does because it contains, in effect, embedded domain knowledge which specializes it to the domain of cancer clinical trails (see however, comment on meta-data below). In addition it is specialized to integrate with each cancer center's local institutional context.
• Cost: a significant investment per year (VU has 3 yr contract); does not include Oracle
• SN: In comparison to a dozen other commercial systems I evaluated, our investments were very reasonable. We spent many times more money in the development and "fixing" of much less capable databases, with groups on campus. True, ONCORE does not include Oracle software, which is a requirement of the system, but that is not the cost. All Cancer Center members of our Consortium, all whom are all users of this product, are affiliated in some way or another, with major institutions. All providing, at no cost, full licenses of Oracle to there Cancer Centers. As Vanderbilt has provided for us. But the point I would like to make here is, even if Vanderbilt did not provide this great resource to us, and the Cancer Center did not own its own license, the cost of obtaining a license would have been negligible, and still saved us buckets of money.
• DM: This is not the cost! The cost to a cancer center depends on the degree to which they collaborate in the development process and is used as an incentive. For example, VICC was a major contributor to developing requirements for the EDC component of Oncore technology. In general the price has never been more than the equivalent of two FTEs, which in our experience is a fraction of the historical staffing levels required to maintain previous, less capable technologies for managing a cancer center clinical trial enterprise.
• Choice of Oracle as a database engine is highly problematic and difficult to understand; if we work with other groups wanting to host their own database but who do not already have an Oracle license, that group will end up paying more for the database engine than for the Oncore system that really does the work. MySQL or PostgreSQL would have been wonderful (and free) choices for a database engine.
• DM: Without understanding how you wish to work with other groups "wanting to host their own database" that part is hard to respond to. The choice of Oracle as the DBMS for Oncore was based on a number of different factors including scalability, reliability, availability, security, maintainability, support, and the fact that it is widely used among cancer centers and their parent academic institutions. The vast majority of universities and academic medical centers have site licenses for Oracle in which case they would be, in fact, free to any university group. In those cases where the group in question does not have access to a site license, an Oracle license can be purchased for a few thousand dollars (note this is not true for Oracle Clinical at this point). This pricing is a change from a few years ago and represents a trend. In the future a run-time version of Oracle may be bundled and included in the cost. In addition, NCI is moving towards Oracle as the data base language for all of their cooperative group studies as well as other trial activity. Some consideration was given to providing versions of Oncore that could use other database engines as a backend, but given the low cost of Oracle and the many advantages there is no practical reason to do this and it would only increase over all costs. Our Biomedical Computing Group (BCG) supports and uses both MySQL and PostgresSQL and neither are free in terms of Total Cost of Ownership (TOC). We would choose neither over Oracle for a clinical trails application, especially when doing business with industry sponsors. Because the UW has a site license Oracle is free to us and the TOC is probably lower than both MySQL and PostgresSQL but it is hard to compare because (except in trivial cases) we would probably never use all three for the same kind of applications. For a technical comparison of these three systems see: http://det-dbalice.if.pw.edu.pl/det-dbalice/ttraczyk/db_compare/db_compare.html. FH: See also http://www.devx.com/dbzone/Article/20743.
• FH: I'm not very convinced by any of these arguments and doubt that the TOC is less with Oracle. Many commercial firms seem to be switching to open source database engines. Oracle has a long history of getting users hooked and then raising renewal fees in an agressive manor. At the University of Virginia, one side of the campus had an Oracle license but users on the other side of the campus did not have access to that license and would be forced to pay significant licensing fees.
• SN: I have not found a good example of any Academic Institution moving away from Oracle to any open source modules, and am not the greatest fan of PostgreSQL, and especially MySQL. Both have significant issues, but not to say that Oracle does not have some either. I agree that Oracle has had a history of increasing there fees, but this is becoming more and more common practice during the past few years for almost every major software company. You can blame Microsoft for this. I wish there was a "perfect" environment, but one does not exists. I will continue to lean towards Oracle for the time being. Having said that, I am still very much in favor of open source as a standard, but I think we are a few years from this becoming reality. We are on the correct path.
• FH: See http://www.onlamp.com/pub/wlg/4715 although that article does not address academics per se. The article points out that a likely reason that Oracle's licensing fees have become less predatory is competition from MySql.
• Cannot use the system for studies in which a pharmaceutical company provides the CRF.
• DM: We are not sure what systems would allow you to do this? Even a fax-based system requires special forms to provide any automation and usually requires some amount of manual data entry. In any case this is not entirely true for Oncore technology. It is true that if a sponsor provides paper CRFs or a separate system for electronic CRFs, it does not make sense to duplicate effort by entering the same data into Oncore. There are, however, many features of Oncore that are useful as well as some that are required for regulatory reasons. More specifically, Oncore is much more than just an electronic data capture (EDC) system. Obviously, the EDC capabilities are rendered useless when the sponsor dictates the use of paper CRFs. In the case of a sponsor with an EDC system it would be technically possible to arrange to export data from Oncore. An example from the cancer world is Weststat. There has been some consideration given to generating paper forms that mimic a sponsors forms but this is not a capability that is of high value to any members of the Consortium at the moment.
• FH: I did not state this very clearly. What I meant to say was that when a pharmaceutical company needs a CRF computerized and the data modules have never been agreed upon by a consortium (because the system is being used with the pharma company in a way that may not involve the consortium), it is very difficult to define these never-seen data elements.
• SN: The rate limiting steps in almost all interactions with Pharma are there guidelines for such interactions. I do not know of any successful standard that exists. I agree that these type of data elements are extremely difficult to define.
• Cannot handle hard and soft range checks on same variable.
• DM: Actually, it did support this capability but it has been modified as the result of extensive field-testing by users in three different cancer centers. The consensus was that the real value of such dual range checks was for lab tests and that a soft range check should be based on institutional norms while another soft range check should be defined for a CRF and modifiable in the context of a given trial. Hard range checks were considered unnecessary and cumbersome.
• FH: This is a major problem and goes against standard practice. Hard range checks (e.g., white blood count greater than zero, diastolic blood pressure less than 500, number of burned fingers on one hand less than 5) are absolutely necessary for some variables.
• SN: I would not consider this a considerable problem. True, a common practice, but not necessarily a standard. I believe over the past decade, with the number of trials exploding and evolving, modifications to such requirements have somewhat changed. I am interested in this quite a bit. Are there any references that can be posted here? Thanks.
• Cannot catch future dates during data entry.
• DM: Future dates are trapped where appropriate. For example, the date of completion for a CRF cannot be a future date. The user community has been unanimous in requesting that future dates be allowed except in some critical areas.
• May not have client-side checking - range checks may not be executed until Submit is pressed.
• DM: It is true that client-side range and data type checks are not performed until a form is submitted. This is a standard behavior for web-based applications that do not require any client side application. The existing user community has not considered this an issue and in fact experienced users prefer it for heads down-data entry. In fact this is essentially the way data entry has been implemented by the BCG in the past using other technologies (e.g. Oracle Forms).
• No help menus
• DM: There are in fact two distinct types of help menus in Oncore. The first is a set of on-line user documents that can be accessed via the "documentation" link that is always present in the upper left-hand of the application. The other type is institution specific help that is accessed via a question mark symbol in the upper right hand corner of each user interface panel. This type of help is meant to reflect the context of local operations and to be populated by the local institution as needed.
• SN: Due to the extremely limited time in our demo, most of the features were not fully explored. An average demo to familiarize someone with the overall "basics" of the system takes approximately 3 to 4 hours. Maybe we can have a follow up demo.
• FH: I would appreciate getting the VU users of Oncore to comment on their experience with this.
• SN: Thus far, during the past seven months, we have primarily been interacting and using the system in an "attack" mode. Trying to break the system in every circumstance in order to prevent any disasters when released for live use. I can confidently state that the system is extremely solid from the nuts and bolts point of view. Plans are underway to begin training groups of users in different roles. Once this is underway, we will interact with them for reporting. Wendy Smith of the VICC Clinical Trials Office, has been one of the lead operators of this system and is most familiar with its intricate capabilities. We will count on her review on this soon.
• No scripting language or metadata file for specifying data dictionary; user time to implement hundreds of new variables using GUI for forms not already used by the consortium will be astronomical
• DM: Oncore technology does use meta-data. However it is not exposed to the user. The goal of SCORE is to do this in a way that is easy to manage. The term "meta-data" is very general; in the case of SCORE managing the amount of information that is needed to specialize it for a domain is not a trivial task, especially as requirements evolve. Assuming that it is required to add hundreds of new variables, there are existing mechanisms based on SQL for easily loading them into Oncore. Again, it is important to bear in mind that Oncore has been designed specifically for use by cancer centers and it already contains hundreds of variables for oncology. In its current form, it is not intended for use in other disease areas.
• Centers cannot send forms additions to other centers. All centers in consortium have to agree to changes, and wait for yearly updates from company. But can add new fields to patient forms.
• DM: The relationship between PercipEnz and the other members of the consortium is very responsive. Depending on the circumstances, technology updates can occur on a weekly basis and are driven by need and priorities not a release schedule. The real limitation is within the cancer centers and their ability to evaluate and absorb changes or arrive at requirements for changes and additions. One of the goals of the Consortium is to develop the common practices, and standards needed for multi-center integration; this also takes time. However, any center can create and/or modify CRFs to meet their needs. This does not require agreement among the Consortium members. As yet, there is no direct support for exchanging CRF definitions among centers. This is one of the goals of the NHLBI proposal mentioned above. However, any center can act as an affiliate of another center while participating in a multi-center trial. This allows a data manager or research nurse at one center (the affiliate site) to have remote access for entering data into the system of the coordinating center. It is also possible to use more manual methods for exchanging setup information using XML import and export, but there seems to be little demand for this capability at this stage and it is inferior to the mechanisms contemplated for the future.
• SN: I do not think that we were very clear on this issue during the short demo, and did not have a chance to really discuss this. But one of the major benefits we have enjoyed at the Cancer Center (a breath of fresh air compared to what we were used to), has specifically been the interactions and experiences gained from the very pro-active communication we have in place with the other Cancer Centers, and the company Percipenz. In the business of Cancer Centers and clinical trials, what our Consortium has developed is really a first working example of an interaction among centers that produces scientific growth. I like to say that "It is not about the software". It is worth mentioning here that the NCI/NIH has underway a national initiative called caBIG (cancer bio informatic grid) that is trying to do exactly what we have in place. Vanderbilt, and the Consortium are a part of the caBIG initiative development, in a number of ways, and specificaly in the Clinical Trials Arena.
• Export to statistics packages is through Excel spreadsheets; no value labels, variable labels, or units of measurements exported
• DM: There are options for configuring this. Column headings (variable labels) can be included or excluded and units of measure are included when provided in the CRF. All values are identified by subject-sequence-number, initials, visit date, cycle and day number. However, exporting an empty CRF will result in an empty file.
• FH: Column headings are short variable names, not variable labels. Labels and names are separate entities.

Conclusions

Oncore has a lot of potential for clinical trials data management and its success will depend on

• the speed with which it can be generalized to non-cancer trials
• the ease with which new data modules (sub forms) and totally new CRFs can be programmed into the system.

While assessing these two issues, other models should be studied, including X-forms and PDF forms data entry/form rendering models.