Quantitative structure-property relationship (QSPR) models used for prediction of property of untested chemicals can be utilized for prioritization plan of synthesis and experimental testing of new compounds. Validation of QSPR models plays a crucial role for judgment of the reliability of predictions of such models. In the QSPR literature, serious attention is now given to external validation for checking reliability of QSPR models, and predictive quality is in the most cases judged based on the quality of predictions of property of a single test set as reflected in one or more external validation metrics. Here, we have shown that a single QSPR model may show a variable degree of prediction quality as reflected in some variants of external validation metrics like Q²(F1), Q²(F2), Q²(F3), CCC, and r²(m) (all of which are differently modified forms of predicted variance, which theoretically may attain a maximum value of 1), depending on the test set composition and test set size. Thus, this report questions the appropriateness of the common practice of the "classic" approach of external validation based on a single test set and thereby derives a conclusion about predictive quality of a model on the basis of a particular validation metric. The present work further demonstrates that among the considered external validation metrics, r²(m) shows statistically significantly different numerical values from others among which CCC is the most optimistic or less stringent. Furthermore, at a given level of threshold value of acceptance for external validation metrics, r²(m) provides the most stringent criterion (especially with Δr²(m) at highest tolerated value of 0.2) of external validation, which may be adopted in the case of regulatory decision support processes.

Comparative Studies on Some Metrics for External Validation of QSPR Models

Risk assessment for most human health effects is based on the threshold of a toxicological effect, usually derived from animal experiments. The Threshold of Toxicological Concern (TTC) is a concept that refers to the establishment of a level of exposure for all chemicals below which there would be no appreciable risk to human health. When carefully applied, the TTC concept can provide a means of waiving testing based on knowledge of exposure limits. Two main approaches exist; the first of these is a General Threshold of Toxicological Concern; the second approach is a TTC in relation to structural information and/or toxicological data of chemicals. The structural scheme most routinely used is that of Cramer and co-workers from 1978. Recently this scheme was encoded into a software program called Toxtree, specifically commissioned by the European Chemicals Bureau (ECB). Here we evaluate two published datasets using Toxtree to demonstrate its concordance and highlight potential software modifications. The results were promising with an overall good concordance between the reported classifications and those generated by Toxtree. Further evaluation of these results highlighted a number of inconsistencies which were examined in turn and rationalised as far as possible. Improvements for Toxtree were proposed where appropriate. Notable of these is a necessity to update the lists of common food components and normal body constituents as these accounted for the majority of false classifications observed. Overall Toxtree was found to be a useful tool in facilitating the systematic evaluation of compounds through the Cramer scheme.

An evaluation of the implementation of the Cramer classification scheme in the Toxtree software

Lhasa Limited is a not-for-profit organization that exists to promote the sharing of data and knowledge in chemistry and the life sciences. It has developed the software tools Derek for Windows, Meteor, and Vitic to facilitate such sharing. Derek for Windows and Meteor are knowledge-based expert systems that predict the toxicity and metabolism of a chemical, respectively. Vitic is a chemically intelligent toxicity database. An overview of each software system is provided along with examples of the sharing of data and knowledge in the context of their development. These examples include illustrations of (1) the use of data entry and editing tools for the sharing of data and knowledge within organizations; (2) the use of proprietary data to develop nonconfidential knowledge that can be shared between organizations; (3) the use of shared expert knowledge to refine predictions; (4) the sharing of proprietary data between organizations through the formation of data-sharing groups; and (5) the use of pr...

In Silico Tools for Sharing Data and Knowledge on Toxicity and Metabolism: Derek for Windows, Meteor, and Vitic

In the past decades, chemical carcinogenicity has been the object of mechanistic studies that have been translated into valuable experimental (e.g., the Salmonella assays system) and theoretical (e.g., compilations of structure alerts for chemical carcinogenicity) models. These findings remain the basis of the science and regulation of mutagens and carcinogens. Recent advances in the organization and treatment of large databases consisting of both biological and chemical information nowadays allows for a much easier and more refined view of data. This paper reviews recent analyses on the predictive performance of various lists of structure alerts, including a new compilation of alerts that combines previous work in an optimized form for computer implementation. The revised compilation is part of the Toxtree 1.50 software (freely available from the European Chemicals Bureau website). The use of structural alerts for the chemical biological profiling of a large database of Salmonella mutagenicity results is also reported. Together with being a repository of the science on the chemical biological interactions at the basis of chemical carcinogenicity, the SAs have a crucial role in practical applications for risk assessment, for: (a) description of sets of chemicals; (b) preliminary hazard characterization; (c) formation of categories for e.g., regulatory purposes; (d) generation of subsets of congeneric chemicals to be analyzed subsequently with QSAR methods; (e) priority setting. An important aspect of SAs as predictive toxicity tools is that they derive directly from mechanistic knowledge. The crucial role of mechanistic knowledge in the process of applying (Q)SAR considerations to risk assessment should be strongly emphasized. Mechanistic knowledge provides a ground for interaction and dialogue between model developers, toxicologists and regulators, and permits the integration of the (Q)SAR results into a wider regulatory framework, where different types of evidence and data concur or complement each other as a basis for making decisions and taking actions.

Structure alerts for carcinogenicity, and the Salmonella assay system: a novel insight through the chemical relational databases technology.

A European perspective on alternatives to animal testing for environmental hazard identification and risk assessment

Under the proposed REACH (Registration, Evaluation and Authorisation of CHemicals) legislation, (Q)SAR models and grouping methods (chemical categories and read across approaches) are expected to play a significant role in prioritising industrial chemicals for further assessment, and for filling information gaps for the purposes of classification and labelling, risk assessment and the assessment of persistent, bioaccumulative and toxic (PBT) chemicals. The European Chemicals Bureau (ECB), which is part of the European Commission's Joint Research Centre (JRC), has a well-established role in providing independent scientific and technical advice to European policy makers. The ECB also promotes consensus and capacity building on scientific and technical matters among stakeholders in the Member State authorities and industry. To promote the availability and use of (Q)SARs and related estimation methods, the ECB is carrying out a range of activities, including applied research in computational toxicology, the assessment of (Q)SAR models and methods, the development of technical guidance documents and computational tools, and the organisation of training courses. This article provides an overview of ECB activities on computational toxicology, which are intended to promote the development, validation, acceptance and use of (Q)SARs and related estimation methods, both at the European and international levels.

Eisenreich S

Papers

The role of the European Chemicals Bureau in promoting the regulatory use of (Q)SAR methods.