nestle contributions: introduction and methods

1 files changed, 143 insertions, 116 deletions

diff --git a/loael.Rmd b/loael.Rmd
index b01e1e3..6def111 100644
--- a/loael.Rmd
+++ b/loael.Rmd
@@ -30,63 +30,82 @@ rmse <-  function(x,y) { sqrt(mean((x-y)^2,na.rm=TRUE)) }
 Introduction
 ============
 
-Elena + Benoit
-
-The quality and reproducibility of (Q)SAR and  read-across predictions is a
-controversial topic in the toxicological risk-assessment community. Although
-model predictions can be validated with various procedures it is rarely
-possible to put the results into the context of experimental variability,
-because replicate experiments are usually not available.
-
-With missing information about the variability of experimental toxicity data it
-is hard to judge the performance of predictive models objectively and it is tempting for
-model developers to use aggressive model optimisation methods that lead to
-impressive validation results, but also to overfitted models with little
-practical relevance.
-
-In this study we intent to compare model predictions with experimental
-variability with chronic oral rat lowest adverse effect levels (LOAEL) as
-toxicity endpoint.  We are using two datasets, one from [@mazzatorta08]
-(*Mazzatorta* dataset) and one from the Swiss Federal Office of TODO (*Swiss
-Federal Office* dataset).
-
-Elena: do you have a reference and the name of the department?
-
-```{r echo=F}
-m = read.csv("data/mazzatorta_log10.csv",header=T)
-s = read.csv("data/swiss_log10.csv",header=T)
-t = read.csv("data/test_log10.csv",header=T)
-c = read.csv("data/training_log10.csv",header=T)
-```
-
-`r length(unique(t$SMILES))` compounds are common in both datasets and we use
-them as a *test* set in our investigation. For the Mazzatorta and Swiss Federal Office datasets we will
-
-- compare the structural diversity of both datasets
-- compare the LOAEL values in both datasets
-- build prediction models 
-- predict LOAELs of the test set
-- compare predictions with experimental variability
-
-With this investigation we also want to support the idea of reproducible
-research, by providing all datasets and programs that have been used to
-generate this manuscript under 
-GPL3 licenses.
-
-A self-contained docker image with all programs, libraries and data required for the
-reproduction of these results is available from <https://hub.docker.com/r/insilicotox/loael-paper/>.
+Relying on standard animal toxicological testing for chemical hazard
+identification and characterization is increasingly questioned on both
+scientific and ethical grounds. In addition, it appears obvious that
+from a resource perspective, the capacity of standard toxicology to
+address the safety of thousands of untested chemicals (Fowler et al.,
+2011) to which human may be exposed is very limited. It has also been
+recognized that getting rapid insight on toxicity of chemicals in case
+of emergency safety incidents or for early prioritization in research
+and development (safety by design) is a big challenge mainly because of
+the time and cost constraints associated with the generation of relevant
+animal data. In this context, alternative approaches to obtain timely
+and fit-for-purpose toxicological information are being developed.
+Amongst others, non-testing, structure-activity based *in silico*
+toxicology methods (also called computational toxicology) are considered
+highly promising. Importantly, they are raising more and more interests
+and getting increased acceptance in various regulatory (e.g. ECHA, 2008;
+EFSA, 2016, 2014; Health Canada, 2016; OECD, 2015) and industrial (e.g.
+Stanton and Kruszewski, 2016; Lo Piparo et al., 2011) frameworks.
+
+For a long time already, computational methods have been an integral
+part of pharmaceutical discovery pipelines, while in chemical food
+safety their actual potentials emerged only recently (Lo Piparo et al.,
+2011). In this later field, an application considered critical is in the
+establishment of levels of safety concern in order to rapidly and
+efficiently manage toxicologically uncharacterized chemicals identified
+in food. This requires a risk-based approach to benchmark exposure with
+a quantitative value of toxicity relevant for risk assessment (Schilter
+et al., 2014a). Since most of the time chemical food safety deals with
+life-long exposures to relatively low levels of chemicals, and because
+long-term toxicity studies are often the most sensitive in food
+toxicology databases, predicting chronic toxicity is of prime
+importance. Up to now, read across and quantitative structure-activity
+relationship (QSAR) have been the most used *in silico* approaches to
+obtain quantitative predictions of chronic toxicity.
+
+The quality and reproducibility of (Q)SAR and read-across predictions
+has been a continuous and controversial topic in the toxicological
+risk-assessment community. Although model predictions can be validated
+with various procedures, to review results in context of experimental
+variability has actually been rarely done or attempted. With missing
+information about the variability of experimental toxicity data it is
+hard to judge the performance of predictive models objectively and it is
+tempting for model developers to use aggressive model optimisation
+methods that lead to impressive validation results, but also to
+overfitted models with little practical relevance.
+
+In the present study, automatic read-across like models were built to
+generate quantitative predictions of long-term toxicity. Two databases
+compiling chronic oral rat lowest adverse effect levels (LOAEL) as
+endpoint were used. An early review of the databases revealed that many
+chemicals had at least two independent studies/LOAELs. These studies
+were exploited to generate information on the reproducibility of chronic
+animal studies and were used to evaluate prediction performance of the
+models in the context of experimental variability.
+
+An important limitation often raised for computational toxicology is the
+lack of transparency on published models and consequently on the
+difficulty for the scientific community to reproduce and apply them. To
+overcome these issues, all databases and programs that have been used to
+generate this manuscript are made available under GPL3 licenses.
+
+A self-contained docker image with all programs, libraries and data
+required for the reproduction of these results is available from
+<https://hub.docker.com/r/insilicotox/loael-paper/>.
 
 Source code and datasets for the reproduction of this manuscript can be
-downloaded from the GitHub repository <https://github.com/opentox/loael-paper>. The lazar framework [@Maunz2013] is
-also available under a GPL3 License from <https://github.com/opentox/lazar>.
+downloaded from the GitHub repository
+<https://github.com/opentox/loael-paper>. The lazar framework [@Maunz2013]
+is also available under a GPL3 License from
+<https://github.com/opentox/lazar>.
 
 A graphical webinterface for `lazar` model predictions and validation results
 is publicly accessible at <https://lazar.in-silico.ch>, models presented in
 this manuscript will be included in future versions. Source code for the GUI
 can be obtained from <https://github.com/opentox/lazar-gui>.
 
-Elena: please check if this is publication strategy is ok for the Swiss Federal Office
-
 Materials and Methods
 =====================
 
@@ -97,35 +116,39 @@ and datasets, links to source code and data sources are included in the text.
 Datasets
 --------
 
-### Mazzatorta dataset
+### Nestlé database
 
-The first dataset (*Mazzatorta* dataset for further reference) originates from
-the publication of [@mazzatorta08]. It contains chronic (> 180 days) lowest
+The first database (Nestlé database for further reference) originates
+from the publication of [@mazzatorta08]. It contains chronic (> 180 days) lowest
 observed effect levels (LOAEL) for rats (*Rattus norvegicus*) after oral
-(gavage, diet, drinking water) administration.  The Mazzatorta dataset consists
+(gavage, diet, drinking water) administration.  The Nestlé database consists
 of `r length(m$SMILES)` LOAEL values for `r length(unique(m$SMILES))` unique
 chemical structures.
-The Mazzatorta dataset can be obtained from the following GitHub links: [original data](https://github.com/opentox/loael-paper/blob/submission/data/LOAEL_mg_corrected_smiles_mmol.csv),
+The Nestlé database can be obtained from the following GitHub links: [original data](https://github.com/opentox/loael-paper/blob/submission/data/LOAEL_mg_corrected_smiles_mmol.csv),
 [unique smiles](https://github.com/opentox/loael-paper/blob/submission/data/mazzatorta.csv),
 [-log10 transfomed LOAEL](https://github.com/opentox/loael-paper/blob/submission/data/mazzatorta_log10.csv).
 
-### Swiss Federal Office dataset
+### Swiss Food Safety and Veterinary Office (FSVO) database
 
-Elena + Swiss Federal Office contribution (input)
+Publicly available data from pesticide evaluations of chronic rat
+toxicity studies from the European Food Safety Authority (EFSA) (EFSA,
+2014), the Joint FAO/WHO Meeting on Pesticide Residues (JMPR) (WHO,
+2011) and the US EPA (US EPA, 2011) were compiled to form the
+FSVO-database. Only studies providing both an experimental NOAEL and an
+experimental LOAEL were included. The LOAELs were taken as they were
+reported in the evaluations. Further details on the database are
+described elsewhere (Zarn et al., 2011; Zarn et al., 2013). The
+FSVO-database consists of `r length(s$SMILES)` rat LOAEL values for `r length(unique(s$SMILES))` unique chemical
+structures. It can be obtained from the following GitHub links:
 
-The original Swiss Federal Office dataset has chronic toxicity data for rats,
-mice and multi generation effects. For the purpose of this study only rat LOAEL
-data with oral administration was used. This leads to the *Swiss Federal
-Office*  dataset with `r length(s$SMILES)` rat LOAEL values for
-`r length(unique(s$SMILES))` unique chemical structures.
-The Swiss dataset can be obtained from the following GitHub links: [original data](https://github.com/opentox/loael-paper/blob/submission/data/NOAEL-LOAEL_SMILES_rat_chron.csv), 
+[original data](https://github.com/opentox/loael-paper/blob/submission/data/NOAEL-LOAEL_SMILES_rat_chron.csv), 
 [unique smiles and mmol/kg_bw/day units](https://github.com/opentox/loael-paper/blob/submission/data/swiss.csv),
 [-log10 transfomed LOAEL](https://github.com/opentox/loael-paper/blob/submission/data/swiss_log10.csv).
 
 ### Preprocessing
 
 Chemical structures (represented as SMILES [@doi:10.1021/ci00057a005]) in both
-datasets were checked for correctness. Syntactically incorrect and missing
+datasets were checked for correctness. When syntactically incorrect or missing
 SMILES were generated from other identifiers (e.g names, CAS numbers). Unique
 smiles from the OpenBabel library [@OBoyle2011] were used for the
 identification of duplicated structures. 
@@ -137,26 +160,25 @@ significant digits. For prediction, validation and visualisation purposes
 
 ### Derived datasets
 
-Two derived datasets were obtained from the original datasets: 
+Two derived datasets were obtained from the original databases:
 
-The [*test* dataset](https://github.com/opentox/loael-paper/blob/submission/data/test_log10.csv)
-contains data from compounds that occur in both datasets.
-LOAEL values equal at five significant digits were considered as duplicates
-originating from the same study/publication and only one instance was kept in
-the test dataset.  The test dataset has
-`r length(t$SMILES)` LOAEL values for `r length(unique(t$SMILES))` unique
-chemical structures and was used for
+The [*test*
+dataset](https://github.com/opentox/loael-paper/blob/submission/data/test_log10.csv)
+contains data from compounds that occur in both databases. LOAEL values equal
+at five significant digits were considered as duplicates originating from the
+same study/publication and only one instance was kept in the test dataset. The
+test dataset has `r length(t$SMILES)` LOAEL values for `r
+length(unique(t$SMILES))` unique chemical structures and was used for
 
 - evaluating experimental variability
-- comparing model predictions with experimental variaility.
-
-The [*training* dataset](https://github.com/opentox/loael-paper/blob/submission/data/training_log10.csv)
-is the union of the Mazzatorta and the Swiss Federal
-Office dataset and it is used to build predictive models. LOAEL duplicates were
-removed using the same criteria as for the test dataset.  The
-training dataset 
-has `r length(c$SMILES)` LOAEL values for `r length(unique(c$SMILES))` unique
-chemical structures.
+- comparing model predictions with experimental variability.
+
+The [*training*
+dataset](https://github.com/opentox/loael-paper/blob/submission/data/training_log10.csv)
+is the union of the Nestlé and the FSVO databases and it was used to build
+predictive models. LOAEL duplicates were removed using the same criteria as for
+the test dataset.  The training dataset has `r length(c$SMILES)` LOAEL values
+for `r length(unique(c$SMILES))` unique chemical structures.
 
 Algorithms
 ----------
@@ -218,7 +240,11 @@ threshold) and the number of predictable compounds (low threshold). As it is in
 many practical cases desirable to make predictions even in the absence of
 closely related neighbors, we follow a tiered approach: 
 
-First a similarity threshold of 0.5 is used to collect neighbors, to create a local QSAR model and to make a prediction for the query compound. If any of this steps fail, the procedure is repeated with a similarity threshold of 0.2 and the prediction is flagged with a warning that it might be out of the applicability domain of the training data.
+First a similarity threshold of 0.5 is used to collect neighbors, to create
+a local QSAR model and to make a prediction for the query compound. If any of
+this steps fail, the procedure is repeated with a similarity threshold of 0.2
+and the prediction is flagged with a warning that it might be out of the
+applicability domain of the training data.
 
 Compounds with the same structure as the query structure are automatically
 [eliminated from neighbors](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/model.rb#L180-L257)
@@ -228,27 +254,27 @@ duplicates.
 ### Local QSAR models and predictions
 
 Only similar compounds (*neighbors*) above the threshold are used for local
-QSAR models.  In this investigation we are using
-[weighted random forests regression (RF)](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/caret.rb#L7-L78)
-for the prediction of quantitative
-properties.  First all uninformative fingerprints (i.e. features with identical
-values across all neighbors) are removed.  The remaining set of features is
-used as descriptors for creating a local weighted RF model with atom
-environments as descriptors and model similarities as weights. The RF method
-from the `caret` R package [@Kuhn08] is used for this purpose.  Models are
-trained with the default `caret` settings, optimizing the number of RF
-components by bootstrap resampling.
-
-Finally the local RF model is applied to
-[predict the activity](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/model.rb#L194-L272)
-of the query
-compound. The RMSE of bootstrapped local model predictions is used to construct 95\%
-prediction intervals at 1.96*RMSE.
-
-If RF modelling or prediction fails, the program resorts to using the
-[weighted mean](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/regression.rb#L6-L16)
+QSAR models.  In this investigation we are using [weighted random forests
+regression
+(RF)](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/caret.rb#L7-L78)
+for the prediction of quantitative properties.  First all uninformative
+fingerprints (i.e. features with identical values across all neighbors) are
+removed.  The remaining set of features is used as descriptors for creating
+a local weighted RF model with atom environments as descriptors and model
+similarities as weights. The RF method from the `caret` R package [@Kuhn08] is
+used for this purpose.  Models are trained with the default `caret` settings,
+optimizing the number of RF components by bootstrap resampling.
+
+Finally the local RF model is applied to [predict the
+activity](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/model.rb#L194-L272)
+of the query compound. The RMSE of bootstrapped local model predictions is used
+to construct 95\% prediction intervals at 1.96*RMSE.
+
+If RF modelling or prediction fails, the program resorts to using the [weighted
+mean](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/regression.rb#L6-L16)
 of the neighbors LOAEL values, where the contribution of each neighbor is
-weighted by its similarity to the query compound. In this case the prediction is also flagged with a warning.
+weighted by its similarity to the query compound. In this case the prediction
+is also flagged with a warning.
 
 ### Applicability domain
 
@@ -270,20 +296,21 @@ interval associated with each prediction.
 ### Validation
 
 For the comparison of experimental variability with predictive accuracies we
-are using a test set of compounds that occur in both datasets. Unbiased read
-across predictions are obtained from the *training* dataset, by
-[removing *all* information](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/model.rb#L234-L238) 
-from the test compound from the training set prior to predictions.
-This procedure is hardcoded into the prediction algorithm in order to prevent
+are using a test set of compounds that occur in both databases. Unbiased read
+across predictions are obtained from the *training* dataset, by [removing *all*
+information](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/model.rb#L234-L238)
+from the test compound from the training set prior to predictions. This
+procedure is hardcoded into the prediction algorithm in order to prevent
 validation errors. As we have only a single test set no model or parameter
 optimisations were performed in order to avoid overfitting a single dataset.
 
-Results from 3 repeated
-[10-fold crossvalidations](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/crossvalidation.rb#L85-L93)
-with independent training/test
-set splits are provided as additional information to the test set results.
+Results from 3 repeated [10-fold
+crossvalidations](https://github.com/opentox/lazar/blob/loael-paper.submission/lib/crossvalidation.rb#L85-L93)
+with independent training/test set splits are provided as additional
+information to the test set results.
 
-The final model for production purposes was trained with all available LOAEL data (Mazzatorta and Swiss Federal Office datasets combined).
+The final model for production purposes was trained with all available LOAEL
+data (Nestlé and FSVO databases combined).
 
 ## Availability
 
@@ -339,7 +366,7 @@ used with different kinds of features. We have investigated structural as well
 as physico-chemical properties and concluded that both datasets are very
 similar, both in terms of chemical structures and physico-chemical properties. 
 
-The only statistically significant difference between both datasets, is that the Mazzatorta dataset contains more small compounds (61 structures with less than 11 atoms) than the Swiss dataset (19 small structures, p-value 3.7E-7).
+The only statistically significant difference between both datasets, is that the Nestlé database contains more small compounds (61 structures with less than 11 atoms) than the Swiss dataset (19 small structures, p-value 3.7E-7).
 
 <!--
 [@fig:ches-mapper-pc] shows an embedding that is based on physico-chemical (PC)
@@ -352,10 +379,10 @@ Martin: please explain light colors at bottom of histograms
 In this example, CheS-Mapper applied a principal components analysis to map
 compounds according to their physico-chemical (PC) feature values into 3D
 space. Both datasets have in general very similar PC feature values. As an
-exception, the Mazzatorta dataset includes most of the tiny compound
+exception, the Nestlé database includes most of the tiny compound
 structures: we have selected the 78 smallest compounds (with 10 atoms and less,
 marked with a blue box in the screen-shot) and found that 61 of these compounds
-occur in the Mazzatorta dataset, whereas only 19 are contained in the Swiss
+occur in the Nestlé database, whereas only 19 are contained in the Swiss
 dataset (p-value 3.7E-7).
 
 This result was confirmed for structural features (fingerprints) including
@@ -405,7 +432,7 @@ c.mg = read.csv("data/all_mg_dup.csv",header=T)
 c.mg$sd <- ave(c.mg$LOAEL,c.mg$SMILES,FUN=sd)
 ```
 
-The Mazzatorta dataset has `r length(m$SMILES)` LOAEL values for
+The Nestlé database has `r length(m$SMILES)` LOAEL values for
 `r length(levels(m$SMILES))` unique structures, `r m.dupnr`
 compounds have multiple measurements with a mean standard deviation (-log10 transformed values) of
 `r round(mean(m.dup$sd),2)`
@@ -581,7 +608,7 @@ Elena + Benoit
 
 ### Dataset comparison
 
-Our investigations clearly indicate that the Mazzatorta and Swiss Federal Office datasets are very similar in terms of chemical structures and properties and the distribution of experimental LOAEL values. The only significant difference that we have observed was that the Mazzatorta dataset has larger amount of small molecules, than the Swiss Federal Office dataset. For this reason we have pooled both dataset into a single training dataset for read across predictions.
+Our investigations clearly indicate that the Mazzatorta and Swiss Federal Office datasets are very similar in terms of chemical structures and properties and the distribution of experimental LOAEL values. The only significant difference that we have observed was that the Nestlé database has larger amount of small molecules, than the Swiss Federal Office dataset. For this reason we have pooled both dataset into a single training dataset for read across predictions.
 
 [@fig:intra] and [@fig:corr] and [@tbl:common-pred] show however considerable
 variability in the experimental data. High experimental variability has an