new tensorflow predictions, nix configuration, t-sne figure

1 files changed, 241 insertions, 2 deletions

diff --git a/mutagenicity.md b/mutagenicity.md
index 69341d1..321a460 100644
--- a/mutagenicity.md
+++ b/mutagenicity.md
@@ -1,5 +1,5 @@
 ---
-title: A comparison of random forest, support vector machine, deep learning and lazar algorithms for predicting mutagenicity
+title: A comparison of random forest, support vector machine, linear regression, deep learning and lazar algorithms for predicting the mutagenic potential of different pyrrolizidine alkaloids 
 #subtitle: Performance comparison with a new expanded dataset
 author:
   - Christoph Helma:
@@ -49,10 +49,66 @@ Introduction
 
 TODO
 
+Pyrrolizidine alkaloids (PAs) are secondary plant ingredients found in
+many plant species as protection against predators [Hartmann & Witte
+1995](#_ENREF_59)[Langel et al. 2011](#_ENREF_76)(; ). PAs are ester
+alkaloids, which are composed of a necine base (two fused five-membered
+rings joined by a nitrogen atom) and one or two necic acid (carboxylic
+ester arms). The necine base can have different structures and thereby
+divides PAs into several structural groups, e.g. otonecine, platynecine,
+and retronecine. The structural groups of the necic acid are macrocyclic
+diester, open-ring diester and monoester [Langel et al.
+2011](#_ENREF_76)().
+
+PA are mainly metabolised in the liver, which is at the same time the
+main target organ of toxicity [Bull & Dick 1959](#_ENREF_17)[Bull et al.
+1958](#_ENREF_18)[Butler et al. 1970](#_ENREF_20)[DeLeve et al.
+1996](#_ENREF_33)[Jago 1971](#_ENREF_65)[Li et al.
+2011](#_ENREF_78)[Neumann et al. 2015](#_ENREF_99)(; ; ; ; ; ; ). There
+are three principal metabolic pathways for 1,2-unsaturated PAs [Chen et
+al. 2010](#_ENREF_26)(): (i) Detoxification by hydrolysis: the ester
+bond on positions C7 and C9 are hydrolysed by non-specific esterases to
+release necine base and necic acid, which are then subjected to further
+phase II-conjugation and excretion. (ii) Detoxification by *N*-oxidation
+of the necine base (only possible for retronecine-type PAs): the
+nitrogen is oxidised to form a PA *N*-oxides, which can be conjugated by
+phase II enzymes e.g. glutathione and then excreted. PA *N*-oxides can
+be converted back into the corresponding parent PA [Wang et al.
+2005](#_ENREF_134)(). (iii) Metabolic activation or toxification: PAs
+are metabolic activated/ toxified by oxidation (for retronecine-type
+PAs) or oxidative *N*-demethylation (for otonecine-type PAs [Lin
+1998](#_ENREF_82)()). This pathway is mainly catalysed by cytochrome
+P450 isoforms CYP2B and 3A [Ruan et al. 2014b](#_ENREF_115)(), and
+results in the formation of dehydropyrrolizidines (DHP, also known as
+pyrrolic ester or reactive pyrroles). DHPs are highly reactive and cause
+damage in the cells where they are formed, usually hepatocytes. However,
+they can also pass from the hepatocytes into the adjacent sinusoids and
+damage the endothelial lining cells [Gao et al. 2015](#_ENREF_48)()
+predominantly by reaction with protein, lipids and DNA. There is even
+evidence, that conjugation of DHP to glutathione, which would generally
+be considered a detoxification step, could result in reactive
+metabolites, which might also lead to DNA adduct formation [Xia et al.
+2015](#_ENREF_138)(). Due to the ability to form DNA adducts, DNA
+crosslinks and DNA breaks 1,2-unsaturated PAs are generally considered
+genotoxic and carcinogenic [Chen et al. 2010](#_ENREF_26)[EFSA
+2011](#_ENREF_36)[Fu et al. 2004](#_ENREF_45)[Li et al.
+2011](#_ENREF_78)[Takanashi et al. 1980](#_ENREF_126)[Yan et al.
+2008](#_ENREF_140)[Zhao et al. 2012](#_ENREF_148)(; ; ; ; ; ; ). Still,
+there is no evidence yet that PAs are carcinogenic in humans [ANZFA
+2001](#_ENREF_4)[EMA 2016](#_ENREF_39)(; ). One general limitation of
+studies with PAs is the number of different PAs investigated. Around 30
+PAs are currently commercially available, therefore all studies focus on
+these PAs. This is also true for *in vitro* and *in vivo* tests on
+mutagenicity and genotoxicity. To gain a wider perspective, in this
+study over 600 different PAs were assessed on their mutagenic potential
+using four different machine learning techniques.
+
+
 The main objectives of this study were
 
   - to generate a new training dataset, by combining the most comprehensive public mutagenicity datasets
   - to compare the performance of global models (RF, SVM, Neural Nets) with local models (`lazar`)
+  - to apply these models for the prediction of the mutagenicity of pyrrolizidine alkaloids
 
 Materials and Methods
 =====================
@@ -60,6 +116,8 @@ Materials and Methods
 Data
 ----
 
+### Mutagenicity training data
+
 An identical training dataset was used for all models. The
 training dataset was compiled from the following sources:
 
@@ -87,6 +145,39 @@ available from the git repository <https://git.in-silico.ch/mutagenicity-paper>
 under a GPL3 License. The new combined dataset can be found at
 <https://git.in-silico.ch/mutagenicity-paper/data/mutagenicity.csv>.
 
+### Pyrrolizidine dataset
+
+The testing dataset consisted of 602 different PAs. The compilation of
+the PA dataset is described in detail in [Schöning et al.
+(2017)](#_ENREF_119). The PAs were assigned to groups according to
+structural features of the necine base and necic acid.
+
+For the necine base, following groups were assigned:
+
+-   Retronecine-type (1,2-unstaturated necine base)
+
+-   Otonecine-type (1,2-unstaturated necine base)
+
+-   Platynecine-type (1,2-saturated necine base)
+
+For the modification of necine base, following groups were assigned:
+
+-   *N*-oxide-type
+
+-   Tertiary-type (PAs which were neither from the *N*-oxide- nor
+    > DHP-type)
+
+-   DHP-type (dehydropyrrolizidine, pyrrolic ester)
+
+For the necic acid, following groups were assigned:
+
+-   Monoester-type
+
+-   Open-ring diester-type
+
+-   Macrocyclic diester-type
+
+
 Algorithms
 ----------
 
@@ -295,7 +386,15 @@ to the training dataset. Therefore, PA dataset is within the AD of the
 training dataset and the models can be used to predict the genotoxic
 potential of the PA dataset.
 
-### TensorFlow Deep Learning
+### TensorFlow models
+
+#### Logistic regression (SGD)
+
+#### Logistic regression (scikit)
+
+#### Random forests
+
+#### Deep Learning
 
 Alternatively, a DL model was established with Python-based TensorFlow
 program (<https://www.tensorflow.org/>) using the high-level API Keras
@@ -447,6 +546,9 @@ The results of all crossvalidation experiments are summarized in @tbl:summary.
 
 ![ROC plot of crossvalidation results. *R-RF*: R Random Forests, *R-SVM*: R Support Vector Machines, *R-DL*: R Deep Learning, *TF*: TensorFlow without feature selection, *TF-FS*: TensorFlow with feature selection, *L*: lazar, *L-HC*: lazar high confidence predictions, *L-P*: lazar with PaDEL descriptors, *L-P-HC*: lazar PaDEL high confidence predictions (overlaps with L-P)](figures/roc.png){#fig:roc}
 
+Predictions for pyrrolizidine alkaloid  mutagenicity
+----------------------------------------------------
+
 Discussion
 ==========
 
@@ -551,6 +653,123 @@ model algorithm or the descriptor type are the reason for the observed
 differences. In order to answer this question, we would have to use global
 modelling algorithms that are capable to handle large, sparse binary matrices.
 
+Mutagenicity of PAs
+-------------------
+
+@fig:tsne-mp2d shows the position of pyrrolizidine alkaloids (PA) in the mutagenicity training dataset
+
+![t-sne visualisation of mutagenicty training data and pyrrolizidine alkaloids (PA)](figures/tsne-mp2d.png){#fig:tsne-mp2d}
+
+Due to the low to moderate predictivity of all models, quantitative
+statement on the genotoxicity of single PAs cannot be made with
+sufficient confidence.
+
+The predictions of the SVM model did not fit with the other models or
+literature, and are therefore not further considered in the discussion.
+
+Necic acid
+
+The rank order of the necic acid is comparable in the four models
+considered (LAZAR, RF and DL (R-project and TensorFlow). PAs from the
+monoester type had the lowest genotoxic potential, followed by PAs from
+the open-ring diester type. PAs with macrocyclic diesters had the
+highest genotoxic potential. The result fit well with current state of
+knowledge: in general, PAs, which have a macrocyclic diesters as necic
+acid, are considered more toxic than those with an open-ring diester or
+monoester [EFSA 2011](#_ENREF_36)[Fu et al. 2004](#_ENREF_45)[Ruan et
+al. 2014b](#_ENREF_115)(; ; ).
+
+Necine base
+
+The rank order of necine base is comparable in LAZAR, RF, and DL
+(R-project) models: with platynecine being less or as genotoxic as
+retronecine, and otonecine being the most genotoxic. In the
+TensorFlow-generate DL model, platynecine also has the lowest genotoxic
+probability, but are then followed by the otonecines and last by
+retronecine. These results partly correspond to earlier published
+studies. Saturated PAs of the platynecine-type are generally accepted to
+be less or non-toxic and have been shown in *in vitro* experiments to
+form no DNA-adducts [Xia et al. 2013](#_ENREF_139)(). Therefore, it is
+striking, that 1,2-unsaturated PAs of the retronecine-type should have
+an almost comparable genotoxic potential in the LAZAR and DL (R-project)
+model. In literature, otonecine-type PAs were shown to be more toxic
+than those of the retronecine-type [Li et al. 2013](#_ENREF_80)().
+
+Modifications of necine base
+
+The group-specific results of the TensorFlow-generated DL model appear
+to reflect the expected relationship between the groups: the low
+genotoxic potential of *N*-oxides and the highest potential of
+dehydropyrrolizidines [Chen et al. 2010](#_ENREF_26)().
+
+In the LAZAR model, the genotoxic potential of dehydropyrrolizidines
+(DHP) (using the extended AD) is comparable to that of tertiary PAs.
+Since, DHP is regarded as the toxic principle in the metabolism of PAs,
+and known to produce protein- and DNA-adducts [Chen et al.
+2010](#_ENREF_26)(), the LAZAR model did not meet this expectation it
+predicted the majority of DHP as being not genotoxic. However, the
+following issues need to be considered. On the one hand, all DHP were
+outside of the stricter AD of 0.5. This indicates that in general, there
+might be a problem with the AD. In addition, DHP has two unsaturated
+double bounds in its necine base, making it highly reactive. DHP and
+other comparable molecules have a very short lifespan, and usually
+cannot be used in *in vitro* experiments. This might explain the absence
+of suitable neighbours in LAZAR.
+
+Furthermore, the probabilities for this substance groups needs to be
+considered, and not only the consolidated prediction. In the LAZAR
+model, all DHPs had probabilities for both outcomes (genotoxic and not
+genotoxic) mainly below 30%. Additionally, the probabilities for both
+outcomes were close together, often within 10% of each other. The fact
+that for both outcomes, the probabilities were low and close together,
+indicates a lower confidence in the prediction of the model for DHPs.
+
+In the DL (R-project) and RF model, *N*-oxides have a by far more
+genotoxic potential that tertiary PAs or dehydropyrrolizidines. As PA
+*N*-oxides are easily conjugated for extraction, they are generally
+considered as detoxification products, which are *in vivo* quickly
+renally eliminated [Chen et al. 2010](#_ENREF_26)(). On the other hand,
+*N*-oxides can be also back-transformed to the corresponding tertiary PA
+[Wang et al. 2005](#_ENREF_134)(). Therefore, it may be questioned,
+whether *N*-oxides themselves are generally less genotoxic than the
+corresponding tertiary PAs. However, in the groups of modification of
+the necine base, dehydropyrrolizidine, the toxic principle of PAs,
+should have had the highest genotoxic potential. Taken together, the
+predictions of the modifications of the necine base from the LAZAR, RF
+and R-generated DL model cannot -- in contrast to the TensorFlow DL
+model - be considered as reliable.
+
+Overall, when comparing the prediction results of the PAs to current
+published knowledge, it can be concluded that the performance of most
+models was low to moderate. This might be contributed to the following
+issues:
+
+1.  In the LAZAR model, only 26.6% PAs were within the stricter AD. With
+    the extended AD, 92.3% of the PAs could be included in the
+    prediction. Even though the Jaccard distance between the training
+    dataset and the PA dataset for the RF, SVM, and DL (R-project and
+    TensorFlow) models was small, suggesting a high similarity, the
+    LAZAR indicated that PAs have only few local neighbours, which might
+    adversely affect the prediction of the mutagenic potential of PAs.
+
+2.  All above-mentioned models were used to predict the mutagenicity of
+    PAs. PAs are generally considered to be genotoxic, and the mode of
+    action is also known. Therefore, the fact that some models predict
+    the majority of PAs as not genotoxic seems contradictory. To
+    understand this result, the basis, the training dataset, has to be
+    considered. The mutagenicity of in the training dataset are based on
+    data of mutagenicity in bacteria. There are some studies, which show
+    mutagenicity of PAs in the AMES test [Chen et al.
+    2010](#_ENREF_26)(). Also, [Rubiolo et al. (1992)](#_ENREF_116)
+    examined several different PAs and several different extracts of
+    PA-containing plants in the AMES test. They found that the AMES test
+    was indeed able to detect mutagenicity of PAs, but in general,
+    appeared to have a low sensitivity. The pre-incubation phase for
+    metabolic activation of PAs by microsomal enzymes was the
+    sensitivity-limiting step. This could very well mean that this is
+    also reflected in the QSAR models.
+
+
 Conclusions
 ===========
 
@@ -562,5 +781,25 @@ test. Differences between algorithms (local vs. global models) and/or
 descriptors (MolPrint2D vs PaDEL) may be responsible for the different
 prediction accuracies. 
 
+In this study, an attempt was made to predict the genotoxic potential of
+PAs using five different machine learning techniques (LAZAR, RF, SVM, DL
+(R-project and TensorFlow). The results of all models fitted only partly
+to the findings in literature, with best results obtained with the
+TensorFlow DL model. Therefore, modelling allows statements on the
+relative risks of genotoxicity of the different PA groups. Individual
+predictions for selective PAs appear, however, not reliable on the
+current basis of the used training dataset.
+
+This study emphasises the importance of critical assessment of
+predictions by QSAR models. This includes not only extensive literature
+research to assess the plausibility of the predictions, but also a good
+knowledge of the metabolism of the test substances and understanding for
+possible mechanisms of toxicity.
+
+In further studies, additional machine learning techniques or a modified
+(extended) training dataset should be used for an additional attempt to
+predict the genotoxic potential of PAs.
+
+
 References
 ==========