--- title: A comparison of nine machine learning mutagenicity models and their application for predicting pyrrolizidine alkaloids author: - Christoph Helma: institute: ist email: helma@in-silico.ch correspondence: "yes" - Verena Schöning: institute: insel - Jürgen Drewe: institute: zeller, unibas - Philipp Boss: institute: sysbio institute: - ist: name: in silico toxicology gmbh address: "Rastatterstrasse 41, 4057 Basel, Switzerland" - zeller: name: Max Zeller Söhne AG address: "Seeblickstrasse 4, 8590 Romanshorn, Switzerland" - sysbio: name: Berlin Institute for Medical Systems Biology, Max Delbrück Center for Molecular Medicine in the Helmholtz Association address: "Robert-Rössle-Strasse 10, Berlin, 13125, Germany" - unibas: name: Clinical Pharmacology, Department of Pharmaceutical Sciences, University Hospital Basel, University of Basel address: "Petersgraben 4, 4031 Basel, Switzerland" - insel: name: Clinical Pharmacology and Toxicology, Department of General Internal Medicine, University Hospital Bern, University of Bern address: "Inselspital, 3010 Bern, Switzerland" bibliography: bibliography.bib keywords: mutagenicity, QSAR, lazar, random forest, support vector machine, linear regression, neural nets, deep learning, pyrrolizidine alkaloids, OpenBabel, CDK documentclass: scrartcl tblPrefix: Table figPrefix: Figure header-includes: - \usepackage{lineno, setspace, color, colortbl, longtable} - \doublespacing - \linenumbers ... Abstract ======== Random forest, support vector machine, logistic regression, neural networks and k-nearest neighbor (`lazar`) algorithms, were applied to new *Salmonella* mutagenicity dataset with {{cv.n_uniq}} unique chemical structures utilizing MolPrint2D and Chemistry Development Kit (CDK) descriptors. Crossvalidation accuracies of all investigated models ranged from 80-85% which is comparable with the interlaboratory variability of the *Salmonella* mutagenicity assay. Pyrrolizidine alkaloid predictions showed a clear distinction between chemical groups, where Otonecines had the highest proportion of positive mutagenicity predictions and Monoester the lowest. Introduction ============ **TODO**: rationale for investigation The main objectives of this study were - to generate a new mutagenicity training dataset, by combining the most comprehensive public datasets - to compare the performance of MolPrint2D (*MP2D*) fingerprints with Chemistry Development Kit (*CDK*) descriptors - to compare the performance of global QSAR models (random forests (*RF*), support vector machines (*SVM*), logistic regression (*LR*), neural nets (*NN*)) with local models (`lazar`) - to apply these models for the prediction of pyrrolizidine alkaloid mutagenicity Materials and Methods ===================== Data ---- ### Mutagenicity training data An identical training dataset was used for all models. The training dataset was compiled from the following sources: - Kazius/Bursi Dataset (4337 compounds, @Kazius2005): - Hansen Dataset (6513 compounds, @Hansen2009): - EFSA Dataset (695 compounds @EFSA2016): Mutagenicity classifications from Kazius and Hansen datasets were used without further processing. To achieve consistency with these datasets, EFSA compounds were classified as mutagenic, if at least one positive result was found for TA98 or T100 Salmonella strains. Dataset merges were based on unique SMILES (*Simplified Molecular Input Line Entry Specification*, @Weininger1989) strings of the compound structures. Duplicated experimental data with the same outcome was merged into a single value, because it is likely that it originated from the same experiment. Contradictory results were kept as multiple measurements in the database. The combined training dataset contains {{cv.n_uniq}} unique structures and {{cv.n}} individual measurements. Source code for all data download, extraction and merge operations is publicly available from the git repository under a GPL3 License. The new combined dataset can be found at . ### Pyrrolizidine alkaloid (PA) dataset The pyrrolizidine alkaloid dataset was created from five independent, necine base substructure searches in PubChem (https://pubchem.ncbi.nlm.nih.gov/) and compared to the PAs listed in the EFSA publication @EFSA2011 and the book by Mattocks @Mattocks1986, to ensure, that all major PAs were included. PAs mentioned in these publications which were not found in the downloaded substances were searched individually in PubChem and, if available, downloaded separately. Non-PA substances, duplicates, and isomers were removed from the files, but artificial PAs, even if unlikely to occur in nature, were kept. The resulting PA dataset comprised a total of {{pa.n}} different PAs. The PAs in the dataset were classified according to structural features. A total of 9 different structural features were assigned to the necine base, modifications of the necine base and to the necic acid: For the necine base, the following structural features were chosen: - Retronecine-type (1,2-unstaturated necine base, {{pa.groups.Retronecine.n}} compounds) - Otonecine-type (1,2-unstaturated necine base, {{pa.groups.Otonecine.n}} compounds) - Platynecine-type (1,2-saturated necine base, {{pa.groups.Platynecine.n}} compounds) For the modifications of the necine base, the following structural features were chosen: - N-oxide-type ({{pa.groups.N_oxide.n}} compounds) - Tertiary-type (PAs which were neither from the N-oxide- nor DHP-type, {{pa.groups.Tertiary_PA.n}} compounds) - Dehydropyrrolizidine-type (pyrrolic ester, {{pa.groups.Dehydropyrrolizidine.n}} compounds) For the necic acid, the following structural features were chosen: - Monoester-type ({{pa.groups.Monoester.n}} compounds) - Open-ring diester-type ({{pa.groups.Diester.n}} compounds) - Macrocyclic diester-type ({{pa.groups.Macrocyclic_diester.n}} compounds) The compilation of the PA dataset is described in detail in @Schoening2017. Descriptors ----------- ### MolPrint2D (*MP2D*) fingerprints MolPrint2D fingerprints (@OBoyle2011a) use atom environments as molecular representation. They determine for each atom in a molecule, the atom types of its connected atoms to represent their chemical environment. This resembles basically the chemical concept of functional groups. In contrast to predefined lists of fragments (e.g. FP3, FP4 or MACCs fingerprints) or descriptors (e.g CDK) they are generated dynamically from chemical structures. This has the advantage that they can capture unknown substructures of toxicological relevance that are not included in other descriptors. In addition they allow the efficient calculation of chemical similarities (e.g. Tanimoto indices) with simple set operations. MolPrint2D fingerprints were calculated with the OpenBabel cheminformatics library (@OBoyle2011a). They can be obtained from the following locations: *Training data:* - sparse representation () - descriptor matrix () *Pyrrolizidine alkaloids:* - sparse representation () - descriptor matrix () #### Chemistry Development Kit (*CDK*) descriptors Molecular 1D and 2D descriptors were calculated with the PaDEL-Descriptors program ( version 2.21, @Yap2011). PaDEL uses the Chemistry Development Kit (*CDK*, ) library for descriptor calculations. As the training dataset contained {{cv.n_uniq}} instances, it was decided to delete instances with missing values during data pre-processing. Furthermore, substances with equivocal outcome were removed. The final training dataset contained {{cv.cdk.n_descriptors}} descriptors for {{cv.cdk.n_compounds}} compounds. CDK training data can be obtained from . The same procedure was applied for the pyrrolizidine dataset yielding {{pa.cdk.n_descriptors}} descriptors for {{pa.cdk.n_compounds}} compounds. CDK features for pyrrolizidine alkaloids are available at . Algorithms ---------- ### `lazar` `lazar` (*lazy structure activity relationships*) is a modular framework for read-across model development and validation. It follows the following basic workflow: For a given chemical structure `lazar`: - searches in a database for similar structures (neighbours) with experimental data, - builds a local QSAR model with these neighbours and - uses this model to predict the unknown activity of the query compound. This procedure resembles an automated version of read across predictions in toxicology, in machine learning terms it would be classified as a k-nearest-neighbour algorithm. Apart from this basic workflow, `lazar` is completely modular and allows the researcher to use arbitrary algorithms for similarity searches and local QSAR (*Quantitative structure--activity relationship*) modelling. Algorithms used within this study are described in the following sections. #### Feature preprocessing MolPrint2D features were used without preprocessing. Near zero variance and strongly correlated CDK descriptors were removed and the remaining descriptor values were centered and scaled. Preprocessing was performed with the R `caret` preProcess function using the methods "nzv","corr","center" and "scale" with default settings. #### Neighbour identification Utilizing this modularity, similarity calculations were based both on MolPrint2D fingerprints and on CDK descriptors. For MolPrint2D fingerprints chemical similarity between two compounds $a$ and $b$ is expressed as the proportion between atom environments common in both structures $A \cap B$ and the total number of atom environments $A \cup B$ (Jaccard/Tanimoto index). $$sim = \frac{\lvert A\ \cap B \rvert}{\lvert A\ \cup B \rvert}$$ For CDK descriptors chemical similarity between two compounds $a$ and $b$ is expressed as the cosine similarity between the descriptor vectors $A$ for $a$ and $B$ for $b$. $$sim = \frac{A \cdot B}{\lvert A \rvert \lvert B \rvert}$$ Threshold selection is a trade-off between prediction accuracy (high threshold) and the number of predictable compounds (low threshold). As it is in many practical cases desirable to make predictions even in the absence of closely related neighbours, we follow a tiered approach: - First a similarity threshold of 0.5 (MP2D/Tanimoto) or 0.9 (CDK/Cosine) is used to collect neighbours, to create a local QSAR model and to make a prediction for the query compound. This are predictions with *high confidence*. - If any of these steps fails, the procedure is repeated with a similarity threshold of 0.2 (MP2D/Tanimoto) or 0.7 (CDK/Cosine) and the prediction is flagged with a warning that it might be out of the applicability domain of the training data (*low confidence*). - These Similarity thresholds are the default values chosen by software developers and remained unchanged during the course of these experiments. Compounds with the same structure as the query structure are automatically eliminated from neighbours to obtain unbiased predictions in the presence of duplicates. #### Local QSAR models and predictions Only similar compounds (neighbours) above the threshold are used for local QSAR models. In this investigation, we are using a weighted majority vote from the neighbour's experimental data for mutagenicity classifications. Probabilities for both classes (mutagenic/non-mutagenic) are calculated according to the following formula and the class with the higher probability is used as prediction outcome. $$p_{c} = \ \frac{\sum_{}^{}\text{sim}_{n,c}}{\sum_{}^{}\text{sim}_{n}}$$ $p_{c}$ Probability of class c (e.g. mutagenic or non-mutagenic)\ $\sum_{}^{}\text{sim}_{n,c}$ Sum of similarities of neighbours with class c\ $\sum_{}^{}\text{sim}_{n}$ Sum of all neighbours #### Applicability domain The applicability domain (AD) of `lazar` models is determined by the structural diversity of the training data. If no similar compounds are found in the training data no predictions will be generated. Warnings are issued if the similarity threshold had to be lowered from 0.5 to 0.2 in order to enable predictions. Predictions without warnings can be considered as close to the applicability domain (*high confidence*) and predictions with warnings as more distant from the applicability domain (*low confidence*). Quantitative applicability domain information can be obtained from the similarities of individual neighbours. #### Validation 10-fold cross validation was performed for model evaluation. #### Pyrrolizidine alkaloid predictions For the prediction of pyrrolizidine alkaloids models were generated with the MP2D and CDK training datasets. The complete feature set was used for MP2D predictions, for CDK predictions the intersection between training and pyrrolizidine alkaloid features was used. #### Availability - Source code for this manuscript (GPL3): - Crossvalidation experiments (GPL3): - Pyrrolizidine alkaloid predictions (GPL3): - Public web interface: ### Tensorflow models #### Feature Preprocessing For preprocessing of the CDK features we used a quantile transformation to a uniform distribution. MP2D features were not preprocessed. #### Random forests (*RF*) For the random forest classifier we used the parameters n_estimators=1000and max_leaf_nodes=200. For the other parameters we used the scikit-learn default values. #### Logistic regression (SGD) (*LR-sgd*) For the logistic regression we used an ensemble of five trained models. For each model we used a batch size of 64 and trained for 50 epoch. As an optimizer ADAM was chosen. For the other parameters we used the tensorflow default values. #### Logistic regression (scikit) (*LR-scikit*) For the logistic regression we used as parameters the scikit-learn default values. #### Neural Nets (*NN*) For the neural network we used an ensemble of five trained models. For each model we used a batch size of 64 and trained for 50 epoch. As an optimizer ADAM was chosen. The neural network had 4 hidden layers with 64 nodes each and a ReLu activation function. For the other parameters we used the tensorflow default values. #### Support vector machines (*SVM*) We used the SVM implemented in scikit-learn. We used the parameters kernel='rbf', gamma='scale'. For the other parameters we used the scikit-learn default values. #### Validation 10-fold cross-validation was used for all Tensorflow models. #### Pyrrolizidine alkaloid predictions For the prediction of pyrrolizidine alkaloids we trained the model described above on the training data. For training and prediction only the features were used that were in the intersection of features from the training data and the pyrrolizidine alkaloids. #### Availability Jupyter notebooks for these experiments can be found at the following locations *Crossvalidation:* - MolPrint2D fingerprints: - CDK descriptors: *Pyrrolizidine alkaloids:* - MolPrint2D fingerprints: - CDK descriptors: - CDK desc Results ======= 10-fold crossvalidations ------------------------ Crossvalidation results are summarized in the following tables: @tbl:cv-mp2d shows results with MolPrint2D descriptors and @tbl:cv-cdk with CDK descriptors. | | lazar-HC | lazar-all | RF | LR-sgd | LR-scikit | NN | SVM | |:-|----------|-----------|----|--------|-----------|----|-----| Accuracy | {{cv.mp2d_lazar_high_confidence.acc_perc}} | {{cv.mp2d_lazar_all.acc_perc}} | {{cv.mp2d_rf.acc_perc}} | {{cv.mp2d_lr.acc_perc}} | {{cv.mp2d_lr2.acc_perc}} | {{cv.mp2d_nn.acc_perc}} | {{cv.mp2d_svm.acc_perc}} | True positive rate | {{cv.mp2d_lazar_high_confidence.tpr_perc}} | {{cv.mp2d_lazar_all.tpr_perc}} | {{cv.mp2d_rf.tpr_perc}} | {{cv.mp2d_lr.tpr_perc}} | {{cv.mp2d_lr2.tpr_perc}} | {{cv.mp2d_nn.tpr_perc}} | {{cv.mp2d_svm.tpr_perc}} | True negative rate | {{cv.mp2d_lazar_high_confidence.tnr_perc}} | {{cv.mp2d_lazar_all.tnr_perc}} | {{cv.mp2d_rf.tnr_perc}} | {{cv.mp2d_lr.tnr_perc}} | {{cv.mp2d_lr2.tnr_perc}} | {{cv.mp2d_nn.tnr_perc}} | {{cv.mp2d_svm.tnr_perc}} | Positive predictive value | {{cv.mp2d_lazar_high_confidence.ppv_perc}} | {{cv.mp2d_lazar_all.ppv_perc}} | {{cv.mp2d_rf.ppv_perc}} | {{cv.mp2d_lr.ppv_perc}} | {{cv.mp2d_lr2.ppv_perc}} | {{cv.mp2d_nn.ppv_perc}} | {{cv.mp2d_svm.ppv_perc}} | Negative predictive value | {{cv.mp2d_lazar_high_confidence.npv_perc}} | {{cv.mp2d_lazar_all.npv_perc}} | {{cv.mp2d_rf.npv_perc}} | {{cv.mp2d_lr.npv_perc}} | {{cv.mp2d_lr2.npv_perc}} | {{cv.mp2d_nn.npv_perc}} | {{cv.mp2d_svm.npv_perc}} | Nr. predictions | {{cv.mp2d_lazar_high_confidence.n}} | {{cv.mp2d_lazar_all.n}} | {{cv.mp2d_rf.n}} | {{cv.mp2d_lr.n}} | {{cv.mp2d_lr2.n}} | {{cv.mp2d_nn.n}} | {{cv.mp2d_svm.n}} | : Summary of crossvalidation results with MolPrint2D descriptors (lazar-HC: lazar with high confidence, lazar-all: all lazar predictions, RF: random forests, LR-sgd: logistic regression (stochastic gradient descent), LR-scikit: logistic regression (scikit), NN: neural networks, SVM: support vector machines) {#tbl:cv-mp2d} | | lazar-HC | lazar-all | RF | LR-sgd | LR-scikit | NN | SVM | |:-|----------|-----------|----|--------|-----------|----|-----| Accuracy | {{cv.cdk_lazar_high_confidence.acc_perc}} | {{cv.cdk_lazar_all.acc_perc}} | {{cv.cdk_rf.acc_perc}} | {{cv.cdk_lr.acc_perc}} | {{cv.cdk_lr2.acc_perc}} | {{cv.cdk_nn.acc_perc}} | {{cv.cdk_svm.acc_perc}} | True positive rate | {{cv.cdk_lazar_high_confidence.tpr_perc}} | {{cv.cdk_lazar_all.tpr_perc}} | {{cv.cdk_rf.tpr_perc}} | {{cv.cdk_lr.tpr_perc}} | {{cv.cdk_lr2.tpr_perc}} | {{cv.cdk_nn.tpr_perc}} | {{cv.cdk_svm.tpr_perc}} | True negative rate | {{cv.cdk_lazar_high_confidence.tnr_perc}} | {{cv.cdk_lazar_all.tnr_perc}} | {{cv.cdk_rf.tnr_perc}} | {{cv.cdk_lr.tnr_perc}} | {{cv.cdk_lr2.tnr_perc}} | {{cv.cdk_nn.tnr_perc}} | {{cv.cdk_svm.tnr_perc}} | Positive predictive value | {{cv.cdk_lazar_high_confidence.ppv_perc}} | {{cv.cdk_lazar_all.ppv_perc}} | {{cv.cdk_rf.ppv_perc}} | {{cv.cdk_lr.ppv_perc}} | {{cv.cdk_lr2.ppv_perc}} | {{cv.cdk_nn.ppv_perc}} | {{cv.cdk_svm.ppv_perc}} | Negative predictive value | {{cv.cdk_lazar_high_confidence.npv_perc}} | {{cv.cdk_lazar_all.npv_perc}} | {{cv.cdk_rf.npv_perc}} | {{cv.cdk_lr.npv_perc}} | {{cv.cdk_lr2.npv_perc}} | {{cv.cdk_nn.npv_perc}} | {{cv.cdk_svm.npv_perc}} | Nr. predictions | {{cv.cdk_lazar_high_confidence.n}} | {{cv.cdk_lazar_all.n}} | {{cv.cdk_rf.n}} | {{cv.cdk_lr.n}} | {{cv.cdk_lr2.n}} | {{cv.cdk_nn.n}} | {{cv.cdk_svm.n}} | : Summary of crossvalidation results with CDK descriptors (lazar-HC: lazar with high confidence, lazar-all: all lazar predictions, RF: random forests, LR-sgd: logistic regression (stochastic gradient descent), LR-scikit: logistic regression (scikit), NN: neural networks, SVM: support vector machines) {#tbl:cv-cdk} @fig:roc depicts the position of all crossvalidation results in receiver operating characteristic (ROC) space. ![ROC plot of crossvalidation results (lazar-HC: lazar with high confidence, lazar-all: all lazar predictions, RF: random forests, LR-sgd: logistic regression (stochastic gradient descent), LR-scikit: logistic regression (scikit), NN: neural networks, SVM: support vector machines).](figures/roc.png){#fig:roc} Confusion matrices for all models are available from the git repository https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/confusion-matrices/, individual predictions can be found in https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/predictions/. All investigated algorithm/descriptor combinations give accuracies between (80 and 85%) which is equivalent to the experimental variability of the *Salmonella typhimurium* mutagenicity bioassay (80-85%, @Benigni1988). Sensitivities and specificities are balanced in all of these models. Pyrrolizidine alkaloid mutagenicity predictions ----------------------------------------------- Mutagenicity predictions of {{pa.n}} pyrrolizidine alkaloids (PAs) from all investigated models can be downloaded from . A visual representation of all PA predictions can be found at . @fig:pa-groups displays the proportion of positive mutagenicity predictions from all models for the different pyrrolizidine alkaloid groups. Tensorflow models predicted all {{pa.n}} pyrrolizidine alkaloids, `lazar` MP2D models predicted {{pa.mp2d_lazar_all.n}} compounds ({{pa.mp2d_lazar_high_confidence.n}} with high confidence) and `lazar` CDK models {{pa.cdk_lazar_all.n}} compounds ({{pa.cdk_lazar_high_confidence.n}} with high confidence). ![Summary of pyrrolizidine alkaloid predictions](figures/pa-groups.png){#fig:pa-groups} For the visualisation of the position of pyrrolizidine alkaloids in respect to the training data set we have applied t-distributed stochastic neighbor embedding (t-SNE, @Maaten2008) for MolPrint2D and CDK descriptors. t-SNE maps each high-dimensional object (chemical) to a two-dimensional point, maintaining the high-dimensional distances of the objects. Similar objects are represented by nearby points and dissimilar objects are represented by distant points. t-SNE coordinates were calculated with the R `Rtsne` package using the default settings (perplexity = 30, theta = 0.5, max_iter = 1000). @fig:tsne-mp2d shows the t-SNE of pyrrolizidine alkaloids (PA) and the mutagenicity training data in MP2D space (Tanimoto/Jaccard similarity), which resembles basically the structural diversity of the investigated compounds. ![t-SNE visualisation of mutagenicity training data and pyrrolizidine alkaloids (PA) in MP2D space](figures/tsne-mp2d-mutagenicity.png){#fig:tsne-mp2d} @fig:tsne-cdk shows the t-SNE of pyrrolizidine alkaloids (PA) and the mutagenicity training data in CDK space (Euclidean similarity), which resembles basically the physical-chemical properties of the investigated compounds. ![t-SNE visualisation of mutagenicity training data and pyrrolizidine alkaloids (PA) in CDK space](figures/tsne-cdk-mutagenicity.png){#fig:tsne-cdk} @fig:tsne-mp2d-rf and @fig:tsne-cdk-lazar-all depict two example pyrrolizidine alkaloid mutagenicity predictions in the context of training data. t-SNE visualisations of all investigated models can be downloaded from . ![t-SNE visualisation of MP2D random forest predictions](figures/tsne-mp2d-rf-classifications.png){#fig:tsne-mp2d-rf} ![t-SNE visualisation of all CDK lazar predictions](figures/tsne-cdk-lazar-all-classifications.png){#fig:tsne-cdk-lazar-all} Discussion ========== Data ---- A new training dataset for *Salmonella* mutagenicity was created from three different sources (@Kazius2005, @Hansen2009, @EFSA2016). It contains {{cv.n_uniq}} unique chemical structures, which is according to our knowledge the largest public mutagenicity dataset presently available. The new training data can be downloaded from . Algorithms ---------- `lazar` is formally a *k-nearest-neighbor* algorithm that searches for similar structures for a given compound and calculates the prediction based on the experimental data for these structures. The QSAR literature calls such models frequently *local models*, because models are generated specifically for each query compound. The investigated tensorflow models are in contrast *global models*, i.e. a single model is used to make predictions for all compounds. It has been postulated in the past, that local models are more accurate, because they can account better for mechanisms, that affect only a subset of the training data. @tbl:cv-mp2d, @tbl:cv-cdk and @fig:roc show that the crossvalidation accuracies of all models are comparable to the experimental variability of the *Salmonella typhimurium* mutagenicity bioassay (80-85% according to @Benigni1988). All of these models have balanced sensitivity (true position rate) and specificity (true negative rate) and provide highly significant concordance with experimental data (as determined by McNemar's Test). This is a clear indication that *in-silico* predictions can be as reliable as the bioassays. Given that the variability of experimental data is similar to model variability it is impossible to decide which model gives the most accurate predictions, as models with higher accuracies might just approximate experimental errors better than more robust models. Our results do not support the assumption that local models are superior to global models for classification purposes. For regression models (lowest observed effect level) we have found however that local models may outperform global models (@Helma2018) with accuracies similar to experimental variability. As all investigated algorithms give similar accuracies the selection will depend more on practical considerations than on intrinsic properties. Nearest neighbor algorithms like `lazar` have the practical advantage that the rationales for individual predictions can be presented in a straightforward manner that is understandable without a background in statistics or machine learning (@fig:lazar). This allows a critical examination of individual predictions and prevents blind trust in models that are intransparent to users with a toxicological background. ![Lazar screenshot of 12,21-Dihydroxy-4-methyl-4,8-secosenecinonan-8,11,16-trione mutagenicity prediction](figures/lazar-screenshot.png){#fig:lazar} Descriptors ----------- This study uses two types of descriptors for the characterisation of chemical structures: *MolPrint2D* fingerprints (MP2D, @Bender2004) use atom environments (i.e. connected atom types for all atoms in a molecule) as molecular representation, which resembles basically the chemical concept of functional groups. MP2D descriptors are used to determine chemical similarities in the default `lazar` settings, and previous experiments have shown, that they give more accurate results than predefined fingerprints (e.g. MACCS, FP2-4). *Chemistry Development Kit* (CDK, @Willighagen2017) descriptors were calculated with the PaDEL graphical interface (@Yap2011). They include 1D and 2D topological descriptors as well as physical-chemical properties. All investigated algorithms obtained models within the experimental variability for both types of descriptors (@tbl:cv-mp2d, @tbl:cv-cdk, @fig:roc). Given that similar predictive accuracies are obtainable from both types of descriptors the choice depends once more on practical considerations: MolPrint2D fragments can be calculated very efficiently for every well defined chemical structure with OpenBabel (@OBoyle2011a). CDK descriptor calculations are in contrast much more resource intensive and may fail for a significant number of compounds ({{cv.cdk.n_failed}} from {{cv.n_uniq}}). MolPrint2D fragments are generated dynamically from chemical structures and can be used to determine if a compound contains structural features that are absent in training data. This feature can be used to determine applicability domains. CDK descriptors contain in contrast a predefined set of descriptors with unknown toxicological relevance. MolPrint2D fingerprints can be represented very efficiently as sets of features that are present in a given compound which makes similarity calculations very efficient. Due to the large number of substructures present in training compounds, they lead however to large and sparsely populated datasets, if they have to be expanded to a binary matrix (e.g. as input for tensorflow models). CDK descriptors contain in contrast in every case matrices with {{cv.cdk.n_descriptors}} columns which can cause substantial computational overhead. Pyrrolizidine alkaloid mutagenicity predictions ----------------------------------------------- @fig:pa-groups shows a clear differentiation between the different pyrrolizidine alkaloid groups. The largest proportion of mutagenic predictions was observed for Otonecines {{pa.groups.Otonecine.mut_perc}}% ({{pa.groups.Otonecine.mut}}/{{pa.groups.Otonecine.n_pred}}), the lowest for Monoesters {{pa.groups.Monoester.mut_perc}}% ({{pa.groups.Monoester.mut}}/{{pa.groups.Monoester.n_pred}}) and N-Oxides {{pa.groups.N_oxide.mut_perc}}% ({{pa.groups.N_oxide.mut}}/{{pa.groups.N_oxide.n_pred}}). Although most of the models show similar accuracies, sensitivities and specificities in crossvalidation experiments some of the models (MPD-RF, CDK-RF and CDK-SVM) predict a lower number of mutagens ({{pa.cdk_rf.mut_perc}}-{{pa.mp2d_rf.mut_perc}}%) than the majority of the models ({{pa.mp2d_svm.mut_perc}}-{{pa.mp2d_lazar_high_confidence.mut_perc}}% (@fig:pa-groups). lazar-CDK on the other hand predicts the largest number of mutagens for all groups with exception of Otonecines. These differences between predictions from different algorithms and descriptors were not expected based on crossvalidation results. In order to investigate, if any of the investigated models show systematic errors in the vicinity of pyrrolizidine-alkaloids we have performed a detailled t-SNE analysis of all models (see @fig:tsne-mp2d-rf and @fig:tsne-cdk-lazar-all for two examples, all visualisations can be found at . Nevertheless none of the models showed obvious deviations from their expected behaviour, so the reason for the disagreement between some of the models remains unclear at the moment. It is however perfectly possible that some systematic errors are covered up by converting high dimensional spaces to two coordinates and are thus invisible in t-SNE visualisations. Conclusions =========== A new public *Salmonella* mutagenicity training dataset with 8309 compounds was created and used it to train `lazar` and Tensorflow models with MolPrint2D and CDK descriptors. References ==========