Assessing predictors for new post translational modification sites: a case study on hydroxylation

Abstract

Post-translational modification (PTM) sites have become popular for predictor development. However, with the exception of phosphorylation and a handful of other examples, PTMs suffer from a limited number of available training examples and their sparsity in protein sequences. Here, proline hydroxylation is taken as an example to compare different methods and evaluate their performance on new experimentally determined sites. As a proxy for an effective experimental design, predictors require both high specificity and sensitivity. However, the self-reported performance is often not indicative of prediction quality and detection of new sites is not guaranteed. We have benchmarked seven published hydroxylation site predictors on two newly constructed independent datasets. The self-reported performance widely overestimates the real accuracy measured on independent datasets. No predictor performs better than random on new examples, indicating the refined models are not sufficiently general to detect new sites. The number of false positives is high and precision low, in particular for non-collagen proteins whose motifs are not conserved. In short, existing predictors for hydroxylation sites do not appear to generalize to new data. Caution is advised when dealing with PTM predictors in the absence of independent evaluations, in particular for unique specific sites such as those involved in signalling. Author SummaryMachine learning methods are extensively used by biologists to design and interpret experiments. Predictors which take the only sequence as input are of particular interest due to the large amount of sequence data available and self-reported performance is often very high. In this work, we evaluated post-translational modification (PTM) predictors for hydroxylation sites and found that they perform no better than random, in strong contrast to performances reported in the original publications. PTMs are chemical amino acids alterations providing the cell with conditional mechanisms to fine tune protein function, thereby regulating complex biological processes such as signalling and cell cycle. Hydroxylation sites are a good PTM test case due to the availability of a range of predictors and an abundance of newly experimentally detected modification sites. Poor performances in our results highlight the overlooked problem of predicting PTMs when best practices are not followed and training data are likely incomplete. Experimentalists should be careful when using PTM predictors blindly and more independent assessments are needed to separate the wheat from the chaff in the field.