Machine learning based language models on nucleotide sequences of human genes

Abstract

The use of computers for different fields of science has provided tremendous benefits. This phenomenon is expected to be more common as the speed of computers and the amount of data available for different kinds of scientific problems increase. This study focuses on genomics, one of the most exciting areas of science. We have applied several techniques to obtain a model for nucleotide sequences of genes that are found in human beings so that the model can learn the general pattern in these nucleotide sequences and predict how likely it is that an unseen sequence is a gene that belongs to human beings. They can even generate new nucleotide sequences. All of the methods used are examples of machine learning, where the programs are designed to learn from data for a specific task, rather than explicitly programming what to do at each step. Traditional approaches such as N-grams and more recent deep learning-based techniques such as recurrent neural networks and transformer architecture language models are used. In addition to the classical metrics, the strength of the methods is measured using a real-world task from the field of genomics. Finally, the results show an interesting comparison of how all these models perform on a task that is inherently different from classical natural language processing tasks, and how sometimes simple models like N-grams can be as good as, if not better than, more sophisticated techniques such as transformer for solving certain types of problems. Furthermore, the significance of evaluating obtained models on real-life tasks is seen because the transformer model was superior to the N- gram model according to perplexity, although it performed worse on real-world task.