A study on malware detection and classification using the analysis of API calls sequences through shallow learning and recurrent neural networks

Malware detection and classification is a critical issue in cybersecurity. Systems acting through signatures suffer the problem of not being able to detect attacks via zero-day malware. Among the approaches that can detect unknown attacks are the possibilities offered by analyzing the sequence of API calls performed by the executable. Such information can be extracted through static and dynamic analysis methods in a sandbox environment. This work proposes an analysis of different techniques to detect malware and subsequently classify them by identifying the family of belonging. Machine Learning algorithms based on trees are compared with Deep Learning algorithms based on Recurrent Neural Networks. The results obtained lead to choosing an algorithm based on RNNs for malware detection and an algorithm based on trees for malware classification.