MalTree: Tracing Malware Evolution using Embeddings at Scale

Malware detection remains largely reactive: machine learning models trained on known samples degrade as threats evolve. Understanding evolutionary relationships among malware families can inform proactive defense, but traditional reverse engineering can take months to years to uncover such lineage relationships. We propose MalTree, a framework that applies bioinformatics-inspired phylogenetic techniques (UPGMA and Neighbor-Joining) at scale to model malware evolution automatically using structural, behavioral, and image-based features. We introduce temporal validation using VirusTotal timestamps to assess whether inferred trees reflect actual evolutionary order. MalTree achieves 87% temporal consistency, indicating that inferred evolutionary relationships closely align with real-world emergence timelines. Our analysis shows that some families mutate over 10 times faster than others, suggesting that detection strategies should be tailored to family-specific evolutionary tempos. Case studies, including the Mirai botnet, confirm that inferred relationships from our phylogenetic tree align with documented threat intelligence. Our framework provides a foundation for shifting malware analysis from sample-by-sample classification toward lineage-aware evolutionary modeling.

(Problem) New malware appears every day, and the programs that detect it are usually one step behind. Security tools learn what past attacks looked like, but attackers constantly modify their attacks to slip past those defenses. Most attacks are not built from scratch. They copy from each other, evolving like organisms over generations. Yet defenders rarely look at malware this way. Figuring out which attacks descended from which used to require months of work by human experts. (Solution) We borrowed a tool from biology: the family tree. Biologists use these trees to trace how species evolved. We applied the same idea to over 100,000 pieces of malware. We combine machine learning with approaches of bio-informatics to build the family tree in an efficient manner. To check if our tree was meaningful, we compared it against the dates malware first appeared in the wild. Earlier programs really did sit closer to the roots. (Impact) Our tree recovers known malware family relations and reveals that some families evolve over ten times faster than others. These insights may help defenders move from reacting to individual attacks toward more predictive defenses.