Malware Image Classifiers Report 97% to 99% Accuracy, but an Audit of 10 Benchmarks Found as Much as 50.6% of Test Images Had a Copy or Near-Copy in Training, and SOC Teams Should Ask What a Model Was Graded On

LONDON - Every security team that has sat through a machine learning pitch has heard a version of the same sentence: the model classifies malware with 99% accuracy. A paper posted to arXiv on Oct. 29 is a useful reminder that the number in that sentence describes a test set as much as it describes a model, and that the test set can contain the answers.

Stefan Larson and Kevin Leach of Vanderbilt University in Nashville examined one corner of the field, classifiers that render a program's raw bytes as a grayscale picture and then sort the picture into a malware family with the same kind of neural network used to recognize photographs. Papers on the standard benchmarks for this task routinely report accuracies of 97% to 99%, they write, and two recent reviews of the literature, one a survey of more than 100 papers, report the same range. The authors are explicit about what kind of paper this is: "This paper is a benchmark audit, and we do not propose a new model, training procedure or dataset."

What the audit found

The researchers audited 10 public benchmarks, from the 2011 Malimg set to MalNet-Image with more than 1.26 million images, for one thing: images in the test set that are identical or nearly identical to images in the training set. If a model is graded on pictures it has already seen, its accuracy measures memory, not judgment.

The standard defense against that problem is to remove any sample whose file hash has already been collected. The audit shows why that is not enough. "Deduplicating by file hash, which is the standard practice, does not prevent leakage, because distinct binaries render to identical images," the authors write. Two Android apps that differ only in a signing certificate and a version string have different hashes but identical code, and identical code produces an identical image. As they put it, "a hash of the distributed file does not identify the model's input."

The numbers are hard to wave away. Duplication rates across the 10 benchmarks ran from 0 to 35.7%. On the published split of MalNet-Tiny, whose creators had removed duplicate files by hash, 14.9% of the test images were pixel-identical to a training image, and hash-based cleanup would not have removed a single one. On the published split of MaleVis, 50.6% of test images had an exact or near-duplicate in training, a figure that applies, the authors note, to any paper that used that partition. MaleX, a set of more than 1.04 million Windows programs in which every SHA-256 hash appears once, still had pixel-identical copies for 13.3% of its images, and 17,317 of its images sat in identical groups that contained both malware and benign programs.

What happens to the accuracy

The team then rebuilt the splits so that no group of duplicates spanned training and testing, and retrained three standard image models on the seven benchmarks small enough to train repeatedly. Accuracy dropped by 3.7 to 10.5 percentage points on five of them; on the other two the drop was not significant. A balanced measure across malware families, macro-F1, fell by as much as 15 points. Test images that had a copy in training were classified correctly 94.1% to 100% of the time.

The audit is fair to the field. Malimg, the original benchmark, lost only 0.49 points, an amount within the margin of error. "We do not claim that malware-as-image accuracy is uniformly an artifact of duplicated data," the authors write. "Our claim is that inflation must be measured per benchmark, and that the image encoding is not a neutral re-representation of the binary." They also checked that their retrained models matched or beat the published results before deduplication, so the drops are measured from the accuracy the original papers report.

Two further results should interest anyone who runs detection, not just researchers. When the team split one benchmark, MOTIF, by the month each sample first appeared, training on older malware and testing on samples first seen from February 2020 to May 2021, leakage fell to 2.3% from 17.4% to 18.4% under random splits, and accuracy fell to 20.62% from 45.37%. The authors caution that the gap combines two effects, less leakage and later malware, but it is the closest thing in the paper to a model meeting new threats. And on the same MOTIF samples, a conventional gradient-boosting model using standard static file features scored 73.93% against 63.63% for the image model, roughly ten points better.

The rendering itself turned out to be a weakness. Changing only the width at which bytes are wrapped into rows, without changing a single byte, cut accuracy by 20.53 points on average on the Dumpware10 benchmark, with a worst case of 57.07%, and by 33.08 points on the team's renderings of MOTIF. If a model depends on that choice, the authors write, "then part of what it has learned is a property of the rendering and not of the malware." On Dumpware10, training one model on all four widths pooled reached 98.22% to 98.49% on every width, so the fix is cheap once someone thinks to test for it.

Why this belongs on a SOC's desk

Most SOC teams will never train a byteplot classifier. Many will buy, inherit or be asked to trust a model that sorts files, alerts or sessions, and will be shown an accuracy figure to justify it. The lesson of this audit is not limited to images. Malware families are built from shared code, so near-copies are the normal state of any malware corpus, and a model graded on a randomly split corpus is likely to be graded partly on samples it has effectively seen before. The paper also shows that a low score is no proof of a clean test: MOTIF-as-byteplot "is only 63% accurate and still drops 10 points under deduplication, so a low absolute accuracy does not indicate a clean benchmark."

The authors note that their temporal result agrees with the lower accuracy the TESSERACT study reported for Android malware detection under time-based splits. A SOC does not get to choose a random split of tomorrow's threats. The only accuracy figure that maps to operations is one measured on samples the model could not have seen, ideally ones that arrived after it was trained.

What to ask before trusting a malware model's accuracy

Detection engineering and SOC leaders evaluating any machine learning classifier, bought or built, should ask for:

• How duplicates were removed, and whether that was done on the model's actual input or only on file hashes.
• The share of test samples that have an exact or near-copy in training, reported next to the headline accuracy.
• Results on a split by time, trained on older samples and tested on newer ones, not only on a random split.
• A plain baseline on the same samples, such as gradient boosting over standard static features, so the benefit of the fancier model is shown rather than assumed.
• Evidence that the model holds up when an arbitrary preprocessing choice changes, such as the rendering width in this paper, and per-family results rather than one average.

The authors make the case for that last point themselves: reporting duplication on the model's input and running one baseline that does not use the image encoding are cheap, and "these two checks would have detected each of the effects we report." Their audit code, deduplicated splits and per-run results are public on GitHub. A vendor that cannot answer those questions has given a SOC a number, not a measurement.