Researchers have created custom malware samples in an effort to test the effecti

veness of some top advanced persistent threat (APT) attack detection appliances.
APT campaigns are increasingly common and since they usually rely on tools that
are not detected by regular antivirus products, many security companies have dev
eloped specialized solutions designed to identify and block such threats. The li
st of firms that offer such solutions includes Cisco, Damballa, Checkpoint, Fire
Eye, Fortinet, Palo Alto Networks, LastLine, Zscaler, Trend Micro and Websense.
Over the past years, independent testing firm NSS Labs has conducted several tes
ts comparing the top solutions, but in many cases the results have been controve
rsial. NSS Labs' testing methods have been criticized by some vendors whose prod
ucts did not perform as expected, including FireEye and Palo Alto Networks. Last
week, Miercom published the results of a test comparing APT detection solutions
from Zscaler and FireEye. In that particular test, Zscaler performed better, bu
t FireEye contested the accuracy of the results and testing methodology.
Researchers from MRG Effitas, a UK-based independent IT security research compan
y focusing on efficacy assessment and assurance services, and CrySyS Lab, the H
ungary-based organization that has been involved in the analysis of numerous APT
campaigns, have tested five "cutting-edge" solutions. The tested products are a
ll "well-established in the market," but they have not been named.
"[Our] goal was simply to implement some ideas we had for bypassing cutting-edge
APT attack detection tools without actually being detected, and to test if our
ideas really work in practice," researchers wrote in their report.
For their tests, researchers used four samples, which they created over a period
of two weeks without having access to any APT attack detection solutions. The s
amples were designed to incorporate functionality that is typical to remote acce
ss Trojans (RATs), including remote code execution, and file download/upload cap
abilities. No lateral movement was initiated in the tests, but real attacks were
simulated through command execution and file transfers.
The first test sample, which used a known Java exploit for delivery, was designe
d to mimic an attacker with limited knowledge and resources. Samples 2 and 3, wh
ich used a Java self-signed applet and a Visual Basic macro for delivery, simula
ted attackers with moderate knowledge and resources. The last sample, which the
researchers dubbed "Babo" (the Hungarian word for "hobbit"), was simple but stea
lthy.
The first two test samples were detected by all five APT attack detection produc
ts. The third sample was detected by only one product, while Babo bypassed all s
olutions. The authors of the report noted that even though the first two samples
were picked up by all solutions, some of them rated the threats as "low severit
y."
"The main message of this work is that novel anti-APT tools can be bypassed with
moderate effort. If we were able to develop samples that were not detected by t
hese tools without actually having access to any of the tested products during t
he development phase, then resourceful attackers who may be able to buy these pr
oducts will also be able to develop similar samples, or even better ones," the r
eport reads.
The researchers advise organizations that want to acquire APT detection solution
s to conduct proper testing, either in-house or outsourced, and not judge the pr
oducts based on the way they are marketed or their prices.
The complete report, titled "An independent test of APT attack detection applian
ces," is available in PDF format.