We’ve seen a [few posts](https://www.reddit.com/r/cybersecurity/comments/p3mjt8/how_to_evaluate_edrmdrxdr_solutions/) recently on this sub asking about how to evaluate tools like EDR and XDR, so we wanted to give some advice on how to test security software.
As security professionals tasked with touching so many technologies, we get a chance to test and see what endpoint tools detect or prevent. In our experience, some of them are sadly far behind the curve when it comes to providing actionable alerts, detection depth, or simply prevention effectiveness.
Testing also helps confirm if the endpoint tool has some EDR features that look at attacker behaviors like process activity, network connections and registry content rather than just raw file inspection.
To test antivirus and EDR tools, a good starting point is to see if the tooling can at least compete with a default Windows 10 install using Windows Defender with Real-Time Protection, as this is installed and free on all Windows systems.
The main goal of the testing is to push endpoint software to see if it can detect any of these activities and allow us to write meaningful detections that can prevent a breach or ransomware incident.
For complete testing, the environment requires three hosts, a threat actor system, an initial user endpoint, and a domain controller server. If you only want to emulate phase one tests, the requirements are just a threat actor system and the initial user endpoint.
## Initial Configuration Requirements
For this test regime, we will be using the [Metasploit](https://www.blumira.com/glossary/metasploit/) Framework, a tried and true adversary and red team platform still in use both by penetration testers and real-world attackers. We will be doing some initial configuration that will be a little bit more than default configuration to make sure that there is at least a little bit of a bar for the tool to be tested to have to pass. But we will refrain from more advanced techniques as our goal is to be detected, if possible.
For initial setup, you’ll need to get a host running Kali. This can be hosted in your local lab’s LAN or created in a cloud environment like AWS.
* Kali install docs [here](https://www.kali.org/docs/installation/)
* AWS Market Kali Image [here](https://aws.amazon.com/marketplace/pp/B01M26MMTT)
After getting Kali installed or started and connecting either via SSH or the GUI (depending on your installation environment), start the [Metasploit](https://www.blumira.com/glossary/metasploit/) console and run some collection tasks to change the SSL certificate used by the framework.
See step-by-step instructions [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Setup.md).
For the targets, set up a server that you’ll want to make a domain controller and an endpoint to join to the test domain.
Once you’ve configured the hosts, it’s time to generate the payloads for testing. This can better help judge how an endpoint tool handles unknown malware. Protection against old threats, while helpful, is less actionable. We want to generate alerts that require security intervention.
We will be giving a handicap and generating several payloads at various iterations of encoding complexity using MSFvenom and the [shikata ga nai](https://www.fireeye.com/blog/threat-research/2019/10/shikata-ga-nai-encoder-still-going-strong.html) encoder. This will make the payloads unique to our test. The shikata ga nai encoder used to be a very effective evasion method, but the encoding method should be readily detected by most antivirus programs today.
See the payload generation steps [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Setup.md#payload-generation).
## Phase One
This first phase of testing helps to establish a baseline of how a particular endpoint tool handles common attack tools and methodologies.
As such, it’s important to provide a lot of handicaps to the test threat actor — things like local admin privileges on the initial host and domain admin later on for lateral movement. You want to apply an “assumed breach” methodology rather than test every minute requirement of an attack chain. Should a tool prove proficient enough, you can return with a finer tooth comb to look at bypasses and more in-depth detection opportunities.
## Initial Access
The first test is meant to simulate a simple phishing link that directs a user to a poorly-configured attacker stager website. We are looking to see if the tool either inspects traffic to the host or hooks into the browser to block malicious files.
Step-by-step instructions [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Phase-1.md#payload-delivery).
For our “assume breach” methodology, in this example we’d turn off Microsoft Defender’s Real-Time Protection to allow the rest of the testing to continue.
For the next test, we’ll need to execute the downloaded files, At this point you can try both with all detections re-enabled or the blocking controls disabled if it prevents further testing. (But mark that as a good outcome.)
Before you start the execution, start the meterpreter listener on our Kali host. Those instructions can be found [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Phase-1.md#execution).
After executing the files, record what, if any, alerting or logs that are generated.
After execution, most threat actors will run a variety of commands to discover information about the local system and connected domains so next we’ll emulate that and look for any alerting or logs that the endpoint technology generates.
The step-by-step instructions can be found [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Phase-1.md#discovery).
A Windows administrator may notice that all the commands run here are built in Windows utilities that get used by normal users and system administrators. While that’s true, they are favored by threat actors for that exact reason, but a modern requirement is to be able to trigger on suspicious invocation of these utilities that fall out of baseline usage.
## Local Privilege Escalation
The following phase in our intrusion chain is elevation of privilege that allows the threat actor to collect sensitive data further down the attack path. Here we’re looking to see if the endpoint technology can detect the common Named Pipe impersonation and process injection.
The step-by-step instructions can be found [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Phase-1.md#local-privilege-escalation).
## Credential Access
For the final part of the first phase we will dump credentials using the meterpreter shell. This may be difficult for most endpoint tools to determine, as this will all be occurring in memory, but is a common step in an intrusion.
You can find the step-by-step guide [here](https://github.com/blumirabrian/endpoint-detection-methology/blob/main/msf/MSF-Phase-1.md#credential-access).
This completes the first round of testing localized to a single endpoint. Later on we’ll post instructions to test the next phase, where we’ll begin lateral movement and look for taking actions against a domain controller.
*This was originally posted* [*on Blumira’s blog*](https://www.blumira.com/test-antivirus-edr-software/)*.*