National Collegiate Cyber Defense Competition Writeup

We are Volitech, an advanced biotech company with heavy emphasis on R&D that serves many customers with high-tech bio research and services, from prosthetics to health management. We have recently been hacked by a notorious hacker group, and have fired all of our administration team. You are the new administrative team, you shall maintain our systems, defend them against future hackers, and investigate the old hack.

That sums up the scenario preceding the 2018 National CCDC event, taking place Apr 13 to 15th in Walt Disney Convention Center of Orlando, FL. 10 Blue Teams, each consisting of 8 students, one from each region of the United States, each winning their Regional CCDC event, have come here to compete against each other, and against the notorious Red Team, to win the competition. 6 of the teams have been here in previous competitions, and most of them have had national championships. 4 teams are newcomers to the national theme, but all of them have participated in CCDC in previous years. University of Virginia is the only new team, formed only 3 months prior to the competition, beating the 2017 national champions UMBC in their Mid-Atlantic regional event.

The Competition

The competition was held over two days, each day from 10 AM to 6:15 PM, without a break, the blue teams would defend their network against the red team, while performing tasks handed to them by the organization, called injects. Injects could vary from resetting a password, to setting up a whole new box and providing data analysis.

The red team, consisting of about 30 professional hackers, would do everything in their power to break into the blue team’s network, steal user data, damage the computers, and wreak havoc. 3 red teamers were exclusively assigned to each blue team, and the rest of the red teamers would help them find vulnerabilities, write exploits, and get into the machines.

Teams were scored based on delivering the injects in a timely manner, and more importantly on service uptime. Automated scripts as well as the orange team (Volitech employees) would use the services and for every check (which is done about every 2 minutes) that a service passes, provide 1 point to the blue team. Each blue team had about 30 services, 11 of which were critical (scored) services, running on about 20 servers.

For every 6 consecutive checks that failed on a service, the blue team would receive an SLA violation, resulting in -20 points. So the blue team had every intention to bring the services up prior to 12 minutes of downtime which would result in violations.

None of this information was known by either the red team or the blue team, all of this was communicated to the blue team immediately before the start of the competition, through a handbook that included the network architecture, the list of the servers, and the list of critical services (and not all of the services or where the user data resides).

Naturally, both red and blue teams spent a good chunk of the first day of the competition mapping the networks and finding the services, the data, and other important parts, before attacking/defending them. Each blue team had 2 ESXis, one locally on a laptop, and one remotely on a cloud. Most of the scored services were on these hypervisors, as you can see in the figure above. Each blue team also had 5 workstation laptops, which are in scope for the red team, and are actually delicious targets. They also had 4 laptops that were actually servers, serving the Active Directory/DHCP, company email, company website and the company DNS. Each blue team also had access to a switch, a firewall, a FireEye box, and a VOIP phone, all of which were also in scope for the red team.

The UVa team quickly recognized that the single points of failure on the network are ESXis and the switch, and if the red team gains access to any of them, they are toast. They also realized that the workstations are a good candidate for the red team to eavesdrop and keylog on, and gain full access to rest of the services, so they isolated and hardened them first.

Another critical point was that the ESXis allow snapshotting and hot swapping, and the local ESXi can be recovered if broken into (with physical access), whereas the remote ESXi would be really sensitive to compromise. The laptop servers however, were not easily restorable, so physical backups had to be taken to restore them later if needed.

Each blue team had access to a scoreboard portal which would show which of their critical services were up and which were down during the last service check. Physical boards were available in another room that showed the status of all teams, available to public:

As visible in the figure, each team also had corporate integrity (i.e., user data exposure) which was quickly gone for almost all teams, because the blue teams didn’t really know where the data is, and the red teams found them earlier!

Red teamers were told to not destroy the servers during the first day, but even if they were not told to, they probably wouldn’t. The first thing a hacker does, is strengthen his foothold, by gaining strong persistence to the machine and performing lateral movement throughout the network. Then once they secured their access to the machines, they could bring them down at any time.

The Calm Before The Flood (Day 1)

The first target of the red teams were the ESXi boxes. For UVa specifically, the red team got access and persistence to their local ESXi using the default credentials, but the remote ESXi was secured before they could access it. UVa’s team realized this access and persistence through the logs, even after they had disabled remote console and SSH, and thus decided to reinstall the machine while preserving the datastore, to remove any persistence the red team had on it. This came to save them big time on day 2.

They also realized that since the ESXis and the switch are single points of failure, they should not all have access to them. ESXi access was exclusive to one of the team members, with limited users for others whom needed console access, and switch access was exclusive to the direct console accessed by their firewall guy, Paul Sanders.

Since there weren’t enough workstations, three team members, Calvin Krist, Daniel Chen and Jake Smith had to sit and work on server laptops, and treat them as a workstation, albeit with extra care. Jake took care of the Windows servers, namely the AD/DHCP and the POP3/SMTP mail server, while Calvin took care of the ECOM website on one of the laptops running Debian, and Dan took care of an OpenSUSE hosting the corporate DNS server. They hardened their machines as much as they could, took backups of the configurations and sensitive files as early as possible, and then spent their time monitoring the network, performing injects and threat hunting.

Speaking of injects, 27 injects were delivered throughout the 16 hour competition. On average, the team was working on more than 2 injects at any time throughout the competition, and this took a large portion of their available time. Some injects were done in a few minutes, whereas some took several hours to complete. The white team, which had at least one person in each room at any time throughout the competition, would judge and score the injects for the teams.

Mariah, the team captain, would go around communicate the priorities and the status to all team members, with a smiley face. That might be one of the most important factors contributing to the success of the UVa team, specially under a lot of stress caused by constant red team attacks, injects, and unfamiliar circumstances. She would also hop on an empty machine and help with writing reports, finishing injects, and securing machines here and there when needed.

Quintin was in charge of securing the other Windows boxes, as well as monitoring the network. Paul was the firewall/network man, taking care of firewall, switch and FireEye configurations, all of which he had only recently mastered. Roman and Abbas were in charge of Linux boxes, including the ESXis, and would contribute to injects and other activities when needed.

The UVa team had printed multiple copies of their password sheet, a piece of paper with 60 randomly generated passwords of varying complexity that were numbered from 01 to 60. Whenever a team member needed a new random password, they would pick one of their liking from the sheet, circle it, write what it was used for besides it, and then communicate it to the rest of the team by announcing its number (rather than trying to write it down making it illegible or shouting out character by character). It didn’t matter if the same password was used by different people for different services, as 60 passwords were too many to be enumerated and used by red teamers effectively.

Team members also had practiced using clipboard managers on different OSes, to help them not make mistakes when typing passwords, make the whole process faster, and prevent keyloggers from sniffing passwords.

Throughout the first day, Jake had a lot of trouble keeping up the services on his machines (namely SSH1 and POP3/SMTP), as there was an OpenSSH installation on one of the Windows boxes he was taking care of, and a mail server installation on the other one, both of which were non-standard and custom configured. The Windows boxes themselves were also one of the main targets of the red team, due to AD/DHCP being on them. Jake would do his best to get some green lights on these services to minimize violations, as the team realized it would be hard if not almost impossible to harden and defend these three services throughout the rest of the competition.

As for the rest of the services, the red team was focused on persistence on the first day, and the blue team was threat hunting to minimize the risk surface.

At the end of Day 1, over the debrief, team scores based on service uptime and injects did not include UVa (team 7) in the top 3 spots, making it seem very difficult to win the competition by that point. After the debrief and dinner, the organizers took the teams over to Disney’s Pandora Ride, perhaps to dissuade them from discussing anything technical for the rest of the day. Team UVa had a quick internal debrief before going to bed at 11.

Here Comes The Flood (Day 2)

Day 2 was supposed to be the havoc day. Red teamers were now out for blood. They would delete whole servers if they could, and do anything to keep the services down.

For the first few hours, things were relatively smooth. Since the ESXis had been secured, not much damage came through that route. However, the rest of the services were receiving a lot of attacks. POP3/SMTP would constantly go down, and be recovered back up. In fact, Jake had to reinstall that whole machine at least 3 times to get green lights and minimize violations.

Roman spent a good chunk of time hunting the threat that had infiltrated the network prior to the competition (active incident) to help the team get a huge score boost and have a chance at winning. He would also help with the large number of injects that were pouring in. Quintin, having relatively secured a Windows 2003 Server VM serving a company website, had a lot of trouble keeping it up, as the attackers would constantly bring it down. Paul had finally secured the firewall and the switch, protecting the network against a whole lot of attacks, and was trying to figure out the FireEye box.

Calvin had installed Splunk on all the machines, and was digesting its reports, while Dan was further securing the DNS box. Abbas was monitoring the 8 Linux VMs, as well as the ESXis to see if any suspicious activity happens on any of them, and Mariah was coordinating it all.

About noon, the attacks started coming. The Windows 2003 would constantly go down. At first, the team would restore hot snapshots through ESXi, to get a green check and no violations, but as the red team realized this, they started automating their exploit to bring down the server as soon as it goes up. To remediate that, the team would restore different snapshots from different states throughout the 2 days of the competition, making it difficult for automated exploits to cause havoc. But the red team soon caught up and made it all automatic.

The team then had to take down the network connection on the Win2003 machine, to gain some time to harden it. The hardening effort took roughly 20 minutes, resulting in one violation, but made the machine good to go. Abbas and Quintin, using the ESXi console, first found all the suspicious processes and persistence that was on the machine, mapped them to files on the disk, and deleted the files. But they all popped back in! Circular dependencies. Nothing to worry about though. They created an empty file, removed its write and execute permissions, and one by one, deleted the malwares replacing them with the empty file (with the same exact name), preventing the automated persistence scripts from overwriting them. After about 15 malwares, the suspicious processes stopped to spawn.

Now was time for the services. Aside from the necessary ones (IIS, MSSQL), everything else had to go down, and made manual instead of automatic. All the extra users were also removed from the machine, and those which would automatically pop back in where locked and renamed, so that the persistence scripts would fail on them.

After the hardening effort, they took a snapshot of the hardened state, and connected the machine back to the network. It went well for a while, until the machine was shut down. After restoring the hardened snapshot, it happened again. The team realized that either a remote shutdown signal or a local daemon is causing this, but after inspection, understood that the red team doesn’t have any direct access to the machine. The team then came up with a solution: open up notepad, write something in it, but don’t save the file. Now the shutdown signals would fail because “You have unsaved files open”. The Win2003 box was secured and stayed up for the rest of the competition!

Roman & Abbas’ monitoring on Linux boxes would show red team IPs doing stuff from time to time. Instead of immediately kicking out the intruder, they would watch the activities of the red teamer to figure out what other persistence they are installing and what they are interested in. The blue team had their own persistence on the Linux machines, on top of the normal SSH and ESXi access, just in case those two were compromised. Once Roman & Abbas would discover a red teamer’s goal on one of the Linux VMs, they would kick him out, rotate passwords, disable the persistence, and write an incident report with all the details. Incident reports could restore half of the points lost due to a compromise by the red team.

On the physical servers side, Jake was heavily battling the red teamers on Active Directory, SSH and Mail Server. Around 3 PM, the red teamers found an SQL injection vulnerability on the ECOM website (Calvin’s physical server), and through that, deleted the Mysql metadata database, actively disabling the web server. Calvin and Abbas quickly rooted out the issue, and restored a backup of /var/lib/mysql directory, restarting the web server, and bringing it back up. Calvin was then actively trying to hunt the vulnerability and prevent the catastrophe from happening again.

Unfortunately for the team, their DNS server, which was under Dan’s OpenSUSE physical server, went down about 3:50 PM, bringing down a whole array of other services that relies on DNS with it, including a dozen websites. Dan and Abbas quickly jumped on that machine, tracing the issue. The named (pronounced name-D) service would die as soon as it started. Their first thought was that a malicious daemon is killing named, so they started killing any service on the box that was not familiar, or was modified since the beginning of the competition, or didn’t pass debsums checks.

But it didn’t seem to help much. They even killed the X server, the GUI, and all other processes, to the point where there was only named and init processes running, but the named service would still go down. They concluded that the named is somehow corrupted, and thus a machine restore is the best option. Using a hard drive that had a snapshot of the machine, they restored the whole machine, which took about 10 minutes, but prior to checking named, they first attempted hardening the default credentials and the users. In the heat of the moment they panicked, and forgot which password they had chosen from the password sheet, effectively ruining the machine, having to restore it again from the snapshot.

This time, as soon as the machine booted, they disabled its network interface. Now named was not dying anymore, making them realize that whatever was causing the crash, was coming from the network. They soon realized that it’s a vulnerability in bind9, and the red team is performing a UDP Flood on that server, causing it to quickly crash. Finding an applying a patch seemed implausible in the heat of the moment, so instead they quickly wrote a cleanup and restart script for bind9, making it spawn another instance of the server as soon as the old one went down. Network plugged back in, server restarted, script launched, machine hardened, and voila, all services are green! The flood was crashing the DNS server about 4 times a second, whereas the script was launching it about 6 times a second, giving the team full uptime.

The scoreboard above shows the scores and services up for each team. Since the recovering of the DNS server took about 1 hour, you can see that at 16:25 PM 6 services are down for Team07, whereas for Team02 and Team03, 9 services are down. However, from that point onward, almost all of Team07 services were up (until 18:00), whereas other teams had a majority service downtime.

At the end of the competition, UVa had 4 services down, POP3, SMTP, FTP and HRM(HTTP). TThey also had all of their user data compromised, which was on par with other teams, except those who managed to locate and secure user data before the red team.

At the end of the competition, Team07 had about 20-30 violations, most of which were for POP3/SMTP services. At that point, they were under the impression that they would rank somewhere between 3rd and 5th place, considering they were unsuccessful at discovering the previous incident that had fired the team before them.

They went for the dinner service and the career fair that followed, had a quick debrief that night, and went to sleep.

The next morning, there was a CTF called Panopoly, from 9 to 11. UVa ranked 4th on that, but it was fun. They were expecting a fourth place, but were still happy with their performance and very confident. Throughout the awards ceremony, the red team organizers talked about how things went for them and what technologies they had used, and all the teams were eagerly waiting for the winners to be announced.

As only the first top three teams would be announced, team members were hoping that they would rank somewhere in the first three, because being the fourth would pretty much be the same as being last. First, Dakota State University was announced as the third place, making everyone’s heart skip a beat. UCF was the heftiest opponent, with the competition being in their hometown, and them having won 2014, 2015 and 2016 national championships, so when they were announced second place, they were pretty sad, but also everyone else felt some hope of winning. Keep in mind that no one was aware of the scoreboard comparing the teams you saw earlier.

By this time the team was saying to themselves “we are either first or fourth”, and their heart rates had doubled. Once the words “The University of” came out by the announcer for the first place winner, they already knew they had won. It was unprecedented, they had only formed the team 3 months ago, and practiced a few times. Some of their members were traveling this whole time and could only attend 2 practice sessions. They had won regionals, beating 2017 champions UMBC, and had now won nationals! Truly a dream come true for them.