Cisco is commemorated to be a spouse of the Dark Hat NOC (Community Operations Heart), and this was once our 7th time supporting Dark Hat Asia. Cisco is the Legit Cell Tool Control, Malware Research and DNS (Area Identify Carrier) Supplier.
We paintings with alternative legit suppliers in order the {hardware}, instrument and engineers to create and reserve the community, for our joint buyer: Dark Hat.
Arista: Community Apparatus
Corelight: Community Analytics and Detection
MyRepublic: Broadband
NetWitness: Ultimatum Detection & Reaction, Identification
Palo Alto Networks: Community Safety Platform
The principle undertaking within the NOC is community resilience. The companions additionally grant built-in safety, visibility and automation, a SOC (Safety Operations Heart) throughout the NOC.
On monitors out of doors the NOC have been displayed spouse dashboards for the attendees to view the amount and safety of the community visitors.
It All Began with Malware
Cisco joined the Dark Hat NOC in 2016, when requested to grant computerized malware research with Wool Grid. The Cisco contributions to the community and safety operations advanced, with the desires of the buyer, to incorporate extra elements of the Cisco Safety Cloud.
The NOC leaders allowed Cisco (and the alternative NOC companions) in order in alternative instrument to produce our inside paintings extra environment friendly and feature larger visibility; alternatively, Cisco isn’t the legit supplier for Prolonged Detection & Reaction, Community Detection & Reaction or Collaboration.
Breach Coverage Suite
Cisco XDR: Ultimatum Searching / Ultimatum Judgement Enrichment / Government dashboards / Automation with Webex
Cisco XDR Analytics (Previously Keep Cloud Analytics / Stealthwatch Cloud): community visitors visibility and warning detection
Cisco Webex: Incident notification and workforce collaboration
The Cisco XDR Command Heart dashboard tiles made it simple to look the situation of each and every of the hooked up Cisco Safety applied sciences, and the situation of ThousandEyes brokers.
When the companions deploy to each and every convention, we arrange an international elegance community and safety operations heart in 3 days. Our objective remainder community up past and growing higher built-in visibility and automation. Dark Hat has the select of the protection business equipment and incorrect corporate can sponsor/purchase their method into the NOC. It’s invitation handiest, with the purpose of variety in companions, and an expectation of complete collaboration.
As a NOC workforce made out of many applied sciences and firms, we’re regularly innovating and integrating, to grant an total SOC cybersecurity structure answer. We sit up for proceeding the paintings with spouse Palo Alto Networks, for additional automation at Dark Hat USA 2024.
Beneath are the Cisco XDR integrations for Dark Hat Asia, empowering analysts to research Signs of Compromise (IOC) in no time, with one seek.
We respect alphaMountain.ai, Pulsedive and Recorded While donating complete licenses to Cisco, for virtue within the Dark Hat Asia 2024 NOC.
An instance of that is an investigation of a probably bad job at the 2d date of Coaching. An IP cope with was once known through NetWitness for conceivable geolocation leakage.
Investigation of the IP correlated the syslog sightings from the spouse applied sciences within the NetWitness planks, with warning perception from Pulsedive, Recorded While, alphaMountain and others.
Reviewing the DNS planks and the main points of the packet seize in each Corelight and NetWitness, it was once showed incorrect geolocation knowledge was once leaked and it was once a part of a Coaching path. The job would were restrained in a manufacturing order.
A core built-in workflow within the Dark Hat NOC is NetWitness and Corelight sending suspicious information to Keep Malware Analytics. Over 4,900 samples have been submitted.
The NOC analysts additionally old Malware Analytics to research suspicious domain names, with out the danger of sickness. In lieu than taking to the web page on a company or Dark Hat property, we have been ready to engage with the web page within the glovebox, together with downloading and putting in the web page payload.
Detonating information or surfing internet sites in Keep Malware Analytics protects the analysts from unintentional sickness.
We noticed a form of alike (however other hash values) exploit kits downloaded at the first date within the Trade Corridor. The downloads have been at the convention Wi-Fi and no longer in a Coaching path, so the development needed to be investigated to verify there was once no longer an assault at the attendees. Running with the Corelight workforce, the NOC responders parsed the visitors and showed it was once a Seize the Flag tournament, which persisted into the latter date of the convention.
Ultimatum Hunters’ Tale, through Aditya Raghavan and Shaun Coulter
Within the Dark Hat Asia 2024 NOC, Shaun staffed the morning shifts, and Aditya the afternoon shifts, as warning hunters centered at the Cisco XDR and Keep Malware Analytics consoles. Mornings have been typically good-looking kick back. On the other hand, and for some heretofore unknown (espresso alike?) explanation why, the job ramped up within the afternoon on maximum days, well-known Aditya to a playground of “involved joy”, and Shaun to a playground of tormented jealousy :D. With dogged resolution each hunters spent their past reviewing indicators, actions, and carried out IOC scans the usage of XDR Examine. They reviewed submitted samples and community planks for indicators of intrusion or suspicious job.
The usage of Keep Malware analytics, they dissected malware samples, analyzed phishing campaigns, and scrutinized community visitors patterns for anomalies. Diverse indicators flagged as spikes in visitors from sudden assets, odd locations and unusual variants of bad code popped up more than one instances, starting up thorough investigations. Typically, they traced the ambiguity to a certified Dark Hat Coaching or Briefing supply and closed such instances as “Black Hat Positive”; that means you wouldn’t permit this to your manufacturing community, however for Dark Hat, it’s industry as regular. Since Dark Hat is a convention designed for studying about offensive safety, those malware samples are anticipated, and marked as such.
Fortunately or unthankfully, because the machine tuning was once finished, maximum indicators raised have been as above and anticipated or in reality ‘near misses’ – pieces that warrant investigation however didn’t lengthen to impactful behaviours, as we have been ready to block them in past.
At the first date of Briefings, as Shaun is dutifully poring throughout the console of Keep Malware Analytics, in walks Aditya to alleviate the shift. Hi apart, Shaun briefly pivots over excitedly “Brother, I want to show you a couple of interesting things.” Aditya’s pastime is piqued, and Shaun opens a untouched dashboard appearing one of the vital just lately discharged options of Cisco XDR – MITRE ATT&CK ® Protection Map.
This untouched capacity briefly shows all of the techniques and strategies within the MITRE ATT&CK® matrix for which Cisco XDR has detections/protection. Along with the XDR Local, detections from Keep Endpoint and Keep Malware Analytics also are old to derive the protection map making it a holistic view. This view permits the consumer to visualise the detections of XDR natively, in addition to the built-in answers and establish the scope of protection and importantly, map out the gaps for month attention. Due to the Cisco Talos workforce, all answers inside the Cisco Breach Coverage Suite are mapped as of late and this may be rolled out to incorporate alternative suites and answers, together with third celebration integrations, quickly.
As our warning hunters geek out at the behind-the-scenes stuff on XDR, Jessica courteously yells out “Adi. Shaun. Guys, there is some new activity on Umbrella. Can you look into it?” Nudged again to fact, our warning hunters get to paintings – discovering needles within the stack of needles at Dark Hat because it was once rightly put through Grifter! Speaking about that, the untouched job seems to be a question for a site labeled as a Command & Keep watch over (C&C) area. Let’s dig into it.
A handy guide a rough glance into Umbrella Process Seek displays the original visitors job indistinguishable the C&C section that was once allowed. Increasing the main points pane, we will see the area being queried and the identification of the endpoint issuing the question which seems to be from the ‘Hacking Enterprises 2024 Red Team’. That may be a official Coaching elegance at Dark Hat Asia 2024. We pivot over to Umbrella Examine and spot the cause of this area being categorised as C&C and its signs.
Let’s head over to XDR and question this obvious in opposition to all of the built-in answers for extra intel. We briefly get a optic hooked up graph and tabulated occasions on all of the related intel. The mixing with NetWitness Woods supplies us with occasions alike to that area, in addition to populating the graph with the ones relationships, together with the Umbrella tournament which was once the supply for this hunt.
Taking a look on the proof, this grew to become out to be some other needle! Not anything untoward right here, we labeled this as a ‘Black Hat Positive’ and went away. Because the afternoon shift winds unwell, the workforce is discussing attainable locations for dinner and there’s all the time dessert to sit up for on the finish. Aditya and Ryan have been pining for affluent prosperous ice cream and House Very best Dessert seems to be the appropriate answer for the ask. Within the NOC, the appropriate answer is nearly all the time teamwork with all our companions.
One such example was once when a Corelight hunter picked up a spike of visitors to bizarre locations. Those seem to be DNS queries to a number of C&C domain names. We briefly delve into Umbrella appearing us all of the domain names being queried in a shorten window and maximum of them being Malware and/or C&C categorised. This seems to be a machine both being compromised or anyone deliberately doing a check / recon for the ones domain names.
Let’s examine a few of these domain names in XDR. We will be able to see a quantity of purple icons in this visualization! If truth be told, each queried area is assessed as Sinful and identified to host alternative bad content material. This doesn’t glance anticipated evidently and that places the intentional check / recon idea to extra briefly. Ben Reardon, the hunter from Corelight, places it succinctly “This box is pwned six ways to Sunday!” What else are we able to to find about the program next?
Taking a look on the DHCP planks for the IP cope with, the Corelight hunter was once ready to pinpoint the software MAC cope with and hostname, which resembled a reputation. A shorten Google seek next, we have now a possible software proprietor and the truth that he was once turning in a consultation at Dark Hat in one of the vital rooms after door! A shorten dialog with the individual then his consultation ensued, the place the NOC leads prompt the NOC’s findings on his compromised machine. He was once thankful for the discovering and reached out for alternative context. This one grew to become out to be a ‘True Positive.’
Please see date, the workforce has zeroed in on Turkish meals for the night. Ryan halts Shaun as he departs on the finish of his shift and calls for his resort title and room quantity. “I’m gonna come knock at your door and wake you up tonight, man. I mean it. No day is too long. I used to do my shifts on three hours of sleep. Now, let’s go!” Ryan is deadpan critical. That’s what we concept day investigating our after attainable malware discovering.
Every other tournament at the Umbrella console involves our consideration and this past this can be a question for a site categorised as Malware. The supply endpoint is instantly known from the Identification and Umbrella examine tells us this area is a part of the Malware prohibit checklist. In a standard manufacturing community, this may preferably be restrained.
Dark Hat isn’t your standard manufacturing community, and it draws a wide variety of safety nation. And that’s precisely what it grew to become out to be this past. The Nationwide College of Singapore has a gaggle organizing ordinary seize the flag (CTF) occasions and is operating a alike get-together at Dark Hat. Exit NUS Greyhats!
Actions involving malware what could be restrained on a company community should be allowed, inside the confines of Dark Hat Code of Habits.
Community Observability with ThousandEyes, through Adam Kilgore and Patrick Yong
Deploying ThousandEyes at Dark Hat is a rigorous procedure involving a quantity of {hardware} (some proven beneath), configuration, trying out, troubleshooting, and operating across the convention heart.
Along with our standard deployment duties, we carried out more than one enhancements to the provider. Those enhancements incorporated an overhaul of the dashboards to turn granular knowledge for each and every convention room, along mixture knowledge for all of the convention; and higher labeling and group of deployed brokers.
The ThousandEyes dashboard was once projected at the massive display screen within the NOC, for alerting on any community problems, previous to stories from customers.
At the troubleshooting aspect, we advanced our plank research and assortment tactics and arrange centralized tracking of wi-fi knowledge. Those efforts contributed to enhancements in visibility and agent uptime all through the convention.
All through the preliminary two days of Coaching periods at Dark Hat, ThousandEyes brokers confirmed handiest minor deviations from baseline because the Coaching periods got here on-line. Because the Coaching periods persisted, efficiency was once solid, with handiest uncommon indicators for minor degraded throughput or reasonable latency spikes. On Thursday, all of the two-day Coaching periods have been closed, and the convention shifted against Briefings, along two four-day Coaching periods that ran for the convention’s area. With get started of Briefings and opening the Trade Corridor, headcounts enormously larger. ThousandEyes noticed degraded efficiency at the community, basically within the massive convention rooms webhosting the Briefings. The beneath symbol displays a check end result from the Hibiscus 3610 ballroom:
The community trail above displays obese latency at the first hyperlink to the default gateway, compounded through some other prime latency hyperlink out of doors the convention community. A breakdown of connectivity for the above trail is proven beneath:
The throughput quantity above is vital to this investigation. The Get entry to Issues (APs) for the Hibiscus 3610 ballroom had a median throughput of round 174 Mbps. Reviewing AP planks, we discovered that 92 customers have been hooked up to the similar AP from which the check was once run. Dividing the 174 Mbps through 92 offers a median throughput in sequence with the 1.7 Mbps proven above, so the destitute connectivity was once pushed through oversaturation of consumer connections on this branch.
The Hibiscus 3610 room and alternative brokers in a close-by hallway constantly had the worst connection a number of the convention rooms, as proven through our agent polling effects.
Presen there have been boundaries within the quantity of bandwidth to be had for the convention usually, the information above suggests extra of the to be had AP and bandwidth assets must be allotted to the Hibiscus 3610 ballroom and adjoining hallways for month convention topologies, which was once shared with our Community Apparatus spouse.
Meraki Methods Supervisor, through Paul Fidler and Connor Loughlin
Our 8th deployment of Meraki Methods Supervisor because the legit Cell Gadgets Control platform went very easily, and we presented a untouched caching operation to replace iOS units at the native community, for velocity and potency. Going into the development, we deliberate for please see kinds of units and functions:
iPhone Top Scanning Gadgets
iPads for Registration
iPads for Consultation Scanning
We registered the units in journey of the convention. Upon arrival, we grew to become each and every software on.
Later we ensured Location Services and products enabled, all the time on.
Rather of the usage of a lump deployment era, like Apple’s Automatic Tool Enrollment, the iOS units are “prepared” the usage of Apple Configurator. This comprises importing a Wi-Fi profile to the units as a part of that procedure. In Las Vegas, this Wi-Fi profile wasn’t eager to auto fix the Wi-Fi, make happen the want to manually alternate this on 1,000 units. Moreover, 200 units weren’t reset or ready, so we had the ones to reimage as neatly.
Dark Hat Asia was once other. We took the teachings from Dark Hat USA 2023 and coordinated with the contractor to organize the units. Now, should you’ve ever old Apple Configurator, there’s a number of steps had to get ready a tool. On the other hand, those may also be blended right into a Blueprint.
For Dark Hat Asia this incorporated:
Wi-Fi profile
Enrollment, together with supervision
Whether or not to permit USB pairing
Setup Associate pane skipping
In Meraki Methods Supervisor, we managed the packages through the assigned virtue, designated through Tags. After we got here in at the first morning of the Briefings, 3 iPhones had to be modified from govern scanning within the Trade Corridor, to Consultation Scanning for the Keynote, so the attendees may just fill the corridor sooner. Reconfiguring was once so simple as updating the Tags on each and every software. Moments next, they have been able for the untouched undertaking…which was once remarkable because the Keynote room crammed and needed to advance to an flood room.
We additionally have been ready to verify the bodily location of each and every software if wiping was once required because of loss or robbery.
When it was once past for the attendees to check in, they simply displayed their QR code from their private telephone, as won in e-mail from Dark Hat. Their badge was once in an instant revealed, with all private main points attach.
This is going with out pronouncing, however the iOS units (Registration, Top Seize and Consultation Scanning) do have get entry to to private knowledge. To assure the protection of the information, units are wiped on the finish of the convention, which may also be finished remotely via Meraki Methods Supervisor.
Content material Caching
One of the vital largest issues affecting the iOS units in Dark Hat USA 2023 was once the fast want to each replace the iOS software’s OS because of a area to recovery a zero-day vulnerability and to replace the Dark Hat iOS app at the units. There have been loads of units, so this was once a problem for each and every to obtain and set up. So, I took the initiative into having a look into Apple’s Content material Caching provider constructed into macOS.
Now, simply to be unclouded, this wasn’t caching EVERYTHING… Simply Apple App bind updates and OS updates.
That is grew to become on withing Device Environment and begins operating straight away.
I’m no longer taking to get into the weeds of surroundings this up, as a result of there’s such a lot to plot for. However, I’d counsel that you simply get started right here. The surroundings I did alternate was once:
Location and Jailbreak detection
Something that we haven’t spoken about in at some time is Jailbreak detection and Location. There are lots of parts that we get again from a tool, however two of them, Location and Jailbreak should be retrieved from a tool the usage of a supplemental utility: On this case, the Meraki Methods Supervisor agent.
HOWEVER, those can handiest be retrieved from the software if the applying is operating within the background. If the software has been rebooted, or the applying terminated, next we don’t get anything else.
One of the vital alternative painful, however comprehensible, sides of MDM is that you’ll’t settingup an utility far flung on a cellular software…. However you’ll!
On each Android and iOS, there’s an ability referred to as Kiosk or Unmarried App method: Usefulness instances for this are most often unattended units, like in eating places, or scanning units like supply drivers, and so on. And when sending the command to the software to advance into kiosk method will settingup the applying. You’ll additionally ship a command to take away kiosk method from the software too. The superb factor about this latter level is that the applying remainder in focal point and detectable!
So, the alternative capacity that the usage of Meraki Methods Supervisor offers us is the power to time table settings. Subsequently, we will activate kiosk method in the course of the evening and take away it an presen next.
To assure that this doesn’t affect the registration team of workers, we will advance one step additional: then we’ve introduced Meraki Methods Supervisor, an presen next we will relaunch the registration utility, Swapcard Exit.
Systematic ThousandEyes Agent Deployment
ThousandEyes has been a collision at Dark Hat. At an tournament the place figuring out straight away the place problems lie within the community and past to assure a superb convention is paramount, the visibility ThousandEyes offers is unbelievable. For the reason that, and the complexity of the community right here, and for the reason that we have now a Mac Small deployed for caching instrument updates, as we’re the usage of Meraki Methods Supervisor (SM) for alternative functions, I believed I’d hurry the chance to deploy the ThousandEyes Agent the usage of SM.
The alternative explanation why is that, while we have now a large amount of cloud and undertaking brokers, we had incorrect endpoint brokers deployed. On the other hand, issues are by no means that straightforward with instrument deployment, basically as a result of you wish to have to provision / configure instrument as soon as deployed. On cellular units, that is simple, both the usage of settings payloads, or through the usage of Controlled Appe Config to configure an app.
On desktop, the usage of MDM, we will most often virtue such things as Controlled Plists to do the similar factor, however the TE agent does NOT aid this. As soon as put in, we should name the agent with a fable.
So, to succeed in all this, we will package deal the agent and command right into a package deal the usage of a command sequence significance at the Mac referred to as PKGBUILD (extra main points right here).
I additionally old a information I’d written for the Meraki Population, to be had right here.
Info of notice:
The Postflight:
#!/bin/bash
# this title will alternate with each and every model of the agent
installer -pkg /tmp/Endpoint\ Agent-x64-1.193.1.pkg -target /
/Programs/ThousandEyes\ Endpoint\ Agent.app/Contents/MacOS/te-agent –check in “YOURUNIQUESTRING”
proceed 0
The command to create the package deal the usage of PKGBUILD
Extra main points right here or supervise the video.
Repurposing of Gadgets for the after display
We have been requested if there was once anything else lets do to release the units as they have been for the after display. Then cautious attention, we made up our minds that lets release the units in a condition that was once amenable to everybody. The key requirement was once retirement the Swapcard Exit app at the software. However, because the app is provisioned for each and every display, it’s rather the method to take away configuration and next re-add it….
So, the alternative factor to notice is the choices that we have got when putting in (and getting rid of) an utility on a controlled iOS software:
Take away with MDM is the fascinating one, because it permits us to, instead than WIPING the software on the finish of the display, to take away control, together with any apps and settings, and their corresponding knowledge.
The disorder with that is that this was once by no means a demand at first of the display. So, we now desire a procedure in a selected form to facilitate this…. As that is for just a handful of units:
Deprovision the app from units through unscoping the applying in Meraki Methods Supervisor
Wait to look this command has finished throughout all units
Reprovision the app the usage of MDM once more, however with this being a untouched app set up, it is going to permit the OS to book the app in situ then an unenrollment
Wait till finished
Unenroll the software
Area Identify Carrier Statistics, through Christian Clasen
Since 2018, we have now been monitoring DNS stats on the Dark Hat Asia meetings.
The historic DNS requests are within the chart beneath.
With over 18.2M DNS requests made, we had essentially the most to age at an Asia display. We made visibility developments on the earlier time’s Asia convention. Previous to Asia 2023, we have been permitting attendees to virtue their selected DNS resolvers over our assigned inside Umbrella Digital Home equipment. In coordination with Palo Alto Networks (the convention Firewall supplier), we started intercepting and redirecting DNS queries for alternative resolvers, to pressure solution throughout the Umbrella equipment. Presen that is handiest efficient for plain-text DNS queries and no longer encrypted protocols like DNS over HTTPS, it never-the-less dramatically larger visibility as evidenced through the numbers within the accompanying charts.
The Process quantity view from Umbrella offers a top-level point look of actions through section, which we will drill into for deeper warning searching. On development with the former Dark Hat Asia occasions, the summit Safety sections have been Malware and Newly Unmistakable Domain names.
In a real-world order, of the 18.2M requests that Umbrella noticed, over 2,000 of them would were restrained through our default safety insurance policies. On the other hand, since this can be a playground for studying, we most often let the whole lot fly.
We additionally observe the Apps the usage of DNS, the usage of App Discovery.
2024: 4,327 apps
2023: 1,162 apps
2022: 2,286 apps
App Discovery in Umbrella offers us a handy guide a rough snapshot of the cloud apps in virtue on the display. No longer strangely, Generative AI (Synthetic Judgement) has exploded over the former time as a summit utility.
Umbrella additionally identifies dangerous cloud packages. Will have to the will stand, we will prohibit any utility by way of DNS, comparable to Generative AI apps, Wi-Fi Analyzers, or anything that has suspicious undertones.
Once more, this isn’t one thing we’d most often do on our Common Wi-Fi community, however there are exceptions. As an example, each so regularly, an attendee will be told a fab hack in one of the vital Dark Hat lessons or within the Arsenal front room AND attempt to virtue stated hack on the convention itself. This is clearly a ‘no-no’ and, in lots of instances, very unlawful. If issues advance too a ways, we can step in the right direction.
All through the convention NOC File, the NOC leaders additionally record of the Manage Divisions clear at Dark Hat.
General, we’re immensely pleased with the collaborative efforts made right here at Dark Hat Asia, through each the Cisco workforce and all of the companions within the NOC.
Dark Hat USA can be in August 2024, in Las Vegas. Christian Clasen will govern the Cisco workforce within the NOC, so practice his weblog to look if what occurs in Vegas, remains in Vegas.
Acknowledgments
Thanks to the Cisco NOC workforce:
Cisco Safety: Christian Clasen, Shaun Coulter, Aditya Raghavan, Adam Kilgore, Patrick Yong and Ryan Maclennan
Meraki Methods Supervisor: Paul Fidler and Connor Loughlin
Alternative Aid and Experience: Adi Sankar, Robert Harris, Jordan Chapian, Junsong Zhao, Vadim Ivlev and Ajit Thyagarajan
Additionally, to our NOC companions NetWitness (particularly Iain Davidson and Alessandro Zatti), Palo Alto Networks (particularly James Holland and Jason Reverri), Corelight (particularly Mark Overholser and Eldon Koyle), Arista Networks (particularly Jonathan Smith), MyRepublic and all of the Dark Hat / Informa Tech team of workers (particularly Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Michael Spicer, Jess Jung and Steve Oldenbourg).
About Dark Hat
Dark Hat is the cybersecurity business’s maximum established and in-depth safety tournament form. Based in 1997, those annual, multi-day occasions grant attendees with the original in cybersecurity analysis, building, and tendencies. Pushed through the desires of the family, Dark Hat occasions show off content material immediately from the family via Briefings shows, Trainings lessons, Summits, and extra. As the development form the place all occupation ranges and educational areas convene to collaborate, community, and speak about the cybersecurity subjects that subject maximum to them, attendees can to find Dark Hat occasions in america, Canada, Europe, Heart East and Africa, and Asia. For more info, please seek advice from www.blackhat.com. See the click leave for Dark Hat Asia 2024.
We’d love to listen to what you assume. Ask a Query, Remark Beneath, and Keep Attached with Cisco Safety on social!
Cisco Safety Social Channels
InstagramFacebookTwitterLinkedIn
Proportion:
