OSINT Basics for Attack and Defense By Andrew McNicol & Matt Foreman
Matt Foreman @s7foreman • Security Consultant • I have some certifications, they are made of letters • I do Penetration Testing, Security Assessments, and sometimes what I call research….
Andrew McNicol • Security consultant • Part-time beard developer try: I enjoy writing error-free Python with Google and stackoverflow except:pass • I do both offensive and defensive stuff
We didn’t do it • We are not lawyers or giving you legal advice • We are not giving you permission or authorizing you in any way to do anything ever • In fact don’t do anything ever
What is OSINT? • OSINT has been formally defined this way… Open-source intelligence (OSINT) is intelligence collected from publicly available sources. In the intelligence community (IC), the term "open" refers to overt, publicly available sources (as opposed to covert or clandestine sources); it is not related to open-source software or public intelligence. • Also check out the PTES , tons of great info http://www.pentest-standard.org
This talk • OSINT has been discussed from a high level to very deep dives in past talks by others • This talk might cover some offensive methods of OSINT you might have seen before, but we also want to cover some defensive uses levering the same/similar OSINT tools that we see mentioned less often
Shodan • Allows users to search for publicly connected internet devices that have been seen by Shodan • Routers • Servers • Firewalls and other Security Devices • SCADA or other Control Systems… – This data can be searched for by IP/CIDR combo – Open ports seen by Shodan – Hostname, OS, Geo-Location, etc… – Server Response
Shodan for Attackers • So it’s fairly easy to see how this can be useful to attackers. • This simple query will show everything seen by Shodan in the US (MERICA!) with TCP 445 open to the internet…
Shodan for Attackers • Hopefully this an uncommon thing you would see on engagements but you get the idea • Without sending a packet to the end customer/target we can identify some of their external infrastructure and at one point what was there
Shodan for Defenders • Understanding what information is available in Shodan can help defenders too • Shodan can be leveraged to fingerprint C2 servers Attackers sometimes make mistakes in server responses These unique strings could help enumerate additional C2 servers • Can be leveraged to see server responses without actually making a request
Shodan for Defenders • Example of searching for “Apach” and “202”:
Maltego by Paterva • Commercially licensed • Runs on multiple different OS • Can integrate API’s from many different Sources • Great for stalking people! <note> remove this its creepy </note> • Uses various “transforms” to gather and hopefully correlate data between various sources
Maltego for Attackers • Here is a simple graph output of a Maltego search
Maltego for Attackers • From this point we can start mapping out infrastructure, people, known aliases, social media, etc.. • All can be valuable information for attackers depending on the goal…..and the scope
Maltego for Attackers • This doesn’t come with out false positives, but after enough digging you could end out with a map like this….
Maltego for Attackers • There are many add-ons to Maltego, including one for Shodan
Maltego for Defenders • Maltego can be a great way to perform link analysis with indicators of compromise • Malformity adds a lot of malware functionality:
Maltego for Defenders • Example of running various transforms and enumerating more information from the hash value (mutex, C2, other samples, etc.):
Have you seen this thing, Google? • So we have all seen Google hacking before and probably the most notable example is the Google Hacking Database or GHDB – Originally created by Johnny Long • And attackers obviously still use these methods today • Here is a very simple Google search for Juniper’s SSL VPN login page…I'm sure this was searched during the Heartbleed craziness #heartbleedcyberAPT
Google for Attackers • This search looks for a WordPress plugin that is vulnerable to an open redirect. About 235 results came back with modifying the query much • exploit-db/exploits/18350/
Google for Attackers • This search looks for a search looks for open Cisco Routers, finding over 15 million results • And here we see one of the results has an open command window running with level 15 privileges
Google for Attackers • People tend to reuse usernames, handles, etc... • So if we can find some target IT personnel on a resource like Linkedin, Facebook, or Twitter and do some searching for common handles they like to use, sometimes you end up with system administrators posting complete firewall configurations onto public websites….
Google for Attackers • A little more digging on the person who shall not be named showed that his/her username was reused on multiple sites and one tech-help forum, which had public profiles • This included corporate email used to register, full name, and location • Some users of these forums include their corporate email signature and tagline (giving us more terms to include in targeted searches) “We are the leader in CyberDongleWidgets, and we know it” • Try a Google search for some of the popular tech forums… site:http://www.tek-tips.com/viewthread.cfm? /etc/shadow
Google for Defenders • Knowing your organizations exposure online can help you defend • Google searching indicators from malware can save you time: • Hashes, Strings, Domains/IPs, persistence mechanisms, mutexes, etc.
Google for Defenders • Humans lie, and humans are creatures of habit: • Fake Domain Registration Information (Emails, Phone numbers, Addresses, etc.)
Online Data Dumps • Monitoring data dumps from the target or 2rd parties can be provide a treasure trove of information for the attacker (Usernames, passwords, etc.) • From a defensive standpoint, monitoring these data dumps for your organization can allow you to take appropriate action
Linkedin • If Social Engineering or Phishing is in scope you can use this data to find targets • Existing personnel to enumerate technologies and partner relationships or company updates listing new projects or acquisitions • New employees are often good targets – Minimal Training – Don’t know IT staff on a first name basis – Sometimes have default AD credentials (changem3)
Additional Search Items • More things to search for… o Business Partners o Vendor Relationships o Are certain functions outsourced? Like HR, the helpdesk, etc…
Wireless Communications • Openbmap.org • wigle.net Find previously discovered wireless in the area of your target
Researching IPs and Domains • Link analysis between IPs, Domains, and Name Servers can help map out additional hostile infrastructure: • Robtex, iplist.net, nslist.net, pop.dnstree.com, webboar.com, centralops.net, etc.
Researching IPs and Domains • Given a hostile Domain/IP ask yourself: • Any fake registration information? • What other domains point to IP? • What other domains leverage that name server? • What domains point to IPs around the hostile? • Additional subdomains (skills.cnndaily.com, jobs.cnndaily.com) • Resolve back to non-routable IP space (Loopback, bogon) • Domains that look right, but are slightly off: • update.macfee.com • mirosoft.supportca.com
Researching IPs/Domains • Passive DNS can allow you to track changes to domains overtime: •Virustotal, DNSDB, Edv-consulting • Hostile infrastructure gets reused: – Can help enumerate additional infrastructure – Can assist with attribution
Automation • Automating tasks is key – especially since you may have to do something thousands of times • Use Case: Whois automation with Team Cymru's Python whois module – 1000s of lookups within seconds:
Automation • Creating and parsing web requests via a scripting language can save a lot of time • Use Case: Looking up IPs via iplist.net with Python
OPSEC and OSINT • As you start digging on the line be aware of the information you are exposing about yourself or your organization • Many ways to control what information you give to the Internet: • Google Cache • Firefox Plugins: • Foxyproxy + ssh tunneling • User Agent Switcher • NoScript • Refcontrol • Tamperdata • Tor, VPNs, Proxy services etc. • Separate non-attrib ISP link
Recon-ng for Attackers • Started by Tim Tomes (@LaNMaSteR53) • Many contributors • Menu feels similar to msfconsole • Way too many great features to list today • Can be a one-stop-shop to gather a ton of data recon/hosts/gather/http/web/bing_domain
Recon-ng for Attackers • This above example is querying searchdns.netcraft.com for additional hosts. • Also its worth looking at these for DNS info as well. These are querying an DNS server of your choice instead of searching recon/hosts/gather/dns/reverse_resolve recon/hosts/gather/dns/brute_hosts
Recon-ng for Attackers • Search xssed.com for past entries. Can be useful for the later phases of attack. Keep in mind the dates on some of the entries
Recon-ng for Defense • Malwaredomainlist.com Module:
Recon-ng for Defense • Hostname Resolver Module:
Malware Sandboxes • Many Internet resources exist to analyze malicious samples: Virustotal Malwr.com ThreatExpert.com CWSandbox • These are very useful, but keep in mind that they often make some of the data public • Adversaries can monitor these online resources just like defenders • Uploading a sample could let the adversary know you found their malware • Cuckoo sandbox can be a free solution
Malware Sandboxes • Cuckoo Sandbox is a free alternative to standup a local malware sandbox:
Public doc’s and metadata • Strings, Exiftool, etc.. • Pull down public documents (pdf, doc, ppt) • The content itself could be as useful as metadata • Sometimes IT creates “how-to” guides disclosing technology and settings used • Metadata (What version of Office, Adobe, etc…) When was it created and so on.
Metadata Defenders • Can be used to extract useful strings for further research (C2, language settings, timestamps, etc.): – Strings, pescanner.py, Exiftool, CFF Explorer etc. • Metadata can be used to link attacks together, and is commonly used to name malware • Pescanner.py:
In Summary • OSINT is important and still gets overlooked by attackers and defenders • We hope that you found this talk useful • This talk and the Python tools mentioned will be available here shortly after the conference: – www.primalsecurity.net