Search engines are a treasure trove of valuable sensitive information, which hackers can use for their cyber-attacks. Good news: so can penetration testers.
From a penetration tester’s point of view, all search engines can be largely divided into pen test-specific and commonly-used. The article will cover three search engines that my counterparts and I widely use as penetration testing tools. These are Google (the commonly-used) and two pen test-specific ones: Shodan and Censys.
Google
Penetration testing engineers employ Google advanced search operators for Google dork queries (or simply Google dorks). These are search strings with the following syntax: operator:search term. Further, you’ll find the list of the most useful operators for pen testers:
- cache: provides access to cached pages. If a pen tester is looking for a certain login page and it is cached, the specialist can use cache: operator to steal user credentials with a web proxy.
- filetype: limits the search result to specific file types.
- allintitle: and intitle: both deal with HTML page titles. allintitle: finds pages that have all of the search terms in the page title. intitle: restricts results to those containing at least some of the search terms in the page title. The remaining terms should appear somewhere in the body of the page.
- allinurl: and inurl: apply the same principle to the page URL.
- site: returns results from a website located on a specified domain.
- related: allows finding other pages similar in linkage patterns to the given URL.
What can be found with Google advanced search operators?
Google advanced search operators are used along with other penetration testing tools for anonymous information gathering, network mapping, as well as port scanning and enumeration. Google dorks can provide a pen tester with a wide array of sensitive information, such as admin login pages, usernames and passwords, sensitive documents, military or government data, corporate mailing lists, bank account details, etc.
Shodan
Shodan is a pen test-specific search engine that helps a penetration tester to find specific nodes (routers, switches, desktops, servers, etc.). The search engine interrogates ports, grabs the resulting banners and indexes them to find the required information. The value of Shodan as a penetration testing tool is that it provides a number of convenient filters:
- country: narrows the search by a two-letter country code. For example, the request apache country:NO will show you apache servers in Norway.
- hostname: filters results by any portion of a hostname or a domain name. For example, apache hostname:.org finds apache servers in the .org domain.
- net: filters results by a particular IP range or subnet.
- os: finds specified operating systems.
- port: searches for specific services. Shodan has a limited collection of ports: 21 (FTP), 22 (SSH), 23 (Telnet) and 80 (HTTP). However, you can send a request to the search engine’s developer John Matherly via Twitter for more ports and services.
Shodan is a commercial project and, although authorization isn’t required, logged-in users have privileges. For a monthly fee you’ll get an extended number of query credits, the ability to use country: and net: filters, save and share searches, as well as export results in XML format.
Censys
Another useful penetration testing tool is Censys – a pen test-specific open-source search engine. Its creators claim that the engine encapsulates a “complete database of everything on the Internet.” Censys scans the internet and provides a pen tester with three data sets of hosts on the public IPv4 address space, websites in the Alexa top million domains and X.509 cryptographic certificates.
Censys supports a full text search (For example, certificate has expired query will provide a pen tester with a list of all devices with expired certificates.) and regular expressions (For example, metadata. Manufacturer: “Cisco” query shows all active Cisco devices. Lots of them will surely have unpatched routers with known vulnerabilities.). A more detailed description of the Censys search syntax is given here.
Shodan vs. Censys
As penetration testing tools, both search engines are employed to scan the internet for vulnerable systems. Still, I see the difference between them in the usage policy and the presentation of search results.
Shodan doesn’t require any proof of a user’s noble intentions, but one should pay to use it. At the same time, Censys is open-source, but it requires a CEH certificate or other document proving the ethics of a user’s intentions to lift substantial usage limitations (access to additional features, a query limit (five per day) from one IP address).
Shodan and Censys present search results differently. Shodan does it in a more convenient for users form (resembles Google SERP), Censys – as raw data or in JSON format. The latter is more suitable for parsers, which then present the information in a more readable form.
Some security researchers claim that Censys offers better IPv4 address space coverage and fresher results. Yet, Shodan performs a way more detailed internet scanning and gives cleaner results.
So, which one to use? To my mind, if you want some recent statistics – choose Censys. For daily pen testing purposes – Shodan is the right pick.
On a final note
Google, Shodan and Censys are well worth adding to your penetration testing tool arsenal. I recommend using all the three, as each contributes its part to a thorough information gathering.
Certified Ethical Hacker at ScienceSoft with 5 years of experience in penetration testing. Uladzislau’s spheres of competence include reverse engineering, black box, white box and gray box penetration testing of web and mobile applications, bug hunting and research work in the area of information security.