16 Sep OSINT and top 15 Open-source intelligence tools
What is OSINT?
OSINT is an acronym for open-source intelligence and forms one of the key concepts in building a robust cybersecurity system. OSINT is the practice of collecting information from already published sources or public sources available on the internet. The OSINT operation process performed by IT operatives, malicious actors, or sanctioned intelligence operatives uses advanced search techniques that are publicly available to gather information. Open source in OSINT doesn’t refer to open-source software movement rather points to the public nature of the data, which is freely available on the internet. Collating data helps in many ways, such as building a robust cybersecurity system by reducing your attack surface and securing information available publicly. It also helps you gain a competitive advantage and get a jump start on your competitors. Simple OSINT examples include:
- Asking questions on any search engine.
- Research public forums on the latest mobile technologies.
- Watch a YouTube video on how to make a certain delicacy.
Importance of OSINT
OSINT, in general, helps an organization keep tabs on the public information. It also helps in reducing the potential attack surface and thus prevent breaches and leaks. For example, the following tasks are done with the help of OSINT.
Discovering and locating the assets outside the perimeter or public: OSINT helps the IT and the cybersecurity teams discover and locate public-facing assets. Through OSINT, the information available in each asset can then be mapped and assessed for sensitive or critical information that can be exploited. In general, the OSINT tools help in mapping and recording data of the public assets of the company that is publicly available and accessible.
Finding relevant data and information outside the organization: The OSINT tools help find relevant data outside the organization, such as domains or ports outside the organization’s network perimeter. This function is particularly helpful for an organization that has recently merged or acquired another organization as it helps find relevant information available outside of the organization just acquired.
Take necessary measures with the collated data: The data collected can be massive and not in order. OSINT tools convert the data into meaningful information that can be used as actionable intelligence. OSINT tools also help piecing the data together and dealing with sensitive data and their problems on a priority basis.
OSINT techniques
While there are a lot of OSINT methodologies and mechanisms available, not all of them will help you achieve your target. So first, you need to define the scope of the search and ask the following questions:
- What are you looking for?
- What is your main research objective?
- Who is your main target?
- What tools or mechanisms will you undertake to conduct the research?
OSINT techniques can be divided into two major categories, namely Active OSINT and Passive OSINT
Active OSINT: Includes port and system scanning and direct contact with the target. The results are more reliable and dependable, along with a high risk of detection.
Passive OSINT: In this category, the contact is established with the help of third-party services. Since it includes a third party, the search results may not be reliable and may include many false positives and negatives. Therefore, the risk of detection is quite low in this category.
Read Also: ZERO TRUST SECURITY – WHAT YOU NEED TO KNOW?
Challenges in performing OSINT
Certain risks that are involved with OSINT tools are:
Getting detected: This is the most common risk involved as performing an OSINT investigation may give your information away as the one who was searching for the data.
Losing access to the information: Getting detected may result in you losing access to the information as it may lead to securing publicly available information or hiding the trails.
You become the victim: If your cover is blown, you can risk becoming a target of an investigation or, even worse, spying.
Content filtering: OSINT performs an exponential search and collates a humongous amount of data. If the data is not pieced together or in order, the data collected is useless and doesn’t result in any meaningful action.
Top OSINT tools
To counter the challenges in performing OSINT, the following tools help with the collection of information and data:
BuiltWith: As the name suggests, BuiltWith lets you decode or find out what the websites are built with or made up of. It enables the user to identify different tech stacks and platforms that power the websites. For instance, BuiltWith can identify whether the website is built using Joomla, WordPress or Drupal as its CMS. It also identifies and generates a list of Java scripts/CSS libraries, website plugins, website framework and server information. BuiltWith can be used as preliminary research or an observation tool for websites.
Maltego: Maltego is primarily used for uncovering relationships among domains and publicly accessible information. It also helps in charting the humongous data into readable and easy-to-understand charts and graphs, which helps convert the raw data into some actionable measures. Maltego comes with 58 data integrations from over 35 data partners that allow users to choose four different layouts to recognize patterns in the data they’ve uncovered and piece complex data together.
Mitaka: Mitaka is available as a Chrome extension and as a Firefox browser adds on that helps in searching IP addresses, URL’s, domains, hashes and wallet addresses across six dozen search engines. It also helps the cyber security team recognize and detect various indicators of compromise from your web browser and helps mitigate threats and risks. Additionally, as they are extensions, the online databases can be quickly queried with just a click.
Spyse: Spyse is considered the complete internet assets registry and is used to collect the data on servers, websites and peripheral connected devices which are publicly available. This is also used as a reconnaissance tool that conducts data analysis to detect any security vulnerabilities of unmanaged assets and also helps in securing exposed credentials.
Spiderfoot: Spiderfoot is a free OSINT reconnaissance tool that integrates with different multiple data sources and automates the collection of OSINT. Spiderfoot gathers and analyses data regarding domains, IP addresses, CIDR ranges, phone numbers, usernames, and other sensitive data. Providing an intuitive web-based GUI, Spiderfoot contains both a command-line interface and an embedded web server, making it ideal for red team reconnaissance activities. In general, Spiderfoot helps discover more information about your target or identify what your organization may be inadvertently exposed to the public.
OSINT framework: The OSINT framework doesn’t run on the servers but is a web-based interface that is useful in gaining valuable information and data by querying free search engines, resources and tools and helps you sniff out the data you need by breaking down the different topics of interest. The querying of websites for data extraction is free, while some require registration and have paid versions that help you construct an advanced google search and collect in-depth data.
Creepy: Creepy is a tool written in python that helps in collecting geolocation data of any individuals through a query raised through social networking platforms and hosting services. Creepy enables the user to present or plot the data collected on a map. It also allows the users to download or filter the data. CSV or .KML format.
Recon-ng: Recon-ng is a tool written in python that primarily focuses on web-based web-based open-source reconnaissance. It includes many modules, interactive help and convenience functions that guide users to use the tool correctly. Recon-ng automates activities like cutting pasting and harvesting, which are time-consuming and also perform operations like database interaction, performing web requests and managing API keys.
Shodan: Shodan is a type of network security monitor and a dedicated search engine used to find data and intelligence on the internet of things (IoT). This tool is also known as the search engine of hackers as it helps you find and explore different devices connected to a network. It also helps detect and find open ports and vulnerabilities on the attack surface. Shodan is of specific interest for IT professionals as it gives information and details about HTTP, SSP, SNMP, RTSP, which are based on operating systems, country, network and ports. Along with the IoT devices, Shodan can also query databases and find data publicly accessible through paths other than the main interface.
theHarvester: theHarvester is one of the simplest tools to capture and access public information outside an organization’s network perimeter. It brings back valuable information about virtual hosts, subdomain names, email addresses and open ports of any organization. This tool is very helpful in determining the scope of the pen test and helps as a reconnaissance step before pen-testing. theHarvester uses popular search engines like Google, Duck Duck Go, Bing and social media networks to collect OSINT.
Metagoofil: As the name suggests, Metagoofil is used to extract metadata from public documents that also include PDF’s and Microsoft office files. It finds the target document and stores it on a local disk and maps the paths used to get the documents. This helps obtain directory tree information, shared resources and server names of the host organization. This is a perfect tool for hackers to gather information and launch brute force attacks on the target system. This tool also helps cyber security professionals determine the vulnerabilities and helps secure the network by closing the gaps before the hacker exploits these vulnerabilities.
Censys: This is a wonderful tool that acts as a search engine to get information about any device or network system connected to the internet. They can also return information on servers and domain names. In addition, you can find geoinformation and technical details about 80 and 443 ports running on a server, HTTP mapping of the target website, SSL certificate information, TLS handshake information, and WHOIS information.
TinEye: TInEye is an image search and image recognition tool that mainly focuses on reverse image searches that helps moderate the content that is posted on the web and is available for access through public domains. It can detect instances of fraud and copyright occurring through images pattern recognition and track the location of these images online among the constantly growing index of billion images online
OpenVAS: Open vulnerability assessment is a security framework that includes a vulnerability scanner for IT professionals to detect threats and vulnerabilities in a system. It is used for authenticated and unauthenticated testing, performance tuning for scans, high-level industrial level protocols and a powerful internal programming language to carry out vulnerability tests from a continuous daily feed. It enforces security by enabling continuous monitoring of networks systems and applications for threats and vulnerabilities.
Searchcode: Searchcode is a unique and dedicated search engine that searches the code repository for any intelligence inside free source code. Works like any other normal search engine, but instead of searching for indexed web servers, searchcode searches for information in the code repositories of running apps or apps in the developing stage. It is completely free, and its filters make it easy for the users to sort data by language, repository or phrase. It is a good OSINT tool since it gathers information from the accessible source codes and checks for sensitive information. Searchcode Is a good tool to have when the apps are in the developmental stage and can be used as a reconnaissance tool before the deployment stage.
