# Open Source Intelligence


## Table of Contents
- [General](#general)
- [Articles/Writeups](#writeups)
- [Presentations & Talks](#talks)
- [Tools](#tools))
- [CVS/Git/Similar](#cvs)
- [DNS Stuff/related](#dns)
- [Email Gathering](#email)
- [Fancy Search Engines](#search)
- [Search Engine Dorks](#gh)
- [Site Specific Tools](#site)
- [Social Media Search/Enumeration](#social)
- [Company/People Searching](#ppl)
- [Reference Sites](#reference)
- [Miscellaneous](#misc)




#### Sort
* Add list of Sources:
	* UCC - Uniform Commercial Code; 
	* DOC - Current Industrial Patents; 
	* DMV - Vehicle Ownership applications; 
	* Patents - Patent DBs; 
	* Operating Licenses/Permits; 
	* Trade Journals;

* [keyhacks](https://github.com/streaak/keyhacks)
	* Keyhacks is a repository which shows quick ways in which API keys leaked by a bug bounty program can be checked to see if they're valid.
* [Threat Intel RSS Feeds via Twitter Lists - Joe Hopper](https://www.fracturelabs.com/posts/2018/threat-intel-rss-feeds-via-twitter-lists/)
https://ahrefs.com/blog/google-advanced-search-operators/
* [Username enumeration techniques and their value - Ben Williams](https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2015/june/username-enumeration-techniques-and-their-value/)
* [WhatsMyName](https://github.com/webbreacher/whatsmyname)
	* This repository has the unified data required to perform user and username enumeration on various websites. Content is in a JSON file and can easily be used in other projects
* [git-all-secrets](https://github.com/anshumanbh/git-all-secrets)
	* A tool to capture all the git secrets by leveraging multiple open source git searching tools
https://github.com/hisxo/gitGraber
https://posts.specterops.io/gathering-open-source-intelligence-bee58de48e05
https://github.com/intrigueio/intrigue-core
* https://www.bellingcat.com/resources/how-tos/2019/04/08/using-phone-contact-book-apps-for-digital-research/
* [How To Tell Stories: A Beginner’s Guide For Open Source Researchers - Natalia Antonova](https://www.bellingcat.com/resources/2019/07/12/how-to-tell-stories-a-beginners-guide-for-open-source-researchers/)
https://github.com/vysecurity/DomLink
https://www.icscybersecurityconference.com/intelligence-gathering-on-u-s-critical-infrastructure/
* [GitHub for Bug Bounty Hunters - Ed Overflow](https://edoverflow.com/2017/github-for-bugbountyhunters/)
* [pastebin_scraper](https://github.com/Critical-Start/pastebin_scraper)
	* https://www.criticalstart.com/2019/03/automated-tool-to-monitor-pastebin-for-interesting-information/

https://github.com/woj-ciech/kamerka
https://github.com/SourcingDenis/free-online-competitive-intelligence/blob/master/README.md
https://github.com/0days/Blue
https://github.com/digininja/leakyrepo

* [MailInt - Profiling China based Employees](https://web.archive.org/web/20180706004654/https://vincentyiu.co.uk/maiint-profiling-china-based-employees/)

* [Giggity](https://github.com/needmorecowbell/giggity)
	* Get information about an organization, user, or repo on github. Stores all data in a json file, organized in a tree of dictionaries for easy database transfer or data analysis. All done through the github api, with or without authentication (authentication highly recommended)
https://www.komodosec.com/post/github-the-red-teamer-s-cheat-sheet
https://github.com/IVMachiavelli/OSINT_Team_Links
https://www.bellingcat.com/resources/2019/07/12/how-to-tell-stories-a-beginners-guide-for-open-source-researchers/
* [asint collection - start.me](https://start.me/p/b5Aow7/asint_collection)
* [cloud_enum](https://github.com/initstring/cloud_enum)
	* Multi-cloud OSINT tool. Enumerate public resources in AWS, Azure, and Google Cloud.
* [SingleFile](https://github.com/gildas-lormeau/SingleFile)
	* SingleFile is a Web Extension compatible with Chrome, Firefox (Desktop and Mobile), Chromium-based Edge, Vivaldi, Brave, Waterfox, Yandex browser, and Opera. It helps you to save a complete web page into a single HTML file.
https://github.com/ZephrFish/GoogD0rker



https://github.com/GeneralTesler/deluxe

* [Weaponizing Corporate Intel - Mike Felch and Beau Bullock(B-Sides Orlando 2019)](https://www.youtube.com/watch?v=EfVXgvABkGg)
	* Strategically targeting a corporation requires deep knowledge of their technologies and employees. Successfully compromising an organization can depend on the quality of reconnaissance a tester performs up front. Often times testers only resort to using publicly available tools which can overlook critical assets.  In this presentation, we will begin by examining some commonly overlooked methods to discover external resources. Next, we will show how to discover employees of a target organization and quickly locate their social media accounts. Finally, we will strategically identify and weaponize personal information about the employees to target the organization directly using new attack techniques. Attendees will learn an external defense evasion method, a new process to gain credentialed access, and be the first to receive a newly released tool!  While the approach is designed to assist offensive security professionals, the presentation will be informative for technical and non-technical audiences; demonstrating the importance of security-awareness for everyone.



* [ODIN](https://github.com/chrismaddalena/ODIN)
	* ODIN aims to automate the basic recon tasks used by red teams to discover and collect data on network assets, including domains, IP addresses, and internet-facing systems. The key feature of ODIN is the data management and reporting. The data is organized in a database and then, optionally, that database can be converted into an HTML report or a Neo4j graph database for visualizing the data.
	* [Open Source Intelligence Gathering: Techniques, Automation, and Visualization - Christopher Maddalena](https://posts.specterops.io/gathering-open-source-intelligence-bee58de48e05)


Remove hidden data and personal information by inspecting documents, presentations, or workbooks
https://support.office.com/en-us/article/remove-hidden-data-and-personal-information-by-inspecting-documents-presentations-or-workbooks-356b7b5d-77af-44fe-a07f-9aa4d085966f






* [yar](https://github.com/Furduhlutur/yar)
	* yar is an OSINT tool for reconnaissance of repositories/users/organizations on Github. Yar clones repositories of users/organizations given to it and goes through the whole commit history in order of commit time, in search for secrets/tokens/passwords, essentially anything that shouldn't be there. Whenever yar finds a secret, it will print it out for you to further assess. Yar searches either by regex, entropy or both, the choice is yours. You can think of yar as a bigger and better truffleHog, it does everything that truffleHog does and more!



--------------------
### <a name="general"></a>General
* **General**
	* SWOT - Strengths, Weaknesses, Opportunities, Threats
* **101**
	* [Open Source Intelligence - Wikipedia](http://en.wikipedia.org/wiki/Open-source_intelligence)
* **Articles/Writeups**
	* [Hunting Pastebin with PasteHunter](https://techanarchy.net/2017/09/hunting-pastebin-with-pastehunter/)
	* [Open Source Intelligence Gathering 101 - appseco.com](https://blog.appsecco.com/open-source-intelligence-gathering-101-d2861d4429e3)
	* [Open Source Intelligence Gathering 201 - appseco.com](https://blog.appsecco.com/open-source-intelligence-gathering-201-covering-12-additional-techniques-b76417b5a544)
	* [Open Source Intelligence Gathering: Techniques, Automation, and Visualization - Christopher Maddalena](https://posts.specterops.io/gathering-open-source-intelligence-bee58de48e05)
	* [The OSINT Connection: Intelligence In Executive Protection - protectioncircle.com](https://protectioncircle.org/2017/03/06/the-osint-connection-intelligence-in-executive-protection/)
* **Alerting**
	* [Google Trends](https://trends.google.com/trends/)
		* See what are the popular related topics people are searching for. This will help widen your search scope.
	* [Google Alerts](https://www.google.com/alerts)
		* Google Alerts are email updates of the latest relevant Google results (web, news, etc.) based on your queries.
	* [PasteLert](https://www.andrewmohawk.com/pasteLert/)
		* PasteLert is a simple system to search pastebin.com and set up alerts (like google alerts) for pastebin.com entries. This means you will automatically recieve email whenever your term(s) is/are found in new pastebin entries!
* **Educational**
	* [Intelligence Gathering - PTES](http://www.pentest-standard.org/index.php/Intelligence_Gathering)
	* [Corporate Espionage without the Hassle of Committing Felonies](https://www.slideshare.net/JohnCABambenek/corporate-espionage-without-the-hassle-of-committing-felonies)
	* [NATO Open Source Intelligence Handbook](http://www.oss.net/dynamaster/file_archive/030201/ca5fb66734f540fbb4f8f6ef759b258c/NATO%20OSINT%20Handbook%20v1.2%20%2d%20Jan%202002.pdf)
	* [OSINT toolbag guide - pdf](http://www.phibetaiota.net/wp-content/uploads/2013/07/2013-07-11-OSINT-2ool-Kit-On-The-Go-Bag-O-Tradecraft.pdf)
	* [Intel Briefing: The Future of OSINT - Robert Munks](https://www.janes.com/article/88791/intel-briefing-the-future-of-osint)
		* This is an extract of a 60 minute live webcast available to subscribers of Jane's Intelligence Centres. In this briefing focusing on the future of open source intelligence collection, Jane's analysts will explore the following themes: A 'golden age' of open-source and social media intelligence; prospects for valuable open sources to 'go dark'; commercial satellite imagery and industry expands and future challenges for organisations conducting OSINT.
* **OSINT Based News**
	* [JustSecurity](https://www.justsecurity.org/)
		* Just Security is an online forum for the rigorous analysis of U.S. national security law and policy. We aim to promote principled and pragmatic solutions to national security problems that decision-makers face. Our Board of Editors includes individuals with significant government experience, civil society attorneys, academics, and other leading voices. Just Security is based at the Center for Human Rights and Global Justice at New York University School of Law.
	* [OSINTInsight](http://www.osintinsight.com/shared.php?user=Mediaquest&folderid=0)
	* [Janes](http://www.janes.com/)
	* [bell?ngcat](https://www.bellingcat.com/) 
		* By and for citizen investigative journalists
	* [NightWatch](http://www.kforcegov.com/Solutions/IAO/NightWatch/About.aspx)
		* NightWatch is an executive commentary and analysis of events that pose or advance threats to US national security interests. It is deliberately edgy in the interest of clarity and brevity. As a product for executives, the distribution and all feedback comments are anonymous. 
	* [RSOE EDIS - Emergency and Disaster Information Service](http://hisz.rsoe.hu/alertmap/index2.php)
* **Resources**
	* [Awesome-OSINT](https://github.com/jivoi/awesome-osint)
	* [OSINT Framework](http://osintframework.com/)
	* [OSINT Resources - greynetwork2](https://sites.google.com/site/greynetwork2/home/osint-resources)
	* [Intel Techniques - Links](http://www.inteltechniques.com/links.html)
	* [toddington - resources](https://www.toddington.com/resources/)
	* [onstrat - osint](http://www.onstrat.com/osint/)
	* http://osintinsight.com/shared.php?expand=169,175&folderid=0&user=Mediaquest
	* [Open Source Intelligence (OSINT) Tools & Resources - osint.link](http://osint.link/)
		* Seems pretty good.
	* [Midasearch.org](https://midasearch.org/)
		* [Open Source Intelligence Resources - toddington.com](https://www.toddington.com/resources/)
		* [OSINT - onstrat](http://www.onstrat.com/osint/)
	* **IntelTechniques OSINT Flowcharts**
		* [Email Address](https://inteltechniques.com/data/Email.png)
		* [Domain Name](https://inteltechniques.com/data/Domain.png)
		* [Real Name](https://inteltechniques.com/data/Real%20Name.png)
		* [Telephone #](https://inteltechniques.com/data/Telephone.png)
		* [Location](https://inteltechniques.com/data/location.png)
		* [User Name](https://inteltechniques.com/data/Username.png)
* **Writeups**
	* [Fantastic OSINT and where to find it - blindseeker/malware focused](http://archive.is/sYzcP#selection-62.0-62.1)
	* [Some blog posts describing/bringing you up to speed on OSINT by krypt3ia](http://krypt3ia.wordpress.com/2012/01/11/the-subtle-art-of-osint/)
	* [Glass Reflections in Pictures + OSINT = More Accurate Location](http://blog.ioactive.com/2014/05/glass-reflections-in-pictures-osint.html)
	* [Exploring the Github Firehose](http://blog.scalyr.com/2013/10/exploring-the-github-firehose/)
	* [OSINT Through Sender Policy Framework (SPF) Records](https://community.rapid7.com/community/infosec/blog/2015/02/23/osint-through-sender-policy-framework-spf-records)
	* [Hunting with ꓘamerka 2.0 aka FIST (Flickr, Instagram, Shodan, Twitter)](https://hackernoon.com/hunting-with-%EA%93%98amerka-2-0-aka-fist-flickr-instagram-shodan-twitter-ca363f12562a)
	* [ꓘamerka 2.0 aka FIST (Flickr, Instagram, Shodan, Twitter)](https://github.com/woj-ciech/kamerka)
		* Build interactive map of cameras, printers, tweets and photos. The script creates a map of cameras, printers, tweets and photos based on your coordinates. Everything is clearly presented in form of interactive map with icons and popups.
* **Talks & Presentations**
	* [Cognitive Bias and Critical Thinking in Open Source Intelligence - Defcamp 2014](https://www.youtube.com/watch?v=pVAM21UERLU&index=24&list=PLnwq8gv9MEKgSryzYIFhpmCcqnVzdUWfH)
	* [Dark Arts of OSINT Skydogcon](https://www.youtube.com/watch?v=062pLOoZhk8)
	* [Developing a Open Source Threat Intelligence Program—Edward McCabe](http://www.irongeek.com/i.php?page=videos/circlecitycon2014/105-developing-a-open-source-threat-intelligence-program-edward-mccabe)
		* What if you could get out in front of common threats such as botnets, scanners and malware? Good news, you can. Learn about one geeks struggle with life on the Internet of (bad) things when it comes to being online, identifying “odd” things, and developing an Open Source Threat Intelligence Program from Open Source Tools and Public Sources.
	* [Corporate Espionage: Gathering Actionable Intelligence Via Covert Operations - Brent White - Defcon22](https://www.youtube.com/watch?v=D2N6FclMMTg)
	* [How to Use Python to Spy on Your Friends: Web APIs, Recon ng, & OSINT](https://www.youtube.com/watch?v=BOjz7NfsLpA)
	* [Practical OSINT - Shane MacDougall](https://www.youtube.com/watch?v=cLmEJLy7dv8)
		*  There’s more to life to OSINT than google scraping and social media harvesting. Learn some practical methods to automate information gathering, explore some of the most useful tools, and learn how to recognize valuable data when you see it. Not only will we explore various tools, attendees will get access to unpublished transforms they can use/modify for their own use.
	* [Pwning People Personally - Josh Schwartz](https://www.youtube.com/watch?v=T2Ha-ZLZTz0)
	* [You're Leaking Trade Secrets - Defcon22 Michael Schrenk](https://www.youtube.com/watch?v=JTd5TL6_zgY)
		* Networks don't need to be hacked for information to be compromised. This is particularly true for organizations that are trying to keep trade secrets. While we hear a lot about personal privacy, little is said in regard to organizational privacy. Organizations, in fact, leak information at a much greater rate than individuals, and usually do so with little fanfare. There are greater consequences for organizations when information is leaked because the secrets often fall into the hands of competitors. This talk uses a variety of real world examples to show how trade secrets are leaked online, and how organizational privacy is compromised by seemingly innocent use of The Internet.
	* [ZOMG Its OSINT Heaven Tazz Tazz](https://www.youtube.com/watch?v=cLmEJLy7dv8)
* **OSINT Tools/Resources** <a name="tools"></a>
	* **Tools**
		* **DNS**
			* [blacksheepwall](https://github.com/tomsteele/blacksheepwall)
				* blacksheepwall is a hostname reconnaissance tool
		* **All-in-One**
			* [Maltego](https://www.paterva.com/web6/products/maltego.php)
				* Description: What you use to tie everything together.
			* [Oryon C Portable]()
				* Oryon C Portable is a web browser designed to assist researchers in conducting Open Source Intelligence investigations. Oryon comes with dozens of pre-installed tools and a select set of links cataloged by category – including those that can be found in the OI Shared Resources.
			* [OSINT Mantra](http://www.getmantra.com/hackery/osint.html)
			* [Recon-ng](https://bitbucket.org/LaNMaSteR53/recon-ng)
				* Description: Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly.
			* [TouchGraph SEO Browser](http://www.touchgraph.com/seo)
				* Use this free Java application to explore the connections between related websites.
			* [Th3inspector](https://github.com/Moham3dRiahi/Th3inspector)
				* Tool that automates OSINT collection. Seems to gather from a variety of sources. Perl script.
			* [gasmask](https://github.com/twelvesec/gasmask)
				* All in one Information gathering tool - OSINT
		* **Certificate Transparency**
			* [ct-exposer](https://github.com/chris408/ct-exposer)
			    * An OSINT tool that discovers sub-domains by searching Certificate Transparency logs. Certificate Transparency (CT) is an experimental IETF standard. The goal of it was to allow the public to audit which certificates were created by Certificate Authorities (CA). TLS has a weakness that comes from the large list of CAs that your browser implicitly trusts. If any of those CAs were to maliciously create a new certificate for a domain, your browser would trust it. CT adds benefits to TLS certificate trust: Companies can monitor who is creating certificates for the domains they own. It also allows browsers to verify that the certificate for a given domain is in the public log record. These logs end up being a gold mine of information for penetration testers and red teams.
		* **Data Manipulation**
			* [Danger-zone](https://github.com/woj-ciech/Danger-zone/blob/master/README.md)
				* Correlate data between domains, ips and email addresses, present it as a graph and store everything into Elasticsearch and JSON files.
				* [Article](https://medium.com/@woj_ciech/osint-tool-for-visualizing-relationships-between-domains-ips-and-email-addresses-94377aa1f20a)
			* [OpenRefine](https://github.com/OpenRefine/OpenRefine)	
				* Description: OpenRefine is a power tool that allows you to load data, understand it, clean it up, reconcile it to master database, and augment it with data coming from Freebase or other web sources. All with the comfort and privacy of your own computer.
			* [OSRFramework](https://github.com/i3visio/osrframework)
				* OSRFramework is a GNU AGPLv3+ set of libraries developed by i3visio to perform Open Source Intelligence tasks. They include references to a bunch of different applications related to username checking, DNS lookups, information leaks research, deep web search, regular expressions extraction and many others. At the same time, by means of ad-hoc Maltego transforms, OSRFramework provides a way of making these queries graphically as well as several interfaces to interact with like OSRFConsole or a Web interface.
		* **Geolocation**
			* [Creepy.py](http://ilektrojohn.github.io/creepy/)
				* Description: Creepy is a geolocation OSINT tool. Gathers geolocation related information from online sources, and allows for presentation on map, search filtering based on exact location and/or date, export in csv format or kml for further analysis in Google Maps.
		* **Research Collection/Organization**
			* [hunch.ly](https://hunch.ly/)
				* Paid web archiving tool
			* [zotero.org](https://www.zotero.org/)
				* Research Collection/Organization Tool
		* **Search Engine**
			* [shodan.io](https://www.shodan.io/)
				* Description: DescriptionShodan is a search engine that lets the user find specific types of computers connected to the internet using a variety of filters.		
* **Company/People Searching** <a name="ppl"></a>
	* [data.com](https://www.data.com/)
	* [LittleSis](https://littlesis.org/)
		* LittleSis is a free database of who-knows-who at the heights of business and government.
	* [Jigsaw](http://jigsawbusinessgroup.com/what-we-do/people/)
		* Jigsaw is a prospecting tool used by sales professionals, marketers and recruiters to get fresh and accurate sales leads and business contact information.
	* [Spokeo](https://www.spokeo.com/)
		* Spokeo is a people search engine that organizes white pages listings, public records and social network information into simple profiles to help you safely find and learn about people.\
	* [Hoovers](http://www.hoovers.com/)
		* Search over 85 million companies within 900 industry segments; Hoover's Reports Easy-to-read reports on key competitors, financials, and executives
	* [Market Visual](http://www.marketvisual.com/)
		* Search Professionals by Name, Company or Title
	* [Glass Door](https://www.glassdoor.com/)
		* Search jobs then look inside. Company salaries, reviews, interview questions, and more all posted anonymously by employees and job seekers.
	* [192](http://www.192.com/)
		* Find people, businesses and places in the UK with 192.com. Directory enquiries, a people finder, business listings and detailed maps with aerial photos.
	* [corporationwiki](https://www.corporationwiki.com/)
	* [orbis](https://orbisdirectory.bvdinfo.com/version-2017821/OrbisDirectory/Companies)
		* Company information across the globe
* **Country Specific Resources**
	* **USA**
		* [SEC EDGAR Search](https://www.sec.gov/edgar/searchedgar/webusers.htm)
		* [US Congressional Research Service - crsreports.congress.gov](https://crsreports.congress.gov/search/#/?termsToSearch=&orderBy=Date)
* **CVS/Git/Similar Focused** <a name="cvs"></a>
	* [repo-supervisor](https://github.com/auth0/repo-supervisor)
	* [GitPrey](https://github.com/repoog/GitPrey)
		* GitPrey is a tool for searching sensitive information or data according to company name or key word something.The design mind is from searching sensitive data leakling in Github:
	* [git-all-secrets](https://github.com/anshumanbh/git-all-secrets)
		* A tool to capture all the git secrets by leveraging multiple open source git searching tools
	* [github-firehose](https://www.npmjs.com/package/github-firehose)
		* A library that will connect to github and emit events from the Github Event API in near-real-time
		* [Exploring the Github Firehose](http://blog.scalyr.com/2013/10/exploring-the-github-firehose/)
	* [Gitem](https://github.com/mschwager/gitem)
		* Gitem is a tool for performing Github organizational reconnaissance.
	* [Truffle Hog](https://github.com/dxa4481/truffleHog)
		* Searches through git repositories for high entropy strings, digging deep into commit history and branches. This is effective at finding secrets accidentally committed that contain high entropy.
	* [dvcs-ripper](https://github.com/kost/dvcs-ripper)
		* Rip web accessible (distributed) version control systems: SVN, GIT, Mercurial/hg, bzr, ... It can rip repositories even when directory browsing is turned off.
	* [Truffle Hog](https://github.com/dxa4481/truffleHog)
		* Searches through git repositories for high entropy strings, digging deep into commit history
	* [DVCS-Pillage](https://github.com/evilpacket/DVCS-Pillage)
		* Pillage web accessible GIT, HG and BZR repositories. I thought it would be useful to automate some other techniques I found to extract code, configs and other information from a git,hg, and bzr repo's identified in a web root that was not 100% cloneable. Each script extracts as much knowledge about the repo as possible through predictable file names and known object hashes, etc.
	* [gitdigger](https://github.com/wick2o/gitDigger)
		* gitDigger: Creating realworld wordlists from github hosted data.
	* [gitrob](https://github.com/michenriksen/gitrob)
		* Gitrob is a command line tool which can help organizations and security professionals find sensitive information lingering in publicly available files on GitHub. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information. Looking for sensitive information in GitHub repositories is not a new thing, it has been [known for a while](http://blog.conviso.com.br/2013/06/github-hacking-for-fun-and-sensitive.html) that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.
	* [reposcanner](https://github.com/Dionach/reposcanner)
		* Python script to scan Git repos for interesting strings
	* [gitleaks](https://github.com/zricethezav/gitleaks)
		*  Searches full repo history for secrets and keys
	* [Reposcanner](https://github.com/Dionach/reposcanner)
		* Reposcanner is a python script to search through the commit history of Git repositories looking for interesting strings such as API keys, inspired by truffleHog.
* **DNS Stuff** <a name="dns"></a>
	* [dauntless](https://github.com/cmeister2/dauntless)
		* Tools for analysing the forward DNS data set published at https://scans.io/study/sonar.fdns_v2
	* [dnstwist](https://github.com/elceef/dnstwist)
		* Domain name permutation engine for detecting typo squatting, phishing and corporate espionage
	* [typofinder](https://github.com/nccgroup/typofinder)
		* Typofinder for domain typo discovery
* **Domain Recon**
	* **Tools**
		* [Waybackpack](https://github.com/jsvine/waybackpack)
			* Waybackpack is a command-line tool that lets you download the entire Wayback Machine archive for a given URL.
		* [domain - jhaddix](https://github.com/jhaddix/domain)
			* Recon-ng and Alt-DNS are awesome. This script combines the power of these tools with the ability to run multiple domains within the same session.  TLDR; I just want to do my subdomain discovery via ONE command and be done with it. Only 1 module needs an api key (/api/google_site) find instructions for that on the recon-ng wiki. Script to enumerate subdomains, leveraging recon-ng. Uses google scraping, bing scraping, baidu scraping, yahoo scraping, netcraft, and bruteforces to find subdomains. Plus resolves to IP
		* [check0365](https://github.com/vysecurity/checkO365)
			* checkO365 is a tool to check if a target domain is using O365
* **Email Gathering/Reconnaissance** <a name="email"></a>
	* **Articles/Writeups**
		* [OSINT Through Sender Policy Framework Records](https://community.rapid7.com/community/infosec/blog/2015/02/23/osint-through-sender-policy-framework-spf-records)
		* [The most complete guide to finding anyone’s email - Timur Daudpota](https://www.blurbiz.io/blog/the-most-complete-guide-to-finding-anyones-email)
	* **Tools**
		* [SimplyEmail](https://github.com/killswitch-GUI/SimplyEmail)
			* What is the simple email recon tool? This tool was based off the work of theHarvester and kind of a port of the functionality. This was just an expansion of what was used to build theHarvester and will incorporate his work but allow users to easily build Modules for the Framework. Which I felt was desperately needed after building my first module for theHarvester.
		* [Email Reconnaissance and Phishing Template Generation Made Simple](https://cybersyndicates.com/2016/05/email-reconnaissance-phishing-template-generation-made-simple/)
		* [theHarvester](https://github.com/laramies/theHarvester)
			* theHarvester is a tool for gathering e-mail accounts, subdomain names, virtual hosts, open ports/ banners, and employee names from different public sources (search engines, pgp key servers).
		* [discover.sh](https://github.com/leebaird/discover)
			* For use with Kali Linux. Custom bash scripts used to automate various pentesting tasks.
		* [Cr3dOv3r](https://github.com/D4Vinci/Cr3dOv3r)
			* Cr3dOv3r simply you give it an email then it does two simple jobs (but useful): Search for public leaks for the email and if it any, it returns with all available details about the leak (Using hacked-emails site API). Now you give it this email's old or leaked password then it checks this credentials against 16 websites (ex: facebook, twitter, google...) then it tells you if login successful in any website!
		* [Infoga](https://github.com/m4ll0k/Infoga)
			* Infoga is a tool gathering email accounts informations (ip,hostname,country,...) from different public source (search engines, pgp key servers and shodan) and check if emails was leaked using haveibeenpwned.com API. Is a really simple tool, but very effective for the early stages of a penetration test or just to know the visibility of your company in the Internet.
* **Facial Mapping Data**
	* [Social Mapper](https://github.com/SpiderLabs/social_mapper)
		* Social Mapper is a Open Source Intelligence Tool that uses facial recognition to correlate social media profiles across different sites on a large scale. It takes an automated approach to searching popular social media sites for targets names and pictures to accurately detect and group a person’s presence, outputting the results into report that a human operator can quickly review.
* **Fancy Search Engines** <a name="search"></a>
	* [Entity Cube](http://entitycube.research.microsoft.com/) 
		* EntityCube is a research prototype for exploring object-level search technologies, which automatically summarizes the Web for entities (such as people, locations and organizations) with a modest web presence.
	* [Silobreaker](http://www.silobreaker.com/)
		* Enterprise Semantic Search Engine, allows virtualisation of data, analytics and exploration of key data.
	* [iSeek](http://www.iseek.com/#/web)
		* Another handy search engine that break results down into easy to manage categories.
	* [Carrot2](http://search.carrot2.org/stable/search)
		*  Carrot2 organizes your search results into topics. With an instant overview of what's available, you will quickly find what you're looking for. 
	* [Sqoop](http://sqoop.com/)
		* OSINT search engine of public documents(handy)
	* [GlobalFileSearch](https://ftplike.com)
		* An FTP Search Engine that may come in handy.
	* [NAPALM FTP Indexer](https://www.searchftps.net/)
* **General Meta Data** <a name="meta"></a>
	* [Just-Metadata](https://github.com/ChrisTruncer/Just-Metadata)
		* Just-Metadata is a tool that can be used to gather intelligence information passively about a large number of IP addresses, and attempt to extrapolate relationships that might not otherwise be seen. Just-Metadata has "gather" modules which are used to gather metadata about IPs loaded into the framework across multiple resources on the internet. Just-Metadata also has "analysis" modules. These are used to analyze the data loaded Just-Metadata and perform various operations that can identify potential relationships between the loaded systems.
	* [MetaGooFil](https://code.google.com/p/metagoofil/)	
		* Description: Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target company. The tool will perform a search in Google to identify and download the documents to local disk and then will extract the metadata with different libraries like Hachoir, PdfMiner and others. With the results it will generate a report with usernames, software versions and servers or machine names that will help Penetration testers in the information gathering phase.
	* [Metashield Analyzer](https://metashieldanalyzer.elevenpaths.com/)
		* Description: Metadata documents can help a malicious user to obtain information that is beyond our control in an enterprise environment. Metashield Analyzer is an online service that allows easily check if your office documents contain metadata.
	* [PowerMeta](https://github.com/dafthack/PowerMeta)
		* PowerMeta searches for publicly available files hosted on various websites for a particular domain by using specially crafted Google, and Bing searches. It then allows for the download of those files from the target domain. After retrieving the files, the metadata associated with them can be analyzed by PowerMeta. Some interesting things commonly found in metadata are usernames, domains, software titles, and computer names.
* **General Data Scrapers** <a name="scrape"></a>
	* [XRAY](https://github.com/evilsocket/xray)
		* XRay is a tool for recon, mapping and OSINT gathering from public networks.
	* [NameCheck](https://www.namecheck.com)
		* Search usernames across multiple services/domain registries
	* [TheHarvester](From: https://code.google.com/p/theharvester/)
		* Description: The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database. This tool is intended to help Penetration testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization. 
	* [OSINT OPSEC Tool](https://github.com/hyprwired/osint-opsec-tool)
		* Description: The OSINT OPSEC Tool monitors multiple 21st Century OSINT sources real-time for keywords, then analyses the results, generates alerts, and maps trends of the data, finding all sorts of info people probably don't want others to see... 
	* [Pattern](https://github.com/clips/pattern/blob/master/README.md)
		* Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google,; Twitter, Wikipedia), web crawler, HTML DOM parser; Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet; Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron); Network Analysis: graph centrality and visualization.
	* **Paste-Site Scrapers**
		* [sniff-paste](https://github.com/needmorecowbell/sniff-paste)
	    	* Multithreaded pastebin scraper, scrapes to mysql database, then reads pastes for noteworthy information.
* **Search Engine Dorks** <a name="gh"></a>
	* **101**
		* [Google Hacking for Penetration Testers](https://www.blackhat.com/presentations/bh-europe-05/BH_EU_05-Long.pdf)
		* [How to Find (Almost) Anything on Google - Barbara Davidson](https://www.netcredit.com/blog/how-to-find-anything-on-google/)
	* **Databases/Lists**
		* [ExpoitDB archive of the google hacking database](http://www.exploit-db.com/google-dorks/)
		* [Google Hacking Database](http://www.hackersforcharity.org/ghdb/)
			* We call them 'googledorks': Inept or foolish people as revealed by Google. Whatever you call these fools, you've found the center of the Google Hacking Universe! 
		* [4500+ Google Dork List 2018 - conzu.de](http://www.conzu.de/en/google-dork-liste-2018-conzu/)
		* [List of Google ASE Queries/Dorks - @payloadartist](https://pastebin.com/zYPZNbMK)
	* **Tools**
		* [GooHak](https://github.com/1N3/Goohak)
			* Automatically launch google hacking queries against a target domain to find vulnerabilities and enumerate a target.
		* [Google Hacking - Search Diggity tool](http://www.bishopfox.com/resources/tools/google-hacking-diggity/attack-tools/)
			* SearchDiggity 3.1 is the primary attack tool of the Google Hacking Diggity Project. It is Bishop Fox’s MS Windows GUI application that serves as a front-end to the most recent versions of our Diggity tools: GoogleDiggity, BingDiggity, Bing LinkFromDomainDiggity, CodeSearchDiggity, DLPDiggity, FlashDiggity, MalwareDiggity, PortScanDiggity, SHODANDiggity, BingBinaryMalwareSearch, and NotInMyBackYard Diggity.
		* [GoogD0rker](https://github.com/ZephrFish/GoogD0rker)
			* GoogD0rker is a tool for firing off google dorks against a target domain, it is purely for OSINT against a specific target domain. Designed for OSX originally however googD0rker txt now works on all nix platforms.
* **Network Information Search Engines** <a name="nin"></a>
	* [Whoisology](https://whoisology.com/)
		* Whoisology is a domain name ownership archive with literally billions of searchable and cross referenced domain name whois records. 
* **Site Specific** <a name="site"></a>
	* **AWS**
		* [AWSBucketDump](https://github.com/jordanpotti/AWSBucketDump)
			* AWSBucketDump is a tool to quickly enumerate AWS S3 buckets to look for loot. It's similar to a subdomain bruteforcer but is made specifically for S3 buckets and also has some extra features that allow you to grep for delicious files as well as download interesting files if you're not afraid to quickly fill up your hard drive.
	* **Facebook**
		* [pymk-inspector](https://github.com/GMG-Special-Projects-Desk/pymk-inspector/blob/master/README.md)
			* The pymk-inspector is a tool built by Gizmodo's Special Projects Desk that we used for our investigation into Facebook's people you may know (pymk) algorithm.
		* [Find FB profiles by Email](https://booleanstrings.com/2018/05/06/how-to-identify-facebook-profiles-from-email-addresses/)
	* **Github**
		* [profile-summary-for-github](https://github.com/tipsy/profile-summary-for-github)
			* Tool for visualizing GitHub profiles
		* [Github dorks - finding vulns](http://blog.conviso.com.br/2013/06/github-hacking-for-fun-and-sensitive.html)
	* **LinkedIn**
		* [InSpy](https://github.com/gojhonny/InSpy)
			* A LinkedIn enumeration tool
		* [linkedin](https://github.com/eracle/linkedin)
			* Linkedin Scraper using Selenium Web Driver, Firefox 45, Ubuntu and Scrapy
		* [LinkedInt: A LinkedIn scraper for reconnaissance during adversary simulation](https://github.com/mdsecactivebreach/LinkedInt)
		* [LinkedIn Gatherer](https://github.com/DisK0nn3cT/linkedin-gatherer)
		* [socilab](http://socilab.com/#home)
			* This site allows users to visualize and analyze their LinkedIn network using methods derived from social-scientific research. Full sample output is shown here. The site is free and open-source. Have fun!
		* [Linkedin_profiles](https://github.com/wpentester/Linkedin_profiles)
			* This script uses selenium to scrape linkedin employee details from a specified company. If the script isn't working, you can always browse to the desired company's employee page and paste in the link on line 69 like this: "employees_page = url"
		* [The Secrets of LinkedIn](https://webbreacher.com/2017/01/14/the-secrets-of-linkedin/)
			* Grabbing usernames/connections(link analysis)
		* [The Endorser](https://github.com/eth0izzle/the-endorser)
			* An OSINT tool that allows you to draw out relationships between people on LinkedIn via endorsements/skills.
		* [ScrapedIn](https://github.com/dchrastil/ScrapedIn)
			* this tool assists in performing reconnaissance using the LinkedIn.com website/API. Provide a search string just as you would on the original website and let ScrapedIn do all the dirty work. Output is stored as an XLSX file, however it is intended to be used with Google Spreadsheets. After importing the XLSX into Google Spreadsheets there will be a "dataset" worksheet and a "report" worksheet.
		* [Gathering Usernames from Google LinkedIn Results Using Burp Suite Pro - BHIS](https://www.blackhillsinfosec.com/gathering-usernames-from-google-linkedin-results-using-burp-suite-pro/)
		* [GatherContacts](https://github.com/clr2of8/GatherContacts)
			* A Burp Suite Extension to pull Employee Names from Google and Bing LinkedIn Search Results.
		* [linkedin2username](https://github.com/initstring/linkedin2username)
		* [Raven](https://github.com/0x09AL/raven)
			* raven is a Linkedin information gathering tool that can be used by pentesters to gather information about an organization employees using Linkedin.
	* **Tinder**
		* [OSINT: Advanced tinder capture](https://www.learnallthethings.net/osmosis)
	* **Twitter**
		* [OneMillionTweetMap](http://onemilliontweetmap.com/)
			* This page maps the last geolocalized tweets delivered by the twitter stream API. ... YES - IN REAL-TIME - and we keep "only" the last one million tweets.
		* [tweets_analyzer](https://github.com/x0rz/tweets_analyzer)
			* Tweets metadata scraper & activity analyzer
		* [Tweet Archivist](https://www.tweetarchivist.com/)
		* [tweets_analyzer](https://github.com/x0rz/tweets_analyzer)
			* Tweets metadata scraper & activity analyzer
		* [Tinfoleak](http://vicenteaguileradiaz.com/tools/)
			* tinfoleak is a simple Python script that allow to obtain: basic information about a Twitter user (name, picture, location, followers, etc.); devices and operating systems used by the Twitter user; applications and social networks used by the Twitter user; place and geolocation coordinates to generate a tracking map of locations visited; show user tweets in Google Earth!; download all pics from a Twitter user; hashtags used by the Twitter user and when are used (date and time); user mentions by the the Twitter user and when are occurred (date and time); topics used by the Twitter user
		* [How to Find the Twitter ID from an Email Address - booleanstrings.com](https://booleanstrings.com/2018/05/02/how-to-find-the-twitter-id-from-an-email-address/)
		* [Twint](https://github.com/twintproject/twint)
			* Formerly known as Tweep, Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.  Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too.  Twint also makes special queries to Twitter allowing you to also scrape a Twitter user's followers, Tweets a user has liked, and who they follow without any authentication, API, Selenium, or browser emulation.
		* [twitterBFTD](https://github.com/misterch0c/twitterBFTD)
    		* Twitter back from the death looks in a user tweets history for domain names that are available for registration.
		    * [Blogpost](https://hackernoon.com/how-i-hijacked-top-celebrities-tweets-including-katy-perry-shakira-fca3a0e751c6)
* **Social Media Search/Enumeration** <a name="social"></a>
	* [CheckUsernames](http://checkusernames.com/)
		* Check the use of your brand or username on 160 Social Networks
	* [NameCHK](https://namechk.com/)
		* Check to see if your desired username or vanity url is still available at dozens of popular Social Networking and Social Bookmarking websites.
	* [Scythe](https://github.com/ChrisJohnRiley/Scythe)
		* The ability to test a range of email addresses across a range of sites (e.g. social media, blogging platforms, etc...) to find where those targets have active accounts. This can be useful in a social engineering test where you have email accounts for a company and want to list where these users have used their work email for 3rd party web based services.
	* [Social Mention](http://www.socialmention.com/)
		* Social Mention is a social media search engine that searches user-generated content such as blogs, comments, bookmarks, events, news, videos, and more
	* [Whos Talkin](http://www.whostalkin.com/)
		* social media search tool that allows users to search for conversations surrounding the topics that they care about most.
	* [sherlock-js](https://github.com/GitSquared/sherlock-js)
		* Find usernames across over 75 social networks - NodeJS remake of sdushantha/sherlock
	* [sherlock](https://github.com/sherlock-project/sherlock)
		* Python tool to find usernames across social networks
* **Tor**
	* [ExoneraTor](https://metrics.torproject.org/exonerator.html)
		* Enter an IP address and date to find out whether that address was used as a Tor relay:	





