How to make a Simple Website Crawler

“If you spend more on coffee than on IT security, you will be hacked. What’s more, you deserve to be hacked”

― Richard Clarke

import requests
def response(URL):
try:
return requests.get("https://" + URL)

except requests.exceptions.ConnectionError:
pass
except requests.exceptions.InvalidURL:
pass
with open("/root/Downloads/subdomains.list","r") as wordlist:
for line in wordlist:
word = line.strip()
test_URL = word + "." + target_URL
res = response(test_URL)
if res:
print("[+] Discovered a new subdomain -->" + test_URL)

--

--

Wannbe Security Researcher

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store