Directories/URLs gathering
getallurls/gau, hawkrawler, gobuster, wfuzz, dirb, dirbuster, ffuf
Last updated
getallurls/gau, hawkrawler, gobuster, wfuzz, dirb, dirbuster, ffuf
Last updated
Refer to the general guide for more information on a few selected tools.
Robots.txt
Sitemap.xml
The Burp Suite Site map feature found under the Target tab provides an overview of the directories found by Burp, that is gathered as the web app is explored, and with additional crawling.
The package getallurls
installed directly in Kali Linux with apt install
seems to have some issue.
The gau
binary can be installed from the installation step listed in the Github link below instead:
Determine the current system architecture (Linux)
Since I am running my Kali on a Raspberry pi in this case, the commands above will display aarch64
, which is ARM-64.
Download the apprioprate gau binary .tar.gz
file. In my case, it will be labelled as gau_2.2.4_linux_arm64.tar.gz
.
Extract the files, and move the binary file to another location. Depending on your system, the default binary path may defer. In this example, I'll assume its /usr/bin
.
Ffuf is a fast web fuzzer for directory discovery. The term fuzzing refers to the act of sending random data to applications (URLs in this case), to discover content that would not have been discovered otherwise.
Basic command:
Flags
-w:
Path to wordlist
-u
: HTTP/HTTPS endpoint URL to fuzz
The FUZZ
keyword in the URL supplied to the -u
flag will be replaced by each word given in the wordist.
Suppose there is a target at the HTTP address http://88.88.88.88 to be fuzzed: with the wordlist in the ~/wordlists/common.txt directory containing common directory values.
The output shows that the directories: assets and robots.txt, returned a valid status code, indicating that there are contents present.
DIRB is a Web Content Scanner. It looks for existing (and/or hidden) Web Objects. It basically works by launching a dictionary based attack against a web server and analyzing the responses.
Basic command:
Scan the URL http://88.88.88.88/ with the wordlist provided.
Gobuster is a tool used to brute-force: URIs (directories and files) in web sites, DNS subdomains (with wildcard support), Virtual Host names on target web servers, ...
Basic command:
Flags
dir
: Uses directory/file enumeration mode
--url:
HTTP/HTTPS endpoint URL to brute-force
-w
: Path to wordlist
To brute-force the HTTP URL http://88.88.88.88/ with the wordlist ~/wordlists/common.txt
The output shows that the directories: assets and robots.txt, returned a valid status code, indicating that there are contents present.