Home

Awesome

<center><img src="https://github.com/xnl-h4ck3r/xnLinkFinder/blob/main/xnLinkFinder/images/title.png"></center>

About - v6.4

This is a tool used to discover endpoints (and potential parameters) for a given target. It can find them by:

The python script is based on the link finding capabilities of my Burp extension GAP. As a starting point, I took the amazing tool LinkFinder by Gerben Javado, and used the Regex for finding links, but with additional improvements to find even more.

Installation

xnLinkFinder supports Python 3.

Install xnLinkFinder in default (global) python environment.

pip install xnLinkFinder

OR

pip install git+https://github.com/xnl-h4ck3r/xnLinkFinder.git -v

You can upgrade with

pip install --upgrade xnLinkFinder

pipx

Quick setup in isolated python environment using pipx

pipx install git+https://github.com/xnl-h4ck3r/xnLinkFinder.git

Usage

ArgLong ArgDescription
-i--inputInput a: URL, text file of URLs, a Directory of files to search, a Burp XML output file, an OWASP ZAP output file, or a Caido CSV file.
-o--outputThe file to save the Links output to, including path if necessary (default: output.txt). If set to cli then output is only written to STDOUT. If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-op--output-paramsThe file to save the Potential Parameters output to, including path if necessary (default: parameters.txt). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-owl--output-wordlistThe file to save the target specific Wordlist output to, including path if necessary (default: No wordlist output). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-oo--output-oosThe file to save the out of scope links output to, including path if necessary (default: No OOS output). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-ow--output-overwriteIf the output file already exists, it will be overwritten instead of being appended to.
-sp--scope-prefixAny links found starting with / will be prefixed with scope domains in the output instead of the original link. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used.
-spo--scope-prefix-originalIf argument -sp is passed, then this determines whether the original link starting with / is also included in the output (default: false)
-spkf--scope-prefix-keep-failedIf argument -spkf is passed, then this determines whether a prefixed link will be kept in the output if it was a 404 or a RequestException occurs (default: false)
-sf--scope-filterWill filter output links to only include them if the domain of the link is in the scope specified. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used. This argument is now mandatory if input is a domain/URL (or file of domains/URLs) to prevent crawling sites that are not in scope and also preventing misleading results.
-c--cookies †Add cookies to pass with HTTP requests. Pass in the format 'name1=value1; name2=value2;'
-H--headers †Add custom headers to pass with HTTP requests. Pass in the format 'Header1: value1; Header2: value2;'
-ra--regex-afterRegEx for filtering purposes against found endpoints before output (e.g. /api/v[0-9]\.[0-9]\* ). If it matches, the link is output.
-d--depth †The level of depth to search. For example, if a value of 2 is passed, then all links initially found will then be searched for more links (default: 1). This option is ignored for Burp files, ZAP files and Caido files because they can be huge and consume lots of memory. It is also advisable to use the -sp (--scope-prefix) argument to ensure a request to links found without a domain can be attempted.
-p--processes †Basic multithreading is done when getting requests for a URL, or file of URLs (not a Burp file, ZAP file or Caido file). This argument determines the number of processes (threads) used (default: 25)
-x--excludeAdditional Link exclusions (to the list in config.yml) in a comma separated list, e.g. careers,forum
-orig--originWhether you want the origin of the link to be in the output. Displayed as LINK-URL [ORIGIN-URL] in the output (default: false)
-prefixedWhether you want to see which links were prefixed in the output. Displays (PREFIXED) after link and origin in the output (default: false)
-xrel--exclude-relative-linksBy default, if any links in the results start with ./ or ../, they will be included. If this argument is used, these relative links will not be added.
-t--timeout †How many seconds to wait for the server to send data before giving up (default: 10 seconds)
-inc--includeInclude input (-i) links in the output (default: false)
-u--user-agent †What User Agents to get links for, e.g. -u desktop mobile. Possible values are desktop, mobile, set-top-boxes and game-console. Also there are mobile-apple, mobile-android and mobile-windows that are subsets of mobile but can be used separately.
-uc--user-agent-custom †A custom User Agent string to use for all requests. This will override the -u/--user-agent argument. This can be used when a program requires a specific User Agent header to identify you for example.
-insecureWhether TLS certificate checks should be disabled when making requests (delfault: false)
-s429Stop when > 95 percent of responses return 429 Too Many Requests (default: false)
-s403Stop when > 95 percent of responses return 403 Forbidden (default: false)
-sTOStop when > 95 percent of requests time out (default: false)
-sCEStop when > 95 percent of requests have connection errors (default: false)
-m--memory-thresholdThe memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95)
-mfs--max-file-size †The maximum file size (in bytes) of a file to be checked if -i is a directory. If the file size is over, it will be ignored (default: 500 MB). Setting to 0 means no files will be ignored, regardless of size.
-rp--replay-proxy†For active link finding with URL (or file of URLs), replay the requests through this proxy.
-ascii-onlyWhether links and parameters will only be added if they only contain ASCII characters. This can be useful when you know the target is likely to use ASCII characters and you also get a number of false positives from binary files for some reason.
-mtl--max-time-limitThe maximum time limit (in minutes) to run before stopping (default: 0). If 0 is passed, there is no limit.
--configPath to the YML config file. If not passed, it looks for file config.yml in the default directory, typically ~/.config/xnLinkFinder.
-nwlpl--no-wordlist-pluralsWhen words are found for a target specific wordlist, by default new words are added if there is a singular word from a plural, and vice versa. If this argument is used, this process is not done.
-nwlpw--no-wordlist-pathwordsBy default, any path words found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed. NOTE: if the YAML config value of respParamPathWords is True then this argument will not have any effect unless -nwlpm/--no-wordlist-parameters is also passed.
-nwlpm--no-wordlist-parametersBy default, any parameters found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwlc--no-wordlist-commentsBy default, any comments in pages will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwlia--no-wordlist-imgaltBy default, any image 'alt' attributes will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwld--no-wordlist-digitsExclude any words from the target specific wordlist with numerical digits in.
-nwll--no-wordlist-lowercaseBy default, any word added with any uppercase characters in will also add the word in lowercase. If this argument is used, the lowercase words will not be added.
-wlml--wordlist-maxlenThe maximum length of words to add to the target specific wordlist, excluding plurals (default: 0 - no limit)
-swf--stopwords-fileA file of additional Stop Words (in addition to "stopWords" in the YML Config file) used to exclude words from the target specific wordlist. Stop Words are used in Natural Language Processing and different lists can be found in different libraries. You may want to add words in different languages, depending on your target.
-brt--burpfile-remove-tagsWhen the input passed with -i is a Burp file, the user is asked interactively whether they want to remove unnecessary tags from that file (sometimes there is a problem in Burp XML files that can often be resolved by removing unnecessary tags which will also make the file smaller). If you are using xnLinkFinder in a script, you don't want to break for user input, so you can set that by passing this argument with a true or false. NOTE: This is a permanent change to the file
-nb--no-bannerHides the tool banner.
-v--verboseVerbose output
-vv--vverboseIncreased verbose output
--versionShow current version number.
-h--helpshow the help message and exit

† NOT RELEVANT FOR INPUT OF DIRECTORY, BURP XML FILE, OWASP ZAP FILE OR CAIDO CSV FILE

config.yml

The config.yml file (typically in ~/.config/xnLinkFinder/) has the keys which can be updated to suit your needs:

† IF THESE ARE NOT FOUND IN THE CONFIG FILE THEY WILL DEFAULT TO True

Examples

Find Links from a specific target - Basic

xnLinkFinder -i target.com -sf target.com

Find Links from a specific target - Detailed

Ideally, provide scope prefix (-sp) with the primary domain (including schema), and a scope filter (-sf) to filter the results only to relevant domains (this can be a file or in scope domains). Also, you can pass cookies and customer headers to ensure you find links only available to authorised users. Specifying the User Agent (-u desktop mobile) will first search for all links using desktop User Agents, and then try again using mobile user agents. There could be specific endpoints that are related to the user agent given. Giving a depth value (-d) will keep sending request to links found on the previous depth search to find more links.

xnLinkFinder -i target.com -sp target_prefix.txt -sf target_scope.txt -spo -inc -vv -H 'Authorization: Bearer XXXXXXXXXXXXXX' -c 'SessionId=MYSESSIONID' -u desktop mobile -d 10

Find Links from a list of URLs - Basic

If you have a file of JS file URLs for example, you can look for links in those:

xnLinkFinder -i target_js.txt -sf target.com

Find Links from a files in a directory - Basic

If you have a files, e.g. JS files, HTTP responses, etc. you can look for links in those:

xnLinkFinder -i ~/.config/waymore/results/target.com

NOTE: Sub directories are also checked. The -mfs option can be specified to skip files over a certain size.

Find Links from a Burp project - Basic

In Burp, select the items you want to search by highlighting the scope for example, right clicking and selecting the Save selected items. Ensure that the option base64-encode requests and responses option is checked before saving. To get all links from the file (even with HUGE files, you'll be able to get all the links):

xnLinkFinder -i target_burp.xml

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i starts with <?xml then you are trying to process a Burp file.

Find Links from a Burp project - Detailed

Ideally, provide scope prefix (-sp) with the primary domain (including schema), and a scope filter (-sf) to filter the results only to relevant domains.

xnLinkFinder -i target_burp.xml -o target_burp.txt -sp https://www.target.com -sf target.* -ow -spo -inc -vv

Find Links from an OWASP ZAP project - Basic

In ZAP, select the items you want to search by highlighting the History for example, clicking menu Export and selecting Export Messages to File.... This will let you save an ASCII text file of all requests and responses you want to search. To get all links from the file (even with HUGE files, you'll be able to get all the links):

xnLinkFinder -i target_zap.txt

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i is in the format ==== 99 ========== (v2.11.1) or ===99 ========== (v2.12) for example, then you are trying to process an OWASP ZAP ASCII text file.

Find Links from a Cadio export CSV file - Basic

In Caido, go to the History section and select the Export option.

If you are using Caido Pro or Enterprise edition, then choose the Export current rows option and pick As CSV. Go to the Exports section and download the CSV file. Then pass as input:

xnLinkFinder -i 2023-03-18-010332_csv_requests.csv

If you are using Caido Community edition, then you will have to choose the Export all option and pick As CSV. Go to the Exports section and download the CSV file. As you have the full history, you will want to remove anything that is not relevant from the CSV file. Use the example below, where redbull is the main part of the domains of the target you are looking at.

cat 2023-03-18-010332_csv_requests.csv | grep -E '^id|^[0-9]+,[^,]*redbull' > caido_redbull.csv
xnLinkFinder -i caido_redbull.csv

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i is in the format id,host,method, then you are trying to process a Caido export CSV file.

Find Links from a Waymore results directory

The waymore tool can be used to get URLs from various third party APIs, and also download archived responses from archive.org (Wayback Machine). Passing a waymore results directory to xnLinKFinder will search the contents of archived responses, and also request URLs from waymore.txt and also the archived URLs from index.txt and get more links from those responses.

xnLinkFinder -i ~/Tools/waymore/results/target.com

NOTE: It is passed as a normal directory, but xnLinkFinder will determine it is a waymore results directory and process respectively. This relies on the default naming convention of the URLs file being waymore.txt and that file being in the same directory as the archived files (which it is by default).

Piping to other Tools

You can pipe xnLinkFinder to other tools. Any errors are sent to stderr and any links found are sent to stdout. The output file is still created in addition to the links being piped to the next program. However, potential parameters are not piped to the next program, but they are still written to file. For example:

xnLinkFinder -i redbull.com -sp https://redbull.com -sf rebbull.* -d 3 | unfurl keys | sort -u

You can also pass the input through stdin instead of -i.

cat redbull_subs.txt | xnLinkFinder -sp https://redbull.com -sf rebbull.* -d 3

NOTE: You can't pipe in a Burp, ZAP or Caido file, these must be passed using -i.

Recommendations and Notes

Issues

If you come across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact command you ran and a detailed description of the problem. If possible, run with -vv to reproduce the problem and let me know about any error messages that are given.

TODO

Example output

Active link finding for a domain:

<center><img src="https://github.com/xnl-h4ck3r/xnLinkFinder/blob/main/xnLinkFinder/images/example1a.png"></center> ... <center><img src="https://github.com/xnl-h4ck3r/xnLinkFinder/blob/main/xnLinkFinder/images/example1b.png"></center>

Piped input and output:

<center><img src="https://github.com/xnl-h4ck3r/xnLinkFinder/blob/main/xnLinkFinder/images/example2.png"></center>

Good luck and good hunting! If you really love the tool (or any others), or they helped you find an awesome bounty, consider BUYING ME A COFFEE! ☕ (I could use the caffeine!)

🤘 /XNL-h4ck3r

<a href='https://ko-fi.com/B0B3CZKR5' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://storage.ko-fi.com/cdn/kofi2.png?v=3' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>