---
title: Spidering websites with WebScarab
course: intro_pentest
section: "Web-Based Exploitation"
layout: lesson
---

A great tool to use when initially interacting with a target is WebScarab.
WebScarab was written by Rogan Dawes and is available through the OWASP website.
If you are running BlackArch, a version of WebScarab is already installed. This
powerful framework is modular in nature and allows you to load numerous plug-ins
to customize it to your needs. Even in its default configuration. WebScarab
provides an excellent resource for interacting with and interrogating web
targets.

After having run the vulnerability scanners, the next logical step is to run a
spidering program on the target website. Spiders are extremely useful in
reviewing and reading (or crawling) your target’s website looking for all links
and associated files. Each of the links, web pages and files discovered on your
target are recorded and catalogued. This catalogued data can be useful for
accessing restricted pages and locating unintentionally disclosed documents or
information.

You can access the spider function in WebScarab by first starting the program,
you can do so by issuing in a terminal:

```
webscarab
```

This will load the WebScarab program. Once you start the tool, you’ll be given
access to a number of new panels along the top of the window including the
“Spider” tab.

Now that you have set up WebScarab, you need to configure your browser to use a
proxy. Setting up WebScarab as your proxy will cause that all the web traffic
going into and coming out of your browser to pass through the WebScarab program.
In this respect, the proxy program acts as a middle man and has the ability to
view, stop and even manipulate network traffic.

Setting up your browser to use a proxy is usually done through the preferences
or network options. In Firefox, you can click ☰ → Preferences → Scroll down, and
click “Settings” in the Network Settings section.

Clicking on the settings button will allow you to configure your browser to use
WebScarab as proxy. Select the radio button for “Manual proxy configuration:”.
Next enter: `127.0.0.1` in the “HTTP Proxy” input box. Finally, enter: `8008`
into the “Port” field. It usually a good idea to check the box just below the
“HTTP Proxy” box and select “Use this proxy for FTP and HTTPS”. Once you have
all of this information entered, you can click “OK” to save the changes made.

At this point, any web traffic coming into or passing out of your browser will
route through the WebScarab proxy. Two words of warning: First, you need to
leave WebScarab running while it’s serving as proxy. If you close the program,
you won’t be able to browse the Internet. If this happens, Firefox is great at
providing you with an error message that it can’t find a proxy and you’ll need
to restart WebScarab or change your network configuration in Firefox. The second
warning is that while surfing the Internet using a local proxy, all https
traffic will show up as having an invalid certificate!. This is expected
behaviour because your proxy is sitting in the middle of your connection.

As a side note, it’s important that you always pay attention to invalid security
certificates when browsing. At this point, certificates are your best defense
and often your only warning against a man-in-the-middle attack.

Now that you have set up a proxy and have configured your browser, you are ready
to begin spidering your target. You begin by entering the target URL into the
browser. In our earlier example, we discovered a website running on
172.16.45.132. Entering the following into your Firefox browser will load the
website through WebScarab. Once the website has loaded in your browser, you can
switch over the WebScarab program. You should see the URL you entered (along
  with other that you have visited since starting your proxy). To spider the
  site, you right click the URL and choose “`Spider tree`".

  You can now view each of the files and folders associated with your target
  website. Individual folders can be further spidered by right clicking and
  choosing "`Spider tree`" again. You should spend time carefully examining
  every nook and cranny within your authorized scope. Spidering a website is a
  great way to find inadvertently or leaked confidential data from a target
  website.

![](/img/courses/intro-pentest/spidered-metasploitable.png)
