• One Android device (can be emulated if you do not have one).
• A PC that will connect to the Android device.
• A PC that will act as a FourthPartyMobile server, which can be the same PC that is connected to the Android device.

1. Download Android SDK TOOLS ONLY on your PC (the one we used, latest one). Once you extract it, run the following command to download/update stuff:

~/android-sdk-linux/tools/android

All you need for the rest of this tutorial is the Android SDK Platform-tools package, so select that thing and install it.

2. Grab our modified Mozilla Add-On SDK (mozilla_addon-sdk-modded.rar) and extract it on your PC. We will assume its contents are in the ~/mozilla_addon-sdk-modded/ path for the rest of the How-To steps.

3. Connect your Android device to your computer. If you do not have an Android device, refer to the Setting up an emulated Android device section in the Appendix below and skip to step 6.

4. A notification should come up in the Android device about the new USB connection. Set the USB connection to CHARGE ONLY (this is really important).

5. On your Android device, go to Settings > Applications > Development or Settings > Developer and make sure that USB Debugging is activated. You might also want to activate the Stay Awake option.

6. Now, invoke the following command on your PC:

sudo ~/android-sdk-linux/platform-tools/adb devices

The sudo is required to grant adb the necessary privileges. Your device's ID should appear there if the computer is detecting it. If not, you will get a blank list serving as an indicating that something is wrong with your configuration.

7. Install a Firefox Mobile nightly build on your Android device.

OPTION 1: We used Installer 1 for most devices and Installer 2 for ARMv6 devices. Point your mobile device's browser to http://bit.ly/installer1 for installer 1 or http://bit.ly/installer2 for installer 2, and complete the installation after the download finishes.

OPTION 2: Point your mobile device's browser to http://nightly.mozilla.org/ and download the latest version of the appropriate installer.

8. Download and extract the FourthPartyMobile source code on your PC. We will assume its contents are in the ~/fourthparty_mobile/ path for the rest of the How-To steps.

2. Download the FourthpartyServer to your publicly-visible PC (binary, source+binary).

3. Run “java -jar FourthpartyServer.jar <port-num>” on the server, where <port-num> is an arbitrary forwarded (i.e. publicly visible) port. You can check if the server is visible from the outside using YouGetSignal.

4. Now you can run the new mobile plugin as many times as you want. Each execution of the mobile plugin will be registered by the server as a new session, with a separate SQLite database being generated once you close the mobile browser.

The FourthpartyServer is written in java with a SQLite JDBC library that supports Mac OS, Linux and Windows, so it should be fully multi-platform. It also supports concurrency, so you can have multiple crawls being recorded simultaneously.

1. Open ~/fourthparty_mobile/extension/lib/remote-server.js with your favorite text editor.

2. Change the SERVER_IP and SERVER_PORT constants to point to the PC that is running the FourthpartyServer application.

3. Start a Mozilla Add-On SDK shell in a terminal:

cd ~/mozilla_addon-sdk-modded

source bin/activate

4. Go to ~/fourthparty_mobile/extension/ using the Mozilla Add-On SDK shell:

cd ~/fourthparty_mobile/extension/

5. Re-build the code to reflect the changes using the Mozilla Add-On SDK shell:

cfx xpi

6. Now you can run the mobile plugin as described in the following section. Remember to have your FourthpartyServer application running every time you run the mobile plugin!

cd ~/mozilla_addon-sdk-modded

source bin/activate

2. Push FourthPartyMobile to the Android device using the Mozilla Add-On SDK shell:

cd ~/fourthparty_mobile/extension

cfx run -a fennec-on-device -b ~/android-sdk-linux/platform-tools/adb –mobile-app fennec –force-mobile

3. Your Android device should have launched Firefox Mobile Nightly. Now, you can browse the internet in your mobile device and the FourthParty database will begin populating on the FourthpartyServer's side.

4. Once you are done crawling, you must close the Firefox Mobile Nightly application to notify the FourthpartyServer that the crawling session is over. This can be done by going to the Android device's Settings > Application Settings > Manage Applications, then selecting Nightly and pressing the Force Stop button.

5. The FourthpartyServer will generate a SQLite database once the session is closed. If something goes wrong and the SQLite database is never generated, there is always a plaintext .log file associated with every crawl containing all the SQL statements needed to generate the SQLite database. You can use our Log2Sqlite utility to convert a log file into a SQLite database automatically (java -jar log2sqlite.jar).

Here is our home-brewed solution: we use Javascript code on one site to trigger the crawl on a separate tab! We tested this method on all platforms (i.e. desktop, tablet, smartphone) and it works perfectly. We have automated the generation of the HTML website containing the Javascript code to facilitate the creation of these scripts for arbitrary URL lists. This is what you have to do to create your own crawling HTML file:

1. Create a text file containing all the URLs that you want to crawl, with one URL per line.

2. Download the our CrawlGenerator (binary, source+binary) and run the following command:

java -jar CrawlGenerator.jar <timeout>

Where <timeout> is the amount of time between URLs in milliseconds. For example, the command shown below would be used to create a script that launches a URL and waits 30 seconds before it launches the next URL, and so on:

java -jar CrawlGenerator.jar 30000

3. The CrawlGenerator program will ask you to select the URL list file. After selecting a valid list, it will generate multiple HTML files (crawl0.html, crawl1.html, …). Each HTML file contains the scripts required to launch 5 URLs or less on a separate tab, which is done on purpose to ensure the proper operation of mobile browsers. Each HTML file also points to the next one (e.g. crawl0.html points to crawl1.html) on the main tab once it is done launching its 5 URLs, so they act as a chain of web pages. The very last HTML file in the chain ends with “about:blank” on the main tab to indicate that the crawl is over.

4. Upload the HTML files to a web server.

5. Launch the Fourthparty Mobile addon on your Android device and point the browser to your crawl0.html file. You must allow all pop-ups from that website. Also, do not forget to have your FourthpartyServer.jar running on an externally-visible PC before you launch the mobile plugin!

6. Wait for all the URLs to be visited, one by one, on the browser's second tab.

We found that a timeout of 20 to 30 seconds is appropriate for a crawl conducted on an Android tablet connected to WiFi.

Once you open a SQLite database with an appropriate piece of software, you will be able to execute SQL queries on it to investigate its contents. If you are unfamiliar with SQL or need a quick refresher, the W3Schools SQL Tutorial is a good starting point.

Below is an UML diagram of the FourthParty database schema (click on it to enlarge):

To grab Mozilla Mobile Nightly's output only and save it to a file named ExceptionTrace:

~/android-sdk-linux/platform-tools/adb -d logcat | grep Gecko > ExceptionTrace

To grab all output from the device and save it to a file named logcatOutput:

~/android-sdk-linux/platform-tools/adb -d logcat > logcatOutput

If you kill (ctrl+c) the logcat process, the FourthPartyMobile application will end as well, so you might want to keep that in mind. You usually close the Nightly browser BEFORE you kill the logcat process.

Emulated Nexus S Smartphone

Emulated Nexus 7 Tablet