In this page, we provide detailed instructions on how to run FourthPartyMobile. If you just wish to analyze existing crawl data, you can go to the Data section and download SQLite databases of crawls conducted by us. Also, the Analyzing a SQLite database section in the Appendix below might still be of interest to you.

For Desktop crawls, refer to the normal FourthParty website.

Note: most files provided by us are compressed in RAR format. To extract their contents, you need 7-Zip on Windows or the unrar package on Linux (sudo apt-get install unrar).

Basic Walkthrough

You will need the following hardware:
  • One Android device (can be emulated if you do not have one).
  • A PC that will connect to the Android device.
  • A PC that will act as a FourthPartyMobile server, which can be the same PC that is connected to the Android device.
STEP 0: Getting Started
We highly recommend you use Ubuntu or some other Linux flavor to do this. Also, we are assuming that you will be extracting/downloading everything to your HOME directory, so replace ”~” with your folder of choice if necessary. Our OS of choice was always Ubuntu 12.04 LTS.

1. Download Android SDK TOOLS ONLY on your PC (the one we used, latest one). Once you extract it, run the following command to download/update stuff:


All you need for the rest of this tutorial is the Android SDK Platform-tools package, so select that thing and install it.

2. Grab our modified Mozilla Add-On SDK (mozilla_addon-sdk-modded.rar) and extract it on your PC. We will assume its contents are in the ~/mozilla_addon-sdk-modded/ path for the rest of the How-To steps.

3. Connect your Android device to your computer. If you do not have an Android device, refer to the Setting up an emulated Android device section in the Appendix below and skip to step 6.

4. A notification should come up in the Android device about the new USB connection. Set the USB connection to CHARGE ONLY (this is really important).

5. On your Android device, go to Settings > Applications > Development or Settings > Developer and make sure that USB Debugging is activated. You might also want to activate the Stay Awake option.

6. Now, invoke the following command on your PC:

sudo ~/android-sdk-linux/platform-tools/adb devices

The sudo is required to grant adb the necessary privileges. Your device's ID should appear there if the computer is detecting it. If not, you will get a blank list serving as an indicating that something is wrong with your configuration.

7. Install a Firefox Mobile nightly build on your Android device.

OPTION 1: We used Installer 1 for most devices and Installer 2 for ARMv6 devices. Point your mobile device's browser to for installer 1 or for installer 2, and complete the installation after the download finishes.

OPTION 2: Point your mobile device's browser to and download the latest version of the appropriate installer.

8. Download and extract the FourthPartyMobile source code on your PC. We will assume its contents are in the ~/fourthparty_mobile/ path for the rest of the How-To steps.

STEP 1: Running the FourthPartyMobile server
1. Make sure you have a PC that is visible from the outside (either through port forwarding or a public IP). It will act as the supporting server for the crawls you conduct. If you are stuck inside a restricted network, you can use an Amazon EC2 Micro instance for free, which comes with a public IP address.

2. Download the FourthpartyServer to your publicly-visible PC (binary, source+binary).

3. Run “java -jar FourthpartyServer.jar <port-num>” on the server, where <port-num> is an arbitrary forwarded (i.e. publicly visible) port. You can check if the server is visible from the outside using YouGetSignal.

4. Now you can run the new mobile plugin as many times as you want. Each execution of the mobile plugin will be registered by the server as a new session, with a separate SQLite database being generated once you close the mobile browser.

The FourthpartyServer is written in java with a SQLite JDBC library that supports Mac OS, Linux and Windows, so it should be fully multi-platform. It also supports concurrency, so you can have multiple crawls being recorded simultaneously.

STEP 2: Preparing the mobile code
Before you run the plugin, you need to change the code minimally on the PC that is connected to the Android device:

1. Open ~/fourthparty_mobile/extension/lib/remote-server.js with your favorite text editor.

2. Change the SERVER_IP and SERVER_PORT constants to point to the PC that is running the FourthpartyServer application.

3. Start a Mozilla Add-On SDK shell in a terminal:

cd ~/mozilla_addon-sdk-modded

source bin/activate

4. Go to ~/fourthparty_mobile/extension/ using the Mozilla Add-On SDK shell:

cd ~/fourthparty_mobile/extension/

5. Re-build the code to reflect the changes using the Mozilla Add-On SDK shell:

cfx xpi

6. Now you can run the mobile plugin as described in the following section. Remember to have your FourthpartyServer application running every time you run the mobile plugin!

STEP 3: Launching FourthPartyMobile on the Android device
1. Start a Mozilla Add-On SDK shell in a terminal:

cd ~/mozilla_addon-sdk-modded

source bin/activate

2. Push FourthPartyMobile to the Android device using the Mozilla Add-On SDK shell:

cd ~/fourthparty_mobile/extension

cfx run -a fennec-on-device -b ~/android-sdk-linux/platform-tools/adb –mobile-app fennec –force-mobile

3. Your Android device should have launched Firefox Mobile Nightly. Now, you can browse the internet in your mobile device and the FourthParty database will begin populating on the FourthpartyServer's side.

4. Once you are done crawling, you must close the Firefox Mobile Nightly application to notify the FourthpartyServer that the crawling session is over. This can be done by going to the Android device's Settings > Application Settings > Manage Applications, then selecting Nightly and pressing the Force Stop button.

5. The FourthpartyServer will generate a SQLite database once the session is closed. If something goes wrong and the SQLite database is never generated, there is always a plaintext .log file associated with every crawl containing all the SQL statements needed to generate the SQLite database. You can use our Log2Sqlite utility to convert a log file into a SQLite database automatically (java -jar log2sqlite.jar).


Setting up an automatic multi-website crawl
We tried automating crawls on Firefox Mobile with MozMill, Selenium, Scriptish and Robocop without any success. The few frameworks that are compatible with Android do not support Firefox Mobile (fennec) – they only interact with the lackluster Android Web Browser. To the best of our knowledge (hours worth of Google searches), it seems that there are no testing frameworks out there that can automate Firefox Mobile crawls. Even the browser plugins that do this on the desktop version (e.g. Flem) have not been ported to the Mozilla Mobile SDK. Bare in mind that Mozilla Mobile SDK was released on February 21 of 2012 (Announcing Add-on SDK 1.5!), so it is not what you would call “mature”.

Here is our home-brewed solution: we use Javascript code on one site to trigger the crawl on a separate tab! We tested this method on all platforms (i.e. desktop, tablet, smartphone) and it works perfectly. We have automated the generation of the HTML website containing the Javascript code to facilitate the creation of these scripts for arbitrary URL lists. This is what you have to do to create your own crawling HTML file:

1. Create a text file containing all the URLs that you want to crawl, with one URL per line.

2. Download the our CrawlGenerator (binary, source+binary) and run the following command:

java -jar CrawlGenerator.jar <timeout>

Where <timeout> is the amount of time between URLs in milliseconds. For example, the command shown below would be used to create a script that launches a URL and waits 30 seconds before it launches the next URL, and so on:

java -jar CrawlGenerator.jar 30000

3. The CrawlGenerator program will ask you to select the URL list file. After selecting a valid list, it will generate multiple HTML files (crawl0.html, crawl1.html, …). Each HTML file contains the scripts required to launch 5 URLs or less on a separate tab, which is done on purpose to ensure the proper operation of mobile browsers. Each HTML file also points to the next one (e.g. crawl0.html points to crawl1.html) on the main tab once it is done launching its 5 URLs, so they act as a chain of web pages. The very last HTML file in the chain ends with “about:blank” on the main tab to indicate that the crawl is over.

4. Upload the HTML files to a web server.

5. Launch the Fourthparty Mobile addon on your Android device and point the browser to your crawl0.html file. You must allow all pop-ups from that website. Also, do not forget to have your FourthpartyServer.jar running on an externally-visible PC before you launch the mobile plugin!

6. Wait for all the URLs to be visited, one by one, on the browser's second tab.

We found that a timeout of 20 to 30 seconds is appropriate for a crawl conducted on an Android tablet connected to WiFi.

Analyzing a SQLite database
There are many software packages capable of interpreting and manipulating SQLite databases. The SQLite Manager Firefox plugin is a popular lightweight solution, while SQLiteStudio is recommended for more advanced users.

Once you open a SQLite database with an appropriate piece of software, you will be able to execute SQL queries on it to investigate its contents. If you are unfamiliar with SQL or need a quick refresher, the W3Schools SQL Tutorial is a good starting point.

Below is an UML diagram of the FourthParty database schema (click on it to enlarge):

Logging the Android device's output
Getting the Android device's output is pretty important for debugging. To do this, you need to invoke adb's logcat command on a separate terminal window (or sending it to the background with &) BEFORE you push the mobile plugin to the device.

To grab Mozilla Mobile Nightly's output only and save it to a file named ExceptionTrace:

~/android-sdk-linux/platform-tools/adb -d logcat | grep Gecko > ExceptionTrace

To grab all output from the device and save it to a file named logcatOutput:

~/android-sdk-linux/platform-tools/adb -d logcat > logcatOutput

If you kill (ctrl+c) the logcat process, the FourthPartyMobile application will end as well, so you might want to keep that in mind. You usually close the Nightly browser BEFORE you kill the logcat process.

Setting up an emulated Android device
Mozilla's Running Fennec (Mobile Firefox) in an emulated Android device tutorial walks through the process of setting up a generic emulated device. The screenshots shown below illustrate the settings that we used to emulate real-life Android devices. Obtaining a stable emulated device compatible with Mozilla Firefox Nightly was no easy feat, so we suggest you use the exact same settings that we did.

Emulated Nexus S Smartphone

Emulated Nexus 7 Tablet

how_to.txt · Last modified: 2013/01/21 21:42 (external edit) · []
Recent changes RSS feed Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki