Home

Awesome

Build Status

Divolte Collector

Scalable clickstream collection for Hadoop and Kafka

Divolte Collector is a scalable and performant server for collecting clickstream data in HDFS and on Kafka topics. It uses a JavaScript tag on the client side to gather user interaction data, similar to many other web tracking solutions. Divolte Collector can be used as the foundation to build anything from basic web analytics dashboarding to real-time recommender engines or banner optimization systems.

https://divolte.io

Divolte Collector

Online documentation and downloads

You can find the latest downloads and documentation on our project website. There is a series of examples for working with collected data in Spark, Hive / Impala, and Kafka in this repository: https://github.com/divolte/divolte-examples.

Features

Building Prerequisites

In order to build the Divolte Collector you need to have following installed:

Building

To build the Divolte Collector server itself:

% ./gradlew zip

or

% ./gradlew tarball

This will build everything and produce an elementary distribution archive under the build/distributions/ directory.

To build the User Guide:

% ./gradlew userdoc

This will build the documentation and place it under the build/userdoc/html/ directory.

Testing

Unit tests can be executed with:

% ./gradlew test

By default this will skip browser-based integration tests. Currently browser-based testing is supported using:

Chromedriver

ChromeDriver must be installed locally. Under OS X this can be installed via HomeBrew:

% brew install chromedriver

Tests can then be executed:

% SELENIUM_DRIVER=chrome CHROME_DRIVER=$(which chromedriver) ./gradlew test

Safari Webdriver

Safari (from version 10) has native Webdriver support. To set this up:

  1. Enable the developer menu: Preferences|Advanced|Show Develop menu in menu bar
  2. In the Develop menu, enable Allow Remote Automation.
  3. First time only, execute safaridriver -p 0 from the command-line and authorise the driver to connect to Safari.

Tests can then be executed:

% SELENIUM_DRIVER=safari ./gradlew test

PhantomJS

PhantomJS must be installed locally. Under OS X this can be installed via HomeBrew:

% brew install phantomjs

Tests can then be executed:

% SELENIUM_DRIVER=phantom ./gradlew test

SauceLabs

If you have a SauceLabs account, you can test against a wide variety of browsers. Once you have a username and API key and Sauce Connect running, tests can then be executed:

% export SAUCE_USERNAME=<username>
% export SAUCE_ACCESS_KEY=<api key>
% SELENIUM_DRIVER=sauce ./gradlew test

These tests can take quite some time to execute. Not all succeed.

License

The Divolte Collector is licensed under the terms of the Apache License, Version 2.0.