Aggregate

ODK Aggregate provides a ready-to-deploy server and data repository to:

  • provide blank forms to ODK Collect (or other OpenRosa clients),
  • accept finalized forms (submissions) from ODK Collect and manage collected data,
  • visualize the collected data using maps and simple graphs,
  • export data (e.g., as CSV files for spreadsheets, or as KML files for Google Earth), and
  • publish data to external systems (e.g., Google Spreadsheets or Google Fusion Tables).

ODK Aggregate can be deployed on Google's App Engine, enabling users to quickly get running without facing the complexities of setting up their own scalable web service. ODK Aggregate can also be deployed locally on a Tomcat server (or any servlet 2.5-compatible web container) backed with a MySQL or PostgreSQL database server.

Installation

  1. Please read all the instructions and notes before beginning.
  2. Make sure you have used ODK Collect and are familiar with how it works.
  3. Next, try the ODK Aggregate demo server to explore the core functionality.
  4. Decide whether to install a cloud instance or a local instance. We strongly recommend you try an App Engine cloud instance first. If you wish to host locally, see Aggregate Deployment Planning.

Local hosting implies that you are taking ownership of the off-site back-up and restoration of your data and are documenting the steps necessary to return your systems to operation in circumstances that might include a full hardware failure or the destruction of your facility. You must also plan for the security of your data and systems. And finally, it requires that you configure your network routers (e.g., WiFi router). We would encourage you to seek assistance from your local computer-technical-support community before proceeding. The set-up of the ODK Aggregate web server and database are very easy in comparison.

 

Installing on App Engine (Cloud)

  1. Make sure Java 7 or higher is installed on the computer you plan to use. If it is not, download and install it. If you are using MacOSX, it may require special care and attention. See MacOSX Java install and MacOSX Java FAQ.
  2. You will need a Gmail account to use App Engine; this Gmail account will be the owner of the Google Cloud Platform "project" under which your App Engine will execute. Google Cloud Platform hosts projects under these terms. There is no cost to set up these projects and App Engine projects that are lightly used will incur no charges; when you exceed the free-usage levels that Google allows, your App Engine server will stop accepting new requests for the remainder of a 24-hour period that resets at midnight in the Los Angeles, CA timezone. See usage fees for ways to minimize usage.
  3. To set up a Google Cloud Platform project, go to Google Cloud Platform and click on "My Console":
  4. If you have never configured a Google Cloud Platform project, click on "Create an Empty Project":

    Otherwise, this link will open onto either a page with a "Create Project" button and a table listing all of your projects, or it will open into one of your existing projects. In the later case, click on that project name to access a drop-down menu and choose "Create Project" from there. E.g., if clicking "My Console" opened onto the project "msundt-agg-test3", you would click on that to access a menu where you can chose "Create a project...":
  5. On the project-creation pop-up dialog, type in a project name that makes sense to you. Then choose "Edit" to edit the project id. The project id will be the first part of the URL to your ODK Aggregate server (i.e., your server will be referenced with a URL of the form https://project_id.appspot.com). You may want to use a project id that combines your organization name and the name of your data collection group or project. You may also need to accept Google's terms-and-conditions, as indicated in the screen-shot below:
  6. Upon creating the Google Cloud Platform project, you will be on an empty-project screen, showing the project name in the upper-right side of the screen. In this case, I named my project and project id the same - "msundt-agg-test3":
  7. Download ODK Aggregate v1.N.N. Select the latest release for your operating system. These downloads are wizard-based installers for the various operating systems. If you are running OSX, you must unzip the downloaded file before running the installer within it. If you are on MacOSX Mountain Lion or onward, you will need to fiddle with GateKeeper settings in order to run the installer. On Linux, you will need to change the downloaded file's permissions to enable running it as a program. Double-click the downloaded file to run it.
  8. The installer will guide you through configuring ODK Aggregate for App Engine and then launch an upload tool to upload this configured ODK Aggregate to App Engine. NOTE: Beginning with Java 7 Update 51, there are security level settings described here that may prevent the upload tool from running. A reported work-around is to add the file: path (e.g., file:///) to the Exception Site list. If you need to install a pre-1.4.8 version of ODK Aggregate, see the workaround here. The scrolling progress messages produced as ODK Aggregate's files are uploaded will report several server errors. These are expected. An example of the scrolling progress messages produced by a successful upload is here.
  9. Once the updater script has run and uploaded the ODK Aggregate configuration to App Engine, return to the Google Cloud Platform console. With the console displaying your project, click on the menu icon (three horizontal bars) to the right of "Google Cloud Platform" in the upper left side of the screen:
  10. From this menu, select "App Engine":
  11. This will display the App Engine dashboard. You can confirm the upload by verifying that the list of modules contains both "default" and "background". If you need to set up billing, the settings to do that are at the bottom of this screen.
  12. If you click on the project id (e.g., msundt-agg-test3.appspot.com) URL in the above screen, you will go to your ODK Aggregate server:

    From here, you can click on "Log In" to log in, enter the ODK Aggregate username (superuser) that you specified within the installer (the initial password for this username will be aggregate) and access the site administration screens for your server. See Using the Application below.

 

Installing VM (Local or Cloud)

The ODK Aggregate VM is a fully-configured install of Aggregate that you can run on any computer. It requires very little setup, works well without Internet connectivity, and gives you complete control over your data collection campaign. Download the ODK Aggregate VM to begin.

Installing on Tomcat (Local or Cloud)

To run on ODK Aggregate on a Tomcat server backed with a MySQL or PostgreSQL database, see Aggregate Tomcat Install.

NOTE: The Visualization and Mapping features of ODK Aggregate do not work when the server is configured to run under certain non-US, non-English locales. We believe this occurs because, in those locales, numbers are formatted with a comma as the decimal point, and that this confuses the visualization software we are using. If you are having issues with these features not working, reconfigure your Tomcat server to run in the US English locale.

Installing on AWS (Cloud)

To run on ODK Aggregate on a Linux micro-instance on the Amazon Web Services EC2 infrastructure, see Aggregate AWS Install.

NOTE: The Visualization and Mapping features of ODK Aggregate do not work when the server is configured to run under certain non-US, non-English locales. We believe this occurs because, in those locales, numbers are formatted with a comma as the decimal point, and that this confuses the visualization software we are using. If you are having issues with these features not working, reconfigure your Tomcat server to run in the US English locale.

Using the Application

  1. Beginning with ODK Aggregate 1.3.2, upon the initial installation of the server, it is configured to allow unauthenticated (anonymousUser) submissions from ODK Collect and unauthenticated browser access to the submissions and forms management functionality of ODK Aggregate. When the URL to the ODK Aggregate server is first opened, you will be presented with the application page showing the Submissions and Form Management tabs.
  2. In April 2015, the use of Google e-mail accounts for accessing the site (via Sign in with Google) stopped working (Google turned off that functionality).  If you have an existing site running an old version of ODK Aggregate that does not have ODK Aggregate usernames configured for website access (and offers a Sign in with Google sign-in choice), you will need to upgrade to regain access to it.
  3. Submitted data, once in ODK Aggregate, can be viewed, exported, mapped and deleted.
  4. Use the Add New Form button on the Form Management tab to upload a new form definition to ODK Aggregate.
  5. View data submitted from ODK Collect on the Submissions tab.
  6. If the Site Admin tab is not visible, click the Log In link in the upper right corner of the screen to be presented with the Log onto Aggregate screen. Choose the Sign in with Aggregate password button and enter the super-user username you specified within the installer. The initial password for this account is aggregate When signing in with this method, if you do not enter the password correctly, you may need to close all your browser windows and quit your browser before you can try again.
  7. If you have not yet changed your super-user password to something other than aggregate, the server will display This server and its data are not secure! Please change the super-user's password! at the top of the web page. Please visit the Permissions sub-tab under the Site Admin tab to change this user's password.
  8. If the instance name of the server changes (the installer asks for this name), then the passwords for all ODK Aggregate usernames will be cleared (preventing their use) and the super-user username's password will be reset to aggregate and the above message will also be displayed. In this case, you should log in, change the super-user's password, and change the passwords for all of your ODK Aggregate usernames.
  9. Use the Permissions sub-tab under the Site Admin tab to restrict who can download forms or submit data from ODK Collect. Do this by creating an ODK username and password and granting it Data Collector privileges. This username and password can then be entered into ODK Collect's settings page. When restricting access you must also remove the Data Collector privilege from the anonymousUser. Remember to click Save Changes to make these changes take effect. Conversely, granting the Data Collector privilege to the anonymousUser enables anyone to submit data to your ODK Aggregate server.
  10. Use the Permissions sub-tab under the Site Admin tab to specify additional usernames with browser access to the server. For each user you add, select whether they have access to the submitted data (Data Viewer privileges), the ability to upload forms and export or publish data (Form Management), or the ability to manage site access and users (Site Admin) privileges. Remember to click Save Changes to make these changes take effect. And, for each username you define, remember to Change Password to assign a password; by default, usernames are created with unusable passwords. Granting any of these privileges to the anonymousUser enables browser access to these functions without first logging in.
  11. Refer to the inline help in ODK Aggregate (near the login button) for more detailed instructions.

Useful Notes

  • ODK Aggregate install for Windows from Geo for Good 2015
  • The developer wiki has release notes and tips on Tomcat troubleshooting (and App Engine troubleshooting).
  • Google App Engine servers may be located anywhere in the world. Depending on the sensitivity of the data and specific storage rules/restrictions, the server infrastructure may not have all necessary security precautions (such as encryption). It is the organization's responsibility to research and comply with applicable laws and regulations before storing data on Google App Engine. The organization is also responsible for taking the appropriate security precautions and educating users that the information will be available to the organization and stored on Google servers as specified in the Google App Engine Terms of Service.
  • Looking for other servers? Try ODK Briefcase (for local offline storage), FormHub (for free data hosting on the cloud), DataHQ (like Aggregate but in Python), RapidSMS (for SMS integration) or OpenMRS (for medical records).