2.0 Getting Started - rev 126

The documentation on this page corresponds to rev 126 (February 2015) of the tools suite. See Older Versions for documentation on earlier releases of this suite.

This is an older version of our software.The websites referenced in this documentation will generally have been updated to work with the newer software, and therefore may not function with this older release.

Where practical, we have provided zip files of the ApplicationDesigner environment used in creating those sites. See Survey and Tables Aux Files

The 2.0 ODK tools are intended to address limitations of the existing tool set.  Click here for more details. The 2.0 Tools Suite consists of:

  1. ODK Survey - a data collection application based upon HTML, CSS and Javascript.
  2. ODK Tables - a data collection and visualization application running on your device.
  3. ODK Aggregate - a ready-to-deploy server and data repository with enhancements to support bi-directional data synchronization across disconnected devices.
  4. ODK Application Designer - a design environment for creating, customizing, and previewing your forms.

This page provides a brief end-to-end walk-through of the ODK 2.0 tools. It will walk the user through all the steps necessary to complete the following tasks:

  1. Joining a device into an existing ODK 2.0 application hosted on an ODK Aggregate server.
  2. Migrating / Setting-up an ODK 2.0 application onto a new ODK Aggregate server.
  3. Configuring a device with an ODK 2.0 application by setting up the ODK ApplicationDesigner environment and using it to deploy an application to your device.
  4. Modifying an ODK 2.0 application to add a new field to form.

ODK Applications

The ODK 2.0 device tools (ODK Survey, ODK Tables, ODK Sync, ODK Scan, ODK Sensors Framework, various ODK Sensor implementations) are APKs that are designed to work together to create a coherent tailored application experience for an end-user.

Once the ODK Application Packager becomes available, we anticipate groups being able to build their own application APK that launches the ODK tools to create this coherent tailored application experience.

ODK 2.0 tools access configuration files and store data under sub-directories of the opendatakit directory on the external storage device. ODK Applications are identified by the name of the sub-directory holding those configuration and data files. Thus, opendatakit/mytestapp would contain all the files and data for the "mytestapp" application. The name of that sub-directory, "mytestapp," is referred to as the AppName of that application. The default AppName for ODK Survey is "survey", and the default AppName of ODK Tables is "tables." However, when configured appropriately, the ODK Survey tool can run under another AppName, accessing configuration and saving data in a different directory under opendatakit.

Joining a device to an Existing Aggregate Server

The steps for joining a device to an existing Aggregate server are very straightforward.

  1. Install the APKs your application uses.
  2. Launch the 'home screen' APK, either ODK Survey or ODK Tables.
  3. If ODK Survey, if using the legacy 1.x mechanisms, configure the server URL, username and password for the legacy 1.x server using the settings menu item and navigate to the downloads screen and download your forms from the server.
  4. Otherwise, click on the Cloud icon to launch ODK Sync in the context of your 'home screen' APK.
  5. Configure ODK Sync to point to the ODK Aggregate instance you want to join.
  6. Choose 'Sync now' to make the device mirror the contents on that ODK Aggregate server.

Follow the steps described above to join the ODK Aggregate server hosting our simple demo, which uses ODK Tables as its 'home screen' APK. The detailed steps are:

  1. Download and install ODK Sync, ODK Tables and ODK Survey. You will also need the Google Play Services APK (for accessing maps).
  2. Launch ODK Tables (the 'home screen' APK)
  3. Click on the Cloud icon to launch ODK Sync
  4. Enter https://opendatakit-simpledemo.appspot.com for the server.
  5. Choose an e-mail account configured for your device (if you have not configured one, open your device's Settings app, scroll down to the Accounts heading, and add an account (this is different from the accounts you can set up under Users).
  6. Choose Save settings.
  7. Choose Authorize account.
  8. Choose Sync now

The synchronization process will now occur. If there is a network error or authentication failure, re-authorize the account and choose Sync now again until it completes successfully.

Once successful, back out of ODK Sync, returning to ODK Tables.

Now, choose the Settings menu item and check the Use custom home screen checkbox (this step will eventually not be necessary, but is required in this release).

Finally, back out of ODK Tables and re-launch ODK Tables.

Tour of the Simple Demo Application

You should now see the custom home screen for the simple demo (Geotagger):

This demo is based upon the geotagger data table and form. It allows users to record the date, time, GPS coordinates, description and picture of their current location.

When you launch the demo by clicking the blue launch button, you see a map showing the collected data points, indicated with markers. By clicking on a marker, you bring its data record to the top of the list of records above the map. Clicking on the record header will expand or contract that item to show the coordinates and photo of that location. For example, if we click on the Phinney Ridge marker, its color changes from blue to green, and, if we then touch the Phinney Ridge heading, it expands to show the coordinates and image of that location:

You can add a new data record by choosing the '+' icon in top menu bar. This opens ODK Survey.

The first time you launch ODK Survey, it will initialize itself. For this release, on this first start-up, you will end up with a white screen (this will be addressed in a future release). You must either rotate the device (to trigger a redraw) or back-out and re-enter ODK Survey to re-open the geotagger form. If that still doesn't work, force stop Survey by going to your phone Settings, Apps (or Applications), selecting Survey, and then clicking "Force Stop".

Advance through and finalize this form. Upon Finalizing the form, you will be returned to ODK Tables and its map view. You can then highlight the marker you added and view the image in the list view:

If you then click or tap in the list item details area (e.g., on the image), a detail view of the item will be displayed.

From here, if you were to choose the pencil icon, ODK Survey would be launched to edit this record.

You can also view the data in a list view or spreadsheet view by choosing the sheet icon in the menu bar and selecting the view you want:

These other views can be useful if you need to access and complete data records that do not yet have location data and cannot therefore be displayed on a map. Try these other views now. NOTE: the Graph view is not functional in this release.

Now back out of the geotagger table view and return to the custom home screen. Choose the three-horizontal-line icon on the top menu bar. This presents the Table Manager screen, and choose the Cloud icon. This opens up ODK Sync. Sync your device with the server (authorize the account, and choose Sync now). This will push your newly-added record to the server. You can see this by browsing to https://opendatakit-simpledemo.appspot.com and selecting the "geotagger" table.

If you then repeat these steps with a different device, you can see that the two devices can share and exchange data, and revisions to this data, whenever they synchronize to the server.

During this process, there are two problem-resolution screens you are likely to encounter:

  • Checkpoint Resolution - if ODK Survey exits without the user explicitly saving their additions or changes.
  • Conflict Resolution - if ODK Sync detects a change on the server to a data record that was also changed on the device.

The first screen is most easily triggered if you choose the '+' icon then back out of ODK Survey:

When presented with this screen, there are two choices, either to discard the entire partially-filled-out form, or to save it as incomplete. In this case, since there is no entered data for this record (all the field values are blank), we can discard it.

The second screen is triggered when another device has edited a row and synchronized its changes to the server before your edits to that same row have been synchronized. In this case, your synchronization attempt will end with an error, and the device notifications list ( the list that is shown when you pull down on the top status bar) will show ODK SYNC ERROR tables and beneath that, Conflicts exist. Please resolve. To resolve the errors, back out of the ODK Sync synchronization screen. At this point, the conflict resolution screen will be presented for the row in conflict (or, if you have multiple rows in conflict, you will be presented with a screen where you can choose among those rows -- you must resolve each data row in conflict before you can return to the ODK Tables application).

The single-row conflict screen looks like this:

This displays the values of the field(s) in conflict, with the field value on the device appearing first. In this case, the Description field is in conflict. The device has "Kite hill at Gasworks" and the server has "Kite Hill ... Gasworks". You can select either to take your device values (Take Local Version) or take the server's values (Take Server Version). After selecting the local version, you must again synchronize with the server to push that change up to the server.

This concludes the tour of the Geotagger example application's screens, and the functionality within ODK Tables. We will now turn to setting up our own ODK Aggregate server, and setting the application up to run on that server.

Migrating / Setting-up an ODK 2.0 application

Now that we have seen how a device can join an already-configured application, and synchronize its view of the data with the ODK Aggregate server hosting the application, it is time to set up our own ODK Aggregate server.

The starting point for this is to have a fully configured application on your device. Only after you have your device configured as you want it to appear, would you proceed with the following steps. In this case, we already have the device configured with the Geotagger demo, so let's proceed to create an ODK Aggregate server and configure it to serve that demo to your devices.

Setting up the ODK Aggregate server

Follow the instructions for installing ODK Aggregate here. You must install the ODK Aggregate v1.4.5 release (until we have a release candidate of the ODK 2.0 tools, please assume that the specific device APKs are operable only with a specific ODK Aggregate release. e.g., the rev 126 tools are compatible with the ODK Aggregate v1.4.5 release).

Once you have installed ODK Aggregate, log in with your super-user account. This is described here.

Once logged in, enable the ODK Tables extensions as described here. You should grant the User account on your device the Administer Tables permissions.

Resetting the Application on the Server

Now, return to your device, start ODK Tables, launch ODK Sync via the cloud icon, and edit the Aggregate URL to be the URL for this newly configured ODK Aggregate server (e.g., https://myodk-test.appspot.com). Save settings.

Authorize account, accept any authorization request if prompted, and click the Reset App Server button in the bottom right of the screen.

This button will push the configuration on your device up to the server. It will then synchronize the data on your device with any existing data on your server.

After this has completed, you have created your own server that replicates the configuration and contents of the https://opendatakit-simpledemo.appspot.com site. Congratulations!

Any device with a user account with Administer Tables permissions can do this. If you configure a device with a user account (or Anonymous user) with just the Synchronize Tables permissions, they will not be able to Reset App Server and will just be able to join into the existing ODK 2.0 application on this ODK Aggregate server.

Configuring your Device with an Application

Next, we will work through the steps to configure your device with an ODK 2.0 application (rather than downloading an existing application from a server).

This task begins with setting up the ODK 2.0 ApplicationDesigner on your computer.

For the purposes of this tutorial, we have created a copy of the ApplicationDesigner that only contains the files for this Geotagger example (it is otherwise identical).

Setting up ODK ApplicationDesigner

Read the Overview section here to get a sense of the features and functionality of the ODK 2.0 ApplicationDesigner environment (we will install it below).

Download the simplified ODK 2.0 ApplicationDesigner from here.

And install the necessary software on you computer as described here.

When you reach the sub-section ODK Application Designer v2.0.zip, use the simplified ODK 2.0 ApplicationDesigner zip (above) instead of the main one on the Survey and Tables download page. Otherwise, follow all these instructions.

Finally, launch the simplified ODK 2.0 ApplicationDesigner as described here.

If successful, the cmd window (on Windows) should display some status messages. Below is a screen-shot of my cmd window beginning with a 'dir' of the contents of the directory, and running 'grunt' in that directory:

And a Chrome browser window should open to display:

If a Chrome browser does not open, try manually launching it and opening http://localhost:8000/index.html.

You can further verify that the ApplicationDesigner works by clicking on the "Geotagger" button, then clicking on "Follow link". This opens the Geotagger form on your computer, and simulates all the features available to you on your device.

You can also try other things, like choosing different device dimensions to see how the form renders on different screen geometries.

We will return to this design environment later.

Deploying to the Device

Now that we have the design environment installed and functioning, and because that environment has a copy of the fully-configured Geotagger application that is running on https://opendatakit-simpledemo.appspot.com, we can work through the steps of deploying that application to your device, and then Resetting your server to push that configuration up to your server.

First, ensure that ODK Tables, ODK Survey and ODK Sync are all stopped. Do this by going to your device's Settings page, choose Apps, select each of these ODK tools and confirm that the Force Stop button is disabled. If it is active, click on it to force the application to stop.

While also in the Settings, confirm that your device has USB debugging enabled. This checkbox is in different places on different devices and may be hidden by default on some. See here for instructions.

Next, launch OI File Manager and delete the contents of the opendatakit directory on the device. For newer devices, the directory to delete is storage/emulated/0/opendatakit This may vary among devices and Android operating system versions. This directory can contain a range of sub-directories, but if you have ever run ODK Tables on the device, it will contain at least tables. Within that tables sub-directory, there will tables, assets, framework and metadata subfolders. Delete all the folders and files within the opendatakit directory.

Your device now has no ODK 2.0 applications on it.

Return to the cmd window on your computer. Control-C to stop the grunt command that popped-open the Chrome browser. On Windows, you will be asked to confirm this (Terminate batch job (Y/N)?). Enter Y to confirm.

Connect your device to your computer via USB. Wait for the storage connection to be established (on Windows, this will generally pop up a file browser or an options box that enables you to select a file browser).

At the command prompt, type:

grunt adbpush-tables-app

This pushes the configured ODK 2.0 application within this ODK 2.0 Application Designer directory to your device. The cmd window will display a series of commands and conclude with a display of overall progress and timings:

Now, on your device, launch ODK Tables.

Click on the three-row icon in the top menu bar (the need for this manual step will be addressed in a subsequent release). This will initiate the configuration of ODK Tables and conclude with a Configuration Summary pop-up reporting that assets/csv/geotagger.updated.csv was imported successfully. This csv file is what populates the initial set of rows into the Geotagger data table. Click OK.

Back out of the application and re-launch it. Everything should now appear as it did with the application you first joined on https://opendatakit-simpledemo.appspot.com, except you will only have the data rows configured by the ODK 2.0 ApplicationDesigner zip, and not any added or modified since that time.

Resetting the Server

Once you have the application running on the device, you will typically need to reset the contents of the application server. While the Reset App Server button on the device can shuffle the various supporting files between the device and the server, it will not destroy data tables that already exist on the server if they are also a part of the application you want to reset.

Therefore, whenever you are developing an application, and have found a need to add a new column to an existing table, you will need to manually delete the data tables from the server before using the Reset App Server button from the device.

Open a browser window to the server, log in with a user that has Administer Tables or Site Admin privileges.

Navigate to the ODK Tables / Current Tables sub-tab.

Delete each of the tables here. In this case, there will be only one, Geotagger. The server will now have a set of App-Level files but no data tables, forms for those tables, or data files. Except for the app-level files, it is 'clean'

Now, from your device, launch ODK Tables, click on the cloud icon to launch ODK Sync, and choose Reset App Server.

The synchronization process will now create the tables and push your content up to this server. Note that the server now only contains the data rows present on the device -- it no longer has any of the additional data records from the demo site.

You have now successfully set up the ApplicationDesigner, used it to deploy an application to a device, and, from that device, configured an ODK Aggregate server to supply that application to other devices you join to that server.

Modifying an ODK 2.0 application

The final task is to modify the Geotagger example by adding a new data field to it.

NOTE: There are many manual steps to this process that will be simplified or eliminated in future releases.

The overall development process has three steps:

  1. revise the data entry form
  2. update the initialization files needed by ODK Tables
  3. update the preloaded data values as needed
  4. update the HTML to include the new field

And then follow the steps in the preceding section to deploy the modified application to the device and push the application up to an ODK Aggregate server.

Modifying the data entry form

Return to your cmd window and once again launch the ODK 2.0 ApplicationDesigner environment (and a Chrome browser) by typing:

grunt

Now, Open a file browser and navigate to the directory where you downloaded the Application Designer.

Then navigate within that directory to app->tables->geotagger

Rename the properties.csv and definition.csv files in this directory to orig.properties.csv and orig.definition.csv. These are the initialization files needed by ODK Tables and they will need to be regenerated because we are altering the data table to incorporate an additional question.

Then navigate within that directory to forms->geotagger

Open the geotagger.xlsx file in Excel (or OpenOffice).

This is the form definition used by ODK Survey.

We will be adding a question to ask the user what direction they were facing when they took the photo. For this example, we will be just collecting a text response. A more realistic modification might restrict the user to a set of choices (North, Northwest, West, Soutwest, South, etc.).

On the survey worksheet, after the image-capture prompt, add a row that looks like the following.  Save the file.

type name display.text display.hint
string Direction Image Direction Enter the direction in which the photo was taken (North, South, East, West, etc.)

Save your changes.

Now go back to the Application Designer.

Click on the tab that says "XLSX Converter".

Drag and drop the geotagger.xlsx file onto this screen.

You should now see some json in the output window.  Hit the "Save to File System" button. This will write this into the formDef.json file in the same location as the geotagger.xlsx file. If this save fails, save it manually to another location and copy the file over.

Go back to the Chrome Browser and click on the "Preview" tab.

Click on "Purge Database".

This will delete the earlier "Geotagger" data table. Select "Geotagger" if you do not already have that form open. Create a new instance of "Geotagger" and advance through it, confirming that the new question is displayed. Note that the date and description are required fields and will generate error pop-ups. If you get any other error pop-ups, click the "Purge Database" button. It may take several attempts before the database is successfully purged. In the worst case, you may need to refresh the browser window (to return to the initial form-selection screen), and "Purge Database" there before entering the "Geotagger" form (this will be cleaned up in a future release).

You have now successfully modified the form.

Updating the Initialization Files

Fortunately, because the geotagger formId matches the tableId, by simply using the Save to Filesystem button on the XLSXConverter, we can regenerate the definition.csv and properties.csv files for this form.

Open the orig.properties.csv and properties.csv files in Excel.

We need to preserve the settings that ODK Tables used when rendering the map view and launching ODK Survey, while incorporating the changes that added the new column to the table.

At the moment, this is a manual process. The general steps are:

  1. Copy all rows whose first column is neither Column nor Table over to the new properties.csv file.
  2. Within the set of Table rows, copy the rows whose 3rd column (_key) match:
    • defaultViewType
    • detailViewFileName
    • keyColorRuleType
    • listViewFileName
    • mapListViewFileName

Once these rows are copied into the new properties.csv file, deploy the application to your device and launch ODK Tables.

Confirm that after it is initialized (click on the three-row icon to trigger initialization), you can access the app as you could before -- i.e., that it shows in map view, that the list displays as it did before, etc. If anything is amiss, it is because the properties were not correctly copied over.

Updating the Preloaded Data

When an ODK Tables application first starts up, it reads the assets/tables.init file. That file identifies csv files that should be imported into the data tables upon first start-up. The description of this file is here.

In this example application, the file being imported is assets/csv/geotagger.updated.csv. If we wanted to, we could edit this file, add a column for the new data field (Direction), and supply values for this field for all of the data rows that form the initial set of "Geotagger" locations.

Updating the HTML files

There are two areas where image information is displayed, one is in the list view, where you can expand or collapse an item, and the other is in the detail view, which is opened when you click or tap on an expanded item in the list view. We will only modify this detail view to report the image direction; a more comprehensive edit would likely also update the expanded item in within the list view.

To determine all the HTML files, we can begin with the files referenced in the properties.csv file we just finished editing. Looking again at that file, we see 3 files referenced:

  • tables/geotagger/html/geo_list.html
  • tables/geotagger/html/geo_list_thumbnail.html
  • tables/geotagger/html/geo_detail.html

Each of these files, or the javascript within them, might open or reference other files that might need to be updated. The above files are simply the ones we know are reachable. In general, files for displaying table-specific data are under the tables/tableid directory. In this example, we will just modify the last of these files.

Open a file browser and navigate to the directory where you downloaded the Application Designer.

Then navigate within that directory to app->tables->geotagger->html

Open the geo_detail.html in a text editor. Navigate down to the bottom of the display() javascript function (to line 38). And add before the closing bracket:

          var dir = data.get("Direction");
          document.getElementById("DIR").innerHTML = dir;

Next, insert a line defining the DIR element above the "Latitude" line in the HTML body region. e.g.,

        <h1><span id="TITLE"></span></h1>
        <p>Image Direction: <span id="DIR"></span></p>
        <p>Latitude: <span id="FIELD_1"></span></p>

Save the file.

Once again, push the application to the device.

Confirm that when you expand a item in the map list window, and then tap on that expanded item, that it now shows "Image Direction:". I.e.,

Congratulations, you have successfully modified this ODK 2.0 application to add a new data field and display it as a field in the HTML detail-view page.

You could now reset your server and start collecting geopoints with the new image direction field.

Next Steps

We provide several examples of different application experiences are:

  • Survey - legacy 1.x application try this. Is an example of a survey application that downloads and publishes data using the legacy ODK 1.x (ODK Collect) mechanisms.
  • Survey - ODK 2.0 synchronization try this. Is an example of a simple survey application that supports dynamic creation and update of data records using the ODK 2.0 synchronization mechanism.
  • Tables - wrapper containing four demo applications try this. Is an example of four different styles of applications that could be implemented using the tools.
  • Tables - this tutorial. This is just the 'Geotagger' demo app extracted from the wrapper application, above.

To understand what the range of possibilities are for these ODK 2.0 applications, it is recommended that you play around with the other 3 demonstration applications, above.

The full ODK ApplicationDesigner zip, available here contains many more example forms, including forms demonstrating linking to sub-forms, layouts using custom HTML templates and custom CSS styles.

The XLSX file format and the supplied prompt types are described here. The tools allow arbitrary customization and the definition of new prompt types. Those capabilities are not yet documented, but the intent is for someone with Javascript skills to be able to define new prompt types.