2.0 Getting Started - rev 206

The documentation on this page corresponds to rev 206 (August 2016) of the tools suite. See Older Versions for documentation on earlier releases of this suite.

The 2.0 ODK tools are intended to address limitations of the existing tool set. The 2.0 Tools Suite consists of:

  1. ODK Services - handles database access, file access, and data synchronization services between all of the ODK 2.0 applications. It also allows you to synchronize data collected by the ODK 2.0 tools using the 2.0 protocol with an ODK Aggregate instance.
  2. ODK Survey - a data collection application based upon HTML, CSS and Javascript.
  3. ODK Tables - a data collection and visualization application running on your device.
  4. ODK Scan - a mark-sense form scanning application running on your device.
  5. ODK Aggregate - a ready-to-deploy server and data repository with enhancements to support bi-directional data synchronization across disconnected devices.
  6. ODK Application Designer - a design environment for creating, customizing, and previewing your forms.
  7. ODK Suitcase - a desktop tool for synchronizing data from an ODK 2.0 server so the data can be exported to CSV format.

This page provides a brief end-to-end walk-through of the ODK 2.0 tools. It will walk the user through all the steps necessary to complete the following tasks:

  1. Joining a device into an existing ODK 2.0 application hosted on an ODK Aggregate server.
  2. Migrating / Setting-up an ODK 2.0 application on a new ODK Aggregate server.
  3. Configuring a device with an ODK 2.0 application by setting up the ODK ApplicationDesigner environment and using it to deploy an application to your device.
  4. Modifying an ODK 2.0 application to add a new field to a form.

 

ODK Applications

The ODK 2.0 Android tools (ODK Survey, ODK Tables, ODK Services, ODK Scan, ODK Sensors Framework and various ODK Sensor implementations) are APKs that are designed to work together to create a coherent tailored application experience for an end-user.

Once the ODK Application Packager becomes available, we anticipate groups being able to build their own application APK that launches the ODK tools to create this coherent tailored application experience.

ODK 2.0 tools access configuration files and store data under sub-directories of the opendatakit directory on the external storage device (i.e., /sdcard/opendatakit). User applications constructed using the ODK 2.0 tools are identified by the name of the sub-directory holding those configuration and data files. Thus, /sdcard/opendatakit/mytestapp would contain all the files and data for the "mytestapp" application. The name of that sub-directory, "mytestapp," is referred to as the AppName of that application. The default AppName for the ODK tools is "default." However, when configured appropriately, the ODK tools can run under another AppName, accessing configuration and saving data in a different subdirectory under opendatakit.

This is handled in such a way that each user application is isolated from all other user applications, with separate configurations, data tables, and server settings. This allows one device to run multiple user applications built on top of the ODK 2.0 tools without any coordination among the teams developing those applications.

A major goal of the 2.0 tools was to eliminate the need for any software engineering skills (e.g., Java programming, Android software development environment, source code control systems) when designing user applications. The skills required to build a user application range from scripting a form definition in XLSX (similar to constructing ODK Collect forms using XLSX files processed by the XLSForm tool), to simple web programming -- modifying boilerplate HTML and Javascript for custom presentations of the collected data. Advanced web programmers can also easily implement entirely-custom webpages.

Joining a device to an Existing Aggregate Server

The steps for joining a device to an existing Aggregate server are very straightforward.

  1. Install the APKs your application uses.
  2. Launch the 'home screen' APK, either ODK Survey or ODK Tables.
  3. Click on the sync icon (two curved arrows) to launch the ODK Services sync activity in the context of your 'home screen' APK.
  4. Configure ODK Services to point to the ODK Aggregate instance you want to join.
  5. Choose 'Sync now' to make the device mirror the contents on that ODK Aggregate server.

Follow the steps described above to join the ODK Aggregate server hosting our simple demo, which uses ODK Tables as its 'home screen' APK. The detailed steps are:

  1. Download and install ODK Services, ODK Tables and ODK Survey. You will also need the Google Play Services APK (for accessing maps).
  2. Launch ODK Tables (the 'home screen' APK).
  3. Click on the sync icon (two curved arrows) to launch ODK Services.
  4. The default Sync Configuration should be https://open-data-kit.appspot.com and None (anonymous access). You will need to change that. It will also default to Fully Sync Attachments.
  5. Click on the three vertical dots in the menu bar, select Settings, then click on Server Settings.
  6. Click on Server URL and replace the default server with https://opendatakit-simpledemo.appspot.com then click OK.
  7. Back out of settings then choose Sync Now.

The synchronization process will now occur. If there is an error, check to make sure the server URL is correct, then choose Sync now again until it completes successfully.

Once successful, back out of ODK Services, returning to ODK Tables. And back out of ODK Tables.

Then relaunch ODK Tables.

Tour of the Simple Demo Application

You should now see the custom home screen for the Geotagger demo:
Geotagging Demo Home

This demo is based upon the geotagger data table and form. It allows users to record the date, time, GPS coordinates, description and picture of their current location.

When you launch the demo by clicking the blue launch button, you see a map showing the collected data points, indicated with markers. By clicking on a marker, you bring its data record to the top of the list of records above the map. Clicking on the record header will expand or contract that item to show the coordinates and photo of that location. For example, if we click on the Phinney Ridge marker, its color changes from blue to green, and, if we then touch the Phinney Ridge heading, it expands to show the coordinates and image of that location:
Phinney Ridge

You can add a new data record by choosing the '+' icon in top menu bar. This opens ODK Survey. Since ODK Survey is being opened for the first time, it will initialize itself.

BUG: There is currently a bug whereby after initialization, ODK Survey has lost the form that had been requested to be opened. This leaves ODK Survey displaying the default form, with a message about specifying a #form_path argument. Back out of ODK Survey, and re-select the '+' icon in ODK Tables. This will properly launch the Geotagger form and you will be able to add a new data point. This only happens when ODK Survey initializes (i.e., since ODK Survey has been initialized, if you save a new data point, return to ODK Tables again, and hit '+', you will not see this default form screen).

Advance through and finalize this form. Upon Finalizing the form, you will be returned to ODK Tables and its map view. You can then highlight the marker you added and view the image in the list view:

If you then click or tap in the list item details area (e.g., on the image), a detail view of the item will be displayed.

From here, if you were to choose the pencil icon, ODK Survey would be launched to edit this record.

You can also view the data in a list view or spreadsheet view by choosing the sheet icon in the menu bar and selecting the view you want:
View Type

These other views can be useful if you need to access and complete data records that do not yet have location data and cannot therefore be displayed on a map. Try these other views now.

Now back out of the geotagger table view and return to the custom home screen. Choose the three-horizontal-line icon on the top menu bar. This presents the Table Manager screen, and choose the sync (two curved arrows) icon. This opens up ODK Services in its sync activity. Sync your device with the server (choose Sync now). This will push your newly-added record to the server. You can see this by browsing to https://opendatakit-simpledemo.appspot.com click on the ODK Tables tab, choose the View Table sub-tab, and select the "geotagger" table.

If you then repeat these steps with a different device, you can see that the two devices can share and exchange data, and revisions to this data, whenever they synchronize to the server.

During this process, there are two problem-resolution screens you are likely to encounter:

  • Checkpoint Resolution - if ODK Survey exits without the user explicitly saving their additions or changes.
  • Conflict Resolution - if ODK Services detects a change on the server to a data record that was also changed on the device.

Checkpoint Resolution

The checkpoint resolution screen is most easily triggered if you choose the '+' icon then back out of ODK Survey:
Checkpoint Resolution
When presented with this screen, there are three choices: cancel and continue editing the form, ignore changes and discard the entire partially-filled-out form, or to save it even though it is incomplete. In this case, since there is no entered data for this record, we can ignore changes.

In rare cases, a second form of checkpoint resolution screen can be triggered. This most often happens if ODK Survey experiences a failure and closes. In this case, you may have several data records with unsaved "checkpoint" changes (changes that the user has not explicitly saved as incomplete or finalized). This will lead to a screen like:
Checkpoint List
Clicking a row will display details about that individual checkpoint. i.e.,
Checkpoint Detail
In all of these screens, you can choose whether to save the changes as incomplete or to discard them.

Conflict Resolution

The conflict resolution screen is triggered when another device has edited one or more rows and synchronized its changes to the server before your edits to those same rows have been synchronized. In this case, your synchronization attempt will end with an error, and a Conflicts detected error will appear:
Modification Conflict
Once you click OK, the conflict resolution screen will be presented. If there are multiple rows in conflict, this screen will display the rows that are in conflict:
Conflict List
Clicking a row will display details about the conflict. i.e.,
Conflict Detail
And if only a single row is in conflict, the list-of-rows screen will be bypassed.

The conflict details screen displays the values of the field(s) in conflict, with the field value on the device (Local) appearing first. In this case, the Description field is in conflict. The device has "Kite hill at Gasworks" and the server has "Kite Hill ... Gasworks". You can select either to take your device values (Take Local Version) or take the server's values (Take Server Version) or pick-and-choose among the changes and merge them (the Merge Changes as Indicated Below button will be enabled after all fields have had either their Local or Server value picked for the merge). After selecting the local version or choosing to merge, you must again synchronize with the server to push that change up to the server.

This concludes the tour of the Geotagger example application's screens, and the functionality within ODK Tables. We will now turn to setting up our own ODK Aggregate server, and setting the application up to run on that server.

Migrating / Setting-up an ODK 2.0 application

Now that we have seen how a device can join an already-configured application, and synchronize its view of the data with the ODK Aggregate server hosting the application, it is time to set up our own ODK Aggregate server.

The starting point for this is to have a fully configured application on your device. Only after you have your device configured as you want it to appear, would you proceed with the following steps. In this case, we already have the device configured with the Geotagger demo, so let's proceed to create an ODK Aggregate server and configure it to serve that demo to your devices.

Setting up the ODK Aggregate server

Follow the instructions for installing ODK Aggregate here. You must install the ODK Aggregate v1.4.12 release. Until we have a release candidate of the ODK 2.0 tools, please assume that the specific device APKs are operable only with a specific ODK Aggregate release. e.g., the rev 206 tools are compatible with the ODK Aggregate v1.4.12 release.

Once you have installed ODK Aggregate, log in with your super-user account. This is described here.

Once logged in, enable the ODK Tables extensions as described here. You should grant the User account on your device the Administer Tables permissions.

Resetting the Application on the Server

Resetting the application on the ODK Aggregate server will push the application configuration on your device up to your server, replacing the configuration that is already on your server. Once the configuration is updated, data tables on the server and device will be sync'd. This process does not destroy data on the server, but instead merges changes on the client with any existing data tables on the server (this enables you to update your configuration without worrying about damaging or destroying the data already captured on the server).

Return to your device, start ODK Tables:

  1. click the diminishing-lines icon to leave the custom home screen.
  2. click the sync (two curved arrows) icon to launch ODK Services onto the sync screen.
  3. choose Settings from the menu.
  4. choose Server Settings
  5. edit the Server URL to be the URL for this newly configured ODK Aggregate server (e.g., https://myodk-test.appspot.com).
  6. click on Server Sign-on Credential and choose Username.
  7. choose Username and enter the superuser username for your ODK Aggregate server
  8. choose Server Password and enter the ODK Aggregate server password for that superuser username.
  9. click the back button to return to the sync screen
  10. click on Reset App Server to push your device configuration up to your ODK Aggregate server

After this has completed, you have created your own server that replicates the configuration and contents of the https://opendatakit-simpledemo.appspot.com site. Congratulations!

Any device with a user account with Administer Tables permissions can do this. If you configure a device with a user account (or Anonymous user) with just the Synchronize Tables permissions, they will not be able to Reset App Server and will just be able to Sync Now and join into the existing ODK 2.0 application on this ODK Aggregate server.

BUG: If you give the anonymousUser the Synchronize Tables permission, then you will not be able to Reset App Server. The work-around is to revoke that privilege from the anonymousUser before you reset the app server, and to re-instate the privilege afterwards.

Configuring your Device with an Application

Next, we will work through the steps to configure your device with an ODK 2.0 application (rather than downloading an existing application from a server).

This task begins with setting up the ODK 2.0 Application Designer on your computer.

For the purposes of this tutorial, we have created a copy of the Application Designer that only contains the files for this Geotagger example (it is otherwise identical).

Setting up ODK Application Designer

Read the Summary and Overview sections here to get a sense of the features and functionality of the ODK 2.0 Application Designer environment (we will install it below).

Install the necessary supporting software on your computer as described here; instead of downloading the full ODK Application Designer, download the simplified version for simpledemo from here.

Finally, launch the ODK 2.0 Application Designer as described here.

If successful, the cmd window (on Windows) should display some status messages. Below is a screen-shot of my cmd window beginning with a 'dir' of the contents of the directory, and running 'grunt' in that directory:

And a Chrome browser window should open to display:

If a Chrome browser does not open, try manually launching it and opening http://localhost:8000/index.html.

You can further verify that the Application Designer works by clicking on the "Geotagger" button, then clicking on "Follow link". This opens the Geotagger form on your computer, and simulates all the features available to you on your device.

You can also try other things, like choosing different device dimensions to see how the form renders on different screen geometries.

We will return to this design environment later.

Deploying to the Device

Now that we have the design environment installed and functioning, and because that environment has a copy of the fully-configured Geotagger application that is running on https://opendatakit-simpledemo.appspot.com (minus any data that users have submitted to the server), we can work through the steps of deploying that application to your device, and then resetting your server to push that configuration up to your server.

First, ensure that ODK Tables, ODK Survey and ODK Services are all stopped. Do this by going to your device's Settings page, choose Apps, select each of these ODK tools and confirm that the Force Stop button is disabled. If it is active, click on it to force the application to stop; in general, you should stop ODK Services last, as all the other ODK 2.0 tools use that and may trigger it to restart if any of them are still active.

While also in the Settings, confirm that your device has USB debugging enabled. This checkbox is in different places on different devices and may be hidden by default on some. See here for instructions.

Next, launch OI File Manager and delete the contents of the opendatakit directory on the device. For newer devices, the directory to delete is storage/emulated/0/opendatakit This may vary among devices and Android operating system versions. This directory can contain a range of sub-directories, but it will contain at least default. Within that default sub-directory (or any other application directory), there will config, data, output and system subfolders. Delete all the folders and files within the opendatakit directory.

Your device now has no ODK 2.0 applications on it.

Return to the cmd window on your computer. Control-C to stop the grunt command that popped-open the Chrome browser. On Windows, you will be asked to confirm this (Terminate batch job (Y/N)?). Enter Y to confirm.

Connect your device to your computer via USB. Wait for the storage connection to be established (on Windows, this will generally pop up a file browser or an options box that enables you to select a file browser).

At the command prompt, type:

grunt adbpush-default-app

This pushes the configured ODK 2.0 application within this ODK 2.0 Application Designer directory to your device. Because this is a stripped-down version of the Application Designer that only contains the simpledemo files, this will copy only those files to the device. When you issue this command, the cmd window will display a long series of commands and conclude with a display of overall progress and timings:

Now, on your device, launch ODK Tables.

This will initiate the configuration of ODK Tables and conclude with a Configuration Summary pop-up reporting that everything was imported successfully. Click OK.

Everything should now appear as it did with the application you first joined on https://opendatakit-simpledemo.appspot.com, except you will only have the data rows configured by the ODK 2.0 ApplicationDesigner zip, and not any added or modified since that time.

Resetting the Server

Once you have the application running on the device, you will typically need to reset the contents of the application server. While the Reset App Server button on the device can shuffle the various supporting files between the device and the server, it will not destroy data tables that already exist on the server; this is intentional -- we want to minimize the potential for accidental loss of data.

Therefore, whenever you are developing an application, and have found a need to add a new column to an existing table, you will need to manually delete the data tables from the server before using the Reset App Server button from the device.

Open a browser window to the server, log in with a user that has Administer Tables or Site Admin privileges.

Navigate to the ODK Tables / Current Tables sub-tab.

Delete each of the tables here. In this case, there will be only one, Geotagger. The server will now have a set of App-Level files but no data tables, forms for those tables, or data files. Except for the app-level files, it is 'clean'

NOTE: If your table has a large number of configuration files or data rows, the server may time out during the deletion process. In this case, the next time you try to create the table on the server, it will resume the deletion process, and potentially time out again until such time as it is able to finish the deletion; only then will it re-create the table.

Now, from your device, launch ODK Tables, click on the sync icon (two curved arrows) to launch ODK Services, and choose Reset App Server.

The synchronization process will create the tables and push your content up to this server. Note that the server now only contains the data rows present on the device -- it no longer has any of the additional data records from the demo site.

You have now successfully set up the Application Designer, used it to deploy an application to a device, and, from that device, configured an ODK Aggregate server to supply that application to other devices you join to that server.

Modifying an ODK 2.0 application

The final task is to modify the Geotagger example by adding a new data field to it.

NOTE: There are many manual steps to this process that will be simplified or eliminated in future releases.

The overall development process has three steps:

  1. revise the data entry form
  2. update the initialization files needed by ODK Tables
  3. update the preloaded data values as needed
  4. update the HTML to include the new field

And then follow the steps in the preceding section to deploy the modified application to the device and push the application up to an ODK Aggregate server.

Modifying the data entry form

Return to your cmd window and once again launch the ODK 2.0 ApplicationDesigner environment (and a Chrome browser) by typing:

grunt

Now, Open a file browser and navigate to the directory where you downloaded the Application Designer.

Then navigate within that directory to app/config/tables/geotagger

Rename the properties.csv and definition.csv files in this directory to orig.properties.csv and orig.definition.csv. These were the initialization files needed by ODK Tables and they will need to be regenerated because we are altering the data table to incorporate an additional question.

Navigate within that directory to app/config/tables/geotagger/forms/geotagger

Open the geotagger.xlsx file in Excel (or OpenOffice).

This is the form definition used by ODK Survey.

We will be adding a question to ask the user what direction they were facing when they took the photo. For this example, we will be just collecting a text response. A more realistic modification might restrict the user to a set of choices (North, Northwest, West, Soutwest, South, etc.).

On the survey worksheet, after the image-capture prompt, add a row that looks like the following. Save the file.

type name display.text display.hint
string Direction Image Direction Enter the direction in which the photo was taken (North, South, East, West, etc.)

Save your changes.

Now go back to the Application Designer.

Click on the tab that says "XLSX Converter".

Choose this XLSX file or use your file browser to drag and drop the geotagger.xlsx file onto this screen (dragging and dropping is not supported on all operating systems).

You should now see some json in the output window. Hit the "Save to File System" button. This will display 3 pop-up notifications announcing that the ApplicationDesigner is (1) writing the updated ODK Survey form definition into the formDef.json file in the same location as the geotagger.xlsx file and, because the form id is the same as the table id, it is also updating (2) the definition.csv and (3) the properties.csv files.

Go back to the Chrome Browser and click on the "Preview" tab.

Click on "Purge Database".

This will delete the earlier "Geotagger" data table -- a necessary step because we are adding a "Direction" column to that data table. Select "Geotagger" if you do not already have that form open.

Create a new instance of "Geotagger" and advance through it (this will create the data table with the new "Direction" column); confirm that the new question is displayed. Note that the date and description are required fields and will generate error pop-ups if you attempt to advance through those prompts without supply a value.

If, during this process, you get any other error pop-ups, click the "Purge Database" button. It may take several attempts before the database is successfully purged. In the worst case, you may need to refresh the browser window (to return to the initial form-selection screen), and "Purge Database" there before entering the "Geotagger" form (this will be cleaned up in a future release).

You have now successfully modified the form.

Updating the Initialization Files

Fortunately, because the geotagger formId matches the tableId, by simply using the Save to Filesystem button on the XLSXConverter, the tool will automatically regenerate the definition.csv and properties.csv files for this form. Furthermore, the configuration that ODK Tables uses to specify what HTML files to use for the list, detail and map views are all specified within the XLSX file on the properties sheet. No manual actions are required!

Now, force close all the ODK tools, delete your /sdcard/opendatakit/default directory and, deploy your updated application to your device; launch ODK Tables to initialize and load your application.

Confirm that when you edit a data row that you are now asked for the direction in which the photo was taken.

Updating the Preloaded Data

At this point, we have added the new field to the data table, but have not yet updated the initial set of "Geotagger" locations with values for that field.

Return to your ApplicationDesigner directory. Recall that when an ODK Tables application first starts up, it reads the assets/tables.init file; that file identifies csv files within config/assets/csv that should be imported into the data tables upon first start-up. The description of this file is here.

In this example application, the file being imported is config/assets/csv/geotagger.updated.csv. If we wanted to, we could edit this file, add a column for the new data field (Direction), and supply values for this field for all of the data rows that form the initial set of "Geotagger" locations.

Alternatively, we can return to the device and use the CSV export functionality within ODK Tables to export the csv file (into /sdcard/opendatakit/default/output/csv); then pull it off of the device and overwrite the csv file under the ApplicationDesigner at app/config/assets/csv/geotagger.updated.csv. Finally, open that file and fill in values for the Direction column.

Note that some CSV editors, like Office or OpenOffice, may convert or alter the content inappropriately when you save changes. If your edits cause the device to fail to initialize the data fields, you may need to make this edit manually using a less-sophisticated tool or choose different options when saving your changes.

Updating the HTML files

There are two areas where image information is displayed, one is in the list view, where you can expand or collapse an item, and the other is in the detail view, which is opened when you click or tap on an expanded item in the list view. We will only modify this detail view to report the image direction; a more comprehensive edit would likely also update the expanded item within the list view.

To determine all the HTML files, we can begin with the files referenced in the properties.csv file we just finished editing. Looking again at that file, we see 3 files referenced:

  • tables/geotagger/html/geo_list.html
  • tables/geotagger/html/geo_list_thumbnail.html
  • tables/geotagger/html/geo_detail.html

Each of these files, or the javascript within them, might open or reference other files that might need to be updated. The above files are simply the ones we know are reachable. In general, files for displaying table-specific data are under the config/tables/tableid directory. In this example, we will just modify the last of these files and its associated javascript file.

Open a file browser and navigate to the directory where you downloaded the Application Designer.

Then navigate within that directory to app/config/tables/geotagger/html

Open geo_detail.html in a text editor. Insert a line that defines a DIR element above the "Latitude" line in the HTML body region. This will be where we will display the value of the "Direction" field. e.g.,

        <h1><span id="TITLE"></span></h1>
        <p>Image Direction: <span id="DIR"></span></p>
        <p>Latitude: <span id="FIELD_1"></span></p>

Save the file.

Now, navigate to app/config/tables/geotagger/js

Open geo_detail.js in a text editor. Navigate down to the bottom of the display() javascript function (to line 44). And add before the closing bracket:

          var dir = geoDetailResultSet.get("Direction");
          document.getElementById("DIR").innerHTML = dir;

Save the file.

Once again, push the application to the device.

Confirm that when you expand a item in the map list window, and then tap on that expanded item, that it now shows "Image Direction:". I.e.,

Congratulations, you have successfully modified this ODK 2.0 application to add a new data field and display it as a field in the HTML detail-view page.

You could now log onto your server, delete the geotagger table, reset your server, and start collecting geopoints with the new image direction field.

Next Steps

The XLSX file format and the supplied prompt types are described here. The tools allow arbitrary customization and the definition of new prompt types. Those capabilities are not yet documented, but the intent is for someone with Javascript skills to be able to define new prompt types.