Application Designer - rev 206

The documentation on this page corresponds to rev 206 (August 2016) of the tools suite. See Older Versions for documentation on earlier releases of this suite.

Release Designation

ODK Application Designer rev 206 (August 2016) is designated Alpha software.

See Release Designations for the meaning of this designation.

Summary

The ODK Application Designer, when combined with Excel or OpenOffice for form design, the Chrome browser, and your favorite editor for template design, provides a development environment for the application designer.

In the context of the ODK 2.0 tools, application design consists of:

  • designing the forms used in data collection (by ODK Survey)
  • designing the HTML landing pages and screens used for navigating, curating, and visualizing that data on your Android device (within ODK Tables).
  • customizing the look-and-feel of both of these via customized images, logos, and CSS rules.
  • designing mark-sense forms for paper-based data entry (by ODK Scan)

And, if you are only interested in data collection, or if you are only interested in data dissemination and visualization, you can focus your application design efforts on just one or two tools (ODK Scan and ODK Survey, or ODK Tables, respectively). The tools operate independently -- you are not required to deploy all tools to your Android device, or use all tools. Simply select the combination or individual tool that fits your needs. All of these tools use ODK Services to access the database, sync to a server, and vend HTML files.

The major goals of the ODK Application Designer are:

  1. Simplify the form-design process by providing a preview of your form with the same screen geometry as your target Android device. You no longer have to copy each iteration of your form onto your device to see how it will look or fit on a smaller screen.
  2. Simplify the design and testing of customized list- and detail- views in ODK Tables, and in the design of graphical representations of data within ODK Tables
  3. Simplify the customization of the look-and-feel of your forms through a simple visual theme editor / generator where your modifications can be immediately viewed and the resulting CSS styles or theme can be saved to a file for later incorporation into your application deployment.
  4. Simplify the conversion of the XLSX file into a form definition by providing a drag-and-drop conversion app running locally on your desktop.
  5. Enable the creation of mark-sense forms (ODK Scan Form Designer) that can be scanned by your Android device for data input. The resulting data is available to the other ODK 2.0 tools without need to communicate with a remote server.
  6. (functionality withdrawn from ODK Survey in rev 202) A stub XML form definition generator. The form definition can be uploaded to ODK Aggregate. This might allow ODK Suitcase to submit data into ODK Aggregate's XForms submission processing and publishing pipeline (the one used by ODK Collect). This might enable organizations to leverage the 2.0 tools on their Android devices while still retaining their established backend procedures and processes. i.e., you can be early adopters of these new Android device technologies while retaining the stability and predictability of the well-established, proven, ODK Aggregate XForms processing pipeline.

The general layout of this document is:

 

Overview

This section presents a brief overview of the features of the ODK Application Designer. To fully utilize the system, you must understand the layout of the files on the Android device and the specific steps to create them.

These details, and how to set up the ODK Application Designer tool, will be covered in the subsequent sections.

The ODK Application Designer is accessed through a Chrome browser. Once launched, it opens Chrome to display:

Initial Screen

This screen has 6 tabs:

  • Preview (shown) - used to preview ODK Survey forms and ODK Tables list-, detail- and graph- views (future). Displays these within a user-selected device geometry.
  • Customize - a visual style and visual theme editor. This editor immediately shows the effects of changes to specific settings in your CSS file. This functionality is undergoing changes and not recommended.
  • XLSX Converter - converts the XLSX description of a form into a formDef.json representation used by ODK Survey.
  • Legacy XML - converts the formDef.json file into a stub XForms XML file that can be uploaded to ODK Aggregate to support publishing data into the XForms-based submissions pathways used by ODK Collect. This has been deprecated.
  • File Browser - enables browsing of the directory structure that will exist on your Android device so that you can access or view other files (currently necessary for accessing the ODK Tables list- and detail- views).
  • Scan Form Designer - drag-and-drop mark-sense form designer tool.

 

Preview

The Preview tab (shown above) has several controls:

  • Launch Page -- what webpage to launch in the window below.
  • Purge Database -- during development, if you are adding new fields or changing their data types, you will need to purge the database so that the database structure can be re-generated with the proper fields and data types.
  • Device Dimensions -- what dimensions to make the window below.
  • Back Button -- this is generally only useful when viewing ODK Tables web pages. It may cause unpredictable results when used while displaying ODK Survey forms.

The Launch Page selections are expected to grow with later updates. Presently, you can select among:

  1. ODK Survey Framework Page - opens the formDef.json in the Android device's application frameworks folder (e.g., either /sdcard/opendatakit/survey/config/assets/framework/forms/framework or /sdcard/opendatakit/tables/config/assets/framework/forms/framework). The contents of this form are defined by the framework.xlsx file in that same directory.
  2. ODK Tables Graph Page - is in active development and not recommended.
  3. ODK Survey Step-Through Test -is in active development and not recommended.
  4. ODK Tables API Test Page -is in active development and not recommended.

For example, if you click on the household test form, and click the Follow Link button on the next screen, the Household Survey form is launched, yielding this screen:
After following link to Household Survey

You can navigate through forms, enter and exit sub-forms, and save results just like on your Android device.

The development environment does not allow you to submit data to a server (future work).

Customize

The Customize tab contains the CSS style and theme generator:
CSS Style and Theme Generator
Using this tool, you can change background colors, fonts, and other settings affecting the appearance of a form. The changes are reflected immediately in the form shown to the left of the toolbar.

This functionality is under active development and not currently recommended.

XLSX Converter

The XLSX Converter tab contains the conversion tool that transforms XLSX files produced by Excel or OpenOffice into the formDef.json file used by ODK Survey:
XLSXConverter

See ODK XLSXConverter documentation for more information about this tool.

Legacy XML

The Legacy XML tab contains the conversion tool that transforms a formDef.json file into a Stub Form Definition that can be uploaded into ODK Aggregate to enable ODK Survey to send submissions into ODK Aggregate's XForms XML submissions path (the one used by ODK Collect):
Legacy XML generator
This functionality is not functional and may be removed in a future release.

See ODK Legacy XML documentation for more information about this tool.

File Browser

The File Browser tab provides a view into what will become the application's directory on the phone.
File Browser

You can use this view to navigate to the HTML list view pages for your tables. Rendering the list view pages is currently broken because underlying internal files rely on requirejs for database access, and these HTML files do not generally use that framework library.

Scan Form Designer

The Scan Form Designer tab presents a drag-and-drop editor for mark-sense form creation.
Scan Form Designer

See ODK Scan Form Designer documentation for more information about this tool.

 

Getting Started

You must install the following software on your computer:

  • Java - Java is required by the Android SDK
  • Chrome - Google's Chrome browser. Download and install it from Chrome.
  • NodeJS- a framework for easily building fast, scalable applications. Download Version 6.2.2 or higher and install it from NodeJS
  • Grunt - a task-based scripting environment (installation is described below).
  • Android SDK - the software development kit for Android devices (installation is described below).
  • ODK Application Designer v2.0.zip - a zip containing this tool (installation is described below).

It is tricky to foresee all the issues that can crop up on many different machines and setups. If something in this process does not go as expected, please tell us. Others are likely encountering the same problem, and we can only make these instructions more complete if people tell us when the instructions are incomplete!

Java

Make sure Java 7 or higher is installed on the computer you plan to use. If it is not, download and install it; if that page does not have a link to older versions, Java 7 is also available here. If you are using MacOSX, it may require special care and attention. See MacOSX Java install and MacOSX Java FAQ.

NodeJS

You must use Version 6.2.2 or higher. To avoid directory path problems on Windows, we require npm version 3, and that is only available on this node release (or higher).

Follow the instructions here to install NodeJS.

For Mac/Unix:

After installing NodeJS, open a terminal (which you can do by clicking the spotlight in the top right corner of the screen, typing terminal, and clicking the program named "Terminal") and type npm --version. If a number is not displayed, but you instead receive a message that the command npm cannot be found, you will have to perform some additional configuration.

As of this writing, by default NodeJS installs its commands into /usr/local/bin/. In the terminal, type ls /usr/local/bin/npm. If this command outputs something like /usr/local/bin/npm, but you are still unable to run npm --version, try running /usr/local/bin/npm --version. If this is successful, then npm is successfully installed, and you will just have to add /usr/local/bin/ to your system PATH variable (see below).

If the command ls /usr/local/bin/npm outputs a message telling you permission is denied, then you will have to change the ownership of the /usr/local/ and /usr/local/bin/ directories. On Mac, follow the instructions here to take ownership of these directories, or to at least give yourself read permission. On other Unix systems, use the chown command or the user-interface appropriate to your distribution to do so.

Grunt

After installing NodeJS, install Grunt by doing the following:

    1. On Windows, open a cmd window (go to Start Menu, search for cmd and open it); on MacOSX, open a Terminal window.
    2. Within this window, type
npm install -g grunt-cli

Note: these installation steps are copied from here. If the above command is unsuccessful, some machines may need to append sudo at the beginning of the command. If grunt is successfully installed, the following command:

grunt --version

Should display the installed version of grunt. e.g., grunt-cli v1.2.0.

Android SDK

To install the Android SDK:

  1. Browse to here.
  2. Scroll down on this page to the "Get just the command line tools" section.
  3. Within that section, download the appropriate file(s) based on your operating system.
  4. Accept the license agreement
  5. Wait for the install of the SDK Tools to complete. Windows will need to manually run the .exe file previously downloaded to start installation.
  6. Run the SDK Manager
    • On Windows, it is available in the Start Menu under Android SDK Tools
    • On Mac/Unix, open the SDK folder you downloaded above. In the bin/ or tools/ directories (on some versions it is in both places--it doesn't matter which you use), double click the file called android.
  7. Select the latest versions of the following packages (by checking their checkboxes):
    • Android SDK Tools
    • Android Platform-tools
    • Android Build-tools
  8. If extra packages are selected, you may unselect them before installation.
  9. Click the Install 3 packages button in the lower right corner of the screen.
  10. A licensing pop-up dialog will appear. Accept the license agreement(s) by selecting the 'Accept License' option. If there are multiple licenses, you may need to select each license in the 'Packages' window on this dialog and check this 'Accept License' option for each of them before the 'Install' button will become enabled.
  11. Click the 'Install' button on that dialog to begin the installation process.

Among many other things, this will install the Android Debug Bridge software on your computer. This tool enables the scripted pushing of files and APKs down to your Android device. See adb (Android Debug Bridge) for a listing of its capabilities.

Next, on Windows open a cmd window (open the Start menu, type cmd in the search box, select and open it) , and on Mac/Unix open a terminal window. Type:

adb version

If there is an error complaining about Java not being installed, you will need to close this cmd/terminal window and download and install Java. After installing Java, open a new cmd/terminal window and type this command again.

If this displays a version string, then your installation is complete; skip to the next section.

Otherwise, if adb is not found, then you need to add it to the PATH variable of your system.

For Windows:

    1. open a Windows File Explorer and navigate to the location of your Android SDK. This will typically be at one of: C:\\Users\\your_username\\android-sdks or C:\\Program Files\\Android\\android-sdk or C:\\Program Files (x86)\\Android\\android-sdk
    2. navigate into the platform-tools folder.
    3. Click in the file path at the top of the File Explorer window. The path will become a selected text string. Copy it into your copy-and-paste buffer.
    4. open the Start menu,
    5. right-click on Computer,
    6. choose Properties. The System Control Panel screen opens.
    7. select Advanced system setting on the left sidebar. The System Properties dialog opens.
    8. click on the Environment Variables... button at the bottom of the screen. The Environment Variables dialog opens.
    9. select the Path variable in the bottom System variables scroll window.
    10. click Edit...
    11. click into the Variable value text box
    12. press your End key to move to the very end of this extremely long string.
    13. enter ; and paste the platform-tools directory path after that semicolon.
    14. click on OK and exit all of the windows.
    15. verify that you have made the change by closing all cmd windows and open a new one (so it picks up the change), and type
adb version

You should now see the version of the adb tool. e.g., Android Debug Bridge version 1.0.31.

For Mac/Unix:

The PATH variable is nothing more than a default list of places the system looks for commands. Open a terminal.

Type echo ${PATH}, and you will see a colon-separated list of folders on your computer. (echo means just print whatever comes next, and the ${ } means that the system will treat PATH as a variable, not a program. You don't need to know this to follow these instructions, but knowledge is power.) For example, you might see something like this:

$ echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin

This means that when you type adb --version, the system will look for the command called adb in the directories /usr/local/bin/, /usr/local/sbin/, /usr/bin/, and /bin/.

Note the location where you downloaded the Android SDK. It should contain a folder called platform-tools, which itself contains the program adb. If this was in the folder /Users/someuser/Desktop/android-sdk/ you should be able to run /Users/someuser/Desktop/android-sdk/platform-tools/adb --version. This works because we're telling the computer exactly where the program adb exists. By putting the platform-tools directory on the system's PATH variable, we will be able to just type adb and have the system find it in the /Users/someuser/Desktop/android-sdk/platform-tools/ directory.

This process is more involved on Mac/Unix than on Windows. Use a text editor (not Word, but something like TextEdit), select the option to open a file, and browse to your home directory. You can find your home directory by typing echo ~ in a terminal. (~ is a shortcut for the home directory.) Macs use a hidden file called .bash_profile in the home directory to set variables like PATH. Other unix systems use files like .bashrc. You might have to check the specifics for your distribution to know which you should use. Open the appropriate file. If the file does not already exist, create a new file that will be saved with the appropriate name in your home directory.

We want to add the location of the adb tool to your PATH while preserving the existing PATH information. Assuming that your adb program is in the /Users/someuser/Desktop/android-sdk/platform-tools/ directory, you would add the following command to the end of the .bash_profile file:

export PATH=${PATH}:/Users/someuser/Desktop/android-sdk/platform-tools

Save the file, close the terminal window, open a new terminal window, and type echo $PATH. You should see your old path with the new directory you added above, and you should now be able to run adb --version.

If you are going to be heavily customizing the look-and-feel of the application with a lot of external JavaScript libraries, you might also choose to install bower.

ODK Application Designer v2.0.zip

There are several different versions of the Application Designer zip:

  • Download the complete version from the Survey And Tables portion of our download page, here.
  • Download one of the minimal ODK Tables v2.0 … ApplicationDesigner.zip files for the various demo setups from the Survey and Tables Aux Files portion of our downloads page, here.

This documentation is for version rev 206.

Unzip whichever file you downloaded and move the resulting folder to somewhere other than your Downloads directory; e.g., to your Documents folder.

On Windows, you will be opening a 'cmd' window and changing your current directory ('cd'ing) into this directory every time you use this tool. It is therefore useful to create a shortcut that opens a 'cmd' window directly into this directory:

  1. Open a file browser and navigate to the unzipped directory containing a number of files and directories, including a Gruntfile.js.
  2. Click into the top location bar that displays the nested list of folders to this folder.
  3. Copy this path to the cut-and-paste buffer
  4. Now, move down to the list of files, right-click
  5. Select 'New...', 'Shortcut'
  6. Type 'cmd' for the location of the item.
  7. Click Next, and then Finish.
  8. Select this newly-created 'cmd.exe' shortcut and right-click
  9. Select 'Properties'
  10. Click on the 'Start in' text box, delete its contents, and paste the path to this folder.
  11. Click 'OK' to accept the change.
  12. Double-click the 'cmd.exe' shortcut to open a 'cmd' window.
  13. Confirm that it opens in the intended directory (you should see the full path to that directory displayed to the left of the blinking cursor).

On MacOSX, Terminal will open a new terminal window if you drag a folder (or pathname) onto the Terminal application icon, and you can also drag a folder to the tab bar of an existing window to create a new tab in that folder.

You have now completed the installation of the ODK Application Designer software.

 

Launching the Application Designer

The ODK Application Designer is both (1) a workspace containing the cohesive interacting set of forms and files you have created and (2) a set of tools used for that development.

We recommend unzipping and creating a new ODK Application Designer directory for each new set of ODK Survey forms, ODK Scan forms, and ODK Tables files that are not intended to be deployed as a cohesive unit. With the eventual availability of the ODK Application Packager, you will then be able to package each set of forms and files into a separately installable APK that you control. If you need to have several of these sets of forms and files co-resident on the same Android device, you would simply install the required set of your APKs on that device. The underlying ODK 2.0 tools will then keep each of these sets of forms and files isolated from each other.

To launch the application designer, open the 'cmd' shortcut or Terminal window onto the directory containing Gruntfile.js (the unzipped ODK Application Designer v2.0 directory) and type:

grunt

This should automatically open Chrome and display the Preview tab. When you need to stop the server, return to the command or terminal window where you typed grunt and press ctrl+c (that is the control and the letter c key at the same time). This will stop the process.

If you have Parallels or other virtualization software running, it might try to open Chrome in this system by default. If so, you should still be able to navigate to the url below.

If the Chrome browser does not open, try opening it yourself and browsing to http://localhost:8000/index.html.

If the page never times-out, but never loads (it remains blank or constantly spinning), then stop grunt and try this command instead:

grunt --verbose connect:livereload:keepalive

This will start grunt, but disable the file-change detection mechanisms that automatically reload an HTML page when it or any javascript file it uses has been modified. If you need to run the command in this manner, please add a comment to this issue so that we can gauge how widespread this problem is. Others have reported that uninstalling npm and node, and then re-installing them may correct the issue. If you try that and it works, please also add a comment to the above issue.

 

Directory Structure

There are many folders and files within the Application Designer directory. Fortunately, the only ones that are of interest for a non-software-developer are:

  • app/ - folder containing everything that will be pushed to the Android device.
  • Gruntfile.js - contains the definitions of tasks that push files to the Android device, launch the Chrome browser, and pull data and log files off the Android device.

Initially, you will only be concerned with the contents of your app/ directory -- the set of files that are placed on the Android device. As your sophistication grows, you may want to define your own grunt tasks to automate repetitive steps in your deployment and device management processes. Adding or modifying tasks is beyond the scope of this document; please refer to the grunt website (see Getting Started for the link to that site).

For completeness, here is the full list of the files and sub-folder in this directory. Again, you generally do not need to be concerned with the contents or specifics of any of these:

  • app/ - folder containing everything that will be pushed to the Android device.
  • convertToLegacy/ - contains the HTML and Javascript for the ODK Legacy XML stub XForms XML generation tool (accessed via the Legacy XML tab).
  • devEnv/ - contains the HTML for the 6 tabs of the Application Designer.
  • grunttemplates/ - contains template files used by grunt tasks.
  • node_modules/ - contains additional software installed by npm, such as external tools used by grunt.
  • scanFormDesigner/ - contains the scan form designer tool.
  • test/ - contains tests of the computer-based simulated device environment.
  • themeGenerator/ - contains the HTML and Javascript for the ODK ThemeGenerator CSS style and theme customization tool (accessed via the Customize tab).
  • xlsxconverter/ - contains the HTML and Javascript for the ODK XLSXConverter tool that converts XLSX form definitions into formDef.json files (accessed via the XLSXConverter tab).
  • .bowerrc - JSON configuration for the 'bower' tool.
  • .editorconfig - when your text editors are configured to use it, enables consistent formatting to files across all contributors to your application design. See EditorConfig.
  • .hgignore - source code management configuration.
  • .hgtags - source code management configuration.
  • .jshintrc - configuration for JSHint - a program that flags suspicious usage in programs written in JavaScript.
  • bower.json - used to control library management through bower. By default, the .bowerrc file has been configured to install these libraries in app/framework/libs so that you have access to them when your app is pushed to the phone.
  • deleteDefAndProp.sh - MacOSX shell script to traverse the relevant parts of the app/ directory and delete the definition.csv and properties.csv files.
  • Gruntfile.js - contains the definitions of tasks that push files to the Android device, launch the Chrome browser, and pull data and log files off the Android device.
  • index.html - the main HTML for the ODK Application Designer web page.
  • macGenConverter.js - MacOSX command-line wrapper for the XLSXConverter tool (converts a single XLSX file piped into stdin into a formDef.json on stdOut).
  • macGenFormDef.sh - MacOSX shell script to traverse relevant parts of the app/ directory and generate formDef.json files from XLSX files.
  • package.json - configuration information for npm.
  • README - description linking back to this document.

 

The app/ Folder

Everything in this folder mimics what is on the Android device. The directory looks as follows:

  • config - user-defined configuration for your application.
  • data - file attachments, and, on the device, the database.
  • output - on the device, logging files and exported csv and media files.
  • system - files managed by the ODK tools (do not modify).

The app/config/ Folder

This folder is sync'd to the device. It contains all of the form and table configuration files and initialization scripts. This is the sub-folder in which you will be primarily working.

This folder contains:

  • assets
  • tables

 

The app/config/assets Folder

  • css/ - contains the common CSS files for ODK Tables detail, list and home screens, and for app forms in ODK Survey (odk_survey.css).
  • csv/ - contains the data files to be initially read and loaded into the ODK Survey and Tables databases. NOTE: this functionality is currently not operational within ODK Survey.
  • fonts/ - contains the fonts used throughout the application.
  • framework/ - contains the framework.xlsx and other relevant framework files.
  • img/ - contains the images used throughout the application.
  • js - contains Javascript used by the ODK Tables custom home screen and/or the ODK Survey custom forms list. NOTE: this functionality is currently not operational within ODK Survey.
  • libs/ - contains the various libraries used throughout the application like jQuery and D3.
  • ratchet/ - contains the various files for the ratchet library.
  • tables.init - contains the initialization directives for which data (csv) files should be loaded at initial start-up of the ODK tools.
  • index.html - the HTML for the ODK Tables custom home screen, if it is enabled in the ODK Tables configuration settings.

 

The app/config/tables Folder

This folder has a predefined directory structure, but the content is entirely dependent upon the needs of your application.

The zip file for the ODK Application Designer populates this with all the subfolders used by each of the ODK Tables and the ODK Survey demonstration zip files. Ultimately, when you have completed your application design, this folder will contain none of these original folders but would instead contain only the folders which you have created.

Unlike ODK Collect, which stores each submission in a separate file, ODK Survey and ODK Tables store their combined collected submission data in data tables (one row per submission).

ODK Tables can display the contents of a table through one or more custom list views; it can display individual submissions through one or more custom detail views. Graphical views are simply list views in which the data is presentd graphically using a library such as d3. All of these custom views are defined here (see below for details).

ODK Survey, unlike ODK Collect, has the additional flexibility of supporting multiple forms to create, access and update data within a single common data table. This enables creating multi-stage workflows such as initial screenings and follow-ups, or registrations and status-updates (submission data can be editable, or not, based upon the form used at that workflow stage).

To accommodate these various capabilities, the tables directory is structured such that individual data tables each have their own directory within the tables directory. The table's table_id is the name of this sub-directory. When defining a new data table, begin with a form whose form id is the table id. See Creating and Loading a Form in ODK Survey for the steps to create a new table id.

The app/config/tables/table_id/ Folder

A canonical table_id sub-folder contains:

  • definition.csv - defines the data columns in this table. Generated when the form_id XLSX file underneath this table_id is processed by the XLSXConverter.
  • properties.csv - defines the appearance properites for this table. Example properties are the detail view HTML file name, the list view HTML file name, the default view type of the table, etc. Generated when the form_id XLSX file underneath this table_id is processed by the XLSXConverter.
  • forms/ - contains directories for each ODK Survey form that manipulates this table. The names of these sub-directories are the form ids of those forms. Within each sub-directory, there is a form_id.xlsx file defining the ODK Survey form and the formDef.json generated by the XLSXConverter when it processed that form definition file. If the form has form-specific images or media files, custom CSS, layouts, or prompt types, those files should reside within the form's sub-directory (nested sub-folders are permitted).
  • html/ the custom HTML files for the ODK Tables list and details views of the table's contents.
  • css/ - contains css files specific to this table.
  • js/ the JavaScript files needed for the custom ODK Tables HTML list and detail views (found in the html/ directory).

ODK Scan is currently split in where it stores its configuration for mark-sense forms. This will likely change and lead to additional sub-directories here.

The app/data Folder

The ODK Application Designer stores user data in this directory. The database itself is in the webDb directory. Any data files associated with a row in the database are stored within this folder under the tables/<table-id>/instances directory.

The app/output Folder

The ODK Application Designer provides various grunt tasks to pull files off the Android device. These files include json objects for debugging, exported csvs, and the database itself. The grunt tasks store these files here. There is also a logging directory which contains logs that are useful for debugging issues.

The app/system/ Folder

This folder contains the files that the ODK tools depend upon and which are expected to be changed only when different versions of the ODK APKs are released.

The general structure is:

  • graph/ - this will go away; it is useful as a set of examples for how graphing with d3 would work. It contains Javascript for rendering graphs using the d3 library. We want to eventually provide code generation within Application Designer to generate graph code like this tailored for a user's data table.
  • js/ - contains Javascript for the Java to Javascript interfaces common to both ODK Table and ODK Survey.
  • libs/ - contains 3rd party Javascript libraries used by ODK Tables and ODK Survey.
  • survey/ - contains Javascript used by ODK Survey to render forms.
  • tables/ - contains Javascript used by ODK Tables to render the custom home screen, list, detail, and graphical views created by the Application Designer.
  • tables.deleting - information related to data deletion
  • tables.pending - information related to pending data changes
  • index.html - the generic HTML for all ODK Survey forms.

Files in this folder are managed by the ODK tools. If you change any of these files, the tools may detect the change and restore the file when they next start. The goal is that only the ODK core team should have to modify things in this folder. If you feel you need to modify anything in this directory, please contact us.

Common Tasks

There are several common tasks that are facilitated by the ODK Application Designer. These are:

Each of these will now be discussed.

ODK Survey: Designing a Form

This section assumes that all of the necessary software has been installed and verified. If not, complete the instructions in Getting Started.

When creating a new form, the appropriate directory structure must be created. Once this directory structure is in place, an xlsx form can be created. From this xlsx form, a formDef.json file will be generated using the XLSXConverter. This formDef.json, in the appropriate directory, is what the system will use to create the survey form.

Creating a Canonical Directory Structure for a New Form

New forms must be placed under the app/config/tables directory as described in the app/config/tables folder section. Given a form with the name formId, it will have a tableId of the same name unless you explicitly specify otherwise. The directory structure that should be created is app/config/tables/tableId/forms/formId (where, under many circumstances, the value for tableId will be the same as the value for formId). To get started, for Windows open a cmd window within your ApplicationDesigner folder (click the cmd shortcut you created earlier), and for Mac/Unix open a terminal window within your ApplicationDesigner folder. Type:

grunt addtable:tableId

This will create the required directory structure for an individual table, including the forms directory. Navigate into that, and create your formId directory. Within that directory, ODK Survey expects to find the formDef.json that defines the form. We recommend placing the xlsx file used to generate that formDef.json in this folder. Any custom screen or prompt templates or other media related to the form should be also placed in this directory (or in a sub-directory). For example, if you wanted to create a new form named testForm.xlsx, the directory structure that would be have to created for this form would be app/config/tables/testForm/forms/testForm.

Creating an ODK Survey

With the proper directory structure in place, you can now create your survey form. The XLSXConverter documentation extensively details the worksheets that will need to be created within your xlsx file to create a survey. Also, you can use the File Browser window of the Application Designer to navigate to example xlsx files under the /app/tables directory; it will likely be easier to start with one of the existing example forms and modify it. The key modification would be on the settings page -- changing the values for table_id and form_id (if present).

Generating a formDef.json File with XLSXConverter

Once you have a saved your survey XLSX file, you can use the XLSXConverter to create a formDef.json. Use the Save to Filesystem button to save the form definition file back to the file system. Note: the Save to Filesystem button uses the form_id an table_id within the XLSX file to identify where to write the formDef.json file. If you have copied the XLSX file from some other location, and forgot to edit it, it may update back to that older location!! If the form_id is equal to the table_id, two additional files are written that define the table's user data fields and that define the key-value properties for the table.

You will notice that the form still does not appear in the list of forms found under the Preview tab of the Application Designer. In order for the form to appear, the framework.xlsx file in the app/config/assets/framework/forms/framework directory must be modified, and the formDef.json file in that same directory updated using XLSXConverter.

The framework.xlsx file is just another form definition, but it generally has no persisted data. In this case, it just presents a list of forms and allows you to open them. Eventually, ODK Survey will offer a configuration option to use this form as its starting screen, rather than presenting the list of all available forms to the user.

The modifications to the framework.xlsx are as follows. Assuming you have created a testForm.xlsx, the appropriate directory structures for testForm.xlsx, and then properly generated and saved the formDef.json, the following lines would need to be added into the framework.xlsx survey worksheet.

branch_label url clause condition type values_list display.text display.hint
testForm
''?' + opendatakit.getHashString('../config/tables/testForm/forms/testForm/',null) external_link Open form
exit section

The following changes will also need to be made to the framework.xlsx choices worksheet

choice_list_name data_value display.text
test_forms testForm testForm

The changes to the choices sheet adds the testForm form as one of the choices that is shown in the user_branch prompt (a user-directed branching prompt type). The changes on the survey sheet add a branch label ('testForm') that matches the 'data_value' from the choices sheet (this branch label will be jumped to if the user selects the testForm selection on the user_branch screen). The new branch label then renders an external_link prompt type that has the necessary arguments to open the testForm.

Once you have made these changes and used XLSXConverter on the framework.xlsx file to update the app/config/assets/framework/forms/framework/formDef.json file, you should see your new form show up in the Preview tab of the Application Designer. Clicking on that should open your form.

Debugging your Survey

The XLSXConverter should report most problems with your survey.

If the form is not being rendered correctly but your survey generates a formDef.json without an error, first try purging the database (dropping all the existing data tables) using the Purge Database button on the Preview tab. You will typically need to purge the database whenever you add or remove fields from your form or change their data type.

If that does not resolve the issue, try stopping the grunt command (on Windows, Control-C should produce a prompt asking to confirm whether to stop or not), and re-running it. Grunt can sometimes get overwhelmed with changes and stop working. After restarting, test your form.

If there are other problems, the contents of the Javascript Console will be helpful to the ODK core team for debugging. Open the Javascript Console by clicking the icon with the three bars in the top right, select 'Tools' and then select Javascript Console. You can click-and-drag to select all of the debugging output, then copy it, save it to a file, and send the file as an attachment to the opendatakit@ list.

Moving the New ODK Survey Form to a Device

NOTE: You must have USB debugging enabled on your device in order to perform this step. See here for instructions.

In order to see these changes on an Android device, you must first have ODK Survey installed on your device. Then:

  1. Force Stop your ODK Tables application by going into the device's Settings / Applications page, selecting ODK Tables , and clicking Force Stop.
  2. Force Stop your ODK Scan application by going into the device's Settings / Applications page, selecting ODK Scan , and clicking Force Stop.
  3. Force Stop your ODK Survey application by going into the device's Settings / Applications page, selecting ODK Survey, and clicking Force Stop.
  4. Force Stop your ODK Services application by going into the device's Settings / Applications page, selecting ODK Services, and clicking Force Stop.
  5. Connect the device to your computer via a USB cable
  6. Open a cmd/terminal window within the ApplicationDesigner directory (the one containing Gruntfile.js), as described in the Directory Structure section.
  7. Type: grunt adbpush-default-app

This will copy all of the files under config onto your device. You should then be able to launch ODK Survey, and it will display your form in its list of forms. Click the form to open it.

More grunt commands can be found in the Both: Pushing updates to the Device section, below.

 

ODK Tables: Designing a Custom View

One of the most powerful aspects of ODK Tables is its ability to run HTML and
JavaScript pages as the skin of the app. Through a Javascript API presented to these files, you can query the database and control the app.

Writing an app using html/js yields a lot of power. However, it can lead to a complicated design cycle.

The html/js files you write rely on the JavaScript API implemented within the ODK Tables APK to vend data-table values into your HTML pages, where they can be displayed as a list of items, as a detail view of a single item, or graphed in any number of ways. This Javascript API, since it is implemented in the APK, makes it difficult to debug your custom views off the phone. Long-term, we intend to support this through a more capable Application Designer environment. At present, the only way to test your HTML pages is on the device. Fortunately, on Android 4.4 and higher, Chrome can access the browser Console and set breakpoints on the device, providing a clumsy but viable debug environment.

 

Understanding the Web File

There are several pieces of boilerplate you have to include in your own code in order to debug the files in Chrome.

In the ODK ApplicationDesigner, use the file browser to open the config/tables/Tea_houses/html/Tea_houses_list.html file for the list view of the "Tea_houses" table. Right-click, and 'View Frame Source'. This shows the contents of that file. The important part to note is the following lines in the <head>:

<script type="text/javascript" src="../../../assets/libs/jquery.js"></script>
<script type="text/javascript" src="../../../../system/js/odkCommon.js"></script>
<script type="text/javascript" src="../../../../system/js/odkData.js"></script>
<script type="text/javascript" src="../../../../system/tables/js/odkTables.js"></script>

In the first line you are just making the jquery object available to your code; JQuery is a very powerful, very commonly used, set of functions for accessing and performing actions within a webpage. In the second two lines you are adding the odkCommon, odkTables, and odkData objects if they are not already provided by the browser environment. When running on the device, the ODK Tables APK will provide these, and the contents of these files will be ignored. When running in ApplicationDesigner on your computer, these files provide the approximate functionality of the APK, allowing you to create and debug your scripts. However, at the moment, these implementations make use of requirejs, which the ODK Tables HTML files do not use (requirejs is extensively used by ODK Survey). This causes these to break in Application Designer.

Writing Your Own Web Files

To write your own file, first decide on the tableid for your table and instantiate a directory using the grunt command:

grunt addtable:tableid

This grunt task creates the needed directory structures and also constructs the html and javascript files with the necessary features for working within the Chrome development environment.

These files need content from your data-table in order enable you to begin creating your custom screens. We recommend that you first design an ODK Survey form to facilitate populating the table and then set up a tables.init file to auto-populate the form with test data. Then, as you shut down and restart your app, it will auto-load that test data.

After that, you can deploy your app to your device and open ODK Tables onto the custom view (see the properties sheet section of the XLSXConverter for how to specify the HTML file that should be opened). Once it opens, you can use the Chrome browser on your computer to inspect for devices and connect to this custom screen on your Android device, and debug from there.

The edit-debug cycle is awkward at present because you must make the HTML or Javascript change on your computer then push the change to your device, and reload the page (e.g., by rotating the screen). When you do rotate the screen, however, it is rendered in a new web page, necessitating connecting to that new page to resume debugging (the prior page sits idle and will eventually be destroyed; if you don't see any activity, it is likely because you are pointing at the wrong web page; return to inspect devices, and select the newest page.).

As with ODK Survey, you can use the Javascript Console to look for and fix errors in your HTML/Javascript. It will be beyond the abilities of the ODK Core Team to assist everyone in debugging their own custom javascript. Please search the web for answers. Keep in mind that the debug objects only emit a subset of the data in your ODK Tables database.

Both: Pushing updates to the Device

NOTE: You must have USB debugging enabled on your device in order to perform this step. See here for instructions.

There are several times during app development where you will need to push and pull files to and from the phone. You will have to open one of the ODK tools on the device before these commands succeed.

Push:

  • The entire app directory

Pull:

  • The database
  • Exported csvs (to set up tables.init loading of test data.

Grunt tasks have been written in Gruntfile.js that perform these operations for you.

These commands can be run anywhere within the repository.

Push:

  • adbpush-default-app. Pushes everything under the app directory to the device.

Pull

  1. grunt adbpull-db
  2. grunt adbpull-csv

The pull commands will place the pulled content in the app/output directory.

The database is a SQLite database and can be viewed using SQLite Browser. This tool can also be used to view the content of the database used by Chrome on your computer (the location of that file is OS dependent).

If you pull the CSV files, they will be under the output/csv directory. You can then copy them to the config/assets/csv directory and set up the tables.init file to read them in order to provision test data for your development effort. If you need any of this data in production, you will want to sync to a server then export the csv files and copy them to the config/assets/csv directory so that they have all of their metadata field values populated.

You can also perform the adbpush-default-app tasks by running grunt adbpush. Running grunt adbpull will perform all the pull tasks.

There are a number of additional grunt tasks available. Assuming you have installed grunt and node, you can view the available tasks by running grunt --help anywhere in the repo.