Encrypted Forms

Overview

To begin with you may want to review the Security and Privacy statement.

Encrypted forms provide a mechanism to keep your data private even when using http: for communications (e.g., when you do not have an SSL certificate or https: is not available). Encrypted forms may also enable Google App Engine deployments (and deployments using other web database services, e.g., AWS) to comply with data privacy laws, eliminating the necessity for setting up your own servers to meet those requirements.

Encrypted forms apply asymmetric public key encryption at the time the form is finalized within ODK Collect. This encrypted form can then be submitted up to ODK Aggregate and downloaded to ODK Briefcase. ODK Briefcase, when supplied with the asymmetric private key (which ODK Briefcase never stores), can then decrypt and export the form data as a CSV file for your use.

This process ensures that the finalized form's data (and media attachments) are encrypted before being submitted to ODK Aggregate, remain encrypted while stored on ODK Aggregate, and remain encrypted as the data and attachments are pulled into ODK Briefcase, where they are again stored in encrypted form.

ODK Aggregate cannot meaningfully publish encrypted forms to Google Spreadsheets or Fusion Tables since the encryption obscures the entire contents of the form and ODK Aggregate never possesses the asymmetric key required to decrypt the form. When using encrypted forms, ODK Aggregate serves only as a data aggregation point — you must download, decrypt, and export the data using ODK Briefcase to access the unencrypted data.

The non-encrypted data is available on the ODK Collect device during data collection and whenever a form is saved without marking it as complete. Once you mark a form as complete (finalize it), ODK Collect will generate a random 256-bit symmetric key, encrypt the form contents and all attachments with this key, then construct a submission manifest which describes the encrypted submission and an asymmetric-key encryption of the symmetric key used for the encryption. This manifest is the "form" that is uploaded to ODK Aggregate, with the encrypted form contents and its encrypted attachments appearing as attachments to this submission manifest "form."

Requirements

Encrypted forms require the following minimum versions for the ODK tools:

  • ODK Collect 1.2 Release Candidate 1 (RC1) or higher
  • ODK Aggregate 1.0.4 Production or higher
  • ODK Briefcase 1.0 Production or higher

Encrypted form definitions must include a unique OpenRosa instanceID and have an explicit <submission/> element. These requirements are covered in more detail below.

Security Concerns

While ODK Collect attempts to remove all unencrypted copies of a finalized form and its attachments from the device, because ODK Collect uses third party applications for image capture, etc., and because of the potential for Forced Close events during the clean-up process, we cannot guarantee that all copies will have been destroyed. Furthermore, because of the way an SD card writes and deletes information, there is a possibility of this data being recoverable from the free space on the SD card. Your organization should investigate the extra steps needed to ensure all data is deleted from the SD cards on your ODK Collect devices and should establish procedures to periodically wipe and reinstall those devices.

Encrypting a form ensures that the finalized form is not readable and is not tampered with. However, there is nothing preventing a malicious adversary from the wholesale replacement of a finalized form with falsified data or the synthesis and submission of extra data — these are not contingencies that encrypted forms seek to address.

Configuration

No special configuration is required for ODK Collect or ODK Aggregate.

For ODK Briefcase, you must download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6 from the Java download site; if the Java 7 version of this is not available on that site, it is also available here. This is required for decryption to be successful.

To install the JCE:

  1. Select the downloaded UCE zip archive file in Windows Explorer (the file viewer)
  2. Right-click, choose "Extract All..." and extract the files from the downloaded zip archive
  3. Navigate into the extracted directory tree and copy the local_policy.jar and US_export_policy.jar files to the lib\security directory under the installation directory of the Java Runtime Enviornment (JRE) of your computer, replacing earlier versions of these files.

Windows
The JRE is usually installed here:
C:\Program Files\Java\jre1.x.x_xxx\lib\security

You might also have a JDK. If you do, you must also install the JCE files there:
C:\Program Files\Java\jdk1.x.x_xxx\jre\lib\security

macOS
The JRE is usually found here:
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security

You might also have a JDK. If you do, you must also install the JCE files there:
/Library/Java/JavaVirtualMachines/jdk1.x.x_xxx.jdk/Contents/Home/jre/lib/security

Form Definition

Here is a sample encrypted form (in which the bulk of the base64RsaPublicKey attribute value has been elided for brevity and readability).

<h:html xmlns="http://www.w3.org/2002/xforms"
        xmlns:h="http://www.w3.org/1999/xhtml"
	xmlns:orx="http://openrosa.org/xforms/">
 <h:head>
  <h:title>Sample Form</h:title>
  <model>
    <itext>
      <translation lang="English" default="">
        <text id="ask_name">
          <value form="long">Please enter your name:</value>
          <value form="short">Respondent's name</value>
        </text>
      </translation>
    </itext>
    <instance>
      <sample id="sample-v1.0">
        <orx:meta>
           <orx:instanceID/>
        </orx:meta>
        <name/>
      </sample>
    </instance>
    <submission method="form-data-post"
                action="https://my-opendatakit.appspot.com/submission"
                base64RsaPublicKey="MIIBIjANB...JCwIDAQAB" />
    <bind nodeset="/sample/meta/instanceID" type="string" readonly="true()"
          calculate="concat('uuid:', uuid())"/>
    <bind nodeset="/sample/name" type="string" />
  </model>
 </h:head>
 <h:body>
    <input ref="name">
       <label ref="jr:itext('ask_name')"/>
    </input>
 </h:body>
</h:html>

If you are using XLSForm, then form encryption is governed by the settings on the Settings Worksheet. Encrypted forms must specify a submission_url and a public_key on this worksheet. If both are specified, XLSForm will generate a encrypted-form definition. No further steps are necessary. Skip to the following sections to see how to create a public-private key pair and specify the public key.

Note the presence of the OpenRosa metadata block, as defined here: OpenRosa 1.0 Metadata Schema; this is required. You can define any value for the instanceID field, but it must be unique across all collected surveys. As shown, the <bind/> calculates an instanceID value comparable to the instanceID value ODK Aggregate will generate when a form does not have an instanceID field. If you use punctuation other than colon and dash, or any special characters, please test thoroughly to ensure that ODK Aggregate correctly handles those characters during submissons and when the form is pulled down to ODK Briefcase (it should).

The other required element to make this form an encrypted form is the <submission/> tag. Within this tag, the method attribute should always be form-data-post. The action attribute should be the url to which the submission should be posted; this is the ODK Aggregate website url with Aggregate.html replaced by submission. Finally, what identifies the form as an encryted form is the presence of a base64RsaPublicKey attribute. This should be the base64 encoding of the RSA public key that ODK Collect uses to encrypt the symmetric encryption key it creates to encrypt a finalized instance of this form (a different symmetric encryption key is created for every finalized form).

Creating RSA Key pair

RSA public-private key pairs are generated using the OpenSSL software package. This is pre-installed on OSX and Linux, but needs to be downloaded and installed on Windows.

1. Install OpenSSL (Windows only)
For Windows, download and run the OpenSSL installer appropriate for your system from OpenSSL for Windows. When it asks whether to copy the DLLs to the Windows system directory or to the /bin directory, choose the /bin directory (either will work, but this will minimize the pollution of the Windows system directory).

2. Constructing the RSA Key Pair
If you are on Windows, open a Windows 'cmd' window. Change directories to the /bin directory in the OpenSSL directory. e.g.,

cd C:\OpenSSL-Win32\bin;

If you are on a Mac, open Terminal.app. Change directories to your Desktop.

cd ~/Desktop;

3. Create a private key
The following command will create a 2048-bit private key and write it to the MyPrivateKey.pem file. This may complain about a missing configuration file. You can ignore this warning.

If you are on Windows, run:

openssl genpkey -out MyPrivateKey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:2048

If you are on a Mac, run:

openssl genrsa -out MyPrivateKey.pem 2048

4. Extract a public key
Next, you need to extract the public key for this private key.

Run the following command:

openssl rsa -in MyPrivateKey.pem -inform PEM -out MyPublicKey.pem -outform PEM -pubout

This may also complain about a missing configuration file. You can ignore this warning.

5. Storing and using the keys
Move the MyPrivateKey.pem file to a secure location. It does not have a password encoding it, so anyone can decrypt your data if they have access to this file. This is the private key file that you will give to ODK Briefcase when decrypting the data.

6. Updating the public_key field in the XLSForm settings worksheet.
Open the MyPublicKey.pem file in Notepad (Windows) or TextEdit (Mac). Remove the leading and trailing '---...' lines. Then remove all new-lines (carriage-returns) from the file. Select the resulting very-long string, and paste it into the public_key field in the XLSForm settings worksheet. This very-long string will become the base64RsaPublicKey attribute in the resulting encrypted form definition. Note that you obviously need to be especially careful that this is ONLY the public key, and not the contents of the original public-private key file (which would also appear to work but provide no security).

Operations

Operationally, you would add the form definition to the ODK Aggregate server identified in the <submission> tag's action attribute, and deploy everything using ODK Collect 1.2 RC1 or later, figure out how you want to implement a periodic SD Card wiping protocol for your devices, and you're done.  Submissions will be encrypted when marked as complete.  ODK Aggregate will only hold the encrypted submission with no access to the private key. Once the data is on ODK Aggregate, use ODK Briefcase to download the encrypted submissions to your PC, and then specify the private key PEM file when decrypting and generating the CSV files.

ODK Briefcase will emit the CSV with an extra final column that indicates whether the signature of the encrypted file was good or bad.  It would be bad if any of the attachments are missing or if there was tampering (other than the wholesale replacement of a submission, which can't be detected).