JIO Home JIO

    jIO

    jIO is an abstract, promise-based JavaScript API that offers connectors to many storages (Dropbox, Google Drive, IndexedDB…) and special handlers for enhanced functionality (replication, compression, querying…). jIO separates storage access from the application, providing a simple way to switch backends and create offline-capable applications. jIO is developed and maintained by Nexedi and is used for the responsive ERP5 interface and as the basis for applications in app stores like OfficeJS.

    What is a jIO Storage?

    A storage is either a connector or a handler. Connectors store documents and attachments, providing access to the documents through the jIO API. Setting up a storage connector is easy:

    // create a jIO LocalStorage
jIO.createJIO({
  "type": "local",
  "sessiononly": false
});

    Handlers add increased functionality on top of connectors, and some of them extend the jIO API. It's just as easy to add a handler storage on top:

    // create a jIO compressed LocalStorage
jIO.createJIO({
  "type": "zip",
  "sub_storage": {
    "type": "local",
    "sessiononly": false
  }
});

    Promises

    jIO is fully asynchronous and uses an implementation based on RSVP.js. However, in order to safely cancel chains of promises, we had to create a custom version of RSVP.js, linked to further down, that does not use .then to chain promises. Instead, promise chains are written as RSVP.Queue().push(function () {...}).push(function (result) {...}).push(undefined, function (error) {...});

    Another JavaScript Framework? Why use jIO?

    Nexedi's free software products and the custom solutions developed from them must normally run for many years. As the complexity of our projects is usually very high, redevelopments to follow the current trending JavaScript framework or to replace legacy frameworks is out of our scope. Hence, we created jIO and RenderJS, two no-frills libraries that are:

    • sturdy, small API, easy to use once understood.
    • flexible, multiple storages and handlers.
    • extendable, write your own storage if needed.

    Getting Started

    jIO is easy to set up and get working.

    Source Code

    The jIO source code is available on GitLab, with a mirror on GitHub. To build,

    > git clone https://lab.nexedi.com/nexedi/jio.git
> make build

    or just download the files directly:

    or for NodeJS users, use npm:

    > npm install jio

    The following file might also be useful:

    What is a Document?

    A document is a collection of metadata and attachments. The metadata is the set of properties of the document and the attachments are binary objects that represent the content of the document. In jIO, the metadata is a plain JSON object with keys and values, and attachments are Blobs, for example:

    // document metadata
{
  title: 'A Title!',
  creator: 'The Author'
}

// attachment blob
new Blob(["string data"], {type: "text/plain"});

    Hello World

    Create an HTML file with the specified JavaScript files and the following contents:

    <!doctype html>
<html>
  <head>
    <script type="text/javascript" src="rsvp-latest.js"></script>
    <script type="text/javascript" src="jio-latest.js"></script>
  </head>
  <body>
    <p>jio example, see console</p>
    <script type="text/javascript">
      (function (jIO) {
        var storage = jIO.createJIO({"type": "indexeddb", "database": "foo"});
        console.log(storage);
        function foo(message) {
          return storage.put("start", message)
            .push(function (result) {
              console.log(result);
              return storage.get("start");
            })
            .push(function (result) {
              console.log(result)
            }, function (error) {
              console.warn(error);
              throw error;
            });
        }
        return foo("hello");
      }(jIO));
    </script>
  </body>
</html>

    Check your console to see how the jIO storage is created, the attachment is stored and fetched again.

    You are now using jIO! Follow the OfficeJS Application Tutorial for a gentle introduction to application development using jIO and RenderJS, or keep on reading to dive straight into the full jIO API.

    Posting Attachments

    Below is an example of posting an attachment into LocalStorage to show the handling of attachments.

    
(function (jIO) {
  // create a new jIO storage
  var jio_instance = jIO.createJIO({"type": "indexeddb", "database": "foo"}});

  // post the metadata for "myVideo"
  return jio_instance.put("document", {
    title: "My Video",
    type: "MovingImage",
    format: "video/ogg",
    description: "Images Compilation"
  })
  // post a thumbnail attachment
  .push(function () {
    return jio_instance.putAttachment(
    "document",
    "thumbnail",
    new Blob([my_image], {type: "image/jpeg"})
  })
  // post video attachment
  .push(function () {
    return jio_instance.putAttachment(
    "document",
    "video",
    new Blob([my_video], {type: "video/ogg"})
  })
  // catch any errors and throw
  .push(undefined, function(error) {
    console.warn(error);
    throw error;
  });
}(jIO));

    API - Quickguide

    The original jIO interface was based on CouchDB, but has since evolved to the current set of methods described below and in more detail afterwards.

    Method Example Info
    Create Storage 
    jIO.createJIO({storage_configuration});
    		 [returns jIO Storage]. Initialize a new storage or storage tree, synchronously.
    Post Document 
    storage.post({
  "key": "value"
});
    		 [returns Promise]. Create a new document with the given metadata and returns the automatically generated ID. If post is unsupported, add a UuidStorage handler on top of your storage to provide it.
    Put Document 
    storage.put(id, {
  "key": "value"
});
    		 [returns Promise]. Create or update the document with the given ID using the given metadata.
    Get Document 
    storage.get(id);

// returns {"key": "value"}
    		 [returns Promise]. Retrieve a document's metadata.
    Remove Document 
    storage.remove(id);
    		 [returns Promise]. Deletes a document and all its attachments.
    Search Documents 
    storage.allDocs({
  "query": query_object,
  "limit": [3, 42],
  "sort_on": [["key1", "ascending"], ["key2", "descending"]],
  "select_list": ["key1", "key2", "key3"],
  "include_docs": false,
});

// include_docs response
// {
//  "total_rows": 39,
//  "rows": [{
//    "id": "text_id",
//    "value": {},
//    "doc": {"key": "value"}
//  }, ...]
//}

// default response
//{
//  "total_rows": 39,
//  "rows": [{
//    "id": "text_id",
//    "value": {"select_list_key": "select_list_value"}
//  }, {}, ...]
//}
    		[returns Promise]. Retrieve a list of documents. If include_docs is true, then doc in the response will contain the full metadata for each document. Otherwise, if select_list contains keys, then value in the response will contain the values of these keys for each document. If query is unsupported, add a QueryStorage handler on top of your storage to provide it.
    Add Attachment 
    storage.putAttachment(id, name, blob);
    		 [returns Promise]. Create or update the given blob as an attachment with the given name to the document with the given ID.
    Remove Attachment 
    storage.removeAttachment(id, name);
    		 [returns Promise]. Deletes the attachment with the given name from the document with the given ID.
    Get Attachment 
    storage.getAttachment(id, name, {"format": "text"});
    		 [returns Promise]. Retrieve the attachment with the given name from the document with the given ID in text, json, blob, data_url, or array_buffer format.
    All Attachments 
    storage.allAttachments(id);

// default response
// {
//  "attachment_id_1": {},
//  "attachment_id_2": {}
// }
    		[returns Promise]. Retrieves an object containing the id's of all attachments belonging to the document that was passed as parameter
    Synchronize Storage 
    storage.repair();
    		 [returns Promise]. Synchronize or repair the storage. If repair is unsupported, add a ReplicateStorage handler on top of the two storages you wish to synchronize, to provide it.

    API - Storage Types

    Below is the list of storages currently supported by jIO, including sample storage configurations. Note: authentication for storages, such as ERP5 or Dropbox, is not handled by jIO. You have to ensure that you are logged in or provide whatever tokens are required to access the storage when creating your jIO storage.

    LocalStorage

    Store documents in the browser's local storage. This storage has only one document, with the ID "/", so post, put, remove and get methods are not supported. The only operations you can do on a raw LocalStorage are attachment manipulation methods with blobs. To treat a LocalStorage as a regular storage, add a DocumentStorage handler on top of it, with "/" as the document_id.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "local").
    sessiononly no Boolean False (default): create a storage with unlimited duration.
    True: the storage duration is limited to the user session.    		 
    {
  type: "local",
  sessiononly: true
});

    MemoryStorage

    Store documents in a raw JavaScript object, in memory. The storage’s data isn't saved when your web page is closed or reloaded, and doesn’t take any other arguments.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "memory"). 
    {
  type: "memory"
});

    IndexedDbStorage

    Store documents in the IndexedDB database with the given name.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "indexeddb").
    database yes String Name of the database. 
    {
  "type": "indexeddb",
  "database": "mydb"
}

    WebSqlStorage

    Store documents in the WebSQL database with the given name. Using an IndexedDBStorage is strictly better.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "websql").
    database yes String Name of the database. 
    {
  "type": "websql",
  "database": "mydb"
}

    WebDavStorage

    Store documents in the WebDAV server with the given URL. Documents are WebDAV directories, so they must not contain any metadata, and their IDs must be bookended by forward slashes that directly correspond to their path. Attachments to documents are files inside WebDAV directories, so they must not contain any forward slashes. To treat a WebDavStorage as a regular storage, add a FileSystemBridgeStorage on top of it.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "dav").
    url yes String URL of your WebDAV server.
    basic_login yes String Login and password of your WebDAV, base64 encoded like this: btoa(username + ":" + password)
    with_credentials no Boolean False (default): do not send domain cookie.
    True: send domain cookie.    		 
    // No authentication
{
  "type": "dav",
  "url": "http://mydav.com/"
}

// Basic authentication
{
  "type": "dav",
  "url": "https://securedav.com/",
  "basic_login": btoa(username + ":" + password)
}

// Digest authentication is not implemented yet

    DropboxStorage

    Store documents in the Dropbox account with the given access token. Documents are Dropbox folders, so they must not contain any metadata, and their IDs must be bookended by forward slashes that directly correspond to their path. Attachments to documents are Dropbox files inside Dropbox folders, so they must not contain any forward slashes. To treat a DropboxStorage as a regular storage, add a FileSystemBridgeStorage on top of it.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "dropbox").
    access_token yes String Access token for your dropbox. See the Dropbox documentation of its API v1 for how to generate an access token.
    root no String "dropbox" (default) for full access to account files.
    "sandbox" for only access to files created by your app.    		 
    {
  "type": "dropbox",
  "access_token": "sample_token",
  "root": "sandbox"
}

    GoogleDriveStorage

    Store documents in the Google Drive account with the given access token. Unlike WebDavStorage and DropboxStorage, GoogleDriveStorage documents can contain metadata and have no specific rules regarding forward slashes. To treat a GoogleDriveStorage as a regular storage, add a FileSystemBridgeStorage on top of it.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "gdrive").
    access_token yes String Access token for your Google Drive. See the Google Drive documentation for how to generate an access token.
    trashing no Boolean true (default): sends file to trash bin when calling "remove".
    false: delete files permanently when calling "remove""    		 
    {
  "type": "gdrive",
  "access_token": "sample_token"
  "trashing": true
}

    Erp5Storage

    Store documents in the ERP5 instance with the given URL. All ERP5 documents must contain values for portal_type and parent_relative_url in the metadata, which define the type of the document that is stored. Attachments are ERP5 actions. A raw Erp5Storage supports post and query, so there is no need to add a UuidStorage or QueryStorage on top of it.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "erp5").
    url yes String URL of HATEOAS in your ERP5 instance.
    default_view_reference no String Reference of the action used to deliver of the document. 
    {
  "type": "erp5",
  "url": "https://erp5.nexedi.net/web_site_module/hateoas",
  "default_view_reference": "
}

    ZipStorage (Handler)

    Compress and decompress attachments to reduce network and storage usage.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "zip").
    sub_storage yes Object Definition of storage whose attachments should be zipped. 
    {
  "type": "zip",
  "sub_storage": {storage_definition}
}

    ShaStorage (handler)

    Provide the post method to create new documents using the SHA-1 hashes of their parameters as their IDs.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "sha").
    sub_storage yes Object Definition of storage whose documents' IDs should be the SHA-1 hash of their parameters. 
    {
  "type": "sha",
  "sub_storage": {storage_definition}
}

    UuidStorage (handler)

    Provide the post method to create new documents using randomly generated UUIDs as their IDs.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "uuid").
    sub_storage yes Object Definition of storage whose documents' IDs should be randomly generated UUIDs. 
    {
  "type": "uuid",
  "sub_storage": {storage_definition}
}

    QueryStorage (handler)

    Provide support for query parameters in the allDocs method.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "query").
    sub_storage yes Object Definition of storage whose documents should be queryable when calling allDocs. 
    {
  "type": "query",
  "sub_storage": {storage_definition}
}

    CryptStorage (handler)

    Encrypt and decrypt attachments to secure them. You must generate a Crypto key in JSON format to use this handler.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "crypt").
    key yes String JSON crypto key.
    sub_storage yes Object Definition of storage whose attachments should be encrypted. 
    var key, jsonKey, jio;
crypto.subtle.generateKey({name: "AES-GCM",length: 256},
                          (true), ["encrypt", "decrypt"])
  .then(function(res){key = res;});
window.crypto.subtle.exportKey("jwk", key)
  .then(function(res){jsonKey = res})

jio = jIO.createJIO({
  "type": "crypt",
  "key": json_key,
  "sub_storage": {storage_definition}
}

    UnionStorage (handler)

    This handler takes a list of storages as its argument. When using a jIO method, UnionStorage tries it on the first storage of the array; if it fails, then UnionStorage tries the method on the next storage, and so on until success or there are no more storages left to try.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "union").
    storage_list yes Array List of storage definitions. 
    {
  "type": "union",
  "storage_list": [
    {storage_definition},
    {storage_definition},
    ...
    {storage_definition}
]}

    FileSystemBridgeStorage (handler)

    Abstract file system storages such as WebDAV, Dropbox, and Google Drive, so that jIO methods work normally on them.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "drivetojiomapping").
    sub_storage yes Object Definition of storage whose file system should be treated as a database. 
    {
  "type": "drivetojiomapping",
  "sub_storage": 
}

    DocumentStorage (handler)

    Create a storage on top of a single document by mapping documents to attachments, so that jIO methods work normally on single-document storages.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "document").
    document_id yes String id of the document to use.
    repair_attachment no Boolean Verify if the document is in good state. (default to false) 
    {
  "type": "document",
  "document_id": "text_id",
  "repair_attachment": false
}

    ReplicateStorage (Handler)

    Synchronize documents between a local and a remote storage by providing the repair method.

    Parameter Required? Type Description
    type yes String Name of the storage type (here: "replicate").
    local_sub_storage yes Object Definition of the local storage, on which normal jIO operations are applied.
    remote_sub_storage yes Object Definition of the remote storage that synchronizes with the local storage.
    query no Object Query object to limit the synchronisation to specific files.
    use_remote_post no Boolean true: at file modification, modifies the local file id. false (default): at file modification, modifies the remote file id.

    conflict_handling

    		 no Number 0 (default): no conflict resolution (throws error) 1: keep the local state. 2: keep the remote state. 3: keep both states (no signature update)
    parallel_operation_amount no Number Control number of parallel operation for document synchronisation (default 1)
    parallel_operation_attachment_amount no Number Control number of parallel operation for attachment synchronisation (default 1)
    check_local_modification no Boolean (default: True) Synchronize when local documents are modified.
    check_local_creation no Boolean (default: True) Synchronize when local documents are created.
    check_local_deletion no Boolean (default: True) Synchronize when local documents are deleted.
    check_remote_modification no Boolean (default: True) Synchronize when remote documents are modified.
    check_remote_creation no Boolean (default: True) Synchronize when remote documents are created.
    check_remote_deletion no Boolean (default: True) Synchronize when remote documents are deleted.
    check_local_attachment_modification no Boolean (default: False) Synchronize when local attachments are modified.
    check_local_attachment_creation no Boolean (default: False) Synchronize when local attachments are created.
    check_local_attachment_deletion no Boolean (default: False) Synchronize when local attachments are deleted.
    check_remote_attachment_modification no Boolean (default: False) Synchronize when remote attachments are modified.
    check_remote_attachment_creation no Boolean (default: False) Synchronize when remote attachments are created.
    check_remote_attachment_deletion no Boolean (default: False) Synchronize when remote attachments are deleted.
    signature_hash_key no String (default: undefined) Use a document key as document signature hash (instead of calculating the SHA1).
    signature_sub_storage no Object (default: store in the local_storage) Definition of the signature storage, where replication signature are stored.
    report_level no Integer (default: 100) Verbosity of the report (0 = silent, 100 = error, 200 = warning, 300 = all write, 500 = all info)
    debug no Boolean (default: False) Log all report operation in the console 
    {
  type: "replicate",
  local_sub_storage: {"type": "local"}
  remote_sub_storage: {
    "type": "dav",
    "url": "http://mydav.com",
    "basic_login": "aGFwcHkgZWFzdGVy"
  }
  use_remote_post: false,
  conflict_handling: 2,
  check_local_creation: false,
  check_remote_deletion: false
}

    jIO Query Engine

    In jIO, a query can ask a storage server to select, filter, sort, or limit a document list before sending it back. If the server is not able to do so, the jIO query tool can do the filtering by itself on the client. Only the allDocs method can use jIO queries.

    A query can either be a string (using a specific language useful for writing queries), or it can be a tree of objects (useful to browse queries). To handle queries, jIO uses a parsed grammar file which is compiled using JISON.

    JIO queries can be used like database queries, for tasks such as:

    • searching a specific document
    • sorting a list of documents in a certain order
    • not retrieving a list of ten thousand documents
    • limiting the list to show only N documents per page

    For some storages (like localStorage), jIO queries can be a powerful tool to query accessible documents. When querying documents on a distant storage, some server-side logic should be run to avoid returning too many documents to the client.

    How to use Queries with jIO?

    Queries can be triggered by including the option named query in the .allDocs() method call.

    var options = {};

// search text query
options.query = '(creator:"John Doe") AND (format:"pdf")';

// OR query tree
options.query = {
  type: "complex",
  operator: "AND",
  query_list: [{
    type: "simple",
    key: "creator",
    value: "John Doe"
  }, {
    type: "simple",
    key: "format",
    value: "pdf"
  }]
};

// FULL example using filtering criteria
options = {
  query: '(creator:"% Doe") AND (format:"pdf")',
  limit: [0, 100],
  sort_on: [
    ["last_modified", "descending"],
    ["creation_date", "descending"]
  ],
  select_list: ["title"]
};

// execution
jio_instance.allDocs(options, callback);

    Creating Your Own Storage

    Extending jIO by adding own storages is fairly easy as you only have to implement the base methods plus the internal methods hasCapacity (for which allDocs parameters are supported) and buildQuery (for constructing actual queries).

    For example if you would want to create a parallel storage that maintains multiple storages on the same jIO gadget you could create a file named jio.parallelstorage.js (or similar) and add it after the jIO file in your html. The file should contain:

    /**
 * JIO Parallel Storage Type = "Parallel".
 * keep storages in parallel, without sync/replication
 */
/*jslint indent: 2 */
/*global jIO, RSVP, Array*/
(function (jIO, RSVP, Array) {
  "use strict";

/**
   * The JIO Parallel Storage extension
   *
   * @class ParallelStorage
   * @constructor
   */
  function ParallelStorage (spec) {
    var i;
    
    if (spec.storage_list === undefined || !Array.isArray(spec.storage_list)) {
      throw new jIO.util.jIOError("storage_list is not an Array", 400);
    }

    this._storage_list = [];

    for (i = 0; i < spec.storage_list.length; i += 1) {
      this._storage_list.push(jIO.createJIO(spec.storage_list[i]));
    }
  }

    Every storage needs a class constructor which sets up the storage. In this case validate the parameters passed in the configuration and calling createJIO with the configurations passed in the storage_list parameter. Note this constructor does not return a promise. It's a synchronous call.

    
  ////////////////////////////////////////
  // Write methods: modify all storages
  ////////////////////////////////////////
  ParallelStorage.prototype.put = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].put.apply(this._storage_list[i], arguments));
    }
    return RSVP.all(promise_list);
  };

  ParallelStorage.prototype.remove = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].remove.apply(this._storage_list[i], arguments));
    }
    return RSVP.all(promise_list);
  };

  ParallelStorage.prototype.putAttachment = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].putAttachment.apply(this._storage_list[i], arguments));
    }
    return RSVP.all(promise_list);
  };

  ParallelStorage.prototype.removeAttachment = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].removeAttachment.apply(this._storage_list[i], arguments));
    }
    return RSVP.all(promise_list);
  };

  ////////////////////////////////////////
  // Read methods: get the fastest result from any storage
  ////////////////////////////////////////
  ParallelStorage.prototype.get = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].put.apply(this._storage_list[i], arguments));
    }
    return RSVP.any(promise_list);
  };

  ParallelStorage.prototype.getAttachment = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].getAttachment.apply(this._storage_list[i], arguments));
    }
    return RSVP.any(promise_list);
  };

  ParallelStorage.prototype.allAttachments = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].allAttachments.apply(this._storage_list[i], arguments));
    }
    return RSVP.any(promise_list);
  };

  ParallelStorage.prototype.hasCapacity = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].hasCapacity.apply(this._storage_list[i], arguments));
    }
    return RSVP.any(promise_list);
  };

  ParallelStorage.prototype.buildQuery = function () {
    var promise_list = [],
      i;
    for (i = 0; i < this._storage_list.length; i += 1) {
      promise_list.push(this._storage_list[i].buildQuery.apply(this._storage_list[i], arguments));
    }
    return RSVP.any(promise_list);
  };

    Afterwards all jIO methods that should be supported must be implemented on the class.

    
  jIO.addStorage("parallel", ParallelStorage);

}(jIO, RSVP, Array));

    The storage closes with adding the storage to the jIO object. After this it is available like all other storages.

    Streaming with jIO

    Streaming is a process by which we can fetch data in an app progressively and continuously in small chunks. Major benefit of streaming data is that we don't need to store the whole response and can save memory usage drastically. Streaming can be done very easily using jIO getAttachment method by specifying the range that needs to be fetched.

    Basic example:

    
// Initialize buffer
var buffer = [];
// When a specific event occurs, call jIO getAttachment with range specified.
storage.getAttachment(id, name, {
  start: start, // Starting position of the range(needs to be calculated)
  end: end, // End position of the range(needs to be calculated)
  format: array_buffer // Optional, default is Blob.
}).push(function (chunk) {
  // Do something with the chunk.
  buffer.append(chunk);
});

    Tips and Tricks

    CreateJIO is not Async

    When creating new storages make sure you don't pass the createJIO call as a return value of a RSVP Promise, because creating a new storage is not asynchronous, so the promise will resolve with an undefined return value instead of the storage.

    Using Queries Outside jIO

    Basic example:

    // object list (generated from documents in storage or index)
var object_list = [
  {"title": "Document number 1", "creator": "John Doe"},
  {"title": "Document number 2", "creator": "James Bond"}
];

// the query to run
var query = 'title: "Document number 1"';

// running the query
var result = jIO.QueryFactory.create(query).exec(object_list);
// console.log(result);
// [ { "title": "Document number 1", "creator": "John Doe"} ]

    Other example:

    var result = jIO.QueryFactory.create(query).exec(
  object_list,
  {
    "select": ['title", "year"],
    "limit": [20, 20], // from 20th to 40th document
    "sort_on": [["title", "ascending"], ["year", "descending"]],
    "other_keys_and_values": "are_ignored"
  }
);
// this case is equal to:
var result = jIO.QueryFactory.
  create(query).exec(object_list);
jIO.Query.sortOn([
  ["title", "ascending"],
  ["year", "descending"]
], result);
jIO.Query.limit([20, 20], result);
jIO.Query.select(["title", "year"], result);

    Wildchard Query Parameter

    Queries select items which exactly match the value given in the query but you can also use wildcards (%). If you don’t want to use a wildcard, just set the operator to =.

    var option = {
  query: 'creator:"% Doe"' // use wildcard
};

var option = {
  query: 'creator:="25%"' // don't use wildcard
};

    JiO Query JSON Schemas and Grammar

    Below you can find schemas for constructing queries.

    Complex Query JSON Schema:

    {
  "id": "ComplexQuery",
  "properties": {
    "type": {
      "type": "string",
      "format": "complex",
      "default": "complex",
      "description": "Type is used to recognize the query type"
    },
    "operator": {
      "type": "string",
      "format": "(AND|OR|NOT)",
      "required": true,
      "description": "Can be 'AND', 'OR' or 'NOT'."
    },
    "query_list": {
      "type": "array",
      "items": {
        "type": "object"
      },
      "required": true,
      "default": [],
      "description": "query_list is a list of queries which " +
                     "can be in serialized format " +
                     "or in object format."
    }
  }
}

    Simple Query JSON Schema:

    {
  "id": "SimpleQuery",
  "properties": {
    "type": {
      "type": "string",
      "format": "simple",
      "default": "simple",
      "description": "Type is used to recognize the query type."
    },
    "operator": {
      "type": "string",
      "default": "",
      "format": "(>=?|<=?|!?=|)",
      "description": "The operator used to compare."
    },
    "id": {
      "type": "string",
      "default": "",
      "description": "The column id."
    },
    "value": {
      "type": "string",
      "default": "",
      "description": "The value we want to search."
    }
  }
}

    JIO Query Grammar:

    
search_text
    : and_expression
    | and_expression search_text
    | and_expression OR search_text

and_expression
    : boolean_expression
    | boolean_expression AND and_expression

boolean_expression
    : NOT expression
    | expression

expression
    : ( search_text )
    | COLUMN expression
    | value

value
    : OPERATOR string
    | string

string
    : WORD
    | STRING

terminal:
    OR               -> /OR[ ]/
    AND              -> /AND[ ]/
    NOT              -> /NOT[ ]/
    COLUMN           -> /[^> /"(\\.|[^\\"])*"/
    WORD             -> /[^> /(>=?|<=?|!?=)/
    LEFT_PARENTHESE  -> /\(/
    RIGHT_PARENTHESE -> /\)/

ignore: " "

    Customizing jIO Query Search Keys

    Features like case insensitive, accent-removing, full-text searches and more can be implemented by customizing jIO’s query behavior.

    Let’s start with a simple search:

    var query = {
  type: "simple",
  key: "someproperty",
  value: comparison_value,
  operator: "="
}

    Each of the .someproperty attribute in objects’ metadata is compared with comparison_value through a function defined by the ‘=’ operator.

    You can provide your own function to be used as ‘=’ operator:

    var strictEqual = function (object_value, comparison_value) {
  return comparison_value === object_value;
};

var query = {
  type: "simple",
  key: {
    read_from: "someproperty",
    equal_match: strictEqual
  },
  value: comparison_value
}

    Inside equal_match, you can decide to interpret the wildcard character % or just ignore it, as in this case.

    If you need to convert or preprocess the values before comparison, you can provide a conversion function:

    var numberType = function (obj) {
  return parseFloat("3.14");
};

var query = {
  type: "simple",
  key: {
    read_from: "someproperty",
    cast_to: numberType
  },
  value: comparison_value
}

    In this case, the operator is still the default ‘=’ that works with strings. You can combine cast_to and equal_match:

    var query = {
  type: "simple",
  key: {
    read_from: "someproperty",
    cast_to: numberType,
    equal_match: strictEqual
  },
  value: comparison_value
}

    Now the query returns all objects for which the following is true:

    
strictEqual(numberType(metadata.someproperty),
            numberType(comparison_value))

    For a more useful example, the following function removes the accents from any string:

    
var accentFold = function (s) {
  var map = [
    [new RegExp("[àáâãäå]", "gi"), "a"],
    [new RegExp('æ', 'gi'), 'ae"],
    [new RegExp("ç", "gi"), "c"],
    [new RegExp("[èéêë]", "gi"), "e"],
    [new RegExp("[ìíîï]", "gi"), "i"],
    [new RegExp("ñ", "gi"), "n"],
    [new RegExp("[òóôõö]", "gi"), "o"],
    [new RegExp("œ", "gi"), "oe"],
    [new RegExp("[ùúûü]", "gi"), "u"],
    [new RegExp("[ýÿ]", "gi"), "y"]
  ];

  map.forEach(function (o) {
    var rep = function (match) {
      if (match.toUpperCase() === match) {
        return o[1].toUpperCase();
      }
      return o[1];
    };
    s = s.replace(o[0], rep);
  });
  return s;
};

...
  cast_to: accentFold
...

    A more robust solution to manage diacritics is recommended for production environments, with unicode normalization, like unorm (untested).

    Overriding jIO Query Operators and Sorting

    The advantage of providing an equal_match function is that it can work with basic types; you can keep the values as strings or, if you use a cast_to function, it can return strings, numbers, arrays... and that’s fine if all you need is the ‘=’ operator.

    It’s also possible to customize the behavior of the other operators: <, >, !=...

    To do that, the object returned by cast_to must contain a .cmp property, that behaves like the compareFunction described in Array.prototype.sort():

    function myType (...) {
  ...
  return {
    ...
    "cmp": function (b) {
      if (a < b) {
        return -1;
      }
      if (a > b) {
        return +1;
      }
      return 0;
    }
  };
}

...
  cast_to: myType
...

    If the < or > comparison makes no sense for the objects, the function should return undefined.

    The .cmp() property is also used, if present, by the sorting feature of queries.

    Partial Date/Time Matching in jIO Queries

    As a real life example, consider a list of documents that have a start_task property.

    The value of start_task can be an ISO 8601 string with date and time information including fractions of a second. Which is, honestly, a bit too much for most queries.

    By using a cast_to function with custom operators, it is possible to perform queries like “start_task > 2010-06”, or “start_task != 2011”. Partial time can be used as well, so we can ask for projects started after noon of a given day: start_task = "2011-04-05" AND start_task > "2011-04-05 12"

    The JIODate type has been implemented on top of the Moment.js library, which has a rich API with support for multiple languages and timezones. No special support for timezones is present (yet) in JIODate.

    To use JIODate, include the jiodate.js and moment.js files in your application, then set cast_to = jiodate.JIODate.

    jIO Query Key Schemas

    Instead of providing the key object for each attribute you want to filter, you can group all of them in a schema object for reuse:

    
var key_schema = {
  key_set: {
    date_day: {
      read_from: "date",
      cast_to: "dateType",
      equal_match: "sameDay"
    },
    date_month: {
      read_from: "date",
      cast_to: "dateType",
      equal_match: "sameMonth"
    }
  },
  cast_lookup: {
    dateType: function (str) {
      return new Date(str);
    }
  },
  match_lookup: {
    sameDay: function (a, b) {
      return (
        (a.getFullYear() === b.getFullYear()) &&
          (a.getMonth() === b.getMonth()) &&
            (a.getDate() === b.getDate())
      );
    },
    sameMonth: function (a, b) {
      return (
        (a.getFullYear() === b.getFullYear()) &&
          (a.getMonth() === b.getMonth())
      );
    }
  }
}

    With this schema, we have created two ‘virtual’ metadata attributes, date_day and date_month. When queried, they match values that happen to be in the same day, ignoring the time, or the same month, ignoring both time and day.

    A key_schema object can have three properties:

    • key_set - required.
    • cast_lookup - optional, an object of the form {name: function} that is used if cast_to is a string. If cast_lookup is not provided, then cast_to must be a function.
    • match_lookup - optional, an object of the form {name: function} that is used if equal_match is a string. If match_lookup is not provided, then equal_match must be a function.

    Using a schema

    A schema can be used:

    • In a query constructor. The same schema will be applied to all the sub-queries:
    jIO.QueryFactory.create({...}, key_schema).exec(...);
    • In the jIO.createJIO() method. The same schema will be used by all the queries created with the .allDocs() method:
    var jio = jIO.createJIO({
  type: "local",
  username: "...",
  application_name: "...",
  key_schema: key_schema
});

    Tests

    You can run tests after installing and building jIO by opening the /test/ folder.

    Browser Support

    jIO will work on fully HTML5 compliant browsers. Thus, jIO should work well with the latest version of Chrome and Firefox. IE is a stretch and Safari as well. Run the tests to find out if your browser is supported.

    Licence

    jIO is Free Software, licensed under the terms of the GNU GPL v3 (or later). For rationale, please see Nexedi licensing.