jIO
jIO is an abstract, promise-based JavaScript API that offers connectors to many storages (Dropbox, Google Drive, IndexedDB…) and special handlers for enhanced functionality (replication, compression, querying…). jIO separates storage access from the application, providing a simple way to switch backends and create offline-capable applications. jIO is developed and maintained by Nexedi and is used for the responsive ERP5 interface and as the basis for applications in app stores like OfficeJS.
What is a jIO Storage?
A storage is either a connector or a handler. Connectors store documents and attachments, providing access to the documents through the jIO API. Setting up a storage connector is easy:
// create a jIO LocalStorage
jIO.createJIO({
"type": "local",
"sessiononly": false
});
Handlers add increased functionality on top of connectors, and some of them extend the jIO API. It's just as easy to add a handler storage on top:
// create a jIO compressed LocalStorage
jIO.createJIO({
"type": "zip",
"sub_storage": {
"type": "local",
"sessiononly": false
}
});
Promises
jIO is fully asynchronous and uses an implementation based on RSVP.js. However, in order to safely cancel chains of promises, we had to create a custom version of RSVP.js, linked to further down, that does not use .then to chain promises. Instead, promise chains are written as RSVP.Queue().push(function () {...}).push(function (result) {...}).push(undefined, function (error) {...});
Another JavaScript Framework? Why use jIO?
Nexedi's free software products and the custom solutions developed from them must normally run for many years. As the complexity of our projects is usually very high, redevelopments to follow the current trending JavaScript framework or to replace legacy frameworks is out of our scope. Hence, we created jIO and RenderJS, two no-frills libraries that are:
- sturdy, small API, easy to use once understood.
- flexible, multiple storages and handlers.
- extendable, write your own storage if needed.
Getting Started
jIO is easy to set up and get working.
Source Code
The jIO source code is available on GitLab, with a mirror on GitHub. To build,
> git clone https://lab.nexedi.com/nexedi/jio.git
> make build
or just download the files directly:
or for NodeJS users, use npm:
> npm install jio
The following file might also be useful:
What is a Document?
A document is a collection of metadata and attachments. The metadata is the set of properties of the document and the attachments are binary objects that represent the content of the document. In jIO, the metadata is a plain JSON object with keys and values, and attachments are Blobs, for example:
// document metadata
{
title: 'A Title!',
creator: 'The Author'
}
// attachment blob
new Blob(["string data"], {type: "text/plain"});
Hello World
Create an HTML file with the specified JavaScript files and the following contents:
<!doctype html>
<html>
<head>
<script type="text/javascript" src="rsvp-latest.js"></script>
<script type="text/javascript" src="jio-latest.js"></script>
</head>
<body>
<p>jio example, see console</p>
<script type="text/javascript">
(function (jIO) {
var storage = jIO.createJIO({"type": "indexeddb", "database": "foo"});
console.log(storage);
function foo(message) {
return storage.put("start", message)
.push(function (result) {
console.log(result);
return storage.get("start");
})
.push(function (result) {
console.log(result)
}, function (error) {
console.warn(error);
throw error;
});
}
return foo("hello");
}(jIO));
</script>
</body>
</html>
Check your console to see how the jIO storage is created, the attachment is stored and fetched again.
You are now using jIO! Follow the OfficeJS Application Tutorial for a gentle introduction to application development using jIO and RenderJS, or keep on reading to dive straight into the full jIO API.
Posting Attachments
Below is an example of posting an attachment into LocalStorage to show the handling of attachments.
(function (jIO) {
// create a new jIO storage
var jio_instance = jIO.createJIO({"type": "indexeddb", "database": "foo"}});
// post the metadata for "myVideo"
return jio_instance.put("document", {
title: "My Video",
type: "MovingImage",
format: "video/ogg",
description: "Images Compilation"
})
// post a thumbnail attachment
.push(function () {
return jio_instance.putAttachment(
"document",
"thumbnail",
new Blob([my_image], {type: "image/jpeg"})
})
// post video attachment
.push(function () {
return jio_instance.putAttachment(
"document",
"video",
new Blob([my_video], {type: "video/ogg"})
})
// catch any errors and throw
.push(undefined, function(error) {
console.warn(error);
throw error;
});
}(jIO));
API - Quickguide
The original jIO interface was based on CouchDB, but has since evolved to the current set of methods described below and in more detail afterwards.
Method |
Example |
Info |
Create Storage |
jIO.createJIO({storage_configuration});
|
[returns jIO Storage]. Initialize a new storage or storage tree, synchronously. |
Post Document |
storage.post({
"key": "value"
});
|
[returns Promise]. Create a new document with the given metadata and returns the automatically generated ID. If post is unsupported, add a UuidStorage handler on top of your storage to provide it. |
Put Document |
storage.put(id, {
"key": "value"
});
|
[returns Promise]. Create or update the document with the given ID using the given metadata. |
Get Document |
storage.get(id);
// returns {"key": "value"}
|
[returns Promise]. Retrieve a document's metadata. |
Remove Document |
storage.remove(id);
|
[returns Promise]. Deletes a document and all its attachments. |
Search Documents |
storage.allDocs({
"query": query_object,
"limit": [3, 42],
"sort_on": [["key1", "ascending"], ["key2", "descending"]],
"select_list": ["key1", "key2", "key3"],
"include_docs": false,
});
// include_docs response
// {
// "total_rows": 39,
// "rows": [{
// "id": "text_id",
// "value": {},
// "doc": {"key": "value"}
// }, ...]
//}
// default response
//{
// "total_rows": 39,
// "rows": [{
// "id": "text_id",
// "value": {"select_list_key": "select_list_value"}
// }, {}, ...]
//}
|
[returns Promise]. Retrieve a list of documents. If include_docs is true, then doc in the response will contain the full metadata for each document. Otherwise, if select_list contains keys, then value in the response will contain the values of these keys for each document. If query is unsupported, add a QueryStorage handler on top of your storage to provide it. |
Add Attachment |
storage.putAttachment(id, name, blob);
|
[returns Promise]. Create or update the given blob as an attachment with the given name to the document with the given ID. |
Remove Attachment |
storage.removeAttachment(id, name);
|
[returns Promise]. Deletes the attachment with the given name from the document with the given ID. |
Get Attachment |
storage.getAttachment(id, name, {"format": "text"});
|
[returns Promise]. Retrieve the attachment with the given name from the document with the given ID in text, json, blob, data_url, or array_buffer format. |
All Attachments |
storage.allAttachments(id);
// default response
// {
// "attachment_id_1": {},
// "attachment_id_2": {}
// }
|
[returns Promise]. Retrieves an object containing the id's of all attachments belonging to the document that was passed as parameter |
Synchronize Storage |
storage.repair();
|
[returns Promise]. Synchronize or repair the storage. If repair is unsupported, add a ReplicateStorage handler on top of the two storages you wish to synchronize, to provide it. |
API - Storage Types
Below is the list of storages currently supported by jIO, including sample storage configurations. Note: authentication for storages, such as ERP5 or Dropbox, is not handled by jIO. You have to ensure that you are logged in or provide whatever tokens are required to access the storage when creating your jIO storage.
LocalStorage
Store documents in the browser's local storage. This storage has only one document, with the ID "/"
, so post, put, remove and get methods are not supported. The only operations you can do on a raw LocalStorage are attachment manipulation methods with blobs. To treat a LocalStorage as a regular storage, add a DocumentStorage handler on top of it, with "/"
as the document_id.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "local"). |
sessiononly |
no |
Boolean |
False (default): create a storage with unlimited duration.
True: the storage duration is limited to the user session. |
{
type: "local",
sessiononly: true
});
MemoryStorage
Store documents in a raw JavaScript object, in memory. The storage’s data isn't saved when your web page is closed or reloaded, and doesn’t take any other arguments.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "memory"). |
{
type: "memory"
});
IndexedDbStorage
Store documents in the IndexedDB database with the given name.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "indexeddb"). |
database |
yes |
String |
Name of the database. |
{
"type": "indexeddb",
"database": "mydb"
}
WebSqlStorage
Store documents in the WebSQL database with the given name. Using an IndexedDBStorage is strictly better.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "websql"). |
database |
yes |
String |
Name of the database. |
{
"type": "websql",
"database": "mydb"
}
WebDavStorage
Store documents in the WebDAV server with the given URL. Documents are WebDAV directories, so they must not contain any metadata, and their IDs must be bookended by forward slashes that directly correspond to their path. Attachments to documents are files inside WebDAV directories, so they must not contain any forward slashes. To treat a WebDavStorage as a regular storage, add a FileSystemBridgeStorage on top of it.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "dav"). |
url |
yes |
String |
URL of your WebDAV server. |
basic_login |
yes |
String |
Login and password of your WebDAV, base64 encoded like this: btoa(username + ":" + password) |
with_credentials |
no |
Boolean |
False (default): do not send domain cookie.
True: send domain cookie. |
// No authentication
{
"type": "dav",
"url": "http://mydav.com/"
}
// Basic authentication
{
"type": "dav",
"url": "https://securedav.com/",
"basic_login": btoa(username + ":" + password)
}
// Digest authentication is not implemented yet
DropboxStorage
Store documents in the Dropbox account with the given access token. Documents are Dropbox folders, so they must not contain any metadata, and their IDs must be bookended by forward slashes that directly correspond to their path. Attachments to documents are Dropbox files inside Dropbox folders, so they must not contain any forward slashes. To treat a DropboxStorage as a regular storage, add a FileSystemBridgeStorage on top of it.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "dropbox"). |
access_token |
yes |
String |
Access token for your dropbox. See the Dropbox documentation of its API v1 for how to generate an access token. |
root |
no |
String |
"dropbox" (default) for full access to account files.
"sandbox" for only access to files created by your app. |
{
"type": "dropbox",
"access_token": "sample_token",
"root": "sandbox"
}
GoogleDriveStorage
Store documents in the Google Drive account with the given access token. Unlike WebDavStorage and DropboxStorage, GoogleDriveStorage documents can contain metadata and have no specific rules regarding forward slashes. To treat a GoogleDriveStorage as a regular storage, add a FileSystemBridgeStorage on top of it.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "gdrive"). |
access_token |
yes |
String |
Access token for your Google Drive. See the Google Drive documentation for how to generate an access token. |
trashing |
no |
Boolean |
true (default): sends file to trash bin when calling "remove".
false: delete files permanently when calling "remove"" |
{
"type": "gdrive",
"access_token": "sample_token"
"trashing": true
}
Erp5Storage
Store documents in the ERP5 instance with the given URL. All ERP5 documents must contain values for portal_type and parent_relative_url in the metadata, which define the type of the document that is stored. Attachments are ERP5 actions. A raw Erp5Storage supports post and query, so there is no need to add a UuidStorage or QueryStorage on top of it.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "erp5"). |
url |
yes |
String |
URL of HATEOAS in your ERP5 instance. |
default_view_reference |
no |
String |
Reference of the action used to deliver of the document. |
{
"type": "erp5",
"url": "https://erp5.nexedi.net/web_site_module/hateoas",
"default_view_reference": "
}
ZipStorage (Handler)
Compress and decompress attachments to reduce network and storage usage.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "zip"). |
sub_storage |
yes |
Object |
Definition of storage whose attachments should be zipped. |
{
"type": "zip",
"sub_storage": {storage_definition}
}
ShaStorage (handler)
Provide the post method to create new documents using the SHA-1 hashes of their parameters as their IDs.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "sha"). |
sub_storage |
yes |
Object |
Definition of storage whose documents' IDs should be the SHA-1 hash of their parameters. |
{
"type": "sha",
"sub_storage": {storage_definition}
}
UuidStorage (handler)
Provide the post method to create new documents using randomly generated UUIDs as their IDs.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "uuid"). |
sub_storage |
yes |
Object |
Definition of storage whose documents' IDs should be randomly generated UUIDs. |
{
"type": "uuid",
"sub_storage": {storage_definition}
}
QueryStorage (handler)
Provide support for query parameters in the allDocs method.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "query"). |
sub_storage |
yes |
Object |
Definition of storage whose documents should be queryable when calling allDocs. |
{
"type": "query",
"sub_storage": {storage_definition}
}
CryptStorage (handler)
Encrypt and decrypt attachments to secure them. You must generate a Crypto key in JSON format to use this handler.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "crypt"). |
key |
yes |
String |
JSON crypto key. |
sub_storage |
yes |
Object |
Definition of storage whose attachments should be encrypted. |
var key, jsonKey, jio;
crypto.subtle.generateKey({name: "AES-GCM",length: 256},
(true), ["encrypt", "decrypt"])
.then(function(res){key = res;});
window.crypto.subtle.exportKey("jwk", key)
.then(function(res){jsonKey = res})
jio = jIO.createJIO({
"type": "crypt",
"key": json_key,
"sub_storage": {storage_definition}
}
UnionStorage (handler)
This handler takes a list of storages as its argument. When using a jIO method, UnionStorage tries it on the first storage of the array; if it fails, then UnionStorage tries the method on the next storage, and so on until success or there are no more storages left to try.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "union"). |
storage_list |
yes |
Array |
List of storage definitions. |
{
"type": "union",
"storage_list": [
{storage_definition},
{storage_definition},
...
{storage_definition}
]}
FileSystemBridgeStorage (handler)
Abstract file system storages such as WebDAV, Dropbox, and Google Drive, so that jIO methods work normally on them.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "drivetojiomapping"). |
sub_storage |
yes |
Object |
Definition of storage whose file system should be treated as a database. |
{
"type": "drivetojiomapping",
"sub_storage":
}
DocumentStorage (handler)
Create a storage on top of a single document by mapping documents to attachments, so that jIO methods work normally on single-document storages.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "document"). |
document_id |
yes |
String |
id of the document to use. |
repair_attachment |
no |
Boolean |
Verify if the document is in good state. (default to false) |
{
"type": "document",
"document_id": "text_id",
"repair_attachment": false
}
ReplicateStorage (Handler)
Synchronize documents between a local and a remote storage by providing the repair method.
Parameter |
Required? |
Type |
Description |
type |
yes |
String |
Name of the storage type (here: "replicate"). |
local_sub_storage |
yes |
Object |
Definition of the local storage, on which normal jIO operations are applied. |
remote_sub_storage |
yes |
Object |
Definition of the remote storage that synchronizes with the local storage. |
query |
no |
Object |
Query object to limit the synchronisation to specific files. |
use_remote_post |
no |
Boolean |
true: at file modification, modifies the local file id. false (default): at file modification, modifies the remote file id. |
conflict_handling
|
no |
Number |
0 (default): no conflict resolution (throws error) 1: keep the local state. 2: keep the remote state. 3: keep both states (no signature update) |
parallel_operation_amount |
no |
Number |
Control number of parallel operation for document synchronisation (default 1) |
parallel_operation_attachment_amount |
no |
Number |
Control number of parallel operation for attachment synchronisation (default 1) |
check_local_modification |
no |
Boolean (default: True) |
Synchronize when local documents are modified. |
check_local_creation |
no |
Boolean (default: True) |
Synchronize when local documents are created. |
check_local_deletion |
no |
Boolean (default: True) |
Synchronize when local documents are deleted. |
check_remote_modification |
no |
Boolean (default: True) |
Synchronize when remote documents are modified. |
check_remote_creation |
no |
Boolean (default: True) |
Synchronize when remote documents are created. |
check_remote_deletion |
no |
Boolean (default: True) |
Synchronize when remote documents are deleted. |
check_local_attachment_modification |
no |
Boolean (default: False) |
Synchronize when local attachments are modified. |
check_local_attachment_creation |
no |
Boolean (default: False) |
Synchronize when local attachments are created. |
check_local_attachment_deletion |
no |
Boolean (default: False) |
Synchronize when local attachments are deleted. |
check_remote_attachment_modification |
no |
Boolean (default: False) |
Synchronize when remote attachments are modified. |
check_remote_attachment_creation |
no |
Boolean (default: False) |
Synchronize when remote attachments are created. |
check_remote_attachment_deletion |
no |
Boolean (default: False) |
Synchronize when remote attachments are deleted. |
signature_hash_key |
no |
String (default: undefined) |
Use a document key as document signature hash (instead of calculating the SHA1). |
signature_sub_storage |
no |
Object (default: store in the local_storage) |
Definition of the signature storage, where replication signature are stored. |
report_level |
no |
Integer (default: 100) |
Verbosity of the report (0 = silent, 100 = error, 200 = warning, 300 = all write, 500 = all info) |
debug |
no |
Boolean (default: False) |
Log all report operation in the console |
{
type: "replicate",
local_sub_storage: {"type": "local"}
remote_sub_storage: {
"type": "dav",
"url": "http://mydav.com",
"basic_login": "aGFwcHkgZWFzdGVy"
}
use_remote_post: false,
conflict_handling: 2,
check_local_creation: false,
check_remote_deletion: false
}
jIO Query Engine
In jIO, a query can ask a storage server to select, filter, sort, or limit a document list before sending it back. If the server is not able to do so, the jIO query tool can do the filtering by itself on the client. Only the allDocs method can use jIO queries.
A query can either be a string (using a specific language useful for writing queries), or it can be a tree of objects (useful to browse queries). To handle queries, jIO uses a parsed grammar file which is compiled using JISON.
JIO queries can be used like database queries, for tasks such as:
- searching a specific document
- sorting a list of documents in a certain order
- not retrieving a list of ten thousand documents
- limiting the list to show only N documents per page
For some storages (like localStorage), jIO queries can be a powerful tool to query accessible documents. When querying documents on a distant storage, some server-side logic should be run to avoid returning too many documents to the client.
How to use Queries with jIO?
Queries can be triggered by including the option named query in the .allDocs() method call.
var options = {};
// search text query
options.query = '(creator:"John Doe") AND (format:"pdf")';
// OR query tree
options.query = {
type: "complex",
operator: "AND",
query_list: [{
type: "simple",
key: "creator",
value: "John Doe"
}, {
type: "simple",
key: "format",
value: "pdf"
}]
};
// FULL example using filtering criteria
options = {
query: '(creator:"% Doe") AND (format:"pdf")',
limit: [0, 100],
sort_on: [
["last_modified", "descending"],
["creation_date", "descending"]
],
select_list: ["title"]
};
// execution
jio_instance.allDocs(options, callback);
Creating Your Own Storage
Extending jIO by adding own storages is fairly easy as you only have to implement the base methods plus the internal methods hasCapacity (for which allDocs parameters are supported) and buildQuery (for constructing actual queries).
For example if you would want to create a parallel storage that maintains multiple storages on the same jIO gadget you could create a file named jio.parallelstorage.js
(or similar) and add it after the jIO file in your html. The file should contain:
/**
* JIO Parallel Storage Type = "Parallel".
* keep storages in parallel, without sync/replication
*/
/*jslint indent: 2 */
/*global jIO, RSVP, Array*/
(function (jIO, RSVP, Array) {
"use strict";
/**
* The JIO Parallel Storage extension
*
* @class ParallelStorage
* @constructor
*/
function ParallelStorage (spec) {
var i;
if (spec.storage_list === undefined || !Array.isArray(spec.storage_list)) {
throw new jIO.util.jIOError("storage_list is not an Array", 400);
}
this._storage_list = [];
for (i = 0; i < spec.storage_list.length; i += 1) {
this._storage_list.push(jIO.createJIO(spec.storage_list[i]));
}
}
Every storage needs a class constructor which sets up the storage. In this case validate the parameters passed in the configuration and calling createJIO with the configurations passed in the storage_list parameter. Note this constructor does not return a promise. It's a synchronous call.
////////////////////////////////////////
// Write methods: modify all storages
////////////////////////////////////////
ParallelStorage.prototype.put = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].put.apply(this._storage_list[i], arguments));
}
return RSVP.all(promise_list);
};
ParallelStorage.prototype.remove = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].remove.apply(this._storage_list[i], arguments));
}
return RSVP.all(promise_list);
};
ParallelStorage.prototype.putAttachment = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].putAttachment.apply(this._storage_list[i], arguments));
}
return RSVP.all(promise_list);
};
ParallelStorage.prototype.removeAttachment = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].removeAttachment.apply(this._storage_list[i], arguments));
}
return RSVP.all(promise_list);
};
////////////////////////////////////////
// Read methods: get the fastest result from any storage
////////////////////////////////////////
ParallelStorage.prototype.get = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].put.apply(this._storage_list[i], arguments));
}
return RSVP.any(promise_list);
};
ParallelStorage.prototype.getAttachment = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].getAttachment.apply(this._storage_list[i], arguments));
}
return RSVP.any(promise_list);
};
ParallelStorage.prototype.allAttachments = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].allAttachments.apply(this._storage_list[i], arguments));
}
return RSVP.any(promise_list);
};
ParallelStorage.prototype.hasCapacity = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].hasCapacity.apply(this._storage_list[i], arguments));
}
return RSVP.any(promise_list);
};
ParallelStorage.prototype.buildQuery = function () {
var promise_list = [],
i;
for (i = 0; i < this._storage_list.length; i += 1) {
promise_list.push(this._storage_list[i].buildQuery.apply(this._storage_list[i], arguments));
}
return RSVP.any(promise_list);
};
Afterwards all jIO methods that should be supported must be implemented on the class.
jIO.addStorage("parallel", ParallelStorage);
}(jIO, RSVP, Array));
The storage closes with adding the storage to the jIO object. After this it is available like all other storages.
Streaming with jIO
Streaming is a process by which we can fetch data in an app progressively and continuously in small chunks. Major benefit of streaming data is that we don't need to store the whole response and can save memory usage drastically. Streaming can be done very easily using jIO getAttachment method by specifying the range that needs to be fetched.
Basic example:
// Initialize buffer
var buffer = [];
// When a specific event occurs, call jIO getAttachment with range specified.
storage.getAttachment(id, name, {
start: start, // Starting position of the range(needs to be calculated)
end: end, // End position of the range(needs to be calculated)
format: array_buffer // Optional, default is Blob.
}).push(function (chunk) {
// Do something with the chunk.
buffer.append(chunk);
});
Tips and Tricks
CreateJIO is not Async
When creating new storages make sure you don't pass the createJIO call as a return value of a RSVP Promise, because creating a new storage is not asynchronous, so the promise will resolve with an undefined return value instead of the storage.
Using Queries Outside jIO
Basic example:
// object list (generated from documents in storage or index)
var object_list = [
{"title": "Document number 1", "creator": "John Doe"},
{"title": "Document number 2", "creator": "James Bond"}
];
// the query to run
var query = 'title: "Document number 1"';
// running the query
var result = jIO.QueryFactory.create(query).exec(object_list);
// console.log(result);
// [ { "title": "Document number 1", "creator": "John Doe"} ]
Other example:
var result = jIO.QueryFactory.create(query).exec(
object_list,
{
"select": ['title", "year"],
"limit": [20, 20], // from 20th to 40th document
"sort_on": [["title", "ascending"], ["year", "descending"]],
"other_keys_and_values": "are_ignored"
}
);
// this case is equal to:
var result = jIO.QueryFactory.
create(query).exec(object_list);
jIO.Query.sortOn([
["title", "ascending"],
["year", "descending"]
], result);
jIO.Query.limit([20, 20], result);
jIO.Query.select(["title", "year"], result);
Wildchard Query Parameter
Queries select items which exactly match the value given in the query but you can also use wildcards (%). If you don’t want to use a wildcard, just set the operator to =.
var option = {
query: 'creator:"% Doe"' // use wildcard
};
var option = {
query: 'creator:="25%"' // don't use wildcard
};
JiO Query JSON Schemas and Grammar
Below you can find schemas for constructing queries.
Complex Query JSON Schema:
{
"id": "ComplexQuery",
"properties": {
"type": {
"type": "string",
"format": "complex",
"default": "complex",
"description": "Type is used to recognize the query type"
},
"operator": {
"type": "string",
"format": "(AND|OR|NOT)",
"required": true,
"description": "Can be 'AND', 'OR' or 'NOT'."
},
"query_list": {
"type": "array",
"items": {
"type": "object"
},
"required": true,
"default": [],
"description": "query_list is a list of queries which " +
"can be in serialized format " +
"or in object format."
}
}
}
Simple Query JSON Schema:
{
"id": "SimpleQuery",
"properties": {
"type": {
"type": "string",
"format": "simple",
"default": "simple",
"description": "Type is used to recognize the query type."
},
"operator": {
"type": "string",
"default": "",
"format": "(>=?|<=?|!?=|)",
"description": "The operator used to compare."
},
"id": {
"type": "string",
"default": "",
"description": "The column id."
},
"value": {
"type": "string",
"default": "",
"description": "The value we want to search."
}
}
}
JIO Query Grammar:
search_text
: and_expression
| and_expression search_text
| and_expression OR search_text
and_expression
: boolean_expression
| boolean_expression AND and_expression
boolean_expression
: NOT expression
| expression
expression
: ( search_text )
| COLUMN expression
| value
value
: OPERATOR string
| string
string
: WORD
| STRING
terminal:
OR -> /OR[ ]/
AND -> /AND[ ]/
NOT -> /NOT[ ]/
COLUMN -> /[^> /"(\\.|[^\\"])*"/
WORD -> /[^> /(>=?|<=?|!?=)/
LEFT_PARENTHESE -> /\(/
RIGHT_PARENTHESE -> /\)/
ignore: " "
Customizing jIO Query Search Keys
Features like case insensitive, accent-removing, full-text searches and more can be implemented by customizing jIO’s query behavior.
Let’s start with a simple search:
var query = {
type: "simple",
key: "someproperty",
value: comparison_value,
operator: "="
}
Each of the .someproperty attribute in objects’ metadata is compared with comparison_value through a function defined by the ‘=’ operator.
You can provide your own function to be used as ‘=’ operator:
var strictEqual = function (object_value, comparison_value) {
return comparison_value === object_value;
};
var query = {
type: "simple",
key: {
read_from: "someproperty",
equal_match: strictEqual
},
value: comparison_value
}
Inside equal_match, you can decide to interpret the wildcard character % or just ignore it, as in this case.
If you need to convert or preprocess the values before comparison, you can provide a conversion function:
var numberType = function (obj) {
return parseFloat("3.14");
};
var query = {
type: "simple",
key: {
read_from: "someproperty",
cast_to: numberType
},
value: comparison_value
}
In this case, the operator is still the default ‘=’ that works with strings. You can combine cast_to and equal_match:
var query = {
type: "simple",
key: {
read_from: "someproperty",
cast_to: numberType,
equal_match: strictEqual
},
value: comparison_value
}
Now the query returns all objects for which the following is true:
strictEqual(numberType(metadata.someproperty),
numberType(comparison_value))
For a more useful example, the following function removes the accents from any string:
var accentFold = function (s) {
var map = [
[new RegExp("[àáâãäå]", "gi"), "a"],
[new RegExp('æ', 'gi'), 'ae"],
[new RegExp("ç", "gi"), "c"],
[new RegExp("[èéêë]", "gi"), "e"],
[new RegExp("[ìíîï]", "gi"), "i"],
[new RegExp("ñ", "gi"), "n"],
[new RegExp("[òóôõö]", "gi"), "o"],
[new RegExp("œ", "gi"), "oe"],
[new RegExp("[ùúûü]", "gi"), "u"],
[new RegExp("[ýÿ]", "gi"), "y"]
];
map.forEach(function (o) {
var rep = function (match) {
if (match.toUpperCase() === match) {
return o[1].toUpperCase();
}
return o[1];
};
s = s.replace(o[0], rep);
});
return s;
};
...
cast_to: accentFold
...
A more robust solution to manage diacritics is recommended for production environments, with unicode normalization, like unorm (untested).
Overriding jIO Query Operators and Sorting
The advantage of providing an equal_match function is that it can work with basic types; you can keep the values as strings or, if you use a cast_to function, it can return strings, numbers, arrays... and that’s fine if all you need is the ‘=’ operator.
It’s also possible to customize the behavior of the other operators: <, >, !=...
To do that, the object returned by cast_to must contain a .cmp property, that behaves like the compareFunction described in Array.prototype.sort():
function myType (...) {
...
return {
...
"cmp": function (b) {
if (a < b) {
return -1;
}
if (a > b) {
return +1;
}
return 0;
}
};
}
...
cast_to: myType
...
If the < or > comparison makes no sense for the objects, the function should return undefined.
The .cmp() property is also used, if present, by the sorting feature of queries.
Partial Date/Time Matching in jIO Queries
As a real life example, consider a list of documents that have a start_task property.
The value of start_task can be an ISO 8601 string with date and time information including fractions of a second. Which is, honestly, a bit too much for most queries.
By using a cast_to function with custom operators, it is possible to perform queries like “start_task > 2010-06”, or “start_task != 2011”. Partial time can be used as well, so we can ask for projects started after noon of a given day: start_task = "2011-04-05" AND start_task > "2011-04-05 12"
The JIODate type has been implemented on top of the Moment.js library, which has a rich API with support for multiple languages and timezones. No special support for timezones is present (yet) in JIODate.
To use JIODate, include the jiodate.js and moment.js files in your application, then set cast_to = jiodate.JIODate.
jIO Query Key Schemas
Instead of providing the key object for each attribute you want to filter, you can group all of them in a schema object for reuse:
var key_schema = {
key_set: {
date_day: {
read_from: "date",
cast_to: "dateType",
equal_match: "sameDay"
},
date_month: {
read_from: "date",
cast_to: "dateType",
equal_match: "sameMonth"
}
},
cast_lookup: {
dateType: function (str) {
return new Date(str);
}
},
match_lookup: {
sameDay: function (a, b) {
return (
(a.getFullYear() === b.getFullYear()) &&
(a.getMonth() === b.getMonth()) &&
(a.getDate() === b.getDate())
);
},
sameMonth: function (a, b) {
return (
(a.getFullYear() === b.getFullYear()) &&
(a.getMonth() === b.getMonth())
);
}
}
}
With this schema, we have created two ‘virtual’ metadata attributes, date_day and date_month. When queried, they match values that happen to be in the same day, ignoring the time, or the same month, ignoring both time and day.
A key_schema object can have three properties:
- key_set - required.
- cast_lookup - optional, an object of the form {name: function} that is used if cast_to is a string. If cast_lookup is not provided, then cast_to must be a function.
- match_lookup - optional, an object of the form {name: function} that is used if equal_match is a string. If match_lookup is not provided, then equal_match must be a function.
Using a schema
A schema can be used:
- In a query constructor. The same schema will be applied to all the sub-queries:
jIO.QueryFactory.create({...}, key_schema).exec(...);
- In the jIO.createJIO() method. The same schema will be used by all the queries created with the .allDocs() method:
var jio = jIO.createJIO({
type: "local",
username: "...",
application_name: "...",
key_schema: key_schema
});
Tests
You can run tests after installing and building jIO by opening the /test/ folder.
Browser Support
jIO will work on fully HTML5 compliant browsers. Thus, jIO should work well with the latest version of Chrome and Firefox. IE is a stretch and Safari as well. Run the tests to find out if your browser is supported.
Licence
jIO is Free Software, licensed under the terms of the GNU GPL v3 (or later). For rationale, please see Nexedi licensing.