Skip to content

Commit

Permalink
Dbcore middleware support (dexie#754)
Browse files Browse the repository at this point in the history
This commit is a replacement of IndexedDB with something called DBCore. It's still a moving target how the DBCore API will be.

Goal: Enable middlewares in Dexie that can be a better alternative to hooks. Actually the current hooks API is now implemented via a middleware that would possibly be an optional addon (dexie-hooks) to slim down dexie.js.

Example uses of middlewares:

```js
const db = new Dexie('dbname');
db.version(x).stores(y);
db.use(someMiddleware);
```

## Example middleware posibilities:

* foreign key constraints
* computed indexes
* collated indexes
* hierarchies
* access control
* encryption

## Commit summaries

* A new middleware-friendly interface DBCore.
To replace calls to indexedDB.
The benefit of this will be an alternative to hooks:
* Enable fully asynchronic create/add/delete hooks. Will eventually enable asynchronic read hooks in future as well.
* Lower-level than Dexie but using Promises and bulk API.

First plan for this will be to make it possible to create middlewares supporting feaures like foreign key constraints, etc.

In the long term, the new middleware API will be simpler and a more robust alternative to hooks.

An example will be:
db.use(myMiddleware)
A middleware can optionally implement partial parts of the DBCore API and act as a proxy between Dexie and DBCore.

* dbcore layers seem to be complete.
Still in the middle of rewriting Dexie to use dbcore instead of going directly to indexedDB.
In this rewrite, all hook handling will be removed as it is now handled by a middleware instead, extept hook('reading') which need to be handled on the Dexie layer still.

* Almost there. Collection.delete() still needed.
Never run. Believe it will probably blow up when I do.

* Should be complete. No tests run yet. Compiles.

* Some fixes after trying running unit tests.
Still race issue about creating the dbcoretables.

* Bugfix dbcore-indexeddb

* Must set table.core both on _allTables and db.

* Issues with Cursor. Not solved.

* Now lots of unit tests succeed. But many still fails.

* More and more unit tests starts working...

* Empty toString message of ModifyError

* Non-bugfixes. Just cleanup code etc.

* Bugfix of tracking failures in Collection.modify

* Corrected failures array of BulkError

* Failing unit test in tests-open.
Failed when there were no objectStores in an existing database.

* Regression bug found by "misc: Adding object with falsy keys"

* Now all but one unit test succeeds.
One test needed to be changed as it invalidly expected certain order of events.
The test that now fails only fails on number of expected assertions. Will need to diff the log from it with a working one to find out why we get 57 assertions instead of the expected 53 assertions.

* Added test that verifies the posibility to modify a primary key.
Don't actually know if this was possible in earlier versions of Dexie, as it used IDBCursor.update() instead of the current modification strategy.

* Include the flow of modifying primary key in test "Dexie.currentTransaction in CRUD hooks".

* Finally all tests run on IE11, Safari 10 and Safari 11 and Chrome.
Let's see what travis thinks about this...

* Fixed failing unit test of Dexie.Observable.
In previous versions, updatingHook was called also when no actual changes were made.
New version will neither call updating hooks nor do any put() operation if the changes wouldn't have any effect.

* Repaired integration tests of Dexie.Observable by locking transaction using trans._promise instead of LockableTableMiddleware.

The difference I see is that trans._promise uses a recursive locking strategy, which may be needed for hooks because they act in a high level that once again might call down the dbcore stack and otherwise get stuck in a deadlock.

This would not be the case for pure middlewares where the middlewares should NOT call on Dexie methods again because it would possibly cause stack overflow.

* All tests seems to pass now.
The last changes had to do with the hooks middleware that needs to:
1. Run in upgraders and on populate as well due to how prior unit tests are done (syncable specifically).
2. Attach to transaction's table's hooks instead of db.table.hook because in certain upgrade scenarios, db.table may be deleted in later version but exist in transaction of intermediate version.

* Improve code coverage

* Fail instead of returning undefined when table is not found. Otherwise an obscure error will be thrown in some unrelated middleware.

* Include messages of nested exceptions on more than just one level.
Specifically, a unit test of Dexie.Syncable failed with inner exception Database.OpenError, with another inner exception telling the real reason.
This was not shown in the unit test log, but is now.

* Minor corrections
  • Loading branch information
dfahlander authored Oct 14, 2018
1 parent 2f3b134 commit fb73581
Show file tree
Hide file tree
Showing 48 changed files with 1,720 additions and 891 deletions.
4 changes: 4 additions & 0 deletions addons/Dexie.Observable/test/unit/hooks/tests-updating.js
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,10 @@ asyncTest('should not create an UPDATE change if the mods are already in the old
db.foo.add({id: ID, foo: 'bar'})
.then(() => {
return db.foo.update(ID, {foo: 'bar'});
}).then(()=>{
return db._changes.toArray((changes) => {
strictEqual(changes.length, 0, 'We have no change');
});
})
.catch(function(err) {
ok(false, "Error: " + err);
Expand Down
9 changes: 5 additions & 4 deletions src/classes/collection/collection-constructor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@ import { Dexie } from '../../classes/dexie';
import { makeClassConstructor } from '../../functions/make-class-constructor';
import { Collection } from './collection';
import { WhereClause } from '../where-clause/where-clause';
import { IDBKeyRange } from '../../public/types/indexeddb';
import { AnyRange } from '../../dbcore/keyrange';
import { KeyRange } from '../../public/types/dbcore';

/** Constructs a Collection instance. */
export interface CollectionConstructor {
new(whereClause?: WhereClause | null, keyRangeGenerator?: () => IDBKeyRange): Collection;
new(whereClause?: WhereClause | null, keyRangeGenerator?: () => KeyRange): Collection;
prototype: Collection;
}

Expand All @@ -23,10 +24,10 @@ export function createCollectionConstructor(db: Dexie) {
function Collection(
this: Collection,
whereClause?: WhereClause | null,
keyRangeGenerator?: () => IDBKeyRange)
keyRangeGenerator?: () => KeyRange)
{
this.db = db;
let keyRange = null, error = null;
let keyRange = AnyRange, error = null;
if (keyRangeGenerator) try {
keyRange = keyRangeGenerator();
} catch (ex) {
Expand Down
106 changes: 50 additions & 56 deletions src/classes/collection/collection-helpers.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
import { combine } from "../../functions/combine";
import { IDBObjectStore, IDBCursor } from "../../public/types/indexeddb";
import { exceptions } from "../../errors";
import { hasOwn, trycatcher } from "../../functions/utils";
import { wrap } from "../../helpers/promise";
import { eventRejectHandler } from "../../functions/event-wrappers";
import { Collection } from './';
import { DBCoreCursor, DBCoreTable, DBCoreTransaction, DBCoreTableSchema, RangeType } from '../../public/types/dbcore';
import { Table } from '../table';
import { nop } from '../../functions/chaining-functions';

type CollectionContext = Collection["_ctx"];

Expand All @@ -27,40 +29,45 @@ export function addMatchFilter(ctx: CollectionContext, fn) {
ctx.isMatch = combine(ctx.isMatch, fn);
}

export function getIndexOrStore(ctx: CollectionContext, store: IDBObjectStore) {
if (ctx.isPrimKey) return store;
var indexSpec = ctx.table.schema.idxByName[ctx.index];
if (!indexSpec) throw new exceptions.Schema("KeyPath " + ctx.index + " on object store " + store.name + " is not indexed");
return store.index(indexSpec.name);
export function getIndexOrStore(ctx: CollectionContext, coreSchema: DBCoreTableSchema) {
// TODO: Rewrite this. No need to know ctx.isPrimKey. ctx.index should hold the keypath.
// Still, throw if not found!
if (ctx.isPrimKey) return coreSchema.primaryKey;
const index = coreSchema.getIndexByKeyPath(ctx.index);
if (!index) throw new exceptions.Schema("KeyPath " + ctx.index + " on object store " + coreSchema.name + " is not indexed");
return index;
}

export function openCursor(ctx: CollectionContext, store: IDBObjectStore) {
var idxOrStore = getIndexOrStore(ctx, store);
return ctx.keysOnly && 'openKeyCursor' in idxOrStore ?
idxOrStore.openKeyCursor(ctx.range || null, ctx.dir + ctx.unique as IDBCursorDirection) :
idxOrStore.openCursor(ctx.range || null, ctx.dir + ctx.unique as IDBCursorDirection);
export function openCursor(ctx: CollectionContext, coreTable: DBCoreTable, trans: DBCoreTransaction) {
const index = getIndexOrStore(ctx, coreTable.schema);
return coreTable.openCursor({
trans,
values: !ctx.keysOnly,
reverse: ctx.dir === 'prev',
unique: !!ctx.unique,
query: {
index,
range: ctx.range
}
});
}

export function iter (
ctx: CollectionContext,
fn: (item, cursor: IDBCursor, advance: Function)=>void,
resolve,
reject,
idbstore: IDBObjectStore)
fn: (item, cursor: DBCoreCursor, advance: Function)=>void,
coreTrans: DBCoreTransaction,
coreTable: DBCoreTable): Promise<any>
{
var filter = ctx.replayFilter ? combine(ctx.filter, ctx.replayFilter()) : ctx.filter;
const filter = ctx.replayFilter ? combine(ctx.filter, ctx.replayFilter()) : ctx.filter;
if (!ctx.or) {
iterate(openCursor(ctx, idbstore), combine(ctx.algorithm, filter), fn, resolve, reject, !ctx.keysOnly && ctx.valueMapper);
} else (()=>{
var set = {};
var resolved = 0;

function resolveboth() {
if (++resolved === 2) resolve(); // Seems like we just support or btwn max 2 expressions, but there are no limit because we do recursion.
}
return iterate(
openCursor(ctx, coreTable, coreTrans),
combine(ctx.algorithm, filter), fn, !ctx.keysOnly && ctx.valueMapper);
} else {
const set = {};

function union(item, cursor, advance) {
if (!filter || filter(cursor, advance, resolveboth, reject)) {
const union = (item: any, cursor: DBCoreCursor, advance) => {
if (!filter || filter(cursor, advance, result=>cursor.stop(result), err => cursor.fail(err))) {
var primaryKey = cursor.primaryKey;
var key = '' + primaryKey;
if (key === '[object ArrayBuffer]') key = '' + new Uint8Array(primaryKey);
Expand All @@ -71,41 +78,28 @@ export function iter (
}
}

ctx.or._iterate(union, resolveboth, reject, idbstore);
iterate(openCursor(ctx, idbstore), ctx.algorithm, union, resolveboth, reject, !ctx.keysOnly && ctx.valueMapper);
})();
return Promise.all([
ctx.or._iterate(union, coreTrans),
iterate(openCursor(ctx, coreTable, coreTrans), ctx.algorithm, union, !ctx.keysOnly && ctx.valueMapper)
]);
}
}

function iterate(req, filter, fn, resolve, reject, valueMapper) {
function iterate(cursorPromise: Promise<DBCoreCursor>, filter, fn, valueMapper): Promise<any> {

// Apply valueMapper (hook('reading') or mappped class)
var mappedFn = valueMapper ? (x,c,a) => fn(valueMapper(x),c,a) : fn;
// Wrap fn with PSD and microtick stuff from Promise.
var wrappedFn = wrap(mappedFn, reject);
var wrappedFn = wrap(mappedFn);

if (!req.onerror) req.onerror = eventRejectHandler(reject);
if (filter) {
req.onsuccess = trycatcher(function filter_record() {
var cursor = req.result;
if (cursor) {
var c = function () { cursor.continue(); };
if (filter(cursor, function (advancer) { c = advancer; }, resolve, reject))
wrappedFn(cursor.value, cursor, function (advancer) { c = advancer; });
c();
} else {
resolve();
}
}, reject);
} else {
req.onsuccess = trycatcher(function filter_record() {
var cursor = req.result;
if (cursor) {
var c = function () { cursor.continue(); };
wrappedFn(cursor.value, cursor, function (advancer) { c = advancer; });
c();
} else {
resolve();
}
}, reject);
}
return cursorPromise.then(cursor => {
if (cursor) {
return cursor.start(()=>{
var c = ()=>cursor.continue();
if (!filter || filter(cursor, advancer => c = advancer, val=>{cursor.stop(val);c=nop}, e => {cursor.fail(e);c = nop;}))
wrappedFn(cursor.value, cursor, advancer => c = advancer);
c();
});
}
});
}
Loading

0 comments on commit fb73581

Please sign in to comment.