Transforms

Search…
Transforms
Script transforms operate on results and cursors to transform them in-flight. Transforms can be applied to Views, Policies and inline for Cursors and BulkOperations.
Though aggregation and projection is often much faster and less expensive that running a script transform, sometimes complex computation or data-interleaving is required that would be impractical to accomplish within a single script instance.
Transforms work by applying a script to an output cursor. Depending on the complexity and number of documents, a cursor transformation may be executed and re-queued multiple times before the cursor is finally exhausted.
When a running transform hits a timeout or ops threshold of 80%, it will stop iterating over the cursor, run the after finalizing method, store the memo object and re-queue for another run.
class Transform
To create a transform, simply define a class annotated with @transform. All methods are optional.
1
const { transform } = require('decorators-transform')
2
​
3
@transform
4
class Transformer {
5
​
6
  /**
7
   * Called if there is an error in the source operation. If the method
8
   * does not throw an error, Cortex throws the original error.
9
   *
10
   * @param err {Fault} thrown fault.
11
   */
12
  error(err) {
13
    throw err
14
  }
15
​
16
  /**
17
   * Called if the result is not a cursor (but a single result value).
18
   * The method must return a result if it's to be transformed. If the 
19
   * method does not return a value, Cortex outputs the original result value.
20
   *
21
   * @param result {*}
22
   * @returns {*}
23
   */
24
  result(result) {
25
    return result
26
  }
27
​
28
  /**
29
   * Cursor handler called only once and before all other processing.
30
   *
31
   * @param memo { Object } An empty object that can be modified and is saved 
32
   *        for the duration of the transformation.
33
   * @param cursor {Cursor} The output cursor.
34
   */
35
  beforeAll(memo, { cursor }) {
36
  }
37
​
38
  /**
39
   * Cursor handler called before each script run, and shares 
40
   * the current script context with each and after.
41
   *
42
   * @param memo { Object }
43
   * @param cursor {Cursor}
44
   */
45
  before(memo, { cursor }) {
46
  }
47
​
48
  /**
49
   * Cursor handler called with each cursor object. Returning undefined 
50
   * filters the object from the output stream.
51
   *
52
   * @param object { Object } the current output document
53
   * @param memo { Object }
54
   * @param cursor {Cursor}\
55
   * @returns {*} a value to be pushed to the output cursor.
56
   */
57
  each(object, memo, { cursor }) {
58
    return object
59
  }
60
​
61
  /**
62
   * Cursor handler called after before() and each() was called.
63
   *
64
   * @param memo { Object }
65
   * @param cursor {Cursor}
66
   * @param count { Number } The number of documents pushed by the script 
67
   *        runtime during the current execution.
68
   */
69
  after(memo, { cursor, count }) {
70
  }
71
​
72
  /**
73
   * Cursor handler called only once and after the cursor is exhausted.
74
   *
75
   * @param memo { Object }
76
   * @param cursor {Cursor}
77
   */
78
  afterAll(memo, { cursor }) {
79
  }
80
​
81
}
Copied!
Types
Policy Transform
A policy transform can handle a result value as well as a cursor. The define one, set the policy action to "Transform" and the script value to a transform export.
script.arguments:
policy { Object }
_id { ObjectID } The policy id
name { String } The policy name
policyOptions { Object }
triggered {String[]} The list of triggered conditions (methods, paths, etc.)
1
const { transform } = require('decorators-transform')
2
​
3
@transform
4
class PolicyTransform {
5
​
6
  error(err) {
7
    throw Fault.create('app.error.policyError', { faults: [err] })
8
  }
9
​
10
  result(result) {
11
    result.policyTriggers = script.arguments.policyOptions.triggered
12
    return result
13
  }
14
​
15
}
16
​
17
module.exports = PolicyTransform
Copied!
View Transform
A view transform acts on the result form a view cursor. The define one, set the script value to a transform export.
script.arguments:
view { Object }
_id { ObjectID } The view id
name { String } The view name
viewOptions { Object } The original view cursor options (object, skipAcl, paths, pipeline, etc.)
1
const { transform } = require('decorators-transform')
2
​
3
@transform
4
class ViewTransform {
5
  beforeAll(memo, { cursor }) {
6
​
7
    const { arguments: { view, viewOptions } } = script,
8
          { paths, object } = viewOptions
9
​
10
    cursor.push({
11
      object: 'transform',
12
      view, 
13
      viewOptions      
14
    })    
15
  }
16
}
17
​
18
module.exports = ViewTransform
Copied!
Inline Cursor Transform
A transform may be applied to any QueryCursor, AggregationCursor, or top-level BulkOperation.
script.arguments:
cursorOptions { Object } The original cursor options (object, skipAcl, paths, pipeline, etc.)
There are a few ways to define a transform for a cursor.
A series of implicitly usable transform methods:
1
const { c_datum: Datum } = org.objects
2
​
3
return Datum.find().transform(`
4
  beforeAll(memo, { cursor }) {
5
    Object.assign(memo, {
6
      object: 'resultTotals',
7
      sum: 0,
8
      deviceTypes: [],
9
      culled: 0
10
    })
11
  }
12
  each(object, memo) {
13
    if (object.c_inactive) {
14
      memo.culled += 1
15
      return // returning undefined filters the object from the output stream
16
    }
17
    memo.sum += object.c_amount || 0
18
    if (!memo.deviceTypes.includes(object.c_deviceType)) {
19
      memo.deviceTypes.push(object.c_deviceType)
20
    }
21
    return object // return the object to include it in the output stream.
22
  }
23
  afterAll(memo, { cursor }) {
24
    cursor.push(memo)
25
  }
26
`)
Copied!
An implicit transform class:
1
const { c_datum: Datum } = org.objects
2
​
3
return Datum.find().transform(`
4
  class {
5
    constructor() {
6
        this.runtTimestamp = Date.now()
7
    }
8
    each(object) {
9
        object.runtTimestamp = this.runtTimestamp
10
        return object
11
    }
12
  }
13
`)
Copied!
A reference to a Library Script that exports an annotated transform class:
1
const { c_datum: Datum } = org.objects
2
​
3
return Datum.find().transform('c_datum_transformer')
Copied!
An inline export definition:
1
const { c_datum: Datum } = org.objects
2
​
3
return Datum.find().transform({
4
  script: `
5
    const { transform } = require('decorators-transform'),
6
          { Accounts } = org.objects
7
    @transform
8
    class Transformer {
9
      each(object, memo, { cursor, index }) {
10
        return object._id
11
      }
12
    }
13
    module.exports = Transformer
14
  `
15
})
Copied!
Bulk Operations
Transformations can be applied to top-level bulk operations only.
1
const { accounts: Datum, accounts: Other, accounts: Device } = org.objects,
2
      _id = Device
3
        .readOne({owner: script.principal._id})
4
        .execute()._id // 1 device per account.
5
​
6
return org.objects.bulk()
7
  .add(
8
    Datum.find({ c_device: _id }).limit(10)
9
  )
10
  .add(
11
    Other.find({ c_device: _id }).limit(10).transform('c_some_transform')
12
  )
13
  .transform(`
14
    each(object) {
15
      return object.data // unwrap
16
    }
17
  `)
Copied!
Examples
CSV View Transform
1
const { transform } = require('decorators-transform'),
2
      schemas = require('schemas'),
3
      res = require('response')
4
​
5
let Undefined
6
​
7
// @transform
8
class CSVTransform {
9
​
10
  constructor() {
11
    this.quote = '"'
12
    this.doubleQuote = '""'
13
  }
14
​
15
  beforeAll(memo) {
16
​
17
    const { arguments: { view, viewOptions } } = script,
18
          { paths, object } = viewOptions
19
​
20
    memo.fields = paths || schemas.read(object, 'properties').filter(v => !v.optional).map(v => v.name)
21
​
22
    res.setHeader('Content-Type', 'text/csv')
23
    res.write('#view: ' + JSON.stringify(view) + '\n')
24
    res.write('#viewOptions: ' + JSON.stringify(viewOptions) + '\n')
25
    res.write(memo.fields.join(','))
26
  }
27
​
28
  each(object, memo) {
29
    res.write(
30
      '\n' +
31
      memo.fields.reduce((row, field) => {
32
        row.push(this.toCSV(object[field]))
33
        return row
34
      }, []).join(',')
35
    )
36
  }
37
​
38
  toCSV(value) {
39
    if (value === null || value === Undefined) {
40
      return Undefined
41
    }
42
    const valueType = typeof value
43
    if (valueType !== 'boolean' && valueType !== 'number' && valueType !== 'string') {
44
      value = JSON.stringify(value)
45
      if (value === Undefined) {
46
        return Undefined
47
      }
48
      if (value[0] === this.quote) {
49
        value = value.replace(/^"(.+)"$/, '$1')
50
      }
51
    }
52
    if (typeof value === 'string') {
53
      if (value.includes(this.quote)) {
54
        value = value.replace(new RegExp(this.quote, 'g'), this.doubleQuote)
55
      }
56
      value = this.quote + value + this.quote
57
    }
58
    return value
59
  }
60
​
61
}
62
​
63
module.exports = CSVTransform
Copied!
Inline CSV Transform
1
const { c_datum: Datum } = org.objects
2
​
3
return Datum.find().transform(`
4
  quote = '"'
5
  doubleQuote = '""'
6
  res = require('response)
7
  beforeAll(memo) {
8
    const schemas = require('schemas'),
9
          { arguments: { cursorOptions: { Object } } } = script
10
    memo.fields = schemas.read(object, 'properties').filter(v => !v.optional).map(v => v.name)
11
    this.res.setHeader('Content-Type', 'text/csv')
12
    this.res.write(memo.fields.join(','))
13
  }
14
  each(object, memo) {
15
    this.res.write(
16
      '\\n' +
17
      memo.fields.reduce((row, field) => {
18
        row.push(this.toCSV(object[field]))
19
        return row
20
      }, []).join(',')
21
    )
22
  }
23
  toCSV(value) {
24
    if (value === null || value === undefined) {
25
      return undefined
26
    }
27
    const valueType = typeof value
28
    if (valueType !== 'boolean' && valueType !== 'number' && valueType !== 'string') {
29
      value = JSON.stringify(value)
30
      if (value === undefined) {
31
        return undefined
32
      }
33
      if (value[0] === this.quote) {
34
        value = value.replace(/^"(.+)"$/, '$1')
35
      }
36
    }
37
    if (typeof value === 'string') {
38
      if (value.includes(this.quote)) {
39
        value = value.replace(new RegExp(this.quote, 'g'), this.doubleQuote)
40
      }
41
      value = this.quote + value + this.quote
42
    }
43
    return value
44
  }
45
`)
Copied!
Account Policy
This example triggers the policy on any individual account read
1
// Triggers on GET /account(s)?/([a-fa-f0-9]{24})
2
​
3
const { transform } = require('decorators-transform'),
4
      { equalIds } = require('util.id')
5
​
6
let Undefined
7
​
8
@transform
9
class PolicyTransform {
10
​
11
  // throw generic opaque error
12
  error(err) {
13
    throw Fault.create('app.error.accountAccess')
14
  }
15
​
16
  // throw access error even if the caller has access to the account
17
  // this might be the result of privileged JWT.
18
  result(result) {
19
    if (!equalIds(script.principal._id, result._id)) {
20
      throw Fault.create('app.accessDenied.account')
21
    }
22
    // add the policy triggers to the result for fun and profit
23
    // in this case: ['methods', 'paths']
24
    result.policyTriggers = script.arguments.policyOptions.triggered
25
    return result
26
  }
27
​
28
}
29
​
30
module.exports = PolicyTransform
Copied!
Televisit
Localization
Last modified 1mo ago
Copy link
Contents
Inline Cursor Transform