Transforms

Search…
Transforms
Script transforms operate on results and cursors to transform them in-flight. Transforms can be applied to Views, Policies and inline for Cursors and BulkOperations.
Though aggregation and projection is often much faster and less expensive that running a script transform, sometimes complex computation or data-interleaving is required that would be impractical to accomplish within a single script instance.
Transforms work by applying a script to an output cursor. Depending on the complexity and number of documents, a cursor transformation may be executed and re-queued multiple times before the cursor is finally exhausted.
When a running transform hits a timeout or ops threshold of 80%, it will stop iterating over the cursor, run the after finalizing method, store the memo object and re-queue for another run.
class Transform
To create a transform, simply define a class annotated with @transform. All methods are optional.
const { transform } = require('decorators-transform')
​
@transform
class Transformer {
​
  /**
   * Called if there is an error in the source operation. If the method
   * does not throw an error, Cortex throws the original error.
   *
   * @param err {Fault} thrown fault.
   */
  error(err) {
    throw err
  }
​
  /**
   * Called if the result is not a cursor (but a single result value).
   * The method must return a result if it's to be transformed. If the 
   * method does not return a value, Cortex outputs the original result value.
   *
   * @param result {*}
   * @returns {*}
   */
  result(result) {
    return result
  }
​
  /**
   * Cursor handler called only once and before all other processing.
   *
   * @param memo { Object } An empty object that can be modified and is saved 
   *        for the duration of the transformation.
   * @param cursor {Cursor} The output cursor.
   */
  beforeAll(memo, { cursor }) {
  }
​
  /**
   * Cursor handler called before each script run, and shares 
   * the current script context with each and after.
   *
   * @param memo { Object }
   * @param cursor {Cursor}
   */
  before(memo, { cursor }) {
  }
​
  /**
   * Cursor handler called with each cursor object. Returning undefined 
   * filters the object from the output stream.
   *
   * @param object { Object } the current output document
   * @param memo { Object }
   * @param cursor {Cursor}\
   * @returns {*} a value to be pushed to the output cursor.
   */
  each(object, memo, { cursor }) {
    return object
  }
​
  /**
   * Cursor handler called after before() and each() was called.
   *
   * @param memo { Object }
   * @param cursor {Cursor}
   * @param count { Number } The number of documents pushed by the script 
   *        runtime during the current execution.
   */
  after(memo, { cursor, count }) {
  }
​
  /**
   * Cursor handler called only once and after the cursor is exhausted.
   *
   * @param memo { Object }
   * @param cursor {Cursor}
   */
  afterAll(memo, { cursor }) {
  }
​
}
Types
Policy Transform
A policy transform can handle a result value as well as a cursor. The define one, set the policy action to "Transform" and the script value to a transform export.
script.arguments:
policy { Object }
_id { ObjectID } The policy id
name { String } The policy name
policyOptions { Object }
triggered {String[]} The list of triggered conditions (methods, paths, etc.)
const { transform } = require('decorators-transform')
​
@transform
class PolicyTransform {
​
  error(err) {
    throw Fault.create('app.error.policyError', { faults: [err] })
  }
​
  result(result) {
    result.policyTriggers = script.arguments.policyOptions.triggered
    return result
  }
​
}
​
module.exports = PolicyTransform
View Transform
A view transform acts on the result form a view cursor. The define one, set the script value to a transform export.
script.arguments:
view { Object }
_id { ObjectID } The view id
name { String } The view name
viewOptions { Object } The original view cursor options (object, skipAcl, paths, pipeline, etc.)
const { transform } = require('decorators-transform')
​
@transform
class ViewTransform {
  beforeAll(memo, { cursor }) {
​
    const { arguments: { view, viewOptions } } = script,
          { paths, object } = viewOptions
​
    cursor.push({
      object: 'transform',
      view, 
      viewOptions      
    })    
  }
}
​
module.exports = ViewTransform
Inline Cursor Transform
A transform may be applied to any QueryCursor, AggregationCursor, or top-level BulkOperation.
script.arguments:
cursorOptions { Object } The original cursor options (object, skipAcl, paths, pipeline, etc.)
There are a few ways to define a transform for a cursor.
A series of implicitly usable transform methods:
const { c_datum: Datum } = org.objects
​
return Datum.find().transform(`
  beforeAll(memo, { cursor }) {
    Object.assign(memo, {
      object: 'resultTotals',
      sum: 0,
      deviceTypes: [],
      culled: 0
    })
  }
  each(object, memo) {
    if (object.c_inactive) {
      memo.culled += 1
      return // returning undefined filters the object from the output stream
    }
    memo.sum += object.c_amount || 0
    if (!memo.deviceTypes.includes(object.c_deviceType)) {
      memo.deviceTypes.push(object.c_deviceType)
    }
    return object // return the object to include it in the output stream.
  }
  afterAll(memo, { cursor }) {
    cursor.push(memo)
  }
`)
An implicit transform class:
const { c_datum: Datum } = org.objects
​
return Datum.find().transform(`
  class {
    constructor() {
        this.runtTimestamp = Date.now()
    }
    each(object) {
        object.runtTimestamp = this.runtTimestamp
        return object
    }
  }
`)
A reference to a Library Script that exports an annotated transform class:
const { c_datum: Datum } = org.objects
​
return Datum.find().transform('c_datum_transformer')
An inline export definition:
const { c_datum: Datum } = org.objects
​
return Datum.find().transform({
  script: `
    const { transform } = require('decorators-transform'),
          { Accounts } = org.objects
    @transform
    class Transformer {
      each(object, memo, { cursor, index }) {
        return object._id
      }
    }
    module.exports = Transformer
  `
})
Bulk Operations
Transformations can be applied to top-level bulk operations only.
const { accounts: Datum, accounts: Other, accounts: Device } = org.objects,
      _id = Device
        .readOne({owner: script.principal._id})
        .execute()._id // 1 device per account.
​
return org.objects.bulk()
  .add(
    Datum.find({ c_device: _id }).limit(10)
  )
  .add(
    Other.find({ c_device: _id }).limit(10).transform('c_some_transform')
  )
  .transform(`
    each(object) {
      return object.data // unwrap
    }
  `)
Examples
CSV View Transform
const { transform } = require('decorators-transform'),
      schemas = require('schemas'),
      res = require('response')
​
let Undefined
​
// @transform
class CSVTransform {
​
  constructor() {
    this.quote = '"'
    this.doubleQuote = '""'
  }
​
  beforeAll(memo) {
​
    const { arguments: { view, viewOptions } } = script,
          { paths, object } = viewOptions
​
    memo.fields = paths || schemas.read(object, 'properties').filter(v => !v.optional).map(v => v.name)
​
    res.setHeader('Content-Type', 'text/csv')
    res.write('#view: ' + JSON.stringify(view) + '\n')
    res.write('#viewOptions: ' + JSON.stringify(viewOptions) + '\n')
    res.write(memo.fields.join(','))
  }
​
  each(object, memo) {
    res.write(
      '\n' +
      memo.fields.reduce((row, field) => {
        row.push(this.toCSV(object[field]))
        return row
      }, []).join(',')
    )
  }
​
  toCSV(value) {
    if (value === null || value === Undefined) {
      return Undefined
    }
    const valueType = typeof value
    if (valueType !== 'boolean' && valueType !== 'number' && valueType !== 'string') {
      value = JSON.stringify(value)
      if (value === Undefined) {
        return Undefined
      }
      if (value[0] === this.quote) {
        value = value.replace(/^"(.+)"$/, '$1')
      }
    }
    if (typeof value === 'string') {
      if (value.includes(this.quote)) {
        value = value.replace(new RegExp(this.quote, 'g'), this.doubleQuote)
      }
      value = this.quote + value + this.quote
    }
    return value
  }
​
}
​
module.exports = CSVTransform
Inline CSV Transform
const { c_datum: Datum } = org.objects
​
return Datum.find().transform(`
  quote = '"'
  doubleQuote = '""'
  res = require('response)
  beforeAll(memo) {
    const schemas = require('schemas'),
          { arguments: { cursorOptions: { Object } } } = script
    memo.fields = schemas.read(object, 'properties').filter(v => !v.optional).map(v => v.name)
    this.res.setHeader('Content-Type', 'text/csv')
    this.res.write(memo.fields.join(','))
  }
  each(object, memo) {
    this.res.write(
      '\\n' +
      memo.fields.reduce((row, field) => {
        row.push(this.toCSV(object[field]))
        return row
      }, []).join(',')
    )
  }
  toCSV(value) {
    if (value === null || value === undefined) {
      return undefined
    }
    const valueType = typeof value
    if (valueType !== 'boolean' && valueType !== 'number' && valueType !== 'string') {
      value = JSON.stringify(value)
      if (value === undefined) {
        return undefined
      }
      if (value[0] === this.quote) {
        value = value.replace(/^"(.+)"$/, '$1')
      }
    }
    if (typeof value === 'string') {
      if (value.includes(this.quote)) {
        value = value.replace(new RegExp(this.quote, 'g'), this.doubleQuote)
      }
      value = this.quote + value + this.quote
    }
    return value
  }
`)
Account Policy
This example triggers the policy on any individual account read
// Triggers on GET /account(s)?/([a-fa-f0-9]{24})
​
const { transform } = require('decorators-transform'),
      { equalIds } = require('util.id')
​
let Undefined
​
@transform
class PolicyTransform {
​
  // throw generic opaque error
  error(err) {
    throw Fault.create('app.error.accountAccess')
  }
​
  // throw access error even if the caller has access to the account
  // this might be the result of privileged JWT.
  result(result) {
    if (!equalIds(script.principal._id, result._id)) {
      throw Fault.create('app.accessDenied.account')
    }
    // add the policy triggers to the result for fun and profit
    // in this case: ['methods', 'paths']
    result.policyTriggers = script.arguments.policyOptions.triggered
    return result
  }
​
}
​
module.exports = PolicyTransform
Televisit
Localization
Last modified 1yr ago
Copy link
On this page
class Transform
Types
Policy Transform
View Transform
Inline Cursor Transform
Bulk Operations
Examples
CSV View Transform
Inline CSV Transform
Account Policy