Skip to content

OrderFlow Scripting Guide

Realtime Despatch Software Ltd

Document Version: 4.0.6

Document Built: 2020-10-12

This document and its content is copyright of Realtime Despatch Software Limited. All rights reserved.
You may not, except with our express written permission, distribute, publish or commercially exploit the content.
Any reproduction of part or all of the contents in any form is prohibited.

Import Mapping

Import Mapping

Import Mapping Pre-translation

Background

Imports to OrderFlow are received using a canonical 'flattened' format of property data such as the data shown below:

order.externalReference=TEST_minorder
order.channel=MYCHANNEL
order.deliveryAddressLine1=4 Quayside Place
order.deliveryContactName=Phil Zoio
order.deliveryEmailAddress=phil@realtimedespatch.co.uk

shipment.state=ready
shipment.method=metapack

orderLine.1.product.externalReference=DVD-BELOVED
orderLine.1.quantity=10

orderLine.2.product.externalReference=DVD-MATR
orderLine.2.quantity=20

The data received at this point can be manipulated as data prior to being mapped to the OrderFlow entity instances (orders, shipments, lines, etc.). For this reason, the mapping is described as the pre-translation mapping.

Pre-translation mappings map imported data field values to entity attributes, with the additional capability of writing to any other entity attributes or context values.

Some examples of what you may wish to do with pre-translation mappings:

  • map an incoming value to a different field name
  • change the string format of an incoming value
  • map one set of predefined values to another set of predefined values
  • introduce new field values, or even entire entity instances (e.g. order, shipment and product attributes).

An example

A complete example of an import mapping pre-transation is shown below:

<fieldmapper>
    <mappings useinput="true">
        <mapping qualifier="order" to="state">'validated'</mapping>
        <mapping qualifier="order" to ="deliveryPostCode">
        return value?.toUpperCase();
        </mapping>
        <mapping qualifier="order" from="paymentDetail" to ="paymentTransactionInfo"/>
        <mapping qualifier="shipment" to="method">
        <![CDATA[
        output['deliverySuggestionCode']=value;
        if (value == 'premium_method') {
        output['priorityName']='urgent';
        output['priority']=100;
        } else {
        output['priorityName']='normal';
        output['priority']=1;
        }
        ]]>
        </mapping>
        <mapping qualifier="order" remove="paymentDetail"/>
    </mappings>
</fieldmapper>

A few notes about the input mapping document follow below:

From and To Atributes

The to attribute of the mapping element defines the field in the target entity instance to which the data will be written. The value used can either be the literal value received from the incoming data, or the value returned from a script contained in the mapping element.

In the example above, the value validated is written to the order state field.

(Note that 'validated' is actually a Groovy script which simply returns the literal string value validated).

The from attribute is not mandatory. If present, it will be used as the source field name for the input data.

For example, in the example

<mapping qualifier="order" from="paymentDetail" to ="paymentTransactionInfo"/>

the input value in the field paymentDetail will be mapped to the output field paymentTransactionInfo.

Note that if no from attribute is present, then the source or input field name will be the same as the output field name.

Use input attribute

The top-level mappings element has a useinput attribute, which can be set to true or false. If true, then an attempt will be made to write every import data attribute (that is not explicitly removed) to its target entity.

Depending on the configuration, any incorrectly-named data will raise an error in OrderFlow, but at the very least will log a warning.

If useinput is set false, the mapping operation will only set field values for fields for which there is an explicit mapping in the field document. This will result in no redundant fields, but will tend to require more configuration to set up the full set of mappings.

Remove Attribute

The remove attribute is only required when useinput is true; it is used to prevent input values from being mapped to output fields for which no equivalent OrderFlow field value exists.

In our example above, the paymentDetail field value is removed, as it is not a field that corresponds with the OrderFlow data model.

<mapping qualifier="order" remove="paymentDetail"/>

Qualifiers

Each contained mapping element must have a qualifier attribute, which (for order imports) can take the values order, shipment or orderLine. This defines which entity the mapped data applies to.

There is an important restriction that needs to be understood here. During the mapping process, each qualifier effectively gets its own set of input values. As a result, it is not possible to map directly, for example, from input values received using the qualifier shipment to output values with the qualifier order, and vice versa.

There is a technique for getting around this where necessary, to be discussed below.

Scripting Context

The contents of the mapping element can be a literal value (enclosed in single- or double-quotes), or a Groovy script. The script has the following values available to it, as described below:

value

This is automatically populated from the imported data referenced by the from attribute, if present, otherwise using the to attribute.

Note that all of the examples below are equivalent, all of which map to the output map with the field name of state.

<mapping qualifier="order" from="state" to="state"/>

The script below relies on the fact that the from field and the to field use refer to the same attribute.

<mapping qualifier="order" to="state"/>

The example below relies on the fact that the variable value refers to the value for the to attribute.

<mapping qualifier="order" to="state">
return value;
</mapping>

input

This is a map that contains all the supplied properties from the import data (which can be read from directly).

The input variable can be used to reference any value from the input map belonging to the same qualifier.

For example, another way of achieving the same result as the previous three mappings is shown below.

<mapping qualifier="order" to="state">
return input['state'];
</mapping>

In practice, input map is used when there is the need to reference multiple input values to obtain a mapped value.

For example, 

<mapping qualifier="shipment" to="priority">
if (input['urgent'] == 'true' && input['shipping_option'] == 'expedited') {
    return '10';
} 
return '1';
</mapping>

In the example above, the input fields urgent and shipping_option are together used to determine the shipment priority.

output

This is a map that contains all the resulting mapped properties (which can be written to explicitly).

The main use for the output map is to allow multiple field values to written to using as single mapping entry.

For example, in a modified version of the previous example:

<mapping qualifier="shipment" to="priority">
if (input['urgent'] == 'true' && input['shipping_option'] == 'expedited') {
    output['priorityName'] = 'Urgent';
    return '10';
} 
output['priorityName'] = 'Normal';
return '1';
</mapping>

The priority name of 'Urgent' or 'Normal' can be set as a side effect of the previous mapping script. This avoids the need for the mapping document to contain repeated mapping entries with very similar logic.

context

This is a map that is available to write to and read from throughout the pre-translation processing.

The context can be used to allow data to be transferred across mapping inputs and outputs with different qualifiers.

Let's consider the import of an example of an order which has a single shipment and two order lines.

order.externalReference=TEST_minorder
order.channel=MYCHANNEL
order.deliveryAddressLine1=4 Quayside Place
order.deliveryContactName=Phil Zoio
order.deliveryEmailAddress=phil@realtimedespatch.co.uk
delivery_method=next_day

shipment.state=ready

orderLine.1.product.externalReference=DVD-BELOVED
orderLine.1.quantity=10

orderLine.2.product.externalReference=DVD-MATR
orderLine.2.quantity=20

The mapping input will be expanded into the following:

  • 1 set of mapping entries with qualifier order for the order values.
  • 1 set of mapping entries with qualifier shipment for the shipment values (beginning with the prefix shipmet..
  • 2 sets of mapping entries with qualifier orderLine, for the line values (beginning with the prefix orderLine.1. and orderLine.2..

In the example above, notice the presence of 

deliveryMethod=next_day

Because this entry has no qualifier, the qualifier is implicitly set to the top level entity, order. The problem occurs because there is no order field named delivery_method, although is a shipment field called deliverySuggestionName which would be a suitable target for this value.

However, it is not possible to map to this field using a mapping entry such as:

<mapping qualifier="order" to="delivery_method">
    output['deliverySuggestionName'] = value;
</mapping>

The way around this is to map the delivery method to a context attribute, then to map the context attribute to the target output field.

<mapping qualifier="order" to="delivery_method">
    context['deliverySuggestionName'] = value;
</mapping>
<mapping qualifier="shipment" to="deliverySuggestionName">
    return context['deliverySuggestionName'];
</mapping>
<mapping qualifier="order" remove="delivery_method"/>

For extra measure, we can remove the delivery_method mapping using the mapping with the remove attribute.

entries

The entries context variable gives us access to the complete set of input values that have been received across the entire input document.

We mentioned earlier that for the input document below that mapping input will support the creation of one order, one shipment and two order lines.

order.externalReference=TEST_minorder
...

shipment.state=ready
...

orderLine.1.product.externalReference=DVD-BELOVED
orderLine.1.quantity=10
...

orderLine.2.product.externalReference=DVD-MATR
orderLine.2.quantity=20
...

However, suppose we also receive additional values that we want to associate with the order, but we don't want to map to any existing order fields.

For this purpose, we need to add attributes.

The use of attributes tends to be domain specific. In the example of animal medicines, additional attributes might be:

  • animal_type (dog, cat, etc.)
  • animal_age (12 months)
  • animal_name (e.g. Rover)

The attributes could be received natively (without modification) using the following input file:

order.externalReference=TEST_minorder
orderAttribute.1.name=animal_type
orderAttribute.1.value=dog
orderAttribute.2.name=animal_age
orderAttribute.2.value=12 months
orderAttribute.3.name=animal_name
orderAttribute.3.value=Rover

However, suppose the attributes are received in the following way, which is more economical for the producer of the document:

order.externalReference=TEST_minorder
animal_type=dog
animal_age=12 months
animal_name=Rover

The entries variable can be used in a script as below: 

<mapping qualifier="order" to="animal_type">
    entries.addEntry('orderAttribute',1,'name','animal_type');
    entries.addEntry('orderAttribute',1,'value',value);
    entries.addEntry('orderAttribute',1,'orderItem','entity:order');
</mapping>
<mapping qualifier="order" to="animal_age">
    entries.addEntry('orderAttribute',2,'name','animal_age');
    entries.addEntry('orderAttribute',2,'value',value);
    entries.addEntry('orderAttribute',2,'orderItem','entity:order');
</mapping>
<mapping qualifier="order" to="animal_name">
    entries.addEntry('orderAttribute',3,'name','animal_name');
    entries.addEntry('orderAttribute',3,'value',value);
    entries.addEntry('orderAttribute',3,'orderItem','entity:order');
</mapping>

The entries.addEntry('orderAttribute',1,'name','animal_type') creates a mapping entry for the first input map for the orderAttribute qualifier, with the value animal_name for the attribute name.

By adding entries as above, additional entity instances can be populated using scripts.

Import Mapping Post-translation

Order import mappings allow imported data to be manipulated, irrespective of how they have been imported.
This manipulation is broadly split into what is known as pre-translation mappings and post-translation transformations.

This document details how to apply order import post-translation mappings.
Post-translation transformations apply changes to the imported entities after instantiation (but before persistence), so can also write to any related entity attributes, or context values.

An example of a post-transformation mapping is shown below:

<transformations>
    <transformation qualifier="order">
        <![CDATA[
        if (input.deliveryContact.mobilePhoneNumber == null) {
        input.deliveryContact.mobilePhoneNumber = input.deliveryContact.dayPhoneNumber;
        }
        ]]>
    </transformation>
</transformations>

The top-level transformations element contains transformation elements, each of which must have a qualifier attribute.
For order imports this can take the values order, shipment or orderLine (or indeed orderAttribute, shipmentAttribute or orderLineAttribute). This defines the type of entity to which the transformation will be applied.

Note that the transformation will be applied to all entities of that type created during the import process, so if there is more than one orderLine entity created, a transformation with the qualifier orderLine will be applied to each of these order lines.

There are two main strategies for applying individual translations:

  • scripted transformations, which involves the running of a Groovy script defined within the transformation element.
  • built-in transformations, involving the running of some predefined functionality.

Built-in transformations are discussed later in this document. The example above contains a single scripted transformation mapping, which will be discussed next.

Scripted Post-translators

The contents of the transformation element can be a literal value (enclosed in single- or double-quotes), or a Groovy script.

Unlike pre-transformations, the return value for post transformations is not important, as the scripts operate directly on OrderFlow entity instances after they have been fully populated using existing and received incoming data, but before these updated/created entities have been persisted. In other words, there is no need for return statement in a post-translation script; any value returned from the script will simply be ignored.

The following variables are available in the scripting context for an import post-transformation.

input

This is an instance of the entity referenced by the 'qualifier' attribute.

It is worth noting that the a post-translation script has much broader and direct access to the OrderFlow data model than the pre-translation mapping script.

While the pre-translation only has access to the data that feeds the import operation, it does not have access to the OrderFlow entity instances.

By contrast, the post-translation script can access OrderFlow orders, shipments, order lines and other entities directly using the OrderFlow Scripting API. This is evident in the example script below.

<transformation qualifier="order">
    <![CDATA[
    if (input.deliveryContact.mobilePhoneNumber == null) {
        input.deliveryContact.mobilePhoneNumber = input.deliveryContact.dayPhoneNumber;
    }
    ]]>
</transformation>

Note how the input variable is used to access the order entity instance directly, and to manipulate its field values.

context

Contains a map to which contextual data can be written. For example, this provides a mechanism by which data can be shared between different post-translation mappings.

Note that the context can be populated declaratively using the context and parameter elements, as shown in the example below:

<transformations>
    <context>
    <parameter name="postalSortFileProperty">royalmail.postal.sort.file</parameter>
    </context>
    <transformation .../>
</transformations>

context.importStateHolder

One of the values that will automatically be available in the context is stored under the key importStateHolder. This is an instance of the Java class rtd.imports.spi.ImportStateHolder, and contains the context of the entire import operation.

Use of this is pretty rare, but it does allow for contextual data that is used in the import process but not the OrderFlow domain model to be accessed (and, if appropriate, modified).

<transformation qualifier="productAttribute">        
<![CDATA[
if (input.isPersisted()) {
    def context = context.importStateHolder.context;
    context.get('suppressPersistence').add(input);
}        
]]>
</transformation>

Built-in Import Handlers

In addition to scripted post-translation handers, there are also built-in transformation handlers which can be used to manipulate an incoming order, shipment or OrderLIne.

<transformations>
<transformation qualifier="shipment" handler="weightCalculator">
</transformation>
</transformations>

As with the scripted transformation, the built-in transformation uses a qualifier to determine the scope of the transformation.

Among the main built-in post translators include:

Name Qualifier Usage
postalSorter order Applies a postal sort value to a shipment.
courierSetterTransformer shipment Sets the courier and/or service for a shipment.
batchSetterTransformer shipment Sets the batch type of a shipment after invoking the batch selection script.
batchTypeTransformer shipment Sets the batch type of a shipment from a context property value.
weightCalculator shipment Calculates the shipment weight from the weight of its constituent products.
goodsPriceCalculator shipment Calculates the shipment goods price from the order line or product data.
countryCodeSetter shipment Validates a supplied two character country code

Note that the transformation needs to be set correctly for the built-in tranformation implementation used.

A bit more detail on each of the above is provided next.

postalSorter

The postal sorter allows for the shipment.presortValue to be set during an order import. For some couriers, doing a presort at the warehouse can result in a cheaper rate for shipments.

The postalSorter transformation uses a spreasheet file which contains a mapping of post code prefixes to sort values, as in the example below

sortcode postcode
1  AB
2  AL
3  B
4  BA
5  BB
6  BD
7  BH
8  BL
9  BN
10 BR
11 BS
...

If the postalSorter built-in transformation is to be used, it needs to be accompanied by the name of the property which stores the postal sort file, this can be done as in the example below:

<transformations>
    <context>
    <parameter name="postalSortFileProperty">royalmail.postal.sort.file</parameter>
    </context>
    <transformation qualifier="order" handler="postalSorter">
    </transformation>
</transformations>

courierSetterTransformer

The courierSetterTransformer allows the selection of the courier to be determined at import item using business rules defined in the Courier Selection Script.

The declaration of the courierSetterTransformer transformer is simple, as shown below.

   <transformations>
      <transformation qualifier="shipment" handler="courierSetterTransformer">
      </transformation>
   </transformations>

Details on the Courier Selection Script are provided later in this document.

batchSetterTransformer

The batchSetterTransformer allows the selection of the batch types of shipments that are to undergo batch picking to be determined at import item using business rules defined in the Batch Selection Script.

As with the courierSetterTransformer, the declaration of the batchSetterTransformer transformer is simple, as shown below.

<transformations>
    <transformation qualifier="shipment" handler="batchSetterTransformer">
    </transformation>
</transformations>

Details on the Batch Selection Script are provided later in this document.

One word of recommendation is that if both courierSetterTransformer and the batchSetterTransformer are both present in the same mapping document, it is advisable to put the courier transformation first, as this allows the batch selection script to use an already populated courier selection, if required.

batchTypeTransformer

The batchTypeTransformer performs a similar function to the batchSetterTransformer, except that it uses the output of a script within the transformation element to determine the batch type, rather than an exteranlly defined Batch Selection Script. An example is shown 

<transformation qualifier="shipment" handler="batchTypeTransformer" scriptFirst="true">
    <![CDATA[
    if (input.orderLineCount > 1) {
    context.batchType = 'multiline'
    } else {
    context.batchType = 'singleline'
    }
    ]]> 
</transformation>

Notice how batch type is stored in the context. Notice also the use of the scriptFirst attribute. This ensures that when both a built-in handler and a script are present within the same transformation, the script is evaluated before the handler functionality is invoked.

The batchTypeTransformer is rarely used, as it is less flexible than the Batch Selection Script, which allows the batch type to be set at other points in the order processing workflow, and not just at import time.

weightCalculator

The weightCalculator is a useful transformation to apply to ensure that the weight value for shipments is set, even if not received as part of the shipment import data.

The usage of the weight calculator is also very simple, as shown below:

<transformations>
    <transformation qualifier="shipment" handler="weightCalculator">
    </transformation>
</transformations>

There are a few points to note about the use of the weight calculator as it is currently implemented:

  • the weightCalculator will only set the weight of shipments if all products for order lines in the shipment have a weight value set. If the weight value is missing, then no weight value will be set for the shipment.
  • if the weightCalculator is used, then it will override any value that is present in the data received from the eCommerce system.

If a weight value is required for the shipment, then it is possible to reject the import using a subsequent transformation, or even using a script within the same transformation.

goodsPriceCalculator

The goods price calculator, as the name suggests, allows for the goods price value to be calculated from order line or product data.

The declaration of the goodsPriceCalculator is shown below:

<transformations>
    <transformation qualifier="shipment" handler="goodsPriceCalculator">
    </transformation>
</transformations>

A couple of noteworthy points about the behaviour of the goods price calculator:

  • if the order goods price is already set in the received order data, then no price calculation will be applied.
  • the calculation will attempt to use the order line price values. If no values are set at the order line level, the handler will attempt to set the goods price using the product price values.
  • if the price is not available for one of the order lines (or products), then no goods price will be set.

A More Complete Example

A complete example of an import post-translation mapping, which uses all of the elements described above, is shown below.

<transformations>
    <context>
    <parameter name="postalSortFileProperty">royalmail.postal.sort.file</parameter>
    </context>
    <transformation qualifier="order">
    <![CDATA[
    if (input.deliveryContact.mobilePhoneNumber == null) {
        input.deliveryContact.mobilePhoneNumber = input.deliveryContact.dayPhoneNumber;
    }
    ]]>
    </transformation>
    <transformation qualifier="order" handler="postalSorter">
    </transformation>
    <transformation qualifier="shipment" handler="courierSetterTransformer">
    </transformation>
    <transformation qualifier="shipment" handler="batchTypeTransformer" scriptFirst="true">
    <![CDATA[
    if (input.orderLineCount > 1) {
    context.batchType = 'multiline'
    } else {
    context.batchType = 'singleline'
    }
    ]]>
    </transformation>
</transformations>