(For a conceptual overview and design of Crux, see this page if you haven't already.)

Implementation of Remiges Crux¶

• Table of contents
• Implementation of Remiges Crux
  • Representing the schema of patterns
  • Representing the schema of actions
  • Representing a pattern
  • Representing an action
  • An entire rule
  • The schema manager
  • The rule manager
  • The matching engine
    • Matching one rule's pattern
    • Collecting the actions from one rule
    • Representing one entity
    • The overall matching engine
    • API for the matching engine

A rules engine implementation must include the following:

• RULE SCHEMA. A notation to specify the list of valid terms in a rule. This list will be separate for each class of entities. For instance, for items in inventory, the list of attributes may be:

  • Price
  • Full name
  • Age in stock
  • Quantity in inventory

  For vendors, the list could include:

  • Amount outstanding
  • Total value of business done in the last financial year

• RULE NOTATION. A notation to specify the pattern and actions of a rule.

• THE MATCHING ENGINE. Something which will take an entity with all its attributes, apply each rule to it, and follow the trail of rules to come up with a list of actions which will emerge.

So, if these three can be designed and then implemented, the core of a rules engine or a flow engine can be built.

Representing the schema of patterns¶

If using JSON, the schema of all valid patterns may be represented in structures of this form:

"patternschema": {
    "class": "inventoryitems",
    "attr": [{
        "name": "cat",
        "type": "enum",
        "vals": [ "textbook", "notebook", "stationery", "refbooks" ]
    },{
        "name": "mrp",
        "type": "float"
    },{
        "name": "fullname",
        "type": "str",
    },{
        "name": "ageinstock",
        "type": "int"
    },{
        "name": "inventoryqty",
        "type": "int"
    }]
}

In this example, the object patternschema is the schema for one category of entities. This schema says that for rules which work on entities of type inventoryitems, there are five attributes available which may be used to make patterns. Each attribute has a type. Boolean, enum types, integers, floating point numbers, timestamps (ts) and strings are supported. The example above does not have any attribute of type ts or bool.

So, the full schema of the rules engine will be an array of patternschema blocks. Initial examples have discussed inventory items and vendors. The patternschema block above is for inventory items. If the schema of patterns for vendors needed to be specified, there would be a second patternschema with “class”: “vendors”

While the fields in the example above are adequate from a purely functional point of view, it may be necessary to have some additional metadata to allow the building of a good UI which will allow users to manage these schema objects. So, an augmented data structure may look like this:

"patternschema": {
    "class": "inventoryitems",
    "attr": [{
        "name": "cat",
        "shortdesc": "Category of item",
        "longdesc": "Each item can belong to one of the following categories: textbooks, notebooks, stationery, or reference books.",
        "type": "enum",
        "vals": [ "textbook", "notebook", "stationery", "refbooks" ],
        "enumdesc": [ "Text books", "Notebooks", "Stationery and miscellaneous items", "Reference books, library books" ]
    },{
        "name": "mrp",
        "shortdesc": "Maximum retail price",
        "longdesc": "The maximum retail price of the item as declared by the manufacturer."
        "type": "float",
        "valmax": 20000,
        "valmin": 0
    },{
        "name": "fullname",
        "shortdesc": "Full name of item",
        "longdesc": "The full human-readable name of the item. Not unique, therefore sometimes confusing.",         
        "type": "str",
        "lenmax": 40,
        "lenmin": 5
    },{
        "name": "ageinstock",
        "shortdesc": "Age in stock, in days",
        "longdescr": "The age in days that the oldest sample of this item has been lying in stock",
        "type": "int",
        "valmax": 1000,
        "valmin": 1
    },{
        "name": "inventoryqty",
        "shortdesc": "Number of items in inventory",
        "longdescr": "How many of these items are currently present in the inventory",
        "type": "int",
        "valmax": 10000,
        "valmin": 0
    }]
}

Here, the shortdesc and longdesc are useful attributes of each attribute, for displaying labels and help text in any UI which is displayed to the human user who manages the rules for entities of this class. The valmax, valmin, lenmax, lenmin, allow the system to enforce some sanity checks on the patterns defined in any rules for this entity.

Representing the schema of actions¶

The schema of the action section of rules is simpler than patterns. Each rule's action section will contain a set of zero or more words, each denoting an action, and zero or more attribute assignments. There is no need for any type specification, etc.

• An example of an action word: invitefordiwali
• An example of an attribute assignment: discount=7

So, the schema of the actions will just specify the valid action names and the attribute names for assignments.

"actionschema": {
    "class": "inventoryitems",
    "actions": [ "invitefordiwali", "allowretailsale", "assigntotrash" ],
    "attribs": [ "discount", "shipby" ],
    "tags": [ "specialvendor", "tryoverseas" ]
}

The schema of actions above indicates that there are three actions, any or all of which may be present in any rule for this class of entities. There are two attributes which may be assigned values by any rule. And there are two tags for this class of entities – if a rule wishes to tag an entity with one or both of these tags, it may do so.

Putting the patternschema and actionschema blocks together, a better representation for the full schema for a class of entities will be:

"ruleschema": {
    "class": "inventoryitems",
    "patternschema": {
        "attr": [{
            "name": "cat",
            "type": "enum",
            "vals": [ "textbook", "notebook", "stationery", "refbooks" ]
        },{
            "name": "mrp",
            "type": "float"
        },{
            "name": "fullname",
            "type": "str",
        },{
            "name": "ageinstock",
            "type": "int"
        },{
            "name": "inventoryqty",
            "type": "int"
        }]
    }
    "actionschema": {
        "actions": [ "invitefordiwali", "allowretailsale", "assigntotrash" ],
        "attribs": [ "discount", "shipby" ],
    }
}

There will need to be one such ruleschema block for each class.

Representing a pattern¶

Below is an example of a pattern of a rule, which conforms to the schema example given above.

"rulepattern": {
    "pattern": [{
        "attr": "cat",
        "op": "eq",
        "val": "textbook"
    },{
        "attr": "mrp",
        "op": "ge",
        "val": 2000
    },{
        "attr": "ageinstock",
        "op": "ge",
        "val": 90
    },{
        "attr": "invitefordiwali",
        "op": "eq",
        "val": true
    }]
}

If a rule has this pattern, it will match any entity which falls in the class inventoryitems which

• is of type textbook
• has MRP (max retail price) greater than INR 2000
• has been in stock longer than 89 days
• one of the earlier rules matched against this entity has added the action invitefordiwali to the action set of this entity

It is important to note that a pattern does not need to have just the attributes listed in patternschema. It may also include actions listed in the ruleschema. Each such action becomes an implicit attribute of type bool for this class.

For attributes which are of type int, float, str and ts, the following comparison operators are available:

• Greater than or equal to: ge
• Greater than: gt
• Less than or equal to: le
• Less than: lt
• Equal to: eq
• Not equal to: ne

Collation sequences for strings are system dependent, and will need to be standardised so that they work reliably across programming languages and Unicode strings in any language. That's an implementation issue.

For enum and bool types, only eq and ne are available.

Representing an action¶

A rule has a set of one or more actions. The following are all examples of the action section of rules:

• invitefordiwali
• discount=7
• shipwithoutpo
• CALL=intlbiz

The terms which identify actions, e.g. invitefordiwali, will automatically be converted to lower-case and stored in the system. Reserved attribute names like CALL, RETURN, EXIT, will always be in uppercase. For an attribute assignment, the value of the attribute will be everything after the first = character till the end of the string, thus supporting multi-word values, e.g.

• reprimand=This cannot go on any longer

The action portion of a rule can have zero or one occurrence of a CALL term, a RETURN term, and an EXIT term. If it contains both a RETURN and an EXIT, then the RETURN will be ignored.

The action portion of a rule will have the following structure, shown here as an example:

"ruleactions": {
    "actions": [ "christmassale", "vipsupport" ],
    "attribs": [ "shipby=fedex" ],
    "call": "internationalrules",
    "return": true,
    "exit": false
}

This example shows all five attributes of ruleactions, but in reality, some of the attributes will typically be missing from most of the rules.

An entire rule¶

This is what an entire rule looks like:

"rule": {
    "class": "inventoryitems",
    "ver": 4,
    "rulepattern": [{
        "attr": "cat",
        "op": "eq",
        "val": "textbook"
    },{
        "attr": "mrp",
        "op": "ge",
        "val": 5000
    }],
    "ruleactions": {
        "actions": [ "christmassale" ],
        "attribs": [ "shipby=fedex" ]
    }
}

This structure represents one rule. The rule applies to entities of class inventoryitems. It has a pattern section which tries to match two attributes and an action section which throws up one action and one assignment.

A rule has a version number, which is incremented whenever the rule is updated. This number is for internal logging and rule engine debugging.

An array of such structures is a set of rules, and will be traversed in the order in which the rules appear in the array. Named rulesets will be represented thus:

"ruleset": {
    "class": "inventoryitems",
    "setname": "overseaspo",
    "rules": [{
        "ver": 4,
        "rulepattern": {
            :
            :
        },
        "ruleactions": {
            :
            :
        }
    }, {
        "ver": 3,
        "rulepattern": {
            :
            :
        },
        "ruleactions": {
            :
            :
        }
    }]
}

The example above shows a ruleset named overseaspo for class inventoryitems which has two rules. This ruleset may be invoked from any other rule with the action CALL=overseaspo.

The schema manager¶

The schema for each class of entities may be written by hand using a text editor. JSON or YAML files are easy to write. If the schema of one class has less than a dozen attributes, it may be short enough to edit or audit by hand. However, a tool to manage and maintain the schema eliminates typos and enforces various types of consistency, and a second-level implementation of a schema manager may also enforce authorisation policies.

A schema manager will have the following features:

• It will allow the user to create new instances of ruleschema
• It will sharply restrict editing of, and prevent deletion of any patternschema block or actionschema block if there are rules defined in the rules engine for this class of entities. In other words, schema are editable only as long as there are no rules for the class. The only kind of editing it will permit for “live” schema are
  • the addition of additional attributes in a patternschema or
  • additional attributes, action names or tags in an actionschema.
• It will ensure that there is no scope for typos when defining the schema.

The rule manager¶

The rule manager will allow a user to manage rules. Core functionality:

• It will provide a user interface to let the user edit rules.
• It will check each rule against the schema for the class, and will not give the user the opportunity to define any rule inconsistent with the schema.
• It will allow the user to move a rule up or down in the sequence, since ordering is important.
• If a rule is being defined with a CALL action, then the rule manager will ensure that a ruleset with that target name exists.
• Most important: it will provide a testing facility by which sample entities may be submitted to the rule engine for testing, and the rule manager will display a full trace showing which rules were attempted to match, which rules actually matched, and how the result set of actions, attributes, etc grew with each step. This feature will be provided without having to save the rule changes.
• Finally, when the editing session is complete and all rulesets need to be saved, it will perform a detailed cross-validation of all rules across each other to ensure consistency. If there is any inconsistency, it will give readable explanations of the problems and not permit saving of the updates.

The matching engine¶

The matching engine has a one-line job. It will take a full set of attributes of one entity, apply all the rules which apply to its class, and return with the list of actions, attributes, etc from all the matching rules.

The operation of the engine, in a highly simplified notation, is:

for each rule in the ruleset do
    match the pattern of the rule with the entity
    if the pattern matches, then
        collect the actions from the rule into the actionset
    endif
endfor

Matching one rule's pattern¶

The algorithm for the matching of one rule's pattern is shown below. Here, it is assumed that the object being matched is in entity and pattern of the rule being matched is in rulepattern. It is assumed that the matching engine may have proceeded some distance in its matching process, and may have collected zero or more actions in its actionset.

func matchOnePattern()
    input parameters: entity, rulepattern, actionset
    returns patternmatch: boolean

#
# In this loop we iterate through terms from our pattern array
# one by one
#
for patternterm in rulepattern do
    #
    # We get into a loop, stepping through the attributes of this
    # entity to pull out the value of the attribute
    # we will now be matching against the term we have selected
    # for this iteration of the outer loop, i.e. patternterm
    #
    entitytermval = null
    for entityoneterm in entity.attrs do
        if entityoneterm.attr == patternterm.attr then
            entitytermval = entityoneterm.val
        endif
    endfor

    if entitytermval == null then
        #
        # We reach here if none of the attributes in the entity
        # has a name matching the term in the pattern array. This
        # can only mean one thing: this is a pattern clause which
        # will match a tag in the entity. A tag is an action clause
        # which has already been collected against this entity from
        # matching a previous rule.
        #
        # So we will now cycle through the action clauses in the
        # actionset to see if any of those matches this patternterm.
        #
        for oneactionclause in actionset.actions do
            if oneactionclause == patternterm.attr then
                #
                # Bingo! We have found an action clause which matches
                # the name of an entry in the pattern array.
                #
                entitytermval = "true"
            endif
        endfor
    endif

    if entitytermval == null then
        #
        # If we reach here, it means that we have a term of the entity which
        # is not listed in the pattern at all. Every entity has all the
        # attributes listed against its class in the schema, but rule patterns
        # may have just one or two terms, so it's likely that many of the
        # attributes of an entity may not match any term in the pattern array.
        #
        # In that case we just loop to the next term in the pattern array.
        #
        continue
    endif

    case patternterm.op in
    "eq":
        if entitytermval != patternterm.val then
            return false
        endif
    "ne":
        if entitytermval == patternterm.val then
            return false
        endif
    endcase
    if patternterm.type in [ "int", "float", "ts", "str" ] then
        case patternterm.op in
        "le":
            if entitytermval > patternterm.val then
                return false
            endif
        "lt":
            if entitytermval >= patternterm.val then
                return false
            endif
        "ge":
            if entitytermval < patternterm.val then
                return false
            endif
        "gt":
            if entitytermval <= patternterm.val then
                return false
            endif
        default:
            log error with priority = CRITICAL: "system inconsistency with BRE rule terms"
        endcase
    endif
endfor

return true

Collecting the actions from one rule¶

If the pattern for one rule matches the entity being processed, then the actions of that rule will need to be added to the result set for that entity. Here we assume that the result of the action-collection function will return an object of the following structure. This object will be passed as input to the action-collecting function, and a (possibly extended) object will be returned, after merging the input object with the action terms from the rule just matched. The object structure will be:

"actionset": {
    "actions": [ "dodiscount", "yearendsale" ],
    "attribs": [ "shipby=fedex" ],
    "call": "overseaspo",
    "return": true,
    "exit": false
}

These five attributes will always be present in the object. The actions and attribs attributes will carry an array of strings, which will be a union set of all the action terms and attribute assignments collected from rules matched so far. The call attribute will either be a zero-length string (signifying that no ruleset needs to be called after this rule returns) or will carry the name of one ruleset to call after the current rule. The return and exit attributes will carry boolean values.

Performing a set union of action names is straightforward. Performing a set union of attribute assignments requires choosing one value of an attribute, if there was already the same attribute in the actionset and the current rule's actions also assigns a value to that attribute. In that case, the old value of the attribute will be overwritten by the new value.

function collectActions()
input parameters: actionset, ruleactions
    returns actionset

actionset.actions = actionset.actions UNION ruleactions.actions
actionset.attribs = actionset.attribs UNION ruleactions.attribs

actionset.call = ""
actionset.return = false
actionset.exit = false
if ruleactions.call is defined, then
    actionset.call = ruleactions.call
endif
if ruleactions.return is defined, then
    actionset.return = true
endif
if ruleactions.exit is defined,  then
    actionset.exit = true
endif

return actionset

The matching engine needs to look at what has emerged from collectActions() and then take action. The flow of the matching engine will change based on the values of the call, return and exit attributes.

Representing one entity¶

The matching engine matches all the rules of a ruleset against one instance of a class, like one instance of vendor or inventoryitem. How do we represent this object instance, when the type and the fields are all dynamically determined at runtime and varies from invocation to invocation? Here is one example:

"inputentity": {
    "class": "inventoryitems",
    "attribs": [{
        "name": "cat",
        "val": "refbook"
    },{
        "name": "mrp",
        "val": "1350"
    },{
        "name": "fullname",
        "val": "Advanced Level Physics, 2/ed"
    },{
        "name": "ageinstock",
        "val": "20"
    },{
        "name": "inventoryqty",
        "val": "540"
    }]
}

As this example highlights, all the values are supplied of type string, so that they may be converted from strings to their respective types later. This allows the data structure for specifying an object instance to be strongly typed and still allow attributes of all types to be captured. One more point illustrated is that all attributes in the patternschema of the class must be present in each object instance of that class.

This is the way the entity will be submitted to the matching engine for processing.

The overall matching engine¶

This engine will go through rules one after another, and for each rule, it will call matchOnePattern(). If matchOnePattern() returns true, it will call collectActions(). And then it will inspect the result obtained from collectActions() and decide what to do next.

This engine will be implemented by the getRules() function, which will occasionally call itself recursively. It will be called with three parameters:

• an inputentity, which will be matched against the ruleset
• a ruleset which will be traversed by the engine
• an actionset, which collects the result of the action matching

The pseudocode has been written with the assumption that parameters are all pass-by-value.

So, the doMatch() engine will work in the following way:

function doMatch()
input parameters: inputentity, ruleset, actionset
    returns actionset

for each onerule in ruleset do
    if matchOnePattern(inputentity, onerule.pattern, actionset) == true then
        actionset = collectActions(actionset, onerule.actions)
        #
        # now check if the actions just collected includes an EXIT clause
        #
        if actionset.exit == true then
            return actionset
        endif
        #
        # If there was no EXIT clause, check if there was a RETURN clause
        #
        if actionset.return == true then
            actionset.return = false
            return actionset
        endif
        #
        # If there was no EXIT or RETURN, check if there was a CALL clause
        #
        if actionset.call is not null then
            settocall = actionset.call
            if settocall.class != inputentity.class then
                log error with priority = CRITICAL:
                       "system inconsistency with BRE rule terms, attempting to call ", settocall, " from ", ruleset, "!"
            endif
            actionset.call = null
            doMatch(inputentity, settocall, actionset)
            #
            # If the called ruleset has set EXIT to true, then we too need to
            # exit, and our caller too needs to exit, ad infinitum
            #
            if actionset.exit == true then
                return actionset
            endif
        endif
    endif
    #
    # We come here because we've done one rule and we've neither been thrown
    # out by an EXIT nor a RETURN clause. So we now loop to the next rule in
    # our ruleset.
    #
endfor

return actionset

This matching engine will be able to traverse all rulesets, make "subroutine calls" from one ruleset to another, and finally come up with a consolidated actionset.

The outermost calling code which calls the outermost layer of doMatch() for a given entity will initialise an empty actionset and pass it in. After all the ruleset traversals, doMatch() will return with a loaded actionset, which will then be returned to the client of the BRE.

API for the matching engine¶

The matching engine must support the following set of operations:

• doMatch(): take an entity, pass it through all relevant rules and rulesets, and respond with the set of final results.
• getAttrSet(): take a class name, pull out from the patternschema all the attributes listed against that class, with full details. This is useful to let the caller know what attributes are to be specified when calling doMatch().