Remiges Products » Crux: BRE/WFE

# A rules engine for large business applications

{{>toc}}

## Background

We need to have a business rules engine. All the good open source rules engines we are seeing seem to prefer putting the rules in code. This means that a change in rules will require a build and deploy to propagate. Therefore, these products are for a quasi-static rules context. We need to support a more dynamic rules scenario, plus we need to provide a GUI to edit rules. In this scenario, if we are to use the popular rules-as-code category of BRE, we will have to generate code from the admin GUI, and then build a new release with the new rules code. This can be done, but is messy. Elegant people like us don’t like messy. Unless it’s spelt Messi.

So we are faced with the prospect of having to design our own rules engine. This rules engine will have absolutely nothing specific to the current project.

## What is a rule?

A rule has two "parts"

* A pattern specifying which entities the rule will apply to

* A what-to-do part, or action section, which specifies what we can do or not do with the matching entities

### The pattern

The first part will have a pattern notation against which we will match entities. For example, these are patterns:

* Items in the inventory

* Textbooks in the inventory

* Textbooks more expensive than INR 2,000 in the inventory

* Textbooks in stock for longer than 30 days in the inventory

* Vendors who owe us more than INR 100,000

* Vendors who supplied us more than INR 5,000,000 worth of goods in the last financial year

* Vendors who owe us less than INR 50,000 **AND** who supplied more than INR 2,000,000 worth of goods to us this year

* Vendor with the ID `"APZ00133"`

The rules engine may allow us to define such patterns, so that we can decide which items the rule will apply on. The last example specifies an explicit vendor instance, by specifying (presumably) its unique ID. Therefore, some rules may be defined for just one object instance. In other words, that rule is not a rule – it’s a codified exception to the general rules.

One can extend the pattern to have a negation operation. The rule will then apply to all entities which do not match the pattern.

### The actions

The second part of the rule is an action or a set of possible actions. Examples are:

* Discount equals 7% (for old stock items, or expensive items, or all items)

* Set credit limit to INR 200,000 (for vendors who owe us more than INR 100,000)

* Accept orders without a written PO (for vendors who have supplied goods more than INR 5,000,000 in the last year)

* Include in the Diwali Special Sale (for vendors who owe us less than INR 100,000)

* Accept orders without a written PO and include in the Christmas Sale (two actions in one rule – why not?)

* Use Fedex to ship out the item (for textbooks above a certain value)

Implementation challenges will be covered later. It’s just concepts at this point.

## Some refinements

### A “class of entity” attribute

A pattern has various attributes. In the items in the inventory examples, the following attributes were seen:

* Type of inventory item: *e.g.* textbook

* Price: *e.g.* INR 2,000

* Age of stock: *e.g.* 30 days

It becomes clear that attributes apply to a broad class of entities, but lose meaning across classes. Attributes which make sense to items in the inventory do not have any meaning when applied to vendors. Some attributes, like “full name”, or “date on which the entity was added to the system”, might apply to most items, but such attributes are very few. Therefore, it is perhaps necessary to define one fundamental attribute of each pattern: what broad class of entities does the pattern apply to? In database parlance, this is a mandatory non-NULL field, whose values will be from an enumerated set.

Another way to view this is to say that rules are partitioned into separate subsets or namespaces based on the value of this primary attribute. Rules which apply to one class do not have any overlap or engagement with rules which apply to another class. A rule can only apply to one class.

### When many rules match

When many rules match an entity, there must be a way to get an unambiguous result. The designer who is managing the rules must know exactly which rules will match for which entity. For example, if we have the following two rules:

* All textbooks

* All textbooks with price higher than INR 3,000

then the order in which matching is attempted will decide whether a book costing more than INR 3,000 will match the first rule or the second. And once one rule matches, the matching engine will need to decide whether to continue matching or to exit the matching loop.

One typical solution to the many-match problem is to break out of the matching process on first match. We call this the FIRST-MATCH algorithm. This allows for only very simple rulesets, because only one rule can possibly match an entity. Also, the ordering of rules in the rulebase becomes important in deciding which rule will match.

A second approach to resolve the ambiguity is to match the more specific rule, *i.e.* the pattern which carries the larger number of attributes -- MORE-SPECIFIC-RULE algorithm. For instance, in the two rules above, the first rule is more generic, because it matches all textbooks, while the second rule is more specific, because it matches only a subset of textbooks. If there was another rule with both price and weight attributes, that would apply to all matching entities.

What happens when there are two rules which match, each of whose patterns has two attributes? For example:

* All textbooks with price higher than INR 2,000

* All textbooks with quantity in stock greater than 20

If an entity matches all the attributes of both the patterns, which of the rules will apply? In such cases, different attributes may be assigned different weights to break the tie. For instance, price may be given higher weightage than quantity in stock. The pattern with the highest sum of attribute weights overrides the lighter pattern. This becomes complicated.

A third approach is to not choose which rules match, but match all the matching rules, and make a union set of all the actions from all the matching rules. This may be referred to as the MATCH-ALL algorithm.

For the BRE, we will follow the MATCH-ALL strategy till the end or till an `EXIT` action. For the flow engine (described below), we will follow the FIRST-MATCH algorithm which traverses rules in the order they are listed and returns on first match.

### Rulesets

Rules may be grouped into named sets. An action of one rule in one ruleset may trigger the equivalent of a subroutine call to a different ruleset, so that the matching algorithm continues with the rules of the second set. When the second ruleset runs to its end, the matching process returns to the point just after the calling rule in the first ruleset.

The matching process will always begin with a `main` ruleset, and may just end at the end of this `main` ruleset in some cases. In some specific cases, an entity will match a rule in the `main` ruleset which will specify a `CALL` action, naming a second ruleset. In that case, the matching process will continue in the second ruleset immediately after completing the rule which had the `CALL` action. After the called ruleset completes, the matching continues with the rule just after the one which triggered the call.

If the analogy of subroutine call is to be continued, then it must be noted that these subroutine calls are without any local variables, and the action list accumulated till the point of the `CALL` are globally visible within the called subroutines.

### Tags

If the engine applies the MATCH-ALL algorithm, it will keep applying each rule, and will pick up action items from each rule which matches. The engine therefore collects a set of action items as it traverses the ruleset(s).

Each action item collected against the entity becomes a **tag** which can then be used in subsequent patterns as a boolean attribute. For instance, if rule 14 throws up an action item `specialvendor`, then in rule 15 or 16, the pattern may test for `specialvendor == true`. Since the entity's action set by then has collected `specialvendor` as an action item, this evaluation of `specialvendor == true` will yield `true`. If the actionset had not included this action item, then the pattern evaluation would have evaluated to `false`.

Thus, actions collected on the way during the matching process become dynamically added attributes for the entity subsequently, of type `bool`, and may be used in the patterns of subsequent rules.

### Conditional rules

A rule may have an action which causes branching to one ruleset if the rule matches, and to another ruleset if it does not match. This is specified in the action section by having a `thenCall=ruleset1` field and an `elseCall=ruleset2` field.

If such a rule is encountered, then

* if the pattern matches the entity, the matching engine will trigger a `CALL` to `ruleset1`.

* If the pattern does not match the entity, then the engine will trigger a `CALL` to `ruleset2`.

In both cases, after the `CALL`, the engine will return to the point just after the conditional rule and resume from there.

A particular rule's actions may contain only a `thenCall` without any `elseCall`, or vice-versa, or both. A rule's actions may also have neither a `thenCall` nor an `elseCall`, in which case the engine simply continues with the next rule after processing the current rule's actions.

### `RETURN` and `EXIT` actions

When the matching engine is traversing a ruleset other than **`main`**, and it matches a rule which has the **`RETURN`** action, the engine will immediately exit the ruleset and return to the calling ruleset. If the matching engine encounters the **`EXIT`** action, it will immediately exit from the entire matching process and return to the client which had triggered the matching operation.

When the matching engine is traversing the main ruleset, **`RETURN`** and **`EXIT`** are synonymous.

### Values of properties

The action of a rule may not just specify a task, it may set a value of a property. Thus the rules matching algorithm may not just specify what task is permitted, but may specify the value of a variable too. For instance, for items in inventory, possible property-value settings could be:

* `shipby=fedex`

* `shipby=royalmail`

* `hsncode=9543`

## Rules engine as flow engine

It is possible to use a rules engine to specify flow, as in task flow or work flow. A task flow basically answers the following question: *“I am entity X of class ABC, and I have just completed operation PQR. What next, for me?”* For example, *"I am vendor PAXX8423, and I have just completed the processing step of initial registration. What's the next step I need to execute?"* Here, the class is "vendor", the entity is PAXX8423, and the operation PQR is "initial registration".

The basic question which a task flow or work flow answers at every step of a process is: *what next for me?*

To support this, special types of rules will need to be defined. These rules will have the following special features:

* The repository of flow rules will be separate from the business rules. There is no need for any overlap or mixing between the two sets, therefore they must be disjoint. However, the matching algorithm can be the same.

* Features like rule chaining, named rulesets, etc, may all be present, to allow for expressive power.

All business processes comprise a set of steps. This will reflect in the flow engine. All “flow” rulesets will have two special mandatory terms in their patterns:

* `process`: which will be one value from an enumerated set of values. Examples of processes could be:

  * Account creation

  * Asset liquidation

  * Identity verification

  * Any other multi-step business process

* `step`: which will be one value from an enumerated set of values. Examples of `step` could be (taking the example of account creation in a bank):

  * Initial personal details

  * Identity papers verification

  * Reference check

  * Initial amount deposit

These two attributes will be essential to tell the flow engine which process the entity is executing and what is the current step of the process which the entity has just completed, after which it is asking *“what next for me”.*

The flow engine will operate just like a rules engine, and will process rules in its repository,  and will arrive at a set of actions to be performed as the next step. If a single action emerges from the rule matching algorithm, then that step will need to be performed. If multiple actions emerge, then all those actions may be performed in parallel, asynchronously.

If multiple actions emerge from the flow engine, in some cases a special action may be part of the result set: `NEXTSTEP=xyz`. When this appears, it indicates that the entity may execute all the listed actions asynchronously, but must wait for all these asynchronous actions to complete before querying the flow engine again. And when all the actions are complete, it must query the flow engine specifying `xyz` as the value of the `step` attribute. This is a facility to make the flow re-synchronise after breaking into asynchronous concurrent threads.

## Implementation

The detailed software design is discussed in [[cruximplement|the algorithms and data structures page]].