Global

# Software design: general rules

{{>toc}}

## Web services: basic rules

* We will expect requests in `POST`, not `GET`. We will not design URLs to communicate request parameters.

* All JSON data structures will have member names in full lowercase. No mixed-case will be used.

## Web service request format

* Example

``` json

{

    "ver": 1,

    "uid": "xpq1234",

    "authtoken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ",

    "data": {

        "goal_id": 23

    }

}

```

* **`ver`** is the API version to be used. See the section on "API version" below.

* **`uid`** is user id for user-oriented applications. It will be an opaque one-word string. The field may be called  **`boxid`** for embedded applications where IoT boxes invoke web services. It may be **`sysid`** or **`serverid`** where there are no human users, but servers invoke web services. For some web service calls it will be optional, _e.g._ for operations which do not need any user identification or box identification.

* **`authtoken`** is the (optional) JWT authentication token, which will be one long, opaque string in three sections, separated by periods, as can be seen here. Some web services may not require authenticated access, and for those calls, this field may be dropped. See the section on "Authentication" below.

* **`data`** is the request data object. Each web service will define its own data object, with some mandatory and some optional fields.

In some cases, a web service only requires `uid` or `boxid` as input. In those cases value of `data` will be an empty object.

e.g.

``` json

{

    "ver": 1,

    "boxid": 1234,

    "data": {}

}

```

## Web service response format

A consistent JSON data structure will be returned by the API for success or errors. For successes, the following JSON structure will be returned:

``` javascript

{

    "status": "success",

    "data": {

    /* Application-specific data payload goes here. */

    },

    "messages": [] /* Or optional success message */

}

```

The contents of `data` for each web service call is provided in its detailed documentation. Note how the `message` block is an empty array here.

For errors, the following JSON structure will be returned:

``` javascript

{

    "status": "error",

    "data": {},

    "messages": [{

        "code": "code1",

        "msg": "message1"

    }, {

        "code": "code2",

        "msg": "message2"

    }]

}

```

Note that the `data` block is empty in case of errors, and the error messages will be listed in `message`. An example of an error response:

``` json

{

    "status": "error",

    "data": {},

    "messages": [{

        "code": "permission_denied",

        "msg": "No permission for the requested operation."

    }, {

        "code": "account_restricted",

        "msg": "The user account is disabled."

    }]

}

```

## Reporting errors by web services

Each error reported has two parts:

* the error code

* the human readable error message

The code is always going to be one word, always in lower case, with letters, digits, and/or underscore characters.

The code is the real indicator of the type of error, and must be used by the client code. It may handle the error, it may retry the operation, it may branch into a different path in the code, it may select a message to show the end-user, _etc_, as needed. The error message supplied by the web service is meant for the programmer of the client software to understand and debug the web service call. It is _not_ meant to be used for end-user display.

In multi-lingual user interfaces, the error message must be indexed to the combination of `(language, errcode)` and the appropriate language-specific error message must be selected from a table and displayed to end-users. The responsibility of designing and selecting meaningful error messages in the correct language lies with the client software. The client software dev team must build error message tables in each supported language. The error message supplied by the web service will always be in English, and may not have appropriate wording to display directly to the end-user. Its audience is the programmer of the client software.

## ID formats

Object instances have unique IDs. All IDs internally generated by the web service, _e.g._ goal-ID, user-ID, transaction-ID, product-ID, portfolio-ID, ops-list-ID, _etc.,_ are all to be treated as opaque strings, _not_ as integers.

The rules for ID syntax are:

* opaque, meaningless string

* between 1 and 50 octets long, variable length

* one word, _i.e._ no spaces or special characters in the word, only letters and digits

* case sensitive

* may start with a digit or a letter

* equality test possible, _i.e._ if two ID strings are identical as strings and refer to the objects of the same class, then they refer to the same instance

* guaranteed unique, _i.e._ if two object instances of the same class exist in the system, they are guaranteed to have two separate ID strings

* ordering is meaningless, _i.e._ "greater than" or "less than" operation is meaningless

Note that this means that the web service may use integers as IDs, and all these rules will be fulfilled. But while representing them in JSON, the IDs will always be enclosed in double-quotes, even if they "look like" integers.

Examples of IDs:

* `"a0ae48b16f90db8f3c542f44f8103701"`

* `"5"`

* `"750294852039"`

Examples of strings which are not IDs:

* `"a0ae48b16f90db8f3c542f4/f8103701"` _(there's a special character in it)_

* `5` _(a digit without quotes is not a string)_

* `"776,245,664"` _(commas are not permitted in IDs)_

* `"62.6"` _(the period is not permitted)_

## Authentication

Typically, the web services layer will need to authenticate clients which are invoking them. If it's a human-user oriented application, then the human will need to be authenticated sometime. If the web services are invoked by devices, then each device will need to be authenticated at least once, maybe periodically, to ensure that the API does not remain completely open to outside access.

Therefore, there will be at least one `login()` web service in every application, where the client will identify itself. 

**User-based authentication**: The `login()` call may accept a username and password, or there may be a back-end OTP loop where the user just submits his username and the system sends an OTP to his registered mobile number which is later submitted through a second web service.

Finally, on successful authentication, the web service will return a JWT token to the client. This JWT token will need to be submitted as part of the fixed header fields of all subsequent web services which expect authenticated access.

**Device-based authentication**: The `login()` call may be designed to require a unique ID. This ID may be generated from some unique aspects of the device hardware (_e.g._ 

```

dmidecode | grep relevant-lines | md5sum

```

on Linux, as a very rough mechanism, or some ROM-embedded unique serial number embedded there by the hardware manufacturer, or some other naturally occurring unique ID like Ethernet MAC address.

This ID will uniquely (we assume) identify one device. The web service will check to see if this device is authorised to access the system, and may do some back-end cross-checks using OTP to some authorised person's mobile phone or some other two-factor authentication. The device may then have to make a second call to submit the second factor for the authentication to complete. Finally, when the web service is satisfied that all is well, it will issue a JWT token to the client. This client will need to be included as part of the fixed header of all subsequent web service requests. All web services which expected authenticated access will first check the validity of the token before processing other parameters. If the token is invalid, the web service will respond with a `authtoken_invalid` error.

**JWT token replacement**: All JWT tokens expire, and it is a good idea to let them refresh, for security reasons. When a web service receives an expired token in its request, it will respond immediately with a special error code indicating `authtoken_expired` without looking at other parameters. What happens next will depend on the application design.

* **Token replacement**: One approach will be to issue a fresh token in lieu of the old one. There will be a special web service whose only job will be to issue a new token. It will take the old token, check all other fields for validity, check that the internal user tables or device masters still permit this client to continue using the services, and then it will create and send back a new token with a fresh expiry date. The client will receive this new token and will use it henceforth for all accesses. To use this approach, the client code will have to be designed to detect `authtoken_expired` errors and automatically make this call for generating a fresh token.

* **Fresh authentication**: For higher security, it may be necessary, specially in user-based applications, to do a full authentication afresh. In this approach, the user will simply be logged out and will be asked to "sign in again". Therefore, whatever was done the first time to authenticate the user or device will now have to be re-done, and manual intervention may be needed at the client end.

## API versions

Any web service (we will refer to it as "service" here) which is exposed to multiple clients (software systems which send web services requests to our service) will, over time, have to serve old clients and new ones.

Our service will grow in features and functionality, and the semantics of some web services may change. For instance, a particular web service may have been released initially with just three parameters in its input, but a fourth parameter was felt necessary after two years of service. Some older clients will still access the web service with three parameters and newer clients will pass four parameters to it. _C'est la vie_. One of the most obvious examples of this is when mobile apps access a web service. Some phones will have older releases of the mobile app and others will have newer releases. This makes it necessary for the web service to serve all generations of clients without breaking any.

One way to handle this is by adding an "API version number" parameter to each web service. This will be a mandatory parameter from Day 1. Initial versions of clients will all be designed to use **`"ver": 1`** in their requests. When the software on the server-side application is upgraded and the semantics of a web service change, then it will support the old semantics under **`"ver" = 1`** and newer semantics with **`"ver" = 2`**. Newer clients which are aware of the updated semantics will use **`ver=2`** in their requests, with the new semantics. Older clients will not be aware of any changes.

The version number applies to each web service in isolation. The service as a whole may have versions 1 and 2 supported for some calls, 1, 2 and 3 for other calls, and just 1 for calls which have not changed at all.

This will go on, and new generations of web services will keep getting released. This will allow a single service to service multiple generations of clients at the same time.

## History tables

Transaction tables which receive updates will also have history tables which "shadow" the main tables. This will ensure that, taken in an overall database context, all updates are non-destructive, because all before-images of records can be reconstructed by analysing the history tables.

History tables will contain:

* previous images of updated records

* deleted records

The following policies will be applied for maintenance of history information:

* When a record is inserted in a main table, nothing is done to the history table. (This means that there is no history-related overhead of compute power or disk space for tables which are not modified much.)

* When a record is updated in a main table, its previous image is first inserted into the history table and then the data is updated in the main table

* When a record is physically deleted (_i.e._ using the SQL `DELETE` operation) from a main table, the to-be-deleted record is first copied into the history table

* When `truncate` operation is performed on the table, all the rows in the current table will be inserted to its history table

These are the implementation specifications for implementing the history feature:

* All main tables which are supposed to get the benefit of history tracking will now have corresponding history tables to "shadow" them. If the main table is called **`X`**, its shadow table will be called **`X_hist`**.

* The history maintenance feature will be implemented using database triggers.

* The shadow table will have identical schema to the main table, with two additional fields:

  * **`hist_when`**: timestamp specifying when this record was inserted in the shadow table

  * **`hist_op`**: will contain either `"U"`, `"D"` or `"T"`, indicating that this history replica is because of an u(pdate), d(eletion) or t(runcated)

* The server-side application code will connect to the database using a specific database username and password meant for business operation. (Let's say that this username is `appcode`.) This username will not have any rights on the shadow tables other than `INSERT` rights. A separate, second database user (let's say, `apphist`) will have `SELECT` rights on the shadow table, for reports. And only the administrator will be able to modify the contents of shadow tables.

## Timestamps

As per [JSON Schema specification for date-time](http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3.1), for timestamps values, we will always use the format defined in [RFC 3339](https://tools.ietf.org/html/rfc3339#section-5.8) _e.g._ **`1990-12-31T23:59:50Z`** for a global UTC timezone timestamp or **`2015-12-31T23:59:50+05:30`** for a timezone-included timestamp. All timestamp information in web service parameters will be communicated in these formats. If an application is used by clients across various timezones, all timestamps sent to or received from web services _must be_ in UTC, and must be stored on the servers in UTC. All conversion of those timestamps to local timezones to make it easier for humans to read, for instance, must be done in the client software.