Wiki » History » Revision 14
Revision 13 (Shuvam Misra, 15/09/2023 04:05 PM) → Revision 14/22 (Shuvam Misra, 15/09/2023 04:05 PM)
# The Remiges product family We currently have six products on the anvil. This anvil, and this list will not grow till end of 2024. We give a short overview note about each here. Some common points about all the products: * All of them are server-side frameworks or systems which help the application developer develop some key functions very fast, securely, reliably and at scale. * All the products are scalable -- they can work with any application, irrespective of how fast the application processes transactions or how many transactions per second it handles * They will integrate with application code written in any programming language. Wherever client libraries are required, they will be provided initially in Go (the mother language of all the products other than IDshield), then on Java, then on PHP, dotNet, NodeJS, *etc*. * All of them are available as open source products from their respective GitHub repositories, and may be downloaded, modified, and re-used without restrictions. They are fully maintained and supported by dedicated teams within Remiges. * All of them are for application designers and developers. They have UI which may be used by non-technical users after they have been integrated into business applications, but they are not usable standalone without any integration. ## Remiges ServerSage: an observability framework This product will allow an application to log performance metrics and health metrics about all aspects of its operation into an observability framework. This framework will include screens to see the data in historical time-series format, correlate different events on a common time axis, and generate automatic alerts wherever readings fall outside acceptable watermarks. The framework is being built by extending **Prometheus**, which is a gold standard in observability solutions today. Prometheus provides infrastructure monitoring agents out of the box, so that you can track CPU load, RAM utilisation, etc, at the server level. ServerSage will extend this to track application health. Duration of each web service call, duration of each database command, durations of key parameters at the business operation level, will all be exported *via* custom-built agents to a central Prometheus data store. The data visualisation front-end for ServerSage is being built by extending **Grafana**, the industry-standard data dashboard tool. Source code for agents will be made available in Go and Java to allow embedding inside the business logic of your application, so that you can decide what to log in what manner, and then just plug in the right agents inside your code to push out that data to Prometheus. A two-tiered Prometheus setup will be available to allow data to be gathered into lower-level Prometheus servers, and then operate Tier 1 servers to pull in and hold the aggregate data in a cluster-wide data store for higher level reports and alerts. *The name ServerSage points to an entity which has wisdom and is aware about the servers and other components in its care.* ## Remiges LogHarbour: a logging framework This product will allow application developers to log three types of data in a unified cluster-wide data store from all types of applications: * application change log audit trail, giving the "before" and "after" values of all data items which are changing or getting deleted * activity and event logging, for information and forensics * debug logging, for developers to trace problems in the flow of code The data thus logged will be sent into Kafka streams and will finally reach Kafka consumers which will store the data in an ElasticSearch database. The product is built by utilising and integrating **Kafka**, the gold standard in distributed, very high performance logging frameworks, and **ElasticSearch**, the top full-text searchable document database. The proven performance and reliability of these systems will be available in LogHarbour. LogHarbour will also have connectors which can parse log files generated from legacy systems and ingest them into the central data store. ElasticSearch will operate in cluster mode with cross-replication, to allow large data stores of several billion entries and still serve queries at speed and scale. Every application development team re-invents the wheel when it decides on a logging module, and most of the time, this logging is done into the main transaction database, thus impacting overall scalability and throughput of the system. Integrating LogHarbour will allow applications to scale at will without worrying about whether the logging system can keep up or where the logs need to be stored. *The name LogHarbour points to the vision of logs of all types and for all applications protected in a safe store.* ## Remiges IDshield: an authentication and authorization framework Every business application needs to build, or plug in, some module to handle the user table and implement authentication and authorization. Remiges IDshield provides application developers with a ready-made service, together with its own database, to take care of these features. It will provide a login authentication interface which can be used by browser based applications or mobile apps, and will support stateless JWT tokens for session management. JWT tokens allow session tracking without any reference to a central session table in a database which invariably becomes a hotspot when the application usage scales. IDshield will also provide the framework for defining authorization rules, so that the application can check at each point whether the current user has the right to perform each operation. Remiges IDshield is built on top of the very well-known open source identity and authentication management system, **Keycloak**. Clients will also have the freedom to opt for **RedHat SSO**, which is a derivative product of Keycloak with commercial support from RedHat. IDshield extends Keycloak to add features which do not exist in the default distribution, like 2FA using SMS, geo-IP based access restrictions, *etc* It also provides a client library which allows application developers to make web service calls to IDshield to manage users, change access rights, *etc* The user table is stored in the relational database private to IDshield. Horizontal scalability is easily achieved by running multiple instances of the IDshield service connecting to a single data store. Accesses to this data store are infrequent, because IDshield uses a caching layer to optimise performance. With Remiges IDshield, application developers can focus more resources on actually building the business logic and less on developing the surrounding framework. In addition, IDshield offers a full-featured identity and authentication management framework which has been reviewed by public scrutiny of the source code and security testing. *The name IDshield builds upon the idea of "ID" referring to identity, and "shield" referring to security.* ## Remiges Rigel: a global configuration management system All well-designed applications need configuration parameters. At the simplest end, a configuration management module is a thin wrapper over a JSON or YAML file; the file holds the configuration values in the form of key-value pairs. Modern configuration management requirements demand much more. The configuration must not get accidentally deleted due to a file system corruption. The configuration must be available at each point of a distributed cluster of servers. The data must be version controlled for forensic purposes, and there must be access rules controlling who can make changes to the parameters. Remiges Rigel is a cluster-aware configuration management system which replicates configuration information and offers an access-controlled GUI to allow administrators to make changes to individual entries. Application code may query individual entries in the configuration database by calling functions in a client library. Changes need to be made only once, and replicate automatically to all nodes in the cluster, where they are cached. The system maintains a version number to detect updates to the data. Rigel logs into files or optionally into Remiges LogHarbour to keep an audit trail of changes. Access control for the configuration management interface may optionally be integrated with Remiges IDshield. Rigel is a simple system which has all the necessary features needed for a robust distributed configuration management system. Designers of large business applications must not rely on simple local config files for the most critical of system parameters, and Rigel offers a simple and easy alternative. *Rigel is a very bright super-giant star in the constellation Orion, and this imagery aligns well with the idea of global configuration being a bright component which other components refer to and are guided by.* ## Remiges Alya: a framework for writing web service calls in Go The Go programming language has already established itself as the worldwide and worthy successor to C for most systems programming projects, and a lot of the learning from forty years of C have been ploughed back into its design. For web service calls, Go has the [Gin](https://gin-gonic.com) framework which offers a lot of flexibility, but is not easy to use for large volume web service calls development and testing. A lot of common actions need to be implemented in template form to enhance programmer productivity and reduce errors. The Remiges Alya framework provides these reusable pieces. * a validation framework and template to validate each parameter in the call request, so that basic syntactic checking is always done consistently * an error response management package, which allows every web service call author to return an array of errors as per a prescribed structure. Each error gets an error code and an i18n-indexed error message in the language of choice as per the caller's preferences * a framework for managing SQL statements, so that all the boilerplate code and error checking best practices are incorporated automatically through code generated by [sqlc](https://sqlc.dev). This dramatically reduces errors and increases programmer productivity, and as a side-effect, results in the developer team maintaining a global list of SQL statements, which can be maintained independent of the business logic. * integration with Remiges LogHarbour for complete logging of audit trails, activity logs, and debug logs, with the feature to turn on or off debug logging for just one module of one application without restarting any services * integration with Remiges IDshield for authentication, authorization and user management * integration with Remiges Rigel for global configuration * various other utility functions When all these packages are aggregated, the designer can worry less about peripheral challenges and decisions and focus on the core business logic. The developer gets a trusted framework which takes care of all the housekeeping tasks unrelated to the business logic and can focus on adding functional features. *The name Alya means sky, loftiness and heaven in Arabic. It is also one star in a triple-star system in the constellation of Serpens.* ## Remiges Crux: a business rules engine (BRE) and workflow engine *The name Crux refers to a constellation in the southern sky which is centred around four bright stars in a cross-shaped asterism often referred to as The Southern Cross.*