Task #17943
Implement batch processing: client library
Added by Sachin Divekar about 1 year ago. Updated about 1 year ago.
100%
Description
This includes the library functions to submit a batch, append more jobs to a batch, mark wait-off for a batch, abort a batch, the function to register an initialiser for an app and a processing function for a batch type, etc. It does not include the asynchronous framework which runs the processing functions.
Related issues
Updated by Shuvam Misra about 1 year ago
- Target version set to Sprint 15 Mar
- Assignee set to Sachin Divekar
- Due date set to 15/03/2024
Updated by Shuvam Misra about 1 year ago
- Copied to Task #18088: Implement job manager for slow queries and batches added
Updated by Shuvam Misra about 1 year ago
- Description updated (diff)
- Subject changed from Implement batch processing to Implement batch processing: client library
Updated by Sachin Divekar about 1 year ago
- % Done changed from 10 to 60
Implemented Submit, Done and Abort for batches and wrote a program to use them and verify their working.
Updated by Sachin Divekar about 1 year ago
- % Done changed from 80 to 90
Wrote examples for Submit, Abort and Append for testing.
Updated by Shuvam Misra about 1 year ago
- Target version changed from Sprint 15 Mar to Sprint 29 Mar
Updated by Sachin Divekar about 1 year ago
Fix issue with concurrent nsuccess update with batchabort and summarize.
- Run all the queries in BatchAbort in the same transaction
- Calculate the counters inside BatchAbort itself after setting the status of the batchrows to Aborted
- Update the summary in batches table inside the BatchAbort itself
Updated by Shuvam Misra about 1 year ago
- Target version changed from Sprint 29 Mar to Sprint 12 Apr
Updated by Shuvam Misra about 1 year ago
- Target version changed from Sprint 12 Apr to Sprint 29 Mar
Updated by Sachin Divekar about 1 year ago
Provided MigrateDatabase() function to migrate Alya batch related tables.
Fixed a problem where non-lowercase values for "op" parameter cause issues because we have lowercase check. The updated code converts the value of "op" passed to batch, slowquery and register processor functions to lowercase before using them.
Updated by Sachin Divekar about 1 year ago
Wrote automated unit tests for initializers and processors registrations, batch summarization, temporary files creation and handling of blobrows.
Updated by Sachin Divekar about 1 year ago
Worked on testing the package. Fixed bugs. Wrote unit tests.
- Fixed issues in the output file where some lines were empty
- Fixed an issue where BatchDone was returning incorrect values when the batch status is present in Redis
- Fixed an issue in BatchDone where it was returning
- Fixed a bug where batch status was not getting updated to inprog
- Wrote an example to process a large batch.