Project

General

Profile

Task #17943

Implement batch processing: client library

Added by Sachin Divekar about 1 year ago. Updated 12 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Sachin Divekar
Target version:
Remiges Products - Sprint 29 Mar
Start date:
Due date:
15/03/2024
% Done:

100%

Estimated time:

Description

This includes the library functions to submit a batch, append more jobs to a batch, mark wait-off for a batch, abort a batch, the function to register an initialiser for an app and a processing function for a batch type, etc. It does not include the asynchronous framework which runs the processing functions.

Related issues

Copied to Alya - Task #18088: Implement job manager for slow queries and batchesClosed05/03/202415/03/2024Sachin Divekar

Actions
#1

Updated by Shuvam Misra about 1 year ago

  • Target version set to Sprint 15 Mar
  • Assignee set to Sachin Divekar
  • Due date set to 15/03/2024
#2

Updated by Sachin Divekar about 1 year ago

  • Status changed from New to In Progress
#3

Updated by Shuvam Misra about 1 year ago

  • Copied to Task #18088: Implement job manager for slow queries and batches added
#4

Updated by Shuvam Misra about 1 year ago

  • Description updated (diff)
#5

Updated by Shuvam Misra about 1 year ago

  • Description updated (diff)
  • Subject changed from Implement batch processing to Implement batch processing: client library
#6

Updated by Shuvam Misra about 1 year ago

  • % Done changed from 0 to 10
#7

Updated by Sachin Divekar about 1 year ago

  • % Done changed from 10 to 60

Implemented Submit, Done and Abort for batches and wrote a program to use them and verify their working.

#8

Updated by Sachin Divekar about 1 year ago

  • % Done changed from 60 to 80

Implement batch.Append()

#9

Updated by Sachin Divekar about 1 year ago

Implemented batch Abort and Waitoff.

#10

Updated by Sachin Divekar about 1 year ago

  • % Done changed from 80 to 90

Wrote examples for Submit, Abort and Append for testing.

#11

Updated by Shuvam Misra about 1 year ago

  • Target version changed from Sprint 15 Mar to Sprint 29 Mar
#12

Updated by Sachin Divekar about 1 year ago

Fix issue with concurrent nsuccess update with batchabort and summarize.

  • Run all the queries in BatchAbort in the same transaction
  • Calculate the counters inside BatchAbort itself after setting the status of the batchrows to Aborted
  • Update the summary in batches table inside the BatchAbort itself
#13

Updated by Shuvam Misra about 1 year ago

  • Target version changed from Sprint 29 Mar to Sprint 12 Apr
#14

Updated by Shuvam Misra about 1 year ago

  • Target version changed from Sprint 12 Apr to Sprint 29 Mar
#15

Updated by Shuvam Misra about 1 year ago

  • Status changed from In Progress to Testing
#16

Updated by Shuvam Misra about 1 year ago

  • % Done changed from 90 to 100
#17

Updated by Sachin Divekar about 1 year ago

Provided MigrateDatabase() function to migrate Alya batch related tables.

Fixed a problem where non-lowercase values for "op" parameter cause issues because we have lowercase check. The updated code converts the value of "op" passed to batch, slowquery and register processor functions to lowercase before using them.

#18

Updated by Sachin Divekar about 1 year ago

Wrote automated unit tests for initializers and processors registrations, batch summarization, temporary files creation and handling of blobrows.

#19

Updated by Sachin Divekar about 1 year ago

Worked on testing the package. Fixed bugs. Wrote unit tests.

  • Fixed issues in the output file where some lines were empty
  • Fixed an issue where BatchDone was returning incorrect values when the batch status is present in Redis
  • Fixed an issue in BatchDone where it was returning
  • Fixed a bug where batch status was not getting updated to inprog
  • Wrote an example to process a large batch.
#20

Updated by Sachin Divekar 12 months ago

  • Status changed from Testing to Closed

Also available in: Atom PDF