UpdateBlock data structure for updating pipeline

[johnml1135]

After a set of USFM tokens are parsed and handled, we will want to apply some updates, namely, intra-verse insertion #578 and punctuation normalization #614. To do this, it would (likely) be cleanest to architect in this way:

• When a verse (or non-verse) set of tokens are processed, a data structure is created called a ScriptureBlock that represents the block as a overall reference, text, embeds, style markers and paragraph markers, with helper routines to iterate through various portions of the data.
• This structure is passed to a number of handlers:
  • A handler to updated text based upon pretranslations and "strip" or "preserve" of various portions
  • A handler to re-align intra-verse markers (paragraphs, embeds and style markers)
  • A handler to de-normalize punctuation
• This ScriptureBlock then can output the tokens that are ready to put on the stack of tokens

This allows multiple transformations to happen on the scripture text on a verse-by-verse bases using a unified data model. Any cross-verse data that needs to be utilized would be done so within the handler itself.

Scripture Block

The data structure could include:

• All types of data in a list with unified indexes:
  • Called "ScriptureBlockElements"
  • Types: Verse (or non-verse) text; Paragraph marker; Embed (totality of the embed); Style marker
  • Includes: Text (includes /ft for notes); Render(); a list of "indexes" that this element includes
• There is a copy of the "original" text elements before the transformations are made for use in later steps if the text ends up being transformed in the middle stages.

[johnml1135]

Ok here is a plan:

• Override CollectTokens when getting data (either in the "new tokens" which are added or incrementing the index, which is just walking through the initial USFM)
• Create two new types: UpdateElement and UpdateBlock
• UpdateElement has 3 parts: type [paragraph, inserted_text, text, Embed and style], whether it is included (bool) and list of tokens
  • Normally there is just one token per element, except for Embed which has many.
• UpdateBlock is made up of:
  • A scripture reference
  • An ordered list of UpdateElements
• There are 2 UpdateBlocks for the handler, one for paragraphs (verse/non-verse), one for embeds. These correspond to existing _newTokens and _newEmbedTokens.
• When CollectTokens is called, it is determined which update block to add the tokens to, or if the update block is "complete" and needs to be processed by:
  • Performing any updates on the "just completed" UpdateBlock (such as de-normalizing the quotes)
  • Adding tokens to _tokens if they are "included" as per each UpdateElement
  • Clearing the completed UpdateBlock

This should be enough information and be easy enough to process what is needed.

Implement in machine.py first.

[johnml1135]

enum UpdateElementType{
  paragraph,
  insertedText,
  existingText,
  embed,
  style
}

record UpdateElement{
  List<Token> tokens
  UpdateElementType type
  bool markedForRemoval
}
  
record UpdateBlock{
  List<UpdateElement> updateElements
  ScriptureRef scriptureRef
}

class IUpdateBlockHandler
  def InitializeHandler(string corpusId){
    ... grab needed corpus metadata ...
  }

  def ProcessBlock(UpdateBlock updateBlock){
    ... process ... update elements ... add remove elements ...
  }