scriptshifter/NOTES.md

Romanization service implementation notes

.cfg files

The .cfg format seems to follow a INI-like syntax that is ad-hoc-parsed by the ScriptShifter.

Unicode points are expressed as U+???? rather than \x???? of the standard INI syntax.

So far only top-level section names have been encountered.

Key-value pairs may express either a transliteration operation, e.g.

U+182CU+1820=qU+0307a

Or a passthrough (verbatim copy) operation, such as

At head of title=At head of title

Or a configuration directive, e.g.

SubfieldsAlwaysExcluded=uvxy0123456789

This last option may appear in any section, the first two only in the transliteration sections.

It is unclear how configuration directives can be distinguished from transliteration rules, except by naming all the possible verbatim copy options. A more readable and efficent format would have discrete subsections for configuration and transliteration—if necessary, with configuration and mapping subsections inside S2R and R2S sections.

Q: Is it possible to copy non-mapped characters verbatim in script to Roman? That would remove the need to explicitly add English phrases to the S2R section such as publisher not identified=publisher not identified.

Q: Shall spaces around the = sign be ignored?

A (RB): Very likely yes, spaces are represented in the legacy files by underscores.

Q: What are the _ at the end of some mappings, e.g. U+4E00=yi_ for Chinese? Are they supposed to add a space where the underscore appears?

A (RB): Yes.

ReRomanizeRecord.bas

Much of the code deals with MARC records. No need to concern about that since the new ScriptShifter is meant to convert text strings to text strings.

Q: Is it possible (and desirable) to determine the S2R/R2S direction from user prompt rather than guessing it from the text as the legacy software seems to be doing?

A (RB): Yes.

Q: The software seems to take multi-line directives in the configuration into account. Is it possible to avoid these for simplicity, or is there a need to express some mapping in multiple lines?

A (RB): Long lines may be needed. (SC) This would be moot in YAML that supports multi-line strings via folding.

Detailed breakdown of individual functions follows.

Functions

ReRomanizeText

• Determine direction (input param): R2S or S2R
• Determine personal name handling
• Whether to uppercase first word

LoadOneRomanizationTable

Load cfg file (line by line, we can do the whole thing) and parse table metadata.

Skip lines starting with # (comments).

Lines starting with [ are section defiinitions. The supported sections are General, ScriptToRoman, and RomanToScript.

General Section

Potentially relevant variables:

• Name
• NoRomanization
• AllowCaseVariation
• ApostropheCharacters
• AllowCaseVariation
• ApostropheCharacters
• BySyllables
• Truncation

Likely irrelevant:

• FontName
• DoNotUse880Field
• AllowDefineButton (what is this? Just UI related?)

RomanToScript section

If there is no = sign, it is assumed to be a multi-line directive, and the next line should be loaded and merged with the previous content.

Otherwise, a keyword indicating a configuration directive is looked up.

Currently supported, and potentially relevant, keywords are:

• FieldsIncluded
• IncludeFormattingCharactersLcPattern
• OtherSubfieldsExcludedByTag
• VowelMarker

Likely irrelevant keywords:

• CreateEmpty880s
• Subfield6Code
• SubfieldsAlwaysExcluded

If no keyword is detected, proceed to transliteration.

ScriptToRoman section

The logic is the same as the RomanToscript section, but the configuration keywords are different.

Currently supported, potentially relevant:

• UppercaseFirstCharacterInSubfield [TODO verify]
• PersonalNameHandling

Likely irrelevant (MARC-related):

• CreateEmptyFields
• FieldsIncluded
• SubfieldsAlwaysExcluded
• OtherSubfieldsExcludedByTag

There is some code (deactivated) to dump the whole table.

RomanizeConvertText

This gets called at several points by ReRomanizeText.

It accepts in, supposedly the input text, a UTF8MarcRecordClass (probably the transliteration table) and a UTF8CharClass (?).

It replaces "_" with " " within the transliteration table (so values with _ are adding spaces to the translitrated text.

It loops through the text and detects the following leaders:

There is a LocalMarcRecordObject.SafeStuff() function that seems critical but I can't seem to find a definition for it.

There is a select switch within a loop whose function is not entirely clear. Is it to advance through sections of the text by MARC record internal markers?

Select Case sLeader$
    Case "&H"
        sLeader$ = "U+"
    Case "U+"
        sLeader$ = "&x"
    Case "&x"
        sLeader$ = "&X"
    Case "&X"
        sLeader$ = "&h"
    Case "&h"
        Exit Do
End Select

LoadRomanizationTables

This function seems to load all the tables by calling LoadOneRomanizationTable over a list of files.

ReRomanizeTextDetails

This is the logic of the romanization process by character or syllable.

• Proceed by syllable or by character based on config (note: currently there doesn't seem to be any cfg file containing the BySyllables option)
• Decide to allow case variation based on config
• Proceed scanning the text and looping; look for MARC delimiters
  • Define initial only, terminal only, medial only characters
  • Decide whether to translate R2V or V2R

EvaluateFirstCharacter

This determines if the translation is R2S or S2R. Does this work reliably and independently of any external directive? Could there be some strings in foreign scripts that start with Latin characters (e.g. numbers or Western terms), and lead to unexpected results?

A (RB): Some non-Latin scripts may start with Latin characters. (SC) Let's use an explicit direction option from the user. The logic would be too complicated and flimsy.

(Also the translation is supposedly purpose-driven, as the user should have a specific direction in mind and wouldn't want the software to decide for them.)

ReRomanizeTextDetailsReplaceApostrophes

Replace apostrophe characters with glyphs supported by foeign script?

Coment (RB): More clarity is needed around Latin and non-Latin punctuation to be used. (SC): TODO More to be discussed via conf call.

Field- and UI-related functions

• RomanizationAssistance
• FindFieldCurrentlyPointedTo
• RomanizationAssistanceConvertWholeRecord
• ReRomanizeAdjustNonfilingIndicators
• AddCharSetCodes2Utf8Record
• FindScriptByKeyPress
• IsFontInstalled

These functions seem to deal with interface interaction, field/text selection, and RTF clenaup. Probably very little to nothing needs to be carried over.

Question: do we need to keep any formatting of the original text? And if yes, which formatting tags are allowed? (in an HTML UI the formatting could be in HTML or Markdown, so this may need to be taken into account.)

Q: It seems like several characters are parsed and added to the text to denote MARC markers. Do we need to deal with these manually as indicators related to the script/language handled, or shall we expect any text string input in the new ScriptShifter to be clean from MARC flags?

A (RB): Most MARC markers are obsolete; but there may be other markers that are not easily transliterated, e.g. BIBFRAME markers. More discussion is needed on this point. (SC) Need feedback from KF + MM about what to expect from input and output string in this regard.

RomanizeConvertDecimalChars

Convert escape sequences &#\d{4,5} to code points.

CreateRomanizationScriptList

Create list of romanization script options by reading a master file. This will likely be replaced by a glob-like approach.

General strategy for rewrite

API endpoints

List available transliterations

Invocation

GET /tables

Returns all available transliterations, and which directions are available for each.

Response

200 OK; body: K/V pairs of: script name, list of r2s (Roman to script), s2r (Script to Roman), or both.

Transliterate a string

Invocation

POST /trans/<script>/<direction>

Query parameters

• <script>: script name as obtained by the /tables endpoint.
• <direction>: transliteration direction as obtained by the /tables endpoint.

POST body

• data: Input text (UTF-8) to transliterate.

Response

• 200 OK if transliteration was successful; response body: transliterated string (UTF-8)
• 400 Bad Request if a script name is not available in the requested direction; response body: details of failure.
• 500 Server Error if an internal error occurred; response body: generic error message (no details about the error)

Reload the translation tables

Reload the tables if they have been modified. This is done internally at server start. This should be auth-protected.

Invocation

POST /reload_config

Authentication

API token (probably just a hard-coded value in a .env file should suffice)

Response

• 204 No content if the tables were reloaded successfully; no response body.
• 500 Server Error on internal error.

Functional approach

1. Upon server startup: load all translation table metadata. This is equivalent to invoking reload_config via REST API (see above) and is done by scanning a designated directory containing only the translation table, finding the metadata in the General section, disccovering the ScriptToRoman and RomanToScript sections, and storing these metadata in a variable available to all requests.

2. Upon invocation of the trans method: load the relevant configuration file (this operation will be cached in order to save expensive parsing) and apply the relevant ScriptToRoman or RomanToScript transliteration to the provided string.

3. Upon invocation of the /reload_config method: reload the table metadata as on startup; invalidate the cache for all the configurations.