scriptshifter/NOTES.md

Romanization service implementation notes

.cfg files

The .cfg format seems to follow a INI-like syntax that is ad-hoc-parsed by the Transliterator.

So far only top-level section names have been encountered.

Key-value pairs may express either a transliteration operation, e.g.

U+182CU+1820=qU+0307a

Or a passthrough (verbatim copy) operation, such as

At head of title=At head of title

Or a configuration directive, e.g.

SubfieldsAlwaysExcluded=uvxy0123456789

This last option may appear in any section, the first two only in the transliteration sections.

It is unclear how configuration directives can be distinguished from transliteration rules, except by naming all the possible verbatim copy options. A more readable and efficent format would have discrete subsections for configuration and transliteration; if possible, vebatim copy should be implicit, which would make maintenance easier.

It is unclear at the moment if spaces around the = sign are ignored.

ReRomanizeRecord.bas

Much of the code deals with MARC records. No need to concern about that since the new Transliterator is meant to convert text strings to text strings.

Functions

ReRomanizeText

• Determine direction (input param): R2S or S2R
• Determine personal name handling
• Whether to uppercase first word

LoadOneRomanizationTable

Load cfg file (line by line, we can do the whole thing) and parse table metadata.

Skip lines starting with # (comments).

Lines starting with [ are section defiinitions. The supported sections are General, ScriptToRoman, and RomanToScript.

General Section

Potentially relevant variables:

• Name
• NoRomanization
• AllowCaseVariation
• ApostropheCharacters
• AllowCaseVariation
• ApostropheCharacters
• BySyllables
• Truncation

Likely irrelevant:

• FontName
• DoNotUse880Field
• AllowDefineButton (what is this? Just UI related?)

RomanToScript section

If there is no = sign, it is assumed to be a multi-line directive, and the next line should be loaded and merged with the previous content.

Otherwise, a keyword indicating a configuration directive is looked up.

Currently supported, and potentially relevant, keywords are:

• FieldsIncluded
• IncludeFormattingCharactersLcPattern
• OtherSubfieldsExcludedByTag
• VowelMarker

Likely irrelevant keywords:

• CreateEmpty880s
• Subfield6Code
• SubfieldsAlwaysExcluded

If no keyword is detected, proceed to transliteration. [TODO transliteration logic details still to be looked at]

ScriptToRoman section

The logic is the same as the RomanToscript section, but the configuration keyword are different.

Currently supported, potentially relevant:

• UppercaseFirstCharacterInSubfield [TODO verify]
• PersonalNameHandling

Likely irrelevant:

• CreateEmptyFields
• FieldsIncluded
• SubfieldsAlwaysExcluded
• OtherSubfieldsExcludedByTag

[TODO Complete other function analysis]

General strategy for rewrite

API endpoints

List available transliterations

Invocation

GET /tables

Returns all available transliterations, and which directions are available for each.

Response

200 OK; body: K/V pairs of: script name, list of r2s (Roman to script), s2r (Script to Roman), or both.

Transliterate a string

Invocation

POST /trans/<script>/<direction>

Query parameters

• <script>: script name as obtained by the /tables endpoint.
• <direction>: transliteration direction as obtained by the /tables endpoint.

POST body

• data: Input text (UTF-8) to transliterate.

Response

• 200 OK if transliteration was successful; response body: transliterated string (UTF-8)
• 400 Bad Request if a script name is not available in the requested direction; response body: details of failure.
• 500 Server Error if an internal error occurred; response body: generic error message (no details about the error)

Reload the translation tables

Reload the tables if they have been modified. This is done internally at server start. This should be auth-protected.

Invocation

POST /reload_config

Authantication

API token (probably just a hard-coded value in a .env file should suffice)

Response

• 204 No content if the tables were reloaded successfully; no response body.
• 500 Server Error on internal error.

Functional approach

1. Load all translation table metadata on server startup. This is equivalent to invoking reload_config via REST API (see above) and is done by scanning a designated directory containing only the translation table, finding the metadata in the General section, disccovering the ScriptToRoman and RomanToScript sections, and storing these metadata in a variable available to all requests.

2. Upon invocation of the trans method: load the relevant configuration file (this operation will be cached in order to save expensive parsing) and apply the relevant ScriptToRoman or RomanToScript transliteration to the provided string.

3. Upon invocation of the /reload_config method: reload the table metadata as on startup; invalidate the cache for all the configurations.