pocket_archive

Pocket Archive

The idea

Stick it in your pocket and carry it around. Install it on a cloud server. Install it on a Raspberry Pi. Browse it offline. Browse it online. Duplicate it, share it, harvest it and aggregate it. Feed it non-GMO spreadsheets regularly and it will thrive.

A more sensical description

Pocket Archive is a digital archival system and static site generator for small- to medium-(?) sized archives. It is designed to function in environments with unreliable connectivity and requires very low technical and human resources to set up, run, and use.

Pocket Archive fulfills the following functions:

• Storage and management of files and metadata-only resources
• Management of descriptive, administrative, and technical metadata
• Dynamic relationships between resources
• Static site generation (discovery interface)

In spite of its design simplicity, Pocket Archive strives to be highly flexible. It is based on Volksdata , a very compact Linked Data store written in C. There is no restriction to the types and schema of metadata allowed, or the file types supported. A file-based configuration allows to set up content types and validation rules, or to have (almost) no rules at all.

Why

Several years ago, the author of this project believed that he should work in larger and larger institutions, with larger and larger data sets. One day, he came across a project that changed his perspective.

"From a standpoint of preserving human cultural heritage at large, does it make more sense to design very large repositories for very rich institutions, with a lot of layers of safety but also a lot of bureaucracy and redundancy, or rather contribute to many decentralized projects that are highly efficient, small, representing periferal cultures, and most importantly, that are at much higher risk of loss than large institutions'"?

Both: this software has been conceived with the experience of large-scale repositories as the background to decide what works and what doesn't, what is necessary and what is superfluous, and what catalogers and archivists need to do their job.

It is not inconceivable that if many Pocket Archives were to sprout all over the place one day, they could be periodically harvested, linked together, and presented in one large, central archive (it's Linked Data, after all), without any detriment to the indepencence of the individual archives.

Basic concepts

Until some proper reference is written, this should serve as a high-level documentation to help evaluate the functionality and to help me to stay on track. Many of these ideas have been ripped right off my day job, so there is a good chance they work.

General philosophy

The functional goals of Pocket Archive are simplicity and flexibility, from both a user's and a maintainer's perspectives. These two properties are usually seen as conflicting, but within reason, they can coexist.

Pocket Archive is built upon a minimalistic framework: C and Lua, with very few dependencies. Similarly to these foundational elements, Pocket Archive strives to offer few tools that can be combined in a multitude of ways to achieve many goals, rather than many tools each doing a specific thing.

Resource

The Linked Data adage goes, "everything is a Resource". Without confusing users too much by taking the concept to the Linked Data extremes, the term resource is used in this project to describe individual, self-contained units of information such as:

• Digital files;
• Intellectual or physical artifacts (artworks, documents, books, etc.);
• Structural elements inside or around an entity, such as the order of pages in a book, the two sides of a postcard, a collection of oher resources, etc.

Files are called opaque resources. They are viewed by Pocket Archive as "opaque" in that the system doesn't care about their contents. It only ensures that files are stored as they were submitted, and keeps checksums to guard against data corruption.

All other entities are called descriptive resources. These are effectively Linked Data, which can be queried and searched for. Each file also has its own descriptive resource, so that it can be classified, discovered, and described.

Submission

A Pocket Archive repository is populated via submissions. A submission is performed by telling the archive to pick up some files from a folder it can access, push them into storage, add metadata to them, and index them so that they can be found later.

A submission is directed by a laundry list, which is a spreadsheet listing all the resources (both opaque and descriptive) to be created, and the metadata assigned to them. The laundry list, formatted as a CSV (comma-separated value) file, can be edited by several free and open source applications, such as LibreOffice. For repetitive, high-volume submissions, templates can be set to facilitate filling in metadata fields. An example submission , which includes a laundry list, is available.

Using spreadsheets is for most users much faster and intuitive than clicking around an alien user interface filled with icons and terms that one has never seen before.

Detailed instructions on how to write a laundry list are under the submission documentation.

Metadata & content model

Metadata are (yes, it's a plural noun) controlled by a content model, which in this project is intended as the entirety of definitions of content types recognized by the system, and how they relate to one another. Each individual Pocket Archive installation can use the baseline one provided by default, or extend it via additional configuration.

See the content model configuration manual for details on how to set up a custom content model.

Site generation

Pocket Archive can generate HTML pages and all the related assets to run a complete static website. The advantages of a static website over a dynamic one are that it's much simpler and economical to set up and run, and it's impervious to malicious attacks.

The entire site must be generated every time resources are created or updated. This is usually very fast, but on large archives it can take a while. This is the downside of static website: they are static.

Status

ALPHA. Pocket Archive is a very recent project, in fast development. Its foundational library, Volksdata, has been developed as a spare-time project for 6 years and it just entered in beta status.

Road map

Simple road map for a rough prototype:

• ⚒ Configuration + config parser
  • ✓ Application
  • ⚒ Content model
  • ⎊ Validation rules
  • ⎊ Relationship inference rules
  • ⎊ Local overrides
• ⚒ Submission module
  • ✓ SIP building
  • ✓ Metadata from LL
  • ✓ Structure inference
  • Relatioships inference
• ⚒ HTML generator
  • ✓ Index
  • ✓ Resource
  • ✓ Static assets
  • ⚒ Transformers
• ⎊ Non-HTML generators
  • LL
  • RDF (turtle)
• ⎊ Front end
  • JS search engine