Remote submission (hot folder) prototype.

README.md

@@ -140,6 +140,8 @@ the downside of static website: they are static.
 
 ### CLI
 
+#### Management interface
+
 Pocket Archive can be managed via a command line interface (CLI) when
 installed locally (e.g. via Luarocks).
 
@@ -181,6 +183,27 @@ inspecting.
 
 More detailed information can be obtain with `pkar --help`.
 
+#### Submission watchdog
+
+Obviously, the CLI only works if one has command line access (e.g. via SSH) to
+the machine hosting Pocket Archive. It is often the case that a contributor has
+neither shell access nor expertise to run the Pocket Archive CLI. Remote
+submissions provide a more user-friendly way to submit contents to any Pocket
+Archive instance on the WWW.
+
+Instead of a web application, Pocket Archive uses a "hot folder" method to
+support remote submissions. The `pkar_watch` utility is run on the machine that
+hosts Pocket Archive to watch a particular system folder for laundry lists.
+Any time a file named `pkar_submission*.csv` is added to that folder, it is
+processed for submission.
+
+In order for this approach to work, the laundry list **must** be uploaded after
+all the other submission files have been successfully uploaded.
+
+The watched folder is local to the Pocket Archive operating system. However, it
+is possible to run an FTP server or even an S3 server via e.g., MinIO on that
+folder, thus providing remote access and permission management.
+
 ### Environment variables
 
 The following environment variables should be set before using Pocket Archive:
@@ -207,10 +230,10 @@ components, even if not entirely developed or only usable in a specific
 development environment, to demonstrate the overall workflows and
 functionality.
 
-The second step is to produce a minimum viable product, which is fully
+The second step is to produce a basic application, which is fully
 functional and available for use by the intended audience.
 
-#### Prototype
+#### Prototype (alpha)
 
 ❏ = pending; ⚒ = in progress; ⎊ = blocked; ✓ = complete; ✖︎ = not implemented.
 
@@ -245,36 +268,45 @@ functional and available for use by the intended audience.
 - ✓ QA
   - ✓ ~50 resource data set
 
-#### MVP
-
-- Proper collection handling
-  - Dedicated template
-  - Link to markdown doc for presentation page
-  - Handle artifacts as members
-- Multilingual support
-- Markdown support
-- Management UI & API
-  - Deposit via S3 source pointer
-  - Deposit via single tar or zip file submission
-  - Dump & restore (whole archive & individual resources)
-- Content model
-  - Local overrides
-  - Relatioships inference
-- htmlgen option for local file or webserver URL generation
-- Improve search indexing
-- Category browsing
-- CLI
-  - Generate LL (multi)
-  - Generate RDF (multi)
-- Front end
-  - Enhanced styling and access
-- Testing
-- Documentation
-  - Break main sections off README
-  - Submission guide
-  - Site generation guide
-  - Content modeling guide (including proxy concepts)
-
-#### Post-MVP
-
+#### Basic application (beta -> v1.0)
+
+- ✖︎ Management UI & API
+  - ✖︎ Deposit via single tar or zip file submission
+- ⚒ Deposit via local hot folder
+- ❏ Proper collection handling
+  - ❏ Dedicated template
+  - ❏ Link to markdown doc for presentation page
+  - ❏ Handle artifacts as members
+- ❏ Content model
+  - ❏ Multilingual support
+  - ❏ Local overrides
+  - ❏ Relatioships inference
+  - ❏ Markdown support for individual fields
+- ❏ Generator
+  - ❏ htmlgen option for local file or webserver URL generation
+  - ❏ Generate LL (multi)
+  - ❏ Generate RDF (multi)
+- ❏ Front end
+  - ❏ Category browsing
+  - ❏ Improve search indexing
+  - ❏ Enhanced styling and access
+- ❏ CLI
+  - ❏ Dump & restore (whole archive & individual resources)
+- ❏ Testing
+    - ❏ Unit tests
+    - ❏ >100 resource data set
+- ❏ Documentation
+  - ❏ Break main sections off README
+  - ❏ Submission guide
+  - ❏ Site generation guide
+  - ❏ Content modeling guide (including proxy concepts)
+
+#### Post-basic
+
+- Deposit via remote hot folder
+  - FTP
+  - S3
 - Incremental build
+- Rebuild only site assets
+- Custom templating
+- Deposit via S3 source pointer

pocket_archive-scm-1.rockspec

@@ -26,9 +26,11 @@ dependencies = {
    "ftcsv",
    "lua-cjson",
    "lua-vips",
+   "luaposix",
    "penlight",
    "sllog",
    "volksdata",
+   "watchdog",
 }
 build = {
     type = "builtin",
@@ -45,6 +47,9 @@ build = {
             "ext/monocypher/lua_monocypher.c",
         },
     },
-    install = {bin = {pkar = "pkar.lua"}},
+    install = {bin = {
+        pkar = "src/util/pkar.lua",
+        pkar_watch = "src/util/watcher.lua",
+    }},
     copy_directories = {"config", "doc", "templates"},
 }

src/util/watcher.lua

@@ -0,0 +1,59 @@
+#!/usr/bin/lua
+
+local cli = require "cli"
+local plpath = require "pl.path"
+local signal = require "posix.signal"
+local watchdog = require "watchdog"
+
+local sub = require "pocket_archive.submission"
+
+
+local running = true
+
+signal.signal(signal.SIGINT, function()
+  print("\nShutting down...")
+  running = false
+end)
+
+cli.locale "en_US"  -- TODO set with multilingual support.
+
+cli.program {
+    [[Run deposit watcher on a directory.
+
+    Laundry list files dropped on the indicated directory will be processed
+    for deposit.
+
+    NOTE: LL files MUST be dropped AFTER all files in the deposit have been
+    successfully uploaded.
+    ]],
+
+    cli.positional "path" {"Local path of folder to watch."},
+
+    cli.flag "l,logfile" {
+        "Log file",
+        type = cli.string,
+        default="/dev/stderr",
+    },
+
+    function(args)
+        local wd = watchdog.init()
+        local logfile = assert(io.open(args.logfile, "a"))
+
+        wd:add(args.path, watchdog.IN_CREATE, function(ev)
+            logfile:write("Created new file:", ev.name, ev.mask)
+            logfile:write("\n")
+            if ev.name:find("pkar_submission.*%.csv") then
+                -- TODO spawn subprocess
+                sip = sub.generate_sip(plpath.join(args.path, ev.name))
+                sub.deposit(sip)
+            end
+        end)
+
+        while running do
+          wd:poll()
+        end
+
+        logfile:close()
+        wd:close()
+    end,
+}