Further major setup.

* Add indexing strategy and setup docs
* Move most text out of README
* RTD theme
* Other amendments here and there

README.rst

@@ -6,102 +6,18 @@ LAKEsuperior
 LAKEsuperior is an alternative `Fedora
 Repository <http://fedorarepository.org>`__ implementation.
 
-Guiding Principles
-------------------
-
-LAKEsuperior aims at being an uncomplicated, efficient Fedora 4
-implementation.
-
-Its main goals are:
+Documentation
+-------------
 
--  **Reliability:** Based on solid technologies with stability in mind.
--  **Efficiency:** Small memory and CPU footprint, high scalability.
--  **Ease of management:** Tools to perform monitoring and maintenance
-   included.
--  **Simplicity of design:** Straight-forward architecture, robustness
-   over features.
+The full documentation is maintained in `Read The Docs
+<http://lakesuperior.readthedocs.io/>`__. Please refer to that for more info.
 
-Key features
+Installation
 ------------
 
--  Drop-in replacement for Fedora4 (with some
-   `caveats <docs/fcrepo4_deltas.md>`__); currently being tested
-   with Hyrax 2
--  Very stable persistence layer based on
-   `LMDB <https://symas.com/lmdb/>`__ and filesystem. Fully
-   ACID-compliant writes guarantee consistency of data.
--  Term-based search (*planned*) and SPARQL Query API + UI
--  No performance penalty for storing many resources under the same
-   container; no
-   `kudzu <https://www.nature.org/ourinitiatives/urgentissues/land-conservation/forests/kudzu.xml>`__
-   pairtree segmentation \ `1 <#f1>`__\ 
--  Extensible `provenance metadata <docs/model.md>`__ tracking
--  `Multi-modal
-   access <docs/architecture.md#multi-modal-access>`__: HTTP
-   (REST), command line interface and native Python API.
--  Fits in a pocket: you can carry 50M triples in an 8Gb memory stick.
-
-Implementation of the official `Fedora API
-specs <https://fedora.info/spec/>`__ (Fedora 5.x and beyond) is not
-foreseen in the short term, however it would be a natural evolution of
-this project if it gains support.
-
-Please make sure you read the `Delta
-document <docs/fcrepo4_deltas.md>`__ for divergences with the
-official Fedora4 implementation.
-
-Target Audience
----------------
-
-LAKEsuperior is for anybody who cares about preserving data in the long
-term.
-
-Less vaguely, LAKEsuperior is targeted at who needs to store large
-quantities of highly linked metadata and documents.
-
-Its Python/C environment and API make it particularly well suited for
-academic and scientific environments who would be able to embed it in a
-Python application as a library or extend it via plug-ins.
-
-LAKEsuperior is able to be exposed to the Web as a `Linked Data
-Platform <https://www.w3.org/TR/ldp-primer/>`__ server. It also acts as
-a SPARQL query (read-only) endpoint, however it is not meant to be used
-as a full-fledged triplestore at the moment.
-
-In its current status, LAKEsuperior is aimed at developers and hands-on
-managers who are interested in evaluating this project.
-
-Quick Install: Running in Docker
---------------------------------
-
-You can run LAKEsuperior in Docker for a hands-off quickstart.
-
-`Docker <http://docker.com/>`__ is a containerization platform that
-allows you to run services in lightweight virtual machine environments
-without having to worry about installing all of the prerequisites on
-your host machine.
-
-1. Install the correct `Docker Community
-   Edition <https://www.docker.com/community-edition>`__ for your
-   operating system.
-2. Clone this repo:
-   ``git clone https://github.com/scossu/lakesuperior.git``
-3. ``cd`` into repo folder
-4. Run ``docker-compose up``
-
-LAKEsuperior should now be available at ``http://localhost:8000/``.
-
-The provided Docker configuration includes persistent storage as a
-self-container Docker volume, meaning your data will persist between
-runs. If you want to clear the decks, simply run
-``docker-compose down -v``.
-
-Manual Install (a bit less quick, a bit more power)
----------------------------------------------------
-
-**Note:** These instructions have been tested on Linux. They may work on
-Darwin with little modification, and possibly on Windows with some
-modifications. Feedback is welcome.
+The following instructions are aimed at a manual install using this git
+repository. For a hands-off install using Docker, see
+:doc:`the setup documentation <setup>`.
 
 Dependencies
 ~~~~~~~~~~~~
@@ -129,43 +45,6 @@ Installation steps
    stores
 8. Run ``./fcrepo``.
 
-Configuration
-~~~~~~~~~~~~~
-
-The app should run for testing and evaluation purposes without any
-further configuration. All the application data are stored by default in
-the ``data`` directory.
-
-To change the default configuration you should:
-
-1. Copy the ``etc.skeleton`` folder to a separate location
-2. Set the configuration folder location in the environment:
-   ``export FCREPO_CONFIG_DIR=<your config dir location>`` (you can add
-   this line at the end of your virtualenv ``activate`` script)
-3. Configure the application
-4. Bootstrap the app or copy the original data folders to the new
-   location if any loction options changed
-5. (Re)start the server: ``./fcrepo``
-
-The configuration options are documented in the files.
-
-**Note:** ``test.yml`` must specify a different location for the graph
-and for the binary stores than the default one, otherwise running a test
-suite will destroy your main data store. The application will issue an
-error message and refuse to start if these locations overlap.
-
-Production deployment
-~~~~~~~~~~~~~~~~~~~~~
-
-If you like fried repositories for lunch, deploy before 11AM.
-
-Status and development
-----------------------
-
-LAKEsuperior is in **alpha** status. Please see the `project
-issues <https://github.com/scossu/lakesuperior/issues>`__ list for a
-rudimentary road map.
-
 Contributing
 ------------
 
@@ -178,17 +57,8 @@ Contributions are welcome in all forms, including ideas, issue reports,
 or even just spinning up the software and providing some feedback.
 LAKEsuperior is meant to live as a community project.
 
-Documentation
------------------------
-
-The documenation is maintained in `Read The Docs
-<http://lakesuperior.readthedocs.io/en/latest/>`__.
-
---------------
-
-1 However if your client splits pairtrees upstream, such as Hyrax does,
-that obviously needs to change to get rid of the path segments.
-`↩ <#a1>`__
+See :doc:`related document <contributing>` for further details onhow to fork,
+improve, document and test the project.
 
 .. |build status| image:: http://img.shields.io/travis/scossu/lakesuperior/master.svg?style=flat
    :target: https://travis-ci.org/username/repo

docs/architecture.rst

@@ -39,10 +39,10 @@ jobs for example.
 
 The Python API is divided in three main areas:
 
--  `Resource API <../../lakesuperior/api/resource.py>`__. This API is in
-   charge of all the resource CRUD operations and implements the
-   majority of the Fedora specs.
--  `Admin API <../../lakesuperior/api/admin.py>`__. This exposes utility
-   methods, mostly long-running maintenance jobs.
--  `Query API <../../lakesuperior/api/query.py>`__. This provides
-   several facilities for querying repository data.
+-  Resource API: this API in charge of all the resource CRUD operations and
+   implements the majority of the Fedora specs.
+-  Admin API: exposes utility methods, mostly long-running maintenance jobs.
+-  Query API: provides several facilities for querying repository data.
+
+
+See :doc:`API documentation<api>` for more details.

docs/conf.py

@@ -53,7 +53,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'lakesuperior'
-copyright = '2018, Stefano Cossu'
+copyright = '2018, Everybody & Nobody'
 author = 'Stefano Cossu'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -61,7 +61,7 @@ author = 'Stefano Cossu'
 # built documents.
 #
 # The short X.Y version.
-version = '1.0.alpha'
+version = '1.0-alpha'
 # The full version, including alpha/beta/rc tags.
 release = '1.0.0-alpha.8'
 
@@ -89,7 +89,7 @@ todo_include_todos = True
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

docs/fcrepo4_deltas.rst

@@ -123,9 +123,9 @@ treated as a fully qualified identifier. The ``fcrepo:hasVersionLabel``
 predicate, however ambiguous in this context, will be kept until the
 adoption of Memento, which will change the retrieval mechanisms.
 
-Also, if a POST is issued on the same resource ``fcr:versions`` location
-using a version ID that already exists, LAKEsuperior will just mint a
-random identifier rather than returning an error.
+Another notable difference is that if a POST is issued on the same resource
+``fcr:versions`` location using a version ID that already exists, LAKEsuperior
+will just mint a random identifier rather than returning an error.
 
 Deprecation track
 -----------------
@@ -144,9 +144,6 @@ This should not pose a problem if a client does not have ``rest``
 hard-coded in its code, but in any event, the ``/rest`` endpoint is
 provided for backwards compatibility.
 
-LAKEsuperior adds the (currently stub) ``query`` endpoint. Other
-endpoints for non-LDP services may be opened in the future.
-
 Automatic LDP class assignment
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/index.rst

@@ -1,35 +1,133 @@
-.. lakesuperior documentation master file, created by
-   sphinx-quickstart on Sat Mar 24 23:05:46 2018.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
 LAKEsuperior
-========================================
+============
+
+|build status|
+
+LAKEsuperior is an alternative `Fedora
+Repository <http://fedorarepository.org>`__ implementation.
+
+Fedora is a mature repository software system historically adopted by
+major cultural heritage institutions. It exposes an
+`LDP <https://www.w3.org/TR/ldp-primer/>`__ endpoint to manage
+any type of binary files and their metadata in Linked Data format.
+
+Guiding Principles
+------------------
+
+LAKEsuperior aims at being an uncomplicated, efficient Fedora 4
+implementation.
+
+Its main goals are:
+
+-  **Reliability:** Based on solid technologies with stability in mind.
+-  **Efficiency:** Small memory and CPU footprint, high scalability.
+-  **Ease of management:** Tools to perform monitoring and maintenance
+   included.
+-  **Simplicity of design:** Straight-forward architecture, robustness
+   over features.
+
+Key features
+------------
+
+-  Drop-in replacement for Fedora4 (with some
+   :doc:`caveats <fcrepo4_deltas>`); currently being tested
+   with Hyrax 2
+-  Very stable persistence layer based on
+   `LMDB <https://symas.com/lmdb/>`__ and filesystem. Fully
+   ACID-compliant writes guarantee consistency of data.
+-  Term-based search (*planned*) and SPARQL Query API + UI
+-  No performance penalty for storing many resources under the same
+   container; no
+   `kudzu <https://www.nature.org/ourinitiatives/urgentissues/land-conservation/forests/kudzu.xml>`__
+   pairtree segmentation \ `1 <#f1>`__\ 
+-  Extensible :doc:`provenance metadata <model>` tracking
+-  :doc:`Multi-modal access <architecture>`: HTTP
+   (REST), command line interface and native Python API.
+-  Fits in a pocket: you can carry 50M triples in an 8Gb memory stick.
+
+Implementation of the official `Fedora API
+specs <https://fedora.info/spec/>`__ (Fedora 5.x and beyond) is not
+foreseen in the short term, however it would be a natural evolution of
+this project if it gains support.
+
+Please make sure you read the :doc:`Delta
+document <fcrepo4_deltas>` for divergences with the
+official Fedora4 implementation.
+
+Target Audience
+---------------
+
+LAKEsuperior is for anybody who cares about preserving data in the long
+term.
+
+Less vaguely, LAKEsuperior is targeted at who needs to store large
+quantities of highly linked metadata and documents.
+
+Its Python/C environment and API make it particularly well suited for
+academic and scientific environments who would be able to embed it in a
+Python application as a library or extend it via plug-ins.
+
+LAKEsuperior is able to be exposed to the Web as a `Linked Data
+Platform <https://www.w3.org/TR/ldp-primer/>`__ server. It also acts as
+a SPARQL query (read-only) endpoint, however it is not meant to be used
+as a full-fledged triplestore at the moment.
+
+In its current status, LAKEsuperior is aimed at developers and hands-on
+managers who are interested in evaluating this project.
+
+Status and development
+----------------------
+
+LAKEsuperior is in **alpha** status. Please see the `project
+issues <https://github.com/scossu/lakesuperior/issues>`__ list for a
+rudimentary road map.
+
+Contributing
+------------
+
+This has been so far a single person’s off-hours project (with much
+input from several sides). In order to turn into anything close to a
+Beta release and eventually to a production-ready implementation, it
+needs some community love.
+
+Contributions are welcome in all forms, including ideas, issue reports,
+or even just spinning up the software and providing some feedback.
+LAKEsuperior is meant to live as a community project.
+
+--------------
+
+1 However if your client splits pairtrees upstream, such as Hyrax does,
+that obviously needs to change to get rid of the path segments.
+`↩ <#a1>`__
+
+.. |build status| image:: http://img.shields.io/travis/scossu/lakesuperior/master.svg?style=flat
+   :target: https://travis-ci.org/username/repo
+
+Indices and tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Contents
 
+    Installation and Configuration <setup>
     Architecture Overview <architecture>
     Divergences from Fedora 4 <fcrepo4_deltas>
-    Content Model <model>
-    Messaging SPI <messaging>
+    Messaging <messaging>
     Migration Guide <migration>
     Command Line Reference <cli>
-    Storage Implementation <storage>
     Performance Benchmarks <performance>
-    API documentation <api>
 
 .. toctree::
-   :maxdepth: 3
-   :caption: Technical notes:
-
-    notes/indexing_strategy
-
+   :maxdepth: 1
+   :caption: In-depth tech & design
 
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+    Contributing <contributing>
+    API documentation <api>
+    Indexing Strategy <indexing_strategy>
+    Storage Implementation <storage>
+    Content Model <model>

docs/setup.rst

@@ -0,0 +1,90 @@
+Installation & Configuration
+============================
+
+Quick Install: Running in Docker
+--------------------------------
+
+You can run LAKEsuperior in Docker for a hands-off quickstart.
+
+`Docker <http://docker.com/>`__ is a containerization platform that
+allows you to run services in lightweight virtual machine environments
+without having to worry about installing all of the prerequisites on
+your host machine.
+
+1. Install the correct `Docker Community
+   Edition <https://www.docker.com/community-edition>`__ for your
+   operating system.
+2. Clone the LAKEsuperior git repository:
+   ``git clone https://github.com/scossu/lakesuperior.git``
+3. ``cd`` into repo folder
+4. Run ``docker-compose up``
+
+LAKEsuperior should now be available at ``http://localhost:8000/``.
+
+The provided Docker configuration includes persistent storage as a
+self-container Docker volume, meaning your data will persist between
+runs. If you want to clear the decks, simply run
+``docker-compose down -v``.
+
+Manual Install (a bit less quick, a bit more power)
+---------------------------------------------------
+
+**Note:** These instructions have been tested on Linux. They may work on
+Darwin with little modification, and possibly on Windows with some
+modifications. Feedback is welcome.
+
+Dependencies
+~~~~~~~~~~~~
+
+1. Python 3.5 or greater.
+2. A message broker supporting the STOMP protocol. For testing and
+   evaluation purposes, `CoilMQ <https://github.com/hozn/coilmq>`__ is
+   included with the dependencies and should be automatically installed.
+
+Installation steps
+~~~~~~~~~~~~~~~~~~
+
+1. Create a virtualenv in a project folder:
+   ``virtualenv -p <python 3.5+ exec path> <virtualenv folder>``
+2. Activate the virtualenv: ``source <path_to_virtualenv>/bin/activate``
+3. Clone this repo:
+   ``git clone https://github.com/scossu/lakesuperior.git``
+4. ``cd`` into repo folder
+5. Install dependencies: ``pip install -r requirements.txt``
+6. Start your STOMP broker, e.g.: ``coilmq &``. If you have another
+   queue manager listening to port 61613 you can either configure a
+   different port on the application configuration, or use the existing
+   message queue.
+7. Run ``./lsup-admin bootstrap`` to initialize the binary and graph
+   stores
+8. Run ``./fcrepo``.
+
+Configuration
+-------------
+
+The app should run for testing and evaluation purposes without any
+further configuration. All the application data are stored by default in
+the ``data`` directory.
+
+To change the default configuration you should:
+
+1. Copy the ``etc.skeleton`` folder to a separate location
+2. Set the configuration folder location in the environment:
+   ``export FCREPO_CONFIG_DIR=<your config dir location>`` (you can add
+   this line at the end of your virtualenv ``activate`` script)
+3. Configure the application
+4. Bootstrap the app or copy the original data folders to the new
+   location if any loction options changed
+5. (Re)start the server: ``./fcrepo``
+
+The configuration options are documented in the files.
+
+**Note:** ``test.yml`` must specify a different location for the graph
+and for the binary stores than the default one, otherwise running a test
+suite will destroy your main data store. The application will issue an
+error message and refuse to start if these locations overlap.
+
+Production deployment
+---------------------
+
+If you like fried repositories for lunch, deploy before 11AM.