[WIP] Update docs.

docs/about.rst

@@ -1,8 +1,9 @@
 About Lakesuperior
 ==================
 
-Lakesuperior is an alternative `Fedora
-Repository <http://fedorarepository.org>`__ implementation.
+Lakesuperior is a repository system to store binary files and their metadata as
+Linked Data. It is a `Fedora Repository <http://fedorarepository.org>`__
+implementation focused on efficiency, stability and integration with Python.
 
 Fedora is a mature repository software system historically adopted by
 major cultural heritage institutions. It exposes an
@@ -37,7 +38,7 @@ Key features
 -  No performance penalty for storing many resources under the same
    container; no `kudzu
    <https://www.nature.org/ourinitiatives/urgentissues/land-conservation/forests/kudzu.xml>`__
-   pairtree segmentation [#]_ 
+   pairtree segmentation.
 -  Extensible :doc:`provenance metadata <model>` tracking
 -  :doc:`Multi-modal access <architecture>`: HTTP
    (REST), command line interface and native Python API.
@@ -81,7 +82,4 @@ rudimentary road map.
 
 --------------
 
-.. [#] However if your client splits pairtrees upstream, such as Hyrax does,
-   that obviously needs to change to get rid of the path segments.
-
 .. [#] Your mileage may vary depending on the variety of your triples.

docs/apidoc/lakesuperior.api.rst

@@ -1,27 +1,27 @@
-lakesuperior\.api package
-=========================
+lakesuperior.api package
+========================
 
 Submodules
 ----------
 
-lakesuperior\.api\.admin module
--------------------------------
+lakesuperior.api.admin module
+-----------------------------
 
 .. automodule:: lakesuperior.api.admin
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.api\.query module
--------------------------------
+lakesuperior.api.query module
+-----------------------------
 
 .. automodule:: lakesuperior.api.query
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.api\.resource module
-----------------------------------
+lakesuperior.api.resource module
+--------------------------------
 
 .. automodule:: lakesuperior.api.resource
     :members:

docs/apidoc/lakesuperior.cy_include.rst

@@ -0,0 +1,10 @@
+lakesuperior.cy\_include package
+================================
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.cy_include
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/apidoc/lakesuperior.dictionaries.rst

@@ -0,0 +1,30 @@
+lakesuperior.dictionaries package
+=================================
+
+Submodules
+----------
+
+lakesuperior.dictionaries.namespaces module
+-------------------------------------------
+
+.. automodule:: lakesuperior.dictionaries.namespaces
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.dictionaries.srv\_mgd\_terms module
+------------------------------------------------
+
+.. automodule:: lakesuperior.dictionaries.srv_mgd_terms
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.dictionaries
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/apidoc/lakesuperior.endpoints.rst

@@ -1,35 +1,35 @@
-lakesuperior\.endpoints package
-===============================
+lakesuperior.endpoints package
+==============================
 
 Submodules
 ----------
 
-lakesuperior\.endpoints\.admin module
--------------------------------------
+lakesuperior.endpoints.admin module
+-----------------------------------
 
 .. automodule:: lakesuperior.endpoints.admin
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.endpoints\.ldp module
------------------------------------
+lakesuperior.endpoints.ldp module
+---------------------------------
 
 .. automodule:: lakesuperior.endpoints.ldp
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.endpoints\.main module
-------------------------------------
+lakesuperior.endpoints.main module
+----------------------------------
 
 .. automodule:: lakesuperior.endpoints.main
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.endpoints\.query module
--------------------------------------
+lakesuperior.endpoints.query module
+-----------------------------------
 
 .. automodule:: lakesuperior.endpoints.query
     :members:

docs/apidoc/lakesuperior.messaging.rst

@@ -1,27 +1,27 @@
-lakesuperior\.messaging package
-===============================
+lakesuperior.messaging package
+==============================
 
 Submodules
 ----------
 
-lakesuperior\.messaging\.formatters module
-------------------------------------------
+lakesuperior.messaging.formatters module
+----------------------------------------
 
 .. automodule:: lakesuperior.messaging.formatters
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.messaging\.handlers module
-----------------------------------------
+lakesuperior.messaging.handlers module
+--------------------------------------
 
 .. automodule:: lakesuperior.messaging.handlers
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.messaging\.messenger module
------------------------------------------
+lakesuperior.messaging.messenger module
+---------------------------------------
 
 .. automodule:: lakesuperior.messaging.messenger
     :members:

docs/apidoc/lakesuperior.model.rdf.rst

@@ -0,0 +1,38 @@
+lakesuperior.model.rdf package
+==============================
+
+Submodules
+----------
+
+lakesuperior.model.rdf.graph module
+-----------------------------------
+
+.. automodule:: lakesuperior.model.rdf.graph
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.model.rdf.term module
+----------------------------------
+
+.. automodule:: lakesuperior.model.rdf.term
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.model.rdf.triple module
+------------------------------------
+
+.. automodule:: lakesuperior.model.rdf.triple
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.model.rdf
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/apidoc/lakesuperior.model.rst

@@ -1,37 +1,29 @@
-lakesuperior\.model package
-===========================
+lakesuperior.model package
+==========================
 
-Submodules
-----------
+Subpackages
+-----------
 
-lakesuperior\.model\.ldp\_factory module
-----------------------------------------
+.. toctree::
 
-.. automodule:: lakesuperior.model.ldp.ldp_factory
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-lakesuperior\.model\.ldp\_nr module
------------------------------------
+    lakesuperior.model.rdf
+    lakesuperior.model.structures
 
-.. automodule:: lakesuperior.model.ldp.ldp_nr
-    :members:
-    :undoc-members:
-    :show-inheritance:
+Submodules
+----------
 
-lakesuperior\.model\.ldp\_rs module
------------------------------------
+lakesuperior.model.base module
+------------------------------
 
-.. automodule:: lakesuperior.model.ldp.ldp_rs
+.. automodule:: lakesuperior.model.base
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.model\.ldpr module
---------------------------------
+lakesuperior.model.callbacks module
+-----------------------------------
 
-.. automodule:: lakesuperior.model.ldp.ldpr
+.. automodule:: lakesuperior.model.callbacks
     :members:
     :undoc-members:
     :show-inheritance:

docs/apidoc/lakesuperior.model.structures.rst

@@ -0,0 +1,30 @@
+lakesuperior.model.structures package
+=====================================
+
+Submodules
+----------
+
+lakesuperior.model.structures.hash module
+-----------------------------------------
+
+.. automodule:: lakesuperior.model.structures.hash
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.model.structures.keyset module
+-------------------------------------------
+
+.. automodule:: lakesuperior.model.structures.keyset
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.model.structures
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/apidoc/lakesuperior.rst

@@ -7,74 +7,86 @@ Subpackages
 .. toctree::
 
     lakesuperior.api
+    lakesuperior.cy_include
+    lakesuperior.dictionaries
     lakesuperior.endpoints
     lakesuperior.messaging
     lakesuperior.model
     lakesuperior.store
+    lakesuperior.util
+    lakesuperior.wsgi
 
 Submodules
 ----------
 
-lakesuperior\.app module
-------------------------
+lakesuperior.app module
+-----------------------
 
 .. automodule:: lakesuperior.app
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.config\_parser module
------------------------------------
+lakesuperior.config\_parser module
+----------------------------------
 
 .. automodule:: lakesuperior.config_parser
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.env module
-------------------------
+lakesuperior.env\_setup module
+------------------------------
 
-.. automodule:: lakesuperior
+.. automodule:: lakesuperior.env_setup
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.env\_setup module
--------------------------------
+lakesuperior.exceptions module
+------------------------------
 
-.. automodule:: lakesuperior.env_setup
+.. automodule:: lakesuperior.exceptions
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.exceptions module
--------------------------------
+lakesuperior.globals module
+---------------------------
 
-.. automodule:: lakesuperior.exceptions
+.. automodule:: lakesuperior.globals
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.globals module
-----------------------------
+lakesuperior.lsup\_admin module
+-------------------------------
 
-.. automodule:: lakesuperior.globals
+.. automodule:: lakesuperior.lsup_admin
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.migrator module
------------------------------
+lakesuperior.migrator module
+----------------------------
 
 .. automodule:: lakesuperior.migrator
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.toolbox module
+lakesuperior.profiler module
 ----------------------------
 
-.. automodule:: lakesuperior.toolbox
+.. automodule:: lakesuperior.profiler
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.server module
+--------------------------
+
+.. automodule:: lakesuperior.server
     :members:
     :undoc-members:
     :show-inheritance:

docs/apidoc/lakesuperior.store.ldp_nr.rst

@@ -1,19 +1,19 @@
-lakesuperior\.store\.ldp\_nr package
-====================================
+lakesuperior.store.ldp\_nr package
+==================================
 
 Submodules
 ----------
 
-lakesuperior\.store\.ldp\_nr\.base\_non\_rdf\_layout module
------------------------------------------------------------
+lakesuperior.store.ldp\_nr.base\_non\_rdf\_layout module
+--------------------------------------------------------
 
 .. automodule:: lakesuperior.store.ldp_nr.base_non_rdf_layout
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.store\.ldp\_nr\.default\_layout module
-----------------------------------------------------
+lakesuperior.store.ldp\_nr.default\_layout module
+-------------------------------------------------
 
 .. automodule:: lakesuperior.store.ldp_nr.default_layout
     :members:

docs/apidoc/lakesuperior.store.ldp_rs.rst

@@ -1,19 +1,27 @@
-lakesuperior\.store\.ldp\_rs package
-====================================
+lakesuperior.store.ldp\_rs package
+==================================
 
 Submodules
 ----------
 
-lakesuperior\.store\.ldp\_rs\.lmdb\_store module
-------------------------------------------------
+lakesuperior.store.ldp\_rs.lmdb\_store module
+---------------------------------------------
 
 .. automodule:: lakesuperior.store.ldp_rs.lmdb_store
     :members:
     :undoc-members:
     :show-inheritance:
 
-lakesuperior\.store\.ldp\_rs\.rsrc\_centric\_layout module
-----------------------------------------------------------
+lakesuperior.store.ldp\_rs.lmdb\_triplestore module
+---------------------------------------------------
+
+.. automodule:: lakesuperior.store.ldp_rs.lmdb_triplestore
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.store.ldp\_rs.rsrc\_centric\_layout module
+-------------------------------------------------------
 
 .. automodule:: lakesuperior.store.ldp_rs.rsrc_centric_layout
     :members:

docs/apidoc/lakesuperior.store.rst

@@ -1,5 +1,5 @@
-lakesuperior\.store package
-===========================
+lakesuperior.store package
+==========================
 
 Subpackages
 -----------
@@ -9,6 +9,26 @@ Subpackages
     lakesuperior.store.ldp_nr
     lakesuperior.store.ldp_rs
 
+Submodules
+----------
+
+lakesuperior.store.base\_lmdb\_store module
+-------------------------------------------
+
+.. automodule:: lakesuperior.store.base_lmdb_store
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.store.metadata\_store module
+-----------------------------------------
+
+.. automodule:: lakesuperior.store.metadata_store
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
 Module contents
 ---------------
 

docs/apidoc/lakesuperior.util.rst

@@ -0,0 +1,54 @@
+lakesuperior.util package
+=========================
+
+Submodules
+----------
+
+lakesuperior.util.benchmark module
+----------------------------------
+
+.. automodule:: lakesuperior.util.benchmark
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.util.generators module
+-----------------------------------
+
+.. automodule:: lakesuperior.util.generators
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.util.ingest\_random\_image module
+----------------------------------------------
+
+.. automodule:: lakesuperior.util.ingest_random_image
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.util.locustfile module
+-----------------------------------
+
+.. automodule:: lakesuperior.util.locustfile
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+lakesuperior.util.toolbox module
+--------------------------------
+
+.. automodule:: lakesuperior.util.toolbox
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.util
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/apidoc/lakesuperior.wsgi.rst

@@ -0,0 +1,10 @@
+lakesuperior.wsgi package
+=========================
+
+Module contents
+---------------
+
+.. automodule:: lakesuperior.wsgi
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/conf.py

@@ -69,7 +69,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'lakesuperior'
-copyright = '2018, Everybody & Nobody'
+copyright = '2019, Knowledge Transfer, LLC'
 author = 'Stefano Cossu'
 
 # The version info for the project you're documenting, acts as replacement for

docs/config.rst

@@ -1,5 +1,5 @@
-Configuration Reference
-=======================
+Application Configuration Reference
+===================================
 
-.. autoyaml:: .
+.. autoyaml:: application.yml
 

docs/structures.rst

@@ -1,99 +0,0 @@
-Data Structure Internals
-========================
-
-**(Draft)**
-
-Lakesuperior has its own methods for handling in-memory graphs. These methods
-rely on C data structures and are therefore much faster than Python/RDFLib
-objects.
-
-The graph data model modules are in :py:module:`lakesuperior.model.graph`.
-
-The Graph Data Model
---------------------
-
-Triples are stored in a C hash set. Each triple is represented by a pointer to
-a ``BufferTriple`` structure stored in a temporary memory pool. This pool is
-tied to the life cycle of the ``SimpleGraph`` object it belongs to.
-
-A triple structure contains three pointers to ``Buffer`` structures, which
-contain a serialized version of a RDF term. These structures are stored in the
-``SimpleGraph`` memory pool as well.
-
-Each ``SimpleGraph`` object has a ``_terms`` property and a ``_triples``
-property. These are C hash sets holding addresses of unique terms and
-triples inserted in the graph. If the same term is entered more than once,
-in any position in any triple, the first one entered is used and is pointed to
-by the triple. This makes the graph data structure very compact.
-
-In summary, the pointers can be represented this way::
-
-   <serialized term data in mem pool (x3)>
-         ^      ^      ^
-         |      |      |
-   <Term structures in mem pool (x3)>
-         ^      ^      ^
-         |      |      |
-   <Term struct addresses in _terms set (x3)>
-         ^      ^      ^
-         |      |      |
-   <Triple structure in mem pool>
-         ^
-         |
-   <address of triple in _triples set>
-
-Let's say we insert the following triples in a ``SimpleGraph``::
-
-   <urn:s:0> <urn:p:0> <urn:o:0>
-   <urn:s:0> <urn:p:1> <urn:o:1>
-   <urn:s:0> <urn:p:1> <urn:o:2>
-   <urn:s:0> <urn:p:0> <urn:o:0>
-
-The memory pool contains the following byte arrays  of raw data, displayed in
-the following list with their relative addresses (simplified to 8-bit
-addresses and fixed-length byte strings for readability)::
-
-   0x00     <urn:s:0>
-   0x09     <urn:p:0>
-   0x12     <urn:o:0>
-
-   0x1b     <urn:s:0>
-   0x24     <urn:p:1>
-   0x2d     <urn:o:1>
-
-   0x36     <urn:s:0>
-   0x3f     <urn:p:1>
-   0x48     <urn:o:2>
-
-   0x51     <urn:s:0>
-   0x5a     <urn:p:0>
-   0x63     <urn:o:0>
-
-However, the ``_terms`` set contains only ``Buffer`` structures pointing to
-unique addresses::
-
-   0x00
-   0x09
-   0x12
-   0x24
-   0x2d
-   0x48
-
-The other terms are just unutilized. They will be deallocated en masse when
-the ``SimpleGraph`` object is garbage collected.
-
-The ``_triples`` set would then contain 3 unique entries pointing to the unique
-term addresses::
-
-   0x00  0x09  0x12
-   0x00  0x24  0x2d
-   0x00  0x24  0x48
-
-(the actual addresses would actually belong to the structures pointing to the
-raw data, but this is just an illustrative example).
-
-The advantage of this approach is that the memory pool is contiguous and
-append-only (until it gets purged), so it's cheap to just add to it, while the
-sets that must maintain uniqueness and are the ones that most operations
-(lookup, adding, removing, slicing, copying, etc.) are done on, contain much
-less data and are therefore faster.

docs/usage.rst

@@ -2,7 +2,7 @@ Sample Usage
 ============
 
 LDP (REST) API
--------
+--------------
 
 The following are very basic examples of LDP interaction. For a more complete
 reference, please consult the `Fedora API guide

lakesuperior/model/rdf/__init__.py

@@ -8,8 +8,8 @@ of large RDF datasets.
 
 See individual modules for detailed documentation:
 
-- :py:module:`lakesuperior.model.rdf.term`
-- :py:module:`lakesuperior.model.rdf.triple`
-- :py:module:`lakesuperior.model.rdf.graph`
+- :py:mod:`lakesuperior.model.rdf.term`
+- :py:mod:`lakesuperior.model.rdf.triple`
+- :py:mod:`lakesuperior.model.rdf.graph`
 
 """

lakesuperior/model/rdf/graph.pyx

@@ -524,7 +524,7 @@ cdef class Graph:
         Any and all of the lookup terms msy be ``None``.
 
         :rtype: Graph
-        "return: New Graph instance with matching triples.
+        :return: New Graph instance with matching triples.
         """
         cdef:
             Graph res_gr = self.empty_copy()
@@ -624,7 +624,7 @@ cdef class Graph:
 ## FACTORY METHODS
 
 def from_rdf(store=None, uri=None, *args, **kwargs):
-    """
+    r"""
     Create a Graph from a serialized RDF string.
 
     This factory function takes the same arguments as
@@ -634,9 +634,9 @@ def from_rdf(store=None, uri=None, *args, **kwargs):
 
     :param uri: see :py:meth:`Graph.__cinit__`.
 
-    :param *args: Positional arguments passed to RDFlib's ``parse``.
+    :param \*args: Positional arguments passed to RDFlib's ``parse``.
 
-    :param *kwargs: Keyword arguments passed to RDFlib's ``parse``.
+    :param \*\*kwargs: Keyword arguments passed to RDFlib's ``parse``.
 
     :rtype: Graph
     """

lakesuperior/store/metadata_store.py

@@ -22,9 +22,9 @@ class MetadataStore(BaseLmdbStore):
     """
     Currently implemented:
 
-    - ``checksums``: registry of
-    LDP resource graphs, indicated in the key by their UID, and their
-    cryptographic hashes.
+    - ``checksums``: registry of LDP resource graphs, indicated in the key by
+      their UID, and their cryptographic hashes.
+
     """