C and Python RDF library. ALPHA

scossu 28ff18b32d Add outdir to library paths. 1 year ago
bin 3e1baa112c Keep bin dir. 1 year ago
cpython 05b9010e4e Copy graph name URI into graph struct. 1 year ago
docs 2fe6a6338f Add memory profiling tests. 1 year ago
ext 2f21e191e9 Remove unused ext folders. 1 year ago
include 7782524770 Pass array of triple handles to LSUP_graph_add_txn(). 1 year ago
src 7782524770 Pass array of triple handles to LSUP_graph_add_txn(). 1 year ago
test 7782524770 Pass array of triple handles to LSUP_graph_add_txn(). 1 year ago
.gitignore 2f61d787fa Remove bash-isms from Makefile. 1 year ago
.gitmodules 20f1e71094 Remove re2c, lmdb, xxhash source code dependencies. 1 year ago
CODE_OF_CONDUCT 5e1c8e5fa6 Fix Makefile; add docs. 3 years ago
Doxyfile c5c7935c41 Store interface reform: 1 year ago
LICENSE 6934635956 Update README and license. 1 year ago
MANIFEST.in e30c194960 Fix Python builder; bump version; wrap up release. 1 year ago
Makefile 28ff18b32d Add outdir to library paths. 1 year ago
README.md 666aefd54c Update README. 1 year ago
TODO.md e30c194960 Fix Python builder; bump version; wrap up release. 1 year ago
profile.c a52682bd35 Fix mem issue in profile; measure peak memory usage. 1 year ago
pyproject.toml 6fff98fdaf Almost fix Python build and many other fixes. 1 year ago
setup.py 983b385c03 Remove references to lmdb and xxHash sources in setup.py. 1 year ago
test.c 25d3818f81 WIP W3C TTL test suite. 1 year ago
valgrind-python.supp e92f9e7363 Fix triple generator. 3 years ago

README.md

lsup_rdf

This project is work in progress.

Embedded RDF (and maybe later, generic graph) store and manipulation library.

Purpose

The goal of this library is to provide efficient and compact handling of RDF data. At least a complete C API and Python bindings are planned.

This library can be thought of as SQLite or BerkeleyDB for graphs. It can be embedded directly in a program and store persistent data without the need of running a server. In addition, lsup_rdf can perform in-memory graph operations such as validation, de/serialization, boolean operations, lookup, etc.

Two graph back ends are available: a memory one based on hash maps and a disk-based one based on LMDB, an extremely fast and compact embedded key-store value. Graphs can be created independently with either back end within the same program. Triples in the persistent back end are fully indexed and optimized for a balance of lookup speed, data compactness, and write performance (in order of importance).

The code offers an interface to write a custom back end implementation with minimal changes to the core. More documentation on the topic will follow.

This library was initially meant to replace RDFLib dependency and Cython code in Lakesuperior in an effort to reduce code clutter and speed up RDF handling; it is now a project for an independent RDF library, but unless the contributor base expands, it will remain focused on serving Lakesuperior.

Development Status

Alpha. Considered feature-complete from an MVP standpoint. The API may still change significantly. The code may not compile, or throw a fit when run. The Python API may be behind the C API and not work. Test coverage is not sufficient. Documentation is fairly extensive but needs reformatting. This code is being integrated in the higher-level lsup_repo project and is being improved as issues arise. The status will move to beta as soon as lsup_repo covers a significant range of lsup_rdf features.

This is also my first stab at writing a C library (coming from Python) and an unpaid fun project, so don't be surprised if you find some gross stuff.

Road Map

In Scope – Short Term

The short-term goal is to support usage in Lakesuperior and a workable set of features as a standalone library:

  • Handling of graphs, triples, terms
  • Memory- and disk-backed (persistent) graph storage
  • Contexts (disk-backed only)
  • Handling of blank nodes
  • Namespace prefixes
  • Validation of literal and URI terms
  • Validation of RDF triples
  • Fast graph lookup using matching patterns
  • Graph boolean operations
  • Serialization and de-serialization to/from N-Triples and N-Quads
  • Serialization and de-serialization to/from Turtle and TriG
  • Compile-time configuration of max graph size (efficiency vs. capacity)
  • Python bindings
  • Basic command line utilities
  • Store interface for custom back end implementations

Possibly In scope – Long Term

  • Binary serialization and hashing of graphs
  • Binary protocol for synchronizing remote replicas
  • Support for non-local, software-defined storage (e.g. S3)
  • True plug-in architecture for 3rd-party store implementations

Likely Out of Scope

(Unless provided and maintained by external contributors)

  • C++ bindings
  • JSON-LD de/serialization
  • SPARQL support

Building

Requirements

  • It is recommended to build and run LSUP_RDF on a Linux system. No other OS has been tested so far.
  • A C compiler. This has been only tested with gcc so far.
  • LMDB libraries and headers.
  • XXHash >=0.8 libraries and headers.
  • re2c to build the RDF language lexers.
  • cinclude2dot and Graphviz for generating dependency graph (optional).

make commands

The default make command compiles the library. Enter make help to get an overview of the other available commands.

make install installs libraries and headers in the directories set by the environment variable $PREFIX. If this is unset, the default /usr/local prefix is used.

Options to compile with debug symbols are available.

Compile-Time defines (-D[...])

DEBUG: Set debug mode: memory map is at reduced size, logging is forced to TRACE level, etc.

LSUP_RDF_STREAM_CHUNK_SIZE: Size of RDF decoding buffer, i.e., maximum size of a chunk of RDF data fed to the parser when decoding a RDF file into a graph. This should be larger than the maximum expected size of a single term in your RDF source. The default value is 8192, which is mildly conservative. If you experience parsing errors on decoding, and they happen to be on a term such a very long string literal, try recompiling the library with a larger value.

Embedding

The generated liblsuprdf.so and liblsuprdf.a libraries can be linked dynamically or statically to your code. Only the lsup_rdf.h header, which recursively includes other headers in the include directory, needs to be #included in the embedding code.

Environment variables and/or compiler options might have to be set in order to find the dynamic libraries and headers in their install locations.

For compilation and linking examples, refer to test, memtest, perftest and other actions in the current Makefile.

Environment Variables

LSUP_MDB_STORE_PATH: The file path for the persistent store back end. For production use it is strongly recommended to set this to a permanent location on the fastest storage volume available. If unset, the current directory will be used. The directory must exist.

LSUP_LOGLEVEL: A number between 0 and 5, corresponding to:

  • 0: TRACE
  • 1: DEBUG
  • 2: INFO
  • 3: WARN
  • 4: ERROR
  • 5: FATAL

If unspecified, it is set to 3.

LSUP_MDB_MAPSIZE Virtual memory map size. It is recommended to leave this alone. By default, it is set to 1Tb for 64-bit systems and 4Gb for 32-bit systems. The map size by itself does not use up any extra resources.

C API Documentation

Almost all header files are documented. Run doxygen (see Doxygen) to generate HTML documentation in docs/html.

Python API Documentation

TODO