Example¶

Ferenda can be used either as a library or as a command-line tool. This code uses the Ferenda API to create a website containing all(*) RFCs and W3C recommended standards.

from ferenda.sources.tech import RFC, W3Standards
from ferenda.manager import makeresources, frontpage, runserver, setup_logger
from ferenda.errors import DocumentRemovedError, ParseError, FSMStateError

config = {'datadir':'netstandards/exampledata', 
          'loglevel':'DEBUG',
          'force':False,
          'storetype':'SQLITE',
          'storelocation':'netstandards/exampledata/netstandards.sqlite',
          'storerepository':'netstandards',
          'downloadmax': 50 }

setup_logger(level='DEBUG')

# Set up two document repositories
docrepos = (RFC(**config), W3Standards(**config))

for docrepo in docrepos:
    # Download a bunch of documents
    docrepo.download()
    
    # Parse all downloaded documents
    for basefile in docrepo.store.list_basefiles_for("parse"):
        try:
            docrepo.parse(basefile)
        except ParseError as e:
            pass  # or handle this in an appropriate way

    # Index the text content and metadata of all parsed documents
    for basefile in docrepo.store.list_basefiles_for("relate"):
        docrepo.relate(basefile, docrepos)

# Prepare various assets for web site navigation
makeresources(docrepos,
              resourcedir="netstandards/exampledata/rsrc",
              sitename="Netstandards",
              sitedescription="A repository of internet standard documents")

# Relate for all repos must run before generate for any repo
for docrepo in docrepos:
    # Generate static HTML files from the parsed documents, 
    # with back- and forward links between them, etc.
    for basefile in docrepo.store.list_basefiles_for("generate"):
        docrepo.generate(basefile)
        
    # Generate a table of contents of all available documents
    docrepo.toc()
    # Generate feeds of new and updated documents, in HTML and Atom flavors
    docrepo.news()

# Create a frontpage for the entire site
frontpage(docrepos,path="netstandards/exampledata/index.html")

# Start WSGI app at http://localhost:8000/ with navigation,
# document viewing, search and API
# runserver(docrepos, port=8000, documentroot="netstandards/exampledata")

Alternately, using the command line tools and the project framework:

$ ferenda-setup netstandards
$ cd netstandards
$ ./ferenda-build.py ferenda.sources.tech.RFC enable
$ ./ferenda-build.py ferenda.sources.tech.W3Standards enable
$ ./ferenda-build.py all all --downloadmax=50
# $ ./ferenda-build.py all runserver &
# $ open http://localhost:8000/

Prerequisites¶

Operating system

Ferenda is tested and works on Unix, Mac OS and Windows.

Python

Version 2.6 or newer required, 3.4 recommended. The code base is primarily developed with python 3, and is heavily dependent on all forward compatibility features introduced in Python 2.6. Python 3.0 and 3.1 is not supported.

Third-party libraries

beautifulsoup4, rdflib, html5lib, lxml, requests, whoosh, pyparsing, jsmin, six and their respective requirements. If you install ferenda using easy_install or pip they should be installed automatically. If you’re working with a clone of the source repository you can install them with a simple pip install -r requirements.py3.txt (substitute with requirements.py2.txt if you’re not yet using python 3).

Command-line tools

For some functionality, certain executables must be present and in your $PATH:

• PDFReader requires pdftotext and pdftohtml (from poppler, version 0.21 or newer).
  • The crop() method requires convert (from ImageMagick).
  • The convert_to_pdf parameter to read() requires the soffice binary from either OpenOffice or LibreOffice
  • The ocr_lang parameter to read() requires tesseract (from tesseract-ocr), convert (see above) and tiffcp (from libtiff)
• WordReader requires antiword to handle old .doc files.
• TripleStore can perform some operations (bulk up- and download) much faster if curl is installed.

Once you have a large number of documents and metadata about those documents, you’ll need a RDF triple store, either Sesame (at least version 2.7) or Fuseki (at least version 1.0). For document collections small enough to keep all metadata in memory you can get by with only rdflib, using either a Sqlite or a Berkely DB (aka Sleepycat/bsddb) backend. For further information, see Triple stores.

Similarly, once you have a large collection of text (either many short documents, or fewer long documents), you’ll need an fulltext search engine to use the search feature (enabled by default). For small document collections the embedded whoosh library is used. Right now, ElasticSearch is the only supported external fulltext search engine.

As a rule of thumb, if your document collection contains over 100 000 RDF triples or 100 000 words, you should start thinking about setting up an external triple store or a fulltext search engine. See Fulltext search engines.

Installing¶

Ferenda should preferably be installed with pip (in fact, it’s the only method tested):

pip install ferenda

You should definitely consider installing ferenda in a virtualenv.

Features¶

• Handles downloading, structural parsing and regeneration of large document collections.
• Contains libraries to make reading of plain text, PDF and MS Word documents as easy as HTML (Note: these are far from finished).
• Uses established information standards as much as possible: If you like XHTML, XSLT, XML namespaces, RDF and SPARQL, you’ll feel right at home.
• Leverages your favourite python libraries: requests, beautifulsoup, rdflib, lxml, pyparsing and whoosh.
• Possible to patch documents if the source content has errors.
• Easy to write reference/citation parsers and run them on document text.
• Documents in the same and other collections are automatically cross-referenced.
• Uses caches and dependency management to avoid performing the same work over and over.
• Once documents are downloaded and structured, you get a usable web site with API, Atom feeds and search for free (Note: API functionality not yet implemented).
• Web site generation can create a set of static HTML pages for offline use (though you lose search and API functionality).
• Pull in commentary for documents from other sources (Note: you’ll have to do most of this work yourself).
• Create topic hubs / keyword pages for document from multiple collections (Note: remains undocumented).

Next step¶

See First steps to set up a project and create your own simple document repository.

Creating a Document repository class¶

Any document collection is handled by a DocumentRepository class (or docrepo for short), so our first task is to create a docrepo for W3C standards.

A docrepo class is responsible for downloading documents in a specific document collection. These classes can inherit from DocumentRepository, which amongst others provides the method download() for this. Since the details of how documents are made available on the web differ greatly from collection to collection, you’ll often have to override the default implementation, but in this particular case, it suffices. The default implementation assumes that all documents are available from a single index page, and that the URLs of the documents follow a set pattern.

The W3C standards are set up just like that: All standards are available at http://www.w3.org/TR/tr-status-all. There are a lot of links to documents on that page, and not all of them are links to recommended standards. A simple way to find only the recommended standards is to see if the link follows the pattern http://www.w3.org/TR/<year>/REC-<standardid>-<date>.

Creating a docrepo that is able to download all web standards is then as simple as creating a subclass and setting three class properties. Create this class in the current directory (or anywhere else on your python path) and save it as w3cstandards.py

from ferenda import DocumentRepository

class W3CStandards(DocumentRepository):
    alias = "w3c"
    start_url = "http://www.w3.org/TR/tr-status-all"
    document_url_regex = "http://www.w3.org/TR/(?P<year>\d{4})/REC-(?P<basefile>.*)-(?P<date>\d+)"

The first property, alias, is required for all docrepos and controls the alias used by the command line tool for that docrepo, as well as the path where files are stored, amongst other things. If your project has a large collection of docrepos, it’s important that they all have unique aliases.

The other two properties are parameters which the default implementation of download() uses in order to find out which documents to download. start_url is just a simple regular URL, while document_url_regex is a standard re regex with named groups. The group named basefile has special meaning, and will be used as a base for stored files and elsewhere as a short identifier for the document. For example, the web standard found at URL http://www.w3.org/TR/2012/REC-rdf-plain-literal-20121211/ will have the basefile rdf-plain-literal.

Using ferenda-build.py and registering docrepo classes¶

Next step is to enable our class. Like most tasks, this is done using the command line tool present in your project directory. To register the class (together with a short alias) in your ferenda.ini configuration file, run the following:

$ ./ferenda-build.py w3cstandards.W3CStandards enable
13:04:16 root INFO Enabled class w3cstandards.W3CStandards (alias 'w3c')

This creates a new section in ferenda.ini that just looks like the following:

[w3c]
class = w3cstandards.W3CStandards

From this point on, you can use the class name or the alias “w3c” interchangably:

$ ./ferenda-build.py w3cstandards.W3CStandards status # verbose
13:04:17 root INFO w3cstandards.W3CStandards status finished in 0.004 sec
Status for document repository 'w3c' (w3cstandards.W3CStandards)
 download: None.
 parse: None.
 generated: None.

$ ./ferenda-build.py w3c status # terse, exactly the same result

Downloading¶

To test the downloading capabilities of our class, you can run the download method directly from the command line using the command line tool:

$ ./ferenda-build.py w3c download 
13:04:21 w3c INFO Downloading max 3 documents
13:04:22 w3c INFO geolocation-API: downloaded from http://www.w3.org/TR/2013/REC-geolocation-API-20131024/
13:04:23 w3c INFO touch-events: downloaded from http://www.w3.org/TR/2013/REC-touch-events-20131010/
13:04:25 w3c INFO ttml1: downloaded from http://www.w3.org/TR/2013/REC-ttml1-20130924/
# and so on...

After a few minutes of downloading, the result is a bunch of files in data/w3c/downloaded:

$ ls -1 data/w3c/downloaded
geolocation-API.html
geolocation-API.html.etag
touch-events.html
touch-events.html.etag
ttml1.html
ttml1.html.etag

We can get a overview of the status of our docrepo using the status command:

$ ./ferenda-build.py w3cstandards.W3CStandards status # verbose
13:04:17 root INFO w3cstandards.W3CStandards status finished in 0.004 sec
Status for document repository 'w3c' (w3cstandards.W3CStandards)
 download: None.
 parse: None.
 generated: None.

$ ./ferenda-build.py w3c status # terse, exactly the same result

from w3cstandards import W3CStandards
repo = W3CStandards()
repo.download()
repo.status()
# or use repo.get_status() to get all status information in a nested dict

Finally, if the logging information scrolls by too quickly and you want to read it again, take a look in the data/logs directory. Each invocation of ferenda-build.py creates a new log file containing the same information that is written to stdout.

Parsing¶

Let’s try the next step in the workflow, to parse one of the documents we’ve downloaded.

$ ./ferenda-build.py w3c parse rdfa-core
13:04:33 w3c INFO rdfa-core: OK (2.033 sec)
13:04:33 root INFO w3c parse finished in 2.053 sec

By now, you might have realized that our command line tool generally is called in the following manner:

$ ./ferenda-build.py <docrepo> <command> [argument(s)]

The parse command resulted in one new file being created in data/w3c/parsed.

$ ls -1 data/w3c/parsed
rdfa-core.xhtml

And we can again use the status command to get a comprehensive overview of our document repository.

$ ./ferenda-build.py w3c status
13:04:34 root INFO w3c status finished in 0.013 sec
Status for document repository 'w3c' (w3cstandards.W3CStandards)
 download: ttml1, touch-events, rdfa-core... (1 more)
 parse: rdfa-core. Todo: ttml1, touch-events, geolocation-API.
 generated: None. Todo: rdfa-core.

Note that by default, subsequent invocations of parse won’t actually parse documents that don’t need parsing.

$ ./ferenda-build.py w3c parse rdfa-core
13:04:35 root INFO w3c parse finished in 0.016 sec

But during development, when you change the parsing code frequently, you’ll need to override this through the --force flag (or set the force parameter in ferenda.ini).

$ ./ferenda-build.py w3c parse rdfa-core --force
13:04:38 w3c INFO rdfa-core: OK (2.024 sec)
13:04:38 root INFO w3c parse finished in 2.043 sec

from w3cstandards import W3CStandards
repo = W3CStandards(force=True)
repo.parse("rdfa-core")

Note also that you can parse all downloaded documents through the --all flag, and control logging verbosity by the --loglevel flag.

$ ./ferenda-build.py w3c parse --all --loglevel=DEBUG
13:04:39 w3c DEBUG ttml1: Starting
13:04:43 w3c DEBUG ttml1: Created data/w3c/parsed/ttml1.xhtml
13:04:45 w3c DEBUG ttml1: 12 triples extracted to data/w3c/distilled/ttml1.rdf
13:04:45 w3c INFO ttml1: OK (5.816 sec)
13:04:45 w3c DEBUG touch-events: Starting
13:04:45 w3c DEBUG touch-events: Created data/w3c/parsed/touch-events.xhtml
13:04:45 w3c DEBUG touch-events: 8 triples extracted to data/w3c/distilled/touch-events.rdf
13:04:45 w3c INFO touch-events: OK (0.486 sec)
13:04:45 w3c DEBUG geolocation-API: Starting
13:04:46 w3c DEBUG geolocation-API: Created data/w3c/parsed/geolocation-API.xhtml
13:04:46 w3c DEBUG geolocation-API: 5 triples extracted to data/w3c/distilled/geolocation-API.rdf
13:04:46 w3c INFO geolocation-API: OK (0.323 sec)
13:04:46 root INFO w3c parse finished in 6.662 sec

import logging
from w3cstandards import W3CStandards
# client code is responsible for setting the effective log level -- ferenda 
# just emits log messages, and depends on the caller to setup the logging 
# subsystem in an appropriate way
logging.getLogger().setLevel(logging.INFO)
repo = W3CStandards()
for basefile in repo.store.list_basefiles_for("parse"):
    # You you might want to try/catch the exception
    # ferenda.errors.ParseError or any of it's children here
    repo.parse(basefile)

If we take a look at the files created in data/w3c/distilled, we see some metadata for each document. This metadata has been automatically extracted from RDFa statements in the XHTML documents, but is so far very spartan.

Now take a look at the files created in data/w3c/parsed. The default implementation of parse() processes the DOM of the main body of the document, but some tags and attribute that are used only for formatting are stripped, such as <style> and <script>.

These documents have quite a lot of “boilerplate” text such as table of contents and links to latest and previous versions which we’d like to remove so that just the actual text is left (problem 1). And we’d like to explicitly extract some parts of the document and represent these as metadata for the document – for example the title, the publication date, the authors/editors of the document and it’s abstract, if available (problem 2).

Just like the default implementation of download() allowed for some customization using class variables, we can solve problem 1 by setting two additional class variables:

    parse_content_selector="body"
    parse_filter_selectors=["div.toc", "div.head"]

The parse_content_selector member specifies, using CSS selector syntax, the part of the document which contains our main text. It defaults to "body", and can often be set to ".content" (the first element that has a class=”content” attribute”), "#main-text" (any element with the id "main-text"), "article" (the first <article> element) or similar. The parse_filter_selectors is a list of similar selectors, with the difference that all matching elements are removed from the tree. In this case, we use it to remove some boilerplate sections that often within the content specified by parse_content_selector, but which we don’t want to appear in the final result.

In order to solve problem 2, we can override one of the methods that the default implementation of parse() calls:

    def parse_metadata_from_soup(self, soup, doc):
        from rdflib import Namespace
        from ferenda import Describer
        from ferenda import util
        import re
        DCT = Namespace("http://purl.org/dc/terms/")
        FOAF = Namespace("http://xmlns.com/foaf/0.1/")
        d = Describer(doc.meta, doc.uri)
        d.rdftype(FOAF.Document)
        d.value(DCT.title, soup.find("title").text, lang=doc.lang)
        d.value(DCT.abstract, soup.find(True, "abstract"), lang=doc.lang)
        # find the issued date -- assume it's the first thing that looks
        # like a date on the form "22 August 2013"
        re_date = re.compile(r'(\d+ \w+ \d{4})')
        datenode = soup.find(text=re_date)
        datestr = re_date.search(datenode).group(1)
        d.value(DCT.issued, util.strptime(datestr, "%d %B %Y"))
        editors = soup.find("dt", text=re.compile("Editors?:"))
        for editor in editors.find_next_siblings("dd"):
            editor_name = editor.text.strip().split(", ")[0]
            d.value(DCT.editor, editor_name)

parse_metadata_from_soup() is called with a document object and the parsed HTML document in the form of a BeautifulSoup object. It is the responsibility of parse_metadata_from_soup() to add document-level metadata for this document, such as it’s title, publication date, and similar. Note that parse_metadata_from_soup() is run before the parse_content_selector and parse_filter_selectors are applied, so the BeautifulSoup object passed into it contains the entire document.

Now, if you run parse --force again, both documents and metadata are in better shape. Further down the line the value of properly extracted metadata will become more obvious.

Republishing the parsed content¶

The XHTML contains metadata in RDFa format. As such, you can extract all that metadata and put it into a triple store. The relate command does this, as well as creating a full text index of all textual content:

$ ./ferenda-build.py w3c relate --all
13:04:47 w3c INFO Clearing context http://localhost:8000/dataset/w3c at repository ferenda
13:04:54 w3c INFO Dumped 34 triples from context http://localhost:8000/dataset/w3c to data/w3c/distilled/dump.nt
13:04:54 root INFO w3c relate finished in 7.655 sec

The next step is to create a number of resource files (placed under data/rsrc). These resource files include css and javascript files for the new website we’re creating, as well as a xml configuration file used by the XSLT transformation done by generate below:

$ ./ferenda-build.py w3c makeresources
$ find data/rsrc -print
data/rsrc
data/rsrc/css
data/rsrc/css/ferenda.css
data/rsrc/css/main.css
data/rsrc/css/normalize-1.1.3.css
data/rsrc/js
data/rsrc/js/ferenda.js
data/rsrc/js/jquery-1.10.2.js
data/rsrc/js/modernizr-2.6.3.js
data/rsrc/js/respond-1.3.0.js
data/rsrc/resources.xml

Running makeresources is needed for the final few steps.

$ ./ferenda-build.py w3c generate --all
13:04:58 w3c INFO ttml1: OK (2.102 sec)
13:04:59 w3c INFO touch-events: OK (0.112 sec)
13:04:59 w3c INFO rdfa-core: OK (0.220 sec)
13:04:59 w3c INFO geolocation-API: OK (0.100 sec)
13:04:59 root INFO w3c generate finished in 2.547 sec

The generate command creates browser-ready HTML5 documents from our structured XHTML documents, using our site’s navigation.

$ ./ferenda-build.py w3c toc
13:05:01 w3c INFO Created data/w3c/toc/issued/2011.html
13:05:01 w3c INFO Created data/w3c/toc/issued/2013.html
13:05:01 w3c INFO Created data/w3c/toc/issued/2014.html
13:05:02 w3c INFO Created data/w3c/toc/title/a.html
13:05:02 w3c INFO Created data/w3c/toc/title/c.html
13:05:02 w3c INFO Created data/w3c/toc/title/r.html
13:05:02 w3c INFO Created data/w3c/toc/title/w.html
13:05:02 w3c INFO Created data/w3c/toc/index.html
13:05:02 root INFO w3c toc finished in 1.739 sec
$ ./ferenda-build.py w3c news
13:05:03 w3c INFO feed main: 4 entries
13:05:03 root INFO w3c news finished in 0.086 sec
$ ./ferenda-build.py w3c frontpage
13:05:04 root INFO frontpage: wrote data/index.html (0.017 sec)

The toc and feeds commands creates static files for general indexes/tables of contents of all documents in our docrepo as well as Atom feeds, and the frontpage command creates a suitable frontpage for the site as a whole.

from ferenda import manager
from w3cstandards import W3CStandards
repo = W3CStandards()
for basefile in repo.store.list_basefiles_for("relate"):
    repo.relate(basefile)
manager.makeresources([repo], sitename="Standards", sitedescription="W3C standards, in a new form")
for basefile in repo.store.list_basefiles_for("generate"):
    repo.generate(basefile)
repo.toc()
repo.news()
manager.frontpage([repo])

Finally, to start a development web server and check out the finished result:

$ ./ferenda-build.py w3c runserver
$ open http://localhost:8080/

Now you’ve created your own web site with structured documents. It contains listings of all documents, feeds with updated documents (in both HTML and Atom flavors) and full text search. In order to deploy your site, you can run it under Apache+mod_wsgi, ngnix+uWSGI, Gunicorn or just about any WSGI capable web server, see The WSGI app.

To keep it up-to-date whenever the W3C issues new standards, use the following command:

$ ./ferenda-build.py w3c all
13:05:07 w3c INFO Downloading max 3 documents
13:05:07 root INFO w3cstandards.W3CStandards download finished in 2.476 sec
13:05:07 root INFO w3cstandards.W3CStandards parse finished in 0.010 sec
13:05:07 root INFO w3cstandards.W3CStandards relate: Nothing to do!
13:05:07 root INFO w3cstandards.W3CStandards relate finished in 0.005 sec
13:05:07 w3c INFO ttml1: OK (0.000 sec)
13:05:07 w3c INFO touch-events: OK (0.000 sec)
13:05:07 w3c INFO rdfa-core: OK (0.000 sec)
13:05:07 w3c INFO geolocation-API: OK (0.000 sec)
13:05:07 root INFO w3cstandards.W3CStandards generate finished in 0.006 sec
13:05:01 w3c INFO Created data/w3c/toc/issued/2011.html
13:05:01 w3c INFO Created data/w3c/toc/issued/2013.html
13:05:01 w3c INFO Created data/w3c/toc/issued/2014.html
13:05:02 w3c INFO Created data/w3c/toc/title/a.html
13:05:02 w3c INFO Created data/w3c/toc/title/c.html
13:05:02 w3c INFO Created data/w3c/toc/title/r.html
13:05:02 w3c INFO Created data/w3c/toc/title/w.html
13:05:09 w3c INFO Created data/w3c/toc/index.html
13:05:09 root INFO w3cstandards.W3CStandards toc finished in 1.705 sec
13:05:09 w3c INFO feed main: 4 entries
13:05:09 root INFO w3cstandards.W3CStandards news finished in 0.057 sec
13:05:09 root INFO frontpage: wrote data/index.html (0.013 sec)

The “all” command is an alias that runs download, parse --all, relate --all, generate --all, toc and feeds in sequence.

This 20-line example of a docrepo took a lot of shortcuts by depending on the default implementation of the download() and parse() methods. Ferenda tries to make it really to get something up and running quickly, and then improving each step incrementally.

In the next section Creating your own document repositories we will take a closer look at each of the six main steps (download, parse, relate, generate, toc and news), including how to completely replace the built-in methods. You can also take a look at the source code for ferenda.sources.tech.W3Standards, which contains a more complete (and substantially longer) implementation of download(), parse() and the others.

Writing your own download implementation¶

The purpose of the download() method is to fetch source documents from a remote source and store them locally, possibly under different filenames but otherwise bit-for-bit identical with how they were stored at the remote source (see File storage for more information about how and where files are stored locally).

The default implementation of download() uses a small number of methods and class variables to do the actual work. By selectively overriding these, you can often avoid rewriting a complete implementation of download().

import re
from datetime import datetime, date

import requests

from ferenda import DocumentRepository, TextReader
from ferenda import util

class RFCs(DocumentRepository):
    alias = "rfc"
    start_url = "http://www.ietf.org/download/rfc-index.txt"
    document_url_template = "http://tools.ietf.org/rfc/rfc%(basefile)s.txt"
    downloaded_suffix = ".txt"

$ ./ferenda-build.py rfcs.RFCs enable
$ ./ferenda-build.py rfc download

    def download(self):
        self.log.debug("download: Start at %s" %  self.start_url)
        indextext = requests.get(self.start_url).text
        reader = TextReader(string=indextext)  # see TextReader class
        iterator = reader.getiterator(reader.readparagraph)
        if not isinstance(self.config.downloadmax, (int, type(None))):
            self.config.downloadmax = int(self.config.downloadmax)
            
        for basefile in self.download_get_basefiles(iterator):
            self.download_single(basefile)

    @downloadmax
    def download_get_basefiles(self, source):
        for p in reversed(list(source)):
            if re.match("^(\d{4}) ",p): # looks like a RFC number
                if not "Not Issued." in p: # Skip RFC known to not exist
                    basefile = str(int(p[:4]))  # eg. '0822' -> '822'
                    yield basefile

url = "http://tools.ietf.org/rfc/rfc%s.txt" % basefile
self.download_single(basefile, url)

download
  start_url (class variable)
  download_get_basefiles (instancemethod) - iterator
  download_single (instancemethod)
     remote_url (instancemethod)
         document_url_template (class variable)
     download_if_needed (instancemethod)

./ferenda-build.py rfc download 6725

def download(self, basefile=None):
    if basefile:
        return self.download_single(basefile)

    # the rest of your code

Writing your own parse implementation¶

The purpose of the parse() method is to take the downloaded file(s) for a particular document and parse it into a structured document with proper metadata, both for the document as a whole, but also for individual sections of the document.

    # In order to properly handle our RDF data, we need to tell
    # ferenda which namespaces we'll be using. These will be available
    # as rdflib.Namespace objects in the self.ns dict, which means you
    # can state that something is eg. a dct:title by using
    # self.ns['dct'].title. See
    # :py:data:`~ferenda.DocumentRepository.namespaces`
    namespaces = ('rdf',  # always needed
                  'dct',  # title, identifier, etc
                  'bibo', # Standard and DocumentPart classes, chapter prop
                  'xsd',  # datatypes
                  'foaf', # rfcs are foaf:Documents for now
                  ('rfc','http://example.org/ontology/rfc/')
                  )
    from rdflib import Namespace
    rdf_type = Namespace('http://example.org/ontology/rfc/').RFC

    
    from ferenda.decorators import managedparsing

    @managedparsing
    def parse(self, doc):
        # some very simple heuristic rules for determining 
        # what an individual paragraph is
   
        def is_heading(p):
            # If it's on a single line and it isn't indented with spaces
            # it's probably a heading.
            if p.count("\n") == 0 and not p.startswith(" "):
                return True
  
        def is_pagebreak(p):
            # if it contains a form feed character, it represents a page break
            return "\f" in p
        
        # Parsing a document consists mainly of two parts:
        # 1: First we parse the body of text and store it in doc.body
        from ferenda.elements import Body, Preformatted, Title, Heading
        from ferenda import Describer
        reader = TextReader(self.store.downloaded_path(doc.basefile))
  
        # First paragraph of an RFC is always a header block 
        header = reader.readparagraph()
        # Preformatted is a ferenda.elements class representing a
        # block of preformatted text. It is derived from the built-in
        # list type, and must thus be initialized with an iterable, in
        # this case a single-element list of strings. (Note: if you
        # try to initialize it with a string, because strings are
        # iterables as well, you'll end up with a list where each
        # character in the string is an element, which is not what you
        # want).
        preheader = Preformatted([header])
        # Doc.body is a ferenda.elements.Body class, which is also
        # is derived from list, so it has (amongst others) the append
        # method. We build our document by adding to this root
        # element.
        doc.body.append(preheader)
  
        # Second paragraph is always the title, and we don't include
        # this in the body of the document, since we'll add it to the
        # medata -- once is enough
        title = reader.readparagraph()
        
        # After that, just iterate over the document and guess what
        # everything is. TextReader.getiterator is useful for
        # iterating through a text in other chunks than single lines
        for para in reader.getiterator(reader.readparagraph):
            if is_heading(para):
                # Heading is yet another of these ferenda.elements
                # classes.
                doc.body.append(Heading([para]))
            elif is_pagebreak(para):
                # Just drop these remnants of a page-and-paper-based past
                pass
            else:
                # If we don't know that it's something else, it's a
                # preformatted section (the safest bet for RFC text).
                doc.body.append(Preformatted([para])) 

        # 2: Then we create metadata for the document and store it in
        # doc.meta (in this case using the convenience
        # ferenda.Describer class).

        desc = Describer(doc.meta, doc.uri)

        # Set the rdf:type of the document
        desc.rdftype(self.rdf_type)

        # Set the title we've captured as the dct:title of the document and 
        # specify that it is in English
        desc.value(self.ns['dct'].title, util.normalize_space(title), lang="en")

        # Construct the dct:identifier (eg "RFC 6991") for this document from the basefile
        desc.value(self.ns['dct'].identifier, "RFC " + doc.basefile)
  
        # find and convert the publication date in the header to a datetime 
        # object, and set it as the dct:issued date for the document   
        re_date = re.compile("(January|February|March|April|May|June|July|August|September|October|November|December) (\d{4})").search
        # This is a context manager that temporarily sets the system
        # locale to the "C" locale in order to be able to use strptime
        # with a string on the form "August 2013", even though the
        # system may use another locale.
        dt_match = re_date(header)
        if dt_match:
            with util.c_locale(): 
                dt = datetime.strptime(re_date(header).group(0), "%B %Y")
            pubdate = date(dt.year,dt.month,dt.day)
            # Note that using some python types (cf. datetime.date)
            # results in a datatyped RDF literal, ie in this case
            #   <http://localhost:8000/res/rfc/6994> dct:issued "2013-08-01"^^xsd:date
            desc.value(self.ns['dct'].issued, pubdate)
  
        # find any older RFCs that this document updates or obsoletes
        obsoletes = re.search("^Obsoletes: ([\d+, ]+)", header, re.MULTILINE)
        updates = re.search("^Updates: ([\d+, ]+)", header, re.MULTILINE)

        # Find the category of this RFC, store it as dct:subject
        cat_match = re.search("^Category: ([\w ]+?)(  |$)", header, re.MULTILINE)
        if cat_match:
            desc.value(self.ns['dct'].subject, cat_match.group(1))
            
        for predicate, matches in ((self.ns['rfc'].updates, updates),
                                   (self.ns['rfc'].obsoletes, obsoletes)):
            if matches is None:
                continue
            # add references between this document and these older rfcs, 
            # using either rfc:updates or rfc:obsoletes
            for match in matches.group(1).strip().split(", "):
                uri = self.canonical_uri(match)
                # Note that this uses our own unofficial
                # namespace/vocabulary
                # http://example.org/ontology/rfc/
                desc.rel(predicate, uri)
  
        # And now we're done. We don't need to return anything as
        # we've modified the Document object that was passed to
        # us. The calling code will serialize this modified object to
        # XHTML and RDF and store it on disk

This implementation builds a very simple object model of a RFC document, which is serialized to a XHTML1.1+RDFa document by the managedparsing() decorator. If you run it (by calling ferenda-build.py rfc parse --all) after having downloaded the rfc documents, the result will be a set of documents in data/rfc/parsed, and a set of RDF files in data/rfc/distilled. Take a look at them! The above might appear to be a lot of code, but it also accomplishes much. Furthermore, it should be obvious how to extend it, for instance to create more metadata from the fields in the header (such as capturing the RFC category, the publishing party, the authors etc) and better semantic representation of the body (such as marking up regular paragraphs, line drawings, bulleted lists, definition lists, EBNF definitions and so on).

Next up, we’ll extend this implementation in two ways: First by representing the nested nature of the sections and subsections in the documents, secondly by finding and linking citations/references to other parts of the text or other RFCs in full.

        from ferenda.elements import Section, Subsection, Subsubsection

        # More heuristic rules: Section headers start at the beginning
        # of a line and are numbered. Subsections and subsubsections
        # have dotted numbers, optionally with a trailing period, ie
        # '9.2.' or '11.3.1'
        def is_section(p):
            return re.match(r"\d+\.? +[A-Z]", p)

        def is_subsection(p):
            return re.match(r"\d+\.\d+\.? +[A-Z]", p)

        def is_subsubsection(p):
            return re.match(r"\d+\.\d+\.\d+\.? +[A-Z]", p)

        def split_sectionheader(p):
            # returns a tuple of title, ordinal, identifier
            ordinal, title = p.split(" ",1)
            ordinal = ordinal.strip(".")
            return title.strip(), ordinal, "RFC %s, section %s" % (doc.basefile, ordinal)

        # Use a list as a simple stack to keep track of the nesting
        # depth of a document. Every time we create a Section,
        # Subsection or Subsubsection object, we push it onto the
        # stack (and clear the stack down to the appropriate nesting
        # depth). Every time we create some other object, we append it
        # to whatever object is at the top of the stack. As your rules
        # for representing the nesting of structure become more
        # complicated, you might want to use the
        # :class:`~ferenda.FSMParser` class, which lets you define
        # heuristic rules (recognizers), states and transitions, and
        # takes care of putting your structure together.
        stack = [doc.body]

        for para in reader.getiterator(reader.readparagraph):
            if is_section(para):
                title, ordinal, identifier = split_sectionheader(para)
                s = Section(title=title, ordinal=ordinal, identifier=identifier)
                stack[1:] = [] # clear all but bottom element
                stack[0].append(s) # add new section to body
                stack.append(s)    # push new section on top of stack
            elif is_subsection(para):
                title, ordinal, identifier = split_sectionheader(para)
                s = Subsection(title=title, ordinal=ordinal, identifier=identifier)
                stack[2:] = [] # clear all but bottom two elements
                stack[1].append(s) # add new subsection to current section
                stack.append(s)
            elif is_subsubsection(para):
                title, ordinal, identifier = split_sectionheader(para)
                s = Subsubsection(title=title, ordinal=ordinal, identifier=identifier)
                stack[3:] = [] # clear all but bottom three
                stack[-1].append(s) # add new subsubsection to current subsection
                stack.append(s)
            elif is_heading(para):
                stack[-1].append(Heading([para]))
            elif is_pagebreak(para):
                pass
            else:
                pre = Preformatted([para])
                stack[-1].append(pre)

<h1>2.  Overview</h1>
<h1>2.1.  Date, Location, and Participants</h1>
<pre>
   The second ForCES interoperability test meeting was held by the IETF
   ForCES Working Group on February 24-25, 2011...
</pre>
<h1>2.2.  Testbed Configuration</h1>
<h1>2.2.1.  Participants' Access</h1>
<pre>
   NTT and ZJSU were physically present for the testing at the Internet
  Technology Lab (ITL) at Zhejiang Gongshang University in China.
</pre>

<div class="section" property="dct:title" content=" Overview"
     typeof="bibo:DocumentPart" about="http://localhost:8000/res/rfc/6984#S2.">
  <span property="bibo:chapter" content="2."
        about="http://localhost:8000/res/rfc/6984#S2."/>
  <div class="subsection" property="dct:title" content=" Date, Location, and Participants"
       typeof="bibo:DocumentPart" about="http://localhost:8000/res/rfc/6984#S2.1.">
    <span property="bibo:chapter" content="2.1."
          about="http://localhost:8000/res/rfc/6984#S2.1."/>
    <pre>
      The second ForCES interoperability test meeting was held by the
      IETF ForCES Working Group on February 24-25, 2011...
    </pre>
    <div class="subsection" property="dct:title" content=" Testbed Configuration"
         typeof="bibo:DocumentPart" about="http://localhost:8000/res/rfc/6984#S2.2.">
      <span property="bibo:chapter" content="2.2."
            about="http://localhost:8000/res/rfc/6984#S2.2."/>
      <div class="subsubsection" property="dct:title" content=" Participants' Access"
           typeof="bibo:DocumentPart" about="http://localhost:8000/res/rfc/6984#S2.2.1.">
        <span content="2.2.1." about="http://localhost:8000/res/rfc/6984#S2.2.1."
              property="bibo:chapter"/>
        <pre>
          NTT and ZJSU were physically present for the testing at the
          Internet Technology Lab (ITL) at Zhejiang Gongshang
          University in China...
        </pre>
      </div>
    </div>
  </div>
</div>

        from pyparsing import Word, CaselessLiteral, nums
        section_citation = (CaselessLiteral("section") + Word(nums+".").setResultsName("Sec")).setResultsName("SecRef")
        rfc_citation = ("[RFC" + Word(nums).setResultsName("RFC") + "]").setResultsName("RFCRef")
        section_rfc_citation = (section_citation + "of" + rfc_citation).setResultsName("SecRFCRef")

        def rfc_uriformatter(parts):
            uri = ""
            if 'RFC' in parts:
                 uri += self.canonical_uri(parts['RFC'].lstrip("0"))
            if 'Sec' in parts:
                 uri += "#S" + parts['Sec']
            return uri

        from ferenda import CitationParser, URIFormatter
        citparser = CitationParser(section_rfc_citation, 
                                   section_citation,
                                   rfc_citation)
        citparser.set_formatter(URIFormatter(("SecRFCRef", rfc_uriformatter),
                                             ("SecRef", rfc_uriformatter),
                                             ("RFCRef", rfc_uriformatter)))
        citparser.parse_recursive(doc.body)

<pre>
   The behavior recommended in Section 2.5 is in line with generic error
   treatment during the IKE_SA_INIT exchange, per Section 2.21.1 of
   [RFC5996].
</pre>

<pre>
   The behavior recommended in <a href="#S2.5"
   rel="dct:references">Section 2.5</a> is in line with generic
   error treatment during the IKE_SA_INIT exchange, per <a
   href="http://localhost:8000/res/rfc/5996#S2.21.1"
   rel="dct:references">Section 2.21.1 of [RFC5996]</a>.
</pre>

Calling relate()¶

The purpose of the relate() method is to make sure that all document data and metadata is properly stored and indexed, so that it can be easily retrieved in later steps. This consists of three steps: Loading all RDF metadata into a triplestore, loading all document content into a full text index, and making note of how documents refer to each other.

Since the output of parse is well structured XHTML+RDFa documents that, on the surface level, do not differ much from docrepo to docrepo, you should not have to change anything about this step.

./ferenda-build.py rfc relate --all --fulltextindex=False

Calling makeresources()¶

This method needs to run at some point before generate and the rest of the methods. Unlike the other methods described above and below, which are run for one docrepo at a time, this method is run for the project as a whole (that is why it is a function in ferenda.manager instead of a DocumentRepository method). It constructs a set of site-wide resources such as minified js and css files, and configuration for the site-wide XSLT template. It is easy to run using the command-line tool:

$ ./ferenda-build.py all makeresources

If you use the API, you need to provide a list of instances of the docrepos that you’re using, and the path to where generated resources should be stored:

from ferenda.manager import makeresources
config = {'datadir':'mydata'}
myrepos = [RFC(**config), W3C(**config]
makeresources(myrepos,'mydata/myresources')

Customizing generate()¶

The purpose of the generate() method is to create new browser-ready HTML files from the structured XHTML+RDFa files created by parse(). Unlike the files created by parse(), these files will contain site-branded headers, footers, navigation menus and such. They will also contain related content not directly found in the parsed files themselves: Sectioned documents will have a automatically-generated table of contents, and other documents that refer to a particular document will be listed in a sidebar in that document. If the references are made to individual sections, there will be sidebars for all such referenced sections.

The default implementation does this in two steps. In the first, prep_annotation_file() fetches metadata about other documents that relates to the document to be generated into an annotation file. In the second, Transformer runs an XSLT transformation on the source file (which sources the annotation file and a configuration file created by makeresources()) in order to create the browser-ready HTML file.

You should not need to override the general generate() method, but you might want to control how the annotation file and the XSLT transformation is done.

    sparql_annotations = "rfc-annotations.rq"

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX bibo: <http://purl.org/ontology/bibo/>
PREFIX rfc: <http://example.org/ontology/rfc/>

CONSTRUCT {?s ?p ?o .
           <%(uri)s> rfc:isObsoletedBy ?obsoleter .
	   <%(uri)s> rfc:isUpdatedBy ?updater .
	   <%(uri)s> dct:isReferencedBy ?referencer .
	  }
WHERE
{
   # get all literal metadata where the document is the subject
   { ?s ?p ?o .
     # FILTER(strstarts(str(?s), "%(uri)s"))
     FILTER(?s = <%(uri)s> && !isUri(?o))
   }
   UNION
   # get all metadata (except unrelated dct:references) about
   #  resources that dct:references the document or any of its
   #  sub-resources.
   { ?s dct:references+ <%(uri)s> ;
        ?p ?o .
     BIND(?s as ?referencer)
     FILTER(?p != dct:references || strstarts(str(?o), "%(uri)s"))
   }
   UNION
   # get all metadata (except dct:references) about any resource that
   # rfc:updates or rfc:obsolets the document
   { ?s ?x <%(uri)s> ;
        ?p ?o .
     FILTER(?x in (rfc:updates, rfc:obsoletes) && ?p != dct:references)
   }
   # finally, bind obsoleting and updating resources to new variables for
   # use in the CONSTRUCT clause
   UNION { ?obsoleter rfc:obsoletes <%(uri)s> . }
   UNION { ?updater   rfc:updates   <%(uri)s> . }
}

<graph xmlns:dct="http://purl.org/dc/terms/"
       xmlns:rfc="http://example.org/ontology/rfc/"
       xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <resource uri="http://localhost:8000/res/rfc/6021">
    <rfc:isObsoletedBy ref="http://localhost:8000/res/rfc/6991"/>
    <dct:published fmt="datatype">
      <date xmlns="http://www.w3.org/2001/XMLSchema#">2010-10-01</date>
    </dct:published>
    <dct:title xml:lang="en">Common YANG Data Types</dct:title>
  </resource>
  <resource uri="http://localhost:8000/res/rfc/6991">
    <a><rfc:RFC/></a>
    <rfc:obsoletes ref="http://localhost:8000/res/rfc/6021"/>
    <dct:published fmt="datatype">
      <date xmlns="http://www.w3.org/2001/XMLSchema#">2013-07-01</date>
    </dct:published>
    <dct:title xml:lang="en">Common YANG Data Types</dct:title>
  </resource>
</graph>

    xslt_template = "rfc.xsl"

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
		xmlns:xhtml="http://www.w3.org/1999/xhtml"
		xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
		xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
		xmlns:dct="http://purl.org/dc/terms/"
		xmlns:rfc="http://example.org/ontology/rfc/"
		xml:space="preserve"
		exclude-result-prefixes="xhtml rdf">

  <xsl:include href="base.xsl"/>

  <!-- Implementations of templates called by base.xsl -->
  <xsl:template name="headtitle"><xsl:value-of select="//xhtml:title"/> | <xsl:value-of select="$configuration/sitename"/></xsl:template>
  <xsl:template name="metarobots"/>
  <xsl:template name="linkalternate"/>
  <xsl:template name="headmetadata"/>
  <xsl:template name="bodyclass">rfc</xsl:template>
  <xsl:template name="pagetitle">
    <h1><xsl:value-of select="../xhtml:head/xhtml:title"/></h1>
  </xsl:template>

  <xsl:template match="xhtml:a"><a href="{@href}"><xsl:value-of select="."/></a></xsl:template>

  <xsl:template match="xhtml:pre[1]">
    <pre><xsl:apply-templates/>
    </pre>
    <xsl:if test="count(ancestor::*) = 2">
      <xsl:call-template name="aside-annotations">
	<xsl:with-param name="uri" select="../@about"/>
      </xsl:call-template>
    </xsl:if>
  </xsl:template>

  <!-- everything that has an @about attribute, i.e. _is_ something
       (with a URI) gets a <section> with an <aside> for inbound links etc -->
  <xsl:template match="xhtml:div[@about]">
    
    <div class="section-wrapper" about="{@about}"><!-- needed? -->
      <section id="{substring-after(@about,'#')}">
	<xsl:variable name="sectionheading"><xsl:if test="xhtml:span[@property='bibo:chapter']/@content"><xsl:value-of select="xhtml:span[@property='bibo:chapter']/@content"/>. </xsl:if><xsl:value-of select="@content"/></xsl:variable>
	<xsl:if test="count(ancestor::*) = 2">
	    <h2><xsl:value-of select="$sectionheading"/></h2>
	</xsl:if>
	<xsl:if test="count(ancestor::*) = 3">
	  <h3><xsl:value-of select="$sectionheading"/></h3>
	</xsl:if>
	<xsl:if test="count(ancestor::*) = 4">
	  <h4><xsl:value-of select="$sectionheading"/></h4>
	</xsl:if>
	<xsl:apply-templates select="*[not(@about)]"/>
      </section>
      <xsl:call-template name="aside-annotations">
	<xsl:with-param name="uri" select="@about"/>
      </xsl:call-template>
    </div>
    <xsl:apply-templates select="xhtml:div[@about]"/>
  </xsl:template>

  <!-- remove spans which only purpose is to contain RDFa data -->
  <xsl:template match="xhtml:span[@property and @content and not(text())]"/>
  
  <!-- construct the side navigation -->
  <xsl:template match="xhtml:div[@about]" mode="toc">
    <li><a href="#{substring-after(@about,'#')}"><xsl:if test="xhtml:span/@content"><xsl:value-of select="xhtml:span[@property='bibo:chapter']/@content"/>. </xsl:if><xsl:value-of select="@content"/></a><xsl:if test="xhtml:div[@about]">
    <ul><xsl:apply-templates mode="toc"/></ul>
    </xsl:if></li>
  </xsl:template>

  <!-- named template called from other templates which match
       xhtml:div[@about] and pre[1] above, and which creates -->
  <xsl:template name="aside-annotations">
    <xsl:param name="uri"/>
    <xsl:if test="$annotations/resource[@uri=$uri]/dct:isReferencedBy">
      <aside class="annotations">
	<h2>References to <xsl:value-of select="$annotations/resource[@uri=$uri]/dct:identifier"/></h2>
	<xsl:for-each select="$annotations/resource[@uri=$uri]/rfc:isObsoletedBy">
	  <xsl:variable name="referencing" select="@ref"/>
	  Obsoleted by
	  <a href="{@ref}">
	    <xsl:value-of select="$annotations/resource[@uri=$referencing]/dct:identifier"/>
	  </a><br/>
	</xsl:for-each>
	<xsl:for-each select="$annotations/resource[@uri=$uri]/rfc:isUpdatedBy">
	  <xsl:variable name="referencing" select="@ref"/>
	  Updated by
	  <a href="{@ref}">
	    <xsl:value-of select="$annotations/resource[@uri=$referencing]/dct:identifier"/>
	  </a><br/>
	</xsl:for-each>
	<xsl:for-each select="$annotations/resource[@uri=$uri]/dct:isReferencedBy">
	  <xsl:variable name="referencing" select="@ref"/>
	  Referenced by
	  <a href="{@ref}">
	    <xsl:value-of select="$annotations/resource[@uri=$referencing]/dct:identifier"/>
	  </a><br/>
	</xsl:for-each>
      </aside>
    </xsl:if>
  </xsl:template>

  <!-- default template: translate everything from whatever namespace
       it's in (usually the XHTML1.1 NS) into the default namespace
       -->
  <xsl:template match="*"><xsl:element name="{local-name(.)}"><xsl:apply-templates select="node()"/></xsl:element></xsl:template>

  <!-- default template for toc handling: do nothing -->
  <xsl:template match="@*|node()" mode="toc"/>
  
</xsl:stylesheet>

Customizing toc()¶

The purpose of the toc() method is to create a set of pages that acts as tables of contents for all documents in your docrepo. For large document collections there are often several different ways of creating such tables, eg. sorted by title, publication date, document status, author and similar. The pages uses the same site-branding,headers, footers, navigation menus etc used by generate().

The default implementation is generic enough to handle most cases, but you’ll have to override other methods which it calls, primarily toc_query(), toc_criteria() and toc_predicates(). These methods all depend on the metadata you’ve created by your parse implementation, but in the simplest cases it’s enough to specify that you want one set of pages organized by the dct:title of each document (alphabetically sorted) and another by dct:issued (numerically/calendarically sorted). The default implementation does exactly this.

In our case, we wish to create four kinds of sorting: By identifier (RFC number), by date of issue, by title and by category. These map directly to four kinds of metadata that we’ve stored about each and every document. By overriding toc_predicates() we can specify these four predicates.

    def toc_predicates(self):
        return [self.ns['dct'].title,
                self.ns['dct'].issued,
                self.ns['dct'].subject,
                self.ns['dct'].identifier]

After running toc with this change, you can see that four sets of index pages are created. However, except for publication year, the actual partitioning of documents are still done by title. In order to correct this, we must create a set of TocCriteria objects that specify how documents should be ordered for any particular criteria, and have toc_criteria() return these.

    def toc_criteria(self, predicates=None):
        from ferenda import TocCriteria

        return [TocCriteria(binding='identifier',
                            label='Sorted by RFC #',
                            pagetitle='RFCs %(select)s--99',
                            selector=lambda x: x['identifier'][4:-2]+"00",  # "RFC 6998" => "69"
                            key=lambda x: int(x['identifier'][4:]),
                            selector_descending=True,
                            key_descending=True),   # "RFC 6998" => 6998

                TocCriteria(binding='title',
                            label='Sorted by title',
                            pagetitle='Documents starting with "%(select)s"',
                            selector=lambda x: util.title_sortkey(x['title'])[0], # "The 'view-state'" property => "v"
                            key=lambda x: util.title_sortkey(x['title'])),

                TocCriteria(binding='issued',
                            label='Sorted by year',
                            pagetitle='Documents published in %(select)s',
                            selector = lambda x: x['issued'][:4],  # '2013-08-01' => '2013'
                            key = lambda x: x['issued'],
                            selector_descending=True,
                            key_descending=True), 

                TocCriteria(binding='subject',
                            label='Sorted by category',
                            pagetitle='Documents in the %(select)s category',
                            selector = lambda x: x['subject'],
                            key = lambda x: int(x['identifier'][4:]),
                            key_descending=True
                            )]

The above code gives some example of how TocCriteria objects can be configured. However, a TocCriteria object does not control how each individual document is listed on a toc page. The default formatting just lists the title of the document, linked to the document in question. For RFCs, who mainly is referenced using their RFC number rather than their title, we’d like to add the RFC number in this display. This is done by overriding toc_item().

    def toc_item(self, binding, row):
        from ferenda.elements import Link
        return [row['identifier'] + ": ",
                Link(row['title'], 
                     uri=row['uri'])]

Se also Customizing the table(s) of content.

Customizing news()¶

The purpose of news(), the next to final step, is to provide a set of news feeds for your document repository.

The default implementation gives you one single news feed for all documents in your docrepo, and creates both browser-ready HTML (using the same headers, footers, navigation menus etc used by generate()) and Atom syndication format files.

You can specify some basic criteria similar to the way you specified the organization of your TOC pages, since you might want to split up the documents in different feeds, for example one feed for each RFC track.

    def news_criteria(self):
        from ferenda import Describer, NewsCriteria
        from rdflib import Graph
        
        # function that returns a closure, which acts as a custom
        # selector function for the NewsCriteria objects.
        def selector_for(category):
            def selector(entry):
                graph = Graph()
                graph.parse(self.store.distilled_path(entry.basefile))
                desc = Describer(graph, entry.id)
                return desc.getvalue(self.ns['dct'].subject) == category
            return selector
            
        return [NewsCriteria('all','All RFCs'),
                NewsCriteria('informational', 'Informational RFCs',
                             selector=selector_for("Informational")),
                NewsCriteria('bcp', 'Best Current Practice RFCs',
                             selector=selector_for("Best Current Practice")),
                NewsCriteria('experimental', 'Experimental RFCs',
                             selector=selector_for("Experimental")),
                NewsCriteria('standards', 'Standards Track RFCs',
                             selector=selector_for("Standards Track"))]

When running news, this will create five different atom feeds (which are mirrored as HTML pages) under data/rfc/news: One containing all documents, and four others that contain documents in a particular category.

Se also Customizing the news feeds.

Customizing frontpage()¶

Finally, frontpage() creates a front page for your entire site with content from the different docrepos. Each docrepos frontpage_content() method will be called, and should return a XHTML fragment with information about the repository and it’s content. Below is a simple example that uses functionality we’ve used in other contexts to create a list of the five latest documents, as well as a total count of documents.

    def frontpage_content(self, primary=False):
        from rdflib import URIRef, Graph
        from itertools import islice
        items = ""
        for entry in islice(self.news_entries(),5):
            graph = Graph()
            with self.store.open_distilled(entry.basefile) as fp:
                graph.parse(data=fp.read())
            data = {'identifier': graph.value(URIRef(entry.id), self.ns['dct'].identifier).toPython(),
                    'uri': entry.id,
                    'title': entry.title}
            items += '<li>%(identifier)s <a href="%(uri)s">%(title)s</a></li>' % data
        return ("""<h2><a href="%(uri)s">Request for comments</a></h2>
                   <p>A complete archive of RFCs in Linked Data form. Contains %(doccount)s documents.</p>
                   <p>Latest 5 documents:</p>
                   <ul>
                      %(items)s
                   </ul>""" % {'uri':self.dataset_uri(),
                               'items': items,
                               'doccount': len(list(self.store.list_basefiles_for("_postgenerate")))})

Next steps¶

When you have written code and customized downloading, parsing and all the other steps, you’ll want to run all these steps for all your docrepos in a single command by using the special value all for docrepo, and again all for action:

./ferenda-build.py all all

By now, you should have a basic idea about the key concepts of ferenda. In the next section, Key concepts, we’ll explore them further.

Project¶

A collection of docrepos and configuration that is used to make a useful web site. The first step in creating a project is running ferenda-setup <projectname>.

A project is primarily defined by its configuration file at <projectname>/ferenda.ini, which specifies which docrepos are used, and settings for them as well as settings for the entire project.

A project is managed using the ferenda-build.py tool.

If using the API instead of these command line tools, there is no concept of a project except for what your code provides. Your client code is responsible for creating the docrepo classes and providing them with proper settings. These can be loaded from a ferenda.ini-style file (the LayeredConfig class makes this simple), be hard-coded, or handled in any other way you see fit.

Configuration¶

A ferenda docrepo object can be configured in two ways - either when creating the object, eg:

d = DocumentSource(datadir="mydata", loglevel="DEBUG",force=True)

Or it can be configured using the LayeredConfig class, which takes configuration data from three places:

• built-in configuration values (provided by get_default_options())
• values from a configuration file (normally ferenda.ini”, placed alongside ferenda-build.py)
• command-line parameters, eg --force --datadir=mydata

d = DocumentSource()
d.config = LayeredConfig(defaults=d.get_default_options(),
                         inifile="ferenda.ini",
                         commandline=sys.argv)

(This is what ferenda-build.py does behind the scenes)

Configuration values from the configuration file overrides built-in configuration values, and command line parameters override configuration file values.

By setting the config property, you override any parameters provided when creating the object.

These are the normal configuration options:

DocumentRepository¶

A document repository (docrepo for short) is a class that handles all aspects of a document collection: Downloading the documents (or aquiring them in some other way), parsing them into structured documents, and then re-generating HTML documents with added niceties, for example references from documents from other docrepos.

You add support for a new collection of documents by subclassing DocumentRepository. For more details, see Creating your own document repositories

Document¶

A Document is the main unit of information in Ferenda. A document is primarily represented in serialized form as a XHTML 1.1 file with embedded metadata in RDFa format, and in code by the Document class. The class has five properties:

• meta (a RDFLib Graph)
• body (a tree of building blocks, normally instances of ferenda.elements classes, representing the structure and content of the document)
• lang (an IETF language tag, eg sv or en-GB)
• uri (a string representing the canonical URI for this document)
• basefile (a short internal id)

The method render_xhtml() (which is called automatically, as long as your parse method use the managedparsing() decorator) renders a Document object into a XHTML 1.1+RDFa document.

Identifiers¶

Documents, and parts of documents, in ferenda have a couple of different identifiers, and it’s useful to understand the difference and relation between them.

• basefile: The internal id for a document. This is is internal to the document repository and is used as the base for the filenames for the stored files . The basefile isn’t totally random and is expected to have some relationship with a human-readable identifier for the document. As an example from the RFC docrepo, the basefile for RFC 1147 would simply be “1147”. By the rules encoded in DocumentStore, this results in the downloaded file rfc/downloads/1147.txt, the parsed file rfc/parsed/1147.xhtml and the generated file rfc/generated/1147.html. Only documents themselves, not parts of documents, have basefile identifiers.
• uri: The canonical URI for a document or a part of a document (generally speaking, a resource). This identifier is used when storing data related to the resource in a triple store and a fulltext search engine, and is also used as the external URL for the document when republishing (see The WSGI app and also Document URI). URI:s for documents can be set by settings the uri property of the Document object. URIs for parts of documents are set by setting the uri property on any elements based object in the body tree. When rendering the document into XHTML, render_xhtml creates RDFa statements based on this property and the meta property.
• dct:identifier: The human readable identifier for a document or a part of a document. If the document has an established human-readable identifier, such as “RFC 1147” or “2003/98/EC” (The EU directive on the re-use of public sector information), the dct:identifier is used for this. Unlike basefile and uri, this identifier isn’t set directly as a property on an object. Instead, you add a triple with dct:identifier as the predicate to the object’s meta property, see Parsing and representing document metadata and also DCMI Terms.

DocumentEntry¶

Apart from information about what a document contains, there is also information about how it has been handled, such as when a document was first downloaded or updated from a remote source, the URL from where it came, and when it was made available through Ferenda. .This information is encapsulated in the DocumentEntry class. Such objects are created and updated by various methods in DocumentRepository. The objects are persisted to JSON files, stored alongside the documents themselves, and are used by the news() method in order to create valid Atom feeds.

File storage¶

During the course of processing, data about each individual document is stored in many different files in various formats. The DocumentStore class handles most aspects of this file handling. A configured DocumentStore object is available as the store property on any DocumentRepository object.

Example: If a created docrepo object d has the alias foo, and handles a document with the basefile identifier bar, data about the document is then stored:

• When downloaded, the original data as retrieved from the remote server, is stored as data/foo/downloaded/bar.html, as determined by d.store.downloaded_path()
• At the same time, a DocumentEntry object is serialized as data/foo/entries/bar.json, as determined by d.store.documententry_path()
• If the downloaded source needs to be transformed into some intermediate format before parsing (which is the case for eg. PDF or Word documents), the intermediate data is stored as data/foo/intermediate/bar.xml, as determined by d.store.intermediate_path()
• When the downloaded data has been parsed, the parsed XHTML+RDFa document is stored as data/foo/parsed/bar.xhtml, as determined by d.store.parsed_path()
• From the parsed document is automatically destilled a RDF/XML file containing all RDFa statements from the parsed file, which is stored as data/foo/distilled/bar.rdf, as determined by d.store. data/foo/annotations/bar.grit.txt, as determined by d.store.annotation_path().
• During the relate step, all documents which are referred to by any other document are marked as dependencies of that document. If the bar document is dependent on another document, then this dependency is recorded in a dependency file stored at data/foo/deps/bar.txt, as determined by d.store.dependencies_path().
• Just prior to the generation of browser-ready HTML5 files, all metadata in the system as a whole which is relevant to bar is serialized in an annotation file in GRIT/XML format at data/foo/annotations/bar.grit.txt, as determined by d.store.annotation_path().
• Finally, the generated HTML5 file is created at data/foo/generated/bar.html, as determined by d.store.generated_path(). (This step also updates the serialized DocumentEntry object described above)

        path = self.store.downloaded_path(basefile)
        with open(path, mode="wb") as fp:
            fp.write(b"...")

        with self.store.open_downloaded(path, mode="wb") as fp:
            fp.write(b"...")

from __future__ import unicode_literals
# begin part-1
class TestDocrepo(DocumentRepository):

    storage_policy = "dir"
    
    def download_single(self, basefile):
        mainurl = self.document_url_template % {'basefile': basefile}
        self.download_if_needed(basefile, mainurl)
        with self.store.open_downloaded(basefile) as fp:
            soup = BeautifulSoup(fp.read())
        for img in soup.find_all("img"):
            imgurl = urljoin(mainurl, img["src"])
# begin part-2
            # open eg. data/foo/downloaded/bar/hello.jpg for writing
            with self.store.open_downloaded(basefile,
                                            attachment=img["src"],
                                            mode="wb") as fp:

Document URI¶

In order to create these metadata statements, you should first create a suitable URI for your document. Preferably, this should be a URI based on the URL where your web site will be published, ie if you plan on publishing it on http://mynetstandards.org/, a URI for RFC 4711 might be http://mynetstandards.org/res/rfc/4711 (ie based on the base URL, the docrepo alias, and the basefile). By changing the url variable in your project configuration file, you can set the base URL from which all document URIs are derived. If you wish to have more control over the exact way URIs are constructed, you can override canonical_uri().

The URI for any document is available as a uri property.

Adding metadata using the RDFLib API¶

With this, you can create metadata for your document using the RDFLib Graph API.

    # Simpler way                   
    def parse_metadata_from_soup(self, soup, doc):
        from ferenda import Describer
        from datetime import datetime
        title = "My Document title"
        authors = ["Fred Bloggs", "Joe Shmoe"]
        identifier = "Docno 2013:4711"
        pubdate = datetime(2013,1,6,10,8,0)
        d = Describer(doc.meta, doc.uri)
        d.rdftype(self.rdf_type)
        d.value(self.ns['prov'].wasGeneratedBy, self.qualified_class_name())
        d.value(self.ns['dct'].title, title, lang=doc.lang)
        d.value(self.ns['dct'].identifier, identifier)
        for author in authors:
            d.value(self.ns['dct'].author, author)

A simpler way of adding metadata¶

The default RDFLib graph API is somewhat cumbersome for adding triples to a metadata graph. Ferenda has a convenience wrapper, Describer (itself a subclass of rdflib.extras.describer.Describer) that makes this somewhat easier. The ns class property also contains a number of references to popular vocabularies. The above can be made more succint like this:

    # Simpler way                   
    def parse_metadata_from_soup(self, soup, doc):
        from ferenda import Describer
        from datetime import datetime
        title = "My Document title"
        authors = ["Fred Bloggs", "Joe Shmoe"]
        identifier = "Docno 2013:4711"
        pubdate = datetime(2013,1,6,10,8,0)
        d = Describer(doc.meta, doc.uri)
        d.rdftype(self.rdf_type)
        d.value(self.ns['prov'].wasGeneratedBy, self.qualified_class_name())
        d.value(self.ns['dct'].title, title, lang=doc.lang)
        d.value(self.ns['dct'].identifier, identifier)
        for author in authors:
            d.value(self.ns['dct'].author, author)

Vocabularies¶

Each RDF vocabulary is defined by a URI, and all terms (types and properties) of that vocabulary is typically directly derived from it. The vocabulary URI therefore acts as a namespace. Like namespaces in XML, a shorter prefix is often assigned to the namespace so that one can use rdf:type rather than http://www.w3.org/1999/02/22-rdf-syntax-ns#type. The DocumentRepository object keeps a dictionary of common (prefix,namespace)s in the class property ns – your code can modify this list in order to add vocabulary terms relevant for your documents.

Serialization of metadata¶

The render_xhtml() method serializes all information in doc.body and doc.meta to a XHTML+RDFa file (the exact location given by parsed_path()). The metadata specified by doc.meta ends up in the <head> section of this XHTML file.

The actual RDF statements are also distilled to a separate RDF/XML file found alongside this file (the location given by distilled_path()) for convenience.

Metadata about parts of the document¶

Just like the main Document object, individual parts of the document (represented as ferenda.elements objects) can have uri and meta properties. Unlike the main Document objects, these properties are not initialized beforehand. But if you do create these properties, they are used to serialize metadata into RDFa properties for each

    def parse_document_from_soup(self, soup, doc):
        from ferenda.elements import Page
        from ferenda import Describer
        part = Page(["This is a part of a document"],
                    ordinal=42,
                    uri="http://example.org/doc#42",
                    meta=self.make_graph())
        d = Describer(part.meta, part.uri)
        d.rdftype(self.ns['bibo'].DocumentPart)
        # the dct:identifier for a document part is often whatever
        # would be the preferred way to cite that part in another
        # document
        d.value(self.ns['dct'].identifier, "Doc:4711, p 42")

This results in the following document fragment:

<div xmlns="http://www.w3.org/1999/xhtml"
     about="http://example.org/doc#42"
     typeof="bibo:DocumentPart"
     class="page">
  <span property="dct:identifier"
	content="Doc:4711, p 42"
	xml:lang=""/>
     This is a part of a document
</div>

Creating your own element classes¶

The exact structure of documents differ greatly. A general document format such as XHTML or ODF cannot contain special constructs for preamble recitals of EC directives or patent claims of US patents. But your own code can create new classes for this. Example:

from ferenda.elements import CompoundElement, OrdinalElement

class Preamble(CompoundElement): pass
class PreambleRecital(CompoundElement,OrdinalElement):
    tagname = "div"
    rdftype = "eurlex:PreambleRecital"

doc = Preamble([PreambleRecital("Un",ordinal=1)],
               [PreambleRecital("Deux",ordinal=2)],
               [PreambleRecital("Trois",ordinal=3)])

Mixin classes¶

As the above example shows, it’s possible and even recommended to use multiple inheritance to compose objects by subclassing two classes – one main class who’s semantics you’re extending, and one mixin class that contains particular properties. The following classes are useful as mixins:

• OrdinalElement: for representing elements with some sort of ordinal numbering. An ordinal element has an ordinal property, and different ordinal objects can be compared or sorted. The sort is based on the ordinal property. The ordinal property is a string, but comparisons/sorts are done in a natural way, i.e. “2” < “2 a” < “10”.
• TemporalElement: for representing things that has a start and/or a end date. A temporal element has an in_effect method which takes a date (or uses today’s date if none given) and returns true if that date falls between the start and end date.

Rendering to XHTML¶

The built-in classes are rendered as XHTML by the built-in method render_xhtml(), which first creates a <head> section containing all document-level metadata (i.e. the data you have specified in your documents meta property), and then calls the as_xhtml method on the root body element. The method is called with doc.uri as a single argument, which is then used as the RDF subject for all triples in the document (except for those sub-elements which themselves have a uri property)

All built-in element classes derive from AbstractElement, which contains a generic implementation of as_xhtml(), that recursively creates a lxml element tree from itself and it’s children.

Your own classes can specify how they are to be rendered in XHTML by overriding the tagname and classname properties, or for full control, the as_xhtml() method.

As an example, the class SectionalElement overrides as_xhtml to the effect that if you provide identifier, ordinal and title properties for the object, a resource URI is automatically constructed and four RDF triples are created (rdf:type, dct:title, dct:identifier, and bibo:chapter):

from ferenda.elements import SectionalElement
p = SectionalElement(["Some content"],
                     ordinal = "1a",
                     identifier = "Doc pt 1(a)",
                     title="Title or name of the part")
body = Body([p])
from lxml import etree               
etree.tostring(body.as_xhtml("http://example.org/doc"))

...which results in:

<body xmlns="http://www.w3.org/1999/xhtml" about="http://example.org/doc">
  <div about="http://example.org/doc#S1a"
       typeof="bibo:DocumentPart"
       property="dct:title"
       content="Title or name of the part"
       class="sectionalelement">
    <span about="http://example.org/doc#S1a"
	  property="dct:identifier"
	  content="Doc pt 1(a)" />
    <span about="http://example.org/doc#S1a"
	  property="bibo:chapter"
	  content="1a" />
    Some content
  </div>
</body>

However, this is a convenience method of SectionalElement, amd may not be appropriate for your needs. The general way of attaching metdata to document parts, as specified in Metadata about parts of the document, is to provide each document part with a uri and meta property. These are then automatically serialized as RDFa statements by the default as_xhtml implementation.

Convenience methods¶

Your element tree structure can be serialized to well-formed XML using the serialize() method. Such a serialization can be turned back into the same tree using deserialize(). This is primarily useful during debugging.

You might also find the as_plaintext method useful. It works similar to as_xhtml, but returns a plaintext string with the contents of an element, including all sub-elements

The ferenda.elements.html module contains the method elements_from_soup() which converts a BeautifulSoup tree into the equivalent tree of element objects.

FSMParser¶

The framework contains a class for creating such state machines, FSMParser. It is used with a set of the following objects:

You initialize the parser with the transition table (which contains the other objects), then call it’s parse() method with a iterator of chunks, an initial state, and an initial constructor. The result of parse is a nested document object tree.

A simple example¶

Consider a very simple document format that only has three kinds of structural elements: a normal paragraph, preformatted text, and sections. Each section has a title and may contain paragraphs or preformatted text, which in turn may not contain anything else. All chunks are separated by double newlines

The section is identified by a header, which is any single-line string followed by a line of = characters of the same length. Any time a new header is encountered, this signals the end of the current section:

This is a header
================

A preformatted section is any chunk where each line starts with at least two spaces:

# some example of preformatted text
def world(name):
    return "Hello", name

A paragraph is anything else:

This is a simple paragraph.
It can contain short lines and longer lines.

(You might recognize this format as a very simple form of ReStructuredText).

Recognizers for these three elements are easy to build:

from ferenda import elements, FSMParser

def is_section(parser):
    chunk = parser.reader.peek()
    lines = chunk.split("\n")
    return (len(lines) == 2 and
            len(lines[0]) == len(lines[1]) and
            lines[1] == "=" * len(lines[0]))

def is_preformatted(parser):
    chunk = parser.reader.peek()
    lines=chunk.split("\n")
    not_indented = lambda x: not x.startswith("  ")
    return len(list(filter(not_indented,lines))) == 0

def is_paragraph(parser):
    return True

The elements module contains ready-built classes which we can use to build our constructors:

def make_body(parser):
    b = elements.Body()
    return parser.make_children(b)

def make_section(parser):
    chunk = parser.reader.next()
    title = chunk.split("\n")[0]
    s = elements.Section(title=title)
    return parser.make_children(s)
setattr(make_section,'newstate','section')
    
def make_paragraph(parser):
    return elements.Paragraph([parser.reader.next()])

def make_preformatted(parser):
    return elements.Preformatted([parser.reader.next()])

Note that any constructor which may contain sub-elements must itself call the make_children() method of the parser. That method takes a parent object, and then repeatedly creates child objects which it attaches to that parent object, until a exit condition is met. Each call to create a child object may, in turn, call make_children (not so in this very simple example).

The final step in putting this together is defining the transition table, and then creating, configuring and running the parser:

transitions = {("body", is_section): (make_section, "section"),
               ("section", is_paragraph): (make_paragraph, None),
               ("section", is_preformatted): (make_preformatted, None),
               ("section", is_section): (False, None)}

text = """First section
=============

This is a regular paragraph. It will not be matched by is_section
(unlike the above chunk) or is_preformatted (unlike the below chunk),
but by the catch-all is_paragraph. The recognizers are run in the
order specified by FSMParser.set_transitions().

    This is a preformatted section.
        It could be used for source code,
    +-------------------+
    |   line drawings   |
    +-------------------+
        or what have                 you.

Second section
==============

The above new section implicitly closed the first section which we
were in. This was made explicit by the last transition rule, which
stated that any time a section is encountered while in the "section"
state, we should not create any more children (False) but instead
return to our previous state (which in this case is "body", but for a
more complex language could be any number of states)."""

p = FSMParser()
p.set_recognizers(is_section, is_preformatted, is_paragraph)
p.set_transitions(transitions)
p.initial_constructor = make_body
p.initial_state = "body"
body = p.parse(text.split("\n\n"))
# print(elements.serialize(body))

The result of this parse is the following document object tree (passed through serialize()):

<Body>
  <Section title="First section">
    <Paragraph>
      <str>This is a regular paragraph. It will not be matched by is_section
(unlike the above chunk) or is_preformatted (unlike the below chunk),
but by the catch-all is_paragraph. The recognizers are run in the
order specified by FSMParser.set_transitions().</str>
    </Paragraph><Preformatted>
      <str>    This is a preformatted section.
        It could be used for source code,
    +-------------------+
    |   line drawings   |
    +-------------------+
        or what have                 you.</str>
    </Preformatted>
  </Section>
  <Section title="Second section">
    <Paragraph>
      <str>The above new section implicitly closed the first section which we
were in. This was made explicit by the last transition rule, which
stated that any time a section is encountered while in the "section"
state, we should not create any more children (False) but instead
return to our previous state (which in this case is "body", but for a
more complex language could be any number of states).</str>
    </Paragraph>
  </Section>
</Body>

	

    

Writing complex parsers¶

Tips for debugging your parser¶

Two useful commands in the Devel module:

$ # sets debug, prints serialize(parser.parse(...))
$ ./ferenda-build.py devel fsmparse parser < chunks
$ # sets debug, returns name of matching function
$ ./ferenda-build.py devel fsmanalyze parser <currentstate> < chunk

The built-in solution¶

Ferenda uses the Pyparsing library in order to find and process citations. As an example, we’ll specify citation patterns and URI formats for references that occurr in RFC documents. These are primarily of three different kinds (examples come from RFC 2616):

1. URL references, eg “GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1”
2. IETF document references, eg “STD 3”, “BCP 14” and “RFC 2068”
3. Internal endnote references, eg “[47]” and “[33]”

We’d like to make sure that any URL reference gets turned into a link to that same URL, that any IETF document reference gets turned into the canonical URI for that document, and that internal endote references gets turned into document-relative links, eg “#endnote-47” and “#endnote-33”. (This requires that other parts of the parse() process has created IDs for these in doc.body, which we assume has been done).

Turning URL references in plain text into real links is so common that ferenda has built-in support for this. The support comes in two parts: First running a parser that detects URLs in the textual content, and secondly, for each match, running a URL formatter on the parse result.

At the end of your parse() method, do the following.

from ferenda import CitationParser
from ferenda import URIFormatter
import ferenda.citationpatterns
import ferenda.uriformats

# CitationParser is initialized with a list of pyparsing
# ParserElements (or any other object that has a scanString method
# that returns a generator of (tokens,start,end) tuples, where start
# and end are integer string indicies and tokens are dict-like
# objects)
citparser = CitationParser(ferenda.citationpatterns.url)

# URIFormatter is initialized with a list of tuples, where each
# tuple is a string (identifying a named ParseResult) and a function
# (that takes as a single argument a dict-like object and returns a
# URI string (possibly relative)
citparser.set_formatter(URIFormatter(("URLRef", ferenda.uriformats.url)))

citparser.parse_recursive(doc.body)

The parse_recursive() takes any elements document tree and modifies it in-place to mark up any references to proper Link objects.

Extending the built-in support¶

Building your own citation patterns and URI formats is fairly simple. First, specify your patterns in the form of a pyparsing parseExpression, and make sure that both the expression as a whole, and any individual significant properties, are named by calling .setResultName.

Then, create a set of formatting functions that takes the named properties from the parse expressions above and use them to create a URI.

Finally, initialize a CitationParser object from your parse expressions and a URIFormatter object that maps named parse expressions to their corresponding URI formatting function, and call parse_recursive()

from pyparsing import Word, nums

from ferenda import CitationParser
from ferenda import URIFormatter
import ferenda.citationpatterns
import ferenda.uriformats

# Create two ParserElements for IETF document references and internal
# references
rfc_citation = "RFC" + Word(nums).setResultsName("RFCRef")
bcp_citation = "BCP" + Word(nums).setResultsName("BCPRef")
std_citation = "STD" + Word(nums).setResultsName("STDRef")
ietf_doc_citation = (rfc_citation | bcp_citation | std_citation).setResultsName("IETFRef")

endnote_citation = ("[" + Word(nums).setResultsName("EndnoteID") + "]").setResultsName("EndnoteRef")

# Create a URI formatter for IETF documents (URI formatter for endnotes
# is so simple that we just use a lambda function below
def rfc_uri_formatter(parts):
    # parts is a dict-like object created from the named result parts
    # of our grammar, eg those ParserElement for which we've called
    # .setResultsName(), in this case eg. {'RFCRef':'2068'}

    # NOTE: If your document collection contains documents of this
    # type and you're republishing them, feel free to change these
    # URIs to URIs under your control,
    # eg. "http://mynetstandards.org/rfc/%(RFCRef)s/" and so on
    if 'RFCRef' in parts:
          return "http://www.ietf.org/rfc/rfc%(RFCRef)s.txt" % parts
    elif 'BCPRef' in parts:
          return "http://tools.ietf.org/rfc/bcp/bcp%(BCPRef)s.txt" % parts
    elif 'STDRef' in parts:
          return "http://rfc-editor.org/std/std%(STDRef)s.txt" % parts
    else:
          return None

# CitationParser is initialized with a list of pyparsing
# ParserElements (or any other object that has a scanString method
# that returns a generator of (tokens,start,end) tuples, where start
# and end are integer string indicies and tokens are dict-like
# objects)
citparser = CitationParser(ferenda.citationpatterns.url,
                           ietf_doc_citation,
                           endnote_citation)

# URIFormatter is initialized with a list of tuples, where each
# tuple is a string (identifying a named ParseResult) and a function
# (that takes as a single argument a dict-like object and returns a
# URI string (possibly relative)
citparser.set_formatter(URIFormatter(("url", ferenda.uriformats.url),
                                      ("IETFRef", rfc_uri_formatter),
                                      ("EndnoteRef", lambda d: "#endnote-%(EndnoteID)s" % d)))

citparser.parse_recursive(doc.body)

This turns this document

<body xmlns="http://www.w3.org/1999/xhtml" about="http://example.org/doc/">
  <h1>Main document</h1>
  <p>A naked URL: http://www.w3.org/pub/WWW/TheProject.html.</p>
  <p>Some IETF document references: See STD 3, BCP 14 and RFC 2068.</p>
  <p>An internal endnote reference: ...relevance ranking, cf. [47]</p>
  <h2>References</h2>
  <p id="endnote-47">47: Malmgren, Towards a theory of jurisprudential
    ranking</p>
</body>

Into this document:

<body xmlns="http://www.w3.org/1999/xhtml" about="http://example.org/doc/">
  <h1>Main document</h1>
  <p>
    A naked URL: <a href="http://www.w3.org/pub/WWW/TheProject.html"
		    rel="dct:references"
		    >http://www.w3.org/pub/WWW/TheProject.html</a>.
  </p>
  <p>
    Some IETF document references: See 
    <a href="http://rfc-editor.org/std/std3.txt"
       rel="dct:references">STD 3</a>,
    <a href="http://tools.ietf.org/rfc/bcp/bcp14.txt"
       rel="dct:references">BCP 14</a> and
    <a href="http://www.ietf.org/rfc/rfc2068.txt"
       rel="dct:references">RFC 2068</a>.
     </p>
  <p>
    An internal endnote reference: ...relevance ranking, cf.
    <a href="#endnote-47"
       rel="dct:references">[47]</a>
  </p>
  <h2>References</h2>
  <p id="endnote-47">47: Malmgren, Towards a theory of jurisprudential
    ranking</p>
</body>

Rolling your own¶

For more complicated situations you can skip calling parse_recursive() and instead do your own processing with the optional support of CitationParser.

This is needed in particular for complicated ParserElement objects which may contain several sub-ParserElement which needs to be turned into individual links. As an example, the text “under Article 56 (2), Article 57 or Article 100a of the Treaty establishing the European Community” may be matched by a single top-level ParseResult (and probably must be, if “Article 56 (2)” is to actually reference article 56(2) in the Treaty), but should be turned in to three separate links.

In those cases, iterate through your doc.body yourself, and for each text part do something like the following:

from ferenda import CitationParser, URIFormatter, citationpatterns, uriformats
from ferenda.elements import Link

citparser = CitationParser()
citparser.add_grammar(citationpatterns.url)
formatter = URIFormatter(("url", uriformats.url))

res = []
text = "An example: http://example.org/. That is all."

for node in citparser.parse_string(text):
    if isinstance(node,str):
        # non-linked text, add and continue
        res.append(node)
    if isinstance(node, tuple):
        (text, match) = node
        uri = formatter.format(match)
        if uri:
            res.append(Link(uri, text, rel="dct:references"))

Defining criteria for grouping and sorting¶

A criteria in this case is a method for grouping a set into documents into distinct categories, then sorting the documents, as well as the categories themseves.

Each criteria is represented by a TocCriteria object. If you want to customize the table of contents, you have to provide a set of these objects. You can do this in two different ways.

The first, simple, way is to specify a list of rdf predicates that you’ve extracted when parsing your documents. Ferenda has some basic knowledge about some common predicates and know how to construct sensible TocCriteria objects for them – ie. if you specify the predicate dct:issued, you get a TocCriteria object that groups documents by year of publication and sorts each group by date of publication.

To do this, you override toc_predicates() to something like this:

    def toc_predicates(self):
        return [self.ns['dct'].issued,
                self.ns['dct'].identifier]

The second, more complicated way is to override toc_criteria() and have it return a list of instantiated TocCriteria objects, like this (equivalent to the above example, but with more possibilities for customization):

    def toc_criteria(self):
        return [TocCriteria(binding='title', # variable binding, see next step
                            label='Sorted by publication date',
                            pagetitle='Documents published in %(select)s',
                            selector=lambda x: x['issued'][:4],
                            key=lambda x: x['issued']),
                TocCriteria(binding='identifier', 
                            label='Sorted by identifier',
                            pagetitle='Documents starting with "%(select)s"',
                            selector=lambda x: x['identifier'][0],
                            key=lambda x: x['identifier'].lower())]

The label and pagetitle parameters are useful to control the headings and labels for the generated pages. They should hopefully be self-explainatory.

The selector and key parameters should be functions (or any callable – like the lambda functions above) that accept a dictionary of string values. These functions are called once each for each row in the result set generated in the next step (see below) with the contents of that row. They should each return a single string value. The selector function should return the label of a group that the document belongs to, i.e. the initial letter of the title, or the year of a publication date. The key function should return a value that will be used for sorting, i.e. for document titles it could return the title without any leading “The”, lowercased, spaces removed etc.

Getting information about all documents¶

The next step is to perform a single SELECT query against the triplestore that retrieves a single large table, where each document is a row with a number of properties.

(This is different from the case of getting information related to a particular document, in that case, a CONSTRUCT query that retrieves a small RDF graph is used).

If you’ve used toc_predicates() above, a query that selects just those properties is automatically constructed by the default implementation of toc_query. If you’ve created your own toc_criteria() implementation, you might need to override this method as well. It should return a string containing a valid SPARQL SELECT query.

It’s called with the graph/context URI for the docrepo in question, which you’ll probably want to use in a SELECT FROM <> clause.

Each TocCriteria object is initialized with a binding parameter. The value of this parameter must match one of the variable bindings specified in the SPARQL query.

Your selector and key functions expect a dict of string values. The keys of this dict must match whatever variable bindings specified in the SPARQL query as well.

Making the TOC pages¶

The final step is to apply these criteria to the table of document properties in order to create a set of static HTML5 pages. This is in turn done in three different sub-steps, neither of which you’ll have to override.

The first sub-step, toc_pagesets(), applies the defined criteria to the data fetched from the triple store to calculate the complete set of TOC pages needed for each criteria (in the form of a TocPageset object, filled with TocPage objects). If your criteria groups documents by year of publication date, this method will yield one page for every year that at least one document was published in.

The next sub-step, toc_select_for_pages(), applies the criteria on the data again, and adds each document to the appropriate TocPage object.

The final sub-step transforms each of these TocPage objects into a HTML5 file. In the process, the method toc_item() is called for every single document listed on every single TOC page. This method controls how each document is presented when laid out. It’s called with a binding (same as used on each TocCriteria object) and a dict (same as used on the selector and key functions), and is expected to return a list of elements objects.

As an example, if you want to group by dct:identifier, but present each document with dct:identifier + dct:title:

    def toc_item(self, binding, row):
        # note: look at binding to determine which pageset is being
        # constructed in case you want to present documents in
        # different ways depending on that.
        from ferenda.elements import Link
        return [row['identifier'] + ": ",
                Link(row['title'], 
                     uri=row['uri'])]

The generated TOC pages automatically get a visual representation of each calculated TocPageset in the left navigational column.

The first page¶

The main way in to each docrepos set of TOC pages is through the tabs in the main header. That link goes to a special copy of the first page in the first pageset. The order of criteria specified by toc_predicates() or toc_criteria() is therefore important.

Running the web application¶

During development, you can just ferenda-build.py runserver. This starts up a single-threaded web server in the foreground with the web application, by default accessible as http://localhost:8000/

You can also run the web application under any WSGI server, such as mod_wsgi, uWSGI or Gunicorn. ferenda-setup creates a file called wsgi.py alongside ferenda-build.py which is used to serve the ferenda web app using WSGI. This is the contents of that file:

from ferenda.manager import make_wsgi_app
inifile = os.path.join(os.path.dirname(__file__), "ferenda.ini")
application = make_wsgi_app(inifile=inifile)

WSGIScriptAlias / /path/to/project/wsgi.py
WSGIPythonPath /path/to/project
<Directory /path/to/project>
  <Files wsgi.py>
    Order deny,allow
    Allow from all
  </Files>
</Directory>

URLs used¶

In keeping with Linked Data principles, all URIs for your documents should be retrievable. By default, all URIs for your documents start with http://localhost:8000/res (e.g. http://localhost:8000/res/rfc/4711 – this is controlled by the url parameter in ferenda.ini). These URIs are retrievable when you run the built-in web server during development, as described above.

The REST(ish) API¶

For each resource, use the Accept header to retrieve different versions of it:

• curl -H "Accept: text/html" http://localhost:8000/res/rfc/4711 returns rfc/generated/4711.html
• curl -H "Accept: application/xhtml+xml" http://localhost:8000/res/rfc/4711 returns rfc/parsed/4711.xhtml
• curl -H "Accept: application/rdf+xml" http://localhost:8000/res/rfc/4711 returns rfc/distilled/4711.rdf
• curl -H "Accept: text/turtle" http://localhost:8000/res/rfc/4711 returns rfc/distilled/4711.rdf, but in Turtle format
• curl -H "Accept: text/plain" http://localhost:8000/res/rfc/4711 returns rfc/distilled/4711.rdf, but in NTriples format

You can also get extended information about a single document in various RDF flavours. This extended information includes everything that construct_annotations() returns, i.e. information about documents that refer to this document.

• curl -H "Accept: application/rdf+xml" http://localhost:8000/res/rfc/4711/data returns a RDF/XML combination of rfc/distilled/4711.rdf and rfc/annotation/4711.grit.xml
• curl -H "Accept: text/turtle" http://localhost:8000/res/rfc/4711/data returns the same in Turtle format
• curl -H "Accept: text/plain" http://localhost:8000/res/rfc/4711/data returns the same in NTriples format

Triple stores¶

There are four choices.

[__root__]
storetype = SQLITE
storelocation = data/ferenda.sqlite # single file
storerepository = <projectname>

[__root__]
storetype = SLEEPYCAT
storelocation = data/ferenda.db # directory
storerepository = <projectname>

[__root__]
storetype = SESAME
storelocation = http://localhost:8080/openrdf-sesame
storerepository = <projectname>

Type: Native Java store
ID: <projectname> # eg same as storerepository in ferenda.ini
Title: Ferenda repository for <projectname>
Triple indexes: spoc,posc,cspo,opsc,psoc

[__root__]
storetype = SESAME
storelocation = http://localhost:3030
storerepository = ds

Fulltext search engines¶

There are two choices.

[__root__]
indextype = WHOOSH
indexlocation = data/whooshindex

[__root__]
indextype = ELASTICSEARCH
indexlocation = http://localhost:9200/ferenda/

Composite docrepos¶

In some cases, a document collection may available from multiple sources, with varying degrees of completeness and/or quality. For example, in a collection of US patents, some patents may be available in structured XML with good metadata through a easy-to-use API, some in tag-soup style HTML with no metadata, requiring screenscraping, and some in the form of TIFF files that you scanned yourself. The implementation of both download() and parse() will differ wildly for these sources. You’ll have something like this:

from ferenda import DocumentRepository, CompositeRepository
from ferenda.decorators import managedparsing

class XMLPatents(DocumentRepository):
    alias = "patxml"

    def download(self, basefile = None):
        download_from_api()

    @managedparsing
    def parse(self,doc):
        return self.transform_patent_xml_to_xhtml(doc)

class HTMLPatents(DocumentRepository):
    alias = "pathtml"
  
    def download(self, basefile=None):
        screenscrape()

    @managedparsing
    def parse(self,doc):
        return self.analyze_tagsoup(doc)

class ScannedPatents(DocumentRepository):
    alias = "patscan"

    # Assume that we, when we scanned the documents, placed them in their
    # correct place under data/patscan/downloaded

    def download(self, basefile=None): pass

    @managedparsing
    def parse(self,doc):
        x = self.ocr_and_structure(doc)
        return True

But since the result of all three parse() implementations are XHTML1.1+RDFa documents (possibly with varying degrees of data fidelity), the implementation of generate() will be substantially the same. Furthermore, you probably want to present a unified document collection to the end user, presenting documents derived from structured XML if they’re available, documents derived from tagsoup HTML if an XML version wasn’t available, and finally documents derived from your scanned documents if nothing else is available.

The class CompositeRepository makes this possible. You specify a number of subordinate docrepo classes using the subrepos class property.

class CompositePatents(CompositeRepository):
    alias = "pat"
    # Specify the classes in order of preference for parsed documents. 
    # Only if XMLPatents does not have a specific patent will HTMLPatents
    # get the chance to provide it through it's parse method
    subrepos = XMLPatents, HTMLPatents, ScannedPatents

    def generate(self, basefile, otherrepos=[]):
        # Optional code to transform parsed XHTML1.1+RDFa documents
        # into browser-ready HTML5, regardless of wheter these are
        # derived from structured XML, tagsoup HTML or scanned
        # TIFFs. If your parse() method can make these parsed
        # documents sufficiently alike and generic, you might not need
        # to implement this method at all.
        self.do_the_work(basefile)

The CompositeRepository docrepo then acts as a proxy for all of your specialized repositories:

$ ./ferenda-build.py patents.CompositePatents enable
# calls download() for all subrepos
$ ./ferenda-build.py pat download 
# selects the best subrepo that has patent 5,723,765, calls parse()
# for that, then copies the result to pat/parsed/ 5723765 (or links)
$ ./ferenda-build.py pat parse 5723765 
# uses the pat/parsed/5723765 data. From here on, we're just like any
# other docrepo.
$ ./ferenda-build.py pat generate 5723765 

Note that patents.XMLPatents and the other subrepos are never registered in ferenda.ini``. They’re just called behind-the-scenes by patents.CompositePatents.

Patch files¶

It is not uncommon that source documents in a document repository contains formatting irregularities, sensitive information that must be redacted, or just outright errors. In some cases, your parse implementation can detect and correct these things, but in other cases, the irregularities are so uncommon or unique that this is not possible to do in a general way.

As an alternative, you can patch the source document (or it’s intermediate representation) before the main part of your parsing logic.

The method patch_if_needed() automates most of this work for you. It expects a basefile and the corresponding source document as a string, looks in a patch directory for a corresponding patch file, and applies it if found.

By default, the patch directory is alongside the data directory. The patch file for document foo in repository bar should be placed in patches/bar/foo.patch. An optional description of the patch (as a plaintext, UTF-8 encoded file) can be placed in patches/bar/foo.desc.

patch_if_needed() returns a tuple (text, description). If there was no available patch, text is identical to the text passed in and description is None. If there was a patch available and it applied cleanly, text is the patched text and description is a description of the patch (or “(No patch description available)”). If there was a patch, but it didn’t apply cleanly, a PatchError is raised.

External annotations¶

Ferenda contains a general docrepo class that fetches data from a separate MediaWiki server and stores this as annotations/descriptions related to the documents in your main docrepos. This makes it possible to present a source document and commentary on it (including annotations about individual sections) side-by-side.

See ferenda.sources.general.MediaWiki

Keyword hubs¶

Ferenda also contains a general docrepo class that lists all keywords used by documents in your main docrepos (by default, it looks for all dct:subject properties used in any document) and generate documents for each of them. These documents have no content of their own, but act as hub pages that list all documents that use a certain keyword in one place.

When used together with the MediaWiki module above, this makes it possible to write editorial descriptions about each keyword used, that is presented alongside the list of documents that use that keyword.

See ferenda.sources.general.Keyword