The Devel class

class ferenda.Devel(config=None, **kwargs)

Collection of utility commands for developing docrepos.

This module acts as a docrepo (and as such is easily callable from ferenda-manager.py), but instead of download, parse, generate et al, contains various tool commands that is useful for developing and debugging your own docrepo classes.

Use it by first enabling it:

./ferenda-build.py ferenda.Devel enable

And then run individual tools like:

./ferenda-build.py devel dumprdf path/to/xhtml/rdfa.xhtml

alias = 'devel'

dumprdf(filename, format='turtle')

Extract all RDF data from a parsed file and dump it to stdout.

Parameters:

• filename (str) – Full path of the parsed XHTML+RDFa file.
• format (str) – The serialization format for RDF data (same as for rdflib.graph.Graph.serialize())

Example:

./ferenda-build.py devel dumprdf path/to/xhtml/rdfa.xhtml nt

dumpstore(format='turtle')

Extract all RDF data from the system triplestore and dump it to stdout using the specified format.

Parameters:format (str) – The serialization format for RDF data (same as for ferenda.TripleStore.get_serialized()).

Example:

./ferenda-build.py devel dumpstore nt > alltriples.nt

csvinventory(alias, predicates=None)

Create an inventory of documents, as a CSV file.

Only documents that have been parsed and yielded some minimum amount of RDF metadata will be included.

Parameters:alias (str) – Docrepo alias

mkpatch(alias, basefile, description, patchedtext=None)

Create a patch file from downloaded or intermediate files. Before running this tool, you should hand-edit the intermediate file. If your docrepo doesn’t use intermediate files, you should hand-edit the downloaded file instead. The tool will first stash away the intermediate (or downloaded) file, then re-run parse() (or download_single()) in order to get a new intermediate (or downloaded) file. It will then calculate the diff between these two versions and save it as a patch file in it’s proper place (as determined by config.patchdir), where it will be picked up automatically by patch_if_needed().

Parameters:

• alias (str) – Docrepo alias
• basefile (str) – The basefile for the document to patch

Example:

./ferenda-build.py devel mkpatch myrepo basefile1 "Removed sensitive personal information"

parsestring(string, citationpattern, uriformatter=None)

Parse a string using a named citationpattern and print parse tree and optionally formatted uri(s) on stdout.

Parameters:

• string (str) – The text to parse
• citationpattern (str) – The fully qualified name of a citationpattern
• uriformatter (str) – The fully qualified name of a uriformatter

Note

This is not implemented yet

Example:

./ferenda-build.py devel parsestring \
    "According to direktiv 2007/42/EU, ..." \
    ferenda.citationpatterns.eulaw

fsmparse(functionname, source)

Parse a list of text chunks using a named fsm parser and output the parse tree and final result to stdout.

Parameters:

• functionname (str) – A function that returns a configured FSMParser
• source (str) – A file containing the text chunks, separated by double newlines

queryindex(querystring)

Query the system fulltext index and return the IDs/URIs for matching documents.

Parameters:querystring (str) – The query

construct(template, uri, format='turtle')

Run the specified SPARQL CONSTRUCT query.

select(template, uri, format='json')

Run the specified SPARQL SELECT query.

destroyindex()

Clear all data in the fulltext search index.

clearstore()

Clear all data in the current triplestore.

wsgi(path='/')

Runs WSGI calls in-process.

samplerepo(alias, sourcedir, sourcerepo=None, destrepo=None, samplesize=None)

Copy a random selection of documents from an external docrepo to the current datadir.

copyrepos(sourcedir, basefilelist)

Copy some specified documents to the current datadir.

The documents are specified in BASEFILELIST, and copied from the external directory SOURCEDIR.

To be used with the output of analyze-error-log.py, eg $ ../tools/analyze-error-log.py data/logs/20160522-120204.log –listerrors > errors.txt $ ./ferenda-build.py devel copyrepos /path/to/big/external/datadir errors.txt

samplerepos(sourcedir)

Copy a random selection of external documents to the current datadir - for all docrepos.

statusreport(alias=None)

Generate report on which files parse()d OK, with errors, or failed.

Creates a servable HTML file containing information about how the last parse went for each doc in the given repo (or all repos if none given).

documentstore_class

alias of DummyStore

downloaded_suffix = '.html'

storage_policy = 'file'

ns = {}

resourceloader = <ferenda.resourceloader.ResourceLoader object>

classmethod get_default_options()

download()

parse(basefile)

relate(basefile)

generate(basefile)

toc(otherrepos)

news(otherrepos)

tabs()

footer()

facets()

basefile_from_uri(uri)

classmethod setup(action, config, *args, **kwargs)

classmethod teardown(action, config, *args, **kwargs)