Skip to content
/ rdf2rml Public
forked from VladimirAlexiev/rdf2rml

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

0 stars 5 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

SCVFlare/rdf2rml

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
bin
bin
 
 
doc
doc
 
 
ideas
ideas
 
 
lib/RDF/Prefixes
lib/RDF/Prefixes
 
 
test
test
 
 
.gitignore
.gitignore
 
 
CITATION.bib
CITATION.bib
 
 
CITATION.cff
CITATION.cff
 
 
README.org
README.org
 
 

Repository files navigation

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

Table of Contents

Introduction

See this presentation:

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation Alexiev, V. In Semantic Web in Libraries 2016 (SWIB 16), Bonn, Germany, November 2016. Presentation, HTML, PDF, Video

RDF is a graph data model, so the best way to understand RDF data schemas (ontologies, application profiles, RDF shapes) is with a diagram. Many RDF visualization tools exist, but they either focus on large graphs (where the details are not easily visible), or the visualization results are not satisfactory, or manual tweaking of the diagrams is required.

We describe a tool rdfpuml that makes true diagrams directly from Turtle examples using PlantUML and GraphViz. Diagram readability is of prime concern, and rdfpuml introduces various diagram control mechanisms using triples in the puml: namespace. Special attention is paid to inlining and visualizing various Reification mechanisms (described with PRV). We give examples from Getty CONA, Getty Museum, AAC (mappings of museum data to CIDOC CRM), Multisensor (NIF and FrameNet), EHRI (Holocaust Research into Jewish social networks), Duraspace (Portland Common Data Model for holding metadata in institutional repositories), Video annotation.

If the example instances include embedded source field names, they can describe a mapping precisely. I’ve implemented a few more tools to generate transformations:

  • rdf2rml generates R2RML transformations for RDBMS tables or SQL queries. Compared to R2RML, this saves about 15x in complexity and is competitive with the dedicated DSL YARRML
  • rdf2sparql generates OntoRefine or TARQL transformations from CSV/TSV that take the form of SPARQL UPDATE (for direct GraphDB loading) or CONSTRUCT (for conversion to RDF). (Subsumes two deprecated tools rdf2tarql and rdf2ontorefine)

See http://twitter.com/hashtag/rdfpuml for news, diagrams and announcements.

Citation

If you use this software, please cite it as shown above.

Documentation

Documentation source:

Related Work

The following works use or mention this software:

Installation

Checkout this repo and add rdf2rml/bin to your path. Install the following prerequisites:

  • both tools: Perl. Tested with version 5.22 on Windows (cygwin and Strawberry).
  • rdfpuml:
    • GraphViz
    • PlantUML. You need a recent version for new features like arrow length and color. I’m currently running 1.2018.10beta7. See in particular plantuml class diagrams.
    • Perl modules: use cpan or cpanm to install them: RDF::Trine RDF::Query Encode FindBin Carp::Always Slurp
    • RDF::Prefixes::Curie. This is my own module located in ./lib, and rdfpuml needs FindBin to locate it.
  • rdf2rml:
    • Apache Jena: riot, update. Tested with version 3.1.0 of 2016-05-10.
    • cat, grep, rm

Debian Repo

Jonas Smedegaard (@jonassmedegaard, dr at jones fullstop dk) has volunteered for some of the tasks below. His development is at https://salsa.debian.org/debian/rdf2rml/branches. To adopt changes, do something like this.

  • To merge all commits in the salsa/develop branch: 
    cd rdf2rml    # i.e. your local clone of your Github project
git remote add salsa https://salsa.debian.org/debian/rdf2rml.git
git fetch salsa
git merge salsa/develop
  • To adopt only single commits from the salsa/develop branch, issue remote and fetch as above, then issue: 
    git cherry-pick $commit1 $commit2 $commit3

Change Log

2023-05-17 rdf2sparql.pl

  • Support “Conditional Nodes”, i.e. URLs that are conditional on the existence of some fields.

2023-05-05 rdfpuml.pl

  • issue 21: Round brackets in fields (eg "(name)") and URLs (eg <type/(type)>) are not mangled to square brackets anymore

2023-04-29 rdfpuml.pl

  • issue 18 Add puml:option for left to right direction etc

2023-04-19 rdf2sparql.pl

  • issue 19 Implement filter function, see test/filter-content
  • issue 20 Allow dynamic graph (computed from a data column), see test/graphs-crunchbase

2022-08-23 rdf2sparql.pl

Datatype attachment eg strdt(?var,xsd:date) now outputs to ?var_xsd_date to avoid conflict with input field names in ALL_UPPERCASE

2022-08-23 rdfpuml.pl

  • Handle blank-node types that occur on owl:Restriction (see VladimirAlexiev#10 and test/blank-node)
  • Duplicate rdfpuml.bat, puml.bat as shell scripts rdfpuml, puml for use in Makefiles across Linux and Windows

2022-08-15 rdf2sparql.pl

Merge rdf2tarql and rdf2ontorefine to one tool rdf2sparql

2022-04-08 rdf2ontorefine.pl

Add script to generate OntoRefine SPARQL Update queries from model.

2021-09-02 Unicode Processing

Use Perl option -C when invoking for proper Unicode processing. See doc section rdfpuml.html#Unicode

2020-09-17 rdf2rml: logicalTable

Use URL for logicalTable instead of blank node, so that R2RML generated from different models for different tables can be merged more easily. Warning: this assumes that all instances of one subjectMap use the same query.

2020-06-01 rdf2tarql.pl

Add rdf2tarql.pl script to generate TARQL script (CSV-RDF conversion) from model.

2020-06-01 rdf2rml.sh, rdf2rml.ru

  • Improve script to abort if the first pipeline step (“update”) fails
  • Improve script to work on Cygwin (invokes the Jena tools as riot.bat and update.bat)
  • Filter out harmless warnings from Jena update’s error log for datatypes like xsd:integer, xsd:date etc since the mention of a source field doesn’t match the syntax of such literals.
  • If a node has single outgoing link and no SQL query/table (puml:label), propagate that property backward across the link into the node (previously that was done only for incoming links)

2020-05-30 rdf2rml: inverse edge

When an edge Y-P-X is recorded in the RDB table of X (as foreign key) or in an association table, it is awkward to specify that table in the node Y. So I added this SPARQL UPDATE clause:

  • If a node ?y has no SQL, is not Inlined, has a single outgoing edge, then add the SQL of its counterparty ?x as default

2018-11-14 Avoid puml:stereotype class node

I often define puml:stereotype for some classes in prefixes.ttl. If the class is not used in some particular turtle, it should avoid emitting a disconnected puml class.

  • `stereotypes`: Avoid emitting
  • `has_statements_different_from`: Check that a node has statements other than puml:stereotype

2018-06-29 Bug: class and puml:InlineProperty

When a type is also used with puml:InlineProperty, it caused this error:

Can't locate object method "uri_value" via package "RDF::Trine::Node::Literal" at rdfpuml.pl line 261.
   main::puml_qname(RDF::Trine::Node::Literal=ARRAY(0x4fd0920)) called at rdfpuml.pl line 279
   main::puml_node2(RDF::Trine::Node::Literal=ARRAY(0x4fd0920)) called at rdfpuml.pl line 128

An inline is converted to a literal, but rdf:type is always assumed to be a URL.

Test: ./test/regression/type-inlineProperty.ttl

2018-04-05 Arrow Attributes

Add arrow attributes (dotted, dashed, bold) and length

Test: ./test/regression/arrowLen.ttl

To Do Tasks

Help needed for the following tasks. Post bugs and enhancement requests to this repo!

Near-term

Dockerization

There’s a pull request VladimirAlexiev#7 that dockerizes the installation. As of 18-Sep-2019 it’s undergoing code review.

Docker Image

A docker image exits on the public Nexus of Ontotext. To run type:

`docker run -v <your directory>:/files –rm docker-registry.ontotext.com/rdf2rml:latest`

where `<your directory>` is a local directory holding your `.ttl` files.

Changelog

**2023-05-19**

Uses the following version

Modularize and Package Better

Regression Tests

  • sort is added at various places to make the tool more deterministic, i.e. independent of order of RDF statements in the input file. However, this will interfere with the ability to control the layout, especially of disconnected components (see layout_new_line)
  • Some regression tests are added.

rdf2rml: disentangle inverse edge

In the case Y-P-X described above:

  • Also need to record ?y puml:property ?p so this prop name can be added to ?y’s subject map
  • When making ?map, take puml:property into account
  • But ?map is made many times, and copy-paste is no good…
  • Also, this should be done in some cases but not others…
  • So it’s better to record ?y puml:map ?map …

Release on CPAN

Add Unicode tests

Add ttl with non-ASCII chars: Accented, Cyrillic, French, etc.

Prefixes

Allow specifying the prefixes file

See VladimirAlexiev#7

Eliminate Curie.pm

./lib/RDF/Prefixes/Curie.pm remembers @base and uses that for URL shortening. Once perlrdf#131 is fixed, eliminate this dependency (local module)

Remember prefixes from input file

rdfpuml shortens URLs using prefixes only from prefixes.ttl, but should also use prefixes defined in the individual input file.

Support more RDF Formats

Now it only supports Turtle, because it concatenates prefixes.ttl to the main file. If it can collect all prefixes from RDF files, such concatenation won’t be needed

Batch Processing

Issue #1: plantuml is slow to start up, so we’d like to process a bunch of puml files at once. The best way is to have a smarter script or Makefile that uses the following http://plantuml.com/command-line features:

  • Keep the intermediate puml files (the current Makefile doesn’t preserve them)
  • Run plantuml on a whole folder (with -r[ecurse] it can even recurse through subfolders)
  • Use -checkmetadata to skip png files that don’t need to be regenerated. (The whole puml text is stored in the png, so plantuml can quickly check that there are no changes)
  • The Makefile should start plantuml only once, if some of the puml files is newer than its respective png file

“Manual” Batching

Before I discovered the -checkmetadata option, I had the idea that rdfpuml could put several diagrams in one puml file:

@startuml file1.png
  # made from file1.ttl
@enduml
@startuml file2.png
  # made from file2.ttl
@enduml

However, this interferes with make processing that regenerates only png for changed ttl files, and makes things less modular overall.

Mid-Term

Upgrade to use Attean

Trine (Perl RDF) is end of life. Attean is the new generation

Integrate in Emacs org-mode

Write Turtle, see diagram (easy to do)

Node colors, icons, tooltips

See ./ideas

More arrow types and styles

./ideas/arrows.png ./ideas/arrows-2.png

dotted|dashed|plain|bold|hidden|norank|single|thickness

Extra Layout Options

Local layout options are described in Help on Layout:

  • “hidden” makes a constraint between two nodes, but does not draw the link (rdfpuml already implements this)
  • norank ignores a link for layout purposes (same as graphviz constraint=false)
  • “together” groups classes as if they were in the same package (i.e. puts them in a graphviz cluster)

Global options include (eg see this diagram):

And there are a lot more undocumented features: https://forum.plantuml.net/7095

Custom Reification

Ability to describe custom reification situations using the Property Reification Vocabulary (PRV)

Use MindMap/WBS for Hierarchies

Plantuml now has MindMap and WBS (or OBS) diagrams that use a simple bulleted syntax to draw hierarchies.

It would be nice to use this to draw hierarchies of individuals, in particular taxonomies.

Here are examples of the two styles:

Long-Term

rdf2soml to Generate Semantic Object Models

A new tool rdf2soml to generate Ontotext Platform SOML from RDF examples.

What’s missing? Most importantly: property cardinality and virtual inverses.

PlantUML can show arrow cardinalities, and this simple and natural PlantUML code:

X "0:1" -left-> "1:m" Y : prop/\ninvProp

Is depicted as follows:

./ideas/cardinality-and-inverse.png

We have two options how to express this in triples:

Cardinality With RDF*

##### model triples
:X :prop :Y.
##### puml triples
<< :X :prop :Y >>
  puml:arrow puml:left; # direction
  puml:min 1; puml:max puml:inf; # cardinality
  puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"]. # virtual inverse
  • Pros: very natural
  • Cons:
    • Perl RDF doesn’t support RDF*, and few editors support it either.
    • Annotating a triple does not assert it, so we need to assert it as well

Cardinality With Blank Node

##### model triples
:X :prop :Y.
##### puml triples
:X puml:left :Y. # direction
:X :prop [ # a puml:Cardinality; # may need this marker class to skip the node from the diagram
  puml:min 1; puml:max puml:inf; # cardinality
  puml:object :Y; # only needed if X has several relations "prop" and they need different annotations
  puml:inverseAlias [puml:min 0; puml:max 1; puml:name "invProp"] # virtual inverse
].

rdf2shape to Describe & Generate RDF Shapes

Visualize RDF Shapes (SHACL and ShEx)

Issue #8: discussion with Thomas Francart of Sparna

I developed this SHACL to PlantUML converter, in Java, based on TopQuadrant SHACL lib, and the result is at https://shacl-play.sparna.fr/play/draw and code at https://github.com/sparna-git/shacl-play/tree/master/shacl-diagram

I don’t have a strong opinion on the example you provide, an alternative idea that comes to my mind is

:node1 :link [
  rdf:value :node2;
  puml:min 1 ;
  puml:max 2 ;
]

But this changes the structure of the example graph itself, which might not be convenient

Generate transformations for other than relational sources

R2RML works great for RDBMS, but how about other sources? Extend rdf2rml to generate:

  • RML: extends R2RML to handle RDB, XML, JSON, CSV
  • XSPARQL: extends XQuery with SPARQL construct and JSON input
  • DONE tarql: handles TSV/CSV with SPARQL construct
  • DONE OntoRefine: transformation of TSV/CSV and direct loading to GraphDB with SPARQL Update

About

RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Perl 60.8%
  • Ruby 30.9%
  • Makefile 4.3%
  • C 2.2%
  • Shell 1.7%
  • Batchfile 0.1%