SWORD v2 – From clueless to claymore

October 4, 2012 by 3 Comments

What follows is a summary of my steps along the path of investigating what the sword technology is, through to being able to actually start to code something useful; I should probably point out that the beginning of this post can be consumed by less technical persons as a quick overview, but the later section assumes that you…

  • Have some knowledge of coding java
  • Have worked with java server containers (e.g. tomcat) before
  • Can place the libraries in an IDE like netbeans/eclipse to do “something” with them

(Since my investigations centered around sword in conjunction with Sakai and e-science central my language of choice therefore is Java).

I should also declare that I still don’t fully understand all of the implementation but this should help you along your way if you’re just starting out!

Taking the Sword course

My first port of call was the SWORD website itself, which will point you to some useful videos and slides to give you insight into what the technology is and what it can be used for. In short, this is what the “Sword Course” will teach you…

An Introduction To SWORD (Video/Slides)

What it is:

The “Simple Webservice Offering Repository Deposit” technology (or SWORD for short) intentionally only deals with how to get data into a repository, nothing else, and is complementary to something like dublin core used to describe stuff in a repository; it also does not deal with packaging, metadata, authentication or authorisation

Existing implementations can be found in:

DSpace
Eprints
Fedora
Intralibrary
Zentity

SWORD Use Cases (Video/Slides)

Use cases sword is trying to enable:

  • Deposit from a desktop machine
  • Deposit to multiple repositories (For example to allow depositing once and ending up in an institution’s repository, funder’s repository and a subject specific repository)
  • Deposit from a piece of lab equipment (non-human depositing data)
  • Deposit from one repository to another (For example Institutional repository to National repository which may have differing visibility of the data…. Dark and light repositories, dark = can’t be seen private repositories, light = can be seen public repository)
  • Deposit from external publisher/publishing system to long term storage (For example from OJS to your own institution’s repository)

How SWORD Works (Video/Slides)

  • SWORD is in the form of an “XML REST webservice to put things in a repository”
  • It has built on the resource creation aspects of the ATOM Pub standard which is for publishing content to the web
  • SWORD is an extension, or “profile”, of the ATOM Pub spec and existing ATOM Pub clients can be made to work if the relevant extensions are added
  • SWORD version 2 now includes full CRUD

When you use a sword client, this is basically what happens…

  • The client asks a repository to describe itself
  • The server returns a “Service document” to describe what you can do with the repository and how to do that. The service document is typically hidden by Basic Auth Authentication (AM: I think this is crying out for an OAuth implementation!) but once authenticated the web service will customise the service document to what you are allowed to / should do with the system. The server can also describe what data formats you want to accept, where it will go and how long you will store it etc… this is your “collection policy”
  • The client then uses the service description to format your data and then deposit it

What sword adds to ATOM Pub:

  • Accept Packaging – Tells the client what types of data the server accepts
  • Mediated Deposit – Allows you to deposit “as” / “on behalf of” someone else, a repository can say whether it allows this or not
  • Developer features – You can state that you want verbose output to say what happened (v1.3 featured a dry run feature called “no-op” that does not actually deposit or do anything. NOTE: this does not appear to be in v2 anymore)
  • Nested Service document – Where there may be many repositories for the service, the top level document provides links to sub documents, instead of repeating the same or similar definitions in one enormous file.

SWORD clients (Video/Slides)

There are generally three sorts of client:

  • Machine to machine – for very specific automated deposits (lab equipment)
  • General – human would use, talks to any repository
  • Specific – for depositing certain data into certain repositories in a given way that has an extra context of a general client, i.e. depositing specific journals, depositing data for a particular project

Interesting possibilities for deposit scenarios:

Writing something useful

My next stop on the journey through sword looked at actual code, how it’s laid out and what you need to do in order to start doing something useful.

As mentioned previously, I have been basing my investigations around the Java client and server libraries but I’d strongly recommend you also get a good grounding in the workings of ATOM Pub (HTML Version) and the Sword Profile specifications themselves. If you’ve ever read specification documents before you’ll know they can make quite dry reading, however, since ATOM Pub and sword are relatively straight forward technologies and the specs only reach into 30-50 odd pages it’s really worth a browse through.

How SWORD works in java

Firstly, it’s probably best to understand a few basic concepts you will need to deal with, the main outputted concepts/objects are:

  • IRI‘s – unique identifiers to a resource
  • Entry – A deposit, has IRI’s/metadata
  • Media – An entry for media (word docs/pdfs/images), can be linked to in an Entry
  • Collection Entries – ATOM Pub collection of entries (member resources)
  • Collections – A set of deposited entries represented by an Atom Feed document, you can distinguish a collection feed from “a.n.other feed” by the presence of a collection IRI in service document
  • Workspaces – A compartmentilisation concept for repositories, has a name but doesn’t have IRI’s or processing models
  • Service Documents – Groups “Collections” in “Workspaces”, can indicate accepted media types and categories of collections

Next, I found you learn the most by studying the server libraries, what you get is a bundle of java and some set up files for your container. We’ll firstly look at the setup in web.xml

Setup and Servlet mappings (web.xml)

The main servlets (i.e ultimately your rest endpoints) that are defined are…

servicedocument

collection

  • Class: org.swordapp.server.servlets.CollectionServletDefault
  • URL: http://<yourServer>/<yourWebapp>/collection/*
  • Purpose: Retrieveing and deposting to/from collections/feeds and entries

mediaresource

container

statement

The code makes heavy use of interfaces to allow the implementer more freedom to create functionality using the server library in the way they want to, in order to tell the server libraries what code we want to instantiate and what auth. mechanism we will be using, you must set some context parameters to define those settings and implementations used at runtime:

auth

  • param-name: authentication-method
  • param-value: Basic or None (default: “Basic”)

You can set an Authorization header and base64 encode user:password (e.g. try going to a basic auth encoding website and encode “user:password” in plain text box) and send as a header… “Authorization” “Basic dXNlcjpwYXNzd29yZA==”, or, if you prefer set the param-value to “None” for no authorisation. I found (at the time of writing) the default code actually has a bug which means turning the auth off doesn’t work correctly. I found the best way of correcting this was in my own war project (which includes the server libraries as a dependancy) I created a org.swordapp.sever package (where I was implementing the objects needed for the interfaces) and dropped in a copy of the SwordAPIEndpoint.java to override the implementation in the library, I then changed the getAuthCredentials to…

protected AuthCredentials getAuthCredentials(HttpServletRequest request, boolean allowUnauthenticated) throws SwordAuthException
{
   AuthCredentials auth = null;
   String authType = this.config.getAuthType();
   String obo = "";
   this.log.info("Auth type = "+authType);
   //If we are insisting on "a" form of authentication that is not of type "none"
   if(!allowUnauthenticated && !authType.equalsIgnoreCase("none"))
   {
      // Has the user passed authentication details
      String authHeader = request.getHeader("Authorization");
      // Is there an On-Behalf-Of header?
      obo = request.getHeader("On-Behalf-Of");
      // Which authentication scheme do we recognise (should only be Basic)
      boolean isBasic = authType.equalsIgnoreCase("basic");

      if(isBasic && (authHeader == null || authHeader.equals("")))
      {
         throw new SwordAuthException(true);
      }
      // decode the auth header and populate the authcredentials object for return
      String[] userPass = this.decodeAuthHeader(authHeader);
      auth = new AuthCredentials(userPass[0], userPass[1], obo);
   }
   else
   {
      log.debug("No Authentication Credentials supplied/required");
      auth = new AuthCredentials(null, null, obo);
   }
   return auth;
}

The following context parameters set the implementations of interfaces used to implement functionality in the endpoints. However, you are not given default implementations for each of these, so in your war you need to create a new class that implements the respective interface and fill in your functionality….

collection-list-impl

  • param-value: org.swordapp.server.CollectionListManagerImpl
  • Interface it implements: org.swordapp.server.CollectionListManager

service-document-impl

  • param-value: org.swordapp.server.ServiceDocumentManagerImpl
  • Interface it implements: org.swordapp.server.ServiceDocumentManager

collection-list-impl

  • param-value: org.swordapp.server.CollectionListManagerImpl
  • Interface it implements: org.swordapp.server.CollectionListManager

collection-deposit-impl

  • param-value: org.swordapp.server.CollectionDepositManagerImpl
  • Interface it implements: org.swordapp.server.CollectionDepositManager

media-resource-impl

  • param-value: org.swordapp.server.MediaResourceManagerImpl
  • Interface it implements: org.swordapp.server.MediaResourceManager

container-impl

  • param-value: org.swordapp.server.ContainerManagerImpl
  • Interface it implements:  org.swordapp.server.ContainerManager

statement-impl

  • param-value: org.swordapp.server.StatementManagerImpl
  • Interface it implements: org.swordapp.server.StatementManager

config-impl

  • param-value: org.swordapp.server.SwordConfigurationDefault (Yes, this one does have a default implementation in the library you can use)
  • Interface it implements: org.swordapp.server.SwordConfiguration

Endpoint Servlet classes (org.swordapp.server.servlets.*)

Let’s now have a look at the servlets themselves, each servlet contains interfaces which are implemented by loading the classes specified in the web.xml (see above).

All servlets used in the server library extend the “SwordServlet” which (obviously) extends the HttpServlet. Since the SwordServlet contains the server configuration object, all servlets (through inheritance) also hold an implementation of the server configuration (i.e. SwordConfiguration) and a method to allow servlets to load classes from the configuration….

SwordServlet encapsulates:

  • SwordConfiguration interface, instantiated using config-impl
  • loadImplClass() method used for loading implementing classes from tomcat context params

CollectionServletDefault extends SwordServlet and encapsulates:

  • CollectionListManager interface, instantied using collection-list-impl
  • CollectionDepositManager interface, instantied using collection-deposit-impl
  • CollectionAPI object

ServiceDocumentServletDefault extends SwordServlet and encapsulates:

  • ServiceDocumentManager interface, instantiated using service-document-impl
  • ServiceDocumentAPI object

MediaResourceServletDefault extends SwordServlet and encapsulates:

  • MediaResourceManager interface, instantiated using media-resource-impl
  • MediaResourceAPI object

ContainerServletDefault extends SwordServlet and encapsulates:

  • ContainerManager interface, instantiated using container-impl
  • StatementManager interface, instantiated using statement-impl
  • ContainerAPI object

StatementServletDefault extends SwordServlet and encapsulates:

  • StatementManager interface, instantiated using statement-impl
  • StatementAPI object

Endpoint Servlet dependant classes (org.swordapp.server.*)

Those with a keen eye will have noticed that each servlet is also holding an “API” object, these objects fill out the standard Get/Post/Put/Delete HttpServlet methods that the servlets override by taking the configuration object and any interfaces that have been implemented and combine them to do something useful. Similarly to the servlets, they all extend a Sword API super class called “SwordAPIEndpoint”, which holds a SwordConfiguration implementation. The hierarchy (and interfaces they encapsulate) looks like this…

SwordAPIEndpoint

  • SwordConfiguration interface

CollectionAPI extends SwordAPIEndpoint

  • CollectionListManager interface
  • CollectionDepositManager interface

ServiceDocumentAPI extends SwordAPIEndpoint

  • ServiceDocumentManager interface

MediaResourceAPI extends SwordAPIEndpoint

  • MediaResourceManager interface

ContainerAPI extends SwordAPIEndpoint

  • ContainerManager interface
  • StatementManager interface

StatementAPI extends SwordAPIEndpoint

  • StatementManager interface

Implementations of interfaces (org.swordapp.server.*)

I keep mentioning the objects that implement the interfaces, I thought it might be useful to go through in “slightly” more detail what the content of those objects are intended for. Apologies once again, this is not exhaustive as I have not worked my way through what all the methods are intended for:

SwordConfigurationDefault implements org.swordapp.server.SwordConfiguration

  • This is the default object which holds the configuration for the server

CollectionListManagerImpl implements org.swordapp.server.CollectionListManager

CollectionDepositManagerImpl implements org.swordapp.server.CollectionDepositManager

ServiceDocumentManagerImpl implements org.swordpapp.server.ServerDocumentManager

  • Method: getServiceDocument()
  • Accessed via: GET http://<yourServer>/<yourWebapp>/servicedocument/
  • Returns: org.swordapp.server.ServiceDocument
  • Purpose: Serves service documents (xml that explains the contents and deposit policies for the repository(/ies)

MediaResourceManagerImpl implements org.swordapp.server.MediaResourceManager

  • Method: replaceMediaResource()
  • Accessed via: PUT http://<yourServer>/<yourWebapp>/edit-media/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Swap a media resource (pdf/doc etc….) in the repository with the one being “PUT’ed”

ContainerManagerImpl implements org.swordapp.server.ContainerManager

  • Method: replaceMetadataAndMediaResource()
  • Accessed via: PUT http://<yourServer>/<yourWebapp>/edit/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Replaces metadata and media associated with an entry
  • Method: addMetadataAndResources()
  • Accessed via: Does not appear to be “directly” accessible via any specific HTTP request
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Not used by the ContainerAPI yet, but presumably it would be for adding a series of entries and associated metadata
  • Method: addResources()
  • Accessed via: Does not appear to be “directly” accessible
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Not used by the ContainerAPI yet, but presumably it would be for adding a series of entries
  • Method: useHeaders()
  • Accessed via: POST http://<yourServer>/<yourWebapp>/edit/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Used when depositing only information specified in the HTTP headers, no entry/ies (i.e. no content body to the POST) will have been specified

StatementManagerImpl implements org.swordapp.server.StatementManager

And finally…

Once you have all that setup (and I’d recommend just creating skeleton override methods for the objects implementing interfaces for the time being whilst you figure the code out), you can then start coding the abdera / sword code and try make the client do something. The client itself comes with a handy cli driven (SwordCLI) interface that you can point at your newly created server instance and test the various example methods. I would recommend though, that you comment out the entire list of method references in the main method and go through the list iteratively to slowly make each part of your server work…

As a brief example, if we were to try and get a basic service document to work, try adding this code to your ServiceDocumentManagerImpl.java….

    public ServiceDocument getServiceDocument(String sdUri, AuthCredentials auth, SwordConfiguration config) throws SwordError, SwordServerException, SwordAuthException
    {
        //Our test service document
        ServiceDocument sd = new ServiceDocument();
        //sd.setVersion("2.0");
        sd.setMaxUploadSize(1000000000);

        //Our test workspace
        SwordWorkspace sw = new SwordWorkspace();
        sw.setTitle("TestWorkspace");

        //Our test collection
        SwordCollection sc = new SwordCollection();
        sc.setHref("http://<yourServer>/<yourWebapp>/collection/TestCollection");
        sc.setTitle("TestCollection");
        sc.addAccepts("*/*");
        sc.setCollectionPolicy("TestCollectionPolicy");
        sc.setAbstract("TestCollectionAbstract");
        sc.setMediation(false);
        sc.setTreatment("A human-readable statement describing treatment the deposited resource has received or a URI that dereferences to such a description.");
        sc.addAcceptPackaging("http://purl.org/net/sword/package/SimpleZip");
        sc.addAcceptPackaging("http://purl.org/net/sword/package/METSDSpaceSIP");
        sc.addAcceptPackaging("http://www.ncl.ac.uk/terms/package/html");
        sc.setLocation("http://<yourServer>/<yourWebapp>/collection/TestCollection");
        sc.setMediation(true);
        List iris = new ArrayList();
        iris.add(new IRI("http://<yourServer>/<yourWebapp>/collection/TestCollection/TestSubService"));
        sc.setSubServices(iris);

        //Add collection to workspace
        sw.addCollection(sc);
        //Add workspace to service document
        sd.addWorkspace(sw);

        return sd;
    }

Browsing to http://<yourServer>/<yourWebapp>/collection/ should return something, and removing your comment in your client for the line…

    cli.trySwordServiceDocument()

…should now yield some results (If you just get errors try temporarily turning off the auth. requirement on the server for the purposes of testing).

And that’s the basic principle, you then take the specs and implement the returning and unpackaging of ATOM using the abdera/SWORD objects and link what’s passed/returned to the content found in the system you are trying to integrate.

Further reading

Sword Specs
Brief history of Sword
More useful Sword docs
Another set of slides explaining sword

Andrew Martin
Research and Collaborative Services
Newcastle University

Filed under blog report, dissemination, implementation, systems, tools

iridium – RDM systems/tools ‘connectivity’ (busy researchers don’t like duplication of metadata entry!)

October 3, 2012 by Leave a comment

Briefly when discussing potential scope for the proof-of-concept Research Data Catalogue (RDC) we talked of *possible* future ‘connectivity’ needs.

We noted that researchers have told us *very* clearly that they do not want to enter research project admin data/outputs metadata twice in multiple systems, either internally or externally to institution.

This requires us to understand some of the systems the RDC may need to exchange metadata with that have existing information already entered. These could be local research group metadata catalogues, local/national repositories and other online systems (for example, we would need to outline the technical protocols (standards) for interoperability with national funder research output systems our researchers have been using).

1. Possible external connectivity needs?

i) ROS?
– http://www.rcuk.ac.uk/research/Pages/ResearchOutcomesProject.aspx
– example -> https://logon.rcuk.ac.uk/
Schema: ?
API: ?

iii) MRC e-Val/Research Fish?

– https://www.researchfish.com/

“Can I integrate the data from e-Val with other systems?

At Researchfish we want to reduce the burden on researchers required to provide information to their Funders and other organisations. As such we offer an API to allow you to use the data in other systems and the data is available for you to download at any time. We are also working with a number of organisations such as EuroCris to ensure future compatibility of our data with other research information systems”

Schema: ?
API: Yes, need details.

iii) Je-S?

– https://je-s.rcuk.ac.uk/JeS2WebLoginSite/Login.aspx
Schema: ?
API: ?

2. Possible internal connectivity needs?

i) A future desktop tool?

ii) Sakai VRE?

iii) e-Science Central?

iii) Most likely research group data set/outputs catalogues/’repositories’ (at least 1 possible example  identified), but many have not been discovered yet, so a standard interoperable feed (in and out) are important i.e. OAI-PMH, SWORD2, (CERIF/XML, RSS maybe ?)

NB: ‘JISC and Research Councils UK work to reduce reporting burden on Universities‘ post from August 2012 is topical.

Filed under reflection, requirements, systems Tagged with , , , , , , , , ,

iridium – requirements for a research data catalogue and proof-of-concept development

October 2, 2012 by Leave a comment

Background

A brief recap on background to the development a proof-of-concept Research Data Catalogue tool:

From the iridium JISC project proposal:

Funding councils are now expecting researchers to make their data available to others. This

requires them to:

• make their data available into the future in a safe and secure manner;

• provide appropriate tagging of the data such that its significance can be determined;

• provide provenance information over the data so that the prior uses and access to the data can be determined.

There is an ever-growing expectation that data curation processes should be handled at the institutional level. For example the EPSRC Policy Framework on Research Data and expectations are clearly stated. Points highlighted below are of particular relevance to the the development of a data catalogue/institutional metadata record:

“… processes to maintain effective internal awareness of their publicly funded research data holdings and of requests by third parties to access such data …”

“… organisations will ensure that appropriately structured metadata describing the research data they hold is published (normally within 12 months of the data being generated) and made freely accessible on the internet …”

“… metadata must be sufficient to allow others to understand what research data exists, why, when and how it was generated, and how to access it …”

“… where the research data referred to in the metadata is a digital object it is expected that the metadata will include use of a robust digital object identifier …”

“… where access to the data is restricted the published metadata should also give the reason and summarise the conditions which must be satisfied for access to be granted. For example ‘commercially confidential’ data, in which a business organisation has a legitimate interest, might be made available to others subject to a suitable legally enforceable non-disclosure agreement …”

The timelines for this are:

  • a roadmap by May 2012,
  • full compliance May 2015

The iridium project is developing and consulting on a draft research data management policy, as are many Universities, in response to the above. For a national review of policies published, see the DCC website. Across the sector there is general consensus on key principles that need to be present (many originating from the RCUK principles and of the ‘aspirational’ policy type pioneered by Edinburgh University). For example:

  • accountability, responsibilities and roles
  • data location, treatment and security
  • data access mechanisms linked to publications
  • internal availability of research data
  • internal deposit of data/discipline repositories
  • curation mechanisms
  • training and support provision
  • embargo periods, ethical considerations and protecting commercial opportunities

A research data catalogue tool, to support implementation of the draft policy, is an obvious need for several of the above themes.

Existing local institutional research information management systems

Our institution has several major existing systems for Research Information Management (RIM):

  • MyProjects – used to support and record University projects, particularly funded projects, throughout their life-cycle (with information additionally fed in from MyProjectsProposals on costing, pricing and sign-off )
  • MyImpact – a person based system in which information on publications and activities are collected for day to day use by the academic community (with information sent to Newcastle ePrints repository of peer-reviewed institutional research)

The iridium project proposal outlined an early RDM (tool) specification for research data to be made available to others through a definable access policy, including tagging of the data and along with recording of provenance. This was proposed to be handled in a semi-automated fashion such that as much of the information as possible can be provided automatically, and that information which cannot be provided automatically can be prompted for.

Research Data Catalogue (RDC) proof-of-concept specification

Early specification work outlined:

  • this should use functionalities which already exist within the University, or which need to be brought in, and which can be integrated to provide a research data management system
  • to fulfil these requirements, the Research Data Catalogue (RDC) proof-of-concept will join up the MyProjects and MyImpact systems and facilitate the manual attachment of a small pre-defined set of metadata. It uses the Institutional Data Feeds Service (IDFS) (see earlier JISC-funded IDMAPS project)
  • this tools will derive appropriate high quality metadata from the existing project and publications records, then augmented and exported to an open searchable web-based interface (API) (machine and person readable).

Schematic below outlines the RDC and institutional RIM system data flows [click for larger image]

iridium Research Data Catalogue proof-of-concept data flows

RIM data flows – iridium Research Data Catalogue proof-of-concept – [Click for larger image]

Research Data Catalogue Basic User Interface

The Basic User Interface of the proof-of-concept RDC has been designed, is functional on a test server and has been demonstrated/tested within project team.

The following summarises the user interface.

Home screen – after the user logs in through Shibboleth authentication, the home welcome screen presents user with tabs for Project, Publications and Data Sources.

iridium Research Data Catalogue proof-of-concept

Welcome tab – iridium Research Data Catalogue proof-of-concept [click for larger image]

Project tab – presents user with research project(s) owned by a Principle Investigator and are pre-populated from their Newcastle ID using MyProjects data feed.

iridium Research Data Catalogue proof-of-concept

Projects tab – iridium Research Data Catalogue proof-of-concept [click for larger image]

Publications tab – presents user with research publications owned by a Principle Investigator and are pre-populated from their Newcastle ID using MyImpact data feed.

iridium Research Data Catalogue proof-of-concept

Publications tab – iridium Research Data Catalogue proof-of-concept [click for larger image]

Data Source tab– allows the addition of a novel data source metadata record to a publication and for this link to be saved in catalogue record system, together with additional user added metadata/tagging.

Data Source tab - iridium Research Data Catalogue proof-of-concept

Data source tab – iridium Research Data Catalogue proof-of-concept [click for larger image]

The working metadata scheme for the proof-of-concept RDC Data Source record has been blogged here. This scheme will be reviewed as project progresses.

We note discussions in the MRD Programme on the DataCite scheme which is gaining popularity, and which is broadly similar and also findings from some previous projects on what fields are actually *used* in data catalogues/repositories for re-use in practice.

The RDC, if implemented institutionally, would bring us closer to CERIF compliance, which is a longer term aim outside the project, but with many institutional advantages.

RDC  ‘roadmap’ and within project proof-of-concept development plan priorities

The process will allow for prototyping, iterative development/feedback, user testing and follow-up development work. Plain English user documentation will be written to fully support tools by the human support workpackage.

Phase 1: Basic UI  is the implementation of:

  • A basic User Interface for linking Publications, Projects and Data Sources (status: working demonstration completed and tested).
  • an Admin Interface for assigned roles and records approval process (ideally inherited from existing RIM systems) (initiated and is in progress)
  • simple reporting feature for researchers/research groups piloting the RDC (initiated and is in progress)

This proof-of-concept work is on track for late  2012. The second phase falls  within a ‘roadmap’ of potential developments based on project priorities:

Phase 2a: Web Servies & Reporting is the future implementation of:

  • the fully documented web service interface (an API, systems can push data to it or read data from it for internal, or external, reporting purposes), including pluggable interfaces using LTI/Widget/Portlet technologies, interoperability standards (i.e. OAI-PMH, SWORD, etc.) allowing embedding in and exchange with other tools/systems [not required for proof-of-concept]
  • advanced reporting [not required for proof-of-concept]

Phase 2B: Data (metadata) Harvesting is the implementation of  harvesting plugins:

  • use of  ‘harvester’ plugins are being looked at as part of tools/systems workpackages to enable researchers to have their information automatically updated from e-Science Central, SAKAI, web services and other repositories. SWORD protocol is desirable. [scoping work initiated looking at Talend, RapidMiner, etc. options) – implementation not required for proof-of-concept].

Developing ‘Data Catalogues’ and future points of discussion

Below are some development and testing team discussion points and questions we have had, which would be useful to share more widely to JISC MRD Programme, as they are likely to be of interest to those developing systems with similar/overlapping functionality (some of these do not need to be resolved for this stage in our proof-of-concept development and have been deferred):

  • delegation of record editing permissions from PI to others (including research administrators)
  • inheriting roles from existing RIM systems
  • approval process for updated records submitted
  • quality assurance of metadata added
  • how much free tagging is useful versus later needed for data cleansing/validation?
  • what existing controlled vocabularies/taxonomies for research data sets are available/useful
  • an extensible metadata scheme to allow established research discipline specific fields to be added where available
  • degree of pre-population of metadata from internal or external existing systems  (i.e. ‘connectivity’)
  • to what extent (or should?) can existing publication tags/keywords/’re-use’ metadata be pull in through APIs/external services to reduce burden on staff time with manual records entry and assist with high quality metadata encouraging data re-use/discovery?
  • use of researcher’s own personal existing tags/citation reference databases as proposed in recent blog post by OpenExeter project (other local ways to achieve this i.e. MyImpact functions)?
  • evidence for what fields/system functions are *used* in existing data repositories/catalogues
  • interfaces to measure and display user completion rates (i.e. progress on linking/metadata completion) of data source records to publications
  • report generation of linked projects, publications and data sources (possible visualisations to assist?)
  • indicating/user navigation of the various licences that data sets could/need to be released with (see DCC) and Terms & Condition for access/use of data catalogue records
  • general usability and navigation issues, especially if a user has many publications/project records
  • a simple user search/filter of assigned multiple data set records/papers/projects, to assist in above
  • user records/work-in-progress organisation, user ‘folders’ and progress options
  • future metadata storage options and benefits for production systems (Triplestore, NoSQL, etc.)
  • production system performance, security and system support staff resources needed
  • steps for embedding as a potential institutional business system, business case and encouraging user uptake

Comments would be welcome on all of above from JISC MRD Programme, especially if you have tackled any of these issues already or have lessons learned on what works well (or not) in your own institutional systems or initial testing.

We are also trying to keep up to date (the blog bundle is very useful!) with the outputs from other JISC MRD02 projects (such as DataFinder work from Oxford/DaMaRo, RD@Essesx and Open Exeter, etc.) on data catalogues/data repositories technical development.

Filed under policy, reflection, systems, tools Tagged with , , , , , , , ,

CERIF in Practice Workshop

December 9, 2011 by 5 Comments

I attended the (ARMA UK) CERIF in Practice workshop on Wednesday (7th Dec) and had interesting conversation with Keith Jeffrey from euroCRIS about what sort of information we should attach as metadata to our data in order to maximise its discoverability and utility and he suggested an interesting 3 level taxonomy.

1. The top level relates to the Discoverability of the Resource and could be based around the 15 Dublin Core elements
2. The second level is contextual and essentially covers the elements in cerif (notably; outcomes – publications, patents etc, funding; research council grant, person; project team members, organisation- Uni, collaborators.
3. Finally a specific level giving the minutiae.

This is quite appealing as we already collect much of the information in the first two levels through our current systems (MyProjects, e-prints and MyImpact) so the main additional input we’d be requiring from the academic would be at the third level.

Useful links:
http://dublincore.org/documents/usageguide/elements.shtml
http://www.eurocris.org

Filed under JISCMRD02, systems, tools Tagged with , ,

Research Data Management at Euro Sakai 2011

November 22, 2011 by Leave a comment

I recently visited Euro Sakai 2011 at the Pakhuis De Zwijger in the Eastern Dockland section of Amsterdam, the main purpose of the visit was to find how best we can make the most out of our VRE software (Sakai) and where it is going, but some presentation strands on research data management also caught my interest. It would seem that the sakai crowd are quite intimately intertwined with R.D.M. and I counted at least 2 research data focused presentations and a couple of others mentioning it.

The most useful was from Hull’s Chris Awre, who described Hull’s approach to managing data through its whole lifecycle. They have built a “Fedora” based, versioned, object repository (i.e. we’re NOT talking about the version of red hat linux but an open source object repository… http://fedora-commons.org/) that was developed and managed using “Duraspace” (http://www.duraspace.org/). They claimed this approach was scalable, standards based, content agnostic and allowed the recording of the relationships between objects.

The history of project stemmed from JISC projects:

To summarise, their implementation uses sakai 2.6.4 and Fedora 3.4 (although they also made an integration to talk to sharepoint) and re-uses the sakai resource section as a GUI driver for the fedora repository, squashing the data objects down into a file and directory “view” inside sakai; all the standard CRUD operations are translated through sakai into the repository. The code for all of this is hosted on github. Looking forward, they would like to create an OAE integration with annotation capture on original documents that gets directly packaged up as metadata into the fedora repository.

The main lessons learned were to stay standards based, it makes everything much easier throughout the entire project; and draw up strong policies around what repositories are for and how they are to be used at the very start.

Presentation hosted by:

Chris Awre – c.awre@hull.ac.uk

Useful links/people:

https://github.com/uohull

http://www2.hull.ac.uk/discover/clif.aspx

https://edocs.hull.ac.uk/muradora/objectView.action?pid=hull:4194 (Final report of the CLIF project)

simon.waddington@kcl.ac.uk (King’s College contact for the CLIF project)

Other interesting stuff worth mentioning:

  • Chris Awre also made mention of the Hydra Project, which is an attempt to standardise data object repositories structures in order to enable and aid interoperability of repositories.

Another presentation by the University of Amsterdam and Edia (A private company that helps with sakai integrations and also the conference organisers) discussed the Fluor research data tool; to briefly summarise sections that differed from Hull’s implementation, they have create the “Fluor tool” inside sakai that talks to their library’s Fedora object repository and have also attached the Fedora Generic Search Service to a SOLR implementation in order to allow searching of the repository (although they also cited the possibility of using Lucene or Zebra; SOLR is apparently based on Lucene but is easier to use and supports REST, JSON and XML). Their implementation works with sakai 2.5 up and allows a fine grain access model on a per object basis so data can be as open or closed as is necessary, all data streams holding object data is encrypted (unlike Hull’s) and even their backups are encrypted.

Presentation host by:

Roland Groen (Edia) – http://www.edia.nl/en/edia/founders

Casper Treijtel (UvA) – dpcmedewerkers-uba@uva.nl

Useful links:

http://www.slideshare.net/RolandGroen/fluor-sakai-la-2011

https://confluence.sakaiproject.org/display/CONF2011/Fluor+-+Your+connection+to+the+Fedora+Digital+Objects+Repository

Here’s a rough representation of my understanding of the model university’s are taking when integrating R.D.M. soutions with the V.R.E (interspersed with a couple of my own ideas)….
To briefly explain, starting at the bottom left, you define structures for the context of your data (so how should a medics repository, or a mathematics repository, or a geography repository look?) this helps with organisation of the contents of the repository (and potentially comparison between repositories); then you can use defined transport standards to interact with your repository and wrap it with search and discover functionality and/or general input/output interactions.
In order to make all of this usable you then need to integrate your research focused tools (your VRE, ELN, profile or research project systems, third party tools etc….) to your repository system via a custom link that caters for the connecting tool (e.g. making the objects appear as files/folders for a VRE system) or front it with some sort of service based system that would offer a defined API to talk to your tools.
Andrew Martin
Research and Collaborative Services

Filed under blog report, dissemination, implementation, infrastructure, systems, tools Tagged with , , ,