iridium – ‘core’ institutional research data management plan development

Research data management plan authoring is a key part of our draft institutional RDM policy and good practice. Most RCUK funders (apart from EPSRC, currently) require a formal RDMP. NERC now require a pre- and post award RDMP.  These types of templates are available in the DMP Online system.

We wanted to write an institutional RDMP within iridium for research projects that do not have a Funder mandated template (a fair proportion, ~66% of research projects?).  This was to be as easy to complete by end user as possible (i.e. low time burden for researchers) and to be used across Faculties (disciplines) if possible.

What are the ‘essential’ RDMP questions (a ‘core plan’, Donnelley, 2012)? We reviewed several RCUK RDMP templates from different disciplines for similarities, but also distinctive and pertinent questions. Also we had project specific criteria together with good practice from DATUM RDMP template with strong actions and review (‘active plan‘) emphasis.

It was decided to pursue a post-award RDMP template approach for projects without a mandate plan, as less it was burden to write ‘core’ plans for projects that were not awarded in the end and maximise uptake. We noted need for key aspects of RDMP planning to be brought forward in pre-award processes and RIM systems (such as ethics which is already strongly monitored institutionally, but also including RDM costs/atypical data volume size (plus extended curation duration?)). For example, recommending for questions/planning to be a ‘flag’/check-box in  a ‘minimal’ (‘ultra-minimal’?) RDMP check-list in existing RIM systems/pre-award Faculty peer review process.

—- —- —

iridium institutional template post-award RDMP v5 [DRAFT]

This template is for projects that DO NOT have a Funder mandated research data management (RDM) plan. Funding body requirements relating to the creation of a research data management plan are available from …

{ Our institution RIM system MyProjects contains research project administration data (see below). In the long term it would be useful to have this imported and auto-populated into a RDMP direct from RIM system. This aligns to the ‘header’ information in the DMP Online template }

Reference:
Proposal Type:
Proposal Title:
Proposal Short Title:
… ….  … …. etc.

Contact details of named individuals (Role/Name/Unit):

MyProjects Owner:

Date of creation of this plan:
Plan version/supersedes:

 Aims and purpose of plan: … …

[SCOPE NOTES: Guidance on completion of this plan is available from …. ‘DCC 1.x references link to additional guidance provide by the Digital Curation Centre]

1 Introduction and Context
1.1 Introduction and Context
[DCC 1.2]: Short description of the project’s fundamental aims and purpose
[DCC 1.3.2(re-worded)]: Describe how you have considered the Newcastle University RDM institutional policy and any Faculty/research group guidelines, together with any other policy-related dependencies:
[From RC template] Document the RDM advice you have sought on planning your proposed project, including any consultation with projects using similar methods.
[DCC 10.2]: Glossary of terms
2 Data Types, Formats, Standards and Capture Methods
2.1 Data Types, Formats, Standards and Capture Methods
[SCOPE NOTE – for further guidance on ‘data’ definitions and the capture of non-digital data, please see XYZ]
[DCC 2.1]: Give a short overview description of the data being generated or reused in this research
[SCOPE NOTE – for further guidance on ‘open’ file formats, please see …]
[DCC 2.3.3(re-worded)]: Which open file formats will you use, and why?
DCC 2.3.4: What criteria and/or procedures will you use for Quality Assurance/Management?
[SCOPE NOTE – for further guidance on ‘Quality Assurance/Management, please see …]
DCC 2.5.1: Are the datasets which you will be capturing/creating self-explanatory, or understandable in isolation?
[DCC 2.5.2]: If you answered No to [DCC 2.5.1], what contextual details are needed to make the data you capture or collect meaningful?
[DCC 2.5.3]: How will you create or capture these metadata?
[DCC 2.5.4]: What form will the metadata take?
3A Ethics
3A Ethics
HAVE YOU COMPLETED A NEWCASTLE UNIVERSITY ETHICS APPLICATION?[YES] [NO] [NOT APPLICABLE] REFERENCE NUMBER:{ We already have strong RIM/institutional check points for ethics, we don’t want to duplicate information gathering, thus this section is brief. }
3B Intellectual Property
3B Intellectual Property
[SCOPE NOTE – for further guidance Intellectual Property/licensing, please see …]
[DCC 3.2.1]: Will the dataset(s) be covered by copyright or the Database Right? If so give details in DCC 3.2.2, below.
[DCC 3.2.2]: If you answered Yes to [DCC 3.2.1], Who owns the copyright and other Intellectual Property?
[DCC 3.2.3]: If you answered Yes to [DCC 3.2.1], How will the dataset be licensed?
4 Access, Data Sharing and Re-Use
4.1 Access, Data Sharing and Re-Use
[From Research Council template] Are there issues of consent, confidentiality (including commercial), anonymisation and other ethical considerations?
[From RC templates] What are the main risks to data security/ confidentiality?
[DCC 4.2.3]: Are there any embargo periods for political/commercial/patent reasons?
[DCC 4.2.4]: If you answered Yes to DCC 4.2.3, Please give details.
[DCC 4.3.1]: Which groups or organisations are likely to be interested in the data that you will create/capture?
[DCC 4.3.2]: How do you anticipate your new data being reused?
[DCC 5.3.2]: How will you implement permissions, restrictions and/or embargoes?
[DCC 4.1.1]: Are you under obligation or do you have plans to share all or part of the data you create/capture?
[DCC 4.1.3]: If you answered Yes to DCC 4.1.1, How will you make the data available?
[DCC 4.1.4]: If you answered Yes to DCC 4.1.1, When will you make the data available?
[DCC 4.1.5]: If you answered Yes to DCC 4.1.1, What is the process for gaining access to the data?
[From RC template] What will be the responsibilities of data sets users (for example as detailed in a ‘Statement of Agreement’)?
[SCOPE NOTE – for further guidance responsibilities of data sets users and ‘Statement of Agreement’ wording, please see ….]
[DCC 4.1.6]: Will access be chargeable?
5 Short-Term Storage and Data Management
5.1 Short-Term Storage and Data Management
[DCC 5.1.1]: Where (physically) will you store the data during the project’s lifetime?
[DCC 5.1.2]: What media will you use for primary storage during the project’s lifetime?
[From RC template] What is the anticipated (‘ballpark’ figure) of data volume that will be collected? Will this vary after processing?
[DCC 5.2.1]: How will you back-up the data during the project’s lifetime?
[DCC 5.2.2]: How regularly will back-ups be made?
Has the back-up process been tested and successfully validate?
Who is responsible for back-up process?
[DCC 5.3.1]: How will you manage access restrictions and data security during the project’s lifetime?
6 Deposit and Long-Term Preservation
6.1 Deposit and Long-Term Preservation
[DCC 6.1]: What is the long-term strategy for maintaining, curating and archiving the data?
[SCOPE NOTE – for further guidance curation and archiving of data sets, please see …]
[DCC 6.2.1]: Will or should data be kept beyond the life of the project?
What is your deletion policy? Will data sets be deleted? When, by whom and how will they be identified?
[DCC 6.2.2]: If you answered Yes to DCC 6.2.1, How long will or should data be kept beyond the life of the project?
[DCC 6.2.3]: If you answered Yes to DCC 6.2.1, What data centre/ repository/ archive have you identified as the long-term place of deposit?
What is the anticipated (‘ballpark’ figure) of data volume that will be archived?
[DCC 6.2.7]: Will transformations be necessary to prepare data for preservation and/or data sharing?
[SCOPE NOTE – for further guidance data set transformations, please see …]
[DCC 6.2.8]: If you answered Yes to DCC 6.2.7, what transformations will be necessary to prepare data for preservation / future re-use?
[DCC 6.3.3]: Will you include links to published materials and/or outcomes?
[SCOPE NOTE – for further guidance on include links to published materials and/or outcomes, including the Research Data Catalogue, please see …]
[DCC 6.3.4]: If you answered Yes to [DCC 6.3.3], please give details.
[DCC 6.3.5]: How will you address the issue of persistent citation?]
[SCOPE NOTE – for further guidance persistent citation, please see …]
[DCC 6.4.1]: Who will have responsibility over time for decisions about the data once the original personnel have gone?
7 Resourcing
7.1 Resourcing
[DCC 7.1]: Outline the staff/organisational roles and responsibilities for research data management
[DCC 7.2]: How will data management activities be funded during the project’s lifetime?
[DCC 7.3]: How will longer-term data management activities be funded after the project ends?
Describe how funding for RDM has been specifically been costed into funding application (where appropriate).
[SCOPE NOTE – for further guidance on costings for RDM, please see …]
8 Adherence and Review
8.1 Adherence and Review
[DCC 8.1.1]: How will adherence to this data management plan be checked or demonstrated?
[DCC 8.1.2]: Who will check this adherence?
[DCC 8.2.1]: When will this data management plan be reviewed?
[SCOPE NOTE – for further guidance on review points for for RDM plans, please see …]
[DCC 8.2.2]: Who will carry out reviews?
9 Actions Required
9.1 Actions Required
Please list actions and timelines against named individuals identified as a result of completing this RDM plan.
For example please indicate additional hardware, software and relevant technical expertise, support and training that is likely to be needed and how it will be acquired.
For any deferred or unanswered questions outline how you plan to seek advice.
Action: / Responsibility: / Review Date:-: / -: / -:-: / -: / -:
Signature Date
Print name Role/Institution
Signature Date
Print name Role/Institution
Signature Date
Print name Role/Institution

[Attribution]

DMPOnline: https://dmponline.dcc.ac.uk/

© Northumbria University School of Computing, Engineering & Information Sciences, 2012 cc: by-nc-sa DATUM DMP template

© Newcastle University, iridium project, 2012 cc: by-nc-sa

— — — —-

We are currently evaluating end user acceptance of this draft plan, time required to complete and support required to assist with writing.

Advertisements

iridium – evaluation of DataStage and DataBank research data management tools from DataFlow project

DataFlow project background:

DCC catalogue record: http://www.dcc.ac.uk/resources/external/datastage

Two tools

(a) DataStage, for researchers to manage their research data locally.

DataFlow lets researchers save their work to a DataStage file system that appears as a mapped drive on their computer, a lightweight system requiring them to install no special software on their computers.

More details: http://www.dataflow.ox.ac.uk/index.php/datastage/users/researchers

(b) DataBank, to preserve and publish valuable research.

DataStage is a secure personalized ‘local’ file management environment for use at the research group level, appearing as a mapped drive on the end-user’s computer.

More details:  http://www.dataflow.ox.ac.uk/index.php/databank 

Firstly, it’s great that the DataFlow team have released this system openly for re-use. Below are some of our findings.

From a local technical infrastructure assessment:

Ubuntu is not our standard Linux platform (which is Red Hat/CentOS). It would almost certainly be possible to port the Dataflow packages to CentOS (and feed this back to the main project) or use Ubuntu as an appliance (but this would mean that the systems used for this would not be managed by our standard configuration system). Either option comes with a reasonably significant cost.

The feeling that we got from installation (testing prior to 24 July 2012) is that the system is in the early stages of its lifecycle and our assessment is that Dataflow is not yet of sufficient maturity to deploy in production at Newcastle. It would be worth re-evaluating this decision at a later time, this would be prioritised against end users who have tried the system i.e. the more that they liked it, the more worthwhile putting resources into trying it again/working with the DataStage developers.

In terms of initial user testing (in early August 2012/and on ‘v0.3.1rc2’ Oxford installation), initial feedback was:

User testing – DataStage:

Users liked the feature specification of what it offered as a tool (desktop integration through mapped drives, web access aiding working from home, do not need a designated computer for their research work, setting of different access writes (private, public, and collaborative) and the ‘invite to share’ options. System interface is fine, basic yet functional and could be ‘skinned’ to institutional brand. Uploading documents/data files is straightforward.

My opinion was if an institution had no existing RDM systems, it would be a very useful ‘bootstrap’ system providing a simple functional system.

Seamless integration of a data file staging system/VRE with the user desktop (ideally through ‘drag & drop’/mapping over existing user networked drives) and through web access are key features that are top of an ‘average’ researchers wish list.

Making sure research data sets can be appended with an appropriate level of metadata in ‘data staging’ RDM tools (or perhaps later in lifecycle as practical?), so that metadata can flow through to an eventual data catalogue/or national repository is important RDM requirement. Thus, making sure that this function is provided to researchers is important to flag and DataStage/DataBank are a good approach to this.

I thought more data file re-use metadata capture would have been an option in DataStage (noting manifest/Zip package upload feature), pulling in automatically from individual data file itself (that’s probably me being simplistic on technical aspects?) ahead of the DataBank stage?

We noted that not all users are comfortable or had success in Windows drive mapping (network path errors), so some end user support would be needed. Users have high expectations on usability – ‘as easy as DropBox’.

Error messages while testing – access forbidden, 505/405, ‘submit as data package’ – where an entered/saved password was looping? (more helpful customisation of error messages, such as ‘this problem normally occurs because of x, y or z – wrong password, wrong file path, etc.’. (rather than ‘Error 505’/’Error 404’ would be helpful.

User testing – DataBank

 Liked:

– Simple, clean functional interface – again could be ‘skinned’ to instituitional brand.

– Current search/’on-off’ filters was good

– Assigning a DOI/RDF were useful RDM specific features.

– Licensing/embargo fields

– Simple admin interface

– CSV/JSON exports are useful

– Rest API was documented

 Suggestions:

– Clarifying, who was intended user audience for DataBank? Researcher or archivist?

– Terminology – not understood by user testers – ‘Silo’, ‘Mediator’, ‘Aggregate’ – obviously this could be changed easy.

– RDF and click through access to XML schema was confusing for our testers (they were not archivist, librarians, metadata experts – who would probably appreciate this function – i.e. package/manifest upload/explore)

– A basic tagging interface/fields to populate the RDF/XML for none specialists would be more friendly

– Again frequent error messages (404 not found/ 500 Internal Server Error, ‘Add manifest’ gives 505)

Documentation for DataStage/DataFlow researcher end users:

User documentation for researchers seemed a little sparse (I think the project/developers noted it is a work in progress i.e. https://github.com/dataflow/RDFDatabank/wiki). More end user documentation would facilitate wider take up. To note, technical installation documentation was more detailed with screen shares, etc.

We look forward to further DataFlow project developments.

DataFlow user forum is at: https://groups.google.com/forum/?fromgroups=#!forum/dataflow-users

iridium – early findings on research data management planning (approaches, tools and writing plans)

Below is brief summary of some resources, findings and discussions on research data management plans (DMP) that have been noted along the way since the project start up. This has been collected from several activities and events such as iridium support team use of the MANTRA RDM online training package, project RDM tools assessment, together with attendance at the JISC Meeting (Disciplinary) Challenges in Research Data Management Planning Workshop and the DCC Roadshow North East.

Definitions

“Research data management refers to all aspects of creating, housing, delivering, maintaining, and archiving and preserving data. It is one of the essential areas of responsible conduct of research.” – MANTRA

“Plans typically state what data will be created and how, and outline the plans for sharing and preservation, noting what is appropriate given the nature of the data and any restrictions that may need to be.” – DCC

Purpose:

  • to assist in planning the research data management (RDM) aspects of your research
  • to assist you in making RDM decisions
  • to identify the RDM actions required
  • to highlight areas that need further thought
  • to provide a record of decisions made and actions taken

http://www.northumbria.ac.uk/static/5007/ceispdf/dmpguide.pdf

Attribution: Northumbria University School of Computing, Engineering & Information Sciences, 2012. CC-BY-SA

Benefits:

The benefits of managing your data include:

  • Meeting funding body grant requirements.
  • Ensuring research integrity and reproducibility.
  • Increasing your research efficiency.
  • Ensuring research data and records are accurate, complete, authentic and reliable.
  • Saving time and resources in the long run.
  • Enhancing data security and minimising the risk of data loss.
  • Preventing duplication of effort by enabling others to use your data.
  • Complying with practices conducted in industry and commerce.

– MANTRA

Local DMP practice/DAF survey results

From our survey (128 projects), findings were were 23% of projects have a formal research data management plan for institutional as a whole, with further 33% having a partial RDM plan (by Faculty split suggested a slightly higher proportion in line with a likely higher proportion of Research Council awards). I expect this is similar across sector? Open Exeter project reported ‘few researchers have experience of completing a data management plan ‘ from their DAF survey.

Policy

Institutional policies on DMP, some examples (see also DCC website):

Edinburgh, point 3: http://www.ed.ac.uk/schools-departments/information-services/about/policies-and-regulations/research-data-policy

Lincoln (draft), point 4: https://github.com/lncd/RDM-Policy/blob/master/Lincoln%20RDM%20Policy.md

Warwick , point 7: http://www2.warwick.ac.uk/services/rss/researchgovernance_ethics/research_code_of_practice/datacollection_retention/reseatch_data_mgt_policy

Funder polices on DMP:

Various requirements at application and funded project stages. For exampe:

ESRC: http://www.esds.ac.uk/create/esrc/dataman/and http://ukdaresearchdatamanagement.blogspot.co.uk/

NERC: http://www.nerc.ac.uk/research/sites/data/dmp.asp?cookieConsent=A

MRC: http://www.mrc.ac.uk/Ourresearch/Ethicsresearchguidance/datasharing/DMPs/index.htm

See also DCC mappings across 6 funder policies to generic DCC Checklist (July 2011)

Training and guidance on research data management planning

External institutional support pages guidance:

http://www.admin.ox.ac.uk/rdm/dmp/plans/

http://www.ed.ac.uk/schools-departments/information-services/services/research-support/data-library/research-data-mgmt/data-mgmt/why-research-data-policy

http://www.gla.ac.uk/services/datamanagement/creatingyourdata/dataplanning/

MANTRA training package covers DMP.

Advocacy for why DMP is important:

“… the role of data management for a new researcher as being one of those essential skills that you really ought to get at the same time as you learn how to handle your references, as you understand methodology, as you get to grips with the theory that is going to set the frame by which you do your research. And it sits alongside those and it’s equal to them …” – Professor Jeff Haywood, Vice Principal, CIO & Librarian, University of Edinburgh talks about the role of of data management for  PhD students and early career researchers

“… it actually gives you a really good framework and for my postgrads now I am pointing them towards that and saying hey,  you know, take a look at that because it will help you to think about how you’re going to gather your data and how you are going to look after it from  the beginning to the end of the project. It gives you a framework to deal with it rather than realizing too late that you  haven’t done some things that you should have done and  therefore you’ve made your life and perhaps actually cause problems for you with the use of data subsequently or sharing your data is made that more difficult.” – Professor Jeff Haywood, Vice Principal, CIO & Librarian, University of Edinburgh talks about the role of data management for  PhD students and early career researchers

Attribution: EDINA and Data Library, University of Edinburgh. Research Data MANTRA [online course]. http://datalib.edina.ac.uk/mantra

Also, available as a video.

DCC resources:

Discussion on DMP approaches and reviewing the styles of questions,  format, how ‘active’ in approach

Oxford DMPOnline Project wrote on and discussed the detail of research data management plans – very interesting reading. They discussed the concept of ‘plan questions’, ‘project questions’, ‘data questions’ and common issues they found when reviewing DMPs – such as compound questions, duplicates,  and individual plan unique questions [link to table XLS]. On DMP style they noted – discursive versus concise, ‘metadata’ versus ‘data’ questions, option to add possible responses and overall gaps in DMP scope and plans that lead to quantified expected data sizes/acquisition rates (resulting in actionable identification of requirements that can be report to a central service provider, as a result of the plan).

The conclusions should be read:

“.. difficult work, since there are many possible questions ..”

“.. avoid asking ambiguous questions ..”

“.. avoid asking for the same or similar information multiple times..”

“..unique questions not covered by the DMPonline ..”

“.. all of the available question sets have drawbacks ..”

“.. in terms of comprehensiveness, the best may be the enemy of the good enough..”

“.. devise and standardize the best possible set of questions for different constituencies of user ..”

[and more …]

http://datamanagementplanning.wordpress.com/2012/03/27/dmp-questions-comparisons-and-conclusions/

DMPs for different audiences – from targeted plans to template author background ‘bias’/priorities

Life-cycle stage specific: from (conception?), pre-award, post-award, to post-project.

Postgrad research project versus PI bidding for new funding.

Curator/archiver versus researcher orientated.

DMP online authoring tools or offline Word/PDF templates

Online tool has many useful advanced staging, customisation & collaboration features.

Online systems:

DMPOnline – pre-award, post-award, post-project, templates for Research Council/major funders, default templates, post-grad, etc. Features – add additional questions from DCC checklist, save, share/collaborate, copy, export to Office files, etc.

DCC/DMPOnline:

  • Introduction and Context
  • Data Types, Formats, Standards and Capture Methods
  • Ethics and Intellectual Property
  • Access, Data Sharing and Re-Use
  • Short-Term Storage and Data Management
  • Deposit and Long-Term Preservation
  • Resourcing
  • Adherence and Review

DCC website/DMPOnline

DMPOnline tools training:

http://www.dcc.ac.uk/webfm_send/879
http://www.dcc.ac.uk/webfm_send/881
http://www.screenr.com/Syo

DMPOnline advocated by MRC (ref 14), etc.

Institutional customisation or tailoring for local use available.
GitHub code: https://github.com/DigitalCurationCentre/DMPOnline (Ruby on Rails/MySQL)

Offline templates (Word/PDF format):

Some users do not like online systems, are overwhelmed by array of features/customisation options and just want a ready to go familiar Office document to type into.

DATUM: http://www.northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/datum/action/outputs/?view=Standard

Shotton 20 Questions http://datamanagementplanning.wordpress.com/2012/03/07/twenty-questions-for-research-data-management/ [CC:BY 3.0]

Bath 360 (postgrad-specific): http://blogs.bath.ac.uk/research360/2012/03/postgraduate-dmp-template-first-draft/

DMPTPsych(York) (postgrad?): http://www.dmtpsych.york.ac.uk/docs/pdf/dmpt_guidance.pdf

MRC: http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC008617

Wellcome Trust: http://www.wellcome.ac.uk/About-us/Policy/Spotlight-issues/Data-sharing/Guidance-for-researchers/index.htm

Wider DMP discussions

  • extent of pre-population of template with default institutional information to aid researcher versus reducing actual thinking/planning for RDM
  • experience in information banks of DMPs, shared pool, ‘successful’ DMP
  • DMP online tools – metadata transfer protocols between systems/integration in existing RIM systems
  • DMP training needs, online/in person, embedding with training – ‘dual service engagement‘ (i.e. Monash), see DCC ‘support researchers with DMP
  • DMP embedding in existing institutional processes, internal peer review, funder review, DMP fields (i.e. data size) resulting a RIM system flag or automatic central service trigger
  • time requirements for writing a plan – minimal plans/resources required to support/advise/review DMP
  • DMP auditing – institutional, Funding Council, etc.
  • wider use as knowledge/information base for forward institutional planning, storing DMP (or parts of ) with an  archived data set, re-use to support metadata population

JISC Research Data Management Planning Projects

Strand B: DATUM, DMPSPsych, History DMP, etc.

Next steps for iridium project:

Reporting on initial user testing of DMPOnline and other templates, authoring a local DMP template and hosting options.

SWORD v2 – From clueless to claymore

What follows is a summary of my steps along the path of investigating what the sword technology is, through to being able to actually start to code something useful; I should probably point out that the beginning of this post can be consumed by less technical persons as a quick overview, but the later section assumes that you…

  • Have some knowledge of coding java
  • Have worked with java server containers (e.g. tomcat) before
  • Can place the libraries in an IDE like netbeans/eclipse to do “something” with them

(Since my investigations centered around sword in conjunction with Sakai and e-science central my language of choice therefore is Java).

I should also declare that I still don’t fully understand all of the implementation but this should help you along your way if you’re just starting out!


Taking the Sword course

My first port of call was the SWORD website itself, which will point you to some useful videos and slides to give you insight into what the technology is and what it can be used for. In short, this is what the “Sword Course” will teach you…

An Introduction To SWORD (Video/Slides)

What it is:

The “Simple Webservice Offering Repository Deposit” technology (or SWORD for short) intentionally only deals with how to get data into a repository, nothing else, and is complementary to something like dublin core used to describe stuff in a repository; it also does not deal with packaging, metadata, authentication or authorisation

Existing implementations can be found in:

DSpace
Eprints
Fedora
Intralibrary
Zentity

SWORD Use Cases (Video/Slides)

Use cases sword is trying to enable:

  • Deposit from a desktop machine
  • Deposit to multiple repositories (For example to allow depositing once and ending up in an institution’s repository, funder’s repository and a subject specific repository)
  • Deposit from a piece of lab equipment (non-human depositing data)
  • Deposit from one repository to another (For example Institutional repository to National repository which may have differing visibility of the data…. Dark and light repositories, dark = can’t be seen private repositories, light = can be seen public repository)
  • Deposit from external publisher/publishing system to long term storage (For example from OJS to your own institution’s repository)

How SWORD Works (Video/Slides)

  • SWORD is in the form of an “XML REST webservice to put things in a repository”
  • It has built on the resource creation aspects of the ATOM Pub standard which is for publishing content to the web
  • SWORD is an extension, or “profile”, of the ATOM Pub spec and existing ATOM Pub clients can be made to work if the relevant extensions are added
  • SWORD version 2 now includes full CRUD

When you use a sword client, this is basically what happens…

  • The client asks a repository to describe itself
  • The server returns a “Service document” to describe what you can do with the repository and how to do that. The service document is typically hidden by Basic Auth Authentication (AM: I think this is crying out for an OAuth implementation!) but once authenticated the web service will customise the service document to what you are allowed to / should do with the system. The server can also describe what data formats you want to accept, where it will go and how long you will store it etc… this is your “collection policy”
  • The client then uses the service description to format your data and then deposit it

What sword adds to ATOM Pub:

  • Accept Packaging – Tells the client what types of data the server accepts
  • Mediated Deposit – Allows you to deposit “as” / “on behalf of” someone else, a repository can say whether it allows this or not
  • Developer features – You can state that you want verbose output to say what happened (v1.3 featured a dry run feature called “no-op” that does not actually deposit or do anything. NOTE: this does not appear to be in v2 anymore)
  • Nested Service document – Where there may be many repositories for the service, the top level document provides links to sub documents, instead of repeating the same or similar definitions in one enormous file.

SWORD clients (Video/Slides)

There are generally three sorts of client:

  • Machine to machine – for very specific automated deposits (lab equipment)
  • General – human would use, talks to any repository
  • Specific – for depositing certain data into certain repositories in a given way that has an extra context of a general client, i.e. depositing specific journals, depositing data for a particular project

Interesting possibilities for deposit scenarios:

Writing something useful

My next stop on the journey through sword looked at actual code, how it’s laid out and what you need to do in order to start doing something useful.

As mentioned previously, I have been basing my investigations around the Java client and server libraries but I’d strongly recommend you also get a good grounding in the workings of ATOM Pub (HTML Version) and the Sword Profile specifications themselves. If you’ve ever read specification documents before you’ll know they can make quite dry reading, however, since ATOM Pub and sword are relatively straight forward technologies and the specs only reach into 30-50 odd pages it’s really worth a browse through.

How SWORD works in java

Firstly, it’s probably best to understand a few basic concepts you will need to deal with, the main outputted concepts/objects are:

  • IRI‘s – unique identifiers to a resource
  • Entry – A deposit, has IRI’s/metadata
  • Media – An entry for media (word docs/pdfs/images), can be linked to in an Entry
  • Collection Entries – ATOM Pub collection of entries (member resources)
  • Collections – A set of deposited entries represented by an Atom Feed document, you can distinguish a collection feed from “a.n.other feed” by the presence of a collection IRI in service document
  • Workspaces – A compartmentilisation concept for repositories, has a name but doesn’t have IRI’s or processing models
  • Service Documents – Groups “Collections” in “Workspaces”, can indicate accepted media types and categories of collections

Next, I found you learn the most by studying the server libraries, what you get is a bundle of java and some set up files for your container. We’ll firstly look at the setup in web.xml

Setup and Servlet mappings (web.xml)

The main servlets (i.e ultimately your rest endpoints) that are defined are…

servicedocument

collection

  • Class: org.swordapp.server.servlets.CollectionServletDefault
  • URL: http://<yourServer>/<yourWebapp>/collection/*
  • Purpose: Retrieveing and deposting to/from collections/feeds and entries

mediaresource

container

statement

The code makes heavy use of interfaces to allow the implementer more freedom to create functionality using the server library in the way they want to, in order to tell the server libraries what code we want to instantiate and what auth. mechanism we will be using, you must set some context parameters to define those settings and implementations used at runtime:

auth

  • param-name: authentication-method
  • param-value: Basic or None (default: “Basic”)

You can set an Authorization header and base64 encode user:password (e.g. try going to a basic auth encoding website and encode “user:password” in plain text box) and send as a header… “Authorization” “Basic dXNlcjpwYXNzd29yZA==”, or, if you prefer set the param-value to “None” for no authorisation. I found (at the time of writing) the default code actually has a bug which means turning the auth off doesn’t work correctly. I found the best way of correcting this was in my own war project (which includes the server libraries as a dependancy) I created a org.swordapp.sever package (where I was implementing the objects needed for the interfaces) and dropped in a copy of the SwordAPIEndpoint.java to override the implementation in the library, I then changed the getAuthCredentials to…

protected AuthCredentials getAuthCredentials(HttpServletRequest request, boolean allowUnauthenticated) throws SwordAuthException
{
   AuthCredentials auth = null;
   String authType = this.config.getAuthType();
   String obo = "";
   this.log.info("Auth type = "+authType);
   //If we are insisting on "a" form of authentication that is not of type "none"
   if(!allowUnauthenticated && !authType.equalsIgnoreCase("none"))
   {
      // Has the user passed authentication details
      String authHeader = request.getHeader("Authorization");
      // Is there an On-Behalf-Of header?
      obo = request.getHeader("On-Behalf-Of");
      // Which authentication scheme do we recognise (should only be Basic)
      boolean isBasic = authType.equalsIgnoreCase("basic");

      if(isBasic && (authHeader == null || authHeader.equals("")))
      {
         throw new SwordAuthException(true);
      }
      // decode the auth header and populate the authcredentials object for return
      String[] userPass = this.decodeAuthHeader(authHeader);
      auth = new AuthCredentials(userPass[0], userPass[1], obo);
   }
   else
   {
      log.debug("No Authentication Credentials supplied/required");
      auth = new AuthCredentials(null, null, obo);
   }
   return auth;
}

The following context parameters set the implementations of interfaces used to implement functionality in the endpoints. However, you are not given default implementations for each of these, so in your war you need to create a new class that implements the respective interface and fill in your functionality….

collection-list-impl

  • param-value: org.swordapp.server.CollectionListManagerImpl
  • Interface it implements: org.swordapp.server.CollectionListManager

service-document-impl

  • param-value: org.swordapp.server.ServiceDocumentManagerImpl
  • Interface it implements: org.swordapp.server.ServiceDocumentManager

collection-list-impl

  • param-value: org.swordapp.server.CollectionListManagerImpl
  • Interface it implements: org.swordapp.server.CollectionListManager

collection-deposit-impl

  • param-value: org.swordapp.server.CollectionDepositManagerImpl
  • Interface it implements: org.swordapp.server.CollectionDepositManager

media-resource-impl

  • param-value: org.swordapp.server.MediaResourceManagerImpl
  • Interface it implements: org.swordapp.server.MediaResourceManager

container-impl

  • param-value: org.swordapp.server.ContainerManagerImpl
  • Interface it implements:  org.swordapp.server.ContainerManager

statement-impl

  • param-value: org.swordapp.server.StatementManagerImpl
  • Interface it implements: org.swordapp.server.StatementManager

config-impl

  • param-value: org.swordapp.server.SwordConfigurationDefault (Yes, this one does have a default implementation in the library you can use)
  • Interface it implements: org.swordapp.server.SwordConfiguration

Endpoint Servlet classes (org.swordapp.server.servlets.*)

Let’s now have a look at the servlets themselves, each servlet contains interfaces which are implemented by loading the classes specified in the web.xml (see above).

All servlets used in the server library extend the “SwordServlet” which (obviously) extends the HttpServlet. Since the SwordServlet contains the server configuration object, all servlets (through inheritance) also hold an implementation of the server configuration (i.e. SwordConfiguration) and a method to allow servlets to load classes from the configuration….

SwordServlet encapsulates:

  • SwordConfiguration interface, instantiated using config-impl
  • loadImplClass() method used for loading implementing classes from tomcat context params

CollectionServletDefault extends SwordServlet and encapsulates:

  • CollectionListManager interface, instantied using collection-list-impl
  • CollectionDepositManager interface, instantied using collection-deposit-impl
  • CollectionAPI object

ServiceDocumentServletDefault extends SwordServlet and encapsulates:

  • ServiceDocumentManager interface, instantiated using service-document-impl
  • ServiceDocumentAPI object

MediaResourceServletDefault extends SwordServlet and encapsulates:

  • MediaResourceManager interface, instantiated using media-resource-impl
  • MediaResourceAPI object

ContainerServletDefault extends SwordServlet and encapsulates:

  • ContainerManager interface, instantiated using container-impl
  • StatementManager interface, instantiated using statement-impl
  • ContainerAPI object

StatementServletDefault extends SwordServlet and encapsulates:

  • StatementManager interface, instantiated using statement-impl
  • StatementAPI object

Endpoint Servlet dependant classes (org.swordapp.server.*)

Those with a keen eye will have noticed that each servlet is also holding an “API” object, these objects fill out the standard Get/Post/Put/Delete HttpServlet methods that the servlets override by taking the configuration object and any interfaces that have been implemented and combine them to do something useful. Similarly to the servlets, they all extend a Sword API super class called “SwordAPIEndpoint”, which holds a SwordConfiguration implementation. The hierarchy (and interfaces they encapsulate) looks like this…

SwordAPIEndpoint

  • SwordConfiguration interface

CollectionAPI extends SwordAPIEndpoint

  • CollectionListManager interface
  • CollectionDepositManager interface

ServiceDocumentAPI extends SwordAPIEndpoint

  • ServiceDocumentManager interface

MediaResourceAPI extends SwordAPIEndpoint

  • MediaResourceManager interface

ContainerAPI extends SwordAPIEndpoint

  • ContainerManager interface
  • StatementManager interface

StatementAPI extends SwordAPIEndpoint

  • StatementManager interface

Implementations of interfaces (org.swordapp.server.*)

I keep mentioning the objects that implement the interfaces, I thought it might be useful to go through in “slightly” more detail what the content of those objects are intended for. Apologies once again, this is not exhaustive as I have not worked my way through what all the methods are intended for:

SwordConfigurationDefault implements org.swordapp.server.SwordConfiguration

  • This is the default object which holds the configuration for the server

CollectionListManagerImpl implements org.swordapp.server.CollectionListManager

CollectionDepositManagerImpl implements org.swordapp.server.CollectionDepositManager

ServiceDocumentManagerImpl implements org.swordpapp.server.ServerDocumentManager

  • Method: getServiceDocument()
  • Accessed via: GET http://<yourServer>/<yourWebapp>/servicedocument/
  • Returns: org.swordapp.server.ServiceDocument
  • Purpose: Serves service documents (xml that explains the contents and deposit policies for the repository(/ies)

MediaResourceManagerImpl implements org.swordapp.server.MediaResourceManager

  • Method: replaceMediaResource()
  • Accessed via: PUT http://<yourServer>/<yourWebapp>/edit-media/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Swap a media resource (pdf/doc etc….) in the repository with the one being “PUT’ed”

ContainerManagerImpl implements org.swordapp.server.ContainerManager

  • Method: replaceMetadataAndMediaResource()
  • Accessed via: PUT http://<yourServer>/<yourWebapp>/edit/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Replaces metadata and media associated with an entry
  • Method: addMetadataAndResources()
  • Accessed via: Does not appear to be “directly” accessible via any specific HTTP request
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Not used by the ContainerAPI yet, but presumably it would be for adding a series of entries and associated metadata
  • Method: addResources()
  • Accessed via: Does not appear to be “directly” accessible
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Not used by the ContainerAPI yet, but presumably it would be for adding a series of entries
  • Method: useHeaders()
  • Accessed via: POST http://<yourServer>/<yourWebapp>/edit/
  • Returns: org.swordapp.server.DepositReceipt
  • Purpose: Used when depositing only information specified in the HTTP headers, no entry/ies (i.e. no content body to the POST) will have been specified

StatementManagerImpl implements org.swordapp.server.StatementManager

And finally…

Once you have all that setup (and I’d recommend just creating skeleton override methods for the objects implementing interfaces for the time being whilst you figure the code out), you can then start coding the abdera / sword code and try make the client do something. The client itself comes with a handy cli driven (SwordCLI) interface that you can point at your newly created server instance and test the various example methods. I would recommend though, that you comment out the entire list of method references in the main method and go through the list iteratively to slowly make each part of your server work…

As a brief example, if we were to try and get a basic service document to work, try adding this code to your ServiceDocumentManagerImpl.java….

    public ServiceDocument getServiceDocument(String sdUri, AuthCredentials auth, SwordConfiguration config) throws SwordError, SwordServerException, SwordAuthException
    {
        //Our test service document
        ServiceDocument sd = new ServiceDocument();
        //sd.setVersion("2.0");
        sd.setMaxUploadSize(1000000000);

        //Our test workspace
        SwordWorkspace sw = new SwordWorkspace();
        sw.setTitle("TestWorkspace");

        //Our test collection
        SwordCollection sc = new SwordCollection();
        sc.setHref("http://<yourServer>/<yourWebapp>/collection/TestCollection");
        sc.setTitle("TestCollection");
        sc.addAccepts("*/*");
        sc.setCollectionPolicy("TestCollectionPolicy");
        sc.setAbstract("TestCollectionAbstract");
        sc.setMediation(false);
        sc.setTreatment("A human-readable statement describing treatment the deposited resource has received or a URI that dereferences to such a description.");
        sc.addAcceptPackaging("http://purl.org/net/sword/package/SimpleZip");
        sc.addAcceptPackaging("http://purl.org/net/sword/package/METSDSpaceSIP");
        sc.addAcceptPackaging("http://www.ncl.ac.uk/terms/package/html");
        sc.setLocation("http://<yourServer>/<yourWebapp>/collection/TestCollection");
        sc.setMediation(true);
        List iris = new ArrayList();
        iris.add(new IRI("http://<yourServer>/<yourWebapp>/collection/TestCollection/TestSubService"));
        sc.setSubServices(iris);

        //Add collection to workspace
        sw.addCollection(sc);
        //Add workspace to service document
        sd.addWorkspace(sw);

        return sd;
    }

Browsing to http://<yourServer>/<yourWebapp>/collection/ should return something, and removing your comment in your client for the line…

    cli.trySwordServiceDocument()

…should now yield some results (If you just get errors try temporarily turning off the auth. requirement on the server for the purposes of testing).

And that’s the basic principle, you then take the specs and implement the returning and unpackaging of ATOM using the abdera/SWORD objects and link what’s passed/returned to the content found in the system you are trying to integrate.

Further reading

Sword Specs
Brief history of Sword
More useful Sword docs
Another set of slides explaining sword

Andrew Martin
Research and Collaborative Services
Newcastle University

%d bloggers like this: