Martin Schobben, Department of Geodesy and Geoinformation, Technical University of Vienna, Austria
Ton Smeele, Online Research Labs, Data Management, Utrecht University, Netherlands
Terrell Russell, iRODS Consortium, Chapel Hill, United States of America
A detailed account of the individual team member’s interests and skills can be found in Section 5.1.
One of the biggest problems in research is the inadvertent destruction of data and the inaccessibility of data due to poor labeling and description of data. This loss of data means that studies cannot be replicated, combined or re-used in different settings (Briney 2015; Wilkinson et al. 2016). Data management becomes more important, but also more challenging, in the age of rapid digital data production. The Integrated Rule-Oriented Data System (iRODS) (Rajasekar et al. 2010, 2015) is an open sourced data management software suite that offers a solution to this demand, which encompasses the whole data life cycle from data generation, storage and recycling. The loosely constructed and highly configurable architecture of iRODS frees the user from strict formatting constrains and single-vendor solutions. Furthermore, the open-sourced and community supported nature of iRODS safeguards data storage and re-usability of data in terms of longevity and independently of future technological innovations.
Nonetheless, the learning curve of how to implement iRODS effectively for day-to-day data management purposes can be steep for the average scientist. Hence there have been multiple incentives to lower this barrier by hiding parts of the behind-the-scenes business logic which requires considerable knowledge of command line tools. A notable example is the UU developed YODA front-end, which simplifies the scientist’s tasks to meet their datamanagement goals. But, it still requires some effort to integrate this data representation with workflows for the analyses of data. Highly popular among academics are higher level programming languages, such as R, that can help construct such data analyses workflows. Currently, there is no straightforward solution to integrate iRODS data storage with R workflows. The development of the R package rirods (Chytracek et al. 2015), build on top of the iRODS C++ API, is currently stale and suffers from strict system requirements constrains, which makes it unsuitable for an easily installable and distributed method. This, furthermore, hinders the publication of rirods on the Comprehensive R Archive Network (CRAN; the official R package archive), as interoperability among operating systems is mandatory for an R package to be eligible for publication.
Recently, a REpresentational State Transfer (REST)full API has been build for iRODS. REST architectures allow connecting services at global scale (Fielding 2000), as it enables clients (users) to interact with their data via a web-based architecture with the common HTTP verbs. HTTP is an interaction protocol for the Web, where verbs like GET, PUT, POST and DELETE can interact with the resource representation through a Uniform Resource Identifier (URI). This allows for a loose coupling between the datamanagment system (iRODS) and the user, which means that the consuming application does not have to be build on the same system, or even on the same operating system. Similarly to data stored on iRODS, using the web means also that the representation of the resource does not need to conform to a stringent format, and can be any, or many, formats (e.g., text, XML, and/or JSON). This user friendliness and lose coupling allows for a radical new approach to bind iRODS to R. We therefore propose to build a new R package for iRODS; rirods, to replace the former package, and, which, allows for a truly distributed computing solution by using iRODS RESTfull API. We foresee that this R package will allow for an integrated R workflow and iRODS data management solution that will aid open practices in e.g., science.
In order for the R package to communicate with the REST API of iRODS we make use of the libcurl library, which facilitates transfer of data over HTTP. Conveniently there already exist R wrapper packages around the libcurl library, notably: curl (Ooms 2023) and httr2 (Wickham 2022). This makes it possible to construct a minimal dependency R solution to establish e.g., secure connections through HTTP with TLS and parallel requests (async multi-download functionality). At the side of the iRODS server, the iRODS C++ REST Mid-Tier API will handle the requests and translate them into sensible representations similar to iRODS’ native iCommands. Authentication occurs through an end-point that queries the native authentication module for iRODS, which then returns an Json Web Token that allows connecting with a remote iRODS server. Future versions will also implement support for the PAM stack, which will greatly increase the configurability in terms of the authentication process.
Development: A local iRODS server is build for convenience during development and testing of the package. For this purpose the irods_demo Docker will be used and run as a localhost. In addition, Utrecht University will accommodate a Virtual Machine with an iRODS server to more faithfully mimic the requirements of querying a remote server.
Unit testing: To enable publication of rirods on the CRAN server, unit tests should be able to be executed without internet connection. To achieve this a mocking technique will be used that imitates HTTP responses (httptest2 Richardson 2022) by saving snapshots of the “real” API’s response for future comparison of test results. This process of generating mock snapshots will be automatized by integration in a GitHub Workflow so that changes in the REST API’s response can be accommodated whilst performing unit tests in future stages of rirods development.
Documentation: Technical documentation will be streamlined with the usage of roxygen2 (Wickham, Danenberg, et al. 2022), which produces MAN pages for the newly developed functions that can be queried from the R console. In addition, long format documentation is included in the form of so-called vignettes, which can as well be consulted from the command line. In addition, the function MAN pages and vignettes will be published as a pkgdown website (Wickham, Hesselberth, and Salmon 2022) hosted with GitHub Pages and automatically updated with a designated GitHub Workflow.
The development of rirods will lower the technical barriers of implementing iRODS data management capabilities as and integral part of R data analysis workflows. Thus rirods is meant to be conducive to innovation and foster inclusiveness through open practices in e.g., science. In turn, R package development encompasses soliciting domain-specific use-cases to optimize the design. A close collaboration with the department of Online Research Labs, Data Management at Utrecht University (UU), the Department of Geodesy and Geoinformation at the Technical University of Vienna, the National Institute for Public Health and the Environment of the Netherlands, and the Wageningen University & Research is already foreseen. Nonetheless, the development will benefit from having a good overview of the types of data and data models produced by analytical equipment in a range of academic, governmental and industrial settings. We will opt for an MIT license and a code of conduct, which will follow the Contributor Covenant guide lines. Combined this ensures that contributions to the package can be done in a safe, inclusive, welcoming, and harassment-free environment conducive for collaborative package development, and ensuring down-stream re-usage of the developed software. Reporting of the progress of the project to both users and developers will help ensure that we stay on track and thus develop a solution that has a broad future implementation.
The duration of the project will be 8 months. The “deliverable” gives a convenient measure of project’s progress.
Months 1–3
R CMD check and with continuous integration provided by GitHub Workflows.Months 4–6
Months 7–9
Months 10–12
We will garner attention on the impact of data management practices on open science, and our proposed solutions to mitigate common problems, through several channels. We actively seek to advertise the product by presenting our finding at conferences; either user-specific (science conference) and/or the developer community (e.g. iRODS general user meeting). In addition, we intend to develop course material and give (online) workshops to instruct users on the usage of the R package.
The realisation of this package requires a collaborative environment that includes the potential users, and their specific requirements for processing analytical data, as well as developers and data scientist with expertise in a range of disciplines. In regards to development, we brought together a multi-disciplinary team, and consulted experts of data management and the R software packaging.
The project team will try to form a comprehensive picture of the current state of data management practices in laboratories through direct interaction with representatives from academia, the public sector and industry. In addition, they take control in all steps of development, documentation and outreach of the package. Dedicated consultants have been contacted and their expertise is regarded as an essential aid for successful deployment of the plan.
The project lead (MS) is an Earth scientist with 10 years of experience in academic research, and he has worked in several analytical laboratory facilities (MfN Berlin, University of Leeds, and Utrecht University). He also has a solid basis in data-analysis and programming with R, and has started developing packages for analysing isotope chemical data (see point). Teaching and helping others to encode R solutions has been another of his passions, such as the development of workshops, and by founding of an R help desk at the UU (uu-code-club).
TR oversees the iRODS development team and handles code review, package management, documentation, and high level architecture design. He’s interested in distributed systems, metadata, security, and open source software that accelerates science. TR holds a Ph.D. in Information Science from the University of North Carolina at Chapel Hill and has been working on iRODS since 2008. In his current role, he also provides management and oversight of the iRODS Consortium.
A prime controller in the initiation of the project is early publication of the R package on GitHub. Feedback on these early developments is sought actively through our dedicated list of consultants, but also the community at large through Twitter and other channels. Throughout this process, we will make sure that the code of conduct, as outlined in Section 4.1, is adhered to.
For successful delivery of the package we need a suitable platform for development testing this will be accommodated by using Docker container as well as a Virtual Machine run on a UU server. GitHub is essential for the collaborative character of the work.
We request funding for personal costs for Martin Schobben (0.2 fte) to develop, test and role-out of the rirods package.
| Item | Currency | Price |
|---|---|---|
| personal cost (M Schobben) | € | 20,000 |
| total | € | 20,000 |
Support is requested for the development, documentation as well as outreach of the rirods package.
The deliverable “Installable package on CRAN” of Section 4.2 defines achievement of the minimal viable product (Solution 1; Section 3.2).
The actual success during the development phase is measured by the number of contributions and the number of research institutes, laboratories and universities that we can engage with. The success of the developed package is measured by use-cases through download statistics, and for development purposes, by tracking how many packages will integrate this package.
It will enable consulting and implementing better resources for data management in academic, governmental and industrial settings as well as help teaching efforts focusing on data management and reproducible practices. In addition, we consider developing course material, give (online) workshops, and continuously advertise usage of the package by actively engaging with target user-base at conferences and on social media.
Problems and delays in terms of coordinating community feedback (especially desired use-cases) and contributions (mainly solutions as listed above) could stem from a lack of consensus on the specific solution to be adopted. Hence we aim to address this at the earliest stages of the project (Months 1–2).