Materials science // data interaction
Data Curation Workflow
(see Appendix below for additional links and resources)
Curation involves selecting, organizing, and looking after items in a collection.
You can think of our data curation workflow as one big ETL process comprising several smaller ETL processes. The source of data is typically a human-readable data representation such as a table or chart embedded in a PDF document, without the raw data itself. The goal of this workflow is to make data machine-interpretable (via digital extraction and metadata annotation) so that the data become searchable.
To curate data from published research articles, the process is decomposed into specific stages and tasks that enable us to manage data quality over time. Depending on the complexity of the datasets or the capabilities of the database schema or ontology at a particular point in time, this process can be highly iterative!
Workflow at a Glance
The graph below shows the workflow, where nodes represent stages of curation and edges describe tasks. Hover over labels for more detailed descriptions.
xWishlistWishlistInitializedInitializedWishlist->InitializedTravelerSetupOutlinedOutlinedInitialized->OutlinedNotebookSetupDigitizedDigitizedOutlined->DigitizedDataGatheringWrangledWrangledDigitized->WrangledWranglingAnnotatedAnnotatedWrangled->AnnotatedSemanticsTemplatedTemplatedWrangled->TemplatedMappingAnnotated->WrangledSearchableSearchableAnnotated->SearchableIntegrationSearchable->OutlinedSearchable->DigitizedSearchable->WrangledSearchable->AnnotatedTemplated->WrangledTemplated->SearchableUpload
What is the Curation Traveler?
The "traveler" is a digital document (with the filename "README.md") in Markdown format that provides documentation throughout a curation job's life cycle. Because of the diverse range of tasks involved in data curation, the process benefits from collaboration. The traveler allows curators to log the process and keep track of the the status and any issues during curation.
The example below shows what the curation traveler looks like when opened in a text editor such as Notepad. We use the filename README.md because Github will recognize and automatically render the markdown formatting when you view the directory in the remote repository.
Curation Traveler
* curation_job_id: lee2012tailoring
* DOI: 10.1021/ma3016085
* status: NotebookSetup
* notebook: https://observablehq.com/d/b3114cc98be438ca
LOG
2021-04-15: Initialized traveler [MED]
2021-04-30: Initialized notebook, moved to NotebookSetup [MED]
The log section of the Traveler is there for curators to document the curation process. It is a good idea to update the Traveler when updates are made to the repo, such as a commit.
Using Github to Organize Curation
Git is a powerful tool for version control ubiquitously used on collaborative software projects. Here, we use a Github repository (repo) to remotely collaborate and stay organized. Each curation job has its own directory containing the curated data and metadata, supporting code, and a curation traveler. You can use Github Desktop, a GUI that helps you more easily organize commits containing multiple files.
Setting up a Curation Job
Once you have a DOI for a research article, you can view the article page by visiting doi.org/ {insert DOI here} in your browser (you will be automatically redirected to the article page). Searching the DOI on a service such as Google Scholar should return the article as a search result, where you can view bibliographic information, a portion of the abstract, and linked articles. Here, you can retrieve the BibTeX information (click on the closing quotation icon, then BibTeX) and use that ID as the curation job ID.
The example below shows the BibTeX information for lee2012tailoring:
@article{lee2012tailoring, title={Tailoring the photomechanical response of glassy, azobenzene-functionalized polyimides by physical aging}, author={Lee, By Kyung Min and Koerner, Hilmar and Wang, David H and Tan, Loon-Seng and White, Timothy J and Vaia, Richard A}, journal={Macromolecules}, volume={45}, number={18}, pages={7527--7534}, year={2012}, publisher={ACS Publications} }
Create a folder in the Curation Jobs directory with the curation job ID as the folder name. In this folder, you can save the BibTeX information (e.g. as a text file called scholar.txt) and the Curation Traveler (README.md).
If you use Google Scholar, add library links via the settings in order to easily access articles through your institution log-in. You can save the research article and article images in this directory within your local repository, but note that the Github repository has been configured to ignore any files with .pdf, .jpg, or .png file extensions, so these types of files will not appear in the remote repository.
Setting up a Notebook
This template can be used for setting up a new Observable notebook. Simply create a Fork of the template and fill out the relevant information. The notebook can then be shared with collaborators using Observable's new share feature. Setting up the notebook involves defining which .csv files will be expected and how those files will be rendered. For visual charts, we use Vega-Lite specifications to create an interactive version of the chart. (Creating Vega-Lite specs requires familiarity with the language, but there are templates in the repository that provide boilerplate examples to build upon.)
Digital Extraction of Data
For data tables, you can use Tabula to auto-extract the information (although, it is important to check the extracted output for errors). For charts such as line and scatter plots, we use WebPlotDigitizer (WPD) to decode the data from a chart image. WPD allows projects to be saved (as .tar files), which allows us to collaborate and save progress on data extraction tasks. In order to tidy the output from WebPlotDigitizer into a format that can be interpreted by Vega-Lite, we use the home-built TidyCatWPD tool (link to tutorial on app page).
Semantic Annotation of Metadata
We are trying out a new method for providing context to a dataset through Semantic Data Dictionaries (SDDs). This involves assembling a multi-sheet workbook in Excel or Google Sheets that describes the semantic relationships between fields of a tabular dataset. The SDD exists as a piece of metadata alongside a dataset which defines how the dataset should be interpreted, and is compiled to a SETLr script.