Design

The purpose of these documents is to describe the design of Sprout in enough detail to guide a sustainable implementation (i.e., maintainable in the long term). It also ensures that the team has a shared understanding of the package’s scope and architecture.

Sprout’s design builds off of our overall Seedcase design principles and patterns, in that any design we create for Sprout should strongly follow the same design principles and patterns described there.

There are two main sections of this design documentation:

• Architecture: The architecture section describes the high-level requirements, components, use cases, expected users, and how the system will be organized and structured.
• Interface: The interface section gives a much more detailed description of the design of the software interface. The detail is at the level of specific Python functions, as well as their arguments, inputs, and outputs.

Purpose

The overall aim of Sprout is to take data and metadata that has been prepared and cleaned from a raw state into “staged” state and process it into a standardized and organized data package structure.

Concretely, Sprout:

1. Processes staged data and metadata into a final data package.
2. Ensures that metadata is correct and that the data matches the metadata.
3. Extracts metadata from staged data.
4. Creates files from templates to simplify filling in with metadata.

Requirements

Overall, Sprout must:

• Handle various sizes of data. The size of research data is often not as large as in industry, but it can still be large enough to require specific care and handling.
• Handle processing and building different overall structures of data (e.g. Parquet partitions) and types of files (e.g. images).
• Process and store tabular data that is in Parquet files and is tidy.
• Allow for data updates from source locations, such as batch updates from a survey or whenever a laboratory sends data after analyzing biological samples.
• Ensure metadata is a core component of a data package by preventing data from being processing without it. Metadata are necessary to understand the data and its context. Without it, data can be near useless.
• Integrate into reproducible data engineering workflows to ensure a transparent and auditable history of how the data and metadata was generated into the final data package.
• Allow for the “right to be forgotten”. If someone requests their data to be deleted or no longer used for any purposes, data owners are legally required to comply with this request. While removing all traces of someone’s data can be or often is near impossible, legally the data should at the least not be included in or distributed for any analyses.

In general, Sprout will not:

• Do any analytic computing or data science work. While some data processing and analysis will occur, it will be limited to running checks on the quality of the data and metadata.