Inputs

In this document we describe the inputs to Sprout at a technical level. For a more general description of what domain we expect data to come from and how it may look like conceptually, see the input data document.

The objects we primarily incorporate in the interface are paths, properties, and data, as described in the naming document.

Paths

These objects are always Path objects from Python’s pathlib.

Properties

These are Python dataclasses we’ve created to represent the properties (metadata) of a data package or resource. They always use the *Properties suffix, such as SproutProperties and ResourceProperties. While SproutProperties holds general Data Package properties, we intentionally avoided a name such as DataPackageProperties to communicate that Sprout uses stricter criteria for some properties than the full flexibility that is available in the Data Package specification.

Data

These data input objects are always Polars DataFrame objects.

Sprout expects the data objects to be in a tidy format. Tidy data is a conceptual framework around how data should be structured to be optimally usable for analysis. This is similar to what is called third normal form in relational database development, excluding the emphasis on creating additional tables to reduce redundancy.

Tidy data has the following properties:

  1. Each variable forms a column.
  2. Each observation forms a row.
  3. Each cell has a single value that represents both the column and row entity.

This tidy structure makes it easier to process and use the data for later analysis. If it is already in this form, there is less need to transform it later.

We have set a requirement on using tidy data frames because it simplifies the workflow and allows users to use their own, potentially highly customised, workflows for processing their data. This also allows for a more flexible and user-friendly interface to Sprout, since it removes the need to support or consider all the wide variety of formats data can be stored or structured in. Because data processing and organizing can be so strongly dependent on the domain, we leave this to the user to handle.

Staging directory

This directory contains data for data resources that are tidied and processed from the raw data. These are bulk collections of data that are processed together. This staging/ folder is where the data is stored before it is processed and saved in the data.parquet file. This folder makes the history between raw data and the final data resource clearer. This can help with troubleshooting issues that may arise during data entry or processing or for auditing purposes. The default location is ./staging/<resource-name>/.

Staging files

These are Parquet files that store data in a structured, columnar format (see our decision post for why we use Parquet files). These files are created when timestamped raw data has been tidied and processed in preparation for being added to a data resource. These files are usually kept in a file path in the format staging/<resource-name>/<timestamp>.parquet. Files in staging/ are not linked within the datapackage.json file. Every time a user adds or updates the raw data, these staging/ files should also be updated. Before being joined into a data resource, they are checked against the properties.