Flows

Important

We created this document mainly as a way to help us as a team all understand and agree on what we’re making and what needs to be worked on. This means that the flows may change quite substantially until we’ve reached a stable full release at v1.0.0.

Based on the functions page that lists and describes the main functions and classes that make up the interface in detail including their input and output, this document describes and shows how all these objects work together and flow into one another.

Each diagram uses specific shapes and lines to represent different things:

• A dotted line between objects/functions means the flow is optional.
• A solid line between objects/functions means the flow is required and is the next step.
• A cylinder means an object such as a class or file.
• A rectangle with rounded corners means an action like a function or method.

Caution

For some reason, the diagrams below don’t display well on some browsers like Firefox. To see them, try using a different browser like Chrome or Edge.

Creating or updating a package

This is the flow for making a new package. The write_properties() function will internally check the package properties. The SproutProperties input is imported from a Python script in the scripts/ folder of the data package. To update the properties in the datapackage.json file, you would edit the Python script with the properties directly and then rerun your main script to overwrite the datapackage.json file with the new properties.

flowchart
    properties[(SproutProperties)]
    path_properties("PackagePath().properties()")
    write_properties("write_properties()")

    properties --> write_properties
    path_properties --> write_properties

Figure 1: Diagram showing the flow of objects and functions to create a new package.

Checking data against the properties

The data must always match what is described in the properties. This means that the data must have the same column names, column types, and column constraints. The check_data() function will internally call several separate functions for these specific checks. Each of these functions outputs an error message describing what the problems are if the check fails. Otherwise, the input data frame is returned unchanged.

flowchart
    data[(DataFrame)]
    resource_properties[(ResourceProperties)]
    check_data("check_data()")
    subgraph Internal
        check_column_names("_check_column_names()")
        check_column_types("_check_column_types()")
        check_column_values_constraints("_check_column_values_constraints()")
        output[("DataFrame<br>or error")]
    end

    data --> check_data
    resource_properties --> check_data
    check_data -.-> Internal
    check_column_names -.-> output
    check_column_types -.-> output
    check_column_values_constraints -.-> output
    style Internal fill:#ffffff

Figure 2: Diagram showing the flow of objects and functions to check data against the properties.

Creating or re-creating the resource data

The batch data files are used to keep track of changes to the data. The data that will be used is kept clean and ready for analysis, while original data is not deleted. This flow converts batch data into the final resource data file. The steps are split up so that, if needed or desired, you can make modifications to the data before it is written to the final resource data. While write_resource_data() will call check_data() internally, this can also be called separately to check the data before writing it to the final resource data.

flowchart
    path_resource_batch_files("PackagePath().resource_batch_files()")
    read_resource_batches("read_resource_batches()")
    list_data[("List[DataFrame]")]
    join_resource_batches("join_resource_batches()")
    joined_data[(DataFrame)]
    write_resource_data("write_resource_data()")
    resource_properties[(ResourceProperties)]
    check_data("check_data()")

    path_resource_batch_files --> read_resource_batches
    read_resource_batches --> list_data
    list_data --> join_resource_batches
    join_resource_batches --> joined_data
    joined_data --> write_resource_data
    resource_properties --> write_resource_data
    joined_data -.-> check_data
    resource_properties -.-> check_data

Figure 3: Diagram showing the flow of objects and functions to create or re-create the resource data.