read_staging

read_staging(
    resource_properties: ResourceProperties,
    paths: list[Path] | None = None,
)

Read all staging file(s) into a list of (Polars) DataFrames.

Use this function to read the Parquet file(s) specified in paths into a list of Polars DataFrames, and perform checks on each of the DataFrames against the resource_properties. The resource_properties object is used to check the data and ensure it is correct. This function also runs checks to ensure the data are correct by comparing to the properties.

Parameters

resource_properties : ResourceProperties

The ResourceProperties object that contains the properties of the resource you want to check the data against.

paths : list[Path] | None = None

A list of paths for all the Parquet files in staging/ folder. Use PackagePath().staging_files() to help provide the correct paths to these files. Defaults to the staging files of the given resource.

Returns

list[pl.DataFrame]

A list of DataFrame objects from all the staging files.

Raises

ValueError

If the staging file name is not in the expected pattern.

ValueError

If the timestamp column name matches an existing column in the DataFrame.