datasaver

Reference Documentation

class hamilton.function_modifiers.datasaver

Decorator for specifying a data saving function within the Hamilton framework. This decorator is used to annotate functions that save data, allowing them to be treated specially in the Hamilton DAG (Directed Acyclic Graph). The decorated function should return a dictionary containing metadata about the saving process.

The datasaver decorator captures saving data metadata and ensures the function’s return type is correctly annotated to be a dictionary, where the dictionary contains metadata about the data saving process, that then is exposed / captures for the Hamilton UI / adapters.

Example Usage:

Assuming you have a function that saves data to a JSON file and you want to expose the metadata in your Hamilton DAG to be captured in the Hamilton UI / adapters:

import pandas as pd
from hamilton.function_modifiers import datasaver


@datasaver()  # you need ()
def save_json_data(data: pd.DataFrame, json_path: str = "data/my_saved_data.json") -> dict:
    '''Saves data to a JSON file and returns metadata about the saving process.

    :param data: The data to save.
    :param json_path: The path to save the data to.
    :return: metadata about what was saved.
    '''
    # Save the data
    with open(json_path, "w") as file:
        data.to_json(json_path)

    # Metadata about the saving process
    metadata = {"destination": json_path, "format": "json"}

    return metadata

This function can now be used within the Hamilton framework as a node that saves data to a JSON file. The metadata returned alongside the data can be used for logging, debugging, or any other purpose that requires information about the data saving process as it can be pulled out by the Hamilton Tracker for the Hamilton UI or other adapters.

generate_nodes(fn: Callable, config) → List[Node]

Generates same node but all this does is add tags to it. :param fn: :param config: :return:

validate(fn: Callable)

Validates that the function output is a dict type.