Skip to content

Changelog

release/5.0.0

Breaking Changes

  • Removed parallel argument from node.run(): Use max_workers parameter instead to control parallel execution. Set max_workers=1 for sequential execution or max_workers=N for parallel execution with N workers. You can also set FLYPIPE_NODE_RUN_MAX_WORKERS environment variable to configure this globally (defaults to 1).- Removed FLYPIPE_DEFAULT_RUN_MODE config: This config previously defined parallel or sequential execution mode. Now use FLYPIPE_NODE_RUN_MAX_WORKERS instead to control parallelism. Set it to 1 for sequential execution or N for parallel execution with N workers. This can be set globally via environment variable or passed directly to node.run() as the max_workers parameter.- Updated Cache class method signatures: The read() and write() methods now include optional CDC (Change Data Capture) parameters: - read(from_node=None, to_node=None, *args, **kwargs): Added from_node and to_node parameters for CDC filtering - write(*args, df, upstream_nodes=None, to_node=None, datetime_started_transformation=None, **kwargs): Added df, upstream_nodes, to_node, and datetime_started_transformation parameters for CDC metadata tracking - All custom cache implementations must update their method signatures to include these parameters, even if CDC functionality is not used (parameters can be ignored)- Added create_cdc_table() method to Cache class: A new optional method for caches with CDC support. The base implementation is a no-op, so existing cache implementations will continue to work, but cache classes that support CDC should override this method to create their CDC metadata tables.

    Commits

  • 209 🚀 Feature Request: Implement CDC Cache in Flypipe

release/4.3.6

release/4.3.5

release/4.3.4

release/4.3.3

release/4.3.2

release/4.3.1

release/4.3.0

release/4.2.0