Migrating Scheduled Workloads from AutoSys to Apache Airflow
January 2023
Problem
Macquarie relied on AutoSys, a legacy enterprise job scheduler, to orchestrate critical data and infrastructure workflows. AutoSys jobs were difficult to version, test, and scale, and the platform was increasingly costly and inflexible to maintain.
Solution
I headed the effort to convert AutoSys job definitions into containerized, code-defined DAGs running on Apache Airflow. This involved mapping legacy job dependencies and schedules into Airflow's Python-based DAG model, containerizing the execution environments for consistency across dev/test/prod, and building a phased migration plan that kept production workflows running throughout the transition. I also completed Astronomer's certifications in both Airflow fundamentals and DAG authoring to guide the platform adoption.
Impact
The migration brought scheduled workloads under version control for the first time, enabled faster iteration and testing of pipeline changes, and reduced operational overhead by replacing a legacy licensed scheduler with an open, container-native platform that the team could extend going forward.