What are the potential challenges and resources available for implementing a MATLAB model training pipeline using Apache Airflow?

Hi Team,

I'm developing a pipeline for MATLAB model training using Apache Airflow. I have a few questions:

1. Is it possible to create a pipeline for MATLAB training in Airflow?

2. Are there any potential bottlenecks in this implementation?

3. If it's doable, is there any documentation available on this specific use case?

Any insights or resources would be greatly appreciated. Thanks!

Answer 1

[Jacob Mathew]

Hey Sai,

Considering that Apache Airflow has no first party documentation on how to connect to MATLAB and neither does MathWorks, it is unlikely that you will be able to directly integrate both of them.

Apache Airflow does mention that it is based on an extensible Python framework. Hence your best bet would be to write the training pipeline in Python integrating with Apache Airflow. Then within the Python script, you can call MATLAB using the following documentation as your guide:

https://www.mathworks.com/help/matlab/matlab-engine-for-python.html

https://www.mathworks.com/products/matlab/matlab-and-python.html