Or you can create your own Operator to do the job and take the parameters. In which level the dynamics is? Task level? DAG level? (dagid, session, includeexternallytriggeredFalse)source Returns the last dag run for a dag, None if there was none. So you probably need to figure out the basic idea: Create a Timetable instance from a scheduleinterval argument. 'worker_machine_type': CFG,ĭefault_args=default_args, # pass the params to DAG environmentīut if you want dynamic dags rather than arguments, you may need other strategy like this. 'cluster_name': CFG, # edit here according to the structure of your yaml file. Os.path.dirname(os.path.realpath(\_file_)), 'your_yaml_file') from airflow import DAG from import chain from airflow. Just simply create a params dictionary then pass to default_args: CONFIGFILE = os.path.join( ![]() A tag name per dag, to allow quick filtering in the DAG. So actually, you don't need XCOM to get the arguments. A dag (directed acyclic graph) is a collection of tasks with directional dependencies. Not sure what you mean for 'dynamic', but when yaml file updated, if the reading file process is in dag file body, the dag will be refreshed to apply for the new args from yaml file.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |