How to make the airflow task fail?

Question

I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all the records have been processed successfully. Task doesn't work otherwise

 def process_csv_entries(csv_file): # Boolean file_completely_parsed = <call_to_module_to_parse_csv> return not file_completely_parsed CSV_FILE=<Sets path to csv file> t1 = PythonOperator(dag=dag, task_id='parse_csv_completely', python_operator=process_csv_entries, op_args=[CSV_FILE]) 

t1 seems to succeed regardless of the return value. How to make PythonOperator task crash?

Answer 1

an exception occurs when you meet the error condition (in your case: when the file is not processed with sufficient detail)

 raise ValueError('File not parsed completely/correctly') 

raise the appropriate type of error using the appropriate message