export AIRFLOW__CORE__XCOM_BACKEND="include.custom_xcom_backend.S3XComBackend" Use code with caution.
Simple design:
The TaskFlow API makes exclusive XComs cleaner by hiding the push/pull logic, but the discipline remains: airflow xcom exclusive
Periodically clear old XCom data. Airflow keeps historical XCom metadata indefinitely unless explicit database maintenance scripts or DAGs are set up to prune the xcom table.
In child DAG, exclusive mode ensures only keys passed via conf are accessible. export AIRFLOW__CORE__XCOM_BACKEND="include
def extract_api_data(**context): # Fetch data and write to temporary location temp_table = f"temp_data_context['ds_nodash']" write_to_bigquery(temp_table) return temp_table # Single string: the exclusive reference
+-------------------+ Returns Object/Data +-----------------------+ | Upstream Task | --------------------------------> | Custom XCom Backend | +-------------------+ +-----------------------+ | +---------------------------+---------------------------+ | Serialize & Upload Payload | Save Metadata Pointer v v +-----------------------+ +-----------------------+ | Cloud Object Storage | | Airflow Metadata DB | | (S3 / GCS / Azure) | | (Stores JSON URI) | +-----------------------+ +-----------------------+ Architecture of a Custom Backend In child DAG, exclusive mode ensures only keys
def try_claim(session, claim_id, worker_id): row = session.execute(update(claim_xcom) .where(claim_xcom.c.id==claim_id) .where(claim_xcom.c.status=='available') .values(status='claimed', claimed_by=worker_id, claimed_at=func.now()) .returning(claim_xcom)).first() return row # None if already claimed
Excessive XCom writes create high I/O concurrency, leading to database locks and slower scheduler loops. Designing "Exclusive" XCom Workflows
The TaskFlow API allows you to pass data between tasks automatically, making the code much cleaner and removing the need for manual xcom_pull commands.