Cloud Composer architecture — Airflow, DAGs, workers, scheduler

What an environment actually provisions

Cloud Composer's unit of deployment is not a DAG, a job or a cluster - it is an environment. Creating one provisions a complete Apache Airflow installation and everything it needs to stay up: a scheduler, a pool of workers, a web server serving the Airflow UI, a relational metadata database, a Cloud Storage bucket that holds your DAG files, and the plumbing that keeps those pieces talking to each other. You do not install Airflow, you do not run airflow db init, and you do not own the process supervision. What you own is the contents of a bucket and a set of configuration overrides.

The important structural fact - and the one that explains most of the surprises later - is that these components are not all in your project. Composer splits the deployment between resources you can see and bill directly (historically a GKE cluster, always the Cloud Storage bucket, always the Cloud Logging and Cloud Monitoring streams) and resources that live in a Google-managed tenant project attached to your environment (the metadata database and the web server, in the generations that moved them there). You get an environment-scoped view of the tenant side and no direct administrative access to it. That is a deliberate trade: you cannot break the control plane, and you also cannot log into the database to unstick it.

Everything below follows from that split. The DAG bucket is yours, so its consistency model is yours to reason about. The scheduler is managed, so you tune it through configuration overrides rather than by editing airflow.cfg on a box. The database is managed, so metadata bloat shows up as a performance cliff rather than as a disk-full alert you can act on.

The DAGs bucket and the sync you cannot see

Every Composer environment has a Cloud Storage bucket, and its layout is a contract. A dags/ prefix holds the DAG files, a plugins/ prefix holds Airflow plugins, and a data/ prefix is a general-purpose scratch area. Objects written under those prefixes are made visible to the Airflow components through a periodic synchronisation process. This is the single most misunderstood part of Composer, because engineers reason about it as if it were a filesystem mount with immediate semantics, and it is not.

Three properties matter. First, the sync is periodic, not event-driven: uploading an object does not push it to the scheduler, it makes it eligible to be picked up on the next pass. "I pushed the fix" and "the scheduler is running the fix" are separated by a sync interval plus a parse interval, which in practice is anywhere from tens of seconds to a couple of minutes. Second, it is eventually consistent across components: the scheduler, the workers and the web server each have their own copy, and for a window of time they can legitimately disagree about what a DAG contains. A task that fails with an import error that "does not exist in the code" is usually a worker that has not caught up with the scheduler. Third, the sync is per object, which is why a multi-file DAG package uploaded file-by-file can be observed half-written - the scheduler parses a module that imports a helper that has not landed yet, and the DAG disappears from the UI until the next pass fixes it.

The operational discipline that follows: deploy DAG code as an atomic unit (a single archive, or a sync tool that uploads dependencies before the DAG module that imports them), never edit files in the bucket by hand as a hotfix, and treat "the UI does not show my DAG yet" as an expected transient for the first minute rather than as an error to debug.

DAG parsing — the loop that runs your top-level code forever

Airflow does not read your DAG file when a task runs. It reads it constantly. A dedicated DAG file processor walks the DAG directory on a schedule, and for each file it launches a Python process that imports the module, collects the DAG objects defined at module scope, and serialises the resulting structure into the metadata database. The scheduler then works from that serialised representation. This means every line of code at the top level of a DAG file executes on a repeating loop, forever, whether or not any task is scheduled.

That is the origin of the most common Composer performance problem. A DAG file that opens a database connection, calls an external API to build its task list, reads a large configuration object from Cloud Storage, or imports a heavy library at module scope pays that cost on every parse, on every file processor pass. Multiply by the number of DAG files and you have a component pinned at high CPU that never gets to the end of its list. Two configuration values bound the damage: core.dag_file_processor_timeout kills a file that takes too long to import, and scheduler.dag_dir_list_interval controls how often the directory is re-listed for new files. Neither fixes a slow DAG; they only stop one slow DAG from starving the rest.

The fix is structural, not configurational. Keep top-level code cheap: build task lists from static data or from a small file, push expensive lookups inside operator execution where they run once per task run rather than once per parse, and import heavy dependencies inside the callable rather than at module scope. If DAG structure genuinely must be data-driven, cache the driving data in a file that a separate, infrequent job refreshes, so the parse reads a local artefact instead of making a network call. A parse-time network call is a network call made every minute of every day.

The scheduler loop, pools and the limits that stack

Once DAGs are serialised, the scheduler runs a loop: examine DAG runs that are eligible to start, examine task instances whose upstream dependencies are satisfied, decide which of them may run given the concurrency limits, and move those to a queued state for an executor to pick up. It is a scheduling decision problem over rows in the metadata database, and every limit you configure is a constraint in that problem.

The constraints stack, and they are checked in combination rather than in isolation. core.parallelism caps the total number of task instances running across the whole environment. max_active_tasks_per_dag caps concurrent tasks within one DAG. max_active_runs_per_dag caps how many runs of the same DAG can be in flight at once - this is the one that governs backfill behaviour and the one people forget. Pools are the cross-cutting mechanism: a named pool has a slot count, tasks declare which pool they belong to, and a task cannot start unless a slot is free. Pools are how you protect a shared downstream system - give the on-premise database a pool of five slots and no combination of DAGs can open a sixth connection to it, regardless of how much worker capacity exists.

Two failure signatures come out of this loop. A queued backlog - many tasks queued, workers not saturated - means a constraint is binding, not that you need more workers; the constraint is usually a pool or a per-DAG limit. A scheduler heartbeat gap - the scheduler's liveness timestamp falling behind - means the loop itself is not completing, which points at parse pressure or a metadata database that has become slow. Adding workers fixes neither. Read the constraint before you read the fleet size.

Executors, workers and why one task kills five

The scheduler decides what may run; the executor decides where. Composer runs Airflow with a distributed executor: the scheduler publishes queued tasks to a broker, and long-lived worker processes pull from it. Each worker runs up to celery.worker_concurrency tasks at once, so the environment's real task capacity is worker count multiplied by per-worker concurrency, bounded above by core.parallelism. Every one of those slots is a Python process running your task code inside the worker's container, sharing that container's memory limit.

That sharing is where worker OOM comes from, and it is almost never a slow leak. It is one task that reads a result set into a DataFrame, or writes a large file to local disk, or unpickles something big, colliding with other tasks that happened to be running in the same worker at the same moment. The kill is not polite: the container dies, and every task that was executing in it dies with it. Airflow then observes task instances whose heartbeat stopped without a terminal state and marks them zombies, to be retried if retries remain. The tell-tale is a batch of unrelated tasks all failing at the same timestamp with no application-level error - that is not a bug in those tasks, that is a co-tenancy eviction.

The structural answer is to stop moving data through the worker at all. Composer's job is orchestration; the heavy lifting belongs in a system built for it. An operator that submits a BigQuery job and waits, or launches a Dataflow pipeline, or starts a Dataproc job, uses a few megabytes in the worker no matter how many terabytes the job processes. When the work genuinely must run as a container you control, KubernetesPodOperator gives the task its own pod with its own resource request and its own image, isolating it from the shared worker entirely - and that isolation, not the packaging, is the main reason to reach for it.

Sensors, deferrable operators and the triggerer

A large fraction of orchestration work is waiting: waiting for a file to land, for a partition to appear, for an external job to finish. The classic Airflow answer is a sensor, an operator that polls a condition and succeeds when it becomes true. The problem is that a sensor in its default mode occupies a task slot for its entire wait. Fifty sensors each waiting six hours for a daily file consume fifty slots for six hours, and no amount of worker capacity feels like enough because the capacity is being spent on sleeping.

The first mitigation is reschedule mode: instead of sleeping in the slot, the sensor checks once, releases the slot, and is re-queued to check again after an interval. That trades slot occupancy for scheduler churn, and it is a real improvement for long waits with a coarse poll interval.

The structural fix is the deferrable operator and the triggerer component. A deferrable operator does its setup work, hands an awaitable trigger to a separate long-running triggerer process, and exits the worker slot entirely. The triggerer runs thousands of these triggers concurrently in a single asynchronous event loop, because each one is a socket waiting on I/O rather than a process. When a trigger fires, the task is re-queued on a worker to finish. The waiting task consumes no worker slot and no pool slot while deferred - only a small amount of event-loop capacity in the triggerer.

This matters most for exactly the pattern Composer is used for: submit a long-running external job, then wait for it. In the blocking form, a two-hour BigQuery or Dataflow job holds a worker slot for two hours doing nothing but polling. In the deferrable form it holds nothing. If your environment is scaled around wait-heavy DAGs, moving those operators to their deferrable variants is usually a larger win than any amount of worker tuning, and it is the change that makes a wait-heavy environment cheap.

Generations — the direction of travel from Composer 1 onwards

Composer has gone through generations, and the direction of travel is consistent enough to be worth understanding as a trend rather than as a version table. Every generational change moves another component out of infrastructure you operate and into infrastructure Google operates.

The first generation put the whole Airflow deployment on a GKE cluster in your project, and you sized it the way you size a cluster: pick machine types, pick a node count, accept that changing them means recreating or resizing the environment. Scaling was a node-pool operation, the cluster's networking was your networking, and its idle cost was the cost of the nodes whether or not any DAG ran.

The second generation moved the metadata database and the web server into the Google-managed tenant project and put workloads on an Autopilot-mode GKE cluster, which shifts node management to Google - the responsibility split that GKE Autopilot covers in detail. The consequential change was that worker count became a service-side autoscaling decision bounded by a range you configure, rather than a fixed node pool. You declare worker CPU, memory and a minimum and maximum count; the service adds and removes workers as the queued task load moves. Scheduler count became configurable in the same way.

The later direction removes the user-visible cluster from the picture altogether, so the environment is a managed service surface rather than a Kubernetes deployment you happen not to touch. The practical guidance is generation-independent: do not treat the cluster as a supported extension point. Anything you install by reaching into the underlying infrastructure - a mutating webhook, a node-level agent, a manual deployment edit - is exactly what the next generation removes. Extend Composer through DAG code, PyPI package configuration, environment variables and Airflow configuration overrides, all of which survive the generational shift.

Identity — the environment service account and two permission systems

Composer sits at an awkward intersection of two permission systems that do not know about each other, and most access confusion is a failure to notice which one is refusing.

Google Cloud IAM governs the environment as a resource: who may create it, modify it, read its configuration, write to its DAG bucket, and open its Airflow UI. It also governs what the environment's workloads can do to other Google Cloud services, and that is mediated by the environment's service account - the identity the Airflow components run as. Every task that calls a Google Cloud API without configuring anything else calls it as that service account. This is the single most important security fact about Composer: tasks inherit the environment identity by default, so the union of everything any DAG needs becomes the permission set of every DAG in the environment.

Airflow's own RBAC governs what a user can do inside the UI - see a DAG, trigger a run, clear a task, edit a connection, read a variable. It is a separate role model with its own roles, evaluated after IAM has already decided you may reach the UI at all. A user who can open the UI but cannot see any DAG is being refused by Airflow, not by IAM; a user who cannot open the UI at all is being refused by IAM, not by Airflow.

The blast-radius controls follow directly. Grant the environment service account the minimum for orchestration and let individual tasks impersonate a narrower, purpose-specific service account for the work they do, so a permission belongs to a job rather than to the environment. Split genuinely different trust domains into separate environments rather than separate DAGs, because a DAG boundary is not a security boundary when every DAG shares an identity. Store credentials for non-Google systems in Secret Manager and let Airflow's secrets backend resolve connections from it, so the value is never a row in the metadata database. The general model behind all of this is covered in GCP IAM.

Orchestrating the rest of Google Cloud

Composer earns its place by orchestrating things it does not execute. The provider operators for Google Cloud are thin: they authenticate, submit a request, record the resulting job identifier, and then either poll or defer until it reaches a terminal state. Almost none of the data moves through the Airflow worker.

The clearest example is Dataflow. A Composer task submits a pipeline and then waits on the job's state; the pipeline's fusion, autoscaling, shuffle and event-time behaviour are entirely the Dataflow service's business, and are covered in Dataflow architecture. From Composer's side the interesting parts are narrower and worth stating plainly. Submission and completion are separate concerns: a submit-and-forget operator returns success when the job starts, which makes downstream dependencies lie, while a wait-for-completion operator is the one that makes the DAG edge mean what it looks like it means. Retries must account for the job already existing, because an Airflow task retry after a network blip can submit a second copy of a pipeline that is running perfectly well - job naming and existence checks are what make the retry idempotent. And cancellation is not automatic: clearing or failing an Airflow task does not necessarily stop the external job it launched, so a long-running pipeline can outlive the task that started it unless the operator explicitly cancels on kill.

The same shape applies to BigQuery jobs, Dataproc submissions and Cloud Run invocations: the DAG holds the dependency graph, the retry policy, the schedule and the audit trail, and each external service holds the compute. A well-built Composer DAG is mostly a graph of small, idempotent submissions with correct waits between them.

Failure modes and where to look

Composer failures cluster into a handful of recognisable shapes, and the useful skill is mapping a symptom to the component that owns it rather than restarting things.

DAG does not appear, or vanishes and returns. Parse-side. Either the bucket sync has not completed, or the file raises on import, or it exceeds the file processor timeout. The import error surfaces in the DAG processing logs, which are a different log stream from task logs - looking in task logs for a DAG that never got scheduled finds nothing, because no task ever existed.

Tasks queued but nothing starts. A binding constraint: an exhausted pool, a per-DAG concurrency limit, core.parallelism, or workers that exist but are unhealthy. Check the constraint before scaling.

Tasks die together with no application error. Worker OOM or worker eviction, with the co-tenants dying as collateral. Look at worker memory against the container limit and at which tasks were concurrent, not at the task that happens to have the loudest traceback.

Scheduler heartbeat lagging, everything slow. Either parse pressure (top-level code, too many files) or metadata database pressure. The database accumulates a row per task instance, per log entry, per XCom, per DAG run, forever, unless something removes them - and a database that has grown unbounded makes every scheduler query slower, which looks like a scheduler problem and is not. Metadata retention is a maintenance task, not a self-managing property.

Slow, unexplained task startup. Every task launch loads the DAG file again in the worker. Heavy imports pay that cost per task, not per DAG.

Two more worth naming: XCom is metadata, not a data channel - it stores values in the database, so passing a DataFrame through it is a way to make the database everyone's bottleneck; pass a Cloud Storage path instead. And catchup is on unless you turn it off, so deploying a DAG with a start date months in the past schedules every missed interval at once, which is the fastest way to saturate an environment on a Monday morning.

When Composer, and when something else

Versus Cloud Workflows. The axis is what the unit of work is. Workflows is a durable state machine per execution: request-shaped, no fleet, no idle cost, and waiting is free because a paused execution consumes nothing. It is the right tool for a business process triggered by an event - an order, an approval, a webhook - where the steps are API calls and the requirement is that the process survives a crash. Composer is a scheduled DAG engine over data: it assumes recurring intervals, arbitrary Python task code, dependencies between datasets, backfills, and a history you inspect and re-run. If your flow has a schedule, a backfill story, and data lineage, it is a Composer flow; if it has a trigger, a caller and a response, it is a Workflows flow.

Versus Cloud Scheduler plus a service. If the job is "run this container at 02:00", a scheduled trigger against Cloud Run is dramatically simpler and costs nothing when idle. Composer's value begins at the second and third task, where dependencies, partial-failure recovery and per-task retry policy start to matter. One cron job does not justify an always-on Airflow environment.

Versus in-warehouse scheduling. If every step is SQL over data already in BigQuery, scheduled queries or a transformation framework that manages the dependency graph inside the warehouse avoids the external orchestrator entirely. Composer wins when the graph spans systems - land a file, launch a pipeline, load a table, train a model, call an API.

Versus self-managed Airflow. You are buying the operational surface: upgrades, component supervision, database management, and worker autoscaling. You are giving up root access to the runtime and accepting an always-on environment cost. That cost is the deciding factor for small workloads - an environment runs at its minimum size through every quiet hour, so a handful of daily DAGs can cost more than the work they orchestrate.