> ## Content Index > Fetch the complete content index at: https://hiteshsharma.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Before the Pod Exists: How Kueue Decides Whether an AI Job May Run - URL: https://hiteshsharma.dev/before-pod-exists-kueue-admission-state-machine-2026/ - Published: 2026-10-17T22:06:47.000Z - Updated: 2026-10-17T22:06:47.000Z - Description: Learn how Kueue admits Kubernetes AI and batch jobs, manages scarce GPU quota, and hands workloads to kube-scheduler. - Author: Hitesh Sharma - Tags: Distributed Platforms, AI Systems You submit a distributed training job to Kubernetes. It needs eight GPU workers, so you expect either eight Pods or eight useful scheduling errors. Instead, nothing happens. The Job remains suspended. There may be no Pods, no application logs, and no message from kube-scheduler because kube-scheduler has not received anything it is allowed to place yet. That can be normal in a cluster managed by Kueue. Kueue adds a workload-admission decision before ordinary Pod scheduling. Its job is to answer a question Kubernetes does not answer by itself: **should this complete workload be allowed to start now?** Only after that answer becomes yes does the usual scheduling machinery decide which specific nodes will run the Pods. This distinction matters most on shared AI, ML, HPC, and batch clusters. A partially started web service may still provide reduced capacity. A partially started synchronous training job may do no useful work while holding several expensive accelerators. A platform therefore needs to reason about the whole job, team quotas, hardware classes, topology, borrowing, priority, and preemption before releasing Pods into the cluster. The central idea of this article is simple: > “Pending” is not one Kubernetes state. It is a sequence of decisions owned by different control planes. Once you understand those decisions, Kueue stops looking like an extra scheduler with an intimidating object model. It becomes a durable state machine that governs the handoff from submitted work to schedulable Pods. ## What Kueue is [Kueue](https://kueue.sigs.k8s.io/docs/concepts/?ref=hiteshsharma.dev) is a Kubernetes-native workload queueing and admission system maintained under Kubernetes SIG Scheduling. It is designed for applications that run to completion: Kubernetes Jobs, JobSets, Kubeflow training jobs, Ray workloads, Spark applications, plain Pod groups, and other supported integrations. The unit Kueue reasons about is a `Workload`, not an individual Pod. A Workload describes all the Pod sets that must be considered together, their resource requests, their counts, their priority, and the queue through which they request admission. Kueue does four broad things: 1. It keeps supported jobs suspended or their Pods scheduling-gated while they wait. 2. It orders Workloads and evaluates them against quota and policy. 3. It assigns resource flavors and optionally validates topology or external capacity. 4. It releases the original job only after its admission requirements are satisfied. Kueue is **not** a replacement for kube-scheduler. It does not normally bind a Pod to a node. It is also not a training framework, workflow engine, cluster autoscaler, checkpointing system, or application-health controller. Its responsibility is narrower and more foundational: decide when a workload may enter the execution phase. ### Before and after Kueue Imagine two teams submit eight-worker GPU training jobs to a cluster with eight available GPUs. Without a workload-admission layer, both jobs can create Pods immediately. If each job obtains four GPUs, neither synchronous training job can begin. All eight GPUs are allocated, but the cluster is doing no useful training. With Kueue, both jobs first become Workloads. Kueue admits one complete eight-worker request and keeps the other suspended. The admitted job can make progress; when it finishes or releases quota, the next job can start. This is the basic motivation for all-or-nothing admission. The real system becomes more complicated because “eight GPUs” is not one interchangeable pool. They may belong to different models, nodes, racks, tenants, or capacity reservations. Kueue’s object model exists to represent those distinctions. ## Where Kueue sits in the control plane A useful mental model is a chain of controllers with explicit ownership boundaries: ```mermaid flowchart TB A["User or workflow system"] --> B["Job / JobSet / RayJob / supported object"] B --> C["Kueue integration
Suspend Job or add scheduling gate"] C --> D["Workload"] D --> E["LocalQueue"] E --> F["ClusterQueue"] F --- G["Cohort
Shared quota"] F --- H["ResourceFlavor
Resource class"] F --> I["Kueue scheduler
Order, assign flavors, reserve quota"] I --> J{"Required checks ready?"} J -- "No" --> K["Wait for AdmissionChecks,
topology, or capacity"] K --> J J -- "Yes" --> L["Admitted=True"] L --> M["Unsuspend Job or remove scheduling gate"] M --> N["kube-scheduler
Bind Pods to nodes"] N --> O["Job controller and application
Run, retry, checkpoint, finish"] ``` For a standard Kubernetes Job, the Kueue integration works with `spec.suspend`. For plain Pods, Kueue can use a Kubernetes scheduling gate. The mechanism differs, but the boundary is the same: Kueue prevents ordinary placement until admission succeeds. This creates the first operational rule: - If the Workload is not admitted, investigate Kueue’s objects, conditions, policies, and capacity checks. - If `Admitted=True` but Pods remain `Pending`, move to the Job, Pod, node, storage, and kube-scheduler branch of the investigation. The [Kubernetes Headlamp troubleshooting guide](https://kubernetes.io/blog/2026/10/14/tracing-kueue-admission-with-headlamp/?ref=hiteshsharma.dev) uses exactly this handoff. Headlamp makes the object graph easier to navigate, but the underlying evidence remains standard Kubernetes API state. ## Where Kueue is used Kueue is not limited to a queueing demo around `batch/v1 Job`. Public integrations show several different ways the admission pattern is being used. ### IBM Research Vela A [Red Hat and IBM engineering account](https://developers.redhat.com/articles/2025/05/22/improve-gpu-utilization-kueue-openshift-ai?ref=hiteshsharma.dev) describes Kueue in the control plane for Vela, IBM Research’s AI supercomputer. Teams receive nominal quotas, borrow unused quota through cohorts, and use priorities and preemption under contention. The interesting architectural detail is the boundary around Kueue. In the disclosed MLBatch stack, Kueue owns the outer suspended/admitted loop. AppWrapper handles the running job’s lifecycle, while Autopilot detects infrastructure faults. The reported 90% GPU utilization belongs to that combined system under active load. It should not be read as a benchmark of Kueue alone. This example demonstrates that admission, application recovery, and hardware health are separate loops even when they contribute to one platform outcome. ### GKE TPU Multislice Google’s [GKE TPU Multislice tutorial](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/tpu-multislice-kueue?ref=hiteshsharma.dev) combines JobSet and Kueue to run distributed JAX workloads across multiple TPU slices. Kueue decides when JobSets may run according to resources, quotas, and fair-sharing hierarchy. The important point is not that a sample manifest runs. It is that a distributed JobSet can be admitted as one resource shape. Releasing only some slices would be wasteful because the complete training topology is the useful unit. ### KubeRay The [KubeRay integration](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/kueue.html?ref=hiteshsharma.dev) covers RayJob, RayCluster, and RayService. It demonstrates gang admission and priority preemption. It also shows a more subtle division of labor for elastic workloads: Kueue manages admission and quota for the Ray cluster, while Ray’s autoscaler changes worker demand within the admitted model. Some of those elastic paths depend on alpha feature gates, so integration breadth should not be confused with uniform maturity. Still, the design illustrates why Kueue is a control-plane component rather than an application scheduler. ### AWS SageMaker HyperPod Kueue’s maintained [ecosystem index](https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/v0.19/docs/ecosystem/%5Findex.md?ref=hiteshsharma.dev) identifies SageMaker HyperPod task governance as a Kueue-based integration. [AWS exposes](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance.html?ref=hiteshsharma.dev) teams, resource allocations, priorities, preemption policies, borrowing and lending, capacity, and task wait time. Public material does not reveal the entire internal architecture or adoption scale, but it shows the same policy model being surfaced as a managed platform capability. Together, these examples establish the useful scope of Kueue: multi-tenant clusters where whole jobs compete for scarce, structured resources. A small cluster running a few independent services may not need the additional CRDs, controllers, and operational vocabulary. ## The object graph Kueue’s API becomes easier to remember when every object is assigned one question. | Object | Question it answers | | ------------------------- | --------------------------------------------------------------- | | Workload | What complete unit is asking to run? | | LocalQueue | Through which namespace-level tenant entry point did it arrive? | | ClusterQueue | Which shared quotas and admission policies govern it? | | ResourceFlavor | Which class of resource may satisfy the request? | | Cohort | Which ClusterQueues may share unused quota? | | AdmissionCheck | Which external or additional guard must approve it? | | Topology / TAS assignment | Can the requested Pods fit in acceptable physical domains? | ### Workload: the durable admission record The Workload is the center of the model. Its specification contains queue identity, priority, Pod sets, counts, and Pod templates. Its status records the selected ClusterQueue, flavor assignments, admitted resource usage, optional topology assignments, admission-check states, and conditions. This makes admission durable. A queue decision is not only an in-memory result inside a scheduler loop; it is reflected in a Kubernetes object that other controllers can reconcile after restarts. The Workload conditions must not be collapsed into one boolean: - `QuotaReserved` says Kueue has assigned a ClusterQueue and reserved the relevant quota. - `Admitted` says quota is reserved **and** all required admission checks are ready. - `Evicted` records that an admitted or reserved Workload must stop and release its position. - `Requeued` describes whether it can return to admission consideration. - `PodsReady` concerns the later execution-readiness phase. - `Finished` closes the lifecycle. The separation between `QuotaReserved` and `Admitted` is particularly important. A workload can hold quota while an AdmissionCheck waits for external capacity, policy approval, or multi-cluster placement. ### LocalQueue: the tenant-facing entry point A LocalQueue is namespaced. A team submits its jobs to a LocalQueue without needing to encode the cluster’s full resource-governance structure in every workload. The LocalQueue points to a cluster-scoped ClusterQueue. That indirection lets multiple namespaces or teams use different entry points while sharing centrally administered quota and policy. ### ClusterQueue and ResourceFlavor: policy meets supply A [ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster%5Fqueue/?ref=hiteshsharma.dev) defines the covered resources, flavor-specific nominal quotas, borrowing and lending limits, ordering strategy, namespace eligibility, preemption, fair sharing, and admission checks. A ResourceFlavor describes a class of resources. It can distinguish GPU models, CPU architectures, pricing classes, node pools, taints, labels, or topology definitions. During admission, Kueue assigns flavors to the resources requested by each Pod set. Quota is therefore not merely “the team has eight GPUs.” It is closer to “this ClusterQueue may consume eight units of this GPU flavor, may borrow four more from its cohort, and may use only namespaces and workloads allowed by its policy.” ### Cohort: controlled borrowing A Cohort groups ClusterQueues that may share unused nominal quota. Nominal quota provides an ownership baseline. Borrowing improves utilization when an owner is idle. Lending and borrowing limits constrain how far sharing may go. This is useful, but it creates policy questions that a single static quota avoids. Who gets displaced when the owner returns? Which Workloads are safe to preempt? How much completed computation will be lost? Kueue can enforce a preemption decision; it cannot make application state cheap to reconstruct. ## The admission state machine The main execution path looks like this: ```mermaid stateDiagram-v2 [*] --> Submitted Submitted --> Held: suspend Job or gate Pod Held --> WorkloadCreated: create Workload WorkloadCreated --> Queued: resolve LocalQueue and ClusterQueue Queued --> Inadmissible: no fit, inactive queue, or policy block Inadmissible --> Queued: capacity or policy changes Queued --> Preempting: fit requires reclamation Preempting --> Queued: victims release resources Queued --> QuotaReserved: flavor, quota, and optional topology fit QuotaReserved --> QuotaReserved: AdmissionChecks pending QuotaReserved --> Admitted: all required checks Ready QuotaReserved --> Evicted: check rejected or reservation invalidated Admitted --> Released: unsuspend Job or remove gate Released --> PodScheduling: create or release Pods PodScheduling --> PodsReady: kube-scheduler binds Pods PodsReady --> Finished: workload completes Admitted --> Evicted: preemption or readiness timeout Evicted --> Queued: eligible for requeue Finished --> [*] ``` Every arrow is reconciled state, not one transaction across Kueue, kube-scheduler, an autoscaler, and the workload controller. That matters during conflict, restart, deletion, or partial failure. The official [admission documentation](https://kueue.sigs.k8s.io/v0.20/docs/concepts/admission/?ref=hiteshsharma.dev) describes two broad admission phases: 1. **Quota reservation.** Kueue evaluates the target ClusterQueue, resource flavors, logical quota, and physical availability when Topology-Aware Scheduling is used. 2. **Admission checks.** Required checks must reach `Ready` before the Workload becomes admitted. A temporary AdmissionCheck failure can release quota and requeue the Workload. A permanent rejection can evict it and require explicit reactivation. “Waiting after quota reservation” is therefore a real phase, not an inconsistent intermediate state. ## Inside one Kueue scheduling cycle The v0.20.2 source makes the scheduler’s structure unusually explicit. In [pkg/scheduler/scheduler.go](https://github.com/kubernetes-sigs/kueue/blob/v0.20.2/pkg/scheduler/scheduler.go?ref=hiteshsharma.dev), one cycle does roughly this: 1. Pop the heads of eligible queues. 2. Take a snapshot of the scheduler cache. 3. Nominate flavor assignments, borrowing, topology choices, and possible preemption targets. 4. Order the resulting candidates, including fair-sharing rules when enabled. 5. Process each candidate against the shared snapshot. 6. Requeue anything not successfully reserved or admitted. Two design choices are worth examining. ### The scheduler works from a snapshot The snapshot gives one scheduling cycle a coherent model of ClusterQueues, cohorts, flavors, and current reservations. As entries are processed, the cycle books their expected usage into that model. That means a flavor assignment computed during nomination may need to be recomputed later in the same cycle. An earlier Workload may have consumed the capacity, or two candidates may have selected overlapping preemption targets or topology domains. The `processEntry` path explicitly rechecks fit and can defer the Workload to another cycle. This is more than defensive programming. It prevents a decision computed against the beginning of the cycle from being treated as valid after the cycle itself has changed the assumed state. ### Assume first, persist second When a candidate fits, Kueue constructs an `Admission`, updates the scheduler cache as though the Workload holds the reservation, and then patches the Workload status asynchronously. The source comment is precise: the scheduler’s `admit` function does **not necessarily make the Workload admitted**. It sets the quota reservation; `Admitted` is synchronized only when all required checks are ready. The API write can also conflict if the Workload changed while the scheduler was evaluating it. In the strict path, Kueue uses the observed resource version so an outdated decision fails instead of overwriting newer state. The failed assumption is then removed or the still-valid reservation is restored for another pass. This is the distributed-systems core of Kueue. Queueing correctness depends on agreement among: - the Workload version the scheduler evaluated; - the resource request and effective Pod specifications; - the quota charge derived from that request; - the in-memory reservation; - the persisted Workload status. ## Quota is a promise; placement is evidence Base Kueue admission reserves aggregate quantities. Aggregate quota does not prove a valid node-level placement exists. Suppose the ledger shows eight GPUs free. They are actually split across two nodes with four GPUs each. A Workload containing one Pod that requests eight GPUs satisfies the aggregate arithmetic but cannot be placed because a Pod cannot span nodes. The [all-or-nothing scheduling documentation](https://kueue.sigs.k8s.io/docs/concepts/all%5For%5Fnothing/?ref=hiteshsharma.dev) uses this kind of fragmentation to explain why several mechanisms exist: - **Quota-based admission** asks whether the complete Workload fits within the governed resource quantities. - **Topology-Aware Scheduling (TAS)** asks whether its Pods can fit within acceptable node or topology domains before admission. - **ProvisioningRequest AdmissionChecks** can ask an external provisioning system to create capacity. - **kube-scheduler** selects the concrete node for each released Pod. - **`waitForPodsReady`** bounds how long an admitted workload may fail to become ready before eviction and requeue. These mechanisms are complementary. Calling all of them “gang scheduling” hides where each one acts. [TAS](https://kueue.sigs.k8s.io/docs/concepts/topology%5Faware%5Fscheduling/?ref=hiteshsharma.dev) uses node allocatable capacity, readiness, schedulability, admitted TAS workload usage, and other Pod usage to calculate free capacity by topology domain. It can keep an impossible shape out of the admitted state and can prefer placements with better locality for communication-heavy jobs. TAS improves the admission decision, but it still operates on observed control-plane state. Nodes can fail, capacity can change, and later controllers still participate. Placement-aware admission reduces uncertainty; it does not turn the whole system into one atomic scheduler. ## Ordering, fairness, borrowing, and preemption “Run the highest-priority job first” is not a complete multi-tenant policy. Kueue must decide what happens when the oldest Workload cannot fit, when one tenant has consumed more than another, when unused quota can be borrowed, and when already running work may be reclaimed. ### Strict FIFO versus Best Effort FIFO With strict FIFO ordering, an older Workload that cannot fit can block later Workloads in the same queueing context. This preserves a strong interpretation of order but can leave usable capacity idle. Best-effort FIFO allows a later Workload to proceed when it fits. That can improve utilization, but a large Workload may wait while a stream of smaller requests repeatedly finds gaps. The choice is a policy trade-off between predictability, starvation risk, and utilization—not a tuning flag with a universally correct value. ### Fair sharing Kueue documents two related mechanisms. Admission fair sharing can order Workloads using historical resource consumption by their source LocalQueues. Preemption-based fair sharing uses dominant-resource-share reasoning to decide which borrowing ClusterQueues may reclaim capacity from others. Fairness is therefore state over time. Priority answers how important one Workload is. Fair sharing asks whether its tenant has already consumed a disproportionate share of the governed pool. ### Preemption has an application cost Preemption restores policy only by interrupting work. For a checkpointed training job, that cost may be acceptable. For a workload with an expensive restart or no useful checkpoint, reclaiming quota can destroy hours of computation. Kueue can choose and evict the victim. The surrounding platform must decide whether that action is economical and make the application recoverable. ## What v0.20.2 teaches about correctness [Kueue v0.20.2](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.20.2?ref=hiteshsharma.dev), released on October 15, 2026, is useful because several fixes expose the invariants underneath the normal state machine. ### 1\. A Workload and its quota charge must describe the same generation A Dynamic Resource Allocation path could requeue an updated Workload while retaining quota charges computed from its previous specification. The scheduler might then over-admit or under-admit work. The general invariant is stronger than “recalculate on update”: > Object version, effective resource request, and computed quota charge must remain coherent through the entire scheduling attempt. If that coherence cannot be proved, the decision must return to the controller for recomputation. ### 2\. Every reservation needs one complete lifecycle A JobSet restart path could leave an evicted Workload holding quota and not requeue it when recreated child Jobs had already succeeded. The resulting damage spreads beyond one JobSet. Other Workloads see capacity as reserved even though no corresponding job can make progress. The invariant is: > A reservation must be intentionally retained, transferred, or released on every eviction, restart, deletion, and completion path. ### 3\. Gated work must remain recoverable A Pod-owned Workload could be preempted or evicted while its Pod was still scheduling-gated, then remain parked with `Requeued=False` forever. No node-placement failure was involved. The problem was a disagreement between admission state and the lifecycle of a Pod that had not crossed the gate. The invariant is: > An evicted Workload cannot remain both ineligible for requeue and responsible for work that has never been released. ### 4\. Fair-sharing preemption must respect nominal ownership Another fix prevented Workloads within nominal quota from being preempted merely because their ClusterQueue borrowed on an uncontested flavor or inside an internal sub-cohort. The invariant is: > Borrowing elsewhere in the policy tree must not erase the protection of resources currently consumed within nominal quota. ### 5\. Observability state has a lifecycle too The release also removed stale metric series for deleted or relabeled queues and corrected cohort-related cleanup. A stale quota metric does not corrupt admission directly, but it corrupts an operator’s model of admission. In a control plane, telemetry identity must follow object identity closely enough that deleted or moved policy objects stop appearing as current capacity. These bugs do not prove that Kueue’s core is broadly unreliable. They do show why “it is just a priority queue” is the wrong mental model. Kueue maintains replicated, derived state across caches, API objects, controllers, integrations, and metrics. ## A causal debugging method When a job does not start, inspect the state machine in order. Do not begin by guessing whether the cluster needs more nodes. | Observation | Current owner | Next question | | --------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | No Workload exists | Job integration or queue selection | Was the job submitted to a valid LocalQueue, and is its integration enabled? | | QuotaReserved=False | Kueue scheduler and queue policy | Does the Workload fit covered resources, flavors, quota, namespace policy, and ordering? | | QuotaReserved=True, Admitted=False | AdmissionCheck or related controller | Which required check is not Ready, and is it retrying, rejected, or waiting for capacity? | | Admitted=True, Job still suspended | Job integration controller | Did it reconcile the admission assignment into the original job? | | Admitted=True, Pods Pending | kube-scheduler and cluster infrastructure | Which node constraint, taint, affinity, volume, image, or physical capacity condition blocks placement? | | Pods started but workload is not useful | Job controller, application, or infrastructure-health loop | Are all workers ready, communicating, checkpointing, and making progress? | Start with the Workload’s conditions and events: ```bash kubectl describe workload -n kubectl get workload -n -o yaml ``` Then follow the references rather than scanning unrelated cluster state: ```bash kubectl get localqueue -n -o yaml kubectl get clusterqueue -o yaml kubectl get resourceflavor -o yaml ``` Conditions describe the current state. Events explain recent attempts and transitions. Controller logs can explain implementation errors. Metrics show fleet-level patterns, but they should not outrank the object that owns one Workload’s state. Once `Admitted=True`, cross the boundary deliberately: ```bash kubectl describe job -n kubectl get pods -n kubectl describe pod -n ``` This ordering prevents two common mistakes: adding nodes to solve a quota-policy denial, and editing queue policy to solve a node-placement failure. ## What Kueue cannot promise Kueue can make admission more deliberate, fair, and observable. It cannot guarantee the outcome of every later layer. It cannot guarantee: - that a quota-only admission has a valid physical placement; - that a node remains healthy after a topology assignment; - that images, volumes, or networks become ready; - that a distributed application makes progress after all Pods start; - that preempted work restarts cheaply or correctly; - that an external AdmissionCheck responds promptly; - that every integration has identical suspend, resume, elasticity, and completion semantics; - that the whole path is atomic across independent Kubernetes controllers. Kueue also has a real operational cost. It adds CRDs, webhooks, queues, caches, feature gates, metrics, conditions, and another controller manager whose availability matters. That cost is justified when scarce resources, multiple tenants, or all-or-nothing workloads require governance. It may be unnecessary for a small cluster whose independent Jobs can safely compete through ordinary scheduling. ## The complete mental model Kueue is easiest to understand as a scheduler before the scheduler, but the phrase needs one qualification. It does not schedule Pods. It schedules **permission to begin**. The Workload is the durable request. The LocalQueue identifies the tenant entry point. The ClusterQueue and ResourceFlavors define governed supply. Cohorts permit controlled borrowing. The Kueue scheduler orders candidates and reserves quota against a snapshot. AdmissionChecks and TAS can add external policy and physical-placement evidence. The integration controller then releases the original job. Only after that does kube-scheduler place Pods. That model explains why one word—pending—can describe several unrelated waits. It also explains why Kueue’s hardest bugs are not heap-ordering mistakes. They are failures of coherence among object generations, resource calculations, reservations, controller transitions, and the telemetry used to observe them. The practical rule is the same one the architecture suggests: > First prove admission. Then debug placement. Then debug application progress. Keeping those boundaries visible is what turns a silent, suspended AI job from a mystery into a finite control-plane investigation. ## Frequently asked questions ### What is Kueue? Kueue is a Kubernetes-native system for queueing workloads and deciding when they may start. It admits whole Jobs or other workload groups against shared quotas and policies before their Pods enter ordinary Kubernetes scheduling. ### Does Kueue replace kube-scheduler? No. Kueue decides whether a workload has permission and quota to begin; kube-scheduler still binds the released Pods to nodes. Keeping those stages separate is the key to diagnosing whether a job is waiting on admission policy or physical placement. ### Why is Kueue useful for AI and machine-learning workloads? Distributed training and batch inference often need several scarce accelerators at once. Kueue can hold the workload until the complete request is admissible, apply team quotas and borrowing rules, and avoid releasing a partial gang of workers that cannot do useful work. ### What is the difference between `QuotaReserved=True` and `Admitted=True`? `QuotaReserved=True` means Kueue has assigned quota from a ClusterQueue. `Admitted=True` means the reservation and any required admission checks are ready, so the integration controller may release the underlying job. A workload can therefore hold quota without yet being allowed to run. ### Where is Kueue used? Public integrations and engineering reports show Kueue being used with IBM Research's Vela platform, GKE TPU Multislice and JobSet, KubeRay, and AWS SageMaker HyperPod. These examples demonstrate concrete patterns for governing GPU and TPU workloads; they do not, by themselves, establish comparative performance or total adoption. ## Sources - [Kueue concepts and APIs](https://kueue.sigs.k8s.io/docs/concepts/?ref=hiteshsharma.dev) - [Kueue admission](https://kueue.sigs.k8s.io/v0.20/docs/concepts/admission/?ref=hiteshsharma.dev) - [Kueue ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster%5Fqueue/?ref=hiteshsharma.dev) - [Kueue all-or-nothing scheduling](https://kueue.sigs.k8s.io/docs/concepts/all%5For%5Fnothing/?ref=hiteshsharma.dev) - [Kueue Topology-Aware Scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology%5Faware%5Fscheduling/?ref=hiteshsharma.dev) - [Kueue fair sharing](https://kueue.sigs.k8s.io/docs/concepts/fair%5Fsharing/?ref=hiteshsharma.dev) - [Kueue v0.20.2 release notes](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.20.2?ref=hiteshsharma.dev) - [Kueue v0.20.2 scheduler source](https://github.com/kubernetes-sigs/kueue/blob/v0.20.2/pkg/scheduler/scheduler.go?ref=hiteshsharma.dev) - [Kueue v0.20.2 workload controller source](https://github.com/kubernetes-sigs/kueue/blob/v0.20.2/pkg/controller/core/workload%5Fcontroller.go?ref=hiteshsharma.dev) - [Kubernetes: tracing Kueue admission with Headlamp](https://kubernetes.io/blog/2026/10/14/tracing-kueue-admission-with-headlamp/?ref=hiteshsharma.dev) - [IBM Vela and Kueue engineering report](https://developers.redhat.com/articles/2025/05/22/improve-gpu-utilization-kueue-openshift-ai?ref=hiteshsharma.dev) - [GKE TPU Multislice with JobSet and Kueue](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/tpu-multislice-kueue?ref=hiteshsharma.dev) - [KubeRay integration with Kueue](https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/kueue.html?ref=hiteshsharma.dev) - [Kueue ecosystem integrations](https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/v0.19/docs/ecosystem/%5Findex.md?ref=hiteshsharma.dev) - [AWS SageMaker HyperPod task governance](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance.html?ref=hiteshsharma.dev)