PAIR routing paths and capability direction

[vonargo]

Love what y'all are doing and appreciate the opportunity to collaborate!

I just did pull request #64 and I think I understand the direction here, but want to check before going further against feature #24.

My read:

• PAIR currently routes a request based mainly on which upstream service has the requested model
• An upstream service might be a raw vLLM server, or something like Dynamo or llm-d managing multiple workers behind one endpoint
• The proposed capability work could let PAIR choose between those upstream services based on fit, not just model name
• PAIR would still route one request to one upstream service
• PAIR would not manage the workers, KV cache, GPU placement, or other inference scheduling behind that service

A few questions:

• Is that the direction you all have in mind for capability routing?
• Should capabilities describe the upstream service, the models it exposes, or both?
• Should PAIR infer requirements from the request, accept them explicitly from the client, or eventually support both?
• Do you see model name remaining the primary selector with capabilities as filters, or could capability fit become the primary selector?
• Is the intended boundary that PAIR chooses the right upstream service and systems like Dynamo or llm-d handle everything behind it?

[Noah-Tervalon-Nvidia]

@NV-sschneider can answer this better, but we are having some discussions about exactly what direction we want to support here in terms of other styles of upstream resources (ex. k8s clusters).
On the idea of routing:

• We do plan to keep the boundary of choosing the right upstream and being a passthrough.
• We plan to update routing to have a handful more options rather than the naive version right now. We are working on these changes right now, but the goal is eventually to support things like kv cache, throughput measurement (effectively how strong is a given machine), queue depth, and ideally measure how large a workload might be and do some prediction, cost, context size, throughput etc. There are a handful of other ideas that @sherief-nv is thinking about.
• model name will be a determining factor for now until we get a model router for requests and can drop it to optional, we have no current stance on developing that as part of this project's scheduler, or using an external model router (or potentially letting users choose a model router). This space is just too young for us to have a defined opinion yet.

[NV-sschneider]

Agree with all the points above.

I think in general, we will want some fuzzier routing, but ultimately be true to the upstream requester. However, the user should be the one to choose how "loose" we want the routing to be. For example, one of the first things i'd love to see is routing across different quants.

[vonargo]

This helps a lot!

What I see emerging is:

• [Feature]: Adopt arbitrary OpenAI-compatible backends with endpoint manifests, model aliases, and deterministic capability routing #24 describes what an upstream can do
• [Feature]: External / heterogeneous scheduling hooks and richer placement signals #7 helps determine how good a fit that upstream is for a request
• PAIR uses that context to choose one upstream
• Whatever is behind that upstream owns the actual inference scheduling (Dynamo, llm-d, etc.)

This architecture makes the endpoint proposal in #24 more important as more generic endpoints come in. Two endpoints can expose the same model but differ pretty substantially in context, quantization, modality, throughput, cost, or what is actually behind them.

I also like the idea of the user deciding how loose that matching can be. Maybe that eventually becomes something a client or higher-level router can express through a standardized contract. Exact model could remain the safe default while allowing things like alternate quants or eventually other compatible choices when the user wants that.

I’d be happy to help with a piece of this. What do you think about me staying on the endpoint contract and eligibility side before touching the more advanced scheduler work? Is there another piece of #24 or #7 that would be useful for me to pick up next?

Thanks to @darkmatter2222 and @guydax for creating those issues and asking some good questions that helped drive this synthesis!