Skip to main content

Fallback

The fallback provider adds resilience by trying multiple clients in sequence.

Basic Example​

client<llm> SuperDuperClient {
    provider fallback
    options {
        strategy [
            ClientA
            ClientB
            ClientC
        ]
    }
}

If ClientA fails, it tries ClientB. If that fails, it tries ClientC.

Options​

OptionTypeRequiredDescription
strategyList[string]YesClient names to try in order

With Retry Policy​

A retry policy on the fallback client tests the fallback after the entire strategy has failed.

client<llm> SuperDuperClient {
    provider fallback
    retry_policy MyRetryPolicy
    options {
        strategy [
            ClientA
            ClientB
            ClientC
        ]
    }
}

Nesting Fallbacks​

Fallbacks can be nested inside each other:

client<llm> SuperDuperClient {
    provider fallback
    options {
        strategy [
            ClientA
            ClientB
            ClientC
        ]
    }
}

client<llm> MegaClient {
    provider fallback
    options {
        strategy [
            SuperDuperClient
            ClientD
        ]
    }
}

Calling MegaClient:

  1. Tries SuperDuperClient (which tries A → B → C)
  2. If all fail, tries ClientD

See Also​