Groq

Groq provides ultra-fast inference using their custom LPU hardware.

Basic Example

client<llm> MyClient {
    provider openai-generic
    options {
        base_url "https://api.groq.com/openai/v1"
        api_key env.GROQ_API_KEY
        model "llama-3-groq-70b-tool-use"
    }
}

BAML-Specific Options

Option	Type	Default	Description
base_url	string	-	https://api.groq.com/openai/v1
api_key	string	env.GROQ_API_KEY	API key
model	string	-	Model identifier

Available Models

Model	Description
llama-3-groq-70b-tool-use	Llama 3 70B optimized for tool use
llama-3.1-70b-versatile	Llama 3.1 70B versatile
llama-3.1-8b-instant	Llama 3.1 8B fast inference
mixtral-8x7b-32768	Mixtral 8x7B
gemma-7b-it	Gemma 7B instruction-tuned

See the Groq documentation for the complete list.

See Also

• OpenAI Generic
• Together AI
• Client Overview