Description

# Azure AI Inference client library for Python Use the Inference client library (in preview) to: * Authenticate against the service * Get information about the AI model * Do chat completions * Get text embeddings * Get image embeddings The Inference client library supports AI models deployed to the following services: * [GitHub Models](https://github.com/marketplace/models) - Free-tier endpoint for AI models from different providers * Serverless API endpoints and Managed Compute endpoints - AI models from different providers deployed from [Azure AI Foundry](https://ai.azure.com). See [Overview: Deploy models, flows, and web apps with Azure AI Foundry](https://learn.microsoft.com/azure/ai-studio/concepts/deployments-overview). * Azure OpenAI Service - OpenAI models deployed from [Azure AI Foundry](https://oai.azure.com/). See [What is Azure OpenAI Service?](https://learn.microsoft.com/azure/ai-services/openai/overview). Although we recommend you use the official [OpenAI client library](https://pypi.org/project/openai/) in your production code for this service, you can use the Azure AI Inference client library to easily compare the performance of OpenAI models to other models, using the same client library and Python code. The Inference client library makes services calls using REST API version `2024-05-01-preview`, as documented in [Azure AI Model Inference API](https://aka.ms/azureai/modelinference). [Product documentation](https://aka.ms/aiservices/inference) | [Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-inference/samples) | [API reference documentation](https://aka.ms/azsdk/azure-ai-inference/python/reference) | [Package (Pypi)](https://aka.ms/azsdk/azure-ai-inference/python/package) | [SDK source code](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-inference/azure/ai/inference) ## Reporting issues To report an issue with the client library, or request additional features, please open a GitHub issue [here](https://github.com/Azure/azure-sdk-for-python/issues). Mention the package name "azure-ai-inference" in the title or content. ## Getting started ### Prerequisites * [Python 3.8](https://www.python.org/) or later installed, including [pip](https://pip.pypa.io/en/stable/). * For GitHub models * The AI model name, such as "gpt-4o" or "mistral-large" * A GitHub personal access token. [Create one here](https://github.com/settings/tokens). You do not need to give any permissions to the token. The token is a string that starts with `github_pat_`. * For Serverless API endpoints or Managed Compute endpoints * An [Azure subscription](https://azure.microsoft.com/free). * An [AI Model from the catalog](https://ai.azure.com/explore/models) deployed through Azure AI Foundry. * The endpoint URL of your model, in of the form `https://<your-host-name>.<your-azure-region>.models.ai.azure.com`, where `your-host-name` is your unique model deployment host name and `your-azure-region` is the Azure region where the model is deployed (e.g. `eastus2`). * Depending on your authentication preference, you either need an API key to authenticate against the service, or Entra ID credentials. * For Azure OpenAI (AOAI) service * An [Azure subscription](https://azure.microsoft.com/free). * An [OpenAI Model from the catalog](https://oai.azure.com/resource/models) deployed through Azure AI Foundry. * The endpoint URL of your model, in the form `https://<your-resouce-name>.openai.azure.com/openai/deployments/<your-deployment-name>`, where `your-resource-name` is your globally unique AOAI resource name, and `your-deployment-name` is your AI Model deployment name. * Depending on your authentication preference, you either need an API key to authenticate against the service, or Entra ID credentials. * An api-version. Latest preview or GA version listed in the `Data plane - inference` row in [the API Specs table](https://aka.ms/azsdk/azure-ai-inference/azure-openai-api-versions). At the time of writing, latest GA version was "2024-06-01". ### Install the package To install the Azure AI Inferencing package use the following command: ```bash pip install azure-ai-inference ``` To update an existing installation of the package, use: ```bash pip install --upgrade azure-ai-inference ``` If you want to install Azure AI Inferencing package with support for OpenTelemetry based tracing, use the following command: ```bash pip install azure-ai-inference[opentelemetry] ``` ## Key concepts ### Create and authenticate a client directly, using API key or GitHub token The package includes two clients `ChatCompletionsClient` and `EmbeddingsClient`<!-- and `ImageGenerationClients`-->. Both can be created in the similar manner. For example, assuming `endpoint`, `key` and `github_token` are strings holding your endpoint URL, API key or GitHub token, this Python code will create and authenticate a synchronous `ChatCompletionsClient`: ```python from azure.ai.inf