Deploying Mistral AI Codestral 22B on Azure AI Foundry

Azure AI Foundry offers two deployment paths for Codestral 22B. The Serverless API functions as the preferred option because it supports the widest range of capabilities and deployment types. You use this path to call models by name without the need to create a dedicated Serverless API or managed compute deployment. Managed compute is currently in public preview and provides a managed GPU platform-as-a-service for open-source and custom-weight models. Microsoft manages the accelerators, provisions the hardware, and keeps the runtime patched for these managed deployments. You access these deployments through the unified Foundry project endpoint using the same authentication and networking as pay-per-token deployments. If you only need to try a supported model, you can skip deployment entirely by using instant access. Serverless API deployments support Global Standard and Data Zone Standard deployment types. These options control where data processes and how you pay. Users can choose regional or global processing to meet compliance requirements. Microsoft Entra ID provides keyless authentication for these deployments. Users can access the model through the Foundry portal or via the Azure CLI and Bicep. Managed compute deployments are served on the unified Foundry project endpoint and support various accelerator families. These families include the A100 (80 GB) or H100 (80 GB) accelerators. Managed compute billing is hourly per accelerator SKU.

Mistral AI trained Codestral 22B on a diverse dataset of over 80 programming languages. The model uses a decoder-only transformer architecture with 22 billion parameters. It includes a fill-in-the-middle mechanism that allows users to define a starting point and an ending point to generate code in between. The 32,000 token context window provides a larger capacity for long-range code generation than several other models in the same category. In the HumanEval benchmark for Python, Codestral 22B achieves an 81.1% pass rate. It also delivers a 78.2% success rate on the MBPP benchmark. In the CruxEval-O benchmark for Python output prediction, Codestral 22B achieved a 51.3% pass rate, which exceeds the 47.3% achieved by CodeLlama 70B. The 22B parameter model processes 80+ programming languages and uses a 32,000 token context window to provide better performance for developers who need to generate code across varying complexity levels. Codestral 22B performs better at algorithmic tasks than it does at handling complex business logic or domain-specific complexity. This model fails to maintain consistency across module boundaries when tasks require multi-file coordination. API calls consistently take under 1 second for typical coding tasks. In the Spider benchmark for SQL, Codestral 22B achieved a 63.5% success rate. This result outperformed the 37.0% achieved by CodeLlama 70B. The model also provides strong performance in languages like Python, Java, C, C++, JavaScript, and Bash.

Teams fine-tune Mistral models on Azure AI Foundry to optimize performance for specific tasks. Users select a model from the Azure AI Foundry portal model catalog and click fine-tune. You must select a project and provide training data to begin the process. For the best results, you should experiment with learning rate multipliers between 0.5 and 2.0. You also need to define the batch size for the training session. If you set the batch size to -1, the system calculates the value as 0.2% of the training set examples with a maximum of 256. You enter a name for your fine-tuned model and add optional tags. You also select the training and validation files. If your files reside in a credential less datastore, you must allow workspace managed identity access to the datastore. You know that model performance depends on the quality of the training data. Fine-tuning provides value by enabling customization and optimization for specific tasks. It can lead to improved performance and cost efficiency. To use the pay-as-you-go model fine-tune offering, your workspace must belong to the East US 2 region. You also need to subscribe your project to the particular offering from the Azure Marketplace. Once you sign up the project, subsequent fine-tuning of the same offering in the same project does not require subscribing again. You can use both the graphical user interface and the SDKs or CLI for fine-tuning. You can also deploy the model in your own application, in the playground, or in prompt flow and other environments. Will the integration of larger context windows eventually resolve the performance degradation seen beyond 100,000 tokens?