The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes. The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks. 405B model is the most capable from the Llama 3.1 family. This model is served in FP8 closely matching reference implementation.
Llama 3.1 405B Instruct is available via Cyfuture AI' serverless API, where you pay per token. There are several ways to call the Cyfuture AI API, including Cyfuture AI' Python client, the REST API, or OpenAI's Python client.
See below for easy generation of calls and a description of the raw REST API for making API requests. See the Querying text models docs for details.
Generate a model response using the chat endpoint of llama-v3p1-405b-instruct. API reference
import requests
import json
url = "https://api.cyfuture.ai/aiapi/inferencing/response"
payload = {
"model": "Model Name",
"max_tokens": 16384,
"top_p": 1,
"top_k": 40,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.6,
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
]
}
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": "Bearer <API_KEY>"
}
requests.request("POST", url, headers=headers, data=json.dumps(payload))await fetch("https://api.cyfuture.ai/aiapi/inferencing/response", {
method: "POST",
headers: {
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": "Bearer <API_KEY>"
},
body: JSON.stringify({
model: ""Model Name"",
max_tokens: 16384,
top_p: 1,
top_k: 40,
presence_penalty: 0,
frequency_penalty: 0,
temperature: 0.6,
messages: [
{
role: "user",
content: "Hello, how are you?"
}
]
})
});URI uri = URI.create("https://api.cyfuture.ai/aiapi/inferencing/response");
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(uri)
.header("Accept", "application/json")
.header("Content-Type", "application/json")
.header("Authorization", "Bearer <API_KEY>")
.POST(HttpRequest.BodyPublishers.ofString("""{
"model": ""Model Name"",
"max_tokens": 16384,
"top_p": 1,
"top_k": 40,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.6,
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
]
}"""))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
package main
import (
"bytes"
"net/http"
"fmt"
)
apiUrl := "https://api.cyfuture.ai/aiapi/inferencing/response"
var jsonData = []byte(`{
"model": "Model Name",
"max_tokens": 16384,
"top_p": 1,
"top_k": 40,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.6,
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
]
}`)
req, err := http.NewRequest(POST, apiUrl, bytes.NewBuffer(jsonData))
req.Header.Set("Accept", "application/json")
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer <API_KEY>")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)curl --request POST \
--url https://api.cyfuture.ai/aiapi/inferencing/response \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <API_KEY>' \
--data '{
"model": "Model Name",
"max_tokens": 16384,
"top_p": 1,
"top_k": 40,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.6,
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
]
}'Llama 3.1 405B Instruct can be fine-tuned on your data to create a model with better response quality. Cyfuture AI uses low-rank adaptation (LoRA) to train a model that can be served efficiently at inference time.
See the Fine-tuning guide for details.
On-demand deployments allow you to use Llama 3.1 405B Instruct on dedicated GPUs with Cyfuture AI' high-performance serving stack with high reliability and no rate limits.
See the On-demand deployments guide for details.
